2024-03-06T07:10:47.149 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:10:47.149 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075 --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/orig.config.yaml 2024-03-06T07:10:47.193 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:10:47.284 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583075' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArYvpsmBmeK/TBDiqVWIuoAATGQa8GFzU77A+6ELgW99JRI/oLRgHOskkANHygDldGYO6XxqDib3DK/40vuNkg= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPfyFDKab2OwyrP2BRUX2aSe+lz0hRLCl1Bj+JetLDX2qnn7gI3+3BuLidL6e2F9CEQCxVSFwMDe8F0xW2Qays= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:10:47.285 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:10:47.285 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:10:47.285 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:10:47.287 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:10:47.310 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:10:47.310 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:10:47.310 INFO:teuthology.packaging:ref: None 2024-03-06T07:10:47.310 INFO:teuthology.packaging:tag: None 2024-03-06T07:10:47.311 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:10:47.311 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:10:47.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:10:47.565 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:10:47.566 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:10:47.571 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:10:47.571 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:10:47.584 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:10:47.600 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:10:47.605 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:10:47.665 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:46.256764', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArYvpsmBmeK/TBDiqVWIuoAATGQa8GFzU77A+6ELgW99JRI/oLRgHOskkANHygDldGYO6XxqDib3DK/40vuNkg='} 2024-03-06T07:10:47.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:46.255259', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPfyFDKab2OwyrP2BRUX2aSe+lz0hRLCl1Bj+JetLDX2qnn7gI3+3BuLidL6e2F9CEQCxVSFwMDe8F0xW2Qays='} 2024-03-06T07:10:47.686 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:10:47.693 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:10:47.693 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:10:47.693 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:10:47.756 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9fbc6bbf70>, signals=[15]) 2024-03-06T07:10:47.757 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:10:47.763 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:10:47.763 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:10:47.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:47.859 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:10:47.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:47.946 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:10:47.953 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-03-06T07:10:47.969 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-03-06T07:10:47.970 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-03-06T07:10:48.026 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-03-06T07:10:48.026 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-03-06T07:10:48.026 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-03-06T07:10:48.026 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:48.027 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:48.028 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-03-06T07:10:48.048 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-03-06T07:10:48.065 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-03-06T07:10:48.066 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="9" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="9" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:48.123 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:48.123 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:48.123 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:48.123 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:48.123 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-03-06T07:10:48.148 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:10:48.158 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:10:48.164 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:10:48.164 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:48.166 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:48.182 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:10:48.188 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:10:48.188 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:48.222 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:48.283 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:10:48.337 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:10:48.352 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-06T07:10:48.352 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-06T07:10:48.352 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-06T07:10:48.352 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:48.353 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:48.369 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:10:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:48.372 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:10:48.681 DEBUG:teuthology.task.kernel:sha1 for client.0 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:48.681 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:10:48.681 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-06T07:10:48.698 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:48.698 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:48.699 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:10:48.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum upgrade -y linux-firmware 2024-03-06T07:10:48.712 DEBUG:teuthology.task.kernel:sha1 for client.1 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:48.712 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:10:48.712 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-03-06T07:10:48.729 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:48.730 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:48.730 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:10:48.730 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum upgrade -y linux-firmware 2024-03-06T07:10:50.811 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-06T07:10:50.916 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:10:52.254 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-06T07:10:52.542 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-06T07:10:53.782 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-03-06T07:10:54.944 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-06T07:10:55.583 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-06T07:10:57.987 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 19 MB 00:02 2024-03-06T07:11:11.608 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 498 kB/s | 6.1 MB 00:12 2024-03-06T07:11:12.729 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 558 kB/s | 6.1 MB 00:11 2024-03-06T07:11:15.268 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 5.9 kB/s | 15 kB 00:02 2024-03-06T07:11:16.554 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 5.6 kB/s | 15 kB 00:02 2024-03-06T07:11:18.892 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-03-06T07:11:23.840 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-06T07:11:24.025 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:01 ago on Wed 06 Mar 2024 07:11:23 AM UTC. 2024-03-06T07:11:26.194 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-06T07:11:26.203 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-06T07:11:26.204 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 1 Package 2024-03-06T07:11:26.204 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:11:26.204 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 375 M 2024-03-06T07:11:26.204 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-06T07:11:38.924 INFO:teuthology.orchestra.run.smithi135.stdout:linux-firmware-20240219-143.el9.noarch.rpm 29 MB/s | 375 MB 00:12 2024-03-06T07:11:38.925 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:11:38.925 INFO:teuthology.orchestra.run.smithi135.stdout:Total 29 MB/s | 375 MB 00:12 2024-03-06T07:11:38.925 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-06T07:11:39.018 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-06T07:11:39.018 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-06T07:11:41.417 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-06T07:11:41.417 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-06T07:11:47.835 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-06T07:11:48.146 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:11:48.237 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:11:49.567 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:11:49.568 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:11:49.985 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:11:49.986 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:11:49.986 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-03-06T07:11:49.986 INFO:teuthology.orchestra.run.smithi135.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:11:49.986 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:11:49.986 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-06T07:11:50.298 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:11:50.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:11:50.668 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:11:50.668 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:11:50.671 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi135.front.sepia.ceph.com to complete... 2024-03-06T07:13:17.930 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 172 kB/s | 21 MB 02:02 2024-03-06T07:13:22.854 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-06T07:13:25.193 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-06T07:13:25.204 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-06T07:13:25.205 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 1 Package 2024-03-06T07:13:25.205 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:13:25.205 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 375 M 2024-03-06T07:13:25.205 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-06T07:13:36.563 INFO:teuthology.orchestra.run.smithi097.stdout:linux-firmware-20240219-143.el9.noarch.rpm 33 MB/s | 375 MB 00:11 2024-03-06T07:13:36.564 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:13:36.564 INFO:teuthology.orchestra.run.smithi097.stdout:Total 33 MB/s | 375 MB 00:11 2024-03-06T07:13:36.565 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-06T07:13:36.654 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-06T07:13:36.654 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-06T07:13:39.036 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-06T07:13:39.036 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-06T07:13:45.342 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-06T07:13:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:13:45.705 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:13:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:13:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:13:47.401 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:13:47.401 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:13:47.401 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-03-06T07:13:47.401 INFO:teuthology.orchestra.run.smithi097.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:13:47.401 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:13:47.401 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-06T07:13:47.699 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:13:47.699 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:13:47.852 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:13:47.852 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:13:47.855 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi097.front.sepia.ceph.com to complete... 2024-03-06T07:15:10.937 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06 07:15:10 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:15:10.938 INFO:teuthology.orchestra.run.smithi135.stderr:FINISHED --2024-03-06 07:15:10-- 2024-03-06T07:15:10.938 INFO:teuthology.orchestra.run.smithi135.stderr:Total wall clock time: 3m 20s 2024-03-06T07:15:10.938 INFO:teuthology.orchestra.run.smithi135.stderr:Downloaded: 1 files, 320M in 3m 20s (1.60 MB/s) 2024-03-06T07:15:10.945 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:15:10.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:15:12.881 INFO:teuthology.orchestra.run.smithi135.stdout:Verifying... ######################################## 2024-03-06T07:15:13.023 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing... ######################################## 2024-03-06T07:15:13.101 INFO:teuthology.orchestra.run.smithi135.stdout:Updating / installing... 2024-03-06T07:15:20.623 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:15:21.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06 07:15:21 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:15:21.842 INFO:teuthology.orchestra.run.smithi097.stderr:FINISHED --2024-03-06 07:15:21-- 2024-03-06T07:15:21.842 INFO:teuthology.orchestra.run.smithi097.stderr:Total wall clock time: 1m 34s 2024-03-06T07:15:21.843 INFO:teuthology.orchestra.run.smithi097.stderr:Downloaded: 1 files, 320M in 1m 33s (3.43 MB/s) 2024-03-06T07:15:21.846 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:15:21.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:15:23.772 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying... ######################################## 2024-03-06T07:15:23.914 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing... ######################################## 2024-03-06T07:15:23.992 INFO:teuthology.orchestra.run.smithi097.stdout:Updating / installing... 2024-03-06T07:15:31.498 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:15:35.617 INFO:teuthology.orchestra.run.smithi135.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:45.801 INFO:teuthology.orchestra.run.smithi097.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:58.158 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:15:58.158 DEBUG:teuthology.orchestra.run.smithi135:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:58.237 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:58.237 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:15:58.237 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:15:58.237 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:15:58.237 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:15:58.238 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:15:58.240 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:15:58.241 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:15:58.242 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:15:58.243 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:15:58.244 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:15:58.245 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:15:58.246 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:15:58.247 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:15:58.248 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:15:58.249 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:15:58.250 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:15:58.251 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:15:58.252 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:15:58.253 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:15:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:15:58.255 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:15:58.256 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:15:58.257 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:15:58.258 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:15:58.259 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:15:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:15:58.261 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:15:58.262 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:15:58.263 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:15:58.265 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:15:58.266 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:15:58.267 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:15:58.268 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:15:58.269 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:15:58.270 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:15:58.271 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:15:58.272 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:15:58.273 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:15:58.274 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:15:58.275 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:15:58.276 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:15:58.277 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:15:58.278 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:15:58.279 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:15:58.280 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:15:58.281 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:15:58.282 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:15:58.283 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:15:58.284 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:15:58.285 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:15:58.287 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:15:58.288 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:15:58.290 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:15:58.291 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:15:58.292 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:15:58.293 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:15:58.294 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:15:58.295 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:15:58.296 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:15:58.297 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:15:58.298 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:15:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:15:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:15:58.301 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:15:58.302 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:15:58.303 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:15:58.304 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:15:58.305 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:15:58.306 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:15:58.307 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:15:58.308 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:15:58.309 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:15:58.310 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:15:58.311 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:15:58.312 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:15:58.313 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:15:58.314 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:15:58.316 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:15:58.317 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:15:58.318 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:15:58.319 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:15:58.320 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:15:58.321 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:15:58.322 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:15:58.323 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:15:58.324 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:15:58.325 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:15:58.326 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:15:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:15:58.328 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:15:58.329 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:15:58.330 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:15:58.331 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:15:58.332 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:15:58.333 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:15:58.334 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:15:58.335 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:15:58.336 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:15:58.337 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:15:58.338 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:15:58.339 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:15:58.340 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:15:58.341 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:15:58.342 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:15:58.343 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:15:58.344 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:15:58.345 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:15:58.346 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:15:58.347 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:15:58.348 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:15:58.349 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:15:58.350 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:15:58.351 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:15:58.352 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:15:58.353 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:15:58.354 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:15:58.355 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:15:58.356 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:15:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:15:58.358 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:15:58.359 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:15:58.360 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:15:58.361 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:15:58.362 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:15:58.363 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:15:58.364 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:15:58.365 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:15:58.366 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:58.370 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:58.370 DEBUG:teuthology.orchestra.run.smithi135:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout:fi 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout: fi 2024-03-06T07:15:58.392 INFO:teuthology.orchestra.run.smithi135.stdout:done 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:fi 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:58.393 INFO:teuthology.orchestra.run.smithi135.stdout:fi 2024-03-06T07:15:58.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2024-03-06T07:15:58.471 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2024-03-06T07:15:58.471 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2024-03-06T07:15:58.471 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.06 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 68.el9 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Group : Unspecified 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 8264254 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : builder@centos.org 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2024-03-06T07:15:58.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:15:58.473 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:15:58.473 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:15:58.473 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:15:58.473 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2024-03-06T07:15:58.473 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:15:58.473 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:15:58.473 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:15:58.473 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:58.473 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:15:59.065 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2024-03-06T07:16:00.496 INFO:teuthology.orchestra.run.smithi135.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:16:00.547 INFO:teuthology.orchestra.run.smithi135.stderr:done 2024-03-06T07:16:00.549 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-06T07:16:00.564 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.PBzwrMIeEs 2024-03-06T07:16:00.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PBzwrMIeEs 2024-03-06T07:16:00.640 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.PBzwrMIeEs 2024-03-06T07:16:00.776 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.PBzwrMIeEs is 6KB 2024-03-06T07:16:00.826 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.PBzwrMIeEs 2024-03-06T07:16:00.840 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:16:00.909 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:16:00.910 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:16:00.910 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:16:00.910 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:16:00.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:01.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2024-03-06T07:16:08.339 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:16:08.340 DEBUG:teuthology.orchestra.run.smithi097:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:16:08.417 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:16:08.418 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:16:08.419 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:16:08.420 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:16:08.421 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:16:08.422 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:16:08.423 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:16:08.424 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:16:08.425 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:16:08.426 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:16:08.427 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:16:08.428 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:16:08.429 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:16:08.430 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:16:08.430 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:16:08.430 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:16:08.430 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:16:08.430 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:16:08.431 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:16:08.432 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:16:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:16:08.434 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:16:08.435 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:16:08.436 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:16:08.437 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:16:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:16:08.439 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:16:08.440 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:16:08.440 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:16:08.440 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:16:08.440 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:16:08.441 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:16:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:16:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:16:08.444 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:16:08.445 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:16:08.446 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:16:08.447 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:16:08.448 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:16:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:16:08.450 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:16:08.450 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:16:08.451 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:16:08.452 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:16:08.454 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:16:08.455 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:16:08.456 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:16:08.457 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:16:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:16:08.459 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:16:08.460 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:16:08.461 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:16:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:16:08.463 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:16:08.464 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:16:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:16:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:16:08.467 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:16:08.468 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:16:08.469 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:16:08.470 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:16:08.471 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:16:08.472 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:16:08.473 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:16:08.474 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:16:08.475 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:16:08.476 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:16:08.480 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:16:08.481 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:16:08.482 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:16:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:16:08.485 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:16:08.486 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:16:08.487 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:16:08.488 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:16:08.489 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:16:08.490 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:16:08.491 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:16:08.492 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:16:08.493 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:16:08.494 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:16:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:16:08.496 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:16:08.497 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:16:08.498 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:16:08.499 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:16:08.500 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:16:08.501 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:16:08.502 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:16:08.503 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:16:08.504 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:16:08.505 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:16:08.506 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:16:08.507 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:16:08.508 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:16:08.510 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:16:08.511 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:16:08.512 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:16:08.513 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:16:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:16:08.515 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:16:08.516 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:16:08.517 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:16:08.518 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:16:08.519 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:16:08.520 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:16:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:16:08.522 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:16:08.523 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:16:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:16:08.525 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:16:08.526 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:16:08.527 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:16:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:16:08.529 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:16:08.530 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:16:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:16:08.532 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:16:08.533 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:16:08.534 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:16:08.535 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:16:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:16:08.537 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:16:08.538 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:16:08.539 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:16:08.540 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:16:08.541 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:16:08.542 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:16:08.543 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:16:08.544 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:16:08.545 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:16:08.546 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:16:08.547 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:16:08.548 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:16:08.549 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:16:08.549 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:16:08.549 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:16:08.549 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:16:08.549 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:16:08.549 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:16:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:16:08.551 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:16:08.552 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:16:08.553 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:16:08.554 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:16:08.554 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:16:08.554 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:16:08.554 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:16:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:16:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:16:08.558 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:16:08.559 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:16:08.560 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:16:08.561 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:16:08.562 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:16:08.563 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:16:08.564 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:16:08.565 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:16:08.569 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:08.569 DEBUG:teuthology.orchestra.run.smithi097:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:16:08.591 INFO:teuthology.orchestra.run.smithi097.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:16:08.591 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:16:08.591 INFO:teuthology.orchestra.run.smithi097.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:16:08.591 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-03-06T07:16:08.591 INFO:teuthology.orchestra.run.smithi097.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout: fi 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:done 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:08.592 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-03-06T07:16:08.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:16:08.668 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:16:08.669 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:16:08.670 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:16:08.670 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:08.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:16:09.255 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-03-06T07:16:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:16:10.735 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-03-06T07:16:10.740 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-06T07:16:10.755 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.nDUIC1QpfH 2024-03-06T07:16:10.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nDUIC1QpfH 2024-03-06T07:16:10.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.nDUIC1QpfH 2024-03-06T07:16:10.964 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.nDUIC1QpfH is 6KB 2024-03-06T07:16:11.013 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.nDUIC1QpfH 2024-03-06T07:16:11.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:16:11.100 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:16:11.100 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:16:11.100 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:16:11.100 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:16:11.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:11.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-03-06T07:16:31.100 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-06T07:16:31.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-06T07:16:31.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:16:41.268 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-06T07:16:41.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-06T07:16:41.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:16:59.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-03-06T07:17:08.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-06T07:17:08.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:11.904 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-03-06T07:17:23.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-06T07:17:23.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:31.111 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:17:40.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-06T07:17:40.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:43.761 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-03-06T07:17:44.347 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-06T07:17:44.347 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:17:44.347 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-03-06T07:17:44.385 INFO:teuthology.orchestra.run.smithi135.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:44.385 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:44.385 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:17:44.386 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:17:45.389 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:55.873 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-06T07:17:56.579 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-06T07:17:56.579 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:17:56.579 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-06T07:17:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:56.621 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:56.621 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:17:56.621 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:17:57.622 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:57.622 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:17:57.630 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:17:57.630 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:17:57.636 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:17:57.724 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:17:57.732 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:17:57.755 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:17:57.755 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:17:57.760 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:17:57.862 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:17:57.876 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:17:57.876 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:17:57.884 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:17:58.026 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:58.098 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:58.127 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:58.149 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:58.152 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:17:58.160 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:17:58.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:17:58.165 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:17:58.236 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:17:58.245 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:17:58.245 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:17:58.250 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:17:58.289 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:58.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:58.491 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:58.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:58.652 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:17:58.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:17:58.747 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:58.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:58.936 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:59.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:59.085 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:17:59.085 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:17:59.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-03-06T07:17:59.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-03-06T07:17:59.274 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:17:59.319 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:17:59.768 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:17:59.777 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:17:59.777 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:17:59.792 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:17:59.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-03-06T07:17:59.896 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-03-06T07:17:59.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-03-06T07:18:00.053 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-03-06T07:18:00.056 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:18:00.056 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-03-06T07:18:00.111 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-03-06T07:18:00.112 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-03-06T07:18:00.287 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-03-06T07:18:00.287 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:18:00.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:18:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:00.368 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:18:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:00.442 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:18:00.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:18:00.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:18:00.666 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:18:00.673 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:18:00.690 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:18:00.691 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:23:57.292 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2024-03-06T07:23:57.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-06T07:23:57.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:23:57.442 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-06T07:23:57.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-06T07:23:57.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-06T07:23:57.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:23:57.742 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-03-06T07:23:57.895 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-06T07:23:57.895 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:23:57.906 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:23:57.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:23:57.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:23:57.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:23:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:23:58.019 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:23:58.062 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:23:58.073 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:23:58.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-03-06T07:23:58.167 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-03-06T07:23:58.198 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-03-06T07:23:58.207 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-03-06T07:23:58.247 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:23:58.257 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:23:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:23:58.313 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:23:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:23:58.434 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:23:58.434 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-06T07:23:58.434 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.434 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.434 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.434 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.447 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:23:58.456 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:23:58.456 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-03-06T07:23:58.456 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.456 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.456 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.456 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:58.457 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:23:58.466 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:23:58.466 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:23:58.466 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:23:58.466 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:23:58.472 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:23:58.472 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:23:58.473 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:23:58.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:23:58.475 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:23:58.476 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:23:58.639 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:23:58.639 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:23:58.724 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:23:58.724 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:23:58.771 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:23:58.771 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:23:58.771 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:23:58.854 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:23:58.854 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:23:58.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:23:58.870 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:23:58.870 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:23:58.954 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:23:58.954 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:23:59.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:23:59.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:23:59.130 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:23:59.216 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2024-03-06T07:23:59.221 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2024-03-06T07:23:59.224 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-03-06T07:23:59.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-03-06T07:23:59.571 INFO:teuthology.orchestra.run.smithi135.stdout:74 files removed 2024-03-06T07:23:59.610 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:23:59.683 INFO:teuthology.orchestra.run.smithi097.stdout:74 files removed 2024-03-06T07:23:59.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 183 kB/s | 83 kB 00:00 2024-03-06T07:24:00.674 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 149 kB/s | 83 kB 00:00 2024-03-06T07:24:01.183 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-06T07:24:01.271 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-06T07:24:01.759 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-06T07:24:01.771 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-06T07:24:02.510 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-06T07:24:03.100 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 5.3 MB/s | 6.7 MB 00:01 2024-03-06T07:24:04.239 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-06T07:24:05.110 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-03-06T07:24:06.322 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-06T07:24:09.357 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-06T07:24:13.849 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.1 MB 00:00 2024-03-06T07:24:15.508 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 15 kB 00:00 2024-03-06T07:24:20.326 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 20 MB 00:04 2024-03-06T07:24:26.026 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-06T07:24:28.804 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:28.806 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:28.823 INFO:teuthology.orchestra.run.smithi097.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:24:28.824 INFO:teuthology.orchestra.run.smithi097.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:24:28.826 INFO:teuthology.orchestra.run.smithi097.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:24:28.827 INFO:teuthology.orchestra.run.smithi097.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:28.829 INFO:teuthology.orchestra.run.smithi097.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:28.832 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:24:28.833 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:24:28.920 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-06T07:24:28.952 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:24:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:24:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:24:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:24:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:24:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:24:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:24:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:24:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout:Install 150 Packages 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 26 Packages 2024-03-06T07:24:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:24:28.968 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 224 M 2024-03-06T07:24:28.968 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-06T07:24:29.986 INFO:teuthology.orchestra.run.smithi097.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-06T07:24:30.928 INFO:teuthology.orchestra.run.smithi097.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 945 kB/s | 888 kB 00:00 2024-03-06T07:24:31.137 INFO:teuthology.orchestra.run.smithi097.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 700 kB/s | 145 kB 00:00 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 817 kB/s | 19 MB 00:23 2024-03-06T07:24:32.362 INFO:teuthology.orchestra.run.smithi097.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.9 MB/s | 2.3 MB 00:01 2024-03-06T07:24:32.912 INFO:teuthology.orchestra.run.smithi097.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-03-06T07:24:33.362 INFO:teuthology.orchestra.run.smithi097.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.4 MB/s | 5.3 MB 00:03 2024-03-06T07:24:34.154 INFO:teuthology.orchestra.run.smithi097.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-03-06T07:24:35.829 INFO:teuthology.orchestra.run.smithi097.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 3.2 MB/s | 20 MB 00:06 2024-03-06T07:24:35.938 INFO:teuthology.orchestra.run.smithi097.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 234 kB/s | 25 kB 00:00 2024-03-06T07:24:36.500 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-06T07:24:36.780 INFO:teuthology.orchestra.run.smithi097.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 3.7 MB/s | 9.8 MB 00:02 2024-03-06T07:24:36.896 INFO:teuthology.orchestra.run.smithi097.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.9 MB/s | 220 kB 00:00 2024-03-06T07:24:37.013 INFO:teuthology.orchestra.run.smithi097.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 234 kB/s | 27 kB 00:00 2024-03-06T07:24:37.139 INFO:teuthology.orchestra.run.smithi097.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 252 kB/s | 31 kB 00:00 2024-03-06T07:24:37.456 INFO:teuthology.orchestra.run.smithi097.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 2.3 MB/s | 747 kB 00:00 2024-03-06T07:24:37.564 INFO:teuthology.orchestra.run.smithi097.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.5 MB/s | 163 kB 00:00 2024-03-06T07:24:37.681 INFO:teuthology.orchestra.run.smithi097.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.1 MB/s | 127 kB 00:00 2024-03-06T07:24:37.898 INFO:teuthology.orchestra.run.smithi097.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 2.3 MB/s | 501 kB 00:00 2024-03-06T07:24:38.234 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-06T07:24:40.035 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-06T07:24:40.157 INFO:teuthology.orchestra.run.smithi097.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 2.3 MB/s | 5.1 MB 00:02 2024-03-06T07:24:40.307 INFO:teuthology.orchestra.run.smithi097.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 302 kB/s | 45 kB 00:00 2024-03-06T07:24:40.416 INFO:teuthology.orchestra.run.smithi097.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.2 MB/s | 131 kB 00:00 2024-03-06T07:24:40.541 INFO:teuthology.orchestra.run.smithi097.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.3 MB/s | 162 kB 00:00 2024-03-06T07:24:40.741 INFO:teuthology.orchestra.run.smithi097.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.6 MB/s | 321 kB 00:00 2024-03-06T07:24:40.958 INFO:teuthology.orchestra.run.smithi097.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.3 MB/s | 298 kB 00:00 2024-03-06T07:24:41.075 INFO:teuthology.orchestra.run.smithi097.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 857 kB/s | 99 kB 00:00 2024-03-06T07:24:41.200 INFO:teuthology.orchestra.run.smithi097.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 687 kB/s | 85 kB 00:00 2024-03-06T07:24:43.076 INFO:teuthology.orchestra.run.smithi097.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.6 MB/s | 3.1 MB 00:01 2024-03-06T07:24:43.293 INFO:teuthology.orchestra.run.smithi097.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 791 kB/s | 171 kB 00:00 2024-03-06T07:24:43.560 INFO:teuthology.orchestra.run.smithi097.stdout:(28/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 1.6 MB/s | 17 MB 00:10 2024-03-06T07:24:43.585 INFO:teuthology.orchestra.run.smithi097.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 108 kB/s | 31 kB 00:00 2024-03-06T07:24:43.727 INFO:teuthology.orchestra.run.smithi097.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 833 kB/s | 138 kB 00:00 2024-03-06T07:24:44.885 INFO:teuthology.orchestra.run.smithi097.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.4 MB/s | 1.8 MB 00:01 2024-03-06T07:24:45.194 INFO:teuthology.orchestra.run.smithi097.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 808 kB/s | 248 kB 00:00 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi097.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 180 kB/s | 49 kB 00:00 2024-03-06T07:24:45.611 INFO:teuthology.orchestra.run.smithi097.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 109 kB/s | 15 kB 00:00 2024-03-06T07:24:45.760 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-06T07:24:45.937 INFO:teuthology.orchestra.run.smithi097.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 813 kB/s | 264 kB 00:00 2024-03-06T07:24:46.545 INFO:teuthology.orchestra.run.smithi097.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.2 MB/s | 748 kB 00:00 2024-03-06T07:24:46.654 INFO:teuthology.orchestra.run.smithi097.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 170 kB/s | 18 kB 00:00 2024-03-06T07:24:46.915 INFO:teuthology.orchestra.run.smithi097.stdout:(38/176): ceph-mgr-diskprediction-local-19.0.0- 2.3 MB/s | 7.4 MB 00:03 2024-03-06T07:24:47.421 INFO:teuthology.orchestra.run.smithi097.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 864 kB/s | 662 kB 00:00 2024-03-06T07:24:47.821 INFO:teuthology.orchestra.run.smithi097.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 920 kB 00:00 2024-03-06T07:24:47.855 INFO:teuthology.orchestra.run.smithi097.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 245 kB/s | 106 kB 00:00 2024-03-06T07:24:47.888 INFO:teuthology.orchestra.run.smithi097.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 310 kB/s | 20 kB 00:00 2024-03-06T07:24:48.022 INFO:teuthology.orchestra.run.smithi097.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 307 kB 00:00 2024-03-06T07:24:48.048 INFO:teuthology.orchestra.run.smithi097.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 200 kB/s | 32 kB 00:00 2024-03-06T07:24:48.440 INFO:teuthology.orchestra.run.smithi097.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 44 kB/s | 17 kB 00:00 2024-03-06T07:24:48.498 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:48.500 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:48.523 INFO:teuthology.orchestra.run.smithi135.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:24:48.524 INFO:teuthology.orchestra.run.smithi135.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:24:48.526 INFO:teuthology.orchestra.run.smithi135.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:24:48.527 INFO:teuthology.orchestra.run.smithi135.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:48.530 INFO:teuthology.orchestra.run.smithi135.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:48.534 INFO:teuthology.orchestra.run.smithi135.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:24:48.535 INFO:teuthology.orchestra.run.smithi135.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:24:48.626 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-06T07:24:48.653 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:24:48.658 INFO:teuthology.orchestra.run.smithi135.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi135.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:24:48.660 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:24:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi135.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:24:48.664 INFO:teuthology.orchestra.run.smithi135.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:24:48.666 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:24:48.667 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:24:48.668 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout:Installing weak dependencies: 2024-03-06T07:24:48.669 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout:Install 150 Packages 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 26 Packages 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:24:48.670 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 224 M 2024-03-06T07:24:48.671 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-06T07:24:48.681 INFO:teuthology.orchestra.run.smithi097.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 42 kB/s | 10 kB 00:00 2024-03-06T07:24:48.798 INFO:teuthology.orchestra.run.smithi097.stdout:(47/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.9 MB/s | 3.0 MB 00:00 2024-03-06T07:24:48.823 INFO:teuthology.orchestra.run.smithi097.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi097.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 918 kB/s | 44 kB 00:00 2024-03-06T07:24:49.008 INFO:teuthology.orchestra.run.smithi097.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-03-06T07:24:49.108 INFO:teuthology.orchestra.run.smithi097.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 2.2 MB/s | 578 kB 00:00 2024-03-06T07:24:49.133 INFO:teuthology.orchestra.run.smithi097.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 197 kB/s | 24 kB 00:00 2024-03-06T07:24:49.167 INFO:teuthology.orchestra.run.smithi097.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 804 kB/s | 46 kB 00:00 2024-03-06T07:24:49.251 INFO:teuthology.orchestra.run.smithi097.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-03-06T07:24:49.284 INFO:teuthology.orchestra.run.smithi097.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-06T07:24:49.360 INFO:teuthology.orchestra.run.smithi097.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 321 kB/s | 34 kB 00:00 2024-03-06T07:24:49.662 INFO:teuthology.orchestra.run.smithi135.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-06T07:24:50.201 INFO:teuthology.orchestra.run.smithi097.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.6 MB/s | 3.0 MB 00:00 2024-03-06T07:24:50.302 INFO:teuthology.orchestra.run.smithi097.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 4.7 MB/s | 4.7 MB 00:01 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi097.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi097.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-03-06T07:24:50.486 INFO:teuthology.orchestra.run.smithi097.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-06T07:24:50.611 INFO:teuthology.orchestra.run.smithi135.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 936 kB/s | 888 kB 00:00 2024-03-06T07:24:50.786 INFO:teuthology.orchestra.run.smithi097.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 18 MB/s | 5.7 MB 00:00 2024-03-06T07:24:50.812 INFO:teuthology.orchestra.run.smithi135.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 730 kB/s | 145 kB 00:00 2024-03-06T07:24:50.828 INFO:teuthology.orchestra.run.smithi097.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-03-06T07:24:50.853 INFO:teuthology.orchestra.run.smithi097.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-06T07:24:51.104 INFO:teuthology.orchestra.run.smithi097.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-06T07:24:51.138 INFO:teuthology.orchestra.run.smithi097.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 815 kB/s | 227 kB 00:00 2024-03-06T07:24:51.163 INFO:teuthology.orchestra.run.smithi097.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 428 kB/s | 24 kB 00:00 2024-03-06T07:24:51.180 INFO:teuthology.orchestra.run.smithi097.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 442 kB/s | 18 kB 00:00 2024-03-06T07:24:51.205 INFO:teuthology.orchestra.run.smithi097.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi097.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 651 kB/s | 32 kB 00:00 2024-03-06T07:24:51.429 INFO:teuthology.orchestra.run.smithi135.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 2.3 MB/s | 5.3 MB 00:02 2024-03-06T07:24:51.439 INFO:teuthology.orchestra.run.smithi097.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-06T07:24:51.473 INFO:teuthology.orchestra.run.smithi097.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 933 kB/s | 217 kB 00:00 2024-03-06T07:24:51.506 INFO:teuthology.orchestra.run.smithi097.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-06T07:24:51.532 INFO:teuthology.orchestra.run.smithi097.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-06T07:24:51.557 INFO:teuthology.orchestra.run.smithi097.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 838 kB/s | 41 kB 00:00 2024-03-06T07:24:51.591 INFO:teuthology.orchestra.run.smithi097.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-03-06T07:24:51.616 INFO:teuthology.orchestra.run.smithi097.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-06T07:24:51.649 INFO:teuthology.orchestra.run.smithi097.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-06T07:24:51.692 INFO:teuthology.orchestra.run.smithi097.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 650 kB/s | 47 kB 00:00 2024-03-06T07:24:51.717 INFO:teuthology.orchestra.run.smithi097.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 648 kB/s | 43 kB 00:00 2024-03-06T07:24:52.004 INFO:teuthology.orchestra.run.smithi135.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-03-06T07:24:52.217 INFO:teuthology.orchestra.run.smithi097.stdout:(81/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 2.9 MB/s | 47 MB 00:16 2024-03-06T07:24:52.359 INFO:teuthology.orchestra.run.smithi097.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 61 kB/s | 39 kB 00:00 2024-03-06T07:24:52.551 INFO:teuthology.orchestra.run.smithi097.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 128 kB/s | 41 kB 00:00 2024-03-06T07:24:52.612 INFO:teuthology.orchestra.run.smithi135.stdout:(6/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.3 MB/s | 2.3 MB 00:01 2024-03-06T07:24:52.802 INFO:teuthology.orchestra.run.smithi097.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-03-06T07:24:52.835 INFO:teuthology.orchestra.run.smithi097.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 455 kB/s | 215 kB 00:00 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi097.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 113 kB/s | 34 kB 00:00 2024-03-06T07:24:52.885 INFO:teuthology.orchestra.run.smithi097.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 539 kB/s | 43 kB 00:00 2024-03-06T07:24:52.911 INFO:teuthology.orchestra.run.smithi097.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-06T07:24:52.944 INFO:teuthology.orchestra.run.smithi097.stdout:(89/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 300 kB 00:00 2024-03-06T07:24:52.969 INFO:teuthology.orchestra.run.smithi097.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 87 kB 00:00 2024-03-06T07:24:52.995 INFO:teuthology.orchestra.run.smithi097.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 774 kB/s | 63 kB 00:00 2024-03-06T07:24:53.020 INFO:teuthology.orchestra.run.smithi097.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 597 kB/s | 44 kB 00:00 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi097.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.1 MB/s | 79 kB 00:00 2024-03-06T07:24:53.124 INFO:teuthology.orchestra.run.smithi097.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 186 kB 00:00 2024-03-06T07:24:53.157 INFO:teuthology.orchestra.run.smithi097.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 802 kB 00:00 2024-03-06T07:24:53.191 INFO:teuthology.orchestra.run.smithi097.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 194 kB 00:00 2024-03-06T07:24:53.233 INFO:teuthology.orchestra.run.smithi097.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2024-03-06T07:24:53.275 INFO:teuthology.orchestra.run.smithi097.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 241 kB 00:00 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi097.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 8.1 MB/s | 1.1 MB 00:00 2024-03-06T07:24:53.359 INFO:teuthology.orchestra.run.smithi097.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 836 kB/s | 103 kB 00:00 2024-03-06T07:24:53.384 INFO:teuthology.orchestra.run.smithi097.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.2 MB/s | 124 kB 00:00 2024-03-06T07:24:53.426 INFO:teuthology.orchestra.run.smithi097.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.7 MB/s | 113 kB 00:00 2024-03-06T07:24:53.452 INFO:teuthology.orchestra.run.smithi097.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.0 MB/s | 187 kB 00:00 2024-03-06T07:24:53.544 INFO:teuthology.orchestra.run.smithi097.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 246 kB/s | 22 kB 00:00 2024-03-06T07:24:53.652 INFO:teuthology.orchestra.run.smithi097.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 922 kB/s | 245 kB 00:00 2024-03-06T07:24:53.654 INFO:teuthology.orchestra.run.smithi135.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-06T07:24:53.686 INFO:teuthology.orchestra.run.smithi097.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-03-06T07:24:53.728 INFO:teuthology.orchestra.run.smithi097.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 2.9 MB/s | 548 kB 00:00 2024-03-06T07:24:53.778 INFO:teuthology.orchestra.run.smithi097.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 213 kB/s | 19 kB 00:00 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi097.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.9 MB/s | 309 kB 00:00 2024-03-06T07:24:53.854 INFO:teuthology.orchestra.run.smithi097.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-03-06T07:24:53.879 INFO:teuthology.orchestra.run.smithi097.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 732 kB/s | 49 kB 00:00 2024-03-06T07:24:53.962 INFO:teuthology.orchestra.run.smithi097.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-06T07:24:53.988 INFO:teuthology.orchestra.run.smithi097.stdout:(113/176): libunwind-1.6.2-1.el9.x86_64.rpm 506 kB/s | 67 kB 00:00 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi097.stdout:(114/176): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-06T07:24:54.130 INFO:teuthology.orchestra.run.smithi097.stdout:(115/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-06T07:24:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 658 kB/s | 43 kB 00:00 2024-03-06T07:24:54.180 INFO:teuthology.orchestra.run.smithi097.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi097.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 302 kB/s | 15 kB 00:00 2024-03-06T07:24:54.239 INFO:teuthology.orchestra.run.smithi097.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-06T07:24:54.273 INFO:teuthology.orchestra.run.smithi097.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-03-06T07:24:54.307 INFO:teuthology.orchestra.run.smithi097.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-03-06T07:24:54.356 INFO:teuthology.orchestra.run.smithi097.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 431 kB/s | 35 kB 00:00 2024-03-06T07:24:54.391 INFO:teuthology.orchestra.run.smithi097.stdout:(123/176): python3-asyncssh-2.13.2-4.el9.noarch 1.3 MB/s | 548 kB 00:00 2024-03-06T07:24:54.424 INFO:teuthology.orchestra.run.smithi097.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-06T07:24:54.508 INFO:teuthology.orchestra.run.smithi097.stdout:(125/176): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-06T07:24:54.533 INFO:teuthology.orchestra.run.smithi097.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-06T07:24:54.550 INFO:teuthology.orchestra.run.smithi097.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 430 kB/s | 18 kB 00:00 2024-03-06T07:24:54.584 INFO:teuthology.orchestra.run.smithi097.stdout:(128/176): python3-grpcio-tools-1.46.7-10.el9.x 753 kB/s | 144 kB 00:00 2024-03-06T07:24:54.601 INFO:teuthology.orchestra.run.smithi097.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-03-06T07:24:54.618 INFO:teuthology.orchestra.run.smithi097.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 292 kB/s | 19 kB 00:00 2024-03-06T07:24:54.643 INFO:teuthology.orchestra.run.smithi097.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 481 kB/s | 20 kB 00:00 2024-03-06T07:24:54.685 INFO:teuthology.orchestra.run.smithi097.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 699 kB/s | 46 kB 00:00 2024-03-06T07:24:54.719 INFO:teuthology.orchestra.run.smithi097.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-06T07:24:54.744 INFO:teuthology.orchestra.run.smithi097.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 996 kB/s | 58 kB 00:00 2024-03-06T07:24:54.811 INFO:teuthology.orchestra.run.smithi097.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-06T07:24:54.836 INFO:teuthology.orchestra.run.smithi097.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-03-06T07:24:54.911 INFO:teuthology.orchestra.run.smithi097.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 904 kB/s | 90 kB 00:00 2024-03-06T07:24:54.995 INFO:teuthology.orchestra.run.smithi097.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 179 kB/s | 28 kB 00:00 2024-03-06T07:24:55.020 INFO:teuthology.orchestra.run.smithi097.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-03-06T07:24:55.062 INFO:teuthology.orchestra.run.smithi097.stdout:(140/176): python3-kubernetes-26.1.0-2.el9.noar 2.2 MB/s | 1.0 MB 00:00 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi097.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-06T07:24:55.123 INFO:teuthology.orchestra.run.smithi097.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-03-06T07:24:55.296 INFO:teuthology.orchestra.run.smithi097.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 153 kB/s | 36 kB 00:00 2024-03-06T07:24:55.330 INFO:teuthology.orchestra.run.smithi097.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-06T07:24:55.383 INFO:teuthology.orchestra.run.smithi097.stdout:(145/176): python3-typing-extensions-4.4.0-2.el 178 kB/s | 51 kB 00:00 2024-03-06T07:24:55.408 INFO:teuthology.orchestra.run.smithi097.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 805 kB/s | 90 kB 00:00 2024-03-06T07:24:55.442 INFO:teuthology.orchestra.run.smithi097.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.7 MB/s | 427 kB 00:00 2024-03-06T07:24:55.518 INFO:teuthology.orchestra.run.smithi097.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-03-06T07:24:55.743 INFO:teuthology.orchestra.run.smithi097.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 575 kB/s | 192 kB 00:00 2024-03-06T07:24:55.793 INFO:teuthology.orchestra.run.smithi097.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi097.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 68 kB 00:00 2024-03-06T07:24:56.094 INFO:teuthology.orchestra.run.smithi097.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 9.8 MB/s | 2.3 MB 00:00 2024-03-06T07:24:56.244 INFO:teuthology.orchestra.run.smithi097.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 12 MB/s | 1.8 MB 00:00 2024-03-06T07:24:56.320 INFO:teuthology.orchestra.run.smithi097.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 557 kB/s | 41 kB 00:00 2024-03-06T07:24:56.470 INFO:teuthology.orchestra.run.smithi097.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 1.6 MB/s | 247 kB 00:00 2024-03-06T07:24:56.471 INFO:teuthology.orchestra.run.smithi135.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 3.5 MB/s | 9.8 MB 00:02 2024-03-06T07:24:56.520 INFO:teuthology.orchestra.run.smithi097.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2024-03-06T07:24:56.538 INFO:teuthology.orchestra.run.smithi135.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 383 kB/s | 25 kB 00:00 2024-03-06T07:24:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 193 kB/s | 35 kB 00:00 2024-03-06T07:24:56.796 INFO:teuthology.orchestra.run.smithi097.stdout:(158/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 3.0 MB/s | 3.1 MB 00:01 2024-03-06T07:24:56.830 INFO:teuthology.orchestra.run.smithi097.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 191 kB 00:00 2024-03-06T07:24:56.872 INFO:teuthology.orchestra.run.smithi097.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.4 MB/s | 254 kB 00:00 2024-03-06T07:24:56.939 INFO:teuthology.orchestra.run.smithi097.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 7.2 MB/s | 802 kB 00:00 2024-03-06T07:24:56.972 INFO:teuthology.orchestra.run.smithi097.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 4.5 MB/s | 433 kB 00:00 2024-03-06T07:24:56.998 INFO:teuthology.orchestra.run.smithi097.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 409 kB/s | 22 kB 00:00 2024-03-06T07:24:57.023 INFO:teuthology.orchestra.run.smithi097.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-06T07:24:57.056 INFO:teuthology.orchestra.run.smithi097.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 154 kB 00:00 2024-03-06T07:24:57.082 INFO:teuthology.orchestra.run.smithi097.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 135 kB 00:00 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi097.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 63 kB 00:00 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi097.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 479 kB/s | 28 kB 00:00 2024-03-06T07:24:57.241 INFO:teuthology.orchestra.run.smithi097.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 9.3 MB/s | 1.2 MB 00:00 2024-03-06T07:24:57.367 INFO:teuthology.orchestra.run.smithi097.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 2.2 MB 00:00 2024-03-06T07:24:57.458 INFO:teuthology.orchestra.run.smithi097.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-06T07:24:57.500 INFO:teuthology.orchestra.run.smithi097.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 438 kB 00:00 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi097.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 478 kB/s | 30 kB 00:00 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi097.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-06T07:24:57.618 INFO:teuthology.orchestra.run.smithi097.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-06T07:24:58.230 INFO:teuthology.orchestra.run.smithi135.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 2.2 MB/s | 20 MB 00:09 2024-03-06T07:24:58.522 INFO:teuthology.orchestra.run.smithi135.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 759 kB/s | 220 kB 00:00 2024-03-06T07:24:58.622 INFO:teuthology.orchestra.run.smithi135.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 273 kB/s | 27 kB 00:00 2024-03-06T07:24:58.730 INFO:teuthology.orchestra.run.smithi135.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 291 kB/s | 31 kB 00:00 2024-03-06T07:24:59.331 INFO:teuthology.orchestra.run.smithi135.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 747 kB 00:00 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi135.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 755 kB/s | 163 kB 00:00 2024-03-06T07:24:59.748 INFO:teuthology.orchestra.run.smithi135.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 636 kB/s | 127 kB 00:00 2024-03-06T07:24:59.796 INFO:teuthology.orchestra.run.smithi097.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 813 kB/s | 3.4 MB 00:04 2024-03-06T07:24:59.801 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:24:59.802 INFO:teuthology.orchestra.run.smithi097.stdout:Total 7.2 MB/s | 224 MB 00:30 2024-03-06T07:24:59.935 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-06T07:25:00.174 INFO:teuthology.orchestra.run.smithi135.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.2 MB/s | 501 kB 00:00 2024-03-06T07:25:00.309 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-06T07:25:00.309 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-06T07:25:02.559 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-06T07:25:02.559 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-06T07:25:03.298 INFO:teuthology.orchestra.run.smithi135.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.6 MB/s | 5.1 MB 00:03 2024-03-06T07:25:03.407 INFO:teuthology.orchestra.run.smithi135.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 418 kB/s | 45 kB 00:00 2024-03-06T07:25:03.532 INFO:teuthology.orchestra.run.smithi135.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.0 MB/s | 131 kB 00:00 2024-03-06T07:25:03.724 INFO:teuthology.orchestra.run.smithi135.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 847 kB/s | 162 kB 00:00 2024-03-06T07:25:04.025 INFO:teuthology.orchestra.run.smithi135.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.0 MB/s | 321 kB 00:00 2024-03-06T07:25:04.225 INFO:teuthology.orchestra.run.smithi135.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.5 MB/s | 298 kB 00:00 2024-03-06T07:25:04.350 INFO:teuthology.orchestra.run.smithi135.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 798 kB/s | 99 kB 00:00 2024-03-06T07:25:04.467 INFO:teuthology.orchestra.run.smithi135.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 735 kB/s | 85 kB 00:00 2024-03-06T07:25:06.217 INFO:teuthology.orchestra.run.smithi135.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.8 MB/s | 3.1 MB 00:01 2024-03-06T07:25:06.443 INFO:teuthology.orchestra.run.smithi135.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 761 kB/s | 171 kB 00:00 2024-03-06T07:25:06.551 INFO:teuthology.orchestra.run.smithi135.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 292 kB/s | 31 kB 00:00 2024-03-06T07:25:06.677 INFO:teuthology.orchestra.run.smithi135.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.1 MB/s | 138 kB 00:00 2024-03-06T07:25:07.802 INFO:teuthology.orchestra.run.smithi135.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.6 MB/s | 1.8 MB 00:01 2024-03-06T07:25:10.046 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-06T07:25:10.159 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:10.199 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:10.246 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:10.521 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:10.640 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:10.691 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:10.729 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:10.836 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:11.259 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:11.327 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:11.491 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:11.520 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:11.698 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:11.743 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:11.996 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:12.060 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:12.104 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:12.295 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:12.325 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:12.343 INFO:teuthology.orchestra.run.smithi135.stdout:(31/176): ceph-mgr-diskprediction-local-19.0.0- 1.6 MB/s | 7.4 MB 00:04 2024-03-06T07:25:12.416 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:12.461 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:12.486 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:12.546 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:12.818 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:12.861 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:12.877 INFO:teuthology.orchestra.run.smithi135.stdout:(32/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 2.9 MB/s | 47 MB 00:16 2024-03-06T07:25:12.907 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:12.910 INFO:teuthology.orchestra.run.smithi135.stdout:(33/176): ceph-mgr-modules-core-19.0.0-1739.g1b 439 kB/s | 248 kB 00:00 2024-03-06T07:25:12.961 INFO:teuthology.orchestra.run.smithi135.stdout:(34/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 598 kB/s | 49 kB 00:00 2024-03-06T07:25:13.019 INFO:teuthology.orchestra.run.smithi135.stdout:(35/176): ceph-prometheus-alerts-19.0.0-1739.g1 142 kB/s | 15 kB 00:00 2024-03-06T07:25:13.249 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:13.270 INFO:teuthology.orchestra.run.smithi135.stdout:(36/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 834 kB/s | 17 MB 00:20 2024-03-06T07:25:13.303 INFO:teuthology.orchestra.run.smithi135.stdout:(37/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 772 kB/s | 264 kB 00:00 2024-03-06T07:25:13.414 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:13.428 INFO:teuthology.orchestra.run.smithi135.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 116 kB/s | 18 kB 00:00 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:13.537 INFO:teuthology.orchestra.run.smithi135.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-06T07:25:13.621 INFO:teuthology.orchestra.run.smithi135.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-06T07:25:13.654 INFO:teuthology.orchestra.run.smithi135.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 973 kB/s | 106 kB 00:00 2024-03-06T07:25:13.680 INFO:teuthology.orchestra.run.smithi135.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 363 kB/s | 20 kB 00:00 2024-03-06T07:25:13.712 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:13.722 INFO:teuthology.orchestra.run.smithi135.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 307 kB 00:00 2024-03-06T07:25:13.747 INFO:teuthology.orchestra.run.smithi135.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 497 kB/s | 32 kB 00:00 2024-03-06T07:25:13.763 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:13.864 INFO:teuthology.orchestra.run.smithi135.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-06T07:25:13.881 INFO:teuthology.orchestra.run.smithi135.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 144 kB/s | 17 kB 00:00 2024-03-06T07:25:13.898 INFO:teuthology.orchestra.run.smithi135.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 310 kB/s | 10 kB 00:00 2024-03-06T07:25:13.923 INFO:teuthology.orchestra.run.smithi135.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-06T07:25:13.948 INFO:teuthology.orchestra.run.smithi135.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 917 kB/s | 44 kB 00:00 2024-03-06T07:25:13.990 INFO:teuthology.orchestra.run.smithi135.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-06T07:25:14.040 INFO:teuthology.orchestra.run.smithi135.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2024-03-06T07:25:14.074 INFO:teuthology.orchestra.run.smithi135.stdout:(52/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 709 kB/s | 748 kB 00:01 2024-03-06T07:25:14.099 INFO:teuthology.orchestra.run.smithi135.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 236 kB/s | 24 kB 00:00 2024-03-06T07:25:14.102 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:14.124 INFO:teuthology.orchestra.run.smithi135.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 561 kB/s | 46 kB 00:00 2024-03-06T07:25:14.158 INFO:teuthology.orchestra.run.smithi135.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2024-03-06T07:25:14.192 INFO:teuthology.orchestra.run.smithi135.stdout:(56/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-06T07:25:14.217 INFO:teuthology.orchestra.run.smithi135.stdout:(57/176): openblas-0.3.21-2.el9.x86_64.rpm 374 kB/s | 34 kB 00:00 2024-03-06T07:25:14.296 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:25:14.359 INFO:teuthology.orchestra.run.smithi135.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-06T07:25:14.419 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:25:14.426 INFO:teuthology.orchestra.run.smithi135.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 7.9 MB/s | 1.5 MB 00:00 2024-03-06T07:25:14.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:25:14.509 INFO:teuthology.orchestra.run.smithi135.stdout:(60/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-06T07:25:14.551 INFO:teuthology.orchestra.run.smithi135.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 4.3 MB/s | 811 kB 00:00 2024-03-06T07:25:14.559 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:25:14.576 INFO:teuthology.orchestra.run.smithi135.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 998 kB/s | 134 kB 00:00 2024-03-06T07:25:14.623 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:25:14.660 INFO:teuthology.orchestra.run.smithi135.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 9.5 MB/s | 1.0 MB 00:00 2024-03-06T07:25:14.690 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:25:14.757 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:25:14.802 INFO:teuthology.orchestra.run.smithi135.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-06T07:25:14.814 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:25:14.827 INFO:teuthology.orchestra.run.smithi135.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 311 kB/s | 78 kB 00:00 2024-03-06T07:25:14.927 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:25:14.968 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:25:14.969 INFO:teuthology.orchestra.run.smithi135.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-06T07:25:14.994 INFO:teuthology.orchestra.run.smithi135.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 24 kB 00:00 2024-03-06T07:25:15.028 INFO:teuthology.orchestra.run.smithi135.stdout:(68/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 227 kB 00:00 2024-03-06T07:25:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:25:15.053 INFO:teuthology.orchestra.run.smithi135.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 324 kB/s | 18 kB 00:00 2024-03-06T07:25:15.078 INFO:teuthology.orchestra.run.smithi135.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-06T07:25:15.081 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:25:15.103 INFO:teuthology.orchestra.run.smithi135.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 431 kB/s | 32 kB 00:00 2024-03-06T07:25:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:25:15.137 INFO:teuthology.orchestra.run.smithi135.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 217 kB 00:00 2024-03-06T07:25:15.235 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:25:15.266 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:25:15.271 INFO:teuthology.orchestra.run.smithi135.stdout:(73/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-06T07:25:15.304 INFO:teuthology.orchestra.run.smithi135.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 883 kB/s | 174 kB 00:00 2024-03-06T07:25:15.328 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:25:15.329 INFO:teuthology.orchestra.run.smithi135.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 386 kB/s | 69 kB 00:00 2024-03-06T07:25:15.364 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:25:15.371 INFO:teuthology.orchestra.run.smithi135.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-03-06T07:25:15.405 INFO:teuthology.orchestra.run.smithi135.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 238 kB 00:00 2024-03-06T07:25:15.439 INFO:teuthology.orchestra.run.smithi135.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-06T07:25:15.472 INFO:teuthology.orchestra.run.smithi135.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2024-03-06T07:25:15.480 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi135.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 998 kB/s | 47 kB 00:00 2024-03-06T07:25:15.506 INFO:teuthology.orchestra.run.smithi135.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 668 kB/s | 43 kB 00:00 2024-03-06T07:25:15.567 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:25:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:25:15.716 INFO:teuthology.orchestra.run.smithi135.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 212 kB/s | 41 kB 00:00 2024-03-06T07:25:15.835 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:25:15.911 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:25:16.235 INFO:teuthology.orchestra.run.smithi135.stdout:(83/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-03-06T07:25:16.257 INFO:teuthology.orchestra.run.smithi135.stdout:(84/176): python3-toml-0.10.2-6.el9.noarch.rpm 52 kB/s | 39 kB 00:00 2024-03-06T07:25:16.299 INFO:teuthology.orchestra.run.smithi135.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 370 kB/s | 215 kB 00:00 2024-03-06T07:25:16.324 INFO:teuthology.orchestra.run.smithi135.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 528 kB/s | 34 kB 00:00 2024-03-06T07:25:16.349 INFO:teuthology.orchestra.run.smithi135.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 489 kB/s | 43 kB 00:00 2024-03-06T07:25:16.383 INFO:teuthology.orchestra.run.smithi135.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-06T07:25:16.408 INFO:teuthology.orchestra.run.smithi135.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2024-03-06T07:25:16.442 INFO:teuthology.orchestra.run.smithi135.stdout:(90/176): uuid-1.6.2-55.el9.x86_64.rpm 636 kB/s | 56 kB 00:00 2024-03-06T07:25:16.467 INFO:teuthology.orchestra.run.smithi135.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 761 kB/s | 63 kB 00:00 2024-03-06T07:25:16.492 INFO:teuthology.orchestra.run.smithi135.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 537 kB/s | 44 kB 00:00 2024-03-06T07:25:16.517 INFO:teuthology.orchestra.run.smithi135.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-06T07:25:16.542 INFO:teuthology.orchestra.run.smithi135.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.5 MB/s | 186 kB 00:00 2024-03-06T07:25:16.585 INFO:teuthology.orchestra.run.smithi135.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 8.8 MB/s | 802 kB 00:00 2024-03-06T07:25:16.612 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:25:16.618 INFO:teuthology.orchestra.run.smithi135.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 194 kB 00:00 2024-03-06T07:25:16.643 INFO:teuthology.orchestra.run.smithi135.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-03-06T07:25:16.694 INFO:teuthology.orchestra.run.smithi135.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-06T07:25:16.707 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi135.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-06T07:25:16.778 INFO:teuthology.orchestra.run.smithi135.stdout:(100/176): python3-cryptography-36.0.1-4.el9.x8 8.7 MB/s | 1.1 MB 00:00 2024-03-06T07:25:16.803 INFO:teuthology.orchestra.run.smithi135.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.1 MB/s | 124 kB 00:00 2024-03-06T07:25:16.828 INFO:teuthology.orchestra.run.smithi135.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.1 MB/s | 113 kB 00:00 2024-03-06T07:25:16.870 INFO:teuthology.orchestra.run.smithi135.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.8 MB/s | 187 kB 00:00 2024-03-06T07:25:16.877 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:25:16.927 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:16.953 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:16.953 INFO:teuthology.orchestra.run.smithi135.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 182 kB/s | 22 kB 00:00 2024-03-06T07:25:16.996 INFO:teuthology.orchestra.run.smithi135.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.3 MB/s | 245 kB 00:00 2024-03-06T07:25:17.013 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:25:17.063 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:25:17.118 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:25:17.171 INFO:teuthology.orchestra.run.smithi135.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.8 MB/s | 863 kB 00:00 2024-03-06T07:25:17.299 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:25:17.304 INFO:teuthology.orchestra.run.smithi135.stdout:(107/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-03-06T07:25:17.330 INFO:teuthology.orchestra.run.smithi135.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-06T07:25:17.372 INFO:teuthology.orchestra.run.smithi135.stdout:(109/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-06T07:25:17.397 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:25:17.405 INFO:teuthology.orchestra.run.smithi135.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-03-06T07:25:17.431 INFO:teuthology.orchestra.run.smithi135.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 841 kB/s | 49 kB 00:00 2024-03-06T07:25:17.497 INFO:teuthology.orchestra.run.smithi135.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-06T07:25:17.531 INFO:teuthology.orchestra.run.smithi135.stdout:(113/176): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-06T07:25:17.623 INFO:teuthology.orchestra.run.smithi135.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-06T07:25:17.748 INFO:teuthology.orchestra.run.smithi135.stdout:(115/176): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-06T07:25:17.774 INFO:teuthology.orchestra.run.smithi135.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 291 kB/s | 43 kB 00:00 2024-03-06T07:25:17.778 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:25:17.807 INFO:teuthology.orchestra.run.smithi135.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 556 kB/s | 32 kB 00:00 2024-03-06T07:25:17.824 INFO:teuthology.orchestra.run.smithi135.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 302 kB/s | 15 kB 00:00 2024-03-06T07:25:17.874 INFO:teuthology.orchestra.run.smithi135.stdout:(119/176): python3-asyncssh-2.13.2-4.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-03-06T07:25:17.908 INFO:teuthology.orchestra.run.smithi135.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-06T07:25:17.950 INFO:teuthology.orchestra.run.smithi135.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-06T07:25:17.967 INFO:teuthology.orchestra.run.smithi135.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 612 kB/s | 35 kB 00:00 2024-03-06T07:25:18.009 INFO:teuthology.orchestra.run.smithi135.stdout:(123/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-06T07:25:18.048 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:25:18.051 INFO:teuthology.orchestra.run.smithi135.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-06T07:25:18.084 INFO:teuthology.orchestra.run.smithi135.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-06T07:25:18.101 INFO:teuthology.orchestra.run.smithi135.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 215 kB/s | 11 kB 00:00 2024-03-06T07:25:18.135 INFO:teuthology.orchestra.run.smithi135.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 356 kB/s | 18 kB 00:00 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi135.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 467 kB/s | 23 kB 00:00 2024-03-06T07:25:18.168 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:25:18.210 INFO:teuthology.orchestra.run.smithi135.stdout:(129/176): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-03-06T07:25:18.227 INFO:teuthology.orchestra.run.smithi135.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-03-06T07:25:18.229 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:25:18.244 INFO:teuthology.orchestra.run.smithi135.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-03-06T07:25:18.303 INFO:teuthology.orchestra.run.smithi135.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-06T07:25:18.320 INFO:teuthology.orchestra.run.smithi135.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 623 kB/s | 46 kB 00:00 2024-03-06T07:25:18.345 INFO:teuthology.orchestra.run.smithi135.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 794 kB/s | 79 kB 00:00 2024-03-06T07:25:18.362 INFO:teuthology.orchestra.run.smithi135.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 997 kB/s | 58 kB 00:00 2024-03-06T07:25:18.404 INFO:teuthology.orchestra.run.smithi135.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-06T07:25:18.421 INFO:teuthology.orchestra.run.smithi135.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-06T07:25:18.431 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:25:18.446 INFO:teuthology.orchestra.run.smithi135.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-06T07:25:18.463 INFO:teuthology.orchestra.run.smithi135.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 487 kB/s | 28 kB 00:00 2024-03-06T07:25:18.488 INFO:teuthology.orchestra.run.smithi135.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 463 kB/s | 31 kB 00:00 2024-03-06T07:25:18.514 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:25:18.522 INFO:teuthology.orchestra.run.smithi135.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi135.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 721 kB/s | 36 kB 00:00 2024-03-06T07:25:18.556 INFO:teuthology.orchestra.run.smithi135.stdout:(143/176): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2024-03-06T07:25:18.573 INFO:teuthology.orchestra.run.smithi135.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-06T07:25:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:25:18.615 INFO:teuthology.orchestra.run.smithi135.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-06T07:25:18.640 INFO:teuthology.orchestra.run.smithi135.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-06T07:25:18.674 INFO:teuthology.orchestra.run.smithi135.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-06T07:25:18.704 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:25:18.732 INFO:teuthology.orchestra.run.smithi135.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 172 kB/s | 20 kB 00:00 2024-03-06T07:25:18.739 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:25:18.767 INFO:teuthology.orchestra.run.smithi135.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-06T07:25:18.768 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:25:18.826 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 9.9 MB/s | 1.6 MB 00:00 2024-03-06T07:25:18.876 INFO:teuthology.orchestra.run.smithi135.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-06T07:25:18.893 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:25:18.956 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:25:19.017 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:25:19.018 INFO:teuthology.orchestra.run.smithi135.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-06T07:25:19.135 INFO:teuthology.orchestra.run.smithi135.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-03-06T07:25:19.169 INFO:teuthology.orchestra.run.smithi135.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:25:19.203 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:25:19.211 INFO:teuthology.orchestra.run.smithi135.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-06T07:25:19.244 INFO:teuthology.orchestra.run.smithi135.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:25:19.270 INFO:teuthology.orchestra.run.smithi135.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-06T07:25:19.311 INFO:teuthology.orchestra.run.smithi135.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-03-06T07:25:19.312 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:25:19.351 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:25:19.354 INFO:teuthology.orchestra.run.smithi135.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-03-06T07:25:19.402 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:25:19.420 INFO:teuthology.orchestra.run.smithi135.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 12 MB/s | 802 kB 00:00 2024-03-06T07:25:19.471 INFO:teuthology.orchestra.run.smithi135.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.6 MB/s | 433 kB 00:00 2024-03-06T07:25:19.504 INFO:teuthology.orchestra.run.smithi135.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 666 kB/s | 22 kB 00:00 2024-03-06T07:25:19.541 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:25:19.547 INFO:teuthology.orchestra.run.smithi135.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-06T07:25:19.583 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:25:19.588 INFO:teuthology.orchestra.run.smithi135.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.7 MB/s | 154 kB 00:00 2024-03-06T07:25:19.630 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:25:19.630 INFO:teuthology.orchestra.run.smithi135.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:25:19.666 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:25:19.672 INFO:teuthology.orchestra.run.smithi135.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-06T07:25:19.714 INFO:teuthology.orchestra.run.smithi135.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 676 kB/s | 28 kB 00:00 2024-03-06T07:25:19.729 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:25:19.789 INFO:teuthology.orchestra.run.smithi135.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 16 MB/s | 1.2 MB 00:00 2024-03-06T07:25:19.898 INFO:teuthology.orchestra.run.smithi135.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-06T07:25:19.901 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:25:20.000 INFO:teuthology.orchestra.run.smithi135.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2024-03-06T07:25:20.076 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:25:20.082 INFO:teuthology.orchestra.run.smithi135.stdout:(171/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-06T07:25:20.124 INFO:teuthology.orchestra.run.smithi135.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 438 kB 00:00 2024-03-06T07:25:20.141 INFO:teuthology.orchestra.run.smithi135.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 553 kB/s | 30 kB 00:00 2024-03-06T07:25:20.175 INFO:teuthology.orchestra.run.smithi135.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-06T07:25:20.233 INFO:teuthology.orchestra.run.smithi135.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-06T07:25:20.244 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:25:20.292 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:25:20.354 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:25:20.429 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:25:20.679 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:25:20.911 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:25:20.966 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:25:21.028 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:25:21.069 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:25:21.108 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:25:21.144 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:25:21.189 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:25:21.241 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:25:21.308 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:25:21.364 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:25:21.437 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:25:21.570 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:25:21.624 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:25:21.660 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:25:21.725 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:25:21.953 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:25:22.079 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:25:22.142 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:25:22.179 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:25:22.238 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:25:22.266 INFO:teuthology.orchestra.run.smithi135.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 983 kB/s | 3.4 MB 00:03 2024-03-06T07:25:22.275 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:25:22.275 INFO:teuthology.orchestra.run.smithi135.stdout:Total 6.7 MB/s | 224 MB 00:33 2024-03-06T07:25:22.385 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:25:22.404 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-06T07:25:22.452 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:25:22.782 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-06T07:25:22.782 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-06T07:25:23.425 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:25:23.477 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:25:23.527 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:25:23.584 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:25:23.695 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:25:23.736 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:25:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:25:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:25:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:25:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:25:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:25:25.159 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-06T07:25:25.160 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-06T07:25:26.481 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:25:26.573 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:25:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:25:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:25:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:27.105 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:27.133 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:25:27.178 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:25:27.678 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:25:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:28.334 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:25:30.905 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:30.948 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:31.564 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:31.609 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:31.742 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:31.777 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:32.437 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:32.588 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:32.710 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:32.771 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:32.850 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:32.949 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:33.003 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:33.069 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:33.174 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:33.606 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:33.651 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:33.801 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:33.827 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:33.946 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:33.982 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:34.227 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:34.283 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:34.306 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:34.474 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:34.563 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:34.617 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:34.642 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:35.002 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:35.049 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:35.094 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:35.454 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:35.624 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:35.704 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:35.905 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:35.977 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:36.327 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:36.517 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:25:36.649 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:25:36.683 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:25:36.779 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:25:36.845 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:25:36.919 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:25:37.300 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:25:37.366 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:25:37.536 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:25:37.609 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:25:37.693 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:25:37.808 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:25:37.847 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:25:37.900 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:25:37.928 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:25:38.028 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:25:38.080 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:25:38.178 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:25:38.348 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:25:38.416 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:25:39.084 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:25:39.181 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:25:39.373 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:39.398 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:39.429 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:25:39.475 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:25:39.533 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:25:39.706 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:25:39.760 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:25:40.353 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:25:40.472 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:25:40.545 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:25:40.763 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:25:40.818 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:25:40.892 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:25:40.988 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:25:41.026 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:25:41.064 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:25:41.173 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:25:41.237 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:25:41.304 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:25:41.493 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:25:41.546 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:25:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:25:41.632 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:25:41.761 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:25:41.799 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:25:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:25:41.906 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:25:42.052 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:25:42.223 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:25:42.364 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:25:42.441 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:25:42.505 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:25:42.580 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:25:42.828 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:25:43.060 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:25:43.111 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:25:43.150 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:25:43.192 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:25:43.222 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:25:43.258 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:25:43.300 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:25:43.345 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:25:43.412 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:25:43.487 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:25:43.535 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:25:43.664 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:25:43.706 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:25:43.749 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:25:43.804 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:25:43.977 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:25:44.096 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:25:44.148 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:25:44.184 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:25:44.239 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:25:44.377 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:25:44.461 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:44.461 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-03-06T07:25:44.462 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:44.500 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:25:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:25:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:25:44.712 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:44.712 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:25:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:45.463 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:25:45.524 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:25:45.557 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:25:45.616 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:25:45.727 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:25:45.766 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:25:45.989 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:25:46.013 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:25:46.104 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:25:46.313 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:25:46.393 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:25:46.393 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:46.393 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:25:46.393 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:25:46.393 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:25:46.394 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:46.655 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:25:46.732 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:25:46.732 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:46.732 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:25:46.732 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:25:46.733 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:25:46.733 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:46.892 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:25:47.040 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:25:47.223 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:25:47.291 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:25:47.291 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:47.292 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:25:47.292 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:25:47.292 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:25:47.292 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:47.408 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:25:48.036 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:25:48.075 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:25:48.125 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:25:48.227 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:25:48.263 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:25:48.554 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:25:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:25:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:25:48.635 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:25:48.661 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:25:48.997 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:25:49.039 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:49.158 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:49.187 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:25:49.243 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:25:49.746 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:25:49.756 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:25:49.775 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:49.866 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:49.894 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:25:49.898 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:25:50.105 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:25:50.142 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:25:50.211 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:25:50.213 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:50.213 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:25:50.213 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:25:50.213 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:25:50.213 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:25:50.303 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:25:50.366 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:25:51.366 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:25:51.470 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:25:51.470 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:51.470 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:25:51.470 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:25:51.470 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:25:51.470 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:51.520 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:25:51.600 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:25:51.600 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:51.600 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:25:51.600 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:51.679 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:25:51.757 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:25:51.757 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:51.757 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:25:51.757 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:25:51.757 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:25:51.757 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:25:52.215 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:25:52.215 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:52.215 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:25:52.215 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:25:52.216 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:25:52.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:25:52.436 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:52.482 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:53.100 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:53.144 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:53.323 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:53.323 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:25:53.323 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:25:53.323 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:25:53.370 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:56.981 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:25:57.032 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:25:57.084 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:25:57.122 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:25:57.407 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:25:57.548 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:25:57.594 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:25:57.640 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:25:57.726 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:25:57.848 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:25:57.937 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:25:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:25:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:25:58.089 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:25:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:25:58.306 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:25:59.360 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:25:59.411 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:25:59.460 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:00.791 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:00.866 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:00.939 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:01.065 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:01.102 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:01.127 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:01.161 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:01.369 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:01.419 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:01.465 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:01.465 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:01.596 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:01.656 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:01.713 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:01.769 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:01.819 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:01.935 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:02.008 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:02.058 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:02.091 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:02.125 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:02.183 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:02.217 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:02.253 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:02.253 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:02.609 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:02.609 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:02.806 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:02.879 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:02.912 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:02.948 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:03.352 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:03.352 INFO:teuthology.orchestra.run.smithi097.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:03.353 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /root 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2024-03-06T07:26:06.414 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:06.635 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:08.159 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:08.160 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:08.161 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:08.181 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:08.182 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:08.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:08.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:08.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:08.187 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:08.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:08.189 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:08.190 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:08.277 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:08.356 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:08.356 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:08.356 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:26:08.356 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:08.356 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:08.356 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:08.603 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:08.670 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:08.671 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:08.671 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:26:08.671 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:08.671 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:08.671 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:09.145 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:09.212 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:09.212 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:09.212 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:26:09.212 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:09.212 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:09.322 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:26:09.963 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:09.998 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:10.046 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:26:10.148 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:26:10.185 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:10.514 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:10.807 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:10.807 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:10.807 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-03-06T07:26:10.807 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:10.807 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.810 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.811 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.812 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:10.814 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:10.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:10.821 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:10.821 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:10.821 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-06T07:26:11.460 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:11.663 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:11.854 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:12.061 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:26:12.097 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:12.164 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:12.261 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:26:13.339 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:26:13.374 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:13.443 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:13.443 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:13.443 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:26:13.443 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:13.444 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:13.444 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:13.500 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:13.570 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:13.570 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:13.570 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:26:13.570 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:13.643 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:14.101 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:14.171 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:14.171 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:14.171 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:26:14.171 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:14.171 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:14.171 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:18.941 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:26:19.030 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:26:19.372 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:26:19.500 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:26:19.548 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:26:19.599 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:26:19.681 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:26:19.799 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:26:19.895 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:26:19.925 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:26:19.978 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:26:20.102 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:26:20.182 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:26:20.219 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:26:21.102 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:21.157 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:22.612 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:22.658 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:22.772 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:22.805 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:22.839 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:22.873 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:22.914 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:22.915 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:23.048 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:23.099 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:23.144 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:23.144 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:23.291 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:23.344 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:23.392 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:23.431 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:23.474 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:23.524 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:23.599 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:23.737 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:23.769 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:23.803 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:23.837 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:23.907 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:23.907 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:24.229 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:24.229 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:24.368 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:24.419 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:24.448 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:24.518 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:24.653 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:24.914 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:24.915 INFO:teuthology.orchestra.run.smithi135.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:24.915 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:30.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:30.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:30.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:30.161 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:30.167 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:32.619 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:32.620 INFO:teuthology.orchestra.run.smithi135.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:32.621 INFO:teuthology.orchestra.run.smithi135.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.623 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:32.624 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.625 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:32.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:32.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:32.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-06T07:26:33.346 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:33.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:26:33.565 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:26:33.635 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:33.636 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:33.636 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:26:33.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:26:33.854 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:26:33.907 INFO:teuthology.orchestra.run.smithi135.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:33.907 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:33.907 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:26:33.909 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:26:33.909 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:33.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:26:34.012 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:34.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:26:34.089 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:26:34.089 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:34.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:26:34.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:26:34.283 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:34.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:26:34.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:26:34.492 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:26:34.493 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:34.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:26:34.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:26:34.717 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:34.717 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:26:34.796 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:26:34.912 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:26:34.912 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:34.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:26:34.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:26:35.109 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:35.110 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:26:35.293 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:26:35.395 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:26:35.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2024-03-06T07:26:35.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /var/log/ceph 2024-03-06T07:26:35.485 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:26:35.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:26:35.491 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:26:35.565 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:26:35.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:26:35.572 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:26:35.648 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:26:35.648 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:26:35.648 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:26:35.648 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:35.649 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:26:35.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:35.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:36.005 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:36.005 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:26:36.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:36.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:36.329 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:26:36.329 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:26:36.329 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583075', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi097.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArYvpsmBmeK/TBDiqVWIuoAATGQa8GFzU77A+6ELgW99JRI/oLRgHOskkANHygDldGYO6XxqDib3DK/40vuNkg=', 'smithi135.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPfyFDKab2OwyrP2BRUX2aSe+lz0hRLCl1Bj+JetLDX2qnn7gI3+3BuLidL6e2F9CEQCxVSFwMDe8F0xW2Qays='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:26:36.330 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:26:36.338 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:26:36.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:26:36.392 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:26:36.484 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:36.484 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:26:36.538 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:26:36.539 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-03-06T07:26:36.588 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:26:36.588 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:36.588 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-06T07:26:36.588 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:36.589 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:36.589 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-06 07:26:03.797380098 +0000 2024-03-06T07:26:36.589 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-06 07:23:24.348270542 +0000 2024-03-06T07:26:36.589 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-06 07:23:24.348270542 +0000 2024-03-06T07:26:36.589 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-06 07:23:24.348270542 +0000 2024-03-06T07:26:36.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:26:36.672 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-06T07:26:36.672 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-06T07:26:36.672 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000312819 s, 1.6 MB/s 2024-03-06T07:26:36.676 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:26:36.729 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-06 07:26:03.797380098 +0000 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-06 07:23:24.918260629 +0000 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-06 07:23:24.918260629 +0000 2024-03-06T07:26:36.777 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-06 07:23:24.918260629 +0000 2024-03-06T07:26:36.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:26:36.860 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-06T07:26:36.860 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-06T07:26:36.860 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000350025 s, 1.5 MB/s 2024-03-06T07:26:36.864 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:26:36.920 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-06 07:26:03.798380080 +0000 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-06 07:23:25.532249950 +0000 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-06 07:23:25.532249950 +0000 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-06 07:23:25.532249950 +0000 2024-03-06T07:26:36.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:26:37.094 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-06T07:26:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-06T07:26:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000331071 s, 1.5 MB/s 2024-03-06T07:26:37.099 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:26:37.153 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-03-06T07:26:37.201 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:26:37.201 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:37.201 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-06T07:26:37.202 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:37.202 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:37.202 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-06 07:26:03.799380063 +0000 2024-03-06T07:26:37.202 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-06 07:23:26.100240072 +0000 2024-03-06T07:26:37.202 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-06 07:23:26.100240072 +0000 2024-03-06T07:26:37.202 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-06 07:23:26.100240072 +0000 2024-03-06T07:26:37.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:26:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-06T07:26:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-06T07:26:37.291 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000349343 s, 1.5 MB/s 2024-03-06T07:26:37.295 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:26:37.349 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:37.349 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:37.349 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:26:37.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:26:37.398 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-03-06T07:26:37.483 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-06 07:26:25.394895785 +0000 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-06 07:23:24.374465745 +0000 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-06 07:23:24.374465745 +0000 2024-03-06T07:26:37.484 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-06 07:23:24.374465745 +0000 2024-03-06T07:26:37.484 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:26:37.596 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-06T07:26:37.597 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-06T07:26:37.597 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000371437 s, 1.4 MB/s 2024-03-06T07:26:37.600 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:26:37.642 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-06 07:26:25.395895764 +0000 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-06 07:23:24.931454109 +0000 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-06 07:23:24.931454109 +0000 2024-03-06T07:26:37.721 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-06 07:23:24.931454109 +0000 2024-03-06T07:26:37.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:26:37.830 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-06T07:26:37.830 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-06T07:26:37.830 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000285601 s, 1.8 MB/s 2024-03-06T07:26:37.832 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:26:37.872 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-03-06T07:26:37.951 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-06 07:26:25.395895764 +0000 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-06 07:23:25.506442098 +0000 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-06 07:23:25.506442098 +0000 2024-03-06T07:26:37.952 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-06 07:23:25.506442098 +0000 2024-03-06T07:26:37.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:26:38.061 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-06T07:26:38.062 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-06T07:26:38.062 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000303817 s, 1.7 MB/s 2024-03-06T07:26:38.064 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:26:38.105 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-03-06T07:26:38.187 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:26:38.187 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:38.187 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-03-06T07:26:38.187 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:38.188 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:38.188 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-06 07:26:25.396895743 +0000 2024-03-06T07:26:38.188 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-06 07:23:26.071430296 +0000 2024-03-06T07:26:38.188 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-06 07:23:26.071430296 +0000 2024-03-06T07:26:38.188 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-06 07:23:26.071430296 +0000 2024-03-06T07:26:38.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:26:38.299 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-06T07:26:38.299 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-06T07:26:38.299 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000268135 s, 1.9 MB/s 2024-03-06T07:26:38.301 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:26:38.343 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:38.343 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:26:38.343 INFO:tasks.ceph:Generating config... 2024-03-06T07:26:38.345 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:26:38.345 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:26:38.345 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:26:38.345 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:26:38.345 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:26:38.345 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:26:38.345 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-06T07:26:38.345 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:26:38.345 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:26:38.346 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:26:38.347 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:26:38.347 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:26:38.347 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:26:38.347 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:26:38.347 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:26:38.348 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:26:38.348 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:26:38.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:26:38.497 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:26:38.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:26:38.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:38.720 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.97'), ('mon.b', '172.21.15.135'), ('mon.c', '[v2:172.21.15.135:3301,v1:172.21.15.135:6790]')] 2024-03-06T07:26:38.721 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:26:38.721 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:38.722 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:26:38.768 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.97 --add b 172.21.15.135 --addv c '[v2:172.21.15.135:3301,v1:172.21.15.135:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:38.921 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:38.921 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: generated fsid 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = quincy 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 0 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:fsid 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:last_changed 2024-03-06T07:26:38.920749+0000 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-03-06T07:26:38.920749+0000 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:min_mon_release 17 (quincy) 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:election_strategy: 1 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-03-06T07:26:38.922 INFO:teuthology.orchestra.run.smithi097.stdout:2: [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] mon.c 2024-03-06T07:26:38.923 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:26:38.925 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:26:38.972 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0b9abceb-2115-4321-96a2-348db0d0fbe4... 2024-03-06T07:26:38.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:38.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:39.135 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:26:39.135 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:26:39.135 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:26:39.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:39.137 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:26:39.138 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:26:39.139 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:26:39.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:39.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:26:39.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:26:39.143 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:26:39.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:39.145 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:26:39.146 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:26:39.147 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:26:39.148 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:26:39.148 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:26:39.148 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:26:39.148 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:26:39.149 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:26:39.149 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:26:39.149 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:39.149 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.150 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:39.151 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:26:39.152 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:39.153 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:26:39.154 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:26:39.155 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:26:39.156 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:39.157 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:26:39.158 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:26:39.159 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:26:39.160 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:26:39.190 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:26:39.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:26:39.338 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:26:39.338 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:39.338 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:26:39.386 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:39.386 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:26:39.481 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:26:39.482 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:39.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:26:39.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:39.624 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:26:39.624 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:39.718 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:26:39.718 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:39.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:26:39.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:39.869 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:26:39.869 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:39.965 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:26:39.966 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:26:39.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:26:40.146 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:26:40.156 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi135.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:26:40.355 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:26:40.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:26:40.539 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:26:40.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:26:40.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:26:40.827 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:26:40.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:26:40.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:26:41.100 INFO:teuthology.orchestra.run.smithi135.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:26:41.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:26:41.202 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:26:41.377 INFO:teuthology.orchestra.run.smithi135.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:26:41.386 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:26:41.478 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:26:41.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:26:41.609 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:26:41.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:26:41.787 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:26:41.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:26:41.959 INFO:teuthology.orchestra.run.smithi135.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:26:42.008 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:26:42.135 INFO:teuthology.orchestra.run.smithi135.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:26:42.193 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:26:42.193 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:26:42.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:42.293 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:42.293 INFO:tasks.ceph:role: osd.0 2024-03-06T07:26:42.293 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:26:42.293 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:42.395 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:42.395 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:42.395 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:42.395 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-06T07:26:42.915 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-06T07:26:42.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:43.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:43.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:43.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:43.296 INFO:tasks.ceph:role: osd.1 2024-03-06T07:26:43.296 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:26:43.296 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:43.388 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:43.389 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:43.867 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-06T07:26:43.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-06T07:26:43.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:44.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:44.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:44.191 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:44.191 INFO:tasks.ceph:role: osd.2 2024-03-06T07:26:44.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:26:44.191 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:26:44.289 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:44.819 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-06T07:26:44.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-06T07:26:44.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:45.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:45.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:45.218 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:45.218 INFO:tasks.ceph:role: osd.3 2024-03-06T07:26:45.218 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:26:45.218 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:26:45.316 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:45.316 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:45.316 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:45.317 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:45.773 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-06T07:26:45.779 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-06T07:26:45.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:45.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:46.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:46.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:46.172+0000 7f7b58721640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:26:46.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:46.173+0000 7f7b58721640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:26:46.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:46.173+0000 7f7b58721640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:46.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:46.173+0000 7f7b58721640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:46.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:46.173+0000 7f7b58721640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:46.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:46.182+0000 7f7b58721640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:26:49.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:49.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:49.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:49.593+0000 7fe586cac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:26:49.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:49.594+0000 7fe586cac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:26:49.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:49.594+0000 7fe586cac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:49.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:49.594+0000 7fe586cac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:49.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:49.594+0000 7fe586cac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:49.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:49.596+0000 7fe586cac640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:26:52.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:52.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:52.747+0000 7fb24f7f6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:26:52.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:52.748+0000 7fb24f7f6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:26:52.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:52.748+0000 7fb24f7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:26:52.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:52.748+0000 7fb24f7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:26:52.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:52.748+0000 7fb24f7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:26:52.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:52.750+0000 7fb24f7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:26:55.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:55.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:55.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:55.928+0000 7f1beac8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:26:55.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:55.929+0000 7f1beac8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:26:55.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:55.929+0000 7f1beac8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:26:55.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:55.929+0000 7f1beac8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:26:55.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:55.929+0000 7f1beac8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:26:55.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:26:55.931+0000 7f1beac8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:26:58.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:58.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:59.034 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:59.034 INFO:tasks.ceph:role: osd.4 2024-03-06T07:26:59.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:26:59.034 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:59.135 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:59.646 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-06T07:26:59.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-06T07:26:59.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:59.826 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:59.927 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:00.009 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:00.010 INFO:tasks.ceph:role: osd.5 2024-03-06T07:27:00.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:27:00.010 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:00.095 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:00.557 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-06T07:27:00.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-06T07:27:00.564 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:00.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:00.765 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:00.842 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:00.842 INFO:tasks.ceph:role: osd.6 2024-03-06T07:27:00.842 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:27:00.842 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:27:00.924 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:00.924 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:00.924 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:00.924 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:00.924 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:00.925 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:00.925 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:00.925 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:00.925 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:00.925 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:01.418 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-06T07:27:01.423 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-06T07:27:01.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:01.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:01.668 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:01.750 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:01.751 INFO:tasks.ceph:role: osd.7 2024-03-06T07:27:01.751 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:27:01.751 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:01.848 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:01.849 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:01.849 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:01.849 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-06T07:27:02.310 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-06T07:27:02.310 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:02.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:02.499 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:02.628 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:02.627+0000 7fb6b3023640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:27:02.629 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:02.628+0000 7fb6b3023640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:27:02.629 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:02.628+0000 7fb6b3023640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:27:02.629 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:02.628+0000 7fb6b3023640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:27:02.629 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:02.628+0000 7fb6b3023640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:27:02.632 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:02.631+0000 7fb6b3023640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:27:05.820 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:27:05.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:06.055 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:06.054+0000 7fe2da00f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:27:06.056 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:06.055+0000 7fe2da00f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:27:06.056 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:06.055+0000 7fe2da00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:27:06.056 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:06.055+0000 7fe2da00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:27:06.057 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:06.055+0000 7fe2da00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:27:06.058 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:06.057+0000 7fe2da00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:27:06.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:06.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:08.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:09.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:09.222 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:09.218+0000 7fa3efc1a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:27:09.222 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:09.219+0000 7fa3efc1a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:27:09.222 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:09.219+0000 7fa3efc1a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:27:09.222 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:09.219+0000 7fa3efc1a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:27:09.222 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:09.219+0000 7fa3efc1a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:27:09.224 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:09.221+0000 7fa3efc1a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:27:12.388 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:12.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:12.618 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:12.616+0000 7f76acb24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:27:12.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:12.616+0000 7f76acb24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:27:12.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:12.616+0000 7f76acb24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:27:12.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:12.617+0000 7f76acb24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:27:12.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:12.617+0000 7f76acb24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:27:12.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:12.619+0000 7f76acb24640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:27:15.799 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:15.885 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:27:15.885 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:15.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:27:15.972 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:15.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:27:16.055 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:16.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:27:16.189 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:16.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:27:16.286 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:16.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:27:16.379 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:16.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:27:16.516 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:16.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:27:16.609 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:16.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:27:16.730 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:16.730 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:27:16.819 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:16.820 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:27:16.910 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:16.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:27:17.046 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:17.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:27:17.139 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:17.140 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:27:17.230 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:17.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:27:17.319 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:17.319 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:27:17.376 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:17.376 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:27:17.426 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:17.426 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:27:17.482 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:17.482 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:27:17.534 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:27:17.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:27:17.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.y] 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ== 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA== 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ== 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ== 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2024-03-06T07:27:17.628 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.x] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.4] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.5] 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw== 2024-03-06T07:27:17.629 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.6] 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ== 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.7] 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg== 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout:[client.2] 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA== 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout:[client.1] 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA== 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout:[client.3] 2024-03-06T07:27:17.630 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag== 2024-03-06T07:27:17.633 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.y] 2024-03-06T07:27:17.633 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.0] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.1] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.2] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.3] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.x] 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag== 2024-03-06T07:27:17.634 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.4] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.5] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.6] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.7] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[client.0] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[client.2] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA== 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout:[client.1] 2024-03-06T07:27:17.635 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA== 2024-03-06T07:27:17.636 INFO:teuthology.orchestra.run.smithi135.stdout:[client.3] 2024-03-06T07:27:17.636 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag== 2024-03-06T07:27:17.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:17.680 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:17.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:17.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:17.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:17.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:18.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.136 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.287 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:18.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:18.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:18.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:18.863 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:18.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:19.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:19.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:19.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:19.732 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:19.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:19.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:20.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:20.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:20.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:20.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:20.324 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:27:20.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:27:20.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:20.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:27:20.650 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:27:20.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:20.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:27:20.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:27:21.074 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:21.227 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:27:21.297 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:21.304 DEBUG:teuthology.orchestra.run.smithi135:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:21.354 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:27:21.354 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:27:21.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:27:21.361 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:27:21.361 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:27:21.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:27:21.366 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:27:21.366 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:27:21.366 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:27:21.371 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:27:21.371 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:27:21.371 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:27:21.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:27:21.378 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:27:21.378 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:27:21.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:27:21.382 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:27:21.382 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:27:21.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:27:21.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.558+0000 7f4bb8200640 1 Processor -- start 2024-03-06T07:27:21.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.558+0000 7f4bb8200640 1 -- start start 2024-03-06T07:27:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.559+0000 7f4bb8200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.559+0000 7f4bb8200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb00719e0 con 0x7f4bb00cda90 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.559+0000 7f4bb8200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb0071cc0 con 0x7f4bb00b3df0 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.559+0000 7f4bb8200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb0072310 con 0x7f4bb00b3650 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb6200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4bb00cda90 0x7f4bb006c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52172/0 (socket says 172.21.15.97:52172) 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb6200640 1 -- 172.21.15.97:0/1232555083 learned_addr learned my addr 172.21.15.97:0/1232555083 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 msgr2=0x7f4bb00727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 msgr2=0x7f4bb00727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.560+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:27:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.762+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.762+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 msgr2=0x7f4bb00727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:27:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.762+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 msgr2=0x7f4bb00727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.762+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.763+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.806+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003553210 0 0) 0x7f4bb00719e0 con 0x7f4bb00cda90 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.807+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b98003520 con 0x7f4bb00cda90 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.807+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063964831 0 0) 0x7f4bb0071cc0 con 0x7f4bb00b3df0 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.807+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb00719e0 con 0x7f4bb00b3df0 2024-03-06T07:27:21.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.807+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612682557 0 0) 0x7f4b98003520 con 0x7f4bb00cda90 2024-03-06T07:27:21.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 msgr2=0x7f4bb00727a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:21.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4bb00727a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 >> v1:172.21.15.135:6789/0 conn(0x7f4bb00b3df0 legacy=0x7f4bb00b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb0071fa0 con 0x7f4bb00cda90 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4ba80019f0 con 0x7f4bb00cda90 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 >> v1:172.21.15.97:6789/0 conn(0x7f4bb00cda90 legacy=0x7f4bb006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4b98003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4b98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98004d40 0x7f4b980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb0072310 con 0x7f4b980040d0 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9800dd60 con 0x7f4b98004d40 2024-03-06T07:27:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.808+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9800df60 con 0x7f4bb00b3650 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.809+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4b98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.809+0000 7f4bb5800640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4b98003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.809+0000 7f4bb6200640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98004d40 0x7f4b980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.810+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 msgr2=0x7f4b98003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.810+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb00b3650 0x7f4b98003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.810+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98004d40 msgr2=0x7f4b980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.810+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98004d40 0x7f4b980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.810+0000 7f4bb4e00640 1 -- 172.21.15.97:0/1232555083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98003520 con 0x7f4b980040d0 2024-03-06T07:27:21.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.812+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4b98004800 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c006fd0 tx=0x7f4b98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0eb254fd81329bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.815+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c032070 con 0x7f4b980040d0 2024-03-06T07:27:21.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.815+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c02c7f0 con 0x7f4b980040d0 2024-03-06T07:27:21.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.815+0000 7f4bafe00640 1 -- 172.21.15.97:0/1232555083 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00d690 con 0x7f4b980040d0 2024-03-06T07:27:21.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb8200640 1 -- 172.21.15.97:0/1232555083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 msgr2=0x7f4b98004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb8200640 1 --2- 172.21.15.97:0/1232555083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4b98004800 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c006fd0 tx=0x7f4b98005830 comp rx=0 tx=0).stop 2024-03-06T07:27:21.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb6200640 1 -- 172.21.15.97:0/1232555083 reap_dead start 2024-03-06T07:27:21.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb8200640 1 -- 172.21.15.97:0/1232555083 shutdown_connections 2024-03-06T07:27:21.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb8200640 1 -- 172.21.15.97:0/1232555083 >> 172.21.15.97:0/1232555083 conn(0x7f4bb00b1940 msgr2=0x7f4bb00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb8200640 1 -- 172.21.15.97:0/1232555083 shutdown_connections 2024-03-06T07:27:21.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.817+0000 7f4bb8200640 1 -- 172.21.15.97:0/1232555083 wait complete. 2024-03-06T07:27:21.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.822+0000 7f4bb8200640 1 Processor -- start 2024-03-06T07:27:21.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.822+0000 7f4bb8200640 1 -- start start 2024-03-06T07:27:21.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.823+0000 7f4bb8200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4bb009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb8200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb014a170 0x7f4bb014a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb8200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb009c8f0 0x7f4bb009cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb8200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb015b3c0 con 0x7f4b980040d0 2024-03-06T07:27:21.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb8200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb015b240 con 0x7f4bb009c8f0 2024-03-06T07:27:21.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb8200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb00cf510 con 0x7f4bb014a170 2024-03-06T07:27:21.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4bb009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4bb009aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54772/0 (socket says 172.21.15.97:54772) 2024-03-06T07:27:21.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb5800640 1 -- 172.21.15.97:0/647857978 learned_addr learned my addr 172.21.15.97:0/647857978 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:21.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb014a170 0x7f4bb014a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.824+0000 7f4bb6200640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb009c8f0 0x7f4bb009cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.825+0000 7f4bb4e00640 1 -- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb009c8f0 msgr2=0x7f4bb009cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.825+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb009c8f0 0x7f4bb009cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.825+0000 7f4bb4e00640 1 -- 172.21.15.97:0/647857978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 msgr2=0x7f4bb009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.825+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4bb009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.825+0000 7f4bb4e00640 1 -- 172.21.15.97:0/647857978 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb009d700 con 0x7f4bb014a170 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.825+0000 7f4bb6200640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb009c8f0 0x7f4bb009cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:21.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.826+0000 7f4bb4e00640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb014a170 0x7f4bb014a5b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c034000 tx=0x7f4b9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.827+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c044020 con 0x7f4bb014a170 2024-03-06T07:27:21.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.827+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c036a40 con 0x7f4bb014a170 2024-03-06T07:27:21.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.827+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00d690 con 0x7f4bb014a170 2024-03-06T07:27:21.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.827+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb01519f0 con 0x7f4bb014a170 2024-03-06T07:27:21.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.828+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb014aaf0 con 0x7f4bb014a170 2024-03-06T07:27:21.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.828+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4b9c047020 con 0x7f4bb014a170 2024-03-06T07:27:21.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.828+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b9c0494f0 con 0x7f4bb014a170 2024-03-06T07:27:21.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.829+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb004f330 con 0x7f4bb014a170 2024-03-06T07:27:21.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.833+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4b9c04e030 con 0x7f4bb014a170 2024-03-06T07:27:21.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.933+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4bb00cc0e0 con 0x7f4bb014a170 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.940+0000 7f4bad600640 1 -- 172.21.15.97:0/647857978 <== mon.2 v2:172.21.15.135:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4b9c03f9b0 con 0x7f4bb014a170 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb014a170 msgr2=0x7f4bb014a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb014a170 0x7f4bb014a5b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c034000 tx=0x7f4b9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 shutdown_connections 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb009c8f0 0x7f4bb009cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bb014a170 0x7f4bb014a5b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 --2- 172.21.15.97:0/647857978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b980040d0 0x7f4bb009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 >> 172.21.15.97:0/647857978 conn(0x7f4bb00b1940 msgr2=0x7f4bb0156330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 shutdown_connections 2024-03-06T07:27:21.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:21.942+0000 7f4bb8200640 1 -- 172.21.15.97:0/647857978 wait complete. 2024-03-06T07:27:21.961 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:27:21.961 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:27:21.961 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:21.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:27:22.044 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:22.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:27:22.176 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:22.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:27:22.261 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:27:22.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:27:22.387 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:22.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:27:22.421 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:22.419+0000 7f7b8c412040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:27:22.431 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:22.431+0000 7f7213e3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:27:22.476 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:22.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:27:22.491 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:22.489+0000 7f7b8c412040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:27:22.493 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:22.493+0000 7f7213e3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:27:22.578 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:22.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:27:22.702 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:27:22.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:27:22.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 0f9b9c89-55e8-4987-838d-ff30970a00b1 0 2024-03-06T07:27:23.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.065+0000 7f590bc00640 1 Processor -- start 2024-03-06T07:27:23.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.065+0000 7f590bc00640 1 -- start start 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.066+0000 7f590bc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f590415e1f0 0x7f59040cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.066+0000 7f590bc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040d1d00 con 0x7f59040d0ea0 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.066+0000 7f590bc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040d1fe0 con 0x7f59040d2da0 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.066+0000 7f590bc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590415e5d0 con 0x7f590415e1f0 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f59040d2da0 0x7f59040d0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33496/0 (socket says 172.21.15.135:33496) 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903e00640 1 -- 172.21.15.135:0/586248181 learned_addr learned my addr 172.21.15.135:0/586248181 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-06T07:27:23.069 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5909200640 1 --2- 172.21.15.135:0/586248181 >> v2:172.21.15.97:3300/0 conn(0x7f590415e1f0 0x7f59040cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266798323 0 0) 0x7f59040d1fe0 con 0x7f59040d2da0 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec003520 con 0x7f59040d2da0 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425319921 0 0) 0x7f58ec003520 con 0x7f59040d2da0 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 >> v2:172.21.15.97:3300/0 conn(0x7f590415e1f0 msgr2=0x7f59040cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 --2- 172.21.15.135:0/586248181 >> v2:172.21.15.97:3300/0 conn(0x7f590415e1f0 0x7f59040cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 >> v1:172.21.15.97:6789/0 conn(0x7f59040d0ea0 legacy=0x7f59040b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040d22c0 con 0x7f59040d2da0 2024-03-06T07:27:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f58f0001670 con 0x7f59040d2da0 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 >> v1:172.21.15.135:6789/0 conn(0x7f59040d2da0 legacy=0x7f59040d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58ec003a20 0x7f58ec003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.067+0000 7f5903400640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f58ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903400640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58ec005230 0x7f58ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590415e5d0 con 0x7f58ec003a20 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec00c190 con 0x7f58ec005230 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec00c390 con 0x7f58ec004800 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5909c00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58ec005230 0x7f58ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903e00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f58ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5909200640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58ec003a20 0x7f58ec003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903e00640 1 -- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58ec005230 msgr2=0x7f58ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903e00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58ec005230 0x7f58ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903e00640 1 -- 172.21.15.135:0/586248181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58ec003a20 msgr2=0x7f58ec003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903e00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58ec003a20 0x7f58ec003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.073 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5903e00640 1 -- 172.21.15.135:0/586248181 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec003520 con 0x7f58ec004800 2024-03-06T07:27:23.074 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.068+0000 7f5909c00640 1 -- 172.21.15.135:0/586248181 reap_dead start 2024-03-06T07:27:23.074 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.069+0000 7f5903e00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f58ec004c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c240 tx=0x7f58ec005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad1bd0f546261cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.074 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.069+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0007d10 con 0x7f58ec004800 2024-03-06T07:27:23.074 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0030070 con 0x7f58ec004800 2024-03-06T07:27:23.074 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f5903400640 1 -- 172.21.15.135:0/586248181 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0005d70 con 0x7f58ec004800 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 -- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 msgr2=0x7f58ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f58ec004c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c240 tx=0x7f58ec005830 comp rx=0 tx=0).stop 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 -- 172.21.15.135:0/586248181 shutdown_connections 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 --2- 172.21.15.135:0/586248181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f58ec004c80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 -- 172.21.15.135:0/586248181 >> 172.21.15.135:0/586248181 conn(0x7f59040d3ee0 msgr2=0x7f59040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 -- 172.21.15.135:0/586248181 shutdown_connections 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.070+0000 7f590bc00640 1 -- 172.21.15.135:0/586248181 wait complete. 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.072+0000 7f590bc00640 1 Processor -- start 2024-03-06T07:27:23.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.072+0000 7f590bc00640 1 -- start start 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f590bc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f59040cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f590bc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f590415b2c0 0x7f590409e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f590bc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f590409ee20 0x7f59040a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f590bc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040d2880 con 0x7f590409ee20 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f590bc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040d1ab0 con 0x7f590415b2c0 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f590bc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040d0790 con 0x7f58ec004800 2024-03-06T07:27:23.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f5909200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f59040cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f5903e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f590415b2c0 0x7f590409e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f5909200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f59040cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41760/0 (socket says 172.21.15.135:41760) 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.073+0000 7f5909200640 1 -- 172.21.15.135:0/4155706922 learned_addr learned my addr 172.21.15.135:0/4155706922 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909c00640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f590409ee20 0x7f59040a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909200640 1 -- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f590415b2c0 msgr2=0x7f590409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909200640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f590415b2c0 0x7f590409e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909200640 1 -- 172.21.15.135:0/4155706922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f590409ee20 msgr2=0x7f59040a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909200640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f590409ee20 0x7f59040a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909200640 1 -- 172.21.15.135:0/4155706922 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040cb150 con 0x7f58ec004800 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5909200640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f59040cdad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f80118f0 tx=0x7f58f8011db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8019070 con 0x7f58ec004800 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f8004240 con 0x7f58ec004800 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59040cb430 con 0x7f58ec004800 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.074+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800f820 con 0x7f58ec004800 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.075+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59040cb9f0 con 0x7f58ec004800 2024-03-06T07:27:23.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.075+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f58f801c070 con 0x7f58ec004800 2024-03-06T07:27:23.079 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.075+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f58f801e790 con 0x7f58ec004800 2024-03-06T07:27:23.079 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.076+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590404f3a0 con 0x7f58ec004800 2024-03-06T07:27:23.080 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.078+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f58f8003da0 con 0x7f58ec004800 2024-03-06T07:27:23.109 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.108+0000 7f7213e3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:27:23.125 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.122+0000 7f7b8c412040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:27:23.160 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.157+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0f9b9c89-55e8-4987-838d-ff30970a00b1", "id": 0} v 0) v1 -- 0x7f590406c260 con 0x7f58ec004800 2024-03-06T07:27:23.165 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.163+0000 7f5900c00640 1 -- 172.21.15.135:0/4155706922 <== mon.2 v2:172.21.15.135:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f9b9c89-55e8-4987-838d-ff30970a00b1", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f58f80036e0 con 0x7f58ec004800 2024-03-06T07:27:23.166 INFO:teuthology.orchestra.run.smithi135.stdout:0 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 msgr2=0x7f59040cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f59040cdad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f80118f0 tx=0x7f58f8011db0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 shutdown_connections 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f590409ee20 0x7f59040a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f590415b2c0 0x7f590409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 --2- 172.21.15.135:0/4155706922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58ec004800 0x7f59040cdad0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 >> 172.21.15.135:0/4155706922 conn(0x7f59040d3ee0 msgr2=0x7f590406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 shutdown_connections 2024-03-06T07:27:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.165+0000 7f590bc00640 1 -- 172.21.15.135:0/4155706922 wait complete. 2024-03-06T07:27:23.173 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.172+0000 7f7213e3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:27:23.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe 1 2024-03-06T07:27:23.200 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.197+0000 7f7b8c412040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:27:23.224 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.223+0000 7f7213e3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:27:23.254 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.251+0000 7f7b8c412040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:27:23.287 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.285+0000 7f7213e3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:27:23.320 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.318+0000 7f7b8c412040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:27:23.336 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.335+0000 7f7213e3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:27:23.371 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.369+0000 7f7b8c412040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:27:23.384 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.381+0000 7f784f200640 1 Processor -- start 2024-03-06T07:27:23.384 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.381+0000 7f784f200640 1 -- start start 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.382+0000 7f784f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 0x7f78500b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f78500b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784f200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78500d4ce0 con 0x7f785015e100 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78500d4a40 con 0x7f78500d0e60 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785015e4e0 con 0x7f78500b3120 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f7847e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 0x7f78500b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784de00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f785015e100 0x7f78500d0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33504/0 (socket says 172.21.15.135:33504) 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784de00640 1 -- 172.21.15.135:0/2723181871 learned_addr learned my addr 172.21.15.135:0/2723181871 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f7847e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 0x7f78500b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56742/0 (socket says 172.21.15.135:56742) 2024-03-06T07:27:23.386 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784e800640 1 --2- 172.21.15.135:0/2723181871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f78500b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.387 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784e800640 1 -- 172.21.15.135:0/2723181871 >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 msgr2=0x7f78500b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.387 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784e800640 1 --2- 172.21.15.135:0/2723181871 >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 0x7f78500b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.387 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784e800640 1 -- 172.21.15.135:0/2723181871 >> v1:172.21.15.135:6789/0 conn(0x7f785015e100 legacy=0x7f78500d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.387 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f784e800640 1 -- 172.21.15.135:0/2723181871 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78500d2470 con 0x7f78500b3120 2024-03-06T07:27:23.387 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.383+0000 7f7847e00640 1 --2- 172.21.15.135:0/2723181871 >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 0x7f78500b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.384+0000 7f784e800640 1 --2- 172.21.15.135:0/2723181871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f78500b3cb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7848009d90 tx=0x7f784802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=658bfe8bfdbbade server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784d400640 1 -- 172.21.15.135:0/2723181871 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7848039070 con 0x7f78500b3120 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784d400640 1 -- 172.21.15.135:0/2723181871 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784802cc60 con 0x7f78500b3120 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784d400640 1 -- 172.21.15.135:0/2723181871 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7848034660 con 0x7f78500b3120 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784f200640 1 -- 172.21.15.135:0/2723181871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 msgr2=0x7f78500b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784f200640 1 --2- 172.21.15.135:0/2723181871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f78500b3cb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7848009d90 tx=0x7f784802beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784f200640 1 -- 172.21.15.135:0/2723181871 shutdown_connections 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784f200640 1 --2- 172.21.15.135:0/2723181871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f78500b3cb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784f200640 1 --2- 172.21.15.135:0/2723181871 >> v2:172.21.15.135:3300/0 conn(0x7f78500d0e60 0x7f78500b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.388 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.385+0000 7f784f200640 1 -- 172.21.15.135:0/2723181871 >> 172.21.15.135:0/2723181871 conn(0x7f78500d3d80 msgr2=0x7f78500d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.389 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.386+0000 7f784f200640 1 -- 172.21.15.135:0/2723181871 shutdown_connections 2024-03-06T07:27:23.389 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.386+0000 7f784f200640 1 -- 172.21.15.135:0/2723181871 wait complete. 2024-03-06T07:27:23.389 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.387+0000 7f784f200640 1 Processor -- start 2024-03-06T07:27:23.390 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.387+0000 7f784f200640 1 -- start start 2024-03-06T07:27:23.391 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f7850156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.391 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78500d0e60 0x7f78500cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.391 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f785015e100 0x7f78501f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78500cfbd0 con 0x7f785015e100 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78500d2a30 con 0x7f78500d0e60 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78500b2bd0 con 0x7f78500b3120 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f7850156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f7850156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41788/0 (socket says 172.21.15.135:41788) 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f7847e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78500d0e60 0x7f78500cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.388+0000 7f784de00640 1 -- 172.21.15.135:0/1123418635 learned_addr learned my addr 172.21.15.135:0/1123418635 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f784e800640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f785015e100 0x7f78501f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7847e00640 1 -- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 msgr2=0x7f7850156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7847e00640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f7850156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7847e00640 1 -- 172.21.15.135:0/1123418635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f785015e100 msgr2=0x7f78501f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7847e00640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f785015e100 0x7f78501f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7847e00640 1 -- 172.21.15.135:0/1123418635 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78500cb0a0 con 0x7f78500d0e60 2024-03-06T07:27:23.392 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f784de00640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f7850156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:23.393 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7847e00640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78500d0e60 0x7f78500cd570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f783800e900 tx=0x7f783800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.393 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838017070 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783800ca00 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783800cdf0 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.389+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78500cb380 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.390+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78500cb940 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.390+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f783801a070 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.390+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f783801c840 con 0x7f78500d0e60 2024-03-06T07:27:23.394 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.391+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78500cae30 con 0x7f78500d0e60 2024-03-06T07:27:23.396 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.394+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7838009330 con 0x7f78500d0e60 2024-03-06T07:27:23.465 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.464+0000 7f7213e3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:27:23.475 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.473+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe", "id": 1} v 0) v1 -- 0x7f78500753e0 con 0x7f78500d0e60 2024-03-06T07:27:23.483 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.480+0000 7f7846000640 1 -- 172.21.15.135:0/1123418635 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f783801cb10 con 0x7f78500d0e60 2024-03-06T07:27:23.483 INFO:teuthology.orchestra.run.smithi135.stdout:1 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78500d0e60 msgr2=0x7f78500cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78500d0e60 0x7f78500cd570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f783800e900 tx=0x7f783800edc0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 shutdown_connections 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f785015e100 0x7f78501f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78500d0e60 0x7f78500cd570 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 --2- 172.21.15.135:0/1123418635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78500b3120 0x7f7850156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 >> 172.21.15.135:0/1123418635 conn(0x7f78500d3d80 msgr2=0x7f78500b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 shutdown_connections 2024-03-06T07:27:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.482+0000 7f784f200640 1 -- 172.21.15.135:0/1123418635 wait complete. 2024-03-06T07:27:23.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 9ec347a4-1598-4f0c-825f-6a5f130994d9 2 2024-03-06T07:27:23.509 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.506+0000 7f7b8c412040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:27:23.545 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.544+0000 7f7213e3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:23.594 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.592+0000 7f7b8c412040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:23.598 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.597+0000 7f7213e3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:27:23.651 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.648+0000 7f7b8c412040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:27:23.657 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.656+0000 7f7213e3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:27:23.698 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.695+0000 7f0442800640 1 Processor -- start 2024-03-06T07:27:23.698 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.695+0000 7f0442800640 1 -- start start 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0442800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0442800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f043c0cf880 0x7f043c0d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0442800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0d0dc0 con 0x7f043c0d2730 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0442800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0ceac0 con 0x7f043c0cf880 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0442800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0ceda0 con 0x7f043c0d04a0 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0440a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f043c0cf880 0x7f043c0d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0441400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.700 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0440a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f043c0cf880 0x7f043c0d21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56762/0 (socket says 172.21.15.135:56762) 2024-03-06T07:27:23.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.697+0000 7f0440a00640 1 -- 172.21.15.135:0/1958147963 learned_addr learned my addr 172.21.15.135:0/1958147963 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:23.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.698+0000 7f0440a00640 1 -- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 msgr2=0x7f043c0d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.698+0000 7f0440a00640 1 --2- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.698+0000 7f0440a00640 1 -- 172.21.15.135:0/1958147963 >> v1:172.21.15.97:6789/0 conn(0x7f043c0d2730 legacy=0x7f043c0b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.698+0000 7f0440a00640 1 -- 172.21.15.135:0/1958147963 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0cf080 con 0x7f043c0cf880 2024-03-06T07:27:23.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.699+0000 7f0440a00640 1 --2- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0d21f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0428009870 tx=0x7f042802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4c62922885741ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.699+0000 7f043be00640 1 -- 172.21.15.135:0/1958147963 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428034070 con 0x7f043c0cf880 2024-03-06T07:27:23.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.699+0000 7f043be00640 1 -- 172.21.15.135:0/1958147963 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0428030840 con 0x7f043c0cf880 2024-03-06T07:27:23.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.699+0000 7f043be00640 1 -- 172.21.15.135:0/1958147963 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428030c10 con 0x7f043c0cf880 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 -- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 msgr2=0x7f043c0d21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 --2- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0d21f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0428009870 tx=0x7f042802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 -- 172.21.15.135:0/1958147963 shutdown_connections 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 --2- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0d21f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 --2- 172.21.15.135:0/1958147963 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 -- 172.21.15.135:0/1958147963 >> 172.21.15.135:0/1958147963 conn(0x7f043c0d3c40 msgr2=0x7f043c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 -- 172.21.15.135:0/1958147963 shutdown_connections 2024-03-06T07:27:23.703 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.700+0000 7f0442800640 1 -- 172.21.15.135:0/1958147963 wait complete. 2024-03-06T07:27:23.704 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.701+0000 7f0442800640 1 Processor -- start 2024-03-06T07:27:23.704 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.702+0000 7f0442800640 1 -- start start 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.702+0000 7f0442800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0442800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0442800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c0d2730 0x7f043c0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0442800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0d4ab0 con 0x7f043c0d2730 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0442800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0d4930 con 0x7f043c0cf880 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0442800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0ce6e0 con 0x7f043c0d04a0 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0440a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56778/0 (socket says 172.21.15.135:56778) 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0440a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41810/0 (socket says 172.21.15.135:41810) 2024-03-06T07:27:23.707 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 -- 172.21.15.135:0/3720639288 learned_addr learned my addr 172.21.15.135:0/3720639288 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441e00640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c0d2730 0x7f043c0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 -- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 msgr2=0x7f043c0b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 -- 172.21.15.135:0/3720639288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c0d2730 msgr2=0x7f043c0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c0d2730 0x7f043c0c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.703+0000 7f0441400640 1 -- 172.21.15.135:0/3720639288 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0b7770 con 0x7f043c0cf880 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0441400640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0cd790 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f043000e930 tx=0x7f043000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430017070 con 0x7f043c0cf880 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043000ca00 con 0x7f043c0cf880 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c0b79f0 con 0x7f043c0cf880 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043000cdf0 con 0x7f043c0cf880 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c0b8480 con 0x7f043c0cf880 2024-03-06T07:27:23.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f043001a070 con 0x7f043c0cf880 2024-03-06T07:27:23.709 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.704+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f0430006700 con 0x7f043c0cf880 2024-03-06T07:27:23.709 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.705+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c0b7fb0 con 0x7f043c0cf880 2024-03-06T07:27:23.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.708+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f043c0b7770 con 0x7f043c0cf880 2024-03-06T07:27:23.712 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:23.711+0000 7f7213e3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:27:23.715 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.713+0000 7f7b8c412040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:27:23.772 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:23.769+0000 7f7b8c412040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:27:23.789 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.787+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9ec347a4-1598-4f0c-825f-6a5f130994d9", "id": 2} v 0) v1 -- 0x7f043c0753e0 con 0x7f043c0cf880 2024-03-06T07:27:23.796 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.794+0000 7f0439600640 1 -- 172.21.15.135:0/3720639288 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9ec347a4-1598-4f0c-825f-6a5f130994d9", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f04300060f0 con 0x7f043c0cf880 2024-03-06T07:27:23.796 INFO:teuthology.orchestra.run.smithi135.stdout:2 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 msgr2=0x7f043c0cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0cd790 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f043000e930 tx=0x7f043000edf0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 shutdown_connections 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c0d2730 0x7f043c0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0d04a0 0x7f043c0b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 --2- 172.21.15.135:0/3720639288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0cf880 0x7f043c0cd790 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 >> 172.21.15.135:0/3720639288 conn(0x7f043c0d3c40 msgr2=0x7f043c0b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 shutdown_connections 2024-03-06T07:27:23.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.795+0000 7f0442800640 1 -- 172.21.15.135:0/3720639288 wait complete. 2024-03-06T07:27:23.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new e2eeb450-3ac2-4a2f-89c2-9d82c39f094f 3 2024-03-06T07:27:24.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:23.999+0000 7f6b7e000640 1 Processor -- start 2024-03-06T07:27:24.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.000+0000 7f6b7e000640 1 -- start start 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.001+0000 7f6b7e000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b7815e1f0 0x7f6b780cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.001+0000 7f6b7e000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b780d1d00 con 0x7f6b780d2da0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.001+0000 7f6b7e000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b780d1fe0 con 0x7f6b780d0ea0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.001+0000 7f6b7e000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7815e5d0 con 0x7f6b7815e1f0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.001+0000 7f6b77e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f6b780d0ea0 0x7f6b780b6d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33508/0 (socket says 172.21.15.135:33508) 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.001+0000 7f6b77e00640 1 -- 172.21.15.135:0/4084152648 learned_addr learned my addr 172.21.15.135:0/4084152648 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b77400640 1 --2- 172.21.15.135:0/4084152648 >> v2:172.21.15.97:3300/0 conn(0x7f6b7815e1f0 0x7f6b780cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093564533 0 0) 0x7f6b780d1fe0 con 0x7f6b780d0ea0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5c003520 con 0x7f6b780d0ea0 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713056697 0 0) 0x7f6b5c003520 con 0x7f6b780d0ea0 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 >> v2:172.21.15.97:3300/0 conn(0x7f6b7815e1f0 msgr2=0x7f6b780cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 --2- 172.21.15.135:0/4084152648 >> v2:172.21.15.97:3300/0 conn(0x7f6b7815e1f0 0x7f6b780cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 >> v1:172.21.15.97:6789/0 conn(0x7f6b780d2da0 legacy=0x7f6b780d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b780d22c0 con 0x7f6b780d0ea0 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f6b700019f0 con 0x7f6b780d0ea0 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 >> v1:172.21.15.135:6789/0 conn(0x7f6b780d0ea0 legacy=0x7f6b780b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5c003a20 0x7f6b5c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b5c004800 0x7f6b5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7815e5d0 con 0x7f6b5c003a20 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c00c190 con 0x7f6b5c005230 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.002+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c00c390 con 0x7f6b5c004800 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77e00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b6fe00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b5c004800 0x7f6b5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77400640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5c003a20 0x7f6b5c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77e00640 1 -- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b5c004800 msgr2=0x7f6b5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77e00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b5c004800 0x7f6b5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77e00640 1 -- 172.21.15.135:0/4084152648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5c003a20 msgr2=0x7f6b5c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77e00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5c003a20 0x7f6b5c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.003+0000 7f6b77e00640 1 -- 172.21.15.135:0/4084152648 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c003520 con 0x7f6b5c005230 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.004+0000 7f6b77e00640 1 -- 172.21.15.135:0/4084152648 reap_dead start 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.004+0000 7f6b77e00640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b5c00b9c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000a860 tx=0x7f6b5c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8482bdeb35295ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.004+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000a480 con 0x7f6b5c005230 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.004+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b70034630 con 0x7f6b5c005230 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.004+0000 7f6b76a00640 1 -- 172.21.15.135:0/4084152648 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70034a00 con 0x7f6b5c005230 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 -- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 msgr2=0x7f6b5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b5c00b9c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000a860 tx=0x7f6b5c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 -- 172.21.15.135:0/4084152648 shutdown_connections 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 --2- 172.21.15.135:0/4084152648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b5c00b9c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 -- 172.21.15.135:0/4084152648 >> 172.21.15.135:0/4084152648 conn(0x7f6b780d3ee0 msgr2=0x7f6b780d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 -- 172.21.15.135:0/4084152648 shutdown_connections 2024-03-06T07:27:24.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.005+0000 7f6b7e000640 1 -- 172.21.15.135:0/4084152648 wait complete. 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.007+0000 7f6b7e000640 1 Processor -- start 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.007+0000 7f6b7e000640 1 -- start start 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b7e000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b7814ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b7e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b780cced0 0x7f6b780b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b7e000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b7814e1f0 0x7f6b7815fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b7e000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780d2880 con 0x7f6b780cced0 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b7e000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780d1ab0 con 0x7f6b5c005230 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b7e000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780d0790 con 0x7f6b7814e1f0 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b7814ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b7814ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56784/0 (socket says 172.21.15.135:56784) 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 -- 172.21.15.135:0/1817316369 learned_addr learned my addr 172.21.15.135:0/1817316369 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77e00640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b7814e1f0 0x7f6b7815fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b6fe00640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b780cced0 0x7f6b780b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 -- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b7814e1f0 msgr2=0x7f6b7815fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b7814e1f0 0x7f6b7815fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 -- 172.21.15.135:0/1817316369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b780cced0 msgr2=0x7f6b780b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b780cced0 0x7f6b780b89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77400640 1 -- 172.21.15.135:0/1817316369 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b780cd470 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.008+0000 7f6b77e00640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b7814e1f0 0x7f6b7815fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.009+0000 7f6b77400640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b7814ec50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6b68011910 tx=0x7f6b68011dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.009+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800faf0 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.009+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6800fc90 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.009+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68014440 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.009+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b780cd750 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.009+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b780cdd10 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.010+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6b68012040 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.010+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7804f3a0 con 0x7f6b5c005230 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.010+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6b6801db10 con 0x7f6b5c005230 2024-03-06T07:27:24.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.013+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6b68019030 con 0x7f6b5c005230 2024-03-06T07:27:24.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.092+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e2eeb450-3ac2-4a2f-89c2-9d82c39f094f", "id": 3} v 0) v1 -- 0x7f6b7806d3b0 con 0x7f6b5c005230 2024-03-06T07:27:24.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.100+0000 7f6b6f400640 1 -- 172.21.15.135:0/1817316369 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2eeb450-3ac2-4a2f-89c2-9d82c39f094f", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6b6801d680 con 0x7f6b5c005230 2024-03-06T07:27:24.102 INFO:teuthology.orchestra.run.smithi135.stdout:3 2024-03-06T07:27:24.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 msgr2=0x7f6b7814ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b7814ec50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6b68011910 tx=0x7f6b68011dd0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.106 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 shutdown_connections 2024-03-06T07:27:24.106 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b7814e1f0 0x7f6b7815fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.107 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b780cced0 0x7f6b780b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.107 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 --2- 172.21.15.135:0/1817316369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5c005230 0x7f6b7814ec50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.107 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 >> 172.21.15.135:0/1817316369 conn(0x7f6b780d3ee0 msgr2=0x7f6b7815c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.107 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.101+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 shutdown_connections 2024-03-06T07:27:24.107 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.102+0000 7f6b7e000640 1 -- 172.21.15.135:0/1817316369 wait complete. 2024-03-06T07:27:24.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 80379238-cfbc-4c68-85b8-d71da22e550b 4 2024-03-06T07:27:24.161 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.160+0000 7f7213e3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:27:24.252 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:24.249+0000 7f7b8c412040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:27:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.308+0000 7f4dc3800640 1 Processor -- start 2024-03-06T07:27:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.308+0000 7f4dc3800640 1 -- start start 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dc3800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4dbc0d2da0 0x7f4dbc0d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dc3800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4dbc0d0cd0 0x7f4dbc0b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dc3800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc0d1d00 con 0x7f4dbc15e1f0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dc3800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc15e5d0 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dc3800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc0d1ab0 con 0x7f4dbc0d2da0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4dbc0d2da0 0x7f4dbc0d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4dbc0d2da0 0x7f4dbc0d0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56786/0 (socket says 172.21.15.135:56786) 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 -- 172.21.15.135:0/3176434682 learned_addr learned my addr 172.21.15.135:0/3176434682 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dc1800640 1 --2- 172.21.15.135:0/3176434682 >> v2:172.21.15.97:3300/0 conn(0x7f4dbc0d0cd0 0x7f4dbc0b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 -- 172.21.15.135:0/3176434682 >> v2:172.21.15.97:3300/0 conn(0x7f4dbc0d0cd0 msgr2=0x7f4dbc0b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 --2- 172.21.15.135:0/3176434682 >> v2:172.21.15.97:3300/0 conn(0x7f4dbc0d0cd0 0x7f4dbc0b6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 -- 172.21.15.135:0/3176434682 >> v1:172.21.15.97:6789/0 conn(0x7f4dbc15e1f0 legacy=0x7f4dbc0cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.310+0000 7f4dbbe00640 1 -- 172.21.15.135:0/3176434682 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc0d1fe0 con 0x7f4dbc0d2da0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.311+0000 7f4dbbe00640 1 --2- 172.21.15.135:0/3176434682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0d0790 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0a1df0 tx=0x7f4da802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dbb25735c4454a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dbb400640 1 -- 172.21.15.135:0/3176434682 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da802e070 con 0x7f4dbc0d2da0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dbb400640 1 -- 172.21.15.135:0/3176434682 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da8037070 con 0x7f4dbc0d2da0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dbb400640 1 -- 172.21.15.135:0/3176434682 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8031a00 con 0x7f4dbc0d2da0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 -- 172.21.15.135:0/3176434682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc0d2da0 msgr2=0x7f4dbc0d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 --2- 172.21.15.135:0/3176434682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0d0790 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0a1df0 tx=0x7f4da802b880 comp rx=0 tx=0).stop 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 -- 172.21.15.135:0/3176434682 shutdown_connections 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 --2- 172.21.15.135:0/3176434682 >> v2:172.21.15.97:3300/0 conn(0x7f4dbc0d0cd0 0x7f4dbc0b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 --2- 172.21.15.135:0/3176434682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0d0790 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 -- 172.21.15.135:0/3176434682 >> 172.21.15.135:0/3176434682 conn(0x7f4dbc0d3ee0 msgr2=0x7f4dbc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.312+0000 7f4dc3800640 1 -- 172.21.15.135:0/3176434682 shutdown_connections 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.313+0000 7f4dc3800640 1 -- 172.21.15.135:0/3176434682 wait complete. 2024-03-06T07:27:24.317 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.314+0000 7f4dc3800640 1 Processor -- start 2024-03-06T07:27:24.317 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.314+0000 7f4dc3800640 1 -- start start 2024-03-06T07:27:24.319 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc3800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc0d0cd0 0x7f4dbc0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.319 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc3800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.319 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc3800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 0x7f4dbc1f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.319 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc3800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0d2880 con 0x7f4dbc0d2da0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc3800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0d25a0 con 0x7f4dbc15e1f0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc3800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0cfd10 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc1800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 0x7f4dbc1f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc0d0cd0 0x7f4dbc0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc1800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 0x7f4dbc1f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56792/0 (socket says 172.21.15.135:56792) 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.315+0000 7f4dc1800640 1 -- 172.21.15.135:0/2795307801 learned_addr learned my addr 172.21.15.135:0/2795307801 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dbbe00640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc0e00640 1 -- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 msgr2=0x7f4dbc1f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc0e00640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 0x7f4dbc1f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc0e00640 1 -- 172.21.15.135:0/2795307801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc0d2da0 msgr2=0x7f4dbc0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc0e00640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc0e00640 1 -- 172.21.15.135:0/2795307801 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc0b1bb0 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dbbe00640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc1800640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 0x7f4dbc1f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc0e00640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc0d0cd0 0x7f4dbc0cda20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4db000b410 tx=0x7f4db000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0015020 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db0011830 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.316+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc0cb130 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.317+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0011c60 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.317+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc0cb6f0 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.317+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4db0018020 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.317+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f4db0011e00 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.322 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.319+0000 7f7213e3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:27:24.322 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.318+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc04f3a0 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.320+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4db001f030 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.402 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.399+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "osd new", "uuid": "80379238-cfbc-4c68-85b8-d71da22e550b", "id": 4} v 0) v1 -- 0x7f4dbc08d940 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.409 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.407+0000 7f4db8c00640 1 -- 172.21.15.135:0/2795307801 <== mon.2 v2:172.21.15.135:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "80379238-cfbc-4c68-85b8-d71da22e550b", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4db001a860 con 0x7f4dbc0d0cd0 2024-03-06T07:27:24.410 INFO:teuthology.orchestra.run.smithi135.stdout:4 2024-03-06T07:27:24.411 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc0d0cd0 msgr2=0x7f4dbc0cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.411 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc0d0cd0 0x7f4dbc0cda20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4db000b410 tx=0x7f4db000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.411 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 shutdown_connections 2024-03-06T07:27:24.411 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15e1f0 0x7f4dbc1f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.411 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc0d2da0 0x7f4dbc0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.412 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 --2- 172.21.15.135:0/2795307801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc0d0cd0 0x7f4dbc0cda20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.412 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 >> 172.21.15.135:0/2795307801 conn(0x7f4dbc0d3ee0 msgr2=0x7f4dbc06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.412 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.408+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 shutdown_connections 2024-03-06T07:27:24.412 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.409+0000 7f4dc3800640 1 -- 172.21.15.135:0/2795307801 wait complete. 2024-03-06T07:27:24.421 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:24.419+0000 7f7b8c412040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:27:24.427 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 2ff0b5f7-e87d-411b-9a5a-79041279f3ad 5 2024-03-06T07:27:24.459 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.458+0000 7f7213e3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:27:24.515 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.512+0000 7f7213e3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:27:24.568 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:24.566+0000 7f7b8c412040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:27:24.608 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.606+0000 7fdd4e200640 1 Processor -- start 2024-03-06T07:27:24.609 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.606+0000 7fdd4e200640 1 -- start start 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.607+0000 7fdd4e200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdd4815df40 0x7fdd480af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.607+0000 7fdd4e200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd4806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd4e200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd480b0a60 con 0x7fdd4806b500 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd4e200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4806b950 con 0x7fdd4815df40 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd4e200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480b0d40 con 0x7fdd480af830 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd4806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd4806afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41822/0 (socket says 172.21.15.135:41822) 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 -- 172.21.15.135:0/627279236 learned_addr learned my addr 172.21.15.135:0/627279236 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd4ce00640 1 --2- 172.21.15.135:0/627279236 >> v2:172.21.15.97:3300/0 conn(0x7fdd4815df40 0x7fdd480af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.612 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 -- 172.21.15.135:0/627279236 >> v2:172.21.15.97:3300/0 conn(0x7fdd4815df40 msgr2=0x7fdd480af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.612 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 --2- 172.21.15.135:0/627279236 >> v2:172.21.15.97:3300/0 conn(0x7fdd4815df40 0x7fdd480af2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.612 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 -- 172.21.15.135:0/627279236 >> v1:172.21.15.97:6789/0 conn(0x7fdd4806b500 legacy=0x7fdd480b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.612 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.608+0000 7fdd47e00640 1 -- 172.21.15.135:0/627279236 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480b1020 con 0x7fdd480af830 2024-03-06T07:27:24.612 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.609+0000 7fdd47e00640 1 --2- 172.21.15.135:0/627279236 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd4806afc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009870 tx=0x7fdd3402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c753ad736ff59ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.610+0000 7fdd47400640 1 -- 172.21.15.135:0/627279236 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd340317c0 con 0x7fdd480af830 2024-03-06T07:27:24.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.610+0000 7fdd47400640 1 -- 172.21.15.135:0/627279236 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34037070 con 0x7fdd480af830 2024-03-06T07:27:24.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.610+0000 7fdd47400640 1 -- 172.21.15.135:0/627279236 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34031b80 con 0x7fdd480af830 2024-03-06T07:27:24.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.610+0000 7fdd4e200640 1 -- 172.21.15.135:0/627279236 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 msgr2=0x7fdd4806afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.610+0000 7fdd4e200640 1 --2- 172.21.15.135:0/627279236 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd4806afc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009870 tx=0x7fdd3402b860 comp rx=0 tx=0).stop 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.611+0000 7fdd4e200640 1 -- 172.21.15.135:0/627279236 shutdown_connections 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.611+0000 7fdd4e200640 1 --2- 172.21.15.135:0/627279236 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd4806afc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.611+0000 7fdd4e200640 1 --2- 172.21.15.135:0/627279236 >> v2:172.21.15.97:3300/0 conn(0x7fdd4815df40 0x7fdd480af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.611+0000 7fdd4e200640 1 -- 172.21.15.135:0/627279236 >> 172.21.15.135:0/627279236 conn(0x7fdd480b3c70 msgr2=0x7fdd480b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.611+0000 7fdd4e200640 1 -- 172.21.15.135:0/627279236 shutdown_connections 2024-03-06T07:27:24.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.611+0000 7fdd4e200640 1 -- 172.21.15.135:0/627279236 wait complete. 2024-03-06T07:27:24.616 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.613+0000 7fdd4e200640 1 Processor -- start 2024-03-06T07:27:24.616 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.613+0000 7fdd4e200640 1 -- start start 2024-03-06T07:27:24.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4e200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd4806b500 0x7fdd4815e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4e200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd481f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4e200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd4815df40 0x7fdd480d0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4e200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480b18c0 con 0x7fdd4806b500 2024-03-06T07:27:24.619 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4e200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480b15e0 con 0x7fdd4815df40 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4e200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4815e320 con 0x7fdd480af830 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd47e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd481f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4d800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd4815df40 0x7fdd480d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd47e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd481f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41826/0 (socket says 172.21.15.135:41826) 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd47e00640 1 -- 172.21.15.135:0/861495488 learned_addr learned my addr 172.21.15.135:0/861495488 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.615+0000 7fdd4ce00640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd4806b500 0x7fdd4815e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4d800640 1 -- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 msgr2=0x7fdd481f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4d800640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd481f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4d800640 1 -- 172.21.15.135:0/861495488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd4806b500 msgr2=0x7fdd4815e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4d800640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd4806b500 0x7fdd4815e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4d800640 1 -- 172.21.15.135:0/861495488 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd481f42a0 con 0x7fdd4815df40 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd47e00640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd481f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.620 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4d800640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd4815df40 0x7fdd480d0330 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000be20 tx=0x7fdd4000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40019070 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd40014440 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40014870 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.616+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4814bf40 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.617+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4814c500 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.617+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdd4001c070 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.617+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fdd4001e440 con 0x7fdd4815df40 2024-03-06T07:27:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.618+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd48068e40 con 0x7fdd4815df40 2024-03-06T07:27:24.622 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.620+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdd48068e40 con 0x7fdd4815df40 2024-03-06T07:27:24.627 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:24.625+0000 7f7b8c412040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:27:24.640 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.639+0000 7f7213e3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:24.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.698+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2ff0b5f7-e87d-411b-9a5a-79041279f3ad", "id": 5} v 0) v1 -- 0x7fdd480753e0 con 0x7fdd4815df40 2024-03-06T07:27:24.708 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.705+0000 7fdd44c00640 1 -- 172.21.15.135:0/861495488 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ff0b5f7-e87d-411b-9a5a-79041279f3ad", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdd40023020 con 0x7fdd4815df40 2024-03-06T07:27:24.708 INFO:teuthology.orchestra.run.smithi135.stdout:5 2024-03-06T07:27:24.709 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd4815df40 msgr2=0x7fdd480d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.709 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd4815df40 0x7fdd480d0330 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000be20 tx=0x7fdd4000bf10 comp rx=0 tx=0).stop 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 shutdown_connections 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd4815df40 0x7fdd480d0330 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd480af830 0x7fdd481f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 --2- 172.21.15.135:0/861495488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd4806b500 0x7fdd4815e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 >> 172.21.15.135:0/861495488 conn(0x7fdd480b3c70 msgr2=0x7fdd481527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 shutdown_connections 2024-03-06T07:27:24.710 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.707+0000 7fdd4e200640 1 -- 172.21.15.135:0/861495488 wait complete. 2024-03-06T07:27:24.732 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 6b6b33d9-7d8a-4717-aae5-22e93e0c815e 6 2024-03-06T07:27:24.751 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.750+0000 7f7213e3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:27:24.768 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:24.765+0000 7f7b8c412040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:24.886 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:24.883+0000 7f7b8c412040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:27:24.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.920+0000 7efc99800640 1 Processor -- start 2024-03-06T07:27:24.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.921+0000 7efc99800640 1 -- start start 2024-03-06T07:27:24.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.922+0000 7efc99800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efc9415bef0 0x7efc9415cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc99800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9415c2d0 con 0x7efc940ce270 2024-03-06T07:27:24.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc99800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940d03e0 con 0x7efc940d2170 2024-03-06T07:27:24.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc99800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9415d400 con 0x7efc9415bef0 2024-03-06T07:27:24.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc98e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7efc940d2170 0x7efc940cd410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33510/0 (socket says 172.21.15.135:33510) 2024-03-06T07:27:24.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc98e00640 1 -- 172.21.15.135:0/4108324110 learned_addr learned my addr 172.21.15.135:0/4108324110 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-06T07:27:24.928 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc93e00640 1 --2- 172.21.15.135:0/4108324110 >> v2:172.21.15.97:3300/0 conn(0x7efc9415bef0 0x7efc9415cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773757119 0 0) 0x7efc8c001490 con 0x7efc940d2170 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.923+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc78003520 con 0x7efc940d2170 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106536382 0 0) 0x7efc78003520 con 0x7efc940d2170 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 >> v2:172.21.15.97:3300/0 conn(0x7efc9415bef0 msgr2=0x7efc9415cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 --2- 172.21.15.135:0/4108324110 >> v2:172.21.15.97:3300/0 conn(0x7efc9415bef0 0x7efc9415cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 >> v1:172.21.15.97:6789/0 conn(0x7efc940ce270 legacy=0x7efc940d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940ce770 con 0x7efc940d2170 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7efc8c001bc0 con 0x7efc940d2170 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 >> v1:172.21.15.135:6789/0 conn(0x7efc940d2170 legacy=0x7efc940cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc78003a20 0x7efc78003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc78004800 0x7efc78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc780051c0 0x7efc7800fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.929 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9415d400 con 0x7efc78004800 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc78010500 con 0x7efc780051c0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc78010700 con 0x7efc78003a20 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc98e00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc780051c0 0x7efc7800fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc93e00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc78003a20 0x7efc78003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.924+0000 7efc8be00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc78004800 0x7efc78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.925+0000 7efc98e00640 1 -- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc78003a20 msgr2=0x7efc78003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.925+0000 7efc98e00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc78003a20 0x7efc78003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.925+0000 7efc98e00640 1 -- 172.21.15.135:0/4108324110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc78004800 msgr2=0x7efc78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.925+0000 7efc98e00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc78004800 0x7efc78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.925+0000 7efc98e00640 1 -- 172.21.15.135:0/4108324110 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc78003520 con 0x7efc780051c0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.925+0000 7efc8be00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc78004800 0x7efc78004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.926+0000 7efc98e00640 1 -- 172.21.15.135:0/4108324110 reap_dead start 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.926+0000 7efc98e00640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc780051c0 0x7efc7800fdd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc940d2b10 tx=0x7efc78005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8356a89ccf572500 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.926+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c00a670 con 0x7efc780051c0 2024-03-06T07:27:24.930 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.926+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c030070 con 0x7efc780051c0 2024-03-06T07:27:24.931 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.926+0000 7efc93400640 1 -- 172.21.15.135:0/4108324110 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c034870 con 0x7efc780051c0 2024-03-06T07:27:24.931 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.927+0000 7f7213e3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:24.932 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 -- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc780051c0 msgr2=0x7efc7800fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.932 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc780051c0 0x7efc7800fdd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc940d2b10 tx=0x7efc78005830 comp rx=0 tx=0).stop 2024-03-06T07:27:24.932 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 -- 172.21.15.135:0/4108324110 shutdown_connections 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 --2- 172.21.15.135:0/4108324110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc780051c0 0x7efc7800fdd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 -- 172.21.15.135:0/4108324110 >> 172.21.15.135:0/4108324110 conn(0x7efc9409a9f0 msgr2=0x7efc9409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 -- 172.21.15.135:0/4108324110 shutdown_connections 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.927+0000 7efc99800640 1 -- 172.21.15.135:0/4108324110 wait complete. 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.929+0000 7efc99800640 1 Processor -- start 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.929+0000 7efc99800640 1 -- start start 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc99800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc780051c0 0x7efc940cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc99800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc940c90b0 0x7efc9415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc99800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc940c9530 0x7efc9415b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc99800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc940ced30 con 0x7efc780051c0 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc99800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc940cf380 con 0x7efc940c90b0 2024-03-06T07:27:24.933 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc99800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc940cf010 con 0x7efc940c9530 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc98e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc940c9530 0x7efc9415b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc8be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc940c90b0 0x7efc9415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc98e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc940c9530 0x7efc9415b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41854/0 (socket says 172.21.15.135:41854) 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc98e00640 1 -- 172.21.15.135:0/659196923 learned_addr learned my addr 172.21.15.135:0/659196923 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc93e00640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc780051c0 0x7efc940cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc8be00640 1 -- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc940c9530 msgr2=0x7efc9415b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc8be00640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc940c9530 0x7efc9415b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.935 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc8be00640 1 -- 172.21.15.135:0/659196923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc780051c0 msgr2=0x7efc940cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc8be00640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc780051c0 0x7efc940cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc8be00640 1 -- 172.21.15.135:0/659196923 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940b6da0 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.930+0000 7efc93e00640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc780051c0 0x7efc940cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc8be00640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc940c90b0 0x7efc9415ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00e410 tx=0x7efc7c00e810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c019070 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c0052e0 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c003810 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9415e220 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9415ed00 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7efc7c01c070 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.931+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7efc7c014480 con 0x7efc940c90b0 2024-03-06T07:27:24.936 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.932+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9406e100 con 0x7efc940c90b0 2024-03-06T07:27:24.937 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:24.934+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7efc9406e100 con 0x7efc940c90b0 2024-03-06T07:27:24.979 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:24.977+0000 7f7213e3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:25.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.013+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6b6b33d9-7d8a-4717-aae5-22e93e0c815e", "id": 6} v 0) v1 -- 0x7efc940753e0 con 0x7efc940c90b0 2024-03-06T07:27:25.023 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.021+0000 7efc90c00640 1 -- 172.21.15.135:0/659196923 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b6b33d9-7d8a-4717-aae5-22e93e0c815e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7efc9415c2d0 con 0x7efc940c90b0 2024-03-06T07:27:25.023 INFO:teuthology.orchestra.run.smithi135.stdout:6 2024-03-06T07:27:25.025 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc940c90b0 msgr2=0x7efc9415ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.025 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc940c90b0 0x7efc9415ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00e410 tx=0x7efc7c00e810 comp rx=0 tx=0).stop 2024-03-06T07:27:25.025 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 shutdown_connections 2024-03-06T07:27:25.025 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc940c9530 0x7efc9415b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.025 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc940c90b0 0x7efc9415ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.025 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 --2- 172.21.15.135:0/659196923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc780051c0 0x7efc940cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.026 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 >> 172.21.15.135:0/659196923 conn(0x7efc9409a9f0 msgr2=0x7efc9406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.026 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 shutdown_connections 2024-03-06T07:27:25.026 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.022+0000 7efc99800640 1 -- 172.21.15.135:0/659196923 wait complete. 2024-03-06T07:27:25.041 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new ff93b254-4b0f-4580-b2e8-cac17574c5ee 7 2024-03-06T07:27:25.073 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:25.070+0000 7f7b8c412040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:25.127 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:25.123+0000 7f7b8c412040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:25.153 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:25.151+0000 7f7213e3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:27:25.259 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.256+0000 7ff550400640 1 Processor -- start 2024-03-06T07:27:25.259 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.256+0000 7ff550400640 1 -- start start 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff550400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff5480cf190 0x7ff5480cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff550400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 0x7ff548165e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff550400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5480d0660 con 0x7ff5480c7560 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff550400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5480d0410 con 0x7ff5480cf190 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff550400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480d0940 con 0x7ff548163a70 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 0x7ff548165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff5480cf190 0x7ff5480cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 0x7ff548165e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.135:41858/0 (socket says 172.21.15.135:41858) 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 -- 172.21.15.135:0/3801975017 learned_addr learned my addr 172.21.15.135:0/3801975017 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 -- 172.21.15.135:0/3801975017 >> v2:172.21.15.135:3300/0 conn(0x7ff5480cf190 msgr2=0x7ff5480cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 --2- 172.21.15.135:0/3801975017 >> v2:172.21.15.135:3300/0 conn(0x7ff5480cf190 0x7ff5480cfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 -- 172.21.15.135:0/3801975017 >> v1:172.21.15.97:6789/0 conn(0x7ff5480c7560 legacy=0x7ff5480c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.258+0000 7ff54e400640 1 -- 172.21.15.135:0/3801975017 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480d0c20 con 0x7ff548163a70 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.260+0000 7ff54e400640 1 --2- 172.21.15.135:0/3801975017 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 0x7ff548165e60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff540009e50 tx=0x7ff54002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6127c14fed6b3b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.260+0000 7ff547e00640 1 -- 172.21.15.135:0/3801975017 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54002c9e0 con 0x7ff548163a70 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.260+0000 7ff547e00640 1 -- 172.21.15.135:0/3801975017 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54002cb80 con 0x7ff548163a70 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.260+0000 7ff547e00640 1 -- 172.21.15.135:0/3801975017 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5400343f0 con 0x7ff548163a70 2024-03-06T07:27:25.263 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 -- 172.21.15.135:0/3801975017 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 msgr2=0x7ff548165e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.263 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 --2- 172.21.15.135:0/3801975017 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 0x7ff548165e60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff540009e50 tx=0x7ff54002be40 comp rx=0 tx=0).stop 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 -- 172.21.15.135:0/3801975017 shutdown_connections 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 --2- 172.21.15.135:0/3801975017 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff548163a70 0x7ff548165e60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 --2- 172.21.15.135:0/3801975017 >> v2:172.21.15.135:3300/0 conn(0x7ff5480cf190 0x7ff5480cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 -- 172.21.15.135:0/3801975017 >> 172.21.15.135:0/3801975017 conn(0x7ff5480b38e0 msgr2=0x7ff5480b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 -- 172.21.15.135:0/3801975017 shutdown_connections 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.261+0000 7ff550400640 1 -- 172.21.15.135:0/3801975017 wait complete. 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.262+0000 7ff550400640 1 Processor -- start 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 -- start start 2024-03-06T07:27:25.267 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5480c7560 0x7ff54814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.267 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5480cf190 0x7ff54814ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.267 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 0x7ff548158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.267 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480d1700 con 0x7ff5480cf190 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480d11e0 con 0x7ff548163a70 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.263+0000 7ff550400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480cd330 con 0x7ff5480c7560 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 0x7ff548158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5480c7560 0x7ff54814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 0x7ff548158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56834/0 (socket says 172.21.15.135:56834) 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54e400640 1 -- 172.21.15.135:0/3072911586 learned_addr learned my addr 172.21.15.135:0/3072911586 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54d000640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5480cf190 0x7ff54814ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54da00640 1 -- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 msgr2=0x7ff548158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54da00640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 0x7ff548158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54da00640 1 -- 172.21.15.135:0/3072911586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5480cf190 msgr2=0x7ff54814ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54da00640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5480cf190 0x7ff54814ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54da00640 1 -- 172.21.15.135:0/3072911586 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480c9e50 con 0x7ff5480c7560 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54e400640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 0x7ff548158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:25.268 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.264+0000 7ff54d000640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5480cf190 0x7ff54814ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff54da00640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5480c7560 0x7ff54814d600 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00b3e0 tx=0x7ff53c00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c015030 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c004020 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c004410 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff548152130 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54814d340 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff53c018070 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.265+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff53c0035f0 con 0x7ff5480c7560 2024-03-06T07:27:25.269 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.266+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54806bc30 con 0x7ff5480c7560 2024-03-06T07:27:25.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.268+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff53c003bd0 con 0x7ff5480c7560 2024-03-06T07:27:25.307 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:25.304+0000 7f7b8c412040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:27:25.349 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.347+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ff93b254-4b0f-4580-b2e8-cac17574c5ee", "id": 7} v 0) v1 -- 0x7ff5480cf5f0 con 0x7ff5480c7560 2024-03-06T07:27:25.359 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.356+0000 7ff545600640 1 -- 172.21.15.135:0/3072911586 <== mon.2 v2:172.21.15.135:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff93b254-4b0f-4580-b2e8-cac17574c5ee", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5480cf5f0 con 0x7ff5480c7560 2024-03-06T07:27:25.359 INFO:teuthology.orchestra.run.smithi135.stdout:7 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5480c7560 msgr2=0x7ff54814d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5480c7560 0x7ff54814d600 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00b3e0 tx=0x7ff53c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 shutdown_connections 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff548163a70 0x7ff548158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5480cf190 0x7ff54814ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 --2- 172.21.15.135:0/3072911586 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5480c7560 0x7ff54814d600 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 >> 172.21.15.135:0/3072911586 conn(0x7ff5480b38e0 msgr2=0x7ff5480b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 shutdown_connections 2024-03-06T07:27:25.361 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-06T07:27:25.358+0000 7ff550400640 1 -- 172.21.15.135:0/3072911586 wait complete. 2024-03-06T07:27:25.376 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:27:25.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:27:25.382 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:27:25.382 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:27:25.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:27:25.388 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:27:25.388 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:27:25.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:27:25.393 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:27:25.393 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:27:25.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:27:25.397 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:27:25.398 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:27:25.398 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:27:25.402 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:27:25.402 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:27:25.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:27:25.406 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:27:25.406 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:27:25.406 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:27:25.410 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:27:25.410 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:27:25.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:27:25.415 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:27:25.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:25.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.640+0000 7f647de00640 1 Processor -- start 2024-03-06T07:27:25.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.641+0000 7f647de00640 1 -- start start 2024-03-06T07:27:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f647de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f647806cfe0 0x7f64780b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f647de00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64780717c0 con 0x7f647806cb50 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f647de00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6478071aa0 con 0x7f64780cf430 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f647de00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6478071d80 con 0x7f647806cfe0 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f646fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f647806cb50 0x7f64780b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45172/0 (socket says 172.21.15.97:45172) 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f646fe00640 1 -- 172.21.15.97:0/3627407991 learned_addr learned my addr 172.21.15.97:0/3627407991 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f6477400640 1 --2- 172.21.15.97:0/3627407991 >> v2:172.21.15.97:3300/0 conn(0x7f647806cfe0 0x7f64780b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f6476000640 1 -- 172.21.15.97:0/3627407991 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489397047 0 0) 0x7f64780717c0 con 0x7f647806cb50 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.642+0000 7f6476000640 1 -- 172.21.15.97:0/3627407991 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645c003520 con 0x7f647806cb50 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.643+0000 7f6477400640 1 -- 172.21.15.97:0/3627407991 >> v1:172.21.15.135:6789/0 conn(0x7f64780cf430 legacy=0x7f6478070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.643+0000 7f6477400640 1 -- 172.21.15.97:0/3627407991 >> v1:172.21.15.97:6789/0 conn(0x7f647806cb50 legacy=0x7f64780b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.643+0000 7f6477400640 1 -- 172.21.15.97:0/3627407991 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6478072060 con 0x7f647806cfe0 2024-03-06T07:27:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.644+0000 7f6477400640 1 --2- 172.21.15.97:0/3627407991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647806cfe0 0x7f64780b3420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6470009d90 tx=0x7f647002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71074f9d69dc5836 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.644+0000 7f6476000640 1 -- 172.21.15.97:0/3627407991 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470039070 con 0x7f647806cfe0 2024-03-06T07:27:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.644+0000 7f6476000640 1 -- 172.21.15.97:0/3627407991 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647002ccf0 con 0x7f647806cfe0 2024-03-06T07:27:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.644+0000 7f6476000640 1 -- 172.21.15.97:0/3627407991 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470034640 con 0x7f647806cfe0 2024-03-06T07:27:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.644+0000 7f647de00640 1 -- 172.21.15.97:0/3627407991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647806cfe0 msgr2=0x7f64780b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.644+0000 7f647de00640 1 --2- 172.21.15.97:0/3627407991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647806cfe0 0x7f64780b3420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6470009d90 tx=0x7f647002bf40 comp rx=0 tx=0).stop 2024-03-06T07:27:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.645+0000 7f647de00640 1 -- 172.21.15.97:0/3627407991 shutdown_connections 2024-03-06T07:27:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.645+0000 7f647de00640 1 --2- 172.21.15.97:0/3627407991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647806cfe0 0x7f64780b3420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.645+0000 7f647de00640 1 -- 172.21.15.97:0/3627407991 >> 172.21.15.97:0/3627407991 conn(0x7f64780b1730 msgr2=0x7f64780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.645+0000 7f647de00640 1 -- 172.21.15.97:0/3627407991 shutdown_connections 2024-03-06T07:27:25.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.645+0000 7f647de00640 1 -- 172.21.15.97:0/3627407991 wait complete. 2024-03-06T07:27:25.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.646+0000 7f647de00640 1 Processor -- start 2024-03-06T07:27:25.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.646+0000 7f647de00640 1 -- start start 2024-03-06T07:27:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.647+0000 7f647de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f647806cb50 0x7f64781ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.647+0000 7f647de00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f647806cfe0 0x7f6478147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.647+0000 7f647de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 0x7f64781ed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.647+0000 7f647de00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6478071430 con 0x7f64780cf430 2024-03-06T07:27:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.647+0000 7f647de00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647806d3f0 con 0x7f647806cfe0 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.647+0000 7f647de00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6478162b70 con 0x7f647806cb50 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 0x7f64781ed2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f646fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f647806cfe0 0x7f6478147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 0x7f64781ed2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37758/0 (socket says 172.21.15.97:37758) 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6476a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f647806cb50 0x7f64781ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 -- 172.21.15.97:0/2328486250 learned_addr learned my addr 172.21.15.97:0/2328486250 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 -- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f647806cb50 msgr2=0x7f64781ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f647806cb50 0x7f64781ebd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 -- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f647806cfe0 msgr2=0x7f6478147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f647806cfe0 0x7f6478147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 -- 172.21.15.97:0/2328486250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64781fc8a0 con 0x7f64780cf430 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f646fe00640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f647806cfe0 0x7f6478147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.648+0000 7f6477400640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 0x7f64781ed2f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6470037040 tx=0x7f6470004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470044020 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647003a040 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64781ec530 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470039050 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64781ecaf0 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6470003bd0 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.649+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f647003ece0 con 0x7f64780cf430 2024-03-06T07:27:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.650+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647804f330 con 0x7f64780cf430 2024-03-06T07:27:25.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.653+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f647004e060 con 0x7f64780cf430 2024-03-06T07:27:25.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.720+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64780b4a40 con 0x7f64780cf430 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.721+0000 7f646f400640 1 -- 172.21.15.97:0/2328486250 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f647003e610 con 0x7f64780cf430 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:25.350454+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 msgr2=0x7f64781ed2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 0x7f64781ed2f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6470037040 tx=0x7f6470004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 shutdown_connections 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64780cf430 0x7f64781ed2f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f647806cfe0 0x7f6478147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 --2- 172.21.15.97:0/2328486250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f647806cb50 0x7f64781ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 >> 172.21.15.97:0/2328486250 conn(0x7f64780b1730 msgr2=0x7f6478070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 shutdown_connections 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:25.722+0000 7f647de00640 1 -- 172.21.15.97:0/2328486250 wait complete. 2024-03-06T07:27:25.739 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:27:25.739 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:27:26.080 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-03-06T07:27:26.078+0000 7f5263f9e640 -1 Falling back to public interface 2024-03-06T07:27:26.084 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-03-06T07:27:26.082+0000 7fdbc6c1a640 -1 Falling back to public interface 2024-03-06T07:27:26.088 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-03-06T07:27:26.086+0000 7fb2c77ff640 -1 Falling back to public interface 2024-03-06T07:27:26.096 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-03-06T07:27:26.094+0000 7fa6d35bb640 -1 Falling back to public interface 2024-03-06T07:27:26.098 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-03-06T07:27:26.096+0000 7f14e8cd9640 -1 Falling back to public interface 2024-03-06T07:27:26.105 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-03-06T07:27:26.103+0000 7f89d0a17640 -1 Falling back to public interface 2024-03-06T07:27:26.108 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-03-06T07:27:26.106+0000 7fa3bb819640 -1 Falling back to public interface 2024-03-06T07:27:26.118 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-03-06T07:27:26.116+0000 7f3429824640 -1 Falling back to public interface 2024-03-06T07:27:26.356 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:26.354+0000 7f7213e3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:27:26.435 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:26.432+0000 7f7b8c412040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:27:26.545 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:26.673 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:26.671+0000 7f7213e3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:27:26.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.694+0000 7fb26f800640 1 Processor -- start 2024-03-06T07:27:26.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.694+0000 7fb26f800640 1 -- start start 2024-03-06T07:27:26.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb26f800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 0x7fb2680af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb26f800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb268072610 0x7fb26806aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb26f800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb268072aa0 con 0x7fb2680af2b0 2024-03-06T07:27:26.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb26f800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2680b0020 con 0x7fb268072610 2024-03-06T07:27:26.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb26f800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680b0300 con 0x7fb2680aeed0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb267e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb2680af2b0 0x7fb26815ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45188/0 (socket says 172.21.15.97:45188) 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.695+0000 7fb267e00640 1 -- 172.21.15.97:0/88267872 learned_addr learned my addr 172.21.15.97:0/88267872 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb26ce00640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 0x7fb2680af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb26d800640 1 --2- 172.21.15.97:0/88267872 >> v2:172.21.15.135:3300/0 conn(0x7fb268072610 0x7fb26806aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042273300 0 0) 0x7fb268072aa0 con 0x7fb2680af2b0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250003520 con 0x7fb2680af2b0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288197194 0 0) 0x7fb250003520 con 0x7fb2680af2b0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 msgr2=0x7fb2680af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 0x7fb2680af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 >> v2:172.21.15.135:3300/0 conn(0x7fb268072610 msgr2=0x7fb26806aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 --2- 172.21.15.97:0/88267872 >> v2:172.21.15.135:3300/0 conn(0x7fb268072610 0x7fb26806aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26815e630 con 0x7fb2680af2b0 2024-03-06T07:27:26.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb254001670 con 0x7fb2680af2b0 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb26d800640 1 --2- 172.21.15.97:0/88267872 >> v2:172.21.15.135:3300/0 conn(0x7fb268072610 0x7fb26806aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 >> v1:172.21.15.97:6789/0 conn(0x7fb2680af2b0 legacy=0x7fb26815ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 0x7fb250004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.696+0000 7fb267400640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680b0020 con 0x7fb2500039b0 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680b0300 con 0x7fb250004d40 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25000ba90 con 0x7fb2680aeed0 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26ce00640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb267e00640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 0x7fb250004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26d800640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26ce00640 1 -- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 msgr2=0x7fb250004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26ce00640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2680aeed0 0x7fb250004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26ce00640 1 -- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb250004d40 msgr2=0x7fb2500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26ce00640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26ce00640 1 -- 172.21.15.97:0/88267872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250003520 con 0x7fb2500039b0 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.697+0000 7fb26d800640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.698+0000 7fb26ce00640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009ac0 tx=0x7fb250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fdf578dc095a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.698+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c039070 con 0x7fb2500039b0 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.698+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c02c6b0 con 0x7fb2500039b0 2024-03-06T07:27:26.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.698+0000 7fb267400640 1 -- 172.21.15.97:0/88267872 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c009690 con 0x7fb2500039b0 2024-03-06T07:27:26.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26f800640 1 -- 172.21.15.97:0/88267872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500039b0 msgr2=0x7fb250003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26f800640 1 --2- 172.21.15.97:0/88267872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009ac0 tx=0x7fb250005830 comp rx=0 tx=0).stop 2024-03-06T07:27:26.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26d800640 1 -- 172.21.15.97:0/88267872 reap_dead start 2024-03-06T07:27:26.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26f800640 1 -- 172.21.15.97:0/88267872 shutdown_connections 2024-03-06T07:27:26.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26f800640 1 -- 172.21.15.97:0/88267872 >> 172.21.15.97:0/88267872 conn(0x7fb26806c000 msgr2=0x7fb26806c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26f800640 1 -- 172.21.15.97:0/88267872 shutdown_connections 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.699+0000 7fb26f800640 1 -- 172.21.15.97:0/88267872 wait complete. 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.701+0000 7fb26f800640 1 Processor -- start 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.701+0000 7fb26f800640 1 -- start start 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26f800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2500039b0 0x7fb2680cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26f800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 0x7fb2681f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26f800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2681f4aa0 0x7fb2681f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26f800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680afb30 con 0x7fb2681f6f50 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26f800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680af9b0 con 0x7fb2500039b0 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26f800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680afcb0 con 0x7fb2681f4aa0 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 0x7fb2681f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 0x7fb2681f0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37778/0 (socket says 172.21.15.97:37778) 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 -- 172.21.15.97:0/608930678 learned_addr learned my addr 172.21.15.97:0/608930678 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:26.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26d800640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2681f4aa0 0x7fb2681f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26ce00640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2500039b0 0x7fb2680cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 -- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2681f4aa0 msgr2=0x7fb2681f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2681f4aa0 0x7fb2681f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 -- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2500039b0 msgr2=0x7fb2680cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2500039b0 0x7fb2680cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb267e00640 1 -- 172.21.15.97:0/608930678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2681f73d0 con 0x7fb2681f6f50 2024-03-06T07:27:26.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.702+0000 7fb26ce00640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2500039b0 0x7fb2680cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb267e00640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 0x7fb2681f0dd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb25400e6c0 tx=0x7fb25400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25400c6c0 con 0x7fb2681f6f50 2024-03-06T07:27:26.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb254018070 con 0x7fb2681f6f50 2024-03-06T07:27:26.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25400caa0 con 0x7fb2681f6f50 2024-03-06T07:27:26.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2681f7680 con 0x7fb2681f6f50 2024-03-06T07:27:26.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2681f7c40 con 0x7fb2681f6f50 2024-03-06T07:27:26.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.703+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb25400cc40 con 0x7fb2681f6f50 2024-03-06T07:27:26.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.704+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb254013c80 con 0x7fb2681f6f50 2024-03-06T07:27:26.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.704+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26804f330 con 0x7fb2681f6f50 2024-03-06T07:27:26.712 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.707+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb2540135b0 con 0x7fb2681f6f50 2024-03-06T07:27:26.738 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:26.736+0000 7f7213e3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:27:26.747 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:26.745+0000 7f7b8c412040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:27:26.776 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.774+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb26806c5f0 con 0x7fb2681f6f50 2024-03-06T07:27:26.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.775+0000 7fb264c00640 1 -- 172.21.15.97:0/608930678 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb254013790 con 0x7fb2681f6f50 2024-03-06T07:27:26.778 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-06T07:27:26.778 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:25.350454+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.776+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 msgr2=0x7fb2681f0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.776+0000 7fb26f800640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 0x7fb2681f0dd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb25400e6c0 tx=0x7fb25400eb80 comp rx=0 tx=0).stop 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 shutdown_connections 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2681f4aa0 0x7fb2681f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2681f6f50 0x7fb2681f0dd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 --2- 172.21.15.97:0/608930678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2500039b0 0x7fb2680cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 >> 172.21.15.97:0/608930678 conn(0x7fb26806c000 msgr2=0x7fb26815ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 shutdown_connections 2024-03-06T07:27:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:26.777+0000 7fb26f800640 1 -- 172.21.15.97:0/608930678 wait complete. 2024-03-06T07:27:26.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:27:26.807 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:26.805+0000 7f7b8c412040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:27:26.872 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:26.870+0000 7f7213e3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:27:26.948 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-03-06T07:27:26.946+0000 7f7b8c412040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:27:27.920 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:27.917+0000 7f71b4800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:27:27.920 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-06T07:27:27.918+0000 7f71b4800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:27:29.594 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-03-06T07:27:29.592+0000 7f5263f9e640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:27:29.652 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-03-06T07:27:29.650+0000 7fa3bb819640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:27:29.656 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-03-06T07:27:29.654+0000 7f89d0a17640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:27:29.656 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-03-06T07:27:29.654+0000 7f3429824640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:27:29.853 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-03-06T07:27:29.851+0000 7fb2c77ff640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:27:29.895 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-03-06T07:27:29.893+0000 7f14e8cd9640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:27:29.897 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-03-06T07:27:29.894+0000 7fa6d35bb640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:27:29.923 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-03-06T07:27:29.920+0000 7fdbc6c1a640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:27:32.859 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-03-06T07:27:32.857+0000 7fa6cda00640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:27:32.869 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-03-06T07:27:32.868+0000 7fa6c4400640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:32.969 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-03-06T07:27:32.968+0000 7f5260200640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:27:32.975 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-03-06T07:27:32.974+0000 7f3425400640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:27:32.978 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-03-06T07:27:32.976+0000 7f5254e00640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:32.985 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-03-06T07:27:32.984+0000 7f341a000640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:33.218 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-03-06T07:27:33.216+0000 7f89caa00640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:27:33.223 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-03-06T07:27:33.221+0000 7f89c1400640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:33.226 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-03-06T07:27:33.224+0000 7fb2c1c00640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:27:33.234 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-03-06T07:27:33.232+0000 7fb2b8600640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:33.246 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-03-06T07:27:33.245+0000 7fa3b5800640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:27:33.252 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-03-06T07:27:33.250+0000 7fa3ac200640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:33.306 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-03-06T07:27:33.304+0000 7f14e3200640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:27:33.317 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-03-06T07:27:33.315+0000 7f14d9c00640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:33.345 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-03-06T07:27:33.344+0000 7fdbc0a00640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:27:33.357 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-03-06T07:27:33.356+0000 7fdbb7400640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:33.596 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:33.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.770+0000 7f4a5cc00640 1 Processor -- start 2024-03-06T07:27:33.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.770+0000 7f4a5cc00640 1 -- start start 2024-03-06T07:27:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.771+0000 7f4a5cc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4a5815fbf0 0x7f4a58160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.771+0000 7f4a5cc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a580cedd0 con 0x7f4a58070ec0 2024-03-06T07:27:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a5cc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a580cf0b0 con 0x7f4a5813eeb0 2024-03-06T07:27:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a5cc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a580cf390 con 0x7f4a5815fbf0 2024-03-06T07:27:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a56800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4a58070ec0 0x7f4a580712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45204/0 (socket says 172.21.15.97:45204) 2024-03-06T07:27:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a56800640 1 -- 172.21.15.97:0/512525754 learned_addr learned my addr 172.21.15.97:0/512525754 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:33.776 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a55400640 1 --2- 172.21.15.97:0/512525754 >> v2:172.21.15.135:3300/0 conn(0x7f4a5815fbf0 0x7f4a58160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.776 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947652542 0 0) 0x7f4a580cedd0 con 0x7f4a58070ec0 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a3c003520 con 0x7f4a58070ec0 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.772+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922145494 0 0) 0x7f4a3c003520 con 0x7f4a58070ec0 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 >> v2:172.21.15.135:3300/0 conn(0x7f4a5815fbf0 msgr2=0x7f4a58160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 --2- 172.21.15.97:0/512525754 >> v2:172.21.15.135:3300/0 conn(0x7f4a5815fbf0 0x7f4a58160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 >> v1:172.21.15.135:6789/0 conn(0x7f4a5813eeb0 legacy=0x7f4a580ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580cf670 con 0x7f4a58070ec0 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4a500019f0 con 0x7f4a58070ec0 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 >> v1:172.21.15.97:6789/0 conn(0x7f4a58070ec0 legacy=0x7f4a580712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a3c003a00 0x7f4a3c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a3c005210 0x7f4a3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580cf390 con 0x7f4a3c004800 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c00c160 con 0x7f4a3c003a00 2024-03-06T07:27:33.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c00c360 con 0x7f4a3c005210 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55400640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55400640 1 -- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a3c005210 msgr2=0x7f4a3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55400640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a3c005210 0x7f4a3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55400640 1 -- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a3c003a00 msgr2=0x7f4a3c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55e00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a3c003a00 0x7f4a3c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55400640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a3c003a00 0x7f4a3c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a55400640 1 -- 172.21.15.97:0/512525754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c003520 con 0x7f4a3c004800 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.773+0000 7f4a56800640 1 -- 172.21.15.97:0/512525754 reap_dead start 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.774+0000 7f4a55e00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a3c003a00 0x7f4a3c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a55400640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a3c004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009a90 tx=0x7f4a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cebfa949d70b31f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44039070 con 0x7f4a3c004800 2024-03-06T07:27:33.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4402c6b0 con 0x7f4a3c004800 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a54a00640 1 -- 172.21.15.97:0/512525754 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44009690 con 0x7f4a3c004800 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a5cc00640 1 -- 172.21.15.97:0/512525754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 msgr2=0x7f4a3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a3c004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009a90 tx=0x7f4a3c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a5cc00640 1 -- 172.21.15.97:0/512525754 shutdown_connections 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/512525754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a3c004c80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.775+0000 7f4a5cc00640 1 -- 172.21.15.97:0/512525754 >> 172.21.15.97:0/512525754 conn(0x7f4a580b16b0 msgr2=0x7f4a580b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.776+0000 7f4a5cc00640 1 -- 172.21.15.97:0/512525754 shutdown_connections 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.776+0000 7f4a5cc00640 1 -- 172.21.15.97:0/512525754 wait complete. 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.777+0000 7f4a5cc00640 1 Processor -- start 2024-03-06T07:27:33.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.777+0000 7f4a5cc00640 1 -- start start 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a5cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a580b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a5cc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a5813c530 0x7f4a5814f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a5cc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a580afc60 0x7f4a580b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a5cc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58160a30 con 0x7f4a3c004800 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a5cc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a581608b0 con 0x7f4a580afc60 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a5cc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58160070 con 0x7f4a5813c530 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.778+0000 7f4a55e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a580b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a580b3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37896/0 (socket says 172.21.15.97:37896) 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 -- 172.21.15.97:0/4141141150 learned_addr learned my addr 172.21.15.97:0/4141141150 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 -- 172.21.15.97:0/4141141150 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a5813c530 msgr2=0x7f4a5814f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a5813c530 0x7f4a5814f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 -- 172.21.15.97:0/4141141150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a580afc60 msgr2=0x7f4a580b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a580afc60 0x7f4a580b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 -- 172.21.15.97:0/4141141150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5813c990 con 0x7f4a3c004800 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a55e00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a580b3280 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800d430 tx=0x7f4a4800d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48018070 con 0x7f4a3c004800 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48003610 con 0x7f4a3c004800 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48003a20 con 0x7f4a3c004800 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.779+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a580b3950 con 0x7f4a3c004800 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.780+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a580b3f10 con 0x7f4a3c004800 2024-03-06T07:27:33.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.781+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5804f330 con 0x7f4a3c004800 2024-03-06T07:27:33.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.781+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f4a4801b070 con 0x7f4a3c004800 2024-03-06T07:27:33.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.783+0000 7f4a4e000640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a3806b470 0x7f4a3806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.783+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f4a48088ab0 con 0x7f4a3c004800 2024-03-06T07:27:33.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.783+0000 7f4a55400640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a3806b470 0x7f4a3806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.783+0000 7f4a55400640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a3806b470 0x7f4a3806d930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4a44036000 tx=0x7f4a4402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.788+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a480580b0 con 0x7f4a3c004800 2024-03-06T07:27:33.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.989+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a58072b20 con 0x7f4a3c004800 2024-03-06T07:27:33.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.990+0000 7f4a4e000640 1 -- 172.21.15.97:0/4141141150 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10897 (secure 0 0 0) 0x7f4a4805c150 con 0x7f4a3c004800 2024-03-06T07:27:33.992 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-06T07:27:33.992 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":14,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:33.865196+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a3806b470 msgr2=0x7f4a3806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a3806b470 0x7f4a3806d930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4a44036000 tx=0x7f4a4402e000 comp rx=0 tx=0).stop 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 msgr2=0x7f4a580b3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a580b3280 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800d430 tx=0x7f4a4800d8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 shutdown_connections 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a3806b470 0x7f4a3806d930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a580afc60 0x7f4a580b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a5813c530 0x7f4a5814f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 --2- 172.21.15.97:0/4141141150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a3c004800 0x7f4a580b3280 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.992+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 >> 172.21.15.97:0/4141141150 conn(0x7f4a580b16b0 msgr2=0x7f4a5806e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.993+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 shutdown_connections 2024-03-06T07:27:33.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-06T07:27:33.993+0000 7f4a5cc00640 1 -- 172.21.15.97:0/4141141150 wait complete. 2024-03-06T07:27:34.008 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:27:34.008 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:27:34.009 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:34.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:34.016 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:34.016 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:34.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:34.023 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:34.023 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:34.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:34.031 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:34.031 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:34.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:34.038 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:34.038 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:27:34.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:34.160 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:27:34.167 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:27:34.195 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:27:34.195 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:27:34.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.217+0000 7f7f0ce00640 1 Processor -- start 2024-03-06T07:27:34.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.217+0000 7f7f0ce00640 1 -- start start 2024-03-06T07:27:34.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.218+0000 7f7f0ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d27a0 0x7f7f080b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.218+0000 7f7f0ce00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f08161960 con 0x7f7f08161580 2024-03-06T07:27:34.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.218+0000 7f7f0ce00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080cf6f0 con 0x7f7f080d2360 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.218+0000 7f7f0ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080cf9d0 con 0x7f7f080d27a0 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.218+0000 7f7f07400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7f08161580 0x7f7f080d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42566/0 (socket says 172.21.15.97:42566) 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.218+0000 7f7f07400640 1 -- 172.21.15.97:0/4146268409 learned_addr learned my addr 172.21.15.97:0/4146268409 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.219+0000 7f7f07e00640 1 --2- 172.21.15.97:0/4146268409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d27a0 0x7f7f080b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.219+0000 7f7f06a00640 1 -- 172.21.15.97:0/4146268409 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390431765 0 0) 0x7f7f080cf6f0 con 0x7f7f080d2360 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.219+0000 7f7f06a00640 1 -- 172.21.15.97:0/4146268409 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec003520 con 0x7f7f080d2360 2024-03-06T07:27:34.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.219+0000 7f7f07e00640 1 -- 172.21.15.97:0/4146268409 >> v1:172.21.15.135:6789/0 conn(0x7f7f080d2360 legacy=0x7f7f080b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.219+0000 7f7f07e00640 1 -- 172.21.15.97:0/4146268409 >> v1:172.21.15.97:6789/0 conn(0x7f7f08161580 legacy=0x7f7f080d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.219+0000 7f7f07e00640 1 -- 172.21.15.97:0/4146268409 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080cfcb0 con 0x7f7f080d27a0 2024-03-06T07:27:34.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.220+0000 7f7f07e00640 1 --2- 172.21.15.97:0/4146268409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d27a0 0x7f7f080b45d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f000097d0 tx=0x7f7f0002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9aae9dd6151988 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.227+0000 7f7f06a00640 1 -- 172.21.15.97:0/4146268409 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00035020 con 0x7f7f080d27a0 2024-03-06T07:27:34.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.227+0000 7f7f06a00640 1 -- 172.21.15.97:0/4146268409 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f00031890 con 0x7f7f080d27a0 2024-03-06T07:27:34.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.227+0000 7f7f06a00640 1 -- 172.21.15.97:0/4146268409 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00031c60 con 0x7f7f080d27a0 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.227+0000 7f7f0ce00640 1 -- 172.21.15.97:0/4146268409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d27a0 msgr2=0x7f7f080b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.227+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/4146268409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d27a0 0x7f7f080b45d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f000097d0 tx=0x7f7f0002b930 comp rx=0 tx=0).stop 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.228+0000 7f7f0ce00640 1 -- 172.21.15.97:0/4146268409 shutdown_connections 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.228+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/4146268409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d27a0 0x7f7f080b45d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.228+0000 7f7f0ce00640 1 -- 172.21.15.97:0/4146268409 >> 172.21.15.97:0/4146268409 conn(0x7f7f080d3d80 msgr2=0x7f7f080d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.228+0000 7f7f0ce00640 1 -- 172.21.15.97:0/4146268409 shutdown_connections 2024-03-06T07:27:34.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.228+0000 7f7f0ce00640 1 -- 172.21.15.97:0/4146268409 wait complete. 2024-03-06T07:27:34.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.229+0000 7f7f0ce00640 1 Processor -- start 2024-03-06T07:27:34.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.229+0000 7f7f0ce00640 1 -- start start 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.230+0000 7f7f0ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d2360 0x7f7f081533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.230+0000 7f7f0ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 0x7f7f080b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.230+0000 7f7f0ce00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08161580 0x7f7f080b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.230+0000 7f7f0ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080d2c30 con 0x7f7f080d27a0 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.230+0000 7f7f0ce00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080b39a0 con 0x7f7f08161580 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.230+0000 7f7f0ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080cf3a0 con 0x7f7f080d2360 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 0x7f7f080b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 0x7f7f080b8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60964/0 (socket says 172.21.15.97:60964) 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 -- 172.21.15.97:0/144120896 learned_addr learned my addr 172.21.15.97:0/144120896 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7f07400640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d2360 0x7f7f081533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7f07e00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08161580 0x7f7f080b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 -- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d2360 msgr2=0x7f7f081533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d2360 0x7f7f081533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 -- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08161580 msgr2=0x7f7f080b8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08161580 0x7f7f080b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 -- 172.21.15.97:0/144120896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080cbca0 con 0x7f7f080d27a0 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7efea00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 0x7f7f080b8860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000c430 tx=0x7f7ef000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0017070 con 0x7f7f080d27a0 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.231+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef00052f0 con 0x7f7f080d27a0 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.232+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f080cbf80 con 0x7f7f080d27a0 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.232+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0003830 con 0x7f7f080d27a0 2024-03-06T07:27:34.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.232+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f080cc540 con 0x7f7f080d27a0 2024-03-06T07:27:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.233+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0804f330 con 0x7f7f080d27a0 2024-03-06T07:27:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.233+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f7ef001a070 con 0x7f7f080d27a0 2024-03-06T07:27:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.234+0000 7f7effe00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ee406b440 0x7f7ee406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.234+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f7ef0088f30 con 0x7f7f080d27a0 2024-03-06T07:27:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.234+0000 7f7f07400640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ee406b440 0x7f7ee406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.235+0000 7f7f07400640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ee406b440 0x7f7ee406d900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005c00 tx=0x7f7ef8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.238+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ef00583f0 con 0x7f7f080d27a0 2024-03-06T07:27:34.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.420+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f080d3b60 con 0x7f7f080d27a0 2024-03-06T07:27:34.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.421+0000 7f7effe00640 1 -- 172.21.15.97:0/144120896 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10897 (secure 0 0 0) 0x7f7ef0057cf0 con 0x7f7f080d27a0 2024-03-06T07:27:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:34.423 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":14,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:33.865196+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ee406b440 msgr2=0x7f7ee406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ee406b440 0x7f7ee406d900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005c00 tx=0x7f7ef8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 msgr2=0x7f7f080b8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 0x7f7f080b8860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000c430 tx=0x7f7ef000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 shutdown_connections 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ee406b440 0x7f7ee406d900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08161580 0x7f7f080b8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f080d27a0 0x7f7f080b8860 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 --2- 172.21.15.97:0/144120896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f080d2360 0x7f7f081533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.423+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 >> 172.21.15.97:0/144120896 conn(0x7f7f080d3d80 msgr2=0x7f7f0806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.424+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 shutdown_connections 2024-03-06T07:27:34.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.424+0000 7f7f0ce00640 1 -- 172.21.15.97:0/144120896 wait complete. 2024-03-06T07:27:34.442 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-06T07:27:34.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:34.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.626+0000 7f285d400640 1 Processor -- start 2024-03-06T07:27:34.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.627+0000 7f285d400640 1 -- start start 2024-03-06T07:27:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.628+0000 7f285d400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f28581561f0 0x7f28580ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.628+0000 7f285d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 0x7f285815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.628+0000 7f285d400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285815c240 con 0x7f28580aebf0 2024-03-06T07:27:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.628+0000 7f285d400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285815ce30 con 0x7f28581561f0 2024-03-06T07:27:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.628+0000 7f285d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285815c500 con 0x7f28580cc9d0 2024-03-06T07:27:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.629+0000 7f285ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 0x7f285815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.629+0000 7f285ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 0x7f285815f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:35330/0 (socket says 172.21.15.97:35330) 2024-03-06T07:27:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.629+0000 7f285ca00640 1 -- 172.21.15.97:0/864521657 learned_addr learned my addr 172.21.15.97:0/864521657 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.629+0000 7f2857e00640 1 --2- 172.21.15.97:0/864521657 >> v2:172.21.15.135:3300/0 conn(0x7f28581561f0 0x7f28580ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.630+0000 7f285ca00640 1 -- 172.21.15.97:0/864521657 >> v2:172.21.15.135:3300/0 conn(0x7f28581561f0 msgr2=0x7f28580ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.630+0000 7f285ca00640 1 --2- 172.21.15.97:0/864521657 >> v2:172.21.15.135:3300/0 conn(0x7f28581561f0 0x7f28580ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.630+0000 7f285ca00640 1 -- 172.21.15.97:0/864521657 >> v1:172.21.15.135:6789/0 conn(0x7f28580aebf0 legacy=0x7f28580cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.630+0000 7f285ca00640 1 -- 172.21.15.97:0/864521657 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285815c7e0 con 0x7f28580cc9d0 2024-03-06T07:27:34.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.631+0000 7f285ca00640 1 --2- 172.21.15.97:0/864521657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 0x7f285815f830 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2850009e50 tx=0x7f285002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5e8703686b87e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f2857400640 1 -- 172.21.15.97:0/864521657 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285002cb00 con 0x7f28580cc9d0 2024-03-06T07:27:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f2857400640 1 -- 172.21.15.97:0/864521657 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285002cca0 con 0x7f28580cc9d0 2024-03-06T07:27:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f2857400640 1 -- 172.21.15.97:0/864521657 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850034610 con 0x7f28580cc9d0 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f285d400640 1 -- 172.21.15.97:0/864521657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 msgr2=0x7f285815f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f285d400640 1 --2- 172.21.15.97:0/864521657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 0x7f285815f830 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2850009e50 tx=0x7f285002bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f285d400640 1 -- 172.21.15.97:0/864521657 shutdown_connections 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f285d400640 1 --2- 172.21.15.97:0/864521657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28580cc9d0 0x7f285815f830 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f285d400640 1 --2- 172.21.15.97:0/864521657 >> v2:172.21.15.135:3300/0 conn(0x7f28581561f0 0x7f28580ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.632+0000 7f285d400640 1 -- 172.21.15.97:0/864521657 >> 172.21.15.97:0/864521657 conn(0x7f285806afc0 msgr2=0x7f285806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.633+0000 7f285d400640 1 -- 172.21.15.97:0/864521657 shutdown_connections 2024-03-06T07:27:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.633+0000 7f285d400640 1 -- 172.21.15.97:0/864521657 wait complete. 2024-03-06T07:27:34.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.634+0000 7f285d400640 1 Processor -- start 2024-03-06T07:27:34.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.634+0000 7f285d400640 1 -- start start 2024-03-06T07:27:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f285d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 0x7f28580af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f285d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28580cc9d0 0x7f28580af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f285d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28581561f0 0x7f28580afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f285d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2858156750 con 0x7f28580aebf0 2024-03-06T07:27:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f285d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28581565d0 con 0x7f28580cc9d0 2024-03-06T07:27:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f285d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285815fd70 con 0x7f28581561f0 2024-03-06T07:27:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f2857e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 0x7f28580af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f2857e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 0x7f28580af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60966/0 (socket says 172.21.15.97:60966) 2024-03-06T07:27:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.635+0000 7f2857e00640 1 -- 172.21.15.97:0/1392508014 learned_addr learned my addr 172.21.15.97:0/1392508014 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f285ca00640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28581561f0 0x7f28580afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2857e00640 1 -- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28581561f0 msgr2=0x7f28580afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2857e00640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28581561f0 0x7f28580afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2857e00640 1 -- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28580cc9d0 msgr2=0x7f28580af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2857e00640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28580cc9d0 0x7f28580af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2857e00640 1 -- 172.21.15.97:0/1392508014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858157540 con 0x7f28580aebf0 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2857e00640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 0x7f28580af410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f284800e870 tx=0x7f284800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848017070 con 0x7f28580aebf0 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284800c940 con 0x7f28580aebf0 2024-03-06T07:27:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284800cd30 con 0x7f28580aebf0 2024-03-06T07:27:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2858157820 con 0x7f28580aebf0 2024-03-06T07:27:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.636+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2858157de0 con 0x7f28580aebf0 2024-03-06T07:27:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.637+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285804f330 con 0x7f28580aebf0 2024-03-06T07:27:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.638+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f284801a070 con 0x7f28580aebf0 2024-03-06T07:27:34.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.639+0000 7f2854c00640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f283406b440 0x7f283406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.639+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f2848089f20 con 0x7f28580aebf0 2024-03-06T07:27:34.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.639+0000 7f284f400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f283406b440 0x7f283406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.640+0000 7f284f400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f283406b440 0x7f283406d900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2840000f60 tx=0x7f28400079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.644+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28480592b0 con 0x7f28580aebf0 2024-03-06T07:27:34.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.861+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28580b2b10 con 0x7f28580aebf0 2024-03-06T07:27:34.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.862+0000 7f2854c00640 1 -- 172.21.15.97:0/1392508014 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f2848058bb0 con 0x7f28580aebf0 2024-03-06T07:27:34.864 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:34.864 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:27:34.864 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 2 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f283406b440 msgr2=0x7f283406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f283406b440 0x7f283406d900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2840000f60 tx=0x7f28400079d0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 msgr2=0x7f28580af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 0x7f28580af410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f284800e870 tx=0x7f284800ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 shutdown_connections 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f283406b440 0x7f283406d900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28581561f0 0x7f28580afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.864+0000 7f285d400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28580cc9d0 0x7f28580af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.865+0000 7f285d400640 1 --2- 172.21.15.97:0/1392508014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28580aebf0 0x7f28580af410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.865+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 >> 172.21.15.97:0/1392508014 conn(0x7f285806afc0 msgr2=0x7f285806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.865+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 shutdown_connections 2024-03-06T07:27:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:34.865+0000 7f285d400640 1 -- 172.21.15.97:0/1392508014 wait complete. 2024-03-06T07:27:36.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:36.515 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:36.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.095+0000 7fb7ddc00640 1 Processor -- start 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.095+0000 7fb7ddc00640 1 -- start start 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7ddc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb7d80cc9d0 0x7fb7d806d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7ddc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d815c240 con 0x7fb7d81561f0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7ddc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d815c500 con 0x7fb7d80aedc0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7ddc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d815ce30 con 0x7fb7d80cc9d0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7dd200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb7d80cc9d0 0x7fb7d806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7d7e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb7d81561f0 0x7fb7d80ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42572/0 (socket says 172.21.15.97:42572) 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7d7e00640 1 -- 172.21.15.97:0/1007229917 learned_addr learned my addr 172.21.15.97:0/1007229917 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.097+0000 7fb7d6a00640 1 -- 172.21.15.97:0/1007229917 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982656071 0 0) 0x7fb7d815c240 con 0x7fb7d81561f0 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.098+0000 7fb7d6a00640 1 -- 172.21.15.97:0/1007229917 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc003520 con 0x7fb7d81561f0 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.098+0000 7fb7dd200640 1 -- 172.21.15.97:0/1007229917 >> v1:172.21.15.135:6789/0 conn(0x7fb7d80aedc0 legacy=0x7fb7d80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.098+0000 7fb7dd200640 1 -- 172.21.15.97:0/1007229917 >> v1:172.21.15.97:6789/0 conn(0x7fb7d81561f0 legacy=0x7fb7d80ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.098+0000 7fb7dd200640 1 -- 172.21.15.97:0/1007229917 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d815c7e0 con 0x7fb7d80cc9d0 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.099+0000 7fb7dd200640 1 --2- 172.21.15.97:0/1007229917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d806d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7d815cfb0 tx=0x7fb7d002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201f22e837dd734f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.099+0000 7fb7d6a00640 1 -- 172.21.15.97:0/1007229917 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0039070 con 0x7fb7d80cc9d0 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.099+0000 7fb7d6a00640 1 -- 172.21.15.97:0/1007229917 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d002cce0 con 0x7fb7d80cc9d0 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.099+0000 7fb7d6a00640 1 -- 172.21.15.97:0/1007229917 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0034660 con 0x7fb7d80cc9d0 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 -- 172.21.15.97:0/1007229917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d80cc9d0 msgr2=0x7fb7d806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/1007229917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d806d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7d815cfb0 tx=0x7fb7d002bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 -- 172.21.15.97:0/1007229917 shutdown_connections 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/1007229917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d806d350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 -- 172.21.15.97:0/1007229917 >> 172.21.15.97:0/1007229917 conn(0x7fb7d806afc0 msgr2=0x7fb7d806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 -- 172.21.15.97:0/1007229917 shutdown_connections 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.100+0000 7fb7ddc00640 1 -- 172.21.15.97:0/1007229917 wait complete. 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.101+0000 7fb7ddc00640 1 Processor -- start 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.102+0000 7fb7ddc00640 1 -- start start 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.102+0000 7fb7ddc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7d80aedc0 0x7fb7d815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.102+0000 7fb7ddc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d8156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7ddc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 0x7fb7d814e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7ddc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8156750 con 0x7fb7d81561f0 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7ddc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81565d0 con 0x7fb7d80cc9d0 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7ddc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d806d890 con 0x7fb7d80aedc0 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 0x7fb7d814e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 0x7fb7d814e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60980/0 (socket says 172.21.15.97:60980) 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 -- 172.21.15.97:0/3061516349 learned_addr learned my addr 172.21.15.97:0/3061516349 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7d7400640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d8156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7d7e00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7d80aedc0 0x7fb7d815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 -- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7d80aedc0 msgr2=0x7fb7d815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7d80aedc0 0x7fb7d815aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 -- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7d80cc9d0 msgr2=0x7fb7d8156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d8156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.103+0000 7fb7dd200640 1 -- 172.21.15.97:0/3061516349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8158390 con 0x7fb7d81561f0 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.104+0000 7fb7dd200640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 0x7fb7d814e710 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009e80 tx=0x7fb7d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.104+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0039050 con 0x7fb7d81561f0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.104+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d0045070 con 0x7fb7d81561f0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.104+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0009450 con 0x7fb7d81561f0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.104+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d814ec50 con 0x7fb7d81561f0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.104+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d815b540 con 0x7fb7d81561f0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.105+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d804f330 con 0x7fb7d81561f0 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.106+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fb7d003e850 con 0x7fb7d81561f0 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.107+0000 7fb7cfe00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb7b806b440 0x7fb7b806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.107+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb7d00acda0 con 0x7fb7d81561f0 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.107+0000 7fb7d7e00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb7b806b440 0x7fb7b806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.108+0000 7fb7d7e00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb7b806b440 0x7fb7b806d900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80011e0 tx=0x7fb7c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.112+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7d007c160 con 0x7fb7d81561f0 2024-03-06T07:27:37.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.381+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d80cee10 con 0x7fb7d81561f0 2024-03-06T07:27:37.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.382+0000 7fb7cfe00640 1 -- 172.21.15.97:0/3061516349 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7fb7d007ba60 con 0x7fb7d81561f0 2024-03-06T07:27:37.384 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:37.384 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:27:37.384 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 3 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb7b806b440 msgr2=0x7fb7b806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb7b806b440 0x7fb7b806d900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80011e0 tx=0x7fb7c8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 msgr2=0x7fb7d814e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 0x7fb7d814e710 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009e80 tx=0x7fb7d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 shutdown_connections 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb7b806b440 0x7fb7b806d900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d81561f0 0x7fb7d814e710 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7d80cc9d0 0x7fb7d8156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 --2- 172.21.15.97:0/3061516349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7d80aedc0 0x7fb7d815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 >> 172.21.15.97:0/3061516349 conn(0x7fb7d806afc0 msgr2=0x7fb7d814f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 shutdown_connections 2024-03-06T07:27:37.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.385+0000 7fb7ddc00640 1 -- 172.21.15.97:0/3061516349 wait complete. 2024-03-06T07:27:37.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:37.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.591+0000 7fa63f800640 1 Processor -- start 2024-03-06T07:27:37.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.591+0000 7fa63f800640 1 -- start start 2024-03-06T07:27:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63f800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 0x7fa6380ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63f800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63815b090 con 0x7fa63815acb0 2024-03-06T07:27:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63f800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6380b2a30 con 0x7fa6380b4680 2024-03-06T07:27:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63f800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380b4360 con 0x7fa63806ec00 2024-03-06T07:27:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63ce00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa63815acb0 0x7fa6380b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42586/0 (socket says 172.21.15.97:42586) 2024-03-06T07:27:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63ce00640 1 -- 172.21.15.97:0/1090365263 learned_addr learned my addr 172.21.15.97:0/1090365263 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa63d800640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 0x7fa6380ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190531548 0 0) 0x7fa63815b090 con 0x7fa63815acb0 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.593+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa620003520 con 0x7fa63815acb0 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236502688 0 0) 0x7fa620003520 con 0x7fa63815acb0 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 msgr2=0x7fa6380ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 0x7fa6380ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 >> v1:172.21.15.135:6789/0 conn(0x7fa6380b4680 legacy=0x7fa63806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380b2cf0 con 0x7fa63815acb0 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa62c001670 con 0x7fa63815acb0 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 >> v1:172.21.15.97:6789/0 conn(0x7fa63815acb0 legacy=0x7fa6380b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 0x7fa6200039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa620003f60 0x7fa620004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa620004d40 0x7fa6200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380b4360 con 0x7fa620003f60 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000baa0 con 0x7fa620004d40 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000bca0 con 0x7fa63806ec00 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa637e00640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa620003f60 0x7fa620004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa63ce00640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 0x7fa6200039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.594+0000 7fa63d800640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa620004d40 0x7fa6200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.595+0000 7fa637e00640 1 -- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 msgr2=0x7fa6200039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.595+0000 7fa637e00640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa63806ec00 0x7fa6200039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.595+0000 7fa637e00640 1 -- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa620004d40 msgr2=0x7fa6200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.595+0000 7fa637e00640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa620004d40 0x7fa6200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.595+0000 7fa637e00640 1 -- 172.21.15.97:0/1090365263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620003520 con 0x7fa620003f60 2024-03-06T07:27:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.596+0000 7fa637e00640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa620003f60 0x7fa620004800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa624001750 tx=0x7fa620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3be8ec4f830d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.596+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624001370 con 0x7fa620003f60 2024-03-06T07:27:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.596+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6240052d0 con 0x7fa620003f60 2024-03-06T07:27:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.596+0000 7fa637400640 1 -- 172.21.15.97:0/1090365263 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624003830 con 0x7fa620003f60 2024-03-06T07:27:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63f800640 1 -- 172.21.15.97:0/1090365263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa620003f60 msgr2=0x7fa620004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63f800640 1 --2- 172.21.15.97:0/1090365263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa620003f60 0x7fa620004800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa624001750 tx=0x7fa620005830 comp rx=0 tx=0).stop 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63d800640 1 -- 172.21.15.97:0/1090365263 reap_dead start 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63f800640 1 -- 172.21.15.97:0/1090365263 shutdown_connections 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63f800640 1 -- 172.21.15.97:0/1090365263 >> 172.21.15.97:0/1090365263 conn(0x7fa6380b1730 msgr2=0x7fa6380b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63f800640 1 -- 172.21.15.97:0/1090365263 shutdown_connections 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.597+0000 7fa63f800640 1 -- 172.21.15.97:0/1090365263 wait complete. 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.599+0000 7fa63f800640 1 Processor -- start 2024-03-06T07:27:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.599+0000 7fa63f800640 1 -- start start 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63f800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa620003f60 0x7fa6380c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63f800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 0x7fa638148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63f800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa638149ef0 0x7fa63814a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63f800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380b4010 con 0x7fa63814c420 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63f800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380b32b0 con 0x7fa638149ef0 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63f800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa638072340 con 0x7fa620003f60 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 0x7fa638148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 0x7fa638148600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32798/0 (socket says 172.21.15.97:32798) 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 -- 172.21.15.97:0/3323721860 learned_addr learned my addr 172.21.15.97:0/3323721860 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63ce00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa620003f60 0x7fa6380c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63d800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa638149ef0 0x7fa63814a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 -- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa620003f60 msgr2=0x7fa6380c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa620003f60 0x7fa6380c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 -- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa638149ef0 msgr2=0x7fa63814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa638149ef0 0x7fa63814a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa637e00640 1 -- 172.21.15.97:0/3323721860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380ca460 con 0x7fa63814c420 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.600+0000 7fa63d800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa638149ef0 0x7fa63814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.601+0000 7fa637e00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 0x7fa638148600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa624008fd0 tx=0x7fa624001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.601+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624041070 con 0x7fa63814c420 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.601+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa624005480 con 0x7fa63814c420 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.601+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624001370 con 0x7fa63814c420 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.601+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6380ca740 con 0x7fa63814c420 2024-03-06T07:27:37.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.601+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6380cad00 con 0x7fa63814c420 2024-03-06T07:27:37.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.602+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63804f330 con 0x7fa63814c420 2024-03-06T07:27:37.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.603+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fa624044070 con 0x7fa63814c420 2024-03-06T07:27:37.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.604+0000 7fa634c00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa61806b440 0x7fa61806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.604+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa6240ab7c0 con 0x7fa63814c420 2024-03-06T07:27:37.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.604+0000 7fa63ce00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa61806b440 0x7fa61806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.605+0000 7fa63ce00640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa61806b440 0x7fa61806d900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa62c001050 tx=0x7fa62c001430 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.608+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa62407ab00 con 0x7fa63814c420 2024-03-06T07:27:37.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.816+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6380ccf70 con 0x7fa63814c420 2024-03-06T07:27:37.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.818+0000 7fa634c00640 1 -- 172.21.15.97:0/3323721860 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12885 (secure 0 0 0) 0x7fa62407a400 con 0x7fa63814c420 2024-03-06T07:27:37.820 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:37.820 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:36.882739+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa61806b440 msgr2=0x7fa61806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa61806b440 0x7fa61806d900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa62c001050 tx=0x7fa62c001430 comp rx=0 tx=0).stop 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 msgr2=0x7fa638148600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 0x7fa638148600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa624008fd0 tx=0x7fa624001120 comp rx=0 tx=0).stop 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 shutdown_connections 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa61806b440 0x7fa61806d900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa638149ef0 0x7fa63814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa63814c420 0x7fa638148600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 --2- 172.21.15.97:0/3323721860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa620003f60 0x7fa6380c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 >> 172.21.15.97:0/3323721860 conn(0x7fa6380b1730 msgr2=0x7fa6381587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 shutdown_connections 2024-03-06T07:27:37.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:37.820+0000 7fa63f800640 1 -- 172.21.15.97:0/3323721860 wait complete. 2024-03-06T07:27:37.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:37.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.018+0000 7fc3e2600640 1 Processor -- start 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.018+0000 7fc3e2600640 1 -- start start 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3e2600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3e2600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc3dc0b5490 0x7fc3dc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3e2600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc0b3730 con 0x7fc3dc0d07f0 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3e2600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc0d4930 con 0x7fc3dc0b5490 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3e2600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc067b20 con 0x7fc3dc15e450 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3dbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc3dc0b5490 0x7fc3dc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3db400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc3dc0d07f0 0x7fc3dc0d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42592/0 (socket says 172.21.15.97:42592) 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.019+0000 7fc3db400640 1 -- 172.21.15.97:0/2716317405 learned_addr learned my addr 172.21.15.97:0/2716317405 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.020+0000 7fc3daa00640 1 --2- 172.21.15.97:0/2716317405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.020+0000 7fc3dbe00640 1 -- 172.21.15.97:0/2716317405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 msgr2=0x7fc3dc0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.020+0000 7fc3dbe00640 1 --2- 172.21.15.97:0/2716317405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.020+0000 7fc3dbe00640 1 -- 172.21.15.97:0/2716317405 >> v1:172.21.15.97:6789/0 conn(0x7fc3dc0d07f0 legacy=0x7fc3dc0d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.020+0000 7fc3dbe00640 1 -- 172.21.15.97:0/2716317405 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc0d4b80 con 0x7fc3dc0b5490 2024-03-06T07:27:38.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.020+0000 7fc3da000640 1 -- 172.21.15.97:0/2716317405 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597546885 0 0) 0x7fc3dc0b3730 con 0x7fc3dc0d07f0 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.021+0000 7fc3dbe00640 1 --2- 172.21.15.97:0/2716317405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0cef60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009d90 tx=0x7fc3d402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d321003d6aacf355 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.021+0000 7fc3da000640 1 -- 172.21.15.97:0/2716317405 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4039070 con 0x7fc3dc0b5490 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.021+0000 7fc3da000640 1 -- 172.21.15.97:0/2716317405 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d402cbc0 con 0x7fc3dc0b5490 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.021+0000 7fc3da000640 1 -- 172.21.15.97:0/2716317405 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4034440 con 0x7fc3dc0b5490 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 -- 172.21.15.97:0/2716317405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 msgr2=0x7fc3dc0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 --2- 172.21.15.97:0/2716317405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0cef60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009d90 tx=0x7fc3d402be10 comp rx=0 tx=0).stop 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 -- 172.21.15.97:0/2716317405 shutdown_connections 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 --2- 172.21.15.97:0/2716317405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0cef60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 --2- 172.21.15.97:0/2716317405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 -- 172.21.15.97:0/2716317405 >> 172.21.15.97:0/2716317405 conn(0x7fc3dc0d3e10 msgr2=0x7fc3dc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 -- 172.21.15.97:0/2716317405 shutdown_connections 2024-03-06T07:27:38.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.022+0000 7fc3e2600640 1 -- 172.21.15.97:0/2716317405 wait complete. 2024-03-06T07:27:38.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.023+0000 7fc3e2600640 1 Processor -- start 2024-03-06T07:27:38.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 -- start start 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3dc0d07f0 0x7fc3dc0b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0b9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b3f60 con 0x7fc3dc0b5490 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b3c80 con 0x7fc3dc0d07f0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.024+0000 7fc3e2600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0d1bf0 con 0x7fc3dc15e450 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0c7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32826/0 (socket says 172.21.15.97:32826) 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 -- 172.21.15.97:0/4157534363 learned_addr learned my addr 172.21.15.97:0/4157534363 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 -- 172.21.15.97:0/4157534363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 msgr2=0x7fc3dc0b9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 -- 172.21.15.97:0/4157534363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3dc0d07f0 msgr2=0x7fc3dc0b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3dc0d07f0 0x7fc3dc0b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 -- 172.21.15.97:0/4157534363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc0ca7c0 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3db400640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0c7b00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00c500 tx=0x7fc3cc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc018070 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc007580 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.025+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc004a20 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.026+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc0cb180 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.026+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc0cb740 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.027+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc068230 con 0x7fc3dc0b5490 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.028+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fc3cc01b070 con 0x7fc3dc0b5490 2024-03-06T07:27:38.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.028+0000 7fc3d3400640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3bc06b440 0x7fc3bc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.029+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3cc089030 con 0x7fc3dc0b5490 2024-03-06T07:27:38.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.029+0000 7fc3daa00640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3bc06b440 0x7fc3bc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.029+0000 7fc3daa00640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3bc06b440 0x7fc3bc06d900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8006fd0 tx=0x7fc3c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.033+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3dc068230 con 0x7fc3dc0b5490 2024-03-06T07:27:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.227+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3dc0d21b0 con 0x7fc3dc0b5490 2024-03-06T07:27:38.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.228+0000 7fc3d3400640 1 -- 172.21.15.97:0/4157534363 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3cc058370 con 0x7fc3dc0b5490 2024-03-06T07:27:38.230 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:27:38.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3bc06b440 msgr2=0x7fc3bc06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3bc06b440 0x7fc3bc06d900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8006fd0 tx=0x7fc3c8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 msgr2=0x7fc3dc0c7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0c7b00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00c500 tx=0x7fc3cc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 shutdown_connections 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3bc06b440 0x7fc3bc06d900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc3dc15e450 0x7fc3dc0b9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3dc0d07f0 0x7fc3dc0b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 --2- 172.21.15.97:0/4157534363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3dc0b5490 0x7fc3dc0c7b00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 >> 172.21.15.97:0/4157534363 conn(0x7fc3dc0d3e10 msgr2=0x7fc3dc06c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 shutdown_connections 2024-03-06T07:27:38.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.230+0000 7fc3e2600640 1 -- 172.21.15.97:0/4157534363 wait complete. 2024-03-06T07:27:38.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.423+0000 7f0be7400640 1 Processor -- start 2024-03-06T07:27:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.423+0000 7f0be7400640 1 -- start start 2024-03-06T07:27:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be7400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0be006b6d0 0x7f0be0168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be7400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be006eb10 con 0x7f0be015e070 2024-03-06T07:27:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be7400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b5010 con 0x7f0be00cd860 2024-03-06T07:27:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be7400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be00b52f0 con 0x7f0be006b6d0 2024-03-06T07:27:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be4a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0be015e070 0x7f0be0066d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42604/0 (socket says 172.21.15.97:42604) 2024-03-06T07:27:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be4a00640 1 -- 172.21.15.97:0/230409450 learned_addr learned my addr 172.21.15.97:0/230409450 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0be5400640 1 --2- 172.21.15.97:0/230409450 >> v2:172.21.15.135:3300/0 conn(0x7f0be006b6d0 0x7f0be0168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917506146 0 0) 0x7f0be006eb10 con 0x7f0be015e070 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.425+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc8003520 con 0x7f0be015e070 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809744417 0 0) 0x7f0bc8003520 con 0x7f0be015e070 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 >> v2:172.21.15.135:3300/0 conn(0x7f0be006b6d0 msgr2=0x7f0be0168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 --2- 172.21.15.97:0/230409450 >> v2:172.21.15.135:3300/0 conn(0x7f0be006b6d0 0x7f0be0168fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 >> v1:172.21.15.135:6789/0 conn(0x7f0be00cd860 legacy=0x7f0be006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00b55d0 con 0x7f0be015e070 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0bd4001690 con 0x7f0be015e070 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 >> v1:172.21.15.97:6789/0 conn(0x7f0be015e070 legacy=0x7f0be0066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8003a00 0x7f0bc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bc80051c0 0x7f0bc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be00b52f0 con 0x7f0bc8004800 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc800c0c0 con 0x7f0bc8003a00 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc800c2c0 con 0x7f0bc80051c0 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdfe00640 1 -- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bc80051c0 msgr2=0x7f0bc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bc80051c0 0x7f0bc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdfe00640 1 -- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8003a00 msgr2=0x7f0bc8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0be4a00640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8003a00 0x7f0bc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.426+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8003a00 0x7f0bc8003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.427+0000 7f0be5400640 1 -- 172.21.15.97:0/230409450 reap_dead start 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.427+0000 7f0bdfe00640 1 -- 172.21.15.97:0/230409450 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8003520 con 0x7f0bc8004800 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc001750 tx=0x7f0bc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8ef86b0075f138 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc001370 con 0x7f0bc8004800 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0052d0 con 0x7f0bc8004800 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0bdf400640 1 -- 172.21.15.97:0/230409450 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc003830 con 0x7f0bc8004800 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 -- 172.21.15.97:0/230409450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bc8004800 msgr2=0x7f0bc8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc001750 tx=0x7f0bc8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 -- 172.21.15.97:0/230409450 shutdown_connections 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 --2- 172.21.15.97:0/230409450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 -- 172.21.15.97:0/230409450 >> 172.21.15.97:0/230409450 conn(0x7f0be00b1e10 msgr2=0x7f0be00b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 -- 172.21.15.97:0/230409450 shutdown_connections 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.428+0000 7f0be7400640 1 -- 172.21.15.97:0/230409450 wait complete. 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.430+0000 7f0be7400640 1 Processor -- start 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.430+0000 7f0be7400640 1 -- start start 2024-03-06T07:27:38.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0be7400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8004800 0x7f0be00ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0be7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 0x7f0be00ced60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0be7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0be015b630 0x7f0be0153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0be7400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be015e450 con 0x7f0be00c3db0 2024-03-06T07:27:38.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0be7400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be00b5b90 con 0x7f0bc8004800 2024-03-06T07:27:38.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0be7400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be00b60b0 con 0x7f0be015b630 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0bdfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 0x7f0be00ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0bdfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 0x7f0be00ced60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32834/0 (socket says 172.21.15.97:32834) 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.431+0000 7f0bdfe00640 1 -- 172.21.15.97:0/4087912290 learned_addr learned my addr 172.21.15.97:0/4087912290 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0be4a00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8004800 0x7f0be00ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0be5400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0be015b630 0x7f0be0153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdfe00640 1 -- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0be015b630 msgr2=0x7f0be0153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0be015b630 0x7f0be0153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdfe00640 1 -- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8004800 msgr2=0x7f0be00ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8004800 0x7f0be00ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdfe00640 1 -- 172.21.15.97:0/4087912290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00d0cc0 con 0x7f0be00c3db0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0be4a00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8004800 0x7f0be00ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdfe00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 0x7f0be00ced60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc001870 tx=0x7f0bcc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc041070 con 0x7f0be00c3db0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc005480 con 0x7f0be00c3db0 2024-03-06T07:27:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc007a30 con 0x7f0be00c3db0 2024-03-06T07:27:38.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be00d0fa0 con 0x7f0be00c3db0 2024-03-06T07:27:38.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.432+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be00ce130 con 0x7f0be00c3db0 2024-03-06T07:27:38.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.434+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be00cd650 con 0x7f0be00c3db0 2024-03-06T07:27:38.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.435+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f0bcc044070 con 0x7f0be00c3db0 2024-03-06T07:27:38.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.435+0000 7f0bdcc00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bc006b470 0x7f0bc006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.436+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0bcc0ab860 con 0x7f0be00c3db0 2024-03-06T07:27:38.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.436+0000 7f0be4a00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bc006b470 0x7f0bc006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.436+0000 7f0be4a00640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bc006b470 0x7f0bc006d930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4001eb0 tx=0x7f0bd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.440+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0be00cd650 con 0x7f0be00c3db0 2024-03-06T07:27:38.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.621+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0be015ce10 con 0x7f0be00c3db0 2024-03-06T07:27:38.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.622+0000 7f0bdcc00640 1 -- 172.21.15.97:0/4087912290 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0be015ce10 con 0x7f0be00c3db0 2024-03-06T07:27:38.624 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:38.625 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:27:38.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:27:38.625 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:27:38.625 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:27:38.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:27:38.625 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.624+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bc006b470 msgr2=0x7f0bc006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.624+0000 7f0be7400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bc006b470 0x7f0bc006d930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4001eb0 tx=0x7f0bd4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.624+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 msgr2=0x7f0be00ced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.624+0000 7f0be7400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 0x7f0be00ced60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc001870 tx=0x7f0bcc001120 comp rx=0 tx=0).stop 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 shutdown_connections 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bc006b470 0x7f0bc006d930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0be015b630 0x7f0be0153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0be00c3db0 0x7f0be00ced60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 --2- 172.21.15.97:0/4087912290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bc8004800 0x7f0be00ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 >> 172.21.15.97:0/4087912290 conn(0x7f0be00b1e10 msgr2=0x7f0be00b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 shutdown_connections 2024-03-06T07:27:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.625+0000 7f0be7400640 1 -- 172.21.15.97:0/4087912290 wait complete. 2024-03-06T07:27:38.643 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:38.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:38.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.850+0000 7fa82a400640 1 Processor -- start 2024-03-06T07:27:38.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.850+0000 7fa82a400640 1 -- start start 2024-03-06T07:27:38.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa82a400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa82406e570 0x7fa8240b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa82a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa824161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa82a400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8240adf70 con 0x7fa82415e170 2024-03-06T07:27:38.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa82a400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82406ea70 con 0x7fa82406e570 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa82a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240b3690 con 0x7fa8240b4990 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa81be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa82406e570 0x7fa8240b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa81be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa82406e570 0x7fa8240b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32838/0 (socket says 172.21.15.97:32838) 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa81be00640 1 -- 172.21.15.97:0/2694192641 learned_addr learned my addr 172.21.15.97:0/2694192641 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.852+0000 7fa823e00640 1 --2- 172.21.15.97:0/2694192641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa824161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.853+0000 7fa81be00640 1 -- 172.21.15.97:0/2694192641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 msgr2=0x7fa824161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.853+0000 7fa81be00640 1 --2- 172.21.15.97:0/2694192641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa824161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.853+0000 7fa81be00640 1 -- 172.21.15.97:0/2694192641 >> v1:172.21.15.97:6789/0 conn(0x7fa82415e170 legacy=0x7fa8240ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.853+0000 7fa81be00640 1 -- 172.21.15.97:0/2694192641 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240b3970 con 0x7fa82406e570 2024-03-06T07:27:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.853+0000 7fa822a00640 1 -- 172.21.15.97:0/2694192641 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038339285 0 0) 0x7fa8240adf70 con 0x7fa82415e170 2024-03-06T07:27:38.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.854+0000 7fa81be00640 1 --2- 172.21.15.97:0/2694192641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8240b4450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009870 tx=0x7fa80c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db41bffb8642a297 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.854+0000 7fa822a00640 1 -- 172.21.15.97:0/2694192641 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c035020 con 0x7fa82406e570 2024-03-06T07:27:38.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.854+0000 7fa822a00640 1 -- 172.21.15.97:0/2694192641 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80c031640 con 0x7fa82406e570 2024-03-06T07:27:38.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.854+0000 7fa822a00640 1 -- 172.21.15.97:0/2694192641 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c02d040 con 0x7fa82406e570 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 -- 172.21.15.97:0/2694192641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 msgr2=0x7fa8240b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 --2- 172.21.15.97:0/2694192641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8240b4450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009870 tx=0x7fa80c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 -- 172.21.15.97:0/2694192641 shutdown_connections 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 --2- 172.21.15.97:0/2694192641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa824161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 --2- 172.21.15.97:0/2694192641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8240b4450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 -- 172.21.15.97:0/2694192641 >> 172.21.15.97:0/2694192641 conn(0x7fa8240cfa80 msgr2=0x7fa8240cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 -- 172.21.15.97:0/2694192641 shutdown_connections 2024-03-06T07:27:38.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.855+0000 7fa82a400640 1 -- 172.21.15.97:0/2694192641 wait complete. 2024-03-06T07:27:38.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.857+0000 7fa82a400640 1 Processor -- start 2024-03-06T07:27:38.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.857+0000 7fa82a400640 1 -- start start 2024-03-06T07:27:38.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa82a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8241e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa82a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa8240af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa82a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa82415e170 0x7fa824156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa82a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240b4dd0 con 0x7fa82406e570 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa82a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240b3f30 con 0x7fa82415e170 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa82a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82406df80 con 0x7fa8240b4990 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8241e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8241e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32842/0 (socket says 172.21.15.97:32842) 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 -- 172.21.15.97:0/1102625286 learned_addr learned my addr 172.21.15.97:0/1102625286 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823e00640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa82415e170 0x7fa824156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa81be00640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa8240af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 -- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 msgr2=0x7fa8240af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa8240af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 -- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa82415e170 msgr2=0x7fa824156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa82415e170 0x7fa824156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa823400640 1 -- 172.21.15.97:0/1102625286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8241e8ae0 con 0x7fa82406e570 2024-03-06T07:27:38.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.858+0000 7fa81be00640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa8240af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:38.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.859+0000 7fa823400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8241e84f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c410 tx=0x7fa81400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.859+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814017070 con 0x7fa82406e570 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.859+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8140035f0 con 0x7fa82406e570 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.859+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8140039e0 con 0x7fa82406e570 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.859+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8241e8dc0 con 0x7fa82406e570 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.859+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8241e9380 con 0x7fa82406e570 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.861+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82404f330 con 0x7fa82406e570 2024-03-06T07:27:38.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.861+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fa81401a070 con 0x7fa82406e570 2024-03-06T07:27:38.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.862+0000 7fa81b400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80006b440 0x7fa80006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.863+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa814089170 con 0x7fa82406e570 2024-03-06T07:27:38.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.863+0000 7fa81be00640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80006b440 0x7fa80006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.863+0000 7fa81be00640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80006b440 0x7fa80006d900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa80c036000 tx=0x7fa80c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:38.867+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa814057d30 con 0x7fa82406e570 2024-03-06T07:27:39.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:39.065+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa8241612c0 con 0x7fa82406e570 2024-03-06T07:27:39.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:39.796+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8140543a0 con 0x7fa82406e570 2024-03-06T07:27:40.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.049+0000 7fa81b400640 1 -- 172.21.15.97:0/1102625286 <== mon.0 v2:172.21.15.97:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa814077a60 con 0x7fa82406e570 2024-03-06T07:27:40.052 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80006b440 msgr2=0x7fa80006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80006b440 0x7fa80006d900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa80c036000 tx=0x7fa80c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 msgr2=0x7fa8241e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8241e84f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c410 tx=0x7fa81400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 shutdown_connections 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80006b440 0x7fa80006d900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa82415e170 0x7fa824156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8240b4990 0x7fa8240af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 --2- 172.21.15.97:0/1102625286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82406e570 0x7fa8241e84f0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c410 tx=0x7fa81400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.053+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 >> 172.21.15.97:0/1102625286 conn(0x7fa8240cfa80 msgr2=0x7fa8240cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.054+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 shutdown_connections 2024-03-06T07:27:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.054+0000 7fa82a400640 1 -- 172.21.15.97:0/1102625286 wait complete. 2024-03-06T07:27:40.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:40.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.284+0000 7fbb60600640 1 Processor -- start 2024-03-06T07:27:40.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.284+0000 7fbb60600640 1 -- start start 2024-03-06T07:27:40.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb60600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 0x7fbb580727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb60600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbb580b3650 0x7fbb580b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb60600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb580b4710 con 0x7fbb580cda90 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb60600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb58072310 con 0x7fbb580b3650 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb60600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580719e0 con 0x7fbb580b3df0 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb5e600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fbb580cda90 0x7fbb5806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42624/0 (socket says 172.21.15.97:42624) 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb5e600640 1 -- 172.21.15.97:0/863415272 learned_addr learned my addr 172.21.15.97:0/863415272 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.286+0000 7fbb5d200640 1 --2- 172.21.15.97:0/863415272 >> v2:172.21.15.135:3300/0 conn(0x7fbb580b3650 0x7fbb580b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb5dc00640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 0x7fbb580727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691160767 0 0) 0x7fbb580b4710 con 0x7fbb580cda90 2024-03-06T07:27:40.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb40005730 con 0x7fbb580cda90 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132078410 0 0) 0x7fbb40005730 con 0x7fbb580cda90 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 msgr2=0x7fbb580727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 0x7fbb580727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 >> v2:172.21.15.135:3300/0 conn(0x7fbb580b3650 msgr2=0x7fbb580b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 --2- 172.21.15.97:0/863415272 >> v2:172.21.15.135:3300/0 conn(0x7fbb580b3650 0x7fbb580b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb58071cc0 con 0x7fbb580cda90 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fbb500019f0 con 0x7fbb580cda90 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 >> v1:172.21.15.97:6789/0 conn(0x7fbb580cda90 legacy=0x7fbb5806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40005bc0 0x7fbb40005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 0x7fbb40006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb40006f50 0x7fbb400073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58072310 con 0x7fbb40006f50 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580719e0 con 0x7fbb40005bc0 2024-03-06T07:27:40.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.287+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4000dc50 con 0x7fbb580b3df0 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5e600640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb40006f50 0x7fbb400073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5dc00640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40005bc0 0x7fbb40005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5e600640 1 -- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 msgr2=0x7fbb40006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5e600640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb580b3df0 0x7fbb40006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5e600640 1 -- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40005bc0 msgr2=0x7fbb40005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5e600640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40005bc0 0x7fbb40005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.288+0000 7fbb5e600640 1 -- 172.21.15.97:0/863415272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40005730 con 0x7fbb40006f50 2024-03-06T07:27:40.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.289+0000 7fbb5e600640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb40006f50 0x7fbb400073d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000a7a0 tx=0x7fbb40007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f4067417da9512 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.289+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5000a3f0 con 0x7fbb40006f50 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.289+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb50034630 con 0x7fbb40006f50 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.289+0000 7fbb57e00640 1 -- 172.21.15.97:0/863415272 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50034a40 con 0x7fbb40006f50 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb60600640 1 -- 172.21.15.97:0/863415272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb40006f50 msgr2=0x7fbb400073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb60600640 1 --2- 172.21.15.97:0/863415272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb40006f50 0x7fbb400073d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000a7a0 tx=0x7fbb40007a40 comp rx=0 tx=0).stop 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb5e600640 1 -- 172.21.15.97:0/863415272 reap_dead start 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb60600640 1 -- 172.21.15.97:0/863415272 shutdown_connections 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb60600640 1 -- 172.21.15.97:0/863415272 >> 172.21.15.97:0/863415272 conn(0x7fbb580b1940 msgr2=0x7fbb580b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb60600640 1 -- 172.21.15.97:0/863415272 shutdown_connections 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.290+0000 7fbb60600640 1 -- 172.21.15.97:0/863415272 wait complete. 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.292+0000 7fbb60600640 1 Processor -- start 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.292+0000 7fbb60600640 1 -- start start 2024-03-06T07:27:40.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb60600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40006f50 0x7fbb581f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb60600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 0x7fbb581f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb60600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb58155190 0x7fbb58155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb60600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5815b240 con 0x7fbb580cb770 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb60600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580b3b30 con 0x7fbb40006f50 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb60600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5815b3c0 con 0x7fbb58155190 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 0x7fbb581f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 0x7fbb581f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32868/0 (socket says 172.21.15.97:32868) 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 -- 172.21.15.97:0/68863215 learned_addr learned my addr 172.21.15.97:0/68863215 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5e600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb58155190 0x7fbb58155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5dc00640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40006f50 0x7fbb581f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 -- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb58155190 msgr2=0x7fbb58155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb58155190 0x7fbb58155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 -- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40006f50 msgr2=0x7fbb581f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40006f50 0x7fbb581f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.293+0000 7fbb5d200640 1 -- 172.21.15.97:0/68863215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb58143f40 con 0x7fbb580cb770 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.294+0000 7fbb5d200640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 0x7fbb581f3b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400e7c0 tx=0x7fbb4400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.294+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44017070 con 0x7fbb580cb770 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.294+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4400c940 con 0x7fbb580cb770 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.294+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400cd30 con 0x7fbb580cb770 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.294+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb581f4090 con 0x7fbb580cb770 2024-03-06T07:27:40.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.294+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb58143180 con 0x7fbb580cb770 2024-03-06T07:27:40.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.295+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5804f330 con 0x7fbb580cb770 2024-03-06T07:27:40.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.296+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb4401a070 con 0x7fbb580cb770 2024-03-06T07:27:40.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.297+0000 7fbb55600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb3806b500 0x7fbb3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.298+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbb44089ed0 con 0x7fbb580cb770 2024-03-06T07:27:40.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.298+0000 7fbb5dc00640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb3806b500 0x7fbb3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.298+0000 7fbb5dc00640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb3806b500 0x7fbb3806d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c006fd0 tx=0x7fbb4c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.303+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb44058f20 con 0x7fbb580cb770 2024-03-06T07:27:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:40.507+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbb580cc0e0 con 0x7fbb580cb770 2024-03-06T07:27:41.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.064+0000 7fbb55600640 1 -- 172.21.15.97:0/68863215 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbb44058820 con 0x7fbb580cb770 2024-03-06T07:27:41.067 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb3806b500 msgr2=0x7fbb3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb3806b500 0x7fbb3806d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c006fd0 tx=0x7fbb4c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 msgr2=0x7fbb581f3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 0x7fbb581f3b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400e7c0 tx=0x7fbb4400ec80 comp rx=0 tx=0).stop 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 shutdown_connections 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb3806b500 0x7fbb3806d9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbb58155190 0x7fbb58155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbb580cb770 0x7fbb581f3b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 --2- 172.21.15.97:0/68863215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbb40006f50 0x7fbb581f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.068+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 >> 172.21.15.97:0/68863215 conn(0x7fbb580b1940 msgr2=0x7fbb581566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.069+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 shutdown_connections 2024-03-06T07:27:41.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.069+0000 7fbb60600640 1 -- 172.21.15.97:0/68863215 wait complete. 2024-03-06T07:27:41.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:41.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.287+0000 7f3433e00640 1 Processor -- start 2024-03-06T07:27:41.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.288+0000 7f3433e00640 1 -- start start 2024-03-06T07:27:41.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f3433e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f34340ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f3433e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3434072840 0x7f343406de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f3433e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f343406e340 0x7f34340cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f3433e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34340b4bf0 con 0x7f3434072840 2024-03-06T07:27:41.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f3433e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343406e790 con 0x7f343406e340 2024-03-06T07:27:41.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f3433e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343415ddd0 con 0x7f343415d9f0 2024-03-06T07:27:41.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f342a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3434072840 0x7f343406de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f342a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3434072840 0x7f343406de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32870/0 (socket says 172.21.15.97:32870) 2024-03-06T07:27:41.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.289+0000 7f342a000640 1 -- 172.21.15.97:0/1633553865 learned_addr learned my addr 172.21.15.97:0/1633553865 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f3433400640 1 --2- 172.21.15.97:0/1633553865 >> v2:172.21.15.135:3300/0 conn(0x7f343406e340 0x7f34340cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f3432a00640 1 --2- 172.21.15.97:0/1633553865 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f34340ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f342a000640 1 -- 172.21.15.97:0/1633553865 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 msgr2=0x7f34340ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f342a000640 1 --2- 172.21.15.97:0/1633553865 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f34340ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f342a000640 1 -- 172.21.15.97:0/1633553865 >> v2:172.21.15.135:3300/0 conn(0x7f343406e340 msgr2=0x7f34340cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f342a000640 1 --2- 172.21.15.97:0/1633553865 >> v2:172.21.15.135:3300/0 conn(0x7f343406e340 0x7f34340cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f342a000640 1 -- 172.21.15.97:0/1633553865 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34340cef20 con 0x7f3434072840 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.290+0000 7f3433400640 1 --2- 172.21.15.97:0/1633553865 >> v2:172.21.15.135:3300/0 conn(0x7f343406e340 0x7f34340cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.291+0000 7f342a000640 1 --2- 172.21.15.97:0/1633553865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3434072840 0x7f343406de00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f343406e9c0 tx=0x7f341c02dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3ac00a0342071e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.291+0000 7f3432000640 1 -- 172.21.15.97:0/1633553865 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c034070 con 0x7f3434072840 2024-03-06T07:27:41.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.291+0000 7f3432000640 1 -- 172.21.15.97:0/1633553865 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c02ec50 con 0x7f3434072840 2024-03-06T07:27:41.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.291+0000 7f3432000640 1 -- 172.21.15.97:0/1633553865 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c038400 con 0x7f3434072840 2024-03-06T07:27:41.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 -- 172.21.15.97:0/1633553865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3434072840 msgr2=0x7f343406de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 --2- 172.21.15.97:0/1633553865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3434072840 0x7f343406de00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f343406e9c0 tx=0x7f341c02dea0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 -- 172.21.15.97:0/1633553865 shutdown_connections 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 --2- 172.21.15.97:0/1633553865 >> v2:172.21.15.135:3300/0 conn(0x7f343406e340 0x7f34340cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 --2- 172.21.15.97:0/1633553865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3434072840 0x7f343406de00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 --2- 172.21.15.97:0/1633553865 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f34340ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 -- 172.21.15.97:0/1633553865 >> 172.21.15.97:0/1633553865 conn(0x7f343406afc0 msgr2=0x7f343406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 -- 172.21.15.97:0/1633553865 shutdown_connections 2024-03-06T07:27:41.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.292+0000 7f3433e00640 1 -- 172.21.15.97:0/1633553865 wait complete. 2024-03-06T07:27:41.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.294+0000 7f3433e00640 1 Processor -- start 2024-03-06T07:27:41.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.294+0000 7f3433e00640 1 -- start start 2024-03-06T07:27:41.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3433e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 0x7f34341f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3433e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3434072840 0x7f3434150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3433e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f3434157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3433e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340b4110 con 0x7f343406e340 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3433e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340b3e70 con 0x7f3434072840 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3433e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34340b43f0 con 0x7f343415d9f0 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 0x7f34341f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 0x7f34341f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32872/0 (socket says 172.21.15.97:32872) 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 -- 172.21.15.97:0/326819051 learned_addr learned my addr 172.21.15.97:0/326819051 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 -- 172.21.15.97:0/326819051 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 msgr2=0x7f3434157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f3434157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 -- 172.21.15.97:0/326819051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3434072840 msgr2=0x7f3434150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3434072840 0x7f3434150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.295+0000 7f3432a00640 1 -- 172.21.15.97:0/326819051 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434150b10 con 0x7f343406e340 2024-03-06T07:27:41.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.296+0000 7f3432a00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 0x7f34341f3a60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f342400c7f0 tx=0x7f342400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.296+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424015070 con 0x7f343406e340 2024-03-06T07:27:41.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.296+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3424004220 con 0x7f343406e340 2024-03-06T07:27:41.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.296+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424007620 con 0x7f343406e340 2024-03-06T07:27:41.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.296+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3434150df0 con 0x7f343406e340 2024-03-06T07:27:41.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.296+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34341513b0 con 0x7f343406e340 2024-03-06T07:27:41.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.297+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343404f330 con 0x7f343406e340 2024-03-06T07:27:41.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.298+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34240079d0 con 0x7f343406e340 2024-03-06T07:27:41.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.300+0000 7f342be00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f341006b500 0x7f341006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.300+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f342408a1a0 con 0x7f343406e340 2024-03-06T07:27:41.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.300+0000 7f342a000640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f341006b500 0x7f341006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.300+0000 7f342a000640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f341006b500 0x7f341006d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f341c039000 tx=0x7f341c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.305+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3424059070 con 0x7f343406e340 2024-03-06T07:27:41.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:41.539+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3434072210 con 0x7f343406e340 2024-03-06T07:27:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.076+0000 7f342be00640 1 -- 172.21.15.97:0/326819051 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3424058970 con 0x7f343406e340 2024-03-06T07:27:42.084 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.084+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f341006b500 msgr2=0x7f341006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.084+0000 7f3433e00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f341006b500 0x7f341006d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f341c039000 tx=0x7f341c030000 comp rx=0 tx=0).stop 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.084+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 msgr2=0x7f34341f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 0x7f34341f3a60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f342400c7f0 tx=0x7f342400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 shutdown_connections 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f341006b500 0x7f341006d9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f343415d9f0 0x7f3434157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3434072840 0x7f3434150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 --2- 172.21.15.97:0/326819051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f343406e340 0x7f34341f3a60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.085+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 >> 172.21.15.97:0/326819051 conn(0x7f343406afc0 msgr2=0x7f34340affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.088+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 shutdown_connections 2024-03-06T07:27:42.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.088+0000 7f3433e00640 1 -- 172.21.15.97:0/326819051 wait complete. 2024-03-06T07:27:42.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:42.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.294+0000 7f4bca800640 1 Processor -- start 2024-03-06T07:27:42.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.294+0000 7f4bca800640 1 -- start start 2024-03-06T07:27:42.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bca800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4bc415acb0 0x7f4bc40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bca800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4bc40b4680 0x7f4bc406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bca800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc415b090 con 0x7f4bc4072340 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bca800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc40b2a30 con 0x7f4bc40b4680 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bca800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc40b4360 con 0x7f4bc415acb0 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4bc40b4680 0x7f4bc406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4bc40b4680 0x7f4bc406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32886/0 (socket says 172.21.15.97:32886) 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bbaa00640 1 -- 172.21.15.97:0/2917772487 learned_addr learned my addr 172.21.15.97:0/2917772487 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bc3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4bc4072340 0x7f4bc40ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42634/0 (socket says 172.21.15.97:42634) 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bc3400640 1 --2- 172.21.15.97:0/2917772487 >> v2:172.21.15.135:3300/0 conn(0x7f4bc415acb0 0x7f4bc40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bbaa00640 1 -- 172.21.15.97:0/2917772487 >> v2:172.21.15.135:3300/0 conn(0x7f4bc415acb0 msgr2=0x7f4bc40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bbaa00640 1 --2- 172.21.15.97:0/2917772487 >> v2:172.21.15.135:3300/0 conn(0x7f4bc415acb0 0x7f4bc40b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bbaa00640 1 -- 172.21.15.97:0/2917772487 >> v1:172.21.15.97:6789/0 conn(0x7f4bc4072340 legacy=0x7f4bc40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.296+0000 7f4bc2a00640 1 -- 172.21.15.97:0/2917772487 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576212254 0 0) 0x7f4bc415b090 con 0x7f4bc4072340 2024-03-06T07:27:42.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.297+0000 7f4bbaa00640 1 -- 172.21.15.97:0/2917772487 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc40b2cf0 con 0x7f4bc40b4680 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bbaa00640 1 --2- 172.21.15.97:0/2917772487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc40b4680 0x7f4bc406e4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40b27f0 tx=0x7f4bac02e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5717010df58954 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bc2a00640 1 -- 172.21.15.97:0/2917772487 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac03b070 con 0x7f4bc40b4680 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bc2a00640 1 -- 172.21.15.97:0/2917772487 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac036440 con 0x7f4bc40b4680 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bc2a00640 1 -- 172.21.15.97:0/2917772487 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac036810 con 0x7f4bc40b4680 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bca800640 1 -- 172.21.15.97:0/2917772487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc40b4680 msgr2=0x7f4bc406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bca800640 1 --2- 172.21.15.97:0/2917772487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc40b4680 0x7f4bc406e4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40b27f0 tx=0x7f4bac02e170 comp rx=0 tx=0).stop 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bca800640 1 -- 172.21.15.97:0/2917772487 shutdown_connections 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bca800640 1 --2- 172.21.15.97:0/2917772487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc40b4680 0x7f4bc406e4f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bca800640 1 --2- 172.21.15.97:0/2917772487 >> v2:172.21.15.135:3300/0 conn(0x7f4bc415acb0 0x7f4bc40b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.298+0000 7f4bca800640 1 -- 172.21.15.97:0/2917772487 >> 172.21.15.97:0/2917772487 conn(0x7f4bc40b1730 msgr2=0x7f4bc40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.299+0000 7f4bca800640 1 -- 172.21.15.97:0/2917772487 shutdown_connections 2024-03-06T07:27:42.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.299+0000 7f4bca800640 1 -- 172.21.15.97:0/2917772487 wait complete. 2024-03-06T07:27:42.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.300+0000 7f4bca800640 1 Processor -- start 2024-03-06T07:27:42.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.300+0000 7f4bca800640 1 -- start start 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bca800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bc4072340 0x7f4bc40c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bca800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bc40b4680 0x7f4bc40cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 0x7f4bc40ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bca800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40b3d10 con 0x7f4bc415acb0 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bca800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40b32b0 con 0x7f4bc40b4680 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bca800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40b4010 con 0x7f4bc4072340 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bc3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 0x7f4bc40ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.301+0000 7f4bc3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 0x7f4bc40ce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32900/0 (socket says 172.21.15.97:32900) 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 -- 172.21.15.97:0/3383113914 learned_addr learned my addr 172.21.15.97:0/3383113914 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bbaa00640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bc40b4680 0x7f4bc40cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 -- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bc4072340 msgr2=0x7f4bc40c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bc4072340 0x7f4bc40c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 -- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bc40b4680 msgr2=0x7f4bc40cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bc40b4680 0x7f4bc40cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 -- 172.21.15.97:0/3383113914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc414bdc0 con 0x7f4bc415acb0 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bc3e00640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 0x7f4bc40ce500 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c700 tx=0x7f4bbc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc018070 con 0x7f4bc415acb0 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc00ce10 con 0x7f4bc415acb0 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc00a860 con 0x7f4bc415acb0 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.302+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc414c0a0 con 0x7f4bc415acb0 2024-03-06T07:27:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.303+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc414c660 con 0x7f4bc415acb0 2024-03-06T07:27:42.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.304+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc404f330 con 0x7f4bc415acb0 2024-03-06T07:27:42.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.305+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bbc00aa00 con 0x7f4bc415acb0 2024-03-06T07:27:42.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.305+0000 7f4bbbe00640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ba006b500 0x7f4ba006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.306+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bbc089680 con 0x7f4bc415acb0 2024-03-06T07:27:42.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.306+0000 7f4bc3400640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ba006b500 0x7f4ba006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.306+0000 7f4bc3400640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ba006b500 0x7f4ba006d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40098f0 tx=0x7f4bb4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.310+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bbc058590 con 0x7f4bc415acb0 2024-03-06T07:27:42.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.495+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bc415a4d0 con 0x7f4bc415acb0 2024-03-06T07:27:42.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.497+0000 7f4bbbe00640 1 -- 172.21.15.97:0/3383113914 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16606 (secure 0 0 0) 0x7f4bbc057e90 con 0x7f4bc415acb0 2024-03-06T07:27:42.499 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:42.499 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":21,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:41.541049+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:42.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ba006b500 msgr2=0x7f4ba006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ba006b500 0x7f4ba006d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40098f0 tx=0x7f4bb4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:42.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 msgr2=0x7f4bc40ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 0x7f4bc40ce500 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c700 tx=0x7f4bbc00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 shutdown_connections 2024-03-06T07:27:42.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ba006b500 0x7f4ba006d9c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bc415acb0 0x7f4bc40ce500 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bc40b4680 0x7f4bc40cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 --2- 172.21.15.97:0/3383113914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4bc4072340 0x7f4bc40c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.499+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 >> 172.21.15.97:0/3383113914 conn(0x7f4bc40b1730 msgr2=0x7f4bc40ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.500+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 shutdown_connections 2024-03-06T07:27:42.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.500+0000 7f4bca800640 1 -- 172.21.15.97:0/3383113914 wait complete. 2024-03-06T07:27:42.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:42.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.714+0000 7fdf72000640 1 Processor -- start 2024-03-06T07:27:42.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.714+0000 7fdf72000640 1 -- start start 2024-03-06T07:27:42.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf72000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf72000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf6c0ae670 0x7fdf6c15e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf72000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c161900 con 0x7fdf6c1620c0 2024-03-06T07:27:42.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf72000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c0b49e0 con 0x7fdf6c0ae670 2024-03-06T07:27:42.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf72000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c15ebd0 con 0x7fdf6c0b40c0 2024-03-06T07:27:42.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf70c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf6be00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf6c0ae670 0x7fdf6c15e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf6be00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf6c0ae670 0x7fdf6c15e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:46676/0 (socket says 172.21.15.97:46676) 2024-03-06T07:27:42.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.716+0000 7fdf6be00640 1 -- 172.21.15.97:0/2887889994 learned_addr learned my addr 172.21.15.97:0/2887889994 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:42.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.717+0000 7fdf6be00640 1 -- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 msgr2=0x7fdf6c0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.717+0000 7fdf6be00640 1 --2- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.717+0000 7fdf6be00640 1 -- 172.21.15.97:0/2887889994 >> v1:172.21.15.135:6789/0 conn(0x7fdf6c1620c0 legacy=0x7fdf6c0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.717+0000 7fdf6be00640 1 -- 172.21.15.97:0/2887889994 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c15eeb0 con 0x7fdf6c0ae670 2024-03-06T07:27:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.718+0000 7fdf6be00640 1 --2- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c15e2c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009870 tx=0x7fdf5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b95c3721c4f7b24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf6b400640 1 -- 172.21.15.97:0/2887889994 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58034070 con 0x7fdf6c0ae670 2024-03-06T07:27:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf6b400640 1 -- 172.21.15.97:0/2887889994 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf58030840 con 0x7fdf6c0ae670 2024-03-06T07:27:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf6b400640 1 -- 172.21.15.97:0/2887889994 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58030c10 con 0x7fdf6c0ae670 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf72000640 1 -- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 msgr2=0x7fdf6c15e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf72000640 1 --2- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c15e2c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009870 tx=0x7fdf5802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf72000640 1 -- 172.21.15.97:0/2887889994 shutdown_connections 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf72000640 1 --2- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c15e2c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf72000640 1 --2- 172.21.15.97:0/2887889994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.719+0000 7fdf72000640 1 -- 172.21.15.97:0/2887889994 >> 172.21.15.97:0/2887889994 conn(0x7fdf6c06afc0 msgr2=0x7fdf6c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.720+0000 7fdf72000640 1 -- 172.21.15.97:0/2887889994 shutdown_connections 2024-03-06T07:27:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.720+0000 7fdf72000640 1 -- 172.21.15.97:0/2887889994 wait complete. 2024-03-06T07:27:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.721+0000 7fdf72000640 1 Processor -- start 2024-03-06T07:27:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.721+0000 7fdf72000640 1 -- start start 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.722+0000 7fdf72000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.722+0000 7fdf72000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.722+0000 7fdf72000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 0x7fdf6c15b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.722+0000 7fdf72000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c15e800 con 0x7fdf6c1620c0 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.722+0000 7fdf72000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c161ba0 con 0x7fdf6c0ae670 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.722+0000 7fdf72000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c15e980 con 0x7fdf6c0b40c0 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 0x7fdf6c15b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 0x7fdf6c15b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32904/0 (socket says 172.21.15.97:32904) 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 -- 172.21.15.97:0/4013586666 learned_addr learned my addr 172.21.15.97:0/4013586666 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:42.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf70c00640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf6be00640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 -- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 msgr2=0x7fdf6c159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 -- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 msgr2=0x7fdf6c158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 -- 172.21.15.97:0/4013586666 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c159640 con 0x7fdf6c1620c0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.723+0000 7fdf71600640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 0x7fdf6c15b200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400c7a0 tx=0x7fdf6400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.724+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64003b70 con 0x7fdf6c1620c0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.724+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64003d30 con 0x7fdf6c1620c0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.724+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400aa30 con 0x7fdf6c1620c0 2024-03-06T07:27:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.724+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c158d60 con 0x7fdf6c1620c0 2024-03-06T07:27:42.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.724+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c1f4300 con 0x7fdf6c1620c0 2024-03-06T07:27:42.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.725+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c04f330 con 0x7fdf6c1620c0 2024-03-06T07:27:42.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.726+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf6400abd0 con 0x7fdf6c1620c0 2024-03-06T07:27:42.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.727+0000 7fdf68c00640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.727+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf6408a130 con 0x7fdf6c1620c0 2024-03-06T07:27:42.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.727+0000 7fdf70c00640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.727+0000 7fdf70c00640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009a30 tx=0x7fdf60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.731+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf64058fa0 con 0x7fdf6c1620c0 2024-03-06T07:27:42.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.920+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf6c0690e0 con 0x7fdf6c1620c0 2024-03-06T07:27:42.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.921+0000 7fdf68c00640 1 -- 172.21.15.97:0/4013586666 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16606 (secure 0 0 0) 0x7fdf640588a0 con 0x7fdf6c1620c0 2024-03-06T07:27:42.923 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:42.924 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":21,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:41.541049+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf4c06b4d0 msgr2=0x7fdf4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009a30 tx=0x7fdf60008040 comp rx=0 tx=0).stop 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 msgr2=0x7fdf6c15b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 0x7fdf6c15b200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400c7a0 tx=0x7fdf6400cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 shutdown_connections 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf6c1620c0 0x7fdf6c15b200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf6c0b40c0 0x7fdf6c159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 --2- 172.21.15.97:0/4013586666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c0ae670 0x7fdf6c158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 >> 172.21.15.97:0/4013586666 conn(0x7fdf6c06afc0 msgr2=0x7fdf6c155390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 shutdown_connections 2024-03-06T07:27:42.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:42.924+0000 7fdf72000640 1 -- 172.21.15.97:0/4013586666 wait complete. 2024-03-06T07:27:42.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:43.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.145+0000 7f6139a00640 1 Processor -- start 2024-03-06T07:27:43.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.145+0000 7f6139a00640 1 -- start start 2024-03-06T07:27:43.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.146+0000 7f6139a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 0x7f61340d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.146+0000 7f6139a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f61340d1db0 0x7f61340d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.146+0000 7f6139a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61340d0ff0 con 0x7f61340b4b90 2024-03-06T07:27:43.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.146+0000 7f6139a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61340cf2f0 con 0x7f61340d1db0 2024-03-06T07:27:43.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.146+0000 7f6139a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61340cf5d0 con 0x7f61340d06d0 2024-03-06T07:27:43.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.147+0000 7f6133e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 0x7f61340d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.147+0000 7f6133e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 0x7f61340d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:35522/0 (socket says 172.21.15.97:35522) 2024-03-06T07:27:43.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.147+0000 7f6133e00640 1 -- 172.21.15.97:0/3335463252 learned_addr learned my addr 172.21.15.97:0/3335463252 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:43.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.147+0000 7f6139000640 1 --2- 172.21.15.97:0/3335463252 >> v2:172.21.15.135:3300/0 conn(0x7f61340d1db0 0x7f61340d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.148+0000 7f6133e00640 1 -- 172.21.15.97:0/3335463252 >> v2:172.21.15.135:3300/0 conn(0x7f61340d1db0 msgr2=0x7f61340d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:43.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.148+0000 7f6133e00640 1 --2- 172.21.15.97:0/3335463252 >> v2:172.21.15.135:3300/0 conn(0x7f61340d1db0 0x7f61340d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.148+0000 7f6133e00640 1 -- 172.21.15.97:0/3335463252 >> v1:172.21.15.135:6789/0 conn(0x7f61340b4b90 legacy=0x7f61340cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.148+0000 7f6133e00640 1 -- 172.21.15.97:0/3335463252 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61340cf8b0 con 0x7f61340d06d0 2024-03-06T07:27:43.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.148+0000 7f6132a00640 1 -- 172.21.15.97:0/3335463252 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417261645 0 0) 0x7f61340d0ff0 con 0x7f61340b4b90 2024-03-06T07:27:43.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.149+0000 7f6133e00640 1 --2- 172.21.15.97:0/3335463252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 0x7f61340d0ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6124009870 tx=0x7f612402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=430c72d7405e8285 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.149+0000 7f6132a00640 1 -- 172.21.15.97:0/3335463252 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124034070 con 0x7f61340d06d0 2024-03-06T07:27:43.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6132a00640 1 -- 172.21.15.97:0/3335463252 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124030840 con 0x7f61340d06d0 2024-03-06T07:27:43.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6132a00640 1 -- 172.21.15.97:0/3335463252 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124030c10 con 0x7f61340d06d0 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 -- 172.21.15.97:0/3335463252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 msgr2=0x7f61340d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 --2- 172.21.15.97:0/3335463252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 0x7f61340d0ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6124009870 tx=0x7f612402b980 comp rx=0 tx=0).stop 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 -- 172.21.15.97:0/3335463252 shutdown_connections 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 --2- 172.21.15.97:0/3335463252 >> v2:172.21.15.135:3300/0 conn(0x7f61340d1db0 0x7f61340d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 --2- 172.21.15.97:0/3335463252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d06d0 0x7f61340d0ab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 -- 172.21.15.97:0/3335463252 >> 172.21.15.97:0/3335463252 conn(0x7f613409d070 msgr2=0x7f61340d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 -- 172.21.15.97:0/3335463252 shutdown_connections 2024-03-06T07:27:43.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.150+0000 7f6139a00640 1 -- 172.21.15.97:0/3335463252 wait complete. 2024-03-06T07:27:43.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.152+0000 7f6139a00640 1 Processor -- start 2024-03-06T07:27:43.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.152+0000 7f6139a00640 1 -- start start 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6139a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 0x7f613414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6139a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61340d06d0 0x7f61340cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6139a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d1db0 0x7f613415f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6139a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61340d2780 con 0x7f61340b4b90 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6139a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61340b0960 con 0x7f61340d06d0 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6139a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61340d2900 con 0x7f61340d1db0 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6133e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 0x7f613414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6133e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 0x7f613414e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32908/0 (socket says 172.21.15.97:32908) 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.153+0000 7f6133e00640 1 -- 172.21.15.97:0/571963925 learned_addr learned my addr 172.21.15.97:0/571963925 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133400640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61340d06d0 0x7f61340cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6139000640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d1db0 0x7f613415f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133e00640 1 -- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d1db0 msgr2=0x7f613415f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133e00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d1db0 0x7f613415f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133e00640 1 -- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61340d06d0 msgr2=0x7f61340cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133e00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61340d06d0 0x7f61340cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133e00640 1 -- 172.21.15.97:0/571963925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61340cc5f0 con 0x7f61340b4b90 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6139000640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d1db0 0x7f613415f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6133e00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 0x7f613414e2b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f61240090b0 tx=0x7f61240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124034040 con 0x7f61340b4b90 2024-03-06T07:27:43.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124030640 con 0x7f61340b4b90 2024-03-06T07:27:43.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124039d40 con 0x7f61340b4b90 2024-03-06T07:27:43.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.154+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61340cc8d0 con 0x7f61340b4b90 2024-03-06T07:27:43.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.155+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61340cce90 con 0x7f61340b4b90 2024-03-06T07:27:43.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.156+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613404f330 con 0x7f61340b4b90 2024-03-06T07:27:43.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.156+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6124037070 con 0x7f61340b4b90 2024-03-06T07:27:43.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.157+0000 7f612be00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f611406b4d0 0x7f611406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.157+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61240abe60 con 0x7f61340b4b90 2024-03-06T07:27:43.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.157+0000 7f6133400640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f611406b4d0 0x7f611406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.158+0000 7f6133400640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f611406b4d0 0x7f611406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61340744c0 tx=0x7f6120009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.162+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f612407ad70 con 0x7f61340b4b90 2024-03-06T07:27:43.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:43.411+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f613408d4d0 con 0x7f61340b4b90 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.088+0000 7f612be00640 1 -- 172.21.15.97:0/571963925 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f612407a670 con 0x7f61340b4b90 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.091+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f611406b4d0 msgr2=0x7f611406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f611406b4d0 0x7f611406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61340744c0 tx=0x7f6120009040 comp rx=0 tx=0).stop 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 msgr2=0x7f613414e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 0x7f613414e2b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f61240090b0 tx=0x7f61240046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 shutdown_connections 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f611406b4d0 0x7f611406d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61340d1db0 0x7f613415f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61340d06d0 0x7f61340cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 --2- 172.21.15.97:0/571963925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61340b4b90 0x7f613414e2b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 >> 172.21.15.97:0/571963925 conn(0x7f613409d070 msgr2=0x7f61340b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 shutdown_connections 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.092+0000 7f6139a00640 1 -- 172.21.15.97:0/571963925 wait complete. 2024-03-06T07:27:44.114 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:44.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:44.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.321+0000 7f6844e00640 1 Processor -- start 2024-03-06T07:27:44.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.321+0000 7f6844e00640 1 -- start start 2024-03-06T07:27:44.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f6844e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f684015f490 0x7f68400d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f6844e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68400d4c40 con 0x7f68400ced50 2024-03-06T07:27:44.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f6844e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68400d2700 con 0x7f68400b4de0 2024-03-06T07:27:44.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f6844e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68400d49f0 con 0x7f684015f490 2024-03-06T07:27:44.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f683de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f684015f490 0x7f68400d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f683de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f684015f490 0x7f68400d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50308/0 (socket says 172.21.15.97:50308) 2024-03-06T07:27:44.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.323+0000 7f683de00640 1 -- 172.21.15.97:0/1208361955 learned_addr learned my addr 172.21.15.97:0/1208361955 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:44.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.324+0000 7f683de00640 1 -- 172.21.15.97:0/1208361955 >> v1:172.21.15.135:6789/0 conn(0x7f68400b4de0 legacy=0x7f68400d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.324+0000 7f683de00640 1 -- 172.21.15.97:0/1208361955 >> v1:172.21.15.97:6789/0 conn(0x7f68400ced50 legacy=0x7f68400cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.324+0000 7f683de00640 1 -- 172.21.15.97:0/1208361955 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68400d29e0 con 0x7f684015f490 2024-03-06T07:27:44.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.324+0000 7f683d400640 1 -- 172.21.15.97:0/1208361955 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396961561 0 0) 0x7f68400d4c40 con 0x7f68400ced50 2024-03-06T07:27:44.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.325+0000 7f683de00640 1 --2- 172.21.15.97:0/1208361955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400d08a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6830009870 tx=0x7f683002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e8749796dbefc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.325+0000 7f683d400640 1 -- 172.21.15.97:0/1208361955 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830035020 con 0x7f684015f490 2024-03-06T07:27:44.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.325+0000 7f683d400640 1 -- 172.21.15.97:0/1208361955 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6830031870 con 0x7f684015f490 2024-03-06T07:27:44.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.325+0000 7f683d400640 1 -- 172.21.15.97:0/1208361955 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830031c40 con 0x7f684015f490 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 -- 172.21.15.97:0/1208361955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 msgr2=0x7f68400d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 --2- 172.21.15.97:0/1208361955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400d08a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6830009870 tx=0x7f683002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 -- 172.21.15.97:0/1208361955 shutdown_connections 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 --2- 172.21.15.97:0/1208361955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400d08a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 -- 172.21.15.97:0/1208361955 >> 172.21.15.97:0/1208361955 conn(0x7f68400d3ed0 msgr2=0x7f68400d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 -- 172.21.15.97:0/1208361955 shutdown_connections 2024-03-06T07:27:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.326+0000 7f6844e00640 1 -- 172.21.15.97:0/1208361955 wait complete. 2024-03-06T07:27:44.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.329+0000 7f6844e00640 1 Processor -- start 2024-03-06T07:27:44.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.329+0000 7f6844e00640 1 -- start start 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.330+0000 7f6844e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68400b4de0 0x7f68401596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.330+0000 7f6844e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68400ced50 0x7f684015c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.330+0000 7f6844e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.330+0000 7f6844e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684015fd50 con 0x7f684015f490 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.330+0000 7f6844e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400d24b0 con 0x7f68400b4de0 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.330+0000 7f6844e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684015f870 con 0x7f68400ced50 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50314/0 (socket says 172.21.15.97:50314) 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 -- 172.21.15.97:0/497455075 learned_addr learned my addr 172.21.15.97:0/497455075 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683de00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68400b4de0 0x7f68401596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 -- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68400ced50 msgr2=0x7f684015c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f6835400640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68400ced50 0x7f684015c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68400ced50 0x7f684015c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 -- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68400b4de0 msgr2=0x7f68401596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68400b4de0 0x7f68401596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 -- 172.21.15.97:0/497455075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68400cc120 con 0x7f684015f490 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.331+0000 7f683e800640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400cd1c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f683800c790 tx=0x7f683800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.332+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838003da0 con 0x7f684015f490 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.332+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6838005070 con 0x7f684015f490 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.332+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838005460 con 0x7f684015f490 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.332+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68400cc400 con 0x7f684015f490 2024-03-06T07:27:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.332+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68400cc9c0 con 0x7f684015f490 2024-03-06T07:27:44.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.333+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684004f330 con 0x7f684015f490 2024-03-06T07:27:44.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.334+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6838005600 con 0x7f684015f490 2024-03-06T07:27:44.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.335+0000 7f6836a00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f681c06b4d0 0x7f681c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.335+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f683808a0d0 con 0x7f684015f490 2024-03-06T07:27:44.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.335+0000 7f683de00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f681c06b4d0 0x7f681c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.336+0000 7f683de00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f681c06b4d0 0x7f681c06d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6830036000 tx=0x7f6830008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.339+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6838058fe0 con 0x7f684015f490 2024-03-06T07:27:44.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:44.577+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f684008d950 con 0x7f684015f490 2024-03-06T07:27:45.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.098+0000 7f6836a00640 1 -- 172.21.15.97:0/497455075 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f68380588e0 con 0x7f684015f490 2024-03-06T07:27:45.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f681c06b4d0 msgr2=0x7f681c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f681c06b4d0 0x7f681c06d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6830036000 tx=0x7f6830008030 comp rx=0 tx=0).stop 2024-03-06T07:27:45.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 msgr2=0x7f68400cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400cd1c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f683800c790 tx=0x7f683800cc50 comp rx=0 tx=0).stop 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 shutdown_connections 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f681c06b4d0 0x7f681c06d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684015f490 0x7f68400cd1c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68400ced50 0x7f684015c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 --2- 172.21.15.97:0/497455075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68400b4de0 0x7f68401596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.101+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 >> 172.21.15.97:0/497455075 conn(0x7f68400d3ed0 msgr2=0x7f684006ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.102+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 shutdown_connections 2024-03-06T07:27:45.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.102+0000 7f6844e00640 1 -- 172.21.15.97:0/497455075 wait complete. 2024-03-06T07:27:45.122 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:45.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:45.190 INFO:tasks.ceph.mds.c.smithi097.stderr:2024-03-06T07:27:45.188+0000 7f509ac00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:45.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.317+0000 7f64d0c00640 1 Processor -- start 2024-03-06T07:27:45.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.317+0000 7f64d0c00640 1 -- start start 2024-03-06T07:27:45.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64d0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 0x7f64cc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64d0c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f64cc06b500 0x7f64cc06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64d0c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc0b5010 con 0x7f64cc15e070 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64d0c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc06b970 con 0x7f64cc06b500 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64d0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0b52f0 con 0x7f64cc0cd860 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64c9400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 0x7f64cc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.319+0000 7f64c9400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 0x7f64cc06afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:32780/0 (socket says 172.21.15.97:32780) 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64c9400640 1 -- 172.21.15.97:0/151926459 learned_addr learned my addr 172.21.15.97:0/151926459 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:45.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64ca800640 1 --2- 172.21.15.97:0/151926459 >> v2:172.21.15.135:3300/0 conn(0x7f64cc06b500 0x7f64cc06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64c8a00640 1 -- 172.21.15.97:0/151926459 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333488185 0 0) 0x7f64cc0b5010 con 0x7f64cc15e070 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64c8a00640 1 -- 172.21.15.97:0/151926459 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b0003520 con 0x7f64cc15e070 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64ca800640 1 -- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 msgr2=0x7f64cc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64ca800640 1 --2- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 0x7f64cc06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64ca800640 1 -- 172.21.15.97:0/151926459 >> v1:172.21.15.135:6789/0 conn(0x7f64cc15e070 legacy=0x7f64cc066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64ca800640 1 -- 172.21.15.97:0/151926459 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc0b55d0 con 0x7f64cc06b500 2024-03-06T07:27:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.320+0000 7f64c9400640 1 --2- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 0x7f64cc06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:45.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.322+0000 7f64ca800640 1 --2- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc06b500 0x7f64cc06db00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097a0 tx=0x7f64c402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7cd600ed9888d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.322+0000 7f64c8a00640 1 -- 172.21.15.97:0/151926459 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4035020 con 0x7f64cc06b500 2024-03-06T07:27:45.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.322+0000 7f64c8a00640 1 -- 172.21.15.97:0/151926459 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4031810 con 0x7f64cc06b500 2024-03-06T07:27:45.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.322+0000 7f64c8a00640 1 -- 172.21.15.97:0/151926459 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4031be0 con 0x7f64cc06b500 2024-03-06T07:27:45.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 -- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc06b500 msgr2=0x7f64cc06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 --2- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc06b500 0x7f64cc06db00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097a0 tx=0x7f64c402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 -- 172.21.15.97:0/151926459 shutdown_connections 2024-03-06T07:27:45.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 --2- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc0cd860 0x7f64cc06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 --2- 172.21.15.97:0/151926459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc06b500 0x7f64cc06db00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 -- 172.21.15.97:0/151926459 >> 172.21.15.97:0/151926459 conn(0x7f64cc0b1e10 msgr2=0x7f64cc0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 -- 172.21.15.97:0/151926459 shutdown_connections 2024-03-06T07:27:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.323+0000 7f64d0c00640 1 -- 172.21.15.97:0/151926459 wait complete. 2024-03-06T07:27:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.325+0000 7f64d0c00640 1 Processor -- start 2024-03-06T07:27:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.325+0000 7f64d0c00640 1 -- start start 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.326+0000 7f64d0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc06b500 0x7f64cc1f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.326+0000 7f64d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 0x7f64cc1e7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.326+0000 7f64d0c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc15e070 0x7f64cc1e7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.326+0000 7f64d0c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15e450 con 0x7f64cc0cd860 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.326+0000 7f64d0c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0b5b90 con 0x7f64cc15e070 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.326+0000 7f64d0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0b60b0 con 0x7f64cc06b500 2024-03-06T07:27:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 0x7f64cc1e7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 0x7f64cc1e7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50324/0 (socket says 172.21.15.97:50324) 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 -- 172.21.15.97:0/3696372943 learned_addr learned my addr 172.21.15.97:0/3696372943 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9e00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc06b500 0x7f64cc1f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64ca800640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc15e070 0x7f64cc1e7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 -- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc06b500 msgr2=0x7f64cc1f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc06b500 0x7f64cc1f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 -- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc15e070 msgr2=0x7f64cc1e7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc15e070 0x7f64cc1e7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 -- 172.21.15.97:0/3696372943 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc151cb0 con 0x7f64cc0cd860 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.327+0000 7f64c9400640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 0x7f64cc1e7a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64b400e9b0 tx=0x7f64b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.328+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4017070 con 0x7f64cc0cd860 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.328+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b400c9a0 con 0x7f64cc0cd860 2024-03-06T07:27:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.328+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b400cd50 con 0x7f64cc0cd860 2024-03-06T07:27:45.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.328+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc150ef0 con 0x7f64cc0cd860 2024-03-06T07:27:45.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.328+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc1514b0 con 0x7f64cc0cd860 2024-03-06T07:27:45.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.329+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc04f330 con 0x7f64cc0cd860 2024-03-06T07:27:45.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.330+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64b401a070 con 0x7f64cc0cd860 2024-03-06T07:27:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.331+0000 7f64c2000640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64a806b520 0x7f64a806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.331+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64b408a0d0 con 0x7f64cc0cd860 2024-03-06T07:27:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.331+0000 7f64c9e00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64a806b520 0x7f64a806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.332+0000 7f64c9e00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64a806b520 0x7f64a806d9e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64bc001cb0 tx=0x7f64bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.336+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64b4058f30 con 0x7f64cc0cd860 2024-03-06T07:27:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.568+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64cc0b2ae0 con 0x7f64cc0cd860 2024-03-06T07:27:45.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.569+0000 7f64c2000640 1 -- 172.21.15.97:0/3696372943 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f64b4058830 con 0x7f64cc0cd860 2024-03-06T07:27:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:45.572 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:27:45.572 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64a806b520 msgr2=0x7f64a806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64a806b520 0x7f64a806d9e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64bc001cb0 tx=0x7f64bc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 msgr2=0x7f64cc1e7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 0x7f64cc1e7a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64b400e9b0 tx=0x7f64b400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 shutdown_connections 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64a806b520 0x7f64a806d9e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64cc15e070 0x7f64cc1e7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64cc0cd860 0x7f64cc1e7a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 --2- 172.21.15.97:0/3696372943 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64cc06b500 0x7f64cc1f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 >> 172.21.15.97:0/3696372943 conn(0x7f64cc0b1e10 msgr2=0x7f64cc0ae230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.572+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 shutdown_connections 2024-03-06T07:27:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.573+0000 7f64d0c00640 1 -- 172.21.15.97:0/3696372943 wait complete. 2024-03-06T07:27:45.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.787+0000 7f25d0400640 1 Processor -- start 2024-03-06T07:27:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.787+0000 7f25d0400640 1 -- start start 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.788+0000 7f25d0400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25c815acb0 0x7f25c80b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25d0400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c815b090 con 0x7f25c80b4680 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25d0400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c80b2a30 con 0x7f25c806ec00 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25d0400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c80b4360 con 0x7f25c815acb0 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cda00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25c815acb0 0x7f25c80b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cd000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f25c80b4680 0x7f25c806e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45046/0 (socket says 172.21.15.97:45046) 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cda00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25c815acb0 0x7f25c80b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50336/0 (socket says 172.21.15.97:50336) 2024-03-06T07:27:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cd000640 1 -- 172.21.15.97:0/3253411355 learned_addr learned my addr 172.21.15.97:0/3253411355 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cda00640 1 -- 172.21.15.97:0/3253411355 >> v1:172.21.15.135:6789/0 conn(0x7f25c806ec00 legacy=0x7f25c80ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cda00640 1 -- 172.21.15.97:0/3253411355 >> v1:172.21.15.97:6789/0 conn(0x7f25c80b4680 legacy=0x7f25c806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.789+0000 7f25cda00640 1 -- 172.21.15.97:0/3253411355 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c80b2cf0 con 0x7f25c815acb0 2024-03-06T07:27:45.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.790+0000 7f25cda00640 1 --2- 172.21.15.97:0/3253411355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c815acb0 0x7f25c80b37d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0097a0 tx=0x7f25bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbeeacb8157670e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25c7e00640 1 -- 172.21.15.97:0/3253411355 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc035020 con 0x7f25c815acb0 2024-03-06T07:27:45.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25c7e00640 1 -- 172.21.15.97:0/3253411355 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc031830 con 0x7f25c815acb0 2024-03-06T07:27:45.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25c7e00640 1 -- 172.21.15.97:0/3253411355 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc031c00 con 0x7f25c815acb0 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25d0400640 1 -- 172.21.15.97:0/3253411355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c815acb0 msgr2=0x7f25c80b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25d0400640 1 --2- 172.21.15.97:0/3253411355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c815acb0 0x7f25c80b37d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0097a0 tx=0x7f25bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25d0400640 1 -- 172.21.15.97:0/3253411355 shutdown_connections 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25d0400640 1 --2- 172.21.15.97:0/3253411355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c815acb0 0x7f25c80b37d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25d0400640 1 -- 172.21.15.97:0/3253411355 >> 172.21.15.97:0/3253411355 conn(0x7f25c80b1730 msgr2=0x7f25c80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.791+0000 7f25d0400640 1 -- 172.21.15.97:0/3253411355 shutdown_connections 2024-03-06T07:27:45.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.792+0000 7f25d0400640 1 -- 172.21.15.97:0/3253411355 wait complete. 2024-03-06T07:27:45.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.793+0000 7f25d0400640 1 Processor -- start 2024-03-06T07:27:45.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.793+0000 7f25d0400640 1 -- start start 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25d0400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 0x7f25c813f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25d0400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25c80b4680 0x7f25c8150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25d0400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25c815acb0 0x7f25c8157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25d0400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80b3d10 con 0x7f25c806ec00 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25d0400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80b32b0 con 0x7f25c815acb0 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25d0400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80b4010 con 0x7f25c80b4680 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.794+0000 7f25cda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 0x7f25c813f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 0x7f25c813f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50344/0 (socket says 172.21.15.97:50344) 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 -- 172.21.15.97:0/3376120756 learned_addr learned my addr 172.21.15.97:0/3376120756 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:45.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cd000640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25c80b4680 0x7f25c8150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 -- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25c80b4680 msgr2=0x7f25c8150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25c80b4680 0x7f25c8150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 -- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25c815acb0 msgr2=0x7f25c8157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25ce400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25c815acb0 0x7f25c8157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25c815acb0 0x7f25c8157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 -- 172.21.15.97:0/3376120756 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c813f850 con 0x7f25c806ec00 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cd000640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25c80b4680 0x7f25c8150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.795+0000 7f25cda00640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 0x7f25c813f310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0098c0 tx=0x7f25bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.796+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc041070 con 0x7f25c806ec00 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.796+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc03aa30 con 0x7f25c806ec00 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.796+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c8151400 con 0x7f25c806ec00 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.796+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc035040 con 0x7f25c806ec00 2024-03-06T07:27:45.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.796+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c81519c0 con 0x7f25c806ec00 2024-03-06T07:27:45.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.797+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c804f330 con 0x7f25c806ec00 2024-03-06T07:27:45.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.798+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25bc04c060 con 0x7f25c806ec00 2024-03-06T07:27:45.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.798+0000 7f25c5600640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25a806b4d0 0x7f25a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.799+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25bc0abc20 con 0x7f25c806ec00 2024-03-06T07:27:45.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.799+0000 7f25cd000640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25a806b4d0 0x7f25a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.800+0000 7f25cd000640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25a806b4d0 0x7f25a806d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25c8155440 tx=0x7f25b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.803+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25bc07ab30 con 0x7f25c806ec00 2024-03-06T07:27:46.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:45.999+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25c80b1d60 con 0x7f25c806ec00 2024-03-06T07:27:46.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.001+0000 7f25c5600640 1 -- 172.21.15.97:0/3376120756 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f25bc07a430 con 0x7f25c806ec00 2024-03-06T07:27:46.004 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:46.004 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:46.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25a806b4d0 msgr2=0x7f25a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25a806b4d0 0x7f25a806d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25c8155440 tx=0x7f25b4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 msgr2=0x7f25c813f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 0x7f25c813f310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0098c0 tx=0x7f25bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 shutdown_connections 2024-03-06T07:27:46.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25a806b4d0 0x7f25a806d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25c815acb0 0x7f25c8157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25c80b4680 0x7f25c8150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 --2- 172.21.15.97:0/3376120756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25c806ec00 0x7f25c813f310 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 >> 172.21.15.97:0/3376120756 conn(0x7f25c80b1730 msgr2=0x7f25c815b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 shutdown_connections 2024-03-06T07:27:46.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.004+0000 7f25d0400640 1 -- 172.21.15.97:0/3376120756 wait complete. 2024-03-06T07:27:46.026 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:46.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:46.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.233+0000 7f88a2400640 1 Processor -- start 2024-03-06T07:27:46.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.233+0000 7f88a2400640 1 -- start start 2024-03-06T07:27:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.234+0000 7f88a2400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f889c0d07b0 0x7f889c0cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.234+0000 7f88a2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c0d20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.234+0000 7f88a2400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c0cf140 con 0x7f889c15f110 2024-03-06T07:27:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.234+0000 7f88a2400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c0cfd50 con 0x7f889c0d07b0 2024-03-06T07:27:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.234+0000 7f88a2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0cf420 con 0x7f889c0d1c20 2024-03-06T07:27:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.235+0000 7f889be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c0d20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.235+0000 7f889be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c0d20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:32808/0 (socket says 172.21.15.97:32808) 2024-03-06T07:27:46.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.235+0000 7f889be00640 1 -- 172.21.15.97:0/3308453799 learned_addr learned my addr 172.21.15.97:0/3308453799 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:46.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.235+0000 7f889b400640 1 --2- 172.21.15.97:0/3308453799 >> v2:172.21.15.135:3300/0 conn(0x7f889c0d07b0 0x7f889c0cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889a000640 1 -- 172.21.15.97:0/3308453799 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976105033 0 0) 0x7f889c0cf140 con 0x7f889c15f110 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889a000640 1 -- 172.21.15.97:0/3308453799 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880003520 con 0x7f889c15f110 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889be00640 1 -- 172.21.15.97:0/3308453799 >> v2:172.21.15.135:3300/0 conn(0x7f889c0d07b0 msgr2=0x7f889c0cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889be00640 1 --2- 172.21.15.97:0/3308453799 >> v2:172.21.15.135:3300/0 conn(0x7f889c0d07b0 0x7f889c0cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889be00640 1 -- 172.21.15.97:0/3308453799 >> v1:172.21.15.135:6789/0 conn(0x7f889c15f110 legacy=0x7f889c0d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889be00640 1 -- 172.21.15.97:0/3308453799 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0cf700 con 0x7f889c0d1c20 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.236+0000 7f889b400640 1 --2- 172.21.15.97:0/3308453799 >> v2:172.21.15.135:3300/0 conn(0x7f889c0d07b0 0x7f889c0cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:46.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.238+0000 7f889be00640 1 --2- 172.21.15.97:0/3308453799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c0d20b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8894009e50 tx=0x7f889402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6996d23db32ff2d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.238+0000 7f889a000640 1 -- 172.21.15.97:0/3308453799 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889402cb00 con 0x7f889c0d1c20 2024-03-06T07:27:46.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.238+0000 7f889a000640 1 -- 172.21.15.97:0/3308453799 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889402cca0 con 0x7f889c0d1c20 2024-03-06T07:27:46.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.238+0000 7f889a000640 1 -- 172.21.15.97:0/3308453799 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894034610 con 0x7f889c0d1c20 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 -- 172.21.15.97:0/3308453799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 msgr2=0x7f889c0d20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 --2- 172.21.15.97:0/3308453799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c0d20b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8894009e50 tx=0x7f889402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 -- 172.21.15.97:0/3308453799 shutdown_connections 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 --2- 172.21.15.97:0/3308453799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c0d20b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 --2- 172.21.15.97:0/3308453799 >> v2:172.21.15.135:3300/0 conn(0x7f889c0d07b0 0x7f889c0cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 -- 172.21.15.97:0/3308453799 >> 172.21.15.97:0/3308453799 conn(0x7f889c0d3ee0 msgr2=0x7f889c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 -- 172.21.15.97:0/3308453799 shutdown_connections 2024-03-06T07:27:46.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.239+0000 7f88a2400640 1 -- 172.21.15.97:0/3308453799 wait complete. 2024-03-06T07:27:46.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.241+0000 7f88a2400640 1 Processor -- start 2024-03-06T07:27:46.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.241+0000 7f88a2400640 1 -- start start 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f88a2400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f889c0d07b0 0x7f889c14e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f88a2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c1f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f88a2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 0x7f889c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f88a2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c15e390 con 0x7f889c15f110 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f88a2400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c15f570 con 0x7f889c0d07b0 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f88a2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c15e510 con 0x7f889c0d1c20 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f889be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 0x7f889c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f889be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 0x7f889c159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50346/0 (socket says 172.21.15.97:50346) 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f889be00640 1 -- 172.21.15.97:0/3208256250 learned_addr learned my addr 172.21.15.97:0/3208256250 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:46.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f889aa00640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c1f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.242+0000 7f889b400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f889c0d07b0 0x7f889c14e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889be00640 1 -- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 msgr2=0x7f889c1f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889be00640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c1f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889be00640 1 -- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f889c0d07b0 msgr2=0x7f889c14e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889be00640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f889c0d07b0 0x7f889c14e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889be00640 1 -- 172.21.15.97:0/3208256250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c1f87d0 con 0x7f889c15f110 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889aa00640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c1f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:46.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f889be00640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 0x7f889c159eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8894009e50 tx=0x7f8894004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894043020 con 0x7f889c15f110 2024-03-06T07:27:46.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889402ce50 con 0x7f889c15f110 2024-03-06T07:27:46.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c15a3f0 con 0x7f889c15f110 2024-03-06T07:27:46.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.243+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894009510 con 0x7f889c15f110 2024-03-06T07:27:46.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.244+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0cb4e0 con 0x7f889c15f110 2024-03-06T07:27:46.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.245+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c04f330 con 0x7f889c15f110 2024-03-06T07:27:46.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.246+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8894046020 con 0x7f889c15f110 2024-03-06T07:27:46.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.247+0000 7f8893400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f887c06b4d0 0x7f887c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.247+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88940ac650 con 0x7f889c15f110 2024-03-06T07:27:46.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.247+0000 7f889b400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f887c06b4d0 0x7f887c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.248+0000 7f889b400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f887c06b4d0 0x7f887c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f888c009560 tx=0x7f888c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.252+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f889404d030 con 0x7f889c15f110 2024-03-06T07:27:46.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.443+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f889c0cafb0 con 0x7f887c06b4d0 2024-03-06T07:27:46.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.460+0000 7f8893400640 1 -- 172.21.15.97:0/3208256250 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251949 (secure 0 0 0) 0x7f889c0cafb0 con 0x7f887c06b4d0 2024-03-06T07:27:46.465 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:46.467 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:27:46.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f887c06b4d0 msgr2=0x7f887c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f887c06b4d0 0x7f887c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f888c009560 tx=0x7f888c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 msgr2=0x7f889c159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 0x7f889c159eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8894009e50 tx=0x7f8894004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:46.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 shutdown_connections 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f887c06b4d0 0x7f887c06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f889c15f110 0x7f889c159eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f889c0d1c20 0x7f889c1f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 --2- 172.21.15.97:0/3208256250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f889c0d07b0 0x7f889c14e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 >> 172.21.15.97:0/3208256250 conn(0x7f889c0d3ee0 msgr2=0x7f889c1593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 shutdown_connections 2024-03-06T07:27:46.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.466+0000 7f88a2400640 1 -- 172.21.15.97:0/3208256250 wait complete. 2024-03-06T07:27:46.488 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:27:44.903730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219460,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755260,"statfs":{"total":773094113280,"available":772869386240,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.851558"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669901+0000","last_change":"2024-03-06T07:27:42.115947+0000","last_active":"2024-03-06T07:27:43.669901+0000","last_peered":"2024-03-06T07:27:43.669901+0000","last_clean":"2024-03-06T07:27:43.669901+0000","last_became_active":"2024-03-06T07:27:42.115323+0000","last_became_peered":"2024-03-06T07:27:42.115323+0000","last_unstale":"2024-03-06T07:27:43.669901+0000","last_undegraded":"2024-03-06T07:27:43.669901+0000","last_fullsized":"2024-03-06T07:27:43.669901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:23.978663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090443+0000","last_change":"2024-03-06T07:27:42.092048+0000","last_active":"2024-03-06T07:27:43.090443+0000","last_peered":"2024-03-06T07:27:43.090443+0000","last_clean":"2024-03-06T07:27:43.090443+0000","last_became_active":"2024-03-06T07:27:42.091783+0000","last_became_peered":"2024-03-06T07:27:42.091783+0000","last_unstale":"2024-03-06T07:27:43.090443+0000","last_undegraded":"2024-03-06T07:27:43.090443+0000","last_fullsized":"2024-03-06T07:27:43.090443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561478+0000","last_change":"2024-03-06T07:27:42.095834+0000","last_active":"2024-03-06T07:27:43.561478+0000","last_peered":"2024-03-06T07:27:43.561478+0000","last_clean":"2024-03-06T07:27:43.561478+0000","last_became_active":"2024-03-06T07:27:42.095026+0000","last_became_peered":"2024-03-06T07:27:42.095026+0000","last_unstale":"2024-03-06T07:27:43.561478+0000","last_undegraded":"2024-03-06T07:27:43.561478+0000","last_fullsized":"2024-03-06T07:27:43.561478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604760+0000","last_change":"2024-03-06T07:27:42.114409+0000","last_active":"2024-03-06T07:27:43.604760+0000","last_peered":"2024-03-06T07:27:43.604760+0000","last_clean":"2024-03-06T07:27:43.604760+0000","last_became_active":"2024-03-06T07:27:42.114122+0000","last_became_peered":"2024-03-06T07:27:42.114122+0000","last_unstale":"2024-03-06T07:27:43.604760+0000","last_undegraded":"2024-03-06T07:27:43.604760+0000","last_fullsized":"2024-03-06T07:27:43.604760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668591+0000","last_change":"2024-03-06T07:27:42.101691+0000","last_active":"2024-03-06T07:27:43.668591+0000","last_peered":"2024-03-06T07:27:43.668591+0000","last_clean":"2024-03-06T07:27:43.668591+0000","last_became_active":"2024-03-06T07:27:42.101372+0000","last_became_peered":"2024-03-06T07:27:42.101372+0000","last_unstale":"2024-03-06T07:27:43.668591+0000","last_undegraded":"2024-03-06T07:27:43.668591+0000","last_fullsized":"2024-03-06T07:27:43.668591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604137+0000","last_change":"2024-03-06T07:27:42.094768+0000","last_active":"2024-03-06T07:27:43.604137+0000","last_peered":"2024-03-06T07:27:43.604137+0000","last_clean":"2024-03-06T07:27:43.604137+0000","last_became_active":"2024-03-06T07:27:42.094416+0000","last_became_peered":"2024-03-06T07:27:42.094416+0000","last_unstale":"2024-03-06T07:27:43.604137+0000","last_undegraded":"2024-03-06T07:27:43.604137+0000","last_fullsized":"2024-03-06T07:27:43.604137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668352+0000","last_change":"2024-03-06T07:27:42.101468+0000","last_active":"2024-03-06T07:27:43.668352+0000","last_peered":"2024-03-06T07:27:43.668352+0000","last_clean":"2024-03-06T07:27:43.668352+0000","last_became_active":"2024-03-06T07:27:42.101168+0000","last_became_peered":"2024-03-06T07:27:42.101168+0000","last_unstale":"2024-03-06T07:27:43.668352+0000","last_undegraded":"2024-03-06T07:27:43.668352+0000","last_fullsized":"2024-03-06T07:27:43.668352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265352+0000","last_change":"2024-03-06T07:27:42.098517+0000","last_active":"2024-03-06T07:27:43.265352+0000","last_peered":"2024-03-06T07:27:43.265352+0000","last_clean":"2024-03-06T07:27:43.265352+0000","last_became_active":"2024-03-06T07:27:42.092643+0000","last_became_peered":"2024-03-06T07:27:42.092643+0000","last_unstale":"2024-03-06T07:27:43.265352+0000","last_undegraded":"2024-03-06T07:27:43.265352+0000","last_fullsized":"2024-03-06T07:27:43.265352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669864+0000","last_change":"2024-03-06T07:27:42.119208+0000","last_active":"2024-03-06T07:27:43.669864+0000","last_peered":"2024-03-06T07:27:43.669864+0000","last_clean":"2024-03-06T07:27:43.669864+0000","last_became_active":"2024-03-06T07:27:42.118050+0000","last_became_peered":"2024-03-06T07:27:42.118050+0000","last_unstale":"2024-03-06T07:27:43.669864+0000","last_undegraded":"2024-03-06T07:27:43.669864+0000","last_fullsized":"2024-03-06T07:27:43.669864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:43.473299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096548+0000","last_change":"2024-03-06T07:27:42.095950+0000","last_active":"2024-03-06T07:27:43.096548+0000","last_peered":"2024-03-06T07:27:43.096548+0000","last_clean":"2024-03-06T07:27:43.096548+0000","last_became_active":"2024-03-06T07:27:42.095462+0000","last_became_peered":"2024-03-06T07:27:42.095462+0000","last_unstale":"2024-03-06T07:27:43.096548+0000","last_undegraded":"2024-03-06T07:27:43.096548+0000","last_fullsized":"2024-03-06T07:27:43.096548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095953+0000","last_change":"2024-03-06T07:27:42.096143+0000","last_active":"2024-03-06T07:27:43.095953+0000","last_peered":"2024-03-06T07:27:43.095953+0000","last_clean":"2024-03-06T07:27:43.095953+0000","last_became_active":"2024-03-06T07:27:42.095793+0000","last_became_peered":"2024-03-06T07:27:42.095793+0000","last_unstale":"2024-03-06T07:27:43.095953+0000","last_undegraded":"2024-03-06T07:27:43.095953+0000","last_fullsized":"2024-03-06T07:27:43.095953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434202+0000","last_change":"2024-03-06T07:27:42.092238+0000","last_active":"2024-03-06T07:27:43.434202+0000","last_peered":"2024-03-06T07:27:43.434202+0000","last_clean":"2024-03-06T07:27:43.434202+0000","last_became_active":"2024-03-06T07:27:42.091930+0000","last_became_peered":"2024-03-06T07:27:42.091930+0000","last_unstale":"2024-03-06T07:27:43.434202+0000","last_undegraded":"2024-03-06T07:27:43.434202+0000","last_fullsized":"2024-03-06T07:27:43.434202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:38.321736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264768+0000","last_change":"2024-03-06T07:27:42.097468+0000","last_active":"2024-03-06T07:27:43.264768+0000","last_peered":"2024-03-06T07:27:43.264768+0000","last_clean":"2024-03-06T07:27:43.264768+0000","last_became_active":"2024-03-06T07:27:42.094036+0000","last_became_peered":"2024-03-06T07:27:42.094036+0000","last_unstale":"2024-03-06T07:27:43.264768+0000","last_undegraded":"2024-03-06T07:27:43.264768+0000","last_fullsized":"2024-03-06T07:27:43.264768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561085+0000","last_change":"2024-03-06T07:27:42.092704+0000","last_active":"2024-03-06T07:27:43.561085+0000","last_peered":"2024-03-06T07:27:43.561085+0000","last_clean":"2024-03-06T07:27:43.561085+0000","last_became_active":"2024-03-06T07:27:42.090459+0000","last_became_peered":"2024-03-06T07:27:42.090459+0000","last_unstale":"2024-03-06T07:27:43.561085+0000","last_undegraded":"2024-03-06T07:27:43.561085+0000","last_fullsized":"2024-03-06T07:27:43.561085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:26.470370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089922+0000","last_change":"2024-03-06T07:27:42.076294+0000","last_active":"2024-03-06T07:27:43.089922+0000","last_peered":"2024-03-06T07:27:43.089922+0000","last_clean":"2024-03-06T07:27:43.089922+0000","last_became_active":"2024-03-06T07:27:42.075529+0000","last_became_peered":"2024-03-06T07:27:42.075529+0000","last_unstale":"2024-03-06T07:27:43.089922+0000","last_undegraded":"2024-03-06T07:27:43.089922+0000","last_fullsized":"2024-03-06T07:27:43.089922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604410+0000","last_change":"2024-03-06T07:27:42.093846+0000","last_active":"2024-03-06T07:27:43.604410+0000","last_peered":"2024-03-06T07:27:43.604410+0000","last_clean":"2024-03-06T07:27:43.604410+0000","last_became_active":"2024-03-06T07:27:42.093544+0000","last_became_peered":"2024-03-06T07:27:42.093544+0000","last_unstale":"2024-03-06T07:27:43.604410+0000","last_undegraded":"2024-03-06T07:27:43.604410+0000","last_fullsized":"2024-03-06T07:27:43.604410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605458+0000","last_change":"2024-03-06T07:27:42.118690+0000","last_active":"2024-03-06T07:27:43.605458+0000","last_peered":"2024-03-06T07:27:43.605458+0000","last_clean":"2024-03-06T07:27:43.605458+0000","last_became_active":"2024-03-06T07:27:42.118405+0000","last_became_peered":"2024-03-06T07:27:42.118405+0000","last_unstale":"2024-03-06T07:27:43.605458+0000","last_undegraded":"2024-03-06T07:27:43.605458+0000","last_fullsized":"2024-03-06T07:27:43.605458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093672+0000","last_change":"2024-03-06T07:27:42.096245+0000","last_active":"2024-03-06T07:27:43.093672+0000","last_peered":"2024-03-06T07:27:43.093672+0000","last_clean":"2024-03-06T07:27:43.093672+0000","last_became_active":"2024-03-06T07:27:42.094600+0000","last_became_peered":"2024-03-06T07:27:42.094600+0000","last_unstale":"2024-03-06T07:27:43.093672+0000","last_undegraded":"2024-03-06T07:27:43.093672+0000","last_fullsized":"2024-03-06T07:27:43.093672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:05.681114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669317+0000","last_change":"2024-03-06T07:27:42.104820+0000","last_active":"2024-03-06T07:27:43.669317+0000","last_peered":"2024-03-06T07:27:43.669317+0000","last_clean":"2024-03-06T07:27:43.669317+0000","last_became_active":"2024-03-06T07:27:42.104489+0000","last_became_peered":"2024-03-06T07:27:42.104489+0000","last_unstale":"2024-03-06T07:27:43.669317+0000","last_undegraded":"2024-03-06T07:27:43.669317+0000","last_fullsized":"2024-03-06T07:27:43.669317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096181+0000","last_change":"2024-03-06T07:27:42.105851+0000","last_active":"2024-03-06T07:27:43.096181+0000","last_peered":"2024-03-06T07:27:43.096181+0000","last_clean":"2024-03-06T07:27:43.096181+0000","last_became_active":"2024-03-06T07:27:42.105276+0000","last_became_peered":"2024-03-06T07:27:42.105276+0000","last_unstale":"2024-03-06T07:27:43.096181+0000","last_undegraded":"2024-03-06T07:27:43.096181+0000","last_fullsized":"2024-03-06T07:27:43.096181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561313+0000","last_change":"2024-03-06T07:27:42.107395+0000","last_active":"2024-03-06T07:27:43.561313+0000","last_peered":"2024-03-06T07:27:43.561313+0000","last_clean":"2024-03-06T07:27:43.561313+0000","last_became_active":"2024-03-06T07:27:42.104429+0000","last_became_peered":"2024-03-06T07:27:42.104429+0000","last_unstale":"2024-03-06T07:27:43.561313+0000","last_undegraded":"2024-03-06T07:27:43.561313+0000","last_fullsized":"2024-03-06T07:27:43.561313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604391+0000","last_change":"2024-03-06T07:27:42.105460+0000","last_active":"2024-03-06T07:27:43.604391+0000","last_peered":"2024-03-06T07:27:43.604391+0000","last_clean":"2024-03-06T07:27:43.604391+0000","last_became_active":"2024-03-06T07:27:42.103643+0000","last_became_peered":"2024-03-06T07:27:42.103643+0000","last_unstale":"2024-03-06T07:27:43.604391+0000","last_undegraded":"2024-03-06T07:27:43.604391+0000","last_fullsized":"2024-03-06T07:27:43.604391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264480+0000","last_change":"2024-03-06T07:27:42.094539+0000","last_active":"2024-03-06T07:27:43.264480+0000","last_peered":"2024-03-06T07:27:43.264480+0000","last_clean":"2024-03-06T07:27:43.264480+0000","last_became_active":"2024-03-06T07:27:42.091758+0000","last_became_peered":"2024-03-06T07:27:42.091758+0000","last_unstale":"2024-03-06T07:27:43.264480+0000","last_undegraded":"2024-03-06T07:27:43.264480+0000","last_fullsized":"2024-03-06T07:27:43.264480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604546+0000","last_change":"2024-03-06T07:27:42.113737+0000","last_active":"2024-03-06T07:27:43.604546+0000","last_peered":"2024-03-06T07:27:43.604546+0000","last_clean":"2024-03-06T07:27:43.604546+0000","last_became_active":"2024-03-06T07:27:42.113444+0000","last_became_peered":"2024-03-06T07:27:42.113444+0000","last_unstale":"2024-03-06T07:27:43.604546+0000","last_undegraded":"2024-03-06T07:27:43.604546+0000","last_fullsized":"2024-03-06T07:27:43.604546+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265404+0000","last_change":"2024-03-06T07:27:42.104966+0000","last_active":"2024-03-06T07:27:43.265404+0000","last_peered":"2024-03-06T07:27:43.265404+0000","last_clean":"2024-03-06T07:27:43.265404+0000","last_became_active":"2024-03-06T07:27:42.104220+0000","last_became_peered":"2024-03-06T07:27:42.104220+0000","last_unstale":"2024-03-06T07:27:43.265404+0000","last_undegraded":"2024-03-06T07:27:43.265404+0000","last_fullsized":"2024-03-06T07:27:43.265404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096684+0000","last_change":"2024-03-06T07:27:42.103398+0000","last_active":"2024-03-06T07:27:43.096684+0000","last_peered":"2024-03-06T07:27:43.096684+0000","last_clean":"2024-03-06T07:27:43.096684+0000","last_became_active":"2024-03-06T07:27:42.102707+0000","last_became_peered":"2024-03-06T07:27:42.102707+0000","last_unstale":"2024-03-06T07:27:43.096684+0000","last_undegraded":"2024-03-06T07:27:43.096684+0000","last_fullsized":"2024-03-06T07:27:43.096684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090325+0000","last_change":"2024-03-06T07:27:42.092577+0000","last_active":"2024-03-06T07:27:43.090325+0000","last_peered":"2024-03-06T07:27:43.090325+0000","last_clean":"2024-03-06T07:27:43.090325+0000","last_became_active":"2024-03-06T07:27:42.092269+0000","last_became_peered":"2024-03-06T07:27:42.092269+0000","last_unstale":"2024-03-06T07:27:43.090325+0000","last_undegraded":"2024-03-06T07:27:43.090325+0000","last_fullsized":"2024-03-06T07:27:43.090325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265145+0000","last_change":"2024-03-06T07:27:42.105501+0000","last_active":"2024-03-06T07:27:43.265145+0000","last_peered":"2024-03-06T07:27:43.265145+0000","last_clean":"2024-03-06T07:27:43.265145+0000","last_became_active":"2024-03-06T07:27:42.100336+0000","last_became_peered":"2024-03-06T07:27:42.100336+0000","last_unstale":"2024-03-06T07:27:43.265145+0000","last_undegraded":"2024-03-06T07:27:43.265145+0000","last_fullsized":"2024-03-06T07:27:43.265145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561748+0000","last_change":"2024-03-06T07:27:42.090601+0000","last_active":"2024-03-06T07:27:43.561748+0000","last_peered":"2024-03-06T07:27:43.561748+0000","last_clean":"2024-03-06T07:27:43.561748+0000","last_became_active":"2024-03-06T07:27:42.090283+0000","last_became_peered":"2024-03-06T07:27:42.090283+0000","last_unstale":"2024-03-06T07:27:43.561748+0000","last_undegraded":"2024-03-06T07:27:43.561748+0000","last_fullsized":"2024-03-06T07:27:43.561748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604604+0000","last_change":"2024-03-06T07:27:42.113977+0000","last_active":"2024-03-06T07:27:43.604604+0000","last_peered":"2024-03-06T07:27:43.604604+0000","last_clean":"2024-03-06T07:27:43.604604+0000","last_became_active":"2024-03-06T07:27:42.111520+0000","last_became_peered":"2024-03-06T07:27:42.111520+0000","last_unstale":"2024-03-06T07:27:43.604604+0000","last_undegraded":"2024-03-06T07:27:43.604604+0000","last_fullsized":"2024-03-06T07:27:43.604604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093842+0000","last_change":"2024-03-06T07:27:42.096735+0000","last_active":"2024-03-06T07:27:43.093842+0000","last_peered":"2024-03-06T07:27:43.093842+0000","last_clean":"2024-03-06T07:27:43.093842+0000","last_became_active":"2024-03-06T07:27:42.096304+0000","last_became_peered":"2024-03-06T07:27:42.096304+0000","last_unstale":"2024-03-06T07:27:43.093842+0000","last_undegraded":"2024-03-06T07:27:43.093842+0000","last_fullsized":"2024-03-06T07:27:43.093842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:37.292940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095982+0000","last_change":"2024-03-06T07:27:42.096332+0000","last_active":"2024-03-06T07:27:43.095982+0000","last_peered":"2024-03-06T07:27:43.095982+0000","last_clean":"2024-03-06T07:27:43.095982+0000","last_became_active":"2024-03-06T07:27:42.095984+0000","last_became_peered":"2024-03-06T07:27:42.095984+0000","last_unstale":"2024-03-06T07:27:43.095982+0000","last_undegraded":"2024-03-06T07:27:43.095982+0000","last_fullsized":"2024-03-06T07:27:43.095982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.433842+0000","last_change":"2024-03-06T07:27:42.091042+0000","last_active":"2024-03-06T07:27:43.433842+0000","last_peered":"2024-03-06T07:27:43.433842+0000","last_clean":"2024-03-06T07:27:43.433842+0000","last_became_active":"2024-03-06T07:27:42.089637+0000","last_became_peered":"2024-03-06T07:27:42.089637+0000","last_unstale":"2024-03-06T07:27:43.433842+0000","last_undegraded":"2024-03-06T07:27:43.433842+0000","last_fullsized":"2024-03-06T07:27:43.433842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:00.218649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265208+0000","last_change":"2024-03-06T07:27:41.111941+0000","last_active":"2024-03-06T07:27:43.265208+0000","last_peered":"2024-03-06T07:27:43.265208+0000","last_clean":"2024-03-06T07:27:43.265208+0000","last_became_active":"2024-03-06T07:27:41.111244+0000","last_became_peered":"2024-03-06T07:27:41.111244+0000","last_unstale":"2024-03-06T07:27:43.265208+0000","last_undegraded":"2024-03-06T07:27:43.265208+0000","last_fullsized":"2024-03-06T07:27:43.265208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561659+0000","last_change":"2024-03-06T07:27:42.098732+0000","last_active":"2024-03-06T07:27:43.561659+0000","last_peered":"2024-03-06T07:27:43.561659+0000","last_clean":"2024-03-06T07:27:43.561659+0000","last_became_active":"2024-03-06T07:27:42.098451+0000","last_became_peered":"2024-03-06T07:27:42.098451+0000","last_unstale":"2024-03-06T07:27:43.561659+0000","last_undegraded":"2024-03-06T07:27:43.561659+0000","last_fullsized":"2024-03-06T07:27:43.561659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095423+0000","last_change":"2024-03-06T07:27:41.102703+0000","last_active":"2024-03-06T07:27:43.095423+0000","last_peered":"2024-03-06T07:27:43.095423+0000","last_clean":"2024-03-06T07:27:43.095423+0000","last_became_active":"2024-03-06T07:27:41.100059+0000","last_became_peered":"2024-03-06T07:27:41.100059+0000","last_unstale":"2024-03-06T07:27:43.095423+0000","last_undegraded":"2024-03-06T07:27:43.095423+0000","last_fullsized":"2024-03-06T07:27:43.095423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264571+0000","last_change":"2024-03-06T07:27:41.105849+0000","last_active":"2024-03-06T07:27:43.264571+0000","last_peered":"2024-03-06T07:27:43.264571+0000","last_clean":"2024-03-06T07:27:43.264571+0000","last_became_active":"2024-03-06T07:27:41.105558+0000","last_became_peered":"2024-03-06T07:27:41.105558+0000","last_unstale":"2024-03-06T07:27:43.264571+0000","last_undegraded":"2024-03-06T07:27:43.264571+0000","last_fullsized":"2024-03-06T07:27:43.264571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264534+0000","last_change":"2024-03-06T07:27:42.098554+0000","last_active":"2024-03-06T07:27:43.264534+0000","last_peered":"2024-03-06T07:27:43.264534+0000","last_clean":"2024-03-06T07:27:43.264534+0000","last_became_active":"2024-03-06T07:27:42.097364+0000","last_became_peered":"2024-03-06T07:27:42.097364+0000","last_unstale":"2024-03-06T07:27:43.264534+0000","last_undegraded":"2024-03-06T07:27:43.264534+0000","last_fullsized":"2024-03-06T07:27:43.264534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604701+0000","last_change":"2024-03-06T07:27:41.103209+0000","last_active":"2024-03-06T07:27:43.604701+0000","last_peered":"2024-03-06T07:27:43.604701+0000","last_clean":"2024-03-06T07:27:43.604701+0000","last_became_active":"2024-03-06T07:27:41.102994+0000","last_became_peered":"2024-03-06T07:27:41.102994+0000","last_unstale":"2024-03-06T07:27:43.604701+0000","last_undegraded":"2024-03-06T07:27:43.604701+0000","last_fullsized":"2024-03-06T07:27:43.604701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090178+0000","last_change":"2024-03-06T07:27:42.086153+0000","last_active":"2024-03-06T07:27:43.090178+0000","last_peered":"2024-03-06T07:27:43.090178+0000","last_clean":"2024-03-06T07:27:43.090178+0000","last_became_active":"2024-03-06T07:27:42.085779+0000","last_became_peered":"2024-03-06T07:27:42.085779+0000","last_unstale":"2024-03-06T07:27:43.090178+0000","last_undegraded":"2024-03-06T07:27:43.090178+0000","last_fullsized":"2024-03-06T07:27:43.090178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096443+0000","last_change":"2024-03-06T07:27:41.109144+0000","last_active":"2024-03-06T07:27:43.096443+0000","last_peered":"2024-03-06T07:27:43.096443+0000","last_clean":"2024-03-06T07:27:43.096443+0000","last_became_active":"2024-03-06T07:27:41.108399+0000","last_became_peered":"2024-03-06T07:27:41.108399+0000","last_unstale":"2024-03-06T07:27:43.096443+0000","last_undegraded":"2024-03-06T07:27:43.096443+0000","last_fullsized":"2024-03-06T07:27:43.096443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604733+0000","last_change":"2024-03-06T07:27:42.106823+0000","last_active":"2024-03-06T07:27:43.604733+0000","last_peered":"2024-03-06T07:27:43.604733+0000","last_clean":"2024-03-06T07:27:43.604733+0000","last_became_active":"2024-03-06T07:27:42.103949+0000","last_became_peered":"2024-03-06T07:27:42.103949+0000","last_unstale":"2024-03-06T07:27:43.604733+0000","last_undegraded":"2024-03-06T07:27:43.604733+0000","last_fullsized":"2024-03-06T07:27:43.604733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605002+0000","last_change":"2024-03-06T07:27:41.108455+0000","last_active":"2024-03-06T07:27:43.605002+0000","last_peered":"2024-03-06T07:27:43.605002+0000","last_clean":"2024-03-06T07:27:43.605002+0000","last_became_active":"2024-03-06T07:27:41.106095+0000","last_became_peered":"2024-03-06T07:27:41.106095+0000","last_unstale":"2024-03-06T07:27:43.605002+0000","last_undegraded":"2024-03-06T07:27:43.605002+0000","last_fullsized":"2024-03-06T07:27:43.605002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561945+0000","last_change":"2024-03-06T07:27:42.106635+0000","last_active":"2024-03-06T07:27:43.561945+0000","last_peered":"2024-03-06T07:27:43.561945+0000","last_clean":"2024-03-06T07:27:43.561945+0000","last_became_active":"2024-03-06T07:27:42.106373+0000","last_became_peered":"2024-03-06T07:27:42.106373+0000","last_unstale":"2024-03-06T07:27:43.561945+0000","last_undegraded":"2024-03-06T07:27:43.561945+0000","last_fullsized":"2024-03-06T07:27:43.561945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090296+0000","last_change":"2024-03-06T07:27:41.108679+0000","last_active":"2024-03-06T07:27:43.090296+0000","last_peered":"2024-03-06T07:27:43.090296+0000","last_clean":"2024-03-06T07:27:43.090296+0000","last_became_active":"2024-03-06T07:27:41.108429+0000","last_became_peered":"2024-03-06T07:27:41.108429+0000","last_unstale":"2024-03-06T07:27:43.090296+0000","last_undegraded":"2024-03-06T07:27:43.090296+0000","last_fullsized":"2024-03-06T07:27:43.090296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434363+0000","last_change":"2024-03-06T07:27:42.102682+0000","last_active":"2024-03-06T07:27:43.434363+0000","last_peered":"2024-03-06T07:27:43.434363+0000","last_clean":"2024-03-06T07:27:43.434363+0000","last_became_active":"2024-03-06T07:27:42.102232+0000","last_became_peered":"2024-03-06T07:27:42.102232+0000","last_unstale":"2024-03-06T07:27:43.434363+0000","last_undegraded":"2024-03-06T07:27:43.434363+0000","last_fullsized":"2024-03-06T07:27:43.434363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:34.589802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096130+0000","last_change":"2024-03-06T07:27:41.096688+0000","last_active":"2024-03-06T07:27:43.096130+0000","last_peered":"2024-03-06T07:27:43.096130+0000","last_clean":"2024-03-06T07:27:43.096130+0000","last_became_active":"2024-03-06T07:27:41.096391+0000","last_became_peered":"2024-03-06T07:27:41.096391+0000","last_unstale":"2024-03-06T07:27:43.096130+0000","last_undegraded":"2024-03-06T07:27:43.096130+0000","last_fullsized":"2024-03-06T07:27:43.096130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668491+0000","last_change":"2024-03-06T07:27:42.116169+0000","last_active":"2024-03-06T07:27:43.668491+0000","last_peered":"2024-03-06T07:27:43.668491+0000","last_clean":"2024-03-06T07:27:43.668491+0000","last_became_active":"2024-03-06T07:27:42.115868+0000","last_became_peered":"2024-03-06T07:27:42.115868+0000","last_unstale":"2024-03-06T07:27:43.668491+0000","last_undegraded":"2024-03-06T07:27:43.668491+0000","last_fullsized":"2024-03-06T07:27:43.668491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.091480+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.090511+0000","last_became_peered":"2024-03-06T07:27:41.090511+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561169+0000","last_change":"2024-03-06T07:27:42.104948+0000","last_active":"2024-03-06T07:27:43.561169+0000","last_peered":"2024-03-06T07:27:43.561169+0000","last_clean":"2024-03-06T07:27:43.561169+0000","last_became_active":"2024-03-06T07:27:42.104640+0000","last_became_peered":"2024-03-06T07:27:42.104640+0000","last_unstale":"2024-03-06T07:27:43.561169+0000","last_undegraded":"2024-03-06T07:27:43.561169+0000","last_fullsized":"2024-03-06T07:27:43.561169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604876+0000","last_change":"2024-03-06T07:27:41.103424+0000","last_active":"2024-03-06T07:27:43.604876+0000","last_peered":"2024-03-06T07:27:43.604876+0000","last_clean":"2024-03-06T07:27:43.604876+0000","last_became_active":"2024-03-06T07:27:41.102084+0000","last_became_peered":"2024-03-06T07:27:41.102084+0000","last_unstale":"2024-03-06T07:27:43.604876+0000","last_undegraded":"2024-03-06T07:27:43.604876+0000","last_fullsized":"2024-03-06T07:27:43.604876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668345+0000","last_change":"2024-03-06T07:27:42.107947+0000","last_active":"2024-03-06T07:27:43.668345+0000","last_peered":"2024-03-06T07:27:43.668345+0000","last_clean":"2024-03-06T07:27:43.668345+0000","last_became_active":"2024-03-06T07:27:42.107570+0000","last_became_peered":"2024-03-06T07:27:42.107570+0000","last_unstale":"2024-03-06T07:27:43.668345+0000","last_undegraded":"2024-03-06T07:27:43.668345+0000","last_fullsized":"2024-03-06T07:27:43.668345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095592+0000","last_change":"2024-03-06T07:27:41.105127+0000","last_active":"2024-03-06T07:27:43.095592+0000","last_peered":"2024-03-06T07:27:43.095592+0000","last_clean":"2024-03-06T07:27:43.095592+0000","last_became_active":"2024-03-06T07:27:41.104134+0000","last_became_peered":"2024-03-06T07:27:41.104134+0000","last_unstale":"2024-03-06T07:27:43.095592+0000","last_undegraded":"2024-03-06T07:27:43.095592+0000","last_fullsized":"2024-03-06T07:27:43.095592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096137+0000","last_change":"2024-03-06T07:27:42.104623+0000","last_active":"2024-03-06T07:27:43.096137+0000","last_peered":"2024-03-06T07:27:43.096137+0000","last_clean":"2024-03-06T07:27:43.096137+0000","last_became_active":"2024-03-06T07:27:42.104185+0000","last_became_peered":"2024-03-06T07:27:42.104185+0000","last_unstale":"2024-03-06T07:27:43.096137+0000","last_undegraded":"2024-03-06T07:27:43.096137+0000","last_fullsized":"2024-03-06T07:27:43.096137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093473+0000","last_change":"2024-03-06T07:27:41.095963+0000","last_active":"2024-03-06T07:27:43.093473+0000","last_peered":"2024-03-06T07:27:43.093473+0000","last_clean":"2024-03-06T07:27:43.093473+0000","last_became_active":"2024-03-06T07:27:41.095610+0000","last_became_peered":"2024-03-06T07:27:41.095610+0000","last_unstale":"2024-03-06T07:27:43.093473+0000","last_undegraded":"2024-03-06T07:27:43.093473+0000","last_fullsized":"2024-03-06T07:27:43.093473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668062+0000","last_change":"2024-03-06T07:27:42.107796+0000","last_active":"2024-03-06T07:27:43.668062+0000","last_peered":"2024-03-06T07:27:43.668062+0000","last_clean":"2024-03-06T07:27:43.668062+0000","last_became_active":"2024-03-06T07:27:42.107514+0000","last_became_peered":"2024-03-06T07:27:42.107514+0000","last_unstale":"2024-03-06T07:27:43.668062+0000","last_undegraded":"2024-03-06T07:27:43.668062+0000","last_fullsized":"2024-03-06T07:27:43.668062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434682+0000","last_change":"2024-03-06T07:27:41.094350+0000","last_active":"2024-03-06T07:27:43.434682+0000","last_peered":"2024-03-06T07:27:43.434682+0000","last_clean":"2024-03-06T07:27:43.434682+0000","last_became_active":"2024-03-06T07:27:41.089935+0000","last_became_peered":"2024-03-06T07:27:41.089935+0000","last_unstale":"2024-03-06T07:27:43.434682+0000","last_undegraded":"2024-03-06T07:27:43.434682+0000","last_fullsized":"2024-03-06T07:27:43.434682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:49.952499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669175+0000","last_change":"2024-03-06T07:27:42.109835+0000","last_active":"2024-03-06T07:27:43.669175+0000","last_peered":"2024-03-06T07:27:43.669175+0000","last_clean":"2024-03-06T07:27:43.669175+0000","last_became_active":"2024-03-06T07:27:42.109535+0000","last_became_peered":"2024-03-06T07:27:42.109535+0000","last_unstale":"2024-03-06T07:27:43.669175+0000","last_undegraded":"2024-03-06T07:27:43.669175+0000","last_fullsized":"2024-03-06T07:27:43.669175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093784+0000","last_change":"2024-03-06T07:27:41.092525+0000","last_active":"2024-03-06T07:27:43.093784+0000","last_peered":"2024-03-06T07:27:43.093784+0000","last_clean":"2024-03-06T07:27:43.093784+0000","last_became_active":"2024-03-06T07:27:41.091644+0000","last_became_peered":"2024-03-06T07:27:41.091644+0000","last_unstale":"2024-03-06T07:27:43.093784+0000","last_undegraded":"2024-03-06T07:27:43.093784+0000","last_fullsized":"2024-03-06T07:27:43.093784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.670064+0000","last_change":"2024-03-06T07:27:42.117934+0000","last_active":"2024-03-06T07:27:43.670064+0000","last_peered":"2024-03-06T07:27:43.670064+0000","last_clean":"2024-03-06T07:27:43.670064+0000","last_became_active":"2024-03-06T07:27:42.115924+0000","last_became_peered":"2024-03-06T07:27:42.115924+0000","last_unstale":"2024-03-06T07:27:43.670064+0000","last_undegraded":"2024-03-06T07:27:43.670064+0000","last_fullsized":"2024-03-06T07:27:43.670064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:39.844169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090372+0000","last_change":"2024-03-06T07:27:41.097118+0000","last_active":"2024-03-06T07:27:43.090372+0000","last_peered":"2024-03-06T07:27:43.090372+0000","last_clean":"2024-03-06T07:27:43.090372+0000","last_became_active":"2024-03-06T07:27:41.096188+0000","last_became_peered":"2024-03-06T07:27:41.096188+0000","last_unstale":"2024-03-06T07:27:43.090372+0000","last_undegraded":"2024-03-06T07:27:43.090372+0000","last_fullsized":"2024-03-06T07:27:43.090372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605208+0000","last_change":"2024-03-06T07:27:42.114130+0000","last_active":"2024-03-06T07:27:43.605208+0000","last_peered":"2024-03-06T07:27:43.605208+0000","last_clean":"2024-03-06T07:27:43.605208+0000","last_became_active":"2024-03-06T07:27:42.113811+0000","last_became_peered":"2024-03-06T07:27:42.113811+0000","last_unstale":"2024-03-06T07:27:43.605208+0000","last_undegraded":"2024-03-06T07:27:43.605208+0000","last_fullsized":"2024-03-06T07:27:43.605208+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093668+0000","last_change":"2024-03-06T07:27:41.102837+0000","last_active":"2024-03-06T07:27:43.093668+0000","last_peered":"2024-03-06T07:27:43.093668+0000","last_clean":"2024-03-06T07:27:43.093668+0000","last_became_active":"2024-03-06T07:27:41.102526+0000","last_became_peered":"2024-03-06T07:27:41.102526+0000","last_unstale":"2024-03-06T07:27:43.093668+0000","last_undegraded":"2024-03-06T07:27:43.093668+0000","last_fullsized":"2024-03-06T07:27:43.093668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668936+0000","last_change":"2024-03-06T07:27:42.110688+0000","last_active":"2024-03-06T07:27:43.668936+0000","last_peered":"2024-03-06T07:27:43.668936+0000","last_clean":"2024-03-06T07:27:43.668936+0000","last_became_active":"2024-03-06T07:27:42.110394+0000","last_became_peered":"2024-03-06T07:27:42.110394+0000","last_unstale":"2024-03-06T07:27:43.668936+0000","last_undegraded":"2024-03-06T07:27:43.668936+0000","last_fullsized":"2024-03-06T07:27:43.668936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667970+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:27:43.667970+0000","last_peered":"2024-03-06T07:27:43.667970+0000","last_clean":"2024-03-06T07:27:43.667970+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:27:43.667970+0000","last_undegraded":"2024-03-06T07:27:43.667970+0000","last_fullsized":"2024-03-06T07:27:43.667970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604170+0000","last_change":"2024-03-06T07:27:41.111128+0000","last_active":"2024-03-06T07:27:43.604170+0000","last_peered":"2024-03-06T07:27:43.604170+0000","last_clean":"2024-03-06T07:27:43.604170+0000","last_became_active":"2024-03-06T07:27:41.109382+0000","last_became_peered":"2024-03-06T07:27:41.109382+0000","last_unstale":"2024-03-06T07:27:43.604170+0000","last_undegraded":"2024-03-06T07:27:43.604170+0000","last_fullsized":"2024-03-06T07:27:43.604170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264748+0000","last_change":"2024-03-06T07:27:42.100303+0000","last_active":"2024-03-06T07:27:43.264748+0000","last_peered":"2024-03-06T07:27:43.264748+0000","last_clean":"2024-03-06T07:27:43.264748+0000","last_became_active":"2024-03-06T07:27:42.099977+0000","last_became_peered":"2024-03-06T07:27:42.099977+0000","last_unstale":"2024-03-06T07:27:43.264748+0000","last_undegraded":"2024-03-06T07:27:43.264748+0000","last_fullsized":"2024-03-06T07:27:43.264748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667817+0000","last_change":"2024-03-06T07:27:41.113323+0000","last_active":"2024-03-06T07:27:43.667817+0000","last_peered":"2024-03-06T07:27:43.667817+0000","last_clean":"2024-03-06T07:27:43.667817+0000","last_became_active":"2024-03-06T07:27:41.112921+0000","last_became_peered":"2024-03-06T07:27:41.112921+0000","last_unstale":"2024-03-06T07:27:43.667817+0000","last_undegraded":"2024-03-06T07:27:43.667817+0000","last_fullsized":"2024-03-06T07:27:43.667817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:12.710184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561173+0000","last_change":"2024-03-06T07:27:42.098275+0000","last_active":"2024-03-06T07:27:43.561173+0000","last_peered":"2024-03-06T07:27:43.561173+0000","last_clean":"2024-03-06T07:27:43.561173+0000","last_became_active":"2024-03-06T07:27:42.097430+0000","last_became_peered":"2024-03-06T07:27:42.097430+0000","last_unstale":"2024-03-06T07:27:43.561173+0000","last_undegraded":"2024-03-06T07:27:43.561173+0000","last_fullsized":"2024-03-06T07:27:43.561173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:43.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093787+0000","last_change":"2024-03-06T07:27:41.092149+0000","last_active":"2024-03-06T07:27:43.093787+0000","last_peered":"2024-03-06T07:27:43.093787+0000","last_clean":"2024-03-06T07:27:43.093787+0000","last_became_active":"2024-03-06T07:27:41.090942+0000","last_became_peered":"2024-03-06T07:27:41.090942+0000","last_unstale":"2024-03-06T07:27:43.093787+0000","last_undegraded":"2024-03-06T07:27:43.093787+0000","last_fullsized":"2024-03-06T07:27:43.093787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089953+0000","last_change":"2024-03-06T07:27:42.103118+0000","last_active":"2024-03-06T07:27:43.089953+0000","last_peered":"2024-03-06T07:27:43.089953+0000","last_clean":"2024-03-06T07:27:43.089953+0000","last_became_active":"2024-03-06T07:27:42.102850+0000","last_became_peered":"2024-03-06T07:27:42.102850+0000","last_unstale":"2024-03-06T07:27:43.089953+0000","last_undegraded":"2024-03-06T07:27:43.089953+0000","last_fullsized":"2024-03-06T07:27:43.089953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605179+0000","last_change":"2024-03-06T07:27:41.111452+0000","last_active":"2024-03-06T07:27:43.605179+0000","last_peered":"2024-03-06T07:27:43.605179+0000","last_clean":"2024-03-06T07:27:43.605179+0000","last_became_active":"2024-03-06T07:27:41.110359+0000","last_became_peered":"2024-03-06T07:27:41.110359+0000","last_unstale":"2024-03-06T07:27:43.605179+0000","last_undegraded":"2024-03-06T07:27:43.605179+0000","last_fullsized":"2024-03-06T07:27:43.605179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434187+0000","last_change":"2024-03-06T07:27:42.093244+0000","last_active":"2024-03-06T07:27:43.434187+0000","last_peered":"2024-03-06T07:27:43.434187+0000","last_clean":"2024-03-06T07:27:43.434187+0000","last_became_active":"2024-03-06T07:27:42.092228+0000","last_became_peered":"2024-03-06T07:27:42.092228+0000","last_unstale":"2024-03-06T07:27:43.434187+0000","last_undegraded":"2024-03-06T07:27:43.434187+0000","last_fullsized":"2024-03-06T07:27:43.434187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:07.427898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090250+0000","last_change":"2024-03-06T07:27:41.093905+0000","last_active":"2024-03-06T07:27:43.090250+0000","last_peered":"2024-03-06T07:27:43.090250+0000","last_clean":"2024-03-06T07:27:43.090250+0000","last_became_active":"2024-03-06T07:27:41.093456+0000","last_became_peered":"2024-03-06T07:27:41.093456+0000","last_unstale":"2024-03-06T07:27:43.090250+0000","last_undegraded":"2024-03-06T07:27:43.090250+0000","last_fullsized":"2024-03-06T07:27:43.090250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605312+0000","last_change":"2024-03-06T07:27:42.112370+0000","last_active":"2024-03-06T07:27:43.605312+0000","last_peered":"2024-03-06T07:27:43.605312+0000","last_clean":"2024-03-06T07:27:43.605312+0000","last_became_active":"2024-03-06T07:27:42.112087+0000","last_became_peered":"2024-03-06T07:27:42.112087+0000","last_unstale":"2024-03-06T07:27:43.605312+0000","last_undegraded":"2024-03-06T07:27:43.605312+0000","last_fullsized":"2024-03-06T07:27:43.605312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669326+0000","last_change":"2024-03-06T07:27:41.116636+0000","last_active":"2024-03-06T07:27:43.669326+0000","last_peered":"2024-03-06T07:27:43.669326+0000","last_clean":"2024-03-06T07:27:43.669326+0000","last_became_active":"2024-03-06T07:27:41.116254+0000","last_became_peered":"2024-03-06T07:27:41.116254+0000","last_unstale":"2024-03-06T07:27:43.669326+0000","last_undegraded":"2024-03-06T07:27:43.669326+0000","last_fullsized":"2024-03-06T07:27:43.669326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:59.546367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090070+0000","last_change":"2024-03-06T07:27:42.094877+0000","last_active":"2024-03-06T07:27:43.090070+0000","last_peered":"2024-03-06T07:27:43.090070+0000","last_clean":"2024-03-06T07:27:43.090070+0000","last_became_active":"2024-03-06T07:27:42.094586+0000","last_became_peered":"2024-03-06T07:27:42.094586+0000","last_unstale":"2024-03-06T07:27:43.090070+0000","last_undegraded":"2024-03-06T07:27:43.090070+0000","last_fullsized":"2024-03-06T07:27:43.090070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.102122+0000","last_change":"2024-03-06T07:27:41.098449+0000","last_active":"2024-03-06T07:27:43.102122+0000","last_peered":"2024-03-06T07:27:43.102122+0000","last_clean":"2024-03-06T07:27:43.102122+0000","last_became_active":"2024-03-06T07:27:41.098137+0000","last_became_peered":"2024-03-06T07:27:41.098137+0000","last_unstale":"2024-03-06T07:27:43.102122+0000","last_undegraded":"2024-03-06T07:27:43.102122+0000","last_fullsized":"2024-03-06T07:27:43.102122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265545+0000","last_change":"2024-03-06T07:27:42.118543+0000","last_active":"2024-03-06T07:27:43.265545+0000","last_peered":"2024-03-06T07:27:43.265545+0000","last_clean":"2024-03-06T07:27:43.265545+0000","last_became_active":"2024-03-06T07:27:42.118157+0000","last_became_peered":"2024-03-06T07:27:42.118157+0000","last_unstale":"2024-03-06T07:27:43.265545+0000","last_undegraded":"2024-03-06T07:27:43.265545+0000","last_fullsized":"2024-03-06T07:27:43.265545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096632+0000","last_change":"2024-03-06T07:27:41.112840+0000","last_active":"2024-03-06T07:27:43.096632+0000","last_peered":"2024-03-06T07:27:43.096632+0000","last_clean":"2024-03-06T07:27:43.096632+0000","last_became_active":"2024-03-06T07:27:41.112440+0000","last_became_peered":"2024-03-06T07:27:41.112440+0000","last_unstale":"2024-03-06T07:27:43.096632+0000","last_undegraded":"2024-03-06T07:27:43.096632+0000","last_fullsized":"2024-03-06T07:27:43.096632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561550+0000","last_change":"2024-03-06T07:27:42.090517+0000","last_active":"2024-03-06T07:27:43.561550+0000","last_peered":"2024-03-06T07:27:43.561550+0000","last_clean":"2024-03-06T07:27:43.561550+0000","last_became_active":"2024-03-06T07:27:42.090202+0000","last_became_peered":"2024-03-06T07:27:42.090202+0000","last_unstale":"2024-03-06T07:27:43.561550+0000","last_undegraded":"2024-03-06T07:27:43.561550+0000","last_fullsized":"2024-03-06T07:27:43.561550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667842+0000","last_change":"2024-03-06T07:27:41.114447+0000","last_active":"2024-03-06T07:27:43.667842+0000","last_peered":"2024-03-06T07:27:43.667842+0000","last_clean":"2024-03-06T07:27:43.667842+0000","last_became_active":"2024-03-06T07:27:41.113802+0000","last_became_peered":"2024-03-06T07:27:41.113802+0000","last_unstale":"2024-03-06T07:27:43.667842+0000","last_undegraded":"2024-03-06T07:27:43.667842+0000","last_fullsized":"2024-03-06T07:27:43.667842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:43.723690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434174+0000","last_change":"2024-03-06T07:27:42.110210+0000","last_active":"2024-03-06T07:27:43.434174+0000","last_peered":"2024-03-06T07:27:43.434174+0000","last_clean":"2024-03-06T07:27:43.434174+0000","last_became_active":"2024-03-06T07:27:42.109736+0000","last_became_peered":"2024-03-06T07:27:42.109736+0000","last_unstale":"2024-03-06T07:27:43.434174+0000","last_undegraded":"2024-03-06T07:27:43.434174+0000","last_fullsized":"2024-03-06T07:27:43.434174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:25.704825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090026+0000","last_change":"2024-03-06T07:27:41.101111+0000","last_active":"2024-03-06T07:27:43.090026+0000","last_peered":"2024-03-06T07:27:43.090026+0000","last_clean":"2024-03-06T07:27:43.090026+0000","last_became_active":"2024-03-06T07:27:41.100857+0000","last_became_peered":"2024-03-06T07:27:41.100857+0000","last_unstale":"2024-03-06T07:27:43.090026+0000","last_undegraded":"2024-03-06T07:27:43.090026+0000","last_fullsized":"2024-03-06T07:27:43.090026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264697+0000","last_change":"2024-03-06T07:27:42.086202+0000","last_active":"2024-03-06T07:27:43.264697+0000","last_peered":"2024-03-06T07:27:43.264697+0000","last_clean":"2024-03-06T07:27:43.264697+0000","last_became_active":"2024-03-06T07:27:42.085754+0000","last_became_peered":"2024-03-06T07:27:42.085754+0000","last_unstale":"2024-03-06T07:27:43.264697+0000","last_undegraded":"2024-03-06T07:27:43.264697+0000","last_fullsized":"2024-03-06T07:27:43.264697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090044+0000","last_change":"2024-03-06T07:27:41.096584+0000","last_active":"2024-03-06T07:27:43.090044+0000","last_peered":"2024-03-06T07:27:43.090044+0000","last_clean":"2024-03-06T07:27:43.090044+0000","last_became_active":"2024-03-06T07:27:41.096318+0000","last_became_peered":"2024-03-06T07:27:41.096318+0000","last_unstale":"2024-03-06T07:27:43.090044+0000","last_undegraded":"2024-03-06T07:27:43.090044+0000","last_fullsized":"2024-03-06T07:27:43.090044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265099+0000","last_change":"2024-03-06T07:27:42.102269+0000","last_active":"2024-03-06T07:27:43.265099+0000","last_peered":"2024-03-06T07:27:43.265099+0000","last_clean":"2024-03-06T07:27:43.265099+0000","last_became_active":"2024-03-06T07:27:42.092816+0000","last_became_peered":"2024-03-06T07:27:42.092816+0000","last_unstale":"2024-03-06T07:27:43.265099+0000","last_undegraded":"2024-03-06T07:27:43.265099+0000","last_fullsized":"2024-03-06T07:27:43.265099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668665+0000","last_change":"2024-03-06T07:27:41.116372+0000","last_active":"2024-03-06T07:27:43.668665+0000","last_peered":"2024-03-06T07:27:43.668665+0000","last_clean":"2024-03-06T07:27:43.668665+0000","last_became_active":"2024-03-06T07:27:41.116072+0000","last_became_peered":"2024-03-06T07:27:41.116072+0000","last_unstale":"2024-03-06T07:27:43.668665+0000","last_undegraded":"2024-03-06T07:27:43.668665+0000","last_fullsized":"2024-03-06T07:27:43.668665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095884+0000","last_change":"2024-03-06T07:27:42.103012+0000","last_active":"2024-03-06T07:27:43.095884+0000","last_peered":"2024-03-06T07:27:43.095884+0000","last_clean":"2024-03-06T07:27:43.095884+0000","last_became_active":"2024-03-06T07:27:42.102731+0000","last_became_peered":"2024-03-06T07:27:42.102731+0000","last_unstale":"2024-03-06T07:27:43.095884+0000","last_undegraded":"2024-03-06T07:27:43.095884+0000","last_fullsized":"2024-03-06T07:27:43.095884+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604917+0000","last_change":"2024-03-06T07:27:42.097462+0000","last_active":"2024-03-06T07:27:43.604917+0000","last_peered":"2024-03-06T07:27:43.604917+0000","last_clean":"2024-03-06T07:27:43.604917+0000","last_became_active":"2024-03-06T07:27:42.097142+0000","last_became_peered":"2024-03-06T07:27:42.097142+0000","last_unstale":"2024-03-06T07:27:43.604917+0000","last_undegraded":"2024-03-06T07:27:43.604917+0000","last_fullsized":"2024-03-06T07:27:43.604917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668754+0000","last_change":"2024-03-06T07:27:41.115663+0000","last_active":"2024-03-06T07:27:43.668754+0000","last_peered":"2024-03-06T07:27:43.668754+0000","last_clean":"2024-03-06T07:27:43.668754+0000","last_became_active":"2024-03-06T07:27:41.114756+0000","last_became_peered":"2024-03-06T07:27:41.114756+0000","last_unstale":"2024-03-06T07:27:43.668754+0000","last_undegraded":"2024-03-06T07:27:43.668754+0000","last_fullsized":"2024-03-06T07:27:43.668754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604966+0000","last_change":"2024-03-06T07:27:42.098684+0000","last_active":"2024-03-06T07:27:43.604966+0000","last_peered":"2024-03-06T07:27:43.604966+0000","last_clean":"2024-03-06T07:27:43.604966+0000","last_became_active":"2024-03-06T07:27:42.098307+0000","last_became_peered":"2024-03-06T07:27:42.098307+0000","last_unstale":"2024-03-06T07:27:43.604966+0000","last_undegraded":"2024-03-06T07:27:43.604966+0000","last_fullsized":"2024-03-06T07:27:43.604966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669076+0000","last_change":"2024-03-06T07:27:41.117369+0000","last_active":"2024-03-06T07:27:43.669076+0000","last_peered":"2024-03-06T07:27:43.669076+0000","last_clean":"2024-03-06T07:27:43.669076+0000","last_became_active":"2024-03-06T07:27:41.117042+0000","last_became_peered":"2024-03-06T07:27:41.117042+0000","last_unstale":"2024-03-06T07:27:43.669076+0000","last_undegraded":"2024-03-06T07:27:43.669076+0000","last_fullsized":"2024-03-06T07:27:43.669076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561654+0000","last_change":"2024-03-06T07:27:42.106505+0000","last_active":"2024-03-06T07:27:43.561654+0000","last_peered":"2024-03-06T07:27:43.561654+0000","last_clean":"2024-03-06T07:27:43.561654+0000","last_became_active":"2024-03-06T07:27:42.105993+0000","last_became_peered":"2024-03-06T07:27:42.105993+0000","last_unstale":"2024-03-06T07:27:43.561654+0000","last_undegraded":"2024-03-06T07:27:43.561654+0000","last_fullsized":"2024-03-06T07:27:43.561654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265054+0000","last_change":"2024-03-06T07:27:41.097028+0000","last_active":"2024-03-06T07:27:43.265054+0000","last_peered":"2024-03-06T07:27:43.265054+0000","last_clean":"2024-03-06T07:27:43.265054+0000","last_became_active":"2024-03-06T07:27:41.096532+0000","last_became_peered":"2024-03-06T07:27:41.096532+0000","last_unstale":"2024-03-06T07:27:43.265054+0000","last_undegraded":"2024-03-06T07:27:43.265054+0000","last_fullsized":"2024-03-06T07:27:43.265054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264945+0000","last_change":"2024-03-06T07:27:42.094785+0000","last_active":"2024-03-06T07:27:43.264945+0000","last_peered":"2024-03-06T07:27:43.264945+0000","last_clean":"2024-03-06T07:27:43.264945+0000","last_became_active":"2024-03-06T07:27:42.091901+0000","last_became_peered":"2024-03-06T07:27:42.091901+0000","last_unstale":"2024-03-06T07:27:43.264945+0000","last_undegraded":"2024-03-06T07:27:43.264945+0000","last_fullsized":"2024-03-06T07:27:43.264945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.102186+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.101868+0000","last_became_peered":"2024-03-06T07:27:41.101868+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:20.791281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:46.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:46.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.741+0000 7faa34000640 1 Processor -- start 2024-03-06T07:27:46.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.741+0000 7faa34000640 1 -- start start 2024-03-06T07:27:46.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa34000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15e0d0 0x7faa2c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa34000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa2c0d0fc0 0x7faa2c160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa34000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2c0cea60 con 0x7faa2c15f470 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa34000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0d2f60 con 0x7faa2c0d0fc0 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa34000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0d2350 con 0x7faa2c15e0d0 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa32000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa2c0d0fc0 0x7faa2c160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa32000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa2c0d0fc0 0x7faa2c160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50356/0 (socket says 172.21.15.97:50356) 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.743+0000 7faa32000640 1 -- 172.21.15.97:0/4195574725 learned_addr learned my addr 172.21.15.97:0/4195574725 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.744+0000 7faa31600640 1 --2- 172.21.15.97:0/4195574725 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15e0d0 0x7faa2c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.744+0000 7faa32000640 1 -- 172.21.15.97:0/4195574725 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15e0d0 msgr2=0x7faa2c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.744+0000 7faa32000640 1 --2- 172.21.15.97:0/4195574725 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15e0d0 0x7faa2c0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.744+0000 7faa32000640 1 -- 172.21.15.97:0/4195574725 >> v1:172.21.15.135:6789/0 conn(0x7faa2c15f470 legacy=0x7faa2c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.744+0000 7faa32000640 1 -- 172.21.15.97:0/4195574725 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c0d2630 con 0x7faa2c0d0fc0 2024-03-06T07:27:46.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.745+0000 7faa32000640 1 --2- 172.21.15.97:0/4195574725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c160dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa24009e50 tx=0x7faa2402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd1204c21fdefa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.745+0000 7faa2be00640 1 -- 172.21.15.97:0/4195574725 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2402cb90 con 0x7faa2c0d0fc0 2024-03-06T07:27:46.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.745+0000 7faa2be00640 1 -- 172.21.15.97:0/4195574725 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2402cd30 con 0x7faa2c0d0fc0 2024-03-06T07:27:46.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.745+0000 7faa2be00640 1 -- 172.21.15.97:0/4195574725 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24034610 con 0x7faa2c0d0fc0 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 -- 172.21.15.97:0/4195574725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0d0fc0 msgr2=0x7faa2c160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 --2- 172.21.15.97:0/4195574725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c160dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa24009e50 tx=0x7faa2402bff0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 -- 172.21.15.97:0/4195574725 shutdown_connections 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 --2- 172.21.15.97:0/4195574725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c160dc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 --2- 172.21.15.97:0/4195574725 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15e0d0 0x7faa2c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 -- 172.21.15.97:0/4195574725 >> 172.21.15.97:0/4195574725 conn(0x7faa2c0d3ec0 msgr2=0x7faa2c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 -- 172.21.15.97:0/4195574725 shutdown_connections 2024-03-06T07:27:46.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.746+0000 7faa34000640 1 -- 172.21.15.97:0/4195574725 wait complete. 2024-03-06T07:27:46.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.748+0000 7faa34000640 1 Processor -- start 2024-03-06T07:27:46.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.748+0000 7faa34000640 1 -- start start 2024-03-06T07:27:46.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa34000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c0cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa34000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 0x7faa2c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa34000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15f470 0x7faa2c0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa34000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c15fb30 con 0x7faa2c15e0d0 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa34000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0d2bf0 con 0x7faa2c0d0fc0 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa34000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0d1fd0 con 0x7faa2c15f470 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 0x7faa2c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 0x7faa2c155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50362/0 (socket says 172.21.15.97:50362) 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 -- 172.21.15.97:0/1786964853 learned_addr learned my addr 172.21.15.97:0/1786964853 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa31600640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c0cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 -- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15f470 msgr2=0x7faa2c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa32000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15f470 0x7faa2c0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15f470 0x7faa2c0c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 -- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c0d0fc0 msgr2=0x7faa2c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c0cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa30c00640 1 -- 172.21.15.97:0/1786964853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c155b40 con 0x7faa2c15e0d0 2024-03-06T07:27:46.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.749+0000 7faa31600640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c0cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.750+0000 7faa30c00640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 0x7faa2c155600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c450 tx=0x7faa1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.750+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18017070 con 0x7faa2c15e0d0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.750+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa180052d0 con 0x7faa2c15e0d0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.750+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18003830 con 0x7faa2c15e0d0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.750+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c0ce120 con 0x7faa2c15e0d0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.750+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c0b8100 con 0x7faa2c15e0d0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.751+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c04f330 con 0x7faa2c15e0d0 2024-03-06T07:27:46.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.752+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa1801a070 con 0x7faa2c15e0d0 2024-03-06T07:27:46.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.754+0000 7faa29600640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0c06b4d0 0x7faa0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.754+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa18089640 con 0x7faa2c15e0d0 2024-03-06T07:27:46.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.754+0000 7faa31600640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0c06b4d0 0x7faa0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.754+0000 7faa31600640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0c06b4d0 0x7faa0c06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faa20009ac0 tx=0x7faa20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.759+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa18058420 con 0x7faa2c15e0d0 2024-03-06T07:27:46.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.952+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa2c08d920 con 0x7faa0c06b4d0 2024-03-06T07:27:46.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.966+0000 7faa29600640 1 -- 172.21.15.97:0/1786964853 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251949 (secure 0 0 0) 0x7faa2c08d920 con 0x7faa0c06b4d0 2024-03-06T07:27:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:46.973 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:27:46.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.972+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0c06b4d0 msgr2=0x7faa0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0c06b4d0 0x7faa0c06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faa20009ac0 tx=0x7faa20008040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 msgr2=0x7faa2c155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 0x7faa2c155600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c450 tx=0x7faa1800c910 comp rx=0 tx=0).stop 2024-03-06T07:27:46.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 shutdown_connections 2024-03-06T07:27:46.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0c06b4d0 0x7faa0c06d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c15f470 0x7faa2c0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c15e0d0 0x7faa2c155600 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 --2- 172.21.15.97:0/1786964853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c0d0fc0 0x7faa2c0cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 >> 172.21.15.97:0/1786964853 conn(0x7faa2c0d3ec0 msgr2=0x7faa2c0b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 shutdown_connections 2024-03-06T07:27:46.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:46.973+0000 7faa34000640 1 -- 172.21.15.97:0/1786964853 wait complete. 2024-03-06T07:27:46.992 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-06T07:27:46.904449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219460,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755260,"statfs":{"total":773094113280,"available":772869386240,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001958"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669901+0000","last_change":"2024-03-06T07:27:42.115947+0000","last_active":"2024-03-06T07:27:43.669901+0000","last_peered":"2024-03-06T07:27:43.669901+0000","last_clean":"2024-03-06T07:27:43.669901+0000","last_became_active":"2024-03-06T07:27:42.115323+0000","last_became_peered":"2024-03-06T07:27:42.115323+0000","last_unstale":"2024-03-06T07:27:43.669901+0000","last_undegraded":"2024-03-06T07:27:43.669901+0000","last_fullsized":"2024-03-06T07:27:43.669901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:23.978663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090443+0000","last_change":"2024-03-06T07:27:42.092048+0000","last_active":"2024-03-06T07:27:43.090443+0000","last_peered":"2024-03-06T07:27:43.090443+0000","last_clean":"2024-03-06T07:27:43.090443+0000","last_became_active":"2024-03-06T07:27:42.091783+0000","last_became_peered":"2024-03-06T07:27:42.091783+0000","last_unstale":"2024-03-06T07:27:43.090443+0000","last_undegraded":"2024-03-06T07:27:43.090443+0000","last_fullsized":"2024-03-06T07:27:43.090443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561478+0000","last_change":"2024-03-06T07:27:42.095834+0000","last_active":"2024-03-06T07:27:43.561478+0000","last_peered":"2024-03-06T07:27:43.561478+0000","last_clean":"2024-03-06T07:27:43.561478+0000","last_became_active":"2024-03-06T07:27:42.095026+0000","last_became_peered":"2024-03-06T07:27:42.095026+0000","last_unstale":"2024-03-06T07:27:43.561478+0000","last_undegraded":"2024-03-06T07:27:43.561478+0000","last_fullsized":"2024-03-06T07:27:43.561478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604760+0000","last_change":"2024-03-06T07:27:42.114409+0000","last_active":"2024-03-06T07:27:43.604760+0000","last_peered":"2024-03-06T07:27:43.604760+0000","last_clean":"2024-03-06T07:27:43.604760+0000","last_became_active":"2024-03-06T07:27:42.114122+0000","last_became_peered":"2024-03-06T07:27:42.114122+0000","last_unstale":"2024-03-06T07:27:43.604760+0000","last_undegraded":"2024-03-06T07:27:43.604760+0000","last_fullsized":"2024-03-06T07:27:43.604760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668591+0000","last_change":"2024-03-06T07:27:42.101691+0000","last_active":"2024-03-06T07:27:43.668591+0000","last_peered":"2024-03-06T07:27:43.668591+0000","last_clean":"2024-03-06T07:27:43.668591+0000","last_became_active":"2024-03-06T07:27:42.101372+0000","last_became_peered":"2024-03-06T07:27:42.101372+0000","last_unstale":"2024-03-06T07:27:43.668591+0000","last_undegraded":"2024-03-06T07:27:43.668591+0000","last_fullsized":"2024-03-06T07:27:43.668591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604137+0000","last_change":"2024-03-06T07:27:42.094768+0000","last_active":"2024-03-06T07:27:43.604137+0000","last_peered":"2024-03-06T07:27:43.604137+0000","last_clean":"2024-03-06T07:27:43.604137+0000","last_became_active":"2024-03-06T07:27:42.094416+0000","last_became_peered":"2024-03-06T07:27:42.094416+0000","last_unstale":"2024-03-06T07:27:43.604137+0000","last_undegraded":"2024-03-06T07:27:43.604137+0000","last_fullsized":"2024-03-06T07:27:43.604137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668352+0000","last_change":"2024-03-06T07:27:42.101468+0000","last_active":"2024-03-06T07:27:43.668352+0000","last_peered":"2024-03-06T07:27:43.668352+0000","last_clean":"2024-03-06T07:27:43.668352+0000","last_became_active":"2024-03-06T07:27:42.101168+0000","last_became_peered":"2024-03-06T07:27:42.101168+0000","last_unstale":"2024-03-06T07:27:43.668352+0000","last_undegraded":"2024-03-06T07:27:43.668352+0000","last_fullsized":"2024-03-06T07:27:43.668352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265352+0000","last_change":"2024-03-06T07:27:42.098517+0000","last_active":"2024-03-06T07:27:43.265352+0000","last_peered":"2024-03-06T07:27:43.265352+0000","last_clean":"2024-03-06T07:27:43.265352+0000","last_became_active":"2024-03-06T07:27:42.092643+0000","last_became_peered":"2024-03-06T07:27:42.092643+0000","last_unstale":"2024-03-06T07:27:43.265352+0000","last_undegraded":"2024-03-06T07:27:43.265352+0000","last_fullsized":"2024-03-06T07:27:43.265352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669864+0000","last_change":"2024-03-06T07:27:42.119208+0000","last_active":"2024-03-06T07:27:43.669864+0000","last_peered":"2024-03-06T07:27:43.669864+0000","last_clean":"2024-03-06T07:27:43.669864+0000","last_became_active":"2024-03-06T07:27:42.118050+0000","last_became_peered":"2024-03-06T07:27:42.118050+0000","last_unstale":"2024-03-06T07:27:43.669864+0000","last_undegraded":"2024-03-06T07:27:43.669864+0000","last_fullsized":"2024-03-06T07:27:43.669864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:43.473299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096548+0000","last_change":"2024-03-06T07:27:42.095950+0000","last_active":"2024-03-06T07:27:43.096548+0000","last_peered":"2024-03-06T07:27:43.096548+0000","last_clean":"2024-03-06T07:27:43.096548+0000","last_became_active":"2024-03-06T07:27:42.095462+0000","last_became_peered":"2024-03-06T07:27:42.095462+0000","last_unstale":"2024-03-06T07:27:43.096548+0000","last_undegraded":"2024-03-06T07:27:43.096548+0000","last_fullsized":"2024-03-06T07:27:43.096548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095953+0000","last_change":"2024-03-06T07:27:42.096143+0000","last_active":"2024-03-06T07:27:43.095953+0000","last_peered":"2024-03-06T07:27:43.095953+0000","last_clean":"2024-03-06T07:27:43.095953+0000","last_became_active":"2024-03-06T07:27:42.095793+0000","last_became_peered":"2024-03-06T07:27:42.095793+0000","last_unstale":"2024-03-06T07:27:43.095953+0000","last_undegraded":"2024-03-06T07:27:43.095953+0000","last_fullsized":"2024-03-06T07:27:43.095953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434202+0000","last_change":"2024-03-06T07:27:42.092238+0000","last_active":"2024-03-06T07:27:43.434202+0000","last_peered":"2024-03-06T07:27:43.434202+0000","last_clean":"2024-03-06T07:27:43.434202+0000","last_became_active":"2024-03-06T07:27:42.091930+0000","last_became_peered":"2024-03-06T07:27:42.091930+0000","last_unstale":"2024-03-06T07:27:43.434202+0000","last_undegraded":"2024-03-06T07:27:43.434202+0000","last_fullsized":"2024-03-06T07:27:43.434202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:38.321736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264768+0000","last_change":"2024-03-06T07:27:42.097468+0000","last_active":"2024-03-06T07:27:43.264768+0000","last_peered":"2024-03-06T07:27:43.264768+0000","last_clean":"2024-03-06T07:27:43.264768+0000","last_became_active":"2024-03-06T07:27:42.094036+0000","last_became_peered":"2024-03-06T07:27:42.094036+0000","last_unstale":"2024-03-06T07:27:43.264768+0000","last_undegraded":"2024-03-06T07:27:43.264768+0000","last_fullsized":"2024-03-06T07:27:43.264768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561085+0000","last_change":"2024-03-06T07:27:42.092704+0000","last_active":"2024-03-06T07:27:43.561085+0000","last_peered":"2024-03-06T07:27:43.561085+0000","last_clean":"2024-03-06T07:27:43.561085+0000","last_became_active":"2024-03-06T07:27:42.090459+0000","last_became_peered":"2024-03-06T07:27:42.090459+0000","last_unstale":"2024-03-06T07:27:43.561085+0000","last_undegraded":"2024-03-06T07:27:43.561085+0000","last_fullsized":"2024-03-06T07:27:43.561085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:26.470370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089922+0000","last_change":"2024-03-06T07:27:42.076294+0000","last_active":"2024-03-06T07:27:43.089922+0000","last_peered":"2024-03-06T07:27:43.089922+0000","last_clean":"2024-03-06T07:27:43.089922+0000","last_became_active":"2024-03-06T07:27:42.075529+0000","last_became_peered":"2024-03-06T07:27:42.075529+0000","last_unstale":"2024-03-06T07:27:43.089922+0000","last_undegraded":"2024-03-06T07:27:43.089922+0000","last_fullsized":"2024-03-06T07:27:43.089922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604410+0000","last_change":"2024-03-06T07:27:42.093846+0000","last_active":"2024-03-06T07:27:43.604410+0000","last_peered":"2024-03-06T07:27:43.604410+0000","last_clean":"2024-03-06T07:27:43.604410+0000","last_became_active":"2024-03-06T07:27:42.093544+0000","last_became_peered":"2024-03-06T07:27:42.093544+0000","last_unstale":"2024-03-06T07:27:43.604410+0000","last_undegraded":"2024-03-06T07:27:43.604410+0000","last_fullsized":"2024-03-06T07:27:43.604410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605458+0000","last_change":"2024-03-06T07:27:42.118690+0000","last_active":"2024-03-06T07:27:43.605458+0000","last_peered":"2024-03-06T07:27:43.605458+0000","last_clean":"2024-03-06T07:27:43.605458+0000","last_became_active":"2024-03-06T07:27:42.118405+0000","last_became_peered":"2024-03-06T07:27:42.118405+0000","last_unstale":"2024-03-06T07:27:43.605458+0000","last_undegraded":"2024-03-06T07:27:43.605458+0000","last_fullsized":"2024-03-06T07:27:43.605458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093672+0000","last_change":"2024-03-06T07:27:42.096245+0000","last_active":"2024-03-06T07:27:43.093672+0000","last_peered":"2024-03-06T07:27:43.093672+0000","last_clean":"2024-03-06T07:27:43.093672+0000","last_became_active":"2024-03-06T07:27:42.094600+0000","last_became_peered":"2024-03-06T07:27:42.094600+0000","last_unstale":"2024-03-06T07:27:43.093672+0000","last_undegraded":"2024-03-06T07:27:43.093672+0000","last_fullsized":"2024-03-06T07:27:43.093672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:05.681114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669317+0000","last_change":"2024-03-06T07:27:42.104820+0000","last_active":"2024-03-06T07:27:43.669317+0000","last_peered":"2024-03-06T07:27:43.669317+0000","last_clean":"2024-03-06T07:27:43.669317+0000","last_became_active":"2024-03-06T07:27:42.104489+0000","last_became_peered":"2024-03-06T07:27:42.104489+0000","last_unstale":"2024-03-06T07:27:43.669317+0000","last_undegraded":"2024-03-06T07:27:43.669317+0000","last_fullsized":"2024-03-06T07:27:43.669317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096181+0000","last_change":"2024-03-06T07:27:42.105851+0000","last_active":"2024-03-06T07:27:43.096181+0000","last_peered":"2024-03-06T07:27:43.096181+0000","last_clean":"2024-03-06T07:27:43.096181+0000","last_became_active":"2024-03-06T07:27:42.105276+0000","last_became_peered":"2024-03-06T07:27:42.105276+0000","last_unstale":"2024-03-06T07:27:43.096181+0000","last_undegraded":"2024-03-06T07:27:43.096181+0000","last_fullsized":"2024-03-06T07:27:43.096181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561313+0000","last_change":"2024-03-06T07:27:42.107395+0000","last_active":"2024-03-06T07:27:43.561313+0000","last_peered":"2024-03-06T07:27:43.561313+0000","last_clean":"2024-03-06T07:27:43.561313+0000","last_became_active":"2024-03-06T07:27:42.104429+0000","last_became_peered":"2024-03-06T07:27:42.104429+0000","last_unstale":"2024-03-06T07:27:43.561313+0000","last_undegraded":"2024-03-06T07:27:43.561313+0000","last_fullsized":"2024-03-06T07:27:43.561313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604391+0000","last_change":"2024-03-06T07:27:42.105460+0000","last_active":"2024-03-06T07:27:43.604391+0000","last_peered":"2024-03-06T07:27:43.604391+0000","last_clean":"2024-03-06T07:27:43.604391+0000","last_became_active":"2024-03-06T07:27:42.103643+0000","last_became_peered":"2024-03-06T07:27:42.103643+0000","last_unstale":"2024-03-06T07:27:43.604391+0000","last_undegraded":"2024-03-06T07:27:43.604391+0000","last_fullsized":"2024-03-06T07:27:43.604391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264480+0000","last_change":"2024-03-06T07:27:42.094539+0000","last_active":"2024-03-06T07:27:43.264480+0000","last_peered":"2024-03-06T07:27:43.264480+0000","last_clean":"2024-03-06T07:27:43.264480+0000","last_became_active":"2024-03-06T07:27:42.091758+0000","last_became_peered":"2024-03-06T07:27:42.091758+0000","last_unstale":"2024-03-06T07:27:43.264480+0000","last_undegraded":"2024-03-06T07:27:43.264480+0000","last_fullsized":"2024-03-06T07:27:43.264480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604546+0000","last_change":"2024-03-06T07:27:42.113737+0000","last_active":"2024-03-06T07:27:43.604546+0000","last_peered":"2024-03-06T07:27:43.604546+0000","last_clean":"2024-03-06T07:27:43.604546+0000","last_became_active":"2024-03-06T07:27:42.113444+0000","last_became_peered":"2024-03-06T07:27:42.113444+0000","last_unstale":"2024-03-06T07:27:43.604546+0000","last_undegraded":"2024-03-06T07:27:43.604546+0000","last_fullsized":"2024-03-06T07:27:43.604546+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265404+0000","last_change":"2024-03-06T07:27:42.104966+0000","last_active":"2024-03-06T07:27:43.265404+0000","last_peered":"2024-03-06T07:27:43.265404+0000","last_clean":"2024-03-06T07:27:43.265404+0000","last_became_active":"2024-03-06T07:27:42.104220+0000","last_became_peered":"2024-03-06T07:27:42.104220+0000","last_unstale":"2024-03-06T07:27:43.265404+0000","last_undegraded":"2024-03-06T07:27:43.265404+0000","last_fullsized":"2024-03-06T07:27:43.265404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096684+0000","last_change":"2024-03-06T07:27:42.103398+0000","last_active":"2024-03-06T07:27:43.096684+0000","last_peered":"2024-03-06T07:27:43.096684+0000","last_clean":"2024-03-06T07:27:43.096684+0000","last_became_active":"2024-03-06T07:27:42.102707+0000","last_became_peered":"2024-03-06T07:27:42.102707+0000","last_unstale":"2024-03-06T07:27:43.096684+0000","last_undegraded":"2024-03-06T07:27:43.096684+0000","last_fullsized":"2024-03-06T07:27:43.096684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090325+0000","last_change":"2024-03-06T07:27:42.092577+0000","last_active":"2024-03-06T07:27:43.090325+0000","last_peered":"2024-03-06T07:27:43.090325+0000","last_clean":"2024-03-06T07:27:43.090325+0000","last_became_active":"2024-03-06T07:27:42.092269+0000","last_became_peered":"2024-03-06T07:27:42.092269+0000","last_unstale":"2024-03-06T07:27:43.090325+0000","last_undegraded":"2024-03-06T07:27:43.090325+0000","last_fullsized":"2024-03-06T07:27:43.090325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265145+0000","last_change":"2024-03-06T07:27:42.105501+0000","last_active":"2024-03-06T07:27:43.265145+0000","last_peered":"2024-03-06T07:27:43.265145+0000","last_clean":"2024-03-06T07:27:43.265145+0000","last_became_active":"2024-03-06T07:27:42.100336+0000","last_became_peered":"2024-03-06T07:27:42.100336+0000","last_unstale":"2024-03-06T07:27:43.265145+0000","last_undegraded":"2024-03-06T07:27:43.265145+0000","last_fullsized":"2024-03-06T07:27:43.265145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561748+0000","last_change":"2024-03-06T07:27:42.090601+0000","last_active":"2024-03-06T07:27:43.561748+0000","last_peered":"2024-03-06T07:27:43.561748+0000","last_clean":"2024-03-06T07:27:43.561748+0000","last_became_active":"2024-03-06T07:27:42.090283+0000","last_became_peered":"2024-03-06T07:27:42.090283+0000","last_unstale":"2024-03-06T07:27:43.561748+0000","last_undegraded":"2024-03-06T07:27:43.561748+0000","last_fullsized":"2024-03-06T07:27:43.561748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604604+0000","last_change":"2024-03-06T07:27:42.113977+0000","last_active":"2024-03-06T07:27:43.604604+0000","last_peered":"2024-03-06T07:27:43.604604+0000","last_clean":"2024-03-06T07:27:43.604604+0000","last_became_active":"2024-03-06T07:27:42.111520+0000","last_became_peered":"2024-03-06T07:27:42.111520+0000","last_unstale":"2024-03-06T07:27:43.604604+0000","last_undegraded":"2024-03-06T07:27:43.604604+0000","last_fullsized":"2024-03-06T07:27:43.604604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093842+0000","last_change":"2024-03-06T07:27:42.096735+0000","last_active":"2024-03-06T07:27:43.093842+0000","last_peered":"2024-03-06T07:27:43.093842+0000","last_clean":"2024-03-06T07:27:43.093842+0000","last_became_active":"2024-03-06T07:27:42.096304+0000","last_became_peered":"2024-03-06T07:27:42.096304+0000","last_unstale":"2024-03-06T07:27:43.093842+0000","last_undegraded":"2024-03-06T07:27:43.093842+0000","last_fullsized":"2024-03-06T07:27:43.093842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:37.292940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095982+0000","last_change":"2024-03-06T07:27:42.096332+0000","last_active":"2024-03-06T07:27:43.095982+0000","last_peered":"2024-03-06T07:27:43.095982+0000","last_clean":"2024-03-06T07:27:43.095982+0000","last_became_active":"2024-03-06T07:27:42.095984+0000","last_became_peered":"2024-03-06T07:27:42.095984+0000","last_unstale":"2024-03-06T07:27:43.095982+0000","last_undegraded":"2024-03-06T07:27:43.095982+0000","last_fullsized":"2024-03-06T07:27:43.095982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.433842+0000","last_change":"2024-03-06T07:27:42.091042+0000","last_active":"2024-03-06T07:27:43.433842+0000","last_peered":"2024-03-06T07:27:43.433842+0000","last_clean":"2024-03-06T07:27:43.433842+0000","last_became_active":"2024-03-06T07:27:42.089637+0000","last_became_peered":"2024-03-06T07:27:42.089637+0000","last_unstale":"2024-03-06T07:27:43.433842+0000","last_undegraded":"2024-03-06T07:27:43.433842+0000","last_fullsized":"2024-03-06T07:27:43.433842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:00.218649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265208+0000","last_change":"2024-03-06T07:27:41.111941+0000","last_active":"2024-03-06T07:27:43.265208+0000","last_peered":"2024-03-06T07:27:43.265208+0000","last_clean":"2024-03-06T07:27:43.265208+0000","last_became_active":"2024-03-06T07:27:41.111244+0000","last_became_peered":"2024-03-06T07:27:41.111244+0000","last_unstale":"2024-03-06T07:27:43.265208+0000","last_undegraded":"2024-03-06T07:27:43.265208+0000","last_fullsized":"2024-03-06T07:27:43.265208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561659+0000","last_change":"2024-03-06T07:27:42.098732+0000","last_active":"2024-03-06T07:27:43.561659+0000","last_peered":"2024-03-06T07:27:43.561659+0000","last_clean":"2024-03-06T07:27:43.561659+0000","last_became_active":"2024-03-06T07:27:42.098451+0000","last_became_peered":"2024-03-06T07:27:42.098451+0000","last_unstale":"2024-03-06T07:27:43.561659+0000","last_undegraded":"2024-03-06T07:27:43.561659+0000","last_fullsized":"2024-03-06T07:27:43.561659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095423+0000","last_change":"2024-03-06T07:27:41.102703+0000","last_active":"2024-03-06T07:27:43.095423+0000","last_peered":"2024-03-06T07:27:43.095423+0000","last_clean":"2024-03-06T07:27:43.095423+0000","last_became_active":"2024-03-06T07:27:41.100059+0000","last_became_peered":"2024-03-06T07:27:41.100059+0000","last_unstale":"2024-03-06T07:27:43.095423+0000","last_undegraded":"2024-03-06T07:27:43.095423+0000","last_fullsized":"2024-03-06T07:27:43.095423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264571+0000","last_change":"2024-03-06T07:27:41.105849+0000","last_active":"2024-03-06T07:27:43.264571+0000","last_peered":"2024-03-06T07:27:43.264571+0000","last_clean":"2024-03-06T07:27:43.264571+0000","last_became_active":"2024-03-06T07:27:41.105558+0000","last_became_peered":"2024-03-06T07:27:41.105558+0000","last_unstale":"2024-03-06T07:27:43.264571+0000","last_undegraded":"2024-03-06T07:27:43.264571+0000","last_fullsized":"2024-03-06T07:27:43.264571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264534+0000","last_change":"2024-03-06T07:27:42.098554+0000","last_active":"2024-03-06T07:27:43.264534+0000","last_peered":"2024-03-06T07:27:43.264534+0000","last_clean":"2024-03-06T07:27:43.264534+0000","last_became_active":"2024-03-06T07:27:42.097364+0000","last_became_peered":"2024-03-06T07:27:42.097364+0000","last_unstale":"2024-03-06T07:27:43.264534+0000","last_undegraded":"2024-03-06T07:27:43.264534+0000","last_fullsized":"2024-03-06T07:27:43.264534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604701+0000","last_change":"2024-03-06T07:27:41.103209+0000","last_active":"2024-03-06T07:27:43.604701+0000","last_peered":"2024-03-06T07:27:43.604701+0000","last_clean":"2024-03-06T07:27:43.604701+0000","last_became_active":"2024-03-06T07:27:41.102994+0000","last_became_peered":"2024-03-06T07:27:41.102994+0000","last_unstale":"2024-03-06T07:27:43.604701+0000","last_undegraded":"2024-03-06T07:27:43.604701+0000","last_fullsized":"2024-03-06T07:27:43.604701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090178+0000","last_change":"2024-03-06T07:27:42.086153+0000","last_active":"2024-03-06T07:27:43.090178+0000","last_peered":"2024-03-06T07:27:43.090178+0000","last_clean":"2024-03-06T07:27:43.090178+0000","last_became_active":"2024-03-06T07:27:42.085779+0000","last_became_peered":"2024-03-06T07:27:42.085779+0000","last_unstale":"2024-03-06T07:27:43.090178+0000","last_undegraded":"2024-03-06T07:27:43.090178+0000","last_fullsized":"2024-03-06T07:27:43.090178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096443+0000","last_change":"2024-03-06T07:27:41.109144+0000","last_active":"2024-03-06T07:27:43.096443+0000","last_peered":"2024-03-06T07:27:43.096443+0000","last_clean":"2024-03-06T07:27:43.096443+0000","last_became_active":"2024-03-06T07:27:41.108399+0000","last_became_peered":"2024-03-06T07:27:41.108399+0000","last_unstale":"2024-03-06T07:27:43.096443+0000","last_undegraded":"2024-03-06T07:27:43.096443+0000","last_fullsized":"2024-03-06T07:27:43.096443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604733+0000","last_change":"2024-03-06T07:27:42.106823+0000","last_active":"2024-03-06T07:27:43.604733+0000","last_peered":"2024-03-06T07:27:43.604733+0000","last_clean":"2024-03-06T07:27:43.604733+0000","last_became_active":"2024-03-06T07:27:42.103949+0000","last_became_peered":"2024-03-06T07:27:42.103949+0000","last_unstale":"2024-03-06T07:27:43.604733+0000","last_undegraded":"2024-03-06T07:27:43.604733+0000","last_fullsized":"2024-03-06T07:27:43.604733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605002+0000","last_change":"2024-03-06T07:27:41.108455+0000","last_active":"2024-03-06T07:27:43.605002+0000","last_peered":"2024-03-06T07:27:43.605002+0000","last_clean":"2024-03-06T07:27:43.605002+0000","last_became_active":"2024-03-06T07:27:41.106095+0000","last_became_peered":"2024-03-06T07:27:41.106095+0000","last_unstale":"2024-03-06T07:27:43.605002+0000","last_undegraded":"2024-03-06T07:27:43.605002+0000","last_fullsized":"2024-03-06T07:27:43.605002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561945+0000","last_change":"2024-03-06T07:27:42.106635+0000","last_active":"2024-03-06T07:27:43.561945+0000","last_peered":"2024-03-06T07:27:43.561945+0000","last_clean":"2024-03-06T07:27:43.561945+0000","last_became_active":"2024-03-06T07:27:42.106373+0000","last_became_peered":"2024-03-06T07:27:42.106373+0000","last_unstale":"2024-03-06T07:27:43.561945+0000","last_undegraded":"2024-03-06T07:27:43.561945+0000","last_fullsized":"2024-03-06T07:27:43.561945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090296+0000","last_change":"2024-03-06T07:27:41.108679+0000","last_active":"2024-03-06T07:27:43.090296+0000","last_peered":"2024-03-06T07:27:43.090296+0000","last_clean":"2024-03-06T07:27:43.090296+0000","last_became_active":"2024-03-06T07:27:41.108429+0000","last_became_peered":"2024-03-06T07:27:41.108429+0000","last_unstale":"2024-03-06T07:27:43.090296+0000","last_undegraded":"2024-03-06T07:27:43.090296+0000","last_fullsized":"2024-03-06T07:27:43.090296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434363+0000","last_change":"2024-03-06T07:27:42.102682+0000","last_active":"2024-03-06T07:27:43.434363+0000","last_peered":"2024-03-06T07:27:43.434363+0000","last_clean":"2024-03-06T07:27:43.434363+0000","last_became_active":"2024-03-06T07:27:42.102232+0000","last_became_peered":"2024-03-06T07:27:42.102232+0000","last_unstale":"2024-03-06T07:27:43.434363+0000","last_undegraded":"2024-03-06T07:27:43.434363+0000","last_fullsized":"2024-03-06T07:27:43.434363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:34.589802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096130+0000","last_change":"2024-03-06T07:27:41.096688+0000","last_active":"2024-03-06T07:27:43.096130+0000","last_peered":"2024-03-06T07:27:43.096130+0000","last_clean":"2024-03-06T07:27:43.096130+0000","last_became_active":"2024-03-06T07:27:41.096391+0000","last_became_peered":"2024-03-06T07:27:41.096391+0000","last_unstale":"2024-03-06T07:27:43.096130+0000","last_undegraded":"2024-03-06T07:27:43.096130+0000","last_fullsized":"2024-03-06T07:27:43.096130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668491+0000","last_change":"2024-03-06T07:27:42.116169+0000","last_active":"2024-03-06T07:27:43.668491+0000","last_peered":"2024-03-06T07:27:43.668491+0000","last_clean":"2024-03-06T07:27:43.668491+0000","last_became_active":"2024-03-06T07:27:42.115868+0000","last_became_peered":"2024-03-06T07:27:42.115868+0000","last_unstale":"2024-03-06T07:27:43.668491+0000","last_undegraded":"2024-03-06T07:27:43.668491+0000","last_fullsized":"2024-03-06T07:27:43.668491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.091480+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.090511+0000","last_became_peered":"2024-03-06T07:27:41.090511+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561169+0000","last_change":"2024-03-06T07:27:42.104948+0000","last_active":"2024-03-06T07:27:43.561169+0000","last_peered":"2024-03-06T07:27:43.561169+0000","last_clean":"2024-03-06T07:27:43.561169+0000","last_became_active":"2024-03-06T07:27:42.104640+0000","last_became_peered":"2024-03-06T07:27:42.104640+0000","last_unstale":"2024-03-06T07:27:43.561169+0000","last_undegraded":"2024-03-06T07:27:43.561169+0000","last_fullsized":"2024-03-06T07:27:43.561169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604876+0000","last_change":"2024-03-06T07:27:41.103424+0000","last_active":"2024-03-06T07:27:43.604876+0000","last_peered":"2024-03-06T07:27:43.604876+0000","last_clean":"2024-03-06T07:27:43.604876+0000","last_became_active":"2024-03-06T07:27:41.102084+0000","last_became_peered":"2024-03-06T07:27:41.102084+0000","last_unstale":"2024-03-06T07:27:43.604876+0000","last_undegraded":"2024-03-06T07:27:43.604876+0000","last_fullsized":"2024-03-06T07:27:43.604876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668345+0000","last_change":"2024-03-06T07:27:42.107947+0000","last_active":"2024-03-06T07:27:43.668345+0000","last_peered":"2024-03-06T07:27:43.668345+0000","last_clean":"2024-03-06T07:27:43.668345+0000","last_became_active":"2024-03-06T07:27:42.107570+0000","last_became_peered":"2024-03-06T07:27:42.107570+0000","last_unstale":"2024-03-06T07:27:43.668345+0000","last_undegraded":"2024-03-06T07:27:43.668345+0000","last_fullsized":"2024-03-06T07:27:43.668345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095592+0000","last_change":"2024-03-06T07:27:41.105127+0000","last_active":"2024-03-06T07:27:43.095592+0000","last_peered":"2024-03-06T07:27:43.095592+0000","last_clean":"2024-03-06T07:27:43.095592+0000","last_became_active":"2024-03-06T07:27:41.104134+0000","last_became_peered":"2024-03-06T07:27:41.104134+0000","last_unstale":"2024-03-06T07:27:43.095592+0000","last_undegraded":"2024-03-06T07:27:43.095592+0000","last_fullsized":"2024-03-06T07:27:43.095592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096137+0000","last_change":"2024-03-06T07:27:42.104623+0000","last_active":"2024-03-06T07:27:43.096137+0000","last_peered":"2024-03-06T07:27:43.096137+0000","last_clean":"2024-03-06T07:27:43.096137+0000","last_became_active":"2024-03-06T07:27:42.104185+0000","last_became_peered":"2024-03-06T07:27:42.104185+0000","last_unstale":"2024-03-06T07:27:43.096137+0000","last_undegraded":"2024-03-06T07:27:43.096137+0000","last_fullsized":"2024-03-06T07:27:43.096137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093473+0000","last_change":"2024-03-06T07:27:41.095963+0000","last_active":"2024-03-06T07:27:43.093473+0000","last_peered":"2024-03-06T07:27:43.093473+0000","last_clean":"2024-03-06T07:27:43.093473+0000","last_became_active":"2024-03-06T07:27:41.095610+0000","last_became_peered":"2024-03-06T07:27:41.095610+0000","last_unstale":"2024-03-06T07:27:43.093473+0000","last_undegraded":"2024-03-06T07:27:43.093473+0000","last_fullsized":"2024-03-06T07:27:43.093473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668062+0000","last_change":"2024-03-06T07:27:42.107796+0000","last_active":"2024-03-06T07:27:43.668062+0000","last_peered":"2024-03-06T07:27:43.668062+0000","last_clean":"2024-03-06T07:27:43.668062+0000","last_became_active":"2024-03-06T07:27:42.107514+0000","last_became_peered":"2024-03-06T07:27:42.107514+0000","last_unstale":"2024-03-06T07:27:43.668062+0000","last_undegraded":"2024-03-06T07:27:43.668062+0000","last_fullsized":"2024-03-06T07:27:43.668062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434682+0000","last_change":"2024-03-06T07:27:41.094350+0000","last_active":"2024-03-06T07:27:43.434682+0000","last_peered":"2024-03-06T07:27:43.434682+0000","last_clean":"2024-03-06T07:27:43.434682+0000","last_became_active":"2024-03-06T07:27:41.089935+0000","last_became_peered":"2024-03-06T07:27:41.089935+0000","last_unstale":"2024-03-06T07:27:43.434682+0000","last_undegraded":"2024-03-06T07:27:43.434682+0000","last_fullsized":"2024-03-06T07:27:43.434682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:49.952499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669175+0000","last_change":"2024-03-06T07:27:42.109835+0000","last_active":"2024-03-06T07:27:43.669175+0000","last_peered":"2024-03-06T07:27:43.669175+0000","last_clean":"2024-03-06T07:27:43.669175+0000","last_became_active":"2024-03-06T07:27:42.109535+0000","last_became_peered":"2024-03-06T07:27:42.109535+0000","last_unstale":"2024-03-06T07:27:43.669175+0000","last_undegraded":"2024-03-06T07:27:43.669175+0000","last_fullsized":"2024-03-06T07:27:43.669175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093784+0000","last_change":"2024-03-06T07:27:41.092525+0000","last_active":"2024-03-06T07:27:43.093784+0000","last_peered":"2024-03-06T07:27:43.093784+0000","last_clean":"2024-03-06T07:27:43.093784+0000","last_became_active":"2024-03-06T07:27:41.091644+0000","last_became_peered":"2024-03-06T07:27:41.091644+0000","last_unstale":"2024-03-06T07:27:43.093784+0000","last_undegraded":"2024-03-06T07:27:43.093784+0000","last_fullsized":"2024-03-06T07:27:43.093784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.670064+0000","last_change":"2024-03-06T07:27:42.117934+0000","last_active":"2024-03-06T07:27:43.670064+0000","last_peered":"2024-03-06T07:27:43.670064+0000","last_clean":"2024-03-06T07:27:43.670064+0000","last_became_active":"2024-03-06T07:27:42.115924+0000","last_became_peered":"2024-03-06T07:27:42.115924+0000","last_unstale":"2024-03-06T07:27:43.670064+0000","last_undegraded":"2024-03-06T07:27:43.670064+0000","last_fullsized":"2024-03-06T07:27:43.670064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:39.844169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090372+0000","last_change":"2024-03-06T07:27:41.097118+0000","last_active":"2024-03-06T07:27:43.090372+0000","last_peered":"2024-03-06T07:27:43.090372+0000","last_clean":"2024-03-06T07:27:43.090372+0000","last_became_active":"2024-03-06T07:27:41.096188+0000","last_became_peered":"2024-03-06T07:27:41.096188+0000","last_unstale":"2024-03-06T07:27:43.090372+0000","last_undegraded":"2024-03-06T07:27:43.090372+0000","last_fullsized":"2024-03-06T07:27:43.090372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605208+0000","last_change":"2024-03-06T07:27:42.114130+0000","last_active":"2024-03-06T07:27:43.605208+0000","last_peered":"2024-03-06T07:27:43.605208+0000","last_clean":"2024-03-06T07:27:43.605208+0000","last_became_active":"2024-03-06T07:27:42.113811+0000","last_became_peered":"2024-03-06T07:27:42.113811+0000","last_unstale":"2024-03-06T07:27:43.605208+0000","last_undegraded":"2024-03-06T07:27:43.605208+0000","last_fullsized":"2024-03-06T07:27:43.605208+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093668+0000","last_change":"2024-03-06T07:27:41.102837+0000","last_active":"2024-03-06T07:27:43.093668+0000","last_peered":"2024-03-06T07:27:43.093668+0000","last_clean":"2024-03-06T07:27:43.093668+0000","last_became_active":"2024-03-06T07:27:41.102526+0000","last_became_peered":"2024-03-06T07:27:41.102526+0000","last_unstale":"2024-03-06T07:27:43.093668+0000","last_undegraded":"2024-03-06T07:27:43.093668+0000","last_fullsized":"2024-03-06T07:27:43.093668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668936+0000","last_change":"2024-03-06T07:27:42.110688+0000","last_active":"2024-03-06T07:27:43.668936+0000","last_peered":"2024-03-06T07:27:43.668936+0000","last_clean":"2024-03-06T07:27:43.668936+0000","last_became_active":"2024-03-06T07:27:42.110394+0000","last_became_peered":"2024-03-06T07:27:42.110394+0000","last_unstale":"2024-03-06T07:27:43.668936+0000","last_undegraded":"2024-03-06T07:27:43.668936+0000","last_fullsized":"2024-03-06T07:27:43.668936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667970+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:27:43.667970+0000","last_peered":"2024-03-06T07:27:43.667970+0000","last_clean":"2024-03-06T07:27:43.667970+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:27:43.667970+0000","last_undegraded":"2024-03-06T07:27:43.667970+0000","last_fullsized":"2024-03-06T07:27:43.667970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604170+0000","last_change":"2024-03-06T07:27:41.111128+0000","last_active":"2024-03-06T07:27:43.604170+0000","last_peered":"2024-03-06T07:27:43.604170+0000","last_clean":"2024-03-06T07:27:43.604170+0000","last_became_active":"2024-03-06T07:27:41.109382+0000","last_became_peered":"2024-03-06T07:27:41.109382+0000","last_unstale":"2024-03-06T07:27:43.604170+0000","last_undegraded":"2024-03-06T07:27:43.604170+0000","last_fullsized":"2024-03-06T07:27:43.604170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264748+0000","last_change":"2024-03-06T07:27:42.100303+0000","last_active":"2024-03-06T07:27:43.264748+0000","last_peered":"2024-03-06T07:27:43.264748+0000","last_clean":"2024-03-06T07:27:43.264748+0000","last_became_active":"2024-03-06T07:27:42.099977+0000","last_became_peered":"2024-03-06T07:27:42.099977+0000","last_unstale":"2024-03-06T07:27:43.264748+0000","last_undegraded":"2024-03-06T07:27:43.264748+0000","last_fullsized":"2024-03-06T07:27:43.264748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667817+0000","last_change":"2024-03-06T07:27:41.113323+0000","last_active":"2024-03-06T07:27:43.667817+0000","last_peered":"2024-03-06T07:27:43.667817+0000","last_clean":"2024-03-06T07:27:43.667817+0000","last_became_active":"2024-03-06T07:27:41.112921+0000","last_became_peered":"2024-03-06T07:27:41.112921+0000","last_unstale":"2024-03-06T07:27:43.667817+0000","last_undegraded":"2024-03-06T07:27:43.667817+0000","last_fullsized":"2024-03-06T07:27:43.667817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:12.710184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561173+0000","last_change":"2024-03-06T07:27:42.098275+0000","last_active":"2024-03-06T07:27:43.561173+0000","last_peered":"2024-03-06T07:27:43.561173+0000","last_clean":"2024-03-06T07:27:43.561173+0000","last_became_active":"2024-03-06T07:27:42.097430+0000","last_became_peered":"2024-03-06T07:27:42.097430+0000","last_unstale":"2024-03-06T07:27:43.561173+0000","last_undegraded":"2024-03-06T07:27:43.561173+0000","last_fullsized":"2024-03-06T07:27:43.561173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:43.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093787+0000","last_change":"2024-03-06T07:27:41.092149+0000","last_active":"2024-03-06T07:27:43.093787+0000","last_peered":"2024-03-06T07:27:43.093787+0000","last_clean":"2024-03-06T07:27:43.093787+0000","last_became_active":"2024-03-06T07:27:41.090942+0000","last_became_peered":"2024-03-06T07:27:41.090942+0000","last_unstale":"2024-03-06T07:27:43.093787+0000","last_undegraded":"2024-03-06T07:27:43.093787+0000","last_fullsized":"2024-03-06T07:27:43.093787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089953+0000","last_change":"2024-03-06T07:27:42.103118+0000","last_active":"2024-03-06T07:27:43.089953+0000","last_peered":"2024-03-06T07:27:43.089953+0000","last_clean":"2024-03-06T07:27:43.089953+0000","last_became_active":"2024-03-06T07:27:42.102850+0000","last_became_peered":"2024-03-06T07:27:42.102850+0000","last_unstale":"2024-03-06T07:27:43.089953+0000","last_undegraded":"2024-03-06T07:27:43.089953+0000","last_fullsized":"2024-03-06T07:27:43.089953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605179+0000","last_change":"2024-03-06T07:27:41.111452+0000","last_active":"2024-03-06T07:27:43.605179+0000","last_peered":"2024-03-06T07:27:43.605179+0000","last_clean":"2024-03-06T07:27:43.605179+0000","last_became_active":"2024-03-06T07:27:41.110359+0000","last_became_peered":"2024-03-06T07:27:41.110359+0000","last_unstale":"2024-03-06T07:27:43.605179+0000","last_undegraded":"2024-03-06T07:27:43.605179+0000","last_fullsized":"2024-03-06T07:27:43.605179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434187+0000","last_change":"2024-03-06T07:27:42.093244+0000","last_active":"2024-03-06T07:27:43.434187+0000","last_peered":"2024-03-06T07:27:43.434187+0000","last_clean":"2024-03-06T07:27:43.434187+0000","last_became_active":"2024-03-06T07:27:42.092228+0000","last_became_peered":"2024-03-06T07:27:42.092228+0000","last_unstale":"2024-03-06T07:27:43.434187+0000","last_undegraded":"2024-03-06T07:27:43.434187+0000","last_fullsized":"2024-03-06T07:27:43.434187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:07.427898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090250+0000","last_change":"2024-03-06T07:27:41.093905+0000","last_active":"2024-03-06T07:27:43.090250+0000","last_peered":"2024-03-06T07:27:43.090250+0000","last_clean":"2024-03-06T07:27:43.090250+0000","last_became_active":"2024-03-06T07:27:41.093456+0000","last_became_peered":"2024-03-06T07:27:41.093456+0000","last_unstale":"2024-03-06T07:27:43.090250+0000","last_undegraded":"2024-03-06T07:27:43.090250+0000","last_fullsized":"2024-03-06T07:27:43.090250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605312+0000","last_change":"2024-03-06T07:27:42.112370+0000","last_active":"2024-03-06T07:27:43.605312+0000","last_peered":"2024-03-06T07:27:43.605312+0000","last_clean":"2024-03-06T07:27:43.605312+0000","last_became_active":"2024-03-06T07:27:42.112087+0000","last_became_peered":"2024-03-06T07:27:42.112087+0000","last_unstale":"2024-03-06T07:27:43.605312+0000","last_undegraded":"2024-03-06T07:27:43.605312+0000","last_fullsized":"2024-03-06T07:27:43.605312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669326+0000","last_change":"2024-03-06T07:27:41.116636+0000","last_active":"2024-03-06T07:27:43.669326+0000","last_peered":"2024-03-06T07:27:43.669326+0000","last_clean":"2024-03-06T07:27:43.669326+0000","last_became_active":"2024-03-06T07:27:41.116254+0000","last_became_peered":"2024-03-06T07:27:41.116254+0000","last_unstale":"2024-03-06T07:27:43.669326+0000","last_undegraded":"2024-03-06T07:27:43.669326+0000","last_fullsized":"2024-03-06T07:27:43.669326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:59.546367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090070+0000","last_change":"2024-03-06T07:27:42.094877+0000","last_active":"2024-03-06T07:27:43.090070+0000","last_peered":"2024-03-06T07:27:43.090070+0000","last_clean":"2024-03-06T07:27:43.090070+0000","last_became_active":"2024-03-06T07:27:42.094586+0000","last_became_peered":"2024-03-06T07:27:42.094586+0000","last_unstale":"2024-03-06T07:27:43.090070+0000","last_undegraded":"2024-03-06T07:27:43.090070+0000","last_fullsized":"2024-03-06T07:27:43.090070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.102122+0000","last_change":"2024-03-06T07:27:41.098449+0000","last_active":"2024-03-06T07:27:43.102122+0000","last_peered":"2024-03-06T07:27:43.102122+0000","last_clean":"2024-03-06T07:27:43.102122+0000","last_became_active":"2024-03-06T07:27:41.098137+0000","last_became_peered":"2024-03-06T07:27:41.098137+0000","last_unstale":"2024-03-06T07:27:43.102122+0000","last_undegraded":"2024-03-06T07:27:43.102122+0000","last_fullsized":"2024-03-06T07:27:43.102122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265545+0000","last_change":"2024-03-06T07:27:42.118543+0000","last_active":"2024-03-06T07:27:43.265545+0000","last_peered":"2024-03-06T07:27:43.265545+0000","last_clean":"2024-03-06T07:27:43.265545+0000","last_became_active":"2024-03-06T07:27:42.118157+0000","last_became_peered":"2024-03-06T07:27:42.118157+0000","last_unstale":"2024-03-06T07:27:43.265545+0000","last_undegraded":"2024-03-06T07:27:43.265545+0000","last_fullsized":"2024-03-06T07:27:43.265545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096632+0000","last_change":"2024-03-06T07:27:41.112840+0000","last_active":"2024-03-06T07:27:43.096632+0000","last_peered":"2024-03-06T07:27:43.096632+0000","last_clean":"2024-03-06T07:27:43.096632+0000","last_became_active":"2024-03-06T07:27:41.112440+0000","last_became_peered":"2024-03-06T07:27:41.112440+0000","last_unstale":"2024-03-06T07:27:43.096632+0000","last_undegraded":"2024-03-06T07:27:43.096632+0000","last_fullsized":"2024-03-06T07:27:43.096632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561550+0000","last_change":"2024-03-06T07:27:42.090517+0000","last_active":"2024-03-06T07:27:43.561550+0000","last_peered":"2024-03-06T07:27:43.561550+0000","last_clean":"2024-03-06T07:27:43.561550+0000","last_became_active":"2024-03-06T07:27:42.090202+0000","last_became_peered":"2024-03-06T07:27:42.090202+0000","last_unstale":"2024-03-06T07:27:43.561550+0000","last_undegraded":"2024-03-06T07:27:43.561550+0000","last_fullsized":"2024-03-06T07:27:43.561550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667842+0000","last_change":"2024-03-06T07:27:41.114447+0000","last_active":"2024-03-06T07:27:43.667842+0000","last_peered":"2024-03-06T07:27:43.667842+0000","last_clean":"2024-03-06T07:27:43.667842+0000","last_became_active":"2024-03-06T07:27:41.113802+0000","last_became_peered":"2024-03-06T07:27:41.113802+0000","last_unstale":"2024-03-06T07:27:43.667842+0000","last_undegraded":"2024-03-06T07:27:43.667842+0000","last_fullsized":"2024-03-06T07:27:43.667842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:43.723690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434174+0000","last_change":"2024-03-06T07:27:42.110210+0000","last_active":"2024-03-06T07:27:43.434174+0000","last_peered":"2024-03-06T07:27:43.434174+0000","last_clean":"2024-03-06T07:27:43.434174+0000","last_became_active":"2024-03-06T07:27:42.109736+0000","last_became_peered":"2024-03-06T07:27:42.109736+0000","last_unstale":"2024-03-06T07:27:43.434174+0000","last_undegraded":"2024-03-06T07:27:43.434174+0000","last_fullsized":"2024-03-06T07:27:43.434174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:25.704825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090026+0000","last_change":"2024-03-06T07:27:41.101111+0000","last_active":"2024-03-06T07:27:43.090026+0000","last_peered":"2024-03-06T07:27:43.090026+0000","last_clean":"2024-03-06T07:27:43.090026+0000","last_became_active":"2024-03-06T07:27:41.100857+0000","last_became_peered":"2024-03-06T07:27:41.100857+0000","last_unstale":"2024-03-06T07:27:43.090026+0000","last_undegraded":"2024-03-06T07:27:43.090026+0000","last_fullsized":"2024-03-06T07:27:43.090026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264697+0000","last_change":"2024-03-06T07:27:42.086202+0000","last_active":"2024-03-06T07:27:43.264697+0000","last_peered":"2024-03-06T07:27:43.264697+0000","last_clean":"2024-03-06T07:27:43.264697+0000","last_became_active":"2024-03-06T07:27:42.085754+0000","last_became_peered":"2024-03-06T07:27:42.085754+0000","last_unstale":"2024-03-06T07:27:43.264697+0000","last_undegraded":"2024-03-06T07:27:43.264697+0000","last_fullsized":"2024-03-06T07:27:43.264697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090044+0000","last_change":"2024-03-06T07:27:41.096584+0000","last_active":"2024-03-06T07:27:43.090044+0000","last_peered":"2024-03-06T07:27:43.090044+0000","last_clean":"2024-03-06T07:27:43.090044+0000","last_became_active":"2024-03-06T07:27:41.096318+0000","last_became_peered":"2024-03-06T07:27:41.096318+0000","last_unstale":"2024-03-06T07:27:43.090044+0000","last_undegraded":"2024-03-06T07:27:43.090044+0000","last_fullsized":"2024-03-06T07:27:43.090044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265099+0000","last_change":"2024-03-06T07:27:42.102269+0000","last_active":"2024-03-06T07:27:43.265099+0000","last_peered":"2024-03-06T07:27:43.265099+0000","last_clean":"2024-03-06T07:27:43.265099+0000","last_became_active":"2024-03-06T07:27:42.092816+0000","last_became_peered":"2024-03-06T07:27:42.092816+0000","last_unstale":"2024-03-06T07:27:43.265099+0000","last_undegraded":"2024-03-06T07:27:43.265099+0000","last_fullsized":"2024-03-06T07:27:43.265099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668665+0000","last_change":"2024-03-06T07:27:41.116372+0000","last_active":"2024-03-06T07:27:43.668665+0000","last_peered":"2024-03-06T07:27:43.668665+0000","last_clean":"2024-03-06T07:27:43.668665+0000","last_became_active":"2024-03-06T07:27:41.116072+0000","last_became_peered":"2024-03-06T07:27:41.116072+0000","last_unstale":"2024-03-06T07:27:43.668665+0000","last_undegraded":"2024-03-06T07:27:43.668665+0000","last_fullsized":"2024-03-06T07:27:43.668665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095884+0000","last_change":"2024-03-06T07:27:42.103012+0000","last_active":"2024-03-06T07:27:43.095884+0000","last_peered":"2024-03-06T07:27:43.095884+0000","last_clean":"2024-03-06T07:27:43.095884+0000","last_became_active":"2024-03-06T07:27:42.102731+0000","last_became_peered":"2024-03-06T07:27:42.102731+0000","last_unstale":"2024-03-06T07:27:43.095884+0000","last_undegraded":"2024-03-06T07:27:43.095884+0000","last_fullsized":"2024-03-06T07:27:43.095884+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604917+0000","last_change":"2024-03-06T07:27:42.097462+0000","last_active":"2024-03-06T07:27:43.604917+0000","last_peered":"2024-03-06T07:27:43.604917+0000","last_clean":"2024-03-06T07:27:43.604917+0000","last_became_active":"2024-03-06T07:27:42.097142+0000","last_became_peered":"2024-03-06T07:27:42.097142+0000","last_unstale":"2024-03-06T07:27:43.604917+0000","last_undegraded":"2024-03-06T07:27:43.604917+0000","last_fullsized":"2024-03-06T07:27:43.604917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668754+0000","last_change":"2024-03-06T07:27:41.115663+0000","last_active":"2024-03-06T07:27:43.668754+0000","last_peered":"2024-03-06T07:27:43.668754+0000","last_clean":"2024-03-06T07:27:43.668754+0000","last_became_active":"2024-03-06T07:27:41.114756+0000","last_became_peered":"2024-03-06T07:27:41.114756+0000","last_unstale":"2024-03-06T07:27:43.668754+0000","last_undegraded":"2024-03-06T07:27:43.668754+0000","last_fullsized":"2024-03-06T07:27:43.668754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604966+0000","last_change":"2024-03-06T07:27:42.098684+0000","last_active":"2024-03-06T07:27:43.604966+0000","last_peered":"2024-03-06T07:27:43.604966+0000","last_clean":"2024-03-06T07:27:43.604966+0000","last_became_active":"2024-03-06T07:27:42.098307+0000","last_became_peered":"2024-03-06T07:27:42.098307+0000","last_unstale":"2024-03-06T07:27:43.604966+0000","last_undegraded":"2024-03-06T07:27:43.604966+0000","last_fullsized":"2024-03-06T07:27:43.604966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669076+0000","last_change":"2024-03-06T07:27:41.117369+0000","last_active":"2024-03-06T07:27:43.669076+0000","last_peered":"2024-03-06T07:27:43.669076+0000","last_clean":"2024-03-06T07:27:43.669076+0000","last_became_active":"2024-03-06T07:27:41.117042+0000","last_became_peered":"2024-03-06T07:27:41.117042+0000","last_unstale":"2024-03-06T07:27:43.669076+0000","last_undegraded":"2024-03-06T07:27:43.669076+0000","last_fullsized":"2024-03-06T07:27:43.669076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561654+0000","last_change":"2024-03-06T07:27:42.106505+0000","last_active":"2024-03-06T07:27:43.561654+0000","last_peered":"2024-03-06T07:27:43.561654+0000","last_clean":"2024-03-06T07:27:43.561654+0000","last_became_active":"2024-03-06T07:27:42.105993+0000","last_became_peered":"2024-03-06T07:27:42.105993+0000","last_unstale":"2024-03-06T07:27:43.561654+0000","last_undegraded":"2024-03-06T07:27:43.561654+0000","last_fullsized":"2024-03-06T07:27:43.561654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265054+0000","last_change":"2024-03-06T07:27:41.097028+0000","last_active":"2024-03-06T07:27:43.265054+0000","last_peered":"2024-03-06T07:27:43.265054+0000","last_clean":"2024-03-06T07:27:43.265054+0000","last_became_active":"2024-03-06T07:27:41.096532+0000","last_became_peered":"2024-03-06T07:27:41.096532+0000","last_unstale":"2024-03-06T07:27:43.265054+0000","last_undegraded":"2024-03-06T07:27:43.265054+0000","last_fullsized":"2024-03-06T07:27:43.265054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264945+0000","last_change":"2024-03-06T07:27:42.094785+0000","last_active":"2024-03-06T07:27:43.264945+0000","last_peered":"2024-03-06T07:27:43.264945+0000","last_clean":"2024-03-06T07:27:43.264945+0000","last_became_active":"2024-03-06T07:27:42.091901+0000","last_became_peered":"2024-03-06T07:27:42.091901+0000","last_unstale":"2024-03-06T07:27:43.264945+0000","last_undegraded":"2024-03-06T07:27:43.264945+0000","last_fullsized":"2024-03-06T07:27:43.264945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.102186+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.101868+0000","last_became_peered":"2024-03-06T07:27:41.101868+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:20.791281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:46.996 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:46.996 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:27:46.997 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:27:46.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:27:46.997 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:27:47.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.238+0000 7fe747400640 1 Processor -- start 2024-03-06T07:27:47.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.238+0000 7fe747400640 1 -- start start 2024-03-06T07:27:47.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe747400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe74015f470 0x7fe7400d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe747400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7400d1200 con 0x7fe74015e0d0 2024-03-06T07:27:47.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe747400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7400cea60 con 0x7fe7400d0df0 2024-03-06T07:27:47.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe747400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7400d2f60 con 0x7fe74015f470 2024-03-06T07:27:47.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe737e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe74015f470 0x7fe7400d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe744a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe74015e0d0 0x7fe7400d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45052/0 (socket says 172.21.15.97:45052) 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe737e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe74015f470 0x7fe7400d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50366/0 (socket says 172.21.15.97:50366) 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.240+0000 7fe744a00640 1 -- 172.21.15.97:0/104964373 learned_addr learned my addr 172.21.15.97:0/104964373 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.241+0000 7fe737e00640 1 -- 172.21.15.97:0/104964373 >> v1:172.21.15.135:6789/0 conn(0x7fe7400d0df0 legacy=0x7fe740160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.241+0000 7fe737e00640 1 -- 172.21.15.97:0/104964373 >> v1:172.21.15.97:6789/0 conn(0x7fe74015e0d0 legacy=0x7fe7400d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.241+0000 7fe737e00640 1 -- 172.21.15.97:0/104964373 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400d2350 con 0x7fe74015f470 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.241+0000 7fe73fe00640 1 -- 172.21.15.97:0/104964373 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321840510 0 0) 0x7fe730001110 con 0x7fe74015e0d0 2024-03-06T07:27:47.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.242+0000 7fe737e00640 1 --2- 172.21.15.97:0/104964373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015f470 0x7fe7400d08b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe7400c77f0 tx=0x7fe72802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e067fc6a143cecf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.242+0000 7fe73fe00640 1 -- 172.21.15.97:0/104964373 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728035020 con 0x7fe74015f470 2024-03-06T07:27:47.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.242+0000 7fe73fe00640 1 -- 172.21.15.97:0/104964373 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe728031890 con 0x7fe74015f470 2024-03-06T07:27:47.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.242+0000 7fe73fe00640 1 -- 172.21.15.97:0/104964373 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728031c60 con 0x7fe74015f470 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 -- 172.21.15.97:0/104964373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015f470 msgr2=0x7fe7400d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 --2- 172.21.15.97:0/104964373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015f470 0x7fe7400d08b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe7400c77f0 tx=0x7fe72802b930 comp rx=0 tx=0).stop 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 -- 172.21.15.97:0/104964373 shutdown_connections 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 --2- 172.21.15.97:0/104964373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015f470 0x7fe7400d08b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 -- 172.21.15.97:0/104964373 >> 172.21.15.97:0/104964373 conn(0x7fe7400d3ec0 msgr2=0x7fe7400d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 -- 172.21.15.97:0/104964373 shutdown_connections 2024-03-06T07:27:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.243+0000 7fe747400640 1 -- 172.21.15.97:0/104964373 wait complete. 2024-03-06T07:27:47.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.244+0000 7fe747400640 1 Processor -- start 2024-03-06T07:27:47.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.245+0000 7fe747400640 1 -- start start 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.245+0000 7fe747400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7400d0df0 0x7fe7400c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe747400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 0x7fe7400b8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe747400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe74015f470 0x7fe7401582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe747400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400d2bf0 con 0x7fe74015e0d0 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe747400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400d2910 con 0x7fe74015f470 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe747400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400cff10 con 0x7fe7400d0df0 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 0x7fe7400b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 0x7fe7400b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50380/0 (socket says 172.21.15.97:50380) 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 -- 172.21.15.97:0/694950256 learned_addr learned my addr 172.21.15.97:0/694950256 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe744a00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7400d0df0 0x7fe7400c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 -- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7400d0df0 msgr2=0x7fe7400c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7400d0df0 0x7fe7400c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 -- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe74015f470 msgr2=0x7fe7401582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe74015f470 0x7fe7401582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe737e00640 1 -- 172.21.15.97:0/694950256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7401f4410 con 0x7fe74015e0d0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.246+0000 7fe744a00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7400d0df0 0x7fe7400c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.247+0000 7fe737e00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 0x7fe7400b8300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe728009890 tx=0x7fe7280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.247+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7280443f0 con 0x7fe74015e0d0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.247+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7280445b0 con 0x7fe74015e0d0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.247+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7401f46f0 con 0x7fe74015e0d0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.247+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7280449a0 con 0x7fe74015e0d0 2024-03-06T07:27:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.247+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7401f4cb0 con 0x7fe74015e0d0 2024-03-06T07:27:47.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.248+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74004f330 con 0x7fe74015e0d0 2024-03-06T07:27:47.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.249+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe728038020 con 0x7fe74015e0d0 2024-03-06T07:27:47.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.250+0000 7fe73d600640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe71c06b4d0 0x7fe71c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.250+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7280ac560 con 0x7fe74015e0d0 2024-03-06T07:27:47.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.250+0000 7fe744a00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe71c06b4d0 0x7fe71c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.251+0000 7fe744a00640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe71c06b4d0 0x7fe71c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe7300013b0 tx=0x7fe730009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.254+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe72803f070 con 0x7fe74015e0d0 2024-03-06T07:27:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.501+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe74008d920 con 0x7fe74015e0d0 2024-03-06T07:27:47.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.510+0000 7fe73d600640 1 -- 172.21.15.97:0/694950256 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171785 (secure 0 0 0) 0x7fe72802d040 con 0x7fe74015e0d0 2024-03-06T07:27:47.514 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:47.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe71c06b4d0 msgr2=0x7fe71c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe71c06b4d0 0x7fe71c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe7300013b0 tx=0x7fe730009040 comp rx=0 tx=0).stop 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 msgr2=0x7fe7400b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 0x7fe7400b8300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe728009890 tx=0x7fe7280046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 shutdown_connections 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe71c06b4d0 0x7fe71c06d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe74015f470 0x7fe7401582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe74015e0d0 0x7fe7400b8300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 --2- 172.21.15.97:0/694950256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7400d0df0 0x7fe7400c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 >> 172.21.15.97:0/694950256 conn(0x7fe7400d3ec0 msgr2=0x7fe7400d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.514+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 shutdown_connections 2024-03-06T07:27:47.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.515+0000 7fe747400640 1 -- 172.21.15.97:0/694950256 wait complete. 2024-03-06T07:27:47.541 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1969101468},{"type":"v1","addr":"172.21.15.97:6833","nonce":1969101468}]},"active_addr":"172.21.15.97:6833/1969101468","active_change":"2024-03-06T07:27:26.876896+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":3829066850}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":4116668428}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":1552885478}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":2823511474}]}]} 2024-03-06T07:27:47.544 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:27:47.544 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:27:47.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.746+0000 7f7d02400640 1 Processor -- start 2024-03-06T07:27:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.746+0000 7f7d02400640 1 -- start start 2024-03-06T07:27:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.747+0000 7f7d02400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 0x7f7cfc0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.747+0000 7f7d02400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7cfc0b4680 0x7f7cfc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.747+0000 7f7d02400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cfc15b090 con 0x7f7cfc072340 2024-03-06T07:27:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.747+0000 7f7d02400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b2a30 con 0x7f7cfc0b4680 2024-03-06T07:27:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.747+0000 7f7d02400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b4360 con 0x7f7cfc15acb0 2024-03-06T07:27:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.748+0000 7f7cfbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7cfc072340 0x7f7cfc0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45066/0 (socket says 172.21.15.97:45066) 2024-03-06T07:27:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.748+0000 7f7cfbe00640 1 -- 172.21.15.97:0/1861993989 learned_addr learned my addr 172.21.15.97:0/1861993989 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.748+0000 7f7cfb400640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 0x7f7cfc0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.748+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/1861993989 >> v2:172.21.15.135:3300/0 conn(0x7f7cfc0b4680 0x7f7cfc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.748+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771121002 0 0) 0x7f7cfc15b090 con 0x7f7cfc072340 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.748+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce0003520 con 0x7f7cfc072340 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9366886 0 0) 0x7f7ce0003520 con 0x7f7cfc072340 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 msgr2=0x7f7cfc0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 0x7f7cfc0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 >> v2:172.21.15.135:3300/0 conn(0x7f7cfc0b4680 msgr2=0x7f7cfc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 --2- 172.21.15.97:0/1861993989 >> v2:172.21.15.135:3300/0 conn(0x7f7cfc0b4680 0x7f7cfc06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc0b2cf0 con 0x7f7cfc072340 2024-03-06T07:27:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7cf40019f0 con 0x7f7cfc072340 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 >> v1:172.21.15.97:6789/0 conn(0x7f7cfc072340 legacy=0x7f7cfc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 0x7f7ce00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7ce0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ce0004d40 0x7f7ce00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b2a30 con 0x7f7ce0003f30 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b4360 con 0x7f7ce0004d40 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce000fdd0 con 0x7f7cfc15acb0 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7ce0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfaa00640 1 -- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 msgr2=0x7f7ce00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfb400640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 0x7f7ce00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfbe00640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ce0004d40 0x7f7ce00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc15acb0 0x7f7ce00039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfaa00640 1 -- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ce0004d40 msgr2=0x7f7ce00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.749+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ce0004d40 0x7f7ce00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.750+0000 7f7cfaa00640 1 -- 172.21.15.97:0/1861993989 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0003520 con 0x7f7ce0003f30 2024-03-06T07:27:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.751+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7ce0004800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009ac0 tx=0x7f7ce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a483181ca5f265 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.751+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8039070 con 0x7f7ce0003f30 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.751+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce802c6d0 con 0x7f7ce0003f30 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.751+0000 7f7cfa000640 1 -- 172.21.15.97:0/1861993989 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8009690 con 0x7f7ce0003f30 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7d02400640 1 -- 172.21.15.97:0/1861993989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 msgr2=0x7f7ce0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7d02400640 1 --2- 172.21.15.97:0/1861993989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7ce0004800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009ac0 tx=0x7f7ce0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7cfbe00640 1 -- 172.21.15.97:0/1861993989 reap_dead start 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7d02400640 1 -- 172.21.15.97:0/1861993989 shutdown_connections 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7d02400640 1 -- 172.21.15.97:0/1861993989 >> 172.21.15.97:0/1861993989 conn(0x7f7cfc0b1730 msgr2=0x7f7cfc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7d02400640 1 -- 172.21.15.97:0/1861993989 shutdown_connections 2024-03-06T07:27:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.752+0000 7f7d02400640 1 -- 172.21.15.97:0/1861993989 wait complete. 2024-03-06T07:27:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.755+0000 7f7d02400640 1 Processor -- start 2024-03-06T07:27:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.755+0000 7f7d02400640 1 -- start start 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7d02400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7cfc0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7d02400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cfc146360 0x7f7cfc157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7d02400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc1467e0 0x7f7cfc157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7d02400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b4010 con 0x7f7ce0003f30 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7d02400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b32b0 con 0x7f7cfc146360 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7d02400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc06ea30 con 0x7f7cfc1467e0 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7cfc0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7cfc0ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50398/0 (socket says 172.21.15.97:50398) 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 -- 172.21.15.97:0/3113697152 learned_addr learned my addr 172.21.15.97:0/3113697152 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 -- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc1467e0 msgr2=0x7f7cfc157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfbe00640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc1467e0 0x7f7cfc157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cfc146360 0x7f7cfc157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc1467e0 0x7f7cfc157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 -- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cfc146360 msgr2=0x7f7cfc157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cfc146360 0x7f7cfc157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.757+0000 7f7cfb400640 1 -- 172.21.15.97:0/3113697152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc159900 con 0x7f7ce0003f30 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.758+0000 7f7cfb400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7cfc0ced10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7cec010860 tx=0x7f7cec010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.758+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00f040 con 0x7f7ce0003f30 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.758+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec00eb60 con 0x7f7ce0003f30 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.758+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc159be0 con 0x7f7ce0003f30 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.758+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec01c400 con 0x7f7ce0003f30 2024-03-06T07:27:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.758+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc0ce7f0 con 0x7f7ce0003f30 2024-03-06T07:27:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.760+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cfc04f330 con 0x7f7ce0003f30 2024-03-06T07:27:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.760+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cec01c5a0 con 0x7f7ce0003f30 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.762+0000 7f7cf3400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cdc06b500 0x7f7cdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.762+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cec08a800 con 0x7f7ce0003f30 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.762+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cdc06b500 0x7f7cdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.763+0000 7f7cfaa00640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cdc06b500 0x7f7cdc06d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8036000 tx=0x7f7ce8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.771+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7cec018030 con 0x7f7ce0003f30 2024-03-06T07:27:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.981+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cfc06ce60 con 0x7f7ce0003f30 2024-03-06T07:27:47.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.984+0000 7f7cf3400640 1 -- 172.21.15.97:0/3113697152 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f7cec0595e0 con 0x7f7ce0003f30 2024-03-06T07:27:47.986 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:47.986 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cdc06b500 msgr2=0x7f7cdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cdc06b500 0x7f7cdc06d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8036000 tx=0x7f7ce8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 msgr2=0x7f7cfc0ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7cfc0ced10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7cec010860 tx=0x7f7cec010d20 comp rx=0 tx=0).stop 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 shutdown_connections 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cdc06b500 0x7f7cdc06d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cfc1467e0 0x7f7cfc157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cfc146360 0x7f7cfc157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 --2- 172.21.15.97:0/3113697152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ce0003f30 0x7f7cfc0ced10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.986+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 >> 172.21.15.97:0/3113697152 conn(0x7f7cfc0b1730 msgr2=0x7f7cfc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.987+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 shutdown_connections 2024-03-06T07:27:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:47.987+0000 7f7d02400640 1 -- 172.21.15.97:0/3113697152 wait complete. 2024-03-06T07:27:48.006 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:27:48.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:48.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.205+0000 7fb858c00640 1 Processor -- start 2024-03-06T07:27:48.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.205+0000 7fb858c00640 1 -- start start 2024-03-06T07:27:48.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.207+0000 7fb858c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb85406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.207+0000 7fb858c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85406ee70 con 0x7fb85415f9d0 2024-03-06T07:27:48.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.207+0000 7fb858c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85406cef0 con 0x7fb85415fe30 2024-03-06T07:27:48.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.207+0000 7fb858c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85406d1d0 con 0x7fb854144c00 2024-03-06T07:27:48.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.207+0000 7fb84be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb85415f9d0 0x7fb8540ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45080/0 (socket says 172.21.15.97:45080) 2024-03-06T07:27:48.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.207+0000 7fb84be00640 1 -- 172.21.15.97:0/2543894076 learned_addr learned my addr 172.21.15.97:0/2543894076 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851e00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb85406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340459170 0 0) 0x7fb83c001110 con 0x7fb85415f9d0 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb838003520 con 0x7fb85415f9d0 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4099101643 0 0) 0x7fb838003520 con 0x7fb85415f9d0 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 msgr2=0x7fb85406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb85406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 >> v1:172.21.15.135:6789/0 conn(0x7fb85415fe30 legacy=0x7fb8540ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85406d4b0 con 0x7fb85415f9d0 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb83c001840 con 0x7fb85415f9d0 2024-03-06T07:27:48.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 >> v1:172.21.15.97:6789/0 conn(0x7fb85415f9d0 legacy=0x7fb8540ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8380039e0 0x7fb838003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.208+0000 7fb851400640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb838004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851400640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb838004d40 0x7fb8380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85406d1d0 con 0x7fb8380039e0 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83800ba70 con 0x7fb838004d40 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83800bc70 con 0x7fb854144c00 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851e00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8380039e0 0x7fb838003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb84be00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb838004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851e00640 1 -- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 msgr2=0x7fb838004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb852800640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb838004d40 0x7fb8380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851e00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb838004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851e00640 1 -- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb838004d40 msgr2=0x7fb8380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851e00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb838004d40 0x7fb8380051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb851e00640 1 -- 172.21.15.97:0/2543894076 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838003520 con 0x7fb8380039e0 2024-03-06T07:27:48.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.209+0000 7fb84be00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb854144c00 0x7fb838004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:48.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.210+0000 7fb851e00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8380039e0 0x7fb838003e00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb844009a90 tx=0x7fb838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c5b4ac4e9de707 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.210+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844039070 con 0x7fb8380039e0 2024-03-06T07:27:48.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84402c680 con 0x7fb8380039e0 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb851400640 1 -- 172.21.15.97:0/2543894076 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844009690 con 0x7fb8380039e0 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb858c00640 1 -- 172.21.15.97:0/2543894076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8380039e0 msgr2=0x7fb838003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb858c00640 1 --2- 172.21.15.97:0/2543894076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8380039e0 0x7fb838003e00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb844009a90 tx=0x7fb838005830 comp rx=0 tx=0).stop 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb852800640 1 -- 172.21.15.97:0/2543894076 reap_dead start 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb858c00640 1 -- 172.21.15.97:0/2543894076 shutdown_connections 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb858c00640 1 -- 172.21.15.97:0/2543894076 >> 172.21.15.97:0/2543894076 conn(0x7fb8540b1730 msgr2=0x7fb8540b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb858c00640 1 -- 172.21.15.97:0/2543894076 shutdown_connections 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.211+0000 7fb858c00640 1 -- 172.21.15.97:0/2543894076 wait complete. 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.213+0000 7fb858c00640 1 Processor -- start 2024-03-06T07:27:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.213+0000 7fb858c00640 1 -- start start 2024-03-06T07:27:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb858c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8380039e0 0x7fb8541535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb858c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 0x7fb85413a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb858c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8540ae9e0 0x7fb8540aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb858c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb854160730 con 0x7fb85413bbb0 2024-03-06T07:27:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb858c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb854160280 con 0x7fb8540ae9e0 2024-03-06T07:27:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb858c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540ceec0 con 0x7fb8380039e0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb84be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 0x7fb85413a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb84be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 0x7fb85413a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50418/0 (socket says 172.21.15.97:50418) 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.214+0000 7fb84be00640 1 -- 172.21.15.97:0/890788458 learned_addr learned my addr 172.21.15.97:0/890788458 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb851e00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8380039e0 0x7fb8541535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb852800640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8540ae9e0 0x7fb8540aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84be00640 1 -- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8380039e0 msgr2=0x7fb8541535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84be00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8380039e0 0x7fb8541535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84be00640 1 -- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8540ae9e0 msgr2=0x7fb8540aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84be00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8540ae9e0 0x7fb8540aee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84be00640 1 -- 172.21.15.97:0/890788458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85413c340 con 0x7fb85413bbb0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84be00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 0x7fb85413a140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00d7c0 tx=0x7fb83c00dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c019070 con 0x7fb85413bbb0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.215+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83c0074e0 con 0x7fb85413bbb0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.216+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85413c620 con 0x7fb85413bbb0 2024-03-06T07:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.216+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0049e0 con 0x7fb85413bbb0 2024-03-06T07:27:48.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.216+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85413cbe0 con 0x7fb85413bbb0 2024-03-06T07:27:48.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.217+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8540af3a0 con 0x7fb85413bbb0 2024-03-06T07:27:48.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.218+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb83c01c070 con 0x7fb85413bbb0 2024-03-06T07:27:48.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.218+0000 7fb84a000640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83006b4d0 0x7fb83006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.219+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb83c089d90 con 0x7fb85413bbb0 2024-03-06T07:27:48.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.219+0000 7fb851e00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83006b4d0 0x7fb83006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.219+0000 7fb851e00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83006b4d0 0x7fb83006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb844036000 tx=0x7fb844030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.227+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb85406ee70 con 0x7fb85413bbb0 2024-03-06T07:27:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.423+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb85404f330 con 0x7fb85413bbb0 2024-03-06T07:27:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.426+0000 7fb84a000640 1 -- 172.21.15.97:0/890788458 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fb83c058b60 con 0x7fb85413bbb0 2024-03-06T07:27:48.428 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:48.429 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.428+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83006b4d0 msgr2=0x7fb83006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.428+0000 7fb858c00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83006b4d0 0x7fb83006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb844036000 tx=0x7fb844030000 comp rx=0 tx=0).stop 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.428+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 msgr2=0x7fb85413a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.428+0000 7fb858c00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 0x7fb85413a140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00d7c0 tx=0x7fb83c00dc80 comp rx=0 tx=0).stop 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.428+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 shutdown_connections 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83006b4d0 0x7fb83006d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8540ae9e0 0x7fb8540aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85413bbb0 0x7fb85413a140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 --2- 172.21.15.97:0/890788458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8380039e0 0x7fb8541535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 >> 172.21.15.97:0/890788458 conn(0x7fb8540b1730 msgr2=0x7fb8540af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 shutdown_connections 2024-03-06T07:27:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.429+0000 7fb858c00640 1 -- 172.21.15.97:0/890788458 wait complete. 2024-03-06T07:27:48.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:27:48.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:27:48.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:27:48.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:27:48.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:27:48.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:27:48.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:27:48.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:27:48.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.774+0000 7fe565200640 1 Processor -- start 2024-03-06T07:27:48.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.775+0000 7fe565200640 1 -- start start 2024-03-06T07:27:48.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.776+0000 7fe565200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe5600b1650 0x7fe5600b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.776+0000 7fe565200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe5600c4930 0x7fe5600c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.776+0000 7f6efbe00640 1 Processor -- start 2024-03-06T07:27:48.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.776+0000 7fb2d7000640 1 Processor -- start 2024-03-06T07:27:48.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.776+0000 7fb2d7000640 1 -- start start 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe565200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5600cd930 con 0x7fe5600cf320 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe565200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5600b1ff0 con 0x7fe5600b1650 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe565200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5600cdef0 con 0x7fe5600c4930 2024-03-06T07:27:48.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7f6efbe00640 1 -- start start 2024-03-06T07:27:48.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe557e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe5600b1650 0x7fe5600b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe557e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe5600b1650 0x7fe5600b1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50434/0 (socket says 172.21.15.97:50434) 2024-03-06T07:27:48.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe557e00640 1 -- 172.21.15.97:0/3454480412 learned_addr learned my addr 172.21.15.97:0/3454480412 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fc77a200640 1 Processor -- start 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.777+0000 7fe55e800640 1 --2- 172.21.15.97:0/3454480412 >> v2:172.21.15.135:3300/0 conn(0x7fe5600c4930 0x7fe5600c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fe557e00640 1 -- 172.21.15.97:0/3454480412 >> v2:172.21.15.135:3300/0 conn(0x7fe5600c4930 msgr2=0x7fe5600c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fe557e00640 1 --2- 172.21.15.97:0/3454480412 >> v2:172.21.15.135:3300/0 conn(0x7fe5600c4930 0x7fe5600c4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fe557e00640 1 -- 172.21.15.97:0/3454480412 >> v1:172.21.15.135:6789/0 conn(0x7fe5600cf320 legacy=0x7fe56016bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fe557e00640 1 -- 172.21.15.97:0/3454480412 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5600cdc10 con 0x7fe5600b1650 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fe55e800640 1 --2- 172.21.15.97:0/3454480412 >> v2:172.21.15.135:3300/0 conn(0x7fe5600c4930 0x7fe5600c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fb2d7000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb2d01636d0 0x7fb2d006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fb2d7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d0070a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fb2d7000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d0162910 con 0x7fb2d0072150 2024-03-06T07:27:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fb2d7000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00b2d60 con 0x7fb2d01636d0 2024-03-06T07:27:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fb2d7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0162bf0 con 0x7fb2d006dcf0 2024-03-06T07:27:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.778+0000 7fc77a200640 1 -- start start 2024-03-06T07:27:48.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.779+0000 7fb2d5000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d0070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.779+0000 7fb2d5000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d0070a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:32922/0 (socket says 172.21.15.97:32922) 2024-03-06T07:27:48.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.779+0000 7fb2d5000640 1 -- 172.21.15.97:0/3953241734 learned_addr learned my addr 172.21.15.97:0/3953241734 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.779+0000 7fb2cf400640 1 --2- 172.21.15.97:0/3953241734 >> v2:172.21.15.135:3300/0 conn(0x7fb2d01636d0 0x7fb2d006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fe557e00640 1 --2- 172.21.15.97:0/3454480412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600b1650 0x7fe5600b1ab0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe56016b120 tx=0x7fe54802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21af3d5ce90eb816 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7f6efbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6efc070fd0 0x7f6efc15e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7f6efbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6efc0b40e0 0x7f6efc06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fe55d400640 1 -- 172.21.15.97:0/3454480412 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548034070 con 0x7fe5600b1650 2024-03-06T07:27:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fe55d400640 1 -- 172.21.15.97:0/3454480412 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe548030870 con 0x7fe5600b1650 2024-03-06T07:27:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fe55d400640 1 -- 172.21.15.97:0/3454480412 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548030c40 con 0x7fe5600b1650 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fb2d5000640 1 -- 172.21.15.97:0/3953241734 >> v2:172.21.15.135:3300/0 conn(0x7fb2d01636d0 msgr2=0x7fb2d006d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fb2cea00640 1 -- 172.21.15.97:0/3953241734 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465796709 0 0) 0x7fb2d0162910 con 0x7fb2d0072150 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fb2d5000640 1 --2- 172.21.15.97:0/3953241734 >> v2:172.21.15.135:3300/0 conn(0x7fb2d01636d0 0x7fb2d006d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fb2d5000640 1 -- 172.21.15.97:0/3953241734 >> v1:172.21.15.135:6789/0 conn(0x7fb2d0072150 legacy=0x7fb2d0072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fb2d5000640 1 -- 172.21.15.97:0/3953241734 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0162ed0 con 0x7fb2d006dcf0 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fb2cf400640 1 --2- 172.21.15.97:0/3953241734 >> v2:172.21.15.135:3300/0 conn(0x7fb2d01636d0 0x7fb2d006d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7f6efbe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0b4ac0 con 0x7f6efc159d60 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7f6efbe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6efc06cb30 con 0x7f6efc0b40e0 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7f6efbe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6efc06ce10 con 0x7f6efc070fd0 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efaa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6efc159d60 0x7f6efc0708c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45092/0 (socket says 172.21.15.97:45092) 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efaa00640 1 -- 172.21.15.97:0/2665678651 learned_addr learned my addr 172.21.15.97:0/2665678651 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efa000640 1 --2- 172.21.15.97:0/2665678651 >> v2:172.21.15.135:3300/0 conn(0x7f6efc070fd0 0x7f6efc15e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efa000640 1 -- 172.21.15.97:0/2665678651 >> v2:172.21.15.97:3300/0 conn(0x7f6efc0b40e0 msgr2=0x7f6efc06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efa000640 1 --2- 172.21.15.97:0/2665678651 >> v2:172.21.15.97:3300/0 conn(0x7f6efc0b40e0 0x7f6efc06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efa000640 1 -- 172.21.15.97:0/2665678651 >> v1:172.21.15.97:6789/0 conn(0x7f6efc159d60 legacy=0x7f6efc0708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7f6efa000640 1 -- 172.21.15.97:0/2665678651 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc06d0f0 con 0x7f6efc070fd0 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.782+0000 7fe565200640 1 -- 172.21.15.97:0/3454480412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600b1650 msgr2=0x7fe5600b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.782+0000 7fe565200640 1 --2- 172.21.15.97:0/3454480412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600b1650 0x7fe5600b1ab0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe56016b120 tx=0x7fe54802b9b0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fe565200640 1 -- 172.21.15.97:0/3454480412 shutdown_connections 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fe565200640 1 --2- 172.21.15.97:0/3454480412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600b1650 0x7fe5600b1ab0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fe565200640 1 --2- 172.21.15.97:0/3454480412 >> v2:172.21.15.135:3300/0 conn(0x7fe5600c4930 0x7fe5600c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fe565200640 1 -- 172.21.15.97:0/3454480412 >> 172.21.15.97:0/3454480412 conn(0x7fe5600cfa70 msgr2=0x7fe5600cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fe565200640 1 -- 172.21.15.97:0/3454480412 shutdown_connections 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fe565200640 1 -- 172.21.15.97:0/3454480412 wait complete. 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.784+0000 7fb2d5000640 1 --2- 172.21.15.97:0/3953241734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d0070a20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009e50 tx=0x7fb2c802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ab1166655a74a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.785+0000 7f6efa000640 1 --2- 172.21.15.97:0/2665678651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc070fd0 0x7f6efc15e410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0730c0 tx=0x7f6eec02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42efe48f1b4e8998 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.785+0000 7fb2cea00640 1 -- 172.21.15.97:0/3953241734 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c802cb00 con 0x7fb2d006dcf0 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.785+0000 7fb2cea00640 1 -- 172.21.15.97:0/3953241734 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c802cca0 con 0x7fb2d006dcf0 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.785+0000 7fb2cea00640 1 -- 172.21.15.97:0/3953241734 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c802cb00 con 0x7fb2d006dcf0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.785+0000 7f6ef9600640 1 -- 172.21.15.97:0/2665678651 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec03b070 con 0x7f6efc070fd0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.786+0000 7f6ef9600640 1 -- 172.21.15.97:0/2665678651 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec02edf0 con 0x7f6efc070fd0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.786+0000 7f6ef9600640 1 -- 172.21.15.97:0/2665678651 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec036660 con 0x7f6efc070fd0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.786+0000 7faec4800640 1 Processor -- start 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fc77a200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7740b1650 0x7fc7740b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fc77a200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7741409b0 con 0x7fc7740d0590 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fc77a200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77406cd60 con 0x7fc7740cd620 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.780+0000 7fc77a200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc77406d690 con 0x7fc7740b1650 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fc772a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7740b1650 0x7fc7740b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fc772a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7740b1650 0x7fc7740b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36596/0 (socket says 172.21.15.97:36596) 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.781+0000 7fc772a00640 1 -- 172.21.15.97:0/2001150487 learned_addr learned my addr 172.21.15.97:0/2001150487 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fc772a00640 1 -- 172.21.15.97:0/2001150487 >> v1:172.21.15.135:6789/0 conn(0x7fc7740cd620 legacy=0x7fc7740cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fc772a00640 1 -- 172.21.15.97:0/2001150487 >> v1:172.21.15.97:6789/0 conn(0x7fc7740d0590 legacy=0x7fc7741402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.783+0000 7fc772a00640 1 -- 172.21.15.97:0/2001150487 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774161050 con 0x7fc7740b1650 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.786+0000 7fb2d7000640 1 -- 172.21.15.97:0/3953241734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 msgr2=0x7fb2d0070a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.786+0000 7fb2d7000640 1 --2- 172.21.15.97:0/3953241734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d0070a20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009e50 tx=0x7fb2c802bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:48.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7f6efbe00640 1 -- 172.21.15.97:0/2665678651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc070fd0 msgr2=0x7f6efc15e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7faec4800640 1 -- start start 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7f6efbe00640 1 --2- 172.21.15.97:0/2665678651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc070fd0 0x7f6efc15e410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0730c0 tx=0x7f6eec02e040 comp rx=0 tx=0).stop 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7fe565200640 1 Processor -- start 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7f6efbe00640 1 -- 172.21.15.97:0/2665678651 shutdown_connections 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7f6efbe00640 1 --2- 172.21.15.97:0/2665678651 >> v2:172.21.15.97:3300/0 conn(0x7f6efc0b40e0 0x7f6efc06d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7f6efbe00640 1 --2- 172.21.15.97:0/2665678651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc070fd0 0x7f6efc15e410 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0730c0 tx=0x7f6eec02e040 comp rx=0 tx=0).stop 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7f6efbe00640 1 -- 172.21.15.97:0/2665678651 >> 172.21.15.97:0/2665678651 conn(0x7f6efc0b1490 msgr2=0x7f6efc0b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.786+0000 7fc772a00640 1 --2- 172.21.15.97:0/2001150487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740b1650 0x7fc7740b1ad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc764014a00 tx=0x7fc764034bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79a7b424fbaac6ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7fe565200640 1 -- start start 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7fb2d7000640 1 -- 172.21.15.97:0/3953241734 shutdown_connections 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7fb2d7000640 1 --2- 172.21.15.97:0/3953241734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d0070a20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7fb2d7000640 1 --2- 172.21.15.97:0/3953241734 >> v2:172.21.15.135:3300/0 conn(0x7fb2d01636d0 0x7fb2d006d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.787+0000 7fb2d7000640 1 -- 172.21.15.97:0/3953241734 >> 172.21.15.97:0/3953241734 conn(0x7fb2d00b1730 msgr2=0x7fb2d00b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.788+0000 7f6efbe00640 1 -- 172.21.15.97:0/2665678651 shutdown_connections 2024-03-06T07:27:48.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.788+0000 7f6efbe00640 1 -- 172.21.15.97:0/2665678651 wait complete. 2024-03-06T07:27:48.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.788+0000 7fc772000640 1 -- 172.21.15.97:0/2001150487 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764035960 con 0x7fc7740b1650 2024-03-06T07:27:48.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.788+0000 7fc772000640 1 -- 172.21.15.97:0/2001150487 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764035b00 con 0x7fc7740b1650 2024-03-06T07:27:48.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.788+0000 7fc772000640 1 -- 172.21.15.97:0/2001150487 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640110d0 con 0x7fc7740b1650 2024-03-06T07:27:48.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.789+0000 7f7b12400640 1 Processor -- start 2024-03-06T07:27:48.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.789+0000 7f7b12400640 1 -- start start 2024-03-06T07:27:48.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec4800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc154490 0x7faebc0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7fb2d7000640 1 -- 172.21.15.97:0/3953241734 shutdown_connections 2024-03-06T07:27:48.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec4800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faebc0cde80 0x7faebc0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec4800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faebc0ca1f0 con 0x7faebc154870 2024-03-06T07:27:48.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7f6efbe00640 1 Processor -- start 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec4800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc0ca4d0 con 0x7faebc0cde80 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec4800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0ca7b0 con 0x7faebc154490 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec1400640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7faebc154870 0x7faebc0cd770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:41572/0 (socket says 172.21.15.97:41572) 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7f6efbe00640 1 -- start start 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.790+0000 7faec1400640 1 -- 172.21.15.97:0/3190067144 learned_addr learned my addr 172.21.15.97:0/3190067144 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe565200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5600b1650 0x7fe5600b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe565200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5600c4930 0x7fe560160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe565200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600cf320 0x7fe5600c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe565200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600c5300 con 0x7fe5600cf320 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe565200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56006d040 con 0x7fe5600b1650 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe565200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56006e670 con 0x7fe5600c4930 2024-03-06T07:27:48.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7f7b12400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b040e1030 0x7f7b04054d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7f7b12400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b04051a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fea48a00640 1 Processor -- start 2024-03-06T07:27:48.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7faec2800640 1 --2- 172.21.15.97:0/3190067144 >> v2:172.21.15.97:3300/0 conn(0x7faebc0cde80 0x7faebc0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7faec1e00640 1 --2- 172.21.15.97:0/3190067144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc154490 0x7faebc0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7faec0a00640 1 -- 172.21.15.97:0/3190067144 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245069214 0 0) 0x7faebc0ca1f0 con 0x7faebc154870 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7f7b12400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b04009b40 con 0x7f7b04056790 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7f7b12400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b04056300 con 0x7f7b040e1030 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7f7b12400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040e1410 con 0x7f7b0403c9f0 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe55de00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5600b1650 0x7fe5600b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe55de00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5600b1650 0x7fe5600b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36598/0 (socket says 172.21.15.97:36598) 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe55de00640 1 -- 172.21.15.97:0/840978860 learned_addr learned my addr 172.21.15.97:0/840978860 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7fe55e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600cf320 0x7fe5600c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fe557e00640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5600c4930 0x7fe560160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.791+0000 7faec0a00640 1 -- 172.21.15.97:0/3190067144 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea4003520 con 0x7faebc154870 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7f7b11000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b040e1030 0x7f7b04054d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7f7b11000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b040e1030 0x7f7b04054d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36604/0 (socket says 172.21.15.97:36604) 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7f7b11000640 1 -- 172.21.15.97:0/3322965541 learned_addr learned my addr 172.21.15.97:0/3322965541 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7faec1e00640 1 -- 172.21.15.97:0/3190067144 >> v2:172.21.15.97:3300/0 conn(0x7faebc0cde80 msgr2=0x7faebc0cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fe55e800640 1 -- 172.21.15.97:0/840978860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5600c4930 msgr2=0x7fe560160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fe55e800640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5600c4930 0x7fe560160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7faec1e00640 1 --2- 172.21.15.97:0/3190067144 >> v2:172.21.15.97:3300/0 conn(0x7faebc0cde80 0x7faebc0cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7faec1e00640 1 -- 172.21.15.97:0/3190067144 >> v1:172.21.15.135:6789/0 conn(0x7faebc154870 legacy=0x7faebc0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fe55e800640 1 -- 172.21.15.97:0/840978860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5600b1650 msgr2=0x7fe5600b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fe55e800640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5600b1650 0x7fe5600b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fe55e800640 1 -- 172.21.15.97:0/840978860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5600c8450 con 0x7fe5600cf320 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7f7b0be00640 1 --2- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b04051a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7f7b11000640 1 -- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 msgr2=0x7f7b04051a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7f7b11000640 1 --2- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b04051a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7f7b11000640 1 -- 172.21.15.97:0/3322965541 >> v1:172.21.15.135:6789/0 conn(0x7f7b04056790 legacy=0x7f7b0400dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7f7b11000640 1 -- 172.21.15.97:0/3322965541 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b040556f0 con 0x7f7b040e1030 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fea48a00640 1 -- start start 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7fe55e800640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600cf320 0x7fe5600c6ab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe55800af70 tx=0x7fe55800e4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7faec1e00640 1 -- 172.21.15.97:0/3190067144 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0caa90 con 0x7faebc154490 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7faec2800640 1 --2- 172.21.15.97:0/3190067144 >> v2:172.21.15.97:3300/0 conn(0x7faebc0cde80 0x7faebc0cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.789+0000 7fc77a200640 1 -- 172.21.15.97:0/2001150487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740b1650 msgr2=0x7fc7740b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.789+0000 7fc77a200640 1 --2- 172.21.15.97:0/2001150487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740b1650 0x7fc7740b1ad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc764014a00 tx=0x7fc764034bb0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fc77a200640 1 -- 172.21.15.97:0/2001150487 shutdown_connections 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fc77a200640 1 --2- 172.21.15.97:0/2001150487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740b1650 0x7fc7740b1ad0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.792+0000 7fc77a200640 1 -- 172.21.15.97:0/2001150487 >> 172.21.15.97:0/2001150487 conn(0x7fc7740cfa70 msgr2=0x7fc7740cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7fb2d7000640 1 -- 172.21.15.97:0/3953241734 wait complete. 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6efc0b40e0 0x7f6efc15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558016070 con 0x7fe5600cf320 2024-03-06T07:27:48.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe558003b90 con 0x7fe5600cf320 2024-03-06T07:27:48.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.793+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800ee20 con 0x7fe5600cf320 2024-03-06T07:27:48.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc159d60 0x7f6efc140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6efc141380 0x7f6efc15a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7fea48a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fea4415e090 0x7fea440cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7f6efbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06de30 con 0x7f6efc141380 2024-03-06T07:27:48.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7f6efbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06ec00 con 0x7f6efc159d60 2024-03-06T07:27:48.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7fc77a200640 1 -- 172.21.15.97:0/2001150487 shutdown_connections 2024-03-06T07:27:48.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7f6efbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06dfb0 con 0x7f6efc0b40e0 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7fc77a200640 1 -- 172.21.15.97:0/2001150487 wait complete. 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5601606a0 con 0x7fe5600cf320 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.794+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5600c7970 con 0x7fe5600cf320 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7f7b11000640 1 --2- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b040e1030 0x7f7b04054d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009870 tx=0x7f7b0002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=195b7fb1a6cb5b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7fea43400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fea4415e090 0x7fea440cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7fea43400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fea4415e090 0x7fea440cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50478/0 (socket says 172.21.15.97:50478) 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7f6efb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6efc141380 0x7f6efc15a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7f6efa000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc159d60 0x7f6efc140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7faec1e00640 1 --2- 172.21.15.97:0/3190067144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc154490 0x7faebc0cba30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faeac0097a0 tx=0x7faeac02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d7bf57fb476395e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7faec0a00640 1 -- 172.21.15.97:0/3190067144 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac035020 con 0x7faebc154490 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7fb2d7000640 1 Processor -- start 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7fea43400640 1 -- 172.21.15.97:0/748800403 learned_addr learned my addr 172.21.15.97:0/748800403 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7faec0a00640 1 -- 172.21.15.97:0/3190067144 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac031830 con 0x7faebc154490 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7faec0a00640 1 -- 172.21.15.97:0/3190067144 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac031c20 con 0x7faebc154490 2024-03-06T07:27:48.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7f6efb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6efc141380 0x7f6efc15a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50468/0 (socket says 172.21.15.97:50468) 2024-03-06T07:27:48.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.795+0000 7f6efb400640 1 -- 172.21.15.97:0/1434603436 learned_addr learned my addr 172.21.15.97:0/1434603436 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440ce890 con 0x7fea440d1270 2024-03-06T07:27:48.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440ceb30 con 0x7fea440d2340 2024-03-06T07:27:48.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4415e470 con 0x7fea4415e090 2024-03-06T07:27:48.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7f6efaa00640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6efc0b40e0 0x7f6efc15c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7faec4800640 1 -- 172.21.15.97:0/3190067144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc154490 msgr2=0x7faebc0cba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7faec4800640 1 --2- 172.21.15.97:0/3190067144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc154490 0x7faebc0cba30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faeac0097a0 tx=0x7faeac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7f6efb400640 1 -- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6efc0b40e0 msgr2=0x7f6efc15c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fea43400640 1 -- 172.21.15.97:0/748800403 >> v1:172.21.15.135:6789/0 conn(0x7fea440d2340 legacy=0x7fea440b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fea43400640 1 -- 172.21.15.97:0/748800403 >> v1:172.21.15.97:6789/0 conn(0x7fea440d1270 legacy=0x7fea440d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7f6efb400640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6efc0b40e0 0x7f6efc15c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7f6efb400640 1 -- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc159d60 msgr2=0x7f6efc140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7f6efb400640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6efc159d60 0x7f6efc140e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7f6efb400640 1 -- 172.21.15.97:0/1434603436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc141830 con 0x7f6efc141380 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fea43400640 1 -- 172.21.15.97:0/748800403 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440cee10 con 0x7fea4415e090 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7faec4800640 1 -- 172.21.15.97:0/3190067144 shutdown_connections 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7faec4800640 1 --2- 172.21.15.97:0/3190067144 >> v2:172.21.15.97:3300/0 conn(0x7faebc0cde80 0x7faebc0cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7faec4800640 1 --2- 172.21.15.97:0/3190067144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc154490 0x7faebc0cba30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7faec4800640 1 -- 172.21.15.97:0/3190067144 >> 172.21.15.97:0/3190067144 conn(0x7faebc0cc4f0 msgr2=0x7faebc0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe534001030 con 0x7fe5600cf320 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f6efb400640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6efc141380 0x7f6efc15a450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009fd0 tx=0x7f6ef400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7faec4800640 1 -- 172.21.15.97:0/3190067144 shutdown_connections 2024-03-06T07:27:48.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4011830 con 0x7f6efc141380 2024-03-06T07:27:48.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7faec4800640 1 -- 172.21.15.97:0/3190067144 wait complete. 2024-03-06T07:27:48.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe558019070 con 0x7fe5600cf320 2024-03-06T07:27:48.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc15b220 con 0x7f6efc141380 2024-03-06T07:27:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef4016070 con 0x7f6efc141380 2024-03-06T07:27:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4011830 con 0x7f6efc141380 2024-03-06T07:27:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fc77a200640 1 Processor -- start 2024-03-06T07:27:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7fc77a200640 1 -- start start 2024-03-06T07:27:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc15ba60 con 0x7f6efc141380 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.800+0000 7fe556000640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53c06f930 0x7fe53c071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f7b0b400640 1 -- 172.21.15.97:0/3322965541 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00035020 con 0x7f7b040e1030 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f7b0b400640 1 -- 172.21.15.97:0/3322965541 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b00031870 con 0x7f7b040e1030 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.798+0000 7f7b0b400640 1 -- 172.21.15.97:0/3322965541 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00031c80 con 0x7f7b040e1030 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7f7b12400640 1 -- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b040e1030 msgr2=0x7f7b04054d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7f7b12400640 1 --2- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b040e1030 0x7f7b04054d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009870 tx=0x7f7b0002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f7b12400640 1 -- 172.21.15.97:0/3322965541 shutdown_connections 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f7b12400640 1 --2- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b04051a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f7b12400640 1 --2- 172.21.15.97:0/3322965541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b040e1030 0x7f7b04054d80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f7b12400640 1 -- 172.21.15.97:0/3322965541 >> 172.21.15.97:0/3322965541 conn(0x7f7b04056cb0 msgr2=0x7f7b040570c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7fea43400640 1 --2- 172.21.15.97:0/748800403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 0x7fea440cf8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea34009870 tx=0x7fea3402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f821a382f06dfce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.800+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe55808a4a0 con 0x7fe5600cf320 2024-03-06T07:27:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f7b12400640 1 -- 172.21.15.97:0/3322965541 shutdown_connections 2024-03-06T07:27:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7faec4800640 1 Processor -- start 2024-03-06T07:27:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7fc77a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7740b1650 0x7fc774153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6efc04f330 con 0x7f6efc141380 2024-03-06T07:27:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7f7b12400640 1 -- 172.21.15.97:0/3322965541 wait complete. 2024-03-06T07:27:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7faec4800640 1 -- start start 2024-03-06T07:27:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc77a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740cd620 0x7fc774070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc77a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7740d0590 0x7fc77413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fea42000640 1 -- 172.21.15.97:0/748800403 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34034070 con 0x7fea4415e090 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc77a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77406d320 con 0x7fc7740b1650 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc77a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740ce390 con 0x7fc7740cd620 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc77a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740c5d90 con 0x7fc7740d0590 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc773400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7740b1650 0x7fc774153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc773400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7740b1650 0x7fc774153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50492/0 (socket says 172.21.15.97:50492) 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fc773400640 1 -- 172.21.15.97:0/2627589458 learned_addr learned my addr 172.21.15.97:0/2627589458 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc773e00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7740d0590 0x7fc77413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc772a00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740cd620 0x7fc774070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7faec4800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc0cde80 0x7faebc1e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7faec4800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faebc154490 0x7faebc1e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7faec4800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faebc154870 0x7faebc140190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7faec4800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc06ec70 con 0x7faebc154870 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7faec4800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0cb290 con 0x7faebc154490 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7faec4800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc06edf0 con 0x7faebc0cde80 2024-03-06T07:27:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7fe556000640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe53c077ad0 0x7fe53c079ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7f7b12400640 1 Processor -- start 2024-03-06T07:27:48.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fea42000640 1 -- 172.21.15.97:0/748800403 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34030850 con 0x7fea4415e090 2024-03-06T07:27:48.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fea42000640 1 -- 172.21.15.97:0/748800403 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34030c20 con 0x7fea4415e090 2024-03-06T07:27:48.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 msgr2=0x7fea440cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7fea48a00640 1 --2- 172.21.15.97:0/748800403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 0x7fea440cf8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea34009870 tx=0x7fea3402b990 comp rx=0 tx=0).stop 2024-03-06T07:27:48.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fe55de00640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53c06f930 0x7fe53c071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5580020b0 con 0x7fe53c077ad0 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe558091050 con 0x7fe5600cf320 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7fe55de00640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53c06f930 0x7fe53c071df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe550007fd0 tx=0x7fe550009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7faec2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faebc154870 0x7faebc140190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7faec2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faebc154870 0x7faebc140190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50502/0 (socket says 172.21.15.97:50502) 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec2800640 1 -- 172.21.15.97:0/1834489874 learned_addr learned my addr 172.21.15.97:0/1834489874 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7f7b12400640 1 -- start start 2024-03-06T07:27:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec1e00640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc0cde80 0x7faebc1e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc773e00640 1 -- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740cd620 msgr2=0x7fc774070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7fe557e00640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe53c077ad0 0x7fe53c079ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec1400640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faebc154490 0x7faebc1e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec2800640 1 -- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc0cde80 msgr2=0x7faebc1e8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec2800640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faebc0cde80 0x7faebc1e8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec2800640 1 -- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faebc154490 msgr2=0x7faebc1e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec2800640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faebc154490 0x7faebc1e4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7faec2800640 1 -- 172.21.15.97:0/1834489874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc15acc0 con 0x7faebc154870 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc773e00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740cd620 0x7fc774070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc773e00640 1 -- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7740b1650 msgr2=0x7fc774153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc773e00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7740b1650 0x7fc774153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fc773e00640 1 -- 172.21.15.97:0/2627589458 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774070fc0 con 0x7fc7740d0590 2024-03-06T07:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7fc773e00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7740d0590 0x7fc77413cbb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00c310 tx=0x7fc76c00c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7fc772a00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7740cd620 0x7fc774070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.796+0000 7fb2d7000640 1 -- start start 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fb2d7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d00b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fb2d7000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2d0072150 0x7fb2d0157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fb2d7000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2d0150f20 0x7fb2d01599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fb2d7000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0162580 con 0x7fb2d0072150 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fb2d7000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d006e390 con 0x7fb2d0150f20 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.797+0000 7fb2d7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0070f60 con 0x7fb2d006dcf0 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7fb2cf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2d0072150 0x7fb2d0157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7fb2cf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2d0072150 0x7fb2d0157d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50482/0 (socket says 172.21.15.97:50482) 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.799+0000 7fb2cf400640 1 -- 172.21.15.97:0/3685732332 learned_addr learned my addr 172.21.15.97:0/3685732332 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7fb2cfe00640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d00b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.801+0000 7fb2d5000640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2d0150f20 0x7fb2d01599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fb2cfe00640 1 -- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2d0150f20 msgr2=0x7fb2d01599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fb2cfe00640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2d0150f20 0x7fb2d01599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fb2cfe00640 1 -- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2d0072150 msgr2=0x7fb2d0157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fb2cfe00640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2d0072150 0x7fb2d0157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.802+0000 7fb2cfe00640 1 -- 172.21.15.97:0/3685732332 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0158260 con 0x7fb2d006dcf0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fb2cfe00640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d00b0600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c3e0 tx=0x7fb2c000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0005320 con 0x7fb2d006dcf0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c0007030 con 0x7fb2d006dcf0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0007440 con 0x7fb2d006dcf0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d015fd50 con 0x7fb2d006dcf0 2024-03-06T07:27:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d0160310 con 0x7fb2d006dcf0 2024-03-06T07:27:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.803+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ef40119d0 con 0x7f6efc141380 2024-03-06T07:27:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c017070 con 0x7fc7740d0590 2024-03-06T07:27:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7f6ef3400640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ed806b6d0 0x7f6ed806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76c0052d0 con 0x7fc7740d0590 2024-03-06T07:27:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.804+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ef408bd50 con 0x7f6efc141380 2024-03-06T07:27:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7f6ef3400640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f6ed8073870 0x7f6ed8075c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ef4020e50 con 0x7f6ed8073870 2024-03-06T07:27:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.805+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ef408f050 con 0x7f6efc141380 2024-03-06T07:27:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7faec2800640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faebc154870 0x7faebc140190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faeb8007e90 tx=0x7faeb800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c003810 con 0x7fc7740d0590 2024-03-06T07:27:48.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7f7b12400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b040d4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b12400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b04056790 0x7f7b040e2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77413be30 con 0x7fc7740d0590 2024-03-06T07:27:48.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77413c0d0 con 0x7fc7740d0590 2024-03-06T07:27:48.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.806+0000 7fe557e00640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe53c077ad0 0x7fe53c079ef0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b12400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b040e1030 0x7f7b040e2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb801a070 con 0x7faebc154870 2024-03-06T07:27:48.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8015400 con 0x7faebc154870 2024-03-06T07:27:48.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80157f0 con 0x7faebc154870 2024-03-06T07:27:48.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b12400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04055f90 con 0x7f7b040e1030 2024-03-06T07:27:48.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b12400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04055cb0 con 0x7f7b04056790 2024-03-06T07:27:48.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b12400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040552c0 con 0x7f7b0403c9f0 2024-03-06T07:27:48.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b11a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b040e1030 0x7f7b040e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f6efaa00640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ed806b6d0 0x7f6ed806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2a8001030 con 0x7fb2d006dcf0 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b11a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b040e1030 0x7f7b040e2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50518/0 (socket says 172.21.15.97:50518) 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b11a00640 1 -- 172.21.15.97:0/2015712663 learned_addr learned my addr 172.21.15.97:0/2015712663 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7f7b0be00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b04056790 0x7f7b040e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 shutdown_connections 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7fea48a00640 1 --2- 172.21.15.97:0/748800403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 0x7fea440cf8f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 >> 172.21.15.97:0/748800403 conn(0x7fea440d3d80 msgr2=0x7fea440d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f7b11000640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b040d4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f6efa000640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f6ed8073870 0x7f6ed8075c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f7b11a00640 1 -- 172.21.15.97:0/2015712663 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 msgr2=0x7f7b040d4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f7b11a00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b0403c9f0 0x7f7b040d4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f7b11a00640 1 -- 172.21.15.97:0/2015712663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b04056790 msgr2=0x7f7b040e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f7b11a00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b04056790 0x7f7b040e2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7f7b11a00640 1 -- 172.21.15.97:0/2015712663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b040e1ee0 con 0x7f7b040e1030 2024-03-06T07:27:48.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 shutdown_connections 2024-03-06T07:27:48.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== osd.3 v2:172.21.15.97:6800/3169996968 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe5580020b0 con 0x7fe53c077ad0 2024-03-06T07:27:48.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc15afa0 con 0x7faebc154870 2024-03-06T07:27:48.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.807+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc1e7c30 con 0x7faebc154870 2024-03-06T07:27:48.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.809+0000 7f7b11a00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b040e1030 0x7f7b040e2670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c06bfd0 tx=0x7f7b0c0683e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.809+0000 7f6efaa00640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ed806b6d0 0x7f6ed806db90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4005880 tx=0x7f6ee4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.808+0000 7fea48a00640 1 -- 172.21.15.97:0/748800403 wait complete. 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.809+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c08f020 con 0x7f7b040e1030 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.809+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c092310 con 0x7f7b040e1030 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.809+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c092720 con 0x7f7b040e1030 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.811+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b040e1c00 con 0x7f7b040e1030 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.811+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b040e19a0 con 0x7f7b040e1030 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.811+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc76c01a070 con 0x7fc7740d0590 2024-03-06T07:27:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.812+0000 7f6efa000640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f6ed8073870 0x7f6ed8075c90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.812+0000 7fea48a00640 1 Processor -- start 2024-03-06T07:27:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fea48a00640 1 -- start start 2024-03-06T07:27:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.812+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7b04005360 con 0x7f7b040e1030 2024-03-06T07:27:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.811+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae98001030 con 0x7faebc154870 2024-03-06T07:27:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faeb801d070 con 0x7faebc154870 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fc76b400640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc75006b5a0 0x7fc75006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc77413c790 con 0x7fc7740d0590 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.811+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2c00067b0 con 0x7fb2d006dcf0 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.812+0000 7fb2c7e00640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2c00abff0 con 0x7fb2d006dcf0 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fb2c7e00640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7fb2b00736f0 0x7fb2b0075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2c0030e50 con 0x7fb2b00736f0 2024-03-06T07:27:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.813+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2c00ac3e0 con 0x7fb2d006dcf0 2024-03-06T07:27:48.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc76c089350 con 0x7fc7740d0590 2024-03-06T07:27:48.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b0c092020 con 0x7f7b040e1030 2024-03-06T07:27:48.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fc773400640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc75006b5a0 0x7fc75006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fb2cf400640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7faeb6000640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faea006b4d0 0x7faea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== osd.4 v2:172.21.15.135:6808/3314712387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6ef4020e50 con 0x7f6ed8073870 2024-03-06T07:27:48.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fea48a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440d1270 0x7fea440c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fea48a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440d2340 0x7fea440c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fea48a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 0x7fea440c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fea48a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440cf460 con 0x7fea4415e090 2024-03-06T07:27:48.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fea48a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440d2180 con 0x7fea440d2340 2024-03-06T07:27:48.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fea48a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea440d2790 con 0x7fea440d1270 2024-03-06T07:27:48.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.814+0000 7fb2d5000640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7fb2b00736f0 0x7fb2b0075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.815+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faeb808a4b0 con 0x7faebc154870 2024-03-06T07:27:48.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.815+0000 7faec1e00640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faea006b4d0 0x7faea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.815+0000 7f7b08c00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7aec06b500 0x7f7aec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.815+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b0c105230 con 0x7f7b040e1030 2024-03-06T07:27:49.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.815+0000 7fea43400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440d1270 0x7fea440c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fb2cf400640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2b006b4d0 0x7fb2b006d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0095e0 tx=0x7fb2bc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fc76b400640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fc750073740 0x7fc750075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc76c009e50 con 0x7fc750073740 2024-03-06T07:27:49.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc76c090050 con 0x7fc7740d0590 2024-03-06T07:27:49.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fc773400640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc75006b5a0 0x7fc75006da60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0017a0 tx=0x7fc75c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fea43400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440d1270 0x7fea440c9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:33002/0 (socket says 172.21.15.97:33002) 2024-03-06T07:27:49.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fb2d5000640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7fb2b00736f0 0x7fb2b0075b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fea42a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440d2340 0x7fea440c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fea42a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440d2340 0x7fea440c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36662/0 (socket says 172.21.15.97:36662) 2024-03-06T07:27:49.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fea42a00640 1 -- 172.21.15.97:0/2336291343 learned_addr learned my addr 172.21.15.97:0/2336291343 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fea43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 0x7fea440c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7faeb6000640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7faea00736f0 0x7faea0075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7fc772a00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fc750073740 0x7fc750075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.816+0000 7faec1e00640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faea006b4d0 0x7faea006d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faeac036000 tx=0x7faeac008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea00761c0 con 0x7faea00736f0 2024-03-06T07:27:49.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeb8091050 con 0x7faebc154870 2024-03-06T07:27:49.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7f7b08c00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f7aec073780 0x7f7aec075ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7aec076250 con 0x7f7aec073780 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b0c105620 con 0x7f7b040e1030 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7fea42a00640 1 -- 172.21.15.97:0/2336291343 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440d1270 msgr2=0x7fea440c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7fea42a00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440d1270 0x7fea440c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7fea42a00640 1 -- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 msgr2=0x7fea440c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7fea42a00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea4415e090 0x7fea440c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7fea42a00640 1 -- 172.21.15.97:0/2336291343 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440c6ff0 con 0x7fea440d2340 2024-03-06T07:27:49.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7fea43400640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440d1270 0x7fea440c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7faec1400640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7faea00736f0 0x7faea0075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.817+0000 7f7b11000640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7aec06b500 0x7f7aec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fea42a00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440d2340 0x7fea440c7430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea3000c450 tx=0x7fea3000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fc772a00640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fc750073740 0x7fc750075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea300052d0 con 0x7fea440d2340 2024-03-06T07:27:49.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7f7b0be00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f7aec073780 0x7f7aec075ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30005470 con 0x7fea440d2340 2024-03-06T07:27:49.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea440c7fa0 con 0x7fea440d2340 2024-03-06T07:27:49.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea440c8560 con 0x7fea440d2340 2024-03-06T07:27:49.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.818+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30012870 con 0x7fea440d2340 2024-03-06T07:27:49.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.819+0000 7faec1400640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7faea00736f0 0x7faea0075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.819+0000 7f7b11000640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7aec06b500 0x7f7aec06d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b00036000 tx=0x7f7b0003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.819+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== osd.7 v2:172.21.15.135:6816/3893802268 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2c0030e50 con 0x7fb2b00736f0 2024-03-06T07:27:49.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.819+0000 7f7b0be00640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f7aec073780 0x7f7aec075ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.821+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7aec076250 con 0x7f7aec073780 2024-03-06T07:27:49.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.820+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fea4404f330 con 0x7fea440d2340 2024-03-06T07:27:49.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.822+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== osd.5 v2:172.21.15.135:6800/2374976391 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faea00761c0 con 0x7faea00736f0 2024-03-06T07:27:49.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.821+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== osd.6 v2:172.21.15.135:6824/3958261964 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc76c009e50 con 0x7fc750073740 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.823+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea30012a10 con 0x7fea440d2340 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.826+0000 7fea3b400640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2406b4d0 0x7fea2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.826+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea3008a8b0 con 0x7fea440d2340 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.826+0000 7fea3b400640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fea240736d0 0x7fea24075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.826+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea240761a0 con 0x7fea240736d0 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.826+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea3008ac60 con 0x7fea440d2340 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.829+0000 7fea43400640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2406b4d0 0x7fea2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.830+0000 7fea43400640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2406b4d0 0x7fea2406d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fea34036000 tx=0x7fea3403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.831+0000 7fea43e00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fea240736d0 0x7fea24075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.832+0000 7fea43e00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fea240736d0 0x7fea24075af0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.833+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== osd.1 v2:172.21.15.97:6808/2322813614 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea240761a0 con 0x7fea240736d0 2024-03-06T07:27:49.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.840+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe534002dd0 con 0x7fe53c077ad0 2024-03-06T07:27:49.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.842+0000 7fe556000640 1 -- 172.21.15.97:0/840978860 <== osd.3 v2:172.21.15.97:6800/3169996968 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe534002dd0 con 0x7fe53c077ad0 2024-03-06T07:27:49.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.842+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe53c077ad0 msgr2=0x7fe53c079ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.842+0000 7fe565200640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe53c077ad0 0x7fe53c079ef0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.845+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53c06f930 msgr2=0x7fe53c071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.845+0000 7fe565200640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53c06f930 0x7fe53c071df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe550007fd0 tx=0x7fe550009040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.845+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600cf320 msgr2=0x7fe5600c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.845+0000 7fe565200640 1 --2- 172.21.15.97:0/840978860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5600cf320 0x7fe5600c6ab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe55800af70 tx=0x7fe55800e4a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.845+0000 7fe55e800640 1 -- 172.21.15.97:0/840978860 reap_dead start 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.846+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 shutdown_connections 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.846+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 >> 172.21.15.97:0/840978860 conn(0x7fe5600cfa70 msgr2=0x7fe56016a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.848+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 shutdown_connections 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.848+0000 7fe565200640 1 -- 172.21.15.97:0/840978860 wait complete. 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.851+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb2a8002dd0 con 0x7fb2b00736f0 2024-03-06T07:27:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.853+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6efc0b2190 con 0x7f6ed8073870 2024-03-06T07:27:49.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.853+0000 7fb2c7e00640 1 -- 172.21.15.97:0/3685732332 <== osd.7 v2:172.21.15.135:6816/3893802268 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb2a8002dd0 con 0x7fb2b00736f0 2024-03-06T07:27:49.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.852+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7b04054b40 con 0x7f7aec073780 2024-03-06T07:27:49.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.853+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae98002dd0 con 0x7faea00736f0 2024-03-06T07:27:49.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.853+0000 7f7b08c00640 1 -- 172.21.15.97:0/2015712663 <== osd.0 v2:172.21.15.97:6819/1306424436 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7b04054b40 con 0x7f7aec073780 2024-03-06T07:27:49.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.853+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7fb2b00736f0 msgr2=0x7fb2b0075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.853+0000 7fb2d7000640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7fb2b00736f0 0x7fb2b0075b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7f6ef3400640 1 -- 172.21.15.97:0/1434603436 <== osd.4 v2:172.21.15.135:6808/3314712387 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6efc0b2190 con 0x7f6ed8073870 2024-03-06T07:27:49.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7faeb6000640 1 -- 172.21.15.97:0/1834489874 <== osd.5 v2:172.21.15.135:6800/2374976391 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae98002dd0 con 0x7faea00736f0 2024-03-06T07:27:49.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f7aec073780 msgr2=0x7f7aec075ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7f7b12400640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f7aec073780 0x7f7aec075ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2b006b4d0 msgr2=0x7fb2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7fb2d7000640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2b006b4d0 0x7fb2b006d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0095e0 tx=0x7fb2bc009190 comp rx=0 tx=0).stop 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 msgr2=0x7fb2d00b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7fb2d7000640 1 --2- 172.21.15.97:0/3685732332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2d006dcf0 0x7fb2d00b0600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c3e0 tx=0x7fb2c000c8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.854+0000 7fb2d5000640 1 -- 172.21.15.97:0/3685732332 reap_dead start 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7aec06b500 msgr2=0x7f7aec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b12400640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7aec06b500 0x7f7aec06d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b00036000 tx=0x7f7b0003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b040e1030 msgr2=0x7f7b040e2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b12400640 1 --2- 172.21.15.97:0/2015712663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b040e1030 0x7f7b040e2670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c06bfd0 tx=0x7f7b0c0683e0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7faea00736f0 msgr2=0x7faea0075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7faec4800640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7faea00736f0 0x7faea0075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b11a00640 1 -- 172.21.15.97:0/2015712663 reap_dead start 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 shutdown_connections 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 >> 172.21.15.97:0/2015712663 conn(0x7f7b04056cb0 msgr2=0x7f7b04008480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faea006b4d0 msgr2=0x7faea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7faec4800640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faea006b4d0 0x7faea006d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faeac036000 tx=0x7faeac008030 comp rx=0 tx=0).stop 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 shutdown_connections 2024-03-06T07:27:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 >> 172.21.15.97:0/3685732332 conn(0x7fb2d00b1730 msgr2=0x7fb2d0072530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faebc154870 msgr2=0x7faebc140190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7faec4800640 1 --2- 172.21.15.97:0/1834489874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faebc154870 0x7faebc140190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faeb8007e90 tx=0x7faeb800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.855+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 shutdown_connections 2024-03-06T07:27:49.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f6ed8073870 msgr2=0x7f6ed8075c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7fb2d7000640 1 -- 172.21.15.97:0/3685732332 wait complete. 2024-03-06T07:27:49.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f6efbe00640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f6ed8073870 0x7f6ed8075c90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 shutdown_connections 2024-03-06T07:27:49.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7faec2800640 1 -- 172.21.15.97:0/1834489874 reap_dead start 2024-03-06T07:27:49.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f7b12400640 1 -- 172.21.15.97:0/2015712663 wait complete. 2024-03-06T07:27:49.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ed806b6d0 msgr2=0x7f6ed806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 shutdown_connections 2024-03-06T07:27:49.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 >> 172.21.15.97:0/1834489874 conn(0x7faebc0cc4f0 msgr2=0x7faebc1536f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f6efbe00640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ed806b6d0 0x7f6ed806db90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4005880 tx=0x7f6ee4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.857+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 shutdown_connections 2024-03-06T07:27:49.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.857+0000 7fa5f8800640 1 Processor -- start 2024-03-06T07:27:49.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.857+0000 7faec4800640 1 -- 172.21.15.97:0/1834489874 wait complete. 2024-03-06T07:27:49.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.857+0000 7fa5f8800640 1 -- start start 2024-03-06T07:27:49.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.856+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6efc141380 msgr2=0x7f6efc15a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.858+0000 7f6efbe00640 1 --2- 172.21.15.97:0/1434603436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6efc141380 0x7f6efc15a450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009fd0 tx=0x7f6ef400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.858+0000 7f6efb400640 1 -- 172.21.15.97:0/1434603436 reap_dead start 2024-03-06T07:27:49.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.858+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 shutdown_connections 2024-03-06T07:27:49.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.858+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 >> 172.21.15.97:0/1434603436 conn(0x7f6efc0b1490 msgr2=0x7f6efc0708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.858+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 shutdown_connections 2024-03-06T07:27:49.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.858+0000 7f6efbe00640 1 -- 172.21.15.97:0/1434603436 wait complete. 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.859+0000 7fa5f8800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa5f00cd5b0 0x7fa5f0145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.859+0000 7fa5f8800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa5f015cc70 0x7fa5f0162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f8800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f0166ec0 con 0x7fa5f00cd990 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f8800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f006cbe0 con 0x7fa5f015cc70 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f8800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f006cec0 con 0x7fa5f00cd5b0 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f5e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa5f00cd5b0 0x7fa5f0145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa5f015cc70 0x7fa5f0162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa5f015cc70 0x7fa5f0162fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50526/0 (socket says 172.21.15.97:50526) 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.860+0000 7fa5f6800640 1 -- 172.21.15.97:0/3607314034 learned_addr learned my addr 172.21.15.97:0/3607314034 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.861+0000 7fa5f6800640 1 -- 172.21.15.97:0/3607314034 >> v2:172.21.15.135:3300/0 conn(0x7fa5f00cd5b0 msgr2=0x7fa5f0145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.861+0000 7fa5f6800640 1 --2- 172.21.15.97:0/3607314034 >> v2:172.21.15.135:3300/0 conn(0x7fa5f00cd5b0 0x7fa5f0145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.861+0000 7fa5f6800640 1 -- 172.21.15.97:0/3607314034 >> v1:172.21.15.135:6789/0 conn(0x7fa5f00cd990 legacy=0x7fa5f015c560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.861+0000 7fa5f6800640 1 -- 172.21.15.97:0/3607314034 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f006d1a0 con 0x7fa5f015cc70 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.863+0000 7fa5f6800640 1 --2- 172.21.15.97:0/3607314034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f015cc70 0x7fa5f0162fb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0073670 tx=0x7fa5e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cd26c2f023597c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.863+0000 7fa5f4a00640 1 -- 172.21.15.97:0/3607314034 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4035020 con 0x7fa5f015cc70 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.863+0000 7fa5f4a00640 1 -- 172.21.15.97:0/3607314034 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4031870 con 0x7fa5f015cc70 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.863+0000 7fa5f4a00640 1 -- 172.21.15.97:0/3607314034 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4031c40 con 0x7fa5f015cc70 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.864+0000 7fa5f8800640 1 -- 172.21.15.97:0/3607314034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f015cc70 msgr2=0x7fa5f0162fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.864+0000 7fa5f8800640 1 --2- 172.21.15.97:0/3607314034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f015cc70 0x7fa5f0162fb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0073670 tx=0x7fa5e402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.865+0000 7fa5f8800640 1 -- 172.21.15.97:0/3607314034 shutdown_connections 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.865+0000 7fa5f8800640 1 --2- 172.21.15.97:0/3607314034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f015cc70 0x7fa5f0162fb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.865+0000 7fa5f8800640 1 --2- 172.21.15.97:0/3607314034 >> v2:172.21.15.135:3300/0 conn(0x7fa5f00cd5b0 0x7fa5f0145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.865+0000 7fa5f8800640 1 -- 172.21.15.97:0/3607314034 >> 172.21.15.97:0/3607314034 conn(0x7fa5f00cc4f0 msgr2=0x7fa5f00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.866+0000 7fa5f8800640 1 -- 172.21.15.97:0/3607314034 shutdown_connections 2024-03-06T07:27:49.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.866+0000 7fa5f8800640 1 -- 172.21.15.97:0/3607314034 wait complete. 2024-03-06T07:27:49.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.868+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc77404f330 con 0x7fc750073740 2024-03-06T07:27:49.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.868+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fea440d3b60 con 0x7fea240736d0 2024-03-06T07:27:49.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.868+0000 7fea3b400640 1 -- 172.21.15.97:0/2336291343 <== osd.1 v2:172.21.15.97:6808/2322813614 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fea440d3b60 con 0x7fea240736d0 2024-03-06T07:27:49.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.869+0000 7fa5f8800640 1 Processor -- start 2024-03-06T07:27:49.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.869+0000 7fc76b400640 1 -- 172.21.15.97:0/2627589458 <== osd.6 v2:172.21.15.135:6824/3958261964 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc77404f330 con 0x7fc750073740 2024-03-06T07:27:49.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.869+0000 7fa5f8800640 1 -- start start 2024-03-06T07:27:49.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.870+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fea240736d0 msgr2=0x7fea24075af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.870+0000 7fea48a00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fea240736d0 0x7fea24075af0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.871+0000 7fa5f8800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f00cd5b0 0x7fa5f0142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.871+0000 7fa5f8800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f015cc70 0x7fa5f015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.871+0000 7fa5f8800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f01fbeb0 0x7fa5f00cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.871+0000 7fa5f8800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f01634f0 con 0x7fa5f00cd5b0 2024-03-06T07:27:49.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.871+0000 7fa5f8800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f00cde40 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.871+0000 7fa5f8800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f0163670 con 0x7fa5f015cc70 2024-03-06T07:27:49.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fc750073740 msgr2=0x7fc750075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc77a200640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fc750073740 0x7fc750075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f01fbeb0 0x7fa5f00cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc75006b5a0 msgr2=0x7fc75006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc77a200640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc75006b5a0 0x7fc75006da60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0017a0 tx=0x7fc75c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f01fbeb0 0x7fa5f00cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36680/0 (socket says 172.21.15.97:36680) 2024-03-06T07:27:49.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fa5f6800640 1 -- 172.21.15.97:0/3977077834 learned_addr learned my addr 172.21.15.97:0/3977077834 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7740d0590 msgr2=0x7fc77413cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc77a200640 1 --2- 172.21.15.97:0/2627589458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7740d0590 0x7fc77413cbb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00c310 tx=0x7fc76c00c7d0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.872+0000 7fc773e00640 1 -- 172.21.15.97:0/2627589458 reap_dead start 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 shutdown_connections 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 >> 172.21.15.97:0/2627589458 conn(0x7fc7740cfa70 msgr2=0x7fc77413e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 shutdown_connections 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fc77a200640 1 -- 172.21.15.97:0/2627589458 wait complete. 2024-03-06T07:27:49.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2406b4d0 msgr2=0x7fea2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fea48a00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2406b4d0 0x7fea2406d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fea34036000 tx=0x7fea3403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fa5f5e00640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f00cd5b0 0x7fa5f0142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fa5f5400640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f015cc70 0x7fa5f015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440d2340 msgr2=0x7fea440c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.873+0000 7fea48a00640 1 --2- 172.21.15.97:0/2336291343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440d2340 0x7fea440c7430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea3000c450 tx=0x7fea3000c910 comp rx=0 tx=0).stop 2024-03-06T07:27:49.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.874+0000 7fea43e00640 1 -- 172.21.15.97:0/2336291343 reap_dead start 2024-03-06T07:27:49.035 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2024-03-06T07:27:49.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:27:49.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.874+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 shutdown_connections 2024-03-06T07:27:49.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.874+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 >> 172.21.15.97:0/2336291343 conn(0x7fea440d3d80 msgr2=0x7fea440b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 shutdown_connections 2024-03-06T07:27:49.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fea48a00640 1 -- 172.21.15.97:0/2336291343 wait complete. 2024-03-06T07:27:49.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fa5f6800640 1 -- 172.21.15.97:0/3977077834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f015cc70 msgr2=0x7fa5f015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fa5f6800640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f015cc70 0x7fa5f015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fa5f6800640 1 -- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f00cd5b0 msgr2=0x7fa5f0142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fa5f6800640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f00cd5b0 0x7fa5f0142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.875+0000 7fa5f6800640 1 -- 172.21.15.97:0/3977077834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f01fc450 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.876+0000 7fa5f6800640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f01fbeb0 0x7fa5f00cbae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009960 tx=0x7fa5e40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.878+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4040070 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.878+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4030d00 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.878+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4035040 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.878+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f01fc730 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.880+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f01fccf0 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.882+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5f004f330 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.882+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5e4038020 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.884+0000 7fa5ea000640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d406b520 0x7fa5d406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.885+0000 7fa5f5e00640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d406b520 0x7fa5d406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.885+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5e40aa570 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.039 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2024-03-06T07:27:49.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:27:49.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:27:49.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.886+0000 7fa5ea000640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fa5d4073720 0x7fa5d4075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.886+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5e4041e50 con 0x7fa5d4073720 2024-03-06T07:27:49.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.886+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e40b1060 con 0x7fa5f01fbeb0 2024-03-06T07:27:49.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.886+0000 7fa5f5e00640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d406b520 0x7fa5d406d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00f510 tx=0x7fa5ec0077b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.887+0000 7fa5f5400640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fa5d4073720 0x7fa5d4075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.888+0000 7fa5f5400640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fa5d4073720 0x7fa5d4075b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.889+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== osd.2 v2:172.21.15.97:6816/75411953 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa5e4041e50 con 0x7fa5d4073720 2024-03-06T07:27:49.042 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:27:49.043 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:27:49.044 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2024-03-06T07:27:49.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.906+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa5f0144a00 con 0x7fa5d4073720 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.906+0000 7fa5ea000640 1 -- 172.21.15.97:0/3977077834 <== osd.2 v2:172.21.15.97:6816/75411953 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa5f0144a00 con 0x7fa5d4073720 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fa5d4073720 msgr2=0x7fa5d4075b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fa5d4073720 0x7fa5d4075b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d406b520 msgr2=0x7fa5d406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d406b520 0x7fa5d406d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00f510 tx=0x7fa5ec0077b0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f01fbeb0 msgr2=0x7fa5f00cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 --2- 172.21.15.97:0/3977077834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f01fbeb0 0x7fa5f00cbae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009960 tx=0x7fa5e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f6800640 1 -- 172.21.15.97:0/3977077834 reap_dead start 2024-03-06T07:27:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 shutdown_connections 2024-03-06T07:27:49.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 >> 172.21.15.97:0/3977077834 conn(0x7fa5f00cc4f0 msgr2=0x7fa5f0165060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 shutdown_connections 2024-03-06T07:27:49.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:48.907+0000 7fa5f8800640 1 -- 172.21.15.97:0/3977077834 wait complete. 2024-03-06T07:27:49.049 INFO:teuthology.orchestra.run.smithi097.stdout:60129542150 2024-03-06T07:27:49.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:27:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.386+0000 7fd62aa00640 1 Processor -- start 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.386+0000 7fd62aa00640 1 -- start start 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.388+0000 7fd62aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd61c101010 0x7fd61c037c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.388+0000 7fd62aa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c054430 0x7fd61c04f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.388+0000 7fd62aa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c052cf0 con 0x7fd61c033920 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.388+0000 7fd62aa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61c0341c0 con 0x7fd61c101010 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.388+0000 7fd62aa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c0548a0 con 0x7fd61c054430 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.389+0000 7fd629600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd61c101010 0x7fd61c037c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.389+0000 7fd629600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd61c101010 0x7fd61c037c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50534/0 (socket says 172.21.15.97:50534) 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.389+0000 7fd629600640 1 -- 172.21.15.97:0/1601459512 learned_addr learned my addr 172.21.15.97:0/1601459512 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.389+0000 7fd629600640 1 -- 172.21.15.97:0/1601459512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c054430 msgr2=0x7fd61c04f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.390+0000 7fd629600640 1 --2- 172.21.15.97:0/1601459512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c054430 0x7fd61c04f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.390+0000 7f8d14a00640 1 Processor -- start 2024-03-06T07:27:49.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.390+0000 7fd629600640 1 -- 172.21.15.97:0/1601459512 >> v1:172.21.15.97:6789/0 conn(0x7fd61c033920 legacy=0x7fd61c053d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.390+0000 7fd629600640 1 -- 172.21.15.97:0/1601459512 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c052f60 con 0x7fd61c101010 2024-03-06T07:27:49.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.393+0000 7fd629600640 1 --2- 172.21.15.97:0/1601459512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c101010 0x7fd61c037c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd6180088f0 tx=0x7fd61802dfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235a6691fa9f03c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.393+0000 7fd623e00640 1 -- 172.21.15.97:0/1601459512 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61802ed90 con 0x7fd61c101010 2024-03-06T07:27:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7fd623e00640 1 -- 172.21.15.97:0/1601459512 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61800ae50 con 0x7fd61c101010 2024-03-06T07:27:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7fd623e00640 1 -- 172.21.15.97:0/1601459512 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd618036610 con 0x7fd61c101010 2024-03-06T07:27:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.395+0000 7fd62aa00640 1 -- 172.21.15.97:0/1601459512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c101010 msgr2=0x7fd61c037c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.395+0000 7fd62aa00640 1 --2- 172.21.15.97:0/1601459512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c101010 0x7fd61c037c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd6180088f0 tx=0x7fd61802dfe0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.395+0000 7fd62aa00640 1 -- 172.21.15.97:0/1601459512 shutdown_connections 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7fd62aa00640 1 --2- 172.21.15.97:0/1601459512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c054430 0x7fd61c04f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7fd62aa00640 1 --2- 172.21.15.97:0/1601459512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c101010 0x7fd61c037c40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7fd62aa00640 1 -- 172.21.15.97:0/1601459512 >> 172.21.15.97:0/1601459512 conn(0x7fd61c009fb0 msgr2=0x7fd61c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7fd62aa00640 1 -- 172.21.15.97:0/1601459512 shutdown_connections 2024-03-06T07:27:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7fd62aa00640 1 -- 172.21.15.97:0/1601459512 wait complete. 2024-03-06T07:27:49.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.400+0000 7fd62aa00640 1 Processor -- start 2024-03-06T07:27:49.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.400+0000 7fd62aa00640 1 -- start start 2024-03-06T07:27:49.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd62aa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd61c033920 0x7fd61c0fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd62aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 0x7fd61c04eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd62aa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c101010 0x7fd61c18bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd62aa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c053800 con 0x7fd61c054430 2024-03-06T07:27:49.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7ff91a200640 1 Processor -- start 2024-03-06T07:27:49.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd62aa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c053520 con 0x7fd61c033920 2024-03-06T07:27:49.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd62aa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c038180 con 0x7fd61c101010 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.390+0000 7f8d14a00640 1 -- start start 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.392+0000 7f8d14a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d1006d670 0x7f8d100cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.392+0000 7f8d14a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1015a780 con 0x7f8d100cc800 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.392+0000 7f8d14a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1015aa60 con 0x7f8d1006e390 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.392+0000 7f8d14a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1015ad40 con 0x7f8d1006d670 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.393+0000 7f8d0fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8d100cc800 0x7f8d100cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45126/0 (socket says 172.21.15.97:45126) 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.393+0000 7f8d0fe00640 1 -- 172.21.15.97:0/2811545455 learned_addr learned my addr 172.21.15.97:0/2811545455 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.393+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916147501 0 0) 0x7f8d1015a780 con 0x7f8d100cc800 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf4003520 con 0x7f8d100cc800 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162653828 0 0) 0x7f8cf4003520 con 0x7f8d100cc800 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d1006d670 msgr2=0x7f8d100cc2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d1006d670 0x7f8d100cc2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 >> v1:172.21.15.135:6789/0 conn(0x7f8d1006e390 legacy=0x7f8d100cec20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1015b020 con 0x7f8d100cc800 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8d080019f0 con 0x7f8d100cc800 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7ff91a200640 1 -- start start 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.402+0000 7fd628c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 0x7fd61c04eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7fd628c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 0x7fd61c04eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50548/0 (socket says 172.21.15.97:50548) 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7fd628c00640 1 -- 172.21.15.97:0/28838030 learned_addr learned my addr 172.21.15.97:0/28838030 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7fd629600640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd61c033920 0x7fd61c0fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7fd62a000640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c101010 0x7fd61c18bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 >> v1:172.21.15.97:6789/0 conn(0x7f8d100cc800 legacy=0x7f8d100cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf40039e0 0x7f8cf4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cf4004800 0x7f8cf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cf4005210 0x7f8cf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1015ad40 con 0x7f8cf4005210 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf402e170 con 0x7f8cf4004800 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.394+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf402e370 con 0x7f8cf40039e0 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.395+0000 7f8d0fe00640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cf4005210 0x7f8cf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7f8d0fe00640 1 -- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf40039e0 msgr2=0x7f8cf4003de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7f8d0fe00640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf40039e0 0x7f8cf4003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7f8d0fe00640 1 -- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cf4004800 msgr2=0x7f8cf4004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7f8d0fe00640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cf4004800 0x7f8cf4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.396+0000 7f8d0fe00640 1 -- 172.21.15.97:0/2811545455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1015b020 con 0x7f8cf4005210 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.400+0000 7f8d0fe00640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cf4005210 0x7f8cf400b9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800a910 tx=0x7f8cf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c276224129b081bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.400+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800a4e0 con 0x7f8cf4005210 2024-03-06T07:27:49.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.400+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d08034630 con 0x7f8cf4005210 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.401+0000 7f8d0e000640 1 -- 172.21.15.97:0/2811545455 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800a4e0 con 0x7f8cf4005210 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7f8d14a00640 1 -- 172.21.15.97:0/2811545455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cf4005210 msgr2=0x7f8cf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2811545455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cf4005210 0x7f8cf400b9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800a910 tx=0x7f8cf4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.403+0000 7f8d0fe00640 1 -- 172.21.15.97:0/2811545455 reap_dead start 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.404+0000 7fd628c00640 1 -- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c101010 msgr2=0x7fd61c18bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.404+0000 7fd628c00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c101010 0x7fd61c18bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.404+0000 7fd628c00640 1 -- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd61c033920 msgr2=0x7fd61c0fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.404+0000 7fd628c00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd61c033920 0x7fd61c0fe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.404+0000 7fd628c00640 1 -- 172.21.15.97:0/28838030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c04f430 con 0x7fd61c054430 2024-03-06T07:27:49.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.404+0000 7fd628c00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 0x7fd61c04eba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd61000b3e0 tx=0x7fd61000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7ff91a200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff91415dfc0 0x7ff91406d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7ff91a200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff9140b1d60 0x7ff9140add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7ff91a200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9140ad860 con 0x7ff914161cd0 2024-03-06T07:27:49.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7ff91a200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9141621b0 con 0x7ff9140b1d60 2024-03-06T07:27:49.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7ff91a200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff914162490 con 0x7ff91415dfc0 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.407+0000 7ff912a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff914161cd0 0x7ff9140b1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45136/0 (socket says 172.21.15.97:45136) 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.407+0000 7ff912a00640 1 -- 172.21.15.97:0/1856798504 learned_addr learned my addr 172.21.15.97:0/1856798504 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.407+0000 7ff913e00640 1 --2- 172.21.15.97:0/1856798504 >> v2:172.21.15.97:3300/0 conn(0x7ff9140b1d60 0x7ff9140add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.405+0000 7f8d14a00640 1 -- 172.21.15.97:0/2811545455 shutdown_connections 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.405+0000 7f8d14a00640 1 -- 172.21.15.97:0/2811545455 >> 172.21.15.97:0/2811545455 conn(0x7f8d1006afc0 msgr2=0x7f8d1006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.405+0000 7f8d14a00640 1 -- 172.21.15.97:0/2811545455 shutdown_connections 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.405+0000 7f8d14a00640 1 -- 172.21.15.97:0/2811545455 wait complete. 2024-03-06T07:27:49.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610015020 con 0x7fd61c054430 2024-03-06T07:27:49.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.406+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd610011850 con 0x7fd61c054430 2024-03-06T07:27:49.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.407+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c04f710 con 0x7fd61c054430 2024-03-06T07:27:49.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.407+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610011c40 con 0x7fd61c054430 2024-03-06T07:27:49.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.407+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c04fcd0 con 0x7fd61c054430 2024-03-06T07:27:49.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.408+0000 7f8d14a00640 1 Processor -- start 2024-03-06T07:27:49.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.408+0000 7ff913400640 1 --2- 172.21.15.97:0/1856798504 >> v2:172.21.15.135:3300/0 conn(0x7ff91415dfc0 0x7ff91406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.409+0000 7ff913e00640 1 -- 172.21.15.97:0/1856798504 >> v2:172.21.15.135:3300/0 conn(0x7ff91415dfc0 msgr2=0x7ff91406d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.409+0000 7ff913e00640 1 --2- 172.21.15.97:0/1856798504 >> v2:172.21.15.135:3300/0 conn(0x7ff91415dfc0 0x7ff91406d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.409+0000 7ff913e00640 1 -- 172.21.15.97:0/1856798504 >> v1:172.21.15.97:6789/0 conn(0x7ff914161cd0 legacy=0x7ff9140b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.410+0000 7f8d14a00640 1 -- start start 2024-03-06T07:27:49.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.409+0000 7ff913e00640 1 -- 172.21.15.97:0/1856798504 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914162770 con 0x7ff9140b1d60 2024-03-06T07:27:49.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.410+0000 7f2169400640 1 Processor -- start 2024-03-06T07:27:49.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f2169400640 1 -- start start 2024-03-06T07:27:49.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c005390 con 0x7fd61c054430 2024-03-06T07:27:49.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7ff913e00640 1 --2- 172.21.15.97:0/1856798504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff9140add40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff9140c4ac0 tx=0x7ff90c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd8ccc6d178dd0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f8d14a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf4005210 0x7f8d10148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f8d14a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 0x7f8d101476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f8d14a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d10147c30 0x7f8d101556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f8d14a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1006e770 con 0x7f8d100ae4c0 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f8d14a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d100ce780 con 0x7f8d10147c30 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.411+0000 7f8d14a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1006e8f0 con 0x7f8cf4005210 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.412+0000 7f8d0ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 0x7f8d101476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.412+0000 7f8d0ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 0x7f8d101476f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50572/0 (socket says 172.21.15.97:50572) 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.412+0000 7f8d0ea00640 1 -- 172.21.15.97:0/2414717216 learned_addr learned my addr 172.21.15.97:0/2414717216 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0f400640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf4005210 0x7f8d10148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f2169400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f21640d0640 0x7f21640d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f2169400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640d0f60 con 0x7f21640b39e0 2024-03-06T07:27:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f2169400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640d27a0 con 0x7f21640b3580 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f2169400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21640cee70 con 0x7f21640d0640 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0fe00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d10147c30 0x7f8d101556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0ea00640 1 -- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf4005210 msgr2=0x7f8d10148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0ea00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf4005210 0x7f8d10148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0ea00640 1 -- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d10147c30 msgr2=0x7f8d101556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0ea00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d10147c30 0x7f8d101556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0ea00640 1 -- 172.21.15.97:0/2414717216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d10149120 con 0x7f8d100ae4c0 2024-03-06T07:27:49.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.413+0000 7f8d0f400640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf4005210 0x7f8d10148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.414+0000 7f2163e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f21640d0640 0x7f21640d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.414+0000 7f8d0ea00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 0x7f8d101476f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00b410 tx=0x7f8cfc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.414+0000 7f2163400640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f21640b3580 0x7f21640cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:41608/0 (socket says 172.21.15.97:41608) 2024-03-06T07:27:49.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.414+0000 7f2163e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f21640d0640 0x7f21640d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36726/0 (socket says 172.21.15.97:36726) 2024-03-06T07:27:49.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.414+0000 7f2163e00640 1 -- 172.21.15.97:0/3248979616 learned_addr learned my addr 172.21.15.97:0/3248979616 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.415+0000 7ff912000640 1 -- 172.21.15.97:0/1856798504 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c039070 con 0x7ff9140b1d60 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.415+0000 7f2163e00640 1 -- 172.21.15.97:0/3248979616 >> v1:172.21.15.135:6789/0 conn(0x7f21640b3580 legacy=0x7f21640cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.415+0000 7f2162a00640 1 -- 172.21.15.97:0/3248979616 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981745995 0 0) 0x7f21640d27a0 con 0x7f21640b3580 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.415+0000 7f2163e00640 1 -- 172.21.15.97:0/3248979616 >> v1:172.21.15.97:6789/0 conn(0x7f21640b39e0 legacy=0x7f21640d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.415+0000 7f2163e00640 1 -- 172.21.15.97:0/3248979616 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21640d2a80 con 0x7f21640d0640 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.417+0000 7f2163e00640 1 --2- 172.21.15.97:0/3248979616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21640d0640 0x7f21640d0a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2154009870 tx=0x7f215402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b6a067b2df76ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.418+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd610018020 con 0x7fd61c054430 2024-03-06T07:27:49.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.419+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc015020 con 0x7f8d100ae4c0 2024-03-06T07:27:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.419+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc011830 con 0x7f8d100ae4c0 2024-03-06T07:27:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.419+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc011c80 con 0x7f8d100ae4c0 2024-03-06T07:27:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.419+0000 7ff912000640 1 -- 172.21.15.97:0/1856798504 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90c02cc90 con 0x7ff9140b1d60 2024-03-06T07:27:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.419+0000 7fd621600640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd60406b5a0 0x7fd60406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.419+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd61008a670 con 0x7fd61c054430 2024-03-06T07:27:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff912000640 1 -- 172.21.15.97:0/1856798504 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c034640 con 0x7ff9140b1d60 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7fd629600640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd60406b5a0 0x7fd60406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff91a200640 1 -- 172.21.15.97:0/1856798504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 msgr2=0x7ff9140add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff91a200640 1 --2- 172.21.15.97:0/1856798504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff9140add40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff9140c4ac0 tx=0x7ff90c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff91a200640 1 -- 172.21.15.97:0/1856798504 shutdown_connections 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff91a200640 1 --2- 172.21.15.97:0/1856798504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff9140add40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff91a200640 1 --2- 172.21.15.97:0/1856798504 >> v2:172.21.15.135:3300/0 conn(0x7ff91415dfc0 0x7ff91406d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.420+0000 7ff91a200640 1 -- 172.21.15.97:0/1856798504 >> 172.21.15.97:0/1856798504 conn(0x7ff9140cfa70 msgr2=0x7ff9140cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.421+0000 7f2162a00640 1 -- 172.21.15.97:0/3248979616 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154034070 con 0x7f21640d0640 2024-03-06T07:27:49.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.421+0000 7f2162a00640 1 -- 172.21.15.97:0/3248979616 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154030860 con 0x7f21640d0640 2024-03-06T07:27:49.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.421+0000 7f2162a00640 1 -- 172.21.15.97:0/3248979616 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154030c30 con 0x7f21640d0640 2024-03-06T07:27:49.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.421+0000 7ff91a200640 1 -- 172.21.15.97:0/1856798504 shutdown_connections 2024-03-06T07:27:49.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.422+0000 7f2169400640 1 -- 172.21.15.97:0/3248979616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21640d0640 msgr2=0x7f21640d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.422+0000 7ff91a200640 1 -- 172.21.15.97:0/1856798504 wait complete. 2024-03-06T07:27:49.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.422+0000 7f2169400640 1 --2- 172.21.15.97:0/3248979616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21640d0640 0x7f21640d0a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2154009870 tx=0x7f215402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.422+0000 7f2169400640 1 -- 172.21.15.97:0/3248979616 shutdown_connections 2024-03-06T07:27:49.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.422+0000 7f2169400640 1 --2- 172.21.15.97:0/3248979616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21640d0640 0x7f21640d0a20 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2154009870 tx=0x7f215402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7f2169400640 1 -- 172.21.15.97:0/3248979616 >> 172.21.15.97:0/3248979616 conn(0x7f216409cfb0 msgr2=0x7f21640d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7fc67d800640 1 Processor -- start 2024-03-06T07:27:49.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7fd629600640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd60406b5a0 0x7fd60406da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd618037000 tx=0x7fd618032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.421+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d10146650 con 0x7f8d100ae4c0 2024-03-06T07:27:49.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.421+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d10146c10 con 0x7f8d100ae4c0 2024-03-06T07:27:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d100b3470 con 0x7f8d100ae4c0 2024-03-06T07:27:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7f2169400640 1 -- 172.21.15.97:0/3248979616 shutdown_connections 2024-03-06T07:27:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7f2169400640 1 -- 172.21.15.97:0/3248979616 wait complete. 2024-03-06T07:27:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.424+0000 7ff91a200640 1 Processor -- start 2024-03-06T07:27:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.424+0000 7ff91a200640 1 -- start start 2024-03-06T07:27:49.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd61001f020 con 0x7fd61c054430 2024-03-06T07:27:49.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff91a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff914204450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff91a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff91415dfc0 0x7ff914201f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff91a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff914161cd0 0x7ff91415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff91a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91406ccc0 con 0x7ff9140b1d60 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff91a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91406d5b0 con 0x7ff914161cd0 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff91a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91406ce40 con 0x7ff91415dfc0 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.426+0000 7ff913400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff914204450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7f2169400640 1 Processor -- start 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.423+0000 7fc67d800640 1 -- start start 2024-03-06T07:27:49.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.425+0000 7fc67d800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 0x7fc678145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.425+0000 7fc67d800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc67815cc70 0x7fc678162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.425+0000 7fc67d800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc678166ec0 con 0x7fc6780cd990 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.425+0000 7fc67d800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67806cbe0 con 0x7fc67815cc70 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.425+0000 7fc67d800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc67806cec0 con 0x7fc6780cd5b0 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7fc676a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 0x7fc678145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7fc676a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 0x7fc678145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:33044/0 (socket says 172.21.15.97:33044) 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7f2169400640 1 -- start start 2024-03-06T07:27:49.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7fc676a00640 1 -- 172.21.15.97:0/4059372724 learned_addr learned my addr 172.21.15.97:0/4059372724 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.428+0000 7f2169400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 0x7f216415df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2169400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21640b39e0 0x7f21640b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2169400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2164154c30 0x7f2164160a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2169400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640cfa10 con 0x7f21640b3580 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2169400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640ce7f0 con 0x7f2164154c30 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2169400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21640cfb90 con 0x7f21640b39e0 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2163e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 0x7f216415df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2163e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 0x7f216415df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50600/0 (socket says 172.21.15.97:50600) 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2163e00640 1 -- 172.21.15.97:0/3359374772 learned_addr learned my addr 172.21.15.97:0/3359374772 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2163400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21640b39e0 0x7f21640b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7f2168a00640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2164154c30 0x7f2164160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f2163e00640 1 -- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21640b39e0 msgr2=0x7f21640b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f2163e00640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21640b39e0 0x7f21640b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f2163e00640 1 -- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2164154c30 msgr2=0x7f2164160a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f2163e00640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2164154c30 0x7f2164160a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f2163e00640 1 -- 172.21.15.97:0/3359374772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216415c690 con 0x7f21640b3580 2024-03-06T07:27:49.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7fc676a00640 1 -- 172.21.15.97:0/4059372724 >> v2:172.21.15.97:3300/0 conn(0x7fc67815cc70 msgr2=0x7fc678162fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.428+0000 7fc677400640 1 --2- 172.21.15.97:0/4059372724 >> v2:172.21.15.97:3300/0 conn(0x7fc67815cc70 0x7fc678162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.428+0000 7fc676a00640 1 --2- 172.21.15.97:0/4059372724 >> v2:172.21.15.97:3300/0 conn(0x7fc67815cc70 0x7fc678162fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.428+0000 7fc676a00640 1 -- 172.21.15.97:0/4059372724 >> v1:172.21.15.97:6789/0 conn(0x7fc6780cd990 legacy=0x7fc67815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.428+0000 7fc676a00640 1 -- 172.21.15.97:0/4059372724 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67806d1a0 con 0x7fc6780cd5b0 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7fc676a00640 1 --2- 172.21.15.97:0/4059372724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 0x7fc678145b70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc670009e50 tx=0x7fc67002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8827d9843fd5c35c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7f8f52200640 1 Processor -- start 2024-03-06T07:27:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.424+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cfc018020 con 0x7f8d100ae4c0 2024-03-06T07:27:49.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7fc675600640 1 -- 172.21.15.97:0/4059372724 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67002ca70 con 0x7fc6780cd5b0 2024-03-06T07:27:49.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7fc675600640 1 -- 172.21.15.97:0/4059372724 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67002cc10 con 0x7fc6780cd5b0 2024-03-06T07:27:49.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7fc675600640 1 -- 172.21.15.97:0/4059372724 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700343f0 con 0x7fc6780cd5b0 2024-03-06T07:27:49.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f8d07400640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8cf006b570 0x7f8cf006da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.430+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cfc08a070 con 0x7f8d100ae4c0 2024-03-06T07:27:49.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7f8f52200640 1 -- start start 2024-03-06T07:27:49.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.432+0000 7f8d0f400640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8cf006b570 0x7f8cf006da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.432+0000 7fc67d800640 1 -- 172.21.15.97:0/4059372724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 msgr2=0x7fc678145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.432+0000 7fc67d800640 1 --2- 172.21.15.97:0/4059372724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 0x7fc678145b70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc670009e50 tx=0x7fc67002bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.433+0000 7f2163e00640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 0x7f216415df90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f215402bf30 tx=0x7f2154004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.427+0000 7ff913400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff914204450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50584/0 (socket says 172.21.15.97:50584) 2024-03-06T07:27:49.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7ff913400640 1 -- 172.21.15.97:0/1238965861 learned_addr learned my addr 172.21.15.97:0/1238965861 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7ff913400640 1 -- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff91415dfc0 msgr2=0x7ff914201f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.429+0000 7ff912a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff91415dfc0 0x7ff914201f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.431+0000 7ff913e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff914161cd0 0x7ff91415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.433+0000 7f8d0f400640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8cf006b570 0x7f8cf006da30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8d00006fd0 tx=0x7f8d00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7fc67d800640 1 -- 172.21.15.97:0/4059372724 shutdown_connections 2024-03-06T07:27:49.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f52200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8f4c0ae6b0 0x7f8f4c0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f52200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8f4c0b3a60 0x7f8f4c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7fc67d800640 1 --2- 172.21.15.97:0/4059372724 >> v2:172.21.15.97:3300/0 conn(0x7fc67815cc70 0x7fc678162fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7fc67d800640 1 --2- 172.21.15.97:0/4059372724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6780cd5b0 0x7fc678145b70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7fc67d800640 1 -- 172.21.15.97:0/4059372724 >> 172.21.15.97:0/4059372724 conn(0x7fc6780cc4f0 msgr2=0x7fc6780cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f52200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c0af300 con 0x7f8f4c1623d0 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f52200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f4c0aeaf0 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f52200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f4c0aed40 con 0x7f8f4c0ae6b0 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f50e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f8f4c1623d0 0x7f8f4c0d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:41616/0 (socket says 172.21.15.97:41616) 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f50e00640 1 -- 172.21.15.97:0/1308790691 learned_addr learned my addr 172.21.15.97:0/1308790691 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.434+0000 7f8f4be00640 1 --2- 172.21.15.97:0/1308790691 >> v2:172.21.15.135:3300/0 conn(0x7f8f4c0ae6b0 0x7f8f4c0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7fc67d800640 1 -- 172.21.15.97:0/4059372724 shutdown_connections 2024-03-06T07:27:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7ff913400640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff91415dfc0 0x7ff914201f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7ff913400640 1 -- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff914161cd0 msgr2=0x7ff91415b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7ff913400640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff914161cd0 0x7ff91415b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7ff913400640 1 -- 172.21.15.97:0/1238965861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914204c80 con 0x7ff9140b1d60 2024-03-06T07:27:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7fc67d800640 1 -- 172.21.15.97:0/4059372724 wait complete. 2024-03-06T07:27:49.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7ff912a00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff91415dfc0 0x7ff914201f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7ff913e00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff914161cd0 0x7ff91415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7f8f4b400640 1 -- 172.21.15.97:0/1308790691 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177599641 0 0) 0x7f8f4c0af300 con 0x7f8f4c1623d0 2024-03-06T07:27:49.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.435+0000 7f8f4b400640 1 -- 172.21.15.97:0/1308790691 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34003520 con 0x7f8f4c1623d0 2024-03-06T07:27:49.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7f8f4be00640 1 -- 172.21.15.97:0/1308790691 >> v2:172.21.15.97:3300/0 conn(0x7f8f4c0b3a60 msgr2=0x7f8f4c0ce230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:49.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7ff913400640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff914204450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff90400c820 tx=0x7ff90400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7f8f4b400640 1 -- 172.21.15.97:0/1308790691 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2244262736 0 0) 0x7f8f34003520 con 0x7f8f4c1623d0 2024-03-06T07:27:49.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7f8f51800640 1 --2- 172.21.15.97:0/1308790691 >> v2:172.21.15.97:3300/0 conn(0x7f8f4c0b3a60 0x7f8f4c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7f8f4be00640 1 --2- 172.21.15.97:0/1308790691 >> v2:172.21.15.97:3300/0 conn(0x7f8f4c0b3a60 0x7f8f4c0ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7f8f4be00640 1 -- 172.21.15.97:0/1308790691 >> v1:172.21.15.135:6789/0 conn(0x7f8f4c1623d0 legacy=0x7f8f4c0d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.436+0000 7f8f4be00640 1 -- 172.21.15.97:0/1308790691 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c0af020 con 0x7f8f4c0ae6b0 2024-03-06T07:27:49.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.437+0000 7fc67d800640 1 Processor -- start 2024-03-06T07:27:49.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.437+0000 7f8f51800640 1 --2- 172.21.15.97:0/1308790691 >> v2:172.21.15.97:3300/0 conn(0x7f8f4c0b3a60 0x7f8f4c0ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.437+0000 7fc67d800640 1 -- start start 2024-03-06T07:27:49.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.437+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904015070 con 0x7ff9140b1d60 2024-03-06T07:27:49.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7ff91a200640 1 -- 172.21.15.97:0/1238965861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff914204f60 con 0x7ff9140b1d60 2024-03-06T07:27:49.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7ff91a200640 1 -- 172.21.15.97:0/1238965861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff914204990 con 0x7ff9140b1d60 2024-03-06T07:27:49.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154034040 con 0x7f21640b3580 2024-03-06T07:27:49.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f8f4be00640 1 --2- 172.21.15.97:0/1308790691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0ae6b0 0x7f8f4c0b3520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c06cff0 tx=0x7f8f3802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6da96d0de5c968c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7fc67d800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6780cd5b0 0x7fc67815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7fc67d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 0x7fc67815b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7fc67d800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc67815cc70 0x7fc678160f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7fc67d800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781634f0 con 0x7fc6780cd990 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7fc67d800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780cde10 con 0x7fc6780cd5b0 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7fc67d800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc678163670 con 0x7fc67815cc70 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154039d90 con 0x7f21640b3580 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154044660 con 0x7f21640b3580 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216415c970 con 0x7f21640b3580 2024-03-06T07:27:49.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216415cf30 con 0x7f21640b3580 2024-03-06T07:27:49.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90400ce10 con 0x7ff9140b1d60 2024-03-06T07:27:49.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7f8f4b400640 1 -- 172.21.15.97:0/1308790691 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38037070 con 0x7f8f4c0ae6b0 2024-03-06T07:27:49.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7f8f4b400640 1 -- 172.21.15.97:0/1308790691 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38030510 con 0x7f8f4c0ae6b0 2024-03-06T07:27:49.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7f8f4b400640 1 -- 172.21.15.97:0/1308790691 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400019f0 con 0x7f8f4c0ae6b0 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d100b3470 con 0x7f8d100ae4c0 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7fc676000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 0x7fc67815b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7fc676000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 0x7fc67815b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50606/0 (socket says 172.21.15.97:50606) 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7fc676a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6780cd5b0 0x7fc67815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7fc676a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6780cd5b0 0x7fc67815b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36760/0 (socket says 172.21.15.97:36760) 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7fc676a00640 1 -- 172.21.15.97:0/4161961913 learned_addr learned my addr 172.21.15.97:0/4161961913 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.439+0000 7fc677400640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc67815cc70 0x7fc678160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.438+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90401e060 con 0x7ff9140b1d60 2024-03-06T07:27:49.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.440+0000 7fc676000640 1 -- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc67815cc70 msgr2=0x7fc678160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.440+0000 7fc676000640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc67815cc70 0x7fc678160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.440+0000 7fc676000640 1 -- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6780cd5b0 msgr2=0x7fc67815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.440+0000 7fc676000640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6780cd5b0 0x7fc67815b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.440+0000 7fc676000640 1 -- 172.21.15.97:0/4161961913 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678162280 con 0x7fc6780cd990 2024-03-06T07:27:49.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.440+0000 7fc676000640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 0x7fc67815b7f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc66400c410 tx=0x7fc66400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7f8f52200640 1 -- 172.21.15.97:0/1308790691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0ae6b0 msgr2=0x7f8f4c0b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7f8f52200640 1 --2- 172.21.15.97:0/1308790691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0ae6b0 0x7f8f4c0b3520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c06cff0 tx=0x7f8f3802b710 comp rx=0 tx=0).stop 2024-03-06T07:27:49.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213c005290 con 0x7f21640b3580 2024-03-06T07:27:49.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f8f52200640 1 -- 172.21.15.97:0/1308790691 shutdown_connections 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f8f52200640 1 --2- 172.21.15.97:0/1308790691 >> v2:172.21.15.97:3300/0 conn(0x7f8f4c0b3a60 0x7f8f4c0ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f8f52200640 1 --2- 172.21.15.97:0/1308790691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0ae6b0 0x7f8f4c0b3520 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664017070 con 0x7fc6780cd990 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6640035f0 con 0x7fc6780cd990 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640039e0 con 0x7fc6780cd990 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc678161cc0 con 0x7fc6780cd990 2024-03-06T07:27:49.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f8f52200640 1 -- 172.21.15.97:0/1308790691 >> 172.21.15.97:0/1308790691 conn(0x7f8f4c06afc0 msgr2=0x7f8f4c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.441+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6781340c0 con 0x7fc6780cd990 2024-03-06T07:27:49.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f8f52200640 1 -- 172.21.15.97:0/1308790691 shutdown_connections 2024-03-06T07:27:49.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.443+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc650005290 con 0x7fc6780cd990 2024-03-06T07:27:49.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.442+0000 7f8f52200640 1 -- 172.21.15.97:0/1308790691 wait complete. 2024-03-06T07:27:49.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.444+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91404f330 con 0x7ff9140b1d60 2024-03-06T07:27:49.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.446+0000 7f8f52200640 1 Processor -- start 2024-03-06T07:27:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.446+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff904007830 con 0x7ff9140b1d60 2024-03-06T07:27:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.446+0000 7f8d75e00640 1 Processor -- start 2024-03-06T07:27:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.446+0000 7f8f52200640 1 -- start start 2024-03-06T07:27:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.446+0000 7f8d75e00640 1 -- start start 2024-03-06T07:27:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.446+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc664003b80 con 0x7fc6780cd990 2024-03-06T07:27:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.448+0000 7f8f52200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 0x7f8f4c162a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.448+0000 7f8d75e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d70162440 0x7f8d70169060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.448+0000 7f8f52200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f4c1623d0 0x7f8f4c14aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.448+0000 7f8f52200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f4c1633f0 0x7f8f4c1fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.447+0000 7ff90b400640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8f406b500 0x7ff8f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8d6ea00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f8d7015e360 0x7f8d700d12e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:41626/0 (socket says 172.21.15.97:41626) 2024-03-06T07:27:49.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8d6ea00640 1 -- 172.21.15.97:0/488147259 learned_addr learned my addr 172.21.15.97:0/488147259 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:49.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f52200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4c0ce770 con 0x7f8f4c1623d0 2024-03-06T07:27:49.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f52200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4c06d5c0 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f52200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4c0ce8f0 con 0x7f8f4c1633f0 2024-03-06T07:27:49.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f50e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 0x7f8f4c162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700cf2b0 con 0x7f8d700cec10 2024-03-06T07:27:49.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f50e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 0x7f8f4c162a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36772/0 (socket says 172.21.15.97:36772) 2024-03-06T07:27:49.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f50e00640 1 -- 172.21.15.97:0/2631275730 learned_addr learned my addr 172.21.15.97:0/2631275730 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.447+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff90408a0b0 con 0x7ff9140b1d60 2024-03-06T07:27:49.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700cf590 con 0x7f8d7015e360 2024-03-06T07:27:49.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700cf870 con 0x7f8d70162440 2024-03-06T07:27:49.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8d6f400640 1 --2- 172.21.15.97:0/488147259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d70162440 0x7f8d70169060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7f8f51800640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f4c1633f0 0x7f8f4c1fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8d6d600640 1 -- 172.21.15.97:0/488147259 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162600864 0 0) 0x7f8d700cf590 con 0x7f8d7015e360 2024-03-06T07:27:49.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8d6d600640 1 -- 172.21.15.97:0/488147259 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d54003520 con 0x7f8d7015e360 2024-03-06T07:27:49.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8d6f400640 1 -- 172.21.15.97:0/488147259 >> v1:172.21.15.135:6789/0 conn(0x7f8d7015e360 legacy=0x7f8d700d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.449+0000 7fc66ea00640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc65806b4d0 0x7fc65806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8d6f400640 1 -- 172.21.15.97:0/488147259 >> v1:172.21.15.97:6789/0 conn(0x7f8d700cec10 legacy=0x7f8d70161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8d6f400640 1 -- 172.21.15.97:0/488147259 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700cfb50 con 0x7f8d70162440 2024-03-06T07:27:49.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8f50e00640 1 -- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f4c1633f0 msgr2=0x7f8f4c1fb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7f8f50e00640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f4c1633f0 0x7f8f4c1fb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.450+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6640897a0 con 0x7fc6780cd990 2024-03-06T07:27:49.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.451+0000 7fc676a00640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc65806b4d0 0x7fc65806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.451+0000 7f8f50e00640 1 -- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f4c1623d0 msgr2=0x7f8f4c14aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.451+0000 7f8f50e00640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f4c1623d0 0x7f8f4c14aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.451+0000 7f8f50e00640 1 -- 172.21.15.97:0/2631275730 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c15a860 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7ff912a00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8f406b500 0x7ff8f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7f8d6f400640 1 --2- 172.21.15.97:0/488147259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d70162440 0x7f8d70169060 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d700d4fc0 tx=0x7f8d6802be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59c23c86571fb50c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7f8f50e00640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 0x7f8f4c162a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000cc10 tx=0x7f8f4000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7f8d6d600640 1 -- 172.21.15.97:0/488147259 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68039070 con 0x7f8d70162440 2024-03-06T07:27:49.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7f8d6d600640 1 -- 172.21.15.97:0/488147259 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6802cc20 con 0x7f8d70162440 2024-03-06T07:27:49.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7fc676a00640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc65806b4d0 0x7fc65806d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc67002bf10 tx=0x7fc670030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.452+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40017070 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8d6d600640 1 -- 172.21.15.97:0/488147259 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68039070 con 0x7f8d70162440 2024-03-06T07:27:49.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7ff912a00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8f406b500 0x7ff8f406d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9140b5840 tx=0x7ff900009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000ce20 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40012640 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2154037070 con 0x7f21640b3580 2024-03-06T07:27:49.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d70162440 msgr2=0x7f8d70169060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8d75e00640 1 --2- 172.21.15.97:0/488147259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d70162440 0x7f8d70169060 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d700d4fc0 tx=0x7f8d6802be70 comp rx=0 tx=0).stop 2024-03-06T07:27:49.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 shutdown_connections 2024-03-06T07:27:49.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8d75e00640 1 --2- 172.21.15.97:0/488147259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d70162440 0x7f8d70169060 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.453+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 >> 172.21.15.97:0/488147259 conn(0x7f8d700d3f50 msgr2=0x7f8d700d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.454+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4c162fa0 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.454+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 shutdown_connections 2024-03-06T07:27:49.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.454+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4c15a580 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.455+0000 7f215be00640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f214406b5a0 0x7f214406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.456+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21540aca60 con 0x7f21640b3580 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.456+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21540d9e10 con 0x7f21640b3580 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.456+0000 7f2163400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f214406b5a0 0x7f214406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.457+0000 7f2163400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f214406b5a0 0x7f214406da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21641fba30 tx=0x7f2150007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.455+0000 7f8d75e00640 1 -- 172.21.15.97:0/488147259 wait complete. 2024-03-06T07:27:49.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.457+0000 7f8d75e00640 1 Processor -- start 2024-03-06T07:27:49.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.456+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4c04f330 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.458+0000 7f8d75e00640 1 -- start start 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.458+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f4001a070 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.460+0000 7f8d75e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d700cec10 0x7f8d70200320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.460+0000 7f8d75e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d7015e360 0x7f8d7015cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.460+0000 7f8d75e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 0x7f8d7015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.460+0000 7f8f48c00640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 0x7f8d7015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 0x7f8d7015d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50632/0 (socket says 172.21.15.97:50632) 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d6f400640 1 -- 172.21.15.97:0/2870848102 learned_addr learned my addr 172.21.15.97:0/2870848102 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d6e000640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d7015e360 0x7f8d7015cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d75e00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700d2e20 con 0x7f8d70162440 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d75e00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700d2ca0 con 0x7f8d7015e360 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8d75e00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700d2fa0 con 0x7f8d700cec10 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.462+0000 7f8d6ea00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d700cec10 0x7f8d70200320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.461+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f4008a190 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.462+0000 7f8f4be00640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.463+0000 7f8f4be00640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8f38034000 tx=0x7f8f3802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.464+0000 7f8d6ea00640 1 -- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d7015e360 msgr2=0x7f8d7015cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.464+0000 7f8d6ea00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d7015e360 0x7f8d7015cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.464+0000 7f8d6ea00640 1 -- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 msgr2=0x7f8d7015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.464+0000 7f8d6ea00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 0x7f8d7015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.464+0000 7f8d6ea00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70200860 con 0x7f8d700cec10 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.464+0000 7f8d6f400640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 0x7f8d7015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.465+0000 7f8d6ea00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d700cec10 0x7f8d70200320 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c550 tx=0x7f8d6000ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.465+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60017070 con 0x7f8d700cec10 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.465+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d60005ce0 con 0x7f8d700cec10 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.465+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d600072f0 con 0x7f8d700cec10 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.465+0000 7f8d75e00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d700cbec0 con 0x7f8d700cec10 2024-03-06T07:27:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.465+0000 7f8d75e00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d700cc480 con 0x7f8d700cec10 2024-03-06T07:27:49.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.466+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc664020020 con 0x7fc6780cd990 2024-03-06T07:27:49.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.467+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d44005290 con 0x7f8d700cec10 2024-03-06T07:27:49.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.470+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d6001a070 con 0x7f8d700cec10 2024-03-06T07:27:49.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.471+0000 7f8d67400640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d5006b5a0 0x7f8d5006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.472+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d60089790 con 0x7f8d700cec10 2024-03-06T07:27:49.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.472+0000 7f8d6e000640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d5006b5a0 0x7f8d5006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.473+0000 7f9a5b200640 1 Processor -- start 2024-03-06T07:27:49.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.474+0000 7f8d6e000640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d5006b5a0 0x7f8d5006da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d700726c0 tx=0x7f8d5c007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.475+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff904058f20 con 0x7ff9140b1d60 2024-03-06T07:27:49.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.474+0000 7f9a5b200640 1 -- start start 2024-03-06T07:27:49.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.475+0000 7f9a5b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9a5c161580 0x7f9a5c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.475+0000 7f9a5b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.476+0000 7f9a5b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c161960 con 0x7f9a5c0d28a0 2024-03-06T07:27:49.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.476+0000 7f9a5b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c0cf6f0 con 0x7f9a5c161580 2024-03-06T07:27:49.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.476+0000 7f9a5b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0cf9d0 con 0x7f9a5c0d1e90 2024-03-06T07:27:49.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.477+0000 7f9a59400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.477+0000 7f9a59400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c0d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:33102/0 (socket says 172.21.15.97:33102) 2024-03-06T07:27:49.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.477+0000 7f9a59400640 1 -- 172.21.15.97:0/3037492618 learned_addr learned my addr 172.21.15.97:0/3037492618 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.477+0000 7f9a59e00640 1 --2- 172.21.15.97:0/3037492618 >> v2:172.21.15.97:3300/0 conn(0x7f9a5c161580 0x7f9a5c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.477+0000 7f9a59400640 1 -- 172.21.15.97:0/3037492618 >> v2:172.21.15.97:3300/0 conn(0x7f9a5c161580 msgr2=0x7f9a5c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.477+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f40058920 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.478+0000 7f9a59400640 1 --2- 172.21.15.97:0/3037492618 >> v2:172.21.15.97:3300/0 conn(0x7f9a5c161580 0x7f9a5c0d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.478+0000 7f9a59400640 1 -- 172.21.15.97:0/3037492618 >> v1:172.21.15.97:6789/0 conn(0x7f9a5c0d28a0 legacy=0x7f9a5c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.478+0000 7f9a59400640 1 -- 172.21.15.97:0/3037492618 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c0cfcb0 con 0x7f9a5c0d1e90 2024-03-06T07:27:49.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.478+0000 7f9a59e00640 1 --2- 172.21.15.97:0/3037492618 >> v2:172.21.15.97:3300/0 conn(0x7f9a5c161580 0x7f9a5c0d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.480+0000 7f9a59400640 1 --2- 172.21.15.97:0/3037492618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c0d2360 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009870 tx=0x7f9a4802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565b4911acc8b462 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.484+0000 7f9a58a00640 1 -- 172.21.15.97:0/3037492618 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48034070 con 0x7f9a5c0d1e90 2024-03-06T07:27:49.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.486+0000 7f9a58a00640 1 -- 172.21.15.97:0/3037492618 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a48030840 con 0x7f9a5c0d1e90 2024-03-06T07:27:49.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.486+0000 7f9a58a00640 1 -- 172.21.15.97:0/3037492618 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48030c10 con 0x7f9a5c0d1e90 2024-03-06T07:27:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f9a5b200640 1 -- 172.21.15.97:0/3037492618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 msgr2=0x7f9a5c0d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3037492618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c0d2360 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009870 tx=0x7f9a4802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d60058030 con 0x7f8d700cec10 2024-03-06T07:27:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f9a5b200640 1 -- 172.21.15.97:0/3037492618 shutdown_connections 2024-03-06T07:27:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3037492618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c0d2360 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3037492618 >> v2:172.21.15.97:3300/0 conn(0x7f9a5c161580 0x7f9a5c0d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.487+0000 7f9a5b200640 1 -- 172.21.15.97:0/3037492618 >> 172.21.15.97:0/3037492618 conn(0x7f9a5c0d3d80 msgr2=0x7f9a5c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.488+0000 7f9a5b200640 1 -- 172.21.15.97:0/3037492618 shutdown_connections 2024-03-06T07:27:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.488+0000 7f9a5b200640 1 -- 172.21.15.97:0/3037492618 wait complete. 2024-03-06T07:27:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.490+0000 7f9a5b200640 1 Processor -- start 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.491+0000 7f9a5b200640 1 -- start start 2024-03-06T07:27:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.493+0000 7f9a5b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.493+0000 7f9a5b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a5c0d28a0 0x7f9a5c0b9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.493+0000 7f9a5b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 0x7f9a5c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.494+0000 7f9a5b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0cf3a0 con 0x7f9a5c0d28a0 2024-03-06T07:27:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.494+0000 7f9a5b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b39a0 con 0x7f9a5c161580 2024-03-06T07:27:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.494+0000 7f9a5a800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 0x7f9a5c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.494+0000 7f9a5a800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 0x7f9a5c0ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:36790/0 (socket says 172.21.15.97:36790) 2024-03-06T07:27:49.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.494+0000 7f9a5a800640 1 -- 172.21.15.97:0/3817612481 learned_addr learned my addr 172.21.15.97:0/3817612481 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:49.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b4ed0 con 0x7f9a5c0d1e90 2024-03-06T07:27:49.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a59e00640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a59400640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a5c0d28a0 0x7f9a5c0b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a5a800640 1 -- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 msgr2=0x7f9a5c153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a5a800640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a5a800640 1 -- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a5c0d28a0 msgr2=0x7f9a5c0b9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a5a800640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a5c0d28a0 0x7f9a5c0b9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.495+0000 7f9a5a800640 1 -- 172.21.15.97:0/3817612481 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c0b74f0 con 0x7f9a5c161580 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.496+0000 7f9a59400640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a5c0d28a0 0x7f9a5c0b9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.496+0000 7f9a59e00640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.497+0000 7f9a5a800640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 0x7f9a5c0ccd10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c710 tx=0x7f9a5400cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.497+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54016020 con 0x7f9a5c161580 2024-03-06T07:27:49.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.497+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c0cc2f0 con 0x7f9a5c161580 2024-03-06T07:27:49.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.498+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c1535b0 con 0x7f9a5c161580 2024-03-06T07:27:49.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.498+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54006d50 con 0x7f9a5c161580 2024-03-06T07:27:49.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.498+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54003bb0 con 0x7f9a5c161580 2024-03-06T07:27:49.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.502+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a34005290 con 0x7f9a5c161580 2024-03-06T07:27:49.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.511+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a5400a850 con 0x7f9a5c161580 2024-03-06T07:27:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.515+0000 7f9a52000640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a3c06b4d0 0x7f9a3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.515+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a54089e60 con 0x7f9a5c161580 2024-03-06T07:27:49.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.516+0000 7f9a59e00640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a3c06b4d0 0x7f9a3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.520+0000 7f9a59e00640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a3c06b4d0 0x7f9a3c06d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009810 tx=0x7f9a4c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.522+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a54020020 con 0x7f9a5c161580 2024-03-06T07:27:49.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.864+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd61c0fee00 con 0x7fd61c054430 2024-03-06T07:27:49.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.865+0000 7fd621600640 1 -- 172.21.15.97:0/28838030 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd610058e80 con 0x7fd61c054430 2024-03-06T07:27:49.867 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.867+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8d1006d450 con 0x7f8d100ae4c0 2024-03-06T07:27:49.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.868+0000 7f8d07400640 1 -- 172.21.15.97:0/2414717216 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d100b3470 con 0x7f8d100ae4c0 2024-03-06T07:27:49.871 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2024-03-06T07:27:49.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.868+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd60406b5a0 msgr2=0x7fd60406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.868+0000 7fd62aa00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd60406b5a0 0x7fd60406da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd618037000 tx=0x7fd618032000 comp rx=0 tx=0).stop 2024-03-06T07:27:49.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.869+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 msgr2=0x7fd61c04eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.869+0000 7fd62aa00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 0x7fd61c04eba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd61000b3e0 tx=0x7fd61000b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.869+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 shutdown_connections 2024-03-06T07:27:49.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd60406b5a0 0x7fd60406da60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd61c101010 0x7fd61c18bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd61c054430 0x7fd61c04eba0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 --2- 172.21.15.97:0/28838030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd61c033920 0x7fd61c0fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 >> 172.21.15.97:0/28838030 conn(0x7fd61c009fb0 msgr2=0x7fd61c036b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 shutdown_connections 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.870+0000 7fd62aa00640 1 -- 172.21.15.97:0/28838030 wait complete. 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.872+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8cf006b570 msgr2=0x7f8cf006da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.872+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8cf006b570 0x7f8cf006da30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8d00006fd0 tx=0x7f8d00008040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.872+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 msgr2=0x7f8d101476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.872+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 0x7f8d101476f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00b410 tx=0x7f8cfc00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.873+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 shutdown_connections 2024-03-06T07:27:49.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.873+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8cf006b570 0x7f8cf006da30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.873+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d10147c30 0x7f8d101556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.873+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d100ae4c0 0x7f8d101476f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.874+0000 7f8d14a00640 1 --2- 172.21.15.97:0/2414717216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cf4005210 0x7f8d10148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.874+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 >> 172.21.15.97:0/2414717216 conn(0x7f8d1006afc0 msgr2=0x7f8d100b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.875+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 shutdown_connections 2024-03-06T07:27:49.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.875+0000 7f8d14a00640 1 -- 172.21.15.97:0/2414717216 wait complete. 2024-03-06T07:27:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.883+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f213c005850 con 0x7f21640b3580 2024-03-06T07:27:49.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.885+0000 7f215be00640 1 -- 172.21.15.97:0/3359374772 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f215407b8c0 con 0x7f21640b3580 2024-03-06T07:27:49.888 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.890+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f214406b5a0 msgr2=0x7f214406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.890+0000 7f2169400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f214406b5a0 0x7f214406da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21641fba30 tx=0x7f2150007e40 comp rx=0 tx=0).stop 2024-03-06T07:27:49.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.890+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 msgr2=0x7f216415df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.890+0000 7f2169400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 0x7f216415df90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f215402bf30 tx=0x7f2154004730 comp rx=0 tx=0).stop 2024-03-06T07:27:49.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.891+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 shutdown_connections 2024-03-06T07:27:49.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.891+0000 7f2169400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f214406b5a0 0x7f214406da60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.891+0000 7f2169400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2164154c30 0x7f2164160a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.891+0000 7f2169400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21640b39e0 0x7f21640b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.891+0000 7f2169400640 1 --2- 172.21.15.97:0/3359374772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21640b3580 0x7f216415df90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.891+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 >> 172.21.15.97:0/3359374772 conn(0x7f216409cfb0 msgr2=0x7f216415b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.896+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 shutdown_connections 2024-03-06T07:27:49.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.896+0000 7f2169400640 1 -- 172.21.15.97:0/3359374772 wait complete. 2024-03-06T07:27:49.899 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2024-03-06T07:27:49.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.896+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc650005850 con 0x7fc6780cd990 2024-03-06T07:27:49.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.897+0000 7fc66ea00640 1 -- 172.21.15.97:0/4161961913 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc664058170 con 0x7fc6780cd990 2024-03-06T07:27:49.900 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:49.900 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.898+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff9140b2560 con 0x7ff9140b1d60 2024-03-06T07:27:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.899+0000 7ff90b400640 1 -- 172.21.15.97:0/1238965861 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff904058820 con 0x7ff9140b1d60 2024-03-06T07:27:49.901 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2024-03-06T07:27:49.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.901+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8f406b500 msgr2=0x7ff8f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.901+0000 7ff908c00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8f406b500 0x7ff8f406d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9140b5840 tx=0x7ff900009040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.901+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 msgr2=0x7ff914204450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.901+0000 7ff908c00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff914204450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff90400c820 tx=0x7ff90400cce0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 shutdown_connections 2024-03-06T07:27:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8f406b500 0x7ff8f406d9c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff914161cd0 0x7ff91415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff91415dfc0 0x7ff914201f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 --2- 172.21.15.97:0/1238965861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9140b1d60 0x7ff914204450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 >> 172.21.15.97:0/1238965861 conn(0x7ff9140cfa70 msgr2=0x7ff9140ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 shutdown_connections 2024-03-06T07:27:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7ff908c00640 1 -- 172.21.15.97:0/1238965861 wait complete. 2024-03-06T07:27:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc65806b4d0 msgr2=0x7fc65806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7fc67d800640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc65806b4d0 0x7fc65806d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc67002bf10 tx=0x7fc670030000 comp rx=0 tx=0).stop 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 msgr2=0x7fc67815b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.902+0000 7fc67d800640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 0x7fc67815b7f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc66400c410 tx=0x7fc66400c810 comp rx=0 tx=0).stop 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.903+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 shutdown_connections 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.903+0000 7fc67d800640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc65806b4d0 0x7fc65806d990 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc67002bf10 tx=0x7fc670030000 comp rx=0 tx=0).stop 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.903+0000 7fc67d800640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc67815cc70 0x7fc678160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.903+0000 7fc67d800640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6780cd990 0x7fc67815b7f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.903+0000 7fc67d800640 1 --2- 172.21.15.97:0/4161961913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6780cd5b0 0x7fc67815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.903+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 >> 172.21.15.97:0/4161961913 conn(0x7fc6780cc4f0 msgr2=0x7fc678163aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.905+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 shutdown_connections 2024-03-06T07:27:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.905+0000 7fc67d800640 1 -- 172.21.15.97:0/4161961913 wait complete. 2024-03-06T07:27:49.908 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.7 2024-03-06T07:27:49.908 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.909+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8f4c1631d0 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.910+0000 7f8f48c00640 1 -- 172.21.15.97:0/2631275730 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8f4005c9c0 con 0x7f8f4c0b3a60 2024-03-06T07:27:49.912 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.913+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f2c06b4d0 msgr2=0x7f8f2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.913+0000 7f8f52200640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8f38034000 tx=0x7f8f3802f000 comp rx=0 tx=0).stop 2024-03-06T07:27:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.913+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 msgr2=0x7f8f4c162a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.913+0000 7f8f52200640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 0x7f8f4c162a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000cc10 tx=0x7f8f4000a430 comp rx=0 tx=0).stop 2024-03-06T07:27:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 shutdown_connections 2024-03-06T07:27:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f4c1633f0 0x7f8f4c1fb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f4c1623d0 0x7f8f4c14aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 --2- 172.21.15.97:0/2631275730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f4c0b3a60 0x7f8f4c162a60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 >> 172.21.15.97:0/2631275730 conn(0x7f8f4c06afc0 msgr2=0x7f8f4c0708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 shutdown_connections 2024-03-06T07:27:49.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:49.914+0000 7f8f52200640 1 -- 172.21.15.97:0/2631275730 wait complete. 2024-03-06T07:27:49.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.0 2024-03-06T07:27:49.926 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:49.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.2 2024-03-06T07:27:49.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.3 2024-03-06T07:27:49.933 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:49.934 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2024-03-06T07:27:49.935 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:50.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.010+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9a34005580 con 0x7f9a5c161580 2024-03-06T07:27:50.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.011+0000 7f9a52000640 1 -- 172.21.15.97:0/3817612481 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a54019310 con 0x7f9a5c161580 2024-03-06T07:27:50.013 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2024-03-06T07:27:50.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a3c06b4d0 msgr2=0x7f9a3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a3c06b4d0 0x7f9a3c06d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009810 tx=0x7f9a4c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 msgr2=0x7f9a5c0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 0x7f9a5c0ccd10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c710 tx=0x7f9a5400cbd0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 shutdown_connections 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a3c06b4d0 0x7f9a3c06d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a5c161580 0x7f9a5c0ccd10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a5c0d28a0 0x7f9a5c0b9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 --2- 172.21.15.97:0/3817612481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a5c0d1e90 0x7f9a5c153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 >> 172.21.15.97:0/3817612481 conn(0x7f9a5c0d3d80 msgr2=0x7f9a5c15e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 shutdown_connections 2024-03-06T07:27:50.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.014+0000 7f9a5b200640 1 -- 172.21.15.97:0/3817612481 wait complete. 2024-03-06T07:27:50.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.023+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8d44005850 con 0x7f8d700cec10 2024-03-06T07:27:50.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.024+0000 7f8d67400640 1 -- 172.21.15.97:0/2870848102 <== mon.2 v2:172.21.15.135:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d6005c0d0 con 0x7f8d700cec10 2024-03-06T07:27:50.027 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d5006b5a0 msgr2=0x7f8d5006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d5006b5a0 0x7f8d5006da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d700726c0 tx=0x7f8d5c007e30 comp rx=0 tx=0).stop 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d700cec10 msgr2=0x7f8d70200320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d700cec10 0x7f8d70200320 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c550 tx=0x7f8d6000ca10 comp rx=0 tx=0).stop 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 shutdown_connections 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d5006b5a0 0x7f8d5006da60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d70162440 0x7f8d7015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d7015e360 0x7f8d7015cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 --2- 172.21.15.97:0/2870848102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d700cec10 0x7f8d70200320 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.027+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 >> 172.21.15.97:0/2870848102 conn(0x7f8d700d3f50 msgr2=0x7f8d7015bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.028+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 shutdown_connections 2024-03-06T07:27:50.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:50.028+0000 7f8d64c00640 1 -- 172.21.15.97:0/2870848102 wait complete. 2024-03-06T07:27:50.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.1 2024-03-06T07:27:50.037 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:50.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.5 2024-03-06T07:27:50.047 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:50.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:27:51.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.141+0000 7f8cab000640 1 Processor -- start 2024-03-06T07:27:51.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.141+0000 7f8cab000640 1 -- start start 2024-03-06T07:27:51.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.142+0000 7f8cab000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ca415e2c0 0x7f8ca40cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.142+0000 7f8cab000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40b4570 con 0x7f8ca406c0d0 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.142+0000 7f8cab000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40d0260 con 0x7f8ca40cc860 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.142+0000 7f8cab000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca40b4320 con 0x7f8ca415e2c0 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ca415e2c0 0x7f8ca40cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca3400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8ca406c0d0 0x7f8ca40d0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45180/0 (socket says 172.21.15.97:45180) 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca3400640 1 -- 172.21.15.97:0/953164348 learned_addr learned my addr 172.21.15.97:0/953164348 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:51.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca3e00640 1 -- 172.21.15.97:0/953164348 >> v1:172.21.15.135:6789/0 conn(0x7f8ca40cc860 legacy=0x7f8ca40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca3e00640 1 -- 172.21.15.97:0/953164348 >> v1:172.21.15.97:6789/0 conn(0x7f8ca406c0d0 legacy=0x7f8ca40d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca2a00640 1 -- 172.21.15.97:0/953164348 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268771278 0 0) 0x7f8ca40b4570 con 0x7f8ca406c0d0 2024-03-06T07:27:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.143+0000 7f8ca3e00640 1 -- 172.21.15.97:0/953164348 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40cff10 con 0x7f8ca415e2c0 2024-03-06T07:27:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.144+0000 7f8ca3e00640 1 --2- 172.21.15.97:0/953164348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca415e2c0 0x7f8ca40cc320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c940097a0 tx=0x7f8c9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596e918fee7425f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.144+0000 7f8ca2a00640 1 -- 172.21.15.97:0/953164348 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94035020 con 0x7f8ca415e2c0 2024-03-06T07:27:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.144+0000 7f8ca2a00640 1 -- 172.21.15.97:0/953164348 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c94031830 con 0x7f8ca415e2c0 2024-03-06T07:27:51.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.144+0000 7f8ca2a00640 1 -- 172.21.15.97:0/953164348 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94031c00 con 0x7f8ca415e2c0 2024-03-06T07:27:51.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 -- 172.21.15.97:0/953164348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca415e2c0 msgr2=0x7f8ca40cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 --2- 172.21.15.97:0/953164348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca415e2c0 0x7f8ca40cc320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c940097a0 tx=0x7f8c9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 -- 172.21.15.97:0/953164348 shutdown_connections 2024-03-06T07:27:51.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 --2- 172.21.15.97:0/953164348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca415e2c0 0x7f8ca40cc320 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 -- 172.21.15.97:0/953164348 >> 172.21.15.97:0/953164348 conn(0x7f8ca40b3800 msgr2=0x7f8ca40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 -- 172.21.15.97:0/953164348 shutdown_connections 2024-03-06T07:27:51.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.145+0000 7f8cab000640 1 -- 172.21.15.97:0/953164348 wait complete. 2024-03-06T07:27:51.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.147+0000 7f8cab000640 1 Processor -- start 2024-03-06T07:27:51.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.147+0000 7f8cab000640 1 -- start start 2024-03-06T07:27:51.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8cab000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca406c0d0 0x7f8ca4150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8cab000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 0x7f8ca40c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8cab000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ca415e2c0 0x7f8ca40b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8cab000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40d1170 con 0x7f8ca40cc860 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8cab000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40d05d0 con 0x7f8ca406c0d0 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8cab000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40d12f0 con 0x7f8ca415e2c0 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8ca3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 0x7f8ca40c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8ca3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 0x7f8ca40c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50666/0 (socket says 172.21.15.97:50666) 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.148+0000 7f8ca3400640 1 -- 172.21.15.97:0/2482618033 learned_addr learned my addr 172.21.15.97:0/2482618033 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3e00640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca406c0d0 0x7f8ca4150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3400640 1 -- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ca415e2c0 msgr2=0x7f8ca40b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3400640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ca415e2c0 0x7f8ca40b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3400640 1 -- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca406c0d0 msgr2=0x7f8ca4150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3400640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca406c0d0 0x7f8ca4150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3400640 1 -- 172.21.15.97:0/2482618033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40b6f30 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8ca3400640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 0x7f8ca40c8da0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000c430 tx=0x7f8c9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c900074e0 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c900047e0 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca40b7210 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90004bd0 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.149+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca40b11d0 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.150+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca406ea90 con 0x7f8ca40cc860 2024-03-06T07:27:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.151+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c90011070 con 0x7f8ca40cc860 2024-03-06T07:27:51.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.152+0000 7f8c9be00640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c8406b4d0 0x7f8c8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.153+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c90089e30 con 0x7f8ca40cc860 2024-03-06T07:27:51.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.153+0000 7f8ca3e00640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c8406b4d0 0x7f8c8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.153+0000 7f8ca3e00640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c8406b4d0 0x7f8c8406d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c94036000 tx=0x7f8c94008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.157+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ca406ea90 con 0x7f8ca40cc860 2024-03-06T07:27:51.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.339+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8ca40b13d0 con 0x7f8ca40cc860 2024-03-06T07:27:51.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.340+0000 7f8c9be00640 1 -- 172.21.15.97:0/2482618033 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ca40b13d0 con 0x7f8ca40cc860 2024-03-06T07:27:51.342 INFO:teuthology.orchestra.run.smithi097.stdout:60129542151 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c8406b4d0 msgr2=0x7f8c8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c8406b4d0 0x7f8c8406d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c94036000 tx=0x7f8c94008030 comp rx=0 tx=0).stop 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 msgr2=0x7f8ca40c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 0x7f8ca40c8da0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000c430 tx=0x7f8c9000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 shutdown_connections 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c8406b4d0 0x7f8c8406d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ca415e2c0 0x7f8ca40b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ca40cc860 0x7f8ca40c8da0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 --2- 172.21.15.97:0/2482618033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca406c0d0 0x7f8ca4150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.342+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 >> 172.21.15.97:0/2482618033 conn(0x7f8ca40b3800 msgr2=0x7f8ca40aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.343+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 shutdown_connections 2024-03-06T07:27:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.343+0000 7f8cab000640 1 -- 172.21.15.97:0/2482618033 wait complete. 2024-03-06T07:27:51.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.2 2024-03-06T07:27:51.363 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:51.363 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:27:51.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:51.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.564+0000 7f5f9d600640 1 Processor -- start 2024-03-06T07:27:51.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.564+0000 7f5f9d600640 1 -- start start 2024-03-06T07:27:51.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.565+0000 7f5f9d600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5f98161580 0x7f5f980d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.565+0000 7f5f9d600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5f980d2a70 0x7f5f980b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.565+0000 7f5f9d600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98161960 con 0x7f5f980d1e90 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9d600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f980cf6f0 con 0x7f5f980d2a70 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9d600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f980cf9d0 con 0x7f5f98161580 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5f980d2a70 0x7f5f980b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5f980d2a70 0x7f5f980b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50670/0 (socket says 172.21.15.97:50670) 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 -- 172.21.15.97:0/702555474 learned_addr learned my addr 172.21.15.97:0/702555474 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 -- 172.21.15.97:0/702555474 >> v2:172.21.15.135:3300/0 conn(0x7f5f98161580 msgr2=0x7f5f980d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f97e00640 1 --2- 172.21.15.97:0/702555474 >> v2:172.21.15.135:3300/0 conn(0x7f5f98161580 0x7f5f980d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 --2- 172.21.15.97:0/702555474 >> v2:172.21.15.135:3300/0 conn(0x7f5f98161580 0x7f5f980d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 -- 172.21.15.97:0/702555474 >> v1:172.21.15.135:6789/0 conn(0x7f5f980d1e90 legacy=0x7f5f980d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f9cc00640 1 -- 172.21.15.97:0/702555474 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980cfcb0 con 0x7f5f980d2a70 2024-03-06T07:27:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.566+0000 7f5f97e00640 1 --2- 172.21.15.97:0/702555474 >> v2:172.21.15.135:3300/0 conn(0x7f5f98161580 0x7f5f980d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:51.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.567+0000 7f5f9cc00640 1 --2- 172.21.15.97:0/702555474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f980b3ec0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5f98073740 tx=0x7f5f9002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c417db24e4222bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.568+0000 7f5f96a00640 1 -- 172.21.15.97:0/702555474 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9002ca80 con 0x7f5f980d2a70 2024-03-06T07:27:51.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.568+0000 7f5f96a00640 1 -- 172.21.15.97:0/702555474 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9002cc20 con 0x7f5f980d2a70 2024-03-06T07:27:51.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.568+0000 7f5f96a00640 1 -- 172.21.15.97:0/702555474 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f900343f0 con 0x7f5f980d2a70 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.568+0000 7f5f9d600640 1 -- 172.21.15.97:0/702555474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 msgr2=0x7f5f980b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.568+0000 7f5f9d600640 1 --2- 172.21.15.97:0/702555474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f980b3ec0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5f98073740 tx=0x7f5f9002bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.569+0000 7f5f9d600640 1 -- 172.21.15.97:0/702555474 shutdown_connections 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.569+0000 7f5f9d600640 1 --2- 172.21.15.97:0/702555474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f980b3ec0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.569+0000 7f5f9d600640 1 --2- 172.21.15.97:0/702555474 >> v2:172.21.15.135:3300/0 conn(0x7f5f98161580 0x7f5f980d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.569+0000 7f5f9d600640 1 -- 172.21.15.97:0/702555474 >> 172.21.15.97:0/702555474 conn(0x7f5f980d3d80 msgr2=0x7f5f980d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.569+0000 7f5f9d600640 1 -- 172.21.15.97:0/702555474 shutdown_connections 2024-03-06T07:27:51.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.569+0000 7f5f9d600640 1 -- 172.21.15.97:0/702555474 wait complete. 2024-03-06T07:27:51.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.570+0000 7f5f9d600640 1 Processor -- start 2024-03-06T07:27:51.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.571+0000 7f5f9d600640 1 -- start start 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.571+0000 7f5f9d600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f980d1e90 0x7f5f980cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.571+0000 7f5f9d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f9815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f9d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f98161580 0x7f5f980b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f9d600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980cf3a0 con 0x7f5f980d2a70 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f9d600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980b39a0 con 0x7f5f980d1e90 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f9d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980b4400 con 0x7f5f98161580 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f9815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f9815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50686/0 (socket says 172.21.15.97:50686) 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 -- 172.21.15.97:0/2104524851 learned_addr learned my addr 172.21.15.97:0/2104524851 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:51.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f9cc00640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f98161580 0x7f5f980b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 -- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f98161580 msgr2=0x7f5f980b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f98161580 0x7f5f980b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 -- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f980d1e90 msgr2=0x7f5f980cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f980d1e90 0x7f5f980cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 -- 172.21.15.97:0/2104524851 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9815c0c0 con 0x7f5f980d2a70 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.572+0000 7f5f97400640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f9815b5d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c430 tx=0x7f5f8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.573+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84011790 con 0x7f5f980d2a70 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.573+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84017070 con 0x7f5f980d2a70 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.573+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84011bb0 con 0x7f5f980d2a70 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.573+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f980b6bc0 con 0x7f5f980d2a70 2024-03-06T07:27:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.573+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f980b7180 con 0x7f5f980d2a70 2024-03-06T07:27:51.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.574+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9804f330 con 0x7f5f980d2a70 2024-03-06T07:27:51.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.575+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f8401c460 con 0x7f5f980d2a70 2024-03-06T07:27:51.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.576+0000 7f5f8fe00640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f7806b4d0 0x7f5f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.576+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f8408a150 con 0x7f5f980d2a70 2024-03-06T07:27:51.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.577+0000 7f5f97e00640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f7806b4d0 0x7f5f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.577+0000 7f5f97e00640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f7806b4d0 0x7f5f7806d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f88006fd0 tx=0x7f5f88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.581+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f84058f30 con 0x7f5f980d2a70 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.805+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f980d3b60 con 0x7f5f7806b4d0 2024-03-06T07:27:51.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.821+0000 7f5f8fe00640 1 -- 172.21.15.97:0/2104524851 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251838 (secure 0 0 0) 0x7f5f980d3b60 con 0x7f5f7806b4d0 2024-03-06T07:27:51.827 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:51.828 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f7806b4d0 msgr2=0x7f5f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f7806b4d0 0x7f5f7806d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f88006fd0 tx=0x7f5f88008040 comp rx=0 tx=0).stop 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 msgr2=0x7f5f9815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f9815b5d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c430 tx=0x7f5f8400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 shutdown_connections 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f7806b4d0 0x7f5f7806d990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f98161580 0x7f5f980b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f980d2a70 0x7f5f9815b5d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 --2- 172.21.15.97:0/2104524851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f980d1e90 0x7f5f980cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 >> 172.21.15.97:0/2104524851 conn(0x7f5f980d3d80 msgr2=0x7f5f98160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 shutdown_connections 2024-03-06T07:27:51.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:51.826+0000 7f5f9d600640 1 -- 172.21.15.97:0/2104524851 wait complete. 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:27:50.905442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219460,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755260,"statfs":{"total":773094113280,"available":772869386240,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.842820"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669901+0000","last_change":"2024-03-06T07:27:42.115947+0000","last_active":"2024-03-06T07:27:43.669901+0000","last_peered":"2024-03-06T07:27:43.669901+0000","last_clean":"2024-03-06T07:27:43.669901+0000","last_became_active":"2024-03-06T07:27:42.115323+0000","last_became_peered":"2024-03-06T07:27:42.115323+0000","last_unstale":"2024-03-06T07:27:43.669901+0000","last_undegraded":"2024-03-06T07:27:43.669901+0000","last_fullsized":"2024-03-06T07:27:43.669901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:23.978663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090443+0000","last_change":"2024-03-06T07:27:42.092048+0000","last_active":"2024-03-06T07:27:43.090443+0000","last_peered":"2024-03-06T07:27:43.090443+0000","last_clean":"2024-03-06T07:27:43.090443+0000","last_became_active":"2024-03-06T07:27:42.091783+0000","last_became_peered":"2024-03-06T07:27:42.091783+0000","last_unstale":"2024-03-06T07:27:43.090443+0000","last_undegraded":"2024-03-06T07:27:43.090443+0000","last_fullsized":"2024-03-06T07:27:43.090443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561478+0000","last_change":"2024-03-06T07:27:42.095834+0000","last_active":"2024-03-06T07:27:43.561478+0000","last_peered":"2024-03-06T07:27:43.561478+0000","last_clean":"2024-03-06T07:27:43.561478+0000","last_became_active":"2024-03-06T07:27:42.095026+0000","last_became_peered":"2024-03-06T07:27:42.095026+0000","last_unstale":"2024-03-06T07:27:43.561478+0000","last_undegraded":"2024-03-06T07:27:43.561478+0000","last_fullsized":"2024-03-06T07:27:43.561478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604760+0000","last_change":"2024-03-06T07:27:42.114409+0000","last_active":"2024-03-06T07:27:43.604760+0000","last_peered":"2024-03-06T07:27:43.604760+0000","last_clean":"2024-03-06T07:27:43.604760+0000","last_became_active":"2024-03-06T07:27:42.114122+0000","last_became_peered":"2024-03-06T07:27:42.114122+0000","last_unstale":"2024-03-06T07:27:43.604760+0000","last_undegraded":"2024-03-06T07:27:43.604760+0000","last_fullsized":"2024-03-06T07:27:43.604760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668591+0000","last_change":"2024-03-06T07:27:42.101691+0000","last_active":"2024-03-06T07:27:43.668591+0000","last_peered":"2024-03-06T07:27:43.668591+0000","last_clean":"2024-03-06T07:27:43.668591+0000","last_became_active":"2024-03-06T07:27:42.101372+0000","last_became_peered":"2024-03-06T07:27:42.101372+0000","last_unstale":"2024-03-06T07:27:43.668591+0000","last_undegraded":"2024-03-06T07:27:43.668591+0000","last_fullsized":"2024-03-06T07:27:43.668591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604137+0000","last_change":"2024-03-06T07:27:42.094768+0000","last_active":"2024-03-06T07:27:43.604137+0000","last_peered":"2024-03-06T07:27:43.604137+0000","last_clean":"2024-03-06T07:27:43.604137+0000","last_became_active":"2024-03-06T07:27:42.094416+0000","last_became_peered":"2024-03-06T07:27:42.094416+0000","last_unstale":"2024-03-06T07:27:43.604137+0000","last_undegraded":"2024-03-06T07:27:43.604137+0000","last_fullsized":"2024-03-06T07:27:43.604137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668352+0000","last_change":"2024-03-06T07:27:42.101468+0000","last_active":"2024-03-06T07:27:43.668352+0000","last_peered":"2024-03-06T07:27:43.668352+0000","last_clean":"2024-03-06T07:27:43.668352+0000","last_became_active":"2024-03-06T07:27:42.101168+0000","last_became_peered":"2024-03-06T07:27:42.101168+0000","last_unstale":"2024-03-06T07:27:43.668352+0000","last_undegraded":"2024-03-06T07:27:43.668352+0000","last_fullsized":"2024-03-06T07:27:43.668352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265352+0000","last_change":"2024-03-06T07:27:42.098517+0000","last_active":"2024-03-06T07:27:43.265352+0000","last_peered":"2024-03-06T07:27:43.265352+0000","last_clean":"2024-03-06T07:27:43.265352+0000","last_became_active":"2024-03-06T07:27:42.092643+0000","last_became_peered":"2024-03-06T07:27:42.092643+0000","last_unstale":"2024-03-06T07:27:43.265352+0000","last_undegraded":"2024-03-06T07:27:43.265352+0000","last_fullsized":"2024-03-06T07:27:43.265352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669864+0000","last_change":"2024-03-06T07:27:42.119208+0000","last_active":"2024-03-06T07:27:43.669864+0000","last_peered":"2024-03-06T07:27:43.669864+0000","last_clean":"2024-03-06T07:27:43.669864+0000","last_became_active":"2024-03-06T07:27:42.118050+0000","last_became_peered":"2024-03-06T07:27:42.118050+0000","last_unstale":"2024-03-06T07:27:43.669864+0000","last_undegraded":"2024-03-06T07:27:43.669864+0000","last_fullsized":"2024-03-06T07:27:43.669864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:43.473299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096548+0000","last_change":"2024-03-06T07:27:42.095950+0000","last_active":"2024-03-06T07:27:43.096548+0000","last_peered":"2024-03-06T07:27:43.096548+0000","last_clean":"2024-03-06T07:27:43.096548+0000","last_became_active":"2024-03-06T07:27:42.095462+0000","last_became_peered":"2024-03-06T07:27:42.095462+0000","last_unstale":"2024-03-06T07:27:43.096548+0000","last_undegraded":"2024-03-06T07:27:43.096548+0000","last_fullsized":"2024-03-06T07:27:43.096548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095953+0000","last_change":"2024-03-06T07:27:42.096143+0000","last_active":"2024-03-06T07:27:43.095953+0000","last_peered":"2024-03-06T07:27:43.095953+0000","last_clean":"2024-03-06T07:27:43.095953+0000","last_became_active":"2024-03-06T07:27:42.095793+0000","last_became_peered":"2024-03-06T07:27:42.095793+0000","last_unstale":"2024-03-06T07:27:43.095953+0000","last_undegraded":"2024-03-06T07:27:43.095953+0000","last_fullsized":"2024-03-06T07:27:43.095953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434202+0000","last_change":"2024-03-06T07:27:42.092238+0000","last_active":"2024-03-06T07:27:43.434202+0000","last_peered":"2024-03-06T07:27:43.434202+0000","last_clean":"2024-03-06T07:27:43.434202+0000","last_became_active":"2024-03-06T07:27:42.091930+0000","last_became_peered":"2024-03-06T07:27:42.091930+0000","last_unstale":"2024-03-06T07:27:43.434202+0000","last_undegraded":"2024-03-06T07:27:43.434202+0000","last_fullsized":"2024-03-06T07:27:43.434202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:38.321736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264768+0000","last_change":"2024-03-06T07:27:42.097468+0000","last_active":"2024-03-06T07:27:43.264768+0000","last_peered":"2024-03-06T07:27:43.264768+0000","last_clean":"2024-03-06T07:27:43.264768+0000","last_became_active":"2024-03-06T07:27:42.094036+0000","last_became_peered":"2024-03-06T07:27:42.094036+0000","last_unstale":"2024-03-06T07:27:43.264768+0000","last_undegraded":"2024-03-06T07:27:43.264768+0000","last_fullsized":"2024-03-06T07:27:43.264768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561085+0000","last_change":"2024-03-06T07:27:42.092704+0000","last_active":"2024-03-06T07:27:43.561085+0000","last_peered":"2024-03-06T07:27:43.561085+0000","last_clean":"2024-03-06T07:27:43.561085+0000","last_became_active":"2024-03-06T07:27:42.090459+0000","last_became_peered":"2024-03-06T07:27:42.090459+0000","last_unstale":"2024-03-06T07:27:43.561085+0000","last_undegraded":"2024-03-06T07:27:43.561085+0000","last_fullsized":"2024-03-06T07:27:43.561085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:26.470370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089922+0000","last_change":"2024-03-06T07:27:42.076294+0000","last_active":"2024-03-06T07:27:43.089922+0000","last_peered":"2024-03-06T07:27:43.089922+0000","last_clean":"2024-03-06T07:27:43.089922+0000","last_became_active":"2024-03-06T07:27:42.075529+0000","last_became_peered":"2024-03-06T07:27:42.075529+0000","last_unstale":"2024-03-06T07:27:43.089922+0000","last_undegraded":"2024-03-06T07:27:43.089922+0000","last_fullsized":"2024-03-06T07:27:43.089922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604410+0000","last_change":"2024-03-06T07:27:42.093846+0000","last_active":"2024-03-06T07:27:43.604410+0000","last_peered":"2024-03-06T07:27:43.604410+0000","last_clean":"2024-03-06T07:27:43.604410+0000","last_became_active":"2024-03-06T07:27:42.093544+0000","last_became_peered":"2024-03-06T07:27:42.093544+0000","last_unstale":"2024-03-06T07:27:43.604410+0000","last_undegraded":"2024-03-06T07:27:43.604410+0000","last_fullsized":"2024-03-06T07:27:43.604410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605458+0000","last_change":"2024-03-06T07:27:42.118690+0000","last_active":"2024-03-06T07:27:43.605458+0000","last_peered":"2024-03-06T07:27:43.605458+0000","last_clean":"2024-03-06T07:27:43.605458+0000","last_became_active":"2024-03-06T07:27:42.118405+0000","last_became_peered":"2024-03-06T07:27:42.118405+0000","last_unstale":"2024-03-06T07:27:43.605458+0000","last_undegraded":"2024-03-06T07:27:43.605458+0000","last_fullsized":"2024-03-06T07:27:43.605458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093672+0000","last_change":"2024-03-06T07:27:42.096245+0000","last_active":"2024-03-06T07:27:43.093672+0000","last_peered":"2024-03-06T07:27:43.093672+0000","last_clean":"2024-03-06T07:27:43.093672+0000","last_became_active":"2024-03-06T07:27:42.094600+0000","last_became_peered":"2024-03-06T07:27:42.094600+0000","last_unstale":"2024-03-06T07:27:43.093672+0000","last_undegraded":"2024-03-06T07:27:43.093672+0000","last_fullsized":"2024-03-06T07:27:43.093672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:05.681114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669317+0000","last_change":"2024-03-06T07:27:42.104820+0000","last_active":"2024-03-06T07:27:43.669317+0000","last_peered":"2024-03-06T07:27:43.669317+0000","last_clean":"2024-03-06T07:27:43.669317+0000","last_became_active":"2024-03-06T07:27:42.104489+0000","last_became_peered":"2024-03-06T07:27:42.104489+0000","last_unstale":"2024-03-06T07:27:43.669317+0000","last_undegraded":"2024-03-06T07:27:43.669317+0000","last_fullsized":"2024-03-06T07:27:43.669317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096181+0000","last_change":"2024-03-06T07:27:42.105851+0000","last_active":"2024-03-06T07:27:43.096181+0000","last_peered":"2024-03-06T07:27:43.096181+0000","last_clean":"2024-03-06T07:27:43.096181+0000","last_became_active":"2024-03-06T07:27:42.105276+0000","last_became_peered":"2024-03-06T07:27:42.105276+0000","last_unstale":"2024-03-06T07:27:43.096181+0000","last_undegraded":"2024-03-06T07:27:43.096181+0000","last_fullsized":"2024-03-06T07:27:43.096181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561313+0000","last_change":"2024-03-06T07:27:42.107395+0000","last_active":"2024-03-06T07:27:43.561313+0000","last_peered":"2024-03-06T07:27:43.561313+0000","last_clean":"2024-03-06T07:27:43.561313+0000","last_became_active":"2024-03-06T07:27:42.104429+0000","last_became_peered":"2024-03-06T07:27:42.104429+0000","last_unstale":"2024-03-06T07:27:43.561313+0000","last_undegraded":"2024-03-06T07:27:43.561313+0000","last_fullsized":"2024-03-06T07:27:43.561313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604391+0000","last_change":"2024-03-06T07:27:42.105460+0000","last_active":"2024-03-06T07:27:43.604391+0000","last_peered":"2024-03-06T07:27:43.604391+0000","last_clean":"2024-03-06T07:27:43.604391+0000","last_became_active":"2024-03-06T07:27:42.103643+0000","last_became_peered":"2024-03-06T07:27:42.103643+0000","last_unstale":"2024-03-06T07:27:43.604391+0000","last_undegraded":"2024-03-06T07:27:43.604391+0000","last_fullsized":"2024-03-06T07:27:43.604391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264480+0000","last_change":"2024-03-06T07:27:42.094539+0000","last_active":"2024-03-06T07:27:43.264480+0000","last_peered":"2024-03-06T07:27:43.264480+0000","last_clean":"2024-03-06T07:27:43.264480+0000","last_became_active":"2024-03-06T07:27:42.091758+0000","last_became_peered":"2024-03-06T07:27:42.091758+0000","last_unstale":"2024-03-06T07:27:43.264480+0000","last_undegraded":"2024-03-06T07:27:43.264480+0000","last_fullsized":"2024-03-06T07:27:43.264480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604546+0000","last_change":"2024-03-06T07:27:42.113737+0000","last_active":"2024-03-06T07:27:43.604546+0000","last_peered":"2024-03-06T07:27:43.604546+0000","last_clean":"2024-03-06T07:27:43.604546+0000","last_became_active":"2024-03-06T07:27:42.113444+0000","last_became_peered":"2024-03-06T07:27:42.113444+0000","last_unstale":"2024-03-06T07:27:43.604546+0000","last_undegraded":"2024-03-06T07:27:43.604546+0000","last_fullsized":"2024-03-06T07:27:43.604546+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265404+0000","last_change":"2024-03-06T07:27:42.104966+0000","last_active":"2024-03-06T07:27:43.265404+0000","last_peered":"2024-03-06T07:27:43.265404+0000","last_clean":"2024-03-06T07:27:43.265404+0000","last_became_active":"2024-03-06T07:27:42.104220+0000","last_became_peered":"2024-03-06T07:27:42.104220+0000","last_unstale":"2024-03-06T07:27:43.265404+0000","last_undegraded":"2024-03-06T07:27:43.265404+0000","last_fullsized":"2024-03-06T07:27:43.265404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096684+0000","last_change":"2024-03-06T07:27:42.103398+0000","last_active":"2024-03-06T07:27:43.096684+0000","last_peered":"2024-03-06T07:27:43.096684+0000","last_clean":"2024-03-06T07:27:43.096684+0000","last_became_active":"2024-03-06T07:27:42.102707+0000","last_became_peered":"2024-03-06T07:27:42.102707+0000","last_unstale":"2024-03-06T07:27:43.096684+0000","last_undegraded":"2024-03-06T07:27:43.096684+0000","last_fullsized":"2024-03-06T07:27:43.096684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090325+0000","last_change":"2024-03-06T07:27:42.092577+0000","last_active":"2024-03-06T07:27:43.090325+0000","last_peered":"2024-03-06T07:27:43.090325+0000","last_clean":"2024-03-06T07:27:43.090325+0000","last_became_active":"2024-03-06T07:27:42.092269+0000","last_became_peered":"2024-03-06T07:27:42.092269+0000","last_unstale":"2024-03-06T07:27:43.090325+0000","last_undegraded":"2024-03-06T07:27:43.090325+0000","last_fullsized":"2024-03-06T07:27:43.090325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265145+0000","last_change":"2024-03-06T07:27:42.105501+0000","last_active":"2024-03-06T07:27:43.265145+0000","last_peered":"2024-03-06T07:27:43.265145+0000","last_clean":"2024-03-06T07:27:43.265145+0000","last_became_active":"2024-03-06T07:27:42.100336+0000","last_became_peered":"2024-03-06T07:27:42.100336+0000","last_unstale":"2024-03-06T07:27:43.265145+0000","last_undegraded":"2024-03-06T07:27:43.265145+0000","last_fullsized":"2024-03-06T07:27:43.265145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561748+0000","last_change":"2024-03-06T07:27:42.090601+0000","last_active":"2024-03-06T07:27:43.561748+0000","last_peered":"2024-03-06T07:27:43.561748+0000","last_clean":"2024-03-06T07:27:43.561748+0000","last_became_active":"2024-03-06T07:27:42.090283+0000","last_became_peered":"2024-03-06T07:27:42.090283+0000","last_unstale":"2024-03-06T07:27:43.561748+0000","last_undegraded":"2024-03-06T07:27:43.561748+0000","last_fullsized":"2024-03-06T07:27:43.561748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604604+0000","last_change":"2024-03-06T07:27:42.113977+0000","last_active":"2024-03-06T07:27:43.604604+0000","last_peered":"2024-03-06T07:27:43.604604+0000","last_clean":"2024-03-06T07:27:43.604604+0000","last_became_active":"2024-03-06T07:27:42.111520+0000","last_became_peered":"2024-03-06T07:27:42.111520+0000","last_unstale":"2024-03-06T07:27:43.604604+0000","last_undegraded":"2024-03-06T07:27:43.604604+0000","last_fullsized":"2024-03-06T07:27:43.604604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093842+0000","last_change":"2024-03-06T07:27:42.096735+0000","last_active":"2024-03-06T07:27:43.093842+0000","last_peered":"2024-03-06T07:27:43.093842+0000","last_clean":"2024-03-06T07:27:43.093842+0000","last_became_active":"2024-03-06T07:27:42.096304+0000","last_became_peered":"2024-03-06T07:27:42.096304+0000","last_unstale":"2024-03-06T07:27:43.093842+0000","last_undegraded":"2024-03-06T07:27:43.093842+0000","last_fullsized":"2024-03-06T07:27:43.093842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:37.292940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095982+0000","last_change":"2024-03-06T07:27:42.096332+0000","last_active":"2024-03-06T07:27:43.095982+0000","last_peered":"2024-03-06T07:27:43.095982+0000","last_clean":"2024-03-06T07:27:43.095982+0000","last_became_active":"2024-03-06T07:27:42.095984+0000","last_became_peered":"2024-03-06T07:27:42.095984+0000","last_unstale":"2024-03-06T07:27:43.095982+0000","last_undegraded":"2024-03-06T07:27:43.095982+0000","last_fullsized":"2024-03-06T07:27:43.095982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.433842+0000","last_change":"2024-03-06T07:27:42.091042+0000","last_active":"2024-03-06T07:27:43.433842+0000","last_peered":"2024-03-06T07:27:43.433842+0000","last_clean":"2024-03-06T07:27:43.433842+0000","last_became_active":"2024-03-06T07:27:42.089637+0000","last_became_peered":"2024-03-06T07:27:42.089637+0000","last_unstale":"2024-03-06T07:27:43.433842+0000","last_undegraded":"2024-03-06T07:27:43.433842+0000","last_fullsized":"2024-03-06T07:27:43.433842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:00.218649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265208+0000","last_change":"2024-03-06T07:27:41.111941+0000","last_active":"2024-03-06T07:27:43.265208+0000","last_peered":"2024-03-06T07:27:43.265208+0000","last_clean":"2024-03-06T07:27:43.265208+0000","last_became_active":"2024-03-06T07:27:41.111244+0000","last_became_peered":"2024-03-06T07:27:41.111244+0000","last_unstale":"2024-03-06T07:27:43.265208+0000","last_undegraded":"2024-03-06T07:27:43.265208+0000","last_fullsized":"2024-03-06T07:27:43.265208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561659+0000","last_change":"2024-03-06T07:27:42.098732+0000","last_active":"2024-03-06T07:27:43.561659+0000","last_peered":"2024-03-06T07:27:43.561659+0000","last_clean":"2024-03-06T07:27:43.561659+0000","last_became_active":"2024-03-06T07:27:42.098451+0000","last_became_peered":"2024-03-06T07:27:42.098451+0000","last_unstale":"2024-03-06T07:27:43.561659+0000","last_undegraded":"2024-03-06T07:27:43.561659+0000","last_fullsized":"2024-03-06T07:27:43.561659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095423+0000","last_change":"2024-03-06T07:27:41.102703+0000","last_active":"2024-03-06T07:27:43.095423+0000","last_peered":"2024-03-06T07:27:43.095423+0000","last_clean":"2024-03-06T07:27:43.095423+0000","last_became_active":"2024-03-06T07:27:41.100059+0000","last_became_peered":"2024-03-06T07:27:41.100059+0000","last_unstale":"2024-03-06T07:27:43.095423+0000","last_undegraded":"2024-03-06T07:27:43.095423+0000","last_fullsized":"2024-03-06T07:27:43.095423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264571+0000","last_change":"2024-03-06T07:27:41.105849+0000","last_active":"2024-03-06T07:27:43.264571+0000","last_peered":"2024-03-06T07:27:43.264571+0000","last_clean":"2024-03-06T07:27:43.264571+0000","last_became_active":"2024-03-06T07:27:41.105558+0000","last_became_peered":"2024-03-06T07:27:41.105558+0000","last_unstale":"2024-03-06T07:27:43.264571+0000","last_undegraded":"2024-03-06T07:27:43.264571+0000","last_fullsized":"2024-03-06T07:27:43.264571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264534+0000","last_change":"2024-03-06T07:27:42.098554+0000","last_active":"2024-03-06T07:27:43.264534+0000","last_peered":"2024-03-06T07:27:43.264534+0000","last_clean":"2024-03-06T07:27:43.264534+0000","last_became_active":"2024-03-06T07:27:42.097364+0000","last_became_peered":"2024-03-06T07:27:42.097364+0000","last_unstale":"2024-03-06T07:27:43.264534+0000","last_undegraded":"2024-03-06T07:27:43.264534+0000","last_fullsized":"2024-03-06T07:27:43.264534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604701+0000","last_change":"2024-03-06T07:27:41.103209+0000","last_active":"2024-03-06T07:27:43.604701+0000","last_peered":"2024-03-06T07:27:43.604701+0000","last_clean":"2024-03-06T07:27:43.604701+0000","last_became_active":"2024-03-06T07:27:41.102994+0000","last_became_peered":"2024-03-06T07:27:41.102994+0000","last_unstale":"2024-03-06T07:27:43.604701+0000","last_undegraded":"2024-03-06T07:27:43.604701+0000","last_fullsized":"2024-03-06T07:27:43.604701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090178+0000","last_change":"2024-03-06T07:27:42.086153+0000","last_active":"2024-03-06T07:27:43.090178+0000","last_peered":"2024-03-06T07:27:43.090178+0000","last_clean":"2024-03-06T07:27:43.090178+0000","last_became_active":"2024-03-06T07:27:42.085779+0000","last_became_peered":"2024-03-06T07:27:42.085779+0000","last_unstale":"2024-03-06T07:27:43.090178+0000","last_undegraded":"2024-03-06T07:27:43.090178+0000","last_fullsized":"2024-03-06T07:27:43.090178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096443+0000","last_change":"2024-03-06T07:27:41.109144+0000","last_active":"2024-03-06T07:27:43.096443+0000","last_peered":"2024-03-06T07:27:43.096443+0000","last_clean":"2024-03-06T07:27:43.096443+0000","last_became_active":"2024-03-06T07:27:41.108399+0000","last_became_peered":"2024-03-06T07:27:41.108399+0000","last_unstale":"2024-03-06T07:27:43.096443+0000","last_undegraded":"2024-03-06T07:27:43.096443+0000","last_fullsized":"2024-03-06T07:27:43.096443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604733+0000","last_change":"2024-03-06T07:27:42.106823+0000","last_active":"2024-03-06T07:27:43.604733+0000","last_peered":"2024-03-06T07:27:43.604733+0000","last_clean":"2024-03-06T07:27:43.604733+0000","last_became_active":"2024-03-06T07:27:42.103949+0000","last_became_peered":"2024-03-06T07:27:42.103949+0000","last_unstale":"2024-03-06T07:27:43.604733+0000","last_undegraded":"2024-03-06T07:27:43.604733+0000","last_fullsized":"2024-03-06T07:27:43.604733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605002+0000","last_change":"2024-03-06T07:27:41.108455+0000","last_active":"2024-03-06T07:27:43.605002+0000","last_peered":"2024-03-06T07:27:43.605002+0000","last_clean":"2024-03-06T07:27:43.605002+0000","last_became_active":"2024-03-06T07:27:41.106095+0000","last_became_peered":"2024-03-06T07:27:41.106095+0000","last_unstale":"2024-03-06T07:27:43.605002+0000","last_undegraded":"2024-03-06T07:27:43.605002+0000","last_fullsized":"2024-03-06T07:27:43.605002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561945+0000","last_change":"2024-03-06T07:27:42.106635+0000","last_active":"2024-03-06T07:27:43.561945+0000","last_peered":"2024-03-06T07:27:43.561945+0000","last_clean":"2024-03-06T07:27:43.561945+0000","last_became_active":"2024-03-06T07:27:42.106373+0000","last_became_peered":"2024-03-06T07:27:42.106373+0000","last_unstale":"2024-03-06T07:27:43.561945+0000","last_undegraded":"2024-03-06T07:27:43.561945+0000","last_fullsized":"2024-03-06T07:27:43.561945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090296+0000","last_change":"2024-03-06T07:27:41.108679+0000","last_active":"2024-03-06T07:27:43.090296+0000","last_peered":"2024-03-06T07:27:43.090296+0000","last_clean":"2024-03-06T07:27:43.090296+0000","last_became_active":"2024-03-06T07:27:41.108429+0000","last_became_peered":"2024-03-06T07:27:41.108429+0000","last_unstale":"2024-03-06T07:27:43.090296+0000","last_undegraded":"2024-03-06T07:27:43.090296+0000","last_fullsized":"2024-03-06T07:27:43.090296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434363+0000","last_change":"2024-03-06T07:27:42.102682+0000","last_active":"2024-03-06T07:27:43.434363+0000","last_peered":"2024-03-06T07:27:43.434363+0000","last_clean":"2024-03-06T07:27:43.434363+0000","last_became_active":"2024-03-06T07:27:42.102232+0000","last_became_peered":"2024-03-06T07:27:42.102232+0000","last_unstale":"2024-03-06T07:27:43.434363+0000","last_undegraded":"2024-03-06T07:27:43.434363+0000","last_fullsized":"2024-03-06T07:27:43.434363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:34.589802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096130+0000","last_change":"2024-03-06T07:27:41.096688+0000","last_active":"2024-03-06T07:27:43.096130+0000","last_peered":"2024-03-06T07:27:43.096130+0000","last_clean":"2024-03-06T07:27:43.096130+0000","last_became_active":"2024-03-06T07:27:41.096391+0000","last_became_peered":"2024-03-06T07:27:41.096391+0000","last_unstale":"2024-03-06T07:27:43.096130+0000","last_undegraded":"2024-03-06T07:27:43.096130+0000","last_fullsized":"2024-03-06T07:27:43.096130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668491+0000","last_change":"2024-03-06T07:27:42.116169+0000","last_active":"2024-03-06T07:27:43.668491+0000","last_peered":"2024-03-06T07:27:43.668491+0000","last_clean":"2024-03-06T07:27:43.668491+0000","last_became_active":"2024-03-06T07:27:42.115868+0000","last_became_peered":"2024-03-06T07:27:42.115868+0000","last_unstale":"2024-03-06T07:27:43.668491+0000","last_undegraded":"2024-03-06T07:27:43.668491+0000","last_fullsized":"2024-03-06T07:27:43.668491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.091480+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.090511+0000","last_became_peered":"2024-03-06T07:27:41.090511+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561169+0000","last_change":"2024-03-06T07:27:42.104948+0000","last_active":"2024-03-06T07:27:43.561169+0000","last_peered":"2024-03-06T07:27:43.561169+0000","last_clean":"2024-03-06T07:27:43.561169+0000","last_became_active":"2024-03-06T07:27:42.104640+0000","last_became_peered":"2024-03-06T07:27:42.104640+0000","last_unstale":"2024-03-06T07:27:43.561169+0000","last_undegraded":"2024-03-06T07:27:43.561169+0000","last_fullsized":"2024-03-06T07:27:43.561169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604876+0000","last_change":"2024-03-06T07:27:41.103424+0000","last_active":"2024-03-06T07:27:43.604876+0000","last_peered":"2024-03-06T07:27:43.604876+0000","last_clean":"2024-03-06T07:27:43.604876+0000","last_became_active":"2024-03-06T07:27:41.102084+0000","last_became_peered":"2024-03-06T07:27:41.102084+0000","last_unstale":"2024-03-06T07:27:43.604876+0000","last_undegraded":"2024-03-06T07:27:43.604876+0000","last_fullsized":"2024-03-06T07:27:43.604876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668345+0000","last_change":"2024-03-06T07:27:42.107947+0000","last_active":"2024-03-06T07:27:43.668345+0000","last_peered":"2024-03-06T07:27:43.668345+0000","last_clean":"2024-03-06T07:27:43.668345+0000","last_became_active":"2024-03-06T07:27:42.107570+0000","last_became_peered":"2024-03-06T07:27:42.107570+0000","last_unstale":"2024-03-06T07:27:43.668345+0000","last_undegraded":"2024-03-06T07:27:43.668345+0000","last_fullsized":"2024-03-06T07:27:43.668345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095592+0000","last_change":"2024-03-06T07:27:41.105127+0000","last_active":"2024-03-06T07:27:43.095592+0000","last_peered":"2024-03-06T07:27:43.095592+0000","last_clean":"2024-03-06T07:27:43.095592+0000","last_became_active":"2024-03-06T07:27:41.104134+0000","last_became_peered":"2024-03-06T07:27:41.104134+0000","last_unstale":"2024-03-06T07:27:43.095592+0000","last_undegraded":"2024-03-06T07:27:43.095592+0000","last_fullsized":"2024-03-06T07:27:43.095592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096137+0000","last_change":"2024-03-06T07:27:42.104623+0000","last_active":"2024-03-06T07:27:43.096137+0000","last_peered":"2024-03-06T07:27:43.096137+0000","last_clean":"2024-03-06T07:27:43.096137+0000","last_became_active":"2024-03-06T07:27:42.104185+0000","last_became_peered":"2024-03-06T07:27:42.104185+0000","last_unstale":"2024-03-06T07:27:43.096137+0000","last_undegraded":"2024-03-06T07:27:43.096137+0000","last_fullsized":"2024-03-06T07:27:43.096137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093473+0000","last_change":"2024-03-06T07:27:41.095963+0000","last_active":"2024-03-06T07:27:43.093473+0000","last_peered":"2024-03-06T07:27:43.093473+0000","last_clean":"2024-03-06T07:27:43.093473+0000","last_became_active":"2024-03-06T07:27:41.095610+0000","last_became_peered":"2024-03-06T07:27:41.095610+0000","last_unstale":"2024-03-06T07:27:43.093473+0000","last_undegraded":"2024-03-06T07:27:43.093473+0000","last_fullsized":"2024-03-06T07:27:43.093473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668062+0000","last_change":"2024-03-06T07:27:42.107796+0000","last_active":"2024-03-06T07:27:43.668062+0000","last_peered":"2024-03-06T07:27:43.668062+0000","last_clean":"2024-03-06T07:27:43.668062+0000","last_became_active":"2024-03-06T07:27:42.107514+0000","last_became_peered":"2024-03-06T07:27:42.107514+0000","last_unstale":"2024-03-06T07:27:43.668062+0000","last_undegraded":"2024-03-06T07:27:43.668062+0000","last_fullsized":"2024-03-06T07:27:43.668062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434682+0000","last_change":"2024-03-06T07:27:41.094350+0000","last_active":"2024-03-06T07:27:43.434682+0000","last_peered":"2024-03-06T07:27:43.434682+0000","last_clean":"2024-03-06T07:27:43.434682+0000","last_became_active":"2024-03-06T07:27:41.089935+0000","last_became_peered":"2024-03-06T07:27:41.089935+0000","last_unstale":"2024-03-06T07:27:43.434682+0000","last_undegraded":"2024-03-06T07:27:43.434682+0000","last_fullsized":"2024-03-06T07:27:43.434682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:49.952499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669175+0000","last_change":"2024-03-06T07:27:42.109835+0000","last_active":"2024-03-06T07:27:43.669175+0000","last_peered":"2024-03-06T07:27:43.669175+0000","last_clean":"2024-03-06T07:27:43.669175+0000","last_became_active":"2024-03-06T07:27:42.109535+0000","last_became_peered":"2024-03-06T07:27:42.109535+0000","last_unstale":"2024-03-06T07:27:43.669175+0000","last_undegraded":"2024-03-06T07:27:43.669175+0000","last_fullsized":"2024-03-06T07:27:43.669175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093784+0000","last_change":"2024-03-06T07:27:41.092525+0000","last_active":"2024-03-06T07:27:43.093784+0000","last_peered":"2024-03-06T07:27:43.093784+0000","last_clean":"2024-03-06T07:27:43.093784+0000","last_became_active":"2024-03-06T07:27:41.091644+0000","last_became_peered":"2024-03-06T07:27:41.091644+0000","last_unstale":"2024-03-06T07:27:43.093784+0000","last_undegraded":"2024-03-06T07:27:43.093784+0000","last_fullsized":"2024-03-06T07:27:43.093784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.670064+0000","last_change":"2024-03-06T07:27:42.117934+0000","last_active":"2024-03-06T07:27:43.670064+0000","last_peered":"2024-03-06T07:27:43.670064+0000","last_clean":"2024-03-06T07:27:43.670064+0000","last_became_active":"2024-03-06T07:27:42.115924+0000","last_became_peered":"2024-03-06T07:27:42.115924+0000","last_unstale":"2024-03-06T07:27:43.670064+0000","last_undegraded":"2024-03-06T07:27:43.670064+0000","last_fullsized":"2024-03-06T07:27:43.670064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:39.844169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090372+0000","last_change":"2024-03-06T07:27:41.097118+0000","last_active":"2024-03-06T07:27:43.090372+0000","last_peered":"2024-03-06T07:27:43.090372+0000","last_clean":"2024-03-06T07:27:43.090372+0000","last_became_active":"2024-03-06T07:27:41.096188+0000","last_became_peered":"2024-03-06T07:27:41.096188+0000","last_unstale":"2024-03-06T07:27:43.090372+0000","last_undegraded":"2024-03-06T07:27:43.090372+0000","last_fullsized":"2024-03-06T07:27:43.090372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605208+0000","last_change":"2024-03-06T07:27:42.114130+0000","last_active":"2024-03-06T07:27:43.605208+0000","last_peered":"2024-03-06T07:27:43.605208+0000","last_clean":"2024-03-06T07:27:43.605208+0000","last_became_active":"2024-03-06T07:27:42.113811+0000","last_became_peered":"2024-03-06T07:27:42.113811+0000","last_unstale":"2024-03-06T07:27:43.605208+0000","last_undegraded":"2024-03-06T07:27:43.605208+0000","last_fullsized":"2024-03-06T07:27:43.605208+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093668+0000","last_change":"2024-03-06T07:27:41.102837+0000","last_active":"2024-03-06T07:27:43.093668+0000","last_peered":"2024-03-06T07:27:43.093668+0000","last_clean":"2024-03-06T07:27:43.093668+0000","last_became_active":"2024-03-06T07:27:41.102526+0000","last_became_peered":"2024-03-06T07:27:41.102526+0000","last_unstale":"2024-03-06T07:27:43.093668+0000","last_undegraded":"2024-03-06T07:27:43.093668+0000","last_fullsized":"2024-03-06T07:27:43.093668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668936+0000","last_change":"2024-03-06T07:27:42.110688+0000","last_active":"2024-03-06T07:27:43.668936+0000","last_peered":"2024-03-06T07:27:43.668936+0000","last_clean":"2024-03-06T07:27:43.668936+0000","last_became_active":"2024-03-06T07:27:42.110394+0000","last_became_peered":"2024-03-06T07:27:42.110394+0000","last_unstale":"2024-03-06T07:27:43.668936+0000","last_undegraded":"2024-03-06T07:27:43.668936+0000","last_fullsized":"2024-03-06T07:27:43.668936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667970+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:27:43.667970+0000","last_peered":"2024-03-06T07:27:43.667970+0000","last_clean":"2024-03-06T07:27:43.667970+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:27:43.667970+0000","last_undegraded":"2024-03-06T07:27:43.667970+0000","last_fullsized":"2024-03-06T07:27:43.667970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604170+0000","last_change":"2024-03-06T07:27:41.111128+0000","last_active":"2024-03-06T07:27:43.604170+0000","last_peered":"2024-03-06T07:27:43.604170+0000","last_clean":"2024-03-06T07:27:43.604170+0000","last_became_active":"2024-03-06T07:27:41.109382+0000","last_became_peered":"2024-03-06T07:27:41.109382+0000","last_unstale":"2024-03-06T07:27:43.604170+0000","last_undegraded":"2024-03-06T07:27:43.604170+0000","last_fullsized":"2024-03-06T07:27:43.604170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264748+0000","last_change":"2024-03-06T07:27:42.100303+0000","last_active":"2024-03-06T07:27:43.264748+0000","last_peered":"2024-03-06T07:27:43.264748+0000","last_clean":"2024-03-06T07:27:43.264748+0000","last_became_active":"2024-03-06T07:27:42.099977+0000","last_became_peered":"2024-03-06T07:27:42.099977+0000","last_unstale":"2024-03-06T07:27:43.264748+0000","last_undegraded":"2024-03-06T07:27:43.264748+0000","last_fullsized":"2024-03-06T07:27:43.264748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667817+0000","last_change":"2024-03-06T07:27:41.113323+0000","last_active":"2024-03-06T07:27:43.667817+0000","last_peered":"2024-03-06T07:27:43.667817+0000","last_clean":"2024-03-06T07:27:43.667817+0000","last_became_active":"2024-03-06T07:27:41.112921+0000","last_became_peered":"2024-03-06T07:27:41.112921+0000","last_unstale":"2024-03-06T07:27:43.667817+0000","last_undegraded":"2024-03-06T07:27:43.667817+0000","last_fullsized":"2024-03-06T07:27:43.667817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:12.710184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561173+0000","last_change":"2024-03-06T07:27:42.098275+0000","last_active":"2024-03-06T07:27:43.561173+0000","last_peered":"2024-03-06T07:27:43.561173+0000","last_clean":"2024-03-06T07:27:43.561173+0000","last_became_active":"2024-03-06T07:27:42.097430+0000","last_became_peered":"2024-03-06T07:27:42.097430+0000","last_unstale":"2024-03-06T07:27:43.561173+0000","last_undegraded":"2024-03-06T07:27:43.561173+0000","last_fullsized":"2024-03-06T07:27:43.561173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:43.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093787+0000","last_change":"2024-03-06T07:27:41.092149+0000","last_active":"2024-03-06T07:27:43.093787+0000","last_peered":"2024-03-06T07:27:43.093787+0000","last_clean":"2024-03-06T07:27:43.093787+0000","last_became_active":"2024-03-06T07:27:41.090942+0000","last_became_peered":"2024-03-06T07:27:41.090942+0000","last_unstale":"2024-03-06T07:27:43.093787+0000","last_undegraded":"2024-03-06T07:27:43.093787+0000","last_fullsized":"2024-03-06T07:27:43.093787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089953+0000","last_change":"2024-03-06T07:27:42.103118+0000","last_active":"2024-03-06T07:27:43.089953+0000","last_peered":"2024-03-06T07:27:43.089953+0000","last_clean":"2024-03-06T07:27:43.089953+0000","last_became_active":"2024-03-06T07:27:42.102850+0000","last_became_peered":"2024-03-06T07:27:42.102850+0000","last_unstale":"2024-03-06T07:27:43.089953+0000","last_undegraded":"2024-03-06T07:27:43.089953+0000","last_fullsized":"2024-03-06T07:27:43.089953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605179+0000","last_change":"2024-03-06T07:27:41.111452+0000","last_active":"2024-03-06T07:27:43.605179+0000","last_peered":"2024-03-06T07:27:43.605179+0000","last_clean":"2024-03-06T07:27:43.605179+0000","last_became_active":"2024-03-06T07:27:41.110359+0000","last_became_peered":"2024-03-06T07:27:41.110359+0000","last_unstale":"2024-03-06T07:27:43.605179+0000","last_undegraded":"2024-03-06T07:27:43.605179+0000","last_fullsized":"2024-03-06T07:27:43.605179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434187+0000","last_change":"2024-03-06T07:27:42.093244+0000","last_active":"2024-03-06T07:27:43.434187+0000","last_peered":"2024-03-06T07:27:43.434187+0000","last_clean":"2024-03-06T07:27:43.434187+0000","last_became_active":"2024-03-06T07:27:42.092228+0000","last_became_peered":"2024-03-06T07:27:42.092228+0000","last_unstale":"2024-03-06T07:27:43.434187+0000","last_undegraded":"2024-03-06T07:27:43.434187+0000","last_fullsized":"2024-03-06T07:27:43.434187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:07.427898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090250+0000","last_change":"2024-03-06T07:27:41.093905+0000","last_active":"2024-03-06T07:27:43.090250+0000","last_peered":"2024-03-06T07:27:43.090250+0000","last_clean":"2024-03-06T07:27:43.090250+0000","last_became_active":"2024-03-06T07:27:41.093456+0000","last_became_peered":"2024-03-06T07:27:41.093456+0000","last_unstale":"2024-03-06T07:27:43.090250+0000","last_undegraded":"2024-03-06T07:27:43.090250+0000","last_fullsized":"2024-03-06T07:27:43.090250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605312+0000","last_change":"2024-03-06T07:27:42.112370+0000","last_active":"2024-03-06T07:27:43.605312+0000","last_peered":"2024-03-06T07:27:43.605312+0000","last_clean":"2024-03-06T07:27:43.605312+0000","last_became_active":"2024-03-06T07:27:42.112087+0000","last_became_peered":"2024-03-06T07:27:42.112087+0000","last_unstale":"2024-03-06T07:27:43.605312+0000","last_undegraded":"2024-03-06T07:27:43.605312+0000","last_fullsized":"2024-03-06T07:27:43.605312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669326+0000","last_change":"2024-03-06T07:27:41.116636+0000","last_active":"2024-03-06T07:27:43.669326+0000","last_peered":"2024-03-06T07:27:43.669326+0000","last_clean":"2024-03-06T07:27:43.669326+0000","last_became_active":"2024-03-06T07:27:41.116254+0000","last_became_peered":"2024-03-06T07:27:41.116254+0000","last_unstale":"2024-03-06T07:27:43.669326+0000","last_undegraded":"2024-03-06T07:27:43.669326+0000","last_fullsized":"2024-03-06T07:27:43.669326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:59.546367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090070+0000","last_change":"2024-03-06T07:27:42.094877+0000","last_active":"2024-03-06T07:27:43.090070+0000","last_peered":"2024-03-06T07:27:43.090070+0000","last_clean":"2024-03-06T07:27:43.090070+0000","last_became_active":"2024-03-06T07:27:42.094586+0000","last_became_peered":"2024-03-06T07:27:42.094586+0000","last_unstale":"2024-03-06T07:27:43.090070+0000","last_undegraded":"2024-03-06T07:27:43.090070+0000","last_fullsized":"2024-03-06T07:27:43.090070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.102122+0000","last_change":"2024-03-06T07:27:41.098449+0000","last_active":"2024-03-06T07:27:43.102122+0000","last_peered":"2024-03-06T07:27:43.102122+0000","last_clean":"2024-03-06T07:27:43.102122+0000","last_became_active":"2024-03-06T07:27:41.098137+0000","last_became_peered":"2024-03-06T07:27:41.098137+0000","last_unstale":"2024-03-06T07:27:43.102122+0000","last_undegraded":"2024-03-06T07:27:43.102122+0000","last_fullsized":"2024-03-06T07:27:43.102122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265545+0000","last_change":"2024-03-06T07:27:42.118543+0000","last_active":"2024-03-06T07:27:43.265545+0000","last_peered":"2024-03-06T07:27:43.265545+0000","last_clean":"2024-03-06T07:27:43.265545+0000","last_became_active":"2024-03-06T07:27:42.118157+0000","last_became_peered":"2024-03-06T07:27:42.118157+0000","last_unstale":"2024-03-06T07:27:43.265545+0000","last_undegraded":"2024-03-06T07:27:43.265545+0000","last_fullsized":"2024-03-06T07:27:43.265545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096632+0000","last_change":"2024-03-06T07:27:41.112840+0000","last_active":"2024-03-06T07:27:43.096632+0000","last_peered":"2024-03-06T07:27:43.096632+0000","last_clean":"2024-03-06T07:27:43.096632+0000","last_became_active":"2024-03-06T07:27:41.112440+0000","last_became_peered":"2024-03-06T07:27:41.112440+0000","last_unstale":"2024-03-06T07:27:43.096632+0000","last_undegraded":"2024-03-06T07:27:43.096632+0000","last_fullsized":"2024-03-06T07:27:43.096632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561550+0000","last_change":"2024-03-06T07:27:42.090517+0000","last_active":"2024-03-06T07:27:43.561550+0000","last_peered":"2024-03-06T07:27:43.561550+0000","last_clean":"2024-03-06T07:27:43.561550+0000","last_became_active":"2024-03-06T07:27:42.090202+0000","last_became_peered":"2024-03-06T07:27:42.090202+0000","last_unstale":"2024-03-06T07:27:43.561550+0000","last_undegraded":"2024-03-06T07:27:43.561550+0000","last_fullsized":"2024-03-06T07:27:43.561550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667842+0000","last_change":"2024-03-06T07:27:41.114447+0000","last_active":"2024-03-06T07:27:43.667842+0000","last_peered":"2024-03-06T07:27:43.667842+0000","last_clean":"2024-03-06T07:27:43.667842+0000","last_became_active":"2024-03-06T07:27:41.113802+0000","last_became_peered":"2024-03-06T07:27:41.113802+0000","last_unstale":"2024-03-06T07:27:43.667842+0000","last_undegraded":"2024-03-06T07:27:43.667842+0000","last_fullsized":"2024-03-06T07:27:43.667842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:43.723690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434174+0000","last_change":"2024-03-06T07:27:42.110210+0000","last_active":"2024-03-06T07:27:43.434174+0000","last_peered":"2024-03-06T07:27:43.434174+0000","last_clean":"2024-03-06T07:27:43.434174+0000","last_became_active":"2024-03-06T07:27:42.109736+0000","last_became_peered":"2024-03-06T07:27:42.109736+0000","last_unstale":"2024-03-06T07:27:43.434174+0000","last_undegraded":"2024-03-06T07:27:43.434174+0000","last_fullsized":"2024-03-06T07:27:43.434174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:25.704825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090026+0000","last_change":"2024-03-06T07:27:41.101111+0000","last_active":"2024-03-06T07:27:43.090026+0000","last_peered":"2024-03-06T07:27:43.090026+0000","last_clean":"2024-03-06T07:27:43.090026+0000","last_became_active":"2024-03-06T07:27:41.100857+0000","last_became_peered":"2024-03-06T07:27:41.100857+0000","last_unstale":"2024-03-06T07:27:43.090026+0000","last_undegraded":"2024-03-06T07:27:43.090026+0000","last_fullsized":"2024-03-06T07:27:43.090026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264697+0000","last_change":"2024-03-06T07:27:42.086202+0000","last_active":"2024-03-06T07:27:43.264697+0000","last_peered":"2024-03-06T07:27:43.264697+0000","last_clean":"2024-03-06T07:27:43.264697+0000","last_became_active":"2024-03-06T07:27:42.085754+0000","last_became_peered":"2024-03-06T07:27:42.085754+0000","last_unstale":"2024-03-06T07:27:43.264697+0000","last_undegraded":"2024-03-06T07:27:43.264697+0000","last_fullsized":"2024-03-06T07:27:43.264697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090044+0000","last_change":"2024-03-06T07:27:41.096584+0000","last_active":"2024-03-06T07:27:43.090044+0000","last_peered":"2024-03-06T07:27:43.090044+0000","last_clean":"2024-03-06T07:27:43.090044+0000","last_became_active":"2024-03-06T07:27:41.096318+0000","last_became_peered":"2024-03-06T07:27:41.096318+0000","last_unstale":"2024-03-06T07:27:43.090044+0000","last_undegraded":"2024-03-06T07:27:43.090044+0000","last_fullsized":"2024-03-06T07:27:43.090044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265099+0000","last_change":"2024-03-06T07:27:42.102269+0000","last_active":"2024-03-06T07:27:43.265099+0000","last_peered":"2024-03-06T07:27:43.265099+0000","last_clean":"2024-03-06T07:27:43.265099+0000","last_became_active":"2024-03-06T07:27:42.092816+0000","last_became_peered":"2024-03-06T07:27:42.092816+0000","last_unstale":"2024-03-06T07:27:43.265099+0000","last_undegraded":"2024-03-06T07:27:43.265099+0000","last_fullsized":"2024-03-06T07:27:43.265099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668665+0000","last_change":"2024-03-06T07:27:41.116372+0000","last_active":"2024-03-06T07:27:43.668665+0000","last_peered":"2024-03-06T07:27:43.668665+0000","last_clean":"2024-03-06T07:27:43.668665+0000","last_became_active":"2024-03-06T07:27:41.116072+0000","last_became_peered":"2024-03-06T07:27:41.116072+0000","last_unstale":"2024-03-06T07:27:43.668665+0000","last_undegraded":"2024-03-06T07:27:43.668665+0000","last_fullsized":"2024-03-06T07:27:43.668665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095884+0000","last_change":"2024-03-06T07:27:42.103012+0000","last_active":"2024-03-06T07:27:43.095884+0000","last_peered":"2024-03-06T07:27:43.095884+0000","last_clean":"2024-03-06T07:27:43.095884+0000","last_became_active":"2024-03-06T07:27:42.102731+0000","last_became_peered":"2024-03-06T07:27:42.102731+0000","last_unstale":"2024-03-06T07:27:43.095884+0000","last_undegraded":"2024-03-06T07:27:43.095884+0000","last_fullsized":"2024-03-06T07:27:43.095884+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604917+0000","last_change":"2024-03-06T07:27:42.097462+0000","last_active":"2024-03-06T07:27:43.604917+0000","last_peered":"2024-03-06T07:27:43.604917+0000","last_clean":"2024-03-06T07:27:43.604917+0000","last_became_active":"2024-03-06T07:27:42.097142+0000","last_became_peered":"2024-03-06T07:27:42.097142+0000","last_unstale":"2024-03-06T07:27:43.604917+0000","last_undegraded":"2024-03-06T07:27:43.604917+0000","last_fullsized":"2024-03-06T07:27:43.604917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668754+0000","last_change":"2024-03-06T07:27:41.115663+0000","last_active":"2024-03-06T07:27:43.668754+0000","last_peered":"2024-03-06T07:27:43.668754+0000","last_clean":"2024-03-06T07:27:43.668754+0000","last_became_active":"2024-03-06T07:27:41.114756+0000","last_became_peered":"2024-03-06T07:27:41.114756+0000","last_unstale":"2024-03-06T07:27:43.668754+0000","last_undegraded":"2024-03-06T07:27:43.668754+0000","last_fullsized":"2024-03-06T07:27:43.668754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604966+0000","last_change":"2024-03-06T07:27:42.098684+0000","last_active":"2024-03-06T07:27:43.604966+0000","last_peered":"2024-03-06T07:27:43.604966+0000","last_clean":"2024-03-06T07:27:43.604966+0000","last_became_active":"2024-03-06T07:27:42.098307+0000","last_became_peered":"2024-03-06T07:27:42.098307+0000","last_unstale":"2024-03-06T07:27:43.604966+0000","last_undegraded":"2024-03-06T07:27:43.604966+0000","last_fullsized":"2024-03-06T07:27:43.604966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669076+0000","last_change":"2024-03-06T07:27:41.117369+0000","last_active":"2024-03-06T07:27:43.669076+0000","last_peered":"2024-03-06T07:27:43.669076+0000","last_clean":"2024-03-06T07:27:43.669076+0000","last_became_active":"2024-03-06T07:27:41.117042+0000","last_became_peered":"2024-03-06T07:27:41.117042+0000","last_unstale":"2024-03-06T07:27:43.669076+0000","last_undegraded":"2024-03-06T07:27:43.669076+0000","last_fullsized":"2024-03-06T07:27:43.669076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561654+0000","last_change":"2024-03-06T07:27:42.106505+0000","last_active":"2024-03-06T07:27:43.561654+0000","last_peered":"2024-03-06T07:27:43.561654+0000","last_clean":"2024-03-06T07:27:43.561654+0000","last_became_active":"2024-03-06T07:27:42.105993+0000","last_became_peered":"2024-03-06T07:27:42.105993+0000","last_unstale":"2024-03-06T07:27:43.561654+0000","last_undegraded":"2024-03-06T07:27:43.561654+0000","last_fullsized":"2024-03-06T07:27:43.561654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265054+0000","last_change":"2024-03-06T07:27:41.097028+0000","last_active":"2024-03-06T07:27:43.265054+0000","last_peered":"2024-03-06T07:27:43.265054+0000","last_clean":"2024-03-06T07:27:43.265054+0000","last_became_active":"2024-03-06T07:27:41.096532+0000","last_became_peered":"2024-03-06T07:27:41.096532+0000","last_unstale":"2024-03-06T07:27:43.265054+0000","last_undegraded":"2024-03-06T07:27:43.265054+0000","last_fullsized":"2024-03-06T07:27:43.265054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264945+0000","last_change":"2024-03-06T07:27:42.094785+0000","last_active":"2024-03-06T07:27:43.264945+0000","last_peered":"2024-03-06T07:27:43.264945+0000","last_clean":"2024-03-06T07:27:43.264945+0000","last_became_active":"2024-03-06T07:27:42.091901+0000","last_became_peered":"2024-03-06T07:27:42.091901+0000","last_unstale":"2024-03-06T07:27:43.264945+0000","last_undegraded":"2024-03-06T07:27:43.264945+0000","last_fullsized":"2024-03-06T07:27:43.264945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.102186+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.101868+0000","last_became_peered":"2024-03-06T07:27:41.101868+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:20.791281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:51.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:52.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.089+0000 7f46bbe00640 1 Processor -- start 2024-03-06T07:27:52.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.090+0000 7f46bbe00640 1 -- start start 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.091+0000 7f46bbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f46bc161580 0x7f46bc0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.091+0000 7f46bbe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc161960 con 0x7f46bc0d1e90 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.091+0000 7f46bbe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc0cf6f0 con 0x7f46bc0d2a70 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.091+0000 7f46bbe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc0cf9d0 con 0x7f46bc161580 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.091+0000 7f46b3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f46bc0d1e90 0x7f46bc0d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45196/0 (socket says 172.21.15.97:45196) 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.091+0000 7f46b3e00640 1 -- 172.21.15.97:0/624050440 learned_addr learned my addr 172.21.15.97:0/624050440 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882068194 0 0) 0x7f46bc161960 con 0x7f46bc0d1e90 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0003520 con 0x7f46bc0d1e90 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46baa00640 1 --2- 172.21.15.97:0/624050440 >> v2:172.21.15.135:3300/0 conn(0x7f46bc161580 0x7f46bc0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554238940 0 0) 0x7f46a0003520 con 0x7f46bc0d1e90 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 >> v2:172.21.15.135:3300/0 conn(0x7f46bc161580 msgr2=0x7f46bc0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 --2- 172.21.15.97:0/624050440 >> v2:172.21.15.135:3300/0 conn(0x7f46bc161580 0x7f46bc0d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 >> v1:172.21.15.135:6789/0 conn(0x7f46bc0d2a70 legacy=0x7f46bc0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0cfcb0 con 0x7f46bc0d1e90 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f46a4001670 con 0x7f46bc0d1e90 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 >> v1:172.21.15.97:6789/0 conn(0x7f46bc0d1e90 legacy=0x7f46bc0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46a0005210 0x7f46a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0cf9d0 con 0x7f46a0003a00 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a000c160 con 0x7f46a0004800 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a000c360 con 0x7f46a0005210 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.092+0000 7f46baa00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46bb400640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46a0005210 0x7f46a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46b3e00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46baa00640 1 -- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46a0005210 msgr2=0x7f46a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46baa00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46a0005210 0x7f46a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46baa00640 1 -- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46a0004800 msgr2=0x7f46a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46baa00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46baa00640 1 -- 172.21.15.97:0/624050440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0003520 con 0x7f46a0003a00 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.093+0000 7f46bb400640 1 -- 172.21.15.97:0/624050440 reap_dead start 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.094+0000 7f46baa00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46a0003e20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00ba30 tx=0x7f46a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f87826a74e99adf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.094+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac036020 con 0x7f46a0003a00 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.094+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac032660 con 0x7f46a0003a00 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.094+0000 7f46ba000640 1 -- 172.21.15.97:0/624050440 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00b600 con 0x7f46a0003a00 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 -- 172.21.15.97:0/624050440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 msgr2=0x7f46a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46a0003e20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00ba30 tx=0x7f46a0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 -- 172.21.15.97:0/624050440 shutdown_connections 2024-03-06T07:27:52.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 --2- 172.21.15.97:0/624050440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46a0003e20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 -- 172.21.15.97:0/624050440 >> 172.21.15.97:0/624050440 conn(0x7f46bc0d3d80 msgr2=0x7f46bc0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 -- 172.21.15.97:0/624050440 shutdown_connections 2024-03-06T07:27:52.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.095+0000 7f46bbe00640 1 -- 172.21.15.97:0/624050440 wait complete. 2024-03-06T07:27:52.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.097+0000 7f46bbe00640 1 Processor -- start 2024-03-06T07:27:52.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.097+0000 7f46bbe00640 1 -- start start 2024-03-06T07:27:52.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46bc0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46bbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46bc154080 0x7f46bc0b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46bbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46bc0b6f90 0x7f46bc0b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46bbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0cf3a0 con 0x7f46a0003a00 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46bbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0b39a0 con 0x7f46bc154080 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46bbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0d1950 con 0x7f46bc0b6f90 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46bc0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46bc0c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50706/0 (socket says 172.21.15.97:50706) 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 -- 172.21.15.97:0/962228194 learned_addr learned my addr 172.21.15.97:0/962228194 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 -- 172.21.15.97:0/962228194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46bc0b6f90 msgr2=0x7f46bc0b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46bc0b6f90 0x7f46bc0b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 -- 172.21.15.97:0/962228194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46bc154080 msgr2=0x7f46bc0b96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46bc154080 0x7f46bc0b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.098+0000 7f46baa00640 1 -- 172.21.15.97:0/962228194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0b9c30 con 0x7f46a0003a00 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.099+0000 7f46baa00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46bc0c9810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00bb20 tx=0x7f46ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.099+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac036040 con 0x7f46a0003a00 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.099+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac030aa0 con 0x7f46a0003a00 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.099+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc0c9df0 con 0x7f46a0003a00 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.099+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00b600 con 0x7f46a0003a00 2024-03-06T07:27:52.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.099+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc0ca3b0 con 0x7f46a0003a00 2024-03-06T07:27:52.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.100+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46bc04f330 con 0x7f46a0003a00 2024-03-06T07:27:52.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.101+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46ac039020 con 0x7f46a0003a00 2024-03-06T07:27:52.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.102+0000 7f46b2a00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f469806b500 0x7f469806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.102+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46ac0ac4d0 con 0x7f46a0003a00 2024-03-06T07:27:52.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.102+0000 7f46b3e00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f469806b500 0x7f469806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.103+0000 7f46b3e00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f469806b500 0x7f469806d9c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0d13e0 tx=0x7f46a4000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.111+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46ac030e70 con 0x7f46a0003a00 2024-03-06T07:27:52.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.391+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46bc0d3b60 con 0x7f469806b500 2024-03-06T07:27:52.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.409+0000 7f46b2a00640 1 -- 172.21.15.97:0/962228194 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251838 (secure 0 0 0) 0x7f46bc0d3b60 con 0x7f469806b500 2024-03-06T07:27:52.414 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:52.415 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:27:52.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f469806b500 msgr2=0x7f469806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f469806b500 0x7f469806d9c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0d13e0 tx=0x7f46a4000ed0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 msgr2=0x7f46bc0c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46bc0c9810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00bb20 tx=0x7f46ac004730 comp rx=0 tx=0).stop 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 shutdown_connections 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f469806b500 0x7f469806d9c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46bc0b6f90 0x7f46bc0b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46bc154080 0x7f46bc0b96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 --2- 172.21.15.97:0/962228194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a0003a00 0x7f46bc0c9810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 >> 172.21.15.97:0/962228194 conn(0x7f46bc0d3d80 msgr2=0x7f46bc0b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 shutdown_connections 2024-03-06T07:27:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.414+0000 7f46bbe00640 1 -- 172.21.15.97:0/962228194 wait complete. 2024-03-06T07:27:52.436 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:27:50.905442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219460,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755260,"statfs":{"total":773094113280,"available":772869386240,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.842820"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669901+0000","last_change":"2024-03-06T07:27:42.115947+0000","last_active":"2024-03-06T07:27:43.669901+0000","last_peered":"2024-03-06T07:27:43.669901+0000","last_clean":"2024-03-06T07:27:43.669901+0000","last_became_active":"2024-03-06T07:27:42.115323+0000","last_became_peered":"2024-03-06T07:27:42.115323+0000","last_unstale":"2024-03-06T07:27:43.669901+0000","last_undegraded":"2024-03-06T07:27:43.669901+0000","last_fullsized":"2024-03-06T07:27:43.669901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:23.978663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090443+0000","last_change":"2024-03-06T07:27:42.092048+0000","last_active":"2024-03-06T07:27:43.090443+0000","last_peered":"2024-03-06T07:27:43.090443+0000","last_clean":"2024-03-06T07:27:43.090443+0000","last_became_active":"2024-03-06T07:27:42.091783+0000","last_became_peered":"2024-03-06T07:27:42.091783+0000","last_unstale":"2024-03-06T07:27:43.090443+0000","last_undegraded":"2024-03-06T07:27:43.090443+0000","last_fullsized":"2024-03-06T07:27:43.090443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561478+0000","last_change":"2024-03-06T07:27:42.095834+0000","last_active":"2024-03-06T07:27:43.561478+0000","last_peered":"2024-03-06T07:27:43.561478+0000","last_clean":"2024-03-06T07:27:43.561478+0000","last_became_active":"2024-03-06T07:27:42.095026+0000","last_became_peered":"2024-03-06T07:27:42.095026+0000","last_unstale":"2024-03-06T07:27:43.561478+0000","last_undegraded":"2024-03-06T07:27:43.561478+0000","last_fullsized":"2024-03-06T07:27:43.561478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604760+0000","last_change":"2024-03-06T07:27:42.114409+0000","last_active":"2024-03-06T07:27:43.604760+0000","last_peered":"2024-03-06T07:27:43.604760+0000","last_clean":"2024-03-06T07:27:43.604760+0000","last_became_active":"2024-03-06T07:27:42.114122+0000","last_became_peered":"2024-03-06T07:27:42.114122+0000","last_unstale":"2024-03-06T07:27:43.604760+0000","last_undegraded":"2024-03-06T07:27:43.604760+0000","last_fullsized":"2024-03-06T07:27:43.604760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668591+0000","last_change":"2024-03-06T07:27:42.101691+0000","last_active":"2024-03-06T07:27:43.668591+0000","last_peered":"2024-03-06T07:27:43.668591+0000","last_clean":"2024-03-06T07:27:43.668591+0000","last_became_active":"2024-03-06T07:27:42.101372+0000","last_became_peered":"2024-03-06T07:27:42.101372+0000","last_unstale":"2024-03-06T07:27:43.668591+0000","last_undegraded":"2024-03-06T07:27:43.668591+0000","last_fullsized":"2024-03-06T07:27:43.668591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604137+0000","last_change":"2024-03-06T07:27:42.094768+0000","last_active":"2024-03-06T07:27:43.604137+0000","last_peered":"2024-03-06T07:27:43.604137+0000","last_clean":"2024-03-06T07:27:43.604137+0000","last_became_active":"2024-03-06T07:27:42.094416+0000","last_became_peered":"2024-03-06T07:27:42.094416+0000","last_unstale":"2024-03-06T07:27:43.604137+0000","last_undegraded":"2024-03-06T07:27:43.604137+0000","last_fullsized":"2024-03-06T07:27:43.604137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668352+0000","last_change":"2024-03-06T07:27:42.101468+0000","last_active":"2024-03-06T07:27:43.668352+0000","last_peered":"2024-03-06T07:27:43.668352+0000","last_clean":"2024-03-06T07:27:43.668352+0000","last_became_active":"2024-03-06T07:27:42.101168+0000","last_became_peered":"2024-03-06T07:27:42.101168+0000","last_unstale":"2024-03-06T07:27:43.668352+0000","last_undegraded":"2024-03-06T07:27:43.668352+0000","last_fullsized":"2024-03-06T07:27:43.668352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.896115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265352+0000","last_change":"2024-03-06T07:27:42.098517+0000","last_active":"2024-03-06T07:27:43.265352+0000","last_peered":"2024-03-06T07:27:43.265352+0000","last_clean":"2024-03-06T07:27:43.265352+0000","last_became_active":"2024-03-06T07:27:42.092643+0000","last_became_peered":"2024-03-06T07:27:42.092643+0000","last_unstale":"2024-03-06T07:27:43.265352+0000","last_undegraded":"2024-03-06T07:27:43.265352+0000","last_fullsized":"2024-03-06T07:27:43.265352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669864+0000","last_change":"2024-03-06T07:27:42.119208+0000","last_active":"2024-03-06T07:27:43.669864+0000","last_peered":"2024-03-06T07:27:43.669864+0000","last_clean":"2024-03-06T07:27:43.669864+0000","last_became_active":"2024-03-06T07:27:42.118050+0000","last_became_peered":"2024-03-06T07:27:42.118050+0000","last_unstale":"2024-03-06T07:27:43.669864+0000","last_undegraded":"2024-03-06T07:27:43.669864+0000","last_fullsized":"2024-03-06T07:27:43.669864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:43.473299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096548+0000","last_change":"2024-03-06T07:27:42.095950+0000","last_active":"2024-03-06T07:27:43.096548+0000","last_peered":"2024-03-06T07:27:43.096548+0000","last_clean":"2024-03-06T07:27:43.096548+0000","last_became_active":"2024-03-06T07:27:42.095462+0000","last_became_peered":"2024-03-06T07:27:42.095462+0000","last_unstale":"2024-03-06T07:27:43.096548+0000","last_undegraded":"2024-03-06T07:27:43.096548+0000","last_fullsized":"2024-03-06T07:27:43.096548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095953+0000","last_change":"2024-03-06T07:27:42.096143+0000","last_active":"2024-03-06T07:27:43.095953+0000","last_peered":"2024-03-06T07:27:43.095953+0000","last_clean":"2024-03-06T07:27:43.095953+0000","last_became_active":"2024-03-06T07:27:42.095793+0000","last_became_peered":"2024-03-06T07:27:42.095793+0000","last_unstale":"2024-03-06T07:27:43.095953+0000","last_undegraded":"2024-03-06T07:27:43.095953+0000","last_fullsized":"2024-03-06T07:27:43.095953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434202+0000","last_change":"2024-03-06T07:27:42.092238+0000","last_active":"2024-03-06T07:27:43.434202+0000","last_peered":"2024-03-06T07:27:43.434202+0000","last_clean":"2024-03-06T07:27:43.434202+0000","last_became_active":"2024-03-06T07:27:42.091930+0000","last_became_peered":"2024-03-06T07:27:42.091930+0000","last_unstale":"2024-03-06T07:27:43.434202+0000","last_undegraded":"2024-03-06T07:27:43.434202+0000","last_fullsized":"2024-03-06T07:27:43.434202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:38.321736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264768+0000","last_change":"2024-03-06T07:27:42.097468+0000","last_active":"2024-03-06T07:27:43.264768+0000","last_peered":"2024-03-06T07:27:43.264768+0000","last_clean":"2024-03-06T07:27:43.264768+0000","last_became_active":"2024-03-06T07:27:42.094036+0000","last_became_peered":"2024-03-06T07:27:42.094036+0000","last_unstale":"2024-03-06T07:27:43.264768+0000","last_undegraded":"2024-03-06T07:27:43.264768+0000","last_fullsized":"2024-03-06T07:27:43.264768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561085+0000","last_change":"2024-03-06T07:27:42.092704+0000","last_active":"2024-03-06T07:27:43.561085+0000","last_peered":"2024-03-06T07:27:43.561085+0000","last_clean":"2024-03-06T07:27:43.561085+0000","last_became_active":"2024-03-06T07:27:42.090459+0000","last_became_peered":"2024-03-06T07:27:42.090459+0000","last_unstale":"2024-03-06T07:27:43.561085+0000","last_undegraded":"2024-03-06T07:27:43.561085+0000","last_fullsized":"2024-03-06T07:27:43.561085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:26.470370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089922+0000","last_change":"2024-03-06T07:27:42.076294+0000","last_active":"2024-03-06T07:27:43.089922+0000","last_peered":"2024-03-06T07:27:43.089922+0000","last_clean":"2024-03-06T07:27:43.089922+0000","last_became_active":"2024-03-06T07:27:42.075529+0000","last_became_peered":"2024-03-06T07:27:42.075529+0000","last_unstale":"2024-03-06T07:27:43.089922+0000","last_undegraded":"2024-03-06T07:27:43.089922+0000","last_fullsized":"2024-03-06T07:27:43.089922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604410+0000","last_change":"2024-03-06T07:27:42.093846+0000","last_active":"2024-03-06T07:27:43.604410+0000","last_peered":"2024-03-06T07:27:43.604410+0000","last_clean":"2024-03-06T07:27:43.604410+0000","last_became_active":"2024-03-06T07:27:42.093544+0000","last_became_peered":"2024-03-06T07:27:42.093544+0000","last_unstale":"2024-03-06T07:27:43.604410+0000","last_undegraded":"2024-03-06T07:27:43.604410+0000","last_fullsized":"2024-03-06T07:27:43.604410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605458+0000","last_change":"2024-03-06T07:27:42.118690+0000","last_active":"2024-03-06T07:27:43.605458+0000","last_peered":"2024-03-06T07:27:43.605458+0000","last_clean":"2024-03-06T07:27:43.605458+0000","last_became_active":"2024-03-06T07:27:42.118405+0000","last_became_peered":"2024-03-06T07:27:42.118405+0000","last_unstale":"2024-03-06T07:27:43.605458+0000","last_undegraded":"2024-03-06T07:27:43.605458+0000","last_fullsized":"2024-03-06T07:27:43.605458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093672+0000","last_change":"2024-03-06T07:27:42.096245+0000","last_active":"2024-03-06T07:27:43.093672+0000","last_peered":"2024-03-06T07:27:43.093672+0000","last_clean":"2024-03-06T07:27:43.093672+0000","last_became_active":"2024-03-06T07:27:42.094600+0000","last_became_peered":"2024-03-06T07:27:42.094600+0000","last_unstale":"2024-03-06T07:27:43.093672+0000","last_undegraded":"2024-03-06T07:27:43.093672+0000","last_fullsized":"2024-03-06T07:27:43.093672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:05.681114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669317+0000","last_change":"2024-03-06T07:27:42.104820+0000","last_active":"2024-03-06T07:27:43.669317+0000","last_peered":"2024-03-06T07:27:43.669317+0000","last_clean":"2024-03-06T07:27:43.669317+0000","last_became_active":"2024-03-06T07:27:42.104489+0000","last_became_peered":"2024-03-06T07:27:42.104489+0000","last_unstale":"2024-03-06T07:27:43.669317+0000","last_undegraded":"2024-03-06T07:27:43.669317+0000","last_fullsized":"2024-03-06T07:27:43.669317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:25.177146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096181+0000","last_change":"2024-03-06T07:27:42.105851+0000","last_active":"2024-03-06T07:27:43.096181+0000","last_peered":"2024-03-06T07:27:43.096181+0000","last_clean":"2024-03-06T07:27:43.096181+0000","last_became_active":"2024-03-06T07:27:42.105276+0000","last_became_peered":"2024-03-06T07:27:42.105276+0000","last_unstale":"2024-03-06T07:27:43.096181+0000","last_undegraded":"2024-03-06T07:27:43.096181+0000","last_fullsized":"2024-03-06T07:27:43.096181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561313+0000","last_change":"2024-03-06T07:27:42.107395+0000","last_active":"2024-03-06T07:27:43.561313+0000","last_peered":"2024-03-06T07:27:43.561313+0000","last_clean":"2024-03-06T07:27:43.561313+0000","last_became_active":"2024-03-06T07:27:42.104429+0000","last_became_peered":"2024-03-06T07:27:42.104429+0000","last_unstale":"2024-03-06T07:27:43.561313+0000","last_undegraded":"2024-03-06T07:27:43.561313+0000","last_fullsized":"2024-03-06T07:27:43.561313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604391+0000","last_change":"2024-03-06T07:27:42.105460+0000","last_active":"2024-03-06T07:27:43.604391+0000","last_peered":"2024-03-06T07:27:43.604391+0000","last_clean":"2024-03-06T07:27:43.604391+0000","last_became_active":"2024-03-06T07:27:42.103643+0000","last_became_peered":"2024-03-06T07:27:42.103643+0000","last_unstale":"2024-03-06T07:27:43.604391+0000","last_undegraded":"2024-03-06T07:27:43.604391+0000","last_fullsized":"2024-03-06T07:27:43.604391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264480+0000","last_change":"2024-03-06T07:27:42.094539+0000","last_active":"2024-03-06T07:27:43.264480+0000","last_peered":"2024-03-06T07:27:43.264480+0000","last_clean":"2024-03-06T07:27:43.264480+0000","last_became_active":"2024-03-06T07:27:42.091758+0000","last_became_peered":"2024-03-06T07:27:42.091758+0000","last_unstale":"2024-03-06T07:27:43.264480+0000","last_undegraded":"2024-03-06T07:27:43.264480+0000","last_fullsized":"2024-03-06T07:27:43.264480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:22.670042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604546+0000","last_change":"2024-03-06T07:27:42.113737+0000","last_active":"2024-03-06T07:27:43.604546+0000","last_peered":"2024-03-06T07:27:43.604546+0000","last_clean":"2024-03-06T07:27:43.604546+0000","last_became_active":"2024-03-06T07:27:42.113444+0000","last_became_peered":"2024-03-06T07:27:42.113444+0000","last_unstale":"2024-03-06T07:27:43.604546+0000","last_undegraded":"2024-03-06T07:27:43.604546+0000","last_fullsized":"2024-03-06T07:27:43.604546+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:58.518779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265404+0000","last_change":"2024-03-06T07:27:42.104966+0000","last_active":"2024-03-06T07:27:43.265404+0000","last_peered":"2024-03-06T07:27:43.265404+0000","last_clean":"2024-03-06T07:27:43.265404+0000","last_became_active":"2024-03-06T07:27:42.104220+0000","last_became_peered":"2024-03-06T07:27:42.104220+0000","last_unstale":"2024-03-06T07:27:43.265404+0000","last_undegraded":"2024-03-06T07:27:43.265404+0000","last_fullsized":"2024-03-06T07:27:43.265404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096684+0000","last_change":"2024-03-06T07:27:42.103398+0000","last_active":"2024-03-06T07:27:43.096684+0000","last_peered":"2024-03-06T07:27:43.096684+0000","last_clean":"2024-03-06T07:27:43.096684+0000","last_became_active":"2024-03-06T07:27:42.102707+0000","last_became_peered":"2024-03-06T07:27:42.102707+0000","last_unstale":"2024-03-06T07:27:43.096684+0000","last_undegraded":"2024-03-06T07:27:43.096684+0000","last_fullsized":"2024-03-06T07:27:43.096684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090325+0000","last_change":"2024-03-06T07:27:42.092577+0000","last_active":"2024-03-06T07:27:43.090325+0000","last_peered":"2024-03-06T07:27:43.090325+0000","last_clean":"2024-03-06T07:27:43.090325+0000","last_became_active":"2024-03-06T07:27:42.092269+0000","last_became_peered":"2024-03-06T07:27:42.092269+0000","last_unstale":"2024-03-06T07:27:43.090325+0000","last_undegraded":"2024-03-06T07:27:43.090325+0000","last_fullsized":"2024-03-06T07:27:43.090325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265145+0000","last_change":"2024-03-06T07:27:42.105501+0000","last_active":"2024-03-06T07:27:43.265145+0000","last_peered":"2024-03-06T07:27:43.265145+0000","last_clean":"2024-03-06T07:27:43.265145+0000","last_became_active":"2024-03-06T07:27:42.100336+0000","last_became_peered":"2024-03-06T07:27:42.100336+0000","last_unstale":"2024-03-06T07:27:43.265145+0000","last_undegraded":"2024-03-06T07:27:43.265145+0000","last_fullsized":"2024-03-06T07:27:43.265145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:20.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561748+0000","last_change":"2024-03-06T07:27:42.090601+0000","last_active":"2024-03-06T07:27:43.561748+0000","last_peered":"2024-03-06T07:27:43.561748+0000","last_clean":"2024-03-06T07:27:43.561748+0000","last_became_active":"2024-03-06T07:27:42.090283+0000","last_became_peered":"2024-03-06T07:27:42.090283+0000","last_unstale":"2024-03-06T07:27:43.561748+0000","last_undegraded":"2024-03-06T07:27:43.561748+0000","last_fullsized":"2024-03-06T07:27:43.561748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604604+0000","last_change":"2024-03-06T07:27:42.113977+0000","last_active":"2024-03-06T07:27:43.604604+0000","last_peered":"2024-03-06T07:27:43.604604+0000","last_clean":"2024-03-06T07:27:43.604604+0000","last_became_active":"2024-03-06T07:27:42.111520+0000","last_became_peered":"2024-03-06T07:27:42.111520+0000","last_unstale":"2024-03-06T07:27:43.604604+0000","last_undegraded":"2024-03-06T07:27:43.604604+0000","last_fullsized":"2024-03-06T07:27:43.604604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093842+0000","last_change":"2024-03-06T07:27:42.096735+0000","last_active":"2024-03-06T07:27:43.093842+0000","last_peered":"2024-03-06T07:27:43.093842+0000","last_clean":"2024-03-06T07:27:43.093842+0000","last_became_active":"2024-03-06T07:27:42.096304+0000","last_became_peered":"2024-03-06T07:27:42.096304+0000","last_unstale":"2024-03-06T07:27:43.093842+0000","last_undegraded":"2024-03-06T07:27:43.093842+0000","last_fullsized":"2024-03-06T07:27:43.093842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:37.292940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095982+0000","last_change":"2024-03-06T07:27:42.096332+0000","last_active":"2024-03-06T07:27:43.095982+0000","last_peered":"2024-03-06T07:27:43.095982+0000","last_clean":"2024-03-06T07:27:43.095982+0000","last_became_active":"2024-03-06T07:27:42.095984+0000","last_became_peered":"2024-03-06T07:27:42.095984+0000","last_unstale":"2024-03-06T07:27:43.095982+0000","last_undegraded":"2024-03-06T07:27:43.095982+0000","last_fullsized":"2024-03-06T07:27:43.095982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.433842+0000","last_change":"2024-03-06T07:27:42.091042+0000","last_active":"2024-03-06T07:27:43.433842+0000","last_peered":"2024-03-06T07:27:43.433842+0000","last_clean":"2024-03-06T07:27:43.433842+0000","last_became_active":"2024-03-06T07:27:42.089637+0000","last_became_peered":"2024-03-06T07:27:42.089637+0000","last_unstale":"2024-03-06T07:27:43.433842+0000","last_undegraded":"2024-03-06T07:27:43.433842+0000","last_fullsized":"2024-03-06T07:27:43.433842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:00.218649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265208+0000","last_change":"2024-03-06T07:27:41.111941+0000","last_active":"2024-03-06T07:27:43.265208+0000","last_peered":"2024-03-06T07:27:43.265208+0000","last_clean":"2024-03-06T07:27:43.265208+0000","last_became_active":"2024-03-06T07:27:41.111244+0000","last_became_peered":"2024-03-06T07:27:41.111244+0000","last_unstale":"2024-03-06T07:27:43.265208+0000","last_undegraded":"2024-03-06T07:27:43.265208+0000","last_fullsized":"2024-03-06T07:27:43.265208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561659+0000","last_change":"2024-03-06T07:27:42.098732+0000","last_active":"2024-03-06T07:27:43.561659+0000","last_peered":"2024-03-06T07:27:43.561659+0000","last_clean":"2024-03-06T07:27:43.561659+0000","last_became_active":"2024-03-06T07:27:42.098451+0000","last_became_peered":"2024-03-06T07:27:42.098451+0000","last_unstale":"2024-03-06T07:27:43.561659+0000","last_undegraded":"2024-03-06T07:27:43.561659+0000","last_fullsized":"2024-03-06T07:27:43.561659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:41.321169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095423+0000","last_change":"2024-03-06T07:27:41.102703+0000","last_active":"2024-03-06T07:27:43.095423+0000","last_peered":"2024-03-06T07:27:43.095423+0000","last_clean":"2024-03-06T07:27:43.095423+0000","last_became_active":"2024-03-06T07:27:41.100059+0000","last_became_peered":"2024-03-06T07:27:41.100059+0000","last_unstale":"2024-03-06T07:27:43.095423+0000","last_undegraded":"2024-03-06T07:27:43.095423+0000","last_fullsized":"2024-03-06T07:27:43.095423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264571+0000","last_change":"2024-03-06T07:27:41.105849+0000","last_active":"2024-03-06T07:27:43.264571+0000","last_peered":"2024-03-06T07:27:43.264571+0000","last_clean":"2024-03-06T07:27:43.264571+0000","last_became_active":"2024-03-06T07:27:41.105558+0000","last_became_peered":"2024-03-06T07:27:41.105558+0000","last_unstale":"2024-03-06T07:27:43.264571+0000","last_undegraded":"2024-03-06T07:27:43.264571+0000","last_fullsized":"2024-03-06T07:27:43.264571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264534+0000","last_change":"2024-03-06T07:27:42.098554+0000","last_active":"2024-03-06T07:27:43.264534+0000","last_peered":"2024-03-06T07:27:43.264534+0000","last_clean":"2024-03-06T07:27:43.264534+0000","last_became_active":"2024-03-06T07:27:42.097364+0000","last_became_peered":"2024-03-06T07:27:42.097364+0000","last_unstale":"2024-03-06T07:27:43.264534+0000","last_undegraded":"2024-03-06T07:27:43.264534+0000","last_fullsized":"2024-03-06T07:27:43.264534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604701+0000","last_change":"2024-03-06T07:27:41.103209+0000","last_active":"2024-03-06T07:27:43.604701+0000","last_peered":"2024-03-06T07:27:43.604701+0000","last_clean":"2024-03-06T07:27:43.604701+0000","last_became_active":"2024-03-06T07:27:41.102994+0000","last_became_peered":"2024-03-06T07:27:41.102994+0000","last_unstale":"2024-03-06T07:27:43.604701+0000","last_undegraded":"2024-03-06T07:27:43.604701+0000","last_fullsized":"2024-03-06T07:27:43.604701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090178+0000","last_change":"2024-03-06T07:27:42.086153+0000","last_active":"2024-03-06T07:27:43.090178+0000","last_peered":"2024-03-06T07:27:43.090178+0000","last_clean":"2024-03-06T07:27:43.090178+0000","last_became_active":"2024-03-06T07:27:42.085779+0000","last_became_peered":"2024-03-06T07:27:42.085779+0000","last_unstale":"2024-03-06T07:27:43.090178+0000","last_undegraded":"2024-03-06T07:27:43.090178+0000","last_fullsized":"2024-03-06T07:27:43.090178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096443+0000","last_change":"2024-03-06T07:27:41.109144+0000","last_active":"2024-03-06T07:27:43.096443+0000","last_peered":"2024-03-06T07:27:43.096443+0000","last_clean":"2024-03-06T07:27:43.096443+0000","last_became_active":"2024-03-06T07:27:41.108399+0000","last_became_peered":"2024-03-06T07:27:41.108399+0000","last_unstale":"2024-03-06T07:27:43.096443+0000","last_undegraded":"2024-03-06T07:27:43.096443+0000","last_fullsized":"2024-03-06T07:27:43.096443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604733+0000","last_change":"2024-03-06T07:27:42.106823+0000","last_active":"2024-03-06T07:27:43.604733+0000","last_peered":"2024-03-06T07:27:43.604733+0000","last_clean":"2024-03-06T07:27:43.604733+0000","last_became_active":"2024-03-06T07:27:42.103949+0000","last_became_peered":"2024-03-06T07:27:42.103949+0000","last_unstale":"2024-03-06T07:27:43.604733+0000","last_undegraded":"2024-03-06T07:27:43.604733+0000","last_fullsized":"2024-03-06T07:27:43.604733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605002+0000","last_change":"2024-03-06T07:27:41.108455+0000","last_active":"2024-03-06T07:27:43.605002+0000","last_peered":"2024-03-06T07:27:43.605002+0000","last_clean":"2024-03-06T07:27:43.605002+0000","last_became_active":"2024-03-06T07:27:41.106095+0000","last_became_peered":"2024-03-06T07:27:41.106095+0000","last_unstale":"2024-03-06T07:27:43.605002+0000","last_undegraded":"2024-03-06T07:27:43.605002+0000","last_fullsized":"2024-03-06T07:27:43.605002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561945+0000","last_change":"2024-03-06T07:27:42.106635+0000","last_active":"2024-03-06T07:27:43.561945+0000","last_peered":"2024-03-06T07:27:43.561945+0000","last_clean":"2024-03-06T07:27:43.561945+0000","last_became_active":"2024-03-06T07:27:42.106373+0000","last_became_peered":"2024-03-06T07:27:42.106373+0000","last_unstale":"2024-03-06T07:27:43.561945+0000","last_undegraded":"2024-03-06T07:27:43.561945+0000","last_fullsized":"2024-03-06T07:27:43.561945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090296+0000","last_change":"2024-03-06T07:27:41.108679+0000","last_active":"2024-03-06T07:27:43.090296+0000","last_peered":"2024-03-06T07:27:43.090296+0000","last_clean":"2024-03-06T07:27:43.090296+0000","last_became_active":"2024-03-06T07:27:41.108429+0000","last_became_peered":"2024-03-06T07:27:41.108429+0000","last_unstale":"2024-03-06T07:27:43.090296+0000","last_undegraded":"2024-03-06T07:27:43.090296+0000","last_fullsized":"2024-03-06T07:27:43.090296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434363+0000","last_change":"2024-03-06T07:27:42.102682+0000","last_active":"2024-03-06T07:27:43.434363+0000","last_peered":"2024-03-06T07:27:43.434363+0000","last_clean":"2024-03-06T07:27:43.434363+0000","last_became_active":"2024-03-06T07:27:42.102232+0000","last_became_peered":"2024-03-06T07:27:42.102232+0000","last_unstale":"2024-03-06T07:27:43.434363+0000","last_undegraded":"2024-03-06T07:27:43.434363+0000","last_fullsized":"2024-03-06T07:27:43.434363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:34.589802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096130+0000","last_change":"2024-03-06T07:27:41.096688+0000","last_active":"2024-03-06T07:27:43.096130+0000","last_peered":"2024-03-06T07:27:43.096130+0000","last_clean":"2024-03-06T07:27:43.096130+0000","last_became_active":"2024-03-06T07:27:41.096391+0000","last_became_peered":"2024-03-06T07:27:41.096391+0000","last_unstale":"2024-03-06T07:27:43.096130+0000","last_undegraded":"2024-03-06T07:27:43.096130+0000","last_fullsized":"2024-03-06T07:27:43.096130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668491+0000","last_change":"2024-03-06T07:27:42.116169+0000","last_active":"2024-03-06T07:27:43.668491+0000","last_peered":"2024-03-06T07:27:43.668491+0000","last_clean":"2024-03-06T07:27:43.668491+0000","last_became_active":"2024-03-06T07:27:42.115868+0000","last_became_peered":"2024-03-06T07:27:42.115868+0000","last_unstale":"2024-03-06T07:27:43.668491+0000","last_undegraded":"2024-03-06T07:27:43.668491+0000","last_fullsized":"2024-03-06T07:27:43.668491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.091480+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.090511+0000","last_became_peered":"2024-03-06T07:27:41.090511+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:18.039278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561169+0000","last_change":"2024-03-06T07:27:42.104948+0000","last_active":"2024-03-06T07:27:43.561169+0000","last_peered":"2024-03-06T07:27:43.561169+0000","last_clean":"2024-03-06T07:27:43.561169+0000","last_became_active":"2024-03-06T07:27:42.104640+0000","last_became_peered":"2024-03-06T07:27:42.104640+0000","last_unstale":"2024-03-06T07:27:43.561169+0000","last_undegraded":"2024-03-06T07:27:43.561169+0000","last_fullsized":"2024-03-06T07:27:43.561169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604876+0000","last_change":"2024-03-06T07:27:41.103424+0000","last_active":"2024-03-06T07:27:43.604876+0000","last_peered":"2024-03-06T07:27:43.604876+0000","last_clean":"2024-03-06T07:27:43.604876+0000","last_became_active":"2024-03-06T07:27:41.102084+0000","last_became_peered":"2024-03-06T07:27:41.102084+0000","last_unstale":"2024-03-06T07:27:43.604876+0000","last_undegraded":"2024-03-06T07:27:43.604876+0000","last_fullsized":"2024-03-06T07:27:43.604876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668345+0000","last_change":"2024-03-06T07:27:42.107947+0000","last_active":"2024-03-06T07:27:43.668345+0000","last_peered":"2024-03-06T07:27:43.668345+0000","last_clean":"2024-03-06T07:27:43.668345+0000","last_became_active":"2024-03-06T07:27:42.107570+0000","last_became_peered":"2024-03-06T07:27:42.107570+0000","last_unstale":"2024-03-06T07:27:43.668345+0000","last_undegraded":"2024-03-06T07:27:43.668345+0000","last_fullsized":"2024-03-06T07:27:43.668345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095592+0000","last_change":"2024-03-06T07:27:41.105127+0000","last_active":"2024-03-06T07:27:43.095592+0000","last_peered":"2024-03-06T07:27:43.095592+0000","last_clean":"2024-03-06T07:27:43.095592+0000","last_became_active":"2024-03-06T07:27:41.104134+0000","last_became_peered":"2024-03-06T07:27:41.104134+0000","last_unstale":"2024-03-06T07:27:43.095592+0000","last_undegraded":"2024-03-06T07:27:43.095592+0000","last_fullsized":"2024-03-06T07:27:43.095592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096137+0000","last_change":"2024-03-06T07:27:42.104623+0000","last_active":"2024-03-06T07:27:43.096137+0000","last_peered":"2024-03-06T07:27:43.096137+0000","last_clean":"2024-03-06T07:27:43.096137+0000","last_became_active":"2024-03-06T07:27:42.104185+0000","last_became_peered":"2024-03-06T07:27:42.104185+0000","last_unstale":"2024-03-06T07:27:43.096137+0000","last_undegraded":"2024-03-06T07:27:43.096137+0000","last_fullsized":"2024-03-06T07:27:43.096137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093473+0000","last_change":"2024-03-06T07:27:41.095963+0000","last_active":"2024-03-06T07:27:43.093473+0000","last_peered":"2024-03-06T07:27:43.093473+0000","last_clean":"2024-03-06T07:27:43.093473+0000","last_became_active":"2024-03-06T07:27:41.095610+0000","last_became_peered":"2024-03-06T07:27:41.095610+0000","last_unstale":"2024-03-06T07:27:43.093473+0000","last_undegraded":"2024-03-06T07:27:43.093473+0000","last_fullsized":"2024-03-06T07:27:43.093473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668062+0000","last_change":"2024-03-06T07:27:42.107796+0000","last_active":"2024-03-06T07:27:43.668062+0000","last_peered":"2024-03-06T07:27:43.668062+0000","last_clean":"2024-03-06T07:27:43.668062+0000","last_became_active":"2024-03-06T07:27:42.107514+0000","last_became_peered":"2024-03-06T07:27:42.107514+0000","last_unstale":"2024-03-06T07:27:43.668062+0000","last_undegraded":"2024-03-06T07:27:43.668062+0000","last_fullsized":"2024-03-06T07:27:43.668062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:07.050458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434682+0000","last_change":"2024-03-06T07:27:41.094350+0000","last_active":"2024-03-06T07:27:43.434682+0000","last_peered":"2024-03-06T07:27:43.434682+0000","last_clean":"2024-03-06T07:27:43.434682+0000","last_became_active":"2024-03-06T07:27:41.089935+0000","last_became_peered":"2024-03-06T07:27:41.089935+0000","last_unstale":"2024-03-06T07:27:43.434682+0000","last_undegraded":"2024-03-06T07:27:43.434682+0000","last_fullsized":"2024-03-06T07:27:43.434682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:49.952499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669175+0000","last_change":"2024-03-06T07:27:42.109835+0000","last_active":"2024-03-06T07:27:43.669175+0000","last_peered":"2024-03-06T07:27:43.669175+0000","last_clean":"2024-03-06T07:27:43.669175+0000","last_became_active":"2024-03-06T07:27:42.109535+0000","last_became_peered":"2024-03-06T07:27:42.109535+0000","last_unstale":"2024-03-06T07:27:43.669175+0000","last_undegraded":"2024-03-06T07:27:43.669175+0000","last_fullsized":"2024-03-06T07:27:43.669175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093784+0000","last_change":"2024-03-06T07:27:41.092525+0000","last_active":"2024-03-06T07:27:43.093784+0000","last_peered":"2024-03-06T07:27:43.093784+0000","last_clean":"2024-03-06T07:27:43.093784+0000","last_became_active":"2024-03-06T07:27:41.091644+0000","last_became_peered":"2024-03-06T07:27:41.091644+0000","last_unstale":"2024-03-06T07:27:43.093784+0000","last_undegraded":"2024-03-06T07:27:43.093784+0000","last_fullsized":"2024-03-06T07:27:43.093784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.670064+0000","last_change":"2024-03-06T07:27:42.117934+0000","last_active":"2024-03-06T07:27:43.670064+0000","last_peered":"2024-03-06T07:27:43.670064+0000","last_clean":"2024-03-06T07:27:43.670064+0000","last_became_active":"2024-03-06T07:27:42.115924+0000","last_became_peered":"2024-03-06T07:27:42.115924+0000","last_unstale":"2024-03-06T07:27:43.670064+0000","last_undegraded":"2024-03-06T07:27:43.670064+0000","last_fullsized":"2024-03-06T07:27:43.670064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:39.844169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090372+0000","last_change":"2024-03-06T07:27:41.097118+0000","last_active":"2024-03-06T07:27:43.090372+0000","last_peered":"2024-03-06T07:27:43.090372+0000","last_clean":"2024-03-06T07:27:43.090372+0000","last_became_active":"2024-03-06T07:27:41.096188+0000","last_became_peered":"2024-03-06T07:27:41.096188+0000","last_unstale":"2024-03-06T07:27:43.090372+0000","last_undegraded":"2024-03-06T07:27:43.090372+0000","last_fullsized":"2024-03-06T07:27:43.090372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605208+0000","last_change":"2024-03-06T07:27:42.114130+0000","last_active":"2024-03-06T07:27:43.605208+0000","last_peered":"2024-03-06T07:27:43.605208+0000","last_clean":"2024-03-06T07:27:43.605208+0000","last_became_active":"2024-03-06T07:27:42.113811+0000","last_became_peered":"2024-03-06T07:27:42.113811+0000","last_unstale":"2024-03-06T07:27:43.605208+0000","last_undegraded":"2024-03-06T07:27:43.605208+0000","last_fullsized":"2024-03-06T07:27:43.605208+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093668+0000","last_change":"2024-03-06T07:27:41.102837+0000","last_active":"2024-03-06T07:27:43.093668+0000","last_peered":"2024-03-06T07:27:43.093668+0000","last_clean":"2024-03-06T07:27:43.093668+0000","last_became_active":"2024-03-06T07:27:41.102526+0000","last_became_peered":"2024-03-06T07:27:41.102526+0000","last_unstale":"2024-03-06T07:27:43.093668+0000","last_undegraded":"2024-03-06T07:27:43.093668+0000","last_fullsized":"2024-03-06T07:27:43.093668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:55.067616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668936+0000","last_change":"2024-03-06T07:27:42.110688+0000","last_active":"2024-03-06T07:27:43.668936+0000","last_peered":"2024-03-06T07:27:43.668936+0000","last_clean":"2024-03-06T07:27:43.668936+0000","last_became_active":"2024-03-06T07:27:42.110394+0000","last_became_peered":"2024-03-06T07:27:42.110394+0000","last_unstale":"2024-03-06T07:27:43.668936+0000","last_undegraded":"2024-03-06T07:27:43.668936+0000","last_fullsized":"2024-03-06T07:27:43.668936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:28.305228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667970+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:27:43.667970+0000","last_peered":"2024-03-06T07:27:43.667970+0000","last_clean":"2024-03-06T07:27:43.667970+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:27:43.667970+0000","last_undegraded":"2024-03-06T07:27:43.667970+0000","last_fullsized":"2024-03-06T07:27:43.667970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604170+0000","last_change":"2024-03-06T07:27:41.111128+0000","last_active":"2024-03-06T07:27:43.604170+0000","last_peered":"2024-03-06T07:27:43.604170+0000","last_clean":"2024-03-06T07:27:43.604170+0000","last_became_active":"2024-03-06T07:27:41.109382+0000","last_became_peered":"2024-03-06T07:27:41.109382+0000","last_unstale":"2024-03-06T07:27:43.604170+0000","last_undegraded":"2024-03-06T07:27:43.604170+0000","last_fullsized":"2024-03-06T07:27:43.604170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264748+0000","last_change":"2024-03-06T07:27:42.100303+0000","last_active":"2024-03-06T07:27:43.264748+0000","last_peered":"2024-03-06T07:27:43.264748+0000","last_clean":"2024-03-06T07:27:43.264748+0000","last_became_active":"2024-03-06T07:27:42.099977+0000","last_became_peered":"2024-03-06T07:27:42.099977+0000","last_unstale":"2024-03-06T07:27:43.264748+0000","last_undegraded":"2024-03-06T07:27:43.264748+0000","last_fullsized":"2024-03-06T07:27:43.264748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667817+0000","last_change":"2024-03-06T07:27:41.113323+0000","last_active":"2024-03-06T07:27:43.667817+0000","last_peered":"2024-03-06T07:27:43.667817+0000","last_clean":"2024-03-06T07:27:43.667817+0000","last_became_active":"2024-03-06T07:27:41.112921+0000","last_became_peered":"2024-03-06T07:27:41.112921+0000","last_unstale":"2024-03-06T07:27:43.667817+0000","last_undegraded":"2024-03-06T07:27:43.667817+0000","last_fullsized":"2024-03-06T07:27:43.667817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:12.710184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561173+0000","last_change":"2024-03-06T07:27:42.098275+0000","last_active":"2024-03-06T07:27:43.561173+0000","last_peered":"2024-03-06T07:27:43.561173+0000","last_clean":"2024-03-06T07:27:43.561173+0000","last_became_active":"2024-03-06T07:27:42.097430+0000","last_became_peered":"2024-03-06T07:27:42.097430+0000","last_unstale":"2024-03-06T07:27:43.561173+0000","last_undegraded":"2024-03-06T07:27:43.561173+0000","last_fullsized":"2024-03-06T07:27:43.561173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:43.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.093787+0000","last_change":"2024-03-06T07:27:41.092149+0000","last_active":"2024-03-06T07:27:43.093787+0000","last_peered":"2024-03-06T07:27:43.093787+0000","last_clean":"2024-03-06T07:27:43.093787+0000","last_became_active":"2024-03-06T07:27:41.090942+0000","last_became_peered":"2024-03-06T07:27:41.090942+0000","last_unstale":"2024-03-06T07:27:43.093787+0000","last_undegraded":"2024-03-06T07:27:43.093787+0000","last_fullsized":"2024-03-06T07:27:43.093787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:42.226722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.089953+0000","last_change":"2024-03-06T07:27:42.103118+0000","last_active":"2024-03-06T07:27:43.089953+0000","last_peered":"2024-03-06T07:27:43.089953+0000","last_clean":"2024-03-06T07:27:43.089953+0000","last_became_active":"2024-03-06T07:27:42.102850+0000","last_became_peered":"2024-03-06T07:27:42.102850+0000","last_unstale":"2024-03-06T07:27:43.089953+0000","last_undegraded":"2024-03-06T07:27:43.089953+0000","last_fullsized":"2024-03-06T07:27:43.089953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:21.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605179+0000","last_change":"2024-03-06T07:27:41.111452+0000","last_active":"2024-03-06T07:27:43.605179+0000","last_peered":"2024-03-06T07:27:43.605179+0000","last_clean":"2024-03-06T07:27:43.605179+0000","last_became_active":"2024-03-06T07:27:41.110359+0000","last_became_peered":"2024-03-06T07:27:41.110359+0000","last_unstale":"2024-03-06T07:27:43.605179+0000","last_undegraded":"2024-03-06T07:27:43.605179+0000","last_fullsized":"2024-03-06T07:27:43.605179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434187+0000","last_change":"2024-03-06T07:27:42.093244+0000","last_active":"2024-03-06T07:27:43.434187+0000","last_peered":"2024-03-06T07:27:43.434187+0000","last_clean":"2024-03-06T07:27:43.434187+0000","last_became_active":"2024-03-06T07:27:42.092228+0000","last_became_peered":"2024-03-06T07:27:42.092228+0000","last_unstale":"2024-03-06T07:27:43.434187+0000","last_undegraded":"2024-03-06T07:27:43.434187+0000","last_fullsized":"2024-03-06T07:27:43.434187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:07.427898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090250+0000","last_change":"2024-03-06T07:27:41.093905+0000","last_active":"2024-03-06T07:27:43.090250+0000","last_peered":"2024-03-06T07:27:43.090250+0000","last_clean":"2024-03-06T07:27:43.090250+0000","last_became_active":"2024-03-06T07:27:41.093456+0000","last_became_peered":"2024-03-06T07:27:41.093456+0000","last_unstale":"2024-03-06T07:27:43.090250+0000","last_undegraded":"2024-03-06T07:27:43.090250+0000","last_fullsized":"2024-03-06T07:27:43.090250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.605312+0000","last_change":"2024-03-06T07:27:42.112370+0000","last_active":"2024-03-06T07:27:43.605312+0000","last_peered":"2024-03-06T07:27:43.605312+0000","last_clean":"2024-03-06T07:27:43.605312+0000","last_became_active":"2024-03-06T07:27:42.112087+0000","last_became_peered":"2024-03-06T07:27:42.112087+0000","last_unstale":"2024-03-06T07:27:43.605312+0000","last_undegraded":"2024-03-06T07:27:43.605312+0000","last_fullsized":"2024-03-06T07:27:43.605312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:26.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669326+0000","last_change":"2024-03-06T07:27:41.116636+0000","last_active":"2024-03-06T07:27:43.669326+0000","last_peered":"2024-03-06T07:27:43.669326+0000","last_clean":"2024-03-06T07:27:43.669326+0000","last_became_active":"2024-03-06T07:27:41.116254+0000","last_became_peered":"2024-03-06T07:27:41.116254+0000","last_unstale":"2024-03-06T07:27:43.669326+0000","last_undegraded":"2024-03-06T07:27:43.669326+0000","last_fullsized":"2024-03-06T07:27:43.669326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:59.546367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090070+0000","last_change":"2024-03-06T07:27:42.094877+0000","last_active":"2024-03-06T07:27:43.090070+0000","last_peered":"2024-03-06T07:27:43.090070+0000","last_clean":"2024-03-06T07:27:43.090070+0000","last_became_active":"2024-03-06T07:27:42.094586+0000","last_became_peered":"2024-03-06T07:27:42.094586+0000","last_unstale":"2024-03-06T07:27:43.090070+0000","last_undegraded":"2024-03-06T07:27:43.090070+0000","last_fullsized":"2024-03-06T07:27:43.090070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.102122+0000","last_change":"2024-03-06T07:27:41.098449+0000","last_active":"2024-03-06T07:27:43.102122+0000","last_peered":"2024-03-06T07:27:43.102122+0000","last_clean":"2024-03-06T07:27:43.102122+0000","last_became_active":"2024-03-06T07:27:41.098137+0000","last_became_peered":"2024-03-06T07:27:41.098137+0000","last_unstale":"2024-03-06T07:27:43.102122+0000","last_undegraded":"2024-03-06T07:27:43.102122+0000","last_fullsized":"2024-03-06T07:27:43.102122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265545+0000","last_change":"2024-03-06T07:27:42.118543+0000","last_active":"2024-03-06T07:27:43.265545+0000","last_peered":"2024-03-06T07:27:43.265545+0000","last_clean":"2024-03-06T07:27:43.265545+0000","last_became_active":"2024-03-06T07:27:42.118157+0000","last_became_peered":"2024-03-06T07:27:42.118157+0000","last_unstale":"2024-03-06T07:27:43.265545+0000","last_undegraded":"2024-03-06T07:27:43.265545+0000","last_fullsized":"2024-03-06T07:27:43.265545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:45.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.096632+0000","last_change":"2024-03-06T07:27:41.112840+0000","last_active":"2024-03-06T07:27:43.096632+0000","last_peered":"2024-03-06T07:27:43.096632+0000","last_clean":"2024-03-06T07:27:43.096632+0000","last_became_active":"2024-03-06T07:27:41.112440+0000","last_became_peered":"2024-03-06T07:27:41.112440+0000","last_unstale":"2024-03-06T07:27:43.096632+0000","last_undegraded":"2024-03-06T07:27:43.096632+0000","last_fullsized":"2024-03-06T07:27:43.096632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561550+0000","last_change":"2024-03-06T07:27:42.090517+0000","last_active":"2024-03-06T07:27:43.561550+0000","last_peered":"2024-03-06T07:27:43.561550+0000","last_clean":"2024-03-06T07:27:43.561550+0000","last_became_active":"2024-03-06T07:27:42.090202+0000","last_became_peered":"2024-03-06T07:27:42.090202+0000","last_unstale":"2024-03-06T07:27:43.561550+0000","last_undegraded":"2024-03-06T07:27:43.561550+0000","last_fullsized":"2024-03-06T07:27:43.561550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.667842+0000","last_change":"2024-03-06T07:27:41.114447+0000","last_active":"2024-03-06T07:27:43.667842+0000","last_peered":"2024-03-06T07:27:43.667842+0000","last_clean":"2024-03-06T07:27:43.667842+0000","last_became_active":"2024-03-06T07:27:41.113802+0000","last_became_peered":"2024-03-06T07:27:41.113802+0000","last_unstale":"2024-03-06T07:27:43.667842+0000","last_undegraded":"2024-03-06T07:27:43.667842+0000","last_fullsized":"2024-03-06T07:27:43.667842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:43.723690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.434174+0000","last_change":"2024-03-06T07:27:42.110210+0000","last_active":"2024-03-06T07:27:43.434174+0000","last_peered":"2024-03-06T07:27:43.434174+0000","last_clean":"2024-03-06T07:27:43.434174+0000","last_became_active":"2024-03-06T07:27:42.109736+0000","last_became_peered":"2024-03-06T07:27:42.109736+0000","last_unstale":"2024-03-06T07:27:43.434174+0000","last_undegraded":"2024-03-06T07:27:43.434174+0000","last_fullsized":"2024-03-06T07:27:43.434174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:25.704825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090026+0000","last_change":"2024-03-06T07:27:41.101111+0000","last_active":"2024-03-06T07:27:43.090026+0000","last_peered":"2024-03-06T07:27:43.090026+0000","last_clean":"2024-03-06T07:27:43.090026+0000","last_became_active":"2024-03-06T07:27:41.100857+0000","last_became_peered":"2024-03-06T07:27:41.100857+0000","last_unstale":"2024-03-06T07:27:43.090026+0000","last_undegraded":"2024-03-06T07:27:43.090026+0000","last_fullsized":"2024-03-06T07:27:43.090026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:24.859411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264697+0000","last_change":"2024-03-06T07:27:42.086202+0000","last_active":"2024-03-06T07:27:43.264697+0000","last_peered":"2024-03-06T07:27:43.264697+0000","last_clean":"2024-03-06T07:27:43.264697+0000","last_became_active":"2024-03-06T07:27:42.085754+0000","last_became_peered":"2024-03-06T07:27:42.085754+0000","last_unstale":"2024-03-06T07:27:43.264697+0000","last_undegraded":"2024-03-06T07:27:43.264697+0000","last_fullsized":"2024-03-06T07:27:43.264697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:15.355541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.090044+0000","last_change":"2024-03-06T07:27:41.096584+0000","last_active":"2024-03-06T07:27:43.090044+0000","last_peered":"2024-03-06T07:27:43.090044+0000","last_clean":"2024-03-06T07:27:43.090044+0000","last_became_active":"2024-03-06T07:27:41.096318+0000","last_became_peered":"2024-03-06T07:27:41.096318+0000","last_unstale":"2024-03-06T07:27:43.090044+0000","last_undegraded":"2024-03-06T07:27:43.090044+0000","last_fullsized":"2024-03-06T07:27:43.090044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:54.566527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265099+0000","last_change":"2024-03-06T07:27:42.102269+0000","last_active":"2024-03-06T07:27:43.265099+0000","last_peered":"2024-03-06T07:27:43.265099+0000","last_clean":"2024-03-06T07:27:43.265099+0000","last_became_active":"2024-03-06T07:27:42.092816+0000","last_became_peered":"2024-03-06T07:27:42.092816+0000","last_unstale":"2024-03-06T07:27:43.265099+0000","last_undegraded":"2024-03-06T07:27:43.265099+0000","last_fullsized":"2024-03-06T07:27:43.265099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:27.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668665+0000","last_change":"2024-03-06T07:27:41.116372+0000","last_active":"2024-03-06T07:27:43.668665+0000","last_peered":"2024-03-06T07:27:43.668665+0000","last_clean":"2024-03-06T07:27:43.668665+0000","last_became_active":"2024-03-06T07:27:41.116072+0000","last_became_peered":"2024-03-06T07:27:41.116072+0000","last_unstale":"2024-03-06T07:27:43.668665+0000","last_undegraded":"2024-03-06T07:27:43.668665+0000","last_fullsized":"2024-03-06T07:27:43.668665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:08.662845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.095884+0000","last_change":"2024-03-06T07:27:42.103012+0000","last_active":"2024-03-06T07:27:43.095884+0000","last_peered":"2024-03-06T07:27:43.095884+0000","last_clean":"2024-03-06T07:27:43.095884+0000","last_became_active":"2024-03-06T07:27:42.102731+0000","last_became_peered":"2024-03-06T07:27:42.102731+0000","last_unstale":"2024-03-06T07:27:43.095884+0000","last_undegraded":"2024-03-06T07:27:43.095884+0000","last_fullsized":"2024-03-06T07:27:43.095884+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:58.394711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604917+0000","last_change":"2024-03-06T07:27:42.097462+0000","last_active":"2024-03-06T07:27:43.604917+0000","last_peered":"2024-03-06T07:27:43.604917+0000","last_clean":"2024-03-06T07:27:43.604917+0000","last_became_active":"2024-03-06T07:27:42.097142+0000","last_became_peered":"2024-03-06T07:27:42.097142+0000","last_unstale":"2024-03-06T07:27:43.604917+0000","last_undegraded":"2024-03-06T07:27:43.604917+0000","last_fullsized":"2024-03-06T07:27:43.604917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:45.596466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.668754+0000","last_change":"2024-03-06T07:27:41.115663+0000","last_active":"2024-03-06T07:27:43.668754+0000","last_peered":"2024-03-06T07:27:43.668754+0000","last_clean":"2024-03-06T07:27:43.668754+0000","last_became_active":"2024-03-06T07:27:41.114756+0000","last_became_peered":"2024-03-06T07:27:41.114756+0000","last_unstale":"2024-03-06T07:27:43.668754+0000","last_undegraded":"2024-03-06T07:27:43.668754+0000","last_fullsized":"2024-03-06T07:27:43.668754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:04.529296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.604966+0000","last_change":"2024-03-06T07:27:42.098684+0000","last_active":"2024-03-06T07:27:43.604966+0000","last_peered":"2024-03-06T07:27:43.604966+0000","last_clean":"2024-03-06T07:27:43.604966+0000","last_became_active":"2024-03-06T07:27:42.098307+0000","last_became_peered":"2024-03-06T07:27:42.098307+0000","last_unstale":"2024-03-06T07:27:43.604966+0000","last_undegraded":"2024-03-06T07:27:43.604966+0000","last_fullsized":"2024-03-06T07:27:43.604966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:47.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.669076+0000","last_change":"2024-03-06T07:27:41.117369+0000","last_active":"2024-03-06T07:27:43.669076+0000","last_peered":"2024-03-06T07:27:43.669076+0000","last_clean":"2024-03-06T07:27:43.669076+0000","last_became_active":"2024-03-06T07:27:41.117042+0000","last_became_peered":"2024-03-06T07:27:41.117042+0000","last_unstale":"2024-03-06T07:27:43.669076+0000","last_undegraded":"2024-03-06T07:27:43.669076+0000","last_fullsized":"2024-03-06T07:27:43.669076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:36.141122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561654+0000","last_change":"2024-03-06T07:27:42.106505+0000","last_active":"2024-03-06T07:27:43.561654+0000","last_peered":"2024-03-06T07:27:43.561654+0000","last_clean":"2024-03-06T07:27:43.561654+0000","last_became_active":"2024-03-06T07:27:42.105993+0000","last_became_peered":"2024-03-06T07:27:42.105993+0000","last_unstale":"2024-03-06T07:27:43.561654+0000","last_undegraded":"2024-03-06T07:27:43.561654+0000","last_fullsized":"2024-03-06T07:27:43.561654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.875069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.265054+0000","last_change":"2024-03-06T07:27:41.097028+0000","last_active":"2024-03-06T07:27:43.265054+0000","last_peered":"2024-03-06T07:27:43.265054+0000","last_clean":"2024-03-06T07:27:43.265054+0000","last_became_active":"2024-03-06T07:27:41.096532+0000","last_became_peered":"2024-03-06T07:27:41.096532+0000","last_unstale":"2024-03-06T07:27:43.265054+0000","last_undegraded":"2024-03-06T07:27:43.265054+0000","last_fullsized":"2024-03-06T07:27:43.265054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:37.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.264945+0000","last_change":"2024-03-06T07:27:42.094785+0000","last_active":"2024-03-06T07:27:43.264945+0000","last_peered":"2024-03-06T07:27:43.264945+0000","last_clean":"2024-03-06T07:27:43.264945+0000","last_became_active":"2024-03-06T07:27:42.091901+0000","last_became_peered":"2024-03-06T07:27:42.091901+0000","last_unstale":"2024-03-06T07:27:43.264945+0000","last_undegraded":"2024-03-06T07:27:43.264945+0000","last_fullsized":"2024-03-06T07:27:43.264945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:41.051342+0000","last_clean_scrub_stamp":"2024-03-06T07:27:41.051342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.183680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:43.561802+0000","last_change":"2024-03-06T07:27:41.102186+0000","last_active":"2024-03-06T07:27:43.561802+0000","last_peered":"2024-03-06T07:27:43.561802+0000","last_clean":"2024-03-06T07:27:43.561802+0000","last_became_active":"2024-03-06T07:27:41.101868+0000","last_became_peered":"2024-03-06T07:27:41.101868+0000","last_unstale":"2024-03-06T07:27:43.561802+0000","last_undegraded":"2024-03-06T07:27:43.561802+0000","last_fullsized":"2024-03-06T07:27:43.561802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:39.899525+0000","last_clean_scrub_stamp":"2024-03-06T07:27:39.899525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:20.791281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:52.439 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:27:52.439 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:27:52.439 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:27:52.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:27:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.673+0000 7ffa86a00640 1 Processor -- start 2024-03-06T07:27:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.674+0000 7ffa86a00640 1 -- start start 2024-03-06T07:27:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.675+0000 7ffa86a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8006ff60 0x7ffa800ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.675+0000 7ffa86a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa800d0e20 con 0x7ffa800d1870 2024-03-06T07:27:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.675+0000 7ffa86a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa800d1100 con 0x7ffa8015e390 2024-03-06T07:27:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.675+0000 7ffa86a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa800cc7e0 con 0x7ffa8006ff60 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa84a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ffa800d1870 0x7ffa800cd1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45200/0 (socket says 172.21.15.97:45200) 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa84a00640 1 -- 172.21.15.97:0/2078079499 learned_addr learned my addr 172.21.15.97:0/2078079499 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa77e00640 1 --2- 172.21.15.97:0/2078079499 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8006ff60 0x7ffa800ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789998080 0 0) 0x7ffa800d0e20 con 0x7ffa800d1870 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa64003520 con 0x7ffa800d1870 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510486264 0 0) 0x7ffa800d1100 con 0x7ffa8015e390 2024-03-06T07:27:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.676+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa800d0e20 con 0x7ffa8015e390 2024-03-06T07:27:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.677+0000 7ffa77e00640 1 -- 172.21.15.97:0/2078079499 >> v1:172.21.15.135:6789/0 conn(0x7ffa8015e390 legacy=0x7ffa8006f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.677+0000 7ffa77e00640 1 -- 172.21.15.97:0/2078079499 >> v1:172.21.15.97:6789/0 conn(0x7ffa800d1870 legacy=0x7ffa800cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.677+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3521722854 0 0) 0x7ffa64003520 con 0x7ffa800d1870 2024-03-06T07:27:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.677+0000 7ffa77e00640 1 -- 172.21.15.97:0/2078079499 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800cc470 con 0x7ffa8006ff60 2024-03-06T07:27:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.678+0000 7ffa77e00640 1 --2- 172.21.15.97:0/2078079499 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8006ff60 0x7ffa800ccc70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffa680097d0 tx=0x7ffa6802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8f9d70fb147d37f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.678+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa68037070 con 0x7ffa8006ff60 2024-03-06T07:27:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.678+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa68030510 con 0x7ffa8006ff60 2024-03-06T07:27:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.678+0000 7ffa7f400640 1 -- 172.21.15.97:0/2078079499 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa780019f0 con 0x7ffa8006ff60 2024-03-06T07:27:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 -- 172.21.15.97:0/2078079499 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8006ff60 msgr2=0x7ffa800ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 --2- 172.21.15.97:0/2078079499 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8006ff60 0x7ffa800ccc70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffa680097d0 tx=0x7ffa6802b800 comp rx=0 tx=0).stop 2024-03-06T07:27:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 -- 172.21.15.97:0/2078079499 shutdown_connections 2024-03-06T07:27:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 --2- 172.21.15.97:0/2078079499 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8006ff60 0x7ffa800ccc70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 -- 172.21.15.97:0/2078079499 >> 172.21.15.97:0/2078079499 conn(0x7ffa800b3870 msgr2=0x7ffa800b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 -- 172.21.15.97:0/2078079499 shutdown_connections 2024-03-06T07:27:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.679+0000 7ffa86a00640 1 -- 172.21.15.97:0/2078079499 wait complete. 2024-03-06T07:27:52.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.681+0000 7ffa86a00640 1 Processor -- start 2024-03-06T07:27:52.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.681+0000 7ffa86a00640 1 -- start start 2024-03-06T07:27:52.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa86a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 0x7ffa801f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa86a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa800d1870 0x7ffa800c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa86a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8015e390 0x7ffa8015b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa86a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8006f1d0 con 0x7ffa8006ff60 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa86a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8006f050 con 0x7ffa800d1870 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa86a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8006f350 con 0x7ffa8015e390 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa7fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 0x7ffa801f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa7fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 0x7ffa801f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50722/0 (socket says 172.21.15.97:50722) 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa7fe00640 1 -- 172.21.15.97:0/1836434175 learned_addr learned my addr 172.21.15.97:0/1836434175 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa84a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8015e390 0x7ffa8015b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.682+0000 7ffa77e00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa800d1870 0x7ffa800c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7fe00640 1 -- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8015e390 msgr2=0x7ffa8015b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7fe00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8015e390 0x7ffa8015b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7fe00640 1 -- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa800d1870 msgr2=0x7ffa800c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7fe00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa800d1870 0x7ffa800c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7fe00640 1 -- 172.21.15.97:0/1836434175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800c6100 con 0x7ffa8006ff60 2024-03-06T07:27:52.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa77e00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa800d1870 0x7ffa800c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7fe00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 0x7ffa801f2f60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000c4c0 tx=0x7ffa7000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa700052d0 con 0x7ffa8006ff60 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa70005470 con 0x7ffa8006ff60 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70012870 con 0x7ffa8006ff60 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa800c63e0 con 0x7ffa8006ff60 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.683+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa800c8d70 con 0x7ffa8006ff60 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.685+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8004f330 con 0x7ffa8006ff60 2024-03-06T07:27:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.685+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa70012a10 con 0x7ffa8006ff60 2024-03-06T07:27:52.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.686+0000 7ffa7cc00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa5c06b4d0 0x7ffa5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.686+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa70089f50 con 0x7ffa8006ff60 2024-03-06T07:27:52.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.686+0000 7ffa77e00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa5c06b4d0 0x7ffa5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.687+0000 7ffa77e00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa5c06b4d0 0x7ffa5c06d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa68034000 tx=0x7ffa68008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.694+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa70058d30 con 0x7ffa8006ff60 2024-03-06T07:27:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.932+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffa801f34a0 con 0x7ffa8006ff60 2024-03-06T07:27:52.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.933+0000 7ffa7cc00640 1 -- 172.21.15.97:0/1836434175 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ffa70058630 con 0x7ffa8006ff60 2024-03-06T07:27:52.935 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:52.935 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.936+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa5c06b4d0 msgr2=0x7ffa5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.936+0000 7ffa86a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa5c06b4d0 0x7ffa5c06d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa68034000 tx=0x7ffa68008030 comp rx=0 tx=0).stop 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.936+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 msgr2=0x7ffa801f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.936+0000 7ffa86a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 0x7ffa801f2f60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000c4c0 tx=0x7ffa7000c980 comp rx=0 tx=0).stop 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 shutdown_connections 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa5c06b4d0 0x7ffa5c06d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa8015e390 0x7ffa8015b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa800d1870 0x7ffa800c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 --2- 172.21.15.97:0/1836434175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa8006ff60 0x7ffa801f2f60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 >> 172.21.15.97:0/1836434175 conn(0x7ffa800b3870 msgr2=0x7ffa800b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 shutdown_connections 2024-03-06T07:27:52.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:52.937+0000 7ffa86a00640 1 -- 172.21.15.97:0/1836434175 wait complete. 2024-03-06T07:27:52.958 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:27:52.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:53.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.159+0000 7effd3000640 1 Processor -- start 2024-03-06T07:27:53.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.159+0000 7effd3000640 1 -- start start 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effd3000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc06db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effd3000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7effcc15e1d0 0x7effcc06b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effd3000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc06b8e0 con 0x7effcc0d1630 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effd3000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effcc06ed70 con 0x7effcc15e1d0 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effd3000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc06e160 con 0x7effcc06afc0 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7effcc15e1d0 0x7effcc06b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7effcc15e1d0 0x7effcc06b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50736/0 (socket says 172.21.15.97:50736) 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcb400640 1 -- 172.21.15.97:0/21855960 learned_addr learned my addr 172.21.15.97:0/21855960 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcb400640 1 -- 172.21.15.97:0/21855960 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 msgr2=0x7effcc06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcbe00640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcb400640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc06db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.161+0000 7effcb400640 1 -- 172.21.15.97:0/21855960 >> v1:172.21.15.97:6789/0 conn(0x7effcc0d1630 legacy=0x7effcc0adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.162+0000 7effcb400640 1 -- 172.21.15.97:0/21855960 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc06e440 con 0x7effcc15e1d0 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.162+0000 7effcaa00640 1 -- 172.21.15.97:0/21855960 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704717010 0 0) 0x7effcc06b8e0 con 0x7effcc0d1630 2024-03-06T07:27:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.162+0000 7effcbe00640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc06db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.163+0000 7effcb400640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc15e1d0 0x7effcc06b3a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7effb8009870 tx=0x7effb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a4bcf4ae516cae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.163+0000 7effcaa00640 1 -- 172.21.15.97:0/21855960 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8034070 con 0x7effcc15e1d0 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.163+0000 7effcaa00640 1 -- 172.21.15.97:0/21855960 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb8030840 con 0x7effcc15e1d0 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.163+0000 7effcaa00640 1 -- 172.21.15.97:0/21855960 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8030c10 con 0x7effcc15e1d0 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 -- 172.21.15.97:0/21855960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc15e1d0 msgr2=0x7effcc06b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc15e1d0 0x7effcc06b3a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7effb8009870 tx=0x7effb802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 -- 172.21.15.97:0/21855960 shutdown_connections 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc15e1d0 0x7effcc06b3a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 --2- 172.21.15.97:0/21855960 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc06db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 -- 172.21.15.97:0/21855960 >> 172.21.15.97:0/21855960 conn(0x7effcc0d0b40 msgr2=0x7effcc0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 -- 172.21.15.97:0/21855960 shutdown_connections 2024-03-06T07:27:53.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.164+0000 7effd3000640 1 -- 172.21.15.97:0/21855960 wait complete. 2024-03-06T07:27:53.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.166+0000 7effd3000640 1 Processor -- start 2024-03-06T07:27:53.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.167+0000 7effd3000640 1 -- start start 2024-03-06T07:27:53.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effd3000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc1e6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effd3000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 0x7effcc0af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effd3000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effcc15e1d0 0x7effcc0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effd3000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0704b0 con 0x7effcc0d1630 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effd3000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc06ea00 con 0x7effcc15e1d0 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effd3000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0ae350 con 0x7effcc06afc0 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effcb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 0x7effcc0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effcb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 0x7effcc0af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50748/0 (socket says 172.21.15.97:50748) 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.168+0000 7effcb400640 1 -- 172.21.15.97:0/4026129350 learned_addr learned my addr 172.21.15.97:0/4026129350 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcbe00640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc1e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effd1000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effcc15e1d0 0x7effcc0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcb400640 1 -- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 msgr2=0x7effcc1e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcb400640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc1e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcb400640 1 -- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effcc15e1d0 msgr2=0x7effcc0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcb400640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effcc15e1d0 0x7effcc0cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcb400640 1 -- 172.21.15.97:0/4026129350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc1e7130 con 0x7effcc0d1630 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcbe00640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc1e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effcb400640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 0x7effcc0af820 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7effb80090b0 tx=0x7effb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8034040 con 0x7effcc0d1630 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb803ad90 con 0x7effcc0d1630 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8004270 con 0x7effcc0d1630 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc1e7410 con 0x7effcc0d1630 2024-03-06T07:27:53.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.169+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc1e6cc0 con 0x7effcc0d1630 2024-03-06T07:27:53.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.170+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effcc04f330 con 0x7effcc0d1630 2024-03-06T07:27:53.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.171+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effb8037070 con 0x7effcc0d1630 2024-03-06T07:27:53.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.172+0000 7effc3e00640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7effac06b4d0 0x7effac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.172+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effb80ac360 con 0x7effcc0d1630 2024-03-06T07:27:53.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.172+0000 7effcbe00640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7effac06b4d0 0x7effac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.173+0000 7effcbe00640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7effac06b4d0 0x7effac06d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7effbc006fd0 tx=0x7effbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.177+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effb803f070 con 0x7effcc0d1630 2024-03-06T07:27:53.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.375+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effcc072db0 con 0x7effcc0d1630 2024-03-06T07:27:53.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.377+0000 7effc3e00640 1 -- 172.21.15.97:0/4026129350 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7effb807b300 con 0x7effcc0d1630 2024-03-06T07:27:53.380 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:53.380 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7effac06b4d0 msgr2=0x7effac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7effac06b4d0 0x7effac06d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7effbc006fd0 tx=0x7effbc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 msgr2=0x7effcc0af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 0x7effcc0af820 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7effb80090b0 tx=0x7effb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 shutdown_connections 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7effac06b4d0 0x7effac06d990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effcc15e1d0 0x7effcc0cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effcc0d1630 0x7effcc0af820 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 --2- 172.21.15.97:0/4026129350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effcc06afc0 0x7effcc1e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 >> 172.21.15.97:0/4026129350 conn(0x7effcc0d0b40 msgr2=0x7effcc0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 shutdown_connections 2024-03-06T07:27:53.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.380+0000 7effd3000640 1 -- 172.21.15.97:0/4026129350 wait complete. 2024-03-06T07:27:53.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:53.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.596+0000 7f6751e00640 1 Processor -- start 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.596+0000 7f6751e00640 1 -- start start 2024-03-06T07:27:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.597+0000 7f6751e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f674c0b3f70 0x7f674c160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.597+0000 7f6751e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c070ff0 0x7f674c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f6751e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c0b46c0 con 0x7f674c15fc10 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f6751e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674c071480 con 0x7f674c0b3f70 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f6751e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c0b49a0 con 0x7f674c070ff0 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f674c0b3f70 0x7f674c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f674c0b3f70 0x7f674c160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50752/0 (socket says 172.21.15.97:50752) 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 -- 172.21.15.97:0/4049322698 learned_addr learned my addr 172.21.15.97:0/4049322698 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674aa00640 1 --1- 172.21.15.97:0/4049322698 >> v1:172.21.15.97:6789/0 conn(0x7f674c15fc10 0x7f674c15fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45216/0 (socket says 172.21.15.97:45216) 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674b400640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c070ff0 0x7f674c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 -- 172.21.15.97:0/4049322698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c070ff0 msgr2=0x7f674c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c070ff0 0x7f674c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 -- 172.21.15.97:0/4049322698 >> v1:172.21.15.97:6789/0 conn(0x7f674c15fc10 legacy=0x7f674c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674a000640 1 -- 172.21.15.97:0/4049322698 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c072c90 con 0x7f674c0b3f70 2024-03-06T07:27:53.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f6749600640 1 -- 172.21.15.97:0/4049322698 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484509996 0 0) 0x7f674c0b46c0 con 0x7f674c15fc10 2024-03-06T07:27:53.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.598+0000 7f674b400640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c070ff0 0x7f674c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f674a000640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c0b3f70 0x7f674c160700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6738009870 tx=0x7f673802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cb7aff36c4b294 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6749600640 1 -- 172.21.15.97:0/4049322698 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738034070 con 0x7f674c0b3f70 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6749600640 1 -- 172.21.15.97:0/4049322698 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738030840 con 0x7f674c0b3f70 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6749600640 1 -- 172.21.15.97:0/4049322698 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738030c10 con 0x7f674c0b3f70 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6751e00640 1 -- 172.21.15.97:0/4049322698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c0b3f70 msgr2=0x7f674c160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6751e00640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c0b3f70 0x7f674c160700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6738009870 tx=0x7f673802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6751e00640 1 -- 172.21.15.97:0/4049322698 shutdown_connections 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6751e00640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c070ff0 0x7f674c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6751e00640 1 --2- 172.21.15.97:0/4049322698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c0b3f70 0x7f674c160700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.600+0000 7f6751e00640 1 -- 172.21.15.97:0/4049322698 >> 172.21.15.97:0/4049322698 conn(0x7f674c0b1940 msgr2=0x7f674c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.601+0000 7f6751e00640 1 -- 172.21.15.97:0/4049322698 shutdown_connections 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.601+0000 7f6751e00640 1 -- 172.21.15.97:0/4049322698 wait complete. 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.602+0000 7f6751e00640 1 Processor -- start 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.602+0000 7f6751e00640 1 -- start start 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f6751e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 0x7f674c13ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f6751e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c0b3f70 0x7f674c158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f6751e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f674c15fc10 0x7f674c158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f6751e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c072a30 con 0x7f674c070ff0 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f6751e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c0728b0 con 0x7f674c15fc10 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f6751e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c15c2e0 con 0x7f674c0b3f70 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f674aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 0x7f674c13ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f674aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 0x7f674c13ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50758/0 (socket says 172.21.15.97:50758) 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.604+0000 7f674aa00640 1 -- 172.21.15.97:0/3446097021 learned_addr learned my addr 172.21.15.97:0/3446097021 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674a000640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c0b3f70 0x7f674c158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674b400640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f674c15fc10 0x7f674c158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674aa00640 1 -- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c0b3f70 msgr2=0x7f674c158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674aa00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c0b3f70 0x7f674c158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674aa00640 1 -- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f674c15fc10 msgr2=0x7f674c158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674aa00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f674c15fc10 0x7f674c158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674aa00640 1 -- 172.21.15.97:0/3446097021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c15f410 con 0x7f674c070ff0 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f674aa00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 0x7f674c13ddd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f673c00c430 tx=0x7f673c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c0074e0 con 0x7f674c070ff0 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673c0047e0 con 0x7f674c070ff0 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c004bd0 con 0x7f674c070ff0 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.605+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c13e5a0 con 0x7f674c070ff0 2024-03-06T07:27:53.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.606+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c13eb60 con 0x7f674c070ff0 2024-03-06T07:27:53.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.607+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c04f330 con 0x7f674c070ff0 2024-03-06T07:27:53.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.608+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f673c011070 con 0x7f674c070ff0 2024-03-06T07:27:53.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.608+0000 7f6743400640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f672c06b4d0 0x7f672c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.609+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f673c089ec0 con 0x7f674c070ff0 2024-03-06T07:27:53.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.609+0000 7f674a000640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f672c06b4d0 0x7f672c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.609+0000 7f674a000640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f672c06b4d0 0x7f672c06d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6738036000 tx=0x7f673803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.613+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f673c00e280 con 0x7f674c070ff0 2024-03-06T07:27:53.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.809+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f674c06ef30 con 0x7f674c070ff0 2024-03-06T07:27:53.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.810+0000 7f6743400640 1 -- 172.21.15.97:0/3446097021 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f673c058cb0 con 0x7f674c070ff0 2024-03-06T07:27:53.812 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.812+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f672c06b4d0 msgr2=0x7f672c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.812+0000 7f6751e00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f672c06b4d0 0x7f672c06d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6738036000 tx=0x7f673803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.812+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 msgr2=0x7f674c13ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.812+0000 7f6751e00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 0x7f674c13ddd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f673c00c430 tx=0x7f673c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 shutdown_connections 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f672c06b4d0 0x7f672c06d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f674c15fc10 0x7f674c158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f674c0b3f70 0x7f674c158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 --2- 172.21.15.97:0/3446097021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f674c070ff0 0x7f674c13ddd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 >> 172.21.15.97:0/3446097021 conn(0x7f674c0b1940 msgr2=0x7f674c0ace90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 shutdown_connections 2024-03-06T07:27:53.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:53.813+0000 7f6751e00640 1 -- 172.21.15.97:0/3446097021 wait complete. 2024-03-06T07:27:53.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.014+0000 7fa2b2e00640 1 Processor -- start 2024-03-06T07:27:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.014+0000 7fa2b2e00640 1 -- start start 2024-03-06T07:27:54.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2b2e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa2ac15fc10 0x7fa2ac070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2b2e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0b46c0 con 0x7fa2ac15fff0 2024-03-06T07:27:54.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2b2e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0b49a0 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2b2e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ac072c90 con 0x7fa2ac15fc10 2024-03-06T07:27:54.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2abe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa2ac15fc10 0x7fa2ac070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2abe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa2ac15fc10 0x7fa2ac070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59450/0 (socket says 172.21.15.97:59450) 2024-03-06T07:27:54.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.016+0000 7fa2abe00640 1 -- 172.21.15.97:0/3703941622 learned_addr learned my addr 172.21.15.97:0/3703941622 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:54.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.017+0000 7fa2abe00640 1 -- 172.21.15.97:0/3703941622 >> v1:172.21.15.135:6789/0 conn(0x7fa2ac0b3f70 legacy=0x7fa2ac0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.017+0000 7fa2abe00640 1 -- 172.21.15.97:0/3703941622 >> v1:172.21.15.97:6789/0 conn(0x7fa2ac15fff0 legacy=0x7fa2ac160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.017+0000 7fa2abe00640 1 -- 172.21.15.97:0/3703941622 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac09dd80 con 0x7fa2ac15fc10 2024-03-06T07:27:54.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2abe00640 1 --2- 172.21.15.97:0/3703941622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac15fc10 0x7fa2ac070ff0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa294009870 tx=0x7fa29402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8cbde463dea9d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2aaa00640 1 -- 172.21.15.97:0/3703941622 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa294034070 con 0x7fa2ac15fc10 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2aaa00640 1 -- 172.21.15.97:0/3703941622 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294030860 con 0x7fa2ac15fc10 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2aaa00640 1 -- 172.21.15.97:0/3703941622 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa294030c30 con 0x7fa2ac15fc10 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2b2e00640 1 -- 172.21.15.97:0/3703941622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac15fc10 msgr2=0x7fa2ac070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/3703941622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac15fc10 0x7fa2ac070ff0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa294009870 tx=0x7fa29402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2b2e00640 1 -- 172.21.15.97:0/3703941622 shutdown_connections 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/3703941622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac15fc10 0x7fa2ac070ff0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.018+0000 7fa2b2e00640 1 -- 172.21.15.97:0/3703941622 >> 172.21.15.97:0/3703941622 conn(0x7fa2ac0b1940 msgr2=0x7fa2ac0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.019+0000 7fa2b2e00640 1 -- 172.21.15.97:0/3703941622 shutdown_connections 2024-03-06T07:27:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.019+0000 7fa2b2e00640 1 -- 172.21.15.97:0/3703941622 wait complete. 2024-03-06T07:27:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.020+0000 7fa2b2e00640 1 Processor -- start 2024-03-06T07:27:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.020+0000 7fa2b2e00640 1 -- start start 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2b2e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 0x7fa2ac15f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2b2e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2ac15fc10 0x7fa2ac1f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2b2e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2ac15fff0 0x7fa2ac1444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2b2e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac15c460 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2b2e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac15c2e0 con 0x7fa2ac15fc10 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2b2e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac0cebb0 con 0x7fa2ac15fff0 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 0x7fa2ac15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 0x7fa2ac15f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59458/0 (socket says 172.21.15.97:59458) 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.021+0000 7fa2abe00640 1 -- 172.21.15.97:0/2944727147 learned_addr learned my addr 172.21.15.97:0/2944727147 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2abe00640 1 -- 172.21.15.97:0/2944727147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2ac15fff0 msgr2=0x7fa2ac1444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2abe00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2ac15fff0 0x7fa2ac1444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2abe00640 1 -- 172.21.15.97:0/2944727147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2ac15fc10 msgr2=0x7fa2ac1f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2abe00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2ac15fc10 0x7fa2ac1f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2abe00640 1 -- 172.21.15.97:0/2944727147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac1f4900 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2abe00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 0x7fa2ac15f100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa294009870 tx=0x7fa294004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa294048070 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294039d90 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac1449f0 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa294034040 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.022+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac1f4e50 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.024+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac04f330 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.024+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa29404b070 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.025+0000 7fa2a3e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa28806b4d0 0x7fa28806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.025+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2940ac3a0 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.025+0000 7fa2ab400640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa28806b4d0 0x7fa28806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.026+0000 7fa2ab400640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa28806b4d0 0x7fa28806d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0012e0 tx=0x7fa29c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.029+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2ac1449f0 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.322+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa2ac0cc410 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.323+0000 7fa2a3e00640 1 -- 172.21.15.97:0/2944727147 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa29407b210 con 0x7fa2ac0b3f70 2024-03-06T07:27:54.325 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa28806b4d0 msgr2=0x7fa28806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa28806b4d0 0x7fa28806d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0012e0 tx=0x7fa29c007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 msgr2=0x7fa2ac15f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 0x7fa2ac15f100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa294009870 tx=0x7fa294004730 comp rx=0 tx=0).stop 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 shutdown_connections 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa28806b4d0 0x7fa28806d990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2ac15fff0 0x7fa2ac1444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2ac15fc10 0x7fa2ac1f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 --2- 172.21.15.97:0/2944727147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2ac0b3f70 0x7fa2ac15f100 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 >> 172.21.15.97:0/2944727147 conn(0x7fa2ac0b1940 msgr2=0x7fa2ac157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.326+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 shutdown_connections 2024-03-06T07:27:54.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.327+0000 7fa2b2e00640 1 -- 172.21.15.97:0/2944727147 wait complete. 2024-03-06T07:27:54.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.558+0000 7f2de0a00640 1 Processor -- start 2024-03-06T07:27:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.558+0000 7f2de0a00640 1 -- start start 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.559+0000 7f2de0a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2ddc0d23c0 0x7f2ddc0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.559+0000 7f2de0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 0x7f2ddc0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.559+0000 7f2de0a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc0d1e60 con 0x7f2ddc1616e0 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.559+0000 7f2de0a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc0cee60 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.559+0000 7f2de0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0cf140 con 0x7f2ddc0b31d0 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.560+0000 7f2ddb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 0x7f2ddc0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.560+0000 7f2ddb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 0x7f2ddc0b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:52332/0 (socket says 172.21.15.97:52332) 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.560+0000 7f2ddb200640 1 -- 172.21.15.97:0/206863714 learned_addr learned my addr 172.21.15.97:0/206863714 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.560+0000 7f2dd1e00640 1 --2- 172.21.15.97:0/206863714 >> v2:172.21.15.135:3300/0 conn(0x7f2ddc0d23c0 0x7f2ddc0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.561+0000 7f2dd9e00640 1 -- 172.21.15.97:0/206863714 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544851389 0 0) 0x7f2ddc0d1e60 con 0x7f2ddc1616e0 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.561+0000 7f2dd9e00640 1 -- 172.21.15.97:0/206863714 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0003520 con 0x7f2ddc1616e0 2024-03-06T07:27:54.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.561+0000 7f2ddb200640 1 -- 172.21.15.97:0/206863714 >> v2:172.21.15.135:3300/0 conn(0x7f2ddc0d23c0 msgr2=0x7f2ddc0d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.561+0000 7f2ddb200640 1 --2- 172.21.15.97:0/206863714 >> v2:172.21.15.135:3300/0 conn(0x7f2ddc0d23c0 0x7f2ddc0d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.561+0000 7f2ddb200640 1 -- 172.21.15.97:0/206863714 >> v1:172.21.15.135:6789/0 conn(0x7f2ddc1616e0 legacy=0x7f2ddc0d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.561+0000 7f2ddb200640 1 -- 172.21.15.97:0/206863714 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0cf420 con 0x7f2ddc0b31d0 2024-03-06T07:27:54.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.562+0000 7f2ddb200640 1 --2- 172.21.15.97:0/206863714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 0x7f2ddc0b3660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009e50 tx=0x7f2dd402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3b570baab25c11e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.563+0000 7f2dd9e00640 1 -- 172.21.15.97:0/206863714 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd402cb00 con 0x7f2ddc0b31d0 2024-03-06T07:27:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.563+0000 7f2dd9e00640 1 -- 172.21.15.97:0/206863714 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd402cca0 con 0x7f2ddc0b31d0 2024-03-06T07:27:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.563+0000 7f2dd9e00640 1 -- 172.21.15.97:0/206863714 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40345f0 con 0x7f2ddc0b31d0 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 -- 172.21.15.97:0/206863714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 msgr2=0x7f2ddc0b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 --2- 172.21.15.97:0/206863714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 0x7f2ddc0b3660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009e50 tx=0x7f2dd402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 -- 172.21.15.97:0/206863714 shutdown_connections 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 --2- 172.21.15.97:0/206863714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc0b31d0 0x7f2ddc0b3660 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 --2- 172.21.15.97:0/206863714 >> v2:172.21.15.135:3300/0 conn(0x7f2ddc0d23c0 0x7f2ddc0d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 -- 172.21.15.97:0/206863714 >> 172.21.15.97:0/206863714 conn(0x7f2ddc09d050 msgr2=0x7f2ddc0d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 -- 172.21.15.97:0/206863714 shutdown_connections 2024-03-06T07:27:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.564+0000 7f2de0a00640 1 -- 172.21.15.97:0/206863714 wait complete. 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.566+0000 7f2de0a00640 1 Processor -- start 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.566+0000 7f2de0a00640 1 -- start start 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.567+0000 7f2de0a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ddc0b31d0 0x7f2ddc1502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.567+0000 7f2de0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 0x7f2ddc0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2de0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc1616e0 0x7f2ddc0b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2de0a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc161ac0 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2de0a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0cf9e0 con 0x7f2ddc0b31d0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2de0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc161c40 con 0x7f2ddc1616e0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 0x7f2ddc0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 0x7f2ddc0caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59474/0 (socket says 172.21.15.97:59474) 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 -- 172.21.15.97:0/1575821949 learned_addr learned my addr 172.21.15.97:0/1575821949 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dda800640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ddc0b31d0 0x7f2ddc1502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2ddb200640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc1616e0 0x7f2ddc0b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 -- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc1616e0 msgr2=0x7f2ddc0b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc1616e0 0x7f2ddc0b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 -- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ddc0b31d0 msgr2=0x7f2ddc1502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ddc0b31d0 0x7f2ddc1502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dd1e00640 1 -- 172.21.15.97:0/1575821949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0cc410 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.568+0000 7f2dda800640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ddc0b31d0 0x7f2ddc1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.569+0000 7f2dd1e00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 0x7f2ddc0caeb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400e870 tx=0x7f2dc400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.569+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4017070 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.569+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc400c940 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.569+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc400cd30 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.569+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc0cb890 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.569+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc0cbe50 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.571+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc04f330 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.571+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dc401a070 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.573+0000 7f2dd3400640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2db806b4d0 0x7f2db806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.573+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dc408a120 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.573+0000 7f2dda800640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2db806b4d0 0x7f2db806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.574+0000 7f2dda800640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2db806b4d0 0x7f2db806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0011e0 tx=0x7f2dcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.577+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dc4058f00 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.772+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ddc0d21a0 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.773+0000 7f2dd3400640 1 -- 172.21.15.97:0/1575821949 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2dc4058800 con 0x7f2ddc0d23c0 2024-03-06T07:27:54.775 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:27:54.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2db806b4d0 msgr2=0x7f2db806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2db806b4d0 0x7f2db806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0011e0 tx=0x7f2dcc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 msgr2=0x7f2ddc0caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 0x7f2ddc0caeb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400e870 tx=0x7f2dc400ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 shutdown_connections 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2db806b4d0 0x7f2db806d990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ddc1616e0 0x7f2ddc0b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ddc0d23c0 0x7f2ddc0caeb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 --2- 172.21.15.97:0/1575821949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ddc0b31d0 0x7f2ddc1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 >> 172.21.15.97:0/1575821949 conn(0x7f2ddc09d050 msgr2=0x7f2ddc15d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 shutdown_connections 2024-03-06T07:27:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.775+0000 7f2de0a00640 1 -- 172.21.15.97:0/1575821949 wait complete. 2024-03-06T07:27:54.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:54.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.992+0000 7f4abb000640 1 Processor -- start 2024-03-06T07:27:54.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.993+0000 7f4abb000640 1 -- start start 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.994+0000 7f4abb000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4ab415e230 0x7f4ab406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.994+0000 7f4abb000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.994+0000 7f4abb000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab40cf340 con 0x7f4ab40b4320 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.994+0000 7f4abb000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab415e610 con 0x7f4ab415e230 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.994+0000 7f4abb000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40cfe60 con 0x7f4ab40ce550 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4ab415e230 0x7f4ab406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4ab415e230 0x7f4ab406edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59486/0 (socket says 172.21.15.97:59486) 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1105395643 learned_addr learned my addr 172.21.15.97:0/1105395643 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab9000640 1 --2- 172.21.15.97:0/1105395643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1105395643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 msgr2=0x7f4ab40ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 --2- 172.21.15.97:0/1105395643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1105395643 >> v1:172.21.15.97:6789/0 conn(0x7f4ab40b4320 legacy=0x7f4ab40d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.995+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1105395643 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab406f390 con 0x7f4ab415e230 2024-03-06T07:27:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.996+0000 7f4ab3e00640 1 --2- 172.21.15.97:0/1105395643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab415e230 0x7f4ab406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009870 tx=0x7f4aa402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaca801a7e19538a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.996+0000 7f4ab2a00640 1 -- 172.21.15.97:0/1105395643 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4035020 con 0x7f4ab415e230 2024-03-06T07:27:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.996+0000 7f4ab2a00640 1 -- 172.21.15.97:0/1105395643 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4031640 con 0x7f4ab415e230 2024-03-06T07:27:54.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.996+0000 7f4ab2a00640 1 -- 172.21.15.97:0/1105395643 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa402d040 con 0x7f4ab415e230 2024-03-06T07:27:54.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 -- 172.21.15.97:0/1105395643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab415e230 msgr2=0x7f4ab406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 --2- 172.21.15.97:0/1105395643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab415e230 0x7f4ab406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009870 tx=0x7f4aa402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 -- 172.21.15.97:0/1105395643 shutdown_connections 2024-03-06T07:27:55.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 --2- 172.21.15.97:0/1105395643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 --2- 172.21.15.97:0/1105395643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab415e230 0x7f4ab406edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 -- 172.21.15.97:0/1105395643 >> 172.21.15.97:0/1105395643 conn(0x7f4ab40b3800 msgr2=0x7f4ab40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 -- 172.21.15.97:0/1105395643 shutdown_connections 2024-03-06T07:27:55.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:54.997+0000 7f4abb000640 1 -- 172.21.15.97:0/1105395643 wait complete. 2024-03-06T07:27:55.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.000+0000 7f4abb000640 1 Processor -- start 2024-03-06T07:27:55.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.000+0000 7f4abb000640 1 -- start start 2024-03-06T07:27:55.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4abb000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 0x7f4ab41f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4abb000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4abb000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ab415e230 0x7f4ab40b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4abb000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40d06c0 con 0x7f4ab40b4320 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4abb000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40d03e0 con 0x7f4ab415e230 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4abb000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40cc320 con 0x7f4ab40ce550 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 0x7f4ab41f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 0x7f4ab41f7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59500/0 (socket says 172.21.15.97:59500) 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1391399957 learned_addr learned my addr 172.21.15.97:0/1391399957 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3400640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 msgr2=0x7f4ab40b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40b6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ab415e230 msgr2=0x7f4ab40b7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ab415e230 0x7f4ab40b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.002+0000 7f4ab3e00640 1 -- 172.21.15.97:0/1391399957 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab40b4f90 con 0x7f4ab40b4320 2024-03-06T07:27:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.003+0000 7f4ab3e00640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 0x7f4ab41f7930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4000c00 tx=0x7f4aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.003+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4040070 con 0x7f4ab40b4320 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.003+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa40399f0 con 0x7f4ab40b4320 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.003+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab41f48b0 con 0x7f4ab40b4320 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.003+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4035040 con 0x7f4ab40b4320 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.003+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab41f4e70 con 0x7f4ab40b4320 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.005+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab404f330 con 0x7f4ab40b4320 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.005+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4aa404b060 con 0x7f4ab40b4320 2024-03-06T07:27:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.006+0000 7f4aabe00640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a9406b4d0 0x7f4a9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.006+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4aa40abb80 con 0x7f4ab40b4320 2024-03-06T07:27:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.006+0000 7f4ab3400640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a9406b4d0 0x7f4a9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.007+0000 7f4ab3400640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a9406b4d0 0x7f4a9406d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ab41f89c0 tx=0x7f4aa0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.011+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4aa407a960 con 0x7f4ab40b4320 2024-03-06T07:27:55.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.320+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ab4152900 con 0x7f4ab40b4320 2024-03-06T07:27:55.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.321+0000 7f4aabe00640 1 -- 172.21.15.97:0/1391399957 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f4aa407a260 con 0x7f4ab40b4320 2024-03-06T07:27:55.323 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:55.324 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:27:55.324 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:27:55.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a9406b4d0 msgr2=0x7f4a9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a9406b4d0 0x7f4a9406d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ab41f89c0 tx=0x7f4aa0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 msgr2=0x7f4ab41f7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 0x7f4ab41f7930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4000c00 tx=0x7f4aa40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 shutdown_connections 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a9406b4d0 0x7f4a9406d990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ab415e230 0x7f4ab40b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ab40ce550 0x7f4ab40b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 --2- 172.21.15.97:0/1391399957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ab40b4320 0x7f4ab41f7930 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 >> 172.21.15.97:0/1391399957 conn(0x7f4ab40b3800 msgr2=0x7f4ab415d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 shutdown_connections 2024-03-06T07:27:55.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.324+0000 7f4abb000640 1 -- 172.21.15.97:0/1391399957 wait complete. 2024-03-06T07:27:55.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:55.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.539+0000 7f7483600640 1 Processor -- start 2024-03-06T07:27:55.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.539+0000 7f7483600640 1 -- start start 2024-03-06T07:27:55.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7483600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7483600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f747c15f8d0 0x7f747c15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7483600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c160a20 con 0x7f747c070c10 2024-03-06T07:27:55.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7483600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c06ef50 con 0x7f747c15f8d0 2024-03-06T07:27:55.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7483600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c06f230 con 0x7f747c15f4f0 2024-03-06T07:27:55.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7481600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f747c15f8d0 0x7f747c15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7481600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f747c15f8d0 0x7f747c15fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59516/0 (socket says 172.21.15.97:59516) 2024-03-06T07:27:55.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7481600640 1 -- 172.21.15.97:0/2476794559 learned_addr learned my addr 172.21.15.97:0/2476794559 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:55.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7480c00640 1 --2- 172.21.15.97:0/2476794559 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.541+0000 7f7481600640 1 -- 172.21.15.97:0/2476794559 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 msgr2=0x7f747c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.542+0000 7f7481600640 1 --2- 172.21.15.97:0/2476794559 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.542+0000 7f7481600640 1 -- 172.21.15.97:0/2476794559 >> v1:172.21.15.135:6789/0 conn(0x7f747c070c10 legacy=0x7f747c0cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.542+0000 7f7481600640 1 -- 172.21.15.97:0/2476794559 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0cce10 con 0x7f747c15f8d0 2024-03-06T07:27:55.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.543+0000 7f7481600640 1 --2- 172.21.15.97:0/2476794559 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c15fd50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7474009e50 tx=0x7f747402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2286caeaf508078 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.543+0000 7f747b400640 1 -- 172.21.15.97:0/2476794559 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747402cb00 con 0x7f747c15f8d0 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.543+0000 7f747b400640 1 -- 172.21.15.97:0/2476794559 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747402cca0 con 0x7f747c15f8d0 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.543+0000 7f747b400640 1 -- 172.21.15.97:0/2476794559 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474034610 con 0x7f747c15f8d0 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 -- 172.21.15.97:0/2476794559 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 msgr2=0x7f747c15fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 --2- 172.21.15.97:0/2476794559 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c15fd50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7474009e50 tx=0x7f747402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 -- 172.21.15.97:0/2476794559 shutdown_connections 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 --2- 172.21.15.97:0/2476794559 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c15fd50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 --2- 172.21.15.97:0/2476794559 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 -- 172.21.15.97:0/2476794559 >> 172.21.15.97:0/2476794559 conn(0x7f747c0b3800 msgr2=0x7f747c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 -- 172.21.15.97:0/2476794559 shutdown_connections 2024-03-06T07:27:55.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.544+0000 7f7483600640 1 -- 172.21.15.97:0/2476794559 wait complete. 2024-03-06T07:27:55.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.545+0000 7f7483600640 1 Processor -- start 2024-03-06T07:27:55.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.546+0000 7f7483600640 1 -- start start 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.546+0000 7f7483600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.546+0000 7f7483600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.546+0000 7f7483600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f747c0c4750 0x7f747c1571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f7483600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c1606a0 con 0x7f747c15f8d0 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f7483600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c160520 con 0x7f747c0c4750 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f7483600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0cd0c0 con 0x7f747c15f4f0 2024-03-06T07:27:55.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c06dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59522/0 (socket says 172.21.15.97:59522) 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 -- 172.21.15.97:0/923461781 learned_addr learned my addr 172.21.15.97:0/923461781 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f7480c00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 -- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 msgr2=0x7f747c0c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f7481600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f747c0c4750 0x7f747c1571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0c6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 -- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f747c0c4750 msgr2=0x7f747c1571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f747c0c4750 0x7f747c1571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.547+0000 7f747be00640 1 -- 172.21.15.97:0/923461781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c06e0e0 con 0x7f747c15f8d0 2024-03-06T07:27:55.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.548+0000 7f747be00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c06dba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f746800c470 tx=0x7f746800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.548+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74680052d0 con 0x7f747c15f8d0 2024-03-06T07:27:55.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.548+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7468005490 con 0x7f747c15f8d0 2024-03-06T07:27:55.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.548+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468007260 con 0x7f747c15f8d0 2024-03-06T07:27:55.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.548+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c0b52f0 con 0x7f747c15f8d0 2024-03-06T07:27:55.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.548+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c0b58b0 con 0x7f747c15f8d0 2024-03-06T07:27:55.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.549+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c0b5a80 con 0x7f747c15f8d0 2024-03-06T07:27:55.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.550+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7468007400 con 0x7f747c15f8d0 2024-03-06T07:27:55.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.551+0000 7f7478c00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f745c06b500 0x7f745c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.551+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7468089720 con 0x7f747c15f8d0 2024-03-06T07:27:55.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.551+0000 7f7480c00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f745c06b500 0x7f745c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.552+0000 7f7480c00640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f745c06b500 0x7f745c06d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.555+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7468058500 con 0x7f747c15f8d0 2024-03-06T07:27:55.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.750+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f747c04f330 con 0x7f747c15f8d0 2024-03-06T07:27:55.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.752+0000 7f7478c00640 1 -- 172.21.15.97:0/923461781 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f7468057e00 con 0x7f747c15f8d0 2024-03-06T07:27:55.754 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:55.754 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:55.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f745c06b500 msgr2=0x7f745c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f745c06b500 0x7f745c06d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470008040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 msgr2=0x7f747c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c06dba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f746800c470 tx=0x7f746800c930 comp rx=0 tx=0).stop 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 shutdown_connections 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f745c06b500 0x7f745c06d9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f747c0c4750 0x7f747c1571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f747c15f8d0 0x7f747c06dba0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 --2- 172.21.15.97:0/923461781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f747c15f4f0 0x7f747c0c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 >> 172.21.15.97:0/923461781 conn(0x7f747c0b3800 msgr2=0x7f747c15b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 shutdown_connections 2024-03-06T07:27:55.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.754+0000 7f7483600640 1 -- 172.21.15.97:0/923461781 wait complete. 2024-03-06T07:27:55.774 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:55.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.970+0000 7fe89d400640 1 Processor -- start 2024-03-06T07:27:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.970+0000 7fe89d400640 1 -- start start 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.971+0000 7fe89d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe8980c9090 0x7fe8981690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.971+0000 7fe89d400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980cecd0 con 0x7fe898161b50 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.971+0000 7fe89d400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980cfc80 con 0x7fe8980d4a00 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.971+0000 7fe89d400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8980d13c0 con 0x7fe8980c9090 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe897200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe8980c9090 0x7fe8981690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe896800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe898161b50 0x7fe8980d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34806/0 (socket says 172.21.15.97:34806) 2024-03-06T07:27:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe896800640 1 -- 172.21.15.97:0/2891030111 learned_addr learned my addr 172.21.15.97:0/2891030111 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:55.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe897200640 1 -- 172.21.15.97:0/2891030111 >> v1:172.21.15.135:6789/0 conn(0x7fe8980d4a00 legacy=0x7fe8980d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe897200640 1 -- 172.21.15.97:0/2891030111 >> v1:172.21.15.97:6789/0 conn(0x7fe898161b50 legacy=0x7fe8980d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe897200640 1 -- 172.21.15.97:0/2891030111 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980c7b60 con 0x7fe8980c9090 2024-03-06T07:27:55.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.972+0000 7fe895e00640 1 -- 172.21.15.97:0/2891030111 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347265803 0 0) 0x7fe8980cecd0 con 0x7fe898161b50 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.973+0000 7fe897200640 1 --2- 172.21.15.97:0/2891030111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8981690f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe890009d90 tx=0x7fe89002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a618ba6aaadde2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe895e00640 1 -- 172.21.15.97:0/2891030111 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890039070 con 0x7fe8980c9090 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe895e00640 1 -- 172.21.15.97:0/2891030111 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89002cc60 con 0x7fe8980c9090 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe895e00640 1 -- 172.21.15.97:0/2891030111 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890034660 con 0x7fe8980c9090 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 -- 172.21.15.97:0/2891030111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 msgr2=0x7fe8981690f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 --2- 172.21.15.97:0/2891030111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8981690f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe890009d90 tx=0x7fe89002beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 -- 172.21.15.97:0/2891030111 shutdown_connections 2024-03-06T07:27:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 --2- 172.21.15.97:0/2891030111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8981690f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 -- 172.21.15.97:0/2891030111 >> 172.21.15.97:0/2891030111 conn(0x7fe8980d3e70 msgr2=0x7fe8980d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 -- 172.21.15.97:0/2891030111 shutdown_connections 2024-03-06T07:27:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.974+0000 7fe89d400640 1 -- 172.21.15.97:0/2891030111 wait complete. 2024-03-06T07:27:55.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.976+0000 7fe89d400640 1 Processor -- start 2024-03-06T07:27:55.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.976+0000 7fe89d400640 1 -- start start 2024-03-06T07:27:55.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe89d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8980b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe89d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8980d4a00 0x7fe89815fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe89d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe898161b50 0x7fe8980cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe89d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980cef40 con 0x7fe8980c9090 2024-03-06T07:27:55.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe89d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980b56a0 con 0x7fe898161b50 2024-03-06T07:27:55.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe89d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe898169630 con 0x7fe8980d4a00 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8980b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8980b8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59538/0 (socket says 172.21.15.97:59538) 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 -- 172.21.15.97:0/399515995 learned_addr learned my addr 172.21.15.97:0/399515995 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 -- 172.21.15.97:0/399515995 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8980d4a00 msgr2=0x7fe89815fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8980d4a00 0x7fe89815fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 -- 172.21.15.97:0/399515995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe898161b50 msgr2=0x7fe8980cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe898161b50 0x7fe8980cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.977+0000 7fe896800640 1 -- 172.21.15.97:0/399515995 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cced0 con 0x7fe8980c9090 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.978+0000 7fe896800640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8980b8c30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c500 tx=0x7fe88800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.978+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888018070 con 0x7fe8980c9090 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.978+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888007560 con 0x7fe8980c9090 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.978+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888004a20 con 0x7fe8980c9090 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.978+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8980cc950 con 0x7fe8980c9090 2024-03-06T07:27:55.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.978+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8980ccbf0 con 0x7fe8980c9090 2024-03-06T07:27:55.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.979+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89804f330 con 0x7fe8980c9090 2024-03-06T07:27:55.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.980+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe888004bc0 con 0x7fe8980c9090 2024-03-06T07:27:55.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.981+0000 7fe88ea00640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.981+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe888089480 con 0x7fe8980c9090 2024-03-06T07:27:55.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.981+0000 7fe88fe00640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.982+0000 7fe88fe00640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe87406b4d0 0x7fe87406d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe898154460 tx=0x7fe8800079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:55.988+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8880582f0 con 0x7fe8980c9090 2024-03-06T07:27:56.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.175+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe898068220 con 0x7fe8980c9090 2024-03-06T07:27:56.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.176+0000 7fe88ea00640 1 -- 172.21.15.97:0/399515995 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe888057bf0 con 0x7fe8980c9090 2024-03-06T07:27:56.178 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe87406b4d0 msgr2=0x7fe87406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe87406b4d0 0x7fe87406d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe898154460 tx=0x7fe8800079b0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 msgr2=0x7fe8980b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8980b8c30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c500 tx=0x7fe88800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 shutdown_connections 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe898161b50 0x7fe8980cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8980d4a00 0x7fe89815fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 --2- 172.21.15.97:0/399515995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8980c9090 0x7fe8980b8c30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.178+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 >> 172.21.15.97:0/399515995 conn(0x7fe8980d3e70 msgr2=0x7fe898160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.179+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 shutdown_connections 2024-03-06T07:27:56.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.179+0000 7fe89d400640 1 -- 172.21.15.97:0/399515995 wait complete. 2024-03-06T07:27:56.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:56.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.395+0000 7f0790a00640 1 Processor -- start 2024-03-06T07:27:56.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.395+0000 7f0790a00640 1 -- start start 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0790a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f078c071b10 0x7f078c15c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0790a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c145270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0790a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078c0d0030 con 0x7f078c1457b0 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0790a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c0d0310 con 0x7f078c071b10 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0790a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c0d05f0 con 0x7f078c071ef0 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0789e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f078c071b10 0x7f078c15c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f078a800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f078c1457b0 0x7f078c0b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34820/0 (socket says 172.21.15.97:34820) 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f0789e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f078c071b10 0x7f078c15c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59550/0 (socket says 172.21.15.97:59550) 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.397+0000 7f078a800640 1 -- 172.21.15.97:0/2654318806 learned_addr learned my addr 172.21.15.97:0/2654318806 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.398+0000 7f0781400640 1 --2- 172.21.15.97:0/2654318806 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.398+0000 7f0789e00640 1 -- 172.21.15.97:0/2654318806 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 msgr2=0x7f078c145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.398+0000 7f0789e00640 1 --2- 172.21.15.97:0/2654318806 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.398+0000 7f0789e00640 1 -- 172.21.15.97:0/2654318806 >> v1:172.21.15.97:6789/0 conn(0x7f078c1457b0 legacy=0x7f078c0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.398+0000 7f0789e00640 1 -- 172.21.15.97:0/2654318806 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c0b3b40 con 0x7f078c071b10 2024-03-06T07:27:56.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.399+0000 7f0789e00640 1 --2- 172.21.15.97:0/2654318806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c15c2e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f077c0097a0 tx=0x7f077c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe44deb0be07821 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.399+0000 7f0789400640 1 -- 172.21.15.97:0/2654318806 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c035020 con 0x7f078c071b10 2024-03-06T07:27:56.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.399+0000 7f0789400640 1 -- 172.21.15.97:0/2654318806 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c031810 con 0x7f078c071b10 2024-03-06T07:27:56.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.399+0000 7f0789400640 1 -- 172.21.15.97:0/2654318806 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c031be0 con 0x7f078c071b10 2024-03-06T07:27:56.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 -- 172.21.15.97:0/2654318806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 msgr2=0x7f078c15c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 --2- 172.21.15.97:0/2654318806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c15c2e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f077c0097a0 tx=0x7f077c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 -- 172.21.15.97:0/2654318806 shutdown_connections 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 --2- 172.21.15.97:0/2654318806 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 --2- 172.21.15.97:0/2654318806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c15c2e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 -- 172.21.15.97:0/2654318806 >> 172.21.15.97:0/2654318806 conn(0x7f078c0b2840 msgr2=0x7f078c0b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 -- 172.21.15.97:0/2654318806 shutdown_connections 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.400+0000 7f0790a00640 1 -- 172.21.15.97:0/2654318806 wait complete. 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.401+0000 7f0790a00640 1 Processor -- start 2024-03-06T07:27:56.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.402+0000 7f0790a00640 1 -- start start 2024-03-06T07:27:56.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0790a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c1f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0790a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c1f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0790a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f078c1457b0 0x7f078c15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0790a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c072350 con 0x7f078c071b10 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0790a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c0cfcc0 con 0x7f078c1457b0 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0790a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c144e30 con 0x7f078c071ef0 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c1f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c1f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59564/0 (socket says 172.21.15.97:59564) 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 -- 172.21.15.97:0/749793771 learned_addr learned my addr 172.21.15.97:0/749793771 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0781400640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c1f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 -- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 msgr2=0x7f078c1f4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c1f4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 -- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f078c1457b0 msgr2=0x7f078c15ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f078a800640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f078c1457b0 0x7f078c15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f078c1457b0 0x7f078c15ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0789e00640 1 -- 172.21.15.97:0/749793771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c1f5510 con 0x7f078c071b10 2024-03-06T07:27:56.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.403+0000 7f0781400640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c1f4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.404+0000 7f0789e00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c1f4fd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f077c02bec0 tx=0x7f077c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.404+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c041070 con 0x7f078c071b10 2024-03-06T07:27:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.404+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c03ad00 con 0x7f078c071b10 2024-03-06T07:27:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.404+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078c1f4dd0 con 0x7f078c071b10 2024-03-06T07:27:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.404+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c035040 con 0x7f078c071b10 2024-03-06T07:27:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.404+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078c06dae0 con 0x7f078c071b10 2024-03-06T07:27:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.405+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c04f330 con 0x7f078c071b10 2024-03-06T07:27:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.406+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f077c04c060 con 0x7f078c071b10 2024-03-06T07:27:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.408+0000 7f0782a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f076806b4d0 0x7f076806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.408+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f077c0abd90 con 0x7f078c071b10 2024-03-06T07:27:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.408+0000 7f0781400640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f076806b4d0 0x7f076806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.409+0000 7f0781400640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f076806b4d0 0x7f076806d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f078c156fd0 tx=0x7f0774008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.412+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f078c1f4dd0 con 0x7f078c071b10 2024-03-06T07:27:56.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.607+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f078c0d11a0 con 0x7f078c071b10 2024-03-06T07:27:56.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.608+0000 7f0782a00640 1 -- 172.21.15.97:0/749793771 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f077c07aca0 con 0x7f078c071b10 2024-03-06T07:27:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:27:56.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f076806b4d0 msgr2=0x7f076806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f076806b4d0 0x7f076806d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f078c156fd0 tx=0x7f0774008040 comp rx=0 tx=0).stop 2024-03-06T07:27:56.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 msgr2=0x7f078c1f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c1f4fd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f077c02bec0 tx=0x7f077c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 shutdown_connections 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f076806b4d0 0x7f076806d990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f078c1457b0 0x7f078c15ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f078c071ef0 0x7f078c1f4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 --2- 172.21.15.97:0/749793771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f078c071b10 0x7f078c1f4fd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 >> 172.21.15.97:0/749793771 conn(0x7f078c0b2840 msgr2=0x7f078c0b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 shutdown_connections 2024-03-06T07:27:56.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.610+0000 7f0790a00640 1 -- 172.21.15.97:0/749793771 wait complete. 2024-03-06T07:27:56.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:56.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.828+0000 7fc7fde00640 1 Processor -- start 2024-03-06T07:27:56.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.828+0000 7fc7fde00640 1 -- start start 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.829+0000 7fc7fde00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7f815e000 0x7fc7f806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.829+0000 7fc7fde00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7f815e3e0 0x7fc7f80d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.829+0000 7fc7fde00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f80d1260 con 0x7fc7f806f7e0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.829+0000 7fc7fde00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80d1020 con 0x7fc7f815e000 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.829+0000 7fc7fde00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f806c0d0 con 0x7fc7f815e3e0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7f815e000 0x7fc7f806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7f815e000 0x7fc7f806f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59566/0 (socket says 172.21.15.97:59566) 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 -- 172.21.15.97:0/2095555783 learned_addr learned my addr 172.21.15.97:0/2095555783 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 -- 172.21.15.97:0/2095555783 >> v2:172.21.15.135:3300/0 conn(0x7fc7f815e3e0 msgr2=0x7fc7f80d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6000640 1 --2- 172.21.15.97:0/2095555783 >> v2:172.21.15.135:3300/0 conn(0x7fc7f815e3e0 0x7fc7f80d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 --2- 172.21.15.97:0/2095555783 >> v2:172.21.15.135:3300/0 conn(0x7fc7f815e3e0 0x7fc7f80d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 -- 172.21.15.97:0/2095555783 >> v1:172.21.15.97:6789/0 conn(0x7fc7f806f7e0 legacy=0x7fc7f80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6a00640 1 -- 172.21.15.97:0/2095555783 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f806c320 con 0x7fc7f815e000 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f6000640 1 --2- 172.21.15.97:0/2095555783 >> v2:172.21.15.135:3300/0 conn(0x7fc7f815e3e0 0x7fc7f80d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.830+0000 7fc7f5600640 1 -- 172.21.15.97:0/2095555783 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840009455 0 0) 0x7fc7f80d1260 con 0x7fc7f806f7e0 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.831+0000 7fc7f6a00640 1 --2- 172.21.15.97:0/2095555783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f806f130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8067cb0 tx=0x7fc7e802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24130d2ead41433e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7f5600640 1 -- 172.21.15.97:0/2095555783 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8034070 con 0x7fc7f815e000 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7f5600640 1 -- 172.21.15.97:0/2095555783 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8030880 con 0x7fc7f815e000 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7f5600640 1 -- 172.21.15.97:0/2095555783 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8030c50 con 0x7fc7f815e000 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 -- 172.21.15.97:0/2095555783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 msgr2=0x7fc7f806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 --2- 172.21.15.97:0/2095555783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f806f130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8067cb0 tx=0x7fc7e802b9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 -- 172.21.15.97:0/2095555783 shutdown_connections 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 --2- 172.21.15.97:0/2095555783 >> v2:172.21.15.135:3300/0 conn(0x7fc7f815e3e0 0x7fc7f80d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 --2- 172.21.15.97:0/2095555783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f806f130 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 -- 172.21.15.97:0/2095555783 >> 172.21.15.97:0/2095555783 conn(0x7fc7f80b3800 msgr2=0x7fc7f80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 -- 172.21.15.97:0/2095555783 shutdown_connections 2024-03-06T07:27:56.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.832+0000 7fc7fde00640 1 -- 172.21.15.97:0/2095555783 wait complete. 2024-03-06T07:27:56.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.835+0000 7fc7fde00640 1 Processor -- start 2024-03-06T07:27:56.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.835+0000 7fc7fde00640 1 -- start start 2024-03-06T07:27:56.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7fde00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7f806f7e0 0x7fc7f8156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7fde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f80b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7fde00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7f815e3e0 0x7fc7f80b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7fde00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f806fdf0 con 0x7fc7f815e000 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7fde00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f806fc70 con 0x7fc7f806f7e0 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7fde00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f806ff70 con 0x7fc7f815e3e0 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7f6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f80b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7f6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f80b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59578/0 (socket says 172.21.15.97:59578) 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7f6000640 1 -- 172.21.15.97:0/75863966 learned_addr learned my addr 172.21.15.97:0/75863966 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7f6a00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7f806f7e0 0x7fc7f8156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.837+0000 7fc7f7400640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7f815e3e0 0x7fc7f80b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f6000640 1 -- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7f815e3e0 msgr2=0x7fc7f80b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f6000640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7f815e3e0 0x7fc7f80b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f6000640 1 -- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7f806f7e0 msgr2=0x7fc7f8156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f6000640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7f806f7e0 0x7fc7f8156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f6000640 1 -- 172.21.15.97:0/75863966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f81572b0 con 0x7fc7f815e000 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f7400640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7f815e3e0 0x7fc7f80b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7f6000640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f80b01a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c7f0 tx=0x7fc7e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4015070 con 0x7fc7f815e000 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e4007a30 con 0x7fc7f815e000 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f81f7800 con 0x7fc7f815e000 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4007e00 con 0x7fc7f815e000 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.838+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f81f7dc0 con 0x7fc7f815e000 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.840+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f804f330 con 0x7fc7f815e000 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.840+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7e4018070 con 0x7fc7f815e000 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.841+0000 7fc7ef400640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7d806b4d0 0x7fc7d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.841+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7e408a0e0 con 0x7fc7f815e000 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.841+0000 7fc7f6a00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7d806b4d0 0x7fc7d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.842+0000 7fc7f6a00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7d806b4d0 0x7fc7d806d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8036000 tx=0x7fc7e803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:56.846+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7e4058ff0 con 0x7fc7f815e000 2024-03-06T07:27:57.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.085+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7f80b7c50 con 0x7fc7f815e000 2024-03-06T07:27:57.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.086+0000 7fc7ef400640 1 -- 172.21.15.97:0/75863966 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc7e40588f0 con 0x7fc7f815e000 2024-03-06T07:27:57.088 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:27:57.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7d806b4d0 msgr2=0x7fc7d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7d806b4d0 0x7fc7d806d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8036000 tx=0x7fc7e803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 msgr2=0x7fc7f80b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f80b01a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c7f0 tx=0x7fc7e400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 shutdown_connections 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7d806b4d0 0x7fc7d806d990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7f815e3e0 0x7fc7f80b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7f815e000 0x7fc7f80b01a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 --2- 172.21.15.97:0/75863966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7f806f7e0 0x7fc7f8156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 >> 172.21.15.97:0/75863966 conn(0x7fc7f80b3800 msgr2=0x7fc7f80b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 shutdown_connections 2024-03-06T07:27:57.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.089+0000 7fc7fde00640 1 -- 172.21.15.97:0/75863966 wait complete. 2024-03-06T07:27:57.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:57.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.310+0000 7f32efe00640 1 Processor -- start 2024-03-06T07:27:57.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.310+0000 7f32efe00640 1 -- start start 2024-03-06T07:27:57.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.312+0000 7f32efe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f32f00d27a0 0x7f32f00b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.312+0000 7f32efe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f0161960 con 0x7f32f0161580 2024-03-06T07:27:57.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.312+0000 7f32efe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f00cf6f0 con 0x7f32f00d2360 2024-03-06T07:27:57.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.312+0000 7f32efe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f00cf9d0 con 0x7f32f00d27a0 2024-03-06T07:27:57.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.312+0000 7f32eea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f32f0161580 0x7f32f00d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34844/0 (socket says 172.21.15.97:34844) 2024-03-06T07:27:57.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.312+0000 7f32eea00640 1 -- 172.21.15.97:0/2455430513 learned_addr learned my addr 172.21.15.97:0/2455430513 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ef400640 1 --2- 172.21.15.97:0/2455430513 >> v2:172.21.15.135:3300/0 conn(0x7f32f00d27a0 0x7f32f00b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51426524 0 0) 0x7f32f0161960 con 0x7f32f0161580 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d4003520 con 0x7f32f0161580 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117570868 0 0) 0x7f32d4003520 con 0x7f32f0161580 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 >> v2:172.21.15.135:3300/0 conn(0x7f32f00d27a0 msgr2=0x7f32f00b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 --2- 172.21.15.97:0/2455430513 >> v2:172.21.15.135:3300/0 conn(0x7f32f00d27a0 0x7f32f00b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 >> v1:172.21.15.135:6789/0 conn(0x7f32f00d2360 legacy=0x7f32f00b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f00cfcb0 con 0x7f32f0161580 2024-03-06T07:27:57.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f32e0001670 con 0x7f32f0161580 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 >> v1:172.21.15.97:6789/0 conn(0x7f32f0161580 legacy=0x7f32f00d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4003a00 0x7f32d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d4004800 0x7f32d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d40051c0 0x7f32d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00cf9d0 con 0x7f32d4004800 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d400c0c0 con 0x7f32d4003a00 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d400c2c0 con 0x7f32d40051c0 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.313+0000 7f32ee000640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d4004800 0x7f32d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ef400640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d40051c0 0x7f32d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ee000640 1 -- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d40051c0 msgr2=0x7f32d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ee000640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d40051c0 0x7f32d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ee000640 1 -- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4003a00 msgr2=0x7f32d4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ee000640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4003a00 0x7f32d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ee000640 1 -- 172.21.15.97:0/2455430513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4003520 con 0x7f32d4004800 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ef400640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d40051c0 0x7f32d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.314+0000 7f32ef400640 1 -- 172.21.15.97:0/2455430513 reap_dead start 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.315+0000 7f32ee000640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d4004800 0x7f32d4004c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32dc001750 tx=0x7f32d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296c0f8b45314cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.315+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc001370 con 0x7f32d4004800 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.315+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc0052d0 con 0x7f32d4004800 2024-03-06T07:27:57.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.315+0000 7f32ed600640 1 -- 172.21.15.97:0/2455430513 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc003830 con 0x7f32d4004800 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 -- 172.21.15.97:0/2455430513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d4004800 msgr2=0x7f32d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d4004800 0x7f32d4004c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32dc001750 tx=0x7f32d4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 -- 172.21.15.97:0/2455430513 shutdown_connections 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 --2- 172.21.15.97:0/2455430513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d4004800 0x7f32d4004c80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 -- 172.21.15.97:0/2455430513 >> 172.21.15.97:0/2455430513 conn(0x7f32f00d3d80 msgr2=0x7f32f00d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 -- 172.21.15.97:0/2455430513 shutdown_connections 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.316+0000 7f32efe00640 1 -- 172.21.15.97:0/2455430513 wait complete. 2024-03-06T07:27:57.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.317+0000 7f32efe00640 1 Processor -- start 2024-03-06T07:27:57.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.318+0000 7f32efe00640 1 -- start start 2024-03-06T07:27:57.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32efe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4004800 0x7f32f014e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 0x7f32f01f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32efe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32f00b77c0 0x7f32f01f7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32efe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00d2bf0 con 0x7f32f00b7380 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32efe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00b39a0 con 0x7f32d4004800 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32efe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00cf3a0 con 0x7f32f00b77c0 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 0x7f32f01f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 0x7f32f01f6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59594/0 (socket says 172.21.15.97:59594) 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 -- 172.21.15.97:0/2940439396 learned_addr learned my addr 172.21.15.97:0/2940439396 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ef400640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32f00b77c0 0x7f32f01f7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 -- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32f00b77c0 msgr2=0x7f32f01f7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32f00b77c0 0x7f32f01f7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 -- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4004800 msgr2=0x7f32f014e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4004800 0x7f32f014e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.319+0000 7f32ee000640 1 -- 172.21.15.97:0/2940439396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f00b8630 con 0x7f32f00b7380 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.320+0000 7f32ee000640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 0x7f32f01f6dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f32dc005040 tx=0x7f32dc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.320+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc041020 con 0x7f32f00b7380 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.320+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc005480 con 0x7f32f00b7380 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.320+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc007a30 con 0x7f32f00b7380 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.320+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f00b7d90 con 0x7f32f00b7380 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.320+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f00b8350 con 0x7f32f00b7380 2024-03-06T07:27:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.321+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f004f330 con 0x7f32f00b7380 2024-03-06T07:27:57.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.323+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32dc044020 con 0x7f32f00b7380 2024-03-06T07:27:57.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.324+0000 7f32e7400640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32d006b500 0x7f32d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.324+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32dc0abc90 con 0x7f32f00b7380 2024-03-06T07:27:57.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.324+0000 7f32eea00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32d006b500 0x7f32d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.325+0000 7f32eea00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32d006b500 0x7f32d006d9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32e00078d0 tx=0x7f32e0001500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.329+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32dc04b030 con 0x7f32f00b7380 2024-03-06T07:27:57.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.566+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32f00d3b60 con 0x7f32f00b7380 2024-03-06T07:27:57.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.567+0000 7f32e7400640 1 -- 172.21.15.97:0/2940439396 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f32dc0303c0 con 0x7f32f00b7380 2024-03-06T07:27:57.569 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:57.569 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:27:57.569 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32d006b500 msgr2=0x7f32d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32d006b500 0x7f32d006d9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32e00078d0 tx=0x7f32e0001500 comp rx=0 tx=0).stop 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 msgr2=0x7f32f01f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 0x7f32f01f6dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f32dc005040 tx=0x7f32dc001120 comp rx=0 tx=0).stop 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 shutdown_connections 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32d006b500 0x7f32d006d9c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32f00b77c0 0x7f32f01f7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f00b7380 0x7f32f01f6dc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 --2- 172.21.15.97:0/2940439396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d4004800 0x7f32f014e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.569+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 >> 172.21.15.97:0/2940439396 conn(0x7f32f00d3d80 msgr2=0x7f32f006d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.570+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 shutdown_connections 2024-03-06T07:27:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.570+0000 7f32efe00640 1 -- 172.21.15.97:0/2940439396 wait complete. 2024-03-06T07:27:57.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:57.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.789+0000 7f891ec00640 1 Processor -- start 2024-03-06T07:27:57.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.789+0000 7f891ec00640 1 -- start start 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.790+0000 7f891ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f891815ada0 0x7f89180b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.790+0000 7f891ec00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89180cdad0 con 0x7f89180cd340 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f891ec00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891806f430 con 0x7f891806eab0 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f891ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891806ef10 con 0x7f891815ada0 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f891cc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f89180cd340 0x7f89180ca900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34852/0 (socket says 172.21.15.97:34852) 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f891cc00640 1 -- 172.21.15.97:0/1978740497 learned_addr learned my addr 172.21.15.97:0/1978740497 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:27:57.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f8917e00640 1 --2- 172.21.15.97:0/1978740497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f891815ada0 0x7f89180b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f8916a00640 1 -- 172.21.15.97:0/1978740497 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565478763 0 0) 0x7f89180cdad0 con 0x7f89180cd340 2024-03-06T07:27:57.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.791+0000 7f8916a00640 1 -- 172.21.15.97:0/1978740497 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88fc003520 con 0x7f89180cd340 2024-03-06T07:27:57.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.792+0000 7f8917e00640 1 -- 172.21.15.97:0/1978740497 >> v1:172.21.15.135:6789/0 conn(0x7f891806eab0 legacy=0x7f89180ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.792+0000 7f8917e00640 1 -- 172.21.15.97:0/1978740497 >> v1:172.21.15.97:6789/0 conn(0x7f89180cd340 legacy=0x7f89180ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.792+0000 7f8917e00640 1 -- 172.21.15.97:0/1978740497 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891806f150 con 0x7f891815ada0 2024-03-06T07:27:57.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.793+0000 7f8917e00640 1 --2- 172.21.15.97:0/1978740497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f891815ada0 0x7f89180b29a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f89080097d0 tx=0x7f890802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e98e1db13c60395 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.793+0000 7f8916a00640 1 -- 172.21.15.97:0/1978740497 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908031770 con 0x7f891815ada0 2024-03-06T07:27:57.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.793+0000 7f8916a00640 1 -- 172.21.15.97:0/1978740497 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908037070 con 0x7f891815ada0 2024-03-06T07:27:57.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.793+0000 7f8916a00640 1 -- 172.21.15.97:0/1978740497 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908031b30 con 0x7f891815ada0 2024-03-06T07:27:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 -- 172.21.15.97:0/1978740497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f891815ada0 msgr2=0x7f89180b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 --2- 172.21.15.97:0/1978740497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f891815ada0 0x7f89180b29a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f89080097d0 tx=0x7f890802b810 comp rx=0 tx=0).stop 2024-03-06T07:27:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 -- 172.21.15.97:0/1978740497 shutdown_connections 2024-03-06T07:27:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 --2- 172.21.15.97:0/1978740497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f891815ada0 0x7f89180b29a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 -- 172.21.15.97:0/1978740497 >> 172.21.15.97:0/1978740497 conn(0x7f89180b1e80 msgr2=0x7f89180b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 -- 172.21.15.97:0/1978740497 shutdown_connections 2024-03-06T07:27:57.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.794+0000 7f891ec00640 1 -- 172.21.15.97:0/1978740497 wait complete. 2024-03-06T07:27:57.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.796+0000 7f891ec00640 1 Processor -- start 2024-03-06T07:27:57.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.796+0000 7f891ec00640 1 -- start start 2024-03-06T07:27:57.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.797+0000 7f891ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 0x7f89180c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.797+0000 7f891ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89180cd340 0x7f8918157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.797+0000 7f891ec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f891815ada0 0x7f89180b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.797+0000 7f891ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891815b180 con 0x7f891806eab0 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.797+0000 7f891ec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89180b53f0 con 0x7f891815ada0 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.797+0000 7f891ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891815b300 con 0x7f89180cd340 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 0x7f89180c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 0x7f89180c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59604/0 (socket says 172.21.15.97:59604) 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 -- 172.21.15.97:0/3192895718 learned_addr learned my addr 172.21.15.97:0/3192895718 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f891cc00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f891815ada0 0x7f89180b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917400640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89180cd340 0x7f8918157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 -- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89180cd340 msgr2=0x7f8918157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89180cd340 0x7f8918157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 -- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f891815ada0 msgr2=0x7f89180b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f891815ada0 0x7f89180b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.798+0000 7f8917e00640 1 -- 172.21.15.97:0/3192895718 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89180cc400 con 0x7f891806eab0 2024-03-06T07:27:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.799+0000 7f8917e00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 0x7f89180c9cb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8908034040 tx=0x7f89080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.799+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908049070 con 0x7f891806eab0 2024-03-06T07:27:57.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.799+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908044410 con 0x7f891806eab0 2024-03-06T07:27:57.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.799+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908044820 con 0x7f891806eab0 2024-03-06T07:27:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.799+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89180ca320 con 0x7f891806eab0 2024-03-06T07:27:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.799+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89180b0860 con 0x7f891806eab0 2024-03-06T07:27:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.800+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891804f330 con 0x7f891806eab0 2024-03-06T07:27:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.801+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8908031070 con 0x7f891806eab0 2024-03-06T07:27:57.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.802+0000 7f890fe00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88f806b4d0 0x7f88f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.802+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89080ab300 con 0x7f891806eab0 2024-03-06T07:27:57.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.802+0000 7f8917400640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88f806b4d0 0x7f88f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.803+0000 7f8917400640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88f806b4d0 0x7f88f806d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89181f7500 tx=0x7f8904009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:57.806+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f890807a210 con 0x7f891806eab0 2024-03-06T07:27:58.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.001+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89180c2c80 con 0x7f891806eab0 2024-03-06T07:27:58.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.003+0000 7f890fe00640 1 -- 172.21.15.97:0/3192895718 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f8908079b10 con 0x7f891806eab0 2024-03-06T07:27:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:58.006 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88f806b4d0 msgr2=0x7f88f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88f806b4d0 0x7f88f806d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89181f7500 tx=0x7f8904009040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 msgr2=0x7f89180c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 0x7f89180c9cb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8908034040 tx=0x7f89080046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 shutdown_connections 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88f806b4d0 0x7f88f806d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f891815ada0 0x7f89180b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89180cd340 0x7f8918157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 --2- 172.21.15.97:0/3192895718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f891806eab0 0x7f89180c9cb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.007+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 >> 172.21.15.97:0/3192895718 conn(0x7f89180b1e80 msgr2=0x7f89180d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.008+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 shutdown_connections 2024-03-06T07:27:58.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.008+0000 7f891ec00640 1 -- 172.21.15.97:0/3192895718 wait complete. 2024-03-06T07:27:58.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:58.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.240+0000 7f135f400640 1 Processor -- start 2024-03-06T07:27:58.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.241+0000 7f135f400640 1 -- start start 2024-03-06T07:27:58.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.242+0000 7f135f400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1358144c00 0x7f135806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.242+0000 7f135f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f135815fe60 0x7f13580ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.242+0000 7f135f400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135806ee70 con 0x7f135815f9d0 2024-03-06T07:27:58.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.242+0000 7f135f400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135806cef0 con 0x7f135815fe60 2024-03-06T07:27:58.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.242+0000 7f135f400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135806d1d0 con 0x7f1358144c00 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f135815fe60 0x7f13580ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f135815fe60 0x7f13580ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59618/0 (socket says 172.21.15.97:59618) 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 -- 172.21.15.97:0/2998486701 learned_addr learned my addr 172.21.15.97:0/2998486701 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135ca00640 1 --2- 172.21.15.97:0/2998486701 >> v2:172.21.15.135:3300/0 conn(0x7f1358144c00 0x7f135806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 -- 172.21.15.97:0/2998486701 >> v2:172.21.15.135:3300/0 conn(0x7f1358144c00 msgr2=0x7f135806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 --2- 172.21.15.97:0/2998486701 >> v2:172.21.15.135:3300/0 conn(0x7f1358144c00 0x7f135806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 -- 172.21.15.97:0/2998486701 >> v1:172.21.15.135:6789/0 conn(0x7f135815f9d0 legacy=0x7f13580ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135d400640 1 -- 172.21.15.97:0/2998486701 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135806d4b0 con 0x7f135815fe60 2024-03-06T07:27:58.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.243+0000 7f135ca00640 1 --2- 172.21.15.97:0/2998486701 >> v2:172.21.15.135:3300/0 conn(0x7f1358144c00 0x7f135806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:58.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.245+0000 7f135d400640 1 --2- 172.21.15.97:0/2998486701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f135815fe60 0x7f13580ce7b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f13580b25f0 tx=0x7f135002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bca29d07bfa901 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.245+0000 7f1357e00640 1 -- 172.21.15.97:0/2998486701 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135002ca80 con 0x7f135815fe60 2024-03-06T07:27:58.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.245+0000 7f1357e00640 1 -- 172.21.15.97:0/2998486701 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135002cc20 con 0x7f135815fe60 2024-03-06T07:27:58.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.245+0000 7f1357e00640 1 -- 172.21.15.97:0/2998486701 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500343f0 con 0x7f135815fe60 2024-03-06T07:27:58.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 -- 172.21.15.97:0/2998486701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f135815fe60 msgr2=0x7f13580ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 --2- 172.21.15.97:0/2998486701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f135815fe60 0x7f13580ce7b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f13580b25f0 tx=0x7f135002bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 -- 172.21.15.97:0/2998486701 shutdown_connections 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 --2- 172.21.15.97:0/2998486701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f135815fe60 0x7f13580ce7b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 --2- 172.21.15.97:0/2998486701 >> v2:172.21.15.135:3300/0 conn(0x7f1358144c00 0x7f135806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 -- 172.21.15.97:0/2998486701 >> 172.21.15.97:0/2998486701 conn(0x7f13580b1730 msgr2=0x7f13580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 -- 172.21.15.97:0/2998486701 shutdown_connections 2024-03-06T07:27:58.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.246+0000 7f135f400640 1 -- 172.21.15.97:0/2998486701 wait complete. 2024-03-06T07:27:58.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.248+0000 7f135f400640 1 Processor -- start 2024-03-06T07:27:58.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.248+0000 7f135f400640 1 -- start start 2024-03-06T07:27:58.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 0x7f13580ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f135815f9d0 0x7f135815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135f400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f135815fe60 0x7f13580b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135f400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1358160730 con 0x7f1358144c00 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135f400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1358160270 con 0x7f135815fe60 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135f400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135806cb50 con 0x7f135815f9d0 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 0x7f13580ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 0x7f13580ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59634/0 (socket says 172.21.15.97:59634) 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.249+0000 7f135ca00640 1 -- 172.21.15.97:0/1014787874 learned_addr learned my addr 172.21.15.97:0/1014787874 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f134fe00640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f135815f9d0 0x7f135815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135d400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f135815fe60 0x7f13580b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135ca00640 1 -- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f135815f9d0 msgr2=0x7f135815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135ca00640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f135815f9d0 0x7f135815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135ca00640 1 -- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f135815fe60 msgr2=0x7f13580b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135ca00640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f135815fe60 0x7f13580b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135ca00640 1 -- 172.21.15.97:0/1014787874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580b49a0 con 0x7f1358144c00 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135d400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f135815fe60 0x7f13580b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135ca00640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 0x7f13580ae270 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f134800c7f0 tx=0x7f134800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348015070 con 0x7f1358144c00 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1348007a30 con 0x7f1358144c00 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135813ccb0 con 0x7f1358144c00 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.250+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348007e00 con 0x7f1358144c00 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.251+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13580b2fb0 con 0x7f1358144c00 2024-03-06T07:27:58.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.251+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135804f330 con 0x7f1358144c00 2024-03-06T07:27:58.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.253+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1348018070 con 0x7f1358144c00 2024-03-06T07:27:58.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.254+0000 7f1355600640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133406b4d0 0x7f133406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.254+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f134808a210 con 0x7f1358144c00 2024-03-06T07:27:58.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.254+0000 7f134fe00640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133406b4d0 0x7f133406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.255+0000 7f134fe00640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133406b4d0 0x7f133406d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1358142a90 tx=0x7f1340009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.258+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1348058ff0 con 0x7f1358144c00 2024-03-06T07:27:58.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.494+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13580b12b0 con 0x7f1358144c00 2024-03-06T07:27:58.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.495+0000 7f1355600640 1 -- 172.21.15.97:0/1014787874 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f13480588f0 con 0x7f1358144c00 2024-03-06T07:27:58.497 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:27:58.497 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:27:58.497 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:27:58.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.497+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133406b4d0 msgr2=0x7f133406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.497+0000 7f135f400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133406b4d0 0x7f133406d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1358142a90 tx=0x7f1340009040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.497+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 msgr2=0x7f13580ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.497+0000 7f135f400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 0x7f13580ae270 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f134800c7f0 tx=0x7f134800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 shutdown_connections 2024-03-06T07:27:58.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133406b4d0 0x7f133406d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f135815fe60 0x7f13580b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f135815f9d0 0x7f135815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 --2- 172.21.15.97:0/1014787874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1358144c00 0x7f13580ae270 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 >> 172.21.15.97:0/1014787874 conn(0x7f13580b1730 msgr2=0x7f13580afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 shutdown_connections 2024-03-06T07:27:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:58.498+0000 7f135f400640 1 -- 172.21.15.97:0/1014787874 wait complete. 2024-03-06T07:27:58.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:41.541937+0000', 'modified': '2024-03-06T07:27:44.578982+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4260}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4260': {'gid': 4260, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/4149107409', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 4149107409}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 4149107409}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4260, 'qdb_cluster': [4260]} 2024-03-06T07:27:58.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:58.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:59.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.132+0000 7fc1d1a00640 1 Processor -- start 2024-03-06T07:27:59.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.132+0000 7fc1d1a00640 1 -- start start 2024-03-06T07:27:59.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1d1a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc1cc155060 0x7fc1cc140e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1d1a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1cc15b290 0x7fc1cc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1d1a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc15b7c0 con 0x7fc1cc06f070 2024-03-06T07:27:59.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1d1a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1cc141360 con 0x7fc1cc15b290 2024-03-06T07:27:59.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1d1a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1cc15e880 con 0x7fc1cc155060 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1ca000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1cc15b290 0x7fc1cc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1ca000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1cc15b290 0x7fc1cc06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59646/0 (socket says 172.21.15.97:59646) 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.134+0000 7fc1ca000640 1 -- 172.21.15.97:0/3851863054 learned_addr learned my addr 172.21.15.97:0/3851863054 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.135+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3851863054 >> v2:172.21.15.135:3300/0 conn(0x7fc1cc155060 0x7fc1cc140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.135+0000 7fc1ca000640 1 -- 172.21.15.97:0/3851863054 >> v2:172.21.15.135:3300/0 conn(0x7fc1cc155060 msgr2=0x7fc1cc140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.135+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3851863054 >> v2:172.21.15.135:3300/0 conn(0x7fc1cc155060 0x7fc1cc140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.135+0000 7fc1ca000640 1 -- 172.21.15.97:0/3851863054 >> v1:172.21.15.135:6789/0 conn(0x7fc1cc06f070 legacy=0x7fc1cc071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.135+0000 7fc1ca000640 1 -- 172.21.15.97:0/3851863054 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc15baa0 con 0x7fc1cc15b290 2024-03-06T07:27:59.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.135+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3851863054 >> v2:172.21.15.135:3300/0 conn(0x7fc1cc155060 0x7fc1cc140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:59.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.136+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3851863054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc15b290 0x7fc1cc06eb30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0b6e00 tx=0x7fc1b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8b339ae5a7fb02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.136+0000 7fc1c9600640 1 -- 172.21.15.97:0/3851863054 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8035020 con 0x7fc1cc15b290 2024-03-06T07:27:59.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.136+0000 7fc1c9600640 1 -- 172.21.15.97:0/3851863054 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8031870 con 0x7fc1cc15b290 2024-03-06T07:27:59.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.136+0000 7fc1c9600640 1 -- 172.21.15.97:0/3851863054 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8031c40 con 0x7fc1cc15b290 2024-03-06T07:27:59.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3851863054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc15b290 msgr2=0x7fc1cc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3851863054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc15b290 0x7fc1cc06eb30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0b6e00 tx=0x7fc1b802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3851863054 shutdown_connections 2024-03-06T07:27:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3851863054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc15b290 0x7fc1cc06eb30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3851863054 >> v2:172.21.15.135:3300/0 conn(0x7fc1cc155060 0x7fc1cc140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3851863054 >> 172.21.15.97:0/3851863054 conn(0x7fc1cc0d0b90 msgr2=0x7fc1cc0d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3851863054 shutdown_connections 2024-03-06T07:27:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.137+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3851863054 wait complete. 2024-03-06T07:27:59.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.139+0000 7fc1d1a00640 1 Processor -- start 2024-03-06T07:27:59.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.139+0000 7fc1d1a00640 1 -- start start 2024-03-06T07:27:59.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.140+0000 7fc1d1a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1cc06f070 0x7fc1cc13fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.140+0000 7fc1d1a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 0x7fc1cc1438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.140+0000 7fc1d1a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1cc15b290 0x7fc1cc142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1d1a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc15c060 con 0x7fc1cc155060 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1d1a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cc620 con 0x7fc1cc15b290 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1d1a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc072330 con 0x7fc1cc06f070 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 0x7fc1cc1438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 0x7fc1cc1438a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59652/0 (socket says 172.21.15.97:59652) 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 -- 172.21.15.97:0/3955469079 learned_addr learned my addr 172.21.15.97:0/3955469079 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1cc06f070 0x7fc1cc13fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 -- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1cc06f070 msgr2=0x7fc1cc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1cc06f070 0x7fc1cc13fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 -- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1cc15b290 msgr2=0x7fc1cc142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1cc15b290 0x7fc1cc142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.141+0000 7fc1ca000640 1 -- 172.21.15.97:0/3955469079 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0b3ec0 con 0x7fc1cc155060 2024-03-06T07:27:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.142+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 0x7fc1cc1438a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc1b802b950 tx=0x7fc1b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.142+0000 7fc1c2a00640 1 -- 172.21.15.97:0/3955469079 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8040070 con 0x7fc1cc155060 2024-03-06T07:27:59.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.142+0000 7fc1c2a00640 1 -- 172.21.15.97:0/3955469079 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b803a420 con 0x7fc1cc155060 2024-03-06T07:27:59.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.142+0000 7fc1c2a00640 1 -- 172.21.15.97:0/3955469079 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8035040 con 0x7fc1cc155060 2024-03-06T07:27:59.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.142+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0b41a0 con 0x7fc1cc155060 2024-03-06T07:27:59.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.142+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc0b4760 con 0x7fc1cc155060 2024-03-06T07:27:59.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.144+0000 7fc1c2a00640 1 -- 172.21.15.97:0/3955469079 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1b8038020 con 0x7fc1cc155060 2024-03-06T07:27:59.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.146+0000 7fc1c2a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1ac06b4d0 0x7fc1ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.146+0000 7fc1d47ca740 1 Processor -- start 2024-03-06T07:27:59.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.146+0000 7fc1c2a00640 1 -- 172.21.15.97:0/3955469079 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1b80ac2b0 con 0x7fc1cc155060 2024-03-06T07:27:59.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.146+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1ac06b4d0 0x7fc1ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.146+0000 7fc1d47ca740 1 -- start start 2024-03-06T07:27:59.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1ac06b4d0 0x7fc1ac06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc140c70 tx=0x7fc1bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a02750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a32540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a02c90 0x55efd5a32a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efd596c600 con 0x55efd5a2f7b0 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efd5915930 con 0x55efd5a02c90 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efd59686a0 con 0x55efd5a2fb90 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.147+0000 7fc1caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a02750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a02750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59668/0 (socket says 172.21.15.97:59668) 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 -- 172.21.15.97:0/2166928965 learned_addr learned my addr 172.21.15.97:0/2166928965 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1cb400640 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a02c90 0x55efd5a32a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1ca000640 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a32540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 -- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 msgr2=0x55efd5a32540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a32540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 -- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a02c90 msgr2=0x55efd5a32a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a02c90 0x55efd5a32a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 -- 172.21.15.97:0/2166928965 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efd56812d0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1caa00640 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a02750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00dbc0 tx=0x7fc1bc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3907aff86b45c521 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1b7400640 1 -- 172.21.15.97:0/2166928965 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc019050 con 0x55efd5a2f7b0 2024-03-06T07:27:59.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1b7400640 1 -- 172.21.15.97:0/2166928965 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc023070 con 0x55efd5a2f7b0 2024-03-06T07:27:59.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.148+0000 7fc1b7400640 1 -- 172.21.15.97:0/2166928965 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc012de0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2166928965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 msgr2=0x55efd5a02750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a02750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00dbc0 tx=0x7fc1bc010000 comp rx=0 tx=0).stop 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2166928965 shutdown_connections 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a02c90 0x55efd5a32a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a32540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2166928965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a02750 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2166928965 >> 172.21.15.97:0/2166928965 conn(0x55efd5a049c0 msgr2=0x55efd5a4d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2166928965 shutdown_connections 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.149+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2166928965 wait complete. 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.150+0000 7fc1d47ca740 1 -- start start 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.150+0000 7fc1d47ca740 1 Processor -- start 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a5d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5a31f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efd5a06f60 con 0x55efd5a2f7b0 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efd5928d90 con 0x55efd5a28390 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efd5a06350 con 0x55efd5a2fb90 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.151+0000 7fc1d47ca740 10 client.0 _ll_register_callbacks cb 0x55efd5964350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a5d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a5d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59670/0 (socket says 172.21.15.97:59670) 2024-03-06T07:27:59.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 -- 172.21.15.97:0/2708759069 learned_addr learned my addr 172.21.15.97:0/2708759069 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 -- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 msgr2=0x55efd5a342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1ca000640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1cb400640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5a31f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 -- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 msgr2=0x55efd5a31f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5a31f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 -- 172.21.15.97:0/2708759069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efd574b6b0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1caa00640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a5d1e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc012810 tx=0x7fc1bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1031ae409a87b268 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.152+0000 7fc1abe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1abe00640 1 -- 172.21.15.97:0/2708759069 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc01eaa0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1abe00640 1 -- 172.21.15.97:0/2708759069 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1bc00d760 con 0x55efd5a2f7b0 2024-03-06T07:27:59.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1bc023070 con 0x55efd5a2f7b0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1abe00640 1 -- 172.21.15.97:0/2708759069 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1bc01ec40 con 0x55efd5a2f7b0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 10 client.4614 fetch_fsmap learned FSMap version 8 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55efd5a319f0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1abe00640 1 -- 172.21.15.97:0/2708759069 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc023070 con 0x55efd5a2f7b0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1abe00640 1 -- 172.21.15.97:0/2708759069 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc1bc03a050 con 0x55efd5a2f7b0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 10 client.4614 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 10 client.4614 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 20 client.4614 populate_metadata read hostname 'smithi097' 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5a61480 0x55efd5a61860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 4 client.4614 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.153+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 --> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efd5979b40 con 0x55efd5a61480 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.154+0000 7fc1ca000640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5a61480 0x55efd5a61860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.154+0000 7fc1ca000640 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5a61480 0x55efd5a61860 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e6f78a4d6f7bcc1 server_cookie=1161b62a8676bb44 in_seq=0 out_seq=0 2024-03-06T07:27:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.154+0000 7fc1abe00640 10 client.4614 ms_handle_connect on v2:172.21.15.97:6836/4149107409 2024-03-06T07:27:59.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1abe00640 1 -- 172.21.15.97:0/2708759069 <== mds.0 v2:172.21.15.97:6836/4149107409 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55efd5979b40 con 0x55efd5a61480 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1abe00640 10 client.4614 handle_command_reply: tid=0 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 client.4614 shutdown 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 msgr2=0x55efd5a5d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a5d1e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc012810 tx=0x7fc1bc004730 comp rx=0 tx=0).stop 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 shutdown_connections 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5a61480 0x55efd5a61860 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5a31f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2fb90 0x55efd5a342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/2708759069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a5d1e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1abe00640 0 client.4614 ms_handle_reset on v2:172.21.15.97:6836/4149107409 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 >> 172.21.15.97:0/2708759069 conn(0x55efd5a02c90 msgr2=0x55efd5a50a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 shutdown_connections 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.155+0000 7fc1d47ca740 1 -- 172.21.15.97:0/2708759069 wait complete. 2024-03-06T07:27:59.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.156+0000 7fc1d47ca740 20 client.4614 trim_cache size 0 max 16384 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.175+0000 7fc1d47ca740 1 Processor -- start 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.175+0000 7fc1d47ca740 1 -- start start 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.176+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a28390 0x55efd5a50c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.176+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a51180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.176+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a2fb90 0x55efd5a5f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.176+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1ff490 con 0x55efd5a2f7b0 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.176+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cf6b0 con 0x55efd5a2fb90 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.176+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cfcb0 con 0x55efd5a28390 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a51180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a51180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59676/0 (socket says 172.21.15.97:59676) 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 -- 172.21.15.97:0/3369944299 learned_addr learned my addr 172.21.15.97:0/3369944299 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a28390 0x55efd5a50c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 -- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a28390 msgr2=0x55efd5a50c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a28390 0x55efd5a50c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 -- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a2fb90 msgr2=0x55efd5a5f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1cb400640 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a2fb90 0x55efd5a5f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a2fb90 0x55efd5a5f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1ca000640 1 -- 172.21.15.97:0/3369944299 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efd59fe050 con 0x55efd5a2f7b0 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.177+0000 7fc1caa00640 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a28390 0x55efd5a50c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.178+0000 7fc1ca000640 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a51180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1b807a920 tx=0x7fc1b80aa610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58799f0d4f8d43c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1b6a00640 1 -- 172.21.15.97:0/3369944299 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b807eae0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1b6a00640 1 -- 172.21.15.97:0/3369944299 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b80830a0 con 0x55efd5a2f7b0 2024-03-06T07:27:59.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1b6a00640 1 -- 172.21.15.97:0/3369944299 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8078c90 con 0x55efd5a2f7b0 2024-03-06T07:27:59.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 -- 172.21.15.97:0/3369944299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 msgr2=0x55efd5a51180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a51180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1b807a920 tx=0x7fc1b80aa610 comp rx=0 tx=0).stop 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 -- 172.21.15.97:0/3369944299 shutdown_connections 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a2fb90 0x55efd5a5f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2f7b0 0x55efd5a51180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/3369944299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a28390 0x55efd5a50c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 -- 172.21.15.97:0/3369944299 >> 172.21.15.97:0/3369944299 conn(0x55efd5a02c90 msgr2=0x55efd5a23840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 -- 172.21.15.97:0/3369944299 shutdown_connections 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.180+0000 7fc1d47ca740 1 -- 172.21.15.97:0/3369944299 wait complete. 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.181+0000 7fc1d47ca740 1 -- start start 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.181+0000 7fc1d47ca740 1 Processor -- start 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5aa03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2f7b0 0x55efd5aa2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 0x55efd5aa3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efd5a27f50 con 0x55efd5a2fb90 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efd5a348b0 con 0x55efd5a28390 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efd5a2b090 con 0x55efd5a2f7b0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.182+0000 7fc1d47ca740 10 client.0 _ll_register_callbacks cb 0x7fc1bc00da60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 0x55efd5aa3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 0x55efd5aa3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59686/0 (socket says 172.21.15.97:59686) 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 -- 172.21.15.97:0/578010618 learned_addr learned my addr 172.21.15.97:0/578010618 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1caa00640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5aa03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1ca000640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2f7b0 0x55efd5aa2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 -- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2f7b0 msgr2=0x55efd5aa2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2f7b0 0x55efd5aa2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 -- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 msgr2=0x55efd5aa03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5aa03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 -- 172.21.15.97:0/578010618 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efd5a4e0b0 con 0x55efd5a2fb90 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1cb400640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 0x55efd5aa3230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0ccb20 tx=0x7fc1c400ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4af2f95a000ac5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.183+0000 7fc1abe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:27:59.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.184+0000 7fc1abe00640 1 -- 172.21.15.97:0/578010618 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c401b060 con 0x55efd5a2fb90 2024-03-06T07:27:59.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.184+0000 7fc1abe00640 1 -- 172.21.15.97:0/578010618 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1c4009d00 con 0x55efd5a2fb90 2024-03-06T07:27:59.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.184+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1b80830a0 con 0x55efd5a2fb90 2024-03-06T07:27:59.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1abe00640 1 -- 172.21.15.97:0/578010618 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1c4021050 con 0x55efd5a2fb90 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 10 client.4620 fetch_fsmap learned FSMap version 8 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55efd5a50c40 con 0x55efd5a2fb90 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1abe00640 1 -- 172.21.15.97:0/578010618 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b80830a0 con 0x55efd5a2fb90 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1abe00640 1 -- 172.21.15.97:0/578010618 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc1c4024020 con 0x55efd5a2fb90 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 10 client.4620 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 10 client.4620 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 20 client.4620 populate_metadata read hostname 'smithi097' 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5abd9f0 0x55efd5abddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 4 client.4620 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.186+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 --> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55efd5973e90 con 0x55efd5abd9f0 2024-03-06T07:27:59.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.187+0000 7fc1caa00640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5abd9f0 0x55efd5abddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.187+0000 7fc1abe00640 10 client.4620 ms_handle_connect on v2:172.21.15.97:6836/4149107409 2024-03-06T07:27:59.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.187+0000 7fc1caa00640 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5abd9f0 0x55efd5abddd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18e88b083293bf6 server_cookie=54e7ef8b3adbb9 in_seq=0 out_seq=0 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.187+0000 7fc1abe00640 1 -- 172.21.15.97:0/578010618 <== mds.0 v2:172.21.15.97:6836/4149107409 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55efd5973e90 con 0x55efd5abd9f0 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.187+0000 7fc1abe00640 10 client.4620 handle_command_reply: tid=0 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.187+0000 7fc1d47ca740 1 client.4620 shutdown 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 msgr2=0x55efd5aa3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 0x55efd5aa3230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0ccb20 tx=0x7fc1c400ae60 comp rx=0 tx=0).stop 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 shutdown_connections 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:6836/4149107409,v1:172.21.15.97:6837/4149107409] conn(0x55efd5abd9f0 0x55efd5abddd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55efd5a2fb90 0x55efd5aa3230 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55efd5a2f7b0 0x55efd5aa2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 --2- 172.21.15.97:0/578010618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55efd5a28390 0x55efd5aa03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1abe00640 0 client.4620 ms_handle_reset on v2:172.21.15.97:6836/4149107409 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 >> 172.21.15.97:0/578010618 conn(0x55efd5a02c90 msgr2=0x55efd5a31bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 shutdown_connections 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 1 -- 172.21.15.97:0/578010618 wait complete. 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.188+0000 7fc1d47ca740 20 client.4620 trim_cache size 0 max 16384 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 4260, 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 17.109057869000001, 2024-03-06T07:27:59.193 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 8, 2024-03-06T07:27:59.193 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 22, 2024-03-06T07:27:59.193 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 22, 2024-03-06T07:27:59.193 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 25.018175794000001 2024-03-06T07:27:59.193 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:27:59.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1ac06b4d0 msgr2=0x7fc1ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1ac06b4d0 0x7fc1ac06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc140c70 tx=0x7fc1bc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 msgr2=0x7fc1cc1438a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 0x7fc1cc1438a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc1b802b950 tx=0x7fc1b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 shutdown_connections 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1ac06b4d0 0x7fc1ac06d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1cc15b290 0x7fc1cc142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1cc155060 0x7fc1cc1438a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 --2- 172.21.15.97:0/3955469079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1cc06f070 0x7fc1cc13fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 >> 172.21.15.97:0/3955469079 conn(0x7fc1cc0d0b90 msgr2=0x7fc1cc15ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 shutdown_connections 2024-03-06T07:27:59.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:27:59.189+0000 7fc1d1a00640 1 -- 172.21.15.97:0/3955469079 wait complete. 2024-03-06T07:27:59.208 INFO:teuthology.run_tasks:Running task kclient... 2024-03-06T07:27:59.233 INFO:tasks.kclient:Mounting kernel clients... 2024-03-06T07:27:59.233 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:27:59.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.233 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:27:59.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-03-06T07:27:59.367 INFO:teuthology.orchestra.run.smithi097.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.371 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:27:59.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-03-06T07:27:59.544 INFO:teuthology.orchestra.run.smithi097.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:59.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.550 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-06T07:27:59.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.608 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link delete ceph-brx 2024-03-06T07:27:59.713 INFO:teuthology.orchestra.run.smithi135.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:59.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.718 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-06T07:27:59.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link delete ceph-brx 2024-03-06T07:27:59.857 INFO:teuthology.orchestra.run.smithi135.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:59.862 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-06T07:27:59.862 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:27:59.862 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:59.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:00.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.065+0000 7f0ffea00640 1 Processor -- start 2024-03-06T07:28:00.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.065+0000 7f0ffea00640 1 -- start start 2024-03-06T07:28:00.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.067+0000 7f0ffea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0ff8144c00 0x7f0ff806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.067+0000 7f0ffea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 0x7f0ff80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.067+0000 7f0ffea00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff806ee70 con 0x7f0ff815f9d0 2024-03-06T07:28:00.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.067+0000 7f0ffea00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff806cef0 con 0x7f0ff8144c00 2024-03-06T07:28:00.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.067+0000 7f0ffea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff806d1d0 con 0x7f0ff815fe60 2024-03-06T07:28:00.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.068+0000 7f0ffca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 0x7f0ff80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.068+0000 7f0ffca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 0x7f0ff80ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:52522/0 (socket says 172.21.15.97:52522) 2024-03-06T07:28:00.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.068+0000 7f0ff7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0ff8144c00 0x7f0ff806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.068+0000 7f0ffca00640 1 -- 172.21.15.97:0/3167251408 learned_addr learned my addr 172.21.15.97:0/3167251408 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:00.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.068+0000 7f0fef400640 1 --1- 172.21.15.97:0/3167251408 >> v1:172.21.15.135:6789/0 conn(0x7f0ff815f9d0 0x7f0ff80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:54436/0 (socket says 172.21.15.97:54436) 2024-03-06T07:28:00.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.069+0000 7f0ffca00640 1 -- 172.21.15.97:0/3167251408 >> v2:172.21.15.135:3300/0 conn(0x7f0ff8144c00 msgr2=0x7f0ff806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:00.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.069+0000 7f0ffca00640 1 --2- 172.21.15.97:0/3167251408 >> v2:172.21.15.135:3300/0 conn(0x7f0ff8144c00 0x7f0ff806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.069+0000 7f0ffca00640 1 -- 172.21.15.97:0/3167251408 >> v1:172.21.15.135:6789/0 conn(0x7f0ff815f9d0 legacy=0x7f0ff80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.069+0000 7f0ffca00640 1 -- 172.21.15.97:0/3167251408 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff806d4b0 con 0x7f0ff815fe60 2024-03-06T07:28:00.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.069+0000 7f0ff7e00640 1 --2- 172.21.15.97:0/3167251408 >> v2:172.21.15.135:3300/0 conn(0x7f0ff8144c00 0x7f0ff806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:00.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.070+0000 7f0ffca00640 1 --2- 172.21.15.97:0/3167251408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 0x7f0ff80ce7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009870 tx=0x7f0ff002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583fa0da11b77b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.070+0000 7f0ff7400640 1 -- 172.21.15.97:0/3167251408 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0034070 con 0x7f0ff815fe60 2024-03-06T07:28:00.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.070+0000 7f0ff7400640 1 -- 172.21.15.97:0/3167251408 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0030840 con 0x7f0ff815fe60 2024-03-06T07:28:00.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.070+0000 7f0ff7400640 1 -- 172.21.15.97:0/3167251408 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0030c10 con 0x7f0ff815fe60 2024-03-06T07:28:00.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 -- 172.21.15.97:0/3167251408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 msgr2=0x7f0ff80ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3167251408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 0x7f0ff80ce7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009870 tx=0x7f0ff002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:00.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 -- 172.21.15.97:0/3167251408 shutdown_connections 2024-03-06T07:28:00.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3167251408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff815fe60 0x7f0ff80ce7b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3167251408 >> v2:172.21.15.135:3300/0 conn(0x7f0ff8144c00 0x7f0ff806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 -- 172.21.15.97:0/3167251408 >> 172.21.15.97:0/3167251408 conn(0x7f0ff80b1730 msgr2=0x7f0ff80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 -- 172.21.15.97:0/3167251408 shutdown_connections 2024-03-06T07:28:00.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.071+0000 7f0ffea00640 1 -- 172.21.15.97:0/3167251408 wait complete. 2024-03-06T07:28:00.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.073+0000 7f0ffea00640 1 Processor -- start 2024-03-06T07:28:00.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.074+0000 7f0ffea00640 1 -- start start 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0ffea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff8144c00 0x7f0ff813c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0ffea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 0x7f0ff81418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0ffea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ff815fe60 0x7f0ff815c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0ffea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8160730 con 0x7f0ff815f9d0 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0ffea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8160290 con 0x7f0ff815fe60 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0ffea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff806cb50 con 0x7f0ff8144c00 2024-03-06T07:28:00.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.075+0000 7f0fef400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 0x7f0ff81418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 0x7f0ff81418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59696/0 (socket says 172.21.15.97:59696) 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 -- 172.21.15.97:0/3418990148 learned_addr learned my addr 172.21.15.97:0/3418990148 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0ff7e00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff8144c00 0x7f0ff813c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0ffca00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ff815fe60 0x7f0ff815c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 -- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff8144c00 msgr2=0x7f0ff813c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff8144c00 0x7f0ff813c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 -- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ff815fe60 msgr2=0x7f0ff815c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ff815fe60 0x7f0ff815c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 -- 172.21.15.97:0/3418990148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8142310 con 0x7f0ff815f9d0 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0ff7e00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff8144c00 0x7f0ff813c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:00.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0fef400640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 0x7f0ff81418c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000c430 tx=0x7f0fe000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.076+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00074e0 con 0x7f0ff815f9d0 2024-03-06T07:28:00.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.077+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0007680 con 0x7f0ff815f9d0 2024-03-06T07:28:00.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.077+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004a00 con 0x7f0ff815f9d0 2024-03-06T07:28:00.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.077+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff81425f0 con 0x7f0ff815f9d0 2024-03-06T07:28:00.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.077+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff8142bb0 con 0x7f0ff815f9d0 2024-03-06T07:28:00.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.078+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff804f330 con 0x7f0ff815f9d0 2024-03-06T07:28:00.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.079+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fe001a020 con 0x7f0ff815f9d0 2024-03-06T07:28:00.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.080+0000 7f0ff4c00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fd406b4d0 0x7f0fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.080+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fe00899a0 con 0x7f0ff815f9d0 2024-03-06T07:28:00.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.080+0000 7f0ff7e00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fd406b4d0 0x7f0fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.081+0000 7f0ff7e00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fd406b4d0 0x7f0fd406d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ff815f9a0 tx=0x7f0fe800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.085+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ff806ee70 con 0x7f0ff815f9d0 2024-03-06T07:28:00.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.322+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0ff8072890 con 0x7f0ff815f9d0 2024-03-06T07:28:00.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.322+0000 7f0ff4c00640 1 -- 172.21.15.97:0/3418990148 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0fe0058820 con 0x7f0ff815f9d0 2024-03-06T07:28:00.325 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:00.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fd406b4d0 msgr2=0x7f0fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fd406b4d0 0x7f0fd406d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ff815f9a0 tx=0x7f0fe800b040 comp rx=0 tx=0).stop 2024-03-06T07:28:00.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 msgr2=0x7f0ff81418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 0x7f0ff81418c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000c430 tx=0x7f0fe000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 shutdown_connections 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fd406b4d0 0x7f0fd406d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ff815fe60 0x7f0ff815c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ff815f9d0 0x7f0ff81418c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 --2- 172.21.15.97:0/3418990148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ff8144c00 0x7f0ff813c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 >> 172.21.15.97:0/3418990148 conn(0x7f0ff80b1730 msgr2=0x7f0ff80ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 shutdown_connections 2024-03-06T07:28:00.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:00.325+0000 7f0ffea00640 1 -- 172.21.15.97:0/3418990148 wait complete. 2024-03-06T07:28:00.346 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:00.346 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:00.346 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:28:00.346 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:00.346 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:00.347 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:00.347 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:00.347 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:00.347 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.347 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:28:00.393 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:00.393 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:00.393 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:00.394 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:00.395 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:00.395 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:00.395 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42594sec preferred_lft 42594sec 2024-03-06T07:28:00.395 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:28:00.395 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.395 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:28:00.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.395 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:00.396 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:00.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:28:00.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr flush dev ceph-brx 2024-03-06T07:28:00.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set ceph-brx up 2024-03-06T07:28:00.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:28:00.396 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:00.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.818 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:00.895 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:28:00.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.899 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:00.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.997 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:00.997 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:00.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:00.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:00.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:00.998 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:01.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.691 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:28:01.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.742 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list-id 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.788 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:01.788 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:01.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:01.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:28:01.789 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:01.929 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:28:01.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.929 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:01.929 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:01.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:28:01.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:01.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:28:01.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:28:01.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:28:01.930 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:02.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:02.203 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:02.203 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:02.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:28:02.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:28:02.204 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:02.356 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:02.357 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:02.396 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:02.483 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:28:02.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:02.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:02.651 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:02.651 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:03.306 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:03.306 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:03.306 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:28:03.306 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:28:03.306 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:28:03.311 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:28:03.311 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:03.311 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:03.312 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:03.312 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:03.316 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:03.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:03.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:03.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:03.407 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:03.471 INFO:teuthology.orchestra.run.smithi097.stdout:client4480 2024-03-06T07:28:03.471 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:03.471 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:03.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:03.520 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:03.622 INFO:teuthology.orchestra.run.smithi097.stdout:client4480 2024-03-06T07:28:03.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:03.623 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:03.691 INFO:teuthology.orchestra.run.smithi097.stdout:client4480 2024-03-06T07:28:03.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:03.691 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b9abceb-2115-4321-96a2-348db0d0fbe4.client4480/status') 2024-03-06T07:28:03.778 INFO:teuthology.orchestra.run.smithi097.stdout:instance: client.4480 (3)172.21.15.97:0/4273953002 2024-03-06T07:28:03.778 INFO:teuthology.orchestra.run.smithi097.stdout:blocklisted: false 2024-03-06T07:28:03.779 INFO:teuthology.orchestra.run.smithi097.stderr:0+1 records in 2024-03-06T07:28:03.779 INFO:teuthology.orchestra.run.smithi097.stderr:0+1 records out 2024-03-06T07:28:03.779 INFO:teuthology.orchestra.run.smithi097.stderr:70 bytes copied, 6.4823e-05 s, 1.1 MB/s 2024-03-06T07:28:03.783 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-06T07:28:03.783 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:03.783 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:03.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:03.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.978+0000 7ff9da200640 1 Processor -- start 2024-03-06T07:28:03.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.979+0000 7ff9da200640 1 -- start start 2024-03-06T07:28:03.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.980+0000 7ff9da200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9d40d27a0 0x7ff9d40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.980+0000 7ff9da200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d4161960 con 0x7ff9d40d2360 2024-03-06T07:28:03.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.980+0000 7ff9da200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d40cf6f0 con 0x7ff9d4161580 2024-03-06T07:28:03.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.980+0000 7ff9da200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d40cf9d0 con 0x7ff9d40d27a0 2024-03-06T07:28:03.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.980+0000 7ff9cbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff9d40d2360 0x7ff9d40b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48936/0 (socket says 172.21.15.97:48936) 2024-03-06T07:28:03.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.980+0000 7ff9cbe00640 1 -- 172.21.15.97:0/760422977 learned_addr learned my addr 172.21.15.97:0/760422977 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d9800640 1 --2- 172.21.15.97:0/760422977 >> v2:172.21.15.135:3300/0 conn(0x7ff9d40d27a0 0x7ff9d40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729640152 0 0) 0x7ff9d4161960 con 0x7ff9d40d2360 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b8003520 con 0x7ff9d40d2360 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 228734148 0 0) 0x7ff9b8003520 con 0x7ff9d40d2360 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 >> v2:172.21.15.135:3300/0 conn(0x7ff9d40d27a0 msgr2=0x7ff9d40b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 --2- 172.21.15.97:0/760422977 >> v2:172.21.15.135:3300/0 conn(0x7ff9d40d27a0 0x7ff9d40b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 >> v1:172.21.15.135:6789/0 conn(0x7ff9d4161580 legacy=0x7ff9d40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40cfcb0 con 0x7ff9d40d2360 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff9bc001670 con 0x7ff9d40d2360 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 >> v1:172.21.15.97:6789/0 conn(0x7ff9d40d2360 legacy=0x7ff9d40b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9b8003a00 0x7ff9b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9b8004800 0x7ff9b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b80051c0 0x7ff9b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40cf9d0 con 0x7ff9b8004800 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b800c0c0 con 0x7ff9b8003a00 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.981+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b800c2c0 con 0x7ff9b80051c0 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9b8004800 0x7ff9b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9d8e00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9b8003a00 0x7ff9b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9d9800640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b80051c0 0x7ff9b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9cbe00640 1 -- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b80051c0 msgr2=0x7ff9b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b80051c0 0x7ff9b800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9cbe00640 1 -- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9b8003a00 msgr2=0x7ff9b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9b8003a00 0x7ff9b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9cbe00640 1 -- 172.21.15.97:0/760422977 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8003520 con 0x7ff9b8004800 2024-03-06T07:28:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.982+0000 7ff9d9800640 1 -- 172.21.15.97:0/760422977 reap_dead start 2024-03-06T07:28:03.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.983+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9b8004800 0x7ff9b8004c80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00c110 tx=0x7ff9b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699fabe93d21a0f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.983+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc007c40 con 0x7ff9b8004800 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.983+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc030070 con 0x7ff9b8004800 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.983+0000 7ff9d3e00640 1 -- 172.21.15.97:0/760422977 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc005d70 con 0x7ff9b8004800 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 -- 172.21.15.97:0/760422977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9b8004800 msgr2=0x7ff9b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9b8004800 0x7ff9b8004c80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00c110 tx=0x7ff9b8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 -- 172.21.15.97:0/760422977 shutdown_connections 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 --2- 172.21.15.97:0/760422977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9b8004800 0x7ff9b8004c80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 -- 172.21.15.97:0/760422977 >> 172.21.15.97:0/760422977 conn(0x7ff9d40d3d80 msgr2=0x7ff9d40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 -- 172.21.15.97:0/760422977 shutdown_connections 2024-03-06T07:28:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.984+0000 7ff9da200640 1 -- 172.21.15.97:0/760422977 wait complete. 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.986+0000 7ff9da200640 1 Processor -- start 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.986+0000 7ff9da200640 1 -- start start 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9da200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b8004800 0x7ff9d414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9da200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 0x7ff9d40cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9da200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9d414e490 0x7ff9d414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9da200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40d2bf0 con 0x7ff9d40cd080 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9da200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40b39a0 con 0x7ff9d414e490 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9da200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40cf3a0 con 0x7ff9b8004800 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 0x7ff9d40cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 0x7ff9d40cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52072/0 (socket says 172.21.15.97:52072) 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9cbe00640 1 -- 172.21.15.97:0/3785078950 learned_addr learned my addr 172.21.15.97:0/3785078950 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.987+0000 7ff9d8e00640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b8004800 0x7ff9d414dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9d9800640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9d414e490 0x7ff9d414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9cbe00640 1 -- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b8004800 msgr2=0x7ff9d414dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b8004800 0x7ff9d414dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9cbe00640 1 -- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9d414e490 msgr2=0x7ff9d414e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9d414e490 0x7ff9d414e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9cbe00640 1 -- 172.21.15.97:0/3785078950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40cc040 con 0x7ff9d40cd080 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9cbe00640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 0x7ff9d40cd500 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00c230 tx=0x7ff9bc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc02cbd0 con 0x7ff9d40cd080 2024-03-06T07:28:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc000ed0 con 0x7ff9d40cd080 2024-03-06T07:28:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc007070 con 0x7ff9d40cd080 2024-03-06T07:28:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.988+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d40cc2e0 con 0x7ff9d40cd080 2024-03-06T07:28:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.989+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d40cc8a0 con 0x7ff9d40cd080 2024-03-06T07:28:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.990+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d404f330 con 0x7ff9d40cd080 2024-03-06T07:28:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.992+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9bc007210 con 0x7ff9d40cd080 2024-03-06T07:28:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.992+0000 7ff9d1600640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9b006b500 0x7ff9b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.993+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9bc0ab890 con 0x7ff9d40cd080 2024-03-06T07:28:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.993+0000 7ff9d8e00640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9b006b500 0x7ff9b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.993+0000 7ff9d8e00640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9b006b500 0x7ff9b006d9c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4000c00 tx=0x7ff9c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:03.998+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9bc07a7a0 con 0x7ff9d40cd080 2024-03-06T07:28:04.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.232+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff9d404f330 con 0x7ff9d40cd080 2024-03-06T07:28:04.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.233+0000 7ff9d1600640 1 -- 172.21.15.97:0/3785078950 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff9bc07a0a0 con 0x7ff9d40cd080 2024-03-06T07:28:04.235 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:04.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9b006b500 msgr2=0x7ff9b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9b006b500 0x7ff9b006d9c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4000c00 tx=0x7ff9c4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 msgr2=0x7ff9d40cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 0x7ff9d40cd500 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00c230 tx=0x7ff9bc02e070 comp rx=0 tx=0).stop 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 shutdown_connections 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9b006b500 0x7ff9b006d9c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9d414e490 0x7ff9d414e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9d40cd080 0x7ff9d40cd500 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 --2- 172.21.15.97:0/3785078950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9b8004800 0x7ff9d414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 >> 172.21.15.97:0/3785078950 conn(0x7ff9d40d3d80 msgr2=0x7ff9d40b4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 shutdown_connections 2024-03-06T07:28:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:04.235+0000 7ff9da200640 1 -- 172.21.15.97:0/3785078950 wait complete. 2024-03-06T07:28:04.255 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:04.256 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:28:04.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.256 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42591sec preferred_lft 42591sec 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.308 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:04.395 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:28:04.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.401 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:28:04.449 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:28:04.449 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:04.449 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:04.449 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:04.449 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:04.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.450 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:04.451 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:04.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:04.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:04.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:04.451 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:04.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.632 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:28:04.681 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:04.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.682 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list-id 2024-03-06T07:28:04.728 INFO:teuthology.orchestra.run.smithi097.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:28:04.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.729 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:04.729 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:04.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:04.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:28:04.729 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 4a:4e:2f:ab:0c:b2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::484e:2fff:feab:cb2/64 scope link proto kernel_ll 2024-03-06T07:28:04.984 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:04.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:28:05.071 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 4a:4e:2f:ab:0c:b2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::484e:2fff:feab:cb2/64 scope link proto kernel_ll 2024-03-06T07:28:05.072 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:05.076 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:28:05.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:28:05.077 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:05.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.335 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:05.336 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:05.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.1 up 2024-03-06T07:28:05.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:28:05.336 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:05.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:05.485 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:05.529 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:28:05.530 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:05.530 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:05.621 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-06T07:28:05.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:05.738 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:05.739 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:05.770 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:05.770 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:05.770 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:28:05.770 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 2@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:28:05.770 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:05.777 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:05.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.867 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:05.920 INFO:teuthology.orchestra.run.smithi097.stdout:client4632 2024-03-06T07:28:05.921 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:05.921 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:05.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.967 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:06.063 INFO:teuthology.orchestra.run.smithi097.stdout:client4632 2024-03-06T07:28:06.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:06.063 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:06.110 INFO:teuthology.orchestra.run.smithi097.stdout:client4632 2024-03-06T07:28:06.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:06.110 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b9abceb-2115-4321-96a2-348db0d0fbe4.client4632/status') 2024-03-06T07:28:06.234 INFO:teuthology.orchestra.run.smithi097.stdout:instance: client.4632 (3)192.168.0.2:0/979911549 2024-03-06T07:28:06.234 INFO:teuthology.orchestra.run.smithi097.stdout:blocklisted: false 2024-03-06T07:28:06.234 INFO:teuthology.orchestra.run.smithi097.stderr:0+1 records in 2024-03-06T07:28:06.234 INFO:teuthology.orchestra.run.smithi097.stderr:0+1 records out 2024-03-06T07:28:06.234 INFO:teuthology.orchestra.run.smithi097.stderr:68 bytes copied, 5.7006e-05 s, 1.2 MB/s 2024-03-06T07:28:06.237 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-06T07:28:06.238 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:06.238 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:06.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:06.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.419+0000 7efdfea00640 1 Processor -- start 2024-03-06T07:28:06.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.419+0000 7efdfea00640 1 -- start start 2024-03-06T07:28:06.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdfea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efdf815f230 0x7efdf815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdfea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efdf806b6d0 0x7efdf815d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdfea00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf80b4090 con 0x7efdf80b3b40 2024-03-06T07:28:06.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdfea00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf80b5ce0 con 0x7efdf806b6d0 2024-03-06T07:28:06.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdfea00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf80b4ca0 con 0x7efdf815f230 2024-03-06T07:28:06.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdef400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efdf80b3b40 0x7efdf806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48940/0 (socket says 172.21.15.97:48940) 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdef400640 1 -- 172.21.15.97:0/4068738483 learned_addr learned my addr 172.21.15.97:0/4068738483 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.421+0000 7efdfca00640 1 --2- 172.21.15.97:0/4068738483 >> v2:172.21.15.97:3300/0 conn(0x7efdf806b6d0 0x7efdf815d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdf7e00640 1 --2- 172.21.15.97:0/4068738483 >> v2:172.21.15.135:3300/0 conn(0x7efdf815f230 0x7efdf815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdf7400640 1 -- 172.21.15.97:0/4068738483 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254509111 0 0) 0x7efdf80b4090 con 0x7efdf80b3b40 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdf7400640 1 -- 172.21.15.97:0/4068738483 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efddc003520 con 0x7efdf80b3b40 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdfca00640 1 -- 172.21.15.97:0/4068738483 >> v2:172.21.15.135:3300/0 conn(0x7efdf815f230 msgr2=0x7efdf815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdfca00640 1 --2- 172.21.15.97:0/4068738483 >> v2:172.21.15.135:3300/0 conn(0x7efdf815f230 0x7efdf815ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdfca00640 1 -- 172.21.15.97:0/4068738483 >> v1:172.21.15.97:6789/0 conn(0x7efdf80b3b40 legacy=0x7efdf806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdfca00640 1 -- 172.21.15.97:0/4068738483 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf80b4370 con 0x7efdf806b6d0 2024-03-06T07:28:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.422+0000 7efdf7e00640 1 --2- 172.21.15.97:0/4068738483 >> v2:172.21.15.135:3300/0 conn(0x7efdf815f230 0x7efdf815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.423+0000 7efdfca00640 1 --2- 172.21.15.97:0/4068738483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf806b6d0 0x7efdf815d6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efdf806bca0 tx=0x7efdf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f31fa6ab7b0b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.423+0000 7efdf7400640 1 -- 172.21.15.97:0/4068738483 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0035020 con 0x7efdf806b6d0 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.423+0000 7efdf7400640 1 -- 172.21.15.97:0/4068738483 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf0031830 con 0x7efdf806b6d0 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.423+0000 7efdf7400640 1 -- 172.21.15.97:0/4068738483 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0031be0 con 0x7efdf806b6d0 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 -- 172.21.15.97:0/4068738483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf806b6d0 msgr2=0x7efdf815d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 --2- 172.21.15.97:0/4068738483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf806b6d0 0x7efdf815d6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efdf806bca0 tx=0x7efdf002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 -- 172.21.15.97:0/4068738483 shutdown_connections 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 --2- 172.21.15.97:0/4068738483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf806b6d0 0x7efdf815d6a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 --2- 172.21.15.97:0/4068738483 >> v2:172.21.15.135:3300/0 conn(0x7efdf815f230 0x7efdf815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 -- 172.21.15.97:0/4068738483 >> 172.21.15.97:0/4068738483 conn(0x7efdf806c0d0 msgr2=0x7efdf806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 -- 172.21.15.97:0/4068738483 shutdown_connections 2024-03-06T07:28:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.424+0000 7efdfea00640 1 -- 172.21.15.97:0/4068738483 wait complete. 2024-03-06T07:28:06.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.426+0000 7efdfea00640 1 Processor -- start 2024-03-06T07:28:06.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.427+0000 7efdfea00640 1 -- start start 2024-03-06T07:28:06.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.428+0000 7efdfea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdf806b6d0 0x7efdf80a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.428+0000 7efdfea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdf80b3b40 0x7efdf81562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.428+0000 7efdfea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 0x7efdf815fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.428+0000 7efdfea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf815f610 con 0x7efdf815f230 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.428+0000 7efdfea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf80b4930 con 0x7efdf80b3b40 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.428+0000 7efdfea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf815f790 con 0x7efdf806b6d0 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 0x7efdf815fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 0x7efdf815fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52102/0 (socket says 172.21.15.97:52102) 2024-03-06T07:28:06.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 -- 172.21.15.97:0/611733421 learned_addr learned my addr 172.21.15.97:0/611733421 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:06.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdef400640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdf80b3b40 0x7efdf81562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdf7e00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdf806b6d0 0x7efdf80a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 -- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdf806b6d0 msgr2=0x7efdf80a95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdf806b6d0 0x7efdf80a95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 -- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdf80b3b40 msgr2=0x7efdf81562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdf80b3b40 0x7efdf81562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 -- 172.21.15.97:0/611733421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8144b00 con 0x7efdf815f230 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdf7e00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdf806b6d0 0x7efdf80a95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.429+0000 7efdfca00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 0x7efdf815fbb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efdf002bee0 tx=0x7efdf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.430+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0035040 con 0x7efdf815f230 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.430+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf003f070 con 0x7efdf815f230 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.430+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf8141f60 con 0x7efdf815f230 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.430+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf003ba60 con 0x7efdf815f230 2024-03-06T07:28:06.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.430+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf8142520 con 0x7efdf815f230 2024-03-06T07:28:06.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.431+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf8142720 con 0x7efdf815f230 2024-03-06T07:28:06.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.432+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdf0038020 con 0x7efdf815f230 2024-03-06T07:28:06.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.433+0000 7efdf4c00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efdd406b4d0 0x7efdd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.433+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdf00abe20 con 0x7efdf815f230 2024-03-06T07:28:06.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.433+0000 7efdf7e00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efdd406b4d0 0x7efdd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.434+0000 7efdf7e00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efdd406b4d0 0x7efdd406d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efde80094c0 tx=0x7efde800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.438+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdf0031800 con 0x7efdf815f230 2024-03-06T07:28:06.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:06.672 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:06.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.677+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efdf804f330 con 0x7efdf815f230 2024-03-06T07:28:06.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.677+0000 7efdf4c00640 1 -- 172.21.15.97:0/611733421 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7efdf007acb0 con 0x7efdf815f230 2024-03-06T07:28:06.679 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:06.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.680+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efdd406b4d0 msgr2=0x7efdd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.680+0000 7efdfea00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efdd406b4d0 0x7efdd406d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efde80094c0 tx=0x7efde800b040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.680+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 msgr2=0x7efdf815fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.680+0000 7efdfea00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 0x7efdf815fbb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efdf002bee0 tx=0x7efdf00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 shutdown_connections 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efdd406b4d0 0x7efdd406d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdf815f230 0x7efdf815fbb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdf80b3b40 0x7efdf81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 --2- 172.21.15.97:0/611733421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdf806b6d0 0x7efdf80a95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 >> 172.21.15.97:0/611733421 conn(0x7efdf806c0d0 msgr2=0x7efdf806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 shutdown_connections 2024-03-06T07:28:06.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:06.681+0000 7efdfea00640 1 -- 172.21.15.97:0/611733421 wait complete. 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi135.front.sepia.ceph.com 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:06.703 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:06.704 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:06.704 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:28:06.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:06.704 DEBUG:teuthology.orchestra.run.smithi135:> ip addr 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:d9:36:76 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: altname enp7s0f0 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:d9:36:77 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout: altname enp7s0f1 2024-03-06T07:28:06.765 INFO:teuthology.orchestra.run.smithi135.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:88:80:82 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:88:80:83 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout: inet 172.21.15.135/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft 42574sec preferred_lft 42574sec 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::ec4:7aff:fe88:8083/64 scope link proto kernel_ll 2024-03-06T07:28:06.766 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:06.767 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:28:06.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip addr flush dev ceph-brx 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set ceph-brx up 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:28:06.767 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:07.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:07.295 DEBUG:teuthology.orchestra.run.smithi135:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:07.373 INFO:teuthology.orchestra.run.smithi135.stdout:1 2024-03-06T07:28:07.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:07.376 DEBUG:teuthology.orchestra.run.smithi135:> route 2024-03-06T07:28:07.423 INFO:teuthology.orchestra.run.smithi135.stdout:Kernel IP routing table 2024-03-06T07:28:07.423 INFO:teuthology.orchestra.run.smithi135.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:07.423 INFO:teuthology.orchestra.run.smithi135.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:07.423 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:07.423 INFO:teuthology.orchestra.run.smithi135.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:07.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:07.425 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:07.425 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:07.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:07.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:07.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:07.425 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:08.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:08.105 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-06T07:28:08.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:08.149 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list-id 2024-03-06T07:28:08.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:08.236 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:08.236 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:08.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:28:08.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:28:08.237 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:08.405 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:28:08.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:28:08.405 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:08.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:08.661 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:08.661 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:08.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set brx.0 up 2024-03-06T07:28:08.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:28:08.661 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:08.804 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:08.804 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:08.848 INFO:teuthology.orchestra.run.smithi135.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:28:08.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:08.849 DEBUG:teuthology.orchestra.run.smithi135:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:08.937 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-06T07:28:08.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:08.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:09.059 INFO:teuthology.orchestra.run.smithi135.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:09.059 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:09.709 INFO:teuthology.orchestra.run.smithi135.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:09.709 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:09.709 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:28:09.709 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: trying mount with new device syntax: 1@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:28:09.709 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:28:09.713 INFO:teuthology.orchestra.run.smithi135.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-06T07:28:09.713 INFO:teuthology.orchestra.run.smithi135.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:09.713 INFO:teuthology.orchestra.run.smithi135.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:09.713 INFO:teuthology.orchestra.run.smithi135.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:09.713 INFO:teuthology.orchestra.run.smithi135.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:09.718 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:09.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.812 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:09.868 INFO:teuthology.orchestra.run.smithi135.stdout:client4388 2024-03-06T07:28:09.868 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:09.869 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.917 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:10.017 INFO:teuthology.orchestra.run.smithi135.stdout:client4388 2024-03-06T07:28:10.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:10.018 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:10.073 INFO:teuthology.orchestra.run.smithi135.stdout:client4388 2024-03-06T07:28:10.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:10.074 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b9abceb-2115-4321-96a2-348db0d0fbe4.client4388/status') 2024-03-06T07:28:10.156 INFO:teuthology.orchestra.run.smithi135.stdout:instance: client.4388 (3)192.168.0.1:0/4269442331 2024-03-06T07:28:10.156 INFO:teuthology.orchestra.run.smithi135.stdout:blocklisted: false 2024-03-06T07:28:10.157 INFO:teuthology.orchestra.run.smithi135.stderr:0+1 records in 2024-03-06T07:28:10.157 INFO:teuthology.orchestra.run.smithi135.stderr:0+1 records out 2024-03-06T07:28:10.157 INFO:teuthology.orchestra.run.smithi135.stderr:69 bytes copied, 6.7261e-05 s, 1.0 MB/s 2024-03-06T07:28:10.160 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-06T07:28:10.160 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:10.161 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:10.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:10.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.374+0000 7ff948400640 1 Processor -- start 2024-03-06T07:28:10.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.374+0000 7ff948400640 1 -- start start 2024-03-06T07:28:10.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff948400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9400af1c0 0x7ff94015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff948400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff948400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff94006d1a0 0x7ff94006e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff948400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff940071d30 con 0x7ff94006d1a0 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff948400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff940071490 con 0x7ff9400af1c0 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff948400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff940071770 con 0x7ff940070d20 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff946400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff94006d1a0 0x7ff94006e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff946400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff94006d1a0 0x7ff94006e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52114/0 (socket says 172.21.15.97:52114) 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff946400640 1 -- 172.21.15.97:0/2368687511 learned_addr learned my addr 172.21.15.97:0/2368687511 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff945a00640 1 --2- 172.21.15.97:0/2368687511 >> v2:172.21.15.135:3300/0 conn(0x7ff9400af1c0 0x7ff94015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.376+0000 7ff945000640 1 --2- 172.21.15.97:0/2368687511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.377+0000 7ff946400640 1 -- 172.21.15.97:0/2368687511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 msgr2=0x7ff94006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.377+0000 7ff946400640 1 --2- 172.21.15.97:0/2368687511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.377+0000 7ff946400640 1 -- 172.21.15.97:0/2368687511 >> v2:172.21.15.135:3300/0 conn(0x7ff9400af1c0 msgr2=0x7ff94015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.377+0000 7ff946400640 1 --2- 172.21.15.97:0/2368687511 >> v2:172.21.15.135:3300/0 conn(0x7ff9400af1c0 0x7ff94015eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.377+0000 7ff946400640 1 -- 172.21.15.97:0/2368687511 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff940071a50 con 0x7ff94006d1a0 2024-03-06T07:28:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.378+0000 7ff946400640 1 --2- 172.21.15.97:0/2368687511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff94006d1a0 0x7ff94006e4d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff9400d0ac0 tx=0x7ff93802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa4b2687aea7c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.378+0000 7ff93fe00640 1 -- 172.21.15.97:0/2368687511 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938035020 con 0x7ff94006d1a0 2024-03-06T07:28:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.378+0000 7ff93fe00640 1 -- 172.21.15.97:0/2368687511 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff938031870 con 0x7ff94006d1a0 2024-03-06T07:28:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.378+0000 7ff93fe00640 1 -- 172.21.15.97:0/2368687511 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938031c40 con 0x7ff94006d1a0 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 -- 172.21.15.97:0/2368687511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff94006d1a0 msgr2=0x7ff94006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 --2- 172.21.15.97:0/2368687511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff94006d1a0 0x7ff94006e4d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff9400d0ac0 tx=0x7ff93802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 -- 172.21.15.97:0/2368687511 shutdown_connections 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 --2- 172.21.15.97:0/2368687511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff94006d1a0 0x7ff94006e4d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 --2- 172.21.15.97:0/2368687511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 --2- 172.21.15.97:0/2368687511 >> v2:172.21.15.135:3300/0 conn(0x7ff9400af1c0 0x7ff94015eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 -- 172.21.15.97:0/2368687511 >> 172.21.15.97:0/2368687511 conn(0x7ff9400b1730 msgr2=0x7ff9400b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 -- 172.21.15.97:0/2368687511 shutdown_connections 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.379+0000 7ff948400640 1 -- 172.21.15.97:0/2368687511 wait complete. 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.381+0000 7ff948400640 1 Processor -- start 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.381+0000 7ff948400640 1 -- start start 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff948400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff94006d1a0 0x7ff94015d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff948400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94014bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff948400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 0x7ff94014c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff948400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94006ea10 con 0x7ff9400af1c0 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff948400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9400711a0 con 0x7ff94006d1a0 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff948400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94006eb90 con 0x7ff940070d20 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff946400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 0x7ff94014c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff946400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 0x7ff94014c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52124/0 (socket says 172.21.15.97:52124) 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.382+0000 7ff946400640 1 -- 172.21.15.97:0/3472154144 learned_addr learned my addr 172.21.15.97:0/3472154144 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff945a00640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff94006d1a0 0x7ff94015d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff945000640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94014bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff946400640 1 -- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 msgr2=0x7ff94014bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff946400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94014bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff946400640 1 -- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff94006d1a0 msgr2=0x7ff94015d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff946400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff94006d1a0 0x7ff94015d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff946400640 1 -- 172.21.15.97:0/3472154144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94015c050 con 0x7ff9400af1c0 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff945a00640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff94006d1a0 0x7ff94015d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff946400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 0x7ff94014c500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff93802b950 tx=0x7ff9380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938040070 con 0x7ff9400af1c0 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff938030c50 con 0x7ff9400af1c0 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.383+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94015c330 con 0x7ff9400af1c0 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.384+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938035040 con 0x7ff9400af1c0 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.384+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94015c8f0 con 0x7ff9400af1c0 2024-03-06T07:28:10.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.385+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94004f330 con 0x7ff9400af1c0 2024-03-06T07:28:10.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.386+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff938038020 con 0x7ff9400af1c0 2024-03-06T07:28:10.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.387+0000 7ff93d600640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff92006b4d0 0x7ff92006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.387+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9380abfa0 con 0x7ff9400af1c0 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.387+0000 7ff945a00640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff92006b4d0 0x7ff92006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.387+0000 7ff945a00640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff92006b4d0 0x7ff92006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff934009ed0 tx=0x7ff93400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.391+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff93807aeb0 con 0x7ff9400af1c0 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.627+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff94015ae20 con 0x7ff9400af1c0 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.627+0000 7ff93d600640 1 -- 172.21.15.97:0/3472154144 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff93807a7b0 con 0x7ff9400af1c0 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:10.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.630+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff92006b4d0 msgr2=0x7ff92006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.630+0000 7ff948400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff92006b4d0 0x7ff92006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff934009ed0 tx=0x7ff93400b040 comp rx=0 tx=0).stop 2024-03-06T07:28:10.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.630+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 msgr2=0x7ff94014c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.630+0000 7ff948400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 0x7ff94014c500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff93802b950 tx=0x7ff9380046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 shutdown_connections 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff92006b4d0 0x7ff92006d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9400af1c0 0x7ff94014c500 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff940070d20 0x7ff94014bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 --2- 172.21.15.97:0/3472154144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff94006d1a0 0x7ff94015d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 >> 172.21.15.97:0/3472154144 conn(0x7ff9400b1730 msgr2=0x7ff9400721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 shutdown_connections 2024-03-06T07:28:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:10.631+0000 7ff948400640 1 -- 172.21.15.97:0/3472154144 wait complete. 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi135.front.sepia.ceph.com 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:10.659 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:28:10.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:10.659 DEBUG:teuthology.orchestra.run.smithi135:> ip addr 2024-03-06T07:28:10.718 INFO:teuthology.orchestra.run.smithi135.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:10.718 INFO:teuthology.orchestra.run.smithi135.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:d9:36:76 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: altname enp7s0f0 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:d9:36:77 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: altname enp7s0f1 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:88:80:82 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:10.719 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:88:80:83 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: inet 172.21.15.135/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft 42570sec preferred_lft 42570sec 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::ec4:7aff:fe88:8083/64 scope link proto kernel_ll 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 42:d6:23:3a:6c:33 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::6872:88ff:fe90:8bb8/64 scope link proto kernel_ll 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 42:d6:23:3a:6c:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::40d6:23ff:fe3a:6c33/64 scope link proto kernel_ll 2024-03-06T07:28:10.720 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:10.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:10.721 DEBUG:teuthology.orchestra.run.smithi135:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:10.805 INFO:teuthology.orchestra.run.smithi135.stdout:1 2024-03-06T07:28:10.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:10.810 DEBUG:teuthology.orchestra.run.smithi135:> route 2024-03-06T07:28:10.863 INFO:teuthology.orchestra.run.smithi135.stdout:Kernel IP routing table 2024-03-06T07:28:10.863 INFO:teuthology.orchestra.run.smithi135.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:10.863 INFO:teuthology.orchestra.run.smithi135.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:10.863 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:10.863 INFO:teuthology.orchestra.run.smithi135.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:10.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:10.863 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:10.863 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:10.863 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:10.863 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:10.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:10.864 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:11.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.041 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-06T07:28:11.090 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:28:11.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.091 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list-id 2024-03-06T07:28:11.140 INFO:teuthology.orchestra.run.smithi135.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:28:11.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.141 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:11.141 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:11.141 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:28:11.141 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:28:11.141 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:28:11.363 INFO:teuthology.orchestra.run.smithi135.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:11.363 INFO:teuthology.orchestra.run.smithi135.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:11.363 INFO:teuthology.orchestra.run.smithi135.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:11.363 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether f6:ea:e9:d7:32:f6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::f4ea:e9ff:fed7:32f6/64 scope link proto kernel_ll 2024-03-06T07:28:11.364 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.368 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether f6:ea:e9:d7:32:f6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::f4ea:e9ff:fed7:32f6/64 scope link proto kernel_ll 2024-03-06T07:28:11.457 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:11.460 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:28:11.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:28:11.460 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:28:11.461 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:11.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.741 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-06T07:28:11.741 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-06T07:28:11.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set brx.1 up 2024-03-06T07:28:11.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:28:11.741 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-06T07:28:11.899 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:11.899 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:11.947 INFO:teuthology.orchestra.run.smithi135.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:28:11.947 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:11.947 DEBUG:teuthology.orchestra.run.smithi135:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:11.990 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-06T07:28:11.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:11.990 DEBUG:teuthology.orchestra.run.smithi135:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:12.171 INFO:teuthology.orchestra.run.smithi135.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:12.171 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:12.204 INFO:teuthology.orchestra.run.smithi135.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:12.204 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:12.204 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:28:12.204 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: trying mount with new device syntax: 3@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:28:12.204 INFO:teuthology.orchestra.run.smithi135.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:28:12.206 INFO:teuthology.orchestra.run.smithi135.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-06T07:28:12.206 INFO:teuthology.orchestra.run.smithi135.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:12.206 INFO:teuthology.orchestra.run.smithi135.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:12.206 INFO:teuthology.orchestra.run.smithi135.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:12.206 INFO:teuthology.orchestra.run.smithi135.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:12.211 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:12.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:12.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.290 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:12.338 INFO:teuthology.orchestra.run.smithi135.stdout:client4498 2024-03-06T07:28:12.338 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:12.339 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:12.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.434 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:12.489 INFO:teuthology.orchestra.run.smithi135.stdout:client4498 2024-03-06T07:28:12.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.489 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:12.545 INFO:teuthology.orchestra.run.smithi135.stdout:client4498 2024-03-06T07:28:12.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.546 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b9abceb-2115-4321-96a2-348db0d0fbe4.client4498/status') 2024-03-06T07:28:12.635 INFO:teuthology.orchestra.run.smithi135.stdout:instance: client.4498 (3)192.168.0.2:0/1575955023 2024-03-06T07:28:12.635 INFO:teuthology.orchestra.run.smithi135.stdout:blocklisted: false 2024-03-06T07:28:12.635 INFO:teuthology.orchestra.run.smithi135.stderr:0+1 records in 2024-03-06T07:28:12.636 INFO:teuthology.orchestra.run.smithi135.stderr:0+1 records out 2024-03-06T07:28:12.636 INFO:teuthology.orchestra.run.smithi135.stderr:69 bytes copied, 0.000143984 s, 479 kB/s 2024-03-06T07:28:12.639 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:28:12.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.866+0000 7f5010800640 1 Processor -- start 2024-03-06T07:28:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.867+0000 7f5010800640 1 -- start start 2024-03-06T07:28:12.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f5010800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f500815f770 0x7f500815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f5010800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f5008160290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f5010800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f50081607d0 0x7f50080ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f5010800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500815fb50 con 0x7f500815f770 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f5010800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50080b3040 con 0x7f50081607d0 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f5010800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50080cd3e0 con 0x7f50080b4390 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.868+0000 7f500de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f500815f770 0x7f500815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f500815f770 0x7f500815fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52136/0 (socket says 172.21.15.97:52136) 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 -- 172.21.15.97:0/1032360564 learned_addr learned my addr 172.21.15.97:0/1032360564 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500e800640 1 --2- 172.21.15.97:0/1032360564 >> v2:172.21.15.135:3300/0 conn(0x7f50081607d0 0x7f50080ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500d400640 1 --2- 172.21.15.97:0/1032360564 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f5008160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 -- 172.21.15.97:0/1032360564 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 msgr2=0x7f5008160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 --2- 172.21.15.97:0/1032360564 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f5008160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 -- 172.21.15.97:0/1032360564 >> v2:172.21.15.135:3300/0 conn(0x7f50081607d0 msgr2=0x7f50080ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 --2- 172.21.15.97:0/1032360564 >> v2:172.21.15.135:3300/0 conn(0x7f50081607d0 0x7f50080ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.869+0000 7f500de00640 1 -- 172.21.15.97:0/1032360564 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50080cd6c0 con 0x7f500815f770 2024-03-06T07:28:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.871+0000 7f500de00640 1 --2- 172.21.15.97:0/1032360564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f500815fd50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5008072ab0 tx=0x7f4ffc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b80b8e9b1d74aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.871+0000 7f500ca00640 1 -- 172.21.15.97:0/1032360564 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc02e070 con 0x7f500815f770 2024-03-06T07:28:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.871+0000 7f500ca00640 1 -- 172.21.15.97:0/1032360564 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc037070 con 0x7f500815f770 2024-03-06T07:28:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.871+0000 7f500ca00640 1 -- 172.21.15.97:0/1032360564 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc031a00 con 0x7f500815f770 2024-03-06T07:28:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 -- 172.21.15.97:0/1032360564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 msgr2=0x7f500815fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 --2- 172.21.15.97:0/1032360564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f500815fd50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5008072ab0 tx=0x7f4ffc02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 -- 172.21.15.97:0/1032360564 shutdown_connections 2024-03-06T07:28:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 --2- 172.21.15.97:0/1032360564 >> v2:172.21.15.135:3300/0 conn(0x7f50081607d0 0x7f50080ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 --2- 172.21.15.97:0/1032360564 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f5008160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 --2- 172.21.15.97:0/1032360564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f500815fd50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 -- 172.21.15.97:0/1032360564 >> 172.21.15.97:0/1032360564 conn(0x7f50080b3870 msgr2=0x7f50080b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 -- 172.21.15.97:0/1032360564 shutdown_connections 2024-03-06T07:28:12.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.872+0000 7f5010800640 1 -- 172.21.15.97:0/1032360564 wait complete. 2024-03-06T07:28:12.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.874+0000 7f5010800640 1 Processor -- start 2024-03-06T07:28:12.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.874+0000 7f5010800640 1 -- start start 2024-03-06T07:28:12.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f5010800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f500814ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f5010800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f50080b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f5010800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50081607d0 0x7f50080b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f5010800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50080b3320 con 0x7f500815f770 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f5010800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5008160be0 con 0x7f50081607d0 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f5010800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50080b34a0 con 0x7f50080b4390 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f50080b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f50080b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52146/0 (socket says 172.21.15.97:52146) 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 -- 172.21.15.97:0/3827284702 learned_addr learned my addr 172.21.15.97:0/3827284702 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500de00640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f500814ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 -- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 msgr2=0x7f500814ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500e800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50081607d0 0x7f50080b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f500814ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 -- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50081607d0 msgr2=0x7f50080b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50081607d0 0x7f50080b72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.875+0000 7f500d400640 1 -- 172.21.15.97:0/3827284702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500814b380 con 0x7f500815f770 2024-03-06T07:28:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.876+0000 7f500d400640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f50080b6d60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800d7d0 tx=0x7f4ff800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.876+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8018020 con 0x7f500815f770 2024-03-06T07:28:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.876+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff80096a0 con 0x7f500815f770 2024-03-06T07:28:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.876+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8009a90 con 0x7f500815f770 2024-03-06T07:28:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.876+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500814f670 con 0x7f500815f770 2024-03-06T07:28:12.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.876+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500814fc30 con 0x7f500815f770 2024-03-06T07:28:12.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.877+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500804f330 con 0x7f500815f770 2024-03-06T07:28:12.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.879+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ff801b020 con 0x7f500815f770 2024-03-06T07:28:12.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.880+0000 7f5002000640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fec06b4d0 0x7f4fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.880+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ff808a6b0 con 0x7f500815f770 2024-03-06T07:28:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.880+0000 7f500de00640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fec06b4d0 0x7f4fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.881+0000 7f500de00640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fec06b4d0 0x7f4fec06d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc036000 tx=0x7f4ffc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:12.887+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ff8022020 con 0x7f500815f770 2024-03-06T07:28:13.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.142+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50080ce130 con 0x7f500815f770 2024-03-06T07:28:13.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.144+0000 7f5002000640 1 -- 172.21.15.97:0/3827284702 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f4ff80595c0 con 0x7f500815f770 2024-03-06T07:28:13.147 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:13.147 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:13.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fec06b4d0 msgr2=0x7f4fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fec06b4d0 0x7f4fec06d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc036000 tx=0x7f4ffc008030 comp rx=0 tx=0).stop 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 msgr2=0x7f50080b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f50080b6d60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800d7d0 tx=0x7f4ff800dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 shutdown_connections 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fec06b4d0 0x7f4fec06d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50081607d0 0x7f50080b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f500815f770 0x7f50080b6d60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 --2- 172.21.15.97:0/3827284702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50080b4390 0x7f500814ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 >> 172.21.15.97:0/3827284702 conn(0x7f50080b3870 msgr2=0x7f50080cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 shutdown_connections 2024-03-06T07:28:13.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.147+0000 7f5010800640 1 -- 172.21.15.97:0/3827284702 wait complete. 2024-03-06T07:28:13.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:13.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:13.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.374+0000 7fab3d000640 1 Processor -- start 2024-03-06T07:28:13.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.374+0000 7fab3d000640 1 -- start start 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab3d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fab380b3f70 0x7fab38160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab3d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fab38070ff0 0x7fab380ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab3d000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab38071440 con 0x7fab3815fc10 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab3d000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380b46c0 con 0x7fab38070ff0 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab3d000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380b49a0 con 0x7fab380b3f70 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fab38070ff0 0x7fab380ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fab38070ff0 0x7fab380ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52158/0 (socket says 172.21.15.97:52158) 2024-03-06T07:28:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 -- 172.21.15.97:0/4263910315 learned_addr learned my addr 172.21.15.97:0/4263910315 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:13.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 -- 172.21.15.97:0/4263910315 >> v2:172.21.15.135:3300/0 conn(0x7fab380b3f70 msgr2=0x7fab38160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 --2- 172.21.15.97:0/4263910315 >> v2:172.21.15.135:3300/0 conn(0x7fab380b3f70 0x7fab38160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 -- 172.21.15.97:0/4263910315 >> v1:172.21.15.135:6789/0 conn(0x7fab3815fc10 legacy=0x7fab3815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.376+0000 7fab36800640 1 -- 172.21.15.97:0/4263910315 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab38072c90 con 0x7fab38070ff0 2024-03-06T07:28:13.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.377+0000 7fab36800640 1 --2- 172.21.15.97:0/4263910315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab38070ff0 0x7fab380ce4a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab38074a50 tx=0x7fab3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc7cb79b8e25a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.378+0000 7fab34a00640 1 -- 172.21.15.97:0/4263910315 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3002ca80 con 0x7fab38070ff0 2024-03-06T07:28:13.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.378+0000 7fab34a00640 1 -- 172.21.15.97:0/4263910315 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3002cc20 con 0x7fab38070ff0 2024-03-06T07:28:13.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.378+0000 7fab34a00640 1 -- 172.21.15.97:0/4263910315 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab300343f0 con 0x7fab38070ff0 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 -- 172.21.15.97:0/4263910315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab38070ff0 msgr2=0x7fab380ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 --2- 172.21.15.97:0/4263910315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab38070ff0 0x7fab380ce4a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab38074a50 tx=0x7fab3002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 -- 172.21.15.97:0/4263910315 shutdown_connections 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 --2- 172.21.15.97:0/4263910315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab38070ff0 0x7fab380ce4a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 --2- 172.21.15.97:0/4263910315 >> v2:172.21.15.135:3300/0 conn(0x7fab380b3f70 0x7fab38160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 -- 172.21.15.97:0/4263910315 >> 172.21.15.97:0/4263910315 conn(0x7fab380b1940 msgr2=0x7fab380b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 -- 172.21.15.97:0/4263910315 shutdown_connections 2024-03-06T07:28:13.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.379+0000 7fab3d000640 1 -- 172.21.15.97:0/4263910315 wait complete. 2024-03-06T07:28:13.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.381+0000 7fab3d000640 1 Processor -- start 2024-03-06T07:28:13.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.381+0000 7fab3d000640 1 -- start start 2024-03-06T07:28:13.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.382+0000 7fab3d000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab38070ff0 0x7fab381445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.382+0000 7fab3d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab380b3f70 0x7fab3815d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.382+0000 7fab3d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 0x7fab3809be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.382+0000 7fab3d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab38072a30 con 0x7fab3815fc10 2024-03-06T07:28:13.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.382+0000 7fab3d000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab380728b0 con 0x7fab38070ff0 2024-03-06T07:28:13.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.382+0000 7fab3d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3815c2e0 con 0x7fab380b3f70 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 0x7fab3809be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 0x7fab3809be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52172/0 (socket says 172.21.15.97:52172) 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 -- 172.21.15.97:0/235116515 learned_addr learned my addr 172.21.15.97:0/235116515 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab35e00640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab38070ff0 0x7fab381445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab35400640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab380b3f70 0x7fab3815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 -- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab380b3f70 msgr2=0x7fab3815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab380b3f70 0x7fab3815d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 -- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab38070ff0 msgr2=0x7fab381445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab38070ff0 0x7fab381445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.383+0000 7fab36800640 1 -- 172.21.15.97:0/235116515 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab38154520 con 0x7fab3815fc10 2024-03-06T07:28:13.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.384+0000 7fab36800640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 0x7fab3809be80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab30036040 tx=0x7fab30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.384+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30043020 con 0x7fab3815fc10 2024-03-06T07:28:13.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.384+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3003e450 con 0x7fab3815fc10 2024-03-06T07:28:13.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.384+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab38154800 con 0x7fab3815fc10 2024-03-06T07:28:13.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.384+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3003e820 con 0x7fab3815fc10 2024-03-06T07:28:13.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.384+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab38154dc0 con 0x7fab3815fc10 2024-03-06T07:28:13.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.385+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3804f330 con 0x7fab3815fc10 2024-03-06T07:28:13.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.386+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab30046020 con 0x7fab3815fc10 2024-03-06T07:28:13.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.387+0000 7fab2e000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab1806b4d0 0x7fab1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.387+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab300acbe0 con 0x7fab3815fc10 2024-03-06T07:28:13.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.387+0000 7fab35e00640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab1806b4d0 0x7fab1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.388+0000 7fab35e00640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab1806b4d0 0x7fab1806d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fab28006fd0 tx=0x7fab28008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.391+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab3004d030 con 0x7fab3815fc10 2024-03-06T07:28:13.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.595+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab380b2d40 con 0x7fab3815fc10 2024-03-06T07:28:13.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.596+0000 7fab2e000640 1 -- 172.21.15.97:0/235116515 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab3007ba60 con 0x7fab3815fc10 2024-03-06T07:28:13.598 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab1806b4d0 msgr2=0x7fab1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab1806b4d0 0x7fab1806d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fab28006fd0 tx=0x7fab28008040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 msgr2=0x7fab3809be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 0x7fab3809be80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab30036040 tx=0x7fab30004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 shutdown_connections 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab1806b4d0 0x7fab1806d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab3815fc10 0x7fab3809be80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab380b3f70 0x7fab3815d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 --2- 172.21.15.97:0/235116515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab38070ff0 0x7fab381445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 >> 172.21.15.97:0/235116515 conn(0x7fab380b1940 msgr2=0x7fab380acfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 shutdown_connections 2024-03-06T07:28:13.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.598+0000 7fab3d000640 1 -- 172.21.15.97:0/235116515 wait complete. 2024-03-06T07:28:13.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.822+0000 7f07f1200640 1 Processor -- start 2024-03-06T07:28:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.822+0000 7f07f1200640 1 -- start start 2024-03-06T07:28:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07f1200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f07ec15fc10 0x7f07ec070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07f1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f07ec0b3f70 0x7f07ec0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07f1200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ec0b46c0 con 0x7f07ec15fff0 2024-03-06T07:28:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07f1200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec0b49a0 con 0x7f07ec15fc10 2024-03-06T07:28:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07f1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec072c90 con 0x7f07ec0b3f70 2024-03-06T07:28:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f07ec15fc10 0x7f07ec070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f07ec15fc10 0x7f07ec070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52174/0 (socket says 172.21.15.97:52174) 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 -- 172.21.15.97:0/2128840387 learned_addr learned my addr 172.21.15.97:0/2128840387 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 -- 172.21.15.97:0/2128840387 >> v2:172.21.15.135:3300/0 conn(0x7f07ec0b3f70 msgr2=0x7f07ec0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 --2- 172.21.15.97:0/2128840387 >> v2:172.21.15.135:3300/0 conn(0x7f07ec0b3f70 0x7f07ec0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 -- 172.21.15.97:0/2128840387 >> v1:172.21.15.135:6789/0 conn(0x7f07ec15fff0 legacy=0x7f07ec160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.824+0000 7f07e9e00640 1 -- 172.21.15.97:0/2128840387 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec09dd80 con 0x7f07ec15fc10 2024-03-06T07:28:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.826+0000 7f07e9e00640 1 --2- 172.21.15.97:0/2128840387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec070ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0cf6a0 tx=0x7f07d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27ebcae1332ec4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.826+0000 7f07e8a00640 1 -- 172.21.15.97:0/2128840387 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4035020 con 0x7f07ec15fc10 2024-03-06T07:28:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.826+0000 7f07e8a00640 1 -- 172.21.15.97:0/2128840387 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d4031870 con 0x7f07ec15fc10 2024-03-06T07:28:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.826+0000 7f07e8a00640 1 -- 172.21.15.97:0/2128840387 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4031c40 con 0x7f07ec15fc10 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 -- 172.21.15.97:0/2128840387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 msgr2=0x7f07ec070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 --2- 172.21.15.97:0/2128840387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec070ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0cf6a0 tx=0x7f07d402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 -- 172.21.15.97:0/2128840387 shutdown_connections 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 --2- 172.21.15.97:0/2128840387 >> v2:172.21.15.135:3300/0 conn(0x7f07ec0b3f70 0x7f07ec0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 --2- 172.21.15.97:0/2128840387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec070ff0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 -- 172.21.15.97:0/2128840387 >> 172.21.15.97:0/2128840387 conn(0x7f07ec0b1940 msgr2=0x7f07ec0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 -- 172.21.15.97:0/2128840387 shutdown_connections 2024-03-06T07:28:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.827+0000 7f07f1200640 1 -- 172.21.15.97:0/2128840387 wait complete. 2024-03-06T07:28:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.828+0000 7f07f1200640 1 Processor -- start 2024-03-06T07:28:13.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.829+0000 7f07f1200640 1 -- start start 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.829+0000 7f07f1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec0b3f70 0x7f07ec0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07f1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07f1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f07ec15fff0 0x7f07ec0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07f1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec15c460 con 0x7f07ec15fc10 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07f1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec15c2e0 con 0x7f07ec0b3f70 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07f1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0ce9e0 con 0x7f07ec15fff0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec0caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39016/0 (socket says 172.21.15.97:39016) 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 -- 172.21.15.97:0/401266764 learned_addr learned my addr 172.21.15.97:0/401266764 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 -- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f07ec15fff0 msgr2=0x7f07ec0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9e00640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec0b3f70 0x7f07ec0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07ea800640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f07ec15fff0 0x7f07ec0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f07ec15fff0 0x7f07ec0cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 -- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec0b3f70 msgr2=0x7f07ec0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec0b3f70 0x7f07ec0c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9400640 1 -- 172.21.15.97:0/401266764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec0cb420 con 0x7f07ec15fc10 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.830+0000 7f07e9e00640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec0b3f70 0x7f07ec0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.831+0000 7f07e9400640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec0caaf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00b3e0 tx=0x7f07dc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.831+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc015020 con 0x7f07ec15fc10 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.831+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07dc011850 con 0x7f07ec15fc10 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.831+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec0cb700 con 0x7f07ec15fc10 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.831+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc011c40 con 0x7f07ec15fc10 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.831+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec0cbcc0 con 0x7f07ec15fc10 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.832+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec0721b0 con 0x7f07ec15fc10 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.834+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07dc018020 con 0x7f07ec15fc10 2024-03-06T07:28:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.835+0000 7f07e2000640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07c806b4d0 0x7f07c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.835+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07dc08a180 con 0x7f07ec15fc10 2024-03-06T07:28:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.835+0000 7f07e9e00640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07c806b4d0 0x7f07c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.836+0000 7f07e9e00640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07c806b4d0 0x7f07c806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07d4036000 tx=0x7f07d403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:13.840+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07ec0721b0 con 0x7f07ec15fc10 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.123+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07ec04f330 con 0x7f07ec15fc10 2024-03-06T07:28:14.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.124+0000 7f07e2000640 1 -- 172.21.15.97:0/401266764 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07dc01f020 con 0x7f07ec15fc10 2024-03-06T07:28:14.126 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:14.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07c806b4d0 msgr2=0x7f07c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07c806b4d0 0x7f07c806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07d4036000 tx=0x7f07d403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:14.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 msgr2=0x7f07ec0caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec0caaf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00b3e0 tx=0x7f07dc00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 shutdown_connections 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07c806b4d0 0x7f07c806d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f07ec15fff0 0x7f07ec0cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07ec15fc10 0x7f07ec0caaf0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 --2- 172.21.15.97:0/401266764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec0b3f70 0x7f07ec0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 >> 172.21.15.97:0/401266764 conn(0x7f07ec0b1940 msgr2=0x7f07ec0ade70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 shutdown_connections 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.127+0000 7f07f1200640 1 -- 172.21.15.97:0/401266764 wait complete. 2024-03-06T07:28:14.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:14.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.347+0000 7ff0b4c00640 1 Processor -- start 2024-03-06T07:28:14.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.347+0000 7ff0b4c00640 1 -- start start 2024-03-06T07:28:14.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.348+0000 7ff0b4c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff0b015be10 0x7ff0b00ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.348+0000 7ff0b4c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b006da40 0x7ff0b00ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.348+0000 7ff0b4c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b015c1f0 con 0x7ff0b00aef20 2024-03-06T07:28:14.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0b4c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b00cd200 con 0x7ff0b015be10 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0b4c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b006cac0 con 0x7ff0b006da40 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff0b015be10 0x7ff0b00ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff0b015be10 0x7ff0b00ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39022/0 (socket says 172.21.15.97:39022) 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 -- 172.21.15.97:0/4131248904 learned_addr learned my addr 172.21.15.97:0/4131248904 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0aea00640 1 --1- 172.21.15.97:0/4131248904 >> v1:172.21.15.97:6789/0 conn(0x7ff0b00aef20 0x7ff0b006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34856/0 (socket says 172.21.15.97:34856) 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0afe00640 1 --2- 172.21.15.97:0/4131248904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b006da40 0x7ff0b00ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 -- 172.21.15.97:0/4131248904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b006da40 msgr2=0x7ff0b00ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 --2- 172.21.15.97:0/4131248904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b006da40 0x7ff0b00ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 -- 172.21.15.97:0/4131248904 >> v1:172.21.15.97:6789/0 conn(0x7ff0b00aef20 legacy=0x7ff0b006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.349+0000 7ff0af400640 1 -- 172.21.15.97:0/4131248904 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b006cda0 con 0x7ff0b015be10 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.350+0000 7ff0af400640 1 --2- 172.21.15.97:0/4131248904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b00ccc20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009e50 tx=0x7ff0a002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b725d9844f4276db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0ae000640 1 -- 172.21.15.97:0/4131248904 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a002caf0 con 0x7ff0b015be10 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0ae000640 1 -- 172.21.15.97:0/4131248904 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a002cc90 con 0x7ff0b015be10 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0ae000640 1 -- 172.21.15.97:0/4131248904 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0034610 con 0x7ff0b015be10 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0b4c00640 1 -- 172.21.15.97:0/4131248904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 msgr2=0x7ff0b00ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/4131248904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b00ccc20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009e50 tx=0x7ff0a002bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0b4c00640 1 -- 172.21.15.97:0/4131248904 shutdown_connections 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/4131248904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b006da40 0x7ff0b00ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/4131248904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b00ccc20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.351+0000 7ff0b4c00640 1 -- 172.21.15.97:0/4131248904 >> 172.21.15.97:0/4131248904 conn(0x7ff0b006afc0 msgr2=0x7ff0b006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.352+0000 7ff0b4c00640 1 -- 172.21.15.97:0/4131248904 shutdown_connections 2024-03-06T07:28:14.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.352+0000 7ff0b4c00640 1 -- 172.21.15.97:0/4131248904 wait complete. 2024-03-06T07:28:14.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.353+0000 7ff0b4c00640 1 Processor -- start 2024-03-06T07:28:14.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.353+0000 7ff0b4c00640 1 -- start start 2024-03-06T07:28:14.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.354+0000 7ff0b4c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0b006da40 0x7ff0b01f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.354+0000 7ff0b4c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b00aef20 0x7ff0b014e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.354+0000 7ff0b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b01586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.354+0000 7ff0b4c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00cda90 con 0x7ff0b015be10 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.354+0000 7ff0b4c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b006d080 con 0x7ff0b006da40 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.354+0000 7ff0b4c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0071ab0 con 0x7ff0b00aef20 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b01586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b01586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39024/0 (socket says 172.21.15.97:39024) 2024-03-06T07:28:14.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 -- 172.21.15.97:0/279076930 learned_addr learned my addr 172.21.15.97:0/279076930 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:14.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0aea00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b00aef20 0x7ff0b014e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 -- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b00aef20 msgr2=0x7ff0b014e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b00aef20 0x7ff0b014e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 -- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0b006da40 msgr2=0x7ff0b01f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0af400640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0b006da40 0x7ff0b01f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0b006da40 0x7ff0b01f7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 -- 172.21.15.97:0/279076930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b01f83d0 con 0x7ff0b015be10 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0aea00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b00aef20 0x7ff0b014e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.355+0000 7ff0afe00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b01586f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800d5d0 tx=0x7ff0a800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.356+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a80125f0 con 0x7ff0b015be10 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.356+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8019070 con 0x7ff0b015be10 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.356+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a80129d0 con 0x7ff0b015be10 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.356+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b01583c0 con 0x7ff0b015be10 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.356+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b0157e00 con 0x7ff0b015be10 2024-03-06T07:28:14.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.357+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b004f330 con 0x7ff0b015be10 2024-03-06T07:28:14.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.358+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0a8012b70 con 0x7ff0b015be10 2024-03-06T07:28:14.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.359+0000 7ff0a7400640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.359+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0a808a630 con 0x7ff0b015be10 2024-03-06T07:28:14.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.359+0000 7ff0af400640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.360+0000 7ff0af400640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff08c06b4d0 0x7ff08c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0035000 tx=0x7ff0a0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.363+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0b01f83d0 con 0x7ff0b015be10 2024-03-06T07:28:14.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.557+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0b00cecf0 con 0x7ff0b015be10 2024-03-06T07:28:14.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.558+0000 7ff0a7400640 1 -- 172.21.15.97:0/279076930 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0a80594a0 con 0x7ff0b015be10 2024-03-06T07:28:14.563 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:14.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.560+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff08c06b4d0 msgr2=0x7ff08c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.560+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff08c06b4d0 0x7ff08c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0035000 tx=0x7ff0a0030000 comp rx=0 tx=0).stop 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.560+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 msgr2=0x7ff0b01586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.560+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b01586f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800d5d0 tx=0x7ff0a800da90 comp rx=0 tx=0).stop 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 shutdown_connections 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0b015be10 0x7ff0b01586f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0b00aef20 0x7ff0b014e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 --2- 172.21.15.97:0/279076930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0b006da40 0x7ff0b01f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 >> 172.21.15.97:0/279076930 conn(0x7ff0b006afc0 msgr2=0x7ff0b00cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 shutdown_connections 2024-03-06T07:28:14.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.561+0000 7ff0b4c00640 1 -- 172.21.15.97:0/279076930 wait complete. 2024-03-06T07:28:14.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:14.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.781+0000 7f30f5200640 1 Processor -- start 2024-03-06T07:28:14.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.781+0000 7f30f5200640 1 -- start start 2024-03-06T07:28:14.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.782+0000 7f30f5200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f30f015f9d0 0x7f30f015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.782+0000 7f30f5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 0x7f30f0070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.782+0000 7f30f5200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f006cef0 con 0x7f30f0144c00 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.782+0000 7f30f5200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f006d1d0 con 0x7f30f015f9d0 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.782+0000 7f30f5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f006d4b0 con 0x7f30f0070b10 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.783+0000 7f30ed400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f30f015f9d0 0x7f30f015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.783+0000 7f30ee800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 0x7f30f0070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.783+0000 7f30ee800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 0x7f30f0070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:57112/0 (socket says 172.21.15.97:57112) 2024-03-06T07:28:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.783+0000 7f30ee800640 1 -- 172.21.15.97:0/1774245303 learned_addr learned my addr 172.21.15.97:0/1774245303 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:14.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.784+0000 7f30ee800640 1 -- 172.21.15.97:0/1774245303 >> v2:172.21.15.135:3300/0 conn(0x7f30f015f9d0 msgr2=0x7f30f015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.784+0000 7f30ee800640 1 --2- 172.21.15.97:0/1774245303 >> v2:172.21.15.135:3300/0 conn(0x7f30f015f9d0 0x7f30f015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.784+0000 7f30ee800640 1 -- 172.21.15.97:0/1774245303 >> v1:172.21.15.135:6789/0 conn(0x7f30f0144c00 legacy=0x7f30f006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.784+0000 7f30ee800640 1 -- 172.21.15.97:0/1774245303 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f006d790 con 0x7f30f0070b10 2024-03-06T07:28:14.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.784+0000 7f30ed400640 1 --2- 172.21.15.97:0/1774245303 >> v2:172.21.15.135:3300/0 conn(0x7f30f015f9d0 0x7f30f015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:14.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.785+0000 7f30ee800640 1 --2- 172.21.15.97:0/1774245303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 0x7f30f0070f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009e50 tx=0x7f30e802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872483153fa3f93c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.786+0000 7f30eca00640 1 -- 172.21.15.97:0/1774245303 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e802cb00 con 0x7f30f0070b10 2024-03-06T07:28:14.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.786+0000 7f30eca00640 1 -- 172.21.15.97:0/1774245303 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e802cca0 con 0x7f30f0070b10 2024-03-06T07:28:14.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.786+0000 7f30eca00640 1 -- 172.21.15.97:0/1774245303 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8034610 con 0x7f30f0070b10 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 -- 172.21.15.97:0/1774245303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 msgr2=0x7f30f0070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 --2- 172.21.15.97:0/1774245303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 0x7f30f0070f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009e50 tx=0x7f30e802bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 -- 172.21.15.97:0/1774245303 shutdown_connections 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 --2- 172.21.15.97:0/1774245303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f0070b10 0x7f30f0070f40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 --2- 172.21.15.97:0/1774245303 >> v2:172.21.15.135:3300/0 conn(0x7f30f015f9d0 0x7f30f015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 -- 172.21.15.97:0/1774245303 >> 172.21.15.97:0/1774245303 conn(0x7f30f00b1730 msgr2=0x7f30f00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 -- 172.21.15.97:0/1774245303 shutdown_connections 2024-03-06T07:28:14.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.787+0000 7f30f5200640 1 -- 172.21.15.97:0/1774245303 wait complete. 2024-03-06T07:28:14.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.790+0000 7f30f5200640 1 Processor -- start 2024-03-06T07:28:14.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.790+0000 7f30f5200640 1 -- start start 2024-03-06T07:28:14.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.792+0000 7f30f5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 0x7f30f00b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30f5200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f0144c00 0x7f30f013c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30f5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f015f9d0 0x7f30f00b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30f5200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f006ccd0 con 0x7f30f0070b10 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30f5200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f006cb50 con 0x7f30f0144c00 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30f5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f006df90 con 0x7f30f015f9d0 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 0x7f30f00b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 0x7f30f00b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39030/0 (socket says 172.21.15.97:39030) 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 -- 172.21.15.97:0/532285702 learned_addr learned my addr 172.21.15.97:0/532285702 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 -- 172.21.15.97:0/532285702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f015f9d0 msgr2=0x7f30f00b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f015f9d0 0x7f30f00b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 -- 172.21.15.97:0/532285702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f0144c00 msgr2=0x7f30f013c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f0144c00 0x7f30f013c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.793+0000 7f30ede00640 1 -- 172.21.15.97:0/532285702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00b3b90 con 0x7f30f0070b10 2024-03-06T07:28:14.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.794+0000 7f30ede00640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 0x7f30f00b3540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c400 tx=0x7f30e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.794+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e00052d0 con 0x7f30f0070b10 2024-03-06T07:28:14.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.794+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e0005490 con 0x7f30f0070b10 2024-03-06T07:28:14.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.794+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0007260 con 0x7f30f0070b10 2024-03-06T07:28:14.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.794+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f00b3e70 con 0x7f30f0070b10 2024-03-06T07:28:14.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.794+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f00b4430 con 0x7f30f0070b10 2024-03-06T07:28:14.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.795+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f004f330 con 0x7f30f0070b10 2024-03-06T07:28:14.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.796+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30e0007400 con 0x7f30f0070b10 2024-03-06T07:28:14.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.798+0000 7f30e6000640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d006b4d0 0x7f30d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.798+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30e00897a0 con 0x7f30f0070b10 2024-03-06T07:28:14.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.798+0000 7f30ed400640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d006b4d0 0x7f30d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.799+0000 7f30ed400640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d006b4d0 0x7f30d006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30dc006e80 tx=0x7f30dc006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:14.802+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30e0058620 con 0x7f30f0070b10 2024-03-06T07:28:15.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.000+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30f0072890 con 0x7f30f0070b10 2024-03-06T07:28:15.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.002+0000 7f30e6000640 1 -- 172.21.15.97:0/532285702 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f30e0057f20 con 0x7f30f0070b10 2024-03-06T07:28:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:15.004 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d006b4d0 msgr2=0x7f30d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d006b4d0 0x7f30d006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30dc006e80 tx=0x7f30dc006dd0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 msgr2=0x7f30f00b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 0x7f30f00b3540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c400 tx=0x7f30e000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 shutdown_connections 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d006b4d0 0x7f30d006d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f015f9d0 0x7f30f00b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f0144c00 0x7f30f013c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 --2- 172.21.15.97:0/532285702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f0070b10 0x7f30f00b3540 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 >> 172.21.15.97:0/532285702 conn(0x7f30f00b1730 msgr2=0x7f30f00b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 shutdown_connections 2024-03-06T07:28:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.005+0000 7f30f5200640 1 -- 172.21.15.97:0/532285702 wait complete. 2024-03-06T07:28:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:15.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:15.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.234+0000 7f835c000640 1 Processor -- start 2024-03-06T07:28:15.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.234+0000 7f835c000640 1 -- start start 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f835c000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f83540b3530 0x7f835415b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f835c000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f83540ad670 0x7f83540adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f835c000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835415bcc0 con 0x7f83540b3910 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f835c000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83540ce690 con 0x7f83540b3530 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f835c000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83540ce970 con 0x7f83540ad670 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f83540b3530 0x7f835415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f83540b3530 0x7f835415b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39044/0 (socket says 172.21.15.97:39044) 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 -- 172.21.15.97:0/144354352 learned_addr learned my addr 172.21.15.97:0/144354352 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f835a000640 1 --2- 172.21.15.97:0/144354352 >> v2:172.21.15.135:3300/0 conn(0x7f83540ad670 0x7f83540adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 -- 172.21.15.97:0/144354352 >> v2:172.21.15.135:3300/0 conn(0x7f83540ad670 msgr2=0x7f83540adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 --2- 172.21.15.97:0/144354352 >> v2:172.21.15.135:3300/0 conn(0x7f83540ad670 0x7f83540adb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 -- 172.21.15.97:0/144354352 >> v1:172.21.15.135:6789/0 conn(0x7f83540b3910 legacy=0x7f83540acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.237+0000 7f8359600640 1 -- 172.21.15.97:0/144354352 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83540cec50 con 0x7f83540b3530 2024-03-06T07:28:15.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.238+0000 7f8359600640 1 --2- 172.21.15.97:0/144354352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3530 0x7f835415b780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8354074910 tx=0x7f834802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfbecdeb50a0ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.238+0000 7f8353e00640 1 -- 172.21.15.97:0/144354352 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348035020 con 0x7f83540b3530 2024-03-06T07:28:15.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.238+0000 7f8353e00640 1 -- 172.21.15.97:0/144354352 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8348031870 con 0x7f83540b3530 2024-03-06T07:28:15.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.238+0000 7f8353e00640 1 -- 172.21.15.97:0/144354352 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348031c40 con 0x7f83540b3530 2024-03-06T07:28:15.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 -- 172.21.15.97:0/144354352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3530 msgr2=0x7f835415b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 --2- 172.21.15.97:0/144354352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3530 0x7f835415b780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8354074910 tx=0x7f834802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 -- 172.21.15.97:0/144354352 shutdown_connections 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 --2- 172.21.15.97:0/144354352 >> v2:172.21.15.135:3300/0 conn(0x7f83540ad670 0x7f83540adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 --2- 172.21.15.97:0/144354352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3530 0x7f835415b780 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 -- 172.21.15.97:0/144354352 >> 172.21.15.97:0/144354352 conn(0x7f83540b1940 msgr2=0x7f83540b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 -- 172.21.15.97:0/144354352 shutdown_connections 2024-03-06T07:28:15.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.239+0000 7f835c000640 1 -- 172.21.15.97:0/144354352 wait complete. 2024-03-06T07:28:15.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.241+0000 7f835c000640 1 Processor -- start 2024-03-06T07:28:15.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.241+0000 7f835c000640 1 -- start start 2024-03-06T07:28:15.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835c000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83540ad670 0x7f83540cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835c000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83540b3530 0x7f835414a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835c000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 0x7f835414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835c000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83540b3d90 con 0x7f83540b3910 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835c000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83540cf210 con 0x7f83540b3530 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835c000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835415aa30 con 0x7f83540ad670 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 0x7f835414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 0x7f835414b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39054/0 (socket says 172.21.15.97:39054) 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.242+0000 7f835a000640 1 -- 172.21.15.97:0/2307988154 learned_addr learned my addr 172.21.15.97:0/2307988154 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f8359600640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83540ad670 0x7f83540cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f8358c00640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83540b3530 0x7f835414a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835a000640 1 -- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83540ad670 msgr2=0x7f83540cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835a000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83540ad670 0x7f83540cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835a000640 1 -- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83540b3530 msgr2=0x7f835414a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835a000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83540b3530 0x7f835414a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835a000640 1 -- 172.21.15.97:0/2307988154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835414a9b0 con 0x7f83540b3910 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835a000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 0x7f835414b2e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f834c00cd70 tx=0x7f834c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c003b90 con 0x7f83540b3910 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c003d30 con 0x7f83540b3910 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835414c100 con 0x7f83540b3910 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.243+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c004240 con 0x7f83540b3910 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.244+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835414c6c0 con 0x7f83540b3910 2024-03-06T07:28:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.245+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835414cbb0 con 0x7f83540b3910 2024-03-06T07:28:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.246+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f834c0043e0 con 0x7f83540b3910 2024-03-06T07:28:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.247+0000 7f8351600640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f833406b4d0 0x7f833406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.247+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f834c08ad30 con 0x7f83540b3910 2024-03-06T07:28:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.248+0000 7f8359600640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f833406b4d0 0x7f833406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.248+0000 7f8359600640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f833406b4d0 0x7f833406d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8348036000 tx=0x7f834803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.252+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f834c059ba0 con 0x7f83540b3910 2024-03-06T07:28:15.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.448+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f835404f330 con 0x7f83540b3910 2024-03-06T07:28:15.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.448+0000 7f8351600640 1 -- 172.21.15.97:0/2307988154 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f834c0594a0 con 0x7f83540b3910 2024-03-06T07:28:15.450 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:15.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f833406b4d0 msgr2=0x7f833406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f833406b4d0 0x7f833406d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8348036000 tx=0x7f834803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 msgr2=0x7f835414b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 0x7f835414b2e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f834c00cd70 tx=0x7f834c00a660 comp rx=0 tx=0).stop 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 shutdown_connections 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f833406b4d0 0x7f833406d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83540b3910 0x7f835414b2e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83540b3530 0x7f835414a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 --2- 172.21.15.97:0/2307988154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83540ad670 0x7f83540cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 >> 172.21.15.97:0/2307988154 conn(0x7f83540b1940 msgr2=0x7f83540b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 shutdown_connections 2024-03-06T07:28:15.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.451+0000 7f835c000640 1 -- 172.21.15.97:0/2307988154 wait complete. 2024-03-06T07:28:15.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.677+0000 7ffa4ba00640 1 Processor -- start 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.677+0000 7ffa4ba00640 1 -- start start 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.677+0000 7ffa4ba00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffa440b2c40 0x7ffa4415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa4ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 0x7ffa440cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa4ba00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4406cba0 con 0x7ffa4406d960 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa4ba00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4415f700 con 0x7ffa440b2c40 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa4ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4406d4d0 con 0x7ffa4406e2f0 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ffa4406d960 0x7ffa440b3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34866/0 (socket says 172.21.15.97:34866) 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43e00640 1 -- 172.21.15.97:0/2228162253 learned_addr learned my addr 172.21.15.97:0/2228162253 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:15.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280659342 0 0) 0x7ffa4406cba0 con 0x7ffa4406d960 2024-03-06T07:28:15.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2c003520 con 0x7ffa4406d960 2024-03-06T07:28:15.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa49000640 1 --2- 172.21.15.97:0/2228162253 >> v2:172.21.15.135:3300/0 conn(0x7ffa440b2c40 0x7ffa4415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa49a00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 0x7ffa440cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912039624 0 0) 0x7ffa2c003520 con 0x7ffa4406d960 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 msgr2=0x7ffa440cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 0x7ffa440cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 >> v2:172.21.15.135:3300/0 conn(0x7ffa440b2c40 msgr2=0x7ffa4415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 --2- 172.21.15.97:0/2228162253 >> v2:172.21.15.135:3300/0 conn(0x7ffa440b2c40 0x7ffa4415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4406ce80 con 0x7ffa4406d960 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ffa30001670 con 0x7ffa4406d960 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa49000640 1 --2- 172.21.15.97:0/2228162253 >> v2:172.21.15.135:3300/0 conn(0x7ffa440b2c40 0x7ffa4415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 >> v1:172.21.15.97:6789/0 conn(0x7ffa4406d960 legacy=0x7ffa440b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2c0039b0 0x7ffa2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 0x7ffa2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4406d4d0 con 0x7ffa2c004d40 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c00ba70 con 0x7ffa2c0039b0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c00bc20 con 0x7ffa4406e2f0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa49a00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.679+0000 7ffa43e00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 0x7ffa2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.680+0000 7ffa49000640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2c0039b0 0x7ffa2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.680+0000 7ffa49a00640 1 -- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 msgr2=0x7ffa2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.680+0000 7ffa49a00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa4406e2f0 0x7ffa2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.680+0000 7ffa49a00640 1 -- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2c0039b0 msgr2=0x7ffa2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.680+0000 7ffa49a00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2c0039b0 0x7ffa2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.680+0000 7ffa49a00640 1 -- 172.21.15.97:0/2228162253 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c003520 con 0x7ffa2c004d40 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.681+0000 7ffa49a00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa2c0051c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c007360 tx=0x7ffa2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c2dd5d4f9b8ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.681+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c039070 con 0x7ffa2c004d40 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.681+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c02ccb0 con 0x7ffa2c004d40 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.681+0000 7ffa43400640 1 -- 172.21.15.97:0/2228162253 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c009be0 con 0x7ffa2c004d40 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa4ba00640 1 -- 172.21.15.97:0/2228162253 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 msgr2=0x7ffa2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/2228162253 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa2c0051c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c007360 tx=0x7ffa2c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa49a00640 1 -- 172.21.15.97:0/2228162253 reap_dead start 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa4ba00640 1 -- 172.21.15.97:0/2228162253 shutdown_connections 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa4ba00640 1 -- 172.21.15.97:0/2228162253 >> 172.21.15.97:0/2228162253 conn(0x7ffa440b1940 msgr2=0x7ffa440b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa4ba00640 1 -- 172.21.15.97:0/2228162253 shutdown_connections 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.682+0000 7ffa4ba00640 1 -- 172.21.15.97:0/2228162253 wait complete. 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.684+0000 7ffa4ba00640 1 Processor -- start 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.684+0000 7ffa4ba00640 1 -- start start 2024-03-06T07:28:15.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa4ba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa441574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa4ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa441f4f60 0x7ffa441f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa4ba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa441493d0 0x7ffa440ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa4ba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4406e900 con 0x7ffa2c004d40 2024-03-06T07:28:15.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa4ba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4406e780 con 0x7ffa441493d0 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa4ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440cda90 con 0x7ffa441f4f60 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa49000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa441574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa49000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa441574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39064/0 (socket says 172.21.15.97:39064) 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa49000640 1 -- 172.21.15.97:0/1223347352 learned_addr learned my addr 172.21.15.97:0/1223347352 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa43e00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa441f4f60 0x7ffa441f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa49000640 1 -- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa441f4f60 msgr2=0x7ffa441f4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.685+0000 7ffa49a00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa441493d0 0x7ffa440ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa49000640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa441f4f60 0x7ffa441f4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa49000640 1 -- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa441493d0 msgr2=0x7ffa440ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa49000640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa441493d0 0x7ffa440ae7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa49000640 1 -- 172.21.15.97:0/1223347352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa441f53e0 con 0x7ffa2c004d40 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa49000640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa441574f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffa38010860 tx=0x7ffa38010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3801a020 con 0x7ffa2c004d40 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3800eb60 con 0x7ffa2c004d40 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800f040 con 0x7ffa2c004d40 2024-03-06T07:28:15.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa441f4d60 con 0x7ffa2c004d40 2024-03-06T07:28:15.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.686+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa44157e20 con 0x7ffa2c004d40 2024-03-06T07:28:15.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.688+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4404f330 con 0x7ffa2c004d40 2024-03-06T07:28:15.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.689+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa3801d020 con 0x7ffa2c004d40 2024-03-06T07:28:15.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.690+0000 7ffa40c00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa2406b500 0x7ffa2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.690+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa3808acc0 con 0x7ffa2c004d40 2024-03-06T07:28:15.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.690+0000 7ffa43e00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa2406b500 0x7ffa2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.691+0000 7ffa43e00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa2406b500 0x7ffa2406d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffa440af3b0 tx=0x7ffa30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.695+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa38091080 con 0x7ffa2c004d40 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.893+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa4415e050 con 0x7ffa2c004d40 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.893+0000 7ffa40c00640 1 -- 172.21.15.97:0/1223347352 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa38024020 con 0x7ffa2c004d40 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:15.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa2406b500 msgr2=0x7ffa2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa2406b500 0x7ffa2406d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffa440af3b0 tx=0x7ffa30009040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 msgr2=0x7ffa441574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa441574f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffa38010860 tx=0x7ffa38010d20 comp rx=0 tx=0).stop 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 shutdown_connections 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa2406b500 0x7ffa2406d9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa441493d0 0x7ffa440ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa441f4f60 0x7ffa441f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 --2- 172.21.15.97:0/1223347352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2c004d40 0x7ffa441574f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 >> 172.21.15.97:0/1223347352 conn(0x7ffa440b1940 msgr2=0x7ffa440b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 shutdown_connections 2024-03-06T07:28:15.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:15.896+0000 7ffa4ba00640 1 -- 172.21.15.97:0/1223347352 wait complete. 2024-03-06T07:28:15.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:16.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.126+0000 7f19c9600640 1 Processor -- start 2024-03-06T07:28:16.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.126+0000 7f19c9600640 1 -- start start 2024-03-06T07:28:16.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.127+0000 7f19c9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40d1ea0 0x7f19c40d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.127+0000 7f19c9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f19c40c7970 0x7f19c40c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.127+0000 7f19c9600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c40d13e0 con 0x7f19c415e130 2024-03-06T07:28:16.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.127+0000 7f19c9600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c40d2840 con 0x7f19c40c7970 2024-03-06T07:28:16.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.127+0000 7f19c9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40cf120 con 0x7f19c40d1ea0 2024-03-06T07:28:16.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f19c40c7970 0x7f19c40c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f19c40c7970 0x7f19c40c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39080/0 (socket says 172.21.15.97:39080) 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 -- 172.21.15.97:0/531042521 learned_addr learned my addr 172.21.15.97:0/531042521 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19bbe00640 1 --2- 172.21.15.97:0/531042521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40d1ea0 0x7f19c40d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 -- 172.21.15.97:0/531042521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40d1ea0 msgr2=0x7f19c40d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 --2- 172.21.15.97:0/531042521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40d1ea0 0x7f19c40d2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 -- 172.21.15.97:0/531042521 >> v1:172.21.15.135:6789/0 conn(0x7f19c415e130 legacy=0x7f19c40cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.128+0000 7f19c8c00640 1 -- 172.21.15.97:0/531042521 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c40cead0 con 0x7f19c40c7970 2024-03-06T07:28:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.129+0000 7f19c8c00640 1 --2- 172.21.15.97:0/531042521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40c7970 0x7f19c40c7e00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009e50 tx=0x7f19bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9262d240e264a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.129+0000 7f19c3400640 1 -- 172.21.15.97:0/531042521 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc02ca70 con 0x7f19c40c7970 2024-03-06T07:28:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c3400640 1 -- 172.21.15.97:0/531042521 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc02cc10 con 0x7f19c40c7970 2024-03-06T07:28:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c3400640 1 -- 172.21.15.97:0/531042521 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0343f0 con 0x7f19c40c7970 2024-03-06T07:28:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 -- 172.21.15.97:0/531042521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40c7970 msgr2=0x7f19c40c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 --2- 172.21.15.97:0/531042521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40c7970 0x7f19c40c7e00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009e50 tx=0x7f19bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 -- 172.21.15.97:0/531042521 shutdown_connections 2024-03-06T07:28:16.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 --2- 172.21.15.97:0/531042521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40c7970 0x7f19c40c7e00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 --2- 172.21.15.97:0/531042521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40d1ea0 0x7f19c40d2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 -- 172.21.15.97:0/531042521 >> 172.21.15.97:0/531042521 conn(0x7f19c409d070 msgr2=0x7f19c40d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 -- 172.21.15.97:0/531042521 shutdown_connections 2024-03-06T07:28:16.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.130+0000 7f19c9600640 1 -- 172.21.15.97:0/531042521 wait complete. 2024-03-06T07:28:16.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.132+0000 7f19c9600640 1 Processor -- start 2024-03-06T07:28:16.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.132+0000 7f19c9600640 1 -- start start 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19c9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40c7970 0x7f19c415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19c9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 0x7f19c40b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19c9600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c415e130 0x7f19c41ea880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19c9600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40cfcc0 con 0x7f19c40d1ea0 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19c9600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40c8340 con 0x7f19c415e130 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19c9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40d16c0 con 0x7f19c40c7970 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 0x7f19c40b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 0x7f19c40b79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39092/0 (socket says 172.21.15.97:39092) 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.133+0000 7f19bbe00640 1 -- 172.21.15.97:0/3486256056 learned_addr learned my addr 172.21.15.97:0/3486256056 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:16.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19c3e00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40c7970 0x7f19c415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19c8c00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c415e130 0x7f19c41ea880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19bbe00640 1 -- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40c7970 msgr2=0x7f19c415c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19bbe00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40c7970 0x7f19c415c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19bbe00640 1 -- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c415e130 msgr2=0x7f19c41ea880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19bbe00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c415e130 0x7f19c41ea880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19bbe00640 1 -- 172.21.15.97:0/3486256056 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c41eb2d0 con 0x7f19c40d1ea0 2024-03-06T07:28:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19c8c00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c415e130 0x7f19c41ea880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19bbe00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 0x7f19c40b79f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00d830 tx=0x7f19ac00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.134+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00b660 con 0x7f19c40d1ea0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.135+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac019070 con 0x7f19c40d1ea0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.135+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00ba20 con 0x7f19c40d1ea0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.135+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c41e7f60 con 0x7f19c40d1ea0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.135+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c41e8520 con 0x7f19c40d1ea0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.136+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c404f330 con 0x7f19c40d1ea0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.137+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19ac01e460 con 0x7f19c40d1ea0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.138+0000 7f19c0c00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f19a006b4d0 0x7f19a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.138+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19ac08a890 con 0x7f19c40d1ea0 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.139+0000 7f19c3e00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f19a006b4d0 0x7f19a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.139+0000 7f19c3e00640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f19a006b4d0 0x7f19a006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19c40b46a0 tx=0x7f19b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.142+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19ac059700 con 0x7f19c40d1ea0 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.340+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19c40b5650 con 0x7f19c40d1ea0 2024-03-06T07:28:16.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.341+0000 7f19c0c00640 1 -- 172.21.15.97:0/3486256056 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19ac059000 con 0x7f19c40d1ea0 2024-03-06T07:28:16.342 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f19a006b4d0 msgr2=0x7f19a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f19a006b4d0 0x7f19a006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19c40b46a0 tx=0x7f19b4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 msgr2=0x7f19c40b79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 0x7f19c40b79f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00d830 tx=0x7f19ac00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 shutdown_connections 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f19a006b4d0 0x7f19a006d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c415e130 0x7f19c41ea880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19c40d1ea0 0x7f19c40b79f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 --2- 172.21.15.97:0/3486256056 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f19c40c7970 0x7f19c415c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 >> 172.21.15.97:0/3486256056 conn(0x7f19c409d070 msgr2=0x7f19c40b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 shutdown_connections 2024-03-06T07:28:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.343+0000 7f19c9600640 1 -- 172.21.15.97:0/3486256056 wait complete. 2024-03-06T07:28:16.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:16.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.554+0000 7f766c400640 1 Processor -- start 2024-03-06T07:28:16.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.554+0000 7f766c400640 1 -- start start 2024-03-06T07:28:16.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.555+0000 7f766c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 0x7f766415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f766c400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766406cef0 con 0x7f7664070b10 2024-03-06T07:28:16.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f766c400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766406d1d0 con 0x7f7664144c00 2024-03-06T07:28:16.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f766c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7664160370 con 0x7f766415f9d0 2024-03-06T07:28:16.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f766a400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7664070b10 0x7f7664070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34870/0 (socket says 172.21.15.97:34870) 2024-03-06T07:28:16.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f766a400640 1 -- 172.21.15.97:0/2658830726 learned_addr learned my addr 172.21.15.97:0/2658830726 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:16.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f7669000640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 0x7f766415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813430508 0 0) 0x7f766406cef0 con 0x7f7664070b10 2024-03-06T07:28:16.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.556+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c003520 con 0x7f7664070b10 2024-03-06T07:28:16.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738830309 0 0) 0x7f764c003520 con 0x7f7664070b10 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 msgr2=0x7f766415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 0x7f766415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 >> v1:172.21.15.135:6789/0 conn(0x7f7664144c00 legacy=0x7f766406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766406d4b0 con 0x7f7664070b10 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f765c0019f0 con 0x7f7664070b10 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 >> v1:172.21.15.97:6789/0 conn(0x7f7664070b10 legacy=0x7f7664070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 0x7f764c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f764c003f60 0x7f764c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f764c004d40 0x7f764c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00ba70 con 0x7f764c004d40 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00bc20 con 0x7f764c003f60 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00be20 con 0x7f766415f9d0 2024-03-06T07:28:16.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f766a400640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f764c004d40 0x7f764c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7669000640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f764c003f60 0x7f764c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.557+0000 7f7669a00640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 0x7f764c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.558+0000 7f766a400640 1 -- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 msgr2=0x7f764c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.558+0000 7f766a400640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f766415f9d0 0x7f764c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.558+0000 7f766a400640 1 -- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f764c003f60 msgr2=0x7f764c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.558+0000 7f766a400640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f764c003f60 0x7f764c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.558+0000 7f766a400640 1 -- 172.21.15.97:0/2658830726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c003520 con 0x7f764c004d40 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.559+0000 7f766a400640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f764c004d40 0x7f764c0051c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f765c00a850 tx=0x7f764c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1511a5421eaae257 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.559+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c00a4a0 con 0x7f764c004d40 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.559+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c034650 con 0x7f764c004d40 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.559+0000 7f7663e00640 1 -- 172.21.15.97:0/2658830726 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c034a80 con 0x7f764c004d40 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766c400640 1 -- 172.21.15.97:0/2658830726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f764c004d40 msgr2=0x7f764c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766c400640 1 --2- 172.21.15.97:0/2658830726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f764c004d40 0x7f764c0051c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f765c00a850 tx=0x7f764c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766a400640 1 -- 172.21.15.97:0/2658830726 reap_dead start 2024-03-06T07:28:16.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766c400640 1 -- 172.21.15.97:0/2658830726 shutdown_connections 2024-03-06T07:28:16.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766c400640 1 -- 172.21.15.97:0/2658830726 >> 172.21.15.97:0/2658830726 conn(0x7f76640b1730 msgr2=0x7f76640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766c400640 1 -- 172.21.15.97:0/2658830726 shutdown_connections 2024-03-06T07:28:16.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.560+0000 7f766c400640 1 -- 172.21.15.97:0/2658830726 wait complete. 2024-03-06T07:28:16.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.563+0000 7f766c400640 1 Processor -- start 2024-03-06T07:28:16.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.563+0000 7f766c400640 1 -- start start 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.564+0000 7f766c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f764c004d40 0x7f7664150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.564+0000 7f766c400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f766414e100 0x7f76640aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.564+0000 7f766c400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 0x7f76640ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.564+0000 7f766c400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766406cb50 con 0x7f766413c380 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.564+0000 7f766c400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664160730 con 0x7f766414e100 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.564+0000 7f766c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766406ccd0 con 0x7f764c004d40 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 0x7f76640ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 0x7f76640ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39112/0 (socket says 172.21.15.97:39112) 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 -- 172.21.15.97:0/101370424 learned_addr learned my addr 172.21.15.97:0/101370424 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f7669000640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f766414e100 0x7f76640aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f7669a00640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f764c004d40 0x7f7664150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 -- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f764c004d40 msgr2=0x7f7664150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f764c004d40 0x7f7664150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 -- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f766414e100 msgr2=0x7f76640aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f766414e100 0x7f76640aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 -- 172.21.15.97:0/101370424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664151130 con 0x7f766413c380 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f7669a00640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f764c004d40 0x7f7664150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.565+0000 7f766a400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 0x7f76640ce060 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f765c02cc70 tx=0x7f765c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.566+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c00a480 con 0x7f766413c380 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.566+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c044070 con 0x7f766413c380 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.566+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c034800 con 0x7f766413c380 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.566+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664150890 con 0x7f766413c380 2024-03-06T07:28:16.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.566+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7664150e50 con 0x7f766413c380 2024-03-06T07:28:16.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.567+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766404f330 con 0x7f766413c380 2024-03-06T07:28:16.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.568+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f765c0349a0 con 0x7f766413c380 2024-03-06T07:28:16.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.569+0000 7f7661600640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.569+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f765c0ac400 con 0x7f766413c380 2024-03-06T07:28:16.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.569+0000 7f7669a00640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.570+0000 7f7669a00640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f764406b4d0 0x7f764406d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f76580016b0 tx=0x7f7658009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.574+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f765c07b280 con 0x7f766413c380 2024-03-06T07:28:16.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.930+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7664072890 con 0x7f766413c380 2024-03-06T07:28:16.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.933+0000 7f7661600640 1 -- 172.21.15.97:0/101370424 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f765c07ab80 con 0x7f766413c380 2024-03-06T07:28:16.934 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:16.934 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.935+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f764406b4d0 msgr2=0x7f764406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.935+0000 7f766c400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f764406b4d0 0x7f764406d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f76580016b0 tx=0x7f7658009040 comp rx=0 tx=0).stop 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.935+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 msgr2=0x7f76640ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.935+0000 7f766c400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 0x7f76640ce060 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f765c02cc70 tx=0x7f765c001dc0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 shutdown_connections 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f766413c380 0x7f76640ce060 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f766414e100 0x7f76640aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 --2- 172.21.15.97:0/101370424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f764c004d40 0x7f7664150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 >> 172.21.15.97:0/101370424 conn(0x7f76640b1730 msgr2=0x7f766415ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 shutdown_connections 2024-03-06T07:28:16.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:16.936+0000 7f766c400640 1 -- 172.21.15.97:0/101370424 wait complete. 2024-03-06T07:28:16.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:16.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:17.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.149+0000 7fcf72c00640 1 Processor -- start 2024-03-06T07:28:17.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.149+0000 7fcf72c00640 1 -- start start 2024-03-06T07:28:17.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf72c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcf6c06fef0 0x7fcf6c06f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf72c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0d1260 con 0x7fcf6c15e000 2024-03-06T07:28:17.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf72c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c06c320 con 0x7fcf6c15e3e0 2024-03-06T07:28:17.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf72c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c06c0d0 con 0x7fcf6c06fef0 2024-03-06T07:28:17.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf6be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcf6c15e000 0x7fcf6c06f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34878/0 (socket says 172.21.15.97:34878) 2024-03-06T07:28:17.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf6be00640 1 -- 172.21.15.97:0/850264949 learned_addr learned my addr 172.21.15.97:0/850264949 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:17.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf6b400640 1 --2- 172.21.15.97:0/850264949 >> v2:172.21.15.135:3300/0 conn(0x7fcf6c06fef0 0x7fcf6c06f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.151+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457709563 0 0) 0x7fcf6c0d1260 con 0x7fcf6c15e000 2024-03-06T07:28:17.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf50003520 con 0x7fcf6c15e000 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443689342 0 0) 0x7fcf50003520 con 0x7fcf6c15e000 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 >> v2:172.21.15.135:3300/0 conn(0x7fcf6c06fef0 msgr2=0x7fcf6c06f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 --2- 172.21.15.97:0/850264949 >> v2:172.21.15.135:3300/0 conn(0x7fcf6c06fef0 0x7fcf6c06f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 >> v1:172.21.15.135:6789/0 conn(0x7fcf6c15e3e0 legacy=0x7fcf6c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c06c600 con 0x7fcf6c15e000 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fcf5c001670 con 0x7fcf6c15e000 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 >> v1:172.21.15.97:6789/0 conn(0x7fcf6c15e000 legacy=0x7fcf6c06f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf50003a00 0x7fcf50003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50004800 0x7fcf50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf500051c0 0x7fcf5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c06c0d0 con 0x7fcf50003a00 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5000c0c0 con 0x7fcf50004800 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5000c2c0 con 0x7fcf500051c0 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.152+0000 7fcf6be00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf50003a00 0x7fcf50003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf6b400640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50004800 0x7fcf50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf6be00640 1 -- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf500051c0 msgr2=0x7fcf5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf6be00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf500051c0 0x7fcf5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf6be00640 1 -- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50004800 msgr2=0x7fcf50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf6be00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50004800 0x7fcf50004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf6be00640 1 -- 172.21.15.97:0/850264949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf50003520 con 0x7fcf50003a00 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.153+0000 7fcf70c00640 1 -- 172.21.15.97:0/850264949 reap_dead start 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.154+0000 7fcf6be00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf50003a00 0x7fcf50003e20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c190 tx=0x7fcf50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafec926ee98edde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.154+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c007c60 con 0x7fcf50003a00 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.154+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c030070 con 0x7fcf50003a00 2024-03-06T07:28:17.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.154+0000 7fcf6aa00640 1 -- 172.21.15.97:0/850264949 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c005d90 con 0x7fcf50003a00 2024-03-06T07:28:17.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.154+0000 7fcf72c00640 1 -- 172.21.15.97:0/850264949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf50003a00 msgr2=0x7fcf50003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.154+0000 7fcf72c00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf50003a00 0x7fcf50003e20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c190 tx=0x7fcf50005830 comp rx=0 tx=0).stop 2024-03-06T07:28:17.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.155+0000 7fcf72c00640 1 -- 172.21.15.97:0/850264949 shutdown_connections 2024-03-06T07:28:17.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.155+0000 7fcf72c00640 1 --2- 172.21.15.97:0/850264949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf50003a00 0x7fcf50003e20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.155+0000 7fcf72c00640 1 -- 172.21.15.97:0/850264949 >> 172.21.15.97:0/850264949 conn(0x7fcf6c0b3800 msgr2=0x7fcf6c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.155+0000 7fcf72c00640 1 -- 172.21.15.97:0/850264949 shutdown_connections 2024-03-06T07:28:17.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.155+0000 7fcf72c00640 1 -- 172.21.15.97:0/850264949 wait complete. 2024-03-06T07:28:17.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.157+0000 7fcf72c00640 1 Processor -- start 2024-03-06T07:28:17.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.158+0000 7fcf72c00640 1 -- start start 2024-03-06T07:28:17.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf72c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50003a00 0x7fcf6c0c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf72c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 0x7fcf6c0c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf72c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf6c0ca860 0x7fcf6c0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf72c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c06f4c0 con 0x7fcf6c15bb70 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf72c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c06f340 con 0x7fcf50003a00 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf72c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0d0ae0 con 0x7fcf6c0ca860 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 0x7fcf6c0c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 0x7fcf6c0c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39124/0 (socket says 172.21.15.97:39124) 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.159+0000 7fcf6b400640 1 -- 172.21.15.97:0/443791588 learned_addr learned my addr 172.21.15.97:0/443791588 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6be00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50003a00 0x7fcf6c0c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6b400640 1 -- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf6c0ca860 msgr2=0x7fcf6c0cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6b400640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf6c0ca860 0x7fcf6c0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6b400640 1 -- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50003a00 msgr2=0x7fcf6c0c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6b400640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50003a00 0x7fcf6c0c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6b400640 1 -- 172.21.15.97:0/443791588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0c8510 con 0x7fcf6c15bb70 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf6b400640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 0x7fcf6c0c7fd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf58010790 tx=0x7fcf58010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5800f040 con 0x7fcf6c15bb70 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5800eb00 con 0x7fcf6c15bb70 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58010ea0 con 0x7fcf6c15bb70 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.160+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c0b5980 con 0x7fcf6c15bb70 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.161+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c0c6ad0 con 0x7fcf6c15bb70 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.162+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c04f330 con 0x7fcf6c15bb70 2024-03-06T07:28:17.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.163+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf5801c400 con 0x7fcf6c15bb70 2024-03-06T07:28:17.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.163+0000 7fcf63e00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcf4c06b4d0 0x7fcf4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.163+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf5808a650 con 0x7fcf6c15bb70 2024-03-06T07:28:17.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.164+0000 7fcf6be00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcf4c06b4d0 0x7fcf4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.164+0000 7fcf6be00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcf4c06b4d0 0x7fcf4c06d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c160 tx=0x7fcf5c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.168+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf6c0b5980 con 0x7fcf6c15bb70 2024-03-06T07:28:17.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.410+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf6c06fca0 con 0x7fcf6c15bb70 2024-03-06T07:28:17.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.411+0000 7fcf63e00640 1 -- 172.21.15.97:0/443791588 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf58018030 con 0x7fcf6c15bb70 2024-03-06T07:28:17.412 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:17.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcf4c06b4d0 msgr2=0x7fcf4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcf4c06b4d0 0x7fcf4c06d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c160 tx=0x7fcf5c009030 comp rx=0 tx=0).stop 2024-03-06T07:28:17.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 msgr2=0x7fcf6c0c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 0x7fcf6c0c7fd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf58010790 tx=0x7fcf58010c50 comp rx=0 tx=0).stop 2024-03-06T07:28:17.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 shutdown_connections 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcf4c06b4d0 0x7fcf4c06d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcf6c0ca860 0x7fcf6c0cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcf6c15bb70 0x7fcf6c0c7fd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 --2- 172.21.15.97:0/443791588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf50003a00 0x7fcf6c0c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 >> 172.21.15.97:0/443791588 conn(0x7fcf6c0b3800 msgr2=0x7fcf6c0aee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 shutdown_connections 2024-03-06T07:28:17.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.413+0000 7fcf72c00640 1 -- 172.21.15.97:0/443791588 wait complete. 2024-03-06T07:28:17.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:17.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.638+0000 7f4ccf600640 1 Processor -- start 2024-03-06T07:28:17.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.638+0000 7f4ccf600640 1 -- start start 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.639+0000 7f4ccf600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4cc80d07f0 0x7f4cc80d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.639+0000 7f4ccf600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 0x7f4cc80b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.639+0000 7f4ccf600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80b3730 con 0x7f4cc80d1bf0 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.639+0000 7f4ccf600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc80d4930 con 0x7f4cc80d07f0 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.639+0000 7f4ccf600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8067b20 con 0x7f4cc815e450 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.640+0000 7f4ccd600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4cc80d1bf0 0x7f4cc80b49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34884/0 (socket says 172.21.15.97:34884) 2024-03-06T07:28:17.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.640+0000 7f4ccd600640 1 -- 172.21.15.97:0/2808981092 learned_addr learned my addr 172.21.15.97:0/2808981092 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:17.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.640+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419839902 0 0) 0x7f4cc80b3730 con 0x7f4cc80d1bf0 2024-03-06T07:28:17.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.640+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb0003520 con 0x7f4cc80d1bf0 2024-03-06T07:28:17.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.640+0000 7f4cccc00640 1 --2- 172.21.15.97:0/2808981092 >> v2:172.21.15.135:3300/0 conn(0x7f4cc80d07f0 0x7f4cc80d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.640+0000 7f4cc7e00640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 0x7f4cc80b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067377847 0 0) 0x7f4cb0003520 con 0x7f4cc80d1bf0 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 msgr2=0x7f4cc80b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 0x7f4cc80b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 >> v2:172.21.15.135:3300/0 conn(0x7f4cc80d07f0 msgr2=0x7f4cc80d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 --2- 172.21.15.97:0/2808981092 >> v2:172.21.15.135:3300/0 conn(0x7f4cc80d07f0 0x7f4cc80d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc80d1110 con 0x7f4cc80d1bf0 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4cc00019f0 con 0x7f4cc80d1bf0 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 >> v1:172.21.15.97:6789/0 conn(0x7f4cc80d1bf0 legacy=0x7f4cc80b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb00039b0 0x7f4cb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 0x7f4cb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb0004d40 0x7f4cb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80d4930 con 0x7f4cb0004d40 2024-03-06T07:28:17.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8067b20 con 0x7f4cb00039b0 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb000bb50 con 0x7f4cc815e450 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4ccd600640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb0004d40 0x7f4cb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4cccc00640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb00039b0 0x7f4cb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4ccd600640 1 -- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 msgr2=0x7f4cb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4ccd600640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc815e450 0x7f4cb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4ccd600640 1 -- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb00039b0 msgr2=0x7f4cb0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4ccd600640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb00039b0 0x7f4cb0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.641+0000 7f4ccd600640 1 -- 172.21.15.97:0/2808981092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb0003520 con 0x7f4cb0004d40 2024-03-06T07:28:17.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccd600640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb0004d40 0x7f4cb00051c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000a7a0 tx=0x7f4cb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7765852bf67fc56f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000a3f0 con 0x7f4cb0004d40 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc0034630 con 0x7f4cb0004d40 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4cc7400640 1 -- 172.21.15.97:0/2808981092 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0034a40 con 0x7f4cb0004d40 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccf600640 1 -- 172.21.15.97:0/2808981092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb0004d40 msgr2=0x7f4cb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccf600640 1 --2- 172.21.15.97:0/2808981092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb0004d40 0x7f4cb00051c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000a7a0 tx=0x7f4cb0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccd600640 1 -- 172.21.15.97:0/2808981092 reap_dead start 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccf600640 1 -- 172.21.15.97:0/2808981092 shutdown_connections 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccf600640 1 -- 172.21.15.97:0/2808981092 >> 172.21.15.97:0/2808981092 conn(0x7f4cc80d3e10 msgr2=0x7f4cc80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccf600640 1 -- 172.21.15.97:0/2808981092 shutdown_connections 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.643+0000 7f4ccf600640 1 -- 172.21.15.97:0/2808981092 wait complete. 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.645+0000 7f4ccf600640 1 Processor -- start 2024-03-06T07:28:17.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.645+0000 7f4ccf600640 1 -- start start 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccf600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb0004d40 0x7f4cc815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccf600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc80b9220 0x7f4cc80b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccf600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 0x7f4cc80b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccf600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80b3c80 con 0x7f4cc80b96a0 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccf600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80b39a0 con 0x7f4cb0004d40 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccf600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80b3f60 con 0x7f4cc80b9220 2024-03-06T07:28:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccd600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 0x7f4cc80b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccd600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 0x7f4cc80b70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39144/0 (socket says 172.21.15.97:39144) 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.646+0000 7f4ccd600640 1 -- 172.21.15.97:0/1437001471 learned_addr learned my addr 172.21.15.97:0/1437001471 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccd600640 1 -- 172.21.15.97:0/1437001471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc80b9220 msgr2=0x7f4cc80b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccd600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc80b9220 0x7f4cc80b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccd600640 1 -- 172.21.15.97:0/1437001471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb0004d40 msgr2=0x7f4cc815cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccd600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb0004d40 0x7f4cc815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccd600640 1 -- 172.21.15.97:0/1437001471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc815d310 con 0x7f4cc80b96a0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccd600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 0x7f4cc80b70b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000a8c0 tx=0x7f4cc80b09e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000a3d0 con 0x7f4cc80b96a0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc0044070 con 0x7f4cc80b96a0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00349f0 con 0x7f4cc80b96a0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc80cd550 con 0x7f4cc80b96a0 2024-03-06T07:28:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.647+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc80cdb10 con 0x7f4cc80b96a0 2024-03-06T07:28:17.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.649+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc804f330 con 0x7f4cc80b96a0 2024-03-06T07:28:17.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.649+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cc0034b90 con 0x7f4cc80b96a0 2024-03-06T07:28:17.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.650+0000 7f4cc4c00640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ca806b4d0 0x7f4ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.650+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cc00ac690 con 0x7f4cc80b96a0 2024-03-06T07:28:17.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.651+0000 7f4cccc00640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ca806b4d0 0x7f4ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.651+0000 7f4cccc00640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ca806b4d0 0x7f4ca806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc006e80 tx=0x7f4cbc006e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.655+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cc007b630 con 0x7f4cc80b96a0 2024-03-06T07:28:17.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.873+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4cc80d21d0 con 0x7f4cc80b96a0 2024-03-06T07:28:17.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.874+0000 7f4cc4c00640 1 -- 172.21.15.97:0/1437001471 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4cc007af30 con 0x7f4cc80b96a0 2024-03-06T07:28:17.875 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:17.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.876+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ca806b4d0 msgr2=0x7f4ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.876+0000 7f4ccf600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ca806b4d0 0x7f4ca806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc006e80 tx=0x7f4cbc006e10 comp rx=0 tx=0).stop 2024-03-06T07:28:17.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.876+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 msgr2=0x7f4cc80b70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.876+0000 7f4ccf600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 0x7f4cc80b70b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000a8c0 tx=0x7f4cc80b09e0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 shutdown_connections 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ca806b4d0 0x7f4ca806d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cc80b96a0 0x7f4cc80b70b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cc80b9220 0x7f4cc80b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 --2- 172.21.15.97:0/1437001471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb0004d40 0x7f4cc815cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 >> 172.21.15.97:0/1437001471 conn(0x7f4cc80d3e10 msgr2=0x7f4cc8160870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 shutdown_connections 2024-03-06T07:28:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:17.877+0000 7f4ccf600640 1 -- 172.21.15.97:0/1437001471 wait complete. 2024-03-06T07:28:17.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.108+0000 7f6773e00640 1 Processor -- start 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.108+0000 7f6773e00640 1 -- start start 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6773e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677415e130 0x7f67740cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6773e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67740d13e0 con 0x7f67740cfaf0 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6773e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67740cead0 con 0x7f67740d25b0 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6773e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740cf120 con 0x7f677415e130 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f676be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f67740cfaf0 0x7f67740d1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34894/0 (socket says 172.21.15.97:34894) 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f676be00640 1 -- 172.21.15.97:0/3161953528 learned_addr learned my addr 172.21.15.97:0/3161953528 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:18.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6772a00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677415e130 0x7f67740cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275565350 0 0) 0x7f67740d13e0 con 0x7f67740cfaf0 2024-03-06T07:28:18.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.110+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6758003520 con 0x7f67740cfaf0 2024-03-06T07:28:18.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1001269300 0 0) 0x7f6758003520 con 0x7f67740cfaf0 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677415e130 msgr2=0x7f67740cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677415e130 0x7f67740cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 >> v1:172.21.15.135:6789/0 conn(0x7f67740d25b0 legacy=0x7f67740c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740cedb0 con 0x7f67740cfaf0 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f675c001670 con 0x7f67740cfaf0 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 >> v1:172.21.15.97:6789/0 conn(0x7f67740cfaf0 legacy=0x7f67740d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67580039e0 0x7f6758003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f6758004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67580051c0 0x7f675800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67740cf120 con 0x7f6758004800 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675800c090 con 0x7f67580039e0 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675800c260 con 0x7f67580051c0 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f6772a00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677415e130 0x7f67740cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.111+0000 7f676be00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f6758004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f6773400640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67580051c0 0x7f675800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f676be00640 1 -- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67580051c0 msgr2=0x7f675800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f676be00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67580051c0 0x7f675800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f676be00640 1 -- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67580039e0 msgr2=0x7f6758003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f676be00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67580039e0 0x7f6758003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f676be00640 1 -- 172.21.15.97:0/3161953528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6758003520 con 0x7f6758004800 2024-03-06T07:28:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.112+0000 7f6773400640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67580051c0 0x7f675800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.113+0000 7f676be00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f6758004c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f675c007fd0 tx=0x7f6758005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af08553bdc09182 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.113+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c007d50 con 0x7f6758004800 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.113+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c030070 con 0x7f6758004800 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.113+0000 7f6772000640 1 -- 172.21.15.97:0/3161953528 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c02cd40 con 0x7f6758004800 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773e00640 1 -- 172.21.15.97:0/3161953528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 msgr2=0x7f6758004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773e00640 1 --2- 172.21.15.97:0/3161953528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f6758004c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f675c007fd0 tx=0x7f6758005830 comp rx=0 tx=0).stop 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773400640 1 -- 172.21.15.97:0/3161953528 reap_dead start 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773e00640 1 -- 172.21.15.97:0/3161953528 shutdown_connections 2024-03-06T07:28:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773e00640 1 -- 172.21.15.97:0/3161953528 >> 172.21.15.97:0/3161953528 conn(0x7f677409d070 msgr2=0x7f67740d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773e00640 1 -- 172.21.15.97:0/3161953528 shutdown_connections 2024-03-06T07:28:18.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.114+0000 7f6773e00640 1 -- 172.21.15.97:0/3161953528 wait complete. 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.115+0000 7f6773e00640 1 Processor -- start 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.116+0000 7f6773e00640 1 -- start start 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.116+0000 7f6773e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f67740b9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.116+0000 7f6773e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67740b79f0 0x7f677415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6773e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6774161fb0 0x7f6774162430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6773e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67740c8200 con 0x7f6758004800 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6773e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67740c8080 con 0x7f6774161fb0 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6773e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740cf5b0 con 0x7f67740b79f0 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f67740b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f67740b9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39168/0 (socket says 172.21.15.97:39168) 2024-03-06T07:28:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 -- 172.21.15.97:0/2722452486 learned_addr learned my addr 172.21.15.97:0/2722452486 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:18.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 -- 172.21.15.97:0/2722452486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67740b79f0 msgr2=0x7f677415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67740b79f0 0x7f677415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 -- 172.21.15.97:0/2722452486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6774161fb0 msgr2=0x7f6774162430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6774161fb0 0x7f6774162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.117+0000 7f6772a00640 1 -- 172.21.15.97:0/2722452486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740b81d0 con 0x7f6758004800 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.118+0000 7f6772a00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f67740b9730 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f676400e570 tx=0x7f676400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.118+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400f040 con 0x7f6758004800 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.118+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6764004250 con 0x7f6758004800 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.118+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764003610 con 0x7f6758004800 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.118+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67740b84b0 con 0x7f6758004800 2024-03-06T07:28:18.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.118+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67740b8a70 con 0x7f6758004800 2024-03-06T07:28:18.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.119+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677408d440 con 0x7f6758004800 2024-03-06T07:28:18.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.120+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67640037b0 con 0x7f6758004800 2024-03-06T07:28:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.121+0000 7f676aa00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675006b500 0x7f675006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.121+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6764089ef0 con 0x7f6758004800 2024-03-06T07:28:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.121+0000 7f676be00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675006b500 0x7f675006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.121+0000 7f676be00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675006b500 0x7f675006d9c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f675c034000 tx=0x7f675c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.129+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f677408d440 con 0x7f6758004800 2024-03-06T07:28:18.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.397+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f677404f330 con 0x7f6758004800 2024-03-06T07:28:18.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.398+0000 7f676aa00640 1 -- 172.21.15.97:0/2722452486 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6764017020 con 0x7f6758004800 2024-03-06T07:28:18.399 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:18.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.401+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675006b500 msgr2=0x7f675006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.401+0000 7f6773e00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675006b500 0x7f675006d9c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f675c034000 tx=0x7f675c007000 comp rx=0 tx=0).stop 2024-03-06T07:28:18.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 msgr2=0x7f67740b9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f67740b9730 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f676400e570 tx=0x7f676400ea30 comp rx=0 tx=0).stop 2024-03-06T07:28:18.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 shutdown_connections 2024-03-06T07:28:18.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675006b500 0x7f675006d9c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6774161fb0 0x7f6774162430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67740b79f0 0x7f677415ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 --2- 172.21.15.97:0/2722452486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6758004800 0x7f67740b9730 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 >> 172.21.15.97:0/2722452486 conn(0x7f677409d070 msgr2=0x7f67740c7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 shutdown_connections 2024-03-06T07:28:18.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.402+0000 7f6773e00640 1 -- 172.21.15.97:0/2722452486 wait complete. 2024-03-06T07:28:18.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:18.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.631+0000 7ff8b5400640 1 Processor -- start 2024-03-06T07:28:18.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.631+0000 7ff8b5400640 1 -- start start 2024-03-06T07:28:18.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8b5400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b015b420 0x7ff8b00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8b5400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8b015bf10 0x7ff8b01649d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8b5400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b00b3f60 con 0x7ff8b0066d00 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8b5400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b00afd50 con 0x7ff8b015bf10 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8b5400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b00b4200 con 0x7ff8b015b420 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8af200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8b015bf10 0x7ff8b01649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8af200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8b015bf10 0x7ff8b01649d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39178/0 (socket says 172.21.15.97:39178) 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8af200640 1 -- 172.21.15.97:0/266827905 learned_addr learned my addr 172.21.15.97:0/266827905 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8ae800640 1 --2- 172.21.15.97:0/266827905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b015b420 0x7ff8b00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8af200640 1 -- 172.21.15.97:0/266827905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b015b420 msgr2=0x7ff8b00af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8af200640 1 --2- 172.21.15.97:0/266827905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b015b420 0x7ff8b00af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.633+0000 7ff8af200640 1 -- 172.21.15.97:0/266827905 >> v1:172.21.15.135:6789/0 conn(0x7ff8b0066d00 legacy=0x7ff8b015b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.634+0000 7ff8af200640 1 -- 172.21.15.97:0/266827905 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b00b44e0 con 0x7ff8b015bf10 2024-03-06T07:28:18.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.635+0000 7ff8af200640 1 --2- 172.21.15.97:0/266827905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015bf10 0x7ff8b01649d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009e50 tx=0x7ff8a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75de363647fb5067 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.635+0000 7ff8ade00640 1 -- 172.21.15.97:0/266827905 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a802ca70 con 0x7ff8b015bf10 2024-03-06T07:28:18.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.635+0000 7ff8ade00640 1 -- 172.21.15.97:0/266827905 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a802cc10 con 0x7ff8b015bf10 2024-03-06T07:28:18.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.635+0000 7ff8ade00640 1 -- 172.21.15.97:0/266827905 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80343f0 con 0x7ff8b015bf10 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 -- 172.21.15.97:0/266827905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015bf10 msgr2=0x7ff8b01649d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 --2- 172.21.15.97:0/266827905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015bf10 0x7ff8b01649d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009e50 tx=0x7ff8a802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 -- 172.21.15.97:0/266827905 shutdown_connections 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 --2- 172.21.15.97:0/266827905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015bf10 0x7ff8b01649d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 --2- 172.21.15.97:0/266827905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b015b420 0x7ff8b00af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 -- 172.21.15.97:0/266827905 >> 172.21.15.97:0/266827905 conn(0x7ff8b00d0340 msgr2=0x7ff8b00d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 -- 172.21.15.97:0/266827905 shutdown_connections 2024-03-06T07:28:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.636+0000 7ff8b5400640 1 -- 172.21.15.97:0/266827905 wait complete. 2024-03-06T07:28:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.638+0000 7ff8b5400640 1 Processor -- start 2024-03-06T07:28:18.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.638+0000 7ff8b5400640 1 -- start start 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8b5400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b0066d00 0x7ff8b01f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8b5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 0x7ff8b0150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8b5400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8b015bf10 0x7ff8b01500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8b5400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b015b1a0 con 0x7ff8b015b420 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8b5400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b006c8a0 con 0x7ff8b015bf10 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8b5400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b006e3b0 con 0x7ff8b0066d00 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 0x7ff8b0150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 0x7ff8b0150940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39192/0 (socket says 172.21.15.97:39192) 2024-03-06T07:28:18.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 -- 172.21.15.97:0/2803070752 learned_addr learned my addr 172.21.15.97:0/2803070752 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8ae800640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b0066d00 0x7ff8b01f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8af200640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8b015bf10 0x7ff8b01500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 -- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b0066d00 msgr2=0x7ff8b01f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b0066d00 0x7ff8b01f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 -- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8b015bf10 msgr2=0x7ff8b01500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8b015bf10 0x7ff8b01500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.639+0000 7ff8a7e00640 1 -- 172.21.15.97:0/2803070752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b0151070 con 0x7ff8b015b420 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.640+0000 7ff8a7e00640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 0x7ff8b0150940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff89800c460 tx=0x7ff89800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.640+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898016020 con 0x7ff8b015b420 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.640+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898012870 con 0x7ff8b015b420 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.640+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898012c60 con 0x7ff8b015b420 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.640+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b0151350 con 0x7ff8b015b420 2024-03-06T07:28:18.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.640+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b0151910 con 0x7ff8b015b420 2024-03-06T07:28:18.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.641+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b0074ae0 con 0x7ff8b015b420 2024-03-06T07:28:18.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.642+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff898019020 con 0x7ff8b015b420 2024-03-06T07:28:18.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.644+0000 7ff8a6a00640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff88c06b4d0 0x7ff88c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.644+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff898089f30 con 0x7ff8b015b420 2024-03-06T07:28:18.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.644+0000 7ff8ae800640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff88c06b4d0 0x7ff88c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.645+0000 7ff8ae800640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff88c06b4d0 0x7ff88c06d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006fd0 tx=0x7ff8a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.648+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff898020030 con 0x7ff8b015b420 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.847+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b006bdf0 con 0x7ff8b015b420 2024-03-06T07:28:18.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.850+0000 7ff8a6a00640 1 -- 172.21.15.97:0/2803070752 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7ff8b006bdf0 con 0x7ff8b015b420 2024-03-06T07:28:18.851 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:18.851 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:18.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff88c06b4d0 msgr2=0x7ff88c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff88c06b4d0 0x7ff88c06d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006fd0 tx=0x7ff8a0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 msgr2=0x7ff8b0150940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 0x7ff8b0150940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff89800c460 tx=0x7ff89800c920 comp rx=0 tx=0).stop 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 shutdown_connections 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff88c06b4d0 0x7ff88c06d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8b015bf10 0x7ff8b01500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8b015b420 0x7ff8b0150940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 --2- 172.21.15.97:0/2803070752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8b0066d00 0x7ff8b01f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.852+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 >> 172.21.15.97:0/2803070752 conn(0x7ff8b00d0340 msgr2=0x7ff8b015b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.853+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 shutdown_connections 2024-03-06T07:28:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:18.853+0000 7ff8b5400640 1 -- 172.21.15.97:0/2803070752 wait complete. 2024-03-06T07:28:18.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:18.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:19.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.082+0000 7fdf22600640 1 Processor -- start 2024-03-06T07:28:19.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.082+0000 7fdf22600640 1 -- start start 2024-03-06T07:28:19.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.083+0000 7fdf22600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdf1c0d07f0 0x7fdf1c0d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.083+0000 7fdf22600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1c0b3730 con 0x7fdf1c15e450 2024-03-06T07:28:19.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.083+0000 7fdf22600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1c067b20 con 0x7fdf1c0d1bf0 2024-03-06T07:28:19.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.083+0000 7fdf22600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c0d4930 con 0x7fdf1c0d07f0 2024-03-06T07:28:19.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.084+0000 7fdf1b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdf1c0d07f0 0x7fdf1c0d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.084+0000 7fdf13e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdf1c15e450 0x7fdf1c0b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34898/0 (socket says 172.21.15.97:34898) 2024-03-06T07:28:19.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.084+0000 7fdf13e00640 1 -- 172.21.15.97:0/3481508756 learned_addr learned my addr 172.21.15.97:0/3481508756 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:19.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.084+0000 7fdf1b400640 1 -- 172.21.15.97:0/3481508756 >> v1:172.21.15.135:6789/0 conn(0x7fdf1c0d1bf0 legacy=0x7fdf1c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.084+0000 7fdf1b400640 1 -- 172.21.15.97:0/3481508756 >> v1:172.21.15.97:6789/0 conn(0x7fdf1c15e450 legacy=0x7fdf1c0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.084+0000 7fdf1b400640 1 -- 172.21.15.97:0/3481508756 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0d1110 con 0x7fdf1c0d07f0 2024-03-06T07:28:19.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.085+0000 7fdf1b400640 1 --2- 172.21.15.97:0/3481508756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c0d07f0 0x7fdf1c0d0bd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0097a0 tx=0x7fdf0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e1548d6b68391e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.086+0000 7fdf1aa00640 1 -- 172.21.15.97:0/3481508756 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0315a0 con 0x7fdf1c0d07f0 2024-03-06T07:28:19.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.086+0000 7fdf1aa00640 1 -- 172.21.15.97:0/3481508756 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0c037070 con 0x7fdf1c0d07f0 2024-03-06T07:28:19.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.086+0000 7fdf1aa00640 1 -- 172.21.15.97:0/3481508756 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c031960 con 0x7fdf1c0d07f0 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 -- 172.21.15.97:0/3481508756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c0d07f0 msgr2=0x7fdf1c0d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 --2- 172.21.15.97:0/3481508756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c0d07f0 0x7fdf1c0d0bd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0097a0 tx=0x7fdf0c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 -- 172.21.15.97:0/3481508756 shutdown_connections 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 --2- 172.21.15.97:0/3481508756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c0d07f0 0x7fdf1c0d0bd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 -- 172.21.15.97:0/3481508756 >> 172.21.15.97:0/3481508756 conn(0x7fdf1c0d3e10 msgr2=0x7fdf1c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 -- 172.21.15.97:0/3481508756 shutdown_connections 2024-03-06T07:28:19.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.087+0000 7fdf22600640 1 -- 172.21.15.97:0/3481508756 wait complete. 2024-03-06T07:28:19.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.090+0000 7fdf22600640 1 Processor -- start 2024-03-06T07:28:19.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.090+0000 7fdf22600640 1 -- start start 2024-03-06T07:28:19.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.092+0000 7fdf22600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf1c0d07f0 0x7fdf1c15f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.092+0000 7fdf22600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf1c0d1bf0 0x7fdf1c06be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.092+0000 7fdf22600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 0x7fdf1c06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.092+0000 7fdf22600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0b3c80 con 0x7fdf1c15e450 2024-03-06T07:28:19.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.092+0000 7fdf22600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0b39a0 con 0x7fdf1c0d1bf0 2024-03-06T07:28:19.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.092+0000 7fdf22600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0b3f60 con 0x7fdf1c0d07f0 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 0x7fdf1c06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 0x7fdf1c06c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39204/0 (socket says 172.21.15.97:39204) 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 -- 172.21.15.97:0/504541530 learned_addr learned my addr 172.21.15.97:0/504541530 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1b400640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf1c0d07f0 0x7fdf1c15f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf13e00640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf1c0d1bf0 0x7fdf1c06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 -- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf1c0d07f0 msgr2=0x7fdf1c15f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf1c0d07f0 0x7fdf1c15f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 -- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf1c0d1bf0 msgr2=0x7fdf1c06be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf1c0d1bf0 0x7fdf1c06be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.093+0000 7fdf1be00640 1 -- 172.21.15.97:0/504541530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c160820 con 0x7fdf1c15e450 2024-03-06T07:28:19.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.094+0000 7fdf1be00640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 0x7fdf1c06c3c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400c780 tx=0x7fdf1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.094+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14004b60 con 0x7fdf1c15e450 2024-03-06T07:28:19.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.094+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf14004d00 con 0x7fdf1c15e450 2024-03-06T07:28:19.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.094+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c160b00 con 0x7fdf1c15e450 2024-03-06T07:28:19.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.094+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400aa90 con 0x7fdf1c15e450 2024-03-06T07:28:19.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.094+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c1610c0 con 0x7fdf1c15e450 2024-03-06T07:28:19.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.095+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c04f330 con 0x7fdf1c15e450 2024-03-06T07:28:19.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.096+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf1400ac30 con 0x7fdf1c15e450 2024-03-06T07:28:19.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.097+0000 7fdf13400640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdef806b4d0 0x7fdef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.097+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf1408a4d0 con 0x7fdf1c15e450 2024-03-06T07:28:19.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.097+0000 7fdf1b400640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdef806b4d0 0x7fdef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.098+0000 7fdf1b400640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdef806b4d0 0x7fdef806d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c036000 tx=0x7fdf0c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.101+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf14020070 con 0x7fdf1c15e450 2024-03-06T07:28:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.300+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf1c0b1620 con 0x7fdf1c15e450 2024-03-06T07:28:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.302+0000 7fdf13400640 1 -- 172.21.15.97:0/504541530 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf14088e20 con 0x7fdf1c15e450 2024-03-06T07:28:19.303 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdef806b4d0 msgr2=0x7fdef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdef806b4d0 0x7fdef806d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c036000 tx=0x7fdf0c03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 msgr2=0x7fdf1c06c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 0x7fdf1c06c3c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400c780 tx=0x7fdf1400cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 shutdown_connections 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdef806b4d0 0x7fdef806d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf1c15e450 0x7fdf1c06c3c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf1c0d1bf0 0x7fdf1c06be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 --2- 172.21.15.97:0/504541530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf1c0d07f0 0x7fdf1c15f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 >> 172.21.15.97:0/504541530 conn(0x7fdf1c0d3e10 msgr2=0x7fdf1c0b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 shutdown_connections 2024-03-06T07:28:19.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.305+0000 7fdf22600640 1 -- 172.21.15.97:0/504541530 wait complete. 2024-03-06T07:28:19.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:19.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.526+0000 7f7fb8400640 1 Processor -- start 2024-03-06T07:28:19.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.526+0000 7f7fb8400640 1 -- start start 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb8400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb00b3f70 0x7f7fb0160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb8400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7fb0070ff0 0x7f7fb00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb8400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00b46c0 con 0x7f7fb015fc10 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb8400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb0071480 con 0x7f7fb0070ff0 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb8400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00b49a0 con 0x7f7fb00b3f70 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb5a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7fb015fc10 0x7f7fb015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34912/0 (socket says 172.21.15.97:34912) 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb5a00640 1 -- 172.21.15.97:0/923093883 learned_addr learned my addr 172.21.15.97:0/923093883 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.528+0000 7f7fb6400640 1 --2- 172.21.15.97:0/923093883 >> v2:172.21.15.97:3300/0 conn(0x7f7fb0070ff0 0x7f7fb00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.529+0000 7f7fb5000640 1 --2- 172.21.15.97:0/923093883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb00b3f70 0x7f7fb0160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.529+0000 7f7fb6400640 1 -- 172.21.15.97:0/923093883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb00b3f70 msgr2=0x7f7fb0160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.529+0000 7f7fb6400640 1 --2- 172.21.15.97:0/923093883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb00b3f70 0x7f7fb0160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.529+0000 7f7fb6400640 1 -- 172.21.15.97:0/923093883 >> v1:172.21.15.97:6789/0 conn(0x7f7fb015fc10 legacy=0x7f7fb015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.529+0000 7f7fb6400640 1 -- 172.21.15.97:0/923093883 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb0072c90 con 0x7f7fb0070ff0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.529+0000 7f7fafe00640 1 -- 172.21.15.97:0/923093883 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208698808 0 0) 0x7f7fb00b46c0 con 0x7f7fb015fc10 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.532+0000 7f7fb6400640 1 --2- 172.21.15.97:0/923093883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00ce4a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009d90 tx=0x7f7fa802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc01b8c87a18e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.532+0000 7f7fafe00640 1 -- 172.21.15.97:0/923093883 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8039070 con 0x7f7fb0070ff0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.532+0000 7f7fafe00640 1 -- 172.21.15.97:0/923093883 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa802cc50 con 0x7f7fb0070ff0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fafe00640 1 -- 172.21.15.97:0/923093883 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8034440 con 0x7f7fb0070ff0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 -- 172.21.15.97:0/923093883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 msgr2=0x7f7fb00ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 --2- 172.21.15.97:0/923093883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00ce4a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009d90 tx=0x7f7fa802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 -- 172.21.15.97:0/923093883 shutdown_connections 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 --2- 172.21.15.97:0/923093883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00ce4a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 --2- 172.21.15.97:0/923093883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb00b3f70 0x7f7fb0160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 -- 172.21.15.97:0/923093883 >> 172.21.15.97:0/923093883 conn(0x7f7fb00b1940 msgr2=0x7f7fb00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 -- 172.21.15.97:0/923093883 shutdown_connections 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.533+0000 7f7fb8400640 1 -- 172.21.15.97:0/923093883 wait complete. 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.535+0000 7f7fb8400640 1 Processor -- start 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.535+0000 7f7fb8400640 1 -- start start 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.536+0000 7f7fb8400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.536+0000 7f7fb8400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7fb00b3f70 0x7f7fb014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.536+0000 7f7fb8400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb015fc10 0x7f7fb00ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.536+0000 7f7fb8400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0072a30 con 0x7f7fb0070ff0 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.536+0000 7f7fb8400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00728b0 con 0x7f7fb00b3f70 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.536+0000 7f7fb8400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb015c2e0 con 0x7f7fb015fc10 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39220/0 (socket says 172.21.15.97:39220) 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 -- 172.21.15.97:0/2367499146 learned_addr learned my addr 172.21.15.97:0/2367499146 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb6400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb015fc10 0x7f7fb00ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 -- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb015fc10 msgr2=0x7f7fb00ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5000640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7fb00b3f70 0x7f7fb014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb015fc10 0x7f7fb00ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 -- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7fb00b3f70 msgr2=0x7f7fb014dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7fb00b3f70 0x7f7fb014dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 -- 172.21.15.97:0/2367499146 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb00c9e90 con 0x7f7fb0070ff0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb6400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb015fc10 0x7f7fb00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.537+0000 7f7fb5a00640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00c9640 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c500 tx=0x7f7fa400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.538+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4018070 con 0x7f7fb0070ff0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.538+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4007580 con 0x7f7fb0070ff0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.538+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb00ca170 con 0x7f7fb0070ff0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.538+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4004a20 con 0x7f7fb0070ff0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.538+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb006cec0 con 0x7f7fb0070ff0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.539+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb006c9d0 con 0x7f7fb0070ff0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.540+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fa401b070 con 0x7f7fb0070ff0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.541+0000 7f7fad600640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f9006b4d0 0x7f7f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.541+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fa4089540 con 0x7f7fb0070ff0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.541+0000 7f7fb5000640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f9006b4d0 0x7f7f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.542+0000 7f7fb5000640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f9006b4d0 0x7f7f9006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0151df0 tx=0x7f7f9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.551+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fa40583b0 con 0x7f7fb0070ff0 2024-03-06T07:28:19.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.745+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7fb004f330 con 0x7f7fb0070ff0 2024-03-06T07:28:19.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.746+0000 7f7fad600640 1 -- 172.21.15.97:0/2367499146 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7fa4057cb0 con 0x7f7fb0070ff0 2024-03-06T07:28:19.747 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.748+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f9006b4d0 msgr2=0x7f7f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.748+0000 7f7fb8400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f9006b4d0 0x7f7f9006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0151df0 tx=0x7f7f9c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.748+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 msgr2=0x7f7fb00c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.748+0000 7f7fb8400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00c9640 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c500 tx=0x7f7fa400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 shutdown_connections 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f9006b4d0 0x7f7f9006d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7fb015fc10 0x7f7fb00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7fb00b3f70 0x7f7fb014dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 --2- 172.21.15.97:0/2367499146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7fb0070ff0 0x7f7fb00c9640 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 >> 172.21.15.97:0/2367499146 conn(0x7f7fb00b1940 msgr2=0x7f7fb00acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 shutdown_connections 2024-03-06T07:28:19.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.749+0000 7f7fb8400640 1 -- 172.21.15.97:0/2367499146 wait complete. 2024-03-06T07:28:19.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:19.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.963+0000 7f6417400640 1 Processor -- start 2024-03-06T07:28:19.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.964+0000 7f6417400640 1 -- start start 2024-03-06T07:28:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6417400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f64100b3f70 0x7f6410160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6417400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410071440 con 0x7f6410070ff0 2024-03-06T07:28:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6417400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100b46c0 con 0x7f641015fc10 2024-03-06T07:28:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6417400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64100b49a0 con 0x7f64100b3f70 2024-03-06T07:28:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6407e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f64100b3f70 0x7f6410160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6407e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f64100b3f70 0x7f6410160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39236/0 (socket says 172.21.15.97:39236) 2024-03-06T07:28:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.965+0000 7f6407e00640 1 -- 172.21.15.97:0/4111335113 learned_addr learned my addr 172.21.15.97:0/4111335113 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.966+0000 7f6407e00640 1 -- 172.21.15.97:0/4111335113 >> v1:172.21.15.135:6789/0 conn(0x7f641015fc10 legacy=0x7f641015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.966+0000 7f6407e00640 1 -- 172.21.15.97:0/4111335113 >> v1:172.21.15.97:6789/0 conn(0x7f6410070ff0 legacy=0x7f64100ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.966+0000 7f6407e00640 1 -- 172.21.15.97:0/4111335113 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410072c90 con 0x7f64100b3f70 2024-03-06T07:28:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.967+0000 7f6407e00640 1 --2- 172.21.15.97:0/4111335113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64100b3f70 0x7f6410160700 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009870 tx=0x7f63f802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04a491778e3efe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.967+0000 7f640fe00640 1 -- 172.21.15.97:0/4111335113 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f802f850 con 0x7f64100b3f70 2024-03-06T07:28:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.967+0000 7f640fe00640 1 -- 172.21.15.97:0/4111335113 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f802f9f0 con 0x7f64100b3f70 2024-03-06T07:28:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.967+0000 7f640fe00640 1 -- 172.21.15.97:0/4111335113 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f802fdc0 con 0x7f64100b3f70 2024-03-06T07:28:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 -- 172.21.15.97:0/4111335113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64100b3f70 msgr2=0x7f6410160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 --2- 172.21.15.97:0/4111335113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64100b3f70 0x7f6410160700 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009870 tx=0x7f63f802ba30 comp rx=0 tx=0).stop 2024-03-06T07:28:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 -- 172.21.15.97:0/4111335113 shutdown_connections 2024-03-06T07:28:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 --2- 172.21.15.97:0/4111335113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64100b3f70 0x7f6410160700 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 -- 172.21.15.97:0/4111335113 >> 172.21.15.97:0/4111335113 conn(0x7f64100b1940 msgr2=0x7f64100b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 -- 172.21.15.97:0/4111335113 shutdown_connections 2024-03-06T07:28:19.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.968+0000 7f6417400640 1 -- 172.21.15.97:0/4111335113 wait complete. 2024-03-06T07:28:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.970+0000 7f6417400640 1 Processor -- start 2024-03-06T07:28:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.970+0000 7f6417400640 1 -- start start 2024-03-06T07:28:19.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6417400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 0x7f641006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6417400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64100b3f70 0x7f641015c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6417400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f641015fc10 0x7f64101f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6417400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6410072a30 con 0x7f6410070ff0 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6417400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100728b0 con 0x7f64100b3f70 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6417400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641015c2e0 con 0x7f641015fc10 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6414a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 0x7f641006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6414a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 0x7f641006d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39248/0 (socket says 172.21.15.97:39248) 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.971+0000 7f6414a00640 1 -- 172.21.15.97:0/3528003130 learned_addr learned my addr 172.21.15.97:0/3528003130 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6407e00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64100b3f70 0x7f641015c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6414a00640 1 -- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f641015fc10 msgr2=0x7f64101f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6414a00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f641015fc10 0x7f64101f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6414a00640 1 -- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64100b3f70 msgr2=0x7f641015c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6414a00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64100b3f70 0x7f641015c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6414a00640 1 -- 172.21.15.97:0/3528003130 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641006d7e0 con 0x7f6410070ff0 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6407e00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64100b3f70 0x7f641015c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6414a00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 0x7f641006d2a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f640000c3c0 tx=0x7f640000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000052d0 con 0x7f6410070ff0 2024-03-06T07:28:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400005490 con 0x7f6410070ff0 2024-03-06T07:28:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400007260 con 0x7f6410070ff0 2024-03-06T07:28:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.972+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641015cb90 con 0x7f6410070ff0 2024-03-06T07:28:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.973+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641006dd80 con 0x7f6410070ff0 2024-03-06T07:28:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.974+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641004f330 con 0x7f6410070ff0 2024-03-06T07:28:19.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.976+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6400007400 con 0x7f6410070ff0 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.977+0000 7f640d600640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.977+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6400089710 con 0x7f6410070ff0 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.977+0000 7f6407e00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.978+0000 7f6407e00640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f63f8036000 tx=0x7f63f8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:19.981+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6400058590 con 0x7f6410070ff0 2024-03-06T07:28:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.176+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64100b2d40 con 0x7f6410070ff0 2024-03-06T07:28:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.176+0000 7f640d600640 1 -- 172.21.15.97:0/3528003130 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6400057e90 con 0x7f6410070ff0 2024-03-06T07:28:20.178 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 msgr2=0x7f63ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f63f8036000 tx=0x7f63f8039040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 msgr2=0x7f641006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 0x7f641006d2a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f640000c3c0 tx=0x7f640000c880 comp rx=0 tx=0).stop 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 shutdown_connections 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f641015fc10 0x7f64101f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64100b3f70 0x7f641015c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 --2- 172.21.15.97:0/3528003130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070ff0 0x7f641006d2a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 >> 172.21.15.97:0/3528003130 conn(0x7f64100b1940 msgr2=0x7f64100cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 shutdown_connections 2024-03-06T07:28:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.179+0000 7f6417400640 1 -- 172.21.15.97:0/3528003130 wait complete. 2024-03-06T07:28:20.210 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:28:20.211 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:20.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:20.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.421+0000 7fd8e5000640 1 Processor -- start 2024-03-06T07:28:20.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.421+0000 7fd8e5000640 1 -- start start 2024-03-06T07:28:20.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.422+0000 7fd8e5000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8e00d07b0 0x7fd8e00cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8e5000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd8e015f110 0x7fd8e00d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8e5000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e00d10d0 con 0x7fd8e0161c50 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8e5000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e00cfd50 con 0x7fd8e00d07b0 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8e5000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e00cf140 con 0x7fd8e015f110 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dde00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8e00d07b0 0x7fd8e00cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dde00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8e00d07b0 0x7fd8e00cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39256/0 (socket says 172.21.15.97:39256) 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dde00640 1 -- 172.21.15.97:0/783048184 learned_addr learned my addr 172.21.15.97:0/783048184 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dde00640 1 -- 172.21.15.97:0/783048184 >> v2:172.21.15.135:3300/0 conn(0x7fd8e015f110 msgr2=0x7fd8e00d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dd400640 1 --2- 172.21.15.97:0/783048184 >> v2:172.21.15.135:3300/0 conn(0x7fd8e015f110 0x7fd8e00d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dde00640 1 --2- 172.21.15.97:0/783048184 >> v2:172.21.15.135:3300/0 conn(0x7fd8e015f110 0x7fd8e00d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.423+0000 7fd8dde00640 1 -- 172.21.15.97:0/783048184 >> v1:172.21.15.135:6789/0 conn(0x7fd8e0161c50 legacy=0x7fd8e01620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.424+0000 7fd8dde00640 1 -- 172.21.15.97:0/783048184 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00cf420 con 0x7fd8e00d07b0 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.424+0000 7fd8dd400640 1 --2- 172.21.15.97:0/783048184 >> v2:172.21.15.135:3300/0 conn(0x7fd8e015f110 0x7fd8e00d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.425+0000 7fd8dde00640 1 --2- 172.21.15.97:0/783048184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00cea80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8e008cdd0 tx=0x7fd8d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d318226a60b39d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.425+0000 7fd8dca00640 1 -- 172.21.15.97:0/783048184 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0035020 con 0x7fd8e00d07b0 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.425+0000 7fd8dca00640 1 -- 172.21.15.97:0/783048184 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d0031870 con 0x7fd8e00d07b0 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.425+0000 7fd8dca00640 1 -- 172.21.15.97:0/783048184 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0031c40 con 0x7fd8e00d07b0 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 -- 172.21.15.97:0/783048184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 msgr2=0x7fd8e00cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 --2- 172.21.15.97:0/783048184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00cea80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8e008cdd0 tx=0x7fd8d002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 -- 172.21.15.97:0/783048184 shutdown_connections 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 --2- 172.21.15.97:0/783048184 >> v2:172.21.15.135:3300/0 conn(0x7fd8e015f110 0x7fd8e00d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 --2- 172.21.15.97:0/783048184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00cea80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 -- 172.21.15.97:0/783048184 >> 172.21.15.97:0/783048184 conn(0x7fd8e00d3ee0 msgr2=0x7fd8e00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 -- 172.21.15.97:0/783048184 shutdown_connections 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.426+0000 7fd8e5000640 1 -- 172.21.15.97:0/783048184 wait complete. 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.427+0000 7fd8e5000640 1 Processor -- start 2024-03-06T07:28:20.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.428+0000 7fd8e5000640 1 -- start start 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8e5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8e5000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8e015f110 0x7fd8e014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8e5000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8e0161c50 0x7fd8e00c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8e5000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e015f570 con 0x7fd8e00d07b0 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8e5000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00cf9e0 con 0x7fd8e015f110 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8e5000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e015e390 con 0x7fd8e0161c50 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00b7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39272/0 (socket says 172.21.15.97:39272) 2024-03-06T07:28:20.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 -- 172.21.15.97:0/1165410335 learned_addr learned my addr 172.21.15.97:0/1165410335 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 -- 172.21.15.97:0/1165410335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8e0161c50 msgr2=0x7fd8e00c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8e0161c50 0x7fd8e00c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 -- 172.21.15.97:0/1165410335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8e015f110 msgr2=0x7fd8e014e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8e015f110 0x7fd8e014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.429+0000 7fd8dde00640 1 -- 172.21.15.97:0/1165410335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00b9b50 con 0x7fd8e00d07b0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.430+0000 7fd8dde00640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00b7ea0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd8d002b950 tx=0x7fd8d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.430+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0040070 con 0x7fd8e00d07b0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.430+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d003a420 con 0x7fd8e00d07b0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.430+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0035040 con 0x7fd8e00d07b0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.430+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e00c8d50 con 0x7fd8e00d07b0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.430+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e00c9310 con 0x7fd8e00d07b0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.431+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e004f330 con 0x7fd8e00d07b0 2024-03-06T07:28:20.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.433+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8d004b060 con 0x7fd8e00d07b0 2024-03-06T07:28:20.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.434+0000 7fd8d6000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8c006b4d0 0x7fd8c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.434+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8d00ac240 con 0x7fd8e00d07b0 2024-03-06T07:28:20.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.434+0000 7fd8dd400640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8c006b4d0 0x7fd8c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.435+0000 7fd8dd400640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8c006b4d0 0x7fd8c006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc006fd0 tx=0x7fd8cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.439+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8e00b9b50 con 0x7fd8e00d07b0 2024-03-06T07:28:20.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.658+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8e008d940 con 0x7fd8e00d07b0 2024-03-06T07:28:20.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.661+0000 7fd8d6000640 1 -- 172.21.15.97:0/1165410335 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fd8d007b020 con 0x7fd8e00d07b0 2024-03-06T07:28:20.662 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:20.662 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8c006b4d0 msgr2=0x7fd8c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8c006b4d0 0x7fd8c006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc006fd0 tx=0x7fd8cc008040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 msgr2=0x7fd8e00b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00b7ea0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd8d002b950 tx=0x7fd8d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 shutdown_connections 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8c006b4d0 0x7fd8c006d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8e0161c50 0x7fd8e00c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8e015f110 0x7fd8e014e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 --2- 172.21.15.97:0/1165410335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8e00d07b0 0x7fd8e00b7ea0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 >> 172.21.15.97:0/1165410335 conn(0x7fd8e00d3ee0 msgr2=0x7fd8e006d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.663+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 shutdown_connections 2024-03-06T07:28:20.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.664+0000 7fd8e5000640 1 -- 172.21.15.97:0/1165410335 wait complete. 2024-03-06T07:28:20.684 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:20.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:20.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.887+0000 7fd903200640 1 Processor -- start 2024-03-06T07:28:20.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.887+0000 7fd903200640 1 -- start start 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd903200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd90406da40 0x7fd9040ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd903200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90415c1f0 con 0x7fd9040aef20 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd903200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90406cac0 con 0x7fd90415be10 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd903200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9040cd200 con 0x7fd90406da40 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd8fbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd90406da40 0x7fd9040ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd8fbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd90406da40 0x7fd9040ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39286/0 (socket says 172.21.15.97:39286) 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd8fbe00640 1 -- 172.21.15.97:0/930218058 learned_addr learned my addr 172.21.15.97:0/930218058 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd902800640 1 --1- 172.21.15.97:0/930218058 >> v1:172.21.15.97:6789/0 conn(0x7fd9040aef20 0x7fd904071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34942/0 (socket says 172.21.15.97:34942) 2024-03-06T07:28:20.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd8fbe00640 1 -- 172.21.15.97:0/930218058 >> v1:172.21.15.135:6789/0 conn(0x7fd90415be10 legacy=0x7fd90406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd8fbe00640 1 -- 172.21.15.97:0/930218058 >> v1:172.21.15.97:6789/0 conn(0x7fd9040aef20 legacy=0x7fd904071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.889+0000 7fd8fbe00640 1 -- 172.21.15.97:0/930218058 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90406cda0 con 0x7fd90406da40 2024-03-06T07:28:20.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.890+0000 7fd8fbe00640 1 --2- 172.21.15.97:0/930218058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90406da40 0x7fd9040ccc20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbfa566b70c007d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.891+0000 7fd901400640 1 -- 172.21.15.97:0/930218058 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec035020 con 0x7fd90406da40 2024-03-06T07:28:20.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.891+0000 7fd901400640 1 -- 172.21.15.97:0/930218058 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec031870 con 0x7fd90406da40 2024-03-06T07:28:20.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.891+0000 7fd901400640 1 -- 172.21.15.97:0/930218058 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec031c40 con 0x7fd90406da40 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.891+0000 7fd903200640 1 -- 172.21.15.97:0/930218058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90406da40 msgr2=0x7fd9040ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.891+0000 7fd903200640 1 --2- 172.21.15.97:0/930218058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90406da40 0x7fd9040ccc20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.892+0000 7fd903200640 1 -- 172.21.15.97:0/930218058 shutdown_connections 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.892+0000 7fd903200640 1 --2- 172.21.15.97:0/930218058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90406da40 0x7fd9040ccc20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.892+0000 7fd903200640 1 -- 172.21.15.97:0/930218058 >> 172.21.15.97:0/930218058 conn(0x7fd90406afc0 msgr2=0x7fd90406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.892+0000 7fd903200640 1 -- 172.21.15.97:0/930218058 shutdown_connections 2024-03-06T07:28:20.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.892+0000 7fd903200640 1 -- 172.21.15.97:0/930218058 wait complete. 2024-03-06T07:28:20.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.893+0000 7fd903200640 1 Processor -- start 2024-03-06T07:28:20.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.893+0000 7fd903200640 1 -- start start 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.894+0000 7fd903200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd90406da40 0x7fd90414c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.894+0000 7fd903200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 0x7fd904151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.894+0000 7fd903200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd90415be10 0x7fd904159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.894+0000 7fd903200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040cda90 con 0x7fd9040aef20 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.894+0000 7fd903200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90406d080 con 0x7fd90406da40 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.894+0000 7fd903200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ce2b0 con 0x7fd90415be10 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 0x7fd904151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 0x7fd904151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39296/0 (socket says 172.21.15.97:39296) 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 -- 172.21.15.97:0/1902753254 learned_addr learned my addr 172.21.15.97:0/1902753254 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 -- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd90415be10 msgr2=0x7fd904159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd90415be10 0x7fd904159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 -- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd90406da40 msgr2=0x7fd90414c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd901e00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd90406da40 0x7fd90414c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd90406da40 0x7fd90414c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd8fbe00640 1 -- 172.21.15.97:0/1902753254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904151b30 con 0x7fd9040aef20 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.895+0000 7fd901e00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd90406da40 0x7fd90414c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.896+0000 7fd8fbe00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 0x7fd904151230 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.896+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec040070 con 0x7fd9040aef20 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.896+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec030e50 con 0x7fd9040aef20 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.896+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9041f71b0 con 0x7fd9040aef20 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.896+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec035040 con 0x7fd9040aef20 2024-03-06T07:28:20.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.896+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90414ca80 con 0x7fd9040aef20 2024-03-06T07:28:20.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.897+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9041f6fa0 con 0x7fd9040aef20 2024-03-06T07:28:20.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.898+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8ec04b060 con 0x7fd9040aef20 2024-03-06T07:28:20.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.899+0000 7fd8fa000640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.899+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8ec0abff0 con 0x7fd9040aef20 2024-03-06T07:28:20.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.899+0000 7fd901e00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.900+0000 7fd901e00640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8e006b4d0 0x7fd8e006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400a830 tx=0x7fd8f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:20.904+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8ec02f320 con 0x7fd9040aef20 2024-03-06T07:28:21.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.108+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd90404f330 con 0x7fd9040aef20 2024-03-06T07:28:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.109+0000 7fd8fa000640 1 -- 172.21.15.97:0/1902753254 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8ec07af00 con 0x7fd9040aef20 2024-03-06T07:28:21.111 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8e006b4d0 msgr2=0x7fd8e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8e006b4d0 0x7fd8e006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400a830 tx=0x7fd8f4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 msgr2=0x7fd904151230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 0x7fd904151230 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 shutdown_connections 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd90415be10 0x7fd904159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9040aef20 0x7fd904151230 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.112+0000 7fd903200640 1 --2- 172.21.15.97:0/1902753254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd90406da40 0x7fd90414c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.113+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 >> 172.21.15.97:0/1902753254 conn(0x7fd90406afc0 msgr2=0x7fd904153dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.113+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 shutdown_connections 2024-03-06T07:28:21.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.113+0000 7fd903200640 1 -- 172.21.15.97:0/1902753254 wait complete. 2024-03-06T07:28:21.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:21.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.330+0000 7ff295e00640 1 Processor -- start 2024-03-06T07:28:21.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.330+0000 7ff295e00640 1 -- start start 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.332+0000 7ff295e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff29015e7a0 0x7ff29006bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.332+0000 7ff295e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006b480 0x7ff29006c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.332+0000 7ff295e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff29006afc0 0x7ff2900b5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.332+0000 7ff295e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2900b2a40 con 0x7ff29006afc0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.332+0000 7ff295e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29015eb80 con 0x7ff29015e7a0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.332+0000 7ff295e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900b14d0 con 0x7ff29006b480 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff29006afc0 0x7ff2900b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff29006afc0 0x7ff2900b5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39302/0 (socket says 172.21.15.97:39302) 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 -- 172.21.15.97:0/1927174674 learned_addr learned my addr 172.21.15.97:0/1927174674 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28ea00640 1 --2- 172.21.15.97:0/1927174674 >> v2:172.21.15.135:3300/0 conn(0x7ff29015e7a0 0x7ff29006bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28e000640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006b480 0x7ff29006c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 -- 172.21.15.97:0/1927174674 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006b480 msgr2=0x7ff29006c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006b480 0x7ff29006c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 -- 172.21.15.97:0/1927174674 >> v2:172.21.15.135:3300/0 conn(0x7ff29015e7a0 msgr2=0x7ff29006bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 --2- 172.21.15.97:0/1927174674 >> v2:172.21.15.135:3300/0 conn(0x7ff29015e7a0 0x7ff29006bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28f400640 1 -- 172.21.15.97:0/1927174674 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290073050 con 0x7ff29006afc0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.333+0000 7ff28e000640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006b480 0x7ff29006c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.335+0000 7ff28f400640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006afc0 0x7ff2900b5020 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff2900705f0 tx=0x7ff28802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465ba7193d0166fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.335+0000 7ff28d600640 1 -- 172.21.15.97:0/1927174674 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28802c9e0 con 0x7ff29006afc0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.335+0000 7ff28d600640 1 -- 172.21.15.97:0/1927174674 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28802cb80 con 0x7ff29006afc0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.335+0000 7ff28d600640 1 -- 172.21.15.97:0/1927174674 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2880343f0 con 0x7ff29006afc0 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 -- 172.21.15.97:0/1927174674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006afc0 msgr2=0x7ff2900b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006afc0 0x7ff2900b5020 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff2900705f0 tx=0x7ff28802be40 comp rx=0 tx=0).stop 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 -- 172.21.15.97:0/1927174674 shutdown_connections 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006afc0 0x7ff2900b5020 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 --2- 172.21.15.97:0/1927174674 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006b480 0x7ff29006c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 --2- 172.21.15.97:0/1927174674 >> v2:172.21.15.135:3300/0 conn(0x7ff29015e7a0 0x7ff29006bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 -- 172.21.15.97:0/1927174674 >> 172.21.15.97:0/1927174674 conn(0x7ff2900b1e80 msgr2=0x7ff2900b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 -- 172.21.15.97:0/1927174674 shutdown_connections 2024-03-06T07:28:21.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.336+0000 7ff295e00640 1 -- 172.21.15.97:0/1927174674 wait complete. 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.338+0000 7ff295e00640 1 Processor -- start 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.338+0000 7ff295e00640 1 -- start start 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff295e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006afc0 0x7ff2900c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff295e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 0x7ff2900ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff295e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff29015e7a0 0x7ff2901f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff295e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900b19f0 con 0x7ff29006b480 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff295e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff290073360 con 0x7ff29015e7a0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff295e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900b1b70 con 0x7ff29006afc0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff28e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 0x7ff2900ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff28e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 0x7ff2900ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39310/0 (socket says 172.21.15.97:39310) 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.339+0000 7ff28e000640 1 -- 172.21.15.97:0/3176936009 learned_addr learned my addr 172.21.15.97:0/3176936009 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28f400640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff29015e7a0 0x7ff2901f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28ea00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006afc0 0x7ff2900c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28e000640 1 -- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006afc0 msgr2=0x7ff2900c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28e000640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006afc0 0x7ff2900c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28e000640 1 -- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff29015e7a0 msgr2=0x7ff2901f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28e000640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff29015e7a0 0x7ff2901f2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28e000640 1 -- 172.21.15.97:0/3176936009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2900cd330 con 0x7ff29006b480 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff28e000640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 0x7ff2900ae420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00c7c0 tx=0x7ff27c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27c015070 con 0x7ff29006b480 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27c007820 con 0x7ff29006b480 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27c007bf0 con 0x7ff29006b480 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.340+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2900cd610 con 0x7ff29006b480 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.341+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2900cdbd0 con 0x7ff29006b480 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.342+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29004f330 con 0x7ff29006b480 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.343+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff27c018070 con 0x7ff29006b480 2024-03-06T07:28:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.343+0000 7ff287400640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff27006b4d0 0x7ff27006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.343+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff27c08a0f0 con 0x7ff29006b480 2024-03-06T07:28:21.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.344+0000 7ff28ea00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff27006b4d0 0x7ff27006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.344+0000 7ff28ea00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff27006b4d0 0x7ff27006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff280006fd0 tx=0x7ff280008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.348+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff27c058f60 con 0x7ff29006b480 2024-03-06T07:28:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.542+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff2900b6030 con 0x7ff29006b480 2024-03-06T07:28:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.543+0000 7ff287400640 1 -- 172.21.15.97:0/3176936009 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff27c058860 con 0x7ff29006b480 2024-03-06T07:28:21.544 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.545+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff27006b4d0 msgr2=0x7ff27006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.545+0000 7ff295e00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff27006b4d0 0x7ff27006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff280006fd0 tx=0x7ff280008040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.545+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 msgr2=0x7ff2900ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.545+0000 7ff295e00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 0x7ff2900ae420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00c7c0 tx=0x7ff27c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 shutdown_connections 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff27006b4d0 0x7ff27006d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff29015e7a0 0x7ff2901f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff29006b480 0x7ff2900ae420 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 --2- 172.21.15.97:0/3176936009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff29006afc0 0x7ff2900c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 >> 172.21.15.97:0/3176936009 conn(0x7ff2900b1e80 msgr2=0x7ff29015b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 shutdown_connections 2024-03-06T07:28:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.546+0000 7ff295e00640 1 -- 172.21.15.97:0/3176936009 wait complete. 2024-03-06T07:28:21.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.760+0000 7f73c2e00640 1 Processor -- start 2024-03-06T07:28:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.760+0000 7f73c2e00640 1 -- start start 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73c2e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73bc15e450 0x7f73bc0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73c2e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc0b3730 con 0x7f73bc0b5490 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73c2e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc067b20 con 0x7f73bc0d07f0 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73c2e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73bc0d4930 con 0x7f73bc15e450 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73bb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73bc15e450 0x7f73bc0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73c0e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f73bc0b5490 0x7f73bc0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34954/0 (socket says 172.21.15.97:34954) 2024-03-06T07:28:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73c0e00640 1 -- 172.21.15.97:0/1811114650 learned_addr learned my addr 172.21.15.97:0/1811114650 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73bb400640 1 -- 172.21.15.97:0/1811114650 >> v1:172.21.15.135:6789/0 conn(0x7f73bc0d07f0 legacy=0x7f73bc0d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73bb400640 1 -- 172.21.15.97:0/1811114650 >> v1:172.21.15.97:6789/0 conn(0x7f73bc0b5490 legacy=0x7f73bc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.762+0000 7f73bb400640 1 -- 172.21.15.97:0/1811114650 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc0d4b80 con 0x7f73bc15e450 2024-03-06T07:28:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.763+0000 7f73bb400640 1 --2- 172.21.15.97:0/1811114650 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc15e450 0x7f73bc0cea20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f73a40097a0 tx=0x7f73a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605b3f290bb3e7ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.764+0000 7f73baa00640 1 -- 172.21.15.97:0/1811114650 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40315a0 con 0x7f73bc15e450 2024-03-06T07:28:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.764+0000 7f73baa00640 1 -- 172.21.15.97:0/1811114650 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a4037070 con 0x7f73bc15e450 2024-03-06T07:28:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.764+0000 7f73baa00640 1 -- 172.21.15.97:0/1811114650 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4031960 con 0x7f73bc15e450 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 -- 172.21.15.97:0/1811114650 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc15e450 msgr2=0x7f73bc0cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 --2- 172.21.15.97:0/1811114650 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc15e450 0x7f73bc0cea20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f73a40097a0 tx=0x7f73a402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 -- 172.21.15.97:0/1811114650 shutdown_connections 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 --2- 172.21.15.97:0/1811114650 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc15e450 0x7f73bc0cea20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 -- 172.21.15.97:0/1811114650 >> 172.21.15.97:0/1811114650 conn(0x7f73bc0d3e10 msgr2=0x7f73bc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 -- 172.21.15.97:0/1811114650 shutdown_connections 2024-03-06T07:28:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.765+0000 7f73c2e00640 1 -- 172.21.15.97:0/1811114650 wait complete. 2024-03-06T07:28:21.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.767+0000 7f73c2e00640 1 Processor -- start 2024-03-06T07:28:21.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.767+0000 7f73c2e00640 1 -- start start 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73c2e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 0x7f73bc0b94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73c2e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73bc0d07f0 0x7f73bc154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73c2e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73bc15e450 0x7f73bc0c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73c2e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0b3f60 con 0x7f73bc0b5490 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73c2e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0b3c80 con 0x7f73bc15e450 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73c2e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0d1bf0 con 0x7f73bc0d07f0 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 0x7f73bc0b94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 0x7f73bc0b94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39326/0 (socket says 172.21.15.97:39326) 2024-03-06T07:28:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.769+0000 7f73bbe00640 1 -- 172.21.15.97:0/3705910828 learned_addr learned my addr 172.21.15.97:0/3705910828 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73c0e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73bc15e450 0x7f73bc0c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bb400640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73bc0d07f0 0x7f73bc154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bbe00640 1 -- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73bc0d07f0 msgr2=0x7f73bc154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bbe00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73bc0d07f0 0x7f73bc154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bbe00640 1 -- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73bc15e450 msgr2=0x7f73bc0c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bbe00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73bc15e450 0x7f73bc0c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bbe00640 1 -- 172.21.15.97:0/3705910828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc0b8720 con 0x7f73bc0b5490 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bb400640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73bc0d07f0 0x7f73bc154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73bbe00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 0x7f73bc0b94a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c7f0 tx=0x7f73ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac016020 con 0x7f73bc0b5490 2024-03-06T07:28:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac00a9f0 con 0x7f73bc0b5490 2024-03-06T07:28:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73bc0c99f0 con 0x7f73bc0b5490 2024-03-06T07:28:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.770+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00ade0 con 0x7f73bc0b5490 2024-03-06T07:28:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.771+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc0b8f80 con 0x7f73bc0b5490 2024-03-06T07:28:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.772+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc04f330 con 0x7f73bc0b5490 2024-03-06T07:28:21.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.773+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ac019020 con 0x7f73bc0b5490 2024-03-06T07:28:21.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.774+0000 7f73b3e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f739806b4d0 0x7f739806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.774+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73ac089fc0 con 0x7f73bc0b5490 2024-03-06T07:28:21.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.774+0000 7f73bb400640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f739806b4d0 0x7f739806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.775+0000 7f73bb400640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f739806b4d0 0x7f739806d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73a4036000 tx=0x7f73a4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.779+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73ac020070 con 0x7f73bc0b5490 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.976+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f73bc0746a0 con 0x7f73bc0b5490 2024-03-06T07:28:21.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.977+0000 7f73b3e00640 1 -- 172.21.15.97:0/3705910828 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f73bc0c99f0 con 0x7f73bc0b5490 2024-03-06T07:28:21.978 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f739806b4d0 msgr2=0x7f739806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f739806b4d0 0x7f739806d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73a4036000 tx=0x7f73a4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 msgr2=0x7f73bc0b94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 0x7f73bc0b94a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c7f0 tx=0x7f73ac00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 shutdown_connections 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f739806b4d0 0x7f739806d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73bc15e450 0x7f73bc0c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73bc0d07f0 0x7f73bc154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 --2- 172.21.15.97:0/3705910828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0b5490 0x7f73bc0b94a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.980+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 >> 172.21.15.97:0/3705910828 conn(0x7f73bc0d3e10 msgr2=0x7f73bc0b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.981+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 shutdown_connections 2024-03-06T07:28:21.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:21.981+0000 7f73c2e00640 1 -- 172.21.15.97:0/3705910828 wait complete. 2024-03-06T07:28:22.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:28:22.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.212+0000 7fc9f1200640 1 Processor -- start 2024-03-06T07:28:22.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.212+0000 7fc9f1200640 1 -- start start 2024-03-06T07:28:22.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.214+0000 7fc9f1200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc9ec156320 0x7fc9ec15f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9f1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc9ec15cd30 0x7fc9ec0ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9f1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec06b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9f1200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ec071dc0 con 0x7fc9ec156320 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9f1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ec06b940 con 0x7fc9ec15cd30 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9f1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec15fed0 con 0x7fc9ec06afc0 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc9ec156320 0x7fc9ec15f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc9ec156320 0x7fc9ec15f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39342/0 (socket says 172.21.15.97:39342) 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 -- 172.21.15.97:0/145755753 learned_addr learned my addr 172.21.15.97:0/145755753 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eaa00640 1 --2- 172.21.15.97:0/145755753 >> v2:172.21.15.135:3300/0 conn(0x7fc9ec15cd30 0x7fc9ec0ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 -- 172.21.15.97:0/145755753 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 msgr2=0x7fc9ec06b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 --2- 172.21.15.97:0/145755753 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec06b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 -- 172.21.15.97:0/145755753 >> v2:172.21.15.135:3300/0 conn(0x7fc9ec15cd30 msgr2=0x7fc9ec0ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 --2- 172.21.15.97:0/145755753 >> v2:172.21.15.135:3300/0 conn(0x7fc9ec15cd30 0x7fc9ec0ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eb400640 1 -- 172.21.15.97:0/145755753 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec1601b0 con 0x7fc9ec156320 2024-03-06T07:28:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.215+0000 7fc9eaa00640 1 --2- 172.21.15.97:0/145755753 >> v2:172.21.15.135:3300/0 conn(0x7fc9ec15cd30 0x7fc9ec0ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.217+0000 7fc9eb400640 1 --2- 172.21.15.97:0/145755753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec15f8f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0b6000 tx=0x7fc9dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a9f762d2bb3d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.217+0000 7fc9ea000640 1 -- 172.21.15.97:0/145755753 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc02e070 con 0x7fc9ec156320 2024-03-06T07:28:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.217+0000 7fc9ea000640 1 -- 172.21.15.97:0/145755753 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9dc037070 con 0x7fc9ec156320 2024-03-06T07:28:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.217+0000 7fc9ea000640 1 -- 172.21.15.97:0/145755753 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc031a00 con 0x7fc9ec156320 2024-03-06T07:28:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 -- 172.21.15.97:0/145755753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 msgr2=0x7fc9ec15f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 --2- 172.21.15.97:0/145755753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec15f8f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0b6000 tx=0x7fc9dc02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 -- 172.21.15.97:0/145755753 shutdown_connections 2024-03-06T07:28:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 --2- 172.21.15.97:0/145755753 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec06b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 --2- 172.21.15.97:0/145755753 >> v2:172.21.15.135:3300/0 conn(0x7fc9ec15cd30 0x7fc9ec0ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 --2- 172.21.15.97:0/145755753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec15f8f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 -- 172.21.15.97:0/145755753 >> 172.21.15.97:0/145755753 conn(0x7fc9ec06c0d0 msgr2=0x7fc9ec06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 -- 172.21.15.97:0/145755753 shutdown_connections 2024-03-06T07:28:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.218+0000 7fc9f1200640 1 -- 172.21.15.97:0/145755753 wait complete. 2024-03-06T07:28:22.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.220+0000 7fc9f1200640 1 Processor -- start 2024-03-06T07:28:22.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.220+0000 7fc9f1200640 1 -- start start 2024-03-06T07:28:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9f1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec0b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9f1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec1402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9f1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9ec15cd30 0x7fc9ec0b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9f1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec156700 con 0x7fc9ec156320 2024-03-06T07:28:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9f1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec160770 con 0x7fc9ec15cd30 2024-03-06T07:28:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9f1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec15d1e0 con 0x7fc9ec06afc0 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9eaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec1402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9eaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec1402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39344/0 (socket says 172.21.15.97:39344) 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9eaa00640 1 -- 172.21.15.97:0/80609869 learned_addr learned my addr 172.21.15.97:0/80609869 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.221+0000 7fc9eb400640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec0b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eaa00640 1 -- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 msgr2=0x7fc9ec0b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eaa00640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec0b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eaa00640 1 -- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9ec15cd30 msgr2=0x7fc9ec0b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9ebe00640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9ec15cd30 0x7fc9ec0b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eaa00640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9ec15cd30 0x7fc9ec0b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eaa00640 1 -- 172.21.15.97:0/80609869 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec0b4190 con 0x7fc9ec156320 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eb400640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec0b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9eaa00640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec1402c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c7f0 tx=0x7fc9d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8015070 con 0x7fc9ec156320 2024-03-06T07:28:22.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d8007a30 con 0x7fc9ec156320 2024-03-06T07:28:22.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8007e00 con 0x7fc9ec156320 2024-03-06T07:28:22.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.222+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec0b4470 con 0x7fc9ec156320 2024-03-06T07:28:22.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.223+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec0b4a30 con 0x7fc9ec156320 2024-03-06T07:28:22.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.224+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec04f330 con 0x7fc9ec156320 2024-03-06T07:28:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.225+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9d8018070 con 0x7fc9ec156320 2024-03-06T07:28:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.225+0000 7fc9e3400640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc9cc06b4d0 0x7fc9cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.226+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9d808a170 con 0x7fc9ec156320 2024-03-06T07:28:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.226+0000 7fc9eb400640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc9cc06b4d0 0x7fc9cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.227+0000 7fc9eb400640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc9cc06b4d0 0x7fc9cc06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc036000 tx=0x7fc9dc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.230+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9d8059080 con 0x7fc9ec156320 2024-03-06T07:28:22.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:22.520+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fc9ec0cd3f0 con 0x7fc9ec156320 2024-03-06T07:28:23.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.321+0000 7fc9e3400640 1 -- 172.21.15.97:0/80609869 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v59) v1 ==== 135+0+0 (secure 0 0 0) 0x7fc9d8058980 con 0x7fc9ec156320 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.324+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc9cc06b4d0 msgr2=0x7fc9cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.324+0000 7fc9f1200640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc9cc06b4d0 0x7fc9cc06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc036000 tx=0x7fc9dc008030 comp rx=0 tx=0).stop 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 msgr2=0x7fc9ec1402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec1402c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c7f0 tx=0x7fc9d800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 shutdown_connections 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc9cc06b4d0 0x7fc9cc06d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9ec15cd30 0x7fc9ec0b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9ec156320 0x7fc9ec1402c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 --2- 172.21.15.97:0/80609869 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc9ec06afc0 0x7fc9ec0b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 >> 172.21.15.97:0/80609869 conn(0x7fc9ec06c0d0 msgr2=0x7fc9ec06ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 shutdown_connections 2024-03-06T07:28:23.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.325+0000 7fc9f1200640 1 -- 172.21.15.97:0/80609869 wait complete. 2024-03-06T07:28:23.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:23.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.544+0000 7fdaf4200640 1 Processor -- start 2024-03-06T07:28:23.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.545+0000 7fdaf4200640 1 -- start start 2024-03-06T07:28:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.546+0000 7fdaf4200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdaec15fc10 0x7fdaec070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.546+0000 7fdaf4200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b46c0 con 0x7fdaec0b3f70 2024-03-06T07:28:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.546+0000 7fdaf4200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b49a0 con 0x7fdaec15fff0 2024-03-06T07:28:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.546+0000 7fdaf4200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec072c90 con 0x7fdaec15fc10 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.546+0000 7fdaf2200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdaec0b3f70 0x7fdaec0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34960/0 (socket says 172.21.15.97:34960) 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.546+0000 7fdaf2200640 1 -- 172.21.15.97:0/2866949703 learned_addr learned my addr 172.21.15.97:0/2866949703 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaf1800640 1 --2- 172.21.15.97:0/2866949703 >> v2:172.21.15.135:3300/0 conn(0x7fdaec15fc10 0x7fdaec070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601421034 0 0) 0x7fdaec0b46c0 con 0x7fdaec0b3f70 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad4003520 con 0x7fdaec0b3f70 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1306107670 0 0) 0x7fdad4003520 con 0x7fdaec0b3f70 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 >> v2:172.21.15.135:3300/0 conn(0x7fdaec15fc10 msgr2=0x7fdaec070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 --2- 172.21.15.97:0/2866949703 >> v2:172.21.15.135:3300/0 conn(0x7fdaec15fc10 0x7fdaec070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 >> v1:172.21.15.135:6789/0 conn(0x7fdaec15fff0 legacy=0x7fdaec160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec09dd80 con 0x7fdaec0b3f70 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdae40019f0 con 0x7fdaec0b3f70 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 >> v1:172.21.15.97:6789/0 conn(0x7fdaec0b3f70 legacy=0x7fdaec0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdad40039b0 0x7fdad4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdad4004800 0x7fdad4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.547+0000 7fdaebe00640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40051c0 0x7fdad400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec072c90 con 0x7fdad40039b0 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad400c040 con 0x7fdad40051c0 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad400c240 con 0x7fdad4004800 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf1800640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdad40039b0 0x7fdad4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf1800640 1 -- 172.21.15.97:0/2866949703 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdad4004800 msgr2=0x7fdad4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf1800640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdad4004800 0x7fdad4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf1800640 1 -- 172.21.15.97:0/2866949703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40051c0 msgr2=0x7fdad400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf1800640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40051c0 0x7fdad400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf1800640 1 -- 172.21.15.97:0/2866949703 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad4003520 con 0x7fdad40039b0 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.548+0000 7fdaf2200640 1 -- 172.21.15.97:0/2866949703 reap_dead start 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.549+0000 7fdaf1800640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdad40039b0 0x7fdad4003db0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009ac0 tx=0x7fdad4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4b95dbf989a9d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.549+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0039070 con 0x7fdad40039b0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.549+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae002c710 con 0x7fdad40039b0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.549+0000 7fdaebe00640 1 -- 172.21.15.97:0/2866949703 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0009690 con 0x7fdad40039b0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 -- 172.21.15.97:0/2866949703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdad40039b0 msgr2=0x7fdad4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdad40039b0 0x7fdad4003db0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009ac0 tx=0x7fdad4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 -- 172.21.15.97:0/2866949703 shutdown_connections 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 --2- 172.21.15.97:0/2866949703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdad40039b0 0x7fdad4003db0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 -- 172.21.15.97:0/2866949703 >> 172.21.15.97:0/2866949703 conn(0x7fdaec0b1940 msgr2=0x7fdaec0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 -- 172.21.15.97:0/2866949703 shutdown_connections 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.550+0000 7fdaf4200640 1 -- 172.21.15.97:0/2866949703 wait complete. 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.552+0000 7fdaf4200640 1 Processor -- start 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.552+0000 7fdaf4200640 1 -- start start 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf4200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40039b0 0x7fdaec143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf4200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 0x7fdaec09ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf4200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdaec09bf80 0x7fdaec15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf4200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec15c460 con 0x7fdaec150bb0 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf4200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec15c2e0 con 0x7fdad40039b0 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf4200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec070ff0 con 0x7fdaec09bf80 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 0x7fdaec09ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 0x7fdaec09ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39360/0 (socket says 172.21.15.97:39360) 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.553+0000 7fdaf0e00640 1 -- 172.21.15.97:0/4222767198 learned_addr learned my addr 172.21.15.97:0/4222767198 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf1800640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40039b0 0x7fdaec143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf2200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdaec09bf80 0x7fdaec15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf0e00640 1 -- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdaec09bf80 msgr2=0x7fdaec15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf0e00640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdaec09bf80 0x7fdaec15b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf0e00640 1 -- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40039b0 msgr2=0x7fdaec143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf0e00640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40039b0 0x7fdaec143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf0e00640 1 -- 172.21.15.97:0/4222767198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec159670 con 0x7fdaec150bb0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf0e00640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 0x7fdaec09ba40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c3f0 tx=0x7fdad800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8007020 con 0x7fdaec150bb0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad80071e0 con 0x7fdaec150bb0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad80075d0 con 0x7fdaec150bb0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec09c400 con 0x7fdaec150bb0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.554+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec09c9c0 con 0x7fdaec150bb0 2024-03-06T07:28:23.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.556+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec04f330 con 0x7fdaec150bb0 2024-03-06T07:28:23.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.557+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdad8019020 con 0x7fdaec150bb0 2024-03-06T07:28:23.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.558+0000 7fdae9600640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdacc06b4d0 0x7fdacc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.558+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdad8089940 con 0x7fdaec150bb0 2024-03-06T07:28:23.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.559+0000 7fdaf1800640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdacc06b4d0 0x7fdacc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.559+0000 7fdaf1800640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdacc06b4d0 0x7fdacc06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdae0036000 tx=0x7fdae0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.563+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdad80587c0 con 0x7fdaec150bb0 2024-03-06T07:28:23.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.757+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdaec0b2d40 con 0x7fdaec150bb0 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.764+0000 7fdae9600640 1 -- 172.21.15.97:0/4222767198 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdad80580c0 con 0x7fdaec150bb0 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.766+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdacc06b4d0 msgr2=0x7fdacc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.766+0000 7fdaf4200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdacc06b4d0 0x7fdacc06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdae0036000 tx=0x7fdae0030000 comp rx=0 tx=0).stop 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.766+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 msgr2=0x7fdaec09ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.766+0000 7fdaf4200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 0x7fdaec09ba40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c3f0 tx=0x7fdad800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 shutdown_connections 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdacc06b4d0 0x7fdacc06d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdaec09bf80 0x7fdaec15b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdaec150bb0 0x7fdaec09ba40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 --2- 172.21.15.97:0/4222767198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdad40039b0 0x7fdaec143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 >> 172.21.15.97:0/4222767198 conn(0x7fdaec0b1940 msgr2=0x7fdaec156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 shutdown_connections 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:23.767+0000 7fdaf4200640 1 -- 172.21.15.97:0/4222767198 wait complete. 2024-03-06T07:28:23.786 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:23.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:23.889 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:24.004 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:24.005 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:24.005 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:24.059 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:24.059 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:24.105 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:24.106 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:24.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:24.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.390+0000 7f376fe00640 1 Processor -- start 2024-03-06T07:28:24.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.390+0000 7f376fe00640 1 -- start start 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.391+0000 7f376fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3768144c00 0x7f376806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.391+0000 7f376fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f376815fe60 0x7f37680ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.391+0000 7f376fe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376806ee70 con 0x7f376815f9d0 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.391+0000 7f376fe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376806cef0 con 0x7f3768144c00 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.391+0000 7f376fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376806d1d0 con 0x7f376815fe60 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3768144c00 0x7f376806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3768144c00 0x7f376806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46406/0 (socket says 172.21.15.97:46406) 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 -- 172.21.15.97:0/2051741028 learned_addr learned my addr 172.21.15.97:0/2051741028 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376de00640 1 --2- 172.21.15.97:0/2051741028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f376815fe60 0x7f37680ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 -- 172.21.15.97:0/2051741028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f376815fe60 msgr2=0x7f37680ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 --2- 172.21.15.97:0/2051741028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f376815fe60 0x7f37680ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 -- 172.21.15.97:0/2051741028 >> v1:172.21.15.135:6789/0 conn(0x7f376815f9d0 legacy=0x7f37680ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.392+0000 7f376d400640 1 -- 172.21.15.97:0/2051741028 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376806d4b0 con 0x7f3768144c00 2024-03-06T07:28:24.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.393+0000 7f376d400640 1 --2- 172.21.15.97:0/2051741028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3768144c00 0x7f376806e930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f375c009870 tx=0x7f375c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9eba7a9d52b0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.393+0000 7f3767e00640 1 -- 172.21.15.97:0/2051741028 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c035020 con 0x7f3768144c00 2024-03-06T07:28:24.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.393+0000 7f3767e00640 1 -- 172.21.15.97:0/2051741028 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c031640 con 0x7f3768144c00 2024-03-06T07:28:24.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.393+0000 7f3767e00640 1 -- 172.21.15.97:0/2051741028 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c02d040 con 0x7f3768144c00 2024-03-06T07:28:24.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 -- 172.21.15.97:0/2051741028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3768144c00 msgr2=0x7f376806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 --2- 172.21.15.97:0/2051741028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3768144c00 0x7f376806e930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f375c009870 tx=0x7f375c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 -- 172.21.15.97:0/2051741028 shutdown_connections 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 --2- 172.21.15.97:0/2051741028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f376815fe60 0x7f37680ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 --2- 172.21.15.97:0/2051741028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3768144c00 0x7f376806e930 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 -- 172.21.15.97:0/2051741028 >> 172.21.15.97:0/2051741028 conn(0x7f37680b1730 msgr2=0x7f37680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 -- 172.21.15.97:0/2051741028 shutdown_connections 2024-03-06T07:28:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.394+0000 7f376fe00640 1 -- 172.21.15.97:0/2051741028 wait complete. 2024-03-06T07:28:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.396+0000 7f376fe00640 1 Processor -- start 2024-03-06T07:28:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.396+0000 7f376fe00640 1 -- start start 2024-03-06T07:28:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3768144c00 0x7f376813e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f376815f9d0 0x7f37681f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 0x7f3768140190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3768160730 con 0x7f376815fe60 2024-03-06T07:28:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3768160290 con 0x7f376815f9d0 2024-03-06T07:28:24.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376806cb50 con 0x7f3768144c00 2024-03-06T07:28:24.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 0x7f3768140190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 0x7f3768140190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46420/0 (socket says 172.21.15.97:46420) 2024-03-06T07:28:24.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.397+0000 7f376de00640 1 -- 172.21.15.97:0/24691874 learned_addr learned my addr 172.21.15.97:0/24691874 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376d400640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3768144c00 0x7f376813e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376ca00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f376815f9d0 0x7f37681f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376de00640 1 -- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3768144c00 msgr2=0x7f376813e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376de00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3768144c00 0x7f376813e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376de00640 1 -- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f376815f9d0 msgr2=0x7f37681f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376de00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f376815f9d0 0x7f37681f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376de00640 1 -- 172.21.15.97:0/24691874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376813ed20 con 0x7f376815fe60 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376ca00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f376815f9d0 0x7f37681f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f376de00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 0x7f3768140190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f376000cd70 tx=0x7f376000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.398+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760003b90 con 0x7f376815fe60 2024-03-06T07:28:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.399+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3760003d30 con 0x7f376815fe60 2024-03-06T07:28:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.399+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760004240 con 0x7f376815fe60 2024-03-06T07:28:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.399+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37681f6490 con 0x7f376815fe60 2024-03-06T07:28:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.399+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37681f5750 con 0x7f376815fe60 2024-03-06T07:28:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.400+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376804f330 con 0x7f376815fe60 2024-03-06T07:28:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.401+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37600043e0 con 0x7f376815fe60 2024-03-06T07:28:24.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.402+0000 7f3765600640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f374806b500 0x7f374806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.402+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f376008ad60 con 0x7f376815fe60 2024-03-06T07:28:24.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.402+0000 7f376d400640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f374806b500 0x7f374806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.402+0000 7f376d400640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f374806b500 0x7f374806d9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f375c036000 tx=0x7f375c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.407+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3760059b40 con 0x7f376815fe60 2024-03-06T07:28:24.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.603+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37680aff70 con 0x7f376815fe60 2024-03-06T07:28:24.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.605+0000 7f3765600640 1 -- 172.21.15.97:0/24691874 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f3760059440 con 0x7f376815fe60 2024-03-06T07:28:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:24.607 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f374806b500 msgr2=0x7f374806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f374806b500 0x7f374806d9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f375c036000 tx=0x7f375c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 msgr2=0x7f3768140190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 0x7f3768140190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f376000cd70 tx=0x7f376000a660 comp rx=0 tx=0).stop 2024-03-06T07:28:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 shutdown_connections 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f374806b500 0x7f374806d9c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f376815fe60 0x7f3768140190 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f376815f9d0 0x7f37681f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 --2- 172.21.15.97:0/24691874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3768144c00 0x7f376813e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 >> 172.21.15.97:0/24691874 conn(0x7f37680b1730 msgr2=0x7f3768070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 shutdown_connections 2024-03-06T07:28:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.608+0000 7f376fe00640 1 -- 172.21.15.97:0/24691874 wait complete. 2024-03-06T07:28:24.630 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:24.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.840+0000 7f8df0000640 1 Processor -- start 2024-03-06T07:28:24.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.840+0000 7f8df0000640 1 -- start start 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.842+0000 7f8df0000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 0x7f8de806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.842+0000 7f8df0000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de80d1fb0 con 0x7f8de815e360 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.842+0000 7f8df0000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de809b100 con 0x7f8de80d02a0 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.842+0000 7f8df0000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80d2a20 con 0x7f8de815df80 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.843+0000 7f8decc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8de815e360 0x7f8de80cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39490/0 (socket says 172.21.15.97:39490) 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.843+0000 7f8decc00640 1 -- 172.21.15.97:0/401359617 learned_addr learned my addr 172.21.15.97:0/401359617 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.843+0000 7f8ded600640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 0x7f8de806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.843+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510951632 0 0) 0x7f8de80d1fb0 con 0x7f8de815e360 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.843+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd0003520 con 0x7f8de815e360 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.843+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787722534 0 0) 0x7f8dd0003520 con 0x7f8de815e360 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 msgr2=0x7f8de806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 0x7f8de806c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 >> v1:172.21.15.135:6789/0 conn(0x7f8de80d02a0 legacy=0x7f8de806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de809b3a0 con 0x7f8de815e360 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8dd4001670 con 0x7f8de815e360 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 >> v1:172.21.15.97:6789/0 conn(0x7f8de815e360 legacy=0x7f8de80cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd00039e0 0x7f8dd0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 0x7f8dd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8dd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd000bb50 con 0x7f8dd0004d40 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd000bcd0 con 0x7f8dd00039e0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd000bed0 con 0x7f8de815df80 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8dee000640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8dd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8ded600640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd00039e0 0x7f8dd0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.844+0000 7f8decc00640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 0x7f8dd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.845+0000 7f8dee000640 1 -- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 msgr2=0x7f8dd0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.845+0000 7f8dee000640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de815df80 0x7f8dd0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.845+0000 7f8dee000640 1 -- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd00039e0 msgr2=0x7f8dd0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.845+0000 7f8dee000640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd00039e0 0x7f8dd0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.845+0000 7f8dee000640 1 -- 172.21.15.97:0/401359617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0003520 con 0x7f8dd0004d40 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8dee000640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8dd00051c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8de0001e20 tx=0x7f8dd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b829fc42b85d54d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0001a70 con 0x7f8dd0004d40 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de0003b70 con 0x7f8dd0004d40 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8de7e00640 1 -- 172.21.15.97:0/401359617 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de00050d0 con 0x7f8dd0004d40 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8df0000640 1 -- 172.21.15.97:0/401359617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 msgr2=0x7f8dd00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8df0000640 1 --2- 172.21.15.97:0/401359617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8dd00051c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8de0001e20 tx=0x7f8dd0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.846+0000 7f8dee000640 1 -- 172.21.15.97:0/401359617 reap_dead start 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.847+0000 7f8df0000640 1 -- 172.21.15.97:0/401359617 shutdown_connections 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.847+0000 7f8df0000640 1 -- 172.21.15.97:0/401359617 >> 172.21.15.97:0/401359617 conn(0x7f8de80b7320 msgr2=0x7f8de809a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.847+0000 7f8df0000640 1 -- 172.21.15.97:0/401359617 shutdown_connections 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.847+0000 7f8df0000640 1 -- 172.21.15.97:0/401359617 wait complete. 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.848+0000 7f8df0000640 1 Processor -- start 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.849+0000 7f8df0000640 1 -- start start 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8df0000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8de814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8df0000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8de815f820 0x7f8de8159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8df0000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de806ee40 0x7f8de806f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8df0000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de809b680 con 0x7f8dd0004d40 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8df0000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80d2500 con 0x7f8de815f820 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8df0000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de806de40 con 0x7f8de806ee40 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8ded600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8de814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8ded600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8de814f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46448/0 (socket says 172.21.15.97:46448) 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.850+0000 7f8ded600640 1 -- 172.21.15.97:0/2258441798 learned_addr learned my addr 172.21.15.97:0/2258441798 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8decc00640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8de815f820 0x7f8de8159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8dee000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de806ee40 0x7f8de806f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8ded600640 1 -- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de806ee40 msgr2=0x7f8de806f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8ded600640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de806ee40 0x7f8de806f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8ded600640 1 -- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8de815f820 msgr2=0x7f8de8159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8ded600640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8de815f820 0x7f8de8159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8ded600640 1 -- 172.21.15.97:0/2258441798 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de815fd20 con 0x7f8dd0004d40 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8decc00640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8de815f820 0x7f8de8159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8ded600640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8de814f780 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc011600 tx=0x7f8ddc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc00f8c0 con 0x7f8dd0004d40 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ddc00fa60 con 0x7f8dd0004d40 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc00fe10 con 0x7f8dd0004d40 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.851+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de8160000 con 0x7f8dd0004d40 2024-03-06T07:28:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.852+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de81605c0 con 0x7f8dd0004d40 2024-03-06T07:28:24.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.853+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de804f330 con 0x7f8dd0004d40 2024-03-06T07:28:24.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.854+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ddc015460 con 0x7f8dd0004d40 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.855+0000 7f8de5600640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dc806b500 0x7f8dc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.855+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ddc08ab00 con 0x7f8dd0004d40 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.855+0000 7f8decc00640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dc806b500 0x7f8dc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.856+0000 7f8decc00640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dc806b500 0x7f8dc806d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8de806fa70 tx=0x7f8dd4000f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:24.859+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ddc019030 con 0x7f8dd0004d40 2024-03-06T07:28:25.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.056+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8de806faa0 con 0x7f8dd0004d40 2024-03-06T07:28:25.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.057+0000 7f8de5600640 1 -- 172.21.15.97:0/2258441798 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ddc0599f0 con 0x7f8dd0004d40 2024-03-06T07:28:25.059 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dc806b500 msgr2=0x7f8dc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dc806b500 0x7f8dc806d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8de806fa70 tx=0x7f8dd4000f30 comp rx=0 tx=0).stop 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 msgr2=0x7f8de814f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8de814f780 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc011600 tx=0x7f8ddc011ac0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 shutdown_connections 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dc806b500 0x7f8dc806d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8de806ee40 0x7f8de806f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8de815f820 0x7f8de8159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 --2- 172.21.15.97:0/2258441798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8dd0004d40 0x7f8de814f780 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.059+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 >> 172.21.15.97:0/2258441798 conn(0x7f8de80b7320 msgr2=0x7f8de806ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.060+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 shutdown_connections 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.060+0000 7f8df0000640 1 -- 172.21.15.97:0/2258441798 wait complete. 2024-03-06T07:28:25.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:25.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.285+0000 7f67f1800640 1 Processor -- start 2024-03-06T07:28:25.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.285+0000 7f67f1800640 1 -- start start 2024-03-06T07:28:25.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.286+0000 7f67f1800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 0x7f67ec160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.286+0000 7f67f1800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f67ec070ff0 0x7f67ec0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.286+0000 7f67f1800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ec0b46c0 con 0x7f67ec15fc10 2024-03-06T07:28:25.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.286+0000 7f67f1800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ec071480 con 0x7f67ec070ff0 2024-03-06T07:28:25.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.286+0000 7f67f1800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec0b49a0 con 0x7f67ec0b3f70 2024-03-06T07:28:25.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.287+0000 7f67ea000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 0x7f67ec160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.287+0000 7f67eb400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f67ec070ff0 0x7f67ec0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.287+0000 7f67ea000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 0x7f67ec160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:36472/0 (socket says 172.21.15.97:36472) 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.287+0000 7f67ea000640 1 -- 172.21.15.97:0/3537305951 learned_addr learned my addr 172.21.15.97:0/3537305951 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.287+0000 7f67ea000640 1 -- 172.21.15.97:0/3537305951 >> v2:172.21.15.135:3300/0 conn(0x7f67ec070ff0 msgr2=0x7f67ec0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.287+0000 7f67ea000640 1 --2- 172.21.15.97:0/3537305951 >> v2:172.21.15.135:3300/0 conn(0x7f67ec070ff0 0x7f67ec0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.288+0000 7f67ea000640 1 -- 172.21.15.97:0/3537305951 >> v1:172.21.15.135:6789/0 conn(0x7f67ec15fc10 legacy=0x7f67ec15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.288+0000 7f67ea000640 1 -- 172.21.15.97:0/3537305951 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec072c90 con 0x7f67ec0b3f70 2024-03-06T07:28:25.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.289+0000 7f67ea000640 1 --2- 172.21.15.97:0/3537305951 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 0x7f67ec160700 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009870 tx=0x7f67d402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c94ae555830842f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67e9600640 1 -- 172.21.15.97:0/3537305951 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4035020 con 0x7f67ec0b3f70 2024-03-06T07:28:25.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67e9600640 1 -- 172.21.15.97:0/3537305951 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d4031640 con 0x7f67ec0b3f70 2024-03-06T07:28:25.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67e9600640 1 -- 172.21.15.97:0/3537305951 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d402d040 con 0x7f67ec0b3f70 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 -- 172.21.15.97:0/3537305951 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 msgr2=0x7f67ec160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 --2- 172.21.15.97:0/3537305951 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 0x7f67ec160700 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009870 tx=0x7f67d402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 -- 172.21.15.97:0/3537305951 shutdown_connections 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 --2- 172.21.15.97:0/3537305951 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec0b3f70 0x7f67ec160700 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 --2- 172.21.15.97:0/3537305951 >> v2:172.21.15.135:3300/0 conn(0x7f67ec070ff0 0x7f67ec0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 -- 172.21.15.97:0/3537305951 >> 172.21.15.97:0/3537305951 conn(0x7f67ec0b1940 msgr2=0x7f67ec0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 -- 172.21.15.97:0/3537305951 shutdown_connections 2024-03-06T07:28:25.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.290+0000 7f67f1800640 1 -- 172.21.15.97:0/3537305951 wait complete. 2024-03-06T07:28:25.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.292+0000 7f67f1800640 1 Processor -- start 2024-03-06T07:28:25.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.292+0000 7f67f1800640 1 -- start start 2024-03-06T07:28:25.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67f1800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec070ff0 0x7f67ec14a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67f1800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67ec0b3f70 0x7f67ec0acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67f1800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 0x7f67ec0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67f1800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec072a30 con 0x7f67ec15fc10 2024-03-06T07:28:25.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67f1800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec0728b0 con 0x7f67ec0b3f70 2024-03-06T07:28:25.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67f1800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec15c2e0 con 0x7f67ec070ff0 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.293+0000 7f67eb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 0x7f67ec0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 0x7f67ec0ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46458/0 (socket says 172.21.15.97:46458) 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 -- 172.21.15.97:0/1431352321 learned_addr learned my addr 172.21.15.97:0/1431352321 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eaa00640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec070ff0 0x7f67ec14a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67ea000640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67ec0b3f70 0x7f67ec0acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 -- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec070ff0 msgr2=0x7f67ec14a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec070ff0 0x7f67ec14a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 -- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67ec0b3f70 msgr2=0x7f67ec0acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67ec0b3f70 0x7f67ec0acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 -- 172.21.15.97:0/1431352321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec14a600 con 0x7f67ec15fc10 2024-03-06T07:28:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.294+0000 7f67eb400640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 0x7f67ec0ad260 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f67e400efc0 tx=0x7f67e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.295+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4018020 con 0x7f67ec15fc10 2024-03-06T07:28:25.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.295+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e4014480 con 0x7f67ec15fc10 2024-03-06T07:28:25.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.295+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4014870 con 0x7f67ec15fc10 2024-03-06T07:28:25.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.295+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec09ab90 con 0x7f67ec15fc10 2024-03-06T07:28:25.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.295+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec09b150 con 0x7f67ec15fc10 2024-03-06T07:28:25.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.296+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec04f330 con 0x7f67ec15fc10 2024-03-06T07:28:25.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.297+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67e4005d10 con 0x7f67ec15fc10 2024-03-06T07:28:25.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.298+0000 7f67e2a00640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67c806b4d0 0x7f67c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.298+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67e408acf0 con 0x7f67ec15fc10 2024-03-06T07:28:25.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.298+0000 7f67eaa00640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67c806b4d0 0x7f67c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.299+0000 7f67eaa00640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67c806b4d0 0x7f67c806d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f67dc004ea0 tx=0x7f67dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.303+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67e4022070 con 0x7f67ec15fc10 2024-03-06T07:28:25.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.500+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67ec06ef30 con 0x7f67ec15fc10 2024-03-06T07:28:25.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.501+0000 7f67e2a00640 1 -- 172.21.15.97:0/1431352321 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67e4010040 con 0x7f67ec15fc10 2024-03-06T07:28:25.503 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67c806b4d0 msgr2=0x7f67c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67c806b4d0 0x7f67c806d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f67dc004ea0 tx=0x7f67dc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 msgr2=0x7f67ec0ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 0x7f67ec0ad260 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f67e400efc0 tx=0x7f67e400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 shutdown_connections 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67c806b4d0 0x7f67c806d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67ec15fc10 0x7f67ec0ad260 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67ec0b3f70 0x7f67ec0acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 --2- 172.21.15.97:0/1431352321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67ec070ff0 0x7f67ec14a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 >> 172.21.15.97:0/1431352321 conn(0x7f67ec0b1940 msgr2=0x7f67ec0addc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.503+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 shutdown_connections 2024-03-06T07:28:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.504+0000 7f67f1800640 1 -- 172.21.15.97:0/1431352321 wait complete. 2024-03-06T07:28:25.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:25.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.732+0000 7f052f200640 1 Processor -- start 2024-03-06T07:28:25.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.732+0000 7f052f200640 1 -- start start 2024-03-06T07:28:25.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f053015f580 0x7f05300b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f05300b0c20 0x7f05300d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300ae890 con 0x7f05300cfbe0 2024-03-06T07:28:25.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300af780 con 0x7f053015f580 2024-03-06T07:28:25.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300aeb70 con 0x7f05300b0c20 2024-03-06T07:28:25.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f0525400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f05300cfbe0 0x7f05300b1370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39500/0 (socket says 172.21.15.97:39500) 2024-03-06T07:28:25.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f0525400640 1 -- 172.21.15.97:0/1147651851 learned_addr learned my addr 172.21.15.97:0/1147651851 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:25.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052de00640 1 --2- 172.21.15.97:0/1147651851 >> v2:172.21.15.97:3300/0 conn(0x7f053015f580 0x7f05300b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.734+0000 7f052e800640 1 --2- 172.21.15.97:0/1147651851 >> v2:172.21.15.135:3300/0 conn(0x7f05300b0c20 0x7f05300d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.735+0000 7f052d400640 1 -- 172.21.15.97:0/1147651851 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721598029 0 0) 0x7f05300ae890 con 0x7f05300cfbe0 2024-03-06T07:28:25.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.735+0000 7f052de00640 1 -- 172.21.15.97:0/1147651851 >> v2:172.21.15.135:3300/0 conn(0x7f05300b0c20 msgr2=0x7f05300d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.735+0000 7f052de00640 1 --2- 172.21.15.97:0/1147651851 >> v2:172.21.15.135:3300/0 conn(0x7f05300b0c20 0x7f05300d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.735+0000 7f052de00640 1 -- 172.21.15.97:0/1147651851 >> v1:172.21.15.97:6789/0 conn(0x7f05300cfbe0 legacy=0x7f05300b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.735+0000 7f052de00640 1 -- 172.21.15.97:0/1147651851 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05300aee50 con 0x7f053015f580 2024-03-06T07:28:25.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.736+0000 7f052de00640 1 --2- 172.21.15.97:0/1147651851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053015f580 0x7f05300b2e00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f05300c5340 tx=0x7f052002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a1c6d91d87f366 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052d400640 1 -- 172.21.15.97:0/1147651851 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520035020 con 0x7f053015f580 2024-03-06T07:28:25.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052d400640 1 -- 172.21.15.97:0/1147651851 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0520031830 con 0x7f053015f580 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052d400640 1 -- 172.21.15.97:0/1147651851 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520031c00 con 0x7f053015f580 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 -- 172.21.15.97:0/1147651851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053015f580 msgr2=0x7f05300b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 --2- 172.21.15.97:0/1147651851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053015f580 0x7f05300b2e00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f05300c5340 tx=0x7f052002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 -- 172.21.15.97:0/1147651851 shutdown_connections 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 --2- 172.21.15.97:0/1147651851 >> v2:172.21.15.135:3300/0 conn(0x7f05300b0c20 0x7f05300d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 --2- 172.21.15.97:0/1147651851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053015f580 0x7f05300b2e00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 -- 172.21.15.97:0/1147651851 >> 172.21.15.97:0/1147651851 conn(0x7f05300d1ae0 msgr2=0x7f05300d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 -- 172.21.15.97:0/1147651851 shutdown_connections 2024-03-06T07:28:25.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.737+0000 7f052f200640 1 -- 172.21.15.97:0/1147651851 wait complete. 2024-03-06T07:28:25.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.739+0000 7f052f200640 1 Processor -- start 2024-03-06T07:28:25.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.739+0000 7f052f200640 1 -- start start 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.740+0000 7f052f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05300b0c20 0x7f05301412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.740+0000 7f052f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 0x7f05300cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.740+0000 7f052f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f053015f580 0x7f05300cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.740+0000 7f052f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053015f960 con 0x7f05300cfbe0 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.740+0000 7f052f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300af410 con 0x7f05300b0c20 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.740+0000 7f052f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300b3340 con 0x7f053015f580 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 0x7f05300cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 0x7f05300cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46478/0 (socket says 172.21.15.97:46478) 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 -- 172.21.15.97:0/2368025884 learned_addr learned my addr 172.21.15.97:0/2368025884 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f052de00640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05300b0c20 0x7f05301412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f052e800640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f053015f580 0x7f05300cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 -- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f053015f580 msgr2=0x7f05300cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f053015f580 0x7f05300cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 -- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05300b0c20 msgr2=0x7f05301412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05300b0c20 0x7f05301412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 -- 172.21.15.97:0/2368025884 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05301f3d30 con 0x7f05300cfbe0 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f052de00640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05300b0c20 0x7f05301412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.741+0000 7f0525400640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 0x7f05300cb1c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f051800c500 tx=0x7f051800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.742+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518018070 con 0x7f05300cfbe0 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.742+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0518007580 con 0x7f05300cfbe0 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.742+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518004a20 con 0x7f05300cfbe0 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.742+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053015dad0 con 0x7f05300cfbe0 2024-03-06T07:28:25.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.742+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05300cb700 con 0x7f05300cfbe0 2024-03-06T07:28:25.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.743+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053006cab0 con 0x7f05300cfbe0 2024-03-06T07:28:25.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.744+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0518004bc0 con 0x7f05300cfbe0 2024-03-06T07:28:25.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.745+0000 7f0526a00640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050c06b500 0x7f050c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.745+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05180893f0 con 0x7f05300cfbe0 2024-03-06T07:28:25.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.745+0000 7f052de00640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050c06b500 0x7f050c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.746+0000 7f052de00640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050c06b500 0x7f050c06d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0520036000 tx=0x7f0520008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.749+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0518058300 con 0x7f05300cfbe0 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.947+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f053004f330 con 0x7f05300cfbe0 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.948+0000 7f0526a00640 1 -- 172.21.15.97:0/2368025884 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0518057c00 con 0x7f05300cfbe0 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:25.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050c06b500 msgr2=0x7f050c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050c06b500 0x7f050c06d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0520036000 tx=0x7f0520008030 comp rx=0 tx=0).stop 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 msgr2=0x7f05300cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 0x7f05300cb1c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f051800c500 tx=0x7f051800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 shutdown_connections 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050c06b500 0x7f050c06d9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f053015f580 0x7f05300cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05300cfbe0 0x7f05300cb1c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 --2- 172.21.15.97:0/2368025884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05300b0c20 0x7f05301412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 >> 172.21.15.97:0/2368025884 conn(0x7f05300d1ae0 msgr2=0x7f053015b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 shutdown_connections 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:25.950+0000 7f052f200640 1 -- 172.21.15.97:0/2368025884 wait complete. 2024-03-06T07:28:25.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.172+0000 7f05f6800640 1 Processor -- start 2024-03-06T07:28:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.172+0000 7f05f6800640 1 -- start start 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.173+0000 7f05f6800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f0162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f05f00affe0 0x7f05f006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f6800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f00cef70 con 0x7f05f006b500 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f6800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f0162270 con 0x7f05f00affe0 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f6800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f015e3b0 con 0x7f05f015dfd0 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f4a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f05f00affe0 0x7f05f006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f5e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f05f006b500 0x7f05f00d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39516/0 (socket says 172.21.15.97:39516) 2024-03-06T07:28:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f5e00640 1 -- 172.21.15.97:0/1118234168 learned_addr learned my addr 172.21.15.97:0/1118234168 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f5400640 1 --2- 172.21.15.97:0/1118234168 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f0162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f4a00640 1 -- 172.21.15.97:0/1118234168 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 msgr2=0x7f05f0162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f4a00640 1 --2- 172.21.15.97:0/1118234168 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f0162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.174+0000 7f05f4a00640 1 -- 172.21.15.97:0/1118234168 >> v1:172.21.15.97:6789/0 conn(0x7f05f006b500 legacy=0x7f05f00d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.175+0000 7f05f4a00640 1 -- 172.21.15.97:0/1118234168 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0161c60 con 0x7f05f00affe0 2024-03-06T07:28:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.175+0000 7f05efe00640 1 -- 172.21.15.97:0/1118234168 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378696525 0 0) 0x7f05f00cef70 con 0x7f05f006b500 2024-03-06T07:28:26.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.176+0000 7f05f4a00640 1 --2- 172.21.15.97:0/1118234168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f006afc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0097a0 tx=0x7f05dc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2756cc8cc9c3b1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05efe00640 1 -- 172.21.15.97:0/1118234168 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc039070 con 0x7f05f00affe0 2024-03-06T07:28:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05efe00640 1 -- 172.21.15.97:0/1118234168 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc0316f0 con 0x7f05f00affe0 2024-03-06T07:28:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05efe00640 1 -- 172.21.15.97:0/1118234168 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc031ac0 con 0x7f05f00affe0 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 -- 172.21.15.97:0/1118234168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 msgr2=0x7f05f006afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 --2- 172.21.15.97:0/1118234168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f006afc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0097a0 tx=0x7f05dc02b790 comp rx=0 tx=0).stop 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 -- 172.21.15.97:0/1118234168 shutdown_connections 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 --2- 172.21.15.97:0/1118234168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f006afc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 --2- 172.21.15.97:0/1118234168 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f0162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 -- 172.21.15.97:0/1118234168 >> 172.21.15.97:0/1118234168 conn(0x7f05f006c0d0 msgr2=0x7f05f006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 -- 172.21.15.97:0/1118234168 shutdown_connections 2024-03-06T07:28:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.177+0000 7f05f6800640 1 -- 172.21.15.97:0/1118234168 wait complete. 2024-03-06T07:28:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.179+0000 7f05f6800640 1 Processor -- start 2024-03-06T07:28:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.179+0000 7f05f6800640 1 -- start start 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f6800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05f006b500 0x7f05f0159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f014c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f6800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f01f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f6800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00ce8d0 con 0x7f05f00affe0 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f6800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0161f00 con 0x7f05f006b500 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f6800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00cea50 con 0x7f05f015dfd0 2024-03-06T07:28:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f014c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f014c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46494/0 (socket says 172.21.15.97:46494) 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.180+0000 7f05f4a00640 1 -- 172.21.15.97:0/2625643230 learned_addr learned my addr 172.21.15.97:0/2625643230 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f5e00640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f01f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f4a00640 1 -- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 msgr2=0x7f05f01f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f5400640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05f006b500 0x7f05f0159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f4a00640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f01f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f4a00640 1 -- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05f006b500 msgr2=0x7f05f0159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f4a00640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05f006b500 0x7f05f0159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f4a00640 1 -- 172.21.15.97:0/2625643230 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0159ce0 con 0x7f05f00affe0 2024-03-06T07:28:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05f4a00640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f014c5b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f05dc000c00 tx=0x7f05dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.181+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc043070 con 0x7f05f00affe0 2024-03-06T07:28:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.182+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc034680 con 0x7f05f00affe0 2024-03-06T07:28:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.182+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f00b2db0 con 0x7f05f00affe0 2024-03-06T07:28:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.182+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc039040 con 0x7f05f00affe0 2024-03-06T07:28:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.182+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f00b3050 con 0x7f05f00affe0 2024-03-06T07:28:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.183+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f004f330 con 0x7f05f00affe0 2024-03-06T07:28:26.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.184+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05dc046070 con 0x7f05f00affe0 2024-03-06T07:28:26.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.185+0000 7f05ed600640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f05d006b4d0 0x7f05d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.185+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05dc0abf10 con 0x7f05f00affe0 2024-03-06T07:28:26.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.185+0000 7f05f5400640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f05d006b4d0 0x7f05d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.185+0000 7f05f5400640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f05d006b4d0 0x7f05d006d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f05e4006fd0 tx=0x7f05e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.193+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05dc07acf0 con 0x7f05f00affe0 2024-03-06T07:28:26.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.426+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f05f00af8e0 con 0x7f05f00affe0 2024-03-06T07:28:26.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.427+0000 7f05ed600640 1 -- 172.21.15.97:0/2625643230 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f05dc07a5f0 con 0x7f05f00affe0 2024-03-06T07:28:26.429 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:28:26.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.429+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f05d006b4d0 msgr2=0x7f05d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.429+0000 7f05f6800640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f05d006b4d0 0x7f05d006d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f05e4006fd0 tx=0x7f05e4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.429+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 msgr2=0x7f05f014c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.429+0000 7f05f6800640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f014c5b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f05dc000c00 tx=0x7f05dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 shutdown_connections 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f05d006b4d0 0x7f05d006d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05f015dfd0 0x7f05f01f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05f00affe0 0x7f05f014c5b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 --2- 172.21.15.97:0/2625643230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05f006b500 0x7f05f0159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 >> 172.21.15.97:0/2625643230 conn(0x7f05f006c0d0 msgr2=0x7f05f006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 shutdown_connections 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.430+0000 7f05f6800640 1 -- 172.21.15.97:0/2625643230 wait complete. 2024-03-06T07:28:26.449 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:26.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-06T07:28:26.532 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:28:26.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:26.533 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:26.580 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:26.580 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:26.637 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:26.637 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:26.689 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:26.689 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:26.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:26.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.940+0000 7f6787200640 1 Processor -- start 2024-03-06T07:28:26.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.940+0000 7f6787200640 1 -- start start 2024-03-06T07:28:26.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6787200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f678815dfd0 0x7f6788162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6787200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f678806b6d0 0x7f67880d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6787200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67880cef70 con 0x7f67880affe0 2024-03-06T07:28:26.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6787200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6788162270 con 0x7f678815dfd0 2024-03-06T07:28:26.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6787200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678815e3b0 con 0x7f678806b6d0 2024-03-06T07:28:26.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6785e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f678815dfd0 0x7f6788162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f677d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f67880affe0 0x7f678806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39532/0 (socket says 172.21.15.97:39532) 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6785e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f678815dfd0 0x7f6788162700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46496/0 (socket says 172.21.15.97:46496) 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f677d400640 1 -- 172.21.15.97:0/2875641651 learned_addr learned my addr 172.21.15.97:0/2875641651 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.942+0000 7f6786800640 1 --2- 172.21.15.97:0/2875641651 >> v2:172.21.15.135:3300/0 conn(0x7f678806b6d0 0x7f67880d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.943+0000 7f6785e00640 1 -- 172.21.15.97:0/2875641651 >> v2:172.21.15.135:3300/0 conn(0x7f678806b6d0 msgr2=0x7f67880d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.943+0000 7f6785e00640 1 --2- 172.21.15.97:0/2875641651 >> v2:172.21.15.135:3300/0 conn(0x7f678806b6d0 0x7f67880d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.943+0000 7f6785e00640 1 -- 172.21.15.97:0/2875641651 >> v1:172.21.15.97:6789/0 conn(0x7f67880affe0 legacy=0x7f678806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.943+0000 7f6785e00640 1 -- 172.21.15.97:0/2875641651 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788161c60 con 0x7f678815dfd0 2024-03-06T07:28:26.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.943+0000 7f6786800640 1 --2- 172.21.15.97:0/2875641651 >> v2:172.21.15.135:3300/0 conn(0x7f678806b6d0 0x7f67880d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.944+0000 7f6785e00640 1 --2- 172.21.15.97:0/2875641651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f678815dfd0 0x7f6788162700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f67880d1990 tx=0x7f677802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22aa34991e537d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.944+0000 7f6785400640 1 -- 172.21.15.97:0/2875641651 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778038070 con 0x7f678815dfd0 2024-03-06T07:28:26.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.944+0000 7f6785400640 1 -- 172.21.15.97:0/2875641651 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6778031520 con 0x7f678815dfd0 2024-03-06T07:28:26.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.944+0000 7f6785400640 1 -- 172.21.15.97:0/2875641651 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67780318f0 con 0x7f678815dfd0 2024-03-06T07:28:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 -- 172.21.15.97:0/2875641651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f678815dfd0 msgr2=0x7f6788162700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 --2- 172.21.15.97:0/2875641651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f678815dfd0 0x7f6788162700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f67880d1990 tx=0x7f677802b7c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 -- 172.21.15.97:0/2875641651 shutdown_connections 2024-03-06T07:28:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 --2- 172.21.15.97:0/2875641651 >> v2:172.21.15.135:3300/0 conn(0x7f678806b6d0 0x7f67880d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 --2- 172.21.15.97:0/2875641651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f678815dfd0 0x7f6788162700 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 -- 172.21.15.97:0/2875641651 >> 172.21.15.97:0/2875641651 conn(0x7f678806c0d0 msgr2=0x7f678806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 -- 172.21.15.97:0/2875641651 shutdown_connections 2024-03-06T07:28:26.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.945+0000 7f6787200640 1 -- 172.21.15.97:0/2875641651 wait complete. 2024-03-06T07:28:26.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.947+0000 7f6787200640 1 Processor -- start 2024-03-06T07:28:26.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.947+0000 7f6787200640 1 -- start start 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6787200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b6d0 0x7f678815d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6787200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 0x7f678815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6787200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f678815dfd0 0x7f67881e9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6787200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67880ce8d0 con 0x7f67880affe0 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6787200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6788161f00 con 0x7f678806b6d0 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6787200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880cea50 con 0x7f678815dfd0 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 0x7f678815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 0x7f678815f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46500/0 (socket says 172.21.15.97:46500) 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 -- 172.21.15.97:0/1953466562 learned_addr learned my addr 172.21.15.97:0/1953466562 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6785e00640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b6d0 0x7f678815d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f6786800640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f678815dfd0 0x7f67881e9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 -- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f678815dfd0 msgr2=0x7f67881e9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f678815dfd0 0x7f67881e9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 -- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b6d0 msgr2=0x7f678815d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.948+0000 7f677d400640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b6d0 0x7f678815d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f677d400640 1 -- 172.21.15.97:0/1953466562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678815d810 con 0x7f67880affe0 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f6786800640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f678815dfd0 0x7f67881e9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f677d400640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 0x7f678815f7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f677000c430 tx=0x7f677000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770017070 con 0x7f67880affe0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770005ce0 con 0x7f67880affe0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700072f0 con 0x7f67880affe0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6788160fd0 con 0x7f67880affe0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.949+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678815efa0 con 0x7f67880affe0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.950+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67881617f0 con 0x7f67880affe0 2024-03-06T07:28:26.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.952+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f677001a070 con 0x7f67880affe0 2024-03-06T07:28:26.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.953+0000 7f677ea00640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.953+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6770089620 con 0x7f67880affe0 2024-03-06T07:28:26.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.953+0000 7f6785e00640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.954+0000 7f6785e00640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f676406b4d0 0x7f676406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6778035000 tx=0x7f6778030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:26.958+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67881617f0 con 0x7f67880affe0 2024-03-06T07:28:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.171+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f678804f330 con 0x7f67880affe0 2024-03-06T07:28:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.173+0000 7f677ea00640 1 -- 172.21.15.97:0/1953466562 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f67700584a0 con 0x7f67880affe0 2024-03-06T07:28:27.176 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:27.176 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:27:43.077306+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f676406b4d0 msgr2=0x7f676406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f676406b4d0 0x7f676406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6778035000 tx=0x7f6778030000 comp rx=0 tx=0).stop 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 msgr2=0x7f678815f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 0x7f678815f7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f677000c430 tx=0x7f677000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 shutdown_connections 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f678815dfd0 0x7f67881e9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67880affe0 0x7f678815f7a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 --2- 172.21.15.97:0/1953466562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b6d0 0x7f678815d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.177+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 >> 172.21.15.97:0/1953466562 conn(0x7f678806c0d0 msgr2=0x7f67880b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.178+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 shutdown_connections 2024-03-06T07:28:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.178+0000 7f6787200640 1 -- 172.21.15.97:0/1953466562 wait complete. 2024-03-06T07:28:27.199 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:27.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:27.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.395+0000 7f879f000640 1 Processor -- start 2024-03-06T07:28:27.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.396+0000 7f879f000640 1 -- start start 2024-03-06T07:28:27.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.397+0000 7f879f000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f879815fc10 0x7f8798070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.397+0000 7f879f000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f879815fff0 0x7f8798160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.397+0000 7f879f000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b46c0 con 0x7f87980ce4a0 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.397+0000 7f879f000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980b49a0 con 0x7f879815fc10 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.397+0000 7f879f000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8798072c90 con 0x7f879815fff0 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f879815fc10 0x7f8798070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f879815fc10 0x7f8798070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46508/0 (socket says 172.21.15.97:46508) 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 -- 172.21.15.97:0/2489944080 learned_addr learned my addr 172.21.15.97:0/2489944080 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f879d000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f87980ce4a0 0x7f87980ce8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39542/0 (socket says 172.21.15.97:39542) 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797400640 1 --2- 172.21.15.97:0/2489944080 >> v2:172.21.15.135:3300/0 conn(0x7f879815fff0 0x7f8798160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 -- 172.21.15.97:0/2489944080 >> v2:172.21.15.135:3300/0 conn(0x7f879815fff0 msgr2=0x7f8798160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 --2- 172.21.15.97:0/2489944080 >> v2:172.21.15.135:3300/0 conn(0x7f879815fff0 0x7f8798160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 -- 172.21.15.97:0/2489944080 >> v1:172.21.15.97:6789/0 conn(0x7f87980ce4a0 legacy=0x7f87980ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797e00640 1 -- 172.21.15.97:0/2489944080 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798160990 con 0x7f879815fc10 2024-03-06T07:28:27.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.398+0000 7f8797400640 1 --2- 172.21.15.97:0/2489944080 >> v2:172.21.15.135:3300/0 conn(0x7f879815fff0 0x7f8798160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.399+0000 7f8797e00640 1 --2- 172.21.15.97:0/2489944080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f879815fc10 0x7f8798070ff0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8798074a50 tx=0x7f878802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2175ec7b841a33d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f8796a00640 1 -- 172.21.15.97:0/2489944080 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788034070 con 0x7f879815fc10 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f8796a00640 1 -- 172.21.15.97:0/2489944080 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8788030880 con 0x7f879815fc10 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f8796a00640 1 -- 172.21.15.97:0/2489944080 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788030c50 con 0x7f879815fc10 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 -- 172.21.15.97:0/2489944080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f879815fc10 msgr2=0x7f8798070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 --2- 172.21.15.97:0/2489944080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f879815fc10 0x7f8798070ff0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8798074a50 tx=0x7f878802b9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 -- 172.21.15.97:0/2489944080 shutdown_connections 2024-03-06T07:28:27.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 --2- 172.21.15.97:0/2489944080 >> v2:172.21.15.135:3300/0 conn(0x7f879815fff0 0x7f8798160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 --2- 172.21.15.97:0/2489944080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f879815fc10 0x7f8798070ff0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 -- 172.21.15.97:0/2489944080 >> 172.21.15.97:0/2489944080 conn(0x7f87980b1940 msgr2=0x7f87980b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 -- 172.21.15.97:0/2489944080 shutdown_connections 2024-03-06T07:28:27.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.400+0000 7f879f000640 1 -- 172.21.15.97:0/2489944080 wait complete. 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.402+0000 7f879f000640 1 Processor -- start 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.402+0000 7f879f000640 1 -- start start 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f879f000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 0x7f879809a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f879f000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f879815fc10 0x7f879809af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f879f000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879815fff0 0x7f879809b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f879f000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8798072a30 con 0x7f87980ce4a0 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f879f000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87980728b0 con 0x7f879815fff0 2024-03-06T07:28:27.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f879f000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879815c2e0 con 0x7f879815fc10 2024-03-06T07:28:27.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f8797e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 0x7f879809a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f8797e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 0x7f879809a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46512/0 (socket says 172.21.15.97:46512) 2024-03-06T07:28:27.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.403+0000 7f8797e00640 1 -- 172.21.15.97:0/3707266384 learned_addr learned my addr 172.21.15.97:0/3707266384 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:27.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f879d000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879815fff0 0x7f879809b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797400640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f879815fc10 0x7f879809af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797e00640 1 -- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f879815fc10 msgr2=0x7f879809af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797e00640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f879815fc10 0x7f879809af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797e00640 1 -- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879815fff0 msgr2=0x7f879809b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797e00640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879815fff0 0x7f879809b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797e00640 1 -- 172.21.15.97:0/3707266384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879815ccd0 con 0x7f87980ce4a0 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f8797e00640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 0x7f879809a9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8788000c00 tx=0x7f8788004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788048070 con 0x7f87980ce4a0 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8788039d10 con 0x7f87980ce4a0 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879815cfb0 con 0x7f87980ce4a0 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.404+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788034040 con 0x7f87980ce4a0 2024-03-06T07:28:27.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.405+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879815d570 con 0x7f87980ce4a0 2024-03-06T07:28:27.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.406+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879804f330 con 0x7f87980ce4a0 2024-03-06T07:28:27.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.408+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f878804b070 con 0x7f87980ce4a0 2024-03-06T07:28:27.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.409+0000 7f878fe00640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f877806b4d0 0x7f877806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.409+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f87880ac4c0 con 0x7f87980ce4a0 2024-03-06T07:28:27.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.409+0000 7f8797400640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f877806b4d0 0x7f877806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.410+0000 7f8797400640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f877806b4d0 0x7f877806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87980cc440 tx=0x7f8784008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.414+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f878807b300 con 0x7f87980ce4a0 2024-03-06T07:28:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.609+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f879806ef30 con 0x7f87980ce4a0 2024-03-06T07:28:27.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.610+0000 7f878fe00640 1 -- 172.21.15.97:0/3707266384 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f878807ac00 con 0x7f87980ce4a0 2024-03-06T07:28:27.612 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f877806b4d0 msgr2=0x7f877806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f877806b4d0 0x7f877806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87980cc440 tx=0x7f8784008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 msgr2=0x7f879809a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 0x7f879809a9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8788000c00 tx=0x7f8788004730 comp rx=0 tx=0).stop 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 shutdown_connections 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f877806b4d0 0x7f877806d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879815fff0 0x7f879809b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f879815fc10 0x7f879809af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 --2- 172.21.15.97:0/3707266384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87980ce4a0 0x7f879809a9f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.612+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 >> 172.21.15.97:0/3707266384 conn(0x7f87980b1940 msgr2=0x7f87980cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.613+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 shutdown_connections 2024-03-06T07:28:27.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.613+0000 7f879f000640 1 -- 172.21.15.97:0/3707266384 wait complete. 2024-03-06T07:28:27.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:27.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.840+0000 7f0bd6400640 1 Processor -- start 2024-03-06T07:28:27.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.841+0000 7f0bd6400640 1 -- start start 2024-03-06T07:28:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.843+0000 7f0bd6400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.843+0000 7f0bd6400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0bd00aeff0 0x7f0bd00cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.843+0000 7f0bd6400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00cfb30 con 0x7f0bd01551a0 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.843+0000 7f0bd6400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd015e540 con 0x7f0bd00aeff0 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.843+0000 7f0bd6400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00cfe10 con 0x7f0bd0154dc0 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0bd00aeff0 0x7f0bd00cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0bd00aeff0 0x7f0bd00cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46516/0 (socket says 172.21.15.97:46516) 2024-03-06T07:28:27.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 -- 172.21.15.97:0/2674512067 learned_addr learned my addr 172.21.15.97:0/2674512067 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:27.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5000640 1 --2- 172.21.15.97:0/2674512067 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 -- 172.21.15.97:0/2674512067 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 msgr2=0x7f0bd0161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 --2- 172.21.15.97:0/2674512067 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0161b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 -- 172.21.15.97:0/2674512067 >> v1:172.21.15.135:6789/0 conn(0x7f0bd01551a0 legacy=0x7f0bd00ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.844+0000 7f0bd5a00640 1 -- 172.21.15.97:0/2674512067 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00d00f0 con 0x7f0bd00aeff0 2024-03-06T07:28:27.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.846+0000 7f0bd5a00640 1 --2- 172.21.15.97:0/2674512067 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd00cebc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009e50 tx=0x7f0bc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5a4a7dd03bc806 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.846+0000 7f0bcf400640 1 -- 172.21.15.97:0/2674512067 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc802ca70 con 0x7f0bd00aeff0 2024-03-06T07:28:27.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.846+0000 7f0bcf400640 1 -- 172.21.15.97:0/2674512067 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc802cc10 con 0x7f0bd00aeff0 2024-03-06T07:28:27.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.846+0000 7f0bcf400640 1 -- 172.21.15.97:0/2674512067 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80343f0 con 0x7f0bd00aeff0 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 -- 172.21.15.97:0/2674512067 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 msgr2=0x7f0bd00cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 --2- 172.21.15.97:0/2674512067 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd00cebc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009e50 tx=0x7f0bc802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 -- 172.21.15.97:0/2674512067 shutdown_connections 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 --2- 172.21.15.97:0/2674512067 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd00cebc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 --2- 172.21.15.97:0/2674512067 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 -- 172.21.15.97:0/2674512067 >> 172.21.15.97:0/2674512067 conn(0x7f0bd006afc0 msgr2=0x7f0bd006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 -- 172.21.15.97:0/2674512067 shutdown_connections 2024-03-06T07:28:27.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.847+0000 7f0bd6400640 1 -- 172.21.15.97:0/2674512067 wait complete. 2024-03-06T07:28:27.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.849+0000 7f0bd6400640 1 Processor -- start 2024-03-06T07:28:27.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.849+0000 7f0bd6400640 1 -- start start 2024-03-06T07:28:27.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd6400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd01498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd6400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd6400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd01551a0 0x7f0bd00afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd6400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0162070 con 0x7f0bd00aeff0 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd6400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00cf820 con 0x7f0bd01551a0 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd6400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd01570d0 con 0x7f0bd0154dc0 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd01498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd01498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46518/0 (socket says 172.21.15.97:46518) 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd5000640 1 -- 172.21.15.97:0/22235345 learned_addr learned my addr 172.21.15.97:0/22235345 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd5000640 1 -- 172.21.15.97:0/22235345 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 msgr2=0x7f0bd0157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd5000640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.851+0000 7f0bd5000640 1 -- 172.21.15.97:0/22235345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd01551a0 msgr2=0x7f0bd00afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bd5000640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd01551a0 0x7f0bd00afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bd5000640 1 -- 172.21.15.97:0/22235345 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd0157320 con 0x7f0bd00aeff0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bd5000640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd01498d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c7f0 tx=0x7f0bc400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4015070 con 0x7f0bd00aeff0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4004240 con 0x7f0bd00aeff0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd0157600 con 0x7f0bd00aeff0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4007620 con 0x7f0bd00aeff0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.852+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd01582c0 con 0x7f0bd00aeff0 2024-03-06T07:28:27.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.854+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd006d680 con 0x7f0bd00aeff0 2024-03-06T07:28:27.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.854+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bc40079d0 con 0x7f0bd00aeff0 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.855+0000 7f0bccc00640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb006b4d0 0x7f0bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.855+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f0bc408a6c0 con 0x7f0bd00aeff0 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.856+0000 7f0bcfe00640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb006b4d0 0x7f0bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.856+0000 7f0bcfe00640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb006b4d0 0x7f0bb006d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc006fd0 tx=0x7f0bbc006bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:27.860+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bd006d680 con 0x7f0bd00aeff0 2024-03-06T07:28:28.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.079+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bd0068350 con 0x7f0bd00aeff0 2024-03-06T07:28:28.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.080+0000 7f0bccc00640 1 -- 172.21.15.97:0/22235345 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bd0068350 con 0x7f0bd00aeff0 2024-03-06T07:28:28.082 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:28.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb006b4d0 msgr2=0x7f0bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb006b4d0 0x7f0bb006d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc006fd0 tx=0x7f0bbc006bc0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 msgr2=0x7f0bd01498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd01498d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c7f0 tx=0x7f0bc400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 shutdown_connections 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb006b4d0 0x7f0bb006d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd01551a0 0x7f0bd00afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd0154dc0 0x7f0bd0157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 --2- 172.21.15.97:0/22235345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd00aeff0 0x7f0bd01498d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 >> 172.21.15.97:0/22235345 conn(0x7f0bd006afc0 msgr2=0x7f0bd00ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 shutdown_connections 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.083+0000 7f0bd6400640 1 -- 172.21.15.97:0/22235345 wait complete. 2024-03-06T07:28:28.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.325+0000 7f6fb0a00640 1 Processor -- start 2024-03-06T07:28:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.325+0000 7f6fb0a00640 1 -- start start 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fb0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6fac0d07f0 0x7f6fac0d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fb0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac0d1bf0 0x7f6fac0cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fb0a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac0b3730 con 0x7f6fac15e450 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fb0a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac0d4930 con 0x7f6fac0d07f0 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fb0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac067b20 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fa9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6fac0d07f0 0x7f6fac0d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fa9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6fac0d07f0 0x7f6fac0d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46524/0 (socket says 172.21.15.97:46524) 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.327+0000 7f6fa9e00640 1 -- 172.21.15.97:0/3099886743 learned_addr learned my addr 172.21.15.97:0/3099886743 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.328+0000 7f6faa800640 1 --2- 172.21.15.97:0/3099886743 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac0d1bf0 0x7f6fac0cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.328+0000 7f6fa9e00640 1 -- 172.21.15.97:0/3099886743 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac0d1bf0 msgr2=0x7f6fac0cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.328+0000 7f6fa9e00640 1 --2- 172.21.15.97:0/3099886743 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac0d1bf0 0x7f6fac0cfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.328+0000 7f6fa9e00640 1 -- 172.21.15.97:0/3099886743 >> v1:172.21.15.135:6789/0 conn(0x7f6fac15e450 legacy=0x7f6fac0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.328+0000 7f6fa9e00640 1 -- 172.21.15.97:0/3099886743 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0d1110 con 0x7f6fac0d07f0 2024-03-06T07:28:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.329+0000 7f6fa9e00640 1 --2- 172.21.15.97:0/3099886743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d07f0 0x7f6fac0d0bd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009870 tx=0x7f6f9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141d01852829d5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.329+0000 7f6fa8a00640 1 -- 172.21.15.97:0/3099886743 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c035020 con 0x7f6fac0d07f0 2024-03-06T07:28:28.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.329+0000 7f6fa8a00640 1 -- 172.21.15.97:0/3099886743 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c031640 con 0x7f6fac0d07f0 2024-03-06T07:28:28.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.329+0000 7f6fa8a00640 1 -- 172.21.15.97:0/3099886743 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c02d040 con 0x7f6fac0d07f0 2024-03-06T07:28:28.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 -- 172.21.15.97:0/3099886743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d07f0 msgr2=0x7f6fac0d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/3099886743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d07f0 0x7f6fac0d0bd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009870 tx=0x7f6f9c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 -- 172.21.15.97:0/3099886743 shutdown_connections 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/3099886743 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac0d1bf0 0x7f6fac0cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/3099886743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d07f0 0x7f6fac0d0bd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 -- 172.21.15.97:0/3099886743 >> 172.21.15.97:0/3099886743 conn(0x7f6fac0d3e10 msgr2=0x7f6fac0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 -- 172.21.15.97:0/3099886743 shutdown_connections 2024-03-06T07:28:28.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.330+0000 7f6fb0a00640 1 -- 172.21.15.97:0/3099886743 wait complete. 2024-03-06T07:28:28.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.332+0000 7f6fb0a00640 1 Processor -- start 2024-03-06T07:28:28.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.332+0000 7f6fb0a00640 1 -- start start 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fb0a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0d07f0 0x7f6fac154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fb0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 0x7f6fac0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fb0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac15e450 0x7f6fac0b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fb0a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0b3c80 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fb0a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0b39a0 con 0x7f6fac0d07f0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fb0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0b3f60 con 0x7f6fac15e450 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fa9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 0x7f6fac0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fa9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 0x7f6fac0b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46532/0 (socket says 172.21.15.97:46532) 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fa9400640 1 -- 172.21.15.97:0/1506280502 learned_addr learned my addr 172.21.15.97:0/1506280502 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.334+0000 7f6fa9e00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0d07f0 0x7f6fac154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6faa800640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac15e450 0x7f6fac0b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa9400640 1 -- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac15e450 msgr2=0x7f6fac0b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa9400640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac15e450 0x7f6fac0b6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa9400640 1 -- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0d07f0 msgr2=0x7f6fac154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa9400640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0d07f0 0x7f6fac154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa9400640 1 -- 172.21.15.97:0/1506280502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0b7910 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6faa800640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac15e450 0x7f6fac0b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa9400640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 0x7f6fac0b91f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800c3f0 tx=0x7f6f9800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.335+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f980052d0 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.336+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f98005490 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.336+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98007260 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.336+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac0b7bf0 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.336+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac158ff0 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.337+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac04f330 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.338+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f98007400 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.339+0000 7f6fa2000640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f8c06b4d0 0x7f6f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.339+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f6f98089750 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.339+0000 7f6fa9e00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f8c06b4d0 0x7f6f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.340+0000 7f6fa9e00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f8c06b4d0 0x7f6f8c06d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c036000 tx=0x7f6f9c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.344+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f980584f0 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.553+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6fac1e74f0 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.554+0000 7f6fa2000640 1 -- 172.21.15.97:0/1506280502 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6f98057df0 con 0x7f6fac0d1bf0 2024-03-06T07:28:28.556 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.556+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f8c06b4d0 msgr2=0x7f6f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.556+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f8c06b4d0 0x7f6f8c06d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c036000 tx=0x7f6f9c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.556+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 msgr2=0x7f6fac0b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.556+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 0x7f6fac0b91f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800c3f0 tx=0x7f6f9800c7f0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 shutdown_connections 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f8c06b4d0 0x7f6f8c06d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6fac15e450 0x7f6fac0b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fac0d1bf0 0x7f6fac0b91f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 --2- 172.21.15.97:0/1506280502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0d07f0 0x7f6fac154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 >> 172.21.15.97:0/1506280502 conn(0x7f6fac0d3e10 msgr2=0x7f6fac0b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 shutdown_connections 2024-03-06T07:28:28.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.557+0000 7f6fb0a00640 1 -- 172.21.15.97:0/1506280502 wait complete. 2024-03-06T07:28:28.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:28.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.790+0000 7f8751a00640 1 Processor -- start 2024-03-06T07:28:28.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.790+0000 7f8751a00640 1 -- start start 2024-03-06T07:28:28.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8751a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4930 0x7f874c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8751a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f874c0c4d10 0x7f874c0c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8751a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f874c0c56b0 0x7f874c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8751a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874c145b30 con 0x7f874c0c4d10 2024-03-06T07:28:28.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8751a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874c142ce0 con 0x7f874c0c56b0 2024-03-06T07:28:28.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8751a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0cbca0 con 0x7f874c0c4930 2024-03-06T07:28:28.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8742000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f874c0c4d10 0x7f874c0c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8742000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f874c0c4d10 0x7f874c0c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46534/0 (socket says 172.21.15.97:46534) 2024-03-06T07:28:28.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.792+0000 7f8742000640 1 -- 172.21.15.97:0/2813042311 learned_addr learned my addr 172.21.15.97:0/2813042311 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f874aa00640 1 --2- 172.21.15.97:0/2813042311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4930 0x7f874c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f874b400640 1 --2- 172.21.15.97:0/2813042311 >> v2:172.21.15.135:3300/0 conn(0x7f874c0c56b0 0x7f874c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f8742000640 1 -- 172.21.15.97:0/2813042311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4930 msgr2=0x7f874c0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f8742000640 1 --2- 172.21.15.97:0/2813042311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4930 0x7f874c0c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f8742000640 1 -- 172.21.15.97:0/2813042311 >> v2:172.21.15.135:3300/0 conn(0x7f874c0c56b0 msgr2=0x7f874c06d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f8742000640 1 --2- 172.21.15.97:0/2813042311 >> v2:172.21.15.135:3300/0 conn(0x7f874c0c56b0 0x7f874c06d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f8742000640 1 -- 172.21.15.97:0/2813042311 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c0cbf80 con 0x7f874c0c4d10 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.793+0000 7f874b400640 1 --2- 172.21.15.97:0/2813042311 >> v2:172.21.15.135:3300/0 conn(0x7f874c0c56b0 0x7f874c06d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.794+0000 7f8742000640 1 --2- 172.21.15.97:0/2813042311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4d10 0x7f874c0c5170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f874c145e30 tx=0x7f873402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389f22494c8c4320 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.794+0000 7f874a000640 1 -- 172.21.15.97:0/2813042311 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340317d0 con 0x7f874c0c4d10 2024-03-06T07:28:28.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.794+0000 7f874a000640 1 -- 172.21.15.97:0/2813042311 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734037070 con 0x7f874c0c4d10 2024-03-06T07:28:28.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.794+0000 7f874a000640 1 -- 172.21.15.97:0/2813042311 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734031b90 con 0x7f874c0c4d10 2024-03-06T07:28:28.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 -- 172.21.15.97:0/2813042311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4d10 msgr2=0x7f874c0c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 --2- 172.21.15.97:0/2813042311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4d10 0x7f874c0c5170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f874c145e30 tx=0x7f873402b870 comp rx=0 tx=0).stop 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 -- 172.21.15.97:0/2813042311 shutdown_connections 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 --2- 172.21.15.97:0/2813042311 >> v2:172.21.15.135:3300/0 conn(0x7f874c0c56b0 0x7f874c06d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 --2- 172.21.15.97:0/2813042311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4d10 0x7f874c0c5170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 --2- 172.21.15.97:0/2813042311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4930 0x7f874c0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 -- 172.21.15.97:0/2813042311 >> 172.21.15.97:0/2813042311 conn(0x7f874c0cc4f0 msgr2=0x7f874c0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 -- 172.21.15.97:0/2813042311 shutdown_connections 2024-03-06T07:28:28.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.795+0000 7f8751a00640 1 -- 172.21.15.97:0/2813042311 wait complete. 2024-03-06T07:28:28.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.797+0000 7f8751a00640 1 Processor -- start 2024-03-06T07:28:28.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.797+0000 7f8751a00640 1 -- start start 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f8751a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 0x7f874c0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f8751a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4d10 0x7f874c1e3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f8751a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f874c0c56b0 0x7f874c0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f8751a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0cc260 con 0x7f874c0c4930 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f8751a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0ca3b0 con 0x7f874c0c56b0 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f8751a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0c5b40 con 0x7f874c0c4d10 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f874aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 0x7f874c0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f874aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 0x7f874c0c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46548/0 (socket says 172.21.15.97:46548) 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.798+0000 7f874aa00640 1 -- 172.21.15.97:0/992521005 learned_addr learned my addr 172.21.15.97:0/992521005 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f8742000640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4d10 0x7f874c1e3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874b400640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f874c0c56b0 0x7f874c0c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874aa00640 1 -- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4d10 msgr2=0x7f874c1e3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874aa00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4d10 0x7f874c1e3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874aa00640 1 -- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f874c0c56b0 msgr2=0x7f874c0c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874aa00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f874c0c56b0 0x7f874c0c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874aa00640 1 -- 172.21.15.97:0/992521005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c0c9690 con 0x7f874c0c4930 2024-03-06T07:28:28.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f874aa00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 0x7f874c0c74e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f873c00d800 tx=0x7f873c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c018030 con 0x7f874c0c4930 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c0096a0 con 0x7f874c0c4930 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c009870 con 0x7f874c0c4930 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874c1f76f0 con 0x7f874c0c4930 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.799+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874c1f7cb0 con 0x7f874c0c4930 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.800+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874c04f330 con 0x7f874c0c4930 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.802+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f873c01b070 con 0x7f874c0c4930 2024-03-06T07:28:28.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.802+0000 7f8743400640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f872806b4d0 0x7f872806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.803+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f873c08a5d0 con 0x7f874c0c4930 2024-03-06T07:28:28.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.803+0000 7f8742000640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f872806b4d0 0x7f872806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.803+0000 7f8742000640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f872806b4d0 0x7f872806d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8734036000 tx=0x7f873403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:28.807+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f873c0592e0 con 0x7f874c0c4930 2024-03-06T07:28:29.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.016+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f874c0ae870 con 0x7f874c0c4930 2024-03-06T07:28:29.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.017+0000 7f8743400640 1 -- 172.21.15.97:0/992521005 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f873c058be0 con 0x7f874c0c4930 2024-03-06T07:28:29.019 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.019+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f872806b4d0 msgr2=0x7f872806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.019+0000 7f8751a00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f872806b4d0 0x7f872806d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8734036000 tx=0x7f873403d040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.019+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 msgr2=0x7f874c0c74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.019+0000 7f8751a00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 0x7f874c0c74e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f873c00d800 tx=0x7f873c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.019+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 shutdown_connections 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.019+0000 7f8751a00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f872806b4d0 0x7f872806d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.020+0000 7f8751a00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f874c0c56b0 0x7f874c0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.020+0000 7f8751a00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f874c0c4d10 0x7f874c1e3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.020+0000 7f8751a00640 1 --2- 172.21.15.97:0/992521005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f874c0c4930 0x7f874c0c74e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.020+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 >> 172.21.15.97:0/992521005 conn(0x7f874c0cc4f0 msgr2=0x7f874c1e5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.020+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 shutdown_connections 2024-03-06T07:28:29.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.020+0000 7f8751a00640 1 -- 172.21.15.97:0/992521005 wait complete. 2024-03-06T07:28:29.039 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-06T07:28:29.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:29.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-06T07:28:29.150 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:28:29.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:29.151 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:29.201 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:29.201 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:29.253 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:29.253 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:29.298 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:29.299 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:29.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:29.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.600+0000 7fe38c800640 1 Processor -- start 2024-03-06T07:28:29.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.600+0000 7fe38c800640 1 -- start start 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.602+0000 7fe38c800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 0x7fe38406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.602+0000 7fe38c800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38406ee70 con 0x7fe38415fe30 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.602+0000 7fe38c800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38406cef0 con 0x7fe38415f9d0 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.602+0000 7fe38c800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38406d1d0 con 0x7fe384144c00 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.602+0000 7fe38a800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe38415fe30 0x7fe3840ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39546/0 (socket says 172.21.15.97:39546) 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.602+0000 7fe38a800640 1 -- 172.21.15.97:0/1462819339 learned_addr learned my addr 172.21.15.97:0/1462819339 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe389e00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 0x7fe38406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312944710 0 0) 0x7fe380001490 con 0x7fe38415fe30 2024-03-06T07:28:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36c003520 con 0x7fe38415fe30 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216943424 0 0) 0x7fe36c003520 con 0x7fe38415fe30 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 msgr2=0x7fe38406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 0x7fe38406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 >> v1:172.21.15.135:6789/0 conn(0x7fe38415f9d0 legacy=0x7fe3840ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38406d4b0 con 0x7fe38415fe30 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe380001bc0 con 0x7fe38415fe30 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 >> v1:172.21.15.97:6789/0 conn(0x7fe38415fe30 legacy=0x7fe3840ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.603+0000 7fe388a00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 0x7fe36c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe388a00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe36c004d40 0x7fe36c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38406d1d0 con 0x7fe36c004d40 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c00ba70 con 0x7fe36c0039e0 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c00bc70 con 0x7fe384144c00 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe38a800640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe36c004d40 0x7fe36c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe389e00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe389400640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 0x7fe36c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe38a800640 1 -- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 msgr2=0x7fe36c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe38a800640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe384144c00 0x7fe36c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe38a800640 1 -- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe36c0039e0 msgr2=0x7fe36c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe38a800640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe38a800640 1 -- 172.21.15.97:0/1462819339 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c003520 con 0x7fe36c004d40 2024-03-06T07:28:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.604+0000 7fe389e00640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:29.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.606+0000 7fe38a800640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe36c004d40 0x7fe36c0051c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe38000aa20 tx=0x7fe36c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f8551eb557e4aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.606+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380038070 con 0x7fe36c004d40 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.606+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380033620 con 0x7fe36c004d40 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.606+0000 7fe388a00640 1 -- 172.21.15.97:0/1462819339 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000a670 con 0x7fe36c004d40 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38c800640 1 -- 172.21.15.97:0/1462819339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe36c004d40 msgr2=0x7fe36c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38c800640 1 --2- 172.21.15.97:0/1462819339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe36c004d40 0x7fe36c0051c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe38000aa20 tx=0x7fe36c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38a800640 1 -- 172.21.15.97:0/1462819339 reap_dead start 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38c800640 1 -- 172.21.15.97:0/1462819339 shutdown_connections 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38c800640 1 -- 172.21.15.97:0/1462819339 >> 172.21.15.97:0/1462819339 conn(0x7fe3840b1730 msgr2=0x7fe3840b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38c800640 1 -- 172.21.15.97:0/1462819339 shutdown_connections 2024-03-06T07:28:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.607+0000 7fe38c800640 1 -- 172.21.15.97:0/1462819339 wait complete. 2024-03-06T07:28:29.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.609+0000 7fe38c800640 1 Processor -- start 2024-03-06T07:28:29.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.609+0000 7fe38c800640 1 -- start start 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38c800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe38413c490 0x7fe3840b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38c800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38413c870 0x7fe3840b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38c800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 0x7fe38415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38c800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe384160730 con 0x7fe3840b3070 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38c800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe384160280 con 0x7fe38413c870 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38c800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3840ceec0 con 0x7fe38413c490 2024-03-06T07:28:29.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 0x7fe38415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 0x7fe38415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46554/0 (socket says 172.21.15.97:46554) 2024-03-06T07:28:29.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 -- 172.21.15.97:0/386963514 learned_addr learned my addr 172.21.15.97:0/386963514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe389400640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38413c870 0x7fe3840b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe389e00640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe38413c490 0x7fe3840b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 -- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe38413c490 msgr2=0x7fe3840b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe38413c490 0x7fe3840b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 -- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38413c870 msgr2=0x7fe3840b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38413c870 0x7fe3840b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.610+0000 7fe38a800640 1 -- 172.21.15.97:0/386963514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3840b35f0 con 0x7fe3840b3070 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe389e00640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe38413c490 0x7fe3840b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe38a800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 0x7fe38415b1d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe38000a0b0 tx=0x7fe380001f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380045070 con 0x7fe3840b3070 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380004d60 con 0x7fe3840b3070 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380038040 con 0x7fe3840b3070 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3840b38d0 con 0x7fe3840b3070 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.611+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3840b3e90 con 0x7fe3840b3070 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.612+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38404f330 con 0x7fe3840b3070 2024-03-06T07:28:29.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.613+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe380033a10 con 0x7fe3840b3070 2024-03-06T07:28:29.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.614+0000 7fe37e000640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe36806b500 0x7fe36806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.614+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fe3800ab110 con 0x7fe3840b3070 2024-03-06T07:28:29.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.614+0000 7fe389e00640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe36806b500 0x7fe36806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.615+0000 7fe389e00640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe36806b500 0x7fe36806d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe378009a00 tx=0x7fe378008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.619+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe38406ee70 con 0x7fe3840b3070 2024-03-06T07:28:29.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.899+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe384072890 con 0x7fe3840b3070 2024-03-06T07:28:29.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.900+0000 7fe37e000640 1 -- 172.21.15.97:0/386963514 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16983 (secure 0 0 0) 0x7fe380079740 con 0x7fe3840b3070 2024-03-06T07:28:29.903 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:29.903 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":24,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:28.349664+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.902+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe36806b500 msgr2=0x7fe36806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.902+0000 7fe38c800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe36806b500 0x7fe36806d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe378009a00 tx=0x7fe378008040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.902+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 msgr2=0x7fe38415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 0x7fe38415b1d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe38000a0b0 tx=0x7fe380001f90 comp rx=0 tx=0).stop 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 shutdown_connections 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe36806b500 0x7fe36806d9c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3840b3070 0x7fe38415b1d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38413c870 0x7fe3840b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 --2- 172.21.15.97:0/386963514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe38413c490 0x7fe3840b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 >> 172.21.15.97:0/386963514 conn(0x7fe3840b1730 msgr2=0x7fe384070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 shutdown_connections 2024-03-06T07:28:29.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:29.903+0000 7fe38c800640 1 -- 172.21.15.97:0/386963514 wait complete. 2024-03-06T07:28:29.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:29.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:30.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.119+0000 7f17ea200640 1 Processor -- start 2024-03-06T07:28:30.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.120+0000 7f17ea200640 1 -- start start 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17ea200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f17e415e130 0x7f17e40cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17ea200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f17e40cfaf0 0x7f17e40d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17ea200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40d23e0 0x7f17e40c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17ea200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e40cf120 con 0x7f17e40cfaf0 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17ea200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e40d13e0 con 0x7f17e415e130 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17ea200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40cead0 con 0x7f17e40d23e0 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17e3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f17e40cfaf0 0x7f17e40d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17e3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f17e40cfaf0 0x7f17e40d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46556/0 (socket says 172.21.15.97:46556) 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.121+0000 7f17e3e00640 1 -- 172.21.15.97:0/234284604 learned_addr learned my addr 172.21.15.97:0/234284604 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e8e00640 1 --2- 172.21.15.97:0/234284604 >> v2:172.21.15.135:3300/0 conn(0x7f17e415e130 0x7f17e40cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e9800640 1 --2- 172.21.15.97:0/234284604 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40d23e0 0x7f17e40c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e3e00640 1 -- 172.21.15.97:0/234284604 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40d23e0 msgr2=0x7f17e40c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e3e00640 1 --2- 172.21.15.97:0/234284604 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40d23e0 0x7f17e40c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e3e00640 1 -- 172.21.15.97:0/234284604 >> v2:172.21.15.135:3300/0 conn(0x7f17e415e130 msgr2=0x7f17e40cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e3e00640 1 --2- 172.21.15.97:0/234284604 >> v2:172.21.15.135:3300/0 conn(0x7f17e415e130 0x7f17e40cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.122+0000 7f17e3e00640 1 -- 172.21.15.97:0/234284604 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40cedb0 con 0x7f17e40cfaf0 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.123+0000 7f17e3e00640 1 --2- 172.21.15.97:0/234284604 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e40cfaf0 0x7f17e40d1ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f17e4067ce0 tx=0x7f17d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2688d6b4296cd6db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.123+0000 7f17e3400640 1 -- 172.21.15.97:0/234284604 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0035020 con 0x7f17e40cfaf0 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.123+0000 7f17e3400640 1 -- 172.21.15.97:0/234284604 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0031870 con 0x7f17e40cfaf0 2024-03-06T07:28:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.123+0000 7f17e3400640 1 -- 172.21.15.97:0/234284604 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0031c40 con 0x7f17e40cfaf0 2024-03-06T07:28:30.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 -- 172.21.15.97:0/234284604 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e40cfaf0 msgr2=0x7f17e40d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 --2- 172.21.15.97:0/234284604 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e40cfaf0 0x7f17e40d1ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f17e4067ce0 tx=0x7f17d002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:30.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 -- 172.21.15.97:0/234284604 shutdown_connections 2024-03-06T07:28:30.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 --2- 172.21.15.97:0/234284604 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40d23e0 0x7f17e40c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 --2- 172.21.15.97:0/234284604 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e40cfaf0 0x7f17e40d1ea0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 --2- 172.21.15.97:0/234284604 >> v2:172.21.15.135:3300/0 conn(0x7f17e415e130 0x7f17e40cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 -- 172.21.15.97:0/234284604 >> 172.21.15.97:0/234284604 conn(0x7f17e409d070 msgr2=0x7f17e40d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 -- 172.21.15.97:0/234284604 shutdown_connections 2024-03-06T07:28:30.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.124+0000 7f17ea200640 1 -- 172.21.15.97:0/234284604 wait complete. 2024-03-06T07:28:30.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.127+0000 7f17ea200640 1 Processor -- start 2024-03-06T07:28:30.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.127+0000 7f17ea200640 1 -- start start 2024-03-06T07:28:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17ea200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40cfaf0 0x7f17e415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17ea200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17e40d23e0 0x7f17e4158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17ea200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 0x7f17e40b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17ea200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40c8030 con 0x7f17e415e130 2024-03-06T07:28:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17ea200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40c7eb0 con 0x7f17e40d23e0 2024-03-06T07:28:30.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17ea200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40c81b0 con 0x7f17e40cfaf0 2024-03-06T07:28:30.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17e9800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 0x7f17e40b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17e9800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 0x7f17e40b7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46566/0 (socket says 172.21.15.97:46566) 2024-03-06T07:28:30.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17e9800640 1 -- 172.21.15.97:0/958591423 learned_addr learned my addr 172.21.15.97:0/958591423 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.129+0000 7f17e8e00640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40cfaf0 0x7f17e415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e3e00640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17e40d23e0 0x7f17e4158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e9800640 1 -- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40cfaf0 msgr2=0x7f17e415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e9800640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40cfaf0 0x7f17e415afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e9800640 1 -- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17e40d23e0 msgr2=0x7f17e4158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e9800640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17e40d23e0 0x7f17e4158ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e9800640 1 -- 172.21.15.97:0/958591423 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40b87d0 con 0x7f17e415e130 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e9800640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 0x7f17e40b7ed0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f17dc007d20 tx=0x7f17dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc014420 con 0x7f17e415e130 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc019070 con 0x7f17e415e130 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e40b8d90 con 0x7f17e415e130 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc014820 con 0x7f17e415e130 2024-03-06T07:28:30.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.130+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e40b9350 con 0x7f17e415e130 2024-03-06T07:28:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.131+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e404f330 con 0x7f17e415e130 2024-03-06T07:28:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.132+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17dc0149c0 con 0x7f17e415e130 2024-03-06T07:28:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.133+0000 7f17e0c00640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f17c406b4d0 0x7f17c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.133+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f17dc08aa00 con 0x7f17e415e130 2024-03-06T07:28:30.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.134+0000 7f17e8e00640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f17c406b4d0 0x7f17c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.134+0000 7f17e8e00640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f17c406b4d0 0x7f17c406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f17d4006fd0 tx=0x7f17d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.138+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17dc059840 con 0x7f17e415e130 2024-03-06T07:28:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.362+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17e409d690 con 0x7f17e415e130 2024-03-06T07:28:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.363+0000 7f17e0c00640 1 -- 172.21.15.97:0/958591423 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17dc05f070 con 0x7f17e415e130 2024-03-06T07:28:30.365 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:30.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f17c406b4d0 msgr2=0x7f17c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f17c406b4d0 0x7f17c406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f17d4006fd0 tx=0x7f17d4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 msgr2=0x7f17e40b7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 0x7f17e40b7ed0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f17dc007d20 tx=0x7f17dc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 shutdown_connections 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f17c406b4d0 0x7f17c406d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f17e415e130 0x7f17e40b7ed0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17e40d23e0 0x7f17e4158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 --2- 172.21.15.97:0/958591423 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17e40cfaf0 0x7f17e415afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 >> 172.21.15.97:0/958591423 conn(0x7f17e409d070 msgr2=0x7f17e40c84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 shutdown_connections 2024-03-06T07:28:30.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.365+0000 7f17ea200640 1 -- 172.21.15.97:0/958591423 wait complete. 2024-03-06T07:28:30.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:30.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.592+0000 7f7f64600640 1 Processor -- start 2024-03-06T07:28:30.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.592+0000 7f7f64600640 1 -- start start 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f64600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f5c0cdd70 0x7f7f5c06d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f64600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c141850 0x7f7f5c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f64600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c15e730 con 0x7f7f5c15bdc0 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f64600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c15ea10 con 0x7f7f5c0cdd70 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f64600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c15ecf0 con 0x7f7f5c141850 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f5c0cdd70 0x7f7f5c06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f5c0cdd70 0x7f7f5c06d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46572/0 (socket says 172.21.15.97:46572) 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 -- 172.21.15.97:0/720889817 learned_addr learned my addr 172.21.15.97:0/720889817 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f62600640 1 --2- 172.21.15.97:0/720889817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c141850 0x7f7f5c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 -- 172.21.15.97:0/720889817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c141850 msgr2=0x7f7f5c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 --2- 172.21.15.97:0/720889817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c141850 0x7f7f5c06d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 -- 172.21.15.97:0/720889817 >> v1:172.21.15.135:6789/0 conn(0x7f7f5c15bdc0 legacy=0x7f7f5c0cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.594+0000 7f7f61200640 1 -- 172.21.15.97:0/720889817 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c15efd0 con 0x7f7f5c0cdd70 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.596+0000 7f7f61200640 1 --2- 172.21.15.97:0/720889817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0cdd70 0x7f7f5c06d170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009870 tx=0x7f7f4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faee0a78582f8a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.596+0000 7f7f5be00640 1 -- 172.21.15.97:0/720889817 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48035020 con 0x7f7f5c0cdd70 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.596+0000 7f7f5be00640 1 -- 172.21.15.97:0/720889817 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48031640 con 0x7f7f5c0cdd70 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.596+0000 7f7f5be00640 1 -- 172.21.15.97:0/720889817 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4802d040 con 0x7f7f5c0cdd70 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 -- 172.21.15.97:0/720889817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0cdd70 msgr2=0x7f7f5c06d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 --2- 172.21.15.97:0/720889817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0cdd70 0x7f7f5c06d170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009870 tx=0x7f7f4802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 -- 172.21.15.97:0/720889817 shutdown_connections 2024-03-06T07:28:30.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 --2- 172.21.15.97:0/720889817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c141850 0x7f7f5c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 --2- 172.21.15.97:0/720889817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0cdd70 0x7f7f5c06d170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 -- 172.21.15.97:0/720889817 >> 172.21.15.97:0/720889817 conn(0x7f7f5c0b2890 msgr2=0x7f7f5c0ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 -- 172.21.15.97:0/720889817 shutdown_connections 2024-03-06T07:28:30.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.597+0000 7f7f64600640 1 -- 172.21.15.97:0/720889817 wait complete. 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.600+0000 7f7f64600640 1 Processor -- start 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.600+0000 7f7f64600640 1 -- start start 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f64600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c0cdd70 0x7f7f5c15ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f64600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f5c141850 0x7f7f5c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f64600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 0x7f7f5c0b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f64600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c1439e0 con 0x7f7f5c0b4840 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f64600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c06dbf0 con 0x7f7f5c141850 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f64600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c143b60 con 0x7f7f5c0cdd70 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f62600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 0x7f7f5c0b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f62600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 0x7f7f5c0b03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46580/0 (socket says 172.21.15.97:46580) 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f62600640 1 -- 172.21.15.97:0/1767191997 learned_addr learned my addr 172.21.15.97:0/1767191997 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.602+0000 7f7f61c00640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c0cdd70 0x7f7f5c15ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f61200640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f5c141850 0x7f7f5c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f62600640 1 -- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c0cdd70 msgr2=0x7f7f5c15ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f62600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c0cdd70 0x7f7f5c15ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f62600640 1 -- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f5c141850 msgr2=0x7f7f5c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f62600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f5c141850 0x7f7f5c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f62600640 1 -- 172.21.15.97:0/1767191997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0b1090 con 0x7f7f5c0b4840 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f62600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 0x7f7f5c0b03b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7f54007e40 tx=0x7f7f5400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54014620 con 0x7f7f5c0b4840 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54019070 con 0x7f7f5c0b4840 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54014a20 con 0x7f7f5c0b4840 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.603+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c0b1370 con 0x7f7f5c0b4840 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.604+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c145420 con 0x7f7f5c0b4840 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.605+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c04f330 con 0x7f7f5c0b4840 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.606+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f54014bc0 con 0x7f7f5c0b4840 2024-03-06T07:28:30.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.607+0000 7f7f59600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f3c06b500 0x7f7f3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.607+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f7f5408ab70 con 0x7f7f5c0b4840 2024-03-06T07:28:30.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.607+0000 7f7f61c00640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f3c06b500 0x7f7f3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.608+0000 7f7f61c00640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f3c06b500 0x7f7f3c06d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7f50006fd0 tx=0x7f7f50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.611+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f54059880 con 0x7f7f5c0b4840 2024-03-06T07:28:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.807+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7f5c06c4b0 con 0x7f7f5c0b4840 2024-03-06T07:28:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.808+0000 7f7f59600640 1 -- 172.21.15.97:0/1767191997 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7f54059180 con 0x7f7f5c0b4840 2024-03-06T07:28:30.810 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f3c06b500 msgr2=0x7f7f3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f3c06b500 0x7f7f3c06d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7f50006fd0 tx=0x7f7f50008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 msgr2=0x7f7f5c0b03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 0x7f7f5c0b03b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7f54007e40 tx=0x7f7f5400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 shutdown_connections 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7f3c06b500 0x7f7f3c06d9c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f5c0b4840 0x7f7f5c0b03b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f5c141850 0x7f7f5c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 --2- 172.21.15.97:0/1767191997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f5c0cdd70 0x7f7f5c15ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 >> 172.21.15.97:0/1767191997 conn(0x7f7f5c0b2890 msgr2=0x7f7f5c06d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 shutdown_connections 2024-03-06T07:28:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:30.811+0000 7f7f64600640 1 -- 172.21.15.97:0/1767191997 wait complete. 2024-03-06T07:28:30.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:31.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.047+0000 7fc1fb200640 1 Processor -- start 2024-03-06T07:28:31.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.047+0000 7fc1fb200640 1 -- start start 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1fb200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc1fc15e130 0x7fc1fc0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1fb200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1fc0cfaf0 0x7fc1fc0d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1fb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc0d23e0 0x7fc1fc0c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1fb200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc0cf120 con 0x7fc1fc0cfaf0 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1fb200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc0d13e0 con 0x7fc1fc15e130 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1fb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0cead0 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1f9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1fc0cfaf0 0x7fc1fc0d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1f9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1fc0cfaf0 0x7fc1fc0d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46584/0 (socket says 172.21.15.97:46584) 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.048+0000 7fc1f9400640 1 -- 172.21.15.97:0/132258349 learned_addr learned my addr 172.21.15.97:0/132258349 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:31.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1f9e00640 1 --2- 172.21.15.97:0/132258349 >> v2:172.21.15.135:3300/0 conn(0x7fc1fc15e130 0x7fc1fc0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1fa800640 1 --2- 172.21.15.97:0/132258349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc0d23e0 0x7fc1fc0c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1f9400640 1 -- 172.21.15.97:0/132258349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc0d23e0 msgr2=0x7fc1fc0c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1f9400640 1 --2- 172.21.15.97:0/132258349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc0d23e0 0x7fc1fc0c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1f9400640 1 -- 172.21.15.97:0/132258349 >> v2:172.21.15.135:3300/0 conn(0x7fc1fc15e130 msgr2=0x7fc1fc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1f9400640 1 --2- 172.21.15.97:0/132258349 >> v2:172.21.15.135:3300/0 conn(0x7fc1fc15e130 0x7fc1fc0cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.049+0000 7fc1f9400640 1 -- 172.21.15.97:0/132258349 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc0cedb0 con 0x7fc1fc0cfaf0 2024-03-06T07:28:31.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.050+0000 7fc1f9400640 1 --2- 172.21.15.97:0/132258349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc0d1ea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc067ce0 tx=0x7fc1e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8afc4921465b60a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1f8a00640 1 -- 172.21.15.97:0/132258349 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e802ca70 con 0x7fc1fc0cfaf0 2024-03-06T07:28:31.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1f8a00640 1 -- 172.21.15.97:0/132258349 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e802cc10 con 0x7fc1fc0cfaf0 2024-03-06T07:28:31.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1f8a00640 1 -- 172.21.15.97:0/132258349 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e80343f0 con 0x7fc1fc0cfaf0 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 -- 172.21.15.97:0/132258349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0cfaf0 msgr2=0x7fc1fc0d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 --2- 172.21.15.97:0/132258349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc0d1ea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc067ce0 tx=0x7fc1e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 -- 172.21.15.97:0/132258349 shutdown_connections 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 --2- 172.21.15.97:0/132258349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc0d23e0 0x7fc1fc0c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 --2- 172.21.15.97:0/132258349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc0d1ea0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 --2- 172.21.15.97:0/132258349 >> v2:172.21.15.135:3300/0 conn(0x7fc1fc15e130 0x7fc1fc0cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.051+0000 7fc1fb200640 1 -- 172.21.15.97:0/132258349 >> 172.21.15.97:0/132258349 conn(0x7fc1fc09d070 msgr2=0x7fc1fc0d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.052+0000 7fc1fb200640 1 -- 172.21.15.97:0/132258349 shutdown_connections 2024-03-06T07:28:31.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.052+0000 7fc1fb200640 1 -- 172.21.15.97:0/132258349 wait complete. 2024-03-06T07:28:31.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.053+0000 7fc1fb200640 1 Processor -- start 2024-03-06T07:28:31.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.053+0000 7fc1fb200640 1 -- start start 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.054+0000 7fc1fb200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc1f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.054+0000 7fc1fb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 0x7fc1fc0cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.054+0000 7fc1fb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc15e130 0x7fc1fc0cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.054+0000 7fc1fb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0c8030 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.054+0000 7fc1fb200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0c7eb0 con 0x7fc1fc0cfaf0 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.054+0000 7fc1fb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0c81b0 con 0x7fc1fc15e130 2024-03-06T07:28:31.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 0x7fc1fc0cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 0x7fc1fc0cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46592/0 (socket says 172.21.15.97:46592) 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 -- 172.21.15.97:0/4027597460 learned_addr learned my addr 172.21.15.97:0/4027597460 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9e00640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc1f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 -- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc15e130 msgr2=0x7fc1fc0cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc15e130 0x7fc1fc0cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 -- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc0cfaf0 msgr2=0x7fc1fc1f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc1f8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 -- 172.21.15.97:0/4027597460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc0cbab0 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.055+0000 7fc1f9400640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 0x7fc1fc0cb380 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009e20 tx=0x7fc1e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.056+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8043020 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.056+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e803e450 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.056+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc0cbd90 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.056+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8009510 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.056+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc0cc350 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.057+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc04f330 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.058+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1e8046020 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.059+0000 7fc1f2000640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.059+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fc1e80acbd0 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.060+0000 7fc1f9e00640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.060+0000 7fc1f9e00640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.063+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1e804d030 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.316+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc1fc09d690 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.316+0000 7fc1f2000640 1 -- 172.21.15.97:0/4027597460 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc1e807ba00 con 0x7fc1fc0d23e0 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.318+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1dc06b4d0 msgr2=0x7fc1dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 msgr2=0x7fc1fc0cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 0x7fc1fc0cb380 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009e20 tx=0x7fc1e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 shutdown_connections 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1fc15e130 0x7fc1fc0cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1fc0d23e0 0x7fc1fc0cb380 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 --2- 172.21.15.97:0/4027597460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc0cfaf0 0x7fc1fc1f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 >> 172.21.15.97:0/4027597460 conn(0x7fc1fc09d070 msgr2=0x7fc1fc15d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 shutdown_connections 2024-03-06T07:28:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.319+0000 7fc1fb200640 1 -- 172.21.15.97:0/4027597460 wait complete. 2024-03-06T07:28:31.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:31.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.533+0000 7fdfe1000640 1 Processor -- start 2024-03-06T07:28:31.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.533+0000 7fdfe1000640 1 -- start start 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.534+0000 7fdfe1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc15be20 0x7fdfdc0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.534+0000 7fdfe1000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdfdc0af5a0 0x7fdfdc0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.534+0000 7fdfe1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdfdc072670 0x7fdfdc072ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.534+0000 7fdfe1000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc073000 con 0x7fdfdc072670 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.534+0000 7fdfe1000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc066d00 con 0x7fdfdc0af5a0 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.534+0000 7fdfe1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc066f50 con 0x7fdfdc15be20 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdfdc072670 0x7fdfdc072ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdfdc072670 0x7fdfdc072ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46594/0 (socket says 172.21.15.97:46594) 2024-03-06T07:28:31.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 -- 172.21.15.97:0/1130057579 learned_addr learned my addr 172.21.15.97:0/1130057579 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdaa00640 1 --2- 172.21.15.97:0/1130057579 >> v2:172.21.15.135:3300/0 conn(0x7fdfdc0af5a0 0x7fdfdc0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdb400640 1 --2- 172.21.15.97:0/1130057579 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc15be20 0x7fdfdc0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 -- 172.21.15.97:0/1130057579 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc15be20 msgr2=0x7fdfdc0b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 --2- 172.21.15.97:0/1130057579 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc15be20 0x7fdfdc0b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 -- 172.21.15.97:0/1130057579 >> v2:172.21.15.135:3300/0 conn(0x7fdfdc0af5a0 msgr2=0x7fdfdc0b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 --2- 172.21.15.97:0/1130057579 >> v2:172.21.15.135:3300/0 conn(0x7fdfdc0af5a0 0x7fdfdc0b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdbe00640 1 -- 172.21.15.97:0/1130057579 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc067230 con 0x7fdfdc072670 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.535+0000 7fdfdaa00640 1 --2- 172.21.15.97:0/1130057579 >> v2:172.21.15.135:3300/0 conn(0x7fdfdc0af5a0 0x7fdfdc0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.536+0000 7fdfdbe00640 1 --2- 172.21.15.97:0/1130057579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc072670 0x7fdfdc072ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0737a0 tx=0x7fdfd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec9cf59cb71b14f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfda000640 1 -- 172.21.15.97:0/1130057579 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd402ca70 con 0x7fdfdc072670 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfda000640 1 -- 172.21.15.97:0/1130057579 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd402cc10 con 0x7fdfdc072670 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfda000640 1 -- 172.21.15.97:0/1130057579 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40343f0 con 0x7fdfdc072670 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 -- 172.21.15.97:0/1130057579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc072670 msgr2=0x7fdfdc072ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 --2- 172.21.15.97:0/1130057579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc072670 0x7fdfdc072ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0737a0 tx=0x7fdfd402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 -- 172.21.15.97:0/1130057579 shutdown_connections 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 --2- 172.21.15.97:0/1130057579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc072670 0x7fdfdc072ac0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 --2- 172.21.15.97:0/1130057579 >> v2:172.21.15.135:3300/0 conn(0x7fdfdc0af5a0 0x7fdfdc0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 --2- 172.21.15.97:0/1130057579 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc15be20 0x7fdfdc0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.537+0000 7fdfe1000640 1 -- 172.21.15.97:0/1130057579 >> 172.21.15.97:0/1130057579 conn(0x7fdfdc0b4fe0 msgr2=0x7fdfdc0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.538+0000 7fdfe1000640 1 -- 172.21.15.97:0/1130057579 shutdown_connections 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.538+0000 7fdfe1000640 1 -- 172.21.15.97:0/1130057579 wait complete. 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.539+0000 7fdfe1000640 1 Processor -- start 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.539+0000 7fdfe1000640 1 -- start start 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.540+0000 7fdfe1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc072670 0x7fdfdc099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.540+0000 7fdfe1000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfdc0af5a0 0x7fdfdc1571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.540+0000 7fdfe1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 0x7fdfdc15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.540+0000 7fdfe1000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0b1be0 con 0x7fdfdc15be20 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.540+0000 7fdfe1000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0b1180 con 0x7fdfdc0af5a0 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.540+0000 7fdfe1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0b0b50 con 0x7fdfdc072670 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 0x7fdfdc15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 0x7fdfdc15a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46598/0 (socket says 172.21.15.97:46598) 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 -- 172.21.15.97:0/2698023362 learned_addr learned my addr 172.21.15.97:0/2698023362 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdaa00640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfdc0af5a0 0x7fdfdc1571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 -- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc072670 msgr2=0x7fdfdc099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc072670 0x7fdfdc099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 -- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfdc0af5a0 msgr2=0x7fdfdc1571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfdc0af5a0 0x7fdfdc1571f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 -- 172.21.15.97:0/2698023362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc099e60 con 0x7fdfdc15be20 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.541+0000 7fdfdbe00640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 0x7fdfdc15a9e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4036040 tx=0x7fdfd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.542+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4043020 con 0x7fdfdc15be20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.542+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd403e450 con 0x7fdfdc15be20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.542+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc157cf0 con 0x7fdfdc15be20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.542+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4009510 con 0x7fdfdc15be20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.542+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc1582b0 con 0x7fdfdc15be20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.543+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc04f330 con 0x7fdfdc15be20 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.544+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfd4005d10 con 0x7fdfdc15be20 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.545+0000 7fdfd3400640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.545+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fdfd40acf20 con 0x7fdfdc15be20 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.545+0000 7fdfdb400640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.546+0000 7fdfdb400640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00a810 tx=0x7fdfcc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.550+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfd407bc30 con 0x7fdfdc15be20 2024-03-06T07:28:31.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.761+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdfdc0b2e20 con 0x7fdfdc15be20 2024-03-06T07:28:31.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.762+0000 7fdfd3400640 1 -- 172.21.15.97:0/2698023362 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdfd407b530 con 0x7fdfdc15be20 2024-03-06T07:28:31.765 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.764+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfbc06b4d0 msgr2=0x7fdfbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.764+0000 7fdfe1000640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00a810 tx=0x7fdfcc008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.764+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 msgr2=0x7fdfdc15a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.764+0000 7fdfe1000640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 0x7fdfdc15a9e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4036040 tx=0x7fdfd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 shutdown_connections 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc15be20 0x7fdfdc15a9e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfdc0af5a0 0x7fdfdc1571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 --2- 172.21.15.97:0/2698023362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdfdc072670 0x7fdfdc099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 >> 172.21.15.97:0/2698023362 conn(0x7fdfdc0b4fe0 msgr2=0x7fdfdc0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 shutdown_connections 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:31.765+0000 7fdfe1000640 1 -- 172.21.15.97:0/2698023362 wait complete. 2024-03-06T07:28:31.787 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-06T07:28:31.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-06T07:28:31.876 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:28:31.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.876 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:31.923 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:31.923 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:31.980 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:31.981 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:32.031 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:28:32.031 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:32.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:32.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.307+0000 7f2451a00640 1 Processor -- start 2024-03-06T07:28:32.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.307+0000 7f2451a00640 1 -- start start 2024-03-06T07:28:32.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.309+0000 7f2451a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f244c15dfd0 0x7f244c162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.309+0000 7f2451a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f244c06b6d0 0x7f244c0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.309+0000 7f2451a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0cef70 con 0x7f244c0affe0 2024-03-06T07:28:32.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.309+0000 7f2451a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c162270 con 0x7f244c06b6d0 2024-03-06T07:28:32.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.309+0000 7f2451a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c15e3b0 con 0x7f244c15dfd0 2024-03-06T07:28:32.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f244c06b6d0 0x7f244c0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f244c06b6d0 0x7f244c0d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46608/0 (socket says 172.21.15.97:46608) 2024-03-06T07:28:32.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 -- 172.21.15.97:0/2141163220 learned_addr learned my addr 172.21.15.97:0/2141163220 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:32.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f244be00640 1 --2- 172.21.15.97:0/2141163220 >> v2:172.21.15.135:3300/0 conn(0x7f244c15dfd0 0x7f244c162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 -- 172.21.15.97:0/2141163220 >> v2:172.21.15.135:3300/0 conn(0x7f244c15dfd0 msgr2=0x7f244c162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 --2- 172.21.15.97:0/2141163220 >> v2:172.21.15.135:3300/0 conn(0x7f244c15dfd0 0x7f244c162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 -- 172.21.15.97:0/2141163220 >> v1:172.21.15.135:6789/0 conn(0x7f244c0affe0 legacy=0x7f244c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f2451000640 1 -- 172.21.15.97:0/2141163220 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c161c60 con 0x7f244c06b6d0 2024-03-06T07:28:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.310+0000 7f244be00640 1 --2- 172.21.15.97:0/2141163220 >> v2:172.21.15.135:3300/0 conn(0x7f244c15dfd0 0x7f244c162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.311+0000 7f2451000640 1 --2- 172.21.15.97:0/2141163220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c06b6d0 0x7f244c0d06f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f244c0d1a70 tx=0x7f244402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b096b8b22c3e366 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f244aa00640 1 -- 172.21.15.97:0/2141163220 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244402c960 con 0x7f244c06b6d0 2024-03-06T07:28:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f244aa00640 1 -- 172.21.15.97:0/2141163220 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244402cb00 con 0x7f244c06b6d0 2024-03-06T07:28:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f244aa00640 1 -- 172.21.15.97:0/2141163220 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24440343f0 con 0x7f244c06b6d0 2024-03-06T07:28:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 -- 172.21.15.97:0/2141163220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c06b6d0 msgr2=0x7f244c0d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 --2- 172.21.15.97:0/2141163220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c06b6d0 0x7f244c0d06f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f244c0d1a70 tx=0x7f244402bdc0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 -- 172.21.15.97:0/2141163220 shutdown_connections 2024-03-06T07:28:32.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 --2- 172.21.15.97:0/2141163220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c06b6d0 0x7f244c0d06f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 --2- 172.21.15.97:0/2141163220 >> v2:172.21.15.135:3300/0 conn(0x7f244c15dfd0 0x7f244c162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 -- 172.21.15.97:0/2141163220 >> 172.21.15.97:0/2141163220 conn(0x7f244c06c0d0 msgr2=0x7f244c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 -- 172.21.15.97:0/2141163220 shutdown_connections 2024-03-06T07:28:32.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.312+0000 7f2451a00640 1 -- 172.21.15.97:0/2141163220 wait complete. 2024-03-06T07:28:32.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.314+0000 7f2451a00640 1 Processor -- start 2024-03-06T07:28:32.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.314+0000 7f2451a00640 1 -- start start 2024-03-06T07:28:32.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f2451a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f244c06b6d0 0x7f244c15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f2451a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 0x7f244c148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f2451a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f244c15dfd0 0x7f244c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f2451a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0ce8d0 con 0x7f244c0affe0 2024-03-06T07:28:32.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f2451a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c161f00 con 0x7f244c15dfd0 2024-03-06T07:28:32.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f2451a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0cea50 con 0x7f244c06b6d0 2024-03-06T07:28:32.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.315+0000 7f244b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 0x7f244c148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 0x7f244c148dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46622/0 (socket says 172.21.15.97:46622) 2024-03-06T07:28:32.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 -- 172.21.15.97:0/352620233 learned_addr learned my addr 172.21.15.97:0/352620233 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:32.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f2451000640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f244c15dfd0 0x7f244c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244be00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f244c06b6d0 0x7f244c15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 -- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f244c06b6d0 msgr2=0x7f244c15b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f244c06b6d0 0x7f244c15b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 -- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f244c15dfd0 msgr2=0x7f244c15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f244c15dfd0 0x7f244c15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 -- 172.21.15.97:0/352620233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c149300 con 0x7f244c0affe0 2024-03-06T07:28:32.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f2451000640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f244c15dfd0 0x7f244c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.316+0000 7f244b400640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 0x7f244c148dc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f243800c460 tx=0x7f243800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.317+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438016020 con 0x7f244c0affe0 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.317+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2438012870 con 0x7f244c0affe0 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.317+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438012c60 con 0x7f244c0affe0 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.317+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c15b9a0 con 0x7f244c0affe0 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.317+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c15bf60 con 0x7f244c0affe0 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.318+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c04f330 con 0x7f244c0affe0 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.319+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2438019020 con 0x7f244c0affe0 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.320+0000 7f2443e00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f242c06b4d0 0x7f242c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.320+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f2438089d60 con 0x7f244c0affe0 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.320+0000 7f244be00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f242c06b4d0 0x7f242c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.321+0000 7f244be00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f242c06b4d0 0x7f242c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f243c00b840 tx=0x7f243c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.329+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2438020070 con 0x7f244c0affe0 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.605+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f244c0cdea0 con 0x7f244c0affe0 2024-03-06T07:28:32.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.606+0000 7f2443e00640 1 -- 172.21.15.97:0/352620233 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16983 (secure 0 0 0) 0x7f2438058b00 con 0x7f244c0affe0 2024-03-06T07:28:32.609 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:32.609 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":24,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:28.349664+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:32.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f242c06b4d0 msgr2=0x7f242c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f242c06b4d0 0x7f242c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f243c00b840 tx=0x7f243c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 msgr2=0x7f244c148dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 0x7f244c148dc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f243800c460 tx=0x7f243800c920 comp rx=0 tx=0).stop 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 shutdown_connections 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f242c06b4d0 0x7f242c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f244c15dfd0 0x7f244c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f244c0affe0 0x7f244c148dc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 --2- 172.21.15.97:0/352620233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f244c06b6d0 0x7f244c15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 >> 172.21.15.97:0/352620233 conn(0x7f244c06c0d0 msgr2=0x7f244c0b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 shutdown_connections 2024-03-06T07:28:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.609+0000 7f2451a00640 1 -- 172.21.15.97:0/352620233 wait complete. 2024-03-06T07:28:32.630 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:32.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:32.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.839+0000 7fb8afa00640 1 Processor -- start 2024-03-06T07:28:32.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.839+0000 7fb8afa00640 1 -- start start 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.840+0000 7fb8afa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb8a815fc10 0x7fb8a8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8afa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb8a815fff0 0x7fb8a8160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8afa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a80ce4a0 0x7fb8a80ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8afa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a80b46c0 con 0x7fb8a815fff0 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8afa00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a80b49a0 con 0x7fb8a815fc10 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8afa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8072c90 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8a7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb8a815fff0 0x7fb8a8160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8a7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb8a815fff0 0x7fb8a8160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46628/0 (socket says 172.21.15.97:46628) 2024-03-06T07:28:32.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8a7e00640 1 -- 172.21.15.97:0/2683208079 learned_addr learned my addr 172.21.15.97:0/2683208079 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8ad000640 1 --2- 172.21.15.97:0/2683208079 >> v2:172.21.15.135:3300/0 conn(0x7fb8a815fc10 0x7fb8a8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.841+0000 7fb8ada00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a80ce4a0 0x7fb8a80ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.842+0000 7fb8a7e00640 1 -- 172.21.15.97:0/2683208079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a80ce4a0 msgr2=0x7fb8a80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.842+0000 7fb8a7e00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a80ce4a0 0x7fb8a80ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.842+0000 7fb8a7e00640 1 -- 172.21.15.97:0/2683208079 >> v2:172.21.15.135:3300/0 conn(0x7fb8a815fc10 msgr2=0x7fb8a8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.842+0000 7fb8a7e00640 1 --2- 172.21.15.97:0/2683208079 >> v2:172.21.15.135:3300/0 conn(0x7fb8a815fc10 0x7fb8a8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.842+0000 7fb8a7e00640 1 -- 172.21.15.97:0/2683208079 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a8160990 con 0x7fb8a815fff0 2024-03-06T07:28:32.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.842+0000 7fb8ada00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a80ce4a0 0x7fb8a80ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.843+0000 7fb8a7e00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a815fff0 0x7fb8a8160450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80cf770 tx=0x7fb89402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4be0def604818f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.843+0000 7fb8a7400640 1 -- 172.21.15.97:0/2683208079 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894035020 con 0x7fb8a815fff0 2024-03-06T07:28:32.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.843+0000 7fb8a7400640 1 -- 172.21.15.97:0/2683208079 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb894031870 con 0x7fb8a815fff0 2024-03-06T07:28:32.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.843+0000 7fb8a7400640 1 -- 172.21.15.97:0/2683208079 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894031c40 con 0x7fb8a815fff0 2024-03-06T07:28:32.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 -- 172.21.15.97:0/2683208079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a815fff0 msgr2=0x7fb8a8160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a815fff0 0x7fb8a8160450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80cf770 tx=0x7fb89402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:32.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 -- 172.21.15.97:0/2683208079 shutdown_connections 2024-03-06T07:28:32.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a80ce4a0 0x7fb8a80ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 --2- 172.21.15.97:0/2683208079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a815fff0 0x7fb8a8160450 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 --2- 172.21.15.97:0/2683208079 >> v2:172.21.15.135:3300/0 conn(0x7fb8a815fc10 0x7fb8a8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 -- 172.21.15.97:0/2683208079 >> 172.21.15.97:0/2683208079 conn(0x7fb8a80b1940 msgr2=0x7fb8a80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 -- 172.21.15.97:0/2683208079 shutdown_connections 2024-03-06T07:28:32.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.844+0000 7fb8afa00640 1 -- 172.21.15.97:0/2683208079 wait complete. 2024-03-06T07:28:32.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.846+0000 7fb8afa00640 1 Processor -- start 2024-03-06T07:28:32.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.846+0000 7fb8afa00640 1 -- start start 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8afa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 0x7fb8a809c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8afa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8a815fc10 0x7fb8a8155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8afa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a815fff0 0x7fb8a809a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8afa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8072a30 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8afa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80728b0 con 0x7fb8a815fc10 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8afa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a815c2e0 con 0x7fb8a815fff0 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 0x7fb8a809c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 0x7fb8a809c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46638/0 (socket says 172.21.15.97:46638) 2024-03-06T07:28:32.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 -- 172.21.15.97:0/3779300453 learned_addr learned my addr 172.21.15.97:0/3779300453 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:32.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 -- 172.21.15.97:0/3779300453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a815fff0 msgr2=0x7fb8a809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a815fff0 0x7fb8a809a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 -- 172.21.15.97:0/3779300453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8a815fc10 msgr2=0x7fb8a8155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8a815fc10 0x7fb8a8155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.848+0000 7fb8ad000640 1 -- 172.21.15.97:0/3779300453 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a809b990 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.849+0000 7fb8ad000640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 0x7fb8a809c750 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00d800 tx=0x7fb89c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.849+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00b6c0 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.849+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c019070 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.849+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00ba80 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.849+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a809bc70 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.849+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a809c230 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.851+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a804f330 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.851+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb89c00bc20 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.852+0000 7fb8a4c00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb88806b4d0 0x7fb88806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.852+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fb89c08a7a0 con 0x7fb8a80ce4a0 2024-03-06T07:28:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.852+0000 7fb8a7e00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb88806b4d0 0x7fb88806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.853+0000 7fb8a7e00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb88806b4d0 0x7fb88806d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb894036000 tx=0x7fb894030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:32.857+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb89c059530 con 0x7fb8a80ce4a0 2024-03-06T07:28:33.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.145+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8a806ef30 con 0x7fb8a80ce4a0 2024-03-06T07:28:33.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.145+0000 7fb8a4c00640 1 -- 172.21.15.97:0/3779300453 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb89c058e30 con 0x7fb8a80ce4a0 2024-03-06T07:28:33.148 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:33.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb88806b4d0 msgr2=0x7fb88806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb88806b4d0 0x7fb88806d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb894036000 tx=0x7fb894030000 comp rx=0 tx=0).stop 2024-03-06T07:28:33.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 msgr2=0x7fb8a809c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 0x7fb8a809c750 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00d800 tx=0x7fb89c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 shutdown_connections 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb88806b4d0 0x7fb88806d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8a815fff0 0x7fb8a809a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8a815fc10 0x7fb8a8155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 --2- 172.21.15.97:0/3779300453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8a80ce4a0 0x7fb8a809c750 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 >> 172.21.15.97:0/3779300453 conn(0x7fb8a80b1940 msgr2=0x7fb8a80ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 shutdown_connections 2024-03-06T07:28:33.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.148+0000 7fb8afa00640 1 -- 172.21.15.97:0/3779300453 wait complete. 2024-03-06T07:28:33.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:33.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.365+0000 7f68aec00640 1 Processor -- start 2024-03-06T07:28:33.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.365+0000 7f68aec00640 1 -- start start 2024-03-06T07:28:33.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.368+0000 7f68aec00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68a80b3310 0x7f68a815fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.368+0000 7f68aec00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f68a81629e0 0x7f68a80cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.368+0000 7f68aec00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80ccc40 con 0x7f68a80ce3a0 2024-03-06T07:28:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.368+0000 7f68aec00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80d05c0 con 0x7f68a80b3310 2024-03-06T07:28:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.368+0000 7f68aec00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80d08a0 con 0x7f68a81629e0 2024-03-06T07:28:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68a80b3310 0x7f68a815fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68a80b3310 0x7f68a815fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46644/0 (socket says 172.21.15.97:46644) 2024-03-06T07:28:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 -- 172.21.15.97:0/320594338 learned_addr learned my addr 172.21.15.97:0/320594338 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7400640 1 --2- 172.21.15.97:0/320594338 >> v2:172.21.15.135:3300/0 conn(0x7f68a81629e0 0x7f68a80cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 -- 172.21.15.97:0/320594338 >> v2:172.21.15.135:3300/0 conn(0x7f68a81629e0 msgr2=0x7f68a80cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 --2- 172.21.15.97:0/320594338 >> v2:172.21.15.135:3300/0 conn(0x7f68a81629e0 0x7f68a80cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 -- 172.21.15.97:0/320594338 >> v1:172.21.15.135:6789/0 conn(0x7f68a80ce3a0 legacy=0x7f68a80cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7e00640 1 -- 172.21.15.97:0/320594338 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80d0b80 con 0x7f68a80b3310 2024-03-06T07:28:33.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.369+0000 7f68a7400640 1 --2- 172.21.15.97:0/320594338 >> v2:172.21.15.135:3300/0 conn(0x7f68a81629e0 0x7f68a80cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.370+0000 7f68a7e00640 1 --2- 172.21.15.97:0/320594338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a80b3310 0x7f68a815fc20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f68a8072020 tx=0x7f689802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de577f300b640a14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.371+0000 7f68a6a00640 1 -- 172.21.15.97:0/320594338 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980317c0 con 0x7f68a80b3310 2024-03-06T07:28:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.371+0000 7f68a6a00640 1 -- 172.21.15.97:0/320594338 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6898037070 con 0x7f68a80b3310 2024-03-06T07:28:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.371+0000 7f68a6a00640 1 -- 172.21.15.97:0/320594338 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898031b80 con 0x7f68a80b3310 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 -- 172.21.15.97:0/320594338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a80b3310 msgr2=0x7f68a815fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 --2- 172.21.15.97:0/320594338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a80b3310 0x7f68a815fc20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f68a8072020 tx=0x7f689802b860 comp rx=0 tx=0).stop 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 -- 172.21.15.97:0/320594338 shutdown_connections 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 --2- 172.21.15.97:0/320594338 >> v2:172.21.15.135:3300/0 conn(0x7f68a81629e0 0x7f68a80cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 --2- 172.21.15.97:0/320594338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a80b3310 0x7f68a815fc20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 -- 172.21.15.97:0/320594338 >> 172.21.15.97:0/320594338 conn(0x7f68a80b3790 msgr2=0x7f68a80b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 -- 172.21.15.97:0/320594338 shutdown_connections 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.372+0000 7f68aec00640 1 -- 172.21.15.97:0/320594338 wait complete. 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.374+0000 7f68aec00640 1 Processor -- start 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.374+0000 7f68aec00640 1 -- start start 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68aec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68a80b3310 0x7f68a80caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68aec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a80ce3a0 0x7f68a8154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68aec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 0x7f68a806fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68aec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80d0fe0 con 0x7f68a81629e0 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68aec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80d0e60 con 0x7f68a80ce3a0 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68aec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80ce7f0 con 0x7f68a80b3310 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68acc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 0x7f68a806fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68acc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 0x7f68a806fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46656/0 (socket says 172.21.15.97:46656) 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68acc00640 1 -- 172.21.15.97:0/1175065992 learned_addr learned my addr 172.21.15.97:0/1175065992 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.375+0000 7f68a7e00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68a80b3310 0x7f68a80caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68a7400640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a80ce3a0 0x7f68a8154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68acc00640 1 -- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68a80b3310 msgr2=0x7f68a80caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68acc00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68a80b3310 0x7f68a80caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68acc00640 1 -- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a80ce3a0 msgr2=0x7f68a8154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68acc00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a80ce3a0 0x7f68a8154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68acc00640 1 -- 172.21.15.97:0/1175065992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80c99b0 con 0x7f68a81629e0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68a7400640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a80ce3a0 0x7f68a8154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68acc00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 0x7f68a806fce0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c780 tx=0x7f68a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0004b60 con 0x7f68a81629e0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0004d00 con 0x7f68a81629e0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a000aa90 con 0x7f68a81629e0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a80c9c90 con 0x7f68a81629e0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.376+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a80ca250 con 0x7f68a81629e0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.378+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a804f330 con 0x7f68a81629e0 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.379+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68a000ac30 con 0x7f68a81629e0 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.380+0000 7f689fe00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f688806b4d0 0x7f688806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.380+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f68a008a5a0 con 0x7f68a81629e0 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.380+0000 7f68a7e00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f688806b4d0 0x7f688806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.380+0000 7f68a7e00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f688806b4d0 0x7f688806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6898036000 tx=0x7f689803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.384+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68a0020070 con 0x7f68a81629e0 2024-03-06T07:28:33.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.579+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68a80c85e0 con 0x7f68a81629e0 2024-03-06T07:28:33.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.579+0000 7f689fe00640 1 -- 172.21.15.97:0/1175065992 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f68a0059340 con 0x7f68a81629e0 2024-03-06T07:28:33.582 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:33.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f688806b4d0 msgr2=0x7f688806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f688806b4d0 0x7f688806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6898036000 tx=0x7f689803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 msgr2=0x7f68a806fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 0x7f68a806fce0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c780 tx=0x7f68a000cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 shutdown_connections 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f688806b4d0 0x7f688806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68a81629e0 0x7f68a806fce0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a80ce3a0 0x7f68a8154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 --2- 172.21.15.97:0/1175065992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68a80b3310 0x7f68a80caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 >> 172.21.15.97:0/1175065992 conn(0x7f68a80b3790 msgr2=0x7f68a806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 shutdown_connections 2024-03-06T07:28:33.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.582+0000 7f68aec00640 1 -- 172.21.15.97:0/1175065992 wait complete. 2024-03-06T07:28:33.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:33.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.811+0000 7f3e08400640 1 Processor -- start 2024-03-06T07:28:33.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.811+0000 7f3e08400640 1 -- start start 2024-03-06T07:28:33.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e08400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3e0015c2a0 0x7f3e000af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e08400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e08400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3e0006afc0 0x7f3e0006b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e08400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e000b0690 con 0x7f3e0015c2a0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e08400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e000b2d80 con 0x7f3e0006afc0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e08400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000731b0 con 0x7f3e00066e00 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e05a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3e0015c2a0 0x7f3e000af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e05a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3e0015c2a0 0x7f3e000af9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46666/0 (socket says 172.21.15.97:46666) 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.812+0000 7f3e05a00640 1 -- 172.21.15.97:0/2874381039 learned_addr learned my addr 172.21.15.97:0/2874381039 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.813+0000 7f3e05000640 1 --2- 172.21.15.97:0/2874381039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.813+0000 7f3e05a00640 1 -- 172.21.15.97:0/2874381039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 msgr2=0x7f3e000ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.813+0000 7f3e05a00640 1 --2- 172.21.15.97:0/2874381039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.813+0000 7f3e05a00640 1 -- 172.21.15.97:0/2874381039 >> v2:172.21.15.135:3300/0 conn(0x7f3e0006afc0 msgr2=0x7f3e0006b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.813+0000 7f3e05a00640 1 --2- 172.21.15.97:0/2874381039 >> v2:172.21.15.135:3300/0 conn(0x7f3e0006afc0 0x7f3e0006b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.813+0000 7f3e05a00640 1 -- 172.21.15.97:0/2874381039 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00073450 con 0x7f3e0015c2a0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.814+0000 7f3e05a00640 1 --2- 172.21.15.97:0/2874381039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e000af9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009870 tx=0x7f3df402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970322ca41b91b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.814+0000 7f3dffe00640 1 -- 172.21.15.97:0/2874381039 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4035020 con 0x7f3e0015c2a0 2024-03-06T07:28:33.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.814+0000 7f3dffe00640 1 -- 172.21.15.97:0/2874381039 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4031640 con 0x7f3e0015c2a0 2024-03-06T07:28:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.814+0000 7f3dffe00640 1 -- 172.21.15.97:0/2874381039 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df402d040 con 0x7f3e0015c2a0 2024-03-06T07:28:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 -- 172.21.15.97:0/2874381039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 msgr2=0x7f3e000af9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 --2- 172.21.15.97:0/2874381039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e000af9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009870 tx=0x7f3df402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 -- 172.21.15.97:0/2874381039 shutdown_connections 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 --2- 172.21.15.97:0/2874381039 >> v2:172.21.15.135:3300/0 conn(0x7f3e0006afc0 0x7f3e0006b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 --2- 172.21.15.97:0/2874381039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 --2- 172.21.15.97:0/2874381039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e000af9f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 -- 172.21.15.97:0/2874381039 >> 172.21.15.97:0/2874381039 conn(0x7f3e000b1e80 msgr2=0x7f3e000b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 -- 172.21.15.97:0/2874381039 shutdown_connections 2024-03-06T07:28:33.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.815+0000 7f3e08400640 1 -- 172.21.15.97:0/2874381039 wait complete. 2024-03-06T07:28:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.818+0000 7f3e08400640 1 Processor -- start 2024-03-06T07:28:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.818+0000 7f3e08400640 1 -- start start 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e08400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e08400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006afc0 0x7f3e000cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e08400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e0015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e08400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000b29a0 con 0x7f3e0015c2a0 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e08400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0015c680 con 0x7f3e0006afc0 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e08400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000b2b20 con 0x7f3e00066e00 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e06400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e0015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e06400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e0015d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46680/0 (socket says 172.21.15.97:46680) 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.820+0000 7f3e06400640 1 -- 172.21.15.97:0/1083703066 learned_addr learned my addr 172.21.15.97:0/1083703066 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e05a00640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e05000640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006afc0 0x7f3e000cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e06400640 1 -- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 msgr2=0x7f3e000998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e06400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e06400640 1 -- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006afc0 msgr2=0x7f3e000cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e06400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006afc0 0x7f3e000cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e06400640 1 -- 172.21.15.97:0/1083703066 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e000cdf20 con 0x7f3e0015c2a0 2024-03-06T07:28:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e06400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e0015d580 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3df800b9a0 tx=0x7f3df800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8019070 con 0x7f3e0015c2a0 2024-03-06T07:28:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df800cbc0 con 0x7f3e0015c2a0 2024-03-06T07:28:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e000ce200 con 0x7f3e0015c2a0 2024-03-06T07:28:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8014440 con 0x7f3e0015c2a0 2024-03-06T07:28:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.821+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e000ce7c0 con 0x7f3e0015c2a0 2024-03-06T07:28:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.822+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0004f330 con 0x7f3e0015c2a0 2024-03-06T07:28:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.824+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3df801c070 con 0x7f3e0015c2a0 2024-03-06T07:28:33.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.825+0000 7f3dfd600640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b4d0 0x7f3de006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.825+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f3df808a0c0 con 0x7f3e0015c2a0 2024-03-06T07:28:33.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.825+0000 7f3e05a00640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b4d0 0x7f3de006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.826+0000 7f3e05a00640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b4d0 0x7f3de006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3df4036000 tx=0x7f3df403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:33.829+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3df8058dd0 con 0x7f3e0015c2a0 2024-03-06T07:28:34.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.025+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3e0015b790 con 0x7f3e0015c2a0 2024-03-06T07:28:34.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.025+0000 7f3dfd600640 1 -- 172.21.15.97:0/1083703066 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3df80586d0 con 0x7f3e0015c2a0 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:34.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b4d0 msgr2=0x7f3de006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b4d0 0x7f3de006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3df4036000 tx=0x7f3df403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 msgr2=0x7f3e0015d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e0015d580 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3df800b9a0 tx=0x7f3df800be60 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 shutdown_connections 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b4d0 0x7f3de006d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e0015c2a0 0x7f3e0015d580 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006afc0 0x7f3e000cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 --2- 172.21.15.97:0/1083703066 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e00066e00 0x7f3e000998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 >> 172.21.15.97:0/1083703066 conn(0x7f3e000b1e80 msgr2=0x7f3e000b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 shutdown_connections 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.028+0000 7f3e08400640 1 -- 172.21.15.97:0/1083703066 wait complete. 2024-03-06T07:28:34.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:34.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.241+0000 7fa8e5600640 1 Processor -- start 2024-03-06T07:28:34.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.241+0000 7fa8e5600640 1 -- start start 2024-03-06T07:28:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8e5600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa8e015e360 0x7fa8e00cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8e5600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8e00d12e0 0x7fa8e0161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8e5600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e00cf2b0 con 0x7fa8e0162270 2024-03-06T07:28:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8e5600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e01626c0 con 0x7fa8e00d12e0 2024-03-06T07:28:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8e5600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00cf590 con 0x7fa8e015e360 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8d7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8e00d12e0 0x7fa8e0161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8d7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8e00d12e0 0x7fa8e0161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55052/0 (socket says 172.21.15.97:55052) 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8df400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa8e0162270 0x7fa8e015e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48400/0 (socket says 172.21.15.97:48400) 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.243+0000 7fa8d7e00640 1 -- 172.21.15.97:0/348747303 learned_addr learned my addr 172.21.15.97:0/348747303 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.244+0000 7fa8dea00640 1 --2- 172.21.15.97:0/348747303 >> v2:172.21.15.135:3300/0 conn(0x7fa8e015e360 0x7fa8e00cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.244+0000 7fa8d7e00640 1 -- 172.21.15.97:0/348747303 >> v2:172.21.15.135:3300/0 conn(0x7fa8e015e360 msgr2=0x7fa8e00cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.244+0000 7fa8d7e00640 1 --2- 172.21.15.97:0/348747303 >> v2:172.21.15.135:3300/0 conn(0x7fa8e015e360 0x7fa8e00cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.244+0000 7fa8d7e00640 1 -- 172.21.15.97:0/348747303 >> v1:172.21.15.97:6789/0 conn(0x7fa8e0162270 legacy=0x7fa8e015e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.244+0000 7fa8d7e00640 1 -- 172.21.15.97:0/348747303 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cf870 con 0x7fa8e00d12e0 2024-03-06T07:28:34.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.244+0000 7fa8de000640 1 -- 172.21.15.97:0/348747303 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536063620 0 0) 0x7fa8e00cf2b0 con 0x7fa8e0162270 2024-03-06T07:28:34.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.246+0000 7fa8d7e00640 1 --2- 172.21.15.97:0/348747303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0161d30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00d4fc0 tx=0x7fa8c802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519cd55e58866321 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.246+0000 7fa8de000640 1 -- 172.21.15.97:0/348747303 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c802f850 con 0x7fa8e00d12e0 2024-03-06T07:28:34.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.246+0000 7fa8de000640 1 -- 172.21.15.97:0/348747303 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c802f9f0 con 0x7fa8e00d12e0 2024-03-06T07:28:34.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.246+0000 7fa8de000640 1 -- 172.21.15.97:0/348747303 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c802fdc0 con 0x7fa8e00d12e0 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 -- 172.21.15.97:0/348747303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e00d12e0 msgr2=0x7fa8e0161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 --2- 172.21.15.97:0/348747303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0161d30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00d4fc0 tx=0x7fa8c802ba30 comp rx=0 tx=0).stop 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 -- 172.21.15.97:0/348747303 shutdown_connections 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 --2- 172.21.15.97:0/348747303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0161d30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 --2- 172.21.15.97:0/348747303 >> v2:172.21.15.135:3300/0 conn(0x7fa8e015e360 0x7fa8e00cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 -- 172.21.15.97:0/348747303 >> 172.21.15.97:0/348747303 conn(0x7fa8e00d3f50 msgr2=0x7fa8e00d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 -- 172.21.15.97:0/348747303 shutdown_connections 2024-03-06T07:28:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.247+0000 7fa8e5600640 1 -- 172.21.15.97:0/348747303 wait complete. 2024-03-06T07:28:34.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.249+0000 7fa8e5600640 1 Processor -- start 2024-03-06T07:28:34.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.249+0000 7fa8e5600640 1 -- start start 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8e5600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8e5600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8e015e360 0x7fa8e00c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8e5600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 0x7fa8e00cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8e5600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00d1770 con 0x7fa8e0162270 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8e5600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00cfe30 con 0x7fa8e00d12e0 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8e5600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00d18f0 con 0x7fa8e015e360 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 0x7fa8e00cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 0x7fa8e00cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55060/0 (socket says 172.21.15.97:55060) 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 -- 172.21.15.97:0/1057331948 learned_addr learned my addr 172.21.15.97:0/1057331948 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8d7e00640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8e015e360 0x7fa8e00c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8dea00640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 -- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8e015e360 msgr2=0x7fa8e00c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8e015e360 0x7fa8e00c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 -- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8e00d12e0 msgr2=0x7fa8e0158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8df400640 1 -- 172.21.15.97:0/1057331948 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cbaa0 con 0x7fa8e0162270 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.251+0000 7fa8d7e00640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8e015e360 0x7fa8e00c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.252+0000 7fa8df400640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 0x7fa8e00cb020 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800c730 tx=0x7fa8d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.252+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8018070 con 0x7fa8e0162270 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.252+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d800ce40 con 0x7fa8e0162270 2024-03-06T07:28:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.252+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d800a860 con 0x7fa8e0162270 2024-03-06T07:28:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.252+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e00cbd80 con 0x7fa8e0162270 2024-03-06T07:28:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.252+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e00cc620 con 0x7fa8e0162270 2024-03-06T07:28:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.253+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e00cbf50 con 0x7fa8e0162270 2024-03-06T07:28:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.255+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8d800aa00 con 0x7fa8e0162270 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.255+0000 7fa8d6a00640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.256+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fa8d8089a00 con 0x7fa8e0162270 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.256+0000 7fa8dea00640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.257+0000 7fa8dea00640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0006fd0 tx=0x7fa8d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.260+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8d8058680 con 0x7fa8e0162270 2024-03-06T07:28:34.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.469+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa8e004f330 con 0x7fa8e0162270 2024-03-06T07:28:34.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.470+0000 7fa8d6a00640 1 -- 172.21.15.97:0/1057331948 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa8d8057f80 con 0x7fa8e0162270 2024-03-06T07:28:34.472 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.472+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8bc06b4d0 msgr2=0x7fa8bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0006fd0 tx=0x7fa8d0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 msgr2=0x7fa8e00cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 0x7fa8e00cb020 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800c730 tx=0x7fa8d800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 shutdown_connections 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8e0162270 0x7fa8e00cb020 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8e015e360 0x7fa8e00c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 --2- 172.21.15.97:0/1057331948 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8e00d12e0 0x7fa8e0158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 >> 172.21.15.97:0/1057331948 conn(0x7fa8e00d3f50 msgr2=0x7fa8e015b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 shutdown_connections 2024-03-06T07:28:34.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.473+0000 7fa8e5600640 1 -- 172.21.15.97:0/1057331948 wait complete. 2024-03-06T07:28:34.494 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-06T07:28:34.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:34.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-06T07:28:34.594 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:28:34.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:34.594 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:34.643 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:34.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.855+0000 7f164d400640 1 Processor -- start 2024-03-06T07:28:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.855+0000 7f164d400640 1 -- start start 2024-03-06T07:28:34.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.856+0000 7f164d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1648144c00 0x7f164806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f164d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164815f9d0 0x7f164815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f164d400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164806ee70 con 0x7f16480ce0a0 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f164d400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1648160370 con 0x7f1648144c00 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f164d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164806cef0 con 0x7f164815f9d0 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1648144c00 0x7f164806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1648144c00 0x7f164806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55066/0 (socket says 172.21.15.97:55066) 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 -- 172.21.15.97:0/2397005763 learned_addr learned my addr 172.21.15.97:0/2397005763 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1647200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f16480ce0a0 0x7f16480ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48416/0 (socket says 172.21.15.97:48416) 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1645e00640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164815f9d0 0x7f164815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 -- 172.21.15.97:0/2397005763 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164815f9d0 msgr2=0x7f164815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164815f9d0 0x7f164815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 -- 172.21.15.97:0/2397005763 >> v1:172.21.15.97:6789/0 conn(0x7f16480ce0a0 legacy=0x7f16480ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1646800640 1 -- 172.21.15.97:0/2397005763 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164806d1d0 con 0x7f1648144c00 2024-03-06T07:28:34.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.857+0000 7f1645e00640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164815f9d0 0x7f164815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.859+0000 7f1646800640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1648144c00 0x7f164806e930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1638009870 tx=0x7f163802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6656c781fd69190 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.859+0000 7f1645400640 1 -- 172.21.15.97:0/2397005763 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638034070 con 0x7f1648144c00 2024-03-06T07:28:34.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.859+0000 7f1645400640 1 -- 172.21.15.97:0/2397005763 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1638030840 con 0x7f1648144c00 2024-03-06T07:28:34.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.859+0000 7f1645400640 1 -- 172.21.15.97:0/2397005763 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638030c10 con 0x7f1648144c00 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 -- 172.21.15.97:0/2397005763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1648144c00 msgr2=0x7f164806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1648144c00 0x7f164806e930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1638009870 tx=0x7f163802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 -- 172.21.15.97:0/2397005763 shutdown_connections 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164815f9d0 0x7f164815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 --2- 172.21.15.97:0/2397005763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1648144c00 0x7f164806e930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 -- 172.21.15.97:0/2397005763 >> 172.21.15.97:0/2397005763 conn(0x7f16480b1730 msgr2=0x7f16480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 -- 172.21.15.97:0/2397005763 shutdown_connections 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.860+0000 7f164d400640 1 -- 172.21.15.97:0/2397005763 wait complete. 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.862+0000 7f164d400640 1 Processor -- start 2024-03-06T07:28:34.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.862+0000 7f164d400640 1 -- start start 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f164d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16480ce0a0 0x7f164813a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f164d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1648144c00 0x7f1648138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f164d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 0x7f1648139420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f164d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1648160730 con 0x7f164815f9d0 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f164d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164806d790 con 0x7f16480ce0a0 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f164d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164806cb50 con 0x7f1648144c00 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f1647200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 0x7f1648139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f1647200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 0x7f1648139420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55082/0 (socket says 172.21.15.97:55082) 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f1647200640 1 -- 172.21.15.97:0/593761944 learned_addr learned my addr 172.21.15.97:0/593761944 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f1646800640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16480ce0a0 0x7f164813a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.863+0000 7f1645e00640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1648144c00 0x7f1648138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1647200640 1 -- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1648144c00 msgr2=0x7f1648138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1647200640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1648144c00 0x7f1648138d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1647200640 1 -- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16480ce0a0 msgr2=0x7f164813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1647200640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16480ce0a0 0x7f164813a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1647200640 1 -- 172.21.15.97:0/593761944 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648139bc0 con 0x7f164815f9d0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1646800640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16480ce0a0 0x7f164813a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f1647200640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 0x7f1648139420 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f164000c760 tx=0x7f164000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640017070 con 0x7f164815f9d0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164000a640 con 0x7f164815f9d0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16480b2bd0 con 0x7f164815f9d0 2024-03-06T07:28:34.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164000a9f0 con 0x7f164815f9d0 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.864+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16480b3190 con 0x7f164815f9d0 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.866+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164804f330 con 0x7f164815f9d0 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.866+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f164001a070 con 0x7f164815f9d0 2024-03-06T07:28:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.868+0000 7f163ea00640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b500 0x7f162806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.868+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f1640089820 con 0x7f164815f9d0 2024-03-06T07:28:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.868+0000 7f1646800640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b500 0x7f162806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.869+0000 7f1646800640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b500 0x7f162806d9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1638036000 tx=0x7f163803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:34.872+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f164806ee70 con 0x7f164815f9d0 2024-03-06T07:28:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.108+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1648072890 con 0x7f164815f9d0 2024-03-06T07:28:35.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.109+0000 7f163ea00640 1 -- 172.21.15.97:0/593761944 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f1640057ee0 con 0x7f164815f9d0 2024-03-06T07:28:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:35.111 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:35.112 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:28:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b500 msgr2=0x7f162806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b500 0x7f162806d9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1638036000 tx=0x7f163803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 msgr2=0x7f1648139420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 0x7f1648139420 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f164000c760 tx=0x7f164000cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 shutdown_connections 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b500 0x7f162806d9c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164815f9d0 0x7f1648139420 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1648144c00 0x7f1648138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 --2- 172.21.15.97:0/593761944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16480ce0a0 0x7f164813a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 >> 172.21.15.97:0/593761944 conn(0x7f16480b1730 msgr2=0x7f1648142d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 shutdown_connections 2024-03-06T07:28:35.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.112+0000 7f164d400640 1 -- 172.21.15.97:0/593761944 wait complete. 2024-03-06T07:28:35.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.314+0000 7f8a2a000640 1 Processor -- start 2024-03-06T07:28:35.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.315+0000 7f8a2a000640 1 -- start start 2024-03-06T07:28:35.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.316+0000 7f8a2a000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8a2415f9d0 0x7f8a2415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.316+0000 7f8a2a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a24070b10 0x7f8a24070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.316+0000 7f8a2a000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2406cef0 con 0x7f8a24144c00 2024-03-06T07:28:35.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.316+0000 7f8a2a000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2406d1d0 con 0x7f8a24070b10 2024-03-06T07:28:35.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.316+0000 7f8a2a000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2406d4b0 con 0x7f8a2415f9d0 2024-03-06T07:28:35.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.316+0000 7f8a23e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a24070b10 0x7f8a24070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8a24144c00 0x7f8a2406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48422/0 (socket says 172.21.15.97:48422) 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23400640 1 -- 172.21.15.97:0/1333840905 learned_addr learned my addr 172.21.15.97:0/1333840905 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a24070b10 0x7f8a24070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55090/0 (socket says 172.21.15.97:55090) 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a1be00640 1 --2- 172.21.15.97:0/1333840905 >> v2:172.21.15.135:3300/0 conn(0x7f8a2415f9d0 0x7f8a2415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23e00640 1 -- 172.21.15.97:0/1333840905 >> v2:172.21.15.135:3300/0 conn(0x7f8a2415f9d0 msgr2=0x7f8a2415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23e00640 1 --2- 172.21.15.97:0/1333840905 >> v2:172.21.15.135:3300/0 conn(0x7f8a2415f9d0 0x7f8a2415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23e00640 1 -- 172.21.15.97:0/1333840905 >> v1:172.21.15.97:6789/0 conn(0x7f8a24144c00 legacy=0x7f8a2406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a23e00640 1 -- 172.21.15.97:0/1333840905 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2406d790 con 0x7f8a24070b10 2024-03-06T07:28:35.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.317+0000 7f8a22a00640 1 -- 172.21.15.97:0/1333840905 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534052325 0 0) 0x7f8a2406cef0 con 0x7f8a24144c00 2024-03-06T07:28:35.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.318+0000 7f8a23e00640 1 --2- 172.21.15.97:0/1333840905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a24070b10 0x7f8a24070f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8a240b26f0 tx=0x7f8a1c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8396bc19abd5be64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.319+0000 7f8a22a00640 1 -- 172.21.15.97:0/1333840905 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c039070 con 0x7f8a24070b10 2024-03-06T07:28:35.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.319+0000 7f8a22a00640 1 -- 172.21.15.97:0/1333840905 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c02cc70 con 0x7f8a24070b10 2024-03-06T07:28:35.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.319+0000 7f8a22a00640 1 -- 172.21.15.97:0/1333840905 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c034660 con 0x7f8a24070b10 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 -- 172.21.15.97:0/1333840905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a24070b10 msgr2=0x7f8a24070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 --2- 172.21.15.97:0/1333840905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a24070b10 0x7f8a24070f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8a240b26f0 tx=0x7f8a1c02bec0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 -- 172.21.15.97:0/1333840905 shutdown_connections 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 --2- 172.21.15.97:0/1333840905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a24070b10 0x7f8a24070f20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 --2- 172.21.15.97:0/1333840905 >> v2:172.21.15.135:3300/0 conn(0x7f8a2415f9d0 0x7f8a2415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 -- 172.21.15.97:0/1333840905 >> 172.21.15.97:0/1333840905 conn(0x7f8a240b1730 msgr2=0x7f8a240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 -- 172.21.15.97:0/1333840905 shutdown_connections 2024-03-06T07:28:35.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.320+0000 7f8a2a000640 1 -- 172.21.15.97:0/1333840905 wait complete. 2024-03-06T07:28:35.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.322+0000 7f8a2a000640 1 Processor -- start 2024-03-06T07:28:35.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.322+0000 7f8a2a000640 1 -- start start 2024-03-06T07:28:35.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a2a000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a24070b10 0x7f8a2413b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a2a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a24144c00 0x7f8a2413e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a2a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 0x7f8a240ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a2a000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2406ccd0 con 0x7f8a2415f9d0 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a2a000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2406cb50 con 0x7f8a24070b10 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a2a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2406df90 con 0x7f8a24144c00 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 0x7f8a240ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 0x7f8a240ae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55100/0 (socket says 172.21.15.97:55100) 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 -- 172.21.15.97:0/479718828 learned_addr learned my addr 172.21.15.97:0/479718828 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a1be00640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a24144c00 0x7f8a2413e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23400640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a24070b10 0x7f8a2413b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 -- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a24144c00 msgr2=0x7f8a2413e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a24144c00 0x7f8a2413e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 -- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a24070b10 msgr2=0x7f8a2413b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a24070b10 0x7f8a2413b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.323+0000 7f8a23e00640 1 -- 172.21.15.97:0/479718828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2413f780 con 0x7f8a2415f9d0 2024-03-06T07:28:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.324+0000 7f8a23e00640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 0x7f8a240ae2c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c02c4d0 tx=0x7f8a1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.324+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c039050 con 0x7f8a2415f9d0 2024-03-06T07:28:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.324+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c045070 con 0x7f8a2415f9d0 2024-03-06T07:28:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.324+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2413fa60 con 0x7f8a2415f9d0 2024-03-06T07:28:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.324+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c03e830 con 0x7f8a2415f9d0 2024-03-06T07:28:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.324+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a24140020 con 0x7f8a2415f9d0 2024-03-06T07:28:35.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.326+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2404f330 con 0x7f8a2415f9d0 2024-03-06T07:28:35.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.326+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a1c03e9d0 con 0x7f8a2415f9d0 2024-03-06T07:28:35.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.327+0000 7f8a1b400640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a0006b4d0 0x7f8a0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.327+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f8a1c0ad070 con 0x7f8a2415f9d0 2024-03-06T07:28:35.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.327+0000 7f8a23400640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a0006b4d0 0x7f8a0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.328+0000 7f8a23400640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a0006b4d0 0x7f8a0006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007fd0 tx=0x7f8a14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.331+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a1c02ce20 con 0x7f8a2415f9d0 2024-03-06T07:28:35.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.526+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a24072890 con 0x7f8a2415f9d0 2024-03-06T07:28:35.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.528+0000 7f8a1b400640 1 -- 172.21.15.97:0/479718828 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16983 (secure 0 0 0) 0x7f8a1c07bd80 con 0x7f8a2415f9d0 2024-03-06T07:28:35.530 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:35.530 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":24,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:28.349664+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a0006b4d0 msgr2=0x7f8a0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a0006b4d0 0x7f8a0006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007fd0 tx=0x7f8a14009040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 msgr2=0x7f8a240ae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 0x7f8a240ae2c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c02c4d0 tx=0x7f8a1c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 shutdown_connections 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a0006b4d0 0x7f8a0006d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a2415f9d0 0x7f8a240ae2c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a24144c00 0x7f8a2413e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 --2- 172.21.15.97:0/479718828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a24070b10 0x7f8a2413b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.530+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 >> 172.21.15.97:0/479718828 conn(0x7f8a240b1730 msgr2=0x7f8a240ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.531+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 shutdown_connections 2024-03-06T07:28:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.531+0000 7f8a2a000640 1 -- 172.21.15.97:0/479718828 wait complete. 2024-03-06T07:28:35.557 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:39.111579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:40.559291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:35.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.756+0000 7faa65200640 1 Processor -- start 2024-03-06T07:28:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.756+0000 7faa65200640 1 -- start start 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.757+0000 7faa65200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e000 0x7faa6006f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.757+0000 7faa65200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa6015e3e0 0x7faa600d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.757+0000 7faa65200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa600d1260 con 0x7faa6006f7e0 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.757+0000 7faa65200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa600d1020 con 0x7faa6015e3e0 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.757+0000 7faa65200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6006c0d0 con 0x7faa6015e000 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa6015e3e0 0x7faa600d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa6015e3e0 0x7faa600d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55108/0 (socket says 172.21.15.97:55108) 2024-03-06T07:28:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 -- 172.21.15.97:0/1391806086 learned_addr learned my addr 172.21.15.97:0/1391806086 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa5de00640 1 --2- 172.21.15.97:0/1391806086 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e000 0x7faa6006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 -- 172.21.15.97:0/1391806086 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e000 msgr2=0x7faa6006f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 --2- 172.21.15.97:0/1391806086 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e000 0x7faa6006f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 -- 172.21.15.97:0/1391806086 >> v1:172.21.15.97:6789/0 conn(0x7faa6006f7e0 legacy=0x7faa600ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa57e00640 1 -- 172.21.15.97:0/1391806086 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6006c320 con 0x7faa6015e3e0 2024-03-06T07:28:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.758+0000 7faa5d400640 1 -- 172.21.15.97:0/1391806086 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762686752 0 0) 0x7faa600d1260 con 0x7faa6006f7e0 2024-03-06T07:28:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.759+0000 7faa57e00640 1 --2- 172.21.15.97:0/1391806086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6015e3e0 0x7faa600d0ae0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa48009870 tx=0x7faa4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4226f60e0f2d9480 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa5d400640 1 -- 172.21.15.97:0/1391806086 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa48034070 con 0x7faa6015e3e0 2024-03-06T07:28:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa5d400640 1 -- 172.21.15.97:0/1391806086 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa48030840 con 0x7faa6015e3e0 2024-03-06T07:28:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa5d400640 1 -- 172.21.15.97:0/1391806086 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa48030c10 con 0x7faa6015e3e0 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa65200640 1 -- 172.21.15.97:0/1391806086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6015e3e0 msgr2=0x7faa600d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa65200640 1 --2- 172.21.15.97:0/1391806086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6015e3e0 0x7faa600d0ae0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa48009870 tx=0x7faa4802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa65200640 1 -- 172.21.15.97:0/1391806086 shutdown_connections 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa65200640 1 --2- 172.21.15.97:0/1391806086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6015e3e0 0x7faa600d0ae0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa65200640 1 --2- 172.21.15.97:0/1391806086 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e000 0x7faa6006f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.760+0000 7faa65200640 1 -- 172.21.15.97:0/1391806086 >> 172.21.15.97:0/1391806086 conn(0x7faa600b3800 msgr2=0x7faa600b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.761+0000 7faa65200640 1 -- 172.21.15.97:0/1391806086 shutdown_connections 2024-03-06T07:28:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.761+0000 7faa65200640 1 -- 172.21.15.97:0/1391806086 wait complete. 2024-03-06T07:28:35.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.762+0000 7faa65200640 1 Processor -- start 2024-03-06T07:28:35.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.762+0000 7faa65200640 1 -- start start 2024-03-06T07:28:35.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa65200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 0x7faa60157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa65200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa6015e000 0x7faa6015f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa65200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e3e0 0x7faa600c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa65200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6006fdf0 con 0x7faa6006f7e0 2024-03-06T07:28:35.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa65200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6006fc70 con 0x7faa6015e000 2024-03-06T07:28:35.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa65200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6006ff70 con 0x7faa6015e3e0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa5de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 0x7faa60157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa5de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 0x7faa60157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55114/0 (socket says 172.21.15.97:55114) 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.763+0000 7faa5de00640 1 -- 172.21.15.97:0/2976690528 learned_addr learned my addr 172.21.15.97:0/2976690528 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5e800640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e3e0 0x7faa600c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa57e00640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa6015e000 0x7faa6015f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5de00640 1 -- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e3e0 msgr2=0x7faa600c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5de00640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e3e0 0x7faa600c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5de00640 1 -- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa6015e000 msgr2=0x7faa6015f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5de00640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa6015e000 0x7faa6015f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5de00640 1 -- 172.21.15.97:0/2976690528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600c5980 con 0x7faa6006f7e0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.764+0000 7faa5de00640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 0x7faa60157470 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa5000d830 tx=0x7faa5000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.765+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5000b6c0 con 0x7faa6006f7e0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.765+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa50019070 con 0x7faa6006f7e0 2024-03-06T07:28:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.765+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5000ba80 con 0x7faa6006f7e0 2024-03-06T07:28:35.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.765+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa600c5c60 con 0x7faa6006f7e0 2024-03-06T07:28:35.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.765+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa600c6220 con 0x7faa6006f7e0 2024-03-06T07:28:35.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.766+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6015dbb0 con 0x7faa6006f7e0 2024-03-06T07:28:35.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.767+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa5001e460 con 0x7faa6006f7e0 2024-03-06T07:28:35.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.768+0000 7faa56000640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa3c06b4d0 0x7faa3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.768+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7faa5008a8f0 con 0x7faa6006f7e0 2024-03-06T07:28:35.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.768+0000 7faa57e00640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa3c06b4d0 0x7faa3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.769+0000 7faa57e00640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa3c06b4d0 0x7faa3c06d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faa48036000 tx=0x7faa4803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.772+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa6015dbb0 con 0x7faa6006f7e0 2024-03-06T07:28:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.967+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa600cfbe0 con 0x7faa6006f7e0 2024-03-06T07:28:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.968+0000 7faa56000640 1 -- 172.21.15.97:0/2976690528 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa600cfbe0 con 0x7faa6006f7e0 2024-03-06T07:28:35.970 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.970+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa3c06b4d0 msgr2=0x7faa3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.970+0000 7faa65200640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa3c06b4d0 0x7faa3c06d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faa48036000 tx=0x7faa4803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.970+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 msgr2=0x7faa60157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 0x7faa60157470 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa5000d830 tx=0x7faa5000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 shutdown_connections 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa3c06b4d0 0x7faa3c06d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa6015e3e0 0x7faa600c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa6015e000 0x7faa6015f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 --2- 172.21.15.97:0/2976690528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa6006f7e0 0x7faa60157470 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 >> 172.21.15.97:0/2976690528 conn(0x7faa600b3800 msgr2=0x7faa600aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 shutdown_connections 2024-03-06T07:28:35.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:35.971+0000 7faa65200640 1 -- 172.21.15.97:0/2976690528 wait complete. 2024-03-06T07:28:35.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:36.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.183+0000 7fd46e200640 1 Processor -- start 2024-03-06T07:28:36.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.183+0000 7fd46e200640 1 -- start start 2024-03-06T07:28:36.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd46e200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd4680d0900 0x7fd4680ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd46e200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd4680ceac0 0x7fd46806edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd46e200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4680ceed0 con 0x7fd46815bfe0 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd46e200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4680cf8a0 con 0x7fd4680d0900 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd46e200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4680cfb40 con 0x7fd4680ceac0 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd466a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd4680d0900 0x7fd4680ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd466a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd4680d0900 0x7fd4680ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55128/0 (socket says 172.21.15.97:55128) 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd466a00640 1 -- 172.21.15.97:0/13754448 learned_addr learned my addr 172.21.15.97:0/13754448 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.186+0000 7fd467e00640 1 --2- 172.21.15.97:0/13754448 >> v2:172.21.15.135:3300/0 conn(0x7fd4680ceac0 0x7fd46806edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.187+0000 7fd466a00640 1 -- 172.21.15.97:0/13754448 >> v2:172.21.15.135:3300/0 conn(0x7fd4680ceac0 msgr2=0x7fd46806edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.187+0000 7fd466a00640 1 --2- 172.21.15.97:0/13754448 >> v2:172.21.15.135:3300/0 conn(0x7fd4680ceac0 0x7fd46806edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.187+0000 7fd466a00640 1 -- 172.21.15.97:0/13754448 >> v1:172.21.15.135:6789/0 conn(0x7fd46815bfe0 legacy=0x7fd4680cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.187+0000 7fd466a00640 1 -- 172.21.15.97:0/13754448 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4680cfe20 con 0x7fd4680d0900 2024-03-06T07:28:36.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.187+0000 7fd467e00640 1 --2- 172.21.15.97:0/13754448 >> v2:172.21.15.135:3300/0 conn(0x7fd4680ceac0 0x7fd46806edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.188+0000 7fd466a00640 1 --2- 172.21.15.97:0/13754448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680d0900 0x7fd4680ce580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd4680c5710 tx=0x7fd45402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e34fd39b8e6f5f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.188+0000 7fd466000640 1 -- 172.21.15.97:0/13754448 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454034070 con 0x7fd4680d0900 2024-03-06T07:28:36.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.188+0000 7fd466000640 1 -- 172.21.15.97:0/13754448 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd454030860 con 0x7fd4680d0900 2024-03-06T07:28:36.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.188+0000 7fd466000640 1 -- 172.21.15.97:0/13754448 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454030c30 con 0x7fd4680d0900 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 -- 172.21.15.97:0/13754448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680d0900 msgr2=0x7fd4680ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 --2- 172.21.15.97:0/13754448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680d0900 0x7fd4680ce580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd4680c5710 tx=0x7fd45402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 -- 172.21.15.97:0/13754448 shutdown_connections 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 --2- 172.21.15.97:0/13754448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680d0900 0x7fd4680ce580 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 --2- 172.21.15.97:0/13754448 >> v2:172.21.15.135:3300/0 conn(0x7fd4680ceac0 0x7fd46806edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 -- 172.21.15.97:0/13754448 >> 172.21.15.97:0/13754448 conn(0x7fd4680b3800 msgr2=0x7fd4680b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 -- 172.21.15.97:0/13754448 shutdown_connections 2024-03-06T07:28:36.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.189+0000 7fd46e200640 1 -- 172.21.15.97:0/13754448 wait complete. 2024-03-06T07:28:36.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.192+0000 7fd46e200640 1 Processor -- start 2024-03-06T07:28:36.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.192+0000 7fd46e200640 1 -- start start 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.193+0000 7fd46e200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 0x7fd4680b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd46e200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4680d0900 0x7fd4680ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd46e200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd46815bfe0 0x7fd4680cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd46e200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680d0d50 con 0x7fd4680ceac0 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd46e200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680d03e0 con 0x7fd4680d0900 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd46e200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680b4540 con 0x7fd46815bfe0 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 0x7fd4680b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 0x7fd4680b74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55144/0 (socket says 172.21.15.97:55144) 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 -- 172.21.15.97:0/2817942928 learned_addr learned my addr 172.21.15.97:0/2817942928 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 -- 172.21.15.97:0/2817942928 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd46815bfe0 msgr2=0x7fd4680cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd46815bfe0 0x7fd4680cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 -- 172.21.15.97:0/2817942928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4680d0900 msgr2=0x7fd4680ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4680d0900 0x7fd4680ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.194+0000 7fd467400640 1 -- 172.21.15.97:0/2817942928 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4680b5850 con 0x7fd4680ceac0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.195+0000 7fd467400640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 0x7fd4680b74a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd45800b3f0 tx=0x7fd45800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.195+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458015020 con 0x7fd4680ceac0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.195+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd458011830 con 0x7fd4680ceac0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.195+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458011c20 con 0x7fd4680ceac0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.195+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4680b5b30 con 0x7fd4680ceac0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.195+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4680b60f0 con 0x7fd4680ceac0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.196+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46804f330 con 0x7fd4680ceac0 2024-03-06T07:28:36.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.197+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd458018020 con 0x7fd4680ceac0 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.198+0000 7fd45f400640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd44806b4d0 0x7fd44806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.198+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fd458089fb0 con 0x7fd4680ceac0 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.198+0000 7fd466a00640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd44806b4d0 0x7fd44806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.199+0000 7fd466a00640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd44806b4d0 0x7fd44806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd454036000 tx=0x7fd45403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.203+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd45801f030 con 0x7fd4680ceac0 2024-03-06T07:28:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.400+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd4680cc1b0 con 0x7fd4680ceac0 2024-03-06T07:28:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.401+0000 7fd45f400640 1 -- 172.21.15.97:0/2817942928 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd458058cc0 con 0x7fd4680ceac0 2024-03-06T07:28:36.403 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:28:36.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd44806b4d0 msgr2=0x7fd44806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd44806b4d0 0x7fd44806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd454036000 tx=0x7fd45403b040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 msgr2=0x7fd4680b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 0x7fd4680b74a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd45800b3f0 tx=0x7fd45800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 shutdown_connections 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd44806b4d0 0x7fd44806d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd46815bfe0 0x7fd4680cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4680d0900 0x7fd4680ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 --2- 172.21.15.97:0/2817942928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4680ceac0 0x7fd4680b74a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.403+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 >> 172.21.15.97:0/2817942928 conn(0x7fd4680b3800 msgr2=0x7fd46815b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.404+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 shutdown_connections 2024-03-06T07:28:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.404+0000 7fd46e200640 1 -- 172.21.15.97:0/2817942928 wait complete. 2024-03-06T07:28:36.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:36.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.643+0000 7ffae5800640 1 Processor -- start 2024-03-06T07:28:36.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.643+0000 7ffae5800640 1 -- start start 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffae5800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 0x7ffae00cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffae5800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffae00cfc10 0x7ffae00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffae5800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae015b130 con 0x7ffae0158570 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffae5800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae00d0080 con 0x7ffae00cfc10 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffae5800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00af170 con 0x7ffae0158950 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffadfe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ffae0158570 0x7ffae00ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48432/0 (socket says 172.21.15.97:48432) 2024-03-06T07:28:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffadfe00640 1 -- 172.21.15.97:0/666700189 learned_addr learned my addr 172.21.15.97:0/666700189 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:36.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffad7400640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 0x7ffae00cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffae4e00640 1 --2- 172.21.15.97:0/666700189 >> v2:172.21.15.135:3300/0 conn(0x7ffae00cfc10 0x7ffae00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.646+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241668439 0 0) 0x7ffae015b130 con 0x7ffae0158570 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac4003520 con 0x7ffae0158570 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1327722275 0 0) 0x7ffac4003520 con 0x7ffae0158570 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 msgr2=0x7ffae00cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 0x7ffae00cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 >> v2:172.21.15.135:3300/0 conn(0x7ffae00cfc10 msgr2=0x7ffae00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 --2- 172.21.15.97:0/666700189 >> v2:172.21.15.135:3300/0 conn(0x7ffae00cfc10 0x7ffae00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae015b3f0 con 0x7ffae0158570 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ffad00019f0 con 0x7ffae0158570 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffae4e00640 1 --2- 172.21.15.97:0/666700189 >> v2:172.21.15.135:3300/0 conn(0x7ffae00cfc10 0x7ffae00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 >> v1:172.21.15.97:6789/0 conn(0x7ffae0158570 legacy=0x7ffae00ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 0x7ffac40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00d0080 con 0x7ffac4004d40 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00af170 con 0x7ffac4003f30 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400dcd0 con 0x7ffae0158950 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.647+0000 7ffae4e00640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffad7400640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffadfe00640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 0x7ffac40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffae4e00640 1 -- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 msgr2=0x7ffac40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffae4e00640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffae0158950 0x7ffac40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffae4e00640 1 -- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffac4003f30 msgr2=0x7ffac4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffae4e00640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.648+0000 7ffae4e00640 1 -- 172.21.15.97:0/666700189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4003520 con 0x7ffac4004d40 2024-03-06T07:28:36.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.649+0000 7ffae4e00640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffad80098d0 tx=0x7ffac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c43f1ac108de10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.649+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad800f040 con 0x7ffac4004d40 2024-03-06T07:28:36.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.649+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad8004230 con 0x7ffac4004d40 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.649+0000 7ffadf400640 1 -- 172.21.15.97:0/666700189 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8003610 con 0x7ffac4004d40 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae5800640 1 -- 172.21.15.97:0/666700189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac4004d40 msgr2=0x7ffac40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae5800640 1 --2- 172.21.15.97:0/666700189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffad80098d0 tx=0x7ffac4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae4e00640 1 -- 172.21.15.97:0/666700189 reap_dead start 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae5800640 1 -- 172.21.15.97:0/666700189 shutdown_connections 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae5800640 1 -- 172.21.15.97:0/666700189 >> 172.21.15.97:0/666700189 conn(0x7ffae006afc0 msgr2=0x7ffae006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae5800640 1 -- 172.21.15.97:0/666700189 shutdown_connections 2024-03-06T07:28:36.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.650+0000 7ffae5800640 1 -- 172.21.15.97:0/666700189 wait complete. 2024-03-06T07:28:36.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.652+0000 7ffae5800640 1 Processor -- start 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.652+0000 7ffae5800640 1 -- start start 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae5800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac4004d40 0x7ffae01483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae5800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae0159740 0x7ffae00ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 0x7ffae00addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae5800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00b4950 con 0x7ffae0159ba0 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae5800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae015b9b0 con 0x7ffae0159740 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae5800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00aedc0 con 0x7ffac4004d40 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 0x7ffae00addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 0x7ffae00addd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55150/0 (socket says 172.21.15.97:55150) 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffae4e00640 1 -- 172.21.15.97:0/3620720955 learned_addr learned my addr 172.21.15.97:0/3620720955 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.653+0000 7ffadfe00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac4004d40 0x7ffae01483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffad7400640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae0159740 0x7ffae00ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae4e00640 1 -- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac4004d40 msgr2=0x7ffae01483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae4e00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac4004d40 0x7ffae01483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae4e00640 1 -- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae0159740 msgr2=0x7ffae00ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae4e00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae0159740 0x7ffae00ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae4e00640 1 -- 172.21.15.97:0/3620720955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0153c10 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae4e00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 0x7ffae00addd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffad80099f0 tx=0x7ffad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8047070 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad800f000 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8042640 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.654+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae0153ef0 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.655+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae01544b0 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.656+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae00cbc10 con 0x7ffae0159ba0 2024-03-06T07:28:36.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.656+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffad80373f0 con 0x7ffae0159ba0 2024-03-06T07:28:36.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.657+0000 7ffadcc00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffabc06b4d0 0x7ffabc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.657+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7ffad80ab6f0 con 0x7ffae0159ba0 2024-03-06T07:28:36.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.658+0000 7ffadfe00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffabc06b4d0 0x7ffabc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.658+0000 7ffadfe00640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffabc06b4d0 0x7ffabc06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffad0005e30 tx=0x7ffad0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.662+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffae00cbc10 con 0x7ffae0159ba0 2024-03-06T07:28:36.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:36.834 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:36.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.868+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffae015b130 con 0x7ffae0159ba0 2024-03-06T07:28:36.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.869+0000 7ffadcc00640 1 -- 172.21.15.97:0/3620720955 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffad8033070 con 0x7ffae0159ba0 2024-03-06T07:28:36.871 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:28:36.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffabc06b4d0 msgr2=0x7ffabc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffabc06b4d0 0x7ffabc06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffad0005e30 tx=0x7ffad0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 msgr2=0x7ffae00addd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 0x7ffae00addd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffad80099f0 tx=0x7ffad8004730 comp rx=0 tx=0).stop 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 shutdown_connections 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffabc06b4d0 0x7ffabc06d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffae0159ba0 0x7ffae00addd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae0159740 0x7ffae00ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 --2- 172.21.15.97:0/3620720955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac4004d40 0x7ffae01483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 >> 172.21.15.97:0/3620720955 conn(0x7ffae006afc0 msgr2=0x7ffae00b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 shutdown_connections 2024-03-06T07:28:36.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:36.872+0000 7ffae5800640 1 -- 172.21.15.97:0/3620720955 wait complete. 2024-03-06T07:28:36.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:37.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.134+0000 7fe806600640 1 Processor -- start 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.134+0000 7fe806600640 1 -- start start 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe806600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe80015f930 0x7fe800066f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe806600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe80006ed70 0x7fe80015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe806600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80015c2f0 0x7fe80015c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe806600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80006f220 con 0x7fe80015f930 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe806600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8000ccd90 con 0x7fe80006ed70 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe806600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000afc00 con 0x7fe80015c2f0 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe7ff400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe80015f930 0x7fe800066f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe7ff400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe80015f930 0x7fe800066f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55152/0 (socket says 172.21.15.97:55152) 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.136+0000 7fe7ff400640 1 -- 172.21.15.97:0/1229198797 learned_addr learned my addr 172.21.15.97:0/1229198797 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7f6a00640 1 --2- 172.21.15.97:0/1229198797 >> v2:172.21.15.135:3300/0 conn(0x7fe80006ed70 0x7fe80015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7ffe00640 1 --2- 172.21.15.97:0/1229198797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80015c2f0 0x7fe80015c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7ff400640 1 -- 172.21.15.97:0/1229198797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80015c2f0 msgr2=0x7fe80015c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1229198797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80015c2f0 0x7fe80015c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7ff400640 1 -- 172.21.15.97:0/1229198797 >> v2:172.21.15.135:3300/0 conn(0x7fe80006ed70 msgr2=0x7fe80015fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1229198797 >> v2:172.21.15.135:3300/0 conn(0x7fe80006ed70 0x7fe80015fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.137+0000 7fe7ff400640 1 -- 172.21.15.97:0/1229198797 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80006e5b0 con 0x7fe80015f930 2024-03-06T07:28:37.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.138+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1229198797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015f930 0x7fe800066f10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8000679d0 tx=0x7fe7f002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d4993fd9e2d434 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.138+0000 7fe7fea00640 1 -- 172.21.15.97:0/1229198797 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f002c9e0 con 0x7fe80015f930 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.138+0000 7fe7fea00640 1 -- 172.21.15.97:0/1229198797 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f002cb80 con 0x7fe80015f930 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.138+0000 7fe7fea00640 1 -- 172.21.15.97:0/1229198797 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00343f0 con 0x7fe80015f930 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 -- 172.21.15.97:0/1229198797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015f930 msgr2=0x7fe800066f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 --2- 172.21.15.97:0/1229198797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015f930 0x7fe800066f10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8000679d0 tx=0x7fe7f002be40 comp rx=0 tx=0).stop 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 -- 172.21.15.97:0/1229198797 shutdown_connections 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 --2- 172.21.15.97:0/1229198797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80015c2f0 0x7fe80015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 --2- 172.21.15.97:0/1229198797 >> v2:172.21.15.135:3300/0 conn(0x7fe80006ed70 0x7fe80015fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 --2- 172.21.15.97:0/1229198797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015f930 0x7fe800066f10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 -- 172.21.15.97:0/1229198797 >> 172.21.15.97:0/1229198797 conn(0x7fe8000d0340 msgr2=0x7fe8000d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 -- 172.21.15.97:0/1229198797 shutdown_connections 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.139+0000 7fe806600640 1 -- 172.21.15.97:0/1229198797 wait complete. 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.141+0000 7fe806600640 1 Processor -- start 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.141+0000 7fe806600640 1 -- start start 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe806600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80006ed70 0x7fe8001412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe806600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 0x7fe800141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe806600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015f930 0x7fe8000b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe806600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000cd2c0 con 0x7fe80015c2f0 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe806600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000ccfe0 con 0x7fe80015f930 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe806600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80006ddb0 con 0x7fe80006ed70 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.142+0000 7fe7f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 0x7fe800141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 0x7fe800141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55162/0 (socket says 172.21.15.97:55162) 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 -- 172.21.15.97:0/1823880363 learned_addr learned my addr 172.21.15.97:0/1823880363 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80006ed70 0x7fe8001412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 -- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80006ed70 msgr2=0x7fe8001412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80006ed70 0x7fe8001412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 -- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015f930 msgr2=0x7fe8000b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015f930 0x7fe8000b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 -- 172.21.15.97:0/1823880363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8001424e0 con 0x7fe80015c2f0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80006ed70 0x7fe8001412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.143+0000 7fe7f6a00640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 0x7fe800141a90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d800 tx=0x7fe7e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.144+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8018020 con 0x7fe80015c2f0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.144+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e80096a0 con 0x7fe80015c2f0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.144+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8009a70 con 0x7fe80015c2f0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.144+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8001427c0 con 0x7fe80015c2f0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.144+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe800142d80 con 0x7fe80015c2f0 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.145+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe800074ae0 con 0x7fe80015c2f0 2024-03-06T07:28:37.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.147+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7e801b020 con 0x7fe80015c2f0 2024-03-06T07:28:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.148+0000 7fe7f7e00640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.148+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fe7e808a590 con 0x7fe80015c2f0 2024-03-06T07:28:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.148+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.149+0000 7fe7ff400640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0035000 tx=0x7fe7f0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.154+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7e8022030 con 0x7fe80015c2f0 2024-03-06T07:28:37.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.386+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8000d0fa0 con 0x7fe80015c2f0 2024-03-06T07:28:37.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.388+0000 7fe7f7e00640 1 -- 172.21.15.97:0/1823880363 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fe7e80592a0 con 0x7fe80015c2f0 2024-03-06T07:28:37.390 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:37.390 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:37.390 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.390+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b4d0 msgr2=0x7fe7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.390+0000 7fe806600640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0035000 tx=0x7fe7f0030000 comp rx=0 tx=0).stop 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.390+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 msgr2=0x7fe800141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.390+0000 7fe806600640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 0x7fe800141a90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d800 tx=0x7fe7e800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 shutdown_connections 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015f930 0x7fe8000b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80015c2f0 0x7fe800141a90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 --2- 172.21.15.97:0/1823880363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80006ed70 0x7fe8001412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 >> 172.21.15.97:0/1823880363 conn(0x7fe8000d0340 msgr2=0x7fe8000cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 shutdown_connections 2024-03-06T07:28:37.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.391+0000 7fe806600640 1 -- 172.21.15.97:0/1823880363 wait complete. 2024-03-06T07:28:37.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:37.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.620+0000 7fac4fe00640 1 Processor -- start 2024-03-06T07:28:37.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.620+0000 7fac4fe00640 1 -- start start 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.621+0000 7fac4fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac4815e160 0x7fac48162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4fe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac48162b60 con 0x7fac480af820 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4fe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4806f1f0 con 0x7fac480b3ac0 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4fe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4806b0f0 con 0x7fac4815e160 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac4815e160 0x7fac48162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac4815e160 0x7fac48162620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55176/0 (socket says 172.21.15.97:55176) 2024-03-06T07:28:37.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4d400640 1 -- 172.21.15.97:0/4142595675 learned_addr learned my addr 172.21.15.97:0/4142595675 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:37.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4d400640 1 -- 172.21.15.97:0/4142595675 >> v1:172.21.15.135:6789/0 conn(0x7fac480b3ac0 legacy=0x7fac4806b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4d400640 1 -- 172.21.15.97:0/4142595675 >> v1:172.21.15.97:6789/0 conn(0x7fac480af820 legacy=0x7fac480b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.622+0000 7fac4d400640 1 -- 172.21.15.97:0/4142595675 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480afce0 con 0x7fac4815e160 2024-03-06T07:28:37.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.624+0000 7fac4d400640 1 --2- 172.21.15.97:0/4142595675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac4815e160 0x7fac48162620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009870 tx=0x7fac3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceca7f0f65c51c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.624+0000 7fac47e00640 1 -- 172.21.15.97:0/4142595675 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c035020 con 0x7fac4815e160 2024-03-06T07:28:37.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.624+0000 7fac47e00640 1 -- 172.21.15.97:0/4142595675 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c031870 con 0x7fac4815e160 2024-03-06T07:28:37.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.624+0000 7fac47e00640 1 -- 172.21.15.97:0/4142595675 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031c40 con 0x7fac4815e160 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.624+0000 7fac4fe00640 1 -- 172.21.15.97:0/4142595675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac4815e160 msgr2=0x7fac48162620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.624+0000 7fac4fe00640 1 --2- 172.21.15.97:0/4142595675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac4815e160 0x7fac48162620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009870 tx=0x7fac3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.625+0000 7fac4fe00640 1 -- 172.21.15.97:0/4142595675 shutdown_connections 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.625+0000 7fac4fe00640 1 --2- 172.21.15.97:0/4142595675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac4815e160 0x7fac48162620 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.625+0000 7fac4fe00640 1 -- 172.21.15.97:0/4142595675 >> 172.21.15.97:0/4142595675 conn(0x7fac480d0340 msgr2=0x7fac480d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.625+0000 7fac4fe00640 1 -- 172.21.15.97:0/4142595675 shutdown_connections 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.625+0000 7fac4fe00640 1 -- 172.21.15.97:0/4142595675 wait complete. 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.626+0000 7fac4fe00640 1 Processor -- start 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.627+0000 7fac4fe00640 1 -- start start 2024-03-06T07:28:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.627+0000 7fac4fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480af820 0x7fac481569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 0x7fac48158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac4815e160 0x7fac48153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806ef10 con 0x7fac480b3ac0 2024-03-06T07:28:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480b5620 con 0x7fac4815e160 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806bc90 con 0x7fac480af820 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 0x7fac48158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 0x7fac48158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55192/0 (socket says 172.21.15.97:55192) 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 -- 172.21.15.97:0/3575264552 learned_addr learned my addr 172.21.15.97:0/3575264552 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4d400640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480af820 0x7fac481569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4de00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac4815e160 0x7fac48153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 -- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480af820 msgr2=0x7fac481569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480af820 0x7fac481569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 -- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac4815e160 msgr2=0x7fac48153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac4815e160 0x7fac48153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4ca00640 1 -- 172.21.15.97:0/3575264552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480cd940 con 0x7fac480b3ac0 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.628+0000 7fac4de00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac4815e160 0x7fac48153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.629+0000 7fac4ca00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 0x7fac48158340 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fac3400c440 tx=0x7fac34001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.629+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac340052d0 con 0x7fac480b3ac0 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.629+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac34005470 con 0x7fac480b3ac0 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.629+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac480cdc20 con 0x7fac480b3ac0 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.629+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34007260 con 0x7fac480b3ac0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.629+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac48151fd0 con 0x7fac480b3ac0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.630+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4804f330 con 0x7fac480b3ac0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.631+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac34007400 con 0x7fac480b3ac0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.632+0000 7fac45600640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.632+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fac340898b0 con 0x7fac480b3ac0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.632+0000 7fac4d400640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.633+0000 7fac4d400640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036000 tx=0x7fac3c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.636+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac48162b60 con 0x7fac480b3ac0 2024-03-06T07:28:37.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.832+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac480b1d60 con 0x7fac480b3ac0 2024-03-06T07:28:37.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.834+0000 7fac45600640 1 -- 172.21.15.97:0/3575264552 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16983 (secure 0 0 0) 0x7fac340585c0 con 0x7fac480b3ac0 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":24,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:28.349664+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 msgr2=0x7fac2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036000 tx=0x7fac3c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 msgr2=0x7fac48158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 0x7fac48158340 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fac3400c440 tx=0x7fac34001e70 comp rx=0 tx=0).stop 2024-03-06T07:28:37.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 shutdown_connections 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac4815e160 0x7fac48153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac480b3ac0 0x7fac48158340 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 --2- 172.21.15.97:0/3575264552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480af820 0x7fac481569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 >> 172.21.15.97:0/3575264552 conn(0x7fac480d0340 msgr2=0x7fac480ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 shutdown_connections 2024-03-06T07:28:37.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:37.837+0000 7fac4fe00640 1 -- 172.21.15.97:0/3575264552 wait complete. 2024-03-06T07:28:37.857 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:37.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:38.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.059+0000 7f125fe00640 1 Processor -- start 2024-03-06T07:28:38.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.059+0000 7f125fe00640 1 -- start start 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.060+0000 7f125fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1260161580 0x7f12600d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.060+0000 7f125fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.060+0000 7f125fe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1260161960 con 0x7f12600d28a0 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.060+0000 7f125fe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600cf6f0 con 0x7f1260161580 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.060+0000 7f125fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600cf9d0 con 0x7f12600d1e90 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1260161580 0x7f12600d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1260161580 0x7f12600d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55194/0 (socket says 172.21.15.97:55194) 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 -- 172.21.15.97:0/3047955116 learned_addr learned my addr 172.21.15.97:0/3047955116 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:38.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125e000640 1 --2- 172.21.15.97:0/3047955116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 -- 172.21.15.97:0/3047955116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 msgr2=0x7f12600d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 --2- 172.21.15.97:0/3047955116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 -- 172.21.15.97:0/3047955116 >> v1:172.21.15.135:6789/0 conn(0x7f12600d28a0 legacy=0x7f12600d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.061+0000 7f125ea00640 1 -- 172.21.15.97:0/3047955116 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600cfcb0 con 0x7f1260161580 2024-03-06T07:28:38.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.062+0000 7f125ea00640 1 --2- 172.21.15.97:0/3047955116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f12600d1950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1250009870 tx=0x7f125002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3171d04f887f9b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125d600640 1 -- 172.21.15.97:0/3047955116 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250035020 con 0x7f1260161580 2024-03-06T07:28:38.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125d600640 1 -- 172.21.15.97:0/3047955116 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250031640 con 0x7f1260161580 2024-03-06T07:28:38.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125d600640 1 -- 172.21.15.97:0/3047955116 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125002d040 con 0x7f1260161580 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125fe00640 1 -- 172.21.15.97:0/3047955116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 msgr2=0x7f12600d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125fe00640 1 --2- 172.21.15.97:0/3047955116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f12600d1950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1250009870 tx=0x7f125002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125fe00640 1 -- 172.21.15.97:0/3047955116 shutdown_connections 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125fe00640 1 --2- 172.21.15.97:0/3047955116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125fe00640 1 --2- 172.21.15.97:0/3047955116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f12600d1950 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.063+0000 7f125fe00640 1 -- 172.21.15.97:0/3047955116 >> 172.21.15.97:0/3047955116 conn(0x7f12600d3d80 msgr2=0x7f12600d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.064+0000 7f125fe00640 1 -- 172.21.15.97:0/3047955116 shutdown_connections 2024-03-06T07:28:38.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.064+0000 7f125fe00640 1 -- 172.21.15.97:0/3047955116 wait complete. 2024-03-06T07:28:38.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.065+0000 7f125fe00640 1 Processor -- start 2024-03-06T07:28:38.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.065+0000 7f125fe00640 1 -- start start 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.066+0000 7f125fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.066+0000 7f125fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12600d28a0 0x7f12601f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.066+0000 7f125fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f1260153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.066+0000 7f125fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600cf3a0 con 0x7f1260161580 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.066+0000 7f125fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600b39a0 con 0x7f12600d28a0 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.066+0000 7f125fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600b4ed0 con 0x7f12600d1e90 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f1260153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f1260153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55196/0 (socket says 172.21.15.97:55196) 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 -- 172.21.15.97:0/1967308091 learned_addr learned my addr 172.21.15.97:0/1967308091 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125ea00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125e000640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12600d28a0 0x7f12601f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 -- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 msgr2=0x7f12600c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 -- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12600d28a0 msgr2=0x7f12601f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12600d28a0 0x7f12601f7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 -- 172.21.15.97:0/1967308091 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600b9810 con 0x7f1260161580 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.067+0000 7f125f400640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f1260153c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f125800c410 tx=0x7f125800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.068+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258011790 con 0x7f1260161580 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.068+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1258017070 con 0x7f1260161580 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.068+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258011bb0 con 0x7f1260161580 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.068+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12600b9af0 con 0x7f1260161580 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.068+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12600cc180 con 0x7f1260161580 2024-03-06T07:28:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.069+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126004f330 con 0x7f1260161580 2024-03-06T07:28:38.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.070+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f125801c460 con 0x7f1260161580 2024-03-06T07:28:38.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.072+0000 7f1256a00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f123c06b4d0 0x7f123c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.072+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f1258089e30 con 0x7f1260161580 2024-03-06T07:28:38.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.072+0000 7f125ea00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f123c06b4d0 0x7f123c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.073+0000 7f125ea00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f123c06b4d0 0x7f123c06d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f125003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.076+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1258021020 con 0x7f1260161580 2024-03-06T07:28:38.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.323+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12600d3b60 con 0x7f1260161580 2024-03-06T07:28:38.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.323+0000 7f1256a00640 1 -- 172.21.15.97:0/1967308091 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f1258058c70 con 0x7f1260161580 2024-03-06T07:28:38.326 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:38.326 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:38.326 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f123c06b4d0 msgr2=0x7f123c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f123c06b4d0 0x7f123c06d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f125003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 msgr2=0x7f1260153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f1260153c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f125800c410 tx=0x7f125800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 shutdown_connections 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f123c06b4d0 0x7f123c06d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.326+0000 7f125fe00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1260161580 0x7f1260153c80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.327+0000 7f125fe00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12600d28a0 0x7f12601f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.327+0000 7f125fe00640 1 --2- 172.21.15.97:0/1967308091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12600d1e90 0x7f12600c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.327+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 >> 172.21.15.97:0/1967308091 conn(0x7f12600d3d80 msgr2=0x7f126006d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.327+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 shutdown_connections 2024-03-06T07:28:38.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.327+0000 7f125fe00640 1 -- 172.21.15.97:0/1967308091 wait complete. 2024-03-06T07:28:38.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:38.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.548+0000 7f0bad400640 1 Processor -- start 2024-03-06T07:28:38.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.549+0000 7f0bad400640 1 -- start start 2024-03-06T07:28:38.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.550+0000 7f0bad400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0ba8144660 0x7f0ba8071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.550+0000 7f0bad400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0ba80cf3e0 0x7f0ba80724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.550+0000 7f0bad400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80cabf0 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.550+0000 7f0bad400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba80ca9b0 con 0x7f0ba80cf3e0 2024-03-06T07:28:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.550+0000 7f0bad400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba80ce900 con 0x7f0ba8144660 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0b9de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0ba80cf3e0 0x7f0ba80724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0ba7200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0ba80c9ec0 0x7f0ba80ca2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48448/0 (socket says 172.21.15.97:48448) 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0ba7200640 1 -- 172.21.15.97:0/4271369133 learned_addr learned my addr 172.21.15.97:0/4271369133 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0ba6800640 1 --2- 172.21.15.97:0/4271369133 >> v2:172.21.15.135:3300/0 conn(0x7f0ba8144660 0x7f0ba8071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0b9de00640 1 -- 172.21.15.97:0/4271369133 >> v2:172.21.15.135:3300/0 conn(0x7f0ba8144660 msgr2=0x7f0ba8071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0b9de00640 1 --2- 172.21.15.97:0/4271369133 >> v2:172.21.15.135:3300/0 conn(0x7f0ba8144660 0x7f0ba8071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0b9de00640 1 -- 172.21.15.97:0/4271369133 >> v1:172.21.15.97:6789/0 conn(0x7f0ba80c9ec0 legacy=0x7f0ba80ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0b9de00640 1 -- 172.21.15.97:0/4271369133 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80cebe0 con 0x7f0ba80cf3e0 2024-03-06T07:28:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0ba5e00640 1 -- 172.21.15.97:0/4271369133 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097349525 0 0) 0x7f0ba80cabf0 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.551+0000 7f0ba6800640 1 --2- 172.21.15.97:0/4271369133 >> v2:172.21.15.135:3300/0 conn(0x7f0ba8144660 0x7f0ba8071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.552+0000 7f0b9de00640 1 --2- 172.21.15.97:0/4271369133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80cf3e0 0x7f0ba80724b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80d1850 tx=0x7f0b9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43590c4109041978 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0ba5e00640 1 -- 172.21.15.97:0/4271369133 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90035020 con 0x7f0ba80cf3e0 2024-03-06T07:28:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0ba5e00640 1 -- 172.21.15.97:0/4271369133 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b90031830 con 0x7f0ba80cf3e0 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0ba5e00640 1 -- 172.21.15.97:0/4271369133 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90031c00 con 0x7f0ba80cf3e0 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0bad400640 1 -- 172.21.15.97:0/4271369133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80cf3e0 msgr2=0x7f0ba80724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0bad400640 1 --2- 172.21.15.97:0/4271369133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80cf3e0 0x7f0ba80724b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80d1850 tx=0x7f0b9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0bad400640 1 -- 172.21.15.97:0/4271369133 shutdown_connections 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0bad400640 1 --2- 172.21.15.97:0/4271369133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80cf3e0 0x7f0ba80724b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0bad400640 1 --2- 172.21.15.97:0/4271369133 >> v2:172.21.15.135:3300/0 conn(0x7f0ba8144660 0x7f0ba8071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.553+0000 7f0bad400640 1 -- 172.21.15.97:0/4271369133 >> 172.21.15.97:0/4271369133 conn(0x7f0ba80d0b90 msgr2=0x7f0ba80d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.554+0000 7f0bad400640 1 -- 172.21.15.97:0/4271369133 shutdown_connections 2024-03-06T07:28:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.554+0000 7f0bad400640 1 -- 172.21.15.97:0/4271369133 wait complete. 2024-03-06T07:28:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.556+0000 7f0bad400640 1 Processor -- start 2024-03-06T07:28:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.556+0000 7f0bad400640 1 -- start start 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0bad400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 0x7f0ba80b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0bad400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ba80cf3e0 0x7f0ba81610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0bad400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ba8144660 0x7f0ba81411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0bad400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80cc0f0 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0bad400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba8144a40 con 0x7f0ba8144660 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0bad400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80cc270 con 0x7f0ba80cf3e0 2024-03-06T07:28:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0ba6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 0x7f0ba80b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0ba6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 0x7f0ba80b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55216/0 (socket says 172.21.15.97:55216) 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.558+0000 7f0ba6800640 1 -- 172.21.15.97:0/1380243990 learned_addr learned my addr 172.21.15.97:0/1380243990 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0b9de00640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ba80cf3e0 0x7f0ba81610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba7200640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ba8144660 0x7f0ba81411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba6800640 1 -- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ba80cf3e0 msgr2=0x7f0ba81610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba6800640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ba80cf3e0 0x7f0ba81610d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba6800640 1 -- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ba8144660 msgr2=0x7f0ba81411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba6800640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ba8144660 0x7f0ba81411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba6800640 1 -- 172.21.15.97:0/1380243990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8141700 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0ba6800640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 0x7f0ba80b6480 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800d7d0 tx=0x7f0b9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98004030 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b980041d0 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba8160310 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.559+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b980096a0 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.560+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba81608d0 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.561+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba804f330 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.562+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b98009840 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.563+0000 7f0b9f400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b8406b4d0 0x7f0b8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.563+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f0b9808a560 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.563+0000 7f0b9de00640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b8406b4d0 0x7f0b8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.564+0000 7f0b9de00640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b8406b4d0 0x7f0b8406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b90036000 tx=0x7f0b90008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.567+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b98059310 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.803+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ba80d0720 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.804+0000 7f0b9f400640 1 -- 172.21.15.97:0/1380243990 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f0b98058c10 con 0x7f0ba80c9ec0 2024-03-06T07:28:38.807 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:38.807 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4245355910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4245355910},{"type":"v1","addr":"172.21.15.135:6835","nonce":4245355910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/499809251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":499809251},{"type":"v1","addr":"172.21.15.135:6834","nonce":499809251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1237384741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1237384741},{"type":"v1","addr":"172.21.15.97:6835","nonce":1237384741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:41.541937+0000","modified":"2024-03-06T07:27:44.578982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/4149107409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4149107409},{"type":"v1","addr":"172.21.15.97:6837","nonce":4149107409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-06T07:28:38.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b8406b4d0 msgr2=0x7f0b8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b8406b4d0 0x7f0b8406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b90036000 tx=0x7f0b90008030 comp rx=0 tx=0).stop 2024-03-06T07:28:38.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 msgr2=0x7f0ba80b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 0x7f0ba80b6480 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800d7d0 tx=0x7f0b9800dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:38.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 shutdown_connections 2024-03-06T07:28:38.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b8406b4d0 0x7f0b8406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ba8144660 0x7f0ba81411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ba80cf3e0 0x7f0ba81610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 --2- 172.21.15.97:0/1380243990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ba80c9ec0 0x7f0ba80b6480 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 >> 172.21.15.97:0/1380243990 conn(0x7f0ba80d0b90 msgr2=0x7f0ba80b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 shutdown_connections 2024-03-06T07:28:38.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:38.807+0000 7f0bad400640 1 -- 172.21.15.97:0/1380243990 wait complete. 2024-03-06T07:28:38.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.033+0000 7f88e1400640 1 Processor -- start 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.033+0000 7f88e1400640 1 -- start start 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.034+0000 7f88e1400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f88dc15e0d0 0x7f88dc0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.034+0000 7f88e1400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 0x7f88dc0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.034+0000 7f88e1400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc0cea60 con 0x7f88dc0d0df0 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.034+0000 7f88e1400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88dc0d1260 con 0x7f88dc15e0d0 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.034+0000 7f88e1400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc0d2f60 con 0x7f88dc15f470 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.035+0000 7f88d9e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 0x7f88dc0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.035+0000 7f88d9e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 0x7f88dc0d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:47598/0 (socket says 172.21.15.97:47598) 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.035+0000 7f88d9e00640 1 -- 172.21.15.97:0/63825611 learned_addr learned my addr 172.21.15.97:0/63825611 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.035+0000 7f88da800640 1 --2- 172.21.15.97:0/63825611 >> v2:172.21.15.135:3300/0 conn(0x7f88dc15e0d0 0x7f88dc0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88d9400640 1 -- 172.21.15.97:0/63825611 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116528359 0 0) 0x7f88dc0cea60 con 0x7f88dc0d0df0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88d9400640 1 -- 172.21.15.97:0/63825611 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c0003520 con 0x7f88dc0d0df0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88da800640 1 -- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 msgr2=0x7f88dc0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88da800640 1 --2- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 0x7f88dc0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88da800640 1 -- 172.21.15.97:0/63825611 >> v1:172.21.15.135:6789/0 conn(0x7f88dc0d0df0 legacy=0x7f88dc160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88da800640 1 -- 172.21.15.97:0/63825611 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc0d2350 con 0x7f88dc15e0d0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.036+0000 7f88d9e00640 1 --2- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 0x7f88dc0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.038+0000 7f88da800640 1 --2- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15e0d0 0x7f88dc0d1a90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0097a0 tx=0x7f88cc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6907fea973cb154 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.038+0000 7f88d9400640 1 -- 172.21.15.97:0/63825611 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc034070 con 0x7f88dc15e0d0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.038+0000 7f88d9400640 1 -- 172.21.15.97:0/63825611 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88cc030610 con 0x7f88dc15e0d0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.038+0000 7f88d9400640 1 -- 172.21.15.97:0/63825611 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0309e0 con 0x7f88dc15e0d0 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 -- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15e0d0 msgr2=0x7f88dc0d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 --2- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15e0d0 0x7f88dc0d1a90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0097a0 tx=0x7f88cc02b960 comp rx=0 tx=0).stop 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 -- 172.21.15.97:0/63825611 shutdown_connections 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 --2- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15f470 0x7f88dc0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 --2- 172.21.15.97:0/63825611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15e0d0 0x7f88dc0d1a90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 -- 172.21.15.97:0/63825611 >> 172.21.15.97:0/63825611 conn(0x7f88dc0d3ec0 msgr2=0x7f88dc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 -- 172.21.15.97:0/63825611 shutdown_connections 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.039+0000 7f88e1400640 1 -- 172.21.15.97:0/63825611 wait complete. 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.041+0000 7f88e1400640 1 Processor -- start 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.041+0000 7f88e1400640 1 -- start start 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88e1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 0x7f88dc155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88e1400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15e0d0 0x7f88dc152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88e1400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15f470 0x7f88dc0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88e1400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc0d2bf0 con 0x7f88dc0d0df0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88e1400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc0d2910 con 0x7f88dc15f470 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88e1400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc15fb30 con 0x7f88dc15e0d0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 0x7f88dc155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 0x7f88dc155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55222/0 (socket says 172.21.15.97:55222) 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 -- 172.21.15.97:0/3453893593 learned_addr learned my addr 172.21.15.97:0/3453893593 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88d9e00640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15e0d0 0x7f88dc152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88db200640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15f470 0x7f88dc0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 -- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15e0d0 msgr2=0x7f88dc152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15e0d0 0x7f88dc152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 -- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15f470 msgr2=0x7f88dc0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15f470 0x7f88dc0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88da800640 1 -- 172.21.15.97:0/3453893593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc152f60 con 0x7f88dc0d0df0 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88d9e00640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15e0d0 0x7f88dc152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.043+0000 7f88db200640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15f470 0x7f88dc0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.044+0000 7f88da800640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 0x7f88dc155840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009890 tx=0x7f88cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.044+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc034040 con 0x7f88dc0d0df0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.044+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88cc03aad0 con 0x7f88dc0d0df0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.044+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88dc153240 con 0x7f88dc0d0df0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.044+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc006e10 con 0x7f88dc0d0df0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.044+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88dc0cdd50 con 0x7f88dc0d0df0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.046+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88dc0682f0 con 0x7f88dc0d0df0 2024-03-06T07:28:39.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.047+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88cc04b060 con 0x7f88dc0d0df0 2024-03-06T07:28:39.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.047+0000 7f88d2a00640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88bc06b4d0 0x7f88bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.048+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f88cc0ac070 con 0x7f88dc0d0df0 2024-03-06T07:28:39.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.048+0000 7f88d9e00640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88bc06b4d0 0x7f88bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.048+0000 7f88d9e00640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88bc06b4d0 0x7f88bc06d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009630 tx=0x7f88c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.052+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88dc0682f0 con 0x7f88dc0d0df0 2024-03-06T07:28:39.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.250+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88dc04f330 con 0x7f88dc0d0df0 2024-03-06T07:28:39.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.252+0000 7f88d2a00640 1 -- 172.21.15.97:0/3453893593 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16983 (secure 0 0 0) 0x7f88cc03f070 con 0x7f88dc0d0df0 2024-03-06T07:28:39.255 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:39.255 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":24,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:28.349664+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:39.111579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:40.559291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:39.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88bc06b4d0 msgr2=0x7f88bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88bc06b4d0 0x7f88bc06d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009630 tx=0x7f88c8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 msgr2=0x7f88dc155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 0x7f88dc155840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009890 tx=0x7f88cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 shutdown_connections 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88bc06b4d0 0x7f88bc06d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88dc15f470 0x7f88dc0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88dc15e0d0 0x7f88dc152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 --2- 172.21.15.97:0/3453893593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88dc0d0df0 0x7f88dc155840 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 >> 172.21.15.97:0/3453893593 conn(0x7f88dc0d3ec0 msgr2=0x7f88dc0b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 shutdown_connections 2024-03-06T07:28:39.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.255+0000 7f88e1400640 1 -- 172.21.15.97:0/3453893593 wait complete. 2024-03-06T07:28:39.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:39.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.502+0000 7f7503200640 1 Processor -- start 2024-03-06T07:28:39.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.502+0000 7f7503200640 1 -- start start 2024-03-06T07:28:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f7503200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f7503200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f74fc1474c0 0x7f74fc0c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f7503200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f74fc0c4e70 0x7f74fc161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f7503200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74fc0c52d0 con 0x7f74fc1474c0 2024-03-06T07:28:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f7503200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74fc0ae380 con 0x7f74fc0c4e70 2024-03-06T07:28:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f7503200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0ae660 con 0x7f74fc1444e0 2024-03-06T07:28:39.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f74f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f74fc1474c0 0x7f74fc0c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f74f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f74fc1474c0 0x7f74fc0c4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55230/0 (socket says 172.21.15.97:55230) 2024-03-06T07:28:39.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f74f3e00640 1 -- 172.21.15.97:0/2180183249 learned_addr learned my addr 172.21.15.97:0/2180183249 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.504+0000 7f74fbe00640 1 --2- 172.21.15.97:0/2180183249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f7501200640 1 --2- 172.21.15.97:0/2180183249 >> v2:172.21.15.135:3300/0 conn(0x7f74fc0c4e70 0x7f74fc161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f74f3e00640 1 -- 172.21.15.97:0/2180183249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 msgr2=0x7f74fc0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f74f3e00640 1 --2- 172.21.15.97:0/2180183249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f74f3e00640 1 -- 172.21.15.97:0/2180183249 >> v2:172.21.15.135:3300/0 conn(0x7f74fc0c4e70 msgr2=0x7f74fc161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f74f3e00640 1 --2- 172.21.15.97:0/2180183249 >> v2:172.21.15.135:3300/0 conn(0x7f74fc0c4e70 0x7f74fc161d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f74f3e00640 1 -- 172.21.15.97:0/2180183249 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc0ae940 con 0x7f74fc1474c0 2024-03-06T07:28:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.505+0000 7f7501200640 1 --2- 172.21.15.97:0/2180183249 >> v2:172.21.15.135:3300/0 conn(0x7f74fc0c4e70 0x7f74fc161d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.506+0000 7f74f3e00640 1 --2- 172.21.15.97:0/2180183249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0c4930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0cf660 tx=0x7f74e402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa01d0d932a05ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.506+0000 7f74fb400640 1 -- 172.21.15.97:0/2180183249 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4035020 con 0x7f74fc1474c0 2024-03-06T07:28:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.506+0000 7f74fb400640 1 -- 172.21.15.97:0/2180183249 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e402d040 con 0x7f74fc1474c0 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.506+0000 7f74fb400640 1 -- 172.21.15.97:0/2180183249 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4031a80 con 0x7f74fc1474c0 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 -- 172.21.15.97:0/2180183249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 msgr2=0x7f74fc0c4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 --2- 172.21.15.97:0/2180183249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0c4930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0cf660 tx=0x7f74e402b900 comp rx=0 tx=0).stop 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 -- 172.21.15.97:0/2180183249 shutdown_connections 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 --2- 172.21.15.97:0/2180183249 >> v2:172.21.15.135:3300/0 conn(0x7f74fc0c4e70 0x7f74fc161d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 --2- 172.21.15.97:0/2180183249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0c4930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 --2- 172.21.15.97:0/2180183249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 -- 172.21.15.97:0/2180183249 >> 172.21.15.97:0/2180183249 conn(0x7f74fc0cc4f0 msgr2=0x7f74fc0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 -- 172.21.15.97:0/2180183249 shutdown_connections 2024-03-06T07:28:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.507+0000 7f7503200640 1 -- 172.21.15.97:0/2180183249 wait complete. 2024-03-06T07:28:39.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.508+0000 7f7503200640 1 Processor -- start 2024-03-06T07:28:39.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.509+0000 7f7503200640 1 -- start start 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.509+0000 7f7503200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74fc0c4e70 0x7f74fc136660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.509+0000 7f7503200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7503200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7503200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc147cb0 con 0x7f74fc1474c0 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7503200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0aef00 con 0x7f74fc0c4e70 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7503200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc1448c0 con 0x7f74fc1444e0 2024-03-06T07:28:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0ca520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55246/0 (socket says 172.21.15.97:55246) 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 -- 172.21.15.97:0/1704347634 learned_addr learned my addr 172.21.15.97:0/1704347634 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f74fbe00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74fc0c4e70 0x7f74fc136660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f74f3e00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 -- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 msgr2=0x7f74fc0b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 -- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74fc0c4e70 msgr2=0x7f74fc136660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74fc0c4e70 0x7f74fc136660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f7501200640 1 -- 172.21.15.97:0/1704347634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc0b3f00 con 0x7f74fc1474c0 2024-03-06T07:28:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f74fbe00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74fc0c4e70 0x7f74fc136660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.510+0000 7f74f3e00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.511+0000 7f7501200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0ca520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f74f400cd80 tx=0x7f74f400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.511+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f401a070 con 0x7f74fc1474c0 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.511+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f400d840 con 0x7f74fc1474c0 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.511+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f400dc30 con 0x7f74fc1474c0 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.511+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74fc0cb210 con 0x7f74fc1474c0 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.511+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc0b4460 con 0x7f74fc1474c0 2024-03-06T07:28:39.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.512+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc04f330 con 0x7f74fc1474c0 2024-03-06T07:28:39.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.513+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74f401d070 con 0x7f74fc1474c0 2024-03-06T07:28:39.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.515+0000 7f74f8c00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d806b4d0 0x7f74d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.515+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f74f408aa50 con 0x7f74fc1474c0 2024-03-06T07:28:39.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.515+0000 7f74fbe00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d806b4d0 0x7f74d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.516+0000 7f74fbe00640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d806b4d0 0x7f74d806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f74ec006fd0 tx=0x7f74ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.519+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74fc0b3f00 con 0x7f74fc1474c0 2024-03-06T07:28:39.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.752+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f74fc072810 con 0x7f74fc1474c0 2024-03-06T07:28:39.755 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:28:39.753+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:39.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.756+0000 7f74f8c00640 1 -- 172.21.15.97:0/1704347634 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f74f40597f0 con 0x7f74fc1474c0 2024-03-06T07:28:39.758 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:39.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.759+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d806b4d0 msgr2=0x7f74d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.759+0000 7f7503200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d806b4d0 0x7f74d806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f74ec006fd0 tx=0x7f74ec008040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.759+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 msgr2=0x7f74fc0ca520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.759+0000 7f7503200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0ca520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f74f400cd80 tx=0x7f74f400a710 comp rx=0 tx=0).stop 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 shutdown_connections 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d806b4d0 0x7f74d806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74fc1474c0 0x7f74fc0ca520 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74fc1444e0 0x7f74fc0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 --2- 172.21.15.97:0/1704347634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74fc0c4e70 0x7f74fc136660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 >> 172.21.15.97:0/1704347634 conn(0x7f74fc0cc4f0 msgr2=0x7f74fc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 shutdown_connections 2024-03-06T07:28:39.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.762+0000 7f7503200640 1 -- 172.21.15.97:0/1704347634 wait complete. 2024-03-06T07:28:39.774 INFO:tasks.ceph.mds.c.smithi097.stderr: -7820> 2024-03-06T07:27:45.188+0000 7f509ac00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:39.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:39.833 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:28:39.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.968+0000 7f9246800640 1 Processor -- start 2024-03-06T07:28:39.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.968+0000 7f9246800640 1 -- start start 2024-03-06T07:28:39.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.969+0000 7f9246800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92400cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.969+0000 7f9246800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f92400cfaf0 0x7f92400d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.969+0000 7f9246800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92400d13e0 con 0x7f92400d23e0 2024-03-06T07:28:39.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.969+0000 7f9246800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92400cf120 con 0x7f92400cfaf0 2024-03-06T07:28:39.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.969+0000 7f9246800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92400cead0 con 0x7f924015e130 2024-03-06T07:28:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.970+0000 7f9245400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92400cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.970+0000 7f9245e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f92400d23e0 0x7f92400c7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:51422/0 (socket says 172.21.15.97:51422) 2024-03-06T07:28:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f9245e00640 1 -- 172.21.15.97:0/598110329 learned_addr learned my addr 172.21.15.97:0/598110329 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f9244a00640 1 --2- 172.21.15.97:0/598110329 >> v2:172.21.15.135:3300/0 conn(0x7f92400cfaf0 0x7f92400d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f923fe00640 1 -- 172.21.15.97:0/598110329 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750165521 0 0) 0x7f92400d13e0 con 0x7f92400d23e0 2024-03-06T07:28:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f9245400640 1 -- 172.21.15.97:0/598110329 >> v2:172.21.15.135:3300/0 conn(0x7f92400cfaf0 msgr2=0x7f92400d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f9245400640 1 --2- 172.21.15.97:0/598110329 >> v2:172.21.15.135:3300/0 conn(0x7f92400cfaf0 0x7f92400d1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f9245400640 1 -- 172.21.15.97:0/598110329 >> v1:172.21.15.135:6789/0 conn(0x7f92400d23e0 legacy=0x7f92400c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.971+0000 7f9245400640 1 -- 172.21.15.97:0/598110329 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92400cedb0 con 0x7f924015e130 2024-03-06T07:28:39.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.973+0000 7f9245400640 1 --2- 172.21.15.97:0/598110329 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92400cf5b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92340097a0 tx=0x7f923402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f17b397ddc0de242 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.973+0000 7f923fe00640 1 -- 172.21.15.97:0/598110329 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234035020 con 0x7f924015e130 2024-03-06T07:28:39.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.973+0000 7f923fe00640 1 -- 172.21.15.97:0/598110329 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9234031830 con 0x7f924015e130 2024-03-06T07:28:39.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.973+0000 7f923fe00640 1 -- 172.21.15.97:0/598110329 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234031c00 con 0x7f924015e130 2024-03-06T07:28:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 -- 172.21.15.97:0/598110329 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 msgr2=0x7f92400cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 --2- 172.21.15.97:0/598110329 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92400cf5b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92340097a0 tx=0x7f923402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 -- 172.21.15.97:0/598110329 shutdown_connections 2024-03-06T07:28:39.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 --2- 172.21.15.97:0/598110329 >> v2:172.21.15.135:3300/0 conn(0x7f92400cfaf0 0x7f92400d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 --2- 172.21.15.97:0/598110329 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92400cf5b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 -- 172.21.15.97:0/598110329 >> 172.21.15.97:0/598110329 conn(0x7f924009d070 msgr2=0x7f92400d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 -- 172.21.15.97:0/598110329 shutdown_connections 2024-03-06T07:28:39.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.974+0000 7f9246800640 1 -- 172.21.15.97:0/598110329 wait complete. 2024-03-06T07:28:39.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.976+0000 7f9246800640 1 Processor -- start 2024-03-06T07:28:39.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.976+0000 7f9246800640 1 -- start start 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.977+0000 7f9246800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92400cfaf0 0x7f92400cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.977+0000 7f9246800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 0x7f92400cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.977+0000 7f9246800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92401e7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.977+0000 7f9246800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92400c8200 con 0x7f92400d23e0 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.977+0000 7f9246800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92400c8080 con 0x7f92400cfaf0 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.977+0000 7f9246800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92400d06b0 con 0x7f924015e130 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 0x7f92400cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 0x7f92400cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55274/0 (socket says 172.21.15.97:55274) 2024-03-06T07:28:39.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 -- 172.21.15.97:0/684167671 learned_addr learned my addr 172.21.15.97:0/684167671 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:39.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9245e00640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92401e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 -- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 msgr2=0x7f92401e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92401e7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 -- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92400cfaf0 msgr2=0x7f92400cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92400cfaf0 0x7f92400cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 -- 172.21.15.97:0/684167671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92401e8c00 con 0x7f92400d23e0 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.978+0000 7f9244a00640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 0x7f92400cca50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f922c00e980 tx=0x7f922c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.979+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c017070 con 0x7f92400d23e0 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.979+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c00ca00 con 0x7f92400d23e0 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.979+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c00cdd0 con 0x7f92400d23e0 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.979+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92400cd0c0 con 0x7f92400d23e0 2024-03-06T07:28:39.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.979+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92400cd900 con 0x7f92400d23e0 2024-03-06T07:28:39.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.980+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924004f330 con 0x7f92400d23e0 2024-03-06T07:28:39.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.981+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f922c01a070 con 0x7f92400d23e0 2024-03-06T07:28:39.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.982+0000 7f923d600640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f922006b4d0 0x7f922006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.982+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5998+0+0 (secure 0 0 0) 0x7f922c08a3d0 con 0x7f92400d23e0 2024-03-06T07:28:39.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.982+0000 7f9245400640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f922006b4d0 0x7f922006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.983+0000 7f9245400640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f922006b4d0 0x7f922006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9234036000 tx=0x7f9234030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:39.986+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92400cd0c0 con 0x7f92400d23e0 2024-03-06T07:28:40.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.225+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f924009d690 con 0x7f92400d23e0 2024-03-06T07:28:40.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.759+0000 7f923d600640 1 -- 172.21.15.97:0/684167671 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f922c059090 con 0x7f92400d23e0 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f922006b4d0 msgr2=0x7f922006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f922006b4d0 0x7f922006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9234036000 tx=0x7f9234030000 comp rx=0 tx=0).stop 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 msgr2=0x7f92400cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 0x7f92400cca50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f922c00e980 tx=0x7f922c00ee40 comp rx=0 tx=0).stop 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 shutdown_connections 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f922006b4d0 0x7f922006d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f924015e130 0x7f92401e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92400d23e0 0x7f92400cca50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 --2- 172.21.15.97:0/684167671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92400cfaf0 0x7f92400cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.763+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 >> 172.21.15.97:0/684167671 conn(0x7f924009d070 msgr2=0x7f9240162310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.764+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 shutdown_connections 2024-03-06T07:28:40.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:40.764+0000 7f9246800640 1 -- 172.21.15.97:0/684167671 wait complete. 2024-03-06T07:28:40.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:41.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.000+0000 7fd097600640 1 Processor -- start 2024-03-06T07:28:41.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.001+0000 7fd097600640 1 -- start start 2024-03-06T07:28:41.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.002+0000 7fd097600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd090165fe0 0x7fd0901634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.002+0000 7fd097600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd09006c9c0 0x7fd0901663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.002+0000 7fd097600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd090166900 con 0x7fd09006d620 2024-03-06T07:28:41.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.002+0000 7fd097600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd090071aa0 con 0x7fd09006c9c0 2024-03-06T07:28:41.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.002+0000 7fd097600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0900b3f00 con 0x7fd090165fe0 2024-03-06T07:28:41.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.002+0000 7fd08fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd09006c9c0 0x7fd0901663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd095600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd09006d620 0x7fd09006dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48454/0 (socket says 172.21.15.97:48454) 2024-03-06T07:28:41.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd095600640 1 -- 172.21.15.97:0/2991511461 learned_addr learned my addr 172.21.15.97:0/2991511461 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:41.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd094c00640 1 --2- 172.21.15.97:0/2991511461 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd090165fe0 0x7fd0901634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd08fe00640 1 -- 172.21.15.97:0/2991511461 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd090165fe0 msgr2=0x7fd0901634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd08fe00640 1 --2- 172.21.15.97:0/2991511461 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd090165fe0 0x7fd0901634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd08fe00640 1 -- 172.21.15.97:0/2991511461 >> v1:172.21.15.97:6789/0 conn(0x7fd09006d620 legacy=0x7fd09006dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd08fe00640 1 -- 172.21.15.97:0/2991511461 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900b41e0 con 0x7fd09006c9c0 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.003+0000 7fd08f400640 1 -- 172.21.15.97:0/2991511461 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187111117 0 0) 0x7fd090166900 con 0x7fd09006d620 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.005+0000 7fd08fe00640 1 --2- 172.21.15.97:0/2991511461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd09006c9c0 0x7fd0901663c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0097a0 tx=0x7fd07c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6583b8c7c23adb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.005+0000 7fd08f400640 1 -- 172.21.15.97:0/2991511461 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c035020 con 0x7fd09006c9c0 2024-03-06T07:28:41.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.005+0000 7fd08f400640 1 -- 172.21.15.97:0/2991511461 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c031830 con 0x7fd09006c9c0 2024-03-06T07:28:41.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.005+0000 7fd08f400640 1 -- 172.21.15.97:0/2991511461 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c031c00 con 0x7fd09006c9c0 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.005+0000 7fd097600640 1 -- 172.21.15.97:0/2991511461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd09006c9c0 msgr2=0x7fd0901663c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.005+0000 7fd097600640 1 --2- 172.21.15.97:0/2991511461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd09006c9c0 0x7fd0901663c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0097a0 tx=0x7fd07c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.006+0000 7fd097600640 1 -- 172.21.15.97:0/2991511461 shutdown_connections 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.006+0000 7fd097600640 1 --2- 172.21.15.97:0/2991511461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd09006c9c0 0x7fd0901663c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.006+0000 7fd097600640 1 --2- 172.21.15.97:0/2991511461 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd090165fe0 0x7fd0901634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.006+0000 7fd097600640 1 -- 172.21.15.97:0/2991511461 >> 172.21.15.97:0/2991511461 conn(0x7fd0900cfa80 msgr2=0x7fd0900cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.006+0000 7fd097600640 1 -- 172.21.15.97:0/2991511461 shutdown_connections 2024-03-06T07:28:41.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.006+0000 7fd097600640 1 -- 172.21.15.97:0/2991511461 wait complete. 2024-03-06T07:28:41.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.007+0000 7fd097600640 1 Processor -- start 2024-03-06T07:28:41.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.007+0000 7fd097600640 1 -- start start 2024-03-06T07:28:41.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.008+0000 7fd097600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd09006c9c0 0x7fd0901f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.008+0000 7fd097600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd09006d620 0x7fd09014a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.008+0000 7fd097600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 0x7fd09014aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.008+0000 7fd097600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09006ce20 con 0x7fd090165fe0 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.008+0000 7fd097600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0900b47a0 con 0x7fd09006c9c0 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.008+0000 7fd097600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09014c0d0 con 0x7fd09006d620 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 0x7fd09014aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 0x7fd09014aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55298/0 (socket says 172.21.15.97:55298) 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 -- 172.21.15.97:0/1729599753 learned_addr learned my addr 172.21.15.97:0/1729599753 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 -- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd09006d620 msgr2=0x7fd09014a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd09006d620 0x7fd09014a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd094c00640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd09006c9c0 0x7fd0901f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 -- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd09006c9c0 msgr2=0x7fd0901f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd09006c9c0 0x7fd0901f5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 -- 172.21.15.97:0/1729599753 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900aea10 con 0x7fd090165fe0 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.009+0000 7fd095600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 0x7fd09014aef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c780 tx=0x7fd08800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.010+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088018070 con 0x7fd090165fe0 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.010+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08800a640 con 0x7fd090165fe0 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.010+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08800aa30 con 0x7fd090165fe0 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.010+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0900aecf0 con 0x7fd090165fe0 2024-03-06T07:28:41.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.010+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0900af2b0 con 0x7fd090165fe0 2024-03-06T07:28:41.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.011+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09004f380 con 0x7fd090165fe0 2024-03-06T07:28:41.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.012+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd08801b070 con 0x7fd090165fe0 2024-03-06T07:28:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.013+0000 7fd08cc00640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd07006b4d0 0x7fd07006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.013+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5998+0+0 (secure 0 0 0) 0x7fd088089910 con 0x7fd090165fe0 2024-03-06T07:28:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.013+0000 7fd094c00640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd07006b4d0 0x7fd07006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.013+0000 7fd094c00640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd07006b4d0 0x7fd07006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd084006fd0 tx=0x7fd084008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.017+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd088058660 con 0x7fd090165fe0 2024-03-06T07:28:41.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.213+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd0901491f0 con 0x7fd090165fe0 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.769+0000 7fd08cc00640 1 -- 172.21.15.97:0/1729599753 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd088057f60 con 0x7fd090165fe0 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.774+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd07006b4d0 msgr2=0x7fd07006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.774+0000 7fd097600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd07006b4d0 0x7fd07006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd084006fd0 tx=0x7fd084008040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.774+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 msgr2=0x7fd09014aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.774+0000 7fd097600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 0x7fd09014aef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c780 tx=0x7fd08800cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.775+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 shutdown_connections 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.776+0000 7fd097600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd07006b4d0 0x7fd07006d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.776+0000 7fd097600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd090165fe0 0x7fd09014aef0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.776+0000 7fd097600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd09006d620 0x7fd09014a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.776+0000 7fd097600640 1 --2- 172.21.15.97:0/1729599753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd09006c9c0 0x7fd0901f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.776+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 >> 172.21.15.97:0/1729599753 conn(0x7fd0900cfa80 msgr2=0x7fd0900b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.777+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 shutdown_connections 2024-03-06T07:28:41.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:41.777+0000 7fd097600640 1 -- 172.21.15.97:0/1729599753 wait complete. 2024-03-06T07:28:41.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.005+0000 7fd59b200640 1 Processor -- start 2024-03-06T07:28:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.006+0000 7fd59b200640 1 -- start start 2024-03-06T07:28:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.007+0000 7fd59b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 0x7fd59c0d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.007+0000 7fd59b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd59c0d0c70 0x7fd59c0b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.007+0000 7fd59b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c0cfd60 con 0x7fd59c0b5320 2024-03-06T07:28:42.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.007+0000 7fd59b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c0d10e0 con 0x7fd59c0d0c70 2024-03-06T07:28:42.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.007+0000 7fd59b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0d2990 con 0x7fd59c0d18f0 2024-03-06T07:28:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd591400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 0x7fd59c0d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd591400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 0x7fd59c0d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:47680/0 (socket says 172.21.15.97:47680) 2024-03-06T07:28:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd59a800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd59c0d0c70 0x7fd59c0b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd591400640 1 -- 172.21.15.97:0/3720225968 learned_addr learned my addr 172.21.15.97:0/3720225968 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd599e00640 1 --1- 172.21.15.97:0/3720225968 >> v1:172.21.15.135:6789/0 conn(0x7fd59c0b5320 0x7fd59c0cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:51438/0 (socket says 172.21.15.97:51438) 2024-03-06T07:28:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd591400640 1 -- 172.21.15.97:0/3720225968 >> v2:172.21.15.135:3300/0 conn(0x7fd59c0d0c70 msgr2=0x7fd59c0b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.008+0000 7fd591400640 1 --2- 172.21.15.97:0/3720225968 >> v2:172.21.15.135:3300/0 conn(0x7fd59c0d0c70 0x7fd59c0b6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.009+0000 7fd591400640 1 -- 172.21.15.97:0/3720225968 >> v1:172.21.15.135:6789/0 conn(0x7fd59c0b5320 legacy=0x7fd59c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.009+0000 7fd591400640 1 -- 172.21.15.97:0/3720225968 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0d1d80 con 0x7fd59c0d18f0 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.010+0000 7fd591400640 1 --2- 172.21.15.97:0/3720225968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 0x7fd59c0d0730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd584009870 tx=0x7fd58402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd9f4e4d55211a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.011+0000 7fd599400640 1 -- 172.21.15.97:0/3720225968 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584034070 con 0x7fd59c0d18f0 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.011+0000 7fd599400640 1 -- 172.21.15.97:0/3720225968 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd584030840 con 0x7fd59c0d18f0 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.011+0000 7fd599400640 1 -- 172.21.15.97:0/3720225968 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584030c10 con 0x7fd59c0d18f0 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 -- 172.21.15.97:0/3720225968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 msgr2=0x7fd59c0d0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 --2- 172.21.15.97:0/3720225968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 0x7fd59c0d0730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd584009870 tx=0x7fd58402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 -- 172.21.15.97:0/3720225968 shutdown_connections 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 --2- 172.21.15.97:0/3720225968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d18f0 0x7fd59c0d0730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 --2- 172.21.15.97:0/3720225968 >> v2:172.21.15.135:3300/0 conn(0x7fd59c0d0c70 0x7fd59c0b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 -- 172.21.15.97:0/3720225968 >> 172.21.15.97:0/3720225968 conn(0x7fd59c0d3d20 msgr2=0x7fd59c0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 -- 172.21.15.97:0/3720225968 shutdown_connections 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.012+0000 7fd59b200640 1 -- 172.21.15.97:0/3720225968 wait complete. 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.014+0000 7fd59b200640 1 Processor -- start 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.014+0000 7fd59b200640 1 -- start start 2024-03-06T07:28:42.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd59b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 0x7fd59c159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd59b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d0c70 0x7fd59c0cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd59b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59c0cce90 0x7fd59c0b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd59b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0d2620 con 0x7fd59c0b5320 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd59b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0d2340 con 0x7fd59c0cce90 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd59b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b70b0 con 0x7fd59c0d0c70 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd599e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 0x7fd59c159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd599e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 0x7fd59c159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55314/0 (socket says 172.21.15.97:55314) 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.015+0000 7fd599e00640 1 -- 172.21.15.97:0/1884010668 learned_addr learned my addr 172.21.15.97:0/1884010668 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd59a800640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59c0cce90 0x7fd59c0b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd591400640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d0c70 0x7fd59c0cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd599e00640 1 -- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d0c70 msgr2=0x7fd59c0cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd599e00640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d0c70 0x7fd59c0cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd599e00640 1 -- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59c0cce90 msgr2=0x7fd59c0b84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd599e00640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59c0cce90 0x7fd59c0b84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd599e00640 1 -- 172.21.15.97:0/1884010668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0cbee0 con 0x7fd59c0b5320 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd59a800640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59c0cce90 0x7fd59c0b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd591400640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d0c70 0x7fd59c0cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd599e00640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 0x7fd59c159510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c400 tx=0x7fd58c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c0074e0 con 0x7fd59c0b5320 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c007680 con 0x7fd59c0b5320 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.016+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c004a00 con 0x7fd59c0b5320 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.017+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c0cb640 con 0x7fd59c0b5320 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.017+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c0cbc00 con 0x7fd59c0b5320 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.018+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c04f330 con 0x7fd59c0b5320 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.019+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd58c01a020 con 0x7fd59c0b5320 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.020+0000 7fd592a00640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.020+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5560+0+0 (secure 0 0 0) 0x7fd58c089890 con 0x7fd59c0b5320 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.020+0000 7fd591400640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.021+0000 7fd591400640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd584036000 tx=0x7fd58403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.025+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd58c058830 con 0x7fd59c0b5320 2024-03-06T07:28:42.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.229+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd59c0d12d0 con 0x7fd59c0b5320 2024-03-06T07:28:42.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.780+0000 7fd592a00640 1 -- 172.21.15.97:0/1884010668 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd58c058130 con 0x7fd59c0b5320 2024-03-06T07:28:42.783 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:42.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.785+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 msgr2=0x7fd57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.785+0000 7fd59b200640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd584036000 tx=0x7fd58403b040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.785+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 msgr2=0x7fd59c159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.785+0000 7fd59b200640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 0x7fd59c159510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c400 tx=0x7fd58c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.786+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 shutdown_connections 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.786+0000 7fd59b200640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.786+0000 7fd59b200640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59c0cce90 0x7fd59c0b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.786+0000 7fd59b200640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd59c0d0c70 0x7fd59c0cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.787+0000 7fd59b200640 1 --2- 172.21.15.97:0/1884010668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd59c0b5320 0x7fd59c159510 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.787+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 >> 172.21.15.97:0/1884010668 conn(0x7fd59c0d3d20 msgr2=0x7fd59c15fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.787+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 shutdown_connections 2024-03-06T07:28:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:42.787+0000 7fd59b200640 1 -- 172.21.15.97:0/1884010668 wait complete. 2024-03-06T07:28:42.814 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:42.814 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:42.814 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:42.815 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:42.815 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:42.815 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:42.815 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:42.815 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:42.815 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:42.816 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:42.816 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:42.816 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:42.816 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:42.816 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:42.834 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:42.834 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:42.836 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:42.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:42.839 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:42.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:42.842 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:42.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:42.883 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:42.883 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:42.888 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:42.889 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:42.889 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:42.889 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:42.895 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:42.895 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:42.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:43.041 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:28:43.041 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:28:43.042 INFO:tasks.ceph.mds.b.smithi135.stderr:2024-03-06T07:28:43.041+0000 7f480a000640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:28:43.043 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:28:43.045 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.078+0000 7f962ea00640 1 Processor -- start 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.078+0000 7f962ea00640 1 -- start start 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.079+0000 7f962ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f96280d2830 0x7f96280b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.079+0000 7f962ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96280af640 0x7f96280afad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.079+0000 7f962ea00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96280b32c0 con 0x7f962815f510 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.079+0000 7f962ea00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962815f8f0 con 0x7f96280d2830 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.079+0000 7f962ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280aeb60 con 0x7f96280af640 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.079+0000 7f9627400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f96280d2830 0x7f96280b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f962ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96280af640 0x7f96280afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9627400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f96280d2830 0x7f96280b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:45534/0 (socket says 172.21.15.97:45534) 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9627400640 1 -- 172.21.15.97:0/459936308 learned_addr learned my addr 172.21.15.97:0/459936308 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9627400640 1 -- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96280af640 msgr2=0x7f96280afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9627400640 1 --2- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96280af640 0x7f96280afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9627400640 1 -- 172.21.15.97:0/459936308 >> v1:172.21.15.135:6789/0 conn(0x7f962815f510 legacy=0x7f96280b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9627400640 1 -- 172.21.15.97:0/459936308 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280aee40 con 0x7f96280d2830 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f9626a00640 1 -- 172.21.15.97:0/459936308 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375733982 0 0) 0x7f96280b32c0 con 0x7f962815f510 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.080+0000 7f962ca00640 1 --2- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96280af640 0x7f96280afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.081+0000 7f9627400640 1 --2- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96280b1b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9614009870 tx=0x7f961402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a986b271ee42a609 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.081+0000 7f9626a00640 1 -- 172.21.15.97:0/459936308 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614034070 con 0x7f96280d2830 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.081+0000 7f9626a00640 1 -- 172.21.15.97:0/459936308 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614030840 con 0x7f96280d2830 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.081+0000 7f9626a00640 1 -- 172.21.15.97:0/459936308 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614030c10 con 0x7f96280d2830 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 -- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 msgr2=0x7f96280b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 --2- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96280b1b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9614009870 tx=0x7f961402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 -- 172.21.15.97:0/459936308 shutdown_connections 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 --2- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96280af640 0x7f96280afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 --2- 172.21.15.97:0/459936308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96280b1b20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 -- 172.21.15.97:0/459936308 >> 172.21.15.97:0/459936308 conn(0x7f96280d1d10 msgr2=0x7f96280d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 -- 172.21.15.97:0/459936308 shutdown_connections 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.082+0000 7f962ea00640 1 -- 172.21.15.97:0/459936308 wait complete. 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.084+0000 7f962ea00640 1 Processor -- start 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.084+0000 7f962ea00640 1 -- start start 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 0x7f96280c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96281480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f962815f510 0x7f96280cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b0fc0 con 0x7f96280af640 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b3560 con 0x7f96280d2830 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b1140 con 0x7f962815f510 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f9627e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 0x7f96280c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f9627e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 0x7f96280c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55386/0 (socket says 172.21.15.97:55386) 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f9627e00640 1 -- 172.21.15.97:0/1325125259 learned_addr learned my addr 172.21.15.97:0/1325125259 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f962ca00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f962815f510 0x7f96280cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.085+0000 7f9627400640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96281480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f9627e00640 1 -- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f962815f510 msgr2=0x7f96280cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f9627e00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f962815f510 0x7f96280cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f9627e00640 1 -- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 msgr2=0x7f96281480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f9627e00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96281480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f9627e00640 1 -- 172.21.15.97:0/1325125259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280cdbe0 con 0x7f96280af640 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f9627e00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 0x7f96280c5840 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f961800c3e0 tx=0x7f961800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618017070 con 0x7f96280af640 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96180052d0 con 0x7f96280af640 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618003830 con 0x7f96280af640 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.086+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96280cde80 con 0x7f96280af640 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.087+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96280ce400 con 0x7f96280af640 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.088+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962804f330 con 0x7f96280af640 2024-03-06T07:28:43.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.089+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f961801a070 con 0x7f96280af640 2024-03-06T07:28:43.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.090+0000 7f961fe00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f960806b500 0x7f960806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.090+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f96180890c0 con 0x7f96280af640 2024-03-06T07:28:43.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.090+0000 7f9627400640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f960806b500 0x7f960806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.091+0000 7f9627400640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f960806b500 0x7f960806d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9614036000 tx=0x7f9614008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.094+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9618057c80 con 0x7f96280af640 2024-03-06T07:28:43.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.311+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f962815e6c0 con 0x7f96280af640 2024-03-06T07:28:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.803+0000 7f961fe00640 1 -- 172.21.15.97:0/1325125259 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f961805bd20 con 0x7f96280af640 2024-03-06T07:28:43.805 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.806+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f960806b500 msgr2=0x7f960806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.806+0000 7f962ea00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f960806b500 0x7f960806d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9614036000 tx=0x7f9614008030 comp rx=0 tx=0).stop 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.806+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 msgr2=0x7f96280c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.806+0000 7f962ea00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 0x7f96280c5840 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f961800c3e0 tx=0x7f961800c7e0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 shutdown_connections 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f960806b500 0x7f960806d9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f962815f510 0x7f96280cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96280d2830 0x7f96281480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 --2- 172.21.15.97:0/1325125259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96280af640 0x7f96280c5840 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 >> 172.21.15.97:0/1325125259 conn(0x7f96280d1d10 msgr2=0x7f962806f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 shutdown_connections 2024-03-06T07:28:43.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:43.807+0000 7f962ea00640 1 -- 172.21.15.97:0/1325125259 wait complete. 2024-03-06T07:28:43.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:44.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.033+0000 7f71b4000640 1 Processor -- start 2024-03-06T07:28:44.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.033+0000 7f71b4000640 1 -- start start 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.034+0000 7f71b4000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f71ac144c00 0x7f71ac06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.034+0000 7f71b4000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f71ac15f9d0 0x7f71ac15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.034+0000 7f71b4000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac06ee70 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.034+0000 7f71b4000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac06cef0 con 0x7f71ac144c00 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.034+0000 7f71b4000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac06d1d0 con 0x7f71ac15f9d0 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f71ac144c00 0x7f71ac06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f71ac144c00 0x7f71ac06e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41184/0 (socket says 172.21.15.97:41184) 2024-03-06T07:28:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 -- 172.21.15.97:0/2859995617 learned_addr learned my addr 172.21.15.97:0/2859995617 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 -- 172.21.15.97:0/2859995617 >> v2:172.21.15.135:3300/0 conn(0x7f71ac15f9d0 msgr2=0x7f71ac15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 --2- 172.21.15.97:0/2859995617 >> v2:172.21.15.135:3300/0 conn(0x7f71ac15f9d0 0x7f71ac15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 -- 172.21.15.97:0/2859995617 >> v1:172.21.15.135:6789/0 conn(0x7f71ac0ce0a0 legacy=0x7f71ac0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.035+0000 7f71b1600640 1 -- 172.21.15.97:0/2859995617 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac06d4b0 con 0x7f71ac144c00 2024-03-06T07:28:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.036+0000 7f71b1600640 1 --2- 172.21.15.97:0/2859995617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac144c00 0x7f71ac06e930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f71ac0b26f0 tx=0x7f719c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4743b2283df7bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71abe00640 1 -- 172.21.15.97:0/2859995617 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c035020 con 0x7f71ac144c00 2024-03-06T07:28:44.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71abe00640 1 -- 172.21.15.97:0/2859995617 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c031870 con 0x7f71ac144c00 2024-03-06T07:28:44.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71abe00640 1 -- 172.21.15.97:0/2859995617 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c031c40 con 0x7f71ac144c00 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 -- 172.21.15.97:0/2859995617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac144c00 msgr2=0x7f71ac06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 --2- 172.21.15.97:0/2859995617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac144c00 0x7f71ac06e930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f71ac0b26f0 tx=0x7f719c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 -- 172.21.15.97:0/2859995617 shutdown_connections 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 --2- 172.21.15.97:0/2859995617 >> v2:172.21.15.135:3300/0 conn(0x7f71ac15f9d0 0x7f71ac15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 --2- 172.21.15.97:0/2859995617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac144c00 0x7f71ac06e930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 -- 172.21.15.97:0/2859995617 >> 172.21.15.97:0/2859995617 conn(0x7f71ac0b1730 msgr2=0x7f71ac0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 -- 172.21.15.97:0/2859995617 shutdown_connections 2024-03-06T07:28:44.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.037+0000 7f71b4000640 1 -- 172.21.15.97:0/2859995617 wait complete. 2024-03-06T07:28:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.039+0000 7f71b4000640 1 Processor -- start 2024-03-06T07:28:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.039+0000 7f71b4000640 1 -- start start 2024-03-06T07:28:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b4000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 0x7f71ac13e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b4000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71ac144c00 0x7f71ac1e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b4000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71ac15f9d0 0x7f71ac0ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b4000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac06cb50 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b4000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac160730 con 0x7f71ac15f9d0 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b4000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac06ccd0 con 0x7f71ac144c00 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 0x7f71ac13e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 0x7f71ac13e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41188/0 (socket says 172.21.15.97:41188) 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.040+0000 7f71b1600640 1 -- 172.21.15.97:0/1224028529 learned_addr learned my addr 172.21.15.97:0/1224028529 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b1600640 1 -- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71ac144c00 msgr2=0x7f71ac1e6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b1600640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71ac144c00 0x7f71ac1e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b1600640 1 -- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71ac15f9d0 msgr2=0x7f71ac0ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b2000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71ac15f9d0 0x7f71ac0ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b1600640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71ac15f9d0 0x7f71ac0ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b1600640 1 -- 172.21.15.97:0/1224028529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac15e790 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b2000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71ac15f9d0 0x7f71ac0ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71b1600640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 0x7f71ac13e0e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f719c02b950 tx=0x7f719c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c03ad60 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c0443f0 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.041+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c0447e0 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.042+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac15ea70 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.042+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac15f030 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.043+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac04f330 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.044+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f719c038020 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.045+0000 7f71a9600640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f719006b4d0 0x7f719006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.045+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f719c0ac2e0 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.045+0000 7f71b0c00640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f719006b4d0 0x7f719006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.046+0000 7f71b0c00640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f719006b4d0 0x7f719006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71ac0ae970 tx=0x7f71a0006cb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.049+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f719c03f070 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.314+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f71ac0b4b20 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.315+0000 7f71a9600640 1 -- 172.21.15.97:0/1224028529 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f71ac06ee70 con 0x7f71ac0ce0a0 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:28:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDZGuhlOJYsLhAAPM4kILsTiCzTUqcnCuftLg==", 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDZGuhlo6csLhAAU+oRzRMnxGvO7WUbgzkx2w==", 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDZGuhl07MsLhAAbx+Ky9p2b6vR8MCY0cWKlg==", 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDZGuhlur8sLhAAxuEPJrs8wd3raReN6dskQg==", 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDZGuhlLcwsLhAAbADxcglNyBeTpOIT9qxzkA==", 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDZGuhl39gsLhAAgEE8SAyLfolSdfbBrCwwng==", 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.318+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f719006b4d0 msgr2=0x7f719006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.318+0000 7f71b4000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f719006b4d0 0x7f719006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71ac0ae970 tx=0x7f71a0006cb0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.318+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 msgr2=0x7f71ac13e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.318+0000 7f71b4000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 0x7f71ac13e0e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f719c02b950 tx=0x7f719c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 shutdown_connections 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f719006b4d0 0x7f719006d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71ac15f9d0 0x7f71ac0ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71ac144c00 0x7f71ac1e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 --2- 172.21.15.97:0/1224028529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71ac0ce0a0 0x7f71ac13e0e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 >> 172.21.15.97:0/1224028529 conn(0x7f71ac0b1730 msgr2=0x7f71ac142d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 shutdown_connections 2024-03-06T07:28:44.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.319+0000 7f71b4000640 1 -- 172.21.15.97:0/1224028529 wait complete. 2024-03-06T07:28:44.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:28:44.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.546+0000 7fd386200640 1 Processor -- start 2024-03-06T07:28:44.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.547+0000 7fd386200640 1 -- start start 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd386200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd38015e470 0x7fd3800d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd386200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd386200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3800d1f80 con 0x7fd3800d2480 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd386200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3800d4da0 con 0x7fd38015e470 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd386200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800cef80 con 0x7fd3800d2860 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd37ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd38015e470 0x7fd3800d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd37ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd38015e470 0x7fd3800d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41198/0 (socket says 172.21.15.97:41198) 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.548+0000 7fd37ea00640 1 -- 172.21.15.97:0/3770530251 learned_addr learned my addr 172.21.15.97:0/3770530251 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.549+0000 7fd37fe00640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.549+0000 7fd37ea00640 1 -- 172.21.15.97:0/3770530251 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 msgr2=0x7fd3800b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.549+0000 7fd37ea00640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.549+0000 7fd37ea00640 1 -- 172.21.15.97:0/3770530251 >> v1:172.21.15.135:6789/0 conn(0x7fd3800d2480 legacy=0x7fd38015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.549+0000 7fd37ea00640 1 -- 172.21.15.97:0/3770530251 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800cf260 con 0x7fd38015e470 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.549+0000 7fd37fe00640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.550+0000 7fd37ea00640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd38015e470 0x7fd3800d4860 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd38015e8e0 tx=0x7fd36802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4539359a48f89a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.550+0000 7fd37e000640 1 -- 172.21.15.97:0/3770530251 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368035020 con 0x7fd38015e470 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.550+0000 7fd37e000640 1 -- 172.21.15.97:0/3770530251 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368031830 con 0x7fd38015e470 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.550+0000 7fd37e000640 1 -- 172.21.15.97:0/3770530251 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368031c00 con 0x7fd38015e470 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 -- 172.21.15.97:0/3770530251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd38015e470 msgr2=0x7fd3800d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd38015e470 0x7fd3800d4860 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd38015e8e0 tx=0x7fd36802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 -- 172.21.15.97:0/3770530251 shutdown_connections 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 --2- 172.21.15.97:0/3770530251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd38015e470 0x7fd3800d4860 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 -- 172.21.15.97:0/3770530251 >> 172.21.15.97:0/3770530251 conn(0x7fd38009d240 msgr2=0x7fd3800d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 -- 172.21.15.97:0/3770530251 shutdown_connections 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.551+0000 7fd386200640 1 -- 172.21.15.97:0/3770530251 wait complete. 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.553+0000 7fd386200640 1 Processor -- start 2024-03-06T07:28:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.553+0000 7fd386200640 1 -- start start 2024-03-06T07:28:44.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd386200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 0x7fd3801f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd386200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd386200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd38015e470 0x7fd380154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd386200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800cfb00 con 0x7fd3800d2480 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd386200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800cf820 con 0x7fd38015e470 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd386200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800ce9f0 con 0x7fd3800d2860 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 0x7fd3801f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 0x7fd3801f7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41206/0 (socket says 172.21.15.97:41206) 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 -- 172.21.15.97:0/4050494624 learned_addr learned my addr 172.21.15.97:0/4050494624 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37fe00640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd38015e470 0x7fd380154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37ea00640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 -- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 msgr2=0x7fd3800cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 -- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd38015e470 msgr2=0x7fd380154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd38015e470 0x7fd380154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37f400640 1 -- 172.21.15.97:0/4050494624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3801f80b0 con 0x7fd3800d2480 2024-03-06T07:28:44.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.554+0000 7fd37fe00640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd38015e470 0x7fd380154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.555+0000 7fd37f400640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 0x7fd3801f7ad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c460 tx=0x7fd37000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.555+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370016020 con 0x7fd3800d2480 2024-03-06T07:28:44.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.555+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd370012870 con 0x7fd3800d2480 2024-03-06T07:28:44.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.555+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370012c60 con 0x7fd3800d2480 2024-03-06T07:28:44.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.555+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3801f8390 con 0x7fd3800d2480 2024-03-06T07:28:44.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.555+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3801f8950 con 0x7fd3800d2480 2024-03-06T07:28:44.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.556+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38004f330 con 0x7fd3800d2480 2024-03-06T07:28:44.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.557+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd370019020 con 0x7fd3800d2480 2024-03-06T07:28:44.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.558+0000 7fd377400640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35c06b4d0 0x7fd35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.558+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd370089ce0 con 0x7fd3800d2480 2024-03-06T07:28:44.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.558+0000 7fd37ea00640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35c06b4d0 0x7fd35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.559+0000 7fd37ea00640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35c06b4d0 0x7fd35c06d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd368036000 tx=0x7fd368008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.562+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd370020030 con 0x7fd3800d2480 2024-03-06T07:28:44.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.831+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fd3800d1490 con 0x7fd3800d2480 2024-03-06T07:28:44.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.836+0000 7fd377400640 1 -- 172.21.15.97:0/4050494624 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd370019310 con 0x7fd3800d2480 2024-03-06T07:28:44.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35c06b4d0 msgr2=0x7fd35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35c06b4d0 0x7fd35c06d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd368036000 tx=0x7fd368008030 comp rx=0 tx=0).stop 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 msgr2=0x7fd3801f7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 0x7fd3801f7ad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c460 tx=0x7fd37000c920 comp rx=0 tx=0).stop 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 shutdown_connections 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35c06b4d0 0x7fd35c06d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd38015e470 0x7fd380154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3800d2860 0x7fd3800cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 --2- 172.21.15.97:0/4050494624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3800d2480 0x7fd3801f7ad0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 >> 172.21.15.97:0/4050494624 conn(0x7fd38009d240 msgr2=0x7fd3800b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 shutdown_connections 2024-03-06T07:28:44.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:44.839+0000 7fd386200640 1 -- 172.21.15.97:0/4050494624 wait complete. 2024-03-06T07:28:44.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:28:45.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.053+0000 7f0504400640 1 Processor -- start 2024-03-06T07:28:45.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.053+0000 7f0504400640 1 -- start start 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0504400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4930 0x7f04fc0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0504400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04fc0c4d10 0x7f04fc0c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0504400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc145b30 con 0x7f04fc0c56b0 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0504400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04fc142ce0 con 0x7f04fc0c4d10 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0504400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0cbca0 con 0x7f04fc0c4930 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0501000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04fc0c4d10 0x7f04fc0c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0502400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f04fc0c56b0 0x7f04fc06d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54250/0 (socket says 172.21.15.97:54250) 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0502400640 1 -- 172.21.15.97:0/2425117246 learned_addr learned my addr 172.21.15.97:0/2425117246 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.055+0000 7f0501000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04fc0c4d10 0x7f04fc0c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41220/0 (socket says 172.21.15.97:41220) 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.056+0000 7f0501a00640 1 --2- 172.21.15.97:0/2425117246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4930 0x7f04fc0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.056+0000 7f0501000640 1 -- 172.21.15.97:0/2425117246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4930 msgr2=0x7f04fc0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.056+0000 7f0501000640 1 --2- 172.21.15.97:0/2425117246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4930 0x7f04fc0c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.056+0000 7f0501000640 1 -- 172.21.15.97:0/2425117246 >> v1:172.21.15.97:6789/0 conn(0x7f04fc0c56b0 legacy=0x7f04fc06d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.056+0000 7f0501000640 1 -- 172.21.15.97:0/2425117246 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0cbf80 con 0x7f04fc0c4d10 2024-03-06T07:28:45.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.057+0000 7f0501000640 1 --2- 172.21.15.97:0/2425117246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c4d10 0x7f04fc0c5170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f04e80097a0 tx=0x7f04e802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac52b5d6230211c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.057+0000 7f04fbe00640 1 -- 172.21.15.97:0/2425117246 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8031780 con 0x7f04fc0c4d10 2024-03-06T07:28:45.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.057+0000 7f04fbe00640 1 -- 172.21.15.97:0/2425117246 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e8037070 con 0x7f04fc0c4d10 2024-03-06T07:28:45.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.057+0000 7f04fbe00640 1 -- 172.21.15.97:0/2425117246 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8031b40 con 0x7f04fc0c4d10 2024-03-06T07:28:45.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 -- 172.21.15.97:0/2425117246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c4d10 msgr2=0x7f04fc0c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 --2- 172.21.15.97:0/2425117246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c4d10 0x7f04fc0c5170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f04e80097a0 tx=0x7f04e802b820 comp rx=0 tx=0).stop 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 -- 172.21.15.97:0/2425117246 shutdown_connections 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 --2- 172.21.15.97:0/2425117246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c4d10 0x7f04fc0c5170 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 --2- 172.21.15.97:0/2425117246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4930 0x7f04fc0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 -- 172.21.15.97:0/2425117246 >> 172.21.15.97:0/2425117246 conn(0x7f04fc0cc4f0 msgr2=0x7f04fc0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 -- 172.21.15.97:0/2425117246 shutdown_connections 2024-03-06T07:28:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.058+0000 7f0504400640 1 -- 172.21.15.97:0/2425117246 wait complete. 2024-03-06T07:28:45.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.060+0000 7f0504400640 1 Processor -- start 2024-03-06T07:28:45.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.060+0000 7f0504400640 1 -- start start 2024-03-06T07:28:45.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0504400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04fc0c4930 0x7f04fc0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0504400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4d10 0x7f04fc1f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0504400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 0x7f04fc1416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0504400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0cc260 con 0x7f04fc0c56b0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0504400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0ca3b0 con 0x7f04fc0c4930 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0504400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0c5b40 con 0x7f04fc0c4d10 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 0x7f04fc1416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 0x7f04fc1416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41230/0 (socket says 172.21.15.97:41230) 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 -- 172.21.15.97:0/1482798424 learned_addr learned my addr 172.21.15.97:0/1482798424 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0501000640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4d10 0x7f04fc1f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 -- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4d10 msgr2=0x7f04fc1f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4d10 0x7f04fc1f3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0501a00640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04fc0c4930 0x7f04fc0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 -- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04fc0c4930 msgr2=0x7f04fc0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.061+0000 7f0502400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04fc0c4930 0x7f04fc0c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f0502400640 1 -- 172.21.15.97:0/1482798424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc1f4460 con 0x7f04fc0c56b0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f0502400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 0x7f04fc1416f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c3c0 tx=0x7f04f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4017070 con 0x7f04fc0c56b0 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f4005ce0 con 0x7f04fc0c56b0 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40072d0 con 0x7f04fc0c56b0 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc1f46e0 con 0x7f04fc0c56b0 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.062+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc1f4ca0 con 0x7f04fc0c56b0 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.064+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc1f4ee0 con 0x7f04fc0c56b0 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.064+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04f401a070 con 0x7f04fc0c56b0 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.065+0000 7f04f9600640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04dc06b4d0 0x7f04dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.065+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f04f4088f20 con 0x7f04fc0c56b0 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.065+0000 7f0501a00640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04dc06b4d0 0x7f04dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.066+0000 7f0501a00640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04dc06b4d0 0x7f04dc06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0ce6b0 tx=0x7f04f000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.070+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04fc1f4ee0 con 0x7f04fc0c56b0 2024-03-06T07:28:45.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.316+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f04fc04f330 con 0x7f04fc0c56b0 2024-03-06T07:28:45.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.321+0000 7f04f9600640 1 -- 172.21.15.97:0/1482798424 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f04fc1f4460 con 0x7f04fc0c56b0 2024-03-06T07:28:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04dc06b4d0 msgr2=0x7f04dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04dc06b4d0 0x7f04dc06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0ce6b0 tx=0x7f04f000b040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 msgr2=0x7f04fc1416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 0x7f04fc1416f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c3c0 tx=0x7f04f400c880 comp rx=0 tx=0).stop 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 shutdown_connections 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04dc06b4d0 0x7f04dc06d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04fc0c56b0 0x7f04fc1416f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04fc0c4d10 0x7f04fc1f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 --2- 172.21.15.97:0/1482798424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04fc0c4930 0x7f04fc0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 >> 172.21.15.97:0/1482798424 conn(0x7f04fc0cc4f0 msgr2=0x7f04fc1e5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 shutdown_connections 2024-03-06T07:28:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.324+0000 7f0504400640 1 -- 172.21.15.97:0/1482798424 wait complete. 2024-03-06T07:28:45.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:28:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.544+0000 7f5e3b200640 1 Processor -- start 2024-03-06T07:28:45.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.544+0000 7f5e3b200640 1 -- start start 2024-03-06T07:28:45.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e3b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5e3c161580 0x7f5e3c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e3b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3c0d1e90 0x7f5e3c0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e3b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c161960 con 0x7f5e3c0d28a0 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e3b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c0d2d10 con 0x7f5e3c0d1e90 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e3b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c0cf6f0 con 0x7f5e3c161580 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e39400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3c0d1e90 0x7f5e3c0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e39400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3c0d1e90 0x7f5e3c0d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41232/0 (socket says 172.21.15.97:41232) 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e39400640 1 -- 172.21.15.97:0/2351280194 learned_addr learned my addr 172.21.15.97:0/2351280194 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.546+0000 7f5e39e00640 1 --2- 172.21.15.97:0/2351280194 >> v2:172.21.15.135:3300/0 conn(0x7f5e3c161580 0x7f5e3c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.547+0000 7f5e39400640 1 -- 172.21.15.97:0/2351280194 >> v2:172.21.15.135:3300/0 conn(0x7f5e3c161580 msgr2=0x7f5e3c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.547+0000 7f5e39400640 1 --2- 172.21.15.97:0/2351280194 >> v2:172.21.15.135:3300/0 conn(0x7f5e3c161580 0x7f5e3c0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.547+0000 7f5e39400640 1 -- 172.21.15.97:0/2351280194 >> v1:172.21.15.135:6789/0 conn(0x7f5e3c0d28a0 legacy=0x7f5e3c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.547+0000 7f5e39400640 1 -- 172.21.15.97:0/2351280194 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c0cf9d0 con 0x7f5e3c0d1e90 2024-03-06T07:28:45.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e39400640 1 --2- 172.21.15.97:0/2351280194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0d2360 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c073740 tx=0x7f5e2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32ef29cd1baba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e38a00640 1 -- 172.21.15.97:0/2351280194 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24035020 con 0x7f5e3c0d1e90 2024-03-06T07:28:45.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e38a00640 1 -- 172.21.15.97:0/2351280194 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e24031870 con 0x7f5e3c0d1e90 2024-03-06T07:28:45.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e38a00640 1 -- 172.21.15.97:0/2351280194 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24031c40 con 0x7f5e3c0d1e90 2024-03-06T07:28:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e3b200640 1 -- 172.21.15.97:0/2351280194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c0d1e90 msgr2=0x7f5e3c0d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e3b200640 1 --2- 172.21.15.97:0/2351280194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0d2360 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c073740 tx=0x7f5e2402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e3b200640 1 -- 172.21.15.97:0/2351280194 shutdown_connections 2024-03-06T07:28:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e3b200640 1 --2- 172.21.15.97:0/2351280194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0d2360 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.548+0000 7f5e3b200640 1 --2- 172.21.15.97:0/2351280194 >> v2:172.21.15.135:3300/0 conn(0x7f5e3c161580 0x7f5e3c0d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.549+0000 7f5e3b200640 1 -- 172.21.15.97:0/2351280194 >> 172.21.15.97:0/2351280194 conn(0x7f5e3c0d3d80 msgr2=0x7f5e3c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.549+0000 7f5e3b200640 1 -- 172.21.15.97:0/2351280194 shutdown_connections 2024-03-06T07:28:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.549+0000 7f5e3b200640 1 -- 172.21.15.97:0/2351280194 wait complete. 2024-03-06T07:28:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.550+0000 7f5e3b200640 1 Processor -- start 2024-03-06T07:28:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.550+0000 7f5e3b200640 1 -- start start 2024-03-06T07:28:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.551+0000 7f5e3b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.551+0000 7f5e3b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3c0d28a0 0x7f5e3c0b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.551+0000 7f5e3b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 0x7f5e3c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.551+0000 7f5e3b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0b39a0 con 0x7f5e3c161580 2024-03-06T07:28:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.551+0000 7f5e3b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0b36c0 con 0x7f5e3c0d1e90 2024-03-06T07:28:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.551+0000 7f5e3b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0cf3a0 con 0x7f5e3c0d28a0 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 0x7f5e3c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 0x7f5e3c0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41234/0 (socket says 172.21.15.97:41234) 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 -- 172.21.15.97:0/4251364199 learned_addr learned my addr 172.21.15.97:0/4251364199 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e39400640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3c0d28a0 0x7f5e3c0b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e39e00640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 -- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3c0d28a0 msgr2=0x7f5e3c0b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3c0d28a0 0x7f5e3c0b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 -- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e3c0d1e90 msgr2=0x7f5e3c0cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 -- 172.21.15.97:0/4251364199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c1f7770 con 0x7f5e3c161580 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.552+0000 7f5e3a800640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 0x7f5e3c0c8dc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400b4c0 tx=0x7f5e3400b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.553+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34015020 con 0x7f5e3c161580 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.553+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34011850 con 0x7f5e3c161580 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.553+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34011c40 con 0x7f5e3c161580 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.553+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c1f7a10 con 0x7f5e3c161580 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.553+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1f7fd0 con 0x7f5e3c161580 2024-03-06T07:28:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.554+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c04f330 con 0x7f5e3c161580 2024-03-06T07:28:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.555+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e34018020 con 0x7f5e3c161580 2024-03-06T07:28:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.556+0000 7f5e32000640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1806b500 0x7f5e1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.556+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e34089740 con 0x7f5e3c161580 2024-03-06T07:28:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.556+0000 7f5e39e00640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1806b500 0x7f5e1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.557+0000 7f5e39e00640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1806b500 0x7f5e1806d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0070b0 tx=0x7f5e2c007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.560+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e3401f070 con 0x7f5e3c161580 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.825+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f5e3c0ce090 con 0x7f5e3c161580 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.830+0000 7f5e32000640 1 -- 172.21.15.97:0/4251364199 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5e34018310 con 0x7f5e3c161580 2024-03-06T07:28:45.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.832+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1806b500 msgr2=0x7f5e1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.832+0000 7f5e3b200640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1806b500 0x7f5e1806d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0070b0 tx=0x7f5e2c007040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.832+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 msgr2=0x7f5e3c0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.832+0000 7f5e3b200640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 0x7f5e3c0c8dc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400b4c0 tx=0x7f5e3400b980 comp rx=0 tx=0).stop 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.832+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 shutdown_connections 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.832+0000 7f5e3b200640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1806b500 0x7f5e1806d9c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.833+0000 7f5e3b200640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3c161580 0x7f5e3c0c8dc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.833+0000 7f5e3b200640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3c0d28a0 0x7f5e3c0b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.833+0000 7f5e3b200640 1 --2- 172.21.15.97:0/4251364199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e3c0d1e90 0x7f5e3c0cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.833+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 >> 172.21.15.97:0/4251364199 conn(0x7f5e3c0d3d80 msgr2=0x7f5e3c0b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.833+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 shutdown_connections 2024-03-06T07:28:45.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:45.833+0000 7f5e3b200640 1 -- 172.21.15.97:0/4251364199 wait complete. 2024-03-06T07:28:45.853 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:28:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.046+0000 7f3b55e00640 1 Processor -- start 2024-03-06T07:28:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.046+0000 7f3b55e00640 1 -- start start 2024-03-06T07:28:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b55e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3b500b4390 0x7f3b500d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b55e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b500d0780 0x7f3b5015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b55e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5015c3a0 con 0x7f3b5015bfc0 2024-03-06T07:28:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b55e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b500cdc20 con 0x7f3b500d0780 2024-03-06T07:28:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b55e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b500cd660 con 0x7f3b500b4390 2024-03-06T07:28:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b4ea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3b5015bfc0 0x7f3b500b1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54264/0 (socket says 172.21.15.97:54264) 2024-03-06T07:28:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b4ea00640 1 -- 172.21.15.97:0/2384192836 learned_addr learned my addr 172.21.15.97:0/2384192836 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.048+0000 7f3b4f400640 1 --2- 172.21.15.97:0/2384192836 >> v2:172.21.15.97:3300/0 conn(0x7f3b500d0780 0x7f3b5015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b46000640 1 --2- 172.21.15.97:0/2384192836 >> v2:172.21.15.135:3300/0 conn(0x7f3b500b4390 0x7f3b500d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b4f400640 1 -- 172.21.15.97:0/2384192836 >> v2:172.21.15.135:3300/0 conn(0x7f3b500b4390 msgr2=0x7f3b500d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b4f400640 1 --2- 172.21.15.97:0/2384192836 >> v2:172.21.15.135:3300/0 conn(0x7f3b500b4390 0x7f3b500d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b4f400640 1 -- 172.21.15.97:0/2384192836 >> v1:172.21.15.97:6789/0 conn(0x7f3b5015bfc0 legacy=0x7f3b500b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b4f400640 1 -- 172.21.15.97:0/2384192836 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500cd940 con 0x7f3b500d0780 2024-03-06T07:28:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b4e000640 1 -- 172.21.15.97:0/2384192836 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001250156 0 0) 0x7f3b5015c3a0 con 0x7f3b5015bfc0 2024-03-06T07:28:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.049+0000 7f3b46000640 1 --2- 172.21.15.97:0/2384192836 >> v2:172.21.15.135:3300/0 conn(0x7f3b500b4390 0x7f3b500d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.050+0000 7f3b4f400640 1 --2- 172.21.15.97:0/2384192836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b500d0780 0x7f3b5015fab0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3b50072cc0 tx=0x7f3b4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31fc3af4854c848 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.050+0000 7f3b4e000640 1 -- 172.21.15.97:0/2384192836 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48039070 con 0x7f3b500d0780 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.050+0000 7f3b4e000640 1 -- 172.21.15.97:0/2384192836 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4802cc80 con 0x7f3b500d0780 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.050+0000 7f3b4e000640 1 -- 172.21.15.97:0/2384192836 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48034660 con 0x7f3b500d0780 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 -- 172.21.15.97:0/2384192836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b500d0780 msgr2=0x7f3b5015fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 --2- 172.21.15.97:0/2384192836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b500d0780 0x7f3b5015fab0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3b50072cc0 tx=0x7f3b4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 -- 172.21.15.97:0/2384192836 shutdown_connections 2024-03-06T07:28:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 --2- 172.21.15.97:0/2384192836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b500d0780 0x7f3b5015fab0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 --2- 172.21.15.97:0/2384192836 >> v2:172.21.15.135:3300/0 conn(0x7f3b500b4390 0x7f3b500d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 -- 172.21.15.97:0/2384192836 >> 172.21.15.97:0/2384192836 conn(0x7f3b500b3800 msgr2=0x7f3b500b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 -- 172.21.15.97:0/2384192836 shutdown_connections 2024-03-06T07:28:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.051+0000 7f3b55e00640 1 -- 172.21.15.97:0/2384192836 wait complete. 2024-03-06T07:28:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.053+0000 7f3b55e00640 1 Processor -- start 2024-03-06T07:28:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.053+0000 7f3b55e00640 1 -- start start 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.054+0000 7f3b55e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b500b4390 0x7f3b500b7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b55e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b500d0780 0x7f3b500c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b55e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 0x7f3b500cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b55e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500b1d80 con 0x7f3b5015bfc0 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b55e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500cfd20 con 0x7f3b500d0780 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b55e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500d0c30 con 0x7f3b500b4390 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 0x7f3b500cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 0x7f3b500cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41248/0 (socket says 172.21.15.97:41248) 2024-03-06T07:28:46.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 -- 172.21.15.97:0/3241968813 learned_addr learned my addr 172.21.15.97:0/3241968813 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:46.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4ea00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b500b4390 0x7f3b500b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b46000640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b500d0780 0x7f3b500c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 -- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b500b4390 msgr2=0x7f3b500b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b500b4390 0x7f3b500b7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 -- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b500d0780 msgr2=0x7f3b500c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b500d0780 0x7f3b500c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.055+0000 7f3b4f400640 1 -- 172.21.15.97:0/3241968813 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5009fa40 con 0x7f3b5015bfc0 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.056+0000 7f3b4f400640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 0x7f3b500cb3b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009e80 tx=0x7f3b48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.056+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48039050 con 0x7f3b5015bfc0 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.056+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b48045070 con 0x7f3b5015bfc0 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.056+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4803e830 con 0x7f3b5015bfc0 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.056+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b500c8660 con 0x7f3b5015bfc0 2024-03-06T07:28:46.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.056+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b500c9be0 con 0x7f3b5015bfc0 2024-03-06T07:28:46.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.057+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5004f330 con 0x7f3b5015bfc0 2024-03-06T07:28:46.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.058+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b4803e9d0 con 0x7f3b5015bfc0 2024-03-06T07:28:46.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.059+0000 7f3b47e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2c06b500 0x7f3b2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.059+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b480acf20 con 0x7f3b5015bfc0 2024-03-06T07:28:46.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.059+0000 7f3b4ea00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2c06b500 0x7f3b2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.060+0000 7f3b4ea00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2c06b500 0x7f3b2c06d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3b40000f60 tx=0x7f3b40007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.064+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b5009fa40 con 0x7f3b5015bfc0 2024-03-06T07:28:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.311+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f3b500c6be0 con 0x7f3b5015bfc0 2024-03-06T07:28:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.315+0000 7f3b47e00640 1 -- 172.21.15.97:0/3241968813 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3b4807c170 con 0x7f3b5015bfc0 2024-03-06T07:28:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2c06b500 msgr2=0x7f3b2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2c06b500 0x7f3b2c06d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3b40000f60 tx=0x7f3b40007b50 comp rx=0 tx=0).stop 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 msgr2=0x7f3b500cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 0x7f3b500cb3b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009e80 tx=0x7f3b48004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 shutdown_connections 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2c06b500 0x7f3b2c06d9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b5015bfc0 0x7f3b500cb3b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b500d0780 0x7f3b500c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 --2- 172.21.15.97:0/3241968813 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b500b4390 0x7f3b500b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 >> 172.21.15.97:0/3241968813 conn(0x7f3b500b3800 msgr2=0x7f3b5006f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 shutdown_connections 2024-03-06T07:28:46.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.318+0000 7f3b55e00640 1 -- 172.21.15.97:0/3241968813 wait complete. 2024-03-06T07:28:46.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:28:46.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.533+0000 7f2cac400640 1 Processor -- start 2024-03-06T07:28:46.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.533+0000 7f2cac400640 1 -- start start 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.534+0000 7f2cac400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2ca40c6460 0x7f2ca40c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.534+0000 7f2cac400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2ca40c5960 0x7f2ca4165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.534+0000 7f2cac400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca4145b30 con 0x7f2ca40c4930 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.534+0000 7f2cac400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca40c68c0 con 0x7f2ca40c6460 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.534+0000 7f2cac400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca4142ce0 con 0x7f2ca40c5960 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2ca40c6460 0x7f2ca40c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2ca40c6460 0x7f2ca40c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41264/0 (socket says 172.21.15.97:41264) 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 -- 172.21.15.97:0/2244064312 learned_addr learned my addr 172.21.15.97:0/2244064312 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2caa400640 1 --2- 172.21.15.97:0/2244064312 >> v2:172.21.15.135:3300/0 conn(0x7f2ca40c5960 0x7f2ca4165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 -- 172.21.15.97:0/2244064312 >> v2:172.21.15.135:3300/0 conn(0x7f2ca40c5960 msgr2=0x7f2ca4165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 --2- 172.21.15.97:0/2244064312 >> v2:172.21.15.135:3300/0 conn(0x7f2ca40c5960 0x7f2ca4165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 -- 172.21.15.97:0/2244064312 >> v1:172.21.15.135:6789/0 conn(0x7f2ca40c4930 legacy=0x7f2ca40c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2ca9000640 1 -- 172.21.15.97:0/2244064312 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40cbca0 con 0x7f2ca40c6460 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.535+0000 7f2caa400640 1 --2- 172.21.15.97:0/2244064312 >> v2:172.21.15.135:3300/0 conn(0x7f2ca40c5960 0x7f2ca4165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.536+0000 7f2ca9000640 1 --2- 172.21.15.97:0/2244064312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c6460 0x7f2ca40c5420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4073310 tx=0x7f2c9002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88a2b212c05b8bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.536+0000 7f2ca3e00640 1 -- 172.21.15.97:0/2244064312 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9002e070 con 0x7f2ca40c6460 2024-03-06T07:28:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2ca3e00640 1 -- 172.21.15.97:0/2244064312 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c90037070 con 0x7f2ca40c6460 2024-03-06T07:28:46.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2ca3e00640 1 -- 172.21.15.97:0/2244064312 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90031a00 con 0x7f2ca40c6460 2024-03-06T07:28:46.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 -- 172.21.15.97:0/2244064312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c6460 msgr2=0x7f2ca40c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 --2- 172.21.15.97:0/2244064312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c6460 0x7f2ca40c5420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4073310 tx=0x7f2c9002b880 comp rx=0 tx=0).stop 2024-03-06T07:28:46.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 -- 172.21.15.97:0/2244064312 shutdown_connections 2024-03-06T07:28:46.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 --2- 172.21.15.97:0/2244064312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c6460 0x7f2ca40c5420 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 --2- 172.21.15.97:0/2244064312 >> v2:172.21.15.135:3300/0 conn(0x7f2ca40c5960 0x7f2ca4165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 -- 172.21.15.97:0/2244064312 >> 172.21.15.97:0/2244064312 conn(0x7f2ca40cc4f0 msgr2=0x7f2ca40cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 -- 172.21.15.97:0/2244064312 shutdown_connections 2024-03-06T07:28:46.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.537+0000 7f2cac400640 1 -- 172.21.15.97:0/2244064312 wait complete. 2024-03-06T07:28:46.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.539+0000 7f2cac400640 1 Processor -- start 2024-03-06T07:28:46.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.539+0000 7f2cac400640 1 -- start start 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2cac400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 0x7f2ca41f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2cac400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ca40c5960 0x7f2ca40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2cac400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ca40c6460 0x7f2ca4141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2cac400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40ca3b0 con 0x7f2ca40c4930 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2cac400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40ca0d0 con 0x7f2ca40c5960 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2cac400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca406e600 con 0x7f2ca40c6460 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2ca9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 0x7f2ca41f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2ca9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 0x7f2ca41f6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41280/0 (socket says 172.21.15.97:41280) 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.540+0000 7f2ca9a00640 1 -- 172.21.15.97:0/2016663997 learned_addr learned my addr 172.21.15.97:0/2016663997 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca9a00640 1 -- 172.21.15.97:0/2016663997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ca40c6460 msgr2=0x7f2ca4141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca9a00640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ca40c6460 0x7f2ca4141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca9a00640 1 -- 172.21.15.97:0/2016663997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ca40c5960 msgr2=0x7f2ca40c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca9a00640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ca40c5960 0x7f2ca40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca9a00640 1 -- 172.21.15.97:0/2016663997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40c7ea0 con 0x7f2ca40c4930 2024-03-06T07:28:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca9a00640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 0x7f2ca41f6920 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800b3e0 tx=0x7f2c9800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98015020 con 0x7f2ca40c4930 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98011850 con 0x7f2ca40c4930 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98011c40 con 0x7f2ca40c4930 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca40c8180 con 0x7f2ca40c4930 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.541+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca40c8740 con 0x7f2ca40c4930 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.543+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca404f330 con 0x7f2ca40c4930 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.544+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c98018020 con 0x7f2ca40c4930 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.544+0000 7f2ca1600640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c8406b4d0 0x7f2c8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.545+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2c98089ea0 con 0x7f2ca40c4930 2024-03-06T07:28:46.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.545+0000 7f2ca9000640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c8406b4d0 0x7f2c8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.545+0000 7f2ca9000640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c8406b4d0 0x7f2c8406d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c90036000 tx=0x7f2c90030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.549+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c9801f020 con 0x7f2ca40c4930 2024-03-06T07:28:46.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.809+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f2ca40ae000 con 0x7f2ca40c4930 2024-03-06T07:28:46.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.813+0000 7f2ca1600640 1 -- 172.21.15.97:0/2016663997 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2c980172c0 con 0x7f2ca40c4930 2024-03-06T07:28:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c8406b4d0 msgr2=0x7f2c8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c8406b4d0 0x7f2c8406d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c90036000 tx=0x7f2c90030000 comp rx=0 tx=0).stop 2024-03-06T07:28:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 msgr2=0x7f2ca41f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 0x7f2ca41f6920 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800b3e0 tx=0x7f2c9800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 shutdown_connections 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c8406b4d0 0x7f2c8406d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2ca40c6460 0x7f2ca4141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ca40c5960 0x7f2ca40c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 --2- 172.21.15.97:0/2016663997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ca40c4930 0x7f2ca41f6920 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 >> 172.21.15.97:0/2016663997 conn(0x7f2ca40cc4f0 msgr2=0x7f2ca41447d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 shutdown_connections 2024-03-06T07:28:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:46.816+0000 7f2cac400640 1 -- 172.21.15.97:0/2016663997 wait complete. 2024-03-06T07:28:46.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.028+0000 7f2298e00640 1 Processor -- start 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.029+0000 7f2298e00640 1 -- start start 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.030+0000 7f2298e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2294164b30 0x7f2294166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.030+0000 7f2298e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294145b30 con 0x7f22940c6460 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.030+0000 7f2298e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294142ce0 con 0x7f22940c4930 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.030+0000 7f2298e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22940c68c0 con 0x7f2294164b30 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.030+0000 7f2289400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f22940c6460 0x7f22940c5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54270/0 (socket says 172.21.15.97:54270) 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.031+0000 7f2289400640 1 -- 172.21.15.97:0/2956748894 learned_addr learned my addr 172.21.15.97:0/2956748894 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.031+0000 7f2292800640 1 --2- 172.21.15.97:0/2956748894 >> v2:172.21.15.97:3300/0 conn(0x7f2294164b30 0x7f2294166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.031+0000 7f2292800640 1 -- 172.21.15.97:0/2956748894 >> v1:172.21.15.135:6789/0 conn(0x7f22940c4930 legacy=0x7f22940c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.031+0000 7f2291400640 1 -- 172.21.15.97:0/2956748894 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232880242 0 0) 0x7f227c001110 con 0x7f22940c6460 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.031+0000 7f2292800640 1 -- 172.21.15.97:0/2956748894 >> v1:172.21.15.97:6789/0 conn(0x7f22940c6460 legacy=0x7f22940c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.031+0000 7f2292800640 1 -- 172.21.15.97:0/2956748894 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22940cbca0 con 0x7f2294164b30 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.032+0000 7f2292800640 1 --2- 172.21.15.97:0/2956748894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2294164b30 0x7f2294166f20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f228c009d90 tx=0x7f228c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e4888a51f1d1c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.032+0000 7f2291400640 1 -- 172.21.15.97:0/2956748894 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c039070 con 0x7f2294164b30 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.032+0000 7f2291400640 1 -- 172.21.15.97:0/2956748894 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c02cbd0 con 0x7f2294164b30 2024-03-06T07:28:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.032+0000 7f2291400640 1 -- 172.21.15.97:0/2956748894 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c034440 con 0x7f2294164b30 2024-03-06T07:28:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 -- 172.21.15.97:0/2956748894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2294164b30 msgr2=0x7f2294166f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 --2- 172.21.15.97:0/2956748894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2294164b30 0x7f2294166f20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f228c009d90 tx=0x7f228c02be20 comp rx=0 tx=0).stop 2024-03-06T07:28:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 -- 172.21.15.97:0/2956748894 shutdown_connections 2024-03-06T07:28:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 --2- 172.21.15.97:0/2956748894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2294164b30 0x7f2294166f20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 -- 172.21.15.97:0/2956748894 >> 172.21.15.97:0/2956748894 conn(0x7f22940cc4f0 msgr2=0x7f22940cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 -- 172.21.15.97:0/2956748894 shutdown_connections 2024-03-06T07:28:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.033+0000 7f2298e00640 1 -- 172.21.15.97:0/2956748894 wait complete. 2024-03-06T07:28:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.035+0000 7f2298e00640 1 Processor -- start 2024-03-06T07:28:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.035+0000 7f2298e00640 1 -- start start 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2298e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22940c4930 0x7f229413e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2298e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 0x7f229413e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2298e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2294164b30 0x7f229413b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2298e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940ca3b0 con 0x7f22940c6460 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2298e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940ca0d0 con 0x7f22940c4930 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2298e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229406e600 con 0x7f2294164b30 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 0x7f229413e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 0x7f229413e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41304/0 (socket says 172.21.15.97:41304) 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 -- 172.21.15.97:0/2962593398 learned_addr learned my addr 172.21.15.97:0/2962593398 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2291e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22940c4930 0x7f229413e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 -- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2294164b30 msgr2=0x7f229413b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2294164b30 0x7f229413b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 -- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22940c4930 msgr2=0x7f229413e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22940c4930 0x7f229413e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.036+0000 7f2289400640 1 -- 172.21.15.97:0/2962593398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229406d2f0 con 0x7f22940c6460 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.037+0000 7f2289400640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 0x7f229413e6b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f227c00c430 tx=0x7f227c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.037+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0074e0 con 0x7f22940c6460 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.037+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0047e0 con 0x7f22940c6460 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.037+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2294164700 con 0x7f22940c6460 2024-03-06T07:28:47.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.037+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c004bd0 con 0x7f22940c6460 2024-03-06T07:28:47.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.037+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2294163aa0 con 0x7f22940c6460 2024-03-06T07:28:47.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.038+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229404f330 con 0x7f22940c6460 2024-03-06T07:28:47.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.039+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f227c011070 con 0x7f22940c6460 2024-03-06T07:28:47.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.040+0000 7f228aa00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f227006b4d0 0x7f227006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.041+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f227c089ae0 con 0x7f22940c6460 2024-03-06T07:28:47.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.041+0000 7f2291e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f227006b4d0 0x7f227006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.041+0000 7f2291e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f227006b4d0 0x7f227006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2284006fd0 tx=0x7f2284008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.045+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2294145b30 con 0x7f22940c6460 2024-03-06T07:28:47.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.306+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f22941621f0 con 0x7f22940c6460 2024-03-06T07:28:47.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.311+0000 7f228aa00640 1 -- 172.21.15.97:0/2962593398 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f227c058d40 con 0x7f22940c6460 2024-03-06T07:28:47.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f227006b4d0 msgr2=0x7f227006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f227006b4d0 0x7f227006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2284006fd0 tx=0x7f2284008040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 msgr2=0x7f229413e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 0x7f229413e6b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f227c00c430 tx=0x7f227c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 shutdown_connections 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f227006b4d0 0x7f227006d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2294164b30 0x7f229413b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22940c6460 0x7f229413e6b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 --2- 172.21.15.97:0/2962593398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22940c4930 0x7f229413e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 >> 172.21.15.97:0/2962593398 conn(0x7f22940cc4f0 msgr2=0x7f2294162710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 shutdown_connections 2024-03-06T07:28:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.314+0000 7f2298e00640 1 -- 172.21.15.97:0/2962593398 wait complete. 2024-03-06T07:28:47.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.533+0000 7f072d000640 1 Processor -- start 2024-03-06T07:28:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.533+0000 7f072d000640 1 -- start start 2024-03-06T07:28:47.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.534+0000 7f072d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f072806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.534+0000 7f072d000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072806ee70 con 0x7f072815fe30 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f072d000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072806cef0 con 0x7f072815f9d0 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f072d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072806d1d0 con 0x7f0728144c00 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f0726800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f072815fe30 0x7f07280ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54272/0 (socket says 172.21.15.97:54272) 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f0726800640 1 -- 172.21.15.97:0/2535062269 learned_addr learned my addr 172.21.15.97:0/2535062269 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373013608 0 0) 0x7f0720001490 con 0x7f072815fe30 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c003520 con 0x7f072815fe30 2024-03-06T07:28:47.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.535+0000 7f0725e00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f072806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1143716222 0 0) 0x7f070c003520 con 0x7f072815fe30 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 msgr2=0x7f072806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f072806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 >> v1:172.21.15.135:6789/0 conn(0x7f072815f9d0 legacy=0x7f07280ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072806d4b0 con 0x7f072815fe30 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0720001bc0 con 0x7f072815fe30 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f072806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 >> v1:172.21.15.97:6789/0 conn(0x7f072815fe30 legacy=0x7f07280ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f070c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f070c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f070c004d40 0x7f070c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072806d1d0 con 0x7f070c0039e0 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c00bad0 con 0x7f070c004d40 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070c00bcd0 con 0x7f0728144c00 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f070c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f071fe00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f070c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 -- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 msgr2=0x7f070c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728144c00 0x7f070c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 -- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f070c004d40 msgr2=0x7f070c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f070c004d40 0x7f070c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.536+0000 7f0725e00640 1 -- 172.21.15.97:0/2535062269 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c003520 con 0x7f070c0039e0 2024-03-06T07:28:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f0725e00640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f070c003e00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0718006fd0 tx=0x7f070c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b868268099daa6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071800f040 con 0x7f070c0039e0 2024-03-06T07:28:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071802e070 con 0x7f070c0039e0 2024-03-06T07:28:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f0725400640 1 -- 172.21.15.97:0/2535062269 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718004450 con 0x7f070c0039e0 2024-03-06T07:28:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f072d000640 1 -- 172.21.15.97:0/2535062269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 msgr2=0x7f070c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f072d000640 1 --2- 172.21.15.97:0/2535062269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f070c003e00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0718006fd0 tx=0x7f070c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f0726800640 1 -- 172.21.15.97:0/2535062269 reap_dead start 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f072d000640 1 -- 172.21.15.97:0/2535062269 shutdown_connections 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f072d000640 1 -- 172.21.15.97:0/2535062269 >> 172.21.15.97:0/2535062269 conn(0x7f07280b1730 msgr2=0x7f07280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.538+0000 7f072d000640 1 -- 172.21.15.97:0/2535062269 shutdown_connections 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.539+0000 7f072d000640 1 -- 172.21.15.97:0/2535062269 wait complete. 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.540+0000 7f072d000640 1 Processor -- start 2024-03-06T07:28:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.540+0000 7f072d000640 1 -- start start 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f072d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f07281f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f072d000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07281f3f60 0x7f0728142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f072d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728143320 0x7f07281f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f072d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0728160730 con 0x7f070c0039e0 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f072d000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0728160280 con 0x7f07281f3f60 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f072d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07280ceec0 con 0x7f0728143320 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f0725e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f07281f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f0725e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f07281f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41326/0 (socket says 172.21.15.97:41326) 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.541+0000 7f0725e00640 1 -- 172.21.15.97:0/1041428917 learned_addr learned my addr 172.21.15.97:0/1041428917 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f071fe00640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07281f3f60 0x7f0728142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0726800640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728143320 0x7f07281f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0725e00640 1 -- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728143320 msgr2=0x7f07281f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0725e00640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728143320 0x7f07281f45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0725e00640 1 -- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07281f3f60 msgr2=0x7f0728142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0725e00640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07281f3f60 0x7f0728142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0725e00640 1 -- 172.21.15.97:0/1041428917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072814c720 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.542+0000 7f0725e00640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f07281f3a20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f07180098d0 tx=0x7f0718004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.543+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718032680 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.543+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0718032820 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.543+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718032c10 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.543+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072814ca00 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.543+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072814cfc0 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.544+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072804f330 con 0x7f070c0039e0 2024-03-06T07:28:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.545+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f071803e430 con 0x7f070c0039e0 2024-03-06T07:28:47.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.546+0000 7f071e000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f070406b500 0x7f070406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.546+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f07180ac3c0 con 0x7f070c0039e0 2024-03-06T07:28:47.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.546+0000 7f071fe00640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f070406b500 0x7f070406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.547+0000 7f071fe00640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f070406b500 0x7f070406d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0710007fd0 tx=0x7f0710009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.551+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f071804d030 con 0x7f070c0039e0 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.747+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0728143700 con 0x7f070c0039e0 2024-03-06T07:28:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.748+0000 7f071e000640 1 -- 172.21.15.97:0/1041428917 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f0718042030 con 0x7f070c0039e0 2024-03-06T07:28:47.750 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:47.750 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":31,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:43.817534+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f070406b500 msgr2=0x7f070406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f070406b500 0x7f070406d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0710007fd0 tx=0x7f0710009040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 msgr2=0x7f07281f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f07281f3a20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f07180098d0 tx=0x7f0718004730 comp rx=0 tx=0).stop 2024-03-06T07:28:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 shutdown_connections 2024-03-06T07:28:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f070406b500 0x7f070406d9c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0728143320 0x7f07281f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07281f3f60 0x7f0728142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 --2- 172.21.15.97:0/1041428917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f070c0039e0 0x7f07281f3a20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 >> 172.21.15.97:0/1041428917 conn(0x7f07280b1730 msgr2=0x7f07280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 shutdown_connections 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.751+0000 7f072d000640 1 -- 172.21.15.97:0/1041428917 wait complete. 2024-03-06T07:28:47.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:47.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:47.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.983+0000 7f763c800640 1 Processor -- start 2024-03-06T07:28:47.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.983+0000 7f763c800640 1 -- start start 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f763c800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f76340b3f70 0x7f7634160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f763c800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7634070ff0 0x7f76340ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f763c800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7634071440 con 0x7f763415fc10 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f763c800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76340b46c0 con 0x7f7634070ff0 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f763c800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76340b49a0 con 0x7f76340b3f70 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f7639e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f763415fc10 0x7f763415fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54276/0 (socket says 172.21.15.97:54276) 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f7639e00640 1 -- 172.21.15.97:0/2937875206 learned_addr learned my addr 172.21.15.97:0/2937875206 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f763a800640 1 --2- 172.21.15.97:0/2937875206 >> v2:172.21.15.97:3300/0 conn(0x7f7634070ff0 0x7f76340ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.985+0000 7f7638a00640 1 -- 172.21.15.97:0/2937875206 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442245285 0 0) 0x7f7634071440 con 0x7f763415fc10 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.986+0000 7f7638a00640 1 -- 172.21.15.97:0/2937875206 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c003520 con 0x7f763415fc10 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.986+0000 7f7639400640 1 --2- 172.21.15.97:0/2937875206 >> v2:172.21.15.135:3300/0 conn(0x7f76340b3f70 0x7f7634160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.986+0000 7f763a800640 1 -- 172.21.15.97:0/2937875206 >> v2:172.21.15.135:3300/0 conn(0x7f76340b3f70 msgr2=0x7f7634160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.986+0000 7f763a800640 1 --2- 172.21.15.97:0/2937875206 >> v2:172.21.15.135:3300/0 conn(0x7f76340b3f70 0x7f7634160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.986+0000 7f763a800640 1 -- 172.21.15.97:0/2937875206 >> v1:172.21.15.97:6789/0 conn(0x7f763415fc10 legacy=0x7f763415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.986+0000 7f763a800640 1 -- 172.21.15.97:0/2937875206 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7634072c90 con 0x7f7634070ff0 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.987+0000 7f763a800640 1 --2- 172.21.15.97:0/2937875206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7634070ff0 0x7f76340ce4a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7634074a50 tx=0x7f763002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7ab51b7c27335a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.987+0000 7f7638a00640 1 -- 172.21.15.97:0/2937875206 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630039070 con 0x7f7634070ff0 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.987+0000 7f7638a00640 1 -- 172.21.15.97:0/2937875206 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763002cc80 con 0x7f7634070ff0 2024-03-06T07:28:47.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.987+0000 7f7638a00640 1 -- 172.21.15.97:0/2937875206 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630034660 con 0x7f7634070ff0 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 -- 172.21.15.97:0/2937875206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7634070ff0 msgr2=0x7f76340ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 --2- 172.21.15.97:0/2937875206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7634070ff0 0x7f76340ce4a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7634074a50 tx=0x7f763002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 -- 172.21.15.97:0/2937875206 shutdown_connections 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 --2- 172.21.15.97:0/2937875206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7634070ff0 0x7f76340ce4a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 --2- 172.21.15.97:0/2937875206 >> v2:172.21.15.135:3300/0 conn(0x7f76340b3f70 0x7f7634160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 -- 172.21.15.97:0/2937875206 >> 172.21.15.97:0/2937875206 conn(0x7f76340b1940 msgr2=0x7f76340b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 -- 172.21.15.97:0/2937875206 shutdown_connections 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.988+0000 7f763c800640 1 -- 172.21.15.97:0/2937875206 wait complete. 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.990+0000 7f763c800640 1 Processor -- start 2024-03-06T07:28:47.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.990+0000 7f763c800640 1 -- start start 2024-03-06T07:28:47.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f763c800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7634070ff0 0x7f763414a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f763c800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 0x7f763406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f763c800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763415fc10 0x7f763406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f763c800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634072a30 con 0x7f76340b3f70 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f763c800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76340728b0 con 0x7f763415fc10 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f763c800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763415c2e0 con 0x7f7634070ff0 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f7639400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 0x7f763406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f7639400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 0x7f763406d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41342/0 (socket says 172.21.15.97:41342) 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.991+0000 7f7639400640 1 -- 172.21.15.97:0/615836186 learned_addr learned my addr 172.21.15.97:0/615836186 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f763a800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763415fc10 0x7f763406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639e00640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7634070ff0 0x7f763414a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639400640 1 -- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7634070ff0 msgr2=0x7f763414a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639400640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7634070ff0 0x7f763414a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639400640 1 -- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763415fc10 msgr2=0x7f763406df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639400640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763415fc10 0x7f763406df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639400640 1 -- 172.21.15.97:0/615836186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763414a920 con 0x7f76340b3f70 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f7639400640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 0x7f763406d9e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f762800d7d0 tx=0x7f762800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762800b6c0 con 0x7f76340b3f70 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7628019070 con 0x7f76340b3f70 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762800ba80 con 0x7f76340b3f70 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763409ab20 con 0x7f76340b3f70 2024-03-06T07:28:47.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.992+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763409b0e0 con 0x7f76340b3f70 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.994+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763404f330 con 0x7f76340b3f70 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.994+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f762800bc20 con 0x7f76340b3f70 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.995+0000 7f762e000640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f761406b4d0 0x7f761406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.995+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f762808a260 con 0x7f76340b3f70 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.996+0000 7f7639e00640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f761406b4d0 0x7f761406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:47.996+0000 7f7639e00640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f761406b4d0 0x7f761406d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7620000f60 tx=0x7f7620009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.000+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76280594c0 con 0x7f76340b3f70 2024-03-06T07:28:48.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.195+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76340b2d40 con 0x7f76340b3f70 2024-03-06T07:28:48.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.196+0000 7f762e000640 1 -- 172.21.15.97:0/615836186 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7628058dc0 con 0x7f76340b3f70 2024-03-06T07:28:48.199 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:28:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f761406b4d0 msgr2=0x7f761406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f761406b4d0 0x7f761406d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7620000f60 tx=0x7f7620009040 comp rx=0 tx=0).stop 2024-03-06T07:28:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 msgr2=0x7f763406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 0x7f763406d9e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f762800d7d0 tx=0x7f762800dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 shutdown_connections 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f761406b4d0 0x7f761406d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763415fc10 0x7f763406df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76340b3f70 0x7f763406d9e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 --2- 172.21.15.97:0/615836186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7634070ff0 0x7f763414a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 >> 172.21.15.97:0/615836186 conn(0x7f76340b1940 msgr2=0x7f76340af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 shutdown_connections 2024-03-06T07:28:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.199+0000 7f763c800640 1 -- 172.21.15.97:0/615836186 wait complete. 2024-03-06T07:28:48.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:48.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.428+0000 7fc2fbc00640 1 Processor -- start 2024-03-06T07:28:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.428+0000 7fc2fbc00640 1 -- start start 2024-03-06T07:28:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.430+0000 7fc2fbc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2f4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.430+0000 7fc2fbc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc2f406e4d0 0x7fc2f406e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.430+0000 7fc2fbc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f4071d30 con 0x7fc2f40af1c0 2024-03-06T07:28:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.430+0000 7fc2fbc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f4071260 con 0x7fc2f406e4d0 2024-03-06T07:28:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.430+0000 7fc2fbc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f415f470 con 0x7fc2f415eff0 2024-03-06T07:28:48.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.431+0000 7fc2f9200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc2f40af1c0 0x7fc2f406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54290/0 (socket says 172.21.15.97:54290) 2024-03-06T07:28:48.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.431+0000 7fc2f9200640 1 -- 172.21.15.97:0/3638639142 learned_addr learned my addr 172.21.15.97:0/3638639142 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.431+0000 7fc2f9c00640 1 --2- 172.21.15.97:0/3638639142 >> v2:172.21.15.135:3300/0 conn(0x7fc2f406e4d0 0x7fc2f406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.431+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2f4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.431+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303087180 0 0) 0x7fc2e4001110 con 0x7fc2f40af1c0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.431+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d8003520 con 0x7fc2f40af1c0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218149803 0 0) 0x7fc2d8003520 con 0x7fc2f40af1c0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 msgr2=0x7fc2f4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2f4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 >> v2:172.21.15.135:3300/0 conn(0x7fc2f406e4d0 msgr2=0x7fc2f406e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 --2- 172.21.15.97:0/3638639142 >> v2:172.21.15.135:3300/0 conn(0x7fc2f406e4d0 0x7fc2f406e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f4071490 con 0x7fc2f40af1c0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc2e4001840 con 0x7fc2f40af1c0 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2f4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 >> v1:172.21.15.97:6789/0 conn(0x7fc2f40af1c0 legacy=0x7fc2f406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004d40 0x7fc2d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f415f470 con 0x7fc2d80039b0 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d800b9c0 con 0x7fc2d8004d40 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d800bb40 con 0x7fc2f415eff0 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9200640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9c00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004d40 0x7fc2d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9200640 1 -- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 msgr2=0x7fc2d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9200640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9200640 1 -- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004d40 msgr2=0x7fc2d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9200640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004d40 0x7fc2d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.432+0000 7fc2f9200640 1 -- 172.21.15.97:0/3638639142 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8003520 con 0x7fc2d80039b0 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.433+0000 7fc2f9c00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004d40 0x7fc2d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.433+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f415eff0 0x7fc2d8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2f9200640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2d8003dd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c3e0 tx=0x7fc2d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a155227f3359a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e400c0e0 con 0x7fc2d80039b0 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e4030070 con 0x7fc2d80039b0 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2f3e00640 1 -- 172.21.15.97:0/3638639142 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e40058d0 con 0x7fc2d80039b0 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2fbc00640 1 -- 172.21.15.97:0/3638639142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 msgr2=0x7fc2d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/3638639142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2d8003dd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c3e0 tx=0x7fc2d8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.434+0000 7fc2f9c00640 1 -- 172.21.15.97:0/3638639142 reap_dead start 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.435+0000 7fc2fbc00640 1 -- 172.21.15.97:0/3638639142 shutdown_connections 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.435+0000 7fc2fbc00640 1 -- 172.21.15.97:0/3638639142 >> 172.21.15.97:0/3638639142 conn(0x7fc2f40b1730 msgr2=0x7fc2f40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.435+0000 7fc2fbc00640 1 -- 172.21.15.97:0/3638639142 shutdown_connections 2024-03-06T07:28:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.435+0000 7fc2fbc00640 1 -- 172.21.15.97:0/3638639142 wait complete. 2024-03-06T07:28:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.436+0000 7fc2fbc00640 1 Processor -- start 2024-03-06T07:28:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 -- start start 2024-03-06T07:28:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2f4153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2f40cc4c0 0x7fc2f40cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f4157a70 0x7fc2f4157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f40b30b0 con 0x7fc2d80039b0 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4071a50 con 0x7fc2f40cc4c0 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.437+0000 7fc2fbc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f406d370 con 0x7fc2f4157a70 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2f4153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2f4153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41362/0 (socket says 172.21.15.97:41362) 2024-03-06T07:28:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 -- 172.21.15.97:0/4194319044 learned_addr learned my addr 172.21.15.97:0/4194319044 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2f40cc4c0 0x7fc2f40cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9c00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f4157a70 0x7fc2f4157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 -- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f4157a70 msgr2=0x7fc2f4157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f4157a70 0x7fc2f4157ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 -- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2f40cc4c0 msgr2=0x7fc2f40cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2f40cc4c0 0x7fc2f40cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2f9200640 1 -- 172.21.15.97:0/4194319044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f4158430 con 0x7fc2d80039b0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.438+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2f40cc4c0 0x7fc2f40cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.439+0000 7fc2f9200640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2f4153580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c500 tx=0x7fc2e4001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.439+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4004bf0 con 0x7fc2d80039b0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.439+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e4004d90 con 0x7fc2d80039b0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.439+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4007290 con 0x7fc2d80039b0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.439+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f40ccf80 con 0x7fc2d80039b0 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.439+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f40cd540 con 0x7fc2d80039b0 2024-03-06T07:28:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.440+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f404f330 con 0x7fc2d80039b0 2024-03-06T07:28:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.441+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2e4007430 con 0x7fc2d80039b0 2024-03-06T07:28:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.442+0000 7fc2f1600640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b500 0x7fc2d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.443+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc2e40aba20 con 0x7fc2d80039b0 2024-03-06T07:28:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.443+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b500 0x7fc2d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.443+0000 7fc2ebe00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b500 0x7fc2d006d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc006e80 tx=0x7fc2dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.447+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2f4071d30 con 0x7fc2d80039b0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.645+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc2f40b3f80 con 0x7fc2d80039b0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.646+0000 7fc2f1600640 1 -- 172.21.15.97:0/4194319044 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc2e40343d0 con 0x7fc2d80039b0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:48.649 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:28:48.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:28:48.649 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:28:48.649 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:28:48.649 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:28:48.649 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b500 msgr2=0x7fc2d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b500 0x7fc2d006d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc006e80 tx=0x7fc2dc008040 comp rx=0 tx=0).stop 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 msgr2=0x7fc2f4153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2f4153580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c500 tx=0x7fc2e4001c10 comp rx=0 tx=0).stop 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 shutdown_connections 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b500 0x7fc2d006d9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f4157a70 0x7fc2f4157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2f40cc4c0 0x7fc2f40cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 --2- 172.21.15.97:0/4194319044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d80039b0 0x7fc2f4153580 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 >> 172.21.15.97:0/4194319044 conn(0x7fc2f40b1730 msgr2=0x7fc2f4160aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.650+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 shutdown_connections 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.651+0000 7fc2fbc00640 1 -- 172.21.15.97:0/4194319044 wait complete. 2024-03-06T07:28:48.670 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:48.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:48.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.879+0000 7f135fe00640 1 Processor -- start 2024-03-06T07:28:48.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.879+0000 7f135fe00640 1 -- start start 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f135fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1360070d30 0x7f1360160470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f135fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f136006afc0 0x7f136006b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f135fe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360072090 con 0x7f1360144940 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f135fe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136015ffe0 con 0x7f1360070d30 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f135fe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1360072370 con 0x7f136006afc0 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f1357e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1360070d30 0x7f1360160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f1357e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1360070d30 0x7f1360160470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41370/0 (socket says 172.21.15.97:41370) 2024-03-06T07:28:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f135ea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f1360144940 0x7f1360071860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54294/0 (socket says 172.21.15.97:54294) 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.881+0000 7f1357e00640 1 -- 172.21.15.97:0/3589179212 learned_addr learned my addr 172.21.15.97:0/3589179212 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.882+0000 7f135f400640 1 --2- 172.21.15.97:0/3589179212 >> v2:172.21.15.135:3300/0 conn(0x7f136006afc0 0x7f136006b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.882+0000 7f1357e00640 1 -- 172.21.15.97:0/3589179212 >> v2:172.21.15.135:3300/0 conn(0x7f136006afc0 msgr2=0x7f136006b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.882+0000 7f1357e00640 1 --2- 172.21.15.97:0/3589179212 >> v2:172.21.15.135:3300/0 conn(0x7f136006afc0 0x7f136006b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.882+0000 7f1357e00640 1 -- 172.21.15.97:0/3589179212 >> v1:172.21.15.97:6789/0 conn(0x7f1360144940 legacy=0x7f1360071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.882+0000 7f1357e00640 1 -- 172.21.15.97:0/3589179212 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136015f990 con 0x7f1360070d30 2024-03-06T07:28:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.882+0000 7f135e000640 1 -- 172.21.15.97:0/3589179212 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210090855 0 0) 0x7f1360072090 con 0x7f1360144940 2024-03-06T07:28:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.883+0000 7f1357e00640 1 --2- 172.21.15.97:0/3589179212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360070d30 0x7f1360160470 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f136015ff50 tx=0x7f134802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bfe78cd4f876ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.883+0000 7f135e000640 1 -- 172.21.15.97:0/3589179212 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348034070 con 0x7f1360070d30 2024-03-06T07:28:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.883+0000 7f135e000640 1 -- 172.21.15.97:0/3589179212 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1348030860 con 0x7f1360070d30 2024-03-06T07:28:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.883+0000 7f135e000640 1 -- 172.21.15.97:0/3589179212 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348030c30 con 0x7f1360070d30 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 -- 172.21.15.97:0/3589179212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360070d30 msgr2=0x7f1360160470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 --2- 172.21.15.97:0/3589179212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360070d30 0x7f1360160470 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f136015ff50 tx=0x7f134802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 -- 172.21.15.97:0/3589179212 shutdown_connections 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 --2- 172.21.15.97:0/3589179212 >> v2:172.21.15.135:3300/0 conn(0x7f136006afc0 0x7f136006b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 --2- 172.21.15.97:0/3589179212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360070d30 0x7f1360160470 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 -- 172.21.15.97:0/3589179212 >> 172.21.15.97:0/3589179212 conn(0x7f13600b18a0 msgr2=0x7f13600b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 -- 172.21.15.97:0/3589179212 shutdown_connections 2024-03-06T07:28:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.884+0000 7f135fe00640 1 -- 172.21.15.97:0/3589179212 wait complete. 2024-03-06T07:28:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.886+0000 7f135fe00640 1 Processor -- start 2024-03-06T07:28:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.886+0000 7f135fe00640 1 -- start start 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.887+0000 7f135fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f136006afc0 0x7f136006d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.887+0000 7f135fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1360070d30 0x7f136013a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.887+0000 7f135fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 0x7f136015d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.887+0000 7f135fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360144ea0 con 0x7f1360144940 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.887+0000 7f135fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360144d20 con 0x7f1360070d30 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.887+0000 7f135fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600b3640 con 0x7f136006afc0 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 0x7f136015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 0x7f136015d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41384/0 (socket says 172.21.15.97:41384) 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 -- 172.21.15.97:0/3505035856 learned_addr learned my addr 172.21.15.97:0/3505035856 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f1357e00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1360070d30 0x7f136013a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135ea00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f136006afc0 0x7f136006d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 -- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f136006afc0 msgr2=0x7f136006d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f136006afc0 0x7f136006d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 -- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1360070d30 msgr2=0x7f136013a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1360070d30 0x7f136013a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135f400640 1 -- 172.21.15.97:0/3505035856 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136015dd20 con 0x7f1360144940 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.888+0000 7f135ea00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f136006afc0 0x7f136006d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.889+0000 7f135f400640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 0x7f136015d420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1358007e40 tx=0x7f135800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.889+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358014620 con 0x7f1360144940 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.889+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1358019070 con 0x7f1360144940 2024-03-06T07:28:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.889+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136015e000 con 0x7f1360144940 2024-03-06T07:28:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.889+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358014a20 con 0x7f1360144940 2024-03-06T07:28:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.889+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136015e5c0 con 0x7f1360144940 2024-03-06T07:28:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.890+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136004f330 con 0x7f1360144940 2024-03-06T07:28:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.891+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1358014bc0 con 0x7f1360144940 2024-03-06T07:28:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.892+0000 7f1356a00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.892+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f135808a900 con 0x7f1360144940 2024-03-06T07:28:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.892+0000 7f135ea00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.893+0000 7f135ea00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133c06b4d0 0x7f133c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1350000f60 tx=0x7f1350009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:48.896+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13580599a0 con 0x7f1360144940 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.094+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f136015bdc0 con 0x7f1360144940 2024-03-06T07:28:49.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.845+0000 7f1356a00640 1 -- 172.21.15.97:0/3505035856 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7f13580592a0 con 0x7f1360144940 2024-03-06T07:28:49.848 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133c06b4d0 msgr2=0x7f133c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133c06b4d0 0x7f133c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1350000f60 tx=0x7f1350009040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 msgr2=0x7f136015d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 0x7f136015d420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1358007e40 tx=0x7f135800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 shutdown_connections 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1360144940 0x7f136015d420 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1360070d30 0x7f136013a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 --2- 172.21.15.97:0/3505035856 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f136006afc0 0x7f136006d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 >> 172.21.15.97:0/3505035856 conn(0x7f13600b18a0 msgr2=0x7f13601413c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 shutdown_connections 2024-03-06T07:28:49.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:49.849+0000 7f135fe00640 1 -- 172.21.15.97:0/3505035856 wait complete. 2024-03-06T07:28:49.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:50.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.079+0000 7f514be00640 1 Processor -- start 2024-03-06T07:28:50.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.080+0000 7f514be00640 1 -- start start 2024-03-06T07:28:50.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.081+0000 7f514be00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f514c0d1f20 0x7f514c0d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.081+0000 7f514be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f514c0cf240 0x7f514c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.081+0000 7f514be00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0b3190 con 0x7f514c15e2d0 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.081+0000 7f514be00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c0cfae0 con 0x7f514c0cf240 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.081+0000 7f514be00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c0cf800 con 0x7f514c0d1f20 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.081+0000 7f514b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f514c0cf240 0x7f514c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f514c0cf240 0x7f514c0b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41388/0 (socket says 172.21.15.97:41388) 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514b400640 1 -- 172.21.15.97:0/3510550666 learned_addr learned my addr 172.21.15.97:0/3510550666 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514aa00640 1 --2- 172.21.15.97:0/3510550666 >> v2:172.21.15.135:3300/0 conn(0x7f514c0d1f20 0x7f514c0d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514b400640 1 -- 172.21.15.97:0/3510550666 >> v2:172.21.15.135:3300/0 conn(0x7f514c0d1f20 msgr2=0x7f514c0d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514b400640 1 --2- 172.21.15.97:0/3510550666 >> v2:172.21.15.135:3300/0 conn(0x7f514c0d1f20 0x7f514c0d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514b400640 1 -- 172.21.15.97:0/3510550666 >> v1:172.21.15.97:6789/0 conn(0x7f514c15e2d0 legacy=0x7f514c0d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.082+0000 7f514b400640 1 -- 172.21.15.97:0/3510550666 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0b3430 con 0x7f514c0cf240 2024-03-06T07:28:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.083+0000 7f514b400640 1 --2- 172.21.15.97:0/3510550666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c0b3c30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f514c08cba0 tx=0x7f514402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8892fe2eb63a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.083+0000 7f514a000640 1 -- 172.21.15.97:0/3510550666 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514402cb10 con 0x7f514c0cf240 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.083+0000 7f514a000640 1 -- 172.21.15.97:0/3510550666 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514402ccb0 con 0x7f514c0cf240 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.083+0000 7f514a000640 1 -- 172.21.15.97:0/3510550666 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514402ce60 con 0x7f514c0cf240 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 -- 172.21.15.97:0/3510550666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 msgr2=0x7f514c0b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 --2- 172.21.15.97:0/3510550666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c0b3c30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f514c08cba0 tx=0x7f514402bf70 comp rx=0 tx=0).stop 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 -- 172.21.15.97:0/3510550666 shutdown_connections 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 --2- 172.21.15.97:0/3510550666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c0b3c30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 --2- 172.21.15.97:0/3510550666 >> v2:172.21.15.135:3300/0 conn(0x7f514c0d1f20 0x7f514c0d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 -- 172.21.15.97:0/3510550666 >> 172.21.15.97:0/3510550666 conn(0x7f514c0d3c90 msgr2=0x7f514c0d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 -- 172.21.15.97:0/3510550666 shutdown_connections 2024-03-06T07:28:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.084+0000 7f514be00640 1 -- 172.21.15.97:0/3510550666 wait complete. 2024-03-06T07:28:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.086+0000 7f514be00640 1 Processor -- start 2024-03-06T07:28:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.086+0000 7f514be00640 1 -- start start 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c14d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f514c0d1f20 0x7f514c0b8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f514c15e2d0 0x7f514c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0b42f0 con 0x7f514c0cf240 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0b4170 con 0x7f514c0d1f20 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0b4470 con 0x7f514c15e2d0 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c14d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41404/0 (socket says 172.21.15.97:41404) 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.087+0000 7f514aa00640 1 -- 172.21.15.97:0/2423998400 learned_addr learned my addr 172.21.15.97:0/2423998400 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514aa00640 1 -- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f514c15e2d0 msgr2=0x7f514c06c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514b400640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f514c15e2d0 0x7f514c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f5142000640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f514c0d1f20 0x7f514c0b8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514aa00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f514c15e2d0 0x7f514c06c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514aa00640 1 -- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f514c0d1f20 msgr2=0x7f514c0b8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514aa00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f514c0d1f20 0x7f514c0b8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514aa00640 1 -- 172.21.15.97:0/2423998400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0b8e00 con 0x7f514c0cf240 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f5142000640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f514c0d1f20 0x7f514c0b8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514aa00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c14d450 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f513c00d830 tx=0x7f513c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c00b6c0 con 0x7f514c0cf240 2024-03-06T07:28:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c019070 con 0x7f514c0cf240 2024-03-06T07:28:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c0b90e0 con 0x7f514c0cf240 2024-03-06T07:28:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.088+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c00ba80 con 0x7f514c0cf240 2024-03-06T07:28:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.089+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c15e8a0 con 0x7f514c0cf240 2024-03-06T07:28:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.090+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c0b92b0 con 0x7f514c0cf240 2024-03-06T07:28:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.091+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f513c01e460 con 0x7f514c0cf240 2024-03-06T07:28:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.092+0000 7f5143400640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f512806b4d0 0x7f512806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.092+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f513c08a6e0 con 0x7f514c0cf240 2024-03-06T07:28:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.092+0000 7f5142000640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f512806b4d0 0x7f512806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.093+0000 7f5142000640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f512806b4d0 0x7f512806d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f514c0c99d0 tx=0x7f5134009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.097+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f513c059730 con 0x7f514c0cf240 2024-03-06T07:28:50.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.294+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f514c0b5990 con 0x7f514c0cf240 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.863+0000 7f5143400640 1 -- 172.21.15.97:0/2423998400 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7f513c059030 con 0x7f514c0cf240 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:28:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.867+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f512806b4d0 msgr2=0x7f512806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.867+0000 7f514be00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f512806b4d0 0x7f512806d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f514c0c99d0 tx=0x7f5134009040 comp rx=0 tx=0).stop 2024-03-06T07:28:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.867+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 msgr2=0x7f514c14d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.867+0000 7f514be00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c14d450 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f513c00d830 tx=0x7f513c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.867+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 shutdown_connections 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f512806b4d0 0x7f512806d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f514c15e2d0 0x7f514c06c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f514c0d1f20 0x7f514c0b8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 --2- 172.21.15.97:0/2423998400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f514c0cf240 0x7f514c14d450 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 >> 172.21.15.97:0/2423998400 conn(0x7f514c0d3c90 msgr2=0x7f514c0b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 shutdown_connections 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:50.868+0000 7f514be00640 1 -- 172.21.15.97:0/2423998400 wait complete. 2024-03-06T07:28:50.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:51.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.088+0000 7f4c22c00640 1 Processor -- start 2024-03-06T07:28:51.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.088+0000 7f4c22c00640 1 -- start start 2024-03-06T07:28:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.090+0000 7f4c22c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c1c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.090+0000 7f4c22c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1c15c1f0 con 0x7f4c1c06da40 2024-03-06T07:28:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.090+0000 7f4c22c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1c06cac0 con 0x7f4c1c0aef20 2024-03-06T07:28:51.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.090+0000 7f4c22c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c0cd200 con 0x7f4c1c15be10 2024-03-06T07:28:51.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.090+0000 7f4c22200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4c1c06da40 0x7f4c1c071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54316/0 (socket says 172.21.15.97:54316) 2024-03-06T07:28:51.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.090+0000 7f4c22200640 1 -- 172.21.15.97:0/2464158807 learned_addr learned my addr 172.21.15.97:0/2464158807 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c21800640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c1c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394901235 0 0) 0x7f4c1c15c1f0 con 0x7f4c1c06da40 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c04003520 con 0x7f4c1c06da40 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422999720 0 0) 0x7f4c04003520 con 0x7f4c1c06da40 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 msgr2=0x7f4c1c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c1c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 >> v1:172.21.15.135:6789/0 conn(0x7f4c1c0aef20 legacy=0x7f4c1c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c06cda0 con 0x7f4c1c06da40 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4c140019f0 con 0x7f4c1c06da40 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 >> v1:172.21.15.97:6789/0 conn(0x7f4c1c06da40 legacy=0x7f4c1c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c040039e0 0x7f4c04003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c04004800 0x7f4c04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c04005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c0cd200 con 0x7f4c04004800 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0400baf0 con 0x7f4c040039e0 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.091+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0400bcf0 con 0x7f4c1c15be10 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c20e00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c04004800 0x7f4c04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c22200640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c04005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c21800640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c040039e0 0x7f4c04003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c20e00640 1 -- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 msgr2=0x7f4c04005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c20e00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c04005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c20e00640 1 -- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c040039e0 msgr2=0x7f4c04003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c20e00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c040039e0 0x7f4c04003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c20e00640 1 -- 172.21.15.97:0/2464158807 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04003520 con 0x7f4c04004800 2024-03-06T07:28:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.092+0000 7f4c22200640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c1c15be10 0x7f4c04005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.093+0000 7f4c20e00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c04004800 0x7f4c04004c80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4c08001750 tx=0x7f4c04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9d193335501758 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.093+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08001370 con 0x7f4c04004800 2024-03-06T07:28:51.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c080052d0 con 0x7f4c04004800 2024-03-06T07:28:51.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c1be00640 1 -- 172.21.15.97:0/2464158807 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08003830 con 0x7f4c04004800 2024-03-06T07:28:51.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22c00640 1 -- 172.21.15.97:0/2464158807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c04004800 msgr2=0x7f4c04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2464158807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c04004800 0x7f4c04004c80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4c08001750 tx=0x7f4c04005830 comp rx=0 tx=0).stop 2024-03-06T07:28:51.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22200640 1 -- 172.21.15.97:0/2464158807 reap_dead start 2024-03-06T07:28:51.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22c00640 1 -- 172.21.15.97:0/2464158807 shutdown_connections 2024-03-06T07:28:51.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22c00640 1 -- 172.21.15.97:0/2464158807 >> 172.21.15.97:0/2464158807 conn(0x7f4c1c06afc0 msgr2=0x7f4c1c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22c00640 1 -- 172.21.15.97:0/2464158807 shutdown_connections 2024-03-06T07:28:51.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.094+0000 7f4c22c00640 1 -- 172.21.15.97:0/2464158807 wait complete. 2024-03-06T07:28:51.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.096+0000 7f4c22c00640 1 Processor -- start 2024-03-06T07:28:51.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.096+0000 7f4c22c00640 1 -- start start 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c04004800 0x7f4c1c14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c1c0a95a0 0x7f4c1c0ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 0x7f4c1c0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c0cda90 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c06d080 con 0x7f4c1c0a95a0 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c0ccc20 con 0x7f4c04004800 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 0x7f4c1c0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 0x7f4c1c0cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41430/0 (socket says 172.21.15.97:41430) 2024-03-06T07:28:51.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.097+0000 7f4c22200640 1 -- 172.21.15.97:0/2080662264 learned_addr learned my addr 172.21.15.97:0/2080662264 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c21800640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c04004800 0x7f4c1c14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c20e00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c1c0a95a0 0x7f4c1c0ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c22200640 1 -- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c04004800 msgr2=0x7f4c1c14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c22200640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c04004800 0x7f4c1c14b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c22200640 1 -- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c1c0a95a0 msgr2=0x7f4c1c0ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c22200640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c1c0a95a0 0x7f4c1c0ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c22200640 1 -- 172.21.15.97:0/2080662264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c0a9ea0 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c22200640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 0x7f4c1c0cf2b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0d09c0 tx=0x7f4c1400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400de90 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1400bb80 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.098+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14007110 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.099+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c14af10 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.099+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c14b4d0 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.100+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c04f330 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.101+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c1401b060 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.102+0000 7f4c19600640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bfc06b6a0 0x7f4bfc06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.102+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4c1408a190 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.102+0000 7f4c21800640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bfc06b6a0 0x7f4bfc06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.103+0000 7f4c21800640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bfc06b6a0 0x7f4bfc06db60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c10006fd0 tx=0x7f4c10008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.107+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c14059100 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.342+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4c1c0ce850 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.884+0000 7f4c19600640 1 -- 172.21.15.97:0/2080662264 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4c14058a00 con 0x7f4c1c0a9a20 2024-03-06T07:28:51.889 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:28:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.897+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bfc06b6a0 msgr2=0x7f4bfc06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.897+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bfc06b6a0 0x7f4bfc06db60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c10006fd0 tx=0x7f4c10008040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.897+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 msgr2=0x7f4c1c0cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.897+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 0x7f4c1c0cf2b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0d09c0 tx=0x7f4c1400b4a0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.899+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 shutdown_connections 2024-03-06T07:28:51.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.899+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bfc06b6a0 0x7f4bfc06db60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.899+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c1c0a9a20 0x7f4c1c0cf2b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.899+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c1c0a95a0 0x7f4c1c0ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.899+0000 7f4c22c00640 1 --2- 172.21.15.97:0/2080662264 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c04004800 0x7f4c1c14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.899+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 >> 172.21.15.97:0/2080662264 conn(0x7f4c1c06afc0 msgr2=0x7f4c1c0b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.902+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 shutdown_connections 2024-03-06T07:28:51.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:51.903+0000 7f4c22c00640 1 -- 172.21.15.97:0/2080662264 wait complete. 2024-03-06T07:28:51.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.115+0000 7f801a600640 1 Processor -- start 2024-03-06T07:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.115+0000 7f801a600640 1 -- start start 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.116+0000 7f801a600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 0x7f801415e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.116+0000 7f801a600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8014161900 con 0x7f80141620c0 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.116+0000 7f801a600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801415ebd0 con 0x7f80140b40c0 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.116+0000 7f801a600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801415eeb0 con 0x7f80140ae670 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8019c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f80141620c0 0x7f80140b28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54330/0 (socket says 172.21.15.97:54330) 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8019c00640 1 -- 172.21.15.97:0/549103384 learned_addr learned my addr 172.21.15.97:0/549103384 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f800be00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 0x7f801415e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286596262 0 0) 0x7f8014161900 con 0x7f80141620c0 2024-03-06T07:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8003520 con 0x7f80141620c0 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403756014 0 0) 0x7f7ff8003520 con 0x7f80141620c0 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 msgr2=0x7f801415e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8013e00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 0x7f801415e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.117+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 >> v1:172.21.15.135:6789/0 conn(0x7f80140b40c0 legacy=0x7f80140b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801415f190 con 0x7f80141620c0 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f800c001670 con 0x7f80141620c0 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 >> v1:172.21.15.97:6789/0 conn(0x7f80141620c0 legacy=0x7f80140b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ff80039e0 0x7f7ff8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f7ff8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 0x7f7ff8005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801415eeb0 con 0x7f7ff8004800 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff800bb50 con 0x7f7ff80039e0 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff800bd50 con 0x7f80140ae670 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f800be00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f7ff8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8019200640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ff80039e0 0x7f7ff8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f8019c00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 0x7f7ff8005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f800be00640 1 -- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 msgr2=0x7f7ff8005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f800be00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140ae670 0x7f7ff8005250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f800be00640 1 -- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ff80039e0 msgr2=0x7f7ff8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f800be00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ff80039e0 0x7f7ff8003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.118+0000 7f800be00640 1 -- 172.21.15.97:0/549103384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8003520 con 0x7f7ff8004800 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.119+0000 7f800be00640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f7ff8004c80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00ba30 tx=0x7f7ff8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aceb5a466b3fbca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc036020 con 0x7f7ff8004800 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc032640 con 0x7f7ff8004800 2024-03-06T07:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f8013e00640 1 -- 172.21.15.97:0/549103384 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00b600 con 0x7f7ff8004800 2024-03-06T07:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f801a600640 1 -- 172.21.15.97:0/549103384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 msgr2=0x7f7ff8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f801a600640 1 --2- 172.21.15.97:0/549103384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f7ff8004c80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00ba30 tx=0x7f7ff8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f8019c00640 1 -- 172.21.15.97:0/549103384 reap_dead start 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f801a600640 1 -- 172.21.15.97:0/549103384 shutdown_connections 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f801a600640 1 -- 172.21.15.97:0/549103384 >> 172.21.15.97:0/549103384 conn(0x7f801406afc0 msgr2=0x7f801406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f801a600640 1 -- 172.21.15.97:0/549103384 shutdown_connections 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.120+0000 7f801a600640 1 -- 172.21.15.97:0/549103384 wait complete. 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.122+0000 7f801a600640 1 Processor -- start 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.122+0000 7f801a600640 1 -- start start 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f801a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f801415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f801a600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140cff30 0x7f8014158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f801a600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80140d03b0 0x7f8014144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f801a600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80141587e0 con 0x7f7ff8004800 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f801a600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8014158660 con 0x7f80140d03b0 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f801a600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80140b2ff0 con 0x7f80140cff30 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f8019200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f801415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f8019200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f801415a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41452/0 (socket says 172.21.15.97:41452) 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.123+0000 7f8019200640 1 -- 172.21.15.97:0/159009381 learned_addr learned my addr 172.21.15.97:0/159009381 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019c00640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80140d03b0 0x7f8014144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f800be00640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140cff30 0x7f8014158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019200640 1 -- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140cff30 msgr2=0x7f8014158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019200640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140cff30 0x7f8014158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019200640 1 -- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80140d03b0 msgr2=0x7f8014144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019200640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80140d03b0 0x7f8014144100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019200640 1 -- 172.21.15.97:0/159009381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8014159390 con 0x7f7ff8004800 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8019200640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f801415a430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f800400d860 tx=0x7f800400dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004003da0 con 0x7f7ff8004800 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004017070 con 0x7f7ff8004800 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8014159670 con 0x7f7ff8004800 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400b8f0 con 0x7f7ff8004800 2024-03-06T07:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.124+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8014159c30 con 0x7f7ff8004800 2024-03-06T07:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.126+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801404f330 con 0x7f7ff8004800 2024-03-06T07:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.126+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f800400ba90 con 0x7f7ff8004800 2024-03-06T07:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.127+0000 7f8011600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ff006b500 0x7f7ff006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.127+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f800408a380 con 0x7f7ff8004800 2024-03-06T07:28:52.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.128+0000 7f800be00640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ff006b500 0x7f7ff006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.128+0000 7f800be00640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ff006b500 0x7f7ff006d9c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc037000 tx=0x7f7ffc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.132+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8004059200 con 0x7f7ff8004800 2024-03-06T07:28:52.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.324+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80140b0cb0 con 0x7f7ff8004800 2024-03-06T07:28:52.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.326+0000 7f8011600640 1 -- 172.21.15.97:0/159009381 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16670 (secure 0 0 0) 0x7f8004058b00 con 0x7f7ff8004800 2024-03-06T07:28:52.329 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:52.329 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:51.344206+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:52.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ff006b500 msgr2=0x7f7ff006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ff006b500 0x7f7ff006d9c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc037000 tx=0x7f7ffc030000 comp rx=0 tx=0).stop 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 msgr2=0x7f801415a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f801415a430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f800400d860 tx=0x7f800400dd20 comp rx=0 tx=0).stop 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 shutdown_connections 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ff006b500 0x7f7ff006d9c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80140d03b0 0x7f8014144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80140cff30 0x7f8014158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 --2- 172.21.15.97:0/159009381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ff8004800 0x7f801415a430 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 >> 172.21.15.97:0/159009381 conn(0x7f801406afc0 msgr2=0x7f80140cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 shutdown_connections 2024-03-06T07:28:52.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.329+0000 7f801a600640 1 -- 172.21.15.97:0/159009381 wait complete. 2024-03-06T07:28:52.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.544+0000 7efc56a00640 1 Processor -- start 2024-03-06T07:28:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.545+0000 7efc56a00640 1 -- start start 2024-03-06T07:28:52.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.546+0000 7efc56a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500aedc0 0x7efc500cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.546+0000 7efc56a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc500cc800 0x7efc5015f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.546+0000 7efc56a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5015c240 con 0x7efc501561f0 2024-03-06T07:28:52.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.546+0000 7efc56a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5015ce30 con 0x7efc500cc800 2024-03-06T07:28:52.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.546+0000 7efc56a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5015c500 con 0x7efc500aedc0 2024-03-06T07:28:52.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.547+0000 7efc56000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc500cc800 0x7efc5015f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.547+0000 7efc54c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500aedc0 0x7efc500cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.547+0000 7efc56000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc500cc800 0x7efc5015f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:59148/0 (socket says 172.21.15.97:59148) 2024-03-06T07:28:52.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.547+0000 7efc56000640 1 -- 172.21.15.97:0/1570776395 learned_addr learned my addr 172.21.15.97:0/1570776395 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:52.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.547+0000 7efc55600640 1 --1- 172.21.15.97:0/1570776395 >> v1:172.21.15.135:6789/0 conn(0x7efc501561f0 0x7efc500ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:53646/0 (socket says 172.21.15.97:53646) 2024-03-06T07:28:52.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.548+0000 7efc56000640 1 -- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500aedc0 msgr2=0x7efc500cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.548+0000 7efc56000640 1 --2- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500aedc0 0x7efc500cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.548+0000 7efc56000640 1 -- 172.21.15.97:0/1570776395 >> v1:172.21.15.135:6789/0 conn(0x7efc501561f0 legacy=0x7efc500ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.548+0000 7efc56000640 1 -- 172.21.15.97:0/1570776395 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5015c7e0 con 0x7efc500cc800 2024-03-06T07:28:52.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.549+0000 7efc56000640 1 --2- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500cc800 0x7efc5015f830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efc48009e50 tx=0x7efc4802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2eba0cc0b6847cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.549+0000 7efc4fe00640 1 -- 172.21.15.97:0/1570776395 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4802cb00 con 0x7efc500cc800 2024-03-06T07:28:52.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.549+0000 7efc4fe00640 1 -- 172.21.15.97:0/1570776395 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4802cca0 con 0x7efc500cc800 2024-03-06T07:28:52.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.549+0000 7efc4fe00640 1 -- 172.21.15.97:0/1570776395 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48034610 con 0x7efc500cc800 2024-03-06T07:28:52.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 -- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500cc800 msgr2=0x7efc5015f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 --2- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500cc800 0x7efc5015f830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efc48009e50 tx=0x7efc4802bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:52.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 -- 172.21.15.97:0/1570776395 shutdown_connections 2024-03-06T07:28:52.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 --2- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500aedc0 0x7efc500cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 --2- 172.21.15.97:0/1570776395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500cc800 0x7efc5015f830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 -- 172.21.15.97:0/1570776395 >> 172.21.15.97:0/1570776395 conn(0x7efc5006afc0 msgr2=0x7efc5006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 -- 172.21.15.97:0/1570776395 shutdown_connections 2024-03-06T07:28:52.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.550+0000 7efc56a00640 1 -- 172.21.15.97:0/1570776395 wait complete. 2024-03-06T07:28:52.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.552+0000 7efc56a00640 1 Processor -- start 2024-03-06T07:28:52.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.552+0000 7efc56a00640 1 -- start start 2024-03-06T07:28:52.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.553+0000 7efc56a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500aedc0 0x7efc501f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.553+0000 7efc56a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500cc800 0x7efc500ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.553+0000 7efc56a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 0x7efc5014e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.553+0000 7efc56a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc501565d0 con 0x7efc501561f0 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.553+0000 7efc56a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500ccc90 con 0x7efc500aedc0 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.553+0000 7efc56a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc50156750 con 0x7efc500cc800 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 0x7efc5014e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 0x7efc5014e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41464/0 (socket says 172.21.15.97:41464) 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 -- 172.21.15.97:0/3235604701 learned_addr learned my addr 172.21.15.97:0/3235604701 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc54c00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500cc800 0x7efc500ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc55600640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500aedc0 0x7efc501f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 -- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500cc800 msgr2=0x7efc500ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500cc800 0x7efc500ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 -- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500aedc0 msgr2=0x7efc501f5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500aedc0 0x7efc501f5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc56000640 1 -- 172.21.15.97:0/3235604701 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5015a000 con 0x7efc501561f0 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.554+0000 7efc55600640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500aedc0 0x7efc501f5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.555+0000 7efc56000640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 0x7efc5014e450 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efc48009e50 tx=0x7efc48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.555+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48043020 con 0x7efc501561f0 2024-03-06T07:28:52.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.555+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4802ce50 con 0x7efc501561f0 2024-03-06T07:28:52.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.555+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48009510 con 0x7efc501561f0 2024-03-06T07:28:52.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.555+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5015a2e0 con 0x7efc501561f0 2024-03-06T07:28:52.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.555+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc500cf080 con 0x7efc501561f0 2024-03-06T07:28:52.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.556+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5004f330 con 0x7efc501561f0 2024-03-06T07:28:52.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.557+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc48046020 con 0x7efc501561f0 2024-03-06T07:28:52.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.558+0000 7efc4d600640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc3406b4d0 0x7efc3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.558+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc480ac7e0 con 0x7efc501561f0 2024-03-06T07:28:52.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.558+0000 7efc55600640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc3406b4d0 0x7efc3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.559+0000 7efc55600640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc3406b4d0 0x7efc3406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efc5014dab0 tx=0x7efc44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.563+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc4804d030 con 0x7efc501561f0 2024-03-06T07:28:52.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.757+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc500b2b10 con 0x7efc501561f0 2024-03-06T07:28:52.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.758+0000 7efc4d600640 1 -- 172.21.15.97:0/3235604701 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16670 (secure 0 0 0) 0x7efc4803c350 con 0x7efc501561f0 2024-03-06T07:28:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:52.761 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:51.344206+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:52.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.761+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc3406b4d0 msgr2=0x7efc3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.761+0000 7efc56a00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc3406b4d0 0x7efc3406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efc5014dab0 tx=0x7efc44009040 comp rx=0 tx=0).stop 2024-03-06T07:28:52.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.761+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 msgr2=0x7efc5014e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.761+0000 7efc56a00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 0x7efc5014e450 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efc48009e50 tx=0x7efc48004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:52.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 shutdown_connections 2024-03-06T07:28:52.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc3406b4d0 0x7efc3406d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc501561f0 0x7efc5014e450 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc500cc800 0x7efc500ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 --2- 172.21.15.97:0/3235604701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc500aedc0 0x7efc501f5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 >> 172.21.15.97:0/3235604701 conn(0x7efc5006afc0 msgr2=0x7efc500719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 shutdown_connections 2024-03-06T07:28:52.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.762+0000 7efc56a00640 1 -- 172.21.15.97:0/3235604701 wait complete. 2024-03-06T07:28:52.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:52.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.995+0000 7ff55a000640 1 Processor -- start 2024-03-06T07:28:52.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.995+0000 7ff55a000640 1 -- start start 2024-03-06T07:28:52.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.996+0000 7ff55a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff5540727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.996+0000 7ff55a000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5540719e0 con 0x7ff5540b3650 2024-03-06T07:28:52.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.996+0000 7ff55a000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff554071cc0 con 0x7ff5540cda90 2024-03-06T07:28:52.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.996+0000 7ff55a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff554072310 con 0x7ff5540b3df0 2024-03-06T07:28:52.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff5540b3650 0x7ff5540b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54336/0 (socket says 172.21.15.97:54336) 2024-03-06T07:28:52.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552a00640 1 -- 172.21.15.97:0/2941335132 learned_addr learned my addr 172.21.15.97:0/2941335132 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:53.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff553400640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff5540727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474728205 0 0) 0x7ff5540719e0 con 0x7ff5540b3650 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538005730 con 0x7ff5540b3650 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345156418 0 0) 0x7ff538005730 con 0x7ff5540b3650 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 msgr2=0x7ff5540727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552000640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff5540727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.997+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 >> v1:172.21.15.135:6789/0 conn(0x7ff5540cda90 legacy=0x7ff55406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff554071fa0 con 0x7ff5540b3650 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff540001670 con 0x7ff5540b3650 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 >> v1:172.21.15.97:6789/0 conn(0x7ff5540b3650 legacy=0x7ff5540b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff538005bf0 0x7ff538006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff538006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff538006f50 0x7ff5380073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff554072310 con 0x7ff538006f50 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53800ddb0 con 0x7ff538005bf0 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53800dfb0 con 0x7ff5540b3df0 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553e00640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff538006f50 0x7ff5380073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552a00640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff538006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553e00640 1 -- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 msgr2=0x7ff538006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553400640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff538005bf0 0x7ff538006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553e00640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff538006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553e00640 1 -- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff538005bf0 msgr2=0x7ff538006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553e00640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff538005bf0 0x7ff538006010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff553e00640 1 -- 172.21.15.97:0/2941335132 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538005730 con 0x7ff538006f50 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.998+0000 7ff552a00640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5540b3df0 0x7ff538006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:53.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:52.999+0000 7ff553e00640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff538006f50 0x7ff5380073d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff54c001e20 tx=0x7ff538007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ddc9aa1a4ea23f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c001a70 con 0x7ff538006f50 2024-03-06T07:28:53.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c003b70 con 0x7ff538006f50 2024-03-06T07:28:53.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff552000640 1 -- 172.21.15.97:0/2941335132 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c0050d0 con 0x7ff538006f50 2024-03-06T07:28:53.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff55a000640 1 -- 172.21.15.97:0/2941335132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff538006f50 msgr2=0x7ff5380073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff55a000640 1 --2- 172.21.15.97:0/2941335132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff538006f50 0x7ff5380073d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff54c001e20 tx=0x7ff538007a40 comp rx=0 tx=0).stop 2024-03-06T07:28:53.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff553e00640 1 -- 172.21.15.97:0/2941335132 reap_dead start 2024-03-06T07:28:53.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff55a000640 1 -- 172.21.15.97:0/2941335132 shutdown_connections 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff55a000640 1 -- 172.21.15.97:0/2941335132 >> 172.21.15.97:0/2941335132 conn(0x7ff5540b1940 msgr2=0x7ff5540b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff55a000640 1 -- 172.21.15.97:0/2941335132 shutdown_connections 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.000+0000 7ff55a000640 1 -- 172.21.15.97:0/2941335132 wait complete. 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.002+0000 7ff55a000640 1 Processor -- start 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.002+0000 7ff55a000640 1 -- start start 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff55a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff538006f50 0x7ff5541f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff55a000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5541f2dc0 0x7ff55406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff55a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 0x7ff55415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff55a000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55415b3c0 con 0x7ff55406dfd0 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff55a000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55415b240 con 0x7ff5541f2dc0 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff55a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540cf510 con 0x7ff538006f50 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff553e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 0x7ff55415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff553e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 0x7ff55415d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41472/0 (socket says 172.21.15.97:41472) 2024-03-06T07:28:53.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.003+0000 7ff553e00640 1 -- 172.21.15.97:0/2106029454 learned_addr learned my addr 172.21.15.97:0/2106029454 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff552a00640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5541f2dc0 0x7ff55406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553400640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff538006f50 0x7ff5541f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553e00640 1 -- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff538006f50 msgr2=0x7ff5541f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553e00640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff538006f50 0x7ff5541f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553e00640 1 -- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5541f2dc0 msgr2=0x7ff55406da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553e00640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5541f2dc0 0x7ff55406da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553e00640 1 -- 172.21.15.97:0/2106029454 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55415ddd0 con 0x7ff55406dfd0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553400640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff538006f50 0x7ff5541f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff553e00640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 0x7ff55415d0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff54c001f10 tx=0x7ff54c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c001a70 con 0x7ff55406dfd0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c003970 con 0x7ff55406dfd0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5541572c0 con 0x7ff55406dfd0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c007db0 con 0x7ff55406dfd0 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.004+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff554157880 con 0x7ff55406dfd0 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.006+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55406f4c0 con 0x7ff55406dfd0 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.007+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff54c041030 con 0x7ff55406dfd0 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.008+0000 7ff54b400640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b500 0x7ff53406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.008+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff54c0ab0f0 con 0x7ff55406dfd0 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.008+0000 7ff553400640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b500 0x7ff53406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.009+0000 7ff553400640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b500 0x7ff53406d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff544006fd0 tx=0x7ff544008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.012+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff55406f4c0 con 0x7ff55406dfd0 2024-03-06T07:28:53.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.237+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff55404f330 con 0x7ff55406dfd0 2024-03-06T07:28:53.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.916+0000 7ff54b400640 1 -- 172.21.15.97:0/2106029454 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff54c033020 con 0x7ff55406dfd0 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b500 msgr2=0x7ff53406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b500 0x7ff53406d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff544006fd0 tx=0x7ff544008040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 msgr2=0x7ff55415d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 0x7ff55415d0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff54c001f10 tx=0x7ff54c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 shutdown_connections 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b500 0x7ff53406d9c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff55406dfd0 0x7ff55415d0a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5541f2dc0 0x7ff55406da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 --2- 172.21.15.97:0/2106029454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff538006f50 0x7ff5541f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 >> 172.21.15.97:0/2106029454 conn(0x7ff5540b1940 msgr2=0x7ff5540ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.921+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 shutdown_connections 2024-03-06T07:28:53.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:53.922+0000 7ff55a000640 1 -- 172.21.15.97:0/2106029454 wait complete. 2024-03-06T07:28:53.954 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:53.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:54.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.170+0000 7fe107600640 1 Processor -- start 2024-03-06T07:28:54.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.170+0000 7fe107600640 1 -- start start 2024-03-06T07:28:54.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.173+0000 7fe107600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe100161b50 0x7fe1000d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.173+0000 7fe107600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe1000d4a00 0x7fe1000d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.173+0000 7fe107600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000b5490 0x7fe1000d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.173+0000 7fe107600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1000cef40 con 0x7fe100161b50 2024-03-06T07:28:54.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.173+0000 7fe107600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1000cecd0 con 0x7fe1000d4a00 2024-03-06T07:28:54.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.173+0000 7fe107600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000cfc80 con 0x7fe1000b5490 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe100161b50 0x7fe1000d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe100161b50 0x7fe1000d05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53146/0 (socket says 172.21.15.97:53146) 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 -- 172.21.15.97:0/3725776512 learned_addr learned my addr 172.21.15.97:0/3725776512 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe105600640 1 --2- 172.21.15.97:0/3725776512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000b5490 0x7fe1000d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe0ffe00640 1 --2- 172.21.15.97:0/3725776512 >> v2:172.21.15.135:3300/0 conn(0x7fe1000d4a00 0x7fe1000d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 -- 172.21.15.97:0/3725776512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000b5490 msgr2=0x7fe1000d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 --2- 172.21.15.97:0/3725776512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000b5490 0x7fe1000d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 -- 172.21.15.97:0/3725776512 >> v2:172.21.15.135:3300/0 conn(0x7fe1000d4a00 msgr2=0x7fe1000d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 --2- 172.21.15.97:0/3725776512 >> v2:172.21.15.135:3300/0 conn(0x7fe1000d4a00 0x7fe1000d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe104c00640 1 -- 172.21.15.97:0/3725776512 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000c7b60 con 0x7fe100161b50 2024-03-06T07:28:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.174+0000 7fe0ffe00640 1 --2- 172.21.15.97:0/3725776512 >> v2:172.21.15.135:3300/0 conn(0x7fe1000d4a00 0x7fe1000d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.175+0000 7fe104c00640 1 --2- 172.21.15.97:0/3725776512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe100161b50 0x7fe1000d05a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe1000b6a60 tx=0x7fe0f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc507baa1eb6ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe0ff400640 1 -- 172.21.15.97:0/3725776512 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4035020 con 0x7fe100161b50 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe0ff400640 1 -- 172.21.15.97:0/3725776512 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f4031870 con 0x7fe100161b50 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe0ff400640 1 -- 172.21.15.97:0/3725776512 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4031c40 con 0x7fe100161b50 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 -- 172.21.15.97:0/3725776512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe100161b50 msgr2=0x7fe1000d05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 --2- 172.21.15.97:0/3725776512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe100161b50 0x7fe1000d05a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe1000b6a60 tx=0x7fe0f402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 -- 172.21.15.97:0/3725776512 shutdown_connections 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 --2- 172.21.15.97:0/3725776512 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000b5490 0x7fe1000d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 --2- 172.21.15.97:0/3725776512 >> v2:172.21.15.135:3300/0 conn(0x7fe1000d4a00 0x7fe1000d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 --2- 172.21.15.97:0/3725776512 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe100161b50 0x7fe1000d05a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.176+0000 7fe107600640 1 -- 172.21.15.97:0/3725776512 >> 172.21.15.97:0/3725776512 conn(0x7fe1000d3e70 msgr2=0x7fe1000d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.177+0000 7fe107600640 1 -- 172.21.15.97:0/3725776512 shutdown_connections 2024-03-06T07:28:54.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.177+0000 7fe107600640 1 -- 172.21.15.97:0/3725776512 wait complete. 2024-03-06T07:28:54.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.179+0000 7fe107600640 1 Processor -- start 2024-03-06T07:28:54.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.179+0000 7fe107600640 1 -- start start 2024-03-06T07:28:54.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.180+0000 7fe107600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 0x7fe1000cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe107600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000d4a00 0x7fe10015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe107600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe100161b50 0x7fe1000b9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe107600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000d22b0 con 0x7fe1000b5490 2024-03-06T07:28:54.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe107600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000d1270 con 0x7fe100161b50 2024-03-06T07:28:54.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe107600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000d2430 con 0x7fe1000d4a00 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 0x7fe1000cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 0x7fe1000cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53154/0 (socket says 172.21.15.97:53154) 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 -- 172.21.15.97:0/3241908143 learned_addr learned my addr 172.21.15.97:0/3241908143 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 -- 172.21.15.97:0/3241908143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000d4a00 msgr2=0x7fe10015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000d4a00 0x7fe10015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 -- 172.21.15.97:0/3241908143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe100161b50 msgr2=0x7fe1000b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe100161b50 0x7fe1000b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.181+0000 7fe104c00640 1 -- 172.21.15.97:0/3241908143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000b96a0 con 0x7fe1000b5490 2024-03-06T07:28:54.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.182+0000 7fe104c00640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 0x7fe1000cd730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe0f402b950 tx=0x7fe0f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.182+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4040070 con 0x7fe1000b5490 2024-03-06T07:28:54.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.182+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f4030d00 con 0x7fe1000b5490 2024-03-06T07:28:54.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.182+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4035040 con 0x7fe1000b5490 2024-03-06T07:28:54.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.182+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1000cdd10 con 0x7fe1000b5490 2024-03-06T07:28:54.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.182+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1000cbe80 con 0x7fe1000b5490 2024-03-06T07:28:54.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.183+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1000d3ae0 con 0x7fe1000b5490 2024-03-06T07:28:54.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.184+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0f4038020 con 0x7fe1000b5490 2024-03-06T07:28:54.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.185+0000 7fe0fcc00640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0e006b4d0 0x7fe0e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.185+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0f40ac0a0 con 0x7fe1000b5490 2024-03-06T07:28:54.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.185+0000 7fe0ffe00640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0e006b4d0 0x7fe0e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.186+0000 7fe0ffe00640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0e006b4d0 0x7fe0e006d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec006fd0 tx=0x7fe0ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.190+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1000d3ae0 con 0x7fe1000b5490 2024-03-06T07:28:54.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.424+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe1000d13f0 con 0x7fe1000b5490 2024-03-06T07:28:54.819 INFO:tasks.ceph.mds.a.smithi097.stderr:2024-03-06T07:28:54.817+0000 7f2c78a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:54.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.919+0000 7fe0fcc00640 1 -- 172.21.15.97:0/3241908143 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe1000d13f0 con 0x7fe1000b5490 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0e006b4d0 msgr2=0x7fe0e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0e006b4d0 0x7fe0e006d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec006fd0 tx=0x7fe0ec008040 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 msgr2=0x7fe1000cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 0x7fe1000cd730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe0f402b950 tx=0x7fe0f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 shutdown_connections 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0e006b4d0 0x7fe0e006d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe100161b50 0x7fe1000b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1000d4a00 0x7fe10015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 --2- 172.21.15.97:0/3241908143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1000b5490 0x7fe1000cd730 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 >> 172.21.15.97:0/3241908143 conn(0x7fe1000d3e70 msgr2=0x7fe1000cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 shutdown_connections 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:54.923+0000 7fe107600640 1 -- 172.21.15.97:0/3241908143 wait complete. 2024-03-06T07:28:54.946 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:54.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.147+0000 7fe764a00640 1 Processor -- start 2024-03-06T07:28:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.147+0000 7fe764a00640 1 -- start start 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe764a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe76006f4c0 0x7fe7600af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe764a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe76006afc0 0x7fe76006b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe764a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600afc00 con 0x7fe7601562b0 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe764a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76006b940 con 0x7fe76006afc0 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe764a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7600afea0 con 0x7fe76006f4c0 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe75fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe76006afc0 0x7fe76006b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe75fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe76006afc0 0x7fe76006b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53156/0 (socket says 172.21.15.97:53156) 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe75fe00640 1 -- 172.21.15.97:0/2884599271 learned_addr learned my addr 172.21.15.97:0/2884599271 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe75f400640 1 --1- 172.21.15.97:0/2884599271 >> v1:172.21.15.97:6789/0 conn(0x7fe7601562b0 0x7fe76015cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39662/0 (socket says 172.21.15.97:39662) 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.149+0000 7fe75ea00640 1 --2- 172.21.15.97:0/2884599271 >> v2:172.21.15.135:3300/0 conn(0x7fe76006f4c0 0x7fe7600af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.150+0000 7fe75fe00640 1 -- 172.21.15.97:0/2884599271 >> v2:172.21.15.135:3300/0 conn(0x7fe76006f4c0 msgr2=0x7fe7600af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.150+0000 7fe75fe00640 1 --2- 172.21.15.97:0/2884599271 >> v2:172.21.15.135:3300/0 conn(0x7fe76006f4c0 0x7fe7600af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.150+0000 7fe75fe00640 1 -- 172.21.15.97:0/2884599271 >> v1:172.21.15.97:6789/0 conn(0x7fe7601562b0 legacy=0x7fe76015cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.150+0000 7fe75fe00640 1 -- 172.21.15.97:0/2884599271 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600b0180 con 0x7fe76006afc0 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.150+0000 7fe75e000640 1 -- 172.21.15.97:0/2884599271 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454107675 0 0) 0x7fe7600afc00 con 0x7fe7601562b0 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.151+0000 7fe75fe00640 1 --2- 172.21.15.97:0/2884599271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe76006afc0 0x7fe76006b400 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe76006cf50 tx=0x7fe75802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76faf693d0bac512 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.151+0000 7fe75e000640 1 -- 172.21.15.97:0/2884599271 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75802ca80 con 0x7fe76006afc0 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.151+0000 7fe75e000640 1 -- 172.21.15.97:0/2884599271 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75802cc20 con 0x7fe76006afc0 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.151+0000 7fe75e000640 1 -- 172.21.15.97:0/2884599271 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580343f0 con 0x7fe76006afc0 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 -- 172.21.15.97:0/2884599271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe76006afc0 msgr2=0x7fe76006b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 --2- 172.21.15.97:0/2884599271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe76006afc0 0x7fe76006b400 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe76006cf50 tx=0x7fe75802bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 -- 172.21.15.97:0/2884599271 shutdown_connections 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 --2- 172.21.15.97:0/2884599271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe76006afc0 0x7fe76006b400 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 --2- 172.21.15.97:0/2884599271 >> v2:172.21.15.135:3300/0 conn(0x7fe76006f4c0 0x7fe7600af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 -- 172.21.15.97:0/2884599271 >> 172.21.15.97:0/2884599271 conn(0x7fe76006c0d0 msgr2=0x7fe76006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 -- 172.21.15.97:0/2884599271 shutdown_connections 2024-03-06T07:28:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.152+0000 7fe764a00640 1 -- 172.21.15.97:0/2884599271 wait complete. 2024-03-06T07:28:55.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.154+0000 7fe764a00640 1 Processor -- start 2024-03-06T07:28:55.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.154+0000 7fe764a00640 1 -- start start 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe764a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe76006afc0 0x7fe760154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe764a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe76006f4c0 0x7fe7600ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe764a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 0x7fe7600ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe764a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600d0500 con 0x7fe7601562b0 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe764a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76015c610 con 0x7fe76006f4c0 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe764a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600d0680 con 0x7fe76006afc0 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 0x7fe7600ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 0x7fe7600ab960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53170/0 (socket says 172.21.15.97:53170) 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75fe00640 1 -- 172.21.15.97:0/2123943918 learned_addr learned my addr 172.21.15.97:0/2123943918 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75f400640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe76006afc0 0x7fe760154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75ea00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe76006f4c0 0x7fe7600ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75fe00640 1 -- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe76006afc0 msgr2=0x7fe760154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.155+0000 7fe75fe00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe76006afc0 0x7fe760154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe75fe00640 1 -- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe76006f4c0 msgr2=0x7fe7600ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe75fe00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe76006f4c0 0x7fe7600ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe75fe00640 1 -- 172.21.15.97:0/2123943918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600b13d0 con 0x7fe7601562b0 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe75f400640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe76006afc0 0x7fe760154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe75ea00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe76006f4c0 0x7fe7600ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:55.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe75fe00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 0x7fe7600ab960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe758009f40 tx=0x7fe758004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758043020 con 0x7fe7601562b0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75803e450 con 0x7fe7601562b0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7600b16b0 con 0x7fe7601562b0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758009510 con 0x7fe7601562b0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.156+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7600b1c70 con 0x7fe7601562b0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.157+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76004f330 con 0x7fe7601562b0 2024-03-06T07:28:55.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.158+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe758046020 con 0x7fe7601562b0 2024-03-06T07:28:55.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.159+0000 7fe757400640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe74006b4d0 0x7fe74006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.159+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7580acad0 con 0x7fe7601562b0 2024-03-06T07:28:55.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.159+0000 7fe75f400640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe74006b4d0 0x7fe74006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.160+0000 7fe75f400640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe74006b4d0 0x7fe74006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe750000f60 tx=0x7fe750009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.164+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe75804d030 con 0x7fe7601562b0 2024-03-06T07:28:55.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.437+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7600cdc60 con 0x7fe7601562b0 2024-03-06T07:28:55.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.439+0000 7fe757400640 1 -- 172.21.15.97:0/2123943918 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fe75807b8b0 con 0x7fe7601562b0 2024-03-06T07:28:55.441 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:55.441 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:28:55.441 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe74006b4d0 msgr2=0x7fe74006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe74006b4d0 0x7fe74006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe750000f60 tx=0x7fe750009040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 msgr2=0x7fe7600ab960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 0x7fe7600ab960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe758009f40 tx=0x7fe758004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 shutdown_connections 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe74006b4d0 0x7fe74006d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7601562b0 0x7fe7600ab960 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe76006f4c0 0x7fe7600ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 --2- 172.21.15.97:0/2123943918 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe76006afc0 0x7fe760154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.441+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 >> 172.21.15.97:0/2123943918 conn(0x7fe76006c0d0 msgr2=0x7fe76015ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.442+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 shutdown_connections 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.442+0000 7fe764a00640 1 -- 172.21.15.97:0/2123943918 wait complete. 2024-03-06T07:28:55.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.654+0000 7efe91a00640 1 Processor -- start 2024-03-06T07:28:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.654+0000 7efe91a00640 1 -- start start 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.655+0000 7efe91a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 0x7efe8c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe91a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efe8c06ec00 0x7efe8c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe91a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c15b090 con 0x7efe8c0b4680 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe91a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe8c0b2a30 con 0x7efe8c06ec00 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe91a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c0b4360 con 0x7efe8c15acb0 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe8a000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efe8c0b4680 0x7efe8c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39672/0 (socket says 172.21.15.97:39672) 2024-03-06T07:28:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe8a000640 1 -- 172.21.15.97:0/2210277469 learned_addr learned my addr 172.21.15.97:0/2210277469 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe8aa00640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 0x7efe8c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652137960 0 0) 0x7efe8c15b090 con 0x7efe8c0b4680 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe8b400640 1 --2- 172.21.15.97:0/2210277469 >> v2:172.21.15.135:3300/0 conn(0x7efe8c06ec00 0x7efe8c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.656+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe70003520 con 0x7efe8c0b4680 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682090761 0 0) 0x7efe70003520 con 0x7efe8c0b4680 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 msgr2=0x7efe8c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 0x7efe8c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 >> v2:172.21.15.135:3300/0 conn(0x7efe8c06ec00 msgr2=0x7efe8c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 --2- 172.21.15.97:0/2210277469 >> v2:172.21.15.135:3300/0 conn(0x7efe8c06ec00 0x7efe8c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0b2cf0 con 0x7efe8c0b4680 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7efe78001670 con 0x7efe8c0b4680 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 >> v1:172.21.15.97:6789/0 conn(0x7efe8c0b4680 legacy=0x7efe8c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe70004800 0x7efe70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 0x7efe700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c0b2a30 con 0x7efe700039b0 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c0b4360 con 0x7efe70004800 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7000ba10 con 0x7efe8c15acb0 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe8aa00640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe8aa00640 1 -- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 msgr2=0x7efe700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe8aa00640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c15acb0 0x7efe700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe8aa00640 1 -- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe70004800 msgr2=0x7efe70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe8aa00640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe70004800 0x7efe70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.657+0000 7efe8aa00640 1 -- 172.21.15.97:0/2210277469 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70003520 con 0x7efe700039b0 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe8aa00640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe70003dd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009ac0 tx=0x7efe70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86449013e58df567 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c039070 con 0x7efe700039b0 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c02c710 con 0x7efe700039b0 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe89600640 1 -- 172.21.15.97:0/2210277469 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c009690 con 0x7efe700039b0 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe91a00640 1 -- 172.21.15.97:0/2210277469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 msgr2=0x7efe70003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe91a00640 1 --2- 172.21.15.97:0/2210277469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe70003dd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009ac0 tx=0x7efe70005830 comp rx=0 tx=0).stop 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.659+0000 7efe8b400640 1 -- 172.21.15.97:0/2210277469 reap_dead start 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.660+0000 7efe91a00640 1 -- 172.21.15.97:0/2210277469 shutdown_connections 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.660+0000 7efe91a00640 1 -- 172.21.15.97:0/2210277469 >> 172.21.15.97:0/2210277469 conn(0x7efe8c0b1730 msgr2=0x7efe8c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.660+0000 7efe91a00640 1 -- 172.21.15.97:0/2210277469 shutdown_connections 2024-03-06T07:28:55.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.660+0000 7efe91a00640 1 -- 172.21.15.97:0/2210277469 wait complete. 2024-03-06T07:28:55.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.661+0000 7efe91a00640 1 Processor -- start 2024-03-06T07:28:55.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.662+0000 7efe91a00640 1 -- start start 2024-03-06T07:28:55.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.662+0000 7efe91a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe8c14c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.662+0000 7efe91a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c1554d0 0x7efe8c155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.662+0000 7efe91a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe8c151320 0x7efe8c1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe91a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c0b4010 con 0x7efe700039b0 2024-03-06T07:28:55.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe91a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c0b32b0 con 0x7efe8c151320 2024-03-06T07:28:55.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe91a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c072340 con 0x7efe8c1554d0 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe8c14c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe8c14c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53190/0 (socket says 172.21.15.97:53190) 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 -- 172.21.15.97:0/3372699591 learned_addr learned my addr 172.21.15.97:0/3372699591 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 -- 172.21.15.97:0/3372699591 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c1554d0 msgr2=0x7efe8c155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c1554d0 0x7efe8c155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 -- 172.21.15.97:0/3372699591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe8c151320 msgr2=0x7efe8c1517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe8c151320 0x7efe8c1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 -- 172.21.15.97:0/3372699591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c14e700 con 0x7efe700039b0 2024-03-06T07:28:55.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.663+0000 7efe8aa00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe8c14c4a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efe7c037040 tx=0x7efe7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.664+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c039050 con 0x7efe700039b0 2024-03-06T07:28:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.664+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c045070 con 0x7efe700039b0 2024-03-06T07:28:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.664+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c003a50 con 0x7efe700039b0 2024-03-06T07:28:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.664+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8c14e9e0 con 0x7efe700039b0 2024-03-06T07:28:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.664+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8c14efa0 con 0x7efe700039b0 2024-03-06T07:28:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.665+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8c155e90 con 0x7efe700039b0 2024-03-06T07:28:55.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.666+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe7c003bf0 con 0x7efe700039b0 2024-03-06T07:28:55.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.667+0000 7efe82a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe6c06b4d0 0x7efe6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.667+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe7c0abe60 con 0x7efe700039b0 2024-03-06T07:28:55.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.667+0000 7efe8a000640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe6c06b4d0 0x7efe6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.668+0000 7efe8a000640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe6c06b4d0 0x7efe6c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efe8c0b1700 tx=0x7efe780014b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.671+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe7c04e020 con 0x7efe700039b0 2024-03-06T07:28:55.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.853+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe8c15e0b0 con 0x7efe700039b0 2024-03-06T07:28:55.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.856+0000 7efe82a00640 1 -- 172.21.15.97:0/3372699591 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16657 (secure 0 0 0) 0x7efe8c15e0b0 con 0x7efe700039b0 2024-03-06T07:28:55.858 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:55.858 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":35,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:52.884546+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:55.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe6c06b4d0 msgr2=0x7efe6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe6c06b4d0 0x7efe6c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efe8c0b1700 tx=0x7efe780014b0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 msgr2=0x7efe8c14c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe8c14c4a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efe7c037040 tx=0x7efe7c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 shutdown_connections 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe6c06b4d0 0x7efe6c06d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe8c151320 0x7efe8c1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe8c1554d0 0x7efe8c155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 --2- 172.21.15.97:0/3372699591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe700039b0 0x7efe8c14c4a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 >> 172.21.15.97:0/3372699591 conn(0x7efe8c0b1730 msgr2=0x7efe8c06e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 shutdown_connections 2024-03-06T07:28:55.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:55.858+0000 7efe91a00640 1 -- 172.21.15.97:0/3372699591 wait complete. 2024-03-06T07:28:55.877 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:55.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:56.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.062+0000 7fccd1e00640 1 Processor -- start 2024-03-06T07:28:56.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.062+0000 7fccd1e00640 1 -- start start 2024-03-06T07:28:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccd1e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcccc0b75b0 0x7fcccc0b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccd1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcccc0b5020 0x7fcccc0b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccd1e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0b5e00 con 0x7fcccc0cff20 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccd1e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0b6a10 con 0x7fcccc0b5020 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccd1e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0b60e0 con 0x7fcccc0b75b0 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcccc0b5020 0x7fcccc0b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcccc0b5020 0x7fcccc0b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53202/0 (socket says 172.21.15.97:53202) 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 -- 172.21.15.97:0/1893796711 learned_addr learned my addr 172.21.15.97:0/1893796711 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccca000640 1 --2- 172.21.15.97:0/1893796711 >> v2:172.21.15.135:3300/0 conn(0x7fcccc0b75b0 0x7fcccc0b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 -- 172.21.15.97:0/1893796711 >> v2:172.21.15.135:3300/0 conn(0x7fcccc0b75b0 msgr2=0x7fcccc0b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 --2- 172.21.15.97:0/1893796711 >> v2:172.21.15.135:3300/0 conn(0x7fcccc0b75b0 0x7fcccc0b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 -- 172.21.15.97:0/1893796711 >> v1:172.21.15.135:6789/0 conn(0x7fcccc0cff20 legacy=0x7fcccc0b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fcccb400640 1 -- 172.21.15.97:0/1893796711 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0b63c0 con 0x7fcccc0b5020 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.064+0000 7fccca000640 1 --2- 172.21.15.97:0/1893796711 >> v2:172.21.15.135:3300/0 conn(0x7fcccc0b75b0 0x7fcccc0b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.065+0000 7fcccb400640 1 --2- 172.21.15.97:0/1893796711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0b5020 0x7fcccc0b54b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0d1550 tx=0x7fccc402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c3c0483ffd14a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccc9600640 1 -- 172.21.15.97:0/1893796711 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc402c9f0 con 0x7fcccc0b5020 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccc9600640 1 -- 172.21.15.97:0/1893796711 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc402cb90 con 0x7fcccc0b5020 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccc9600640 1 -- 172.21.15.97:0/1893796711 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc40343f0 con 0x7fcccc0b5020 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 -- 172.21.15.97:0/1893796711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0b5020 msgr2=0x7fcccc0b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1893796711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0b5020 0x7fcccc0b54b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0d1550 tx=0x7fccc402be50 comp rx=0 tx=0).stop 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 -- 172.21.15.97:0/1893796711 shutdown_connections 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1893796711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0b5020 0x7fcccc0b54b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1893796711 >> v2:172.21.15.135:3300/0 conn(0x7fcccc0b75b0 0x7fcccc0b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 -- 172.21.15.97:0/1893796711 >> 172.21.15.97:0/1893796711 conn(0x7fcccc0d1c90 msgr2=0x7fcccc0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.066+0000 7fccd1e00640 1 -- 172.21.15.97:0/1893796711 shutdown_connections 2024-03-06T07:28:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.067+0000 7fccd1e00640 1 -- 172.21.15.97:0/1893796711 wait complete. 2024-03-06T07:28:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.068+0000 7fccd1e00640 1 Processor -- start 2024-03-06T07:28:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.068+0000 7fccd1e00640 1 -- start start 2024-03-06T07:28:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fccd1e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcccc0b5020 0x7fcccc1e7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fccd1e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc0b75b0 0x7fcccc1ea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fccd1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 0x7fcccc1e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fccd1e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0b7a30 con 0x7fcccc0cff20 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fccd1e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc161a50 con 0x7fcccc0b75b0 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fccd1e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0b7bb0 con 0x7fcccc0b5020 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fcccb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 0x7fcccc1e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fcccb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 0x7fcccc1e8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53206/0 (socket says 172.21.15.97:53206) 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.069+0000 7fcccb400640 1 -- 172.21.15.97:0/1450390452 learned_addr learned my addr 172.21.15.97:0/1450390452 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccaa00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcccc0b5020 0x7fcccc1e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fccca000640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc0b75b0 0x7fcccc1ea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccb400640 1 -- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcccc0b5020 msgr2=0x7fcccc1e7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccb400640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcccc0b5020 0x7fcccc1e7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccb400640 1 -- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc0b75b0 msgr2=0x7fcccc1ea0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccb400640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc0b75b0 0x7fcccc1ea0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccb400640 1 -- 172.21.15.97:0/1450390452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc1e7f30 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fcccb400640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 0x7fcccc1e8fd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fccc4036040 tx=0x7fccc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4043020 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc403e450 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc1e8210 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc403e860 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.070+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc1e87d0 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.072+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc04f330 con 0x7fcccc0cff20 2024-03-06T07:28:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.072+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccc4046020 con 0x7fcccc0cff20 2024-03-06T07:28:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.074+0000 7fccc3400640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccac06b500 0x7fccac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.074+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccc40ac8a0 con 0x7fcccc0cff20 2024-03-06T07:28:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.074+0000 7fcccaa00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccac06b500 0x7fccac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.075+0000 7fcccaa00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccac06b500 0x7fccac06d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0015d0 tx=0x7fccbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.078+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccc404d070 con 0x7fcccc0cff20 2024-03-06T07:28:56.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.255+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcccc0b4bd0 con 0x7fccac06b500 2024-03-06T07:28:56.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.264+0000 7fccc3400640 1 -- 172.21.15.97:0/1450390452 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251950 (secure 0 0 0) 0x7fcccc0b4bd0 con 0x7fccac06b500 2024-03-06T07:28:56.270 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccac06b500 msgr2=0x7fccac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccac06b500 0x7fccac06d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0015d0 tx=0x7fccbc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 msgr2=0x7fcccc1e8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 0x7fcccc1e8fd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fccc4036040 tx=0x7fccc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 shutdown_connections 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccac06b500 0x7fccac06d9c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcccc0cff20 0x7fcccc1e8fd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc0b75b0 0x7fcccc1ea0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 --2- 172.21.15.97:0/1450390452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcccc0b5020 0x7fcccc1e7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 >> 172.21.15.97:0/1450390452 conn(0x7fcccc0d1c90 msgr2=0x7fcccc06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 shutdown_connections 2024-03-06T07:28:56.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.268+0000 7fccd1e00640 1 -- 172.21.15.97:0/1450390452 wait complete. 2024-03-06T07:28:56.286 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-06T07:28:54.922927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220476,"kb_used_data":4284,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754244,"statfs":{"total":773094113280,"available":772868345856,"internally_reserved":0,"allocated":4386816,"data_stored":2419992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.073936"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986403+0000","last_change":"2024-03-06T07:28:51.920469+0000","last_active":"2024-03-06T07:28:52.986403+0000","last_peered":"2024-03-06T07:28:52.986403+0000","last_clean":"2024-03-06T07:28:52.986403+0000","last_became_active":"2024-03-06T07:28:51.920150+0000","last_became_peered":"2024-03-06T07:28:51.920150+0000","last_unstale":"2024-03-06T07:28:52.986403+0000","last_undegraded":"2024-03-06T07:28:52.986403+0000","last_fullsized":"2024-03-06T07:28:52.986403+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:36.766617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904122+0000","last_change":"2024-03-06T07:28:51.916476+0000","last_active":"2024-03-06T07:28:52.904122+0000","last_peered":"2024-03-06T07:28:52.904122+0000","last_clean":"2024-03-06T07:28:52.904122+0000","last_became_active":"2024-03-06T07:28:51.915640+0000","last_became_peered":"2024-03-06T07:28:51.915640+0000","last_unstale":"2024-03-06T07:28:52.904122+0000","last_undegraded":"2024-03-06T07:28:52.904122+0000","last_fullsized":"2024-03-06T07:28:52.904122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:29.996692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982384+0000","last_change":"2024-03-06T07:28:51.925809+0000","last_active":"2024-03-06T07:28:52.982384+0000","last_peered":"2024-03-06T07:28:52.982384+0000","last_clean":"2024-03-06T07:28:52.982384+0000","last_became_active":"2024-03-06T07:28:51.925495+0000","last_became_peered":"2024-03-06T07:28:51.925495+0000","last_unstale":"2024-03-06T07:28:52.982384+0000","last_undegraded":"2024-03-06T07:28:52.982384+0000","last_fullsized":"2024-03-06T07:28:52.982384+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:23.627303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899915+0000","last_change":"2024-03-06T07:28:51.903797+0000","last_active":"2024-03-06T07:28:52.899915+0000","last_peered":"2024-03-06T07:28:52.899915+0000","last_clean":"2024-03-06T07:28:52.899915+0000","last_became_active":"2024-03-06T07:28:51.903522+0000","last_became_peered":"2024-03-06T07:28:51.903522+0000","last_unstale":"2024-03-06T07:28:52.899915+0000","last_undegraded":"2024-03-06T07:28:52.899915+0000","last_fullsized":"2024-03-06T07:28:52.899915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:08.868206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980084+0000","last_change":"2024-03-06T07:28:51.905461+0000","last_active":"2024-03-06T07:28:52.980084+0000","last_peered":"2024-03-06T07:28:52.980084+0000","last_clean":"2024-03-06T07:28:52.980084+0000","last_became_active":"2024-03-06T07:28:51.905194+0000","last_became_peered":"2024-03-06T07:28:51.905194+0000","last_unstale":"2024-03-06T07:28:52.980084+0000","last_undegraded":"2024-03-06T07:28:52.980084+0000","last_fullsized":"2024-03-06T07:28:52.980084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:36.318818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903666+0000","last_change":"2024-03-06T07:28:51.918829+0000","last_active":"2024-03-06T07:28:52.903666+0000","last_peered":"2024-03-06T07:28:52.903666+0000","last_clean":"2024-03-06T07:28:52.903666+0000","last_became_active":"2024-03-06T07:28:51.918526+0000","last_became_peered":"2024-03-06T07:28:51.918526+0000","last_unstale":"2024-03-06T07:28:52.903666+0000","last_undegraded":"2024-03-06T07:28:52.903666+0000","last_fullsized":"2024-03-06T07:28:52.903666+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.249279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906197+0000","last_change":"2024-03-06T07:28:51.910217+0000","last_active":"2024-03-06T07:28:52.906197+0000","last_peered":"2024-03-06T07:28:52.906197+0000","last_clean":"2024-03-06T07:28:52.906197+0000","last_became_active":"2024-03-06T07:28:51.909923+0000","last_became_peered":"2024-03-06T07:28:51.909923+0000","last_unstale":"2024-03-06T07:28:52.906197+0000","last_undegraded":"2024-03-06T07:28:52.906197+0000","last_fullsized":"2024-03-06T07:28:52.906197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:40.472650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905066+0000","last_change":"2024-03-06T07:28:51.918529+0000","last_active":"2024-03-06T07:28:52.905066+0000","last_peered":"2024-03-06T07:28:52.905066+0000","last_clean":"2024-03-06T07:28:52.905066+0000","last_became_active":"2024-03-06T07:28:51.917948+0000","last_became_peered":"2024-03-06T07:28:51.917948+0000","last_unstale":"2024-03-06T07:28:52.905066+0000","last_undegraded":"2024-03-06T07:28:52.905066+0000","last_fullsized":"2024-03-06T07:28:52.905066+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:33.018003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987760+0000","last_change":"2024-03-06T07:28:51.898806+0000","last_active":"2024-03-06T07:28:52.987760+0000","last_peered":"2024-03-06T07:28:52.987760+0000","last_clean":"2024-03-06T07:28:52.987760+0000","last_became_active":"2024-03-06T07:28:51.897770+0000","last_became_peered":"2024-03-06T07:28:51.897770+0000","last_unstale":"2024-03-06T07:28:52.987760+0000","last_undegraded":"2024-03-06T07:28:52.987760+0000","last_fullsized":"2024-03-06T07:28:52.987760+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:31.024451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900210+0000","last_change":"2024-03-06T07:28:51.909932+0000","last_active":"2024-03-06T07:28:52.900210+0000","last_peered":"2024-03-06T07:28:52.900210+0000","last_clean":"2024-03-06T07:28:52.900210+0000","last_became_active":"2024-03-06T07:28:51.909656+0000","last_became_peered":"2024-03-06T07:28:51.909656+0000","last_unstale":"2024-03-06T07:28:52.900210+0000","last_undegraded":"2024-03-06T07:28:52.900210+0000","last_fullsized":"2024-03-06T07:28:52.900210+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:33.018003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906146+0000","last_change":"2024-03-06T07:28:51.910123+0000","last_active":"2024-03-06T07:28:52.906146+0000","last_peered":"2024-03-06T07:28:52.906146+0000","last_clean":"2024-03-06T07:28:52.906146+0000","last_became_active":"2024-03-06T07:28:51.909823+0000","last_became_peered":"2024-03-06T07:28:51.909823+0000","last_unstale":"2024-03-06T07:28:52.906146+0000","last_undegraded":"2024-03-06T07:28:52.906146+0000","last_fullsized":"2024-03-06T07:28:52.906146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.396135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.984271+0000","last_change":"2024-03-06T07:28:51.919134+0000","last_active":"2024-03-06T07:28:52.984271+0000","last_peered":"2024-03-06T07:28:52.984271+0000","last_clean":"2024-03-06T07:28:52.984271+0000","last_became_active":"2024-03-06T07:28:51.918897+0000","last_became_peered":"2024-03-06T07:28:51.918897+0000","last_unstale":"2024-03-06T07:28:52.984271+0000","last_undegraded":"2024-03-06T07:28:52.984271+0000","last_fullsized":"2024-03-06T07:28:52.984271+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.249279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980096+0000","last_change":"2024-03-06T07:28:51.904581+0000","last_active":"2024-03-06T07:28:52.980096+0000","last_peered":"2024-03-06T07:28:52.980096+0000","last_clean":"2024-03-06T07:28:52.980096+0000","last_became_active":"2024-03-06T07:28:51.902855+0000","last_became_peered":"2024-03-06T07:28:51.902855+0000","last_unstale":"2024-03-06T07:28:52.980096+0000","last_undegraded":"2024-03-06T07:28:52.980096+0000","last_fullsized":"2024-03-06T07:28:52.980096+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:30.580365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903700+0000","last_change":"2024-03-06T07:28:51.901097+0000","last_active":"2024-03-06T07:28:52.903700+0000","last_peered":"2024-03-06T07:28:52.903700+0000","last_clean":"2024-03-06T07:28:52.903700+0000","last_became_active":"2024-03-06T07:28:51.900775+0000","last_became_peered":"2024-03-06T07:28:51.900775+0000","last_unstale":"2024-03-06T07:28:52.903700+0000","last_undegraded":"2024-03-06T07:28:52.903700+0000","last_fullsized":"2024-03-06T07:28:52.903700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:40.472650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982311+0000","last_change":"2024-03-06T07:28:51.915312+0000","last_active":"2024-03-06T07:28:52.982311+0000","last_peered":"2024-03-06T07:28:52.982311+0000","last_clean":"2024-03-06T07:28:52.982311+0000","last_became_active":"2024-03-06T07:28:51.914936+0000","last_became_peered":"2024-03-06T07:28:51.914936+0000","last_unstale":"2024-03-06T07:28:52.982311+0000","last_undegraded":"2024-03-06T07:28:52.982311+0000","last_fullsized":"2024-03-06T07:28:52.982311+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:57.146808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905991+0000","last_change":"2024-03-06T07:28:51.911752+0000","last_active":"2024-03-06T07:28:52.905991+0000","last_peered":"2024-03-06T07:28:52.905991+0000","last_clean":"2024-03-06T07:28:52.905991+0000","last_became_active":"2024-03-06T07:28:51.911420+0000","last_became_peered":"2024-03-06T07:28:51.911420+0000","last_unstale":"2024-03-06T07:28:52.905991+0000","last_undegraded":"2024-03-06T07:28:52.905991+0000","last_fullsized":"2024-03-06T07:28:52.905991+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:25.099777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906226+0000","last_change":"2024-03-06T07:28:51.914773+0000","last_active":"2024-03-06T07:28:52.906226+0000","last_peered":"2024-03-06T07:28:52.906226+0000","last_clean":"2024-03-06T07:28:52.906226+0000","last_became_active":"2024-03-06T07:28:51.913086+0000","last_became_peered":"2024-03-06T07:28:51.913086+0000","last_unstale":"2024-03-06T07:28:52.906226+0000","last_undegraded":"2024-03-06T07:28:52.906226+0000","last_fullsized":"2024-03-06T07:28:52.906226+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:10.489955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904214+0000","last_change":"2024-03-06T07:28:51.911335+0000","last_active":"2024-03-06T07:28:52.904214+0000","last_peered":"2024-03-06T07:28:52.904214+0000","last_clean":"2024-03-06T07:28:52.904214+0000","last_became_active":"2024-03-06T07:28:51.911031+0000","last_became_peered":"2024-03-06T07:28:51.911031+0000","last_unstale":"2024-03-06T07:28:52.904214+0000","last_undegraded":"2024-03-06T07:28:52.904214+0000","last_fullsized":"2024-03-06T07:28:52.904214+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:39.554357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903787+0000","last_change":"2024-03-06T07:28:51.916519+0000","last_active":"2024-03-06T07:28:52.903787+0000","last_peered":"2024-03-06T07:28:52.903787+0000","last_clean":"2024-03-06T07:28:52.903787+0000","last_became_active":"2024-03-06T07:28:51.914227+0000","last_became_peered":"2024-03-06T07:28:51.914227+0000","last_unstale":"2024-03-06T07:28:52.903787+0000","last_undegraded":"2024-03-06T07:28:52.903787+0000","last_fullsized":"2024-03-06T07:28:52.903787+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.396135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980094+0000","last_change":"2024-03-06T07:28:51.922304+0000","last_active":"2024-03-06T07:28:52.980094+0000","last_peered":"2024-03-06T07:28:52.980094+0000","last_clean":"2024-03-06T07:28:52.980094+0000","last_became_active":"2024-03-06T07:28:51.921998+0000","last_became_peered":"2024-03-06T07:28:51.921998+0000","last_unstale":"2024-03-06T07:28:52.980094+0000","last_undegraded":"2024-03-06T07:28:52.980094+0000","last_fullsized":"2024-03-06T07:28:52.980094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:06.222398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986373+0000","last_change":"2024-03-06T07:28:51.895536+0000","last_active":"2024-03-06T07:28:52.986373+0000","last_peered":"2024-03-06T07:28:52.986373+0000","last_clean":"2024-03-06T07:28:52.986373+0000","last_became_active":"2024-03-06T07:28:51.893536+0000","last_became_peered":"2024-03-06T07:28:51.893536+0000","last_unstale":"2024-03-06T07:28:52.986373+0000","last_undegraded":"2024-03-06T07:28:52.986373+0000","last_fullsized":"2024-03-06T07:28:52.986373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:45.952071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986440+0000","last_change":"2024-03-06T07:28:51.904074+0000","last_active":"2024-03-06T07:28:52.986440+0000","last_peered":"2024-03-06T07:28:52.986440+0000","last_clean":"2024-03-06T07:28:52.986440+0000","last_became_active":"2024-03-06T07:28:51.903788+0000","last_became_peered":"2024-03-06T07:28:51.903788+0000","last_unstale":"2024-03-06T07:28:52.986440+0000","last_undegraded":"2024-03-06T07:28:52.986440+0000","last_fullsized":"2024-03-06T07:28:52.986440+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:10.523665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904817+0000","last_change":"2024-03-06T07:28:51.904045+0000","last_active":"2024-03-06T07:28:52.904817+0000","last_peered":"2024-03-06T07:28:52.904817+0000","last_clean":"2024-03-06T07:28:52.904817+0000","last_became_active":"2024-03-06T07:28:51.903647+0000","last_became_peered":"2024-03-06T07:28:51.903647+0000","last_unstale":"2024-03-06T07:28:52.904817+0000","last_undegraded":"2024-03-06T07:28:52.904817+0000","last_fullsized":"2024-03-06T07:28:52.904817+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:36.766617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981977+0000","last_change":"2024-03-06T07:28:51.917904+0000","last_active":"2024-03-06T07:28:52.981977+0000","last_peered":"2024-03-06T07:28:52.981977+0000","last_clean":"2024-03-06T07:28:52.981977+0000","last_became_active":"2024-03-06T07:28:51.916508+0000","last_became_peered":"2024-03-06T07:28:51.916508+0000","last_unstale":"2024-03-06T07:28:52.981977+0000","last_undegraded":"2024-03-06T07:28:52.981977+0000","last_fullsized":"2024-03-06T07:28:52.981977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.696454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900465+0000","last_change":"2024-03-06T07:28:51.908930+0000","last_active":"2024-03-06T07:28:52.900465+0000","last_peered":"2024-03-06T07:28:52.900465+0000","last_clean":"2024-03-06T07:28:52.900465+0000","last_became_active":"2024-03-06T07:28:51.908569+0000","last_became_peered":"2024-03-06T07:28:51.908569+0000","last_unstale":"2024-03-06T07:28:52.900465+0000","last_undegraded":"2024-03-06T07:28:52.900465+0000","last_fullsized":"2024-03-06T07:28:52.900465+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.696454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904271+0000","last_change":"2024-03-06T07:28:51.912062+0000","last_active":"2024-03-06T07:28:52.904271+0000","last_peered":"2024-03-06T07:28:52.904271+0000","last_clean":"2024-03-06T07:28:52.904271+0000","last_became_active":"2024-03-06T07:28:51.911779+0000","last_became_peered":"2024-03-06T07:28:51.911779+0000","last_unstale":"2024-03-06T07:28:52.904271+0000","last_undegraded":"2024-03-06T07:28:52.904271+0000","last_fullsized":"2024-03-06T07:28:52.904271+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:11.145501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900374+0000","last_change":"2024-03-06T07:28:51.902550+0000","last_active":"2024-03-06T07:28:52.900374+0000","last_peered":"2024-03-06T07:28:52.900374+0000","last_clean":"2024-03-06T07:28:52.900374+0000","last_became_active":"2024-03-06T07:28:51.899321+0000","last_became_peered":"2024-03-06T07:28:51.899321+0000","last_unstale":"2024-03-06T07:28:52.900374+0000","last_undegraded":"2024-03-06T07:28:52.900374+0000","last_fullsized":"2024-03-06T07:28:52.900374+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:36.424334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.984511+0000","last_change":"2024-03-06T07:28:51.918223+0000","last_active":"2024-03-06T07:28:52.984511+0000","last_peered":"2024-03-06T07:28:52.984511+0000","last_clean":"2024-03-06T07:28:52.984511+0000","last_became_active":"2024-03-06T07:28:51.917322+0000","last_became_peered":"2024-03-06T07:28:51.917322+0000","last_unstale":"2024-03-06T07:28:52.984511+0000","last_undegraded":"2024-03-06T07:28:52.984511+0000","last_fullsized":"2024-03-06T07:28:52.984511+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:40.472650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986652+0000","last_change":"2024-03-06T07:28:51.909921+0000","last_active":"2024-03-06T07:28:52.986652+0000","last_peered":"2024-03-06T07:28:52.986652+0000","last_clean":"2024-03-06T07:28:52.986652+0000","last_became_active":"2024-03-06T07:28:51.908545+0000","last_became_peered":"2024-03-06T07:28:51.908545+0000","last_unstale":"2024-03-06T07:28:52.986652+0000","last_undegraded":"2024-03-06T07:28:52.986652+0000","last_fullsized":"2024-03-06T07:28:52.986652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:04.552921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906075+0000","last_change":"2024-03-06T07:28:51.908632+0000","last_active":"2024-03-06T07:28:52.906075+0000","last_peered":"2024-03-06T07:28:52.906075+0000","last_clean":"2024-03-06T07:28:52.906075+0000","last_became_active":"2024-03-06T07:28:51.904666+0000","last_became_peered":"2024-03-06T07:28:51.904666+0000","last_unstale":"2024-03-06T07:28:52.906075+0000","last_undegraded":"2024-03-06T07:28:52.906075+0000","last_fullsized":"2024-03-06T07:28:52.906075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.249279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983613+0000","last_change":"2024-03-06T07:28:51.907793+0000","last_active":"2024-03-06T07:28:52.983613+0000","last_peered":"2024-03-06T07:28:52.983613+0000","last_clean":"2024-03-06T07:28:52.983613+0000","last_became_active":"2024-03-06T07:28:51.907137+0000","last_became_peered":"2024-03-06T07:28:51.907137+0000","last_unstale":"2024-03-06T07:28:52.983613+0000","last_undegraded":"2024-03-06T07:28:52.983613+0000","last_fullsized":"2024-03-06T07:28:52.983613+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:19.963525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905199+0000","last_change":"2024-03-06T07:28:51.898133+0000","last_active":"2024-03-06T07:28:52.905199+0000","last_peered":"2024-03-06T07:28:52.905199+0000","last_clean":"2024-03-06T07:28:52.905199+0000","last_became_active":"2024-03-06T07:28:51.897723+0000","last_became_peered":"2024-03-06T07:28:51.897723+0000","last_unstale":"2024-03-06T07:28:52.905199+0000","last_undegraded":"2024-03-06T07:28:52.905199+0000","last_fullsized":"2024-03-06T07:28:52.905199+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:36.424334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905160+0000","last_change":"2024-03-06T07:28:51.897381+0000","last_active":"2024-03-06T07:28:52.905160+0000","last_peered":"2024-03-06T07:28:52.905160+0000","last_clean":"2024-03-06T07:28:52.905160+0000","last_became_active":"2024-03-06T07:28:51.896985+0000","last_became_peered":"2024-03-06T07:28:51.896985+0000","last_unstale":"2024-03-06T07:28:52.905160+0000","last_undegraded":"2024-03-06T07:28:52.905160+0000","last_fullsized":"2024-03-06T07:28:52.905160+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:57.146419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987005+0000","last_change":"2024-03-06T07:28:50.902394+0000","last_active":"2024-03-06T07:28:52.987005+0000","last_peered":"2024-03-06T07:28:52.987005+0000","last_clean":"2024-03-06T07:28:52.987005+0000","last_became_active":"2024-03-06T07:28:50.901925+0000","last_became_peered":"2024-03-06T07:28:50.901925+0000","last_unstale":"2024-03-06T07:28:52.987005+0000","last_undegraded":"2024-03-06T07:28:52.987005+0000","last_fullsized":"2024-03-06T07:28:52.987005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:35.035122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986843+0000","last_change":"2024-03-06T07:28:51.907709+0000","last_active":"2024-03-06T07:28:52.986843+0000","last_peered":"2024-03-06T07:28:52.986843+0000","last_clean":"2024-03-06T07:28:52.986843+0000","last_became_active":"2024-03-06T07:28:51.907378+0000","last_became_peered":"2024-03-06T07:28:51.907378+0000","last_unstale":"2024-03-06T07:28:52.986843+0000","last_undegraded":"2024-03-06T07:28:52.986843+0000","last_fullsized":"2024-03-06T07:28:52.986843+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:25.949576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905283+0000","last_change":"2024-03-06T07:28:50.894334+0000","last_active":"2024-03-06T07:28:52.905283+0000","last_peered":"2024-03-06T07:28:52.905283+0000","last_clean":"2024-03-06T07:28:52.905283+0000","last_became_active":"2024-03-06T07:28:50.894076+0000","last_became_peered":"2024-03-06T07:28:50.894076+0000","last_unstale":"2024-03-06T07:28:52.905283+0000","last_undegraded":"2024-03-06T07:28:52.905283+0000","last_fullsized":"2024-03-06T07:28:52.905283+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.304948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986973+0000","last_change":"2024-03-06T07:28:51.908832+0000","last_active":"2024-03-06T07:28:52.986973+0000","last_peered":"2024-03-06T07:28:52.986973+0000","last_clean":"2024-03-06T07:28:52.986973+0000","last_became_active":"2024-03-06T07:28:51.908461+0000","last_became_peered":"2024-03-06T07:28:51.908461+0000","last_unstale":"2024-03-06T07:28:52.986973+0000","last_undegraded":"2024-03-06T07:28:52.986973+0000","last_fullsized":"2024-03-06T07:28:52.986973+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:04.044975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903941+0000","last_change":"2024-03-06T07:28:50.898522+0000","last_active":"2024-03-06T07:28:52.903941+0000","last_peered":"2024-03-06T07:28:52.903941+0000","last_clean":"2024-03-06T07:28:52.903941+0000","last_became_active":"2024-03-06T07:28:50.898189+0000","last_became_peered":"2024-03-06T07:28:50.898189+0000","last_unstale":"2024-03-06T07:28:52.903941+0000","last_undegraded":"2024-03-06T07:28:52.903941+0000","last_fullsized":"2024-03-06T07:28:52.903941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:21.189606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900615+0000","last_change":"2024-03-06T07:28:51.903456+0000","last_active":"2024-03-06T07:28:52.900615+0000","last_peered":"2024-03-06T07:28:52.900615+0000","last_clean":"2024-03-06T07:28:52.900615+0000","last_became_active":"2024-03-06T07:28:51.903230+0000","last_became_peered":"2024-03-06T07:28:51.903230+0000","last_unstale":"2024-03-06T07:28:52.900615+0000","last_undegraded":"2024-03-06T07:28:52.900615+0000","last_fullsized":"2024-03-06T07:28:52.900615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:57.146808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980222+0000","last_change":"2024-03-06T07:28:50.896486+0000","last_active":"2024-03-06T07:28:52.980222+0000","last_peered":"2024-03-06T07:28:52.980222+0000","last_clean":"2024-03-06T07:28:52.980222+0000","last_became_active":"2024-03-06T07:28:50.896209+0000","last_became_peered":"2024-03-06T07:28:50.896209+0000","last_unstale":"2024-03-06T07:28:52.980222+0000","last_undegraded":"2024-03-06T07:28:52.980222+0000","last_fullsized":"2024-03-06T07:28:52.980222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:19.212504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905867+0000","last_change":"2024-03-06T07:28:51.912696+0000","last_active":"2024-03-06T07:28:52.905867+0000","last_peered":"2024-03-06T07:28:52.905867+0000","last_clean":"2024-03-06T07:28:52.905867+0000","last_became_active":"2024-03-06T07:28:51.909766+0000","last_became_peered":"2024-03-06T07:28:51.909766+0000","last_unstale":"2024-03-06T07:28:52.905867+0000","last_undegraded":"2024-03-06T07:28:52.905867+0000","last_fullsized":"2024-03-06T07:28:52.905867+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.886019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986727+0000","last_change":"2024-03-06T07:28:50.899522+0000","last_active":"2024-03-06T07:28:52.986727+0000","last_peered":"2024-03-06T07:28:52.986727+0000","last_clean":"2024-03-06T07:28:52.986727+0000","last_became_active":"2024-03-06T07:28:50.899223+0000","last_became_peered":"2024-03-06T07:28:50.899223+0000","last_unstale":"2024-03-06T07:28:52.986727+0000","last_undegraded":"2024-03-06T07:28:52.986727+0000","last_fullsized":"2024-03-06T07:28:52.986727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:03.570979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982680+0000","last_change":"2024-03-06T07:28:51.917066+0000","last_active":"2024-03-06T07:28:52.982680+0000","last_peered":"2024-03-06T07:28:52.982680+0000","last_clean":"2024-03-06T07:28:52.982680+0000","last_became_active":"2024-03-06T07:28:51.916334+0000","last_became_peered":"2024-03-06T07:28:51.916334+0000","last_unstale":"2024-03-06T07:28:52.982680+0000","last_undegraded":"2024-03-06T07:28:52.982680+0000","last_fullsized":"2024-03-06T07:28:52.982680+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:25.099777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986175+0000","last_change":"2024-03-06T07:28:50.900754+0000","last_active":"2024-03-06T07:28:52.986175+0000","last_peered":"2024-03-06T07:28:52.986175+0000","last_clean":"2024-03-06T07:28:52.986175+0000","last_became_active":"2024-03-06T07:28:50.900424+0000","last_became_peered":"2024-03-06T07:28:50.900424+0000","last_unstale":"2024-03-06T07:28:52.986175+0000","last_undegraded":"2024-03-06T07:28:52.986175+0000","last_fullsized":"2024-03-06T07:28:52.986175+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:01.076475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981667+0000","last_change":"2024-03-06T07:28:51.899968+0000","last_active":"2024-03-06T07:28:52.981667+0000","last_peered":"2024-03-06T07:28:52.981667+0000","last_clean":"2024-03-06T07:28:52.981667+0000","last_became_active":"2024-03-06T07:28:51.899501+0000","last_became_peered":"2024-03-06T07:28:51.899501+0000","last_unstale":"2024-03-06T07:28:52.981667+0000","last_undegraded":"2024-03-06T07:28:52.981667+0000","last_fullsized":"2024-03-06T07:28:52.981667+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:48.134346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902758+0000","last_change":"2024-03-06T07:28:50.890559+0000","last_active":"2024-03-06T07:28:52.902758+0000","last_peered":"2024-03-06T07:28:52.902758+0000","last_clean":"2024-03-06T07:28:52.902758+0000","last_became_active":"2024-03-06T07:28:50.890328+0000","last_became_peered":"2024-03-06T07:28:50.890328+0000","last_unstale":"2024-03-06T07:28:52.902758+0000","last_undegraded":"2024-03-06T07:28:52.902758+0000","last_fullsized":"2024-03-06T07:28:52.902758+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:24.935705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899212+0000","last_change":"2024-03-06T07:28:51.916840+0000","last_active":"2024-03-06T07:28:52.899212+0000","last_peered":"2024-03-06T07:28:52.899212+0000","last_clean":"2024-03-06T07:28:52.899212+0000","last_became_active":"2024-03-06T07:28:51.916548+0000","last_became_peered":"2024-03-06T07:28:51.916548+0000","last_unstale":"2024-03-06T07:28:52.899212+0000","last_undegraded":"2024-03-06T07:28:52.899212+0000","last_fullsized":"2024-03-06T07:28:52.899212+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:59.568375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981538+0000","last_change":"2024-03-06T07:28:50.897537+0000","last_active":"2024-03-06T07:28:52.981538+0000","last_peered":"2024-03-06T07:28:52.981538+0000","last_clean":"2024-03-06T07:28:52.981538+0000","last_became_active":"2024-03-06T07:28:50.897170+0000","last_became_peered":"2024-03-06T07:28:50.897170+0000","last_unstale":"2024-03-06T07:28:52.981538+0000","last_undegraded":"2024-03-06T07:28:52.981538+0000","last_fullsized":"2024-03-06T07:28:52.981538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:00.079347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982636+0000","last_change":"2024-03-06T07:28:51.925668+0000","last_active":"2024-03-06T07:28:52.982636+0000","last_peered":"2024-03-06T07:28:52.982636+0000","last_clean":"2024-03-06T07:28:52.982636+0000","last_became_active":"2024-03-06T07:28:51.925217+0000","last_became_peered":"2024-03-06T07:28:51.925217+0000","last_unstale":"2024-03-06T07:28:52.982636+0000","last_undegraded":"2024-03-06T07:28:52.982636+0000","last_fullsized":"2024-03-06T07:28:52.982636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.886019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904484+0000","last_change":"2024-03-06T07:28:50.897874+0000","last_active":"2024-03-06T07:28:52.904484+0000","last_peered":"2024-03-06T07:28:52.904484+0000","last_clean":"2024-03-06T07:28:52.904484+0000","last_became_active":"2024-03-06T07:28:50.897584+0000","last_became_peered":"2024-03-06T07:28:50.897584+0000","last_unstale":"2024-03-06T07:28:52.904484+0000","last_undegraded":"2024-03-06T07:28:52.904484+0000","last_fullsized":"2024-03-06T07:28:52.904484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:04.116946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904541+0000","last_change":"2024-03-06T07:28:51.914505+0000","last_active":"2024-03-06T07:28:52.904541+0000","last_peered":"2024-03-06T07:28:52.904541+0000","last_clean":"2024-03-06T07:28:52.904541+0000","last_became_active":"2024-03-06T07:28:51.914176+0000","last_became_peered":"2024-03-06T07:28:51.914176+0000","last_unstale":"2024-03-06T07:28:52.904541+0000","last_undegraded":"2024-03-06T07:28:52.904541+0000","last_fullsized":"2024-03-06T07:28:52.904541+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:09.729567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905365+0000","last_change":"2024-03-06T07:28:50.898049+0000","last_active":"2024-03-06T07:28:52.905365+0000","last_peered":"2024-03-06T07:28:52.905365+0000","last_clean":"2024-03-06T07:28:52.905365+0000","last_became_active":"2024-03-06T07:28:50.896854+0000","last_became_peered":"2024-03-06T07:28:50.896854+0000","last_unstale":"2024-03-06T07:28:52.905365+0000","last_undegraded":"2024-03-06T07:28:52.905365+0000","last_fullsized":"2024-03-06T07:28:52.905365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:29.566494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903821+0000","last_change":"2024-03-06T07:28:51.919644+0000","last_active":"2024-03-06T07:28:52.903821+0000","last_peered":"2024-03-06T07:28:52.903821+0000","last_clean":"2024-03-06T07:28:52.903821+0000","last_became_active":"2024-03-06T07:28:51.915283+0000","last_became_peered":"2024-03-06T07:28:51.915283+0000","last_unstale":"2024-03-06T07:28:52.903821+0000","last_undegraded":"2024-03-06T07:28:52.903821+0000","last_fullsized":"2024-03-06T07:28:52.903821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:10.489955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982711+0000","last_change":"2024-03-06T07:28:50.896900+0000","last_active":"2024-03-06T07:28:52.982711+0000","last_peered":"2024-03-06T07:28:52.982711+0000","last_clean":"2024-03-06T07:28:52.982711+0000","last_became_active":"2024-03-06T07:28:50.895714+0000","last_became_peered":"2024-03-06T07:28:50.895714+0000","last_unstale":"2024-03-06T07:28:52.982711+0000","last_undegraded":"2024-03-06T07:28:52.982711+0000","last_fullsized":"2024-03-06T07:28:52.982711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:08.878663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981967+0000","last_change":"2024-03-06T07:28:51.917945+0000","last_active":"2024-03-06T07:28:52.981967+0000","last_peered":"2024-03-06T07:28:52.981967+0000","last_clean":"2024-03-06T07:28:52.981967+0000","last_became_active":"2024-03-06T07:28:51.917631+0000","last_became_peered":"2024-03-06T07:28:51.917631+0000","last_unstale":"2024-03-06T07:28:52.981967+0000","last_undegraded":"2024-03-06T07:28:52.981967+0000","last_fullsized":"2024-03-06T07:28:52.981967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:36.424334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903942+0000","last_change":"2024-03-06T07:28:50.882756+0000","last_active":"2024-03-06T07:28:52.903942+0000","last_peered":"2024-03-06T07:28:52.903942+0000","last_clean":"2024-03-06T07:28:52.903942+0000","last_became_active":"2024-03-06T07:28:50.882453+0000","last_became_peered":"2024-03-06T07:28:50.882453+0000","last_unstale":"2024-03-06T07:28:52.903942+0000","last_undegraded":"2024-03-06T07:28:52.903942+0000","last_fullsized":"2024-03-06T07:28:52.903942+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:07.854336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899569+0000","last_change":"2024-03-06T07:28:51.923593+0000","last_active":"2024-03-06T07:28:52.899569+0000","last_peered":"2024-03-06T07:28:52.899569+0000","last_clean":"2024-03-06T07:28:52.899569+0000","last_became_active":"2024-03-06T07:28:51.920647+0000","last_became_peered":"2024-03-06T07:28:51.920647+0000","last_unstale":"2024-03-06T07:28:52.899569+0000","last_undegraded":"2024-03-06T07:28:52.899569+0000","last_fullsized":"2024-03-06T07:28:52.899569+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:05.130817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986956+0000","last_change":"2024-03-06T07:28:50.897541+0000","last_active":"2024-03-06T07:28:52.986956+0000","last_peered":"2024-03-06T07:28:52.986956+0000","last_clean":"2024-03-06T07:28:52.986956+0000","last_became_active":"2024-03-06T07:28:50.897255+0000","last_became_peered":"2024-03-06T07:28:50.897255+0000","last_unstale":"2024-03-06T07:28:52.986956+0000","last_undegraded":"2024-03-06T07:28:52.986956+0000","last_fullsized":"2024-03-06T07:28:52.986956+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.096128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903950+0000","last_change":"2024-03-06T07:28:51.908136+0000","last_active":"2024-03-06T07:28:52.903950+0000","last_peered":"2024-03-06T07:28:52.903950+0000","last_clean":"2024-03-06T07:28:52.903950+0000","last_became_active":"2024-03-06T07:28:51.907097+0000","last_became_peered":"2024-03-06T07:28:51.907097+0000","last_unstale":"2024-03-06T07:28:52.903950+0000","last_undegraded":"2024-03-06T07:28:52.903950+0000","last_fullsized":"2024-03-06T07:28:52.903950+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:25.674172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899471+0000","last_change":"2024-03-06T07:28:50.900478+0000","last_active":"2024-03-06T07:28:52.899471+0000","last_peered":"2024-03-06T07:28:52.899471+0000","last_clean":"2024-03-06T07:28:52.899471+0000","last_became_active":"2024-03-06T07:28:50.898772+0000","last_became_peered":"2024-03-06T07:28:50.898772+0000","last_unstale":"2024-03-06T07:28:52.899471+0000","last_undegraded":"2024-03-06T07:28:52.899471+0000","last_fullsized":"2024-03-06T07:28:52.899471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:14.947717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905637+0000","last_change":"2024-03-06T07:28:51.898634+0000","last_active":"2024-03-06T07:28:52.905637+0000","last_peered":"2024-03-06T07:28:52.905637+0000","last_clean":"2024-03-06T07:28:52.905637+0000","last_became_active":"2024-03-06T07:28:51.898265+0000","last_became_peered":"2024-03-06T07:28:51.898265+0000","last_unstale":"2024-03-06T07:28:52.905637+0000","last_undegraded":"2024-03-06T07:28:52.905637+0000","last_fullsized":"2024-03-06T07:28:52.905637+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:57.146808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902667+0000","last_change":"2024-03-06T07:28:50.899722+0000","last_active":"2024-03-06T07:28:52.902667+0000","last_peered":"2024-03-06T07:28:52.902667+0000","last_clean":"2024-03-06T07:28:52.902667+0000","last_became_active":"2024-03-06T07:28:50.897552+0000","last_became_peered":"2024-03-06T07:28:50.897552+0000","last_unstale":"2024-03-06T07:28:52.902667+0000","last_undegraded":"2024-03-06T07:28:52.902667+0000","last_fullsized":"2024-03-06T07:28:52.902667+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.752746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983929+0000","last_change":"2024-03-06T07:28:51.907944+0000","last_active":"2024-03-06T07:28:52.983929+0000","last_peered":"2024-03-06T07:28:52.983929+0000","last_clean":"2024-03-06T07:28:52.983929+0000","last_became_active":"2024-03-06T07:28:51.907644+0000","last_became_peered":"2024-03-06T07:28:51.907644+0000","last_unstale":"2024-03-06T07:28:52.983929+0000","last_undegraded":"2024-03-06T07:28:52.983929+0000","last_fullsized":"2024-03-06T07:28:52.983929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:09.892533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902516+0000","last_change":"2024-03-06T07:28:50.900168+0000","last_active":"2024-03-06T07:28:52.902516+0000","last_peered":"2024-03-06T07:28:52.902516+0000","last_clean":"2024-03-06T07:28:52.902516+0000","last_became_active":"2024-03-06T07:28:50.899511+0000","last_became_peered":"2024-03-06T07:28:50.899511+0000","last_unstale":"2024-03-06T07:28:52.902516+0000","last_undegraded":"2024-03-06T07:28:52.902516+0000","last_fullsized":"2024-03-06T07:28:52.902516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:08.878663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987532+0000","last_change":"2024-03-06T07:28:51.920787+0000","last_active":"2024-03-06T07:28:52.987532+0000","last_peered":"2024-03-06T07:28:52.987532+0000","last_clean":"2024-03-06T07:28:52.987532+0000","last_became_active":"2024-03-06T07:28:51.920504+0000","last_became_peered":"2024-03-06T07:28:51.920504+0000","last_unstale":"2024-03-06T07:28:52.987532+0000","last_undegraded":"2024-03-06T07:28:52.987532+0000","last_fullsized":"2024-03-06T07:28:52.987532+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:16.041355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900593+0000","last_change":"2024-03-06T07:28:50.890538+0000","last_active":"2024-03-06T07:28:52.900593+0000","last_peered":"2024-03-06T07:28:52.900593+0000","last_clean":"2024-03-06T07:28:52.900593+0000","last_became_active":"2024-03-06T07:28:50.889826+0000","last_became_peered":"2024-03-06T07:28:50.889826+0000","last_unstale":"2024-03-06T07:28:52.900593+0000","last_undegraded":"2024-03-06T07:28:52.900593+0000","last_fullsized":"2024-03-06T07:28:52.900593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:29.566494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.985680+0000","last_change":"2024-03-06T07:28:51.918465+0000","last_active":"2024-03-06T07:28:52.985680+0000","last_peered":"2024-03-06T07:28:52.985680+0000","last_clean":"2024-03-06T07:28:52.985680+0000","last_became_active":"2024-03-06T07:28:51.917766+0000","last_became_peered":"2024-03-06T07:28:51.917766+0000","last_unstale":"2024-03-06T07:28:52.985680+0000","last_undegraded":"2024-03-06T07:28:52.985680+0000","last_fullsized":"2024-03-06T07:28:52.985680+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.396135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987407+0000","last_change":"2024-03-06T07:28:50.897991+0000","last_active":"2024-03-06T07:28:52.987407+0000","last_peered":"2024-03-06T07:28:52.987407+0000","last_clean":"2024-03-06T07:28:52.987407+0000","last_became_active":"2024-03-06T07:28:50.897697+0000","last_became_peered":"2024-03-06T07:28:50.897697+0000","last_unstale":"2024-03-06T07:28:52.987407+0000","last_undegraded":"2024-03-06T07:28:52.987407+0000","last_fullsized":"2024-03-06T07:28:52.987407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:54.614934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900060+0000","last_change":"2024-03-06T07:28:51.909321+0000","last_active":"2024-03-06T07:28:52.900060+0000","last_peered":"2024-03-06T07:28:52.900060+0000","last_clean":"2024-03-06T07:28:52.900060+0000","last_became_active":"2024-03-06T07:28:51.905920+0000","last_became_peered":"2024-03-06T07:28:51.905920+0000","last_unstale":"2024-03-06T07:28:52.900060+0000","last_undegraded":"2024-03-06T07:28:52.900060+0000","last_fullsized":"2024-03-06T07:28:52.900060+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:04.317549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987384+0000","last_change":"2024-03-06T07:28:50.895990+0000","last_active":"2024-03-06T07:28:52.987384+0000","last_peered":"2024-03-06T07:28:52.987384+0000","last_clean":"2024-03-06T07:28:52.987384+0000","last_became_active":"2024-03-06T07:28:50.895628+0000","last_became_peered":"2024-03-06T07:28:50.895628+0000","last_unstale":"2024-03-06T07:28:52.987384+0000","last_undegraded":"2024-03-06T07:28:52.987384+0000","last_fullsized":"2024-03-06T07:28:52.987384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:00.841352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980431+0000","last_change":"2024-03-06T07:28:51.903593+0000","last_active":"2024-03-06T07:28:52.980431+0000","last_peered":"2024-03-06T07:28:52.980431+0000","last_clean":"2024-03-06T07:28:52.980431+0000","last_became_active":"2024-03-06T07:28:51.903263+0000","last_became_peered":"2024-03-06T07:28:51.903263+0000","last_unstale":"2024-03-06T07:28:52.980431+0000","last_undegraded":"2024-03-06T07:28:52.980431+0000","last_fullsized":"2024-03-06T07:28:52.980431+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:32.221514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987037+0000","last_change":"2024-03-06T07:28:50.901081+0000","last_active":"2024-03-06T07:28:52.987037+0000","last_peered":"2024-03-06T07:28:52.987037+0000","last_clean":"2024-03-06T07:28:52.987037+0000","last_became_active":"2024-03-06T07:28:50.900751+0000","last_became_peered":"2024-03-06T07:28:50.900751+0000","last_unstale":"2024-03-06T07:28:52.987037+0000","last_undegraded":"2024-03-06T07:28:52.987037+0000","last_fullsized":"2024-03-06T07:28:52.987037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:04.945213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980052+0000","last_change":"2024-03-06T07:28:51.900185+0000","last_active":"2024-03-06T07:28:52.980052+0000","last_peered":"2024-03-06T07:28:52.980052+0000","last_clean":"2024-03-06T07:28:52.980052+0000","last_became_active":"2024-03-06T07:28:51.899828+0000","last_became_peered":"2024-03-06T07:28:51.899828+0000","last_unstale":"2024-03-06T07:28:52.980052+0000","last_undegraded":"2024-03-06T07:28:52.980052+0000","last_fullsized":"2024-03-06T07:28:52.980052+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:10.448027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905081+0000","last_change":"2024-03-06T07:28:50.893526+0000","last_active":"2024-03-06T07:28:52.905081+0000","last_peered":"2024-03-06T07:28:52.905081+0000","last_clean":"2024-03-06T07:28:52.905081+0000","last_became_active":"2024-03-06T07:28:50.893253+0000","last_became_peered":"2024-03-06T07:28:50.893253+0000","last_unstale":"2024-03-06T07:28:52.905081+0000","last_undegraded":"2024-03-06T07:28:52.905081+0000","last_fullsized":"2024-03-06T07:28:52.905081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:09.434156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980297+0000","last_change":"2024-03-06T07:28:51.892668+0000","last_active":"2024-03-06T07:28:52.980297+0000","last_peered":"2024-03-06T07:28:52.980297+0000","last_clean":"2024-03-06T07:28:52.980297+0000","last_became_active":"2024-03-06T07:28:51.892223+0000","last_became_peered":"2024-03-06T07:28:51.892223+0000","last_unstale":"2024-03-06T07:28:52.980297+0000","last_undegraded":"2024-03-06T07:28:52.980297+0000","last_fullsized":"2024-03-06T07:28:52.980297+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:15.961587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983545+0000","last_change":"2024-03-06T07:28:50.901458+0000","last_active":"2024-03-06T07:28:52.983545+0000","last_peered":"2024-03-06T07:28:52.983545+0000","last_clean":"2024-03-06T07:28:52.983545+0000","last_became_active":"2024-03-06T07:28:50.900891+0000","last_became_peered":"2024-03-06T07:28:50.900891+0000","last_unstale":"2024-03-06T07:28:52.983545+0000","last_undegraded":"2024-03-06T07:28:52.983545+0000","last_fullsized":"2024-03-06T07:28:52.983545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:09.434156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980075+0000","last_change":"2024-03-06T07:28:51.900038+0000","last_active":"2024-03-06T07:28:52.980075+0000","last_peered":"2024-03-06T07:28:52.980075+0000","last_clean":"2024-03-06T07:28:52.980075+0000","last_became_active":"2024-03-06T07:28:51.889244+0000","last_became_peered":"2024-03-06T07:28:51.889244+0000","last_unstale":"2024-03-06T07:28:52.980075+0000","last_undegraded":"2024-03-06T07:28:52.980075+0000","last_fullsized":"2024-03-06T07:28:52.980075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:19.911872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981342+0000","last_change":"2024-03-06T07:28:50.898611+0000","last_active":"2024-03-06T07:28:52.981342+0000","last_peered":"2024-03-06T07:28:52.981342+0000","last_clean":"2024-03-06T07:28:52.981342+0000","last_became_active":"2024-03-06T07:28:50.895588+0000","last_became_peered":"2024-03-06T07:28:50.895588+0000","last_unstale":"2024-03-06T07:28:52.981342+0000","last_undegraded":"2024-03-06T07:28:52.981342+0000","last_fullsized":"2024-03-06T07:28:52.981342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:19.106856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905471+0000","last_change":"2024-03-06T07:28:51.904954+0000","last_active":"2024-03-06T07:28:52.905471+0000","last_peered":"2024-03-06T07:28:52.905471+0000","last_clean":"2024-03-06T07:28:52.905471+0000","last_became_active":"2024-03-06T07:28:51.904605+0000","last_became_peered":"2024-03-06T07:28:51.904605+0000","last_unstale":"2024-03-06T07:28:52.905471+0000","last_undegraded":"2024-03-06T07:28:52.905471+0000","last_fullsized":"2024-03-06T07:28:52.905471+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.696454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981454+0000","last_change":"2024-03-06T07:28:50.903865+0000","last_active":"2024-03-06T07:28:52.981454+0000","last_peered":"2024-03-06T07:28:52.981454+0000","last_clean":"2024-03-06T07:28:52.981454+0000","last_became_active":"2024-03-06T07:28:50.903476+0000","last_became_peered":"2024-03-06T07:28:50.903476+0000","last_unstale":"2024-03-06T07:28:52.981454+0000","last_undegraded":"2024-03-06T07:28:52.981454+0000","last_fullsized":"2024-03-06T07:28:52.981454+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:18.949654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980340+0000","last_change":"2024-03-06T07:28:51.923858+0000","last_active":"2024-03-06T07:28:52.980340+0000","last_peered":"2024-03-06T07:28:52.980340+0000","last_clean":"2024-03-06T07:28:52.980340+0000","last_became_active":"2024-03-06T07:28:51.921911+0000","last_became_peered":"2024-03-06T07:28:51.921911+0000","last_unstale":"2024-03-06T07:28:52.980340+0000","last_undegraded":"2024-03-06T07:28:52.980340+0000","last_fullsized":"2024-03-06T07:28:52.980340+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:48.047781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900618+0000","last_change":"2024-03-06T07:28:50.890751+0000","last_active":"2024-03-06T07:28:52.900618+0000","last_peered":"2024-03-06T07:28:52.900618+0000","last_clean":"2024-03-06T07:28:52.900618+0000","last_became_active":"2024-03-06T07:28:50.890243+0000","last_became_peered":"2024-03-06T07:28:50.890243+0000","last_unstale":"2024-03-06T07:28:52.900618+0000","last_undegraded":"2024-03-06T07:28:52.900618+0000","last_fullsized":"2024-03-06T07:28:52.900618+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:05.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.985908+0000","last_change":"2024-03-06T07:28:51.925821+0000","last_active":"2024-03-06T07:28:52.985908+0000","last_peered":"2024-03-06T07:28:52.985908+0000","last_clean":"2024-03-06T07:28:52.985908+0000","last_became_active":"2024-03-06T07:28:51.925480+0000","last_became_peered":"2024-03-06T07:28:51.925480+0000","last_unstale":"2024-03-06T07:28:52.985908+0000","last_undegraded":"2024-03-06T07:28:52.985908+0000","last_fullsized":"2024-03-06T07:28:52.985908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:10.489955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"35'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.907045+0000","last_change":"2024-03-06T07:28:50.884638+0000","last_active":"2024-03-06T07:28:52.907045+0000","last_peered":"2024-03-06T07:28:52.907045+0000","last_clean":"2024-03-06T07:28:52.907045+0000","last_became_active":"2024-03-06T07:28:50.884298+0000","last_became_peered":"2024-03-06T07:28:50.884298+0000","last_unstale":"2024-03-06T07:28:52.907045+0000","last_undegraded":"2024-03-06T07:28:52.907045+0000","last_fullsized":"2024-03-06T07:28:52.907045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:05.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905152+0000","last_change":"2024-03-06T07:28:51.896390+0000","last_active":"2024-03-06T07:28:52.905152+0000","last_peered":"2024-03-06T07:28:52.905152+0000","last_clean":"2024-03-06T07:28:52.905152+0000","last_became_active":"2024-03-06T07:28:51.895776+0000","last_became_peered":"2024-03-06T07:28:51.895776+0000","last_unstale":"2024-03-06T07:28:52.905152+0000","last_undegraded":"2024-03-06T07:28:52.905152+0000","last_fullsized":"2024-03-06T07:28:52.905152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:25.949576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900108+0000","last_change":"2024-03-06T07:28:50.889001+0000","last_active":"2024-03-06T07:28:52.900108+0000","last_peered":"2024-03-06T07:28:52.900108+0000","last_clean":"2024-03-06T07:28:52.900108+0000","last_became_active":"2024-03-06T07:28:50.888686+0000","last_became_peered":"2024-03-06T07:28:50.888686+0000","last_unstale":"2024-03-06T07:28:52.900108+0000","last_undegraded":"2024-03-06T07:28:52.900108+0000","last_fullsized":"2024-03-06T07:28:52.900108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.304948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983331+0000","last_change":"2024-03-06T07:28:51.907867+0000","last_active":"2024-03-06T07:28:52.983331+0000","last_peered":"2024-03-06T07:28:52.983331+0000","last_clean":"2024-03-06T07:28:52.983331+0000","last_became_active":"2024-03-06T07:28:51.907530+0000","last_became_peered":"2024-03-06T07:28:51.907530+0000","last_unstale":"2024-03-06T07:28:52.983331+0000","last_undegraded":"2024-03-06T07:28:52.983331+0000","last_fullsized":"2024-03-06T07:28:52.983331+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:32.221514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905264+0000","last_change":"2024-03-06T07:28:50.893960+0000","last_active":"2024-03-06T07:28:52.905264+0000","last_peered":"2024-03-06T07:28:52.905264+0000","last_clean":"2024-03-06T07:28:52.905264+0000","last_became_active":"2024-03-06T07:28:50.893680+0000","last_became_peered":"2024-03-06T07:28:50.893680+0000","last_unstale":"2024-03-06T07:28:52.905264+0000","last_undegraded":"2024-03-06T07:28:52.905264+0000","last_fullsized":"2024-03-06T07:28:52.905264+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:18.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905185+0000","last_change":"2024-03-06T07:28:51.897697+0000","last_active":"2024-03-06T07:28:52.905185+0000","last_peered":"2024-03-06T07:28:52.905185+0000","last_clean":"2024-03-06T07:28:52.905185+0000","last_became_active":"2024-03-06T07:28:51.897432+0000","last_became_peered":"2024-03-06T07:28:51.897432+0000","last_unstale":"2024-03-06T07:28:52.905185+0000","last_undegraded":"2024-03-06T07:28:52.905185+0000","last_fullsized":"2024-03-06T07:28:52.905185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:22.203477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899562+0000","last_change":"2024-03-06T07:28:50.890259+0000","last_active":"2024-03-06T07:28:52.899562+0000","last_peered":"2024-03-06T07:28:52.899562+0000","last_clean":"2024-03-06T07:28:52.899562+0000","last_became_active":"2024-03-06T07:28:50.889891+0000","last_became_peered":"2024-03-06T07:28:50.889891+0000","last_unstale":"2024-03-06T07:28:52.899562+0000","last_undegraded":"2024-03-06T07:28:52.899562+0000","last_fullsized":"2024-03-06T07:28:52.899562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:47.033910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902405+0000","last_change":"2024-03-06T07:28:51.899489+0000","last_active":"2024-03-06T07:28:52.902405+0000","last_peered":"2024-03-06T07:28:52.902405+0000","last_clean":"2024-03-06T07:28:52.902405+0000","last_became_active":"2024-03-06T07:28:51.899208+0000","last_became_peered":"2024-03-06T07:28:51.899208+0000","last_unstale":"2024-03-06T07:28:52.902405+0000","last_undegraded":"2024-03-06T07:28:52.902405+0000","last_fullsized":"2024-03-06T07:28:52.902405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:25.099777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982054+0000","last_change":"2024-03-06T07:28:50.893091+0000","last_active":"2024-03-06T07:28:52.982054+0000","last_peered":"2024-03-06T07:28:52.982054+0000","last_clean":"2024-03-06T07:28:52.982054+0000","last_became_active":"2024-03-06T07:28:50.892720+0000","last_became_peered":"2024-03-06T07:28:50.892720+0000","last_unstale":"2024-03-06T07:28:52.982054+0000","last_undegraded":"2024-03-06T07:28:52.982054+0000","last_fullsized":"2024-03-06T07:28:52.982054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:28.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981204+0000","last_change":"2024-03-06T07:28:51.904718+0000","last_active":"2024-03-06T07:28:52.981204+0000","last_peered":"2024-03-06T07:28:52.981204+0000","last_clean":"2024-03-06T07:28:52.981204+0000","last_became_active":"2024-03-06T07:28:51.904377+0000","last_became_peered":"2024-03-06T07:28:51.904377+0000","last_unstale":"2024-03-06T07:28:52.981204+0000","last_undegraded":"2024-03-06T07:28:52.981204+0000","last_fullsized":"2024-03-06T07:28:52.981204+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:29.759579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902440+0000","last_change":"2024-03-06T07:28:50.899035+0000","last_active":"2024-03-06T07:28:52.902440+0000","last_peered":"2024-03-06T07:28:52.902440+0000","last_clean":"2024-03-06T07:28:52.902440+0000","last_became_active":"2024-03-06T07:28:50.898719+0000","last_became_peered":"2024-03-06T07:28:50.898719+0000","last_unstale":"2024-03-06T07:28:52.902440+0000","last_undegraded":"2024-03-06T07:28:52.902440+0000","last_fullsized":"2024-03-06T07:28:52.902440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:18.949654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905796+0000","last_change":"2024-03-06T07:28:51.920756+0000","last_active":"2024-03-06T07:28:52.905796+0000","last_peered":"2024-03-06T07:28:52.905796+0000","last_clean":"2024-03-06T07:28:52.905796+0000","last_became_active":"2024-03-06T07:28:51.917449+0000","last_became_peered":"2024-03-06T07:28:51.917449+0000","last_unstale":"2024-03-06T07:28:52.905796+0000","last_undegraded":"2024-03-06T07:28:52.905796+0000","last_fullsized":"2024-03-06T07:28:52.905796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:23.627303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983258+0000","last_change":"2024-03-06T07:28:50.902919+0000","last_active":"2024-03-06T07:28:52.983258+0000","last_peered":"2024-03-06T07:28:52.983258+0000","last_clean":"2024-03-06T07:28:52.983258+0000","last_became_active":"2024-03-06T07:28:50.902645+0000","last_became_peered":"2024-03-06T07:28:50.902645+0000","last_unstale":"2024-03-06T07:28:52.983258+0000","last_undegraded":"2024-03-06T07:28:52.983258+0000","last_fullsized":"2024-03-06T07:28:52.983258+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:18.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905816+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:28:52.905816+0000","last_peered":"2024-03-06T07:28:52.905816+0000","last_clean":"2024-03-06T07:28:52.905816+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:28:52.905816+0000","last_undegraded":"2024-03-06T07:28:52.905816+0000","last_fullsized":"2024-03-06T07:28:52.905816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":991232,"data_stored":745550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":155196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":156726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":148862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":156726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":745460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:56.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:56.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.523+0000 7ff079400640 1 Processor -- start 2024-03-06T07:28:56.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.523+0000 7ff079400640 1 -- start start 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff079400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff07406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff079400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07415b090 con 0x7ff074072340 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff079400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0740b2a30 con 0x7ff07415acb0 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff079400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740b4360 con 0x7ff0740b4680 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff073200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff074072340 0x7ff0740ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39678/0 (socket says 172.21.15.97:39678) 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff073200640 1 -- 172.21.15.97:0/2759176683 learned_addr learned my addr 172.21.15.97:0/2759176683 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.525+0000 7ff06be00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff07406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432862849 0 0) 0x7ff07415b090 con 0x7ff074072340 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff058003520 con 0x7ff074072340 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 217678026 0 0) 0x7ff058003520 con 0x7ff074072340 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 msgr2=0x7ff07406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff07406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 >> v1:172.21.15.135:6789/0 conn(0x7ff07415acb0 legacy=0x7ff0740b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0740b2cf0 con 0x7ff074072340 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff06c0019f0 con 0x7ff074072340 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 >> v1:172.21.15.97:6789/0 conn(0x7ff074072340 legacy=0x7ff0740ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0580039e0 0x7ff058003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff058004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff0580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740b4360 con 0x7ff058004800 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05800baa0 con 0x7ff0580039e0 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.526+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05800bca0 con 0x7ff0740b4680 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff06be00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff058004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff072800640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0580039e0 0x7ff058003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff073200640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff0580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff06be00640 1 -- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 msgr2=0x7ff0580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff06be00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff0580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff06be00640 1 -- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0580039e0 msgr2=0x7ff058003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff06be00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0580039e0 0x7ff058003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff06be00640 1 -- 172.21.15.97:0/2759176683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058003520 con 0x7ff058004800 2024-03-06T07:28:56.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.527+0000 7ff073200640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0740b4680 0x7ff0580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.528+0000 7ff06be00640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff058004c80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009a90 tx=0x7ff058005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761c6f95b153c4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.528+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c039070 con 0x7ff058004800 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.528+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c02c660 con 0x7ff058004800 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.528+0000 7ff071e00640 1 -- 172.21.15.97:0/2759176683 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c009690 con 0x7ff058004800 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff079400640 1 -- 172.21.15.97:0/2759176683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 msgr2=0x7ff058004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff079400640 1 --2- 172.21.15.97:0/2759176683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff058004c80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009a90 tx=0x7ff058005830 comp rx=0 tx=0).stop 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff073200640 1 -- 172.21.15.97:0/2759176683 reap_dead start 2024-03-06T07:28:56.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff079400640 1 -- 172.21.15.97:0/2759176683 shutdown_connections 2024-03-06T07:28:56.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff079400640 1 -- 172.21.15.97:0/2759176683 >> 172.21.15.97:0/2759176683 conn(0x7ff0740b1730 msgr2=0x7ff0740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff079400640 1 -- 172.21.15.97:0/2759176683 shutdown_connections 2024-03-06T07:28:56.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.529+0000 7ff079400640 1 -- 172.21.15.97:0/2759176683 wait complete. 2024-03-06T07:28:56.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.531+0000 7ff079400640 1 Processor -- start 2024-03-06T07:28:56.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.531+0000 7ff079400640 1 -- start start 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff079400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff07414ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff079400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0741557b0 0x7ff074155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff079400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff074153060 0x7ff0741534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff079400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740b4010 con 0x7ff058004800 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff079400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740b32b0 con 0x7ff0741557b0 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff079400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740cd6c0 con 0x7ff074153060 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff07414ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff07414ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53224/0 (socket says 172.21.15.97:53224) 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 -- 172.21.15.97:0/2710126095 learned_addr learned my addr 172.21.15.97:0/2710126095 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff06be00640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0741557b0 0x7ff074155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 -- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff074153060 msgr2=0x7ff0741534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff074153060 0x7ff0741534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 -- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0741557b0 msgr2=0x7ff074155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0741557b0 0x7ff074155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.532+0000 7ff072800640 1 -- 172.21.15.97:0/2710126095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074153e10 con 0x7ff058004800 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.533+0000 7ff072800640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff07414ea10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff06400e5b0 tx=0x7ff06400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.533+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0640052e0 con 0x7ff058004800 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.533+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064018070 con 0x7ff058004800 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.533+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0741540f0 con 0x7ff058004800 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.533+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064003810 con 0x7ff058004800 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.533+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff074151240 con 0x7ff058004800 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.535+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07404f330 con 0x7ff058004800 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.535+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0640039b0 con 0x7ff058004800 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.537+0000 7ff06aa00640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff05006b4d0 0x7ff05006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.537+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff064089e60 con 0x7ff058004800 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.537+0000 7ff06be00640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff05006b4d0 0x7ff05006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.538+0000 7ff06be00640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff05006b4d0 0x7ff05006d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff05c036000 tx=0x7ff05c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.541+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff064058cd0 con 0x7ff058004800 2024-03-06T07:28:56.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.730+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff07414ef50 con 0x7ff05006b4d0 2024-03-06T07:28:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.744+0000 7ff06aa00640 1 -- 172.21.15.97:0/2710126095 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251950 (secure 0 0 0) 0x7ff05c004140 con 0x7ff05006b4d0 2024-03-06T07:28:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:28:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff05006b4d0 msgr2=0x7ff05006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff05006b4d0 0x7ff05006d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff05c036000 tx=0x7ff05c030000 comp rx=0 tx=0).stop 2024-03-06T07:28:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 msgr2=0x7ff07414ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff07414ea10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff06400e5b0 tx=0x7ff06400ea70 comp rx=0 tx=0).stop 2024-03-06T07:28:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 shutdown_connections 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff05006b4d0 0x7ff05006d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff074153060 0x7ff0741534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0741557b0 0x7ff074155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 --2- 172.21.15.97:0/2710126095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff058004800 0x7ff07414ea10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.749+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 >> 172.21.15.97:0/2710126095 conn(0x7ff0740b1730 msgr2=0x7ff07406e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.750+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 shutdown_connections 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:56.750+0000 7ff079400640 1 -- 172.21.15.97:0/2710126095 wait complete. 2024-03-06T07:28:56.769 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-06T07:28:54.922927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220476,"kb_used_data":4284,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754244,"statfs":{"total":773094113280,"available":772868345856,"internally_reserved":0,"allocated":4386816,"data_stored":2419992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.073936"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986403+0000","last_change":"2024-03-06T07:28:51.920469+0000","last_active":"2024-03-06T07:28:52.986403+0000","last_peered":"2024-03-06T07:28:52.986403+0000","last_clean":"2024-03-06T07:28:52.986403+0000","last_became_active":"2024-03-06T07:28:51.920150+0000","last_became_peered":"2024-03-06T07:28:51.920150+0000","last_unstale":"2024-03-06T07:28:52.986403+0000","last_undegraded":"2024-03-06T07:28:52.986403+0000","last_fullsized":"2024-03-06T07:28:52.986403+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:36.766617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904122+0000","last_change":"2024-03-06T07:28:51.916476+0000","last_active":"2024-03-06T07:28:52.904122+0000","last_peered":"2024-03-06T07:28:52.904122+0000","last_clean":"2024-03-06T07:28:52.904122+0000","last_became_active":"2024-03-06T07:28:51.915640+0000","last_became_peered":"2024-03-06T07:28:51.915640+0000","last_unstale":"2024-03-06T07:28:52.904122+0000","last_undegraded":"2024-03-06T07:28:52.904122+0000","last_fullsized":"2024-03-06T07:28:52.904122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:29.996692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982384+0000","last_change":"2024-03-06T07:28:51.925809+0000","last_active":"2024-03-06T07:28:52.982384+0000","last_peered":"2024-03-06T07:28:52.982384+0000","last_clean":"2024-03-06T07:28:52.982384+0000","last_became_active":"2024-03-06T07:28:51.925495+0000","last_became_peered":"2024-03-06T07:28:51.925495+0000","last_unstale":"2024-03-06T07:28:52.982384+0000","last_undegraded":"2024-03-06T07:28:52.982384+0000","last_fullsized":"2024-03-06T07:28:52.982384+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:23.627303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899915+0000","last_change":"2024-03-06T07:28:51.903797+0000","last_active":"2024-03-06T07:28:52.899915+0000","last_peered":"2024-03-06T07:28:52.899915+0000","last_clean":"2024-03-06T07:28:52.899915+0000","last_became_active":"2024-03-06T07:28:51.903522+0000","last_became_peered":"2024-03-06T07:28:51.903522+0000","last_unstale":"2024-03-06T07:28:52.899915+0000","last_undegraded":"2024-03-06T07:28:52.899915+0000","last_fullsized":"2024-03-06T07:28:52.899915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:08.868206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980084+0000","last_change":"2024-03-06T07:28:51.905461+0000","last_active":"2024-03-06T07:28:52.980084+0000","last_peered":"2024-03-06T07:28:52.980084+0000","last_clean":"2024-03-06T07:28:52.980084+0000","last_became_active":"2024-03-06T07:28:51.905194+0000","last_became_peered":"2024-03-06T07:28:51.905194+0000","last_unstale":"2024-03-06T07:28:52.980084+0000","last_undegraded":"2024-03-06T07:28:52.980084+0000","last_fullsized":"2024-03-06T07:28:52.980084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:36.318818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903666+0000","last_change":"2024-03-06T07:28:51.918829+0000","last_active":"2024-03-06T07:28:52.903666+0000","last_peered":"2024-03-06T07:28:52.903666+0000","last_clean":"2024-03-06T07:28:52.903666+0000","last_became_active":"2024-03-06T07:28:51.918526+0000","last_became_peered":"2024-03-06T07:28:51.918526+0000","last_unstale":"2024-03-06T07:28:52.903666+0000","last_undegraded":"2024-03-06T07:28:52.903666+0000","last_fullsized":"2024-03-06T07:28:52.903666+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.249279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906197+0000","last_change":"2024-03-06T07:28:51.910217+0000","last_active":"2024-03-06T07:28:52.906197+0000","last_peered":"2024-03-06T07:28:52.906197+0000","last_clean":"2024-03-06T07:28:52.906197+0000","last_became_active":"2024-03-06T07:28:51.909923+0000","last_became_peered":"2024-03-06T07:28:51.909923+0000","last_unstale":"2024-03-06T07:28:52.906197+0000","last_undegraded":"2024-03-06T07:28:52.906197+0000","last_fullsized":"2024-03-06T07:28:52.906197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:40.472650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905066+0000","last_change":"2024-03-06T07:28:51.918529+0000","last_active":"2024-03-06T07:28:52.905066+0000","last_peered":"2024-03-06T07:28:52.905066+0000","last_clean":"2024-03-06T07:28:52.905066+0000","last_became_active":"2024-03-06T07:28:51.917948+0000","last_became_peered":"2024-03-06T07:28:51.917948+0000","last_unstale":"2024-03-06T07:28:52.905066+0000","last_undegraded":"2024-03-06T07:28:52.905066+0000","last_fullsized":"2024-03-06T07:28:52.905066+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:33.018003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987760+0000","last_change":"2024-03-06T07:28:51.898806+0000","last_active":"2024-03-06T07:28:52.987760+0000","last_peered":"2024-03-06T07:28:52.987760+0000","last_clean":"2024-03-06T07:28:52.987760+0000","last_became_active":"2024-03-06T07:28:51.897770+0000","last_became_peered":"2024-03-06T07:28:51.897770+0000","last_unstale":"2024-03-06T07:28:52.987760+0000","last_undegraded":"2024-03-06T07:28:52.987760+0000","last_fullsized":"2024-03-06T07:28:52.987760+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:31.024451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900210+0000","last_change":"2024-03-06T07:28:51.909932+0000","last_active":"2024-03-06T07:28:52.900210+0000","last_peered":"2024-03-06T07:28:52.900210+0000","last_clean":"2024-03-06T07:28:52.900210+0000","last_became_active":"2024-03-06T07:28:51.909656+0000","last_became_peered":"2024-03-06T07:28:51.909656+0000","last_unstale":"2024-03-06T07:28:52.900210+0000","last_undegraded":"2024-03-06T07:28:52.900210+0000","last_fullsized":"2024-03-06T07:28:52.900210+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:33.018003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906146+0000","last_change":"2024-03-06T07:28:51.910123+0000","last_active":"2024-03-06T07:28:52.906146+0000","last_peered":"2024-03-06T07:28:52.906146+0000","last_clean":"2024-03-06T07:28:52.906146+0000","last_became_active":"2024-03-06T07:28:51.909823+0000","last_became_peered":"2024-03-06T07:28:51.909823+0000","last_unstale":"2024-03-06T07:28:52.906146+0000","last_undegraded":"2024-03-06T07:28:52.906146+0000","last_fullsized":"2024-03-06T07:28:52.906146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.396135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.984271+0000","last_change":"2024-03-06T07:28:51.919134+0000","last_active":"2024-03-06T07:28:52.984271+0000","last_peered":"2024-03-06T07:28:52.984271+0000","last_clean":"2024-03-06T07:28:52.984271+0000","last_became_active":"2024-03-06T07:28:51.918897+0000","last_became_peered":"2024-03-06T07:28:51.918897+0000","last_unstale":"2024-03-06T07:28:52.984271+0000","last_undegraded":"2024-03-06T07:28:52.984271+0000","last_fullsized":"2024-03-06T07:28:52.984271+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.249279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980096+0000","last_change":"2024-03-06T07:28:51.904581+0000","last_active":"2024-03-06T07:28:52.980096+0000","last_peered":"2024-03-06T07:28:52.980096+0000","last_clean":"2024-03-06T07:28:52.980096+0000","last_became_active":"2024-03-06T07:28:51.902855+0000","last_became_peered":"2024-03-06T07:28:51.902855+0000","last_unstale":"2024-03-06T07:28:52.980096+0000","last_undegraded":"2024-03-06T07:28:52.980096+0000","last_fullsized":"2024-03-06T07:28:52.980096+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:30.580365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903700+0000","last_change":"2024-03-06T07:28:51.901097+0000","last_active":"2024-03-06T07:28:52.903700+0000","last_peered":"2024-03-06T07:28:52.903700+0000","last_clean":"2024-03-06T07:28:52.903700+0000","last_became_active":"2024-03-06T07:28:51.900775+0000","last_became_peered":"2024-03-06T07:28:51.900775+0000","last_unstale":"2024-03-06T07:28:52.903700+0000","last_undegraded":"2024-03-06T07:28:52.903700+0000","last_fullsized":"2024-03-06T07:28:52.903700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:40.472650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982311+0000","last_change":"2024-03-06T07:28:51.915312+0000","last_active":"2024-03-06T07:28:52.982311+0000","last_peered":"2024-03-06T07:28:52.982311+0000","last_clean":"2024-03-06T07:28:52.982311+0000","last_became_active":"2024-03-06T07:28:51.914936+0000","last_became_peered":"2024-03-06T07:28:51.914936+0000","last_unstale":"2024-03-06T07:28:52.982311+0000","last_undegraded":"2024-03-06T07:28:52.982311+0000","last_fullsized":"2024-03-06T07:28:52.982311+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:57.146808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905991+0000","last_change":"2024-03-06T07:28:51.911752+0000","last_active":"2024-03-06T07:28:52.905991+0000","last_peered":"2024-03-06T07:28:52.905991+0000","last_clean":"2024-03-06T07:28:52.905991+0000","last_became_active":"2024-03-06T07:28:51.911420+0000","last_became_peered":"2024-03-06T07:28:51.911420+0000","last_unstale":"2024-03-06T07:28:52.905991+0000","last_undegraded":"2024-03-06T07:28:52.905991+0000","last_fullsized":"2024-03-06T07:28:52.905991+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:25.099777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906226+0000","last_change":"2024-03-06T07:28:51.914773+0000","last_active":"2024-03-06T07:28:52.906226+0000","last_peered":"2024-03-06T07:28:52.906226+0000","last_clean":"2024-03-06T07:28:52.906226+0000","last_became_active":"2024-03-06T07:28:51.913086+0000","last_became_peered":"2024-03-06T07:28:51.913086+0000","last_unstale":"2024-03-06T07:28:52.906226+0000","last_undegraded":"2024-03-06T07:28:52.906226+0000","last_fullsized":"2024-03-06T07:28:52.906226+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:10.489955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904214+0000","last_change":"2024-03-06T07:28:51.911335+0000","last_active":"2024-03-06T07:28:52.904214+0000","last_peered":"2024-03-06T07:28:52.904214+0000","last_clean":"2024-03-06T07:28:52.904214+0000","last_became_active":"2024-03-06T07:28:51.911031+0000","last_became_peered":"2024-03-06T07:28:51.911031+0000","last_unstale":"2024-03-06T07:28:52.904214+0000","last_undegraded":"2024-03-06T07:28:52.904214+0000","last_fullsized":"2024-03-06T07:28:52.904214+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:39.554357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903787+0000","last_change":"2024-03-06T07:28:51.916519+0000","last_active":"2024-03-06T07:28:52.903787+0000","last_peered":"2024-03-06T07:28:52.903787+0000","last_clean":"2024-03-06T07:28:52.903787+0000","last_became_active":"2024-03-06T07:28:51.914227+0000","last_became_peered":"2024-03-06T07:28:51.914227+0000","last_unstale":"2024-03-06T07:28:52.903787+0000","last_undegraded":"2024-03-06T07:28:52.903787+0000","last_fullsized":"2024-03-06T07:28:52.903787+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.396135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980094+0000","last_change":"2024-03-06T07:28:51.922304+0000","last_active":"2024-03-06T07:28:52.980094+0000","last_peered":"2024-03-06T07:28:52.980094+0000","last_clean":"2024-03-06T07:28:52.980094+0000","last_became_active":"2024-03-06T07:28:51.921998+0000","last_became_peered":"2024-03-06T07:28:51.921998+0000","last_unstale":"2024-03-06T07:28:52.980094+0000","last_undegraded":"2024-03-06T07:28:52.980094+0000","last_fullsized":"2024-03-06T07:28:52.980094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:06.222398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986373+0000","last_change":"2024-03-06T07:28:51.895536+0000","last_active":"2024-03-06T07:28:52.986373+0000","last_peered":"2024-03-06T07:28:52.986373+0000","last_clean":"2024-03-06T07:28:52.986373+0000","last_became_active":"2024-03-06T07:28:51.893536+0000","last_became_peered":"2024-03-06T07:28:51.893536+0000","last_unstale":"2024-03-06T07:28:52.986373+0000","last_undegraded":"2024-03-06T07:28:52.986373+0000","last_fullsized":"2024-03-06T07:28:52.986373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:45.952071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986440+0000","last_change":"2024-03-06T07:28:51.904074+0000","last_active":"2024-03-06T07:28:52.986440+0000","last_peered":"2024-03-06T07:28:52.986440+0000","last_clean":"2024-03-06T07:28:52.986440+0000","last_became_active":"2024-03-06T07:28:51.903788+0000","last_became_peered":"2024-03-06T07:28:51.903788+0000","last_unstale":"2024-03-06T07:28:52.986440+0000","last_undegraded":"2024-03-06T07:28:52.986440+0000","last_fullsized":"2024-03-06T07:28:52.986440+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:10.523665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904817+0000","last_change":"2024-03-06T07:28:51.904045+0000","last_active":"2024-03-06T07:28:52.904817+0000","last_peered":"2024-03-06T07:28:52.904817+0000","last_clean":"2024-03-06T07:28:52.904817+0000","last_became_active":"2024-03-06T07:28:51.903647+0000","last_became_peered":"2024-03-06T07:28:51.903647+0000","last_unstale":"2024-03-06T07:28:52.904817+0000","last_undegraded":"2024-03-06T07:28:52.904817+0000","last_fullsized":"2024-03-06T07:28:52.904817+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:36.766617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981977+0000","last_change":"2024-03-06T07:28:51.917904+0000","last_active":"2024-03-06T07:28:52.981977+0000","last_peered":"2024-03-06T07:28:52.981977+0000","last_clean":"2024-03-06T07:28:52.981977+0000","last_became_active":"2024-03-06T07:28:51.916508+0000","last_became_peered":"2024-03-06T07:28:51.916508+0000","last_unstale":"2024-03-06T07:28:52.981977+0000","last_undegraded":"2024-03-06T07:28:52.981977+0000","last_fullsized":"2024-03-06T07:28:52.981977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.696454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900465+0000","last_change":"2024-03-06T07:28:51.908930+0000","last_active":"2024-03-06T07:28:52.900465+0000","last_peered":"2024-03-06T07:28:52.900465+0000","last_clean":"2024-03-06T07:28:52.900465+0000","last_became_active":"2024-03-06T07:28:51.908569+0000","last_became_peered":"2024-03-06T07:28:51.908569+0000","last_unstale":"2024-03-06T07:28:52.900465+0000","last_undegraded":"2024-03-06T07:28:52.900465+0000","last_fullsized":"2024-03-06T07:28:52.900465+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.696454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904271+0000","last_change":"2024-03-06T07:28:51.912062+0000","last_active":"2024-03-06T07:28:52.904271+0000","last_peered":"2024-03-06T07:28:52.904271+0000","last_clean":"2024-03-06T07:28:52.904271+0000","last_became_active":"2024-03-06T07:28:51.911779+0000","last_became_peered":"2024-03-06T07:28:51.911779+0000","last_unstale":"2024-03-06T07:28:52.904271+0000","last_undegraded":"2024-03-06T07:28:52.904271+0000","last_fullsized":"2024-03-06T07:28:52.904271+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:11.145501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900374+0000","last_change":"2024-03-06T07:28:51.902550+0000","last_active":"2024-03-06T07:28:52.900374+0000","last_peered":"2024-03-06T07:28:52.900374+0000","last_clean":"2024-03-06T07:28:52.900374+0000","last_became_active":"2024-03-06T07:28:51.899321+0000","last_became_peered":"2024-03-06T07:28:51.899321+0000","last_unstale":"2024-03-06T07:28:52.900374+0000","last_undegraded":"2024-03-06T07:28:52.900374+0000","last_fullsized":"2024-03-06T07:28:52.900374+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:36.424334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.984511+0000","last_change":"2024-03-06T07:28:51.918223+0000","last_active":"2024-03-06T07:28:52.984511+0000","last_peered":"2024-03-06T07:28:52.984511+0000","last_clean":"2024-03-06T07:28:52.984511+0000","last_became_active":"2024-03-06T07:28:51.917322+0000","last_became_peered":"2024-03-06T07:28:51.917322+0000","last_unstale":"2024-03-06T07:28:52.984511+0000","last_undegraded":"2024-03-06T07:28:52.984511+0000","last_fullsized":"2024-03-06T07:28:52.984511+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:40.472650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986652+0000","last_change":"2024-03-06T07:28:51.909921+0000","last_active":"2024-03-06T07:28:52.986652+0000","last_peered":"2024-03-06T07:28:52.986652+0000","last_clean":"2024-03-06T07:28:52.986652+0000","last_became_active":"2024-03-06T07:28:51.908545+0000","last_became_peered":"2024-03-06T07:28:51.908545+0000","last_unstale":"2024-03-06T07:28:52.986652+0000","last_undegraded":"2024-03-06T07:28:52.986652+0000","last_fullsized":"2024-03-06T07:28:52.986652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:04.552921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.906075+0000","last_change":"2024-03-06T07:28:51.908632+0000","last_active":"2024-03-06T07:28:52.906075+0000","last_peered":"2024-03-06T07:28:52.906075+0000","last_clean":"2024-03-06T07:28:52.906075+0000","last_became_active":"2024-03-06T07:28:51.904666+0000","last_became_peered":"2024-03-06T07:28:51.904666+0000","last_unstale":"2024-03-06T07:28:52.906075+0000","last_undegraded":"2024-03-06T07:28:52.906075+0000","last_fullsized":"2024-03-06T07:28:52.906075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.249279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983613+0000","last_change":"2024-03-06T07:28:51.907793+0000","last_active":"2024-03-06T07:28:52.983613+0000","last_peered":"2024-03-06T07:28:52.983613+0000","last_clean":"2024-03-06T07:28:52.983613+0000","last_became_active":"2024-03-06T07:28:51.907137+0000","last_became_peered":"2024-03-06T07:28:51.907137+0000","last_unstale":"2024-03-06T07:28:52.983613+0000","last_undegraded":"2024-03-06T07:28:52.983613+0000","last_fullsized":"2024-03-06T07:28:52.983613+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:19.963525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905199+0000","last_change":"2024-03-06T07:28:51.898133+0000","last_active":"2024-03-06T07:28:52.905199+0000","last_peered":"2024-03-06T07:28:52.905199+0000","last_clean":"2024-03-06T07:28:52.905199+0000","last_became_active":"2024-03-06T07:28:51.897723+0000","last_became_peered":"2024-03-06T07:28:51.897723+0000","last_unstale":"2024-03-06T07:28:52.905199+0000","last_undegraded":"2024-03-06T07:28:52.905199+0000","last_fullsized":"2024-03-06T07:28:52.905199+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:36.424334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905160+0000","last_change":"2024-03-06T07:28:51.897381+0000","last_active":"2024-03-06T07:28:52.905160+0000","last_peered":"2024-03-06T07:28:52.905160+0000","last_clean":"2024-03-06T07:28:52.905160+0000","last_became_active":"2024-03-06T07:28:51.896985+0000","last_became_peered":"2024-03-06T07:28:51.896985+0000","last_unstale":"2024-03-06T07:28:52.905160+0000","last_undegraded":"2024-03-06T07:28:52.905160+0000","last_fullsized":"2024-03-06T07:28:52.905160+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:57.146419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987005+0000","last_change":"2024-03-06T07:28:50.902394+0000","last_active":"2024-03-06T07:28:52.987005+0000","last_peered":"2024-03-06T07:28:52.987005+0000","last_clean":"2024-03-06T07:28:52.987005+0000","last_became_active":"2024-03-06T07:28:50.901925+0000","last_became_peered":"2024-03-06T07:28:50.901925+0000","last_unstale":"2024-03-06T07:28:52.987005+0000","last_undegraded":"2024-03-06T07:28:52.987005+0000","last_fullsized":"2024-03-06T07:28:52.987005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:35.035122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986843+0000","last_change":"2024-03-06T07:28:51.907709+0000","last_active":"2024-03-06T07:28:52.986843+0000","last_peered":"2024-03-06T07:28:52.986843+0000","last_clean":"2024-03-06T07:28:52.986843+0000","last_became_active":"2024-03-06T07:28:51.907378+0000","last_became_peered":"2024-03-06T07:28:51.907378+0000","last_unstale":"2024-03-06T07:28:52.986843+0000","last_undegraded":"2024-03-06T07:28:52.986843+0000","last_fullsized":"2024-03-06T07:28:52.986843+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:25.949576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905283+0000","last_change":"2024-03-06T07:28:50.894334+0000","last_active":"2024-03-06T07:28:52.905283+0000","last_peered":"2024-03-06T07:28:52.905283+0000","last_clean":"2024-03-06T07:28:52.905283+0000","last_became_active":"2024-03-06T07:28:50.894076+0000","last_became_peered":"2024-03-06T07:28:50.894076+0000","last_unstale":"2024-03-06T07:28:52.905283+0000","last_undegraded":"2024-03-06T07:28:52.905283+0000","last_fullsized":"2024-03-06T07:28:52.905283+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.304948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986973+0000","last_change":"2024-03-06T07:28:51.908832+0000","last_active":"2024-03-06T07:28:52.986973+0000","last_peered":"2024-03-06T07:28:52.986973+0000","last_clean":"2024-03-06T07:28:52.986973+0000","last_became_active":"2024-03-06T07:28:51.908461+0000","last_became_peered":"2024-03-06T07:28:51.908461+0000","last_unstale":"2024-03-06T07:28:52.986973+0000","last_undegraded":"2024-03-06T07:28:52.986973+0000","last_fullsized":"2024-03-06T07:28:52.986973+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:04.044975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903941+0000","last_change":"2024-03-06T07:28:50.898522+0000","last_active":"2024-03-06T07:28:52.903941+0000","last_peered":"2024-03-06T07:28:52.903941+0000","last_clean":"2024-03-06T07:28:52.903941+0000","last_became_active":"2024-03-06T07:28:50.898189+0000","last_became_peered":"2024-03-06T07:28:50.898189+0000","last_unstale":"2024-03-06T07:28:52.903941+0000","last_undegraded":"2024-03-06T07:28:52.903941+0000","last_fullsized":"2024-03-06T07:28:52.903941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:21.189606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900615+0000","last_change":"2024-03-06T07:28:51.903456+0000","last_active":"2024-03-06T07:28:52.900615+0000","last_peered":"2024-03-06T07:28:52.900615+0000","last_clean":"2024-03-06T07:28:52.900615+0000","last_became_active":"2024-03-06T07:28:51.903230+0000","last_became_peered":"2024-03-06T07:28:51.903230+0000","last_unstale":"2024-03-06T07:28:52.900615+0000","last_undegraded":"2024-03-06T07:28:52.900615+0000","last_fullsized":"2024-03-06T07:28:52.900615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:57.146808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980222+0000","last_change":"2024-03-06T07:28:50.896486+0000","last_active":"2024-03-06T07:28:52.980222+0000","last_peered":"2024-03-06T07:28:52.980222+0000","last_clean":"2024-03-06T07:28:52.980222+0000","last_became_active":"2024-03-06T07:28:50.896209+0000","last_became_peered":"2024-03-06T07:28:50.896209+0000","last_unstale":"2024-03-06T07:28:52.980222+0000","last_undegraded":"2024-03-06T07:28:52.980222+0000","last_fullsized":"2024-03-06T07:28:52.980222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:19.212504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905867+0000","last_change":"2024-03-06T07:28:51.912696+0000","last_active":"2024-03-06T07:28:52.905867+0000","last_peered":"2024-03-06T07:28:52.905867+0000","last_clean":"2024-03-06T07:28:52.905867+0000","last_became_active":"2024-03-06T07:28:51.909766+0000","last_became_peered":"2024-03-06T07:28:51.909766+0000","last_unstale":"2024-03-06T07:28:52.905867+0000","last_undegraded":"2024-03-06T07:28:52.905867+0000","last_fullsized":"2024-03-06T07:28:52.905867+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.886019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986727+0000","last_change":"2024-03-06T07:28:50.899522+0000","last_active":"2024-03-06T07:28:52.986727+0000","last_peered":"2024-03-06T07:28:52.986727+0000","last_clean":"2024-03-06T07:28:52.986727+0000","last_became_active":"2024-03-06T07:28:50.899223+0000","last_became_peered":"2024-03-06T07:28:50.899223+0000","last_unstale":"2024-03-06T07:28:52.986727+0000","last_undegraded":"2024-03-06T07:28:52.986727+0000","last_fullsized":"2024-03-06T07:28:52.986727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:03.570979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982680+0000","last_change":"2024-03-06T07:28:51.917066+0000","last_active":"2024-03-06T07:28:52.982680+0000","last_peered":"2024-03-06T07:28:52.982680+0000","last_clean":"2024-03-06T07:28:52.982680+0000","last_became_active":"2024-03-06T07:28:51.916334+0000","last_became_peered":"2024-03-06T07:28:51.916334+0000","last_unstale":"2024-03-06T07:28:52.982680+0000","last_undegraded":"2024-03-06T07:28:52.982680+0000","last_fullsized":"2024-03-06T07:28:52.982680+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:25.099777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986175+0000","last_change":"2024-03-06T07:28:50.900754+0000","last_active":"2024-03-06T07:28:52.986175+0000","last_peered":"2024-03-06T07:28:52.986175+0000","last_clean":"2024-03-06T07:28:52.986175+0000","last_became_active":"2024-03-06T07:28:50.900424+0000","last_became_peered":"2024-03-06T07:28:50.900424+0000","last_unstale":"2024-03-06T07:28:52.986175+0000","last_undegraded":"2024-03-06T07:28:52.986175+0000","last_fullsized":"2024-03-06T07:28:52.986175+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:01.076475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981667+0000","last_change":"2024-03-06T07:28:51.899968+0000","last_active":"2024-03-06T07:28:52.981667+0000","last_peered":"2024-03-06T07:28:52.981667+0000","last_clean":"2024-03-06T07:28:52.981667+0000","last_became_active":"2024-03-06T07:28:51.899501+0000","last_became_peered":"2024-03-06T07:28:51.899501+0000","last_unstale":"2024-03-06T07:28:52.981667+0000","last_undegraded":"2024-03-06T07:28:52.981667+0000","last_fullsized":"2024-03-06T07:28:52.981667+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:48.134346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902758+0000","last_change":"2024-03-06T07:28:50.890559+0000","last_active":"2024-03-06T07:28:52.902758+0000","last_peered":"2024-03-06T07:28:52.902758+0000","last_clean":"2024-03-06T07:28:52.902758+0000","last_became_active":"2024-03-06T07:28:50.890328+0000","last_became_peered":"2024-03-06T07:28:50.890328+0000","last_unstale":"2024-03-06T07:28:52.902758+0000","last_undegraded":"2024-03-06T07:28:52.902758+0000","last_fullsized":"2024-03-06T07:28:52.902758+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:24.935705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899212+0000","last_change":"2024-03-06T07:28:51.916840+0000","last_active":"2024-03-06T07:28:52.899212+0000","last_peered":"2024-03-06T07:28:52.899212+0000","last_clean":"2024-03-06T07:28:52.899212+0000","last_became_active":"2024-03-06T07:28:51.916548+0000","last_became_peered":"2024-03-06T07:28:51.916548+0000","last_unstale":"2024-03-06T07:28:52.899212+0000","last_undegraded":"2024-03-06T07:28:52.899212+0000","last_fullsized":"2024-03-06T07:28:52.899212+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:59.568375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981538+0000","last_change":"2024-03-06T07:28:50.897537+0000","last_active":"2024-03-06T07:28:52.981538+0000","last_peered":"2024-03-06T07:28:52.981538+0000","last_clean":"2024-03-06T07:28:52.981538+0000","last_became_active":"2024-03-06T07:28:50.897170+0000","last_became_peered":"2024-03-06T07:28:50.897170+0000","last_unstale":"2024-03-06T07:28:52.981538+0000","last_undegraded":"2024-03-06T07:28:52.981538+0000","last_fullsized":"2024-03-06T07:28:52.981538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:00.079347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982636+0000","last_change":"2024-03-06T07:28:51.925668+0000","last_active":"2024-03-06T07:28:52.982636+0000","last_peered":"2024-03-06T07:28:52.982636+0000","last_clean":"2024-03-06T07:28:52.982636+0000","last_became_active":"2024-03-06T07:28:51.925217+0000","last_became_peered":"2024-03-06T07:28:51.925217+0000","last_unstale":"2024-03-06T07:28:52.982636+0000","last_undegraded":"2024-03-06T07:28:52.982636+0000","last_fullsized":"2024-03-06T07:28:52.982636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.886019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904484+0000","last_change":"2024-03-06T07:28:50.897874+0000","last_active":"2024-03-06T07:28:52.904484+0000","last_peered":"2024-03-06T07:28:52.904484+0000","last_clean":"2024-03-06T07:28:52.904484+0000","last_became_active":"2024-03-06T07:28:50.897584+0000","last_became_peered":"2024-03-06T07:28:50.897584+0000","last_unstale":"2024-03-06T07:28:52.904484+0000","last_undegraded":"2024-03-06T07:28:52.904484+0000","last_fullsized":"2024-03-06T07:28:52.904484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:04.116946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.904541+0000","last_change":"2024-03-06T07:28:51.914505+0000","last_active":"2024-03-06T07:28:52.904541+0000","last_peered":"2024-03-06T07:28:52.904541+0000","last_clean":"2024-03-06T07:28:52.904541+0000","last_became_active":"2024-03-06T07:28:51.914176+0000","last_became_peered":"2024-03-06T07:28:51.914176+0000","last_unstale":"2024-03-06T07:28:52.904541+0000","last_undegraded":"2024-03-06T07:28:52.904541+0000","last_fullsized":"2024-03-06T07:28:52.904541+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:09.729567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905365+0000","last_change":"2024-03-06T07:28:50.898049+0000","last_active":"2024-03-06T07:28:52.905365+0000","last_peered":"2024-03-06T07:28:52.905365+0000","last_clean":"2024-03-06T07:28:52.905365+0000","last_became_active":"2024-03-06T07:28:50.896854+0000","last_became_peered":"2024-03-06T07:28:50.896854+0000","last_unstale":"2024-03-06T07:28:52.905365+0000","last_undegraded":"2024-03-06T07:28:52.905365+0000","last_fullsized":"2024-03-06T07:28:52.905365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:29.566494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903821+0000","last_change":"2024-03-06T07:28:51.919644+0000","last_active":"2024-03-06T07:28:52.903821+0000","last_peered":"2024-03-06T07:28:52.903821+0000","last_clean":"2024-03-06T07:28:52.903821+0000","last_became_active":"2024-03-06T07:28:51.915283+0000","last_became_peered":"2024-03-06T07:28:51.915283+0000","last_unstale":"2024-03-06T07:28:52.903821+0000","last_undegraded":"2024-03-06T07:28:52.903821+0000","last_fullsized":"2024-03-06T07:28:52.903821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:10.489955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982711+0000","last_change":"2024-03-06T07:28:50.896900+0000","last_active":"2024-03-06T07:28:52.982711+0000","last_peered":"2024-03-06T07:28:52.982711+0000","last_clean":"2024-03-06T07:28:52.982711+0000","last_became_active":"2024-03-06T07:28:50.895714+0000","last_became_peered":"2024-03-06T07:28:50.895714+0000","last_unstale":"2024-03-06T07:28:52.982711+0000","last_undegraded":"2024-03-06T07:28:52.982711+0000","last_fullsized":"2024-03-06T07:28:52.982711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:08.878663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981967+0000","last_change":"2024-03-06T07:28:51.917945+0000","last_active":"2024-03-06T07:28:52.981967+0000","last_peered":"2024-03-06T07:28:52.981967+0000","last_clean":"2024-03-06T07:28:52.981967+0000","last_became_active":"2024-03-06T07:28:51.917631+0000","last_became_peered":"2024-03-06T07:28:51.917631+0000","last_unstale":"2024-03-06T07:28:52.981967+0000","last_undegraded":"2024-03-06T07:28:52.981967+0000","last_fullsized":"2024-03-06T07:28:52.981967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:36.424334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903942+0000","last_change":"2024-03-06T07:28:50.882756+0000","last_active":"2024-03-06T07:28:52.903942+0000","last_peered":"2024-03-06T07:28:52.903942+0000","last_clean":"2024-03-06T07:28:52.903942+0000","last_became_active":"2024-03-06T07:28:50.882453+0000","last_became_peered":"2024-03-06T07:28:50.882453+0000","last_unstale":"2024-03-06T07:28:52.903942+0000","last_undegraded":"2024-03-06T07:28:52.903942+0000","last_fullsized":"2024-03-06T07:28:52.903942+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:07.854336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899569+0000","last_change":"2024-03-06T07:28:51.923593+0000","last_active":"2024-03-06T07:28:52.899569+0000","last_peered":"2024-03-06T07:28:52.899569+0000","last_clean":"2024-03-06T07:28:52.899569+0000","last_became_active":"2024-03-06T07:28:51.920647+0000","last_became_peered":"2024-03-06T07:28:51.920647+0000","last_unstale":"2024-03-06T07:28:52.899569+0000","last_undegraded":"2024-03-06T07:28:52.899569+0000","last_fullsized":"2024-03-06T07:28:52.899569+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:05.130817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.986956+0000","last_change":"2024-03-06T07:28:50.897541+0000","last_active":"2024-03-06T07:28:52.986956+0000","last_peered":"2024-03-06T07:28:52.986956+0000","last_clean":"2024-03-06T07:28:52.986956+0000","last_became_active":"2024-03-06T07:28:50.897255+0000","last_became_peered":"2024-03-06T07:28:50.897255+0000","last_unstale":"2024-03-06T07:28:52.986956+0000","last_undegraded":"2024-03-06T07:28:52.986956+0000","last_fullsized":"2024-03-06T07:28:52.986956+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.096128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.903950+0000","last_change":"2024-03-06T07:28:51.908136+0000","last_active":"2024-03-06T07:28:52.903950+0000","last_peered":"2024-03-06T07:28:52.903950+0000","last_clean":"2024-03-06T07:28:52.903950+0000","last_became_active":"2024-03-06T07:28:51.907097+0000","last_became_peered":"2024-03-06T07:28:51.907097+0000","last_unstale":"2024-03-06T07:28:52.903950+0000","last_undegraded":"2024-03-06T07:28:52.903950+0000","last_fullsized":"2024-03-06T07:28:52.903950+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:25.674172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899471+0000","last_change":"2024-03-06T07:28:50.900478+0000","last_active":"2024-03-06T07:28:52.899471+0000","last_peered":"2024-03-06T07:28:52.899471+0000","last_clean":"2024-03-06T07:28:52.899471+0000","last_became_active":"2024-03-06T07:28:50.898772+0000","last_became_peered":"2024-03-06T07:28:50.898772+0000","last_unstale":"2024-03-06T07:28:52.899471+0000","last_undegraded":"2024-03-06T07:28:52.899471+0000","last_fullsized":"2024-03-06T07:28:52.899471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:14.947717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905637+0000","last_change":"2024-03-06T07:28:51.898634+0000","last_active":"2024-03-06T07:28:52.905637+0000","last_peered":"2024-03-06T07:28:52.905637+0000","last_clean":"2024-03-06T07:28:52.905637+0000","last_became_active":"2024-03-06T07:28:51.898265+0000","last_became_peered":"2024-03-06T07:28:51.898265+0000","last_unstale":"2024-03-06T07:28:52.905637+0000","last_undegraded":"2024-03-06T07:28:52.905637+0000","last_fullsized":"2024-03-06T07:28:52.905637+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:57.146808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902667+0000","last_change":"2024-03-06T07:28:50.899722+0000","last_active":"2024-03-06T07:28:52.902667+0000","last_peered":"2024-03-06T07:28:52.902667+0000","last_clean":"2024-03-06T07:28:52.902667+0000","last_became_active":"2024-03-06T07:28:50.897552+0000","last_became_peered":"2024-03-06T07:28:50.897552+0000","last_unstale":"2024-03-06T07:28:52.902667+0000","last_undegraded":"2024-03-06T07:28:52.902667+0000","last_fullsized":"2024-03-06T07:28:52.902667+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.752746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983929+0000","last_change":"2024-03-06T07:28:51.907944+0000","last_active":"2024-03-06T07:28:52.983929+0000","last_peered":"2024-03-06T07:28:52.983929+0000","last_clean":"2024-03-06T07:28:52.983929+0000","last_became_active":"2024-03-06T07:28:51.907644+0000","last_became_peered":"2024-03-06T07:28:51.907644+0000","last_unstale":"2024-03-06T07:28:52.983929+0000","last_undegraded":"2024-03-06T07:28:52.983929+0000","last_fullsized":"2024-03-06T07:28:52.983929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:09.892533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902516+0000","last_change":"2024-03-06T07:28:50.900168+0000","last_active":"2024-03-06T07:28:52.902516+0000","last_peered":"2024-03-06T07:28:52.902516+0000","last_clean":"2024-03-06T07:28:52.902516+0000","last_became_active":"2024-03-06T07:28:50.899511+0000","last_became_peered":"2024-03-06T07:28:50.899511+0000","last_unstale":"2024-03-06T07:28:52.902516+0000","last_undegraded":"2024-03-06T07:28:52.902516+0000","last_fullsized":"2024-03-06T07:28:52.902516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:08.878663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987532+0000","last_change":"2024-03-06T07:28:51.920787+0000","last_active":"2024-03-06T07:28:52.987532+0000","last_peered":"2024-03-06T07:28:52.987532+0000","last_clean":"2024-03-06T07:28:52.987532+0000","last_became_active":"2024-03-06T07:28:51.920504+0000","last_became_peered":"2024-03-06T07:28:51.920504+0000","last_unstale":"2024-03-06T07:28:52.987532+0000","last_undegraded":"2024-03-06T07:28:52.987532+0000","last_fullsized":"2024-03-06T07:28:52.987532+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:16.041355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900593+0000","last_change":"2024-03-06T07:28:50.890538+0000","last_active":"2024-03-06T07:28:52.900593+0000","last_peered":"2024-03-06T07:28:52.900593+0000","last_clean":"2024-03-06T07:28:52.900593+0000","last_became_active":"2024-03-06T07:28:50.889826+0000","last_became_peered":"2024-03-06T07:28:50.889826+0000","last_unstale":"2024-03-06T07:28:52.900593+0000","last_undegraded":"2024-03-06T07:28:52.900593+0000","last_fullsized":"2024-03-06T07:28:52.900593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:29.566494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.985680+0000","last_change":"2024-03-06T07:28:51.918465+0000","last_active":"2024-03-06T07:28:52.985680+0000","last_peered":"2024-03-06T07:28:52.985680+0000","last_clean":"2024-03-06T07:28:52.985680+0000","last_became_active":"2024-03-06T07:28:51.917766+0000","last_became_peered":"2024-03-06T07:28:51.917766+0000","last_unstale":"2024-03-06T07:28:52.985680+0000","last_undegraded":"2024-03-06T07:28:52.985680+0000","last_fullsized":"2024-03-06T07:28:52.985680+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.396135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987407+0000","last_change":"2024-03-06T07:28:50.897991+0000","last_active":"2024-03-06T07:28:52.987407+0000","last_peered":"2024-03-06T07:28:52.987407+0000","last_clean":"2024-03-06T07:28:52.987407+0000","last_became_active":"2024-03-06T07:28:50.897697+0000","last_became_peered":"2024-03-06T07:28:50.897697+0000","last_unstale":"2024-03-06T07:28:52.987407+0000","last_undegraded":"2024-03-06T07:28:52.987407+0000","last_fullsized":"2024-03-06T07:28:52.987407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:54.614934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900060+0000","last_change":"2024-03-06T07:28:51.909321+0000","last_active":"2024-03-06T07:28:52.900060+0000","last_peered":"2024-03-06T07:28:52.900060+0000","last_clean":"2024-03-06T07:28:52.900060+0000","last_became_active":"2024-03-06T07:28:51.905920+0000","last_became_peered":"2024-03-06T07:28:51.905920+0000","last_unstale":"2024-03-06T07:28:52.900060+0000","last_undegraded":"2024-03-06T07:28:52.900060+0000","last_fullsized":"2024-03-06T07:28:52.900060+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:04.317549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987384+0000","last_change":"2024-03-06T07:28:50.895990+0000","last_active":"2024-03-06T07:28:52.987384+0000","last_peered":"2024-03-06T07:28:52.987384+0000","last_clean":"2024-03-06T07:28:52.987384+0000","last_became_active":"2024-03-06T07:28:50.895628+0000","last_became_peered":"2024-03-06T07:28:50.895628+0000","last_unstale":"2024-03-06T07:28:52.987384+0000","last_undegraded":"2024-03-06T07:28:52.987384+0000","last_fullsized":"2024-03-06T07:28:52.987384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:00.841352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980431+0000","last_change":"2024-03-06T07:28:51.903593+0000","last_active":"2024-03-06T07:28:52.980431+0000","last_peered":"2024-03-06T07:28:52.980431+0000","last_clean":"2024-03-06T07:28:52.980431+0000","last_became_active":"2024-03-06T07:28:51.903263+0000","last_became_peered":"2024-03-06T07:28:51.903263+0000","last_unstale":"2024-03-06T07:28:52.980431+0000","last_undegraded":"2024-03-06T07:28:52.980431+0000","last_fullsized":"2024-03-06T07:28:52.980431+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:32.221514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.987037+0000","last_change":"2024-03-06T07:28:50.901081+0000","last_active":"2024-03-06T07:28:52.987037+0000","last_peered":"2024-03-06T07:28:52.987037+0000","last_clean":"2024-03-06T07:28:52.987037+0000","last_became_active":"2024-03-06T07:28:50.900751+0000","last_became_peered":"2024-03-06T07:28:50.900751+0000","last_unstale":"2024-03-06T07:28:52.987037+0000","last_undegraded":"2024-03-06T07:28:52.987037+0000","last_fullsized":"2024-03-06T07:28:52.987037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:04.945213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980052+0000","last_change":"2024-03-06T07:28:51.900185+0000","last_active":"2024-03-06T07:28:52.980052+0000","last_peered":"2024-03-06T07:28:52.980052+0000","last_clean":"2024-03-06T07:28:52.980052+0000","last_became_active":"2024-03-06T07:28:51.899828+0000","last_became_peered":"2024-03-06T07:28:51.899828+0000","last_unstale":"2024-03-06T07:28:52.980052+0000","last_undegraded":"2024-03-06T07:28:52.980052+0000","last_fullsized":"2024-03-06T07:28:52.980052+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:10.448027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905081+0000","last_change":"2024-03-06T07:28:50.893526+0000","last_active":"2024-03-06T07:28:52.905081+0000","last_peered":"2024-03-06T07:28:52.905081+0000","last_clean":"2024-03-06T07:28:52.905081+0000","last_became_active":"2024-03-06T07:28:50.893253+0000","last_became_peered":"2024-03-06T07:28:50.893253+0000","last_unstale":"2024-03-06T07:28:52.905081+0000","last_undegraded":"2024-03-06T07:28:52.905081+0000","last_fullsized":"2024-03-06T07:28:52.905081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:09.434156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980297+0000","last_change":"2024-03-06T07:28:51.892668+0000","last_active":"2024-03-06T07:28:52.980297+0000","last_peered":"2024-03-06T07:28:52.980297+0000","last_clean":"2024-03-06T07:28:52.980297+0000","last_became_active":"2024-03-06T07:28:51.892223+0000","last_became_peered":"2024-03-06T07:28:51.892223+0000","last_unstale":"2024-03-06T07:28:52.980297+0000","last_undegraded":"2024-03-06T07:28:52.980297+0000","last_fullsized":"2024-03-06T07:28:52.980297+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:15.961587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983545+0000","last_change":"2024-03-06T07:28:50.901458+0000","last_active":"2024-03-06T07:28:52.983545+0000","last_peered":"2024-03-06T07:28:52.983545+0000","last_clean":"2024-03-06T07:28:52.983545+0000","last_became_active":"2024-03-06T07:28:50.900891+0000","last_became_peered":"2024-03-06T07:28:50.900891+0000","last_unstale":"2024-03-06T07:28:52.983545+0000","last_undegraded":"2024-03-06T07:28:52.983545+0000","last_fullsized":"2024-03-06T07:28:52.983545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:09.434156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980075+0000","last_change":"2024-03-06T07:28:51.900038+0000","last_active":"2024-03-06T07:28:52.980075+0000","last_peered":"2024-03-06T07:28:52.980075+0000","last_clean":"2024-03-06T07:28:52.980075+0000","last_became_active":"2024-03-06T07:28:51.889244+0000","last_became_peered":"2024-03-06T07:28:51.889244+0000","last_unstale":"2024-03-06T07:28:52.980075+0000","last_undegraded":"2024-03-06T07:28:52.980075+0000","last_fullsized":"2024-03-06T07:28:52.980075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:19.911872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981342+0000","last_change":"2024-03-06T07:28:50.898611+0000","last_active":"2024-03-06T07:28:52.981342+0000","last_peered":"2024-03-06T07:28:52.981342+0000","last_clean":"2024-03-06T07:28:52.981342+0000","last_became_active":"2024-03-06T07:28:50.895588+0000","last_became_peered":"2024-03-06T07:28:50.895588+0000","last_unstale":"2024-03-06T07:28:52.981342+0000","last_undegraded":"2024-03-06T07:28:52.981342+0000","last_fullsized":"2024-03-06T07:28:52.981342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:19.106856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905471+0000","last_change":"2024-03-06T07:28:51.904954+0000","last_active":"2024-03-06T07:28:52.905471+0000","last_peered":"2024-03-06T07:28:52.905471+0000","last_clean":"2024-03-06T07:28:52.905471+0000","last_became_active":"2024-03-06T07:28:51.904605+0000","last_became_peered":"2024-03-06T07:28:51.904605+0000","last_unstale":"2024-03-06T07:28:52.905471+0000","last_undegraded":"2024-03-06T07:28:52.905471+0000","last_fullsized":"2024-03-06T07:28:52.905471+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.696454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981454+0000","last_change":"2024-03-06T07:28:50.903865+0000","last_active":"2024-03-06T07:28:52.981454+0000","last_peered":"2024-03-06T07:28:52.981454+0000","last_clean":"2024-03-06T07:28:52.981454+0000","last_became_active":"2024-03-06T07:28:50.903476+0000","last_became_peered":"2024-03-06T07:28:50.903476+0000","last_unstale":"2024-03-06T07:28:52.981454+0000","last_undegraded":"2024-03-06T07:28:52.981454+0000","last_fullsized":"2024-03-06T07:28:52.981454+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:18.949654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.980340+0000","last_change":"2024-03-06T07:28:51.923858+0000","last_active":"2024-03-06T07:28:52.980340+0000","last_peered":"2024-03-06T07:28:52.980340+0000","last_clean":"2024-03-06T07:28:52.980340+0000","last_became_active":"2024-03-06T07:28:51.921911+0000","last_became_peered":"2024-03-06T07:28:51.921911+0000","last_unstale":"2024-03-06T07:28:52.980340+0000","last_undegraded":"2024-03-06T07:28:52.980340+0000","last_fullsized":"2024-03-06T07:28:52.980340+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:48.047781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900618+0000","last_change":"2024-03-06T07:28:50.890751+0000","last_active":"2024-03-06T07:28:52.900618+0000","last_peered":"2024-03-06T07:28:52.900618+0000","last_clean":"2024-03-06T07:28:52.900618+0000","last_became_active":"2024-03-06T07:28:50.890243+0000","last_became_peered":"2024-03-06T07:28:50.890243+0000","last_unstale":"2024-03-06T07:28:52.900618+0000","last_undegraded":"2024-03-06T07:28:52.900618+0000","last_fullsized":"2024-03-06T07:28:52.900618+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:05.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.985908+0000","last_change":"2024-03-06T07:28:51.925821+0000","last_active":"2024-03-06T07:28:52.985908+0000","last_peered":"2024-03-06T07:28:52.985908+0000","last_clean":"2024-03-06T07:28:52.985908+0000","last_became_active":"2024-03-06T07:28:51.925480+0000","last_became_peered":"2024-03-06T07:28:51.925480+0000","last_unstale":"2024-03-06T07:28:52.985908+0000","last_undegraded":"2024-03-06T07:28:52.985908+0000","last_fullsized":"2024-03-06T07:28:52.985908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:10.489955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"35'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.907045+0000","last_change":"2024-03-06T07:28:50.884638+0000","last_active":"2024-03-06T07:28:52.907045+0000","last_peered":"2024-03-06T07:28:52.907045+0000","last_clean":"2024-03-06T07:28:52.907045+0000","last_became_active":"2024-03-06T07:28:50.884298+0000","last_became_peered":"2024-03-06T07:28:50.884298+0000","last_unstale":"2024-03-06T07:28:52.907045+0000","last_undegraded":"2024-03-06T07:28:52.907045+0000","last_fullsized":"2024-03-06T07:28:52.907045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:05.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905152+0000","last_change":"2024-03-06T07:28:51.896390+0000","last_active":"2024-03-06T07:28:52.905152+0000","last_peered":"2024-03-06T07:28:52.905152+0000","last_clean":"2024-03-06T07:28:52.905152+0000","last_became_active":"2024-03-06T07:28:51.895776+0000","last_became_peered":"2024-03-06T07:28:51.895776+0000","last_unstale":"2024-03-06T07:28:52.905152+0000","last_undegraded":"2024-03-06T07:28:52.905152+0000","last_fullsized":"2024-03-06T07:28:52.905152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:25.949576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.900108+0000","last_change":"2024-03-06T07:28:50.889001+0000","last_active":"2024-03-06T07:28:52.900108+0000","last_peered":"2024-03-06T07:28:52.900108+0000","last_clean":"2024-03-06T07:28:52.900108+0000","last_became_active":"2024-03-06T07:28:50.888686+0000","last_became_peered":"2024-03-06T07:28:50.888686+0000","last_unstale":"2024-03-06T07:28:52.900108+0000","last_undegraded":"2024-03-06T07:28:52.900108+0000","last_fullsized":"2024-03-06T07:28:52.900108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.304948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983331+0000","last_change":"2024-03-06T07:28:51.907867+0000","last_active":"2024-03-06T07:28:52.983331+0000","last_peered":"2024-03-06T07:28:52.983331+0000","last_clean":"2024-03-06T07:28:52.983331+0000","last_became_active":"2024-03-06T07:28:51.907530+0000","last_became_peered":"2024-03-06T07:28:51.907530+0000","last_unstale":"2024-03-06T07:28:52.983331+0000","last_undegraded":"2024-03-06T07:28:52.983331+0000","last_fullsized":"2024-03-06T07:28:52.983331+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:32.221514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905264+0000","last_change":"2024-03-06T07:28:50.893960+0000","last_active":"2024-03-06T07:28:52.905264+0000","last_peered":"2024-03-06T07:28:52.905264+0000","last_clean":"2024-03-06T07:28:52.905264+0000","last_became_active":"2024-03-06T07:28:50.893680+0000","last_became_peered":"2024-03-06T07:28:50.893680+0000","last_unstale":"2024-03-06T07:28:52.905264+0000","last_undegraded":"2024-03-06T07:28:52.905264+0000","last_fullsized":"2024-03-06T07:28:52.905264+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:18.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905185+0000","last_change":"2024-03-06T07:28:51.897697+0000","last_active":"2024-03-06T07:28:52.905185+0000","last_peered":"2024-03-06T07:28:52.905185+0000","last_clean":"2024-03-06T07:28:52.905185+0000","last_became_active":"2024-03-06T07:28:51.897432+0000","last_became_peered":"2024-03-06T07:28:51.897432+0000","last_unstale":"2024-03-06T07:28:52.905185+0000","last_undegraded":"2024-03-06T07:28:52.905185+0000","last_fullsized":"2024-03-06T07:28:52.905185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:22.203477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.899562+0000","last_change":"2024-03-06T07:28:50.890259+0000","last_active":"2024-03-06T07:28:52.899562+0000","last_peered":"2024-03-06T07:28:52.899562+0000","last_clean":"2024-03-06T07:28:52.899562+0000","last_became_active":"2024-03-06T07:28:50.889891+0000","last_became_peered":"2024-03-06T07:28:50.889891+0000","last_unstale":"2024-03-06T07:28:52.899562+0000","last_undegraded":"2024-03-06T07:28:52.899562+0000","last_fullsized":"2024-03-06T07:28:52.899562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:47.033910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902405+0000","last_change":"2024-03-06T07:28:51.899489+0000","last_active":"2024-03-06T07:28:52.902405+0000","last_peered":"2024-03-06T07:28:52.902405+0000","last_clean":"2024-03-06T07:28:52.902405+0000","last_became_active":"2024-03-06T07:28:51.899208+0000","last_became_peered":"2024-03-06T07:28:51.899208+0000","last_unstale":"2024-03-06T07:28:52.902405+0000","last_undegraded":"2024-03-06T07:28:52.902405+0000","last_fullsized":"2024-03-06T07:28:52.902405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:25.099777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.982054+0000","last_change":"2024-03-06T07:28:50.893091+0000","last_active":"2024-03-06T07:28:52.982054+0000","last_peered":"2024-03-06T07:28:52.982054+0000","last_clean":"2024-03-06T07:28:52.982054+0000","last_became_active":"2024-03-06T07:28:50.892720+0000","last_became_peered":"2024-03-06T07:28:50.892720+0000","last_unstale":"2024-03-06T07:28:52.982054+0000","last_undegraded":"2024-03-06T07:28:52.982054+0000","last_fullsized":"2024-03-06T07:28:52.982054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:28.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.981204+0000","last_change":"2024-03-06T07:28:51.904718+0000","last_active":"2024-03-06T07:28:52.981204+0000","last_peered":"2024-03-06T07:28:52.981204+0000","last_clean":"2024-03-06T07:28:52.981204+0000","last_became_active":"2024-03-06T07:28:51.904377+0000","last_became_peered":"2024-03-06T07:28:51.904377+0000","last_unstale":"2024-03-06T07:28:52.981204+0000","last_undegraded":"2024-03-06T07:28:52.981204+0000","last_fullsized":"2024-03-06T07:28:52.981204+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:29.759579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.902440+0000","last_change":"2024-03-06T07:28:50.899035+0000","last_active":"2024-03-06T07:28:52.902440+0000","last_peered":"2024-03-06T07:28:52.902440+0000","last_clean":"2024-03-06T07:28:52.902440+0000","last_became_active":"2024-03-06T07:28:50.898719+0000","last_became_peered":"2024-03-06T07:28:50.898719+0000","last_unstale":"2024-03-06T07:28:52.902440+0000","last_undegraded":"2024-03-06T07:28:52.902440+0000","last_fullsized":"2024-03-06T07:28:52.902440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:18.949654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905796+0000","last_change":"2024-03-06T07:28:51.920756+0000","last_active":"2024-03-06T07:28:52.905796+0000","last_peered":"2024-03-06T07:28:52.905796+0000","last_clean":"2024-03-06T07:28:52.905796+0000","last_became_active":"2024-03-06T07:28:51.917449+0000","last_became_peered":"2024-03-06T07:28:51.917449+0000","last_unstale":"2024-03-06T07:28:52.905796+0000","last_undegraded":"2024-03-06T07:28:52.905796+0000","last_fullsized":"2024-03-06T07:28:52.905796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.847470+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.847470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:23.627303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.983258+0000","last_change":"2024-03-06T07:28:50.902919+0000","last_active":"2024-03-06T07:28:52.983258+0000","last_peered":"2024-03-06T07:28:52.983258+0000","last_clean":"2024-03-06T07:28:52.983258+0000","last_became_active":"2024-03-06T07:28:50.902645+0000","last_became_peered":"2024-03-06T07:28:50.902645+0000","last_unstale":"2024-03-06T07:28:52.983258+0000","last_undegraded":"2024-03-06T07:28:52.983258+0000","last_fullsized":"2024-03-06T07:28:52.983258+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:49.833599+0000","last_clean_scrub_stamp":"2024-03-06T07:28:49.833599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:18.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:52.905816+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:28:52.905816+0000","last_peered":"2024-03-06T07:28:52.905816+0000","last_clean":"2024-03-06T07:28:52.905816+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:28:52.905816+0000","last_undegraded":"2024-03-06T07:28:52.905816+0000","last_fullsized":"2024-03-06T07:28:52.905816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":991232,"data_stored":745550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":155196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":156726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":148862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":156726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":745460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:56.773 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:56.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.015+0000 7f2264400640 1 Processor -- start 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.015+0000 7f2264400640 1 -- start start 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2264400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f225c066f10 0x7f225c0c3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2264400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0afcd0 0x7f225c0b0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2264400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c0673a0 con 0x7f225c0c3320 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2264400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c0727a0 con 0x7f225c066f10 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2264400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c073520 con 0x7f225c0afcd0 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f225c0c3320 0x7f225c06c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39686/0 (socket says 172.21.15.97:39686) 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261a00640 1 -- 172.21.15.97:0/114571118 learned_addr learned my addr 172.21.15.97:0/114571118 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261000640 1 --2- 172.21.15.97:0/114571118 >> v2:172.21.15.97:3300/0 conn(0x7f225c066f10 0x7f225c0c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2262400640 1 --2- 172.21.15.97:0/114571118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0afcd0 0x7f225c0b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261000640 1 -- 172.21.15.97:0/114571118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0afcd0 msgr2=0x7f225c0b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261000640 1 --2- 172.21.15.97:0/114571118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0afcd0 0x7f225c0b0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261000640 1 -- 172.21.15.97:0/114571118 >> v1:172.21.15.97:6789/0 conn(0x7f225c0c3320 legacy=0x7f225c06c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f2261000640 1 -- 172.21.15.97:0/114571118 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c072a40 con 0x7f225c066f10 2024-03-06T07:28:57.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.017+0000 7f225be00640 1 -- 172.21.15.97:0/114571118 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896571151 0 0) 0x7f225c0673a0 con 0x7f225c0c3320 2024-03-06T07:28:57.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.018+0000 7f2261000640 1 --2- 172.21.15.97:0/114571118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c066f10 0x7f225c0c3700 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22480097a0 tx=0x7f224802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8237ffee4c2915e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f225be00640 1 -- 172.21.15.97:0/114571118 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248038070 con 0x7f225c066f10 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f225be00640 1 -- 172.21.15.97:0/114571118 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2248031700 con 0x7f225c066f10 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f225be00640 1 -- 172.21.15.97:0/114571118 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248031ad0 con 0x7f225c066f10 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 -- 172.21.15.97:0/114571118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c066f10 msgr2=0x7f225c0c3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 --2- 172.21.15.97:0/114571118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c066f10 0x7f225c0c3700 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22480097a0 tx=0x7f224802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 -- 172.21.15.97:0/114571118 shutdown_connections 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 --2- 172.21.15.97:0/114571118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0afcd0 0x7f225c0b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 --2- 172.21.15.97:0/114571118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c066f10 0x7f225c0c3700 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 -- 172.21.15.97:0/114571118 >> 172.21.15.97:0/114571118 conn(0x7f225c0b1e80 msgr2=0x7f225c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 -- 172.21.15.97:0/114571118 shutdown_connections 2024-03-06T07:28:57.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.019+0000 7f2264400640 1 -- 172.21.15.97:0/114571118 wait complete. 2024-03-06T07:28:57.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.022+0000 7f2264400640 1 Processor -- start 2024-03-06T07:28:57.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.022+0000 7f2264400640 1 -- start start 2024-03-06T07:28:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2264400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 0x7f225c160850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2264400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0c3320 0x7f225c0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2264400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f225c0c75f0 0x7f225c1fb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2264400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0c3c40 con 0x7f225c0afcd0 2024-03-06T07:28:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2264400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c073000 con 0x7f225c0c75f0 2024-03-06T07:28:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2264400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0b2bd0 con 0x7f225c0c3320 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 0x7f225c160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 0x7f225c160850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53246/0 (socket says 172.21.15.97:53246) 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 -- 172.21.15.97:0/1568065234 learned_addr learned my addr 172.21.15.97:0/1568065234 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261000640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0c3320 0x7f225c0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2262400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f225c0c75f0 0x7f225c1fb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 -- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0c3320 msgr2=0x7f225c0c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0c3320 0x7f225c0c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 -- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f225c0c75f0 msgr2=0x7f225c1fb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f225c0c75f0 0x7f225c1fb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261a00640 1 -- 172.21.15.97:0/1568065234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c0c8ac0 con 0x7f225c0afcd0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.024+0000 7f2261000640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0c3320 0x7f225c0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.025+0000 7f2261a00640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 0x7f225c160850 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f225000c430 tx=0x7f225000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.025+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500074e0 con 0x7f225c0afcd0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.025+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22500047e0 con 0x7f225c0afcd0 2024-03-06T07:28:57.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.025+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c0c8da0 con 0x7f225c0afcd0 2024-03-06T07:28:57.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.025+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250004bd0 con 0x7f225c0afcd0 2024-03-06T07:28:57.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.025+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c1fba30 con 0x7f225c0afcd0 2024-03-06T07:28:57.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.026+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c1644f0 con 0x7f225c0afcd0 2024-03-06T07:28:57.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.027+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2250011070 con 0x7f225c0afcd0 2024-03-06T07:28:57.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.028+0000 7f2259600640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f223c06b4d0 0x7f223c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.028+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2250089e10 con 0x7f225c0afcd0 2024-03-06T07:28:57.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.028+0000 7f2261000640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f223c06b4d0 0x7f223c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.029+0000 7f2261000640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f223c06b4d0 0x7f223c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2248035000 tx=0x7f2248008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.033+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2250058d00 con 0x7f225c0afcd0 2024-03-06T07:28:57.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.288+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f225c04f330 con 0x7f225c0afcd0 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.292+0000 7f2259600640 1 -- 172.21.15.97:0/1568065234 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2250058600 con 0x7f225c0afcd0 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.294+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f223c06b4d0 msgr2=0x7f223c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.294+0000 7f2264400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f223c06b4d0 0x7f223c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2248035000 tx=0x7f2248008030 comp rx=0 tx=0).stop 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.294+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 msgr2=0x7f225c160850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.294+0000 7f2264400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 0x7f225c160850 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f225000c430 tx=0x7f225000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 shutdown_connections 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f223c06b4d0 0x7f223c06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f225c0c75f0 0x7f225c1fb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f225c0c3320 0x7f225c0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 --2- 172.21.15.97:0/1568065234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f225c0afcd0 0x7f225c160850 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 >> 172.21.15.97:0/1568065234 conn(0x7f225c0b1e80 msgr2=0x7f225c0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 shutdown_connections 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.295+0000 7f2264400640 1 -- 172.21.15.97:0/1568065234 wait complete. 2024-03-06T07:28:57.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:57.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.510+0000 7fdd80e00640 1 Processor -- start 2024-03-06T07:28:57.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.511+0000 7fdd80e00640 1 -- start start 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd80e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdd7c15f8d0 0x7fdd7c0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 0x7fdd7c15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd80e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c070de0 con 0x7fdd7c15f4f0 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd80e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7c0cd0a0 con 0x7fdd7c15f8d0 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd80e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c160a20 con 0x7fdd7c0ccc40 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd79e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdd7c15f4f0 0x7fdd7c0706d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39696/0 (socket says 172.21.15.97:39696) 2024-03-06T07:28:57.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd79e00640 1 -- 172.21.15.97:0/4160055098 learned_addr learned my addr 172.21.15.97:0/4160055098 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:57.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.513+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269331247 0 0) 0x7fdd7c070de0 con 0x7fdd7c15f4f0 2024-03-06T07:28:57.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd79400640 1 --2- 172.21.15.97:0/4160055098 >> v2:172.21.15.135:3300/0 conn(0x7fdd7c15f8d0 0x7fdd7c0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd7a800640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 0x7fdd7c15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd60003520 con 0x7fdd7c15f4f0 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218991098 0 0) 0x7fdd60003520 con 0x7fdd7c15f4f0 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 msgr2=0x7fdd7c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 0x7fdd7c15fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 >> v2:172.21.15.135:3300/0 conn(0x7fdd7c15f8d0 msgr2=0x7fdd7c0cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 --2- 172.21.15.97:0/4160055098 >> v2:172.21.15.135:3300/0 conn(0x7fdd7c15f8d0 0x7fdd7c0cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c06ef50 con 0x7fdd7c15f4f0 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdd6c001690 con 0x7fdd7c15f4f0 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 >> v1:172.21.15.97:6789/0 conn(0x7fdd7c15f4f0 legacy=0x7fdd7c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd600039b0 0x7fdd60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd60004800 0x7fdd60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 0x7fdd600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c0cd0a0 con 0x7fdd600039b0 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c160a20 con 0x7fdd60004800 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6000baf0 con 0x7fdd7c0ccc40 2024-03-06T07:28:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.514+0000 7fdd79e00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd600039b0 0x7fdd60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79400640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd60004800 0x7fdd60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79e00640 1 -- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 msgr2=0x7fdd600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd7a800640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 0x7fdd600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79e00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c0ccc40 0x7fdd600051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79e00640 1 -- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd60004800 msgr2=0x7fdd60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79e00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd60004800 0x7fdd60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79e00640 1 -- 172.21.15.97:0/4160055098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60003520 con 0x7fdd600039b0 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.515+0000 7fdd79400640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd60004800 0x7fdd60004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.516+0000 7fdd79e00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd600039b0 0x7fdd60003dd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0b1620 tx=0x7fdd60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8598936a318c2f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.516+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c007ca0 con 0x7fdd600039b0 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.516+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c030070 con 0x7fdd600039b0 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.516+0000 7fdd78a00640 1 -- 172.21.15.97:0/4160055098 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c005d10 con 0x7fdd600039b0 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd80e00640 1 -- 172.21.15.97:0/4160055098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd600039b0 msgr2=0x7fdd60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd80e00640 1 --2- 172.21.15.97:0/4160055098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd600039b0 0x7fdd60003dd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0b1620 tx=0x7fdd60005830 comp rx=0 tx=0).stop 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd7a800640 1 -- 172.21.15.97:0/4160055098 reap_dead start 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd80e00640 1 -- 172.21.15.97:0/4160055098 shutdown_connections 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd80e00640 1 -- 172.21.15.97:0/4160055098 >> 172.21.15.97:0/4160055098 conn(0x7fdd7c0b3800 msgr2=0x7fdd7c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd80e00640 1 -- 172.21.15.97:0/4160055098 shutdown_connections 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.517+0000 7fdd80e00640 1 -- 172.21.15.97:0/4160055098 wait complete. 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.518+0000 7fdd80e00640 1 Processor -- start 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.519+0000 7fdd80e00640 1 -- start start 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd600039b0 0x7fdd7c0c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 0x7fdd7c0b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c159740 0x7fdd7c0b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd80e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c15fed0 con 0x7fdd7c1592c0 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd80e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c15fd50 con 0x7fdd600039b0 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd80e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c160050 con 0x7fdd7c159740 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 0x7fdd7c0b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 0x7fdd7c0b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53268/0 (socket says 172.21.15.97:53268) 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 -- 172.21.15.97:0/824941403 learned_addr learned my addr 172.21.15.97:0/824941403 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 -- 172.21.15.97:0/824941403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c159740 msgr2=0x7fdd7c0b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c159740 0x7fdd7c0b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 -- 172.21.15.97:0/824941403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd600039b0 msgr2=0x7fdd7c0c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd600039b0 0x7fdd7c0c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.520+0000 7fdd79400640 1 -- 172.21.15.97:0/824941403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c15dd30 con 0x7fdd7c1592c0 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.521+0000 7fdd79400640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 0x7fdd7c0b5b30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800e7e0 tx=0x7fdd6800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.521+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd68017070 con 0x7fdd7c1592c0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.521+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6800c6d0 con 0x7fdd7c1592c0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.521+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c15bf80 con 0x7fdd7c1592c0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.521+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800f040 con 0x7fdd7c1592c0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.521+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c0b5330 con 0x7fdd7c1592c0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.522+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c04f330 con 0x7fdd7c1592c0 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.523+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd6801a070 con 0x7fdd7c1592c0 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.524+0000 7fdd72000640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5c06b500 0x7fdd5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.524+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd68089d50 con 0x7fdd7c1592c0 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.524+0000 7fdd79e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5c06b500 0x7fdd5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.525+0000 7fdd79e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5c06b500 0x7fdd5c06d9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c034000 tx=0x7fdd6c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.531+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd68058b30 con 0x7fdd7c1592c0 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.750+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd7c0cff00 con 0x7fdd7c1592c0 2024-03-06T07:28:57.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.751+0000 7fdd72000640 1 -- 172.21.15.97:0/824941403 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fdd68058430 con 0x7fdd7c1592c0 2024-03-06T07:28:57.753 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:28:57.753 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.753+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5c06b500 msgr2=0x7fdd5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.753+0000 7fdd80e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5c06b500 0x7fdd5c06d9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c034000 tx=0x7fdd6c02e000 comp rx=0 tx=0).stop 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.753+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 msgr2=0x7fdd7c0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.753+0000 7fdd80e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 0x7fdd7c0b5b30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800e7e0 tx=0x7fdd6800eca0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.753+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 shutdown_connections 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.753+0000 7fdd80e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5c06b500 0x7fdd5c06d9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.754+0000 7fdd80e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7c159740 0x7fdd7c0b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.754+0000 7fdd80e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7c1592c0 0x7fdd7c0b5b30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.754+0000 7fdd80e00640 1 --2- 172.21.15.97:0/824941403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd600039b0 0x7fdd7c0c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.754+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 >> 172.21.15.97:0/824941403 conn(0x7fdd7c0b3800 msgr2=0x7fdd7c0cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.754+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 shutdown_connections 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:57.754+0000 7fdd80e00640 1 -- 172.21.15.97:0/824941403 wait complete. 2024-03-06T07:28:58.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:51.345690+0000', 'modified': '2024-03-06T07:28:54.426245+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5052}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5052': {'gid': 5052, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.97:6837/4130179477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 4130179477}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 4130179477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5052, 'qdb_cluster': [5052]} 2024-03-06T07:28:58.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:58.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:58.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.403+0000 7fc6fbe00640 1 Processor -- start 2024-03-06T07:28:58.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.403+0000 7fc6fbe00640 1 -- start start 2024-03-06T07:28:58.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.404+0000 7fc6fbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc6fc161580 0x7fc6fc0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6fbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc0d2a70 0x7fc6fc0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6fbe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc161960 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6fbe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6fc0cf6f0 con 0x7fc6fc161580 2024-03-06T07:28:58.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6fbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0cf9d0 con 0x7fc6fc0d2a70 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc6fc161580 0x7fc6fc0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc6fc161580 0x7fc6fc0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53272/0 (socket says 172.21.15.97:53272) 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 -- 172.21.15.97:0/3712771635 learned_addr learned my addr 172.21.15.97:0/3712771635 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6fb400640 1 --2- 172.21.15.97:0/3712771635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc0d2a70 0x7fc6fc0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 -- 172.21.15.97:0/3712771635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc0d2a70 msgr2=0x7fc6fc0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3712771635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc0d2a70 0x7fc6fc0b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 -- 172.21.15.97:0/3712771635 >> v1:172.21.15.135:6789/0 conn(0x7fc6fc0d1e90 legacy=0x7fc6fc0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.405+0000 7fc6faa00640 1 -- 172.21.15.97:0/3712771635 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0cfcb0 con 0x7fc6fc161580 2024-03-06T07:28:58.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.406+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3712771635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc161580 0x7fc6fc0d1950 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc073740 tx=0x7fc6ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e00a50e20a79920 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fa000640 1 -- 172.21.15.97:0/3712771635 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec035020 con 0x7fc6fc161580 2024-03-06T07:28:58.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fa000640 1 -- 172.21.15.97:0/3712771635 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec031640 con 0x7fc6fc161580 2024-03-06T07:28:58.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fa000640 1 -- 172.21.15.97:0/3712771635 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec02d040 con 0x7fc6fc161580 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3712771635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc161580 msgr2=0x7fc6fc0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3712771635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc161580 0x7fc6fc0d1950 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc073740 tx=0x7fc6ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3712771635 shutdown_connections 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3712771635 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc0d2a70 0x7fc6fc0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3712771635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc161580 0x7fc6fc0d1950 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.407+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3712771635 >> 172.21.15.97:0/3712771635 conn(0x7fc6fc0d3d80 msgr2=0x7fc6fc0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.408+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3712771635 shutdown_connections 2024-03-06T07:28:58.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.408+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3712771635 wait complete. 2024-03-06T07:28:58.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.409+0000 7fc6fbe00640 1 Processor -- start 2024-03-06T07:28:58.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.409+0000 7fc6fbe00640 1 -- start start 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6fbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 0x7fc6fc0b9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6fbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6fc0d2a70 0x7fc6fc0cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6fbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc161580 0x7fc6fc0b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6fbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0cf3a0 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6fbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0b39a0 con 0x7fc6fc0d2a70 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6fbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0b4400 con 0x7fc6fc161580 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6faa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 0x7fc6fc0b9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6faa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 0x7fc6fc0b9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53288/0 (socket says 172.21.15.97:53288) 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.410+0000 7fc6faa00640 1 -- 172.21.15.97:0/3174698125 learned_addr learned my addr 172.21.15.97:0/3174698125 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6faa00640 1 -- 172.21.15.97:0/3174698125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc161580 msgr2=0x7fc6fc0b8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc161580 0x7fc6fc0b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6faa00640 1 -- 172.21.15.97:0/3174698125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6fc0d2a70 msgr2=0x7fc6fc0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6fc0d2a70 0x7fc6fc0cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6faa00640 1 -- 172.21.15.97:0/3174698125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0ccdf0 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 0x7fc6fc0b9820 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec02b930 tx=0x7fc6ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6f2a00640 1 -- 172.21.15.97:0/3174698125 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec03ac60 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6f2a00640 1 -- 172.21.15.97:0/3174698125 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec03ae40 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6f2a00640 1 -- 172.21.15.97:0/3174698125 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec044630 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.411+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc0cd090 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.412+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc0cd650 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.414+0000 7fc6f2a00640 1 -- 172.21.15.97:0/3174698125 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6ec038020 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.414+0000 7fc6f2a00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6d806b4d0 0x7fc6d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.414+0000 7fc6f2a00640 1 -- 172.21.15.97:0/3174698125 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6ec0ac110 con 0x7fc6fc0d1e90 2024-03-06T07:28:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.415+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6d806b4d0 0x7fc6d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.415+0000 7fc70459a740 1 Processor -- start 2024-03-06T07:28:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.415+0000 7fc70459a740 1 -- start start 2024-03-06T07:28:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.415+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6d806b4d0 0x7fc6d806d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0d12f0 tx=0x7fc6e4007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0990 0x5636b6f73930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fa3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc70459a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 0x5636b6fa3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc70459a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636b6edd600 con 0x5636b6f73e70 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636b6e86930 con 0x5636b6fa0990 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636b6ed96a0 con 0x5636b6fa0d70 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.416+0000 7fc6fb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 0x5636b6fa3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 0x5636b6fa3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53298/0 (socket says 172.21.15.97:53298) 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 -- 172.21.15.97:0/2273563487 learned_addr learned my addr 172.21.15.97:0/2273563487 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6faa00640 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0990 0x5636b6f73930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 -- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 msgr2=0x5636b6fa3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fa3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fa3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 -- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0990 msgr2=0x5636b6f73930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0990 0x5636b6f73930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 -- 172.21.15.97:0/2273563487 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636b6bf22d0 con 0x5636b6f73e70 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.417+0000 7fc6fb400640 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 0x5636b6fa3c60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4007ec0 tx=0x7fc6f400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afa3050530c8cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc6eb400640 1 -- 172.21.15.97:0/2273563487 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4003b90 con 0x5636b6f73e70 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc6eb400640 1 -- 172.21.15.97:0/2273563487 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4003d30 con 0x5636b6f73e70 2024-03-06T07:28:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc6eb400640 1 -- 172.21.15.97:0/2273563487 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4004240 con 0x5636b6f73e70 2024-03-06T07:28:58.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 -- 172.21.15.97:0/2273563487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 msgr2=0x5636b6fa3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 0x5636b6fa3c60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4007ec0 tx=0x7fc6f400c6e0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 -- 172.21.15.97:0/2273563487 shutdown_connections 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f73e70 0x5636b6fa3c60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fa3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 --2- 172.21.15.97:0/2273563487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0990 0x5636b6f73930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 -- 172.21.15.97:0/2273563487 >> 172.21.15.97:0/2273563487 conn(0x5636b6f75ba0 msgr2=0x5636b6fbe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 -- 172.21.15.97:0/2273563487 shutdown_connections 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.418+0000 7fc70459a740 1 -- 172.21.15.97:0/2273563487 wait complete. 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.419+0000 7fc70459a740 1 -- start start 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.419+0000 7fc70459a740 1 Processor -- start 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 0x5636b6fd1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fcf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6f9c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636b6f78140 con 0x5636b6fa0990 2024-03-06T07:28:58.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636b6e99d90 con 0x5636b6f9bc60 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636b6f77530 con 0x5636b6fa0d70 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc70459a740 10 client.0 _ll_register_callbacks cb 0x5636b6ed5350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.420+0000 7fc6faa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 0x5636b6fd1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 0x5636b6fd1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53304/0 (socket says 172.21.15.97:53304) 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 -- 172.21.15.97:0/3466656216 learned_addr learned my addr 172.21.15.97:0/3466656216 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6fb400640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6f9c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 -- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 msgr2=0x5636b6fcf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fcf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fcf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 -- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 msgr2=0x5636b6f9c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6f9c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 -- 172.21.15.97:0/3466656216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636b6cbc6b0 con 0x5636b6fa0990 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6fb400640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6f9c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 0x5636b6fd1df0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec07a810 tx=0x7fc6ec07a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecdc27891fe24db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6dfe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:28:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6dfe00640 1 -- 172.21.15.97:0/3466656216 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec07ace0 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.421+0000 7fc6dfe00640 1 -- 172.21.15.97:0/3466656216 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6ec082b90 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6f4003d30 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc6dfe00640 1 -- 172.21.15.97:0/3466656216 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6ec07e9d0 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 10 client.5226 fetch_fsmap learned FSMap version 17 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5636b6fa2460 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc6dfe00640 1 -- 172.21.15.97:0/3466656216 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4003d30 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc6dfe00640 1 -- 172.21.15.97:0/3466656216 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc6ec0aa120 con 0x5636b6fa0990 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 10 client.5226 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 10 client.5226 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 20 client.5226 populate_metadata read hostname 'smithi097' 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6fd28a0 0x5636b6fd2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 4 client.5226 mds_command: new command op to 5052 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 --> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636b6eeab40 con 0x5636b6fd28a0 2024-03-06T07:28:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.422+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6fd28a0 0x5636b6fd2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.423+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6fd28a0 0x5636b6fd2c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66c5aae35b974047 server_cookie=adbca1dfbf5ee881 in_seq=0 out_seq=0 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.423+0000 7fc6dfe00640 10 client.5226 ms_handle_connect on v2:172.21.15.97:6836/4130179477 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc6dfe00640 1 -- 172.21.15.97:0/3466656216 <== mds.0 v2:172.21.15.97:6836/4130179477 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5636b6eeab40 con 0x5636b6fd28a0 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc6dfe00640 10 client.5226 handle_command_reply: tid=0 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 client.5226 shutdown 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 msgr2=0x5636b6fd1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 0x5636b6fd1df0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec07a810 tx=0x7fc6ec07a2a0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 shutdown_connections 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6fd28a0 0x5636b6fd2c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6f9c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0d70 0x5636b6fcf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc6dfe00640 0 client.5226 ms_handle_reset on v2:172.21.15.97:6836/4130179477 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 --2- 172.21.15.97:0/3466656216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0990 0x5636b6fd1df0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.424+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 >> 172.21.15.97:0/3466656216 conn(0x5636b6f73e70 msgr2=0x5636b6fc1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.425+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 shutdown_connections 2024-03-06T07:28:58.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.425+0000 7fc70459a740 1 -- 172.21.15.97:0/3466656216 wait complete. 2024-03-06T07:28:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.425+0000 7fc70459a740 20 client.5226 trim_cache size 0 max 16384 2024-03-06T07:28:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.442+0000 7fc70459a740 1 Processor -- start 2024-03-06T07:28:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.442+0000 7fc70459a740 1 -- start start 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6fc10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fbea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc70459a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 0x5636b6f9aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc70459a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc200300 con 0x5636b6fa0d70 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc1ff400 con 0x5636b6f9bc60 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc1fe800 con 0x5636b6fa0990 2024-03-06T07:28:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.443+0000 7fc6fb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 0x5636b6f9aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 0x5636b6f9aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53320/0 (socket says 172.21.15.97:53320) 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 -- 172.21.15.97:0/3738462907 learned_addr learned my addr 172.21.15.97:0/3738462907 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6faa00640 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6fc10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fbea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 -- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 msgr2=0x5636b6fbea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fbea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 -- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 msgr2=0x5636b6fc10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6fc10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 -- 172.21.15.97:0/3738462907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636b6f99260 con 0x5636b6fa0d70 2024-03-06T07:28:58.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.444+0000 7fc6fb400640 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 0x5636b6f9aba0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4013000 tx=0x7fc6f400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c769b3bebbaff61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc6eaa00640 1 -- 172.21.15.97:0/3738462907 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4027070 con 0x5636b6fa0d70 2024-03-06T07:28:58.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc6eaa00640 1 -- 172.21.15.97:0/3738462907 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4003990 con 0x5636b6fa0d70 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc6eaa00640 1 -- 172.21.15.97:0/3738462907 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4003d80 con 0x5636b6fa0d70 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 -- 172.21.15.97:0/3738462907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 msgr2=0x5636b6f9aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 0x5636b6f9aba0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4013000 tx=0x7fc6f400a000 comp rx=0 tx=0).stop 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 -- 172.21.15.97:0/3738462907 shutdown_connections 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6fa0d70 0x5636b6f9aba0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fbea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 --2- 172.21.15.97:0/3738462907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6f9bc60 0x5636b6fc10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 -- 172.21.15.97:0/3738462907 >> 172.21.15.97:0/3738462907 conn(0x5636b6f73e70 msgr2=0x5636b6fa32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 -- 172.21.15.97:0/3738462907 shutdown_connections 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.445+0000 7fc70459a740 1 -- 172.21.15.97:0/3738462907 wait complete. 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.446+0000 7fc70459a740 1 -- start start 2024-03-06T07:28:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.446+0000 7fc70459a740 1 Processor -- start 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 0x5636b70316a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fd0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0d70 0x5636b6fd0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636b6e99d90 con 0x5636b6f9bc60 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc1fee00 con 0x5636b6fa0d70 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636b6f78140 con 0x5636b6fa0990 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc70459a740 10 client.0 _ll_register_callbacks cb 0x7fc6ec006e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc6faa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 0x5636b70316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc6faa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 0x5636b70316a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53336/0 (socket says 172.21.15.97:53336) 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.447+0000 7fc6faa00640 1 -- 172.21.15.97:0/1721986224 learned_addr learned my addr 172.21.15.97:0/1721986224 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6faa00640 1 -- 172.21.15.97:0/1721986224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 msgr2=0x5636b6fd0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6faa00640 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fd0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6faa00640 1 -- 172.21.15.97:0/1721986224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0d70 msgr2=0x5636b6fd0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6faa00640 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0d70 0x5636b6fd0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6faa00640 1 -- 172.21.15.97:0/1721986224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636b6fa71c0 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6faa00640 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 0x5636b70316a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec076ce0 tx=0x7fc6ec0aa470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21780cf61b7dc4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6dfe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6dfe00640 1 -- 172.21.15.97:0/1721986224 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec07bbb0 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6dfe00640 1 -- 172.21.15.97:0/1721986224 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6ec07bd50 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6f4003990 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc6dfe00640 1 -- 172.21.15.97:0/1721986224 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6ec092180 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc70459a740 10 client.5232 fetch_fsmap learned FSMap version 17 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.448+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5636b6fa71c0 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc6dfe00640 1 -- 172.21.15.97:0/1721986224 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4003990 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc6dfe00640 1 -- 172.21.15.97:0/1721986224 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc6ec092320 con 0x5636b6f9bc60 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc70459a740 10 client.5232 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:28:58.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc70459a740 10 client.5232 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc70459a740 20 client.5232 populate_metadata read hostname 'smithi097' 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc70459a740 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6f77910 0x5636b6fc1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc70459a740 4 client.5232 mds_command: new command op to 5052 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 --> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5636b6ee4e90 con 0x5636b6f77910 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6f77910 0x5636b6fc1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc6f3e00640 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6f77910 0x5636b6fc1c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34164816f5555460 server_cookie=ce915b8817de71b0 in_seq=0 out_seq=0 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.449+0000 7fc6dfe00640 10 client.5232 ms_handle_connect on v2:172.21.15.97:6836/4130179477 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc6dfe00640 1 -- 172.21.15.97:0/1721986224 <== mds.0 v2:172.21.15.97:6836/4130179477 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5636b6ee4e90 con 0x5636b6f77910 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc6dfe00640 10 client.5232 handle_command_reply: tid=0 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 client.5232 shutdown 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 msgr2=0x5636b70316a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 0x5636b70316a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec076ce0 tx=0x7fc6ec0aa470 comp rx=0 tx=0).stop 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 shutdown_connections 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:6836/4130179477,v1:172.21.15.97:6837/4130179477] conn(0x5636b6f77910 0x5636b6fc1c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5636b6fa0d70 0x5636b6fd0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5636b6fa0990 0x5636b6fd0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 --2- 172.21.15.97:0/1721986224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5636b6f9bc60 0x5636b70316a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 >> 172.21.15.97:0/1721986224 conn(0x5636b6f73e70 msgr2=0x5636b6fc0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc6dfe00640 0 client.5232 ms_handle_reset on v2:172.21.15.97:6836/4130179477 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 shutdown_connections 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 1 -- 172.21.15.97:0/1721986224 wait complete. 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.450+0000 7fc70459a740 20 client.5232 trim_cache size 0 max 16384 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 5052, 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:28:58.456 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 6.5607347149999997, 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 17, 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 35, 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 35, 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 15.401349728 2024-03-06T07:28:58.457 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6d806b4d0 msgr2=0x7fc6d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6d806b4d0 0x7fc6d806d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0d12f0 tx=0x7fc6e4007990 comp rx=0 tx=0).stop 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 msgr2=0x7fc6fc0b9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 0x7fc6fc0b9820 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec02b930 tx=0x7fc6ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 shutdown_connections 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6d806b4d0 0x7fc6d806d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6fc161580 0x7fc6fc0b8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6fc0d2a70 0x7fc6fc0cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 --2- 172.21.15.97:0/3174698125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6fc0d1e90 0x7fc6fc0b9820 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 >> 172.21.15.97:0/3174698125 conn(0x7fc6fc0d3d80 msgr2=0x7fc6fc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 shutdown_connections 2024-03-06T07:28:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.451+0000 7fc6fbe00640 1 -- 172.21.15.97:0/3174698125 wait complete. 2024-03-06T07:28:58.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:58.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.666+0000 7f7689200640 1 Processor -- start 2024-03-06T07:28:58.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.666+0000 7f7689200640 1 -- start start 2024-03-06T07:28:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.667+0000 7f7689200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 0x7f76840b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.667+0000 7f7689200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768415b090 con 0x7f76840b4680 2024-03-06T07:28:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.667+0000 7f7689200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76840b2a30 con 0x7f768406ec00 2024-03-06T07:28:58.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.667+0000 7f7689200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840b4360 con 0x7f768415acb0 2024-03-06T07:28:58.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.668+0000 7f7681400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f76840b4680 0x7f768406e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39704/0 (socket says 172.21.15.97:39704) 2024-03-06T07:28:58.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.668+0000 7f7681400640 1 -- 172.21.15.97:0/3082676324 learned_addr learned my addr 172.21.15.97:0/3082676324 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:28:58.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.668+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144479228 0 0) 0x7f768415b090 con 0x7f76840b4680 2024-03-06T07:28:58.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.668+0000 7f7681e00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 0x7f76840b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.668+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7668003520 con 0x7f76840b4680 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.668+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589591846 0 0) 0x7f7668003520 con 0x7f76840b4680 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 msgr2=0x7f76840b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 0x7f76840b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 >> v1:172.21.15.135:6789/0 conn(0x7f768406ec00 legacy=0x7f76840ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840b2cf0 con 0x7f76840b4680 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7670001670 con 0x7f76840b4680 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 >> v1:172.21.15.97:6789/0 conn(0x7f76840b4680 legacy=0x7f768406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76680039e0 0x7f7668003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 0x7f7668004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7668004d40 0x7f76680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840b4360 con 0x7f76680039e0 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766800baa0 con 0x7f7668004d40 2024-03-06T07:28:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766800deb0 con 0x7f768415acb0 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681e00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76680039e0 0x7f7668003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681400640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 0x7f7668004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681e00640 1 -- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 msgr2=0x7f7668004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681e00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768415acb0 0x7f7668004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681e00640 1 -- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7668004d40 msgr2=0x7f76680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681e00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7668004d40 0x7f76680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.669+0000 7f7681e00640 1 -- 172.21.15.97:0/3082676324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668003520 con 0x7f76680039e0 2024-03-06T07:28:58.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.671+0000 7f7681e00640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76680039e0 0x7f7668003e00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7674009a90 tx=0x7f7668005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6435653aca3de171 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.671+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674039070 con 0x7f76680039e0 2024-03-06T07:28:58.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.671+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767402c680 con 0x7f76680039e0 2024-03-06T07:28:58.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.671+0000 7f7680a00640 1 -- 172.21.15.97:0/3082676324 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674009690 con 0x7f76680039e0 2024-03-06T07:28:58.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7689200640 1 -- 172.21.15.97:0/3082676324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76680039e0 msgr2=0x7f7668003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7689200640 1 --2- 172.21.15.97:0/3082676324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76680039e0 0x7f7668003e00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7674009a90 tx=0x7f7668005830 comp rx=0 tx=0).stop 2024-03-06T07:28:58.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7682800640 1 -- 172.21.15.97:0/3082676324 reap_dead start 2024-03-06T07:28:58.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7689200640 1 -- 172.21.15.97:0/3082676324 shutdown_connections 2024-03-06T07:28:58.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7689200640 1 -- 172.21.15.97:0/3082676324 >> 172.21.15.97:0/3082676324 conn(0x7f76840b1730 msgr2=0x7f76840b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7689200640 1 -- 172.21.15.97:0/3082676324 shutdown_connections 2024-03-06T07:28:58.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.672+0000 7f7689200640 1 -- 172.21.15.97:0/3082676324 wait complete. 2024-03-06T07:28:58.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.675+0000 7f7689200640 1 Processor -- start 2024-03-06T07:28:58.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.675+0000 7f7689200640 1 -- start start 2024-03-06T07:28:58.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7689200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f76680039e0 0x7f76841e5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7689200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76841f4fc0 0x7f768415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7689200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 0x7f76840cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7689200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840b4010 con 0x7f76840cb160 2024-03-06T07:28:58.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7689200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840b32b0 con 0x7f76841f4fc0 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7689200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7684072340 con 0x7f76680039e0 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7682800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 0x7f76840cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7682800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 0x7f76840cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53358/0 (socket says 172.21.15.97:53358) 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7682800640 1 -- 172.21.15.97:0/3618382837 learned_addr learned my addr 172.21.15.97:0/3618382837 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7681e00640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f76680039e0 0x7f76841e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.677+0000 7f7681400640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76841f4fc0 0x7f768415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7682800640 1 -- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f76680039e0 msgr2=0x7f76841e5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7682800640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f76680039e0 0x7f76841e5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7682800640 1 -- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76841f4fc0 msgr2=0x7f768415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7682800640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76841f4fc0 0x7f768415d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7682800640 1 -- 172.21.15.97:0/3618382837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76841f5580 con 0x7f76840cb160 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7681400640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76841f4fc0 0x7f768415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7682800640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 0x7f76840cb5e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f76840b27f0 tx=0x7f767c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c017070 con 0x7f76840cb160 2024-03-06T07:28:58.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c005d60 con 0x7f76840cb160 2024-03-06T07:28:58.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c00a7f0 con 0x7f76840cb160 2024-03-06T07:28:58.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76840ca3a0 con 0x7f76840cb160 2024-03-06T07:28:58.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.678+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76840ca960 con 0x7f76840cb160 2024-03-06T07:28:58.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.679+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768404f330 con 0x7f76840cb160 2024-03-06T07:28:58.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.680+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f767c00a990 con 0x7f76840cb160 2024-03-06T07:28:58.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.681+0000 7f767a000640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f766406b4d0 0x7f766406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.681+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f767c0892a0 con 0x7f76840cb160 2024-03-06T07:28:58.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.682+0000 7f7681e00640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f766406b4d0 0x7f766406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.682+0000 7f7681e00640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f766406b4d0 0x7f766406d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7674036000 tx=0x7f7674030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.686+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f767c058110 con 0x7f76840cb160 2024-03-06T07:28:58.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.929+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f768406ce60 con 0x7f76840cb160 2024-03-06T07:28:58.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.930+0000 7f767a000640 1 -- 172.21.15.97:0/3618382837 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f767c057a10 con 0x7f76840cb160 2024-03-06T07:28:58.932 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f766406b4d0 msgr2=0x7f766406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f766406b4d0 0x7f766406d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7674036000 tx=0x7f7674030000 comp rx=0 tx=0).stop 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 msgr2=0x7f76840cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 0x7f76840cb5e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f76840b27f0 tx=0x7f767c00cb90 comp rx=0 tx=0).stop 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 shutdown_connections 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f766406b4d0 0x7f766406d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76840cb160 0x7f76840cb5e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76841f4fc0 0x7f768415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 --2- 172.21.15.97:0/3618382837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f76680039e0 0x7f76841e5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 >> 172.21.15.97:0/3618382837 conn(0x7f76840b1730 msgr2=0x7f76840cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 shutdown_connections 2024-03-06T07:28:58.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:28:58.933+0000 7f7689200640 1 -- 172.21.15.97:0/3618382837 wait complete. 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:58.955 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:58.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:58.955 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:28:59.003 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:59.003 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:59.003 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:59.003 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.003 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42536sec preferred_lft 42536sec 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.004 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:28:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:59.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.006 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:59.134 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:28:59.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.138 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:28:59.185 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:28:59.185 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:59.185 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:59.185 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:59.185 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:59.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.186 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:28:59.186 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:28:59.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:59.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:59.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:59.186 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:28:59.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.350 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:28:59.398 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:59.398 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:59.398 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:59.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:28:59.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:59.485 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:59.529 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:59.530 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:59.530 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:59.619 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:28:59.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:59.744 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:59.744 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:59.777 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:59.777 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:59.777 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:28:59.777 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:28:59.777 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:28:59.779 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:28:59.780 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:59.780 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:59.780 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:59.780 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:59.783 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:59.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:59.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:59.864 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:59.923 INFO:teuthology.orchestra.run.smithi097.stdout:client4607 2024-03-06T07:28:59.924 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:28:59.924 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:59.965 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:28:59.965 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:28:59.965 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:28:59.965 DEBUG:teuthology.orchestra.run.smithi097:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:59.965 DEBUG:teuthology.orchestra.run.smithi097:> subdir = "delete_me" 2024-03-06T07:28:59.965 DEBUG:teuthology.orchestra.run.smithi097:> size = 1024 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> file_count = 1000 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(0, file_count): 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> f.write(size * '"'"'x'"'"') 2024-03-06T07:28:59.966 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:29:01.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:01.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.365+0000 7f0918200640 1 Processor -- start 2024-03-06T07:29:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.365+0000 7f0918200640 1 -- start start 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.366+0000 7f0918200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f091015f9d0 0x7f091015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.366+0000 7f0918200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091006cef0 con 0x7f0910070b10 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.366+0000 7f0918200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091006d1d0 con 0x7f0910144c00 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.366+0000 7f0918200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091006d4b0 con 0x7f091015f9d0 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.366+0000 7f0914e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f091015f9d0 0x7f091015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.366+0000 7f0916200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0910070b10 0x7f0910070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39706/0 (socket says 172.21.15.97:39706) 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.367+0000 7f0914e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f091015f9d0 0x7f091015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53362/0 (socket says 172.21.15.97:53362) 2024-03-06T07:29:01.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.367+0000 7f0916200640 1 -- 172.21.15.97:0/2766800431 learned_addr learned my addr 172.21.15.97:0/2766800431 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.367+0000 7f0914e00640 1 -- 172.21.15.97:0/2766800431 >> v1:172.21.15.135:6789/0 conn(0x7f0910144c00 legacy=0x7f091006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.367+0000 7f0914e00640 1 -- 172.21.15.97:0/2766800431 >> v1:172.21.15.97:6789/0 conn(0x7f0910070b10 legacy=0x7f0910070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.367+0000 7f0914e00640 1 -- 172.21.15.97:0/2766800431 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091006d790 con 0x7f091015f9d0 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.367+0000 7f090fe00640 1 -- 172.21.15.97:0/2766800431 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599748641 0 0) 0x7f091006cef0 con 0x7f0910070b10 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.368+0000 7f0914e00640 1 --2- 172.21.15.97:0/2766800431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015fe60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0097a0 tx=0x7f08fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc90d53b1efeeae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.368+0000 7f090fe00640 1 -- 172.21.15.97:0/2766800431 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc035020 con 0x7f091015f9d0 2024-03-06T07:29:01.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f090fe00640 1 -- 172.21.15.97:0/2766800431 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc031830 con 0x7f091015f9d0 2024-03-06T07:29:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f090fe00640 1 -- 172.21.15.97:0/2766800431 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc031c00 con 0x7f091015f9d0 2024-03-06T07:29:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 -- 172.21.15.97:0/2766800431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 msgr2=0x7f091015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 --2- 172.21.15.97:0/2766800431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015fe60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0097a0 tx=0x7f08fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 -- 172.21.15.97:0/2766800431 shutdown_connections 2024-03-06T07:29:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 --2- 172.21.15.97:0/2766800431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015fe60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 -- 172.21.15.97:0/2766800431 >> 172.21.15.97:0/2766800431 conn(0x7f09100b1730 msgr2=0x7f09100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 -- 172.21.15.97:0/2766800431 shutdown_connections 2024-03-06T07:29:01.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.369+0000 7f0918200640 1 -- 172.21.15.97:0/2766800431 wait complete. 2024-03-06T07:29:01.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.372+0000 7f0918200640 1 Processor -- start 2024-03-06T07:29:01.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.372+0000 7f0918200640 1 -- start start 2024-03-06T07:29:01.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0918200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0910070b10 0x7f0910155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0918200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0910144c00 0x7f09100a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0918200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0918200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091006ccd0 con 0x7f091015f9d0 2024-03-06T07:29:01.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0918200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091006cb50 con 0x7f0910144c00 2024-03-06T07:29:01.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0918200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091006df90 con 0x7f0910070b10 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0916200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0916200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53370/0 (socket says 172.21.15.97:53370) 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0916200640 1 -- 172.21.15.97:0/102752658 learned_addr learned my addr 172.21.15.97:0/102752658 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0914e00640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0910144c00 0x7f09100a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0916200640 1 -- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0910070b10 msgr2=0x7f0910155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.374+0000 7f0915800640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0910070b10 0x7f0910155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0916200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0910070b10 0x7f0910155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0916200640 1 -- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0910144c00 msgr2=0x7f09100a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0916200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0910144c00 0x7f09100a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0916200640 1 -- 172.21.15.97:0/102752658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09100a9b10 con 0x7f091015f9d0 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0914e00640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0910144c00 0x7f09100a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0916200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015ef80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f090800cad0 tx=0x7f090800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090800ac30 con 0x7f091015f9d0 2024-03-06T07:29:01.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090800adf0 con 0x7f091015f9d0 2024-03-06T07:29:01.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908012640 con 0x7f091015f9d0 2024-03-06T07:29:01.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09100a9df0 con 0x7f091015f9d0 2024-03-06T07:29:01.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.375+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09100aa3b0 con 0x7f091015f9d0 2024-03-06T07:29:01.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.376+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091004f330 con 0x7f091015f9d0 2024-03-06T07:29:01.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.377+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09080127e0 con 0x7f091015f9d0 2024-03-06T07:29:01.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.378+0000 7f090d600640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f08f006b4d0 0x7f08f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.378+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f090808a8b0 con 0x7f091015f9d0 2024-03-06T07:29:01.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.379+0000 7f0915800640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f08f006b4d0 0x7f08f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.379+0000 7f0915800640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f08f006b4d0 0x7f08f006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0904003d90 tx=0x7f0904009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.383+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0908059720 con 0x7f091015f9d0 2024-03-06T07:29:01.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.616+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0910072890 con 0x7f091015f9d0 2024-03-06T07:29:01.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.617+0000 7f090d600640 1 -- 172.21.15.97:0/102752658 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f0908059020 con 0x7f091015f9d0 2024-03-06T07:29:01.619 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:01.619 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:01.619 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:01.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f08f006b4d0 msgr2=0x7f08f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f08f006b4d0 0x7f08f006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0904003d90 tx=0x7f0904009040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 msgr2=0x7f091015ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015ef80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f090800cad0 tx=0x7f090800cf90 comp rx=0 tx=0).stop 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 shutdown_connections 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f08f006b4d0 0x7f08f006d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f091015f9d0 0x7f091015ef80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0910144c00 0x7f09100a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 --2- 172.21.15.97:0/102752658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0910070b10 0x7f0910155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 >> 172.21.15.97:0/102752658 conn(0x7f09100b1730 msgr2=0x7f091006e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 shutdown_connections 2024-03-06T07:29:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:01.620+0000 7f0918200640 1 -- 172.21.15.97:0/102752658 wait complete. 2024-03-06T07:29:01.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:02.225 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:29:02.225 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:02.225 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:29:02.226 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:29:02.226 DEBUG:teuthology.orchestra.run.smithi097:> import stat 2024-03-06T07:29:02.226 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:29:02.226 DEBUG:teuthology.orchestra.run.smithi097:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:29:02.226 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:29:02.357 INFO:teuthology.orchestra.run.smithi097.stdout:1099511627776 2024-03-06T07:29:02.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:02.373 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:29:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.508+0000 7fb08d84ff40 1 Processor -- start 2024-03-06T07:29:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.508+0000 7fb08d84ff40 1 -- start start 2024-03-06T07:29:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08d84ff40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14bc1990 0x563f14c950b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08d84ff40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c80b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08d84ff40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 0x563f14bef090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08d84ff40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f14bc0d10 con 0x563f14c7f9b0 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08d84ff40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f14b67fd0 con 0x563f14bc1d70 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08d84ff40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f14bc0fb0 con 0x563f14bc1990 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.509+0000 7fb08ae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 0x563f14bef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 0x563f14bef090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53382/0 (socket says 172.21.15.97:53382) 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 -- 172.21.15.97:0/551459802 learned_addr learned my addr 172.21.15.97:0/551459802 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 -- 172.21.15.97:0/551459802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14bc1990 msgr2=0x563f14c950b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14bc1990 0x563f14c950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 -- 172.21.15.97:0/551459802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 msgr2=0x563f14c80b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c80b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 -- 172.21.15.97:0/551459802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f14c0b510 con 0x563f14c7f9b0 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb08ae00640 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 0x563f14bef090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb08400b600 tx=0x7fb08400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478e3ffcf9de4621 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb083e00640 1 -- 172.21.15.97:0/551459802 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084014030 con 0x563f14c7f9b0 2024-03-06T07:29:02.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb083e00640 1 -- 172.21.15.97:0/551459802 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084010620 con 0x563f14c7f9b0 2024-03-06T07:29:02.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.510+0000 7fb083e00640 1 -- 172.21.15.97:0/551459802 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0840109f0 con 0x563f14c7f9b0 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 -- 172.21.15.97:0/551459802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 msgr2=0x563f14bef090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 0x563f14bef090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb08400b600 tx=0x7fb08400bac0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 -- 172.21.15.97:0/551459802 shutdown_connections 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14c7f9b0 0x563f14bef090 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c80b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/551459802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14bc1990 0x563f14c950b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 -- 172.21.15.97:0/551459802 >> 172.21.15.97:0/551459802 conn(0x563f14bf0c50 msgr2=0x563f14bf3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 -- 172.21.15.97:0/551459802 shutdown_connections 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.511+0000 7fb08d84ff40 1 -- 172.21.15.97:0/551459802 wait complete. 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.512+0000 7fb08d84ff40 1 Processor -- start 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.512+0000 7fb08d84ff40 1 -- start start 2024-03-06T07:29:02.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08d84ff40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14bc1990 0x563f14c97f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08d84ff40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c94cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08d84ff40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14c7f9b0 0x563f14c72070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08d84ff40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f14c0b780 con 0x563f14bc1990 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08d84ff40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f14c955f0 con 0x563f14bc1d70 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08d84ff40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f14befd60 con 0x563f14c7f9b0 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14bc1990 0x563f14c97f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14bc1990 0x563f14c97f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53390/0 (socket says 172.21.15.97:53390) 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08a400640 1 -- 172.21.15.97:0/2555220704 learned_addr learned my addr 172.21.15.97:0/2555220704 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb089a00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c94cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08ae00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14c7f9b0 0x563f14c72070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08a400640 1 -- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14c7f9b0 msgr2=0x563f14c72070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.513+0000 7fb08a400640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x563f14c7f9b0 0x563f14c72070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb08a400640 1 -- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 msgr2=0x563f14c94cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb08a400640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c94cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb08a400640 1 -- 172.21.15.97:0/2555220704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f14c7d9e0 con 0x563f14bc1990 2024-03-06T07:29:02.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb089a00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563f14bc1d70 0x563f14c94cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb08a400640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14bc1990 0x563f14c97f00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb07c01cf50 tx=0x7fb07c0058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb082a00640 1 -- 172.21.15.97:0/2555220704 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c057040 con 0x563f14bc1990 2024-03-06T07:29:02.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb082a00640 1 -- 172.21.15.97:0/2555220704 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c052ab0 con 0x563f14bc1990 2024-03-06T07:29:02.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563f14c79e10 con 0x563f14bc1990 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb082a00640 1 -- 172.21.15.97:0/2555220704 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c05d470 con 0x563f14bc1990 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.514+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f14c7d670 con 0x563f14bc1990 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.516+0000 7fb082a00640 1 -- 172.21.15.97:0/2555220704 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb07c05d610 con 0x563f14bc1990 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.517+0000 7fb082a00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb068069ee0 0x7fb06806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.517+0000 7fb082a00640 1 -- 172.21.15.97:0/2555220704 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb07c0cae40 con 0x563f14bc1990 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.517+0000 7fb089a00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb068069ee0 0x7fb06806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.517+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x563f14c951f0 0x563f14c725b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.517+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v9 -- 0x563f14c72af0 con 0x563f14c951f0 2024-03-06T07:29:02.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.517+0000 7fb08ae00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x563f14c951f0 0x563f14c725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.518+0000 7fb089a00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb068069ee0 0x7fb06806c3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x563f14c0e680 tx=0x7fb0740055d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.518+0000 7fb08ae00640 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x563f14c951f0 0x563f14c725b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.518+0000 7fb08ae00640 1 -- 172.21.15.97:0/2555220704 <== osd.3 v2:172.21.15.97:6800/3169996968 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7fb084010a70 con 0x563f14c951f0 2024-03-06T07:29:02.521 INFO:teuthology.orchestra.run.smithi097.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:29:02.000000+0000, size 0 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.518+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x563f14c951f0 msgr2=0x563f14c725b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.518+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x563f14c951f0 0x563f14c725b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb068069ee0 msgr2=0x7fb06806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb068069ee0 0x7fb06806c3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x563f14c0e680 tx=0x7fb0740055d0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14bc1990 msgr2=0x563f14c97f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 --2- 172.21.15.97:0/2555220704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563f14bc1990 0x563f14c97f00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb07c01cf50 tx=0x7fb07c0058f0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08ae00640 1 -- 172.21.15.97:0/2555220704 reap_dead start 2024-03-06T07:29:02.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 shutdown_connections 2024-03-06T07:29:02.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 >> 172.21.15.97:0/2555220704 conn(0x563f14bf0c50 msgr2=0x563f14bef090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 shutdown_connections 2024-03-06T07:29:02.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:02.519+0000 7fb08d84ff40 1 -- 172.21.15.97:0/2555220704 wait complete. 2024-03-06T07:29:02.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.525 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:29:04.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:05.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.152+0000 7f4dbec00640 1 Processor -- start 2024-03-06T07:29:05.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.152+0000 7f4dbec00640 1 -- start start 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.154+0000 7f4dbec00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4db8144c00 0x7f4db806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.154+0000 7f4dbec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db815f9d0 0x7f4db815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.154+0000 7f4dbec00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db806ee70 con 0x7f4db80ce0a0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.154+0000 7f4dbec00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db8160370 con 0x7f4db8144c00 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.154+0000 7f4dbec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806cef0 con 0x7f4db815f9d0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4db8144c00 0x7f4db806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4db8144c00 0x7f4db806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49322/0 (socket says 172.21.15.97:49322) 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 -- 172.21.15.97:0/2990023826 learned_addr learned my addr 172.21.15.97:0/2990023826 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 -- 172.21.15.97:0/2990023826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db815f9d0 msgr2=0x7f4db815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7400640 1 --2- 172.21.15.97:0/2990023826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db815f9d0 0x7f4db815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 --2- 172.21.15.97:0/2990023826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db815f9d0 0x7f4db815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 -- 172.21.15.97:0/2990023826 >> v1:172.21.15.135:6789/0 conn(0x7f4db80ce0a0 legacy=0x7f4db80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.155+0000 7f4db7e00640 1 -- 172.21.15.97:0/2990023826 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db806d1d0 con 0x7f4db8144c00 2024-03-06T07:29:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.156+0000 7f4db7e00640 1 --2- 172.21.15.97:0/2990023826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db8144c00 0x7f4db806e930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009870 tx=0x7f4da802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c30de77c834691 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.157+0000 7f4db6a00640 1 -- 172.21.15.97:0/2990023826 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8035020 con 0x7f4db8144c00 2024-03-06T07:29:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.157+0000 7f4db6a00640 1 -- 172.21.15.97:0/2990023826 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da8031640 con 0x7f4db8144c00 2024-03-06T07:29:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.157+0000 7f4db6a00640 1 -- 172.21.15.97:0/2990023826 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da802d040 con 0x7f4db8144c00 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.157+0000 7f4dbec00640 1 -- 172.21.15.97:0/2990023826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db8144c00 msgr2=0x7f4db806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.157+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2990023826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db8144c00 0x7f4db806e930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009870 tx=0x7f4da802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.158+0000 7f4dbec00640 1 -- 172.21.15.97:0/2990023826 shutdown_connections 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.158+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2990023826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db815f9d0 0x7f4db815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.158+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2990023826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db8144c00 0x7f4db806e930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.158+0000 7f4dbec00640 1 -- 172.21.15.97:0/2990023826 >> 172.21.15.97:0/2990023826 conn(0x7f4db80b1730 msgr2=0x7f4db80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.158+0000 7f4dbec00640 1 -- 172.21.15.97:0/2990023826 shutdown_connections 2024-03-06T07:29:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.158+0000 7f4dbec00640 1 -- 172.21.15.97:0/2990023826 wait complete. 2024-03-06T07:29:05.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.159+0000 7f4dbec00640 1 Processor -- start 2024-03-06T07:29:05.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.159+0000 7f4dbec00640 1 -- start start 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.160+0000 7f4dbec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db80ce0a0 0x7f4db8137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.160+0000 7f4dbec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4db8144c00 0x7f4db815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.160+0000 7f4dbec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 0x7f4db815ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.160+0000 7f4dbec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8160730 con 0x7f4db815f9d0 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.160+0000 7f4dbec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806d790 con 0x7f4db8144c00 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.160+0000 7f4dbec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806cb50 con 0x7f4db80ce0a0 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 0x7f4db815ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 0x7f4db815ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49326/0 (socket says 172.21.15.97:49326) 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 -- 172.21.15.97:0/2370601668 learned_addr learned my addr 172.21.15.97:0/2370601668 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4db7e00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db80ce0a0 0x7f4db8137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4db7400640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4db8144c00 0x7f4db815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 -- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db80ce0a0 msgr2=0x7f4db8137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db80ce0a0 0x7f4db8137c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 -- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4db8144c00 msgr2=0x7f4db815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4db8144c00 0x7f4db815b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 -- 172.21.15.97:0/2370601668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db815d030 con 0x7f4db815f9d0 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4db7e00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db80ce0a0 0x7f4db8137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4db7400640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4db8144c00 0x7f4db815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.161+0000 7f4dbcc00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 0x7f4db815ebe0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4db000b970 tx=0x7f4db000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.162+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0019070 con 0x7f4db815f9d0 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.162+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db000cae0 con 0x7f4db815f9d0 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.162+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db8139cc0 con 0x7f4db815f9d0 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.162+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0014440 con 0x7f4db815f9d0 2024-03-06T07:29:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.162+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db813a280 con 0x7f4db815f9d0 2024-03-06T07:29:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.163+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db804f330 con 0x7f4db815f9d0 2024-03-06T07:29:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.164+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4db001c070 con 0x7f4db815f9d0 2024-03-06T07:29:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.165+0000 7f4dafe00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9806b500 0x7f4d9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.165+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4db0089df0 con 0x7f4db815f9d0 2024-03-06T07:29:05.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.165+0000 7f4db7e00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9806b500 0x7f4d9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.166+0000 7f4db7e00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9806b500 0x7f4d9806d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4da8036000 tx=0x7f4da8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.170+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4db815d030 con 0x7f4db815f9d0 2024-03-06T07:29:05.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.603+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4db8145660 con 0x7f4db815f9d0 2024-03-06T07:29:05.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.604+0000 7f4dafe00640 1 -- 172.21.15.97:0/2370601668 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f4db0058bd0 con 0x7f4db815f9d0 2024-03-06T07:29:05.607 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:05.608 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:05.609 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:05.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.609+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9806b500 msgr2=0x7f4d9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.609+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9806b500 0x7f4d9806d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4da8036000 tx=0x7f4da8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:05.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.609+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 msgr2=0x7f4db815ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.609+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 0x7f4db815ebe0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4db000b970 tx=0x7f4db000be30 comp rx=0 tx=0).stop 2024-03-06T07:29:05.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.611+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 shutdown_connections 2024-03-06T07:29:05.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.611+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9806b500 0x7f4d9806d9c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.611+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4db815f9d0 0x7f4db815ebe0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.611+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4db8144c00 0x7f4db815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.611+0000 7f4dbec00640 1 --2- 172.21.15.97:0/2370601668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4db80ce0a0 0x7f4db8137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.611+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 >> 172.21.15.97:0/2370601668 conn(0x7f4db80b1730 msgr2=0x7f4db8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.612+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 shutdown_connections 2024-03-06T07:29:05.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:05.612+0000 7f4dbec00640 1 -- 172.21.15.97:0/2370601668 wait complete. 2024-03-06T07:29:05.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:06.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:06.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:08.280 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:29:08.281 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:08.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.454+0000 7fccaa000640 1 Processor -- start 2024-03-06T07:29:08.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.454+0000 7fccaa000640 1 -- start start 2024-03-06T07:29:08.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fccaa000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcca40cb1f0 0x7fcca406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fccaa000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca40cb680 con 0x7fcca41458a0 2024-03-06T07:29:08.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fccaa000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca406b900 con 0x7fcca406b500 2024-03-06T07:29:08.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fccaa000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca406fac0 con 0x7fcca40cb1f0 2024-03-06T07:29:08.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fcca3400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcca41458a0 0x7fcca40725b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58064/0 (socket says 172.21.15.97:58064) 2024-03-06T07:29:08.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fcca3400640 1 -- 172.21.15.97:0/2020824562 learned_addr learned my addr 172.21.15.97:0/2020824562 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:08.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.456+0000 7fcc9be00640 1 --2- 172.21.15.97:0/2020824562 >> v2:172.21.15.135:3300/0 conn(0x7fcca40cb1f0 0x7fcca406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963112780 0 0) 0x7fcca40cb680 con 0x7fcca41458a0 2024-03-06T07:29:08.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc88003520 con 0x7fcca41458a0 2024-03-06T07:29:08.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497837688 0 0) 0x7fcc88003520 con 0x7fcca41458a0 2024-03-06T07:29:08.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 >> v2:172.21.15.135:3300/0 conn(0x7fcca40cb1f0 msgr2=0x7fcca406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 --2- 172.21.15.97:0/2020824562 >> v2:172.21.15.135:3300/0 conn(0x7fcca40cb1f0 0x7fcca406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 >> v1:172.21.15.135:6789/0 conn(0x7fcca406b500 legacy=0x7fcca406df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40d0140 con 0x7fcca41458a0 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fcc94001670 con 0x7fcca41458a0 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 >> v1:172.21.15.97:6789/0 conn(0x7fcca41458a0 legacy=0x7fcca40725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880039b0 0x7fcc88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc88004800 0x7fcc88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc880051c0 0x7fcc8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca406fac0 con 0x7fcc880051c0 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8800c160 con 0x7fcc88004800 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8800c360 con 0x7fcc880039b0 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.457+0000 7fcca3e00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc880051c0 0x7fcc8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.458+0000 7fcca3400640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880039b0 0x7fcc88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.458+0000 7fcca3e00640 1 -- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880039b0 msgr2=0x7fcc88003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.458+0000 7fcca3e00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880039b0 0x7fcc88003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.458+0000 7fcca3e00640 1 -- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc88004800 msgr2=0x7fcc88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.458+0000 7fcca3e00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc88004800 0x7fcc88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.458+0000 7fcca3e00640 1 -- 172.21.15.97:0/2020824562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88003520 con 0x7fcc880051c0 2024-03-06T07:29:08.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.459+0000 7fcca3e00640 1 -- 172.21.15.97:0/2020824562 reap_dead start 2024-03-06T07:29:08.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.460+0000 7fcca3e00640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc880051c0 0x7fcc8800b9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c001e50 tx=0x7fcc88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe386438579abf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.460+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c001a70 con 0x7fcc880051c0 2024-03-06T07:29:08.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.460+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c003bb0 con 0x7fcc880051c0 2024-03-06T07:29:08.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.460+0000 7fcca2a00640 1 -- 172.21.15.97:0/2020824562 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c0050d0 con 0x7fcc880051c0 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.460+0000 7fccaa000640 1 -- 172.21.15.97:0/2020824562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc880051c0 msgr2=0x7fcc8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.460+0000 7fccaa000640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc880051c0 0x7fcc8800b9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c001e50 tx=0x7fcc88005830 comp rx=0 tx=0).stop 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.461+0000 7fccaa000640 1 -- 172.21.15.97:0/2020824562 shutdown_connections 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.461+0000 7fccaa000640 1 --2- 172.21.15.97:0/2020824562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc880051c0 0x7fcc8800b9c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.461+0000 7fccaa000640 1 -- 172.21.15.97:0/2020824562 >> 172.21.15.97:0/2020824562 conn(0x7fcca40d0ba0 msgr2=0x7fcca40d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.461+0000 7fccaa000640 1 -- 172.21.15.97:0/2020824562 shutdown_connections 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.461+0000 7fccaa000640 1 -- 172.21.15.97:0/2020824562 wait complete. 2024-03-06T07:29:08.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.462+0000 7fccaa000640 1 Processor -- start 2024-03-06T07:29:08.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.463+0000 7fccaa000640 1 -- start start 2024-03-06T07:29:08.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fccaa000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880051c0 0x7fcca41f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fccaa000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcca40c7e70 0x7fcca4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fccaa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 0x7fcca4142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fccaa000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca406f7e0 con 0x7fcca4071fe0 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fccaa000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d09a0 con 0x7fcca40c7e70 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fccaa000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d03e0 con 0x7fcc880051c0 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 0x7fcca4142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 0x7fcca4142e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49350/0 (socket says 172.21.15.97:49350) 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 -- 172.21.15.97:0/1851147638 learned_addr learned my addr 172.21.15.97:0/1851147638 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcc9be00640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcca40c7e70 0x7fcca4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3400640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880051c0 0x7fcca41f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 -- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880051c0 msgr2=0x7fcca41f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880051c0 0x7fcca41f7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 -- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcca40c7e70 msgr2=0x7fcca4071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcca40c7e70 0x7fcca4071aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcca3e00640 1 -- 172.21.15.97:0/1851147638 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40c9a00 con 0x7fcca4071fe0 2024-03-06T07:29:08.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.464+0000 7fcc9be00640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcca40c7e70 0x7fcca4071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.465+0000 7fcca3e00640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 0x7fcca4142e90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005040 tx=0x7fcc9c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.465+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c003970 con 0x7fcca4071fe0 2024-03-06T07:29:08.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.465+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c03d020 con 0x7fcca4071fe0 2024-03-06T07:29:08.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.465+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca40c9ca0 con 0x7fcca4071fe0 2024-03-06T07:29:08.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.465+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c007030 con 0x7fcca4071fe0 2024-03-06T07:29:08.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.465+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca40ca260 con 0x7fcca4071fe0 2024-03-06T07:29:08.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.467+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca404f330 con 0x7fcca4071fe0 2024-03-06T07:29:08.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.467+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc9c0071d0 con 0x7fcca4071fe0 2024-03-06T07:29:08.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.468+0000 7fcc9b400640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.468+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc9c0abfc0 con 0x7fcca4071fe0 2024-03-06T07:29:08.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.468+0000 7fcca3400640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.469+0000 7fcca3400640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc940017b0 tx=0x7fcc94000f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.477+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc9c047030 con 0x7fcca4071fe0 2024-03-06T07:29:08.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.750+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcca415bf60 con 0x7fcca4071fe0 2024-03-06T07:29:08.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.751+0000 7fcc9b400640 1 -- 172.21.15.97:0/1851147638 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fcc9c033030 con 0x7fcca4071fe0 2024-03-06T07:29:08.754 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:08.754 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:08.754 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.754+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc8006b4d0 msgr2=0x7fcc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.754+0000 7fccaa000640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc940017b0 tx=0x7fcc94000f10 comp rx=0 tx=0).stop 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.754+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 msgr2=0x7fcca4142e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.754+0000 7fccaa000640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 0x7fcca4142e90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005040 tx=0x7fcc9c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 shutdown_connections 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4071fe0 0x7fcca4142e90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcca40c7e70 0x7fcca4071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 --2- 172.21.15.97:0/1851147638 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880051c0 0x7fcca41f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 >> 172.21.15.97:0/1851147638 conn(0x7fcca40d0ba0 msgr2=0x7fcca40ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 shutdown_connections 2024-03-06T07:29:08.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:08.755+0000 7fccaa000640 1 -- 172.21.15.97:0/1851147638 wait complete. 2024-03-06T07:29:08.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:09.058 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:29:09.059 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:29:09.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:09.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.231+0000 7f604b200640 1 Processor -- start 2024-03-06T07:29:09.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.231+0000 7f604b200640 1 -- start start 2024-03-06T07:29:09.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.232+0000 7f604b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c0d0c00 0x7f604c0d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.232+0000 7f604b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f604c0b2e50 0x7f604c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.232+0000 7f604b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0b5240 con 0x7f604c15e200 2024-03-06T07:29:09.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.232+0000 7f604b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c0d2ec0 con 0x7f604c0b2e50 2024-03-06T07:29:09.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.232+0000 7f604b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0cf4d0 con 0x7f604c0d0c00 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f604c0b2e50 0x7f604c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f604c0b2e50 0x7f604c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49364/0 (socket says 172.21.15.97:49364) 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 -- 172.21.15.97:0/48010101 learned_addr learned my addr 172.21.15.97:0/48010101 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 -- 172.21.15.97:0/48010101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c0d0c00 msgr2=0x7f604c0d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f6043e00640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c0d0c00 0x7f604c0d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c0d0c00 0x7f604c0d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 -- 172.21.15.97:0/48010101 >> v1:172.21.15.97:6789/0 conn(0x7f604c15e200 legacy=0x7f604c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f604a800640 1 -- 172.21.15.97:0/48010101 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c0b5500 con 0x7f604c0b2e50 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f6043e00640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c0d0c00 0x7f604c0d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.233+0000 7f6049400640 1 -- 172.21.15.97:0/48010101 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666039437 0 0) 0x7f604c0b5240 con 0x7f604c15e200 2024-03-06T07:29:09.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.234+0000 7f604a800640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0b2e50 0x7f604c0b3640 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6044009e50 tx=0x7f604402bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc511eb132427c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.235+0000 7f6049400640 1 -- 172.21.15.97:0/48010101 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604402cb80 con 0x7f604c0b2e50 2024-03-06T07:29:09.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.235+0000 7f6049400640 1 -- 172.21.15.97:0/48010101 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604402cd20 con 0x7f604c0b2e50 2024-03-06T07:29:09.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.235+0000 7f6049400640 1 -- 172.21.15.97:0/48010101 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044034610 con 0x7f604c0b2e50 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.235+0000 7f604b200640 1 -- 172.21.15.97:0/48010101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0b2e50 msgr2=0x7f604c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.235+0000 7f604b200640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0b2e50 0x7f604c0b3640 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6044009e50 tx=0x7f604402bfe0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.236+0000 7f604b200640 1 -- 172.21.15.97:0/48010101 shutdown_connections 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.236+0000 7f604b200640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0b2e50 0x7f604c0b3640 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.236+0000 7f604b200640 1 --2- 172.21.15.97:0/48010101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c0d0c00 0x7f604c0d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.236+0000 7f604b200640 1 -- 172.21.15.97:0/48010101 >> 172.21.15.97:0/48010101 conn(0x7f604c0d3d30 msgr2=0x7f604c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.236+0000 7f604b200640 1 -- 172.21.15.97:0/48010101 shutdown_connections 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.236+0000 7f604b200640 1 -- 172.21.15.97:0/48010101 wait complete. 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.237+0000 7f604b200640 1 Processor -- start 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.238+0000 7f604b200640 1 -- start start 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.238+0000 7f604b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f604c0b2e50 0x7f604c0b7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.238+0000 7f604b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 0x7f604c0cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f604b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c15e200 0x7f604c14f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f604b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0cfd60 con 0x7f604c0d0c00 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f604b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0cfa80 con 0x7f604c0b2e50 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f604b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c15e5e0 con 0x7f604c15e200 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 0x7f604c0cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 0x7f604c0cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49370/0 (socket says 172.21.15.97:49370) 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 -- 172.21.15.97:0/598434689 learned_addr learned my addr 172.21.15.97:0/598434689 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f604a800640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c15e200 0x7f604c14f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6049e00640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f604c0b2e50 0x7f604c0b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 -- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c15e200 msgr2=0x7f604c14f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c15e200 0x7f604c14f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 -- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f604c0b2e50 msgr2=0x7f604c0b7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f604c0b2e50 0x7f604c0b7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.239+0000 7f6043e00640 1 -- 172.21.15.97:0/598434689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c0cc8f0 con 0x7f604c0d0c00 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.240+0000 7f6043e00640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 0x7f604c0cc3b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f603400c7f0 tx=0x7f603400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.240+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034004230 con 0x7f604c0d0c00 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.240+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60340043d0 con 0x7f604c0d0c00 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.240+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034007a70 con 0x7f604c0d0c00 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.240+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c0ccbb0 con 0x7f604c0d0c00 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.240+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c0cd170 con 0x7f604c0d0c00 2024-03-06T07:29:09.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.241+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c04fb80 con 0x7f604c0d0c00 2024-03-06T07:29:09.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.242+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6034007c10 con 0x7f604c0d0c00 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.243+0000 7f6042000640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.243+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f603408a290 con 0x7f604c0d0c00 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.243+0000 7f6049e00640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.244+0000 7f6049e00640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f603c000f60 tx=0x7f603c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.248+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f604c04fb80 con 0x7f604c0d0c00 2024-03-06T07:29:09.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.483+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f604c0d3a90 con 0x7f604c0d0c00 2024-03-06T07:29:09.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.484+0000 7f6042000640 1 -- 172.21.15.97:0/598434689 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f6034015020 con 0x7f604c0d0c00 2024-03-06T07:29:09.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:09.486 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:09.487 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.486+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f602806b4d0 msgr2=0x7f602806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.486+0000 7f604b200640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f603c000f60 tx=0x7f603c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.486+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 msgr2=0x7f604c0cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.486+0000 7f604b200640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 0x7f604c0cc3b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f603400c7f0 tx=0x7f603400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 shutdown_connections 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f604c15e200 0x7f604c14f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f604c0d0c00 0x7f604c0cc3b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 --2- 172.21.15.97:0/598434689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f604c0b2e50 0x7f604c0b7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 >> 172.21.15.97:0/598434689 conn(0x7f604c0d3d30 msgr2=0x7f604c0d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 shutdown_connections 2024-03-06T07:29:09.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.487+0000 7f604b200640 1 -- 172.21.15.97:0/598434689 wait complete. 2024-03-06T07:29:09.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:29:09.714 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:29:09.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:09.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.907+0000 7f3ee6200640 1 Processor -- start 2024-03-06T07:29:09.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.908+0000 7f3ee6200640 1 -- start start 2024-03-06T07:29:09.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3ee6200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3ee00b3b40 0x7f3ee0071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3ee6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 0x7f3ee006f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3ee6200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee00cfe30 con 0x7f3ee0144660 2024-03-06T07:29:09.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3ee6200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee0072590 con 0x7f3ee00b3b40 2024-03-06T07:29:09.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3ee6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00cf880 con 0x7f3ee0072140 2024-03-06T07:29:09.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3edf400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3ee0144660 0x7f3ee00cf170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58080/0 (socket says 172.21.15.97:58080) 2024-03-06T07:29:09.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3edf400640 1 -- 172.21.15.97:0/947175011 learned_addr learned my addr 172.21.15.97:0/947175011 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:09.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3edfe00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 0x7f3ee006f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/947175011 >> v2:172.21.15.135:3300/0 conn(0x7f3ee00b3b40 0x7f3ee0071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.909+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338608179 0 0) 0x7f3ee00cfe30 con 0x7f3ee0144660 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec4003520 con 0x7f3ee0144660 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209602616 0 0) 0x7f3ec4003520 con 0x7f3ee0144660 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 msgr2=0x7f3ee006f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 0x7f3ee006f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 >> v2:172.21.15.135:3300/0 conn(0x7f3ee00b3b40 msgr2=0x7f3ee0071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 --2- 172.21.15.97:0/947175011 >> v2:172.21.15.135:3300/0 conn(0x7f3ee00b3b40 0x7f3ee0071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00cfb50 con 0x7f3ee0144660 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3ed0001670 con 0x7f3ee0144660 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 >> v1:172.21.15.97:6789/0 conn(0x7f3ee0144660 legacy=0x7f3ee00cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 0x7f3ec40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ec4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ec4004d40 0x7f3ec40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0072590 con 0x7f3ec4004030 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00cf880 con 0x7f3ec4004d40 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec400bb20 con 0x7f3ee0072140 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.910+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ec4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3edf400640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 0x7f3ec40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3edfe00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ec4004d40 0x7f3ec40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3ed6a00640 1 -- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 msgr2=0x7f3ec40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee0072140 0x7f3ec40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3ed6a00640 1 -- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ec4004d40 msgr2=0x7f3ec40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ec4004d40 0x7f3ec40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.911+0000 7f3ed6a00640 1 -- 172.21.15.97:0/947175011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4003520 con 0x7f3ec4004030 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.912+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ec4004800 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800ba30 tx=0x7f3ec4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aa1a389a97c9d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.912+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8036020 con 0x7f3ec4004030 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.912+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8032680 con 0x7f3ec4004030 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3edea00640 1 -- 172.21.15.97:0/947175011 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec800b600 con 0x7f3ec4004030 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3ee6200640 1 -- 172.21.15.97:0/947175011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 msgr2=0x7f3ec4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3ee6200640 1 --2- 172.21.15.97:0/947175011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ec4004800 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800ba30 tx=0x7f3ec4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3edfe00640 1 -- 172.21.15.97:0/947175011 reap_dead start 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3ee6200640 1 -- 172.21.15.97:0/947175011 shutdown_connections 2024-03-06T07:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3ee6200640 1 -- 172.21.15.97:0/947175011 >> 172.21.15.97:0/947175011 conn(0x7f3ee00d0b90 msgr2=0x7f3ee00d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3ee6200640 1 -- 172.21.15.97:0/947175011 shutdown_connections 2024-03-06T07:29:09.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.913+0000 7f3ee6200640 1 -- 172.21.15.97:0/947175011 wait complete. 2024-03-06T07:29:09.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.915+0000 7f3ee6200640 1 Processor -- start 2024-03-06T07:29:09.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.915+0000 7f3ee6200640 1 -- start start 2024-03-06T07:29:09.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3ee6200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ee015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3ee6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee015deb0 0x7f3ee0156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3ee6200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee013bad0 0x7f3ee0158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3ee6200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee006f350 con 0x7f3ec4004030 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3ee6200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee006f070 con 0x7f3ee013bad0 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3ee6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00cc550 con 0x7f3ee015deb0 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3edf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ee015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3edf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ee015b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49386/0 (socket says 172.21.15.97:49386) 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.916+0000 7f3edf400640 1 -- 172.21.15.97:0/3973979692 learned_addr learned my addr 172.21.15.97:0/3973979692 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3edf400640 1 -- 172.21.15.97:0/3973979692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee015deb0 msgr2=0x7f3ee0156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3edf400640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee015deb0 0x7f3ee0156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3edf400640 1 -- 172.21.15.97:0/3973979692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee013bad0 msgr2=0x7f3ee0158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3edf400640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee013bad0 0x7f3ee0158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3edf400640 1 -- 172.21.15.97:0/3973979692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee013ec30 con 0x7f3ec4004030 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3edf400640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ee015b210 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000e700 tx=0x7f3ed000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000ee10 con 0x7f3ec4004030 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed0018070 con 0x7f3ec4004030 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000ca90 con 0x7f3ec4004030 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee015d6b0 con 0x7f3ec4004030 2024-03-06T07:29:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.917+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee0156e60 con 0x7f3ec4004030 2024-03-06T07:29:09.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.919+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee0072a10 con 0x7f3ec4004030 2024-03-06T07:29:09.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.919+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ed000cc30 con 0x7f3ec4004030 2024-03-06T07:29:09.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.920+0000 7f3ed7e00640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ebc06b4d0 0x7f3ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.920+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ed008a0d0 con 0x7f3ec4004030 2024-03-06T07:29:09.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.920+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ebc06b4d0 0x7f3ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.921+0000 7f3ed6a00640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ebc06b4d0 0x7f3ebc06d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800ba00 tx=0x7f3ec8032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:09.929+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ee0072a10 con 0x7f3ec4004030 2024-03-06T07:29:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.151+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ee004f330 con 0x7f3ec4004030 2024-03-06T07:29:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.152+0000 7f3ed7e00640 1 -- 172.21.15.97:0/3973979692 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f3ed0058f40 con 0x7f3ec4004030 2024-03-06T07:29:10.154 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:10.154 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:10.155 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ebc06b4d0 msgr2=0x7f3ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ebc06b4d0 0x7f3ebc06d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800ba00 tx=0x7f3ec8032000 comp rx=0 tx=0).stop 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 msgr2=0x7f3ee015b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ee015b210 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000e700 tx=0x7f3ed000ebc0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 shutdown_connections 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ebc06b4d0 0x7f3ebc06d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee013bad0 0x7f3ee0158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ee015deb0 0x7f3ee0156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 --2- 172.21.15.97:0/3973979692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ec4004030 0x7f3ee015b210 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.154+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 >> 172.21.15.97:0/3973979692 conn(0x7f3ee00d0b90 msgr2=0x7f3ee006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.155+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 shutdown_connections 2024-03-06T07:29:10.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.155+0000 7f3ee6200640 1 -- 172.21.15.97:0/3973979692 wait complete. 2024-03-06T07:29:10.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:29:10.386 INFO:teuthology.orchestra.run.smithi097.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:29:10.386 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:29:10.386 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:29:10.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.483+0000 7fd3d0c3ff40 1 Processor -- start 2024-03-06T07:29:10.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.483+0000 7fd3d0c3ff40 1 -- start start 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3d0c3ff40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6e9a9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3d0c3ff40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3d0c3ff40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e9a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3d0c3ff40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6e877fd0 con 0x55fa6e987ea0 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3d0c3ff40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6e87bf30 con 0x55fa6e901f80 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3d0c3ff40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa6e9a7660 con 0x55fa6e987ac0 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3ce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e9a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3ce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e9a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49392/0 (socket says 172.21.15.97:49392) 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3ce000640 1 -- 172.21.15.97:0/725826718 learned_addr learned my addr 172.21.15.97:0/725826718 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3ce000640 1 -- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 msgr2=0x55fa6e9a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.484+0000 7fd3cd600640 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6e9a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3c7e00640 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3ce000640 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6e9a9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3ce000640 1 -- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 msgr2=0x55fa6e9a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3ce000640 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3ce000640 1 -- 172.21.15.97:0/725826718 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa6e988380 con 0x55fa6e987ea0 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3cd600640 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6e9a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3ce000640 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e9a4980 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd3c801b7c0 tx=0x7fd3c804c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3939c5911b7440 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3c7400640 1 -- 172.21.15.97:0/725826718 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8061070 con 0x55fa6e987ea0 2024-03-06T07:29:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3c7400640 1 -- 172.21.15.97:0/725826718 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c8006030 con 0x55fa6e987ea0 2024-03-06T07:29:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.485+0000 7fd3c7400640 1 -- 172.21.15.97:0/725826718 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8006d50 con 0x55fa6e987ea0 2024-03-06T07:29:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/725826718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 msgr2=0x55fa6e9a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e9a4980 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd3c801b7c0 tx=0x7fd3c804c070 comp rx=0 tx=0).stop 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/725826718 shutdown_connections 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e9a4980 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/725826718 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6e9a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/725826718 >> 172.21.15.97:0/725826718 conn(0x55fa6e8fe8c0 msgr2=0x55fa6e901870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/725826718 shutdown_connections 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.486+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/725826718 wait complete. 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.487+0000 7fd3d0c3ff40 1 Processor -- start 2024-03-06T07:29:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.487+0000 7fd3d0c3ff40 1 -- start start 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3d0c3ff40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3d0c3ff40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6ea36330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3d0c3ff40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e99d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3d0c3ff40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6e988dd0 con 0x55fa6e987ea0 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3d0c3ff40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6e988af0 con 0x55fa6e901f80 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3d0c3ff40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa6e9890b0 con 0x55fa6e987ac0 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e99d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e99d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49406/0 (socket says 172.21.15.97:49406) 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 -- 172.21.15.97:0/3389849312 learned_addr learned my addr 172.21.15.97:0/3389849312 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3c7e00640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6ea36330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3cd600640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 -- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 msgr2=0x55fa6ea36330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55fa6e987ac0 0x55fa6ea36330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 -- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 msgr2=0x55fa6e9a93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.488+0000 7fd3ce000640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3ce000640 1 -- 172.21.15.97:0/3389849312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa6e9a4b40 con 0x55fa6e987ea0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3cd600640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa6e901f80 0x55fa6e9a93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3ce000640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e99d2f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80012e0 tx=0x7fd3b4004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3c6000640 1 -- 172.21.15.97:0/3389849312 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c806e070 con 0x55fa6e987ea0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3c6000640 1 -- 172.21.15.97:0/3389849312 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c80061e0 con 0x55fa6e987ea0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3c6000640 1 -- 172.21.15.97:0/3389849312 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8061040 con 0x55fa6e987ea0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fa6e9a4e20 con 0x55fa6e987ea0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.489+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa6e9a53e0 con 0x55fa6e987ea0 2024-03-06T07:29:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.491+0000 7fd3c6000640 1 -- 172.21.15.97:0/3389849312 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3c8071070 con 0x55fa6e987ea0 2024-03-06T07:29:10.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.493+0000 7fd3c6000640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3ac069eb0 0x7fd3ac06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.493+0000 7fd3c6000640 1 -- 172.21.15.97:0/3389849312 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3c80caa90 con 0x55fa6e987ea0 2024-03-06T07:29:10.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.493+0000 7fd3cd600640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3ac069eb0 0x7fd3ac06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.493+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x55fa6ea36870 0x55fa6e98aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.493+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v9 -- 0x55fa6e9a5be0 con 0x55fa6ea36870 2024-03-06T07:29:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.494+0000 7fd3c7e00640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x55fa6ea36870 0x55fa6e98aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.494+0000 7fd3cd600640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3ac069eb0 0x7fd3ac06c370 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd3c00096a0 tx=0x7fd3c0003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.494+0000 7fd3c7e00640 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x55fa6ea36870 0x55fa6e98aff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.494+0000 7fd3c7e00640 1 -- 172.21.15.97:0/3389849312 <== osd.3 v2:172.21.15.97:6800/3169996968 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7fd3b800e6d0 con 0x55fa6ea36870 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x55fa6ea36870 msgr2=0x55fa6e98aff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x55fa6ea36870 0x55fa6e98aff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3ac069eb0 msgr2=0x7fd3ac06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3ac069eb0 0x7fd3ac06c370 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd3c00096a0 tx=0x7fd3c0003020 comp rx=0 tx=0).stop 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 msgr2=0x55fa6e99d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 --2- 172.21.15.97:0/3389849312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fa6e987ea0 0x55fa6e99d2f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80012e0 tx=0x7fd3b4004120 comp rx=0 tx=0).stop 2024-03-06T07:29:10.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3ce000640 1 -- 172.21.15.97:0/3389849312 reap_dead start 2024-03-06T07:29:10.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 shutdown_connections 2024-03-06T07:29:10.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 >> 172.21.15.97:0/3389849312 conn(0x55fa6e8fe8c0 msgr2=0x55fa6e9a86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 shutdown_connections 2024-03-06T07:29:10.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.495+0000 7fd3d0c3ff40 1 -- 172.21.15.97:0/3389849312 wait complete. 2024-03-06T07:29:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:10.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:29:10.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.692+0000 7f8d50400640 1 Processor -- start 2024-03-06T07:29:10.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.692+0000 7f8d50400640 1 -- start start 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.693+0000 7f8d50400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d48158560 0x7f8d4806c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d50400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d4806afc0 0x7f8d4815f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d50400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4806b4a0 con 0x7f8d480b33b0 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d50400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d480b5670 con 0x7f8d4806afc0 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d50400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4806b780 con 0x7f8d48158560 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d4d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d4806afc0 0x7f8d4815f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d4da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d48158560 0x7f8d4806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d4da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d48158560 0x7f8d4806c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:37184/0 (socket says 172.21.15.97:37184) 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d4da00640 1 -- 172.21.15.97:0/1488884377 learned_addr learned my addr 172.21.15.97:0/1488884377 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.694+0000 7f8d4d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d4806afc0 0x7f8d4815f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:57044/0 (socket says 172.21.15.97:57044) 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.695+0000 7f8d4d000640 1 -- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d48158560 msgr2=0x7f8d4806c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.695+0000 7f8d4d000640 1 --2- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d48158560 0x7f8d4806c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.695+0000 7f8d4d000640 1 -- 172.21.15.97:0/1488884377 >> v1:172.21.15.135:6789/0 conn(0x7f8d480b33b0 legacy=0x7f8d4806d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.695+0000 7f8d4d000640 1 -- 172.21.15.97:0/1488884377 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4806ba60 con 0x7f8d4806afc0 2024-03-06T07:29:10.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.697+0000 7f8d4d000640 1 --2- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d4806afc0 0x7f8d4815f000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8d340097a0 tx=0x7f8d3402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=973644aafad5c114 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.697+0000 7f8d47e00640 1 -- 172.21.15.97:0/1488884377 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d340315a0 con 0x7f8d4806afc0 2024-03-06T07:29:10.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.697+0000 7f8d47e00640 1 -- 172.21.15.97:0/1488884377 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34037070 con 0x7f8d4806afc0 2024-03-06T07:29:10.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.697+0000 7f8d47e00640 1 -- 172.21.15.97:0/1488884377 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34031960 con 0x7f8d4806afc0 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 -- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d4806afc0 msgr2=0x7f8d4815f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 --2- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d4806afc0 0x7f8d4815f000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8d340097a0 tx=0x7f8d3402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 -- 172.21.15.97:0/1488884377 shutdown_connections 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 --2- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d4806afc0 0x7f8d4815f000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 --2- 172.21.15.97:0/1488884377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d48158560 0x7f8d4806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 -- 172.21.15.97:0/1488884377 >> 172.21.15.97:0/1488884377 conn(0x7f8d480d0340 msgr2=0x7f8d480d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 -- 172.21.15.97:0/1488884377 shutdown_connections 2024-03-06T07:29:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.698+0000 7f8d50400640 1 -- 172.21.15.97:0/1488884377 wait complete. 2024-03-06T07:29:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.699+0000 7f8d50400640 1 Processor -- start 2024-03-06T07:29:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.700+0000 7f8d50400640 1 -- start start 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.700+0000 7f8d50400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d4806afc0 0x7f8d4813cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.700+0000 7f8d50400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 0x7f8d48158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d50400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d48158560 0x7f8d4813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d50400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48158940 con 0x7f8d480b33b0 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d50400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4806c020 con 0x7f8d48158560 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d50400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d480b3800 con 0x7f8d4806afc0 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 0x7f8d48158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 0x7f8d48158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49412/0 (socket says 172.21.15.97:49412) 2024-03-06T07:29:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 -- 172.21.15.97:0/3470783925 learned_addr learned my addr 172.21.15.97:0/3470783925 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:10.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 -- 172.21.15.97:0/3470783925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d4806afc0 msgr2=0x7f8d4813cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d4806afc0 0x7f8d4813cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 -- 172.21.15.97:0/3470783925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d48158560 msgr2=0x7f8d4813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d48158560 0x7f8d4813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.701+0000 7f8d4d000640 1 -- 172.21.15.97:0/3470783925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4813d2e0 con 0x7f8d480b33b0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.702+0000 7f8d4d000640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 0x7f8d48158ac0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8d34034040 tx=0x7f8d340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.702+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34032630 con 0x7f8d480b33b0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.702+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d340327f0 con 0x7f8d480b33b0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.702+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d48159430 con 0x7f8d480b33b0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.702+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34032be0 con 0x7f8d480b33b0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.702+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d481599f0 con 0x7f8d480b33b0 2024-03-06T07:29:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.703+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4804f330 con 0x7f8d480b33b0 2024-03-06T07:29:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.704+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d3403b3f0 con 0x7f8d480b33b0 2024-03-06T07:29:10.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.705+0000 7f8d45600640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d2806b500 0x7f8d2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.705+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3402f030 con 0x7f8d480b33b0 2024-03-06T07:29:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.706+0000 7f8d4da00640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d2806b500 0x7f8d2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.706+0000 7f8d4da00640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d2806b500 0x7f8d2806d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c006f40 tx=0x7f8d3c006e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:10.709+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d4813d2e0 con 0x7f8d480b33b0 2024-03-06T07:29:11.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.031+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f8d48074240 con 0x7f8d480b33b0 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.032+0000 7f8d45600640 1 -- 172.21.15.97:0/3470783925 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1693 (secure 0 0 0) 0x7f8d34040070 con 0x7f8d480b33b0 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: "total_avail_bytes": 772830412800, 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_bytes": 263700480, 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_bytes": 263700480, 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_ratio": 0.0003410975041333586, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "num_osds": 8, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "num_per_pool_osds": 8, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "stats_by_class": { 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "ssd": { 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "total_avail_bytes": 772830412800, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_bytes": 263700480, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_bytes": 263700480, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_ratio": 0.0003410975041333586 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "pools": [ 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:11.035 INFO:teuthology.orchestra.run.smithi097.stdout: "name": ".mgr", 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 1, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 590368, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 2, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 1160, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 1187840, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 1.6180679267563391e-06, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367054454784 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephfs_metadata", 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 4, 2024-03-06T07:29:11.036 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 1219006, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 23, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 2444, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 2502656, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 3.4090955978172133e-06, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367054454784 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephfs_data", 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 5, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 210944, 2024-03-06T07:29:11.037 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 213, 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 1648, 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 1687552, 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 2.2987708234722959e-06, 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367054454784 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d2806b500 msgr2=0x7f8d2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d2806b500 0x7f8d2806d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c006f40 tx=0x7f8d3c006e90 comp rx=0 tx=0).stop 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 msgr2=0x7f8d48158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 0x7f8d48158ac0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8d34034040 tx=0x7f8d340046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 shutdown_connections 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d2806b500 0x7f8d2806d9c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d48158560 0x7f8d4813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d480b33b0 0x7f8d48158ac0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 --2- 172.21.15.97:0/3470783925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d4806afc0 0x7f8d4813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.034+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 >> 172.21.15.97:0/3470783925 conn(0x7f8d480d0340 msgr2=0x7f8d480b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.035+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 shutdown_connections 2024-03-06T07:29:11.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:11.035+0000 7f8d50400640 1 -- 172.21.15.97:0/3470783925 wait complete. 2024-03-06T07:29:11.055 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:29:16.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:29:16.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.260+0000 7f012f200640 1 Processor -- start 2024-03-06T07:29:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.261+0000 7f012f200640 1 -- start start 2024-03-06T07:29:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01280b40c0 0x7f01280b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01281620c0 0x7f01280b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0128161900 con 0x7f01280ae670 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01280b49e0 con 0x7f01281620c0 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012815ebd0 con 0x7f01280b40c0 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01281620c0 0x7f01280b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01281620c0 0x7f01280b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40668/0 (socket says 172.21.15.97:40668) 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.262+0000 7f012e800640 1 -- 172.21.15.97:0/359878197 learned_addr learned my addr 172.21.15.97:0/359878197 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.263+0000 7f012de00640 1 --2- 172.21.15.97:0/359878197 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01280b40c0 0x7f01280b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.263+0000 7f012e800640 1 -- 172.21.15.97:0/359878197 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01280b40c0 msgr2=0x7f01280b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.263+0000 7f012e800640 1 --2- 172.21.15.97:0/359878197 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01280b40c0 0x7f01280b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.263+0000 7f012e800640 1 -- 172.21.15.97:0/359878197 >> v1:172.21.15.97:6789/0 conn(0x7f01280ae670 legacy=0x7f012815e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.263+0000 7f012e800640 1 -- 172.21.15.97:0/359878197 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012815eeb0 con 0x7f01281620c0 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.264+0000 7f012e800640 1 --2- 172.21.15.97:0/359878197 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01281620c0 0x7f01280b28e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0124009e50 tx=0x7f012402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7891ccc3798e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.264+0000 7f012ca00640 1 -- 172.21.15.97:0/359878197 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012402cb00 con 0x7f01281620c0 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.264+0000 7f012ca00640 1 -- 172.21.15.97:0/359878197 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012402cca0 con 0x7f01281620c0 2024-03-06T07:29:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.264+0000 7f012ca00640 1 -- 172.21.15.97:0/359878197 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124034610 con 0x7f01281620c0 2024-03-06T07:29:16.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 -- 172.21.15.97:0/359878197 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01281620c0 msgr2=0x7f01280b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 --2- 172.21.15.97:0/359878197 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01281620c0 0x7f01280b28e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0124009e50 tx=0x7f012402bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:16.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 -- 172.21.15.97:0/359878197 shutdown_connections 2024-03-06T07:29:16.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 --2- 172.21.15.97:0/359878197 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01281620c0 0x7f01280b28e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 --2- 172.21.15.97:0/359878197 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01280b40c0 0x7f01280b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 -- 172.21.15.97:0/359878197 >> 172.21.15.97:0/359878197 conn(0x7f012806afc0 msgr2=0x7f012806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 -- 172.21.15.97:0/359878197 shutdown_connections 2024-03-06T07:29:16.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.265+0000 7f012f200640 1 -- 172.21.15.97:0/359878197 wait complete. 2024-03-06T07:29:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.267+0000 7f012f200640 1 Processor -- start 2024-03-06T07:29:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.267+0000 7f012f200640 1 -- start start 2024-03-06T07:29:16.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 0x7f01281546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01280b40c0 0x7f0128156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01281620c0 0x7f012814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128158660 con 0x7f01280ae670 2024-03-06T07:29:16.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128161ba0 con 0x7f01280b40c0 2024-03-06T07:29:16.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01281587e0 con 0x7f01281620c0 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 0x7f01281546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 0x7f01281546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40674/0 (socket says 172.21.15.97:40674) 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.268+0000 7f012de00640 1 -- 172.21.15.97:0/2517459553 learned_addr learned my addr 172.21.15.97:0/2517459553 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012e800640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01281620c0 0x7f012814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012d400640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01280b40c0 0x7f0128156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012de00640 1 -- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01281620c0 msgr2=0x7f012814f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012de00640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01281620c0 0x7f012814f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012de00640 1 -- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01280b40c0 msgr2=0x7f0128156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012de00640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01280b40c0 0x7f0128156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012de00640 1 -- 172.21.15.97:0/2517459553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128153c70 con 0x7f01280ae670 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012d400640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01280b40c0 0x7f0128156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:16.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.269+0000 7f012de00640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 0x7f01281546e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f011c00d5f0 tx=0x7f011c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.270+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0125f0 con 0x7f01280ae670 2024-03-06T07:29:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.270+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c019070 con 0x7f01280ae670 2024-03-06T07:29:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.270+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0129d0 con 0x7f01280ae670 2024-03-06T07:29:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.270+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012814f750 con 0x7f01280ae670 2024-03-06T07:29:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.270+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0128153ee0 con 0x7f01280ae670 2024-03-06T07:29:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.271+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012804f330 con 0x7f01280ae670 2024-03-06T07:29:16.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.273+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f011c012b70 con 0x7f01280ae670 2024-03-06T07:29:16.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.274+0000 7f0122000640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.274+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f011c08a5d0 con 0x7f01280ae670 2024-03-06T07:29:16.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.274+0000 7f012d400640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.275+0000 7f012d400640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f010c06b4d0 0x7f010c06d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f01280d0820 tx=0x7f0118009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.279+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f011c059440 con 0x7f01280ae670 2024-03-06T07:29:16.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.544+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f01280adf40 con 0x7f01280ae670 2024-03-06T07:29:16.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.545+0000 7f0122000640 1 -- 172.21.15.97:0/2517459553 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1658 (secure 0 0 0) 0x7f011c058d40 con 0x7f01280ae670 2024-03-06T07:29:16.546 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "total_avail_bytes": 772840194048, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_bytes": 253919232, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_bytes": 253919232, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_ratio": 0.00032844542874954641, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "num_osds": 8, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "num_per_pool_osds": 8, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "stats_by_class": { 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "ssd": { 2024-03-06T07:29:16.547 INFO:teuthology.orchestra.run.smithi097.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "total_avail_bytes": 772840194048, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_bytes": 253919232, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_bytes": 253919232, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_ratio": 0.00032844542874954641 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "pools": [ 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "name": ".mgr", 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 1, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 590368, 2024-03-06T07:29:16.548 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 2, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 1160, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 1187840, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 1.6179835711227497e-06, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367073591296 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephfs_metadata", 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 4, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 1358047, 2024-03-06T07:29:16.549 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 23, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 2712, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 2777088, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 3.7827260257472517e-06, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367073591296 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephfs_data", 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 5, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 0, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 0, 2024-03-06T07:29:16.550 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 0, 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 0, 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 0, 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367073591296 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:29:16.551 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.548+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f010c06b4d0 msgr2=0x7f010c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.548+0000 7f012f200640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f010c06b4d0 0x7f010c06d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f01280d0820 tx=0x7f0118009040 comp rx=0 tx=0).stop 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.548+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 msgr2=0x7f01281546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.548+0000 7f012f200640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 0x7f01281546e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f011c00d5f0 tx=0x7f011c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 shutdown_connections 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01281620c0 0x7f012814f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01280b40c0 0x7f0128156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 --2- 172.21.15.97:0/2517459553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01280ae670 0x7f01281546e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 >> 172.21.15.97:0/2517459553 conn(0x7f012806afc0 msgr2=0x7f01280af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 shutdown_connections 2024-03-06T07:29:16.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:16.549+0000 7f012f200640 1 -- 172.21.15.97:0/2517459553 wait complete. 2024-03-06T07:29:16.567 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:29:16.567 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:16.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:16.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:16.780 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:16.781 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:16.781 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:16.831 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:16.831 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:16.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:17.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.125+0000 7f3b6d200640 1 Processor -- start 2024-03-06T07:29:17.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.125+0000 7f3b6d200640 1 -- start start 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b6d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b680b4680 0x7f3b6806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b6d200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b68072340 0x7f3b6806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b6d200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6815b090 con 0x7f3b6815acb0 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b6d200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b680b2a30 con 0x7f3b68072340 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b6d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680b4360 con 0x7f3b680b4680 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b66800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b68072340 0x7f3b6806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b65e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3b6815acb0 0x7f3b680b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44940/0 (socket says 172.21.15.97:44940) 2024-03-06T07:29:17.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b65e00640 1 -- 172.21.15.97:0/2513900829 learned_addr learned my addr 172.21.15.97:0/2513900829 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:17.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.127+0000 7f3b5fe00640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b680b4680 0x7f3b6806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.128+0000 7f3b66800640 1 -- 172.21.15.97:0/2513900829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b680b4680 msgr2=0x7f3b6806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.128+0000 7f3b66800640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b680b4680 0x7f3b6806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.128+0000 7f3b66800640 1 -- 172.21.15.97:0/2513900829 >> v1:172.21.15.97:6789/0 conn(0x7f3b6815acb0 legacy=0x7f3b680b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.128+0000 7f3b66800640 1 -- 172.21.15.97:0/2513900829 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b680b2cf0 con 0x7f3b68072340 2024-03-06T07:29:17.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.128+0000 7f3b65400640 1 -- 172.21.15.97:0/2513900829 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262417705 0 0) 0x7f3b6815b090 con 0x7f3b6815acb0 2024-03-06T07:29:17.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.128+0000 7f3b5fe00640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b680b4680 0x7f3b6806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.129+0000 7f3b66800640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b68072340 0x7f3b6806ea30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009d90 tx=0x7f3b6002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f9f2bf8979dece server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.129+0000 7f3b65400640 1 -- 172.21.15.97:0/2513900829 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60039070 con 0x7f3b68072340 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.129+0000 7f3b65400640 1 -- 172.21.15.97:0/2513900829 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6002cc50 con 0x7f3b68072340 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.129+0000 7f3b65400640 1 -- 172.21.15.97:0/2513900829 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60034440 con 0x7f3b68072340 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.129+0000 7f3b6d200640 1 -- 172.21.15.97:0/2513900829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b68072340 msgr2=0x7f3b6806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.129+0000 7f3b6d200640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b68072340 0x7f3b6806ea30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009d90 tx=0x7f3b6002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.130+0000 7f3b6d200640 1 -- 172.21.15.97:0/2513900829 shutdown_connections 2024-03-06T07:29:17.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.130+0000 7f3b6d200640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b68072340 0x7f3b6806ea30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.130+0000 7f3b6d200640 1 --2- 172.21.15.97:0/2513900829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b680b4680 0x7f3b6806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.130+0000 7f3b6d200640 1 -- 172.21.15.97:0/2513900829 >> 172.21.15.97:0/2513900829 conn(0x7f3b680b1730 msgr2=0x7f3b680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.130+0000 7f3b6d200640 1 -- 172.21.15.97:0/2513900829 shutdown_connections 2024-03-06T07:29:17.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.130+0000 7f3b6d200640 1 -- 172.21.15.97:0/2513900829 wait complete. 2024-03-06T07:29:17.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.132+0000 7f3b6d200640 1 Processor -- start 2024-03-06T07:29:17.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.133+0000 7f3b6d200640 1 -- start start 2024-03-06T07:29:17.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b6d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b68072340 0x7f3b680a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b6d200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b680b4680 0x7f3b68141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b6d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 0x7f3b68144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b6d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680b4010 con 0x7f3b6815acb0 2024-03-06T07:29:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b6d200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680b32b0 con 0x7f3b680b4680 2024-03-06T07:29:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b6d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680ccfb0 con 0x7f3b68072340 2024-03-06T07:29:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b66800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 0x7f3b68144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b66800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 0x7f3b68144660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40692/0 (socket says 172.21.15.97:40692) 2024-03-06T07:29:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.134+0000 7f3b66800640 1 -- 172.21.15.97:0/658111584 learned_addr learned my addr 172.21.15.97:0/658111584 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b65e00640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b68072340 0x7f3b680a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b66800640 1 -- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b68072340 msgr2=0x7f3b680a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b66800640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b68072340 0x7f3b680a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b66800640 1 -- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b680b4680 msgr2=0x7f3b68141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b66800640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b680b4680 0x7f3b68141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b66800640 1 -- 172.21.15.97:0/658111584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b68142670 con 0x7f3b6815acb0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b66800640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 0x7f3b68144660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009d90 tx=0x7f3b60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60039050 con 0x7f3b6815acb0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.135+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b60045070 con 0x7f3b6815acb0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.136+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6003e610 con 0x7f3b6815acb0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.136+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b68142950 con 0x7f3b6815acb0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.136+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6815b1b0 con 0x7f3b6815acb0 2024-03-06T07:29:17.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.137+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6804f330 con 0x7f3b6815acb0 2024-03-06T07:29:17.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.138+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b6003e7b0 con 0x7f3b6815acb0 2024-03-06T07:29:17.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.139+0000 7f3b5e000640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4406b4d0 0x7f3b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.139+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b600ace90 con 0x7f3b6815acb0 2024-03-06T07:29:17.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.139+0000 7f3b65e00640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4406b4d0 0x7f3b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.140+0000 7f3b65e00640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4406b4d0 0x7f3b4406d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b58000f60 tx=0x7f3b58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.144+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b60032e20 con 0x7f3b6815acb0 2024-03-06T07:29:17.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.412+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b6806ce60 con 0x7f3b6815acb0 2024-03-06T07:29:17.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.413+0000 7f3b5e000640 1 -- 172.21.15.97:0/658111584 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f3b600ae090 con 0x7f3b6815acb0 2024-03-06T07:29:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:17.415 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":35,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:52.884546+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:17.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4406b4d0 msgr2=0x7f3b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4406b4d0 0x7f3b4406d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b58000f60 tx=0x7f3b58009040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 msgr2=0x7f3b68144660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 0x7f3b68144660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009d90 tx=0x7f3b60004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 shutdown_connections 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4406b4d0 0x7f3b4406d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6815acb0 0x7f3b68144660 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b680b4680 0x7f3b68141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 --2- 172.21.15.97:0/658111584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b68072340 0x7f3b680a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 >> 172.21.15.97:0/658111584 conn(0x7f3b680b1730 msgr2=0x7f3b680b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 shutdown_connections 2024-03-06T07:29:17.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.416+0000 7f3b6d200640 1 -- 172.21.15.97:0/658111584 wait complete. 2024-03-06T07:29:17.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:49.145821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:50.344612+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:17.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:17.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.657+0000 7f6777e00640 1 Processor -- start 2024-03-06T07:29:17.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.657+0000 7f6777e00640 1 -- start start 2024-03-06T07:29:17.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.658+0000 7f6777e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.659+0000 7f6777e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780cf020 con 0x7f67780d1b90 2024-03-06T07:29:17.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.659+0000 7f6777e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780cf300 con 0x7f67780d40b0 2024-03-06T07:29:17.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.659+0000 7f6777e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67780cedd0 con 0x7f677815e000 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.659+0000 7f6776000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f67780d1b90 0x7f67780d39a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44956/0 (socket says 172.21.15.97:44956) 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.659+0000 7f6776000640 1 -- 172.21.15.97:0/2839430522 learned_addr learned my addr 172.21.15.97:0/2839430522 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.659+0000 7f6776a00640 1 --2- 172.21.15.97:0/2839430522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064204705 0 0) 0x7f67780cf020 con 0x7f67780d1b90 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f675c003520 con 0x7f67780d1b90 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937735 0 0) 0x7f67780cf300 con 0x7f67780d40b0 2024-03-06T07:29:17.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67780cf020 con 0x7f67780d40b0 2024-03-06T07:29:17.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6776a00640 1 -- 172.21.15.97:0/2839430522 >> v1:172.21.15.135:6789/0 conn(0x7f67780d40b0 legacy=0x7f67780b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6776a00640 1 -- 172.21.15.97:0/2839430522 >> v1:172.21.15.97:6789/0 conn(0x7f67780d1b90 legacy=0x7f67780d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.660+0000 7f6776a00640 1 -- 172.21.15.97:0/2839430522 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780cf5e0 con 0x7f677815e000 2024-03-06T07:29:17.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.662+0000 7f6776a00640 1 --2- 172.21.15.97:0/2839430522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780ce890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f67680097d0 tx=0x7f676802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdcfb4cca79300be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.663+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768035020 con 0x7f677815e000 2024-03-06T07:29:17.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.663+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6768031890 con 0x7f677815e000 2024-03-06T07:29:17.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.663+0000 7f6775600640 1 -- 172.21.15.97:0/2839430522 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768031c40 con 0x7f677815e000 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 -- 172.21.15.97:0/2839430522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 msgr2=0x7f67780ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 --2- 172.21.15.97:0/2839430522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780ce890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f67680097d0 tx=0x7f676802b930 comp rx=0 tx=0).stop 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 -- 172.21.15.97:0/2839430522 shutdown_connections 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 --2- 172.21.15.97:0/2839430522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780ce890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 -- 172.21.15.97:0/2839430522 >> 172.21.15.97:0/2839430522 conn(0x7f677808d740 msgr2=0x7f677809d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 -- 172.21.15.97:0/2839430522 shutdown_connections 2024-03-06T07:29:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.664+0000 7f6777e00640 1 -- 172.21.15.97:0/2839430522 wait complete. 2024-03-06T07:29:17.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.666+0000 7f6777e00640 1 Processor -- start 2024-03-06T07:29:17.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.666+0000 7f6777e00640 1 -- start start 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.667+0000 7f6777e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 0x7f677815a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.667+0000 7f6777e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67780d40b0 0x7f67780b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6777e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6777e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67780d2010 con 0x7f67780d1b90 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6777e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67780cfba0 con 0x7f67780d40b0 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6777e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67780b44f0 con 0x7f677815e000 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 0x7f677815a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 0x7f677815a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40698/0 (socket says 172.21.15.97:40698) 2024-03-06T07:29:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 -- 172.21.15.97:0/2345356123 learned_addr learned my addr 172.21.15.97:0/2345356123 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6777400640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776000640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67780d40b0 0x7f67780b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 -- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 msgr2=0x7f67780c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780c7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 -- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67780d40b0 msgr2=0x7f67780b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67780d40b0 0x7f67780b8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.668+0000 7f6776a00640 1 -- 172.21.15.97:0/2345356123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780b7b90 con 0x7f67780d1b90 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.669+0000 7f6776a00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 0x7f677815a3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6768034040 tx=0x7f67680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.669+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768035050 con 0x7f67780d1b90 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.669+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6768003820 con 0x7f67780d1b90 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.669+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768003bf0 con 0x7f67780d1b90 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.669+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67780b7e70 con 0x7f67780d1b90 2024-03-06T07:29:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.669+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67780b8430 con 0x7f67780d1b90 2024-03-06T07:29:17.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.670+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677804f330 con 0x7f67780d1b90 2024-03-06T07:29:17.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.671+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6768038020 con 0x7f67780d1b90 2024-03-06T07:29:17.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.673+0000 7f676ea00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675806b500 0x7f675806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.673+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67680ac3c0 con 0x7f67780d1b90 2024-03-06T07:29:17.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.673+0000 7f6776000640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675806b500 0x7f675806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.674+0000 7f6776000640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675806b500 0x7f675806d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f67780b1140 tx=0x7f6764009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.677+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f676802d040 con 0x7f67780d1b90 2024-03-06T07:29:17.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.874+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67780d4780 con 0x7f67780d1b90 2024-03-06T07:29:17.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.875+0000 7f676ea00640 1 -- 172.21.15.97:0/2345356123 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f676807ac00 con 0x7f67780d1b90 2024-03-06T07:29:17.876 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:29:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675806b500 msgr2=0x7f675806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675806b500 0x7f675806d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f67780b1140 tx=0x7f6764009040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 msgr2=0x7f677815a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 0x7f677815a3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6768034040 tx=0x7f67680046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 shutdown_connections 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f675806b500 0x7f675806d9c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f677815e000 0x7f67780c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67780d40b0 0x7f67780b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 --2- 172.21.15.97:0/2345356123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67780d1b90 0x7f677815a3a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 >> 172.21.15.97:0/2345356123 conn(0x7f677808d740 msgr2=0x7f677806d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 shutdown_connections 2024-03-06T07:29:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:17.878+0000 7f6777e00640 1 -- 172.21.15.97:0/2345356123 wait complete. 2024-03-06T07:29:17.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.111+0000 7f86c7e00640 1 Processor -- start 2024-03-06T07:29:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.111+0000 7f86c7e00640 1 -- start start 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f86c0160470 0x7f86c006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f86c006b500 0x7f86c00cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c7e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c006f6d0 con 0x7f86c0145690 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c7e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c00cce60 con 0x7f86c0160470 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c7e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c006f9b0 con 0x7f86c006b500 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c4a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f86c0160470 0x7f86c006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c4a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f86c0160470 0x7f86c006afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40704/0 (socket says 172.21.15.97:40704) 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c4a00640 1 -- 172.21.15.97:0/1701889241 learned_addr learned my addr 172.21.15.97:0/1701889241 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c5400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f86c0145690 0x7f86c006ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44962/0 (socket says 172.21.15.97:44962) 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.113+0000 7f86c5e00640 1 --2- 172.21.15.97:0/1701889241 >> v2:172.21.15.135:3300/0 conn(0x7f86c006b500 0x7f86c00cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.114+0000 7f86c4a00640 1 -- 172.21.15.97:0/1701889241 >> v2:172.21.15.135:3300/0 conn(0x7f86c006b500 msgr2=0x7f86c00cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.114+0000 7f86c4a00640 1 --2- 172.21.15.97:0/1701889241 >> v2:172.21.15.135:3300/0 conn(0x7f86c006b500 0x7f86c00cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.114+0000 7f86c4a00640 1 -- 172.21.15.97:0/1701889241 >> v1:172.21.15.97:6789/0 conn(0x7f86c0145690 legacy=0x7f86c006ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.114+0000 7f86c4a00640 1 -- 172.21.15.97:0/1701889241 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00ccaf0 con 0x7f86c0160470 2024-03-06T07:29:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.114+0000 7f86bfe00640 1 -- 172.21.15.97:0/1701889241 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026918786 0 0) 0x7f86c006f6d0 con 0x7f86c0145690 2024-03-06T07:29:18.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.115+0000 7f86c4a00640 1 --2- 172.21.15.97:0/1701889241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c006afc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f86c015f8a0 tx=0x7f86ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce90aa69bd1ef2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86bfe00640 1 -- 172.21.15.97:0/1701889241 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac034070 con 0x7f86c0160470 2024-03-06T07:29:18.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86bfe00640 1 -- 172.21.15.97:0/1701889241 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac030860 con 0x7f86c0160470 2024-03-06T07:29:18.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86bfe00640 1 -- 172.21.15.97:0/1701889241 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac030c30 con 0x7f86c0160470 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86c7e00640 1 -- 172.21.15.97:0/1701889241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 msgr2=0x7f86c006afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86c7e00640 1 --2- 172.21.15.97:0/1701889241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c006afc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f86c015f8a0 tx=0x7f86ac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86c7e00640 1 -- 172.21.15.97:0/1701889241 shutdown_connections 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86c7e00640 1 --2- 172.21.15.97:0/1701889241 >> v2:172.21.15.135:3300/0 conn(0x7f86c006b500 0x7f86c00cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86c7e00640 1 --2- 172.21.15.97:0/1701889241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c006afc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.116+0000 7f86c7e00640 1 -- 172.21.15.97:0/1701889241 >> 172.21.15.97:0/1701889241 conn(0x7f86c00b27d0 msgr2=0x7f86c00b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.117+0000 7f86c7e00640 1 -- 172.21.15.97:0/1701889241 shutdown_connections 2024-03-06T07:29:18.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.117+0000 7f86c7e00640 1 -- 172.21.15.97:0/1701889241 wait complete. 2024-03-06T07:29:18.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.118+0000 7f86c7e00640 1 Processor -- start 2024-03-06T07:29:18.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.118+0000 7f86c7e00640 1 -- start start 2024-03-06T07:29:18.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.119+0000 7f86c7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86c006b500 0x7f86c01e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.119+0000 7f86c7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86c0145690 0x7f86c015f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.119+0000 7f86c7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c015b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.119+0000 7f86c7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00b3ad0 con 0x7f86c0160470 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.119+0000 7f86c7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00cc7a0 con 0x7f86c006b500 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.119+0000 7f86c7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00cd830 con 0x7f86c0145690 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c015b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c015b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40718/0 (socket says 172.21.15.97:40718) 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 -- 172.21.15.97:0/3715173202 learned_addr learned my addr 172.21.15.97:0/3715173202 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5400640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86c006b500 0x7f86c01e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c4a00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86c0145690 0x7f86c015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 -- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86c0145690 msgr2=0x7f86c015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86c0145690 0x7f86c015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 -- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86c006b500 msgr2=0x7f86c01e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86c006b500 0x7f86c01e6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 -- 172.21.15.97:0/3715173202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c01e5b50 con 0x7f86c0160470 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c4a00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86c0145690 0x7f86c015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:18.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.120+0000 7f86c5e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c015b030 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f86b8007e40 tx=0x7f86b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.121+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8014620 con 0x7f86c0160470 2024-03-06T07:29:18.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.121+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8019070 con 0x7f86c0160470 2024-03-06T07:29:18.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.121+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8014a20 con 0x7f86c0160470 2024-03-06T07:29:18.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.121+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c01e5e30 con 0x7f86c0160470 2024-03-06T07:29:18.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.121+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c01e63f0 con 0x7f86c0160470 2024-03-06T07:29:18.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.122+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c004f330 con 0x7f86c0160470 2024-03-06T07:29:18.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.123+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86b8014bc0 con 0x7f86c0160470 2024-03-06T07:29:18.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.125+0000 7f86bd600640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86a006b4d0 0x7f86a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.125+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86b808ab00 con 0x7f86c0160470 2024-03-06T07:29:18.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.125+0000 7f86c5400640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86a006b4d0 0x7f86a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.126+0000 7f86c5400640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86a006b4d0 0x7f86a006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000f60 tx=0x7f86b4007b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.129+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86b8059a10 con 0x7f86c0160470 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.324+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f86c00d06c0 con 0x7f86c0160470 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.325+0000 7f86bd600640 1 -- 172.21.15.97:0/3715173202 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7f86b8059310 con 0x7f86c0160470 2024-03-06T07:29:18.326 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86a006b4d0 msgr2=0x7f86a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86a006b4d0 0x7f86a006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000f60 tx=0x7f86b4007b70 comp rx=0 tx=0).stop 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 msgr2=0x7f86c015b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c015b030 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f86b8007e40 tx=0x7f86b800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 shutdown_connections 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86a006b4d0 0x7f86a006d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0160470 0x7f86c015b030 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86c0145690 0x7f86c015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 --2- 172.21.15.97:0/3715173202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86c006b500 0x7f86c01e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 >> 172.21.15.97:0/3715173202 conn(0x7f86c00b27d0 msgr2=0x7f86c00b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 shutdown_connections 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.327+0000 7f86c7e00640 1 -- 172.21.15.97:0/3715173202 wait complete. 2024-03-06T07:29:18.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:18.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.547+0000 7f16cdc00640 1 Processor -- start 2024-03-06T07:29:18.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.547+0000 7f16cdc00640 1 -- start start 2024-03-06T07:29:18.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16cdc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f16c80b3df0 0x7f16c80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16cdc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c80b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16cdc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c80b4710 con 0x7f16c80cda90 2024-03-06T07:29:18.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16cdc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c8072310 con 0x7f16c80b3df0 2024-03-06T07:29:18.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16cdc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80719e0 con 0x7f16c80b3650 2024-03-06T07:29:18.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16c6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f16c80b3df0 0x7f16c80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16c6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f16c80b3df0 0x7f16c80727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40734/0 (socket says 172.21.15.97:40734) 2024-03-06T07:29:18.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.549+0000 7f16c6a00640 1 -- 172.21.15.97:0/2546552027 learned_addr learned my addr 172.21.15.97:0/2546552027 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.550+0000 7f16c6a00640 1 -- 172.21.15.97:0/2546552027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 msgr2=0x7f16c80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.550+0000 7f16bfe00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c80b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.550+0000 7f16c6a00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c80b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.550+0000 7f16c6a00640 1 -- 172.21.15.97:0/2546552027 >> v1:172.21.15.135:6789/0 conn(0x7f16c80cda90 legacy=0x7f16c806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.550+0000 7f16c6a00640 1 -- 172.21.15.97:0/2546552027 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8071cc0 con 0x7f16c80b3df0 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.550+0000 7f16bfe00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c80b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:18.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.551+0000 7f16c6a00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c80727a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f16b800bb10 tx=0x7f16b802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bede85450aad16a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16c6000640 1 -- 172.21.15.97:0/2546552027 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b803b070 con 0x7f16c80b3df0 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16c6000640 1 -- 172.21.15.97:0/2546552027 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8036440 con 0x7f16c80b3df0 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16c6000640 1 -- 172.21.15.97:0/2546552027 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8036810 con 0x7f16c80b3df0 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 -- 172.21.15.97:0/2546552027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 msgr2=0x7f16c80727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c80727a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f16b800bb10 tx=0x7f16b802e150 comp rx=0 tx=0).stop 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 -- 172.21.15.97:0/2546552027 shutdown_connections 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c80b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 --2- 172.21.15.97:0/2546552027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c80727a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 -- 172.21.15.97:0/2546552027 >> 172.21.15.97:0/2546552027 conn(0x7f16c80b1940 msgr2=0x7f16c80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 -- 172.21.15.97:0/2546552027 shutdown_connections 2024-03-06T07:29:18.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.552+0000 7f16cdc00640 1 -- 172.21.15.97:0/2546552027 wait complete. 2024-03-06T07:29:18.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.554+0000 7f16cdc00640 1 Processor -- start 2024-03-06T07:29:18.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.554+0000 7f16cdc00640 1 -- start start 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16cdc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c815da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16cdc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16cdc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16c80cda90 0x7f16c809a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16cdc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c815b240 con 0x7f16c80b3df0 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16cdc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80b3b30 con 0x7f16c80cda90 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16cdc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c815b3c0 con 0x7f16c80b3650 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.555+0000 7f16bfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c814f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40746/0 (socket says 172.21.15.97:40746) 2024-03-06T07:29:18.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 -- 172.21.15.97:0/578264613 learned_addr learned my addr 172.21.15.97:0/578264613 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16c6a00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c815da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16c7400640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16c80cda90 0x7f16c809a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 -- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 msgr2=0x7f16c815da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c815da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 -- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16c80cda90 msgr2=0x7f16c809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16c80cda90 0x7f16c809a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 -- 172.21.15.97:0/578264613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c809b240 con 0x7f16c80b3df0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.556+0000 7f16bfe00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c814f960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f16b000c820 tx=0x7f16b000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.557+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0015070 con 0x7f16c80b3df0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.557+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b0007a30 con 0x7f16c80b3df0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.557+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c809b520 con 0x7f16c80b3df0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.557+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0007e00 con 0x7f16c80b3df0 2024-03-06T07:29:18.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.557+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c809bae0 con 0x7f16c80b3df0 2024-03-06T07:29:18.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.558+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c804f330 con 0x7f16c80b3df0 2024-03-06T07:29:18.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.559+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16b0018070 con 0x7f16c80b3df0 2024-03-06T07:29:18.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.560+0000 7f16bea00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f16a406b500 0x7f16a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.560+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16b008a150 con 0x7f16c80b3df0 2024-03-06T07:29:18.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.560+0000 7f16c6a00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f16a406b500 0x7f16a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.560+0000 7f16c6a00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f16a406b500 0x7f16a406d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f16b8038000 tx=0x7f16b8032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.568+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16b0058fc0 con 0x7f16c80b3df0 2024-03-06T07:29:18.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.764+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16c80cc0e0 con 0x7f16c80b3df0 2024-03-06T07:29:18.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.765+0000 7f16bea00640 1 -- 172.21.15.97:0/578264613 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) v1 ==== 100+0+11 (secure 0 0 0) 0x7f16b00588c0 con 0x7f16c80b3df0 2024-03-06T07:29:18.766 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f16a406b500 msgr2=0x7f16a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f16a406b500 0x7f16a406d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f16b8038000 tx=0x7f16b8032000 comp rx=0 tx=0).stop 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 msgr2=0x7f16c814f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c814f960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f16b000c820 tx=0x7f16b000cc20 comp rx=0 tx=0).stop 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 shutdown_connections 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f16a406b500 0x7f16a406d9c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16c80cda90 0x7f16c809a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16c80b3df0 0x7f16c814f960 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 --2- 172.21.15.97:0/578264613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f16c80b3650 0x7f16c815da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.767+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 >> 172.21.15.97:0/578264613 conn(0x7f16c80b1940 msgr2=0x7f16c80b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.768+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 shutdown_connections 2024-03-06T07:29:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.768+0000 7f16cdc00640 1 -- 172.21.15.97:0/578264613 wait complete. 2024-03-06T07:29:18.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.995+0000 7f3241a00640 1 Processor -- start 2024-03-06T07:29:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.995+0000 7f3241a00640 1 -- start start 2024-03-06T07:29:18.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.997+0000 7f3241a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f323c0b2a60 0x7f323c0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.997+0000 7f3241a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 0x7f323c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.997+0000 7f3241a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c15f030 con 0x7f323c0d0600 2024-03-06T07:29:18.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.997+0000 7f3241a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c0cf090 con 0x7f323c0b2a60 2024-03-06T07:29:18.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.997+0000 7f3241a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c15f310 con 0x7f323c0d18e0 2024-03-06T07:29:18.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f323c0d0600 0x7f323c0d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44972/0 (socket says 172.21.15.97:44972) 2024-03-06T07:29:18.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323be00640 1 -- 172.21.15.97:0/3789900992 learned_addr learned my addr 172.21.15.97:0/3789900992 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323b400640 1 --2- 172.21.15.97:0/3789900992 >> v2:172.21.15.135:3300/0 conn(0x7f323c0b2a60 0x7f323c0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183119621 0 0) 0x7f323c15f030 con 0x7f323c0d0600 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f3241000640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 0x7f323c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3220003520 con 0x7f323c0d0600 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803684347 0 0) 0x7f3220003520 con 0x7f323c0d0600 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 msgr2=0x7f323c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 0x7f323c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 >> v2:172.21.15.135:3300/0 conn(0x7f323c0b2a60 msgr2=0x7f323c0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 --2- 172.21.15.97:0/3789900992 >> v2:172.21.15.135:3300/0 conn(0x7f323c0b2a60 0x7f323c0d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0cea40 con 0x7f323c0d0600 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f322c001670 con 0x7f323c0d0600 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.998+0000 7f323b400640 1 --2- 172.21.15.97:0/3789900992 >> v2:172.21.15.135:3300/0 conn(0x7f323c0b2a60 0x7f323c0d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 >> v1:172.21.15.97:6789/0 conn(0x7f323c0d0600 legacy=0x7f323c0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f3220003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 0x7f3220004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3220004d40 0x7f32200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0cf090 con 0x7f32200039b0 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c15f310 con 0x7f3220004d40 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322000bb20 con 0x7f323c0d18e0 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323be00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f3220003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f3241000640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3220004d40 0x7f32200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323be00640 1 -- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 msgr2=0x7f3220004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323be00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0d18e0 0x7f3220004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323be00640 1 -- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3220004d40 msgr2=0x7f32200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323be00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3220004d40 0x7f32200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f323be00640 1 -- 172.21.15.97:0/3789900992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220003520 con 0x7f32200039b0 2024-03-06T07:29:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:18.999+0000 7f3241000640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3220004d40 0x7f32200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:19.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.000+0000 7f323be00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f3220003dd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f322c00c160 tx=0x7f3220005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3afba3add31dff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c007c80 con 0x7f32200039b0 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c030070 con 0x7f32200039b0 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f323aa00640 1 -- 172.21.15.97:0/3789900992 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c005d30 con 0x7f32200039b0 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241a00640 1 -- 172.21.15.97:0/3789900992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 msgr2=0x7f3220003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241a00640 1 --2- 172.21.15.97:0/3789900992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f3220003dd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f322c00c160 tx=0x7f3220005830 comp rx=0 tx=0).stop 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241000640 1 -- 172.21.15.97:0/3789900992 reap_dead start 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241a00640 1 -- 172.21.15.97:0/3789900992 shutdown_connections 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241a00640 1 -- 172.21.15.97:0/3789900992 >> 172.21.15.97:0/3789900992 conn(0x7f323c0d3d10 msgr2=0x7f323c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241a00640 1 -- 172.21.15.97:0/3789900992 shutdown_connections 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.001+0000 7f3241a00640 1 -- 172.21.15.97:0/3789900992 wait complete. 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.003+0000 7f3241a00640 1 Processor -- start 2024-03-06T07:29:19.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.003+0000 7f3241a00640 1 -- start start 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f3241a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f323c0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f3241a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f323c0cade0 0x7f323c159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f3241a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0cc8c0 0x7f323c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f3241a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c15ecf0 con 0x7f32200039b0 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f3241a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0b3250 con 0x7f323c0cade0 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f3241a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0d1d70 con 0x7f323c0cc8c0 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f323be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f323c0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f323be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f323c0cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40764/0 (socket says 172.21.15.97:40764) 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.004+0000 7f323be00640 1 -- 172.21.15.97:0/1794818523 learned_addr learned my addr 172.21.15.97:0/1794818523 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323b400640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f323c0cade0 0x7f323c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f3241000640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0cc8c0 0x7f323c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323be00640 1 -- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0cc8c0 msgr2=0x7f323c159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323be00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0cc8c0 0x7f323c159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323be00640 1 -- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f323c0cade0 msgr2=0x7f323c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323be00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f323c0cade0 0x7f323c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323be00640 1 -- 172.21.15.97:0/1794818523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0cb2e0 con 0x7f32200039b0 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323b400640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f323c0cade0 0x7f323c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f323be00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f323c0cc380 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f322c02c4c0 tx=0x7f323c0ce780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c004bf0 con 0x7f32200039b0 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c004d90 con 0x7f32200039b0 2024-03-06T07:29:19.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c007270 con 0x7f32200039b0 2024-03-06T07:29:19.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.005+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c0cb5c0 con 0x7f32200039b0 2024-03-06T07:29:19.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.006+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c0cbb80 con 0x7f32200039b0 2024-03-06T07:29:19.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.007+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c04f330 con 0x7f32200039b0 2024-03-06T07:29:19.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.008+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f322c048020 con 0x7f32200039b0 2024-03-06T07:29:19.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.009+0000 7f3233e00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f321c06b500 0x7f321c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.009+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f322c0ab690 con 0x7f32200039b0 2024-03-06T07:29:19.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.009+0000 7f323b400640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f321c06b500 0x7f321c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.010+0000 7f323b400640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f321c06b500 0x7f321c06d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3228006fd0 tx=0x7f3228008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.013+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f322c07a480 con 0x7f32200039b0 2024-03-06T07:29:19.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.277+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f323c15e360 con 0x7f32200039b0 2024-03-06T07:29:19.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.278+0000 7f3233e00640 1 -- 172.21.15.97:0/1794818523 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v35) v1 ==== 80+0+0 (secure 0 0 0) 0x7f322c079d80 con 0x7f32200039b0 2024-03-06T07:29:19.279 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:29:19.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f321c06b500 msgr2=0x7f321c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f321c06b500 0x7f321c06d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3228006fd0 tx=0x7f3228008040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 msgr2=0x7f323c0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f323c0cc380 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f322c02c4c0 tx=0x7f323c0ce780 comp rx=0 tx=0).stop 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 shutdown_connections 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f321c06b500 0x7f321c06d9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f323c0cc8c0 0x7f323c159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f323c0cade0 0x7f323c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 --2- 172.21.15.97:0/1794818523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32200039b0 0x7f323c0cc380 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 >> 172.21.15.97:0/1794818523 conn(0x7f323c0d3d10 msgr2=0x7f323c06d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 shutdown_connections 2024-03-06T07:29:19.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.281+0000 7f3241a00640 1 -- 172.21.15.97:0/1794818523 wait complete. 2024-03-06T07:29:19.301 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:19.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:19.420 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:29:19.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:19.420 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:19.464 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:29:19.464 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:19.505 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:19.505 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:19.505 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:19.545 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:19.545 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:29:19.545 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:19.593 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:19.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.786+0000 7f76a2400640 1 Processor -- start 2024-03-06T07:29:19.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.786+0000 7f76a2400640 1 -- start start 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.787+0000 7f76a2400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f769c0cb1f0 0x7f769c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f76a2400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c0cb680 con 0x7f769c1458a0 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f76a2400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c06b900 con 0x7f769c06b500 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f76a2400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c06fac0 con 0x7f769c0cb1f0 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f769aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f769c0cb1f0 0x7f769c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f769b400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f769c1458a0 0x7f769c0725b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44980/0 (socket says 172.21.15.97:44980) 2024-03-06T07:29:19.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f769b400640 1 -- 172.21.15.97:0/697686531 learned_addr learned my addr 172.21.15.97:0/697686531 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:19.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f769aa00640 1 -- 172.21.15.97:0/697686531 >> v1:172.21.15.135:6789/0 conn(0x7f769c06b500 legacy=0x7f769c06df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f769aa00640 1 -- 172.21.15.97:0/697686531 >> v1:172.21.15.97:6789/0 conn(0x7f769c1458a0 legacy=0x7f769c0725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.788+0000 7f769aa00640 1 -- 172.21.15.97:0/697686531 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0d0140 con 0x7f769c0cb1f0 2024-03-06T07:29:19.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.789+0000 7f769aa00640 1 --2- 172.21.15.97:0/697686531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c0cb1f0 0x7f769c06afc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f76880097a0 tx=0x7f768802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88316fb62a76c3da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f769a000640 1 -- 172.21.15.97:0/697686531 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880315a0 con 0x7f769c0cb1f0 2024-03-06T07:29:19.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f769a000640 1 -- 172.21.15.97:0/697686531 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7688037070 con 0x7f769c0cb1f0 2024-03-06T07:29:19.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f769a000640 1 -- 172.21.15.97:0/697686531 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688031960 con 0x7f769c0cb1f0 2024-03-06T07:29:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f76a2400640 1 -- 172.21.15.97:0/697686531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c0cb1f0 msgr2=0x7f769c06afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f76a2400640 1 --2- 172.21.15.97:0/697686531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c0cb1f0 0x7f769c06afc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f76880097a0 tx=0x7f768802b840 comp rx=0 tx=0).stop 2024-03-06T07:29:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f76a2400640 1 -- 172.21.15.97:0/697686531 shutdown_connections 2024-03-06T07:29:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.790+0000 7f76a2400640 1 --2- 172.21.15.97:0/697686531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c0cb1f0 0x7f769c06afc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.791+0000 7f76a2400640 1 -- 172.21.15.97:0/697686531 >> 172.21.15.97:0/697686531 conn(0x7f769c0d0ba0 msgr2=0x7f769c0d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.791+0000 7f76a2400640 1 -- 172.21.15.97:0/697686531 shutdown_connections 2024-03-06T07:29:19.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.791+0000 7f76a2400640 1 -- 172.21.15.97:0/697686531 wait complete. 2024-03-06T07:29:19.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.792+0000 7f76a2400640 1 Processor -- start 2024-03-06T07:29:19.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.792+0000 7f76a2400640 1 -- start start 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.793+0000 7f76a2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f769c06b500 0x7f769c140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.793+0000 7f76a2400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f769c0cb1f0 0x7f769c0c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.793+0000 7f76a2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 0x7f769c0ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.793+0000 7f76a2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c06f7e0 con 0x7f769c1458a0 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.793+0000 7f76a2400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0d09a0 con 0x7f769c0cb1f0 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.793+0000 7f76a2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0d03e0 con 0x7f769c06b500 2024-03-06T07:29:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 0x7f769c0ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 0x7f769c0ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40778/0 (socket says 172.21.15.97:40778) 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 -- 172.21.15.97:0/3475627163 learned_addr learned my addr 172.21.15.97:0/3475627163 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769b400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f769c06b500 0x7f769c140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 -- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f769c06b500 msgr2=0x7f769c140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f769c06b500 0x7f769c140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 -- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f769c0cb1f0 msgr2=0x7f769c0c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f769c0cb1f0 0x7f769c0c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 -- 172.21.15.97:0/3475627163 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0ad900 con 0x7f769c1458a0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.794+0000 7f769be00640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 0x7f769c0ae500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f769400c7b0 tx=0x7f769400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.795+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694004b60 con 0x7f769c1458a0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.795+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7694004d00 con 0x7f769c1458a0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.795+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c0adbe0 con 0x7f769c1458a0 2024-03-06T07:29:19.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.795+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769400aa90 con 0x7f769c1458a0 2024-03-06T07:29:19.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.795+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c0c6800 con 0x7f769c1458a0 2024-03-06T07:29:19.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.796+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c0addb0 con 0x7f769c1458a0 2024-03-06T07:29:19.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.797+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f769400ac30 con 0x7f769c1458a0 2024-03-06T07:29:19.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.798+0000 7f7693400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f767c06b4d0 0x7f767c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.798+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f769408a570 con 0x7f769c1458a0 2024-03-06T07:29:19.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.798+0000 7f769b400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f767c06b4d0 0x7f767c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.799+0000 7f769b400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f767c06b4d0 0x7f767c06d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f768c000f60 tx=0x7f768c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:19.802+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7694020070 con 0x7f769c1458a0 2024-03-06T07:29:20.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.070+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f769c04f330 con 0x7f769c1458a0 2024-03-06T07:29:20.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.071+0000 7f7693400640 1 -- 172.21.15.97:0/3475627163 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f7694088e20 con 0x7f769c1458a0 2024-03-06T07:29:20.072 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:20.072 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:20.073 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.073+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f767c06b4d0 msgr2=0x7f767c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.073+0000 7f76a2400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f767c06b4d0 0x7f767c06d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f768c000f60 tx=0x7f768c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.073+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 msgr2=0x7f769c0ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.073+0000 7f76a2400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 0x7f769c0ae500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f769400c7b0 tx=0x7f769400cc70 comp rx=0 tx=0).stop 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 shutdown_connections 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f767c06b4d0 0x7f767c06d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f769c1458a0 0x7f769c0ae500 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f769c0cb1f0 0x7f769c0c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 --2- 172.21.15.97:0/3475627163 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f769c06b500 0x7f769c140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 >> 172.21.15.97:0/3475627163 conn(0x7f769c0d0ba0 msgr2=0x7f769c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 shutdown_connections 2024-03-06T07:29:20.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.074+0000 7f76a2400640 1 -- 172.21.15.97:0/3475627163 wait complete. 2024-03-06T07:29:20.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:20.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.290+0000 7fb4d1600640 1 Processor -- start 2024-03-06T07:29:20.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.290+0000 7fb4d1600640 1 -- start start 2024-03-06T07:29:20.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4d1600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb4cc06ec00 0x7fb4cc0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4d1600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc15b090 con 0x7fb4cc0b4680 2024-03-06T07:29:20.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4d1600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0b2a30 con 0x7fb4cc15acb0 2024-03-06T07:29:20.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4d1600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc0b4360 con 0x7fb4cc06ec00 2024-03-06T07:29:20.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4c2000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb4cc0b4680 0x7fb4cc06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44988/0 (socket says 172.21.15.97:44988) 2024-03-06T07:29:20.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4c2000640 1 -- 172.21.15.97:0/314414238 learned_addr learned my addr 172.21.15.97:0/314414238 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:20.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.292+0000 7fb4cb400640 1 --2- 172.21.15.97:0/314414238 >> v2:172.21.15.135:3300/0 conn(0x7fb4cc06ec00 0x7fb4cc0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948422323 0 0) 0x7fb4cc15b090 con 0x7fb4cc0b4680 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b0003520 con 0x7fb4cc0b4680 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711915538 0 0) 0x7fb4b0003520 con 0x7fb4cc0b4680 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 >> v2:172.21.15.135:3300/0 conn(0x7fb4cc06ec00 msgr2=0x7fb4cc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 --2- 172.21.15.97:0/314414238 >> v2:172.21.15.135:3300/0 conn(0x7fb4cc06ec00 0x7fb4cc0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 >> v1:172.21.15.135:6789/0 conn(0x7fb4cc15acb0 legacy=0x7fb4cc0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0b2cf0 con 0x7fb4cc0b4680 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb4b4001670 con 0x7fb4cc0b4680 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 >> v1:172.21.15.97:6789/0 conn(0x7fb4cc0b4680 legacy=0x7fb4cc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4b0003a00 0x7fb4b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4b00051c0 0x7fb4b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0b4360 con 0x7fb4b0004800 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b000c0c0 con 0x7fb4b00051c0 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.293+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b000c2c0 con 0x7fb4b0003a00 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4c2000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4caa00640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4b0003a00 0x7fb4b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4c2000640 1 -- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4b0003a00 msgr2=0x7fb4b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4c2000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4b0003a00 0x7fb4b0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4c2000640 1 -- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4b00051c0 msgr2=0x7fb4b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4c2000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4b00051c0 0x7fb4b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4c2000640 1 -- 172.21.15.97:0/314414238 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0003520 con 0x7fb4b0004800 2024-03-06T07:29:20.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.294+0000 7fb4cb400640 1 -- 172.21.15.97:0/314414238 reap_dead start 2024-03-06T07:29:20.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.295+0000 7fb4c2000640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4b0004c80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400c110 tx=0x7fb4b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a4b9d882b83b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.295+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4007c40 con 0x7fb4b0004800 2024-03-06T07:29:20.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.295+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b4030070 con 0x7fb4b0004800 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.295+0000 7fb4ca000640 1 -- 172.21.15.97:0/314414238 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4005d70 con 0x7fb4b0004800 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 -- 172.21.15.97:0/314414238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 msgr2=0x7fb4b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4b0004c80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400c110 tx=0x7fb4b0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 -- 172.21.15.97:0/314414238 shutdown_connections 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 --2- 172.21.15.97:0/314414238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 -- 172.21.15.97:0/314414238 >> 172.21.15.97:0/314414238 conn(0x7fb4cc0b1730 msgr2=0x7fb4cc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 -- 172.21.15.97:0/314414238 shutdown_connections 2024-03-06T07:29:20.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.296+0000 7fb4d1600640 1 -- 172.21.15.97:0/314414238 wait complete. 2024-03-06T07:29:20.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.298+0000 7fb4d1600640 1 Processor -- start 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.298+0000 7fb4d1600640 1 -- start start 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4d1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4cc1477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4d1600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4cc0c53b0 0x7fb4cc0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4d1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4cc158690 0x7fb4cc158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4d1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0b4010 con 0x7fb4b0004800 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4d1600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0b32b0 con 0x7fb4cc0c53b0 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4d1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc072340 con 0x7fb4cc158690 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4cc1477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4cc1477e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40790/0 (socket says 172.21.15.97:40790) 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4caa00640 1 -- 172.21.15.97:0/2304387245 learned_addr learned my addr 172.21.15.97:0/2304387245 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4c2000640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4cc0c53b0 0x7fb4cc0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.299+0000 7fb4cb400640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4cc158690 0x7fb4cc158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4caa00640 1 -- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4cc158690 msgr2=0x7fb4cc158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4caa00640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4cc158690 0x7fb4cc158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4caa00640 1 -- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4cc0c53b0 msgr2=0x7fb4cc0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4caa00640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4cc0c53b0 0x7fb4cc0c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4caa00640 1 -- 172.21.15.97:0/2304387245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc147d20 con 0x7fb4b0004800 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4c2000640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4cc0c53b0 0x7fb4cc0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4caa00640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4cc1477e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00d470 tx=0x7fb4bc00d930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc005480 con 0x7fb4b0004800 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc017070 con 0x7fb4b0004800 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc003810 con 0x7fb4b0004800 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc1593c0 con 0x7fb4b0004800 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.300+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc159980 con 0x7fb4b0004800 2024-03-06T07:29:20.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.302+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc04f330 con 0x7fb4b0004800 2024-03-06T07:29:20.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.303+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4bc0039b0 con 0x7fb4b0004800 2024-03-06T07:29:20.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.304+0000 7fb4c3400640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4a806b500 0x7fb4a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.304+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4bc089e90 con 0x7fb4b0004800 2024-03-06T07:29:20.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.304+0000 7fb4c2000640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4a806b500 0x7fb4a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.305+0000 7fb4c2000640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4a806b500 0x7fb4a806d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4034000 tx=0x7fb4b4039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.308+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4cc147d20 con 0x7fb4b0004800 2024-03-06T07:29:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.575+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4cc06ce60 con 0x7fb4b0004800 2024-03-06T07:29:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.576+0000 7fb4c3400640 1 -- 172.21.15.97:0/2304387245 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16657 (secure 0 0 0) 0x7fb4bc058da0 con 0x7fb4b0004800 2024-03-06T07:29:20.577 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:20.578 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":35,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:52.884546+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.578+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4a806b500 msgr2=0x7fb4a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.578+0000 7fb4d1600640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4a806b500 0x7fb4a806d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4034000 tx=0x7fb4b4039040 comp rx=0 tx=0).stop 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.578+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 msgr2=0x7fb4cc1477e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.578+0000 7fb4d1600640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4cc1477e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00d470 tx=0x7fb4bc00d930 comp rx=0 tx=0).stop 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 shutdown_connections 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4a806b500 0x7fb4a806d9c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4cc158690 0x7fb4cc158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4cc0c53b0 0x7fb4cc0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 --2- 172.21.15.97:0/2304387245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4b0004800 0x7fb4cc1477e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 >> 172.21.15.97:0/2304387245 conn(0x7fb4cc0b1730 msgr2=0x7fb4cc0b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 shutdown_connections 2024-03-06T07:29:20.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.579+0000 7fb4d1600640 1 -- 172.21.15.97:0/2304387245 wait complete. 2024-03-06T07:29:20.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:49.145821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:50.344612+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:20.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.789+0000 7feda5c00640 1 Processor -- start 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.789+0000 7feda5c00640 1 -- start start 2024-03-06T07:29:20.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.790+0000 7feda5c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feda0166fc0 0x7feda01693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.790+0000 7feda5c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00cf2f0 con 0x7feda00d06d0 2024-03-06T07:29:20.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.790+0000 7feda5c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00cf5d0 con 0x7feda00b4b90 2024-03-06T07:29:20.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.790+0000 7feda5c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda00cf8b0 con 0x7feda0166fc0 2024-03-06T07:29:20.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.790+0000 7fed9fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feda00d06d0 0x7feda00d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44998/0 (socket says 172.21.15.97:44998) 2024-03-06T07:29:20.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.790+0000 7fed9fe00640 1 -- 172.21.15.97:0/3723574297 learned_addr learned my addr 172.21.15.97:0/3723574297 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7feda5200640 1 --2- 172.21.15.97:0/3723574297 >> v2:172.21.15.135:3300/0 conn(0x7feda0166fc0 0x7feda01693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497181603 0 0) 0x7feda00cf2f0 con 0x7feda00d06d0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed84003520 con 0x7feda00d06d0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3071475467 0 0) 0x7fed84003520 con 0x7feda00d06d0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 >> v2:172.21.15.135:3300/0 conn(0x7feda0166fc0 msgr2=0x7feda01693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 --2- 172.21.15.97:0/3723574297 >> v2:172.21.15.135:3300/0 conn(0x7feda0166fc0 0x7feda01693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 >> v1:172.21.15.135:6789/0 conn(0x7feda00b4b90 legacy=0x7feda00cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00cfb90 con 0x7feda00d06d0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fed90001670 con 0x7feda00d06d0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 >> v1:172.21.15.97:6789/0 conn(0x7feda00d06d0 legacy=0x7feda00d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840039b0 0x7fed84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.791+0000 7fed9f400640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed84004800 0x7fed84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed9f400640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed840051c0 0x7fed8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda00cf8b0 con 0x7fed840051c0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8400c060 con 0x7fed840039b0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8400c260 con 0x7fed84004800 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7feda5200640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed840051c0 0x7fed8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed9fe00640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840039b0 0x7fed84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed97400640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed84004800 0x7fed84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7feda5200640 1 -- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed84004800 msgr2=0x7fed84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7feda5200640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed84004800 0x7fed84004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7feda5200640 1 -- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840039b0 msgr2=0x7fed84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7feda5200640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840039b0 0x7fed84003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7feda5200640 1 -- 172.21.15.97:0/3723574297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84003520 con 0x7fed840051c0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.792+0000 7fed9fe00640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840039b0 0x7fed84003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:20.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.793+0000 7feda5200640 1 -- 172.21.15.97:0/3723574297 reap_dead start 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.793+0000 7feda5200640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed840051c0 0x7fed8400b9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fed98009f90 tx=0x7fed84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19131264021fe06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed98009c10 con 0x7fed840051c0 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9802cc20 con 0x7fed840051c0 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7fed9f400640 1 -- 172.21.15.97:0/3723574297 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed98003990 con 0x7fed840051c0 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 -- 172.21.15.97:0/3723574297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed840051c0 msgr2=0x7fed8400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed840051c0 0x7fed8400b9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fed98009f90 tx=0x7fed84005830 comp rx=0 tx=0).stop 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 -- 172.21.15.97:0/3723574297 shutdown_connections 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 --2- 172.21.15.97:0/3723574297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed840051c0 0x7fed8400b9c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 -- 172.21.15.97:0/3723574297 >> 172.21.15.97:0/3723574297 conn(0x7feda009d070 msgr2=0x7feda00d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 -- 172.21.15.97:0/3723574297 shutdown_connections 2024-03-06T07:29:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.794+0000 7feda5c00640 1 -- 172.21.15.97:0/3723574297 wait complete. 2024-03-06T07:29:20.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.796+0000 7feda5c00640 1 Processor -- start 2024-03-06T07:29:20.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.796+0000 7feda5c00640 1 -- start start 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7feda5c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840051c0 0x7feda00caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7feda5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 0x7feda0154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7feda5c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feda01545f0 0x7feda0154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7feda5c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda00d0c30 con 0x7feda0154170 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7feda5c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda00d0ab0 con 0x7fed840051c0 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7feda5c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda00d0db0 con 0x7feda01545f0 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7fed97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 0x7feda0154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7fed97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 0x7feda0154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40808/0 (socket says 172.21.15.97:40808) 2024-03-06T07:29:20.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.797+0000 7fed97400640 1 -- 172.21.15.97:0/3827415974 learned_addr learned my addr 172.21.15.97:0/3827415974 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:20.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7feda5200640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feda01545f0 0x7feda0154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed9fe00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840051c0 0x7feda00caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed97400640 1 -- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feda01545f0 msgr2=0x7feda0154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed97400640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feda01545f0 0x7feda0154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed97400640 1 -- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840051c0 msgr2=0x7feda00caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed97400640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840051c0 0x7feda00caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed97400640 1 -- 172.21.15.97:0/3827415974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda01609a0 con 0x7feda0154170 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.798+0000 7fed97400640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 0x7feda0154a30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fed8800d4c0 tx=0x7fed8800d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.799+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88005480 con 0x7feda0154170 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.799+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed88017070 con 0x7feda0154170 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.799+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda0160c80 con 0x7feda0154170 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.799+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88003830 con 0x7feda0154170 2024-03-06T07:29:20.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.799+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda00b8180 con 0x7feda0154170 2024-03-06T07:29:20.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.800+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda004f330 con 0x7feda0154170 2024-03-06T07:29:20.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.801+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed88003a90 con 0x7feda0154170 2024-03-06T07:29:20.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.802+0000 7fed9cc00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7c06b4d0 0x7fed7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.802+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed8808a030 con 0x7feda0154170 2024-03-06T07:29:20.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.803+0000 7fed9fe00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7c06b4d0 0x7fed7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.803+0000 7fed9fe00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7c06b4d0 0x7fed7c06d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed90001200 tx=0x7fed90009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:20.807+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed88058e20 con 0x7feda0154170 2024-03-06T07:29:21.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.056+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feda00b3540 con 0x7feda0154170 2024-03-06T07:29:21.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.057+0000 7fed9cc00640 1 -- 172.21.15.97:0/3827415974 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed88058720 con 0x7feda0154170 2024-03-06T07:29:21.058 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:29:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7c06b4d0 msgr2=0x7fed7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7c06b4d0 0x7fed7c06d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed90001200 tx=0x7fed90009040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 msgr2=0x7feda0154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 0x7feda0154a30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fed8800d4c0 tx=0x7fed8800d980 comp rx=0 tx=0).stop 2024-03-06T07:29:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 shutdown_connections 2024-03-06T07:29:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7c06b4d0 0x7fed7c06d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feda01545f0 0x7feda0154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feda0154170 0x7feda0154a30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 --2- 172.21.15.97:0/3827415974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed840051c0 0x7feda00caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 >> 172.21.15.97:0/3827415974 conn(0x7feda009d070 msgr2=0x7feda006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 shutdown_connections 2024-03-06T07:29:21.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.061+0000 7feda5c00640 1 -- 172.21.15.97:0/3827415974 wait complete. 2024-03-06T07:29:21.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:21.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.276+0000 7fc46d600640 1 Processor -- start 2024-03-06T07:29:21.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.276+0000 7fc46d600640 1 -- start start 2024-03-06T07:29:21.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.277+0000 7fc46d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 0x7fc4680d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.277+0000 7fc46d600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680d0ff0 con 0x7fc4680d1db0 2024-03-06T07:29:21.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.277+0000 7fc46d600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680cf2f0 con 0x7fc4680b4b90 2024-03-06T07:29:21.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.277+0000 7fc46d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680cf5d0 con 0x7fc4680d06d0 2024-03-06T07:29:21.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.278+0000 7fc46cc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc4680d1db0 0x7fc4680d2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45014/0 (socket says 172.21.15.97:45014) 2024-03-06T07:29:21.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.278+0000 7fc46cc00640 1 -- 172.21.15.97:0/1332628656 learned_addr learned my addr 172.21.15.97:0/1332628656 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:21.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.278+0000 7fc467e00640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 0x7fc4680d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.278+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299921061 0 0) 0x7fc4680d0ff0 con 0x7fc4680d1db0 2024-03-06T07:29:21.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.278+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c003520 con 0x7fc4680d1db0 2024-03-06T07:29:21.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3330850463 0 0) 0x7fc44c003520 con 0x7fc4680d1db0 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 msgr2=0x7fc4680d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 0x7fc4680d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 >> v1:172.21.15.135:6789/0 conn(0x7fc4680b4b90 legacy=0x7fc4680cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4680cf8b0 con 0x7fc4680d1db0 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc4600019f0 con 0x7fc4680d1db0 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 >> v1:172.21.15.97:6789/0 conn(0x7fc4680d1db0 legacy=0x7fc4680d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 0x7fc44c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc44c003fb0 0x7fc44c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c004d40 0x7fc44c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680cf5d0 con 0x7fc44c003fb0 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00bb00 con 0x7fc44c004d40 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00bd00 con 0x7fc4680d06d0 2024-03-06T07:29:21.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.279+0000 7fc45f400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc44c003fb0 0x7fc44c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc467e00640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 0x7fc44c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc45f400640 1 -- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 msgr2=0x7fc44c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc46cc00640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c004d40 0x7fc44c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc45f400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680d06d0 0x7fc44c0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc45f400640 1 -- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c004d40 msgr2=0x7fc44c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc45f400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c004d40 0x7fc44c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.280+0000 7fc45f400640 1 -- 172.21.15.97:0/1332628656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c003520 con 0x7fc44c003fb0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc45f400640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc44c003fb0 0x7fc44c004800 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc458001750 tx=0x7fc44c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf507c881d09b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458001370 con 0x7fc44c003fb0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4580052d0 con 0x7fc44c003fb0 2024-03-06T07:29:21.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc467400640 1 -- 172.21.15.97:0/1332628656 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458003830 con 0x7fc44c003fb0 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc46d600640 1 -- 172.21.15.97:0/1332628656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc44c003fb0 msgr2=0x7fc44c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc46d600640 1 --2- 172.21.15.97:0/1332628656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc44c003fb0 0x7fc44c004800 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc458001750 tx=0x7fc44c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.281+0000 7fc46cc00640 1 -- 172.21.15.97:0/1332628656 reap_dead start 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.282+0000 7fc46d600640 1 -- 172.21.15.97:0/1332628656 shutdown_connections 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.282+0000 7fc46d600640 1 -- 172.21.15.97:0/1332628656 >> 172.21.15.97:0/1332628656 conn(0x7fc46809d070 msgr2=0x7fc4680d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.282+0000 7fc46d600640 1 -- 172.21.15.97:0/1332628656 shutdown_connections 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.282+0000 7fc46d600640 1 -- 172.21.15.97:0/1332628656 wait complete. 2024-03-06T07:29:21.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.283+0000 7fc46d600640 1 Processor -- start 2024-03-06T07:29:21.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 -- start start 2024-03-06T07:29:21.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c003fb0 0x7fc4680c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680b7540 0x7fc4680b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 0x7fc4680cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680d0ab0 con 0x7fc4680cc340 2024-03-06T07:29:21.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680b0960 con 0x7fc44c003fb0 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.284+0000 7fc46d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680d0c30 con 0x7fc4680b7540 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 0x7fc4680cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 0x7fc4680cc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40816/0 (socket says 172.21.15.97:40816) 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 -- 172.21.15.97:0/1490303930 learned_addr learned my addr 172.21.15.97:0/1490303930 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc467e00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c003fb0 0x7fc4680c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc45f400640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680b7540 0x7fc4680b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 -- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680b7540 msgr2=0x7fc4680b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680b7540 0x7fc4680b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 -- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c003fb0 msgr2=0x7fc4680c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c003fb0 0x7fc4680c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.285+0000 7fc46cc00640 1 -- 172.21.15.97:0/1490303930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468160ca0 con 0x7fc4680cc340 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.286+0000 7fc46cc00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 0x7fc4680cc7c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc46808c8f0 tx=0x7fc46000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.286+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000de90 con 0x7fc4680cc340 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.286+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46000bb80 con 0x7fc4680cc340 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.286+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460007070 con 0x7fc4680cc340 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.286+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc468160f80 con 0x7fc4680cc340 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.286+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc468161540 con 0x7fc4680cc340 2024-03-06T07:29:21.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.287+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46804f330 con 0x7fc4680cc340 2024-03-06T07:29:21.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.288+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc46001b020 con 0x7fc4680cc340 2024-03-06T07:29:21.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.289+0000 7fc464c00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc44406b670 0x7fc44406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.289+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc46008a3b0 con 0x7fc4680cc340 2024-03-06T07:29:21.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.289+0000 7fc467e00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc44406b670 0x7fc44406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.290+0000 7fc467e00640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc44406b670 0x7fc44406db30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc4500099f0 tx=0x7fc450008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.293+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc460059220 con 0x7fc4680cc340 2024-03-06T07:29:21.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.489+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4680cac40 con 0x7fc4680cc340 2024-03-06T07:29:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.490+0000 7fc464c00640 1 -- 172.21.15.97:0/1490303930 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc460058b20 con 0x7fc4680cc340 2024-03-06T07:29:21.491 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:29:21.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc44406b670 msgr2=0x7fc44406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc44406b670 0x7fc44406db30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc4500099f0 tx=0x7fc450008040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 msgr2=0x7fc4680cc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 0x7fc4680cc7c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc46808c8f0 tx=0x7fc46000b4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 shutdown_connections 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc44406b670 0x7fc44406db30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4680cc340 0x7fc4680cc7c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4680b7540 0x7fc4680b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 --2- 172.21.15.97:0/1490303930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c003fb0 0x7fc4680c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 >> 172.21.15.97:0/1490303930 conn(0x7fc46809d070 msgr2=0x7fc4680d2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.492+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 shutdown_connections 2024-03-06T07:29:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.493+0000 7fc46d600640 1 -- 172.21.15.97:0/1490303930 wait complete. 2024-03-06T07:29:21.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.702+0000 7ff3dfe00640 1 Processor -- start 2024-03-06T07:29:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.702+0000 7ff3dfe00640 1 -- start start 2024-03-06T07:29:21.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.704+0000 7ff3dfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff3e015e390 0x7ff3e0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.704+0000 7ff3dfe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e006ba70 con 0x7ff3e006c200 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.704+0000 7ff3dfe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e006ebf0 con 0x7ff3e0072c40 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.704+0000 7ff3dfe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e006e9c0 con 0x7ff3e015e390 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3dea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff3e015e390 0x7ff3e0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3d6000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff3e006c200 0x7ff3e0072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45030/0 (socket says 172.21.15.97:45030) 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3dea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff3e015e390 0x7ff3e0066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40832/0 (socket says 172.21.15.97:40832) 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3d6000640 1 -- 172.21.15.97:0/2271425333 learned_addr learned my addr 172.21.15.97:0/2271425333 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3dea00640 1 -- 172.21.15.97:0/2271425333 >> v1:172.21.15.135:6789/0 conn(0x7ff3e0072c40 legacy=0x7ff3e00cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3dea00640 1 -- 172.21.15.97:0/2271425333 >> v1:172.21.15.97:6789/0 conn(0x7ff3e006c200 legacy=0x7ff3e0072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.705+0000 7ff3dea00640 1 -- 172.21.15.97:0/2271425333 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e006eed0 con 0x7ff3e015e390 2024-03-06T07:29:21.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.706+0000 7ff3dea00640 1 --2- 172.21.15.97:0/2271425333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e015e390 0x7ff3e0066d00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00097a0 tx=0x7ff3d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0150032852e977e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3de000640 1 -- 172.21.15.97:0/2271425333 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0035020 con 0x7ff3e015e390 2024-03-06T07:29:21.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3de000640 1 -- 172.21.15.97:0/2271425333 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d0031830 con 0x7ff3e015e390 2024-03-06T07:29:21.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3de000640 1 -- 172.21.15.97:0/2271425333 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0031c00 con 0x7ff3e015e390 2024-03-06T07:29:21.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2271425333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e015e390 msgr2=0x7ff3e0066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2271425333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e015e390 0x7ff3e0066d00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00097a0 tx=0x7ff3d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2271425333 shutdown_connections 2024-03-06T07:29:21.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2271425333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e015e390 0x7ff3e0066d00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2271425333 >> 172.21.15.97:0/2271425333 conn(0x7ff3e00b1d00 msgr2=0x7ff3e00b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2271425333 shutdown_connections 2024-03-06T07:29:21.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.707+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2271425333 wait complete. 2024-03-06T07:29:21.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.709+0000 7ff3dfe00640 1 Processor -- start 2024-03-06T07:29:21.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.709+0000 7ff3dfe00640 1 -- start start 2024-03-06T07:29:21.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3dfe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e006c200 0x7ff3e00c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 0x7ff3e01f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3dfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e015e390 0x7ff3e01f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3dfe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00673c0 con 0x7ff3e0072c40 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3dfe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e0067240 con 0x7ff3e006c200 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3dfe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00b2a20 con 0x7ff3e015e390 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3d6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 0x7ff3e01f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3d6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 0x7ff3e01f8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40842/0 (socket says 172.21.15.97:40842) 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.710+0000 7ff3d6000640 1 -- 172.21.15.97:0/2731099053 learned_addr learned my addr 172.21.15.97:0/2731099053 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3df400640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e015e390 0x7ff3e01f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3dea00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e006c200 0x7ff3e00c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d6000640 1 -- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e015e390 msgr2=0x7ff3e01f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d6000640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e015e390 0x7ff3e01f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d6000640 1 -- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e006c200 msgr2=0x7ff3e00c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d6000640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e006c200 0x7ff3e00c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d6000640 1 -- 172.21.15.97:0/2731099053 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e01f87a0 con 0x7ff3e0072c40 2024-03-06T07:29:21.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3dea00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e006c200 0x7ff3e00c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d6000640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 0x7ff3e01f8260 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800c400 tx=0x7ff3c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8017070 con 0x7ff3e0072c40 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8005ce0 con 0x7ff3e0072c40 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80072f0 con 0x7ff3e0072c40 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.711+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e00cc740 con 0x7ff3e0072c40 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.712+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e00c5d50 con 0x7ff3e0072c40 2024-03-06T07:29:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.713+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e004f330 con 0x7ff3e0072c40 2024-03-06T07:29:21.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.714+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3c801a070 con 0x7ff3e0072c40 2024-03-06T07:29:21.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.715+0000 7ff3d7400640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3bc06b4d0 0x7ff3bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.715+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3c8089540 con 0x7ff3e0072c40 2024-03-06T07:29:21.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.715+0000 7ff3dea00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3bc06b4d0 0x7ff3bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.716+0000 7ff3dea00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3bc06b4d0 0x7ff3bc06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0036000 tx=0x7ff3d003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.720+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3c8058430 con 0x7ff3e0072c40 2024-03-06T07:29:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.916+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3e00af3e0 con 0x7ff3e0072c40 2024-03-06T07:29:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.917+0000 7ff3d7400640 1 -- 172.21.15.97:0/2731099053 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3c8057d30 con 0x7ff3e0072c40 2024-03-06T07:29:21.919 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3bc06b4d0 msgr2=0x7ff3bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3bc06b4d0 0x7ff3bc06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0036000 tx=0x7ff3d003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 msgr2=0x7ff3e01f8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 0x7ff3e01f8260 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800c400 tx=0x7ff3c800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 shutdown_connections 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3bc06b4d0 0x7ff3bc06d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e015e390 0x7ff3e01f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.919+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0072c40 0x7ff3e01f8260 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.920+0000 7ff3dfe00640 1 --2- 172.21.15.97:0/2731099053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e006c200 0x7ff3e00c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.920+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 >> 172.21.15.97:0/2731099053 conn(0x7ff3e00b1d00 msgr2=0x7ff3e0153cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.920+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 shutdown_connections 2024-03-06T07:29:21.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:21.920+0000 7ff3dfe00640 1 -- 172.21.15.97:0/2731099053 wait complete. 2024-03-06T07:29:21.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.154+0000 7fb862000640 1 Processor -- start 2024-03-06T07:29:22.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.154+0000 7fb862000640 1 -- start start 2024-03-06T07:29:22.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.155+0000 7fb862000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb85c15f9d0 0x7fb85c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.155+0000 7fb862000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c070b10 0x7fb85c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.155+0000 7fb862000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c06cef0 con 0x7fb85c144c00 2024-03-06T07:29:22.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.155+0000 7fb862000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c06d1d0 con 0x7fb85c15f9d0 2024-03-06T07:29:22.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.155+0000 7fb862000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c06d4b0 con 0x7fb85c070b10 2024-03-06T07:29:22.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb85b400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb85c144c00 0x7fb85c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45038/0 (socket says 172.21.15.97:45038) 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb85b400640 1 -- 172.21.15.97:0/2735578709 learned_addr learned my addr 172.21.15.97:0/2735578709 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb852a00640 1 --2- 172.21.15.97:0/2735578709 >> v2:172.21.15.97:3300/0 conn(0x7fb85c15f9d0 0x7fb85c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb85be00640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c070b10 0x7fb85c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb852a00640 1 -- 172.21.15.97:0/2735578709 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c070b10 msgr2=0x7fb85c070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb852a00640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c070b10 0x7fb85c070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb852a00640 1 -- 172.21.15.97:0/2735578709 >> v1:172.21.15.97:6789/0 conn(0x7fb85c144c00 legacy=0x7fb85c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb852a00640 1 -- 172.21.15.97:0/2735578709 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c06d790 con 0x7fb85c15f9d0 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.156+0000 7fb85be00640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c070b10 0x7fb85c070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb852a00640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c15f9d0 0x7fb85c15fe60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8440097a0 tx=0x7fb84402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d329ccbf71c779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb85aa00640 1 -- 172.21.15.97:0/2735578709 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844035020 con 0x7fb85c15f9d0 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb85aa00640 1 -- 172.21.15.97:0/2735578709 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb844031810 con 0x7fb85c15f9d0 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb85aa00640 1 -- 172.21.15.97:0/2735578709 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844031be0 con 0x7fb85c15f9d0 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb862000640 1 -- 172.21.15.97:0/2735578709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c15f9d0 msgr2=0x7fb85c15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb862000640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c15f9d0 0x7fb85c15fe60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8440097a0 tx=0x7fb84402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb862000640 1 -- 172.21.15.97:0/2735578709 shutdown_connections 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb862000640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c070b10 0x7fb85c070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb862000640 1 --2- 172.21.15.97:0/2735578709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c15f9d0 0x7fb85c15fe60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.158+0000 7fb862000640 1 -- 172.21.15.97:0/2735578709 >> 172.21.15.97:0/2735578709 conn(0x7fb85c0b1730 msgr2=0x7fb85c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.159+0000 7fb862000640 1 -- 172.21.15.97:0/2735578709 shutdown_connections 2024-03-06T07:29:22.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.159+0000 7fb862000640 1 -- 172.21.15.97:0/2735578709 wait complete. 2024-03-06T07:29:22.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.160+0000 7fb862000640 1 Processor -- start 2024-03-06T07:29:22.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.160+0000 7fb862000640 1 -- start start 2024-03-06T07:29:22.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb862000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 0x7fb85c13e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb862000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c144c00 0x7fb85c1395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb862000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb85c15f9d0 0x7fb85c13fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb862000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c06ccd0 con 0x7fb85c070b10 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb862000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c06cb50 con 0x7fb85c15f9d0 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb862000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c06df90 con 0x7fb85c144c00 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb85b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 0x7fb85c13e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.161+0000 7fb85b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 0x7fb85c13e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40862/0 (socket says 172.21.15.97:40862) 2024-03-06T07:29:22.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 -- 172.21.15.97:0/3252537473 learned_addr learned my addr 172.21.15.97:0/3252537473 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:22.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 -- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c144c00 msgr2=0x7fb85c1395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb852a00640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c144c00 0x7fb85c1395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85be00640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb85c15f9d0 0x7fb85c13fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c144c00 0x7fb85c1395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 -- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb85c15f9d0 msgr2=0x7fb85c13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb85c15f9d0 0x7fb85c13fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 -- 172.21.15.97:0/3252537473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c139b20 con 0x7fb85c070b10 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb85b400640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 0x7fb85c13e340 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00cb10 tx=0x7fb84c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.162+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c017070 con 0x7fb85c070b10 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.163+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c00ac70 con 0x7fb85c070b10 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.163+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c00ae60 con 0x7fb85c070b10 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.163+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c13d2a0 con 0x7fb85c070b10 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.163+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c13d860 con 0x7fb85c070b10 2024-03-06T07:29:22.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.164+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c04f330 con 0x7fb85c070b10 2024-03-06T07:29:22.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.165+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb84c01a070 con 0x7fb85c070b10 2024-03-06T07:29:22.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.166+0000 7fb853e00640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83806b4d0 0x7fb83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.166+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb84c089fc0 con 0x7fb85c070b10 2024-03-06T07:29:22.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.166+0000 7fb852a00640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83806b4d0 0x7fb83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.167+0000 7fb852a00640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83806b4d0 0x7fb83806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb844036000 tx=0x7fb844008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.171+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb85c139b20 con 0x7fb85c070b10 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.427+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb85c0cd270 con 0x7fb85c070b10 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.427+0000 7fb853e00640 1 -- 172.21.15.97:0/3252537473 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fb84c058e30 con 0x7fb85c070b10 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:22.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83806b4d0 msgr2=0x7fb83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83806b4d0 0x7fb83806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb844036000 tx=0x7fb844008030 comp rx=0 tx=0).stop 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 msgr2=0x7fb85c13e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 0x7fb85c13e340 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00cb10 tx=0x7fb84c00a060 comp rx=0 tx=0).stop 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 shutdown_connections 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb83806b4d0 0x7fb83806d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb85c15f9d0 0x7fb85c13fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb85c144c00 0x7fb85c1395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 --2- 172.21.15.97:0/3252537473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb85c070b10 0x7fb85c13e340 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 >> 172.21.15.97:0/3252537473 conn(0x7fb85c0b1730 msgr2=0x7fb85c0af7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 shutdown_connections 2024-03-06T07:29:22.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.430+0000 7fb862000640 1 -- 172.21.15.97:0/3252537473 wait complete. 2024-03-06T07:29:22.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.652+0000 7f5345200640 1 Processor -- start 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.652+0000 7f5345200640 1 -- start start 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f5345200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5340161580 0x7f53400d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f5345200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f53400d2a70 0x7f53400b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f5345200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5340161960 con 0x7f53400d1e90 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f5345200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400cf6f0 con 0x7f5340161580 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f5345200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400cf9d0 con 0x7f53400d2a70 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f533f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5340161580 0x7f53400d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f533f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5340161580 0x7f53400d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40866/0 (socket says 172.21.15.97:40866) 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f533f400640 1 -- 172.21.15.97:0/3506494950 learned_addr learned my addr 172.21.15.97:0/3506494950 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.654+0000 7f533fe00640 1 --2- 172.21.15.97:0/3506494950 >> v2:172.21.15.135:3300/0 conn(0x7f53400d2a70 0x7f53400b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.655+0000 7f533f400640 1 -- 172.21.15.97:0/3506494950 >> v2:172.21.15.135:3300/0 conn(0x7f53400d2a70 msgr2=0x7f53400b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.655+0000 7f533f400640 1 --2- 172.21.15.97:0/3506494950 >> v2:172.21.15.135:3300/0 conn(0x7f53400d2a70 0x7f53400b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.655+0000 7f533f400640 1 -- 172.21.15.97:0/3506494950 >> v1:172.21.15.135:6789/0 conn(0x7f53400d1e90 legacy=0x7f53400d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.655+0000 7f533f400640 1 -- 172.21.15.97:0/3506494950 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53400cfcb0 con 0x7f5340161580 2024-03-06T07:29:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.656+0000 7f533f400640 1 --2- 172.21.15.97:0/3506494950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400d1950 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f534008d180 tx=0x7f533002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07250926049b890 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.656+0000 7f533e000640 1 -- 172.21.15.97:0/3506494950 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330035020 con 0x7f5340161580 2024-03-06T07:29:22.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.656+0000 7f533e000640 1 -- 172.21.15.97:0/3506494950 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330031870 con 0x7f5340161580 2024-03-06T07:29:22.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.656+0000 7f533e000640 1 -- 172.21.15.97:0/3506494950 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330031c40 con 0x7f5340161580 2024-03-06T07:29:22.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 -- 172.21.15.97:0/3506494950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 msgr2=0x7f53400d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 --2- 172.21.15.97:0/3506494950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400d1950 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f534008d180 tx=0x7f533002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 -- 172.21.15.97:0/3506494950 shutdown_connections 2024-03-06T07:29:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 --2- 172.21.15.97:0/3506494950 >> v2:172.21.15.135:3300/0 conn(0x7f53400d2a70 0x7f53400b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 --2- 172.21.15.97:0/3506494950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400d1950 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 -- 172.21.15.97:0/3506494950 >> 172.21.15.97:0/3506494950 conn(0x7f53400d3d80 msgr2=0x7f53400d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 -- 172.21.15.97:0/3506494950 shutdown_connections 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.657+0000 7f5345200640 1 -- 172.21.15.97:0/3506494950 wait complete. 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.658+0000 7f5345200640 1 Processor -- start 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.659+0000 7f5345200640 1 -- start start 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f5345200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53400d1e90 0x7f53400cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f5345200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53400d2a70 0x7f53400b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f5345200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f5345200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400cf3a0 con 0x7f5340161580 2024-03-06T07:29:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f5345200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400b39a0 con 0x7f53400d2a70 2024-03-06T07:29:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f5345200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53400b4400 con 0x7f53400d1e90 2024-03-06T07:29:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40870/0 (socket says 172.21.15.97:40870) 2024-03-06T07:29:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 -- 172.21.15.97:0/2583832921 learned_addr learned my addr 172.21.15.97:0/2583832921 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:22.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533ea00640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53400d2a70 0x7f53400b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533f400640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53400d1e90 0x7f53400cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 -- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53400d1e90 msgr2=0x7f53400cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53400d1e90 0x7f53400cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 -- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53400d2a70 msgr2=0x7f53400b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.660+0000 7f533fe00640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53400d2a70 0x7f53400b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f533fe00640 1 -- 172.21.15.97:0/2583832921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53400cd600 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f533fe00640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400cd850 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f533800cd40 tx=0x7f533800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338003b90 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338003d30 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338004240 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53400b8120 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.661+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53400b86e0 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.663+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534004f330 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.664+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53380043e0 con 0x7f5340161580 2024-03-06T07:29:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.665+0000 7f5337400640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f532006b4d0 0x7f532006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.665+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f533808acf0 con 0x7f5340161580 2024-03-06T07:29:22.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.665+0000 7f533f400640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f532006b4d0 0x7f532006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.666+0000 7f533f400640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f532006b4d0 0x7f532006d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5330036000 tx=0x7f533003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.669+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f533808f070 con 0x7f5340161580 2024-03-06T07:29:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.865+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53400d3b60 con 0x7f5340161580 2024-03-06T07:29:22.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.868+0000 7f5337400640 1 -- 172.21.15.97:0/2583832921 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f5338059b70 con 0x7f5340161580 2024-03-06T07:29:22.870 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:22.870 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":35,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:52.884546+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.870+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f532006b4d0 msgr2=0x7f532006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.870+0000 7f5345200640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f532006b4d0 0x7f532006d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5330036000 tx=0x7f533003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.870+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 msgr2=0x7f53400cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.870+0000 7f5345200640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400cd850 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f533800cd40 tx=0x7f533800a660 comp rx=0 tx=0).stop 2024-03-06T07:29:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 shutdown_connections 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f532006b4d0 0x7f532006d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5340161580 0x7f53400cd850 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53400d2a70 0x7f53400b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 --2- 172.21.15.97:0/2583832921 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53400d1e90 0x7f53400cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 >> 172.21.15.97:0/2583832921 conn(0x7f53400d3d80 msgr2=0x7f534006c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 shutdown_connections 2024-03-06T07:29:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:22.871+0000 7f5345200640 1 -- 172.21.15.97:0/2583832921 wait complete. 2024-03-06T07:29:22.893 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:22.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:23.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.083+0000 7f0a27a00640 1 Processor -- start 2024-03-06T07:29:23.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.083+0000 7f0a27a00640 1 -- start start 2024-03-06T07:29:23.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a27a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a2015f470 0x7f0a200d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a27a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0a200d0df0 0x7f0a20160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a27a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200cea60 con 0x7f0a2015e0d0 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a27a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a200d1260 con 0x7f0a200d0df0 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a27a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a200d2f60 con 0x7f0a2015f470 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a25a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0a200d0df0 0x7f0a20160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a25a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0a200d0df0 0x7f0a20160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40874/0 (socket says 172.21.15.97:40874) 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.085+0000 7f0a25a00640 1 -- 172.21.15.97:0/1326517819 learned_addr learned my addr 172.21.15.97:0/1326517819 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.086+0000 7f0a17e00640 1 --2- 172.21.15.97:0/1326517819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a2015f470 0x7f0a200d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.086+0000 7f0a25a00640 1 -- 172.21.15.97:0/1326517819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a2015f470 msgr2=0x7f0a200d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.086+0000 7f0a25a00640 1 --2- 172.21.15.97:0/1326517819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a2015f470 0x7f0a200d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.086+0000 7f0a25a00640 1 -- 172.21.15.97:0/1326517819 >> v1:172.21.15.135:6789/0 conn(0x7f0a2015e0d0 legacy=0x7f0a200d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.086+0000 7f0a25a00640 1 -- 172.21.15.97:0/1326517819 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a200d2350 con 0x7f0a200d0df0 2024-03-06T07:29:23.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.087+0000 7f0a25a00640 1 --2- 172.21.15.97:0/1326517819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a200d0df0 0x7f0a20160dc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009e50 tx=0x7f0a1802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ba36923bc2d1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.087+0000 7f0a1fe00640 1 -- 172.21.15.97:0/1326517819 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1802cb00 con 0x7f0a200d0df0 2024-03-06T07:29:23.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.087+0000 7f0a1fe00640 1 -- 172.21.15.97:0/1326517819 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1802cca0 con 0x7f0a200d0df0 2024-03-06T07:29:23.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.087+0000 7f0a1fe00640 1 -- 172.21.15.97:0/1326517819 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18034610 con 0x7f0a200d0df0 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 -- 172.21.15.97:0/1326517819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a200d0df0 msgr2=0x7f0a20160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1326517819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a200d0df0 0x7f0a20160dc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009e50 tx=0x7f0a1802bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 -- 172.21.15.97:0/1326517819 shutdown_connections 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1326517819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a200d0df0 0x7f0a20160dc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1326517819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a2015f470 0x7f0a200d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 -- 172.21.15.97:0/1326517819 >> 172.21.15.97:0/1326517819 conn(0x7f0a200d3ec0 msgr2=0x7f0a200d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 -- 172.21.15.97:0/1326517819 shutdown_connections 2024-03-06T07:29:23.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.088+0000 7f0a27a00640 1 -- 172.21.15.97:0/1326517819 wait complete. 2024-03-06T07:29:23.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.090+0000 7f0a27a00640 1 Processor -- start 2024-03-06T07:29:23.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.090+0000 7f0a27a00640 1 -- start start 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a27a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a200d0df0 0x7f0a20159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a27a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a2015e0d0 0x7f0a200b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a27a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 0x7f0a200b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a27a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a200d2bf0 con 0x7f0a2015f470 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a27a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a200d2910 con 0x7f0a2015e0d0 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a27a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2015fb30 con 0x7f0a200d0df0 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a25a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 0x7f0a200b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a25a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 0x7f0a200b2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40876/0 (socket says 172.21.15.97:40876) 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.091+0000 7f0a25a00640 1 -- 172.21.15.97:0/1002648415 learned_addr learned my addr 172.21.15.97:0/1002648415 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:23.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a17e00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a2015e0d0 0x7f0a200b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25000640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a200d0df0 0x7f0a20159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25a00640 1 -- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a200d0df0 msgr2=0x7f0a20159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a200d0df0 0x7f0a20159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25a00640 1 -- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a2015e0d0 msgr2=0x7f0a200b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a2015e0d0 0x7f0a200b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25a00640 1 -- 172.21.15.97:0/1002648415 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a201f8780 con 0x7f0a2015f470 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25000640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a200d0df0 0x7f0a20159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a17e00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a2015e0d0 0x7f0a200b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a25a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 0x7f0a200b2010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009e50 tx=0x7f0a18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18043020 con 0x7f0a2015f470 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1802ce50 con 0x7f0a2015f470 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18009510 con 0x7f0a2015f470 2024-03-06T07:29:23.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.092+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a201f79c0 con 0x7f0a2015f470 2024-03-06T07:29:23.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.093+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a201f7f80 con 0x7f0a2015f470 2024-03-06T07:29:23.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.094+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2004f330 con 0x7f0a2015f470 2024-03-06T07:29:23.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.095+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a18046020 con 0x7f0a2015f470 2024-03-06T07:29:23.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.096+0000 7f0a1d600640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f09fc06b4d0 0x7f09fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.096+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a180aca30 con 0x7f0a2015f470 2024-03-06T07:29:23.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.096+0000 7f0a25000640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f09fc06b4d0 0x7f09fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.097+0000 7f0a25000640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f09fc06b4d0 0x7f09fc06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a10003eb0 tx=0x7f0a10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.100+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a1804d030 con 0x7f0a2015f470 2024-03-06T07:29:23.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.370+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a2008d920 con 0x7f0a2015f470 2024-03-06T07:29:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.371+0000 7f0a1d600640 1 -- 172.21.15.97:0/1002648415 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f0a1807b810 con 0x7f0a2015f470 2024-03-06T07:29:23.373 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:23.373 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:23.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f09fc06b4d0 msgr2=0x7f09fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f09fc06b4d0 0x7f09fc06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a10003eb0 tx=0x7f0a10009040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 msgr2=0x7f0a200b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 0x7f0a200b2010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009e50 tx=0x7f0a18004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 shutdown_connections 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f09fc06b4d0 0x7f09fc06d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a2015f470 0x7f0a200b2010 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a2015e0d0 0x7f0a200b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 --2- 172.21.15.97:0/1002648415 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0a200d0df0 0x7f0a20159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 >> 172.21.15.97:0/1002648415 conn(0x7f0a200d3ec0 msgr2=0x7f0a200b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 shutdown_connections 2024-03-06T07:29:23.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.374+0000 7f0a27a00640 1 -- 172.21.15.97:0/1002648415 wait complete. 2024-03-06T07:29:23.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:23.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.583+0000 7fd0c0a00640 1 Processor -- start 2024-03-06T07:29:23.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.583+0000 7fd0c0a00640 1 -- start start 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0c0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd0bc0b4680 0x7fd0bc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0c0a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd0bc072340 0x7fd0bc06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0c0a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc15b090 con 0x7fd0bc15acb0 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0c0a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc0b2a30 con 0x7fd0bc0b4680 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0c0a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc0b4360 con 0x7fd0bc072340 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0b9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd0bc15acb0 0x7fd0bc0b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45042/0 (socket says 172.21.15.97:45042) 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0b9e00640 1 -- 172.21.15.97:0/3101094361 learned_addr learned my addr 172.21.15.97:0/3101094361 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:23.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0b9400640 1 --2- 172.21.15.97:0/3101094361 >> v2:172.21.15.97:3300/0 conn(0x7fd0bc0b4680 0x7fd0bc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0ba800640 1 --2- 172.21.15.97:0/3101094361 >> v2:172.21.15.135:3300/0 conn(0x7fd0bc072340 0x7fd0bc06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0b8a00640 1 -- 172.21.15.97:0/3101094361 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985055842 0 0) 0x7fd0bc15b090 con 0x7fd0bc15acb0 2024-03-06T07:29:23.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.585+0000 7fd0b8a00640 1 -- 172.21.15.97:0/3101094361 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0a0003520 con 0x7fd0bc15acb0 2024-03-06T07:29:23.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.586+0000 7fd0b9400640 1 -- 172.21.15.97:0/3101094361 >> v2:172.21.15.135:3300/0 conn(0x7fd0bc072340 msgr2=0x7fd0bc06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.586+0000 7fd0b9400640 1 --2- 172.21.15.97:0/3101094361 >> v2:172.21.15.135:3300/0 conn(0x7fd0bc072340 0x7fd0bc06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.586+0000 7fd0b9400640 1 -- 172.21.15.97:0/3101094361 >> v1:172.21.15.97:6789/0 conn(0x7fd0bc15acb0 legacy=0x7fd0bc0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.586+0000 7fd0b9400640 1 -- 172.21.15.97:0/3101094361 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0b2cf0 con 0x7fd0bc0b4680 2024-03-06T07:29:23.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.587+0000 7fd0b9400640 1 --2- 172.21.15.97:0/3101094361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc06e4f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0b4e60 tx=0x7fd0a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2bcde64507dc3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.587+0000 7fd0b8a00640 1 -- 172.21.15.97:0/3101094361 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8035020 con 0x7fd0bc0b4680 2024-03-06T07:29:23.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.587+0000 7fd0b8a00640 1 -- 172.21.15.97:0/3101094361 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a8031830 con 0x7fd0bc0b4680 2024-03-06T07:29:23.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.587+0000 7fd0b8a00640 1 -- 172.21.15.97:0/3101094361 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8031c00 con 0x7fd0bc0b4680 2024-03-06T07:29:23.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3101094361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 msgr2=0x7fd0bc06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3101094361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc06e4f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0b4e60 tx=0x7fd0a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3101094361 shutdown_connections 2024-03-06T07:29:23.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3101094361 >> v2:172.21.15.135:3300/0 conn(0x7fd0bc072340 0x7fd0bc06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3101094361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc06e4f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3101094361 >> 172.21.15.97:0/3101094361 conn(0x7fd0bc0b1730 msgr2=0x7fd0bc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3101094361 shutdown_connections 2024-03-06T07:29:23.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.588+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3101094361 wait complete. 2024-03-06T07:29:23.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.590+0000 7fd0c0a00640 1 Processor -- start 2024-03-06T07:29:23.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.590+0000 7fd0c0a00640 1 -- start start 2024-03-06T07:29:23.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0c0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0bc072340 0x7fd0bc14e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0c0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc0ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0c0a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd0bc15acb0 0x7fd0bc15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0c0a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0b4010 con 0x7fd0bc0b4680 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0c0a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0b32b0 con 0x7fd0bc15acb0 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0c0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0ccfb0 con 0x7fd0bc072340 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0b9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc0ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0b9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc0ad890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40888/0 (socket says 172.21.15.97:40888) 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0b9400640 1 -- 172.21.15.97:0/3011391215 learned_addr learned my addr 172.21.15.97:0/3011391215 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0b9e00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0bc072340 0x7fd0bc14e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.592+0000 7fd0ba800640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd0bc15acb0 0x7fd0bc15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9400640 1 -- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0bc072340 msgr2=0x7fd0bc14e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9400640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0bc072340 0x7fd0bc14e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9400640 1 -- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd0bc15acb0 msgr2=0x7fd0bc15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9400640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd0bc15acb0 0x7fd0bc15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9400640 1 -- 172.21.15.97:0/3011391215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0cef50 con 0x7fd0bc0b4680 2024-03-06T07:29:23.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9e00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0bc072340 0x7fd0bc14e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b9400640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc0ad890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009890 tx=0x7fd0a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8041070 con 0x7fd0bc0b4680 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a803b920 con 0x7fd0bc0b4680 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc0cf230 con 0x7fd0bc0b4680 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8035040 con 0x7fd0bc0b4680 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.593+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc15d120 con 0x7fd0bc0b4680 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.594+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc0cc5d0 con 0x7fd0bc0b4680 2024-03-06T07:29:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.595+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0a8038020 con 0x7fd0bc0b4680 2024-03-06T07:29:23.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.596+0000 7fd0b2000640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd09c06b4d0 0x7fd09c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.596+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0a80abc30 con 0x7fd0bc0b4680 2024-03-06T07:29:23.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.597+0000 7fd0b9e00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd09c06b4d0 0x7fd09c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.597+0000 7fd0b9e00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd09c06b4d0 0x7fd09c06d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac005880 tx=0x7fd0ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.602+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0bc0cc5d0 con 0x7fd0bc0b4680 2024-03-06T07:29:23.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.832+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0bc04f330 con 0x7fd0bc0b4680 2024-03-06T07:29:23.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.833+0000 7fd0b2000640 1 -- 172.21.15.97:0/3011391215 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd0a802f2e0 con 0x7fd0bc0b4680 2024-03-06T07:29:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:23.836 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:51.345690+0000","modified":"2024-03-06T07:28:54.426245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5052":{"gid":5052,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/4130179477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4130179477},{"type":"v1","addr":"172.21.15.97:6837","nonce":4130179477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5052,"qdb_cluster":[5052]},"id":2}]} 2024-03-06T07:29:23.836 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-06T07:29:23.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd09c06b4d0 msgr2=0x7fd09c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd09c06b4d0 0x7fd09c06d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac005880 tx=0x7fd0ac009040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 msgr2=0x7fd0bc0ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc0ad890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009890 tx=0x7fd0a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 shutdown_connections 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd09c06b4d0 0x7fd09c06d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd0bc15acb0 0x7fd0bc15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0bc0b4680 0x7fd0bc0ad890 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 --2- 172.21.15.97:0/3011391215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0bc072340 0x7fd0bc14e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 >> 172.21.15.97:0/3011391215 conn(0x7fd0bc0b1730 msgr2=0x7fd0bc15e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 shutdown_connections 2024-03-06T07:29:23.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:23.837+0000 7fd0c0a00640 1 -- 172.21.15.97:0/3011391215 wait complete. 2024-03-06T07:29:23.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:24.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.068+0000 7f3cbbe00640 1 Processor -- start 2024-03-06T07:29:24.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.068+0000 7f3cbbe00640 1 -- start start 2024-03-06T07:29:24.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.069+0000 7f3cbbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3cbc0aedc0 0x7f3cbc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.069+0000 7f3cbbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc0cc800 0x7f3cbc15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.069+0000 7f3cbbe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc15c240 con 0x7f3cbc1561f0 2024-03-06T07:29:24.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.069+0000 7f3cbbe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cbc15ce30 con 0x7f3cbc0aedc0 2024-03-06T07:29:24.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.069+0000 7f3cbbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc15c500 con 0x7f3cbc0cc800 2024-03-06T07:29:24.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.070+0000 7f3cb3e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3cbc0aedc0 0x7f3cbc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.070+0000 7f3cbb400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc0cc800 0x7f3cbc15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.070+0000 7f3cbb400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc0cc800 0x7f3cbc15f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59038/0 (socket says 172.21.15.97:59038) 2024-03-06T07:29:24.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.070+0000 7f3cbb400640 1 -- 172.21.15.97:0/1859577785 learned_addr learned my addr 172.21.15.97:0/1859577785 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:24.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.071+0000 7f3cb3e00640 1 -- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc0cc800 msgr2=0x7f3cbc15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.071+0000 7f3cb3e00640 1 --2- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc0cc800 0x7f3cbc15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.071+0000 7f3cb3e00640 1 -- 172.21.15.97:0/1859577785 >> v1:172.21.15.135:6789/0 conn(0x7f3cbc1561f0 legacy=0x7f3cbc0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.071+0000 7f3cb3e00640 1 -- 172.21.15.97:0/1859577785 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc15c7e0 con 0x7f3cbc0aedc0 2024-03-06T07:29:24.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.072+0000 7f3cb3e00640 1 --2- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc0cc2c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40097a0 tx=0x7f3ca402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8752573f4074cb8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.072+0000 7f3cba000640 1 -- 172.21.15.97:0/1859577785 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4035020 con 0x7f3cbc0aedc0 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.072+0000 7f3cba000640 1 -- 172.21.15.97:0/1859577785 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca4031830 con 0x7f3cbc0aedc0 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.072+0000 7f3cba000640 1 -- 172.21.15.97:0/1859577785 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4031c00 con 0x7f3cbc0aedc0 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 -- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 msgr2=0x7f3cbc0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc0cc2c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40097a0 tx=0x7f3ca402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 -- 172.21.15.97:0/1859577785 shutdown_connections 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc0cc800 0x7f3cbc15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/1859577785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc0cc2c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 -- 172.21.15.97:0/1859577785 >> 172.21.15.97:0/1859577785 conn(0x7f3cbc06afc0 msgr2=0x7f3cbc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 -- 172.21.15.97:0/1859577785 shutdown_connections 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.073+0000 7f3cbbe00640 1 -- 172.21.15.97:0/1859577785 wait complete. 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.075+0000 7f3cbbe00640 1 Processor -- start 2024-03-06T07:29:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.075+0000 7f3cbbe00640 1 -- start start 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 0x7f3cbc1f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc1561f0 0x7f3cbc0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc1565d0 con 0x7f3cbc0cc800 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc0ccc70 con 0x7f3cbc0aedc0 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc156750 con 0x7f3cbc1561f0 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 0x7f3cbc1f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 0x7f3cbc1f3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39288/0 (socket says 172.21.15.97:39288) 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 -- 172.21.15.97:0/3872292522 learned_addr learned my addr 172.21.15.97:0/3872292522 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbaa00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbb400640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc1561f0 0x7f3cbc0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 -- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc1561f0 msgr2=0x7f3cbc0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc1561f0 0x7f3cbc0ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 -- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 msgr2=0x7f3cbc158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cb3e00640 1 -- 172.21.15.97:0/3872292522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc158e50 con 0x7f3cbc0cc800 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.076+0000 7f3cbaa00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.077+0000 7f3cb3e00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 0x7f3cbc1f3b80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40098c0 tx=0x7f3ca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.077+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4041070 con 0x7f3cbc0cc800 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.077+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca403b920 con 0x7f3cbc0cc800 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.077+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4035040 con 0x7f3cbc0cc800 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.077+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc159130 con 0x7f3cbc0cc800 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.077+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc154b20 con 0x7f3cbc0cc800 2024-03-06T07:29:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.078+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc04f330 con 0x7f3cbc0cc800 2024-03-06T07:29:24.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.079+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ca404c060 con 0x7f3cbc0cc800 2024-03-06T07:29:24.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.081+0000 7f3cb2a00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3c9806b500 0x7f3c9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.081+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ca40aba40 con 0x7f3cbc0cc800 2024-03-06T07:29:24.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.081+0000 7f3cbaa00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3c9806b500 0x7f3c9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.082+0000 7f3cbaa00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3c9806b500 0x7f3c9806d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0015e0 tx=0x7f3cac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.085+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ca4031800 con 0x7f3cbc0cc800 2024-03-06T07:29:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.326+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cbc0b2b10 con 0x7f3cbc0cc800 2024-03-06T07:29:24.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.327+0000 7f3cb2a00640 1 -- 172.21.15.97:0/3872292522 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f3ca407a950 con 0x7f3cbc0cc800 2024-03-06T07:29:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:24.330 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":35,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:28:52.884546+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:49.145821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:50.344612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.329+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3c9806b500 msgr2=0x7f3c9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.329+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3c9806b500 0x7f3c9806d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0015e0 tx=0x7f3cac009040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.329+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 msgr2=0x7f3cbc1f3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.329+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 0x7f3cbc1f3b80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40098c0 tx=0x7f3ca40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 shutdown_connections 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3c9806b500 0x7f3c9806d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cbc1561f0 0x7f3cbc0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cbc0cc800 0x7f3cbc1f3b80 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 --2- 172.21.15.97:0/3872292522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cbc0aedc0 0x7f3cbc158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 >> 172.21.15.97:0/3872292522 conn(0x7f3cbc06afc0 msgr2=0x7f3cbc15b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 shutdown_connections 2024-03-06T07:29:24.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.330+0000 7f3cbbe00640 1 -- 172.21.15.97:0/3872292522 wait complete. 2024-03-06T07:29:24.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:24.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.547+0000 7fa680600640 1 Processor -- start 2024-03-06T07:29:24.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.547+0000 7fa680600640 1 -- start start 2024-03-06T07:29:24.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa680600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 0x7fa6780c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa680600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6780c4d10 0x7fa6780c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa680600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa678145b30 con 0x7fa6780c56b0 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa680600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa678142ce0 con 0x7fa6780c4d10 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa680600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780cbca0 con 0x7fa6780c4930 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa67e600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa6780c56b0 0x7fa67806d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53468/0 (socket says 172.21.15.97:53468) 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa67e600640 1 -- 172.21.15.97:0/3629302151 learned_addr learned my addr 172.21.15.97:0/3629302151 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa67dc00640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 0x7fa6780c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.549+0000 7fa67d200640 1 --2- 172.21.15.97:0/3629302151 >> v2:172.21.15.135:3300/0 conn(0x7fa6780c4d10 0x7fa6780c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987661024 0 0) 0x7fa670001490 con 0x7fa6780c56b0 2024-03-06T07:29:24.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660003520 con 0x7fa6780c56b0 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2338060477 0 0) 0x7fa660003520 con 0x7fa6780c56b0 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 msgr2=0x7fa6780c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 0x7fa6780c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 >> v2:172.21.15.135:3300/0 conn(0x7fa6780c4d10 msgr2=0x7fa6780c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 --2- 172.21.15.97:0/3629302151 >> v2:172.21.15.135:3300/0 conn(0x7fa6780c4d10 0x7fa6780c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6780cbf80 con 0x7fa6780c56b0 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa670001bc0 con 0x7fa6780c56b0 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 >> v1:172.21.15.97:6789/0 conn(0x7fa6780c56b0 legacy=0x7fa67806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 0x7fa660004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa660004d40 0x7fa6600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa678142ce0 con 0x7fa660004d40 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780cbca0 con 0x7fa6600039b0 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.550+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000b9c0 con 0x7fa6780c4930 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67e600640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa660004d40 0x7fa6600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67dc00640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67e600640 1 -- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 msgr2=0x7fa660004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67e600640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c4930 0x7fa660004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67e600640 1 -- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6600039b0 msgr2=0x7fa660003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67e600640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.551+0000 7fa67e600640 1 -- 172.21.15.97:0/3629302151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660003520 con 0x7fa660004d40 2024-03-06T07:29:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.552+0000 7fa67e600640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa660004d40 0x7fa6600051c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa67000aa70 tx=0x7fa660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7713eaad1cda1f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.552+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670038070 con 0x7fa660004d40 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.552+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6700072f0 con 0x7fa660004d40 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.552+0000 7fa677e00640 1 -- 172.21.15.97:0/3629302151 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67000a690 con 0x7fa660004d40 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa680600640 1 -- 172.21.15.97:0/3629302151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa660004d40 msgr2=0x7fa6600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa680600640 1 --2- 172.21.15.97:0/3629302151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa660004d40 0x7fa6600051c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa67000aa70 tx=0x7fa660005830 comp rx=0 tx=0).stop 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa67e600640 1 -- 172.21.15.97:0/3629302151 reap_dead start 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa680600640 1 -- 172.21.15.97:0/3629302151 shutdown_connections 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa680600640 1 -- 172.21.15.97:0/3629302151 >> 172.21.15.97:0/3629302151 conn(0x7fa6780cc4f0 msgr2=0x7fa6780cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa680600640 1 -- 172.21.15.97:0/3629302151 shutdown_connections 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.553+0000 7fa680600640 1 -- 172.21.15.97:0/3629302151 wait complete. 2024-03-06T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.555+0000 7fa680600640 1 Processor -- start 2024-03-06T07:29:24.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.555+0000 7fa680600640 1 -- start start 2024-03-06T07:29:24.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa680600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa660004d40 0x7fa6780b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa680600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 0x7fa6781f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa680600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c8660 0x7fa6780b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa680600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780cc260 con 0x7fa6780b47f0 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa680600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780ca3b0 con 0x7fa660004d40 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa680600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780c5b40 con 0x7fa6780c8660 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa67d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 0x7fa6781f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa67d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 0x7fa6781f79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39298/0 (socket says 172.21.15.97:39298) 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa67d200640 1 -- 172.21.15.97:0/3470011886 learned_addr learned my addr 172.21.15.97:0/3470011886 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.556+0000 7fa67dc00640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa660004d40 0x7fa6780b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67e600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c8660 0x7fa6780b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67d200640 1 -- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c8660 msgr2=0x7fa6780b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67d200640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c8660 0x7fa6780b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67d200640 1 -- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa660004d40 msgr2=0x7fa6780b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67d200640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa660004d40 0x7fa6780b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67d200640 1 -- 172.21.15.97:0/3470011886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6780b5040 con 0x7fa6780b47f0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67e600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c8660 0x7fa6780b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa67d200640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 0x7fa6781f79b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa66400e7f0 tx=0x7fa66400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664017070 con 0x7fa6780b47f0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66400c940 con 0x7fa6780b47f0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66400cd30 con 0x7fa6780b47f0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.557+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6780b5320 con 0x7fa6780b47f0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.558+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6781f71b0 con 0x7fa6780b47f0 2024-03-06T07:29:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.559+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67804f330 con 0x7fa6780b47f0 2024-03-06T07:29:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.560+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa66401a070 con 0x7fa6780b47f0 2024-03-06T07:29:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.560+0000 7fa675600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa65806b4d0 0x7fa65806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.561+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa66408a4d0 con 0x7fa6780b47f0 2024-03-06T07:29:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.561+0000 7fa67dc00640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa65806b4d0 0x7fa65806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.561+0000 7fa67dc00640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa65806b4d0 0x7fa65806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa6781f1710 tx=0x7fa66c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.565+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa664059330 con 0x7fa6780b47f0 2024-03-06T07:29:24.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.799+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa6781453a0 con 0x7fa6780b47f0 2024-03-06T07:29:24.802 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:29:24.800+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:24.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.804+0000 7fa675600640 1 -- 172.21.15.97:0/3470011886 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa664058c30 con 0x7fa6780b47f0 2024-03-06T07:29:24.806 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa65806b4d0 msgr2=0x7fa65806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa65806b4d0 0x7fa65806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa6781f1710 tx=0x7fa66c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 msgr2=0x7fa6781f79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 0x7fa6781f79b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa66400e7f0 tx=0x7fa66400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 shutdown_connections 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa65806b4d0 0x7fa65806d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6780c8660 0x7fa6780b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6780b47f0 0x7fa6781f79b0 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa66400e7f0 tx=0x7fa66400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 --2- 172.21.15.97:0/3470011886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa660004d40 0x7fa6780b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.807+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 >> 172.21.15.97:0/3470011886 conn(0x7fa6780cc4f0 msgr2=0x7fa6780ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.808+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 shutdown_connections 2024-03-06T07:29:24.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:24.809+0000 7fa680600640 1 -- 172.21.15.97:0/3470011886 wait complete. 2024-03-06T07:29:24.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:24.892 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:29:25.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.015+0000 7f081f200640 1 Processor -- start 2024-03-06T07:29:25.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.015+0000 7f081f200640 1 -- start start 2024-03-06T07:29:25.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.016+0000 7f081f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f08180b3f70 0x7f0818160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.016+0000 7f081f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 0x7f08180ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.016+0000 7f081f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08180b46c0 con 0x7f081815fc10 2024-03-06T07:29:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.016+0000 7f081f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0818071480 con 0x7f08180b3f70 2024-03-06T07:29:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.016+0000 7f081f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08180b49a0 con 0x7f0818070ff0 2024-03-06T07:29:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f0817e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f081815fc10 0x7f081815fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53486/0 (socket says 172.21.15.97:53486) 2024-03-06T07:29:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f0817e00640 1 -- 172.21.15.97:0/638419486 learned_addr learned my addr 172.21.15.97:0/638419486 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720194914 0 0) 0x7f08180b46c0 con 0x7f081815fc10 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003520 con 0x7f081815fc10 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f081d200640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 0x7f08180ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f080f400640 1 --2- 172.21.15.97:0/638419486 >> v2:172.21.15.135:3300/0 conn(0x7f08180b3f70 0x7f0818160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.017+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623425573 0 0) 0x7f07fc003520 con 0x7f081815fc10 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 msgr2=0x7f08180ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 0x7f08180ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 >> v2:172.21.15.135:3300/0 conn(0x7f08180b3f70 msgr2=0x7f0818160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 --2- 172.21.15.97:0/638419486 >> v2:172.21.15.135:3300/0 conn(0x7f08180b3f70 0x7f0818160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0818072c90 con 0x7f081815fc10 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0808001670 con 0x7f081815fc10 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 >> v1:172.21.15.97:6789/0 conn(0x7f081815fc10 legacy=0x7f081815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc0039b0 0x7f07fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 0x7f07fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f07fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818071480 con 0x7f07fc004d40 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08180b49a0 con 0x7f07fc0039b0 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00ba40 con 0x7f0818070ff0 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f080f400640 1 --2- 172.21.15.97:0/638419486 >> v2:172.21.15.135:3300/0 conn(0x7f08180b3f70 0x7f0818160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f081d200640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f07fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f081d200640 1 -- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 msgr2=0x7f07fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f081d200640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0818070ff0 0x7f07fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f081d200640 1 -- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc0039b0 msgr2=0x7f07fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f0817e00640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc0039b0 0x7f07fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f081d200640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc0039b0 0x7f07fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.018+0000 7f081d200640 1 -- 172.21.15.97:0/638419486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc003520 con 0x7f07fc004d40 2024-03-06T07:29:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.019+0000 7f0817e00640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc0039b0 0x7f07fc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f081d200640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f07fc0051c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0810009fc0 tx=0x7f07fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f49f4f943d151e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810039070 con 0x7f07fc004d40 2024-03-06T07:29:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081002cc60 con 0x7f07fc004d40 2024-03-06T07:29:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f0817400640 1 -- 172.21.15.97:0/638419486 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810009c10 con 0x7f07fc004d40 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f081f200640 1 -- 172.21.15.97:0/638419486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 msgr2=0x7f07fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f081f200640 1 --2- 172.21.15.97:0/638419486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f07fc0051c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0810009fc0 tx=0x7f07fc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f081d200640 1 -- 172.21.15.97:0/638419486 reap_dead start 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f081f200640 1 -- 172.21.15.97:0/638419486 shutdown_connections 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.020+0000 7f081f200640 1 -- 172.21.15.97:0/638419486 >> 172.21.15.97:0/638419486 conn(0x7f08180b1940 msgr2=0x7f08180b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.021+0000 7f081f200640 1 -- 172.21.15.97:0/638419486 shutdown_connections 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.021+0000 7f081f200640 1 -- 172.21.15.97:0/638419486 wait complete. 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.022+0000 7f081f200640 1 Processor -- start 2024-03-06T07:29:25.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.022+0000 7f081f200640 1 -- start start 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.023+0000 7f081f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f081806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.023+0000 7f081f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0818139b20 0x7f081813a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.023+0000 7f081f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081813a900 0x7f081813c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.023+0000 7f081f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818072a30 con 0x7f07fc004d40 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.023+0000 7f081f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08180728b0 con 0x7f0818139b20 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.023+0000 7f081f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081815c2e0 con 0x7f081813a900 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f081806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f081806dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39320/0 (socket says 172.21.15.97:39320) 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 -- 172.21.15.97:0/1940051473 learned_addr learned my addr 172.21.15.97:0/1940051473 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f081d200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081813a900 0x7f081813c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f080f400640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0818139b20 0x7f081813a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 -- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081813a900 msgr2=0x7f081813c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081813a900 0x7f081813c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 -- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0818139b20 msgr2=0x7f081813a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0818139b20 0x7f081813a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 -- 172.21.15.97:0/1940051473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081815c5b0 con 0x7f07fc004d40 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f080f400640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0818139b20 0x7f081813a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f081d200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081813a900 0x7f081813c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.024+0000 7f0817e00640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f081806dd40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f080800d8b0 tx=0x7f080800dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.025+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808019070 con 0x7f07fc004d40 2024-03-06T07:29:25.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.025+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08080074e0 con 0x7f07fc004d40 2024-03-06T07:29:25.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.025+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808004a00 con 0x7f07fc004d40 2024-03-06T07:29:25.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.025+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081815c890 con 0x7f07fc004d40 2024-03-06T07:29:25.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.025+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081806d820 con 0x7f07fc004d40 2024-03-06T07:29:25.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.026+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081804f330 con 0x7f07fc004d40 2024-03-06T07:29:25.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.027+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f080801c070 con 0x7f07fc004d40 2024-03-06T07:29:25.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.028+0000 7f0814c00640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f406b500 0x7f07f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.028+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0808089db0 con 0x7f07fc004d40 2024-03-06T07:29:25.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.028+0000 7f080f400640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f406b500 0x7f07f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.029+0000 7f080f400640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f406b500 0x7f07f406d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08181f79e0 tx=0x7f0800008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.032+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f080808e070 con 0x7f07fc004d40 2024-03-06T07:29:25.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.272+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f081806ef30 con 0x7f07fc004d40 2024-03-06T07:29:25.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.813+0000 7f0814c00640 1 -- 172.21.15.97:0/1940051473 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0808058b30 con 0x7f07fc004d40 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.817+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f406b500 msgr2=0x7f07f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.817+0000 7f081f200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f406b500 0x7f07f406d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08181f79e0 tx=0x7f0800008040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.817+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 msgr2=0x7f081806dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.817+0000 7f081f200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f081806dd40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f080800d8b0 tx=0x7f080800dd70 comp rx=0 tx=0).stop 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 shutdown_connections 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f406b500 0x7f07f406d9c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081813a900 0x7f081813c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0818139b20 0x7f081813a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 --2- 172.21.15.97:0/1940051473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07fc004d40 0x7f081806dd40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 >> 172.21.15.97:0/1940051473 conn(0x7f08180b1940 msgr2=0x7f08180b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 shutdown_connections 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:25.818+0000 7f081f200640 1 -- 172.21.15.97:0/1940051473 wait complete. 2024-03-06T07:29:25.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.052+0000 7f3267400640 1 Processor -- start 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.052+0000 7f3267400640 1 -- start start 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.053+0000 7f3267400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f32600b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.053+0000 7f3267400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f32600b3070 0x7f32600cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3267400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32600cefd0 0x7f326015e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3267400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32601405c0 con 0x7f32600cefd0 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3267400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32601408a0 con 0x7f32600b3070 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3267400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3260140b80 con 0x7f326013dd40 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32600cefd0 0x7f326015e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32600cefd0 0x7f326015e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39332/0 (socket says 172.21.15.97:39332) 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 -- 172.21.15.97:0/2798178367 learned_addr learned my addr 172.21.15.97:0/2798178367 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3264a00640 1 --2- 172.21.15.97:0/2798178367 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f32600b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f325fe00640 1 --2- 172.21.15.97:0/2798178367 >> v2:172.21.15.135:3300/0 conn(0x7f32600b3070 0x7f32600cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 -- 172.21.15.97:0/2798178367 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 msgr2=0x7f32600b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 --2- 172.21.15.97:0/2798178367 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f32600b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 -- 172.21.15.97:0/2798178367 >> v2:172.21.15.135:3300/0 conn(0x7f32600b3070 msgr2=0x7f32600cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 --2- 172.21.15.97:0/2798178367 >> v2:172.21.15.135:3300/0 conn(0x7f32600b3070 0x7f32600cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f3265400640 1 -- 172.21.15.97:0/2798178367 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260140e60 con 0x7f32600cefd0 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.054+0000 7f325fe00640 1 --2- 172.21.15.97:0/2798178367 >> v2:172.21.15.135:3300/0 conn(0x7f32600b3070 0x7f32600cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.056+0000 7f3265400640 1 --2- 172.21.15.97:0/2798178367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326015e9b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f326006c580 tx=0x7f325802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555c64d0744d33ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.056+0000 7f325f400640 1 -- 172.21.15.97:0/2798178367 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325802ca70 con 0x7f32600cefd0 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.056+0000 7f325f400640 1 -- 172.21.15.97:0/2798178367 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325802cc10 con 0x7f32600cefd0 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.056+0000 7f325f400640 1 -- 172.21.15.97:0/2798178367 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580343f0 con 0x7f32600cefd0 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.056+0000 7f3267400640 1 -- 172.21.15.97:0/2798178367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 msgr2=0x7f326015e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 --2- 172.21.15.97:0/2798178367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326015e9b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f326006c580 tx=0x7f325802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 -- 172.21.15.97:0/2798178367 shutdown_connections 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 --2- 172.21.15.97:0/2798178367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326015e9b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 --2- 172.21.15.97:0/2798178367 >> v2:172.21.15.135:3300/0 conn(0x7f32600b3070 0x7f32600cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 --2- 172.21.15.97:0/2798178367 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f32600b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 -- 172.21.15.97:0/2798178367 >> 172.21.15.97:0/2798178367 conn(0x7f32600cfa80 msgr2=0x7f32600cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 -- 172.21.15.97:0/2798178367 shutdown_connections 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.057+0000 7f3267400640 1 -- 172.21.15.97:0/2798178367 wait complete. 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.059+0000 7f3267400640 1 Processor -- start 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.059+0000 7f3267400640 1 -- start start 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f3267400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32600b3070 0x7f326013bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f3267400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326013d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f3267400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f3260156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f3267400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600b35b0 con 0x7f32600cefd0 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f3267400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260141380 con 0x7f32600b3070 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f3267400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600b3730 con 0x7f326013dd40 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f325fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326013d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f325fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326013d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39334/0 (socket says 172.21.15.97:39334) 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.060+0000 7f325fe00640 1 -- 172.21.15.97:0/2099664389 learned_addr learned my addr 172.21.15.97:0/2099664389 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f3265400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f3260156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f3264a00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32600b3070 0x7f326013bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f325fe00640 1 -- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 msgr2=0x7f3260156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f325fe00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f3260156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f325fe00640 1 -- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32600b3070 msgr2=0x7f326013bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f325fe00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32600b3070 0x7f326013bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f325fe00640 1 -- 172.21.15.97:0/2099664389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260154cb0 con 0x7f32600cefd0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f3264a00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32600b3070 0x7f326013bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f3265400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f3260156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.061+0000 7f325fe00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326013d170 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c820 tx=0x7f324c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.062+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c015070 con 0x7f32600cefd0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.062+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c007a30 con 0x7f32600cefd0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.062+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c007e00 con 0x7f32600cefd0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.062+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3260154f50 con 0x7f32600cefd0 2024-03-06T07:29:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.062+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3260155510 con 0x7f32600cefd0 2024-03-06T07:29:26.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.063+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326004f3a0 con 0x7f32600cefd0 2024-03-06T07:29:26.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.064+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f324c018070 con 0x7f32600cefd0 2024-03-06T07:29:26.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.065+0000 7f325cc00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f324006b4d0 0x7f324006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.065+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f324c08a240 con 0x7f32600cefd0 2024-03-06T07:29:26.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.065+0000 7f3264a00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f324006b4d0 0x7f324006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.066+0000 7f3264a00640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f324006b4d0 0x7f324006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f32600b1d50 tx=0x7f3254008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.069+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f324c058fc0 con 0x7f32600cefd0 2024-03-06T07:29:26.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:26.269+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f32600d0900 con 0x7f32600cefd0 2024-03-06T07:29:27.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.083+0000 7f325cc00640 1 -- 172.21.15.97:0/2099664389 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v37) v1 ==== 181+0+0 (secure 0 0 0) 0x7f324c0588c0 con 0x7f32600cefd0 2024-03-06T07:29:27.085 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:29:27.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.087+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f324006b4d0 msgr2=0x7f324006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.087+0000 7f3267400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f324006b4d0 0x7f324006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f32600b1d50 tx=0x7f3254008040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.087+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 msgr2=0x7f326013d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.087+0000 7f3267400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326013d170 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c820 tx=0x7f324c00cc20 comp rx=0 tx=0).stop 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 shutdown_connections 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f324006b4d0 0x7f324006d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f326013dd40 0x7f3260156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32600cefd0 0x7f326013d170 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 --2- 172.21.15.97:0/2099664389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32600b3070 0x7f326013bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 >> 172.21.15.97:0/2099664389 conn(0x7f32600cfa80 msgr2=0x7f3260153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.088+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 shutdown_connections 2024-03-06T07:29:27.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.089+0000 7f3267400640 1 -- 172.21.15.97:0/2099664389 wait complete. 2024-03-06T07:29:27.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:29:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.307+0000 7f9fecc00640 1 Processor -- start 2024-03-06T07:29:27.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.307+0000 7f9fecc00640 1 -- start start 2024-03-06T07:29:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.309+0000 7f9fecc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe80716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.309+0000 7f9fecc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9fe814bda0 0x7f9fe806ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.309+0000 7f9fecc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe814bbb0 con 0x7f9fe806cf40 2024-03-06T07:29:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.309+0000 7f9fecc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe806d390 con 0x7f9fe814bda0 2024-03-06T07:29:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.309+0000 7f9fecc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe80725e0 con 0x7f9fe814b7d0 2024-03-06T07:29:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.310+0000 7f9fdd400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9fe814bda0 0x7f9fe806ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.310+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe80716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.310+0000 7f9fdd400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9fe814bda0 0x7f9fe806ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:60258/0 (socket says 172.21.15.97:60258) 2024-03-06T07:29:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.310+0000 7f9fdd400640 1 -- 172.21.15.97:0/1353378591 learned_addr learned my addr 172.21.15.97:0/1353378591 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:27.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.311+0000 7f9fdd400640 1 -- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 msgr2=0x7f9fe80716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.311+0000 7f9fdd400640 1 --2- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe80716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.311+0000 7f9fdd400640 1 -- 172.21.15.97:0/1353378591 >> v1:172.21.15.135:6789/0 conn(0x7f9fe806cf40 legacy=0x7f9fe8165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.311+0000 7f9fdd400640 1 -- 172.21.15.97:0/1353378591 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe814c200 con 0x7f9fe814bda0 2024-03-06T07:29:27.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.312+0000 7f9fdd400640 1 --2- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 0x7f9fe806ca00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009870 tx=0x7f9fd002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d12828c60280196 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fe5400640 1 -- 172.21.15.97:0/1353378591 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0035020 con 0x7f9fe814bda0 2024-03-06T07:29:27.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fe5400640 1 -- 172.21.15.97:0/1353378591 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd0031640 con 0x7f9fe814bda0 2024-03-06T07:29:27.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fe5400640 1 -- 172.21.15.97:0/1353378591 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd002d040 con 0x7f9fe814bda0 2024-03-06T07:29:27.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 -- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 msgr2=0x7f9fe806ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 0x7f9fe806ca00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009870 tx=0x7f9fd002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 -- 172.21.15.97:0/1353378591 shutdown_connections 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 0x7f9fe806ca00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1353378591 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe80716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 -- 172.21.15.97:0/1353378591 >> 172.21.15.97:0/1353378591 conn(0x7f9fe80cfa80 msgr2=0x7f9fe80cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.313+0000 7f9fecc00640 1 -- 172.21.15.97:0/1353378591 shutdown_connections 2024-03-06T07:29:27.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.314+0000 7f9fecc00640 1 -- 172.21.15.97:0/1353378591 wait complete. 2024-03-06T07:29:27.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.315+0000 7f9fecc00640 1 Processor -- start 2024-03-06T07:29:27.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.315+0000 7f9fecc00640 1 -- start start 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.316+0000 7f9fecc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 0x7f9fe8145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.316+0000 7f9fecc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe81eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.316+0000 7f9fecc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 0x7f9fe81ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.316+0000 7f9fecc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe8072300 con 0x7f9fe806cf40 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.316+0000 7f9fecc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe8072020 con 0x7f9fe814bda0 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.316+0000 7f9fecc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe80b2c60 con 0x7f9fe814b7d0 2024-03-06T07:29:27.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 0x7f9fe8145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 0x7f9fe8145ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39342/0 (socket says 172.21.15.97:39342) 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 -- 172.21.15.97:0/1558945539 learned_addr learned my addr 172.21.15.97:0/1558945539 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 -- 172.21.15.97:0/1558945539 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 msgr2=0x7f9fe81eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe81eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 -- 172.21.15.97:0/1558945539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 msgr2=0x7f9fe81ebbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 0x7f9fe81ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 -- 172.21.15.97:0/1558945539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe815ffb0 con 0x7f9fe806cf40 2024-03-06T07:29:27.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.317+0000 7f9fe5e00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 0x7f9fe8145ab0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800e870 tx=0x7f9fd800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.318+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8017070 con 0x7f9fe806cf40 2024-03-06T07:29:27.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.318+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd800c940 con 0x7f9fe806cf40 2024-03-06T07:29:27.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.318+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd800cd30 con 0x7f9fe806cf40 2024-03-06T07:29:27.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.318+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe81ec1c0 con 0x7f9fe806cf40 2024-03-06T07:29:27.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.318+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe81ec780 con 0x7f9fe806cf40 2024-03-06T07:29:27.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.319+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe804f330 con 0x7f9fe806cf40 2024-03-06T07:29:27.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.320+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fd801a070 con 0x7f9fe806cf40 2024-03-06T07:29:27.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.321+0000 7f9fdea00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fc406b4d0 0x7f9fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.321+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f9fd8089f20 con 0x7f9fe806cf40 2024-03-06T07:29:27.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.321+0000 7f9fdd400640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fc406b4d0 0x7f9fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.322+0000 7f9fdd400640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fc406b4d0 0x7f9fc406d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0036000 tx=0x7f9fd0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.326+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fd8058ef0 con 0x7f9fe806cf40 2024-03-06T07:29:27.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:27.525+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9fe8165ac0 con 0x7f9fe806cf40 2024-03-06T07:29:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.094+0000 7f9fdea00640 1 -- 172.21.15.97:0/1558945539 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v38) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9fd80587f0 con 0x7f9fe806cf40 2024-03-06T07:29:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:29:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.097+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fc406b4d0 msgr2=0x7f9fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.097+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fc406b4d0 0x7f9fc406d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0036000 tx=0x7f9fd0030000 comp rx=0 tx=0).stop 2024-03-06T07:29:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.097+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 msgr2=0x7f9fe8145ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.097+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 0x7f9fe8145ab0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800e870 tx=0x7f9fd800ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 shutdown_connections 2024-03-06T07:29:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fc406b4d0 0x7f9fc406d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fe814bda0 0x7f9fe81ebbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fe814b7d0 0x7f9fe81eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 --2- 172.21.15.97:0/1558945539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fe806cf40 0x7f9fe8145ab0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 >> 172.21.15.97:0/1558945539 conn(0x7f9fe80cfa80 msgr2=0x7f9fe80ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 shutdown_connections 2024-03-06T07:29:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.098+0000 7f9fecc00640 1 -- 172.21.15.97:0/1558945539 wait complete. 2024-03-06T07:29:28.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:29:28.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.313+0000 7f491f600640 1 Processor -- start 2024-03-06T07:29:28.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.314+0000 7f491f600640 1 -- start start 2024-03-06T07:29:28.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f491f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49181618e0 0x7f49180ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f491f600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49180b2d80 0x7f49180cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f491f600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f49180cf520 0x7f49180d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f491f600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49180d0d20 con 0x7f49180b2d80 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f491f600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49180cf9b0 con 0x7f49180cf520 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f491f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49180b32f0 con 0x7f49181618e0 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.315+0000 7f4917e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49180b2d80 0x7f49180cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49180b2d80 0x7f49180cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39356/0 (socket says 172.21.15.97:39356) 2024-03-06T07:29:28.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 -- 172.21.15.97:0/1772634818 learned_addr learned my addr 172.21.15.97:0/1772634818 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f491d600640 1 --2- 172.21.15.97:0/1772634818 >> v2:172.21.15.135:3300/0 conn(0x7f49180cf520 0x7f49180d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f491cc00640 1 --2- 172.21.15.97:0/1772634818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49181618e0 0x7f49180ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 -- 172.21.15.97:0/1772634818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49181618e0 msgr2=0x7f49180ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 --2- 172.21.15.97:0/1772634818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49181618e0 0x7f49180ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 -- 172.21.15.97:0/1772634818 >> v2:172.21.15.135:3300/0 conn(0x7f49180cf520 msgr2=0x7f49180d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 --2- 172.21.15.97:0/1772634818 >> v2:172.21.15.135:3300/0 conn(0x7f49180cf520 0x7f49180d1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.316+0000 7f4917e00640 1 -- 172.21.15.97:0/1772634818 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180d0f70 con 0x7f49180b2d80 2024-03-06T07:29:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.317+0000 7f4917e00640 1 --2- 172.21.15.97:0/1772634818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49180b2d80 0x7f49180cefe0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4904009870 tx=0x7f490402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6129d9970072d68f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.317+0000 7f4917400640 1 -- 172.21.15.97:0/1772634818 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49040317c0 con 0x7f49180b2d80 2024-03-06T07:29:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.317+0000 7f4917400640 1 -- 172.21.15.97:0/1772634818 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904037070 con 0x7f49180b2d80 2024-03-06T07:29:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.317+0000 7f4917400640 1 -- 172.21.15.97:0/1772634818 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904031b80 con 0x7f49180b2d80 2024-03-06T07:29:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 -- 172.21.15.97:0/1772634818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49180b2d80 msgr2=0x7f49180cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 --2- 172.21.15.97:0/1772634818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49180b2d80 0x7f49180cefe0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4904009870 tx=0x7f490402b860 comp rx=0 tx=0).stop 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 -- 172.21.15.97:0/1772634818 shutdown_connections 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 --2- 172.21.15.97:0/1772634818 >> v2:172.21.15.135:3300/0 conn(0x7f49180cf520 0x7f49180d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 --2- 172.21.15.97:0/1772634818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49180b2d80 0x7f49180cefe0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 --2- 172.21.15.97:0/1772634818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49181618e0 0x7f49180ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 -- 172.21.15.97:0/1772634818 >> 172.21.15.97:0/1772634818 conn(0x7f49180d3e90 msgr2=0x7f49180d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 -- 172.21.15.97:0/1772634818 shutdown_connections 2024-03-06T07:29:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.318+0000 7f491f600640 1 -- 172.21.15.97:0/1772634818 wait complete. 2024-03-06T07:29:28.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.320+0000 7f491f600640 1 Processor -- start 2024-03-06T07:29:28.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.320+0000 7f491f600640 1 -- start start 2024-03-06T07:29:28.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491f600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49180b2d80 0x7f4918150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49180cf520 0x7f49181f1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491f600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 0x7f49180ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491f600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4918161cc0 con 0x7f49181618e0 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491f600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180d1530 con 0x7f49180b2d80 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49180d22d0 con 0x7f49180cf520 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 0x7f49180ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 0x7f49180ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39368/0 (socket says 172.21.15.97:39368) 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491d600640 1 -- 172.21.15.97:0/1025134695 learned_addr learned my addr 172.21.15.97:0/1025134695 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f4917e00640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49180cf520 0x7f49181f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.321+0000 7f491cc00640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49180b2d80 0x7f4918150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491d600640 1 -- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49180cf520 msgr2=0x7f49181f1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491d600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49180cf520 0x7f49181f1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491d600640 1 -- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49180b2d80 msgr2=0x7f4918150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491d600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49180b2d80 0x7f4918150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491d600640 1 -- 172.21.15.97:0/1025134695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180c7e10 con 0x7f49181618e0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491cc00640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49180b2d80 0x7f4918150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491d600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 0x7f49180ca260 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f491000cd20 tx=0x7f491000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910003b90 con 0x7f49181618e0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4910003d30 con 0x7f49181618e0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49180c7870 con 0x7f49181618e0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910004240 con 0x7f49181618e0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.322+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49180b85c0 con 0x7f49181618e0 2024-03-06T07:29:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.323+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49180b83d0 con 0x7f49181618e0 2024-03-06T07:29:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.325+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49100043e0 con 0x7f49181618e0 2024-03-06T07:29:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.325+0000 7f4914c00640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48f806b4d0 0x7f48f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.326+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f491008aa80 con 0x7f49181618e0 2024-03-06T07:29:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.326+0000 7f491cc00640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48f806b4d0 0x7f48f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.326+0000 7f491cc00640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48f806b4d0 0x7f48f806d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f490c00b820 tx=0x7f490c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.330+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f491008ce20 con 0x7f49181618e0 2024-03-06T07:29:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.526+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f491804f330 con 0x7f49181618e0 2024-03-06T07:29:28.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.532+0000 7f4914c00640 1 -- 172.21.15.97:0/1025134695 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4910059b60 con 0x7f49181618e0 2024-03-06T07:29:28.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48f806b4d0 msgr2=0x7f48f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48f806b4d0 0x7f48f806d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f490c00b820 tx=0x7f490c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:28.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 msgr2=0x7f49180ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 0x7f49180ca260 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f491000cd20 tx=0x7f491000a660 comp rx=0 tx=0).stop 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 shutdown_connections 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48f806b4d0 0x7f48f806d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49181618e0 0x7f49180ca260 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49180cf520 0x7f49181f1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 --2- 172.21.15.97:0/1025134695 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49180b2d80 0x7f4918150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 >> 172.21.15.97:0/1025134695 conn(0x7f49180d3e90 msgr2=0x7f491806c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 shutdown_connections 2024-03-06T07:29:28.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.535+0000 7f491f600640 1 -- 172.21.15.97:0/1025134695 wait complete. 2024-03-06T07:29:28.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:29:28.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.758+0000 7f4c7de00640 1 Processor -- start 2024-03-06T07:29:28.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.758+0000 7f4c7de00640 1 -- start start 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cf370 0x7f4c7806d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4c780cc2a0 0x7f4c780cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7de00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780b2b30 con 0x7f4c7815a940 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7de00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c780b28e0 con 0x7f4c780cc2a0 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7de00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c780b2e10 con 0x7f4c780cf370 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4c780cc2a0 0x7f4c780cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4c780cc2a0 0x7f4c780cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39384/0 (socket says 172.21.15.97:39384) 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c7d400640 1 -- 172.21.15.97:0/2399273690 learned_addr learned my addr 172.21.15.97:0/2399273690 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.760+0000 7f4c77e00640 1 --2- 172.21.15.97:0/2399273690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cf370 0x7f4c7806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.761+0000 7f4c7d400640 1 -- 172.21.15.97:0/2399273690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cf370 msgr2=0x7f4c7806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.761+0000 7f4c7d400640 1 --2- 172.21.15.97:0/2399273690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cf370 0x7f4c7806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.761+0000 7f4c7d400640 1 -- 172.21.15.97:0/2399273690 >> v1:172.21.15.135:6789/0 conn(0x7f4c7815a940 legacy=0x7f4c780b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.761+0000 7f4c7d400640 1 -- 172.21.15.97:0/2399273690 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7806ca50 con 0x7f4c780cc2a0 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.762+0000 7f4c7d400640 1 --2- 172.21.15.97:0/2399273690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cc2a0 0x7f4c780cc750 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009870 tx=0x7f4c6802db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e410fac09d5b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.762+0000 7f4c77400640 1 -- 172.21.15.97:0/2399273690 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6803b070 con 0x7f4c780cc2a0 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.762+0000 7f4c77400640 1 -- 172.21.15.97:0/2399273690 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6802e940 con 0x7f4c780cc2a0 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.762+0000 7f4c77400640 1 -- 172.21.15.97:0/2399273690 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6802ed10 con 0x7f4c780cc2a0 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 -- 172.21.15.97:0/2399273690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cc2a0 msgr2=0x7f4c780cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 --2- 172.21.15.97:0/2399273690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cc2a0 0x7f4c780cc750 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009870 tx=0x7f4c6802db90 comp rx=0 tx=0).stop 2024-03-06T07:29:28.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 -- 172.21.15.97:0/2399273690 shutdown_connections 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 --2- 172.21.15.97:0/2399273690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cc2a0 0x7f4c780cc750 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 --2- 172.21.15.97:0/2399273690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cf370 0x7f4c7806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 -- 172.21.15.97:0/2399273690 >> 172.21.15.97:0/2399273690 conn(0x7f4c7806afc0 msgr2=0x7f4c7806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 -- 172.21.15.97:0/2399273690 shutdown_connections 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.763+0000 7f4c7de00640 1 -- 172.21.15.97:0/2399273690 wait complete. 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.765+0000 7f4c7de00640 1 Processor -- start 2024-03-06T07:29:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.765+0000 7f4c7de00640 1 -- start start 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.765+0000 7f4c7de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cc2a0 0x7f4c781574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c7de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 0x7f4c781f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c7de00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c7815a940 0x7f4c78147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c7de00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7806d790 con 0x7f4c780cf370 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c7de00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c780b30f0 con 0x7f4c7815a940 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c7de00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c78147420 con 0x7f4c780cc2a0 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 0x7f4c781f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 0x7f4c781f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39392/0 (socket says 172.21.15.97:39392) 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 -- 172.21.15.97:0/3331749645 learned_addr learned my addr 172.21.15.97:0/3331749645 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c7ca00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cc2a0 0x7f4c781574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 -- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cc2a0 msgr2=0x7f4c781574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cc2a0 0x7f4c781574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 -- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c7815a940 msgr2=0x7f4c78147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c7815a940 0x7f4c78147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.766+0000 7f4c77e00640 1 -- 172.21.15.97:0/3331749645 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c781e4640 con 0x7f4c780cf370 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.767+0000 7f4c77e00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 0x7f4c781f79f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000d800 tx=0x7f4c6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.767+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000b6c0 con 0x7f4c780cf370 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.767+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c60019070 con 0x7f4c780cf370 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.767+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000ba80 con 0x7f4c780cf370 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.767+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c781e4920 con 0x7f4c780cf370 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.767+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c781f7f30 con 0x7f4c780cf370 2024-03-06T07:29:28.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.768+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7804f330 con 0x7f4c780cf370 2024-03-06T07:29:28.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.769+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c6000bc20 con 0x7f4c780cf370 2024-03-06T07:29:28.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.770+0000 7f4c74c00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5406b500 0x7f4c5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.770+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4c6008a3d0 con 0x7f4c780cf370 2024-03-06T07:29:28.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.770+0000 7f4c7ca00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5406b500 0x7f4c5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.771+0000 7f4c7ca00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5406b500 0x7f4c5406d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4c70001d40 tx=0x7f4c70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:28.775+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c6001c360 con 0x7f4c780cf370 2024-03-06T07:29:29.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.037+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f4c78157c60 con 0x7f4c780cf370 2024-03-06T07:29:29.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.097+0000 7f4c74c00640 1 -- 172.21.15.97:0/3331749645 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v118) v1 ==== 132+0+0 (secure 0 0 0) 0x7f4c60059540 con 0x7f4c780cf370 2024-03-06T07:29:29.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5406b500 msgr2=0x7f4c5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5406b500 0x7f4c5406d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4c70001d40 tx=0x7f4c70009040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 msgr2=0x7f4c781f79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 0x7f4c781f79f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000d800 tx=0x7f4c6000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 shutdown_connections 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5406b500 0x7f4c5406d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c7815a940 0x7f4c78147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c780cf370 0x7f4c781f79f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 --2- 172.21.15.97:0/3331749645 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c780cc2a0 0x7f4c781574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 >> 172.21.15.97:0/3331749645 conn(0x7f4c7806afc0 msgr2=0x7f4c7806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 shutdown_connections 2024-03-06T07:29:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.100+0000 7f4c7de00640 1 -- 172.21.15.97:0/3331749645 wait complete. 2024-03-06T07:29:29.120 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:29:29.120 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:29:29.121 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:29:29.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:29.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.318+0000 7f73fcc00640 1 Processor -- start 2024-03-06T07:29:29.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.318+0000 7f73fcc00640 1 -- start start 2024-03-06T07:29:29.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.319+0000 7f73fcc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73f80b4680 0x7f73f806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.319+0000 7f73fcc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 0x7f73f806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.319+0000 7f73fcc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f815b090 con 0x7f73f815acb0 2024-03-06T07:29:29.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.319+0000 7f73fcc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f80b2a30 con 0x7f73f80b4680 2024-03-06T07:29:29.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.319+0000 7f73fcc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80b4360 con 0x7f73f8072340 2024-03-06T07:29:29.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73f80b4680 0x7f73f806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73f80b4680 0x7f73f806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39398/0 (socket says 172.21.15.97:39398) 2024-03-06T07:29:29.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 -- 172.21.15.97:0/1390986804 learned_addr learned my addr 172.21.15.97:0/1390986804 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 -- 172.21.15.97:0/1390986804 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 msgr2=0x7f73f806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 --2- 172.21.15.97:0/1390986804 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 0x7f73f806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 -- 172.21.15.97:0/1390986804 >> v1:172.21.15.135:6789/0 conn(0x7f73f815acb0 legacy=0x7f73f80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.320+0000 7f73ed400640 1 -- 172.21.15.97:0/1390986804 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80b2cf0 con 0x7f73f80b4680 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.321+0000 7f73ed400640 1 --2- 172.21.15.97:0/1390986804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f80b4680 0x7f73f806e4f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f73e00098a0 tx=0x7f73e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298a3697bbe3fc33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.321+0000 7f73f5400640 1 -- 172.21.15.97:0/1390986804 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e0035020 con 0x7f73f80b4680 2024-03-06T07:29:29.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73f5400640 1 -- 172.21.15.97:0/1390986804 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e0031640 con 0x7f73f80b4680 2024-03-06T07:29:29.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73f5400640 1 -- 172.21.15.97:0/1390986804 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e002d040 con 0x7f73f80b4680 2024-03-06T07:29:29.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 -- 172.21.15.97:0/1390986804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f80b4680 msgr2=0x7f73f806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1390986804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f80b4680 0x7f73f806e4f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f73e00098a0 tx=0x7f73e002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 -- 172.21.15.97:0/1390986804 shutdown_connections 2024-03-06T07:29:29.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1390986804 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 0x7f73f806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1390986804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f80b4680 0x7f73f806e4f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 -- 172.21.15.97:0/1390986804 >> 172.21.15.97:0/1390986804 conn(0x7f73f80b1730 msgr2=0x7f73f80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 -- 172.21.15.97:0/1390986804 shutdown_connections 2024-03-06T07:29:29.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.322+0000 7f73fcc00640 1 -- 172.21.15.97:0/1390986804 wait complete. 2024-03-06T07:29:29.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.324+0000 7f73fcc00640 1 Processor -- start 2024-03-06T07:29:29.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.324+0000 7f73fcc00640 1 -- start start 2024-03-06T07:29:29.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73fcc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 0x7f73f8149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73fcc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f80b4680 0x7f73f8159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73fcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 0x7f73f80af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73fcc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80b4010 con 0x7f73f815acb0 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73fcc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80b32b0 con 0x7f73f80b4680 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73fcc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80ccfb0 con 0x7f73f8072340 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 0x7f73f80af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 0x7f73f80af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39408/0 (socket says 172.21.15.97:39408) 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 -- 172.21.15.97:0/1919423019 learned_addr learned my addr 172.21.15.97:0/1919423019 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 -- 172.21.15.97:0/1919423019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 msgr2=0x7f73f8149c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 0x7f73f8149c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 -- 172.21.15.97:0/1919423019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f80b4680 msgr2=0x7f73f8159160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f80b4680 0x7f73f8159160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.325+0000 7f73f6800640 1 -- 172.21.15.97:0/1919423019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f8152ad0 con 0x7f73f815acb0 2024-03-06T07:29:29.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.326+0000 7f73f6800640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 0x7f73f80af6a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f73f000b970 tx=0x7f73f000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.326+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0019070 con 0x7f73f815acb0 2024-03-06T07:29:29.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.326+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f000cb60 con 0x7f73f815acb0 2024-03-06T07:29:29.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.326+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f8152db0 con 0x7f73f815acb0 2024-03-06T07:29:29.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.326+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0014440 con 0x7f73f815acb0 2024-03-06T07:29:29.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.326+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f8153370 con 0x7f73f815acb0 2024-03-06T07:29:29.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.327+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f804f330 con 0x7f73f815acb0 2024-03-06T07:29:29.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.328+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73f001c070 con 0x7f73f815acb0 2024-03-06T07:29:29.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.329+0000 7f73eea00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73d406b4d0 0x7f73d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.330+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f73f0089c50 con 0x7f73f815acb0 2024-03-06T07:29:29.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.330+0000 7f73f5e00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73d406b4d0 0x7f73d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.330+0000 7f73f5e00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73d406b4d0 0x7f73d406d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f73e80095f0 tx=0x7f73e8006b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.334+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73f0058dc0 con 0x7f73f815acb0 2024-03-06T07:29:29.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.541+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73f806ce60 con 0x7f73f815acb0 2024-03-06T07:29:29.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.542+0000 7f73eea00640 1 -- 172.21.15.97:0/1919423019 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13008 (secure 0 0 0) 0x7f73f00586c0 con 0x7f73f815acb0 2024-03-06T07:29:29.544 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:29.544 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":38,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:28.084480+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/4130179477":"2024-03-07T07:29:24.801335+0000","172.21.15.97:6836/4130179477":"2024-03-07T07:29:24.801335+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73d406b4d0 msgr2=0x7f73d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73d406b4d0 0x7f73d406d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f73e80095f0 tx=0x7f73e8006b50 comp rx=0 tx=0).stop 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 msgr2=0x7f73f80af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 0x7f73f80af6a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f73f000b970 tx=0x7f73f000be30 comp rx=0 tx=0).stop 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 shutdown_connections 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73d406b4d0 0x7f73d406d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73f815acb0 0x7f73f80af6a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f80b4680 0x7f73f8159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 --2- 172.21.15.97:0/1919423019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73f8072340 0x7f73f8149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.544+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 >> 172.21.15.97:0/1919423019 conn(0x7f73f80b1730 msgr2=0x7f73f806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.545+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 shutdown_connections 2024-03-06T07:29:29.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.545+0000 7f73fcc00640 1 -- 172.21.15.97:0/1919423019 wait complete. 2024-03-06T07:29:29.563 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:29.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:29.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.768+0000 7f01db000640 1 Processor -- start 2024-03-06T07:29:29.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.768+0000 7f01db000640 1 -- start start 2024-03-06T07:29:29.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01db000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01d415df40 0x7f01d406b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01db000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f01d40b5ca0 0x7f01d40725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01db000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d4072b00 0x7f01d4161ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01db000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d406b7a0 con 0x7f01d415df40 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01db000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d40b1900 con 0x7f01d40b5ca0 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01db000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d406c4f0 con 0x7f01d4072b00 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01d415df40 0x7f01d406b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01d415df40 0x7f01d406b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39414/0 (socket says 172.21.15.97:39414) 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 -- 172.21.15.97:0/3653527377 learned_addr learned my addr 172.21.15.97:0/3653527377 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 -- 172.21.15.97:0/3653527377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d4072b00 msgr2=0x7f01d4161ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 --2- 172.21.15.97:0/3653527377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d4072b00 0x7f01d4161ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 -- 172.21.15.97:0/3653527377 >> v2:172.21.15.135:3300/0 conn(0x7f01d40b5ca0 msgr2=0x7f01d40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 --2- 172.21.15.97:0/3653527377 >> v2:172.21.15.135:3300/0 conn(0x7f01d40b5ca0 0x7f01d40725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.770+0000 7f01d9c00640 1 -- 172.21.15.97:0/3653527377 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d406c7d0 con 0x7f01d415df40 2024-03-06T07:29:29.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.771+0000 7f01d9c00640 1 --2- 172.21.15.97:0/3653527377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d406b260 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f01d4067020 tx=0x7f01c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64bc93729df7e62c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01d3e00640 1 -- 172.21.15.97:0/3653527377 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8035020 con 0x7f01d415df40 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01d3e00640 1 -- 172.21.15.97:0/3653527377 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c8031870 con 0x7f01d415df40 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01d3e00640 1 -- 172.21.15.97:0/3653527377 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8031c40 con 0x7f01d415df40 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 -- 172.21.15.97:0/3653527377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 msgr2=0x7f01d406b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 --2- 172.21.15.97:0/3653527377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d406b260 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f01d4067020 tx=0x7f01c802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 -- 172.21.15.97:0/3653527377 shutdown_connections 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 --2- 172.21.15.97:0/3653527377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d4072b00 0x7f01d4161ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 --2- 172.21.15.97:0/3653527377 >> v2:172.21.15.135:3300/0 conn(0x7f01d40b5ca0 0x7f01d40725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 --2- 172.21.15.97:0/3653527377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d406b260 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 -- 172.21.15.97:0/3653527377 >> 172.21.15.97:0/3653527377 conn(0x7f01d40b3bf0 msgr2=0x7f01d40b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 -- 172.21.15.97:0/3653527377 shutdown_connections 2024-03-06T07:29:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.772+0000 7f01db000640 1 -- 172.21.15.97:0/3653527377 wait complete. 2024-03-06T07:29:29.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.774+0000 7f01db000640 1 Processor -- start 2024-03-06T07:29:29.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.774+0000 7f01db000640 1 -- start start 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01db000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d4072b00 0x7f01d415fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01db000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d40b5ca0 0x7f01d4161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01db000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d40991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01db000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40b11b0 con 0x7f01d415df40 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01db000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d415e320 con 0x7f01d4072b00 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01db000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40b1330 con 0x7f01d40b5ca0 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01da600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d40991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01da600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d40991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39416/0 (socket says 172.21.15.97:39416) 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.775+0000 7f01da600640 1 -- 172.21.15.97:0/412061282 learned_addr learned my addr 172.21.15.97:0/412061282 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d9200640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d40b5ca0 0x7f01d4161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d9c00640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d4072b00 0x7f01d415fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01da600640 1 -- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d40b5ca0 msgr2=0x7f01d4161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01da600640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d40b5ca0 0x7f01d4161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01da600640 1 -- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d4072b00 msgr2=0x7f01d415fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01da600640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d4072b00 0x7f01d415fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01da600640 1 -- 172.21.15.97:0/412061282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d41601d0 con 0x7f01d415df40 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d9c00640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d4072b00 0x7f01d415fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d9200640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d40b5ca0 0x7f01d4161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01da600640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d40991a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b9a0 tx=0x7f01cc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc019070 con 0x7f01d415df40 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc00cbc0 con 0x7f01d415df40 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc014440 con 0x7f01d415df40 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.776+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d41604b0 con 0x7f01d415df40 2024-03-06T07:29:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.777+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d4160a70 con 0x7f01d415df40 2024-03-06T07:29:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.778+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d406c1b0 con 0x7f01d415df40 2024-03-06T07:29:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.779+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01cc01c070 con 0x7f01d415df40 2024-03-06T07:29:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.780+0000 7f01d1600640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01b806b4d0 0x7f01b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.780+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f01cc089db0 con 0x7f01d415df40 2024-03-06T07:29:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.780+0000 7f01d9c00640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01b806b4d0 0x7f01b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.781+0000 7f01d9c00640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01b806b4d0 0x7f01b806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01c8036000 tx=0x7f01c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.784+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01d406c1b0 con 0x7f01d415df40 2024-03-06T07:29:29.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.980+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01d404f330 con 0x7f01d415df40 2024-03-06T07:29:29.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.980+0000 7f01d1600640 1 -- 172.21.15.97:0/412061282 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f01cc08e070 con 0x7f01d415df40 2024-03-06T07:29:29.983 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.983+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01b806b4d0 msgr2=0x7f01b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.983+0000 7f01db000640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01b806b4d0 0x7f01b806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01c8036000 tx=0x7f01c8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.983+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 msgr2=0x7f01d40991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.983+0000 7f01db000640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d40991a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b9a0 tx=0x7f01cc00be60 comp rx=0 tx=0).stop 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 shutdown_connections 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01b806b4d0 0x7f01b806d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01d415df40 0x7f01d40991a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01d40b5ca0 0x7f01d4161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 --2- 172.21.15.97:0/412061282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d4072b00 0x7f01d415fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 >> 172.21.15.97:0/412061282 conn(0x7f01d40b3bf0 msgr2=0x7f01d40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 shutdown_connections 2024-03-06T07:29:29.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:29.984+0000 7f01db000640 1 -- 172.21.15.97:0/412061282 wait complete. 2024-03-06T07:29:30.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:29:30.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.214+0000 7f68dd800640 1 Processor -- start 2024-03-06T07:29:30.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.214+0000 7f68dd800640 1 -- start start 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68dd800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f68d8144c00 0x7f68d806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68dd800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d806ee70 con 0x7f68d815f9d0 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68dd800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d806cef0 con 0x7f68d815fe60 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68dd800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d806d1d0 con 0x7f68d8144c00 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68d6000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f68d815f9d0 0x7f68d80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53496/0 (socket says 172.21.15.97:53496) 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68d6000640 1 -- 172.21.15.97:0/2804244859 learned_addr learned my addr 172.21.15.97:0/2804244859 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:30.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.216+0000 7f68d6a00640 1 --2- 172.21.15.97:0/2804244859 >> v2:172.21.15.135:3300/0 conn(0x7f68d8144c00 0x7f68d806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702454588 0 0) 0x7f68c0001490 con 0x7f68d815f9d0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b8003520 con 0x7f68d815f9d0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844588051 0 0) 0x7f68b8003520 con 0x7f68d815f9d0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 >> v2:172.21.15.135:3300/0 conn(0x7f68d8144c00 msgr2=0x7f68d806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 --2- 172.21.15.97:0/2804244859 >> v2:172.21.15.135:3300/0 conn(0x7f68d8144c00 0x7f68d806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 >> v1:172.21.15.135:6789/0 conn(0x7f68d815fe60 legacy=0x7f68d80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d806d4b0 con 0x7f68d815f9d0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f68c0001bc0 con 0x7f68d815f9d0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 >> v1:172.21.15.97:6789/0 conn(0x7f68d815f9d0 legacy=0x7f68d80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68b80039b0 0x7f68b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b8004800 0x7f68b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68b80051c0 0x7f68b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d806d1d0 con 0x7f68b80039b0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b800c060 con 0x7f68b8004800 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b800c260 con 0x7f68b80051c0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.217+0000 7f68d6a00640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68b80039b0 0x7f68b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6000640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b8004800 0x7f68b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6a00640 1 -- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68b80051c0 msgr2=0x7f68b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6a00640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68b80051c0 0x7f68b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6a00640 1 -- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b8004800 msgr2=0x7f68b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6a00640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b8004800 0x7f68b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6a00640 1 -- 172.21.15.97:0/2804244859 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8003520 con 0x7f68b80039b0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d7400640 1 -- 172.21.15.97:0/2804244859 reap_dead start 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.218+0000 7f68d6000640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b8004800 0x7f68b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.219+0000 7f68d6a00640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68b80039b0 0x7f68b8003dd0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009ac0 tx=0x7f68b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14579d92b6f61c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.219+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8039070 con 0x7f68b80039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.219+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c802c6b0 con 0x7f68b80039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.219+0000 7f68d5600640 1 -- 172.21.15.97:0/2804244859 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8009690 con 0x7f68b80039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 -- 172.21.15.97:0/2804244859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68b80039b0 msgr2=0x7f68b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68b80039b0 0x7f68b8003dd0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009ac0 tx=0x7f68b8005830 comp rx=0 tx=0).stop 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 -- 172.21.15.97:0/2804244859 shutdown_connections 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 --2- 172.21.15.97:0/2804244859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68b80039b0 0x7f68b8003dd0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 -- 172.21.15.97:0/2804244859 >> 172.21.15.97:0/2804244859 conn(0x7f68d80b1730 msgr2=0x7f68d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 -- 172.21.15.97:0/2804244859 shutdown_connections 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.220+0000 7f68dd800640 1 -- 172.21.15.97:0/2804244859 wait complete. 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.222+0000 7f68dd800640 1 Processor -- start 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.222+0000 7f68dd800640 1 -- start start 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68dd800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b80039b0 0x7f68d8139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68dd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 0x7f68d8156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68dd800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68d81e62d0 0x7f68d81e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68dd800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8160730 con 0x7f68d81560f0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68dd800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8160270 con 0x7f68b80039b0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68dd800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d806cb50 con 0x7f68d81e62d0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 0x7f68d8156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 0x7f68d8156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39426/0 (socket says 172.21.15.97:39426) 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 -- 172.21.15.97:0/1776693409 learned_addr learned my addr 172.21.15.97:0/1776693409 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6a00640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b80039b0 0x7f68d8139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 -- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68d81e62d0 msgr2=0x7f68d81e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68d81e62d0 0x7f68d81e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 -- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b80039b0 msgr2=0x7f68d8139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b80039b0 0x7f68d8139450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.223+0000 7f68d6000640 1 -- 172.21.15.97:0/1776693409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d813a4a0 con 0x7f68d81560f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.224+0000 7f68d6000640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 0x7f68d8156570 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f68c0001bc0 tx=0x7f68c000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.224+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0019070 con 0x7f68d81560f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.224+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0004d60 con 0x7f68d81560f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.224+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d81e7090 con 0x7f68d81560f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.224+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c00072f0 con 0x7f68d81560f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.224+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d813aa20 con 0x7f68d81560f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.225+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d804f330 con 0x7f68d81560f0 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.226+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68c001c070 con 0x7f68d81560f0 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.227+0000 7f68cea00640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f68b406b4d0 0x7f68b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.227+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f68c0089f30 con 0x7f68d81560f0 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.227+0000 7f68d6a00640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f68b406b4d0 0x7f68b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.228+0000 7f68d6a00640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f68b406b4d0 0x7f68b406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f68c8036000 tx=0x7f68c8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.232+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68d806ee70 con 0x7f68d81560f0 2024-03-06T07:29:30.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:30.494+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f68d8072890 con 0x7f68d81560f0 2024-03-06T07:29:31.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.111+0000 7f68cea00640 1 -- 172.21.15.97:0/1776693409 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v120) v1 ==== 136+0+0 (secure 0 0 0) 0x7f68c0058a50 con 0x7f68d81560f0 2024-03-06T07:29:31.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f68b406b4d0 msgr2=0x7f68b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f68b406b4d0 0x7f68b406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f68c8036000 tx=0x7f68c8030000 comp rx=0 tx=0).stop 2024-03-06T07:29:31.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 msgr2=0x7f68d8156570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 0x7f68d8156570 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f68c0001bc0 tx=0x7f68c000b4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 shutdown_connections 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f68b406b4d0 0x7f68b406d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68d81e62d0 0x7f68d81e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68d81560f0 0x7f68d8156570 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 --2- 172.21.15.97:0/1776693409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68b80039b0 0x7f68d8139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 >> 172.21.15.97:0/1776693409 conn(0x7f68d80b1730 msgr2=0x7f68d80b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 shutdown_connections 2024-03-06T07:29:31.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.115+0000 7f68dd800640 1 -- 172.21.15.97:0/1776693409 wait complete. 2024-03-06T07:29:31.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:29:31.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.347+0000 7fdd6b800640 1 Processor -- start 2024-03-06T07:29:31.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.347+0000 7fdd6b800640 1 -- start start 2024-03-06T07:29:31.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.349+0000 7fdd6b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415bdc0 0x7fdd640cd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.349+0000 7fdd6b800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6415e730 con 0x7fdd640cdba0 2024-03-06T07:29:31.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.349+0000 7fdd6b800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6415ea10 con 0x7fdd64141850 2024-03-06T07:29:31.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.349+0000 7fdd6b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cdfd0 con 0x7fdd6415bdc0 2024-03-06T07:29:31.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.349+0000 7fdd69800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdd640cdba0 0x7fdd641439e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53500/0 (socket says 172.21.15.97:53500) 2024-03-06T07:29:31.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.349+0000 7fdd69800640 1 -- 172.21.15.97:0/1023154911 learned_addr learned my addr 172.21.15.97:0/1023154911 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:31.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd68e00640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415bdc0 0x7fdd640cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895043389 0 0) 0x7fdd6415e730 con 0x7fdd640cdba0 2024-03-06T07:29:31.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd48003520 con 0x7fdd640cdba0 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340421900 0 0) 0x7fdd48003520 con 0x7fdd640cdba0 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415bdc0 msgr2=0x7fdd640cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415bdc0 0x7fdd640cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 >> v1:172.21.15.135:6789/0 conn(0x7fdd64141850 legacy=0x7fdd6406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6415ecf0 con 0x7fdd640cdba0 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdd5c001670 con 0x7fdd640cdba0 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 >> v1:172.21.15.97:6789/0 conn(0x7fdd640cdba0 legacy=0x7fdd641439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd480039e0 0x7fdd48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd48004800 0x7fdd48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd48005180 0x7fdd4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cdfd0 con 0x7fdd480039e0 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4800bf00 con 0x7fdd48005180 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.350+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4800c080 con 0x7fdd48004800 2024-03-06T07:29:31.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd68e00640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd480039e0 0x7fdd48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd63e00640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd48004800 0x7fdd48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd68e00640 1 -- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd48004800 msgr2=0x7fdd48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd68e00640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd48004800 0x7fdd48004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd68e00640 1 -- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd48005180 msgr2=0x7fdd4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd68e00640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd48005180 0x7fdd4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.351+0000 7fdd68e00640 1 -- 172.21.15.97:0/1023154911 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd48003520 con 0x7fdd480039e0 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.352+0000 7fdd68e00640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd480039e0 0x7fdd48003e00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009ac0 tx=0x7fdd48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94496d97f050d996 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.352+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c039070 con 0x7fdd480039e0 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.352+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c02c6b0 con 0x7fdd480039e0 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.352+0000 7fdd63400640 1 -- 172.21.15.97:0/1023154911 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c009690 con 0x7fdd480039e0 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd6b800640 1 -- 172.21.15.97:0/1023154911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd480039e0 msgr2=0x7fdd48003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd6b800640 1 --2- 172.21.15.97:0/1023154911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd480039e0 0x7fdd48003e00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009ac0 tx=0x7fdd48005830 comp rx=0 tx=0).stop 2024-03-06T07:29:31.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd69800640 1 -- 172.21.15.97:0/1023154911 reap_dead start 2024-03-06T07:29:31.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd6b800640 1 -- 172.21.15.97:0/1023154911 shutdown_connections 2024-03-06T07:29:31.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd6b800640 1 -- 172.21.15.97:0/1023154911 >> 172.21.15.97:0/1023154911 conn(0x7fdd640b2890 msgr2=0x7fdd640ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd6b800640 1 -- 172.21.15.97:0/1023154911 shutdown_connections 2024-03-06T07:29:31.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.353+0000 7fdd6b800640 1 -- 172.21.15.97:0/1023154911 wait complete. 2024-03-06T07:29:31.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.355+0000 7fdd6b800640 1 Processor -- start 2024-03-06T07:29:31.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.355+0000 7fdd6b800640 1 -- start start 2024-03-06T07:29:31.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd6b800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd480039e0 0x7fdd640af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd6b800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 0x7fdd640b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd6b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415ac30 0x7fdd6415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd6b800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cd660 con 0x7fdd640b1340 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd6b800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6415f2b0 con 0x7fdd480039e0 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd6b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cd7e0 con 0x7fdd6415ac30 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 0x7fdd640b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 0x7fdd640b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39454/0 (socket says 172.21.15.97:39454) 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.356+0000 7fdd63e00640 1 -- 172.21.15.97:0/2977819613 learned_addr learned my addr 172.21.15.97:0/2977819613 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd68e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd480039e0 0x7fdd640af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd69800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415ac30 0x7fdd6415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd63e00640 1 -- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415ac30 msgr2=0x7fdd6415b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd63e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415ac30 0x7fdd6415b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd63e00640 1 -- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd480039e0 msgr2=0x7fdd640af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd63e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd480039e0 0x7fdd640af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd63e00640 1 -- 172.21.15.97:0/2977819613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd640afb10 con 0x7fdd640b1340 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd69800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415ac30 0x7fdd6415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd68e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd480039e0 0x7fdd640af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd63e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 0x7fdd640b4570 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d8e0 tx=0x7fdd5400dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54003da0 con 0x7fdd640b1340 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd54017070 con 0x7fdd640b1340 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5400b8b0 con 0x7fdd640b1340 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd640afdf0 con 0x7fdd640b1340 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.357+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd640b03b0 con 0x7fdd640b1340 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.358+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6404f330 con 0x7fdd640b1340 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.359+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd5400ba50 con 0x7fdd640b1340 2024-03-06T07:29:31.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.360+0000 7fdd60c00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd4006b4d0 0x7fdd4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.360+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdd5408a260 con 0x7fdd640b1340 2024-03-06T07:29:31.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.360+0000 7fdd68e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd4006b4d0 0x7fdd4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.361+0000 7fdd68e00640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd4006b4d0 0x7fdd4006d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c036000 tx=0x7fdd4c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.365+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd54059340 con 0x7fdd640b1340 2024-03-06T07:29:31.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.560+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdd640b08d0 con 0x7fdd640b1340 2024-03-06T07:29:31.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.565+0000 7fdd60c00640 1 -- 172.21.15.97:0/2977819613 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdd640b08d0 con 0x7fdd640b1340 2024-03-06T07:29:31.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd4006b4d0 msgr2=0x7fdd4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd4006b4d0 0x7fdd4006d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c036000 tx=0x7fdd4c030000 comp rx=0 tx=0).stop 2024-03-06T07:29:31.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 msgr2=0x7fdd640b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 0x7fdd640b4570 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d8e0 tx=0x7fdd5400dda0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 shutdown_connections 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd4006b4d0 0x7fdd4006d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd6415ac30 0x7fdd6415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd640b1340 0x7fdd640b4570 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 --2- 172.21.15.97:0/2977819613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd480039e0 0x7fdd640af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 >> 172.21.15.97:0/2977819613 conn(0x7fdd640b2890 msgr2=0x7fdd6415f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 shutdown_connections 2024-03-06T07:29:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:31.568+0000 7fdd6b800640 1 -- 172.21.15.97:0/2977819613 wait complete. 2024-03-06T07:29:31.589 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:31.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:31.705 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:31.818 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:31.819 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:31.819 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:31.875 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:31.875 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:29:31.875 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:31.930 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:31.930 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:31.930 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:31.985 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:31.985 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:29:31.985 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:32.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:32.041 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:29:32.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:32.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.252+0000 7fc7bf000640 1 Processor -- start 2024-03-06T07:29:32.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.253+0000 7fc7bf000640 1 -- start start 2024-03-06T07:29:32.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bf000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bf000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7b80cf9a0 0x7fc7b80cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bf000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b80b3040 con 0x7fc7b80b4390 2024-03-06T07:29:32.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bf000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b815fb50 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bf000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80cd3e0 con 0x7fc7b815f770 2024-03-06T07:29:32.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bd000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7b80cf9a0 0x7fc7b80cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bd000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7b80cf9a0 0x7fc7b80cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39466/0 (socket says 172.21.15.97:39466) 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.254+0000 7fc7bd000640 1 -- 172.21.15.97:0/422762146 learned_addr learned my addr 172.21.15.97:0/422762146 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.255+0000 7fc7b7e00640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.255+0000 7fc7bd000640 1 -- 172.21.15.97:0/422762146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 msgr2=0x7fc7b815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.255+0000 7fc7bd000640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b815fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.255+0000 7fc7bd000640 1 -- 172.21.15.97:0/422762146 >> v1:172.21.15.135:6789/0 conn(0x7fc7b80b4390 legacy=0x7fc7b8160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.255+0000 7fc7bd000640 1 -- 172.21.15.97:0/422762146 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b80cd6c0 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.255+0000 7fc7b7e00640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b815fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:32.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.256+0000 7fc7bd000640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cfe30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009870 tx=0x7fc7b002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d5462b40914da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.256+0000 7fc7b6a00640 1 -- 172.21.15.97:0/422762146 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00317c0 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.256+0000 7fc7b6a00640 1 -- 172.21.15.97:0/422762146 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b0037070 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.256+0000 7fc7b6a00640 1 -- 172.21.15.97:0/422762146 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0031b80 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 -- 172.21.15.97:0/422762146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 msgr2=0x7fc7b80cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cfe30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009870 tx=0x7fc7b002b860 comp rx=0 tx=0).stop 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 -- 172.21.15.97:0/422762146 shutdown_connections 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cfe30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 --2- 172.21.15.97:0/422762146 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b815fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 -- 172.21.15.97:0/422762146 >> 172.21.15.97:0/422762146 conn(0x7fc7b80b3870 msgr2=0x7fc7b80b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 -- 172.21.15.97:0/422762146 shutdown_connections 2024-03-06T07:29:32.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.257+0000 7fc7bf000640 1 -- 172.21.15.97:0/422762146 wait complete. 2024-03-06T07:29:32.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.259+0000 7fc7bf000640 1 Processor -- start 2024-03-06T07:29:32.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.259+0000 7fc7bf000640 1 -- start start 2024-03-06T07:29:32.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7bf000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7b80b4390 0x7fc7b8151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7bf000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7bf000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b80cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7bf000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80b34a0 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7bf000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80b3320 con 0x7fc7b80b4390 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7bf000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b81609a0 con 0x7fc7b815f770 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39468/0 (socket says 172.21.15.97:39468) 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.260+0000 7fc7b7400640 1 -- 172.21.15.97:0/2209724247 learned_addr learned my addr 172.21.15.97:0/2209724247 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7400640 1 -- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 msgr2=0x7fc7b80cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7bd000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b80cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7e00640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7b80b4390 0x7fc7b8151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7400640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b80cfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7400640 1 -- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7b80b4390 msgr2=0x7fc7b8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7400640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7b80b4390 0x7fc7b8151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7400640 1 -- 172.21.15.97:0/2209724247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b80c9180 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7b7400640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cb4e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c780 tx=0x7fc7a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0016020 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a000aa50 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.261+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b80c9460 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.262+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0012420 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.262+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b80c9a20 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.263+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b80c9f10 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.264+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7a0019020 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.265+0000 7fc7afe00640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79406b4d0 0x7fc79406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.265+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc7a0089fd0 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.265+0000 7fc7b7e00640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79406b4d0 0x7fc79406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.266+0000 7fc7b7e00640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79406b4d0 0x7fc79406d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800b840 tx=0x7fc7a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.270+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7a0020030 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.495+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7b8068ee0 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.496+0000 7fc7afe00640 1 -- 172.21.15.97:0/2209724247 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fc7b8068ee0 con 0x7fc7b80cf9a0 2024-03-06T07:29:32.499 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:32.499 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/414851246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":414851246},{"type":"v1","addr":"172.21.15.135:6835","nonce":414851246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4642,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/851002758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":851002758},{"type":"v1","addr":"172.21.15.135:6834","nonce":851002758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/541548774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":541548774},{"type":"v1","addr":"172.21.15.97:6835","nonce":541548774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5424,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3667234474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3667234474},{"type":"v1","addr":"172.21.15.97:6837","nonce":3667234474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-03-06T07:29:32.499 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 19 2024-03-06T07:29:32.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79406b4d0 msgr2=0x7fc79406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79406b4d0 0x7fc79406d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800b840 tx=0x7fc7a8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 msgr2=0x7fc7b80cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cb4e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c780 tx=0x7fc7a000cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 shutdown_connections 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79406b4d0 0x7fc79406d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7b815f770 0x7fc7b80cfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7b80cf9a0 0x7fc7b80cb4e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 --2- 172.21.15.97:0/2209724247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7b80b4390 0x7fc7b8151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 >> 172.21.15.97:0/2209724247 conn(0x7fc7b80b3870 msgr2=0x7fc7b80b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 shutdown_connections 2024-03-06T07:29:32.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.499+0000 7fc7bf000640 1 -- 172.21.15.97:0/2209724247 wait complete. 2024-03-06T07:29:32.518 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:29:32.518 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:29:32.519 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:29:32.519 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:32.519 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:29:32.519 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:29:32.519 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:29:32.519 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:32.520 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:29:32.520 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:29:32.520 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:29:32.520 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:32.520 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:29:32.520 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:29:32.520 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:29:32.521 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:32.536 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:29:32.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:29:32.537 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:29:32.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:29:32.548 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:29:32.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:29:32.556 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:29:32.557 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:29:32.583 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:29:32.583 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:32.588 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:29:32.589 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:32.603 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:29:32.603 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:32.609 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:29:32.609 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:32.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:29:32.731 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:29:32.732 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:29:32.759 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:29:32.765 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:29:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.778+0000 7fe01fe00640 1 Processor -- start 2024-03-06T07:29:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.778+0000 7fe01fe00640 1 -- start start 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe0200cf370 0x7fe02006d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cc2a0 0x7fe0200cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01fe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0200b2b30 con 0x7fe02015a940 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01fe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0200b28e0 con 0x7fe0200cf370 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200b2e10 con 0x7fe0200cc2a0 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01ea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe02015a940 0x7fe0200b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53516/0 (socket says 172.21.15.97:53516) 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01ea00640 1 -- 172.21.15.97:0/1344925446 learned_addr learned my addr 172.21.15.97:0/1344925446 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:32.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe01f400640 1 --2- 172.21.15.97:0/1344925446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cc2a0 0x7fe0200cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.779+0000 7fe017e00640 1 --2- 172.21.15.97:0/1344925446 >> v2:172.21.15.135:3300/0 conn(0x7fe0200cf370 0x7fe02006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.780+0000 7fe01e000640 1 -- 172.21.15.97:0/1344925446 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891728304 0 0) 0x7fe0200b2b30 con 0x7fe02015a940 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.780+0000 7fe01e000640 1 -- 172.21.15.97:0/1344925446 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe004003520 con 0x7fe02015a940 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.780+0000 7fe01f400640 1 -- 172.21.15.97:0/1344925446 >> v2:172.21.15.135:3300/0 conn(0x7fe0200cf370 msgr2=0x7fe02006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.780+0000 7fe01f400640 1 --2- 172.21.15.97:0/1344925446 >> v2:172.21.15.135:3300/0 conn(0x7fe0200cf370 0x7fe02006d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.780+0000 7fe01f400640 1 -- 172.21.15.97:0/1344925446 >> v1:172.21.15.97:6789/0 conn(0x7fe02015a940 legacy=0x7fe0200b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.780+0000 7fe01f400640 1 -- 172.21.15.97:0/1344925446 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02006ca50 con 0x7fe0200cc2a0 2024-03-06T07:29:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.781+0000 7fe01f400640 1 --2- 172.21.15.97:0/1344925446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cc2a0 0x7fe0200cc750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe018009d90 tx=0x7fe01802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe643c85d3f5e7d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.781+0000 7fe01e000640 1 -- 172.21.15.97:0/1344925446 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018039070 con 0x7fe0200cc2a0 2024-03-06T07:29:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.781+0000 7fe01e000640 1 -- 172.21.15.97:0/1344925446 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01802cc50 con 0x7fe0200cc2a0 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.781+0000 7fe01e000640 1 -- 172.21.15.97:0/1344925446 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018034440 con 0x7fe0200cc2a0 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 -- 172.21.15.97:0/1344925446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cc2a0 msgr2=0x7fe0200cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 --2- 172.21.15.97:0/1344925446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cc2a0 0x7fe0200cc750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe018009d90 tx=0x7fe01802bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 -- 172.21.15.97:0/1344925446 shutdown_connections 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 --2- 172.21.15.97:0/1344925446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cc2a0 0x7fe0200cc750 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 --2- 172.21.15.97:0/1344925446 >> v2:172.21.15.135:3300/0 conn(0x7fe0200cf370 0x7fe02006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 -- 172.21.15.97:0/1344925446 >> 172.21.15.97:0/1344925446 conn(0x7fe02006afc0 msgr2=0x7fe02006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 -- 172.21.15.97:0/1344925446 shutdown_connections 2024-03-06T07:29:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.782+0000 7fe01fe00640 1 -- 172.21.15.97:0/1344925446 wait complete. 2024-03-06T07:29:32.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.784+0000 7fe01fe00640 1 Processor -- start 2024-03-06T07:29:32.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.784+0000 7fe01fe00640 1 -- start start 2024-03-06T07:29:32.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0200cc2a0 0x7fe02015d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cf370 0x7fe02014c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 0x7fe0200c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02006d790 con 0x7fe02015a940 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200b30f0 con 0x7fe0200cc2a0 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02015db70 con 0x7fe0200cf370 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 0x7fe0200c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 0x7fe0200c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39500/0 (socket says 172.21.15.97:39500) 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 -- 172.21.15.97:0/4238532334 learned_addr learned my addr 172.21.15.97:0/4238532334 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01ea00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0200cc2a0 0x7fe02015d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe017e00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cf370 0x7fe02014c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 -- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cf370 msgr2=0x7fe02014c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cf370 0x7fe02014c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 -- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0200cc2a0 msgr2=0x7fe02015d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0200cc2a0 0x7fe02015d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.785+0000 7fe01f400640 1 -- 172.21.15.97:0/4238532334 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02015c590 con 0x7fe02015a940 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.786+0000 7fe01f400640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 0x7fe0200c4850 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe018009d90 tx=0x7fe018004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.786+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018039050 con 0x7fe02015a940 2024-03-06T07:29:32.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.786+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe018045070 con 0x7fe02015a940 2024-03-06T07:29:32.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.786+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018009450 con 0x7fe02015a940 2024-03-06T07:29:32.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.786+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02015c870 con 0x7fe02015a940 2024-03-06T07:29:32.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.786+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02015ce30 con 0x7fe02015a940 2024-03-06T07:29:32.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.787+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020067830 con 0x7fe02015a940 2024-03-06T07:29:32.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.788+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe01803e610 con 0x7fe02015a940 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.789+0000 7fe016a00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.789+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe0180ac810 con 0x7fe02015a940 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.789+0000 7fe01ea00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.790+0000 7fe01ea00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdffc06b4d0 0x7fdffc06d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe010005880 tx=0x7fe010009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.793+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe020067830 con 0x7fe02015a940 2024-03-06T07:29:32.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:32.988+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe02004f330 con 0x7fe02015a940 2024-03-06T07:29:33.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.135+0000 7fe016a00640 1 -- 172.21.15.97:0/4238532334 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v40) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe01807b2a0 con 0x7fe02015a940 2024-03-06T07:29:33.137 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:29:33.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.138+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdffc06b4d0 msgr2=0x7fdffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.138+0000 7fe01fe00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdffc06b4d0 0x7fdffc06d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe010005880 tx=0x7fe010009040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.138+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 msgr2=0x7fe0200c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.138+0000 7fe01fe00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 0x7fe0200c4850 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe018009d90 tx=0x7fe018004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:33.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 shutdown_connections 2024-03-06T07:29:33.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe02015a940 0x7fe0200c4850 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0200cf370 0x7fe02014c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 --2- 172.21.15.97:0/4238532334 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0200cc2a0 0x7fe02015d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 >> 172.21.15.97:0/4238532334 conn(0x7fe02006afc0 msgr2=0x7fe020070990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 shutdown_connections 2024-03-06T07:29:33.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.139+0000 7fe01fe00640 1 -- 172.21.15.97:0/4238532334 wait complete. 2024-03-06T07:29:33.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:29:33.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.342+0000 7f3cd9e00640 1 Processor -- start 2024-03-06T07:29:33.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.343+0000 7f3cd9e00640 1 -- start start 2024-03-06T07:29:33.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.344+0000 7f3cd9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3cd41561f0 0x7f3cd40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.344+0000 7f3cd9e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3cd40cc9d0 0x7f3cd415f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.344+0000 7f3cd9e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd415c240 con 0x7f3cd40aebf0 2024-03-06T07:29:33.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.344+0000 7f3cd9e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd415ce30 con 0x7f3cd41561f0 2024-03-06T07:29:33.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.344+0000 7f3cd9e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd415c500 con 0x7f3cd40cc9d0 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3cd41561f0 0x7f3cd40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3cd41561f0 0x7f3cd40ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39506/0 (socket says 172.21.15.97:39506) 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 -- 172.21.15.97:0/3770886049 learned_addr learned my addr 172.21.15.97:0/3770886049 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd9400640 1 --2- 172.21.15.97:0/3770886049 >> v2:172.21.15.135:3300/0 conn(0x7f3cd40cc9d0 0x7f3cd415f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 -- 172.21.15.97:0/3770886049 >> v2:172.21.15.135:3300/0 conn(0x7f3cd40cc9d0 msgr2=0x7f3cd415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 --2- 172.21.15.97:0/3770886049 >> v2:172.21.15.135:3300/0 conn(0x7f3cd40cc9d0 0x7f3cd415f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 -- 172.21.15.97:0/3770886049 >> v1:172.21.15.135:6789/0 conn(0x7f3cd40aebf0 legacy=0x7f3cd40cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd8a00640 1 -- 172.21.15.97:0/3770886049 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd415c7e0 con 0x7f3cd41561f0 2024-03-06T07:29:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.345+0000 7f3cd9400640 1 --2- 172.21.15.97:0/3770886049 >> v2:172.21.15.135:3300/0 conn(0x7f3cd40cc9d0 0x7f3cd415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:33.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.347+0000 7f3cd8a00640 1 --2- 172.21.15.97:0/3770886049 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ae6b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3cd406cb60 tx=0x7f3cc402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c6fdbf276175ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd3e00640 1 -- 172.21.15.97:0/3770886049 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4034070 con 0x7f3cd41561f0 2024-03-06T07:29:33.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd3e00640 1 -- 172.21.15.97:0/3770886049 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4030860 con 0x7f3cd41561f0 2024-03-06T07:29:33.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd3e00640 1 -- 172.21.15.97:0/3770886049 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4030c30 con 0x7f3cd41561f0 2024-03-06T07:29:33.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd9e00640 1 -- 172.21.15.97:0/3770886049 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 msgr2=0x7f3cd40ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/3770886049 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ae6b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3cd406cb60 tx=0x7f3cc402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd9e00640 1 -- 172.21.15.97:0/3770886049 shutdown_connections 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/3770886049 >> v2:172.21.15.135:3300/0 conn(0x7f3cd40cc9d0 0x7f3cd415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/3770886049 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ae6b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.348+0000 7f3cd9e00640 1 -- 172.21.15.97:0/3770886049 >> 172.21.15.97:0/3770886049 conn(0x7f3cd406afc0 msgr2=0x7f3cd406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.349+0000 7f3cd9e00640 1 -- 172.21.15.97:0/3770886049 shutdown_connections 2024-03-06T07:29:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.349+0000 7f3cd9e00640 1 -- 172.21.15.97:0/3770886049 wait complete. 2024-03-06T07:29:33.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.350+0000 7f3cd9e00640 1 Processor -- start 2024-03-06T07:29:33.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.350+0000 7f3cd9e00640 1 -- start start 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.351+0000 7f3cd9e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd40aebf0 0x7f3cd4157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.351+0000 7f3cd9e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd40cc9d0 0x7f3cd4158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.351+0000 7f3cd9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.351+0000 7f3cd9e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4156750 con 0x7f3cd41561f0 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.351+0000 7f3cd9e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd41565d0 con 0x7f3cd40aebf0 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.351+0000 7f3cd9e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd415fd70 con 0x7f3cd40cc9d0 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39510/0 (socket says 172.21.15.97:39510) 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 -- 172.21.15.97:0/1559424462 learned_addr learned my addr 172.21.15.97:0/1559424462 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd8a00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd40aebf0 0x7f3cd4157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 -- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd40cc9d0 msgr2=0x7f3cd4158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd40cc9d0 0x7f3cd4158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 -- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd40aebf0 msgr2=0x7f3cd4157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd40aebf0 0x7f3cd4157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 -- 172.21.15.97:0/1559424462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd4158bf0 con 0x7f3cd41561f0 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9400640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ade00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc007f20 tx=0x7f3ccc00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc01a070 con 0x7f3cd41561f0 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc003b90 con 0x7f3cd41561f0 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.352+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd4158ed0 con 0x7f3cd41561f0 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.353+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc004030 con 0x7f3cd41561f0 2024-03-06T07:29:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.353+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd4159490 con 0x7f3cd41561f0 2024-03-06T07:29:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.354+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd404f330 con 0x7f3cd41561f0 2024-03-06T07:29:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.354+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ccc01d070 con 0x7f3cd41561f0 2024-03-06T07:29:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.355+0000 7f3cd1600640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b4d0 0x7f3cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.355+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3ccc08a260 con 0x7f3cd41561f0 2024-03-06T07:29:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.356+0000 7f3cd8a00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b4d0 0x7f3cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.356+0000 7f3cd8a00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b4d0 0x7f3cb006d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4036000 tx=0x7f3cc403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.360+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ccc010070 con 0x7f3cd41561f0 2024-03-06T07:29:33.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.608+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3cd40ae9d0 con 0x7f3cd41561f0 2024-03-06T07:29:33.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.609+0000 7f3cd1600640 1 -- 172.21.15.97:0/1559424462 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3ccc059390 con 0x7f3cd41561f0 2024-03-06T07:29:33.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:33.611 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:29:33.611 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:29:33.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.611 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:29:33.612 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:29:33.614 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.616 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.618 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:29:33.622 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:33.623 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:29:33.624 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:29:33.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b4d0 msgr2=0x7f3cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b4d0 0x7f3cb006d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4036000 tx=0x7f3cc403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 msgr2=0x7f3cd40ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ade00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc007f20 tx=0x7f3ccc00d560 comp rx=0 tx=0).stop 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 shutdown_connections 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b4d0 0x7f3cb006d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd41561f0 0x7f3cd40ade00 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd40cc9d0 0x7f3cd4158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 --2- 172.21.15.97:0/1559424462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd40aebf0 0x7f3cd4157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.611+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 >> 172.21.15.97:0/1559424462 conn(0x7f3cd406afc0 msgr2=0x7f3cd40b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.612+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 shutdown_connections 2024-03-06T07:29:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.612+0000 7f3cd9e00640 1 -- 172.21.15.97:0/1559424462 wait complete. 2024-03-06T07:29:33.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.815+0000 7f3826200640 1 Processor -- start 2024-03-06T07:29:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.816+0000 7f3826200640 1 -- start start 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.817+0000 7f3826200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f382015f9d0 0x7f382015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.817+0000 7f3826200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006cef0 con 0x7f3820144c00 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.817+0000 7f3826200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006d1d0 con 0x7f3820070b10 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.817+0000 7f3826200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382006d4b0 con 0x7f382015f9d0 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3820144c00 0x7f382006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53526/0 (socket says 172.21.15.97:53526) 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381f400640 1 -- 172.21.15.97:0/3180182473 learned_addr learned my addr 172.21.15.97:0/3180182473 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f3816a00640 1 --2- 172.21.15.97:0/3180182473 >> v2:172.21.15.135:3300/0 conn(0x7f382015f9d0 0x7f382015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430087975 0 0) 0x7f382006cef0 con 0x7f3820144c00 2024-03-06T07:29:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804003520 con 0x7f3820144c00 2024-03-06T07:29:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440525779 0 0) 0x7f3804003520 con 0x7f3820144c00 2024-03-06T07:29:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 >> v2:172.21.15.135:3300/0 conn(0x7f382015f9d0 msgr2=0x7f382015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 --2- 172.21.15.97:0/3180182473 >> v2:172.21.15.135:3300/0 conn(0x7f382015f9d0 0x7f382015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 >> v1:172.21.15.135:6789/0 conn(0x7f3820070b10 legacy=0x7f3820070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382006d790 con 0x7f3820144c00 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.818+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3810001670 con 0x7f3820144c00 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 >> v1:172.21.15.97:6789/0 conn(0x7f3820144c00 legacy=0x7f382006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38040039b0 0x7f3804003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f3804004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38040051c0 0x7f380400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382006d4b0 con 0x7f3804004800 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380400c060 con 0x7f38040051c0 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380400c260 con 0x7f38040039b0 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f3816a00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f3804004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381f400640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38040039b0 0x7f3804003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f3816a00640 1 -- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38040039b0 msgr2=0x7f3804003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381fe00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38040051c0 0x7f380400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f3816a00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38040039b0 0x7f3804003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f3816a00640 1 -- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38040051c0 msgr2=0x7f380400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f3816a00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38040051c0 0x7f380400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f3816a00640 1 -- 172.21.15.97:0/3180182473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804003520 con 0x7f3804004800 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381fe00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38040051c0 0x7f380400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.819+0000 7f381fe00640 1 -- 172.21.15.97:0/3180182473 reap_dead start 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.820+0000 7f3816a00640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f3804004c80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3808009ac0 tx=0x7f3804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9023129d1cac17c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808039070 con 0x7f3804004800 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380802c6b0 con 0x7f3804004800 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f381ea00640 1 -- 172.21.15.97:0/3180182473 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808009690 con 0x7f3804004800 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 -- 172.21.15.97:0/3180182473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 msgr2=0x7f3804004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f3804004c80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3808009ac0 tx=0x7f3804005830 comp rx=0 tx=0).stop 2024-03-06T07:29:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 -- 172.21.15.97:0/3180182473 shutdown_connections 2024-03-06T07:29:33.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 --2- 172.21.15.97:0/3180182473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f3804004c80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 -- 172.21.15.97:0/3180182473 >> 172.21.15.97:0/3180182473 conn(0x7f38200b1730 msgr2=0x7f38200b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 -- 172.21.15.97:0/3180182473 shutdown_connections 2024-03-06T07:29:33.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.821+0000 7f3826200640 1 -- 172.21.15.97:0/3180182473 wait complete. 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.823+0000 7f3826200640 1 Processor -- start 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.823+0000 7f3826200640 1 -- start start 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f3826200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f382013c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f3826200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f382015af70 0x7f382015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f3826200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f382013cf80 0x7f382013d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f3826200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382006ccd0 con 0x7f3804004800 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f3826200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382006cb50 con 0x7f382015af70 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f3826200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382006df90 con 0x7f382013cf80 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f381f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f382013c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f381f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f382013c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39526/0 (socket says 172.21.15.97:39526) 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.824+0000 7f381f400640 1 -- 172.21.15.97:0/1896972366 learned_addr learned my addr 172.21.15.97:0/1896972366 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f3816a00640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f382015af70 0x7f382015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381fe00640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f382013cf80 0x7f382013d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381f400640 1 -- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f382013cf80 msgr2=0x7f382013d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381f400640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f382013cf80 0x7f382013d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381f400640 1 -- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f382015af70 msgr2=0x7f382015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381f400640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f382015af70 0x7f382015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381f400640 1 -- 172.21.15.97:0/1896972366 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382013f0f0 con 0x7f3804004800 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f3816a00640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f382015af70 0x7f382015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f381f400640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f382013c580 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f381000d810 tx=0x7f381000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810019070 con 0x7f3804004800 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38100074e0 con 0x7f3804004800 2024-03-06T07:29:33.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810004a00 con 0x7f3804004800 2024-03-06T07:29:33.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.825+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382013f3d0 con 0x7f3804004800 2024-03-06T07:29:33.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.826+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382013f990 con 0x7f3804004800 2024-03-06T07:29:33.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.827+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382004f330 con 0x7f3804004800 2024-03-06T07:29:33.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.827+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3810004ba0 con 0x7f3804004800 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.828+0000 7f3817e00640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f37fc06b4d0 0x7f37fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.828+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f38100898f0 con 0x7f3804004800 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.828+0000 7f3816a00640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f37fc06b4d0 0x7f37fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.829+0000 7f3816a00640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f37fc06b4d0 0x7f37fc06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3808036000 tx=0x7f3808030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:33.833+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3810058b80 con 0x7f3804004800 2024-03-06T07:29:34.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.030+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3820072890 con 0x7f3804004800 2024-03-06T07:29:34.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.032+0000 7f3817e00640 1 -- 172.21.15.97:0/1896972366 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f3810058480 con 0x7f3804004800 2024-03-06T07:29:34.034 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:34.035 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":42,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:33.150487+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:34.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f37fc06b4d0 msgr2=0x7f37fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f37fc06b4d0 0x7f37fc06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3808036000 tx=0x7f3808030000 comp rx=0 tx=0).stop 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 msgr2=0x7f382013c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f382013c580 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f381000d810 tx=0x7f381000dcd0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 shutdown_connections 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f37fc06b4d0 0x7f37fc06d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f382013cf80 0x7f382013d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f382015af70 0x7f382015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 --2- 172.21.15.97:0/1896972366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3804004800 0x7f382013c580 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 >> 172.21.15.97:0/1896972366 conn(0x7f38200b1730 msgr2=0x7f38200afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 shutdown_connections 2024-03-06T07:29:34.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.035+0000 7f3826200640 1 -- 172.21.15.97:0/1896972366 wait complete. 2024-03-06T07:29:34.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:34.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:34.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.251+0000 7ff001200640 1 Processor -- start 2024-03-06T07:29:34.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.251+0000 7ff001200640 1 -- start start 2024-03-06T07:29:34.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.252+0000 7ff001200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feffc15fc10 0x7feffc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.252+0000 7ff001200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feffc0b3f70 0x7feffc0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7ff001200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0b46c0 con 0x7feffc15fff0 2024-03-06T07:29:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7ff001200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc0b49a0 con 0x7feffc0b3f70 2024-03-06T07:29:34.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7ff001200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc072c90 con 0x7feffc15fc10 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7feff9400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feffc15fff0 0x7feffc160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47996/0 (socket says 172.21.15.97:47996) 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7feff9400640 1 -- 172.21.15.97:0/1259343 learned_addr learned my addr 172.21.15.97:0/1259343 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7feffa800640 1 --2- 172.21.15.97:0/1259343 >> v2:172.21.15.97:3300/0 conn(0x7feffc0b3f70 0x7feffc0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.253+0000 7feff9e00640 1 --2- 172.21.15.97:0/1259343 >> v2:172.21.15.135:3300/0 conn(0x7feffc15fc10 0x7feffc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.254+0000 7feff8a00640 1 -- 172.21.15.97:0/1259343 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345141322 0 0) 0x7feffc0b46c0 con 0x7feffc15fff0 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.254+0000 7feff8a00640 1 -- 172.21.15.97:0/1259343 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe0003520 con 0x7feffc15fff0 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.254+0000 7feffa800640 1 -- 172.21.15.97:0/1259343 >> v2:172.21.15.135:3300/0 conn(0x7feffc15fc10 msgr2=0x7feffc070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.254+0000 7feffa800640 1 --2- 172.21.15.97:0/1259343 >> v2:172.21.15.135:3300/0 conn(0x7feffc15fc10 0x7feffc070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.254+0000 7feffa800640 1 -- 172.21.15.97:0/1259343 >> v1:172.21.15.97:6789/0 conn(0x7feffc15fff0 legacy=0x7feffc160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.254+0000 7feffa800640 1 -- 172.21.15.97:0/1259343 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc09dd80 con 0x7feffc0b3f70 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.255+0000 7feffa800640 1 --2- 172.21.15.97:0/1259343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc0ce4a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7feffc0cf770 tx=0x7feff402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da36c68d1b8ca0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.255+0000 7feff8a00640 1 -- 172.21.15.97:0/1259343 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4039070 con 0x7feffc0b3f70 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.255+0000 7feff8a00640 1 -- 172.21.15.97:0/1259343 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff402cc80 con 0x7feffc0b3f70 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.255+0000 7feff8a00640 1 -- 172.21.15.97:0/1259343 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4034640 con 0x7feffc0b3f70 2024-03-06T07:29:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 -- 172.21.15.97:0/1259343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 msgr2=0x7feffc0ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 --2- 172.21.15.97:0/1259343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc0ce4a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7feffc0cf770 tx=0x7feff402bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 -- 172.21.15.97:0/1259343 shutdown_connections 2024-03-06T07:29:34.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 --2- 172.21.15.97:0/1259343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc0ce4a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 --2- 172.21.15.97:0/1259343 >> v2:172.21.15.135:3300/0 conn(0x7feffc15fc10 0x7feffc070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 -- 172.21.15.97:0/1259343 >> 172.21.15.97:0/1259343 conn(0x7feffc0b1940 msgr2=0x7feffc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 -- 172.21.15.97:0/1259343 shutdown_connections 2024-03-06T07:29:34.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.256+0000 7ff001200640 1 -- 172.21.15.97:0/1259343 wait complete. 2024-03-06T07:29:34.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.257+0000 7ff001200640 1 Processor -- start 2024-03-06T07:29:34.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.258+0000 7ff001200640 1 -- start start 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.258+0000 7ff001200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc09a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7ff001200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feffc15fc10 0x7feffc1490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7ff001200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feffc15fff0 0x7feffc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7ff001200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15c460 con 0x7feffc0b3f70 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7ff001200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15c2e0 con 0x7feffc15fff0 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7ff001200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0ce9e0 con 0x7feffc15fc10 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc09a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc09a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44606/0 (socket says 172.21.15.97:44606) 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 -- 172.21.15.97:0/3783991647 learned_addr learned my addr 172.21.15.97:0/3783991647 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 -- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feffc15fc10 msgr2=0x7feffc1490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feffa800640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feffc15fff0 0x7feffc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feffc15fc10 0x7feffc1490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 -- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feffc15fff0 msgr2=0x7feffc06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feffc15fff0 0x7feffc06d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.259+0000 7feff9e00640 1 -- 172.21.15.97:0/3783991647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc09b2d0 con 0x7feffc0b3f70 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.260+0000 7feff9e00640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc09a9f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9e0 tx=0x7fefec00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.260+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec017070 con 0x7feffc0b3f70 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.260+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec00c9d0 con 0x7feffc0b3f70 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.260+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec00cdc0 con 0x7feffc0b3f70 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.260+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc09b5b0 con 0x7feffc0b3f70 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.260+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc09bb70 con 0x7feffc0b3f70 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.261+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc04f330 con 0x7feffc0b3f70 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.262+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefec01a070 con 0x7feffc0b3f70 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.263+0000 7feff2000640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fefdc06b4d0 0x7fefdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.263+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fefec08a330 con 0x7feffc0b3f70 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.263+0000 7feff9400640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fefdc06b4d0 0x7fefdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.264+0000 7feff9400640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fefdc06b4d0 0x7fefdc06d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fefe80011e0 tx=0x7fefe8007910 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.267+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefec059460 con 0x7feffc0b3f70 2024-03-06T07:29:34.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.447+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feffc06ef30 con 0x7feffc0b3f70 2024-03-06T07:29:34.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.448+0000 7feff2000640 1 -- 172.21.15.97:0/3783991647 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefec058d60 con 0x7feffc0b3f70 2024-03-06T07:29:34.450 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.450+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fefdc06b4d0 msgr2=0x7fefdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.450+0000 7ff001200640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fefdc06b4d0 0x7fefdc06d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fefe80011e0 tx=0x7fefe8007910 comp rx=0 tx=0).stop 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 msgr2=0x7feffc09a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc09a9f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9e0 tx=0x7fefec00eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 shutdown_connections 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fefdc06b4d0 0x7fefdc06d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feffc15fff0 0x7feffc06d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feffc15fc10 0x7feffc1490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 --2- 172.21.15.97:0/3783991647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feffc0b3f70 0x7feffc09a9f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 >> 172.21.15.97:0/3783991647 conn(0x7feffc0b1940 msgr2=0x7feffc0ad800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 shutdown_connections 2024-03-06T07:29:34.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.451+0000 7ff001200640 1 -- 172.21.15.97:0/3783991647 wait complete. 2024-03-06T07:29:34.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:29:34.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.668+0000 7f816b000640 1 Processor -- start 2024-03-06T07:29:34.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.668+0000 7f816b000640 1 -- start start 2024-03-06T07:29:34.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.670+0000 7f816b000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f81640af1c0 0x7f816415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.670+0000 7f816b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8164070d20 0x7f816406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.670+0000 7f816b000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8164071d30 con 0x7f816406d1a0 2024-03-06T07:29:34.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.670+0000 7f816b000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8164071490 con 0x7f81640af1c0 2024-03-06T07:29:34.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.670+0000 7f816b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8164071770 con 0x7f8164070d20 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.671+0000 7f8163e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f81640af1c0 0x7f816415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.671+0000 7f8169000640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f816406d1a0 0x7f816406e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:57170/0 (socket says 172.21.15.97:57170) 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.671+0000 7f8169000640 1 -- 172.21.15.97:0/1915680173 learned_addr learned my addr 172.21.15.97:0/1915680173 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.671+0000 7f815b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8164070d20 0x7f816406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.671+0000 7f8163e00640 1 -- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8164070d20 msgr2=0x7f816406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.672+0000 7f8163e00640 1 --2- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8164070d20 0x7f816406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.672+0000 7f8163e00640 1 -- 172.21.15.97:0/1915680173 >> v1:172.21.15.135:6789/0 conn(0x7f816406d1a0 legacy=0x7f816406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.672+0000 7f8163e00640 1 -- 172.21.15.97:0/1915680173 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164071a50 con 0x7f81640af1c0 2024-03-06T07:29:34.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.673+0000 7f8163e00640 1 --2- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81640af1c0 0x7f816415eff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81540097a0 tx=0x7f815402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6600c31a60106463 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.674+0000 7f8163400640 1 -- 172.21.15.97:0/1915680173 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154035020 con 0x7f81640af1c0 2024-03-06T07:29:34.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.674+0000 7f8163400640 1 -- 172.21.15.97:0/1915680173 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8154031810 con 0x7f81640af1c0 2024-03-06T07:29:34.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.674+0000 7f8163400640 1 -- 172.21.15.97:0/1915680173 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154031be0 con 0x7f81640af1c0 2024-03-06T07:29:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 -- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81640af1c0 msgr2=0x7f816415eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 --2- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81640af1c0 0x7f816415eff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81540097a0 tx=0x7f815402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 -- 172.21.15.97:0/1915680173 shutdown_connections 2024-03-06T07:29:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 --2- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8164070d20 0x7f816406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 --2- 172.21.15.97:0/1915680173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81640af1c0 0x7f816415eff0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 -- 172.21.15.97:0/1915680173 >> 172.21.15.97:0/1915680173 conn(0x7f81640b1730 msgr2=0x7f81640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 -- 172.21.15.97:0/1915680173 shutdown_connections 2024-03-06T07:29:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.675+0000 7f816b000640 1 -- 172.21.15.97:0/1915680173 wait complete. 2024-03-06T07:29:34.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.677+0000 7f816b000640 1 Processor -- start 2024-03-06T07:29:34.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.677+0000 7f816b000640 1 -- start start 2024-03-06T07:29:34.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f816b000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f816406d1a0 0x7f8164156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f816b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 0x7f81641574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f816b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f81640af1c0 0x7f8164159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f816b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816406ebe0 con 0x7f8164070d20 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f816b000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81640711a0 con 0x7f816406d1a0 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f816b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816406ed60 con 0x7f81640af1c0 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 0x7f81641574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 0x7f81641574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44622/0 (socket says 172.21.15.97:44622) 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 -- 172.21.15.97:0/1640307032 learned_addr learned my addr 172.21.15.97:0/1640307032 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f8163e00640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f816406d1a0 0x7f8164156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f8169000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f81640af1c0 0x7f8164159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 -- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f81640af1c0 msgr2=0x7f8164159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f81640af1c0 0x7f8164159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 -- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f816406d1a0 msgr2=0x7f8164156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f816406d1a0 0x7f8164156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.679+0000 7f815b400640 1 -- 172.21.15.97:0/1640307032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164158430 con 0x7f8164070d20 2024-03-06T07:29:34.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.680+0000 7f815b400640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 0x7f81641574d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f814c00e9e0 tx=0x7f814c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.680+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c017070 con 0x7f8164070d20 2024-03-06T07:29:34.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.680+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c00c9d0 con 0x7f8164070d20 2024-03-06T07:29:34.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.680+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8164157e70 con 0x7f8164070d20 2024-03-06T07:29:34.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.680+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c00cdc0 con 0x7f8164070d20 2024-03-06T07:29:34.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.680+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81641564b0 con 0x7f8164070d20 2024-03-06T07:29:34.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.681+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816404f330 con 0x7f8164070d20 2024-03-06T07:29:34.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.682+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f814c01a070 con 0x7f8164070d20 2024-03-06T07:29:34.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.683+0000 7f8160c00640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f814006b4d0 0x7f814006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.683+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f814c089f90 con 0x7f8164070d20 2024-03-06T07:29:34.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.683+0000 7f8163e00640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f814006b4d0 0x7f814006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.684+0000 7f8163e00640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f814006b4d0 0x7f814006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8154036000 tx=0x7f815403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.687+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f814c0590c0 con 0x7f8164070d20 2024-03-06T07:29:34.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.884+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f81640ce8a0 con 0x7f8164070d20 2024-03-06T07:29:34.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.885+0000 7f8160c00640 1 -- 172.21.15.97:0/1640307032 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7f814c0589c0 con 0x7f8164070d20 2024-03-06T07:29:34.887 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:34.887 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:29:34.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.887+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f814006b4d0 msgr2=0x7f814006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.887+0000 7f816b000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f814006b4d0 0x7f814006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8154036000 tx=0x7f815403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.887+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 msgr2=0x7f81641574d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.887+0000 7f816b000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 0x7f81641574d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f814c00e9e0 tx=0x7f814c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 shutdown_connections 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f814006b4d0 0x7f814006d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f81640af1c0 0x7f8164159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8164070d20 0x7f81641574d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 --2- 172.21.15.97:0/1640307032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f816406d1a0 0x7f8164156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 >> 172.21.15.97:0/1640307032 conn(0x7f81640b1730 msgr2=0x7f81640b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 shutdown_connections 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:34.888+0000 7f816b000640 1 -- 172.21.15.97:0/1640307032 wait complete. 2024-03-06T07:29:34.908 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:29:34.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.102+0000 7f2dcca00640 1 Processor -- start 2024-03-06T07:29:35.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.103+0000 7f2dcca00640 1 -- start start 2024-03-06T07:29:35.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dcca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2dc80b5680 0x7f2dc815f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dcca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2dc80b5a60 0x7f2dc815d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dcca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc815e090 con 0x7f2dc815db80 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dcca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc815f670 con 0x7f2dc80b5a60 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dcca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc815e370 con 0x7f2dc80b5680 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dc7200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2dc815db80 0x7f2dc806ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48006/0 (socket says 172.21.15.97:48006) 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.104+0000 7f2dc7200640 1 -- 172.21.15.97:0/2862115111 learned_addr learned my addr 172.21.15.97:0/2862115111 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5e00640 1 --2- 172.21.15.97:0/2862115111 >> v2:172.21.15.97:3300/0 conn(0x7f2dc80b5a60 0x7f2dc815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc6800640 1 --2- 172.21.15.97:0/2862115111 >> v2:172.21.15.135:3300/0 conn(0x7f2dc80b5680 0x7f2dc815f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5400640 1 -- 172.21.15.97:0/2862115111 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118090740 0 0) 0x7f2dc815e090 con 0x7f2dc815db80 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5400640 1 -- 172.21.15.97:0/2862115111 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac003520 con 0x7f2dc815db80 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5e00640 1 -- 172.21.15.97:0/2862115111 >> v2:172.21.15.135:3300/0 conn(0x7f2dc80b5680 msgr2=0x7f2dc815f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5e00640 1 --2- 172.21.15.97:0/2862115111 >> v2:172.21.15.135:3300/0 conn(0x7f2dc80b5680 0x7f2dc815f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5e00640 1 -- 172.21.15.97:0/2862115111 >> v1:172.21.15.97:6789/0 conn(0x7f2dc815db80 legacy=0x7f2dc806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.105+0000 7f2dc5e00640 1 -- 172.21.15.97:0/2862115111 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc815e650 con 0x7f2dc80b5a60 2024-03-06T07:29:35.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.106+0000 7f2dc5e00640 1 --2- 172.21.15.97:0/2862115111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc80b5a60 0x7f2dc815d640 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2dc806bd90 tx=0x7f2db802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443485831781ed6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.106+0000 7f2dc5400640 1 -- 172.21.15.97:0/2862115111 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8039070 con 0x7f2dc80b5a60 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.106+0000 7f2dc5400640 1 -- 172.21.15.97:0/2862115111 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db802cc80 con 0x7f2dc80b5a60 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.106+0000 7f2dc5400640 1 -- 172.21.15.97:0/2862115111 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8034660 con 0x7f2dc80b5a60 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 -- 172.21.15.97:0/2862115111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc80b5a60 msgr2=0x7f2dc815d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2862115111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc80b5a60 0x7f2dc815d640 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2dc806bd90 tx=0x7f2db802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 -- 172.21.15.97:0/2862115111 shutdown_connections 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2862115111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc80b5a60 0x7f2dc815d640 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2862115111 >> v2:172.21.15.135:3300/0 conn(0x7f2dc80b5680 0x7f2dc815f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 -- 172.21.15.97:0/2862115111 >> 172.21.15.97:0/2862115111 conn(0x7f2dc8073130 msgr2=0x7f2dc809a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 -- 172.21.15.97:0/2862115111 shutdown_connections 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.107+0000 7f2dcca00640 1 -- 172.21.15.97:0/2862115111 wait complete. 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.109+0000 7f2dcca00640 1 Processor -- start 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.109+0000 7f2dcca00640 1 -- start start 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dcca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2dc80b5680 0x7f2dc814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dcca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2dc80b5a60 0x7f2dc8150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dcca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 0x7f2dc814dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dcca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc815c4a0 con 0x7f2dc815db80 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dcca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc80b5ee0 con 0x7f2dc80b5a60 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dcca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc815c620 con 0x7f2dc80b5680 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 0x7f2dc814dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 0x7f2dc814dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44646/0 (socket says 172.21.15.97:44646) 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 -- 172.21.15.97:0/2658330200 learned_addr learned my addr 172.21.15.97:0/2658330200 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:35.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 -- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2dc80b5680 msgr2=0x7f2dc814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2dc80b5680 0x7f2dc814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 -- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2dc80b5a60 msgr2=0x7f2dc8150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc5e00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2dc80b5a60 0x7f2dc8150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2dc80b5a60 0x7f2dc8150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.110+0000 7f2dc7200640 1 -- 172.21.15.97:0/2658330200 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc814e250 con 0x7f2dc815db80 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.111+0000 7f2dc7200640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 0x7f2dc814dab0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c400 tx=0x7f2dc000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.111+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0017070 con 0x7f2dc815db80 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.111+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0005ce0 con 0x7f2dc815db80 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.111+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc814e530 con 0x7f2dc815db80 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.111+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00072d0 con 0x7f2dc815db80 2024-03-06T07:29:35.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.111+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc814eaf0 con 0x7f2dc815db80 2024-03-06T07:29:35.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.112+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80b20f0 con 0x7f2dc815db80 2024-03-06T07:29:35.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.113+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dc001a070 con 0x7f2dc815db80 2024-03-06T07:29:35.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.114+0000 7f2dbea00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2da406b4d0 0x7f2da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.114+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2dc00892c0 con 0x7f2dc815db80 2024-03-06T07:29:35.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.114+0000 7f2dc6800640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2da406b4d0 0x7f2da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.114+0000 7f2dc6800640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2da406b4d0 0x7f2da406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2db000a810 tx=0x7f2db0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.118+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dc80b20f0 con 0x7f2dc815db80 2024-03-06T07:29:35.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:35.313+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2dc8068e40 con 0x7f2dc815db80 2024-03-06T07:29:36.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.165+0000 7f2dbea00640 1 -- 172.21.15.97:0/2658330200 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2dc8068e40 con 0x7f2dc815db80 2024-03-06T07:29:36.167 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:29:36.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2da406b4d0 msgr2=0x7f2da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2da406b4d0 0x7f2da406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2db000a810 tx=0x7f2db0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 msgr2=0x7f2dc814dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 0x7f2dc814dab0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c400 tx=0x7f2dc000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 shutdown_connections 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2da406b4d0 0x7f2da406d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2dc815db80 0x7f2dc814dab0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2dc80b5a60 0x7f2dc8150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 --2- 172.21.15.97:0/2658330200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2dc80b5680 0x7f2dc814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.168+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 >> 172.21.15.97:0/2658330200 conn(0x7f2dc8073130 msgr2=0x7f2dc80b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.169+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 shutdown_connections 2024-03-06T07:29:36.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.169+0000 7f2dcca00640 1 -- 172.21.15.97:0/2658330200 wait complete. 2024-03-06T07:29:36.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:29:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.391+0000 7f6214000640 1 Processor -- start 2024-03-06T07:29:36.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.391+0000 7f6214000640 1 -- start start 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6214000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f620c06fef0 0x7f620c06f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6214000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f620c15e3e0 0x7f620c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6214000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0d1260 con 0x7f620c15e000 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6214000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620c0d1020 con 0x7f620c06fef0 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6214000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620c06c0d0 con 0x7f620c15e3e0 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6211600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f620c15e000 0x7f620c06f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48018/0 (socket says 172.21.15.97:48018) 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6210c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f620c06fef0 0x7f620c06f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6211600640 1 -- 172.21.15.97:0/3072570577 learned_addr learned my addr 172.21.15.97:0/3072570577 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6212000640 1 --2- 172.21.15.97:0/3072570577 >> v2:172.21.15.135:3300/0 conn(0x7f620c15e3e0 0x7f620c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6210c00640 1 -- 172.21.15.97:0/3072570577 >> v2:172.21.15.135:3300/0 conn(0x7f620c15e3e0 msgr2=0x7f620c0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f6210c00640 1 --2- 172.21.15.97:0/3072570577 >> v2:172.21.15.135:3300/0 conn(0x7f620c15e3e0 0x7f620c0d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.394+0000 7f6210c00640 1 -- 172.21.15.97:0/3072570577 >> v1:172.21.15.97:6789/0 conn(0x7f620c15e000 legacy=0x7f620c06f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.394+0000 7f6210c00640 1 -- 172.21.15.97:0/3072570577 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c06c320 con 0x7f620c06fef0 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.394+0000 7f6212000640 1 --2- 172.21.15.97:0/3072570577 >> v2:172.21.15.135:3300/0 conn(0x7f620c15e3e0 0x7f620c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.393+0000 7f620be00640 1 -- 172.21.15.97:0/3072570577 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544203149 0 0) 0x7f620c0d1260 con 0x7f620c15e000 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.395+0000 7f6210c00640 1 --2- 172.21.15.97:0/3072570577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c06f130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f620c072af0 tx=0x7f61f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8050e73e7db9e115 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.395+0000 7f620be00640 1 -- 172.21.15.97:0/3072570577 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8035020 con 0x7f620c06fef0 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.395+0000 7f620be00640 1 -- 172.21.15.97:0/3072570577 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8031830 con 0x7f620c06fef0 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.395+0000 7f620be00640 1 -- 172.21.15.97:0/3072570577 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8031c00 con 0x7f620c06fef0 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.395+0000 7f6214000640 1 -- 172.21.15.97:0/3072570577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 msgr2=0x7f620c06f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.395+0000 7f6214000640 1 --2- 172.21.15.97:0/3072570577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c06f130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f620c072af0 tx=0x7f61f802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.396+0000 7f6214000640 1 -- 172.21.15.97:0/3072570577 shutdown_connections 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.396+0000 7f6214000640 1 --2- 172.21.15.97:0/3072570577 >> v2:172.21.15.135:3300/0 conn(0x7f620c15e3e0 0x7f620c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.396+0000 7f6214000640 1 --2- 172.21.15.97:0/3072570577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c06f130 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.396+0000 7f6214000640 1 -- 172.21.15.97:0/3072570577 >> 172.21.15.97:0/3072570577 conn(0x7f620c0b3800 msgr2=0x7f620c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.396+0000 7f6214000640 1 -- 172.21.15.97:0/3072570577 shutdown_connections 2024-03-06T07:29:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.396+0000 7f6214000640 1 -- 172.21.15.97:0/3072570577 wait complete. 2024-03-06T07:29:36.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.397+0000 7f6214000640 1 Processor -- start 2024-03-06T07:29:36.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 -- start start 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f620c15e000 0x7f620c14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f620c15e3e0 0x7f620c14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0ce4c0 con 0x7f620c06fef0 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0ce340 con 0x7f620c15e3e0 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.398+0000 7f6214000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0ce640 con 0x7f620c15e000 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44660/0 (socket says 172.21.15.97:44660) 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 -- 172.21.15.97:0/574306321 learned_addr learned my addr 172.21.15.97:0/574306321 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 -- 172.21.15.97:0/574306321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f620c15e000 msgr2=0x7f620c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f620c15e000 0x7f620c14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 -- 172.21.15.97:0/574306321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f620c15e3e0 msgr2=0x7f620c14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f620c15e3e0 0x7f620c14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 -- 172.21.15.97:0/574306321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c147f60 con 0x7f620c06fef0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.399+0000 7f6211600640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c156e70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c4d0 tx=0x7f61fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.400+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc018070 con 0x7f620c06fef0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.400+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc007500 con 0x7f620c06fef0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.400+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc004a20 con 0x7f620c06fef0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.400+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c160490 con 0x7f620c06fef0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.400+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c160a50 con 0x7f620c06fef0 2024-03-06T07:29:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.401+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c04f330 con 0x7f620c06fef0 2024-03-06T07:29:36.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.403+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61fc004bc0 con 0x7f620c06fef0 2024-03-06T07:29:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.403+0000 7f6209600640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61ec06b4d0 0x7f61ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.404+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f61fc0895e0 con 0x7f620c06fef0 2024-03-06T07:29:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.404+0000 7f6210c00640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61ec06b4d0 0x7f61ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.404+0000 7f6210c00640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61ec06b4d0 0x7f61ec06d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f61f8036000 tx=0x7f61f8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.407+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61fc058720 con 0x7f620c06fef0 2024-03-06T07:29:36.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:36.602+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f620c0b7c50 con 0x7f620c06fef0 2024-03-06T07:29:37.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:37.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:37.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.168+0000 7f6209600640 1 -- 172.21.15.97:0/574306321 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f61fc058020 con 0x7f620c06fef0 2024-03-06T07:29:37.170 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.172+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61ec06b4d0 msgr2=0x7f61ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61ec06b4d0 0x7f61ec06d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f61f8036000 tx=0x7f61f8030000 comp rx=0 tx=0).stop 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 msgr2=0x7f620c156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c156e70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c4d0 tx=0x7f61fc00c990 comp rx=0 tx=0).stop 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 shutdown_connections 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61ec06b4d0 0x7f61ec06d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f620c15e3e0 0x7f620c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f620c15e000 0x7f620c14a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 --2- 172.21.15.97:0/574306321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f620c06fef0 0x7f620c156e70 secure :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c4d0 tx=0x7f61fc00c990 comp rx=0 tx=0).stop 2024-03-06T07:29:37.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.173+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 >> 172.21.15.97:0/574306321 conn(0x7f620c0b3800 msgr2=0x7f620c0d15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.176+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 shutdown_connections 2024-03-06T07:29:37.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.176+0000 7f6214000640 1 -- 172.21.15.97:0/574306321 wait complete. 2024-03-06T07:29:37.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:29:37.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.390+0000 7ff3aca00640 1 Processor -- start 2024-03-06T07:29:37.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.390+0000 7ff3aca00640 1 -- start start 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3aca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff3a815cb90 0x7ff3a806d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3aca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff3a806e060 0x7ff3a806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3aca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a815cf70 con 0x7ff3a8163800 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3aca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a80b3890 con 0x7ff3a815cb90 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3aca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a806ea10 con 0x7ff3a806e060 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3a5e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff3a815cb90 0x7ff3a806d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3a5400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff3a8163800 0x7ff3a80b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48026/0 (socket says 172.21.15.97:48026) 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.392+0000 7ff3a5400640 1 -- 172.21.15.97:0/4293364024 learned_addr learned my addr 172.21.15.97:0/4293364024 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.393+0000 7ff3a6800640 1 --2- 172.21.15.97:0/4293364024 >> v2:172.21.15.135:3300/0 conn(0x7ff3a806e060 0x7ff3a806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.393+0000 7ff3a4a00640 1 -- 172.21.15.97:0/4293364024 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7733939 0 0) 0x7ff3a815cf70 con 0x7ff3a8163800 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.393+0000 7ff3a5e00640 1 -- 172.21.15.97:0/4293364024 >> v2:172.21.15.135:3300/0 conn(0x7ff3a806e060 msgr2=0x7ff3a806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.393+0000 7ff3a5e00640 1 --2- 172.21.15.97:0/4293364024 >> v2:172.21.15.135:3300/0 conn(0x7ff3a806e060 0x7ff3a806e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.393+0000 7ff3a5e00640 1 -- 172.21.15.97:0/4293364024 >> v1:172.21.15.97:6789/0 conn(0x7ff3a8163800 legacy=0x7ff3a80b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.393+0000 7ff3a5e00640 1 -- 172.21.15.97:0/4293364024 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80b35b0 con 0x7ff3a815cb90 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.394+0000 7ff3a5e00640 1 --2- 172.21.15.97:0/4293364024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a806d6c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff3a806c670 tx=0x7ff39802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631d7e173e0de58f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.394+0000 7ff3a4a00640 1 -- 172.21.15.97:0/4293364024 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398035020 con 0x7ff3a815cb90 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.394+0000 7ff3a4a00640 1 -- 172.21.15.97:0/4293364024 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398031830 con 0x7ff3a815cb90 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.394+0000 7ff3a4a00640 1 -- 172.21.15.97:0/4293364024 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398031c00 con 0x7ff3a815cb90 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 -- 172.21.15.97:0/4293364024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 msgr2=0x7ff3a806d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 --2- 172.21.15.97:0/4293364024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a806d6c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff3a806c670 tx=0x7ff39802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 -- 172.21.15.97:0/4293364024 shutdown_connections 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 --2- 172.21.15.97:0/4293364024 >> v2:172.21.15.135:3300/0 conn(0x7ff3a806e060 0x7ff3a806e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 --2- 172.21.15.97:0/4293364024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a806d6c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 -- 172.21.15.97:0/4293364024 >> 172.21.15.97:0/4293364024 conn(0x7ff3a80b1940 msgr2=0x7ff3a80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 -- 172.21.15.97:0/4293364024 shutdown_connections 2024-03-06T07:29:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.395+0000 7ff3aca00640 1 -- 172.21.15.97:0/4293364024 wait complete. 2024-03-06T07:29:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.397+0000 7ff3aca00640 1 Processor -- start 2024-03-06T07:29:37.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.397+0000 7ff3aca00640 1 -- start start 2024-03-06T07:29:37.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3aca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3a806e060 0x7ff3a81f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3aca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a814ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3aca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3a8163800 0x7ff3a80b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3aca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a806d230 con 0x7ff3a815cb90 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3aca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a806c990 con 0x7ff3a806e060 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3aca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a80b3350 con 0x7ff3a8163800 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a814ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a814ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44680/0 (socket says 172.21.15.97:44680) 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 -- 172.21.15.97:0/737501357 learned_addr learned my addr 172.21.15.97:0/737501357 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5e00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3a806e060 0x7ff3a81f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 -- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3a8163800 msgr2=0x7ff3a80b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3a8163800 0x7ff3a80b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 -- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3a806e060 msgr2=0x7ff3a81f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3a806e060 0x7ff3a81f8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.398+0000 7ff3a5400640 1 -- 172.21.15.97:0/737501357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a81fba20 con 0x7ff3a815cb90 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.399+0000 7ff3a5400640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a814ebb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff39000c7c0 tx=0x7ff39000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.399+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390018070 con 0x7ff3a815cb90 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.399+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39000a640 con 0x7ff3a815cb90 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.399+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39000aa30 con 0x7ff3a815cb90 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.399+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a81fbd00 con 0x7ff3a815cb90 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.399+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a81fc2c0 con 0x7ff3a815cb90 2024-03-06T07:29:37.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.400+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a804f330 con 0x7ff3a815cb90 2024-03-06T07:29:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.401+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff39000abd0 con 0x7ff3a815cb90 2024-03-06T07:29:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.402+0000 7ff39e000640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff38406b500 0x7ff38406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.402+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff390089740 con 0x7ff3a815cb90 2024-03-06T07:29:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.402+0000 7ff3a5e00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff38406b500 0x7ff38406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.403+0000 7ff3a5e00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff38406b500 0x7ff38406d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff398036000 tx=0x7ff398008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.408+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff390058610 con 0x7ff3a815cb90 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:37.641+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff3a80b2530 con 0x7ff3a815cb90 2024-03-06T07:29:38.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.203+0000 7ff39e000640 1 -- 172.21.15.97:0/737501357 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff390057f10 con 0x7ff3a815cb90 2024-03-06T07:29:38.209 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:29:38.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.209+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff38406b500 msgr2=0x7ff38406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.209+0000 7ff3aca00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff38406b500 0x7ff38406d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff398036000 tx=0x7ff398008030 comp rx=0 tx=0).stop 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.209+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 msgr2=0x7ff3a814ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.209+0000 7ff3aca00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a814ebb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff39000c7c0 tx=0x7ff39000cc80 comp rx=0 tx=0).stop 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 shutdown_connections 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff38406b500 0x7ff38406d9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3a8163800 0x7ff3a80b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3a815cb90 0x7ff3a814ebb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 --2- 172.21.15.97:0/737501357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3a806e060 0x7ff3a81f8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 >> 172.21.15.97:0/737501357 conn(0x7ff3a80b1940 msgr2=0x7ff3a80b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 shutdown_connections 2024-03-06T07:29:38.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.210+0000 7ff3aca00640 1 -- 172.21.15.97:0/737501357 wait complete. 2024-03-06T07:29:38.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:38.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.414+0000 7f04fd400640 1 Processor -- start 2024-03-06T07:29:38.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.414+0000 7f04fd400640 1 -- start start 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04fd400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04f80af1c0 0x7f04f815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04fd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f8070d20 0x7f04f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04fd400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f8071d30 con 0x7f04f806d1a0 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04fd400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f8071490 con 0x7f04f80af1c0 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04fd400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f8071770 con 0x7f04f8070d20 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04f6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04f80af1c0 0x7f04f815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04f7200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f04f806d1a0 0x7f04f806e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48042/0 (socket says 172.21.15.97:48042) 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04f7200640 1 -- 172.21.15.97:0/3442805894 learned_addr learned my addr 172.21.15.97:0/3442805894 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04ede00640 1 --2- 172.21.15.97:0/3442805894 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f8070d20 0x7f04f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.417+0000 7f04f6800640 1 -- 172.21.15.97:0/3442805894 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f8070d20 msgr2=0x7f04f806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.417+0000 7f04f6800640 1 --2- 172.21.15.97:0/3442805894 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f8070d20 0x7f04f806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.416+0000 7f04f5e00640 1 -- 172.21.15.97:0/3442805894 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716213947 0 0) 0x7f04f0001490 con 0x7f04f806d1a0 2024-03-06T07:29:38.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.417+0000 7f04f6800640 1 -- 172.21.15.97:0/3442805894 >> v1:172.21.15.97:6789/0 conn(0x7f04f806d1a0 legacy=0x7f04f806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.417+0000 7f04f6800640 1 -- 172.21.15.97:0/3442805894 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8071a50 con 0x7f04f80af1c0 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.418+0000 7f04f6800640 1 --2- 172.21.15.97:0/3442805894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f80af1c0 0x7f04f815eff0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f04e80097a0 tx=0x7f04e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e60a5e43d0708d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.418+0000 7f04f5e00640 1 -- 172.21.15.97:0/3442805894 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8035020 con 0x7f04f80af1c0 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.418+0000 7f04f5e00640 1 -- 172.21.15.97:0/3442805894 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e8031830 con 0x7f04f80af1c0 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.418+0000 7f04f5e00640 1 -- 172.21.15.97:0/3442805894 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8031c00 con 0x7f04f80af1c0 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 -- 172.21.15.97:0/3442805894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f80af1c0 msgr2=0x7f04f815eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 --2- 172.21.15.97:0/3442805894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f80af1c0 0x7f04f815eff0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f04e80097a0 tx=0x7f04e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 -- 172.21.15.97:0/3442805894 shutdown_connections 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 --2- 172.21.15.97:0/3442805894 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f8070d20 0x7f04f806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 --2- 172.21.15.97:0/3442805894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f80af1c0 0x7f04f815eff0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 -- 172.21.15.97:0/3442805894 >> 172.21.15.97:0/3442805894 conn(0x7f04f80b1730 msgr2=0x7f04f80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 -- 172.21.15.97:0/3442805894 shutdown_connections 2024-03-06T07:29:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.419+0000 7f04fd400640 1 -- 172.21.15.97:0/3442805894 wait complete. 2024-03-06T07:29:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.420+0000 7f04fd400640 1 Processor -- start 2024-03-06T07:29:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 -- start start 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 0x7f04f8145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04f8070d20 0x7f04f8155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f80af1c0 0x7f04f8155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f806ebe0 con 0x7f04f806d1a0 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80711a0 con 0x7f04f8070d20 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.421+0000 7f04fd400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f806ed60 con 0x7f04f80af1c0 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 0x7f04f8145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 0x7f04f8145370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44694/0 (socket says 172.21.15.97:44694) 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 -- 172.21.15.97:0/2699811030 learned_addr learned my addr 172.21.15.97:0/2699811030 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 -- 172.21.15.97:0/2699811030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f80af1c0 msgr2=0x7f04f8155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f80af1c0 0x7f04f8155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 -- 172.21.15.97:0/2699811030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04f8070d20 msgr2=0x7f04f8155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04f8070d20 0x7f04f8155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 -- 172.21.15.97:0/2699811030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f80ceaa0 con 0x7f04f806d1a0 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.422+0000 7f04f6800640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 0x7f04f8145370 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f04e8000c00 tx=0x7f04e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.423+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8040070 con 0x7f04f806d1a0 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.423+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e8039a00 con 0x7f04f806d1a0 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.423+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8035040 con 0x7f04f806d1a0 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.423+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f80ced80 con 0x7f04f806d1a0 2024-03-06T07:29:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.423+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f815f230 con 0x7f04f806d1a0 2024-03-06T07:29:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.424+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f804f330 con 0x7f04f806d1a0 2024-03-06T07:29:38.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.425+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04e804b060 con 0x7f04f806d1a0 2024-03-06T07:29:38.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.426+0000 7f04ef400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04d406b4d0 0x7f04d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.426+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04e80abba0 con 0x7f04f806d1a0 2024-03-06T07:29:38.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.426+0000 7f04ede00640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04d406b4d0 0x7f04d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.426+0000 7f04ede00640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04d406b4d0 0x7f04d406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04f815ec40 tx=0x7f04e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.430+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04e807a980 con 0x7f04f806d1a0 2024-03-06T07:29:38.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.610+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f806e690 con 0x7f04f806d1a0 2024-03-06T07:29:38.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.612+0000 7f04ef400640 1 -- 172.21.15.97:0/2699811030 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16614 (secure 0 0 0) 0x7f04e807a280 con 0x7f04f806d1a0 2024-03-06T07:29:38.614 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:38.614 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":45,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:37.643239+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04d406b4d0 msgr2=0x7f04d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04d406b4d0 0x7f04d406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04f815ec40 tx=0x7f04e0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 msgr2=0x7f04f8145370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 0x7f04f8145370 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f04e8000c00 tx=0x7f04e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 shutdown_connections 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f04d406b4d0 0x7f04d406d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04f80af1c0 0x7f04f8155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04f8070d20 0x7f04f8155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 --2- 172.21.15.97:0/2699811030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04f806d1a0 0x7f04f8145370 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 >> 172.21.15.97:0/2699811030 conn(0x7f04f80b1730 msgr2=0x7f04f80b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 shutdown_connections 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.614+0000 7f04fd400640 1 -- 172.21.15.97:0/2699811030 wait complete. 2024-03-06T07:29:38.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:38.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.837+0000 7f5e4d400640 1 Processor -- start 2024-03-06T07:29:38.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.837+0000 7f5e4d400640 1 -- start start 2024-03-06T07:29:38.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.838+0000 7f5e4d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 0x7f5e48070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.838+0000 7f5e4d400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5e48071260 0x7f5e4806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e4d400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e480717c0 con 0x7f5e4806d090 2024-03-06T07:29:38.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e4d400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4806d4c0 con 0x7f5e48071260 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e4d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48071aa0 con 0x7f5e480cf430 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e47200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5e4806d090 0x7f5e48165f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48048/0 (socket says 172.21.15.97:48048) 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e47200640 1 -- 172.21.15.97:0/4279462370 learned_addr learned my addr 172.21.15.97:0/4279462370 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e46800640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 0x7f5e48070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e3de00640 1 --2- 172.21.15.97:0/4279462370 >> v2:172.21.15.135:3300/0 conn(0x7f5e48071260 0x7f5e4806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317033835 0 0) 0x7f5e480717c0 con 0x7f5e4806d090 2024-03-06T07:29:38.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.839+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2c003520 con 0x7f5e4806d090 2024-03-06T07:29:38.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817396015 0 0) 0x7f5e2c003520 con 0x7f5e4806d090 2024-03-06T07:29:38.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 msgr2=0x7f5e48070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 0x7f5e48070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 >> v2:172.21.15.135:3300/0 conn(0x7f5e48071260 msgr2=0x7f5e4806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 --2- 172.21.15.97:0/4279462370 >> v2:172.21.15.135:3300/0 conn(0x7f5e48071260 0x7f5e4806cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48071d80 con 0x7f5e4806d090 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f5e400019f0 con 0x7f5e4806d090 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 >> v1:172.21.15.97:6789/0 conn(0x7f5e4806d090 legacy=0x7f5e48165f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e2c004800 0x7f5e2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 0x7f5e2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4806d4c0 con 0x7f5e2c0039b0 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48071aa0 con 0x7f5e2c004800 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c00b9c0 con 0x7f5e480cf430 2024-03-06T07:29:38.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.840+0000 7f5e46800640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.841+0000 7f5e3de00640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e2c004800 0x7f5e2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.841+0000 7f5e46800640 1 -- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 msgr2=0x7f5e2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.841+0000 7f5e46800640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e480cf430 0x7f5e2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.841+0000 7f5e46800640 1 -- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e2c004800 msgr2=0x7f5e2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.841+0000 7f5e46800640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e2c004800 0x7f5e2c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.841+0000 7f5e46800640 1 -- 172.21.15.97:0/4279462370 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c003520 con 0x7f5e2c0039b0 2024-03-06T07:29:38.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e46800640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e2c003dd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009ac0 tx=0x7f5e2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c9524ac54f3944 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38039070 con 0x7f5e2c0039b0 2024-03-06T07:29:38.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3802c710 con 0x7f5e2c0039b0 2024-03-06T07:29:38.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e45e00640 1 -- 172.21.15.97:0/4279462370 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38009690 con 0x7f5e2c0039b0 2024-03-06T07:29:38.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e4d400640 1 -- 172.21.15.97:0/4279462370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 msgr2=0x7f5e2c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e4d400640 1 --2- 172.21.15.97:0/4279462370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e2c003dd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009ac0 tx=0x7f5e2c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e47200640 1 -- 172.21.15.97:0/4279462370 reap_dead start 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e4d400640 1 -- 172.21.15.97:0/4279462370 shutdown_connections 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.842+0000 7f5e4d400640 1 -- 172.21.15.97:0/4279462370 >> 172.21.15.97:0/4279462370 conn(0x7f5e480b1730 msgr2=0x7f5e480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.843+0000 7f5e4d400640 1 -- 172.21.15.97:0/4279462370 shutdown_connections 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.843+0000 7f5e4d400640 1 -- 172.21.15.97:0/4279462370 wait complete. 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.844+0000 7f5e4d400640 1 Processor -- start 2024-03-06T07:29:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.844+0000 7f5e4d400640 1 -- start start 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.845+0000 7f5e4d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e481527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.845+0000 7f5e4d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e48165b80 0x7f5e48166000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.845+0000 7f5e4d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e481f76e0 0x7f5e481f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.845+0000 7f5e4d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48162b70 con 0x7f5e2c0039b0 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.845+0000 7f5e4d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48072340 con 0x7f5e481f76e0 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.845+0000 7f5e4d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48162cf0 con 0x7f5e48165b80 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e481527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e481527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44718/0 (socket says 172.21.15.97:44718) 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 -- 172.21.15.97:0/1226837364 learned_addr learned my addr 172.21.15.97:0/1226837364 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e3de00640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e48165b80 0x7f5e48166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 -- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e48165b80 msgr2=0x7f5e48166000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e47200640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e481f76e0 0x7f5e481f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e48165b80 0x7f5e48166000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 -- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e481f76e0 msgr2=0x7f5e481f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e481f76e0 0x7f5e481f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 -- 172.21.15.97:0/1226837364 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e481f8720 con 0x7f5e2c0039b0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.846+0000 7f5e46800640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e481527c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e38037040 tx=0x7f5e38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.847+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38042020 con 0x7f5e2c0039b0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.847+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3803a040 con 0x7f5e2c0039b0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.847+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38039070 con 0x7f5e2c0039b0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.847+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e481f8a00 con 0x7f5e2c0039b0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.847+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e481f8fc0 con 0x7f5e2c0039b0 2024-03-06T07:29:38.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.848+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4804f330 con 0x7f5e2c0039b0 2024-03-06T07:29:38.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.849+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e38045020 con 0x7f5e2c0039b0 2024-03-06T07:29:38.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.850+0000 7f5e3f400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e2406b4d0 0x7f5e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.850+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e380ab690 con 0x7f5e2c0039b0 2024-03-06T07:29:38.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.850+0000 7f5e3de00640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e2406b4d0 0x7f5e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.851+0000 7f5e3de00640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e2406b4d0 0x7f5e2406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5e480b1020 tx=0x7f5e30008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:38.854+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e3804c070 con 0x7f5e2c0039b0 2024-03-06T07:29:39.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.057+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e48072550 con 0x7f5e2c0039b0 2024-03-06T07:29:39.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.059+0000 7f5e3f400640 1 -- 172.21.15.97:0/1226837364 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16614 (secure 0 0 0) 0x7f5e3807a500 con 0x7f5e2c0039b0 2024-03-06T07:29:39.061 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:39.061 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":45,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:37.643239+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:39.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.061+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e2406b4d0 msgr2=0x7f5e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.061+0000 7f5e4d400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e2406b4d0 0x7f5e2406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5e480b1020 tx=0x7f5e30008040 comp rx=0 tx=0).stop 2024-03-06T07:29:39.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.061+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 msgr2=0x7f5e481527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.061+0000 7f5e4d400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e481527c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e38037040 tx=0x7f5e38004730 comp rx=0 tx=0).stop 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 shutdown_connections 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e2406b4d0 0x7f5e2406d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e481f76e0 0x7f5e481f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e48165b80 0x7f5e48166000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 --2- 172.21.15.97:0/1226837364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e2c0039b0 0x7f5e481527c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 >> 172.21.15.97:0/1226837364 conn(0x7f5e480b1730 msgr2=0x7f5e480afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 shutdown_connections 2024-03-06T07:29:39.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.062+0000 7f5e4d400640 1 -- 172.21.15.97:0/1226837364 wait complete. 2024-03-06T07:29:39.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:29:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.296+0000 7f756e600640 1 Processor -- start 2024-03-06T07:29:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.296+0000 7f756e600640 1 -- start start 2024-03-06T07:29:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f756e600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f756815fc10 0x7f7568070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f756e600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f756815fff0 0x7f7568160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f756e600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75680b46c0 con 0x7f75680ce4a0 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f756e600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75680b49a0 con 0x7f756815fc10 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f756e600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7568072c90 con 0x7f756815fff0 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7567400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f756815fc10 0x7f7568070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7567400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f756815fc10 0x7f7568070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44726/0 (socket says 172.21.15.97:44726) 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7567400640 1 -- 172.21.15.97:0/2959422588 learned_addr learned my addr 172.21.15.97:0/2959422588 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7567400640 1 -- 172.21.15.97:0/2959422588 >> v2:172.21.15.135:3300/0 conn(0x7f756815fff0 msgr2=0x7f7568160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7566a00640 1 --2- 172.21.15.97:0/2959422588 >> v2:172.21.15.135:3300/0 conn(0x7f756815fff0 0x7f7568160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7567400640 1 --2- 172.21.15.97:0/2959422588 >> v2:172.21.15.135:3300/0 conn(0x7f756815fff0 0x7f7568160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.298+0000 7f7567400640 1 -- 172.21.15.97:0/2959422588 >> v1:172.21.15.135:6789/0 conn(0x7f75680ce4a0 legacy=0x7f75680ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.299+0000 7f7567400640 1 -- 172.21.15.97:0/2959422588 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568160990 con 0x7f756815fc10 2024-03-06T07:29:39.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.300+0000 7f7567400640 1 --2- 172.21.15.97:0/2959422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fc10 0x7f7568070ff0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7568074a50 tx=0x7f755802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50e54fecba627af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.300+0000 7f7566000640 1 -- 172.21.15.97:0/2959422588 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755802ca80 con 0x7f756815fc10 2024-03-06T07:29:39.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.300+0000 7f7566000640 1 -- 172.21.15.97:0/2959422588 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755802cc20 con 0x7f756815fc10 2024-03-06T07:29:39.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.300+0000 7f7566000640 1 -- 172.21.15.97:0/2959422588 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75580343f0 con 0x7f756815fc10 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.300+0000 7f756e600640 1 -- 172.21.15.97:0/2959422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fc10 msgr2=0x7f7568070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.300+0000 7f756e600640 1 --2- 172.21.15.97:0/2959422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fc10 0x7f7568070ff0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7568074a50 tx=0x7f755802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.301+0000 7f756e600640 1 -- 172.21.15.97:0/2959422588 shutdown_connections 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.301+0000 7f756e600640 1 --2- 172.21.15.97:0/2959422588 >> v2:172.21.15.135:3300/0 conn(0x7f756815fff0 0x7f7568160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.301+0000 7f756e600640 1 --2- 172.21.15.97:0/2959422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fc10 0x7f7568070ff0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.301+0000 7f756e600640 1 -- 172.21.15.97:0/2959422588 >> 172.21.15.97:0/2959422588 conn(0x7f75680b1940 msgr2=0x7f75680b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.301+0000 7f756e600640 1 -- 172.21.15.97:0/2959422588 shutdown_connections 2024-03-06T07:29:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.301+0000 7f756e600640 1 -- 172.21.15.97:0/2959422588 wait complete. 2024-03-06T07:29:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.302+0000 7f756e600640 1 Processor -- start 2024-03-06T07:29:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.302+0000 7f756e600640 1 -- start start 2024-03-06T07:29:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.303+0000 7f756e600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75680ce4a0 0x7f756809d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.303+0000 7f756e600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f756815fc10 0x7f7568146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.303+0000 7f756e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 0x7f75681571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.303+0000 7f756e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7568072a30 con 0x7f756815fff0 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.303+0000 7f756e600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75680728b0 con 0x7f756815fc10 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.303+0000 7f756e600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756815c2e0 con 0x7f75680ce4a0 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 0x7f75681571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 0x7f75681571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44728/0 (socket says 172.21.15.97:44728) 2024-03-06T07:29:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 -- 172.21.15.97:0/1825754322 learned_addr learned my addr 172.21.15.97:0/1825754322 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:39.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567400640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75680ce4a0 0x7f756809d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7566a00640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f756815fc10 0x7f7568146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 -- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75680ce4a0 msgr2=0x7f756809d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75680ce4a0 0x7f756809d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 -- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f756815fc10 msgr2=0x7f7568146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f756815fc10 0x7f7568146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 -- 172.21.15.97:0/1825754322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756815c870 con 0x7f756815fff0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.304+0000 7f7567e00640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 0x7f75681571e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f756000c410 tx=0x7f756000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.305+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7560011790 con 0x7f756815fff0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.305+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7560017070 con 0x7f756815fff0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.305+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7560011bb0 con 0x7f756815fff0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.305+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756815e9b0 con 0x7f756815fff0 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.305+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756815ef70 con 0x7f756815fff0 2024-03-06T07:29:39.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.306+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756804f330 con 0x7f756815fff0 2024-03-06T07:29:39.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.307+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f756001c460 con 0x7f756815fff0 2024-03-06T07:29:39.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.308+0000 7f755f400640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f754406b4d0 0x7f754406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.308+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7560089e90 con 0x7f756815fff0 2024-03-06T07:29:39.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.308+0000 7f7567400640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f754406b4d0 0x7f754406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.309+0000 7f7567400640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f754406b4d0 0x7f754406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7558035000 tx=0x7f7558030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.313+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7560021020 con 0x7f756815fff0 2024-03-06T07:29:39.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:39.531+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f756806ef30 con 0x7f756815fff0 2024-03-06T07:29:40.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.218+0000 7f755f400640 1 -- 172.21.15.97:0/1825754322 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7560058c70 con 0x7f756815fff0 2024-03-06T07:29:40.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.222+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f754406b4d0 msgr2=0x7f754406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.222+0000 7f756e600640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f754406b4d0 0x7f754406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7558035000 tx=0x7f7558030000 comp rx=0 tx=0).stop 2024-03-06T07:29:40.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.222+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 msgr2=0x7f75681571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.222+0000 7f756e600640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 0x7f75681571e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f756000c410 tx=0x7f756000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 shutdown_connections 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f754406b4d0 0x7f754406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f756815fff0 0x7f75681571e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f756815fc10 0x7f7568146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 --2- 172.21.15.97:0/1825754322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75680ce4a0 0x7f756809d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 >> 172.21.15.97:0/1825754322 conn(0x7f75680b1940 msgr2=0x7f7568151370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 shutdown_connections 2024-03-06T07:29:40.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.223+0000 7f756e600640 1 -- 172.21.15.97:0/1825754322 wait complete. 2024-03-06T07:29:40.246 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:29:40.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:29:40.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.457+0000 7f366d400640 1 Processor -- start 2024-03-06T07:29:40.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.457+0000 7f366d400640 1 -- start start 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f366d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f36680b3df0 0x7f36680727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f366d400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f36680b3650 0x7f36680b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f366d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680cda90 0x7f366806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f366d400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3668072310 con 0x7f36680b3df0 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f366d400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36680b4710 con 0x7f36680b3650 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f366d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36680719e0 con 0x7f36680cda90 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f3666800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f36680b3df0 0x7f36680727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f3666800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f36680b3df0 0x7f36680727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44736/0 (socket says 172.21.15.97:44736) 2024-03-06T07:29:40.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.459+0000 7f3666800640 1 -- 172.21.15.97:0/3935519553 learned_addr learned my addr 172.21.15.97:0/3935519553 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f365fe00640 1 --2- 172.21.15.97:0/3935519553 >> v2:172.21.15.135:3300/0 conn(0x7f36680b3650 0x7f36680b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3667200640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680cda90 0x7f366806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3666800640 1 -- 172.21.15.97:0/3935519553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680cda90 msgr2=0x7f366806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3666800640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680cda90 0x7f366806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3666800640 1 -- 172.21.15.97:0/3935519553 >> v2:172.21.15.135:3300/0 conn(0x7f36680b3650 msgr2=0x7f36680b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3666800640 1 --2- 172.21.15.97:0/3935519553 >> v2:172.21.15.135:3300/0 conn(0x7f36680b3650 0x7f36680b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3666800640 1 -- 172.21.15.97:0/3935519553 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3668071cc0 con 0x7f36680b3df0 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f365fe00640 1 --2- 172.21.15.97:0/3935519553 >> v2:172.21.15.135:3300/0 conn(0x7f36680b3650 0x7f36680b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.460+0000 7f3667200640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680cda90 0x7f366806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.461+0000 7f3666800640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3df0 0x7f36680727a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f36680b4c90 tx=0x7f365802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efa9fbfd4332795 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.461+0000 7f3665e00640 1 -- 172.21.15.97:0/3935519553 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658035020 con 0x7f36680b3df0 2024-03-06T07:29:40.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.461+0000 7f3665e00640 1 -- 172.21.15.97:0/3935519553 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365802d040 con 0x7f36680b3df0 2024-03-06T07:29:40.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.461+0000 7f3665e00640 1 -- 172.21.15.97:0/3935519553 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658031a80 con 0x7f36680b3df0 2024-03-06T07:29:40.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 -- 172.21.15.97:0/3935519553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3df0 msgr2=0x7f36680727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3df0 0x7f36680727a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f36680b4c90 tx=0x7f365802b900 comp rx=0 tx=0).stop 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 -- 172.21.15.97:0/3935519553 shutdown_connections 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680cda90 0x7f366806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 --2- 172.21.15.97:0/3935519553 >> v2:172.21.15.135:3300/0 conn(0x7f36680b3650 0x7f36680b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 --2- 172.21.15.97:0/3935519553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3df0 0x7f36680727a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 -- 172.21.15.97:0/3935519553 >> 172.21.15.97:0/3935519553 conn(0x7f36680b1940 msgr2=0x7f36680b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 -- 172.21.15.97:0/3935519553 shutdown_connections 2024-03-06T07:29:40.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.462+0000 7f366d400640 1 -- 172.21.15.97:0/3935519553 wait complete. 2024-03-06T07:29:40.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.464+0000 7f366d400640 1 Processor -- start 2024-03-06T07:29:40.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.464+0000 7f366d400640 1 -- start start 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f366d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 0x7f366809b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f366d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680b3df0 0x7f36680ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f366d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f36680cda90 0x7f36680adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f366d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366815b240 con 0x7f36680b3650 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f366d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36680b3b30 con 0x7f36680cda90 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f366d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366815b3c0 con 0x7f36680b3df0 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 0x7f366809b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 0x7f366809b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44750/0 (socket says 172.21.15.97:44750) 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 -- 172.21.15.97:0/478410256 learned_addr learned my addr 172.21.15.97:0/478410256 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 -- 172.21.15.97:0/478410256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680b3df0 msgr2=0x7f36680ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680b3df0 0x7f36680ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 -- 172.21.15.97:0/478410256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f36680cda90 msgr2=0x7f36680adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f36680cda90 0x7f36680adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.465+0000 7f3666800640 1 -- 172.21.15.97:0/478410256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36681436c0 con 0x7f36680b3650 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.466+0000 7f3666800640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 0x7f366809b270 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3658000c00 tx=0x7f36580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.466+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658040070 con 0x7f36680b3650 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.466+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658039bf0 con 0x7f36680b3650 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.466+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658035040 con 0x7f36680b3650 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.466+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36680ae3f0 con 0x7f36680b3650 2024-03-06T07:29:40.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.466+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366814ded0 con 0x7f36680b3650 2024-03-06T07:29:40.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.467+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366804f330 con 0x7f36680b3650 2024-03-06T07:29:40.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.468+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3658038020 con 0x7f36680b3650 2024-03-06T07:29:40.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.469+0000 7f365ea00640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f364406b4d0 0x7f364406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.469+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36580ac000 con 0x7f36680b3650 2024-03-06T07:29:40.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.469+0000 7f365fe00640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f364406b4d0 0x7f364406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.470+0000 7f365fe00640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f364406b4d0 0x7f364406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3650006b10 tx=0x7f3650008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.473+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f365807ae80 con 0x7f36680b3650 2024-03-06T07:29:40.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:40.713+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f36680b0ca0 con 0x7f36680b3650 2024-03-06T07:29:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.229+0000 7f365ea00640 1 -- 172.21.15.97:0/478410256 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f365807a780 con 0x7f36680b3650 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f364406b4d0 msgr2=0x7f364406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f364406b4d0 0x7f364406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3650006b10 tx=0x7f3650008040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 msgr2=0x7f366809b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 0x7f366809b270 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3658000c00 tx=0x7f36580046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 shutdown_connections 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f364406b4d0 0x7f364406d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f36680cda90 0x7f36680adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36680b3df0 0x7f36680ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 --2- 172.21.15.97:0/478410256 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36680b3650 0x7f366809b270 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.233+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 >> 172.21.15.97:0/478410256 conn(0x7f36680b1940 msgr2=0x7f3668152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.234+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 shutdown_connections 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.234+0000 7f366d400640 1 -- 172.21.15.97:0/478410256 wait complete. 2024-03-06T07:29:41.257 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:29:41.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.458+0000 7f9016200640 1 Processor -- start 2024-03-06T07:29:41.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.458+0000 7f9016200640 1 -- start start 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.459+0000 7f9016200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9010144c00 0x7f901006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.459+0000 7f9016200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f901015f9d0 0x7f901015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.459+0000 7f9016200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90100ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.459+0000 7f9016200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901006ee70 con 0x7f9010144c00 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.459+0000 7f9016200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901006cef0 con 0x7f901015f9d0 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.459+0000 7f9016200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901006d1d0 con 0x7f90100ce0a0 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9010144c00 0x7f901006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9010144c00 0x7f901006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44764/0 (socket says 172.21.15.97:44764) 2024-03-06T07:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 -- 172.21.15.97:0/3378440174 learned_addr learned my addr 172.21.15.97:0/3378440174 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 -- 172.21.15.97:0/3378440174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 msgr2=0x7f90100ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 --2- 172.21.15.97:0/3378440174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90100ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 -- 172.21.15.97:0/3378440174 >> v2:172.21.15.135:3300/0 conn(0x7f901015f9d0 msgr2=0x7f901015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 --2- 172.21.15.97:0/3378440174 >> v2:172.21.15.135:3300/0 conn(0x7f901015f9d0 0x7f901015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.460+0000 7f900f400640 1 -- 172.21.15.97:0/3378440174 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901006d4b0 con 0x7f9010144c00 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.461+0000 7f900f400640 1 --2- 172.21.15.97:0/3378440174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9010144c00 0x7f901006e930 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f90100b26f0 tx=0x7f900002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13e9852a40cf1ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.461+0000 7f900ea00640 1 -- 172.21.15.97:0/3378440174 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000035020 con 0x7f9010144c00 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.461+0000 7f900ea00640 1 -- 172.21.15.97:0/3378440174 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9000031870 con 0x7f9010144c00 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.461+0000 7f900ea00640 1 -- 172.21.15.97:0/3378440174 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000031c40 con 0x7f9010144c00 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 -- 172.21.15.97:0/3378440174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9010144c00 msgr2=0x7f901006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 --2- 172.21.15.97:0/3378440174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9010144c00 0x7f901006e930 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f90100b26f0 tx=0x7f900002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 -- 172.21.15.97:0/3378440174 shutdown_connections 2024-03-06T07:29:41.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 --2- 172.21.15.97:0/3378440174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90100ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 --2- 172.21.15.97:0/3378440174 >> v2:172.21.15.135:3300/0 conn(0x7f901015f9d0 0x7f901015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 --2- 172.21.15.97:0/3378440174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9010144c00 0x7f901006e930 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 -- 172.21.15.97:0/3378440174 >> 172.21.15.97:0/3378440174 conn(0x7f90100b1730 msgr2=0x7f90100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 -- 172.21.15.97:0/3378440174 shutdown_connections 2024-03-06T07:29:41.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.462+0000 7f9016200640 1 -- 172.21.15.97:0/3378440174 wait complete. 2024-03-06T07:29:41.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.464+0000 7f9016200640 1 Processor -- start 2024-03-06T07:29:41.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.464+0000 7f9016200640 1 -- start start 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f9016200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90101e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f9016200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9010144c00 0x7f90101f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f9016200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 0x7f901013baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f9016200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901006cb50 con 0x7f901015f9d0 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f9016200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9010160730 con 0x7f9010144c00 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f9016200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901006ccd0 con 0x7f90100ce0a0 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f900fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 0x7f901013baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f900fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 0x7f901013baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44770/0 (socket says 172.21.15.97:44770) 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f900fe00640 1 -- 172.21.15.97:0/1365500891 learned_addr learned my addr 172.21.15.97:0/1365500891 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.465+0000 7f900f400640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90101e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9006a00640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9010144c00 0x7f90101f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f900fe00640 1 -- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 msgr2=0x7f90101e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f900fe00640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90101e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f900fe00640 1 -- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9010144c00 msgr2=0x7f90101f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f900fe00640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9010144c00 0x7f90101f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f900fe00640 1 -- 172.21.15.97:0/1365500891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90101f38e0 con 0x7f901015f9d0 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9006a00640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9010144c00 0x7f90101f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f900fe00640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 0x7f901013baf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f900800b970 tx=0x7f900800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008019070 con 0x7f901015f9d0 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900800cb60 con 0x7f901015f9d0 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008014440 con 0x7f901015f9d0 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90101e5ea0 con 0x7f901015f9d0 2024-03-06T07:29:41.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.466+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90101e6460 con 0x7f901015f9d0 2024-03-06T07:29:41.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.467+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901004f330 con 0x7f901015f9d0 2024-03-06T07:29:41.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.468+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f900801c070 con 0x7f901015f9d0 2024-03-06T07:29:41.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.469+0000 7f9007e00640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8fec06b4d0 0x7f8fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.469+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9008089fc0 con 0x7f901015f9d0 2024-03-06T07:29:41.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.470+0000 7f900f400640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8fec06b4d0 0x7f8fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.470+0000 7f900f400640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8fec06b4d0 0x7f8fec06d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9000036000 tx=0x7f900003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.474+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9008058da0 con 0x7f901015f9d0 2024-03-06T07:29:41.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.713+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90100716c0 con 0x7f901015f9d0 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.714+0000 7f9007e00640 1 -- 172.21.15.97:0/1365500891 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f90080586a0 con 0x7f901015f9d0 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:37.643662+0000","modified":"2024-03-06T07:29:40.715394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6835/1918561213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1918561213},{"type":"v1","addr":"172.21.15.135:6835","nonce":1918561213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":3}]} 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 26 2024-03-06T07:29:41.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.717+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8fec06b4d0 msgr2=0x7f8fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.717+0000 7f9016200640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8fec06b4d0 0x7f8fec06d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9000036000 tx=0x7f900003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.717+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 msgr2=0x7f901013baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.717+0000 7f9016200640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 0x7f901013baf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f900800b970 tx=0x7f900800be30 comp rx=0 tx=0).stop 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 shutdown_connections 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8fec06b4d0 0x7f8fec06d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f901015f9d0 0x7f901013baf0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9010144c00 0x7f90101f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 --2- 172.21.15.97:0/1365500891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f90100ce0a0 0x7f90101e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 >> 172.21.15.97:0/1365500891 conn(0x7f90100b1730 msgr2=0x7f9010070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 shutdown_connections 2024-03-06T07:29:41.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.718+0000 7f9016200640 1 -- 172.21.15.97:0/1365500891 wait complete. 2024-03-06T07:29:41.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:41.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.938+0000 7ff55ca00640 1 Processor -- start 2024-03-06T07:29:41.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.938+0000 7ff55ca00640 1 -- start start 2024-03-06T07:29:41.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.939+0000 7ff55ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 0x7ff5580ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.939+0000 7ff55ca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff5580cc9d0 0x7ff55815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.939+0000 7ff55ca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55815c240 con 0x7ff5580aebf0 2024-03-06T07:29:41.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.939+0000 7ff55ca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55815ce30 con 0x7ff5580cc9d0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.939+0000 7ff55ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55815c500 con 0x7ff5581561f0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff5580aebf0 0x7ff5580cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48050/0 (socket says 172.21.15.97:48050) 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555e00640 1 -- 172.21.15.97:0/3809084340 learned_addr learned my addr 172.21.15.97:0/3809084340 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff556800640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 0x7ff5580ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041992294 0 0) 0x7ff55815c240 con 0x7ff5580aebf0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c003520 con 0x7ff5580aebf0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff557200640 1 --2- 172.21.15.97:0/3809084340 >> v2:172.21.15.135:3300/0 conn(0x7ff5580cc9d0 0x7ff55815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4174478888 0 0) 0x7ff53c003520 con 0x7ff5580aebf0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 msgr2=0x7ff5580ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 0x7ff5580ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 >> v2:172.21.15.135:3300/0 conn(0x7ff5580cc9d0 msgr2=0x7ff55815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 --2- 172.21.15.97:0/3809084340 >> v2:172.21.15.135:3300/0 conn(0x7ff5580cc9d0 0x7ff55815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.940+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55815c7e0 con 0x7ff5580aebf0 2024-03-06T07:29:41.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff540001670 con 0x7ff5580aebf0 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 >> v1:172.21.15.97:6789/0 conn(0x7ff5580aebf0 legacy=0x7ff5580cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff53c0039b0 0x7ff53c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 0x7ff53c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff53c004d40 0x7ff53c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55815ce30 con 0x7ff53c0039b0 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55815c500 con 0x7ff53c004d40 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c00ba40 con 0x7ff5581561f0 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff556800640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff53c0039b0 0x7ff53c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff556800640 1 -- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 msgr2=0x7ff53c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff556800640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5581561f0 0x7ff53c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff557200640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff53c004d40 0x7ff53c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff556800640 1 -- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff53c004d40 msgr2=0x7ff53c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff556800640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff53c004d40 0x7ff53c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.941+0000 7ff556800640 1 -- 172.21.15.97:0/3809084340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c003520 con 0x7ff53c0039b0 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.942+0000 7ff556800640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff53c0039b0 0x7ff53c003dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ac0 tx=0x7ff53c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac6f85c6e9c738a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548039070 con 0x7ff53c0039b0 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54802c710 con 0x7ff53c0039b0 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff555400640 1 -- 172.21.15.97:0/3809084340 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548009690 con 0x7ff53c0039b0 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff55ca00640 1 -- 172.21.15.97:0/3809084340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff53c0039b0 msgr2=0x7ff53c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff55ca00640 1 --2- 172.21.15.97:0/3809084340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff53c0039b0 0x7ff53c003dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ac0 tx=0x7ff53c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff557200640 1 -- 172.21.15.97:0/3809084340 reap_dead start 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff55ca00640 1 -- 172.21.15.97:0/3809084340 shutdown_connections 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff55ca00640 1 -- 172.21.15.97:0/3809084340 >> 172.21.15.97:0/3809084340 conn(0x7ff55806afc0 msgr2=0x7ff55806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff55ca00640 1 -- 172.21.15.97:0/3809084340 shutdown_connections 2024-03-06T07:29:41.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.943+0000 7ff55ca00640 1 -- 172.21.15.97:0/3809084340 wait complete. 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.945+0000 7ff55ca00640 1 Processor -- start 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.945+0000 7ff55ca00640 1 -- start start 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff55ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff53c0039b0 0x7ff558151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff55ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 0x7ff55814d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff55ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff558154f50 0x7ff5581553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff55ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558156750 con 0x7ff5581532d0 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff55ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581565d0 con 0x7ff558154f50 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff55ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580ae6b0 con 0x7ff53c0039b0 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff555e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 0x7ff55814d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff555e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 0x7ff55814d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44784/0 (socket says 172.21.15.97:44784) 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.946+0000 7ff555e00640 1 -- 172.21.15.97:0/1036257170 learned_addr learned my addr 172.21.15.97:0/1036257170 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff556800640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff53c0039b0 0x7ff558151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff557200640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff558154f50 0x7ff5581553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff555e00640 1 -- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff53c0039b0 msgr2=0x7ff558151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff555e00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff53c0039b0 0x7ff558151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff555e00640 1 -- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff558154f50 msgr2=0x7ff5581553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff555e00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff558154f50 0x7ff5581553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff555e00640 1 -- 172.21.15.97:0/1036257170 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff558151da0 con 0x7ff5581532d0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff557200640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff558154f50 0x7ff5581553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff555e00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 0x7ff55814d690 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff54000c770 tx=0x7ff54000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540018070 con 0x7ff5581532d0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5400074e0 con 0x7ff5581532d0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540004a20 con 0x7ff5581532d0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.947+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558152080 con 0x7ff5581532d0 2024-03-06T07:29:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.948+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558152640 con 0x7ff5581532d0 2024-03-06T07:29:41.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.949+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55804f330 con 0x7ff5581532d0 2024-03-06T07:29:41.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.950+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff540004bc0 con 0x7ff5581532d0 2024-03-06T07:29:41.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.951+0000 7ff54ea00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b4d0 0x7ff53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.951+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5400897b0 con 0x7ff5581532d0 2024-03-06T07:29:41.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.951+0000 7ff556800640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b4d0 0x7ff53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.952+0000 7ff556800640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b4d0 0x7ff53406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff548036000 tx=0x7ff548030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:41.956+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff540058590 con 0x7ff5581532d0 2024-03-06T07:29:42.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.154+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5580b2b10 con 0x7ff5581532d0 2024-03-06T07:29:42.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.157+0000 7ff54ea00640 1 -- 172.21.15.97:0/1036257170 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16601 (secure 0 0 0) 0x7ff540057e90 con 0x7ff5581532d0 2024-03-06T07:29:42.159 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:42.160 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":46,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:39.204502+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:42.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b4d0 msgr2=0x7ff53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b4d0 0x7ff53406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff548036000 tx=0x7ff548030000 comp rx=0 tx=0).stop 2024-03-06T07:29:42.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 msgr2=0x7ff55814d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 0x7ff55814d690 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff54000c770 tx=0x7ff54000cc30 comp rx=0 tx=0).stop 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 shutdown_connections 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff53406b4d0 0x7ff53406d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff558154f50 0x7ff5581553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5581532d0 0x7ff55814d690 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 --2- 172.21.15.97:0/1036257170 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff53c0039b0 0x7ff558151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 >> 172.21.15.97:0/1036257170 conn(0x7ff55806afc0 msgr2=0x7ff5580adc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 shutdown_connections 2024-03-06T07:29:42.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.160+0000 7ff55ca00640 1 -- 172.21.15.97:0/1036257170 wait complete. 2024-03-06T07:29:42.181 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:29:42.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.386+0000 7fa33f000640 1 Processor -- start 2024-03-06T07:29:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.386+0000 7fa33f000640 1 -- start start 2024-03-06T07:29:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.387+0000 7fa33f000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa33815acb0 0x7fa3380b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.387+0000 7fa33f000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa3380b4680 0x7fa33806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.387+0000 7fa33f000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33815b090 con 0x7fa338072340 2024-03-06T07:29:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.387+0000 7fa33f000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3380b2a30 con 0x7fa3380b4680 2024-03-06T07:29:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.387+0000 7fa33f000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3380b4360 con 0x7fa33815acb0 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa3380b4680 0x7fa33806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa3380b4680 0x7fa33806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44792/0 (socket says 172.21.15.97:44792) 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 -- 172.21.15.97:0/2032602595 learned_addr learned my addr 172.21.15.97:0/2032602595 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa337e00640 1 --2- 172.21.15.97:0/2032602595 >> v2:172.21.15.135:3300/0 conn(0x7fa33815acb0 0x7fa3380b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 -- 172.21.15.97:0/2032602595 >> v2:172.21.15.135:3300/0 conn(0x7fa33815acb0 msgr2=0x7fa3380b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 --2- 172.21.15.97:0/2032602595 >> v2:172.21.15.135:3300/0 conn(0x7fa33815acb0 0x7fa3380b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 -- 172.21.15.97:0/2032602595 >> v1:172.21.15.135:6789/0 conn(0x7fa338072340 legacy=0x7fa3380ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.388+0000 7fa32f400640 1 -- 172.21.15.97:0/2032602595 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380b2cf0 con 0x7fa3380b4680 2024-03-06T07:29:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.389+0000 7fa32f400640 1 --2- 172.21.15.97:0/2032602595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3380b4680 0x7fa33806e4f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa3380b27f0 tx=0x7fa32002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538ae8c03b45f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa337400640 1 -- 172.21.15.97:0/2032602595 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32003b070 con 0x7fa3380b4680 2024-03-06T07:29:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa337400640 1 -- 172.21.15.97:0/2032602595 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa320036440 con 0x7fa3380b4680 2024-03-06T07:29:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa337400640 1 -- 172.21.15.97:0/2032602595 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320036810 con 0x7fa3380b4680 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 -- 172.21.15.97:0/2032602595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3380b4680 msgr2=0x7fa33806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 --2- 172.21.15.97:0/2032602595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3380b4680 0x7fa33806e4f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa3380b27f0 tx=0x7fa32002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 -- 172.21.15.97:0/2032602595 shutdown_connections 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 --2- 172.21.15.97:0/2032602595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3380b4680 0x7fa33806e4f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 --2- 172.21.15.97:0/2032602595 >> v2:172.21.15.135:3300/0 conn(0x7fa33815acb0 0x7fa3380b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 -- 172.21.15.97:0/2032602595 >> 172.21.15.97:0/2032602595 conn(0x7fa3380b1730 msgr2=0x7fa3380b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 -- 172.21.15.97:0/2032602595 shutdown_connections 2024-03-06T07:29:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.390+0000 7fa33f000640 1 -- 172.21.15.97:0/2032602595 wait complete. 2024-03-06T07:29:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.392+0000 7fa33f000640 1 Processor -- start 2024-03-06T07:29:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.392+0000 7fa33f000640 1 -- start start 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33f000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa338072340 0x7fa3381e47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33f000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3380b4680 0x7fa3380ce270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33f000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 0x7fa33814b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33f000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380b3d10 con 0x7fa33815acb0 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33f000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380b32b0 con 0x7fa338072340 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33f000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380b4010 con 0x7fa3380b4680 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 0x7fa33814b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 0x7fa33814b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44806/0 (socket says 172.21.15.97:44806) 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.393+0000 7fa33d000640 1 -- 172.21.15.97:0/2482069549 learned_addr learned my addr 172.21.15.97:0/2482069549 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33d000640 1 -- 172.21.15.97:0/2482069549 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3380b4680 msgr2=0x7fa3380ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33d000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3380b4680 0x7fa3380ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33d000640 1 -- 172.21.15.97:0/2482069549 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa338072340 msgr2=0x7fa3381e47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33d000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa338072340 0x7fa3381e47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33d000640 1 -- 172.21.15.97:0/2482069549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33815b7e0 con 0x7fa33815acb0 2024-03-06T07:29:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33d000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 0x7fa33814b0d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa33000b410 tx=0x7fa33000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330015030 con 0x7fa33815acb0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa330004020 con 0x7fa33815acb0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330004410 con 0x7fa33815acb0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33815bac0 con 0x7fa33815acb0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.394+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33815c080 con 0x7fa33815acb0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.396+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33804f330 con 0x7fa33815acb0 2024-03-06T07:29:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.397+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa330018070 con 0x7fa33815acb0 2024-03-06T07:29:42.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.398+0000 7fa334c00640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa31406b4d0 0x7fa31406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.398+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa330089cb0 con 0x7fa33815acb0 2024-03-06T07:29:42.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.398+0000 7fa337e00640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa31406b4d0 0x7fa31406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.399+0000 7fa337e00640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa31406b4d0 0x7fa31406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa3280070b0 tx=0x7fa328007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.402+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa330058b30 con 0x7fa33815acb0 2024-03-06T07:29:42.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.592+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa33806ce60 con 0x7fa31406b4d0 2024-03-06T07:29:42.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.611+0000 7fa334c00640 1 -- 172.21.15.97:0/2482069549 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251894 (secure 0 0 0) 0x7fa33806ce60 con 0x7fa31406b4d0 2024-03-06T07:29:42.617 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:42.618 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa31406b4d0 msgr2=0x7fa31406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa31406b4d0 0x7fa31406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa3280070b0 tx=0x7fa328007040 comp rx=0 tx=0).stop 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 msgr2=0x7fa33814b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 0x7fa33814b0d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa33000b410 tx=0x7fa33000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 shutdown_connections 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa31406b4d0 0x7fa31406d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa33815acb0 0x7fa33814b0d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3380b4680 0x7fa3380ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 --2- 172.21.15.97:0/2482069549 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa338072340 0x7fa3381e47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 >> 172.21.15.97:0/2482069549 conn(0x7fa3380b1730 msgr2=0x7fa3380ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 shutdown_connections 2024-03-06T07:29:42.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.616+0000 7fa33f000640 1 -- 172.21.15.97:0/2482069549 wait complete. 2024-03-06T07:29:42.638 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:29:40.936336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250560,"kb_used_data":5256,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724160,"statfs":{"total":773094113280,"available":772837539840,"internally_reserved":0,"allocated":5382144,"data_stored":2858819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.768275"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235988+0000","last_change":"2024-03-06T07:29:38.230569+0000","last_active":"2024-03-06T07:29:38.235988+0000","last_peered":"2024-03-06T07:29:38.235988+0000","last_clean":"2024-03-06T07:29:38.235988+0000","last_became_active":"2024-03-06T07:29:38.229487+0000","last_became_peered":"2024-03-06T07:29:38.229487+0000","last_unstale":"2024-03-06T07:29:38.235988+0000","last_undegraded":"2024-03-06T07:29:38.235988+0000","last_fullsized":"2024-03-06T07:29:38.235988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:15.599863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233221+0000","last_change":"2024-03-06T07:29:38.224626+0000","last_active":"2024-03-06T07:29:38.233221+0000","last_peered":"2024-03-06T07:29:38.233221+0000","last_clean":"2024-03-06T07:29:38.233221+0000","last_became_active":"2024-03-06T07:29:38.223739+0000","last_became_peered":"2024-03-06T07:29:38.223739+0000","last_unstale":"2024-03-06T07:29:38.233221+0000","last_undegraded":"2024-03-06T07:29:38.233221+0000","last_fullsized":"2024-03-06T07:29:38.233221+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:58.295575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228936+0000","last_change":"2024-03-06T07:29:38.225631+0000","last_active":"2024-03-06T07:29:38.228936+0000","last_peered":"2024-03-06T07:29:38.228936+0000","last_clean":"2024-03-06T07:29:38.228936+0000","last_became_active":"2024-03-06T07:29:38.224305+0000","last_became_peered":"2024-03-06T07:29:38.224305+0000","last_unstale":"2024-03-06T07:29:38.228936+0000","last_undegraded":"2024-03-06T07:29:38.228936+0000","last_fullsized":"2024-03-06T07:29:38.228936+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:57.801906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.222667+0000","last_change":"2024-03-06T07:29:38.212896+0000","last_active":"2024-03-06T07:29:38.222667+0000","last_peered":"2024-03-06T07:29:38.222667+0000","last_clean":"2024-03-06T07:29:38.222667+0000","last_became_active":"2024-03-06T07:29:38.210119+0000","last_became_peered":"2024-03-06T07:29:38.210119+0000","last_unstale":"2024-03-06T07:29:38.222667+0000","last_undegraded":"2024-03-06T07:29:38.222667+0000","last_fullsized":"2024-03-06T07:29:38.222667+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:12.292948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231782+0000","last_change":"2024-03-06T07:29:38.224985+0000","last_active":"2024-03-06T07:29:38.231782+0000","last_peered":"2024-03-06T07:29:38.231782+0000","last_clean":"2024-03-06T07:29:38.231782+0000","last_became_active":"2024-03-06T07:29:38.224690+0000","last_became_peered":"2024-03-06T07:29:38.224690+0000","last_unstale":"2024-03-06T07:29:38.231782+0000","last_undegraded":"2024-03-06T07:29:38.231782+0000","last_fullsized":"2024-03-06T07:29:38.231782+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:36.450661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232011+0000","last_change":"2024-03-06T07:29:38.227982+0000","last_active":"2024-03-06T07:29:38.232011+0000","last_peered":"2024-03-06T07:29:38.232011+0000","last_clean":"2024-03-06T07:29:38.232011+0000","last_became_active":"2024-03-06T07:29:38.227296+0000","last_became_peered":"2024-03-06T07:29:38.227296+0000","last_unstale":"2024-03-06T07:29:38.232011+0000","last_undegraded":"2024-03-06T07:29:38.232011+0000","last_fullsized":"2024-03-06T07:29:38.232011+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:25.689904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235996+0000","last_change":"2024-03-06T07:29:38.227651+0000","last_active":"2024-03-06T07:29:38.235996+0000","last_peered":"2024-03-06T07:29:38.235996+0000","last_clean":"2024-03-06T07:29:38.235996+0000","last_became_active":"2024-03-06T07:29:38.226435+0000","last_became_peered":"2024-03-06T07:29:38.226435+0000","last_unstale":"2024-03-06T07:29:38.235996+0000","last_undegraded":"2024-03-06T07:29:38.235996+0000","last_fullsized":"2024-03-06T07:29:38.235996+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:47.445382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.212606+0000","last_change":"2024-03-06T07:29:38.206428+0000","last_active":"2024-03-06T07:29:38.212606+0000","last_peered":"2024-03-06T07:29:38.212606+0000","last_clean":"2024-03-06T07:29:38.212606+0000","last_became_active":"2024-03-06T07:29:38.206136+0000","last_became_peered":"2024-03-06T07:29:38.206136+0000","last_unstale":"2024-03-06T07:29:38.212606+0000","last_undegraded":"2024-03-06T07:29:38.212606+0000","last_fullsized":"2024-03-06T07:29:38.212606+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:22.577604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.224604+0000","last_change":"2024-03-06T07:29:38.217108+0000","last_active":"2024-03-06T07:29:38.224604+0000","last_peered":"2024-03-06T07:29:38.224604+0000","last_clean":"2024-03-06T07:29:38.224604+0000","last_became_active":"2024-03-06T07:29:38.216805+0000","last_became_peered":"2024-03-06T07:29:38.216805+0000","last_unstale":"2024-03-06T07:29:38.224604+0000","last_undegraded":"2024-03-06T07:29:38.224604+0000","last_fullsized":"2024-03-06T07:29:38.224604+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:08.540463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225246+0000","last_change":"2024-03-06T07:29:38.219333+0000","last_active":"2024-03-06T07:29:38.225246+0000","last_peered":"2024-03-06T07:29:38.225246+0000","last_clean":"2024-03-06T07:29:38.225246+0000","last_became_active":"2024-03-06T07:29:38.217493+0000","last_became_peered":"2024-03-06T07:29:38.217493+0000","last_unstale":"2024-03-06T07:29:38.225246+0000","last_undegraded":"2024-03-06T07:29:38.225246+0000","last_fullsized":"2024-03-06T07:29:38.225246+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:39.896470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225064+0000","last_change":"2024-03-06T07:29:38.217603+0000","last_active":"2024-03-06T07:29:38.225064+0000","last_peered":"2024-03-06T07:29:38.225064+0000","last_clean":"2024-03-06T07:29:38.225064+0000","last_became_active":"2024-03-06T07:29:38.217115+0000","last_became_peered":"2024-03-06T07:29:38.217115+0000","last_unstale":"2024-03-06T07:29:38.225064+0000","last_undegraded":"2024-03-06T07:29:38.225064+0000","last_fullsized":"2024-03-06T07:29:38.225064+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:18.591917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.237043+0000","last_change":"2024-03-06T07:29:38.232746+0000","last_active":"2024-03-06T07:29:38.237043+0000","last_peered":"2024-03-06T07:29:38.237043+0000","last_clean":"2024-03-06T07:29:38.237043+0000","last_became_active":"2024-03-06T07:29:38.230999+0000","last_became_peered":"2024-03-06T07:29:38.230999+0000","last_unstale":"2024-03-06T07:29:38.237043+0000","last_undegraded":"2024-03-06T07:29:38.237043+0000","last_fullsized":"2024-03-06T07:29:38.237043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:33.122277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232083+0000","last_change":"2024-03-06T07:29:38.225460+0000","last_active":"2024-03-06T07:29:38.232083+0000","last_peered":"2024-03-06T07:29:38.232083+0000","last_clean":"2024-03-06T07:29:38.232083+0000","last_became_active":"2024-03-06T07:29:38.224462+0000","last_became_peered":"2024-03-06T07:29:38.224462+0000","last_unstale":"2024-03-06T07:29:38.232083+0000","last_undegraded":"2024-03-06T07:29:38.232083+0000","last_fullsized":"2024-03-06T07:29:38.232083+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:25.872343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230726+0000","last_change":"2024-03-06T07:29:38.227747+0000","last_active":"2024-03-06T07:29:38.230726+0000","last_peered":"2024-03-06T07:29:38.230726+0000","last_clean":"2024-03-06T07:29:38.230726+0000","last_became_active":"2024-03-06T07:29:38.226172+0000","last_became_peered":"2024-03-06T07:29:38.226172+0000","last_unstale":"2024-03-06T07:29:38.230726+0000","last_undegraded":"2024-03-06T07:29:38.230726+0000","last_fullsized":"2024-03-06T07:29:38.230726+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:28:40.782881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232486+0000","last_change":"2024-03-06T07:29:38.230424+0000","last_active":"2024-03-06T07:29:38.232486+0000","last_peered":"2024-03-06T07:29:38.232486+0000","last_clean":"2024-03-06T07:29:38.232486+0000","last_became_active":"2024-03-06T07:29:38.230076+0000","last_became_peered":"2024-03-06T07:29:38.230076+0000","last_unstale":"2024-03-06T07:29:38.232486+0000","last_undegraded":"2024-03-06T07:29:38.232486+0000","last_fullsized":"2024-03-06T07:29:38.232486+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:25.691880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.227871+0000","last_change":"2024-03-06T07:29:38.219507+0000","last_active":"2024-03-06T07:29:38.227871+0000","last_peered":"2024-03-06T07:29:38.227871+0000","last_clean":"2024-03-06T07:29:38.227871+0000","last_became_active":"2024-03-06T07:29:38.219104+0000","last_became_peered":"2024-03-06T07:29:38.219104+0000","last_unstale":"2024-03-06T07:29:38.227871+0000","last_undegraded":"2024-03-06T07:29:38.227871+0000","last_fullsized":"2024-03-06T07:29:38.227871+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:11.992158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.234600+0000","last_change":"2024-03-06T07:29:38.227287+0000","last_active":"2024-03-06T07:29:38.234600+0000","last_peered":"2024-03-06T07:29:38.234600+0000","last_clean":"2024-03-06T07:29:38.234600+0000","last_became_active":"2024-03-06T07:29:38.226950+0000","last_became_peered":"2024-03-06T07:29:38.226950+0000","last_unstale":"2024-03-06T07:29:38.234600+0000","last_undegraded":"2024-03-06T07:29:38.234600+0000","last_fullsized":"2024-03-06T07:29:38.234600+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.420975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236353+0000","last_change":"2024-03-06T07:29:38.231283+0000","last_active":"2024-03-06T07:29:38.236353+0000","last_peered":"2024-03-06T07:29:38.236353+0000","last_clean":"2024-03-06T07:29:38.236353+0000","last_became_active":"2024-03-06T07:29:38.230997+0000","last_became_peered":"2024-03-06T07:29:38.230997+0000","last_unstale":"2024-03-06T07:29:38.236353+0000","last_undegraded":"2024-03-06T07:29:38.236353+0000","last_fullsized":"2024-03-06T07:29:38.236353+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:54.828137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228835+0000","last_change":"2024-03-06T07:29:38.219968+0000","last_active":"2024-03-06T07:29:38.228835+0000","last_peered":"2024-03-06T07:29:38.228835+0000","last_clean":"2024-03-06T07:29:38.228835+0000","last_became_active":"2024-03-06T07:29:38.219465+0000","last_became_peered":"2024-03-06T07:29:38.219465+0000","last_unstale":"2024-03-06T07:29:38.228835+0000","last_undegraded":"2024-03-06T07:29:38.228835+0000","last_fullsized":"2024-03-06T07:29:38.228835+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:16.314678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.223102+0000","last_change":"2024-03-06T07:29:38.215089+0000","last_active":"2024-03-06T07:29:38.223102+0000","last_peered":"2024-03-06T07:29:38.223102+0000","last_clean":"2024-03-06T07:29:38.223102+0000","last_became_active":"2024-03-06T07:29:38.214776+0000","last_became_peered":"2024-03-06T07:29:38.214776+0000","last_unstale":"2024-03-06T07:29:38.223102+0000","last_undegraded":"2024-03-06T07:29:38.223102+0000","last_fullsized":"2024-03-06T07:29:38.223102+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:56.714122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.240796+0000","last_change":"2024-03-06T07:29:38.234584+0000","last_active":"2024-03-06T07:29:38.240796+0000","last_peered":"2024-03-06T07:29:38.240796+0000","last_clean":"2024-03-06T07:29:38.240796+0000","last_became_active":"2024-03-06T07:29:38.233282+0000","last_became_peered":"2024-03-06T07:29:38.233282+0000","last_unstale":"2024-03-06T07:29:38.240796+0000","last_undegraded":"2024-03-06T07:29:38.240796+0000","last_fullsized":"2024-03-06T07:29:38.240796+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:42.990968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.220015+0000","last_change":"2024-03-06T07:29:38.208648+0000","last_active":"2024-03-06T07:29:38.220015+0000","last_peered":"2024-03-06T07:29:38.220015+0000","last_clean":"2024-03-06T07:29:38.220015+0000","last_became_active":"2024-03-06T07:29:38.208319+0000","last_became_peered":"2024-03-06T07:29:38.208319+0000","last_unstale":"2024-03-06T07:29:38.220015+0000","last_undegraded":"2024-03-06T07:29:38.220015+0000","last_fullsized":"2024-03-06T07:29:38.220015+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:22.621316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.229627+0000","last_change":"2024-03-06T07:29:38.218295+0000","last_active":"2024-03-06T07:29:38.229627+0000","last_peered":"2024-03-06T07:29:38.229627+0000","last_clean":"2024-03-06T07:29:38.229627+0000","last_became_active":"2024-03-06T07:29:38.217587+0000","last_became_peered":"2024-03-06T07:29:38.217587+0000","last_unstale":"2024-03-06T07:29:38.229627+0000","last_undegraded":"2024-03-06T07:29:38.229627+0000","last_fullsized":"2024-03-06T07:29:38.229627+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:56.714122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228581+0000","last_change":"2024-03-06T07:29:38.226506+0000","last_active":"2024-03-06T07:29:38.228581+0000","last_peered":"2024-03-06T07:29:38.228581+0000","last_clean":"2024-03-06T07:29:38.228581+0000","last_became_active":"2024-03-06T07:29:38.226183+0000","last_became_peered":"2024-03-06T07:29:38.226183+0000","last_unstale":"2024-03-06T07:29:38.228581+0000","last_undegraded":"2024-03-06T07:29:38.228581+0000","last_fullsized":"2024-03-06T07:29:38.228581+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:22.577604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.216727+0000","last_change":"2024-03-06T07:29:38.213303+0000","last_active":"2024-03-06T07:29:38.216727+0000","last_peered":"2024-03-06T07:29:38.216727+0000","last_clean":"2024-03-06T07:29:38.216727+0000","last_became_active":"2024-03-06T07:29:38.213003+0000","last_became_peered":"2024-03-06T07:29:38.213003+0000","last_unstale":"2024-03-06T07:29:38.216727+0000","last_undegraded":"2024-03-06T07:29:38.216727+0000","last_fullsized":"2024-03-06T07:29:38.216727+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.790636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225071+0000","last_change":"2024-03-06T07:29:38.221634+0000","last_active":"2024-03-06T07:29:38.225071+0000","last_peered":"2024-03-06T07:29:38.225071+0000","last_clean":"2024-03-06T07:29:38.225071+0000","last_became_active":"2024-03-06T07:29:38.220504+0000","last_became_peered":"2024-03-06T07:29:38.220504+0000","last_unstale":"2024-03-06T07:29:38.225071+0000","last_undegraded":"2024-03-06T07:29:38.225071+0000","last_fullsized":"2024-03-06T07:29:38.225071+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:56.807941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.234534+0000","last_change":"2024-03-06T07:29:38.226824+0000","last_active":"2024-03-06T07:29:38.234534+0000","last_peered":"2024-03-06T07:29:38.234534+0000","last_clean":"2024-03-06T07:29:38.234534+0000","last_became_active":"2024-03-06T07:29:38.226415+0000","last_became_peered":"2024-03-06T07:29:38.226415+0000","last_unstale":"2024-03-06T07:29:38.234534+0000","last_undegraded":"2024-03-06T07:29:38.234534+0000","last_fullsized":"2024-03-06T07:29:38.234534+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:29.634273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230778+0000","last_change":"2024-03-06T07:29:38.217495+0000","last_active":"2024-03-06T07:29:38.230778+0000","last_peered":"2024-03-06T07:29:38.230778+0000","last_clean":"2024-03-06T07:29:38.230778+0000","last_became_active":"2024-03-06T07:29:38.216542+0000","last_became_peered":"2024-03-06T07:29:38.216542+0000","last_unstale":"2024-03-06T07:29:38.230778+0000","last_undegraded":"2024-03-06T07:29:38.230778+0000","last_fullsized":"2024-03-06T07:29:38.230778+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:16.901995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.216574+0000","last_change":"2024-03-06T07:29:38.212495+0000","last_active":"2024-03-06T07:29:38.216574+0000","last_peered":"2024-03-06T07:29:38.216574+0000","last_clean":"2024-03-06T07:29:38.216574+0000","last_became_active":"2024-03-06T07:29:38.212228+0000","last_became_peered":"2024-03-06T07:29:38.212228+0000","last_unstale":"2024-03-06T07:29:38.216574+0000","last_undegraded":"2024-03-06T07:29:38.216574+0000","last_fullsized":"2024-03-06T07:29:38.216574+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.567265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228856+0000","last_change":"2024-03-06T07:29:38.220521+0000","last_active":"2024-03-06T07:29:38.228856+0000","last_peered":"2024-03-06T07:29:38.228856+0000","last_clean":"2024-03-06T07:29:38.228856+0000","last_became_active":"2024-03-06T07:29:38.220223+0000","last_became_peered":"2024-03-06T07:29:38.220223+0000","last_unstale":"2024-03-06T07:29:38.228856+0000","last_undegraded":"2024-03-06T07:29:38.228856+0000","last_fullsized":"2024-03-06T07:29:38.228856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:28.715523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230332+0000","last_change":"2024-03-06T07:29:38.226544+0000","last_active":"2024-03-06T07:29:38.230332+0000","last_peered":"2024-03-06T07:29:38.230332+0000","last_clean":"2024-03-06T07:29:38.230332+0000","last_became_active":"2024-03-06T07:29:38.224799+0000","last_became_peered":"2024-03-06T07:29:38.224799+0000","last_unstale":"2024-03-06T07:29:38.230332+0000","last_undegraded":"2024-03-06T07:29:38.230332+0000","last_fullsized":"2024-03-06T07:29:38.230332+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:40.234045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.209541+0000","last_change":"2024-03-06T07:29:38.206941+0000","last_active":"2024-03-06T07:29:38.209541+0000","last_peered":"2024-03-06T07:29:38.209541+0000","last_clean":"2024-03-06T07:29:38.209541+0000","last_became_active":"2024-03-06T07:29:38.205924+0000","last_became_peered":"2024-03-06T07:29:38.205924+0000","last_unstale":"2024-03-06T07:29:38.209541+0000","last_undegraded":"2024-03-06T07:29:38.209541+0000","last_fullsized":"2024-03-06T07:29:38.209541+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.790636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225313+0000","last_change":"2024-03-06T07:29:38.222586+0000","last_active":"2024-03-06T07:29:38.225313+0000","last_peered":"2024-03-06T07:29:38.225313+0000","last_clean":"2024-03-06T07:29:38.225313+0000","last_became_active":"2024-03-06T07:29:38.222256+0000","last_became_peered":"2024-03-06T07:29:38.222256+0000","last_unstale":"2024-03-06T07:29:38.225313+0000","last_undegraded":"2024-03-06T07:29:38.225313+0000","last_fullsized":"2024-03-06T07:29:38.225313+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:46.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.243852+0000","last_change":"2024-03-06T07:29:37.203528+0000","last_active":"2024-03-06T07:29:38.243852+0000","last_peered":"2024-03-06T07:29:38.243852+0000","last_clean":"2024-03-06T07:29:38.243852+0000","last_became_active":"2024-03-06T07:29:37.203220+0000","last_became_peered":"2024-03-06T07:29:37.203220+0000","last_unstale":"2024-03-06T07:29:38.243852+0000","last_undegraded":"2024-03-06T07:29:38.243852+0000","last_fullsized":"2024-03-06T07:29:38.243852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:14.584142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.234400+0000","last_change":"2024-03-06T07:29:38.229863+0000","last_active":"2024-03-06T07:29:38.234400+0000","last_peered":"2024-03-06T07:29:38.234400+0000","last_clean":"2024-03-06T07:29:38.234400+0000","last_became_active":"2024-03-06T07:29:38.227652+0000","last_became_peered":"2024-03-06T07:29:38.227652+0000","last_unstale":"2024-03-06T07:29:38.234400+0000","last_undegraded":"2024-03-06T07:29:38.234400+0000","last_fullsized":"2024-03-06T07:29:38.234400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:53.247884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233002+0000","last_change":"2024-03-06T07:29:37.198982+0000","last_active":"2024-03-06T07:29:38.233002+0000","last_peered":"2024-03-06T07:29:38.233002+0000","last_clean":"2024-03-06T07:29:38.233002+0000","last_became_active":"2024-03-06T07:29:37.198670+0000","last_became_peered":"2024-03-06T07:29:37.198670+0000","last_unstale":"2024-03-06T07:29:38.233002+0000","last_undegraded":"2024-03-06T07:29:38.233002+0000","last_fullsized":"2024-03-06T07:29:38.233002+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:30.030268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233728+0000","last_change":"2024-03-06T07:29:38.222647+0000","last_active":"2024-03-06T07:29:38.233728+0000","last_peered":"2024-03-06T07:29:38.233728+0000","last_clean":"2024-03-06T07:29:38.233728+0000","last_became_active":"2024-03-06T07:29:38.222297+0000","last_became_peered":"2024-03-06T07:29:38.222297+0000","last_unstale":"2024-03-06T07:29:38.233728+0000","last_undegraded":"2024-03-06T07:29:38.233728+0000","last_fullsized":"2024-03-06T07:29:38.233728+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:27.802877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.241367+0000","last_change":"2024-03-06T07:29:37.207021+0000","last_active":"2024-03-06T07:29:38.241367+0000","last_peered":"2024-03-06T07:29:38.241367+0000","last_clean":"2024-03-06T07:29:38.241367+0000","last_became_active":"2024-03-06T07:29:37.206714+0000","last_became_peered":"2024-03-06T07:29:37.206714+0000","last_unstale":"2024-03-06T07:29:38.241367+0000","last_undegraded":"2024-03-06T07:29:38.241367+0000","last_fullsized":"2024-03-06T07:29:38.241367+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:17.576195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.224282+0000","last_change":"2024-03-06T07:29:38.220647+0000","last_active":"2024-03-06T07:29:38.224282+0000","last_peered":"2024-03-06T07:29:38.224282+0000","last_clean":"2024-03-06T07:29:38.224282+0000","last_became_active":"2024-03-06T07:29:38.220298+0000","last_became_peered":"2024-03-06T07:29:38.220298+0000","last_unstale":"2024-03-06T07:29:38.224282+0000","last_undegraded":"2024-03-06T07:29:38.224282+0000","last_fullsized":"2024-03-06T07:29:38.224282+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:39.896470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.237194+0000","last_change":"2024-03-06T07:29:37.194194+0000","last_active":"2024-03-06T07:29:38.237194+0000","last_peered":"2024-03-06T07:29:38.237194+0000","last_clean":"2024-03-06T07:29:38.237194+0000","last_became_active":"2024-03-06T07:29:37.193871+0000","last_became_peered":"2024-03-06T07:29:37.193871+0000","last_unstale":"2024-03-06T07:29:38.237194+0000","last_undegraded":"2024-03-06T07:29:38.237194+0000","last_fullsized":"2024-03-06T07:29:38.237194+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:57.279853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.227946+0000","last_change":"2024-03-06T07:29:38.217119+0000","last_active":"2024-03-06T07:29:38.227946+0000","last_peered":"2024-03-06T07:29:38.227946+0000","last_clean":"2024-03-06T07:29:38.227946+0000","last_became_active":"2024-03-06T07:29:38.216159+0000","last_became_peered":"2024-03-06T07:29:38.216159+0000","last_unstale":"2024-03-06T07:29:38.227946+0000","last_undegraded":"2024-03-06T07:29:38.227946+0000","last_fullsized":"2024-03-06T07:29:38.227946+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:02.697325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.243477+0000","last_change":"2024-03-06T07:29:37.204958+0000","last_active":"2024-03-06T07:29:38.243477+0000","last_peered":"2024-03-06T07:29:38.243477+0000","last_clean":"2024-03-06T07:29:38.243477+0000","last_became_active":"2024-03-06T07:29:37.204553+0000","last_became_peered":"2024-03-06T07:29:37.204553+0000","last_unstale":"2024-03-06T07:29:38.243477+0000","last_undegraded":"2024-03-06T07:29:38.243477+0000","last_fullsized":"2024-03-06T07:29:38.243477+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:33.466259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.221478+0000","last_change":"2024-03-06T07:29:38.210751+0000","last_active":"2024-03-06T07:29:38.221478+0000","last_peered":"2024-03-06T07:29:38.221478+0000","last_clean":"2024-03-06T07:29:38.221478+0000","last_became_active":"2024-03-06T07:29:38.210453+0000","last_became_peered":"2024-03-06T07:29:38.210453+0000","last_unstale":"2024-03-06T07:29:38.221478+0000","last_undegraded":"2024-03-06T07:29:38.221478+0000","last_fullsized":"2024-03-06T07:29:38.221478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.693659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.237232+0000","last_change":"2024-03-06T07:29:37.190755+0000","last_active":"2024-03-06T07:29:38.237232+0000","last_peered":"2024-03-06T07:29:38.237232+0000","last_clean":"2024-03-06T07:29:38.237232+0000","last_became_active":"2024-03-06T07:29:37.190175+0000","last_became_peered":"2024-03-06T07:29:37.190175+0000","last_unstale":"2024-03-06T07:29:38.237232+0000","last_undegraded":"2024-03-06T07:29:38.237232+0000","last_fullsized":"2024-03-06T07:29:38.237232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:10.970889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.221109+0000","last_change":"2024-03-06T07:29:38.208445+0000","last_active":"2024-03-06T07:29:38.221109+0000","last_peered":"2024-03-06T07:29:38.221109+0000","last_clean":"2024-03-06T07:29:38.221109+0000","last_became_active":"2024-03-06T07:29:38.208123+0000","last_became_peered":"2024-03-06T07:29:38.208123+0000","last_unstale":"2024-03-06T07:29:38.221109+0000","last_undegraded":"2024-03-06T07:29:38.221109+0000","last_fullsized":"2024-03-06T07:29:38.221109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:53.247884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.208424+0000","last_change":"2024-03-06T07:29:37.197870+0000","last_active":"2024-03-06T07:29:38.208424+0000","last_peered":"2024-03-06T07:29:38.208424+0000","last_clean":"2024-03-06T07:29:38.208424+0000","last_became_active":"2024-03-06T07:29:37.197472+0000","last_became_peered":"2024-03-06T07:29:37.197472+0000","last_unstale":"2024-03-06T07:29:38.208424+0000","last_undegraded":"2024-03-06T07:29:38.208424+0000","last_fullsized":"2024-03-06T07:29:38.208424+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:22.616144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.229654+0000","last_change":"2024-03-06T07:29:38.221823+0000","last_active":"2024-03-06T07:29:38.229654+0000","last_peered":"2024-03-06T07:29:38.229654+0000","last_clean":"2024-03-06T07:29:38.229654+0000","last_became_active":"2024-03-06T07:29:38.217931+0000","last_became_peered":"2024-03-06T07:29:38.217931+0000","last_unstale":"2024-03-06T07:29:38.229654+0000","last_undegraded":"2024-03-06T07:29:38.229654+0000","last_fullsized":"2024-03-06T07:29:38.229654+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:56.047553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.192030+0000","last_change":"2024-03-06T07:29:37.188901+0000","last_active":"2024-03-06T07:29:38.192030+0000","last_peered":"2024-03-06T07:29:38.192030+0000","last_clean":"2024-03-06T07:29:38.192030+0000","last_became_active":"2024-03-06T07:29:37.188623+0000","last_became_peered":"2024-03-06T07:29:37.188623+0000","last_unstale":"2024-03-06T07:29:38.192030+0000","last_undegraded":"2024-03-06T07:29:38.192030+0000","last_fullsized":"2024-03-06T07:29:38.192030+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:15.298957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236330+0000","last_change":"2024-03-06T07:29:38.230658+0000","last_active":"2024-03-06T07:29:38.236330+0000","last_peered":"2024-03-06T07:29:38.236330+0000","last_clean":"2024-03-06T07:29:38.236330+0000","last_became_active":"2024-03-06T07:29:38.230268+0000","last_became_peered":"2024-03-06T07:29:38.230268+0000","last_unstale":"2024-03-06T07:29:38.236330+0000","last_undegraded":"2024-03-06T07:29:38.236330+0000","last_fullsized":"2024-03-06T07:29:38.236330+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:10.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.212418+0000","last_change":"2024-03-06T07:29:37.182731+0000","last_active":"2024-03-06T07:29:38.212418+0000","last_peered":"2024-03-06T07:29:38.212418+0000","last_clean":"2024-03-06T07:29:38.212418+0000","last_became_active":"2024-03-06T07:29:37.181027+0000","last_became_peered":"2024-03-06T07:29:37.181027+0000","last_unstale":"2024-03-06T07:29:38.212418+0000","last_undegraded":"2024-03-06T07:29:38.212418+0000","last_fullsized":"2024-03-06T07:29:38.212418+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:40.235523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236586+0000","last_change":"2024-03-06T07:29:38.233759+0000","last_active":"2024-03-06T07:29:38.236586+0000","last_peered":"2024-03-06T07:29:38.236586+0000","last_clean":"2024-03-06T07:29:38.236586+0000","last_became_active":"2024-03-06T07:29:38.233545+0000","last_became_peered":"2024-03-06T07:29:38.233545+0000","last_unstale":"2024-03-06T07:29:38.236586+0000","last_undegraded":"2024-03-06T07:29:38.236586+0000","last_fullsized":"2024-03-06T07:29:38.236586+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:53.420308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.195228+0000","last_change":"2024-03-06T07:29:37.195168+0000","last_active":"2024-03-06T07:29:38.195228+0000","last_peered":"2024-03-06T07:29:38.195228+0000","last_clean":"2024-03-06T07:29:38.195228+0000","last_became_active":"2024-03-06T07:29:37.194957+0000","last_became_peered":"2024-03-06T07:29:37.194957+0000","last_unstale":"2024-03-06T07:29:38.195228+0000","last_undegraded":"2024-03-06T07:29:38.195228+0000","last_fullsized":"2024-03-06T07:29:38.195228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:40.231550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232859+0000","last_change":"2024-03-06T07:29:38.229028+0000","last_active":"2024-03-06T07:29:38.232859+0000","last_peered":"2024-03-06T07:29:38.232859+0000","last_clean":"2024-03-06T07:29:38.232859+0000","last_became_active":"2024-03-06T07:29:38.228769+0000","last_became_peered":"2024-03-06T07:29:38.228769+0000","last_unstale":"2024-03-06T07:29:38.232859+0000","last_undegraded":"2024-03-06T07:29:38.232859+0000","last_fullsized":"2024-03-06T07:29:38.232859+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:53.360639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235724+0000","last_change":"2024-03-06T07:29:37.201548+0000","last_active":"2024-03-06T07:29:38.235724+0000","last_peered":"2024-03-06T07:29:38.235724+0000","last_clean":"2024-03-06T07:29:38.235724+0000","last_became_active":"2024-03-06T07:29:37.201257+0000","last_became_peered":"2024-03-06T07:29:37.201257+0000","last_unstale":"2024-03-06T07:29:38.235724+0000","last_undegraded":"2024-03-06T07:29:38.235724+0000","last_fullsized":"2024-03-06T07:29:38.235724+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:15.886274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236591+0000","last_change":"2024-03-06T07:29:38.234716+0000","last_active":"2024-03-06T07:29:38.236591+0000","last_peered":"2024-03-06T07:29:38.236591+0000","last_clean":"2024-03-06T07:29:38.236591+0000","last_became_active":"2024-03-06T07:29:38.232296+0000","last_became_peered":"2024-03-06T07:29:38.232296+0000","last_unstale":"2024-03-06T07:29:38.236591+0000","last_undegraded":"2024-03-06T07:29:38.236591+0000","last_fullsized":"2024-03-06T07:29:38.236591+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:34.481980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.245575+0000","last_change":"2024-03-06T07:29:37.181201+0000","last_active":"2024-03-06T07:29:38.245575+0000","last_peered":"2024-03-06T07:29:38.245575+0000","last_clean":"2024-03-06T07:29:38.245575+0000","last_became_active":"2024-03-06T07:29:37.180748+0000","last_became_peered":"2024-03-06T07:29:37.180748+0000","last_unstale":"2024-03-06T07:29:38.245575+0000","last_undegraded":"2024-03-06T07:29:38.245575+0000","last_fullsized":"2024-03-06T07:29:38.245575+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:10.402042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.227908+0000","last_change":"2024-03-06T07:29:38.219357+0000","last_active":"2024-03-06T07:29:38.227908+0000","last_peered":"2024-03-06T07:29:38.227908+0000","last_clean":"2024-03-06T07:29:38.227908+0000","last_became_active":"2024-03-06T07:29:38.218995+0000","last_became_peered":"2024-03-06T07:29:38.218995+0000","last_unstale":"2024-03-06T07:29:38.227908+0000","last_undegraded":"2024-03-06T07:29:38.227908+0000","last_fullsized":"2024-03-06T07:29:38.227908+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:56.807941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.246663+0000","last_change":"2024-03-06T07:29:37.204859+0000","last_active":"2024-03-06T07:29:38.246663+0000","last_peered":"2024-03-06T07:29:38.246663+0000","last_clean":"2024-03-06T07:29:38.246663+0000","last_became_active":"2024-03-06T07:29:37.202958+0000","last_became_peered":"2024-03-06T07:29:37.202958+0000","last_unstale":"2024-03-06T07:29:38.246663+0000","last_undegraded":"2024-03-06T07:29:38.246663+0000","last_fullsized":"2024-03-06T07:29:38.246663+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:35.136528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225795+0000","last_change":"2024-03-06T07:29:38.220755+0000","last_active":"2024-03-06T07:29:38.225795+0000","last_peered":"2024-03-06T07:29:38.225795+0000","last_clean":"2024-03-06T07:29:38.225795+0000","last_became_active":"2024-03-06T07:29:38.220469+0000","last_became_peered":"2024-03-06T07:29:38.220469+0000","last_unstale":"2024-03-06T07:29:38.225795+0000","last_undegraded":"2024-03-06T07:29:38.225795+0000","last_fullsized":"2024-03-06T07:29:38.225795+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:27.312802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.215669+0000","last_change":"2024-03-06T07:29:37.202512+0000","last_active":"2024-03-06T07:29:38.215669+0000","last_peered":"2024-03-06T07:29:38.215669+0000","last_clean":"2024-03-06T07:29:38.215669+0000","last_became_active":"2024-03-06T07:29:37.202194+0000","last_became_peered":"2024-03-06T07:29:37.202194+0000","last_unstale":"2024-03-06T07:29:38.215669+0000","last_undegraded":"2024-03-06T07:29:38.215669+0000","last_fullsized":"2024-03-06T07:29:38.215669+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:07.654693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235291+0000","last_change":"2024-03-06T07:29:38.231622+0000","last_active":"2024-03-06T07:29:38.235291+0000","last_peered":"2024-03-06T07:29:38.235291+0000","last_clean":"2024-03-06T07:29:38.235291+0000","last_became_active":"2024-03-06T07:29:38.231288+0000","last_became_peered":"2024-03-06T07:29:38.231288+0000","last_unstale":"2024-03-06T07:29:38.235291+0000","last_undegraded":"2024-03-06T07:29:38.235291+0000","last_fullsized":"2024-03-06T07:29:38.235291+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:57.064911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.214652+0000","last_change":"2024-03-06T07:29:37.193797+0000","last_active":"2024-03-06T07:29:38.214652+0000","last_peered":"2024-03-06T07:29:38.214652+0000","last_clean":"2024-03-06T07:29:38.214652+0000","last_became_active":"2024-03-06T07:29:37.193578+0000","last_became_peered":"2024-03-06T07:29:37.193578+0000","last_unstale":"2024-03-06T07:29:38.214652+0000","last_undegraded":"2024-03-06T07:29:38.214652+0000","last_fullsized":"2024-03-06T07:29:38.214652+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:32.106556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236016+0000","last_change":"2024-03-06T07:29:38.227365+0000","last_active":"2024-03-06T07:29:38.236016+0000","last_peered":"2024-03-06T07:29:38.236016+0000","last_clean":"2024-03-06T07:29:38.236016+0000","last_became_active":"2024-03-06T07:29:38.224328+0000","last_became_peered":"2024-03-06T07:29:38.224328+0000","last_unstale":"2024-03-06T07:29:38.236016+0000","last_undegraded":"2024-03-06T07:29:38.236016+0000","last_fullsized":"2024-03-06T07:29:38.236016+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:33.122277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.249492+0000","last_change":"2024-03-06T07:29:37.192767+0000","last_active":"2024-03-06T07:29:38.249492+0000","last_peered":"2024-03-06T07:29:38.249492+0000","last_clean":"2024-03-06T07:29:38.249492+0000","last_became_active":"2024-03-06T07:29:37.191181+0000","last_became_peered":"2024-03-06T07:29:37.191181+0000","last_unstale":"2024-03-06T07:29:38.249492+0000","last_undegraded":"2024-03-06T07:29:38.249492+0000","last_fullsized":"2024-03-06T07:29:38.249492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:55.031832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225091+0000","last_change":"2024-03-06T07:29:38.223222+0000","last_active":"2024-03-06T07:29:38.225091+0000","last_peered":"2024-03-06T07:29:38.225091+0000","last_clean":"2024-03-06T07:29:38.225091+0000","last_became_active":"2024-03-06T07:29:38.222903+0000","last_became_peered":"2024-03-06T07:29:38.222903+0000","last_unstale":"2024-03-06T07:29:38.225091+0000","last_undegraded":"2024-03-06T07:29:38.225091+0000","last_fullsized":"2024-03-06T07:29:38.225091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:11.992158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.239322+0000","last_change":"2024-03-06T07:29:37.202023+0000","last_active":"2024-03-06T07:29:38.239322+0000","last_peered":"2024-03-06T07:29:38.239322+0000","last_clean":"2024-03-06T07:29:38.239322+0000","last_became_active":"2024-03-06T07:29:37.199912+0000","last_became_peered":"2024-03-06T07:29:37.199912+0000","last_unstale":"2024-03-06T07:29:38.239322+0000","last_undegraded":"2024-03-06T07:29:38.239322+0000","last_fullsized":"2024-03-06T07:29:38.239322+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:41.551544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225153+0000","last_change":"2024-03-06T07:29:38.223364+0000","last_active":"2024-03-06T07:29:38.225153+0000","last_peered":"2024-03-06T07:29:38.225153+0000","last_clean":"2024-03-06T07:29:38.225153+0000","last_became_active":"2024-03-06T07:29:38.220905+0000","last_became_peered":"2024-03-06T07:29:38.220905+0000","last_unstale":"2024-03-06T07:29:38.225153+0000","last_undegraded":"2024-03-06T07:29:38.225153+0000","last_fullsized":"2024-03-06T07:29:38.225153+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:54.179817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235946+0000","last_change":"2024-03-06T07:29:37.198656+0000","last_active":"2024-03-06T07:29:38.235946+0000","last_peered":"2024-03-06T07:29:38.235946+0000","last_clean":"2024-03-06T07:29:38.235946+0000","last_became_active":"2024-03-06T07:29:37.198410+0000","last_became_peered":"2024-03-06T07:29:37.198410+0000","last_unstale":"2024-03-06T07:29:38.235946+0000","last_undegraded":"2024-03-06T07:29:38.235946+0000","last_fullsized":"2024-03-06T07:29:38.235946+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:30.412946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228593+0000","last_change":"2024-03-06T07:29:38.224061+0000","last_active":"2024-03-06T07:29:38.228593+0000","last_peered":"2024-03-06T07:29:38.228593+0000","last_clean":"2024-03-06T07:29:38.228593+0000","last_became_active":"2024-03-06T07:29:38.223763+0000","last_became_peered":"2024-03-06T07:29:38.223763+0000","last_unstale":"2024-03-06T07:29:38.228593+0000","last_undegraded":"2024-03-06T07:29:38.228593+0000","last_fullsized":"2024-03-06T07:29:38.228593+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:41.247272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.238583+0000","last_change":"2024-03-06T07:29:37.204042+0000","last_active":"2024-03-06T07:29:38.238583+0000","last_peered":"2024-03-06T07:29:38.238583+0000","last_clean":"2024-03-06T07:29:38.238583+0000","last_became_active":"2024-03-06T07:29:37.203797+0000","last_became_peered":"2024-03-06T07:29:37.203797+0000","last_unstale":"2024-03-06T07:29:38.238583+0000","last_undegraded":"2024-03-06T07:29:38.238583+0000","last_fullsized":"2024-03-06T07:29:38.238583+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:21.385366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228857+0000","last_change":"2024-03-06T07:29:38.221257+0000","last_active":"2024-03-06T07:29:38.228857+0000","last_peered":"2024-03-06T07:29:38.228857+0000","last_clean":"2024-03-06T07:29:38.228857+0000","last_became_active":"2024-03-06T07:29:38.220802+0000","last_became_peered":"2024-03-06T07:29:38.220802+0000","last_unstale":"2024-03-06T07:29:38.228857+0000","last_undegraded":"2024-03-06T07:29:38.228857+0000","last_fullsized":"2024-03-06T07:29:38.228857+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:57.463487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.213080+0000","last_change":"2024-03-06T07:29:37.202951+0000","last_active":"2024-03-06T07:29:38.213080+0000","last_peered":"2024-03-06T07:29:38.213080+0000","last_clean":"2024-03-06T07:29:38.213080+0000","last_became_active":"2024-03-06T07:29:37.202658+0000","last_became_peered":"2024-03-06T07:29:37.202658+0000","last_unstale":"2024-03-06T07:29:38.213080+0000","last_undegraded":"2024-03-06T07:29:38.213080+0000","last_fullsized":"2024-03-06T07:29:38.213080+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:35.434940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231993+0000","last_change":"2024-03-06T07:29:38.227768+0000","last_active":"2024-03-06T07:29:38.231993+0000","last_peered":"2024-03-06T07:29:38.231993+0000","last_clean":"2024-03-06T07:29:38.231993+0000","last_became_active":"2024-03-06T07:29:38.226951+0000","last_became_peered":"2024-03-06T07:29:38.226951+0000","last_unstale":"2024-03-06T07:29:38.231993+0000","last_undegraded":"2024-03-06T07:29:38.231993+0000","last_fullsized":"2024-03-06T07:29:38.231993+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:50.564389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.243630+0000","last_change":"2024-03-06T07:29:37.204642+0000","last_active":"2024-03-06T07:29:38.243630+0000","last_peered":"2024-03-06T07:29:38.243630+0000","last_clean":"2024-03-06T07:29:38.243630+0000","last_became_active":"2024-03-06T07:29:37.203762+0000","last_became_peered":"2024-03-06T07:29:37.203762+0000","last_unstale":"2024-03-06T07:29:38.243630+0000","last_undegraded":"2024-03-06T07:29:38.243630+0000","last_fullsized":"2024-03-06T07:29:38.243630+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:52.404586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231263+0000","last_change":"2024-03-06T07:29:38.227267+0000","last_active":"2024-03-06T07:29:38.231263+0000","last_peered":"2024-03-06T07:29:38.231263+0000","last_clean":"2024-03-06T07:29:38.231263+0000","last_became_active":"2024-03-06T07:29:38.226983+0000","last_became_peered":"2024-03-06T07:29:38.226983+0000","last_unstale":"2024-03-06T07:29:38.231263+0000","last_undegraded":"2024-03-06T07:29:38.231263+0000","last_fullsized":"2024-03-06T07:29:38.231263+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:27.312802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.208787+0000","last_change":"2024-03-06T07:29:37.196685+0000","last_active":"2024-03-06T07:29:38.208787+0000","last_peered":"2024-03-06T07:29:38.208787+0000","last_clean":"2024-03-06T07:29:38.208787+0000","last_became_active":"2024-03-06T07:29:37.196368+0000","last_became_peered":"2024-03-06T07:29:37.196368+0000","last_unstale":"2024-03-06T07:29:38.208787+0000","last_undegraded":"2024-03-06T07:29:38.208787+0000","last_fullsized":"2024-03-06T07:29:38.208787+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:41.633967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233726+0000","last_change":"2024-03-06T07:29:38.227543+0000","last_active":"2024-03-06T07:29:38.233726+0000","last_peered":"2024-03-06T07:29:38.233726+0000","last_clean":"2024-03-06T07:29:38.233726+0000","last_became_active":"2024-03-06T07:29:38.227236+0000","last_became_peered":"2024-03-06T07:29:38.227236+0000","last_unstale":"2024-03-06T07:29:38.233726+0000","last_undegraded":"2024-03-06T07:29:38.233726+0000","last_fullsized":"2024-03-06T07:29:38.233726+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:16.314678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.192436+0000","last_change":"2024-03-06T07:29:37.196138+0000","last_active":"2024-03-06T07:29:38.192436+0000","last_peered":"2024-03-06T07:29:38.192436+0000","last_clean":"2024-03-06T07:29:38.192436+0000","last_became_active":"2024-03-06T07:29:37.195602+0000","last_became_peered":"2024-03-06T07:29:37.195602+0000","last_unstale":"2024-03-06T07:29:38.192436+0000","last_undegraded":"2024-03-06T07:29:38.192436+0000","last_fullsized":"2024-03-06T07:29:38.192436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:24.856622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.224524+0000","last_change":"2024-03-06T07:29:38.217306+0000","last_active":"2024-03-06T07:29:38.224524+0000","last_peered":"2024-03-06T07:29:38.224524+0000","last_clean":"2024-03-06T07:29:38.224524+0000","last_became_active":"2024-03-06T07:29:38.216959+0000","last_became_peered":"2024-03-06T07:29:38.216959+0000","last_unstale":"2024-03-06T07:29:38.224524+0000","last_undegraded":"2024-03-06T07:29:38.224524+0000","last_fullsized":"2024-03-06T07:29:38.224524+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:26.795765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.213167+0000","last_change":"2024-03-06T07:29:37.202369+0000","last_active":"2024-03-06T07:29:38.213167+0000","last_peered":"2024-03-06T07:29:38.213167+0000","last_clean":"2024-03-06T07:29:38.213167+0000","last_became_active":"2024-03-06T07:29:37.202093+0000","last_became_peered":"2024-03-06T07:29:37.202093+0000","last_unstale":"2024-03-06T07:29:38.213167+0000","last_undegraded":"2024-03-06T07:29:38.213167+0000","last_fullsized":"2024-03-06T07:29:38.213167+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:21.385366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230001+0000","last_change":"2024-03-06T07:29:38.220769+0000","last_active":"2024-03-06T07:29:38.230001+0000","last_peered":"2024-03-06T07:29:38.230001+0000","last_clean":"2024-03-06T07:29:38.230001+0000","last_became_active":"2024-03-06T07:29:38.220474+0000","last_became_peered":"2024-03-06T07:29:38.220474+0000","last_unstale":"2024-03-06T07:29:38.230001+0000","last_undegraded":"2024-03-06T07:29:38.230001+0000","last_fullsized":"2024-03-06T07:29:38.230001+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:25.872343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.214433+0000","last_change":"2024-03-06T07:29:37.198765+0000","last_active":"2024-03-06T07:29:38.214433+0000","last_peered":"2024-03-06T07:29:38.214433+0000","last_clean":"2024-03-06T07:29:38.214433+0000","last_became_active":"2024-03-06T07:29:37.198477+0000","last_became_peered":"2024-03-06T07:29:37.198477+0000","last_unstale":"2024-03-06T07:29:38.214433+0000","last_undegraded":"2024-03-06T07:29:38.214433+0000","last_fullsized":"2024-03-06T07:29:38.214433+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:46.429660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225109+0000","last_change":"2024-03-06T07:29:38.223635+0000","last_active":"2024-03-06T07:29:38.225109+0000","last_peered":"2024-03-06T07:29:38.225109+0000","last_clean":"2024-03-06T07:29:38.225109+0000","last_became_active":"2024-03-06T07:29:38.221378+0000","last_became_peered":"2024-03-06T07:29:38.221378+0000","last_unstale":"2024-03-06T07:29:38.225109+0000","last_undegraded":"2024-03-06T07:29:38.225109+0000","last_fullsized":"2024-03-06T07:29:38.225109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:04.044411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.216323+0000","last_change":"2024-03-06T07:29:37.202168+0000","last_active":"2024-03-06T07:29:38.216323+0000","last_peered":"2024-03-06T07:29:38.216323+0000","last_clean":"2024-03-06T07:29:38.216323+0000","last_became_active":"2024-03-06T07:29:37.201887+0000","last_became_peered":"2024-03-06T07:29:37.201887+0000","last_unstale":"2024-03-06T07:29:38.216323+0000","last_undegraded":"2024-03-06T07:29:38.216323+0000","last_fullsized":"2024-03-06T07:29:38.216323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:26.787155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236572+0000","last_change":"2024-03-06T07:29:38.233511+0000","last_active":"2024-03-06T07:29:38.236572+0000","last_peered":"2024-03-06T07:29:38.236572+0000","last_clean":"2024-03-06T07:29:38.236572+0000","last_became_active":"2024-03-06T07:29:38.233203+0000","last_became_peered":"2024-03-06T07:29:38.233203+0000","last_unstale":"2024-03-06T07:29:38.236572+0000","last_undegraded":"2024-03-06T07:29:38.236572+0000","last_fullsized":"2024-03-06T07:29:38.236572+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:36.152249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.212447+0000","last_change":"2024-03-06T07:29:37.196836+0000","last_active":"2024-03-06T07:29:38.212447+0000","last_peered":"2024-03-06T07:29:38.212447+0000","last_clean":"2024-03-06T07:29:38.212447+0000","last_became_active":"2024-03-06T07:29:37.196504+0000","last_became_peered":"2024-03-06T07:29:37.196504+0000","last_unstale":"2024-03-06T07:29:38.212447+0000","last_undegraded":"2024-03-06T07:29:38.212447+0000","last_fullsized":"2024-03-06T07:29:38.212447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:25.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231968+0000","last_change":"2024-03-06T07:29:38.227545+0000","last_active":"2024-03-06T07:29:38.231968+0000","last_peered":"2024-03-06T07:29:38.231968+0000","last_clean":"2024-03-06T07:29:38.231968+0000","last_became_active":"2024-03-06T07:29:38.226566+0000","last_became_peered":"2024-03-06T07:29:38.226566+0000","last_unstale":"2024-03-06T07:29:38.231968+0000","last_undegraded":"2024-03-06T07:29:38.231968+0000","last_fullsized":"2024-03-06T07:29:38.231968+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:38.832436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.217676+0000","last_change":"2024-03-06T07:29:37.201293+0000","last_active":"2024-03-06T07:29:38.217676+0000","last_peered":"2024-03-06T07:29:38.217676+0000","last_clean":"2024-03-06T07:29:38.217676+0000","last_became_active":"2024-03-06T07:29:37.200984+0000","last_became_peered":"2024-03-06T07:29:37.200984+0000","last_unstale":"2024-03-06T07:29:38.217676+0000","last_undegraded":"2024-03-06T07:29:38.217676+0000","last_fullsized":"2024-03-06T07:29:38.217676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:35.434940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.240827+0000","last_change":"2024-03-06T07:29:38.235625+0000","last_active":"2024-03-06T07:29:38.240827+0000","last_peered":"2024-03-06T07:29:38.240827+0000","last_clean":"2024-03-06T07:29:38.240827+0000","last_became_active":"2024-03-06T07:29:38.235173+0000","last_became_peered":"2024-03-06T07:29:38.235173+0000","last_unstale":"2024-03-06T07:29:38.240827+0000","last_undegraded":"2024-03-06T07:29:38.240827+0000","last_fullsized":"2024-03-06T07:29:38.240827+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:07.118321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.211019+0000","last_change":"2024-03-06T07:29:37.198215+0000","last_active":"2024-03-06T07:29:38.211019+0000","last_peered":"2024-03-06T07:29:38.211019+0000","last_clean":"2024-03-06T07:29:38.211019+0000","last_became_active":"2024-03-06T07:29:37.197932+0000","last_became_peered":"2024-03-06T07:29:37.197932+0000","last_unstale":"2024-03-06T07:29:38.211019+0000","last_undegraded":"2024-03-06T07:29:38.211019+0000","last_fullsized":"2024-03-06T07:29:38.211019+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:13.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236775+0000","last_change":"2024-03-06T07:29:38.229775+0000","last_active":"2024-03-06T07:29:38.236775+0000","last_peered":"2024-03-06T07:29:38.236775+0000","last_clean":"2024-03-06T07:29:38.236775+0000","last_became_active":"2024-03-06T07:29:38.229194+0000","last_became_peered":"2024-03-06T07:29:38.229194+0000","last_unstale":"2024-03-06T07:29:38.236775+0000","last_undegraded":"2024-03-06T07:29:38.236775+0000","last_fullsized":"2024-03-06T07:29:38.236775+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:47.445382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.211906+0000","last_change":"2024-03-06T07:29:37.202029+0000","last_active":"2024-03-06T07:29:38.211906+0000","last_peered":"2024-03-06T07:29:38.211906+0000","last_clean":"2024-03-06T07:29:38.211906+0000","last_became_active":"2024-03-06T07:29:37.199523+0000","last_became_peered":"2024-03-06T07:29:37.199523+0000","last_unstale":"2024-03-06T07:29:38.211906+0000","last_undegraded":"2024-03-06T07:29:38.211906+0000","last_fullsized":"2024-03-06T07:29:38.211906+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:07.524741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228087+0000","last_change":"2024-03-06T07:29:38.218491+0000","last_active":"2024-03-06T07:29:38.228087+0000","last_peered":"2024-03-06T07:29:38.228087+0000","last_clean":"2024-03-06T07:29:38.228087+0000","last_became_active":"2024-03-06T07:29:38.218206+0000","last_became_peered":"2024-03-06T07:29:38.218206+0000","last_unstale":"2024-03-06T07:29:38.228087+0000","last_undegraded":"2024-03-06T07:29:38.228087+0000","last_fullsized":"2024-03-06T07:29:38.228087+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:09.958078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.192476+0000","last_change":"2024-03-06T07:29:37.199637+0000","last_active":"2024-03-06T07:29:38.192476+0000","last_peered":"2024-03-06T07:29:38.192476+0000","last_clean":"2024-03-06T07:29:38.192476+0000","last_became_active":"2024-03-06T07:29:37.199358+0000","last_became_peered":"2024-03-06T07:29:37.199358+0000","last_unstale":"2024-03-06T07:29:38.192476+0000","last_undegraded":"2024-03-06T07:29:38.192476+0000","last_fullsized":"2024-03-06T07:29:38.192476+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:56.447765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225334+0000","last_change":"2024-03-06T07:29:38.220434+0000","last_active":"2024-03-06T07:29:38.225334+0000","last_peered":"2024-03-06T07:29:38.225334+0000","last_clean":"2024-03-06T07:29:38.225334+0000","last_became_active":"2024-03-06T07:29:38.220236+0000","last_became_peered":"2024-03-06T07:29:38.220236+0000","last_unstale":"2024-03-06T07:29:38.225334+0000","last_undegraded":"2024-03-06T07:29:38.225334+0000","last_fullsized":"2024-03-06T07:29:38.225334+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:46.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.241241+0000","last_change":"2024-03-06T07:29:37.198783+0000","last_active":"2024-03-06T07:29:38.241241+0000","last_peered":"2024-03-06T07:29:38.241241+0000","last_clean":"2024-03-06T07:29:38.241241+0000","last_became_active":"2024-03-06T07:29:37.198506+0000","last_became_peered":"2024-03-06T07:29:37.198506+0000","last_unstale":"2024-03-06T07:29:38.241241+0000","last_undegraded":"2024-03-06T07:29:38.241241+0000","last_fullsized":"2024-03-06T07:29:38.241241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:07.654693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":211,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.213263+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:29:38.213263+0000","last_peered":"2024-03-06T07:29:38.213263+0000","last_clean":"2024-03-06T07:29:38.213263+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:29:38.213263+0000","last_undegraded":"2024-03-06T07:29:38.213263+0000","last_fullsized":"2024-03-06T07:29:38.213263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":802442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":212656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542172,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":528384,"data_stored":212054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":205209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542171,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":205209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":213618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":212054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542172,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":795577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:42.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:42.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.885+0000 7f97df200640 1 Processor -- start 2024-03-06T07:29:42.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.885+0000 7f97df200640 1 -- start start 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97df200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e00b40c0 0x7f97e00b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97df200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f97e01620c0 0x7f97e00b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97df200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e0161900 con 0x7f97e00ae670 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97df200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e00b49e0 con 0x7f97e01620c0 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97df200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e015ebd0 con 0x7f97e00b40c0 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97de800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f97e01620c0 0x7f97e00b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97de800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f97e01620c0 0x7f97e00b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44822/0 (socket says 172.21.15.97:44822) 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.887+0000 7f97de800640 1 -- 172.21.15.97:0/2253327466 learned_addr learned my addr 172.21.15.97:0/2253327466 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.888+0000 7f97dde00640 1 --2- 172.21.15.97:0/2253327466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e00b40c0 0x7f97e00b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.888+0000 7f97de800640 1 -- 172.21.15.97:0/2253327466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e00b40c0 msgr2=0x7f97e00b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.888+0000 7f97de800640 1 --2- 172.21.15.97:0/2253327466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e00b40c0 0x7f97e00b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.888+0000 7f97de800640 1 -- 172.21.15.97:0/2253327466 >> v1:172.21.15.135:6789/0 conn(0x7f97e00ae670 legacy=0x7f97e015e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.888+0000 7f97de800640 1 -- 172.21.15.97:0/2253327466 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e015eeb0 con 0x7f97e01620c0 2024-03-06T07:29:42.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.889+0000 7f97de800640 1 --2- 172.21.15.97:0/2253327466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e01620c0 0x7f97e00b28e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009e50 tx=0x7f97d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bb0e22424e22c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.889+0000 7f97dd400640 1 -- 172.21.15.97:0/2253327466 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d802ca70 con 0x7f97e01620c0 2024-03-06T07:29:42.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.889+0000 7f97dd400640 1 -- 172.21.15.97:0/2253327466 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d802cc10 con 0x7f97e01620c0 2024-03-06T07:29:42.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.889+0000 7f97dd400640 1 -- 172.21.15.97:0/2253327466 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80343f0 con 0x7f97e01620c0 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 -- 172.21.15.97:0/2253327466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e01620c0 msgr2=0x7f97e00b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 --2- 172.21.15.97:0/2253327466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e01620c0 0x7f97e00b28e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009e50 tx=0x7f97d802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 -- 172.21.15.97:0/2253327466 shutdown_connections 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 --2- 172.21.15.97:0/2253327466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e01620c0 0x7f97e00b28e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 --2- 172.21.15.97:0/2253327466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e00b40c0 0x7f97e00b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 -- 172.21.15.97:0/2253327466 >> 172.21.15.97:0/2253327466 conn(0x7f97e006afc0 msgr2=0x7f97e006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 -- 172.21.15.97:0/2253327466 shutdown_connections 2024-03-06T07:29:42.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.890+0000 7f97df200640 1 -- 172.21.15.97:0/2253327466 wait complete. 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.892+0000 7f97df200640 1 Processor -- start 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.892+0000 7f97df200640 1 -- start start 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97df200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97e00ae670 0x7f97e014a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97df200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 0x7f97e0159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97df200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e01620c0 0x7f97e015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97df200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0158660 con 0x7f97e00b40c0 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97df200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0161ba0 con 0x7f97e00ae670 2024-03-06T07:29:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97df200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e01587e0 con 0x7f97e01620c0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 0x7f97e0159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 0x7f97e0159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44832/0 (socket says 172.21.15.97:44832) 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 -- 172.21.15.97:0/2544566070 learned_addr learned my addr 172.21.15.97:0/2544566070 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97dde00640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97e00ae670 0x7f97e014a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 -- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e01620c0 msgr2=0x7f97e015a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e01620c0 0x7f97e015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 -- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97e00ae670 msgr2=0x7f97e014a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97e00ae670 0x7f97e014a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.893+0000 7f97d5400640 1 -- 172.21.15.97:0/2544566070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e015a6b0 con 0x7f97e00b40c0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.894+0000 7f97d5400640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 0x7f97e0159b80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f97c800b3f0 tx=0x7f97c800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.894+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8015020 con 0x7f97e00b40c0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.894+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c8011830 con 0x7f97e00b40c0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.894+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e015a990 con 0x7f97e00b40c0 2024-03-06T07:29:42.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.894+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8011c20 con 0x7f97e00b40c0 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.894+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e015af50 con 0x7f97e00b40c0 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.895+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e004f330 con 0x7f97e00b40c0 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.896+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97c8018020 con 0x7f97e00b40c0 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.897+0000 7f97d6a00640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f97bc06b4d0 0x7f97bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.897+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97c8089ee0 con 0x7f97e00b40c0 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.897+0000 7f97dde00640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f97bc06b4d0 0x7f97bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.898+0000 7f97dde00640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f97bc06b4d0 0x7f97bc06d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f97d0006fd0 tx=0x7f97d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:42.903+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97c801f030 con 0x7f97e00b40c0 2024-03-06T07:29:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.096+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97e00adf40 con 0x7f97bc06b4d0 2024-03-06T07:29:43.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.106+0000 7f97d6a00640 1 -- 172.21.15.97:0/2544566070 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251894 (secure 0 0 0) 0x7f97e00adf40 con 0x7f97bc06b4d0 2024-03-06T07:29:43.112 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:43.113 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:29:43.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.112+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f97bc06b4d0 msgr2=0x7f97bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.112+0000 7f97df200640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f97bc06b4d0 0x7f97bc06d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f97d0006fd0 tx=0x7f97d0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.112+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 msgr2=0x7f97e0159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.112+0000 7f97df200640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 0x7f97e0159b80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f97c800b3f0 tx=0x7f97c800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 shutdown_connections 2024-03-06T07:29:43.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f97bc06b4d0 0x7f97bc06d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97e01620c0 0x7f97e015a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97e00b40c0 0x7f97e0159b80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 --2- 172.21.15.97:0/2544566070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97e00ae670 0x7f97e014a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 >> 172.21.15.97:0/2544566070 conn(0x7f97e006afc0 msgr2=0x7f97e0155370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 shutdown_connections 2024-03-06T07:29:43.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.113+0000 7f97df200640 1 -- 172.21.15.97:0/2544566070 wait complete. 2024-03-06T07:29:43.133 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-03-06T07:29:42.936870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250560,"kb_used_data":5256,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724160,"statfs":{"total":773094113280,"available":772837539840,"internally_reserved":0,"allocated":5382144,"data_stored":2858819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001676"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235988+0000","last_change":"2024-03-06T07:29:38.230569+0000","last_active":"2024-03-06T07:29:38.235988+0000","last_peered":"2024-03-06T07:29:38.235988+0000","last_clean":"2024-03-06T07:29:38.235988+0000","last_became_active":"2024-03-06T07:29:38.229487+0000","last_became_peered":"2024-03-06T07:29:38.229487+0000","last_unstale":"2024-03-06T07:29:38.235988+0000","last_undegraded":"2024-03-06T07:29:38.235988+0000","last_fullsized":"2024-03-06T07:29:38.235988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:15.599863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233221+0000","last_change":"2024-03-06T07:29:38.224626+0000","last_active":"2024-03-06T07:29:38.233221+0000","last_peered":"2024-03-06T07:29:38.233221+0000","last_clean":"2024-03-06T07:29:38.233221+0000","last_became_active":"2024-03-06T07:29:38.223739+0000","last_became_peered":"2024-03-06T07:29:38.223739+0000","last_unstale":"2024-03-06T07:29:38.233221+0000","last_undegraded":"2024-03-06T07:29:38.233221+0000","last_fullsized":"2024-03-06T07:29:38.233221+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:58.295575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228936+0000","last_change":"2024-03-06T07:29:38.225631+0000","last_active":"2024-03-06T07:29:38.228936+0000","last_peered":"2024-03-06T07:29:38.228936+0000","last_clean":"2024-03-06T07:29:38.228936+0000","last_became_active":"2024-03-06T07:29:38.224305+0000","last_became_peered":"2024-03-06T07:29:38.224305+0000","last_unstale":"2024-03-06T07:29:38.228936+0000","last_undegraded":"2024-03-06T07:29:38.228936+0000","last_fullsized":"2024-03-06T07:29:38.228936+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:57.801906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.222667+0000","last_change":"2024-03-06T07:29:38.212896+0000","last_active":"2024-03-06T07:29:38.222667+0000","last_peered":"2024-03-06T07:29:38.222667+0000","last_clean":"2024-03-06T07:29:38.222667+0000","last_became_active":"2024-03-06T07:29:38.210119+0000","last_became_peered":"2024-03-06T07:29:38.210119+0000","last_unstale":"2024-03-06T07:29:38.222667+0000","last_undegraded":"2024-03-06T07:29:38.222667+0000","last_fullsized":"2024-03-06T07:29:38.222667+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:12.292948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231782+0000","last_change":"2024-03-06T07:29:38.224985+0000","last_active":"2024-03-06T07:29:38.231782+0000","last_peered":"2024-03-06T07:29:38.231782+0000","last_clean":"2024-03-06T07:29:38.231782+0000","last_became_active":"2024-03-06T07:29:38.224690+0000","last_became_peered":"2024-03-06T07:29:38.224690+0000","last_unstale":"2024-03-06T07:29:38.231782+0000","last_undegraded":"2024-03-06T07:29:38.231782+0000","last_fullsized":"2024-03-06T07:29:38.231782+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:36.450661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232011+0000","last_change":"2024-03-06T07:29:38.227982+0000","last_active":"2024-03-06T07:29:38.232011+0000","last_peered":"2024-03-06T07:29:38.232011+0000","last_clean":"2024-03-06T07:29:38.232011+0000","last_became_active":"2024-03-06T07:29:38.227296+0000","last_became_peered":"2024-03-06T07:29:38.227296+0000","last_unstale":"2024-03-06T07:29:38.232011+0000","last_undegraded":"2024-03-06T07:29:38.232011+0000","last_fullsized":"2024-03-06T07:29:38.232011+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:25.689904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235996+0000","last_change":"2024-03-06T07:29:38.227651+0000","last_active":"2024-03-06T07:29:38.235996+0000","last_peered":"2024-03-06T07:29:38.235996+0000","last_clean":"2024-03-06T07:29:38.235996+0000","last_became_active":"2024-03-06T07:29:38.226435+0000","last_became_peered":"2024-03-06T07:29:38.226435+0000","last_unstale":"2024-03-06T07:29:38.235996+0000","last_undegraded":"2024-03-06T07:29:38.235996+0000","last_fullsized":"2024-03-06T07:29:38.235996+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:47.445382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.212606+0000","last_change":"2024-03-06T07:29:38.206428+0000","last_active":"2024-03-06T07:29:38.212606+0000","last_peered":"2024-03-06T07:29:38.212606+0000","last_clean":"2024-03-06T07:29:38.212606+0000","last_became_active":"2024-03-06T07:29:38.206136+0000","last_became_peered":"2024-03-06T07:29:38.206136+0000","last_unstale":"2024-03-06T07:29:38.212606+0000","last_undegraded":"2024-03-06T07:29:38.212606+0000","last_fullsized":"2024-03-06T07:29:38.212606+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:22.577604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.224604+0000","last_change":"2024-03-06T07:29:38.217108+0000","last_active":"2024-03-06T07:29:38.224604+0000","last_peered":"2024-03-06T07:29:38.224604+0000","last_clean":"2024-03-06T07:29:38.224604+0000","last_became_active":"2024-03-06T07:29:38.216805+0000","last_became_peered":"2024-03-06T07:29:38.216805+0000","last_unstale":"2024-03-06T07:29:38.224604+0000","last_undegraded":"2024-03-06T07:29:38.224604+0000","last_fullsized":"2024-03-06T07:29:38.224604+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:08.540463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225246+0000","last_change":"2024-03-06T07:29:38.219333+0000","last_active":"2024-03-06T07:29:38.225246+0000","last_peered":"2024-03-06T07:29:38.225246+0000","last_clean":"2024-03-06T07:29:38.225246+0000","last_became_active":"2024-03-06T07:29:38.217493+0000","last_became_peered":"2024-03-06T07:29:38.217493+0000","last_unstale":"2024-03-06T07:29:38.225246+0000","last_undegraded":"2024-03-06T07:29:38.225246+0000","last_fullsized":"2024-03-06T07:29:38.225246+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:39.896470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225064+0000","last_change":"2024-03-06T07:29:38.217603+0000","last_active":"2024-03-06T07:29:38.225064+0000","last_peered":"2024-03-06T07:29:38.225064+0000","last_clean":"2024-03-06T07:29:38.225064+0000","last_became_active":"2024-03-06T07:29:38.217115+0000","last_became_peered":"2024-03-06T07:29:38.217115+0000","last_unstale":"2024-03-06T07:29:38.225064+0000","last_undegraded":"2024-03-06T07:29:38.225064+0000","last_fullsized":"2024-03-06T07:29:38.225064+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:18.591917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.237043+0000","last_change":"2024-03-06T07:29:38.232746+0000","last_active":"2024-03-06T07:29:38.237043+0000","last_peered":"2024-03-06T07:29:38.237043+0000","last_clean":"2024-03-06T07:29:38.237043+0000","last_became_active":"2024-03-06T07:29:38.230999+0000","last_became_peered":"2024-03-06T07:29:38.230999+0000","last_unstale":"2024-03-06T07:29:38.237043+0000","last_undegraded":"2024-03-06T07:29:38.237043+0000","last_fullsized":"2024-03-06T07:29:38.237043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:33.122277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232083+0000","last_change":"2024-03-06T07:29:38.225460+0000","last_active":"2024-03-06T07:29:38.232083+0000","last_peered":"2024-03-06T07:29:38.232083+0000","last_clean":"2024-03-06T07:29:38.232083+0000","last_became_active":"2024-03-06T07:29:38.224462+0000","last_became_peered":"2024-03-06T07:29:38.224462+0000","last_unstale":"2024-03-06T07:29:38.232083+0000","last_undegraded":"2024-03-06T07:29:38.232083+0000","last_fullsized":"2024-03-06T07:29:38.232083+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:25.872343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230726+0000","last_change":"2024-03-06T07:29:38.227747+0000","last_active":"2024-03-06T07:29:38.230726+0000","last_peered":"2024-03-06T07:29:38.230726+0000","last_clean":"2024-03-06T07:29:38.230726+0000","last_became_active":"2024-03-06T07:29:38.226172+0000","last_became_peered":"2024-03-06T07:29:38.226172+0000","last_unstale":"2024-03-06T07:29:38.230726+0000","last_undegraded":"2024-03-06T07:29:38.230726+0000","last_fullsized":"2024-03-06T07:29:38.230726+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:28:40.782881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232486+0000","last_change":"2024-03-06T07:29:38.230424+0000","last_active":"2024-03-06T07:29:38.232486+0000","last_peered":"2024-03-06T07:29:38.232486+0000","last_clean":"2024-03-06T07:29:38.232486+0000","last_became_active":"2024-03-06T07:29:38.230076+0000","last_became_peered":"2024-03-06T07:29:38.230076+0000","last_unstale":"2024-03-06T07:29:38.232486+0000","last_undegraded":"2024-03-06T07:29:38.232486+0000","last_fullsized":"2024-03-06T07:29:38.232486+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:25.691880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.227871+0000","last_change":"2024-03-06T07:29:38.219507+0000","last_active":"2024-03-06T07:29:38.227871+0000","last_peered":"2024-03-06T07:29:38.227871+0000","last_clean":"2024-03-06T07:29:38.227871+0000","last_became_active":"2024-03-06T07:29:38.219104+0000","last_became_peered":"2024-03-06T07:29:38.219104+0000","last_unstale":"2024-03-06T07:29:38.227871+0000","last_undegraded":"2024-03-06T07:29:38.227871+0000","last_fullsized":"2024-03-06T07:29:38.227871+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:11.992158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.234600+0000","last_change":"2024-03-06T07:29:38.227287+0000","last_active":"2024-03-06T07:29:38.234600+0000","last_peered":"2024-03-06T07:29:38.234600+0000","last_clean":"2024-03-06T07:29:38.234600+0000","last_became_active":"2024-03-06T07:29:38.226950+0000","last_became_peered":"2024-03-06T07:29:38.226950+0000","last_unstale":"2024-03-06T07:29:38.234600+0000","last_undegraded":"2024-03-06T07:29:38.234600+0000","last_fullsized":"2024-03-06T07:29:38.234600+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.420975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236353+0000","last_change":"2024-03-06T07:29:38.231283+0000","last_active":"2024-03-06T07:29:38.236353+0000","last_peered":"2024-03-06T07:29:38.236353+0000","last_clean":"2024-03-06T07:29:38.236353+0000","last_became_active":"2024-03-06T07:29:38.230997+0000","last_became_peered":"2024-03-06T07:29:38.230997+0000","last_unstale":"2024-03-06T07:29:38.236353+0000","last_undegraded":"2024-03-06T07:29:38.236353+0000","last_fullsized":"2024-03-06T07:29:38.236353+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:54.828137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228835+0000","last_change":"2024-03-06T07:29:38.219968+0000","last_active":"2024-03-06T07:29:38.228835+0000","last_peered":"2024-03-06T07:29:38.228835+0000","last_clean":"2024-03-06T07:29:38.228835+0000","last_became_active":"2024-03-06T07:29:38.219465+0000","last_became_peered":"2024-03-06T07:29:38.219465+0000","last_unstale":"2024-03-06T07:29:38.228835+0000","last_undegraded":"2024-03-06T07:29:38.228835+0000","last_fullsized":"2024-03-06T07:29:38.228835+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:16.314678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.223102+0000","last_change":"2024-03-06T07:29:38.215089+0000","last_active":"2024-03-06T07:29:38.223102+0000","last_peered":"2024-03-06T07:29:38.223102+0000","last_clean":"2024-03-06T07:29:38.223102+0000","last_became_active":"2024-03-06T07:29:38.214776+0000","last_became_peered":"2024-03-06T07:29:38.214776+0000","last_unstale":"2024-03-06T07:29:38.223102+0000","last_undegraded":"2024-03-06T07:29:38.223102+0000","last_fullsized":"2024-03-06T07:29:38.223102+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:56.714122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.240796+0000","last_change":"2024-03-06T07:29:38.234584+0000","last_active":"2024-03-06T07:29:38.240796+0000","last_peered":"2024-03-06T07:29:38.240796+0000","last_clean":"2024-03-06T07:29:38.240796+0000","last_became_active":"2024-03-06T07:29:38.233282+0000","last_became_peered":"2024-03-06T07:29:38.233282+0000","last_unstale":"2024-03-06T07:29:38.240796+0000","last_undegraded":"2024-03-06T07:29:38.240796+0000","last_fullsized":"2024-03-06T07:29:38.240796+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:42.990968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.220015+0000","last_change":"2024-03-06T07:29:38.208648+0000","last_active":"2024-03-06T07:29:38.220015+0000","last_peered":"2024-03-06T07:29:38.220015+0000","last_clean":"2024-03-06T07:29:38.220015+0000","last_became_active":"2024-03-06T07:29:38.208319+0000","last_became_peered":"2024-03-06T07:29:38.208319+0000","last_unstale":"2024-03-06T07:29:38.220015+0000","last_undegraded":"2024-03-06T07:29:38.220015+0000","last_fullsized":"2024-03-06T07:29:38.220015+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:22.621316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.229627+0000","last_change":"2024-03-06T07:29:38.218295+0000","last_active":"2024-03-06T07:29:38.229627+0000","last_peered":"2024-03-06T07:29:38.229627+0000","last_clean":"2024-03-06T07:29:38.229627+0000","last_became_active":"2024-03-06T07:29:38.217587+0000","last_became_peered":"2024-03-06T07:29:38.217587+0000","last_unstale":"2024-03-06T07:29:38.229627+0000","last_undegraded":"2024-03-06T07:29:38.229627+0000","last_fullsized":"2024-03-06T07:29:38.229627+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:56.714122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228581+0000","last_change":"2024-03-06T07:29:38.226506+0000","last_active":"2024-03-06T07:29:38.228581+0000","last_peered":"2024-03-06T07:29:38.228581+0000","last_clean":"2024-03-06T07:29:38.228581+0000","last_became_active":"2024-03-06T07:29:38.226183+0000","last_became_peered":"2024-03-06T07:29:38.226183+0000","last_unstale":"2024-03-06T07:29:38.228581+0000","last_undegraded":"2024-03-06T07:29:38.228581+0000","last_fullsized":"2024-03-06T07:29:38.228581+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:22.577604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.216727+0000","last_change":"2024-03-06T07:29:38.213303+0000","last_active":"2024-03-06T07:29:38.216727+0000","last_peered":"2024-03-06T07:29:38.216727+0000","last_clean":"2024-03-06T07:29:38.216727+0000","last_became_active":"2024-03-06T07:29:38.213003+0000","last_became_peered":"2024-03-06T07:29:38.213003+0000","last_unstale":"2024-03-06T07:29:38.216727+0000","last_undegraded":"2024-03-06T07:29:38.216727+0000","last_fullsized":"2024-03-06T07:29:38.216727+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.790636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225071+0000","last_change":"2024-03-06T07:29:38.221634+0000","last_active":"2024-03-06T07:29:38.225071+0000","last_peered":"2024-03-06T07:29:38.225071+0000","last_clean":"2024-03-06T07:29:38.225071+0000","last_became_active":"2024-03-06T07:29:38.220504+0000","last_became_peered":"2024-03-06T07:29:38.220504+0000","last_unstale":"2024-03-06T07:29:38.225071+0000","last_undegraded":"2024-03-06T07:29:38.225071+0000","last_fullsized":"2024-03-06T07:29:38.225071+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:56.807941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.234534+0000","last_change":"2024-03-06T07:29:38.226824+0000","last_active":"2024-03-06T07:29:38.234534+0000","last_peered":"2024-03-06T07:29:38.234534+0000","last_clean":"2024-03-06T07:29:38.234534+0000","last_became_active":"2024-03-06T07:29:38.226415+0000","last_became_peered":"2024-03-06T07:29:38.226415+0000","last_unstale":"2024-03-06T07:29:38.234534+0000","last_undegraded":"2024-03-06T07:29:38.234534+0000","last_fullsized":"2024-03-06T07:29:38.234534+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:29.634273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230778+0000","last_change":"2024-03-06T07:29:38.217495+0000","last_active":"2024-03-06T07:29:38.230778+0000","last_peered":"2024-03-06T07:29:38.230778+0000","last_clean":"2024-03-06T07:29:38.230778+0000","last_became_active":"2024-03-06T07:29:38.216542+0000","last_became_peered":"2024-03-06T07:29:38.216542+0000","last_unstale":"2024-03-06T07:29:38.230778+0000","last_undegraded":"2024-03-06T07:29:38.230778+0000","last_fullsized":"2024-03-06T07:29:38.230778+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:16.901995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.216574+0000","last_change":"2024-03-06T07:29:38.212495+0000","last_active":"2024-03-06T07:29:38.216574+0000","last_peered":"2024-03-06T07:29:38.216574+0000","last_clean":"2024-03-06T07:29:38.216574+0000","last_became_active":"2024-03-06T07:29:38.212228+0000","last_became_peered":"2024-03-06T07:29:38.212228+0000","last_unstale":"2024-03-06T07:29:38.216574+0000","last_undegraded":"2024-03-06T07:29:38.216574+0000","last_fullsized":"2024-03-06T07:29:38.216574+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.567265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228856+0000","last_change":"2024-03-06T07:29:38.220521+0000","last_active":"2024-03-06T07:29:38.228856+0000","last_peered":"2024-03-06T07:29:38.228856+0000","last_clean":"2024-03-06T07:29:38.228856+0000","last_became_active":"2024-03-06T07:29:38.220223+0000","last_became_peered":"2024-03-06T07:29:38.220223+0000","last_unstale":"2024-03-06T07:29:38.228856+0000","last_undegraded":"2024-03-06T07:29:38.228856+0000","last_fullsized":"2024-03-06T07:29:38.228856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:28.715523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230332+0000","last_change":"2024-03-06T07:29:38.226544+0000","last_active":"2024-03-06T07:29:38.230332+0000","last_peered":"2024-03-06T07:29:38.230332+0000","last_clean":"2024-03-06T07:29:38.230332+0000","last_became_active":"2024-03-06T07:29:38.224799+0000","last_became_peered":"2024-03-06T07:29:38.224799+0000","last_unstale":"2024-03-06T07:29:38.230332+0000","last_undegraded":"2024-03-06T07:29:38.230332+0000","last_fullsized":"2024-03-06T07:29:38.230332+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:40.234045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.209541+0000","last_change":"2024-03-06T07:29:38.206941+0000","last_active":"2024-03-06T07:29:38.209541+0000","last_peered":"2024-03-06T07:29:38.209541+0000","last_clean":"2024-03-06T07:29:38.209541+0000","last_became_active":"2024-03-06T07:29:38.205924+0000","last_became_peered":"2024-03-06T07:29:38.205924+0000","last_unstale":"2024-03-06T07:29:38.209541+0000","last_undegraded":"2024-03-06T07:29:38.209541+0000","last_fullsized":"2024-03-06T07:29:38.209541+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.790636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225313+0000","last_change":"2024-03-06T07:29:38.222586+0000","last_active":"2024-03-06T07:29:38.225313+0000","last_peered":"2024-03-06T07:29:38.225313+0000","last_clean":"2024-03-06T07:29:38.225313+0000","last_became_active":"2024-03-06T07:29:38.222256+0000","last_became_peered":"2024-03-06T07:29:38.222256+0000","last_unstale":"2024-03-06T07:29:38.225313+0000","last_undegraded":"2024-03-06T07:29:38.225313+0000","last_fullsized":"2024-03-06T07:29:38.225313+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:46.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.243852+0000","last_change":"2024-03-06T07:29:37.203528+0000","last_active":"2024-03-06T07:29:38.243852+0000","last_peered":"2024-03-06T07:29:38.243852+0000","last_clean":"2024-03-06T07:29:38.243852+0000","last_became_active":"2024-03-06T07:29:37.203220+0000","last_became_peered":"2024-03-06T07:29:37.203220+0000","last_unstale":"2024-03-06T07:29:38.243852+0000","last_undegraded":"2024-03-06T07:29:38.243852+0000","last_fullsized":"2024-03-06T07:29:38.243852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:14.584142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.234400+0000","last_change":"2024-03-06T07:29:38.229863+0000","last_active":"2024-03-06T07:29:38.234400+0000","last_peered":"2024-03-06T07:29:38.234400+0000","last_clean":"2024-03-06T07:29:38.234400+0000","last_became_active":"2024-03-06T07:29:38.227652+0000","last_became_peered":"2024-03-06T07:29:38.227652+0000","last_unstale":"2024-03-06T07:29:38.234400+0000","last_undegraded":"2024-03-06T07:29:38.234400+0000","last_fullsized":"2024-03-06T07:29:38.234400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:53.247884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233002+0000","last_change":"2024-03-06T07:29:37.198982+0000","last_active":"2024-03-06T07:29:38.233002+0000","last_peered":"2024-03-06T07:29:38.233002+0000","last_clean":"2024-03-06T07:29:38.233002+0000","last_became_active":"2024-03-06T07:29:37.198670+0000","last_became_peered":"2024-03-06T07:29:37.198670+0000","last_unstale":"2024-03-06T07:29:38.233002+0000","last_undegraded":"2024-03-06T07:29:38.233002+0000","last_fullsized":"2024-03-06T07:29:38.233002+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:30.030268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233728+0000","last_change":"2024-03-06T07:29:38.222647+0000","last_active":"2024-03-06T07:29:38.233728+0000","last_peered":"2024-03-06T07:29:38.233728+0000","last_clean":"2024-03-06T07:29:38.233728+0000","last_became_active":"2024-03-06T07:29:38.222297+0000","last_became_peered":"2024-03-06T07:29:38.222297+0000","last_unstale":"2024-03-06T07:29:38.233728+0000","last_undegraded":"2024-03-06T07:29:38.233728+0000","last_fullsized":"2024-03-06T07:29:38.233728+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:27.802877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.241367+0000","last_change":"2024-03-06T07:29:37.207021+0000","last_active":"2024-03-06T07:29:38.241367+0000","last_peered":"2024-03-06T07:29:38.241367+0000","last_clean":"2024-03-06T07:29:38.241367+0000","last_became_active":"2024-03-06T07:29:37.206714+0000","last_became_peered":"2024-03-06T07:29:37.206714+0000","last_unstale":"2024-03-06T07:29:38.241367+0000","last_undegraded":"2024-03-06T07:29:38.241367+0000","last_fullsized":"2024-03-06T07:29:38.241367+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:17.576195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.224282+0000","last_change":"2024-03-06T07:29:38.220647+0000","last_active":"2024-03-06T07:29:38.224282+0000","last_peered":"2024-03-06T07:29:38.224282+0000","last_clean":"2024-03-06T07:29:38.224282+0000","last_became_active":"2024-03-06T07:29:38.220298+0000","last_became_peered":"2024-03-06T07:29:38.220298+0000","last_unstale":"2024-03-06T07:29:38.224282+0000","last_undegraded":"2024-03-06T07:29:38.224282+0000","last_fullsized":"2024-03-06T07:29:38.224282+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:39.896470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.237194+0000","last_change":"2024-03-06T07:29:37.194194+0000","last_active":"2024-03-06T07:29:38.237194+0000","last_peered":"2024-03-06T07:29:38.237194+0000","last_clean":"2024-03-06T07:29:38.237194+0000","last_became_active":"2024-03-06T07:29:37.193871+0000","last_became_peered":"2024-03-06T07:29:37.193871+0000","last_unstale":"2024-03-06T07:29:38.237194+0000","last_undegraded":"2024-03-06T07:29:38.237194+0000","last_fullsized":"2024-03-06T07:29:38.237194+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:57.279853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.227946+0000","last_change":"2024-03-06T07:29:38.217119+0000","last_active":"2024-03-06T07:29:38.227946+0000","last_peered":"2024-03-06T07:29:38.227946+0000","last_clean":"2024-03-06T07:29:38.227946+0000","last_became_active":"2024-03-06T07:29:38.216159+0000","last_became_peered":"2024-03-06T07:29:38.216159+0000","last_unstale":"2024-03-06T07:29:38.227946+0000","last_undegraded":"2024-03-06T07:29:38.227946+0000","last_fullsized":"2024-03-06T07:29:38.227946+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:02.697325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.243477+0000","last_change":"2024-03-06T07:29:37.204958+0000","last_active":"2024-03-06T07:29:38.243477+0000","last_peered":"2024-03-06T07:29:38.243477+0000","last_clean":"2024-03-06T07:29:38.243477+0000","last_became_active":"2024-03-06T07:29:37.204553+0000","last_became_peered":"2024-03-06T07:29:37.204553+0000","last_unstale":"2024-03-06T07:29:38.243477+0000","last_undegraded":"2024-03-06T07:29:38.243477+0000","last_fullsized":"2024-03-06T07:29:38.243477+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:33.466259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.221478+0000","last_change":"2024-03-06T07:29:38.210751+0000","last_active":"2024-03-06T07:29:38.221478+0000","last_peered":"2024-03-06T07:29:38.221478+0000","last_clean":"2024-03-06T07:29:38.221478+0000","last_became_active":"2024-03-06T07:29:38.210453+0000","last_became_peered":"2024-03-06T07:29:38.210453+0000","last_unstale":"2024-03-06T07:29:38.221478+0000","last_undegraded":"2024-03-06T07:29:38.221478+0000","last_fullsized":"2024-03-06T07:29:38.221478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.693659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.237232+0000","last_change":"2024-03-06T07:29:37.190755+0000","last_active":"2024-03-06T07:29:38.237232+0000","last_peered":"2024-03-06T07:29:38.237232+0000","last_clean":"2024-03-06T07:29:38.237232+0000","last_became_active":"2024-03-06T07:29:37.190175+0000","last_became_peered":"2024-03-06T07:29:37.190175+0000","last_unstale":"2024-03-06T07:29:38.237232+0000","last_undegraded":"2024-03-06T07:29:38.237232+0000","last_fullsized":"2024-03-06T07:29:38.237232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:10.970889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.221109+0000","last_change":"2024-03-06T07:29:38.208445+0000","last_active":"2024-03-06T07:29:38.221109+0000","last_peered":"2024-03-06T07:29:38.221109+0000","last_clean":"2024-03-06T07:29:38.221109+0000","last_became_active":"2024-03-06T07:29:38.208123+0000","last_became_peered":"2024-03-06T07:29:38.208123+0000","last_unstale":"2024-03-06T07:29:38.221109+0000","last_undegraded":"2024-03-06T07:29:38.221109+0000","last_fullsized":"2024-03-06T07:29:38.221109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:53.247884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.208424+0000","last_change":"2024-03-06T07:29:37.197870+0000","last_active":"2024-03-06T07:29:38.208424+0000","last_peered":"2024-03-06T07:29:38.208424+0000","last_clean":"2024-03-06T07:29:38.208424+0000","last_became_active":"2024-03-06T07:29:37.197472+0000","last_became_peered":"2024-03-06T07:29:37.197472+0000","last_unstale":"2024-03-06T07:29:38.208424+0000","last_undegraded":"2024-03-06T07:29:38.208424+0000","last_fullsized":"2024-03-06T07:29:38.208424+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:22.616144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.229654+0000","last_change":"2024-03-06T07:29:38.221823+0000","last_active":"2024-03-06T07:29:38.229654+0000","last_peered":"2024-03-06T07:29:38.229654+0000","last_clean":"2024-03-06T07:29:38.229654+0000","last_became_active":"2024-03-06T07:29:38.217931+0000","last_became_peered":"2024-03-06T07:29:38.217931+0000","last_unstale":"2024-03-06T07:29:38.229654+0000","last_undegraded":"2024-03-06T07:29:38.229654+0000","last_fullsized":"2024-03-06T07:29:38.229654+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:56.047553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.192030+0000","last_change":"2024-03-06T07:29:37.188901+0000","last_active":"2024-03-06T07:29:38.192030+0000","last_peered":"2024-03-06T07:29:38.192030+0000","last_clean":"2024-03-06T07:29:38.192030+0000","last_became_active":"2024-03-06T07:29:37.188623+0000","last_became_peered":"2024-03-06T07:29:37.188623+0000","last_unstale":"2024-03-06T07:29:38.192030+0000","last_undegraded":"2024-03-06T07:29:38.192030+0000","last_fullsized":"2024-03-06T07:29:38.192030+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:15.298957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236330+0000","last_change":"2024-03-06T07:29:38.230658+0000","last_active":"2024-03-06T07:29:38.236330+0000","last_peered":"2024-03-06T07:29:38.236330+0000","last_clean":"2024-03-06T07:29:38.236330+0000","last_became_active":"2024-03-06T07:29:38.230268+0000","last_became_peered":"2024-03-06T07:29:38.230268+0000","last_unstale":"2024-03-06T07:29:38.236330+0000","last_undegraded":"2024-03-06T07:29:38.236330+0000","last_fullsized":"2024-03-06T07:29:38.236330+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:10.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.212418+0000","last_change":"2024-03-06T07:29:37.182731+0000","last_active":"2024-03-06T07:29:38.212418+0000","last_peered":"2024-03-06T07:29:38.212418+0000","last_clean":"2024-03-06T07:29:38.212418+0000","last_became_active":"2024-03-06T07:29:37.181027+0000","last_became_peered":"2024-03-06T07:29:37.181027+0000","last_unstale":"2024-03-06T07:29:38.212418+0000","last_undegraded":"2024-03-06T07:29:38.212418+0000","last_fullsized":"2024-03-06T07:29:38.212418+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:40.235523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236586+0000","last_change":"2024-03-06T07:29:38.233759+0000","last_active":"2024-03-06T07:29:38.236586+0000","last_peered":"2024-03-06T07:29:38.236586+0000","last_clean":"2024-03-06T07:29:38.236586+0000","last_became_active":"2024-03-06T07:29:38.233545+0000","last_became_peered":"2024-03-06T07:29:38.233545+0000","last_unstale":"2024-03-06T07:29:38.236586+0000","last_undegraded":"2024-03-06T07:29:38.236586+0000","last_fullsized":"2024-03-06T07:29:38.236586+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:53.420308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.195228+0000","last_change":"2024-03-06T07:29:37.195168+0000","last_active":"2024-03-06T07:29:38.195228+0000","last_peered":"2024-03-06T07:29:38.195228+0000","last_clean":"2024-03-06T07:29:38.195228+0000","last_became_active":"2024-03-06T07:29:37.194957+0000","last_became_peered":"2024-03-06T07:29:37.194957+0000","last_unstale":"2024-03-06T07:29:38.195228+0000","last_undegraded":"2024-03-06T07:29:38.195228+0000","last_fullsized":"2024-03-06T07:29:38.195228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:40.231550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.232859+0000","last_change":"2024-03-06T07:29:38.229028+0000","last_active":"2024-03-06T07:29:38.232859+0000","last_peered":"2024-03-06T07:29:38.232859+0000","last_clean":"2024-03-06T07:29:38.232859+0000","last_became_active":"2024-03-06T07:29:38.228769+0000","last_became_peered":"2024-03-06T07:29:38.228769+0000","last_unstale":"2024-03-06T07:29:38.232859+0000","last_undegraded":"2024-03-06T07:29:38.232859+0000","last_fullsized":"2024-03-06T07:29:38.232859+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:53.360639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235724+0000","last_change":"2024-03-06T07:29:37.201548+0000","last_active":"2024-03-06T07:29:38.235724+0000","last_peered":"2024-03-06T07:29:38.235724+0000","last_clean":"2024-03-06T07:29:38.235724+0000","last_became_active":"2024-03-06T07:29:37.201257+0000","last_became_peered":"2024-03-06T07:29:37.201257+0000","last_unstale":"2024-03-06T07:29:38.235724+0000","last_undegraded":"2024-03-06T07:29:38.235724+0000","last_fullsized":"2024-03-06T07:29:38.235724+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:15.886274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236591+0000","last_change":"2024-03-06T07:29:38.234716+0000","last_active":"2024-03-06T07:29:38.236591+0000","last_peered":"2024-03-06T07:29:38.236591+0000","last_clean":"2024-03-06T07:29:38.236591+0000","last_became_active":"2024-03-06T07:29:38.232296+0000","last_became_peered":"2024-03-06T07:29:38.232296+0000","last_unstale":"2024-03-06T07:29:38.236591+0000","last_undegraded":"2024-03-06T07:29:38.236591+0000","last_fullsized":"2024-03-06T07:29:38.236591+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:34.481980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.245575+0000","last_change":"2024-03-06T07:29:37.181201+0000","last_active":"2024-03-06T07:29:38.245575+0000","last_peered":"2024-03-06T07:29:38.245575+0000","last_clean":"2024-03-06T07:29:38.245575+0000","last_became_active":"2024-03-06T07:29:37.180748+0000","last_became_peered":"2024-03-06T07:29:37.180748+0000","last_unstale":"2024-03-06T07:29:38.245575+0000","last_undegraded":"2024-03-06T07:29:38.245575+0000","last_fullsized":"2024-03-06T07:29:38.245575+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:10.402042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.227908+0000","last_change":"2024-03-06T07:29:38.219357+0000","last_active":"2024-03-06T07:29:38.227908+0000","last_peered":"2024-03-06T07:29:38.227908+0000","last_clean":"2024-03-06T07:29:38.227908+0000","last_became_active":"2024-03-06T07:29:38.218995+0000","last_became_peered":"2024-03-06T07:29:38.218995+0000","last_unstale":"2024-03-06T07:29:38.227908+0000","last_undegraded":"2024-03-06T07:29:38.227908+0000","last_fullsized":"2024-03-06T07:29:38.227908+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:56.807941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.246663+0000","last_change":"2024-03-06T07:29:37.204859+0000","last_active":"2024-03-06T07:29:38.246663+0000","last_peered":"2024-03-06T07:29:38.246663+0000","last_clean":"2024-03-06T07:29:38.246663+0000","last_became_active":"2024-03-06T07:29:37.202958+0000","last_became_peered":"2024-03-06T07:29:37.202958+0000","last_unstale":"2024-03-06T07:29:38.246663+0000","last_undegraded":"2024-03-06T07:29:38.246663+0000","last_fullsized":"2024-03-06T07:29:38.246663+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:35.136528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225795+0000","last_change":"2024-03-06T07:29:38.220755+0000","last_active":"2024-03-06T07:29:38.225795+0000","last_peered":"2024-03-06T07:29:38.225795+0000","last_clean":"2024-03-06T07:29:38.225795+0000","last_became_active":"2024-03-06T07:29:38.220469+0000","last_became_peered":"2024-03-06T07:29:38.220469+0000","last_unstale":"2024-03-06T07:29:38.225795+0000","last_undegraded":"2024-03-06T07:29:38.225795+0000","last_fullsized":"2024-03-06T07:29:38.225795+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:27.312802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.215669+0000","last_change":"2024-03-06T07:29:37.202512+0000","last_active":"2024-03-06T07:29:38.215669+0000","last_peered":"2024-03-06T07:29:38.215669+0000","last_clean":"2024-03-06T07:29:38.215669+0000","last_became_active":"2024-03-06T07:29:37.202194+0000","last_became_peered":"2024-03-06T07:29:37.202194+0000","last_unstale":"2024-03-06T07:29:38.215669+0000","last_undegraded":"2024-03-06T07:29:38.215669+0000","last_fullsized":"2024-03-06T07:29:38.215669+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:07.654693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235291+0000","last_change":"2024-03-06T07:29:38.231622+0000","last_active":"2024-03-06T07:29:38.235291+0000","last_peered":"2024-03-06T07:29:38.235291+0000","last_clean":"2024-03-06T07:29:38.235291+0000","last_became_active":"2024-03-06T07:29:38.231288+0000","last_became_peered":"2024-03-06T07:29:38.231288+0000","last_unstale":"2024-03-06T07:29:38.235291+0000","last_undegraded":"2024-03-06T07:29:38.235291+0000","last_fullsized":"2024-03-06T07:29:38.235291+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:57.064911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.214652+0000","last_change":"2024-03-06T07:29:37.193797+0000","last_active":"2024-03-06T07:29:38.214652+0000","last_peered":"2024-03-06T07:29:38.214652+0000","last_clean":"2024-03-06T07:29:38.214652+0000","last_became_active":"2024-03-06T07:29:37.193578+0000","last_became_peered":"2024-03-06T07:29:37.193578+0000","last_unstale":"2024-03-06T07:29:38.214652+0000","last_undegraded":"2024-03-06T07:29:38.214652+0000","last_fullsized":"2024-03-06T07:29:38.214652+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:32.106556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236016+0000","last_change":"2024-03-06T07:29:38.227365+0000","last_active":"2024-03-06T07:29:38.236016+0000","last_peered":"2024-03-06T07:29:38.236016+0000","last_clean":"2024-03-06T07:29:38.236016+0000","last_became_active":"2024-03-06T07:29:38.224328+0000","last_became_peered":"2024-03-06T07:29:38.224328+0000","last_unstale":"2024-03-06T07:29:38.236016+0000","last_undegraded":"2024-03-06T07:29:38.236016+0000","last_fullsized":"2024-03-06T07:29:38.236016+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:33.122277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.249492+0000","last_change":"2024-03-06T07:29:37.192767+0000","last_active":"2024-03-06T07:29:38.249492+0000","last_peered":"2024-03-06T07:29:38.249492+0000","last_clean":"2024-03-06T07:29:38.249492+0000","last_became_active":"2024-03-06T07:29:37.191181+0000","last_became_peered":"2024-03-06T07:29:37.191181+0000","last_unstale":"2024-03-06T07:29:38.249492+0000","last_undegraded":"2024-03-06T07:29:38.249492+0000","last_fullsized":"2024-03-06T07:29:38.249492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:55.031832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225091+0000","last_change":"2024-03-06T07:29:38.223222+0000","last_active":"2024-03-06T07:29:38.225091+0000","last_peered":"2024-03-06T07:29:38.225091+0000","last_clean":"2024-03-06T07:29:38.225091+0000","last_became_active":"2024-03-06T07:29:38.222903+0000","last_became_peered":"2024-03-06T07:29:38.222903+0000","last_unstale":"2024-03-06T07:29:38.225091+0000","last_undegraded":"2024-03-06T07:29:38.225091+0000","last_fullsized":"2024-03-06T07:29:38.225091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:11.992158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.239322+0000","last_change":"2024-03-06T07:29:37.202023+0000","last_active":"2024-03-06T07:29:38.239322+0000","last_peered":"2024-03-06T07:29:38.239322+0000","last_clean":"2024-03-06T07:29:38.239322+0000","last_became_active":"2024-03-06T07:29:37.199912+0000","last_became_peered":"2024-03-06T07:29:37.199912+0000","last_unstale":"2024-03-06T07:29:38.239322+0000","last_undegraded":"2024-03-06T07:29:38.239322+0000","last_fullsized":"2024-03-06T07:29:38.239322+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:41.551544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225153+0000","last_change":"2024-03-06T07:29:38.223364+0000","last_active":"2024-03-06T07:29:38.225153+0000","last_peered":"2024-03-06T07:29:38.225153+0000","last_clean":"2024-03-06T07:29:38.225153+0000","last_became_active":"2024-03-06T07:29:38.220905+0000","last_became_peered":"2024-03-06T07:29:38.220905+0000","last_unstale":"2024-03-06T07:29:38.225153+0000","last_undegraded":"2024-03-06T07:29:38.225153+0000","last_fullsized":"2024-03-06T07:29:38.225153+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:54.179817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.235946+0000","last_change":"2024-03-06T07:29:37.198656+0000","last_active":"2024-03-06T07:29:38.235946+0000","last_peered":"2024-03-06T07:29:38.235946+0000","last_clean":"2024-03-06T07:29:38.235946+0000","last_became_active":"2024-03-06T07:29:37.198410+0000","last_became_peered":"2024-03-06T07:29:37.198410+0000","last_unstale":"2024-03-06T07:29:38.235946+0000","last_undegraded":"2024-03-06T07:29:38.235946+0000","last_fullsized":"2024-03-06T07:29:38.235946+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:30.412946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228593+0000","last_change":"2024-03-06T07:29:38.224061+0000","last_active":"2024-03-06T07:29:38.228593+0000","last_peered":"2024-03-06T07:29:38.228593+0000","last_clean":"2024-03-06T07:29:38.228593+0000","last_became_active":"2024-03-06T07:29:38.223763+0000","last_became_peered":"2024-03-06T07:29:38.223763+0000","last_unstale":"2024-03-06T07:29:38.228593+0000","last_undegraded":"2024-03-06T07:29:38.228593+0000","last_fullsized":"2024-03-06T07:29:38.228593+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:41.247272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.238583+0000","last_change":"2024-03-06T07:29:37.204042+0000","last_active":"2024-03-06T07:29:38.238583+0000","last_peered":"2024-03-06T07:29:38.238583+0000","last_clean":"2024-03-06T07:29:38.238583+0000","last_became_active":"2024-03-06T07:29:37.203797+0000","last_became_peered":"2024-03-06T07:29:37.203797+0000","last_unstale":"2024-03-06T07:29:38.238583+0000","last_undegraded":"2024-03-06T07:29:38.238583+0000","last_fullsized":"2024-03-06T07:29:38.238583+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:21.385366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228857+0000","last_change":"2024-03-06T07:29:38.221257+0000","last_active":"2024-03-06T07:29:38.228857+0000","last_peered":"2024-03-06T07:29:38.228857+0000","last_clean":"2024-03-06T07:29:38.228857+0000","last_became_active":"2024-03-06T07:29:38.220802+0000","last_became_peered":"2024-03-06T07:29:38.220802+0000","last_unstale":"2024-03-06T07:29:38.228857+0000","last_undegraded":"2024-03-06T07:29:38.228857+0000","last_fullsized":"2024-03-06T07:29:38.228857+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:57.463487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.213080+0000","last_change":"2024-03-06T07:29:37.202951+0000","last_active":"2024-03-06T07:29:38.213080+0000","last_peered":"2024-03-06T07:29:38.213080+0000","last_clean":"2024-03-06T07:29:38.213080+0000","last_became_active":"2024-03-06T07:29:37.202658+0000","last_became_peered":"2024-03-06T07:29:37.202658+0000","last_unstale":"2024-03-06T07:29:38.213080+0000","last_undegraded":"2024-03-06T07:29:38.213080+0000","last_fullsized":"2024-03-06T07:29:38.213080+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:35.434940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231993+0000","last_change":"2024-03-06T07:29:38.227768+0000","last_active":"2024-03-06T07:29:38.231993+0000","last_peered":"2024-03-06T07:29:38.231993+0000","last_clean":"2024-03-06T07:29:38.231993+0000","last_became_active":"2024-03-06T07:29:38.226951+0000","last_became_peered":"2024-03-06T07:29:38.226951+0000","last_unstale":"2024-03-06T07:29:38.231993+0000","last_undegraded":"2024-03-06T07:29:38.231993+0000","last_fullsized":"2024-03-06T07:29:38.231993+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:50.564389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.243630+0000","last_change":"2024-03-06T07:29:37.204642+0000","last_active":"2024-03-06T07:29:38.243630+0000","last_peered":"2024-03-06T07:29:38.243630+0000","last_clean":"2024-03-06T07:29:38.243630+0000","last_became_active":"2024-03-06T07:29:37.203762+0000","last_became_peered":"2024-03-06T07:29:37.203762+0000","last_unstale":"2024-03-06T07:29:38.243630+0000","last_undegraded":"2024-03-06T07:29:38.243630+0000","last_fullsized":"2024-03-06T07:29:38.243630+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:52.404586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231263+0000","last_change":"2024-03-06T07:29:38.227267+0000","last_active":"2024-03-06T07:29:38.231263+0000","last_peered":"2024-03-06T07:29:38.231263+0000","last_clean":"2024-03-06T07:29:38.231263+0000","last_became_active":"2024-03-06T07:29:38.226983+0000","last_became_peered":"2024-03-06T07:29:38.226983+0000","last_unstale":"2024-03-06T07:29:38.231263+0000","last_undegraded":"2024-03-06T07:29:38.231263+0000","last_fullsized":"2024-03-06T07:29:38.231263+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:27.312802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.208787+0000","last_change":"2024-03-06T07:29:37.196685+0000","last_active":"2024-03-06T07:29:38.208787+0000","last_peered":"2024-03-06T07:29:38.208787+0000","last_clean":"2024-03-06T07:29:38.208787+0000","last_became_active":"2024-03-06T07:29:37.196368+0000","last_became_peered":"2024-03-06T07:29:37.196368+0000","last_unstale":"2024-03-06T07:29:38.208787+0000","last_undegraded":"2024-03-06T07:29:38.208787+0000","last_fullsized":"2024-03-06T07:29:38.208787+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:41.633967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.233726+0000","last_change":"2024-03-06T07:29:38.227543+0000","last_active":"2024-03-06T07:29:38.233726+0000","last_peered":"2024-03-06T07:29:38.233726+0000","last_clean":"2024-03-06T07:29:38.233726+0000","last_became_active":"2024-03-06T07:29:38.227236+0000","last_became_peered":"2024-03-06T07:29:38.227236+0000","last_unstale":"2024-03-06T07:29:38.233726+0000","last_undegraded":"2024-03-06T07:29:38.233726+0000","last_fullsized":"2024-03-06T07:29:38.233726+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:16.314678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.192436+0000","last_change":"2024-03-06T07:29:37.196138+0000","last_active":"2024-03-06T07:29:38.192436+0000","last_peered":"2024-03-06T07:29:38.192436+0000","last_clean":"2024-03-06T07:29:38.192436+0000","last_became_active":"2024-03-06T07:29:37.195602+0000","last_became_peered":"2024-03-06T07:29:37.195602+0000","last_unstale":"2024-03-06T07:29:38.192436+0000","last_undegraded":"2024-03-06T07:29:38.192436+0000","last_fullsized":"2024-03-06T07:29:38.192436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:24.856622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.224524+0000","last_change":"2024-03-06T07:29:38.217306+0000","last_active":"2024-03-06T07:29:38.224524+0000","last_peered":"2024-03-06T07:29:38.224524+0000","last_clean":"2024-03-06T07:29:38.224524+0000","last_became_active":"2024-03-06T07:29:38.216959+0000","last_became_peered":"2024-03-06T07:29:38.216959+0000","last_unstale":"2024-03-06T07:29:38.224524+0000","last_undegraded":"2024-03-06T07:29:38.224524+0000","last_fullsized":"2024-03-06T07:29:38.224524+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:26.795765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.213167+0000","last_change":"2024-03-06T07:29:37.202369+0000","last_active":"2024-03-06T07:29:38.213167+0000","last_peered":"2024-03-06T07:29:38.213167+0000","last_clean":"2024-03-06T07:29:38.213167+0000","last_became_active":"2024-03-06T07:29:37.202093+0000","last_became_peered":"2024-03-06T07:29:37.202093+0000","last_unstale":"2024-03-06T07:29:38.213167+0000","last_undegraded":"2024-03-06T07:29:38.213167+0000","last_fullsized":"2024-03-06T07:29:38.213167+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:21.385366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.230001+0000","last_change":"2024-03-06T07:29:38.220769+0000","last_active":"2024-03-06T07:29:38.230001+0000","last_peered":"2024-03-06T07:29:38.230001+0000","last_clean":"2024-03-06T07:29:38.230001+0000","last_became_active":"2024-03-06T07:29:38.220474+0000","last_became_peered":"2024-03-06T07:29:38.220474+0000","last_unstale":"2024-03-06T07:29:38.230001+0000","last_undegraded":"2024-03-06T07:29:38.230001+0000","last_fullsized":"2024-03-06T07:29:38.230001+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:25.872343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.214433+0000","last_change":"2024-03-06T07:29:37.198765+0000","last_active":"2024-03-06T07:29:38.214433+0000","last_peered":"2024-03-06T07:29:38.214433+0000","last_clean":"2024-03-06T07:29:38.214433+0000","last_became_active":"2024-03-06T07:29:37.198477+0000","last_became_peered":"2024-03-06T07:29:37.198477+0000","last_unstale":"2024-03-06T07:29:38.214433+0000","last_undegraded":"2024-03-06T07:29:38.214433+0000","last_fullsized":"2024-03-06T07:29:38.214433+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:46.429660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225109+0000","last_change":"2024-03-06T07:29:38.223635+0000","last_active":"2024-03-06T07:29:38.225109+0000","last_peered":"2024-03-06T07:29:38.225109+0000","last_clean":"2024-03-06T07:29:38.225109+0000","last_became_active":"2024-03-06T07:29:38.221378+0000","last_became_peered":"2024-03-06T07:29:38.221378+0000","last_unstale":"2024-03-06T07:29:38.225109+0000","last_undegraded":"2024-03-06T07:29:38.225109+0000","last_fullsized":"2024-03-06T07:29:38.225109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:04.044411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.216323+0000","last_change":"2024-03-06T07:29:37.202168+0000","last_active":"2024-03-06T07:29:38.216323+0000","last_peered":"2024-03-06T07:29:38.216323+0000","last_clean":"2024-03-06T07:29:38.216323+0000","last_became_active":"2024-03-06T07:29:37.201887+0000","last_became_peered":"2024-03-06T07:29:37.201887+0000","last_unstale":"2024-03-06T07:29:38.216323+0000","last_undegraded":"2024-03-06T07:29:38.216323+0000","last_fullsized":"2024-03-06T07:29:38.216323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:26.787155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236572+0000","last_change":"2024-03-06T07:29:38.233511+0000","last_active":"2024-03-06T07:29:38.236572+0000","last_peered":"2024-03-06T07:29:38.236572+0000","last_clean":"2024-03-06T07:29:38.236572+0000","last_became_active":"2024-03-06T07:29:38.233203+0000","last_became_peered":"2024-03-06T07:29:38.233203+0000","last_unstale":"2024-03-06T07:29:38.236572+0000","last_undegraded":"2024-03-06T07:29:38.236572+0000","last_fullsized":"2024-03-06T07:29:38.236572+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:36.152249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.212447+0000","last_change":"2024-03-06T07:29:37.196836+0000","last_active":"2024-03-06T07:29:38.212447+0000","last_peered":"2024-03-06T07:29:38.212447+0000","last_clean":"2024-03-06T07:29:38.212447+0000","last_became_active":"2024-03-06T07:29:37.196504+0000","last_became_peered":"2024-03-06T07:29:37.196504+0000","last_unstale":"2024-03-06T07:29:38.212447+0000","last_undegraded":"2024-03-06T07:29:38.212447+0000","last_fullsized":"2024-03-06T07:29:38.212447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:25.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.231968+0000","last_change":"2024-03-06T07:29:38.227545+0000","last_active":"2024-03-06T07:29:38.231968+0000","last_peered":"2024-03-06T07:29:38.231968+0000","last_clean":"2024-03-06T07:29:38.231968+0000","last_became_active":"2024-03-06T07:29:38.226566+0000","last_became_peered":"2024-03-06T07:29:38.226566+0000","last_unstale":"2024-03-06T07:29:38.231968+0000","last_undegraded":"2024-03-06T07:29:38.231968+0000","last_fullsized":"2024-03-06T07:29:38.231968+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:38.832436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.217676+0000","last_change":"2024-03-06T07:29:37.201293+0000","last_active":"2024-03-06T07:29:38.217676+0000","last_peered":"2024-03-06T07:29:38.217676+0000","last_clean":"2024-03-06T07:29:38.217676+0000","last_became_active":"2024-03-06T07:29:37.200984+0000","last_became_peered":"2024-03-06T07:29:37.200984+0000","last_unstale":"2024-03-06T07:29:38.217676+0000","last_undegraded":"2024-03-06T07:29:38.217676+0000","last_fullsized":"2024-03-06T07:29:38.217676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:35.434940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.240827+0000","last_change":"2024-03-06T07:29:38.235625+0000","last_active":"2024-03-06T07:29:38.240827+0000","last_peered":"2024-03-06T07:29:38.240827+0000","last_clean":"2024-03-06T07:29:38.240827+0000","last_became_active":"2024-03-06T07:29:38.235173+0000","last_became_peered":"2024-03-06T07:29:38.235173+0000","last_unstale":"2024-03-06T07:29:38.240827+0000","last_undegraded":"2024-03-06T07:29:38.240827+0000","last_fullsized":"2024-03-06T07:29:38.240827+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:07.118321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.211019+0000","last_change":"2024-03-06T07:29:37.198215+0000","last_active":"2024-03-06T07:29:38.211019+0000","last_peered":"2024-03-06T07:29:38.211019+0000","last_clean":"2024-03-06T07:29:38.211019+0000","last_became_active":"2024-03-06T07:29:37.197932+0000","last_became_peered":"2024-03-06T07:29:37.197932+0000","last_unstale":"2024-03-06T07:29:38.211019+0000","last_undegraded":"2024-03-06T07:29:38.211019+0000","last_fullsized":"2024-03-06T07:29:38.211019+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:13.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.236775+0000","last_change":"2024-03-06T07:29:38.229775+0000","last_active":"2024-03-06T07:29:38.236775+0000","last_peered":"2024-03-06T07:29:38.236775+0000","last_clean":"2024-03-06T07:29:38.236775+0000","last_became_active":"2024-03-06T07:29:38.229194+0000","last_became_peered":"2024-03-06T07:29:38.229194+0000","last_unstale":"2024-03-06T07:29:38.236775+0000","last_undegraded":"2024-03-06T07:29:38.236775+0000","last_fullsized":"2024-03-06T07:29:38.236775+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:47.445382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.211906+0000","last_change":"2024-03-06T07:29:37.202029+0000","last_active":"2024-03-06T07:29:38.211906+0000","last_peered":"2024-03-06T07:29:38.211906+0000","last_clean":"2024-03-06T07:29:38.211906+0000","last_became_active":"2024-03-06T07:29:37.199523+0000","last_became_peered":"2024-03-06T07:29:37.199523+0000","last_unstale":"2024-03-06T07:29:38.211906+0000","last_undegraded":"2024-03-06T07:29:38.211906+0000","last_fullsized":"2024-03-06T07:29:38.211906+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:07.524741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.228087+0000","last_change":"2024-03-06T07:29:38.218491+0000","last_active":"2024-03-06T07:29:38.228087+0000","last_peered":"2024-03-06T07:29:38.228087+0000","last_clean":"2024-03-06T07:29:38.228087+0000","last_became_active":"2024-03-06T07:29:38.218206+0000","last_became_peered":"2024-03-06T07:29:38.218206+0000","last_unstale":"2024-03-06T07:29:38.228087+0000","last_undegraded":"2024-03-06T07:29:38.228087+0000","last_fullsized":"2024-03-06T07:29:38.228087+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:09.958078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.192476+0000","last_change":"2024-03-06T07:29:37.199637+0000","last_active":"2024-03-06T07:29:38.192476+0000","last_peered":"2024-03-06T07:29:38.192476+0000","last_clean":"2024-03-06T07:29:38.192476+0000","last_became_active":"2024-03-06T07:29:37.199358+0000","last_became_peered":"2024-03-06T07:29:37.199358+0000","last_unstale":"2024-03-06T07:29:38.192476+0000","last_undegraded":"2024-03-06T07:29:38.192476+0000","last_fullsized":"2024-03-06T07:29:38.192476+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:56.447765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.225334+0000","last_change":"2024-03-06T07:29:38.220434+0000","last_active":"2024-03-06T07:29:38.225334+0000","last_peered":"2024-03-06T07:29:38.225334+0000","last_clean":"2024-03-06T07:29:38.225334+0000","last_became_active":"2024-03-06T07:29:38.220236+0000","last_became_peered":"2024-03-06T07:29:38.220236+0000","last_unstale":"2024-03-06T07:29:38.225334+0000","last_undegraded":"2024-03-06T07:29:38.225334+0000","last_fullsized":"2024-03-06T07:29:38.225334+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:37.165456+0000","last_clean_scrub_stamp":"2024-03-06T07:29:37.165456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:46.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.241241+0000","last_change":"2024-03-06T07:29:37.198783+0000","last_active":"2024-03-06T07:29:38.241241+0000","last_peered":"2024-03-06T07:29:38.241241+0000","last_clean":"2024-03-06T07:29:38.241241+0000","last_became_active":"2024-03-06T07:29:37.198506+0000","last_became_peered":"2024-03-06T07:29:37.198506+0000","last_unstale":"2024-03-06T07:29:38.241241+0000","last_undegraded":"2024-03-06T07:29:38.241241+0000","last_fullsized":"2024-03-06T07:29:38.241241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:36.149735+0000","last_clean_scrub_stamp":"2024-03-06T07:29:36.149735+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:07.654693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":211,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:38.213263+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:29:38.213263+0000","last_peered":"2024-03-06T07:29:38.213263+0000","last_clean":"2024-03-06T07:29:38.213263+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:29:38.213263+0000","last_undegraded":"2024-03-06T07:29:38.213263+0000","last_fullsized":"2024-03-06T07:29:38.213263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":802442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":212656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542172,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":528384,"data_stored":212054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":205209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542171,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":205209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":213618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":212054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542172,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":795577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:43.137 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:43.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:29:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.393+0000 7efe6ca00640 1 Processor -- start 2024-03-06T07:29:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.394+0000 7efe6ca00640 1 -- start start 2024-03-06T07:29:43.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.395+0000 7efe6ca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efe680b3530 0x7efe6815b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.395+0000 7efe6ca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efe680ad670 0x7efe680adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.395+0000 7efe6ca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6815bcc0 con 0x7efe680b3910 2024-03-06T07:29:43.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.395+0000 7efe6ca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe680ce690 con 0x7efe680b3530 2024-03-06T07:29:43.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.395+0000 7efe6ca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe680ce970 con 0x7efe680ad670 2024-03-06T07:29:43.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe5fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efe680b3910 0x7efe680acf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48060/0 (socket says 172.21.15.97:48060) 2024-03-06T07:29:43.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe5fe00640 1 -- 172.21.15.97:0/3781103244 learned_addr learned my addr 172.21.15.97:0/3781103244 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:43.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe65e00640 1 --2- 172.21.15.97:0/3781103244 >> v2:172.21.15.97:3300/0 conn(0x7efe680b3530 0x7efe6815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe66800640 1 --2- 172.21.15.97:0/3781103244 >> v2:172.21.15.135:3300/0 conn(0x7efe680ad670 0x7efe680adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe65e00640 1 -- 172.21.15.97:0/3781103244 >> v2:172.21.15.135:3300/0 conn(0x7efe680ad670 msgr2=0x7efe680adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe65e00640 1 --2- 172.21.15.97:0/3781103244 >> v2:172.21.15.135:3300/0 conn(0x7efe680ad670 0x7efe680adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe65e00640 1 -- 172.21.15.97:0/3781103244 >> v1:172.21.15.97:6789/0 conn(0x7efe680b3910 legacy=0x7efe680acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe65e00640 1 -- 172.21.15.97:0/3781103244 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680cec50 con 0x7efe680b3530 2024-03-06T07:29:43.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.396+0000 7efe65400640 1 -- 172.21.15.97:0/3781103244 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713525838 0 0) 0x7efe50001110 con 0x7efe680b3910 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.397+0000 7efe65e00640 1 --2- 172.21.15.97:0/3781103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3530 0x7efe6815b780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe68074910 tx=0x7efe5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e54149b8af1e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe65400640 1 -- 172.21.15.97:0/3781103244 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58035020 con 0x7efe680b3530 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe65400640 1 -- 172.21.15.97:0/3781103244 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe58031830 con 0x7efe680b3530 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe65400640 1 -- 172.21.15.97:0/3781103244 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58031c00 con 0x7efe680b3530 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 -- 172.21.15.97:0/3781103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3530 msgr2=0x7efe6815b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3781103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3530 0x7efe6815b780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe68074910 tx=0x7efe5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 -- 172.21.15.97:0/3781103244 shutdown_connections 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3781103244 >> v2:172.21.15.135:3300/0 conn(0x7efe680ad670 0x7efe680adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3781103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3530 0x7efe6815b780 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 -- 172.21.15.97:0/3781103244 >> 172.21.15.97:0/3781103244 conn(0x7efe680b1940 msgr2=0x7efe680b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 -- 172.21.15.97:0/3781103244 shutdown_connections 2024-03-06T07:29:43.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.398+0000 7efe6ca00640 1 -- 172.21.15.97:0/3781103244 wait complete. 2024-03-06T07:29:43.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.400+0000 7efe6ca00640 1 Processor -- start 2024-03-06T07:29:43.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.400+0000 7efe6ca00640 1 -- start start 2024-03-06T07:29:43.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe6ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe680ad670 0x7efe681445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe6ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe680b3530 0x7efe6806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe6ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 0x7efe68145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe6ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b3d90 con 0x7efe680b3910 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe6ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680cf210 con 0x7efe680ad670 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe6ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6815aa30 con 0x7efe680b3530 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe66800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 0x7efe68145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe66800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 0x7efe68145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44862/0 (socket says 172.21.15.97:44862) 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.401+0000 7efe66800640 1 -- 172.21.15.97:0/3143567565 learned_addr learned my addr 172.21.15.97:0/3143567565 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe65e00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe680ad670 0x7efe681445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe66800640 1 -- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe680b3530 msgr2=0x7efe6806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe5fe00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe680b3530 0x7efe6806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe66800640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe680b3530 0x7efe6806e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe66800640 1 -- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe680ad670 msgr2=0x7efe681445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe66800640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe680ad670 0x7efe681445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe66800640 1 -- 172.21.15.97:0/3143567565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68144bd0 con 0x7efe680b3910 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe65e00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe680ad670 0x7efe681445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe66800640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 0x7efe68145c70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efe60007e40 tx=0x7efe6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60014620 con 0x7efe680b3910 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60019070 con 0x7efe680b3910 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60014a20 con 0x7efe680b3910 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe68144eb0 con 0x7efe680b3910 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.402+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe68145470 con 0x7efe680b3910 2024-03-06T07:29:43.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.404+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6804f330 con 0x7efe680b3910 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.404+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe60014bc0 con 0x7efe680b3910 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.405+0000 7efe5e000640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe4406b4d0 0x7efe4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.405+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe6008aae0 con 0x7efe680b3910 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.406+0000 7efe65e00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe4406b4d0 0x7efe4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.406+0000 7efe65e00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe4406b4d0 0x7efe4406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efe58036000 tx=0x7efe58008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.410+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe60059950 con 0x7efe680b3910 2024-03-06T07:29:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.676+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efe680adf80 con 0x7efe680b3910 2024-03-06T07:29:43.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.680+0000 7efe5e000640 1 -- 172.21.15.97:0/3143567565 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7efe60059250 con 0x7efe680b3910 2024-03-06T07:29:43.682 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:29:43.683 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:29:43.683 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:29:43.683 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:29:43.683 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:43.683 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe4406b4d0 msgr2=0x7efe4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe4406b4d0 0x7efe4406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efe58036000 tx=0x7efe58008030 comp rx=0 tx=0).stop 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 msgr2=0x7efe68145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 0x7efe68145c70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efe60007e40 tx=0x7efe6000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 shutdown_connections 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe4406b4d0 0x7efe4406d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe680b3910 0x7efe68145c70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe680b3530 0x7efe6806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 --2- 172.21.15.97:0/3143567565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe680ad670 0x7efe681445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 >> 172.21.15.97:0/3143567565 conn(0x7efe680b1940 msgr2=0x7efe680b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 shutdown_connections 2024-03-06T07:29:43.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.683+0000 7efe6ca00640 1 -- 172.21.15.97:0/3143567565 wait complete. 2024-03-06T07:29:43.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.902+0000 7f14e1a00640 1 Processor -- start 2024-03-06T07:29:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.902+0000 7f14e1a00640 1 -- start start 2024-03-06T07:29:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14e1a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 0x7f14dc070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14e1a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14dc071d30 con 0x7f14dc0af1c0 2024-03-06T07:29:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14e1a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14dc15f4a0 con 0x7f14dc15eff0 2024-03-06T07:29:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14e1a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc071260 con 0x7f14dc06ebe0 2024-03-06T07:29:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14daa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f14dc0af1c0 0x7f14dc06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41998/0 (socket says 172.21.15.97:41998) 2024-03-06T07:29:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14daa00640 1 -- 172.21.15.97:0/3347448630 learned_addr learned my addr 172.21.15.97:0/3347448630 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:43.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14db400640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 0x7f14dc070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946963777 0 0) 0x7f14cc001110 con 0x7f14dc0af1c0 2024-03-06T07:29:43.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.904+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0003520 con 0x7f14dc0af1c0 2024-03-06T07:29:43.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101102366 0 0) 0x7f14c0003520 con 0x7f14dc0af1c0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 msgr2=0x7f14dc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 0x7f14dc070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 >> v1:172.21.15.135:6789/0 conn(0x7f14dc15eff0 legacy=0x7f14dc06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc071490 con 0x7f14dc0af1c0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f14cc001840 con 0x7f14dc0af1c0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 >> v1:172.21.15.97:6789/0 conn(0x7f14dc0af1c0 legacy=0x7f14dc06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 0x7f14c00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14c0003fb0 0x7f14c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c000bad0 con 0x7f14c0004d40 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c000bc80 con 0x7f14c0003fb0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c000be80 con 0x7f14dc06ebe0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.905+0000 7f14db400640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14daa00640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 0x7f14c00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14d3e00640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14c0003fb0 0x7f14c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14db400640 1 -- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 msgr2=0x7f14c00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14db400640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14dc06ebe0 0x7f14c00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14db400640 1 -- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14c0003fb0 msgr2=0x7f14c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14db400640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14c0003fb0 0x7f14c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.906+0000 7f14db400640 1 -- 172.21.15.97:0/3347448630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0003520 con 0x7f14c0004d40 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.907+0000 7f14db400640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009f90 tx=0x7f14c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c219cc6acf12eddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.907+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4039070 con 0x7f14c0004d40 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.907+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d402cbd0 con 0x7f14c0004d40 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.907+0000 7f14da000640 1 -- 172.21.15.97:0/3347448630 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4009c10 con 0x7f14c0004d40 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14e1a00640 1 -- 172.21.15.97:0/3347448630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14c0004d40 msgr2=0x7f14c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14e1a00640 1 --2- 172.21.15.97:0/3347448630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009f90 tx=0x7f14c0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14db400640 1 -- 172.21.15.97:0/3347448630 reap_dead start 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14e1a00640 1 -- 172.21.15.97:0/3347448630 shutdown_connections 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14e1a00640 1 -- 172.21.15.97:0/3347448630 >> 172.21.15.97:0/3347448630 conn(0x7f14dc0b1730 msgr2=0x7f14dc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14e1a00640 1 -- 172.21.15.97:0/3347448630 shutdown_connections 2024-03-06T07:29:43.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.908+0000 7f14e1a00640 1 -- 172.21.15.97:0/3347448630 wait complete. 2024-03-06T07:29:43.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.910+0000 7f14e1a00640 1 Processor -- start 2024-03-06T07:29:43.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.910+0000 7f14e1a00640 1 -- start start 2024-03-06T07:29:43.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14e1a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14c0004d40 0x7f14dc146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14e1a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 0x7f14dc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14e1a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14dc0d0060 0x7f14dc1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14e1a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc0b30b0 con 0x7f14dc0cf110 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14e1a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc071a50 con 0x7f14dc0d0060 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14e1a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc06d370 con 0x7f14c0004d40 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14d3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 0x7f14dc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14d3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 0x7f14dc0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50410/0 (socket says 172.21.15.97:50410) 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14d3e00640 1 -- 172.21.15.97:0/1197501301 learned_addr learned my addr 172.21.15.97:0/1197501301 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14daa00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14c0004d40 0x7f14dc146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.911+0000 7f14db400640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14dc0d0060 0x7f14dc1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d3e00640 1 -- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14c0004d40 msgr2=0x7f14dc146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d3e00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14c0004d40 0x7f14dc146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d3e00640 1 -- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14dc0d0060 msgr2=0x7f14dc1574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d3e00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14dc0d0060 0x7f14dc1574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d3e00640 1 -- 172.21.15.97:0/1197501301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc0ce070 con 0x7f14dc0cf110 2024-03-06T07:29:43.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d3e00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 0x7f14dc0cfb20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f14c400e4f0 tx=0x7f14c400e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c40052e0 con 0x7f14dc0cf110 2024-03-06T07:29:43.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c4018070 con 0x7f14dc0cf110 2024-03-06T07:29:43.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14dc0ce350 con 0x7f14dc0cf110 2024-03-06T07:29:43.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c4003830 con 0x7f14dc0cf110 2024-03-06T07:29:43.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.912+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14dc0ce910 con 0x7f14dc0cf110 2024-03-06T07:29:43.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.913+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14dc04f330 con 0x7f14dc0cf110 2024-03-06T07:29:43.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.914+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14c40039d0 con 0x7f14dc0cf110 2024-03-06T07:29:43.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.915+0000 7f14d2a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14b806b4d0 0x7f14b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.915+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14c4089eb0 con 0x7f14dc0cf110 2024-03-06T07:29:43.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.915+0000 7f14daa00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14b806b4d0 0x7f14b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.916+0000 7f14daa00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14b806b4d0 0x7f14b806d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f14cc001050 tx=0x7f14cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:43.919+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14dc15f4a0 con 0x7f14dc0cf110 2024-03-06T07:29:44.147 INFO:tasks.ceph.mds.d.smithi135.stderr:2024-03-06T07:29:44.144+0000 7faf9ac00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:44.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.156+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14dc0d0440 con 0x7f14dc0cf110 2024-03-06T07:29:44.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.157+0000 7f14d2a00640 1 -- 172.21.15.97:0/1197501301 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f14c4058c90 con 0x7f14dc0cf110 2024-03-06T07:29:44.160 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:44.160 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:37.643662+0000","modified":"2024-03-06T07:29:40.715394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6835/1918561213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1918561213},{"type":"v1","addr":"172.21.15.135:6835","nonce":1918561213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":3}]} 2024-03-06T07:29:44.160 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 26 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14b806b4d0 msgr2=0x7f14b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14b806b4d0 0x7f14b806d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f14cc001050 tx=0x7f14cc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 msgr2=0x7f14dc0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 0x7f14dc0cfb20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f14c400e4f0 tx=0x7f14c400e9b0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 shutdown_connections 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14b806b4d0 0x7f14b806d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14dc0d0060 0x7f14dc1574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14dc0cf110 0x7f14dc0cfb20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.160+0000 7f14e1a00640 1 --2- 172.21.15.97:0/1197501301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14c0004d40 0x7f14dc146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.161+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 >> 172.21.15.97:0/1197501301 conn(0x7f14dc0b1730 msgr2=0x7f14dc15e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.161+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 shutdown_connections 2024-03-06T07:29:44.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.161+0000 7f14e1a00640 1 -- 172.21.15.97:0/1197501301 wait complete. 2024-03-06T07:29:44.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:37.643662+0000', 'modified': '2024-03-06T07:29:40.715394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5511}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5511': {'gid': 5511, 'name': 'd', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.135:6835/1918561213', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6834', 'nonce': 1918561213}, {'type': 'v1', 'addr': '172.21.15.135:6835', 'nonce': 1918561213}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5511, 'qdb_cluster': [5511]} 2024-03-06T07:29:44.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:44.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:29:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.798+0000 7fd7f0a00640 1 Processor -- start 2024-03-06T07:29:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.798+0000 7fd7f0a00640 1 -- start start 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7f0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15ac90 0x7fd7ec06e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7f0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd7ec15f120 0x7fd7ec0c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7f0a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ec0cc000 con 0x7fd7ec15b070 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7f0a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ec0ce360 con 0x7fd7ec15f120 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7f0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0ce640 con 0x7fd7ec15ac90 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7e1400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd7ec15b070 0x7fd7ec15ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42014/0 (socket says 172.21.15.97:42014) 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7ea800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd7ec15f120 0x7fd7ec0c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.800+0000 7fd7e1400640 1 -- 172.21.15.97:0/4172980113 learned_addr learned my addr 172.21.15.97:0/4172980113 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:44.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.801+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/4172980113 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15ac90 0x7fd7ec06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.801+0000 7fd7ea800640 1 -- 172.21.15.97:0/4172980113 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15ac90 msgr2=0x7fd7ec06e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.801+0000 7fd7ea800640 1 --2- 172.21.15.97:0/4172980113 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15ac90 0x7fd7ec06e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.801+0000 7fd7ea800640 1 -- 172.21.15.97:0/4172980113 >> v1:172.21.15.97:6789/0 conn(0x7fd7ec15b070 legacy=0x7fd7ec15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.801+0000 7fd7ea800640 1 -- 172.21.15.97:0/4172980113 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec0ce920 con 0x7fd7ec15f120 2024-03-06T07:29:44.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.801+0000 7fd7e9400640 1 -- 172.21.15.97:0/4172980113 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655214021 0 0) 0x7fd7ec0cc000 con 0x7fd7ec15b070 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.802+0000 7fd7ea800640 1 --2- 172.21.15.97:0/4172980113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15f120 0x7fd7ec0c4a10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d003067d2d353cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.802+0000 7fd7e9400640 1 -- 172.21.15.97:0/4172980113 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4039070 con 0x7fd7ec15f120 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.802+0000 7fd7e9400640 1 -- 172.21.15.97:0/4172980113 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e402cbc0 con 0x7fd7ec15f120 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.802+0000 7fd7e9400640 1 -- 172.21.15.97:0/4172980113 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4034440 con 0x7fd7ec15f120 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 -- 172.21.15.97:0/4172980113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15f120 msgr2=0x7fd7ec0c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/4172980113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15f120 0x7fd7ec0c4a10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e402be10 comp rx=0 tx=0).stop 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 -- 172.21.15.97:0/4172980113 shutdown_connections 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/4172980113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15f120 0x7fd7ec0c4a10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/4172980113 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15ac90 0x7fd7ec06e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 -- 172.21.15.97:0/4172980113 >> 172.21.15.97:0/4172980113 conn(0x7fd7ec0cc4f0 msgr2=0x7fd7ec0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 -- 172.21.15.97:0/4172980113 shutdown_connections 2024-03-06T07:29:44.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.803+0000 7fd7f0a00640 1 -- 172.21.15.97:0/4172980113 wait complete. 2024-03-06T07:29:44.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.804+0000 7fd7f0a00640 1 Processor -- start 2024-03-06T07:29:44.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.805+0000 7fd7f0a00640 1 -- start start 2024-03-06T07:29:44.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.805+0000 7fd7f0a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ec15ac90 0x7fd7ec159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.805+0000 7fd7f0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 0x7fd7ec15db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.805+0000 7fd7f0a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15f120 0x7fd7ec0b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.805+0000 7fd7f0a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec15f5a0 con 0x7fd7ec15b070 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.805+0000 7fd7f0a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0ceee0 con 0x7fd7ec15ac90 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7f0a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec15f720 con 0x7fd7ec15f120 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 0x7fd7ec15db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 0x7fd7ec15db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50438/0 (socket says 172.21.15.97:50438) 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 -- 172.21.15.97:0/3285682620 learned_addr learned my addr 172.21.15.97:0/3285682620 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ec15ac90 0x7fd7ec159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 -- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15f120 msgr2=0x7fd7ec0b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15f120 0x7fd7ec0b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 -- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ec15ac90 msgr2=0x7fd7ec159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ec15ac90 0x7fd7ec159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.806+0000 7fd7e1400640 1 -- 172.21.15.97:0/3285682620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec1592f0 con 0x7fd7ec15b070 2024-03-06T07:29:44.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.807+0000 7fd7e1400640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 0x7fd7ec15db20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400c4a0 tx=0x7fd7d400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.807+0000 7fd7e2a00640 1 -- 172.21.15.97:0/3285682620 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4017070 con 0x7fd7ec15b070 2024-03-06T07:29:44.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.807+0000 7fd7e2a00640 1 -- 172.21.15.97:0/3285682620 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d4005ce0 con 0x7fd7ec15b070 2024-03-06T07:29:44.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.807+0000 7fd7e2a00640 1 -- 172.21.15.97:0/3285682620 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40072b0 con 0x7fd7ec15b070 2024-03-06T07:29:44.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.807+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ec0b1cb0 con 0x7fd7ec15b070 2024-03-06T07:29:44.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.807+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec15d040 con 0x7fd7ec15b070 2024-03-06T07:29:44.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.809+0000 7fd7e2a00640 1 -- 172.21.15.97:0/3285682620 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7d401a070 con 0x7fd7ec15b070 2024-03-06T07:29:44.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.810+0000 7fd7e2a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd7c806b4d0 0x7fd7c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.810+0000 7fd7e2a00640 1 -- 172.21.15.97:0/3285682620 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7d4089590 con 0x7fd7ec15b070 2024-03-06T07:29:44.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.810+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd7c806b4d0 0x7fd7c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.811+0000 7fd7f3605740 1 Processor -- start 2024-03-06T07:29:44.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.811+0000 7fd7f3605740 1 -- start start 2024-03-06T07:29:44.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.811+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd7c806b4d0 0x7fd7c806d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc006fd0 tx=0x7fd7dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185e7c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185eac540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185e7cc90 0x55f185eaca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7f3605740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f185de6600 con 0x55f185ea97b0 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f185d8f930 con 0x55f185e7cc90 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f185de26a0 con 0x55f185ea9b90 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7e9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185e7c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7e9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185e7c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50454/0 (socket says 172.21.15.97:50454) 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.812+0000 7fd7e9e00640 1 -- 172.21.15.97:0/1227340726 learned_addr learned my addr 172.21.15.97:0/1227340726 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:44.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e1400640 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185eac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7ea800640 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185e7cc90 0x55f185eaca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e9e00640 1 -- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 msgr2=0x55f185eac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185eac540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e9e00640 1 -- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185e7cc90 msgr2=0x55f185eaca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185e7cc90 0x55f185eaca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e9e00640 1 -- 172.21.15.97:0/1227340726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f185afb2d0 con 0x55f185ea97b0 2024-03-06T07:29:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185e7c750 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00fa80 tx=0x7fd7dc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6692552a07e2cd99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7da600640 1 -- 172.21.15.97:0/1227340726 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc01b050 con 0x55f185ea97b0 2024-03-06T07:29:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7da600640 1 -- 172.21.15.97:0/1227340726 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc025070 con 0x55f185ea97b0 2024-03-06T07:29:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.813+0000 7fd7da600640 1 -- 172.21.15.97:0/1227340726 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc014de0 con 0x55f185ea97b0 2024-03-06T07:29:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 -- 172.21.15.97:0/1227340726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 msgr2=0x55f185e7c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185e7c750 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00fa80 tx=0x7fd7dc012000 comp rx=0 tx=0).stop 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 -- 172.21.15.97:0/1227340726 shutdown_connections 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185e7cc90 0x55f185eaca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185eac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 --2- 172.21.15.97:0/1227340726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185e7c750 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 -- 172.21.15.97:0/1227340726 >> 172.21.15.97:0/1227340726 conn(0x55f185e7e9c0 msgr2=0x55f185ec7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 -- 172.21.15.97:0/1227340726 shutdown_connections 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.814+0000 7fd7f3605740 1 -- 172.21.15.97:0/1227340726 wait complete. 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.815+0000 7fd7f3605740 1 -- start start 2024-03-06T07:29:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.815+0000 7fd7f3605740 1 Processor -- start 2024-03-06T07:29:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185eaadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185ea2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185ec7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f185e80f60 con 0x55f185ea97b0 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f185da2d90 con 0x55f185edb830 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f185e80350 con 0x55f185ea9b90 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.816+0000 7fd7f3605740 10 client.0 _ll_register_callbacks cb 0x55f185dde350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185eaadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185eaadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50464/0 (socket says 172.21.15.97:50464) 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 -- 172.21.15.97:0/3104187679 learned_addr learned my addr 172.21.15.97:0/3104187679 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 -- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 msgr2=0x55f185ea2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e1400640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185ea2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7ea800640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185ec7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185ea2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 -- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 msgr2=0x55f185ec7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185ec7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 -- 172.21.15.97:0/3104187679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f185bc56b0 con 0x55f185ea97b0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185eaadf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc014830 tx=0x7fd7dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39f6da8c136976a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.817+0000 7fd7cea00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7cea00640 1 -- 172.21.15.97:0/3104187679 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc020cb0 con 0x55f185ea97b0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7cea00640 1 -- 172.21.15.97:0/3104187679 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7dc035af0 con 0x55f185ea97b0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7dc025070 con 0x55f185ea97b0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7cea00640 1 -- 172.21.15.97:0/3104187679 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7dc020e50 con 0x55f185ea97b0 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 10 client.5649 fetch_fsmap learned FSMap version 26 2024-03-06T07:29:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55f185eabc00 con 0x55f185ea97b0 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7cea00640 1 -- 172.21.15.97:0/3104187679 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc025070 con 0x55f185ea97b0 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7cea00640 1 -- 172.21.15.97:0/3104187679 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd7dc03f080 con 0x55f185ea97b0 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 10 client.5649 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 10 client.5649 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 20 client.5649 populate_metadata read hostname 'smithi097' 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea4a30 0x55f185ea4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 4 client.5649 mds_command: new command op to 5511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.818+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 --> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f185df3b40 con 0x55f185ea4a30 2024-03-06T07:29:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.819+0000 7fd7e1400640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea4a30 0x55f185ea4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.820+0000 7fd7e1400640 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea4a30 0x55f185ea4e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6cc90d305409eb0 server_cookie=4d3cd778f5da89cd in_seq=0 out_seq=0 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.820+0000 7fd7cea00640 10 client.5649 ms_handle_connect on v2:172.21.15.135:6834/1918561213 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7cea00640 1 -- 172.21.15.97:0/3104187679 <== mds.0 v2:172.21.15.135:6834/1918561213 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f185df3b40 con 0x55f185ea4a30 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7cea00640 10 client.5649 handle_command_reply: tid=0 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 client.5649 shutdown 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 msgr2=0x55f185eaadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185eaadf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc014830 tx=0x7fd7dc004730 comp rx=0 tx=0).stop 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 shutdown_connections 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea4a30 0x55f185ea4e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185ec7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea9b90 0x55f185ea2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 --2- 172.21.15.97:0/3104187679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea97b0 0x55f185eaadf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7cea00640 0 client.5649 ms_handle_reset on v2:172.21.15.135:6834/1918561213 2024-03-06T07:29:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 >> 172.21.15.97:0/3104187679 conn(0x55f185e7cc90 msgr2=0x55f185ecaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 shutdown_connections 2024-03-06T07:29:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.822+0000 7fd7f3605740 1 -- 172.21.15.97:0/3104187679 wait complete. 2024-03-06T07:29:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.823+0000 7fd7f3605740 20 client.5649 trim_cache size 0 max 16384 2024-03-06T07:29:44.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.841+0000 7fd7f3605740 1 Processor -- start 2024-03-06T07:29:44.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.841+0000 7fd7f3605740 1 -- start start 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.841+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ea3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.841+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185ea9b90 0x55f185ed96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.841+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 0x55f185ed9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.841+0000 7fd7f3605740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec1ff1b0 con 0x55f185edb830 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec1fe5b0 con 0x55f185ea9b90 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec1febb0 con 0x55f185ea97b0 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 0x55f185ed9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 0x55f185ed9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50480/0 (socket says 172.21.15.97:50480) 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 -- 172.21.15.97:0/2266475992 learned_addr learned my addr 172.21.15.97:0/2266475992 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ea3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 -- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 msgr2=0x55f185ea3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ea3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 -- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185ea9b90 msgr2=0x55f185ed96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185ea9b90 0x55f185ed96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.842+0000 7fd7ea800640 1 -- 172.21.15.97:0/2266475992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f185ecb7b0 con 0x55f185edb830 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7ea800640 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 0x55f185ed9c00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009eb0 tx=0x7fd7e402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227f96aa42fb31d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7d9c00640 1 -- 172.21.15.97:0/2266475992 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4044070 con 0x55f185edb830 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7d9c00640 1 -- 172.21.15.97:0/2266475992 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e402cbc0 con 0x55f185edb830 2024-03-06T07:29:44.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7d9c00640 1 -- 172.21.15.97:0/2266475992 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4039040 con 0x55f185edb830 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 -- 172.21.15.97:0/2266475992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 msgr2=0x55f185ed9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 0x55f185ed9c00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009eb0 tx=0x7fd7e402e070 comp rx=0 tx=0).stop 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 -- 172.21.15.97:0/2266475992 shutdown_connections 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185edb830 0x55f185ed9c00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185ea9b90 0x55f185ed96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 --2- 172.21.15.97:0/2266475992 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ea3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.843+0000 7fd7f3605740 1 -- 172.21.15.97:0/2266475992 >> 172.21.15.97:0/2266475992 conn(0x55f185e7cc90 msgr2=0x55f185ec7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.844+0000 7fd7f3605740 1 -- 172.21.15.97:0/2266475992 shutdown_connections 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.844+0000 7fd7f3605740 1 -- 172.21.15.97:0/2266475992 wait complete. 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.844+0000 7fd7f3605740 1 -- start start 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.844+0000 7fd7f3605740 1 Processor -- start 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ee15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 0x55f185ed5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185f3fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f185eaee10 con 0x55f185ea9b90 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f185eabee0 con 0x55f185edb830 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f185eae570 con 0x55f185ea97b0 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.845+0000 7fd7f3605740 10 client.0 _ll_register_callbacks cb 0x7fd7dc00f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 0x55f185ed5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 0x55f185ed5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50496/0 (socket says 172.21.15.97:50496) 2024-03-06T07:29:44.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 -- 172.21.15.97:0/4017531760 learned_addr learned my addr 172.21.15.97:0/4017531760 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ee15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 -- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 msgr2=0x55f185ee15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ee15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 -- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 msgr2=0x55f185f3fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185f3fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 -- 172.21.15.97:0/4017531760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f185ecba20 con 0x55f185ea9b90 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7e1400640 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 0x55f185ed5c60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4057a00 tx=0x7fd7d400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1438c7af6698dda0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7cea00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7cea00640 1 -- 172.21.15.97:0/4017531760 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4086060 con 0x55f185ea9b90 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7cea00640 1 -- 172.21.15.97:0/4017531760 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7d4054370 con 0x55f185ea9b90 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.846+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7e402cbc0 con 0x55f185ea9b90 2024-03-06T07:29:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7cea00640 1 -- 172.21.15.97:0/4017531760 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7d405ff80 con 0x55f185ea9b90 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 10 client.5655 fetch_fsmap learned FSMap version 26 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55f185ea79d0 con 0x55f185ea9b90 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7cea00640 1 -- 172.21.15.97:0/4017531760 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e402cbc0 con 0x55f185ea9b90 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7cea00640 1 -- 172.21.15.97:0/4017531760 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd7d4007450 con 0x55f185ea9b90 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 10 client.5655 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 10 client.5655 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 20 client.5655 populate_metadata read hostname 'smithi097' 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea3a60 0x55f185ea3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 4 client.5655 mds_command: new command op to 5511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.847+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 --> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f185dede90 con 0x55f185ea3a60 2024-03-06T07:29:44.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.848+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea3a60 0x55f185ea3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7e9e00640 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea3a60 0x55f185ea3e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71e1e74f84af5c6 server_cookie=bca2c222d066d5cd in_seq=0 out_seq=0 2024-03-06T07:29:44.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7cea00640 10 client.5655 ms_handle_connect on v2:172.21.15.135:6834/1918561213 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7cea00640 1 -- 172.21.15.97:0/4017531760 <== mds.0 v2:172.21.15.135:6834/1918561213 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7fd7dc020e50 con 0x55f185ea3a60 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7cea00640 10 client.5655 handle_command_reply: tid=0 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 client.5655 shutdown 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 msgr2=0x55f185ed5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 0x55f185ed5c60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4057a00 tx=0x7fd7d400cf90 comp rx=0 tx=0).stop 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 shutdown_connections 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:6834/1918561213,v1:172.21.15.135:6835/1918561213] conn(0x55f185ea3a60 0x55f185ea3e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f185edb830 0x55f185f3fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f185ea9b90 0x55f185ed5c60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 --2- 172.21.15.97:0/4017531760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f185ea97b0 0x55f185ee15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7cea00640 0 client.5655 ms_handle_reset on v2:172.21.15.135:6834/1918561213 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.849+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 >> 172.21.15.97:0/4017531760 conn(0x55f185e7cc90 msgr2=0x55f185e77b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 shutdown_connections 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f3605740 1 -- 172.21.15.97:0/4017531760 wait complete. 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f3605740 20 client.5655 trim_cache size 0 max 16384 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 5511, 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 6.6418921910000002, 2024-03-06T07:29:44.856 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 26, 2024-03-06T07:29:44.857 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 46, 2024-03-06T07:29:44.857 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 46, 2024-03-06T07:29:44.857 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 12.082819419 2024-03-06T07:29:44.857 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:29:44.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd7c806b4d0 msgr2=0x7fd7c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd7c806b4d0 0x7fd7c806d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc006fd0 tx=0x7fd7dc008040 comp rx=0 tx=0).stop 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 msgr2=0x7fd7ec15db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 0x7fd7ec15db20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400c4a0 tx=0x7fd7d400c960 comp rx=0 tx=0).stop 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 shutdown_connections 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd7c806b4d0 0x7fd7c806d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd7ec15f120 0x7fd7ec0b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7ec15b070 0x7fd7ec15db20 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 --2- 172.21.15.97:0/3285682620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ec15ac90 0x7fd7ec159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.850+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 >> 172.21.15.97:0/3285682620 conn(0x7fd7ec0cc4f0 msgr2=0x7fd7ec071270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.851+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 shutdown_connections 2024-03-06T07:29:44.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:44.851+0000 7fd7f0a00640 1 -- 172.21.15.97:0/3285682620 wait complete. 2024-03-06T07:29:44.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:45.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.067+0000 7f752a800640 1 Processor -- start 2024-03-06T07:29:45.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.068+0000 7f752a800640 1 -- start start 2024-03-06T07:29:45.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.069+0000 7f752a800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7524144c00 0x7f752406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.069+0000 7f752a800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752406ee70 con 0x7f752415fe60 2024-03-06T07:29:45.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.069+0000 7f752a800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752406cef0 con 0x7f752415f9d0 2024-03-06T07:29:45.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.069+0000 7f752a800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752406d1d0 con 0x7f7524144c00 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7523e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f752415fe60 0x7f75240ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42022/0 (socket says 172.21.15.97:42022) 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7523e00640 1 -- 172.21.15.97:0/1189525837 learned_addr learned my addr 172.21.15.97:0/1189525837 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7523400640 1 --2- 172.21.15.97:0/1189525837 >> v2:172.21.15.135:3300/0 conn(0x7f7524144c00 0x7f752406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401064531 0 0) 0x7f751c001490 con 0x7f752415fe60 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7508003520 con 0x7f752415fe60 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3915454401 0 0) 0x7f7508003520 con 0x7f752415fe60 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 >> v2:172.21.15.135:3300/0 conn(0x7f7524144c00 msgr2=0x7f752406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7522a00640 1 --2- 172.21.15.97:0/1189525837 >> v2:172.21.15.135:3300/0 conn(0x7f7524144c00 0x7f752406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.070+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 >> v1:172.21.15.135:6789/0 conn(0x7f752415f9d0 legacy=0x7f75240ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752406d4b0 con 0x7f752415fe60 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f751c001bc0 con 0x7f752415fe60 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 >> v1:172.21.15.97:6789/0 conn(0x7f752415fe60 legacy=0x7f75240ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75080039b0 0x7f7508003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7508004800 0x7f7508004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75080051c0 0x7f750800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752406d1d0 con 0x7f75080051c0 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750800c060 con 0x7f7508004800 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750800c260 con 0x7f75080039b0 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7523e00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75080051c0 0x7f750800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f7523400640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75080039b0 0x7f7508003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.071+0000 7f751be00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7508004800 0x7f7508004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.072+0000 7f7523e00640 1 -- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75080039b0 msgr2=0x7f7508003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.072+0000 7f7523e00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75080039b0 0x7f7508003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.072+0000 7f7523e00640 1 -- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7508004800 msgr2=0x7f7508004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.072+0000 7f7523e00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7508004800 0x7f7508004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.072+0000 7f7523e00640 1 -- 172.21.15.97:0/1189525837 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7508003520 con 0x7f75080051c0 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f7523e00640 1 -- 172.21.15.97:0/1189525837 reap_dead start 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f7523e00640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75080051c0 0x7f750800b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f751c00aa50 tx=0x7f7508005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424545fb1ad6f621 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c038070 con 0x7f75080051c0 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751c033650 con 0x7f75080051c0 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f7522a00640 1 -- 172.21.15.97:0/1189525837 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c00a670 con 0x7f75080051c0 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f752a800640 1 -- 172.21.15.97:0/1189525837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75080051c0 msgr2=0x7f750800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.073+0000 7f752a800640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75080051c0 0x7f750800b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f751c00aa50 tx=0x7f7508005830 comp rx=0 tx=0).stop 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.074+0000 7f752a800640 1 -- 172.21.15.97:0/1189525837 shutdown_connections 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.074+0000 7f752a800640 1 --2- 172.21.15.97:0/1189525837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75080051c0 0x7f750800b9c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.074+0000 7f752a800640 1 -- 172.21.15.97:0/1189525837 >> 172.21.15.97:0/1189525837 conn(0x7f75240b1730 msgr2=0x7f75240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.074+0000 7f752a800640 1 -- 172.21.15.97:0/1189525837 shutdown_connections 2024-03-06T07:29:45.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.074+0000 7f752a800640 1 -- 172.21.15.97:0/1189525837 wait complete. 2024-03-06T07:29:45.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.075+0000 7f752a800640 1 Processor -- start 2024-03-06T07:29:45.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.076+0000 7f752a800640 1 -- start start 2024-03-06T07:29:45.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.076+0000 7f752a800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75080051c0 0x7f7524140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f752a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 0x7f752413d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f752a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75241f4030 0x7f75240ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f752a800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7524160730 con 0x7f752413ef40 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f752a800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7524160270 con 0x7f75080051c0 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f752a800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752406cb50 con 0x7f75241f4030 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 0x7f752413d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 0x7f752413d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50506/0 (socket says 172.21.15.97:50506) 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 -- 172.21.15.97:0/1604657910 learned_addr learned my addr 172.21.15.97:0/1604657910 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 -- 172.21.15.97:0/1604657910 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75241f4030 msgr2=0x7f75240ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75241f4030 0x7f75240ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 -- 172.21.15.97:0/1604657910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75080051c0 msgr2=0x7f7524140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75080051c0 0x7f7524140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 -- 172.21.15.97:0/1604657910 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75241f49a0 con 0x7f752413ef40 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.077+0000 7f751be00640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 0x7f752413d7d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f750c00e4b0 tx=0x7f750c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.078+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0052e0 con 0x7f752413ef40 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.078+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c018070 con 0x7f752413ef40 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.078+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c003830 con 0x7f752413ef40 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.078+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75241f4c80 con 0x7f752413ef40 2024-03-06T07:29:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.078+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752413cfd0 con 0x7f752413ef40 2024-03-06T07:29:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.079+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752404f330 con 0x7f752413ef40 2024-03-06T07:29:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.080+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f750c0039d0 con 0x7f752413ef40 2024-03-06T07:29:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.081+0000 7f751b400640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f750006b4d0 0x7f750006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.081+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f750c089e10 con 0x7f752413ef40 2024-03-06T07:29:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.081+0000 7f7523400640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f750006b4d0 0x7f750006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.082+0000 7f7523400640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f750006b4d0 0x7f750006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7514009890 tx=0x7f7514008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.085+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f750c08d030 con 0x7f752413ef40 2024-03-06T07:29:45.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.324+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7524072190 con 0x7f752413ef40 2024-03-06T07:29:45.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.325+0000 7f751b400640 1 -- 172.21.15.97:0/1604657910 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f750c058bf0 con 0x7f752413ef40 2024-03-06T07:29:45.327 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f750006b4d0 msgr2=0x7f750006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f750006b4d0 0x7f750006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7514009890 tx=0x7f7514008040 comp rx=0 tx=0).stop 2024-03-06T07:29:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 msgr2=0x7f752413d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 0x7f752413d7d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f750c00e4b0 tx=0x7f750c00e970 comp rx=0 tx=0).stop 2024-03-06T07:29:45.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 shutdown_connections 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f750006b4d0 0x7f750006d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f75241f4030 0x7f75240ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f752413ef40 0x7f752413d7d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 --2- 172.21.15.97:0/1604657910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75080051c0 0x7f7524140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 >> 172.21.15.97:0/1604657910 conn(0x7f75240b1730 msgr2=0x7f7524071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 shutdown_connections 2024-03-06T07:29:45.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:45.328+0000 7f752a800640 1 -- 172.21.15.97:0/1604657910 wait complete. 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:45.348 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:45.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.349 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:29:45.397 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42489sec preferred_lft 42489sec 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:29:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:29:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:45.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.400 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:45.489 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:29:45.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.493 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:29:45.544 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:29:45.544 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:45.544 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:29:45.544 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:29:45.544 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:45.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.545 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:29:45.546 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:29:45.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:29:45.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:29:45.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:29:45.546 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:29:45.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.725 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:29:45.776 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:45.776 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:29:45.777 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:45.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:29:45.869 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:45.869 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:45.923 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:45.923 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:45.923 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:45.976 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:29:45.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:46.113 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:46.113 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:46.148 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:46.148 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:46.148 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:29:46.148 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:29:46.148 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:29:46.150 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:29:46.150 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:46.150 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:46.150 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:46.150 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:46.155 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:46.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:46.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:46.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:46.238 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:46.304 INFO:teuthology.orchestra.run.smithi097.stdout:client4879 2024-03-06T07:29:46.305 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:46.305 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:46.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:29:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.610+0000 7f6b6d600640 1 Processor -- start 2024-03-06T07:29:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.610+0000 7f6b6d600640 1 -- start start 2024-03-06T07:29:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b6d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680b3c30 0x7f6b6815c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b6d600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b680cd660 0x7f6b680cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b6d600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6815fa50 con 0x7f6b680cc760 2024-03-06T07:29:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b6d600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680cdff0 con 0x7f6b680cd660 2024-03-06T07:29:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b6d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680cea60 con 0x7f6b680b3c30 2024-03-06T07:29:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b67400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b680cd660 0x7f6b680cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b66000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6b680cc760 0x7f6b680ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42034/0 (socket says 172.21.15.97:42034) 2024-03-06T07:29:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.612+0000 7f6b66000640 1 -- 172.21.15.97:0/1761776025 learned_addr learned my addr 172.21.15.97:0/1761776025 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.613+0000 7f6b66a00640 1 --2- 172.21.15.97:0/1761776025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680b3c30 0x7f6b6815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.613+0000 7f6b67400640 1 -- 172.21.15.97:0/1761776025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680b3c30 msgr2=0x7f6b6815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.613+0000 7f6b67400640 1 --2- 172.21.15.97:0/1761776025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680b3c30 0x7f6b6815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.613+0000 7f6b67400640 1 -- 172.21.15.97:0/1761776025 >> v1:172.21.15.97:6789/0 conn(0x7f6b680cc760 legacy=0x7f6b680ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.613+0000 7f6b67400640 1 -- 172.21.15.97:0/1761776025 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6815ca10 con 0x7f6b680cd660 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.614+0000 7f6b67400640 1 --2- 172.21.15.97:0/1761776025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cd660 0x7f6b680cdab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009d90 tx=0x7f6b6002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50bf3d3f33dc4c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.614+0000 7f6b65600640 1 -- 172.21.15.97:0/1761776025 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60039070 con 0x7f6b680cd660 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.614+0000 7f6b65600640 1 -- 172.21.15.97:0/1761776025 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6002cc60 con 0x7f6b680cd660 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.614+0000 7f6b65600640 1 -- 172.21.15.97:0/1761776025 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60034660 con 0x7f6b680cd660 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 -- 172.21.15.97:0/1761776025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cd660 msgr2=0x7f6b680cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 --2- 172.21.15.97:0/1761776025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cd660 0x7f6b680cdab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009d90 tx=0x7f6b6002beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 -- 172.21.15.97:0/1761776025 shutdown_connections 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 --2- 172.21.15.97:0/1761776025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cd660 0x7f6b680cdab0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 --2- 172.21.15.97:0/1761776025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680b3c30 0x7f6b6815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 -- 172.21.15.97:0/1761776025 >> 172.21.15.97:0/1761776025 conn(0x7f6b680b2890 msgr2=0x7f6b680ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 -- 172.21.15.97:0/1761776025 shutdown_connections 2024-03-06T07:29:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.615+0000 7f6b6d600640 1 -- 172.21.15.97:0/1761776025 wait complete. 2024-03-06T07:29:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.617+0000 7f6b6d600640 1 Processor -- start 2024-03-06T07:29:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.617+0000 7f6b6d600640 1 -- start start 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b6d600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b680b3c30 0x7f6b6813f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b6d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 0x7f6b68155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b6d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680cd660 0x7f6b68157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b6d600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680b4010 con 0x7f6b680cc760 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b6d600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6815cfd0 con 0x7f6b680b3c30 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b6d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6815d2b0 con 0x7f6b680cd660 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b66000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 0x7f6b68155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b66000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 0x7f6b68155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50512/0 (socket says 172.21.15.97:50512) 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b66000640 1 -- 172.21.15.97:0/3112899301 learned_addr learned my addr 172.21.15.97:0/3112899301 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b67400640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680cd660 0x7f6b68157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.618+0000 7f6b66a00640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b680b3c30 0x7f6b6813f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b66000640 1 -- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680cd660 msgr2=0x7f6b68157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b66000640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680cd660 0x7f6b68157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b66000640 1 -- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b680b3c30 msgr2=0x7f6b6813f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b66000640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b680b3c30 0x7f6b6813f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b66000640 1 -- 172.21.15.97:0/3112899301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b681582c0 con 0x7f6b680cc760 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b67400640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680cd660 0x7f6b68157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:46.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b66000640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 0x7f6b68155a10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000c400 tx=0x7f6b5000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50017070 con 0x7f6b680cc760 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b50005ce0 con 0x7f6b680cc760 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b500072f0 con 0x7f6b680cc760 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b681585a0 con 0x7f6b680cc760 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.619+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6813faa0 con 0x7f6b680cc760 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.620+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6804f330 con 0x7f6b680cc760 2024-03-06T07:29:46.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.621+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b5001a070 con 0x7f6b680cc760 2024-03-06T07:29:46.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.622+0000 7f6b5ea00640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b4406b4d0 0x7f6b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.622+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b50089690 con 0x7f6b680cc760 2024-03-06T07:29:46.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.623+0000 7f6b66a00640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b4406b4d0 0x7f6b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.623+0000 7f6b66a00640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b4406b4d0 0x7f6b4406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6b58006fd0 tx=0x7f6b58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.627+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b50058470 con 0x7f6b680cc760 2024-03-06T07:29:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.823+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f6b6806c350 con 0x7f6b680cc760 2024-03-06T07:29:46.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.830+0000 7f6b5ea00640 1 -- 172.21.15.97:0/3112899301 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6b50057d70 con 0x7f6b680cc760 2024-03-06T07:29:46.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b4406b4d0 msgr2=0x7f6b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b4406b4d0 0x7f6b4406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6b58006fd0 tx=0x7f6b58008040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 msgr2=0x7f6b68155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 0x7f6b68155a10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000c400 tx=0x7f6b5000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 shutdown_connections 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b4406b4d0 0x7f6b4406d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b680cd660 0x7f6b68157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b680cc760 0x7f6b68155a10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 --2- 172.21.15.97:0/3112899301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b680b3c30 0x7f6b6813f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 >> 172.21.15.97:0/3112899301 conn(0x7f6b680b2890 msgr2=0x7f6b680b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 shutdown_connections 2024-03-06T07:29:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:46.833+0000 7f6b6d600640 1 -- 172.21.15.97:0/3112899301 wait complete. 2024-03-06T07:29:56.855 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:56.855 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> import uuid 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> n = 51 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = False 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> if 0 > 0: 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = True 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> dpath = os.path.dirname(path) 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> fnameprefix = os.path.basename(path) 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:29:56.914 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> try: 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(n): 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> f.write(f"{i}") 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> f.flush() 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(f.fileno()) 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> os.unlink(fpath) 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> if create_hard_links: 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> for j in range(0): 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:29:56.915 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:29:56.916 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:29:56.916 DEBUG:teuthology.orchestra.run.smithi097:> finally: 2024-03-06T07:29:56.916 DEBUG:teuthology.orchestra.run.smithi097:> os.close(dirfd) 2024-03-06T07:29:56.916 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi097.stderr:Traceback (most recent call last): 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi097.stderr: File "", line 33, in 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi097.stderr:OSError: [Errno 28] No space left on device 2024-03-06T07:29:57.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:29:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:57.153 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:57.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:57.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:57.428 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:57.428 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:57.428 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:57.486 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:29:57.486 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:57.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:57.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.745+0000 7fb4eee00640 1 Processor -- start 2024-03-06T07:29:57.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.745+0000 7fb4eee00640 1 -- start start 2024-03-06T07:29:57.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.746+0000 7fb4eee00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb4e815f9d0 0x7fb4e815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.746+0000 7fb4eee00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb4e8070b10 0x7fb4e8070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.746+0000 7fb4eee00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e806cef0 con 0x7fb4e8144c00 2024-03-06T07:29:57.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.746+0000 7fb4eee00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e806d1d0 con 0x7fb4e815f9d0 2024-03-06T07:29:57.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4eee00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e806d4b0 con 0x7fb4e8070b10 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4e7e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb4e8144c00 0x7fb4e806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35020/0 (socket says 172.21.15.97:35020) 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4e7e00640 1 -- 172.21.15.97:0/1046942331 learned_addr learned my addr 172.21.15.97:0/1046942331 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4e7400640 1 --2- 172.21.15.97:0/1046942331 >> v2:172.21.15.97:3300/0 conn(0x7fb4e815f9d0 0x7fb4e815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4ece00640 1 --2- 172.21.15.97:0/1046942331 >> v2:172.21.15.135:3300/0 conn(0x7fb4e8070b10 0x7fb4e8070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4e6a00640 1 -- 172.21.15.97:0/1046942331 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768970297 0 0) 0x7fb4e806cef0 con 0x7fb4e8144c00 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.747+0000 7fb4e6a00640 1 -- 172.21.15.97:0/1046942331 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4cc003520 con 0x7fb4e8144c00 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.748+0000 7fb4e7400640 1 -- 172.21.15.97:0/1046942331 >> v2:172.21.15.135:3300/0 conn(0x7fb4e8070b10 msgr2=0x7fb4e8070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.748+0000 7fb4e7400640 1 --2- 172.21.15.97:0/1046942331 >> v2:172.21.15.135:3300/0 conn(0x7fb4e8070b10 0x7fb4e8070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.748+0000 7fb4e7400640 1 -- 172.21.15.97:0/1046942331 >> v1:172.21.15.97:6789/0 conn(0x7fb4e8144c00 legacy=0x7fb4e806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.748+0000 7fb4e7400640 1 -- 172.21.15.97:0/1046942331 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e806d790 con 0x7fb4e815f9d0 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.749+0000 7fb4e7400640 1 --2- 172.21.15.97:0/1046942331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e815f9d0 0x7fb4e815fe60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb4e806e500 tx=0x7fb4d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be47bcf6b671d4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.749+0000 7fb4e6a00640 1 -- 172.21.15.97:0/1046942331 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0035020 con 0x7fb4e815f9d0 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.749+0000 7fb4e6a00640 1 -- 172.21.15.97:0/1046942331 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0031830 con 0x7fb4e815f9d0 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.749+0000 7fb4e6a00640 1 -- 172.21.15.97:0/1046942331 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0031c00 con 0x7fb4e815f9d0 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 -- 172.21.15.97:0/1046942331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e815f9d0 msgr2=0x7fb4e815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 --2- 172.21.15.97:0/1046942331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e815f9d0 0x7fb4e815fe60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb4e806e500 tx=0x7fb4d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 -- 172.21.15.97:0/1046942331 shutdown_connections 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 --2- 172.21.15.97:0/1046942331 >> v2:172.21.15.135:3300/0 conn(0x7fb4e8070b10 0x7fb4e8070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 --2- 172.21.15.97:0/1046942331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e815f9d0 0x7fb4e815fe60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 -- 172.21.15.97:0/1046942331 >> 172.21.15.97:0/1046942331 conn(0x7fb4e80b1730 msgr2=0x7fb4e80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 -- 172.21.15.97:0/1046942331 shutdown_connections 2024-03-06T07:29:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.750+0000 7fb4eee00640 1 -- 172.21.15.97:0/1046942331 wait complete. 2024-03-06T07:29:57.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.751+0000 7fb4eee00640 1 Processor -- start 2024-03-06T07:29:57.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.752+0000 7fb4eee00640 1 -- start start 2024-03-06T07:29:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.752+0000 7fb4eee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4e8070b10 0x7fb4e813afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4eee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 0x7fb4e813cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4eee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e815f9d0 0x7fb4e80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4eee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e806ccd0 con 0x7fb4e8144c00 2024-03-06T07:29:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4eee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e806cb50 con 0x7fb4e815f9d0 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4eee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e806df90 con 0x7fb4e8070b10 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 0x7fb4e813cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 0x7fb4e813cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41038/0 (socket says 172.21.15.97:41038) 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 -- 172.21.15.97:0/3247005886 learned_addr learned my addr 172.21.15.97:0/3247005886 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 -- 172.21.15.97:0/3247005886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4e8070b10 msgr2=0x7fb4e813afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4e8070b10 0x7fb4e813afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 -- 172.21.15.97:0/3247005886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e815f9d0 msgr2=0x7fb4e80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e815f9d0 0x7fb4e80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.753+0000 7fb4e7400640 1 -- 172.21.15.97:0/3247005886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e813d950 con 0x7fb4e8144c00 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.754+0000 7fb4e7400640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 0x7fb4e813cfd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009890 tx=0x7fb4d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.754+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0040070 con 0x7fb4e8144c00 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.754+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d003e070 con 0x7fb4e8144c00 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.754+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0035040 con 0x7fb4e8144c00 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.754+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e81f7900 con 0x7fb4e8144c00 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.754+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e81f7ec0 con 0x7fb4e8144c00 2024-03-06T07:29:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.755+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e804f330 con 0x7fb4e8144c00 2024-03-06T07:29:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.756+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4d0038020 con 0x7fb4e8144c00 2024-03-06T07:29:57.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.757+0000 7fb4dfe00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4c406b4d0 0x7fb4c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.757+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4d00abb50 con 0x7fb4e8144c00 2024-03-06T07:29:57.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.757+0000 7fb4e7e00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4c406b4d0 0x7fb4c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.758+0000 7fb4e7e00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4c406b4d0 0x7fb4c406d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80012e0 tx=0x7fb4d8001650 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.762+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4d0031800 con 0x7fb4e8144c00 2024-03-06T07:29:57.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.963+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4e8072890 con 0x7fb4e8144c00 2024-03-06T07:29:57.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.965+0000 7fb4dfe00640 1 -- 172.21.15.97:0/3247005886 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16601 (secure 0 0 0) 0x7fb4d007a9d0 con 0x7fb4e8144c00 2024-03-06T07:29:57.968 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:29:57.968 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":46,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:39.204502+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:57.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4c406b4d0 msgr2=0x7fb4c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4c406b4d0 0x7fb4c406d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80012e0 tx=0x7fb4d8001650 comp rx=0 tx=0).stop 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 msgr2=0x7fb4e813cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 0x7fb4e813cfd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009890 tx=0x7fb4d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 shutdown_connections 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb4c406b4d0 0x7fb4c406d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e815f9d0 0x7fb4e80b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4e8144c00 0x7fb4e813cfd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 --2- 172.21.15.97:0/3247005886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb4e8070b10 0x7fb4e813afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.968+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 >> 172.21.15.97:0/3247005886 conn(0x7fb4e80b1730 msgr2=0x7fb4e80ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.969+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 shutdown_connections 2024-03-06T07:29:57.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:57.969+0000 7fb4eee00640 1 -- 172.21.15.97:0/3247005886 wait complete. 2024-03-06T07:29:57.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:35.365606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:36.650975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:57.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.203+0000 7f9387200640 1 Processor -- start 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.203+0000 7f9387200640 1 -- start start 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.204+0000 7f9387200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9388166fc0 0x7f93881693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.204+0000 7f9387200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880cf2f0 con 0x7f93880b4b90 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.204+0000 7f9387200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880cf5d0 con 0x7f93880d06d0 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.204+0000 7f9387200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93880cf8b0 con 0x7f9388166fc0 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.204+0000 7f9385400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f93880b4b90 0x7f93880cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35030/0 (socket says 172.21.15.97:35030) 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.204+0000 7f9385400640 1 -- 172.21.15.97:0/738385863 learned_addr learned my addr 172.21.15.97:0/738385863 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9386800640 1 --2- 172.21.15.97:0/738385863 >> v2:172.21.15.135:3300/0 conn(0x7f9388166fc0 0x7f93881693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764332729 0 0) 0x7f93880cf2f0 con 0x7f93880b4b90 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f936c003520 con 0x7f93880b4b90 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007430936 0 0) 0x7f936c003520 con 0x7f93880b4b90 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 >> v2:172.21.15.135:3300/0 conn(0x7f9388166fc0 msgr2=0x7f93881693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 --2- 172.21.15.97:0/738385863 >> v2:172.21.15.135:3300/0 conn(0x7f9388166fc0 0x7f93881693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 >> v1:172.21.15.135:6789/0 conn(0x7f93880d06d0 legacy=0x7f93880d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880cfb90 con 0x7f93880b4b90 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9378001670 con 0x7f93880b4b90 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 >> v1:172.21.15.97:6789/0 conn(0x7f93880b4b90 legacy=0x7f93880cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f936c0039b0 0x7f936c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f936c004800 0x7f936c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0051c0 0x7f936c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880cf8b0 con 0x7f936c0039b0 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c00c110 con 0x7f936c0051c0 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.205+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c00c310 con 0x7f936c004800 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385e00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f936c0039b0 0x7f936c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385400640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f936c004800 0x7f936c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9386800640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0051c0 0x7f936c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385e00640 1 -- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f936c004800 msgr2=0x7f936c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385e00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f936c004800 0x7f936c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385e00640 1 -- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0051c0 msgr2=0x7f936c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385e00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0051c0 0x7f936c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385e00640 1 -- 172.21.15.97:0/738385863 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c003520 con 0x7f936c0039b0 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9386800640 1 -- 172.21.15.97:0/738385863 reap_dead start 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.206+0000 7f9385400640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f936c004800 0x7f936c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.208+0000 7f9385e00640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f936c0039b0 0x7f936c003dd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9370001750 tx=0x7f936c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cf65dbf1140f13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.208+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370001370 con 0x7f936c0039b0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.208+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93700052d0 con 0x7f936c0039b0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.208+0000 7f9384a00640 1 -- 172.21.15.97:0/738385863 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370003830 con 0x7f936c0039b0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.208+0000 7f9387200640 1 -- 172.21.15.97:0/738385863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f936c0039b0 msgr2=0x7f936c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.208+0000 7f9387200640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f936c0039b0 0x7f936c003dd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9370001750 tx=0x7f936c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.209+0000 7f9387200640 1 -- 172.21.15.97:0/738385863 shutdown_connections 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.209+0000 7f9387200640 1 --2- 172.21.15.97:0/738385863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f936c0039b0 0x7f936c003dd0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.209+0000 7f9387200640 1 -- 172.21.15.97:0/738385863 >> 172.21.15.97:0/738385863 conn(0x7f938809d070 msgr2=0x7f93880d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.209+0000 7f9387200640 1 -- 172.21.15.97:0/738385863 shutdown_connections 2024-03-06T07:29:58.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.209+0000 7f9387200640 1 -- 172.21.15.97:0/738385863 wait complete. 2024-03-06T07:29:58.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.210+0000 7f9387200640 1 Processor -- start 2024-03-06T07:29:58.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.210+0000 7f9387200640 1 -- start start 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.211+0000 7f9387200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0039b0 0x7f9388153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.211+0000 7f9387200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 0x7f93880b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.211+0000 7f9387200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93880b7200 0x7f93880b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.211+0000 7f9387200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880d0c30 con 0x7f93880cd760 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.211+0000 7f9387200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880d0ab0 con 0x7f936c0039b0 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.211+0000 7f9387200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93880d0db0 con 0x7f93880b7200 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 0x7f93880b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 0x7f93880b6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41060/0 (socket says 172.21.15.97:41060) 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 -- 172.21.15.97:0/2921263770 learned_addr learned my addr 172.21.15.97:0/2921263770 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9386800640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93880b7200 0x7f93880b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385e00640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0039b0 0x7f9388153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 -- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93880b7200 msgr2=0x7f93880b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93880b7200 0x7f93880b76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 -- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0039b0 msgr2=0x7f9388153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0039b0 0x7f9388153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 -- 172.21.15.97:0/2921263770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388153840 con 0x7f93880cd760 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385e00640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0039b0 0x7f9388153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.212+0000 7f9385400640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 0x7f93880b6cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f937800d780 tx=0x7f937800dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.213+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378019070 con 0x7f93880cd760 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.213+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93780074e0 con 0x7f93880cd760 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.213+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378004a00 con 0x7f93880cd760 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.213+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9388162440 con 0x7f93880cd760 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.213+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93880cb2e0 con 0x7f93880cd760 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.214+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938804f330 con 0x7f93880cd760 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.215+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9378004ba0 con 0x7f93880cd760 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.216+0000 7f937e000640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f936406b500 0x7f936406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.216+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9378089af0 con 0x7f93880cd760 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.216+0000 7f9385e00640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f936406b500 0x7f936406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.217+0000 7f9385e00640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f936406b500 0x7f936406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9370033000 tx=0x7f9370009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.220+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9388153840 con 0x7f93880cd760 2024-03-06T07:29:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.416+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f93880b3540 con 0x7f93880cd760 2024-03-06T07:29:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.417+0000 7f937e000640 1 -- 172.21.15.97:0/2921263770 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9378058960 con 0x7f93880cd760 2024-03-06T07:29:58.420 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:29:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f936406b500 msgr2=0x7f936406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f936406b500 0x7f936406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9370033000 tx=0x7f9370009030 comp rx=0 tx=0).stop 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 msgr2=0x7f93880b6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 0x7f93880b6cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f937800d780 tx=0x7f937800dc40 comp rx=0 tx=0).stop 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 shutdown_connections 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f936406b500 0x7f936406d9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93880b7200 0x7f93880b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93880cd760 0x7f93880b6cc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 --2- 172.21.15.97:0/2921263770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f936c0039b0 0x7f9388153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 >> 172.21.15.97:0/2921263770 conn(0x7f938809d070 msgr2=0x7f93880b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 shutdown_connections 2024-03-06T07:29:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.420+0000 7f9387200640 1 -- 172.21.15.97:0/2921263770 wait complete. 2024-03-06T07:29:58.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:58.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.652+0000 7f1653600640 1 Processor -- start 2024-03-06T07:29:58.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.652+0000 7f1653600640 1 -- start start 2024-03-06T07:29:58.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f1653600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f164c15e000 0x7f164c06f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f1653600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f164c15e3e0 0x7f164c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f1653600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164c0d1260 con 0x7f164c06f7e0 2024-03-06T07:29:58.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f1653600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164c0d1020 con 0x7f164c15e3e0 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f1653600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164c06c0d0 con 0x7f164c15e000 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f164be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f164c15e3e0 0x7f164c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f164be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f164c15e3e0 0x7f164c0d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41068/0 (socket says 172.21.15.97:41068) 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.654+0000 7f164be00640 1 -- 172.21.15.97:0/1994702324 learned_addr learned my addr 172.21.15.97:0/1994702324 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.655+0000 7f164be00640 1 -- 172.21.15.97:0/1994702324 >> v2:172.21.15.135:3300/0 conn(0x7f164c15e000 msgr2=0x7f164c06f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.655+0000 7f164be00640 1 --2- 172.21.15.97:0/1994702324 >> v2:172.21.15.135:3300/0 conn(0x7f164c15e000 0x7f164c06f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.655+0000 7f164be00640 1 -- 172.21.15.97:0/1994702324 >> v1:172.21.15.135:6789/0 conn(0x7f164c06f7e0 legacy=0x7f164c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.655+0000 7f164be00640 1 -- 172.21.15.97:0/1994702324 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c06c320 con 0x7f164c15e3e0 2024-03-06T07:29:58.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.656+0000 7f164be00640 1 --2- 172.21.15.97:0/1994702324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c15e3e0 0x7f164c0d0ae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f164c071b20 tx=0x7f163402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b021b51d296ef33d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.656+0000 7f164b400640 1 -- 172.21.15.97:0/1994702324 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634035020 con 0x7f164c15e3e0 2024-03-06T07:29:58.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.656+0000 7f164b400640 1 -- 172.21.15.97:0/1994702324 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1634031870 con 0x7f164c15e3e0 2024-03-06T07:29:58.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.656+0000 7f164b400640 1 -- 172.21.15.97:0/1994702324 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634031c40 con 0x7f164c15e3e0 2024-03-06T07:29:58.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 -- 172.21.15.97:0/1994702324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c15e3e0 msgr2=0x7f164c0d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 --2- 172.21.15.97:0/1994702324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c15e3e0 0x7f164c0d0ae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f164c071b20 tx=0x7f163402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:58.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 -- 172.21.15.97:0/1994702324 shutdown_connections 2024-03-06T07:29:58.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 --2- 172.21.15.97:0/1994702324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c15e3e0 0x7f164c0d0ae0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 --2- 172.21.15.97:0/1994702324 >> v2:172.21.15.135:3300/0 conn(0x7f164c15e000 0x7f164c06f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 -- 172.21.15.97:0/1994702324 >> 172.21.15.97:0/1994702324 conn(0x7f164c0b3800 msgr2=0x7f164c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 -- 172.21.15.97:0/1994702324 shutdown_connections 2024-03-06T07:29:58.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.657+0000 7f1653600640 1 -- 172.21.15.97:0/1994702324 wait complete. 2024-03-06T07:29:58.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.659+0000 7f1653600640 1 Processor -- start 2024-03-06T07:29:58.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.659+0000 7f1653600640 1 -- start start 2024-03-06T07:29:58.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.660+0000 7f1653600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 0x7f164c1f8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1653600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164c15e000 0x7f164c14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1653600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c15e3e0 0x7f164c09bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1653600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c06fdf0 con 0x7f164c06f7e0 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1653600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c06fc70 con 0x7f164c15e3e0 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1653600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164c06ff70 con 0x7f164c15e000 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 0x7f164c1f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 0x7f164c1f8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41082/0 (socket says 172.21.15.97:41082) 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 -- 172.21.15.97:0/3412468169 learned_addr learned my addr 172.21.15.97:0/3412468169 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f164be00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164c15e000 0x7f164c14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 -- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164c15e000 msgr2=0x7f164c14d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164c15e000 0x7f164c14d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 -- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c15e3e0 msgr2=0x7f164c09bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1651600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c15e3e0 0x7f164c09bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c15e3e0 0x7f164c09bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1650c00640 1 -- 172.21.15.97:0/3412468169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c09d860 con 0x7f164c06f7e0 2024-03-06T07:29:58.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.661+0000 7f1651600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c15e3e0 0x7f164c09bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.662+0000 7f1650c00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 0x7f164c1f8450 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f163c00ba50 tx=0x7f163c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.662+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c019070 con 0x7f164c06f7e0 2024-03-06T07:29:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.662+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c00cc00 con 0x7f164c06f7e0 2024-03-06T07:29:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.662+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c09db40 con 0x7f164c06f7e0 2024-03-06T07:29:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.662+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c014440 con 0x7f164c06f7e0 2024-03-06T07:29:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.662+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c09e100 con 0x7f164c06f7e0 2024-03-06T07:29:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.663+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164c04f330 con 0x7f164c06f7e0 2024-03-06T07:29:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.664+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f163c01c070 con 0x7f164c06f7e0 2024-03-06T07:29:58.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.665+0000 7f1648c00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b4d0 0x7f162806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.665+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f163c08a080 con 0x7f164c06f7e0 2024-03-06T07:29:58.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.666+0000 7f164be00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b4d0 0x7f162806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.666+0000 7f164be00640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b4d0 0x7f162806d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1634036000 tx=0x7f1634008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.670+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f163c058e60 con 0x7f164c06f7e0 2024-03-06T07:29:58.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.868+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f164c0affa0 con 0x7f164c06f7e0 2024-03-06T07:29:58.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.869+0000 7f1648c00640 1 -- 172.21.15.97:0/3412468169 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f163c058760 con 0x7f164c06f7e0 2024-03-06T07:29:58.872 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:29:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b4d0 msgr2=0x7f162806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b4d0 0x7f162806d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1634036000 tx=0x7f1634008030 comp rx=0 tx=0).stop 2024-03-06T07:29:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 msgr2=0x7f164c1f8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 0x7f164c1f8450 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f163c00ba50 tx=0x7f163c00bf10 comp rx=0 tx=0).stop 2024-03-06T07:29:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 shutdown_connections 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f162806b4d0 0x7f162806d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c15e3e0 0x7f164c09bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f164c15e000 0x7f164c14d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 --2- 172.21.15.97:0/3412468169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164c06f7e0 0x7f164c1f8450 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 >> 172.21.15.97:0/3412468169 conn(0x7f164c0b3800 msgr2=0x7f164c06ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 shutdown_connections 2024-03-06T07:29:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:58.872+0000 7f1653600640 1 -- 172.21.15.97:0/3412468169 wait complete. 2024-03-06T07:29:58.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.104+0000 7f63ff000640 1 Processor -- start 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.104+0000 7f63ff000640 1 -- start start 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63ff000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f63f80b3df0 0x7f63f80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63ff000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63f80b3650 0x7f63f80b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63ff000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80cda90 0x7f63f806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63ff000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f8072310 con 0x7f63f80b3650 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63ff000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f80b4710 con 0x7f63f80b3df0 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63ff000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80719e0 con 0x7f63f80cda90 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63f7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63f80b3650 0x7f63f80b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63f7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63f80b3650 0x7f63f80b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41086/0 (socket says 172.21.15.97:41086) 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63f7400640 1 -- 172.21.15.97:0/778181736 learned_addr learned my addr 172.21.15.97:0/778181736 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63fd000640 1 --2- 172.21.15.97:0/778181736 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80cda90 0x7f63f806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.106+0000 7f63f7e00640 1 --2- 172.21.15.97:0/778181736 >> v2:172.21.15.135:3300/0 conn(0x7f63f80b3df0 0x7f63f80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.107+0000 7f63f7400640 1 -- 172.21.15.97:0/778181736 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80cda90 msgr2=0x7f63f806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.107+0000 7f63f7400640 1 --2- 172.21.15.97:0/778181736 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80cda90 0x7f63f806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.107+0000 7f63f7400640 1 -- 172.21.15.97:0/778181736 >> v2:172.21.15.135:3300/0 conn(0x7f63f80b3df0 msgr2=0x7f63f80727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.107+0000 7f63f7400640 1 --2- 172.21.15.97:0/778181736 >> v2:172.21.15.135:3300/0 conn(0x7f63f80b3df0 0x7f63f80727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.107+0000 7f63f7400640 1 -- 172.21.15.97:0/778181736 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8071cc0 con 0x7f63f80b3650 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.108+0000 7f63f7400640 1 --2- 172.21.15.97:0/778181736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80b3650 0x7f63f80b41d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f63f80b3dc0 tx=0x7f63e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593f28261dceb0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.108+0000 7f63f6a00640 1 -- 172.21.15.97:0/778181736 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e4035020 con 0x7f63f80b3650 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.108+0000 7f63f6a00640 1 -- 172.21.15.97:0/778181736 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e4031870 con 0x7f63f80b3650 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.108+0000 7f63f6a00640 1 -- 172.21.15.97:0/778181736 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e4031c40 con 0x7f63f80b3650 2024-03-06T07:29:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 -- 172.21.15.97:0/778181736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80b3650 msgr2=0x7f63f80b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 --2- 172.21.15.97:0/778181736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80b3650 0x7f63f80b41d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f63f80b3dc0 tx=0x7f63e402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 -- 172.21.15.97:0/778181736 shutdown_connections 2024-03-06T07:29:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 --2- 172.21.15.97:0/778181736 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80cda90 0x7f63f806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 --2- 172.21.15.97:0/778181736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80b3650 0x7f63f80b41d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 --2- 172.21.15.97:0/778181736 >> v2:172.21.15.135:3300/0 conn(0x7f63f80b3df0 0x7f63f80727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 -- 172.21.15.97:0/778181736 >> 172.21.15.97:0/778181736 conn(0x7f63f80b1940 msgr2=0x7f63f80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 -- 172.21.15.97:0/778181736 shutdown_connections 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.109+0000 7f63ff000640 1 -- 172.21.15.97:0/778181736 wait complete. 2024-03-06T07:29:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.111+0000 7f63ff000640 1 Processor -- start 2024-03-06T07:29:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.111+0000 7f63ff000640 1 -- start start 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63ff000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80b3650 0x7f63f81f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63ff000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63f80b3df0 0x7f63f81f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63ff000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 0x7f63f81f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63ff000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f815b240 con 0x7f63f80cda90 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63ff000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b3b30 con 0x7f63f80b3df0 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63ff000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f815b3c0 con 0x7f63f80b3650 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63fd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 0x7f63f81f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63fd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 0x7f63f81f3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41100/0 (socket says 172.21.15.97:41100) 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.112+0000 7f63fd000640 1 -- 172.21.15.97:0/1719097314 learned_addr learned my addr 172.21.15.97:0/1719097314 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63f7400640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63f80b3df0 0x7f63f81f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63f7e00640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80b3650 0x7f63f81f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63fd000640 1 -- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80b3650 msgr2=0x7f63f81f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63fd000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80b3650 0x7f63f81f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63fd000640 1 -- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63f80b3df0 msgr2=0x7f63f81f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63fd000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63f80b3df0 0x7f63f81f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63fd000640 1 -- 172.21.15.97:0/1719097314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8155b30 con 0x7f63f80cda90 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63f7e00640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80b3650 0x7f63f81f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63fd000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 0x7f63f81f3b90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f63f0007e30 tx=0x7f63f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.113+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0014620 con 0x7f63f80cda90 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.114+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0019070 con 0x7f63f80cda90 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.114+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0014a00 con 0x7f63f80cda90 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.114+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f80c94a0 con 0x7f63f80cda90 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.114+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f80c9a60 con 0x7f63f80cda90 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.115+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f804f330 con 0x7f63f80cda90 2024-03-06T07:29:59.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.116+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63f0014ba0 con 0x7f63f80cda90 2024-03-06T07:29:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.117+0000 7f63efe00640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63d806b4d0 0x7f63d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.117+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63f008aa60 con 0x7f63f80cda90 2024-03-06T07:29:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.117+0000 7f63f7e00640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63d806b4d0 0x7f63d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.118+0000 7f63f7e00640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63d806b4d0 0x7f63d806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f63e8006fd0 tx=0x7f63e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.121+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63f0059970 con 0x7f63f80cda90 2024-03-06T07:29:59.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.317+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63f80aece0 con 0x7f63f80cda90 2024-03-06T07:29:59.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.318+0000 7f63efe00640 1 -- 172.21.15.97:0/1719097314 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63f0059270 con 0x7f63f80cda90 2024-03-06T07:29:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:29:59.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63d806b4d0 msgr2=0x7f63d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63d806b4d0 0x7f63d806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f63e8006fd0 tx=0x7f63e8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 msgr2=0x7f63f81f3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 0x7f63f81f3b90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f63f0007e30 tx=0x7f63f000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 shutdown_connections 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63d806b4d0 0x7f63d806d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63f80cda90 0x7f63f81f3b90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63f80b3df0 0x7f63f81f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 --2- 172.21.15.97:0/1719097314 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63f80b3650 0x7f63f81f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 >> 172.21.15.97:0/1719097314 conn(0x7f63f80b1940 msgr2=0x7f63f8072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 shutdown_connections 2024-03-06T07:29:59.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.321+0000 7f63ff000640 1 -- 172.21.15.97:0/1719097314 wait complete. 2024-03-06T07:29:59.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.549+0000 7f099b400640 1 Processor -- start 2024-03-06T07:29:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.549+0000 7f099b400640 1 -- start start 2024-03-06T07:29:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f099b400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f099406afc0 0x7f099406b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f099b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f09940cacb0 0x7f09940cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f099b400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099406b8e0 con 0x7f09940cc5c0 2024-03-06T07:29:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f099b400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09940cd320 con 0x7f09940cacb0 2024-03-06T07:29:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f099b400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09940cca80 con 0x7f099406afc0 2024-03-06T07:29:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f0999400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f09940cc5c0 0x7f0994161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35040/0 (socket says 172.21.15.97:35040) 2024-03-06T07:29:59.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f0999400640 1 -- 172.21.15.97:0/2145908555 learned_addr learned my addr 172.21.15.97:0/2145908555 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:29:59.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f0993e00640 1 --2- 172.21.15.97:0/2145908555 >> v2:172.21.15.97:3300/0 conn(0x7f09940cacb0 0x7f09940cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.551+0000 7f0998a00640 1 --2- 172.21.15.97:0/2145908555 >> v2:172.21.15.135:3300/0 conn(0x7f099406afc0 0x7f099406b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.552+0000 7f0993400640 1 -- 172.21.15.97:0/2145908555 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814587641 0 0) 0x7f099406b8e0 con 0x7f09940cc5c0 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.552+0000 7f0993400640 1 -- 172.21.15.97:0/2145908555 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0978003520 con 0x7f09940cc5c0 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.552+0000 7f0993e00640 1 -- 172.21.15.97:0/2145908555 >> v2:172.21.15.135:3300/0 conn(0x7f099406afc0 msgr2=0x7f099406b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.552+0000 7f0993e00640 1 --2- 172.21.15.97:0/2145908555 >> v2:172.21.15.135:3300/0 conn(0x7f099406afc0 0x7f099406b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.552+0000 7f0993e00640 1 -- 172.21.15.97:0/2145908555 >> v1:172.21.15.97:6789/0 conn(0x7f09940cc5c0 legacy=0x7f0994161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.552+0000 7f0993e00640 1 -- 172.21.15.97:0/2145908555 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940ccd60 con 0x7f09940cacb0 2024-03-06T07:29:59.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.553+0000 7f0993e00640 1 --2- 172.21.15.97:0/2145908555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09940cacb0 0x7f09940cc080 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f09940676a0 tx=0x7f098002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7697a5dde1ef708 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.553+0000 7f0993400640 1 -- 172.21.15.97:0/2145908555 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09800315a0 con 0x7f09940cacb0 2024-03-06T07:29:59.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.553+0000 7f0993400640 1 -- 172.21.15.97:0/2145908555 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0980037070 con 0x7f09940cacb0 2024-03-06T07:29:59.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.553+0000 7f0993400640 1 -- 172.21.15.97:0/2145908555 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980031940 con 0x7f09940cacb0 2024-03-06T07:29:59.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 -- 172.21.15.97:0/2145908555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09940cacb0 msgr2=0x7f09940cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 --2- 172.21.15.97:0/2145908555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09940cacb0 0x7f09940cc080 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f09940676a0 tx=0x7f098002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 -- 172.21.15.97:0/2145908555 shutdown_connections 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 --2- 172.21.15.97:0/2145908555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09940cacb0 0x7f09940cc080 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 --2- 172.21.15.97:0/2145908555 >> v2:172.21.15.135:3300/0 conn(0x7f099406afc0 0x7f099406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 -- 172.21.15.97:0/2145908555 >> 172.21.15.97:0/2145908555 conn(0x7f09940d0b90 msgr2=0x7f09940d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 -- 172.21.15.97:0/2145908555 shutdown_connections 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.554+0000 7f099b400640 1 -- 172.21.15.97:0/2145908555 wait complete. 2024-03-06T07:29:59.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.555+0000 7f099b400640 1 Processor -- start 2024-03-06T07:29:59.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.556+0000 7f099b400640 1 -- start start 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.556+0000 7f099b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 0x7f0994138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.556+0000 7f099b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09940cacb0 0x7f09940b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f099b400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09940cc5c0 0x7f099406f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f099b400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0994144b00 con 0x7f099406afc0 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f099b400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0994144980 con 0x7f09940cc5c0 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f099b400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09940cfd30 con 0x7f09940cacb0 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 0x7f0994138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 0x7f0994138b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41112/0 (socket says 172.21.15.97:41112) 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 -- 172.21.15.97:0/3271430797 learned_addr learned my addr 172.21.15.97:0/3271430797 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 -- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09940cacb0 msgr2=0x7f09940b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0993e00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09940cacb0 0x7f09940b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0999400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09940cc5c0 0x7f099406f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09940cacb0 0x7f09940b5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 -- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09940cc5c0 msgr2=0x7f099406f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09940cc5c0 0x7f099406f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0998a00640 1 -- 172.21.15.97:0/3271430797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099415e5d0 con 0x7f099406afc0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0993e00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09940cacb0 0x7f09940b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.557+0000 7f0999400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09940cc5c0 0x7f099406f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.558+0000 7f0998a00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 0x7f0994138b10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0984009db0 tx=0x7f098400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.558+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984018020 con 0x7f099406afc0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.558+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984014480 con 0x7f099406afc0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.558+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099415e8b0 con 0x7f099406afc0 2024-03-06T07:29:59.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.558+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984014850 con 0x7f099406afc0 2024-03-06T07:29:59.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.558+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09940b6630 con 0x7f099406afc0 2024-03-06T07:29:59.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.559+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099404f330 con 0x7f099406afc0 2024-03-06T07:29:59.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.560+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0984005d10 con 0x7f099406afc0 2024-03-06T07:29:59.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.561+0000 7f0990c00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f097406b4d0 0x7f097406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.561+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f098408ae10 con 0x7f099406afc0 2024-03-06T07:29:59.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.561+0000 7f0993e00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f097406b4d0 0x7f097406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.562+0000 7f0993e00640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f097406b4d0 0x7f097406d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0980036000 tx=0x7f0980008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.565+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f098408e030 con 0x7f099406afc0 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.772+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f09940cffd0 con 0x7f099406afc0 2024-03-06T07:29:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.773+0000 7f0990c00640 1 -- 172.21.15.97:0/3271430797 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0984022070 con 0x7f099406afc0 2024-03-06T07:29:59.776 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:29:59.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f097406b4d0 msgr2=0x7f097406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f097406b4d0 0x7f097406d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0980036000 tx=0x7f0980008030 comp rx=0 tx=0).stop 2024-03-06T07:29:59.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 msgr2=0x7f0994138b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 0x7f0994138b10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0984009db0 tx=0x7f098400c490 comp rx=0 tx=0).stop 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 shutdown_connections 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f097406b4d0 0x7f097406d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09940cc5c0 0x7f099406f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09940cacb0 0x7f09940b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 --2- 172.21.15.97:0/3271430797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f099406afc0 0x7f0994138b10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.776+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 >> 172.21.15.97:0/3271430797 conn(0x7f09940d0b90 msgr2=0x7f0994142c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.777+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 shutdown_connections 2024-03-06T07:29:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:29:59.777+0000 7f099b400640 1 -- 172.21.15.97:0/3271430797 wait complete. 2024-03-06T07:29:59.795 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:29:59.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:59.972 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:29:59.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.972 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:00.008 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:30:00.008 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.045 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:00.045 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:00.046 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.089 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:00.089 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:30:00.089 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.131 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:00.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:00.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.332+0000 7efc49000640 1 Processor -- start 2024-03-06T07:30:00.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.332+0000 7efc49000640 1 -- start start 2024-03-06T07:30:00.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc49000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc440d2360 0x7efc440d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc49000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc44161960 con 0x7efc44161580 2024-03-06T07:30:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc49000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc440cf6f0 con 0x7efc440b3ec0 2024-03-06T07:30:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc49000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc440d2ce0 con 0x7efc440d2360 2024-03-06T07:30:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc43400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efc44161580 0x7efc440d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35042/0 (socket says 172.21.15.97:35042) 2024-03-06T07:30:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc43400640 1 -- 172.21.15.97:0/2100433300 learned_addr learned my addr 172.21.15.97:0/2100433300 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc3be00640 1 --2- 172.21.15.97:0/2100433300 >> v2:172.21.15.135:3300/0 conn(0x7efc440d2360 0x7efc440d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.334+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422773061 0 0) 0x7efc44161960 con 0x7efc44161580 2024-03-06T07:30:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc28003520 con 0x7efc44161580 2024-03-06T07:30:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266637885 0 0) 0x7efc28003520 con 0x7efc44161580 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 >> v2:172.21.15.135:3300/0 conn(0x7efc440d2360 msgr2=0x7efc440d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 --2- 172.21.15.97:0/2100433300 >> v2:172.21.15.135:3300/0 conn(0x7efc440d2360 0x7efc440d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 >> v1:172.21.15.135:6789/0 conn(0x7efc440b3ec0 legacy=0x7efc440b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440cf9d0 con 0x7efc44161580 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7efc34001670 con 0x7efc44161580 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 >> v1:172.21.15.97:6789/0 conn(0x7efc44161580 legacy=0x7efc440d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc28003a00 0x7efc28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc28004800 0x7efc28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc280051c0 0x7efc2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc440d2ce0 con 0x7efc28004800 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2800c0c0 con 0x7efc28003a00 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2800c2c0 con 0x7efc280051c0 2024-03-06T07:30:00.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.335+0000 7efc3be00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc28004800 0x7efc28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc43400640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc28003a00 0x7efc28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc43e00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc280051c0 0x7efc2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc3be00640 1 -- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc280051c0 msgr2=0x7efc2800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc3be00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc280051c0 0x7efc2800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc3be00640 1 -- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc28003a00 msgr2=0x7efc28003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc3be00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc28003a00 0x7efc28003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc3be00640 1 -- 172.21.15.97:0/2100433300 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28003520 con 0x7efc28004800 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.336+0000 7efc43e00640 1 -- 172.21.15.97:0/2100433300 reap_dead start 2024-03-06T07:30:00.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.337+0000 7efc3be00640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc28004800 0x7efc28004c80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009ac0 tx=0x7efc28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c936b191285fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.337+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c039070 con 0x7efc28004800 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.337+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c02c6b0 con 0x7efc28004800 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.337+0000 7efc42a00640 1 -- 172.21.15.97:0/2100433300 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c009690 con 0x7efc28004800 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 -- 172.21.15.97:0/2100433300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc28004800 msgr2=0x7efc28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc28004800 0x7efc28004c80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009ac0 tx=0x7efc28005830 comp rx=0 tx=0).stop 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 -- 172.21.15.97:0/2100433300 shutdown_connections 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 --2- 172.21.15.97:0/2100433300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc28004800 0x7efc28004c80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 -- 172.21.15.97:0/2100433300 >> 172.21.15.97:0/2100433300 conn(0x7efc440d3d80 msgr2=0x7efc440d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 -- 172.21.15.97:0/2100433300 shutdown_connections 2024-03-06T07:30:00.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.338+0000 7efc49000640 1 -- 172.21.15.97:0/2100433300 wait complete. 2024-03-06T07:30:00.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.340+0000 7efc49000640 1 Processor -- start 2024-03-06T07:30:00.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.340+0000 7efc49000640 1 -- start start 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.340+0000 7efc49000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc28004800 0x7efc4414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc49000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc441f7f60 0x7efc440c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc49000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 0x7efc440c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc49000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc440b39a0 con 0x7efc440c7410 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc49000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc440b36c0 con 0x7efc441f7f60 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc49000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc440cf3a0 con 0x7efc28004800 2024-03-06T07:30:00.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 0x7efc440c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 0x7efc440c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41138/0 (socket says 172.21.15.97:41138) 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 -- 172.21.15.97:0/1899376790 learned_addr learned my addr 172.21.15.97:0/1899376790 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43400640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc28004800 0x7efc4414dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc3be00640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc441f7f60 0x7efc440c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 -- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc28004800 msgr2=0x7efc4414dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc28004800 0x7efc4414dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 -- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc441f7f60 msgr2=0x7efc440c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc441f7f60 0x7efc440c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc43e00640 1 -- 172.21.15.97:0/1899376790 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc441f83e0 con 0x7efc440c7410 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.341+0000 7efc3be00640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc441f7f60 0x7efc440c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.342+0000 7efc43e00640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 0x7efc440c7890 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efc4408c8e0 tx=0x7efc3c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.342+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c003970 con 0x7efc440c7410 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.342+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c018070 con 0x7efc440c7410 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.342+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c003d70 con 0x7efc440c7410 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.342+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc440c9cb0 con 0x7efc440c7410 2024-03-06T07:30:00.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.342+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc440ca270 con 0x7efc440c7410 2024-03-06T07:30:00.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.343+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4404f330 con 0x7efc440c7410 2024-03-06T07:30:00.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.344+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc3c0050d0 con 0x7efc440c7410 2024-03-06T07:30:00.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.345+0000 7efc3b400640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc2006b4d0 0x7efc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.345+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc3c089ec0 con 0x7efc440c7410 2024-03-06T07:30:00.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.345+0000 7efc43400640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc2006b4d0 0x7efc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.346+0000 7efc43400640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc2006b4d0 0x7efc2006d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efc34000c00 tx=0x7efc34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.349+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc3c058d20 con 0x7efc440c7410 2024-03-06T07:30:00.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.575+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc440d3b60 con 0x7efc440c7410 2024-03-06T07:30:00.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.576+0000 7efc3b400640 1 -- 172.21.15.97:0/1899376790 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7efc3c058620 con 0x7efc440c7410 2024-03-06T07:30:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:00.578 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:37.643662+0000","modified":"2024-03-06T07:29:40.715394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6835/1918561213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1918561213},{"type":"v1","addr":"172.21.15.135:6835","nonce":1918561213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":3}]} 2024-03-06T07:30:00.578 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 26 2024-03-06T07:30:00.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc2006b4d0 msgr2=0x7efc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc2006b4d0 0x7efc2006d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efc34000c00 tx=0x7efc34009040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 msgr2=0x7efc440c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 0x7efc440c7890 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efc4408c8e0 tx=0x7efc3c00ec10 comp rx=0 tx=0).stop 2024-03-06T07:30:00.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 shutdown_connections 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efc2006b4d0 0x7efc2006d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc440c7410 0x7efc440c7890 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc441f7f60 0x7efc440c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 --2- 172.21.15.97:0/1899376790 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc28004800 0x7efc4414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 >> 172.21.15.97:0/1899376790 conn(0x7efc440d3d80 msgr2=0x7efc4406d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 shutdown_connections 2024-03-06T07:30:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.579+0000 7efc49000640 1 -- 172.21.15.97:0/1899376790 wait complete. 2024-03-06T07:30:00.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.787+0000 7f9516000640 1 Processor -- start 2024-03-06T07:30:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.788+0000 7f9516000640 1 -- start start 2024-03-06T07:30:00.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f9516000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 0x7f951015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f9516000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9510070b10 0x7f9510070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f9516000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951006cef0 con 0x7f9510144c00 2024-03-06T07:30:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f9516000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9510160370 con 0x7f9510070b10 2024-03-06T07:30:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f9516000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951006d1d0 con 0x7f951015f9d0 2024-03-06T07:30:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f950f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9510144c00 0x7f951006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35054/0 (socket says 172.21.15.97:35054) 2024-03-06T07:30:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.789+0000 7f950f400640 1 -- 172.21.15.97:0/1210473958 learned_addr learned my addr 172.21.15.97:0/1210473958 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:00.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950fe00640 1 --2- 172.21.15.97:0/1210473958 >> v2:172.21.15.135:3300/0 conn(0x7f9510070b10 0x7f9510070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950ea00640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 0x7f951015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470695606 0 0) 0x7f951006cef0 con 0x7f9510144c00 2024-03-06T07:30:00.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f4003520 con 0x7f9510144c00 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421775569 0 0) 0x7f94f4003520 con 0x7f9510144c00 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 msgr2=0x7f951015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 0x7f951015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 >> v2:172.21.15.135:3300/0 conn(0x7f9510070b10 msgr2=0x7f9510070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 --2- 172.21.15.97:0/1210473958 >> v2:172.21.15.135:3300/0 conn(0x7f9510070b10 0x7f9510070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951006d4b0 con 0x7f9510144c00 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9500001670 con 0x7f9510144c00 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 >> v1:172.21.15.97:6789/0 conn(0x7f9510144c00 legacy=0x7f951006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 0x7f94f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f94f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f4004d40 0x7f94f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951006d1d0 con 0x7f94f4003f30 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f400b9c0 con 0x7f94f4004d40 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.790+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f400bb40 con 0x7f951015f9d0 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950ea00640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f94f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950f400640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 0x7f94f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950ea00640 1 -- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 msgr2=0x7f94f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950ea00640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015f9d0 0x7f94f40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950ea00640 1 -- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f4004d40 msgr2=0x7f94f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950ea00640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f4004d40 0x7f94f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.791+0000 7f950ea00640 1 -- 172.21.15.97:0/1210473958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4003520 con 0x7f94f4003f30 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.792+0000 7f950ea00640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f94f4004800 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009ac0 tx=0x7f94f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4612c413dc6751 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.792+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc039070 con 0x7f94f4003f30 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.792+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94fc02c6b0 con 0x7f94f4003f30 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.792+0000 7f950e000640 1 -- 172.21.15.97:0/1210473958 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc009690 con 0x7f94f4003f30 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f9516000640 1 -- 172.21.15.97:0/1210473958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 msgr2=0x7f94f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f9516000640 1 --2- 172.21.15.97:0/1210473958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f94f4004800 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009ac0 tx=0x7f94f4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f950fe00640 1 -- 172.21.15.97:0/1210473958 reap_dead start 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f9516000640 1 -- 172.21.15.97:0/1210473958 shutdown_connections 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f9516000640 1 -- 172.21.15.97:0/1210473958 >> 172.21.15.97:0/1210473958 conn(0x7f95100b1730 msgr2=0x7f95100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f9516000640 1 -- 172.21.15.97:0/1210473958 shutdown_connections 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.793+0000 7f9516000640 1 -- 172.21.15.97:0/1210473958 wait complete. 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.795+0000 7f9516000640 1 Processor -- start 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.795+0000 7f9516000640 1 -- start start 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f9516000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f951013dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f9516000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015c340 0x7f951015a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f9516000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95101585f0 0x7f9510158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f9516000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951006cb50 con 0x7f94f4003f30 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f9516000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510160730 con 0x7f95101585f0 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f9516000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951006ccd0 con 0x7f951015c340 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f951013dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f951013dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41152/0 (socket says 172.21.15.97:41152) 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950f400640 1 -- 172.21.15.97:0/569642609 learned_addr learned my addr 172.21.15.97:0/569642609 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950ea00640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015c340 0x7f951015a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950fe00640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95101585f0 0x7f9510158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950f400640 1 -- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015c340 msgr2=0x7f951015a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.796+0000 7f950f400640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015c340 0x7f951015a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f950f400640 1 -- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95101585f0 msgr2=0x7f9510158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f950f400640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95101585f0 0x7f9510158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f950f400640 1 -- 172.21.15.97:0/569642609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951015cab0 con 0x7f94f4003f30 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f950fe00640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95101585f0 0x7f9510158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f950f400640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f951013dc90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f950000d810 tx=0x7f950000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500019070 con 0x7f94f4003f30 2024-03-06T07:30:00.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95000074e0 con 0x7f94f4003f30 2024-03-06T07:30:00.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500004a00 con 0x7f94f4003f30 2024-03-06T07:30:00.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951015cd90 con 0x7f94f4003f30 2024-03-06T07:30:00.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.797+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951015d350 con 0x7f94f4003f30 2024-03-06T07:30:00.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.799+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95100cd9a0 con 0x7f94f4003f30 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.800+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9500004ba0 con 0x7f94f4003f30 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.801+0000 7f9507400640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94f006b500 0x7f94f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.801+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9500089b70 con 0x7f94f4003f30 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.801+0000 7f950ea00640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94f006b500 0x7f94f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.802+0000 7f950ea00640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94f006b500 0x7f94f006d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f94fc036000 tx=0x7f94fc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:00.805+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95100cd9a0 con 0x7f94f4003f30 2024-03-06T07:30:01.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.006+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f951004f330 con 0x7f94f4003f30 2024-03-06T07:30:01.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.007+0000 7f9507400640 1 -- 172.21.15.97:0/569642609 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16601 (secure 0 0 0) 0x7f9500058a80 con 0x7f94f4003f30 2024-03-06T07:30:01.010 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:01.010 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":46,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:39.204502+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94f006b500 msgr2=0x7f94f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94f006b500 0x7f94f006d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f94fc036000 tx=0x7f94fc030000 comp rx=0 tx=0).stop 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 msgr2=0x7f951013dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f951013dc90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f950000d810 tx=0x7f950000dcd0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 shutdown_connections 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94f006b500 0x7f94f006d9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95101585f0 0x7f9510158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f951015c340 0x7f951015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.010+0000 7f9516000640 1 --2- 172.21.15.97:0/569642609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f4003f30 0x7f951013dc90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.011+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 >> 172.21.15.97:0/569642609 conn(0x7f95100b1730 msgr2=0x7f9510143ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.011+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 shutdown_connections 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.011+0000 7f9516000640 1 -- 172.21.15.97:0/569642609 wait complete. 2024-03-06T07:30:01.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:35.365606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:36.650975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:01.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:01.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.236+0000 7fa403e00640 1 Processor -- start 2024-03-06T07:30:01.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.236+0000 7fa403e00640 1 -- start start 2024-03-06T07:30:01.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.237+0000 7fa403e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa404158570 0x7fa4040ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.237+0000 7fa403e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa404158950 0x7fa4040cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.237+0000 7fa403e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40415b130 con 0x7fa4040aebf0 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa403e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4040af170 con 0x7fa404158570 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa403e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40415b3f0 con 0x7fa404158950 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa402a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa404158570 0x7fa4040ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa402a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa404158570 0x7fa4040ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41166/0 (socket says 172.21.15.97:41166) 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa402a00640 1 -- 172.21.15.97:0/201778258 learned_addr learned my addr 172.21.15.97:0/201778258 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa3fbe00640 1 --2- 172.21.15.97:0/201778258 >> v2:172.21.15.135:3300/0 conn(0x7fa404158950 0x7fa4040cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa402a00640 1 -- 172.21.15.97:0/201778258 >> v2:172.21.15.135:3300/0 conn(0x7fa404158950 msgr2=0x7fa4040cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa402a00640 1 --2- 172.21.15.97:0/201778258 >> v2:172.21.15.135:3300/0 conn(0x7fa404158950 0x7fa4040cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.238+0000 7fa402a00640 1 -- 172.21.15.97:0/201778258 >> v1:172.21.15.135:6789/0 conn(0x7fa4040aebf0 legacy=0x7fa40415e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.239+0000 7fa402a00640 1 -- 172.21.15.97:0/201778258 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40415b6d0 con 0x7fa404158570 2024-03-06T07:30:01.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.240+0000 7fa402a00640 1 --2- 172.21.15.97:0/201778258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa404158570 0x7fa4040ae6b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa40406c050 tx=0x7fa3f402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77087e2bc138c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.240+0000 7fa402000640 1 -- 172.21.15.97:0/201778258 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f402e070 con 0x7fa404158570 2024-03-06T07:30:01.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.240+0000 7fa402000640 1 -- 172.21.15.97:0/201778258 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4037070 con 0x7fa404158570 2024-03-06T07:30:01.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.240+0000 7fa402000640 1 -- 172.21.15.97:0/201778258 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4031a00 con 0x7fa404158570 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 -- 172.21.15.97:0/201778258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa404158570 msgr2=0x7fa4040ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 --2- 172.21.15.97:0/201778258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa404158570 0x7fa4040ae6b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa40406c050 tx=0x7fa3f402b880 comp rx=0 tx=0).stop 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 -- 172.21.15.97:0/201778258 shutdown_connections 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 --2- 172.21.15.97:0/201778258 >> v2:172.21.15.135:3300/0 conn(0x7fa404158950 0x7fa4040cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 --2- 172.21.15.97:0/201778258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa404158570 0x7fa4040ae6b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 -- 172.21.15.97:0/201778258 >> 172.21.15.97:0/201778258 conn(0x7fa40406afc0 msgr2=0x7fa40406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 -- 172.21.15.97:0/201778258 shutdown_connections 2024-03-06T07:30:01.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.241+0000 7fa403e00640 1 -- 172.21.15.97:0/201778258 wait complete. 2024-03-06T07:30:01.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.242+0000 7fa403e00640 1 Processor -- start 2024-03-06T07:30:01.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.243+0000 7fa403e00640 1 -- start start 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa403e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 0x7fa404156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa403e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa404158570 0x7fa404150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa403e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa404158950 0x7fa404158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa403e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40415bed0 con 0x7fa4040aebf0 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa403e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040b4950 con 0x7fa404158950 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa403e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40415c050 con 0x7fa404158570 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 0x7fa404156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 0x7fa404156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41182/0 (socket says 172.21.15.97:41182) 2024-03-06T07:30:01.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 -- 172.21.15.97:0/1820195634 learned_addr learned my addr 172.21.15.97:0/1820195634 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:01.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 -- 172.21.15.97:0/1820195634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa404158570 msgr2=0x7fa404150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa404158570 0x7fa404150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 -- 172.21.15.97:0/1820195634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa404158950 msgr2=0x7fa404158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa404158950 0x7fa404158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.244+0000 7fa402a00640 1 -- 172.21.15.97:0/1820195634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa404151230 con 0x7fa4040aebf0 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.245+0000 7fa402a00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 0x7fa404156d60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4000c00 tx=0x7fa3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.245+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4032880 con 0x7fa4040aebf0 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.245+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4032a40 con 0x7fa4040aebf0 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.245+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4032e30 con 0x7fa4040aebf0 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.245+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa404159290 con 0x7fa4040aebf0 2024-03-06T07:30:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.245+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa404159810 con 0x7fa4040aebf0 2024-03-06T07:30:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.246+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40404f330 con 0x7fa4040aebf0 2024-03-06T07:30:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.247+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f40453f0 con 0x7fa4040aebf0 2024-03-06T07:30:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.248+0000 7fa3fb400640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3e006b4d0 0x7fa3e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.248+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3f40ac3d0 con 0x7fa4040aebf0 2024-03-06T07:30:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.248+0000 7fa3fbe00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3e006b4d0 0x7fa3e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.249+0000 7fa3fbe00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3e006b4d0 0x7fa3e006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec006fd0 tx=0x7fa3ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.253+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3f4040070 con 0x7fa4040aebf0 2024-03-06T07:30:01.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.449+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa4040ced50 con 0x7fa4040aebf0 2024-03-06T07:30:01.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.450+0000 7fa3fb400640 1 -- 172.21.15.97:0/1820195634 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3f407b360 con 0x7fa4040aebf0 2024-03-06T07:30:01.452 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3e006b4d0 msgr2=0x7fa3e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3e006b4d0 0x7fa3e006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec006fd0 tx=0x7fa3ec008040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 msgr2=0x7fa404156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 0x7fa404156d60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4000c00 tx=0x7fa3f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 shutdown_connections 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3e006b4d0 0x7fa3e006d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa404158950 0x7fa404158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa404158570 0x7fa404150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 --2- 172.21.15.97:0/1820195634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4040aebf0 0x7fa404156d60 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 >> 172.21.15.97:0/1820195634 conn(0x7fa40406afc0 msgr2=0x7fa40406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 shutdown_connections 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.452+0000 7fa403e00640 1 -- 172.21.15.97:0/1820195634 wait complete. 2024-03-06T07:30:01.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:01.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.675+0000 7f7bd8600640 1 Processor -- start 2024-03-06T07:30:01.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.675+0000 7f7bd8600640 1 -- start start 2024-03-06T07:30:01.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd8600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7bd015fc10 0x7f7bd0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd8600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7bd015fff0 0x7f7bd0160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd8600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd00b46c0 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd8600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd00b49a0 con 0x7f7bd015fff0 2024-03-06T07:30:01.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd8600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd0072c90 con 0x7f7bd015fc10 2024-03-06T07:30:01.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd5200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7bd015fff0 0x7f7bd0160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.677+0000 7f7bd5200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7bd015fff0 0x7f7bd0160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41190/0 (socket says 172.21.15.97:41190) 2024-03-06T07:30:01.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.678+0000 7f7bd5200640 1 -- 172.21.15.97:0/978053709 learned_addr learned my addr 172.21.15.97:0/978053709 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:01.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.678+0000 7f7bd5c00640 1 --2- 172.21.15.97:0/978053709 >> v2:172.21.15.135:3300/0 conn(0x7f7bd015fc10 0x7f7bd0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.678+0000 7f7bd5200640 1 -- 172.21.15.97:0/978053709 >> v2:172.21.15.135:3300/0 conn(0x7f7bd015fc10 msgr2=0x7f7bd0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.678+0000 7f7bd5200640 1 --2- 172.21.15.97:0/978053709 >> v2:172.21.15.135:3300/0 conn(0x7f7bd015fc10 0x7f7bd0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.678+0000 7f7bd5200640 1 -- 172.21.15.97:0/978053709 >> v1:172.21.15.135:6789/0 conn(0x7f7bd00ce4a0 legacy=0x7f7bd00ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.678+0000 7f7bd5200640 1 -- 172.21.15.97:0/978053709 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd0160990 con 0x7f7bd015fff0 2024-03-06T07:30:01.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.679+0000 7f7bd5200640 1 --2- 172.21.15.97:0/978053709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd015fff0 0x7f7bd0160450 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00cf770 tx=0x7f7bbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee8f76d85f5e5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.679+0000 7f7bcfe00640 1 -- 172.21.15.97:0/978053709 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc035020 con 0x7f7bd015fff0 2024-03-06T07:30:01.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.679+0000 7f7bcfe00640 1 -- 172.21.15.97:0/978053709 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc031870 con 0x7f7bd015fff0 2024-03-06T07:30:01.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.679+0000 7f7bcfe00640 1 -- 172.21.15.97:0/978053709 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc031c40 con 0x7f7bd015fff0 2024-03-06T07:30:01.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 -- 172.21.15.97:0/978053709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd015fff0 msgr2=0x7f7bd0160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 --2- 172.21.15.97:0/978053709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd015fff0 0x7f7bd0160450 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00cf770 tx=0x7f7bbc02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:01.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 -- 172.21.15.97:0/978053709 shutdown_connections 2024-03-06T07:30:01.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 --2- 172.21.15.97:0/978053709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd015fff0 0x7f7bd0160450 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 --2- 172.21.15.97:0/978053709 >> v2:172.21.15.135:3300/0 conn(0x7f7bd015fc10 0x7f7bd0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 -- 172.21.15.97:0/978053709 >> 172.21.15.97:0/978053709 conn(0x7f7bd00b1940 msgr2=0x7f7bd00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 -- 172.21.15.97:0/978053709 shutdown_connections 2024-03-06T07:30:01.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.680+0000 7f7bd8600640 1 -- 172.21.15.97:0/978053709 wait complete. 2024-03-06T07:30:01.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.682+0000 7f7bd8600640 1 Processor -- start 2024-03-06T07:30:01.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.683+0000 7f7bd8600640 1 -- start start 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd8600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 0x7f7bd0155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd8600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bd015fc10 0x7f7bd006db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd8600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bd015fff0 0x7f7bd006e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd8600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd0072a30 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd8600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd00728b0 con 0x7f7bd015fff0 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd8600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd015c2e0 con 0x7f7bd015fc10 2024-03-06T07:30:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 0x7f7bd0155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 0x7f7bd0155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41204/0 (socket says 172.21.15.97:41204) 2024-03-06T07:30:01.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd5c00640 1 -- 172.21.15.97:0/814670131 learned_addr learned my addr 172.21.15.97:0/814670131 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:01.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd5200640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bd015fc10 0x7f7bd006db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.684+0000 7f7bd6600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bd015fff0 0x7f7bd006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd5c00640 1 -- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bd015fc10 msgr2=0x7f7bd006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd5c00640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bd015fc10 0x7f7bd006db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd5c00640 1 -- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bd015fff0 msgr2=0x7f7bd006e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd5c00640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bd015fff0 0x7f7bd006e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd5c00640 1 -- 172.21.15.97:0/814670131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd00c9fd0 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd5c00640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 0x7f7bd0155630 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400d800 tx=0x7f7bc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400b6c0 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc4019070 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd00ca2b0 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400ba80 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.685+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd00c9c80 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.687+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd00ca9f0 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.687+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bc400bc20 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.688+0000 7f7bcd600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7bb006b4d0 0x7f7bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.688+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bc408a620 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.689+0000 7f7bd5200640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7bb006b4d0 0x7f7bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.689+0000 7f7bd5200640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7bb006b4d0 0x7f7bb006d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc036000 tx=0x7f7bbc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.693+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bc4059530 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.890+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7bd004f330 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.891+0000 7f7bcd600640 1 -- 172.21.15.97:0/814670131 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7bc4058e30 con 0x7f7bd00ce4a0 2024-03-06T07:30:01.893 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:30:01.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7bb006b4d0 msgr2=0x7f7bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7bb006b4d0 0x7f7bb006d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc036000 tx=0x7f7bbc008030 comp rx=0 tx=0).stop 2024-03-06T07:30:01.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 msgr2=0x7f7bd0155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 0x7f7bd0155630 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400d800 tx=0x7f7bc400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 shutdown_connections 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7bb006b4d0 0x7f7bb006d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bd015fff0 0x7f7bd006e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bd015fc10 0x7f7bd006db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 --2- 172.21.15.97:0/814670131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd00ce4a0 0x7f7bd0155630 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.893+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 >> 172.21.15.97:0/814670131 conn(0x7f7bd00b1940 msgr2=0x7f7bd00b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.894+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 shutdown_connections 2024-03-06T07:30:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:01.894+0000 7f7bd8600640 1 -- 172.21.15.97:0/814670131 wait complete. 2024-03-06T07:30:01.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:02.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.121+0000 7f8f13c00640 1 Processor -- start 2024-03-06T07:30:02.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.121+0000 7f8f13c00640 1 -- start start 2024-03-06T07:30:02.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.123+0000 7f8f13c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 0x7f8f0c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.123+0000 7f8f13c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c0b46c0 con 0x7f8f0c15fc10 2024-03-06T07:30:02.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.123+0000 7f8f13c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c0b49a0 con 0x7f8f0c0b3f70 2024-03-06T07:30:02.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.123+0000 7f8f13c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c072c90 con 0x7f8f0c160700 2024-03-06T07:30:02.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.123+0000 7f8f11200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8f0c15fc10 0x7f8f0c15fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35058/0 (socket says 172.21.15.97:35058) 2024-03-06T07:30:02.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.123+0000 7f8f11200640 1 -- 172.21.15.97:0/218047605 learned_addr learned my addr 172.21.15.97:0/218047605 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:02.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f11c00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 0x7f8f0c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832831155 0 0) 0x7f8f0c0b46c0 con 0x7f8f0c15fc10 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef0003520 con 0x7f8f0c15fc10 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731352063 0 0) 0x7f8ef0003520 con 0x7f8f0c15fc10 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 msgr2=0x7f8f0c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 0x7f8f0c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 >> v1:172.21.15.135:6789/0 conn(0x7f8f0c0b3f70 legacy=0x7f8f0c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c09dd80 con 0x7f8f0c15fc10 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8efc0019f0 con 0x7f8f0c15fc10 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 >> v1:172.21.15.97:6789/0 conn(0x7f8f0c15fc10 legacy=0x7f8f0c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ef00039e0 0x7f8ef0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 0x7f8ef0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ef0004d40 0x7f8ef00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c072c90 con 0x7f8ef0004d40 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef000bb50 con 0x7f8ef00039e0 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.124+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef000bd20 con 0x7f8f0c160700 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11c00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ef0004d40 0x7f8ef00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11200640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ef00039e0 0x7f8ef0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f0be00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 0x7f8ef0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11c00640 1 -- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 msgr2=0x7f8ef0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11c00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f0c160700 0x7f8ef0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11c00640 1 -- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ef00039e0 msgr2=0x7f8ef0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11c00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ef00039e0 0x7f8ef0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.125+0000 7f8f11c00640 1 -- 172.21.15.97:0/218047605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef0003520 con 0x7f8ef0004d40 2024-03-06T07:30:02.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.126+0000 7f8f11c00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ef0004d40 0x7f8ef00051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400ba60 tx=0x7f8ef0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff80619a0d121e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.126+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04036020 con 0x7f8ef0004d40 2024-03-06T07:30:02.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.126+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f04032630 con 0x7f8ef0004d40 2024-03-06T07:30:02.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.126+0000 7f8f0b400640 1 -- 172.21.15.97:0/218047605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400b600 con 0x7f8ef0004d40 2024-03-06T07:30:02.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f13c00640 1 -- 172.21.15.97:0/218047605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ef0004d40 msgr2=0x7f8ef00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f13c00640 1 --2- 172.21.15.97:0/218047605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ef0004d40 0x7f8ef00051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400ba60 tx=0x7f8ef0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f11c00640 1 -- 172.21.15.97:0/218047605 reap_dead start 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f13c00640 1 -- 172.21.15.97:0/218047605 shutdown_connections 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f13c00640 1 -- 172.21.15.97:0/218047605 >> 172.21.15.97:0/218047605 conn(0x7f8f0c0b1940 msgr2=0x7f8f0c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f13c00640 1 -- 172.21.15.97:0/218047605 shutdown_connections 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.127+0000 7f8f13c00640 1 -- 172.21.15.97:0/218047605 wait complete. 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.129+0000 7f8f13c00640 1 Processor -- start 2024-03-06T07:30:02.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.129+0000 7f8f13c00640 1 -- start start 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f13c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ef0004d40 0x7f8f0c145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f13c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 0x7f8f0c143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f13c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f0c1441a0 0x7f8f0c0b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f13c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c15c460 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f13c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c15c2e0 con 0x7f8f0c1441a0 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f13c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c070ff0 con 0x7f8ef0004d40 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f0be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 0x7f8f0c143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f0be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 0x7f8f0c143850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41224/0 (socket says 172.21.15.97:41224) 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.130+0000 7f8f0be00640 1 -- 172.21.15.97:0/2735522904 learned_addr learned my addr 172.21.15.97:0/2735522904 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f11200640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ef0004d40 0x7f8f0c145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f11c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f0c1441a0 0x7f8f0c0b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f0be00640 1 -- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ef0004d40 msgr2=0x7f8f0c145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f0be00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ef0004d40 0x7f8f0c145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f0be00640 1 -- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f0c1441a0 msgr2=0x7f8f0c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f0be00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f0c1441a0 0x7f8f0c0b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f0be00640 1 -- 172.21.15.97:0/2735522904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c144af0 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f11c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f0c1441a0 0x7f8f0c0b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:02.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f0be00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 0x7f8f0c143850 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800d5a0 tx=0x7f8ef800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.131+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8005480 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.132+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef8017070 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.132+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8003810 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.132+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c144dd0 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.132+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c145390 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.133+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c04f330 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.134+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ef80039b0 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.135+0000 7f8f08c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8eec06b500 0x7f8eec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.135+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ef8089ff0 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.135+0000 7f8f11200640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8eec06b500 0x7f8eec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.136+0000 7f8f11200640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8eec06b500 0x7f8eec06d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8efc002050 tx=0x7f8efc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.139+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ef8058e30 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.342+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f0c06ef30 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.342+0000 7f8f08c00640 1 -- 172.21.15.97:0/2735522904 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ef8058730 con 0x7f8f0c15a6a0 2024-03-06T07:30:02.345 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8eec06b500 msgr2=0x7f8eec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8eec06b500 0x7f8eec06d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8efc002050 tx=0x7f8efc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 msgr2=0x7f8f0c143850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 0x7f8f0c143850 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800d5a0 tx=0x7f8ef800da60 comp rx=0 tx=0).stop 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 shutdown_connections 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8eec06b500 0x7f8eec06d9c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f0c1441a0 0x7f8f0c0b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f0c15a6a0 0x7f8f0c143850 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 --2- 172.21.15.97:0/2735522904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ef0004d40 0x7f8f0c145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 >> 172.21.15.97:0/2735522904 conn(0x7f8f0c0b1940 msgr2=0x7f8f0c0ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 shutdown_connections 2024-03-06T07:30:02.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.345+0000 7f8f13c00640 1 -- 172.21.15.97:0/2735522904 wait complete. 2024-03-06T07:30:02.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:02.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.570+0000 7ff420a00640 1 Processor -- start 2024-03-06T07:30:02.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.570+0000 7ff420a00640 1 -- start start 2024-03-06T07:30:02.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.571+0000 7ff420a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff41c15a940 0x7ff41c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff420a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cc2a0 0x7ff41c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff420a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c0b2b30 con 0x7ff41c0cf370 2024-03-06T07:30:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff420a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41c0b28e0 con 0x7ff41c15a940 2024-03-06T07:30:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff420a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b2e10 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff41b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cc2a0 0x7ff41c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff41a800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff41c15a940 0x7ff41c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff419e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff41c0cf370 0x7ff41c06d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:59178/0 (socket says 172.21.15.97:59178) 2024-03-06T07:30:02.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.572+0000 7ff419e00640 1 -- 172.21.15.97:0/4253177784 learned_addr learned my addr 172.21.15.97:0/4253177784 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:02.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.573+0000 7ff419400640 1 -- 172.21.15.97:0/4253177784 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215753370 0 0) 0x7ff41c0b2b30 con 0x7ff41c0cf370 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.573+0000 7ff419400640 1 -- 172.21.15.97:0/4253177784 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff400003520 con 0x7ff41c0cf370 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.573+0000 7ff41b200640 1 -- 172.21.15.97:0/4253177784 >> v2:172.21.15.135:3300/0 conn(0x7ff41c15a940 msgr2=0x7ff41c0b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.573+0000 7ff41b200640 1 --2- 172.21.15.97:0/4253177784 >> v2:172.21.15.135:3300/0 conn(0x7ff41c15a940 0x7ff41c0b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.573+0000 7ff41b200640 1 -- 172.21.15.97:0/4253177784 >> v1:172.21.15.135:6789/0 conn(0x7ff41c0cf370 legacy=0x7ff41c06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.573+0000 7ff41b200640 1 -- 172.21.15.97:0/4253177784 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c06ca50 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.575+0000 7ff41b200640 1 --2- 172.21.15.97:0/4253177784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cc2a0 0x7ff41c0cc750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4140097a0 tx=0x7ff41402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3f16d792f0a542 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.575+0000 7ff419400640 1 -- 172.21.15.97:0/4253177784 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414035020 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.575+0000 7ff419400640 1 -- 172.21.15.97:0/4253177784 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414031810 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.575+0000 7ff419400640 1 -- 172.21.15.97:0/4253177784 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414031bc0 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 -- 172.21.15.97:0/4253177784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cc2a0 msgr2=0x7ff41c0cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 --2- 172.21.15.97:0/4253177784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cc2a0 0x7ff41c0cc750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4140097a0 tx=0x7ff41402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 -- 172.21.15.97:0/4253177784 shutdown_connections 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 --2- 172.21.15.97:0/4253177784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cc2a0 0x7ff41c0cc750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 --2- 172.21.15.97:0/4253177784 >> v2:172.21.15.135:3300/0 conn(0x7ff41c15a940 0x7ff41c0b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 -- 172.21.15.97:0/4253177784 >> 172.21.15.97:0/4253177784 conn(0x7ff41c06afc0 msgr2=0x7ff41c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 -- 172.21.15.97:0/4253177784 shutdown_connections 2024-03-06T07:30:02.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.576+0000 7ff420a00640 1 -- 172.21.15.97:0/4253177784 wait complete. 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.578+0000 7ff420a00640 1 Processor -- start 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.578+0000 7ff420a00640 1 -- start start 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff420a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 0x7ff41c145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff420a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cf370 0x7ff41c1e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff420a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff41c15a940 0x7ff41c15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff420a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b4b20 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff420a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b30f0 con 0x7ff41c15a940 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff420a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c15ad20 con 0x7ff41c0cf370 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff41a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 0x7ff41c145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff41a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 0x7ff41c145410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41234/0 (socket says 172.21.15.97:41234) 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff41a800640 1 -- 172.21.15.97:0/1117591075 learned_addr learned my addr 172.21.15.97:0/1117591075 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff419e00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cf370 0x7ff41c1e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.579+0000 7ff41b200640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff41c15a940 0x7ff41c15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff41a800640 1 -- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cf370 msgr2=0x7ff41c1e43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff41a800640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cf370 0x7ff41c1e43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff41a800640 1 -- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff41c15a940 msgr2=0x7ff41c15d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff41a800640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff41c15a940 0x7ff41c15d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff41a800640 1 -- 172.21.15.97:0/1117591075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c15c0c0 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff41a800640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 0x7ff41c145410 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00efc0 tx=0x7ff40c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c018020 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c014480 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c014850 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.580+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c15c3a0 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.581+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c15c960 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.582+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c04f330 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.583+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff40c005d10 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.583+0000 7ff412a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3f806b500 0x7ff3f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.584+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff40c08add0 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.584+0000 7ff419e00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3f806b500 0x7ff3f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.585+0000 7ff419e00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3f806b500 0x7ff3f806d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff41c156410 tx=0x7ff404009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.589+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff40c022070 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.824+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff41c06b680 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.825+0000 7ff412a00640 1 -- 172.21.15.97:0/1117591075 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff40c010040 con 0x7ff41c0cc2a0 2024-03-06T07:30:02.827 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:02.827 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:37.643662+0000","modified":"2024-03-06T07:29:40.715394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6835/1918561213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1918561213},{"type":"v1","addr":"172.21.15.135:6835","nonce":1918561213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":3}]} 2024-03-06T07:30:02.828 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 26 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3f806b500 msgr2=0x7ff3f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3f806b500 0x7ff3f806d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff41c156410 tx=0x7ff404009040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 msgr2=0x7ff41c145410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 0x7ff41c145410 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00efc0 tx=0x7ff40c00c490 comp rx=0 tx=0).stop 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 shutdown_connections 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3f806b500 0x7ff3f806d9c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff41c15a940 0x7ff41c15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff41c0cf370 0x7ff41c1e43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 --2- 172.21.15.97:0/1117591075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41c0cc2a0 0x7ff41c145410 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 >> 172.21.15.97:0/1117591075 conn(0x7ff41c06afc0 msgr2=0x7ff41c0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 shutdown_connections 2024-03-06T07:30:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:02.828+0000 7ff420a00640 1 -- 172.21.15.97:0/1117591075 wait complete. 2024-03-06T07:30:02.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:03.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.052+0000 7fc5e5800640 1 Processor -- start 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.052+0000 7fc5e5800640 1 -- start start 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5e5800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc5e0161580 0x7fc5e00d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5e5800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5e00d1e90 0x7fc5e00d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5e5800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e0161960 con 0x7fc5e00d28a0 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5e5800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00d2d10 con 0x7fc5e00d1e90 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5e5800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00cf6f0 con 0x7fc5e0161580 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5d7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5e00d1e90 0x7fc5e00d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5d7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5e00d1e90 0x7fc5e00d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41250/0 (socket says 172.21.15.97:41250) 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.054+0000 7fc5d7400640 1 -- 172.21.15.97:0/1078045959 learned_addr learned my addr 172.21.15.97:0/1078045959 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.055+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/1078045959 >> v2:172.21.15.135:3300/0 conn(0x7fc5e0161580 0x7fc5e00d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.055+0000 7fc5d7400640 1 -- 172.21.15.97:0/1078045959 >> v2:172.21.15.135:3300/0 conn(0x7fc5e0161580 msgr2=0x7fc5e00d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.055+0000 7fc5d7400640 1 --2- 172.21.15.97:0/1078045959 >> v2:172.21.15.135:3300/0 conn(0x7fc5e0161580 0x7fc5e00d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.055+0000 7fc5d7400640 1 -- 172.21.15.97:0/1078045959 >> v1:172.21.15.135:6789/0 conn(0x7fc5e00d28a0 legacy=0x7fc5e00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.055+0000 7fc5d7400640 1 -- 172.21.15.97:0/1078045959 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00cf9d0 con 0x7fc5e00d1e90 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.056+0000 7fc5d7400640 1 --2- 172.21.15.97:0/1078045959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d1e90 0x7fc5e00d2360 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0073740 tx=0x7fc5c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970bd63b380ec62e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.056+0000 7fc5df400640 1 -- 172.21.15.97:0/1078045959 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8035020 con 0x7fc5e00d1e90 2024-03-06T07:30:03.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.056+0000 7fc5df400640 1 -- 172.21.15.97:0/1078045959 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c8031870 con 0x7fc5e00d1e90 2024-03-06T07:30:03.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.056+0000 7fc5df400640 1 -- 172.21.15.97:0/1078045959 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8031c40 con 0x7fc5e00d1e90 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 -- 172.21.15.97:0/1078045959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d1e90 msgr2=0x7fc5e00d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 --2- 172.21.15.97:0/1078045959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d1e90 0x7fc5e00d2360 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0073740 tx=0x7fc5c802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 -- 172.21.15.97:0/1078045959 shutdown_connections 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 --2- 172.21.15.97:0/1078045959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d1e90 0x7fc5e00d2360 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 --2- 172.21.15.97:0/1078045959 >> v2:172.21.15.135:3300/0 conn(0x7fc5e0161580 0x7fc5e00d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 -- 172.21.15.97:0/1078045959 >> 172.21.15.97:0/1078045959 conn(0x7fc5e00d3d80 msgr2=0x7fc5e00d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 -- 172.21.15.97:0/1078045959 shutdown_connections 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.057+0000 7fc5e5800640 1 -- 172.21.15.97:0/1078045959 wait complete. 2024-03-06T07:30:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.059+0000 7fc5e5800640 1 Processor -- start 2024-03-06T07:30:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.059+0000 7fc5e5800640 1 -- start start 2024-03-06T07:30:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e5800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e00d1e90 0x7fc5e0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e5800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d28a0 0x7fc5e00b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 0x7fc5e00b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e5800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00b39a0 con 0x7fc5e0161580 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e5800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00b36c0 con 0x7fc5e00d28a0 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e5800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00cf3a0 con 0x7fc5e00d1e90 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 0x7fc5e00b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 0x7fc5e00b9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41252/0 (socket says 172.21.15.97:41252) 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.060+0000 7fc5e4e00640 1 -- 172.21.15.97:0/2986721724 learned_addr learned my addr 172.21.15.97:0/2986721724 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e00d1e90 0x7fc5e0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5d7400640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d28a0 0x7fc5e00b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e4e00640 1 -- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e00d1e90 msgr2=0x7fc5e0158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e4e00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e00d1e90 0x7fc5e0158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e4e00640 1 -- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d28a0 msgr2=0x7fc5e00b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e4e00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d28a0 0x7fc5e00b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e4e00640 1 -- 172.21.15.97:0/2986721724 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e01f4440 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5d7400640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d28a0 0x7fc5e00b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e4e00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 0x7fc5e00b9870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800b410 tx=0x7fc5d800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8015030 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d8004020 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8004410 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.061+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e01f4720 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.062+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e01f4ce0 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.063+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e004f330 con 0x7fc5e0161580 2024-03-06T07:30:03.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.063+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5d8018070 con 0x7fc5e0161580 2024-03-06T07:30:03.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.064+0000 7fc5dcc00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5bc06b4d0 0x7fc5bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.064+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5d8089d50 con 0x7fc5e0161580 2024-03-06T07:30:03.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.065+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5bc06b4d0 0x7fc5bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.065+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5bc06b4d0 0x7fc5bc06d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00099c0 tx=0x7fc5d0003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.069+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5d8058b30 con 0x7fc5e0161580 2024-03-06T07:30:03.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.270+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5e00d3b60 con 0x7fc5e0161580 2024-03-06T07:30:03.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.272+0000 7fc5dcc00640 1 -- 172.21.15.97:0/2986721724 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16601 (secure 0 0 0) 0x7fc5d8058430 con 0x7fc5e0161580 2024-03-06T07:30:03.274 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:03.275 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":46,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:39.204502+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:03.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5bc06b4d0 msgr2=0x7fc5bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5bc06b4d0 0x7fc5bc06d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00099c0 tx=0x7fc5d0003990 comp rx=0 tx=0).stop 2024-03-06T07:30:03.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 msgr2=0x7fc5e00b9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 0x7fc5e00b9870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800b410 tx=0x7fc5d800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 shutdown_connections 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5bc06b4d0 0x7fc5bc06d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0161580 0x7fc5e00b9870 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d28a0 0x7fc5e00b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 --2- 172.21.15.97:0/2986721724 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e00d1e90 0x7fc5e0158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 >> 172.21.15.97:0/2986721724 conn(0x7fc5e00d3d80 msgr2=0x7fc5e006b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 shutdown_connections 2024-03-06T07:30:03.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.275+0000 7fc5e5800640 1 -- 172.21.15.97:0/2986721724 wait complete. 2024-03-06T07:30:03.297 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:30:03.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:03.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.501+0000 7f64dee00640 1 Processor -- start 2024-03-06T07:30:03.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.501+0000 7f64dee00640 1 -- start start 2024-03-06T07:30:03.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.502+0000 7f64dee00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f64d80b4680 0x7f64d806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64dee00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d815b090 con 0x7f64d815acb0 2024-03-06T07:30:03.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64dee00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d80b2a30 con 0x7f64d8072340 2024-03-06T07:30:03.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64dee00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d80b4360 con 0x7f64d80b4680 2024-03-06T07:30:03.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64d7e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f64d815acb0 0x7f64d80b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35062/0 (socket says 172.21.15.97:35062) 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64d7e00640 1 -- 172.21.15.97:0/937291375 learned_addr learned my addr 172.21.15.97:0/937291375 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64cf400640 1 --2- 172.21.15.97:0/937291375 >> v2:172.21.15.97:3300/0 conn(0x7f64d80b4680 0x7f64d806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64d7400640 1 -- 172.21.15.97:0/937291375 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151692183 0 0) 0x7f64d815b090 con 0x7f64d815acb0 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.503+0000 7f64d7400640 1 -- 172.21.15.97:0/937291375 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc003520 con 0x7f64d815acb0 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.504+0000 7f64cf400640 1 -- 172.21.15.97:0/937291375 >> v1:172.21.15.135:6789/0 conn(0x7f64d8072340 legacy=0x7f64d80ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.504+0000 7f64cf400640 1 -- 172.21.15.97:0/937291375 >> v1:172.21.15.97:6789/0 conn(0x7f64d815acb0 legacy=0x7f64d80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.504+0000 7f64cf400640 1 -- 172.21.15.97:0/937291375 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d80b2cf0 con 0x7f64d80b4680 2024-03-06T07:30:03.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.505+0000 7f64cf400640 1 --2- 172.21.15.97:0/937291375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d806e4f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f64c00097a0 tx=0x7f64c002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61cd601f5d8f5286 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.505+0000 7f64d7400640 1 -- 172.21.15.97:0/937291375 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c0034070 con 0x7f64d80b4680 2024-03-06T07:30:03.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.505+0000 7f64d7400640 1 -- 172.21.15.97:0/937291375 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c0030610 con 0x7f64d80b4680 2024-03-06T07:30:03.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.505+0000 7f64d7400640 1 -- 172.21.15.97:0/937291375 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c00309e0 con 0x7f64d80b4680 2024-03-06T07:30:03.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 -- 172.21.15.97:0/937291375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 msgr2=0x7f64d806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 --2- 172.21.15.97:0/937291375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d806e4f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f64c00097a0 tx=0x7f64c002b960 comp rx=0 tx=0).stop 2024-03-06T07:30:03.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 -- 172.21.15.97:0/937291375 shutdown_connections 2024-03-06T07:30:03.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 --2- 172.21.15.97:0/937291375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d806e4f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 -- 172.21.15.97:0/937291375 >> 172.21.15.97:0/937291375 conn(0x7f64d80b1730 msgr2=0x7f64d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 -- 172.21.15.97:0/937291375 shutdown_connections 2024-03-06T07:30:03.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.506+0000 7f64dee00640 1 -- 172.21.15.97:0/937291375 wait complete. 2024-03-06T07:30:03.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.508+0000 7f64dee00640 1 Processor -- start 2024-03-06T07:30:03.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.508+0000 7f64dee00640 1 -- start start 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.509+0000 7f64dee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64d8072340 0x7f64d814b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.509+0000 7f64dee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d80adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.509+0000 7f64dee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64d815acb0 0x7f64d80ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.509+0000 7f64dee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80b4010 con 0x7f64d80b4680 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.509+0000 7f64dee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80b32b0 con 0x7f64d8072340 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.509+0000 7f64dee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d806ea30 con 0x7f64d815acb0 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d80adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d80adce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41268/0 (socket says 172.21.15.97:41268) 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 -- 172.21.15.97:0/3525943507 learned_addr learned my addr 172.21.15.97:0/3525943507 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64d7e00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64d8072340 0x7f64d814b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64dce00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64d815acb0 0x7f64d80ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 -- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64d815acb0 msgr2=0x7f64d80ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64d815acb0 0x7f64d80ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 -- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64d8072340 msgr2=0x7f64d814b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64d8072340 0x7f64d814b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64cf400640 1 -- 172.21.15.97:0/3525943507 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d80aeb30 con 0x7f64d80b4680 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.510+0000 7f64d7e00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64d8072340 0x7f64d814b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.511+0000 7f64cf400640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d80adce0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f64c002bf70 tx=0x7f64c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.511+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c0034040 con 0x7f64d80b4680 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.511+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c0030410 con 0x7f64d80b4680 2024-03-06T07:30:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.511+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c0039c50 con 0x7f64d80b4680 2024-03-06T07:30:03.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.511+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d80ae850 con 0x7f64d80b4680 2024-03-06T07:30:03.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.511+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d80cd2f0 con 0x7f64d80b4680 2024-03-06T07:30:03.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.512+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d804f330 con 0x7f64d80b4680 2024-03-06T07:30:03.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.513+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64c0037070 con 0x7f64d80b4680 2024-03-06T07:30:03.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.514+0000 7f64d4c00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64b406b4d0 0x7f64b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.514+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64c00ac0c0 con 0x7f64d80b4680 2024-03-06T07:30:03.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.515+0000 7f64d7e00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64b406b4d0 0x7f64b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.515+0000 7f64d7e00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64b406b4d0 0x7f64b406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f64c80011e0 tx=0x7f64c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.519+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64c00305e0 con 0x7f64d80b4680 2024-03-06T07:30:03.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.751+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64d806f3c0 con 0x7f64d80b4680 2024-03-06T07:30:03.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.751+0000 7f64d4c00640 1 -- 172.21.15.97:0/3525943507 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f64c007afd0 con 0x7f64d80b4680 2024-03-06T07:30:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:03.754 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:37.643662+0000","modified":"2024-03-06T07:29:40.715394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6835/1918561213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1918561213},{"type":"v1","addr":"172.21.15.135:6835","nonce":1918561213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":3}]} 2024-03-06T07:30:03.754 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 26 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64b406b4d0 msgr2=0x7f64b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64b406b4d0 0x7f64b406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f64c80011e0 tx=0x7f64c8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 msgr2=0x7f64d80adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d80adce0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f64c002bf70 tx=0x7f64c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 shutdown_connections 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64b406b4d0 0x7f64b406d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64d815acb0 0x7f64d80ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64d80b4680 0x7f64d80adce0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 --2- 172.21.15.97:0/3525943507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64d8072340 0x7f64d814b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 >> 172.21.15.97:0/3525943507 conn(0x7f64d80b1730 msgr2=0x7f64d8156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 shutdown_connections 2024-03-06T07:30:03.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.754+0000 7f64dee00640 1 -- 172.21.15.97:0/3525943507 wait complete. 2024-03-06T07:30:03.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:03.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.981+0000 7f63b2800640 1 Processor -- start 2024-03-06T07:30:03.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.981+0000 7f63b2800640 1 -- start start 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63b2800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f63ac15f9d0 0x7f63ac15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63b2800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63ac070b10 0x7f63ac070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63b2800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac06cef0 con 0x7f63ac144c00 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63b2800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac06d1d0 con 0x7f63ac070b10 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63b2800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac06d4b0 con 0x7f63ac15f9d0 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63ac070b10 0x7f63ac070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63ac070b10 0x7f63ac070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35164/0 (socket says 172.21.15.97:35164) 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 -- 172.21.15.97:0/974296819 learned_addr learned my addr 172.21.15.97:0/974296819 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63a3e00640 1 --2- 172.21.15.97:0/974296819 >> v2:172.21.15.135:3300/0 conn(0x7f63ac15f9d0 0x7f63ac15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 -- 172.21.15.97:0/974296819 >> v2:172.21.15.135:3300/0 conn(0x7f63ac15f9d0 msgr2=0x7f63ac15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 --2- 172.21.15.97:0/974296819 >> v2:172.21.15.135:3300/0 conn(0x7f63ac15f9d0 0x7f63ac15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 -- 172.21.15.97:0/974296819 >> v1:172.21.15.135:6789/0 conn(0x7f63ac144c00 legacy=0x7f63ac06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.983+0000 7f63abe00640 1 -- 172.21.15.97:0/974296819 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac06d790 con 0x7f63ac070b10 2024-03-06T07:30:03.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.985+0000 7f63abe00640 1 --2- 172.21.15.97:0/974296819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac070f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0b26f0 tx=0x7f63a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a747e6cca5e61564 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.985+0000 7f63aaa00640 1 -- 172.21.15.97:0/974296819 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a402ca80 con 0x7f63ac070b10 2024-03-06T07:30:03.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.985+0000 7f63aaa00640 1 -- 172.21.15.97:0/974296819 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a402cc20 con 0x7f63ac070b10 2024-03-06T07:30:03.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.985+0000 7f63aaa00640 1 -- 172.21.15.97:0/974296819 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a40343f0 con 0x7f63ac070b10 2024-03-06T07:30:03.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.985+0000 7f63b2800640 1 -- 172.21.15.97:0/974296819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 msgr2=0x7f63ac070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.985+0000 7f63b2800640 1 --2- 172.21.15.97:0/974296819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac070f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0b26f0 tx=0x7f63a402bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.986+0000 7f63b2800640 1 -- 172.21.15.97:0/974296819 shutdown_connections 2024-03-06T07:30:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.986+0000 7f63b2800640 1 --2- 172.21.15.97:0/974296819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac070f20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.986+0000 7f63b2800640 1 --2- 172.21.15.97:0/974296819 >> v2:172.21.15.135:3300/0 conn(0x7f63ac15f9d0 0x7f63ac15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.986+0000 7f63b2800640 1 -- 172.21.15.97:0/974296819 >> 172.21.15.97:0/974296819 conn(0x7f63ac0b1730 msgr2=0x7f63ac0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.986+0000 7f63b2800640 1 -- 172.21.15.97:0/974296819 shutdown_connections 2024-03-06T07:30:03.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.986+0000 7f63b2800640 1 -- 172.21.15.97:0/974296819 wait complete. 2024-03-06T07:30:03.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.987+0000 7f63b2800640 1 Processor -- start 2024-03-06T07:30:03.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.988+0000 7f63b2800640 1 -- start start 2024-03-06T07:30:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.988+0000 7f63b2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac1e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.988+0000 7f63b2800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63ac144c00 0x7f63ac1e6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63b2800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ac15f9d0 0x7f63ac15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63b2800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac06ccd0 con 0x7f63ac070b10 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63b2800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac06cb50 con 0x7f63ac15f9d0 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63b2800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac06df90 con 0x7f63ac144c00 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac1e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac1e7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35172/0 (socket says 172.21.15.97:35172) 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 -- 172.21.15.97:0/168546652 learned_addr learned my addr 172.21.15.97:0/168546652 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63a3e00640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63ac144c00 0x7f63ac1e6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 -- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63ac144c00 msgr2=0x7f63ac1e6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63ac144c00 0x7f63ac1e6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 -- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ac15f9d0 msgr2=0x7f63ac15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ac15f9d0 0x7f63ac15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 -- 172.21.15.97:0/168546652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac1e6b20 con 0x7f63ac070b10 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.989+0000 7f63ab400640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac1e7a20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f639c00c430 tx=0x7f639c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.990+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c011790 con 0x7f63ac070b10 2024-03-06T07:30:03.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.990+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c017070 con 0x7f63ac070b10 2024-03-06T07:30:03.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.990+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c011bb0 con 0x7f63ac070b10 2024-03-06T07:30:03.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.990+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac1e6e00 con 0x7f63ac070b10 2024-03-06T07:30:03.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.990+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac12f390 con 0x7f63ac070b10 2024-03-06T07:30:03.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.991+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ac04f330 con 0x7f63ac070b10 2024-03-06T07:30:03.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.992+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f639c01c460 con 0x7f63ac070b10 2024-03-06T07:30:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.993+0000 7f63a3400640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f638806b520 0x7f638806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.993+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f639c089f80 con 0x7f63ac070b10 2024-03-06T07:30:03.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.993+0000 7f63a3e00640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f638806b520 0x7f638806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.994+0000 7f63a3e00640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f638806b520 0x7f638806d9e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6394006fd0 tx=0x7f6394008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:03.997+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f639c058ee0 con 0x7f63ac070b10 2024-03-06T07:30:04.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.234+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63ac0b1dc0 con 0x7f63ac070b10 2024-03-06T07:30:04.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.235+0000 7f63a3400640 1 -- 172.21.15.97:0/168546652 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f639c0587e0 con 0x7f63ac070b10 2024-03-06T07:30:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:04.238 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:37.643662+0000","modified":"2024-03-06T07:29:40.715394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5511":{"gid":5511,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6835/1918561213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1918561213},{"type":"v1","addr":"172.21.15.135:6835","nonce":1918561213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5511,"qdb_cluster":[5511]},"id":3}]} 2024-03-06T07:30:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 26 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f638806b520 msgr2=0x7f638806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f638806b520 0x7f638806d9e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6394006fd0 tx=0x7f6394008040 comp rx=0 tx=0).stop 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 msgr2=0x7f63ac1e7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac1e7a20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f639c00c430 tx=0x7f639c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 shutdown_connections 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f638806b520 0x7f638806d9e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ac15f9d0 0x7f63ac15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63ac144c00 0x7f63ac1e6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 --2- 172.21.15.97:0/168546652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ac070b10 0x7f63ac1e7a20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.238+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 >> 172.21.15.97:0/168546652 conn(0x7f63ac0b1730 msgr2=0x7f63ac0b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.239+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 shutdown_connections 2024-03-06T07:30:04.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.239+0000 7f63b2800640 1 -- 172.21.15.97:0/168546652 wait complete. 2024-03-06T07:30:04.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:04.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.462+0000 7f01f3e00640 1 Processor -- start 2024-03-06T07:30:04.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.462+0000 7f01f3e00640 1 -- start start 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.463+0000 7f01f3e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f01f41561f0 0x7f01f40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.463+0000 7f01f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01f40aebf0 0x7f01f40cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f3e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f415c240 con 0x7f01f40cc800 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f3e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f415ce30 con 0x7f01f40aebf0 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f3e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f415c500 con 0x7f01f41561f0 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01f40aebf0 0x7f01f40cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01f40aebf0 0x7f01f40cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35184/0 (socket says 172.21.15.97:35184) 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 -- 172.21.15.97:0/489383078 learned_addr learned my addr 172.21.15.97:0/489383078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2a00640 1 --2- 172.21.15.97:0/489383078 >> v2:172.21.15.135:3300/0 conn(0x7f01f41561f0 0x7f01f40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 -- 172.21.15.97:0/489383078 >> v2:172.21.15.135:3300/0 conn(0x7f01f41561f0 msgr2=0x7f01f40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 --2- 172.21.15.97:0/489383078 >> v2:172.21.15.135:3300/0 conn(0x7f01f41561f0 0x7f01f40ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 -- 172.21.15.97:0/489383078 >> v1:172.21.15.135:6789/0 conn(0x7f01f40cc800 legacy=0x7f01f40cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.464+0000 7f01f2000640 1 -- 172.21.15.97:0/489383078 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f415c7e0 con 0x7f01f40aebf0 2024-03-06T07:30:04.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.465+0000 7f01f2000640 1 --2- 172.21.15.97:0/489383078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40aebf0 0x7f01f40cc2c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f01f40b4db0 tx=0x7f01e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df560de197fea24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.466+0000 7f01f1600640 1 -- 172.21.15.97:0/489383078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0035020 con 0x7f01f40aebf0 2024-03-06T07:30:04.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.466+0000 7f01f1600640 1 -- 172.21.15.97:0/489383078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e0031870 con 0x7f01f40aebf0 2024-03-06T07:30:04.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.466+0000 7f01f1600640 1 -- 172.21.15.97:0/489383078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0031c40 con 0x7f01f40aebf0 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.466+0000 7f01f3e00640 1 -- 172.21.15.97:0/489383078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40aebf0 msgr2=0x7f01f40cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.466+0000 7f01f3e00640 1 --2- 172.21.15.97:0/489383078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40aebf0 0x7f01f40cc2c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f01f40b4db0 tx=0x7f01e002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.467+0000 7f01f3e00640 1 -- 172.21.15.97:0/489383078 shutdown_connections 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.467+0000 7f01f3e00640 1 --2- 172.21.15.97:0/489383078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40aebf0 0x7f01f40cc2c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.467+0000 7f01f3e00640 1 --2- 172.21.15.97:0/489383078 >> v2:172.21.15.135:3300/0 conn(0x7f01f41561f0 0x7f01f40ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.467+0000 7f01f3e00640 1 -- 172.21.15.97:0/489383078 >> 172.21.15.97:0/489383078 conn(0x7f01f406afc0 msgr2=0x7f01f406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.467+0000 7f01f3e00640 1 -- 172.21.15.97:0/489383078 shutdown_connections 2024-03-06T07:30:04.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.467+0000 7f01f3e00640 1 -- 172.21.15.97:0/489383078 wait complete. 2024-03-06T07:30:04.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.468+0000 7f01f3e00640 1 Processor -- start 2024-03-06T07:30:04.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.468+0000 7f01f3e00640 1 -- start start 2024-03-06T07:30:04.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.469+0000 7f01f3e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01f40aebf0 0x7f01f4154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.469+0000 7f01f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 0x7f01f41e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.469+0000 7f01f3e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01f41561f0 0x7f01f4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.469+0000 7f01f3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f41565d0 con 0x7f01f40cc800 2024-03-06T07:30:04.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.469+0000 7f01f3e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40ccc50 con 0x7f01f41561f0 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.469+0000 7f01f3e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f4156750 con 0x7f01f40aebf0 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 0x7f01f41e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 0x7f01f41e54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35198/0 (socket says 172.21.15.97:35198) 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 -- 172.21.15.97:0/3759197125 learned_addr learned my addr 172.21.15.97:0/3759197125 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f3400640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01f41561f0 0x7f01f4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 -- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01f40aebf0 msgr2=0x7f01f4154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01f40aebf0 0x7f01f4154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 -- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01f41561f0 msgr2=0x7f01f4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01f41561f0 0x7f01f4158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 -- 172.21.15.97:0/3759197125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f41e59f0 con 0x7f01f40cc800 2024-03-06T07:30:04.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.470+0000 7f01f2000640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 0x7f01f41e54b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f01e0000c00 tx=0x7f01e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.471+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0040070 con 0x7f01f40cc800 2024-03-06T07:30:04.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.471+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e0030d00 con 0x7f01f40cc800 2024-03-06T07:30:04.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.471+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0035040 con 0x7f01f40cc800 2024-03-06T07:30:04.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.471+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f41e6a70 con 0x7f01f40cc800 2024-03-06T07:30:04.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.471+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f41e7030 con 0x7f01f40cc800 2024-03-06T07:30:04.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.472+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f41e7270 con 0x7f01f40cc800 2024-03-06T07:30:04.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.473+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01e004b060 con 0x7f01f40cc800 2024-03-06T07:30:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.474+0000 7f01eaa00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01d406b4d0 0x7f01d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.474+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01e00abfe0 con 0x7f01f40cc800 2024-03-06T07:30:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.474+0000 7f01f2a00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01d406b4d0 0x7f01d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.475+0000 7f01f2a00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01d406b4d0 0x7f01d406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f01e4006fd0 tx=0x7f01e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.479+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01f41e7270 con 0x7f01f40cc800 2024-03-06T07:30:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.659+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01f404f330 con 0x7f01f40cc800 2024-03-06T07:30:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.661+0000 7f01eaa00640 1 -- 172.21.15.97:0/3759197125 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16601 (secure 0 0 0) 0x7f01e007aeb0 con 0x7f01f40cc800 2024-03-06T07:30:04.663 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:04.664 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":46,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:29:39.204502+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:35.365606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:36.650975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01d406b4d0 msgr2=0x7f01d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01d406b4d0 0x7f01d406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f01e4006fd0 tx=0x7f01e4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 msgr2=0x7f01f41e54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 0x7f01f41e54b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f01e0000c00 tx=0x7f01e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 shutdown_connections 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f01d406b4d0 0x7f01d406d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01f41561f0 0x7f01f4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01f40cc800 0x7f01f41e54b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 --2- 172.21.15.97:0/3759197125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01f40aebf0 0x7f01f4154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.663+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 >> 172.21.15.97:0/3759197125 conn(0x7f01f406afc0 msgr2=0x7f01f415a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.664+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 shutdown_connections 2024-03-06T07:30:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.664+0000 7f01f3e00640 1 -- 172.21.15.97:0/3759197125 wait complete. 2024-03-06T07:30:04.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:30:04.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.880+0000 7f6a18000640 1 Processor -- start 2024-03-06T07:30:04.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.881+0000 7f6a18000640 1 -- start start 2024-03-06T07:30:04.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.882+0000 7f6a18000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6a10164b30 0x7f6a10166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.882+0000 7f6a18000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a10145b30 con 0x7f6a100c6460 2024-03-06T07:30:04.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.882+0000 7f6a18000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a10142ce0 con 0x7f6a100c4930 2024-03-06T07:30:04.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.882+0000 7f6a18000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a100c68c0 con 0x7f6a10164b30 2024-03-06T07:30:04.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.882+0000 7f6a14c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6a100c6460 0x7f6a100c5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60672/0 (socket says 172.21.15.97:60672) 2024-03-06T07:30:04.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a14c00640 1 -- 172.21.15.97:0/4225625662 learned_addr learned my addr 172.21.15.97:0/4225625662 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:04.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a16000640 1 --2- 172.21.15.97:0/4225625662 >> v2:172.21.15.135:3300/0 conn(0x7f6a10164b30 0x7f6a10166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748762338 0 0) 0x7f69fc001110 con 0x7f6a100c6460 2024-03-06T07:30:04.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8003520 con 0x7f6a100c6460 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752361029 0 0) 0x7f69f8003520 con 0x7f6a100c6460 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 >> v2:172.21.15.135:3300/0 conn(0x7f6a10164b30 msgr2=0x7f6a10166f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 --2- 172.21.15.97:0/4225625662 >> v2:172.21.15.135:3300/0 conn(0x7f6a10164b30 0x7f6a10166f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 >> v1:172.21.15.135:6789/0 conn(0x7f6a100c4930 legacy=0x7f6a100c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100cbca0 con 0x7f6a100c6460 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f69fc001840 con 0x7f6a100c6460 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 >> v1:172.21.15.97:6789/0 conn(0x7f6a100c6460 legacy=0x7f6a100c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f80039b0 0x7f69f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f8004800 0x7f69f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f80051c0 0x7f69f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100c68c0 con 0x7f69f80039b0 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f800c160 con 0x7f69f80051c0 2024-03-06T07:30:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.883+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f800c360 con 0x7f69f8004800 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a15600640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f80039b0 0x7f69f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a15600640 1 -- 172.21.15.97:0/4225625662 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f8004800 msgr2=0x7f69f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a15600640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f8004800 0x7f69f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a15600640 1 -- 172.21.15.97:0/4225625662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f80051c0 msgr2=0x7f69f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a15600640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f80051c0 0x7f69f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a15600640 1 -- 172.21.15.97:0/4225625662 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8003520 con 0x7f69f80039b0 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.884+0000 7f6a16000640 1 -- 172.21.15.97:0/4225625662 reap_dead start 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.885+0000 7f6a15600640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f80039b0 0x7f69f8003dd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6a04001750 tx=0x7f69f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd416fe6b13163b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.885+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a04001370 con 0x7f69f80039b0 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.885+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a040052d0 con 0x7f69f80039b0 2024-03-06T07:30:04.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.885+0000 7f6a0fe00640 1 -- 172.21.15.97:0/4225625662 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a04003830 con 0x7f69f80039b0 2024-03-06T07:30:04.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 -- 172.21.15.97:0/4225625662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f80039b0 msgr2=0x7f69f8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f80039b0 0x7f69f8003dd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6a04001750 tx=0x7f69f8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:04.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 -- 172.21.15.97:0/4225625662 shutdown_connections 2024-03-06T07:30:04.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 --2- 172.21.15.97:0/4225625662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f80039b0 0x7f69f8003dd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 -- 172.21.15.97:0/4225625662 >> 172.21.15.97:0/4225625662 conn(0x7f6a100cc4f0 msgr2=0x7f6a100cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 -- 172.21.15.97:0/4225625662 shutdown_connections 2024-03-06T07:30:04.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.886+0000 7f6a18000640 1 -- 172.21.15.97:0/4225625662 wait complete. 2024-03-06T07:30:04.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.888+0000 7f6a18000640 1 Processor -- start 2024-03-06T07:30:04.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.888+0000 7f6a18000640 1 -- start start 2024-03-06T07:30:04.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a18000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f80039b0 0x7f6a100cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a18000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 0x7f6a100ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a18000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a1013d710 0x7f6a100addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a18000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100ca3b0 con 0x7f6a1013d1e0 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a18000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100ca0d0 con 0x7f6a1013d710 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a18000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100cc260 con 0x7f69f80039b0 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 0x7f6a100ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 0x7f6a100ad890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35220/0 (socket says 172.21.15.97:35220) 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 -- 172.21.15.97:0/1467605400 learned_addr learned my addr 172.21.15.97:0/1467605400 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 -- 172.21.15.97:0/1467605400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f80039b0 msgr2=0x7f6a100cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f80039b0 0x7f6a100cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 -- 172.21.15.97:0/1467605400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a1013d710 msgr2=0x7f6a100addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a1013d710 0x7f6a100addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.889+0000 7f6a14c00640 1 -- 172.21.15.97:0/1467605400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100c92d0 con 0x7f6a1013d1e0 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.890+0000 7f6a14c00640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 0x7f6a100ad890 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c780 tx=0x7f69fc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.890+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc018070 con 0x7f6a1013d1e0 2024-03-06T07:30:04.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.890+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc0074e0 con 0x7f6a1013d1e0 2024-03-06T07:30:04.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.890+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a100c95b0 con 0x7f6a1013d1e0 2024-03-06T07:30:04.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.890+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc004a20 con 0x7f6a1013d1e0 2024-03-06T07:30:04.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.890+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a100c6f30 con 0x7f6a1013d1e0 2024-03-06T07:30:04.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.891+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1004f330 con 0x7f6a1013d1e0 2024-03-06T07:30:04.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.892+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69fc01b070 con 0x7f6a1013d1e0 2024-03-06T07:30:04.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.893+0000 7f6a0d600640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69f006b500 0x7f69f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.893+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69fc0896d0 con 0x7f6a1013d1e0 2024-03-06T07:30:04.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.893+0000 7f6a15600640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69f006b500 0x7f69f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.894+0000 7f6a15600640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69f006b500 0x7f69f006d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a04033000 tx=0x7f6a0402f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:04.897+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a10145b30 con 0x7f6a1013d1e0 2024-03-06T07:30:05.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.114+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6a101646e0 con 0x7f6a1013d1e0 2024-03-06T07:30:05.117 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:30:05.115+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:30:05.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.118+0000 7f6a0d600640 1 -- 172.21.15.97:0/1467605400 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f69fc058540 con 0x7f6a1013d1e0 2024-03-06T07:30:05.120 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.120+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69f006b500 msgr2=0x7f69f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.120+0000 7f6a18000640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69f006b500 0x7f69f006d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a04033000 tx=0x7f6a0402f000 comp rx=0 tx=0).stop 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.120+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 msgr2=0x7f6a100ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.120+0000 7f6a18000640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 0x7f6a100ad890 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c780 tx=0x7f69fc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 shutdown_connections 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69f006b500 0x7f69f006d9c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a1013d710 0x7f6a100addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a1013d1e0 0x7f6a100ad890 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 --2- 172.21.15.97:0/1467605400 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f80039b0 0x7f6a100cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 >> 172.21.15.97:0/1467605400 conn(0x7f6a100cc4f0 msgr2=0x7f6a10163630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 shutdown_connections 2024-03-06T07:30:05.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.121+0000 7f6a18000640 1 -- 172.21.15.97:0/1467605400 wait complete. 2024-03-06T07:30:05.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:30:05.263 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:30:05.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.335+0000 7f5075400640 1 Processor -- start 2024-03-06T07:30:05.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.336+0000 7f5075400640 1 -- start start 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5075400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f507006d960 0x7f507015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5075400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f507006e2f0 0x7f50700cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5075400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507006cba0 con 0x7f50700b2c40 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5075400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507015f700 con 0x7f507006d960 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5075400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507006d4d0 con 0x7f507006e2f0 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5065e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f507006d960 0x7f507015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f506e800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f50700b2c40 0x7f50700b3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60682/0 (socket says 172.21.15.97:60682) 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f5065e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f507006d960 0x7f507015f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35222/0 (socket says 172.21.15.97:35222) 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.337+0000 7f506e800640 1 -- 172.21.15.97:0/2544834005 learned_addr learned my addr 172.21.15.97:0/2544834005 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.338+0000 7f506f200640 1 --2- 172.21.15.97:0/2544834005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f507006e2f0 0x7f50700cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.338+0000 7f5065e00640 1 -- 172.21.15.97:0/2544834005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f507006e2f0 msgr2=0x7f50700cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.338+0000 7f5065e00640 1 --2- 172.21.15.97:0/2544834005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f507006e2f0 0x7f50700cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.338+0000 7f5065e00640 1 -- 172.21.15.97:0/2544834005 >> v1:172.21.15.97:6789/0 conn(0x7f50700b2c40 legacy=0x7f50700b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.338+0000 7f5065e00640 1 -- 172.21.15.97:0/2544834005 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507006ce80 con 0x7f507006d960 2024-03-06T07:30:05.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.339+0000 7f5065e00640 1 --2- 172.21.15.97:0/2544834005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f507015f1c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f50580097a0 tx=0x7f505802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945d1fb18a256189 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.339+0000 7f506de00640 1 -- 172.21.15.97:0/2544834005 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058035020 con 0x7f507006d960 2024-03-06T07:30:05.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.339+0000 7f506de00640 1 -- 172.21.15.97:0/2544834005 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058031810 con 0x7f507006d960 2024-03-06T07:30:05.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.339+0000 7f506de00640 1 -- 172.21.15.97:0/2544834005 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058031be0 con 0x7f507006d960 2024-03-06T07:30:05.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 -- 172.21.15.97:0/2544834005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 msgr2=0x7f507015f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 --2- 172.21.15.97:0/2544834005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f507015f1c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f50580097a0 tx=0x7f505802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 -- 172.21.15.97:0/2544834005 shutdown_connections 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 --2- 172.21.15.97:0/2544834005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f507006e2f0 0x7f50700cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 --2- 172.21.15.97:0/2544834005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f507015f1c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 -- 172.21.15.97:0/2544834005 >> 172.21.15.97:0/2544834005 conn(0x7f50700b1940 msgr2=0x7f50700b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 -- 172.21.15.97:0/2544834005 shutdown_connections 2024-03-06T07:30:05.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.340+0000 7f5075400640 1 -- 172.21.15.97:0/2544834005 wait complete. 2024-03-06T07:30:05.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.342+0000 7f5075400640 1 Processor -- start 2024-03-06T07:30:05.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.342+0000 7f5075400640 1 -- start start 2024-03-06T07:30:05.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5075400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f5070158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5075400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f507006e2f0 0x7f507009b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5075400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50700b2c40 0x7f507015f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5075400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507006e900 con 0x7f507006d960 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5075400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507006e780 con 0x7f507006e2f0 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5075400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50700cdfd0 con 0x7f50700b2c40 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f5070158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f5070158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35238/0 (socket says 172.21.15.97:35238) 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 -- 172.21.15.97:0/3704084127 learned_addr learned my addr 172.21.15.97:0/3704084127 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 -- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50700b2c40 msgr2=0x7f507015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506f200640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50700b2c40 0x7f507015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f5065e00640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f507006e2f0 0x7f507009b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50700b2c40 0x7f507015f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 -- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f507006e2f0 msgr2=0x7f507009b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f507006e2f0 0x7f507009b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.343+0000 7f506e800640 1 -- 172.21.15.97:0/3704084127 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50701f7720 con 0x7f507006d960 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.344+0000 7f506e800640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f5070158980 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f506000c3c0 tx=0x7f506000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.344+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060017070 con 0x7f507006d960 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.344+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5060005ce0 con 0x7f507006d960 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.344+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600072f0 con 0x7f507006d960 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.344+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50701621d0 con 0x7f507006d960 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.344+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5070162790 con 0x7f507006d960 2024-03-06T07:30:05.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.345+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507004f330 con 0x7f507006d960 2024-03-06T07:30:05.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.346+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f506001a070 con 0x7f507006d960 2024-03-06T07:30:05.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.347+0000 7f5067400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504c06b4d0 0x7f504c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.347+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5060089600 con 0x7f507006d960 2024-03-06T07:30:05.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.347+0000 7f5065e00640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504c06b4d0 0x7f504c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.348+0000 7f5065e00640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504c06b4d0 0x7f504c06d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.351+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5060058420 con 0x7f507006d960 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:05.584+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f50700b3150 con 0x7f507006d960 2024-03-06T07:30:06.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.121+0000 7f5067400640 1 -- 172.21.15.97:0/3704084127 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5060057d20 con 0x7f507006d960 2024-03-06T07:30:06.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.125+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504c06b4d0 msgr2=0x7f504c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.125+0000 7f5075400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504c06b4d0 0x7f504c06d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058008030 comp rx=0 tx=0).stop 2024-03-06T07:30:06.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 msgr2=0x7f5070158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f5070158980 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f506000c3c0 tx=0x7f506000c880 comp rx=0 tx=0).stop 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 shutdown_connections 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504c06b4d0 0x7f504c06d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50700b2c40 0x7f507015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f507006e2f0 0x7f507009b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 --2- 172.21.15.97:0/3704084127 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f507006d960 0x7f5070158980 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 >> 172.21.15.97:0/3704084127 conn(0x7f50700b1940 msgr2=0x7f507015dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 shutdown_connections 2024-03-06T07:30:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.126+0000 7f5075400640 1 -- 172.21.15.97:0/3704084127 wait complete. 2024-03-06T07:30:06.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:06.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.360+0000 7fc830600640 1 Processor -- start 2024-03-06T07:30:06.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.361+0000 7fc830600640 1 -- start start 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.362+0000 7fc830600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc8280b3b40 0x7fc8280b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.362+0000 7fc830600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc82806b1f0 0x7fc82806b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.362+0000 7fc830600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8280af630 con 0x7fc828071b10 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.362+0000 7fc830600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc82806bf20 con 0x7fc8280b3b40 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.362+0000 7fc830600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8280b4460 con 0x7fc82806b1f0 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc8280b3b40 0x7fc8280b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc8280b3b40 0x7fc8280b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35250/0 (socket says 172.21.15.97:35250) 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82e600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc828071b10 0x7fc828071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60696/0 (socket says 172.21.15.97:60696) 2024-03-06T07:30:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 -- 172.21.15.97:0/2587780280 learned_addr learned my addr 172.21.15.97:0/2587780280 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82d200640 1 --2- 172.21.15.97:0/2587780280 >> v2:172.21.15.135:3300/0 conn(0x7fc82806b1f0 0x7fc82806b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 -- 172.21.15.97:0/2587780280 >> v2:172.21.15.135:3300/0 conn(0x7fc82806b1f0 msgr2=0x7fc82806b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 --2- 172.21.15.97:0/2587780280 >> v2:172.21.15.135:3300/0 conn(0x7fc82806b1f0 0x7fc82806b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 -- 172.21.15.97:0/2587780280 >> v1:172.21.15.97:6789/0 conn(0x7fc828071b10 legacy=0x7fc828071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc827e00640 1 -- 172.21.15.97:0/2587780280 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730411266 0 0) 0x7fc8280af630 con 0x7fc828071b10 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82dc00640 1 -- 172.21.15.97:0/2587780280 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82815e710 con 0x7fc8280b3b40 2024-03-06T07:30:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.363+0000 7fc82d200640 1 --2- 172.21.15.97:0/2587780280 >> v2:172.21.15.135:3300/0 conn(0x7fc82806b1f0 0x7fc82806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.364+0000 7fc82dc00640 1 --2- 172.21.15.97:0/2587780280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8280b3b40 0x7fc8280b3f20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc82806d170 tx=0x7fc81802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306100bf9f6ceac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.364+0000 7fc827e00640 1 -- 172.21.15.97:0/2587780280 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818035020 con 0x7fc8280b3b40 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.364+0000 7fc827e00640 1 -- 172.21.15.97:0/2587780280 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc818031870 con 0x7fc8280b3b40 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.364+0000 7fc827e00640 1 -- 172.21.15.97:0/2587780280 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818031c40 con 0x7fc8280b3b40 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 -- 172.21.15.97:0/2587780280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8280b3b40 msgr2=0x7fc8280b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 --2- 172.21.15.97:0/2587780280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8280b3b40 0x7fc8280b3f20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc82806d170 tx=0x7fc81802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 -- 172.21.15.97:0/2587780280 shutdown_connections 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 --2- 172.21.15.97:0/2587780280 >> v2:172.21.15.135:3300/0 conn(0x7fc82806b1f0 0x7fc82806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 --2- 172.21.15.97:0/2587780280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8280b3b40 0x7fc8280b3f20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 -- 172.21.15.97:0/2587780280 >> 172.21.15.97:0/2587780280 conn(0x7fc8280b2840 msgr2=0x7fc8280b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 -- 172.21.15.97:0/2587780280 shutdown_connections 2024-03-06T07:30:06.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.365+0000 7fc830600640 1 -- 172.21.15.97:0/2587780280 wait complete. 2024-03-06T07:30:06.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.367+0000 7fc830600640 1 Processor -- start 2024-03-06T07:30:06.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.367+0000 7fc830600640 1 -- start start 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc830600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 0x7fc8281f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc830600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc828071b10 0x7fc8281f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc830600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc8280b3b40 0x7fc82815f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc830600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82815efb0 con 0x7fc82806b1f0 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc830600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82815ecd0 con 0x7fc8280b3b40 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc830600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82806bbe0 con 0x7fc828071b10 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc82dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 0x7fc8281f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc82dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 0x7fc8281f7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35264/0 (socket says 172.21.15.97:35264) 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.368+0000 7fc82dc00640 1 -- 172.21.15.97:0/2468893541 learned_addr learned my addr 172.21.15.97:0/2468893541 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc82dc00640 1 -- 172.21.15.97:0/2468893541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc828071b10 msgr2=0x7fc8281f0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc82dc00640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc828071b10 0x7fc8281f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc82dc00640 1 -- 172.21.15.97:0/2468893541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc8280b3b40 msgr2=0x7fc82815f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc82dc00640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc8280b3b40 0x7fc82815f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc82dc00640 1 -- 172.21.15.97:0/2468893541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8281f7e70 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc82dc00640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 0x7fc8281f7930 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc81802b950 tx=0x7fc8180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818040070 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81803a420 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818035040 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.369+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8281f48c0 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.370+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8281f4020 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.371+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8280b4a60 con 0x7fc82806b1f0 2024-03-06T07:30:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.372+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc81804b060 con 0x7fc82806b1f0 2024-03-06T07:30:06.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.372+0000 7fc825600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.373+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc8180ac1b0 con 0x7fc82806b1f0 2024-03-06T07:30:06.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.373+0000 7fc82d200640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.373+0000 7fc82d200640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc80806b4d0 0x7fc80806d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc814006fd0 tx=0x7fc814008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.377+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8280b4a60 con 0x7fc82806b1f0 2024-03-06T07:30:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:06.575+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc82804f3a0 con 0x7fc82806b1f0 2024-03-06T07:30:07.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.130+0000 7fc825600640 1 -- 172.21.15.97:0/2468893541 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc8281f7e70 con 0x7fc82806b1f0 2024-03-06T07:30:07.133 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:07.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.134+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc80806b4d0 msgr2=0x7fc80806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.134+0000 7fc830600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc80806b4d0 0x7fc80806d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc814006fd0 tx=0x7fc814008040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.134+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 msgr2=0x7fc8281f7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.134+0000 7fc830600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 0x7fc8281f7930 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc81802b950 tx=0x7fc8180046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.135+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 shutdown_connections 2024-03-06T07:30:07.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.135+0000 7fc830600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.135+0000 7fc830600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc8280b3b40 0x7fc82815f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.135+0000 7fc830600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc828071b10 0x7fc8281f0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.135+0000 7fc830600640 1 --2- 172.21.15.97:0/2468893541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc82806b1f0 0x7fc8281f7930 secure :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc81802b950 tx=0x7fc8180046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.135+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 >> 172.21.15.97:0/2468893541 conn(0x7fc8280b2840 msgr2=0x7fc82815d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.138+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 shutdown_connections 2024-03-06T07:30:07.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.138+0000 7fc830600640 1 -- 172.21.15.97:0/2468893541 wait complete. 2024-03-06T07:30:07.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:07.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:07.299 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.341+0000 7f011be00640 1 Processor -- start 2024-03-06T07:30:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.341+0000 7f011be00640 1 -- start start 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f011be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0114161770 0x7f01140d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f011be00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f01140d0870 0x7f01140d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f011be00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140d16a0 con 0x7f01140d1210 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f011be00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01140d2f00 con 0x7f01140d0870 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f011be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140b3760 con 0x7f0114161770 2024-03-06T07:30:07.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f0119e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f01140d1210 0x7f01140ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60702/0 (socket says 172.21.15.97:60702) 2024-03-06T07:30:07.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f0119e00640 1 -- 172.21.15.97:0/738842723 learned_addr learned my addr 172.21.15.97:0/738842723 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f0119400640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0114161770 0x7f01140d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.342+0000 7f0118a00640 1 --2- 172.21.15.97:0/738842723 >> v2:172.21.15.135:3300/0 conn(0x7f01140d0870 0x7f01140d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0113e00640 1 -- 172.21.15.97:0/738842723 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349034234 0 0) 0x7f01140d16a0 con 0x7f01140d1210 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0113e00640 1 -- 172.21.15.97:0/738842723 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc003520 con 0x7f01140d1210 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0118a00640 1 -- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0114161770 msgr2=0x7f01140d4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0118a00640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0114161770 0x7f01140d4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0118a00640 1 -- 172.21.15.97:0/738842723 >> v1:172.21.15.97:6789/0 conn(0x7f01140d1210 legacy=0x7f01140ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0118a00640 1 -- 172.21.15.97:0/738842723 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140b4c00 con 0x7f01140d0870 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.343+0000 7f0119400640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0114161770 0x7f01140d4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.344+0000 7f0118a00640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d0870 0x7f01140d0cd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f01000097a0 tx=0x7f010002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4d41b680f598edc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f0113e00640 1 -- 172.21.15.97:0/738842723 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100035020 con 0x7f01140d0870 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f0113e00640 1 -- 172.21.15.97:0/738842723 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100031810 con 0x7f01140d0870 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f0113e00640 1 -- 172.21.15.97:0/738842723 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100031be0 con 0x7f01140d0870 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f011be00640 1 -- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d0870 msgr2=0x7f01140d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f011be00640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d0870 0x7f01140d0cd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f01000097a0 tx=0x7f010002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f011be00640 1 -- 172.21.15.97:0/738842723 shutdown_connections 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f011be00640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d0870 0x7f01140d0cd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f011be00640 1 --2- 172.21.15.97:0/738842723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0114161770 0x7f01140d4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.345+0000 7f011be00640 1 -- 172.21.15.97:0/738842723 >> 172.21.15.97:0/738842723 conn(0x7f01140d3ee0 msgr2=0x7f01140d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.346+0000 7f011be00640 1 -- 172.21.15.97:0/738842723 shutdown_connections 2024-03-06T07:30:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.346+0000 7f011be00640 1 -- 172.21.15.97:0/738842723 wait complete. 2024-03-06T07:30:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.347+0000 7f011be00640 1 Processor -- start 2024-03-06T07:30:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.347+0000 7f011be00640 1 -- start start 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f011be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01140d0870 0x7f011415c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f011be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d1210 0x7f01140cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f011be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 0x7f01140cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f011be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140d2c20 con 0x7f0114161770 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f011be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140d2940 con 0x7f01140d1210 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f011be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140b50d0 con 0x7f01140d0870 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.348+0000 7f0119e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 0x7f01140cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 0x7f01140cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35280/0 (socket says 172.21.15.97:35280) 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 -- 172.21.15.97:0/398135261 learned_addr learned my addr 172.21.15.97:0/398135261 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0118a00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d1210 0x7f01140cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119400640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01140d0870 0x7f011415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 -- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01140d0870 msgr2=0x7f011415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01140d0870 0x7f011415c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 -- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d1210 msgr2=0x7f01140cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d1210 0x7f01140cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 -- 172.21.15.97:0/398135261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140cb2b0 con 0x7f0114161770 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.349+0000 7f0119e00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 0x7f01140cd3e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f010c00cb40 tx=0x7f010c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.350+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c017070 con 0x7f0114161770 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.350+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c00aca0 con 0x7f0114161770 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.350+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c012620 con 0x7f0114161770 2024-03-06T07:30:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.350+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01140cb590 con 0x7f0114161770 2024-03-06T07:30:07.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.350+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01140cbb50 con 0x7f0114161770 2024-03-06T07:30:07.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.351+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011404f330 con 0x7f0114161770 2024-03-06T07:30:07.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.352+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f010c01a070 con 0x7f0114161770 2024-03-06T07:30:07.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.353+0000 7f0111600640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f00f406b4d0 0x7f00f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.353+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f010c089e00 con 0x7f0114161770 2024-03-06T07:30:07.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.353+0000 7f0119400640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f00f406b4d0 0x7f00f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.354+0000 7f0119400640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f00f406b4d0 0x7f00f406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0108009560 tx=0x7f0108009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.359+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f010c058780 con 0x7f0114161770 2024-03-06T07:30:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:07.575+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f01140cad70 con 0x7f0114161770 2024-03-06T07:30:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.141+0000 7f0111600640 1 -- 172.21.15.97:0/398135261 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) v1 ==== 169+0+0 (secure 0 0 0) 0x7f010c05c820 con 0x7f0114161770 2024-03-06T07:30:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:08.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.145+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f00f406b4d0 msgr2=0x7f00f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.145+0000 7f011be00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f00f406b4d0 0x7f00f406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0108009560 tx=0x7f0108009190 comp rx=0 tx=0).stop 2024-03-06T07:30:08.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.145+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 msgr2=0x7f01140cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.145+0000 7f011be00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 0x7f01140cd3e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f010c00cb40 tx=0x7f010c00a060 comp rx=0 tx=0).stop 2024-03-06T07:30:08.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.147+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 shutdown_connections 2024-03-06T07:30:08.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.147+0000 7f011be00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f00f406b4d0 0x7f00f406d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.147+0000 7f011be00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0114161770 0x7f01140cd3e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.147+0000 7f011be00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01140d1210 0x7f01140cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.147+0000 7f011be00640 1 --2- 172.21.15.97:0/398135261 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01140d0870 0x7f011415c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.147+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 >> 172.21.15.97:0/398135261 conn(0x7f01140d3ee0 msgr2=0x7f011415e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.151+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 shutdown_connections 2024-03-06T07:30:08.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.151+0000 7f011be00640 1 -- 172.21.15.97:0/398135261 wait complete. 2024-03-06T07:30:08.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:30:08.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.357+0000 7fef4ca00640 1 Processor -- start 2024-03-06T07:30:08.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.358+0000 7fef4ca00640 1 -- start start 2024-03-06T07:30:08.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef4ca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fef4815be10 0x7fef480ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef4ca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fef4806da40 0x7fef480ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef4ca00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4815c1f0 con 0x7fef480aef20 2024-03-06T07:30:08.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef4ca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef480cd200 con 0x7fef4815be10 2024-03-06T07:30:08.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef4ca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef4806cac0 con 0x7fef4806da40 2024-03-06T07:30:08.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef46800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fef4815be10 0x7fef480ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef46800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fef4815be10 0x7fef480ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35282/0 (socket says 172.21.15.97:35282) 2024-03-06T07:30:08.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.359+0000 7fef46800640 1 -- 172.21.15.97:0/725392601 learned_addr learned my addr 172.21.15.97:0/725392601 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:08.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.360+0000 7fef46800640 1 -- 172.21.15.97:0/725392601 >> v2:172.21.15.135:3300/0 conn(0x7fef4806da40 msgr2=0x7fef480ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.360+0000 7fef46800640 1 --2- 172.21.15.97:0/725392601 >> v2:172.21.15.135:3300/0 conn(0x7fef4806da40 0x7fef480ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.360+0000 7fef46800640 1 -- 172.21.15.97:0/725392601 >> v1:172.21.15.135:6789/0 conn(0x7fef480aef20 legacy=0x7fef4806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.360+0000 7fef46800640 1 -- 172.21.15.97:0/725392601 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4806cda0 con 0x7fef4815be10 2024-03-06T07:30:08.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.361+0000 7fef46800640 1 --2- 172.21.15.97:0/725392601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef480ccc20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef480b1500 tx=0x7fef3802bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba96fe0a33a79b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.361+0000 7fef45e00640 1 -- 172.21.15.97:0/725392601 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3802caa0 con 0x7fef4815be10 2024-03-06T07:30:08.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.361+0000 7fef45e00640 1 -- 172.21.15.97:0/725392601 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3802cc40 con 0x7fef4815be10 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.361+0000 7fef45e00640 1 -- 172.21.15.97:0/725392601 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380343f0 con 0x7fef4815be10 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 -- 172.21.15.97:0/725392601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 msgr2=0x7fef480ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 --2- 172.21.15.97:0/725392601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef480ccc20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef480b1500 tx=0x7fef3802bf00 comp rx=0 tx=0).stop 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 -- 172.21.15.97:0/725392601 shutdown_connections 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 --2- 172.21.15.97:0/725392601 >> v2:172.21.15.135:3300/0 conn(0x7fef4806da40 0x7fef480ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 --2- 172.21.15.97:0/725392601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef480ccc20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 -- 172.21.15.97:0/725392601 >> 172.21.15.97:0/725392601 conn(0x7fef4806afc0 msgr2=0x7fef4806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 -- 172.21.15.97:0/725392601 shutdown_connections 2024-03-06T07:30:08.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.362+0000 7fef4ca00640 1 -- 172.21.15.97:0/725392601 wait complete. 2024-03-06T07:30:08.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.364+0000 7fef4ca00640 1 Processor -- start 2024-03-06T07:30:08.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.364+0000 7fef4ca00640 1 -- start start 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef4ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef4806da40 0x7fef481509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef4ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef480aef20 0x7fef48150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef4ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef4814e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef4ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480cda90 con 0x7fef4815be10 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef4ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4806d080 con 0x7fef480aef20 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef4ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef48071ab0 con 0x7fef4806da40 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef47200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef4814e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef47200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef4814e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35292/0 (socket says 172.21.15.97:35292) 2024-03-06T07:30:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.365+0000 7fef47200640 1 -- 172.21.15.97:0/3093331998 learned_addr learned my addr 172.21.15.97:0/3093331998 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef3de00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef480aef20 0x7fef48150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef47200640 1 -- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef4806da40 msgr2=0x7fef481509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef46800640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef4806da40 0x7fef481509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef47200640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef4806da40 0x7fef481509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef47200640 1 -- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef480aef20 msgr2=0x7fef48150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef47200640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef480aef20 0x7fef48150f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef47200640 1 -- 172.21.15.97:0/3093331998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4814f260 con 0x7fef4815be10 2024-03-06T07:30:08.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef47200640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef4814e650 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fef4000b3f0 tx=0x7fef4000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40015020 con 0x7fef4815be10 2024-03-06T07:30:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef40011850 con 0x7fef4815be10 2024-03-06T07:30:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40011c40 con 0x7fef4815be10 2024-03-06T07:30:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.366+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4814f540 con 0x7fef4815be10 2024-03-06T07:30:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.367+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4814fb00 con 0x7fef4815be10 2024-03-06T07:30:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.368+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef48150020 con 0x7fef4815be10 2024-03-06T07:30:08.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.369+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef40018020 con 0x7fef4815be10 2024-03-06T07:30:08.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.369+0000 7fef3f400640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fef2406b4d0 0x7fef2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.369+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fef40089e40 con 0x7fef4815be10 2024-03-06T07:30:08.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.370+0000 7fef46800640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fef2406b4d0 0x7fef2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.370+0000 7fef46800640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fef2406b4d0 0x7fef2406d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fef38035000 tx=0x7fef38030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.374+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef4001f020 con 0x7fef4815be10 2024-03-06T07:30:08.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.583+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fef4804f330 con 0x7fef4815be10 2024-03-06T07:30:08.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.588+0000 7fef3f400640 1 -- 172.21.15.97:0/3093331998 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v14) v1 ==== 108+0+0 (secure 0 0 0) 0x7fef400172c0 con 0x7fef4815be10 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fef2406b4d0 msgr2=0x7fef2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fef2406b4d0 0x7fef2406d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fef38035000 tx=0x7fef38030000 comp rx=0 tx=0).stop 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 msgr2=0x7fef4814e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef4814e650 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fef4000b3f0 tx=0x7fef4000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 shutdown_connections 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fef2406b4d0 0x7fef2406d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4815be10 0x7fef4814e650 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef480aef20 0x7fef48150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 --2- 172.21.15.97:0/3093331998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef4806da40 0x7fef481509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.591+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 >> 172.21.15.97:0/3093331998 conn(0x7fef4806afc0 msgr2=0x7fef480cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.592+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 shutdown_connections 2024-03-06T07:30:08.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.592+0000 7fef4ca00640 1 -- 172.21.15.97:0/3093331998 wait complete. 2024-03-06T07:30:08.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.815+0000 7fe648600640 1 Processor -- start 2024-03-06T07:30:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.815+0000 7fe648600640 1 -- start start 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.816+0000 7fe648600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe6400b3f70 0x7fe640160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.816+0000 7fe648600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe640070ff0 0x7fe6400ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.816+0000 7fe648600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640071440 con 0x7fe64015fc10 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.816+0000 7fe648600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6400b46c0 con 0x7fe6400b3f70 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.816+0000 7fe648600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6400b49a0 con 0x7fe640070ff0 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe6400b3f70 0x7fe640160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe6400b3f70 0x7fe640160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35302/0 (socket says 172.21.15.97:35302) 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 -- 172.21.15.97:0/669458356 learned_addr learned my addr 172.21.15.97:0/669458356 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 -- 172.21.15.97:0/669458356 >> v2:172.21.15.135:3300/0 conn(0x7fe640070ff0 msgr2=0x7fe6400ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 --2- 172.21.15.97:0/669458356 >> v2:172.21.15.135:3300/0 conn(0x7fe640070ff0 0x7fe6400ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 -- 172.21.15.97:0/669458356 >> v1:172.21.15.135:6789/0 conn(0x7fe64015fc10 legacy=0x7fe64015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.817+0000 7fe645200640 1 -- 172.21.15.97:0/669458356 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640072c90 con 0x7fe6400b3f70 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.818+0000 7fe645200640 1 --2- 172.21.15.97:0/669458356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6400b3f70 0x7fe640160700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe6400cf6a0 tx=0x7fe62c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45177e21fe57294c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe63fe00640 1 -- 172.21.15.97:0/669458356 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c035020 con 0x7fe6400b3f70 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe63fe00640 1 -- 172.21.15.97:0/669458356 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c031870 con 0x7fe6400b3f70 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe63fe00640 1 -- 172.21.15.97:0/669458356 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c031c40 con 0x7fe6400b3f70 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 -- 172.21.15.97:0/669458356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6400b3f70 msgr2=0x7fe640160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 --2- 172.21.15.97:0/669458356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6400b3f70 0x7fe640160700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe6400cf6a0 tx=0x7fe62c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 -- 172.21.15.97:0/669458356 shutdown_connections 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 --2- 172.21.15.97:0/669458356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6400b3f70 0x7fe640160700 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 --2- 172.21.15.97:0/669458356 >> v2:172.21.15.135:3300/0 conn(0x7fe640070ff0 0x7fe6400ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 -- 172.21.15.97:0/669458356 >> 172.21.15.97:0/669458356 conn(0x7fe6400b1940 msgr2=0x7fe6400b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 -- 172.21.15.97:0/669458356 shutdown_connections 2024-03-06T07:30:08.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.819+0000 7fe648600640 1 -- 172.21.15.97:0/669458356 wait complete. 2024-03-06T07:30:08.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.821+0000 7fe648600640 1 Processor -- start 2024-03-06T07:30:08.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.821+0000 7fe648600640 1 -- start start 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.822+0000 7fe648600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe640070ff0 0x7fe6400c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.822+0000 7fe648600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6400b3f70 0x7fe640145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.822+0000 7fe648600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 0x7fe640156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.822+0000 7fe648600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe640072a30 con 0x7fe64015fc10 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.822+0000 7fe648600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6400728b0 con 0x7fe6400b3f70 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.822+0000 7fe648600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64015c2e0 con 0x7fe640070ff0 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 0x7fe640156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 0x7fe640156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35310/0 (socket says 172.21.15.97:35310) 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 -- 172.21.15.97:0/484223281 learned_addr learned my addr 172.21.15.97:0/484223281 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe645200640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6400b3f70 0x7fe640145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 -- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe640070ff0 msgr2=0x7fe6400c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe640070ff0 0x7fe6400c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 -- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6400b3f70 msgr2=0x7fe640145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6400b3f70 0x7fe640145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 -- 172.21.15.97:0/484223281 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64009d560 con 0x7fe64015fc10 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe645200640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6400b3f70 0x7fe640145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.823+0000 7fe646600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 0x7fe640156c30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe63800b970 tx=0x7fe63800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.824+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638019070 con 0x7fe64015fc10 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.824+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63800cb60 con 0x7fe64015fc10 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.824+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638014440 con 0x7fe64015fc10 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.824+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6401e5ba0 con 0x7fe64015fc10 2024-03-06T07:30:08.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.824+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6401e6160 con 0x7fe64015fc10 2024-03-06T07:30:08.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.825+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6400cf7a0 con 0x7fe64015fc10 2024-03-06T07:30:08.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.826+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe63801c070 con 0x7fe64015fc10 2024-03-06T07:30:08.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.827+0000 7fe63d600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe62006b4d0 0x7fe62006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.827+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe638089cc0 con 0x7fe64015fc10 2024-03-06T07:30:08.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.827+0000 7fe645c00640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe62006b4d0 0x7fe62006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.828+0000 7fe645c00640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe62006b4d0 0x7fe62006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe634009650 tx=0x7fe634006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:08.832+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6400cf7a0 con 0x7fe64015fc10 2024-03-06T07:30:09.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.027+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe6400cdf70 con 0x7fe64015fc10 2024-03-06T07:30:09.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.034+0000 7fe63d600640 1 -- 172.21.15.97:0/484223281 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6400cdf70 con 0x7fe64015fc10 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe62006b4d0 msgr2=0x7fe62006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe62006b4d0 0x7fe62006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe634009650 tx=0x7fe634006ba0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 msgr2=0x7fe640156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 0x7fe640156c30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe63800b970 tx=0x7fe63800be30 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 shutdown_connections 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe62006b4d0 0x7fe62006d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe64015fc10 0x7fe640156c30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6400b3f70 0x7fe640145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 --2- 172.21.15.97:0/484223281 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe640070ff0 0x7fe6400c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.036+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 >> 172.21.15.97:0/484223281 conn(0x7fe6400b1940 msgr2=0x7fe6400ace00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.037+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 shutdown_connections 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.037+0000 7fe648600640 1 -- 172.21.15.97:0/484223281 wait complete. 2024-03-06T07:30:09.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:30:09.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.255+0000 7fdb50000640 1 Processor -- start 2024-03-06T07:30:09.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.255+0000 7fdb50000640 1 -- start start 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.257+0000 7fdb50000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48144c00 0x7fdb4806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.257+0000 7fdb50000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdb4815fe30 0x7fdb480ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.257+0000 7fdb50000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4806ee70 con 0x7fdb4815f9d0 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.257+0000 7fdb50000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb4806cef0 con 0x7fdb4815fe30 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.257+0000 7fdb50000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4806d1d0 con 0x7fdb48144c00 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.258+0000 7fdb4cc00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fdb4815f9d0 0x7fdb480ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:40918/0 (socket says 172.21.15.97:40918) 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.258+0000 7fdb4cc00640 1 -- 172.21.15.97:0/1440304611 learned_addr learned my addr 172.21.15.97:0/1440304611 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.258+0000 7fdb4e000640 1 --2- 172.21.15.97:0/1440304611 >> v2:172.21.15.135:3300/0 conn(0x7fdb4815fe30 0x7fdb480ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.258+0000 7fdb4d600640 1 --2- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48144c00 0x7fdb4806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.258+0000 7fdb47e00640 1 -- 172.21.15.97:0/1440304611 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894448569 0 0) 0x7fdb34001110 con 0x7fdb4815f9d0 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.258+0000 7fdb47e00640 1 -- 172.21.15.97:0/1440304611 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30003520 con 0x7fdb4815f9d0 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.259+0000 7fdb4e000640 1 -- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48144c00 msgr2=0x7fdb4806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.259+0000 7fdb4e000640 1 --2- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48144c00 0x7fdb4806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.259+0000 7fdb4e000640 1 -- 172.21.15.97:0/1440304611 >> v1:172.21.15.135:6789/0 conn(0x7fdb4815f9d0 legacy=0x7fdb480ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.259+0000 7fdb4e000640 1 -- 172.21.15.97:0/1440304611 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4806d4b0 con 0x7fdb4815fe30 2024-03-06T07:30:09.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.260+0000 7fdb4e000640 1 --2- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb4815fe30 0x7fdb480ce7b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009d90 tx=0x7fdb4002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790b8fe567fbcd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.260+0000 7fdb47e00640 1 -- 172.21.15.97:0/1440304611 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40039070 con 0x7fdb4815fe30 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.261+0000 7fdb47e00640 1 -- 172.21.15.97:0/1440304611 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4002cc50 con 0x7fdb4815fe30 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.261+0000 7fdb47e00640 1 -- 172.21.15.97:0/1440304611 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40034440 con 0x7fdb4815fe30 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.261+0000 7fdb50000640 1 -- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb4815fe30 msgr2=0x7fdb480ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.261+0000 7fdb50000640 1 --2- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb4815fe30 0x7fdb480ce7b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009d90 tx=0x7fdb4002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.262+0000 7fdb50000640 1 -- 172.21.15.97:0/1440304611 shutdown_connections 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.262+0000 7fdb50000640 1 --2- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb4815fe30 0x7fdb480ce7b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.262+0000 7fdb50000640 1 --2- 172.21.15.97:0/1440304611 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48144c00 0x7fdb4806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.262+0000 7fdb50000640 1 -- 172.21.15.97:0/1440304611 >> 172.21.15.97:0/1440304611 conn(0x7fdb480b1730 msgr2=0x7fdb480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.262+0000 7fdb50000640 1 -- 172.21.15.97:0/1440304611 shutdown_connections 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.262+0000 7fdb50000640 1 -- 172.21.15.97:0/1440304611 wait complete. 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.263+0000 7fdb50000640 1 Processor -- start 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.264+0000 7fdb50000640 1 -- start start 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb50000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48144c00 0x7fdb4815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb50000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 0x7fdb4813fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb50000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb4815fe30 0x7fdb48141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb50000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48160730 con 0x7fdb4815f9d0 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb50000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48160280 con 0x7fdb48144c00 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb50000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4806cb50 con 0x7fdb4815fe30 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 0x7fdb4813fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 0x7fdb4813fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35312/0 (socket says 172.21.15.97:35312) 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 -- 172.21.15.97:0/2881898266 learned_addr learned my addr 172.21.15.97:0/2881898266 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4d600640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48144c00 0x7fdb4815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4e000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb4815fe30 0x7fdb48141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 -- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb4815fe30 msgr2=0x7fdb48141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb4815fe30 0x7fdb48141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 -- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48144c00 msgr2=0x7fdb4815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48144c00 0x7fdb4815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4cc00640 1 -- 172.21.15.97:0/2881898266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4813ca60 con 0x7fdb4815f9d0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4d600640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48144c00 0x7fdb4815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.265+0000 7fdb4e000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb4815fe30 0x7fdb48141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.266+0000 7fdb4cc00640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 0x7fdb4813fa80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400c730 tx=0x7fdb3400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.266+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34017070 con 0x7fdb4815f9d0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.266+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb34005ad0 con 0x7fdb4815f9d0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.266+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4815e410 con 0x7fdb4815f9d0 2024-03-06T07:30:09.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.266+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34007e20 con 0x7fdb4815f9d0 2024-03-06T07:30:09.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.266+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4815e9d0 con 0x7fdb4815f9d0 2024-03-06T07:30:09.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.267+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4804f330 con 0x7fdb4815f9d0 2024-03-06T07:30:09.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.268+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb3401a070 con 0x7fdb4815f9d0 2024-03-06T07:30:09.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.269+0000 7fdb45600640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb2806b4d0 0x7fdb2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.269+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb34089330 con 0x7fdb4815f9d0 2024-03-06T07:30:09.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.269+0000 7fdb4d600640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb2806b4d0 0x7fdb2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.270+0000 7fdb4d600640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb2806b4d0 0x7fdb2806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009a20 tx=0x7fdb3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.274+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb4806ee70 con 0x7fdb4815f9d0 2024-03-06T07:30:09.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:09.546+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7fdb48072890 con 0x7fdb4815f9d0 2024-03-06T07:30:10.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.147+0000 7fdb45600640 1 -- 172.21.15.97:0/2881898266 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v153) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdb34058410 con 0x7fdb4815f9d0 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.150+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb2806b4d0 msgr2=0x7fdb2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.150+0000 7fdb50000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb2806b4d0 0x7fdb2806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009a20 tx=0x7fdb3c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.150+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 msgr2=0x7fdb4813fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.150+0000 7fdb50000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 0x7fdb4813fa80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400c730 tx=0x7fdb3400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 shutdown_connections 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb2806b4d0 0x7fdb2806d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb4815fe30 0x7fdb48141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb4815f9d0 0x7fdb4813fa80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 --2- 172.21.15.97:0/2881898266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48144c00 0x7fdb4815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 >> 172.21.15.97:0/2881898266 conn(0x7fdb480b1730 msgr2=0x7fdb48070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 shutdown_connections 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.151+0000 7fdb50000640 1 -- 172.21.15.97:0/2881898266 wait complete. 2024-03-06T07:30:10.173 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:30:10.173 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:30:10.173 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:10.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:10.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.375+0000 7fc52b200640 1 Processor -- start 2024-03-06T07:30:10.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.375+0000 7fc52b200640 1 -- start start 2024-03-06T07:30:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.376+0000 7fc52b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc52c0aedc0 0x7fc52c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.376+0000 7fc52b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc52c0cc800 0x7fc52c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.376+0000 7fc52b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c15c240 con 0x7fc52c1561f0 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.376+0000 7fc52b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c15ce30 con 0x7fc52c0cc800 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.376+0000 7fc52b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c15c500 con 0x7fc52c0aedc0 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc529e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc52c1561f0 0x7fc52c0ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60714/0 (socket says 172.21.15.97:60714) 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc529e00640 1 -- 172.21.15.97:0/693203212 learned_addr learned my addr 172.21.15.97:0/693203212 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc52a800640 1 --2- 172.21.15.97:0/693203212 >> v2:172.21.15.97:3300/0 conn(0x7fc52c0cc800 0x7fc52c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc528a00640 1 -- 172.21.15.97:0/693203212 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509936058 0 0) 0x7fc52c15c240 con 0x7fc52c1561f0 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc528a00640 1 -- 172.21.15.97:0/693203212 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc510003520 con 0x7fc52c1561f0 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc52a800640 1 -- 172.21.15.97:0/693203212 >> v2:172.21.15.135:3300/0 conn(0x7fc52c0aedc0 msgr2=0x7fc52c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc52a800640 1 --2- 172.21.15.97:0/693203212 >> v2:172.21.15.135:3300/0 conn(0x7fc52c0aedc0 0x7fc52c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc52a800640 1 -- 172.21.15.97:0/693203212 >> v1:172.21.15.97:6789/0 conn(0x7fc52c1561f0 legacy=0x7fc52c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.377+0000 7fc52a800640 1 -- 172.21.15.97:0/693203212 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c15c7e0 con 0x7fc52c0cc800 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.378+0000 7fc52a800640 1 --2- 172.21.15.97:0/693203212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0cc800 0x7fc52c15f830 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc52c068400 tx=0x7fc52402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666c378bfbb663c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc528a00640 1 -- 172.21.15.97:0/693203212 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524039070 con 0x7fc52c0cc800 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc528a00640 1 -- 172.21.15.97:0/693203212 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52402cc80 con 0x7fc52c0cc800 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc528a00640 1 -- 172.21.15.97:0/693203212 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524034640 con 0x7fc52c0cc800 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 -- 172.21.15.97:0/693203212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0cc800 msgr2=0x7fc52c15f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 --2- 172.21.15.97:0/693203212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0cc800 0x7fc52c15f830 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc52c068400 tx=0x7fc52402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 -- 172.21.15.97:0/693203212 shutdown_connections 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 --2- 172.21.15.97:0/693203212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0cc800 0x7fc52c15f830 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 --2- 172.21.15.97:0/693203212 >> v2:172.21.15.135:3300/0 conn(0x7fc52c0aedc0 0x7fc52c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 -- 172.21.15.97:0/693203212 >> 172.21.15.97:0/693203212 conn(0x7fc52c06afc0 msgr2=0x7fc52c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 -- 172.21.15.97:0/693203212 shutdown_connections 2024-03-06T07:30:10.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.379+0000 7fc52b200640 1 -- 172.21.15.97:0/693203212 wait complete. 2024-03-06T07:30:10.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.381+0000 7fc52b200640 1 Processor -- start 2024-03-06T07:30:10.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.381+0000 7fc52b200640 1 -- start start 2024-03-06T07:30:10.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc52b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 0x7fc52c1451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc52b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc52c0cc800 0x7fc52c15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc52b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc52c1561f0 0x7fc52c15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc52b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c1565d0 con 0x7fc52c0aedc0 2024-03-06T07:30:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc52b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0ccc30 con 0x7fc52c0cc800 2024-03-06T07:30:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc52b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c156750 con 0x7fc52c1561f0 2024-03-06T07:30:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc529e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 0x7fc52c1451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc529e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 0x7fc52c1451e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35320/0 (socket says 172.21.15.97:35320) 2024-03-06T07:30:10.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.382+0000 7fc529e00640 1 -- 172.21.15.97:0/1752904996 learned_addr learned my addr 172.21.15.97:0/1752904996 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529e00640 1 -- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc52c1561f0 msgr2=0x7fc52c15b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529400640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc52c0cc800 0x7fc52c15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc52a800640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc52c1561f0 0x7fc52c15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529e00640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc52c1561f0 0x7fc52c15b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529e00640 1 -- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc52c0cc800 msgr2=0x7fc52c15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529e00640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc52c0cc800 0x7fc52c15b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529e00640 1 -- 172.21.15.97:0/1752904996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c145c60 con 0x7fc52c0aedc0 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc529e00640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 0x7fc52c1451e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00c390 tx=0x7fc51c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c017070 con 0x7fc52c0aedc0 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c005ce0 con 0x7fc52c0aedc0 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0072f0 con 0x7fc52c0aedc0 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.383+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c145f40 con 0x7fc52c0aedc0 2024-03-06T07:30:10.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.384+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c145720 con 0x7fc52c0aedc0 2024-03-06T07:30:10.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.385+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c04f330 con 0x7fc52c0aedc0 2024-03-06T07:30:10.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.386+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc51c01a070 con 0x7fc52c0aedc0 2024-03-06T07:30:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.386+0000 7fc522000640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc50c06b500 0x7fc50c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.387+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc51c089240 con 0x7fc52c0aedc0 2024-03-06T07:30:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.387+0000 7fc529400640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc50c06b500 0x7fc50c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.387+0000 7fc529400640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc50c06b500 0x7fc50c06d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc52c154b80 tx=0x7fc518008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.391+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc51c058450 con 0x7fc52c0aedc0 2024-03-06T07:30:10.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.586+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc52c0b2b10 con 0x7fc52c0aedc0 2024-03-06T07:30:10.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.588+0000 7fc522000640 1 -- 172.21.15.97:0/1752904996 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+13010 (secure 0 0 0) 0x7fc51c057d50 con 0x7fc52c0aedc0 2024-03-06T07:30:10.590 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:10.590 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":49,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:08.132214+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.135:6835/1918561213":"2024-03-07T07:30:05.115967+0000","172.21.15.135:6834/1918561213":"2024-03-07T07:30:05.115967+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:10.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.590+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc50c06b500 msgr2=0x7fc50c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.590+0000 7fc52b200640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc50c06b500 0x7fc50c06d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc52c154b80 tx=0x7fc518008040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 msgr2=0x7fc52c1451e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 0x7fc52c1451e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00c390 tx=0x7fc51c00c850 comp rx=0 tx=0).stop 2024-03-06T07:30:10.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 shutdown_connections 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc50c06b500 0x7fc50c06d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc52c1561f0 0x7fc52c15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc52c0cc800 0x7fc52c15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 --2- 172.21.15.97:0/1752904996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc52c0aedc0 0x7fc52c1451e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 >> 172.21.15.97:0/1752904996 conn(0x7fc52c06afc0 msgr2=0x7fc52c071b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 shutdown_connections 2024-03-06T07:30:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.591+0000 7fc52b200640 1 -- 172.21.15.97:0/1752904996 wait complete. 2024-03-06T07:30:10.610 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:10.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:10.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.819+0000 7f6c47e00640 1 Processor -- start 2024-03-06T07:30:10.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.819+0000 7f6c47e00640 1 -- start start 2024-03-06T07:30:10.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.819+0000 7f6c47e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6c4815be10 0x7f6c480ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.819+0000 7f6c47e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6c480aef20 0x7f6c480ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c47e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4815c1f0 con 0x7f6c4806d330 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c47e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c480cd200 con 0x7f6c4815be10 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c47e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4806cac0 con 0x7f6c480aef20 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c46a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6c4815be10 0x7f6c480ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c47400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6c4806d330 0x7f6c48071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60722/0 (socket says 172.21.15.97:60722) 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c47400640 1 -- 172.21.15.97:0/1037635044 learned_addr learned my addr 172.21.15.97:0/1037635044 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c46a00640 1 -- 172.21.15.97:0/1037635044 >> v2:172.21.15.135:3300/0 conn(0x7f6c480aef20 msgr2=0x7f6c480ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c3e000640 1 --2- 172.21.15.97:0/1037635044 >> v2:172.21.15.135:3300/0 conn(0x7f6c480aef20 0x7f6c480ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c46a00640 1 --2- 172.21.15.97:0/1037635044 >> v2:172.21.15.135:3300/0 conn(0x7f6c480aef20 0x7f6c480ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c46a00640 1 -- 172.21.15.97:0/1037635044 >> v1:172.21.15.97:6789/0 conn(0x7f6c4806d330 legacy=0x7f6c48071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c46a00640 1 -- 172.21.15.97:0/1037635044 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4806cda0 con 0x7f6c4815be10 2024-03-06T07:30:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c46000640 1 -- 172.21.15.97:0/1037635044 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995088245 0 0) 0x7f6c4815c1f0 con 0x7f6c4806d330 2024-03-06T07:30:10.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.820+0000 7f6c3e000640 1 --2- 172.21.15.97:0/1037635044 >> v2:172.21.15.135:3300/0 conn(0x7f6c480aef20 0x7f6c480ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.821+0000 7f6c46a00640 1 --2- 172.21.15.97:0/1037635044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ccc20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6c4806ea00 tx=0x7f6c3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22a4fbfa70dd664 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c46000640 1 -- 172.21.15.97:0/1037635044 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38039070 con 0x7f6c4815be10 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c46000640 1 -- 172.21.15.97:0/1037635044 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3802cc80 con 0x7f6c4815be10 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c46000640 1 -- 172.21.15.97:0/1037635044 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38034660 con 0x7f6c4815be10 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 -- 172.21.15.97:0/1037635044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 msgr2=0x7f6c480ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 --2- 172.21.15.97:0/1037635044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ccc20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6c4806ea00 tx=0x7f6c3802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 -- 172.21.15.97:0/1037635044 shutdown_connections 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 --2- 172.21.15.97:0/1037635044 >> v2:172.21.15.135:3300/0 conn(0x7f6c480aef20 0x7f6c480ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 --2- 172.21.15.97:0/1037635044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ccc20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 -- 172.21.15.97:0/1037635044 >> 172.21.15.97:0/1037635044 conn(0x7f6c4806afc0 msgr2=0x7f6c4806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.822+0000 7f6c47e00640 1 -- 172.21.15.97:0/1037635044 shutdown_connections 2024-03-06T07:30:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.823+0000 7f6c47e00640 1 -- 172.21.15.97:0/1037635044 wait complete. 2024-03-06T07:30:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.824+0000 7f6c47e00640 1 Processor -- start 2024-03-06T07:30:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.824+0000 7f6c47e00640 1 -- start start 2024-03-06T07:30:10.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c4806d330 0x7f6c480add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c480aef20 0x7f6c48156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480cda90 con 0x7f6c4815be10 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4806d080 con 0x7f6c480aef20 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4806d7a0 con 0x7f6c4806d330 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ab070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35338/0 (socket says 172.21.15.97:35338) 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.825+0000 7f6c47400640 1 -- 172.21.15.97:0/2939125176 learned_addr learned my addr 172.21.15.97:0/2939125176 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c46a00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c4806d330 0x7f6c480add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c47400640 1 -- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c4806d330 msgr2=0x7f6c480add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c3e000640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c480aef20 0x7f6c48156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c47400640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c4806d330 0x7f6c480add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c47400640 1 -- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c480aef20 msgr2=0x7f6c48156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c47400640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c480aef20 0x7f6c48156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c47400640 1 -- 172.21.15.97:0/2939125176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c481601d0 con 0x7f6c4815be10 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c47400640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ab070 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000c450 tx=0x7f6c4000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40018070 con 0x7f6c4815be10 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40007500 con 0x7f6c4815be10 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.826+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40004a20 con 0x7f6c4815be10 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.827+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c480a9580 con 0x7f6c4815be10 2024-03-06T07:30:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.827+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4815fef0 con 0x7f6c4815be10 2024-03-06T07:30:10.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.828+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4804f330 con 0x7f6c4815be10 2024-03-06T07:30:10.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.829+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c4001b070 con 0x7f6c4815be10 2024-03-06T07:30:10.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.829+0000 7f6c3f400640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6c2406b4d0 0x7f6c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.830+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6c400890a0 con 0x7f6c4815be10 2024-03-06T07:30:10.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.830+0000 7f6c46a00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6c2406b4d0 0x7f6c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.830+0000 7f6c46a00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6c2406b4d0 0x7f6c2406d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6c38036000 tx=0x7f6c38030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:10.834+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c40058290 con 0x7f6c4815be10 2024-03-06T07:30:11.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.025+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c480aff00 con 0x7f6c4815be10 2024-03-06T07:30:11.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.026+0000 7f6c3f400640 1 -- 172.21.15.97:0/2939125176 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c40057b90 con 0x7f6c4815be10 2024-03-06T07:30:11.028 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6c2406b4d0 msgr2=0x7f6c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6c2406b4d0 0x7f6c2406d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6c38036000 tx=0x7f6c38030000 comp rx=0 tx=0).stop 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 msgr2=0x7f6c480ab070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ab070 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000c450 tx=0x7f6c4000c910 comp rx=0 tx=0).stop 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 shutdown_connections 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6c2406b4d0 0x7f6c2406d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c4815be10 0x7f6c480ab070 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c480aef20 0x7f6c48156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 --2- 172.21.15.97:0/2939125176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c4806d330 0x7f6c480add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 >> 172.21.15.97:0/2939125176 conn(0x7f6c4806afc0 msgr2=0x7f6c480cf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 shutdown_connections 2024-03-06T07:30:11.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.028+0000 7f6c47e00640 1 -- 172.21.15.97:0/2939125176 wait complete. 2024-03-06T07:30:11.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:30:11.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.239+0000 7fcd31600640 1 Processor -- start 2024-03-06T07:30:11.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.239+0000 7fcd31600640 1 -- start start 2024-03-06T07:30:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.240+0000 7fcd31600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd2c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.240+0000 7fcd31600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c15b090 con 0x7fcd2c0b4680 2024-03-06T07:30:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.240+0000 7fcd31600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0b2a30 con 0x7fcd2c15acb0 2024-03-06T07:30:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.240+0000 7fcd31600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0b4360 con 0x7fcd2c06ec00 2024-03-06T07:30:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd2a000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcd2c0b4680 0x7fcd2c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60726/0 (socket says 172.21.15.97:60726) 2024-03-06T07:30:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd2a000640 1 -- 172.21.15.97:0/273219965 learned_addr learned my addr 172.21.15.97:0/273219965 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:11.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945740763 0 0) 0x7fcd2c15b090 con 0x7fcd2c0b4680 2024-03-06T07:30:11.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd10003520 con 0x7fcd2c0b4680 2024-03-06T07:30:11.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd2b400640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd2c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186030874 0 0) 0x7fcd10003520 con 0x7fcd2c0b4680 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 msgr2=0x7fcd2c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd2c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 >> v1:172.21.15.135:6789/0 conn(0x7fcd2c15acb0 legacy=0x7fcd2c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c0b2cf0 con 0x7fcd2c0b4680 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.241+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fcd18001670 con 0x7fcd2c0b4680 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 >> v1:172.21.15.97:6789/0 conn(0x7fcd2c0b4680 legacy=0x7fcd2c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd100039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd10004060 0x7fcd10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004d40 0x7fcd100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0b4360 con 0x7fcd10004060 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1000bc00 con 0x7fcd10004d40 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1000e010 con 0x7fcd2c06ec00 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2a000640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd10004060 0x7fcd10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2aa00640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd100039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2a000640 1 -- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 msgr2=0x7fcd100039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2b400640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004d40 0x7fcd100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2a000640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd100039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2a000640 1 -- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004d40 msgr2=0x7fcd100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2a000640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004d40 0x7fcd100051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2a000640 1 -- 172.21.15.97:0/273219965 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd10003520 con 0x7fcd10004060 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.242+0000 7fcd2aa00640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c06ec00 0x7fcd100039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.243+0000 7fcd2a000640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd10004060 0x7fcd10004800 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800c110 tx=0x7fcd10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa26638ff40f4a5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.243+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18007c40 con 0x7fcd10004060 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.243+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd18030070 con 0x7fcd10004060 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.243+0000 7fcd29600640 1 -- 172.21.15.97:0/273219965 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18005d70 con 0x7fcd10004060 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd31600640 1 -- 172.21.15.97:0/273219965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd10004060 msgr2=0x7fcd10004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd31600640 1 --2- 172.21.15.97:0/273219965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd10004060 0x7fcd10004800 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800c110 tx=0x7fcd10005830 comp rx=0 tx=0).stop 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd2b400640 1 -- 172.21.15.97:0/273219965 reap_dead start 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd31600640 1 -- 172.21.15.97:0/273219965 shutdown_connections 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd31600640 1 -- 172.21.15.97:0/273219965 >> 172.21.15.97:0/273219965 conn(0x7fcd2c0b1730 msgr2=0x7fcd2c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd31600640 1 -- 172.21.15.97:0/273219965 shutdown_connections 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.244+0000 7fcd31600640 1 -- 172.21.15.97:0/273219965 wait complete. 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.246+0000 7fcd31600640 1 Processor -- start 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.246+0000 7fcd31600640 1 -- start start 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd31600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004060 0x7fcd2c13ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd31600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c13cb30 0x7fcd2c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd31600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 0x7fcd2c151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd31600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0b4010 con 0x7fcd2c151280 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd31600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0b32b0 con 0x7fcd10004060 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd31600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c072340 con 0x7fcd2c13cb30 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 0x7fcd2c151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 0x7fcd2c151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35362/0 (socket says 172.21.15.97:35362) 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 -- 172.21.15.97:0/2781127440 learned_addr learned my addr 172.21.15.97:0/2781127440 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2a000640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c13cb30 0x7fcd2c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 -- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c13cb30 msgr2=0x7fcd2c1588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2aa00640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004060 0x7fcd2c13ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c13cb30 0x7fcd2c1588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 -- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004060 msgr2=0x7fcd2c13ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004060 0x7fcd2c13ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.247+0000 7fcd2b400640 1 -- 172.21.15.97:0/2781127440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c159150 con 0x7fcd2c151280 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.248+0000 7fcd2b400640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 0x7fcd2c151700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0cf970 tx=0x7fcd2400e510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.248+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400ee20 con 0x7fcd2c151280 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.248+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2401a070 con 0x7fcd2c151280 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.248+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24015650 con 0x7fcd2c151280 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.248+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c159430 con 0x7fcd2c151280 2024-03-06T07:30:11.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.248+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c1599f0 con 0x7fcd2c151280 2024-03-06T07:30:11.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.249+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c04f330 con 0x7fcd2c151280 2024-03-06T07:30:11.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.250+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd24012030 con 0x7fcd2c151280 2024-03-06T07:30:11.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.251+0000 7fcd22a00640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcd0c06b4d0 0x7fcd0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.251+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcd2408a500 con 0x7fcd2c151280 2024-03-06T07:30:11.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.251+0000 7fcd2aa00640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcd0c06b4d0 0x7fcd0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.252+0000 7fcd2aa00640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcd0c06b4d0 0x7fcd0c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0015f0 tx=0x7fcd1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.256+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd24010040 con 0x7fcd2c151280 2024-03-06T07:30:11.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:11.524+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fcd2c06ce60 con 0x7fcd2c151280 2024-03-06T07:30:12.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.162+0000 7fcd22a00640 1 -- 172.21.15.97:0/2781127440 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v155) v1 ==== 147+0+0 (secure 0 0 0) 0x7fcd24059710 con 0x7fcd2c151280 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcd0c06b4d0 msgr2=0x7fcd0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcd0c06b4d0 0x7fcd0c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0015f0 tx=0x7fcd1c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 msgr2=0x7fcd2c151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 0x7fcd2c151700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0cf970 tx=0x7fcd2400e510 comp rx=0 tx=0).stop 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 shutdown_connections 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcd0c06b4d0 0x7fcd0c06d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd2c151280 0x7fcd2c151700 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd2c13cb30 0x7fcd2c1588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 --2- 172.21.15.97:0/2781127440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd10004060 0x7fcd2c13ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.165+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 >> 172.21.15.97:0/2781127440 conn(0x7fcd2c0b1730 msgr2=0x7fcd2c0ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.166+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 shutdown_connections 2024-03-06T07:30:12.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.166+0000 7fcd31600640 1 -- 172.21.15.97:0/2781127440 wait complete. 2024-03-06T07:30:12.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:12.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.386+0000 7f558f600640 1 Processor -- start 2024-03-06T07:30:12.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.386+0000 7f558f600640 1 -- start start 2024-03-06T07:30:12.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558f600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55880b3df0 0x7f55880727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3650 0x7f55880b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558f600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880b4710 con 0x7f55880cda90 2024-03-06T07:30:12.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558f600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5588072310 con 0x7f55880b3df0 2024-03-06T07:30:12.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880719e0 con 0x7f55880b3650 2024-03-06T07:30:12.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f5587e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3650 0x7f55880b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558cc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55880b3df0 0x7f55880727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558d600640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f55880cda90 0x7f558806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:40924/0 (socket says 172.21.15.97:40924) 2024-03-06T07:30:12.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.388+0000 7f558d600640 1 -- 172.21.15.97:0/821870260 learned_addr learned my addr 172.21.15.97:0/821870260 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:12.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.389+0000 7f558cc00640 1 -- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3650 msgr2=0x7f55880b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.389+0000 7f558cc00640 1 --2- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3650 0x7f55880b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.389+0000 7f558cc00640 1 -- 172.21.15.97:0/821870260 >> v1:172.21.15.135:6789/0 conn(0x7f55880cda90 legacy=0x7f558806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.389+0000 7f558cc00640 1 -- 172.21.15.97:0/821870260 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588071cc0 con 0x7f55880b3df0 2024-03-06T07:30:12.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558cc00640 1 --2- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3df0 0x7f55880727a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f55780097a0 tx=0x7f557802dac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5666272e3f957e2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f5587400640 1 -- 172.21.15.97:0/821870260 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557802f880 con 0x7f55880b3df0 2024-03-06T07:30:12.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f5587400640 1 -- 172.21.15.97:0/821870260 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557802fa20 con 0x7f55880b3df0 2024-03-06T07:30:12.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f5587400640 1 -- 172.21.15.97:0/821870260 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557802fdf0 con 0x7f55880b3df0 2024-03-06T07:30:12.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 -- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3df0 msgr2=0x7f55880727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 --2- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3df0 0x7f55880727a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f55780097a0 tx=0x7f557802dac0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 -- 172.21.15.97:0/821870260 shutdown_connections 2024-03-06T07:30:12.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 --2- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3650 0x7f55880b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 --2- 172.21.15.97:0/821870260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3df0 0x7f55880727a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 -- 172.21.15.97:0/821870260 >> 172.21.15.97:0/821870260 conn(0x7f55880b1940 msgr2=0x7f55880b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 -- 172.21.15.97:0/821870260 shutdown_connections 2024-03-06T07:30:12.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.391+0000 7f558f600640 1 -- 172.21.15.97:0/821870260 wait complete. 2024-03-06T07:30:12.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.393+0000 7f558f600640 1 Processor -- start 2024-03-06T07:30:12.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.393+0000 7f558f600640 1 -- start start 2024-03-06T07:30:12.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558f600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3650 0x7f55881f8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3df0 0x7f55881f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558f600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 0x7f55880ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558f600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558815b240 con 0x7f55880cda90 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558f600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880b3b30 con 0x7f55880b3650 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558815b3c0 con 0x7f55880b3df0 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 0x7f55880ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 0x7f55880ae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35370/0 (socket says 172.21.15.97:35370) 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.394+0000 7f558d600640 1 -- 172.21.15.97:0/2884070039 learned_addr learned my addr 172.21.15.97:0/2884070039 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558cc00640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3650 0x7f55881f8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f5587e00640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3df0 0x7f55881f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558d600640 1 -- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3df0 msgr2=0x7f55881f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558d600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3df0 0x7f55881f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558d600640 1 -- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3650 msgr2=0x7f55881f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558d600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3650 0x7f55881f8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558d600640 1 -- 172.21.15.97:0/2884070039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558809d740 con 0x7f55880cda90 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f5587e00640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3df0 0x7f55881f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:12.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558d600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 0x7f55880ae460 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f558000caa0 tx=0x7f558000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580017070 con 0x7f55880cda90 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558000ac00 con 0x7f55880cda90 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580012400 con 0x7f55880cda90 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55881f8790 con 0x7f55880cda90 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.395+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558815ba30 con 0x7f55880cda90 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.396+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558804f330 con 0x7f55880cda90 2024-03-06T07:30:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.397+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f558001a070 con 0x7f55880cda90 2024-03-06T07:30:12.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.398+0000 7f5584c00640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f556806b500 0x7f556806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.398+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5580089b50 con 0x7f55880cda90 2024-03-06T07:30:12.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.398+0000 7f558cc00640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f556806b500 0x7f556806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.399+0000 7f558cc00640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f556806b500 0x7f556806d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5578038000 tx=0x7f557803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.403+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55881f8790 con 0x7f55880cda90 2024-03-06T07:30:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.584+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f558815c770 con 0x7f55880cda90 2024-03-06T07:30:12.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.592+0000 7f5584c00640 1 -- 172.21.15.97:0/2884070039 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f558809d740 con 0x7f55880cda90 2024-03-06T07:30:12.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f556806b500 msgr2=0x7f556806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f556806b500 0x7f556806d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5578038000 tx=0x7f557803b040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 msgr2=0x7f55880ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 0x7f55880ae460 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f558000caa0 tx=0x7f558000cf60 comp rx=0 tx=0).stop 2024-03-06T07:30:12.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 shutdown_connections 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f556806b500 0x7f556806d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55880cda90 0x7f55880ae460 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55880b3df0 0x7f55881f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 --2- 172.21.15.97:0/2884070039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55880b3650 0x7f55881f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 >> 172.21.15.97:0/2884070039 conn(0x7f55880b1940 msgr2=0x7f55880affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 shutdown_connections 2024-03-06T07:30:12.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:12.594+0000 7f558f600640 1 -- 172.21.15.97:0/2884070039 wait complete. 2024-03-06T07:30:12.614 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:12.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:12.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:12.839 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:12.839 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:12.840 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:12.897 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:12.897 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:30:12.898 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:12.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:12.955 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:12.955 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:13.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:13.013 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:30:13.013 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:13.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:13.067 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:13.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:13.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.285+0000 7fd2daa00640 1 Processor -- start 2024-03-06T07:30:13.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.285+0000 7fd2daa00640 1 -- start start 2024-03-06T07:30:13.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2daa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd2d41561f0 0x7fd2d40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2daa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2d40cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2daa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d415c240 con 0x7fd2d40cc800 2024-03-06T07:30:13.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2daa00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d415ce30 con 0x7fd2d41561f0 2024-03-06T07:30:13.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2daa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d415c500 con 0x7fd2d40aebf0 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2da000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd2d40cc800 0x7fd2d40cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60742/0 (socket says 172.21.15.97:60742) 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2da000640 1 -- 172.21.15.97:0/2927128138 learned_addr learned my addr 172.21.15.97:0/2927128138 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207136546 0 0) 0x7fd2d415c240 con 0x7fd2d40cc800 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc003520 con 0x7fd2d40cc800 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2d9600640 1 --2- 172.21.15.97:0/2927128138 >> v2:172.21.15.135:3300/0 conn(0x7fd2d41561f0 0x7fd2d40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.287+0000 7fd2d8c00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2d40cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1789184032 0 0) 0x7fd2bc003520 con 0x7fd2d40cc800 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 msgr2=0x7fd2d40cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2d40cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 >> v2:172.21.15.135:3300/0 conn(0x7fd2d41561f0 msgr2=0x7fd2d40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 --2- 172.21.15.97:0/2927128138 >> v2:172.21.15.135:3300/0 conn(0x7fd2d41561f0 0x7fd2d40ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d415c7e0 con 0x7fd2d40cc800 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fd2cc0019f0 con 0x7fd2d40cc800 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d9600640 1 --2- 172.21.15.97:0/2927128138 >> v2:172.21.15.135:3300/0 conn(0x7fd2d41561f0 0x7fd2d40ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:13.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 >> v1:172.21.15.97:6789/0 conn(0x7fd2d40cc800 legacy=0x7fd2d40cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2bc004d40 0x7fd2bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d415ce30 con 0x7fd2bc004d40 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d415c500 con 0x7fd2bc0039b0 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00ba70 con 0x7fd2d40aebf0 2024-03-06T07:30:13.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.288+0000 7fd2da000640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2bc004d40 0x7fd2bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2d8c00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2d9600640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2da000640 1 -- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 msgr2=0x7fd2bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2da000640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2da000640 1 -- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2bc0039b0 msgr2=0x7fd2bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2da000640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2da000640 1 -- 172.21.15.97:0/2927128138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc003520 con 0x7fd2bc004d40 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.289+0000 7fd2d8c00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2d40aebf0 0x7fd2bc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.290+0000 7fd2da000640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2bc004d40 0x7fd2bc0051c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00a8a0 tx=0x7fd2bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1600d0e1c7205f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.290+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc00a4c0 con 0x7fd2bc004d40 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.290+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc034630 con 0x7fd2bc004d40 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.290+0000 7fd2d3e00640 1 -- 172.21.15.97:0/2927128138 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc034a20 con 0x7fd2bc004d40 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2daa00640 1 -- 172.21.15.97:0/2927128138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2bc004d40 msgr2=0x7fd2bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2daa00640 1 --2- 172.21.15.97:0/2927128138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2bc004d40 0x7fd2bc0051c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00a8a0 tx=0x7fd2bc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2da000640 1 -- 172.21.15.97:0/2927128138 reap_dead start 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2daa00640 1 -- 172.21.15.97:0/2927128138 shutdown_connections 2024-03-06T07:30:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2daa00640 1 -- 172.21.15.97:0/2927128138 >> 172.21.15.97:0/2927128138 conn(0x7fd2d406afc0 msgr2=0x7fd2d406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2daa00640 1 -- 172.21.15.97:0/2927128138 shutdown_connections 2024-03-06T07:30:13.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.291+0000 7fd2daa00640 1 -- 172.21.15.97:0/2927128138 wait complete. 2024-03-06T07:30:13.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.293+0000 7fd2daa00640 1 Processor -- start 2024-03-06T07:30:13.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.293+0000 7fd2daa00640 1 -- start start 2024-03-06T07:30:13.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2daa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2bc004d40 0x7fd2d414ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2daa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2d414f230 0x7fd2d415e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2daa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 0x7fd2d415eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2daa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d41565d0 con 0x7fd2d41f3d80 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2daa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40ccc90 con 0x7fd2d414f230 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2daa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4156750 con 0x7fd2bc004d40 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 0x7fd2d415eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 0x7fd2d415eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35402/0 (socket says 172.21.15.97:35402) 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 -- 172.21.15.97:0/3192183890 learned_addr learned my addr 172.21.15.97:0/3192183890 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 -- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2bc004d40 msgr2=0x7fd2d414ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2d9600640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2bc004d40 0x7fd2d414ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2d8c00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2d414f230 0x7fd2d415e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2bc004d40 0x7fd2d414ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 -- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2d414f230 msgr2=0x7fd2d415e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2d414f230 0x7fd2d415e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.294+0000 7fd2da000640 1 -- 172.21.15.97:0/3192183890 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d41f8020 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.295+0000 7fd2da000640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 0x7fd2d415eaa0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc000f80 tx=0x7fd2d415cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.295+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc039040 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.295+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc034430 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.295+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d414fdd0 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.295+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc00a4c0 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.295+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d4150390 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.296+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d414fbf0 con 0x7fd2d41f3d80 2024-03-06T07:30:13.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.297+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2cc045020 con 0x7fd2d41f3d80 2024-03-06T07:30:13.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.298+0000 7fd2d1600640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2b406b500 0x7fd2b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.298+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd2cc0ab370 con 0x7fd2d41f3d80 2024-03-06T07:30:13.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.298+0000 7fd2d9600640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2b406b500 0x7fd2b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.299+0000 7fd2d9600640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2b406b500 0x7fd2b406d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006fd0 tx=0x7fd2c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.302+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2d414fbf0 con 0x7fd2d41f3d80 2024-03-06T07:30:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.536+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2d404f330 con 0x7fd2d41f3d80 2024-03-06T07:30:13.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.537+0000 7fd2d1600640 1 -- 172.21.15.97:0/3192183890 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fd2cc07a4d0 con 0x7fd2d41f3d80 2024-03-06T07:30:13.539 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:13.539 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4224886203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":4224886203},{"type":"v1","addr":"172.21.15.135:6835","nonce":4224886203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/703786362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":703786362},{"type":"v1","addr":"172.21.15.97:6836","nonce":703786362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5493,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1250630411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1250630411},{"type":"v1","addr":"172.21.15.97:6837","nonce":1250630411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2848258164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2848258164},{"type":"v1","addr":"172.21.15.135:6833","nonce":2848258164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-03-06T07:30:13.539 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-06T07:30:13.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2b406b500 msgr2=0x7fd2b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2b406b500 0x7fd2b406d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006fd0 tx=0x7fd2c8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:13.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 msgr2=0x7fd2d415eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 0x7fd2d415eaa0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc000f80 tx=0x7fd2d415cda0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 shutdown_connections 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2b406b500 0x7fd2b406d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2d41f3d80 0x7fd2d415eaa0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2d414f230 0x7fd2d415e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 --2- 172.21.15.97:0/3192183890 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2bc004d40 0x7fd2d414ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 >> 172.21.15.97:0/3192183890 conn(0x7fd2d406afc0 msgr2=0x7fd2d415b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 shutdown_connections 2024-03-06T07:30:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.540+0000 7fd2daa00640 1 -- 172.21.15.97:0/3192183890 wait complete. 2024-03-06T07:30:13.561 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:13.561 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:13.561 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:13.561 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:13.561 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:13.562 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:13.562 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:13.562 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:13.562 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:13.562 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:13.562 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:13.562 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:13.563 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:13.563 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:30:13.563 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:13.563 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:13.578 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:13.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:13.579 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:13.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:13.582 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:13.583 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:13.584 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:13.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:13.623 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:13.624 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:13.629 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:13.629 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:13.631 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:13.631 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:13.637 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:13.637 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:13.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:13.775 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:30:13.776 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:30:13.791 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:30:13.792 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:30:13.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.815+0000 7f0457200640 1 Processor -- start 2024-03-06T07:30:13.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.815+0000 7f0457200640 1 -- start start 2024-03-06T07:30:13.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.816+0000 7f0457200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f045015e140 0x7f04500b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0457200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04500b3960 0x7f045006d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0457200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0457200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045015e520 con 0x7f04500b3960 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0457200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04500b3e00 con 0x7f045015e140 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0457200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04500b40e0 con 0x7f045006d820 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04500b3960 0x7f045006d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f04500b3960 0x7f045006d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35454/0 (socket says 172.21.15.97:35454) 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 -- 172.21.15.97:0/1786961255 learned_addr learned my addr 172.21.15.97:0/1786961255 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044fe00640 1 --2- 172.21.15.97:0/1786961255 >> v2:172.21.15.135:3300/0 conn(0x7f045015e140 0x7f04500b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0455200640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 -- 172.21.15.97:0/1786961255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 msgr2=0x7f045006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045006bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 -- 172.21.15.97:0/1786961255 >> v2:172.21.15.135:3300/0 conn(0x7f045015e140 msgr2=0x7f04500b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 --2- 172.21.15.97:0/1786961255 >> v2:172.21.15.135:3300/0 conn(0x7f045015e140 0x7f04500b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f044f400640 1 -- 172.21.15.97:0/1786961255 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04500b43c0 con 0x7f04500b3960 2024-03-06T07:30:13.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.817+0000 7f0455200640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.818+0000 7f044f400640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045006d2e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0450072ba0 tx=0x7f043c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d49624e0485bad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f044ea00640 1 -- 172.21.15.97:0/1786961255 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c035020 con 0x7f04500b3960 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f044ea00640 1 -- 172.21.15.97:0/1786961255 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c02d040 con 0x7f04500b3960 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f044ea00640 1 -- 172.21.15.97:0/1786961255 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c031a80 con 0x7f04500b3960 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 -- 172.21.15.97:0/1786961255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 msgr2=0x7f045006d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045006d2e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0450072ba0 tx=0x7f043c02b900 comp rx=0 tx=0).stop 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 -- 172.21.15.97:0/1786961255 shutdown_connections 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 --2- 172.21.15.97:0/1786961255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045006d2e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 --2- 172.21.15.97:0/1786961255 >> v2:172.21.15.135:3300/0 conn(0x7f045015e140 0x7f04500b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 -- 172.21.15.97:0/1786961255 >> 172.21.15.97:0/1786961255 conn(0x7f0450071380 msgr2=0x7f0450071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 -- 172.21.15.97:0/1786961255 shutdown_connections 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.819+0000 7f0457200640 1 -- 172.21.15.97:0/1786961255 wait complete. 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.821+0000 7f0457200640 1 Processor -- start 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.821+0000 7f0457200640 1 -- start start 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f0457200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045014f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f0457200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f0457200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f045015e140 0x7f04501f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f0457200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0450161e20 con 0x7f04500b3960 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f0457200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04500b4980 con 0x7f045015e140 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f0457200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0450161fa0 con 0x7f045006d820 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f044f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f044f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045014ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35458/0 (socket says 172.21.15.97:35458) 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.822+0000 7f044f400640 1 -- 172.21.15.97:0/264996025 learned_addr learned my addr 172.21.15.97:0/264996025 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044fe00640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045014f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f0455200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f045015e140 0x7f04501f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044f400640 1 -- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 msgr2=0x7f045014f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044f400640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045014f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044f400640 1 -- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f045015e140 msgr2=0x7f04501f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044f400640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f045015e140 0x7f04501f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044f400640 1 -- 172.21.15.97:0/264996025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045015fd00 con 0x7f04500b3960 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044fe00640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.823+0000 7f044f400640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045014ea90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f043c02b940 tx=0x7f043c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.824+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c040070 con 0x7f04500b3960 2024-03-06T07:30:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.824+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c0399f0 con 0x7f04500b3960 2024-03-06T07:30:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.824+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c035040 con 0x7f04500b3960 2024-03-06T07:30:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.824+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04500cda20 con 0x7f04500b3960 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.824+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04500cdfe0 con 0x7f04500b3960 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.825+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045004f330 con 0x7f04500b3960 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.826+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f043c04b060 con 0x7f04500b3960 2024-03-06T07:30:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.827+0000 7f0447e00640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f043006b4d0 0x7f043006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.827+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f043c0ab900 con 0x7f04500b3960 2024-03-06T07:30:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.827+0000 7f044fe00640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f043006b4d0 0x7f043006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.828+0000 7f044fe00640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f043006b4d0 0x7f043006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0440009af0 tx=0x7f0440008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:13.831+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f045015fd00 con 0x7f04500b3960 2024-03-06T07:30:14.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:14.042+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f04500ccbe0 con 0x7f04500b3960 2024-03-06T07:30:15.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.200+0000 7f0447e00640 1 -- 172.21.15.97:0/264996025 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) v1 ==== 97+0+0 (secure 0 0 0) 0x7f043c07aaf0 con 0x7f04500b3960 2024-03-06T07:30:15.202 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:30:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f043006b4d0 msgr2=0x7f043006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f043006b4d0 0x7f043006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0440009af0 tx=0x7f0440008040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 msgr2=0x7f045014ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045014ea90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f043c02b940 tx=0x7f043c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 shutdown_connections 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f043006b4d0 0x7f043006d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f045015e140 0x7f04501f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04500b3960 0x7f045014ea90 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 --2- 172.21.15.97:0/264996025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f045006d820 0x7f045014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.204+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 >> 172.21.15.97:0/264996025 conn(0x7f0450071380 msgr2=0x7f04500adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.205+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 shutdown_connections 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.205+0000 7f0457200640 1 -- 172.21.15.97:0/264996025 wait complete. 2024-03-06T07:30:15.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.434+0000 7effa5600640 1 Processor -- start 2024-03-06T07:30:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.434+0000 7effa5600640 1 -- start start 2024-03-06T07:30:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7effa5600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7effa01620c0 0x7effa00b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7effa5600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa0161900 con 0x7effa00b40c0 2024-03-06T07:30:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7effa5600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa015ebd0 con 0x7effa00ae670 2024-03-06T07:30:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7effa5600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa015eeb0 con 0x7effa01620c0 2024-03-06T07:30:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7eff9fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7effa00b40c0 0x7effa00b44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39038/0 (socket says 172.21.15.97:39038) 2024-03-06T07:30:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7eff9fe00640 1 -- 172.21.15.97:0/689986553 learned_addr learned my addr 172.21.15.97:0/689986553 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875674350 0 0) 0x7effa0161900 con 0x7effa00b40c0 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff84003520 con 0x7effa00b40c0 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.436+0000 7effa4c00640 1 --2- 172.21.15.97:0/689986553 >> v2:172.21.15.135:3300/0 conn(0x7effa01620c0 0x7effa00b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 953768598 0 0) 0x7eff84003520 con 0x7effa00b40c0 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 >> v2:172.21.15.135:3300/0 conn(0x7effa01620c0 msgr2=0x7effa00b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 --2- 172.21.15.97:0/689986553 >> v2:172.21.15.135:3300/0 conn(0x7effa01620c0 0x7effa00b28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 >> v1:172.21.15.135:6789/0 conn(0x7effa00ae670 legacy=0x7effa015e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa015f190 con 0x7effa00b40c0 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7eff90001670 con 0x7effa00b40c0 2024-03-06T07:30:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 >> v1:172.21.15.97:6789/0 conn(0x7effa00b40c0 legacy=0x7effa00b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff84003a00 0x7eff84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84004800 0x7eff84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7eff84005210 0x7eff8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa015eeb0 con 0x7eff84003a00 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8400c160 con 0x7eff84004800 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8400c360 con 0x7eff84005210 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9fe00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff84003a00 0x7eff84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.437+0000 7eff9f400640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84004800 0x7eff84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7effa4c00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7eff84005210 0x7eff8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7eff9fe00640 1 -- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7eff84005210 msgr2=0x7eff8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7eff9fe00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7eff84005210 0x7eff8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7eff9fe00640 1 -- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84004800 msgr2=0x7eff84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7eff9fe00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84004800 0x7eff84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7eff9fe00640 1 -- 172.21.15.97:0/689986553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84003520 con 0x7eff84003a00 2024-03-06T07:30:15.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7effa4c00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7eff84005210 0x7eff8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.438+0000 7effa4c00640 1 -- 172.21.15.97:0/689986553 reap_dead start 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.439+0000 7eff9fe00640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff84003a00 0x7eff84003e20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c190 tx=0x7eff84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af807b297a2de00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.439+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90007c60 con 0x7eff84003a00 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.439+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff90030070 con 0x7eff84003a00 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.439+0000 7eff9ea00640 1 -- 172.21.15.97:0/689986553 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90005d90 con 0x7eff84003a00 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 -- 172.21.15.97:0/689986553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff84003a00 msgr2=0x7eff84003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff84003a00 0x7eff84003e20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c190 tx=0x7eff84005830 comp rx=0 tx=0).stop 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 -- 172.21.15.97:0/689986553 shutdown_connections 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 --2- 172.21.15.97:0/689986553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff84003a00 0x7eff84003e20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 -- 172.21.15.97:0/689986553 >> 172.21.15.97:0/689986553 conn(0x7effa006afc0 msgr2=0x7effa006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 -- 172.21.15.97:0/689986553 shutdown_connections 2024-03-06T07:30:15.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.440+0000 7effa5600640 1 -- 172.21.15.97:0/689986553 wait complete. 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.441+0000 7effa5600640 1 Processor -- start 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.442+0000 7effa5600640 1 -- start start 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.442+0000 7effa5600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84003a00 0x7effa0147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.442+0000 7effa5600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 0x7effa00af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7effa5600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effa00afa70 0x7effa0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7effa5600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa01587e0 con 0x7effa0156160 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7effa5600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa0158660 con 0x7eff84003a00 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7effa5600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa0162570 con 0x7effa00afa70 2024-03-06T07:30:15.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 0x7effa00af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 0x7effa00af530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59728/0 (socket says 172.21.15.97:59728) 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 -- 172.21.15.97:0/4153662976 learned_addr learned my addr 172.21.15.97:0/4153662976 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9fe00640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84003a00 0x7effa0147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 -- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effa00afa70 msgr2=0x7effa0071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7effa4c00640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effa00afa70 0x7effa0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effa00afa70 0x7effa0071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 -- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84003a00 msgr2=0x7effa0147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84003a00 0x7effa0147430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9f400640 1 -- 172.21.15.97:0/4153662976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa01553a0 con 0x7effa0156160 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.443+0000 7eff9fe00640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84003a00 0x7effa0147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.444+0000 7eff9f400640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 0x7effa00af530 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00e4b0 tx=0x7eff8c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.444+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0052e0 con 0x7effa0156160 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.444+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c018070 con 0x7effa0156160 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.444+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c003830 con 0x7effa0156160 2024-03-06T07:30:15.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.444+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa0155680 con 0x7effa0156160 2024-03-06T07:30:15.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.444+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa0072220 con 0x7effa0156160 2024-03-06T07:30:15.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.445+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa00afe80 con 0x7effa0156160 2024-03-06T07:30:15.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.446+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff8c0039d0 con 0x7effa0156160 2024-03-06T07:30:15.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.447+0000 7eff97e00640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7eff8006b4d0 0x7eff8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.447+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff8c089bc0 con 0x7effa0156160 2024-03-06T07:30:15.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.447+0000 7eff9fe00640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7eff8006b4d0 0x7eff8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.448+0000 7eff9fe00640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7eff8006b4d0 0x7eff8006d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c160 tx=0x7eff90009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.451+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effa00afe80 con 0x7effa0156160 2024-03-06T07:30:15.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.719+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7effa004f330 con 0x7effa0156160 2024-03-06T07:30:15.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.720+0000 7eff97e00640 1 -- 172.21.15.97:0/4153662976 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7eff8c058cf0 con 0x7effa0156160 2024-03-06T07:30:15.721 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:15.721 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:30:15.721 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.722 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.724 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.725 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:15.728 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:30:15.729 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:30:15.731 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:30:15.732 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:30:15.733 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7eff8006b4d0 msgr2=0x7eff8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7eff8006b4d0 0x7eff8006d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c160 tx=0x7eff90009030 comp rx=0 tx=0).stop 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 msgr2=0x7effa00af530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 0x7effa00af530 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00e4b0 tx=0x7eff8c00e970 comp rx=0 tx=0).stop 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 shutdown_connections 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7eff8006b4d0 0x7eff8006d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7effa00afa70 0x7effa0071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effa0156160 0x7effa00af530 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 --2- 172.21.15.97:0/4153662976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff84003a00 0x7effa0147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.722+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 >> 172.21.15.97:0/4153662976 conn(0x7effa006afc0 msgr2=0x7effa015d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.723+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 shutdown_connections 2024-03-06T07:30:15.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.723+0000 7effa5600640 1 -- 172.21.15.97:0/4153662976 wait complete. 2024-03-06T07:30:15.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:15.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.945+0000 7f4b2ae00640 1 Processor -- start 2024-03-06T07:30:15.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.945+0000 7f4b2ae00640 1 -- start start 2024-03-06T07:30:15.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b2ae00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4b2415f9d0 0x7f4b2415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b2ae00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4b24070b10 0x7f4b24070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b2ae00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2406cef0 con 0x7f4b24144c00 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b2ae00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2406d1d0 con 0x7f4b24070b10 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b2ae00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2406d4b0 con 0x7f4b2415f9d0 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b28e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4b24070b10 0x7f4b24070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b28e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4b24070b10 0x7f4b24070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59744/0 (socket says 172.21.15.97:59744) 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b28e00640 1 -- 172.21.15.97:0/592757719 learned_addr learned my addr 172.21.15.97:0/592757719 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b28e00640 1 -- 172.21.15.97:0/592757719 >> v2:172.21.15.135:3300/0 conn(0x7f4b2415f9d0 msgr2=0x7f4b2415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b28e00640 1 --2- 172.21.15.97:0/592757719 >> v2:172.21.15.135:3300/0 conn(0x7f4b2415f9d0 0x7f4b2415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.947+0000 7f4b28e00640 1 -- 172.21.15.97:0/592757719 >> v1:172.21.15.135:6789/0 conn(0x7f4b24144c00 legacy=0x7f4b2406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.948+0000 7f4b28e00640 1 -- 172.21.15.97:0/592757719 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2406d790 con 0x7f4b24070b10 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b28e00640 1 --2- 172.21.15.97:0/592757719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b24070b10 0x7f4b24070f20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4b240b26f0 tx=0x7f4b1c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3044346cdac40e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b23400640 1 -- 172.21.15.97:0/592757719 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c02ca80 con 0x7f4b24070b10 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b23400640 1 -- 172.21.15.97:0/592757719 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c02cc20 con 0x7f4b24070b10 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b23400640 1 -- 172.21.15.97:0/592757719 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0343f0 con 0x7f4b24070b10 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b2ae00640 1 -- 172.21.15.97:0/592757719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b24070b10 msgr2=0x7f4b24070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/592757719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b24070b10 0x7f4b24070f20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4b240b26f0 tx=0x7f4b1c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b2ae00640 1 -- 172.21.15.97:0/592757719 shutdown_connections 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/592757719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b24070b10 0x7f4b24070f20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/592757719 >> v2:172.21.15.135:3300/0 conn(0x7f4b2415f9d0 0x7f4b2415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.949+0000 7f4b2ae00640 1 -- 172.21.15.97:0/592757719 >> 172.21.15.97:0/592757719 conn(0x7f4b240b1730 msgr2=0x7f4b240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.950+0000 7f4b2ae00640 1 -- 172.21.15.97:0/592757719 shutdown_connections 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.950+0000 7f4b2ae00640 1 -- 172.21.15.97:0/592757719 wait complete. 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.951+0000 7f4b2ae00640 1 Processor -- start 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.951+0000 7f4b2ae00640 1 -- start start 2024-03-06T07:30:15.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b2ae00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b24070b10 0x7f4b2415d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b2ae00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4b24144c00 0x7f4b2413a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b2ae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 0x7f4b2413c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b2ae00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2406ccd0 con 0x7f4b2415f9d0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b2ae00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2406cb50 con 0x7f4b24070b10 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b2ae00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2406df90 con 0x7f4b24144c00 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b28e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 0x7f4b2413c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b28e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 0x7f4b2413c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59746/0 (socket says 172.21.15.97:59746) 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.952+0000 7f4b28e00640 1 -- 172.21.15.97:0/3061707769 learned_addr learned my addr 172.21.15.97:0/3061707769 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b23e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b24070b10 0x7f4b2415d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b28e00640 1 -- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4b24144c00 msgr2=0x7f4b2413a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b28e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4b24144c00 0x7f4b2413a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b28e00640 1 -- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b24070b10 msgr2=0x7f4b2415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b28e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b24070b10 0x7f4b2415d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b28e00640 1 -- 172.21.15.97:0/3061707769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2413a140 con 0x7f4b2415f9d0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b23e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b24070b10 0x7f4b2415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b28e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 0x7f4b2413c230 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036040 tx=0x7f4b1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c043020 con 0x7f4b2415f9d0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c03e450 con 0x7f4b2415f9d0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b24139b80 con 0x7f4b2415f9d0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.953+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c03e820 con 0x7f4b2415f9d0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.954+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b241e6b80 con 0x7f4b2415f9d0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.955+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2404f330 con 0x7f4b2415f9d0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.956+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b1c046020 con 0x7f4b2415f9d0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.956+0000 7f4b20c00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4b0006b4d0 0x7f4b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.956+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b1c0ac780 con 0x7f4b2415f9d0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.957+0000 7f4b23e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4b0006b4d0 0x7f4b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.957+0000 7f4b23e00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4b0006b4d0 0x7f4b0006d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4b14003eb0 tx=0x7f4b14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:15.961+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b1c04d030 con 0x7f4b2415f9d0 2024-03-06T07:30:16.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.157+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b24072890 con 0x7f4b2415f9d0 2024-03-06T07:30:16.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.158+0000 7f4b20c00640 1 -- 172.21.15.97:0/3061707769 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f4b1c07b9e0 con 0x7f4b2415f9d0 2024-03-06T07:30:16.160 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:16.160 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":54,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:15.197038+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:16.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4b0006b4d0 msgr2=0x7f4b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4b0006b4d0 0x7f4b0006d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4b14003eb0 tx=0x7f4b14009040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 msgr2=0x7f4b2413c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 0x7f4b2413c230 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036040 tx=0x7f4b1c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 shutdown_connections 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4b0006b4d0 0x7f4b0006d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2415f9d0 0x7f4b2413c230 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4b24144c00 0x7f4b2413a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 --2- 172.21.15.97:0/3061707769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b24070b10 0x7f4b2415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 >> 172.21.15.97:0/3061707769 conn(0x7f4b240b1730 msgr2=0x7f4b240b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.161+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 shutdown_connections 2024-03-06T07:30:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.162+0000 7f4b2ae00640 1 -- 172.21.15.97:0/3061707769 wait complete. 2024-03-06T07:30:16.181 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:16.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:16.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.387+0000 7f8249200640 1 Processor -- start 2024-03-06T07:30:16.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.387+0000 7f8249200640 1 -- start start 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.388+0000 7f8249200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 0x7f8244070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.388+0000 7f8249200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f82440b3f70 0x7f82440ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8249200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82440b46c0 con 0x7f824415fff0 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8249200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440b49a0 con 0x7f82440b3f70 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8249200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8244072c90 con 0x7f824415fc10 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8241400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f824415fff0 0x7f8244160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39054/0 (socket says 172.21.15.97:39054) 2024-03-06T07:30:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8241400640 1 -- 172.21.15.97:0/71749097 learned_addr learned my addr 172.21.15.97:0/71749097 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8242800640 1 --2- 172.21.15.97:0/71749097 >> v2:172.21.15.135:3300/0 conn(0x7f82440b3f70 0x7f82440ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8241e00640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 0x7f8244070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297271280 0 0) 0x7f82440b46c0 con 0x7f824415fff0 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.389+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228003520 con 0x7f824415fff0 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288455352 0 0) 0x7f8228003520 con 0x7f824415fff0 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 msgr2=0x7f8244070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 0x7f8244070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 >> v2:172.21.15.135:3300/0 conn(0x7f82440b3f70 msgr2=0x7f82440ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 --2- 172.21.15.97:0/71749097 >> v2:172.21.15.135:3300/0 conn(0x7f82440b3f70 0x7f82440ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824409dd80 con 0x7f824415fff0 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f822c001670 con 0x7f824415fff0 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 >> v1:172.21.15.97:6789/0 conn(0x7f824415fff0 legacy=0x7f8244160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 0x7f82280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f8228004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8228004d40 0x7f82280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440b49a0 con 0x7f8228003f80 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244072c90 con 0x7f8228004d40 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822800b9c0 con 0x7f824415fc10 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8241400640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f8228004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8241400640 1 -- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 msgr2=0x7f82280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8241400640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824415fc10 0x7f82280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8242800640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8228004d40 0x7f82280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8241400640 1 -- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8228004d40 msgr2=0x7f82280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8241400640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8228004d40 0x7f82280051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.390+0000 7f8241400640 1 -- 172.21.15.97:0/71749097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228003520 con 0x7f8228003f80 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8241400640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f8228004800 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f822c00c1c0 tx=0x7f8228005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a156fcc926829fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c007c80 con 0x7f8228003f80 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c030070 con 0x7f8228003f80 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8240a00640 1 -- 172.21.15.97:0/71749097 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c005d70 con 0x7f8228003f80 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8249200640 1 -- 172.21.15.97:0/71749097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 msgr2=0x7f8228004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8249200640 1 --2- 172.21.15.97:0/71749097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f8228004800 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f822c00c1c0 tx=0x7f8228005830 comp rx=0 tx=0).stop 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8242800640 1 -- 172.21.15.97:0/71749097 reap_dead start 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8249200640 1 -- 172.21.15.97:0/71749097 shutdown_connections 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8249200640 1 -- 172.21.15.97:0/71749097 >> 172.21.15.97:0/71749097 conn(0x7f82440b1940 msgr2=0x7f82440b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.392+0000 7f8249200640 1 -- 172.21.15.97:0/71749097 shutdown_connections 2024-03-06T07:30:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.393+0000 7f8249200640 1 -- 172.21.15.97:0/71749097 wait complete. 2024-03-06T07:30:16.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.394+0000 7f8249200640 1 Processor -- start 2024-03-06T07:30:16.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.394+0000 7f8249200640 1 -- start start 2024-03-06T07:30:16.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.395+0000 7f8249200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f82441f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.395+0000 7f8249200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82440ca030 0x7f824409ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.395+0000 7f8249200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824409bf80 0x7f824414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.395+0000 7f8249200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824415c460 con 0x7f8228003f80 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.395+0000 7f8249200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824415c2e0 con 0x7f82440ca030 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.395+0000 7f8249200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8244070ff0 con 0x7f824409bf80 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f82441f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f82441f4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59750/0 (socket says 172.21.15.97:59750) 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 -- 172.21.15.97:0/2147128188 learned_addr learned my addr 172.21.15.97:0/2147128188 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 -- 172.21.15.97:0/2147128188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824409bf80 msgr2=0x7f824414ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824409bf80 0x7f824414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 -- 172.21.15.97:0/2147128188 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82440ca030 msgr2=0x7f824409ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82440ca030 0x7f824409ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 -- 172.21.15.97:0/2147128188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824406d220 con 0x7f8228003f80 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8241e00640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f82441f4840 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f823400ef10 tx=0x7f823400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234018020 con 0x7f8228003f80 2024-03-06T07:30:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8234014400 con 0x7f8228003f80 2024-03-06T07:30:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340147d0 con 0x7f8228003f80 2024-03-06T07:30:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.396+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824406d500 con 0x7f8228003f80 2024-03-06T07:30:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.397+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824406dac0 con 0x7f8228003f80 2024-03-06T07:30:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.398+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82440caa10 con 0x7f8228003f80 2024-03-06T07:30:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.399+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8234005d10 con 0x7f8228003f80 2024-03-06T07:30:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.399+0000 7f823a000640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f822006b4d0 0x7f822006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.399+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f823408a680 con 0x7f8228003f80 2024-03-06T07:30:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.400+0000 7f8241400640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f822006b4d0 0x7f822006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.400+0000 7f8241400640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f822006b4d0 0x7f822006d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f822c034000 tx=0x7f822c039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.404+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8234022030 con 0x7f8228003f80 2024-03-06T07:30:16.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.598+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f824404f330 con 0x7f8228003f80 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.599+0000 7f823a000640 1 -- 172.21.15.97:0/2147128188 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82340598e0 con 0x7f8228003f80 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:30:16.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f822006b4d0 msgr2=0x7f822006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f822006b4d0 0x7f822006d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f822c034000 tx=0x7f822c039040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 msgr2=0x7f82441f4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f82441f4840 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f823400ef10 tx=0x7f823400b480 comp rx=0 tx=0).stop 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 shutdown_connections 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f822006b4d0 0x7f822006d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f824409bf80 0x7f824414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82440ca030 0x7f824409ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 --2- 172.21.15.97:0/2147128188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8228003f80 0x7f82441f4840 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 >> 172.21.15.97:0/2147128188 conn(0x7f82440b1940 msgr2=0x7f82440ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 shutdown_connections 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.601+0000 7f8249200640 1 -- 172.21.15.97:0/2147128188 wait complete. 2024-03-06T07:30:16.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.817+0000 7f4461800640 1 Processor -- start 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.817+0000 7f4461800640 1 -- start start 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f4461800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f445c15f470 0x7f445c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f4461800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 0x7f445c160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f4461800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c0cea60 con 0x7f445c15e0d0 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f4461800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0d2f60 con 0x7f445c15f470 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f4461800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0d1240 con 0x7f445c0d0df0 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f445aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f445c15e0d0 0x7f445c0d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39070/0 (socket says 172.21.15.97:39070) 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f445aa00640 1 -- 172.21.15.97:0/386364888 learned_addr learned my addr 172.21.15.97:0/386364888 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:16.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f4452000640 1 --2- 172.21.15.97:0/386364888 >> v2:172.21.15.135:3300/0 conn(0x7f445c15f470 0x7f445c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550939781 0 0) 0x7f445c0cea60 con 0x7f445c15e0d0 2024-03-06T07:30:16.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4440003520 con 0x7f445c15e0d0 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.819+0000 7f445b400640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 0x7f445c160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211493225 0 0) 0x7f4440003520 con 0x7f445c15e0d0 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 msgr2=0x7f445c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 0x7f445c160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 >> v2:172.21.15.135:3300/0 conn(0x7f445c15f470 msgr2=0x7f445c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 --2- 172.21.15.97:0/386364888 >> v2:172.21.15.135:3300/0 conn(0x7f445c15f470 0x7f445c0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c0d2350 con 0x7f445c15e0d0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f444c0019f0 con 0x7f445c15e0d0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f4452000640 1 --2- 172.21.15.97:0/386364888 >> v2:172.21.15.135:3300/0 conn(0x7f445c15f470 0x7f445c0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 >> v1:172.21.15.97:6789/0 conn(0x7f445c15e0d0 legacy=0x7f445c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44400039b0 0x7f4440003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4440004800 0x7f4440004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 0x7f44400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0d1240 con 0x7f44400039b0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444000b9f0 con 0x7f4440004800 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444000bbf0 con 0x7f445c0d0df0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f445aa00640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44400039b0 0x7f4440003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.820+0000 7f4452000640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4440004800 0x7f4440004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.821+0000 7f445b400640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 0x7f44400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.821+0000 7f445aa00640 1 -- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 msgr2=0x7f44400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.821+0000 7f445aa00640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c0d0df0 0x7f44400051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.821+0000 7f445aa00640 1 -- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4440004800 msgr2=0x7f4440004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.821+0000 7f445aa00640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4440004800 0x7f4440004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.821+0000 7f445aa00640 1 -- 172.21.15.97:0/386364888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4440003520 con 0x7f44400039b0 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.822+0000 7f445aa00640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44400039b0 0x7f4440003dd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f444c00a8a0 tx=0x7f4440005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bcce5e062dcb290 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.822+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c00a4c0 con 0x7f44400039b0 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.822+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c034630 con 0x7f44400039b0 2024-03-06T07:30:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.822+0000 7f445a000640 1 -- 172.21.15.97:0/386364888 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c034a60 con 0x7f44400039b0 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f4461800640 1 -- 172.21.15.97:0/386364888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44400039b0 msgr2=0x7f4440003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f4461800640 1 --2- 172.21.15.97:0/386364888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44400039b0 0x7f4440003dd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f444c00a8a0 tx=0x7f4440005830 comp rx=0 tx=0).stop 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f445b400640 1 -- 172.21.15.97:0/386364888 reap_dead start 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f4461800640 1 -- 172.21.15.97:0/386364888 shutdown_connections 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f4461800640 1 -- 172.21.15.97:0/386364888 >> 172.21.15.97:0/386364888 conn(0x7f445c0d3ec0 msgr2=0x7f445c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f4461800640 1 -- 172.21.15.97:0/386364888 shutdown_connections 2024-03-06T07:30:16.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.823+0000 7f4461800640 1 -- 172.21.15.97:0/386364888 wait complete. 2024-03-06T07:30:16.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.825+0000 7f4461800640 1 Processor -- start 2024-03-06T07:30:16.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.825+0000 7f4461800640 1 -- start start 2024-03-06T07:30:16.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4461800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44400039b0 0x7f445c0b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4461800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 0x7f445c0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4461800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c148150 0x7f445c14a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4461800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0d2bf0 con 0x7f445c0b9260 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4461800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0d2910 con 0x7f44400039b0 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4461800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c15fb30 con 0x7f445c148150 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 0x7f445c0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 0x7f445c0cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59770/0 (socket says 172.21.15.97:59770) 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 -- 172.21.15.97:0/222508118 learned_addr learned my addr 172.21.15.97:0/222508118 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f445b400640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c148150 0x7f445c14a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f445aa00640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44400039b0 0x7f445c0b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 -- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c148150 msgr2=0x7f445c14a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c148150 0x7f445c14a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 -- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44400039b0 msgr2=0x7f445c0b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44400039b0 0x7f445c0b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.826+0000 7f4452000640 1 -- 172.21.15.97:0/222508118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c0c8720 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.827+0000 7f4452000640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 0x7f445c0cd820 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f44440108c0 tx=0x7f4444010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.827+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444401a020 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.827+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444400eb00 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.827+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c0c8a00 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.827+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444400f040 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.827+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c0cdd60 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.828+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c04f330 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.829+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f444401d020 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.830+0000 7f4453400640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443806b4d0 0x7f443806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.830+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f444408a7d0 con 0x7f445c0b9260 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.830+0000 7f445aa00640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443806b4d0 0x7f443806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.831+0000 7f445aa00640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443806b4d0 0x7f443806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f444c034000 tx=0x7f444c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:16.835+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4444024020 con 0x7f445c0b9260 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.034+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f445c08d920 con 0x7f445c0b9260 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.035+0000 7f4453400640 1 -- 172.21.15.97:0/222508118 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+65 (secure 0 0 0) 0x7f44440599a0 con 0x7f445c0b9260 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:30:17.037 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443806b4d0 msgr2=0x7f443806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443806b4d0 0x7f443806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f444c034000 tx=0x7f444c009030 comp rx=0 tx=0).stop 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 msgr2=0x7f445c0cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 0x7f445c0cd820 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f44440108c0 tx=0x7f4444010d80 comp rx=0 tx=0).stop 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 shutdown_connections 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443806b4d0 0x7f443806d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445c148150 0x7f445c14a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445c0b9260 0x7f445c0cd820 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 --2- 172.21.15.97:0/222508118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44400039b0 0x7f445c0b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.037+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 >> 172.21.15.97:0/222508118 conn(0x7f445c0d3ec0 msgr2=0x7f445c0ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.038+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 shutdown_connections 2024-03-06T07:30:17.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.038+0000 7f4461800640 1 -- 172.21.15.97:0/222508118 wait complete. 2024-03-06T07:30:17.057 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:17.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:17.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.257+0000 7efd1ca00640 1 Processor -- start 2024-03-06T07:30:17.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.258+0000 7efd1ca00640 1 -- start start 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.259+0000 7efd1ca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efd1806e090 0x7efd180cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.259+0000 7efd1ca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1806e750 con 0x7efd1806afc0 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.259+0000 7efd1ca00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1806ea30 con 0x7efd18154e10 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.259+0000 7efd1ca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1806e520 con 0x7efd1806e090 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efd1806e090 0x7efd180cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efd1806e090 0x7efd180cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59776/0 (socket says 172.21.15.97:59776) 2024-03-06T07:30:17.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd17e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efd1806afc0 0x7efd18071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39080/0 (socket says 172.21.15.97:39080) 2024-03-06T07:30:17.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16a00640 1 -- 172.21.15.97:0/2128280647 learned_addr learned my addr 172.21.15.97:0/2128280647 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16a00640 1 -- 172.21.15.97:0/2128280647 >> v1:172.21.15.135:6789/0 conn(0x7efd18154e10 legacy=0x7efd1815ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16a00640 1 -- 172.21.15.97:0/2128280647 >> v1:172.21.15.97:6789/0 conn(0x7efd1806afc0 legacy=0x7efd18071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16000640 1 -- 172.21.15.97:0/2128280647 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892474352 0 0) 0x7efd1806e750 con 0x7efd1806afc0 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.260+0000 7efd16a00640 1 -- 172.21.15.97:0/2128280647 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1815e660 con 0x7efd1806e090 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.261+0000 7efd16a00640 1 --2- 172.21.15.97:0/2128280647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806e090 0x7efd180cf040 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efd08009870 tx=0x7efd0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95edd3180a00653 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd16000640 1 -- 172.21.15.97:0/2128280647 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08034070 con 0x7efd1806e090 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd16000640 1 -- 172.21.15.97:0/2128280647 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd08030860 con 0x7efd1806e090 2024-03-06T07:30:17.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd16000640 1 -- 172.21.15.97:0/2128280647 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08030c30 con 0x7efd1806e090 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd1ca00640 1 -- 172.21.15.97:0/2128280647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806e090 msgr2=0x7efd180cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd1ca00640 1 --2- 172.21.15.97:0/2128280647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806e090 0x7efd180cf040 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efd08009870 tx=0x7efd0802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd1ca00640 1 -- 172.21.15.97:0/2128280647 shutdown_connections 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd1ca00640 1 --2- 172.21.15.97:0/2128280647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806e090 0x7efd180cf040 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.262+0000 7efd1ca00640 1 -- 172.21.15.97:0/2128280647 >> 172.21.15.97:0/2128280647 conn(0x7efd180b18a0 msgr2=0x7efd180b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.263+0000 7efd1ca00640 1 -- 172.21.15.97:0/2128280647 shutdown_connections 2024-03-06T07:30:17.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.263+0000 7efd1ca00640 1 -- 172.21.15.97:0/2128280647 wait complete. 2024-03-06T07:30:17.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.264+0000 7efd1ca00640 1 Processor -- start 2024-03-06T07:30:17.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.264+0000 7efd1ca00640 1 -- start start 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd1ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 0x7efd18154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd1ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd1806e090 0x7efd18159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd1ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efd18154e10 0x7efd181568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd1ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd181551f0 con 0x7efd1806afc0 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd1ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806ed10 con 0x7efd1806e090 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd1ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180cf580 con 0x7efd18154e10 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd17400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 0x7efd18154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd17400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 0x7efd18154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59790/0 (socket says 172.21.15.97:59790) 2024-03-06T07:30:17.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.265+0000 7efd17400640 1 -- 172.21.15.97:0/3573002506 learned_addr learned my addr 172.21.15.97:0/3573002506 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:17.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd17400640 1 -- 172.21.15.97:0/3573002506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efd18154e10 msgr2=0x7efd181568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd17400640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efd18154e10 0x7efd181568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd17400640 1 -- 172.21.15.97:0/3573002506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd1806e090 msgr2=0x7efd18159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd17400640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd1806e090 0x7efd18159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd17400640 1 -- 172.21.15.97:0/3573002506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1815a0b0 con 0x7efd1806afc0 2024-03-06T07:30:17.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd17400640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 0x7efd18154510 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efd0000b420 tx=0x7efd0000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.266+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd00011830 con 0x7efd1806afc0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.267+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd00016070 con 0x7efd1806afc0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.267+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1815efe0 con 0x7efd1806afc0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.267+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd00011c10 con 0x7efd1806afc0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.267+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1815f5a0 con 0x7efd1806afc0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.268+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1804f330 con 0x7efd1806afc0 2024-03-06T07:30:17.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.269+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd0001b460 con 0x7efd1806afc0 2024-03-06T07:30:17.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.270+0000 7efd0f400640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.270+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efd000899f0 con 0x7efd1806afc0 2024-03-06T07:30:17.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.270+0000 7efd16a00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.270+0000 7efd16a00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efcf406b4d0 0x7efcf406d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efd08036000 tx=0x7efd08030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.274+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd00020020 con 0x7efd1806afc0 2024-03-06T07:30:17.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:17.468+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efd180ae3b0 con 0x7efd1806afc0 2024-03-06T07:30:18.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.227+0000 7efd0f400640 1 -- 172.21.15.97:0/3573002506 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) v1 ==== 138+0+0 (secure 0 0 0) 0x7efd00058b20 con 0x7efd1806afc0 2024-03-06T07:30:18.229 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:18.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efcf406b4d0 msgr2=0x7efcf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efcf406b4d0 0x7efcf406d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efd08036000 tx=0x7efd08030000 comp rx=0 tx=0).stop 2024-03-06T07:30:18.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 msgr2=0x7efd18154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 0x7efd18154510 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efd0000b420 tx=0x7efd0000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 shutdown_connections 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efd18154e10 0x7efd181568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd1806e090 0x7efd18159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 --2- 172.21.15.97:0/3573002506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd1806afc0 0x7efd18154510 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.231+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 >> 172.21.15.97:0/3573002506 conn(0x7efd180b18a0 msgr2=0x7efd180716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.232+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 shutdown_connections 2024-03-06T07:30:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.232+0000 7efd1ca00640 1 -- 172.21.15.97:0/3573002506 wait complete. 2024-03-06T07:30:18.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.459+0000 7fee09800640 1 Processor -- start 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.460+0000 7fee09800640 1 -- start start 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.461+0000 7fee09800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee0415cb90 0x7fee0406d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.461+0000 7fee09800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fee0406e060 0x7fee0406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.461+0000 7fee09800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0415cf70 con 0x7fee04163800 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.461+0000 7fee09800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee040b3890 con 0x7fee0415cb90 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.461+0000 7fee09800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0406ea10 con 0x7fee0406e060 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee0415cb90 0x7fee0406d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee0415cb90 0x7fee0406d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59802/0 (socket says 172.21.15.97:59802) 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 -- 172.21.15.97:0/3921440257 learned_addr learned my addr 172.21.15.97:0/3921440257 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:18.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 -- 172.21.15.97:0/3921440257 >> v2:172.21.15.135:3300/0 conn(0x7fee0406e060 msgr2=0x7fee0406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 --2- 172.21.15.97:0/3921440257 >> v2:172.21.15.135:3300/0 conn(0x7fee0406e060 0x7fee0406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 -- 172.21.15.97:0/3921440257 >> v1:172.21.15.135:6789/0 conn(0x7fee04163800 legacy=0x7fee040b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.462+0000 7fee02a00640 1 -- 172.21.15.97:0/3921440257 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee040b35b0 con 0x7fee0415cb90 2024-03-06T07:30:18.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.463+0000 7fee02a00640 1 --2- 172.21.15.97:0/3921440257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee0415cb90 0x7fee0406d6c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fee0406c670 tx=0x7fedf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5d891604334687 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee02000640 1 -- 172.21.15.97:0/3921440257 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf4035020 con 0x7fee0415cb90 2024-03-06T07:30:18.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee02000640 1 -- 172.21.15.97:0/3921440257 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf4031870 con 0x7fee0415cb90 2024-03-06T07:30:18.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee02000640 1 -- 172.21.15.97:0/3921440257 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf4031c40 con 0x7fee0415cb90 2024-03-06T07:30:18.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee09800640 1 -- 172.21.15.97:0/3921440257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee0415cb90 msgr2=0x7fee0406d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee09800640 1 --2- 172.21.15.97:0/3921440257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee0415cb90 0x7fee0406d6c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fee0406c670 tx=0x7fedf402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee09800640 1 -- 172.21.15.97:0/3921440257 shutdown_connections 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee09800640 1 --2- 172.21.15.97:0/3921440257 >> v2:172.21.15.135:3300/0 conn(0x7fee0406e060 0x7fee0406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.464+0000 7fee09800640 1 --2- 172.21.15.97:0/3921440257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee0415cb90 0x7fee0406d6c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.465+0000 7fee09800640 1 -- 172.21.15.97:0/3921440257 >> 172.21.15.97:0/3921440257 conn(0x7fee040b1940 msgr2=0x7fee040b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.465+0000 7fee09800640 1 -- 172.21.15.97:0/3921440257 shutdown_connections 2024-03-06T07:30:18.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.465+0000 7fee09800640 1 -- 172.21.15.97:0/3921440257 wait complete. 2024-03-06T07:30:18.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.467+0000 7fee09800640 1 Processor -- start 2024-03-06T07:30:18.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.467+0000 7fee09800640 1 -- start start 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee09800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee0406e060 0x7fee0409d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee09800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee0415cb90 0x7fee04155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee09800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 0x7fee0409b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee09800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0406d230 con 0x7fee04163800 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee09800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0406c990 con 0x7fee0415cb90 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee09800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee040b3350 con 0x7fee0406e060 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee03400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 0x7fee0409b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee03400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 0x7fee0409b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59810/0 (socket says 172.21.15.97:59810) 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.468+0000 7fee03400640 1 -- 172.21.15.97:0/650699754 learned_addr learned my addr 172.21.15.97:0/650699754 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fedfbe00640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee0415cb90 0x7fee04155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee02a00640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee0406e060 0x7fee0409d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee03400640 1 -- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee0406e060 msgr2=0x7fee0409d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee03400640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee0406e060 0x7fee0409d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee03400640 1 -- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee0415cb90 msgr2=0x7fee04155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee03400640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee0415cb90 0x7fee04155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee03400640 1 -- 172.21.15.97:0/650699754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0409c470 con 0x7fee04163800 2024-03-06T07:30:18.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fee03400640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 0x7fee0409b400 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b9a0 tx=0x7fedfc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.469+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc019070 con 0x7fee04163800 2024-03-06T07:30:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.470+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc00cbc0 con 0x7fee04163800 2024-03-06T07:30:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.470+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc014440 con 0x7fee04163800 2024-03-06T07:30:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.470+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0409c710 con 0x7fee04163800 2024-03-06T07:30:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.470+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0409ccd0 con 0x7fee04163800 2024-03-06T07:30:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.471+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0404f330 con 0x7fee04163800 2024-03-06T07:30:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.473+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedfc01c070 con 0x7fee04163800 2024-03-06T07:30:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.474+0000 7fedfaa00640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fede006b4d0 0x7fede006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.474+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fedfc08a180 con 0x7fee04163800 2024-03-06T07:30:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.474+0000 7fee02a00640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fede006b4d0 0x7fede006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.475+0000 7fee02a00640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fede006b4d0 0x7fede006d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fedf4036000 tx=0x7fedf403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.478+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedfc023020 con 0x7fee04163800 2024-03-06T07:30:18.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:18.675+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fee040b2530 con 0x7fee04163800 2024-03-06T07:30:19.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.234+0000 7fedfaa00640 1 -- 172.21.15.97:0/650699754 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) v1 ==== 170+0+0 (secure 0 0 0) 0x7fedfc059250 con 0x7fee04163800 2024-03-06T07:30:19.236 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:30:19.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.238+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fede006b4d0 msgr2=0x7fede006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.238+0000 7fee09800640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fede006b4d0 0x7fede006d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fedf4036000 tx=0x7fedf403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:19.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.238+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 msgr2=0x7fee0409b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.238+0000 7fee09800640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 0x7fee0409b400 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b9a0 tx=0x7fedfc00be60 comp rx=0 tx=0).stop 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.239+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 shutdown_connections 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.239+0000 7fee09800640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fede006b4d0 0x7fede006d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.239+0000 7fee09800640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04163800 0x7fee0409b400 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.239+0000 7fee09800640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee0415cb90 0x7fee04155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.239+0000 7fee09800640 1 --2- 172.21.15.97:0/650699754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee0406e060 0x7fee0409d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.239+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 >> 172.21.15.97:0/650699754 conn(0x7fee040b1940 msgr2=0x7fee040afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.240+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 shutdown_connections 2024-03-06T07:30:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.240+0000 7fee09800640 1 -- 172.21.15.97:0/650699754 wait complete. 2024-03-06T07:30:19.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.457+0000 7f041b000640 1 Processor -- start 2024-03-06T07:30:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.457+0000 7f041b000640 1 -- start start 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.458+0000 7f041b000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f04140b3f70 0x7f0414160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.458+0000 7f041b000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0414070ff0 0x7f04140ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.458+0000 7f041b000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0414071440 con 0x7f041415fc10 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.458+0000 7f041b000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04140b46c0 con 0x7f0414070ff0 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.458+0000 7f041b000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04140b49a0 con 0x7f04140b3f70 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0414070ff0 0x7f04140ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0414070ff0 0x7f04140ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59812/0 (socket says 172.21.15.97:59812) 2024-03-06T07:30:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 -- 172.21.15.97:0/2729724390 learned_addr learned my addr 172.21.15.97:0/2729724390 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 -- 172.21.15.97:0/2729724390 >> v2:172.21.15.135:3300/0 conn(0x7f04140b3f70 msgr2=0x7f0414160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 --2- 172.21.15.97:0/2729724390 >> v2:172.21.15.135:3300/0 conn(0x7f04140b3f70 0x7f0414160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 -- 172.21.15.97:0/2729724390 >> v1:172.21.15.135:6789/0 conn(0x7f041415fc10 legacy=0x7f041415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.459+0000 7f0419000640 1 -- 172.21.15.97:0/2729724390 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414072c90 con 0x7f0414070ff0 2024-03-06T07:30:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.460+0000 7f0419000640 1 --2- 172.21.15.97:0/2729724390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0414070ff0 0x7f04140ce4a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0414074a50 tx=0x7f040c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e03241a086a739 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f0412a00640 1 -- 172.21.15.97:0/2729724390 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c02ca80 con 0x7f0414070ff0 2024-03-06T07:30:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f0412a00640 1 -- 172.21.15.97:0/2729724390 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c02cc20 con 0x7f0414070ff0 2024-03-06T07:30:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f0412a00640 1 -- 172.21.15.97:0/2729724390 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c0343f0 con 0x7f0414070ff0 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 -- 172.21.15.97:0/2729724390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0414070ff0 msgr2=0x7f04140ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 --2- 172.21.15.97:0/2729724390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0414070ff0 0x7f04140ce4a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0414074a50 tx=0x7f040c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 -- 172.21.15.97:0/2729724390 shutdown_connections 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 --2- 172.21.15.97:0/2729724390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0414070ff0 0x7f04140ce4a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 --2- 172.21.15.97:0/2729724390 >> v2:172.21.15.135:3300/0 conn(0x7f04140b3f70 0x7f0414160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 -- 172.21.15.97:0/2729724390 >> 172.21.15.97:0/2729724390 conn(0x7f04140b1940 msgr2=0x7f04140b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 -- 172.21.15.97:0/2729724390 shutdown_connections 2024-03-06T07:30:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.461+0000 7f041b000640 1 -- 172.21.15.97:0/2729724390 wait complete. 2024-03-06T07:30:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.463+0000 7f041b000640 1 Processor -- start 2024-03-06T07:30:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.463+0000 7f041b000640 1 -- start start 2024-03-06T07:30:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.464+0000 7f041b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0414070ff0 0x7f04141445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.464+0000 7f041b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 0x7f041415d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.464+0000 7f041b000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f041415fc10 0x7f041409be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.464+0000 7f041b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414072a30 con 0x7f04140b3f70 2024-03-06T07:30:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.464+0000 7f041b000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04140728b0 con 0x7f041415fc10 2024-03-06T07:30:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.464+0000 7f041b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041415c2e0 con 0x7f0414070ff0 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 0x7f041415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 0x7f041415d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59814/0 (socket says 172.21.15.97:59814) 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 -- 172.21.15.97:0/3599093710 learned_addr learned my addr 172.21.15.97:0/3599093710 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413e00640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0414070ff0 0x7f04141445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0419000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f041415fc10 0x7f041409be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 -- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0414070ff0 msgr2=0x7f04141445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0414070ff0 0x7f04141445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 -- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f041415fc10 msgr2=0x7f041409be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f041415fc10 0x7f041409be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 -- 172.21.15.97:0/3599093710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414154520 con 0x7f04140b3f70 2024-03-06T07:30:19.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.465+0000 7f0413400640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 0x7f041415d460 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f040000b450 tx=0x7f040000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.466+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400015020 con 0x7f04140b3f70 2024-03-06T07:30:19.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.466+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0400011890 con 0x7f04140b3f70 2024-03-06T07:30:19.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.466+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400011c80 con 0x7f04140b3f70 2024-03-06T07:30:19.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.466+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414154800 con 0x7f04140b3f70 2024-03-06T07:30:19.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.466+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0414154dc0 con 0x7f04140b3f70 2024-03-06T07:30:19.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.467+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041404f330 con 0x7f04140b3f70 2024-03-06T07:30:19.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.468+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0400018020 con 0x7f04140b3f70 2024-03-06T07:30:19.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.469+0000 7f040be00640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03f406b4d0 0x7f03f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.469+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f040008a430 con 0x7f04140b3f70 2024-03-06T07:30:19.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.470+0000 7f0413e00640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03f406b4d0 0x7f03f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.470+0000 7f0413e00640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03f406b4d0 0x7f03f406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0404006fd0 tx=0x7f0404008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.475+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f040001f020 con 0x7f04140b3f70 2024-03-06T07:30:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:19.709+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f04140b2d40 con 0x7f04140b3f70 2024-03-06T07:30:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.267+0000 7f040be00640 1 -- 172.21.15.97:0/3599093710 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f040000d040 con 0x7f04140b3f70 2024-03-06T07:30:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:30:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.272+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03f406b4d0 msgr2=0x7f03f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.272+0000 7f041b000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03f406b4d0 0x7f03f406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0404006fd0 tx=0x7f0404008040 comp rx=0 tx=0).stop 2024-03-06T07:30:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.272+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 msgr2=0x7f041415d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.272+0000 7f041b000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 0x7f041415d460 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f040000b450 tx=0x7f040000b910 comp rx=0 tx=0).stop 2024-03-06T07:30:20.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.275+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 shutdown_connections 2024-03-06T07:30:20.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.275+0000 7f041b000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03f406b4d0 0x7f03f406d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.275+0000 7f041b000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f041415fc10 0x7f041409be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.275+0000 7f041b000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04140b3f70 0x7f041415d460 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.275+0000 7f041b000640 1 --2- 172.21.15.97:0/3599093710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0414070ff0 0x7f04141445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.275+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 >> 172.21.15.97:0/3599093710 conn(0x7f04140b1940 msgr2=0x7f04140acfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.277+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 shutdown_connections 2024-03-06T07:30:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.277+0000 7f041b000640 1 -- 172.21.15.97:0/3599093710 wait complete. 2024-03-06T07:30:20.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.484+0000 7f8abbe00640 1 Processor -- start 2024-03-06T07:30:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.484+0000 7f8abbe00640 1 -- start start 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8abbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8ab415eff0 0x7f8ab4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8abbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ab406e4d0 0x7f8ab406e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8abbe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab4071d30 con 0x7f8ab40af1c0 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8abbe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab406ee80 con 0x7f8ab406e4d0 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8abbe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab4071260 con 0x7f8ab415eff0 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8ab9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ab406e4d0 0x7f8ab406e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8ab9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ab406e4d0 0x7f8ab406e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59822/0 (socket says 172.21.15.97:59822) 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8ab9e00640 1 -- 172.21.15.97:0/421354395 learned_addr learned my addr 172.21.15.97:0/421354395 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.486+0000 7f8ab9400640 1 --1- 172.21.15.97:0/421354395 >> v1:172.21.15.97:6789/0 conn(0x7f8ab40af1c0 0x7f8ab406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39086/0 (socket says 172.21.15.97:39086) 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.487+0000 7f8ab8a00640 1 --2- 172.21.15.97:0/421354395 >> v2:172.21.15.135:3300/0 conn(0x7f8ab415eff0 0x7f8ab4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.487+0000 7f8ab9e00640 1 -- 172.21.15.97:0/421354395 >> v2:172.21.15.135:3300/0 conn(0x7f8ab415eff0 msgr2=0x7f8ab4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.487+0000 7f8ab9e00640 1 --2- 172.21.15.97:0/421354395 >> v2:172.21.15.135:3300/0 conn(0x7f8ab415eff0 0x7f8ab4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.487+0000 7f8ab9e00640 1 -- 172.21.15.97:0/421354395 >> v1:172.21.15.97:6789/0 conn(0x7f8ab40af1c0 legacy=0x7f8ab406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.487+0000 7f8ab9e00640 1 -- 172.21.15.97:0/421354395 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab415f4a0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.487+0000 7f8ab3e00640 1 -- 172.21.15.97:0/421354395 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221742987 0 0) 0x7f8aa8001490 con 0x7f8ab40af1c0 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.488+0000 7f8ab9e00640 1 --2- 172.21.15.97:0/421354395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab406e940 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009870 tx=0x7f8aac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf787104faa90c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.488+0000 7f8ab3e00640 1 -- 172.21.15.97:0/421354395 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac034070 con 0x7f8ab406e4d0 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.488+0000 7f8ab3e00640 1 -- 172.21.15.97:0/421354395 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac030860 con 0x7f8ab406e4d0 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.488+0000 7f8ab3e00640 1 -- 172.21.15.97:0/421354395 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac030c30 con 0x7f8ab406e4d0 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 -- 172.21.15.97:0/421354395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 msgr2=0x7f8ab406e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 --2- 172.21.15.97:0/421354395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab406e940 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009870 tx=0x7f8aac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 -- 172.21.15.97:0/421354395 shutdown_connections 2024-03-06T07:30:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 --2- 172.21.15.97:0/421354395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab406e940 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 --2- 172.21.15.97:0/421354395 >> v2:172.21.15.135:3300/0 conn(0x7f8ab415eff0 0x7f8ab4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 -- 172.21.15.97:0/421354395 >> 172.21.15.97:0/421354395 conn(0x7f8ab40b1730 msgr2=0x7f8ab40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 -- 172.21.15.97:0/421354395 shutdown_connections 2024-03-06T07:30:20.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.489+0000 7f8abbe00640 1 -- 172.21.15.97:0/421354395 wait complete. 2024-03-06T07:30:20.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.490+0000 7f8abbe00640 1 Processor -- start 2024-03-06T07:30:20.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 -- start start 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab41f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ab40af1c0 0x7f8ab415f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ab415eff0 0x7f8ab415feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4071a50 con 0x7f8ab406e4d0 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4071770 con 0x7f8ab415eff0 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.491+0000 7f8abbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40b30b0 con 0x7f8ab40af1c0 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab41f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab41f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59828/0 (socket says 172.21.15.97:59828) 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 -- 172.21.15.97:0/1222775282 learned_addr learned my addr 172.21.15.97:0/1222775282 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 -- 172.21.15.97:0/1222775282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ab40af1c0 msgr2=0x7f8ab415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ab40af1c0 0x7f8ab415f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 -- 172.21.15.97:0/1222775282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ab415eff0 msgr2=0x7f8ab415feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ab415eff0 0x7f8ab415feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 -- 172.21.15.97:0/1222775282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab41f6ca0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.492+0000 7f8ab9400640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab41f6760 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800c850 tx=0x7f8aa800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.493+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8018070 con 0x7f8ab406e4d0 2024-03-06T07:30:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.493+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa800a640 con 0x7f8ab406e4d0 2024-03-06T07:30:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.493+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa800aa70 con 0x7f8ab406e4d0 2024-03-06T07:30:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.493+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab40cf3a0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.493+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab40ceb00 con 0x7f8ab406e4d0 2024-03-06T07:30:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.494+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab41f39e0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.495+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8aa800ac10 con 0x7f8ab406e4d0 2024-03-06T07:30:20.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.496+0000 7f8ab1600640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a9406b500 0x7f8a9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.496+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8aa80898d0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.496+0000 7f8ab8a00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a9406b500 0x7f8a9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.497+0000 7f8ab8a00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a9406b500 0x7f8a9406d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0006fd0 tx=0x7f8aa0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.501+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ab41f39e0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.691+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ab4074be0 con 0x7f8ab406e4d0 2024-03-06T07:30:20.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.692+0000 7f8ab1600640 1 -- 172.21.15.97:0/1222775282 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f8ab4071d30 con 0x7f8ab406e4d0 2024-03-06T07:30:20.695 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:20.695 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":57,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:19.709871+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:20.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a9406b500 msgr2=0x7f8a9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a9406b500 0x7f8a9406d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0006fd0 tx=0x7f8aa0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:20.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 msgr2=0x7f8ab41f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab41f6760 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800c850 tx=0x7f8aa800cd10 comp rx=0 tx=0).stop 2024-03-06T07:30:20.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 shutdown_connections 2024-03-06T07:30:20.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a9406b500 0x7f8a9406d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ab415eff0 0x7f8ab415feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ab40af1c0 0x7f8ab415f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 --2- 172.21.15.97:0/1222775282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ab406e4d0 0x7f8ab41f6760 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 >> 172.21.15.97:0/1222775282 conn(0x7f8ab40b1730 msgr2=0x7f8ab41610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 shutdown_connections 2024-03-06T07:30:20.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.695+0000 7f8abbe00640 1 -- 172.21.15.97:0/1222775282 wait complete. 2024-03-06T07:30:20.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.914+0000 7f54e0800640 1 Processor -- start 2024-03-06T07:30:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.914+0000 7f54e0800640 1 -- start start 2024-03-06T07:30:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.915+0000 7f54e0800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 0x7f54d8071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.915+0000 7f54e0800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d8071a60 con 0x7f54d815dff0 2024-03-06T07:30:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.915+0000 7f54e0800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d80b35b0 con 0x7f54d80cf430 2024-03-06T07:30:20.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.915+0000 7f54e0800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80b3ee0 con 0x7f54d8151660 2024-03-06T07:30:20.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.915+0000 7f54de800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f54d815dff0 0x7f54d80b4370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39098/0 (socket says 172.21.15.97:39098) 2024-03-06T07:30:20.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54de800640 1 -- 172.21.15.97:0/2351198025 learned_addr learned my addr 172.21.15.97:0/2351198025 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:20.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dde00640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 0x7f54d8071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123992585 0 0) 0x7f54d8071a60 con 0x7f54d815dff0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c0005730 con 0x7f54d815dff0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3069604349 0 0) 0x7f54c0005730 con 0x7f54d815dff0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 msgr2=0x7f54d8071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 0x7f54d8071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 >> v1:172.21.15.135:6789/0 conn(0x7f54d80cf430 legacy=0x7f54d806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d80b3890 con 0x7f54d815dff0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f54d40019f0 con 0x7f54d815dff0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 >> v1:172.21.15.97:6789/0 conn(0x7f54d815dff0 legacy=0x7f54d80b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 0x7f54c0005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54c0006220 0x7f54c0006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c0006fa0 0x7f54c0007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.916+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80b3ee0 con 0x7f54c0006fa0 2024-03-06T07:30:20.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c000deb0 con 0x7f54c0006220 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c000e0b0 con 0x7f54d8151660 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54de800640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c0006fa0 0x7f54c0007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54dde00640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 0x7f54c0005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54dd400640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54c0006220 0x7f54c0006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54de800640 1 -- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 msgr2=0x7f54c0005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54de800640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d8151660 0x7f54c0005bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54de800640 1 -- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54c0006220 msgr2=0x7f54c0006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54de800640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54c0006220 0x7f54c0006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54de800640 1 -- 172.21.15.97:0/2351198025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c0005730 con 0x7f54c0006fa0 2024-03-06T07:30:20.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.917+0000 7f54dd400640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54c0006220 0x7f54c0006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:20.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.918+0000 7f54de800640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c0006fa0 0x7f54c0007420 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f54d4002230 tx=0x7f54c0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b8d4e6d4623134 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d400a4a0 con 0x7f54c0006fa0 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4034650 con 0x7f54c0006fa0 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54dca00640 1 -- 172.21.15.97:0/2351198025 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4034a80 con 0x7f54c0006fa0 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54e0800640 1 -- 172.21.15.97:0/2351198025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c0006fa0 msgr2=0x7f54c0007420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54e0800640 1 --2- 172.21.15.97:0/2351198025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c0006fa0 0x7f54c0007420 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f54d4002230 tx=0x7f54c0007a40 comp rx=0 tx=0).stop 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54de800640 1 -- 172.21.15.97:0/2351198025 reap_dead start 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54e0800640 1 -- 172.21.15.97:0/2351198025 shutdown_connections 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54e0800640 1 -- 172.21.15.97:0/2351198025 >> 172.21.15.97:0/2351198025 conn(0x7f54d80b1730 msgr2=0x7f54d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54e0800640 1 -- 172.21.15.97:0/2351198025 shutdown_connections 2024-03-06T07:30:20.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.919+0000 7f54e0800640 1 -- 172.21.15.97:0/2351198025 wait complete. 2024-03-06T07:30:20.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.921+0000 7f54e0800640 1 Processor -- start 2024-03-06T07:30:20.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.921+0000 7f54e0800640 1 -- start start 2024-03-06T07:30:20.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54e0800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 0x7f54d8151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54e0800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d80cc710 0x7f54d81f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54e0800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54d81f4300 0x7f54d81f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54e0800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d806daf0 con 0x7f54d8146bd0 2024-03-06T07:30:20.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54e0800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d8071d30 con 0x7f54d81f4300 2024-03-06T07:30:20.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54e0800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80b4a80 con 0x7f54d80cc710 2024-03-06T07:30:20.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54dde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 0x7f54d8151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54dde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 0x7f54d8151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59858/0 (socket says 172.21.15.97:59858) 2024-03-06T07:30:20.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.922+0000 7f54dde00640 1 -- 172.21.15.97:0/4107690073 learned_addr learned my addr 172.21.15.97:0/4107690073 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54dde00640 1 -- 172.21.15.97:0/4107690073 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d80cc710 msgr2=0x7f54d81f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54dde00640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d80cc710 0x7f54d81f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54dde00640 1 -- 172.21.15.97:0/4107690073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54d81f4300 msgr2=0x7f54d81f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54dde00640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54d81f4300 0x7f54d81f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54dde00640 1 -- 172.21.15.97:0/4107690073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d81533a0 con 0x7f54d8146bd0 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54dde00640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 0x7f54d8151a40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00e7e0 tx=0x7f54cc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc017070 con 0x7f54d8146bd0 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc00c650 con 0x7f54d8146bd0 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00f040 con 0x7f54d8146bd0 2024-03-06T07:30:20.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d814b680 con 0x7f54d8146bd0 2024-03-06T07:30:20.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.923+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d8153620 con 0x7f54d8146bd0 2024-03-06T07:30:20.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.924+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d804f330 con 0x7f54d8146bd0 2024-03-06T07:30:20.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.925+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54cc01a070 con 0x7f54d8146bd0 2024-03-06T07:30:20.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.926+0000 7f54d2000640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f54bc06b4d0 0x7f54bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.926+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54cc089e00 con 0x7f54d8146bd0 2024-03-06T07:30:20.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.926+0000 7f54dd400640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f54bc06b4d0 0x7f54bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.927+0000 7f54dd400640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f54bc06b4d0 0x7f54bc06d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f54d815b550 tx=0x7f54c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:20.931+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54d81533a0 con 0x7f54d8146bd0 2024-03-06T07:30:21.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.127+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54d815b2a0 con 0x7f54d8146bd0 2024-03-06T07:30:21.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.130+0000 7f54d2000640 1 -- 172.21.15.97:0/4107690073 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f54cc058c70 con 0x7f54d8146bd0 2024-03-06T07:30:21.133 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:21.133 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":57,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:19.709871+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:21.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.133+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f54bc06b4d0 msgr2=0x7f54bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.133+0000 7f54e0800640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f54bc06b4d0 0x7f54bc06d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f54d815b550 tx=0x7f54c8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.133+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 msgr2=0x7f54d8151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.133+0000 7f54e0800640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 0x7f54d8151a40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00e7e0 tx=0x7f54cc00eca0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 shutdown_connections 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f54bc06b4d0 0x7f54bc06d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54d81f4300 0x7f54d81f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54d80cc710 0x7f54d81f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 --2- 172.21.15.97:0/4107690073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54d8146bd0 0x7f54d8151a40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 >> 172.21.15.97:0/4107690073 conn(0x7f54d80b1730 msgr2=0x7f54d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 shutdown_connections 2024-03-06T07:30:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.134+0000 7f54e0800640 1 -- 172.21.15.97:0/4107690073 wait complete. 2024-03-06T07:30:21.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:21.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.367+0000 7f1fe4a00640 1 Processor -- start 2024-03-06T07:30:21.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.367+0000 7f1fe4a00640 1 -- start start 2024-03-06T07:30:21.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 0x7f1fe00d39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fe4a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1fe00d3ee0 0x7f1fe00b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fe4a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00d4370 con 0x7f1fe015e000 2024-03-06T07:30:21.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fe4a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00cf020 con 0x7f1fe00d3ee0 2024-03-06T07:30:21.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cf300 con 0x7f1fe00ce890 2024-03-06T07:30:21.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fde800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f1fe015e000 0x7f1fe00d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39114/0 (socket says 172.21.15.97:39114) 2024-03-06T07:30:21.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.368+0000 7f1fde800640 1 -- 172.21.15.97:0/204068312 learned_addr learned my addr 172.21.15.97:0/204068312 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292472309 0 0) 0x7f1fe00d4370 con 0x7f1fe015e000 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc4003520 con 0x7f1fe015e000 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526682899 0 0) 0x7f1fc4003520 con 0x7f1fe015e000 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 msgr2=0x7f1fe00d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdde00640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 0x7f1fe00d39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdf200640 1 --2- 172.21.15.97:0/204068312 >> v2:172.21.15.135:3300/0 conn(0x7f1fe00d3ee0 0x7f1fe00b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 0x7f1fe00d39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 >> v2:172.21.15.135:3300/0 conn(0x7f1fe00d3ee0 msgr2=0x7f1fe00b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 --2- 172.21.15.97:0/204068312 >> v2:172.21.15.135:3300/0 conn(0x7f1fe00d3ee0 0x7f1fe00b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00cf5e0 con 0x7f1fe015e000 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1fd0001670 con 0x7f1fe015e000 2024-03-06T07:30:21.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 >> v1:172.21.15.97:6789/0 conn(0x7f1fe015e000 legacy=0x7f1fe00d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fc40039b0 0x7f1fc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 0x7f1fc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc4004d90 0x7f1fc4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.369+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cf020 con 0x7f1fc40039b0 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cf300 con 0x7f1fc4004d90 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc400bc20 con 0x7f1fe00ce890 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fde800640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fc40039b0 0x7f1fc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fde800640 1 -- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 msgr2=0x7f1fc4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fde800640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe00ce890 0x7f1fc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fde800640 1 -- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc4004d90 msgr2=0x7f1fc4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fde800640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc4004d90 0x7f1fc4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.370+0000 7f1fde800640 1 -- 172.21.15.97:0/204068312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4003520 con 0x7f1fc40039b0 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.372+0000 7f1fde800640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fc40039b0 0x7f1fc4003dd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c210 tx=0x7f1fc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1823de79d53717ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.372+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0007d50 con 0x7f1fc40039b0 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.372+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0030070 con 0x7f1fc40039b0 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.372+0000 7f1fdd400640 1 -- 172.21.15.97:0/204068312 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd002cd80 con 0x7f1fc40039b0 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fe4a00640 1 -- 172.21.15.97:0/204068312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fc40039b0 msgr2=0x7f1fc4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/204068312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fc40039b0 0x7f1fc4003dd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c210 tx=0x7f1fc4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fdf200640 1 -- 172.21.15.97:0/204068312 reap_dead start 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fe4a00640 1 -- 172.21.15.97:0/204068312 shutdown_connections 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fe4a00640 1 -- 172.21.15.97:0/204068312 >> 172.21.15.97:0/204068312 conn(0x7f1fe008d740 msgr2=0x7f1fe009d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fe4a00640 1 -- 172.21.15.97:0/204068312 shutdown_connections 2024-03-06T07:30:21.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.373+0000 7f1fe4a00640 1 -- 172.21.15.97:0/204068312 wait complete. 2024-03-06T07:30:21.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.374+0000 7f1fe4a00640 1 Processor -- start 2024-03-06T07:30:21.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.375+0000 7f1fe4a00640 1 -- start start 2024-03-06T07:30:21.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.375+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc40039b0 0x7f1fe00b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe0156930 0x7f1fe0157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 0x7f1fe00cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cecf0 con 0x7f1fe0152200 2024-03-06T07:30:21.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cfba0 con 0x7f1fc40039b0 2024-03-06T07:30:21.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00d39a0 con 0x7f1fe0156930 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 0x7f1fe00cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 0x7f1fe00cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59882/0 (socket says 172.21.15.97:59882) 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 -- 172.21.15.97:0/177174582 learned_addr learned my addr 172.21.15.97:0/177174582 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdde00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe0156930 0x7f1fe0157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fde800640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc40039b0 0x7f1fe00b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 -- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe0156930 msgr2=0x7f1fe0157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe0156930 0x7f1fe0157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 -- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc40039b0 msgr2=0x7f1fe00b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc40039b0 0x7f1fe00b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fdf200640 1 -- 172.21.15.97:0/177174582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00b23d0 con 0x7f1fe0152200 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.376+0000 7f1fde800640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc40039b0 0x7f1fe00b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.377+0000 7f1fdf200640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 0x7f1fe00cc990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1fe008e000 tx=0x7f1fd800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.377+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8003990 con 0x7f1fe0152200 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.377+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8003b50 con 0x7f1fe0152200 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.377+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe0157460 con 0x7f1fe0152200 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.377+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8004030 con 0x7f1fe0152200 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.377+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe0157a20 con 0x7f1fe0152200 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.378+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0157c20 con 0x7f1fe0152200 2024-03-06T07:30:21.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.379+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd80041d0 con 0x7f1fe0152200 2024-03-06T07:30:21.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.380+0000 7f1fd6a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1fc006b500 0x7f1fc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.380+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd808aab0 con 0x7f1fe0152200 2024-03-06T07:30:21.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.380+0000 7f1fde800640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1fc006b500 0x7f1fc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.381+0000 7f1fde800640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1fc006b500 0x7f1fc006d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd0007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.384+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fe0157c20 con 0x7f1fe0152200 2024-03-06T07:30:21.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:21.619+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1fe004f330 con 0x7f1fe0152200 2024-03-06T07:30:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.280+0000 7f1fd6a00640 1 -- 172.21.15.97:0/177174582 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1fd801c060 con 0x7f1fe0152200 2024-03-06T07:30:22.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1fc006b500 msgr2=0x7f1fc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1fc006b500 0x7f1fc006d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd0007000 comp rx=0 tx=0).stop 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 msgr2=0x7f1fe00cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 0x7f1fe00cc990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1fe008e000 tx=0x7f1fd800a660 comp rx=0 tx=0).stop 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 shutdown_connections 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1fc006b500 0x7f1fc006d9c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1fe0152200 0x7f1fe00cc990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1fe0156930 0x7f1fe0157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 --2- 172.21.15.97:0/177174582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fc40039b0 0x7f1fe00b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.284+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 >> 172.21.15.97:0/177174582 conn(0x7f1fe008d740 msgr2=0x7f1fe0159110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.285+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 shutdown_connections 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.285+0000 7f1fe4a00640 1 -- 172.21.15.97:0/177174582 wait complete. 2024-03-06T07:30:22.307 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:22.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:22.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.501+0000 7f5d8f000640 1 Processor -- start 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.502+0000 7f5d8f000640 1 -- start start 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d8f000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d880d07f0 0x7f5d880d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d8f000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880b3730 con 0x7f5d8815e450 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d8f000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d88067b20 con 0x7f5d880d1bf0 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d8f000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880d4930 con 0x7f5d880d07f0 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d87e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d880d07f0 0x7f5d880d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d87e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d880d07f0 0x7f5d880d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59894/0 (socket says 172.21.15.97:59894) 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.503+0000 7f5d87e00640 1 -- 172.21.15.97:0/2419450747 learned_addr learned my addr 172.21.15.97:0/2419450747 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.504+0000 7f5d87e00640 1 -- 172.21.15.97:0/2419450747 >> v1:172.21.15.135:6789/0 conn(0x7f5d880d1bf0 legacy=0x7f5d880b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.504+0000 7f5d87e00640 1 -- 172.21.15.97:0/2419450747 >> v1:172.21.15.97:6789/0 conn(0x7f5d8815e450 legacy=0x7f5d880b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.504+0000 7f5d87e00640 1 -- 172.21.15.97:0/2419450747 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880d1110 con 0x7f5d880d07f0 2024-03-06T07:30:22.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.505+0000 7f5d87e00640 1 --2- 172.21.15.97:0/2419450747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d07f0 0x7f5d880d0bd0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439a898e1f10b8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d87400640 1 -- 172.21.15.97:0/2419450747 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78035020 con 0x7f5d880d07f0 2024-03-06T07:30:22.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d87400640 1 -- 172.21.15.97:0/2419450747 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78031870 con 0x7f5d880d07f0 2024-03-06T07:30:22.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d87400640 1 -- 172.21.15.97:0/2419450747 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78031c40 con 0x7f5d880d07f0 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d8f000640 1 -- 172.21.15.97:0/2419450747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d07f0 msgr2=0x7f5d880d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d8f000640 1 --2- 172.21.15.97:0/2419450747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d07f0 0x7f5d880d0bd0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d8f000640 1 -- 172.21.15.97:0/2419450747 shutdown_connections 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d8f000640 1 --2- 172.21.15.97:0/2419450747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d07f0 0x7f5d880d0bd0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d8f000640 1 -- 172.21.15.97:0/2419450747 >> 172.21.15.97:0/2419450747 conn(0x7f5d880d3e10 msgr2=0x7f5d880d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.506+0000 7f5d8f000640 1 -- 172.21.15.97:0/2419450747 shutdown_connections 2024-03-06T07:30:22.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.507+0000 7f5d8f000640 1 -- 172.21.15.97:0/2419450747 wait complete. 2024-03-06T07:30:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.509+0000 7f5d8f000640 1 Processor -- start 2024-03-06T07:30:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.509+0000 7f5d8f000640 1 -- start start 2024-03-06T07:30:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.510+0000 7f5d8f000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d880d07f0 0x7f5d8815f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.510+0000 7f5d8f000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 0x7f5d8806be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.510+0000 7f5d8f000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8815e450 0x7f5d8806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.510+0000 7f5d8f000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b3c80 con 0x7f5d880d1bf0 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.510+0000 7f5d8f000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b39a0 con 0x7f5d8815e450 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.510+0000 7f5d8f000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b3f60 con 0x7f5d880d07f0 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 0x7f5d8806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 0x7f5d8806be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59904/0 (socket says 172.21.15.97:59904) 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 -- 172.21.15.97:0/1446458139 learned_addr learned my addr 172.21.15.97:0/1446458139 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d87e00640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d880d07f0 0x7f5d8815f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d8d000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8815e450 0x7f5d8806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 -- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d880d07f0 msgr2=0x7f5d8815f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d880d07f0 0x7f5d8815f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 -- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8815e450 msgr2=0x7f5d8806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8815e450 0x7f5d8806c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.511+0000 7f5d7f400640 1 -- 172.21.15.97:0/1446458139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d88160820 con 0x7f5d880d1bf0 2024-03-06T07:30:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.512+0000 7f5d7f400640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 0x7f5d8806be80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c3c0 tx=0x7f5d7000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.512+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d700052d0 con 0x7f5d880d1bf0 2024-03-06T07:30:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.512+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d70005490 con 0x7f5d880d1bf0 2024-03-06T07:30:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.512+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70007260 con 0x7f5d880d1bf0 2024-03-06T07:30:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.512+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d88160b00 con 0x7f5d880d1bf0 2024-03-06T07:30:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.512+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d881610c0 con 0x7f5d880d1bf0 2024-03-06T07:30:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.514+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8804f330 con 0x7f5d880d1bf0 2024-03-06T07:30:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.514+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d70007400 con 0x7f5d880d1bf0 2024-03-06T07:30:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.515+0000 7f5d84c00640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.515+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d700896e0 con 0x7f5d880d1bf0 2024-03-06T07:30:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.515+0000 7f5d87e00640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.516+0000 7f5d87e00640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5d78036000 tx=0x7f5d7803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.520+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d70058560 con 0x7f5d880d1bf0 2024-03-06T07:30:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:22.763+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5d880b1900 con 0x7f5d880d1bf0 2024-03-06T07:30:23.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.290+0000 7f5d84c00640 1 -- 172.21.15.97:0/1446458139 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5d70057e60 con 0x7f5d880d1bf0 2024-03-06T07:30:23.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 msgr2=0x7f5d6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5d78036000 tx=0x7f5d7803c040 comp rx=0 tx=0).stop 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 msgr2=0x7f5d8806be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 0x7f5d8806be80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c3c0 tx=0x7f5d7000c7c0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 shutdown_connections 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8815e450 0x7f5d8806c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d880d1bf0 0x7f5d8806be80 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 --2- 172.21.15.97:0/1446458139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d880d07f0 0x7f5d8815f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.294+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 >> 172.21.15.97:0/1446458139 conn(0x7f5d880d3e10 msgr2=0x7f5d880b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.295+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 shutdown_connections 2024-03-06T07:30:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.295+0000 7f5d8f000640 1 -- 172.21.15.97:0/1446458139 wait complete. 2024-03-06T07:30:23.321 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:23.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.525+0000 7feaeb600640 1 Processor -- start 2024-03-06T07:30:23.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.525+0000 7feaeb600640 1 -- start start 2024-03-06T07:30:23.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.526+0000 7feaeb600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feae415acb0 0x7feae40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.526+0000 7feaeb600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae415b090 con 0x7feae406ec00 2024-03-06T07:30:23.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feaeb600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40b2a30 con 0x7feae40b4680 2024-03-06T07:30:23.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feaeb600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae40b4360 con 0x7feae415acb0 2024-03-06T07:30:23.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feae8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feae415acb0 0x7feae40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feae8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feae415acb0 0x7feae40b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59916/0 (socket says 172.21.15.97:59916) 2024-03-06T07:30:23.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feae8c00640 1 -- 172.21.15.97:0/2269481448 learned_addr learned my addr 172.21.15.97:0/2269481448 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:23.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feae8c00640 1 -- 172.21.15.97:0/2269481448 >> v1:172.21.15.135:6789/0 conn(0x7feae40b4680 legacy=0x7feae406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feae8c00640 1 -- 172.21.15.97:0/2269481448 >> v1:172.21.15.97:6789/0 conn(0x7feae406ec00 legacy=0x7feae40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.527+0000 7feae8c00640 1 -- 172.21.15.97:0/2269481448 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae40b2cf0 con 0x7feae415acb0 2024-03-06T07:30:23.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feae8c00640 1 --2- 172.21.15.97:0/2269481448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae415acb0 0x7feae40b37d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fead400bb10 tx=0x7fead402e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f2863050c803f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feae3e00640 1 -- 172.21.15.97:0/2269481448 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead403b070 con 0x7feae415acb0 2024-03-06T07:30:23.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feae3e00640 1 -- 172.21.15.97:0/2269481448 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead4036440 con 0x7feae415acb0 2024-03-06T07:30:23.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feae3e00640 1 -- 172.21.15.97:0/2269481448 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4036810 con 0x7feae415acb0 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feaeb600640 1 -- 172.21.15.97:0/2269481448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae415acb0 msgr2=0x7feae40b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feaeb600640 1 --2- 172.21.15.97:0/2269481448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae415acb0 0x7feae40b37d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fead400bb10 tx=0x7fead402e160 comp rx=0 tx=0).stop 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feaeb600640 1 -- 172.21.15.97:0/2269481448 shutdown_connections 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feaeb600640 1 --2- 172.21.15.97:0/2269481448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae415acb0 0x7feae40b37d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.529+0000 7feaeb600640 1 -- 172.21.15.97:0/2269481448 >> 172.21.15.97:0/2269481448 conn(0x7feae40b1730 msgr2=0x7feae40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.530+0000 7feaeb600640 1 -- 172.21.15.97:0/2269481448 shutdown_connections 2024-03-06T07:30:23.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.530+0000 7feaeb600640 1 -- 172.21.15.97:0/2269481448 wait complete. 2024-03-06T07:30:23.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.531+0000 7feaeb600640 1 Processor -- start 2024-03-06T07:30:23.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.531+0000 7feaeb600640 1 -- start start 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feaeb600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feae406ec00 0x7feae415b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feaeb600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 0x7feae415d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feaeb600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feae415acb0 0x7feae415db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feaeb600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae40b3d10 con 0x7feae40b4680 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feaeb600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae40b32b0 con 0x7feae406ec00 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feaeb600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae40b4010 con 0x7feae415acb0 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feadbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 0x7feae415d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feadbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 0x7feae415d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59932/0 (socket says 172.21.15.97:59932) 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.533+0000 7feadbe00640 1 -- 172.21.15.97:0/329028945 learned_addr learned my addr 172.21.15.97:0/329028945 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feae8c00640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feae406ec00 0x7feae415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feae9600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feae415acb0 0x7feae415db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feadbe00640 1 -- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feae415acb0 msgr2=0x7feae415db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feadbe00640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feae415acb0 0x7feae415db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feadbe00640 1 -- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feae406ec00 msgr2=0x7feae415b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feadbe00640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feae406ec00 0x7feae415b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feadbe00640 1 -- 172.21.15.97:0/329028945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae40c9480 con 0x7feae40b4680 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feae9600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feae415acb0 0x7feae415db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feadbe00640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 0x7feae415d600 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7feacc00c410 tx=0x7feacc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.534+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc017070 con 0x7feae40b4680 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.535+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc0035f0 con 0x7feae40b4680 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.535+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc003a20 con 0x7feae40b4680 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.535+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae40c9760 con 0x7feae40b4680 2024-03-06T07:30:23.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.535+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae41521e0 con 0x7feae40b4680 2024-03-06T07:30:23.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.536+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae404f330 con 0x7feae40b4680 2024-03-06T07:30:23.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.537+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feacc003bc0 con 0x7feae40b4680 2024-03-06T07:30:23.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.538+0000 7feae1600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feac006b4d0 0x7feac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.538+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feacc0895c0 con 0x7feae40b4680 2024-03-06T07:30:23.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.538+0000 7feae8c00640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feac006b4d0 0x7feac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.539+0000 7feae8c00640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feac006b4d0 0x7feac006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fead4038000 tx=0x7fead4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.542+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feacc058440 con 0x7feae40b4680 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.775+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feae406ce60 con 0x7feae40b4680 2024-03-06T07:30:23.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.777+0000 7feae1600640 1 -- 172.21.15.97:0/329028945 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7feacc0010c0 con 0x7feae40b4680 2024-03-06T07:30:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:23.779 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:23.779 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feac006b4d0 msgr2=0x7feac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feac006b4d0 0x7feac006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fead4038000 tx=0x7fead4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 msgr2=0x7feae415d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 0x7feae415d600 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7feacc00c410 tx=0x7feacc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 shutdown_connections 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feac006b4d0 0x7feac006d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feae415acb0 0x7feae415db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feae40b4680 0x7feae415d600 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 --2- 172.21.15.97:0/329028945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feae406ec00 0x7feae415b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.779+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 >> 172.21.15.97:0/329028945 conn(0x7feae40b1730 msgr2=0x7feae40cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.780+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 shutdown_connections 2024-03-06T07:30:23.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.780+0000 7feaeb600640 1 -- 172.21.15.97:0/329028945 wait complete. 2024-03-06T07:30:23.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:24.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.998+0000 7f8979200640 1 Processor -- start 2024-03-06T07:30:24.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:23.998+0000 7f8979200640 1 -- start start 2024-03-06T07:30:24.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8979200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 0x7f897406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8979200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f897415f9d0 0x7f897415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8979200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897406ee70 con 0x7f89740ce0a0 2024-03-06T07:30:24.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8979200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8974160370 con 0x7f897415f9d0 2024-03-06T07:30:24.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8979200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897406cef0 con 0x7f8974144c00 2024-03-06T07:30:24.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8972800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f89740ce0a0 0x7f89740ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46276/0 (socket says 172.21.15.97:46276) 2024-03-06T07:30:24.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.000+0000 7f8972800640 1 -- 172.21.15.97:0/3912783998 learned_addr learned my addr 172.21.15.97:0/3912783998 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:24.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971e00640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 0x7f897406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f896be00640 1 --2- 172.21.15.97:0/3912783998 >> v2:172.21.15.135:3300/0 conn(0x7f897415f9d0 0x7f897415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950825399 0 0) 0x7f896c001490 con 0x7f89740ce0a0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8958003520 con 0x7f89740ce0a0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2785993661 0 0) 0x7f8958003520 con 0x7f89740ce0a0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 msgr2=0x7f897406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 0x7f897406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 >> v2:172.21.15.135:3300/0 conn(0x7f897415f9d0 msgr2=0x7f897415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 --2- 172.21.15.97:0/3912783998 >> v2:172.21.15.135:3300/0 conn(0x7f897415f9d0 0x7f897415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897406d1d0 con 0x7f89740ce0a0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f896c001bc0 con 0x7f89740ce0a0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 >> v1:172.21.15.97:6789/0 conn(0x7f89740ce0a0 legacy=0x7f89740ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89580039b0 0x7f8958003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8958004800 0x7f8958004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 0x7f89580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897406cef0 con 0x7f89580039b0 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895800b9c0 con 0x7f8958004800 2024-03-06T07:30:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.001+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895800bb40 con 0x7f8974144c00 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8971e00640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89580039b0 0x7f8958003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8972800640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 0x7f89580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f896be00640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8958004800 0x7f8958004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8971e00640 1 -- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 msgr2=0x7f89580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8971e00640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8974144c00 0x7f89580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8971e00640 1 -- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8958004800 msgr2=0x7f8958004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8971e00640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8958004800 0x7f8958004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.002+0000 7f8971e00640 1 -- 172.21.15.97:0/3912783998 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8958003520 con 0x7f89580039b0 2024-03-06T07:30:24.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.003+0000 7f8971e00640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89580039b0 0x7f8958003dd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8964009ac0 tx=0x7f8958005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8846fa5bb6d5482 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964039070 con 0x7f89580039b0 2024-03-06T07:30:24.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896402c6b0 con 0x7f89580039b0 2024-03-06T07:30:24.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8971400640 1 -- 172.21.15.97:0/3912783998 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964009690 con 0x7f89580039b0 2024-03-06T07:30:24.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8979200640 1 -- 172.21.15.97:0/3912783998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89580039b0 msgr2=0x7f8958003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8979200640 1 --2- 172.21.15.97:0/3912783998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89580039b0 0x7f8958003dd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8964009ac0 tx=0x7f8958005830 comp rx=0 tx=0).stop 2024-03-06T07:30:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8972800640 1 -- 172.21.15.97:0/3912783998 reap_dead start 2024-03-06T07:30:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8979200640 1 -- 172.21.15.97:0/3912783998 shutdown_connections 2024-03-06T07:30:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8979200640 1 -- 172.21.15.97:0/3912783998 >> 172.21.15.97:0/3912783998 conn(0x7f89740b1730 msgr2=0x7f89740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8979200640 1 -- 172.21.15.97:0/3912783998 shutdown_connections 2024-03-06T07:30:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.004+0000 7f8979200640 1 -- 172.21.15.97:0/3912783998 wait complete. 2024-03-06T07:30:24.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.007+0000 7f8979200640 1 Processor -- start 2024-03-06T07:30:24.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.007+0000 7f8979200640 1 -- start start 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f8979200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89580039b0 0x7f897415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f8979200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 0x7f89741e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f8979200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89740adb00 0x7f89740adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f8979200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8974160730 con 0x7f89741e55c0 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f8979200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897406d790 con 0x7f89580039b0 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f8979200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897406cb50 con 0x7f89740adb00 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f896be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 0x7f89741e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f896be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 0x7f89741e5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52488/0 (socket says 172.21.15.97:52488) 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.009+0000 7f896be00640 1 -- 172.21.15.97:0/1740371022 learned_addr learned my addr 172.21.15.97:0/1740371022 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896be00640 1 -- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89740adb00 msgr2=0x7f89740adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f8971e00640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89580039b0 0x7f897415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896be00640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89740adb00 0x7f89740adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896be00640 1 -- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89580039b0 msgr2=0x7f897415dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896be00640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89580039b0 0x7f897415dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896be00640 1 -- 172.21.15.97:0/1740371022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897414e170 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896be00640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 0x7f89741e5a40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f895c011700 tx=0x7f895c011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c00f960 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c00fb00 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89740b2bd0 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.010+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c015460 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.011+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89740b3190 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.012+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897404f330 con 0x7f89741e55c0 2024-03-06T07:30:24.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.012+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f895c015600 con 0x7f89741e55c0 2024-03-06T07:30:24.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.014+0000 7f896a000640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f895006b4d0 0x7f895006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.014+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f895c08ab10 con 0x7f89741e55c0 2024-03-06T07:30:24.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.014+0000 7f8971e00640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f895006b4d0 0x7f895006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.015+0000 7f8971e00640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f895006b4d0 0x7f895006d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8964036000 tx=0x7f8964030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.018+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f895c019030 con 0x7f89741e55c0 2024-03-06T07:30:24.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.336+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8974072890 con 0x7f89741e55c0 2024-03-06T07:30:24.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.338+0000 7f896a000640 1 -- 172.21.15.97:0/1740371022 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f895c059930 con 0x7f89741e55c0 2024-03-06T07:30:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:24.341 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":58,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:21.268327+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.340+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f895006b4d0 msgr2=0x7f895006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.340+0000 7f8979200640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f895006b4d0 0x7f895006d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8964036000 tx=0x7f8964030000 comp rx=0 tx=0).stop 2024-03-06T07:30:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 msgr2=0x7f89741e5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 0x7f89741e5a40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f895c011700 tx=0x7f895c011bc0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 shutdown_connections 2024-03-06T07:30:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f895006b4d0 0x7f895006d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89740adb00 0x7f89740adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89741e55c0 0x7f89741e5a40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 --2- 172.21.15.97:0/1740371022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89580039b0 0x7f897415dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 >> 172.21.15.97:0/1740371022 conn(0x7f89740b1730 msgr2=0x7f89740b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 shutdown_connections 2024-03-06T07:30:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.341+0000 7f8979200640 1 -- 172.21.15.97:0/1740371022 wait complete. 2024-03-06T07:30:24.362 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:24.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.558+0000 7f4eec000640 1 Processor -- start 2024-03-06T07:30:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.558+0000 7f4eec000640 1 -- start start 2024-03-06T07:30:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.559+0000 7f4eec000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4ee4144c00 0x7f4ee406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.559+0000 7f4eec000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4ee415f9d0 0x7f4ee415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.559+0000 7f4eec000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee406ee70 con 0x7f4ee40ce0a0 2024-03-06T07:30:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.559+0000 7f4eec000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee406cef0 con 0x7f4ee415f9d0 2024-03-06T07:30:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.559+0000 7f4eec000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee406d1d0 con 0x7f4ee4144c00 2024-03-06T07:30:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4ee415f9d0 0x7f4ee415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4eea000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4ee40ce0a0 0x7f4ee40ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46280/0 (socket says 172.21.15.97:46280) 2024-03-06T07:30:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4eea000640 1 -- 172.21.15.97:0/1212972585 learned_addr learned my addr 172.21.15.97:0/1212972585 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee9600640 1 --2- 172.21.15.97:0/1212972585 >> v2:172.21.15.135:3300/0 conn(0x7f4ee4144c00 0x7f4ee406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee8c00640 1 -- 172.21.15.97:0/1212972585 >> v2:172.21.15.135:3300/0 conn(0x7f4ee4144c00 msgr2=0x7f4ee406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee8c00640 1 --2- 172.21.15.97:0/1212972585 >> v2:172.21.15.135:3300/0 conn(0x7f4ee4144c00 0x7f4ee406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee8c00640 1 -- 172.21.15.97:0/1212972585 >> v1:172.21.15.97:6789/0 conn(0x7f4ee40ce0a0 legacy=0x7f4ee40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee8c00640 1 -- 172.21.15.97:0/1212972585 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee406d4b0 con 0x7f4ee415f9d0 2024-03-06T07:30:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.560+0000 7f4ee3e00640 1 -- 172.21.15.97:0/1212972585 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002020736 0 0) 0x7f4edc001490 con 0x7f4ee40ce0a0 2024-03-06T07:30:24.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.561+0000 7f4ee8c00640 1 --2- 172.21.15.97:0/1212972585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415fe60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40b25f0 tx=0x7f4ed002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7e93c96c3786b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.561+0000 7f4ee3e00640 1 -- 172.21.15.97:0/1212972585 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0035020 con 0x7f4ee415f9d0 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4ee3e00640 1 -- 172.21.15.97:0/1212972585 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0031830 con 0x7f4ee415f9d0 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4ee3e00640 1 -- 172.21.15.97:0/1212972585 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0031c00 con 0x7f4ee415f9d0 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 -- 172.21.15.97:0/1212972585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 msgr2=0x7f4ee415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 --2- 172.21.15.97:0/1212972585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415fe60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40b25f0 tx=0x7f4ed002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 -- 172.21.15.97:0/1212972585 shutdown_connections 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 --2- 172.21.15.97:0/1212972585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415fe60 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 --2- 172.21.15.97:0/1212972585 >> v2:172.21.15.135:3300/0 conn(0x7f4ee4144c00 0x7f4ee406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 -- 172.21.15.97:0/1212972585 >> 172.21.15.97:0/1212972585 conn(0x7f4ee40b1730 msgr2=0x7f4ee40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 -- 172.21.15.97:0/1212972585 shutdown_connections 2024-03-06T07:30:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.562+0000 7f4eec000640 1 -- 172.21.15.97:0/1212972585 wait complete. 2024-03-06T07:30:24.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.564+0000 7f4eec000640 1 Processor -- start 2024-03-06T07:30:24.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.564+0000 7f4eec000640 1 -- start start 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eec000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ee40ce0a0 0x7f4ee413c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eec000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ee4144c00 0x7f4ee41f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eec000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eec000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee406cb50 con 0x7f4ee415f9d0 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eec000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee4160730 con 0x7f4ee40ce0a0 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eec000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee406ccd0 con 0x7f4ee4144c00 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.565+0000 7f4eea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52508/0 (socket says 172.21.15.97:52508) 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 -- 172.21.15.97:0/3509805135 learned_addr learned my addr 172.21.15.97:0/3509805135 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4ee9600640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ee40ce0a0 0x7f4ee413c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4ee8c00640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ee4144c00 0x7f4ee41f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 -- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ee4144c00 msgr2=0x7f4ee41f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ee4144c00 0x7f4ee41f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 -- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ee40ce0a0 msgr2=0x7f4ee413c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ee40ce0a0 0x7f4ee413c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 -- 172.21.15.97:0/3509805135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee415b9e0 con 0x7f4ee415f9d0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eea000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415ad80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00c700 tx=0x7f4edc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc018070 con 0x7f4ee415f9d0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc00ce10 con 0x7f4ee415f9d0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee413cde0 con 0x7f4ee415f9d0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.566+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc00a860 con 0x7f4ee415f9d0 2024-03-06T07:30:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.567+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee413b200 con 0x7f4ee415f9d0 2024-03-06T07:30:24.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.568+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee41f2410 con 0x7f4ee415f9d0 2024-03-06T07:30:24.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.569+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4edc01b070 con 0x7f4ee415f9d0 2024-03-06T07:30:24.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.570+0000 7f4ee1600640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.570+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4edc089770 con 0x7f4ee415f9d0 2024-03-06T07:30:24.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.570+0000 7f4ee9600640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.571+0000 7f4ee9600640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ec406b4d0 0x7f4ec406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009a10 tx=0x7f4ed8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.575+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ee406ee70 con 0x7f4ee415f9d0 2024-03-06T07:30:24.608 INFO:tasks.ceph.mds.a.smithi097.stderr:2024-03-06T07:30:24.606+0000 7f285de00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:24.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.866+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ee404f330 con 0x7f4ec406b4d0 2024-03-06T07:30:24.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.885+0000 7f4ee1600640 1 -- 172.21.15.97:0/3509805135 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239859 (secure 0 0 0) 0x7f4ee404f330 con 0x7f4ec406b4d0 2024-03-06T07:30:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:24.891 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:30:24.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ec406b4d0 msgr2=0x7f4ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ec406b4d0 0x7f4ec406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009a10 tx=0x7f4ed8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:24.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 msgr2=0x7f4ee415ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415ad80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00c700 tx=0x7f4edc00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 shutdown_connections 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ee415f9d0 0x7f4ee415ad80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ee4144c00 0x7f4ee41f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 --2- 172.21.15.97:0/3509805135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ee40ce0a0 0x7f4ee413c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 >> 172.21.15.97:0/3509805135 conn(0x7f4ee40b1730 msgr2=0x7f4ee40affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 shutdown_connections 2024-03-06T07:30:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:24.891+0000 7f4eec000640 1 -- 172.21.15.97:0/3509805135 wait complete. 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-03-06T07:30:22.949468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":251556,"kb_used_data":6244,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754723164,"statfs":{"total":773094113280,"available":772836519936,"internally_reserved":0,"allocated":6393856,"data_stored":3302877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.719910"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.240413+0000","last_change":"2024-03-06T07:30:18.240413+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.240413+0000","last_undegraded":"2024-03-06T07:30:18.240413+0000","last_fullsized":"2024-03-06T07:30:18.240413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.234117+0000","last_change":"2024-03-06T07:30:18.234117+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.234117+0000","last_undegraded":"2024-03-06T07:30:18.234117+0000","last_fullsized":"2024-03-06T07:30:18.234117+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.239754+0000","last_change":"2024-03-06T07:30:18.239754+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.239754+0000","last_undegraded":"2024-03-06T07:30:18.239754+0000","last_fullsized":"2024-03-06T07:30:18.239754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.246825+0000","last_change":"2024-03-06T07:30:18.246825+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.246825+0000","last_undegraded":"2024-03-06T07:30:18.246825+0000","last_fullsized":"2024-03-06T07:30:18.246825+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.239085+0000","last_change":"2024-03-06T07:30:18.239085+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.239085+0000","last_undegraded":"2024-03-06T07:30:18.239085+0000","last_fullsized":"2024-03-06T07:30:18.239085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.239854+0000","last_change":"2024-03-06T07:30:18.239854+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.239854+0000","last_undegraded":"2024-03-06T07:30:18.239854+0000","last_fullsized":"2024-03-06T07:30:18.239854+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.242139+0000","last_change":"2024-03-06T07:30:18.242139+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.242139+0000","last_undegraded":"2024-03-06T07:30:18.242139+0000","last_fullsized":"2024-03-06T07:30:18.242139+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.239033+0000","last_change":"2024-03-06T07:30:18.239033+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.239033+0000","last_undegraded":"2024-03-06T07:30:18.239033+0000","last_fullsized":"2024-03-06T07:30:18.239033+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.244115+0000","last_change":"2024-03-06T07:30:18.244115+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.244115+0000","last_undegraded":"2024-03-06T07:30:18.244115+0000","last_fullsized":"2024-03-06T07:30:18.244115+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.233588+0000","last_change":"2024-03-06T07:30:18.233588+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.233588+0000","last_undegraded":"2024-03-06T07:30:18.233588+0000","last_fullsized":"2024-03-06T07:30:18.233588+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.245311+0000","last_change":"2024-03-06T07:30:18.245311+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.245311+0000","last_undegraded":"2024-03-06T07:30:18.245311+0000","last_fullsized":"2024-03-06T07:30:18.245311+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.240274+0000","last_change":"2024-03-06T07:30:18.240274+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.240274+0000","last_undegraded":"2024-03-06T07:30:18.240274+0000","last_fullsized":"2024-03-06T07:30:18.240274+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.233704+0000","last_change":"2024-03-06T07:30:18.233704+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.233704+0000","last_undegraded":"2024-03-06T07:30:18.233704+0000","last_fullsized":"2024-03-06T07:30:18.233704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.233950+0000","last_change":"2024-03-06T07:30:18.233950+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.233950+0000","last_undegraded":"2024-03-06T07:30:18.233950+0000","last_fullsized":"2024-03-06T07:30:18.233950+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.240648+0000","last_change":"2024-03-06T07:30:18.240648+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.240648+0000","last_undegraded":"2024-03-06T07:30:18.240648+0000","last_fullsized":"2024-03-06T07:30:18.240648+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.243228+0000","last_change":"2024-03-06T07:30:18.243228+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.243228+0000","last_undegraded":"2024-03-06T07:30:18.243228+0000","last_fullsized":"2024-03-06T07:30:18.243228+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.233036+0000","last_change":"2024-03-06T07:30:18.233036+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.233036+0000","last_undegraded":"2024-03-06T07:30:18.233036+0000","last_fullsized":"2024-03-06T07:30:18.233036+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.241525+0000","last_change":"2024-03-06T07:30:18.241525+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.241525+0000","last_undegraded":"2024-03-06T07:30:18.241525+0000","last_fullsized":"2024-03-06T07:30:18.241525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.244135+0000","last_change":"2024-03-06T07:30:18.244135+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.244135+0000","last_undegraded":"2024-03-06T07:30:18.244135+0000","last_fullsized":"2024-03-06T07:30:18.244135+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.240392+0000","last_change":"2024-03-06T07:30:18.240392+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.240392+0000","last_undegraded":"2024-03-06T07:30:18.240392+0000","last_fullsized":"2024-03-06T07:30:18.240392+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.233048+0000","last_change":"2024-03-06T07:30:18.233048+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.233048+0000","last_undegraded":"2024-03-06T07:30:18.233048+0000","last_fullsized":"2024-03-06T07:30:18.233048+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.236277+0000","last_change":"2024-03-06T07:30:18.236277+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.236277+0000","last_undegraded":"2024-03-06T07:30:18.236277+0000","last_fullsized":"2024-03-06T07:30:18.236277+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.240724+0000","last_change":"2024-03-06T07:30:18.240724+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.240724+0000","last_undegraded":"2024-03-06T07:30:18.240724+0000","last_fullsized":"2024-03-06T07:30:18.240724+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.235509+0000","last_change":"2024-03-06T07:30:18.235509+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.235509+0000","last_undegraded":"2024-03-06T07:30:18.235509+0000","last_fullsized":"2024-03-06T07:30:18.235509+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.239835+0000","last_change":"2024-03-06T07:30:18.239835+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.239835+0000","last_undegraded":"2024-03-06T07:30:18.239835+0000","last_fullsized":"2024-03-06T07:30:18.239835+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.234013+0000","last_change":"2024-03-06T07:30:18.234013+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.234013+0000","last_undegraded":"2024-03-06T07:30:18.234013+0000","last_fullsized":"2024-03-06T07:30:18.234013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.245683+0000","last_change":"2024-03-06T07:30:18.245683+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.245683+0000","last_undegraded":"2024-03-06T07:30:18.245683+0000","last_fullsized":"2024-03-06T07:30:18.245683+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.235099+0000","last_change":"2024-03-06T07:30:18.235099+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.235099+0000","last_undegraded":"2024-03-06T07:30:18.235099+0000","last_fullsized":"2024-03-06T07:30:18.235099+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.239111+0000","last_change":"2024-03-06T07:30:18.239111+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.239111+0000","last_undegraded":"2024-03-06T07:30:18.239111+0000","last_fullsized":"2024-03-06T07:30:18.239111+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.235310+0000","last_change":"2024-03-06T07:30:18.235310+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.235310+0000","last_undegraded":"2024-03-06T07:30:18.235310+0000","last_fullsized":"2024-03-06T07:30:18.235310+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.232868+0000","last_change":"2024-03-06T07:30:18.232868+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.232868+0000","last_undegraded":"2024-03-06T07:30:18.232868+0000","last_fullsized":"2024-03-06T07:30:18.232868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:19.227217+0000","last_change":"2024-03-06T07:30:19.227217+0000","last_active":"2024-03-06T07:30:19.227217+0000","last_peered":"2024-03-06T07:30:19.227217+0000","last_clean":"2024-03-06T07:30:19.227217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:19.227217+0000","last_undegraded":"2024-03-06T07:30:19.227217+0000","last_fullsized":"2024-03-06T07:30:19.227217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:30:18.233925+0000","last_change":"2024-03-06T07:30:18.233925+0000","last_active":"2024-03-06T07:30:18.212541+0000","last_peered":"2024-03-06T07:30:18.212541+0000","last_clean":"2024-03-06T07:30:18.212541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:18.233925+0000","last_undegraded":"2024-03-06T07:30:18.233925+0000","last_fullsized":"2024-03-06T07:30:18.233925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":229,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:18.243935+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:30:18.243935+0000","last_peered":"2024-03-06T07:30:18.243935+0000","last_clean":"2024-03-06T07:30:18.243935+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:30:18.243935+0000","last_undegraded":"2024-03-06T07:30:18.243935+0000","last_fullsized":"2024-03-06T07:30:18.243935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1249280,"data_stored":857817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":267449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":267449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":643072,"data_stored":261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":643072,"data_stored":261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":267449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":267449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1236992,"data_stored":852000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:24.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.147+0000 7f189e400640 1 Processor -- start 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.148+0000 7f189e400640 1 -- start start 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f189e400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f18980d1cc0 0x7f18980b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f189e400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980d4c50 con 0x7f18980b2de0 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f189e400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980cfa30 con 0x7f18980d24c0 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f189e400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18980d4a00 con 0x7f18980d1cc0 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f1897e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f18980d1cc0 0x7f18980b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f188ea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f18980b2de0 0x7f18980b0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46284/0 (socket says 172.21.15.97:46284) 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.149+0000 7f188ea00640 1 -- 172.21.15.97:0/774465212 learned_addr learned my addr 172.21.15.97:0/774465212 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.150+0000 7f1897e00640 1 -- 172.21.15.97:0/774465212 >> v1:172.21.15.135:6789/0 conn(0x7f18980d24c0 legacy=0x7f18980d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.150+0000 7f1897e00640 1 -- 172.21.15.97:0/774465212 >> v1:172.21.15.97:6789/0 conn(0x7f18980b2de0 legacy=0x7f18980b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.150+0000 7f1897e00640 1 -- 172.21.15.97:0/774465212 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980cfd10 con 0x7f18980d1cc0 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.150+0000 7f1896a00640 1 -- 172.21.15.97:0/774465212 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062626938 0 0) 0x7f18980d4c50 con 0x7f18980b2de0 2024-03-06T07:30:25.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.151+0000 7f1897e00640 1 --2- 172.21.15.97:0/774465212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d1cc0 0x7f18980b11b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f18900097a0 tx=0x7f189002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8487195d714682c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.151+0000 7f1896a00640 1 -- 172.21.15.97:0/774465212 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890035020 con 0x7f18980d1cc0 2024-03-06T07:30:25.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.151+0000 7f1896a00640 1 -- 172.21.15.97:0/774465212 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1890031830 con 0x7f18980d1cc0 2024-03-06T07:30:25.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.151+0000 7f1896a00640 1 -- 172.21.15.97:0/774465212 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890031c00 con 0x7f18980d1cc0 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 -- 172.21.15.97:0/774465212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d1cc0 msgr2=0x7f18980b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 --2- 172.21.15.97:0/774465212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d1cc0 0x7f18980b11b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f18900097a0 tx=0x7f189002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 -- 172.21.15.97:0/774465212 shutdown_connections 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 --2- 172.21.15.97:0/774465212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d1cc0 0x7f18980b11b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 -- 172.21.15.97:0/774465212 >> 172.21.15.97:0/774465212 conn(0x7f18980d3ee0 msgr2=0x7f18980d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 -- 172.21.15.97:0/774465212 shutdown_connections 2024-03-06T07:30:25.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.152+0000 7f189e400640 1 -- 172.21.15.97:0/774465212 wait complete. 2024-03-06T07:30:25.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.154+0000 7f189e400640 1 Processor -- start 2024-03-06T07:30:25.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.155+0000 7f189e400640 1 -- start start 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.156+0000 7f189e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f18980b2de0 0x7f18980c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.156+0000 7f189e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18980d1cc0 0x7f189814f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.156+0000 7f189e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 0x7f18980b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.156+0000 7f189e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18980d0080 con 0x7f18980d24c0 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.156+0000 7f189e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18980cf7e0 con 0x7f18980d1cc0 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.156+0000 7f189e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18980b16f0 con 0x7f18980b2de0 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 0x7f18980b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 0x7f18980b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52514/0 (socket says 172.21.15.97:52514) 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 -- 172.21.15.97:0/995638293 learned_addr learned my addr 172.21.15.97:0/995638293 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f188ea00640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18980d1cc0 0x7f189814f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 -- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f18980b2de0 msgr2=0x7f18980c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f18980b2de0 0x7f18980c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 -- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18980d1cc0 msgr2=0x7f189814f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18980d1cc0 0x7f189814f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 -- 172.21.15.97:0/995638293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980b8270 con 0x7f18980d24c0 2024-03-06T07:30:25.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f188ea00640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18980d1cc0 0x7f189814f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.157+0000 7f1897e00640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 0x7f18980b7a20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f18900098c0 tx=0x7f18900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.158+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890035050 con 0x7f18980d24c0 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.158+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1890004230 con 0x7f18980d24c0 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.158+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18980c7b40 con 0x7f18980d24c0 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.158+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189003b7c0 con 0x7f18980d24c0 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.158+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189814f800 con 0x7f18980d24c0 2024-03-06T07:30:25.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.159+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189804f330 con 0x7f18980d24c0 2024-03-06T07:30:25.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.160+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1890038020 con 0x7f18980d24c0 2024-03-06T07:30:25.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.161+0000 7f188fe00640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.161+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18900abf60 con 0x7f18980d24c0 2024-03-06T07:30:25.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.161+0000 7f1897400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.162+0000 7f1897400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f187406b500 0x7f187406d9c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f18880012e0 tx=0x7f1888007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.166+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1890040070 con 0x7f18980d24c0 2024-03-06T07:30:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.358+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18980d22a0 con 0x7f187406b500 2024-03-06T07:30:25.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.376+0000 7f188fe00640 1 -- 172.21.15.97:0/995638293 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251955 (secure 0 0 0) 0x7f18980d22a0 con 0x7f187406b500 2024-03-06T07:30:25.382 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:25.383 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f187406b500 msgr2=0x7f187406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f187406b500 0x7f187406d9c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f18880012e0 tx=0x7f1888007d30 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 msgr2=0x7f18980b7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 0x7f18980b7a20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f18900098c0 tx=0x7f18900046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 shutdown_connections 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18980d24c0 0x7f18980b7a20 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18980d1cc0 0x7f189814f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 --2- 172.21.15.97:0/995638293 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f18980b2de0 0x7f18980c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 >> 172.21.15.97:0/995638293 conn(0x7f18980d3ee0 msgr2=0x7f18980d0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.381+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 shutdown_connections 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.382+0000 7f189e400640 1 -- 172.21.15.97:0/995638293 wait complete. 2024-03-06T07:30:25.404 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-06T07:30:24.950043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251936,"kb_used_data":6632,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754722784,"statfs":{"total":773094113280,"available":772836130816,"internally_reserved":0,"allocated":6791168,"data_stored":3479884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001811"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.284695+0000","last_change":"2024-03-06T07:30:20.284433+0000","last_active":"2024-03-06T07:30:21.284695+0000","last_peered":"2024-03-06T07:30:21.284695+0000","last_clean":"2024-03-06T07:30:21.284695+0000","last_became_active":"2024-03-06T07:30:20.284107+0000","last_became_peered":"2024-03-06T07:30:20.284107+0000","last_unstale":"2024-03-06T07:30:21.284695+0000","last_undegraded":"2024-03-06T07:30:21.284695+0000","last_fullsized":"2024-03-06T07:30:21.284695+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:18.214010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365656+0000","last_change":"2024-03-06T07:30:20.290943+0000","last_active":"2024-03-06T07:30:21.365656+0000","last_peered":"2024-03-06T07:30:21.365656+0000","last_clean":"2024-03-06T07:30:21.365656+0000","last_became_active":"2024-03-06T07:30:20.289955+0000","last_became_peered":"2024-03-06T07:30:20.289955+0000","last_unstale":"2024-03-06T07:30:21.365656+0000","last_undegraded":"2024-03-06T07:30:21.365656+0000","last_fullsized":"2024-03-06T07:30:21.365656+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:59.206317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369165+0000","last_change":"2024-03-06T07:30:20.293317+0000","last_active":"2024-03-06T07:30:21.369165+0000","last_peered":"2024-03-06T07:30:21.369165+0000","last_clean":"2024-03-06T07:30:21.369165+0000","last_became_active":"2024-03-06T07:30:20.292300+0000","last_became_peered":"2024-03-06T07:30:20.292300+0000","last_unstale":"2024-03-06T07:30:21.369165+0000","last_undegraded":"2024-03-06T07:30:21.369165+0000","last_fullsized":"2024-03-06T07:30:21.369165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:12.629356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371182+0000","last_change":"2024-03-06T07:30:20.283930+0000","last_active":"2024-03-06T07:30:21.371182+0000","last_peered":"2024-03-06T07:30:21.371182+0000","last_clean":"2024-03-06T07:30:21.371182+0000","last_became_active":"2024-03-06T07:30:20.283611+0000","last_became_peered":"2024-03-06T07:30:20.283611+0000","last_unstale":"2024-03-06T07:30:21.371182+0000","last_undegraded":"2024-03-06T07:30:21.371182+0000","last_fullsized":"2024-03-06T07:30:21.371182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:09.877905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.284421+0000","last_change":"2024-03-06T07:30:20.287714+0000","last_active":"2024-03-06T07:30:21.284421+0000","last_peered":"2024-03-06T07:30:21.284421+0000","last_clean":"2024-03-06T07:30:21.284421+0000","last_became_active":"2024-03-06T07:30:20.287436+0000","last_became_peered":"2024-03-06T07:30:20.287436+0000","last_unstale":"2024-03-06T07:30:21.284421+0000","last_undegraded":"2024-03-06T07:30:21.284421+0000","last_fullsized":"2024-03-06T07:30:21.284421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:29.507143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.292912+0000","last_change":"2024-03-06T07:30:20.272323+0000","last_active":"2024-03-06T07:30:21.292912+0000","last_peered":"2024-03-06T07:30:21.292912+0000","last_clean":"2024-03-06T07:30:21.292912+0000","last_became_active":"2024-03-06T07:30:20.269848+0000","last_became_peered":"2024-03-06T07:30:20.269848+0000","last_unstale":"2024-03-06T07:30:21.292912+0000","last_undegraded":"2024-03-06T07:30:21.292912+0000","last_fullsized":"2024-03-06T07:30:21.292912+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:57.945646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.286782+0000","last_change":"2024-03-06T07:30:20.282760+0000","last_active":"2024-03-06T07:30:21.286782+0000","last_peered":"2024-03-06T07:30:21.286782+0000","last_clean":"2024-03-06T07:30:21.286782+0000","last_became_active":"2024-03-06T07:30:20.282187+0000","last_became_peered":"2024-03-06T07:30:20.282187+0000","last_unstale":"2024-03-06T07:30:21.286782+0000","last_undegraded":"2024-03-06T07:30:21.286782+0000","last_fullsized":"2024-03-06T07:30:21.286782+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:04.331602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365228+0000","last_change":"2024-03-06T07:30:20.288244+0000","last_active":"2024-03-06T07:30:21.365228+0000","last_peered":"2024-03-06T07:30:21.365228+0000","last_clean":"2024-03-06T07:30:21.365228+0000","last_became_active":"2024-03-06T07:30:20.287985+0000","last_became_peered":"2024-03-06T07:30:20.287985+0000","last_unstale":"2024-03-06T07:30:21.365228+0000","last_undegraded":"2024-03-06T07:30:21.365228+0000","last_fullsized":"2024-03-06T07:30:21.365228+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:22.682894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.287751+0000","last_change":"2024-03-06T07:30:20.276251+0000","last_active":"2024-03-06T07:30:21.287751+0000","last_peered":"2024-03-06T07:30:21.287751+0000","last_clean":"2024-03-06T07:30:21.287751+0000","last_became_active":"2024-03-06T07:30:20.274652+0000","last_became_peered":"2024-03-06T07:30:20.274652+0000","last_unstale":"2024-03-06T07:30:21.287751+0000","last_undegraded":"2024-03-06T07:30:21.287751+0000","last_fullsized":"2024-03-06T07:30:21.287751+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:08.163199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283594+0000","last_change":"2024-03-06T07:30:20.287506+0000","last_active":"2024-03-06T07:30:21.283594+0000","last_peered":"2024-03-06T07:30:21.283594+0000","last_clean":"2024-03-06T07:30:21.283594+0000","last_became_active":"2024-03-06T07:30:20.286094+0000","last_became_peered":"2024-03-06T07:30:20.286094+0000","last_unstale":"2024-03-06T07:30:21.283594+0000","last_undegraded":"2024-03-06T07:30:21.283594+0000","last_fullsized":"2024-03-06T07:30:21.283594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.541132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283470+0000","last_change":"2024-03-06T07:30:20.288376+0000","last_active":"2024-03-06T07:30:21.283470+0000","last_peered":"2024-03-06T07:30:21.283470+0000","last_clean":"2024-03-06T07:30:21.283470+0000","last_became_active":"2024-03-06T07:30:20.288107+0000","last_became_peered":"2024-03-06T07:30:20.288107+0000","last_unstale":"2024-03-06T07:30:21.283470+0000","last_undegraded":"2024-03-06T07:30:21.283470+0000","last_fullsized":"2024-03-06T07:30:21.283470+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:29.411242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371288+0000","last_change":"2024-03-06T07:30:20.296040+0000","last_active":"2024-03-06T07:30:21.371288+0000","last_peered":"2024-03-06T07:30:21.371288+0000","last_clean":"2024-03-06T07:30:21.371288+0000","last_became_active":"2024-03-06T07:30:20.295638+0000","last_became_peered":"2024-03-06T07:30:20.295638+0000","last_unstale":"2024-03-06T07:30:21.371288+0000","last_undegraded":"2024-03-06T07:30:21.371288+0000","last_fullsized":"2024-03-06T07:30:21.371288+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:57.945646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.299201+0000","last_change":"2024-03-06T07:30:20.302600+0000","last_active":"2024-03-06T07:30:21.299201+0000","last_peered":"2024-03-06T07:30:21.299201+0000","last_clean":"2024-03-06T07:30:21.299201+0000","last_became_active":"2024-03-06T07:30:20.302364+0000","last_became_peered":"2024-03-06T07:30:20.302364+0000","last_unstale":"2024-03-06T07:30:21.299201+0000","last_undegraded":"2024-03-06T07:30:21.299201+0000","last_fullsized":"2024-03-06T07:30:21.299201+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:39.327448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.287235+0000","last_change":"2024-03-06T07:30:20.278799+0000","last_active":"2024-03-06T07:30:21.287235+0000","last_peered":"2024-03-06T07:30:21.287235+0000","last_clean":"2024-03-06T07:30:21.287235+0000","last_became_active":"2024-03-06T07:30:20.278450+0000","last_became_peered":"2024-03-06T07:30:20.278450+0000","last_unstale":"2024-03-06T07:30:21.287235+0000","last_undegraded":"2024-03-06T07:30:21.287235+0000","last_fullsized":"2024-03-06T07:30:21.287235+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.185695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.286834+0000","last_change":"2024-03-06T07:30:20.281441+0000","last_active":"2024-03-06T07:30:21.286834+0000","last_peered":"2024-03-06T07:30:21.286834+0000","last_clean":"2024-03-06T07:30:21.286834+0000","last_became_active":"2024-03-06T07:30:20.281185+0000","last_became_peered":"2024-03-06T07:30:20.281185+0000","last_unstale":"2024-03-06T07:30:21.286834+0000","last_undegraded":"2024-03-06T07:30:21.286834+0000","last_fullsized":"2024-03-06T07:30:21.286834+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:39.268305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.292740+0000","last_change":"2024-03-06T07:30:20.287487+0000","last_active":"2024-03-06T07:30:21.292740+0000","last_peered":"2024-03-06T07:30:21.292740+0000","last_clean":"2024-03-06T07:30:21.292740+0000","last_became_active":"2024-03-06T07:30:20.287009+0000","last_became_peered":"2024-03-06T07:30:20.287009+0000","last_unstale":"2024-03-06T07:30:21.292740+0000","last_undegraded":"2024-03-06T07:30:21.292740+0000","last_fullsized":"2024-03-06T07:30:21.292740+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:52.120154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365814+0000","last_change":"2024-03-06T07:30:20.292731+0000","last_active":"2024-03-06T07:30:21.365814+0000","last_peered":"2024-03-06T07:30:21.365814+0000","last_clean":"2024-03-06T07:30:21.365814+0000","last_became_active":"2024-03-06T07:30:20.291207+0000","last_became_peered":"2024-03-06T07:30:20.291207+0000","last_unstale":"2024-03-06T07:30:21.365814+0000","last_undegraded":"2024-03-06T07:30:21.365814+0000","last_fullsized":"2024-03-06T07:30:21.365814+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:32.099234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371208+0000","last_change":"2024-03-06T07:30:20.302880+0000","last_active":"2024-03-06T07:30:21.371208+0000","last_peered":"2024-03-06T07:30:21.371208+0000","last_clean":"2024-03-06T07:30:21.371208+0000","last_became_active":"2024-03-06T07:30:20.302594+0000","last_became_peered":"2024-03-06T07:30:20.302594+0000","last_unstale":"2024-03-06T07:30:21.371208+0000","last_undegraded":"2024-03-06T07:30:21.371208+0000","last_fullsized":"2024-03-06T07:30:21.371208+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:48.210486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.284611+0000","last_change":"2024-03-06T07:30:20.285118+0000","last_active":"2024-03-06T07:30:21.284611+0000","last_peered":"2024-03-06T07:30:21.284611+0000","last_clean":"2024-03-06T07:30:21.284611+0000","last_became_active":"2024-03-06T07:30:20.284615+0000","last_became_peered":"2024-03-06T07:30:20.284615+0000","last_unstale":"2024-03-06T07:30:21.284611+0000","last_undegraded":"2024-03-06T07:30:21.284611+0000","last_fullsized":"2024-03-06T07:30:21.284611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:57.661624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365520+0000","last_change":"2024-03-06T07:30:20.296363+0000","last_active":"2024-03-06T07:30:21.365520+0000","last_peered":"2024-03-06T07:30:21.365520+0000","last_clean":"2024-03-06T07:30:21.365520+0000","last_became_active":"2024-03-06T07:30:20.296045+0000","last_became_peered":"2024-03-06T07:30:20.296045+0000","last_unstale":"2024-03-06T07:30:21.365520+0000","last_undegraded":"2024-03-06T07:30:21.365520+0000","last_fullsized":"2024-03-06T07:30:21.365520+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:42.508583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369075+0000","last_change":"2024-03-06T07:30:20.285986+0000","last_active":"2024-03-06T07:30:21.369075+0000","last_peered":"2024-03-06T07:30:21.369075+0000","last_clean":"2024-03-06T07:30:21.369075+0000","last_became_active":"2024-03-06T07:30:20.285691+0000","last_became_peered":"2024-03-06T07:30:20.285691+0000","last_unstale":"2024-03-06T07:30:21.369075+0000","last_undegraded":"2024-03-06T07:30:21.369075+0000","last_fullsized":"2024-03-06T07:30:21.369075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:50.192009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.366075+0000","last_change":"2024-03-06T07:30:20.283703+0000","last_active":"2024-03-06T07:30:21.366075+0000","last_peered":"2024-03-06T07:30:21.366075+0000","last_clean":"2024-03-06T07:30:21.366075+0000","last_became_active":"2024-03-06T07:30:20.283454+0000","last_became_peered":"2024-03-06T07:30:20.283454+0000","last_unstale":"2024-03-06T07:30:21.366075+0000","last_undegraded":"2024-03-06T07:30:21.366075+0000","last_fullsized":"2024-03-06T07:30:21.366075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:37.036960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.368878+0000","last_change":"2024-03-06T07:30:20.290822+0000","last_active":"2024-03-06T07:30:21.368878+0000","last_peered":"2024-03-06T07:30:21.368878+0000","last_clean":"2024-03-06T07:30:21.368878+0000","last_became_active":"2024-03-06T07:30:20.289194+0000","last_became_peered":"2024-03-06T07:30:20.289194+0000","last_unstale":"2024-03-06T07:30:21.368878+0000","last_undegraded":"2024-03-06T07:30:21.368878+0000","last_fullsized":"2024-03-06T07:30:21.368878+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:57.945646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.298811+0000","last_change":"2024-03-06T07:30:20.283435+0000","last_active":"2024-03-06T07:30:21.298811+0000","last_peered":"2024-03-06T07:30:21.298811+0000","last_clean":"2024-03-06T07:30:21.298811+0000","last_became_active":"2024-03-06T07:30:20.283156+0000","last_became_peered":"2024-03-06T07:30:20.283156+0000","last_unstale":"2024-03-06T07:30:21.298811+0000","last_undegraded":"2024-03-06T07:30:21.298811+0000","last_fullsized":"2024-03-06T07:30:21.298811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:41.857686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.284759+0000","last_change":"2024-03-06T07:30:20.289466+0000","last_active":"2024-03-06T07:30:21.284759+0000","last_peered":"2024-03-06T07:30:21.284759+0000","last_clean":"2024-03-06T07:30:21.284759+0000","last_became_active":"2024-03-06T07:30:20.289204+0000","last_became_peered":"2024-03-06T07:30:20.289204+0000","last_unstale":"2024-03-06T07:30:21.284759+0000","last_undegraded":"2024-03-06T07:30:21.284759+0000","last_fullsized":"2024-03-06T07:30:21.284759+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:35.482069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371766+0000","last_change":"2024-03-06T07:30:20.289171+0000","last_active":"2024-03-06T07:30:21.371766+0000","last_peered":"2024-03-06T07:30:21.371766+0000","last_clean":"2024-03-06T07:30:21.371766+0000","last_became_active":"2024-03-06T07:30:20.285914+0000","last_became_peered":"2024-03-06T07:30:20.285914+0000","last_unstale":"2024-03-06T07:30:21.371766+0000","last_undegraded":"2024-03-06T07:30:21.371766+0000","last_fullsized":"2024-03-06T07:30:21.371766+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:50.192009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293355+0000","last_change":"2024-03-06T07:30:20.286726+0000","last_active":"2024-03-06T07:30:21.293355+0000","last_peered":"2024-03-06T07:30:21.293355+0000","last_clean":"2024-03-06T07:30:21.293355+0000","last_became_active":"2024-03-06T07:30:20.282518+0000","last_became_peered":"2024-03-06T07:30:20.282518+0000","last_unstale":"2024-03-06T07:30:21.293355+0000","last_undegraded":"2024-03-06T07:30:21.293355+0000","last_fullsized":"2024-03-06T07:30:21.293355+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:12.629356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.284625+0000","last_change":"2024-03-06T07:30:20.290895+0000","last_active":"2024-03-06T07:30:21.284625+0000","last_peered":"2024-03-06T07:30:21.284625+0000","last_clean":"2024-03-06T07:30:21.284625+0000","last_became_active":"2024-03-06T07:30:20.290456+0000","last_became_peered":"2024-03-06T07:30:20.290456+0000","last_unstale":"2024-03-06T07:30:21.284625+0000","last_undegraded":"2024-03-06T07:30:21.284625+0000","last_fullsized":"2024-03-06T07:30:21.284625+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:15.184038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365777+0000","last_change":"2024-03-06T07:30:20.287786+0000","last_active":"2024-03-06T07:30:21.365777+0000","last_peered":"2024-03-06T07:30:21.365777+0000","last_clean":"2024-03-06T07:30:21.365777+0000","last_became_active":"2024-03-06T07:30:20.287531+0000","last_became_peered":"2024-03-06T07:30:20.287531+0000","last_unstale":"2024-03-06T07:30:21.365777+0000","last_undegraded":"2024-03-06T07:30:21.365777+0000","last_fullsized":"2024-03-06T07:30:21.365777+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:29.663813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.370064+0000","last_change":"2024-03-06T07:30:20.284726+0000","last_active":"2024-03-06T07:30:21.370064+0000","last_peered":"2024-03-06T07:30:21.370064+0000","last_clean":"2024-03-06T07:30:21.370064+0000","last_became_active":"2024-03-06T07:30:20.282915+0000","last_became_peered":"2024-03-06T07:30:20.282915+0000","last_unstale":"2024-03-06T07:30:21.370064+0000","last_undegraded":"2024-03-06T07:30:21.370064+0000","last_fullsized":"2024-03-06T07:30:21.370064+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:34.009835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.284214+0000","last_change":"2024-03-06T07:30:20.284550+0000","last_active":"2024-03-06T07:30:21.284214+0000","last_peered":"2024-03-06T07:30:21.284214+0000","last_clean":"2024-03-06T07:30:21.284214+0000","last_became_active":"2024-03-06T07:30:20.283716+0000","last_became_peered":"2024-03-06T07:30:20.283716+0000","last_unstale":"2024-03-06T07:30:21.284214+0000","last_undegraded":"2024-03-06T07:30:21.284214+0000","last_fullsized":"2024-03-06T07:30:21.284214+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:18.512422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.292851+0000","last_change":"2024-03-06T07:30:20.286641+0000","last_active":"2024-03-06T07:30:21.292851+0000","last_peered":"2024-03-06T07:30:21.292851+0000","last_clean":"2024-03-06T07:30:21.292851+0000","last_became_active":"2024-03-06T07:30:20.283292+0000","last_became_peered":"2024-03-06T07:30:20.283292+0000","last_unstale":"2024-03-06T07:30:21.292851+0000","last_undegraded":"2024-03-06T07:30:21.292851+0000","last_fullsized":"2024-03-06T07:30:21.292851+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:09.877905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293375+0000","last_change":"2024-03-06T07:30:20.294565+0000","last_active":"2024-03-06T07:30:21.293375+0000","last_peered":"2024-03-06T07:30:21.293375+0000","last_clean":"2024-03-06T07:30:21.293375+0000","last_became_active":"2024-03-06T07:30:20.294221+0000","last_became_peered":"2024-03-06T07:30:20.294221+0000","last_unstale":"2024-03-06T07:30:21.293375+0000","last_undegraded":"2024-03-06T07:30:21.293375+0000","last_fullsized":"2024-03-06T07:30:21.293375+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:10.292220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293518+0000","last_change":"2024-03-06T07:30:19.267606+0000","last_active":"2024-03-06T07:30:21.293518+0000","last_peered":"2024-03-06T07:30:21.293518+0000","last_clean":"2024-03-06T07:30:21.293518+0000","last_became_active":"2024-03-06T07:30:19.267316+0000","last_became_peered":"2024-03-06T07:30:19.267316+0000","last_unstale":"2024-03-06T07:30:21.293518+0000","last_undegraded":"2024-03-06T07:30:21.293518+0000","last_fullsized":"2024-03-06T07:30:21.293518+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:42.263016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283667+0000","last_change":"2024-03-06T07:30:20.294668+0000","last_active":"2024-03-06T07:30:21.283667+0000","last_peered":"2024-03-06T07:30:21.283667+0000","last_clean":"2024-03-06T07:30:21.283667+0000","last_became_active":"2024-03-06T07:30:20.293699+0000","last_became_peered":"2024-03-06T07:30:20.293699+0000","last_unstale":"2024-03-06T07:30:21.283667+0000","last_undegraded":"2024-03-06T07:30:21.283667+0000","last_fullsized":"2024-03-06T07:30:21.283667+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:58.481333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365718+0000","last_change":"2024-03-06T07:30:19.269007+0000","last_active":"2024-03-06T07:30:21.365718+0000","last_peered":"2024-03-06T07:30:21.365718+0000","last_clean":"2024-03-06T07:30:21.365718+0000","last_became_active":"2024-03-06T07:30:19.267854+0000","last_became_peered":"2024-03-06T07:30:19.267854+0000","last_unstale":"2024-03-06T07:30:21.365718+0000","last_undegraded":"2024-03-06T07:30:21.365718+0000","last_fullsized":"2024-03-06T07:30:21.365718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:31.493677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293334+0000","last_change":"2024-03-06T07:30:20.285007+0000","last_active":"2024-03-06T07:30:21.293334+0000","last_peered":"2024-03-06T07:30:21.293334+0000","last_clean":"2024-03-06T07:30:21.293334+0000","last_became_active":"2024-03-06T07:30:20.284751+0000","last_became_peered":"2024-03-06T07:30:20.284751+0000","last_unstale":"2024-03-06T07:30:21.293334+0000","last_undegraded":"2024-03-06T07:30:21.293334+0000","last_fullsized":"2024-03-06T07:30:21.293334+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:32.508353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293642+0000","last_change":"2024-03-06T07:30:19.253758+0000","last_active":"2024-03-06T07:30:21.293642+0000","last_peered":"2024-03-06T07:30:21.293642+0000","last_clean":"2024-03-06T07:30:21.293642+0000","last_became_active":"2024-03-06T07:30:19.252161+0000","last_became_peered":"2024-03-06T07:30:19.252161+0000","last_unstale":"2024-03-06T07:30:21.293642+0000","last_undegraded":"2024-03-06T07:30:21.293642+0000","last_fullsized":"2024-03-06T07:30:21.293642+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:41.548200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293510+0000","last_change":"2024-03-06T07:30:20.298744+0000","last_active":"2024-03-06T07:30:21.293510+0000","last_peered":"2024-03-06T07:30:21.293510+0000","last_clean":"2024-03-06T07:30:21.293510+0000","last_became_active":"2024-03-06T07:30:20.298259+0000","last_became_peered":"2024-03-06T07:30:20.298259+0000","last_unstale":"2024-03-06T07:30:21.293510+0000","last_undegraded":"2024-03-06T07:30:21.293510+0000","last_fullsized":"2024-03-06T07:30:21.293510+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:12.153162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371331+0000","last_change":"2024-03-06T07:30:19.269144+0000","last_active":"2024-03-06T07:30:21.371331+0000","last_peered":"2024-03-06T07:30:21.371331+0000","last_clean":"2024-03-06T07:30:21.371331+0000","last_became_active":"2024-03-06T07:30:19.268773+0000","last_became_peered":"2024-03-06T07:30:19.268773+0000","last_unstale":"2024-03-06T07:30:21.371331+0000","last_undegraded":"2024-03-06T07:30:21.371331+0000","last_fullsized":"2024-03-06T07:30:21.371331+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:21.031644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365342+0000","last_change":"2024-03-06T07:30:20.284262+0000","last_active":"2024-03-06T07:30:21.365342+0000","last_peered":"2024-03-06T07:30:21.365342+0000","last_clean":"2024-03-06T07:30:21.365342+0000","last_became_active":"2024-03-06T07:30:20.282893+0000","last_became_peered":"2024-03-06T07:30:20.282893+0000","last_unstale":"2024-03-06T07:30:21.365342+0000","last_undegraded":"2024-03-06T07:30:21.365342+0000","last_fullsized":"2024-03-06T07:30:21.365342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:24.452960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.287025+0000","last_change":"2024-03-06T07:30:19.272447+0000","last_active":"2024-03-06T07:30:21.287025+0000","last_peered":"2024-03-06T07:30:21.287025+0000","last_clean":"2024-03-06T07:30:21.287025+0000","last_became_active":"2024-03-06T07:30:19.272144+0000","last_became_peered":"2024-03-06T07:30:19.272144+0000","last_unstale":"2024-03-06T07:30:21.287025+0000","last_undegraded":"2024-03-06T07:30:21.287025+0000","last_fullsized":"2024-03-06T07:30:21.287025+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:34.212356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365600+0000","last_change":"2024-03-06T07:30:20.285838+0000","last_active":"2024-03-06T07:30:21.365600+0000","last_peered":"2024-03-06T07:30:21.365600+0000","last_clean":"2024-03-06T07:30:21.365600+0000","last_became_active":"2024-03-06T07:30:20.285585+0000","last_became_peered":"2024-03-06T07:30:20.285585+0000","last_unstale":"2024-03-06T07:30:21.365600+0000","last_undegraded":"2024-03-06T07:30:21.365600+0000","last_fullsized":"2024-03-06T07:30:21.365600+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:58.912581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.292977+0000","last_change":"2024-03-06T07:30:19.260389+0000","last_active":"2024-03-06T07:30:21.292977+0000","last_peered":"2024-03-06T07:30:21.292977+0000","last_clean":"2024-03-06T07:30:21.292977+0000","last_became_active":"2024-03-06T07:30:19.260095+0000","last_became_peered":"2024-03-06T07:30:19.260095+0000","last_unstale":"2024-03-06T07:30:21.292977+0000","last_undegraded":"2024-03-06T07:30:21.292977+0000","last_fullsized":"2024-03-06T07:30:21.292977+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:57.466658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.298955+0000","last_change":"2024-03-06T07:30:20.291409+0000","last_active":"2024-03-06T07:30:21.298955+0000","last_peered":"2024-03-06T07:30:21.298955+0000","last_clean":"2024-03-06T07:30:21.298955+0000","last_became_active":"2024-03-06T07:30:20.291136+0000","last_became_peered":"2024-03-06T07:30:20.291136+0000","last_unstale":"2024-03-06T07:30:21.298955+0000","last_undegraded":"2024-03-06T07:30:21.298955+0000","last_fullsized":"2024-03-06T07:30:21.298955+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:33.910928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369527+0000","last_change":"2024-03-06T07:30:19.269674+0000","last_active":"2024-03-06T07:30:21.369527+0000","last_peered":"2024-03-06T07:30:21.369527+0000","last_clean":"2024-03-06T07:30:21.369527+0000","last_became_active":"2024-03-06T07:30:19.267516+0000","last_became_peered":"2024-03-06T07:30:19.267516+0000","last_unstale":"2024-03-06T07:30:21.369527+0000","last_undegraded":"2024-03-06T07:30:21.369527+0000","last_fullsized":"2024-03-06T07:30:21.369527+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:42.263016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365297+0000","last_change":"2024-03-06T07:30:20.292181+0000","last_active":"2024-03-06T07:30:21.365297+0000","last_peered":"2024-03-06T07:30:21.365297+0000","last_clean":"2024-03-06T07:30:21.365297+0000","last_became_active":"2024-03-06T07:30:20.291640+0000","last_became_peered":"2024-03-06T07:30:20.291640+0000","last_unstale":"2024-03-06T07:30:21.365297+0000","last_undegraded":"2024-03-06T07:30:21.365297+0000","last_fullsized":"2024-03-06T07:30:21.365297+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:27.477829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.286529+0000","last_change":"2024-03-06T07:30:19.269363+0000","last_active":"2024-03-06T07:30:21.286529+0000","last_peered":"2024-03-06T07:30:21.286529+0000","last_clean":"2024-03-06T07:30:21.286529+0000","last_became_active":"2024-03-06T07:30:19.269024+0000","last_became_peered":"2024-03-06T07:30:19.269024+0000","last_unstale":"2024-03-06T07:30:21.286529+0000","last_undegraded":"2024-03-06T07:30:21.286529+0000","last_fullsized":"2024-03-06T07:30:21.286529+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:39.931475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.287868+0000","last_change":"2024-03-06T07:30:20.287763+0000","last_active":"2024-03-06T07:30:21.287868+0000","last_peered":"2024-03-06T07:30:21.287868+0000","last_clean":"2024-03-06T07:30:21.287868+0000","last_became_active":"2024-03-06T07:30:20.287393+0000","last_became_peered":"2024-03-06T07:30:20.287393+0000","last_unstale":"2024-03-06T07:30:21.287868+0000","last_undegraded":"2024-03-06T07:30:21.287868+0000","last_fullsized":"2024-03-06T07:30:21.287868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:23.312689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369532+0000","last_change":"2024-03-06T07:30:19.268343+0000","last_active":"2024-03-06T07:30:21.369532+0000","last_peered":"2024-03-06T07:30:21.369532+0000","last_clean":"2024-03-06T07:30:21.369532+0000","last_became_active":"2024-03-06T07:30:19.268056+0000","last_became_peered":"2024-03-06T07:30:19.268056+0000","last_unstale":"2024-03-06T07:30:21.369532+0000","last_undegraded":"2024-03-06T07:30:21.369532+0000","last_fullsized":"2024-03-06T07:30:21.369532+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:21.031644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369654+0000","last_change":"2024-03-06T07:30:20.298975+0000","last_active":"2024-03-06T07:30:21.369654+0000","last_peered":"2024-03-06T07:30:21.369654+0000","last_clean":"2024-03-06T07:30:21.369654+0000","last_became_active":"2024-03-06T07:30:20.298594+0000","last_became_peered":"2024-03-06T07:30:20.298594+0000","last_unstale":"2024-03-06T07:30:21.369654+0000","last_undegraded":"2024-03-06T07:30:21.369654+0000","last_fullsized":"2024-03-06T07:30:21.369654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:54.192021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365714+0000","last_change":"2024-03-06T07:30:19.265534+0000","last_active":"2024-03-06T07:30:21.365714+0000","last_peered":"2024-03-06T07:30:21.365714+0000","last_clean":"2024-03-06T07:30:21.365714+0000","last_became_active":"2024-03-06T07:30:19.264908+0000","last_became_peered":"2024-03-06T07:30:19.264908+0000","last_unstale":"2024-03-06T07:30:21.365714+0000","last_undegraded":"2024-03-06T07:30:21.365714+0000","last_fullsized":"2024-03-06T07:30:21.365714+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:49.177333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283241+0000","last_change":"2024-03-06T07:30:20.283416+0000","last_active":"2024-03-06T07:30:21.283241+0000","last_peered":"2024-03-06T07:30:21.283241+0000","last_clean":"2024-03-06T07:30:21.283241+0000","last_became_active":"2024-03-06T07:30:20.283017+0000","last_became_peered":"2024-03-06T07:30:20.283017+0000","last_unstale":"2024-03-06T07:30:21.283241+0000","last_undegraded":"2024-03-06T07:30:21.283241+0000","last_fullsized":"2024-03-06T07:30:21.283241+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:58.056900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.370355+0000","last_change":"2024-03-06T07:30:19.269682+0000","last_active":"2024-03-06T07:30:21.370355+0000","last_peered":"2024-03-06T07:30:21.370355+0000","last_clean":"2024-03-06T07:30:21.370355+0000","last_became_active":"2024-03-06T07:30:19.269344+0000","last_became_peered":"2024-03-06T07:30:19.269344+0000","last_unstale":"2024-03-06T07:30:21.370355+0000","last_undegraded":"2024-03-06T07:30:21.370355+0000","last_fullsized":"2024-03-06T07:30:21.370355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:41.548200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369592+0000","last_change":"2024-03-06T07:30:20.299725+0000","last_active":"2024-03-06T07:30:21.369592+0000","last_peered":"2024-03-06T07:30:21.369592+0000","last_clean":"2024-03-06T07:30:21.369592+0000","last_became_active":"2024-03-06T07:30:20.298980+0000","last_became_peered":"2024-03-06T07:30:20.298980+0000","last_unstale":"2024-03-06T07:30:21.369592+0000","last_undegraded":"2024-03-06T07:30:21.369592+0000","last_fullsized":"2024-03-06T07:30:21.369592+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:12.153162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.287747+0000","last_change":"2024-03-06T07:30:19.273247+0000","last_active":"2024-03-06T07:30:21.287747+0000","last_peered":"2024-03-06T07:30:21.287747+0000","last_clean":"2024-03-06T07:30:21.287747+0000","last_became_active":"2024-03-06T07:30:19.272872+0000","last_became_peered":"2024-03-06T07:30:19.272872+0000","last_unstale":"2024-03-06T07:30:21.287747+0000","last_undegraded":"2024-03-06T07:30:21.287747+0000","last_fullsized":"2024-03-06T07:30:21.287747+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:41.336230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.370155+0000","last_change":"2024-03-06T07:30:20.292445+0000","last_active":"2024-03-06T07:30:21.370155+0000","last_peered":"2024-03-06T07:30:21.370155+0000","last_clean":"2024-03-06T07:30:21.370155+0000","last_became_active":"2024-03-06T07:30:20.292185+0000","last_became_peered":"2024-03-06T07:30:20.292185+0000","last_unstale":"2024-03-06T07:30:21.370155+0000","last_undegraded":"2024-03-06T07:30:21.370155+0000","last_fullsized":"2024-03-06T07:30:21.370155+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:52.120154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365686+0000","last_change":"2024-03-06T07:30:19.264525+0000","last_active":"2024-03-06T07:30:21.365686+0000","last_peered":"2024-03-06T07:30:21.365686+0000","last_clean":"2024-03-06T07:30:21.365686+0000","last_became_active":"2024-03-06T07:30:19.264258+0000","last_became_peered":"2024-03-06T07:30:19.264258+0000","last_unstale":"2024-03-06T07:30:21.365686+0000","last_undegraded":"2024-03-06T07:30:21.365686+0000","last_fullsized":"2024-03-06T07:30:21.365686+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:51.105478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.368813+0000","last_change":"2024-03-06T07:30:20.284456+0000","last_active":"2024-03-06T07:30:21.368813+0000","last_peered":"2024-03-06T07:30:21.368813+0000","last_clean":"2024-03-06T07:30:21.368813+0000","last_became_active":"2024-03-06T07:30:20.284143+0000","last_became_peered":"2024-03-06T07:30:20.284143+0000","last_unstale":"2024-03-06T07:30:21.368813+0000","last_undegraded":"2024-03-06T07:30:21.368813+0000","last_fullsized":"2024-03-06T07:30:21.368813+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:09.877905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.298412+0000","last_change":"2024-03-06T07:30:19.266684+0000","last_active":"2024-03-06T07:30:21.298412+0000","last_peered":"2024-03-06T07:30:21.298412+0000","last_clean":"2024-03-06T07:30:21.298412+0000","last_became_active":"2024-03-06T07:30:19.265948+0000","last_became_peered":"2024-03-06T07:30:19.265948+0000","last_unstale":"2024-03-06T07:30:21.298412+0000","last_undegraded":"2024-03-06T07:30:21.298412+0000","last_fullsized":"2024-03-06T07:30:21.298412+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:06.736990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.286698+0000","last_change":"2024-03-06T07:30:20.287022+0000","last_active":"2024-03-06T07:30:21.286698+0000","last_peered":"2024-03-06T07:30:21.286698+0000","last_clean":"2024-03-06T07:30:21.286698+0000","last_became_active":"2024-03-06T07:30:20.286322+0000","last_became_peered":"2024-03-06T07:30:20.286322+0000","last_unstale":"2024-03-06T07:30:21.286698+0000","last_undegraded":"2024-03-06T07:30:21.286698+0000","last_fullsized":"2024-03-06T07:30:21.286698+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:42.407951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369822+0000","last_change":"2024-03-06T07:30:19.263713+0000","last_active":"2024-03-06T07:30:21.369822+0000","last_peered":"2024-03-06T07:30:21.369822+0000","last_clean":"2024-03-06T07:30:21.369822+0000","last_became_active":"2024-03-06T07:30:19.263358+0000","last_became_peered":"2024-03-06T07:30:19.263358+0000","last_unstale":"2024-03-06T07:30:21.369822+0000","last_undegraded":"2024-03-06T07:30:21.369822+0000","last_fullsized":"2024-03-06T07:30:21.369822+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:58.383984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369134+0000","last_change":"2024-03-06T07:30:20.291712+0000","last_active":"2024-03-06T07:30:21.369134+0000","last_peered":"2024-03-06T07:30:21.369134+0000","last_clean":"2024-03-06T07:30:21.369134+0000","last_became_active":"2024-03-06T07:30:20.291143+0000","last_became_peered":"2024-03-06T07:30:20.291143+0000","last_unstale":"2024-03-06T07:30:21.369134+0000","last_undegraded":"2024-03-06T07:30:21.369134+0000","last_fullsized":"2024-03-06T07:30:21.369134+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:32.508353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":22,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.368419+0000","last_change":"2024-03-06T07:30:19.272364+0000","last_active":"2024-03-06T07:30:21.368419+0000","last_peered":"2024-03-06T07:30:21.368419+0000","last_clean":"2024-03-06T07:30:21.368419+0000","last_became_active":"2024-03-06T07:30:19.272086+0000","last_became_peered":"2024-03-06T07:30:19.272086+0000","last_unstale":"2024-03-06T07:30:21.368419+0000","last_undegraded":"2024-03-06T07:30:21.368419+0000","last_fullsized":"2024-03-06T07:30:21.368419+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:48.104058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371831+0000","last_change":"2024-03-06T07:30:20.287848+0000","last_active":"2024-03-06T07:30:21.371831+0000","last_peered":"2024-03-06T07:30:21.371831+0000","last_clean":"2024-03-06T07:30:21.371831+0000","last_became_active":"2024-03-06T07:30:20.287462+0000","last_became_peered":"2024-03-06T07:30:20.287462+0000","last_unstale":"2024-03-06T07:30:21.371831+0000","last_undegraded":"2024-03-06T07:30:21.371831+0000","last_fullsized":"2024-03-06T07:30:21.371831+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:44.193860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365901+0000","last_change":"2024-03-06T07:30:19.267213+0000","last_active":"2024-03-06T07:30:21.365901+0000","last_peered":"2024-03-06T07:30:21.365901+0000","last_clean":"2024-03-06T07:30:21.365901+0000","last_became_active":"2024-03-06T07:30:19.266887+0000","last_became_peered":"2024-03-06T07:30:19.266887+0000","last_unstale":"2024-03-06T07:30:21.365901+0000","last_undegraded":"2024-03-06T07:30:21.365901+0000","last_fullsized":"2024-03-06T07:30:21.365901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:08.863229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.372351+0000","last_change":"2024-03-06T07:30:20.290486+0000","last_active":"2024-03-06T07:30:21.372351+0000","last_peered":"2024-03-06T07:30:21.372351+0000","last_clean":"2024-03-06T07:30:21.372351+0000","last_became_active":"2024-03-06T07:30:20.290228+0000","last_became_peered":"2024-03-06T07:30:20.290228+0000","last_unstale":"2024-03-06T07:30:21.372351+0000","last_undegraded":"2024-03-06T07:30:21.372351+0000","last_fullsized":"2024-03-06T07:30:21.372351+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:10.292220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293439+0000","last_change":"2024-03-06T07:30:19.261168+0000","last_active":"2024-03-06T07:30:21.293439+0000","last_peered":"2024-03-06T07:30:21.293439+0000","last_clean":"2024-03-06T07:30:21.293439+0000","last_became_active":"2024-03-06T07:30:19.260814+0000","last_became_peered":"2024-03-06T07:30:19.260814+0000","last_unstale":"2024-03-06T07:30:21.293439+0000","last_undegraded":"2024-03-06T07:30:21.293439+0000","last_fullsized":"2024-03-06T07:30:21.293439+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:48.104058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365679+0000","last_change":"2024-03-06T07:30:20.287099+0000","last_active":"2024-03-06T07:30:21.365679+0000","last_peered":"2024-03-06T07:30:21.365679+0000","last_clean":"2024-03-06T07:30:21.365679+0000","last_became_active":"2024-03-06T07:30:20.285675+0000","last_became_peered":"2024-03-06T07:30:20.285675+0000","last_unstale":"2024-03-06T07:30:21.365679+0000","last_undegraded":"2024-03-06T07:30:21.365679+0000","last_fullsized":"2024-03-06T07:30:21.365679+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:36.025053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371704+0000","last_change":"2024-03-06T07:30:19.264370+0000","last_active":"2024-03-06T07:30:21.371704+0000","last_peered":"2024-03-06T07:30:21.371704+0000","last_clean":"2024-03-06T07:30:21.371704+0000","last_became_active":"2024-03-06T07:30:19.264070+0000","last_became_peered":"2024-03-06T07:30:19.264070+0000","last_unstale":"2024-03-06T07:30:21.371704+0000","last_undegraded":"2024-03-06T07:30:21.371704+0000","last_fullsized":"2024-03-06T07:30:21.371704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:19.982628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.370139+0000","last_change":"2024-03-06T07:30:20.299966+0000","last_active":"2024-03-06T07:30:21.370139+0000","last_peered":"2024-03-06T07:30:21.370139+0000","last_clean":"2024-03-06T07:30:21.370139+0000","last_became_active":"2024-03-06T07:30:20.299376+0000","last_became_peered":"2024-03-06T07:30:20.299376+0000","last_unstale":"2024-03-06T07:30:21.370139+0000","last_undegraded":"2024-03-06T07:30:21.370139+0000","last_fullsized":"2024-03-06T07:30:21.370139+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:50.982845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293396+0000","last_change":"2024-03-06T07:30:19.259208+0000","last_active":"2024-03-06T07:30:21.293396+0000","last_peered":"2024-03-06T07:30:21.293396+0000","last_clean":"2024-03-06T07:30:21.293396+0000","last_became_active":"2024-03-06T07:30:19.258600+0000","last_became_peered":"2024-03-06T07:30:19.258600+0000","last_unstale":"2024-03-06T07:30:21.293396+0000","last_undegraded":"2024-03-06T07:30:21.293396+0000","last_fullsized":"2024-03-06T07:30:21.293396+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:59.901505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.370631+0000","last_change":"2024-03-06T07:30:20.292759+0000","last_active":"2024-03-06T07:30:21.370631+0000","last_peered":"2024-03-06T07:30:21.370631+0000","last_clean":"2024-03-06T07:30:21.370631+0000","last_became_active":"2024-03-06T07:30:20.292091+0000","last_became_peered":"2024-03-06T07:30:20.292091+0000","last_unstale":"2024-03-06T07:30:21.370631+0000","last_undegraded":"2024-03-06T07:30:21.370631+0000","last_fullsized":"2024-03-06T07:30:21.370631+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:20.239144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365827+0000","last_change":"2024-03-06T07:30:19.272187+0000","last_active":"2024-03-06T07:30:21.365827+0000","last_peered":"2024-03-06T07:30:21.365827+0000","last_clean":"2024-03-06T07:30:21.365827+0000","last_became_active":"2024-03-06T07:30:19.271827+0000","last_became_peered":"2024-03-06T07:30:19.271827+0000","last_unstale":"2024-03-06T07:30:21.365827+0000","last_undegraded":"2024-03-06T07:30:21.365827+0000","last_fullsized":"2024-03-06T07:30:21.365827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:19.224468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293250+0000","last_change":"2024-03-06T07:30:20.282099+0000","last_active":"2024-03-06T07:30:21.293250+0000","last_peered":"2024-03-06T07:30:21.293250+0000","last_clean":"2024-03-06T07:30:21.293250+0000","last_became_active":"2024-03-06T07:30:20.281856+0000","last_became_peered":"2024-03-06T07:30:20.281856+0000","last_unstale":"2024-03-06T07:30:21.293250+0000","last_undegraded":"2024-03-06T07:30:21.293250+0000","last_fullsized":"2024-03-06T07:30:21.293250+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:50.192009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283049+0000","last_change":"2024-03-06T07:30:19.262383+0000","last_active":"2024-03-06T07:30:21.283049+0000","last_peered":"2024-03-06T07:30:21.283049+0000","last_clean":"2024-03-06T07:30:21.283049+0000","last_became_active":"2024-03-06T07:30:19.262072+0000","last_became_peered":"2024-03-06T07:30:19.262072+0000","last_unstale":"2024-03-06T07:30:21.283049+0000","last_undegraded":"2024-03-06T07:30:21.283049+0000","last_fullsized":"2024-03-06T07:30:21.283049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:06.738966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.292808+0000","last_change":"2024-03-06T07:30:20.286543+0000","last_active":"2024-03-06T07:30:21.292808+0000","last_peered":"2024-03-06T07:30:21.292808+0000","last_clean":"2024-03-06T07:30:21.292808+0000","last_became_active":"2024-03-06T07:30:20.286295+0000","last_became_peered":"2024-03-06T07:30:20.286295+0000","last_unstale":"2024-03-06T07:30:21.292808+0000","last_undegraded":"2024-03-06T07:30:21.292808+0000","last_fullsized":"2024-03-06T07:30:21.292808+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:20.239144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.368339+0000","last_change":"2024-03-06T07:30:19.266386+0000","last_active":"2024-03-06T07:30:21.368339+0000","last_peered":"2024-03-06T07:30:21.368339+0000","last_clean":"2024-03-06T07:30:21.368339+0000","last_became_active":"2024-03-06T07:30:19.266103+0000","last_became_peered":"2024-03-06T07:30:19.266103+0000","last_unstale":"2024-03-06T07:30:21.368339+0000","last_undegraded":"2024-03-06T07:30:21.368339+0000","last_fullsized":"2024-03-06T07:30:21.368339+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:48.104058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.372052+0000","last_change":"2024-03-06T07:30:20.289262+0000","last_active":"2024-03-06T07:30:21.372052+0000","last_peered":"2024-03-06T07:30:21.372052+0000","last_clean":"2024-03-06T07:30:21.372052+0000","last_became_active":"2024-03-06T07:30:20.285558+0000","last_became_peered":"2024-03-06T07:30:20.285558+0000","last_unstale":"2024-03-06T07:30:21.372052+0000","last_undegraded":"2024-03-06T07:30:21.372052+0000","last_fullsized":"2024-03-06T07:30:21.372052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:12.629356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283667+0000","last_change":"2024-03-06T07:30:19.258588+0000","last_active":"2024-03-06T07:30:21.283667+0000","last_peered":"2024-03-06T07:30:21.283667+0000","last_clean":"2024-03-06T07:30:21.283667+0000","last_became_active":"2024-03-06T07:30:19.258244+0000","last_became_peered":"2024-03-06T07:30:19.258244+0000","last_unstale":"2024-03-06T07:30:21.283667+0000","last_undegraded":"2024-03-06T07:30:21.283667+0000","last_fullsized":"2024-03-06T07:30:21.283667+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:38.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.372042+0000","last_change":"2024-03-06T07:30:20.294169+0000","last_active":"2024-03-06T07:30:21.372042+0000","last_peered":"2024-03-06T07:30:21.372042+0000","last_clean":"2024-03-06T07:30:21.372042+0000","last_became_active":"2024-03-06T07:30:20.293743+0000","last_became_peered":"2024-03-06T07:30:20.293743+0000","last_unstale":"2024-03-06T07:30:21.372042+0000","last_undegraded":"2024-03-06T07:30:21.372042+0000","last_fullsized":"2024-03-06T07:30:21.372042+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:32.508353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293599+0000","last_change":"2024-03-06T07:30:19.264608+0000","last_active":"2024-03-06T07:30:21.293599+0000","last_peered":"2024-03-06T07:30:21.293599+0000","last_clean":"2024-03-06T07:30:21.293599+0000","last_became_active":"2024-03-06T07:30:19.264351+0000","last_became_peered":"2024-03-06T07:30:19.264351+0000","last_unstale":"2024-03-06T07:30:21.293599+0000","last_undegraded":"2024-03-06T07:30:21.293599+0000","last_fullsized":"2024-03-06T07:30:21.293599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:21.031644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.283258+0000","last_change":"2024-03-06T07:30:20.284017+0000","last_active":"2024-03-06T07:30:21.283258+0000","last_peered":"2024-03-06T07:30:21.283258+0000","last_clean":"2024-03-06T07:30:21.283258+0000","last_became_active":"2024-03-06T07:30:20.283764+0000","last_became_peered":"2024-03-06T07:30:20.283764+0000","last_unstale":"2024-03-06T07:30:21.283258+0000","last_undegraded":"2024-03-06T07:30:21.283258+0000","last_fullsized":"2024-03-06T07:30:21.283258+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:13.245682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.298541+0000","last_change":"2024-03-06T07:30:19.270641+0000","last_active":"2024-03-06T07:30:21.298541+0000","last_peered":"2024-03-06T07:30:21.298541+0000","last_clean":"2024-03-06T07:30:21.298541+0000","last_became_active":"2024-03-06T07:30:19.269394+0000","last_became_peered":"2024-03-06T07:30:19.269394+0000","last_unstale":"2024-03-06T07:30:21.298541+0000","last_undegraded":"2024-03-06T07:30:21.298541+0000","last_fullsized":"2024-03-06T07:30:21.298541+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:34.407724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.293603+0000","last_change":"2024-03-06T07:30:20.285606+0000","last_active":"2024-03-06T07:30:21.293603+0000","last_peered":"2024-03-06T07:30:21.293603+0000","last_clean":"2024-03-06T07:30:21.293603+0000","last_became_active":"2024-03-06T07:30:20.285367+0000","last_became_peered":"2024-03-06T07:30:20.285367+0000","last_unstale":"2024-03-06T07:30:21.293603+0000","last_undegraded":"2024-03-06T07:30:21.293603+0000","last_fullsized":"2024-03-06T07:30:21.293603+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:54.192021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.371411+0000","last_change":"2024-03-06T07:30:19.271789+0000","last_active":"2024-03-06T07:30:21.371411+0000","last_peered":"2024-03-06T07:30:21.371411+0000","last_clean":"2024-03-06T07:30:21.371411+0000","last_became_active":"2024-03-06T07:30:19.271534+0000","last_became_peered":"2024-03-06T07:30:19.271534+0000","last_unstale":"2024-03-06T07:30:21.371411+0000","last_undegraded":"2024-03-06T07:30:21.371411+0000","last_fullsized":"2024-03-06T07:30:21.371411+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:41.548200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365539+0000","last_change":"2024-03-06T07:30:20.291192+0000","last_active":"2024-03-06T07:30:21.365539+0000","last_peered":"2024-03-06T07:30:21.365539+0000","last_clean":"2024-03-06T07:30:21.365539+0000","last_became_active":"2024-03-06T07:30:20.289870+0000","last_became_peered":"2024-03-06T07:30:20.289870+0000","last_unstale":"2024-03-06T07:30:21.365539+0000","last_undegraded":"2024-03-06T07:30:21.365539+0000","last_fullsized":"2024-03-06T07:30:21.365539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:23.539550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.282881+0000","last_change":"2024-03-06T07:30:19.263995+0000","last_active":"2024-03-06T07:30:21.282881+0000","last_peered":"2024-03-06T07:30:21.282881+0000","last_clean":"2024-03-06T07:30:21.282881+0000","last_became_active":"2024-03-06T07:30:19.263617+0000","last_became_peered":"2024-03-06T07:30:19.263617+0000","last_unstale":"2024-03-06T07:30:21.282881+0000","last_undegraded":"2024-03-06T07:30:21.282881+0000","last_fullsized":"2024-03-06T07:30:21.282881+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:06.736990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.369227+0000","last_change":"2024-03-06T07:30:20.303749+0000","last_active":"2024-03-06T07:30:21.369227+0000","last_peered":"2024-03-06T07:30:21.369227+0000","last_clean":"2024-03-06T07:30:21.369227+0000","last_became_active":"2024-03-06T07:30:20.301920+0000","last_became_peered":"2024-03-06T07:30:20.301920+0000","last_unstale":"2024-03-06T07:30:21.369227+0000","last_undegraded":"2024-03-06T07:30:21.369227+0000","last_fullsized":"2024-03-06T07:30:21.369227+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:10.292220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.287078+0000","last_change":"2024-03-06T07:30:19.271352+0000","last_active":"2024-03-06T07:30:21.287078+0000","last_peered":"2024-03-06T07:30:21.287078+0000","last_clean":"2024-03-06T07:30:21.287078+0000","last_became_active":"2024-03-06T07:30:19.270609+0000","last_became_peered":"2024-03-06T07:30:19.270609+0000","last_unstale":"2024-03-06T07:30:21.287078+0000","last_undegraded":"2024-03-06T07:30:21.287078+0000","last_fullsized":"2024-03-06T07:30:21.287078+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:54.848786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.282838+0000","last_change":"2024-03-06T07:30:20.288772+0000","last_active":"2024-03-06T07:30:21.282838+0000","last_peered":"2024-03-06T07:30:21.282838+0000","last_clean":"2024-03-06T07:30:21.282838+0000","last_became_active":"2024-03-06T07:30:20.288520+0000","last_became_peered":"2024-03-06T07:30:20.288520+0000","last_unstale":"2024-03-06T07:30:21.282838+0000","last_undegraded":"2024-03-06T07:30:21.282838+0000","last_fullsized":"2024-03-06T07:30:21.282838+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:42.944853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.282826+0000","last_change":"2024-03-06T07:30:19.245157+0000","last_active":"2024-03-06T07:30:21.282826+0000","last_peered":"2024-03-06T07:30:21.282826+0000","last_clean":"2024-03-06T07:30:21.282826+0000","last_became_active":"2024-03-06T07:30:19.244811+0000","last_became_peered":"2024-03-06T07:30:19.244811+0000","last_unstale":"2024-03-06T07:30:21.282826+0000","last_undegraded":"2024-03-06T07:30:21.282826+0000","last_fullsized":"2024-03-06T07:30:21.282826+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:34.407724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365410+0000","last_change":"2024-03-06T07:30:20.286646+0000","last_active":"2024-03-06T07:30:21.365410+0000","last_peered":"2024-03-06T07:30:21.365410+0000","last_clean":"2024-03-06T07:30:21.365410+0000","last_became_active":"2024-03-06T07:30:20.286335+0000","last_became_peered":"2024-03-06T07:30:20.286335+0000","last_unstale":"2024-03-06T07:30:21.365410+0000","last_undegraded":"2024-03-06T07:30:21.365410+0000","last_fullsized":"2024-03-06T07:30:21.365410+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:16.685776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365288+0000","last_change":"2024-03-06T07:30:19.267941+0000","last_active":"2024-03-06T07:30:21.365288+0000","last_peered":"2024-03-06T07:30:21.365288+0000","last_clean":"2024-03-06T07:30:21.365288+0000","last_became_active":"2024-03-06T07:30:19.267676+0000","last_became_peered":"2024-03-06T07:30:19.267676+0000","last_unstale":"2024-03-06T07:30:21.365288+0000","last_undegraded":"2024-03-06T07:30:21.365288+0000","last_fullsized":"2024-03-06T07:30:21.365288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:56.930970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.365262+0000","last_change":"2024-03-06T07:30:20.281089+0000","last_active":"2024-03-06T07:30:21.365262+0000","last_peered":"2024-03-06T07:30:21.365262+0000","last_clean":"2024-03-06T07:30:21.365262+0000","last_became_active":"2024-03-06T07:30:20.280731+0000","last_became_peered":"2024-03-06T07:30:20.280731+0000","last_unstale":"2024-03-06T07:30:21.365262+0000","last_undegraded":"2024-03-06T07:30:21.365262+0000","last_fullsized":"2024-03-06T07:30:21.365262+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.227217+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.227217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:37.527541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.298953+0000","last_change":"2024-03-06T07:30:19.268373+0000","last_active":"2024-03-06T07:30:21.298953+0000","last_peered":"2024-03-06T07:30:21.298953+0000","last_clean":"2024-03-06T07:30:21.298953+0000","last_became_active":"2024-03-06T07:30:19.267949+0000","last_became_peered":"2024-03-06T07:30:19.267949+0000","last_unstale":"2024-03-06T07:30:21.298953+0000","last_undegraded":"2024-03-06T07:30:21.298953+0000","last_fullsized":"2024-03-06T07:30:21.298953+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:18.212541+0000","last_clean_scrub_stamp":"2024-03-06T07:30:18.212541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:31.611474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":235,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.368431+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:30:21.368431+0000","last_peered":"2024-03-06T07:30:21.368431+0000","last_clean":"2024-03-06T07:30:21.368431+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:30:21.368431+0000","last_undegraded":"2024-03-06T07:30:21.368431+0000","last_fullsized":"2024-03-06T07:30:21.368431+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542180,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":877319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542181,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":286791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542181,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":287443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542181,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":704512,"data_stored":288403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542180,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":700416,"data_stored":286847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542182,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":286847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":696320,"data_stored":286881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542181,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":879353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:25.407 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:25.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:25.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.688+0000 7f6303e00640 1 Processor -- start 2024-03-06T07:30:25.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.688+0000 7f6303e00640 1 -- start start 2024-03-06T07:30:25.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f6303e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63040cf240 0x7f63040d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f6303e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63040b3190 con 0x7f630415e2d0 2024-03-06T07:30:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f6303e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63040cfae0 con 0x7f63040d1f20 2024-03-06T07:30:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f6303e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040cf800 con 0x7f63040cf240 2024-03-06T07:30:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f6303400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63040cf240 0x7f63040d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f62fbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f630415e2d0 0x7f63040b3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46290/0 (socket says 172.21.15.97:46290) 2024-03-06T07:30:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.690+0000 7f62fbe00640 1 -- 172.21.15.97:0/1079480072 learned_addr learned my addr 172.21.15.97:0/1079480072 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.691+0000 7f6303400640 1 -- 172.21.15.97:0/1079480072 >> v1:172.21.15.135:6789/0 conn(0x7f63040d1f20 legacy=0x7f63040d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.691+0000 7f6303400640 1 -- 172.21.15.97:0/1079480072 >> v1:172.21.15.97:6789/0 conn(0x7f630415e2d0 legacy=0x7f63040b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.691+0000 7f6303400640 1 -- 172.21.15.97:0/1079480072 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040b3430 con 0x7f63040cf240 2024-03-06T07:30:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.691+0000 7f6302000640 1 -- 172.21.15.97:0/1079480072 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283257888 0 0) 0x7f63040b3190 con 0x7f630415e2d0 2024-03-06T07:30:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.692+0000 7f6303400640 1 --2- 172.21.15.97:0/1079480072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040cf240 0x7f63040d1860 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009d90 tx=0x7f62fc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5eca8ba90ecfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.692+0000 7f6302000640 1 -- 172.21.15.97:0/1079480072 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc039070 con 0x7f63040cf240 2024-03-06T07:30:25.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.692+0000 7f6302000640 1 -- 172.21.15.97:0/1079480072 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc02cd00 con 0x7f63040cf240 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.692+0000 7f6302000640 1 -- 172.21.15.97:0/1079480072 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc034660 con 0x7f63040cf240 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 -- 172.21.15.97:0/1079480072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040cf240 msgr2=0x7f63040d1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 --2- 172.21.15.97:0/1079480072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040cf240 0x7f63040d1860 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009d90 tx=0x7f62fc02bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 -- 172.21.15.97:0/1079480072 shutdown_connections 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 --2- 172.21.15.97:0/1079480072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040cf240 0x7f63040d1860 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 -- 172.21.15.97:0/1079480072 >> 172.21.15.97:0/1079480072 conn(0x7f63040d3c90 msgr2=0x7f63040d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 -- 172.21.15.97:0/1079480072 shutdown_connections 2024-03-06T07:30:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.693+0000 7f6303e00640 1 -- 172.21.15.97:0/1079480072 wait complete. 2024-03-06T07:30:25.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.695+0000 7f6303e00640 1 Processor -- start 2024-03-06T07:30:25.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.695+0000 7f6303e00640 1 -- start start 2024-03-06T07:30:25.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6303e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63040cf240 0x7f63040c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6303e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 0x7f63040c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6303e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f630415e2d0 0x7f63041f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6303e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b44c0 con 0x7f63040d1f20 2024-03-06T07:30:25.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6303e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b4340 con 0x7f63040cf240 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6303e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b4640 con 0x7f630415e2d0 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f62fbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 0x7f63040c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f62fbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 0x7f63040c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52530/0 (socket says 172.21.15.97:52530) 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f62fbe00640 1 -- 172.21.15.97:0/1779514600 learned_addr learned my addr 172.21.15.97:0/1779514600 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f6302a00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63040cf240 0x7f63040c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f62fbe00640 1 -- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f630415e2d0 msgr2=0x7f63041f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f62fbe00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f630415e2d0 0x7f63041f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.696+0000 7f62fbe00640 1 -- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63040cf240 msgr2=0x7f63040c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f62fbe00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63040cf240 0x7f63040c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f62fbe00640 1 -- 172.21.15.97:0/1779514600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040c7c50 con 0x7f63040d1f20 2024-03-06T07:30:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f6302a00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63040cf240 0x7f63040c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f62fbe00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 0x7f63040c93a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00c3c0 tx=0x7f62ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0074e0 con 0x7f63040d1f20 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec0047e0 con 0x7f63040d1f20 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63040c7f30 con 0x7f63040d1f20 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec004bd0 con 0x7f63040d1f20 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.697+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63040c84f0 con 0x7f63040d1f20 2024-03-06T07:30:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.699+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6304075480 con 0x7f63040d1f20 2024-03-06T07:30:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.699+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62ec011070 con 0x7f63040d1f20 2024-03-06T07:30:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.700+0000 7f62faa00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f62e006b4d0 0x7f62e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.700+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62ec089de0 con 0x7f63040d1f20 2024-03-06T07:30:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.700+0000 7f6302a00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f62e006b4d0 0x7f62e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.701+0000 7f6302a00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f62e006b4d0 0x7f62e006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f62f4000f60 tx=0x7f62f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:25.705+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62ec00e280 con 0x7f63040d1f20 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.186+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f63040ce940 con 0x7f63040d1f20 2024-03-06T07:30:26.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.189+0000 7f62faa00640 1 -- 172.21.15.97:0/1779514600 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f62ec058c60 con 0x7f63040d1f20 2024-03-06T07:30:26.192 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:30:26.192 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:30:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:30:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:30:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:26.193 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:30:26.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f62e006b4d0 msgr2=0x7f62e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f62e006b4d0 0x7f62e006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f62f4000f60 tx=0x7f62f4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 msgr2=0x7f63040c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 0x7f63040c93a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00c3c0 tx=0x7f62ec00c880 comp rx=0 tx=0).stop 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 shutdown_connections 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f62e006b4d0 0x7f62e006d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f630415e2d0 0x7f63041f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63040d1f20 0x7f63040c93a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 --2- 172.21.15.97:0/1779514600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63040cf240 0x7f63040c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.192+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 >> 172.21.15.97:0/1779514600 conn(0x7f63040d3c90 msgr2=0x7f63040d2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.193+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 shutdown_connections 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.193+0000 7f6303e00640 1 -- 172.21.15.97:0/1779514600 wait complete. 2024-03-06T07:30:26.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.409+0000 7faee7800640 1 Processor -- start 2024-03-06T07:30:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.409+0000 7faee7800640 1 -- start start 2024-03-06T07:30:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.410+0000 7faee7800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7faee00af820 0x7faee00b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.410+0000 7faee7800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faee00b38f0 0x7faee006b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.410+0000 7faee7800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee00b3da0 con 0x7faee015e160 2024-03-06T07:30:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.410+0000 7faee7800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee006b0f0 con 0x7faee00b38f0 2024-03-06T07:30:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.410+0000 7faee7800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00b5620 con 0x7faee00af820 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faee00b38f0 0x7faee006b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faee00b38f0 0x7faee006b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52536/0 (socket says 172.21.15.97:52536) 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 -- 172.21.15.97:0/2335748508 learned_addr learned my addr 172.21.15.97:0/2335748508 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 -- 172.21.15.97:0/2335748508 >> v2:172.21.15.135:3300/0 conn(0x7faee00af820 msgr2=0x7faee00b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 --2- 172.21.15.97:0/2335748508 >> v2:172.21.15.135:3300/0 conn(0x7faee00af820 0x7faee00b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 -- 172.21.15.97:0/2335748508 >> v1:172.21.15.135:6789/0 conn(0x7faee015e160 legacy=0x7faee0162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.411+0000 7faee5800640 1 -- 172.21.15.97:0/2335748508 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee006f1f0 con 0x7faee00b38f0 2024-03-06T07:30:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.412+0000 7faee5800640 1 --2- 172.21.15.97:0/2335748508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee006b580 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faee00d11b0 tx=0x7faed802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184974a09bea3275 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.412+0000 7faedf400640 1 -- 172.21.15.97:0/2335748508 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed802c9f0 con 0x7faee00b38f0 2024-03-06T07:30:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faedf400640 1 -- 172.21.15.97:0/2335748508 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed802cb90 con 0x7faee00b38f0 2024-03-06T07:30:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faedf400640 1 -- 172.21.15.97:0/2335748508 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed80343f0 con 0x7faee00b38f0 2024-03-06T07:30:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 -- 172.21.15.97:0/2335748508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 msgr2=0x7faee006b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 --2- 172.21.15.97:0/2335748508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee006b580 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faee00d11b0 tx=0x7faed802be50 comp rx=0 tx=0).stop 2024-03-06T07:30:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 -- 172.21.15.97:0/2335748508 shutdown_connections 2024-03-06T07:30:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 --2- 172.21.15.97:0/2335748508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee006b580 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 --2- 172.21.15.97:0/2335748508 >> v2:172.21.15.135:3300/0 conn(0x7faee00af820 0x7faee00b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 -- 172.21.15.97:0/2335748508 >> 172.21.15.97:0/2335748508 conn(0x7faee00d0340 msgr2=0x7faee00d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 -- 172.21.15.97:0/2335748508 shutdown_connections 2024-03-06T07:30:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.413+0000 7faee7800640 1 -- 172.21.15.97:0/2335748508 wait complete. 2024-03-06T07:30:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.415+0000 7faee7800640 1 Processor -- start 2024-03-06T07:30:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.415+0000 7faee7800640 1 -- start start 2024-03-06T07:30:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faee7800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faee00af820 0x7faee0147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faee7800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee00cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faee7800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee015e160 0x7faee00b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faee7800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee006ef10 con 0x7faee00b38f0 2024-03-06T07:30:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faee7800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee006ec30 con 0x7faee015e160 2024-03-06T07:30:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faee7800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee006bac0 con 0x7faee00af820 2024-03-06T07:30:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faedfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee00cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.416+0000 7faedfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee00cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52552/0 (socket says 172.21.15.97:52552) 2024-03-06T07:30:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 -- 172.21.15.97:0/950685646 learned_addr learned my addr 172.21.15.97:0/950685646 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faee4e00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faee00af820 0x7faee0147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 -- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faee00af820 msgr2=0x7faee0147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faee00af820 0x7faee0147f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 -- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee015e160 msgr2=0x7faee00b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee015e160 0x7faee00b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 -- 172.21.15.97:0/950685646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee01f5070 con 0x7faee00b38f0 2024-03-06T07:30:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedfe00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee00cf580 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7faecc00b450 tx=0x7faecc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.417+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc015020 con 0x7faee00b38f0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.418+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc011890 con 0x7faee00b38f0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.418+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee01f44f0 con 0x7faee00b38f0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.418+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc011c80 con 0x7faee00b38f0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.418+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee01f4ab0 con 0x7faee00b38f0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.419+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee006ea50 con 0x7faee00b38f0 2024-03-06T07:30:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.420+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faecc018020 con 0x7faee00b38f0 2024-03-06T07:30:26.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.421+0000 7faedcc00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faec006b4d0 0x7faec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.421+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faecc08a370 con 0x7faee00b38f0 2024-03-06T07:30:26.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.421+0000 7faee4e00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faec006b4d0 0x7faec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.422+0000 7faee4e00640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faec006b4d0 0x7faec006d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faed400a730 tx=0x7faed4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.425+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faee006ea50 con 0x7faee00b38f0 2024-03-06T07:30:26.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.646+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee00d11e0 con 0x7faee00b38f0 2024-03-06T07:30:26.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.647+0000 7faedcc00640 1 -- 172.21.15.97:0/950685646 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7faecc01f020 con 0x7faee00b38f0 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.649+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faec006b4d0 msgr2=0x7faec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.649+0000 7faee7800640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faec006b4d0 0x7faec006d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faed400a730 tx=0x7faed4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.649+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 msgr2=0x7faee00cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.649+0000 7faee7800640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee00cf580 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7faecc00b450 tx=0x7faecc00b910 comp rx=0 tx=0).stop 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 shutdown_connections 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faec006b4d0 0x7faec006d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee015e160 0x7faee00b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faee00b38f0 0x7faee00cf580 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 --2- 172.21.15.97:0/950685646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faee00af820 0x7faee0147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 >> 172.21.15.97:0/950685646 conn(0x7faee00d0340 msgr2=0x7faee0160db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 shutdown_connections 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:26.650+0000 7faee7800640 1 -- 172.21.15.97:0/950685646 wait complete. 2024-03-06T07:30:27.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:19.710683+0000', 'modified': '2024-03-06T07:30:22.764725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5865}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5865': {'gid': 5865, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/24115969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 24115969}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 24115969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5865, 'qdb_cluster': [5865]} 2024-03-06T07:30:27.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:27.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:30:27.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.283+0000 7f8bfd600640 1 Processor -- start 2024-03-06T07:30:27.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.284+0000 7f8bfd600640 1 -- start start 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.286+0000 7f8bfd600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8bf8159b90 0x7f8bf815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.286+0000 7f8bfd600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8bf806afc0 0x7f8bf806b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.286+0000 7f8bfd600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf806b910 con 0x7f8bf80d1a30 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bfd600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf815cb30 con 0x7f8bf8159b90 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bfd600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf80cd640 con 0x7f8bf806afc0 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8bf8159b90 0x7f8bf815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8bf8159b90 0x7f8bf815cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52554/0 (socket says 172.21.15.97:52554) 2024-03-06T07:30:27.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 -- 172.21.15.97:0/3808059978 learned_addr learned my addr 172.21.15.97:0/3808059978 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 -- 172.21.15.97:0/3808059978 >> v2:172.21.15.135:3300/0 conn(0x7f8bf806afc0 msgr2=0x7f8bf806b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/3808059978 >> v2:172.21.15.135:3300/0 conn(0x7f8bf806afc0 0x7f8bf806b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 -- 172.21.15.97:0/3808059978 >> v1:172.21.15.135:6789/0 conn(0x7f8bf80d1a30 legacy=0x7f8bf80cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.287+0000 7f8bf6a00640 1 -- 172.21.15.97:0/3808059978 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf80cd360 con 0x7f8bf8159b90 2024-03-06T07:30:27.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.288+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/3808059978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf8159b90 0x7f8bf815cfc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80709b0 tx=0x7f8be802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15e4affe5a51f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.288+0000 7f8bf6000640 1 -- 172.21.15.97:0/3808059978 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be802e070 con 0x7f8bf8159b90 2024-03-06T07:30:27.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bf6000640 1 -- 172.21.15.97:0/3808059978 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be8037070 con 0x7f8bf8159b90 2024-03-06T07:30:27.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bf6000640 1 -- 172.21.15.97:0/3808059978 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8031a00 con 0x7f8bf8159b90 2024-03-06T07:30:27.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 -- 172.21.15.97:0/3808059978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf8159b90 msgr2=0x7f8bf815cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3808059978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf8159b90 0x7f8bf815cfc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80709b0 tx=0x7f8be802b880 comp rx=0 tx=0).stop 2024-03-06T07:30:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 -- 172.21.15.97:0/3808059978 shutdown_connections 2024-03-06T07:30:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3808059978 >> v2:172.21.15.135:3300/0 conn(0x7f8bf806afc0 0x7f8bf806b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3808059978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf8159b90 0x7f8bf815cfc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 -- 172.21.15.97:0/3808059978 >> 172.21.15.97:0/3808059978 conn(0x7f8bf80d0df0 msgr2=0x7f8bf80d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 -- 172.21.15.97:0/3808059978 shutdown_connections 2024-03-06T07:30:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.289+0000 7f8bfd600640 1 -- 172.21.15.97:0/3808059978 wait complete. 2024-03-06T07:30:27.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.291+0000 7f8bfd600640 1 Processor -- start 2024-03-06T07:30:27.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.291+0000 7f8bfd600640 1 -- start start 2024-03-06T07:30:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bfd600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bf806afc0 0x7f8bf814f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bfd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bf8159b90 0x7f8bf815e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bfd600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 0x7f8bf80af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bfd600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf80cd020 con 0x7f8bf80af0b0 2024-03-06T07:30:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bfd600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf815c7c0 con 0x7f8bf806afc0 2024-03-06T07:30:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bfd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf815f9c0 con 0x7f8bf8159b90 2024-03-06T07:30:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bf7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 0x7f8bf80af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bf7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 0x7f8bf80af560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52556/0 (socket says 172.21.15.97:52556) 2024-03-06T07:30:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.292+0000 7f8bf7400640 1 -- 172.21.15.97:0/3823818388 learned_addr learned my addr 172.21.15.97:0/3823818388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bf806afc0 0x7f8bf814f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bee000640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bf8159b90 0x7f8bf815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf7400640 1 -- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bf8159b90 msgr2=0x7f8bf815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf7400640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bf8159b90 0x7f8bf815e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf7400640 1 -- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bf806afc0 msgr2=0x7f8bf814f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf7400640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bf806afc0 0x7f8bf814f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf7400640 1 -- 172.21.15.97:0/3823818388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf814f950 con 0x7f8bf80af0b0 2024-03-06T07:30:27.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bf7400640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 0x7f8bf80af560 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000b9a0 tx=0x7f8bf000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bef400640 1 -- 172.21.15.97:0/3823818388 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0019070 con 0x7f8bf80af0b0 2024-03-06T07:30:27.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bef400640 1 -- 172.21.15.97:0/3823818388 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf000cbc0 con 0x7f8bf80af0b0 2024-03-06T07:30:27.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bef400640 1 -- 172.21.15.97:0/3823818388 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0014440 con 0x7f8bf80af0b0 2024-03-06T07:30:27.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf815ec50 con 0x7f8bf80af0b0 2024-03-06T07:30:27.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.293+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf806e6e0 con 0x7f8bf80af0b0 2024-03-06T07:30:27.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.297+0000 7f8bef400640 1 -- 172.21.15.97:0/3823818388 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bf001c070 con 0x7f8bf80af0b0 2024-03-06T07:30:27.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.297+0000 7f8c003a1740 1 Processor -- start 2024-03-06T07:30:27.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.297+0000 7f8bef400640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bd406b4d0 0x7f8bd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.297+0000 7f8c003a1740 1 -- start start 2024-03-06T07:30:27.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.298+0000 7f8bef400640 1 -- 172.21.15.97:0/3823818388 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bf0089ea0 con 0x7f8bf80af0b0 2024-03-06T07:30:27.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.298+0000 7f8bee000640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bd406b4d0 0x7f8bd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.298+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x5572001e5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.298+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200215540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 0x557200215a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8c003a1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55720014f600 con 0x5572001e5c90 2024-03-06T07:30:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572000f8930 con 0x557200212b90 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55720014b6a0 con 0x5572002127b0 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bee000640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bd406b4d0 0x7f8bd406d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8bf815f410 tx=0x7f8be0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 0x557200215a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 0x557200215a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52568/0 (socket says 172.21.15.97:52568) 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 -- 172.21.15.97:0/177946454 learned_addr learned my addr 172.21.15.97:0/177946454 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x5572001e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bee000640 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200215540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 -- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 msgr2=0x5572001e5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x5572001e5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 -- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 msgr2=0x557200215540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200215540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bf7400640 1 -- 172.21.15.97:0/177946454 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571ffe642d0 con 0x5572001e5c90 2024-03-06T07:30:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.299+0000 7f8bee000640 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200215540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.300+0000 7f8bf7400640 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 0x557200215a80 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0088480 tx=0x7f8bf0058140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659882b768fb1e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.300+0000 7f8be7000640 1 -- 172.21.15.97:0/177946454 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0060dc0 con 0x5572001e5c90 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.300+0000 7f8be7000640 1 -- 172.21.15.97:0/177946454 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0070170 con 0x5572001e5c90 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.300+0000 7f8be7000640 1 -- 172.21.15.97:0/177946454 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0054d20 con 0x5572001e5c90 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.300+0000 7f8c003a1740 1 -- 172.21.15.97:0/177946454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 msgr2=0x557200215a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.300+0000 7f8c003a1740 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 0x557200215a80 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0088480 tx=0x7f8bf0058140 comp rx=0 tx=0).stop 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 -- 172.21.15.97:0/177946454 shutdown_connections 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5572001e5c90 0x557200215a80 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200215540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 --2- 172.21.15.97:0/177946454 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x5572001e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 -- 172.21.15.97:0/177946454 >> 172.21.15.97:0/177946454 conn(0x5572001e79c0 msgr2=0x557200230540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 -- 172.21.15.97:0/177946454 shutdown_connections 2024-03-06T07:30:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 -- 172.21.15.97:0/177946454 wait complete. 2024-03-06T07:30:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.301+0000 7f8c003a1740 1 -- start start 2024-03-06T07:30:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.302+0000 7f8c003a1740 1 Processor -- start 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5572002127b0 0x557200230a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x55720020deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x557200217410 0x55720020baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572001e9f60 con 0x557200212b90 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55720010bd90 con 0x5572002127b0 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572001e9350 con 0x557200217410 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8c003a1740 10 client.0 _ll_register_callbacks cb 0x557200147350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:27.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x55720020deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x55720020deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52576/0 (socket says 172.21.15.97:52576) 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 -- 172.21.15.97:0/2474073902 learned_addr learned my addr 172.21.15.97:0/2474073902 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 -- 172.21.15.97:0/2474073902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x557200217410 msgr2=0x55720020baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x557200217410 0x55720020baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 -- 172.21.15.97:0/2474073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5572002127b0 msgr2=0x557200230a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5572002127b0 0x557200230a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.303+0000 7f8bee000640 1 -- 172.21.15.97:0/2474073902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571fff2e6b0 con 0x557200212b90 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8bee000640 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x55720020deb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8be0003cb0 tx=0x7f8be0000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0efd9ee81dd60d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8bdb400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8bdb400640 1 -- 172.21.15.97:0/2474073902 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0021630 con 0x557200212b90 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8bdb400640 1 -- 172.21.15.97:0/2474073902 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8be0020a80 con 0x557200212b90 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bf0070170 con 0x557200212b90 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8bdb400640 1 -- 172.21.15.97:0/2474073902 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8be0020cc0 con 0x557200212b90 2024-03-06T07:30:27.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8c003a1740 10 client.6018 fetch_fsmap learned FSMap version 34 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.304+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x557200214720 con 0x557200212b90 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8bdb400640 1 -- 172.21.15.97:0/2474073902 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0070170 con 0x557200212b90 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8bdb400640 1 -- 172.21.15.97:0/2474073902 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8be0020420 con 0x557200212b90 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8c003a1740 10 client.6018 fetch_fsmap finished waiting for FSMap version 34 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8c003a1740 10 client.6018 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8c003a1740 20 client.6018 populate_metadata read hostname 'smithi097' 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8c003a1740 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200244aa0 0x557200244e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8c003a1740 4 client.6018 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 --> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55720015cb40 con 0x557200244aa0 2024-03-06T07:30:27.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.305+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200244aa0 0x557200244e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.306+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200244aa0 0x557200244e80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc2b6f1d3e706b2 server_cookie=473441afd2b91add in_seq=0 out_seq=0 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.306+0000 7f8bdb400640 10 client.6018 ms_handle_connect on v2:172.21.15.97:6835/24115969 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.307+0000 7f8bdb400640 1 -- 172.21.15.97:0/2474073902 <== mds.0 v2:172.21.15.97:6835/24115969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55720015cb40 con 0x557200244aa0 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.307+0000 7f8bdb400640 10 client.6018 handle_command_reply: tid=0 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.307+0000 7f8c003a1740 1 client.6018 shutdown 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.307+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 msgr2=0x55720020deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x55720020deb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8be0003cb0 tx=0x7f8be0000fb0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 shutdown_connections 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200244aa0 0x557200244e80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x557200217410 0x55720020baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x55720020deb0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8bdb400640 0 client.6018 ms_handle_reset on v2:172.21.15.97:6835/24115969 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 --2- 172.21.15.97:0/2474073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5572002127b0 0x557200230a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 >> 172.21.15.97:0/2474073902 conn(0x5572001e5c90 msgr2=0x557200233a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 shutdown_connections 2024-03-06T07:30:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 1 -- 172.21.15.97:0/2474073902 wait complete. 2024-03-06T07:30:27.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.308+0000 7f8c003a1740 20 client.6018 trim_cache size 0 max 16384 2024-03-06T07:30:27.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.345+0000 7f8c003a1740 1 Processor -- start 2024-03-06T07:30:27.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.345+0000 7f8c003a1740 1 -- start start 2024-03-06T07:30:27.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.346+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x557200216950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.346+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200254250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.346+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 0x557200297800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.346+0000 7f8c003a1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf81ff6b0 con 0x557200217410 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.346+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf815d9a0 con 0x557200212b90 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.346+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf81ff110 con 0x5572002127b0 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 0x557200297800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 0x557200297800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52584/0 (socket says 172.21.15.97:52584) 2024-03-06T07:30:27.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 -- 172.21.15.97:0/1263186599 learned_addr learned my addr 172.21.15.97:0/1263186599 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 -- 172.21.15.97:0/1263186599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 msgr2=0x557200216950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x557200216950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 -- 172.21.15.97:0/1263186599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 msgr2=0x557200254250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200254250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 -- 172.21.15.97:0/1263186599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55720015d270 con 0x557200217410 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8bf7400640 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 0x557200297800 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0058360 tx=0x7f8bf0016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0fe54c3cf0ecba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.347+0000 7f8be6600640 1 -- 172.21.15.97:0/1263186599 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0060dc0 con 0x557200217410 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8be6600640 1 -- 172.21.15.97:0/1263186599 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0070170 con 0x557200217410 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8be6600640 1 -- 172.21.15.97:0/1263186599 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0054d20 con 0x557200217410 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 -- 172.21.15.97:0/1263186599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 msgr2=0x557200297800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 0x557200297800 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0058360 tx=0x7f8bf0016040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 -- 172.21.15.97:0/1263186599 shutdown_connections 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200217410 0x557200297800 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200212b90 0x557200254250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 --2- 172.21.15.97:0/1263186599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x557200216950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 -- 172.21.15.97:0/1263186599 >> 172.21.15.97:0/1263186599 conn(0x5572001e5c90 msgr2=0x5572001e09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.348+0000 7f8c003a1740 1 -- 172.21.15.97:0/1263186599 shutdown_connections 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.349+0000 7f8c003a1740 1 -- 172.21.15.97:0/1263186599 wait complete. 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.349+0000 7f8c003a1740 1 -- start start 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.349+0000 7f8c003a1740 1 Processor -- start 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x557200218420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x5572001e9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200217410 0x5572001e9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557200217980 con 0x557200212b90 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x557200214a60 con 0x557200217410 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x557200215720 con 0x5572002127b0 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.350+0000 7f8c003a1740 10 client.0 _ll_register_callbacks cb 0x7f8be00012e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x5572001e9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x5572001e9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52598/0 (socket says 172.21.15.97:52598) 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 -- 172.21.15.97:0/4200334000 learned_addr learned my addr 172.21.15.97:0/4200334000 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 -- 172.21.15.97:0/4200334000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 msgr2=0x557200218420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x557200218420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 -- 172.21.15.97:0/4200334000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200217410 msgr2=0x5572001e9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200217410 0x5572001e9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 -- 172.21.15.97:0/4200334000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557200218d70 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bee000640 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x5572001e9590 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x557200212590 tx=0x7f8be0001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636fb1b2663921b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.351+0000 7f8bdb400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8bdb400640 1 -- 172.21.15.97:0/4200334000 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be002f040 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8bdb400640 1 -- 172.21.15.97:0/4200334000 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8be0035cf0 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bf0070170 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8bdb400640 1 -- 172.21.15.97:0/4200334000 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8be000dca0 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8c003a1740 10 client.6024 fetch_fsmap learned FSMap version 34 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x5572001eac40 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8bdb400640 1 -- 172.21.15.97:0/4200334000 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0070170 con 0x557200212b90 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8bdb400640 1 -- 172.21.15.97:0/4200334000 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8be003c050 con 0x557200212b90 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8c003a1740 10 client.6024 fetch_fsmap finished waiting for FSMap version 34 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8c003a1740 10 client.6024 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.352+0000 7f8c003a1740 20 client.6024 populate_metadata read hostname 'smithi097' 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.353+0000 7f8c003a1740 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200299c20 0x557200210cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.353+0000 7f8c003a1740 4 client.6024 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.353+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 --> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557200156e90 con 0x557200299c20 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.353+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200299c20 0x557200210cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.353+0000 7f8bf6a00640 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200299c20 0x557200210cc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20df26aa314ad028 server_cookie=f31ecf908b22bcc9 in_seq=0 out_seq=0 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.353+0000 7f8bdb400640 10 client.6024 ms_handle_connect on v2:172.21.15.97:6835/24115969 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8bdb400640 1 -- 172.21.15.97:0/4200334000 <== mds.0 v2:172.21.15.97:6835/24115969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x557200156e90 con 0x557200299c20 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8bdb400640 10 client.6024 handle_command_reply: tid=0 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 client.6024 shutdown 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 msgr2=0x5572001e9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x5572001e9590 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x557200212590 tx=0x7f8be0001020 comp rx=0 tx=0).stop 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 shutdown_connections 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:6835/24115969,v1:172.21.15.97:6837/24115969] conn(0x557200299c20 0x557200210cc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557200217410 0x5572001e9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557200212b90 0x5572001e9590 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 --2- 172.21.15.97:0/4200334000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5572002127b0 0x557200218420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 >> 172.21.15.97:0/4200334000 conn(0x5572001e5c90 msgr2=0x557200216850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8bdb400640 0 client.6024 ms_handle_reset on v2:172.21.15.97:6835/24115969 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 shutdown_connections 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 1 -- 172.21.15.97:0/4200334000 wait complete. 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.354+0000 7f8c003a1740 20 client.6024 trim_cache size 0 max 16384 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 5865, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 7.0824290870000004, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 34, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 58, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 58, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 13.575695018999999 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bd406b4d0 msgr2=0x7f8bd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bd406b4d0 0x7f8bd406d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8bf815f410 tx=0x7f8be0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 msgr2=0x7f8bf80af560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 0x7f8bf80af560 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000b9a0 tx=0x7f8bf000be60 comp rx=0 tx=0).stop 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 shutdown_connections 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bd406b4d0 0x7f8bd406d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf80af0b0 0x7f8bf80af560 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bf8159b90 0x7f8bf815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 --2- 172.21.15.97:0/3823818388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bf806afc0 0x7f8bf814f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 >> 172.21.15.97:0/3823818388 conn(0x7f8bf80d0df0 msgr2=0x7f8bf8074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 shutdown_connections 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.355+0000 7f8bfd600640 1 -- 172.21.15.97:0/3823818388 wait complete. 2024-03-06T07:30:27.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:27.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.564+0000 7f3ac9400640 1 Processor -- start 2024-03-06T07:30:27.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.564+0000 7f3ac9400640 1 -- start start 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.565+0000 7f3ac9400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac40d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.565+0000 7f3ac9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3ac40d2a70 0x7f3ac40b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.565+0000 7f3ac9400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac4161960 con 0x7f3ac40d1e90 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.565+0000 7f3ac9400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac40cf6f0 con 0x7f3ac40d2a70 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.565+0000 7f3ac9400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40cf9d0 con 0x7f3ac4161580 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3ac40d2a70 0x7f3ac40b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3ac40d2a70 0x7f3ac40b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52610/0 (socket says 172.21.15.97:52610) 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 -- 172.21.15.97:0/3827876916 learned_addr learned my addr 172.21.15.97:0/3827876916 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/3827876916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac40d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 -- 172.21.15.97:0/3827876916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 msgr2=0x7f3ac40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 --2- 172.21.15.97:0/3827876916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac40d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 -- 172.21.15.97:0/3827876916 >> v1:172.21.15.135:6789/0 conn(0x7f3ac40d1e90 legacy=0x7f3ac40d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.566+0000 7f3ac8a00640 1 -- 172.21.15.97:0/3827876916 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40cfcb0 con 0x7f3ac40d2a70 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.567+0000 7f3ac8a00640 1 --2- 172.21.15.97:0/3827876916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d2a70 0x7f3ac40b3ec0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009e50 tx=0x7f3abc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ae2b0e10625af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac2a00640 1 -- 172.21.15.97:0/3827876916 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc02ca70 con 0x7f3ac40d2a70 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac2a00640 1 -- 172.21.15.97:0/3827876916 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc02cc10 con 0x7f3ac40d2a70 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac2a00640 1 -- 172.21.15.97:0/3827876916 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc0343f0 con 0x7f3ac40d2a70 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 -- 172.21.15.97:0/3827876916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d2a70 msgr2=0x7f3ac40b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 --2- 172.21.15.97:0/3827876916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d2a70 0x7f3ac40b3ec0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009e50 tx=0x7f3abc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 -- 172.21.15.97:0/3827876916 shutdown_connections 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 --2- 172.21.15.97:0/3827876916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d2a70 0x7f3ac40b3ec0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 --2- 172.21.15.97:0/3827876916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac40d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 -- 172.21.15.97:0/3827876916 >> 172.21.15.97:0/3827876916 conn(0x7f3ac40d3d80 msgr2=0x7f3ac40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 -- 172.21.15.97:0/3827876916 shutdown_connections 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.568+0000 7f3ac9400640 1 -- 172.21.15.97:0/3827876916 wait complete. 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.570+0000 7f3ac9400640 1 Processor -- start 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.570+0000 7f3ac9400640 1 -- start start 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 0x7f3ac40b9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac9400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40d2a70 0x7f3ac415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac9400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac41f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac9400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40cf3a0 con 0x7f3ac40d1e90 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac9400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40b39a0 con 0x7f3ac40d2a70 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac9400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40b4400 con 0x7f3ac4161580 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 0x7f3ac40b9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 0x7f3ac40b9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52626/0 (socket says 172.21.15.97:52626) 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.571+0000 7f3ac3e00640 1 -- 172.21.15.97:0/2368739553 learned_addr learned my addr 172.21.15.97:0/2368739553 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac8a00640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac41f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40d2a70 0x7f3ac415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3e00640 1 -- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 msgr2=0x7f3ac41f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac41f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3e00640 1 -- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40d2a70 msgr2=0x7f3ac415b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40d2a70 0x7f3ac415b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3e00640 1 -- 172.21.15.97:0/2368739553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac415ba60 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 0x7f3ac40b9830 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400d5f0 tx=0x7f3ab400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.572+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40125f0 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.573+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab4019070 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.573+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40129d0 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.573+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac415bd40 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.573+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac415c300 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.574+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac404f330 con 0x7f3ac40d1e90 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.575+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ab4012b70 con 0x7f3ac40d1e90 2024-03-06T07:30:27.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.576+0000 7f3abbe00640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b4d0 0x7f3aa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.576+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ab408a5e0 con 0x7f3ac40d1e90 2024-03-06T07:30:27.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.576+0000 7f3ac3400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b4d0 0x7f3aa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.577+0000 7f3ac3400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b4d0 0x7f3aa406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4160410 tx=0x7f3ab0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.581+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ab4059450 con 0x7f3ac40d1e90 2024-03-06T07:30:27.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.839+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3ac40d3b60 con 0x7f3ac40d1e90 2024-03-06T07:30:27.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.839+0000 7f3abbe00640 1 -- 172.21.15.97:0/2368739553 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3ab4058d50 con 0x7f3ac40d1e90 2024-03-06T07:30:27.842 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:27.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b4d0 msgr2=0x7f3aa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b4d0 0x7f3aa406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4160410 tx=0x7f3ab0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 msgr2=0x7f3ac40b9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 0x7f3ac40b9830 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400d5f0 tx=0x7f3ab400dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 shutdown_connections 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b4d0 0x7f3aa406d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac4161580 0x7f3ac41f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40d2a70 0x7f3ac415b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 --2- 172.21.15.97:0/2368739553 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40d1e90 0x7f3ac40b9830 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 >> 172.21.15.97:0/2368739553 conn(0x7f3ac40d3d80 msgr2=0x7f3ac40d0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 shutdown_connections 2024-03-06T07:30:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:27.842+0000 7f3ac9400640 1 -- 172.21.15.97:0/2368739553 wait complete. 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:27.864 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:27.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:27.865 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:27.921 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42447sec preferred_lft 42447sec 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:30:27.922 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:30:27.923 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:27.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:27.924 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:28.012 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:30:28.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.016 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:30:28.061 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:30:28.061 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:28.061 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:28.061 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:28.061 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:28.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.062 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:30:28.062 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:30:28.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:28.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:28.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:28.063 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.291 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:30:28.339 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:28.339 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:28.340 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:28.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:30:28.420 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:28.420 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:28.468 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:28.468 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:28.468 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:28.562 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:28.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:28.695 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:28.695 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:28.731 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:28.731 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:28.731 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:30:28.731 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:30:28.731 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:30:28.735 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:28.735 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:28.735 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:28.735 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:28.735 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:28.738 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:28.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:28.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.821 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:28.892 INFO:teuthology.orchestra.run.smithi097.stdout:client5038 2024-03-06T07:30:28.892 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:28.892 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:28.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:30:29.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.199+0000 7fdaede00640 1 Processor -- start 2024-03-06T07:30:29.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.200+0000 7fdaede00640 1 -- start start 2024-03-06T07:30:29.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.201+0000 7fdaede00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdae815fc10 0x7fdae8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.201+0000 7fdaede00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80b46c0 con 0x7fdae815fff0 2024-03-06T07:30:29.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.201+0000 7fdaede00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80b49a0 con 0x7fdae80b3f70 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.201+0000 7fdaede00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae8072c90 con 0x7fdae815fc10 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdae815fc10 0x7fdae8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae6000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdae815fff0 0x7fdae8160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46294/0 (socket says 172.21.15.97:46294) 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae6000640 1 -- 172.21.15.97:0/3453377158 learned_addr learned my addr 172.21.15.97:0/3453377158 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae6a00640 1 -- 172.21.15.97:0/3453377158 >> v1:172.21.15.135:6789/0 conn(0x7fdae80b3f70 legacy=0x7fdae80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae6a00640 1 -- 172.21.15.97:0/3453377158 >> v1:172.21.15.97:6789/0 conn(0x7fdae815fff0 legacy=0x7fdae8160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae5600640 1 -- 172.21.15.97:0/3453377158 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568406397 0 0) 0x7fdae80b46c0 con 0x7fdae815fff0 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.202+0000 7fdae6a00640 1 -- 172.21.15.97:0/3453377158 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae809dd80 con 0x7fdae815fc10 2024-03-06T07:30:29.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.203+0000 7fdae6a00640 1 --2- 172.21.15.97:0/3453377158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae8070ff0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdad80097a0 tx=0x7fdad802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23525cc15b80b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.203+0000 7fdae5600640 1 -- 172.21.15.97:0/3453377158 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8035020 con 0x7fdae815fc10 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.203+0000 7fdae5600640 1 -- 172.21.15.97:0/3453377158 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad8031830 con 0x7fdae815fc10 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.203+0000 7fdae5600640 1 -- 172.21.15.97:0/3453377158 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8031c00 con 0x7fdae815fc10 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 -- 172.21.15.97:0/3453377158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 msgr2=0x7fdae8070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 --2- 172.21.15.97:0/3453377158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae8070ff0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdad80097a0 tx=0x7fdad802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 -- 172.21.15.97:0/3453377158 shutdown_connections 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 --2- 172.21.15.97:0/3453377158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae8070ff0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 -- 172.21.15.97:0/3453377158 >> 172.21.15.97:0/3453377158 conn(0x7fdae80b1940 msgr2=0x7fdae80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 -- 172.21.15.97:0/3453377158 shutdown_connections 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.204+0000 7fdaede00640 1 -- 172.21.15.97:0/3453377158 wait complete. 2024-03-06T07:30:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.208+0000 7fdaede00640 1 Processor -- start 2024-03-06T07:30:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.208+0000 7fdaede00640 1 -- start start 2024-03-06T07:30:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.210+0000 7fdaede00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdae80b3f70 0x7fdae815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.210+0000 7fdaede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae80c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.210+0000 7fdaede00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae815fff0 0x7fdae8151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.210+0000 7fdaede00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae815c460 con 0x7fdae815fc10 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.210+0000 7fdaede00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae815c2e0 con 0x7fdae815fff0 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.210+0000 7fdaede00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80cebb0 con 0x7fdae80b3f70 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae80c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae80c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52644/0 (socket says 172.21.15.97:52644) 2024-03-06T07:30:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 -- 172.21.15.97:0/1187738110 learned_addr learned my addr 172.21.15.97:0/1187738110 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 -- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdae80b3f70 msgr2=0x7fdae815ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6a00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdae80b3f70 0x7fdae815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae7400640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae815fff0 0x7fdae8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdae80b3f70 0x7fdae815ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 -- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae815fff0 msgr2=0x7fdae8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae815fff0 0x7fdae8151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 -- 172.21.15.97:0/1187738110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae815f360 con 0x7fdae815fc10 2024-03-06T07:30:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.211+0000 7fdae6000640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae80c74d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c430 tx=0x7fdad400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.212+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad40074e0 con 0x7fdae815fc10 2024-03-06T07:30:29.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.212+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad40047e0 con 0x7fdae815fc10 2024-03-06T07:30:29.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.212+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4004bd0 con 0x7fdae815fc10 2024-03-06T07:30:29.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.212+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae80c4dc0 con 0x7fdae815fc10 2024-03-06T07:30:29.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.212+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae80c5380 con 0x7fdae815fc10 2024-03-06T07:30:29.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.213+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae804f330 con 0x7fdae815fc10 2024-03-06T07:30:29.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.214+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdad4011070 con 0x7fdae815fc10 2024-03-06T07:30:29.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.215+0000 7fdadf400640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdac806b4d0 0x7fdac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.215+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdad4089f60 con 0x7fdae815fc10 2024-03-06T07:30:29.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.215+0000 7fdae6a00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdac806b4d0 0x7fdac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.216+0000 7fdae6a00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdac806b4d0 0x7fdac806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdad8036000 tx=0x7fdad8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.219+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdad4058d40 con 0x7fdae815fc10 2024-03-06T07:30:29.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.419+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fdae806ef30 con 0x7fdae815fc10 2024-03-06T07:30:29.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.426+0000 7fdadf400640 1 -- 172.21.15.97:0/1187738110 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdad4058640 con 0x7fdae815fc10 2024-03-06T07:30:29.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdac806b4d0 msgr2=0x7fdac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdac806b4d0 0x7fdac806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdad8036000 tx=0x7fdad8008030 comp rx=0 tx=0).stop 2024-03-06T07:30:29.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 msgr2=0x7fdae80c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae80c74d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c430 tx=0x7fdad400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 shutdown_connections 2024-03-06T07:30:29.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdac806b4d0 0x7fdac806d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae815fff0 0x7fdae8151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdae815fc10 0x7fdae80c74d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 --2- 172.21.15.97:0/1187738110 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdae80b3f70 0x7fdae815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 >> 172.21.15.97:0/1187738110 conn(0x7fdae80b1940 msgr2=0x7fdae809ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 shutdown_connections 2024-03-06T07:30:29.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.429+0000 7fdaede00640 1 -- 172.21.15.97:0/1187738110 wait complete. 2024-03-06T07:30:29.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:30:29.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.657+0000 7f6451400640 1 Processor -- start 2024-03-06T07:30:29.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.658+0000 7f6451400640 1 -- start start 2024-03-06T07:30:29.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.659+0000 7f6451400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f644c15cb90 0x7f644c06d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.659+0000 7f6451400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c163800 0x7f644c0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.659+0000 7f6451400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c15cf70 con 0x7f644c06e060 2024-03-06T07:30:29.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.659+0000 7f6451400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c0b3890 con 0x7f644c15cb90 2024-03-06T07:30:29.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.659+0000 7f6451400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c06c990 con 0x7f644c163800 2024-03-06T07:30:29.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.659+0000 7f644a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f644c15cb90 0x7f644c06d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f644c15cb90 0x7f644c06d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52660/0 (socket says 172.21.15.97:52660) 2024-03-06T07:30:29.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644a800640 1 -- 172.21.15.97:0/4283971988 learned_addr learned my addr 172.21.15.97:0/4283971988 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:29.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644b200640 1 --1- 172.21.15.97:0/4283971988 >> v1:172.21.15.97:6789/0 conn(0x7f644c06e060 0x7f644c0cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46310/0 (socket says 172.21.15.97:46310) 2024-03-06T07:30:29.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f6441e00640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c163800 0x7f644c0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644a800640 1 -- 172.21.15.97:0/4283971988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c163800 msgr2=0x7f644c0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644a800640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c163800 0x7f644c0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644a800640 1 -- 172.21.15.97:0/4283971988 >> v1:172.21.15.97:6789/0 conn(0x7f644c06e060 legacy=0x7f644c0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f644a800640 1 -- 172.21.15.97:0/4283971988 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0b35b0 con 0x7f644c15cb90 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.660+0000 7f6441e00640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c163800 0x7f644c0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.661+0000 7f644a800640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c15cb90 0x7f644c06d6c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f643c009870 tx=0x7f643c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98aee67e2283c43a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.661+0000 7f6449e00640 1 -- 172.21.15.97:0/4283971988 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c034070 con 0x7f644c15cb90 2024-03-06T07:30:29.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6449e00640 1 -- 172.21.15.97:0/4283971988 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c030840 con 0x7f644c15cb90 2024-03-06T07:30:29.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6449e00640 1 -- 172.21.15.97:0/4283971988 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c030c10 con 0x7f644c15cb90 2024-03-06T07:30:29.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 -- 172.21.15.97:0/4283971988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c15cb90 msgr2=0x7f644c06d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c15cb90 0x7f644c06d6c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f643c009870 tx=0x7f643c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:29.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 -- 172.21.15.97:0/4283971988 shutdown_connections 2024-03-06T07:30:29.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c163800 0x7f644c0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 --2- 172.21.15.97:0/4283971988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c15cb90 0x7f644c06d6c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 -- 172.21.15.97:0/4283971988 >> 172.21.15.97:0/4283971988 conn(0x7f644c0b1940 msgr2=0x7f644c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 -- 172.21.15.97:0/4283971988 shutdown_connections 2024-03-06T07:30:29.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.662+0000 7f6451400640 1 -- 172.21.15.97:0/4283971988 wait complete. 2024-03-06T07:30:29.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.664+0000 7f6451400640 1 Processor -- start 2024-03-06T07:30:29.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.664+0000 7f6451400640 1 -- start start 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f6451400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f644c06e060 0x7f644c071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f6451400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c15cb90 0x7f644c160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f6451400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 0x7f644c14d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f6451400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c06cec0 con 0x7f644c163800 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f6451400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c06d230 con 0x7f644c06e060 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f6451400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c06dc00 con 0x7f644c15cb90 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f644b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 0x7f644c14d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f644b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 0x7f644c14d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52676/0 (socket says 172.21.15.97:52676) 2024-03-06T07:30:29.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.665+0000 7f644b200640 1 -- 172.21.15.97:0/1006113040 learned_addr learned my addr 172.21.15.97:0/1006113040 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644a800640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f644c06e060 0x7f644c071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f6441e00640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c15cb90 0x7f644c160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644b200640 1 -- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c15cb90 msgr2=0x7f644c160170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644b200640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c15cb90 0x7f644c160170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644b200640 1 -- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f644c06e060 msgr2=0x7f644c071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644b200640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f644c06e060 0x7f644c071ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644b200640 1 -- 172.21.15.97:0/1006113040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c09b640 con 0x7f644c163800 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f6441e00640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c15cb90 0x7f644c160170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f644b200640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 0x7f644c14d8c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f644400c780 tx=0x7f644400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444017070 con 0x7f644c163800 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644400a640 con 0x7f644c163800 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c09b920 con 0x7f644c163800 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.666+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644400aa30 con 0x7f644c163800 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.667+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c072410 con 0x7f644c163800 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.668+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c04f330 con 0x7f644c163800 2024-03-06T07:30:29.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.669+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f644401a070 con 0x7f644c163800 2024-03-06T07:30:29.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.669+0000 7f6443400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.670+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64440896c0 con 0x7f644c163800 2024-03-06T07:30:29.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.670+0000 7f644a800640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.671+0000 7f644a800640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f642806b4d0 0x7f642806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f643c036000 tx=0x7f643c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.674+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6444057f60 con 0x7f644c163800 2024-03-06T07:30:29.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.933+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f644c0b2530 con 0x7f644c163800 2024-03-06T07:30:29.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.939+0000 7f6443400640 1 -- 172.21.15.97:0/1006113040 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f644405c000 con 0x7f644c163800 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f642806b4d0 msgr2=0x7f642806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f642806b4d0 0x7f642806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f643c036000 tx=0x7f643c03b040 comp rx=0 tx=0).stop 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 msgr2=0x7f644c14d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 0x7f644c14d8c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f644400c780 tx=0x7f644400cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 shutdown_connections 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f644c163800 0x7f644c14d8c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f644c15cb90 0x7f644c160170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 --2- 172.21.15.97:0/1006113040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f644c06e060 0x7f644c071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 >> 172.21.15.97:0/1006113040 conn(0x7f644c0b1940 msgr2=0x7f644c0afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 shutdown_connections 2024-03-06T07:30:29.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:29.942+0000 7f6451400640 1 -- 172.21.15.97:0/1006113040 wait complete. 2024-03-06T07:30:37.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:37.433 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:39.962 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:39.962 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> import uuid 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> n = 50 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = False 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> if 0 > 0: 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = True 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> dpath = os.path.dirname(path) 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:40.017 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> try: 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(n): 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> f.write(f"{i}") 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> f.flush() 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(f.fileno()) 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> os.unlink(fpath) 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:30:40.018 DEBUG:teuthology.orchestra.run.smithi097:> if create_hard_links: 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> for j in range(0): 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> finally: 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> os.close(dirfd) 2024-03-06T07:30:40.019 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:30:40.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:40.322 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:40.322 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:40.411 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:40.412 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:40.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:40.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.749+0000 7f65ed200640 1 Processor -- start 2024-03-06T07:30:40.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.749+0000 7f65ed200640 1 -- start start 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65ed200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f65e80b2a60 0x7f65e80d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65ed200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f65e80d18e0 0x7f65e80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65ed200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e815f030 con 0x7f65e80d0600 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65ed200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e80cf090 con 0x7f65e80b2a60 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65ed200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e815f310 con 0x7f65e80d18e0 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f65e80b2a60 0x7f65e80d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f65e80b2a60 0x7f65e80d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47268/0 (socket says 172.21.15.97:47268) 2024-03-06T07:30:40.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 -- 172.21.15.97:0/4254088731 learned_addr learned my addr 172.21.15.97:0/4254088731 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:40.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65e7e00640 1 --2- 172.21.15.97:0/4254088731 >> v2:172.21.15.135:3300/0 conn(0x7f65e80d18e0 0x7f65e80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 -- 172.21.15.97:0/4254088731 >> v2:172.21.15.135:3300/0 conn(0x7f65e80d18e0 msgr2=0x7f65e80d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 --2- 172.21.15.97:0/4254088731 >> v2:172.21.15.135:3300/0 conn(0x7f65e80d18e0 0x7f65e80d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 -- 172.21.15.97:0/4254088731 >> v1:172.21.15.135:6789/0 conn(0x7f65e80d0600 legacy=0x7f65e80d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65dfe00640 1 -- 172.21.15.97:0/4254088731 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e80cea40 con 0x7f65e80b2a60 2024-03-06T07:30:40.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.751+0000 7f65e7e00640 1 --2- 172.21.15.97:0/4254088731 >> v2:172.21.15.135:3300/0 conn(0x7f65e80d18e0 0x7f65e80d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:40.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.753+0000 7f65dfe00640 1 --2- 172.21.15.97:0/4254088731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e80d4830 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f65e808dc30 tx=0x7f65d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb8289b4804a7d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.753+0000 7f65e6a00640 1 -- 172.21.15.97:0/4254088731 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0035020 con 0x7f65e80b2a60 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.753+0000 7f65e6a00640 1 -- 172.21.15.97:0/4254088731 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0031870 con 0x7f65e80b2a60 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.753+0000 7f65e6a00640 1 -- 172.21.15.97:0/4254088731 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0031c40 con 0x7f65e80b2a60 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.753+0000 7f65ed200640 1 -- 172.21.15.97:0/4254088731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 msgr2=0x7f65e80d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.753+0000 7f65ed200640 1 --2- 172.21.15.97:0/4254088731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e80d4830 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f65e808dc30 tx=0x7f65d002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.754+0000 7f65ed200640 1 -- 172.21.15.97:0/4254088731 shutdown_connections 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.754+0000 7f65ed200640 1 --2- 172.21.15.97:0/4254088731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e80d4830 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.754+0000 7f65ed200640 1 --2- 172.21.15.97:0/4254088731 >> v2:172.21.15.135:3300/0 conn(0x7f65e80d18e0 0x7f65e80d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.754+0000 7f65ed200640 1 -- 172.21.15.97:0/4254088731 >> 172.21.15.97:0/4254088731 conn(0x7f65e80d3d10 msgr2=0x7f65e80d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.754+0000 7f65ed200640 1 -- 172.21.15.97:0/4254088731 shutdown_connections 2024-03-06T07:30:40.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.754+0000 7f65ed200640 1 -- 172.21.15.97:0/4254088731 wait complete. 2024-03-06T07:30:40.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.755+0000 7f65ed200640 1 Processor -- start 2024-03-06T07:30:40.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 -- start start 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e814e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65e80d0600 0x7f65e814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65e80d18e0 0x7f65e80b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e815ecf0 con 0x7f65e80b2a60 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b3250 con 0x7f65e80d0600 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.756+0000 7f65ed200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80cf520 con 0x7f65e80d18e0 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e814e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e814e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47280/0 (socket says 172.21.15.97:47280) 2024-03-06T07:30:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 -- 172.21.15.97:0/1931522405 learned_addr learned my addr 172.21.15.97:0/1931522405 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:40.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 -- 172.21.15.97:0/1931522405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65e80d18e0 msgr2=0x7f65e80b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65e80d18e0 0x7f65e80b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 -- 172.21.15.97:0/1931522405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65e80d0600 msgr2=0x7f65e814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65e80d0600 0x7f65e814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 -- 172.21.15.97:0/1931522405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e814b910 con 0x7f65e80b2a60 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.757+0000 7f65e7400640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e814e800 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f65d800c410 tx=0x7f65d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.758+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8017070 con 0x7f65e80b2a60 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.758+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d80035f0 con 0x7f65e80b2a60 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.758+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8003a20 con 0x7f65e80b2a60 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.758+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e80b4960 con 0x7f65e80b2a60 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.758+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e80b51d0 con 0x7f65e80b2a60 2024-03-06T07:30:40.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.759+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e80b4c70 con 0x7f65e80b2a60 2024-03-06T07:30:40.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.760+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65d8003bc0 con 0x7f65e80b2a60 2024-03-06T07:30:40.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.761+0000 7f65df400640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65c406b4d0 0x7f65c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.761+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65d8089530 con 0x7f65e80b2a60 2024-03-06T07:30:40.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.761+0000 7f65dfe00640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65c406b4d0 0x7f65c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.762+0000 7f65dfe00640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65c406b4d0 0x7f65c406d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f65d0036000 tx=0x7f65d0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:40.766+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65e80b4c70 con 0x7f65e80b2a60 2024-03-06T07:30:41.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.029+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65e804f330 con 0x7f65e80b2a60 2024-03-06T07:30:41.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.032+0000 7f65df400640 1 -- 172.21.15.97:0/1931522405 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f65d80583b0 con 0x7f65e80b2a60 2024-03-06T07:30:41.034 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:41.034 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":58,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:21.268327+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65c406b4d0 msgr2=0x7f65c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65c406b4d0 0x7f65c406d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f65d0036000 tx=0x7f65d0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 msgr2=0x7f65e814e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e814e800 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f65d800c410 tx=0x7f65d800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 shutdown_connections 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65c406b4d0 0x7f65c406d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65e80d18e0 0x7f65e80b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65e80d0600 0x7f65e814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 --2- 172.21.15.97:0/1931522405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65e80b2a60 0x7f65e814e800 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 >> 172.21.15.97:0/1931522405 conn(0x7f65e80d3d10 msgr2=0x7f65e80b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 shutdown_connections 2024-03-06T07:30:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.035+0000 7f65ed200640 1 -- 172.21.15.97:0/1931522405 wait complete. 2024-03-06T07:30:41.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:17.521656+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:18.729778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:41.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:41.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.236+0000 7f3e08000640 1 Processor -- start 2024-03-06T07:30:41.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.237+0000 7f3e08000640 1 -- start start 2024-03-06T07:30:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.238+0000 7f3e08000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3e001605b0 0x7f3e001671d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.238+0000 7f3e08000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e000ce1e0 con 0x7f3e000b43b0 2024-03-06T07:30:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.238+0000 7f3e08000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e000ce4c0 con 0x7f3e0015c330 2024-03-06T07:30:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.238+0000 7f3e08000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e000cedf0 con 0x7f3e001605b0 2024-03-06T07:30:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.238+0000 7f3e04c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3e000b43b0 0x7f3e0015fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33234/0 (socket says 172.21.15.97:33234) 2024-03-06T07:30:41.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.238+0000 7f3e04c00640 1 -- 172.21.15.97:0/3433407833 learned_addr learned my addr 172.21.15.97:0/3433407833 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:41.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3e06000640 1 --2- 172.21.15.97:0/3433407833 >> v2:172.21.15.135:3300/0 conn(0x7f3e001605b0 0x7f3e001671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14135973 0 0) 0x7f3e000ce1e0 con 0x7f3e000b43b0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de8003520 con 0x7f3e000b43b0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509121035 0 0) 0x7f3de8003520 con 0x7f3e000b43b0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 >> v2:172.21.15.135:3300/0 conn(0x7f3e001605b0 msgr2=0x7f3e001671d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 --2- 172.21.15.97:0/3433407833 >> v2:172.21.15.135:3300/0 conn(0x7f3e001605b0 0x7f3e001671d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 >> v1:172.21.15.135:6789/0 conn(0x7f3e0015c330 legacy=0x7f3e000d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e000ce7a0 con 0x7f3e000b43b0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3dec001670 con 0x7f3e000b43b0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 >> v1:172.21.15.97:6789/0 conn(0x7f3e000b43b0 legacy=0x7f3e0015fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.239+0000 7f3dffe00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80039b0 0x7f3de8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3dffe00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3dffe00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000cedf0 con 0x7f3de8004800 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de800c040 con 0x7f3de80039b0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de800c240 con 0x7f3de80051c0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e05600640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80039b0 0x7f3de8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e06000640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e04c00640 1 -- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de80051c0 msgr2=0x7f3de800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e04c00640 1 -- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80039b0 msgr2=0x7f3de8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80039b0 0x7f3de8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e04c00640 1 -- 172.21.15.97:0/3433407833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8003520 con 0x7f3de8004800 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e06000640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.240+0000 7f3e06000640 1 -- 172.21.15.97:0/3433407833 reap_dead start 2024-03-06T07:30:41.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.242+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c240 tx=0x7f3de8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6556d26538c43063 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.242+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec007d10 con 0x7f3de8004800 2024-03-06T07:30:41.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.242+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec030070 con 0x7f3de8004800 2024-03-06T07:30:41.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.242+0000 7f3dffe00640 1 -- 172.21.15.97:0/3433407833 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec005d70 con 0x7f3de8004800 2024-03-06T07:30:41.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 -- 172.21.15.97:0/3433407833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de8004800 msgr2=0x7f3de8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c240 tx=0x7f3de8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 -- 172.21.15.97:0/3433407833 shutdown_connections 2024-03-06T07:30:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 --2- 172.21.15.97:0/3433407833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 -- 172.21.15.97:0/3433407833 >> 172.21.15.97:0/3433407833 conn(0x7f3e000b3790 msgr2=0x7f3e000b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 -- 172.21.15.97:0/3433407833 shutdown_connections 2024-03-06T07:30:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.243+0000 7f3e08000640 1 -- 172.21.15.97:0/3433407833 wait complete. 2024-03-06T07:30:41.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.246+0000 7f3e08000640 1 Processor -- start 2024-03-06T07:30:41.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.246+0000 7f3e08000640 1 -- start start 2024-03-06T07:30:41.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.247+0000 7f3e08000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de8004800 0x7f3e00149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.247+0000 7f3e08000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 0x7f3e000b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.247+0000 7f3e08000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006f630 0x7f3e0006fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.247+0000 7f3e08000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000cf280 con 0x7f3e000b4c90 2024-03-06T07:30:41.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.247+0000 7f3e08000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0015fea0 con 0x7f3e0006f630 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.247+0000 7f3e08000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00155900 con 0x7f3de8004800 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 0x7f3e000b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 0x7f3e000b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47304/0 (socket says 172.21.15.97:47304) 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 -- 172.21.15.97:0/3418103739 learned_addr learned my addr 172.21.15.97:0/3418103739 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e05600640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de8004800 0x7f3e00149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e06000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006f630 0x7f3e0006fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 -- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de8004800 msgr2=0x7f3e00149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de8004800 0x7f3e00149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 -- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006f630 msgr2=0x7f3e0006fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006f630 0x7f3e0006fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 -- 172.21.15.97:0/3418103739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0014a490 con 0x7f3e000b4c90 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e06000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006f630 0x7f3e0006fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.248+0000 7f3e04c00640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 0x7f3e000b5110 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3dec001970 tx=0x7f3e000ced60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.249+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec001c50 con 0x7f3e000b4c90 2024-03-06T07:30:41.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.249+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec02cbe0 con 0x7f3e000b4c90 2024-03-06T07:30:41.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.249+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e00155620 con 0x7f3e000b4c90 2024-03-06T07:30:41.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.249+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec007070 con 0x7f3e000b4c90 2024-03-06T07:30:41.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.249+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0015e390 con 0x7f3e000b4c90 2024-03-06T07:30:41.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.250+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0004f330 con 0x7f3e000b4c90 2024-03-06T07:30:41.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.251+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3dec048020 con 0x7f3e000b4c90 2024-03-06T07:30:41.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.252+0000 7f3dfd600640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b500 0x7f3de006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.252+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dec0ab810 con 0x7f3e000b4c90 2024-03-06T07:30:41.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.252+0000 7f3e05600640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b500 0x7f3de006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.253+0000 7f3e05600640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b500 0x7f3de006d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3df4007fd0 tx=0x7f3df4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.257+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3dec07a600 con 0x7f3e000b4c90 2024-03-06T07:30:41.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.531+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e0004f330 con 0x7f3e000b4c90 2024-03-06T07:30:41.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.531+0000 7f3dfd600640 1 -- 172.21.15.97:0/3418103739 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3dec079f00 con 0x7f3e000b4c90 2024-03-06T07:30:41.533 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:30:41.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b500 msgr2=0x7f3de006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b500 0x7f3de006d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3df4007fd0 tx=0x7f3df4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 msgr2=0x7f3e000b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 0x7f3e000b5110 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3dec001970 tx=0x7f3e000ced60 comp rx=0 tx=0).stop 2024-03-06T07:30:41.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 shutdown_connections 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3de006b500 0x7f3de006d9c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e0006f630 0x7f3e0006fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e000b4c90 0x7f3e000b5110 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 --2- 172.21.15.97:0/3418103739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de8004800 0x7f3e00149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 >> 172.21.15.97:0/3418103739 conn(0x7f3e000b3790 msgr2=0x7f3e000b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 shutdown_connections 2024-03-06T07:30:41.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.534+0000 7f3e08000640 1 -- 172.21.15.97:0/3418103739 wait complete. 2024-03-06T07:30:41.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:41.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.759+0000 7f5d4ce00640 1 Processor -- start 2024-03-06T07:30:41.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.760+0000 7f5d4ce00640 1 -- start start 2024-03-06T07:30:41.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.761+0000 7f5d4ce00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5d480c8080 0x7f5d480d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.761+0000 7f5d4ce00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d480d13e0 con 0x7f5d4815e130 2024-03-06T07:30:41.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.761+0000 7f5d4ce00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d480cead0 con 0x7f5d480d1ea0 2024-03-06T07:30:41.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.761+0000 7f5d4ce00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d480d2840 con 0x7f5d480c8080 2024-03-06T07:30:41.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.761+0000 7f5d47400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5d4815e130 0x7f5d480cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33250/0 (socket says 172.21.15.97:33250) 2024-03-06T07:30:41.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.761+0000 7f5d47400640 1 -- 172.21.15.97:0/3665771764 learned_addr learned my addr 172.21.15.97:0/3665771764 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:41.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d47e00640 1 --2- 172.21.15.97:0/3665771764 >> v2:172.21.15.135:3300/0 conn(0x7f5d480c8080 0x7f5d480d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203345274 0 0) 0x7f5d480d13e0 con 0x7f5d4815e130 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c003520 con 0x7f5d4815e130 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782643426 0 0) 0x7f5d2c003520 con 0x7f5d4815e130 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 >> v2:172.21.15.135:3300/0 conn(0x7f5d480c8080 msgr2=0x7f5d480d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 --2- 172.21.15.97:0/3665771764 >> v2:172.21.15.135:3300/0 conn(0x7f5d480c8080 0x7f5d480d2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 >> v1:172.21.15.135:6789/0 conn(0x7f5d480d1ea0 legacy=0x7f5d480c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480cedb0 con 0x7f5d4815e130 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f5d38001670 con 0x7f5d4815e130 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 >> v1:172.21.15.97:6789/0 conn(0x7f5d4815e130 legacy=0x7f5d480cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2c0039b0 0x7f5d2c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2c004800 0x7f5d2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.762+0000 7f5d46a00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c0051c0 0x7f5d2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c00c010 con 0x7f5d2c004800 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c00c1c0 con 0x7f5d2c0039b0 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c00c3c0 con 0x7f5d2c0051c0 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d3fe00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2c004800 0x7f5d2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d47400640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2c0039b0 0x7f5d2c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d3fe00640 1 -- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c0051c0 msgr2=0x7f5d2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d3fe00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c0051c0 0x7f5d2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d3fe00640 1 -- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2c0039b0 msgr2=0x7f5d2c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d3fe00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2c0039b0 0x7f5d2c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d3fe00640 1 -- 172.21.15.97:0/3665771764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c003520 con 0x7f5d2c004800 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d47400640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2c0039b0 0x7f5d2c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.763+0000 7f5d47e00640 1 -- 172.21.15.97:0/3665771764 reap_dead start 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.765+0000 7f5d3fe00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2c004800 0x7f5d2c004c80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5d30003db0 tx=0x7f5d2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b836f243610f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.765+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30037070 con 0x7f5d2c004800 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.765+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3002be50 con 0x7f5d2c004800 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.765+0000 7f5d46a00640 1 -- 172.21.15.97:0/3665771764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30003980 con 0x7f5d2c004800 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3665771764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2c004800 msgr2=0x7f5d2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2c004800 0x7f5d2c004c80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5d30003db0 tx=0x7f5d2c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3665771764 shutdown_connections 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3665771764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2c004800 0x7f5d2c004c80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3665771764 >> 172.21.15.97:0/3665771764 conn(0x7f5d4809d070 msgr2=0x7f5d480d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3665771764 shutdown_connections 2024-03-06T07:30:41.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.766+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3665771764 wait complete. 2024-03-06T07:30:41.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.768+0000 7f5d4ce00640 1 Processor -- start 2024-03-06T07:30:41.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.768+0000 7f5d4ce00640 1 -- start start 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d4ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c004800 0x7f5d480b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d4ce00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d480caef0 0x7f5d48154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d4ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 0x7f5d480d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d4ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d480d2300 con 0x7f5d480d3570 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d4ce00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d480cf120 con 0x7f5d480caef0 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d4ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d480d2480 con 0x7f5d2c004800 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 0x7f5d480d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 0x7f5d480d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47312/0 (socket says 172.21.15.97:47312) 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d47e00640 1 -- 172.21.15.97:0/3310213011 learned_addr learned my addr 172.21.15.97:0/3310213011 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d47400640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c004800 0x7f5d480b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.769+0000 7f5d3fe00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d480caef0 0x7f5d48154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d47e00640 1 -- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c004800 msgr2=0x7f5d480b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d47e00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c004800 0x7f5d480b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d47e00640 1 -- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d480caef0 msgr2=0x7f5d48154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d47e00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d480caef0 0x7f5d48154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d47e00640 1 -- 172.21.15.97:0/3310213011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480cb6d0 con 0x7f5d480d3570 2024-03-06T07:30:41.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d47e00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 0x7f5d480d39f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5d4808d1a0 tx=0x7f5d4000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40018020 con 0x7f5d480d3570 2024-03-06T07:30:41.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4000bce0 con 0x7f5d480d3570 2024-03-06T07:30:41.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d480cb9b0 con 0x7f5d480d3570 2024-03-06T07:30:41.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40014640 con 0x7f5d480d3570 2024-03-06T07:30:41.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.770+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d480cc6d0 con 0x7f5d480d3570 2024-03-06T07:30:41.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.772+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d480cc8d0 con 0x7f5d480d3570 2024-03-06T07:30:41.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.773+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d40005d10 con 0x7f5d480d3570 2024-03-06T07:30:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.774+0000 7f5d3f400640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d2406b500 0x7f5d2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.774+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d4008a930 con 0x7f5d480d3570 2024-03-06T07:30:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.774+0000 7f5d47400640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d2406b500 0x7f5d2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.775+0000 7f5d47400640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d2406b500 0x7f5d2406d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5d380017e0 tx=0x7f5d38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.779+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d480cc8d0 con 0x7f5d480d3570 2024-03-06T07:30:41.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.974+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d4804f330 con 0x7f5d480d3570 2024-03-06T07:30:41.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.975+0000 7f5d3f400640 1 -- 172.21.15.97:0/3310213011 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d4005f070 con 0x7f5d480d3570 2024-03-06T07:30:41.977 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:30:41.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d2406b500 msgr2=0x7f5d2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d2406b500 0x7f5d2406d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5d380017e0 tx=0x7f5d38009040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 msgr2=0x7f5d480d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 0x7f5d480d39f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5d4808d1a0 tx=0x7f5d4000b480 comp rx=0 tx=0).stop 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 shutdown_connections 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d2406b500 0x7f5d2406d9c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d480d3570 0x7f5d480d39f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d480caef0 0x7f5d48154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 --2- 172.21.15.97:0/3310213011 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d2c004800 0x7f5d480b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.977+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 >> 172.21.15.97:0/3310213011 conn(0x7f5d4809d070 msgr2=0x7f5d48161b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.978+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 shutdown_connections 2024-03-06T07:30:41.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:41.978+0000 7f5d4ce00640 1 -- 172.21.15.97:0/3310213011 wait complete. 2024-03-06T07:30:41.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.215+0000 7f0f49000640 1 Processor -- start 2024-03-06T07:30:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.215+0000 7f0f49000640 1 -- start start 2024-03-06T07:30:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f49000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0f4406da40 0x7f0f440ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f49000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4415c1f0 con 0x7f0f440aef20 2024-03-06T07:30:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f49000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4406cac0 con 0x7f0f4415be10 2024-03-06T07:30:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f49000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f440cd200 con 0x7f0f4406da40 2024-03-06T07:30:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f43e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0f440aef20 0x7f0f44071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33254/0 (socket says 172.21.15.97:33254) 2024-03-06T07:30:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f43e00640 1 -- 172.21.15.97:0/3054080399 learned_addr learned my addr 172.21.15.97:0/3054080399 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:42.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.217+0000 7f0f42a00640 1 --2- 172.21.15.97:0/3054080399 >> v2:172.21.15.135:3300/0 conn(0x7f0f4406da40 0x7f0f440ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.218+0000 7f0f42000640 1 -- 172.21.15.97:0/3054080399 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883951263 0 0) 0x7f0f4415c1f0 con 0x7f0f440aef20 2024-03-06T07:30:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.218+0000 7f0f42000640 1 -- 172.21.15.97:0/3054080399 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f28003520 con 0x7f0f440aef20 2024-03-06T07:30:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.218+0000 7f0f42000640 1 -- 172.21.15.97:0/3054080399 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560220295 0 0) 0x7f0f28003520 con 0x7f0f440aef20 2024-03-06T07:30:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.218+0000 7f0f42a00640 1 -- 172.21.15.97:0/3054080399 >> v1:172.21.15.135:6789/0 conn(0x7f0f4415be10 legacy=0x7f0f4406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.218+0000 7f0f42a00640 1 -- 172.21.15.97:0/3054080399 >> v1:172.21.15.97:6789/0 conn(0x7f0f440aef20 legacy=0x7f0f44071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.218+0000 7f0f42a00640 1 -- 172.21.15.97:0/3054080399 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4406cda0 con 0x7f0f4406da40 2024-03-06T07:30:42.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.219+0000 7f0f42a00640 1 --2- 172.21.15.97:0/3054080399 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f4406da40 0x7f0f440ccc20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0f300097a0 tx=0x7f0f3002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a141743d1b8ef3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.220+0000 7f0f42000640 1 -- 172.21.15.97:0/3054080399 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30037070 con 0x7f0f4406da40 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.220+0000 7f0f42000640 1 -- 172.21.15.97:0/3054080399 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f30030510 con 0x7f0f4406da40 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.220+0000 7f0f42000640 1 -- 172.21.15.97:0/3054080399 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0019f0 con 0x7f0f4406da40 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.220+0000 7f0f49000640 1 -- 172.21.15.97:0/3054080399 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f4406da40 msgr2=0x7f0f440ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.220+0000 7f0f49000640 1 --2- 172.21.15.97:0/3054080399 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f4406da40 0x7f0f440ccc20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0f300097a0 tx=0x7f0f3002b790 comp rx=0 tx=0).stop 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.221+0000 7f0f49000640 1 -- 172.21.15.97:0/3054080399 shutdown_connections 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.221+0000 7f0f49000640 1 --2- 172.21.15.97:0/3054080399 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f4406da40 0x7f0f440ccc20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.221+0000 7f0f49000640 1 -- 172.21.15.97:0/3054080399 >> 172.21.15.97:0/3054080399 conn(0x7f0f4406afc0 msgr2=0x7f0f4406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.221+0000 7f0f49000640 1 -- 172.21.15.97:0/3054080399 shutdown_connections 2024-03-06T07:30:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.221+0000 7f0f49000640 1 -- 172.21.15.97:0/3054080399 wait complete. 2024-03-06T07:30:42.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.222+0000 7f0f49000640 1 Processor -- start 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.222+0000 7f0f49000640 1 -- start start 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.223+0000 7f0f49000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f4406da40 0x7f0f44147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.223+0000 7f0f49000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f440aef20 0x7f0f441596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.223+0000 7f0f49000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 0x7f0f440b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.223+0000 7f0f49000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440cda90 con 0x7f0f4415be10 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.223+0000 7f0f49000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4406d080 con 0x7f0f440aef20 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.223+0000 7f0f49000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440ce2b0 con 0x7f0f4406da40 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 0x7f0f440b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 0x7f0f440b39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47324/0 (socket says 172.21.15.97:47324) 2024-03-06T07:30:42.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 -- 172.21.15.97:0/3498497123 learned_addr learned my addr 172.21.15.97:0/3498497123 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:42.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f42a00640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f440aef20 0x7f0f441596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 -- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f4406da40 msgr2=0x7f0f44147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43400640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f4406da40 0x7f0f44147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f4406da40 0x7f0f44147230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 -- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f440aef20 msgr2=0x7f0f441596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f440aef20 0x7f0f441596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 -- 172.21.15.97:0/3498497123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f440b2970 con 0x7f0f4415be10 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f42a00640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f440aef20 0x7f0f441596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.224+0000 7f0f43e00640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 0x7f0f440b39f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c001890 tx=0x7f0f3c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.225+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c012400 con 0x7f0f4415be10 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.225+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c0125c0 con 0x7f0f4415be10 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.225+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0129b0 con 0x7f0f4415be10 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.225+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f440b2c30 con 0x7f0f4415be10 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.225+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f440b31f0 con 0x7f0f4415be10 2024-03-06T07:30:42.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.226+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4404f330 con 0x7f0f4415be10 2024-03-06T07:30:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.227+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f3c012b50 con 0x7f0f4415be10 2024-03-06T07:30:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.228+0000 7f0f3b400640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f2406b500 0x7f0f2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.228+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f3c08ab10 con 0x7f0f4415be10 2024-03-06T07:30:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.228+0000 7f0f43400640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f2406b500 0x7f0f2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.229+0000 7f0f43400640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f2406b500 0x7f0f2406d9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0f34000f60 tx=0x7f0f34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.233+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f3c0598f0 con 0x7f0f4415be10 2024-03-06T07:30:42.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.427+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f440cbc90 con 0x7f0f4415be10 2024-03-06T07:30:42.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.428+0000 7f0f3b400640 1 -- 172.21.15.97:0/3498497123 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f3c0591f0 con 0x7f0f4415be10 2024-03-06T07:30:42.430 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.431+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f2406b500 msgr2=0x7f0f2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.431+0000 7f0f49000640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f2406b500 0x7f0f2406d9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0f34000f60 tx=0x7f0f34009040 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.431+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 msgr2=0x7f0f440b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.431+0000 7f0f49000640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 0x7f0f440b39f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c001890 tx=0x7f0f3c00a430 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 shutdown_connections 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f2406b500 0x7f0f2406d9c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f4415be10 0x7f0f440b39f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f440aef20 0x7f0f441596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 --2- 172.21.15.97:0/3498497123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f4406da40 0x7f0f44147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 >> 172.21.15.97:0/3498497123 conn(0x7f0f4406afc0 msgr2=0x7f0f44071ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 shutdown_connections 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.432+0000 7f0f49000640 1 -- 172.21.15.97:0/3498497123 wait complete. 2024-03-06T07:30:42.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:42.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.653+0000 7faf2d400640 1 Processor -- start 2024-03-06T07:30:42.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.654+0000 7faf2d400640 1 -- start start 2024-03-06T07:30:42.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf2d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faf2806c0d0 0x7faf280d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf2d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2806d0a0 0x7faf280d0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf2d400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf280b4570 con 0x7faf2815e2c0 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf2d400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280b4320 con 0x7faf2806c0d0 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf2d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280d05d0 con 0x7faf2806d0a0 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf25e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faf2806c0d0 0x7faf280d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf25e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faf2806c0d0 0x7faf280d0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47326/0 (socket says 172.21.15.97:47326) 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.655+0000 7faf25e00640 1 -- 172.21.15.97:0/2931735832 learned_addr learned my addr 172.21.15.97:0/2931735832 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.656+0000 7faf27200640 1 --2- 172.21.15.97:0/2931735832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2806d0a0 0x7faf280d0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.656+0000 7faf25e00640 1 -- 172.21.15.97:0/2931735832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2806d0a0 msgr2=0x7faf280d0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.656+0000 7faf25e00640 1 --2- 172.21.15.97:0/2931735832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2806d0a0 0x7faf280d0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.656+0000 7faf25e00640 1 -- 172.21.15.97:0/2931735832 >> v1:172.21.15.97:6789/0 conn(0x7faf2815e2c0 legacy=0x7faf280cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.656+0000 7faf25e00640 1 -- 172.21.15.97:0/2931735832 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280d0260 con 0x7faf2806c0d0 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.656+0000 7faf25400640 1 -- 172.21.15.97:0/2931735832 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887471869 0 0) 0x7faf280b4570 con 0x7faf2815e2c0 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.658+0000 7faf25e00640 1 --2- 172.21.15.97:0/2931735832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806c0d0 0x7faf280d0a60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7faf14009870 tx=0x7faf1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bf9ead6b980742 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.658+0000 7faf25400640 1 -- 172.21.15.97:0/2931735832 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14034070 con 0x7faf2806c0d0 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.658+0000 7faf25400640 1 -- 172.21.15.97:0/2931735832 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf14030840 con 0x7faf2806c0d0 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.658+0000 7faf25400640 1 -- 172.21.15.97:0/2931735832 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14030c10 con 0x7faf2806c0d0 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 -- 172.21.15.97:0/2931735832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806c0d0 msgr2=0x7faf280d0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 --2- 172.21.15.97:0/2931735832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806c0d0 0x7faf280d0a60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7faf14009870 tx=0x7faf1402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 -- 172.21.15.97:0/2931735832 shutdown_connections 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 --2- 172.21.15.97:0/2931735832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2806d0a0 0x7faf280d0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 --2- 172.21.15.97:0/2931735832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806c0d0 0x7faf280d0a60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 -- 172.21.15.97:0/2931735832 >> 172.21.15.97:0/2931735832 conn(0x7faf280b3800 msgr2=0x7faf280b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 -- 172.21.15.97:0/2931735832 shutdown_connections 2024-03-06T07:30:42.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.659+0000 7faf2d400640 1 -- 172.21.15.97:0/2931735832 wait complete. 2024-03-06T07:30:42.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.661+0000 7faf2d400640 1 Processor -- start 2024-03-06T07:30:42.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.661+0000 7faf2d400640 1 -- start start 2024-03-06T07:30:42.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf2d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf2806c0d0 0x7faf280c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf2d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 0x7faf2815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf2d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2815e2c0 0x7faf28156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf2d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280cca30 con 0x7faf2806d0a0 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf2d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280b0e20 con 0x7faf2806c0d0 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf2d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280ccbb0 con 0x7faf2815e2c0 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 0x7faf2815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 0x7faf2815b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47332/0 (socket says 172.21.15.97:47332) 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 -- 172.21.15.97:0/801077083 learned_addr learned my addr 172.21.15.97:0/801077083 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf27200640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2815e2c0 0x7faf28156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 -- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2815e2c0 msgr2=0x7faf28156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf26800640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf2806c0d0 0x7faf280c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2815e2c0 0x7faf28156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 -- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf2806c0d0 msgr2=0x7faf280c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf2806c0d0 0x7faf280c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.663+0000 7faf25e00640 1 -- 172.21.15.97:0/801077083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280c9380 con 0x7faf2806d0a0 2024-03-06T07:30:42.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.664+0000 7faf25e00640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 0x7faf2815b780 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7faf14000c00 tx=0x7faf140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.664+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14034040 con 0x7faf2806d0a0 2024-03-06T07:30:42.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.664+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf14039d90 con 0x7faf2806d0a0 2024-03-06T07:30:42.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.664+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf280c9660 con 0x7faf2806d0a0 2024-03-06T07:30:42.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.664+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14043630 con 0x7faf2806d0a0 2024-03-06T07:30:42.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.664+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf280c9c20 con 0x7faf2806d0a0 2024-03-06T07:30:42.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.665+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2815dad0 con 0x7faf2806d0a0 2024-03-06T07:30:42.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.666+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf1404a060 con 0x7faf2806d0a0 2024-03-06T07:30:42.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.667+0000 7faf1ea00640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faf0806b500 0x7faf0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.667+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf140ac2d0 con 0x7faf2806d0a0 2024-03-06T07:30:42.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.667+0000 7faf26800640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faf0806b500 0x7faf0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.668+0000 7faf26800640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faf0806b500 0x7faf0806d9c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faf18000f60 tx=0x7faf18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.672+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf2815dad0 con 0x7faf2806d0a0 2024-03-06T07:30:42.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.879+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faf2804f330 con 0x7faf2806d0a0 2024-03-06T07:30:42.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.879+0000 7faf1ea00640 1 -- 172.21.15.97:0/801077083 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7faf1403e070 con 0x7faf2806d0a0 2024-03-06T07:30:42.881 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faf0806b500 msgr2=0x7faf0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faf0806b500 0x7faf0806d9c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faf18000f60 tx=0x7faf18009040 comp rx=0 tx=0).stop 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 msgr2=0x7faf2815b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 0x7faf2815b780 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7faf14000c00 tx=0x7faf140046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 shutdown_connections 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faf0806b500 0x7faf0806d9c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faf2815e2c0 0x7faf28156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf2806d0a0 0x7faf2815b780 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 --2- 172.21.15.97:0/801077083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf2806c0d0 0x7faf280c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.882+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 >> 172.21.15.97:0/801077083 conn(0x7faf280b3800 msgr2=0x7faf280ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.883+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 shutdown_connections 2024-03-06T07:30:42.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:42.883+0000 7faf2d400640 1 -- 172.21.15.97:0/801077083 wait complete. 2024-03-06T07:30:42.901 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:42.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:42.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:43.080 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:30:43.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:43.081 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:43.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:43.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.346+0000 7fec80c00640 1 Processor -- start 2024-03-06T07:30:43.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.346+0000 7fec80c00640 1 -- start start 2024-03-06T07:30:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec80c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec7c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec80c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7c161960 con 0x7fec7c0d2a70 2024-03-06T07:30:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec80c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7c0cf6f0 con 0x7fec7c0d1e90 2024-03-06T07:30:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec80c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0cf9d0 con 0x7fec7c161580 2024-03-06T07:30:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec7be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fec7c0d2a70 0x7fec7c0b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33270/0 (socket says 172.21.15.97:33270) 2024-03-06T07:30:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec7be00640 1 -- 172.21.15.97:0/1661069640 learned_addr learned my addr 172.21.15.97:0/1661069640 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:43.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec7b400640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec7c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.349+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287435342 0 0) 0x7fec7c161960 con 0x7fec7c0d2a70 2024-03-06T07:30:43.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec60003520 con 0x7fec7c0d2a70 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2812433321 0 0) 0x7fec60003520 con 0x7fec7c0d2a70 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 msgr2=0x7fec7c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec7c0d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 >> v1:172.21.15.135:6789/0 conn(0x7fec7c0d1e90 legacy=0x7fec7c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c0cfcb0 con 0x7fec7c0d2a70 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fec740019f0 con 0x7fec7c0d2a70 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 >> v1:172.21.15.97:6789/0 conn(0x7fec7c0d2a70 legacy=0x7fec7c0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec60004800 0x7fec60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec60005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0cf9d0 con 0x7fec600039e0 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6000baf0 con 0x7fec60004800 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6000bcf0 con 0x7fec7c161580 2024-03-06T07:30:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.350+0000 7fec7b400640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec73e00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec60004800 0x7fec60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7be00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec60005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7b400640 1 -- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 msgr2=0x7fec60005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7b400640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec60005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7b400640 1 -- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec60004800 msgr2=0x7fec60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7b400640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec60004800 0x7fec60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7b400640 1 -- 172.21.15.97:0/1661069640 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60003520 con 0x7fec600039e0 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec7be00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c161580 0x7fec60005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.351+0000 7fec73e00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec60004800 0x7fec60004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.352+0000 7fec7b400640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec60003e00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00ba30 tx=0x7fec60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c1b459474b5d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c036020 con 0x7fec600039e0 2024-03-06T07:30:43.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c032630 con 0x7fec600039e0 2024-03-06T07:30:43.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec7aa00640 1 -- 172.21.15.97:0/1661069640 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00b600 con 0x7fec600039e0 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec80c00640 1 -- 172.21.15.97:0/1661069640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 msgr2=0x7fec60003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec80c00640 1 --2- 172.21.15.97:0/1661069640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec60003e00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00ba30 tx=0x7fec60005830 comp rx=0 tx=0).stop 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec7be00640 1 -- 172.21.15.97:0/1661069640 reap_dead start 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec80c00640 1 -- 172.21.15.97:0/1661069640 shutdown_connections 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec80c00640 1 -- 172.21.15.97:0/1661069640 >> 172.21.15.97:0/1661069640 conn(0x7fec7c0d3d80 msgr2=0x7fec7c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec80c00640 1 -- 172.21.15.97:0/1661069640 shutdown_connections 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.353+0000 7fec80c00640 1 -- 172.21.15.97:0/1661069640 wait complete. 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.355+0000 7fec80c00640 1 Processor -- start 2024-03-06T07:30:43.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.355+0000 7fec80c00640 1 -- start start 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec80c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec7c151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec80c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c0b7bb0 0x7fec7c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec80c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec7c0cc460 0x7fec7c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec80c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0cf3a0 con 0x7fec600039e0 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec80c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0b39a0 con 0x7fec7c0cc460 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec80c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0d1950 con 0x7fec7c0b7bb0 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec7c151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec7c151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47344/0 (socket says 172.21.15.97:47344) 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.356+0000 7fec7b400640 1 -- 172.21.15.97:0/4266955356 learned_addr learned my addr 172.21.15.97:0/4266955356 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:43.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7b400640 1 -- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c0b7bb0 msgr2=0x7fec7c0c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7b400640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c0b7bb0 0x7fec7c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7b400640 1 -- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec7c0cc460 msgr2=0x7fec7c0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7be00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec7c0cc460 0x7fec7c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7b400640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec7c0cc460 0x7fec7c0cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7b400640 1 -- 172.21.15.97:0/4266955356 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c0c9dd0 con 0x7fec600039e0 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7be00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec7c0cc460 0x7fec7c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec7b400640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec7c151260 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00bb50 tx=0x7fec6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c041070 con 0x7fec600039e0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c030a40 con 0x7fec600039e0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7c1517a0 con 0x7fec600039e0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.357+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c036040 con 0x7fec600039e0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.358+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c0ca050 con 0x7fec600039e0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.359+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7c068240 con 0x7fec600039e0 2024-03-06T07:30:43.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.359+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec6c039020 con 0x7fec600039e0 2024-03-06T07:30:43.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.361+0000 7fec73400640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fec5806b500 0x7fec5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.361+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec6c0ac4d0 con 0x7fec600039e0 2024-03-06T07:30:43.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.361+0000 7fec73e00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fec5806b500 0x7fec5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.362+0000 7fec73e00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fec5806b500 0x7fec5806d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fec7c15dcd0 tx=0x7fec64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.365+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec7c068240 con 0x7fec600039e0 2024-03-06T07:30:43.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.601+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec7c04f330 con 0x7fec600039e0 2024-03-06T07:30:43.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.602+0000 7fec73400640 1 -- 172.21.15.97:0/4266955356 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fec7c0c9dd0 con 0x7fec600039e0 2024-03-06T07:30:43.605 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:43.605 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:43.605 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fec5806b500 msgr2=0x7fec5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fec5806b500 0x7fec5806d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fec7c15dcd0 tx=0x7fec64009040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 msgr2=0x7fec7c151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec7c151260 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00bb50 tx=0x7fec6c004730 comp rx=0 tx=0).stop 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 shutdown_connections 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fec5806b500 0x7fec5806d9c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec7c0cc460 0x7fec7c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fec7c0b7bb0 0x7fec7c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 --2- 172.21.15.97:0/4266955356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec600039e0 0x7fec7c151260 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 >> 172.21.15.97:0/4266955356 conn(0x7fec7c0d3d80 msgr2=0x7fec7c0b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 shutdown_connections 2024-03-06T07:30:43.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:43.605+0000 7fec80c00640 1 -- 172.21.15.97:0/4266955356 wait complete. 2024-03-06T07:30:43.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:30:43.836 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:30:43.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:44.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.030+0000 7f32d7e00640 1 Processor -- start 2024-03-06T07:30:44.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.030+0000 7f32d7e00640 1 -- start start 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32d7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32d8161580 0x7f32d80d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32d7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f32d80d2a70 0x7f32d80b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32d7e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d8161960 con 0x7f32d80d1e90 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32d7e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80cf6f0 con 0x7f32d8161580 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32d7e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80cf9d0 con 0x7f32d80d2a70 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32d6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32d8161580 0x7f32d80d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32ce000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f32d80d1e90 0x7f32d80d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50964/0 (socket says 172.21.15.97:50964) 2024-03-06T07:30:44.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.032+0000 7f32ce000640 1 -- 172.21.15.97:0/1231647412 learned_addr learned my addr 172.21.15.97:0/1231647412 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:44.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.033+0000 7f32d7400640 1 --2- 172.21.15.97:0/1231647412 >> v2:172.21.15.135:3300/0 conn(0x7f32d80d2a70 0x7f32d80b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.033+0000 7f32d6a00640 1 -- 172.21.15.97:0/1231647412 >> v2:172.21.15.135:3300/0 conn(0x7f32d80d2a70 msgr2=0x7f32d80b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.033+0000 7f32d6a00640 1 --2- 172.21.15.97:0/1231647412 >> v2:172.21.15.135:3300/0 conn(0x7f32d80d2a70 0x7f32d80b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.033+0000 7f32d6a00640 1 -- 172.21.15.97:0/1231647412 >> v1:172.21.15.97:6789/0 conn(0x7f32d80d1e90 legacy=0x7f32d80d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.033+0000 7f32d6a00640 1 -- 172.21.15.97:0/1231647412 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80cfcb0 con 0x7f32d8161580 2024-03-06T07:30:44.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.033+0000 7f32d6000640 1 -- 172.21.15.97:0/1231647412 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501659966 0 0) 0x7f32d8161960 con 0x7f32d80d1e90 2024-03-06T07:30:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.034+0000 7f32d6a00640 1 --2- 172.21.15.97:0/1231647412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d80d1950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f32d808d180 tx=0x7f32c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84026beb65ef8ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.034+0000 7f32d6000640 1 -- 172.21.15.97:0/1231647412 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8035020 con 0x7f32d8161580 2024-03-06T07:30:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.034+0000 7f32d6000640 1 -- 172.21.15.97:0/1231647412 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8031830 con 0x7f32d8161580 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.034+0000 7f32d6000640 1 -- 172.21.15.97:0/1231647412 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8031c00 con 0x7f32d8161580 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 -- 172.21.15.97:0/1231647412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 msgr2=0x7f32d80d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 --2- 172.21.15.97:0/1231647412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d80d1950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f32d808d180 tx=0x7f32c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 -- 172.21.15.97:0/1231647412 shutdown_connections 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 --2- 172.21.15.97:0/1231647412 >> v2:172.21.15.135:3300/0 conn(0x7f32d80d2a70 0x7f32d80b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 --2- 172.21.15.97:0/1231647412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d80d1950 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 -- 172.21.15.97:0/1231647412 >> 172.21.15.97:0/1231647412 conn(0x7f32d80d3d80 msgr2=0x7f32d80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 -- 172.21.15.97:0/1231647412 shutdown_connections 2024-03-06T07:30:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.035+0000 7f32d7e00640 1 -- 172.21.15.97:0/1231647412 wait complete. 2024-03-06T07:30:44.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.037+0000 7f32d7e00640 1 Processor -- start 2024-03-06T07:30:44.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.037+0000 7f32d7e00640 1 -- start start 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d80d1e90 0x7f32d80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d80d2a70 0x7f32d80b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80cf3a0 con 0x7f32d8161580 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b39a0 con 0x7f32d80d2a70 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b4400 con 0x7f32d80d1e90 2024-03-06T07:30:44.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d815fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33050/0 (socket says 172.21.15.97:33050) 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.038+0000 7f32d7400640 1 -- 172.21.15.97:0/136855254 learned_addr learned my addr 172.21.15.97:0/136855254 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d6a00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d80d1e90 0x7f32d80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32ce000640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d80d2a70 0x7f32d80b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d7400640 1 -- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d80d1e90 msgr2=0x7f32d80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d7400640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d80d1e90 0x7f32d80b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d7400640 1 -- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d80d2a70 msgr2=0x7f32d80b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d7400640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d80d2a70 0x7f32d80b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d7400640 1 -- 172.21.15.97:0/136855254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80b92b0 con 0x7f32d8161580 2024-03-06T07:30:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32d7400640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d815fb20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f32d0007e30 tx=0x7f32d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.039+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0014620 con 0x7f32d8161580 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.040+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d0019070 con 0x7f32d8161580 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.040+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0014a00 con 0x7f32d8161580 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.040+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d80b7a60 con 0x7f32d8161580 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.040+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d80b74a0 con 0x7f32d8161580 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.041+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d804f330 con 0x7f32d8161580 2024-03-06T07:30:44.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.042+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32d0014ba0 con 0x7f32d8161580 2024-03-06T07:30:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.043+0000 7f32cf400640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32b406b4d0 0x7f32b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.043+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32d008aae0 con 0x7f32d8161580 2024-03-06T07:30:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.043+0000 7f32d6a00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32b406b4d0 0x7f32b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.044+0000 7f32d6a00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32b406b4d0 0x7f32b406d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f32c8036000 tx=0x7f32c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.047+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32d0059960 con 0x7f32d8161580 2024-03-06T07:30:44.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.282+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d80d3b60 con 0x7f32d8161580 2024-03-06T07:30:44.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.283+0000 7f32cf400640 1 -- 172.21.15.97:0/136855254 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f32d0059260 con 0x7f32d8161580 2024-03-06T07:30:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.286+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32b406b4d0 msgr2=0x7f32b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.286+0000 7f32d7e00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32b406b4d0 0x7f32b406d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f32c8036000 tx=0x7f32c8008030 comp rx=0 tx=0).stop 2024-03-06T07:30:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.286+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 msgr2=0x7f32d815fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.286+0000 7f32d7e00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d815fb20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f32d0007e30 tx=0x7f32d000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 shutdown_connections 2024-03-06T07:30:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f32b406b4d0 0x7f32b406d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32d8161580 0x7f32d815fb20 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32d80d2a70 0x7f32d80b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 --2- 172.21.15.97:0/136855254 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32d80d1e90 0x7f32d80b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 >> 172.21.15.97:0/136855254 conn(0x7f32d80d3d80 msgr2=0x7f32d806cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 shutdown_connections 2024-03-06T07:30:44.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.287+0000 7f32d7e00640 1 -- 172.21.15.97:0/136855254 wait complete. 2024-03-06T07:30:44.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:30:44.550 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:30:44.551 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:44.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:44.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.744+0000 7f3621400640 1 Processor -- start 2024-03-06T07:30:44.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.744+0000 7f3621400640 1 -- start start 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f3621400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f361c15e160 0x7f361c162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f3621400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f361c0b3ac0 0x7f361c06b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f3621400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c162b60 con 0x7f361c0af820 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f3621400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c06b0f0 con 0x7f361c15e160 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f3621400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0b5620 con 0x7f361c0b3ac0 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f361a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f361c15e160 0x7f361c162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f361a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f361c15e160 0x7f361c162620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33060/0 (socket says 172.21.15.97:33060) 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.746+0000 7f361a800640 1 -- 172.21.15.97:0/1437150112 learned_addr learned my addr 172.21.15.97:0/1437150112 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.747+0000 7f361b200640 1 --2- 172.21.15.97:0/1437150112 >> v2:172.21.15.135:3300/0 conn(0x7f361c0b3ac0 0x7f361c06b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.747+0000 7f361a800640 1 -- 172.21.15.97:0/1437150112 >> v2:172.21.15.135:3300/0 conn(0x7f361c0b3ac0 msgr2=0x7f361c06b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.747+0000 7f361a800640 1 --2- 172.21.15.97:0/1437150112 >> v2:172.21.15.135:3300/0 conn(0x7f361c0b3ac0 0x7f361c06b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.747+0000 7f361a800640 1 -- 172.21.15.97:0/1437150112 >> v1:172.21.15.97:6789/0 conn(0x7f361c0af820 legacy=0x7f361c0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.747+0000 7f361a800640 1 -- 172.21.15.97:0/1437150112 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c06f1f0 con 0x7f361c15e160 2024-03-06T07:30:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.747+0000 7f361b200640 1 --2- 172.21.15.97:0/1437150112 >> v2:172.21.15.135:3300/0 conn(0x7f361c0b3ac0 0x7f361c06b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.748+0000 7f361a800640 1 --2- 172.21.15.97:0/1437150112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c162620 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f361c06e460 tx=0x7f360c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f65abc53a4b3074 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.748+0000 7f3619400640 1 -- 172.21.15.97:0/1437150112 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c035020 con 0x7f361c15e160 2024-03-06T07:30:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.748+0000 7f3619400640 1 -- 172.21.15.97:0/1437150112 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c031870 con 0x7f361c15e160 2024-03-06T07:30:44.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.748+0000 7f3619400640 1 -- 172.21.15.97:0/1437150112 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c031c40 con 0x7f361c15e160 2024-03-06T07:30:44.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 -- 172.21.15.97:0/1437150112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 msgr2=0x7f361c162620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 --2- 172.21.15.97:0/1437150112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c162620 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f361c06e460 tx=0x7f360c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 -- 172.21.15.97:0/1437150112 shutdown_connections 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 --2- 172.21.15.97:0/1437150112 >> v2:172.21.15.135:3300/0 conn(0x7f361c0b3ac0 0x7f361c06b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 --2- 172.21.15.97:0/1437150112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c162620 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 -- 172.21.15.97:0/1437150112 >> 172.21.15.97:0/1437150112 conn(0x7f361c0d0340 msgr2=0x7f361c0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 -- 172.21.15.97:0/1437150112 shutdown_connections 2024-03-06T07:30:44.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.749+0000 7f3621400640 1 -- 172.21.15.97:0/1437150112 wait complete. 2024-03-06T07:30:44.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.751+0000 7f3621400640 1 Processor -- start 2024-03-06T07:30:44.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.751+0000 7f3621400640 1 -- start start 2024-03-06T07:30:44.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f3621400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0af820 0x7f361c145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f3621400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0b3ac0 0x7f361c160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f3621400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c1609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f3621400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c06ef10 con 0x7f361c15e160 2024-03-06T07:30:44.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f3621400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c06ec30 con 0x7f361c0b3ac0 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f3621400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c06bac0 con 0x7f361c0af820 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f361b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c1609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f361b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c1609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33064/0 (socket says 172.21.15.97:33064) 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.752+0000 7f361b200640 1 -- 172.21.15.97:0/4051778001 learned_addr learned my addr 172.21.15.97:0/4051778001 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f361b200640 1 -- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0af820 msgr2=0x7f361c145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f361b200640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0af820 0x7f361c145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f361b200640 1 -- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0b3ac0 msgr2=0x7f361c160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3619e00640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0b3ac0 0x7f361c160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f361b200640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0b3ac0 0x7f361c160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f361b200640 1 -- 172.21.15.97:0/4051778001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c160fd0 con 0x7f361c15e160 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3619e00640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0b3ac0 0x7f361c160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f361b200640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c1609c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f361400cdd0 tx=0x7f361400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361401a070 con 0x7f361c15e160 2024-03-06T07:30:44.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361400d840 con 0x7f361c15e160 2024-03-06T07:30:44.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361400dc70 con 0x7f361c15e160 2024-03-06T07:30:44.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c1612b0 con 0x7f361c15e160 2024-03-06T07:30:44.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.753+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c157260 con 0x7f361c15e160 2024-03-06T07:30:44.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.755+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c074ae0 con 0x7f361c15e160 2024-03-06T07:30:44.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.755+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f361401d070 con 0x7f361c15e160 2024-03-06T07:30:44.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.756+0000 7f3612a00640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f35fc06b4d0 0x7f35fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.756+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f361408a950 con 0x7f361c15e160 2024-03-06T07:30:44.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.757+0000 7f361a800640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f35fc06b4d0 0x7f35fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.757+0000 7f361a800640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f35fc06b4d0 0x7f35fc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f360c036000 tx=0x7f360c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.761+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3614059860 con 0x7f361c15e160 2024-03-06T07:30:44.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.996+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f361c161480 con 0x7f361c15e160 2024-03-06T07:30:44.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.996+0000 7f3612a00640 1 -- 172.21.15.97:0/4051778001 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3614059160 con 0x7f361c15e160 2024-03-06T07:30:44.999 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:45.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f35fc06b4d0 msgr2=0x7f35fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f35fc06b4d0 0x7f35fc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f360c036000 tx=0x7f360c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 msgr2=0x7f361c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c1609c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f361400cdd0 tx=0x7f361400a710 comp rx=0 tx=0).stop 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 shutdown_connections 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f35fc06b4d0 0x7f35fc06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c15e160 0x7f361c1609c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0b3ac0 0x7f361c160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 --2- 172.21.15.97:0/4051778001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0af820 0x7f361c145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 >> 172.21.15.97:0/4051778001 conn(0x7f361c0d0340 msgr2=0x7f361c0ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 shutdown_connections 2024-03-06T07:30:45.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:44.999+0000 7f3621400640 1 -- 172.21.15.97:0/4051778001 wait complete. 2024-03-06T07:30:45.019 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:45.020 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:45.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.020 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:30:45.068 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42430sec preferred_lft 42430sec 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:30:45.069 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:30:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:45.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.071 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:30:45.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.152 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:30:45.193 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:30:45.193 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:45.193 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:45.193 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:45.193 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:45.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.193 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:30:45.193 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:30:45.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:45.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:45.194 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:45.194 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:30:45.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.404 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:30:45.452 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:45.452 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:45.453 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:45.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:30:45.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:45.540 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:45.589 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:45.589 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:45.590 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:45.634 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:45.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:45.810 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:45.810 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:45.843 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:45.843 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:45.843 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:30:45.843 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:30:45.843 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:30:45.845 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:45.845 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:45.845 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:45.845 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:45.845 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:45.849 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:45.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.849 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:45.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.927 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:45.979 INFO:teuthology.orchestra.run.smithi097.stdout:client4844 2024-03-06T07:30:45.979 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:45.979 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:46.026 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:46.026 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:46.117 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:30:46.117 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:30:46.117 DEBUG:teuthology.orchestra.run.smithi097:> import uuid 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> n = 75 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = False 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> if 0 > 0: 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = True 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> dpath = os.path.dirname(path) 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> try: 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(n): 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:46.118 DEBUG:teuthology.orchestra.run.smithi097:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> f.write(f"{i}") 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> f.flush() 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(f.fileno()) 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> os.unlink(fpath) 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> if create_hard_links: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> for j in range(0): 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> finally: 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> os.close(dirfd) 2024-03-06T07:30:46.119 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:30:47.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:47.152 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:47.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:47.314 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:47.429 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:47.429 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:47.429 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:47.486 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:47.486 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:47.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:47.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.784+0000 7f2301600640 1 Processor -- start 2024-03-06T07:30:47.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.784+0000 7f2301600640 1 -- start start 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f2301600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 0x7f22fc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f2301600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f22fc15fe30 0x7f22fc0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f2301600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc06ee70 con 0x7f22fc15f9d0 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f2301600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc06cef0 con 0x7f22fc15fe30 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f2301600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc06d1d0 con 0x7f22fc144c00 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f22f2000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f22fc15f9d0 0x7f22fc0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50970/0 (socket says 172.21.15.97:50970) 2024-03-06T07:30:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f22f2000640 1 -- 172.21.15.97:0/441677828 learned_addr learned my addr 172.21.15.97:0/441677828 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:47.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f22faa00640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 0x7f22fc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.786+0000 7f22fb400640 1 --2- 172.21.15.97:0/441677828 >> v2:172.21.15.135:3300/0 conn(0x7f22fc15fe30 0x7f22fc0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627141147 0 0) 0x7f22e4001110 con 0x7f22fc15f9d0 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e0003520 con 0x7f22fc15f9d0 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3732361279 0 0) 0x7f22e0003520 con 0x7f22fc15f9d0 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 msgr2=0x7f22fc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 0x7f22fc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 >> v2:172.21.15.135:3300/0 conn(0x7f22fc15fe30 msgr2=0x7f22fc0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 --2- 172.21.15.97:0/441677828 >> v2:172.21.15.135:3300/0 conn(0x7f22fc15fe30 0x7f22fc0ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc06d4b0 con 0x7f22fc15f9d0 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f22e4001840 con 0x7f22fc15f9d0 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 >> v1:172.21.15.97:6789/0 conn(0x7f22fc15f9d0 legacy=0x7f22fc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e00039b0 0x7f22e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 0x7f22e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22e0004d40 0x7f22e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc06cef0 con 0x7f22e0004d40 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc06d1d0 con 0x7f22e00039b0 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.787+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e000fdd0 con 0x7f22fc144c00 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22fb400640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22e0004d40 0x7f22e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22faa00640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e00039b0 0x7f22e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22f2000640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 0x7f22e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22fb400640 1 -- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 msgr2=0x7f22e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22fb400640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22fc144c00 0x7f22e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22fb400640 1 -- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e00039b0 msgr2=0x7f22e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22fb400640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e00039b0 0x7f22e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.788+0000 7f22fb400640 1 -- 172.21.15.97:0/441677828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0003520 con 0x7f22e0004d40 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.789+0000 7f22fb400640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22e0004d40 0x7f22e00051c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009ac0 tx=0x7f22e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6e28a62aa5547f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.789+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4039070 con 0x7f22e0004d40 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.789+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f402c710 con 0x7f22e0004d40 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.789+0000 7f22fa000640 1 -- 172.21.15.97:0/441677828 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4009690 con 0x7f22e0004d40 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f2301600640 1 -- 172.21.15.97:0/441677828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22e0004d40 msgr2=0x7f22e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f2301600640 1 --2- 172.21.15.97:0/441677828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22e0004d40 0x7f22e00051c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009ac0 tx=0x7f22e0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f22fb400640 1 -- 172.21.15.97:0/441677828 reap_dead start 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f2301600640 1 -- 172.21.15.97:0/441677828 shutdown_connections 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f2301600640 1 -- 172.21.15.97:0/441677828 >> 172.21.15.97:0/441677828 conn(0x7f22fc0b1730 msgr2=0x7f22fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f2301600640 1 -- 172.21.15.97:0/441677828 shutdown_connections 2024-03-06T07:30:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.790+0000 7f2301600640 1 -- 172.21.15.97:0/441677828 wait complete. 2024-03-06T07:30:47.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.793+0000 7f2301600640 1 Processor -- start 2024-03-06T07:30:47.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.793+0000 7f2301600640 1 -- start start 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f2301600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22e0004d40 0x7f22fc1373c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f2301600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22fc13dc30 0x7f22fc13e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f2301600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 0x7f22fc13ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f2301600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc160730 con 0x7f22fc13a6c0 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f2301600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc160280 con 0x7f22fc13dc30 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f2301600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc06cb50 con 0x7f22e0004d40 2024-03-06T07:30:47.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f22fb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 0x7f22fc13ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f22fb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 0x7f22fc13ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33078/0 (socket says 172.21.15.97:33078) 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.794+0000 7f22fb400640 1 -- 172.21.15.97:0/1523582272 learned_addr learned my addr 172.21.15.97:0/1523582272 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22faa00640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22e0004d40 0x7f22fc1373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22f2000640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22fc13dc30 0x7f22fc13e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22fb400640 1 -- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22e0004d40 msgr2=0x7f22fc1373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22fb400640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22e0004d40 0x7f22fc1373c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22fb400640 1 -- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22fc13dc30 msgr2=0x7f22fc13e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22fb400640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22fc13dc30 0x7f22fc13e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22fb400640 1 -- 172.21.15.97:0/1523582272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc13cb90 con 0x7f22fc13a6c0 2024-03-06T07:30:47.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22fb400640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 0x7f22fc13ab40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009be0 tx=0x7f22f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4039070 con 0x7f22fc13a6c0 2024-03-06T07:30:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.795+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f4043070 con 0x7f22fc13a6c0 2024-03-06T07:30:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.796+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc13ce70 con 0x7f22fc13a6c0 2024-03-06T07:30:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.796+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4009690 con 0x7f22fc13a6c0 2024-03-06T07:30:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.796+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc13d430 con 0x7f22fc13a6c0 2024-03-06T07:30:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.797+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc04f330 con 0x7f22fc13a6c0 2024-03-06T07:30:47.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.798+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22f4003850 con 0x7f22fc13a6c0 2024-03-06T07:30:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.799+0000 7f22f3400640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f22d806b4d0 0x7f22d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.799+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22f40ab9b0 con 0x7f22fc13a6c0 2024-03-06T07:30:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.800+0000 7f22faa00640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f22d806b4d0 0x7f22d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.800+0000 7f22faa00640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f22d806b4d0 0x7f22d806d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009fd0 tx=0x7f22ec00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.804+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22f407a820 con 0x7f22fc13a6c0 2024-03-06T07:30:48.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:47.998+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22fc1452f0 con 0x7f22fc13a6c0 2024-03-06T07:30:48.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.000+0000 7f22f3400640 1 -- 172.21.15.97:0/1523582272 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f22f407a120 con 0x7f22fc13a6c0 2024-03-06T07:30:48.002 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:48.002 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":58,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:21.268327+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.002+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f22d806b4d0 msgr2=0x7f22d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.002+0000 7f2301600640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f22d806b4d0 0x7f22d806d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009fd0 tx=0x7f22ec00b040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.002+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 msgr2=0x7f22fc13ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.002+0000 7f2301600640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 0x7f22fc13ab40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009be0 tx=0x7f22f4004730 comp rx=0 tx=0).stop 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 shutdown_connections 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f22d806b4d0 0x7f22d806d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22fc13a6c0 0x7f22fc13ab40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22fc13dc30 0x7f22fc13e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 --2- 172.21.15.97:0/1523582272 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f22e0004d40 0x7f22fc1373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 >> 172.21.15.97:0/1523582272 conn(0x7f22fc0b1730 msgr2=0x7f22fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 shutdown_connections 2024-03-06T07:30:48.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.003+0000 7f2301600640 1 -- 172.21.15.97:0/1523582272 wait complete. 2024-03-06T07:30:48.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:17.521656+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:18.729778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:48.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:48.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.220+0000 7fbcd3600640 1 Processor -- start 2024-03-06T07:30:48.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.220+0000 7fbcd3600640 1 -- start start 2024-03-06T07:30:48.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.221+0000 7fbcd3600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.221+0000 7fbcd3600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbccc070ff0 0x7fbccc0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.221+0000 7fbcd3600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc0b46c0 con 0x7fbccc15fc10 2024-03-06T07:30:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.221+0000 7fbcd3600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc071480 con 0x7fbccc070ff0 2024-03-06T07:30:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.221+0000 7fbcd3600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0b49a0 con 0x7fbccc0b3f70 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbccc070ff0 0x7fbccc0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbccc070ff0 0x7fbccc0ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33082/0 (socket says 172.21.15.97:33082) 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 -- 172.21.15.97:0/878689194 learned_addr learned my addr 172.21.15.97:0/878689194 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbccbe00640 1 --2- 172.21.15.97:0/878689194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 -- 172.21.15.97:0/878689194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 msgr2=0x7fbccc160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 --2- 172.21.15.97:0/878689194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 -- 172.21.15.97:0/878689194 >> v1:172.21.15.135:6789/0 conn(0x7fbccc15fc10 legacy=0x7fbccc15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.222+0000 7fbcd1600640 1 -- 172.21.15.97:0/878689194 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc072c90 con 0x7fbccc070ff0 2024-03-06T07:30:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.223+0000 7fbcd1600640 1 --2- 172.21.15.97:0/878689194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc070ff0 0x7fbccc0ce4a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009e50 tx=0x7fbcc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95bd7742bcce2782 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbccb400640 1 -- 172.21.15.97:0/878689194 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc402cb00 con 0x7fbccc070ff0 2024-03-06T07:30:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbccb400640 1 -- 172.21.15.97:0/878689194 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc402cca0 con 0x7fbccc070ff0 2024-03-06T07:30:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbccb400640 1 -- 172.21.15.97:0/878689194 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4034610 con 0x7fbccc070ff0 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbcd3600640 1 -- 172.21.15.97:0/878689194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc070ff0 msgr2=0x7fbccc0ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbcd3600640 1 --2- 172.21.15.97:0/878689194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc070ff0 0x7fbccc0ce4a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009e50 tx=0x7fbcc402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbcd3600640 1 -- 172.21.15.97:0/878689194 shutdown_connections 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbcd3600640 1 --2- 172.21.15.97:0/878689194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc070ff0 0x7fbccc0ce4a0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbcd3600640 1 --2- 172.21.15.97:0/878689194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.224+0000 7fbcd3600640 1 -- 172.21.15.97:0/878689194 >> 172.21.15.97:0/878689194 conn(0x7fbccc0b1940 msgr2=0x7fbccc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.225+0000 7fbcd3600640 1 -- 172.21.15.97:0/878689194 shutdown_connections 2024-03-06T07:30:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.225+0000 7fbcd3600640 1 -- 172.21.15.97:0/878689194 wait complete. 2024-03-06T07:30:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.228+0000 7fbcd3600640 1 Processor -- start 2024-03-06T07:30:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.228+0000 7fbcd3600640 1 -- start start 2024-03-06T07:30:48.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.230+0000 7fbcd3600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbccc070ff0 0x7fbccc14d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.230+0000 7fbcd3600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc09a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.230+0000 7fbcd3600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 0x7fbccc0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.230+0000 7fbcd3600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc072a30 con 0x7fbccc15fc10 2024-03-06T07:30:48.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.230+0000 7fbcd3600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0728b0 con 0x7fbccc070ff0 2024-03-06T07:30:48.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.230+0000 7fbcd3600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc15c2e0 con 0x7fbccc0b3f70 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 0x7fbccc0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 0x7fbccc0c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33088/0 (socket says 172.21.15.97:33088) 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 -- 172.21.15.97:0/1225925420 learned_addr learned my addr 172.21.15.97:0/1225925420 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd0c00640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbccc070ff0 0x7fbccc14d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbccbe00640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc09a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 -- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 msgr2=0x7fbccc09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc09a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 -- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbccc070ff0 msgr2=0x7fbccc14d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbccc070ff0 0x7fbccc14d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 -- 172.21.15.97:0/1225925420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc159eb0 con 0x7fbccc15fc10 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd0c00640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbccc070ff0 0x7fbccc14d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.231+0000 7fbcd1600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 0x7fbccc0c9620 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009f70 tx=0x7fbcc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.232+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4043020 con 0x7fbccc15fc10 2024-03-06T07:30:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.232+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc402ce50 con 0x7fbccc15fc10 2024-03-06T07:30:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.232+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4009510 con 0x7fbccc15fc10 2024-03-06T07:30:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.232+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc15a190 con 0x7fbccc15fc10 2024-03-06T07:30:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.232+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc15a750 con 0x7fbccc15fc10 2024-03-06T07:30:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.233+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc04f330 con 0x7fbccc15fc10 2024-03-06T07:30:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.234+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcc4046020 con 0x7fbccc15fc10 2024-03-06T07:30:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.235+0000 7fbcc8c00640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcac06b500 0x7fbcac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.235+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcc40ac8f0 con 0x7fbccc15fc10 2024-03-06T07:30:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.235+0000 7fbcd0c00640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcac06b500 0x7fbcac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.236+0000 7fbcd0c00640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcac06b500 0x7fbcac06d9c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0001700 tx=0x7fbcc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.240+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcc404d030 con 0x7fbccc15fc10 2024-03-06T07:30:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.436+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbccc06ef30 con 0x7fbccc15fc10 2024-03-06T07:30:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.436+0000 7fbcc8c00640 1 -- 172.21.15.97:0/1225925420 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbcc407b6d0 con 0x7fbccc15fc10 2024-03-06T07:30:48.439 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:30:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcac06b500 msgr2=0x7fbcac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcac06b500 0x7fbcac06d9c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0001700 tx=0x7fbcc0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 msgr2=0x7fbccc0c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 0x7fbccc0c9620 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009f70 tx=0x7fbcc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 shutdown_connections 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcac06b500 0x7fbcac06d9c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbccc15fc10 0x7fbccc0c9620 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbccc0b3f70 0x7fbccc09a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 --2- 172.21.15.97:0/1225925420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbccc070ff0 0x7fbccc14d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 >> 172.21.15.97:0/1225925420 conn(0x7fbccc0b1940 msgr2=0x7fbccc0ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 shutdown_connections 2024-03-06T07:30:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.439+0000 7fbcd3600640 1 -- 172.21.15.97:0/1225925420 wait complete. 2024-03-06T07:30:48.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.673+0000 7f87c3a00640 1 Processor -- start 2024-03-06T07:30:48.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.674+0000 7f87c3a00640 1 -- start start 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c3a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f87bc1618e0 0x7f87bc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c3a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f87bc0cf6f0 0x7f87bc0b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c3a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc0b32f0 con 0x7f87bc0b2d80 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c3a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc0d0d20 con 0x7f87bc1618e0 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c3a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc0d0f70 con 0x7f87bc0cf6f0 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f87bc1618e0 0x7f87bc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f87bc1618e0 0x7f87bc0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33094/0 (socket says 172.21.15.97:33094) 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.675+0000 7f87c1000640 1 -- 172.21.15.97:0/3978443252 learned_addr learned my addr 172.21.15.97:0/3978443252 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.676+0000 7f87c1a00640 1 --2- 172.21.15.97:0/3978443252 >> v2:172.21.15.135:3300/0 conn(0x7f87bc0cf6f0 0x7f87bc0b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.676+0000 7f87c1000640 1 -- 172.21.15.97:0/3978443252 >> v2:172.21.15.135:3300/0 conn(0x7f87bc0cf6f0 msgr2=0x7f87bc0b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.676+0000 7f87c1000640 1 --2- 172.21.15.97:0/3978443252 >> v2:172.21.15.135:3300/0 conn(0x7f87bc0cf6f0 0x7f87bc0b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.676+0000 7f87c1000640 1 -- 172.21.15.97:0/3978443252 >> v1:172.21.15.135:6789/0 conn(0x7f87bc0b2d80 legacy=0x7f87bc0cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.676+0000 7f87c1000640 1 -- 172.21.15.97:0/3978443252 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0d1250 con 0x7f87bc1618e0 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.677+0000 7f87c1000640 1 --2- 172.21.15.97:0/3978443252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0ceaa0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f87bc08da90 tx=0x7f87b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8438d5f34e23ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.677+0000 7f87bb400640 1 -- 172.21.15.97:0/3978443252 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0035020 con 0x7f87bc1618e0 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.677+0000 7f87bb400640 1 -- 172.21.15.97:0/3978443252 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b0031870 con 0x7f87bc1618e0 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.677+0000 7f87bb400640 1 -- 172.21.15.97:0/3978443252 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0031c40 con 0x7f87bc1618e0 2024-03-06T07:30:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 -- 172.21.15.97:0/3978443252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 msgr2=0x7f87bc0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 --2- 172.21.15.97:0/3978443252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0ceaa0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f87bc08da90 tx=0x7f87b002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 -- 172.21.15.97:0/3978443252 shutdown_connections 2024-03-06T07:30:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 --2- 172.21.15.97:0/3978443252 >> v2:172.21.15.135:3300/0 conn(0x7f87bc0cf6f0 0x7f87bc0b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 --2- 172.21.15.97:0/3978443252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0ceaa0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 -- 172.21.15.97:0/3978443252 >> 172.21.15.97:0/3978443252 conn(0x7f87bc0d3e90 msgr2=0x7f87bc0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 -- 172.21.15.97:0/3978443252 shutdown_connections 2024-03-06T07:30:48.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.678+0000 7f87c3a00640 1 -- 172.21.15.97:0/3978443252 wait complete. 2024-03-06T07:30:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.680+0000 7f87c3a00640 1 Processor -- start 2024-03-06T07:30:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.680+0000 7f87c3a00640 1 -- start start 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c3a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87bc0b2d80 0x7f87bc0b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c3a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87bc0cf6f0 0x7f87bc0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c3a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c3a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0d2da0 con 0x7f87bc1618e0 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c3a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc161cc0 con 0x7f87bc0cf6f0 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c3a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0b1180 con 0x7f87bc0b2d80 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c1a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c1a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33100/0 (socket says 172.21.15.97:33100) 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.681+0000 7f87c1a00640 1 -- 172.21.15.97:0/2517775580 learned_addr learned my addr 172.21.15.97:0/2517775580 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1000640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87bc0b2d80 0x7f87bc0b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87bbe00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87bc0cf6f0 0x7f87bc0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1a00640 1 -- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87bc0b2d80 msgr2=0x7f87bc0b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87bc0b2d80 0x7f87bc0b8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1a00640 1 -- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87bc0cf6f0 msgr2=0x7f87bc0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87bc0cf6f0 0x7f87bc0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1a00640 1 -- 172.21.15.97:0/2517775580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0c9e60 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c1a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0b3ba0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f87b4007e30 tx=0x7f87b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4014620 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4019070 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc0c9b80 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.682+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4014a00 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.683+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc0cb940 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.684+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc04f330 con 0x7f87bc1618e0 2024-03-06T07:30:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.685+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87b4014ba0 con 0x7f87bc1618e0 2024-03-06T07:30:48.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.685+0000 7f87b8c00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f879c06b4d0 0x7f879c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.686+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87b408aa10 con 0x7f87bc1618e0 2024-03-06T07:30:48.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.686+0000 7f87c1000640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f879c06b4d0 0x7f879c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.687+0000 7f87c1000640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f879c06b4d0 0x7f879c06d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f87b0036000 tx=0x7f87b003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.690+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87b4059900 con 0x7f87bc1618e0 2024-03-06T07:30:48.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.887+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87bc06bc70 con 0x7f87bc1618e0 2024-03-06T07:30:48.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.888+0000 7f87b8c00640 1 -- 172.21.15.97:0/2517775580 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87b4059200 con 0x7f87bc1618e0 2024-03-06T07:30:48.890 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:30:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f879c06b4d0 msgr2=0x7f879c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f879c06b4d0 0x7f879c06d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f87b0036000 tx=0x7f87b003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 msgr2=0x7f87bc0b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0b3ba0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f87b4007e30 tx=0x7f87b400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 shutdown_connections 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f879c06b4d0 0x7f879c06d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87bc1618e0 0x7f87bc0b3ba0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87bc0cf6f0 0x7f87bc0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 --2- 172.21.15.97:0/2517775580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87bc0b2d80 0x7f87bc0b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 >> 172.21.15.97:0/2517775580 conn(0x7f87bc0d3e90 msgr2=0x7f87bc06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 shutdown_connections 2024-03-06T07:30:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:48.891+0000 7f87c3a00640 1 -- 172.21.15.97:0/2517775580 wait complete. 2024-03-06T07:30:48.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:49.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.119+0000 7fb197e00640 1 Processor -- start 2024-03-06T07:30:49.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.119+0000 7fb197e00640 1 -- start start 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb197e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb19806afc0 0x7fb19806b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb197e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb19815e7c0 0x7fb19815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb197e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980cad60 con 0x7fb1980cf4a0 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb197e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19806b8e0 con 0x7fb19806afc0 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb197e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb198161ab0 con 0x7fb19815e7c0 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb196a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb19806afc0 0x7fb19806b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb196a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb19806afc0 0x7fb19806b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33116/0 (socket says 172.21.15.97:33116) 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb197400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb1980cf4a0 0x7fb1980cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50984/0 (socket says 172.21.15.97:50984) 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb196a00640 1 -- 172.21.15.97:0/2303665619 learned_addr learned my addr 172.21.15.97:0/2303665619 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.121+0000 7fb196000640 1 --2- 172.21.15.97:0/2303665619 >> v2:172.21.15.135:3300/0 conn(0x7fb19815e7c0 0x7fb19815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.122+0000 7fb196a00640 1 -- 172.21.15.97:0/2303665619 >> v2:172.21.15.135:3300/0 conn(0x7fb19815e7c0 msgr2=0x7fb19815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.122+0000 7fb196a00640 1 --2- 172.21.15.97:0/2303665619 >> v2:172.21.15.135:3300/0 conn(0x7fb19815e7c0 0x7fb19815ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.122+0000 7fb196a00640 1 -- 172.21.15.97:0/2303665619 >> v1:172.21.15.97:6789/0 conn(0x7fb1980cf4a0 legacy=0x7fb1980cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.122+0000 7fb196a00640 1 -- 172.21.15.97:0/2303665619 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb198161d90 con 0x7fb19806afc0 2024-03-06T07:30:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.123+0000 7fb196a00640 1 --2- 172.21.15.97:0/2303665619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19806b3a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb198070610 tx=0x7fb18802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ab360e4c27e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb195600640 1 -- 172.21.15.97:0/2303665619 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188035020 con 0x7fb19806afc0 2024-03-06T07:30:49.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb195600640 1 -- 172.21.15.97:0/2303665619 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb188031870 con 0x7fb19806afc0 2024-03-06T07:30:49.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb195600640 1 -- 172.21.15.97:0/2303665619 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188031c40 con 0x7fb19806afc0 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 -- 172.21.15.97:0/2303665619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 msgr2=0x7fb19806b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 --2- 172.21.15.97:0/2303665619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19806b3a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb198070610 tx=0x7fb18802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 -- 172.21.15.97:0/2303665619 shutdown_connections 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 --2- 172.21.15.97:0/2303665619 >> v2:172.21.15.135:3300/0 conn(0x7fb19815e7c0 0x7fb19815ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 --2- 172.21.15.97:0/2303665619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19806b3a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 -- 172.21.15.97:0/2303665619 >> 172.21.15.97:0/2303665619 conn(0x7fb1980cd480 msgr2=0x7fb1980cd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 -- 172.21.15.97:0/2303665619 shutdown_connections 2024-03-06T07:30:49.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.124+0000 7fb197e00640 1 -- 172.21.15.97:0/2303665619 wait complete. 2024-03-06T07:30:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.126+0000 7fb197e00640 1 Processor -- start 2024-03-06T07:30:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.126+0000 7fb197e00640 1 -- start start 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb197e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19814b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb197e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1980cf4a0 0x7fb19806e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb197e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb19815e7c0 0x7fb1980b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb197e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19815f150 con 0x7fb19806afc0 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb197e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb198162350 con 0x7fb19815e7c0 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb197e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19815f2d0 con 0x7fb1980cf4a0 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb196a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19814b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb196a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19814b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33128/0 (socket says 172.21.15.97:33128) 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.127+0000 7fb196a00640 1 -- 172.21.15.97:0/2486191332 learned_addr learned my addr 172.21.15.97:0/2486191332 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196000640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1980cf4a0 0x7fb19806e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196a00640 1 -- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1980cf4a0 msgr2=0x7fb19806e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196a00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1980cf4a0 0x7fb19806e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196a00640 1 -- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb19815e7c0 msgr2=0x7fb1980b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196a00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb19815e7c0 0x7fb1980b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196a00640 1 -- 172.21.15.97:0/2486191332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19806ef80 con 0x7fb19806afc0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb196a00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19814b660 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb18802b950 tx=0x7fb1880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.128+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188040070 con 0x7fb19806afc0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.129+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb188030d00 con 0x7fb19806afc0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.129+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188035040 con 0x7fb19806afc0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.129+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19806f260 con 0x7fb19806afc0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.129+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19806f820 con 0x7fb19806afc0 2024-03-06T07:30:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.130+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19804f330 con 0x7fb19806afc0 2024-03-06T07:30:49.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.131+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb18804b060 con 0x7fb19806afc0 2024-03-06T07:30:49.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.132+0000 7fb18ea00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb17806b500 0x7fb17806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.132+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1880abf50 con 0x7fb19806afc0 2024-03-06T07:30:49.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.132+0000 7fb196000640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb17806b500 0x7fb17806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.133+0000 7fb196000640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb17806b500 0x7fb17806d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb19806fcb0 tx=0x7fb184008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.137+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb18807adc0 con 0x7fb19806afc0 2024-03-06T07:30:49.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.332+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb19815dff0 con 0x7fb19806afc0 2024-03-06T07:30:49.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.333+0000 7fb18ea00640 1 -- 172.21.15.97:0/2486191332 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb18807a6c0 con 0x7fb19806afc0 2024-03-06T07:30:49.335 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:30:49.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb17806b500 msgr2=0x7fb17806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb17806b500 0x7fb17806d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb19806fcb0 tx=0x7fb184008040 comp rx=0 tx=0).stop 2024-03-06T07:30:49.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 msgr2=0x7fb19814b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19814b660 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb18802b950 tx=0x7fb1880046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 shutdown_connections 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb17806b500 0x7fb17806d9c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb19815e7c0 0x7fb1980b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1980cf4a0 0x7fb19806e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 --2- 172.21.15.97:0/2486191332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19806afc0 0x7fb19814b660 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 >> 172.21.15.97:0/2486191332 conn(0x7fb1980cd480 msgr2=0x7fb1980cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 shutdown_connections 2024-03-06T07:30:49.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.336+0000 7fb197e00640 1 -- 172.21.15.97:0/2486191332 wait complete. 2024-03-06T07:30:49.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:49.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.552+0000 7f5b75c00640 1 Processor -- start 2024-03-06T07:30:49.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.552+0000 7f5b75c00640 1 -- start start 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.555+0000 7f5b75c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5b701581d0 0x7f5b700cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.555+0000 7f5b75c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b7006d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.555+0000 7f5b75c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7015b9e0 con 0x7f5b7006d9b0 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.555+0000 7f5b75c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b700cbff0 con 0x7f5b701581d0 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.555+0000 7f5b75c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700cc220 con 0x7f5b7006e470 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5b7006d9b0 0x7f5b7015e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50998/0 (socket says 172.21.15.97:50998) 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6f400640 1 -- 172.21.15.97:0/828695562 learned_addr learned my addr 172.21.15.97:0/828695562 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:49.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6e000640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b7006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/828695562 >> v2:172.21.15.135:3300/0 conn(0x7f5b701581d0 0x7f5b700cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751223446 0 0) 0x7f5b7015b9e0 con 0x7f5b7006d9b0 2024-03-06T07:30:49.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b54003520 con 0x7f5b7006d9b0 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.556+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881926909 0 0) 0x7f5b54003520 con 0x7f5b7006d9b0 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 msgr2=0x7f5b7006d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b7006d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 >> v2:172.21.15.135:3300/0 conn(0x7f5b701581d0 msgr2=0x7f5b700cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 --2- 172.21.15.97:0/828695562 >> v2:172.21.15.135:3300/0 conn(0x7f5b701581d0 0x7f5b700cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7015ac20 con 0x7f5b7006d9b0 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f5b680019f0 con 0x7f5b7006d9b0 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 >> v1:172.21.15.97:6789/0 conn(0x7f5b7006d9b0 legacy=0x7f5b7015e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b540039b0 0x7f5b54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b54004800 0x7f5b54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700cc220 con 0x7f5b540039b0 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5400b9c0 con 0x7f5b54004800 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5400bb40 con 0x7f5b7006e470 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b540039b0 0x7f5b54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6f400640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6ea00640 1 -- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 msgr2=0x7f5b540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6e000640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b54004800 0x7f5b54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6ea00640 1 -- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b54004800 msgr2=0x7f5b54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b54004800 0x7f5b54004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.557+0000 7f5b6ea00640 1 -- 172.21.15.97:0/828695562 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54003520 con 0x7f5b540039b0 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.558+0000 7f5b6f400640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b7006e470 0x7f5b540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b540039b0 0x7f5b54003dd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009ac0 tx=0x7f5b54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ea350d44da4a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60039070 con 0x7f5b540039b0 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6002c710 con 0x7f5b540039b0 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b6d600640 1 -- 172.21.15.97:0/828695562 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60009690 con 0x7f5b540039b0 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b75c00640 1 -- 172.21.15.97:0/828695562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b540039b0 msgr2=0x7f5b54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b75c00640 1 --2- 172.21.15.97:0/828695562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b540039b0 0x7f5b54003dd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009ac0 tx=0x7f5b54005830 comp rx=0 tx=0).stop 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b6f400640 1 -- 172.21.15.97:0/828695562 reap_dead start 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b75c00640 1 -- 172.21.15.97:0/828695562 shutdown_connections 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b75c00640 1 -- 172.21.15.97:0/828695562 >> 172.21.15.97:0/828695562 conn(0x7f5b700cc4f0 msgr2=0x7f5b700cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.559+0000 7f5b75c00640 1 -- 172.21.15.97:0/828695562 shutdown_connections 2024-03-06T07:30:49.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.560+0000 7f5b75c00640 1 -- 172.21.15.97:0/828695562 wait complete. 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.562+0000 7f5b75c00640 1 Processor -- start 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.562+0000 7f5b75c00640 1 -- start start 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b75c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b540039b0 0x7f5b700b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b75c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b7014eab0 0x7f5b700b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b75c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 0x7f5b700c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b75c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7015b4c0 con 0x7f5b700b0dd0 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b75c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7015b1e0 con 0x7f5b7014eab0 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b75c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700cb250 con 0x7f5b540039b0 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b6f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 0x7f5b700c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b6f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 0x7f5b700c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33152/0 (socket says 172.21.15.97:33152) 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.563+0000 7f5b6f400640 1 -- 172.21.15.97:0/2847898833 learned_addr learned my addr 172.21.15.97:0/2847898833 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b540039b0 0x7f5b700b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6e000640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b7014eab0 0x7f5b700b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6f400640 1 -- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b540039b0 msgr2=0x7f5b700b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6f400640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b540039b0 0x7f5b700b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6f400640 1 -- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b7014eab0 msgr2=0x7f5b700b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6f400640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b7014eab0 0x7f5b700b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6f400640 1 -- 172.21.15.97:0/2847898833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b700b1270 con 0x7f5b700b0dd0 2024-03-06T07:30:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b540039b0 0x7f5b700b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b6f400640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 0x7f5b700c8fb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5b70073760 tx=0x7f5b6800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68004d60 con 0x7f5b700b0dd0 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.564+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6800bb80 con 0x7f5b700b0dd0 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.565+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b700b1550 con 0x7f5b700b0dd0 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.565+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68007110 con 0x7f5b700b0dd0 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.565+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b700c7020 con 0x7f5b700b0dd0 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.566+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7004f330 con 0x7f5b700b0dd0 2024-03-06T07:30:49.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.567+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b6801b020 con 0x7f5b700b0dd0 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.568+0000 7f5b66a00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b5006b4d0 0x7f5b5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.568+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b6808a4d0 con 0x7f5b700b0dd0 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.568+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b5006b4d0 0x7f5b5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.569+0000 7f5b6ea00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b5006b4d0 0x7f5b5006d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5b60036000 tx=0x7f5b60030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.573+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b680592b0 con 0x7f5b700b0dd0 2024-03-06T07:30:49.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.782+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5b700aed90 con 0x7f5b700b0dd0 2024-03-06T07:30:49.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.782+0000 7f5b66a00640 1 -- 172.21.15.97:0/2847898833 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5b68058bb0 con 0x7f5b700b0dd0 2024-03-06T07:30:49.785 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b5006b4d0 msgr2=0x7f5b5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b5006b4d0 0x7f5b5006d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5b60036000 tx=0x7f5b60030000 comp rx=0 tx=0).stop 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 msgr2=0x7f5b700c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 0x7f5b700c8fb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5b70073760 tx=0x7f5b6800b4a0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 shutdown_connections 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b5006b4d0 0x7f5b5006d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b700b0dd0 0x7f5b700c8fb0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b7014eab0 0x7f5b700b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 --2- 172.21.15.97:0/2847898833 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b540039b0 0x7f5b700b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.785+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 >> 172.21.15.97:0/2847898833 conn(0x7f5b700cc4f0 msgr2=0x7f5b700ade20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.786+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 shutdown_connections 2024-03-06T07:30:49.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:49.786+0000 7f5b75c00640 1 -- 172.21.15.97:0/2847898833 wait complete. 2024-03-06T07:30:49.807 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:49.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:49.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:50.022 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:30:50.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:50.022 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:50.063 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:30:50.063 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:50.104 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:50.105 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:30:50.105 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:50.141 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:50.141 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:30:50.141 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:50.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:50.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:50.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.390+0000 7f5e5e400640 1 Processor -- start 2024-03-06T07:30:50.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.390+0000 7f5e5e400640 1 -- start start 2024-03-06T07:30:50.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.392+0000 7f5e5e400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5e580d24c0 0x7f5e580d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.392+0000 7f5e5e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.392+0000 7f5e5e400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e580d2de0 con 0x7f5e580d07f0 2024-03-06T07:30:50.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.392+0000 7f5e5e400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e580d4a00 con 0x7f5e580d24c0 2024-03-06T07:30:50.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.392+0000 7f5e5e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580cf7e0 con 0x7f5e580b2de0 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.392+0000 7f5e56a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e57e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f5e580d07f0 0x7f5e580d0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:45142/0 (socket says 172.21.15.97:45142) 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e56a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:34814/0 (socket says 172.21.15.97:34814) 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e57e00640 1 -- 172.21.15.97:0/406821266 learned_addr learned my addr 172.21.15.97:0/406821266 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e57400640 1 --2- 172.21.15.97:0/406821266 >> v2:172.21.15.135:3300/0 conn(0x7f5e580d24c0 0x7f5e580d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e56000640 1 -- 172.21.15.97:0/406821266 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026915292 0 0) 0x7f5e580d2de0 con 0x7f5e580d07f0 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e56000640 1 -- 172.21.15.97:0/406821266 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c003520 con 0x7f5e580d07f0 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e56a00640 1 -- 172.21.15.97:0/406821266 >> v2:172.21.15.135:3300/0 conn(0x7f5e580d24c0 msgr2=0x7f5e580d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.393+0000 7f5e56a00640 1 --2- 172.21.15.97:0/406821266 >> v2:172.21.15.135:3300/0 conn(0x7f5e580d24c0 0x7f5e580d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.394+0000 7f5e56a00640 1 -- 172.21.15.97:0/406821266 >> v1:172.21.15.135:6789/0 conn(0x7f5e580d07f0 legacy=0x7f5e580d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.394+0000 7f5e56a00640 1 -- 172.21.15.97:0/406821266 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e580d4c50 con 0x7f5e580b2de0 2024-03-06T07:30:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.394+0000 7f5e57400640 1 --2- 172.21.15.97:0/406821266 >> v2:172.21.15.135:3300/0 conn(0x7f5e580d24c0 0x7f5e580d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:50.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.395+0000 7f5e56a00640 1 --2- 172.21.15.97:0/406821266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580d28a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e440097a0 tx=0x7f5e4402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dacb9db570b68d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.395+0000 7f5e56000640 1 -- 172.21.15.97:0/406821266 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44031780 con 0x7f5e580b2de0 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.395+0000 7f5e56000640 1 -- 172.21.15.97:0/406821266 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44037070 con 0x7f5e580b2de0 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.395+0000 7f5e56000640 1 -- 172.21.15.97:0/406821266 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44031b20 con 0x7f5e580b2de0 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 -- 172.21.15.97:0/406821266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 msgr2=0x7f5e580d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 --2- 172.21.15.97:0/406821266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580d28a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e440097a0 tx=0x7f5e4402b820 comp rx=0 tx=0).stop 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 -- 172.21.15.97:0/406821266 shutdown_connections 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 --2- 172.21.15.97:0/406821266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580d28a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 --2- 172.21.15.97:0/406821266 >> v2:172.21.15.135:3300/0 conn(0x7f5e580d24c0 0x7f5e580d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 -- 172.21.15.97:0/406821266 >> 172.21.15.97:0/406821266 conn(0x7f5e580d3ee0 msgr2=0x7f5e580d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 -- 172.21.15.97:0/406821266 shutdown_connections 2024-03-06T07:30:50.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.396+0000 7f5e5e400640 1 -- 172.21.15.97:0/406821266 wait complete. 2024-03-06T07:30:50.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.398+0000 7f5e5e400640 1 Processor -- start 2024-03-06T07:30:50.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.398+0000 7f5e5e400640 1 -- start start 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e5e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e5e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e580d07f0 0x7f5e580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e5e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 0x7f5e581f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e5e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580cfd10 con 0x7f5e580d24c0 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e5e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580d0080 con 0x7f5e580d07f0 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e5e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5815e5b0 con 0x7f5e580b2de0 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 0x7f5e581f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 0x7f5e581f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33158/0 (socket says 172.21.15.97:33158) 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e57e00640 1 -- 172.21.15.97:0/463143715 learned_addr learned my addr 172.21.15.97:0/463143715 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e57400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.399+0000 7f5e56a00640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e580d07f0 0x7f5e580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e57e00640 1 -- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 msgr2=0x7f5e580cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e57e00640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e57e00640 1 -- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e580d07f0 msgr2=0x7f5e580b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e57e00640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e580d07f0 0x7f5e580b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e57e00640 1 -- 172.21.15.97:0/463143715 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e580b90d0 con 0x7f5e580d24c0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e57e00640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 0x7f5e581f4710 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000cb40 tx=0x7f5e5000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50017070 con 0x7f5e580d24c0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5000aca0 con 0x7f5e580d24c0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50012620 con 0x7f5e580d24c0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.400+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e580b9350 con 0x7f5e580d24c0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.401+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e580b9910 con 0x7f5e580d24c0 2024-03-06T07:30:50.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.402+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5804f330 con 0x7f5e580d24c0 2024-03-06T07:30:50.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.404+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e5001a070 con 0x7f5e580d24c0 2024-03-06T07:30:50.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.405+0000 7f5e4f400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e3806b4d0 0x7f5e3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.405+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e5008a070 con 0x7f5e580d24c0 2024-03-06T07:30:50.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.405+0000 7f5e57400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e3806b4d0 0x7f5e3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.406+0000 7f5e57400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e3806b4d0 0x7f5e3806d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009a30 tx=0x7f5e48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.409+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e580b90d0 con 0x7f5e580d24c0 2024-03-06T07:30:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.645+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e580d22a0 con 0x7f5e580d24c0 2024-03-06T07:30:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.646+0000 7f5e4f400640 1 -- 172.21.15.97:0/463143715 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f5e50058ee0 con 0x7f5e580d24c0 2024-03-06T07:30:50.648 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:50.648 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e3806b4d0 msgr2=0x7f5e3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e3806b4d0 0x7f5e3806d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009a30 tx=0x7f5e48008040 comp rx=0 tx=0).stop 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 msgr2=0x7f5e581f4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 0x7f5e581f4710 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000cb40 tx=0x7f5e5000a060 comp rx=0 tx=0).stop 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 shutdown_connections 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e3806b4d0 0x7f5e3806d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e580d24c0 0x7f5e581f4710 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e580d07f0 0x7f5e580b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 --2- 172.21.15.97:0/463143715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e580b2de0 0x7f5e580cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.648+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 >> 172.21.15.97:0/463143715 conn(0x7f5e580d3ee0 msgr2=0x7f5e580b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.649+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 shutdown_connections 2024-03-06T07:30:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.649+0000 7f5e5e400640 1 -- 172.21.15.97:0/463143715 wait complete. 2024-03-06T07:30:50.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.867+0000 7fba3f200640 1 Processor -- start 2024-03-06T07:30:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.867+0000 7fba3f200640 1 -- start start 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fba380d26e0 0x7fba380ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3815d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3806c060 con 0x7fba380d00c0 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3806be10 con 0x7fba380d26e0 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3806c340 con 0x7fba380b4930 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fba380d26e0 0x7fba380ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3de00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fba380d00c0 0x7fba380d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51002/0 (socket says 172.21.15.97:51002) 2024-03-06T07:30:50.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3de00640 1 -- 172.21.15.97:0/780577968 learned_addr learned my addr 172.21.15.97:0/780577968 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:50.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3d400640 1 -- 172.21.15.97:0/780577968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 msgr2=0x7fba3815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3d400640 1 --2- 172.21.15.97:0/780577968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3d400640 1 -- 172.21.15.97:0/780577968 >> v1:172.21.15.97:6789/0 conn(0x7fba380d00c0 legacy=0x7fba380d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.869+0000 7fba3d400640 1 -- 172.21.15.97:0/780577968 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3806c620 con 0x7fba380d26e0 2024-03-06T07:30:50.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.871+0000 7fba3d400640 1 --2- 172.21.15.97:0/780577968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d26e0 0x7fba380ceb40 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fba280097a0 tx=0x7fba2802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0a48cade3c5de0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.871+0000 7fba3ca00640 1 -- 172.21.15.97:0/780577968 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28035020 con 0x7fba380d26e0 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.871+0000 7fba3ca00640 1 -- 172.21.15.97:0/780577968 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28031810 con 0x7fba380d26e0 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.871+0000 7fba3ca00640 1 -- 172.21.15.97:0/780577968 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28031be0 con 0x7fba380d26e0 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 -- 172.21.15.97:0/780577968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d26e0 msgr2=0x7fba380ceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 --2- 172.21.15.97:0/780577968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d26e0 0x7fba380ceb40 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fba280097a0 tx=0x7fba2802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 -- 172.21.15.97:0/780577968 shutdown_connections 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 --2- 172.21.15.97:0/780577968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 --2- 172.21.15.97:0/780577968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d26e0 0x7fba380ceb40 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 -- 172.21.15.97:0/780577968 >> 172.21.15.97:0/780577968 conn(0x7fba380d1ae0 msgr2=0x7fba380d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 -- 172.21.15.97:0/780577968 shutdown_connections 2024-03-06T07:30:50.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.872+0000 7fba3f200640 1 -- 172.21.15.97:0/780577968 wait complete. 2024-03-06T07:30:50.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.873+0000 7fba3f200640 1 Processor -- start 2024-03-06T07:30:50.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.874+0000 7fba3f200640 1 -- start start 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3814efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 0x7fba3815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba380d26e0 0x7fba380b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380b7a10 con 0x7fba380d00c0 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380b7890 con 0x7fba380d26e0 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba380cf8c0 con 0x7fba380b4930 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 0x7fba3815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 0x7fba3815bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33174/0 (socket says 172.21.15.97:33174) 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 -- 172.21.15.97:0/3071996473 learned_addr learned my addr 172.21.15.97:0/3071996473 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:50.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3de00640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3814efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3e800640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba380d26e0 0x7fba380b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 -- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 msgr2=0x7fba3814efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3814efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 -- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba380d26e0 msgr2=0x7fba380b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba380d26e0 0x7fba380b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3d400640 1 -- 172.21.15.97:0/3071996473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3815d1e0 con 0x7fba380d00c0 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.875+0000 7fba3de00640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3814efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.876+0000 7fba3d400640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 0x7fba3815bda0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fba28000c00 tx=0x7fba280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.876+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28040070 con 0x7fba380d00c0 2024-03-06T07:30:50.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.876+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28039e70 con 0x7fba380d00c0 2024-03-06T07:30:50.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.876+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28035040 con 0x7fba380d00c0 2024-03-06T07:30:50.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.876+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3815d4c0 con 0x7fba380d00c0 2024-03-06T07:30:50.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.877+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3815da80 con 0x7fba380d00c0 2024-03-06T07:30:50.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.878+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3804f330 con 0x7fba380d00c0 2024-03-06T07:30:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.879+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba2804b060 con 0x7fba380d00c0 2024-03-06T07:30:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.879+0000 7fba32000640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba1c06b4d0 0x7fba1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.880+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba280abed0 con 0x7fba380d00c0 2024-03-06T07:30:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.880+0000 7fba3de00640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba1c06b4d0 0x7fba1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.880+0000 7fba3de00640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba1c06b4d0 0x7fba1c06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fba380cdc90 tx=0x7fba2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:50.888+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba2804a2c0 con 0x7fba380d00c0 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.083+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba380cfea0 con 0x7fba380d00c0 2024-03-06T07:30:51.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.086+0000 7fba32000640 1 -- 172.21.15.97:0/3071996473 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fba2807ad50 con 0x7fba380d00c0 2024-03-06T07:30:51.089 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:51.089 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":58,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:21.268327+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:51.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba1c06b4d0 msgr2=0x7fba1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba1c06b4d0 0x7fba1c06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fba380cdc90 tx=0x7fba2c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 msgr2=0x7fba3815bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 0x7fba3815bda0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fba28000c00 tx=0x7fba280046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 shutdown_connections 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba1c06b4d0 0x7fba1c06d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba380d26e0 0x7fba380b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba380d00c0 0x7fba3815bda0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 --2- 172.21.15.97:0/3071996473 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba380b4930 0x7fba3814efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 >> 172.21.15.97:0/3071996473 conn(0x7fba380d1ae0 msgr2=0x7fba38074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 shutdown_connections 2024-03-06T07:30:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.089+0000 7fba3f200640 1 -- 172.21.15.97:0/3071996473 wait complete. 2024-03-06T07:30:51.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:17.521656+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:18.729778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:51.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.321+0000 7f3fed600640 1 Processor -- start 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.321+0000 7f3fed600640 1 -- start start 2024-03-06T07:30:51.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.323+0000 7f3fed600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 0x7f3fe80cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.323+0000 7f3fed600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80d1fb0 con 0x7f3fe80d00d0 2024-03-06T07:30:51.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.323+0000 7f3fed600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe809b100 con 0x7f3fe815df80 2024-03-06T07:30:51.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.323+0000 7f3fed600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80d2a20 con 0x7f3fe815e360 2024-03-06T07:30:51.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.323+0000 7f3fe7400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3fe80d00d0 0x7f3fe806ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51016/0 (socket says 172.21.15.97:51016) 2024-03-06T07:30:51.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.323+0000 7f3fe7400640 1 -- 172.21.15.97:0/501665716 learned_addr learned my addr 172.21.15.97:0/501665716 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fde000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 0x7f3fe80cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372260688 0 0) 0x7f3fe80d1fb0 con 0x7f3fe80d00d0 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fcc003520 con 0x7f3fe80d00d0 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627823105 0 0) 0x7f3fcc003520 con 0x7f3fe80d00d0 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 msgr2=0x7f3fe80cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 0x7f3fe80cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 >> v1:172.21.15.135:6789/0 conn(0x7f3fe815df80 legacy=0x7f3fe806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe809b3a0 con 0x7f3fe80d00d0 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3fe00019f0 con 0x7f3fe80d00d0 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 >> v1:172.21.15.97:6789/0 conn(0x7f3fe80d00d0 legacy=0x7f3fe806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 0x7f3fcc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fcc003f20 0x7f3fcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc004d40 0x7f3fcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc00b9f0 con 0x7f3fcc003f20 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc00bba0 con 0x7f3fcc004d40 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.324+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc00bda0 con 0x7f3fe815e360 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fde000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fcc003f20 0x7f3fcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fe6a00640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 0x7f3fcc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fe7400640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc004d40 0x7f3fcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fde000640 1 -- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 msgr2=0x7f3fcc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fde000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe815e360 0x7f3fcc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fde000640 1 -- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc004d40 msgr2=0x7f3fcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fde000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc004d40 0x7f3fcc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.325+0000 7f3fde000640 1 -- 172.21.15.97:0/501665716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc003520 con 0x7f3fcc003f20 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.327+0000 7f3fde000640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fcc003f20 0x7f3fcc004800 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009a70 tx=0x7f3fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da01e8a9fea65a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.327+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0039070 con 0x7f3fcc003f20 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.327+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd002c5c0 con 0x7f3fcc003f20 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.327+0000 7f3fe6000640 1 -- 172.21.15.97:0/501665716 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0009670 con 0x7f3fcc003f20 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fed600640 1 -- 172.21.15.97:0/501665716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fcc003f20 msgr2=0x7f3fcc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fed600640 1 --2- 172.21.15.97:0/501665716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fcc003f20 0x7f3fcc004800 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009a70 tx=0x7f3fcc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fe7400640 1 -- 172.21.15.97:0/501665716 reap_dead start 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fed600640 1 -- 172.21.15.97:0/501665716 shutdown_connections 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fed600640 1 -- 172.21.15.97:0/501665716 >> 172.21.15.97:0/501665716 conn(0x7f3fe80b7320 msgr2=0x7f3fe809a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fed600640 1 -- 172.21.15.97:0/501665716 shutdown_connections 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.328+0000 7f3fed600640 1 -- 172.21.15.97:0/501665716 wait complete. 2024-03-06T07:30:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.330+0000 7f3fed600640 1 Processor -- start 2024-03-06T07:30:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.330+0000 7f3fed600640 1 -- start start 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fed600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc003f20 0x7f3fe815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fed600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 0x7f3fe81484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fed600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe8157740 0x7f3fe8157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fed600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe809b680 con 0x7f3fe8160c00 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fed600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80d2500 con 0x7f3fcc003f20 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fed600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe806de40 con 0x7f3fe8157740 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fde000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 0x7f3fe81484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fde000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 0x7f3fe81484e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33192/0 (socket says 172.21.15.97:33192) 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fde000640 1 -- 172.21.15.97:0/894059333 learned_addr learned my addr 172.21.15.97:0/894059333 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fe7400640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe8157740 0x7f3fe8157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.331+0000 7f3fe6a00640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc003f20 0x7f3fe815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fde000640 1 -- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe8157740 msgr2=0x7f3fe8157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fde000640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe8157740 0x7f3fe8157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fde000640 1 -- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc003f20 msgr2=0x7f3fe815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fde000640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc003f20 0x7f3fe815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fde000640 1 -- 172.21.15.97:0/894059333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe815a990 con 0x7f3fe8160c00 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fe7400640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe8157740 0x7f3fe8157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:51.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fde000640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 0x7f3fe81484e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0037040 tx=0x7f3fd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0044020 con 0x7f3fe8160c00 2024-03-06T07:30:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd003f670 con 0x7f3fe8160c00 2024-03-06T07:30:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0039040 con 0x7f3fe8160c00 2024-03-06T07:30:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.332+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe815ac70 con 0x7f3fe8160c00 2024-03-06T07:30:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.333+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe815e970 con 0x7f3fe8160c00 2024-03-06T07:30:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.334+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe804f330 con 0x7f3fe8160c00 2024-03-06T07:30:51.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.335+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fd0047020 con 0x7f3fe8160c00 2024-03-06T07:30:51.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.335+0000 7f3fdf400640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc406b4d0 0x7f3fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.335+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fd00abf70 con 0x7f3fe8160c00 2024-03-06T07:30:51.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.336+0000 7f3fe6a00640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc406b4d0 0x7f3fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.336+0000 7f3fe6a00640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc406b4d0 0x7f3fc406d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8005540 tx=0x7f3fd8007c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.340+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fd004e070 con 0x7f3fe8160c00 2024-03-06T07:30:51.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.537+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3fe8070440 con 0x7f3fe8160c00 2024-03-06T07:30:51.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.538+0000 7f3fdf400640 1 -- 172.21.15.97:0/894059333 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3fd007ade0 con 0x7f3fe8160c00 2024-03-06T07:30:51.540 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:30:51.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc406b4d0 msgr2=0x7f3fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc406b4d0 0x7f3fc406d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8005540 tx=0x7f3fd8007c50 comp rx=0 tx=0).stop 2024-03-06T07:30:51.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 msgr2=0x7f3fe81484e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 0x7f3fe81484e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0037040 tx=0x7f3fd0004730 comp rx=0 tx=0).stop 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 shutdown_connections 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc406b4d0 0x7f3fc406d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe8157740 0x7f3fe8157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe8160c00 0x7f3fe81484e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 --2- 172.21.15.97:0/894059333 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc003f20 0x7f3fe815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 >> 172.21.15.97:0/894059333 conn(0x7f3fe80b7320 msgr2=0x7f3fe80b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 shutdown_connections 2024-03-06T07:30:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.540+0000 7f3fed600640 1 -- 172.21.15.97:0/894059333 wait complete. 2024-03-06T07:30:51.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.797+0000 7fb68be00640 1 Processor -- start 2024-03-06T07:30:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.797+0000 7fb68be00640 1 -- start start 2024-03-06T07:30:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.799+0000 7fb68be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 0x7fb68c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.799+0000 7fb68be00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c15c1f0 con 0x7fb68c15be10 2024-03-06T07:30:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.799+0000 7fb68be00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c06cac0 con 0x7fb68c0aef20 2024-03-06T07:30:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.799+0000 7fb68be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0cd200 con 0x7fb68c06da40 2024-03-06T07:30:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.799+0000 7fb68aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb68c15be10 0x7fb68c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51022/0 (socket says 172.21.15.97:51022) 2024-03-06T07:30:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.799+0000 7fb68aa00640 1 -- 172.21.15.97:0/2838870739 learned_addr learned my addr 172.21.15.97:0/2838870739 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728945643 0 0) 0x7fb68c15c1f0 con 0x7fb68c15be10 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb670003520 con 0x7fb68c15be10 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb682000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 0x7fb68c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898565023 0 0) 0x7fb670003520 con 0x7fb68c15be10 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 msgr2=0x7fb68c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 0x7fb68c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 >> v1:172.21.15.135:6789/0 conn(0x7fb68c0aef20 legacy=0x7fb68c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c06cda0 con 0x7fb68c15be10 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb67c001670 con 0x7fb68c15be10 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 >> v1:172.21.15.97:6789/0 conn(0x7fb68c15be10 legacy=0x7fb68c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 0x7fb6700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb670004010 0x7fb670004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb670004d40 0x7fb6700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0cd200 con 0x7fb670004010 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000bb80 con 0x7fb670004d40 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.800+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000bd80 con 0x7fb68c06da40 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb682000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb670004010 0x7fb670004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb68aa00640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 0x7fb6700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb68b400640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb670004d40 0x7fb6700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb682000640 1 -- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 msgr2=0x7fb6700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb682000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb68c06da40 0x7fb6700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb682000640 1 -- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb670004d40 msgr2=0x7fb6700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb682000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb670004d40 0x7fb6700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb682000640 1 -- 172.21.15.97:0/2838870739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670003520 con 0x7fb670004010 2024-03-06T07:30:51.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.801+0000 7fb68b400640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb670004d40 0x7fb6700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:51.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.803+0000 7fb682000640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb670004010 0x7fb670004800 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb67400ba30 tx=0x7fb670005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a683c3c78f4bbba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.803+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674036020 con 0x7fb670004010 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.803+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb674032640 con 0x7fb670004010 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.803+0000 7fb68a000640 1 -- 172.21.15.97:0/2838870739 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400b600 con 0x7fb670004010 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.803+0000 7fb68be00640 1 -- 172.21.15.97:0/2838870739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb670004010 msgr2=0x7fb670004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.803+0000 7fb68be00640 1 --2- 172.21.15.97:0/2838870739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb670004010 0x7fb670004800 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb67400ba30 tx=0x7fb670005830 comp rx=0 tx=0).stop 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.804+0000 7fb68b400640 1 -- 172.21.15.97:0/2838870739 reap_dead start 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.804+0000 7fb68be00640 1 -- 172.21.15.97:0/2838870739 shutdown_connections 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.804+0000 7fb68be00640 1 -- 172.21.15.97:0/2838870739 >> 172.21.15.97:0/2838870739 conn(0x7fb68c06afc0 msgr2=0x7fb68c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.804+0000 7fb68be00640 1 -- 172.21.15.97:0/2838870739 shutdown_connections 2024-03-06T07:30:51.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.804+0000 7fb68be00640 1 -- 172.21.15.97:0/2838870739 wait complete. 2024-03-06T07:30:51.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.806+0000 7fb68be00640 1 Processor -- start 2024-03-06T07:30:51.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.807+0000 7fb68be00640 1 -- start start 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.808+0000 7fb68be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb670004010 0x7fb68c155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.808+0000 7fb68be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb68c15e2e0 0x7fb68c154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.808+0000 7fb68be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 0x7fb68c152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.808+0000 7fb68be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0cda90 con 0x7fb68c1545b0 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.808+0000 7fb68be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c06d080 con 0x7fb68c15e2e0 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.808+0000 7fb68be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0ccc20 con 0x7fb670004010 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 0x7fb68c152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 0x7fb68c152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33216/0 (socket says 172.21.15.97:33216) 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 -- 172.21.15.97:0/215797496 learned_addr learned my addr 172.21.15.97:0/215797496 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:51.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb682000640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb68c15e2e0 0x7fb68c154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68aa00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb670004010 0x7fb68c155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 -- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb670004010 msgr2=0x7fb68c155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb670004010 0x7fb68c155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 -- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb68c15e2e0 msgr2=0x7fb68c154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb68c15e2e0 0x7fb68c154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 -- 172.21.15.97:0/215797496 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c1560a0 con 0x7fb68c1545b0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb68b400640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 0x7fb68c152710 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb68c06c000 tx=0x7fb68400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.809+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684006d50 con 0x7fb68c1545b0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.810+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684003970 con 0x7fb68c1545b0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.810+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684003d60 con 0x7fb68c1545b0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.810+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c154ac0 con 0x7fb68c1545b0 2024-03-06T07:30:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.810+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c155080 con 0x7fb68c1545b0 2024-03-06T07:30:51.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.811+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c04f330 con 0x7fb68c1545b0 2024-03-06T07:30:51.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.812+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb68401b020 con 0x7fb68c1545b0 2024-03-06T07:30:51.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.812+0000 7fb683e00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb66806b500 0x7fb66806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.813+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb684089df0 con 0x7fb68c1545b0 2024-03-06T07:30:51.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.813+0000 7fb68aa00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb66806b500 0x7fb66806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.813+0000 7fb68aa00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb66806b500 0x7fb66806d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb67c001eb0 tx=0x7fb67c001dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.818+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb684058bd0 con 0x7fb68c1545b0 2024-03-06T07:30:52.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:51.999+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb68c074a50 con 0x7fb68c1545b0 2024-03-06T07:30:52.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.000+0000 7fb683e00640 1 -- 172.21.15.97:0/215797496 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb6840584d0 con 0x7fb68c1545b0 2024-03-06T07:30:52.002 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.002+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb66806b500 msgr2=0x7fb66806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.002+0000 7fb68be00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb66806b500 0x7fb66806d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb67c001eb0 tx=0x7fb67c001dc0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.002+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 msgr2=0x7fb68c152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.002+0000 7fb68be00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 0x7fb68c152710 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb68c06c000 tx=0x7fb68400dc00 comp rx=0 tx=0).stop 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 shutdown_connections 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb66806b500 0x7fb66806d9c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb68c1545b0 0x7fb68c152710 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb68c15e2e0 0x7fb68c154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 --2- 172.21.15.97:0/215797496 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb670004010 0x7fb68c155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 >> 172.21.15.97:0/215797496 conn(0x7fb68c06afc0 msgr2=0x7fb68c0ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 shutdown_connections 2024-03-06T07:30:52.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.003+0000 7fb68be00640 1 -- 172.21.15.97:0/215797496 wait complete. 2024-03-06T07:30:52.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:52.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.227+0000 7feda2e00640 1 Processor -- start 2024-03-06T07:30:52.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.227+0000 7feda2e00640 1 -- start start 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7feda2e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fed9c06d960 0x7fed9c15f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7feda2e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c06cba0 con 0x7fed9c06e2f0 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7feda2e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c06ce80 con 0x7fed9c0b2c40 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7feda2e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9c15f700 con 0x7fed9c06d960 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7fed93400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fed9c06d960 0x7fed9c15f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7feda0e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fed9c06e2f0 0x7fed9c0cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51024/0 (socket says 172.21.15.97:51024) 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.229+0000 7feda0e00640 1 -- 172.21.15.97:0/2113854058 learned_addr learned my addr 172.21.15.97:0/2113854058 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.230+0000 7fed93400640 1 -- 172.21.15.97:0/2113854058 >> v1:172.21.15.135:6789/0 conn(0x7fed9c0b2c40 legacy=0x7fed9c0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.230+0000 7fed93400640 1 -- 172.21.15.97:0/2113854058 >> v1:172.21.15.97:6789/0 conn(0x7fed9c06e2f0 legacy=0x7fed9c0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.230+0000 7fed93400640 1 -- 172.21.15.97:0/2113854058 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c06d160 con 0x7fed9c06d960 2024-03-06T07:30:52.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.230+0000 7fed9b400640 1 -- 172.21.15.97:0/2113854058 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813126187 0 0) 0x7fed9c06cba0 con 0x7fed9c06e2f0 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.231+0000 7fed93400640 1 --2- 172.21.15.97:0/2113854058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06d960 0x7fed9c15f1c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fed840097a0 tx=0x7fed8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334f7234fc68da40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.231+0000 7fed9b400640 1 -- 172.21.15.97:0/2113854058 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84035020 con 0x7fed9c06d960 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.231+0000 7fed9b400640 1 -- 172.21.15.97:0/2113854058 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed84031830 con 0x7fed9c06d960 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.231+0000 7fed9b400640 1 -- 172.21.15.97:0/2113854058 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84031c00 con 0x7fed9c06d960 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.231+0000 7feda2e00640 1 -- 172.21.15.97:0/2113854058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06d960 msgr2=0x7fed9c15f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.231+0000 7feda2e00640 1 --2- 172.21.15.97:0/2113854058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06d960 0x7fed9c15f1c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fed840097a0 tx=0x7fed8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.232+0000 7feda2e00640 1 -- 172.21.15.97:0/2113854058 shutdown_connections 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.232+0000 7feda2e00640 1 --2- 172.21.15.97:0/2113854058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06d960 0x7fed9c15f1c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.232+0000 7feda2e00640 1 -- 172.21.15.97:0/2113854058 >> 172.21.15.97:0/2113854058 conn(0x7fed9c0b1940 msgr2=0x7fed9c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.232+0000 7feda2e00640 1 -- 172.21.15.97:0/2113854058 shutdown_connections 2024-03-06T07:30:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.232+0000 7feda2e00640 1 -- 172.21.15.97:0/2113854058 wait complete. 2024-03-06T07:30:52.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.233+0000 7feda2e00640 1 Processor -- start 2024-03-06T07:30:52.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.233+0000 7feda2e00640 1 -- start start 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7feda2e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed9c06d960 0x7fed9c149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7feda2e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 0x7fed9c0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7feda2e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed9c0b2c40 0x7fed9c0add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7feda2e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c06d4d0 con 0x7fed9c06e2f0 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7feda2e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c06e990 con 0x7fed9c0b2c40 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7feda2e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c0ce1a0 con 0x7fed9c06d960 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.234+0000 7fed93400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 0x7fed9c0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 0x7fed9c0c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33234/0 (socket says 172.21.15.97:33234) 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 -- 172.21.15.97:0/2795062686 learned_addr learned my addr 172.21.15.97:0/2795062686 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:52.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 -- 172.21.15.97:0/2795062686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed9c06d960 msgr2=0x7fed9c149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed9c06d960 0x7fed9c149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 -- 172.21.15.97:0/2795062686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed9c0b2c40 msgr2=0x7fed9c0add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed9c0b2c40 0x7fed9c0add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 -- 172.21.15.97:0/2795062686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0c9a10 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed93400640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 0x7fed9c0c8fc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fed8402bee0 tx=0x7fed840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.235+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8403ba80 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.236+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8403bc40 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.236+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84045400 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.236+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c0c9cf0 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.236+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c0ca2b0 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.237+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c04f330 con 0x7fed9c06e2f0 2024-03-06T07:30:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.238+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed84038020 con 0x7fed9c06e2f0 2024-03-06T07:30:52.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.238+0000 7fed98c00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7806b4d0 0x7fed7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.238+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed840ac040 con 0x7fed9c06e2f0 2024-03-06T07:30:52.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.239+0000 7fed9be00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7806b4d0 0x7fed7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.239+0000 7fed9be00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7806b4d0 0x7fed7806d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00ab90 tx=0x7fed8c009230 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.243+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed84040070 con 0x7fed9c06e2f0 2024-03-06T07:30:52.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.444+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed9c15d7c0 con 0x7fed9c06e2f0 2024-03-06T07:30:52.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.445+0000 7fed98c00640 1 -- 172.21.15.97:0/2795062686 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed8404c340 con 0x7fed9c06e2f0 2024-03-06T07:30:52.447 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.447+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7806b4d0 msgr2=0x7fed7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.447+0000 7feda2e00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7806b4d0 0x7fed7806d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00ab90 tx=0x7fed8c009230 comp rx=0 tx=0).stop 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.447+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 msgr2=0x7fed9c0c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.447+0000 7feda2e00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 0x7fed9c0c8fc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fed8402bee0 tx=0x7fed840046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.447+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 shutdown_connections 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fed7806b4d0 0x7fed7806d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed9c0b2c40 0x7fed9c0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed9c06e2f0 0x7fed9c0c8fc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 --2- 172.21.15.97:0/2795062686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed9c06d960 0x7fed9c149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 >> 172.21.15.97:0/2795062686 conn(0x7fed9c0b1940 msgr2=0x7fed9c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 shutdown_connections 2024-03-06T07:30:52.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.448+0000 7feda2e00640 1 -- 172.21.15.97:0/2795062686 wait complete. 2024-03-06T07:30:52.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.666+0000 7f9ea7000640 1 Processor -- start 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.666+0000 7f9ea7000640 1 -- start start 2024-03-06T07:30:52.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.667+0000 7f9ea7000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ea00b4680 0x7f9ea006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9ea7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9ea7000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea015b090 con 0x7f9ea015acb0 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9ea7000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea00b2a30 con 0x7f9ea00b4680 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9ea7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00b4360 con 0x7f9ea0072340 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9e97e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ea00b4680 0x7f9ea006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9e97e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ea00b4680 0x7f9ea006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33244/0 (socket says 172.21.15.97:33244) 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9e97e00640 1 -- 172.21.15.97:0/1828661765 learned_addr learned my addr 172.21.15.97:0/1828661765 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9ea5000640 1 --2- 172.21.15.97:0/1828661765 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.668+0000 7f9e97e00640 1 -- 172.21.15.97:0/1828661765 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 msgr2=0x7f9ea006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.669+0000 7f9e97e00640 1 --2- 172.21.15.97:0/1828661765 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea006ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.669+0000 7f9e97e00640 1 -- 172.21.15.97:0/1828661765 >> v1:172.21.15.135:6789/0 conn(0x7f9ea015acb0 legacy=0x7f9ea00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.669+0000 7f9e97e00640 1 -- 172.21.15.97:0/1828661765 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea00b2cf0 con 0x7f9ea00b4680 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.670+0000 7f9e97e00640 1 --2- 172.21.15.97:0/1828661765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea00b4680 0x7f9ea006e4f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009870 tx=0x7f9e8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73588ff5a972179 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9e9f400640 1 -- 172.21.15.97:0/1828661765 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88035020 con 0x7f9ea00b4680 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9e9f400640 1 -- 172.21.15.97:0/1828661765 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e88031640 con 0x7f9ea00b4680 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9e9f400640 1 -- 172.21.15.97:0/1828661765 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8802d040 con 0x7f9ea00b4680 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9ea7000640 1 -- 172.21.15.97:0/1828661765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea00b4680 msgr2=0x7f9ea006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9ea7000640 1 --2- 172.21.15.97:0/1828661765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea00b4680 0x7f9ea006e4f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009870 tx=0x7f9e8802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9ea7000640 1 -- 172.21.15.97:0/1828661765 shutdown_connections 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9ea7000640 1 --2- 172.21.15.97:0/1828661765 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea006ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9ea7000640 1 --2- 172.21.15.97:0/1828661765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea00b4680 0x7f9ea006e4f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.671+0000 7f9ea7000640 1 -- 172.21.15.97:0/1828661765 >> 172.21.15.97:0/1828661765 conn(0x7f9ea00b1730 msgr2=0x7f9ea00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.672+0000 7f9ea7000640 1 -- 172.21.15.97:0/1828661765 shutdown_connections 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.672+0000 7f9ea7000640 1 -- 172.21.15.97:0/1828661765 wait complete. 2024-03-06T07:30:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.674+0000 7f9ea7000640 1 Processor -- start 2024-03-06T07:30:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.674+0000 7f9ea7000640 1 -- start start 2024-03-06T07:30:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea014deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea7000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ea00b4680 0x7f9ea0159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea7000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 0x7f9ea006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea7000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00b4010 con 0x7f9ea015acb0 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea7000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00b32b0 con 0x7f9ea00b4680 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00ccfb0 con 0x7f9ea0072340 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 0x7f9ea006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 0x7f9ea006db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33254/0 (socket says 172.21.15.97:33254) 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.676+0000 7f9ea5000640 1 -- 172.21.15.97:0/3901772909 learned_addr learned my addr 172.21.15.97:0/3901772909 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e9fe00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea014deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e97e00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ea00b4680 0x7f9ea0159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea5000640 1 -- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 msgr2=0x7f9ea014deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea5000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea014deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea5000640 1 -- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ea00b4680 msgr2=0x7f9ea0159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea5000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ea00b4680 0x7f9ea0159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea5000640 1 -- 172.21.15.97:0/3901772909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea00ce7d0 con 0x7f9ea015acb0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e9fe00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e97e00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ea00b4680 0x7f9ea0159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea5000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 0x7f9ea006db30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9e98007e40 tx=0x7f9e9800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98014620 con 0x7f9ea015acb0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98019070 con 0x7f9ea015acb0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98014a20 con 0x7f9ea015acb0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.677+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea00ceab0 con 0x7f9ea015acb0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.678+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea00cf070 con 0x7f9ea015acb0 2024-03-06T07:30:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.679+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea004f330 con 0x7f9ea015acb0 2024-03-06T07:30:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.680+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e98014bc0 con 0x7f9ea015acb0 2024-03-06T07:30:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.680+0000 7f9e9cc00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.680+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e9808aad0 con 0x7f9ea015acb0 2024-03-06T07:30:52.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.681+0000 7f9e9fe00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.681+0000 7f9e9fe00640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000a830 tx=0x7f9e90008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.685+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e98059950 con 0x7f9ea015acb0 2024-03-06T07:30:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.916+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea015ea40 con 0x7f9ea015acb0 2024-03-06T07:30:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.917+0000 7f9e9cc00640 1 -- 172.21.15.97:0/3901772909 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f9e98059250 con 0x7f9ea015acb0 2024-03-06T07:30:52.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:52.920 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.920+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e7c06b4d0 msgr2=0x7f9e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.920+0000 7f9ea7000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000a830 tx=0x7f9e90008040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.920+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 msgr2=0x7f9ea006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.920+0000 7f9ea7000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 0x7f9ea006db30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9e98007e40 tx=0x7f9e9800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 shutdown_connections 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ea015acb0 0x7f9ea006db30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ea00b4680 0x7f9ea0159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 --2- 172.21.15.97:0/3901772909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ea0072340 0x7f9ea014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 >> 172.21.15.97:0/3901772909 conn(0x7f9ea00b1730 msgr2=0x7f9ea0156540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 shutdown_connections 2024-03-06T07:30:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:52.921+0000 7f9ea7000640 1 -- 172.21.15.97:0/3901772909 wait complete. 2024-03-06T07:30:52.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:53.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.151+0000 7f06a9600640 1 Processor -- start 2024-03-06T07:30:53.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.152+0000 7f06a9600640 1 -- start start 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f06a40d0ca0 0x7f06a40cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a9600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f06a4160530 0x7f06a406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a9600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a415d4f0 con 0x7f06a406b500 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a9600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a406b950 con 0x7f06a40d0ca0 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a9600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a40b50a0 con 0x7f06a4160530 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a2a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f06a40d0ca0 0x7f06a40cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a3400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f06a406b500 0x7f06a40b5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51040/0 (socket says 172.21.15.97:51040) 2024-03-06T07:30:53.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.153+0000 7f06a3400640 1 -- 172.21.15.97:0/3158441389 learned_addr learned my addr 172.21.15.97:0/3158441389 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:53.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f069be00640 1 --2- 172.21.15.97:0/3158441389 >> v2:172.21.15.135:3300/0 conn(0x7f06a4160530 0x7f06a406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f06a2000640 1 -- 172.21.15.97:0/3158441389 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585461433 0 0) 0x7f06a415d4f0 con 0x7f06a406b500 2024-03-06T07:30:53.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f06a2000640 1 -- 172.21.15.97:0/3158441389 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0684003520 con 0x7f06a406b500 2024-03-06T07:30:53.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f06a2a00640 1 -- 172.21.15.97:0/3158441389 >> v2:172.21.15.135:3300/0 conn(0x7f06a4160530 msgr2=0x7f06a406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f06a2a00640 1 --2- 172.21.15.97:0/3158441389 >> v2:172.21.15.135:3300/0 conn(0x7f06a4160530 0x7f06a406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f06a2a00640 1 -- 172.21.15.97:0/3158441389 >> v1:172.21.15.97:6789/0 conn(0x7f06a406b500 legacy=0x7f06a40b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f06a2a00640 1 -- 172.21.15.97:0/3158441389 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a415cc50 con 0x7f06a40d0ca0 2024-03-06T07:30:53.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.154+0000 7f069be00640 1 --2- 172.21.15.97:0/3158441389 >> v2:172.21.15.135:3300/0 conn(0x7f06a4160530 0x7f06a406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.155+0000 7f06a2a00640 1 --2- 172.21.15.97:0/3158441389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a40d0ca0 0x7f06a40cfe50 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f06a40723a0 tx=0x7f069402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a5d189e42aeb19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a2000640 1 -- 172.21.15.97:0/3158441389 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06940315a0 con 0x7f06a40d0ca0 2024-03-06T07:30:53.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a2000640 1 -- 172.21.15.97:0/3158441389 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0694037070 con 0x7f06a40d0ca0 2024-03-06T07:30:53.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a2000640 1 -- 172.21.15.97:0/3158441389 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694031960 con 0x7f06a40d0ca0 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 -- 172.21.15.97:0/3158441389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a40d0ca0 msgr2=0x7f06a40cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 --2- 172.21.15.97:0/3158441389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a40d0ca0 0x7f06a40cfe50 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f06a40723a0 tx=0x7f069402b840 comp rx=0 tx=0).stop 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 -- 172.21.15.97:0/3158441389 shutdown_connections 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 --2- 172.21.15.97:0/3158441389 >> v2:172.21.15.135:3300/0 conn(0x7f06a4160530 0x7f06a406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 --2- 172.21.15.97:0/3158441389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a40d0ca0 0x7f06a40cfe50 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 -- 172.21.15.97:0/3158441389 >> 172.21.15.97:0/3158441389 conn(0x7f06a40b1e10 msgr2=0x7f06a40b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 -- 172.21.15.97:0/3158441389 shutdown_connections 2024-03-06T07:30:53.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.156+0000 7f06a9600640 1 -- 172.21.15.97:0/3158441389 wait complete. 2024-03-06T07:30:53.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.158+0000 7f06a9600640 1 Processor -- start 2024-03-06T07:30:53.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.158+0000 7f06a9600640 1 -- start start 2024-03-06T07:30:53.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a9600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f06a406b500 0x7f06a415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f06a40d0ca0 0x7f06a4155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 0x7f06a4158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a9600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a4160990 con 0x7f06a4160530 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a9600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a415d210 con 0x7f06a406b500 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a40d1080 con 0x7f06a40d0ca0 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 0x7f06a4158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 0x7f06a4158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33274/0 (socket says 172.21.15.97:33274) 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.159+0000 7f06a3400640 1 -- 172.21.15.97:0/2467847617 learned_addr learned my addr 172.21.15.97:0/2467847617 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a2a00640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f06a406b500 0x7f06a415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f069be00640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f06a40d0ca0 0x7f06a4155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a3400640 1 -- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f06a40d0ca0 msgr2=0x7f06a4155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a3400640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f06a40d0ca0 0x7f06a4155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a3400640 1 -- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f06a406b500 msgr2=0x7f06a415bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a3400640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f06a406b500 0x7f06a415bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a3400640 1 -- 172.21.15.97:0/2467847617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a415ac10 con 0x7f06a4160530 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f069be00640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f06a40d0ca0 0x7f06a4155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a3400640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 0x7f06a4158e40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f069c00ca20 tx=0x7f069c00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c017070 con 0x7f06a4160530 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c00abd0 con 0x7f06a4160530 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a415aef0 con 0x7f06a4160530 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.160+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c012400 con 0x7f06a4160530 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.161+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a415b790 con 0x7f06a4160530 2024-03-06T07:30:53.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.162+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a415b990 con 0x7f06a4160530 2024-03-06T07:30:53.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.163+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f069c01a070 con 0x7f06a4160530 2024-03-06T07:30:53.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.164+0000 7f069aa00640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f068006b4d0 0x7f068006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.164+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f069c089fe0 con 0x7f06a4160530 2024-03-06T07:30:53.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.164+0000 7f06a2a00640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f068006b4d0 0x7f068006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.165+0000 7f06a2a00640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f068006b4d0 0x7f068006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0694036000 tx=0x7f069403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.168+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06a415b990 con 0x7f06a4160530 2024-03-06T07:30:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.362+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06a404f330 con 0x7f06a4160530 2024-03-06T07:30:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.363+0000 7f069aa00640 1 -- 172.21.15.97:0/2467847617 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f069c058dc0 con 0x7f06a4160530 2024-03-06T07:30:53.366 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:53.366 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":58,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:21.268327+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f068006b4d0 msgr2=0x7f068006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f068006b4d0 0x7f068006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0694036000 tx=0x7f069403d040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 msgr2=0x7f06a4158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 0x7f06a4158e40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f069c00ca20 tx=0x7f069c00cee0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 shutdown_connections 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f068006b4d0 0x7f068006d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f06a4160530 0x7f06a4158e40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f06a40d0ca0 0x7f06a4155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 --2- 172.21.15.97:0/2467847617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f06a406b500 0x7f06a415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 >> 172.21.15.97:0/2467847617 conn(0x7f06a40b1e10 msgr2=0x7f06a40ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 shutdown_connections 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.366+0000 7f06a9600640 1 -- 172.21.15.97:0/2467847617 wait complete. 2024-03-06T07:30:53.387 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:30:53.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:53.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.569+0000 7f9caa800640 1 Processor -- start 2024-03-06T07:30:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.569+0000 7f9caa800640 1 -- start start 2024-03-06T07:30:53.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9caa800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ca415acb0 0x7f9ca40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9caa800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca406ec00 0x7f9ca40ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9caa800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca415b090 con 0x7f9ca40b4680 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9caa800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca40b2a30 con 0x7f9ca415acb0 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9caa800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b4360 con 0x7f9ca406ec00 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9ca3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ca415acb0 0x7f9ca40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9ca3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ca415acb0 0x7f9ca40b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33288/0 (socket says 172.21.15.97:33288) 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9ca3400640 1 -- 172.21.15.97:0/3675361078 learned_addr learned my addr 172.21.15.97:0/3675361078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.572+0000 7f9ca3e00640 1 --2- 172.21.15.97:0/3675361078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca406ec00 0x7f9ca40ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.573+0000 7f9ca3400640 1 -- 172.21.15.97:0/3675361078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca406ec00 msgr2=0x7f9ca40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.573+0000 7f9ca3400640 1 --2- 172.21.15.97:0/3675361078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca406ec00 0x7f9ca40ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.573+0000 7f9ca3400640 1 -- 172.21.15.97:0/3675361078 >> v1:172.21.15.135:6789/0 conn(0x7f9ca40b4680 legacy=0x7f9ca406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.573+0000 7f9ca3400640 1 -- 172.21.15.97:0/3675361078 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40b2cf0 con 0x7f9ca415acb0 2024-03-06T07:30:53.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.574+0000 7f9ca3400640 1 --2- 172.21.15.97:0/3675361078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca415acb0 0x7f9ca40b37d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009e50 tx=0x7f9c9402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0a4c97962e5684 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.574+0000 7f9ca2a00640 1 -- 172.21.15.97:0/3675361078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9402cb00 con 0x7f9ca415acb0 2024-03-06T07:30:53.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.574+0000 7f9ca2a00640 1 -- 172.21.15.97:0/3675361078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9402cca0 con 0x7f9ca415acb0 2024-03-06T07:30:53.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.574+0000 7f9ca2a00640 1 -- 172.21.15.97:0/3675361078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94034610 con 0x7f9ca415acb0 2024-03-06T07:30:53.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 -- 172.21.15.97:0/3675361078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca415acb0 msgr2=0x7f9ca40b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 --2- 172.21.15.97:0/3675361078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca415acb0 0x7f9ca40b37d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009e50 tx=0x7f9c9402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:53.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 -- 172.21.15.97:0/3675361078 shutdown_connections 2024-03-06T07:30:53.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 --2- 172.21.15.97:0/3675361078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca406ec00 0x7f9ca40ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 --2- 172.21.15.97:0/3675361078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca415acb0 0x7f9ca40b37d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 -- 172.21.15.97:0/3675361078 >> 172.21.15.97:0/3675361078 conn(0x7f9ca40b1730 msgr2=0x7f9ca40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 -- 172.21.15.97:0/3675361078 shutdown_connections 2024-03-06T07:30:53.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.575+0000 7f9caa800640 1 -- 172.21.15.97:0/3675361078 wait complete. 2024-03-06T07:30:53.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.577+0000 7f9caa800640 1 Processor -- start 2024-03-06T07:30:53.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.577+0000 7f9caa800640 1 -- start start 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9caa800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca406ec00 0x7f9ca4154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9caa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 0x7f9ca4144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9caa800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca415acb0 0x7f9ca4155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9caa800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b3d10 con 0x7f9ca40b4680 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9caa800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b32b0 con 0x7f9ca406ec00 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9caa800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b4010 con 0x7f9ca415acb0 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9c9aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 0x7f9ca4144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9c9aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 0x7f9ca4144500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33296/0 (socket says 172.21.15.97:33296) 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.578+0000 7f9c9aa00640 1 -- 172.21.15.97:0/458737092 learned_addr learned my addr 172.21.15.97:0/458737092 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9ca3400640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca406ec00 0x7f9ca4154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9ca3e00640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca415acb0 0x7f9ca4155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9aa00640 1 -- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca415acb0 msgr2=0x7f9ca4155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9aa00640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca415acb0 0x7f9ca4155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9aa00640 1 -- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca406ec00 msgr2=0x7f9ca4154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9aa00640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca406ec00 0x7f9ca4154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9aa00640 1 -- 172.21.15.97:0/458737092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca41531a0 con 0x7f9ca40b4680 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9ca3400640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca406ec00 0x7f9ca4154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9aa00640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 0x7f9ca4144500 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00c430 tx=0x7f9c8c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0052d0 con 0x7f9ca40b4680 2024-03-06T07:30:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8c005490 con 0x7f9ca40b4680 2024-03-06T07:30:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.579+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c007260 con 0x7f9ca40b4680 2024-03-06T07:30:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.580+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca4153480 con 0x7f9ca40b4680 2024-03-06T07:30:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.580+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca4153a40 con 0x7f9ca40b4680 2024-03-06T07:30:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.581+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca40b2430 con 0x7f9ca40b4680 2024-03-06T07:30:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.583+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c8c007400 con 0x7f9ca40b4680 2024-03-06T07:30:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.583+0000 7f9c9be00640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.584+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c8c089700 con 0x7f9ca40b4680 2024-03-06T07:30:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.584+0000 7f9ca3400640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.585+0000 7f9ca3400640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9c94035000 tx=0x7f9c94030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.591+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca40b2430 con 0x7f9ca40b4680 2024-03-06T07:30:53.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.815+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ca404f330 con 0x7f9ca40b4680 2024-03-06T07:30:53.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.816+0000 7f9c9be00640 1 -- 172.21.15.97:0/458737092 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f9c8c058570 con 0x7f9ca40b4680 2024-03-06T07:30:53.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:53.819 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:53.819 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:53.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 msgr2=0x7f9c8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9c94035000 tx=0x7f9c94030000 comp rx=0 tx=0).stop 2024-03-06T07:30:53.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 msgr2=0x7f9ca4144500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 0x7f9ca4144500 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00c430 tx=0x7f9c8c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 shutdown_connections 2024-03-06T07:30:53.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca415acb0 0x7f9ca4155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca40b4680 0x7f9ca4144500 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 --2- 172.21.15.97:0/458737092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca406ec00 0x7f9ca4154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 >> 172.21.15.97:0/458737092 conn(0x7f9ca40b1730 msgr2=0x7f9ca40ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 shutdown_connections 2024-03-06T07:30:53.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:53.819+0000 7f9caa800640 1 -- 172.21.15.97:0/458737092 wait complete. 2024-03-06T07:30:53.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.023+0000 7efc0f200640 1 Processor -- start 2024-03-06T07:30:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.023+0000 7efc0f200640 1 -- start start 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc0f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc1015a940 0x7efc100b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc0f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 0x7efc100cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc0f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc100b2b30 con 0x7efc100cf370 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc0f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc100b28e0 con 0x7efc1015a940 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc0f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b2e10 con 0x7efc100cc2a0 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc07e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efc100cf370 0x7efc1006d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41370/0 (socket says 172.21.15.97:41370) 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc07e00640 1 -- 172.21.15.97:0/3230759348 learned_addr learned my addr 172.21.15.97:0/3230759348 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.025+0000 7efc0e800640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 0x7efc100cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0de00640 1 --2- 172.21.15.97:0/3230759348 >> v2:172.21.15.135:3300/0 conn(0x7efc1015a940 0x7efc100b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843321042 0 0) 0x7efc100b2b30 con 0x7efc100cf370 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf4003520 con 0x7efc100cf370 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1172165501 0 0) 0x7efbf4003520 con 0x7efc100cf370 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 msgr2=0x7efc100cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 0x7efc100cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 >> v2:172.21.15.135:3300/0 conn(0x7efc1015a940 msgr2=0x7efc100b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 --2- 172.21.15.97:0/3230759348 >> v2:172.21.15.135:3300/0 conn(0x7efc1015a940 0x7efc100b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1006ca50 con 0x7efc100cf370 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7efbf8001670 con 0x7efc100cf370 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 >> v1:172.21.15.97:6789/0 conn(0x7efc100cf370 legacy=0x7efc1006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efbf40039b0 0x7efbf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efbf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 0x7efbf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b28e0 con 0x7efbf4004800 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b2e10 con 0x7efbf40039b0 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.026+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf400b9c0 con 0x7efc100cc2a0 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc07e00640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efbf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc0de00640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efbf40039b0 0x7efbf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc0e800640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 0x7efbf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc07e00640 1 -- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 msgr2=0x7efbf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc07e00640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc100cc2a0 0x7efbf40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc07e00640 1 -- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efbf40039b0 msgr2=0x7efbf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc07e00640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efbf40039b0 0x7efbf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.027+0000 7efc07e00640 1 -- 172.21.15.97:0/3230759348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf4003520 con 0x7efbf4004800 2024-03-06T07:30:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc07e00640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efbf4004c80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c240 tx=0x7efbf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf576d1e4c594ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8007d30 con 0x7efbf4004800 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf8030070 con 0x7efbf4004800 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0d400640 1 -- 172.21.15.97:0/3230759348 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf802cd60 con 0x7efbf4004800 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0f200640 1 -- 172.21.15.97:0/3230759348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 msgr2=0x7efbf4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0f200640 1 --2- 172.21.15.97:0/3230759348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efbf4004c80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c240 tx=0x7efbf4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0e800640 1 -- 172.21.15.97:0/3230759348 reap_dead start 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0f200640 1 -- 172.21.15.97:0/3230759348 shutdown_connections 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0f200640 1 -- 172.21.15.97:0/3230759348 >> 172.21.15.97:0/3230759348 conn(0x7efc1006afc0 msgr2=0x7efc1006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.029+0000 7efc0f200640 1 -- 172.21.15.97:0/3230759348 shutdown_connections 2024-03-06T07:30:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.030+0000 7efc0f200640 1 -- 172.21.15.97:0/3230759348 wait complete. 2024-03-06T07:30:54.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.031+0000 7efc0f200640 1 Processor -- start 2024-03-06T07:30:54.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.031+0000 7efc0f200640 1 -- start start 2024-03-06T07:30:54.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efc101514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc1014a620 0x7efc1014fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc10071fb0 0x7efc1013be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b4b20 con 0x7efbf4004800 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b30f0 con 0x7efc1014a620 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1015ad20 con 0x7efc10071fb0 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efc101514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efc101514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42576/0 (socket says 172.21.15.97:42576) 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.032+0000 7efc0de00640 1 -- 172.21.15.97:0/1746633310 learned_addr learned my addr 172.21.15.97:0/1746633310 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc07e00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc1014a620 0x7efc1014fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0e800640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc10071fb0 0x7efc1013be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0de00640 1 -- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc10071fb0 msgr2=0x7efc1013be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0de00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc10071fb0 0x7efc1013be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0de00640 1 -- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc1014a620 msgr2=0x7efc1014fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0de00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc1014a620 0x7efc1014fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0de00640 1 -- 172.21.15.97:0/1746633310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10150410 con 0x7efbf4004800 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc07e00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc1014a620 0x7efc1014fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0de00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efc101514b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7efc00011630 tx=0x7efc00011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000f920 con 0x7efbf4004800 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0000fac0 con 0x7efbf4004800 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000fe70 con 0x7efbf4004800 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.033+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc101506f0 con 0x7efbf4004800 2024-03-06T07:30:54.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.034+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc10150cb0 con 0x7efbf4004800 2024-03-06T07:30:54.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.035+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1004f330 con 0x7efbf4004800 2024-03-06T07:30:54.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.036+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc00015460 con 0x7efbf4004800 2024-03-06T07:30:54.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.037+0000 7efc06000640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efbec06b500 0x7efbec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.037+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc0008aba0 con 0x7efbf4004800 2024-03-06T07:30:54.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.037+0000 7efc07e00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efbec06b500 0x7efbec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.038+0000 7efc07e00640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efbec06b500 0x7efbec06d9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efbf8034000 tx=0x7efbf8007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.041+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc00019030 con 0x7efbf4004800 2024-03-06T07:30:54.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.263+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc100cfcf0 con 0x7efbf4004800 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.265+0000 7efc06000640 1 -- 172.21.15.97:0/1746633310 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7efc00059a20 con 0x7efbf4004800 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:19.710683+0000","modified":"2024-03-06T07:30:22.764725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/24115969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":24115969},{"type":"v1","addr":"172.21.15.97:6837","nonce":24115969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efbec06b500 msgr2=0x7efbec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efbec06b500 0x7efbec06d9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efbf8034000 tx=0x7efbf8007000 comp rx=0 tx=0).stop 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 msgr2=0x7efc101514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efc101514b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7efc00011630 tx=0x7efc00011af0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 shutdown_connections 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efbec06b500 0x7efbec06d9c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efc10071fb0 0x7efc1013be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc1014a620 0x7efc1014fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 --2- 172.21.15.97:0/1746633310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efbf4004800 0x7efc101514b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 >> 172.21.15.97:0/1746633310 conn(0x7efc1006afc0 msgr2=0x7efc100d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 shutdown_connections 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.267+0000 7efc0f200640 1 -- 172.21.15.97:0/1746633310 wait complete. 2024-03-06T07:30:54.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:54.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.478+0000 7fa3c2c00640 1 Processor -- start 2024-03-06T07:30:54.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.478+0000 7fa3c2c00640 1 -- start start 2024-03-06T07:30:54.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.479+0000 7fa3c2c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa3bc15e0d0 0x7fa3bc0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3c2c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc0cea60 con 0x7fa3bc0d0fc0 2024-03-06T07:30:54.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3c2c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc0d2350 con 0x7fa3bc15f470 2024-03-06T07:30:54.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3c2c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3bc0d2f60 con 0x7fa3bc15e0d0 2024-03-06T07:30:54.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3c0c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa3bc0d0fc0 0x7fa3bc160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41372/0 (socket says 172.21.15.97:41372) 2024-03-06T07:30:54.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3c0c00640 1 -- 172.21.15.97:0/520887545 learned_addr learned my addr 172.21.15.97:0/520887545 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:54.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3bbe00640 1 --2- 172.21.15.97:0/520887545 >> v2:172.21.15.135:3300/0 conn(0x7fa3bc15e0d0 0x7fa3bc0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.480+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284076898 0 0) 0x7fa3bc0cea60 con 0x7fa3bc0d0fc0 2024-03-06T07:30:54.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a0003520 con 0x7fa3bc0d0fc0 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 155878607 0 0) 0x7fa3a0003520 con 0x7fa3bc0d0fc0 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 >> v2:172.21.15.135:3300/0 conn(0x7fa3bc15e0d0 msgr2=0x7fa3bc0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 --2- 172.21.15.97:0/520887545 >> v2:172.21.15.135:3300/0 conn(0x7fa3bc15e0d0 0x7fa3bc0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 >> v1:172.21.15.135:6789/0 conn(0x7fa3bc15f470 legacy=0x7fa3bc0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc0d2630 con 0x7fa3bc0d0fc0 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa3b40019f0 con 0x7fa3bc0d0fc0 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 >> v1:172.21.15.97:6789/0 conn(0x7fa3bc0d0fc0 legacy=0x7fa3bc160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3a00039b0 0x7fa3a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3a00051c0 0x7fa3a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a000c040 con 0x7fa3a0004800 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a000c1f0 con 0x7fa3a00051c0 2024-03-06T07:30:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a000c3f0 con 0x7fa3a00039b0 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.481+0000 7fa3bb400640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3bbe00640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3a00039b0 0x7fa3a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3bb400640 1 -- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3a00039b0 msgr2=0x7fa3a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3bb400640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3a00039b0 0x7fa3a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3bb400640 1 -- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3a00051c0 msgr2=0x7fa3a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3bb400640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3a00051c0 0x7fa3a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3bb400640 1 -- 172.21.15.97:0/520887545 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0003520 con 0x7fa3a0004800 2024-03-06T07:30:54.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.482+0000 7fa3c0c00640 1 -- 172.21.15.97:0/520887545 reap_dead start 2024-03-06T07:30:54.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.483+0000 7fa3bb400640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8001720 tx=0x7fa3a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9447a147846823e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.483+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8001370 con 0x7fa3a0004800 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.483+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a80052d0 con 0x7fa3a0004800 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.483+0000 7fa3baa00640 1 -- 172.21.15.97:0/520887545 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8003830 con 0x7fa3a0004800 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 -- 172.21.15.97:0/520887545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 msgr2=0x7fa3a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8001720 tx=0x7fa3a0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 -- 172.21.15.97:0/520887545 shutdown_connections 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/520887545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 -- 172.21.15.97:0/520887545 >> 172.21.15.97:0/520887545 conn(0x7fa3bc0d3ec0 msgr2=0x7fa3bc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 -- 172.21.15.97:0/520887545 shutdown_connections 2024-03-06T07:30:54.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.484+0000 7fa3c2c00640 1 -- 172.21.15.97:0/520887545 wait complete. 2024-03-06T07:30:54.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.486+0000 7fa3c2c00640 1 Processor -- start 2024-03-06T07:30:54.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.486+0000 7fa3c2c00640 1 -- start start 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3c2c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3bc160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3c2c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc0ca080 0x7fa3bc0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3c2c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3bc152660 0x7fa3bc0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3c2c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc15fb30 con 0x7fa3a0004800 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3c2c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc0d2bf0 con 0x7fa3bc0ca080 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3c2c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc15e4b0 con 0x7fa3bc152660 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3bc160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3bc160fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42592/0 (socket says 172.21.15.97:42592) 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 -- 172.21.15.97:0/3149621872 learned_addr learned my addr 172.21.15.97:0/3149621872 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 -- 172.21.15.97:0/3149621872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3bc152660 msgr2=0x7fa3bc0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3bc152660 0x7fa3bc0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 -- 172.21.15.97:0/3149621872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc0ca080 msgr2=0x7fa3bc0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc0ca080 0x7fa3bc0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.487+0000 7fa3bbe00640 1 -- 172.21.15.97:0/3149621872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc0c8d40 con 0x7fa3a0004800 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.488+0000 7fa3bbe00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3bc160fd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0114e0 tx=0x7fa3ac0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.488+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac015870 con 0x7fa3a0004800 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.488+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac015a10 con 0x7fa3a0004800 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.488+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac015dc0 con 0x7fa3a0004800 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.488+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3bc0c8fe0 con 0x7fa3a0004800 2024-03-06T07:30:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.488+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc0c95a0 con 0x7fa3a0004800 2024-03-06T07:30:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.489+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc04f330 con 0x7fa3a0004800 2024-03-06T07:30:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.490+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3ac013400 con 0x7fa3a0004800 2024-03-06T07:30:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.491+0000 7fa3b3e00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa39c06b4d0 0x7fa39c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.491+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3ac08a750 con 0x7fa3a0004800 2024-03-06T07:30:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.492+0000 7fa3bb400640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa39c06b4d0 0x7fa39c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.492+0000 7fa3bb400640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa39c06b4d0 0x7fa39c06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8033000 tx=0x7fa3a802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.495+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3ac019030 con 0x7fa3a0004800 2024-03-06T07:30:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.694+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3bc08d920 con 0x7fa3a0004800 2024-03-06T07:30:54.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.696+0000 7fa3b3e00640 1 -- 172.21.15.97:0/3149621872 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fa3ac0595d0 con 0x7fa3a0004800 2024-03-06T07:30:54.698 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:30:54.699 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":58,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:21.268327+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:17.521656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:18.729778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa39c06b4d0 msgr2=0x7fa39c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa39c06b4d0 0x7fa39c06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8033000 tx=0x7fa3a802f000 comp rx=0 tx=0).stop 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 msgr2=0x7fa3bc160fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3bc160fd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0114e0 tx=0x7fa3ac0119a0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 shutdown_connections 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa39c06b4d0 0x7fa39c06d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3bc152660 0x7fa3bc0b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc0ca080 0x7fa3bc0c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 --2- 172.21.15.97:0/3149621872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3a0004800 0x7fa3bc160fd0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 >> 172.21.15.97:0/3149621872 conn(0x7fa3bc0d3ec0 msgr2=0x7fa3bc0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 shutdown_connections 2024-03-06T07:30:54.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.699+0000 7fa3c2c00640 1 -- 172.21.15.97:0/3149621872 wait complete. 2024-03-06T07:30:54.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:30:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.908+0000 7fd556200640 1 Processor -- start 2024-03-06T07:30:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.908+0000 7fd556200640 1 -- start start 2024-03-06T07:30:54.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd556200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd55006afc0 0x7fd55006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd556200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd5500ced20 0x7fd5500cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd556200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55006b8e0 con 0x7fd5500cacb0 2024-03-06T07:30:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd556200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5500cd320 con 0x7fd5500ced20 2024-03-06T07:30:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd556200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5500cca80 con 0x7fd55006afc0 2024-03-06T07:30:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd54fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd5500ced20 0x7fd5500cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd54fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd5500ced20 0x7fd5500cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42608/0 (socket says 172.21.15.97:42608) 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd54ea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd5500cacb0 0x7fd5500cc080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41378/0 (socket says 172.21.15.97:41378) 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.910+0000 7fd54fe00640 1 -- 172.21.15.97:0/845674772 learned_addr learned my addr 172.21.15.97:0/845674772 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.911+0000 7fd54f400640 1 --2- 172.21.15.97:0/845674772 >> v2:172.21.15.135:3300/0 conn(0x7fd55006afc0 0x7fd55006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.911+0000 7fd54fe00640 1 -- 172.21.15.97:0/845674772 >> v2:172.21.15.135:3300/0 conn(0x7fd55006afc0 msgr2=0x7fd55006b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.911+0000 7fd54fe00640 1 --2- 172.21.15.97:0/845674772 >> v2:172.21.15.135:3300/0 conn(0x7fd55006afc0 0x7fd55006b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.911+0000 7fd54fe00640 1 -- 172.21.15.97:0/845674772 >> v1:172.21.15.97:6789/0 conn(0x7fd5500cacb0 legacy=0x7fd5500cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.911+0000 7fd54fe00640 1 -- 172.21.15.97:0/845674772 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500ccd60 con 0x7fd5500ced20 2024-03-06T07:30:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.911+0000 7fd54e000640 1 -- 172.21.15.97:0/845674772 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711677146 0 0) 0x7fd55006b8e0 con 0x7fd5500cacb0 2024-03-06T07:30:54.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.912+0000 7fd54fe00640 1 --2- 172.21.15.97:0/845674772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5500cf1d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd5500676a0 tx=0x7fd54802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783a014a6f6f429a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.912+0000 7fd54e000640 1 -- 172.21.15.97:0/845674772 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54802ca80 con 0x7fd5500ced20 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.912+0000 7fd54e000640 1 -- 172.21.15.97:0/845674772 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54802cc20 con 0x7fd5500ced20 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.912+0000 7fd54e000640 1 -- 172.21.15.97:0/845674772 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5480343f0 con 0x7fd5500ced20 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 -- 172.21.15.97:0/845674772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 msgr2=0x7fd5500cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 --2- 172.21.15.97:0/845674772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5500cf1d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd5500676a0 tx=0x7fd54802bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 -- 172.21.15.97:0/845674772 shutdown_connections 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 --2- 172.21.15.97:0/845674772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5500cf1d0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 --2- 172.21.15.97:0/845674772 >> v2:172.21.15.135:3300/0 conn(0x7fd55006afc0 0x7fd55006b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 -- 172.21.15.97:0/845674772 >> 172.21.15.97:0/845674772 conn(0x7fd5500d0b90 msgr2=0x7fd5500d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 -- 172.21.15.97:0/845674772 shutdown_connections 2024-03-06T07:30:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.913+0000 7fd556200640 1 -- 172.21.15.97:0/845674772 wait complete. 2024-03-06T07:30:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.915+0000 7fd556200640 1 Processor -- start 2024-03-06T07:30:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.915+0000 7fd556200640 1 -- start start 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd556200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd55006afc0 0x7fd55015e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd556200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5500cacb0 0x7fd55015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd556200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5501f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd556200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550144980 con 0x7fd5500ced20 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd556200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5500cc790 con 0x7fd5500cacb0 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd556200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd550144b00 con 0x7fd55006afc0 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd54fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5501f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd54fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5501f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42612/0 (socket says 172.21.15.97:42612) 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd54fe00640 1 -- 172.21.15.97:0/1747334310 learned_addr learned my addr 172.21.15.97:0/1747334310 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd54ea00640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5500cacb0 0x7fd55015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.916+0000 7fd54f400640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd55006afc0 0x7fd55015e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd54fe00640 1 -- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd55006afc0 msgr2=0x7fd55015e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd54fe00640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd55006afc0 0x7fd55015e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd54fe00640 1 -- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5500cacb0 msgr2=0x7fd55015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd54fe00640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5500cacb0 0x7fd55015d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd54fe00640 1 -- 172.21.15.97:0/1747334310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55015ebd0 con 0x7fd5500ced20 2024-03-06T07:30:54.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd54fe00640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5501f7b80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd548036040 tx=0x7fd548004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548043020 con 0x7fd5500ced20 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54803e450 con 0x7fd5500ced20 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55015eeb0 con 0x7fd5500ced20 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548009510 con 0x7fd5500ced20 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.917+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55015f470 con 0x7fd5500ced20 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.919+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55004f330 con 0x7fd5500ced20 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.919+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd548046020 con 0x7fd5500ced20 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.920+0000 7fd547400640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd53006b4d0 0x7fd53006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.920+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5480ac9a0 con 0x7fd5500ced20 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.920+0000 7fd54f400640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd53006b4d0 0x7fd53006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.921+0000 7fd54f400640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd53006b4d0 0x7fd53006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd540006fd0 tx=0x7fd540008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:54.925+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd54804d030 con 0x7fd5500ced20 2024-03-06T07:30:55.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.159+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd550142700 con 0x7fd5500ced20 2024-03-06T07:30:55.162 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:30:55.160+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:30:55.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.163+0000 7fd547400640 1 -- 172.21.15.97:0/1747334310 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd54807b810 con 0x7fd5500ced20 2024-03-06T07:30:55.166 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:30:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.167+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd53006b4d0 msgr2=0x7fd53006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.167+0000 7fd556200640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd53006b4d0 0x7fd53006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd540006fd0 tx=0x7fd540008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.167+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 msgr2=0x7fd5501f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.167+0000 7fd556200640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5501f7b80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd548036040 tx=0x7fd548004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 shutdown_connections 2024-03-06T07:30:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd53006b4d0 0x7fd53006d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5500ced20 0x7fd5501f7b80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5500cacb0 0x7fd55015d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 --2- 172.21.15.97:0/1747334310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd55006afc0 0x7fd55015e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 >> 172.21.15.97:0/1747334310 conn(0x7fd5500d0b90 msgr2=0x7fd5500b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 shutdown_connections 2024-03-06T07:30:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.168+0000 7fd556200640 1 -- 172.21.15.97:0/1747334310 wait complete. 2024-03-06T07:30:55.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:30:55.279 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.380+0000 7f2f70a00640 1 Processor -- start 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.380+0000 7f2f70a00640 1 -- start start 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.381+0000 7f2f70a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2f6c0b3df0 0x7f2f6c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.381+0000 7f2f70a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c0719e0 con 0x7f2f6c0b3650 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.381+0000 7f2f70a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c071cc0 con 0x7f2f6c0cda90 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.381+0000 7f2f70a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c072310 con 0x7f2f6c0b3df0 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.381+0000 7f2f69400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2f6c0b3650 0x7f2f6c0b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41392/0 (socket says 172.21.15.97:41392) 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.381+0000 7f2f69400640 1 -- 172.21.15.97:0/3846835358 learned_addr learned my addr 172.21.15.97:0/3846835358 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291868084 0 0) 0x7f2f6c0719e0 con 0x7f2f6c0b3650 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f69e00640 1 --2- 172.21.15.97:0/3846835358 >> v2:172.21.15.135:3300/0 conn(0x7f2f6c0b3df0 0x7f2f6c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f50003520 con 0x7f2f6c0b3650 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3301169458 0 0) 0x7f2f50003520 con 0x7f2f6c0b3650 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 >> v2:172.21.15.135:3300/0 conn(0x7f2f6c0b3df0 msgr2=0x7f2f6c0727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 --2- 172.21.15.97:0/3846835358 >> v2:172.21.15.135:3300/0 conn(0x7f2f6c0b3df0 0x7f2f6c0727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 >> v1:172.21.15.135:6789/0 conn(0x7f2f6c0cda90 legacy=0x7f2f6c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c071fa0 con 0x7f2f6c0b3650 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2f58001670 con 0x7f2f6c0b3650 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 >> v1:172.21.15.97:6789/0 conn(0x7f2f6c0b3650 legacy=0x7f2f6c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.382+0000 7f2f68a00640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f50003a00 0x7f2f50003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f68a00640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f68a00640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f500051c0 0x7f2f5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c072310 con 0x7f2f50004800 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5000c0c0 con 0x7f2f50003a00 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5000c2c0 con 0x7f2f500051c0 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69400640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f6a800640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f500051c0 0x7f2f5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69400640 1 -- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f500051c0 msgr2=0x7f2f5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69e00640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f50003a00 0x7f2f50003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69400640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f500051c0 0x7f2f5000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69400640 1 -- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f50003a00 msgr2=0x7f2f50003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69400640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f50003a00 0x7f2f50003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f69400640 1 -- 172.21.15.97:0/3846835358 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50003520 con 0x7f2f50004800 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f6a800640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f500051c0 0x7f2f5000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.383+0000 7f2f6a800640 1 -- 172.21.15.97:0/3846835358 reap_dead start 2024-03-06T07:30:55.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.384+0000 7f2f69400640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c190 tx=0x7f2f50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3596f397bc84e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58007c60 con 0x7f2f50004800 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f58030070 con 0x7f2f50004800 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f68a00640 1 -- 172.21.15.97:0/3846835358 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005db0 con 0x7f2f50004800 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f70a00640 1 -- 172.21.15.97:0/3846835358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 msgr2=0x7f2f50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c190 tx=0x7f2f50005830 comp rx=0 tx=0).stop 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f70a00640 1 -- 172.21.15.97:0/3846835358 shutdown_connections 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3846835358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.385+0000 7f2f70a00640 1 -- 172.21.15.97:0/3846835358 >> 172.21.15.97:0/3846835358 conn(0x7f2f6c0b1940 msgr2=0x7f2f6c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.386+0000 7f2f70a00640 1 -- 172.21.15.97:0/3846835358 shutdown_connections 2024-03-06T07:30:55.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.386+0000 7f2f70a00640 1 -- 172.21.15.97:0/3846835358 wait complete. 2024-03-06T07:30:55.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.387+0000 7f2f70a00640 1 Processor -- start 2024-03-06T07:30:55.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.387+0000 7f2f70a00640 1 -- start start 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.388+0000 7f2f70a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c14c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.388+0000 7f2f70a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1519f0 0x7f2f6c0b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.388+0000 7f2f70a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 0x7f2f6c0b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.388+0000 7f2f70a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c15b3c0 con 0x7f2f6c151e70 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.388+0000 7f2f70a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c15b240 con 0x7f2f6c1519f0 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.388+0000 7f2f70a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0cf510 con 0x7f2f50004800 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 0x7f2f6c0b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 0x7f2f6c0b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42644/0 (socket says 172.21.15.97:42644) 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 -- 172.21.15.97:0/3439464250 learned_addr learned my addr 172.21.15.97:0/3439464250 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f69e00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c14c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f69400640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1519f0 0x7f2f6c0b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 -- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 msgr2=0x7f2f6c14c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c14c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 -- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1519f0 msgr2=0x7f2f6c0b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1519f0 0x7f2f6c0b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 -- 172.21.15.97:0/3439464250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c1e60d0 con 0x7f2f6c151e70 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f69400640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1519f0 0x7f2f6c0b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.389+0000 7f2f6a800640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 0x7f2f6c0b2f40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0708e0 tx=0x7f2f6400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.390+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64018070 con 0x7f2f6c151e70 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.390+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64003da0 con 0x7f2f6c151e70 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.390+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640052b0 con 0x7f2f6c151e70 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.390+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c1f77c0 con 0x7f2f6c151e70 2024-03-06T07:30:55.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.390+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c1f7d80 con 0x7f2f6c151e70 2024-03-06T07:30:55.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.391+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c04f330 con 0x7f2f6c151e70 2024-03-06T07:30:55.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.392+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f6401b070 con 0x7f2f6c151e70 2024-03-06T07:30:55.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.393+0000 7f2f62000640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4c06b500 0x7f2f4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.393+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2f64089960 con 0x7f2f6c151e70 2024-03-06T07:30:55.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.393+0000 7f2f69e00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4c06b500 0x7f2f4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.394+0000 7f2f69e00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4c06b500 0x7f2f4c06d9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c09d600 tx=0x7f2f5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.397+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f640586e0 con 0x7f2f6c151e70 2024-03-06T07:30:55.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:55.634+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2f6c072e70 con 0x7f2f6c151e70 2024-03-06T07:30:56.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.167+0000 7f2f62000640 1 -- 172.21.15.97:0/3439464250 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2f64089070 con 0x7f2f6c151e70 2024-03-06T07:30:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4c06b500 msgr2=0x7f2f4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4c06b500 0x7f2f4c06d9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c09d600 tx=0x7f2f5c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 msgr2=0x7f2f6c0b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 0x7f2f6c0b2f40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0708e0 tx=0x7f2f6400d9d0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 shutdown_connections 2024-03-06T07:30:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4c06b500 0x7f2f4c06d9c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c151e70 0x7f2f6c0b2f40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1519f0 0x7f2f6c0b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 --2- 172.21.15.97:0/3439464250 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c14c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 >> 172.21.15.97:0/3439464250 conn(0x7f2f6c0b1940 msgr2=0x7f2f6c0b03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 shutdown_connections 2024-03-06T07:30:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.171+0000 7f2f70a00640 1 -- 172.21.15.97:0/3439464250 wait complete. 2024-03-06T07:30:56.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.406+0000 7f9fd1200640 1 Processor -- start 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.406+0000 7f9fd1200640 1 -- start start 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 0x7f9fcc166ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fd1200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc072970 con 0x7f9fcc145750 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fd1200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc0cdbd0 con 0x7f9fcc0cd450 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fd1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0cd980 con 0x7f9fcc06dd30 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fc9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9fcc145750 0x7f9fcc072260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41400/0 (socket says 172.21.15.97:41400) 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fc9e00640 1 -- 172.21.15.97:0/2495122786 learned_addr learned my addr 172.21.15.97:0/2495122786 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234766211 0 0) 0x7f9fcc072970 con 0x7f9fcc145750 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.408+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb0003520 con 0x7f9fcc145750 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fca800640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 0x7f9fcc166ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2511625313 0 0) 0x7f9fb0003520 con 0x7f9fcc145750 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 msgr2=0x7f9fcc166ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 0x7f9fcc166ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 >> v1:172.21.15.135:6789/0 conn(0x7f9fcc0cd450 legacy=0x7f9fcc06d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc0cdeb0 con 0x7f9fcc145750 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9fbc001670 con 0x7f9fcc145750 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 >> v1:172.21.15.97:6789/0 conn(0x7f9fcc145750 legacy=0x7f9fcc072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb0003a50 0x7f9fb0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb0004800 0x7f9fb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 0x7f9fb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0cd980 con 0x7f9fb0004800 2024-03-06T07:30:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0005630 con 0x7f9fb0003a50 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb000bf00 con 0x7f9fcc06dd30 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.409+0000 7f9fc9400640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb0004800 0x7f9fb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fc9e00640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb0003a50 0x7f9fb0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fca800640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 0x7f9fb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fc9400640 1 -- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 msgr2=0x7f9fb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fc9400640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc06dd30 0x7f9fb000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fc9400640 1 -- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb0003a50 msgr2=0x7f9fb0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fc9400640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb0003a50 0x7f9fb0003e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.410+0000 7f9fc9400640 1 -- 172.21.15.97:0/2495122786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb0003520 con 0x7f9fb0004800 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.411+0000 7f9fc9400640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb0004800 0x7f9fb0004c80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8001750 tx=0x7f9fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bbc850f7cde98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.411+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8001370 con 0x7f9fb0004800 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.411+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb80052d0 con 0x7f9fb0004800 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.411+0000 7f9fc8a00640 1 -- 172.21.15.97:0/2495122786 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8003830 con 0x7f9fb0004800 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fd1200640 1 -- 172.21.15.97:0/2495122786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb0004800 msgr2=0x7f9fb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2495122786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb0004800 0x7f9fb0004c80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8001750 tx=0x7f9fb0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fca800640 1 -- 172.21.15.97:0/2495122786 reap_dead start 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fd1200640 1 -- 172.21.15.97:0/2495122786 shutdown_connections 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fd1200640 1 -- 172.21.15.97:0/2495122786 >> 172.21.15.97:0/2495122786 conn(0x7f9fcc0cfa80 msgr2=0x7f9fcc0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fd1200640 1 -- 172.21.15.97:0/2495122786 shutdown_connections 2024-03-06T07:30:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.412+0000 7f9fd1200640 1 -- 172.21.15.97:0/2495122786 wait complete. 2024-03-06T07:30:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.414+0000 7f9fd1200640 1 Processor -- start 2024-03-06T07:30:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.414+0000 7f9fd1200640 1 -- start start 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fcc1513b0 0x7f9fcc13ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc13f7c0 0x7f9fcc13fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 0x7f9fcc13a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fd1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc15c440 con 0x7f9fcc140160 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fd1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0ce470 con 0x7f9fcc1513b0 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fd1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc15c5c0 con 0x7f9fcc13f7c0 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 0x7f9fcc13a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 0x7f9fcc13a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42664/0 (socket says 172.21.15.97:42664) 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 -- 172.21.15.97:0/2882914528 learned_addr learned my addr 172.21.15.97:0/2882914528 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fc9400640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc13f7c0 0x7f9fcc13fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fc9e00640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fcc1513b0 0x7f9fcc13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 -- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc13f7c0 msgr2=0x7f9fcc13fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc13f7c0 0x7f9fcc13fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 -- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fcc1513b0 msgr2=0x7f9fcc13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fcc1513b0 0x7f9fcc13ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fca800640 1 -- 172.21.15.97:0/2882914528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc1405e0 con 0x7f9fcc140160 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fc9400640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc13f7c0 0x7f9fcc13fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.415+0000 7f9fc9e00640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fcc1513b0 0x7f9fcc13ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.416+0000 7f9fca800640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 0x7f9fcc13a620 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40098a0 tx=0x7f9fc4009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.416+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4019070 con 0x7f9fcc140160 2024-03-06T07:30:56.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.416+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc4007ce0 con 0x7f9fcc140160 2024-03-06T07:30:56.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.416+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4014640 con 0x7f9fcc140160 2024-03-06T07:30:56.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.416+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc13b980 con 0x7f9fcc140160 2024-03-06T07:30:56.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.416+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc13bf40 con 0x7f9fcc140160 2024-03-06T07:30:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.417+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc04f3a0 con 0x7f9fcc140160 2024-03-06T07:30:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.418+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fc401c070 con 0x7f9fcc140160 2024-03-06T07:30:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.419+0000 7f9fc2000640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fac06b4d0 0x7f9fac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.420+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9fc4089e70 con 0x7f9fcc140160 2024-03-06T07:30:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.420+0000 7f9fc9e00640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fac06b4d0 0x7f9fac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.420+0000 7f9fc9e00640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fac06b4d0 0x7f9fac06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc001e70 tx=0x7f9fbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.424+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fc4058b60 con 0x7f9fcc140160 2024-03-06T07:30:56.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:56.605+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9fcc0721c0 con 0x7f9fcc140160 2024-03-06T07:30:57.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.564+0000 7f9fc2000640 1 -- 172.21.15.97:0/2882914528 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9fc4058460 con 0x7f9fcc140160 2024-03-06T07:30:57.567 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:57.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.569+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fac06b4d0 msgr2=0x7f9fac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.569+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fac06b4d0 0x7f9fac06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc001e70 tx=0x7f9fbc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.569+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 msgr2=0x7f9fcc13a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.569+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 0x7f9fcc13a620 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40098a0 tx=0x7f9fc4009d60 comp rx=0 tx=0).stop 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 shutdown_connections 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fac06b4d0 0x7f9fac06d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fcc140160 0x7f9fcc13a620 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fcc13f7c0 0x7f9fcc13fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 --2- 172.21.15.97:0/2882914528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fcc1513b0 0x7f9fcc13ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 >> 172.21.15.97:0/2882914528 conn(0x7f9fcc0cfa80 msgr2=0x7f9fcc071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 shutdown_connections 2024-03-06T07:30:57.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.571+0000 7f9fd1200640 1 -- 172.21.15.97:0/2882914528 wait complete. 2024-03-06T07:30:57.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.794+0000 7fbc47800640 1 Processor -- start 2024-03-06T07:30:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.794+0000 7fbc47800640 1 -- start start 2024-03-06T07:30:57.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.795+0000 7fbc47800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbc4015e170 0x7fbc400ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.795+0000 7fbc47800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbc4006e570 0x7fbc400b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.795+0000 7fbc47800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc400b4990 0x7fbc4006cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.795+0000 7fbc47800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4006ea70 con 0x7fbc4015e170 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc47800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc400b3690 con 0x7fbc4006e570 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc47800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400b3970 con 0x7fbc400b4990 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbc4015e170 0x7fbc400ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbc4015e170 0x7fbc400ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42678/0 (socket says 172.21.15.97:42678) 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 -- 172.21.15.97:0/2752452882 learned_addr learned my addr 172.21.15.97:0/2752452882 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 -- 172.21.15.97:0/2752452882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc400b4990 msgr2=0x7fbc4006cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 --2- 172.21.15.97:0/2752452882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc400b4990 0x7fbc4006cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 -- 172.21.15.97:0/2752452882 >> v2:172.21.15.135:3300/0 conn(0x7fbc4006e570 msgr2=0x7fbc400b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 --2- 172.21.15.97:0/2752452882 >> v2:172.21.15.135:3300/0 conn(0x7fbc4006e570 0x7fbc400b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.796+0000 7fbc44e00640 1 -- 172.21.15.97:0/2752452882 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400b3c50 con 0x7fbc4015e170 2024-03-06T07:30:57.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.797+0000 7fbc44e00640 1 --2- 172.21.15.97:0/2752452882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4015e170 0x7fbc400ad860 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fbc40072f50 tx=0x7fbc3402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5274aeab8b55410 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc3f400640 1 -- 172.21.15.97:0/2752452882 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3402e070 con 0x7fbc4015e170 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc3f400640 1 -- 172.21.15.97:0/2752452882 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34037070 con 0x7fbc4015e170 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc3f400640 1 -- 172.21.15.97:0/2752452882 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34031a00 con 0x7fbc4015e170 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 -- 172.21.15.97:0/2752452882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4015e170 msgr2=0x7fbc400ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 --2- 172.21.15.97:0/2752452882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4015e170 0x7fbc400ad860 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fbc40072f50 tx=0x7fbc3402b880 comp rx=0 tx=0).stop 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 -- 172.21.15.97:0/2752452882 shutdown_connections 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 --2- 172.21.15.97:0/2752452882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc400b4990 0x7fbc4006cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 --2- 172.21.15.97:0/2752452882 >> v2:172.21.15.135:3300/0 conn(0x7fbc4006e570 0x7fbc400b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 --2- 172.21.15.97:0/2752452882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4015e170 0x7fbc400ad860 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.798+0000 7fbc47800640 1 -- 172.21.15.97:0/2752452882 >> 172.21.15.97:0/2752452882 conn(0x7fbc400cfa80 msgr2=0x7fbc400cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.799+0000 7fbc47800640 1 -- 172.21.15.97:0/2752452882 shutdown_connections 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.799+0000 7fbc47800640 1 -- 172.21.15.97:0/2752452882 wait complete. 2024-03-06T07:30:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.801+0000 7fbc47800640 1 Processor -- start 2024-03-06T07:30:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.801+0000 7fbc47800640 1 -- start start 2024-03-06T07:30:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc47800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 0x7fbc401e6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc47800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc400b4990 0x7fbc401576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc47800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc4015e170 0x7fbc40157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc47800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4006df80 con 0x7fbc4006e570 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc47800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400b4dd0 con 0x7fbc400b4990 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc47800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400adda0 con 0x7fbc4015e170 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc44e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 0x7fbc401e6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc44e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 0x7fbc401e6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42694/0 (socket says 172.21.15.97:42694) 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.802+0000 7fbc44e00640 1 -- 172.21.15.97:0/3142717357 learned_addr learned my addr 172.21.15.97:0/3142717357 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc3fe00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc400b4990 0x7fbc401576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc45800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc4015e170 0x7fbc40157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc44e00640 1 -- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc4015e170 msgr2=0x7fbc40157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc44e00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc4015e170 0x7fbc40157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc44e00640 1 -- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc400b4990 msgr2=0x7fbc401576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc44e00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc400b4990 0x7fbc401576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc44e00640 1 -- 172.21.15.97:0/3142717357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400aee20 con 0x7fbc4006e570 2024-03-06T07:30:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc45800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc4015e170 0x7fbc40157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc44e00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 0x7fbc401e6a40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbc3402b8c0 tx=0x7fbc340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34032880 con 0x7fbc4006e570 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34032a20 con 0x7fbc4006e570 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.803+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc40142470 con 0x7fbc4006e570 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.804+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34032e10 con 0x7fbc4006e570 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.804+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc40142a30 con 0x7fbc4006e570 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.805+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4004f330 con 0x7fbc4006e570 2024-03-06T07:30:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.806+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc3404c060 con 0x7fbc4006e570 2024-03-06T07:30:57.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.806+0000 7fbc3cc00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.807+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fbc340abde0 con 0x7fbc4006e570 2024-03-06T07:30:57.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.807+0000 7fbc3fe00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.807+0000 7fbc3fe00640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbc40154f80 tx=0x7fbc2c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:57.811+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc34040070 con 0x7fbc4006e570 2024-03-06T07:30:58.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.009+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbc400b2e30 con 0x7fbc4006e570 2024-03-06T07:30:58.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.568+0000 7fbc3cc00640 1 -- 172.21.15.97:0/3142717357 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbc3407adb0 con 0x7fbc4006e570 2024-03-06T07:30:58.571 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:58.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.572+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbc2006b4d0 msgr2=0x7fbc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.572+0000 7fbc47800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbc40154f80 tx=0x7fbc2c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.572+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 msgr2=0x7fbc401e6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.572+0000 7fbc47800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 0x7fbc401e6a40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbc3402b8c0 tx=0x7fbc340046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.573+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 shutdown_connections 2024-03-06T07:30:58.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.573+0000 7fbc47800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbc40154f80 tx=0x7fbc2c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.573+0000 7fbc47800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbc4015e170 0x7fbc40157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.573+0000 7fbc47800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc400b4990 0x7fbc401576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.573+0000 7fbc47800640 1 --2- 172.21.15.97:0/3142717357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc4006e570 0x7fbc401e6a40 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.573+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 >> 172.21.15.97:0/3142717357 conn(0x7fbc400cfa80 msgr2=0x7fbc400721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.574+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 shutdown_connections 2024-03-06T07:30:58.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.574+0000 7fbc47800640 1 -- 172.21.15.97:0/3142717357 wait complete. 2024-03-06T07:30:58.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:30:58.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.804+0000 7feb8b200640 1 Processor -- start 2024-03-06T07:30:58.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.804+0000 7feb8b200640 1 -- start start 2024-03-06T07:30:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.805+0000 7feb8b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feb840cff60 0x7feb8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.805+0000 7feb8b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feb8406da20 0x7feb840ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.805+0000 7feb8b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb840b3560 con 0x7feb84159570 2024-03-06T07:30:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.805+0000 7feb8b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb840cc480 con 0x7feb8406da20 2024-03-06T07:30:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.805+0000 7feb8b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb840cc760 con 0x7feb840cff60 2024-03-06T07:30:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb89e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feb84159570 0x7feb840cf850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41408/0 (socket says 172.21.15.97:41408) 2024-03-06T07:30:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb89e00640 1 -- 172.21.15.97:0/2618265797 learned_addr learned my addr 172.21.15.97:0/2618265797 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb8a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feb8406da20 0x7feb840ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb8a800640 1 -- 172.21.15.97:0/2618265797 >> v2:172.21.15.135:3300/0 conn(0x7feb840cff60 msgr2=0x7feb8406d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb89400640 1 --2- 172.21.15.97:0/2618265797 >> v2:172.21.15.135:3300/0 conn(0x7feb840cff60 0x7feb8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb8a800640 1 --2- 172.21.15.97:0/2618265797 >> v2:172.21.15.135:3300/0 conn(0x7feb840cff60 0x7feb8406d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb8a800640 1 -- 172.21.15.97:0/2618265797 >> v1:172.21.15.97:6789/0 conn(0x7feb84159570 legacy=0x7feb840cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb8a800640 1 -- 172.21.15.97:0/2618265797 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb840cca40 con 0x7feb8406da20 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb89400640 1 --2- 172.21.15.97:0/2618265797 >> v2:172.21.15.135:3300/0 conn(0x7feb840cff60 0x7feb8406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.806+0000 7feb88a00640 1 -- 172.21.15.97:0/2618265797 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373854049 0 0) 0x7feb840b3560 con 0x7feb84159570 2024-03-06T07:30:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.807+0000 7feb8a800640 1 --2- 172.21.15.97:0/2618265797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb8406da20 0x7feb840ce3d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feb8406ecb0 tx=0x7feb8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15049cb1ef84b8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.807+0000 7feb88a00640 1 -- 172.21.15.97:0/2618265797 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80035020 con 0x7feb8406da20 2024-03-06T07:30:58.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.807+0000 7feb88a00640 1 -- 172.21.15.97:0/2618265797 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80031830 con 0x7feb8406da20 2024-03-06T07:30:58.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.807+0000 7feb88a00640 1 -- 172.21.15.97:0/2618265797 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80031c00 con 0x7feb8406da20 2024-03-06T07:30:58.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 -- 172.21.15.97:0/2618265797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb8406da20 msgr2=0x7feb840ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 --2- 172.21.15.97:0/2618265797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb8406da20 0x7feb840ce3d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feb8406ecb0 tx=0x7feb8002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 -- 172.21.15.97:0/2618265797 shutdown_connections 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 --2- 172.21.15.97:0/2618265797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb8406da20 0x7feb840ce3d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 --2- 172.21.15.97:0/2618265797 >> v2:172.21.15.135:3300/0 conn(0x7feb840cff60 0x7feb8406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 -- 172.21.15.97:0/2618265797 >> 172.21.15.97:0/2618265797 conn(0x7feb8406afc0 msgr2=0x7feb8406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 -- 172.21.15.97:0/2618265797 shutdown_connections 2024-03-06T07:30:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.808+0000 7feb8b200640 1 -- 172.21.15.97:0/2618265797 wait complete. 2024-03-06T07:30:58.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.810+0000 7feb8b200640 1 Processor -- start 2024-03-06T07:30:58.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.810+0000 7feb8b200640 1 -- start start 2024-03-06T07:30:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb8406da20 0x7feb84148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb840cff60 0x7feb8414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 0x7feb84156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb840cea90 con 0x7feb84159570 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb840ce910 con 0x7feb840cff60 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb840cec10 con 0x7feb8406da20 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 0x7feb84156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 0x7feb84156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42706/0 (socket says 172.21.15.97:42706) 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.811+0000 7feb8a800640 1 -- 172.21.15.97:0/3756143898 learned_addr learned my addr 172.21.15.97:0/3756143898 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb89400640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb840cff60 0x7feb8414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb89e00640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb8406da20 0x7feb84148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8a800640 1 -- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb8406da20 msgr2=0x7feb84148020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8a800640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb8406da20 0x7feb84148020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8a800640 1 -- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb840cff60 msgr2=0x7feb8414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8a800640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb840cff60 0x7feb8414fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8a800640 1 -- 172.21.15.97:0/3756143898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb841505f0 con 0x7feb84159570 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb89e00640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb8406da20 0x7feb84148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:58.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8a800640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 0x7feb84156ef0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7feb80009860 tx=0x7feb800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80041070 con 0x7feb84159570 2024-03-06T07:30:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8003a970 con 0x7feb84159570 2024-03-06T07:30:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb84153f70 con 0x7feb84159570 2024-03-06T07:30:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.812+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80035040 con 0x7feb84159570 2024-03-06T07:30:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.813+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb84154530 con 0x7feb84159570 2024-03-06T07:30:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.814+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb84154730 con 0x7feb84159570 2024-03-06T07:30:58.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.815+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb80038020 con 0x7feb84159570 2024-03-06T07:30:58.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.815+0000 7feb7a000640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb6806b500 0x7feb6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.816+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feb800ab9b0 con 0x7feb84159570 2024-03-06T07:30:58.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.816+0000 7feb89e00640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb6806b500 0x7feb6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.816+0000 7feb89e00640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb6806b500 0x7feb6806d9c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7feb7c007fd0 tx=0x7feb7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:58.820+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb84154730 con 0x7feb84159570 2024-03-06T07:30:59.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.019+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7feb8404f330 con 0x7feb84159570 2024-03-06T07:30:59.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.024+0000 7feb7a000640 1 -- 172.21.15.97:0/3756143898 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v19) v1 ==== 108+0+0 (secure 0 0 0) 0x7feb8007ab30 con 0x7feb84159570 2024-03-06T07:30:59.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb6806b500 msgr2=0x7feb6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb6806b500 0x7feb6806d9c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7feb7c007fd0 tx=0x7feb7c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 msgr2=0x7feb84156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 0x7feb84156ef0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7feb80009860 tx=0x7feb800046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 shutdown_connections 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb6806b500 0x7feb6806d9c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb84159570 0x7feb84156ef0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb840cff60 0x7feb8414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 --2- 172.21.15.97:0/3756143898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb8406da20 0x7feb84148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 >> 172.21.15.97:0/3756143898 conn(0x7feb8406afc0 msgr2=0x7feb840b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 shutdown_connections 2024-03-06T07:30:59.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.027+0000 7feb8b200640 1 -- 172.21.15.97:0/3756143898 wait complete. 2024-03-06T07:30:59.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:30:59.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.255+0000 7f53c6600640 1 Processor -- start 2024-03-06T07:30:59.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.255+0000 7f53c6600640 1 -- start start 2024-03-06T07:30:59.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.256+0000 7f53c6600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.256+0000 7f53c6600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53c00b3f70 0x7f53c00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.256+0000 7f53c6600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00b46c0 con 0x7f53c015fff0 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.256+0000 7f53c6600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00b49a0 con 0x7f53c00b3f70 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.256+0000 7f53c6600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0072c90 con 0x7f53c015fc10 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53c00b3f70 0x7f53c00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f53c015fff0 0x7f53c0160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41424/0 (socket says 172.21.15.97:41424) 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53c00b3f70 0x7f53c00ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42714/0 (socket says 172.21.15.97:42714) 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bea00640 1 -- 172.21.15.97:0/2671662803 learned_addr learned my addr 172.21.15.97:0/2671662803 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bf400640 1 --2- 172.21.15.97:0/2671662803 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bfe00640 1 -- 172.21.15.97:0/2671662803 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 msgr2=0x7f53c0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bfe00640 1 --2- 172.21.15.97:0/2671662803 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bfe00640 1 -- 172.21.15.97:0/2671662803 >> v1:172.21.15.97:6789/0 conn(0x7f53c015fff0 legacy=0x7f53c0160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53be000640 1 -- 172.21.15.97:0/2671662803 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206686860 0 0) 0x7f53c00b46c0 con 0x7f53c015fff0 2024-03-06T07:30:59.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.257+0000 7f53bfe00640 1 -- 172.21.15.97:0/2671662803 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c009dd80 con 0x7f53c00b3f70 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.259+0000 7f53bfe00640 1 --2- 172.21.15.97:0/2671662803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c00ce4a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009d90 tx=0x7f53b802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afa7403f5c5ffab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.259+0000 7f53be000640 1 -- 172.21.15.97:0/2671662803 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8039070 con 0x7f53c00b3f70 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.259+0000 7f53be000640 1 -- 172.21.15.97:0/2671662803 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b802cc50 con 0x7f53c00b3f70 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.259+0000 7f53be000640 1 -- 172.21.15.97:0/2671662803 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8034440 con 0x7f53c00b3f70 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.259+0000 7f53c6600640 1 -- 172.21.15.97:0/2671662803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 msgr2=0x7f53c00ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.259+0000 7f53c6600640 1 --2- 172.21.15.97:0/2671662803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c00ce4a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009d90 tx=0x7f53b802bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.260+0000 7f53c6600640 1 -- 172.21.15.97:0/2671662803 shutdown_connections 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.260+0000 7f53c6600640 1 --2- 172.21.15.97:0/2671662803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c00ce4a0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.260+0000 7f53c6600640 1 --2- 172.21.15.97:0/2671662803 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.260+0000 7f53c6600640 1 -- 172.21.15.97:0/2671662803 >> 172.21.15.97:0/2671662803 conn(0x7f53c00b1940 msgr2=0x7f53c00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.260+0000 7f53c6600640 1 -- 172.21.15.97:0/2671662803 shutdown_connections 2024-03-06T07:30:59.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.260+0000 7f53c6600640 1 -- 172.21.15.97:0/2671662803 wait complete. 2024-03-06T07:30:59.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.261+0000 7f53c6600640 1 Processor -- start 2024-03-06T07:30:59.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.261+0000 7f53c6600640 1 -- start start 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.262+0000 7f53c6600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c0158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.262+0000 7f53c6600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.262+0000 7f53c6600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c015fff0 0x7f53c009d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.262+0000 7f53c6600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c015c460 con 0x7f53c00b3f70 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.262+0000 7f53c6600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c015c2e0 con 0x7f53c015fff0 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.262+0000 7f53c6600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00ce9e0 con 0x7f53c015fc10 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c0158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c0158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42716/0 (socket says 172.21.15.97:42716) 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 -- 172.21.15.97:0/2690498030 learned_addr learned my addr 172.21.15.97:0/2690498030 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bea00640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 -- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 msgr2=0x7f53c0144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 -- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c015fff0 msgr2=0x7f53c009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c015fff0 0x7f53c009d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 -- 172.21.15.97:0/2690498030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c0158a20 con 0x7f53c00b3f70 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.263+0000 7f53bf400640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c0158440 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d830 tx=0x7f53b000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.264+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b000b6c0 con 0x7f53c00b3f70 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.264+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0019070 con 0x7f53c00b3f70 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.264+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b000ba80 con 0x7f53c00b3f70 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.264+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c0158d00 con 0x7f53c00b3f70 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.264+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c009ac40 con 0x7f53c00b3f70 2024-03-06T07:30:59.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.265+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c004f330 con 0x7f53c00b3f70 2024-03-06T07:30:59.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.266+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53b001e460 con 0x7f53c00b3f70 2024-03-06T07:30:59.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.267+0000 7f53b7400640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.267+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53b008a520 con 0x7f53c00b3f70 2024-03-06T07:30:59.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.267+0000 7f53bea00640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.268+0000 7f53bea00640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53a006b4d0 0x7f53a006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f53c009ca80 tx=0x7f53ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.271+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53b0059730 con 0x7f53c00b3f70 2024-03-06T07:30:59.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.464+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f53c006ef30 con 0x7f53c00b3f70 2024-03-06T07:30:59.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.468+0000 7f53b7400640 1 -- 172.21.15.97:0/2690498030 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7f53b0059030 con 0x7f53c00b3f70 2024-03-06T07:30:59.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.471+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53a006b4d0 msgr2=0x7f53a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.471+0000 7f53c6600640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53a006b4d0 0x7f53a006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f53c009ca80 tx=0x7f53ac009040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.471+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 msgr2=0x7f53c0158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.471+0000 7f53c6600640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c0158440 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d830 tx=0x7f53b000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 shutdown_connections 2024-03-06T07:30:59.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c015fff0 0x7f53c009d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c015fc10 0x7f53c0144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 --2- 172.21.15.97:0/2690498030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00b3f70 0x7f53c0158440 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 >> 172.21.15.97:0/2690498030 conn(0x7f53c00b1940 msgr2=0x7f53c00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 shutdown_connections 2024-03-06T07:30:59.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.472+0000 7f53c6600640 1 -- 172.21.15.97:0/2690498030 wait complete. 2024-03-06T07:30:59.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:59.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.699+0000 7fa609000640 1 Processor -- start 2024-03-06T07:30:59.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.699+0000 7fa609000640 1 -- start start 2024-03-06T07:30:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.701+0000 7fa609000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6040af9a0 0x7fa6040afd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.701+0000 7fa609000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6040b02c0 con 0x7fa60415e150 2024-03-06T07:30:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.701+0000 7fa609000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60415f330 con 0x7fa60415ed10 2024-03-06T07:30:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.701+0000 7fa609000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60406d890 con 0x7fa6040af9a0 2024-03-06T07:30:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.701+0000 7fa601400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa60415e150 0x7fa60415e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41428/0 (socket says 172.21.15.97:41428) 2024-03-06T07:30:59.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.701+0000 7fa601400640 1 -- 172.21.15.97:0/3923492013 learned_addr learned my addr 172.21.15.97:0/3923492013 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099819977 0 0) 0x7fa6040b02c0 con 0x7fa60415e150 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e8003520 con 0x7fa60415e150 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa601e00640 1 --2- 172.21.15.97:0/3923492013 >> v2:172.21.15.135:3300/0 conn(0x7fa6040af9a0 0x7fa6040afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272811220 0 0) 0x7fa5e8003520 con 0x7fa60415e150 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 >> v2:172.21.15.135:3300/0 conn(0x7fa6040af9a0 msgr2=0x7fa6040afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 --2- 172.21.15.97:0/3923492013 >> v2:172.21.15.135:3300/0 conn(0x7fa6040af9a0 0x7fa6040afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 >> v1:172.21.15.135:6789/0 conn(0x7fa60415ed10 legacy=0x7fa60406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604161af0 con 0x7fa60415e150 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa5f0001670 con 0x7fa60415e150 2024-03-06T07:30:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 >> v1:172.21.15.97:6789/0 conn(0x7fa60415e150 legacy=0x7fa60415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5e80039b0 0x7fa5e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e8004800 0x7fa5e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5e80051c0 0x7fa5e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e800c0e0 con 0x7fa5e80051c0 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e800c290 con 0x7fa5e8004800 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.702+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e800c490 con 0x7fa5e80039b0 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa602800640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5e80051c0 0x7fa5e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa601e00640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5e80039b0 0x7fa5e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa601400640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e8004800 0x7fa5e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa602800640 1 -- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5e80039b0 msgr2=0x7fa5e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa602800640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5e80039b0 0x7fa5e8003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa602800640 1 -- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e8004800 msgr2=0x7fa5e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa602800640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e8004800 0x7fa5e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa602800640 1 -- 172.21.15.97:0/3923492013 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8003520 con 0x7fa5e80051c0 2024-03-06T07:30:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.703+0000 7fa601400640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e8004800 0x7fa5e8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.704+0000 7fa602800640 1 -- 172.21.15.97:0/3923492013 reap_dead start 2024-03-06T07:30:59.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.704+0000 7fa602800640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5e80051c0 0x7fa5e800b9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc001e50 tx=0x7fa5e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754cb271a77d69df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.704+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc001a70 con 0x7fa5e80051c0 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.704+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc003b90 con 0x7fa5e80051c0 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.704+0000 7fa600a00640 1 -- 172.21.15.97:0/3923492013 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0050d0 con 0x7fa5e80051c0 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 -- 172.21.15.97:0/3923492013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5e80051c0 msgr2=0x7fa5e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5e80051c0 0x7fa5e800b9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc001e50 tx=0x7fa5e8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 -- 172.21.15.97:0/3923492013 shutdown_connections 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 --2- 172.21.15.97:0/3923492013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5e80051c0 0x7fa5e800b9c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 -- 172.21.15.97:0/3923492013 >> 172.21.15.97:0/3923492013 conn(0x7fa6040b2840 msgr2=0x7fa6040b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 -- 172.21.15.97:0/3923492013 shutdown_connections 2024-03-06T07:30:59.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.705+0000 7fa609000640 1 -- 172.21.15.97:0/3923492013 wait complete. 2024-03-06T07:30:59.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.707+0000 7fa609000640 1 Processor -- start 2024-03-06T07:30:59.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.707+0000 7fa609000640 1 -- start start 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa609000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e80051c0 0x7fa604152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa609000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 0x7fa604148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa609000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa60415aee0 0x7fa60415b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa609000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604162390 con 0x7fa6040b0e80 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa609000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6041620b0 con 0x7fa5e80051c0 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa609000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6041628b0 con 0x7fa60415aee0 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 0x7fa604148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 0x7fa604148570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42732/0 (socket says 172.21.15.97:42732) 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 -- 172.21.15.97:0/3414677686 learned_addr learned my addr 172.21.15.97:0/3414677686 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601e00640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e80051c0 0x7fa604152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa602800640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa60415aee0 0x7fa60415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 -- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa60415aee0 msgr2=0x7fa60415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa60415aee0 0x7fa60415b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 -- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e80051c0 msgr2=0x7fa604152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e80051c0 0x7fa604152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601400640 1 -- 172.21.15.97:0/3414677686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604153600 con 0x7fa6040b0e80 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.708+0000 7fa601e00640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e80051c0 0x7fa604152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.709+0000 7fa601400640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 0x7fa604148570 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000d770 tx=0x7fa5f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.709+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0019070 con 0x7fa6040b0e80 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.709+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f00074e0 con 0x7fa6040b0e80 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.709+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6041538e0 con 0x7fa6040b0e80 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.709+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00049e0 con 0x7fa6040b0e80 2024-03-06T07:30:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.709+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa604153ea0 con 0x7fa6040b0e80 2024-03-06T07:30:59.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.710+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6040d0210 con 0x7fa6040b0e80 2024-03-06T07:30:59.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.711+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5f001c070 con 0x7fa6040b0e80 2024-03-06T07:30:59.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.712+0000 7fa5fa000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5e406b4d0 0x7fa5e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.712+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa5f00898f0 con 0x7fa6040b0e80 2024-03-06T07:30:59.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.712+0000 7fa601e00640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5e406b4d0 0x7fa5e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.713+0000 7fa601e00640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5e406b4d0 0x7fa5e406d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa6040aeee0 tx=0x7fa5f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.716+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6040d0210 con 0x7fa6040b0e80 2024-03-06T07:30:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.896+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa604162640 con 0x7fa6040b0e80 2024-03-06T07:30:59.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.902+0000 7fa5fa000640 1 -- 172.21.15.97:0/3414677686 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa604162640 con 0x7fa6040b0e80 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5e406b4d0 msgr2=0x7fa5e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5e406b4d0 0x7fa5e406d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa6040aeee0 tx=0x7fa5f4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 msgr2=0x7fa604148570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 0x7fa604148570 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000d770 tx=0x7fa5f000dc30 comp rx=0 tx=0).stop 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 shutdown_connections 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5e406b4d0 0x7fa5e406d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa60415aee0 0x7fa60415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6040b0e80 0x7fa604148570 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 --2- 172.21.15.97:0/3414677686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5e80051c0 0x7fa604152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.904+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 >> 172.21.15.97:0/3414677686 conn(0x7fa6040b2840 msgr2=0x7fa6040b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.905+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 shutdown_connections 2024-03-06T07:30:59.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:30:59.905+0000 7fa609000640 1 -- 172.21.15.97:0/3414677686 wait complete. 2024-03-06T07:30:59.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:31:00.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.121+0000 7fba97e00640 1 Processor -- start 2024-03-06T07:31:00.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.121+0000 7fba97e00640 1 -- start start 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba97e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fba980aedc0 0x7fba980cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba97e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9815c240 con 0x7fba981561f0 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba97e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9815c500 con 0x7fba980cc800 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba97e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9815ce30 con 0x7fba980aedc0 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba96a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fba981561f0 0x7fba980ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41430/0 (socket says 172.21.15.97:41430) 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba96a00640 1 -- 172.21.15.97:0/3659725708 learned_addr learned my addr 172.21.15.97:0/3659725708 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba96000640 1 --2- 172.21.15.97:0/3659725708 >> v2:172.21.15.97:3300/0 conn(0x7fba980aedc0 0x7fba980cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba95600640 1 -- 172.21.15.97:0/3659725708 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835954513 0 0) 0x7fba9815c240 con 0x7fba981561f0 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba95600640 1 -- 172.21.15.97:0/3659725708 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c003520 con 0x7fba981561f0 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba96000640 1 -- 172.21.15.97:0/3659725708 >> v1:172.21.15.135:6789/0 conn(0x7fba980cc800 legacy=0x7fba9815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba96000640 1 -- 172.21.15.97:0/3659725708 >> v1:172.21.15.97:6789/0 conn(0x7fba981561f0 legacy=0x7fba980ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.123+0000 7fba96000640 1 -- 172.21.15.97:0/3659725708 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9815c7e0 con 0x7fba980aedc0 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.125+0000 7fba96000640 1 --2- 172.21.15.97:0/3659725708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba980aedc0 0x7fba980cc2c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fba9815cfb0 tx=0x7fba8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3511767bc6ae6d94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.125+0000 7fba95600640 1 -- 172.21.15.97:0/3659725708 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84035020 con 0x7fba980aedc0 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.125+0000 7fba95600640 1 -- 172.21.15.97:0/3659725708 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba84031830 con 0x7fba980aedc0 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.125+0000 7fba95600640 1 -- 172.21.15.97:0/3659725708 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84031c00 con 0x7fba980aedc0 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.125+0000 7fba97e00640 1 -- 172.21.15.97:0/3659725708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba980aedc0 msgr2=0x7fba980cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.125+0000 7fba97e00640 1 --2- 172.21.15.97:0/3659725708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba980aedc0 0x7fba980cc2c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fba9815cfb0 tx=0x7fba8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.126+0000 7fba97e00640 1 -- 172.21.15.97:0/3659725708 shutdown_connections 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.126+0000 7fba97e00640 1 --2- 172.21.15.97:0/3659725708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba980aedc0 0x7fba980cc2c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.126+0000 7fba97e00640 1 -- 172.21.15.97:0/3659725708 >> 172.21.15.97:0/3659725708 conn(0x7fba9806afc0 msgr2=0x7fba9806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.126+0000 7fba97e00640 1 -- 172.21.15.97:0/3659725708 shutdown_connections 2024-03-06T07:31:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.126+0000 7fba97e00640 1 -- 172.21.15.97:0/3659725708 wait complete. 2024-03-06T07:31:00.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.127+0000 7fba97e00640 1 Processor -- start 2024-03-06T07:31:00.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.127+0000 7fba97e00640 1 -- start start 2024-03-06T07:31:00.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.128+0000 7fba97e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba980aedc0 0x7fba9815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.128+0000 7fba97e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba980cc800 0x7fba980cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.128+0000 7fba97e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 0x7fba9814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.128+0000 7fba97e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980ccc30 con 0x7fba981561f0 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.128+0000 7fba97e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9806e360 con 0x7fba980aedc0 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.128+0000 7fba97e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba981565d0 con 0x7fba980cc800 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 0x7fba9814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 0x7fba9814a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42750/0 (socket says 172.21.15.97:42750) 2024-03-06T07:31:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 -- 172.21.15.97:0/4006469492 learned_addr learned my addr 172.21.15.97:0/4006469492 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:00.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba96000640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba980cc800 0x7fba980cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 -- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba980cc800 msgr2=0x7fba980cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba980cc800 0x7fba980cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 -- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba980aedc0 msgr2=0x7fba9815ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba980aedc0 0x7fba9815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 -- 172.21.15.97:0/4006469492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9815e3d0 con 0x7fba981561f0 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.129+0000 7fba97400640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 0x7fba9814a2f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fba9000c780 tx=0x7fba9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.130+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90004b60 con 0x7fba981561f0 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.130+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba90004d00 con 0x7fba981561f0 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.130+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000aa90 con 0x7fba981561f0 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.130+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9815e6b0 con 0x7fba981561f0 2024-03-06T07:31:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.130+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9815ec70 con 0x7fba981561f0 2024-03-06T07:31:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.131+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9804f330 con 0x7fba981561f0 2024-03-06T07:31:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.132+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba9000ac30 con 0x7fba981561f0 2024-03-06T07:31:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.133+0000 7fba8ea00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba7806b4d0 0x7fba7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.133+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fba9008a310 con 0x7fba981561f0 2024-03-06T07:31:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.133+0000 7fba96a00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba7806b4d0 0x7fba7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.134+0000 7fba96a00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba7806b4d0 0x7fba7806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fba88000f60 tx=0x7fba880078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.137+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba90020070 con 0x7fba981561f0 2024-03-06T07:31:00.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.407+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fba980b2b10 con 0x7fba981561f0 2024-03-06T07:31:00.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.598+0000 7fba8ea00640 1 -- 172.21.15.97:0/4006469492 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v196) v1 ==== 144+0+0 (secure 0 0 0) 0x7fba90059360 con 0x7fba981561f0 2024-03-06T07:31:00.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba7806b4d0 msgr2=0x7fba7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba7806b4d0 0x7fba7806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fba88000f60 tx=0x7fba880078f0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 msgr2=0x7fba9814a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 0x7fba9814a2f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fba9000c780 tx=0x7fba9000cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 shutdown_connections 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba7806b4d0 0x7fba7806d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fba981561f0 0x7fba9814a2f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fba980cc800 0x7fba980cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 --2- 172.21.15.97:0/4006469492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba980aedc0 0x7fba9815ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.601+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 >> 172.21.15.97:0/4006469492 conn(0x7fba9806afc0 msgr2=0x7fba9806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.602+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 shutdown_connections 2024-03-06T07:31:00.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.602+0000 7fba97e00640 1 -- 172.21.15.97:0/4006469492 wait complete. 2024-03-06T07:31:00.623 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:31:00.623 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:31:00.624 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:31:00.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:00.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.824+0000 7f08b1200640 1 Processor -- start 2024-03-06T07:31:00.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.824+0000 7f08b1200640 1 -- start start 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.825+0000 7f08b1200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f08ac0af1c0 0x7f08ac15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.825+0000 7f08b1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f08ac070d20 0x7f08ac06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.825+0000 7f08b1200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac071d30 con 0x7f08ac06d1a0 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.825+0000 7f08b1200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ac0711d0 con 0x7f08ac0af1c0 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08b1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ac071490 con 0x7f08ac070d20 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f08ac0af1c0 0x7f08ac15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f08ac0af1c0 0x7f08ac15eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42766/0 (socket says 172.21.15.97:42766) 2024-03-06T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 -- 172.21.15.97:0/3673125224 learned_addr learned my addr 172.21.15.97:0/3673125224 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:00.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 -- 172.21.15.97:0/3673125224 >> v2:172.21.15.135:3300/0 conn(0x7f08ac070d20 msgr2=0x7f08ac06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 --2- 172.21.15.97:0/3673125224 >> v2:172.21.15.135:3300/0 conn(0x7f08ac070d20 0x7f08ac06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 -- 172.21.15.97:0/3673125224 >> v1:172.21.15.135:6789/0 conn(0x7f08ac06d1a0 legacy=0x7f08ac06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.826+0000 7f08a9e00640 1 -- 172.21.15.97:0/3673125224 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac071770 con 0x7f08ac0af1c0 2024-03-06T07:31:00.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.827+0000 7f08a9e00640 1 --2- 172.21.15.97:0/3673125224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac15eff0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f089c009870 tx=0x7f089c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b62da22a0465f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.828+0000 7f08a8a00640 1 -- 172.21.15.97:0/3673125224 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c035020 con 0x7f08ac0af1c0 2024-03-06T07:31:00.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.828+0000 7f08a8a00640 1 -- 172.21.15.97:0/3673125224 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c02d040 con 0x7f08ac0af1c0 2024-03-06T07:31:00.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.828+0000 7f08a8a00640 1 -- 172.21.15.97:0/3673125224 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c031a80 con 0x7f08ac0af1c0 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.828+0000 7f08b1200640 1 -- 172.21.15.97:0/3673125224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 msgr2=0x7f08ac15eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.828+0000 7f08b1200640 1 --2- 172.21.15.97:0/3673125224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac15eff0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f089c009870 tx=0x7f089c02b900 comp rx=0 tx=0).stop 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.829+0000 7f08b1200640 1 -- 172.21.15.97:0/3673125224 shutdown_connections 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.829+0000 7f08b1200640 1 --2- 172.21.15.97:0/3673125224 >> v2:172.21.15.135:3300/0 conn(0x7f08ac070d20 0x7f08ac06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.829+0000 7f08b1200640 1 --2- 172.21.15.97:0/3673125224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac15eff0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.829+0000 7f08b1200640 1 -- 172.21.15.97:0/3673125224 >> 172.21.15.97:0/3673125224 conn(0x7f08ac0b1730 msgr2=0x7f08ac0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.829+0000 7f08b1200640 1 -- 172.21.15.97:0/3673125224 shutdown_connections 2024-03-06T07:31:00.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.829+0000 7f08b1200640 1 -- 172.21.15.97:0/3673125224 wait complete. 2024-03-06T07:31:00.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.830+0000 7f08b1200640 1 Processor -- start 2024-03-06T07:31:00.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.830+0000 7f08b1200640 1 -- start start 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.831+0000 7f08b1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f08ac06d1a0 0x7f08ac14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.831+0000 7f08b1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08ac070d20 0x7f08ac161050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.831+0000 7f08b1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.831+0000 7f08b1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac06ebe0 con 0x7f08ac0af1c0 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.831+0000 7f08b1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac0b30b0 con 0x7f08ac070d20 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.831+0000 7f08b1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac06ed60 con 0x7f08ac06d1a0 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac14f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42774/0 (socket says 172.21.15.97:42774) 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 -- 172.21.15.97:0/67785430 learned_addr learned my addr 172.21.15.97:0/67785430 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08a9e00640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f08ac06d1a0 0x7f08ac14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08a9400640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08ac070d20 0x7f08ac161050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 -- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f08ac06d1a0 msgr2=0x7f08ac14eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f08ac06d1a0 0x7f08ac14eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 -- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08ac070d20 msgr2=0x7f08ac161050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08ac070d20 0x7f08ac161050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 -- 172.21.15.97:0/67785430 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac0ce630 con 0x7f08ac0af1c0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08a9400640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08ac070d20 0x7f08ac161050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.832+0000 7f08aa800640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac14f5c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f08a400b990 tx=0x7f08a400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.833+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a4019070 con 0x7f08ac0af1c0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.833+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a400cba0 con 0x7f08ac0af1c0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.833+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a4014440 con 0x7f08ac0af1c0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.833+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac0ce910 con 0x7f08ac0af1c0 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.833+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac15e8f0 con 0x7f08ac0af1c0 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.834+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac15eaf0 con 0x7f08ac0af1c0 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.835+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08a401c070 con 0x7f08ac0af1c0 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.836+0000 7f08a2000640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.836+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f08a4089c80 con 0x7f08ac0af1c0 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.836+0000 7f08a9e00640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.837+0000 7f08a9e00640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f088c06b4d0 0x7f088c06d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f089c036000 tx=0x7f089c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:00.840+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08ac15eaf0 con 0x7f08ac0af1c0 2024-03-06T07:31:01.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.036+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08ac0b04a0 con 0x7f08ac0af1c0 2024-03-06T07:31:01.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.038+0000 7f08a2000640 1 -- 172.21.15.97:0/67785430 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+13004 (secure 0 0 0) 0x7f08a4058e00 con 0x7f08ac0af1c0 2024-03-06T07:31:01.040 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:01.040 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":61,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:30:58.565503+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/24115969":"2024-03-07T07:30:55.161481+0000","172.21.15.97:6835/24115969":"2024-03-07T07:30:55.161481+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.040+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f088c06b4d0 msgr2=0x7f088c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.040+0000 7f08b1200640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f088c06b4d0 0x7f088c06d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f089c036000 tx=0x7f089c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.040+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 msgr2=0x7f08ac14f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.040+0000 7f08b1200640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac14f5c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f08a400b990 tx=0x7f08a400be50 comp rx=0 tx=0).stop 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 shutdown_connections 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08ac0af1c0 0x7f08ac14f5c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08ac070d20 0x7f08ac161050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 --2- 172.21.15.97:0/67785430 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f08ac06d1a0 0x7f08ac14eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 >> 172.21.15.97:0/67785430 conn(0x7f08ac0b1730 msgr2=0x7f08ac06db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 shutdown_connections 2024-03-06T07:31:01.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.041+0000 7f08b1200640 1 -- 172.21.15.97:0/67785430 wait complete. 2024-03-06T07:31:01.061 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:01.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:01.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.271+0000 7f985a000640 1 Processor -- start 2024-03-06T07:31:01.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.271+0000 7f985a000640 1 -- start start 2024-03-06T07:31:01.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f985a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98540b2c40 0x7f985415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f985a000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f985406d960 0x7f98540b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f985a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406e2f0 0x7f9854161d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f985a000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985415f700 con 0x7f98540b2c40 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f985a000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985406d4d0 con 0x7f985406d960 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f985a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98540b3560 con 0x7f985406e2f0 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f9853400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98540b2c40 0x7f985415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f9853400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98540b2c40 0x7f985415f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42778/0 (socket says 172.21.15.97:42778) 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.273+0000 7f9853400640 1 -- 172.21.15.97:0/1103556420 learned_addr learned my addr 172.21.15.97:0/1103556420 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9852a00640 1 --2- 172.21.15.97:0/1103556420 >> v2:172.21.15.135:3300/0 conn(0x7f985406d960 0x7f98540b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9853e00640 1 --2- 172.21.15.97:0/1103556420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406e2f0 0x7f9854161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9853400640 1 -- 172.21.15.97:0/1103556420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406e2f0 msgr2=0x7f9854161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9853400640 1 --2- 172.21.15.97:0/1103556420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406e2f0 0x7f9854161d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9853400640 1 -- 172.21.15.97:0/1103556420 >> v2:172.21.15.135:3300/0 conn(0x7f985406d960 msgr2=0x7f98540b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9853400640 1 --2- 172.21.15.97:0/1103556420 >> v2:172.21.15.135:3300/0 conn(0x7f985406d960 0x7f98540b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.274+0000 7f9853400640 1 -- 172.21.15.97:0/1103556420 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985406cba0 con 0x7f98540b2c40 2024-03-06T07:31:01.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.276+0000 7f9853400640 1 --2- 172.21.15.97:0/1103556420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985415f1c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9854074a50 tx=0x7f984402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d5cdbb8214c3c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.276+0000 7f9852000640 1 -- 172.21.15.97:0/1103556420 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844035020 con 0x7f98540b2c40 2024-03-06T07:31:01.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.276+0000 7f9852000640 1 -- 172.21.15.97:0/1103556420 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984402d040 con 0x7f98540b2c40 2024-03-06T07:31:01.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.276+0000 7f9852000640 1 -- 172.21.15.97:0/1103556420 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844031a80 con 0x7f98540b2c40 2024-03-06T07:31:01.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 -- 172.21.15.97:0/1103556420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 msgr2=0x7f985415f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 --2- 172.21.15.97:0/1103556420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985415f1c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9854074a50 tx=0x7f984402b900 comp rx=0 tx=0).stop 2024-03-06T07:31:01.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 -- 172.21.15.97:0/1103556420 shutdown_connections 2024-03-06T07:31:01.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 --2- 172.21.15.97:0/1103556420 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406e2f0 0x7f9854161d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 --2- 172.21.15.97:0/1103556420 >> v2:172.21.15.135:3300/0 conn(0x7f985406d960 0x7f98540b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 --2- 172.21.15.97:0/1103556420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985415f1c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 -- 172.21.15.97:0/1103556420 >> 172.21.15.97:0/1103556420 conn(0x7f98540b1940 msgr2=0x7f98540b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 -- 172.21.15.97:0/1103556420 shutdown_connections 2024-03-06T07:31:01.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.277+0000 7f985a000640 1 -- 172.21.15.97:0/1103556420 wait complete. 2024-03-06T07:31:01.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.279+0000 7f985a000640 1 Processor -- start 2024-03-06T07:31:01.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.280+0000 7f985a000640 1 -- start start 2024-03-06T07:31:01.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f985a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406d960 0x7f985409a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f985a000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f985406e2f0 0x7f9854161ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f985a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985409c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f985a000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985406e780 con 0x7f98540b2c40 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f985a000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985406d160 con 0x7f985406e2f0 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f985a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985406e900 con 0x7f985406d960 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f9853e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985409c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f9853e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985409c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42786/0 (socket says 172.21.15.97:42786) 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.281+0000 7f9853e00640 1 -- 172.21.15.97:0/1585796828 learned_addr learned my addr 172.21.15.97:0/1585796828 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853400640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406d960 0x7f985409a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9852a00640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f985406e2f0 0x7f9854161ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853e00640 1 -- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406d960 msgr2=0x7f985409a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853e00640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406d960 0x7f985409a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853e00640 1 -- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f985406e2f0 msgr2=0x7f9854161ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853e00640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f985406e2f0 0x7f9854161ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853e00640 1 -- 172.21.15.97:0/1585796828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985409af30 con 0x7f98540b2c40 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853400640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406d960 0x7f985409a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f9853e00640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985409c870 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f984c007e40 tx=0x7f984c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c014620 con 0x7f98540b2c40 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c019070 con 0x7f98540b2c40 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c014a20 con 0x7f98540b2c40 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985409b200 con 0x7f98540b2c40 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.282+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985409b7c0 con 0x7f98540b2c40 2024-03-06T07:31:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.284+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9854067830 con 0x7f98540b2c40 2024-03-06T07:31:01.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.285+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f984c014bc0 con 0x7f98540b2c40 2024-03-06T07:31:01.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.285+0000 7f984b400640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f983406b4d0 0x7f983406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.286+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f984c08a770 con 0x7f98540b2c40 2024-03-06T07:31:01.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.286+0000 7f9853400640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f983406b4d0 0x7f983406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.286+0000 7f9853400640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f983406b4d0 0x7f983406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9844036000 tx=0x7f9844008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.290+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9854067830 con 0x7f98540b2c40 2024-03-06T07:31:01.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.484+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98540b3a00 con 0x7f98540b2c40 2024-03-06T07:31:01.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.484+0000 7f984b400640 1 -- 172.21.15.97:0/1585796828 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f984c0598f0 con 0x7f98540b2c40 2024-03-06T07:31:01.486 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f983406b4d0 msgr2=0x7f983406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f983406b4d0 0x7f983406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9844036000 tx=0x7f9844008030 comp rx=0 tx=0).stop 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 msgr2=0x7f985409c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985409c870 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f984c007e40 tx=0x7f984c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 shutdown_connections 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f983406b4d0 0x7f983406d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98540b2c40 0x7f985409c870 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f985406e2f0 0x7f9854161ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 --2- 172.21.15.97:0/1585796828 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f985406d960 0x7f985409a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 >> 172.21.15.97:0/1585796828 conn(0x7f98540b1940 msgr2=0x7f985415dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 shutdown_connections 2024-03-06T07:31:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.487+0000 7f985a000640 1 -- 172.21.15.97:0/1585796828 wait complete. 2024-03-06T07:31:01.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:31:01.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.713+0000 7fc7e8e00640 1 Processor -- start 2024-03-06T07:31:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.713+0000 7fc7e8e00640 1 -- start start 2024-03-06T07:31:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.715+0000 7fc7e8e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7e406ec00 0x7fc7e40ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.715+0000 7fc7e8e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e415b090 con 0x7fc7e415acb0 2024-03-06T07:31:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.715+0000 7fc7e8e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e40b2a30 con 0x7fc7e40b4680 2024-03-06T07:31:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.715+0000 7fc7e8e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e40b4360 con 0x7fc7e406ec00 2024-03-06T07:31:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.715+0000 7fc7e1e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc7e415acb0 0x7fc7e40b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41438/0 (socket says 172.21.15.97:41438) 2024-03-06T07:31:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.715+0000 7fc7e1e00640 1 -- 172.21.15.97:0/22777465 learned_addr learned my addr 172.21.15.97:0/22777465 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e2800640 1 --2- 172.21.15.97:0/22777465 >> v2:172.21.15.135:3300/0 conn(0x7fc7e406ec00 0x7fc7e40ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797529462 0 0) 0x7fc7e415b090 con 0x7fc7e415acb0 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8003520 con 0x7fc7e415acb0 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2832479759 0 0) 0x7fc7c8003520 con 0x7fc7e415acb0 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 >> v2:172.21.15.135:3300/0 conn(0x7fc7e406ec00 msgr2=0x7fc7e40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 --2- 172.21.15.97:0/22777465 >> v2:172.21.15.135:3300/0 conn(0x7fc7e406ec00 0x7fc7e40ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 >> v1:172.21.15.135:6789/0 conn(0x7fc7e40b4680 legacy=0x7fc7e406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e40b2cf0 con 0x7fc7e415acb0 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc7d40019f0 con 0x7fc7e415acb0 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 >> v1:172.21.15.97:6789/0 conn(0x7fc7e415acb0 legacy=0x7fc7e40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7c8003a00 0x7fc7c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7c8004800 0x7fc7c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c80051c0 0x7fc7c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40b4360 con 0x7fc7c8003a00 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c800c0c0 con 0x7fc7c80051c0 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.716+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c800c2c0 con 0x7fc7c8004800 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7c8003a00 0x7fc7c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e2800640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c80051c0 0x7fc7c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1400640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7c8004800 0x7fc7c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1e00640 1 -- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7c8004800 msgr2=0x7fc7c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7c8004800 0x7fc7c8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1e00640 1 -- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c80051c0 msgr2=0x7fc7c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c80051c0 0x7fc7c800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e2800640 1 -- 172.21.15.97:0/22777465 reap_dead start 2024-03-06T07:31:01.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.717+0000 7fc7e1e00640 1 -- 172.21.15.97:0/22777465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8003520 con 0x7fc7c8003a00 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.719+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7c8003a00 0x7fc7c8003e20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400a370 tx=0x7fc7c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf77f2adba1879c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.719+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400a3b0 con 0x7fc7c8003a00 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.719+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d4036650 con 0x7fc7c8003a00 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.719+0000 7fc7e0a00640 1 -- 172.21.15.97:0/22777465 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4036a40 con 0x7fc7c8003a00 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.719+0000 7fc7e8e00640 1 -- 172.21.15.97:0/22777465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7c8003a00 msgr2=0x7fc7c8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.719+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7c8003a00 0x7fc7c8003e20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400a370 tx=0x7fc7c8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.720+0000 7fc7e8e00640 1 -- 172.21.15.97:0/22777465 shutdown_connections 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.720+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/22777465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7c8003a00 0x7fc7c8003e20 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.720+0000 7fc7e8e00640 1 -- 172.21.15.97:0/22777465 >> 172.21.15.97:0/22777465 conn(0x7fc7e40b1730 msgr2=0x7fc7e40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.720+0000 7fc7e8e00640 1 -- 172.21.15.97:0/22777465 shutdown_connections 2024-03-06T07:31:01.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.720+0000 7fc7e8e00640 1 -- 172.21.15.97:0/22777465 wait complete. 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.721+0000 7fc7e8e00640 1 Processor -- start 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.721+0000 7fc7e8e00640 1 -- start start 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.722+0000 7fc7e8e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8003a00 0x7fc7e413cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.722+0000 7fc7e8e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7e4150d90 0x7fc7e4156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.722+0000 7fc7e8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 0x7fc7e4157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.722+0000 7fc7e8e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40b4010 con 0x7fc7e4151210 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.722+0000 7fc7e8e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40b32b0 con 0x7fc7c8003a00 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.722+0000 7fc7e8e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e4072340 con 0x7fc7e4150d90 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 0x7fc7e4157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 0x7fc7e4157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42804/0 (socket says 172.21.15.97:42804) 2024-03-06T07:31:01.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 -- 172.21.15.97:0/2200109398 learned_addr learned my addr 172.21.15.97:0/2200109398 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e1400640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7e4150d90 0x7fc7e4156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 -- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7e4150d90 msgr2=0x7fc7e4156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8003a00 0x7fc7e413cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7e4150d90 0x7fc7e4156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 -- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8003a00 msgr2=0x7fc7e413cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8003a00 0x7fc7e413cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 -- 172.21.15.97:0/2200109398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e4152730 con 0x7fc7e4151210 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.723+0000 7fc7e2800640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 0x7fc7e4157280 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40b4af0 tx=0x7fc7dc010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.724+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc00f040 con 0x7fc7e4151210 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.724+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc01b070 con 0x7fc7e4151210 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.724+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc004490 con 0x7fc7e4151210 2024-03-06T07:31:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.724+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e4151690 con 0x7fc7e4151210 2024-03-06T07:31:01.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.724+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e4151c50 con 0x7fc7e4151210 2024-03-06T07:31:01.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.725+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e40b1ea0 con 0x7fc7e4151210 2024-03-06T07:31:01.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.726+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7dc00e670 con 0x7fc7e4151210 2024-03-06T07:31:01.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.727+0000 7fc7da000640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7c006b4d0 0x7fc7c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.727+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc7dc089d50 con 0x7fc7e4151210 2024-03-06T07:31:01.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.727+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7c006b4d0 0x7fc7c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.728+0000 7fc7e1e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7c006b4d0 0x7fc7c006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4036000 tx=0x7fc7d4009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.732+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7e40b1ea0 con 0x7fc7e4151210 2024-03-06T07:31:01.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:01.988+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fc7e406d620 con 0x7fc7e4151210 2024-03-06T07:31:02.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.609+0000 7fc7da000640 1 -- 172.21.15.97:0/2200109398 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v198) v1 ==== 143+0+0 (secure 0 0 0) 0x7fc7dc08d030 con 0x7fc7e4151210 2024-03-06T07:31:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7c006b4d0 msgr2=0x7fc7c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7c006b4d0 0x7fc7c006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4036000 tx=0x7fc7d4009030 comp rx=0 tx=0).stop 2024-03-06T07:31:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 msgr2=0x7fc7e4157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 0x7fc7e4157280 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40b4af0 tx=0x7fc7dc010c20 comp rx=0 tx=0).stop 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 shutdown_connections 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7c006b4d0 0x7fc7c006d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7e4151210 0x7fc7e4157280 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7e4150d90 0x7fc7e4156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 --2- 172.21.15.97:0/2200109398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8003a00 0x7fc7e413cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 >> 172.21.15.97:0/2200109398 conn(0x7fc7e40b1730 msgr2=0x7fc7e41546b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 shutdown_connections 2024-03-06T07:31:02.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.613+0000 7fc7e8e00640 1 -- 172.21.15.97:0/2200109398 wait complete. 2024-03-06T07:31:02.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:31:02.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.850+0000 7f998f600640 1 Processor -- start 2024-03-06T07:31:02.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.850+0000 7f998f600640 1 -- start start 2024-03-06T07:31:02.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.851+0000 7f998f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880b3df0 0x7f99880727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.851+0000 7f998f600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f99880b3650 0x7f99880b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998f600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99880b4710 con 0x7f99880cda90 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998f600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9988072310 con 0x7f99880b3650 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99880719e0 con 0x7f99880b3df0 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f9987e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f99880b3650 0x7f99880b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998d600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f99880cda90 0x7f998806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41452/0 (socket says 172.21.15.97:41452) 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998d600640 1 -- 172.21.15.97:0/4205706157 learned_addr learned my addr 172.21.15.97:0/4205706157 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998cc00640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880b3df0 0x7f99880727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f9987e00640 1 -- 172.21.15.97:0/4205706157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880b3df0 msgr2=0x7f99880727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f9987e00640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880b3df0 0x7f99880727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f9987e00640 1 -- 172.21.15.97:0/4205706157 >> v1:172.21.15.97:6789/0 conn(0x7f99880cda90 legacy=0x7f998806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f9987e00640 1 -- 172.21.15.97:0/4205706157 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9988071cc0 con 0x7f99880b3650 2024-03-06T07:31:02.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.852+0000 7f998cc00640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880b3df0 0x7f99880727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.853+0000 7f9987e00640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3650 0x7f99880b41d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f997402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3278e178727de40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f9987400640 1 -- 172.21.15.97:0/4205706157 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974039070 con 0x7f99880b3650 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f9987400640 1 -- 172.21.15.97:0/4205706157 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997402cc50 con 0x7f99880b3650 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f9987400640 1 -- 172.21.15.97:0/4205706157 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974034440 con 0x7f99880b3650 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 -- 172.21.15.97:0/4205706157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3650 msgr2=0x7f99880b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3650 0x7f99880b41d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f997402bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 -- 172.21.15.97:0/4205706157 shutdown_connections 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3650 0x7f99880b41d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 --2- 172.21.15.97:0/4205706157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880b3df0 0x7f99880727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 -- 172.21.15.97:0/4205706157 >> 172.21.15.97:0/4205706157 conn(0x7f99880b1940 msgr2=0x7f99880b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 -- 172.21.15.97:0/4205706157 shutdown_connections 2024-03-06T07:31:02.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.854+0000 7f998f600640 1 -- 172.21.15.97:0/4205706157 wait complete. 2024-03-06T07:31:02.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.856+0000 7f998f600640 1 Processor -- start 2024-03-06T07:31:02.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.856+0000 7f998f600640 1 -- start start 2024-03-06T07:31:02.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f998f600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99880b3650 0x7f998809d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f998f600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 0x7f9988141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f998f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880cda90 0x7f998809b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f998f600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998815b240 con 0x7f99880b3df0 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f998f600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99880b3b30 con 0x7f99880b3650 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f998f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998815b3c0 con 0x7f99880cda90 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f9987e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 0x7f9988141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f9987e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 0x7f9988141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42828/0 (socket says 172.21.15.97:42828) 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.857+0000 7f9987e00640 1 -- 172.21.15.97:0/3758861174 learned_addr learned my addr 172.21.15.97:0/3758861174 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f998cc00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99880b3650 0x7f998809d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f998d600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880cda90 0x7f998809b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9987e00640 1 -- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880cda90 msgr2=0x7f998809b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9987e00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880cda90 0x7f998809b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9987e00640 1 -- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99880b3650 msgr2=0x7f998809d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9987e00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99880b3650 0x7f998809d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9987e00640 1 -- 172.21.15.97:0/3758861174 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998815a590 con 0x7f99880b3df0 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f998cc00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99880b3650 0x7f998809d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f998d600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880cda90 0x7f998809b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9987e00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 0x7f9988141ea0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f9974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.858+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974039050 con 0x7f99880b3df0 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.859+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974045070 con 0x7f99880b3df0 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.859+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997403e850 con 0x7f99880b3df0 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.859+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998815a870 con 0x7f99880b3df0 2024-03-06T07:31:02.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.859+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99881423e0 con 0x7f99880b3df0 2024-03-06T07:31:02.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.860+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998804f330 con 0x7f99880b3df0 2024-03-06T07:31:02.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.861+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f997402cad0 con 0x7f99880b3df0 2024-03-06T07:31:02.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.862+0000 7f9984c00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f996806b4d0 0x7f996806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.862+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f99740acbb0 con 0x7f99880b3df0 2024-03-06T07:31:02.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.862+0000 7f998cc00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f996806b4d0 0x7f996806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.862+0000 7f998cc00640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f996806b4d0 0x7f996806d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:02.866+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f997407bdb0 con 0x7f99880b3df0 2024-03-06T07:31:03.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.049+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f998815aa40 con 0x7f99880b3df0 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.055+0000 7f9984c00640 1 -- 172.21.15.97:0/3758861174 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f997407b6b0 con 0x7f99880b3df0 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f996806b4d0 msgr2=0x7f996806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f996806b4d0 0x7f996806d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978008040 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 msgr2=0x7f9988141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 0x7f9988141ea0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f9974004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 shutdown_connections 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f996806b4d0 0x7f996806d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f99880cda90 0x7f998809b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99880b3df0 0x7f9988141ea0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 --2- 172.21.15.97:0/3758861174 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99880b3650 0x7f998809d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.057+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 >> 172.21.15.97:0/3758861174 conn(0x7f99880b1940 msgr2=0x7f99880b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.058+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 shutdown_connections 2024-03-06T07:31:03.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.058+0000 7f998f600640 1 -- 172.21.15.97:0/3758861174 wait complete. 2024-03-06T07:31:03.079 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:03.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:03.194 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:03.316 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:03.316 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:03.316 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:03.372 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:31:03.372 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.429 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:03.429 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:03.429 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.484 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:03.485 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:31:03.485 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.541 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:03.541 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:31:03.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:03.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.756+0000 7fe95ea00640 1 Processor -- start 2024-03-06T07:31:03.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.757+0000 7fe95ea00640 1 -- start start 2024-03-06T07:31:03.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe95ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe95815acb0 0x7fe9580b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe95ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9580b4680 0x7fe95806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe95ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe958072340 0x7fe95806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe95ea00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95815b090 con 0x7fe95815acb0 2024-03-06T07:31:03.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe95ea00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9580b2a30 con 0x7fe958072340 2024-03-06T07:31:03.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe95ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580b4360 con 0x7fe9580b4680 2024-03-06T07:31:03.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe957e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe95815acb0 0x7fe9580b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe957e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe95815acb0 0x7fe9580b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42834/0 (socket says 172.21.15.97:42834) 2024-03-06T07:31:03.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.758+0000 7fe957e00640 1 -- 172.21.15.97:0/307973037 learned_addr learned my addr 172.21.15.97:0/307973037 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe95ca00640 1 --2- 172.21.15.97:0/307973037 >> v2:172.21.15.135:3300/0 conn(0x7fe958072340 0x7fe95806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe94f400640 1 --2- 172.21.15.97:0/307973037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9580b4680 0x7fe95806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe957e00640 1 -- 172.21.15.97:0/307973037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9580b4680 msgr2=0x7fe95806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe957e00640 1 --2- 172.21.15.97:0/307973037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9580b4680 0x7fe95806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe957e00640 1 -- 172.21.15.97:0/307973037 >> v2:172.21.15.135:3300/0 conn(0x7fe958072340 msgr2=0x7fe95806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe957e00640 1 --2- 172.21.15.97:0/307973037 >> v2:172.21.15.135:3300/0 conn(0x7fe958072340 0x7fe95806ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe957e00640 1 -- 172.21.15.97:0/307973037 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9580b2cf0 con 0x7fe95815acb0 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.759+0000 7fe95ca00640 1 --2- 172.21.15.97:0/307973037 >> v2:172.21.15.135:3300/0 conn(0x7fe958072340 0x7fe95806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.760+0000 7fe957e00640 1 --2- 172.21.15.97:0/307973037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe95815acb0 0x7fe9580b37d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe958070aa0 tx=0x7fe94802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5fe50823ecf663 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.760+0000 7fe957400640 1 -- 172.21.15.97:0/307973037 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948035020 con 0x7fe95815acb0 2024-03-06T07:31:03.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.760+0000 7fe957400640 1 -- 172.21.15.97:0/307973037 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94802d040 con 0x7fe95815acb0 2024-03-06T07:31:03.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.760+0000 7fe957400640 1 -- 172.21.15.97:0/307973037 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948031a80 con 0x7fe95815acb0 2024-03-06T07:31:03.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 -- 172.21.15.97:0/307973037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe95815acb0 msgr2=0x7fe9580b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 --2- 172.21.15.97:0/307973037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe95815acb0 0x7fe9580b37d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe958070aa0 tx=0x7fe94802b900 comp rx=0 tx=0).stop 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 -- 172.21.15.97:0/307973037 shutdown_connections 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 --2- 172.21.15.97:0/307973037 >> v2:172.21.15.135:3300/0 conn(0x7fe958072340 0x7fe95806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 --2- 172.21.15.97:0/307973037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9580b4680 0x7fe95806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 --2- 172.21.15.97:0/307973037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe95815acb0 0x7fe9580b37d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 -- 172.21.15.97:0/307973037 >> 172.21.15.97:0/307973037 conn(0x7fe9580b1730 msgr2=0x7fe9580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 -- 172.21.15.97:0/307973037 shutdown_connections 2024-03-06T07:31:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.761+0000 7fe95ea00640 1 -- 172.21.15.97:0/307973037 wait complete. 2024-03-06T07:31:03.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.763+0000 7fe95ea00640 1 Processor -- start 2024-03-06T07:31:03.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.763+0000 7fe95ea00640 1 -- start start 2024-03-06T07:31:03.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe958072340 0x7fe9580ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 0x7fe9580ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe95815acb0 0x7fe95814d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580b3d10 con 0x7fe9580b4680 2024-03-06T07:31:03.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580b32b0 con 0x7fe95815acb0 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580b4010 con 0x7fe958072340 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe94f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 0x7fe9580ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe94f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 0x7fe9580ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42844/0 (socket says 172.21.15.97:42844) 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe94f400640 1 -- 172.21.15.97:0/3254915323 learned_addr learned my addr 172.21.15.97:0/3254915323 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe94f400640 1 -- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe958072340 msgr2=0x7fe9580ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe957e00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe958072340 0x7fe9580ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe95ca00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe95815acb0 0x7fe95814d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.764+0000 7fe94f400640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe958072340 0x7fe9580ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe94f400640 1 -- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe95815acb0 msgr2=0x7fe95814d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe94f400640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe95815acb0 0x7fe95814d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe94f400640 1 -- 172.21.15.97:0/3254915323 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9580cbe50 con 0x7fe9580b4680 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe957e00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe958072340 0x7fe9580ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe94f400640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 0x7fe9580ce9b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c7f0 tx=0x7fe94000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940015070 con 0x7fe9580b4680 2024-03-06T07:31:03.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe940004240 con 0x7fe9580b4680 2024-03-06T07:31:03.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940007620 con 0x7fe9580b4680 2024-03-06T07:31:03.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9580cd590 con 0x7fe9580b4680 2024-03-06T07:31:03.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.765+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9580cdb50 con 0x7fe9580b4680 2024-03-06T07:31:03.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.766+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95804f330 con 0x7fe9580b4680 2024-03-06T07:31:03.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.767+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9400079d0 con 0x7fe9580b4680 2024-03-06T07:31:03.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.768+0000 7fe954c00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe93406b500 0x7fe93406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.768+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe940089fe0 con 0x7fe9580b4680 2024-03-06T07:31:03.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.768+0000 7fe957e00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe93406b500 0x7fe93406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.769+0000 7fe957e00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe93406b500 0x7fe93406d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe948036000 tx=0x7fe948008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:03.773+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9580cbe50 con 0x7fe9580b4680 2024-03-06T07:31:04.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.007+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe95815e5b0 con 0x7fe9580b4680 2024-03-06T07:31:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.008+0000 7fe954c00640 1 -- 172.21.15.97:0/3254915323 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fe9400590e0 con 0x7fe9580b4680 2024-03-06T07:31:04.011 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:04.011 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/4194335673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":4194335673},{"type":"v1","addr":"172.21.15.135:6833","nonce":4194335673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4996,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2059799869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":2059799869},{"type":"v1","addr":"172.21.15.135:6835","nonce":2059799869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/1991180831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1991180831},{"type":"v1","addr":"172.21.15.97:6836","nonce":1991180831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3377454548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3377454548},{"type":"v1","addr":"172.21.15.97:6837","nonce":3377454548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2024-03-06T07:31:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 36 2024-03-06T07:31:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe93406b500 msgr2=0x7fe93406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe93406b500 0x7fe93406d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe948036000 tx=0x7fe948008030 comp rx=0 tx=0).stop 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 msgr2=0x7fe9580ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 0x7fe9580ce9b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c7f0 tx=0x7fe94000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 shutdown_connections 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe93406b500 0x7fe93406d9c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe95815acb0 0x7fe95814d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9580b4680 0x7fe9580ce9b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 --2- 172.21.15.97:0/3254915323 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe958072340 0x7fe9580ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 >> 172.21.15.97:0/3254915323 conn(0x7fe9580b1730 msgr2=0x7fe958071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 shutdown_connections 2024-03-06T07:31:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.011+0000 7fe95ea00640 1 -- 172.21.15.97:0/3254915323 wait complete. 2024-03-06T07:31:04.030 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:31:04.030 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:31:04.030 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:31:04.030 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:04.031 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:31:04.031 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:31:04.031 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:04.031 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:31:04.031 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:31:04.031 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:31:04.032 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:04.032 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:31:04.032 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:31:04.032 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:31:04.032 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:04.048 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:31:04.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:31:04.050 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:31:04.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:31:04.066 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:31:04.066 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:31:04.067 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:31:04.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:31:04.096 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:31:04.097 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:04.103 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:31:04.103 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:04.114 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:31:04.114 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:04.120 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:31:04.120 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:04.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:31:04.243 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:31:04.253 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:31:04.276 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:31:04.276 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:31:04.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.297+0000 7f5ac6200640 1 Processor -- start 2024-03-06T07:31:04.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.298+0000 7f5ac6200640 1 -- start start 2024-03-06T07:31:04.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5ac6200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5ac00c6460 0x7f5ac00c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5ac6200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5ac00c5960 0x7f5ac0165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5ac6200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac0145b30 con 0x7f5ac00c4930 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5ac6200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac00c68c0 con 0x7f5ac00c5960 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5ac6200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac0142ce0 con 0x7f5ac00c6460 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abf400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5ac00c4930 0x7f5ac00c4d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50640/0 (socket says 172.21.15.97:50640) 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abf400640 1 -- 172.21.15.97:0/1388362936 learned_addr learned my addr 172.21.15.97:0/1388362936 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abfe00640 1 --2- 172.21.15.97:0/1388362936 >> v2:172.21.15.97:3300/0 conn(0x7f5ac00c5960 0x7f5ac0165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5ab6a00640 1 --2- 172.21.15.97:0/1388362936 >> v2:172.21.15.135:3300/0 conn(0x7f5ac00c6460 0x7f5ac00c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abfe00640 1 -- 172.21.15.97:0/1388362936 >> v2:172.21.15.135:3300/0 conn(0x7f5ac00c6460 msgr2=0x7f5ac00c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abfe00640 1 --2- 172.21.15.97:0/1388362936 >> v2:172.21.15.135:3300/0 conn(0x7f5ac00c6460 0x7f5ac00c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abfe00640 1 -- 172.21.15.97:0/1388362936 >> v1:172.21.15.97:6789/0 conn(0x7f5ac00c4930 legacy=0x7f5ac00c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abfe00640 1 -- 172.21.15.97:0/1388362936 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac00cbca0 con 0x7f5ac00c5960 2024-03-06T07:31:04.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.299+0000 7f5abea00640 1 -- 172.21.15.97:0/1388362936 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215334119 0 0) 0x7f5ab0001110 con 0x7f5ac00c4930 2024-03-06T07:31:04.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.300+0000 7f5abfe00640 1 --2- 172.21.15.97:0/1388362936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c5960 0x7f5ac0165b30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0068c90 tx=0x7f5ab802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b0ebee05e253f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5abea00640 1 -- 172.21.15.97:0/1388362936 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8039070 con 0x7f5ac00c5960 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5abea00640 1 -- 172.21.15.97:0/1388362936 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab802cbf0 con 0x7f5ac00c5960 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5abea00640 1 -- 172.21.15.97:0/1388362936 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8034440 con 0x7f5ac00c5960 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 -- 172.21.15.97:0/1388362936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c5960 msgr2=0x7f5ac0165b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 --2- 172.21.15.97:0/1388362936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c5960 0x7f5ac0165b30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0068c90 tx=0x7f5ab802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 -- 172.21.15.97:0/1388362936 shutdown_connections 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 --2- 172.21.15.97:0/1388362936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c5960 0x7f5ac0165b30 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 --2- 172.21.15.97:0/1388362936 >> v2:172.21.15.135:3300/0 conn(0x7f5ac00c6460 0x7f5ac00c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 -- 172.21.15.97:0/1388362936 >> 172.21.15.97:0/1388362936 conn(0x7f5ac00cc4f0 msgr2=0x7f5ac00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.301+0000 7f5ac6200640 1 -- 172.21.15.97:0/1388362936 shutdown_connections 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.302+0000 7f5ac6200640 1 -- 172.21.15.97:0/1388362936 wait complete. 2024-03-06T07:31:04.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.303+0000 7f5ac6200640 1 Processor -- start 2024-03-06T07:31:04.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.303+0000 7f5ac6200640 1 -- start start 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5ac6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ac00c4930 0x7f5ac01e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5ac6200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ac00c5960 0x7f5ac00affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5ac6200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 0x7f5ac01d6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5ac6200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac00ca3b0 con 0x7f5ac00c6460 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5ac6200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac00ca0d0 con 0x7f5ac00c5960 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5ac6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac006e600 con 0x7f5ac00c4930 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.304+0000 7f5abfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 0x7f5ac01d6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 0x7f5ac01d6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58190/0 (socket says 172.21.15.97:58190) 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 -- 172.21.15.97:0/2032936325 learned_addr learned my addr 172.21.15.97:0/2032936325 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abf400640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ac00c4930 0x7f5ac01e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5ab6a00640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ac00c5960 0x7f5ac00affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 -- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ac00c4930 msgr2=0x7f5ac01e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ac00c4930 0x7f5ac01e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 -- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ac00c5960 msgr2=0x7f5ac00affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ac00c5960 0x7f5ac00affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 -- 172.21.15.97:0/2032936325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac01d69c0 con 0x7f5ac00c6460 2024-03-06T07:31:04.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5abfe00640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 0x7f5ac01d6480 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009e80 tx=0x7f5ab8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8039050 con 0x7f5ac00c6460 2024-03-06T07:31:04.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab8045070 con 0x7f5ac00c6460 2024-03-06T07:31:04.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.305+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8009450 con 0x7f5ac00c6460 2024-03-06T07:31:04.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.306+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac00b0690 con 0x7f5ac00c6460 2024-03-06T07:31:04.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.306+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac00b0c50 con 0x7f5ac00c6460 2024-03-06T07:31:04.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.307+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac004f330 con 0x7f5ac00c6460 2024-03-06T07:31:04.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.307+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ab803e850 con 0x7f5ac00c6460 2024-03-06T07:31:04.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.308+0000 7f5ab7e00640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a9c06b4d0 0x7f5a9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.308+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5ab80acb30 con 0x7f5ac00c6460 2024-03-06T07:31:04.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.309+0000 7f5abf400640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a9c06b4d0 0x7f5a9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.309+0000 7f5abf400640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a9c06b4d0 0x7f5a9c06d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000f60 tx=0x7f5ab0007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.313+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ab807bdd0 con 0x7f5ac00c6460 2024-03-06T07:31:04.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.515+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5ac0145570 con 0x7f5ac00c6460 2024-03-06T07:31:04.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.641+0000 7f5ab7e00640 1 -- 172.21.15.97:0/2032936325 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5ab807b6d0 con 0x7f5ac00c6460 2024-03-06T07:31:04.644 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:31:04.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a9c06b4d0 msgr2=0x7f5a9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a9c06b4d0 0x7f5a9c06d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000f60 tx=0x7f5ab0007c00 comp rx=0 tx=0).stop 2024-03-06T07:31:04.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 msgr2=0x7f5ac01d6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 0x7f5ac01d6480 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009e80 tx=0x7f5ab8004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:04.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 shutdown_connections 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a9c06b4d0 0x7f5a9c06d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ac00c6460 0x7f5ac01d6480 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ac00c5960 0x7f5ac00affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 --2- 172.21.15.97:0/2032936325 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ac00c4930 0x7f5ac01e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 >> 172.21.15.97:0/2032936325 conn(0x7f5ac00cc4f0 msgr2=0x7f5ac0141080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 shutdown_connections 2024-03-06T07:31:04.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.645+0000 7f5ac6200640 1 -- 172.21.15.97:0/2032936325 wait complete. 2024-03-06T07:31:04.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.862+0000 7f266b200640 1 Processor -- start 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.862+0000 7f266b200640 1 -- start start 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.863+0000 7f266b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 0x7f266c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.863+0000 7f266b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f266c0cc800 0x7f266c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.863+0000 7f266b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c15c240 con 0x7f266c1561f0 2024-03-06T07:31:04.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.863+0000 7f266b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266c15ce30 con 0x7f266c0cc800 2024-03-06T07:31:04.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.863+0000 7f266b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c15c500 con 0x7f266c0aedc0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.864+0000 7f2669e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f266c1561f0 0x7f266c0ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50652/0 (socket says 172.21.15.97:50652) 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.864+0000 7f2669e00640 1 -- 172.21.15.97:0/1761152288 learned_addr learned my addr 172.21.15.97:0/1761152288 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.864+0000 7f266a800640 1 --2- 172.21.15.97:0/1761152288 >> v2:172.21.15.135:3300/0 conn(0x7f266c0cc800 0x7f266c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.864+0000 7f2663e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 0x7f266c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.864+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620292572 0 0) 0x7f266c15c240 con 0x7f266c1561f0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.864+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2650003520 con 0x7f266c1561f0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1531444835 0 0) 0x7f2650003520 con 0x7f266c1561f0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 msgr2=0x7f266c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 0x7f266c0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 >> v2:172.21.15.135:3300/0 conn(0x7f266c0cc800 msgr2=0x7f266c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 --2- 172.21.15.97:0/1761152288 >> v2:172.21.15.135:3300/0 conn(0x7f266c0cc800 0x7f266c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c15c7e0 con 0x7f266c1561f0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f265c001670 con 0x7f266c1561f0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 >> v1:172.21.15.97:6789/0 conn(0x7f266c1561f0 legacy=0x7f266c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26500039b0 0x7f2650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f2650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 0x7f26500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c15ce30 con 0x7f2650004800 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c15c500 con 0x7f26500039b0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265000b9c0 con 0x7f266c0aedc0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.865+0000 7f2663e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f2650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2669e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26500039b0 0x7f2650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2663e00640 1 -- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 msgr2=0x7f26500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f266a800640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 0x7f26500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2663e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c0aedc0 0x7f26500051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2663e00640 1 -- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26500039b0 msgr2=0x7f2650003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2663e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26500039b0 0x7f2650003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2663e00640 1 -- 172.21.15.97:0/1761152288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2650003520 con 0x7f2650004800 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.866+0000 7f2669e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26500039b0 0x7f2650003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:04.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f2663e00640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f2650004c80 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2654009ac0 tx=0x7f2650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9db18ced7240cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654039070 con 0x7f2650004800 2024-03-06T07:31:04.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265402c6b0 con 0x7f2650004800 2024-03-06T07:31:04.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f2669400640 1 -- 172.21.15.97:0/1761152288 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654009690 con 0x7f2650004800 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f266b200640 1 -- 172.21.15.97:0/1761152288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 msgr2=0x7f2650004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f266b200640 1 --2- 172.21.15.97:0/1761152288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f2650004c80 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2654009ac0 tx=0x7f2650005830 comp rx=0 tx=0).stop 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.867+0000 7f266a800640 1 -- 172.21.15.97:0/1761152288 reap_dead start 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.868+0000 7f266b200640 1 -- 172.21.15.97:0/1761152288 shutdown_connections 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.868+0000 7f266b200640 1 -- 172.21.15.97:0/1761152288 >> 172.21.15.97:0/1761152288 conn(0x7f266c06afc0 msgr2=0x7f266c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.868+0000 7f266b200640 1 -- 172.21.15.97:0/1761152288 shutdown_connections 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.868+0000 7f266b200640 1 -- 172.21.15.97:0/1761152288 wait complete. 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.869+0000 7f266b200640 1 Processor -- start 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 -- start start 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f266c1482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c1568d0 0x7f266c0ab3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f266c156d50 0x7f266c0ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1565d0 con 0x7f2650004800 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c0ccc90 con 0x7f266c156d50 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.870+0000 7f266b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c156750 con 0x7f266c1568d0 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f266c1482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f266c1482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58194/0 (socket says 172.21.15.97:58194) 2024-03-06T07:31:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 -- 172.21.15.97:0/3644497525 learned_addr learned my addr 172.21.15.97:0/3644497525 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 -- 172.21.15.97:0/3644497525 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c1568d0 msgr2=0x7f266c0ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c1568d0 0x7f266c0ab3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 -- 172.21.15.97:0/3644497525 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f266c156d50 msgr2=0x7f266c0ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f266c156d50 0x7f266c0ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 -- 172.21.15.97:0/3644497525 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c0a95f0 con 0x7f2650004800 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.871+0000 7f2669e00640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f266c1482b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f265c00d860 tx=0x7f265c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.872+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c019070 con 0x7f2650004800 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.872+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c0074e0 con 0x7f2650004800 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.872+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c0a98d0 con 0x7f2650004800 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.872+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0049e0 con 0x7f2650004800 2024-03-06T07:31:04.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.872+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c0a9e90 con 0x7f2650004800 2024-03-06T07:31:04.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.873+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c04f330 con 0x7f2650004800 2024-03-06T07:31:04.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.874+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f265c01c070 con 0x7f2650004800 2024-03-06T07:31:04.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.874+0000 7f2662000640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f264806b500 0x7f264806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.875+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f265c089800 con 0x7f2650004800 2024-03-06T07:31:04.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.875+0000 7f2663e00640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f264806b500 0x7f264806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.875+0000 7f2663e00640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f264806b500 0x7f264806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2654036000 tx=0x7f265402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:04.879+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f265c058930 con 0x7f2650004800 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.142+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f266c0cbf50 con 0x7f2650004800 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.143+0000 7f2662000640 1 -- 172.21.15.97:0/3644497525 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f265c058230 con 0x7f2650004800 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:31:05.145 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:31:05.147 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.148 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.149 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.150 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.152 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.153 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:31:05.155 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:31:05.157 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:31:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f264806b500 msgr2=0x7f264806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f264806b500 0x7f264806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2654036000 tx=0x7f265402e000 comp rx=0 tx=0).stop 2024-03-06T07:31:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 msgr2=0x7f266c1482b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f266c1482b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f265c00d860 tx=0x7f265c00dd20 comp rx=0 tx=0).stop 2024-03-06T07:31:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 shutdown_connections 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f264806b500 0x7f264806d9c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f266c156d50 0x7f266c0ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f266c1568d0 0x7f266c0ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 --2- 172.21.15.97:0/3644497525 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2650004800 0x7f266c1482b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 >> 172.21.15.97:0/3644497525 conn(0x7f266c06afc0 msgr2=0x7f266c06e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 shutdown_connections 2024-03-06T07:31:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.145+0000 7f266b200640 1 -- 172.21.15.97:0/3644497525 wait complete. 2024-03-06T07:31:05.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:05.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.355+0000 7f4a88400640 1 Processor -- start 2024-03-06T07:31:05.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.355+0000 7f4a88400640 1 -- start start 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a88400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4a80144c00 0x7f4a8006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a88400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 0x7f4a8015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a88400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8006ee70 con 0x7f4a800ce0a0 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a88400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a80160370 con 0x7f4a80144c00 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a88400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8006cef0 con 0x7f4a8015f9d0 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a86400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4a800ce0a0 0x7f4a800ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50660/0 (socket says 172.21.15.97:50660) 2024-03-06T07:31:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a86400640 1 -- 172.21.15.97:0/477208182 learned_addr learned my addr 172.21.15.97:0/477208182 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a85000640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 0x7f4a8015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a85a00640 1 --2- 172.21.15.97:0/477208182 >> v2:172.21.15.135:3300/0 conn(0x7f4a80144c00 0x7f4a8006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708255997 0 0) 0x7f4a78001490 con 0x7f4a800ce0a0 2024-03-06T07:31:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.357+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a68003520 con 0x7f4a800ce0a0 2024-03-06T07:31:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2925378523 0 0) 0x7f4a68003520 con 0x7f4a800ce0a0 2024-03-06T07:31:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 msgr2=0x7f4a8015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 0x7f4a8015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 >> v2:172.21.15.135:3300/0 conn(0x7f4a80144c00 msgr2=0x7f4a8006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 --2- 172.21.15.97:0/477208182 >> v2:172.21.15.135:3300/0 conn(0x7f4a80144c00 0x7f4a8006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8006d1d0 con 0x7f4a800ce0a0 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4a78001bc0 con 0x7f4a800ce0a0 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 >> v1:172.21.15.97:6789/0 conn(0x7f4a800ce0a0 legacy=0x7f4a800ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 0x7f4a680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68003f30 0x7f4a68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8006cef0 con 0x7f4a68004d40 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6800b9c0 con 0x7f4a68003f30 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6800bb40 con 0x7f4a8015f9d0 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.358+0000 7f4a86400640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.359+0000 7f4a85000640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68003f30 0x7f4a68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.359+0000 7f4a86400640 1 -- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 msgr2=0x7f4a680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.359+0000 7f4a86400640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015f9d0 0x7f4a680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.359+0000 7f4a86400640 1 -- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68003f30 msgr2=0x7f4a68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.359+0000 7f4a86400640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68003f30 0x7f4a68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.359+0000 7f4a86400640 1 -- 172.21.15.97:0/477208182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68003520 con 0x7f4a68004d40 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a86400640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a680051c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800aa70 tx=0x7f4a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b51aaa270065c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78038070 con 0x7f4a68004d40 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a780072f0 con 0x7f4a68004d40 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a7fe00640 1 -- 172.21.15.97:0/477208182 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800a690 con 0x7f4a68004d40 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a88400640 1 -- 172.21.15.97:0/477208182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 msgr2=0x7f4a680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a88400640 1 --2- 172.21.15.97:0/477208182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a680051c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800aa70 tx=0x7f4a68005830 comp rx=0 tx=0).stop 2024-03-06T07:31:05.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.360+0000 7f4a86400640 1 -- 172.21.15.97:0/477208182 reap_dead start 2024-03-06T07:31:05.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.361+0000 7f4a88400640 1 -- 172.21.15.97:0/477208182 shutdown_connections 2024-03-06T07:31:05.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.361+0000 7f4a88400640 1 -- 172.21.15.97:0/477208182 >> 172.21.15.97:0/477208182 conn(0x7f4a800b1730 msgr2=0x7f4a800b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.361+0000 7f4a88400640 1 -- 172.21.15.97:0/477208182 shutdown_connections 2024-03-06T07:31:05.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.361+0000 7f4a88400640 1 -- 172.21.15.97:0/477208182 wait complete. 2024-03-06T07:31:05.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.362+0000 7f4a88400640 1 Processor -- start 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.362+0000 7f4a88400640 1 -- start start 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.363+0000 7f4a88400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a801404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.363+0000 7f4a88400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a80131e40 0x7f4a80142db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.363+0000 7f4a88400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015d390 0x7f4a8015d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.363+0000 7f4a88400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80160730 con 0x7f4a68004d40 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.363+0000 7f4a88400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8006d790 con 0x7f4a80131e40 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.363+0000 7f4a88400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8006cb50 con 0x7f4a8015d390 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a801404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a801404c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58220/0 (socket says 172.21.15.97:58220) 2024-03-06T07:31:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 -- 172.21.15.97:0/3150253901 learned_addr learned my addr 172.21.15.97:0/3150253901 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 -- 172.21.15.97:0/3150253901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015d390 msgr2=0x7f4a8015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015d390 0x7f4a8015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 -- 172.21.15.97:0/3150253901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a80131e40 msgr2=0x7f4a80142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a80131e40 0x7f4a80142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 -- 172.21.15.97:0/3150253901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a801432f0 con 0x7f4a68004d40 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a85a00640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a801404c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400e7c0 tx=0x7f4a7400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.364+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74017070 con 0x7f4a68004d40 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.365+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7400c860 con 0x7f4a68004d40 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.365+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7400cc50 con 0x7f4a68004d40 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.365+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a80140b50 con 0x7f4a68004d40 2024-03-06T07:31:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.365+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a80141110 con 0x7f4a68004d40 2024-03-06T07:31:05.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.366+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a80145750 con 0x7f4a68004d40 2024-03-06T07:31:05.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.367+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a7401a070 con 0x7f4a68004d40 2024-03-06T07:31:05.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.367+0000 7f4a7d600640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a6006b4d0 0x7f4a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.368+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4a74089c50 con 0x7f4a68004d40 2024-03-06T07:31:05.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.368+0000 7f4a85000640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a6006b4d0 0x7f4a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.368+0000 7f4a85000640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a6006b4d0 0x7f4a6006d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a8015efb0 tx=0x7f4a6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.372+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a80145750 con 0x7f4a68004d40 2024-03-06T07:31:05.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.568+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a8004f330 con 0x7f4a68004d40 2024-03-06T07:31:05.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.569+0000 7f4a7d600640 1 -- 172.21.15.97:0/3150253901 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f4a74058e90 con 0x7f4a68004d40 2024-03-06T07:31:05.572 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:05.572 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":65,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:04.658490+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:05.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a6006b4d0 msgr2=0x7f4a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a6006b4d0 0x7f4a6006d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a8015efb0 tx=0x7f4a6c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 msgr2=0x7f4a801404c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a801404c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400e7c0 tx=0x7f4a7400ec80 comp rx=0 tx=0).stop 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 shutdown_connections 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a6006b4d0 0x7f4a6006d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a8015d390 0x7f4a8015d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a80131e40 0x7f4a80142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 --2- 172.21.15.97:0/3150253901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68004d40 0x7f4a801404c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 >> 172.21.15.97:0/3150253901 conn(0x7f4a800b1730 msgr2=0x7f4a80145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 shutdown_connections 2024-03-06T07:31:05.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.572+0000 7f4a88400640 1 -- 172.21.15.97:0/3150253901 wait complete. 2024-03-06T07:31:05.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:05.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:05.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.808+0000 7feb77e00640 1 Processor -- start 2024-03-06T07:31:05.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.808+0000 7feb77e00640 1 -- start start 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.809+0000 7feb77e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feb700b3f70 0x7feb70160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.809+0000 7feb77e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb70071440 con 0x7feb70070ff0 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.809+0000 7feb77e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700b46c0 con 0x7feb7015fc10 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.809+0000 7feb77e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb700b49a0 con 0x7feb700b3f70 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.810+0000 7feb75e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feb70070ff0 0x7feb700ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50676/0 (socket says 172.21.15.97:50676) 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.810+0000 7feb75e00640 1 -- 172.21.15.97:0/2700530135 learned_addr learned my addr 172.21.15.97:0/2700530135 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.810+0000 7feb74a00640 1 --2- 172.21.15.97:0/2700530135 >> v2:172.21.15.135:3300/0 conn(0x7feb700b3f70 0x7feb70160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.810+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972871200 0 0) 0x7feb70071440 con 0x7feb70070ff0 2024-03-06T07:31:05.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.810+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb58003520 con 0x7feb70070ff0 2024-03-06T07:31:05.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811253333 0 0) 0x7feb58003520 con 0x7feb70070ff0 2024-03-06T07:31:05.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 >> v2:172.21.15.135:3300/0 conn(0x7feb700b3f70 msgr2=0x7feb70160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 --2- 172.21.15.97:0/2700530135 >> v2:172.21.15.135:3300/0 conn(0x7feb700b3f70 0x7feb70160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 >> v1:172.21.15.135:6789/0 conn(0x7feb7015fc10 legacy=0x7feb7015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb70072c90 con 0x7feb70070ff0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7feb680019f0 con 0x7feb70070ff0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 >> v1:172.21.15.97:6789/0 conn(0x7feb70070ff0 legacy=0x7feb700ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb580039b0 0x7feb58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb58004800 0x7feb58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb580051c0 0x7feb5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700b49a0 con 0x7feb580051c0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5800c060 con 0x7feb580039b0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5800c260 con 0x7feb58004800 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb75e00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb580051c0 0x7feb5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.811+0000 7feb75400640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb580039b0 0x7feb58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb74a00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb58004800 0x7feb58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb75e00640 1 -- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb58004800 msgr2=0x7feb58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb75e00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb58004800 0x7feb58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb75e00640 1 -- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb580039b0 msgr2=0x7feb58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb75e00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb580039b0 0x7feb58003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb75e00640 1 -- 172.21.15.97:0/2700530135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58003520 con 0x7feb580051c0 2024-03-06T07:31:05.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.812+0000 7feb74a00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb58004800 0x7feb58004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.813+0000 7feb75e00640 1 -- 172.21.15.97:0/2700530135 reap_dead start 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.813+0000 7feb75e00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb580051c0 0x7feb5800b9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7feb6800a850 tx=0x7feb58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9e36a1e5b30da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.813+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800a4a0 con 0x7feb580051c0 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.813+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb68034650 con 0x7feb580051c0 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.813+0000 7feb6fe00640 1 -- 172.21.15.97:0/2700530135 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68034a40 con 0x7feb580051c0 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 -- 172.21.15.97:0/2700530135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb580051c0 msgr2=0x7feb5800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb580051c0 0x7feb5800b9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7feb6800a850 tx=0x7feb58005830 comp rx=0 tx=0).stop 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 -- 172.21.15.97:0/2700530135 shutdown_connections 2024-03-06T07:31:05.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 --2- 172.21.15.97:0/2700530135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb580051c0 0x7feb5800b9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 -- 172.21.15.97:0/2700530135 >> 172.21.15.97:0/2700530135 conn(0x7feb700b1940 msgr2=0x7feb700b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 -- 172.21.15.97:0/2700530135 shutdown_connections 2024-03-06T07:31:05.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.814+0000 7feb77e00640 1 -- 172.21.15.97:0/2700530135 wait complete. 2024-03-06T07:31:05.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.816+0000 7feb77e00640 1 Processor -- start 2024-03-06T07:31:05.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.816+0000 7feb77e00640 1 -- start start 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb77e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb580051c0 0x7feb7015cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb77e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb700c4cb0 0x7feb701504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 0x7feb700afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb77e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb70072a30 con 0x7feb701f5270 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb77e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700728b0 con 0x7feb700c4cb0 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb77e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7015c2e0 con 0x7feb580051c0 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb75e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 0x7feb700afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb75e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 0x7feb700afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58236/0 (socket says 172.21.15.97:58236) 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.817+0000 7feb75e00640 1 -- 172.21.15.97:0/2669023532 learned_addr learned my addr 172.21.15.97:0/2669023532 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb74a00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb700c4cb0 0x7feb701504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75400640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb580051c0 0x7feb7015cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75e00640 1 -- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb580051c0 msgr2=0x7feb7015cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb580051c0 0x7feb7015cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75e00640 1 -- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb700c4cb0 msgr2=0x7feb701504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb700c4cb0 0x7feb701504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75e00640 1 -- 172.21.15.97:0/2669023532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb701f5be0 con 0x7feb701f5270 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb75e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 0x7feb700afa40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7feb6800a970 tx=0x7feb68001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68039040 con 0x7feb701f5270 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb68043070 con 0x7feb701f5270 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.818+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680019f0 con 0x7feb701f5270 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.819+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb701f5ec0 con 0x7feb701f5270 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.819+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7015d4f0 con 0x7feb701f5270 2024-03-06T07:31:05.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.820+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7004f330 con 0x7feb701f5270 2024-03-06T07:31:05.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.821+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb68034650 con 0x7feb701f5270 2024-03-06T07:31:05.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.822+0000 7feb6d600640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb5006b4d0 0x7feb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.822+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb680abb10 con 0x7feb701f5270 2024-03-06T07:31:05.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.822+0000 7feb75400640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb5006b4d0 0x7feb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.823+0000 7feb75400640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb5006b4d0 0x7feb5006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7feb64005490 tx=0x7feb64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:05.828+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb6807acd0 con 0x7feb701f5270 2024-03-06T07:31:06.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.025+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb7006ef30 con 0x7feb701f5270 2024-03-06T07:31:06.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.026+0000 7feb6d600640 1 -- 172.21.15.97:0/2669023532 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb6807a5d0 con 0x7feb701f5270 2024-03-06T07:31:06.028 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:31:06.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.028+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb5006b4d0 msgr2=0x7feb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.028+0000 7feb77e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb5006b4d0 0x7feb5006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7feb64005490 tx=0x7feb64009040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.028+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 msgr2=0x7feb700afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.028+0000 7feb77e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 0x7feb700afa40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7feb6800a970 tx=0x7feb68001dc0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 shutdown_connections 2024-03-06T07:31:06.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb5006b4d0 0x7feb5006d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb701f5270 0x7feb700afa40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb700c4cb0 0x7feb701504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 --2- 172.21.15.97:0/2669023532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb580051c0 0x7feb7015cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 >> 172.21.15.97:0/2669023532 conn(0x7feb700b1940 msgr2=0x7feb700713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 shutdown_connections 2024-03-06T07:31:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.029+0000 7feb77e00640 1 -- 172.21.15.97:0/2669023532 wait complete. 2024-03-06T07:31:06.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:31:06.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.243+0000 7f7d41c00640 1 Processor -- start 2024-03-06T07:31:06.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.243+0000 7f7d41c00640 1 -- start start 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.244+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 0x7f7d3c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.244+0000 7f7d41c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c15c240 con 0x7f7d3c1561f0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d41c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c15c500 con 0x7f7d3c0aedc0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d41c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c15ce30 con 0x7f7d3c0cc9d0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d3be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7d3c1561f0 0x7f7d3c0ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50684/0 (socket says 172.21.15.97:50684) 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d3be00640 1 -- 172.21.15.97:0/941875185 learned_addr learned my addr 172.21.15.97:0/941875185 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d41200640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 0x7f7d3c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009921288 0 0) 0x7f7d3c15c240 con 0x7f7d3c1561f0 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.245+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d20003520 con 0x7f7d3c1561f0 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3339638572 0 0) 0x7f7d20003520 con 0x7f7d3c1561f0 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 msgr2=0x7f7d3c06d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 0x7f7d3c06d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 >> v1:172.21.15.135:6789/0 conn(0x7f7d3c0aedc0 legacy=0x7f7d3c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c15c7e0 con 0x7f7d3c1561f0 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7d2c001670 con 0x7f7d3c1561f0 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 >> v1:172.21.15.97:6789/0 conn(0x7f7d3c1561f0 legacy=0x7f7d3c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d200039e0 0x7f7d20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 0x7f7d20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d20004d40 0x7f7d200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c15ce30 con 0x7f7d200039e0 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2000bad0 con 0x7f7d20004d40 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2000bcd0 con 0x7f7d3c0cc9d0 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3be00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d200039e0 0x7f7d20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3be00640 1 -- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 msgr2=0x7f7d20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3be00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c0cc9d0 0x7f7d20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d41200640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d20004d40 0x7f7d200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.246+0000 7f7d3be00640 1 -- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d20004d40 msgr2=0x7f7d200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.247+0000 7f7d3be00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d20004d40 0x7f7d200051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.247+0000 7f7d3be00640 1 -- 172.21.15.97:0/941875185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20003520 con 0x7f7d200039e0 2024-03-06T07:31:06.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.248+0000 7f7d3be00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d200039e0 0x7f7d20003e20 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00c0e0 tx=0x7f7d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c678021947c6c24b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.248+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c007c40 con 0x7f7d200039e0 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.248+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c030070 con 0x7f7d200039e0 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.248+0000 7f7d3aa00640 1 -- 172.21.15.97:0/941875185 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c005d30 con 0x7f7d200039e0 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.248+0000 7f7d41c00640 1 -- 172.21.15.97:0/941875185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d200039e0 msgr2=0x7f7d20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.248+0000 7f7d41c00640 1 --2- 172.21.15.97:0/941875185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d200039e0 0x7f7d20003e20 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00c0e0 tx=0x7f7d20005830 comp rx=0 tx=0).stop 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.249+0000 7f7d41200640 1 -- 172.21.15.97:0/941875185 reap_dead start 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.249+0000 7f7d41c00640 1 -- 172.21.15.97:0/941875185 shutdown_connections 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.249+0000 7f7d41c00640 1 -- 172.21.15.97:0/941875185 >> 172.21.15.97:0/941875185 conn(0x7f7d3c06afc0 msgr2=0x7f7d3c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.249+0000 7f7d41c00640 1 -- 172.21.15.97:0/941875185 shutdown_connections 2024-03-06T07:31:06.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.249+0000 7f7d41c00640 1 -- 172.21.15.97:0/941875185 wait complete. 2024-03-06T07:31:06.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.250+0000 7f7d41c00640 1 Processor -- start 2024-03-06T07:31:06.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 -- start start 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d200039e0 0x7f7d3c14d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 0x7f7d3c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c15b780 0x7f7d3c158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c156750 con 0x7f7d3c153050 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c1565d0 con 0x7f7d200039e0 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.251+0000 7f7d41c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c06d350 con 0x7f7d3c15b780 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 0x7f7d3c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 0x7f7d3c15b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58252/0 (socket says 172.21.15.97:58252) 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 -- 172.21.15.97:0/157578719 learned_addr learned my addr 172.21.15.97:0/157578719 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3be00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d200039e0 0x7f7d3c14d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d41200640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c15b780 0x7f7d3c158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 -- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c15b780 msgr2=0x7f7d3c158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c15b780 0x7f7d3c158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 -- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d200039e0 msgr2=0x7f7d3c14d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d200039e0 0x7f7d3c14d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3b400640 1 -- 172.21.15.97:0/157578719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c14dbd0 con 0x7f7d3c153050 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d3be00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d200039e0 0x7f7d3c14d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.252+0000 7f7d41200640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c15b780 0x7f7d3c158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.253+0000 7f7d3b400640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 0x7f7d3c15b240 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800c500 tx=0x7f7d2800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.253+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280052d0 con 0x7f7d3c153050 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.253+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28005470 con 0x7f7d3c153050 2024-03-06T07:31:06.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.253+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28012870 con 0x7f7d3c153050 2024-03-06T07:31:06.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.253+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c159910 con 0x7f7d3c153050 2024-03-06T07:31:06.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.253+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c152850 con 0x7f7d3c153050 2024-03-06T07:31:06.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.254+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c04f330 con 0x7f7d3c153050 2024-03-06T07:31:06.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.255+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d28012a10 con 0x7f7d3c153050 2024-03-06T07:31:06.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.256+0000 7f7d33e00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d1c06b500 0x7f7d1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.256+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7d28089b50 con 0x7f7d3c153050 2024-03-06T07:31:06.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.256+0000 7f7d3be00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d1c06b500 0x7f7d1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.257+0000 7f7d3be00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d1c06b500 0x7f7d1c06d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c034000 tx=0x7f7d2c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.261+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d3c14dbd0 con 0x7f7d3c153050 2024-03-06T07:31:06.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.458+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7d3c0b2b10 con 0x7f7d3c153050 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.459+0000 7f7d33e00640 1 -- 172.21.15.97:0/157578719 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7d28058d90 con 0x7f7d3c153050 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.461+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d1c06b500 msgr2=0x7f7d1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.461+0000 7f7d41c00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d1c06b500 0x7f7d1c06d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c034000 tx=0x7f7d2c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.461+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 msgr2=0x7f7d3c15b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.461+0000 7f7d41c00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 0x7f7d3c15b240 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800c500 tx=0x7f7d2800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 shutdown_connections 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d1c06b500 0x7f7d1c06d9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3c15b780 0x7f7d3c158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d3c153050 0x7f7d3c15b240 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 --2- 172.21.15.97:0/157578719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d200039e0 0x7f7d3c14d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 >> 172.21.15.97:0/157578719 conn(0x7f7d3c06afc0 msgr2=0x7f7d3c155370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 shutdown_connections 2024-03-06T07:31:06.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.462+0000 7f7d41c00640 1 -- 172.21.15.97:0/157578719 wait complete. 2024-03-06T07:31:06.482 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:31:06.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:31:06.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.691+0000 7fa30ee00640 1 Processor -- start 2024-03-06T07:31:06.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.691+0000 7fa30ee00640 1 -- start start 2024-03-06T07:31:06.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ee00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa308145980 0x7fa30815fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3080ced20 0x7fa30815ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ee00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa3080712d0 0x7fa308071740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ee00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa308070e40 con 0x7fa308145980 2024-03-06T07:31:06.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ee00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080cf1e0 con 0x7fa3080712d0 2024-03-06T07:31:06.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080cf4c0 con 0x7fa3080ced20 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa307e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa308145980 0x7fa30815fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa307e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa308145980 0x7fa30815fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58256/0 (socket says 172.21.15.97:58256) 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa307e00640 1 -- 172.21.15.97:0/2801371117 learned_addr learned my addr 172.21.15.97:0/2801371117 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.693+0000 7fa30ce00640 1 --2- 172.21.15.97:0/2801371117 >> v2:172.21.15.135:3300/0 conn(0x7fa3080712d0 0x7fa308071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.694+0000 7fa307e00640 1 -- 172.21.15.97:0/2801371117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3080ced20 msgr2=0x7fa30815ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.694+0000 7fa307e00640 1 --2- 172.21.15.97:0/2801371117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3080ced20 0x7fa30815ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.694+0000 7fa307e00640 1 -- 172.21.15.97:0/2801371117 >> v2:172.21.15.135:3300/0 conn(0x7fa3080712d0 msgr2=0x7fa308071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.694+0000 7fa307e00640 1 --2- 172.21.15.97:0/2801371117 >> v2:172.21.15.135:3300/0 conn(0x7fa3080712d0 0x7fa308071740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.694+0000 7fa307e00640 1 -- 172.21.15.97:0/2801371117 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080cf7a0 con 0x7fa308145980 2024-03-06T07:31:06.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.695+0000 7fa307e00640 1 --2- 172.21.15.97:0/2801371117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa308145980 0x7fa30815fa00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa308074fe0 tx=0x7fa2f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816761abcedf4e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.695+0000 7fa306a00640 1 -- 172.21.15.97:0/2801371117 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8035020 con 0x7fa308145980 2024-03-06T07:31:06.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.695+0000 7fa306a00640 1 -- 172.21.15.97:0/2801371117 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8031870 con 0x7fa308145980 2024-03-06T07:31:06.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.695+0000 7fa306a00640 1 -- 172.21.15.97:0/2801371117 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8031c40 con 0x7fa308145980 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 -- 172.21.15.97:0/2801371117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa308145980 msgr2=0x7fa30815fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 --2- 172.21.15.97:0/2801371117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa308145980 0x7fa30815fa00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa308074fe0 tx=0x7fa2f802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 -- 172.21.15.97:0/2801371117 shutdown_connections 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 --2- 172.21.15.97:0/2801371117 >> v2:172.21.15.135:3300/0 conn(0x7fa3080712d0 0x7fa308071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 --2- 172.21.15.97:0/2801371117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3080ced20 0x7fa30815ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 --2- 172.21.15.97:0/2801371117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa308145980 0x7fa30815fa00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 -- 172.21.15.97:0/2801371117 >> 172.21.15.97:0/2801371117 conn(0x7fa3080cfa80 msgr2=0x7fa3080cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 -- 172.21.15.97:0/2801371117 shutdown_connections 2024-03-06T07:31:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.696+0000 7fa30ee00640 1 -- 172.21.15.97:0/2801371117 wait complete. 2024-03-06T07:31:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.698+0000 7fa30ee00640 1 Processor -- start 2024-03-06T07:31:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.698+0000 7fa30ee00640 1 -- start start 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3080712d0 0x7fa3080aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 0x7fa3081f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa308145980 0x7fa3081f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3081608c0 con 0x7fa3080ced20 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa308160740 con 0x7fa3080712d0 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa308071c80 con 0x7fa308145980 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 0x7fa3081f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 0x7fa3081f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58258/0 (socket says 172.21.15.97:58258) 2024-03-06T07:31:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 -- 172.21.15.97:0/4016160820 learned_addr learned my addr 172.21.15.97:0/4016160820 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa30ce00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa308145980 0x7fa3081f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307e00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3080712d0 0x7fa3080aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 -- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa308145980 msgr2=0x7fa3081f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa308145980 0x7fa3081f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 -- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3080712d0 msgr2=0x7fa3080aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.699+0000 7fa307400640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3080712d0 0x7fa3080aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa307400640 1 -- 172.21.15.97:0/4016160820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30812fad0 con 0x7fa3080ced20 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa307400640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 0x7fa3081f3300 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b4a0 tx=0x7fa2f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4014070 con 0x7fa3080ced20 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4010610 con 0x7fa3080ced20 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30812fdb0 con 0x7fa3080ced20 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4010a00 con 0x7fa3080ced20 2024-03-06T07:31:06.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.700+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa308130370 con 0x7fa3080ced20 2024-03-06T07:31:06.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.701+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30804f330 con 0x7fa3080ced20 2024-03-06T07:31:06.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.702+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2f4017070 con 0x7fa3080ced20 2024-03-06T07:31:06.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.703+0000 7fa2ffe00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2e806b4d0 0x7fa2e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.703+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2f4089950 con 0x7fa3080ced20 2024-03-06T07:31:06.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.703+0000 7fa307e00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2e806b4d0 0x7fa2e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.704+0000 7fa307e00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2e806b4d0 0x7fa2e806d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8036000 tx=0x7fa2f8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.708+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2f4058a80 con 0x7fa3080ced20 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:06.908+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa308072920 con 0x7fa3080ced20 2024-03-06T07:31:07.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:07.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:07.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.659+0000 7fa2ffe00640 1 -- 172.21.15.97:0/4016160820 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa2f4058380 con 0x7fa3080ced20 2024-03-06T07:31:07.661 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:31:07.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.664+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2e806b4d0 msgr2=0x7fa2e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.664+0000 7fa30ee00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2e806b4d0 0x7fa2e806d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8036000 tx=0x7fa2f8008030 comp rx=0 tx=0).stop 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.664+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 msgr2=0x7fa3081f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.664+0000 7fa30ee00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 0x7fa3081f3300 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b4a0 tx=0x7fa2f400b960 comp rx=0 tx=0).stop 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.665+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 shutdown_connections 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.665+0000 7fa30ee00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2e806b4d0 0x7fa2e806d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.665+0000 7fa30ee00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa308145980 0x7fa3081f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.665+0000 7fa30ee00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3080ced20 0x7fa3081f3300 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.665+0000 7fa30ee00640 1 --2- 172.21.15.97:0/4016160820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3080712d0 0x7fa3080aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.665+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 >> 172.21.15.97:0/4016160820 conn(0x7fa3080cfa80 msgr2=0x7fa3080b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.666+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 shutdown_connections 2024-03-06T07:31:07.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.666+0000 7fa30ee00640 1 -- 172.21.15.97:0/4016160820 wait complete. 2024-03-06T07:31:07.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:31:07.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.886+0000 7f742d000640 1 Processor -- start 2024-03-06T07:31:07.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.886+0000 7f742d000640 1 -- start start 2024-03-06T07:31:07.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.887+0000 7f742d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f742815cb90 0x7f742806d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.887+0000 7f742d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 0x7f742806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.887+0000 7f742d000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742815cf70 con 0x7f7428163800 2024-03-06T07:31:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.887+0000 7f742d000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74280b3890 con 0x7f742815cb90 2024-03-06T07:31:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.887+0000 7f742d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742806c990 con 0x7f742806e060 2024-03-06T07:31:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.888+0000 7f7426800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 0x7f742806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.888+0000 7f7425e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f742815cb90 0x7f742806d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.888+0000 7f7426800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 0x7f742806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:34708/0 (socket says 172.21.15.97:34708) 2024-03-06T07:31:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.888+0000 7f7426800640 1 -- 172.21.15.97:0/2387061053 learned_addr learned my addr 172.21.15.97:0/2387061053 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.889+0000 7f7425e00640 1 -- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 msgr2=0x7f742806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.889+0000 7f7425e00640 1 --2- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 0x7f742806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.889+0000 7f7425e00640 1 -- 172.21.15.97:0/2387061053 >> v1:172.21.15.135:6789/0 conn(0x7f7428163800 legacy=0x7f74280b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.889+0000 7f7425e00640 1 -- 172.21.15.97:0/2387061053 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280b35b0 con 0x7f742815cb90 2024-03-06T07:31:07.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.889+0000 7f7426800640 1 --2- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 0x7f742806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:07.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.890+0000 7f7425e00640 1 --2- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742815cb90 0x7f742806d6c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74180097a0 tx=0x7f741802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=702f0f2f02612be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f7424a00640 1 -- 172.21.15.97:0/2387061053 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418035020 con 0x7f742815cb90 2024-03-06T07:31:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f7424a00640 1 -- 172.21.15.97:0/2387061053 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7418031810 con 0x7f742815cb90 2024-03-06T07:31:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f7424a00640 1 -- 172.21.15.97:0/2387061053 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418031be0 con 0x7f742815cb90 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 -- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742815cb90 msgr2=0x7f742806d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 --2- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742815cb90 0x7f742806d6c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74180097a0 tx=0x7f741802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 -- 172.21.15.97:0/2387061053 shutdown_connections 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 --2- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f742806e060 0x7f742806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 --2- 172.21.15.97:0/2387061053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742815cb90 0x7f742806d6c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 -- 172.21.15.97:0/2387061053 >> 172.21.15.97:0/2387061053 conn(0x7f74280b1940 msgr2=0x7f74280b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 -- 172.21.15.97:0/2387061053 shutdown_connections 2024-03-06T07:31:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.891+0000 7f742d000640 1 -- 172.21.15.97:0/2387061053 wait complete. 2024-03-06T07:31:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.893+0000 7f742d000640 1 Processor -- start 2024-03-06T07:31:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.893+0000 7f742d000640 1 -- start start 2024-03-06T07:31:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f742d000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742806e060 0x7f742815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f742d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 0x7f7428158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f742d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7428163800 0x7f74281eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f742d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74280b3350 con 0x7f742815cb90 2024-03-06T07:31:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f742d000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742806d230 con 0x7f742806e060 2024-03-06T07:31:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f742d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742806cec0 con 0x7f7428163800 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.894+0000 7f7425400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 0x7f7428158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7426800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7428163800 0x7f74281eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742806e060 0x7f742815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 0x7f7428158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58268/0 (socket says 172.21.15.97:58268) 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 -- 172.21.15.97:0/1063492034 learned_addr learned my addr 172.21.15.97:0/1063492034 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 -- 172.21.15.97:0/1063492034 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7428163800 msgr2=0x7f74281eabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7428163800 0x7f74281eabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 -- 172.21.15.97:0/1063492034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742806e060 msgr2=0x7f742815af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742806e060 0x7f742815af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.895+0000 7f7425400640 1 -- 172.21.15.97:0/1063492034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7428159e60 con 0x7f742815cb90 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.896+0000 7f7425400640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 0x7f7428158350 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f741400c3f0 tx=0x7f741400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.896+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74140052d0 con 0x7f742815cb90 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.896+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414005490 con 0x7f742815cb90 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.896+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414007260 con 0x7f742815cb90 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.896+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742815a140 con 0x7f742815cb90 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.896+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742815a700 con 0x7f742815cb90 2024-03-06T07:31:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.897+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742804f330 con 0x7f742815cb90 2024-03-06T07:31:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.898+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7414007400 con 0x7f742815cb90 2024-03-06T07:31:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.899+0000 7f741e000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f740806b4d0 0x7f740806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.899+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7414089420 con 0x7f742815cb90 2024-03-06T07:31:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.899+0000 7f7425e00640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f740806b4d0 0x7f740806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.900+0000 7f7425e00640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f740806b4d0 0x7f740806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7418036000 tx=0x7f7418008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:07.904+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7414057d90 con 0x7f742815cb90 2024-03-06T07:31:08.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.101+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7428158890 con 0x7f742815cb90 2024-03-06T07:31:08.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.665+0000 7f741e000640 1 -- 172.21.15.97:0/1063492034 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) v1 ==== 170+0+0 (secure 0 0 0) 0x7f741405be30 con 0x7f742815cb90 2024-03-06T07:31:08.668 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:31:08.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f740806b4d0 msgr2=0x7f740806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f740806b4d0 0x7f740806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7418036000 tx=0x7f7418008030 comp rx=0 tx=0).stop 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 msgr2=0x7f7428158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 0x7f7428158350 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f741400c3f0 tx=0x7f741400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 shutdown_connections 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f740806b4d0 0x7f740806d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7428163800 0x7f74281eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742815cb90 0x7f7428158350 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 --2- 172.21.15.97:0/1063492034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742806e060 0x7f742815af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.669+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 >> 172.21.15.97:0/1063492034 conn(0x7f74280b1940 msgr2=0x7f74280ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.670+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 shutdown_connections 2024-03-06T07:31:08.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.670+0000 7f742d000640 1 -- 172.21.15.97:0/1063492034 wait complete. 2024-03-06T07:31:08.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:31:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.882+0000 7f6b91e00640 1 Processor -- start 2024-03-06T07:31:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.882+0000 7f6b91e00640 1 -- start start 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6b8c158570 0x7f6b8c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c158950 0x7f6b8c0cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b8c0aebf0 0x7f6b8c0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c15b130 con 0x7f6b8c0aebf0 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c0af170 con 0x7f6b8c158570 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c15b3f0 con 0x7f6b8c158950 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b8c0aebf0 0x7f6b8c0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b8c0aebf0 0x7f6b8c0cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58282/0 (socket says 172.21.15.97:58282) 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b91400640 1 -- 172.21.15.97:0/55084965 learned_addr learned my addr 172.21.15.97:0/55084965 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b83e00640 1 --2- 172.21.15.97:0/55084965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c158950 0x7f6b8c0cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.884+0000 7f6b90a00640 1 --2- 172.21.15.97:0/55084965 >> v2:172.21.15.135:3300/0 conn(0x7f6b8c158570 0x7f6b8c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.885+0000 7f6b91400640 1 -- 172.21.15.97:0/55084965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c158950 msgr2=0x7f6b8c0cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.885+0000 7f6b91400640 1 --2- 172.21.15.97:0/55084965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c158950 0x7f6b8c0cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.885+0000 7f6b91400640 1 -- 172.21.15.97:0/55084965 >> v2:172.21.15.135:3300/0 conn(0x7f6b8c158570 msgr2=0x7f6b8c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.885+0000 7f6b91400640 1 --2- 172.21.15.97:0/55084965 >> v2:172.21.15.135:3300/0 conn(0x7f6b8c158570 0x7f6b8c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.885+0000 7f6b91400640 1 -- 172.21.15.97:0/55084965 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c15b6d0 con 0x7f6b8c0aebf0 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.886+0000 7f6b91400640 1 --2- 172.21.15.97:0/55084965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c0aebf0 0x7f6b8c0cfc10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c06eb20 tx=0x7f6b8402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780d38d0f7e722cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.886+0000 7f6b8be00640 1 -- 172.21.15.97:0/55084965 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8402c9e0 con 0x7f6b8c0aebf0 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.886+0000 7f6b8be00640 1 -- 172.21.15.97:0/55084965 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8402cb80 con 0x7f6b8c0aebf0 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.886+0000 7f6b8be00640 1 -- 172.21.15.97:0/55084965 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b840343f0 con 0x7f6b8c0aebf0 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 -- 172.21.15.97:0/55084965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c0aebf0 msgr2=0x7f6b8c0cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 --2- 172.21.15.97:0/55084965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c0aebf0 0x7f6b8c0cfc10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c06eb20 tx=0x7f6b8402be40 comp rx=0 tx=0).stop 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 -- 172.21.15.97:0/55084965 shutdown_connections 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 --2- 172.21.15.97:0/55084965 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c0aebf0 0x7f6b8c0cfc10 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 --2- 172.21.15.97:0/55084965 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c158950 0x7f6b8c0cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 --2- 172.21.15.97:0/55084965 >> v2:172.21.15.135:3300/0 conn(0x7f6b8c158570 0x7f6b8c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 -- 172.21.15.97:0/55084965 >> 172.21.15.97:0/55084965 conn(0x7f6b8c06afc0 msgr2=0x7f6b8c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 -- 172.21.15.97:0/55084965 shutdown_connections 2024-03-06T07:31:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.887+0000 7f6b91e00640 1 -- 172.21.15.97:0/55084965 wait complete. 2024-03-06T07:31:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.888+0000 7f6b91e00640 1 Processor -- start 2024-03-06T07:31:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 -- start start 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c0aebf0 0x7f6b8c14c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 0x7f6b8c158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c158950 0x7f6b8c159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c15bed0 con 0x7f6b8c158570 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c0b4950 con 0x7f6b8c158950 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.889+0000 7f6b91e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c15c050 con 0x7f6b8c0aebf0 2024-03-06T07:31:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 0x7f6b8c158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 0x7f6b8c158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58298/0 (socket says 172.21.15.97:58298) 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 -- 172.21.15.97:0/415098286 learned_addr learned my addr 172.21.15.97:0/415098286 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 -- 172.21.15.97:0/415098286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c0aebf0 msgr2=0x7f6b8c14c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c0aebf0 0x7f6b8c14c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 -- 172.21.15.97:0/415098286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c158950 msgr2=0x7f6b8c159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c158950 0x7f6b8c159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 -- 172.21.15.97:0/415098286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c14d830 con 0x7f6b8c158570 2024-03-06T07:31:08.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.890+0000 7f6b83e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 0x7f6b8c158d50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400d800 tx=0x7f6b7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.891+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400b6c0 con 0x7f6b8c158570 2024-03-06T07:31:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.891+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b74019070 con 0x7f6b8c158570 2024-03-06T07:31:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.891+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400ba80 con 0x7f6b8c158570 2024-03-06T07:31:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.891+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c15ac40 con 0x7f6b8c158570 2024-03-06T07:31:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.891+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c15a680 con 0x7f6b8c158570 2024-03-06T07:31:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.892+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c04f330 con 0x7f6b8c158570 2024-03-06T07:31:08.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.893+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b7400bc20 con 0x7f6b8c158570 2024-03-06T07:31:08.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.894+0000 7f6b89600640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b6806b4d0 0x7f6b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.894+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6b7408af70 con 0x7f6b8c158570 2024-03-06T07:31:08.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.894+0000 7f6b90a00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b6806b4d0 0x7f6b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.895+0000 7f6b90a00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b6806b4d0 0x7f6b6806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c006fd0 tx=0x7f6b7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:08.898+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b74059fc0 con 0x7f6b8c158570 2024-03-06T07:31:09.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.129+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6b8c0ced50 con 0x7f6b8c158570 2024-03-06T07:31:09.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.711+0000 7f6b89600640 1 -- 172.21.15.97:0/415098286 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6b740598c0 con 0x7f6b8c158570 2024-03-06T07:31:09.714 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:31:09.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b6806b4d0 msgr2=0x7f6b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b6806b4d0 0x7f6b6806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c006fd0 tx=0x7f6b7c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:09.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 msgr2=0x7f6b8c158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 0x7f6b8c158d50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400d800 tx=0x7f6b7400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 shutdown_connections 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b6806b4d0 0x7f6b6806d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c158950 0x7f6b8c159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b8c158570 0x7f6b8c158d50 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 --2- 172.21.15.97:0/415098286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b8c0aebf0 0x7f6b8c14c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.718+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 >> 172.21.15.97:0/415098286 conn(0x7f6b8c06afc0 msgr2=0x7f6b8c0ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.719+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 shutdown_connections 2024-03-06T07:31:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.720+0000 7f6b91e00640 1 -- 172.21.15.97:0/415098286 wait complete. 2024-03-06T07:31:09.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:09.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.928+0000 7f9763c00640 1 Processor -- start 2024-03-06T07:31:09.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.928+0000 7f9763c00640 1 -- start start 2024-03-06T07:31:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f9763c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f975c0af1c0 0x7f975c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f9763c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c071d30 con 0x7f975c070d20 2024-03-06T07:31:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f9763c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0711d0 con 0x7f975c06e4d0 2024-03-06T07:31:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f9763c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975c071490 con 0x7f975c0af1c0 2024-03-06T07:31:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f975c070d20 0x7f975c06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50690/0 (socket says 172.21.15.97:50690) 2024-03-06T07:31:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975be00640 1 -- 172.21.15.97:0/3945509095 learned_addr learned my addr 172.21.15.97:0/3945509095 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:09.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f9761200640 1 --2- 172.21.15.97:0/3945509095 >> v2:172.21.15.135:3300/0 conn(0x7f975c0af1c0 0x7f975c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455969078 0 0) 0x7f9748001110 con 0x7f975c070d20 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9744003520 con 0x7f975c070d20 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1145101207 0 0) 0x7f9744003520 con 0x7f975c070d20 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 >> v2:172.21.15.135:3300/0 conn(0x7f975c0af1c0 msgr2=0x7f975c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 --2- 172.21.15.97:0/3945509095 >> v2:172.21.15.135:3300/0 conn(0x7f975c0af1c0 0x7f975c15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 >> v1:172.21.15.135:6789/0 conn(0x7f975c06e4d0 legacy=0x7f975c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c071770 con 0x7f975c070d20 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9748001840 con 0x7f975c070d20 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 >> v1:172.21.15.97:6789/0 conn(0x7f975c070d20 legacy=0x7f975c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9744003a00 0x7f9744003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f9744004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9744005240 0x7f974400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c071490 con 0x7f9744004800 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974400c160 con 0x7f9744005240 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974400c360 con 0x7f9744003a00 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.931+0000 7f975be00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f9744004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f9761200640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9744003a00 0x7f9744003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f975be00640 1 -- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9744003a00 msgr2=0x7f9744003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f975be00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9744003a00 0x7f9744003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f9761c00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9744005240 0x7f974400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f975be00640 1 -- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9744005240 msgr2=0x7f974400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f975be00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9744005240 0x7f974400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f975be00640 1 -- 172.21.15.97:0/3945509095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9744003520 con 0x7f9744004800 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f9761c00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9744005240 0x7f974400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.932+0000 7f9761c00640 1 -- 172.21.15.97:0/3945509095 reap_dead start 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.933+0000 7f975be00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f9744004c80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f974800c320 tx=0x7f9744005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42097b44f0d2f9a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.933+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974802cc20 con 0x7f9744004800 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.933+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9748030070 con 0x7f9744004800 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.933+0000 7f975b400640 1 -- 172.21.15.97:0/3945509095 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97480058d0 con 0x7f9744004800 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 -- 172.21.15.97:0/3945509095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 msgr2=0x7f9744004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f9744004c80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f974800c320 tx=0x7f9744005830 comp rx=0 tx=0).stop 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 -- 172.21.15.97:0/3945509095 shutdown_connections 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 --2- 172.21.15.97:0/3945509095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f9744004c80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 -- 172.21.15.97:0/3945509095 >> 172.21.15.97:0/3945509095 conn(0x7f975c0b1730 msgr2=0x7f975c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 -- 172.21.15.97:0/3945509095 shutdown_connections 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.934+0000 7f9763c00640 1 -- 172.21.15.97:0/3945509095 wait complete. 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.936+0000 7f9763c00640 1 Processor -- start 2024-03-06T07:31:09.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.936+0000 7f9763c00640 1 -- start start 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9763c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f975c160e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9763c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f975c15d170 0x7f975c14a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9763c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f975c15d5f0 0x7f975c14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9763c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c15eff0 con 0x7f9744004800 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9763c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0b30b0 con 0x7f975c15d170 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9763c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c15f170 con 0x7f975c15d5f0 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f975c160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f975c160e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58304/0 (socket says 172.21.15.97:58304) 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 -- 172.21.15.97:0/1372779820 learned_addr learned my addr 172.21.15.97:0/1372779820 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 -- 172.21.15.97:0/1372779820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f975c15d5f0 msgr2=0x7f975c14a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f975c15d5f0 0x7f975c14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 -- 172.21.15.97:0/1372779820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f975c15d170 msgr2=0x7f975c14a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f975c15d170 0x7f975c14a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.937+0000 7f9761200640 1 -- 172.21.15.97:0/1372779820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c15f6a0 con 0x7f9744004800 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.938+0000 7f9761200640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f975c160e40 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f975000e7e0 tx=0x7f975000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.938+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750017070 con 0x7f9744004800 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.938+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975000c6d0 con 0x7f9744004800 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.938+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000f040 con 0x7f9744004800 2024-03-06T07:31:09.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.938+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975c15f980 con 0x7f9744004800 2024-03-06T07:31:09.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.938+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975c15ff40 con 0x7f9744004800 2024-03-06T07:31:09.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.939+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c04f330 con 0x7f9744004800 2024-03-06T07:31:09.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.940+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f975001a070 con 0x7f9744004800 2024-03-06T07:31:09.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.941+0000 7f9758c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f973c06b4d0 0x7f973c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.941+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9750089eb0 con 0x7f9744004800 2024-03-06T07:31:09.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.941+0000 7f975be00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f973c06b4d0 0x7f973c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.942+0000 7f975be00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f973c06b4d0 0x7f973c06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9748034000 tx=0x7f9748007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:09.945+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9750058c90 con 0x7f9744004800 2024-03-06T07:31:10.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.136+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f975c160320 con 0x7f9744004800 2024-03-06T07:31:10.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.138+0000 7f9758c00640 1 -- 172.21.15.97:0/1372779820 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f9750058590 con 0x7f9744004800 2024-03-06T07:31:10.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:10.141 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":68,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:09.131559+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.140+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f973c06b4d0 msgr2=0x7f973c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.140+0000 7f9763c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f973c06b4d0 0x7f973c06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9748034000 tx=0x7f9748007000 comp rx=0 tx=0).stop 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.140+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 msgr2=0x7f975c160e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f975c160e40 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f975000e7e0 tx=0x7f975000eca0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 shutdown_connections 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f973c06b4d0 0x7f973c06d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f975c15d5f0 0x7f975c14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f975c15d170 0x7f975c14a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 --2- 172.21.15.97:0/1372779820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9744004800 0x7f975c160e40 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 >> 172.21.15.97:0/1372779820 conn(0x7f975c0b1730 msgr2=0x7f975c15e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 shutdown_connections 2024-03-06T07:31:10.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.141+0000 7f9763c00640 1 -- 172.21.15.97:0/1372779820 wait complete. 2024-03-06T07:31:10.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:10.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.357+0000 7f7d33800640 1 Processor -- start 2024-03-06T07:31:10.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.357+0000 7f7d33800640 1 -- start start 2024-03-06T07:31:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d33800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7d2c144c00 0x7f7d2c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d33800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c06ee70 con 0x7f7d2c15f9d0 2024-03-06T07:31:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d33800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c06cef0 con 0x7f7d2c15fe60 2024-03-06T07:31:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d33800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c06d1d0 con 0x7f7d2c144c00 2024-03-06T07:31:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d23e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7d2c15f9d0 0x7f7d2c0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50696/0 (socket says 172.21.15.97:50696) 2024-03-06T07:31:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d23e00640 1 -- 172.21.15.97:0/1397103139 learned_addr learned my addr 172.21.15.97:0/1397103139 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.359+0000 7f7d30e00640 1 --2- 172.21.15.97:0/1397103139 >> v2:172.21.15.135:3300/0 conn(0x7f7d2c144c00 0x7f7d2c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744413561 0 0) 0x7f7d14001110 con 0x7f7d2c15f9d0 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d10003520 con 0x7f7d2c15f9d0 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2983987738 0 0) 0x7f7d10003520 con 0x7f7d2c15f9d0 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 >> v2:172.21.15.135:3300/0 conn(0x7f7d2c144c00 msgr2=0x7f7d2c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 --2- 172.21.15.97:0/1397103139 >> v2:172.21.15.135:3300/0 conn(0x7f7d2c144c00 0x7f7d2c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 >> v1:172.21.15.135:6789/0 conn(0x7f7d2c15fe60 legacy=0x7f7d2c0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c06d4b0 con 0x7f7d2c15f9d0 2024-03-06T07:31:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7d14001840 con 0x7f7d2c15f9d0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 >> v1:172.21.15.97:6789/0 conn(0x7f7d2c15f9d0 legacy=0x7f7d2c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d100039b0 0x7f7d10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d10004800 0x7f7d10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d100051c0 0x7f7d1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c06d1d0 con 0x7f7d100051c0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1000c060 con 0x7f7d100039b0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1000c260 con 0x7f7d10004800 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.360+0000 7f7d31800640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d100051c0 0x7f7d1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d30e00640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d100039b0 0x7f7d10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d23e00640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d10004800 0x7f7d10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d31800640 1 -- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d10004800 msgr2=0x7f7d10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d31800640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d10004800 0x7f7d10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d31800640 1 -- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d100039b0 msgr2=0x7f7d10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d31800640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d100039b0 0x7f7d10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d31800640 1 -- 172.21.15.97:0/1397103139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d10003520 con 0x7f7d100051c0 2024-03-06T07:31:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.361+0000 7f7d23e00640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d10004800 0x7f7d10004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.362+0000 7f7d31800640 1 -- 172.21.15.97:0/1397103139 reap_dead start 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.362+0000 7f7d31800640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d100051c0 0x7f7d1000b9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f7d24001e50 tx=0x7f7d10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f10871baff2d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.362+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24001a70 con 0x7f7d100051c0 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.362+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d24003b90 con 0x7f7d100051c0 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.362+0000 7f7d2be00640 1 -- 172.21.15.97:0/1397103139 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240050d0 con 0x7f7d100051c0 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 -- 172.21.15.97:0/1397103139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d100051c0 msgr2=0x7f7d1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d100051c0 0x7f7d1000b9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f7d24001e50 tx=0x7f7d10005830 comp rx=0 tx=0).stop 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 -- 172.21.15.97:0/1397103139 shutdown_connections 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 --2- 172.21.15.97:0/1397103139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d100051c0 0x7f7d1000b9c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 -- 172.21.15.97:0/1397103139 >> 172.21.15.97:0/1397103139 conn(0x7f7d2c0b1730 msgr2=0x7f7d2c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 -- 172.21.15.97:0/1397103139 shutdown_connections 2024-03-06T07:31:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.363+0000 7f7d33800640 1 -- 172.21.15.97:0/1397103139 wait complete. 2024-03-06T07:31:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.365+0000 7f7d33800640 1 Processor -- start 2024-03-06T07:31:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.365+0000 7f7d33800640 1 -- start start 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d33800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d100051c0 0x7f7d2c1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d33800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d2c13c800 0x7f7d2c139c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d33800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 0x7f7d2c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d33800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c160730 con 0x7f7d2c153820 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d33800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c160270 con 0x7f7d2c13c800 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d33800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c06cb50 con 0x7f7d100051c0 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d31800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 0x7f7d2c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d31800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 0x7f7d2c0ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58312/0 (socket says 172.21.15.97:58312) 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d31800640 1 -- 172.21.15.97:0/2565276675 learned_addr learned my addr 172.21.15.97:0/2565276675 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.366+0000 7f7d30e00640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d100051c0 0x7f7d2c1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d23e00640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d2c13c800 0x7f7d2c139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d31800640 1 -- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d100051c0 msgr2=0x7f7d2c1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d31800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d100051c0 0x7f7d2c1531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d31800640 1 -- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d2c13c800 msgr2=0x7f7d2c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d31800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d2c13c800 0x7f7d2c139c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d31800640 1 -- 172.21.15.97:0/2565276675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0b4750 con 0x7f7d2c153820 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d23e00640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d2c13c800 0x7f7d2c139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d31800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 0x7f7d2c0ad860 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7d24001f70 tx=0x7f7d240014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24006d50 con 0x7f7d2c153820 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d24039480 con 0x7f7d2c153820 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24039830 con 0x7f7d2c153820 2024-03-06T07:31:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c0b4a30 con 0x7f7d2c153820 2024-03-06T07:31:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.367+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c13c2e0 con 0x7f7d2c153820 2024-03-06T07:31:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.368+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c15ed30 con 0x7f7d2c153820 2024-03-06T07:31:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.369+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d240399d0 con 0x7f7d2c153820 2024-03-06T07:31:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.370+0000 7f7d29600640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0806b4d0 0x7f7d0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.370+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d240ac240 con 0x7f7d2c153820 2024-03-06T07:31:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.370+0000 7f7d30e00640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0806b4d0 0x7f7d0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.371+0000 7f7d30e00640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0806b4d0 0x7f7d0806d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009990 tx=0x7f7d1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.374+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d24033030 con 0x7f7d2c153820 2024-03-06T07:31:10.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.565+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d2c04f330 con 0x7f7d2c153820 2024-03-06T07:31:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.567+0000 7f7d29600640 1 -- 172.21.15.97:0/2565276675 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f7d2407a9e0 con 0x7f7d2c153820 2024-03-06T07:31:10.570 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:10.570 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":68,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:09.131559+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0806b4d0 msgr2=0x7f7d0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0806b4d0 0x7f7d0806d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009990 tx=0x7f7d1c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 msgr2=0x7f7d2c0ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 0x7f7d2c0ad860 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7d24001f70 tx=0x7f7d240014a0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 shutdown_connections 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0806b4d0 0x7f7d0806d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d2c153820 0x7f7d2c0ad860 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d2c13c800 0x7f7d2c139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 --2- 172.21.15.97:0/2565276675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d100051c0 0x7f7d2c1531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 >> 172.21.15.97:0/2565276675 conn(0x7f7d2c0b1730 msgr2=0x7f7d2c071800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 shutdown_connections 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.571+0000 7f7d33800640 1 -- 172.21.15.97:0/2565276675 wait complete. 2024-03-06T07:31:10.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:31:10.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.838+0000 7f11cb200640 1 Processor -- start 2024-03-06T07:31:10.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.838+0000 7f11cb200640 1 -- start start 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.839+0000 7f11cb200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f11c415e130 0x7f11c40cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.839+0000 7f11cb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40d25b0 0x7f11c40c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.839+0000 7f11cb200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40d13e0 con 0x7f11c40cfaf0 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.839+0000 7f11cb200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40cf120 con 0x7f11c415e130 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.839+0000 7f11cb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40cead0 con 0x7f11c40d25b0 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.839+0000 7f11c9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f11c415e130 0x7f11c40cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.840+0000 7f11c9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f11c415e130 0x7f11c40cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58326/0 (socket says 172.21.15.97:58326) 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.840+0000 7f11c9e00640 1 -- 172.21.15.97:0/2981293018 learned_addr learned my addr 172.21.15.97:0/2981293018 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.840+0000 7f11c9e00640 1 -- 172.21.15.97:0/2981293018 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40d25b0 msgr2=0x7f11c40c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.840+0000 7f11c9e00640 1 --2- 172.21.15.97:0/2981293018 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40d25b0 0x7f11c40c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.840+0000 7f11c9e00640 1 -- 172.21.15.97:0/2981293018 >> v1:172.21.15.135:6789/0 conn(0x7f11c40cfaf0 legacy=0x7f11c40d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.840+0000 7f11c9e00640 1 -- 172.21.15.97:0/2981293018 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40cedb0 con 0x7f11c415e130 2024-03-06T07:31:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.841+0000 7f11c9e00640 1 --2- 172.21.15.97:0/2981293018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40cf5b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009e50 tx=0x7f11bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9038a6c2adf085e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.841+0000 7f11c8a00640 1 -- 172.21.15.97:0/2981293018 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc02ca80 con 0x7f11c415e130 2024-03-06T07:31:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.841+0000 7f11c8a00640 1 -- 172.21.15.97:0/2981293018 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc02cc20 con 0x7f11c415e130 2024-03-06T07:31:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.841+0000 7f11c8a00640 1 -- 172.21.15.97:0/2981293018 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0343f0 con 0x7f11c415e130 2024-03-06T07:31:10.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 -- 172.21.15.97:0/2981293018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 msgr2=0x7f11c40cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 --2- 172.21.15.97:0/2981293018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40cf5b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009e50 tx=0x7f11bc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 -- 172.21.15.97:0/2981293018 shutdown_connections 2024-03-06T07:31:10.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 --2- 172.21.15.97:0/2981293018 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40d25b0 0x7f11c40c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 --2- 172.21.15.97:0/2981293018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40cf5b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 -- 172.21.15.97:0/2981293018 >> 172.21.15.97:0/2981293018 conn(0x7f11c409d070 msgr2=0x7f11c40d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 -- 172.21.15.97:0/2981293018 shutdown_connections 2024-03-06T07:31:10.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.842+0000 7f11cb200640 1 -- 172.21.15.97:0/2981293018 wait complete. 2024-03-06T07:31:10.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.844+0000 7f11cb200640 1 Processor -- start 2024-03-06T07:31:10.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.844+0000 7f11cb200640 1 -- start start 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11cb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40cfaf0 0x7f11c41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11cb200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11c40d25b0 0x7f11c40d3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11cb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40d4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11cb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40c8030 con 0x7f11c415e130 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11cb200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40c7eb0 con 0x7f11c40d25b0 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11cb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40c81b0 con 0x7f11c40cfaf0 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40d4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40d4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58342/0 (socket says 172.21.15.97:58342) 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 -- 172.21.15.97:0/2484374514 learned_addr learned my addr 172.21.15.97:0/2484374514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11c9e00640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40cfaf0 0x7f11c41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11c9400640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11c40d25b0 0x7f11c40d3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 -- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40cfaf0 msgr2=0x7f11c41593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40cfaf0 0x7f11c41593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 -- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11c40d25b0 msgr2=0x7f11c40d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11c40d25b0 0x7f11c40d3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.845+0000 7f11ca800640 1 -- 172.21.15.97:0/2484374514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40cbe90 con 0x7f11c415e130 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.846+0000 7f11ca800640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40d4490 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f11c000b410 tx=0x7f11c000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.846+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0015020 con 0x7f11c415e130 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.846+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c0011850 con 0x7f11c415e130 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.846+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0011c40 con 0x7f11c415e130 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.846+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c415f6c0 con 0x7f11c415e130 2024-03-06T07:31:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.846+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c415fc80 con 0x7f11c415e130 2024-03-06T07:31:10.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.847+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c40c9e90 con 0x7f11c415e130 2024-03-06T07:31:10.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.848+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11c0018020 con 0x7f11c415e130 2024-03-06T07:31:10.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.849+0000 7f11ba000640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11a806b4d0 0x7f11a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.850+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11c008a180 con 0x7f11c415e130 2024-03-06T07:31:10.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.850+0000 7f11c9e00640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11a806b4d0 0x7f11a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.850+0000 7f11c9e00640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11a806b4d0 0x7f11a806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f11bc035000 tx=0x7f11bc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:10.854+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11c40c9e90 con 0x7f11c415e130 2024-03-06T07:31:11.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.076+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f11c404f330 con 0x7f11c415e130 2024-03-06T07:31:11.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.736+0000 7f11ba000640 1 -- 172.21.15.97:0/2484374514 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) v1 ==== 118+0+0 (secure 0 0 0) 0x7f11c001f020 con 0x7f11c415e130 2024-03-06T07:31:11.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11a806b4d0 msgr2=0x7f11a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11a806b4d0 0x7f11a806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f11bc035000 tx=0x7f11bc030000 comp rx=0 tx=0).stop 2024-03-06T07:31:11.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 msgr2=0x7f11c40d4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40d4490 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f11c000b410 tx=0x7f11c000b810 comp rx=0 tx=0).stop 2024-03-06T07:31:11.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 shutdown_connections 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11a806b4d0 0x7f11a806d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11c415e130 0x7f11c40d4490 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11c40d25b0 0x7f11c40d3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 --2- 172.21.15.97:0/2484374514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f11c40cfaf0 0x7f11c41593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 >> 172.21.15.97:0/2484374514 conn(0x7f11c409d070 msgr2=0x7f11c4161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 shutdown_connections 2024-03-06T07:31:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.740+0000 7f11cb200640 1 -- 172.21.15.97:0/2484374514 wait complete. 2024-03-06T07:31:11.763 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:31:11.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:31:12.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.998+0000 7f0fd1200640 1 Processor -- start 2024-03-06T07:31:12.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:11.998+0000 7f0fd1200640 1 -- start start 2024-03-06T07:31:12.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fd1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0fcc0aeff0 0x7f0fcc0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fd1200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc0cfb30 con 0x7f0fcc154dc0 2024-03-06T07:31:12.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fd1200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc0cfe10 con 0x7f0fcc1551a0 2024-03-06T07:31:12.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fd1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fcc15e540 con 0x7f0fcc0aeff0 2024-03-06T07:31:12.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fcb400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0fcc154dc0 0x7f0fcc161b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50710/0 (socket says 172.21.15.97:50710) 2024-03-06T07:31:12.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fcb400640 1 -- 172.21.15.97:0/3951638904 learned_addr learned my addr 172.21.15.97:0/3951638904 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:12.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.000+0000 7f0fcbe00640 1 --2- 172.21.15.97:0/3951638904 >> v2:172.21.15.135:3300/0 conn(0x7f0fcc0aeff0 0x7f0fcc0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777726003 0 0) 0x7f0fcc0cfb30 con 0x7f0fcc154dc0 2024-03-06T07:31:12.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb0003520 con 0x7f0fcc154dc0 2024-03-06T07:31:12.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2449748236 0 0) 0x7f0fb0003520 con 0x7f0fcc154dc0 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 >> v2:172.21.15.135:3300/0 conn(0x7f0fcc0aeff0 msgr2=0x7f0fcc0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 --2- 172.21.15.97:0/3951638904 >> v2:172.21.15.135:3300/0 conn(0x7f0fcc0aeff0 0x7f0fcc0cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 >> v1:172.21.15.135:6789/0 conn(0x7f0fcc1551a0 legacy=0x7f0fcc0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc0d00f0 con 0x7f0fcc154dc0 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0fbc001670 con 0x7f0fcc154dc0 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 >> v1:172.21.15.97:6789/0 conn(0x7f0fcc154dc0 legacy=0x7f0fcc161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0003a00 0x7f0fb0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fb0004800 0x7f0fb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fb00051c0 0x7f0fb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc15e540 con 0x7f0fb0004800 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb000c0c0 con 0x7f0fb00051c0 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb000c2c0 con 0x7f0fb0003a00 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.001+0000 7f0fc3e00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fb0004800 0x7f0fb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fcbe00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fb00051c0 0x7f0fb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fcb400640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0003a00 0x7f0fb0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fc3e00640 1 -- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0003a00 msgr2=0x7f0fb0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fc3e00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0003a00 0x7f0fb0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fc3e00640 1 -- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fb00051c0 msgr2=0x7f0fb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fc3e00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fb00051c0 0x7f0fb000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fc3e00640 1 -- 172.21.15.97:0/3951638904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0003520 con 0x7f0fb0004800 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fcbe00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fb00051c0 0x7f0fb000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.002+0000 7f0fcbe00640 1 -- 172.21.15.97:0/3951638904 reap_dead start 2024-03-06T07:31:12.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.003+0000 7f0fc3e00640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fb0004800 0x7f0fb0004c80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4001750 tx=0x7f0fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4195e50ea9b2a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.003+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4001370 con 0x7f0fb0004800 2024-03-06T07:31:12.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.003+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb40052d0 con 0x7f0fb0004800 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.003+0000 7f0fcaa00640 1 -- 172.21.15.97:0/3951638904 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4003830 con 0x7f0fb0004800 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 -- 172.21.15.97:0/3951638904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fb0004800 msgr2=0x7f0fb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fb0004800 0x7f0fb0004c80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4001750 tx=0x7f0fb0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 -- 172.21.15.97:0/3951638904 shutdown_connections 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 --2- 172.21.15.97:0/3951638904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fb0004800 0x7f0fb0004c80 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 -- 172.21.15.97:0/3951638904 >> 172.21.15.97:0/3951638904 conn(0x7f0fcc06afc0 msgr2=0x7f0fcc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 -- 172.21.15.97:0/3951638904 shutdown_connections 2024-03-06T07:31:12.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.004+0000 7f0fd1200640 1 -- 172.21.15.97:0/3951638904 wait complete. 2024-03-06T07:31:12.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.006+0000 7f0fd1200640 1 Processor -- start 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.006+0000 7f0fd1200640 1 -- start start 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fd1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0004800 0x7f0fcc150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fd1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fcc1f4cb0 0x7f0fcc1f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 0x7f0fcc0c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fd1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0ae8e0 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fd1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0cf820 con 0x7f0fcc1f4cb0 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fd1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0aea60 con 0x7f0fb0004800 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fcbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 0x7f0fcc0c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fcbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 0x7f0fcc0c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58354/0 (socket says 172.21.15.97:58354) 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fcbe00640 1 -- 172.21.15.97:0/2074706988 learned_addr learned my addr 172.21.15.97:0/2074706988 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fcb400640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0004800 0x7f0fcc150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.007+0000 7f0fc3e00640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fcc1f4cb0 0x7f0fcc1f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fcbe00640 1 -- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0004800 msgr2=0x7f0fcc150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fcbe00640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0004800 0x7f0fcc150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fcbe00640 1 -- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fcc1f4cb0 msgr2=0x7f0fcc1f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fcbe00640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fcc1f4cb0 0x7f0fcc1f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fcbe00640 1 -- 172.21.15.97:0/2074706988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc0b0010 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fcbe00640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 0x7f0fcc0c4f80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc155580 tx=0x7f0fc400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4016420 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc401b070 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40167c0 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.008+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc0b02f0 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.009+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc0b08b0 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.010+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc0b0ab0 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.011+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fc4016960 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.012+0000 7f0fc3400640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fa806b500 0x7f0fa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.012+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fc408b180 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.012+0000 7f0fcb400640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fa806b500 0x7f0fa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.013+0000 7f0fcb400640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fa806b500 0x7f0fa806d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0cd170 tx=0x7f0fcc0cffe0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.017+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fcc0b0ab0 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.251+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0fcc0add40 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.739+0000 7f0fc3400640 1 -- 172.21.15.97:0/2074706988 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0fc4025020 con 0x7f0fcc0c4b00 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fa806b500 msgr2=0x7f0fa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fa806b500 0x7f0fa806d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0cd170 tx=0x7f0fcc0cffe0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 msgr2=0x7f0fcc0c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 0x7f0fcc0c4f80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc155580 tx=0x7f0fc400f520 comp rx=0 tx=0).stop 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 shutdown_connections 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0fa806b500 0x7f0fa806d9c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fcc0c4b00 0x7f0fcc0c4f80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fcc1f4cb0 0x7f0fcc1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 --2- 172.21.15.97:0/2074706988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0fb0004800 0x7f0fcc150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.743+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 >> 172.21.15.97:0/2074706988 conn(0x7f0fcc06afc0 msgr2=0x7f0fcc070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.744+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 shutdown_connections 2024-03-06T07:31:12.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:12.744+0000 7f0fd1200640 1 -- 172.21.15.97:0/2074706988 wait complete. 2024-03-06T07:31:12.766 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:31:12.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:13.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.000+0000 7f2feb200640 1 Processor -- start 2024-03-06T07:31:13.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.000+0000 7f2feb200640 1 -- start start 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.001+0000 7f2feb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 0x7f2fe40cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2feb200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2fe40cc800 0x7f2fe415f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2feb200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe415c240 con 0x7f2fe41561f0 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2feb200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe415ce30 con 0x7f2fe40cc800 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2feb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe415c500 con 0x7f2fe40aedc0 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2fe9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2fe41561f0 0x7f2fe40ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50720/0 (socket says 172.21.15.97:50720) 2024-03-06T07:31:13.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2fe9e00640 1 -- 172.21.15.97:0/3560008249 learned_addr learned my addr 172.21.15.97:0/3560008249 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175809578 0 0) 0x7f2fe415c240 con 0x7f2fe41561f0 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2fea800640 1 --2- 172.21.15.97:0/3560008249 >> v2:172.21.15.135:3300/0 conn(0x7f2fe40cc800 0x7f2fe415f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2fe9400640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 0x7f2fe40cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.002+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc003520 con 0x7f2fe41561f0 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696230203 0 0) 0x7f2fcc003520 con 0x7f2fe41561f0 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 msgr2=0x7f2fe40cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 0x7f2fe40cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 >> v2:172.21.15.135:3300/0 conn(0x7f2fe40cc800 msgr2=0x7f2fe415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 --2- 172.21.15.97:0/3560008249 >> v2:172.21.15.135:3300/0 conn(0x7f2fe40cc800 0x7f2fe415f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe415c7e0 con 0x7f2fe41561f0 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2fd8001670 con 0x7f2fe41561f0 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 >> v1:172.21.15.97:6789/0 conn(0x7f2fe41561f0 legacy=0x7f2fe40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 0x7f2fcc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fcc003f80 0x7f2fcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe415ce30 con 0x7f2fcc004d40 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe415c500 con 0x7f2fcc003f80 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc00b9c0 con 0x7f2fe40aedc0 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.003+0000 7f2fea800640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.004+0000 7f2fe9400640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fcc003f80 0x7f2fcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.004+0000 7f2fea800640 1 -- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 msgr2=0x7f2fcc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.004+0000 7f2fea800640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40aedc0 0x7f2fcc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.004+0000 7f2fea800640 1 -- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fcc003f80 msgr2=0x7f2fcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.004+0000 7f2fea800640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fcc003f80 0x7f2fcc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.004+0000 7f2fea800640 1 -- 172.21.15.97:0/3560008249 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc003520 con 0x7f2fcc004d40 2024-03-06T07:31:13.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2fea800640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fcc0051c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000e170 tx=0x7f2fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd83041353194e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0039070 con 0x7f2fcc004d40 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe002cc20 con 0x7f2fcc004d40 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2fe8a00640 1 -- 172.21.15.97:0/3560008249 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0009c10 con 0x7f2fcc004d40 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2feb200640 1 -- 172.21.15.97:0/3560008249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 msgr2=0x7f2fcc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2feb200640 1 --2- 172.21.15.97:0/3560008249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fcc0051c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000e170 tx=0x7f2fcc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.005+0000 7f2fea800640 1 -- 172.21.15.97:0/3560008249 reap_dead start 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.006+0000 7f2feb200640 1 -- 172.21.15.97:0/3560008249 shutdown_connections 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.006+0000 7f2feb200640 1 -- 172.21.15.97:0/3560008249 >> 172.21.15.97:0/3560008249 conn(0x7f2fe406afc0 msgr2=0x7f2fe406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.006+0000 7f2feb200640 1 -- 172.21.15.97:0/3560008249 shutdown_connections 2024-03-06T07:31:13.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.006+0000 7f2feb200640 1 -- 172.21.15.97:0/3560008249 wait complete. 2024-03-06T07:31:13.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.007+0000 7f2feb200640 1 Processor -- start 2024-03-06T07:31:13.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 -- start start 2024-03-06T07:31:13.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fe41f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fe4152db0 0x7f2fe414a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40b0240 0x7f2fe40b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe41565d0 con 0x7f2fcc004d40 2024-03-06T07:31:13.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40ccc90 con 0x7f2fe4152db0 2024-03-06T07:31:13.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.008+0000 7f2feb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4156750 con 0x7f2fe40b0240 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fe41f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fe41f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58378/0 (socket says 172.21.15.97:58378) 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 -- 172.21.15.97:0/1531980090 learned_addr learned my addr 172.21.15.97:0/1531980090 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 -- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40b0240 msgr2=0x7f2fe40b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40b0240 0x7f2fe40b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9400640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fe4152db0 0x7f2fe414a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 -- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fe4152db0 msgr2=0x7f2fe414a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fe4152db0 0x7f2fe414a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 -- 172.21.15.97:0/1531980090 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe4153300 con 0x7f2fcc004d40 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9400640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fe4152db0 0x7f2fe414a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:13.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.009+0000 7f2fe9e00640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fe41f2c10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800c750 tx=0x7f2fd800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.010+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8018070 con 0x7f2fcc004d40 2024-03-06T07:31:13.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.010+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8007500 con 0x7f2fcc004d40 2024-03-06T07:31:13.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.010+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800ce60 con 0x7f2fcc004d40 2024-03-06T07:31:13.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.010+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe41535e0 con 0x7f2fcc004d40 2024-03-06T07:31:13.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.010+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe4153ba0 con 0x7f2fcc004d40 2024-03-06T07:31:13.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.011+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe40b2e10 con 0x7f2fcc004d40 2024-03-06T07:31:13.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.012+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd8004a20 con 0x7f2fcc004d40 2024-03-06T07:31:13.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.013+0000 7f2fde000640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.013+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fd80896c0 con 0x7f2fcc004d40 2024-03-06T07:31:13.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.013+0000 7f2fe9400640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.014+0000 7f2fe9400640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2fc806b4d0 0x7f2fc806d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2fe414abc0 tx=0x7f2fd4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.019+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fe40b2e10 con 0x7f2fcc004d40 2024-03-06T07:31:13.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.254+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe404f330 con 0x7f2fcc004d40 2024-03-06T07:31:13.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.255+0000 7f2fde000640 1 -- 172.21.15.97:0/1531980090 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f2fd80585d0 con 0x7f2fcc004d40 2024-03-06T07:31:13.257 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:13.257 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:13.257 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2fc806b4d0 msgr2=0x7f2fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2fc806b4d0 0x7f2fc806d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2fe414abc0 tx=0x7f2fd4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 msgr2=0x7f2fe41f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fe41f2c10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800c750 tx=0x7f2fd800cc10 comp rx=0 tx=0).stop 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 shutdown_connections 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2fe40b0240 0x7f2fe40b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fe4152db0 0x7f2fe414a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 --2- 172.21.15.97:0/1531980090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fcc004d40 0x7f2fe41f2c10 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 >> 172.21.15.97:0/1531980090 conn(0x7f2fe406afc0 msgr2=0x7f2fe4071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 shutdown_connections 2024-03-06T07:31:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.258+0000 7f2feb200640 1 -- 172.21.15.97:0/1531980090 wait complete. 2024-03-06T07:31:13.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.479+0000 7fc1eba00640 1 Processor -- start 2024-03-06T07:31:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.480+0000 7fc1eba00640 1 -- start start 2024-03-06T07:31:13.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.481+0000 7fc1eba00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1e4144c00 0x7fc1e406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.481+0000 7fc1eba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e415f9d0 0x7fc1e415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.481+0000 7fc1eba00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e406ee70 con 0x7fc1e40ce0a0 2024-03-06T07:31:13.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.481+0000 7fc1eba00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e4160370 con 0x7fc1e4144c00 2024-03-06T07:31:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.481+0000 7fc1eba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e406cef0 con 0x7fc1e415f9d0 2024-03-06T07:31:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1e4144c00 0x7fc1e406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc1e4144c00 0x7fc1e406e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58392/0 (socket says 172.21.15.97:58392) 2024-03-06T07:31:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 -- 172.21.15.97:0/1754638607 learned_addr learned my addr 172.21.15.97:0/1754638607 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 -- 172.21.15.97:0/1754638607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e415f9d0 msgr2=0x7fc1e415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 --2- 172.21.15.97:0/1754638607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e415f9d0 0x7fc1e415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 -- 172.21.15.97:0/1754638607 >> v1:172.21.15.135:6789/0 conn(0x7fc1e40ce0a0 legacy=0x7fc1e40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.482+0000 7fc1e9000640 1 -- 172.21.15.97:0/1754638607 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e406d1d0 con 0x7fc1e4144c00 2024-03-06T07:31:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.483+0000 7fc1e9000640 1 --2- 172.21.15.97:0/1754638607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e4144c00 0x7fc1e406e930 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009870 tx=0x7fc1d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c7612c265ea611 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.483+0000 7fc1e3e00640 1 -- 172.21.15.97:0/1754638607 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4035020 con 0x7fc1e4144c00 2024-03-06T07:31:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1e3e00640 1 -- 172.21.15.97:0/1754638607 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4031640 con 0x7fc1e4144c00 2024-03-06T07:31:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1e3e00640 1 -- 172.21.15.97:0/1754638607 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d402d040 con 0x7fc1e4144c00 2024-03-06T07:31:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 -- 172.21.15.97:0/1754638607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e4144c00 msgr2=0x7fc1e406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 --2- 172.21.15.97:0/1754638607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e4144c00 0x7fc1e406e930 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009870 tx=0x7fc1d402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 -- 172.21.15.97:0/1754638607 shutdown_connections 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 --2- 172.21.15.97:0/1754638607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e415f9d0 0x7fc1e415fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 --2- 172.21.15.97:0/1754638607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e4144c00 0x7fc1e406e930 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 -- 172.21.15.97:0/1754638607 >> 172.21.15.97:0/1754638607 conn(0x7fc1e40b1730 msgr2=0x7fc1e40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 -- 172.21.15.97:0/1754638607 shutdown_connections 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.484+0000 7fc1eba00640 1 -- 172.21.15.97:0/1754638607 wait complete. 2024-03-06T07:31:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.487+0000 7fc1eba00640 1 Processor -- start 2024-03-06T07:31:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.487+0000 7fc1eba00640 1 -- start start 2024-03-06T07:31:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1eba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e40ce0a0 0x7fc1e41e9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1eba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1e4144c00 0x7fc1e415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1eba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 0x7fc1e415b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1eba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e4160730 con 0x7fc1e415f9d0 2024-03-06T07:31:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1eba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e406d790 con 0x7fc1e4144c00 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1eba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e406cb50 con 0x7fc1e40ce0a0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1e9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 0x7fc1e415b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1e9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 0x7fc1e415b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58400/0 (socket says 172.21.15.97:58400) 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1e9a00640 1 -- 172.21.15.97:0/3674909993 learned_addr learned my addr 172.21.15.97:0/3674909993 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1dbe00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1e4144c00 0x7fc1e415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.489+0000 7fc1e9000640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e40ce0a0 0x7fc1e41e9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e9a00640 1 -- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e40ce0a0 msgr2=0x7fc1e41e9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e9a00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e40ce0a0 0x7fc1e41e9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e9a00640 1 -- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1e4144c00 msgr2=0x7fc1e415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e9a00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1e4144c00 0x7fc1e415afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e9a00640 1 -- 172.21.15.97:0/3674909993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e41e8d10 con 0x7fc1e415f9d0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e9a00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 0x7fc1e415b4f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00b990 tx=0x7fc1dc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc019070 con 0x7fc1e415f9d0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc00cba0 con 0x7fc1e415f9d0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e41e7c70 con 0x7fc1e415f9d0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc014440 con 0x7fc1e415f9d0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.490+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e41e8230 con 0x7fc1e415f9d0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.491+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e4071880 con 0x7fc1e415f9d0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.493+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1dc01c070 con 0x7fc1e415f9d0 2024-03-06T07:31:13.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.494+0000 7fc1e1600640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1c006b500 0x7fc1c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.494+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1dc08a060 con 0x7fc1e415f9d0 2024-03-06T07:31:13.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.494+0000 7fc1e9000640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1c006b500 0x7fc1c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.495+0000 7fc1e9000640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1c006b500 0x7fc1c006d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4036000 tx=0x7fc1d403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.498+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1e4071880 con 0x7fc1e415f9d0 2024-03-06T07:31:13.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.696+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1e404f330 con 0x7fc1e415f9d0 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.698+0000 7fc1e1600640 1 -- 172.21.15.97:0/3674909993 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fc1dc058db0 con 0x7fc1e415f9d0 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":69,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:10.689032+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:13.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1c006b500 msgr2=0x7fc1c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1c006b500 0x7fc1c006d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4036000 tx=0x7fc1d403c040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 msgr2=0x7fc1e415b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 0x7fc1e415b4f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00b990 tx=0x7fc1dc00be50 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 shutdown_connections 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc1c006b500 0x7fc1c006d9c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1e415f9d0 0x7fc1e415b4f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1e4144c00 0x7fc1e415afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 --2- 172.21.15.97:0/3674909993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc1e40ce0a0 0x7fc1e41e9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 >> 172.21.15.97:0/3674909993 conn(0x7fc1e40b1730 msgr2=0x7fc1e40ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 shutdown_connections 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.701+0000 7fc1eba00640 1 -- 172.21.15.97:0/3674909993 wait complete. 2024-03-06T07:31:13.721 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:31:13.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.913+0000 7f7231200640 1 Processor -- start 2024-03-06T07:31:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.913+0000 7f7231200640 1 -- start start 2024-03-06T07:31:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.914+0000 7f7231200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f722c144c00 0x7f722c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.914+0000 7f7231200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 0x7f722c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.914+0000 7f7231200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722c06ee70 con 0x7f722c0ce0a0 2024-03-06T07:31:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.914+0000 7f7231200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c160370 con 0x7f722c144c00 2024-03-06T07:31:13.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.914+0000 7f7231200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c06cef0 con 0x7f722c15f9d0 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.915+0000 7f722a800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f722c0ce0a0 0x7f722c0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48160/0 (socket says 172.21.15.97:48160) 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.915+0000 7f722a800640 1 -- 172.21.15.97:0/1790897779 learned_addr learned my addr 172.21.15.97:0/1790897779 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.915+0000 7f7229e00640 1 --2- 172.21.15.97:0/1790897779 >> v2:172.21.15.135:3300/0 conn(0x7f722c144c00 0x7f722c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.915+0000 7f7221400640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 0x7f722c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.915+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957985581 0 0) 0x7f7224001110 con 0x7f722c0ce0a0 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.915+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7210003520 con 0x7f722c0ce0a0 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3088094471 0 0) 0x7f7210003520 con 0x7f722c0ce0a0 2024-03-06T07:31:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 msgr2=0x7f722c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 0x7f722c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 >> v2:172.21.15.135:3300/0 conn(0x7f722c144c00 msgr2=0x7f722c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 --2- 172.21.15.97:0/1790897779 >> v2:172.21.15.135:3300/0 conn(0x7f722c144c00 0x7f722c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c06d1d0 con 0x7f722c0ce0a0 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7224001840 con 0x7f722c0ce0a0 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 >> v1:172.21.15.97:6789/0 conn(0x7f722c0ce0a0 legacy=0x7f722c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72100039b0 0x7f7210003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 0x7f7210004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7210004d40 0x7f72100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c06cef0 con 0x7f72100039b0 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721000b9c0 con 0x7f7210004d40 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721000bb40 con 0x7f722c15f9d0 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.916+0000 7f7229e00640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72100039b0 0x7f7210003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f7221400640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 0x7f7210004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f722a800640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7210004d40 0x7f72100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f7229e00640 1 -- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 msgr2=0x7f7210004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f7229e00640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c15f9d0 0x7f7210004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f7229e00640 1 -- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7210004d40 msgr2=0x7f72100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f7229e00640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7210004d40 0x7f72100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f7229e00640 1 -- 172.21.15.97:0/1790897779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7210003520 con 0x7f72100039b0 2024-03-06T07:31:13.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.917+0000 7f722a800640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7210004d40 0x7f72100051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.918+0000 7f7229e00640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72100039b0 0x7f7210003dd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f721c00e170 tx=0x7f7210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355e70d9c10ce70f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.918+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c039070 con 0x7f72100039b0 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.918+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721c02cc20 con 0x7f72100039b0 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.918+0000 7f7229400640 1 -- 172.21.15.97:0/1790897779 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c009c10 con 0x7f72100039b0 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f7231200640 1 -- 172.21.15.97:0/1790897779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72100039b0 msgr2=0x7f7210003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f7231200640 1 --2- 172.21.15.97:0/1790897779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72100039b0 0x7f7210003dd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f721c00e170 tx=0x7f7210005830 comp rx=0 tx=0).stop 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f722a800640 1 -- 172.21.15.97:0/1790897779 reap_dead start 2024-03-06T07:31:13.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f7231200640 1 -- 172.21.15.97:0/1790897779 shutdown_connections 2024-03-06T07:31:13.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f7231200640 1 -- 172.21.15.97:0/1790897779 >> 172.21.15.97:0/1790897779 conn(0x7f722c0b1730 msgr2=0x7f722c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f7231200640 1 -- 172.21.15.97:0/1790897779 shutdown_connections 2024-03-06T07:31:13.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.919+0000 7f7231200640 1 -- 172.21.15.97:0/1790897779 wait complete. 2024-03-06T07:31:13.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.921+0000 7f7231200640 1 Processor -- start 2024-03-06T07:31:13.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.921+0000 7f7231200640 1 -- start start 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.922+0000 7f7231200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72100039b0 0x7f722c143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.922+0000 7f7231200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 0x7f722c15e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.922+0000 7f7231200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c14f660 0x7f722c14fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.922+0000 7f7231200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c160730 con 0x7f722c150320 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.922+0000 7f7231200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c06d790 con 0x7f72100039b0 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.922+0000 7f7231200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c06cb50 con 0x7f722c14f660 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 0x7f722c15e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 0x7f722c15e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39734/0 (socket says 172.21.15.97:39734) 2024-03-06T07:31:13.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 -- 172.21.15.97:0/1687991907 learned_addr learned my addr 172.21.15.97:0/1687991907 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7229e00640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72100039b0 0x7f722c143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f722a800640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c14f660 0x7f722c14fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 -- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c14f660 msgr2=0x7f722c14fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c14f660 0x7f722c14fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 -- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72100039b0 msgr2=0x7f722c143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72100039b0 0x7f722c143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f7221400640 1 -- 172.21.15.97:0/1687991907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c143a50 con 0x7f722c150320 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.923+0000 7f722a800640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c14f660 0x7f722c14fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.924+0000 7f7221400640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 0x7f722c15e640 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7214010890 tx=0x7f7214010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.924+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721400f040 con 0x7f722c150320 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.924+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721400eb60 con 0x7f722c150320 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.924+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721401c400 con 0x7f722c150320 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.924+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c13d2d0 con 0x7f722c150320 2024-03-06T07:31:13.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.924+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c13d890 con 0x7f722c150320 2024-03-06T07:31:13.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.926+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c04f330 con 0x7f722c150320 2024-03-06T07:31:13.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.926+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f721401c5a0 con 0x7f722c150320 2024-03-06T07:31:13.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.927+0000 7f7222a00640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f720806b4d0 0x7f720806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.927+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f721408a840 con 0x7f722c150320 2024-03-06T07:31:13.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.927+0000 7f7229e00640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f720806b4d0 0x7f720806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.928+0000 7f7229e00640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f720806b4d0 0x7f720806d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f721c036000 tx=0x7f721c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:13.932+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7214018030 con 0x7f722c150320 2024-03-06T07:31:14.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.146+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f722c160510 con 0x7f720806b4d0 2024-03-06T07:31:14.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.163+0000 7f7222a00640 1 -- 172.21.15.97:0/1687991907 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f722c160510 con 0x7f720806b4d0 2024-03-06T07:31:14.169 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:14.170 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f720806b4d0 msgr2=0x7f720806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f720806b4d0 0x7f720806d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f721c036000 tx=0x7f721c02e000 comp rx=0 tx=0).stop 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 msgr2=0x7f722c15e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 0x7f722c15e640 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7214010890 tx=0x7f7214010d50 comp rx=0 tx=0).stop 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 shutdown_connections 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f720806b4d0 0x7f720806d990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f722c14f660 0x7f722c14fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f722c150320 0x7f722c15e640 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 --2- 172.21.15.97:0/1687991907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72100039b0 0x7f722c143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 >> 172.21.15.97:0/1687991907 conn(0x7f722c0b1730 msgr2=0x7f722c070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 shutdown_connections 2024-03-06T07:31:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.170+0000 7f7231200640 1 -- 172.21.15.97:0/1687991907 wait complete. 2024-03-06T07:31:14.191 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-03-06T07:31:12.965264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256876,"kb_used_data":7412,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717844,"statfs":{"total":773094113280,"available":772831072256,"internally_reserved":0,"allocated":7589888,"data_stored":3832139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.303623"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.701275+0000","last_change":"2024-03-06T07:31:08.701275+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.701275+0000","last_undegraded":"2024-03-06T07:31:08.701275+0000","last_fullsized":"2024-03-06T07:31:08.701275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.698067+0000","last_change":"2024-03-06T07:31:08.698067+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.698067+0000","last_undegraded":"2024-03-06T07:31:08.698067+0000","last_fullsized":"2024-03-06T07:31:08.698067+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.684036+0000","last_change":"2024-03-06T07:31:08.684036+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.684036+0000","last_undegraded":"2024-03-06T07:31:08.684036+0000","last_fullsized":"2024-03-06T07:31:08.684036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677906+0000","last_change":"2024-03-06T07:31:08.677906+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677906+0000","last_undegraded":"2024-03-06T07:31:08.677906+0000","last_fullsized":"2024-03-06T07:31:08.677906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.680268+0000","last_change":"2024-03-06T07:31:08.680268+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.680268+0000","last_undegraded":"2024-03-06T07:31:08.680268+0000","last_fullsized":"2024-03-06T07:31:08.680268+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.693149+0000","last_change":"2024-03-06T07:31:08.693149+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.693149+0000","last_undegraded":"2024-03-06T07:31:08.693149+0000","last_fullsized":"2024-03-06T07:31:08.693149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.691253+0000","last_change":"2024-03-06T07:31:08.691253+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.691253+0000","last_undegraded":"2024-03-06T07:31:08.691253+0000","last_fullsized":"2024-03-06T07:31:08.691253+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.694293+0000","last_change":"2024-03-06T07:31:08.694293+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.694293+0000","last_undegraded":"2024-03-06T07:31:08.694293+0000","last_fullsized":"2024-03-06T07:31:08.694293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685077+0000","last_change":"2024-03-06T07:31:08.685077+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685077+0000","last_undegraded":"2024-03-06T07:31:08.685077+0000","last_fullsized":"2024-03-06T07:31:08.685077+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.686572+0000","last_change":"2024-03-06T07:31:08.686572+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.686572+0000","last_undegraded":"2024-03-06T07:31:08.686572+0000","last_fullsized":"2024-03-06T07:31:08.686572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696583+0000","last_change":"2024-03-06T07:31:08.696583+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696583+0000","last_undegraded":"2024-03-06T07:31:08.696583+0000","last_fullsized":"2024-03-06T07:31:08.696583+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677236+0000","last_change":"2024-03-06T07:31:08.677236+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677236+0000","last_undegraded":"2024-03-06T07:31:08.677236+0000","last_fullsized":"2024-03-06T07:31:08.677236+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675022+0000","last_change":"2024-03-06T07:31:08.675022+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675022+0000","last_undegraded":"2024-03-06T07:31:08.675022+0000","last_fullsized":"2024-03-06T07:31:08.675022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.680969+0000","last_change":"2024-03-06T07:31:08.680969+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.680969+0000","last_undegraded":"2024-03-06T07:31:08.680969+0000","last_fullsized":"2024-03-06T07:31:08.680969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675956+0000","last_change":"2024-03-06T07:31:08.675956+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675956+0000","last_undegraded":"2024-03-06T07:31:08.675956+0000","last_fullsized":"2024-03-06T07:31:08.675956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.679641+0000","last_change":"2024-03-06T07:31:08.679641+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.679641+0000","last_undegraded":"2024-03-06T07:31:08.679641+0000","last_fullsized":"2024-03-06T07:31:08.679641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682028+0000","last_change":"2024-03-06T07:31:08.682028+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682028+0000","last_undegraded":"2024-03-06T07:31:08.682028+0000","last_fullsized":"2024-03-06T07:31:08.682028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676292+0000","last_change":"2024-03-06T07:31:08.676292+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676292+0000","last_undegraded":"2024-03-06T07:31:08.676292+0000","last_fullsized":"2024-03-06T07:31:08.676292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.697491+0000","last_change":"2024-03-06T07:31:08.697491+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.697491+0000","last_undegraded":"2024-03-06T07:31:08.697491+0000","last_fullsized":"2024-03-06T07:31:08.697491+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675005+0000","last_change":"2024-03-06T07:31:08.675005+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675005+0000","last_undegraded":"2024-03-06T07:31:08.675005+0000","last_fullsized":"2024-03-06T07:31:08.675005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677955+0000","last_change":"2024-03-06T07:31:08.677955+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677955+0000","last_undegraded":"2024-03-06T07:31:08.677955+0000","last_fullsized":"2024-03-06T07:31:08.677955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.679705+0000","last_change":"2024-03-06T07:31:08.679705+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.679705+0000","last_undegraded":"2024-03-06T07:31:08.679705+0000","last_fullsized":"2024-03-06T07:31:08.679705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.702113+0000","last_change":"2024-03-06T07:31:08.702113+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.702113+0000","last_undegraded":"2024-03-06T07:31:08.702113+0000","last_fullsized":"2024-03-06T07:31:08.702113+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.700106+0000","last_change":"2024-03-06T07:31:08.700106+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.700106+0000","last_undegraded":"2024-03-06T07:31:08.700106+0000","last_fullsized":"2024-03-06T07:31:08.700106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677926+0000","last_change":"2024-03-06T07:31:08.677926+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677926+0000","last_undegraded":"2024-03-06T07:31:08.677926+0000","last_fullsized":"2024-03-06T07:31:08.677926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689247+0000","last_change":"2024-03-06T07:31:08.689247+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689247+0000","last_undegraded":"2024-03-06T07:31:08.689247+0000","last_fullsized":"2024-03-06T07:31:08.689247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.683364+0000","last_change":"2024-03-06T07:31:08.683364+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.683364+0000","last_undegraded":"2024-03-06T07:31:08.683364+0000","last_fullsized":"2024-03-06T07:31:08.683364+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.686181+0000","last_change":"2024-03-06T07:31:08.686181+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.686181+0000","last_undegraded":"2024-03-06T07:31:08.686181+0000","last_fullsized":"2024-03-06T07:31:08.686181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682255+0000","last_change":"2024-03-06T07:31:08.682255+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682255+0000","last_undegraded":"2024-03-06T07:31:08.682255+0000","last_fullsized":"2024-03-06T07:31:08.682255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689933+0000","last_change":"2024-03-06T07:31:08.689933+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689933+0000","last_undegraded":"2024-03-06T07:31:08.689933+0000","last_fullsized":"2024-03-06T07:31:08.689933+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689947+0000","last_change":"2024-03-06T07:31:08.689947+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689947+0000","last_undegraded":"2024-03-06T07:31:08.689947+0000","last_fullsized":"2024-03-06T07:31:08.689947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.693845+0000","last_change":"2024-03-06T07:31:08.693845+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.693845+0000","last_undegraded":"2024-03-06T07:31:08.693845+0000","last_fullsized":"2024-03-06T07:31:08.693845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.695834+0000","last_change":"2024-03-06T07:31:08.695834+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.695834+0000","last_undegraded":"2024-03-06T07:31:08.695834+0000","last_fullsized":"2024-03-06T07:31:08.695834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711201+0000","last_change":"2024-03-06T07:31:08.709472+0000","last_active":"2024-03-06T07:31:08.711201+0000","last_peered":"2024-03-06T07:31:08.711201+0000","last_clean":"2024-03-06T07:31:08.711201+0000","last_became_active":"2024-03-06T07:31:08.709183+0000","last_became_peered":"2024-03-06T07:31:08.709183+0000","last_unstale":"2024-03-06T07:31:08.711201+0000","last_undegraded":"2024-03-06T07:31:08.711201+0000","last_fullsized":"2024-03-06T07:31:08.711201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681348+0000","last_change":"2024-03-06T07:31:08.681348+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681348+0000","last_undegraded":"2024-03-06T07:31:08.681348+0000","last_fullsized":"2024-03-06T07:31:08.681348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.692181+0000","last_change":"2024-03-06T07:31:08.690969+0000","last_active":"2024-03-06T07:31:08.692181+0000","last_peered":"2024-03-06T07:31:08.692181+0000","last_clean":"2024-03-06T07:31:08.692181+0000","last_became_active":"2024-03-06T07:31:08.690694+0000","last_became_peered":"2024-03-06T07:31:08.690694+0000","last_unstale":"2024-03-06T07:31:08.692181+0000","last_undegraded":"2024-03-06T07:31:08.692181+0000","last_fullsized":"2024-03-06T07:31:08.692181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:15.902202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696210+0000","last_change":"2024-03-06T07:31:08.696210+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696210+0000","last_undegraded":"2024-03-06T07:31:08.696210+0000","last_fullsized":"2024-03-06T07:31:08.696210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712977+0000","last_change":"2024-03-06T07:31:08.711193+0000","last_active":"2024-03-06T07:31:08.712977+0000","last_peered":"2024-03-06T07:31:08.712977+0000","last_clean":"2024-03-06T07:31:08.712977+0000","last_became_active":"2024-03-06T07:31:08.710729+0000","last_became_peered":"2024-03-06T07:31:08.710729+0000","last_unstale":"2024-03-06T07:31:08.712977+0000","last_undegraded":"2024-03-06T07:31:08.712977+0000","last_fullsized":"2024-03-06T07:31:08.712977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:09.374013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696094+0000","last_change":"2024-03-06T07:31:08.696094+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696094+0000","last_undegraded":"2024-03-06T07:31:08.696094+0000","last_fullsized":"2024-03-06T07:31:08.696094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711033+0000","last_change":"2024-03-06T07:31:08.703780+0000","last_active":"2024-03-06T07:31:08.711033+0000","last_peered":"2024-03-06T07:31:08.711033+0000","last_clean":"2024-03-06T07:31:08.711033+0000","last_became_active":"2024-03-06T07:31:08.703434+0000","last_became_peered":"2024-03-06T07:31:08.703434+0000","last_unstale":"2024-03-06T07:31:08.711033+0000","last_undegraded":"2024-03-06T07:31:08.711033+0000","last_fullsized":"2024-03-06T07:31:08.711033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:09.421677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689219+0000","last_change":"2024-03-06T07:31:08.689219+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689219+0000","last_undegraded":"2024-03-06T07:31:08.689219+0000","last_fullsized":"2024-03-06T07:31:08.689219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.697066+0000","last_change":"2024-03-06T07:31:08.695941+0000","last_active":"2024-03-06T07:31:08.697066+0000","last_peered":"2024-03-06T07:31:08.697066+0000","last_clean":"2024-03-06T07:31:08.697066+0000","last_became_active":"2024-03-06T07:31:08.695727+0000","last_became_peered":"2024-03-06T07:31:08.695727+0000","last_unstale":"2024-03-06T07:31:08.697066+0000","last_undegraded":"2024-03-06T07:31:08.697066+0000","last_fullsized":"2024-03-06T07:31:08.697066+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:43.370682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676619+0000","last_change":"2024-03-06T07:31:08.676619+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676619+0000","last_undegraded":"2024-03-06T07:31:08.676619+0000","last_fullsized":"2024-03-06T07:31:08.676619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.709795+0000","last_change":"2024-03-06T07:31:08.701504+0000","last_active":"2024-03-06T07:31:08.709795+0000","last_peered":"2024-03-06T07:31:08.709795+0000","last_clean":"2024-03-06T07:31:08.709795+0000","last_became_active":"2024-03-06T07:31:08.701175+0000","last_became_peered":"2024-03-06T07:31:08.701175+0000","last_unstale":"2024-03-06T07:31:08.709795+0000","last_undegraded":"2024-03-06T07:31:08.709795+0000","last_fullsized":"2024-03-06T07:31:08.709795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:06.427514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681850+0000","last_change":"2024-03-06T07:31:08.681850+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681850+0000","last_undegraded":"2024-03-06T07:31:08.681850+0000","last_fullsized":"2024-03-06T07:31:08.681850+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.694098+0000","last_change":"2024-03-06T07:31:08.690249+0000","last_active":"2024-03-06T07:31:08.694098+0000","last_peered":"2024-03-06T07:31:08.694098+0000","last_clean":"2024-03-06T07:31:08.694098+0000","last_became_active":"2024-03-06T07:31:08.689995+0000","last_became_peered":"2024-03-06T07:31:08.689995+0000","last_unstale":"2024-03-06T07:31:08.694098+0000","last_undegraded":"2024-03-06T07:31:08.694098+0000","last_fullsized":"2024-03-06T07:31:08.694098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.402843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675535+0000","last_change":"2024-03-06T07:31:08.675535+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675535+0000","last_undegraded":"2024-03-06T07:31:08.675535+0000","last_fullsized":"2024-03-06T07:31:08.675535+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.697843+0000","last_change":"2024-03-06T07:31:08.694935+0000","last_active":"2024-03-06T07:31:08.697843+0000","last_peered":"2024-03-06T07:31:08.697843+0000","last_clean":"2024-03-06T07:31:08.697843+0000","last_became_active":"2024-03-06T07:31:08.694548+0000","last_became_peered":"2024-03-06T07:31:08.694548+0000","last_unstale":"2024-03-06T07:31:08.697843+0000","last_undegraded":"2024-03-06T07:31:08.697843+0000","last_fullsized":"2024-03-06T07:31:08.697843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:01.670055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.688273+0000","last_change":"2024-03-06T07:31:08.688273+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.688273+0000","last_undegraded":"2024-03-06T07:31:08.688273+0000","last_fullsized":"2024-03-06T07:31:08.688273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.693297+0000","last_change":"2024-03-06T07:31:08.691602+0000","last_active":"2024-03-06T07:31:08.693297+0000","last_peered":"2024-03-06T07:31:08.693297+0000","last_clean":"2024-03-06T07:31:08.693297+0000","last_became_active":"2024-03-06T07:31:08.690752+0000","last_became_peered":"2024-03-06T07:31:08.690752+0000","last_unstale":"2024-03-06T07:31:08.693297+0000","last_undegraded":"2024-03-06T07:31:08.693297+0000","last_fullsized":"2024-03-06T07:31:08.693297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.768385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.700840+0000","last_change":"2024-03-06T07:31:08.700840+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.700840+0000","last_undegraded":"2024-03-06T07:31:08.700840+0000","last_fullsized":"2024-03-06T07:31:08.700840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712385+0000","last_change":"2024-03-06T07:31:08.710975+0000","last_active":"2024-03-06T07:31:08.712385+0000","last_peered":"2024-03-06T07:31:08.712385+0000","last_clean":"2024-03-06T07:31:08.712385+0000","last_became_active":"2024-03-06T07:31:08.710624+0000","last_became_peered":"2024-03-06T07:31:08.710624+0000","last_unstale":"2024-03-06T07:31:08.712385+0000","last_undegraded":"2024-03-06T07:31:08.712385+0000","last_fullsized":"2024-03-06T07:31:08.712385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.544528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.694234+0000","last_change":"2024-03-06T07:31:08.694234+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.694234+0000","last_undegraded":"2024-03-06T07:31:08.694234+0000","last_fullsized":"2024-03-06T07:31:08.694234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.702788+0000","last_change":"2024-03-06T07:31:08.701415+0000","last_active":"2024-03-06T07:31:08.702788+0000","last_peered":"2024-03-06T07:31:08.702788+0000","last_clean":"2024-03-06T07:31:08.702788+0000","last_became_active":"2024-03-06T07:31:08.700947+0000","last_became_peered":"2024-03-06T07:31:08.700947+0000","last_unstale":"2024-03-06T07:31:08.702788+0000","last_undegraded":"2024-03-06T07:31:08.702788+0000","last_fullsized":"2024-03-06T07:31:08.702788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:46.346819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685031+0000","last_change":"2024-03-06T07:31:08.685031+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685031+0000","last_undegraded":"2024-03-06T07:31:08.685031+0000","last_fullsized":"2024-03-06T07:31:08.685031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.693846+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:08.693846+0000","last_peered":"2024-03-06T07:31:08.693846+0000","last_clean":"2024-03-06T07:31:08.693846+0000","last_became_active":"2024-03-06T07:31:08.687214+0000","last_became_peered":"2024-03-06T07:31:08.687214+0000","last_unstale":"2024-03-06T07:31:08.693846+0000","last_undegraded":"2024-03-06T07:31:08.693846+0000","last_fullsized":"2024-03-06T07:31:08.693846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.137094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.684329+0000","last_change":"2024-03-06T07:31:08.684329+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.684329+0000","last_undegraded":"2024-03-06T07:31:08.684329+0000","last_fullsized":"2024-03-06T07:31:08.684329+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.706300+0000","last_change":"2024-03-06T07:31:08.702804+0000","last_active":"2024-03-06T07:31:08.706300+0000","last_peered":"2024-03-06T07:31:08.706300+0000","last_clean":"2024-03-06T07:31:08.706300+0000","last_became_active":"2024-03-06T07:31:08.702494+0000","last_became_peered":"2024-03-06T07:31:08.702494+0000","last_unstale":"2024-03-06T07:31:08.706300+0000","last_undegraded":"2024-03-06T07:31:08.706300+0000","last_fullsized":"2024-03-06T07:31:08.706300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:08.663518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685982+0000","last_change":"2024-03-06T07:31:08.685982+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685982+0000","last_undegraded":"2024-03-06T07:31:08.685982+0000","last_fullsized":"2024-03-06T07:31:08.685982+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711177+0000","last_change":"2024-03-06T07:31:08.706279+0000","last_active":"2024-03-06T07:31:08.711177+0000","last_peered":"2024-03-06T07:31:08.711177+0000","last_clean":"2024-03-06T07:31:08.711177+0000","last_became_active":"2024-03-06T07:31:08.705857+0000","last_became_peered":"2024-03-06T07:31:08.705857+0000","last_unstale":"2024-03-06T07:31:08.711177+0000","last_undegraded":"2024-03-06T07:31:08.711177+0000","last_fullsized":"2024-03-06T07:31:08.711177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:22.682345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.674758+0000","last_change":"2024-03-06T07:31:08.674758+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.674758+0000","last_undegraded":"2024-03-06T07:31:08.674758+0000","last_fullsized":"2024-03-06T07:31:08.674758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.705595+0000","last_change":"2024-03-06T07:31:08.699062+0000","last_active":"2024-03-06T07:31:08.705595+0000","last_peered":"2024-03-06T07:31:08.705595+0000","last_clean":"2024-03-06T07:31:08.705595+0000","last_became_active":"2024-03-06T07:31:08.698767+0000","last_became_peered":"2024-03-06T07:31:08.698767+0000","last_unstale":"2024-03-06T07:31:08.705595+0000","last_undegraded":"2024-03-06T07:31:08.705595+0000","last_fullsized":"2024-03-06T07:31:08.705595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:12.877333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.679862+0000","last_change":"2024-03-06T07:31:08.679862+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.679862+0000","last_undegraded":"2024-03-06T07:31:08.679862+0000","last_fullsized":"2024-03-06T07:31:08.679862+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712144+0000","last_change":"2024-03-06T07:31:08.708615+0000","last_active":"2024-03-06T07:31:08.712144+0000","last_peered":"2024-03-06T07:31:08.712144+0000","last_clean":"2024-03-06T07:31:08.712144+0000","last_became_active":"2024-03-06T07:31:08.708149+0000","last_became_peered":"2024-03-06T07:31:08.708149+0000","last_unstale":"2024-03-06T07:31:08.712144+0000","last_undegraded":"2024-03-06T07:31:08.712144+0000","last_fullsized":"2024-03-06T07:31:08.712144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.198390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685501+0000","last_change":"2024-03-06T07:31:08.685501+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685501+0000","last_undegraded":"2024-03-06T07:31:08.685501+0000","last_fullsized":"2024-03-06T07:31:08.685501+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711090+0000","last_change":"2024-03-06T07:31:08.706291+0000","last_active":"2024-03-06T07:31:08.711090+0000","last_peered":"2024-03-06T07:31:08.711090+0000","last_clean":"2024-03-06T07:31:08.711090+0000","last_became_active":"2024-03-06T07:31:08.705537+0000","last_became_peered":"2024-03-06T07:31:08.705537+0000","last_unstale":"2024-03-06T07:31:08.711090+0000","last_undegraded":"2024-03-06T07:31:08.711090+0000","last_fullsized":"2024-03-06T07:31:08.711090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:06.738181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.700356+0000","last_change":"2024-03-06T07:31:08.700356+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.700356+0000","last_undegraded":"2024-03-06T07:31:08.700356+0000","last_fullsized":"2024-03-06T07:31:08.700356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712807+0000","last_change":"2024-03-06T07:31:08.708950+0000","last_active":"2024-03-06T07:31:08.712807+0000","last_peered":"2024-03-06T07:31:08.712807+0000","last_clean":"2024-03-06T07:31:08.712807+0000","last_became_active":"2024-03-06T07:31:08.707758+0000","last_became_peered":"2024-03-06T07:31:08.707758+0000","last_unstale":"2024-03-06T07:31:08.712807+0000","last_undegraded":"2024-03-06T07:31:08.712807+0000","last_fullsized":"2024-03-06T07:31:08.712807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:46.487298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676722+0000","last_change":"2024-03-06T07:31:08.676722+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676722+0000","last_undegraded":"2024-03-06T07:31:08.676722+0000","last_fullsized":"2024-03-06T07:31:08.676722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711489+0000","last_change":"2024-03-06T07:31:08.708472+0000","last_active":"2024-03-06T07:31:08.711489+0000","last_peered":"2024-03-06T07:31:08.711489+0000","last_clean":"2024-03-06T07:31:08.711489+0000","last_became_active":"2024-03-06T07:31:08.708186+0000","last_became_peered":"2024-03-06T07:31:08.708186+0000","last_unstale":"2024-03-06T07:31:08.711489+0000","last_undegraded":"2024-03-06T07:31:08.711489+0000","last_fullsized":"2024-03-06T07:31:08.711489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.137094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682111+0000","last_change":"2024-03-06T07:31:08.682111+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682111+0000","last_undegraded":"2024-03-06T07:31:08.682111+0000","last_fullsized":"2024-03-06T07:31:08.682111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712674+0000","last_change":"2024-03-06T07:31:08.710889+0000","last_active":"2024-03-06T07:31:08.712674+0000","last_peered":"2024-03-06T07:31:08.712674+0000","last_clean":"2024-03-06T07:31:08.712674+0000","last_became_active":"2024-03-06T07:31:08.709406+0000","last_became_peered":"2024-03-06T07:31:08.709406+0000","last_unstale":"2024-03-06T07:31:08.712674+0000","last_undegraded":"2024-03-06T07:31:08.712674+0000","last_fullsized":"2024-03-06T07:31:08.712674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:47.336954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681829+0000","last_change":"2024-03-06T07:31:08.681829+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681829+0000","last_undegraded":"2024-03-06T07:31:08.681829+0000","last_fullsized":"2024-03-06T07:31:08.681829+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.710997+0000","last_change":"2024-03-06T07:31:08.704271+0000","last_active":"2024-03-06T07:31:08.710997+0000","last_peered":"2024-03-06T07:31:08.710997+0000","last_clean":"2024-03-06T07:31:08.710997+0000","last_became_active":"2024-03-06T07:31:08.700885+0000","last_became_peered":"2024-03-06T07:31:08.700885+0000","last_unstale":"2024-03-06T07:31:08.710997+0000","last_undegraded":"2024-03-06T07:31:08.710997+0000","last_fullsized":"2024-03-06T07:31:08.710997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:46.809806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681577+0000","last_change":"2024-03-06T07:31:08.681577+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681577+0000","last_undegraded":"2024-03-06T07:31:08.681577+0000","last_fullsized":"2024-03-06T07:31:08.681577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.705142+0000","last_change":"2024-03-06T07:31:08.698893+0000","last_active":"2024-03-06T07:31:08.705142+0000","last_peered":"2024-03-06T07:31:08.705142+0000","last_clean":"2024-03-06T07:31:08.705142+0000","last_became_active":"2024-03-06T07:31:08.698419+0000","last_became_peered":"2024-03-06T07:31:08.698419+0000","last_unstale":"2024-03-06T07:31:08.705142+0000","last_undegraded":"2024-03-06T07:31:08.705142+0000","last_fullsized":"2024-03-06T07:31:08.705142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:15.902202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682852+0000","last_change":"2024-03-06T07:31:08.682852+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682852+0000","last_undegraded":"2024-03-06T07:31:08.682852+0000","last_fullsized":"2024-03-06T07:31:08.682852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.691192+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:08.691192+0000","last_peered":"2024-03-06T07:31:08.691192+0000","last_clean":"2024-03-06T07:31:08.691192+0000","last_became_active":"2024-03-06T07:31:08.687300+0000","last_became_peered":"2024-03-06T07:31:08.687300+0000","last_unstale":"2024-03-06T07:31:08.691192+0000","last_undegraded":"2024-03-06T07:31:08.691192+0000","last_fullsized":"2024-03-06T07:31:08.691192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:25.951914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.674916+0000","last_change":"2024-03-06T07:31:08.674916+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.674916+0000","last_undegraded":"2024-03-06T07:31:08.674916+0000","last_fullsized":"2024-03-06T07:31:08.674916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712323+0000","last_change":"2024-03-06T07:31:08.710412+0000","last_active":"2024-03-06T07:31:08.712323+0000","last_peered":"2024-03-06T07:31:08.712323+0000","last_clean":"2024-03-06T07:31:08.712323+0000","last_became_active":"2024-03-06T07:31:08.710061+0000","last_became_peered":"2024-03-06T07:31:08.710061+0000","last_unstale":"2024-03-06T07:31:08.712323+0000","last_undegraded":"2024-03-06T07:31:08.712323+0000","last_fullsized":"2024-03-06T07:31:08.712323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.402843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.688028+0000","last_change":"2024-03-06T07:31:08.688028+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.688028+0000","last_undegraded":"2024-03-06T07:31:08.688028+0000","last_fullsized":"2024-03-06T07:31:08.688028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.700709+0000","last_change":"2024-03-06T07:31:08.695443+0000","last_active":"2024-03-06T07:31:08.700709+0000","last_peered":"2024-03-06T07:31:08.700709+0000","last_clean":"2024-03-06T07:31:08.700709+0000","last_became_active":"2024-03-06T07:31:08.695112+0000","last_became_peered":"2024-03-06T07:31:08.695112+0000","last_unstale":"2024-03-06T07:31:08.700709+0000","last_undegraded":"2024-03-06T07:31:08.700709+0000","last_fullsized":"2024-03-06T07:31:08.700709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:25.951914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.684771+0000","last_change":"2024-03-06T07:31:08.684771+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.684771+0000","last_undegraded":"2024-03-06T07:31:08.684771+0000","last_fullsized":"2024-03-06T07:31:08.684771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.694369+0000","last_change":"2024-03-06T07:31:08.691823+0000","last_active":"2024-03-06T07:31:08.694369+0000","last_peered":"2024-03-06T07:31:08.694369+0000","last_clean":"2024-03-06T07:31:08.694369+0000","last_became_active":"2024-03-06T07:31:08.691505+0000","last_became_peered":"2024-03-06T07:31:08.691505+0000","last_unstale":"2024-03-06T07:31:08.694369+0000","last_undegraded":"2024-03-06T07:31:08.694369+0000","last_fullsized":"2024-03-06T07:31:08.694369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:46.481274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.695512+0000","last_change":"2024-03-06T07:31:08.695512+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.695512+0000","last_undegraded":"2024-03-06T07:31:08.695512+0000","last_fullsized":"2024-03-06T07:31:08.695512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.699479+0000","last_change":"2024-03-06T07:31:08.698585+0000","last_active":"2024-03-06T07:31:08.699479+0000","last_peered":"2024-03-06T07:31:08.699479+0000","last_clean":"2024-03-06T07:31:08.699479+0000","last_became_active":"2024-03-06T07:31:08.698166+0000","last_became_peered":"2024-03-06T07:31:08.698166+0000","last_unstale":"2024-03-06T07:31:08.699479+0000","last_undegraded":"2024-03-06T07:31:08.699479+0000","last_fullsized":"2024-03-06T07:31:08.699479+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:12.877333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.680554+0000","last_change":"2024-03-06T07:31:08.680554+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.680554+0000","last_undegraded":"2024-03-06T07:31:08.680554+0000","last_fullsized":"2024-03-06T07:31:08.680554+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.697911+0000","last_change":"2024-03-06T07:31:08.696740+0000","last_active":"2024-03-06T07:31:08.697911+0000","last_peered":"2024-03-06T07:31:08.697911+0000","last_clean":"2024-03-06T07:31:08.697911+0000","last_became_active":"2024-03-06T07:31:08.696450+0000","last_became_peered":"2024-03-06T07:31:08.696450+0000","last_unstale":"2024-03-06T07:31:08.697911+0000","last_undegraded":"2024-03-06T07:31:08.697911+0000","last_fullsized":"2024-03-06T07:31:08.697911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:17.835616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696783+0000","last_change":"2024-03-06T07:31:08.696783+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696783+0000","last_undegraded":"2024-03-06T07:31:08.696783+0000","last_fullsized":"2024-03-06T07:31:08.696783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.694246+0000","last_change":"2024-03-06T07:31:08.692727+0000","last_active":"2024-03-06T07:31:08.694246+0000","last_peered":"2024-03-06T07:31:08.694246+0000","last_clean":"2024-03-06T07:31:08.694246+0000","last_became_active":"2024-03-06T07:31:08.692457+0000","last_became_peered":"2024-03-06T07:31:08.692457+0000","last_unstale":"2024-03-06T07:31:08.694246+0000","last_undegraded":"2024-03-06T07:31:08.694246+0000","last_fullsized":"2024-03-06T07:31:08.694246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.544528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676557+0000","last_change":"2024-03-06T07:31:08.676557+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676557+0000","last_undegraded":"2024-03-06T07:31:08.676557+0000","last_fullsized":"2024-03-06T07:31:08.676557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.705180+0000","last_change":"2024-03-06T07:31:08.703245+0000","last_active":"2024-03-06T07:31:08.705180+0000","last_peered":"2024-03-06T07:31:08.705180+0000","last_clean":"2024-03-06T07:31:08.705180+0000","last_became_active":"2024-03-06T07:31:08.702878+0000","last_became_peered":"2024-03-06T07:31:08.702878+0000","last_unstale":"2024-03-06T07:31:08.705180+0000","last_undegraded":"2024-03-06T07:31:08.705180+0000","last_fullsized":"2024-03-06T07:31:08.705180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:32.737748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.695845+0000","last_change":"2024-03-06T07:31:08.695845+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.695845+0000","last_undegraded":"2024-03-06T07:31:08.695845+0000","last_fullsized":"2024-03-06T07:31:08.695845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711260+0000","last_change":"2024-03-06T07:31:08.707266+0000","last_active":"2024-03-06T07:31:08.711260+0000","last_peered":"2024-03-06T07:31:08.711260+0000","last_clean":"2024-03-06T07:31:08.711260+0000","last_became_active":"2024-03-06T07:31:08.706774+0000","last_became_peered":"2024-03-06T07:31:08.706774+0000","last_unstale":"2024-03-06T07:31:08.711260+0000","last_undegraded":"2024-03-06T07:31:08.711260+0000","last_fullsized":"2024-03-06T07:31:08.711260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:49.878415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682755+0000","last_change":"2024-03-06T07:31:08.682755+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682755+0000","last_undegraded":"2024-03-06T07:31:08.682755+0000","last_fullsized":"2024-03-06T07:31:08.682755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711533+0000","last_change":"2024-03-06T07:31:08.707358+0000","last_active":"2024-03-06T07:31:08.711533+0000","last_peered":"2024-03-06T07:31:08.711533+0000","last_clean":"2024-03-06T07:31:08.711533+0000","last_became_active":"2024-03-06T07:31:08.706392+0000","last_became_peered":"2024-03-06T07:31:08.706392+0000","last_unstale":"2024-03-06T07:31:08.711533+0000","last_undegraded":"2024-03-06T07:31:08.711533+0000","last_fullsized":"2024-03-06T07:31:08.711533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:24.349345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":253,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.706106+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:31:08.706106+0000","last_peered":"2024-03-06T07:31:08.706106+0000","last_clean":"2024-03-06T07:31:08.706106+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:31:08.706106+0000","last_undegraded":"2024-03-06T07:31:08.706106+0000","last_fullsized":"2024-03-06T07:31:08.706106+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542189,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":1404928,"data_stored":927284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542190,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542191,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542190,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1392640,"data_stored":921009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:14.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:14.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.427+0000 7f363ea00640 1 Processor -- start 2024-03-06T07:31:14.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.427+0000 7f363ea00640 1 -- start start 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f363ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36380d24c0 0x7f36380d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f363ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f36380b0aa0 0x7f36380b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f363ea00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36380d4c50 con 0x7f36380b2de0 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f363ea00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36380d4a00 con 0x7f36380b0aa0 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f363ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36380cf7e0 con 0x7f36380d24c0 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f362f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f36380b2de0 0x7f36380d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48168/0 (socket says 172.21.15.97:48168) 2024-03-06T07:31:14.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f362f400640 1 -- 172.21.15.97:0/468537201 learned_addr learned my addr 172.21.15.97:0/468537201 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:14.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f363ca00640 1 --2- 172.21.15.97:0/468537201 >> v2:172.21.15.135:3300/0 conn(0x7f36380b0aa0 0x7f36380b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f3637e00640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36380d24c0 0x7f36380d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.429+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094086287 0 0) 0x7f36380d4c50 con 0x7f36380b2de0 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c003520 con 0x7f36380b2de0 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88323124 0 0) 0x7f361c003520 con 0x7f36380b2de0 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36380d24c0 msgr2=0x7f36380d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36380d24c0 0x7f36380d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 >> v2:172.21.15.135:3300/0 conn(0x7f36380b0aa0 msgr2=0x7f36380b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 --2- 172.21.15.97:0/468537201 >> v2:172.21.15.135:3300/0 conn(0x7f36380b0aa0 0x7f36380b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36380cfa30 con 0x7f36380b2de0 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f36200019f0 con 0x7f36380b2de0 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 >> v1:172.21.15.97:6789/0 conn(0x7f36380b2de0 legacy=0x7f36380d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0039b0 0x7f361c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c004800 0x7f361c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0051c0 0x7f361c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36380d4a00 con 0x7f361c004800 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36380cf7e0 con 0x7f361c0039b0 2024-03-06T07:31:14.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c00c030 con 0x7f361c0051c0 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f3637e00640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36380d24c0 0x7f36380d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.430+0000 7f362f400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c004800 0x7f361c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.431+0000 7f362f400640 1 -- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0051c0 msgr2=0x7f361c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.431+0000 7f362f400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f361c0051c0 0x7f361c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.431+0000 7f362f400640 1 -- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0039b0 msgr2=0x7f361c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.431+0000 7f362f400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c0039b0 0x7f361c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.431+0000 7f362f400640 1 -- 172.21.15.97:0/468537201 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c003520 con 0x7f361c004800 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.432+0000 7f362f400640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c004800 0x7f361c004c80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f362000a7d0 tx=0x7f361c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c247b062d81fd07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.432+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362000a3f0 con 0x7f361c004800 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.432+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3620034630 con 0x7f361c004800 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.432+0000 7f3637400640 1 -- 172.21.15.97:0/468537201 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362002ce60 con 0x7f361c004800 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ea00640 1 -- 172.21.15.97:0/468537201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c004800 msgr2=0x7f361c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ea00640 1 --2- 172.21.15.97:0/468537201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f361c004800 0x7f361c004c80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f362000a7d0 tx=0x7f361c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ca00640 1 -- 172.21.15.97:0/468537201 reap_dead start 2024-03-06T07:31:14.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ea00640 1 -- 172.21.15.97:0/468537201 shutdown_connections 2024-03-06T07:31:14.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ea00640 1 -- 172.21.15.97:0/468537201 >> 172.21.15.97:0/468537201 conn(0x7f36380d3ee0 msgr2=0x7f36380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ea00640 1 -- 172.21.15.97:0/468537201 shutdown_connections 2024-03-06T07:31:14.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.433+0000 7f363ea00640 1 -- 172.21.15.97:0/468537201 wait complete. 2024-03-06T07:31:14.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.435+0000 7f363ea00640 1 Processor -- start 2024-03-06T07:31:14.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.435+0000 7f363ea00640 1 -- start start 2024-03-06T07:31:14.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c004800 0x7f363814e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36381f4d90 0x7f363814dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 0x7f36380ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363815e5b0 con 0x7f36380c9dc0 2024-03-06T07:31:14.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36380d0080 con 0x7f361c004800 2024-03-06T07:31:14.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363815e730 con 0x7f36381f4d90 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 0x7f36380ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 0x7f36380ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39746/0 (socket says 172.21.15.97:39746) 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 -- 172.21.15.97:0/2027090217 learned_addr learned my addr 172.21.15.97:0/2027090217 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f3637e00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c004800 0x7f363814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f362f400640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36381f4d90 0x7f363814dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 -- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36381f4d90 msgr2=0x7f363814dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36381f4d90 0x7f363814dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 -- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c004800 msgr2=0x7f363814e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c004800 0x7f363814e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.436+0000 7f363ca00640 1 -- 172.21.15.97:0/2027090217 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36381f7a60 con 0x7f36380c9dc0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.437+0000 7f363ca00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 0x7f36380ca240 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f363808cdb0 tx=0x7f363000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.437+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363000c6b0 con 0x7f36380c9dc0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.437+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363001a070 con 0x7f36380c9dc0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.437+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363000ca90 con 0x7f36380c9dc0 2024-03-06T07:31:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.437+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36381f7d40 con 0x7f36380c9dc0 2024-03-06T07:31:14.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.437+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36381f8300 con 0x7f36380c9dc0 2024-03-06T07:31:14.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.438+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363804f330 con 0x7f36380c9dc0 2024-03-06T07:31:14.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.439+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f363000cc30 con 0x7f36380c9dc0 2024-03-06T07:31:14.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.440+0000 7f3634c00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.440+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3630089ed0 con 0x7f36380c9dc0 2024-03-06T07:31:14.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.440+0000 7f3637e00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.441+0000 7f3637e00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f361406b4d0 0x7f361406d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3628006a90 tx=0x7f3628008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.444+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3630024020 con 0x7f36380c9dc0 2024-03-06T07:31:14.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.637+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36380d22a0 con 0x7f361406b4d0 2024-03-06T07:31:14.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.650+0000 7f3634c00640 1 -- 172.21.15.97:0/2027090217 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f36380d22a0 con 0x7f361406b4d0 2024-03-06T07:31:14.654 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:14.655 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.654+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f361406b4d0 msgr2=0x7f361406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.654+0000 7f363ea00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f361406b4d0 0x7f361406d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3628006a90 tx=0x7f3628008040 comp rx=0 tx=0).stop 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.654+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 msgr2=0x7f36380ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.654+0000 7f363ea00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 0x7f36380ca240 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f363808cdb0 tx=0x7f363000ec00 comp rx=0 tx=0).stop 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 shutdown_connections 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36380c9dc0 0x7f36380ca240 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f36381f4d90 0x7f363814dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 --2- 172.21.15.97:0/2027090217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f361c004800 0x7f363814e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 >> 172.21.15.97:0/2027090217 conn(0x7f36380d3ee0 msgr2=0x7f363806bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 shutdown_connections 2024-03-06T07:31:14.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.655+0000 7f363ea00640 1 -- 172.21.15.97:0/2027090217 wait complete. 2024-03-06T07:31:14.675 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-03-06T07:31:12.965264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256876,"kb_used_data":7412,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717844,"statfs":{"total":773094113280,"available":772831072256,"internally_reserved":0,"allocated":7589888,"data_stored":3832139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.303623"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.701275+0000","last_change":"2024-03-06T07:31:08.701275+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.701275+0000","last_undegraded":"2024-03-06T07:31:08.701275+0000","last_fullsized":"2024-03-06T07:31:08.701275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.698067+0000","last_change":"2024-03-06T07:31:08.698067+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.698067+0000","last_undegraded":"2024-03-06T07:31:08.698067+0000","last_fullsized":"2024-03-06T07:31:08.698067+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.684036+0000","last_change":"2024-03-06T07:31:08.684036+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.684036+0000","last_undegraded":"2024-03-06T07:31:08.684036+0000","last_fullsized":"2024-03-06T07:31:08.684036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677906+0000","last_change":"2024-03-06T07:31:08.677906+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677906+0000","last_undegraded":"2024-03-06T07:31:08.677906+0000","last_fullsized":"2024-03-06T07:31:08.677906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.680268+0000","last_change":"2024-03-06T07:31:08.680268+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.680268+0000","last_undegraded":"2024-03-06T07:31:08.680268+0000","last_fullsized":"2024-03-06T07:31:08.680268+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.693149+0000","last_change":"2024-03-06T07:31:08.693149+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.693149+0000","last_undegraded":"2024-03-06T07:31:08.693149+0000","last_fullsized":"2024-03-06T07:31:08.693149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.691253+0000","last_change":"2024-03-06T07:31:08.691253+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.691253+0000","last_undegraded":"2024-03-06T07:31:08.691253+0000","last_fullsized":"2024-03-06T07:31:08.691253+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.694293+0000","last_change":"2024-03-06T07:31:08.694293+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.694293+0000","last_undegraded":"2024-03-06T07:31:08.694293+0000","last_fullsized":"2024-03-06T07:31:08.694293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685077+0000","last_change":"2024-03-06T07:31:08.685077+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685077+0000","last_undegraded":"2024-03-06T07:31:08.685077+0000","last_fullsized":"2024-03-06T07:31:08.685077+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.686572+0000","last_change":"2024-03-06T07:31:08.686572+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.686572+0000","last_undegraded":"2024-03-06T07:31:08.686572+0000","last_fullsized":"2024-03-06T07:31:08.686572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696583+0000","last_change":"2024-03-06T07:31:08.696583+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696583+0000","last_undegraded":"2024-03-06T07:31:08.696583+0000","last_fullsized":"2024-03-06T07:31:08.696583+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677236+0000","last_change":"2024-03-06T07:31:08.677236+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677236+0000","last_undegraded":"2024-03-06T07:31:08.677236+0000","last_fullsized":"2024-03-06T07:31:08.677236+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675022+0000","last_change":"2024-03-06T07:31:08.675022+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675022+0000","last_undegraded":"2024-03-06T07:31:08.675022+0000","last_fullsized":"2024-03-06T07:31:08.675022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.680969+0000","last_change":"2024-03-06T07:31:08.680969+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.680969+0000","last_undegraded":"2024-03-06T07:31:08.680969+0000","last_fullsized":"2024-03-06T07:31:08.680969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675956+0000","last_change":"2024-03-06T07:31:08.675956+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675956+0000","last_undegraded":"2024-03-06T07:31:08.675956+0000","last_fullsized":"2024-03-06T07:31:08.675956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.679641+0000","last_change":"2024-03-06T07:31:08.679641+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.679641+0000","last_undegraded":"2024-03-06T07:31:08.679641+0000","last_fullsized":"2024-03-06T07:31:08.679641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682028+0000","last_change":"2024-03-06T07:31:08.682028+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682028+0000","last_undegraded":"2024-03-06T07:31:08.682028+0000","last_fullsized":"2024-03-06T07:31:08.682028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676292+0000","last_change":"2024-03-06T07:31:08.676292+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676292+0000","last_undegraded":"2024-03-06T07:31:08.676292+0000","last_fullsized":"2024-03-06T07:31:08.676292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.697491+0000","last_change":"2024-03-06T07:31:08.697491+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.697491+0000","last_undegraded":"2024-03-06T07:31:08.697491+0000","last_fullsized":"2024-03-06T07:31:08.697491+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675005+0000","last_change":"2024-03-06T07:31:08.675005+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675005+0000","last_undegraded":"2024-03-06T07:31:08.675005+0000","last_fullsized":"2024-03-06T07:31:08.675005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677955+0000","last_change":"2024-03-06T07:31:08.677955+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677955+0000","last_undegraded":"2024-03-06T07:31:08.677955+0000","last_fullsized":"2024-03-06T07:31:08.677955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.679705+0000","last_change":"2024-03-06T07:31:08.679705+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.679705+0000","last_undegraded":"2024-03-06T07:31:08.679705+0000","last_fullsized":"2024-03-06T07:31:08.679705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.702113+0000","last_change":"2024-03-06T07:31:08.702113+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.702113+0000","last_undegraded":"2024-03-06T07:31:08.702113+0000","last_fullsized":"2024-03-06T07:31:08.702113+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.700106+0000","last_change":"2024-03-06T07:31:08.700106+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.700106+0000","last_undegraded":"2024-03-06T07:31:08.700106+0000","last_fullsized":"2024-03-06T07:31:08.700106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.677926+0000","last_change":"2024-03-06T07:31:08.677926+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.677926+0000","last_undegraded":"2024-03-06T07:31:08.677926+0000","last_fullsized":"2024-03-06T07:31:08.677926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689247+0000","last_change":"2024-03-06T07:31:08.689247+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689247+0000","last_undegraded":"2024-03-06T07:31:08.689247+0000","last_fullsized":"2024-03-06T07:31:08.689247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.683364+0000","last_change":"2024-03-06T07:31:08.683364+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.683364+0000","last_undegraded":"2024-03-06T07:31:08.683364+0000","last_fullsized":"2024-03-06T07:31:08.683364+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.686181+0000","last_change":"2024-03-06T07:31:08.686181+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.686181+0000","last_undegraded":"2024-03-06T07:31:08.686181+0000","last_fullsized":"2024-03-06T07:31:08.686181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682255+0000","last_change":"2024-03-06T07:31:08.682255+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682255+0000","last_undegraded":"2024-03-06T07:31:08.682255+0000","last_fullsized":"2024-03-06T07:31:08.682255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689933+0000","last_change":"2024-03-06T07:31:08.689933+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689933+0000","last_undegraded":"2024-03-06T07:31:08.689933+0000","last_fullsized":"2024-03-06T07:31:08.689933+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689947+0000","last_change":"2024-03-06T07:31:08.689947+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689947+0000","last_undegraded":"2024-03-06T07:31:08.689947+0000","last_fullsized":"2024-03-06T07:31:08.689947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.693845+0000","last_change":"2024-03-06T07:31:08.693845+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.693845+0000","last_undegraded":"2024-03-06T07:31:08.693845+0000","last_fullsized":"2024-03-06T07:31:08.693845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.695834+0000","last_change":"2024-03-06T07:31:08.695834+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.695834+0000","last_undegraded":"2024-03-06T07:31:08.695834+0000","last_fullsized":"2024-03-06T07:31:08.695834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711201+0000","last_change":"2024-03-06T07:31:08.709472+0000","last_active":"2024-03-06T07:31:08.711201+0000","last_peered":"2024-03-06T07:31:08.711201+0000","last_clean":"2024-03-06T07:31:08.711201+0000","last_became_active":"2024-03-06T07:31:08.709183+0000","last_became_peered":"2024-03-06T07:31:08.709183+0000","last_unstale":"2024-03-06T07:31:08.711201+0000","last_undegraded":"2024-03-06T07:31:08.711201+0000","last_fullsized":"2024-03-06T07:31:08.711201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681348+0000","last_change":"2024-03-06T07:31:08.681348+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681348+0000","last_undegraded":"2024-03-06T07:31:08.681348+0000","last_fullsized":"2024-03-06T07:31:08.681348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.692181+0000","last_change":"2024-03-06T07:31:08.690969+0000","last_active":"2024-03-06T07:31:08.692181+0000","last_peered":"2024-03-06T07:31:08.692181+0000","last_clean":"2024-03-06T07:31:08.692181+0000","last_became_active":"2024-03-06T07:31:08.690694+0000","last_became_peered":"2024-03-06T07:31:08.690694+0000","last_unstale":"2024-03-06T07:31:08.692181+0000","last_undegraded":"2024-03-06T07:31:08.692181+0000","last_fullsized":"2024-03-06T07:31:08.692181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:15.902202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696210+0000","last_change":"2024-03-06T07:31:08.696210+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696210+0000","last_undegraded":"2024-03-06T07:31:08.696210+0000","last_fullsized":"2024-03-06T07:31:08.696210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712977+0000","last_change":"2024-03-06T07:31:08.711193+0000","last_active":"2024-03-06T07:31:08.712977+0000","last_peered":"2024-03-06T07:31:08.712977+0000","last_clean":"2024-03-06T07:31:08.712977+0000","last_became_active":"2024-03-06T07:31:08.710729+0000","last_became_peered":"2024-03-06T07:31:08.710729+0000","last_unstale":"2024-03-06T07:31:08.712977+0000","last_undegraded":"2024-03-06T07:31:08.712977+0000","last_fullsized":"2024-03-06T07:31:08.712977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:09.374013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696094+0000","last_change":"2024-03-06T07:31:08.696094+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696094+0000","last_undegraded":"2024-03-06T07:31:08.696094+0000","last_fullsized":"2024-03-06T07:31:08.696094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711033+0000","last_change":"2024-03-06T07:31:08.703780+0000","last_active":"2024-03-06T07:31:08.711033+0000","last_peered":"2024-03-06T07:31:08.711033+0000","last_clean":"2024-03-06T07:31:08.711033+0000","last_became_active":"2024-03-06T07:31:08.703434+0000","last_became_peered":"2024-03-06T07:31:08.703434+0000","last_unstale":"2024-03-06T07:31:08.711033+0000","last_undegraded":"2024-03-06T07:31:08.711033+0000","last_fullsized":"2024-03-06T07:31:08.711033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:09.421677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.689219+0000","last_change":"2024-03-06T07:31:08.689219+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.689219+0000","last_undegraded":"2024-03-06T07:31:08.689219+0000","last_fullsized":"2024-03-06T07:31:08.689219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.697066+0000","last_change":"2024-03-06T07:31:08.695941+0000","last_active":"2024-03-06T07:31:08.697066+0000","last_peered":"2024-03-06T07:31:08.697066+0000","last_clean":"2024-03-06T07:31:08.697066+0000","last_became_active":"2024-03-06T07:31:08.695727+0000","last_became_peered":"2024-03-06T07:31:08.695727+0000","last_unstale":"2024-03-06T07:31:08.697066+0000","last_undegraded":"2024-03-06T07:31:08.697066+0000","last_fullsized":"2024-03-06T07:31:08.697066+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:43.370682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676619+0000","last_change":"2024-03-06T07:31:08.676619+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676619+0000","last_undegraded":"2024-03-06T07:31:08.676619+0000","last_fullsized":"2024-03-06T07:31:08.676619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.709795+0000","last_change":"2024-03-06T07:31:08.701504+0000","last_active":"2024-03-06T07:31:08.709795+0000","last_peered":"2024-03-06T07:31:08.709795+0000","last_clean":"2024-03-06T07:31:08.709795+0000","last_became_active":"2024-03-06T07:31:08.701175+0000","last_became_peered":"2024-03-06T07:31:08.701175+0000","last_unstale":"2024-03-06T07:31:08.709795+0000","last_undegraded":"2024-03-06T07:31:08.709795+0000","last_fullsized":"2024-03-06T07:31:08.709795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:06.427514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681850+0000","last_change":"2024-03-06T07:31:08.681850+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681850+0000","last_undegraded":"2024-03-06T07:31:08.681850+0000","last_fullsized":"2024-03-06T07:31:08.681850+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.694098+0000","last_change":"2024-03-06T07:31:08.690249+0000","last_active":"2024-03-06T07:31:08.694098+0000","last_peered":"2024-03-06T07:31:08.694098+0000","last_clean":"2024-03-06T07:31:08.694098+0000","last_became_active":"2024-03-06T07:31:08.689995+0000","last_became_peered":"2024-03-06T07:31:08.689995+0000","last_unstale":"2024-03-06T07:31:08.694098+0000","last_undegraded":"2024-03-06T07:31:08.694098+0000","last_fullsized":"2024-03-06T07:31:08.694098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.402843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.675535+0000","last_change":"2024-03-06T07:31:08.675535+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.675535+0000","last_undegraded":"2024-03-06T07:31:08.675535+0000","last_fullsized":"2024-03-06T07:31:08.675535+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.697843+0000","last_change":"2024-03-06T07:31:08.694935+0000","last_active":"2024-03-06T07:31:08.697843+0000","last_peered":"2024-03-06T07:31:08.697843+0000","last_clean":"2024-03-06T07:31:08.697843+0000","last_became_active":"2024-03-06T07:31:08.694548+0000","last_became_peered":"2024-03-06T07:31:08.694548+0000","last_unstale":"2024-03-06T07:31:08.697843+0000","last_undegraded":"2024-03-06T07:31:08.697843+0000","last_fullsized":"2024-03-06T07:31:08.697843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:01.670055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.688273+0000","last_change":"2024-03-06T07:31:08.688273+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.688273+0000","last_undegraded":"2024-03-06T07:31:08.688273+0000","last_fullsized":"2024-03-06T07:31:08.688273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.693297+0000","last_change":"2024-03-06T07:31:08.691602+0000","last_active":"2024-03-06T07:31:08.693297+0000","last_peered":"2024-03-06T07:31:08.693297+0000","last_clean":"2024-03-06T07:31:08.693297+0000","last_became_active":"2024-03-06T07:31:08.690752+0000","last_became_peered":"2024-03-06T07:31:08.690752+0000","last_unstale":"2024-03-06T07:31:08.693297+0000","last_undegraded":"2024-03-06T07:31:08.693297+0000","last_fullsized":"2024-03-06T07:31:08.693297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.768385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.700840+0000","last_change":"2024-03-06T07:31:08.700840+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.700840+0000","last_undegraded":"2024-03-06T07:31:08.700840+0000","last_fullsized":"2024-03-06T07:31:08.700840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712385+0000","last_change":"2024-03-06T07:31:08.710975+0000","last_active":"2024-03-06T07:31:08.712385+0000","last_peered":"2024-03-06T07:31:08.712385+0000","last_clean":"2024-03-06T07:31:08.712385+0000","last_became_active":"2024-03-06T07:31:08.710624+0000","last_became_peered":"2024-03-06T07:31:08.710624+0000","last_unstale":"2024-03-06T07:31:08.712385+0000","last_undegraded":"2024-03-06T07:31:08.712385+0000","last_fullsized":"2024-03-06T07:31:08.712385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.544528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.694234+0000","last_change":"2024-03-06T07:31:08.694234+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.694234+0000","last_undegraded":"2024-03-06T07:31:08.694234+0000","last_fullsized":"2024-03-06T07:31:08.694234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.702788+0000","last_change":"2024-03-06T07:31:08.701415+0000","last_active":"2024-03-06T07:31:08.702788+0000","last_peered":"2024-03-06T07:31:08.702788+0000","last_clean":"2024-03-06T07:31:08.702788+0000","last_became_active":"2024-03-06T07:31:08.700947+0000","last_became_peered":"2024-03-06T07:31:08.700947+0000","last_unstale":"2024-03-06T07:31:08.702788+0000","last_undegraded":"2024-03-06T07:31:08.702788+0000","last_fullsized":"2024-03-06T07:31:08.702788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:46.346819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685031+0000","last_change":"2024-03-06T07:31:08.685031+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685031+0000","last_undegraded":"2024-03-06T07:31:08.685031+0000","last_fullsized":"2024-03-06T07:31:08.685031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.693846+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:08.693846+0000","last_peered":"2024-03-06T07:31:08.693846+0000","last_clean":"2024-03-06T07:31:08.693846+0000","last_became_active":"2024-03-06T07:31:08.687214+0000","last_became_peered":"2024-03-06T07:31:08.687214+0000","last_unstale":"2024-03-06T07:31:08.693846+0000","last_undegraded":"2024-03-06T07:31:08.693846+0000","last_fullsized":"2024-03-06T07:31:08.693846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.137094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.684329+0000","last_change":"2024-03-06T07:31:08.684329+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.684329+0000","last_undegraded":"2024-03-06T07:31:08.684329+0000","last_fullsized":"2024-03-06T07:31:08.684329+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.706300+0000","last_change":"2024-03-06T07:31:08.702804+0000","last_active":"2024-03-06T07:31:08.706300+0000","last_peered":"2024-03-06T07:31:08.706300+0000","last_clean":"2024-03-06T07:31:08.706300+0000","last_became_active":"2024-03-06T07:31:08.702494+0000","last_became_peered":"2024-03-06T07:31:08.702494+0000","last_unstale":"2024-03-06T07:31:08.706300+0000","last_undegraded":"2024-03-06T07:31:08.706300+0000","last_fullsized":"2024-03-06T07:31:08.706300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:08.663518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685982+0000","last_change":"2024-03-06T07:31:08.685982+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685982+0000","last_undegraded":"2024-03-06T07:31:08.685982+0000","last_fullsized":"2024-03-06T07:31:08.685982+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711177+0000","last_change":"2024-03-06T07:31:08.706279+0000","last_active":"2024-03-06T07:31:08.711177+0000","last_peered":"2024-03-06T07:31:08.711177+0000","last_clean":"2024-03-06T07:31:08.711177+0000","last_became_active":"2024-03-06T07:31:08.705857+0000","last_became_peered":"2024-03-06T07:31:08.705857+0000","last_unstale":"2024-03-06T07:31:08.711177+0000","last_undegraded":"2024-03-06T07:31:08.711177+0000","last_fullsized":"2024-03-06T07:31:08.711177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:22.682345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.674758+0000","last_change":"2024-03-06T07:31:08.674758+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.674758+0000","last_undegraded":"2024-03-06T07:31:08.674758+0000","last_fullsized":"2024-03-06T07:31:08.674758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.705595+0000","last_change":"2024-03-06T07:31:08.699062+0000","last_active":"2024-03-06T07:31:08.705595+0000","last_peered":"2024-03-06T07:31:08.705595+0000","last_clean":"2024-03-06T07:31:08.705595+0000","last_became_active":"2024-03-06T07:31:08.698767+0000","last_became_peered":"2024-03-06T07:31:08.698767+0000","last_unstale":"2024-03-06T07:31:08.705595+0000","last_undegraded":"2024-03-06T07:31:08.705595+0000","last_fullsized":"2024-03-06T07:31:08.705595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:12.877333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.679862+0000","last_change":"2024-03-06T07:31:08.679862+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.679862+0000","last_undegraded":"2024-03-06T07:31:08.679862+0000","last_fullsized":"2024-03-06T07:31:08.679862+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712144+0000","last_change":"2024-03-06T07:31:08.708615+0000","last_active":"2024-03-06T07:31:08.712144+0000","last_peered":"2024-03-06T07:31:08.712144+0000","last_clean":"2024-03-06T07:31:08.712144+0000","last_became_active":"2024-03-06T07:31:08.708149+0000","last_became_peered":"2024-03-06T07:31:08.708149+0000","last_unstale":"2024-03-06T07:31:08.712144+0000","last_undegraded":"2024-03-06T07:31:08.712144+0000","last_fullsized":"2024-03-06T07:31:08.712144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.198390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.685501+0000","last_change":"2024-03-06T07:31:08.685501+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.685501+0000","last_undegraded":"2024-03-06T07:31:08.685501+0000","last_fullsized":"2024-03-06T07:31:08.685501+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711090+0000","last_change":"2024-03-06T07:31:08.706291+0000","last_active":"2024-03-06T07:31:08.711090+0000","last_peered":"2024-03-06T07:31:08.711090+0000","last_clean":"2024-03-06T07:31:08.711090+0000","last_became_active":"2024-03-06T07:31:08.705537+0000","last_became_peered":"2024-03-06T07:31:08.705537+0000","last_unstale":"2024-03-06T07:31:08.711090+0000","last_undegraded":"2024-03-06T07:31:08.711090+0000","last_fullsized":"2024-03-06T07:31:08.711090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:06.738181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.700356+0000","last_change":"2024-03-06T07:31:08.700356+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.700356+0000","last_undegraded":"2024-03-06T07:31:08.700356+0000","last_fullsized":"2024-03-06T07:31:08.700356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712807+0000","last_change":"2024-03-06T07:31:08.708950+0000","last_active":"2024-03-06T07:31:08.712807+0000","last_peered":"2024-03-06T07:31:08.712807+0000","last_clean":"2024-03-06T07:31:08.712807+0000","last_became_active":"2024-03-06T07:31:08.707758+0000","last_became_peered":"2024-03-06T07:31:08.707758+0000","last_unstale":"2024-03-06T07:31:08.712807+0000","last_undegraded":"2024-03-06T07:31:08.712807+0000","last_fullsized":"2024-03-06T07:31:08.712807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:46.487298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676722+0000","last_change":"2024-03-06T07:31:08.676722+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676722+0000","last_undegraded":"2024-03-06T07:31:08.676722+0000","last_fullsized":"2024-03-06T07:31:08.676722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711489+0000","last_change":"2024-03-06T07:31:08.708472+0000","last_active":"2024-03-06T07:31:08.711489+0000","last_peered":"2024-03-06T07:31:08.711489+0000","last_clean":"2024-03-06T07:31:08.711489+0000","last_became_active":"2024-03-06T07:31:08.708186+0000","last_became_peered":"2024-03-06T07:31:08.708186+0000","last_unstale":"2024-03-06T07:31:08.711489+0000","last_undegraded":"2024-03-06T07:31:08.711489+0000","last_fullsized":"2024-03-06T07:31:08.711489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.137094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682111+0000","last_change":"2024-03-06T07:31:08.682111+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682111+0000","last_undegraded":"2024-03-06T07:31:08.682111+0000","last_fullsized":"2024-03-06T07:31:08.682111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712674+0000","last_change":"2024-03-06T07:31:08.710889+0000","last_active":"2024-03-06T07:31:08.712674+0000","last_peered":"2024-03-06T07:31:08.712674+0000","last_clean":"2024-03-06T07:31:08.712674+0000","last_became_active":"2024-03-06T07:31:08.709406+0000","last_became_peered":"2024-03-06T07:31:08.709406+0000","last_unstale":"2024-03-06T07:31:08.712674+0000","last_undegraded":"2024-03-06T07:31:08.712674+0000","last_fullsized":"2024-03-06T07:31:08.712674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:47.336954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681829+0000","last_change":"2024-03-06T07:31:08.681829+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681829+0000","last_undegraded":"2024-03-06T07:31:08.681829+0000","last_fullsized":"2024-03-06T07:31:08.681829+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.710997+0000","last_change":"2024-03-06T07:31:08.704271+0000","last_active":"2024-03-06T07:31:08.710997+0000","last_peered":"2024-03-06T07:31:08.710997+0000","last_clean":"2024-03-06T07:31:08.710997+0000","last_became_active":"2024-03-06T07:31:08.700885+0000","last_became_peered":"2024-03-06T07:31:08.700885+0000","last_unstale":"2024-03-06T07:31:08.710997+0000","last_undegraded":"2024-03-06T07:31:08.710997+0000","last_fullsized":"2024-03-06T07:31:08.710997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:46.809806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.681577+0000","last_change":"2024-03-06T07:31:08.681577+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.681577+0000","last_undegraded":"2024-03-06T07:31:08.681577+0000","last_fullsized":"2024-03-06T07:31:08.681577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.705142+0000","last_change":"2024-03-06T07:31:08.698893+0000","last_active":"2024-03-06T07:31:08.705142+0000","last_peered":"2024-03-06T07:31:08.705142+0000","last_clean":"2024-03-06T07:31:08.705142+0000","last_became_active":"2024-03-06T07:31:08.698419+0000","last_became_peered":"2024-03-06T07:31:08.698419+0000","last_unstale":"2024-03-06T07:31:08.705142+0000","last_undegraded":"2024-03-06T07:31:08.705142+0000","last_fullsized":"2024-03-06T07:31:08.705142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:15.902202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682852+0000","last_change":"2024-03-06T07:31:08.682852+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682852+0000","last_undegraded":"2024-03-06T07:31:08.682852+0000","last_fullsized":"2024-03-06T07:31:08.682852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.691192+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:08.691192+0000","last_peered":"2024-03-06T07:31:08.691192+0000","last_clean":"2024-03-06T07:31:08.691192+0000","last_became_active":"2024-03-06T07:31:08.687300+0000","last_became_peered":"2024-03-06T07:31:08.687300+0000","last_unstale":"2024-03-06T07:31:08.691192+0000","last_undegraded":"2024-03-06T07:31:08.691192+0000","last_fullsized":"2024-03-06T07:31:08.691192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:25.951914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.674916+0000","last_change":"2024-03-06T07:31:08.674916+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.674916+0000","last_undegraded":"2024-03-06T07:31:08.674916+0000","last_fullsized":"2024-03-06T07:31:08.674916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.712323+0000","last_change":"2024-03-06T07:31:08.710412+0000","last_active":"2024-03-06T07:31:08.712323+0000","last_peered":"2024-03-06T07:31:08.712323+0000","last_clean":"2024-03-06T07:31:08.712323+0000","last_became_active":"2024-03-06T07:31:08.710061+0000","last_became_peered":"2024-03-06T07:31:08.710061+0000","last_unstale":"2024-03-06T07:31:08.712323+0000","last_undegraded":"2024-03-06T07:31:08.712323+0000","last_fullsized":"2024-03-06T07:31:08.712323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.402843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.688028+0000","last_change":"2024-03-06T07:31:08.688028+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.688028+0000","last_undegraded":"2024-03-06T07:31:08.688028+0000","last_fullsized":"2024-03-06T07:31:08.688028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.700709+0000","last_change":"2024-03-06T07:31:08.695443+0000","last_active":"2024-03-06T07:31:08.700709+0000","last_peered":"2024-03-06T07:31:08.700709+0000","last_clean":"2024-03-06T07:31:08.700709+0000","last_became_active":"2024-03-06T07:31:08.695112+0000","last_became_peered":"2024-03-06T07:31:08.695112+0000","last_unstale":"2024-03-06T07:31:08.700709+0000","last_undegraded":"2024-03-06T07:31:08.700709+0000","last_fullsized":"2024-03-06T07:31:08.700709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:25.951914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.684771+0000","last_change":"2024-03-06T07:31:08.684771+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.684771+0000","last_undegraded":"2024-03-06T07:31:08.684771+0000","last_fullsized":"2024-03-06T07:31:08.684771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.694369+0000","last_change":"2024-03-06T07:31:08.691823+0000","last_active":"2024-03-06T07:31:08.694369+0000","last_peered":"2024-03-06T07:31:08.694369+0000","last_clean":"2024-03-06T07:31:08.694369+0000","last_became_active":"2024-03-06T07:31:08.691505+0000","last_became_peered":"2024-03-06T07:31:08.691505+0000","last_unstale":"2024-03-06T07:31:08.694369+0000","last_undegraded":"2024-03-06T07:31:08.694369+0000","last_fullsized":"2024-03-06T07:31:08.694369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:46.481274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.695512+0000","last_change":"2024-03-06T07:31:08.695512+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.695512+0000","last_undegraded":"2024-03-06T07:31:08.695512+0000","last_fullsized":"2024-03-06T07:31:08.695512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.699479+0000","last_change":"2024-03-06T07:31:08.698585+0000","last_active":"2024-03-06T07:31:08.699479+0000","last_peered":"2024-03-06T07:31:08.699479+0000","last_clean":"2024-03-06T07:31:08.699479+0000","last_became_active":"2024-03-06T07:31:08.698166+0000","last_became_peered":"2024-03-06T07:31:08.698166+0000","last_unstale":"2024-03-06T07:31:08.699479+0000","last_undegraded":"2024-03-06T07:31:08.699479+0000","last_fullsized":"2024-03-06T07:31:08.699479+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:12.877333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.680554+0000","last_change":"2024-03-06T07:31:08.680554+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.680554+0000","last_undegraded":"2024-03-06T07:31:08.680554+0000","last_fullsized":"2024-03-06T07:31:08.680554+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.697911+0000","last_change":"2024-03-06T07:31:08.696740+0000","last_active":"2024-03-06T07:31:08.697911+0000","last_peered":"2024-03-06T07:31:08.697911+0000","last_clean":"2024-03-06T07:31:08.697911+0000","last_became_active":"2024-03-06T07:31:08.696450+0000","last_became_peered":"2024-03-06T07:31:08.696450+0000","last_unstale":"2024-03-06T07:31:08.697911+0000","last_undegraded":"2024-03-06T07:31:08.697911+0000","last_fullsized":"2024-03-06T07:31:08.697911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:17.835616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.696783+0000","last_change":"2024-03-06T07:31:08.696783+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.696783+0000","last_undegraded":"2024-03-06T07:31:08.696783+0000","last_fullsized":"2024-03-06T07:31:08.696783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.694246+0000","last_change":"2024-03-06T07:31:08.692727+0000","last_active":"2024-03-06T07:31:08.694246+0000","last_peered":"2024-03-06T07:31:08.694246+0000","last_clean":"2024-03-06T07:31:08.694246+0000","last_became_active":"2024-03-06T07:31:08.692457+0000","last_became_peered":"2024-03-06T07:31:08.692457+0000","last_unstale":"2024-03-06T07:31:08.694246+0000","last_undegraded":"2024-03-06T07:31:08.694246+0000","last_fullsized":"2024-03-06T07:31:08.694246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.544528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.676557+0000","last_change":"2024-03-06T07:31:08.676557+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.676557+0000","last_undegraded":"2024-03-06T07:31:08.676557+0000","last_fullsized":"2024-03-06T07:31:08.676557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.705180+0000","last_change":"2024-03-06T07:31:08.703245+0000","last_active":"2024-03-06T07:31:08.705180+0000","last_peered":"2024-03-06T07:31:08.705180+0000","last_clean":"2024-03-06T07:31:08.705180+0000","last_became_active":"2024-03-06T07:31:08.702878+0000","last_became_peered":"2024-03-06T07:31:08.702878+0000","last_unstale":"2024-03-06T07:31:08.705180+0000","last_undegraded":"2024-03-06T07:31:08.705180+0000","last_fullsized":"2024-03-06T07:31:08.705180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:32.737748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.695845+0000","last_change":"2024-03-06T07:31:08.695845+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.695845+0000","last_undegraded":"2024-03-06T07:31:08.695845+0000","last_fullsized":"2024-03-06T07:31:08.695845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711260+0000","last_change":"2024-03-06T07:31:08.707266+0000","last_active":"2024-03-06T07:31:08.711260+0000","last_peered":"2024-03-06T07:31:08.711260+0000","last_clean":"2024-03-06T07:31:08.711260+0000","last_became_active":"2024-03-06T07:31:08.706774+0000","last_became_peered":"2024-03-06T07:31:08.706774+0000","last_unstale":"2024-03-06T07:31:08.711260+0000","last_undegraded":"2024-03-06T07:31:08.711260+0000","last_fullsized":"2024-03-06T07:31:08.711260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:49.878415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:08.682755+0000","last_change":"2024-03-06T07:31:08.682755+0000","last_active":"2024-03-06T07:31:08.660088+0000","last_peered":"2024-03-06T07:31:08.660088+0000","last_clean":"2024-03-06T07:31:08.660088+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:08.682755+0000","last_undegraded":"2024-03-06T07:31:08.682755+0000","last_fullsized":"2024-03-06T07:31:08.682755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.711533+0000","last_change":"2024-03-06T07:31:08.707358+0000","last_active":"2024-03-06T07:31:08.711533+0000","last_peered":"2024-03-06T07:31:08.711533+0000","last_clean":"2024-03-06T07:31:08.711533+0000","last_became_active":"2024-03-06T07:31:08.706392+0000","last_became_peered":"2024-03-06T07:31:08.706392+0000","last_unstale":"2024-03-06T07:31:08.711533+0000","last_undegraded":"2024-03-06T07:31:08.711533+0000","last_fullsized":"2024-03-06T07:31:08.711533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:24.349345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":253,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:08.706106+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:31:08.706106+0000","last_peered":"2024-03-06T07:31:08.706106+0000","last_clean":"2024-03-06T07:31:08.706106+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:31:08.706106+0000","last_undegraded":"2024-03-06T07:31:08.706106+0000","last_fullsized":"2024-03-06T07:31:08.706106+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542189,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":1404928,"data_stored":927284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542190,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542191,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":330641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542190,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1392640,"data_stored":921009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:14.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:14.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.923+0000 7f165c400640 1 Processor -- start 2024-03-06T07:31:14.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.923+0000 7f165c400640 1 -- start start 2024-03-06T07:31:14.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f165c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 0x7f1654168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f165c400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165406f1f0 con 0x7f165415e160 2024-03-06T07:31:14.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f165c400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540afce0 con 0x7f16540af820 2024-03-06T07:31:14.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f165c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165406b0f0 con 0x7f165406c590 2024-03-06T07:31:14.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f1659a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f165415e160 0x7f1654162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48176/0 (socket says 172.21.15.97:48176) 2024-03-06T07:31:14.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f1659a00640 1 -- 172.21.15.97:0/2543561421 learned_addr learned my addr 172.21.15.97:0/2543561421 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f165a400640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 0x7f1654168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.925+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150508942 0 0) 0x7f165406f1f0 con 0x7f165415e160 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f163c003520 con 0x7f165415e160 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781407747 0 0) 0x7f163c003520 con 0x7f165415e160 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 msgr2=0x7f1654168fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 0x7f1654168fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 >> v1:172.21.15.135:6789/0 conn(0x7f16540af820 legacy=0x7f16540b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165406ec30 con 0x7f165415e160 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1648001670 con 0x7f165415e160 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 >> v1:172.21.15.97:6789/0 conn(0x7f165415e160 legacy=0x7f1654162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 0x7f163c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f163c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c004d40 0x7f163c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165406b0f0 con 0x7f163c003f60 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c00bb50 con 0x7f163c004d40 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c00bd50 con 0x7f165406c590 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.926+0000 7f1659000640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f163c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f1659a00640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 0x7f163c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f165a400640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c004d40 0x7f163c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f1659000640 1 -- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 msgr2=0x7f163c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f1659000640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f165406c590 0x7f163c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f1659000640 1 -- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c004d40 msgr2=0x7f163c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f1659000640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c004d40 0x7f163c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.927+0000 7f1659000640 1 -- 172.21.15.97:0/2543561421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c003520 con 0x7f163c003f60 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f1659000640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f163c004800 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1640001720 tx=0x7f163c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe175d1167a7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640001370 con 0x7f163c003f60 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16400052d0 con 0x7f163c003f60 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f1653e00640 1 -- 172.21.15.97:0/2543561421 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640003830 con 0x7f163c003f60 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f165c400640 1 -- 172.21.15.97:0/2543561421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 msgr2=0x7f163c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f165c400640 1 --2- 172.21.15.97:0/2543561421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f163c004800 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1640001720 tx=0x7f163c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.929+0000 7f165a400640 1 -- 172.21.15.97:0/2543561421 reap_dead start 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.930+0000 7f165c400640 1 -- 172.21.15.97:0/2543561421 shutdown_connections 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.930+0000 7f165c400640 1 -- 172.21.15.97:0/2543561421 >> 172.21.15.97:0/2543561421 conn(0x7f16540d0340 msgr2=0x7f16540d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.930+0000 7f165c400640 1 -- 172.21.15.97:0/2543561421 shutdown_connections 2024-03-06T07:31:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.930+0000 7f165c400640 1 -- 172.21.15.97:0/2543561421 wait complete. 2024-03-06T07:31:14.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.931+0000 7f165c400640 1 Processor -- start 2024-03-06T07:31:14.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.932+0000 7f165c400640 1 -- start start 2024-03-06T07:31:14.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f165c400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f16541423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f165c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1654145520 0x7f16540cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f165c400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16540cf3b0 0x7f165415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f165c400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540b3ac0 con 0x7f163c003f60 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f165c400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540b5620 con 0x7f16540cf3b0 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f165c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16540b3c40 con 0x7f1654145520 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f1659a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f16541423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f1659a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f16541423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39764/0 (socket says 172.21.15.97:39764) 2024-03-06T07:31:14.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.933+0000 7f1659a00640 1 -- 172.21.15.97:0/3181514927 learned_addr learned my addr 172.21.15.97:0/3181514927 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1659a00640 1 -- 172.21.15.97:0/3181514927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1654145520 msgr2=0x7f16540cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1659a00640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1654145520 0x7f16540cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1659a00640 1 -- 172.21.15.97:0/3181514927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16540cf3b0 msgr2=0x7f165415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1659a00640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16540cf3b0 0x7f165415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1659a00640 1 -- 172.21.15.97:0/3181514927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654160ef0 con 0x7f163c003f60 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1659a00640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f16541423c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f16480072c0 tx=0x7f1648007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164801a070 con 0x7f163c003f60 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164800f8e0 con 0x7f163c003f60 2024-03-06T07:31:14.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164800fcb0 con 0x7f163c003f60 2024-03-06T07:31:14.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.934+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165415e8b0 con 0x7f163c003f60 2024-03-06T07:31:14.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.935+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1654161450 con 0x7f163c003f60 2024-03-06T07:31:14.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.936+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1654074ae0 con 0x7f163c003f60 2024-03-06T07:31:14.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.937+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f164801d360 con 0x7f163c003f60 2024-03-06T07:31:14.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.937+0000 7f1651600640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.938+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1648089a80 con 0x7f163c003f60 2024-03-06T07:31:14.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.938+0000 7f1659000640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.938+0000 7f1659000640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f163406b4d0 0x7f163406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1640001720 tx=0x7f164002f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:14.942+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1654160ef0 con 0x7f163c003f60 2024-03-06T07:31:15.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.138+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16540cc390 con 0x7f163406b4d0 2024-03-06T07:31:15.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.157+0000 7f1651600640 1 -- 172.21.15.97:0/3181514927 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f16540cc390 con 0x7f163406b4d0 2024-03-06T07:31:15.162 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:31:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.162+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f163406b4d0 msgr2=0x7f163406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.162+0000 7f165c400640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f163406b4d0 0x7f163406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1640001720 tx=0x7f164002f000 comp rx=0 tx=0).stop 2024-03-06T07:31:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.162+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 msgr2=0x7f16541423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.162+0000 7f165c400640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f16541423c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f16480072c0 tx=0x7f1648007780 comp rx=0 tx=0).stop 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 shutdown_connections 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16540cf3b0 0x7f165415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1654145520 0x7f16540cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 --2- 172.21.15.97:0/3181514927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c003f60 0x7f16541423c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 >> 172.21.15.97:0/3181514927 conn(0x7f16540d0340 msgr2=0x7f16540ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 shutdown_connections 2024-03-06T07:31:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.163+0000 7f165c400640 1 -- 172.21.15.97:0/3181514927 wait complete. 2024-03-06T07:31:15.183 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-03-06T07:31:14.965743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257256,"kb_used_data":7752,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717464,"statfs":{"total":773094113280,"available":772830683136,"internally_reserved":0,"allocated":7938048,"data_stored":3985484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.295446"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314242+0000","last_change":"2024-03-06T07:31:09.738280+0000","last_active":"2024-03-06T07:31:11.314242+0000","last_peered":"2024-03-06T07:31:11.314242+0000","last_clean":"2024-03-06T07:31:11.314242+0000","last_became_active":"2024-03-06T07:31:09.737919+0000","last_became_peered":"2024-03-06T07:31:09.737919+0000","last_unstale":"2024-03-06T07:31:11.314242+0000","last_undegraded":"2024-03-06T07:31:11.314242+0000","last_fullsized":"2024-03-06T07:31:11.314242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:05.838129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701003+0000","last_change":"2024-03-06T07:31:09.731844+0000","last_active":"2024-03-06T07:31:10.701003+0000","last_peered":"2024-03-06T07:31:10.701003+0000","last_clean":"2024-03-06T07:31:10.701003+0000","last_became_active":"2024-03-06T07:31:09.731497+0000","last_became_peered":"2024-03-06T07:31:09.731497+0000","last_unstale":"2024-03-06T07:31:10.701003+0000","last_undegraded":"2024-03-06T07:31:10.701003+0000","last_fullsized":"2024-03-06T07:31:10.701003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:48.639188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.210992+0000","last_change":"2024-03-06T07:31:09.720966+0000","last_active":"2024-03-06T07:31:11.210992+0000","last_peered":"2024-03-06T07:31:11.210992+0000","last_clean":"2024-03-06T07:31:11.210992+0000","last_became_active":"2024-03-06T07:31:09.717482+0000","last_became_peered":"2024-03-06T07:31:09.717482+0000","last_unstale":"2024-03-06T07:31:11.210992+0000","last_undegraded":"2024-03-06T07:31:11.210992+0000","last_fullsized":"2024-03-06T07:31:11.210992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:08.762253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312031+0000","last_change":"2024-03-06T07:31:09.732801+0000","last_active":"2024-03-06T07:31:11.312031+0000","last_peered":"2024-03-06T07:31:11.312031+0000","last_clean":"2024-03-06T07:31:11.312031+0000","last_became_active":"2024-03-06T07:31:09.729989+0000","last_became_peered":"2024-03-06T07:31:09.729989+0000","last_unstale":"2024-03-06T07:31:11.312031+0000","last_undegraded":"2024-03-06T07:31:11.312031+0000","last_fullsized":"2024-03-06T07:31:11.312031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:37.643357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699389+0000","last_change":"2024-03-06T07:31:09.735427+0000","last_active":"2024-03-06T07:31:10.699389+0000","last_peered":"2024-03-06T07:31:10.699389+0000","last_clean":"2024-03-06T07:31:10.699389+0000","last_became_active":"2024-03-06T07:31:09.735240+0000","last_became_peered":"2024-03-06T07:31:09.735240+0000","last_unstale":"2024-03-06T07:31:10.699389+0000","last_undegraded":"2024-03-06T07:31:10.699389+0000","last_fullsized":"2024-03-06T07:31:10.699389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:25.357842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699572+0000","last_change":"2024-03-06T07:31:09.735149+0000","last_active":"2024-03-06T07:31:10.699572+0000","last_peered":"2024-03-06T07:31:10.699572+0000","last_clean":"2024-03-06T07:31:10.699572+0000","last_became_active":"2024-03-06T07:31:09.730477+0000","last_became_peered":"2024-03-06T07:31:09.730477+0000","last_unstale":"2024-03-06T07:31:10.699572+0000","last_undegraded":"2024-03-06T07:31:10.699572+0000","last_fullsized":"2024-03-06T07:31:10.699572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:16.910700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736731+0000","last_change":"2024-03-06T07:31:09.719251+0000","last_active":"2024-03-06T07:31:10.736731+0000","last_peered":"2024-03-06T07:31:10.736731+0000","last_clean":"2024-03-06T07:31:10.736731+0000","last_became_active":"2024-03-06T07:31:09.718763+0000","last_became_peered":"2024-03-06T07:31:09.718763+0000","last_unstale":"2024-03-06T07:31:10.736731+0000","last_undegraded":"2024-03-06T07:31:10.736731+0000","last_fullsized":"2024-03-06T07:31:10.736731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:53.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699419+0000","last_change":"2024-03-06T07:31:09.734091+0000","last_active":"2024-03-06T07:31:10.699419+0000","last_peered":"2024-03-06T07:31:10.699419+0000","last_clean":"2024-03-06T07:31:10.699419+0000","last_became_active":"2024-03-06T07:31:09.733787+0000","last_became_peered":"2024-03-06T07:31:09.733787+0000","last_unstale":"2024-03-06T07:31:10.699419+0000","last_undegraded":"2024-03-06T07:31:10.699419+0000","last_fullsized":"2024-03-06T07:31:10.699419+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:26.960412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700953+0000","last_change":"2024-03-06T07:31:09.728764+0000","last_active":"2024-03-06T07:31:10.700953+0000","last_peered":"2024-03-06T07:31:10.700953+0000","last_clean":"2024-03-06T07:31:10.700953+0000","last_became_active":"2024-03-06T07:31:09.728236+0000","last_became_peered":"2024-03-06T07:31:09.728236+0000","last_unstale":"2024-03-06T07:31:10.700953+0000","last_undegraded":"2024-03-06T07:31:10.700953+0000","last_fullsized":"2024-03-06T07:31:10.700953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:09.672015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314990+0000","last_change":"2024-03-06T07:31:09.733888+0000","last_active":"2024-03-06T07:31:11.314990+0000","last_peered":"2024-03-06T07:31:11.314990+0000","last_clean":"2024-03-06T07:31:11.314990+0000","last_became_active":"2024-03-06T07:31:09.733591+0000","last_became_peered":"2024-03-06T07:31:09.733591+0000","last_unstale":"2024-03-06T07:31:11.314990+0000","last_undegraded":"2024-03-06T07:31:11.314990+0000","last_fullsized":"2024-03-06T07:31:11.314990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:56.718632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314183+0000","last_change":"2024-03-06T07:31:09.728837+0000","last_active":"2024-03-06T07:31:11.314183+0000","last_peered":"2024-03-06T07:31:11.314183+0000","last_clean":"2024-03-06T07:31:11.314183+0000","last_became_active":"2024-03-06T07:31:09.724590+0000","last_became_peered":"2024-03-06T07:31:09.724590+0000","last_unstale":"2024-03-06T07:31:11.314183+0000","last_undegraded":"2024-03-06T07:31:11.314183+0000","last_fullsized":"2024-03-06T07:31:11.314183+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.779623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312138+0000","last_change":"2024-03-06T07:31:09.725547+0000","last_active":"2024-03-06T07:31:11.312138+0000","last_peered":"2024-03-06T07:31:11.312138+0000","last_clean":"2024-03-06T07:31:11.312138+0000","last_became_active":"2024-03-06T07:31:09.724975+0000","last_became_peered":"2024-03-06T07:31:09.724975+0000","last_unstale":"2024-03-06T07:31:11.312138+0000","last_undegraded":"2024-03-06T07:31:11.312138+0000","last_fullsized":"2024-03-06T07:31:11.312138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:33.626731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699223+0000","last_change":"2024-03-06T07:31:09.730705+0000","last_active":"2024-03-06T07:31:10.699223+0000","last_peered":"2024-03-06T07:31:10.699223+0000","last_clean":"2024-03-06T07:31:10.699223+0000","last_became_active":"2024-03-06T07:31:09.730523+0000","last_became_peered":"2024-03-06T07:31:09.730523+0000","last_unstale":"2024-03-06T07:31:10.699223+0000","last_undegraded":"2024-03-06T07:31:10.699223+0000","last_fullsized":"2024-03-06T07:31:10.699223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:23.028507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699528+0000","last_change":"2024-03-06T07:31:09.725710+0000","last_active":"2024-03-06T07:31:10.699528+0000","last_peered":"2024-03-06T07:31:10.699528+0000","last_clean":"2024-03-06T07:31:10.699528+0000","last_became_active":"2024-03-06T07:31:09.725335+0000","last_became_peered":"2024-03-06T07:31:09.725335+0000","last_unstale":"2024-03-06T07:31:10.699528+0000","last_undegraded":"2024-03-06T07:31:10.699528+0000","last_fullsized":"2024-03-06T07:31:10.699528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:21.145591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698581+0000","last_change":"2024-03-06T07:31:09.734193+0000","last_active":"2024-03-06T07:31:10.698581+0000","last_peered":"2024-03-06T07:31:10.698581+0000","last_clean":"2024-03-06T07:31:10.698581+0000","last_became_active":"2024-03-06T07:31:09.733891+0000","last_became_peered":"2024-03-06T07:31:09.733891+0000","last_unstale":"2024-03-06T07:31:10.698581+0000","last_undegraded":"2024-03-06T07:31:10.698581+0000","last_fullsized":"2024-03-06T07:31:10.698581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:57.434602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312912+0000","last_change":"2024-03-06T07:31:09.746880+0000","last_active":"2024-03-06T07:31:11.312912+0000","last_peered":"2024-03-06T07:31:11.312912+0000","last_clean":"2024-03-06T07:31:11.312912+0000","last_became_active":"2024-03-06T07:31:09.746461+0000","last_became_peered":"2024-03-06T07:31:09.746461+0000","last_unstale":"2024-03-06T07:31:11.312912+0000","last_undegraded":"2024-03-06T07:31:11.312912+0000","last_fullsized":"2024-03-06T07:31:11.312912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:41.617082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700999+0000","last_change":"2024-03-06T07:31:09.728614+0000","last_active":"2024-03-06T07:31:10.700999+0000","last_peered":"2024-03-06T07:31:10.700999+0000","last_clean":"2024-03-06T07:31:10.700999+0000","last_became_active":"2024-03-06T07:31:09.728201+0000","last_became_peered":"2024-03-06T07:31:09.728201+0000","last_unstale":"2024-03-06T07:31:10.700999+0000","last_undegraded":"2024-03-06T07:31:10.700999+0000","last_fullsized":"2024-03-06T07:31:10.700999+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:39.624880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312482+0000","last_change":"2024-03-06T07:31:09.739356+0000","last_active":"2024-03-06T07:31:11.312482+0000","last_peered":"2024-03-06T07:31:11.312482+0000","last_clean":"2024-03-06T07:31:11.312482+0000","last_became_active":"2024-03-06T07:31:09.738061+0000","last_became_peered":"2024-03-06T07:31:09.738061+0000","last_unstale":"2024-03-06T07:31:11.312482+0000","last_undegraded":"2024-03-06T07:31:11.312482+0000","last_fullsized":"2024-03-06T07:31:11.312482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:16.897453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699939+0000","last_change":"2024-03-06T07:31:09.734267+0000","last_active":"2024-03-06T07:31:10.699939+0000","last_peered":"2024-03-06T07:31:10.699939+0000","last_clean":"2024-03-06T07:31:10.699939+0000","last_became_active":"2024-03-06T07:31:09.733544+0000","last_became_peered":"2024-03-06T07:31:09.733544+0000","last_unstale":"2024-03-06T07:31:10.699939+0000","last_undegraded":"2024-03-06T07:31:10.699939+0000","last_fullsized":"2024-03-06T07:31:10.699939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:48.345451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700585+0000","last_change":"2024-03-06T07:31:09.723844+0000","last_active":"2024-03-06T07:31:10.700585+0000","last_peered":"2024-03-06T07:31:10.700585+0000","last_clean":"2024-03-06T07:31:10.700585+0000","last_became_active":"2024-03-06T07:31:09.723522+0000","last_became_peered":"2024-03-06T07:31:09.723522+0000","last_unstale":"2024-03-06T07:31:10.700585+0000","last_undegraded":"2024-03-06T07:31:10.700585+0000","last_fullsized":"2024-03-06T07:31:10.700585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:48.499605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313402+0000","last_change":"2024-03-06T07:31:09.721457+0000","last_active":"2024-03-06T07:31:11.313402+0000","last_peered":"2024-03-06T07:31:11.313402+0000","last_clean":"2024-03-06T07:31:11.313402+0000","last_became_active":"2024-03-06T07:31:09.721122+0000","last_became_peered":"2024-03-06T07:31:09.721122+0000","last_unstale":"2024-03-06T07:31:11.313402+0000","last_undegraded":"2024-03-06T07:31:11.313402+0000","last_fullsized":"2024-03-06T07:31:11.313402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:09.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313386+0000","last_change":"2024-03-06T07:31:09.721253+0000","last_active":"2024-03-06T07:31:11.313386+0000","last_peered":"2024-03-06T07:31:11.313386+0000","last_clean":"2024-03-06T07:31:11.313386+0000","last_became_active":"2024-03-06T07:31:09.720907+0000","last_became_peered":"2024-03-06T07:31:09.720907+0000","last_unstale":"2024-03-06T07:31:11.313386+0000","last_undegraded":"2024-03-06T07:31:11.313386+0000","last_fullsized":"2024-03-06T07:31:11.313386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:05.077124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314875+0000","last_change":"2024-03-06T07:31:09.728686+0000","last_active":"2024-03-06T07:31:11.314875+0000","last_peered":"2024-03-06T07:31:11.314875+0000","last_clean":"2024-03-06T07:31:11.314875+0000","last_became_active":"2024-03-06T07:31:09.728378+0000","last_became_peered":"2024-03-06T07:31:09.728378+0000","last_unstale":"2024-03-06T07:31:11.314875+0000","last_undegraded":"2024-03-06T07:31:11.314875+0000","last_fullsized":"2024-03-06T07:31:11.314875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:43.899640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313546+0000","last_change":"2024-03-06T07:31:09.745709+0000","last_active":"2024-03-06T07:31:11.313546+0000","last_peered":"2024-03-06T07:31:11.313546+0000","last_clean":"2024-03-06T07:31:11.313546+0000","last_became_active":"2024-03-06T07:31:09.744468+0000","last_became_peered":"2024-03-06T07:31:09.744468+0000","last_unstale":"2024-03-06T07:31:11.313546+0000","last_undegraded":"2024-03-06T07:31:11.313546+0000","last_fullsized":"2024-03-06T07:31:11.313546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:47.710080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315154+0000","last_change":"2024-03-06T07:31:09.722571+0000","last_active":"2024-03-06T07:31:11.315154+0000","last_peered":"2024-03-06T07:31:11.315154+0000","last_clean":"2024-03-06T07:31:11.315154+0000","last_became_active":"2024-03-06T07:31:09.722243+0000","last_became_peered":"2024-03-06T07:31:09.722243+0000","last_unstale":"2024-03-06T07:31:11.315154+0000","last_undegraded":"2024-03-06T07:31:11.315154+0000","last_fullsized":"2024-03-06T07:31:11.315154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:00.460321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737350+0000","last_change":"2024-03-06T07:31:09.743119+0000","last_active":"2024-03-06T07:31:10.737350+0000","last_peered":"2024-03-06T07:31:10.737350+0000","last_clean":"2024-03-06T07:31:10.737350+0000","last_became_active":"2024-03-06T07:31:09.742848+0000","last_became_peered":"2024-03-06T07:31:09.742848+0000","last_unstale":"2024-03-06T07:31:10.737350+0000","last_undegraded":"2024-03-06T07:31:10.737350+0000","last_fullsized":"2024-03-06T07:31:10.737350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:28.617476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211399+0000","last_change":"2024-03-06T07:31:09.720989+0000","last_active":"2024-03-06T07:31:11.211399+0000","last_peered":"2024-03-06T07:31:11.211399+0000","last_clean":"2024-03-06T07:31:11.211399+0000","last_became_active":"2024-03-06T07:31:09.720480+0000","last_became_peered":"2024-03-06T07:31:09.720480+0000","last_unstale":"2024-03-06T07:31:11.211399+0000","last_undegraded":"2024-03-06T07:31:11.211399+0000","last_fullsized":"2024-03-06T07:31:11.211399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:45.345281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314404+0000","last_change":"2024-03-06T07:31:09.727631+0000","last_active":"2024-03-06T07:31:11.314404+0000","last_peered":"2024-03-06T07:31:11.314404+0000","last_clean":"2024-03-06T07:31:11.314404+0000","last_became_active":"2024-03-06T07:31:09.727329+0000","last_became_peered":"2024-03-06T07:31:09.727329+0000","last_unstale":"2024-03-06T07:31:11.314404+0000","last_undegraded":"2024-03-06T07:31:11.314404+0000","last_fullsized":"2024-03-06T07:31:11.314404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.841904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211396+0000","last_change":"2024-03-06T07:31:09.719664+0000","last_active":"2024-03-06T07:31:11.211396+0000","last_peered":"2024-03-06T07:31:11.211396+0000","last_clean":"2024-03-06T07:31:11.211396+0000","last_became_active":"2024-03-06T07:31:09.719149+0000","last_became_peered":"2024-03-06T07:31:09.719149+0000","last_unstale":"2024-03-06T07:31:11.211396+0000","last_undegraded":"2024-03-06T07:31:11.211396+0000","last_fullsized":"2024-03-06T07:31:11.211396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:00.352698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736992+0000","last_change":"2024-03-06T07:31:09.742657+0000","last_active":"2024-03-06T07:31:10.736992+0000","last_peered":"2024-03-06T07:31:10.736992+0000","last_clean":"2024-03-06T07:31:10.736992+0000","last_became_active":"2024-03-06T07:31:09.742333+0000","last_became_peered":"2024-03-06T07:31:09.742333+0000","last_unstale":"2024-03-06T07:31:10.736992+0000","last_undegraded":"2024-03-06T07:31:10.736992+0000","last_fullsized":"2024-03-06T07:31:10.736992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:23.028507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737334+0000","last_change":"2024-03-06T07:31:09.731801+0000","last_active":"2024-03-06T07:31:10.737334+0000","last_peered":"2024-03-06T07:31:10.737334+0000","last_clean":"2024-03-06T07:31:10.737334+0000","last_became_active":"2024-03-06T07:31:09.731374+0000","last_became_peered":"2024-03-06T07:31:09.731374+0000","last_unstale":"2024-03-06T07:31:10.737334+0000","last_undegraded":"2024-03-06T07:31:10.737334+0000","last_fullsized":"2024-03-06T07:31:10.737334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:15.954452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699872+0000","last_change":"2024-03-06T07:31:09.731886+0000","last_active":"2024-03-06T07:31:10.699872+0000","last_peered":"2024-03-06T07:31:10.699872+0000","last_clean":"2024-03-06T07:31:10.699872+0000","last_became_active":"2024-03-06T07:31:09.731581+0000","last_became_peered":"2024-03-06T07:31:09.731581+0000","last_unstale":"2024-03-06T07:31:10.699872+0000","last_undegraded":"2024-03-06T07:31:10.699872+0000","last_fullsized":"2024-03-06T07:31:10.699872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:12.972421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315323+0000","last_change":"2024-03-06T07:31:09.737867+0000","last_active":"2024-03-06T07:31:11.315323+0000","last_peered":"2024-03-06T07:31:11.315323+0000","last_clean":"2024-03-06T07:31:11.315323+0000","last_became_active":"2024-03-06T07:31:09.737471+0000","last_became_peered":"2024-03-06T07:31:09.737471+0000","last_unstale":"2024-03-06T07:31:11.315323+0000","last_undegraded":"2024-03-06T07:31:11.315323+0000","last_fullsized":"2024-03-06T07:31:11.315323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:44.379179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315796+0000","last_change":"2024-03-06T07:31:08.709472+0000","last_active":"2024-03-06T07:31:11.315796+0000","last_peered":"2024-03-06T07:31:11.315796+0000","last_clean":"2024-03-06T07:31:11.315796+0000","last_became_active":"2024-03-06T07:31:08.709183+0000","last_became_peered":"2024-03-06T07:31:08.709183+0000","last_unstale":"2024-03-06T07:31:11.315796+0000","last_undegraded":"2024-03-06T07:31:11.315796+0000","last_fullsized":"2024-03-06T07:31:11.315796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:18.088186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701172+0000","last_change":"2024-03-06T07:31:09.728911+0000","last_active":"2024-03-06T07:31:10.701172+0000","last_peered":"2024-03-06T07:31:10.701172+0000","last_clean":"2024-03-06T07:31:10.701172+0000","last_became_active":"2024-03-06T07:31:09.728505+0000","last_became_peered":"2024-03-06T07:31:09.728505+0000","last_unstale":"2024-03-06T07:31:10.701172+0000","last_undegraded":"2024-03-06T07:31:10.701172+0000","last_fullsized":"2024-03-06T07:31:10.701172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:21.941224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699503+0000","last_change":"2024-03-06T07:31:08.690969+0000","last_active":"2024-03-06T07:31:10.699503+0000","last_peered":"2024-03-06T07:31:10.699503+0000","last_clean":"2024-03-06T07:31:10.699503+0000","last_became_active":"2024-03-06T07:31:08.690694+0000","last_became_peered":"2024-03-06T07:31:08.690694+0000","last_unstale":"2024-03-06T07:31:10.699503+0000","last_undegraded":"2024-03-06T07:31:10.699503+0000","last_fullsized":"2024-03-06T07:31:10.699503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699674+0000","last_change":"2024-03-06T07:31:09.733888+0000","last_active":"2024-03-06T07:31:10.699674+0000","last_peered":"2024-03-06T07:31:10.699674+0000","last_clean":"2024-03-06T07:31:10.699674+0000","last_became_active":"2024-03-06T07:31:09.733189+0000","last_became_peered":"2024-03-06T07:31:09.733189+0000","last_unstale":"2024-03-06T07:31:10.699674+0000","last_undegraded":"2024-03-06T07:31:10.699674+0000","last_fullsized":"2024-03-06T07:31:10.699674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:06.118647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314826+0000","last_change":"2024-03-06T07:31:08.711193+0000","last_active":"2024-03-06T07:31:11.314826+0000","last_peered":"2024-03-06T07:31:11.314826+0000","last_clean":"2024-03-06T07:31:11.314826+0000","last_became_active":"2024-03-06T07:31:08.710729+0000","last_became_peered":"2024-03-06T07:31:08.710729+0000","last_unstale":"2024-03-06T07:31:11.314826+0000","last_undegraded":"2024-03-06T07:31:11.314826+0000","last_fullsized":"2024-03-06T07:31:11.314826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.461751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314596+0000","last_change":"2024-03-06T07:31:09.720876+0000","last_active":"2024-03-06T07:31:11.314596+0000","last_peered":"2024-03-06T07:31:11.314596+0000","last_clean":"2024-03-06T07:31:11.314596+0000","last_became_active":"2024-03-06T07:31:09.720571+0000","last_became_peered":"2024-03-06T07:31:09.720571+0000","last_unstale":"2024-03-06T07:31:11.314596+0000","last_undegraded":"2024-03-06T07:31:11.314596+0000","last_fullsized":"2024-03-06T07:31:11.314596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:34.904471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699964+0000","last_change":"2024-03-06T07:31:08.703780+0000","last_active":"2024-03-06T07:31:10.699964+0000","last_peered":"2024-03-06T07:31:10.699964+0000","last_clean":"2024-03-06T07:31:10.699964+0000","last_became_active":"2024-03-06T07:31:08.703434+0000","last_became_peered":"2024-03-06T07:31:08.703434+0000","last_unstale":"2024-03-06T07:31:10.699964+0000","last_undegraded":"2024-03-06T07:31:10.699964+0000","last_fullsized":"2024-03-06T07:31:10.699964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:00.577536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737151+0000","last_change":"2024-03-06T07:31:09.719528+0000","last_active":"2024-03-06T07:31:10.737151+0000","last_peered":"2024-03-06T07:31:10.737151+0000","last_clean":"2024-03-06T07:31:10.737151+0000","last_became_active":"2024-03-06T07:31:09.719252+0000","last_became_peered":"2024-03-06T07:31:09.719252+0000","last_unstale":"2024-03-06T07:31:10.737151+0000","last_undegraded":"2024-03-06T07:31:10.737151+0000","last_fullsized":"2024-03-06T07:31:10.737151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:03.282222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211556+0000","last_change":"2024-03-06T07:31:08.695941+0000","last_active":"2024-03-06T07:31:11.211556+0000","last_peered":"2024-03-06T07:31:11.211556+0000","last_clean":"2024-03-06T07:31:11.211556+0000","last_became_active":"2024-03-06T07:31:08.695727+0000","last_became_peered":"2024-03-06T07:31:08.695727+0000","last_unstale":"2024-03-06T07:31:11.211556+0000","last_undegraded":"2024-03-06T07:31:11.211556+0000","last_fullsized":"2024-03-06T07:31:11.211556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.258836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313548+0000","last_change":"2024-03-06T07:31:09.722880+0000","last_active":"2024-03-06T07:31:11.313548+0000","last_peered":"2024-03-06T07:31:11.313548+0000","last_clean":"2024-03-06T07:31:11.313548+0000","last_became_active":"2024-03-06T07:31:09.722301+0000","last_became_peered":"2024-03-06T07:31:09.722301+0000","last_unstale":"2024-03-06T07:31:11.313548+0000","last_undegraded":"2024-03-06T07:31:11.313548+0000","last_fullsized":"2024-03-06T07:31:11.313548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:54.332390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700385+0000","last_change":"2024-03-06T07:31:08.701504+0000","last_active":"2024-03-06T07:31:10.700385+0000","last_peered":"2024-03-06T07:31:10.700385+0000","last_clean":"2024-03-06T07:31:10.700385+0000","last_became_active":"2024-03-06T07:31:08.701175+0000","last_became_peered":"2024-03-06T07:31:08.701175+0000","last_unstale":"2024-03-06T07:31:10.700385+0000","last_undegraded":"2024-03-06T07:31:10.700385+0000","last_fullsized":"2024-03-06T07:31:10.700385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:46.370019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698911+0000","last_change":"2024-03-06T07:31:09.726824+0000","last_active":"2024-03-06T07:31:10.698911+0000","last_peered":"2024-03-06T07:31:10.698911+0000","last_clean":"2024-03-06T07:31:10.698911+0000","last_became_active":"2024-03-06T07:31:09.726046+0000","last_became_peered":"2024-03-06T07:31:09.726046+0000","last_unstale":"2024-03-06T07:31:10.698911+0000","last_undegraded":"2024-03-06T07:31:10.698911+0000","last_fullsized":"2024-03-06T07:31:10.698911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:50.886912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311718+0000","last_change":"2024-03-06T07:31:08.690249+0000","last_active":"2024-03-06T07:31:11.311718+0000","last_peered":"2024-03-06T07:31:11.311718+0000","last_clean":"2024-03-06T07:31:11.311718+0000","last_became_active":"2024-03-06T07:31:08.689995+0000","last_became_peered":"2024-03-06T07:31:08.689995+0000","last_unstale":"2024-03-06T07:31:11.311718+0000","last_undegraded":"2024-03-06T07:31:11.311718+0000","last_fullsized":"2024-03-06T07:31:11.311718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:58.302278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698603+0000","last_change":"2024-03-06T07:31:09.724364+0000","last_active":"2024-03-06T07:31:10.698603+0000","last_peered":"2024-03-06T07:31:10.698603+0000","last_clean":"2024-03-06T07:31:10.698603+0000","last_became_active":"2024-03-06T07:31:09.724084+0000","last_became_peered":"2024-03-06T07:31:09.724084+0000","last_unstale":"2024-03-06T07:31:10.698603+0000","last_undegraded":"2024-03-06T07:31:10.698603+0000","last_fullsized":"2024-03-06T07:31:10.698603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.250553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698951+0000","last_change":"2024-03-06T07:31:08.694935+0000","last_active":"2024-03-06T07:31:10.698951+0000","last_peered":"2024-03-06T07:31:10.698951+0000","last_clean":"2024-03-06T07:31:10.698951+0000","last_became_active":"2024-03-06T07:31:08.694548+0000","last_became_peered":"2024-03-06T07:31:08.694548+0000","last_unstale":"2024-03-06T07:31:10.698951+0000","last_undegraded":"2024-03-06T07:31:10.698951+0000","last_fullsized":"2024-03-06T07:31:10.698951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:51.965506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737616+0000","last_change":"2024-03-06T07:31:09.731317+0000","last_active":"2024-03-06T07:31:10.737616+0000","last_peered":"2024-03-06T07:31:10.737616+0000","last_clean":"2024-03-06T07:31:10.737616+0000","last_became_active":"2024-03-06T07:31:09.730595+0000","last_became_peered":"2024-03-06T07:31:09.730595+0000","last_unstale":"2024-03-06T07:31:10.737616+0000","last_undegraded":"2024-03-06T07:31:10.737616+0000","last_fullsized":"2024-03-06T07:31:10.737616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:47.710080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315404+0000","last_change":"2024-03-06T07:31:08.691602+0000","last_active":"2024-03-06T07:31:11.315404+0000","last_peered":"2024-03-06T07:31:11.315404+0000","last_clean":"2024-03-06T07:31:11.315404+0000","last_became_active":"2024-03-06T07:31:08.690752+0000","last_became_peered":"2024-03-06T07:31:08.690752+0000","last_unstale":"2024-03-06T07:31:11.315404+0000","last_undegraded":"2024-03-06T07:31:11.315404+0000","last_fullsized":"2024-03-06T07:31:11.315404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:58.046062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315603+0000","last_change":"2024-03-06T07:31:09.745602+0000","last_active":"2024-03-06T07:31:11.315603+0000","last_peered":"2024-03-06T07:31:11.315603+0000","last_clean":"2024-03-06T07:31:11.315603+0000","last_became_active":"2024-03-06T07:31:09.742517+0000","last_became_peered":"2024-03-06T07:31:09.742517+0000","last_unstale":"2024-03-06T07:31:11.315603+0000","last_undegraded":"2024-03-06T07:31:11.315603+0000","last_fullsized":"2024-03-06T07:31:11.315603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:47.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701236+0000","last_change":"2024-03-06T07:31:08.710975+0000","last_active":"2024-03-06T07:31:10.701236+0000","last_peered":"2024-03-06T07:31:10.701236+0000","last_clean":"2024-03-06T07:31:10.701236+0000","last_became_active":"2024-03-06T07:31:08.710624+0000","last_became_peered":"2024-03-06T07:31:08.710624+0000","last_unstale":"2024-03-06T07:31:10.701236+0000","last_undegraded":"2024-03-06T07:31:10.701236+0000","last_fullsized":"2024-03-06T07:31:10.701236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:42.616395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699823+0000","last_change":"2024-03-06T07:31:09.736963+0000","last_active":"2024-03-06T07:31:10.699823+0000","last_peered":"2024-03-06T07:31:10.699823+0000","last_clean":"2024-03-06T07:31:10.699823+0000","last_became_active":"2024-03-06T07:31:09.736635+0000","last_became_peered":"2024-03-06T07:31:09.736635+0000","last_unstale":"2024-03-06T07:31:10.699823+0000","last_undegraded":"2024-03-06T07:31:10.699823+0000","last_fullsized":"2024-03-06T07:31:10.699823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:31.941454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211510+0000","last_change":"2024-03-06T07:31:08.701415+0000","last_active":"2024-03-06T07:31:11.211510+0000","last_peered":"2024-03-06T07:31:11.211510+0000","last_clean":"2024-03-06T07:31:11.211510+0000","last_became_active":"2024-03-06T07:31:08.700947+0000","last_became_peered":"2024-03-06T07:31:08.700947+0000","last_unstale":"2024-03-06T07:31:11.211510+0000","last_undegraded":"2024-03-06T07:31:11.211510+0000","last_fullsized":"2024-03-06T07:31:11.211510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:51.621273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700977+0000","last_change":"2024-03-06T07:31:09.736417+0000","last_active":"2024-03-06T07:31:10.700977+0000","last_peered":"2024-03-06T07:31:10.700977+0000","last_clean":"2024-03-06T07:31:10.700977+0000","last_became_active":"2024-03-06T07:31:09.736032+0000","last_became_peered":"2024-03-06T07:31:09.736032+0000","last_unstale":"2024-03-06T07:31:10.700977+0000","last_undegraded":"2024-03-06T07:31:10.700977+0000","last_fullsized":"2024-03-06T07:31:10.700977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:47.378517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312462+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:11.312462+0000","last_peered":"2024-03-06T07:31:11.312462+0000","last_clean":"2024-03-06T07:31:11.312462+0000","last_became_active":"2024-03-06T07:31:08.687214+0000","last_became_peered":"2024-03-06T07:31:08.687214+0000","last_unstale":"2024-03-06T07:31:11.312462+0000","last_undegraded":"2024-03-06T07:31:11.312462+0000","last_fullsized":"2024-03-06T07:31:11.312462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:20.932727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314098+0000","last_change":"2024-03-06T07:31:09.734608+0000","last_active":"2024-03-06T07:31:11.314098+0000","last_peered":"2024-03-06T07:31:11.314098+0000","last_clean":"2024-03-06T07:31:11.314098+0000","last_became_active":"2024-03-06T07:31:09.734395+0000","last_became_peered":"2024-03-06T07:31:09.734395+0000","last_unstale":"2024-03-06T07:31:11.314098+0000","last_undegraded":"2024-03-06T07:31:11.314098+0000","last_fullsized":"2024-03-06T07:31:11.314098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:57.600744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312003+0000","last_change":"2024-03-06T07:31:08.702804+0000","last_active":"2024-03-06T07:31:11.312003+0000","last_peered":"2024-03-06T07:31:11.312003+0000","last_clean":"2024-03-06T07:31:11.312003+0000","last_became_active":"2024-03-06T07:31:08.702494+0000","last_became_peered":"2024-03-06T07:31:08.702494+0000","last_unstale":"2024-03-06T07:31:11.312003+0000","last_undegraded":"2024-03-06T07:31:11.312003+0000","last_fullsized":"2024-03-06T07:31:11.312003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:38.616383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737294+0000","last_change":"2024-03-06T07:31:09.727382+0000","last_active":"2024-03-06T07:31:10.737294+0000","last_peered":"2024-03-06T07:31:10.737294+0000","last_clean":"2024-03-06T07:31:10.737294+0000","last_became_active":"2024-03-06T07:31:09.725803+0000","last_became_peered":"2024-03-06T07:31:09.725803+0000","last_unstale":"2024-03-06T07:31:10.737294+0000","last_undegraded":"2024-03-06T07:31:10.737294+0000","last_fullsized":"2024-03-06T07:31:10.737294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:48.499605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700674+0000","last_change":"2024-03-06T07:31:08.706279+0000","last_active":"2024-03-06T07:31:10.700674+0000","last_peered":"2024-03-06T07:31:10.700674+0000","last_clean":"2024-03-06T07:31:10.700674+0000","last_became_active":"2024-03-06T07:31:08.705857+0000","last_became_peered":"2024-03-06T07:31:08.705857+0000","last_unstale":"2024-03-06T07:31:10.700674+0000","last_undegraded":"2024-03-06T07:31:10.700674+0000","last_fullsized":"2024-03-06T07:31:10.700674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:38.616383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698468+0000","last_change":"2024-03-06T07:31:09.735918+0000","last_active":"2024-03-06T07:31:10.698468+0000","last_peered":"2024-03-06T07:31:10.698468+0000","last_clean":"2024-03-06T07:31:10.698468+0000","last_became_active":"2024-03-06T07:31:09.735302+0000","last_became_peered":"2024-03-06T07:31:09.735302+0000","last_unstale":"2024-03-06T07:31:10.698468+0000","last_undegraded":"2024-03-06T07:31:10.698468+0000","last_fullsized":"2024-03-06T07:31:10.698468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:09.747371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736808+0000","last_change":"2024-03-06T07:31:08.699062+0000","last_active":"2024-03-06T07:31:10.736808+0000","last_peered":"2024-03-06T07:31:10.736808+0000","last_clean":"2024-03-06T07:31:10.736808+0000","last_became_active":"2024-03-06T07:31:08.698767+0000","last_became_peered":"2024-03-06T07:31:08.698767+0000","last_unstale":"2024-03-06T07:31:10.736808+0000","last_undegraded":"2024-03-06T07:31:10.736808+0000","last_fullsized":"2024-03-06T07:31:10.736808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:05.110150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699664+0000","last_change":"2024-03-06T07:31:09.729881+0000","last_active":"2024-03-06T07:31:10.699664+0000","last_peered":"2024-03-06T07:31:10.699664+0000","last_clean":"2024-03-06T07:31:10.699664+0000","last_became_active":"2024-03-06T07:31:09.728303+0000","last_became_peered":"2024-03-06T07:31:09.728303+0000","last_unstale":"2024-03-06T07:31:10.699664+0000","last_undegraded":"2024-03-06T07:31:10.699664+0000","last_fullsized":"2024-03-06T07:31:10.699664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:39.411341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313005+0000","last_change":"2024-03-06T07:31:08.708615+0000","last_active":"2024-03-06T07:31:11.313005+0000","last_peered":"2024-03-06T07:31:11.313005+0000","last_clean":"2024-03-06T07:31:11.313005+0000","last_became_active":"2024-03-06T07:31:08.708149+0000","last_became_peered":"2024-03-06T07:31:08.708149+0000","last_unstale":"2024-03-06T07:31:11.313005+0000","last_undegraded":"2024-03-06T07:31:11.313005+0000","last_fullsized":"2024-03-06T07:31:11.313005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:19.844233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315110+0000","last_change":"2024-03-06T07:31:09.712575+0000","last_active":"2024-03-06T07:31:11.315110+0000","last_peered":"2024-03-06T07:31:11.315110+0000","last_clean":"2024-03-06T07:31:11.315110+0000","last_became_active":"2024-03-06T07:31:09.712036+0000","last_became_peered":"2024-03-06T07:31:09.712036+0000","last_unstale":"2024-03-06T07:31:11.315110+0000","last_undegraded":"2024-03-06T07:31:11.315110+0000","last_fullsized":"2024-03-06T07:31:11.315110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:05.374622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700610+0000","last_change":"2024-03-06T07:31:08.706291+0000","last_active":"2024-03-06T07:31:10.700610+0000","last_peered":"2024-03-06T07:31:10.700610+0000","last_clean":"2024-03-06T07:31:10.700610+0000","last_became_active":"2024-03-06T07:31:08.705537+0000","last_became_peered":"2024-03-06T07:31:08.705537+0000","last_unstale":"2024-03-06T07:31:10.700610+0000","last_undegraded":"2024-03-06T07:31:10.700610+0000","last_fullsized":"2024-03-06T07:31:10.700610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:58.716593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315787+0000","last_change":"2024-03-06T07:31:09.728588+0000","last_active":"2024-03-06T07:31:11.315787+0000","last_peered":"2024-03-06T07:31:11.315787+0000","last_clean":"2024-03-06T07:31:11.315787+0000","last_became_active":"2024-03-06T07:31:09.728244+0000","last_became_peered":"2024-03-06T07:31:09.728244+0000","last_unstale":"2024-03-06T07:31:11.315787+0000","last_undegraded":"2024-03-06T07:31:11.315787+0000","last_fullsized":"2024-03-06T07:31:11.315787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:02.267334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315263+0000","last_change":"2024-03-06T07:31:08.708950+0000","last_active":"2024-03-06T07:31:11.315263+0000","last_peered":"2024-03-06T07:31:11.315263+0000","last_clean":"2024-03-06T07:31:11.315263+0000","last_became_active":"2024-03-06T07:31:08.707758+0000","last_became_peered":"2024-03-06T07:31:08.707758+0000","last_unstale":"2024-03-06T07:31:11.315263+0000","last_undegraded":"2024-03-06T07:31:11.315263+0000","last_fullsized":"2024-03-06T07:31:11.315263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:25.868683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312246+0000","last_change":"2024-03-06T07:31:09.725327+0000","last_active":"2024-03-06T07:31:11.312246+0000","last_peered":"2024-03-06T07:31:11.312246+0000","last_clean":"2024-03-06T07:31:11.312246+0000","last_became_active":"2024-03-06T07:31:09.724605+0000","last_became_peered":"2024-03-06T07:31:09.724605+0000","last_unstale":"2024-03-06T07:31:11.312246+0000","last_undegraded":"2024-03-06T07:31:11.312246+0000","last_fullsized":"2024-03-06T07:31:11.312246+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:24.729631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700202+0000","last_change":"2024-03-06T07:31:08.708472+0000","last_active":"2024-03-06T07:31:10.700202+0000","last_peered":"2024-03-06T07:31:10.700202+0000","last_clean":"2024-03-06T07:31:10.700202+0000","last_became_active":"2024-03-06T07:31:08.708186+0000","last_became_peered":"2024-03-06T07:31:08.708186+0000","last_unstale":"2024-03-06T07:31:10.700202+0000","last_undegraded":"2024-03-06T07:31:10.700202+0000","last_fullsized":"2024-03-06T07:31:10.700202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:08.663518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700925+0000","last_change":"2024-03-06T07:31:09.731228+0000","last_active":"2024-03-06T07:31:10.700925+0000","last_peered":"2024-03-06T07:31:10.700925+0000","last_clean":"2024-03-06T07:31:10.700925+0000","last_became_active":"2024-03-06T07:31:09.730895+0000","last_became_peered":"2024-03-06T07:31:09.730895+0000","last_unstale":"2024-03-06T07:31:10.700925+0000","last_undegraded":"2024-03-06T07:31:10.700925+0000","last_fullsized":"2024-03-06T07:31:10.700925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:59.310776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314363+0000","last_change":"2024-03-06T07:31:08.710889+0000","last_active":"2024-03-06T07:31:11.314363+0000","last_peered":"2024-03-06T07:31:11.314363+0000","last_clean":"2024-03-06T07:31:11.314363+0000","last_became_active":"2024-03-06T07:31:08.709406+0000","last_became_peered":"2024-03-06T07:31:08.709406+0000","last_unstale":"2024-03-06T07:31:11.314363+0000","last_undegraded":"2024-03-06T07:31:11.314363+0000","last_fullsized":"2024-03-06T07:31:11.314363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:24.449426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211460+0000","last_change":"2024-03-06T07:31:09.711012+0000","last_active":"2024-03-06T07:31:11.211460+0000","last_peered":"2024-03-06T07:31:11.211460+0000","last_clean":"2024-03-06T07:31:11.211460+0000","last_became_active":"2024-03-06T07:31:09.710327+0000","last_became_peered":"2024-03-06T07:31:09.710327+0000","last_unstale":"2024-03-06T07:31:11.211460+0000","last_undegraded":"2024-03-06T07:31:11.211460+0000","last_fullsized":"2024-03-06T07:31:11.211460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.275443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700911+0000","last_change":"2024-03-06T07:31:08.704271+0000","last_active":"2024-03-06T07:31:10.700911+0000","last_peered":"2024-03-06T07:31:10.700911+0000","last_clean":"2024-03-06T07:31:10.700911+0000","last_became_active":"2024-03-06T07:31:08.700885+0000","last_became_peered":"2024-03-06T07:31:08.700885+0000","last_unstale":"2024-03-06T07:31:10.700911+0000","last_undegraded":"2024-03-06T07:31:10.700911+0000","last_fullsized":"2024-03-06T07:31:10.700911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:20.932727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211402+0000","last_change":"2024-03-06T07:31:09.744759+0000","last_active":"2024-03-06T07:31:11.211402+0000","last_peered":"2024-03-06T07:31:11.211402+0000","last_clean":"2024-03-06T07:31:11.211402+0000","last_became_active":"2024-03-06T07:31:09.744362+0000","last_became_peered":"2024-03-06T07:31:09.744362+0000","last_unstale":"2024-03-06T07:31:11.211402+0000","last_undegraded":"2024-03-06T07:31:11.211402+0000","last_fullsized":"2024-03-06T07:31:11.211402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:05.303513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737373+0000","last_change":"2024-03-06T07:31:08.698893+0000","last_active":"2024-03-06T07:31:10.737373+0000","last_peered":"2024-03-06T07:31:10.737373+0000","last_clean":"2024-03-06T07:31:10.737373+0000","last_became_active":"2024-03-06T07:31:08.698419+0000","last_became_peered":"2024-03-06T07:31:08.698419+0000","last_unstale":"2024-03-06T07:31:10.737373+0000","last_undegraded":"2024-03-06T07:31:10.737373+0000","last_fullsized":"2024-03-06T07:31:10.737373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:30.932957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211055+0000","last_change":"2024-03-06T07:31:09.733966+0000","last_active":"2024-03-06T07:31:11.211055+0000","last_peered":"2024-03-06T07:31:11.211055+0000","last_clean":"2024-03-06T07:31:11.211055+0000","last_became_active":"2024-03-06T07:31:09.732858+0000","last_became_peered":"2024-03-06T07:31:09.732858+0000","last_unstale":"2024-03-06T07:31:11.211055+0000","last_undegraded":"2024-03-06T07:31:11.211055+0000","last_fullsized":"2024-03-06T07:31:11.211055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:31.500909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698202+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:10.698202+0000","last_peered":"2024-03-06T07:31:10.698202+0000","last_clean":"2024-03-06T07:31:10.698202+0000","last_became_active":"2024-03-06T07:31:08.687300+0000","last_became_peered":"2024-03-06T07:31:08.687300+0000","last_unstale":"2024-03-06T07:31:10.698202+0000","last_undegraded":"2024-03-06T07:31:10.698202+0000","last_fullsized":"2024-03-06T07:31:10.698202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:05.110150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699548+0000","last_change":"2024-03-06T07:31:09.735514+0000","last_active":"2024-03-06T07:31:10.699548+0000","last_peered":"2024-03-06T07:31:10.699548+0000","last_clean":"2024-03-06T07:31:10.699548+0000","last_became_active":"2024-03-06T07:31:09.733652+0000","last_became_peered":"2024-03-06T07:31:09.733652+0000","last_unstale":"2024-03-06T07:31:10.699548+0000","last_undegraded":"2024-03-06T07:31:10.699548+0000","last_fullsized":"2024-03-06T07:31:10.699548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:53.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700894+0000","last_change":"2024-03-06T07:31:08.710412+0000","last_active":"2024-03-06T07:31:10.700894+0000","last_peered":"2024-03-06T07:31:10.700894+0000","last_clean":"2024-03-06T07:31:10.700894+0000","last_became_active":"2024-03-06T07:31:08.710061+0000","last_became_peered":"2024-03-06T07:31:08.710061+0000","last_unstale":"2024-03-06T07:31:10.700894+0000","last_undegraded":"2024-03-06T07:31:10.700894+0000","last_fullsized":"2024-03-06T07:31:10.700894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:01.053729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737817+0000","last_change":"2024-03-06T07:31:09.718215+0000","last_active":"2024-03-06T07:31:10.737817+0000","last_peered":"2024-03-06T07:31:10.737817+0000","last_clean":"2024-03-06T07:31:10.737817+0000","last_became_active":"2024-03-06T07:31:09.717858+0000","last_became_peered":"2024-03-06T07:31:09.717858+0000","last_unstale":"2024-03-06T07:31:10.737817+0000","last_undegraded":"2024-03-06T07:31:10.737817+0000","last_fullsized":"2024-03-06T07:31:10.737817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:57.600744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737782+0000","last_change":"2024-03-06T07:31:08.695443+0000","last_active":"2024-03-06T07:31:10.737782+0000","last_peered":"2024-03-06T07:31:10.737782+0000","last_clean":"2024-03-06T07:31:10.737782+0000","last_became_active":"2024-03-06T07:31:08.695112+0000","last_became_peered":"2024-03-06T07:31:08.695112+0000","last_unstale":"2024-03-06T07:31:10.737782+0000","last_undegraded":"2024-03-06T07:31:10.737782+0000","last_fullsized":"2024-03-06T07:31:10.737782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314122+0000","last_change":"2024-03-06T07:31:09.745311+0000","last_active":"2024-03-06T07:31:11.314122+0000","last_peered":"2024-03-06T07:31:11.314122+0000","last_clean":"2024-03-06T07:31:11.314122+0000","last_became_active":"2024-03-06T07:31:09.744759+0000","last_became_peered":"2024-03-06T07:31:09.744759+0000","last_unstale":"2024-03-06T07:31:11.314122+0000","last_undegraded":"2024-03-06T07:31:11.314122+0000","last_fullsized":"2024-03-06T07:31:11.314122+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:28.248865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701276+0000","last_change":"2024-03-06T07:31:08.691823+0000","last_active":"2024-03-06T07:31:10.701276+0000","last_peered":"2024-03-06T07:31:10.701276+0000","last_clean":"2024-03-06T07:31:10.701276+0000","last_became_active":"2024-03-06T07:31:08.691505+0000","last_became_peered":"2024-03-06T07:31:08.691505+0000","last_unstale":"2024-03-06T07:31:10.701276+0000","last_undegraded":"2024-03-06T07:31:10.701276+0000","last_fullsized":"2024-03-06T07:31:10.701276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:49.340554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314973+0000","last_change":"2024-03-06T07:31:09.729461+0000","last_active":"2024-03-06T07:31:11.314973+0000","last_peered":"2024-03-06T07:31:11.314973+0000","last_clean":"2024-03-06T07:31:11.314973+0000","last_became_active":"2024-03-06T07:31:09.729200+0000","last_became_peered":"2024-03-06T07:31:09.729200+0000","last_unstale":"2024-03-06T07:31:11.314973+0000","last_undegraded":"2024-03-06T07:31:11.314973+0000","last_fullsized":"2024-03-06T07:31:11.314973+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:43.967010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699926+0000","last_change":"2024-03-06T07:31:08.698585+0000","last_active":"2024-03-06T07:31:10.699926+0000","last_peered":"2024-03-06T07:31:10.699926+0000","last_clean":"2024-03-06T07:31:10.699926+0000","last_became_active":"2024-03-06T07:31:08.698166+0000","last_became_peered":"2024-03-06T07:31:08.698166+0000","last_unstale":"2024-03-06T07:31:10.699926+0000","last_undegraded":"2024-03-06T07:31:10.699926+0000","last_fullsized":"2024-03-06T07:31:10.699926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:30.932957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699716+0000","last_change":"2024-03-06T07:31:09.721386+0000","last_active":"2024-03-06T07:31:10.699716+0000","last_peered":"2024-03-06T07:31:10.699716+0000","last_clean":"2024-03-06T07:31:10.699716+0000","last_became_active":"2024-03-06T07:31:09.720479+0000","last_became_peered":"2024-03-06T07:31:09.720479+0000","last_unstale":"2024-03-06T07:31:10.699716+0000","last_undegraded":"2024-03-06T07:31:10.699716+0000","last_fullsized":"2024-03-06T07:31:10.699716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:41.553025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699681+0000","last_change":"2024-03-06T07:31:08.696740+0000","last_active":"2024-03-06T07:31:10.699681+0000","last_peered":"2024-03-06T07:31:10.699681+0000","last_clean":"2024-03-06T07:31:10.699681+0000","last_became_active":"2024-03-06T07:31:08.696450+0000","last_became_peered":"2024-03-06T07:31:08.696450+0000","last_unstale":"2024-03-06T07:31:10.699681+0000","last_undegraded":"2024-03-06T07:31:10.699681+0000","last_fullsized":"2024-03-06T07:31:10.699681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:46.905707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699612+0000","last_change":"2024-03-06T07:31:09.737762+0000","last_active":"2024-03-06T07:31:10.699612+0000","last_peered":"2024-03-06T07:31:10.699612+0000","last_clean":"2024-03-06T07:31:10.699612+0000","last_became_active":"2024-03-06T07:31:09.737289+0000","last_became_peered":"2024-03-06T07:31:09.737289+0000","last_unstale":"2024-03-06T07:31:10.699612+0000","last_undegraded":"2024-03-06T07:31:10.699612+0000","last_fullsized":"2024-03-06T07:31:10.699612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:13.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311646+0000","last_change":"2024-03-06T07:31:08.692727+0000","last_active":"2024-03-06T07:31:11.311646+0000","last_peered":"2024-03-06T07:31:11.311646+0000","last_clean":"2024-03-06T07:31:11.311646+0000","last_became_active":"2024-03-06T07:31:08.692457+0000","last_became_peered":"2024-03-06T07:31:08.692457+0000","last_unstale":"2024-03-06T07:31:11.311646+0000","last_undegraded":"2024-03-06T07:31:11.311646+0000","last_fullsized":"2024-03-06T07:31:11.311646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:46.370019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311961+0000","last_change":"2024-03-06T07:31:09.709061+0000","last_active":"2024-03-06T07:31:11.311961+0000","last_peered":"2024-03-06T07:31:11.311961+0000","last_clean":"2024-03-06T07:31:11.311961+0000","last_became_active":"2024-03-06T07:31:09.708573+0000","last_became_peered":"2024-03-06T07:31:09.708573+0000","last_unstale":"2024-03-06T07:31:11.311961+0000","last_undegraded":"2024-03-06T07:31:11.311961+0000","last_fullsized":"2024-03-06T07:31:11.311961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:40.415716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312822+0000","last_change":"2024-03-06T07:31:08.703245+0000","last_active":"2024-03-06T07:31:11.312822+0000","last_peered":"2024-03-06T07:31:11.312822+0000","last_clean":"2024-03-06T07:31:11.312822+0000","last_became_active":"2024-03-06T07:31:08.702878+0000","last_became_peered":"2024-03-06T07:31:08.702878+0000","last_unstale":"2024-03-06T07:31:11.312822+0000","last_undegraded":"2024-03-06T07:31:11.312822+0000","last_fullsized":"2024-03-06T07:31:11.312822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:24.790896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699273+0000","last_change":"2024-03-06T07:31:09.742322+0000","last_active":"2024-03-06T07:31:10.699273+0000","last_peered":"2024-03-06T07:31:10.699273+0000","last_clean":"2024-03-06T07:31:10.699273+0000","last_became_active":"2024-03-06T07:31:09.737823+0000","last_became_peered":"2024-03-06T07:31:09.737823+0000","last_unstale":"2024-03-06T07:31:10.699273+0000","last_undegraded":"2024-03-06T07:31:10.699273+0000","last_fullsized":"2024-03-06T07:31:10.699273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:12.115764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699855+0000","last_change":"2024-03-06T07:31:08.707266+0000","last_active":"2024-03-06T07:31:10.699855+0000","last_peered":"2024-03-06T07:31:10.699855+0000","last_clean":"2024-03-06T07:31:10.699855+0000","last_became_active":"2024-03-06T07:31:08.706774+0000","last_became_peered":"2024-03-06T07:31:08.706774+0000","last_unstale":"2024-03-06T07:31:10.699855+0000","last_undegraded":"2024-03-06T07:31:10.699855+0000","last_fullsized":"2024-03-06T07:31:10.699855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:22.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211286+0000","last_change":"2024-03-06T07:31:09.733311+0000","last_active":"2024-03-06T07:31:11.211286+0000","last_peered":"2024-03-06T07:31:11.211286+0000","last_clean":"2024-03-06T07:31:11.211286+0000","last_became_active":"2024-03-06T07:31:09.732908+0000","last_became_peered":"2024-03-06T07:31:09.732908+0000","last_unstale":"2024-03-06T07:31:11.211286+0000","last_undegraded":"2024-03-06T07:31:11.211286+0000","last_fullsized":"2024-03-06T07:31:11.211286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:50.935114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699895+0000","last_change":"2024-03-06T07:31:08.707358+0000","last_active":"2024-03-06T07:31:10.699895+0000","last_peered":"2024-03-06T07:31:10.699895+0000","last_clean":"2024-03-06T07:31:10.699895+0000","last_became_active":"2024-03-06T07:31:08.706392+0000","last_became_peered":"2024-03-06T07:31:08.706392+0000","last_unstale":"2024-03-06T07:31:10.699895+0000","last_undegraded":"2024-03-06T07:31:10.699895+0000","last_fullsized":"2024-03-06T07:31:10.699895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:58.302278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313628+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:31:11.313628+0000","last_peered":"2024-03-06T07:31:11.313628+0000","last_clean":"2024-03-06T07:31:11.313628+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:31:11.313628+0000","last_undegraded":"2024-03-06T07:31:11.313628+0000","last_fullsized":"2024-03-06T07:31:11.313628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542190,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1437696,"data_stored":940975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542191,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":843776,"data_stored":350083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":350081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":350061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":843776,"data_stored":350103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542192,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":351603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":352219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":940359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:15.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:31:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.422+0000 7f4588e00640 1 Processor -- start 2024-03-06T07:31:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.422+0000 7f4588e00640 1 -- start start 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.423+0000 7f4588e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4584156150 0x7f4584072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.423+0000 7f4588e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4584156530 0x7f45840ae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.423+0000 7f4588e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45840cd150 con 0x7f45840aec80 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.423+0000 7f4588e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4584071350 con 0x7f4584156150 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.423+0000 7f4588e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45840af0b0 con 0x7f4584156530 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4584156150 0x7f4584072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4584156150 0x7f4584072060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39770/0 (socket says 172.21.15.97:39770) 2024-03-06T07:31:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 -- 172.21.15.97:0/2933126762 learned_addr learned my addr 172.21.15.97:0/2933126762 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4582a00640 1 --2- 172.21.15.97:0/2933126762 >> v2:172.21.15.135:3300/0 conn(0x7f4584156530 0x7f45840ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 -- 172.21.15.97:0/2933126762 >> v2:172.21.15.135:3300/0 conn(0x7f4584156530 msgr2=0x7f45840ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 --2- 172.21.15.97:0/2933126762 >> v2:172.21.15.135:3300/0 conn(0x7f4584156530 0x7f45840ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 -- 172.21.15.97:0/2933126762 >> v1:172.21.15.135:6789/0 conn(0x7f45840aec80 legacy=0x7f458406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.424+0000 7f4583400640 1 -- 172.21.15.97:0/2933126762 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840cce70 con 0x7f4584156150 2024-03-06T07:31:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.425+0000 7f4583400640 1 --2- 172.21.15.97:0/2933126762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156150 0x7f4584072060 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f458406ee10 tx=0x7f457402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4cadabf70cfa30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.426+0000 7f4582000640 1 -- 172.21.15.97:0/2933126762 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574035020 con 0x7f4584156150 2024-03-06T07:31:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.426+0000 7f4582000640 1 -- 172.21.15.97:0/2933126762 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574031870 con 0x7f4584156150 2024-03-06T07:31:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.426+0000 7f4582000640 1 -- 172.21.15.97:0/2933126762 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574031c40 con 0x7f4584156150 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.426+0000 7f4588e00640 1 -- 172.21.15.97:0/2933126762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156150 msgr2=0x7f4584072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.426+0000 7f4588e00640 1 --2- 172.21.15.97:0/2933126762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156150 0x7f4584072060 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f458406ee10 tx=0x7f457402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.427+0000 7f4588e00640 1 -- 172.21.15.97:0/2933126762 shutdown_connections 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.427+0000 7f4588e00640 1 --2- 172.21.15.97:0/2933126762 >> v2:172.21.15.135:3300/0 conn(0x7f4584156530 0x7f45840ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.427+0000 7f4588e00640 1 --2- 172.21.15.97:0/2933126762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156150 0x7f4584072060 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.427+0000 7f4588e00640 1 -- 172.21.15.97:0/2933126762 >> 172.21.15.97:0/2933126762 conn(0x7f458406afc0 msgr2=0x7f458406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.427+0000 7f4588e00640 1 -- 172.21.15.97:0/2933126762 shutdown_connections 2024-03-06T07:31:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.427+0000 7f4588e00640 1 -- 172.21.15.97:0/2933126762 wait complete. 2024-03-06T07:31:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.428+0000 7f4588e00640 1 Processor -- start 2024-03-06T07:31:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.428+0000 7f4588e00640 1 -- start start 2024-03-06T07:31:15.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.429+0000 7f4588e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 0x7f45841550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.429+0000 7f4588e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4584156150 0x7f458415eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.429+0000 7f4588e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156530 0x7f45840afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.429+0000 7f4588e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584071b40 con 0x7f4584156530 2024-03-06T07:31:15.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.429+0000 7f4588e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584071610 con 0x7f45840aec80 2024-03-06T07:31:15.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.429+0000 7f4588e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840725a0 con 0x7f4584156150 2024-03-06T07:31:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.430+0000 7f4583400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 0x7f45841550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.430+0000 7f4583400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 0x7f45841550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:45326/0 (socket says 172.21.15.97:45326) 2024-03-06T07:31:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.430+0000 7f4583400640 1 -- 172.21.15.97:0/3807635781 learned_addr learned my addr 172.21.15.97:0/3807635781 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.431+0000 7f4583400640 1 -- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4584156150 msgr2=0x7f458415eae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:31:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.431+0000 7f4583400640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4584156150 0x7f458415eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.431+0000 7f4583400640 1 -- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156530 msgr2=0x7f45840afbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:31:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.432+0000 7f4583e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156530 0x7f45840afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.432+0000 7f4583400640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156530 0x7f45840afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.432+0000 7f4583400640 1 -- 172.21.15.97:0/3807635781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840b0a30 con 0x7f45840aec80 2024-03-06T07:31:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.432+0000 7f4583e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156530 0x7f45840afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.432+0000 7f4583400640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 0x7f45841550e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f45740097f0 tx=0x7f45740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.433+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574040070 con 0x7f45840aec80 2024-03-06T07:31:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.433+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574030d00 con 0x7f45840aec80 2024-03-06T07:31:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.433+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574035040 con 0x7f45840aec80 2024-03-06T07:31:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.433+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458415e1a0 con 0x7f45840aec80 2024-03-06T07:31:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.433+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4584157df0 con 0x7f45840aec80 2024-03-06T07:31:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.434+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458415e6c0 con 0x7f45840aec80 2024-03-06T07:31:15.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.436+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f457404b060 con 0x7f45840aec80 2024-03-06T07:31:15.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.437+0000 7f457b400640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f456406b520 0x7f456406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.437+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45740ac450 con 0x7f45840aec80 2024-03-06T07:31:15.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.437+0000 7f4582a00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f456406b520 0x7f456406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.438+0000 7f4582a00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f456406b520 0x7f456406d9e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4570006fd0 tx=0x7f4570008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.452+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f458415e6c0 con 0x7f45840aec80 2024-03-06T07:31:15.661 INFO:tasks.ceph.mds.a.smithi097.stderr:2024-03-06T07:31:15.660+0000 7f3896600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:15.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.763+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f458404f330 con 0x7f45840aec80 2024-03-06T07:31:15.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.769+0000 7f457b400640 1 -- 172.21.15.97:0/3807635781 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7f457407b2b0 con 0x7f45840aec80 2024-03-06T07:31:15.770 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:15.770 INFO:teuthology.orchestra.run.smithi097.stdout:{"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":233,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":69,"num_osds":8,"num_up_osds":8,"osd_up_since":1709710053,"num_in_osds":8,"osd_in_since":1709710045,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":263430144,"bytes_avail":772830683136,"bytes_total":773094113280,"write_bytes_sec":2277,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":43,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6318}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:29:24.932376+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f456406b520 msgr2=0x7f456406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f456406b520 0x7f456406d9e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4570006fd0 tx=0x7f4570008040 comp rx=0 tx=0).stop 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 msgr2=0x7f45841550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 0x7f45841550e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f45740097f0 tx=0x7f45740046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 shutdown_connections 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f456406b520 0x7f456406d9e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4584156530 0x7f45840afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4584156150 0x7f458415eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 --2- 172.21.15.97:0/3807635781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45840aec80 0x7f45841550e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.771+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 >> 172.21.15.97:0/3807635781 conn(0x7f458406afc0 msgr2=0x7f45840ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.772+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 shutdown_connections 2024-03-06T07:31:15.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.772+0000 7f4588e00640 1 -- 172.21.15.97:0/3807635781 wait complete. 2024-03-06T07:31:15.790 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:31:15.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.978+0000 7f6415200640 1 Processor -- start 2024-03-06T07:31:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.978+0000 7f6415200640 1 -- start start 2024-03-06T07:31:15.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6415200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f641015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6415200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6410070d20 0x7f641006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6415200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f641006d1a0 0x7f641006e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6415200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6410071d30 con 0x7f6410070d20 2024-03-06T07:31:15.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6415200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6410071490 con 0x7f641006d1a0 2024-03-06T07:31:15.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6415200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6410071770 con 0x7f64100af1c0 2024-03-06T07:31:15.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6407e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6410070d20 0x7f641006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6407e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6410070d20 0x7f641006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39786/0 (socket says 172.21.15.97:39786) 2024-03-06T07:31:15.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.980+0000 7f6407e00640 1 -- 172.21.15.97:0/2777869744 learned_addr learned my addr 172.21.15.97:0/2777869744 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f6407e00640 1 -- 172.21.15.97:0/2777869744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 msgr2=0x7f641015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f640de00640 1 --2- 172.21.15.97:0/2777869744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f641015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f640e800640 1 --2- 172.21.15.97:0/2777869744 >> v2:172.21.15.135:3300/0 conn(0x7f641006d1a0 0x7f641006e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f6407e00640 1 --2- 172.21.15.97:0/2777869744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f641015eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f6407e00640 1 -- 172.21.15.97:0/2777869744 >> v2:172.21.15.135:3300/0 conn(0x7f641006d1a0 msgr2=0x7f641006e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f6407e00640 1 --2- 172.21.15.97:0/2777869744 >> v2:172.21.15.135:3300/0 conn(0x7f641006d1a0 0x7f641006e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f6407e00640 1 -- 172.21.15.97:0/2777869744 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410071a50 con 0x7f6410070d20 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.981+0000 7f640e800640 1 --2- 172.21.15.97:0/2777869744 >> v2:172.21.15.135:3300/0 conn(0x7f641006d1a0 0x7f641006e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.982+0000 7f6407e00640 1 --2- 172.21.15.97:0/2777869744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070d20 0x7f641006cc60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f64100b2730 tx=0x7f63f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523ff49000a3bc97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.982+0000 7f640d400640 1 -- 172.21.15.97:0/2777869744 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8035020 con 0x7f6410070d20 2024-03-06T07:31:15.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.982+0000 7f640d400640 1 -- 172.21.15.97:0/2777869744 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8031870 con 0x7f6410070d20 2024-03-06T07:31:15.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.982+0000 7f640d400640 1 -- 172.21.15.97:0/2777869744 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8031c40 con 0x7f6410070d20 2024-03-06T07:31:15.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 -- 172.21.15.97:0/2777869744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070d20 msgr2=0x7f641006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 --2- 172.21.15.97:0/2777869744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070d20 0x7f641006cc60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f64100b2730 tx=0x7f63f802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 -- 172.21.15.97:0/2777869744 shutdown_connections 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 --2- 172.21.15.97:0/2777869744 >> v2:172.21.15.135:3300/0 conn(0x7f641006d1a0 0x7f641006e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 --2- 172.21.15.97:0/2777869744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6410070d20 0x7f641006cc60 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 --2- 172.21.15.97:0/2777869744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f641015eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 -- 172.21.15.97:0/2777869744 >> 172.21.15.97:0/2777869744 conn(0x7f64100b1730 msgr2=0x7f64100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 -- 172.21.15.97:0/2777869744 shutdown_connections 2024-03-06T07:31:15.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.983+0000 7f6415200640 1 -- 172.21.15.97:0/2777869744 wait complete. 2024-03-06T07:31:15.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.985+0000 7f6415200640 1 Processor -- start 2024-03-06T07:31:15.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.985+0000 7f6415200640 1 -- start start 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f6415200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 0x7f641014b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f6415200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6410070d20 0x7f64100cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f6415200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f64100cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f6415200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641006ea10 con 0x7f641006d1a0 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f6415200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100711a0 con 0x7f6410070d20 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f6415200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641006eb90 con 0x7f64100af1c0 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 0x7f641014b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 0x7f641014b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39802/0 (socket says 172.21.15.97:39802) 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 -- 172.21.15.97:0/1378077231 learned_addr learned my addr 172.21.15.97:0/1378077231 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 -- 172.21.15.97:0/1378077231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 msgr2=0x7f64100cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f64100cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 -- 172.21.15.97:0/1378077231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6410070d20 msgr2=0x7f64100cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6410070d20 0x7f64100cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.986+0000 7f640de00640 1 -- 172.21.15.97:0/1378077231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641014d320 con 0x7f641006d1a0 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.987+0000 7f640de00640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 0x7f641014b2f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f640000c7f0 tx=0x7f640000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.987+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400015070 con 0x7f641006d1a0 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.987+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400004220 con 0x7f641006d1a0 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.987+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400007620 con 0x7f641006d1a0 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.987+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64100cf230 con 0x7f641006d1a0 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.987+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641014d8b0 con 0x7f641006d1a0 2024-03-06T07:31:15.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.988+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641014e250 con 0x7f641006d1a0 2024-03-06T07:31:15.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.989+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64000079d0 con 0x7f641006d1a0 2024-03-06T07:31:15.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.990+0000 7f6406000640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.991+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f640008a480 con 0x7f641006d1a0 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.991+0000 7f6407e00640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.991+0000 7f6407e00640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f63f8036000 tx=0x7f63f8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:15.998+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f641014e250 con 0x7f641006d1a0 2024-03-06T07:31:16.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.181+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64100b4a10 con 0x7f63ec06b4d0 2024-03-06T07:31:16.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.199+0000 7f6406000640 1 -- 172.21.15.97:0/1378077231 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f64100b4a10 con 0x7f63ec06b4d0 2024-03-06T07:31:16.203 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:16.204 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:31:16.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 msgr2=0x7f63ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f63f8036000 tx=0x7f63f8030000 comp rx=0 tx=0).stop 2024-03-06T07:31:16.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 msgr2=0x7f641014b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 0x7f641014b2f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f640000c7f0 tx=0x7f640000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 shutdown_connections 2024-03-06T07:31:16.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63ec06b4d0 0x7f63ec06d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64100af1c0 0x7f64100cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6410070d20 0x7f64100cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 --2- 172.21.15.97:0/1378077231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f641006d1a0 0x7f641014b2f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 >> 172.21.15.97:0/1378077231 conn(0x7f64100b1730 msgr2=0x7f64100b3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 shutdown_connections 2024-03-06T07:31:16.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:16.204+0000 7f6415200640 1 -- 172.21.15.97:0/1378077231 wait complete. 2024-03-06T07:31:16.221 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-03-06T07:31:14.965743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257256,"kb_used_data":7752,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717464,"statfs":{"total":773094113280,"available":772830683136,"internally_reserved":0,"allocated":7938048,"data_stored":3985484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.295446"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314242+0000","last_change":"2024-03-06T07:31:09.738280+0000","last_active":"2024-03-06T07:31:11.314242+0000","last_peered":"2024-03-06T07:31:11.314242+0000","last_clean":"2024-03-06T07:31:11.314242+0000","last_became_active":"2024-03-06T07:31:09.737919+0000","last_became_peered":"2024-03-06T07:31:09.737919+0000","last_unstale":"2024-03-06T07:31:11.314242+0000","last_undegraded":"2024-03-06T07:31:11.314242+0000","last_fullsized":"2024-03-06T07:31:11.314242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:05.838129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701003+0000","last_change":"2024-03-06T07:31:09.731844+0000","last_active":"2024-03-06T07:31:10.701003+0000","last_peered":"2024-03-06T07:31:10.701003+0000","last_clean":"2024-03-06T07:31:10.701003+0000","last_became_active":"2024-03-06T07:31:09.731497+0000","last_became_peered":"2024-03-06T07:31:09.731497+0000","last_unstale":"2024-03-06T07:31:10.701003+0000","last_undegraded":"2024-03-06T07:31:10.701003+0000","last_fullsized":"2024-03-06T07:31:10.701003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:48.639188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.210992+0000","last_change":"2024-03-06T07:31:09.720966+0000","last_active":"2024-03-06T07:31:11.210992+0000","last_peered":"2024-03-06T07:31:11.210992+0000","last_clean":"2024-03-06T07:31:11.210992+0000","last_became_active":"2024-03-06T07:31:09.717482+0000","last_became_peered":"2024-03-06T07:31:09.717482+0000","last_unstale":"2024-03-06T07:31:11.210992+0000","last_undegraded":"2024-03-06T07:31:11.210992+0000","last_fullsized":"2024-03-06T07:31:11.210992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:08.762253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312031+0000","last_change":"2024-03-06T07:31:09.732801+0000","last_active":"2024-03-06T07:31:11.312031+0000","last_peered":"2024-03-06T07:31:11.312031+0000","last_clean":"2024-03-06T07:31:11.312031+0000","last_became_active":"2024-03-06T07:31:09.729989+0000","last_became_peered":"2024-03-06T07:31:09.729989+0000","last_unstale":"2024-03-06T07:31:11.312031+0000","last_undegraded":"2024-03-06T07:31:11.312031+0000","last_fullsized":"2024-03-06T07:31:11.312031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:37.643357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699389+0000","last_change":"2024-03-06T07:31:09.735427+0000","last_active":"2024-03-06T07:31:10.699389+0000","last_peered":"2024-03-06T07:31:10.699389+0000","last_clean":"2024-03-06T07:31:10.699389+0000","last_became_active":"2024-03-06T07:31:09.735240+0000","last_became_peered":"2024-03-06T07:31:09.735240+0000","last_unstale":"2024-03-06T07:31:10.699389+0000","last_undegraded":"2024-03-06T07:31:10.699389+0000","last_fullsized":"2024-03-06T07:31:10.699389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:25.357842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699572+0000","last_change":"2024-03-06T07:31:09.735149+0000","last_active":"2024-03-06T07:31:10.699572+0000","last_peered":"2024-03-06T07:31:10.699572+0000","last_clean":"2024-03-06T07:31:10.699572+0000","last_became_active":"2024-03-06T07:31:09.730477+0000","last_became_peered":"2024-03-06T07:31:09.730477+0000","last_unstale":"2024-03-06T07:31:10.699572+0000","last_undegraded":"2024-03-06T07:31:10.699572+0000","last_fullsized":"2024-03-06T07:31:10.699572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:16.910700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736731+0000","last_change":"2024-03-06T07:31:09.719251+0000","last_active":"2024-03-06T07:31:10.736731+0000","last_peered":"2024-03-06T07:31:10.736731+0000","last_clean":"2024-03-06T07:31:10.736731+0000","last_became_active":"2024-03-06T07:31:09.718763+0000","last_became_peered":"2024-03-06T07:31:09.718763+0000","last_unstale":"2024-03-06T07:31:10.736731+0000","last_undegraded":"2024-03-06T07:31:10.736731+0000","last_fullsized":"2024-03-06T07:31:10.736731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:53.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699419+0000","last_change":"2024-03-06T07:31:09.734091+0000","last_active":"2024-03-06T07:31:10.699419+0000","last_peered":"2024-03-06T07:31:10.699419+0000","last_clean":"2024-03-06T07:31:10.699419+0000","last_became_active":"2024-03-06T07:31:09.733787+0000","last_became_peered":"2024-03-06T07:31:09.733787+0000","last_unstale":"2024-03-06T07:31:10.699419+0000","last_undegraded":"2024-03-06T07:31:10.699419+0000","last_fullsized":"2024-03-06T07:31:10.699419+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:26.960412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700953+0000","last_change":"2024-03-06T07:31:09.728764+0000","last_active":"2024-03-06T07:31:10.700953+0000","last_peered":"2024-03-06T07:31:10.700953+0000","last_clean":"2024-03-06T07:31:10.700953+0000","last_became_active":"2024-03-06T07:31:09.728236+0000","last_became_peered":"2024-03-06T07:31:09.728236+0000","last_unstale":"2024-03-06T07:31:10.700953+0000","last_undegraded":"2024-03-06T07:31:10.700953+0000","last_fullsized":"2024-03-06T07:31:10.700953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:09.672015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314990+0000","last_change":"2024-03-06T07:31:09.733888+0000","last_active":"2024-03-06T07:31:11.314990+0000","last_peered":"2024-03-06T07:31:11.314990+0000","last_clean":"2024-03-06T07:31:11.314990+0000","last_became_active":"2024-03-06T07:31:09.733591+0000","last_became_peered":"2024-03-06T07:31:09.733591+0000","last_unstale":"2024-03-06T07:31:11.314990+0000","last_undegraded":"2024-03-06T07:31:11.314990+0000","last_fullsized":"2024-03-06T07:31:11.314990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:56.718632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314183+0000","last_change":"2024-03-06T07:31:09.728837+0000","last_active":"2024-03-06T07:31:11.314183+0000","last_peered":"2024-03-06T07:31:11.314183+0000","last_clean":"2024-03-06T07:31:11.314183+0000","last_became_active":"2024-03-06T07:31:09.724590+0000","last_became_peered":"2024-03-06T07:31:09.724590+0000","last_unstale":"2024-03-06T07:31:11.314183+0000","last_undegraded":"2024-03-06T07:31:11.314183+0000","last_fullsized":"2024-03-06T07:31:11.314183+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.779623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312138+0000","last_change":"2024-03-06T07:31:09.725547+0000","last_active":"2024-03-06T07:31:11.312138+0000","last_peered":"2024-03-06T07:31:11.312138+0000","last_clean":"2024-03-06T07:31:11.312138+0000","last_became_active":"2024-03-06T07:31:09.724975+0000","last_became_peered":"2024-03-06T07:31:09.724975+0000","last_unstale":"2024-03-06T07:31:11.312138+0000","last_undegraded":"2024-03-06T07:31:11.312138+0000","last_fullsized":"2024-03-06T07:31:11.312138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:33.626731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699223+0000","last_change":"2024-03-06T07:31:09.730705+0000","last_active":"2024-03-06T07:31:10.699223+0000","last_peered":"2024-03-06T07:31:10.699223+0000","last_clean":"2024-03-06T07:31:10.699223+0000","last_became_active":"2024-03-06T07:31:09.730523+0000","last_became_peered":"2024-03-06T07:31:09.730523+0000","last_unstale":"2024-03-06T07:31:10.699223+0000","last_undegraded":"2024-03-06T07:31:10.699223+0000","last_fullsized":"2024-03-06T07:31:10.699223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:23.028507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699528+0000","last_change":"2024-03-06T07:31:09.725710+0000","last_active":"2024-03-06T07:31:10.699528+0000","last_peered":"2024-03-06T07:31:10.699528+0000","last_clean":"2024-03-06T07:31:10.699528+0000","last_became_active":"2024-03-06T07:31:09.725335+0000","last_became_peered":"2024-03-06T07:31:09.725335+0000","last_unstale":"2024-03-06T07:31:10.699528+0000","last_undegraded":"2024-03-06T07:31:10.699528+0000","last_fullsized":"2024-03-06T07:31:10.699528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:21.145591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698581+0000","last_change":"2024-03-06T07:31:09.734193+0000","last_active":"2024-03-06T07:31:10.698581+0000","last_peered":"2024-03-06T07:31:10.698581+0000","last_clean":"2024-03-06T07:31:10.698581+0000","last_became_active":"2024-03-06T07:31:09.733891+0000","last_became_peered":"2024-03-06T07:31:09.733891+0000","last_unstale":"2024-03-06T07:31:10.698581+0000","last_undegraded":"2024-03-06T07:31:10.698581+0000","last_fullsized":"2024-03-06T07:31:10.698581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:57.434602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312912+0000","last_change":"2024-03-06T07:31:09.746880+0000","last_active":"2024-03-06T07:31:11.312912+0000","last_peered":"2024-03-06T07:31:11.312912+0000","last_clean":"2024-03-06T07:31:11.312912+0000","last_became_active":"2024-03-06T07:31:09.746461+0000","last_became_peered":"2024-03-06T07:31:09.746461+0000","last_unstale":"2024-03-06T07:31:11.312912+0000","last_undegraded":"2024-03-06T07:31:11.312912+0000","last_fullsized":"2024-03-06T07:31:11.312912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:41.617082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700999+0000","last_change":"2024-03-06T07:31:09.728614+0000","last_active":"2024-03-06T07:31:10.700999+0000","last_peered":"2024-03-06T07:31:10.700999+0000","last_clean":"2024-03-06T07:31:10.700999+0000","last_became_active":"2024-03-06T07:31:09.728201+0000","last_became_peered":"2024-03-06T07:31:09.728201+0000","last_unstale":"2024-03-06T07:31:10.700999+0000","last_undegraded":"2024-03-06T07:31:10.700999+0000","last_fullsized":"2024-03-06T07:31:10.700999+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:39.624880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312482+0000","last_change":"2024-03-06T07:31:09.739356+0000","last_active":"2024-03-06T07:31:11.312482+0000","last_peered":"2024-03-06T07:31:11.312482+0000","last_clean":"2024-03-06T07:31:11.312482+0000","last_became_active":"2024-03-06T07:31:09.738061+0000","last_became_peered":"2024-03-06T07:31:09.738061+0000","last_unstale":"2024-03-06T07:31:11.312482+0000","last_undegraded":"2024-03-06T07:31:11.312482+0000","last_fullsized":"2024-03-06T07:31:11.312482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:16.897453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699939+0000","last_change":"2024-03-06T07:31:09.734267+0000","last_active":"2024-03-06T07:31:10.699939+0000","last_peered":"2024-03-06T07:31:10.699939+0000","last_clean":"2024-03-06T07:31:10.699939+0000","last_became_active":"2024-03-06T07:31:09.733544+0000","last_became_peered":"2024-03-06T07:31:09.733544+0000","last_unstale":"2024-03-06T07:31:10.699939+0000","last_undegraded":"2024-03-06T07:31:10.699939+0000","last_fullsized":"2024-03-06T07:31:10.699939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:48.345451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700585+0000","last_change":"2024-03-06T07:31:09.723844+0000","last_active":"2024-03-06T07:31:10.700585+0000","last_peered":"2024-03-06T07:31:10.700585+0000","last_clean":"2024-03-06T07:31:10.700585+0000","last_became_active":"2024-03-06T07:31:09.723522+0000","last_became_peered":"2024-03-06T07:31:09.723522+0000","last_unstale":"2024-03-06T07:31:10.700585+0000","last_undegraded":"2024-03-06T07:31:10.700585+0000","last_fullsized":"2024-03-06T07:31:10.700585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:48.499605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313402+0000","last_change":"2024-03-06T07:31:09.721457+0000","last_active":"2024-03-06T07:31:11.313402+0000","last_peered":"2024-03-06T07:31:11.313402+0000","last_clean":"2024-03-06T07:31:11.313402+0000","last_became_active":"2024-03-06T07:31:09.721122+0000","last_became_peered":"2024-03-06T07:31:09.721122+0000","last_unstale":"2024-03-06T07:31:11.313402+0000","last_undegraded":"2024-03-06T07:31:11.313402+0000","last_fullsized":"2024-03-06T07:31:11.313402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:09.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313386+0000","last_change":"2024-03-06T07:31:09.721253+0000","last_active":"2024-03-06T07:31:11.313386+0000","last_peered":"2024-03-06T07:31:11.313386+0000","last_clean":"2024-03-06T07:31:11.313386+0000","last_became_active":"2024-03-06T07:31:09.720907+0000","last_became_peered":"2024-03-06T07:31:09.720907+0000","last_unstale":"2024-03-06T07:31:11.313386+0000","last_undegraded":"2024-03-06T07:31:11.313386+0000","last_fullsized":"2024-03-06T07:31:11.313386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:05.077124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314875+0000","last_change":"2024-03-06T07:31:09.728686+0000","last_active":"2024-03-06T07:31:11.314875+0000","last_peered":"2024-03-06T07:31:11.314875+0000","last_clean":"2024-03-06T07:31:11.314875+0000","last_became_active":"2024-03-06T07:31:09.728378+0000","last_became_peered":"2024-03-06T07:31:09.728378+0000","last_unstale":"2024-03-06T07:31:11.314875+0000","last_undegraded":"2024-03-06T07:31:11.314875+0000","last_fullsized":"2024-03-06T07:31:11.314875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:43.899640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313546+0000","last_change":"2024-03-06T07:31:09.745709+0000","last_active":"2024-03-06T07:31:11.313546+0000","last_peered":"2024-03-06T07:31:11.313546+0000","last_clean":"2024-03-06T07:31:11.313546+0000","last_became_active":"2024-03-06T07:31:09.744468+0000","last_became_peered":"2024-03-06T07:31:09.744468+0000","last_unstale":"2024-03-06T07:31:11.313546+0000","last_undegraded":"2024-03-06T07:31:11.313546+0000","last_fullsized":"2024-03-06T07:31:11.313546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:47.710080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315154+0000","last_change":"2024-03-06T07:31:09.722571+0000","last_active":"2024-03-06T07:31:11.315154+0000","last_peered":"2024-03-06T07:31:11.315154+0000","last_clean":"2024-03-06T07:31:11.315154+0000","last_became_active":"2024-03-06T07:31:09.722243+0000","last_became_peered":"2024-03-06T07:31:09.722243+0000","last_unstale":"2024-03-06T07:31:11.315154+0000","last_undegraded":"2024-03-06T07:31:11.315154+0000","last_fullsized":"2024-03-06T07:31:11.315154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:00.460321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737350+0000","last_change":"2024-03-06T07:31:09.743119+0000","last_active":"2024-03-06T07:31:10.737350+0000","last_peered":"2024-03-06T07:31:10.737350+0000","last_clean":"2024-03-06T07:31:10.737350+0000","last_became_active":"2024-03-06T07:31:09.742848+0000","last_became_peered":"2024-03-06T07:31:09.742848+0000","last_unstale":"2024-03-06T07:31:10.737350+0000","last_undegraded":"2024-03-06T07:31:10.737350+0000","last_fullsized":"2024-03-06T07:31:10.737350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:28.617476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211399+0000","last_change":"2024-03-06T07:31:09.720989+0000","last_active":"2024-03-06T07:31:11.211399+0000","last_peered":"2024-03-06T07:31:11.211399+0000","last_clean":"2024-03-06T07:31:11.211399+0000","last_became_active":"2024-03-06T07:31:09.720480+0000","last_became_peered":"2024-03-06T07:31:09.720480+0000","last_unstale":"2024-03-06T07:31:11.211399+0000","last_undegraded":"2024-03-06T07:31:11.211399+0000","last_fullsized":"2024-03-06T07:31:11.211399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:45.345281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314404+0000","last_change":"2024-03-06T07:31:09.727631+0000","last_active":"2024-03-06T07:31:11.314404+0000","last_peered":"2024-03-06T07:31:11.314404+0000","last_clean":"2024-03-06T07:31:11.314404+0000","last_became_active":"2024-03-06T07:31:09.727329+0000","last_became_peered":"2024-03-06T07:31:09.727329+0000","last_unstale":"2024-03-06T07:31:11.314404+0000","last_undegraded":"2024-03-06T07:31:11.314404+0000","last_fullsized":"2024-03-06T07:31:11.314404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.841904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211396+0000","last_change":"2024-03-06T07:31:09.719664+0000","last_active":"2024-03-06T07:31:11.211396+0000","last_peered":"2024-03-06T07:31:11.211396+0000","last_clean":"2024-03-06T07:31:11.211396+0000","last_became_active":"2024-03-06T07:31:09.719149+0000","last_became_peered":"2024-03-06T07:31:09.719149+0000","last_unstale":"2024-03-06T07:31:11.211396+0000","last_undegraded":"2024-03-06T07:31:11.211396+0000","last_fullsized":"2024-03-06T07:31:11.211396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:00.352698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736992+0000","last_change":"2024-03-06T07:31:09.742657+0000","last_active":"2024-03-06T07:31:10.736992+0000","last_peered":"2024-03-06T07:31:10.736992+0000","last_clean":"2024-03-06T07:31:10.736992+0000","last_became_active":"2024-03-06T07:31:09.742333+0000","last_became_peered":"2024-03-06T07:31:09.742333+0000","last_unstale":"2024-03-06T07:31:10.736992+0000","last_undegraded":"2024-03-06T07:31:10.736992+0000","last_fullsized":"2024-03-06T07:31:10.736992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:23.028507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737334+0000","last_change":"2024-03-06T07:31:09.731801+0000","last_active":"2024-03-06T07:31:10.737334+0000","last_peered":"2024-03-06T07:31:10.737334+0000","last_clean":"2024-03-06T07:31:10.737334+0000","last_became_active":"2024-03-06T07:31:09.731374+0000","last_became_peered":"2024-03-06T07:31:09.731374+0000","last_unstale":"2024-03-06T07:31:10.737334+0000","last_undegraded":"2024-03-06T07:31:10.737334+0000","last_fullsized":"2024-03-06T07:31:10.737334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:15.954452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699872+0000","last_change":"2024-03-06T07:31:09.731886+0000","last_active":"2024-03-06T07:31:10.699872+0000","last_peered":"2024-03-06T07:31:10.699872+0000","last_clean":"2024-03-06T07:31:10.699872+0000","last_became_active":"2024-03-06T07:31:09.731581+0000","last_became_peered":"2024-03-06T07:31:09.731581+0000","last_unstale":"2024-03-06T07:31:10.699872+0000","last_undegraded":"2024-03-06T07:31:10.699872+0000","last_fullsized":"2024-03-06T07:31:10.699872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:12.972421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315323+0000","last_change":"2024-03-06T07:31:09.737867+0000","last_active":"2024-03-06T07:31:11.315323+0000","last_peered":"2024-03-06T07:31:11.315323+0000","last_clean":"2024-03-06T07:31:11.315323+0000","last_became_active":"2024-03-06T07:31:09.737471+0000","last_became_peered":"2024-03-06T07:31:09.737471+0000","last_unstale":"2024-03-06T07:31:11.315323+0000","last_undegraded":"2024-03-06T07:31:11.315323+0000","last_fullsized":"2024-03-06T07:31:11.315323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:44.379179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315796+0000","last_change":"2024-03-06T07:31:08.709472+0000","last_active":"2024-03-06T07:31:11.315796+0000","last_peered":"2024-03-06T07:31:11.315796+0000","last_clean":"2024-03-06T07:31:11.315796+0000","last_became_active":"2024-03-06T07:31:08.709183+0000","last_became_peered":"2024-03-06T07:31:08.709183+0000","last_unstale":"2024-03-06T07:31:11.315796+0000","last_undegraded":"2024-03-06T07:31:11.315796+0000","last_fullsized":"2024-03-06T07:31:11.315796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:18.088186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701172+0000","last_change":"2024-03-06T07:31:09.728911+0000","last_active":"2024-03-06T07:31:10.701172+0000","last_peered":"2024-03-06T07:31:10.701172+0000","last_clean":"2024-03-06T07:31:10.701172+0000","last_became_active":"2024-03-06T07:31:09.728505+0000","last_became_peered":"2024-03-06T07:31:09.728505+0000","last_unstale":"2024-03-06T07:31:10.701172+0000","last_undegraded":"2024-03-06T07:31:10.701172+0000","last_fullsized":"2024-03-06T07:31:10.701172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:21.941224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699503+0000","last_change":"2024-03-06T07:31:08.690969+0000","last_active":"2024-03-06T07:31:10.699503+0000","last_peered":"2024-03-06T07:31:10.699503+0000","last_clean":"2024-03-06T07:31:10.699503+0000","last_became_active":"2024-03-06T07:31:08.690694+0000","last_became_peered":"2024-03-06T07:31:08.690694+0000","last_unstale":"2024-03-06T07:31:10.699503+0000","last_undegraded":"2024-03-06T07:31:10.699503+0000","last_fullsized":"2024-03-06T07:31:10.699503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699674+0000","last_change":"2024-03-06T07:31:09.733888+0000","last_active":"2024-03-06T07:31:10.699674+0000","last_peered":"2024-03-06T07:31:10.699674+0000","last_clean":"2024-03-06T07:31:10.699674+0000","last_became_active":"2024-03-06T07:31:09.733189+0000","last_became_peered":"2024-03-06T07:31:09.733189+0000","last_unstale":"2024-03-06T07:31:10.699674+0000","last_undegraded":"2024-03-06T07:31:10.699674+0000","last_fullsized":"2024-03-06T07:31:10.699674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:06.118647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314826+0000","last_change":"2024-03-06T07:31:08.711193+0000","last_active":"2024-03-06T07:31:11.314826+0000","last_peered":"2024-03-06T07:31:11.314826+0000","last_clean":"2024-03-06T07:31:11.314826+0000","last_became_active":"2024-03-06T07:31:08.710729+0000","last_became_peered":"2024-03-06T07:31:08.710729+0000","last_unstale":"2024-03-06T07:31:11.314826+0000","last_undegraded":"2024-03-06T07:31:11.314826+0000","last_fullsized":"2024-03-06T07:31:11.314826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.461751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314596+0000","last_change":"2024-03-06T07:31:09.720876+0000","last_active":"2024-03-06T07:31:11.314596+0000","last_peered":"2024-03-06T07:31:11.314596+0000","last_clean":"2024-03-06T07:31:11.314596+0000","last_became_active":"2024-03-06T07:31:09.720571+0000","last_became_peered":"2024-03-06T07:31:09.720571+0000","last_unstale":"2024-03-06T07:31:11.314596+0000","last_undegraded":"2024-03-06T07:31:11.314596+0000","last_fullsized":"2024-03-06T07:31:11.314596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:34.904471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699964+0000","last_change":"2024-03-06T07:31:08.703780+0000","last_active":"2024-03-06T07:31:10.699964+0000","last_peered":"2024-03-06T07:31:10.699964+0000","last_clean":"2024-03-06T07:31:10.699964+0000","last_became_active":"2024-03-06T07:31:08.703434+0000","last_became_peered":"2024-03-06T07:31:08.703434+0000","last_unstale":"2024-03-06T07:31:10.699964+0000","last_undegraded":"2024-03-06T07:31:10.699964+0000","last_fullsized":"2024-03-06T07:31:10.699964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:00.577536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737151+0000","last_change":"2024-03-06T07:31:09.719528+0000","last_active":"2024-03-06T07:31:10.737151+0000","last_peered":"2024-03-06T07:31:10.737151+0000","last_clean":"2024-03-06T07:31:10.737151+0000","last_became_active":"2024-03-06T07:31:09.719252+0000","last_became_peered":"2024-03-06T07:31:09.719252+0000","last_unstale":"2024-03-06T07:31:10.737151+0000","last_undegraded":"2024-03-06T07:31:10.737151+0000","last_fullsized":"2024-03-06T07:31:10.737151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:03.282222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211556+0000","last_change":"2024-03-06T07:31:08.695941+0000","last_active":"2024-03-06T07:31:11.211556+0000","last_peered":"2024-03-06T07:31:11.211556+0000","last_clean":"2024-03-06T07:31:11.211556+0000","last_became_active":"2024-03-06T07:31:08.695727+0000","last_became_peered":"2024-03-06T07:31:08.695727+0000","last_unstale":"2024-03-06T07:31:11.211556+0000","last_undegraded":"2024-03-06T07:31:11.211556+0000","last_fullsized":"2024-03-06T07:31:11.211556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.258836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313548+0000","last_change":"2024-03-06T07:31:09.722880+0000","last_active":"2024-03-06T07:31:11.313548+0000","last_peered":"2024-03-06T07:31:11.313548+0000","last_clean":"2024-03-06T07:31:11.313548+0000","last_became_active":"2024-03-06T07:31:09.722301+0000","last_became_peered":"2024-03-06T07:31:09.722301+0000","last_unstale":"2024-03-06T07:31:11.313548+0000","last_undegraded":"2024-03-06T07:31:11.313548+0000","last_fullsized":"2024-03-06T07:31:11.313548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:54.332390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700385+0000","last_change":"2024-03-06T07:31:08.701504+0000","last_active":"2024-03-06T07:31:10.700385+0000","last_peered":"2024-03-06T07:31:10.700385+0000","last_clean":"2024-03-06T07:31:10.700385+0000","last_became_active":"2024-03-06T07:31:08.701175+0000","last_became_peered":"2024-03-06T07:31:08.701175+0000","last_unstale":"2024-03-06T07:31:10.700385+0000","last_undegraded":"2024-03-06T07:31:10.700385+0000","last_fullsized":"2024-03-06T07:31:10.700385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:46.370019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698911+0000","last_change":"2024-03-06T07:31:09.726824+0000","last_active":"2024-03-06T07:31:10.698911+0000","last_peered":"2024-03-06T07:31:10.698911+0000","last_clean":"2024-03-06T07:31:10.698911+0000","last_became_active":"2024-03-06T07:31:09.726046+0000","last_became_peered":"2024-03-06T07:31:09.726046+0000","last_unstale":"2024-03-06T07:31:10.698911+0000","last_undegraded":"2024-03-06T07:31:10.698911+0000","last_fullsized":"2024-03-06T07:31:10.698911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:50.886912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311718+0000","last_change":"2024-03-06T07:31:08.690249+0000","last_active":"2024-03-06T07:31:11.311718+0000","last_peered":"2024-03-06T07:31:11.311718+0000","last_clean":"2024-03-06T07:31:11.311718+0000","last_became_active":"2024-03-06T07:31:08.689995+0000","last_became_peered":"2024-03-06T07:31:08.689995+0000","last_unstale":"2024-03-06T07:31:11.311718+0000","last_undegraded":"2024-03-06T07:31:11.311718+0000","last_fullsized":"2024-03-06T07:31:11.311718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:58.302278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698603+0000","last_change":"2024-03-06T07:31:09.724364+0000","last_active":"2024-03-06T07:31:10.698603+0000","last_peered":"2024-03-06T07:31:10.698603+0000","last_clean":"2024-03-06T07:31:10.698603+0000","last_became_active":"2024-03-06T07:31:09.724084+0000","last_became_peered":"2024-03-06T07:31:09.724084+0000","last_unstale":"2024-03-06T07:31:10.698603+0000","last_undegraded":"2024-03-06T07:31:10.698603+0000","last_fullsized":"2024-03-06T07:31:10.698603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.250553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698951+0000","last_change":"2024-03-06T07:31:08.694935+0000","last_active":"2024-03-06T07:31:10.698951+0000","last_peered":"2024-03-06T07:31:10.698951+0000","last_clean":"2024-03-06T07:31:10.698951+0000","last_became_active":"2024-03-06T07:31:08.694548+0000","last_became_peered":"2024-03-06T07:31:08.694548+0000","last_unstale":"2024-03-06T07:31:10.698951+0000","last_undegraded":"2024-03-06T07:31:10.698951+0000","last_fullsized":"2024-03-06T07:31:10.698951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:51.965506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737616+0000","last_change":"2024-03-06T07:31:09.731317+0000","last_active":"2024-03-06T07:31:10.737616+0000","last_peered":"2024-03-06T07:31:10.737616+0000","last_clean":"2024-03-06T07:31:10.737616+0000","last_became_active":"2024-03-06T07:31:09.730595+0000","last_became_peered":"2024-03-06T07:31:09.730595+0000","last_unstale":"2024-03-06T07:31:10.737616+0000","last_undegraded":"2024-03-06T07:31:10.737616+0000","last_fullsized":"2024-03-06T07:31:10.737616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:47.710080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315404+0000","last_change":"2024-03-06T07:31:08.691602+0000","last_active":"2024-03-06T07:31:11.315404+0000","last_peered":"2024-03-06T07:31:11.315404+0000","last_clean":"2024-03-06T07:31:11.315404+0000","last_became_active":"2024-03-06T07:31:08.690752+0000","last_became_peered":"2024-03-06T07:31:08.690752+0000","last_unstale":"2024-03-06T07:31:11.315404+0000","last_undegraded":"2024-03-06T07:31:11.315404+0000","last_fullsized":"2024-03-06T07:31:11.315404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:58.046062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315603+0000","last_change":"2024-03-06T07:31:09.745602+0000","last_active":"2024-03-06T07:31:11.315603+0000","last_peered":"2024-03-06T07:31:11.315603+0000","last_clean":"2024-03-06T07:31:11.315603+0000","last_became_active":"2024-03-06T07:31:09.742517+0000","last_became_peered":"2024-03-06T07:31:09.742517+0000","last_unstale":"2024-03-06T07:31:11.315603+0000","last_undegraded":"2024-03-06T07:31:11.315603+0000","last_fullsized":"2024-03-06T07:31:11.315603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:47.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701236+0000","last_change":"2024-03-06T07:31:08.710975+0000","last_active":"2024-03-06T07:31:10.701236+0000","last_peered":"2024-03-06T07:31:10.701236+0000","last_clean":"2024-03-06T07:31:10.701236+0000","last_became_active":"2024-03-06T07:31:08.710624+0000","last_became_peered":"2024-03-06T07:31:08.710624+0000","last_unstale":"2024-03-06T07:31:10.701236+0000","last_undegraded":"2024-03-06T07:31:10.701236+0000","last_fullsized":"2024-03-06T07:31:10.701236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:42.616395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699823+0000","last_change":"2024-03-06T07:31:09.736963+0000","last_active":"2024-03-06T07:31:10.699823+0000","last_peered":"2024-03-06T07:31:10.699823+0000","last_clean":"2024-03-06T07:31:10.699823+0000","last_became_active":"2024-03-06T07:31:09.736635+0000","last_became_peered":"2024-03-06T07:31:09.736635+0000","last_unstale":"2024-03-06T07:31:10.699823+0000","last_undegraded":"2024-03-06T07:31:10.699823+0000","last_fullsized":"2024-03-06T07:31:10.699823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:31.941454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211510+0000","last_change":"2024-03-06T07:31:08.701415+0000","last_active":"2024-03-06T07:31:11.211510+0000","last_peered":"2024-03-06T07:31:11.211510+0000","last_clean":"2024-03-06T07:31:11.211510+0000","last_became_active":"2024-03-06T07:31:08.700947+0000","last_became_peered":"2024-03-06T07:31:08.700947+0000","last_unstale":"2024-03-06T07:31:11.211510+0000","last_undegraded":"2024-03-06T07:31:11.211510+0000","last_fullsized":"2024-03-06T07:31:11.211510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:51.621273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700977+0000","last_change":"2024-03-06T07:31:09.736417+0000","last_active":"2024-03-06T07:31:10.700977+0000","last_peered":"2024-03-06T07:31:10.700977+0000","last_clean":"2024-03-06T07:31:10.700977+0000","last_became_active":"2024-03-06T07:31:09.736032+0000","last_became_peered":"2024-03-06T07:31:09.736032+0000","last_unstale":"2024-03-06T07:31:10.700977+0000","last_undegraded":"2024-03-06T07:31:10.700977+0000","last_fullsized":"2024-03-06T07:31:10.700977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:47.378517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312462+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:11.312462+0000","last_peered":"2024-03-06T07:31:11.312462+0000","last_clean":"2024-03-06T07:31:11.312462+0000","last_became_active":"2024-03-06T07:31:08.687214+0000","last_became_peered":"2024-03-06T07:31:08.687214+0000","last_unstale":"2024-03-06T07:31:11.312462+0000","last_undegraded":"2024-03-06T07:31:11.312462+0000","last_fullsized":"2024-03-06T07:31:11.312462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:20.932727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314098+0000","last_change":"2024-03-06T07:31:09.734608+0000","last_active":"2024-03-06T07:31:11.314098+0000","last_peered":"2024-03-06T07:31:11.314098+0000","last_clean":"2024-03-06T07:31:11.314098+0000","last_became_active":"2024-03-06T07:31:09.734395+0000","last_became_peered":"2024-03-06T07:31:09.734395+0000","last_unstale":"2024-03-06T07:31:11.314098+0000","last_undegraded":"2024-03-06T07:31:11.314098+0000","last_fullsized":"2024-03-06T07:31:11.314098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:57.600744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312003+0000","last_change":"2024-03-06T07:31:08.702804+0000","last_active":"2024-03-06T07:31:11.312003+0000","last_peered":"2024-03-06T07:31:11.312003+0000","last_clean":"2024-03-06T07:31:11.312003+0000","last_became_active":"2024-03-06T07:31:08.702494+0000","last_became_peered":"2024-03-06T07:31:08.702494+0000","last_unstale":"2024-03-06T07:31:11.312003+0000","last_undegraded":"2024-03-06T07:31:11.312003+0000","last_fullsized":"2024-03-06T07:31:11.312003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:38.616383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737294+0000","last_change":"2024-03-06T07:31:09.727382+0000","last_active":"2024-03-06T07:31:10.737294+0000","last_peered":"2024-03-06T07:31:10.737294+0000","last_clean":"2024-03-06T07:31:10.737294+0000","last_became_active":"2024-03-06T07:31:09.725803+0000","last_became_peered":"2024-03-06T07:31:09.725803+0000","last_unstale":"2024-03-06T07:31:10.737294+0000","last_undegraded":"2024-03-06T07:31:10.737294+0000","last_fullsized":"2024-03-06T07:31:10.737294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:48.499605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700674+0000","last_change":"2024-03-06T07:31:08.706279+0000","last_active":"2024-03-06T07:31:10.700674+0000","last_peered":"2024-03-06T07:31:10.700674+0000","last_clean":"2024-03-06T07:31:10.700674+0000","last_became_active":"2024-03-06T07:31:08.705857+0000","last_became_peered":"2024-03-06T07:31:08.705857+0000","last_unstale":"2024-03-06T07:31:10.700674+0000","last_undegraded":"2024-03-06T07:31:10.700674+0000","last_fullsized":"2024-03-06T07:31:10.700674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:38.616383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698468+0000","last_change":"2024-03-06T07:31:09.735918+0000","last_active":"2024-03-06T07:31:10.698468+0000","last_peered":"2024-03-06T07:31:10.698468+0000","last_clean":"2024-03-06T07:31:10.698468+0000","last_became_active":"2024-03-06T07:31:09.735302+0000","last_became_peered":"2024-03-06T07:31:09.735302+0000","last_unstale":"2024-03-06T07:31:10.698468+0000","last_undegraded":"2024-03-06T07:31:10.698468+0000","last_fullsized":"2024-03-06T07:31:10.698468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:09.747371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736808+0000","last_change":"2024-03-06T07:31:08.699062+0000","last_active":"2024-03-06T07:31:10.736808+0000","last_peered":"2024-03-06T07:31:10.736808+0000","last_clean":"2024-03-06T07:31:10.736808+0000","last_became_active":"2024-03-06T07:31:08.698767+0000","last_became_peered":"2024-03-06T07:31:08.698767+0000","last_unstale":"2024-03-06T07:31:10.736808+0000","last_undegraded":"2024-03-06T07:31:10.736808+0000","last_fullsized":"2024-03-06T07:31:10.736808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:05.110150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699664+0000","last_change":"2024-03-06T07:31:09.729881+0000","last_active":"2024-03-06T07:31:10.699664+0000","last_peered":"2024-03-06T07:31:10.699664+0000","last_clean":"2024-03-06T07:31:10.699664+0000","last_became_active":"2024-03-06T07:31:09.728303+0000","last_became_peered":"2024-03-06T07:31:09.728303+0000","last_unstale":"2024-03-06T07:31:10.699664+0000","last_undegraded":"2024-03-06T07:31:10.699664+0000","last_fullsized":"2024-03-06T07:31:10.699664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:39.411341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313005+0000","last_change":"2024-03-06T07:31:08.708615+0000","last_active":"2024-03-06T07:31:11.313005+0000","last_peered":"2024-03-06T07:31:11.313005+0000","last_clean":"2024-03-06T07:31:11.313005+0000","last_became_active":"2024-03-06T07:31:08.708149+0000","last_became_peered":"2024-03-06T07:31:08.708149+0000","last_unstale":"2024-03-06T07:31:11.313005+0000","last_undegraded":"2024-03-06T07:31:11.313005+0000","last_fullsized":"2024-03-06T07:31:11.313005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:19.844233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315110+0000","last_change":"2024-03-06T07:31:09.712575+0000","last_active":"2024-03-06T07:31:11.315110+0000","last_peered":"2024-03-06T07:31:11.315110+0000","last_clean":"2024-03-06T07:31:11.315110+0000","last_became_active":"2024-03-06T07:31:09.712036+0000","last_became_peered":"2024-03-06T07:31:09.712036+0000","last_unstale":"2024-03-06T07:31:11.315110+0000","last_undegraded":"2024-03-06T07:31:11.315110+0000","last_fullsized":"2024-03-06T07:31:11.315110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:05.374622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700610+0000","last_change":"2024-03-06T07:31:08.706291+0000","last_active":"2024-03-06T07:31:10.700610+0000","last_peered":"2024-03-06T07:31:10.700610+0000","last_clean":"2024-03-06T07:31:10.700610+0000","last_became_active":"2024-03-06T07:31:08.705537+0000","last_became_peered":"2024-03-06T07:31:08.705537+0000","last_unstale":"2024-03-06T07:31:10.700610+0000","last_undegraded":"2024-03-06T07:31:10.700610+0000","last_fullsized":"2024-03-06T07:31:10.700610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:58.716593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315787+0000","last_change":"2024-03-06T07:31:09.728588+0000","last_active":"2024-03-06T07:31:11.315787+0000","last_peered":"2024-03-06T07:31:11.315787+0000","last_clean":"2024-03-06T07:31:11.315787+0000","last_became_active":"2024-03-06T07:31:09.728244+0000","last_became_peered":"2024-03-06T07:31:09.728244+0000","last_unstale":"2024-03-06T07:31:11.315787+0000","last_undegraded":"2024-03-06T07:31:11.315787+0000","last_fullsized":"2024-03-06T07:31:11.315787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:02.267334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315263+0000","last_change":"2024-03-06T07:31:08.708950+0000","last_active":"2024-03-06T07:31:11.315263+0000","last_peered":"2024-03-06T07:31:11.315263+0000","last_clean":"2024-03-06T07:31:11.315263+0000","last_became_active":"2024-03-06T07:31:08.707758+0000","last_became_peered":"2024-03-06T07:31:08.707758+0000","last_unstale":"2024-03-06T07:31:11.315263+0000","last_undegraded":"2024-03-06T07:31:11.315263+0000","last_fullsized":"2024-03-06T07:31:11.315263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:25.868683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312246+0000","last_change":"2024-03-06T07:31:09.725327+0000","last_active":"2024-03-06T07:31:11.312246+0000","last_peered":"2024-03-06T07:31:11.312246+0000","last_clean":"2024-03-06T07:31:11.312246+0000","last_became_active":"2024-03-06T07:31:09.724605+0000","last_became_peered":"2024-03-06T07:31:09.724605+0000","last_unstale":"2024-03-06T07:31:11.312246+0000","last_undegraded":"2024-03-06T07:31:11.312246+0000","last_fullsized":"2024-03-06T07:31:11.312246+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:24.729631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700202+0000","last_change":"2024-03-06T07:31:08.708472+0000","last_active":"2024-03-06T07:31:10.700202+0000","last_peered":"2024-03-06T07:31:10.700202+0000","last_clean":"2024-03-06T07:31:10.700202+0000","last_became_active":"2024-03-06T07:31:08.708186+0000","last_became_peered":"2024-03-06T07:31:08.708186+0000","last_unstale":"2024-03-06T07:31:10.700202+0000","last_undegraded":"2024-03-06T07:31:10.700202+0000","last_fullsized":"2024-03-06T07:31:10.700202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:08.663518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700925+0000","last_change":"2024-03-06T07:31:09.731228+0000","last_active":"2024-03-06T07:31:10.700925+0000","last_peered":"2024-03-06T07:31:10.700925+0000","last_clean":"2024-03-06T07:31:10.700925+0000","last_became_active":"2024-03-06T07:31:09.730895+0000","last_became_peered":"2024-03-06T07:31:09.730895+0000","last_unstale":"2024-03-06T07:31:10.700925+0000","last_undegraded":"2024-03-06T07:31:10.700925+0000","last_fullsized":"2024-03-06T07:31:10.700925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:59.310776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314363+0000","last_change":"2024-03-06T07:31:08.710889+0000","last_active":"2024-03-06T07:31:11.314363+0000","last_peered":"2024-03-06T07:31:11.314363+0000","last_clean":"2024-03-06T07:31:11.314363+0000","last_became_active":"2024-03-06T07:31:08.709406+0000","last_became_peered":"2024-03-06T07:31:08.709406+0000","last_unstale":"2024-03-06T07:31:11.314363+0000","last_undegraded":"2024-03-06T07:31:11.314363+0000","last_fullsized":"2024-03-06T07:31:11.314363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:24.449426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211460+0000","last_change":"2024-03-06T07:31:09.711012+0000","last_active":"2024-03-06T07:31:11.211460+0000","last_peered":"2024-03-06T07:31:11.211460+0000","last_clean":"2024-03-06T07:31:11.211460+0000","last_became_active":"2024-03-06T07:31:09.710327+0000","last_became_peered":"2024-03-06T07:31:09.710327+0000","last_unstale":"2024-03-06T07:31:11.211460+0000","last_undegraded":"2024-03-06T07:31:11.211460+0000","last_fullsized":"2024-03-06T07:31:11.211460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.275443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700911+0000","last_change":"2024-03-06T07:31:08.704271+0000","last_active":"2024-03-06T07:31:10.700911+0000","last_peered":"2024-03-06T07:31:10.700911+0000","last_clean":"2024-03-06T07:31:10.700911+0000","last_became_active":"2024-03-06T07:31:08.700885+0000","last_became_peered":"2024-03-06T07:31:08.700885+0000","last_unstale":"2024-03-06T07:31:10.700911+0000","last_undegraded":"2024-03-06T07:31:10.700911+0000","last_fullsized":"2024-03-06T07:31:10.700911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:20.932727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211402+0000","last_change":"2024-03-06T07:31:09.744759+0000","last_active":"2024-03-06T07:31:11.211402+0000","last_peered":"2024-03-06T07:31:11.211402+0000","last_clean":"2024-03-06T07:31:11.211402+0000","last_became_active":"2024-03-06T07:31:09.744362+0000","last_became_peered":"2024-03-06T07:31:09.744362+0000","last_unstale":"2024-03-06T07:31:11.211402+0000","last_undegraded":"2024-03-06T07:31:11.211402+0000","last_fullsized":"2024-03-06T07:31:11.211402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:05.303513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737373+0000","last_change":"2024-03-06T07:31:08.698893+0000","last_active":"2024-03-06T07:31:10.737373+0000","last_peered":"2024-03-06T07:31:10.737373+0000","last_clean":"2024-03-06T07:31:10.737373+0000","last_became_active":"2024-03-06T07:31:08.698419+0000","last_became_peered":"2024-03-06T07:31:08.698419+0000","last_unstale":"2024-03-06T07:31:10.737373+0000","last_undegraded":"2024-03-06T07:31:10.737373+0000","last_fullsized":"2024-03-06T07:31:10.737373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:30.932957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211055+0000","last_change":"2024-03-06T07:31:09.733966+0000","last_active":"2024-03-06T07:31:11.211055+0000","last_peered":"2024-03-06T07:31:11.211055+0000","last_clean":"2024-03-06T07:31:11.211055+0000","last_became_active":"2024-03-06T07:31:09.732858+0000","last_became_peered":"2024-03-06T07:31:09.732858+0000","last_unstale":"2024-03-06T07:31:11.211055+0000","last_undegraded":"2024-03-06T07:31:11.211055+0000","last_fullsized":"2024-03-06T07:31:11.211055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:31.500909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698202+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:10.698202+0000","last_peered":"2024-03-06T07:31:10.698202+0000","last_clean":"2024-03-06T07:31:10.698202+0000","last_became_active":"2024-03-06T07:31:08.687300+0000","last_became_peered":"2024-03-06T07:31:08.687300+0000","last_unstale":"2024-03-06T07:31:10.698202+0000","last_undegraded":"2024-03-06T07:31:10.698202+0000","last_fullsized":"2024-03-06T07:31:10.698202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:05.110150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699548+0000","last_change":"2024-03-06T07:31:09.735514+0000","last_active":"2024-03-06T07:31:10.699548+0000","last_peered":"2024-03-06T07:31:10.699548+0000","last_clean":"2024-03-06T07:31:10.699548+0000","last_became_active":"2024-03-06T07:31:09.733652+0000","last_became_peered":"2024-03-06T07:31:09.733652+0000","last_unstale":"2024-03-06T07:31:10.699548+0000","last_undegraded":"2024-03-06T07:31:10.699548+0000","last_fullsized":"2024-03-06T07:31:10.699548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:53.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700894+0000","last_change":"2024-03-06T07:31:08.710412+0000","last_active":"2024-03-06T07:31:10.700894+0000","last_peered":"2024-03-06T07:31:10.700894+0000","last_clean":"2024-03-06T07:31:10.700894+0000","last_became_active":"2024-03-06T07:31:08.710061+0000","last_became_peered":"2024-03-06T07:31:08.710061+0000","last_unstale":"2024-03-06T07:31:10.700894+0000","last_undegraded":"2024-03-06T07:31:10.700894+0000","last_fullsized":"2024-03-06T07:31:10.700894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:01.053729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737817+0000","last_change":"2024-03-06T07:31:09.718215+0000","last_active":"2024-03-06T07:31:10.737817+0000","last_peered":"2024-03-06T07:31:10.737817+0000","last_clean":"2024-03-06T07:31:10.737817+0000","last_became_active":"2024-03-06T07:31:09.717858+0000","last_became_peered":"2024-03-06T07:31:09.717858+0000","last_unstale":"2024-03-06T07:31:10.737817+0000","last_undegraded":"2024-03-06T07:31:10.737817+0000","last_fullsized":"2024-03-06T07:31:10.737817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:57.600744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737782+0000","last_change":"2024-03-06T07:31:08.695443+0000","last_active":"2024-03-06T07:31:10.737782+0000","last_peered":"2024-03-06T07:31:10.737782+0000","last_clean":"2024-03-06T07:31:10.737782+0000","last_became_active":"2024-03-06T07:31:08.695112+0000","last_became_peered":"2024-03-06T07:31:08.695112+0000","last_unstale":"2024-03-06T07:31:10.737782+0000","last_undegraded":"2024-03-06T07:31:10.737782+0000","last_fullsized":"2024-03-06T07:31:10.737782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314122+0000","last_change":"2024-03-06T07:31:09.745311+0000","last_active":"2024-03-06T07:31:11.314122+0000","last_peered":"2024-03-06T07:31:11.314122+0000","last_clean":"2024-03-06T07:31:11.314122+0000","last_became_active":"2024-03-06T07:31:09.744759+0000","last_became_peered":"2024-03-06T07:31:09.744759+0000","last_unstale":"2024-03-06T07:31:11.314122+0000","last_undegraded":"2024-03-06T07:31:11.314122+0000","last_fullsized":"2024-03-06T07:31:11.314122+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:28.248865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701276+0000","last_change":"2024-03-06T07:31:08.691823+0000","last_active":"2024-03-06T07:31:10.701276+0000","last_peered":"2024-03-06T07:31:10.701276+0000","last_clean":"2024-03-06T07:31:10.701276+0000","last_became_active":"2024-03-06T07:31:08.691505+0000","last_became_peered":"2024-03-06T07:31:08.691505+0000","last_unstale":"2024-03-06T07:31:10.701276+0000","last_undegraded":"2024-03-06T07:31:10.701276+0000","last_fullsized":"2024-03-06T07:31:10.701276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:49.340554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314973+0000","last_change":"2024-03-06T07:31:09.729461+0000","last_active":"2024-03-06T07:31:11.314973+0000","last_peered":"2024-03-06T07:31:11.314973+0000","last_clean":"2024-03-06T07:31:11.314973+0000","last_became_active":"2024-03-06T07:31:09.729200+0000","last_became_peered":"2024-03-06T07:31:09.729200+0000","last_unstale":"2024-03-06T07:31:11.314973+0000","last_undegraded":"2024-03-06T07:31:11.314973+0000","last_fullsized":"2024-03-06T07:31:11.314973+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:43.967010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699926+0000","last_change":"2024-03-06T07:31:08.698585+0000","last_active":"2024-03-06T07:31:10.699926+0000","last_peered":"2024-03-06T07:31:10.699926+0000","last_clean":"2024-03-06T07:31:10.699926+0000","last_became_active":"2024-03-06T07:31:08.698166+0000","last_became_peered":"2024-03-06T07:31:08.698166+0000","last_unstale":"2024-03-06T07:31:10.699926+0000","last_undegraded":"2024-03-06T07:31:10.699926+0000","last_fullsized":"2024-03-06T07:31:10.699926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:30.932957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699716+0000","last_change":"2024-03-06T07:31:09.721386+0000","last_active":"2024-03-06T07:31:10.699716+0000","last_peered":"2024-03-06T07:31:10.699716+0000","last_clean":"2024-03-06T07:31:10.699716+0000","last_became_active":"2024-03-06T07:31:09.720479+0000","last_became_peered":"2024-03-06T07:31:09.720479+0000","last_unstale":"2024-03-06T07:31:10.699716+0000","last_undegraded":"2024-03-06T07:31:10.699716+0000","last_fullsized":"2024-03-06T07:31:10.699716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:41.553025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699681+0000","last_change":"2024-03-06T07:31:08.696740+0000","last_active":"2024-03-06T07:31:10.699681+0000","last_peered":"2024-03-06T07:31:10.699681+0000","last_clean":"2024-03-06T07:31:10.699681+0000","last_became_active":"2024-03-06T07:31:08.696450+0000","last_became_peered":"2024-03-06T07:31:08.696450+0000","last_unstale":"2024-03-06T07:31:10.699681+0000","last_undegraded":"2024-03-06T07:31:10.699681+0000","last_fullsized":"2024-03-06T07:31:10.699681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:46.905707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699612+0000","last_change":"2024-03-06T07:31:09.737762+0000","last_active":"2024-03-06T07:31:10.699612+0000","last_peered":"2024-03-06T07:31:10.699612+0000","last_clean":"2024-03-06T07:31:10.699612+0000","last_became_active":"2024-03-06T07:31:09.737289+0000","last_became_peered":"2024-03-06T07:31:09.737289+0000","last_unstale":"2024-03-06T07:31:10.699612+0000","last_undegraded":"2024-03-06T07:31:10.699612+0000","last_fullsized":"2024-03-06T07:31:10.699612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:13.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311646+0000","last_change":"2024-03-06T07:31:08.692727+0000","last_active":"2024-03-06T07:31:11.311646+0000","last_peered":"2024-03-06T07:31:11.311646+0000","last_clean":"2024-03-06T07:31:11.311646+0000","last_became_active":"2024-03-06T07:31:08.692457+0000","last_became_peered":"2024-03-06T07:31:08.692457+0000","last_unstale":"2024-03-06T07:31:11.311646+0000","last_undegraded":"2024-03-06T07:31:11.311646+0000","last_fullsized":"2024-03-06T07:31:11.311646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:46.370019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311961+0000","last_change":"2024-03-06T07:31:09.709061+0000","last_active":"2024-03-06T07:31:11.311961+0000","last_peered":"2024-03-06T07:31:11.311961+0000","last_clean":"2024-03-06T07:31:11.311961+0000","last_became_active":"2024-03-06T07:31:09.708573+0000","last_became_peered":"2024-03-06T07:31:09.708573+0000","last_unstale":"2024-03-06T07:31:11.311961+0000","last_undegraded":"2024-03-06T07:31:11.311961+0000","last_fullsized":"2024-03-06T07:31:11.311961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:40.415716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312822+0000","last_change":"2024-03-06T07:31:08.703245+0000","last_active":"2024-03-06T07:31:11.312822+0000","last_peered":"2024-03-06T07:31:11.312822+0000","last_clean":"2024-03-06T07:31:11.312822+0000","last_became_active":"2024-03-06T07:31:08.702878+0000","last_became_peered":"2024-03-06T07:31:08.702878+0000","last_unstale":"2024-03-06T07:31:11.312822+0000","last_undegraded":"2024-03-06T07:31:11.312822+0000","last_fullsized":"2024-03-06T07:31:11.312822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:24.790896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699273+0000","last_change":"2024-03-06T07:31:09.742322+0000","last_active":"2024-03-06T07:31:10.699273+0000","last_peered":"2024-03-06T07:31:10.699273+0000","last_clean":"2024-03-06T07:31:10.699273+0000","last_became_active":"2024-03-06T07:31:09.737823+0000","last_became_peered":"2024-03-06T07:31:09.737823+0000","last_unstale":"2024-03-06T07:31:10.699273+0000","last_undegraded":"2024-03-06T07:31:10.699273+0000","last_fullsized":"2024-03-06T07:31:10.699273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:12.115764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699855+0000","last_change":"2024-03-06T07:31:08.707266+0000","last_active":"2024-03-06T07:31:10.699855+0000","last_peered":"2024-03-06T07:31:10.699855+0000","last_clean":"2024-03-06T07:31:10.699855+0000","last_became_active":"2024-03-06T07:31:08.706774+0000","last_became_peered":"2024-03-06T07:31:08.706774+0000","last_unstale":"2024-03-06T07:31:10.699855+0000","last_undegraded":"2024-03-06T07:31:10.699855+0000","last_fullsized":"2024-03-06T07:31:10.699855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:22.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211286+0000","last_change":"2024-03-06T07:31:09.733311+0000","last_active":"2024-03-06T07:31:11.211286+0000","last_peered":"2024-03-06T07:31:11.211286+0000","last_clean":"2024-03-06T07:31:11.211286+0000","last_became_active":"2024-03-06T07:31:09.732908+0000","last_became_peered":"2024-03-06T07:31:09.732908+0000","last_unstale":"2024-03-06T07:31:11.211286+0000","last_undegraded":"2024-03-06T07:31:11.211286+0000","last_fullsized":"2024-03-06T07:31:11.211286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:50.935114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699895+0000","last_change":"2024-03-06T07:31:08.707358+0000","last_active":"2024-03-06T07:31:10.699895+0000","last_peered":"2024-03-06T07:31:10.699895+0000","last_clean":"2024-03-06T07:31:10.699895+0000","last_became_active":"2024-03-06T07:31:08.706392+0000","last_became_peered":"2024-03-06T07:31:08.706392+0000","last_unstale":"2024-03-06T07:31:10.699895+0000","last_undegraded":"2024-03-06T07:31:10.699895+0000","last_fullsized":"2024-03-06T07:31:10.699895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:58.302278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313628+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:31:11.313628+0000","last_peered":"2024-03-06T07:31:11.313628+0000","last_clean":"2024-03-06T07:31:11.313628+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:31:11.313628+0000","last_undegraded":"2024-03-06T07:31:11.313628+0000","last_fullsized":"2024-03-06T07:31:11.313628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542190,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1437696,"data_stored":940975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542191,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":843776,"data_stored":350083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":350081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":350061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":843776,"data_stored":350103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542192,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":351603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":352219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":940359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:19.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:19.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.434+0000 7f4c82200640 1 Processor -- start 2024-03-06T07:31:19.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.434+0000 7f4c82200640 1 -- start start 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.435+0000 7f4c82200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4c7c15acb0 0x7f4c7c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.435+0000 7f4c82200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7c15b090 con 0x7f4c7c06ec00 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c82200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7c0b2a30 con 0x7f4c7c0b4680 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c82200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7c0b4360 con 0x7f4c7c15acb0 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c7be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4c7c06ec00 0x7f4c7c0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48180/0 (socket says 172.21.15.97:48180) 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c7be00640 1 -- 172.21.15.97:0/670806721 learned_addr learned my addr 172.21.15.97:0/670806721 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c7b400640 1 --2- 172.21.15.97:0/670806721 >> v2:172.21.15.135:3300/0 conn(0x7f4c7c15acb0 0x7f4c7c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445320707 0 0) 0x7f4c7c15b090 con 0x7f4c7c06ec00 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.436+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c60003520 con 0x7f4c7c06ec00 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755098826 0 0) 0x7f4c60003520 con 0x7f4c7c06ec00 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 >> v2:172.21.15.135:3300/0 conn(0x7f4c7c15acb0 msgr2=0x7f4c7c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 --2- 172.21.15.97:0/670806721 >> v2:172.21.15.135:3300/0 conn(0x7f4c7c15acb0 0x7f4c7c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 >> v1:172.21.15.135:6789/0 conn(0x7f4c7c0b4680 legacy=0x7f4c7c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0b2cf0 con 0x7f4c7c06ec00 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4c740019f0 con 0x7f4c7c06ec00 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 >> v1:172.21.15.97:6789/0 conn(0x7f4c7c06ec00 legacy=0x7f4c7c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c600039b0 0x7f4c60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c60004800 0x7f4c60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c600051c0 0x7f4c6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0b4360 con 0x7f4c600039b0 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6000c060 con 0x7f4c60004800 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6000c260 con 0x7f4c600051c0 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.437+0000 7f4c7b400640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c600039b0 0x7f4c60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c72a00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c60004800 0x7f4c60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7be00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c600051c0 0x7f4c6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7b400640 1 -- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c600051c0 msgr2=0x7f4c6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7b400640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c600051c0 0x7f4c6000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7b400640 1 -- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c60004800 msgr2=0x7f4c60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7b400640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c60004800 0x7f4c60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7b400640 1 -- 172.21.15.97:0/670806721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60003520 con 0x7f4c600039b0 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c72a00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c60004800 0x7f4c60004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7be00640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c600051c0 0x7f4c6000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.438+0000 7f4c7be00640 1 -- 172.21.15.97:0/670806721 reap_dead start 2024-03-06T07:31:19.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.439+0000 7f4c7b400640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c600039b0 0x7f4c60003dd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009ac0 tx=0x7f4c60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849612189ea61820 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.439+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c039070 con 0x7f4c600039b0 2024-03-06T07:31:19.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.439+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c02c6b0 con 0x7f4c600039b0 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.439+0000 7f4c7aa00640 1 -- 172.21.15.97:0/670806721 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c009690 con 0x7f4c600039b0 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 -- 172.21.15.97:0/670806721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c600039b0 msgr2=0x7f4c60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c600039b0 0x7f4c60003dd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009ac0 tx=0x7f4c60005830 comp rx=0 tx=0).stop 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 -- 172.21.15.97:0/670806721 shutdown_connections 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 --2- 172.21.15.97:0/670806721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c600039b0 0x7f4c60003dd0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 -- 172.21.15.97:0/670806721 >> 172.21.15.97:0/670806721 conn(0x7f4c7c0b1730 msgr2=0x7f4c7c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 -- 172.21.15.97:0/670806721 shutdown_connections 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.440+0000 7f4c82200640 1 -- 172.21.15.97:0/670806721 wait complete. 2024-03-06T07:31:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.443+0000 7f4c82200640 1 Processor -- start 2024-03-06T07:31:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.443+0000 7f4c82200640 1 -- start start 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.444+0000 7f4c82200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c600039b0 0x7f4c7c141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.444+0000 7f4c82200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c7c0c5190 0x7f4c7c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.444+0000 7f4c82200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 0x7f4c7c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.444+0000 7f4c82200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0b4010 con 0x7f4c7c0c5610 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.444+0000 7f4c82200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0b32b0 con 0x7f4c600039b0 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.444+0000 7f4c82200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c072340 con 0x7f4c7c0c5190 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 0x7f4c7c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 0x7f4c7c0cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39818/0 (socket says 172.21.15.97:39818) 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 -- 172.21.15.97:0/4237681574 learned_addr learned my addr 172.21.15.97:0/4237681574 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7b400640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c600039b0 0x7f4c7c141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c72a00640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c7c0c5190 0x7f4c7c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 -- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c7c0c5190 msgr2=0x7f4c7c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c7c0c5190 0x7f4c7c0ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 -- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c600039b0 msgr2=0x7f4c7c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c600039b0 0x7f4c7c141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c7be00640 1 -- 172.21.15.97:0/4237681574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c14c700 con 0x7f4c7c0c5610 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.445+0000 7f4c72a00640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c7c0c5190 0x7f4c7c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.446+0000 7f4c7be00640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 0x7f4c7c0cd4b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c070aa0 tx=0x7f4c7400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.446+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74004d60 con 0x7f4c7c0c5610 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.446+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7400bb20 con 0x7f4c7c0c5610 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.446+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7c14c9e0 con 0x7f4c7c0c5610 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.446+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74007070 con 0x7f4c7c0c5610 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.446+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7c14cfa0 con 0x7f4c7c0c5610 2024-03-06T07:31:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.448+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7c04f330 con 0x7f4c7c0c5610 2024-03-06T07:31:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.448+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c7401b020 con 0x7f4c7c0c5610 2024-03-06T07:31:19.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.449+0000 7f4c73e00640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5806b500 0x7f4c5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.450+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c7408a310 con 0x7f4c7c0c5610 2024-03-06T07:31:19.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.450+0000 7f4c7b400640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5806b500 0x7f4c5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.450+0000 7f4c7b400640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5806b500 0x7f4c5806d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c036000 tx=0x7f4c6c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.454+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c740590f0 con 0x7f4c7c0c5610 2024-03-06T07:31:19.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.665+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c7c071230 con 0x7f4c5806b500 2024-03-06T07:31:19.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.682+0000 7f4c73e00640 1 -- 172.21.15.97:0/4237681574 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f4c7c071230 con 0x7f4c5806b500 2024-03-06T07:31:19.689 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:19.690 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:31:19.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.689+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5806b500 msgr2=0x7f4c5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.689+0000 7f4c82200640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5806b500 0x7f4c5806d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c036000 tx=0x7f4c6c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:19.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 msgr2=0x7f4c7c0cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 0x7f4c7c0cd4b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c070aa0 tx=0x7f4c7400b4a0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 shutdown_connections 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c5806b500 0x7f4c5806d9c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c7c0c5610 0x7f4c7c0cd4b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c7c0c5190 0x7f4c7c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 --2- 172.21.15.97:0/4237681574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c600039b0 0x7f4c7c141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 >> 172.21.15.97:0/4237681574 conn(0x7f4c7c0b1730 msgr2=0x7f4c7c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 shutdown_connections 2024-03-06T07:31:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.690+0000 7f4c82200640 1 -- 172.21.15.97:0/4237681574 wait complete. 2024-03-06T07:31:19.710 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-03-06T07:31:18.967205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257256,"kb_used_data":7752,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717464,"statfs":{"total":773094113280,"available":772830683136,"internally_reserved":0,"allocated":7938048,"data_stored":3985484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.275171"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314242+0000","last_change":"2024-03-06T07:31:09.738280+0000","last_active":"2024-03-06T07:31:11.314242+0000","last_peered":"2024-03-06T07:31:11.314242+0000","last_clean":"2024-03-06T07:31:11.314242+0000","last_became_active":"2024-03-06T07:31:09.737919+0000","last_became_peered":"2024-03-06T07:31:09.737919+0000","last_unstale":"2024-03-06T07:31:11.314242+0000","last_undegraded":"2024-03-06T07:31:11.314242+0000","last_fullsized":"2024-03-06T07:31:11.314242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:05.838129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701003+0000","last_change":"2024-03-06T07:31:09.731844+0000","last_active":"2024-03-06T07:31:10.701003+0000","last_peered":"2024-03-06T07:31:10.701003+0000","last_clean":"2024-03-06T07:31:10.701003+0000","last_became_active":"2024-03-06T07:31:09.731497+0000","last_became_peered":"2024-03-06T07:31:09.731497+0000","last_unstale":"2024-03-06T07:31:10.701003+0000","last_undegraded":"2024-03-06T07:31:10.701003+0000","last_fullsized":"2024-03-06T07:31:10.701003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:48.639188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.210992+0000","last_change":"2024-03-06T07:31:09.720966+0000","last_active":"2024-03-06T07:31:11.210992+0000","last_peered":"2024-03-06T07:31:11.210992+0000","last_clean":"2024-03-06T07:31:11.210992+0000","last_became_active":"2024-03-06T07:31:09.717482+0000","last_became_peered":"2024-03-06T07:31:09.717482+0000","last_unstale":"2024-03-06T07:31:11.210992+0000","last_undegraded":"2024-03-06T07:31:11.210992+0000","last_fullsized":"2024-03-06T07:31:11.210992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:08.762253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312031+0000","last_change":"2024-03-06T07:31:09.732801+0000","last_active":"2024-03-06T07:31:11.312031+0000","last_peered":"2024-03-06T07:31:11.312031+0000","last_clean":"2024-03-06T07:31:11.312031+0000","last_became_active":"2024-03-06T07:31:09.729989+0000","last_became_peered":"2024-03-06T07:31:09.729989+0000","last_unstale":"2024-03-06T07:31:11.312031+0000","last_undegraded":"2024-03-06T07:31:11.312031+0000","last_fullsized":"2024-03-06T07:31:11.312031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:37.643357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699389+0000","last_change":"2024-03-06T07:31:09.735427+0000","last_active":"2024-03-06T07:31:10.699389+0000","last_peered":"2024-03-06T07:31:10.699389+0000","last_clean":"2024-03-06T07:31:10.699389+0000","last_became_active":"2024-03-06T07:31:09.735240+0000","last_became_peered":"2024-03-06T07:31:09.735240+0000","last_unstale":"2024-03-06T07:31:10.699389+0000","last_undegraded":"2024-03-06T07:31:10.699389+0000","last_fullsized":"2024-03-06T07:31:10.699389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:25.357842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699572+0000","last_change":"2024-03-06T07:31:09.735149+0000","last_active":"2024-03-06T07:31:10.699572+0000","last_peered":"2024-03-06T07:31:10.699572+0000","last_clean":"2024-03-06T07:31:10.699572+0000","last_became_active":"2024-03-06T07:31:09.730477+0000","last_became_peered":"2024-03-06T07:31:09.730477+0000","last_unstale":"2024-03-06T07:31:10.699572+0000","last_undegraded":"2024-03-06T07:31:10.699572+0000","last_fullsized":"2024-03-06T07:31:10.699572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:16.910700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736731+0000","last_change":"2024-03-06T07:31:09.719251+0000","last_active":"2024-03-06T07:31:10.736731+0000","last_peered":"2024-03-06T07:31:10.736731+0000","last_clean":"2024-03-06T07:31:10.736731+0000","last_became_active":"2024-03-06T07:31:09.718763+0000","last_became_peered":"2024-03-06T07:31:09.718763+0000","last_unstale":"2024-03-06T07:31:10.736731+0000","last_undegraded":"2024-03-06T07:31:10.736731+0000","last_fullsized":"2024-03-06T07:31:10.736731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:53.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699419+0000","last_change":"2024-03-06T07:31:09.734091+0000","last_active":"2024-03-06T07:31:10.699419+0000","last_peered":"2024-03-06T07:31:10.699419+0000","last_clean":"2024-03-06T07:31:10.699419+0000","last_became_active":"2024-03-06T07:31:09.733787+0000","last_became_peered":"2024-03-06T07:31:09.733787+0000","last_unstale":"2024-03-06T07:31:10.699419+0000","last_undegraded":"2024-03-06T07:31:10.699419+0000","last_fullsized":"2024-03-06T07:31:10.699419+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:26.960412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700953+0000","last_change":"2024-03-06T07:31:09.728764+0000","last_active":"2024-03-06T07:31:10.700953+0000","last_peered":"2024-03-06T07:31:10.700953+0000","last_clean":"2024-03-06T07:31:10.700953+0000","last_became_active":"2024-03-06T07:31:09.728236+0000","last_became_peered":"2024-03-06T07:31:09.728236+0000","last_unstale":"2024-03-06T07:31:10.700953+0000","last_undegraded":"2024-03-06T07:31:10.700953+0000","last_fullsized":"2024-03-06T07:31:10.700953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:09.672015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314990+0000","last_change":"2024-03-06T07:31:09.733888+0000","last_active":"2024-03-06T07:31:11.314990+0000","last_peered":"2024-03-06T07:31:11.314990+0000","last_clean":"2024-03-06T07:31:11.314990+0000","last_became_active":"2024-03-06T07:31:09.733591+0000","last_became_peered":"2024-03-06T07:31:09.733591+0000","last_unstale":"2024-03-06T07:31:11.314990+0000","last_undegraded":"2024-03-06T07:31:11.314990+0000","last_fullsized":"2024-03-06T07:31:11.314990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:56.718632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314183+0000","last_change":"2024-03-06T07:31:09.728837+0000","last_active":"2024-03-06T07:31:11.314183+0000","last_peered":"2024-03-06T07:31:11.314183+0000","last_clean":"2024-03-06T07:31:11.314183+0000","last_became_active":"2024-03-06T07:31:09.724590+0000","last_became_peered":"2024-03-06T07:31:09.724590+0000","last_unstale":"2024-03-06T07:31:11.314183+0000","last_undegraded":"2024-03-06T07:31:11.314183+0000","last_fullsized":"2024-03-06T07:31:11.314183+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.779623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312138+0000","last_change":"2024-03-06T07:31:09.725547+0000","last_active":"2024-03-06T07:31:11.312138+0000","last_peered":"2024-03-06T07:31:11.312138+0000","last_clean":"2024-03-06T07:31:11.312138+0000","last_became_active":"2024-03-06T07:31:09.724975+0000","last_became_peered":"2024-03-06T07:31:09.724975+0000","last_unstale":"2024-03-06T07:31:11.312138+0000","last_undegraded":"2024-03-06T07:31:11.312138+0000","last_fullsized":"2024-03-06T07:31:11.312138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:33.626731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699223+0000","last_change":"2024-03-06T07:31:09.730705+0000","last_active":"2024-03-06T07:31:10.699223+0000","last_peered":"2024-03-06T07:31:10.699223+0000","last_clean":"2024-03-06T07:31:10.699223+0000","last_became_active":"2024-03-06T07:31:09.730523+0000","last_became_peered":"2024-03-06T07:31:09.730523+0000","last_unstale":"2024-03-06T07:31:10.699223+0000","last_undegraded":"2024-03-06T07:31:10.699223+0000","last_fullsized":"2024-03-06T07:31:10.699223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:23.028507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699528+0000","last_change":"2024-03-06T07:31:09.725710+0000","last_active":"2024-03-06T07:31:10.699528+0000","last_peered":"2024-03-06T07:31:10.699528+0000","last_clean":"2024-03-06T07:31:10.699528+0000","last_became_active":"2024-03-06T07:31:09.725335+0000","last_became_peered":"2024-03-06T07:31:09.725335+0000","last_unstale":"2024-03-06T07:31:10.699528+0000","last_undegraded":"2024-03-06T07:31:10.699528+0000","last_fullsized":"2024-03-06T07:31:10.699528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:21.145591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698581+0000","last_change":"2024-03-06T07:31:09.734193+0000","last_active":"2024-03-06T07:31:10.698581+0000","last_peered":"2024-03-06T07:31:10.698581+0000","last_clean":"2024-03-06T07:31:10.698581+0000","last_became_active":"2024-03-06T07:31:09.733891+0000","last_became_peered":"2024-03-06T07:31:09.733891+0000","last_unstale":"2024-03-06T07:31:10.698581+0000","last_undegraded":"2024-03-06T07:31:10.698581+0000","last_fullsized":"2024-03-06T07:31:10.698581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:57.434602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312912+0000","last_change":"2024-03-06T07:31:09.746880+0000","last_active":"2024-03-06T07:31:11.312912+0000","last_peered":"2024-03-06T07:31:11.312912+0000","last_clean":"2024-03-06T07:31:11.312912+0000","last_became_active":"2024-03-06T07:31:09.746461+0000","last_became_peered":"2024-03-06T07:31:09.746461+0000","last_unstale":"2024-03-06T07:31:11.312912+0000","last_undegraded":"2024-03-06T07:31:11.312912+0000","last_fullsized":"2024-03-06T07:31:11.312912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:41.617082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700999+0000","last_change":"2024-03-06T07:31:09.728614+0000","last_active":"2024-03-06T07:31:10.700999+0000","last_peered":"2024-03-06T07:31:10.700999+0000","last_clean":"2024-03-06T07:31:10.700999+0000","last_became_active":"2024-03-06T07:31:09.728201+0000","last_became_peered":"2024-03-06T07:31:09.728201+0000","last_unstale":"2024-03-06T07:31:10.700999+0000","last_undegraded":"2024-03-06T07:31:10.700999+0000","last_fullsized":"2024-03-06T07:31:10.700999+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:39.624880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312482+0000","last_change":"2024-03-06T07:31:09.739356+0000","last_active":"2024-03-06T07:31:11.312482+0000","last_peered":"2024-03-06T07:31:11.312482+0000","last_clean":"2024-03-06T07:31:11.312482+0000","last_became_active":"2024-03-06T07:31:09.738061+0000","last_became_peered":"2024-03-06T07:31:09.738061+0000","last_unstale":"2024-03-06T07:31:11.312482+0000","last_undegraded":"2024-03-06T07:31:11.312482+0000","last_fullsized":"2024-03-06T07:31:11.312482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:16.897453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699939+0000","last_change":"2024-03-06T07:31:09.734267+0000","last_active":"2024-03-06T07:31:10.699939+0000","last_peered":"2024-03-06T07:31:10.699939+0000","last_clean":"2024-03-06T07:31:10.699939+0000","last_became_active":"2024-03-06T07:31:09.733544+0000","last_became_peered":"2024-03-06T07:31:09.733544+0000","last_unstale":"2024-03-06T07:31:10.699939+0000","last_undegraded":"2024-03-06T07:31:10.699939+0000","last_fullsized":"2024-03-06T07:31:10.699939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:48.345451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700585+0000","last_change":"2024-03-06T07:31:09.723844+0000","last_active":"2024-03-06T07:31:10.700585+0000","last_peered":"2024-03-06T07:31:10.700585+0000","last_clean":"2024-03-06T07:31:10.700585+0000","last_became_active":"2024-03-06T07:31:09.723522+0000","last_became_peered":"2024-03-06T07:31:09.723522+0000","last_unstale":"2024-03-06T07:31:10.700585+0000","last_undegraded":"2024-03-06T07:31:10.700585+0000","last_fullsized":"2024-03-06T07:31:10.700585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:48.499605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313402+0000","last_change":"2024-03-06T07:31:09.721457+0000","last_active":"2024-03-06T07:31:11.313402+0000","last_peered":"2024-03-06T07:31:11.313402+0000","last_clean":"2024-03-06T07:31:11.313402+0000","last_became_active":"2024-03-06T07:31:09.721122+0000","last_became_peered":"2024-03-06T07:31:09.721122+0000","last_unstale":"2024-03-06T07:31:11.313402+0000","last_undegraded":"2024-03-06T07:31:11.313402+0000","last_fullsized":"2024-03-06T07:31:11.313402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:09.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313386+0000","last_change":"2024-03-06T07:31:09.721253+0000","last_active":"2024-03-06T07:31:11.313386+0000","last_peered":"2024-03-06T07:31:11.313386+0000","last_clean":"2024-03-06T07:31:11.313386+0000","last_became_active":"2024-03-06T07:31:09.720907+0000","last_became_peered":"2024-03-06T07:31:09.720907+0000","last_unstale":"2024-03-06T07:31:11.313386+0000","last_undegraded":"2024-03-06T07:31:11.313386+0000","last_fullsized":"2024-03-06T07:31:11.313386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:05.077124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314875+0000","last_change":"2024-03-06T07:31:09.728686+0000","last_active":"2024-03-06T07:31:11.314875+0000","last_peered":"2024-03-06T07:31:11.314875+0000","last_clean":"2024-03-06T07:31:11.314875+0000","last_became_active":"2024-03-06T07:31:09.728378+0000","last_became_peered":"2024-03-06T07:31:09.728378+0000","last_unstale":"2024-03-06T07:31:11.314875+0000","last_undegraded":"2024-03-06T07:31:11.314875+0000","last_fullsized":"2024-03-06T07:31:11.314875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:43.899640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313546+0000","last_change":"2024-03-06T07:31:09.745709+0000","last_active":"2024-03-06T07:31:11.313546+0000","last_peered":"2024-03-06T07:31:11.313546+0000","last_clean":"2024-03-06T07:31:11.313546+0000","last_became_active":"2024-03-06T07:31:09.744468+0000","last_became_peered":"2024-03-06T07:31:09.744468+0000","last_unstale":"2024-03-06T07:31:11.313546+0000","last_undegraded":"2024-03-06T07:31:11.313546+0000","last_fullsized":"2024-03-06T07:31:11.313546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:47.710080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315154+0000","last_change":"2024-03-06T07:31:09.722571+0000","last_active":"2024-03-06T07:31:11.315154+0000","last_peered":"2024-03-06T07:31:11.315154+0000","last_clean":"2024-03-06T07:31:11.315154+0000","last_became_active":"2024-03-06T07:31:09.722243+0000","last_became_peered":"2024-03-06T07:31:09.722243+0000","last_unstale":"2024-03-06T07:31:11.315154+0000","last_undegraded":"2024-03-06T07:31:11.315154+0000","last_fullsized":"2024-03-06T07:31:11.315154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:00.460321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737350+0000","last_change":"2024-03-06T07:31:09.743119+0000","last_active":"2024-03-06T07:31:10.737350+0000","last_peered":"2024-03-06T07:31:10.737350+0000","last_clean":"2024-03-06T07:31:10.737350+0000","last_became_active":"2024-03-06T07:31:09.742848+0000","last_became_peered":"2024-03-06T07:31:09.742848+0000","last_unstale":"2024-03-06T07:31:10.737350+0000","last_undegraded":"2024-03-06T07:31:10.737350+0000","last_fullsized":"2024-03-06T07:31:10.737350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:28.617476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211399+0000","last_change":"2024-03-06T07:31:09.720989+0000","last_active":"2024-03-06T07:31:11.211399+0000","last_peered":"2024-03-06T07:31:11.211399+0000","last_clean":"2024-03-06T07:31:11.211399+0000","last_became_active":"2024-03-06T07:31:09.720480+0000","last_became_peered":"2024-03-06T07:31:09.720480+0000","last_unstale":"2024-03-06T07:31:11.211399+0000","last_undegraded":"2024-03-06T07:31:11.211399+0000","last_fullsized":"2024-03-06T07:31:11.211399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:45.345281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314404+0000","last_change":"2024-03-06T07:31:09.727631+0000","last_active":"2024-03-06T07:31:11.314404+0000","last_peered":"2024-03-06T07:31:11.314404+0000","last_clean":"2024-03-06T07:31:11.314404+0000","last_became_active":"2024-03-06T07:31:09.727329+0000","last_became_peered":"2024-03-06T07:31:09.727329+0000","last_unstale":"2024-03-06T07:31:11.314404+0000","last_undegraded":"2024-03-06T07:31:11.314404+0000","last_fullsized":"2024-03-06T07:31:11.314404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.841904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211396+0000","last_change":"2024-03-06T07:31:09.719664+0000","last_active":"2024-03-06T07:31:11.211396+0000","last_peered":"2024-03-06T07:31:11.211396+0000","last_clean":"2024-03-06T07:31:11.211396+0000","last_became_active":"2024-03-06T07:31:09.719149+0000","last_became_peered":"2024-03-06T07:31:09.719149+0000","last_unstale":"2024-03-06T07:31:11.211396+0000","last_undegraded":"2024-03-06T07:31:11.211396+0000","last_fullsized":"2024-03-06T07:31:11.211396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:00.352698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736992+0000","last_change":"2024-03-06T07:31:09.742657+0000","last_active":"2024-03-06T07:31:10.736992+0000","last_peered":"2024-03-06T07:31:10.736992+0000","last_clean":"2024-03-06T07:31:10.736992+0000","last_became_active":"2024-03-06T07:31:09.742333+0000","last_became_peered":"2024-03-06T07:31:09.742333+0000","last_unstale":"2024-03-06T07:31:10.736992+0000","last_undegraded":"2024-03-06T07:31:10.736992+0000","last_fullsized":"2024-03-06T07:31:10.736992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:23.028507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737334+0000","last_change":"2024-03-06T07:31:09.731801+0000","last_active":"2024-03-06T07:31:10.737334+0000","last_peered":"2024-03-06T07:31:10.737334+0000","last_clean":"2024-03-06T07:31:10.737334+0000","last_became_active":"2024-03-06T07:31:09.731374+0000","last_became_peered":"2024-03-06T07:31:09.731374+0000","last_unstale":"2024-03-06T07:31:10.737334+0000","last_undegraded":"2024-03-06T07:31:10.737334+0000","last_fullsized":"2024-03-06T07:31:10.737334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:15.954452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699872+0000","last_change":"2024-03-06T07:31:09.731886+0000","last_active":"2024-03-06T07:31:10.699872+0000","last_peered":"2024-03-06T07:31:10.699872+0000","last_clean":"2024-03-06T07:31:10.699872+0000","last_became_active":"2024-03-06T07:31:09.731581+0000","last_became_peered":"2024-03-06T07:31:09.731581+0000","last_unstale":"2024-03-06T07:31:10.699872+0000","last_undegraded":"2024-03-06T07:31:10.699872+0000","last_fullsized":"2024-03-06T07:31:10.699872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:12.972421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315323+0000","last_change":"2024-03-06T07:31:09.737867+0000","last_active":"2024-03-06T07:31:11.315323+0000","last_peered":"2024-03-06T07:31:11.315323+0000","last_clean":"2024-03-06T07:31:11.315323+0000","last_became_active":"2024-03-06T07:31:09.737471+0000","last_became_peered":"2024-03-06T07:31:09.737471+0000","last_unstale":"2024-03-06T07:31:11.315323+0000","last_undegraded":"2024-03-06T07:31:11.315323+0000","last_fullsized":"2024-03-06T07:31:11.315323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:44.379179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315796+0000","last_change":"2024-03-06T07:31:08.709472+0000","last_active":"2024-03-06T07:31:11.315796+0000","last_peered":"2024-03-06T07:31:11.315796+0000","last_clean":"2024-03-06T07:31:11.315796+0000","last_became_active":"2024-03-06T07:31:08.709183+0000","last_became_peered":"2024-03-06T07:31:08.709183+0000","last_unstale":"2024-03-06T07:31:11.315796+0000","last_undegraded":"2024-03-06T07:31:11.315796+0000","last_fullsized":"2024-03-06T07:31:11.315796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:18.088186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701172+0000","last_change":"2024-03-06T07:31:09.728911+0000","last_active":"2024-03-06T07:31:10.701172+0000","last_peered":"2024-03-06T07:31:10.701172+0000","last_clean":"2024-03-06T07:31:10.701172+0000","last_became_active":"2024-03-06T07:31:09.728505+0000","last_became_peered":"2024-03-06T07:31:09.728505+0000","last_unstale":"2024-03-06T07:31:10.701172+0000","last_undegraded":"2024-03-06T07:31:10.701172+0000","last_fullsized":"2024-03-06T07:31:10.701172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:21.941224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699503+0000","last_change":"2024-03-06T07:31:08.690969+0000","last_active":"2024-03-06T07:31:10.699503+0000","last_peered":"2024-03-06T07:31:10.699503+0000","last_clean":"2024-03-06T07:31:10.699503+0000","last_became_active":"2024-03-06T07:31:08.690694+0000","last_became_peered":"2024-03-06T07:31:08.690694+0000","last_unstale":"2024-03-06T07:31:10.699503+0000","last_undegraded":"2024-03-06T07:31:10.699503+0000","last_fullsized":"2024-03-06T07:31:10.699503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699674+0000","last_change":"2024-03-06T07:31:09.733888+0000","last_active":"2024-03-06T07:31:10.699674+0000","last_peered":"2024-03-06T07:31:10.699674+0000","last_clean":"2024-03-06T07:31:10.699674+0000","last_became_active":"2024-03-06T07:31:09.733189+0000","last_became_peered":"2024-03-06T07:31:09.733189+0000","last_unstale":"2024-03-06T07:31:10.699674+0000","last_undegraded":"2024-03-06T07:31:10.699674+0000","last_fullsized":"2024-03-06T07:31:10.699674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:06.118647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314826+0000","last_change":"2024-03-06T07:31:08.711193+0000","last_active":"2024-03-06T07:31:11.314826+0000","last_peered":"2024-03-06T07:31:11.314826+0000","last_clean":"2024-03-06T07:31:11.314826+0000","last_became_active":"2024-03-06T07:31:08.710729+0000","last_became_peered":"2024-03-06T07:31:08.710729+0000","last_unstale":"2024-03-06T07:31:11.314826+0000","last_undegraded":"2024-03-06T07:31:11.314826+0000","last_fullsized":"2024-03-06T07:31:11.314826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.461751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314596+0000","last_change":"2024-03-06T07:31:09.720876+0000","last_active":"2024-03-06T07:31:11.314596+0000","last_peered":"2024-03-06T07:31:11.314596+0000","last_clean":"2024-03-06T07:31:11.314596+0000","last_became_active":"2024-03-06T07:31:09.720571+0000","last_became_peered":"2024-03-06T07:31:09.720571+0000","last_unstale":"2024-03-06T07:31:11.314596+0000","last_undegraded":"2024-03-06T07:31:11.314596+0000","last_fullsized":"2024-03-06T07:31:11.314596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:34.904471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699964+0000","last_change":"2024-03-06T07:31:08.703780+0000","last_active":"2024-03-06T07:31:10.699964+0000","last_peered":"2024-03-06T07:31:10.699964+0000","last_clean":"2024-03-06T07:31:10.699964+0000","last_became_active":"2024-03-06T07:31:08.703434+0000","last_became_peered":"2024-03-06T07:31:08.703434+0000","last_unstale":"2024-03-06T07:31:10.699964+0000","last_undegraded":"2024-03-06T07:31:10.699964+0000","last_fullsized":"2024-03-06T07:31:10.699964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:00.577536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737151+0000","last_change":"2024-03-06T07:31:09.719528+0000","last_active":"2024-03-06T07:31:10.737151+0000","last_peered":"2024-03-06T07:31:10.737151+0000","last_clean":"2024-03-06T07:31:10.737151+0000","last_became_active":"2024-03-06T07:31:09.719252+0000","last_became_peered":"2024-03-06T07:31:09.719252+0000","last_unstale":"2024-03-06T07:31:10.737151+0000","last_undegraded":"2024-03-06T07:31:10.737151+0000","last_fullsized":"2024-03-06T07:31:10.737151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:03.282222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211556+0000","last_change":"2024-03-06T07:31:08.695941+0000","last_active":"2024-03-06T07:31:11.211556+0000","last_peered":"2024-03-06T07:31:11.211556+0000","last_clean":"2024-03-06T07:31:11.211556+0000","last_became_active":"2024-03-06T07:31:08.695727+0000","last_became_peered":"2024-03-06T07:31:08.695727+0000","last_unstale":"2024-03-06T07:31:11.211556+0000","last_undegraded":"2024-03-06T07:31:11.211556+0000","last_fullsized":"2024-03-06T07:31:11.211556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.258836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313548+0000","last_change":"2024-03-06T07:31:09.722880+0000","last_active":"2024-03-06T07:31:11.313548+0000","last_peered":"2024-03-06T07:31:11.313548+0000","last_clean":"2024-03-06T07:31:11.313548+0000","last_became_active":"2024-03-06T07:31:09.722301+0000","last_became_peered":"2024-03-06T07:31:09.722301+0000","last_unstale":"2024-03-06T07:31:11.313548+0000","last_undegraded":"2024-03-06T07:31:11.313548+0000","last_fullsized":"2024-03-06T07:31:11.313548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:54.332390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700385+0000","last_change":"2024-03-06T07:31:08.701504+0000","last_active":"2024-03-06T07:31:10.700385+0000","last_peered":"2024-03-06T07:31:10.700385+0000","last_clean":"2024-03-06T07:31:10.700385+0000","last_became_active":"2024-03-06T07:31:08.701175+0000","last_became_peered":"2024-03-06T07:31:08.701175+0000","last_unstale":"2024-03-06T07:31:10.700385+0000","last_undegraded":"2024-03-06T07:31:10.700385+0000","last_fullsized":"2024-03-06T07:31:10.700385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:46.370019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698911+0000","last_change":"2024-03-06T07:31:09.726824+0000","last_active":"2024-03-06T07:31:10.698911+0000","last_peered":"2024-03-06T07:31:10.698911+0000","last_clean":"2024-03-06T07:31:10.698911+0000","last_became_active":"2024-03-06T07:31:09.726046+0000","last_became_peered":"2024-03-06T07:31:09.726046+0000","last_unstale":"2024-03-06T07:31:10.698911+0000","last_undegraded":"2024-03-06T07:31:10.698911+0000","last_fullsized":"2024-03-06T07:31:10.698911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:50.886912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311718+0000","last_change":"2024-03-06T07:31:08.690249+0000","last_active":"2024-03-06T07:31:11.311718+0000","last_peered":"2024-03-06T07:31:11.311718+0000","last_clean":"2024-03-06T07:31:11.311718+0000","last_became_active":"2024-03-06T07:31:08.689995+0000","last_became_peered":"2024-03-06T07:31:08.689995+0000","last_unstale":"2024-03-06T07:31:11.311718+0000","last_undegraded":"2024-03-06T07:31:11.311718+0000","last_fullsized":"2024-03-06T07:31:11.311718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:58.302278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698603+0000","last_change":"2024-03-06T07:31:09.724364+0000","last_active":"2024-03-06T07:31:10.698603+0000","last_peered":"2024-03-06T07:31:10.698603+0000","last_clean":"2024-03-06T07:31:10.698603+0000","last_became_active":"2024-03-06T07:31:09.724084+0000","last_became_peered":"2024-03-06T07:31:09.724084+0000","last_unstale":"2024-03-06T07:31:10.698603+0000","last_undegraded":"2024-03-06T07:31:10.698603+0000","last_fullsized":"2024-03-06T07:31:10.698603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.250553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698951+0000","last_change":"2024-03-06T07:31:08.694935+0000","last_active":"2024-03-06T07:31:10.698951+0000","last_peered":"2024-03-06T07:31:10.698951+0000","last_clean":"2024-03-06T07:31:10.698951+0000","last_became_active":"2024-03-06T07:31:08.694548+0000","last_became_peered":"2024-03-06T07:31:08.694548+0000","last_unstale":"2024-03-06T07:31:10.698951+0000","last_undegraded":"2024-03-06T07:31:10.698951+0000","last_fullsized":"2024-03-06T07:31:10.698951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:51.965506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737616+0000","last_change":"2024-03-06T07:31:09.731317+0000","last_active":"2024-03-06T07:31:10.737616+0000","last_peered":"2024-03-06T07:31:10.737616+0000","last_clean":"2024-03-06T07:31:10.737616+0000","last_became_active":"2024-03-06T07:31:09.730595+0000","last_became_peered":"2024-03-06T07:31:09.730595+0000","last_unstale":"2024-03-06T07:31:10.737616+0000","last_undegraded":"2024-03-06T07:31:10.737616+0000","last_fullsized":"2024-03-06T07:31:10.737616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:47.710080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315404+0000","last_change":"2024-03-06T07:31:08.691602+0000","last_active":"2024-03-06T07:31:11.315404+0000","last_peered":"2024-03-06T07:31:11.315404+0000","last_clean":"2024-03-06T07:31:11.315404+0000","last_became_active":"2024-03-06T07:31:08.690752+0000","last_became_peered":"2024-03-06T07:31:08.690752+0000","last_unstale":"2024-03-06T07:31:11.315404+0000","last_undegraded":"2024-03-06T07:31:11.315404+0000","last_fullsized":"2024-03-06T07:31:11.315404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:58.046062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315603+0000","last_change":"2024-03-06T07:31:09.745602+0000","last_active":"2024-03-06T07:31:11.315603+0000","last_peered":"2024-03-06T07:31:11.315603+0000","last_clean":"2024-03-06T07:31:11.315603+0000","last_became_active":"2024-03-06T07:31:09.742517+0000","last_became_peered":"2024-03-06T07:31:09.742517+0000","last_unstale":"2024-03-06T07:31:11.315603+0000","last_undegraded":"2024-03-06T07:31:11.315603+0000","last_fullsized":"2024-03-06T07:31:11.315603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:47.355316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701236+0000","last_change":"2024-03-06T07:31:08.710975+0000","last_active":"2024-03-06T07:31:10.701236+0000","last_peered":"2024-03-06T07:31:10.701236+0000","last_clean":"2024-03-06T07:31:10.701236+0000","last_became_active":"2024-03-06T07:31:08.710624+0000","last_became_peered":"2024-03-06T07:31:08.710624+0000","last_unstale":"2024-03-06T07:31:10.701236+0000","last_undegraded":"2024-03-06T07:31:10.701236+0000","last_fullsized":"2024-03-06T07:31:10.701236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:42.616395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699823+0000","last_change":"2024-03-06T07:31:09.736963+0000","last_active":"2024-03-06T07:31:10.699823+0000","last_peered":"2024-03-06T07:31:10.699823+0000","last_clean":"2024-03-06T07:31:10.699823+0000","last_became_active":"2024-03-06T07:31:09.736635+0000","last_became_peered":"2024-03-06T07:31:09.736635+0000","last_unstale":"2024-03-06T07:31:10.699823+0000","last_undegraded":"2024-03-06T07:31:10.699823+0000","last_fullsized":"2024-03-06T07:31:10.699823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:31.941454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211510+0000","last_change":"2024-03-06T07:31:08.701415+0000","last_active":"2024-03-06T07:31:11.211510+0000","last_peered":"2024-03-06T07:31:11.211510+0000","last_clean":"2024-03-06T07:31:11.211510+0000","last_became_active":"2024-03-06T07:31:08.700947+0000","last_became_peered":"2024-03-06T07:31:08.700947+0000","last_unstale":"2024-03-06T07:31:11.211510+0000","last_undegraded":"2024-03-06T07:31:11.211510+0000","last_fullsized":"2024-03-06T07:31:11.211510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:51.621273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700977+0000","last_change":"2024-03-06T07:31:09.736417+0000","last_active":"2024-03-06T07:31:10.700977+0000","last_peered":"2024-03-06T07:31:10.700977+0000","last_clean":"2024-03-06T07:31:10.700977+0000","last_became_active":"2024-03-06T07:31:09.736032+0000","last_became_peered":"2024-03-06T07:31:09.736032+0000","last_unstale":"2024-03-06T07:31:10.700977+0000","last_undegraded":"2024-03-06T07:31:10.700977+0000","last_fullsized":"2024-03-06T07:31:10.700977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:47.378517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312462+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:11.312462+0000","last_peered":"2024-03-06T07:31:11.312462+0000","last_clean":"2024-03-06T07:31:11.312462+0000","last_became_active":"2024-03-06T07:31:08.687214+0000","last_became_peered":"2024-03-06T07:31:08.687214+0000","last_unstale":"2024-03-06T07:31:11.312462+0000","last_undegraded":"2024-03-06T07:31:11.312462+0000","last_fullsized":"2024-03-06T07:31:11.312462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:20.932727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314098+0000","last_change":"2024-03-06T07:31:09.734608+0000","last_active":"2024-03-06T07:31:11.314098+0000","last_peered":"2024-03-06T07:31:11.314098+0000","last_clean":"2024-03-06T07:31:11.314098+0000","last_became_active":"2024-03-06T07:31:09.734395+0000","last_became_peered":"2024-03-06T07:31:09.734395+0000","last_unstale":"2024-03-06T07:31:11.314098+0000","last_undegraded":"2024-03-06T07:31:11.314098+0000","last_fullsized":"2024-03-06T07:31:11.314098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:57.600744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312003+0000","last_change":"2024-03-06T07:31:08.702804+0000","last_active":"2024-03-06T07:31:11.312003+0000","last_peered":"2024-03-06T07:31:11.312003+0000","last_clean":"2024-03-06T07:31:11.312003+0000","last_became_active":"2024-03-06T07:31:08.702494+0000","last_became_peered":"2024-03-06T07:31:08.702494+0000","last_unstale":"2024-03-06T07:31:11.312003+0000","last_undegraded":"2024-03-06T07:31:11.312003+0000","last_fullsized":"2024-03-06T07:31:11.312003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:38.616383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737294+0000","last_change":"2024-03-06T07:31:09.727382+0000","last_active":"2024-03-06T07:31:10.737294+0000","last_peered":"2024-03-06T07:31:10.737294+0000","last_clean":"2024-03-06T07:31:10.737294+0000","last_became_active":"2024-03-06T07:31:09.725803+0000","last_became_peered":"2024-03-06T07:31:09.725803+0000","last_unstale":"2024-03-06T07:31:10.737294+0000","last_undegraded":"2024-03-06T07:31:10.737294+0000","last_fullsized":"2024-03-06T07:31:10.737294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:48.499605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700674+0000","last_change":"2024-03-06T07:31:08.706279+0000","last_active":"2024-03-06T07:31:10.700674+0000","last_peered":"2024-03-06T07:31:10.700674+0000","last_clean":"2024-03-06T07:31:10.700674+0000","last_became_active":"2024-03-06T07:31:08.705857+0000","last_became_peered":"2024-03-06T07:31:08.705857+0000","last_unstale":"2024-03-06T07:31:10.700674+0000","last_undegraded":"2024-03-06T07:31:10.700674+0000","last_fullsized":"2024-03-06T07:31:10.700674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:38.616383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698468+0000","last_change":"2024-03-06T07:31:09.735918+0000","last_active":"2024-03-06T07:31:10.698468+0000","last_peered":"2024-03-06T07:31:10.698468+0000","last_clean":"2024-03-06T07:31:10.698468+0000","last_became_active":"2024-03-06T07:31:09.735302+0000","last_became_peered":"2024-03-06T07:31:09.735302+0000","last_unstale":"2024-03-06T07:31:10.698468+0000","last_undegraded":"2024-03-06T07:31:10.698468+0000","last_fullsized":"2024-03-06T07:31:10.698468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:09.747371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.736808+0000","last_change":"2024-03-06T07:31:08.699062+0000","last_active":"2024-03-06T07:31:10.736808+0000","last_peered":"2024-03-06T07:31:10.736808+0000","last_clean":"2024-03-06T07:31:10.736808+0000","last_became_active":"2024-03-06T07:31:08.698767+0000","last_became_peered":"2024-03-06T07:31:08.698767+0000","last_unstale":"2024-03-06T07:31:10.736808+0000","last_undegraded":"2024-03-06T07:31:10.736808+0000","last_fullsized":"2024-03-06T07:31:10.736808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:05.110150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699664+0000","last_change":"2024-03-06T07:31:09.729881+0000","last_active":"2024-03-06T07:31:10.699664+0000","last_peered":"2024-03-06T07:31:10.699664+0000","last_clean":"2024-03-06T07:31:10.699664+0000","last_became_active":"2024-03-06T07:31:09.728303+0000","last_became_peered":"2024-03-06T07:31:09.728303+0000","last_unstale":"2024-03-06T07:31:10.699664+0000","last_undegraded":"2024-03-06T07:31:10.699664+0000","last_fullsized":"2024-03-06T07:31:10.699664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:39.411341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313005+0000","last_change":"2024-03-06T07:31:08.708615+0000","last_active":"2024-03-06T07:31:11.313005+0000","last_peered":"2024-03-06T07:31:11.313005+0000","last_clean":"2024-03-06T07:31:11.313005+0000","last_became_active":"2024-03-06T07:31:08.708149+0000","last_became_peered":"2024-03-06T07:31:08.708149+0000","last_unstale":"2024-03-06T07:31:11.313005+0000","last_undegraded":"2024-03-06T07:31:11.313005+0000","last_fullsized":"2024-03-06T07:31:11.313005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:19.844233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315110+0000","last_change":"2024-03-06T07:31:09.712575+0000","last_active":"2024-03-06T07:31:11.315110+0000","last_peered":"2024-03-06T07:31:11.315110+0000","last_clean":"2024-03-06T07:31:11.315110+0000","last_became_active":"2024-03-06T07:31:09.712036+0000","last_became_peered":"2024-03-06T07:31:09.712036+0000","last_unstale":"2024-03-06T07:31:11.315110+0000","last_undegraded":"2024-03-06T07:31:11.315110+0000","last_fullsized":"2024-03-06T07:31:11.315110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:05.374622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700610+0000","last_change":"2024-03-06T07:31:08.706291+0000","last_active":"2024-03-06T07:31:10.700610+0000","last_peered":"2024-03-06T07:31:10.700610+0000","last_clean":"2024-03-06T07:31:10.700610+0000","last_became_active":"2024-03-06T07:31:08.705537+0000","last_became_peered":"2024-03-06T07:31:08.705537+0000","last_unstale":"2024-03-06T07:31:10.700610+0000","last_undegraded":"2024-03-06T07:31:10.700610+0000","last_fullsized":"2024-03-06T07:31:10.700610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:58.716593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315787+0000","last_change":"2024-03-06T07:31:09.728588+0000","last_active":"2024-03-06T07:31:11.315787+0000","last_peered":"2024-03-06T07:31:11.315787+0000","last_clean":"2024-03-06T07:31:11.315787+0000","last_became_active":"2024-03-06T07:31:09.728244+0000","last_became_peered":"2024-03-06T07:31:09.728244+0000","last_unstale":"2024-03-06T07:31:11.315787+0000","last_undegraded":"2024-03-06T07:31:11.315787+0000","last_fullsized":"2024-03-06T07:31:11.315787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:02.267334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.315263+0000","last_change":"2024-03-06T07:31:08.708950+0000","last_active":"2024-03-06T07:31:11.315263+0000","last_peered":"2024-03-06T07:31:11.315263+0000","last_clean":"2024-03-06T07:31:11.315263+0000","last_became_active":"2024-03-06T07:31:08.707758+0000","last_became_peered":"2024-03-06T07:31:08.707758+0000","last_unstale":"2024-03-06T07:31:11.315263+0000","last_undegraded":"2024-03-06T07:31:11.315263+0000","last_fullsized":"2024-03-06T07:31:11.315263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:25.868683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312246+0000","last_change":"2024-03-06T07:31:09.725327+0000","last_active":"2024-03-06T07:31:11.312246+0000","last_peered":"2024-03-06T07:31:11.312246+0000","last_clean":"2024-03-06T07:31:11.312246+0000","last_became_active":"2024-03-06T07:31:09.724605+0000","last_became_peered":"2024-03-06T07:31:09.724605+0000","last_unstale":"2024-03-06T07:31:11.312246+0000","last_undegraded":"2024-03-06T07:31:11.312246+0000","last_fullsized":"2024-03-06T07:31:11.312246+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:24.729631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700202+0000","last_change":"2024-03-06T07:31:08.708472+0000","last_active":"2024-03-06T07:31:10.700202+0000","last_peered":"2024-03-06T07:31:10.700202+0000","last_clean":"2024-03-06T07:31:10.700202+0000","last_became_active":"2024-03-06T07:31:08.708186+0000","last_became_peered":"2024-03-06T07:31:08.708186+0000","last_unstale":"2024-03-06T07:31:10.700202+0000","last_undegraded":"2024-03-06T07:31:10.700202+0000","last_fullsized":"2024-03-06T07:31:10.700202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:08.663518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700925+0000","last_change":"2024-03-06T07:31:09.731228+0000","last_active":"2024-03-06T07:31:10.700925+0000","last_peered":"2024-03-06T07:31:10.700925+0000","last_clean":"2024-03-06T07:31:10.700925+0000","last_became_active":"2024-03-06T07:31:09.730895+0000","last_became_peered":"2024-03-06T07:31:09.730895+0000","last_unstale":"2024-03-06T07:31:10.700925+0000","last_undegraded":"2024-03-06T07:31:10.700925+0000","last_fullsized":"2024-03-06T07:31:10.700925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:59.310776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314363+0000","last_change":"2024-03-06T07:31:08.710889+0000","last_active":"2024-03-06T07:31:11.314363+0000","last_peered":"2024-03-06T07:31:11.314363+0000","last_clean":"2024-03-06T07:31:11.314363+0000","last_became_active":"2024-03-06T07:31:08.709406+0000","last_became_peered":"2024-03-06T07:31:08.709406+0000","last_unstale":"2024-03-06T07:31:11.314363+0000","last_undegraded":"2024-03-06T07:31:11.314363+0000","last_fullsized":"2024-03-06T07:31:11.314363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:24.449426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211460+0000","last_change":"2024-03-06T07:31:09.711012+0000","last_active":"2024-03-06T07:31:11.211460+0000","last_peered":"2024-03-06T07:31:11.211460+0000","last_clean":"2024-03-06T07:31:11.211460+0000","last_became_active":"2024-03-06T07:31:09.710327+0000","last_became_peered":"2024-03-06T07:31:09.710327+0000","last_unstale":"2024-03-06T07:31:11.211460+0000","last_undegraded":"2024-03-06T07:31:11.211460+0000","last_fullsized":"2024-03-06T07:31:11.211460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.275443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700911+0000","last_change":"2024-03-06T07:31:08.704271+0000","last_active":"2024-03-06T07:31:10.700911+0000","last_peered":"2024-03-06T07:31:10.700911+0000","last_clean":"2024-03-06T07:31:10.700911+0000","last_became_active":"2024-03-06T07:31:08.700885+0000","last_became_peered":"2024-03-06T07:31:08.700885+0000","last_unstale":"2024-03-06T07:31:10.700911+0000","last_undegraded":"2024-03-06T07:31:10.700911+0000","last_fullsized":"2024-03-06T07:31:10.700911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:20.932727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211402+0000","last_change":"2024-03-06T07:31:09.744759+0000","last_active":"2024-03-06T07:31:11.211402+0000","last_peered":"2024-03-06T07:31:11.211402+0000","last_clean":"2024-03-06T07:31:11.211402+0000","last_became_active":"2024-03-06T07:31:09.744362+0000","last_became_peered":"2024-03-06T07:31:09.744362+0000","last_unstale":"2024-03-06T07:31:11.211402+0000","last_undegraded":"2024-03-06T07:31:11.211402+0000","last_fullsized":"2024-03-06T07:31:11.211402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:05.303513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737373+0000","last_change":"2024-03-06T07:31:08.698893+0000","last_active":"2024-03-06T07:31:10.737373+0000","last_peered":"2024-03-06T07:31:10.737373+0000","last_clean":"2024-03-06T07:31:10.737373+0000","last_became_active":"2024-03-06T07:31:08.698419+0000","last_became_peered":"2024-03-06T07:31:08.698419+0000","last_unstale":"2024-03-06T07:31:10.737373+0000","last_undegraded":"2024-03-06T07:31:10.737373+0000","last_fullsized":"2024-03-06T07:31:10.737373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:30.932957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211055+0000","last_change":"2024-03-06T07:31:09.733966+0000","last_active":"2024-03-06T07:31:11.211055+0000","last_peered":"2024-03-06T07:31:11.211055+0000","last_clean":"2024-03-06T07:31:11.211055+0000","last_became_active":"2024-03-06T07:31:09.732858+0000","last_became_peered":"2024-03-06T07:31:09.732858+0000","last_unstale":"2024-03-06T07:31:11.211055+0000","last_undegraded":"2024-03-06T07:31:11.211055+0000","last_fullsized":"2024-03-06T07:31:11.211055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:31.500909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.698202+0000","last_change":"2024-03-06T07:31:08.687577+0000","last_active":"2024-03-06T07:31:10.698202+0000","last_peered":"2024-03-06T07:31:10.698202+0000","last_clean":"2024-03-06T07:31:10.698202+0000","last_became_active":"2024-03-06T07:31:08.687300+0000","last_became_peered":"2024-03-06T07:31:08.687300+0000","last_unstale":"2024-03-06T07:31:10.698202+0000","last_undegraded":"2024-03-06T07:31:10.698202+0000","last_fullsized":"2024-03-06T07:31:10.698202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:05.110150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699548+0000","last_change":"2024-03-06T07:31:09.735514+0000","last_active":"2024-03-06T07:31:10.699548+0000","last_peered":"2024-03-06T07:31:10.699548+0000","last_clean":"2024-03-06T07:31:10.699548+0000","last_became_active":"2024-03-06T07:31:09.733652+0000","last_became_peered":"2024-03-06T07:31:09.733652+0000","last_unstale":"2024-03-06T07:31:10.699548+0000","last_undegraded":"2024-03-06T07:31:10.699548+0000","last_fullsized":"2024-03-06T07:31:10.699548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:53.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.700894+0000","last_change":"2024-03-06T07:31:08.710412+0000","last_active":"2024-03-06T07:31:10.700894+0000","last_peered":"2024-03-06T07:31:10.700894+0000","last_clean":"2024-03-06T07:31:10.700894+0000","last_became_active":"2024-03-06T07:31:08.710061+0000","last_became_peered":"2024-03-06T07:31:08.710061+0000","last_unstale":"2024-03-06T07:31:10.700894+0000","last_undegraded":"2024-03-06T07:31:10.700894+0000","last_fullsized":"2024-03-06T07:31:10.700894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:01.053729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737817+0000","last_change":"2024-03-06T07:31:09.718215+0000","last_active":"2024-03-06T07:31:10.737817+0000","last_peered":"2024-03-06T07:31:10.737817+0000","last_clean":"2024-03-06T07:31:10.737817+0000","last_became_active":"2024-03-06T07:31:09.717858+0000","last_became_peered":"2024-03-06T07:31:09.717858+0000","last_unstale":"2024-03-06T07:31:10.737817+0000","last_undegraded":"2024-03-06T07:31:10.737817+0000","last_fullsized":"2024-03-06T07:31:10.737817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:57.600744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.737782+0000","last_change":"2024-03-06T07:31:08.695443+0000","last_active":"2024-03-06T07:31:10.737782+0000","last_peered":"2024-03-06T07:31:10.737782+0000","last_clean":"2024-03-06T07:31:10.737782+0000","last_became_active":"2024-03-06T07:31:08.695112+0000","last_became_peered":"2024-03-06T07:31:08.695112+0000","last_unstale":"2024-03-06T07:31:10.737782+0000","last_undegraded":"2024-03-06T07:31:10.737782+0000","last_fullsized":"2024-03-06T07:31:10.737782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:11.963924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314122+0000","last_change":"2024-03-06T07:31:09.745311+0000","last_active":"2024-03-06T07:31:11.314122+0000","last_peered":"2024-03-06T07:31:11.314122+0000","last_clean":"2024-03-06T07:31:11.314122+0000","last_became_active":"2024-03-06T07:31:09.744759+0000","last_became_peered":"2024-03-06T07:31:09.744759+0000","last_unstale":"2024-03-06T07:31:11.314122+0000","last_undegraded":"2024-03-06T07:31:11.314122+0000","last_fullsized":"2024-03-06T07:31:11.314122+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:28.248865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.701276+0000","last_change":"2024-03-06T07:31:08.691823+0000","last_active":"2024-03-06T07:31:10.701276+0000","last_peered":"2024-03-06T07:31:10.701276+0000","last_clean":"2024-03-06T07:31:10.701276+0000","last_became_active":"2024-03-06T07:31:08.691505+0000","last_became_peered":"2024-03-06T07:31:08.691505+0000","last_unstale":"2024-03-06T07:31:10.701276+0000","last_undegraded":"2024-03-06T07:31:10.701276+0000","last_fullsized":"2024-03-06T07:31:10.701276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:49.340554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.314973+0000","last_change":"2024-03-06T07:31:09.729461+0000","last_active":"2024-03-06T07:31:11.314973+0000","last_peered":"2024-03-06T07:31:11.314973+0000","last_clean":"2024-03-06T07:31:11.314973+0000","last_became_active":"2024-03-06T07:31:09.729200+0000","last_became_peered":"2024-03-06T07:31:09.729200+0000","last_unstale":"2024-03-06T07:31:11.314973+0000","last_undegraded":"2024-03-06T07:31:11.314973+0000","last_fullsized":"2024-03-06T07:31:11.314973+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:43.967010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699926+0000","last_change":"2024-03-06T07:31:08.698585+0000","last_active":"2024-03-06T07:31:10.699926+0000","last_peered":"2024-03-06T07:31:10.699926+0000","last_clean":"2024-03-06T07:31:10.699926+0000","last_became_active":"2024-03-06T07:31:08.698166+0000","last_became_peered":"2024-03-06T07:31:08.698166+0000","last_unstale":"2024-03-06T07:31:10.699926+0000","last_undegraded":"2024-03-06T07:31:10.699926+0000","last_fullsized":"2024-03-06T07:31:10.699926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:30.932957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699716+0000","last_change":"2024-03-06T07:31:09.721386+0000","last_active":"2024-03-06T07:31:10.699716+0000","last_peered":"2024-03-06T07:31:10.699716+0000","last_clean":"2024-03-06T07:31:10.699716+0000","last_became_active":"2024-03-06T07:31:09.720479+0000","last_became_peered":"2024-03-06T07:31:09.720479+0000","last_unstale":"2024-03-06T07:31:10.699716+0000","last_undegraded":"2024-03-06T07:31:10.699716+0000","last_fullsized":"2024-03-06T07:31:10.699716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:41.553025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699681+0000","last_change":"2024-03-06T07:31:08.696740+0000","last_active":"2024-03-06T07:31:10.699681+0000","last_peered":"2024-03-06T07:31:10.699681+0000","last_clean":"2024-03-06T07:31:10.699681+0000","last_became_active":"2024-03-06T07:31:08.696450+0000","last_became_peered":"2024-03-06T07:31:08.696450+0000","last_unstale":"2024-03-06T07:31:10.699681+0000","last_undegraded":"2024-03-06T07:31:10.699681+0000","last_fullsized":"2024-03-06T07:31:10.699681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:46.905707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699612+0000","last_change":"2024-03-06T07:31:09.737762+0000","last_active":"2024-03-06T07:31:10.699612+0000","last_peered":"2024-03-06T07:31:10.699612+0000","last_clean":"2024-03-06T07:31:10.699612+0000","last_became_active":"2024-03-06T07:31:09.737289+0000","last_became_peered":"2024-03-06T07:31:09.737289+0000","last_unstale":"2024-03-06T07:31:10.699612+0000","last_undegraded":"2024-03-06T07:31:10.699612+0000","last_fullsized":"2024-03-06T07:31:10.699612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:13.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311646+0000","last_change":"2024-03-06T07:31:08.692727+0000","last_active":"2024-03-06T07:31:11.311646+0000","last_peered":"2024-03-06T07:31:11.311646+0000","last_clean":"2024-03-06T07:31:11.311646+0000","last_became_active":"2024-03-06T07:31:08.692457+0000","last_became_peered":"2024-03-06T07:31:08.692457+0000","last_unstale":"2024-03-06T07:31:11.311646+0000","last_undegraded":"2024-03-06T07:31:11.311646+0000","last_fullsized":"2024-03-06T07:31:11.311646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:46.370019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.311961+0000","last_change":"2024-03-06T07:31:09.709061+0000","last_active":"2024-03-06T07:31:11.311961+0000","last_peered":"2024-03-06T07:31:11.311961+0000","last_clean":"2024-03-06T07:31:11.311961+0000","last_became_active":"2024-03-06T07:31:09.708573+0000","last_became_peered":"2024-03-06T07:31:09.708573+0000","last_unstale":"2024-03-06T07:31:11.311961+0000","last_undegraded":"2024-03-06T07:31:11.311961+0000","last_fullsized":"2024-03-06T07:31:11.311961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:40.415716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.312822+0000","last_change":"2024-03-06T07:31:08.703245+0000","last_active":"2024-03-06T07:31:11.312822+0000","last_peered":"2024-03-06T07:31:11.312822+0000","last_clean":"2024-03-06T07:31:11.312822+0000","last_became_active":"2024-03-06T07:31:08.702878+0000","last_became_peered":"2024-03-06T07:31:08.702878+0000","last_unstale":"2024-03-06T07:31:11.312822+0000","last_undegraded":"2024-03-06T07:31:11.312822+0000","last_fullsized":"2024-03-06T07:31:11.312822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:24.790896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699273+0000","last_change":"2024-03-06T07:31:09.742322+0000","last_active":"2024-03-06T07:31:10.699273+0000","last_peered":"2024-03-06T07:31:10.699273+0000","last_clean":"2024-03-06T07:31:10.699273+0000","last_became_active":"2024-03-06T07:31:09.737823+0000","last_became_peered":"2024-03-06T07:31:09.737823+0000","last_unstale":"2024-03-06T07:31:10.699273+0000","last_undegraded":"2024-03-06T07:31:10.699273+0000","last_fullsized":"2024-03-06T07:31:10.699273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:12.115764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699855+0000","last_change":"2024-03-06T07:31:08.707266+0000","last_active":"2024-03-06T07:31:10.699855+0000","last_peered":"2024-03-06T07:31:10.699855+0000","last_clean":"2024-03-06T07:31:10.699855+0000","last_became_active":"2024-03-06T07:31:08.706774+0000","last_became_peered":"2024-03-06T07:31:08.706774+0000","last_unstale":"2024-03-06T07:31:10.699855+0000","last_undegraded":"2024-03-06T07:31:10.699855+0000","last_fullsized":"2024-03-06T07:31:10.699855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:22.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.211286+0000","last_change":"2024-03-06T07:31:09.733311+0000","last_active":"2024-03-06T07:31:11.211286+0000","last_peered":"2024-03-06T07:31:11.211286+0000","last_clean":"2024-03-06T07:31:11.211286+0000","last_became_active":"2024-03-06T07:31:09.732908+0000","last_became_peered":"2024-03-06T07:31:09.732908+0000","last_unstale":"2024-03-06T07:31:11.211286+0000","last_undegraded":"2024-03-06T07:31:11.211286+0000","last_fullsized":"2024-03-06T07:31:11.211286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:08.660088+0000","last_clean_scrub_stamp":"2024-03-06T07:31:08.660088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:50.935114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.699895+0000","last_change":"2024-03-06T07:31:08.707358+0000","last_active":"2024-03-06T07:31:10.699895+0000","last_peered":"2024-03-06T07:31:10.699895+0000","last_clean":"2024-03-06T07:31:10.699895+0000","last_became_active":"2024-03-06T07:31:08.706392+0000","last_became_peered":"2024-03-06T07:31:08.706392+0000","last_unstale":"2024-03-06T07:31:10.699895+0000","last_undegraded":"2024-03-06T07:31:10.699895+0000","last_fullsized":"2024-03-06T07:31:10.699895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:07.651591+0000","last_clean_scrub_stamp":"2024-03-06T07:31:07.651591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:58.302278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:11.313628+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:31:11.313628+0000","last_peered":"2024-03-06T07:31:11.313628+0000","last_clean":"2024-03-06T07:31:11.313628+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:31:11.313628+0000","last_undegraded":"2024-03-06T07:31:11.313628+0000","last_fullsized":"2024-03-06T07:31:11.313628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542190,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1437696,"data_stored":940975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542191,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":843776,"data_stored":350083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":350081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":350061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":843776,"data_stored":350103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542192,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":351603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":352219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":940359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:19.714 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:31:19.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:31:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.963+0000 7f2521c00640 1 Processor -- start 2024-03-06T07:31:19.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.964+0000 7f2521c00640 1 -- start start 2024-03-06T07:31:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.965+0000 7f2521c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f251c145320 0x7f251c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.965+0000 7f2521c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f251c160420 0x7f251c06b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.965+0000 7f2521c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251c06c940 con 0x7f251c06b920 2024-03-06T07:31:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.965+0000 7f2521c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251c06c6f0 con 0x7f251c145320 2024-03-06T07:31:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.965+0000 7f2521c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251c06cc20 con 0x7f251c160420 2024-03-06T07:31:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.965+0000 7f251be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f251c145320 0x7f251c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f2521200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f251c06b920 0x7f251c0cb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48186/0 (socket says 172.21.15.97:48186) 2024-03-06T07:31:19.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f2521200640 1 -- 172.21.15.97:0/1559480420 learned_addr learned my addr 172.21.15.97:0/1559480420 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f2513400640 1 --2- 172.21.15.97:0/1559480420 >> v2:172.21.15.135:3300/0 conn(0x7f251c160420 0x7f251c06b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f251be00640 1 -- 172.21.15.97:0/1559480420 >> v2:172.21.15.135:3300/0 conn(0x7f251c160420 msgr2=0x7f251c06b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f251be00640 1 --2- 172.21.15.97:0/1559480420 >> v2:172.21.15.135:3300/0 conn(0x7f251c160420 0x7f251c06b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f251be00640 1 -- 172.21.15.97:0/1559480420 >> v1:172.21.15.97:6789/0 conn(0x7f251c06b920 legacy=0x7f251c0cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f251be00640 1 -- 172.21.15.97:0/1559480420 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c06cf00 con 0x7f251c145320 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f2513400640 1 --2- 172.21.15.97:0/1559480420 >> v2:172.21.15.135:3300/0 conn(0x7f251c160420 0x7f251c06b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.966+0000 7f251b400640 1 -- 172.21.15.97:0/1559480420 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364012766 0 0) 0x7f251c06c940 con 0x7f251c06b920 2024-03-06T07:31:19.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.967+0000 7f251be00640 1 --2- 172.21.15.97:0/1559480420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c0b48a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f251c0b26c0 tx=0x7f250c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc0ceacf97ea427 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.967+0000 7f251b400640 1 -- 172.21.15.97:0/1559480420 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c039070 con 0x7f251c145320 2024-03-06T07:31:19.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f251b400640 1 -- 172.21.15.97:0/1559480420 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c02cc80 con 0x7f251c145320 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f251b400640 1 -- 172.21.15.97:0/1559480420 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c034660 con 0x7f251c145320 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 -- 172.21.15.97:0/1559480420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 msgr2=0x7f251c0b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 --2- 172.21.15.97:0/1559480420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c0b48a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f251c0b26c0 tx=0x7f250c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 -- 172.21.15.97:0/1559480420 shutdown_connections 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 --2- 172.21.15.97:0/1559480420 >> v2:172.21.15.135:3300/0 conn(0x7f251c160420 0x7f251c06b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 --2- 172.21.15.97:0/1559480420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c0b48a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 -- 172.21.15.97:0/1559480420 >> 172.21.15.97:0/1559480420 conn(0x7f251c0b1a70 msgr2=0x7f251c0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 -- 172.21.15.97:0/1559480420 shutdown_connections 2024-03-06T07:31:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.968+0000 7f2521c00640 1 -- 172.21.15.97:0/1559480420 wait complete. 2024-03-06T07:31:19.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.970+0000 7f2521c00640 1 Processor -- start 2024-03-06T07:31:19.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.970+0000 7f2521c00640 1 -- start start 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2521c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f251c06b920 0x7f251c1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2521c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c13abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2521c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f251c13b120 0x7f251c140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2521c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c0b4de0 con 0x7f251c145320 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2521c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c06d4c0 con 0x7f251c06b920 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2521c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c145700 con 0x7f251c13b120 2024-03-06T07:31:19.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2513400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c13abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2513400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c13abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39840/0 (socket says 172.21.15.97:39840) 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.971+0000 7f2513400640 1 -- 172.21.15.97:0/3657224880 learned_addr learned my addr 172.21.15.97:0/3657224880 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f251be00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f251c06b920 0x7f251c1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2521200640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f251c13b120 0x7f251c140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2513400640 1 -- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f251c13b120 msgr2=0x7f251c140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2513400640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f251c13b120 0x7f251c140220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2513400640 1 -- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f251c06b920 msgr2=0x7f251c1567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2513400640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f251c06b920 0x7f251c1567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2513400640 1 -- 172.21.15.97:0/3657224880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c1512f0 con 0x7f251c145320 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2513400640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c13abe0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f250400c7c0 tx=0x7f250400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504015070 con 0x7f251c145320 2024-03-06T07:31:19.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504007820 con 0x7f251c145320 2024-03-06T07:31:19.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504007bf0 con 0x7f251c145320 2024-03-06T07:31:19.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.972+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c1515d0 con 0x7f251c145320 2024-03-06T07:31:19.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.973+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c151e70 con 0x7f251c145320 2024-03-06T07:31:19.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.974+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c1517a0 con 0x7f251c145320 2024-03-06T07:31:19.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.976+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2504018070 con 0x7f251c145320 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.977+0000 7f2518c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24f806b4d0 0x7f24f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.977+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f250408a190 con 0x7f251c145320 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.977+0000 7f251be00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24f806b4d0 0x7f24f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.978+0000 7f251be00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24f806b4d0 0x7f24f806d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f250c036000 tx=0x7f250c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:19.981+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2504058f70 con 0x7f251c145320 2024-03-06T07:31:20.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.249+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f251c06e6f0 con 0x7f251c145320 2024-03-06T07:31:20.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.253+0000 7f2518c00640 1 -- 172.21.15.97:0/3657224880 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2504058870 con 0x7f251c145320 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:31:20.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24f806b4d0 msgr2=0x7f24f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24f806b4d0 0x7f24f806d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f250c036000 tx=0x7f250c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 msgr2=0x7f251c13abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c13abe0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f250400c7c0 tx=0x7f250400cc80 comp rx=0 tx=0).stop 2024-03-06T07:31:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 shutdown_connections 2024-03-06T07:31:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24f806b4d0 0x7f24f806d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f251c13b120 0x7f251c140220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f251c145320 0x7f251c13abe0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 --2- 172.21.15.97:0/3657224880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f251c06b920 0x7f251c1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.255+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 >> 172.21.15.97:0/3657224880 conn(0x7f251c0b1a70 msgr2=0x7f251c0ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.256+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 shutdown_connections 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.256+0000 7f2521c00640 1 -- 172.21.15.97:0/3657224880 wait complete. 2024-03-06T07:31:20.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:20.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.473+0000 7f528f200640 1 Processor -- start 2024-03-06T07:31:20.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.474+0000 7f528f200640 1 -- start start 2024-03-06T07:31:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f52900d0c00 0x7f52900d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f52900b2e50 0x7f52900b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52900b5240 con 0x7f529015e200 2024-03-06T07:31:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52900d2ec0 con 0x7f52900d0c00 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52900cf4d0 con 0x7f52900b2e50 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528de00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f529015e200 0x7f52900d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48188/0 (socket says 172.21.15.97:48188) 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.475+0000 7f528de00640 1 -- 172.21.15.97:0/3140576748 learned_addr learned my addr 172.21.15.97:0/3140576748 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528d400640 1 --2- 172.21.15.97:0/3140576748 >> v2:172.21.15.97:3300/0 conn(0x7f52900d0c00 0x7f52900d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528e800640 1 --2- 172.21.15.97:0/3140576748 >> v2:172.21.15.135:3300/0 conn(0x7f52900b2e50 0x7f52900b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528ca00640 1 -- 172.21.15.97:0/3140576748 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167983666 0 0) 0x7f52900b5240 con 0x7f529015e200 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528ca00640 1 -- 172.21.15.97:0/3140576748 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5270003520 con 0x7f529015e200 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528d400640 1 -- 172.21.15.97:0/3140576748 >> v2:172.21.15.135:3300/0 conn(0x7f52900b2e50 msgr2=0x7f52900b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528d400640 1 --2- 172.21.15.97:0/3140576748 >> v2:172.21.15.135:3300/0 conn(0x7f52900b2e50 0x7f52900b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528d400640 1 -- 172.21.15.97:0/3140576748 >> v1:172.21.15.97:6789/0 conn(0x7f529015e200 legacy=0x7f52900d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.476+0000 7f528d400640 1 -- 172.21.15.97:0/3140576748 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900b5500 con 0x7f52900d0c00 2024-03-06T07:31:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.477+0000 7f528d400640 1 --2- 172.21.15.97:0/3140576748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900d4850 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f52900cad90 tx=0x7f527802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6067a19d65a9b983 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528ca00640 1 -- 172.21.15.97:0/3140576748 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278034070 con 0x7f52900d0c00 2024-03-06T07:31:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528ca00640 1 -- 172.21.15.97:0/3140576748 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278030610 con 0x7f52900d0c00 2024-03-06T07:31:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528ca00640 1 -- 172.21.15.97:0/3140576748 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52780309c0 con 0x7f52900d0c00 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 -- 172.21.15.97:0/3140576748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 msgr2=0x7f52900d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 --2- 172.21.15.97:0/3140576748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900d4850 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f52900cad90 tx=0x7f527802b960 comp rx=0 tx=0).stop 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 -- 172.21.15.97:0/3140576748 shutdown_connections 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 --2- 172.21.15.97:0/3140576748 >> v2:172.21.15.135:3300/0 conn(0x7f52900b2e50 0x7f52900b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 --2- 172.21.15.97:0/3140576748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900d4850 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 -- 172.21.15.97:0/3140576748 >> 172.21.15.97:0/3140576748 conn(0x7f52900d3d30 msgr2=0x7f52900d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 -- 172.21.15.97:0/3140576748 shutdown_connections 2024-03-06T07:31:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.478+0000 7f528f200640 1 -- 172.21.15.97:0/3140576748 wait complete. 2024-03-06T07:31:20.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.480+0000 7f528f200640 1 Processor -- start 2024-03-06T07:31:20.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.480+0000 7f528f200640 1 -- start start 2024-03-06T07:31:20.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52900b2e50 0x7f52900ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f529015e200 0x7f52900b8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900cfd60 con 0x7f52900d0c00 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900cfa80 con 0x7f52900b2e50 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529015e5e0 con 0x7f529015e200 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39872/0 (socket says 172.21.15.97:39872) 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528d400640 1 -- 172.21.15.97:0/84952375 learned_addr learned my addr 172.21.15.97:0/84952375 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528d400640 1 -- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f529015e200 msgr2=0x7f52900b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528d400640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f529015e200 0x7f52900b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528d400640 1 -- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52900b2e50 msgr2=0x7f52900ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.481+0000 7f528de00640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52900b2e50 0x7f52900ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f528d400640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52900b2e50 0x7f52900ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f528d400640 1 -- 172.21.15.97:0/84952375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900b74e0 con 0x7f52900d0c00 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f528de00640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52900b2e50 0x7f52900ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f528d400640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900cae00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5278000c00 tx=0x7f52780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278049070 con 0x7f52900d0c00 2024-03-06T07:31:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278030410 con 0x7f52900d0c00 2024-03-06T07:31:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52900b77a0 con 0x7f52900d0c00 2024-03-06T07:31:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278034040 con 0x7f52900d0c00 2024-03-06T07:31:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.482+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52900cb340 con 0x7f52900d0c00 2024-03-06T07:31:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.484+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529004f330 con 0x7f52900d0c00 2024-03-06T07:31:20.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.484+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f527804c070 con 0x7f52900d0c00 2024-03-06T07:31:20.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.486+0000 7f5286000640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f526c06b500 0x7f526c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.486+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52780abe00 con 0x7f52900d0c00 2024-03-06T07:31:20.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.486+0000 7f528de00640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f526c06b500 0x7f526c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.486+0000 7f528de00640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f526c06b500 0x7f526c06d9c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5280005880 tx=0x7f5280009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.489+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f527807abe0 con 0x7f52900d0c00 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.748+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f529008d4c0 con 0x7f52900d0c00 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.749+0000 7f5286000640 1 -- 172.21.15.97:0/84952375 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f527807a4e0 con 0x7f52900d0c00 2024-03-06T07:31:20.752 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:20.752 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:20.752 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:20.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f526c06b500 msgr2=0x7f526c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f526c06b500 0x7f526c06d9c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5280005880 tx=0x7f5280009040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 msgr2=0x7f52900cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900cae00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5278000c00 tx=0x7f52780046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 shutdown_connections 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f526c06b500 0x7f526c06d9c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f529015e200 0x7f52900b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52900d0c00 0x7f52900cae00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 --2- 172.21.15.97:0/84952375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52900b2e50 0x7f52900ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 >> 172.21.15.97:0/84952375 conn(0x7f52900d3d30 msgr2=0x7f52900b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 shutdown_connections 2024-03-06T07:31:20.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:20.752+0000 7f528f200640 1 -- 172.21.15.97:0/84952375 wait complete. 2024-03-06T07:31:21.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:09.132065+0000', 'modified': '2024-03-06T07:31:12.252322+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6318': {'gid': 6318, 'name': 'a', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3151111028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 3151111028}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3151111028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6318, 'qdb_cluster': [6318]} 2024-03-06T07:31:21.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:21.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:31:21.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.382+0000 7f6ffd600640 1 Processor -- start 2024-03-06T07:31:21.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.382+0000 7f6ffd600640 1 -- start start 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.383+0000 7f6ffd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff8067000 0x7f6ff80d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.383+0000 7f6ffd600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6ff806bd20 0x7f6ff806c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.383+0000 7f6ffd600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80b58c0 con 0x7f6ff815e0d0 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffd600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff80d0220 con 0x7f6ff806bd20 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80d0500 con 0x7f6ff8067000 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffcc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6ff806bd20 0x7f6ff806c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffcc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6ff806bd20 0x7f6ff806c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39884/0 (socket says 172.21.15.97:39884) 2024-03-06T07:31:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2304551623 learned_addr learned my addr 172.21.15.97:0/2304551623 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ff7400640 1 --2- 172.21.15.97:0/2304551623 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff8067000 0x7f6ff80d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2304551623 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff8067000 msgr2=0x7f6ff80d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2304551623 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff8067000 0x7f6ff80d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.384+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2304551623 >> v1:172.21.15.135:6789/0 conn(0x7f6ff815e0d0 legacy=0x7f6ff80b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.385+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2304551623 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80d07e0 con 0x7f6ff806bd20 2024-03-06T07:31:21.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.386+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2304551623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff806bd20 0x7f6ff806c1b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009e50 tx=0x7f6ff002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafa79af38aec837 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.386+0000 7f6ff6a00640 1 -- 172.21.15.97:0/2304551623 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff002cb00 con 0x7f6ff806bd20 2024-03-06T07:31:21.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.386+0000 7f6ff6a00640 1 -- 172.21.15.97:0/2304551623 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff002cca0 con 0x7f6ff806bd20 2024-03-06T07:31:21.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.386+0000 7f6ff6a00640 1 -- 172.21.15.97:0/2304551623 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0034610 con 0x7f6ff806bd20 2024-03-06T07:31:21.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 -- 172.21.15.97:0/2304551623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff806bd20 msgr2=0x7f6ff806c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2304551623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff806bd20 0x7f6ff806c1b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009e50 tx=0x7f6ff002bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 -- 172.21.15.97:0/2304551623 shutdown_connections 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2304551623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff806bd20 0x7f6ff806c1b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2304551623 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff8067000 0x7f6ff80d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 -- 172.21.15.97:0/2304551623 >> 172.21.15.97:0/2304551623 conn(0x7f6ff80725e0 msgr2=0x7f6ff80729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 -- 172.21.15.97:0/2304551623 shutdown_connections 2024-03-06T07:31:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.387+0000 7f6ffd600640 1 -- 172.21.15.97:0/2304551623 wait complete. 2024-03-06T07:31:21.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.388+0000 7f6ffd600640 1 Processor -- start 2024-03-06T07:31:21.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.389+0000 7f6ffd600640 1 -- start start 2024-03-06T07:31:21.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.389+0000 7f6ffd600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 0x7f6ff80cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.389+0000 7f6ffd600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ff806bd20 0x7f6ff81f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.389+0000 7f6ffd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff815e0d0 0x7f6ff81f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ffd600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff815e630 con 0x7f6ff8067000 2024-03-06T07:31:21.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ffd600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff815e4b0 con 0x7f6ff806bd20 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ffd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80b5ba0 con 0x7f6ff815e0d0 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 0x7f6ff80cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 0x7f6ff80cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39900/0 (socket says 172.21.15.97:39900) 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2219482852 learned_addr learned my addr 172.21.15.97:0/2219482852 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7400640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ff806bd20 0x7f6ff81f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff815e0d0 msgr2=0x7f6ff81f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff815e0d0 0x7f6ff81f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff815e0d0 0x7f6ff81f6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ff806bd20 msgr2=0x7f6ff81f77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ff806bd20 0x7f6ff81f77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2219482852 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80cb4b0 con 0x7f6ff8067000 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.390+0000 7f6ff7400640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ff806bd20 0x7f6ff81f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.391+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 0x7f6ff80cadf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800c450 tx=0x7f6fe800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.391+0000 7f6fefe00640 1 -- 172.21.15.97:0/2219482852 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80052d0 con 0x7f6ff8067000 2024-03-06T07:31:21.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.391+0000 7f6fefe00640 1 -- 172.21.15.97:0/2219482852 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe8005470 con 0x7f6ff8067000 2024-03-06T07:31:21.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.391+0000 7f6fefe00640 1 -- 172.21.15.97:0/2219482852 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8012870 con 0x7f6ff8067000 2024-03-06T07:31:21.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.391+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff80cb790 con 0x7f6ff8067000 2024-03-06T07:31:21.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.391+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff80cbd50 con 0x7f6ff8067000 2024-03-06T07:31:21.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.394+0000 7f6fefe00640 1 -- 172.21.15.97:0/2219482852 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fe8012a10 con 0x7f6ff8067000 2024-03-06T07:31:21.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.395+0000 7f70019e4740 1 Processor -- start 2024-03-06T07:31:21.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.395+0000 7f70019e4740 1 -- start start 2024-03-06T07:31:21.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.395+0000 7f6fefe00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6fd806b4d0 0x7f6fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.395+0000 7f6fefe00640 1 -- 172.21.15.97:0/2219482852 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe8089ee0 con 0x7f6ff8067000 2024-03-06T07:31:21.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6fd806b4d0 0x7f6fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193990 0x55faab166930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab196720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f70019e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 0x55faab196c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f70019e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faab0d0600 con 0x55faab166e70 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faab079930 con 0x55faab193990 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55faab0cc6a0 con 0x55faab193d70 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f6ffcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 0x55faab196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f6ffcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 0x55faab196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39908/0 (socket says 172.21.15.97:39908) 2024-03-06T07:31:21.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2676443045 learned_addr learned my addr 172.21.15.97:0/2676443045 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193990 0x55faab166930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.396+0000 7f6ff7400640 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 msgr2=0x55faab196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193990 msgr2=0x55faab166930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193990 0x55faab166930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 -- 172.21.15.97:0/2676443045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faaade52d0 con 0x55faab166e70 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6fd806b4d0 0x7f6fd806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0036040 tx=0x7f6ff002e070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6ffcc00640 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 0x55faab196c60 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009c90 tx=0x7f6ff0048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8477bd2b906c447f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.397+0000 7f6fe3e00640 1 -- 172.21.15.97:0/2676443045 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00438e0 con 0x55faab166e70 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f6fe3e00640 1 -- 172.21.15.97:0/2676443045 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff002ce50 con 0x55faab166e70 2024-03-06T07:31:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f6fe3e00640 1 -- 172.21.15.97:0/2676443045 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0043ca0 con 0x55faab166e70 2024-03-06T07:31:21.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 -- 172.21.15.97:0/2676443045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 msgr2=0x55faab196c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 0x55faab196c60 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009c90 tx=0x7f6ff0048000 comp rx=0 tx=0).stop 2024-03-06T07:31:21.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 -- 172.21.15.97:0/2676443045 shutdown_connections 2024-03-06T07:31:21.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab166e70 0x55faab196c60 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 --2- 172.21.15.97:0/2676443045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193990 0x55faab166930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 -- 172.21.15.97:0/2676443045 >> 172.21.15.97:0/2676443045 conn(0x55faab168ba0 msgr2=0x55faab1b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 -- 172.21.15.97:0/2676443045 shutdown_connections 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.398+0000 7f70019e4740 1 -- 172.21.15.97:0/2676443045 wait complete. 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.399+0000 7f70019e4740 1 -- start start 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.399+0000 7f70019e4740 1 Processor -- start 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab1c1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193d70 0x55faab199420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab1abd50 0x55faab1ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faab16b140 con 0x55faab193990 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faab08cd90 con 0x55faab193d70 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55faab16a530 con 0x55faab1abd50 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f70019e4740 10 client.0 _ll_register_callbacks cb 0x55faab0c8350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab1c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab1c1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39912/0 (socket says 172.21.15.97:39912) 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.400+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2334453987 learned_addr learned my addr 172.21.15.97:0/2334453987 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2334453987 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab1abd50 msgr2=0x55faab1ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab1abd50 0x55faab1ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2334453987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193d70 msgr2=0x55faab199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193d70 0x55faab199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6ff7e00640 1 -- 172.21.15.97:0/2334453987 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faaaeaf6b0 con 0x55faab193990 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab1c1700 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8088370 tx=0x7f6fe8060cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283d1f7ae8c9c46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6fe0c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6fe0c00640 1 -- 172.21.15.97:0/2334453987 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8060840 con 0x55faab193990 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6fe0c00640 1 -- 172.21.15.97:0/2334453987 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe80609e0 con 0x55faab193990 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ff002ce50 con 0x55faab193990 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.401+0000 7f6fe0c00640 1 -- 172.21.15.97:0/2334453987 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fe8057a10 con 0x55faab193990 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 10 client.6480 fetch_fsmap learned FSMap version 43 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55faab185d40 con 0x55faab193990 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f6fe0c00640 1 -- 172.21.15.97:0/2334453987 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff002ce50 con 0x55faab193990 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f6fe0c00640 1 -- 172.21.15.97:0/2334453987 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 43) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6fe8057bb0 con 0x55faab193990 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 10 client.6480 fetch_fsmap finished waiting for FSMap version 43 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 10 client.6480 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 20 client.6480 populate_metadata read hostname 'smithi097' 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab1c5920 0x55faab186b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 4 client.6480 mds_command: new command op to 6318 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 --> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55faab0ddb40 con 0x55faab1c5920 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.402+0000 7f6ff7400640 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab1c5920 0x55faab186b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.403+0000 7f6ff7400640 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab1c5920 0x55faab186b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa293619e69cc441 server_cookie=d0f9a66a2855bf7c in_seq=0 out_seq=0 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.403+0000 7f6fe0c00640 10 client.6480 ms_handle_connect on v2:172.21.15.97:6836/3151111028 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.403+0000 7f6fe0c00640 1 -- 172.21.15.97:0/2334453987 <== mds.0 v2:172.21.15.97:6836/3151111028 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55faab0ddb40 con 0x55faab1c5920 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.403+0000 7f6fe0c00640 10 client.6480 handle_command_reply: tid=0 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 client.6480 shutdown 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 msgr2=0x55faab1c1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab1c1700 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8088370 tx=0x7f6fe8060cd0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 shutdown_connections 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab1c5920 0x55faab186b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab1abd50 0x55faab1ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab193d70 0x55faab199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 --2- 172.21.15.97:0/2334453987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab1c1700 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 >> 172.21.15.97:0/2334453987 conn(0x55faab166e70 msgr2=0x55faab1b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f6fe0c00640 0 client.6480 ms_handle_reset on v2:172.21.15.97:6836/3151111028 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 shutdown_connections 2024-03-06T07:31:21.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 1 -- 172.21.15.97:0/2334453987 wait complete. 2024-03-06T07:31:21.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.404+0000 7f70019e4740 20 client.6480 trim_cache size 0 max 16384 2024-03-06T07:31:21.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.422+0000 7f70019e4740 1 Processor -- start 2024-03-06T07:31:21.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.422+0000 7f70019e4740 1 -- start start 2024-03-06T07:31:21.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f70019e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab198910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab187a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 0x55faab1d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f70019e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81ff250 con 0x55faab193990 2024-03-06T07:31:21.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80cf5e0 con 0x55faab1abd50 2024-03-06T07:31:21.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80cfbe0 con 0x55faab193d70 2024-03-06T07:31:21.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39916/0 (socket says 172.21.15.97:39916) 2024-03-06T07:31:21.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.423+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1567231297 learned_addr learned my addr 172.21.15.97:0/1567231297 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1567231297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 msgr2=0x55faab187a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab187a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1567231297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 msgr2=0x55faab1d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 0x55faab1d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1567231297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faab1d5a40 con 0x55faab193990 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab198910 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800ce10 tx=0x7f6fe806d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442632263eb53d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6fe3400640 1 -- 172.21.15.97:0/1567231297 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8054c30 con 0x55faab193990 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6fe3400640 1 -- 172.21.15.97:0/1567231297 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe8060840 con 0x55faab193990 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.424+0000 7f6fe3400640 1 -- 172.21.15.97:0/1567231297 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8054dd0 con 0x55faab193990 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 -- 172.21.15.97:0/1567231297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 msgr2=0x55faab198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab198910 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800ce10 tx=0x7f6fe806d000 comp rx=0 tx=0).stop 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 -- 172.21.15.97:0/1567231297 shutdown_connections 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 0x55faab1d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab187a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 --2- 172.21.15.97:0/1567231297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab198910 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 -- 172.21.15.97:0/1567231297 >> 172.21.15.97:0/1567231297 conn(0x55faab166e70 msgr2=0x55faab1c1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 -- 172.21.15.97:0/1567231297 shutdown_connections 2024-03-06T07:31:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.425+0000 7f70019e4740 1 -- 172.21.15.97:0/1567231297 wait complete. 2024-03-06T07:31:21.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.426+0000 7f70019e4740 1 -- start start 2024-03-06T07:31:21.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.426+0000 7f70019e4740 1 Processor -- start 2024-03-06T07:31:21.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab2240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab221390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 0x55faab1657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faab1c57a0 con 0x55faab193990 2024-03-06T07:31:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faab186020 con 0x55faab1abd50 2024-03-06T07:31:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55faab1c5da0 con 0x55faab193d70 2024-03-06T07:31:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f70019e4740 10 client.0 _ll_register_callbacks cb 0x7f6fe8057f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:21.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab2240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab2240c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39928/0 (socket says 172.21.15.97:39928) 2024-03-06T07:31:21.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1840412482 learned_addr learned my addr 172.21.15.97:0/1840412482 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7400640 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab221390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 msgr2=0x55faab221390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab221390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 msgr2=0x55faab1657c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 0x55faab1657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.427+0000 7f6ff7e00640 1 -- 172.21.15.97:0/1840412482 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faab220c00 con 0x55faab193990 2024-03-06T07:31:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f6ff7e00640 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab2240c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8088370 tx=0x7f6fe8088240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2069407dba6d8152 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f6fe0c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:31:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f6fe0c00640 1 -- 172.21.15.97:0/1840412482 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8076020 con 0x55faab193990 2024-03-06T07:31:21.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f6fe0c00640 1 -- 172.21.15.97:0/1840412482 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe8057480 con 0x55faab193990 2024-03-06T07:31:21.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fe8060840 con 0x55faab193990 2024-03-06T07:31:21.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f6fe0c00640 1 -- 172.21.15.97:0/1840412482 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fe807c050 con 0x55faab193990 2024-03-06T07:31:21.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f70019e4740 10 client.6486 fetch_fsmap learned FSMap version 43 2024-03-06T07:31:21.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55faab220c00 con 0x55faab193990 2024-03-06T07:31:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.428+0000 7f6fe0c00640 1 -- 172.21.15.97:0/1840412482 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe8060840 con 0x55faab193990 2024-03-06T07:31:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f6fe0c00640 1 -- 172.21.15.97:0/1840412482 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 43) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6fe8057680 con 0x55faab193990 2024-03-06T07:31:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f70019e4740 10 client.6486 fetch_fsmap finished waiting for FSMap version 43 2024-03-06T07:31:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f70019e4740 10 client.6486 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f70019e4740 20 client.6486 populate_metadata read hostname 'smithi097' 2024-03-06T07:31:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f70019e4740 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab197790 0x55faab197b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f70019e4740 4 client.6486 mds_command: new command op to 6318 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 --> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55faab0d7e90 con 0x55faab197790 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f6ff7400640 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab197790 0x55faab197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f6ff7400640 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab197790 0x55faab197b70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fad66bffbb430b6f server_cookie=db54a8edb8ee1044 in_seq=0 out_seq=0 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.429+0000 7f6fe0c00640 10 client.6486 ms_handle_connect on v2:172.21.15.97:6836/3151111028 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f6fe0c00640 1 -- 172.21.15.97:0/1840412482 <== mds.0 v2:172.21.15.97:6836/3151111028 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55faab0d7e90 con 0x55faab197790 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f6fe0c00640 10 client.6486 handle_command_reply: tid=0 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 client.6486 shutdown 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 msgr2=0x55faab2240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab2240c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8088370 tx=0x7f6fe8088240 comp rx=0 tx=0).stop 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 shutdown_connections 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55faab1abd50 0x55faab1657c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:6836/3151111028,v1:172.21.15.97:6837/3151111028] conn(0x55faab197790 0x55faab197b70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55faab193d70 0x55faab221390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 --2- 172.21.15.97:0/1840412482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55faab193990 0x55faab2240c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 >> 172.21.15.97:0/1840412482 conn(0x55faab166e70 msgr2=0x55faab1867e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f6fe0c00640 0 client.6486 ms_handle_reset on v2:172.21.15.97:6836/3151111028 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 shutdown_connections 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 1 -- 172.21.15.97:0/1840412482 wait complete. 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.430+0000 7f70019e4740 20 client.6486 trim_cache size 0 max 16384 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 6318, 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 11.714900631000001, 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 43, 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 69, 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 69, 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 17.175555137 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6fd806b4d0 msgr2=0x7f6fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6fd806b4d0 0x7f6fd806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0036040 tx=0x7f6ff002e070 comp rx=0 tx=0).stop 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 msgr2=0x7f6ff80cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 0x7f6ff80cadf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800c450 tx=0x7f6fe800c910 comp rx=0 tx=0).stop 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 shutdown_connections 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6fd806b4d0 0x7f6fd806d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ff815e0d0 0x7f6ff81f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ff806bd20 0x7f6ff81f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 --2- 172.21.15.97:0/2219482852 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ff8067000 0x7f6ff80cadf0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 >> 172.21.15.97:0/2219482852 conn(0x7f6ff80725e0 msgr2=0x7f6ff806d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 shutdown_connections 2024-03-06T07:31:21.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.431+0000 7f6ffd600640 1 -- 172.21.15.97:0/2219482852 wait complete. 2024-03-06T07:31:21.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:21.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.644+0000 7fedf3a00640 1 Processor -- start 2024-03-06T07:31:21.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.644+0000 7fedf3a00640 1 -- start start 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fedf3a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fedec144c00 0x7fedec06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fedf3a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fedec15f9d0 0x7fedec15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fedf3a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec06ee70 con 0x7fedec0ce0a0 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fedf3a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedec06cef0 con 0x7fedec15f9d0 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fedf3a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedec06d1d0 con 0x7fedec144c00 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fede3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fedec15f9d0 0x7fedec15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fede3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fedec15f9d0 0x7fedec15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39934/0 (socket says 172.21.15.97:39934) 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.646+0000 7fede3e00640 1 -- 172.21.15.97:0/503150895 learned_addr learned my addr 172.21.15.97:0/503150895 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.647+0000 7fedf1000640 1 --2- 172.21.15.97:0/503150895 >> v2:172.21.15.135:3300/0 conn(0x7fedec144c00 0x7fedec06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.647+0000 7fede3e00640 1 -- 172.21.15.97:0/503150895 >> v2:172.21.15.135:3300/0 conn(0x7fedec144c00 msgr2=0x7fedec06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.647+0000 7fede3e00640 1 --2- 172.21.15.97:0/503150895 >> v2:172.21.15.135:3300/0 conn(0x7fedec144c00 0x7fedec06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.647+0000 7fede3e00640 1 -- 172.21.15.97:0/503150895 >> v1:172.21.15.135:6789/0 conn(0x7fedec0ce0a0 legacy=0x7fedec0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.647+0000 7fede3e00640 1 -- 172.21.15.97:0/503150895 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec06d4b0 con 0x7fedec15f9d0 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.648+0000 7fede3e00640 1 --2- 172.21.15.97:0/503150895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec15f9d0 0x7fedec15fe60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fedec0b25f0 tx=0x7fedd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698d1c21b83307f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.649+0000 7fedebe00640 1 -- 172.21.15.97:0/503150895 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4035020 con 0x7fedec15f9d0 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.649+0000 7fedebe00640 1 -- 172.21.15.97:0/503150895 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd4031870 con 0x7fedec15f9d0 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.649+0000 7fedebe00640 1 -- 172.21.15.97:0/503150895 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4031c40 con 0x7fedec15f9d0 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 -- 172.21.15.97:0/503150895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec15f9d0 msgr2=0x7fedec15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 --2- 172.21.15.97:0/503150895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec15f9d0 0x7fedec15fe60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fedec0b25f0 tx=0x7fedd402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 -- 172.21.15.97:0/503150895 shutdown_connections 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 --2- 172.21.15.97:0/503150895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec15f9d0 0x7fedec15fe60 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 --2- 172.21.15.97:0/503150895 >> v2:172.21.15.135:3300/0 conn(0x7fedec144c00 0x7fedec06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 -- 172.21.15.97:0/503150895 >> 172.21.15.97:0/503150895 conn(0x7fedec0b1730 msgr2=0x7fedec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 -- 172.21.15.97:0/503150895 shutdown_connections 2024-03-06T07:31:21.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.650+0000 7fedf3a00640 1 -- 172.21.15.97:0/503150895 wait complete. 2024-03-06T07:31:21.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.652+0000 7fedf3a00640 1 Processor -- start 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.652+0000 7fedf3a00640 1 -- start start 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf3a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec0ce0a0 0x7fedec138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf3a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 0x7fedec13c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf3a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fedec15f9d0 0x7fedec141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf3a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec06cb50 con 0x7fedec144c00 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf3a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec160730 con 0x7fedec0ce0a0 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf3a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec06ccd0 con 0x7fedec15f9d0 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 0x7fedec13c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 0x7fedec13c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39944/0 (socket says 172.21.15.97:39944) 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 -- 172.21.15.97:0/1278960516 learned_addr learned my addr 172.21.15.97:0/1278960516 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf1000640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec0ce0a0 0x7fedec138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fedf1a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fedec15f9d0 0x7fedec141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 -- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fedec15f9d0 msgr2=0x7fedec141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fedec15f9d0 0x7fedec141970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 -- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec0ce0a0 msgr2=0x7fedec138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.653+0000 7fede3e00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec0ce0a0 0x7fedec138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fede3e00640 1 -- 172.21.15.97:0/1278960516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec1f41d0 con 0x7fedec144c00 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fede3e00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 0x7fedec13c610 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000c00 tx=0x7fedd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4035040 con 0x7fedec144c00 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd4030d00 con 0x7fedec144c00 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd40456a0 con 0x7fedec144c00 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec13a590 con 0x7fedec144c00 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.654+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec0b2bd0 con 0x7fedec144c00 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.655+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec04f330 con 0x7fedec144c00 2024-03-06T07:31:21.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.656+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedd4038020 con 0x7fedec144c00 2024-03-06T07:31:21.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.658+0000 7fede9600640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fedc806b4d0 0x7fedc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.658+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedd40ac630 con 0x7fedec144c00 2024-03-06T07:31:21.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.658+0000 7fedf1000640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fedc806b4d0 0x7fedc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.659+0000 7fedf1000640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fedc806b4d0 0x7fedc806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.662+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedec06ee70 con 0x7fedec144c00 2024-03-06T07:31:21.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.937+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fedec072890 con 0x7fedec144c00 2024-03-06T07:31:21.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.937+0000 7fede9600640 1 -- 172.21.15.97:0/1278960516 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7fedd407b4a0 con 0x7fedec144c00 2024-03-06T07:31:21.940 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:21.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fedc806b4d0 msgr2=0x7fedc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fedc806b4d0 0x7fedc806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:21.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 msgr2=0x7fedec13c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 0x7fedec13c610 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000c00 tx=0x7fedd40046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 shutdown_connections 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fedc806b4d0 0x7fedc806d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fedec15f9d0 0x7fedec141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fedec144c00 0x7fedec13c610 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 --2- 172.21.15.97:0/1278960516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec0ce0a0 0x7fedec138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 >> 172.21.15.97:0/1278960516 conn(0x7fedec0b1730 msgr2=0x7fedec145370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 shutdown_connections 2024-03-06T07:31:21.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:21.940+0000 7fedf3a00640 1 -- 172.21.15.97:0/1278960516 wait complete. 2024-03-06T07:31:21.961 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:31:21.961 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:21.962 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:21.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:21.962 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:31:22.007 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:22.008 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42393sec preferred_lft 42393sec 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:31:22.009 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.010 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:22.010 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:22.010 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:31:22.010 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:22.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.011 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:31:22.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.143 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:31:22.194 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:31:22.194 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:22.194 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:31:22.195 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:31:22.195 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:22.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.195 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:31:22.195 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:31:22.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:31:22.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:31:22.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:31:22.195 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:31:22.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.370 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:31:22.421 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:22.421 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:22.423 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:22.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:31:22.502 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:22.502 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:22.552 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:22.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:22.552 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:22.601 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:31:22.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:22.734 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:22.735 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:22.765 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:22.765 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:22.765 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:31:22.765 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:31:22.765 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:31:22.767 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:31:22.767 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:31:22.767 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:31:22.767 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:31:22.767 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:31:22.770 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:31:22.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:22.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:22.845 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:31:22.905 INFO:teuthology.orchestra.run.smithi097.stdout:client5287 2024-03-06T07:31:22.905 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:22.905 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:22.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:31:23.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.213+0000 7f3f1fa00640 1 Processor -- start 2024-03-06T07:31:23.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.213+0000 7f3f1fa00640 1 -- start start 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f1fa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3f18161b50 0x7f3f180d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f1fa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3f180d4a00 0x7f3f180d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f1fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180b5490 0x7f3f180d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f1fa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f180cef40 con 0x7f3f180d4a00 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f1fa00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f180cecd0 con 0x7f3f18161b50 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f1fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180cfc80 con 0x7f3f180b5490 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f17e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3f180d4a00 0x7f3f180d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f17e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3f180d4a00 0x7f3f180d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39954/0 (socket says 172.21.15.97:39954) 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.216+0000 7f3f17e00640 1 -- 172.21.15.97:0/588546465 learned_addr learned my addr 172.21.15.97:0/588546465 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f1d000640 1 --2- 172.21.15.97:0/588546465 >> v2:172.21.15.135:3300/0 conn(0x7f3f18161b50 0x7f3f180d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f1da00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180b5490 0x7f3f180d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f17e00640 1 -- 172.21.15.97:0/588546465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180b5490 msgr2=0x7f3f180d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f17e00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180b5490 0x7f3f180d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f17e00640 1 -- 172.21.15.97:0/588546465 >> v2:172.21.15.135:3300/0 conn(0x7f3f18161b50 msgr2=0x7f3f180d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f17e00640 1 --2- 172.21.15.97:0/588546465 >> v2:172.21.15.135:3300/0 conn(0x7f3f18161b50 0x7f3f180d05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f17e00640 1 -- 172.21.15.97:0/588546465 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180c7b60 con 0x7f3f180d4a00 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.217+0000 7f3f1da00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180b5490 0x7f3f180d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.218+0000 7f3f17e00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180d4a00 0x7f3f180d0ae0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3f180d3e40 tx=0x7f3f0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491c2df68dabbe58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.218+0000 7f3f17400640 1 -- 172.21.15.97:0/588546465 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04035020 con 0x7f3f180d4a00 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.218+0000 7f3f17400640 1 -- 172.21.15.97:0/588546465 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04031870 con 0x7f3f180d4a00 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.218+0000 7f3f17400640 1 -- 172.21.15.97:0/588546465 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04031c40 con 0x7f3f180d4a00 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 -- 172.21.15.97:0/588546465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180d4a00 msgr2=0x7f3f180d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180d4a00 0x7f3f180d0ae0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3f180d3e40 tx=0x7f3f0402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 -- 172.21.15.97:0/588546465 shutdown_connections 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180b5490 0x7f3f180d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/588546465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180d4a00 0x7f3f180d0ae0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/588546465 >> v2:172.21.15.135:3300/0 conn(0x7f3f18161b50 0x7f3f180d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 -- 172.21.15.97:0/588546465 >> 172.21.15.97:0/588546465 conn(0x7f3f180d3e70 msgr2=0x7f3f180d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 -- 172.21.15.97:0/588546465 shutdown_connections 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.219+0000 7f3f1fa00640 1 -- 172.21.15.97:0/588546465 wait complete. 2024-03-06T07:31:23.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.221+0000 7f3f1fa00640 1 Processor -- start 2024-03-06T07:31:23.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.221+0000 7f3f1fa00640 1 -- start start 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1fa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 0x7f3f180b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180d4a00 0x7f3f18149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1fa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f18161b50 0x7f3f1815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1fa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180d22b0 con 0x7f3f180b5490 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1fa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180d1270 con 0x7f3f18161b50 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180d2430 con 0x7f3f180d4a00 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 0x7f3f180b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 0x7f3f180b99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39960/0 (socket says 172.21.15.97:39960) 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.222+0000 7f3f1d000640 1 -- 172.21.15.97:0/3200187997 learned_addr learned my addr 172.21.15.97:0/3200187997 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f17e00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180d4a00 0x7f3f18149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1d000640 1 -- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180d4a00 msgr2=0x7f3f18149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1d000640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180d4a00 0x7f3f18149980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1d000640 1 -- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f18161b50 msgr2=0x7f3f1815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1d000640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f18161b50 0x7f3f1815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1d000640 1 -- 172.21.15.97:0/3200187997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1814d380 con 0x7f3f180b5490 2024-03-06T07:31:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1d000640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 0x7f3f180b99e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00d620 tx=0x7f3f0c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00b620 con 0x7f3f180b5490 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c019070 con 0x7f3f180b5490 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00ba00 con 0x7f3f180b5490 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f180b8c20 con 0x7f3f180b5490 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.223+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f180b91e0 con 0x7f3f180b5490 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.225+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1804f330 con 0x7f3f180b5490 2024-03-06T07:31:23.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.226+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f0c00bba0 con 0x7f3f180b5490 2024-03-06T07:31:23.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.226+0000 7f3f14c00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ef806b4d0 0x7f3ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.227+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f0c08a620 con 0x7f3f180b5490 2024-03-06T07:31:23.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.227+0000 7f3f17e00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ef806b4d0 0x7f3ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.228+0000 7f3f17e00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ef806b4d0 0x7f3ef806d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3f04036000 tx=0x7f3f04008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.232+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f0c0594a0 con 0x7f3f180b5490 2024-03-06T07:31:23.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.428+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f3f180ca920 con 0x7f3f180b5490 2024-03-06T07:31:23.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.433+0000 7f3f14c00640 1 -- 172.21.15.97:0/3200187997 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3f0c058da0 con 0x7f3f180b5490 2024-03-06T07:31:23.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ef806b4d0 msgr2=0x7f3ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ef806b4d0 0x7f3ef806d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3f04036000 tx=0x7f3f04008030 comp rx=0 tx=0).stop 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 msgr2=0x7f3f180b99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 0x7f3f180b99e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00d620 tx=0x7f3f0c00dae0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 shutdown_connections 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ef806b4d0 0x7f3ef806d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f18161b50 0x7f3f1815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3f180d4a00 0x7f3f18149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 --2- 172.21.15.97:0/3200187997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f180b5490 0x7f3f180b99e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 >> 172.21.15.97:0/3200187997 conn(0x7f3f180d3e70 msgr2=0x7f3f1815ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 shutdown_connections 2024-03-06T07:31:23.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:23.436+0000 7f3f1fa00640 1 -- 172.21.15.97:0/3200187997 wait complete. 2024-03-06T07:31:33.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:33.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.671+0000 7f7710000640 1 Processor -- start 2024-03-06T07:31:33.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.672+0000 7f7710000640 1 -- start start 2024-03-06T07:31:33.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.673+0000 7f7710000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77080c4c40 0x7f77080c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.673+0000 7f7710000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b4650 con 0x7f77080b0670 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.673+0000 7f7710000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7708166e50 con 0x7f7708162a40 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.673+0000 7f7710000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080af1d0 con 0x7f77080c4c40 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.673+0000 7f770d600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77080c4c40 0x7f77080c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.674+0000 7f770d600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77080c4c40 0x7f77080c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42296/0 (socket says 172.21.15.97:42296) 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.674+0000 7f770d600640 1 -- 172.21.15.97:0/4015690611 learned_addr learned my addr 172.21.15.97:0/4015690611 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.674+0000 7f770e000640 1 --1- 172.21.15.97:0/4015690611 >> v1:172.21.15.97:6789/0 conn(0x7f77080b0670 0x7f7708166740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54570/0 (socket says 172.21.15.97:54570) 2024-03-06T07:31:33.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.674+0000 7f770d600640 1 -- 172.21.15.97:0/4015690611 >> v1:172.21.15.135:6789/0 conn(0x7f7708162a40 legacy=0x7f7708166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.674+0000 7f770d600640 1 -- 172.21.15.97:0/4015690611 >> v1:172.21.15.97:6789/0 conn(0x7f77080b0670 legacy=0x7f7708166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.674+0000 7f770d600640 1 -- 172.21.15.97:0/4015690611 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080c5560 con 0x7f77080c4c40 2024-03-06T07:31:33.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.676+0000 7f770d600640 1 --2- 172.21.15.97:0/4015690611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77080c4c40 0x7f77080c5020 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009870 tx=0x7f76fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e929bcd5c1b59784 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.676+0000 7f7707e00640 1 -- 172.21.15.97:0/4015690611 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc034070 con 0x7f77080c4c40 2024-03-06T07:31:33.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.676+0000 7f7707e00640 1 -- 172.21.15.97:0/4015690611 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc030860 con 0x7f77080c4c40 2024-03-06T07:31:33.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.676+0000 7f7707e00640 1 -- 172.21.15.97:0/4015690611 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc030c30 con 0x7f77080c4c40 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 -- 172.21.15.97:0/4015690611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77080c4c40 msgr2=0x7f77080c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 --2- 172.21.15.97:0/4015690611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77080c4c40 0x7f77080c5020 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009870 tx=0x7f76fc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 -- 172.21.15.97:0/4015690611 shutdown_connections 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 --2- 172.21.15.97:0/4015690611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77080c4c40 0x7f77080c5020 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 -- 172.21.15.97:0/4015690611 >> 172.21.15.97:0/4015690611 conn(0x7f77080b3800 msgr2=0x7f77080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 -- 172.21.15.97:0/4015690611 shutdown_connections 2024-03-06T07:31:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.677+0000 7f7710000640 1 -- 172.21.15.97:0/4015690611 wait complete. 2024-03-06T07:31:33.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.679+0000 7f7710000640 1 Processor -- start 2024-03-06T07:31:33.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.680+0000 7f7710000640 1 -- start start 2024-03-06T07:31:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f7710000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77080b0670 0x7f77080cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f7710000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77080c4c40 0x7f77080cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f7710000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 0x7f77080b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f7710000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080aeb80 con 0x7f7708162a40 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f7710000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7708163970 con 0x7f77080b0670 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f7710000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080aee60 con 0x7f77080c4c40 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.681+0000 7f770e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 0x7f77080b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 0x7f77080b4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42298/0 (socket says 172.21.15.97:42298) 2024-03-06T07:31:33.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 -- 172.21.15.97:0/2794516149 learned_addr learned my addr 172.21.15.97:0/2794516149 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:33.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 -- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77080c4c40 msgr2=0x7f77080cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77080c4c40 0x7f77080cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 -- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77080b0670 msgr2=0x7f77080cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770d600640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77080b0670 0x7f77080cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77080b0670 0x7f77080cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 -- 172.21.15.97:0/2794516149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080ce4a0 con 0x7f7708162a40 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f770e000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 0x7f77080b4a40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f770000c7a0 tx=0x7f770000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.682+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700017070 con 0x7f7708162a40 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.683+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770000a640 con 0x7f7708162a40 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.683+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77080ce780 con 0x7f7708162a40 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.683+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770000a9f0 con 0x7f7708162a40 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.683+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77080cf020 con 0x7f7708162a40 2024-03-06T07:31:33.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.684+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770804f330 con 0x7f7708162a40 2024-03-06T07:31:33.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.685+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f770001a070 con 0x7f7708162a40 2024-03-06T07:31:33.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.686+0000 7f7705600640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f76e806b4d0 0x7f76e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.686+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f77000897c0 con 0x7f7708162a40 2024-03-06T07:31:33.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.686+0000 7f770d600640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f76e806b4d0 0x7f76e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.687+0000 7f770d600640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f76e806b4d0 0x7f76e806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f76fc036000 tx=0x7f76fc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.695+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77080b4650 con 0x7f7708162a40 2024-03-06T07:31:33.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.938+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7708067af0 con 0x7f7708162a40 2024-03-06T07:31:33.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.939+0000 7f7705600640 1 -- 172.21.15.97:0/2794516149 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f7700057fa0 con 0x7f7708162a40 2024-03-06T07:31:33.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:33.941 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:33.941 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f76e806b4d0 msgr2=0x7f76e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f76e806b4d0 0x7f76e806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f76fc036000 tx=0x7f76fc03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 msgr2=0x7f77080b4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 0x7f77080b4a40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f770000c7a0 tx=0x7f770000cc60 comp rx=0 tx=0).stop 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 shutdown_connections 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f76e806b4d0 0x7f76e806d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7708162a40 0x7f77080b4a40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77080c4c40 0x7f77080cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 --2- 172.21.15.97:0/2794516149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77080b0670 0x7f77080cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 >> 172.21.15.97:0/2794516149 conn(0x7f77080b3800 msgr2=0x7f77080b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.941+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 shutdown_connections 2024-03-06T07:31:33.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:33.942+0000 7f7710000640 1 -- 172.21.15.97:0/2794516149 wait complete. 2024-03-06T07:31:33.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:34.192 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:31:34.192 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:31:34.192 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:34.192 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> import uuid 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> n = 100 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = False 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> if 0 > 0: 2024-03-06T07:31:34.241 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = True 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> dpath = os.path.dirname(path) 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> try: 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(n): 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> f.write(f"{i}") 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> f.flush() 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(f.fileno()) 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> os.unlink(fpath) 2024-03-06T07:31:34.242 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> if create_hard_links: 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> for j in range(0): 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> finally: 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> os.close(dirfd) 2024-03-06T07:31:34.243 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:31:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:Traceback (most recent call last): 2024-03-06T07:31:34.491 INFO:teuthology.orchestra.run.smithi097.stderr: File "", line 26, in 2024-03-06T07:31:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:31:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:31:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:34.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:34.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.749+0000 7fa7bf200640 1 Processor -- start 2024-03-06T07:31:34.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.749+0000 7fa7bf200640 1 -- start start 2024-03-06T07:31:34.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7bf200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa7c01561f0 0x7fa7c00ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7bf200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c015c240 con 0x7fa7c00cc9d0 2024-03-06T07:31:34.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7bf200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c015c500 con 0x7fa7c00aebf0 2024-03-06T07:31:34.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7bf200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c015ce30 con 0x7fa7c01561f0 2024-03-06T07:31:34.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7be800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa7c00cc9d0 0x7fa7c015f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57290/0 (socket says 172.21.15.97:57290) 2024-03-06T07:31:34.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7be800640 1 -- 172.21.15.97:0/898039755 learned_addr learned my addr 172.21.15.97:0/898039755 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:34.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.751+0000 7fa7bde00640 1 --2- 172.21.15.97:0/898039755 >> v2:172.21.15.135:3300/0 conn(0x7fa7c01561f0 0x7fa7c00ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511515034 0 0) 0x7fa7c015c240 con 0x7fa7c00cc9d0 2024-03-06T07:31:34.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a4003520 con 0x7fa7c00cc9d0 2024-03-06T07:31:34.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290108161 0 0) 0x7fa7a4003520 con 0x7fa7c00cc9d0 2024-03-06T07:31:34.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 >> v2:172.21.15.135:3300/0 conn(0x7fa7c01561f0 msgr2=0x7fa7c00ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 --2- 172.21.15.97:0/898039755 >> v2:172.21.15.135:3300/0 conn(0x7fa7c01561f0 0x7fa7c00ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 >> v1:172.21.15.135:6789/0 conn(0x7fa7c00aebf0 legacy=0x7fa7c00cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c015c7e0 con 0x7fa7c00cc9d0 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa7b80019f0 con 0x7fa7c00cc9d0 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 >> v1:172.21.15.97:6789/0 conn(0x7fa7c00cc9d0 legacy=0x7fa7c015f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4003a00 0x7fa7a4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a4004800 0x7fa7a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a40051c0 0x7fa7a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c015ce30 con 0x7fa7a40051c0 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a400c0c0 con 0x7fa7a4003a00 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.752+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a400c2c0 con 0x7fa7a4004800 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7be800640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a40051c0 0x7fa7a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7bde00640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4003a00 0x7fa7a4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7be800640 1 -- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a4004800 msgr2=0x7fa7a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7be800640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a4004800 0x7fa7a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7be800640 1 -- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4003a00 msgr2=0x7fa7a4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7be800640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4003a00 0x7fa7a4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.753+0000 7fa7be800640 1 -- 172.21.15.97:0/898039755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4003520 con 0x7fa7a40051c0 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.754+0000 7fa7be800640 1 -- 172.21.15.97:0/898039755 reap_dead start 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.754+0000 7fa7be800640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a40051c0 0x7fa7a400b9c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800a790 tx=0x7fa7a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcebbbb33d6150 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.754+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b800a3b0 con 0x7fa7a40051c0 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.754+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b8034670 con 0x7fa7a40051c0 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.754+0000 7fa7bca00640 1 -- 172.21.15.97:0/898039755 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8034a60 con 0x7fa7a40051c0 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 -- 172.21.15.97:0/898039755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a40051c0 msgr2=0x7fa7a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a40051c0 0x7fa7a400b9c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800a790 tx=0x7fa7a4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 -- 172.21.15.97:0/898039755 shutdown_connections 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 --2- 172.21.15.97:0/898039755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a40051c0 0x7fa7a400b9c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 -- 172.21.15.97:0/898039755 >> 172.21.15.97:0/898039755 conn(0x7fa7c006afc0 msgr2=0x7fa7c006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 -- 172.21.15.97:0/898039755 shutdown_connections 2024-03-06T07:31:34.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.755+0000 7fa7bf200640 1 -- 172.21.15.97:0/898039755 wait complete. 2024-03-06T07:31:34.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.758+0000 7fa7bf200640 1 Processor -- start 2024-03-06T07:31:34.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.758+0000 7fa7bf200640 1 -- start start 2024-03-06T07:31:34.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40051c0 0x7fa7c00cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 0x7fa7c01591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7c00ad860 0x7fa7c00adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bf200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0156750 con 0x7fa7c01e6a20 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bf200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01565d0 con 0x7fa7a40051c0 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bf200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c00ae6b0 con 0x7fa7c00ad860 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 0x7fa7c01591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 0x7fa7c01591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34744/0 (socket says 172.21.15.97:34744) 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bd400640 1 -- 172.21.15.97:0/3527665147 learned_addr learned my addr 172.21.15.97:0/3527665147 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.760+0000 7fa7bde00640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40051c0 0x7fa7c00cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bd400640 1 -- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7c00ad860 msgr2=0x7fa7c00adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bd400640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7c00ad860 0x7fa7c00adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bd400640 1 -- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40051c0 msgr2=0x7fa7c00cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bd400640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40051c0 0x7fa7c00cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bd400640 1 -- 172.21.15.97:0/3527665147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0156c70 con 0x7fa7c01e6a20 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bde00640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40051c0 0x7fa7c00cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bd400640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 0x7fa7c01591c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00c500 tx=0x7fa7ac00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0052d0 con 0x7fa7c01e6a20 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac005470 con 0x7fa7c01e6a20 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac012850 con 0x7fa7c01e6a20 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c0156f50 con 0x7fa7c01e6a20 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.761+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c01589c0 con 0x7fa7c01e6a20 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.763+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c00ae220 con 0x7fa7c01e6a20 2024-03-06T07:31:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.764+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7ac0129f0 con 0x7fa7c01e6a20 2024-03-06T07:31:34.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.765+0000 7fa7b6000640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a006b500 0x7fa7a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.765+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7ac089f30 con 0x7fa7c01e6a20 2024-03-06T07:31:34.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.765+0000 7fa7bde00640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a006b500 0x7fa7a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.766+0000 7fa7bde00640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a006b500 0x7fa7a006d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009a50 tx=0x7fa7b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:34.769+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7ac058d60 con 0x7fa7c01e6a20 2024-03-06T07:31:35.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.069+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7c0159700 con 0x7fa7c01e6a20 2024-03-06T07:31:35.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.070+0000 7fa7b6000640 1 -- 172.21.15.97:0/3527665147 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fa7ac058660 con 0x7fa7c01e6a20 2024-03-06T07:31:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:35.072 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:35.073 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a006b500 msgr2=0x7fa7a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a006b500 0x7fa7a006d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009a50 tx=0x7fa7b0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 msgr2=0x7fa7c01591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 0x7fa7c01591c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00c500 tx=0x7fa7ac00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 shutdown_connections 2024-03-06T07:31:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a006b500 0x7fa7a006d9c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7c00ad860 0x7fa7c00adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7c01e6a20 0x7fa7c01591c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 --2- 172.21.15.97:0/3527665147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40051c0 0x7fa7c00cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 >> 172.21.15.97:0/3527665147 conn(0x7fa7c006afc0 msgr2=0x7fa7c0072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 shutdown_connections 2024-03-06T07:31:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.073+0000 7fa7bf200640 1 -- 172.21.15.97:0/3527665147 wait complete. 2024-03-06T07:31:35.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:35.306 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:31:35.307 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:31:35.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:35.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.508+0000 7f341cc00640 1 Processor -- start 2024-03-06T07:31:35.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.508+0000 7f341cc00640 1 -- start start 2024-03-06T07:31:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.509+0000 7f341cc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3418161580 0x7f34180d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.509+0000 7f341cc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f34180d2a70 0x7f34180b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.509+0000 7f341cc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3418161960 con 0x7f34180d1e90 2024-03-06T07:31:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.509+0000 7f341cc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180cf6f0 con 0x7f34180d2a70 2024-03-06T07:31:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.509+0000 7f341cc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180cf9d0 con 0x7f3418161580 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.509+0000 7f340fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f34180d1e90 0x7f34180d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57304/0 (socket says 172.21.15.97:57304) 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f340fe00640 1 -- 172.21.15.97:0/551786045 learned_addr learned my addr 172.21.15.97:0/551786045 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3417e00640 1 --2- 172.21.15.97:0/551786045 >> v2:172.21.15.97:3300/0 conn(0x7f34180d2a70 0x7f34180b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3417400640 1 --2- 172.21.15.97:0/551786045 >> v2:172.21.15.135:3300/0 conn(0x7f3418161580 0x7f34180d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3416a00640 1 -- 172.21.15.97:0/551786045 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704947940 0 0) 0x7f3418161960 con 0x7f34180d1e90 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3416a00640 1 -- 172.21.15.97:0/551786045 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc003520 con 0x7f34180d1e90 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3417e00640 1 -- 172.21.15.97:0/551786045 >> v2:172.21.15.135:3300/0 conn(0x7f3418161580 msgr2=0x7f34180d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3417e00640 1 --2- 172.21.15.97:0/551786045 >> v2:172.21.15.135:3300/0 conn(0x7f3418161580 0x7f34180d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3417e00640 1 -- 172.21.15.97:0/551786045 >> v1:172.21.15.97:6789/0 conn(0x7f34180d1e90 legacy=0x7f34180d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.510+0000 7f3417e00640 1 -- 172.21.15.97:0/551786045 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180cfcb0 con 0x7f34180d2a70 2024-03-06T07:31:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.512+0000 7f3417e00640 1 --2- 172.21.15.97:0/551786045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34180d2a70 0x7f34180b3ec0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3418073740 tx=0x7f341002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862df6504e56784d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.512+0000 7f3416a00640 1 -- 172.21.15.97:0/551786045 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410039070 con 0x7f34180d2a70 2024-03-06T07:31:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.512+0000 7f3416a00640 1 -- 172.21.15.97:0/551786045 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341002cc70 con 0x7f34180d2a70 2024-03-06T07:31:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.512+0000 7f3416a00640 1 -- 172.21.15.97:0/551786045 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410034640 con 0x7f34180d2a70 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.512+0000 7f341cc00640 1 -- 172.21.15.97:0/551786045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34180d2a70 msgr2=0x7f34180b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.512+0000 7f341cc00640 1 --2- 172.21.15.97:0/551786045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34180d2a70 0x7f34180b3ec0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3418073740 tx=0x7f341002bec0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.513+0000 7f341cc00640 1 -- 172.21.15.97:0/551786045 shutdown_connections 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.513+0000 7f341cc00640 1 --2- 172.21.15.97:0/551786045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34180d2a70 0x7f34180b3ec0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.513+0000 7f341cc00640 1 --2- 172.21.15.97:0/551786045 >> v2:172.21.15.135:3300/0 conn(0x7f3418161580 0x7f34180d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.513+0000 7f341cc00640 1 -- 172.21.15.97:0/551786045 >> 172.21.15.97:0/551786045 conn(0x7f34180d3d80 msgr2=0x7f34180d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.513+0000 7f341cc00640 1 -- 172.21.15.97:0/551786045 shutdown_connections 2024-03-06T07:31:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.513+0000 7f341cc00640 1 -- 172.21.15.97:0/551786045 wait complete. 2024-03-06T07:31:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.514+0000 7f341cc00640 1 Processor -- start 2024-03-06T07:31:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.514+0000 7f341cc00640 1 -- start start 2024-03-06T07:31:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.515+0000 7f341cc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34180d1e90 0x7f34180b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.515+0000 7f341cc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34180d2a70 0x7f34180c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.515+0000 7f341cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 0x7f34181f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.515+0000 7f341cc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180cf3a0 con 0x7f3418161580 2024-03-06T07:31:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.515+0000 7f341cc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b39a0 con 0x7f34180d1e90 2024-03-06T07:31:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.515+0000 7f341cc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b4400 con 0x7f34180d2a70 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 0x7f34181f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 0x7f34181f4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34764/0 (socket says 172.21.15.97:34764) 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 -- 172.21.15.97:0/1581565010 learned_addr learned my addr 172.21.15.97:0/1581565010 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f340fe00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34180d2a70 0x7f34180c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417400640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34180d1e90 0x7f34180b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 -- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34180d2a70 msgr2=0x7f34180c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34180d2a70 0x7f34180c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 -- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34180d1e90 msgr2=0x7f34180b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34180d1e90 0x7f34180b9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 -- 172.21.15.97:0/1581565010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341815d870 con 0x7f3418161580 2024-03-06T07:31:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.516+0000 7f3417e00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 0x7f34181f4cf0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3410037040 tx=0x7f3410004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.517+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410042020 con 0x7f3418161580 2024-03-06T07:31:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.517+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341003a040 con 0x7f3418161580 2024-03-06T07:31:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.517+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410039070 con 0x7f3418161580 2024-03-06T07:31:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.517+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34181f5230 con 0x7f3418161580 2024-03-06T07:31:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.517+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34180cc5e0 con 0x7f3418161580 2024-03-06T07:31:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.518+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341804f330 con 0x7f3418161580 2024-03-06T07:31:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.519+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3410045020 con 0x7f3418161580 2024-03-06T07:31:35.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.520+0000 7f340f400640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f33f406b4d0 0x7f33f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.520+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f34100ac320 con 0x7f3418161580 2024-03-06T07:31:35.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.520+0000 7f3417400640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f33f406b4d0 0x7f33f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.521+0000 7f3417400640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f33f406b4d0 0x7f33f406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.525+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f341004c070 con 0x7f3418161580 2024-03-06T07:31:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.760+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34180ce1b0 con 0x7f3418161580 2024-03-06T07:31:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.761+0000 7f340f400640 1 -- 172.21.15.97:0/1581565010 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f341002ce20 con 0x7f3418161580 2024-03-06T07:31:35.763 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:35.763 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f33f406b4d0 msgr2=0x7f33f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f33f406b4d0 0x7f33f406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408008040 comp rx=0 tx=0).stop 2024-03-06T07:31:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 msgr2=0x7f34181f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 0x7f34181f4cf0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3410037040 tx=0x7f3410004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 shutdown_connections 2024-03-06T07:31:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f33f406b4d0 0x7f33f406d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3418161580 0x7f34181f4cf0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34180d2a70 0x7f34180c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 --2- 172.21.15.97:0/1581565010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34180d1e90 0x7f34180b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 >> 172.21.15.97:0/1581565010 conn(0x7f34180d3d80 msgr2=0x7f34180d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 shutdown_connections 2024-03-06T07:31:35.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:35.764+0000 7f341cc00640 1 -- 172.21.15.97:0/1581565010 wait complete. 2024-03-06T07:31:35.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:36.003 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:31:36.004 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:31:36.004 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:31:37.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:37.736 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:41.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:41.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.218+0000 7fbf55800640 1 Processor -- start 2024-03-06T07:31:41.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.218+0000 7fbf55800640 1 -- start start 2024-03-06T07:31:41.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf55800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbf5015f730 0x7fbf500cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf55800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf500c5970 0x7fbf500c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf55800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500b3340 con 0x7fbf500b2e60 2024-03-06T07:31:41.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf55800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf500cd320 con 0x7fbf500c5970 2024-03-06T07:31:41.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf55800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf500cdbc0 con 0x7fbf5015f730 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf4f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf500c5970 0x7fbf500c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf4f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf500c5970 0x7fbf500c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34774/0 (socket says 172.21.15.97:34774) 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.220+0000 7fbf4f400640 1 -- 172.21.15.97:0/3219758622 learned_addr learned my addr 172.21.15.97:0/3219758622 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.221+0000 7fbf4ea00640 1 --2- 172.21.15.97:0/3219758622 >> v2:172.21.15.135:3300/0 conn(0x7fbf5015f730 0x7fbf500cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.221+0000 7fbf4f400640 1 -- 172.21.15.97:0/3219758622 >> v2:172.21.15.135:3300/0 conn(0x7fbf5015f730 msgr2=0x7fbf500cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.221+0000 7fbf4f400640 1 --2- 172.21.15.97:0/3219758622 >> v2:172.21.15.135:3300/0 conn(0x7fbf5015f730 0x7fbf500cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.221+0000 7fbf4f400640 1 -- 172.21.15.97:0/3219758622 >> v1:172.21.15.135:6789/0 conn(0x7fbf500b2e60 legacy=0x7fbf500cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.221+0000 7fbf4f400640 1 -- 172.21.15.97:0/3219758622 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500cd570 con 0x7fbf500c5970 2024-03-06T07:31:41.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.222+0000 7fbf4f400640 1 --2- 172.21.15.97:0/3219758622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf500c5de0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbf50071dc0 tx=0x7fbf4802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb131558d7748a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.222+0000 7fbf4e000640 1 -- 172.21.15.97:0/3219758622 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4802cb20 con 0x7fbf500c5970 2024-03-06T07:31:41.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.222+0000 7fbf4e000640 1 -- 172.21.15.97:0/3219758622 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4802ccc0 con 0x7fbf500c5970 2024-03-06T07:31:41.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.222+0000 7fbf4e000640 1 -- 172.21.15.97:0/3219758622 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48034610 con 0x7fbf500c5970 2024-03-06T07:31:41.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 -- 172.21.15.97:0/3219758622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 msgr2=0x7fbf500c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 --2- 172.21.15.97:0/3219758622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf500c5de0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbf50071dc0 tx=0x7fbf4802bf80 comp rx=0 tx=0).stop 2024-03-06T07:31:41.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 -- 172.21.15.97:0/3219758622 shutdown_connections 2024-03-06T07:31:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 --2- 172.21.15.97:0/3219758622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf500c5de0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 --2- 172.21.15.97:0/3219758622 >> v2:172.21.15.135:3300/0 conn(0x7fbf5015f730 0x7fbf500cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 -- 172.21.15.97:0/3219758622 >> 172.21.15.97:0/3219758622 conn(0x7fbf500b3800 msgr2=0x7fbf500b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 -- 172.21.15.97:0/3219758622 shutdown_connections 2024-03-06T07:31:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.223+0000 7fbf55800640 1 -- 172.21.15.97:0/3219758622 wait complete. 2024-03-06T07:31:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.225+0000 7fbf55800640 1 Processor -- start 2024-03-06T07:31:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.225+0000 7fbf55800640 1 -- start start 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf55800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf500b2e60 0x7fbf50153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf55800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf5015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf55800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf5015f730 0x7fbf5015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf55800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5015fb10 con 0x7fbf500c5970 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf55800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500d0ae0 con 0x7fbf500b2e60 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf55800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5015fc90 con 0x7fbf5015f730 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf5015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf5015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34790/0 (socket says 172.21.15.97:34790) 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.226+0000 7fbf47e00640 1 -- 172.21.15.97:0/4030528405 learned_addr learned my addr 172.21.15.97:0/4030528405 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf4ea00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf500b2e60 0x7fbf50153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf4f400640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf5015f730 0x7fbf5015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf47e00640 1 -- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf5015f730 msgr2=0x7fbf5015db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf47e00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf5015f730 0x7fbf5015db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf47e00640 1 -- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf500b2e60 msgr2=0x7fbf50153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf47e00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf500b2e60 0x7fbf50153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf47e00640 1 -- 172.21.15.97:0/4030528405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50154200 con 0x7fbf500c5970 2024-03-06T07:31:41.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf47e00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf5015d5f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c3c0 tx=0x7fbf3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.227+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380052d0 con 0x7fbf500c5970 2024-03-06T07:31:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.228+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38005490 con 0x7fbf500c5970 2024-03-06T07:31:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.228+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38007260 con 0x7fbf500c5970 2024-03-06T07:31:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.228+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf500c9a00 con 0x7fbf500c5970 2024-03-06T07:31:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.228+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf500c9fc0 con 0x7fbf500c5970 2024-03-06T07:31:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.229+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5004f330 con 0x7fbf500c5970 2024-03-06T07:31:41.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.230+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf38007400 con 0x7fbf500c5970 2024-03-06T07:31:41.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.231+0000 7fbf46a00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.231+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf380897c0 con 0x7fbf500c5970 2024-03-06T07:31:41.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.231+0000 7fbf4ea00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.232+0000 7fbf4ea00640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbf40006fd0 tx=0x7fbf40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.236+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf38058560 con 0x7fbf500c5970 2024-03-06T07:31:41.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.481+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf500ceee0 con 0x7fbf500c5970 2024-03-06T07:31:41.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.482+0000 7fbf46a00640 1 -- 172.21.15.97:0/4030528405 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fbf38057e60 con 0x7fbf500c5970 2024-03-06T07:31:41.484 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:41.485 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:41.485 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.484+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf2c06b4d0 msgr2=0x7fbf2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.484+0000 7fbf55800640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbf40006fd0 tx=0x7fbf40008040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.484+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 msgr2=0x7fbf5015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.484+0000 7fbf55800640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf5015d5f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c3c0 tx=0x7fbf3800c880 comp rx=0 tx=0).stop 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 shutdown_connections 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf5015f730 0x7fbf5015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf500c5970 0x7fbf5015d5f0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 --2- 172.21.15.97:0/4030528405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf500b2e60 0x7fbf50153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 >> 172.21.15.97:0/4030528405 conn(0x7fbf500b3800 msgr2=0x7fbf500ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 shutdown_connections 2024-03-06T07:31:41.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:41.485+0000 7fbf55800640 1 -- 172.21.15.97:0/4030528405 wait complete. 2024-03-06T07:31:41.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:41.724 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:31:41.724 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:31:41.724 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:31:46.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:46.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.992+0000 7f4a3d200640 1 Processor -- start 2024-03-06T07:31:46.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.992+0000 7f4a3d200640 1 -- start start 2024-03-06T07:31:46.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.994+0000 7f4a3d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 0x7f4a380cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.994+0000 7f4a3d200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3815c240 con 0x7f4a381561f0 2024-03-06T07:31:46.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.994+0000 7f4a3d200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3815c500 con 0x7f4a380cc800 2024-03-06T07:31:46.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.994+0000 7f4a3d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3815ce30 con 0x7f4a380aedc0 2024-03-06T07:31:46.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.994+0000 7f4a37400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4a381561f0 0x7f4a380ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33334/0 (socket says 172.21.15.97:33334) 2024-03-06T07:31:46.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.994+0000 7f4a37400640 1 -- 172.21.15.97:0/3328641052 learned_addr learned my addr 172.21.15.97:0/3328641052 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:46.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36a00640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 0x7f4a380cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128009934 0 0) 0x7f4a3815c240 con 0x7f4a381561f0 2024-03-06T07:31:46.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a1c003520 con 0x7f4a381561f0 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513515911 0 0) 0x7f4a1c003520 con 0x7f4a381561f0 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 msgr2=0x7f4a380cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 0x7f4a380cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 >> v1:172.21.15.135:6789/0 conn(0x7f4a380cc800 legacy=0x7f4a3815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3815c7e0 con 0x7f4a381561f0 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4a28001670 con 0x7f4a381561f0 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 >> v1:172.21.15.97:6789/0 conn(0x7f4a381561f0 legacy=0x7f4a380ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a1c0039e0 0x7f4a1c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a1c004800 0x7f4a1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 0x7f4a1c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3815ce30 con 0x7f4a1c004800 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c00bb50 con 0x7f4a1c0039e0 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.995+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c00bd50 con 0x7f4a380aedc0 2024-03-06T07:31:46.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a36a00640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a1c004800 0x7f4a1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a37400640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a1c0039e0 0x7f4a1c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a37e00640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 0x7f4a1c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a36a00640 1 -- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 msgr2=0x7f4a1c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a36a00640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a380aedc0 0x7f4a1c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a36a00640 1 -- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a1c0039e0 msgr2=0x7f4a1c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a36a00640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a1c0039e0 0x7f4a1c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.996+0000 7f4a36a00640 1 -- 172.21.15.97:0/3328641052 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c003520 con 0x7f4a1c004800 2024-03-06T07:31:47.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.997+0000 7f4a36a00640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a1c004800 0x7f4a1c004c80 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000ba30 tx=0x7f4a1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd94e09cbbfcb829 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.997+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20036020 con 0x7f4a1c004800 2024-03-06T07:31:47.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.997+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20032640 con 0x7f4a1c004800 2024-03-06T07:31:47.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.997+0000 7f4a36000640 1 -- 172.21.15.97:0/3328641052 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000b600 con 0x7f4a1c004800 2024-03-06T07:31:47.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a3d200640 1 -- 172.21.15.97:0/3328641052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a1c004800 msgr2=0x7f4a1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a3d200640 1 --2- 172.21.15.97:0/3328641052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a1c004800 0x7f4a1c004c80 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000ba30 tx=0x7f4a1c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a37e00640 1 -- 172.21.15.97:0/3328641052 reap_dead start 2024-03-06T07:31:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a3d200640 1 -- 172.21.15.97:0/3328641052 shutdown_connections 2024-03-06T07:31:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a3d200640 1 -- 172.21.15.97:0/3328641052 >> 172.21.15.97:0/3328641052 conn(0x7f4a3806afc0 msgr2=0x7f4a3806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a3d200640 1 -- 172.21.15.97:0/3328641052 shutdown_connections 2024-03-06T07:31:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:46.998+0000 7f4a3d200640 1 -- 172.21.15.97:0/3328641052 wait complete. 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.000+0000 7f4a3d200640 1 Processor -- start 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.000+0000 7f4a3d200640 1 -- start start 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a3d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a1c004800 0x7f4a38151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a3d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 0x7f4a38152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a3d200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a38150a10 0x7f4a380b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a3d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3806e360 con 0x7f4a381537b0 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a3d200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380ccc90 con 0x7f4a38150a10 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a3d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a381565d0 con 0x7f4a1c004800 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 0x7f4a38152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 0x7f4a38152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54066/0 (socket says 172.21.15.97:54066) 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a36a00640 1 -- 172.21.15.97:0/2794431102 learned_addr learned my addr 172.21.15.97:0/2794431102 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a37400640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a1c004800 0x7f4a38151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a37e00640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a38150a10 0x7f4a380b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a36a00640 1 -- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a1c004800 msgr2=0x7f4a38151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.001+0000 7f4a36a00640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a1c004800 0x7f4a38151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a36a00640 1 -- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a38150a10 msgr2=0x7f4a380b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a36a00640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a38150a10 0x7f4a380b0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a36a00640 1 -- 172.21.15.97:0/2794431102 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a38153ce0 con 0x7f4a381537b0 2024-03-06T07:31:47.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a36a00640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 0x7f4a38152900 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000bb20 tx=0x7f4a20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20036040 con 0x7f4a381537b0 2024-03-06T07:31:47.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20030a80 con 0x7f4a381537b0 2024-03-06T07:31:47.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a38151e20 con 0x7f4a381537b0 2024-03-06T07:31:47.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000b600 con 0x7f4a381537b0 2024-03-06T07:31:47.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.002+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a38150e90 con 0x7f4a381537b0 2024-03-06T07:31:47.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.003+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3804f330 con 0x7f4a381537b0 2024-03-06T07:31:47.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.004+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a20039020 con 0x7f4a381537b0 2024-03-06T07:31:47.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.005+0000 7f4a2f400640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a1406b500 0x7f4a1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.005+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a200ace20 con 0x7f4a381537b0 2024-03-06T07:31:47.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.005+0000 7f4a37400640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a1406b500 0x7f4a1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.006+0000 7f4a37400640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a1406b500 0x7f4a1406d9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4a28001050 tx=0x7f4a28009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.010+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a20030e70 con 0x7f4a381537b0 2024-03-06T07:31:47.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.243+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3806b3a0 con 0x7f4a381537b0 2024-03-06T07:31:47.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.244+0000 7f4a2f400640 1 -- 172.21.15.97:0/2794431102 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f4a2007bb30 con 0x7f4a381537b0 2024-03-06T07:31:47.246 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:47.247 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:47.247 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.246+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a1406b500 msgr2=0x7f4a1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.246+0000 7f4a3d200640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a1406b500 0x7f4a1406d9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4a28001050 tx=0x7f4a28009040 comp rx=0 tx=0).stop 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.246+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 msgr2=0x7f4a38152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.246+0000 7f4a3d200640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 0x7f4a38152900 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000bb20 tx=0x7f4a20004730 comp rx=0 tx=0).stop 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 shutdown_connections 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a1406b500 0x7f4a1406d9c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a38150a10 0x7f4a380b0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a381537b0 0x7f4a38152900 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 --2- 172.21.15.97:0/2794431102 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a1c004800 0x7f4a38151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 >> 172.21.15.97:0/2794431102 conn(0x7f4a3806afc0 msgr2=0x7f4a380ada10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 shutdown_connections 2024-03-06T07:31:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.247+0000 7f4a3d200640 1 -- 172.21.15.97:0/2794431102 wait complete. 2024-03-06T07:31:47.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:47.474 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:31:47.474 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:31:47.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.673+0000 7fbabee00640 1 Processor -- start 2024-03-06T07:31:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.673+0000 7fbabee00640 1 -- start start 2024-03-06T07:31:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabee00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbab806c2e0 0x7fbab815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabee00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab80d0010 con 0x7fbab80cdd80 2024-03-06T07:31:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabee00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab80d0ec0 con 0x7fbab806c2e0 2024-03-06T07:31:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80d02b0 con 0x7fbab806d1f0 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabce00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbab806c2e0 0x7fbab815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabce00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbab806c2e0 0x7fbab815d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54082/0 (socket says 172.21.15.97:54082) 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbabce00640 1 -- 172.21.15.97:0/3962013142 learned_addr learned my addr 172.21.15.97:0/3962013142 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.675+0000 7fbaaf400640 1 --2- 172.21.15.97:0/3962013142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.676+0000 7fbabce00640 1 -- 172.21.15.97:0/3962013142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 msgr2=0x7fbab81627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.676+0000 7fbabce00640 1 --2- 172.21.15.97:0/3962013142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.676+0000 7fbabce00640 1 -- 172.21.15.97:0/3962013142 >> v1:172.21.15.135:6789/0 conn(0x7fbab80cdd80 legacy=0x7fbab806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.676+0000 7fbabce00640 1 -- 172.21.15.97:0/3962013142 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab80d0590 con 0x7fbab806c2e0 2024-03-06T07:31:47.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.677+0000 7fbabce00640 1 --2- 172.21.15.97:0/3962013142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab815d9b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009870 tx=0x7fbab002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202a798a0879642f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.677+0000 7fbab7400640 1 -- 172.21.15.97:0/3962013142 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0035020 con 0x7fbab806c2e0 2024-03-06T07:31:47.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.677+0000 7fbab7400640 1 -- 172.21.15.97:0/3962013142 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab0031640 con 0x7fbab806c2e0 2024-03-06T07:31:47.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.677+0000 7fbab7400640 1 -- 172.21.15.97:0/3962013142 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab002d040 con 0x7fbab806c2e0 2024-03-06T07:31:47.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 -- 172.21.15.97:0/3962013142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 msgr2=0x7fbab815d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 --2- 172.21.15.97:0/3962013142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab815d9b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009870 tx=0x7fbab002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 -- 172.21.15.97:0/3962013142 shutdown_connections 2024-03-06T07:31:47.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 --2- 172.21.15.97:0/3962013142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab815d9b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 --2- 172.21.15.97:0/3962013142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 -- 172.21.15.97:0/3962013142 >> 172.21.15.97:0/3962013142 conn(0x7fbab80b3870 msgr2=0x7fbab80b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 -- 172.21.15.97:0/3962013142 shutdown_connections 2024-03-06T07:31:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.678+0000 7fbabee00640 1 -- 172.21.15.97:0/3962013142 wait complete. 2024-03-06T07:31:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.680+0000 7fbabee00640 1 Processor -- start 2024-03-06T07:31:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.680+0000 7fbabee00640 1 -- start start 2024-03-06T07:31:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab81e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab80cdd80 0x7fbab80c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80ce160 con 0x7fbab806c2e0 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80d0b50 con 0x7fbab80cdd80 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80d1350 con 0x7fbab806d1f0 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab81e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab81e80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54096/0 (socket says 172.21.15.97:54096) 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 -- 172.21.15.97:0/4277595497 learned_addr learned my addr 172.21.15.97:0/4277595497 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbaaf400640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbabce00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab80cdd80 0x7fbab80c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 -- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 msgr2=0x7fbab81f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81f50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 -- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab80cdd80 msgr2=0x7fbab80c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab80cdd80 0x7fbab80c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.681+0000 7fbab7e00640 1 -- 172.21.15.97:0/4277595497 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab80c5970 con 0x7fbab806c2e0 2024-03-06T07:31:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.682+0000 7fbab7e00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab81e80d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800ba50 tx=0x7fbaa800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.682+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8019070 con 0x7fbab806c2e0 2024-03-06T07:31:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.682+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa800cc20 con 0x7fbab806c2e0 2024-03-06T07:31:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.682+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8014440 con 0x7fbab806c2e0 2024-03-06T07:31:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.682+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab80c5c50 con 0x7fbab806c2e0 2024-03-06T07:31:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.682+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab80c6210 con 0x7fbab806c2e0 2024-03-06T07:31:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.684+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab806bbc0 con 0x7fbab806c2e0 2024-03-06T07:31:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.684+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaa801c070 con 0x7fbab806c2e0 2024-03-06T07:31:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.685+0000 7fbab4c00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba9406b4d0 0x7fba9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.685+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbaa8089f80 con 0x7fbab806c2e0 2024-03-06T07:31:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.685+0000 7fbaaf400640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba9406b4d0 0x7fba9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.686+0000 7fbaaf400640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba9406b4d0 0x7fba9406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbab806cf50 tx=0x7fbaa0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.690+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaa80586d0 con 0x7fbab806c2e0 2024-03-06T07:31:47.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.909+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab804f330 con 0x7fbab806c2e0 2024-03-06T07:31:47.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.910+0000 7fbab4c00640 1 -- 172.21.15.97:0/4277595497 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fbaa805c770 con 0x7fbab806c2e0 2024-03-06T07:31:47.912 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:47.912 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:47.913 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.912+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba9406b4d0 msgr2=0x7fba9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.912+0000 7fbabee00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba9406b4d0 0x7fba9406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbab806cf50 tx=0x7fbaa0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.912+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 msgr2=0x7fbab81e80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.912+0000 7fbabee00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab81e80d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800ba50 tx=0x7fbaa800bf10 comp rx=0 tx=0).stop 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 shutdown_connections 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fba9406b4d0 0x7fba9406d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab80cdd80 0x7fbab80c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbab806d1f0 0x7fbab81f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 --2- 172.21.15.97:0/4277595497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbab806c2e0 0x7fbab81e80d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 >> 172.21.15.97:0/4277595497 conn(0x7fbab80b3870 msgr2=0x7fbab80b6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 shutdown_connections 2024-03-06T07:31:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:47.913+0000 7fbabee00640 1 -- 172.21.15.97:0/4277595497 wait complete. 2024-03-06T07:31:47.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:31:48.152 INFO:teuthology.orchestra.run.smithi097.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:31:48.152 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:31:48.153 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:48.153 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> import uuid 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> n = 10 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = False 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> if 0 > 0: 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> create_hard_links = True 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> dpath = os.path.dirname(path) 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:48.204 DEBUG:teuthology.orchestra.run.smithi097:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> try: 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(n): 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> f.write(f"{i}") 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> f.flush() 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(f.fileno()) 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> os.unlink(fpath) 2024-03-06T07:31:48.205 DEBUG:teuthology.orchestra.run.smithi097:> if True: 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> if create_hard_links: 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> for j in range(0): 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> if False: 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> os.fsync(dirfd) 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> finally: 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> os.close(dirfd) 2024-03-06T07:31:48.206 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:31:48.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:31:48.510 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:48.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:48.673 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:48.790 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:48.791 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:48.791 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:48.843 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:48.843 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:48.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:49.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.148+0000 7f46fd400640 1 Processor -- start 2024-03-06T07:31:49.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.148+0000 7f46fd400640 1 -- start start 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46fd400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46f815e200 0x7f46f80d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46fd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f80d0c00 0x7f46f80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46fd400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80b5240 con 0x7f46f80b2e50 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46fd400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80d2ec0 con 0x7f46f815e200 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46fd400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80cf4d0 con 0x7f46f80d0c00 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46f7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46f815e200 0x7f46f80d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46f7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46f815e200 0x7f46f80d4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54102/0 (socket says 172.21.15.97:54102) 2024-03-06T07:31:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46f7e00640 1 -- 172.21.15.97:0/2105061980 learned_addr learned my addr 172.21.15.97:0/2105061980 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46fca00640 1 --2- 172.21.15.97:0/2105061980 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f80d0c00 0x7f46f80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46f7e00640 1 -- 172.21.15.97:0/2105061980 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f80d0c00 msgr2=0x7f46f80b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.150+0000 7f46f7e00640 1 --2- 172.21.15.97:0/2105061980 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f80d0c00 0x7f46f80b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.151+0000 7f46f7e00640 1 -- 172.21.15.97:0/2105061980 >> v1:172.21.15.97:6789/0 conn(0x7f46f80b2e50 legacy=0x7f46f80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.151+0000 7f46f7e00640 1 -- 172.21.15.97:0/2105061980 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80b5500 con 0x7f46f815e200 2024-03-06T07:31:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.151+0000 7f46f6a00640 1 -- 172.21.15.97:0/2105061980 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616225360 0 0) 0x7f46f80b5240 con 0x7f46f80b2e50 2024-03-06T07:31:49.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.152+0000 7f46f7e00640 1 --2- 172.21.15.97:0/2105061980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f815e200 0x7f46f80d4850 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009870 tx=0x7f46e802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593937ef922b1bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.152+0000 7f46f6a00640 1 -- 172.21.15.97:0/2105061980 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e802f830 con 0x7f46f815e200 2024-03-06T07:31:49.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.152+0000 7f46f6a00640 1 -- 172.21.15.97:0/2105061980 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e802f9d0 con 0x7f46f815e200 2024-03-06T07:31:49.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.152+0000 7f46f6a00640 1 -- 172.21.15.97:0/2105061980 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e802fda0 con 0x7f46f815e200 2024-03-06T07:31:49.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 -- 172.21.15.97:0/2105061980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f815e200 msgr2=0x7f46f80d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 --2- 172.21.15.97:0/2105061980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f815e200 0x7f46f80d4850 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009870 tx=0x7f46e802ba10 comp rx=0 tx=0).stop 2024-03-06T07:31:49.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 -- 172.21.15.97:0/2105061980 shutdown_connections 2024-03-06T07:31:49.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 --2- 172.21.15.97:0/2105061980 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f80d0c00 0x7f46f80b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 --2- 172.21.15.97:0/2105061980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f815e200 0x7f46f80d4850 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 -- 172.21.15.97:0/2105061980 >> 172.21.15.97:0/2105061980 conn(0x7f46f80d3d30 msgr2=0x7f46f80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 -- 172.21.15.97:0/2105061980 shutdown_connections 2024-03-06T07:31:49.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.153+0000 7f46fd400640 1 -- 172.21.15.97:0/2105061980 wait complete. 2024-03-06T07:31:49.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.155+0000 7f46fd400640 1 Processor -- start 2024-03-06T07:31:49.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.155+0000 7f46fd400640 1 -- start start 2024-03-06T07:31:49.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46fd400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f80b2e50 0x7f46f8153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46fd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 0x7f46f80b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46fd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f815e200 0x7f46f80b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46fd400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80cfd60 con 0x7f46f80d0c00 2024-03-06T07:31:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46fd400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80cfa80 con 0x7f46f80b2e50 2024-03-06T07:31:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46fd400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f815e5e0 con 0x7f46f815e200 2024-03-06T07:31:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 0x7f46f80b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 0x7f46f80b82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54106/0 (socket says 172.21.15.97:54106) 2024-03-06T07:31:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.156+0000 7f46f7400640 1 -- 172.21.15.97:0/653553927 learned_addr learned my addr 172.21.15.97:0/653553927 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:49.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46fca00640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f815e200 0x7f46f80b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46f7400640 1 -- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f815e200 msgr2=0x7f46f80b87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46f7400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f815e200 0x7f46f80b87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46f7400640 1 -- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f80b2e50 msgr2=0x7f46f8153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46f7400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f80b2e50 0x7f46f8153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46f7400640 1 -- 172.21.15.97:0/653553927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80b7cc0 con 0x7f46f80d0c00 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46f7400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 0x7f46f80b82a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f46e400c550 tx=0x7f46e400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.157+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4017070 con 0x7f46f80d0c00 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.158+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e40035f0 con 0x7f46f80d0c00 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.158+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f80b7700 con 0x7f46f80d0c00 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.158+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e40039e0 con 0x7f46f80d0c00 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.158+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f80c7900 con 0x7f46f80d0c00 2024-03-06T07:31:49.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.159+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f80d3900 con 0x7f46f80d0c00 2024-03-06T07:31:49.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.161+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46e401a070 con 0x7f46f80d0c00 2024-03-06T07:31:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.162+0000 7f46efe00640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46d806b4d0 0x7f46d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.162+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f46e4089670 con 0x7f46f80d0c00 2024-03-06T07:31:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.162+0000 7f46f7e00640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46d806b4d0 0x7f46d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.163+0000 7f46f7e00640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46d806b4d0 0x7f46d806d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f46e8036000 tx=0x7f46e8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.166+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46f80d3900 con 0x7f46f80d0c00 2024-03-06T07:31:49.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.432+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46f808d4c0 con 0x7f46f80d0c00 2024-03-06T07:31:49.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.434+0000 7f46efe00640 1 -- 172.21.15.97:0/653553927 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f46e40584a0 con 0x7f46f80d0c00 2024-03-06T07:31:49.437 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:49.437 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":71,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:28.452973+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.30","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:49.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46d806b4d0 msgr2=0x7f46d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46d806b4d0 0x7f46d806d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f46e8036000 tx=0x7f46e8039040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 msgr2=0x7f46f80b82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 0x7f46f80b82a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f46e400c550 tx=0x7f46e400ca10 comp rx=0 tx=0).stop 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 shutdown_connections 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46d806b4d0 0x7f46d806d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f815e200 0x7f46f80b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f80d0c00 0x7f46f80b82a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 --2- 172.21.15.97:0/653553927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f80b2e50 0x7f46f8153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 >> 172.21.15.97:0/653553927 conn(0x7f46f80d3d30 msgr2=0x7f46f80b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 shutdown_connections 2024-03-06T07:31:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.438+0000 7f46fd400640 1 -- 172.21.15.97:0/653553927 wait complete. 2024-03-06T07:31:49.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:06.960932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:08.155371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:49.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.691+0000 7f7e85400640 1 Processor -- start 2024-03-06T07:31:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.691+0000 7f7e85400640 1 -- start start 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.693+0000 7f7e85400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 0x7f7e800ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e85400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7e8006e510 0x7f7e8006c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e85400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e800cef20 con 0x7f7e80072840 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e85400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e800b4bf0 con 0x7f7e8006e510 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e85400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8015ddd0 con 0x7f7e8015d9f0 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e77e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7e80072840 0x7f7e8006de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33352/0 (socket says 172.21.15.97:33352) 2024-03-06T07:31:49.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e77e00640 1 -- 172.21.15.97:0/235199950 learned_addr learned my addr 172.21.15.97:0/235199950 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e84a00640 1 --2- 172.21.15.97:0/235199950 >> v2:172.21.15.135:3300/0 conn(0x7f7e8006e510 0x7f7e8006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e7fe00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 0x7f7e800ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.694+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444900850 0 0) 0x7f7e68001110 con 0x7f7e80072840 2024-03-06T07:31:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e64003520 con 0x7f7e80072840 2024-03-06T07:31:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059737504 0 0) 0x7f7e64003520 con 0x7f7e80072840 2024-03-06T07:31:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 msgr2=0x7f7e800ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 0x7f7e800ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 >> v2:172.21.15.135:3300/0 conn(0x7f7e8006e510 msgr2=0x7f7e8006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 --2- 172.21.15.97:0/235199950 >> v2:172.21.15.135:3300/0 conn(0x7f7e8006e510 0x7f7e8006c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800b3e70 con 0x7f7e80072840 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7e68001840 con 0x7f7e80072840 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 >> v1:172.21.15.97:6789/0 conn(0x7f7e80072840 legacy=0x7f7e8006de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 0x7f7e640039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e64003fa0 0x7f7e64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800b4bf0 con 0x7f7e64004d40 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8015ddd0 con 0x7f7e64003fa0 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6400b9f0 con 0x7f7e8015d9f0 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.695+0000 7f7e84a00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e77e00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e64003fa0 0x7f7e64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e7fe00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 0x7f7e640039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e84a00640 1 -- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 msgr2=0x7f7e640039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e84a00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8015d9f0 0x7f7e640039d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e84a00640 1 -- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e64003fa0 msgr2=0x7f7e64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e84a00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e64003fa0 0x7f7e64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e84a00640 1 -- 172.21.15.97:0/235199950 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64003520 con 0x7f7e64004d40 2024-03-06T07:31:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.696+0000 7f7e77e00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e64003fa0 0x7f7e64004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.697+0000 7f7e84a00640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009f90 tx=0x7f7e64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7d35cf1abcd46f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.697+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78039070 con 0x7f7e64004d40 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.697+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7802cc20 con 0x7f7e64004d40 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.697+0000 7f7e7f400640 1 -- 172.21.15.97:0/235199950 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78009c10 con 0x7f7e64004d40 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e85400640 1 -- 172.21.15.97:0/235199950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 msgr2=0x7f7e640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e85400640 1 --2- 172.21.15.97:0/235199950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009f90 tx=0x7f7e64005830 comp rx=0 tx=0).stop 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e84a00640 1 -- 172.21.15.97:0/235199950 reap_dead start 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e85400640 1 -- 172.21.15.97:0/235199950 shutdown_connections 2024-03-06T07:31:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e85400640 1 -- 172.21.15.97:0/235199950 >> 172.21.15.97:0/235199950 conn(0x7f7e8006afc0 msgr2=0x7f7e8006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e85400640 1 -- 172.21.15.97:0/235199950 shutdown_connections 2024-03-06T07:31:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.698+0000 7f7e85400640 1 -- 172.21.15.97:0/235199950 wait complete. 2024-03-06T07:31:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.700+0000 7f7e85400640 1 Processor -- start 2024-03-06T07:31:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.700+0000 7f7e85400640 1 -- start start 2024-03-06T07:31:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e85400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e8014a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e85400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8014dc00 0x7f7e8013bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e85400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e8013c0e0 0x7f7e8014cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e85400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800b43f0 con 0x7f7e64004d40 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e85400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800b4110 con 0x7f7e8013c0e0 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e85400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e80070970 con 0x7f7e8014dc00 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e8014a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e8014a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54136/0 (socket says 172.21.15.97:54136) 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 -- 172.21.15.97:0/2576030785 learned_addr learned my addr 172.21.15.97:0/2576030785 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 -- 172.21.15.97:0/2576030785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8014dc00 msgr2=0x7f7e8013bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8014dc00 0x7f7e8013bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 -- 172.21.15.97:0/2576030785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e8013c0e0 msgr2=0x7f7e8014cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e8013c0e0 0x7f7e8014cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.701+0000 7f7e7fe00640 1 -- 172.21.15.97:0/2576030785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8014ad20 con 0x7f7e64004d40 2024-03-06T07:31:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.702+0000 7f7e7fe00640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e8014a7e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7e70010840 tx=0x7f7e70010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.702+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000f040 con 0x7f7e64004d40 2024-03-06T07:31:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.702+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7000eb00 con 0x7f7e64004d40 2024-03-06T07:31:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.702+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7001c400 con 0x7f7e64004d40 2024-03-06T07:31:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.702+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e80148930 con 0x7f7e64004d40 2024-03-06T07:31:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.702+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e80148ef0 con 0x7f7e64004d40 2024-03-06T07:31:49.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.703+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8004f330 con 0x7f7e64004d40 2024-03-06T07:31:49.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.704+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e700253f0 con 0x7f7e64004d40 2024-03-06T07:31:49.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.705+0000 7f7e7cc00640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.705+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e7008a870 con 0x7f7e64004d40 2024-03-06T07:31:49.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.706+0000 7f7e77e00640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.706+0000 7f7e77e00640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7e8014e7c0 tx=0x7f7e68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.710+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e70018030 con 0x7f7e64004d40 2024-03-06T07:31:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.917+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e8006e2d0 con 0x7f7e64004d40 2024-03-06T07:31:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.918+0000 7f7e7cc00640 1 -- 172.21.15.97:0/2576030785 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e700596b0 con 0x7f7e64004d40 2024-03-06T07:31:49.920 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5c06b4d0 msgr2=0x7f7e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7e8014e7c0 tx=0x7f7e68009040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 msgr2=0x7f7e8014a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e8014a7e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7e70010840 tx=0x7f7e70010d00 comp rx=0 tx=0).stop 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 shutdown_connections 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.920+0000 7f7e85400640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e8013c0e0 0x7f7e8014cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.921+0000 7f7e85400640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e8014dc00 0x7f7e8013bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.921+0000 7f7e85400640 1 --2- 172.21.15.97:0/2576030785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e64004d40 0x7f7e8014a7e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.921+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 >> 172.21.15.97:0/2576030785 conn(0x7f7e8006afc0 msgr2=0x7f7e8006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.921+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 shutdown_connections 2024-03-06T07:31:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:49.921+0000 7f7e85400640 1 -- 172.21.15.97:0/2576030785 wait complete. 2024-03-06T07:31:49.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:50.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.137+0000 7fd89b400640 1 Processor -- start 2024-03-06T07:31:50.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.138+0000 7fd89b400640 1 -- start start 2024-03-06T07:31:50.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.139+0000 7fd89b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8940b2c40 0x7fd89415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.139+0000 7fd89b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 0x7fd8940b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd89b400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940b3560 con 0x7fd89406e2f0 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd89b400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89415f700 con 0x7fd8940b2c40 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd89b400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406d4d0 con 0x7fd89406d960 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8940b2c40 0x7fd89415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8940b2c40 0x7fd89415f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54148/0 (socket says 172.21.15.97:54148) 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 -- 172.21.15.97:0/15240475 learned_addr learned my addr 172.21.15.97:0/15240475 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 -- 172.21.15.97:0/15240475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 msgr2=0x7fd8940b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 --2- 172.21.15.97:0/15240475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 0x7fd8940b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 -- 172.21.15.97:0/15240475 >> v1:172.21.15.135:6789/0 conn(0x7fd89406e2f0 legacy=0x7fd894161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.140+0000 7fd898a00640 1 -- 172.21.15.97:0/15240475 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89406cba0 con 0x7fd8940b2c40 2024-03-06T07:31:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.141+0000 7fd898a00640 1 --2- 172.21.15.97:0/15240475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89415f1c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd884009870 tx=0x7fd88402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc03c8af88e07f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd893e00640 1 -- 172.21.15.97:0/15240475 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884035020 con 0x7fd8940b2c40 2024-03-06T07:31:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd893e00640 1 -- 172.21.15.97:0/15240475 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884031640 con 0x7fd8940b2c40 2024-03-06T07:31:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd893e00640 1 -- 172.21.15.97:0/15240475 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88402d040 con 0x7fd8940b2c40 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd89b400640 1 -- 172.21.15.97:0/15240475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 msgr2=0x7fd89415f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd89b400640 1 --2- 172.21.15.97:0/15240475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89415f1c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd884009870 tx=0x7fd88402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd89b400640 1 -- 172.21.15.97:0/15240475 shutdown_connections 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd89b400640 1 --2- 172.21.15.97:0/15240475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 0x7fd8940b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd89b400640 1 --2- 172.21.15.97:0/15240475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89415f1c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.142+0000 7fd89b400640 1 -- 172.21.15.97:0/15240475 >> 172.21.15.97:0/15240475 conn(0x7fd8940b1940 msgr2=0x7fd8940b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.143+0000 7fd89b400640 1 -- 172.21.15.97:0/15240475 shutdown_connections 2024-03-06T07:31:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.143+0000 7fd89b400640 1 -- 172.21.15.97:0/15240475 wait complete. 2024-03-06T07:31:50.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.144+0000 7fd89b400640 1 Processor -- start 2024-03-06T07:31:50.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.144+0000 7fd89b400640 1 -- start start 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.145+0000 7fd89b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 0x7fd894152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.145+0000 7fd89b400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd89406e2f0 0x7fd89409b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.145+0000 7fd89b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89409bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.145+0000 7fd89b400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406e750 con 0x7fd8940b2c40 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.145+0000 7fd89b400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406d160 con 0x7fd89406e2f0 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.145+0000 7fd89b400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406e8d0 con 0x7fd89406d960 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89409bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89409bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54154/0 (socket says 172.21.15.97:54154) 2024-03-06T07:31:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 -- 172.21.15.97:0/573754007 learned_addr learned my addr 172.21.15.97:0/573754007 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 -- 172.21.15.97:0/573754007 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 msgr2=0x7fd894152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 0x7fd894152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 -- 172.21.15.97:0/573754007 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd89406e2f0 msgr2=0x7fd89409b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd89406e2f0 0x7fd89409b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 -- 172.21.15.97:0/573754007 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894159120 con 0x7fd8940b2c40 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.146+0000 7fd899400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89409bbf0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00b990 tx=0x7fd88c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.147+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c019070 con 0x7fd8940b2c40 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.147+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c00cba0 con 0x7fd8940b2c40 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.147+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89409c440 con 0x7fd8940b2c40 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.147+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c014440 con 0x7fd8940b2c40 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.147+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89409ca00 con 0x7fd8940b2c40 2024-03-06T07:31:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.148+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8940b08c0 con 0x7fd8940b2c40 2024-03-06T07:31:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.149+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd88c007370 con 0x7fd8940b2c40 2024-03-06T07:31:50.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.150+0000 7fd891600640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd87006b500 0x7fd87006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.150+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd88c089ed0 con 0x7fd8940b2c40 2024-03-06T07:31:50.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.150+0000 7fd898a00640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd87006b500 0x7fd87006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.151+0000 7fd898a00640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd87006b500 0x7fd87006d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd884036000 tx=0x7fd884030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.157+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8940b08c0 con 0x7fd8940b2c40 2024-03-06T07:31:50.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.351+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8940b2530 con 0x7fd8940b2c40 2024-03-06T07:31:50.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.352+0000 7fd891600640 1 -- 172.21.15.97:0/573754007 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd88c058d10 con 0x7fd8940b2c40 2024-03-06T07:31:50.354 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.354+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd87006b500 msgr2=0x7fd87006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.354+0000 7fd89b400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd87006b500 0x7fd87006d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd884036000 tx=0x7fd884030000 comp rx=0 tx=0).stop 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.354+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 msgr2=0x7fd89409bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.354+0000 7fd89b400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89409bbf0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00b990 tx=0x7fd88c00be50 comp rx=0 tx=0).stop 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 shutdown_connections 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd87006b500 0x7fd87006d9c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8940b2c40 0x7fd89409bbf0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd89406e2f0 0x7fd89409b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 --2- 172.21.15.97:0/573754007 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd89406d960 0x7fd894152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 >> 172.21.15.97:0/573754007 conn(0x7fd8940b1940 msgr2=0x7fd894071790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 shutdown_connections 2024-03-06T07:31:50.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.355+0000 7fd89b400640 1 -- 172.21.15.97:0/573754007 wait complete. 2024-03-06T07:31:50.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:50.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.575+0000 7f7b8a400640 1 Processor -- start 2024-03-06T07:31:50.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.575+0000 7f7b8a400640 1 -- start start 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.577+0000 7f7b8a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b8415fc10 0x7f7b84070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.577+0000 7f7b8a400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b8415fff0 0x7f7b84160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.577+0000 7f7b8a400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b840b46c0 con 0x7f7b840ce4a0 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.577+0000 7f7b8a400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b840b49a0 con 0x7f7b8415fff0 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.577+0000 7f7b8a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b84072c90 con 0x7f7b8415fc10 2024-03-06T07:31:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.578+0000 7f7b83400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b8415fc10 0x7f7b84070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.578+0000 7f7b83e00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f7b840ce4a0 0x7f7b840ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:41344/0 (socket says 172.21.15.97:41344) 2024-03-06T07:31:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.578+0000 7f7b83e00640 1 -- 172.21.15.97:0/3125651873 learned_addr learned my addr 172.21.15.97:0/3125651873 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.578+0000 7f7b7be00640 1 --2- 172.21.15.97:0/3125651873 >> v2:172.21.15.135:3300/0 conn(0x7f7b8415fff0 0x7f7b84160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.579+0000 7f7b7be00640 1 -- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b8415fc10 msgr2=0x7f7b84070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.579+0000 7f7b7be00640 1 --2- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b8415fc10 0x7f7b84070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.579+0000 7f7b7be00640 1 -- 172.21.15.97:0/3125651873 >> v1:172.21.15.135:6789/0 conn(0x7f7b840ce4a0 legacy=0x7f7b840ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.579+0000 7f7b7be00640 1 -- 172.21.15.97:0/3125651873 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84160990 con 0x7f7b8415fff0 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.580+0000 7f7b7be00640 1 --2- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84160450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0097a0 tx=0x7f7b6c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=862e1a84e0e7ccfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.581+0000 7f7b82a00640 1 -- 172.21.15.97:0/3125651873 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c035020 con 0x7f7b8415fff0 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.581+0000 7f7b82a00640 1 -- 172.21.15.97:0/3125651873 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c031830 con 0x7f7b8415fff0 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.581+0000 7f7b82a00640 1 -- 172.21.15.97:0/3125651873 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c031c00 con 0x7f7b8415fff0 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 -- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 msgr2=0x7f7b84160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 --2- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84160450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0097a0 tx=0x7f7b6c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 -- 172.21.15.97:0/3125651873 shutdown_connections 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 --2- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84160450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 --2- 172.21.15.97:0/3125651873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b8415fc10 0x7f7b84070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 -- 172.21.15.97:0/3125651873 >> 172.21.15.97:0/3125651873 conn(0x7f7b840b1940 msgr2=0x7f7b840b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 -- 172.21.15.97:0/3125651873 shutdown_connections 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.582+0000 7f7b8a400640 1 -- 172.21.15.97:0/3125651873 wait complete. 2024-03-06T07:31:50.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.584+0000 7f7b8a400640 1 Processor -- start 2024-03-06T07:31:50.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.584+0000 7f7b8a400640 1 -- start start 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b8a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b840ce4a0 0x7f7b8415b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b8a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 0x7f7b84150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b8a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b8a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b84072a30 con 0x7f7b8415fc10 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b8a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840728b0 con 0x7f7b8415fff0 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b8a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8415c2e0 con 0x7f7b840ce4a0 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 0x7f7b84150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 0x7f7b84150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54166/0 (socket says 172.21.15.97:54166) 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b7be00640 1 -- 172.21.15.97:0/1352791832 learned_addr learned my addr 172.21.15.97:0/1352791832 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b83400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b840ce4a0 0x7f7b8415b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.585+0000 7f7b83e00640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7be00640 1 -- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b840ce4a0 msgr2=0x7f7b8415b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7be00640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b840ce4a0 0x7f7b8415b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7be00640 1 -- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 msgr2=0x7f7b84154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7be00640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7be00640 1 -- 172.21.15.97:0/1352791832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840c8fc0 con 0x7f7b8415fc10 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b83400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b840ce4a0 0x7f7b8415b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b83e00640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7be00640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 0x7f7b84150120 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0096f0 tx=0x7f7b6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c040070 con 0x7f7b8415fc10 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c03a910 con 0x7f7b8415fc10 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8409ab90 con 0x7f7b8415fc10 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c035040 con 0x7f7b8415fc10 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.586+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8409b150 con 0x7f7b8415fc10 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.587+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8404f330 con 0x7f7b8415fc10 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.588+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b6c04b060 con 0x7f7b8415fc10 2024-03-06T07:31:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.589+0000 7f7b7b400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b6006b4d0 0x7f7b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.589+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b6c0ab9f0 con 0x7f7b8415fc10 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.589+0000 7f7b83400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b6006b4d0 0x7f7b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.590+0000 7f7b83400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b6006b4d0 0x7f7b6006d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009ac0 tx=0x7f7b74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.597+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b6c07a820 con 0x7f7b8415fc10 2024-03-06T07:31:50.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.792+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7b8406f570 con 0x7f7b8415fc10 2024-03-06T07:31:50.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.793+0000 7f7b7b400640 1 -- 172.21.15.97:0/1352791832 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7b6c07a120 con 0x7f7b8415fc10 2024-03-06T07:31:50.796 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b6006b4d0 msgr2=0x7f7b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b6006b4d0 0x7f7b6006d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009ac0 tx=0x7f7b74008040 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 msgr2=0x7f7b84150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 0x7f7b84150120 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0096f0 tx=0x7f7b6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 shutdown_connections 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b6006b4d0 0x7f7b6006d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b8415fff0 0x7f7b84154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b8415fc10 0x7f7b84150120 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 --2- 172.21.15.97:0/1352791832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b840ce4a0 0x7f7b8415b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 >> 172.21.15.97:0/1352791832 conn(0x7f7b840b1940 msgr2=0x7f7b840b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 shutdown_connections 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:50.796+0000 7f7b8a400640 1 -- 172.21.15.97:0/1352791832 wait complete. 2024-03-06T07:31:50.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:51.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.019+0000 7f5567400640 1 Processor -- start 2024-03-06T07:31:51.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.019+0000 7f5567400640 1 -- start start 2024-03-06T07:31:51.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5567400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55600d2480 0x7f55600d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5567400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f556015e470 0x7f55600b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5567400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55600d1f80 con 0x7f55600d2860 2024-03-06T07:31:51.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5567400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600d4da0 con 0x7f556015e470 2024-03-06T07:31:51.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5567400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600cef80 con 0x7f55600d2480 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5565400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f556015e470 0x7f55600b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5557e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f55600d2860 0x7f556015dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33366/0 (socket says 172.21.15.97:33366) 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.021+0000 7f5557e00640 1 -- 172.21.15.97:0/3513545143 learned_addr learned my addr 172.21.15.97:0/3513545143 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.022+0000 7f5564a00640 1 --2- 172.21.15.97:0/3513545143 >> v2:172.21.15.135:3300/0 conn(0x7f55600d2480 0x7f55600d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.022+0000 7f5565400640 1 -- 172.21.15.97:0/3513545143 >> v2:172.21.15.135:3300/0 conn(0x7f55600d2480 msgr2=0x7f55600d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.022+0000 7f5565400640 1 --2- 172.21.15.97:0/3513545143 >> v2:172.21.15.135:3300/0 conn(0x7f55600d2480 0x7f55600d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.022+0000 7f5565400640 1 -- 172.21.15.97:0/3513545143 >> v1:172.21.15.97:6789/0 conn(0x7f55600d2860 legacy=0x7f556015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.022+0000 7f5565400640 1 -- 172.21.15.97:0/3513545143 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55600cf260 con 0x7f556015e470 2024-03-06T07:31:51.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.023+0000 7f5565400640 1 --2- 172.21.15.97:0/3513545143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556015e470 0x7f55600b3350 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f556009d970 tx=0x7f555802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e3e4159998932a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.023+0000 7f555fe00640 1 -- 172.21.15.97:0/3513545143 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5558039070 con 0x7f556015e470 2024-03-06T07:31:51.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.023+0000 7f555fe00640 1 -- 172.21.15.97:0/3513545143 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555802cd10 con 0x7f556015e470 2024-03-06T07:31:51.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.023+0000 7f555fe00640 1 -- 172.21.15.97:0/3513545143 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5558034660 con 0x7f556015e470 2024-03-06T07:31:51.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 -- 172.21.15.97:0/3513545143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556015e470 msgr2=0x7f55600b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 --2- 172.21.15.97:0/3513545143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556015e470 0x7f55600b3350 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f556009d970 tx=0x7f555802bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:51.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 -- 172.21.15.97:0/3513545143 shutdown_connections 2024-03-06T07:31:51.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 --2- 172.21.15.97:0/3513545143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556015e470 0x7f55600b3350 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 --2- 172.21.15.97:0/3513545143 >> v2:172.21.15.135:3300/0 conn(0x7f55600d2480 0x7f55600d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 -- 172.21.15.97:0/3513545143 >> 172.21.15.97:0/3513545143 conn(0x7f556009d240 msgr2=0x7f55600d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 -- 172.21.15.97:0/3513545143 shutdown_connections 2024-03-06T07:31:51.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.024+0000 7f5567400640 1 -- 172.21.15.97:0/3513545143 wait complete. 2024-03-06T07:31:51.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.026+0000 7f5567400640 1 Processor -- start 2024-03-06T07:31:51.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.026+0000 7f5567400640 1 -- start start 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5567400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 0x7f5560157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5567400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55600d2860 0x7f55600b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5567400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556015e470 0x7f5560158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5567400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600cfb00 con 0x7f55600d2480 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5567400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600cf820 con 0x7f556015e470 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5567400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600ce9f0 con 0x7f55600d2860 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 0x7f5560157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 0x7f5560157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54182/0 (socket says 172.21.15.97:54182) 2024-03-06T07:31:51.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 -- 172.21.15.97:0/3294394199 learned_addr learned my addr 172.21.15.97:0/3294394199 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:51.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 -- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55600d2860 msgr2=0x7f55600b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55600d2860 0x7f55600b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5565400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556015e470 0x7f5560158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 -- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556015e470 msgr2=0x7f5560158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556015e470 0x7f5560158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.027+0000 7f5564a00640 1 -- 172.21.15.97:0/3294394199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560157750 con 0x7f55600d2480 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.028+0000 7f5564a00640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 0x7f5560157210 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f555000d800 tx=0x7f555000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.028+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555000b6c0 con 0x7f55600d2480 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.028+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5550019070 con 0x7f55600d2480 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.028+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555000ba80 con 0x7f55600d2480 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.028+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55600cc070 con 0x7f55600d2480 2024-03-06T07:31:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.028+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55600cc630 con 0x7f55600d2480 2024-03-06T07:31:51.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.030+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556004f330 con 0x7f55600d2480 2024-03-06T07:31:51.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.030+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f555000bc20 con 0x7f55600d2480 2024-03-06T07:31:51.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.031+0000 7f555d600640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f553c06b4d0 0x7f553c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.031+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f555008a660 con 0x7f55600d2480 2024-03-06T07:31:51.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.031+0000 7f5557e00640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f553c06b4d0 0x7f553c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.032+0000 7f5557e00640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f553c06b4d0 0x7f553c06d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5548000f60 tx=0x7f5548007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.035+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5550059530 con 0x7f55600d2480 2024-03-06T07:31:51.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.243+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f55600d1490 con 0x7f55600d2480 2024-03-06T07:31:51.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.244+0000 7f555d600640 1 -- 172.21.15.97:0/3294394199 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v71) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5550058e30 con 0x7f55600d2480 2024-03-06T07:31:51.246 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:31:51.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.246+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f553c06b4d0 msgr2=0x7f553c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f553c06b4d0 0x7f553c06d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5548000f60 tx=0x7f5548007a80 comp rx=0 tx=0).stop 2024-03-06T07:31:51.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 msgr2=0x7f5560157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 0x7f5560157210 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f555000d800 tx=0x7f555000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 shutdown_connections 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f553c06b4d0 0x7f553c06d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556015e470 0x7f5560158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f55600d2860 0x7f55600b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 --2- 172.21.15.97:0/3294394199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55600d2480 0x7f5560157210 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 >> 172.21.15.97:0/3294394199 conn(0x7f556009d240 msgr2=0x7f55600d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 shutdown_connections 2024-03-06T07:31:51.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.247+0000 7f5567400640 1 -- 172.21.15.97:0/3294394199 wait complete. 2024-03-06T07:31:51.268 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:31:51.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:51.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:51.404 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:31:51.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:51.404 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:51.451 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:31:51.451 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:51.490 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:51.490 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:31:51.490 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:51.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:51.529 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:31:51.529 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:51.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:51.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.769+0000 7fd093800640 1 Processor -- start 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.769+0000 7fd093800640 1 -- start start 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.770+0000 7fd093800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c15f9d0 0x7fd08c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.770+0000 7fd093800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd08c070b10 0x7fd08c070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.770+0000 7fd093800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c06cef0 con 0x7fd08c144c00 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.770+0000 7fd093800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08c160370 con 0x7fd08c070b10 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.770+0000 7fd093800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c06d1d0 con 0x7fd08c15f9d0 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd090e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd08c144c00 0x7fd08c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33380/0 (socket says 172.21.15.97:33380) 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd090e00640 1 -- 172.21.15.97:0/3177041681 learned_addr learned my addr 172.21.15.97:0/3177041681 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd091800640 1 --2- 172.21.15.97:0/3177041681 >> v2:172.21.15.97:3300/0 conn(0x7fd08c070b10 0x7fd08c070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd083e00640 1 --2- 172.21.15.97:0/3177041681 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c15f9d0 0x7fd08c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd091800640 1 -- 172.21.15.97:0/3177041681 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c15f9d0 msgr2=0x7fd08c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd091800640 1 --2- 172.21.15.97:0/3177041681 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c15f9d0 0x7fd08c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd091800640 1 -- 172.21.15.97:0/3177041681 >> v1:172.21.15.97:6789/0 conn(0x7fd08c144c00 legacy=0x7fd08c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.771+0000 7fd091800640 1 -- 172.21.15.97:0/3177041681 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c06d4b0 con 0x7fd08c070b10 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.773+0000 7fd091800640 1 --2- 172.21.15.97:0/3177041681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c070f60 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd084009d90 tx=0x7fd08402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14720b37c4b53cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.773+0000 7fd08be00640 1 -- 172.21.15.97:0/3177041681 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084039070 con 0x7fd08c070b10 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.773+0000 7fd08be00640 1 -- 172.21.15.97:0/3177041681 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08402cc50 con 0x7fd08c070b10 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.773+0000 7fd08be00640 1 -- 172.21.15.97:0/3177041681 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084034440 con 0x7fd08c070b10 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 -- 172.21.15.97:0/3177041681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 msgr2=0x7fd08c070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 --2- 172.21.15.97:0/3177041681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c070f60 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd084009d90 tx=0x7fd08402bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 -- 172.21.15.97:0/3177041681 shutdown_connections 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 --2- 172.21.15.97:0/3177041681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c070f60 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 --2- 172.21.15.97:0/3177041681 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c15f9d0 0x7fd08c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 -- 172.21.15.97:0/3177041681 >> 172.21.15.97:0/3177041681 conn(0x7fd08c0b1730 msgr2=0x7fd08c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 -- 172.21.15.97:0/3177041681 shutdown_connections 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.774+0000 7fd093800640 1 -- 172.21.15.97:0/3177041681 wait complete. 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.776+0000 7fd093800640 1 Processor -- start 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.776+0000 7fd093800640 1 -- start start 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd093800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c15b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd093800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c144c00 0x7fd08c159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd093800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd08c15f9d0 0x7fd08c159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd093800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c06cb50 con 0x7fd08c070b10 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd093800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c160730 con 0x7fd08c15f9d0 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd093800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c06ccd0 con 0x7fd08c144c00 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c15b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c15b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54200/0 (socket says 172.21.15.97:54200) 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 -- 172.21.15.97:0/1952856829 learned_addr learned my addr 172.21.15.97:0/1952856829 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd091800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd08c15f9d0 0x7fd08c159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd083e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c144c00 0x7fd08c159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 -- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c144c00 msgr2=0x7fd08c159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c144c00 0x7fd08c159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 -- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd08c15f9d0 msgr2=0x7fd08c159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd08c15f9d0 0x7fd08c159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd090e00640 1 -- 172.21.15.97:0/1952856829 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c1f8760 con 0x7fd08c070b10 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd091800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd08c15f9d0 0x7fd08c159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.777+0000 7fd083e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c144c00 0x7fd08c159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.778+0000 7fd090e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c15b990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00c400 tx=0x7fd07c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.778+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c017070 con 0x7fd08c070b10 2024-03-06T07:31:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.778+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c005ce0 con 0x7fd08c070b10 2024-03-06T07:31:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.778+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08c15bed0 con 0x7fd08c070b10 2024-03-06T07:31:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.778+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0072f0 con 0x7fd08c070b10 2024-03-06T07:31:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.778+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08c1f7c80 con 0x7fd08c070b10 2024-03-06T07:31:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.780+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08c04f330 con 0x7fd08c070b10 2024-03-06T07:31:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.780+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd07c01a070 con 0x7fd08c070b10 2024-03-06T07:31:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.782+0000 7fd089600640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.782+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd07c089690 con 0x7fd08c070b10 2024-03-06T07:31:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.782+0000 7fd083e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.783+0000 7fd083e00640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd06806b4d0 0x7fd06806d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd08c13db40 tx=0x7fd074008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:51.786+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd07c058430 con 0x7fd08c070b10 2024-03-06T07:31:52.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.060+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd08c072890 con 0x7fd08c070b10 2024-03-06T07:31:52.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.061+0000 7fd089600640 1 -- 172.21.15.97:0/1952856829 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd07c057d30 con 0x7fd08c070b10 2024-03-06T07:31:52.063 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:52.063 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:52.063 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.063+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd06806b4d0 msgr2=0x7fd06806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.063+0000 7fd093800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd06806b4d0 0x7fd06806d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd08c13db40 tx=0x7fd074008040 comp rx=0 tx=0).stop 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.063+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 msgr2=0x7fd08c15b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.063+0000 7fd093800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c15b990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00c400 tx=0x7fd07c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 shutdown_connections 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd08c15f9d0 0x7fd08c159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd08c144c00 0x7fd08c159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 --2- 172.21.15.97:0/1952856829 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd08c070b10 0x7fd08c15b990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 >> 172.21.15.97:0/1952856829 conn(0x7fd08c0b1730 msgr2=0x7fd08c13e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 shutdown_connections 2024-03-06T07:31:52.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.064+0000 7fd093800640 1 -- 172.21.15.97:0/1952856829 wait complete. 2024-03-06T07:31:52.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:52.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.288+0000 7fda42e00640 1 Processor -- start 2024-03-06T07:31:52.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.289+0000 7fda42e00640 1 -- start start 2024-03-06T07:31:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda42e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fda3c15acb0 0x7fda3c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda42e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fda3c06ec00 0x7fda3c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda42e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c15b090 con 0x7fda3c0b4680 2024-03-06T07:31:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda42e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c0b2a30 con 0x7fda3c15acb0 2024-03-06T07:31:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda42e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c0b4360 con 0x7fda3c06ec00 2024-03-06T07:31:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda3be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fda3c15acb0 0x7fda3c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda33400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fda3c0b4680 0x7fda3c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33390/0 (socket says 172.21.15.97:33390) 2024-03-06T07:31:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.290+0000 7fda33400640 1 -- 172.21.15.97:0/1883459812 learned_addr learned my addr 172.21.15.97:0/1883459812 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.291+0000 7fda3be00640 1 -- 172.21.15.97:0/1883459812 >> v2:172.21.15.135:3300/0 conn(0x7fda3c06ec00 msgr2=0x7fda3c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.291+0000 7fda40e00640 1 --2- 172.21.15.97:0/1883459812 >> v2:172.21.15.135:3300/0 conn(0x7fda3c06ec00 0x7fda3c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.291+0000 7fda3be00640 1 --2- 172.21.15.97:0/1883459812 >> v2:172.21.15.135:3300/0 conn(0x7fda3c06ec00 0x7fda3c0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.291+0000 7fda3be00640 1 -- 172.21.15.97:0/1883459812 >> v1:172.21.15.97:6789/0 conn(0x7fda3c0b4680 legacy=0x7fda3c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.291+0000 7fda3be00640 1 -- 172.21.15.97:0/1883459812 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0b2cf0 con 0x7fda3c15acb0 2024-03-06T07:31:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.291+0000 7fda3b400640 1 -- 172.21.15.97:0/1883459812 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811963400 0 0) 0x7fda3c15b090 con 0x7fda3c0b4680 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.292+0000 7fda3be00640 1 --2- 172.21.15.97:0/1883459812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c15acb0 0x7fda3c0b37d0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0b4e60 tx=0x7fda2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c56ed7f4cdfb20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.292+0000 7fda3b400640 1 -- 172.21.15.97:0/1883459812 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c035020 con 0x7fda3c15acb0 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.292+0000 7fda3b400640 1 -- 172.21.15.97:0/1883459812 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c031830 con 0x7fda3c15acb0 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.292+0000 7fda3b400640 1 -- 172.21.15.97:0/1883459812 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c031c00 con 0x7fda3c15acb0 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 -- 172.21.15.97:0/1883459812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c15acb0 msgr2=0x7fda3c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 --2- 172.21.15.97:0/1883459812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c15acb0 0x7fda3c0b37d0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0b4e60 tx=0x7fda2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 -- 172.21.15.97:0/1883459812 shutdown_connections 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 --2- 172.21.15.97:0/1883459812 >> v2:172.21.15.135:3300/0 conn(0x7fda3c06ec00 0x7fda3c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 --2- 172.21.15.97:0/1883459812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c15acb0 0x7fda3c0b37d0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 -- 172.21.15.97:0/1883459812 >> 172.21.15.97:0/1883459812 conn(0x7fda3c0b1730 msgr2=0x7fda3c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 -- 172.21.15.97:0/1883459812 shutdown_connections 2024-03-06T07:31:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.293+0000 7fda42e00640 1 -- 172.21.15.97:0/1883459812 wait complete. 2024-03-06T07:31:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.295+0000 7fda42e00640 1 Processor -- start 2024-03-06T07:31:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.295+0000 7fda42e00640 1 -- start start 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda42e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 0x7fda3c1466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda42e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fda3c0b4680 0x7fda3c0c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda42e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda3c15acb0 0x7fda3c148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda42e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b3d10 con 0x7fda3c06ec00 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda42e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b32b0 con 0x7fda3c15acb0 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda42e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b4010 con 0x7fda3c0b4680 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 0x7fda3c1466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 0x7fda3c1466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54208/0 (socket says 172.21.15.97:54208) 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda3be00640 1 -- 172.21.15.97:0/1048855137 learned_addr learned my addr 172.21.15.97:0/1048855137 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.297+0000 7fda40e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda3c15acb0 0x7fda3c148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda3be00640 1 -- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fda3c0b4680 msgr2=0x7fda3c0c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda3be00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fda3c0b4680 0x7fda3c0c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda3be00640 1 -- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda3c15acb0 msgr2=0x7fda3c148e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda3be00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda3c15acb0 0x7fda3c148e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda3be00640 1 -- 172.21.15.97:0/1048855137 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c1432a0 con 0x7fda3c06ec00 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda3be00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 0x7fda3c1466f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fda2c000c00 tx=0x7fda2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c035050 con 0x7fda3c06ec00 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c004230 con 0x7fda3c06ec00 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c03b800 con 0x7fda3c06ec00 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.298+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c143580 con 0x7fda3c06ec00 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.299+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c143b40 con 0x7fda3c06ec00 2024-03-06T07:31:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.300+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c04f330 con 0x7fda3c06ec00 2024-03-06T07:31:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.301+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda2c038020 con 0x7fda3c06ec00 2024-03-06T07:31:52.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.302+0000 7fda38c00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.302+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fda2c0ac0f0 con 0x7fda3c06ec00 2024-03-06T07:31:52.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.302+0000 7fda33400640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.303+0000 7fda33400640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fda1806b4d0 0x7fda1806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fda24007fd0 tx=0x7fda24007960 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.306+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda2c040070 con 0x7fda3c06ec00 2024-03-06T07:31:52.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.538+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda3c074be0 con 0x7fda3c06ec00 2024-03-06T07:31:52.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.540+0000 7fda38c00640 1 -- 172.21.15.97:0/1048855137 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16754 (secure 0 0 0) 0x7fda2c07af20 con 0x7fda3c06ec00 2024-03-06T07:31:52.543 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:52.543 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":71,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:28.452973+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.30","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fda1806b4d0 msgr2=0x7fda1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fda1806b4d0 0x7fda1806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fda24007fd0 tx=0x7fda24007960 comp rx=0 tx=0).stop 2024-03-06T07:31:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 msgr2=0x7fda3c1466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 0x7fda3c1466f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fda2c000c00 tx=0x7fda2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 shutdown_connections 2024-03-06T07:31:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda3c15acb0 0x7fda3c148e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fda3c0b4680 0x7fda3c0c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 --2- 172.21.15.97:0/1048855137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda3c06ec00 0x7fda3c1466f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 >> 172.21.15.97:0/1048855137 conn(0x7fda3c0b1730 msgr2=0x7fda3c155ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 shutdown_connections 2024-03-06T07:31:52.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.544+0000 7fda42e00640 1 -- 172.21.15.97:0/1048855137 wait complete. 2024-03-06T07:31:52.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:06.960932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:08.155371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:52.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:52.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.774+0000 7f305de00640 1 Processor -- start 2024-03-06T07:31:52.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.774+0000 7f305de00640 1 -- start start 2024-03-06T07:31:52.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.775+0000 7f305de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305815e000 0x7f305806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f305de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f305806fef0 0x7f30580d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f305de00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580d1260 con 0x7f305815e3e0 2024-03-06T07:31:52.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f305de00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580d1020 con 0x7f305806fef0 2024-03-06T07:31:52.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f305de00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305806c0d0 con 0x7f305815e000 2024-03-06T07:31:52.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f304fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f305815e3e0 0x7f305806f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33406/0 (socket says 172.21.15.97:33406) 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f304fe00640 1 -- 172.21.15.97:0/4224828175 learned_addr learned my addr 172.21.15.97:0/4224828175 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f3057400640 1 --2- 172.21.15.97:0/4224828175 >> v2:172.21.15.97:3300/0 conn(0x7f305806fef0 0x7f30580d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f3056a00640 1 --2- 172.21.15.97:0/4224828175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305815e000 0x7f305806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f3056000640 1 -- 172.21.15.97:0/4224828175 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969621896 0 0) 0x7f30580d1260 con 0x7f305815e3e0 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.776+0000 7f3056000640 1 -- 172.21.15.97:0/4224828175 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f303c003520 con 0x7f305815e3e0 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.777+0000 7f3057400640 1 -- 172.21.15.97:0/4224828175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305815e000 msgr2=0x7f305806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.777+0000 7f3057400640 1 --2- 172.21.15.97:0/4224828175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305815e000 0x7f305806f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.777+0000 7f3057400640 1 -- 172.21.15.97:0/4224828175 >> v1:172.21.15.97:6789/0 conn(0x7f305815e3e0 legacy=0x7f305806f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.777+0000 7f3057400640 1 -- 172.21.15.97:0/4224828175 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305806c320 con 0x7f305806fef0 2024-03-06T07:31:52.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.778+0000 7f3057400640 1 --2- 172.21.15.97:0/4224828175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305806fef0 0x7f30580d0ae0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3050009d90 tx=0x7f305002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64be15966436d16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.778+0000 7f3056000640 1 -- 172.21.15.97:0/4224828175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050039070 con 0x7f305806fef0 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.778+0000 7f3056000640 1 -- 172.21.15.97:0/4224828175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305002cc50 con 0x7f305806fef0 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.778+0000 7f3056000640 1 -- 172.21.15.97:0/4224828175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050034440 con 0x7f305806fef0 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.778+0000 7f305de00640 1 -- 172.21.15.97:0/4224828175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305806fef0 msgr2=0x7f30580d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.778+0000 7f305de00640 1 --2- 172.21.15.97:0/4224828175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305806fef0 0x7f30580d0ae0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3050009d90 tx=0x7f305002bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.779+0000 7f305de00640 1 -- 172.21.15.97:0/4224828175 shutdown_connections 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.779+0000 7f305de00640 1 --2- 172.21.15.97:0/4224828175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305806fef0 0x7f30580d0ae0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.779+0000 7f305de00640 1 --2- 172.21.15.97:0/4224828175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305815e000 0x7f305806f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.779+0000 7f305de00640 1 -- 172.21.15.97:0/4224828175 >> 172.21.15.97:0/4224828175 conn(0x7f30580b3800 msgr2=0x7f30580b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.779+0000 7f305de00640 1 -- 172.21.15.97:0/4224828175 shutdown_connections 2024-03-06T07:31:52.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.779+0000 7f305de00640 1 -- 172.21.15.97:0/4224828175 wait complete. 2024-03-06T07:31:52.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.780+0000 7f305de00640 1 Processor -- start 2024-03-06T07:31:52.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 -- start start 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305806fef0 0x7f30580c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 0x7f305815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305815e3e0 0x7f30580c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580ce4c0 con 0x7f305815e000 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580ce340 con 0x7f305815e3e0 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.781+0000 7f305de00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30580ce640 con 0x7f305806fef0 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 0x7f305815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 0x7f305815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54234/0 (socket says 172.21.15.97:54234) 2024-03-06T07:31:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 -- 172.21.15.97:0/167226023 learned_addr learned my addr 172.21.15.97:0/167226023 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f3057400640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305815e3e0 0x7f30580c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f3056a00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305806fef0 0x7f30580c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 -- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305806fef0 msgr2=0x7f30580c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305806fef0 0x7f30580c5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 -- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305815e3e0 msgr2=0x7f30580c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305815e3e0 0x7f30580c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f304fe00640 1 -- 172.21.15.97:0/167226023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580c5a10 con 0x7f305815e000 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.782+0000 7f3057400640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305815e3e0 0x7f30580c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.783+0000 7f304fe00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 0x7f305815a970 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f304000c390 tx=0x7f304000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.783+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040017070 con 0x7f305815e000 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.783+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040005ce0 con 0x7f305815e000 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.783+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400072f0 con 0x7f305815e000 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.783+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30581f4410 con 0x7f305815e000 2024-03-06T07:31:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.783+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581f49d0 con 0x7f305815e000 2024-03-06T07:31:52.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.784+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30580b42f0 con 0x7f305815e000 2024-03-06T07:31:52.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.785+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f304001a070 con 0x7f305815e000 2024-03-06T07:31:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.786+0000 7f304f400640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f303406b4d0 0x7f303406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.787+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3040089650 con 0x7f305815e000 2024-03-06T07:31:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.787+0000 7f3056a00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f303406b4d0 0x7f303406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.788+0000 7f3056a00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f303406b4d0 0x7f303406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3048009a10 tx=0x7f3048008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:52.791+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30580b42f0 con 0x7f305815e000 2024-03-06T07:31:53.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.019+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f305804f330 con 0x7f305815e000 2024-03-06T07:31:53.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.020+0000 7f304f400640 1 -- 172.21.15.97:0/167226023 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3040058480 con 0x7f305815e000 2024-03-06T07:31:53.022 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f303406b4d0 msgr2=0x7f303406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f303406b4d0 0x7f303406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3048009a10 tx=0x7f3048008040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 msgr2=0x7f305815a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 0x7f305815a970 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f304000c390 tx=0x7f304000c850 comp rx=0 tx=0).stop 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 shutdown_connections 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f303406b4d0 0x7f303406d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305815e3e0 0x7f30580c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f305815e000 0x7f305815a970 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 --2- 172.21.15.97:0/167226023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f305806fef0 0x7f30580c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.022+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 >> 172.21.15.97:0/167226023 conn(0x7f30580b3800 msgr2=0x7f30580b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.023+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 shutdown_connections 2024-03-06T07:31:53.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.023+0000 7f305de00640 1 -- 172.21.15.97:0/167226023 wait complete. 2024-03-06T07:31:53.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:53.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.240+0000 7faa77e00640 1 Processor -- start 2024-03-06T07:31:53.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.241+0000 7faa77e00640 1 -- start start 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.242+0000 7faa77e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa700727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.242+0000 7faa77e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7faa700b3650 0x7faa700b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.242+0000 7faa77e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa700cda90 0x7faa7006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.242+0000 7faa77e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa70072310 con 0x7faa700cda90 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.242+0000 7faa77e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa700b4710 con 0x7faa700b3650 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.242+0000 7faa77e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa700719e0 con 0x7faa700b3df0 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa700cda90 0x7faa7006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa700cda90 0x7faa7006c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54238/0 (socket says 172.21.15.97:54238) 2024-03-06T07:31:53.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 -- 172.21.15.97:0/101090577 learned_addr learned my addr 172.21.15.97:0/101090577 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:53.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa74a00640 1 --2- 172.21.15.97:0/101090577 >> v2:172.21.15.135:3300/0 conn(0x7faa700b3650 0x7faa700b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75400640 1 --2- 172.21.15.97:0/101090577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa700727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 -- 172.21.15.97:0/101090577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 msgr2=0x7faa700727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 --2- 172.21.15.97:0/101090577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa700727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 -- 172.21.15.97:0/101090577 >> v2:172.21.15.135:3300/0 conn(0x7faa700b3650 msgr2=0x7faa700b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 --2- 172.21.15.97:0/101090577 >> v2:172.21.15.135:3300/0 conn(0x7faa700b3650 0x7faa700b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa75e00640 1 -- 172.21.15.97:0/101090577 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa70071cc0 con 0x7faa700cda90 2024-03-06T07:31:53.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.243+0000 7faa74a00640 1 --2- 172.21.15.97:0/101090577 >> v2:172.21.15.135:3300/0 conn(0x7faa700b3650 0x7faa700b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:53.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.245+0000 7faa75e00640 1 --2- 172.21.15.97:0/101090577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700cda90 0x7faa7006c990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faa700b2a00 tx=0x7faa6802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbbbee2d6f72af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.245+0000 7faa6fe00640 1 -- 172.21.15.97:0/101090577 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68035020 con 0x7faa700cda90 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.245+0000 7faa6fe00640 1 -- 172.21.15.97:0/101090577 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6802d040 con 0x7faa700cda90 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.245+0000 7faa6fe00640 1 -- 172.21.15.97:0/101090577 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68031a80 con 0x7faa700cda90 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.245+0000 7faa77e00640 1 -- 172.21.15.97:0/101090577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700cda90 msgr2=0x7faa7006c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.245+0000 7faa77e00640 1 --2- 172.21.15.97:0/101090577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700cda90 0x7faa7006c990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faa700b2a00 tx=0x7faa6802b900 comp rx=0 tx=0).stop 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 -- 172.21.15.97:0/101090577 shutdown_connections 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 --2- 172.21.15.97:0/101090577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700cda90 0x7faa7006c990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 --2- 172.21.15.97:0/101090577 >> v2:172.21.15.135:3300/0 conn(0x7faa700b3650 0x7faa700b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 --2- 172.21.15.97:0/101090577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa700727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 -- 172.21.15.97:0/101090577 >> 172.21.15.97:0/101090577 conn(0x7faa700b1940 msgr2=0x7faa700b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 -- 172.21.15.97:0/101090577 shutdown_connections 2024-03-06T07:31:53.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.246+0000 7faa77e00640 1 -- 172.21.15.97:0/101090577 wait complete. 2024-03-06T07:31:53.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.247+0000 7faa77e00640 1 Processor -- start 2024-03-06T07:31:53.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.248+0000 7faa77e00640 1 -- start start 2024-03-06T07:31:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.248+0000 7faa77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 0x7faa70155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa77e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa7014d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa77e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa700cda90 0x7faa701e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa77e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7015b240 con 0x7faa700b3650 2024-03-06T07:31:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa77e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa700b3b30 con 0x7faa700cda90 2024-03-06T07:31:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa77e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7015b3c0 con 0x7faa700b3df0 2024-03-06T07:31:53.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa75400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 0x7faa70155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa75400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 0x7faa70155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54246/0 (socket says 172.21.15.97:54246) 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa75400640 1 -- 172.21.15.97:0/1012183953 learned_addr learned my addr 172.21.15.97:0/1012183953 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa75e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa700cda90 0x7faa701e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.249+0000 7faa74a00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa7014d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75400640 1 -- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 msgr2=0x7faa7014d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75400640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa7014d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75400640 1 -- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa700cda90 msgr2=0x7faa701e7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75400640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa700cda90 0x7faa701e7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75400640 1 -- 172.21.15.97:0/1012183953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7015cc10 con 0x7faa700b3650 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa700cda90 0x7faa701e7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa75400640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 0x7faa70155320 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7faa64007e70 tx=0x7faa6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64014620 con 0x7faa700b3650 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64019070 con 0x7faa700b3650 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7015cef0 con 0x7faa700b3650 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64014a00 con 0x7faa700b3650 2024-03-06T07:31:53.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.250+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7015d4b0 con 0x7faa700b3650 2024-03-06T07:31:53.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.251+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7004f330 con 0x7faa700b3650 2024-03-06T07:31:53.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.254+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa64014ba0 con 0x7faa700b3650 2024-03-06T07:31:53.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.255+0000 7faa6d600640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.255+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faa6408ab50 con 0x7faa700b3650 2024-03-06T07:31:53.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.255+0000 7faa74a00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.256+0000 7faa74a00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa60006fd0 tx=0x7faa60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.259+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa640598f0 con 0x7faa700b3650 2024-03-06T07:31:53.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.454+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa700aeca0 con 0x7faa700b3650 2024-03-06T07:31:53.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.455+0000 7faa6d600640 1 -- 172.21.15.97:0/1012183953 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa640591f0 con 0x7faa700b3650 2024-03-06T07:31:53.457 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.457+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa5406b4d0 msgr2=0x7faa5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.457+0000 7faa77e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa60006fd0 tx=0x7faa60008040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.457+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 msgr2=0x7faa70155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.457+0000 7faa77e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 0x7faa70155320 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7faa64007e70 tx=0x7faa6400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 shutdown_connections 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa700cda90 0x7faa701e7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa700b3df0 0x7faa7014d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 --2- 172.21.15.97:0/1012183953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa700b3650 0x7faa70155320 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 >> 172.21.15.97:0/1012183953 conn(0x7faa700b1940 msgr2=0x7faa700adb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 shutdown_connections 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.458+0000 7faa77e00640 1 -- 172.21.15.97:0/1012183953 wait complete. 2024-03-06T07:31:53.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:53.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.688+0000 7f5f82c00640 1 Processor -- start 2024-03-06T07:31:53.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.689+0000 7f5f82c00640 1 -- start start 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f82c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5f7c0b4de0 0x7f5f7c0d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f82c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c0d4c40 con 0x7f5f7c0ced50 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f82c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c0d2700 con 0x7f5f7c15f490 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f82c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d49f0 con 0x7f5f7c0b4de0 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f7b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5f7c0b4de0 0x7f5f7c0d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f80c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5f7c0ced50 0x7f5f7c0cf180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33410/0 (socket says 172.21.15.97:33410) 2024-03-06T07:31:53.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f7b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5f7c0b4de0 0x7f5f7c0d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54248/0 (socket says 172.21.15.97:54248) 2024-03-06T07:31:53.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.690+0000 7f5f80c00640 1 -- 172.21.15.97:0/338684729 learned_addr learned my addr 172.21.15.97:0/338684729 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:53.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.691+0000 7f5f7aa00640 1 -- 172.21.15.97:0/338684729 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384327325 0 0) 0x7f5f7c0d4c40 con 0x7f5f7c0ced50 2024-03-06T07:31:53.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.691+0000 7f5f7b400640 1 -- 172.21.15.97:0/338684729 >> v1:172.21.15.135:6789/0 conn(0x7f5f7c15f490 legacy=0x7f5f7c0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.691+0000 7f5f7b400640 1 -- 172.21.15.97:0/338684729 >> v1:172.21.15.97:6789/0 conn(0x7f5f7c0ced50 legacy=0x7f5f7c0cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.691+0000 7f5f7b400640 1 -- 172.21.15.97:0/338684729 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0d29e0 con 0x7f5f7c0b4de0 2024-03-06T07:31:53.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.692+0000 7f5f7b400640 1 --2- 172.21.15.97:0/338684729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0b4de0 0x7f5f7c0d0fb0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5f680097a0 tx=0x7f5f6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e073f287b6a32a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.692+0000 7f5f7aa00640 1 -- 172.21.15.97:0/338684729 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f680315a0 con 0x7f5f7c0b4de0 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.692+0000 7f5f7aa00640 1 -- 172.21.15.97:0/338684729 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f68037070 con 0x7f5f7c0b4de0 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.692+0000 7f5f7aa00640 1 -- 172.21.15.97:0/338684729 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68031960 con 0x7f5f7c0b4de0 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 -- 172.21.15.97:0/338684729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0b4de0 msgr2=0x7f5f7c0d0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 --2- 172.21.15.97:0/338684729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0b4de0 0x7f5f7c0d0fb0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5f680097a0 tx=0x7f5f6802b840 comp rx=0 tx=0).stop 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 -- 172.21.15.97:0/338684729 shutdown_connections 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 --2- 172.21.15.97:0/338684729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0b4de0 0x7f5f7c0d0fb0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 -- 172.21.15.97:0/338684729 >> 172.21.15.97:0/338684729 conn(0x7f5f7c0d3ed0 msgr2=0x7f5f7c0d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 -- 172.21.15.97:0/338684729 shutdown_connections 2024-03-06T07:31:53.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.693+0000 7f5f82c00640 1 -- 172.21.15.97:0/338684729 wait complete. 2024-03-06T07:31:53.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.694+0000 7f5f82c00640 1 Processor -- start 2024-03-06T07:31:53.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.695+0000 7f5f82c00640 1 -- start start 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.695+0000 7f5f82c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f7c0b4de0 0x7f5f7c0b78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.695+0000 7f5f82c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 0x7f5f7c154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.695+0000 7f5f82c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f7c15f490 0x7f5f7c0b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f82c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c15fd50 con 0x7f5f7c0ced50 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f82c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d24b0 con 0x7f5f7c15f490 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f82c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c15f870 con 0x7f5f7c0b4de0 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 0x7f5f7c154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 0x7f5f7c154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54254/0 (socket says 172.21.15.97:54254) 2024-03-06T07:31:53.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 -- 172.21.15.97:0/2148677449 learned_addr learned my addr 172.21.15.97:0/2148677449 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 -- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f7c0b4de0 msgr2=0x7f5f7c0b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7be00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f7c0b4de0 0x7f5f7c0b78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f80c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f7c15f490 0x7f5f7c0b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f7c0b4de0 0x7f5f7c0b78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 -- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f7c15f490 msgr2=0x7f5f7c0b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f7c15f490 0x7f5f7c0b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7b400640 1 -- 172.21.15.97:0/2148677449 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0b7e00 con 0x7f5f7c0ced50 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f7be00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f7c0b4de0 0x7f5f7c0b78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.696+0000 7f5f80c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f7c15f490 0x7f5f7c0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.697+0000 7f5f7b400640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 0x7f5f7c154350 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5f680097a0 tx=0x7f5f680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.697+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68032840 con 0x7f5f7c0ced50 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.697+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f680329e0 con 0x7f5f7c0ced50 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.697+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c0b2ff0 con 0x7f5f7c0ced50 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.697+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68032dd0 con 0x7f5f7c0ced50 2024-03-06T07:31:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.697+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c0b3570 con 0x7f5f7c0ced50 2024-03-06T07:31:53.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.698+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c04f330 con 0x7f5f7c0ced50 2024-03-06T07:31:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.700+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f6803b3f0 con 0x7f5f7c0ced50 2024-03-06T07:31:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.701+0000 7f5f73e00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.701+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f6802f030 con 0x7f5f7c0ced50 2024-03-06T07:31:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.701+0000 7f5f7be00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.702+0000 7f5f7be00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c003eb0 tx=0x7f5f6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.710+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f68040070 con 0x7f5f7c0ced50 2024-03-06T07:31:53.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.926+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f7c08d950 con 0x7f5f7c0ced50 2024-03-06T07:31:53.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.926+0000 7f5f73e00640 1 -- 172.21.15.97:0/2148677449 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f6807b040 con 0x7f5f7c0ced50 2024-03-06T07:31:53.929 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:31:53.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f5c06b4d0 msgr2=0x7f5f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c003eb0 tx=0x7f5f6c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 msgr2=0x7f5f7c154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 0x7f5f7c154350 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5f680097a0 tx=0x7f5f680046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 shutdown_connections 2024-03-06T07:31:53.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f7c15f490 0x7f5f7c0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f7c0ced50 0x7f5f7c154350 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 --2- 172.21.15.97:0/2148677449 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5f7c0b4de0 0x7f5f7c0b78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 >> 172.21.15.97:0/2148677449 conn(0x7f5f7c0d3ed0 msgr2=0x7f5f7c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 shutdown_connections 2024-03-06T07:31:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:53.929+0000 7f5f82c00640 1 -- 172.21.15.97:0/2148677449 wait complete. 2024-03-06T07:31:53.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.152+0000 7f7e7ec00640 1 Processor -- start 2024-03-06T07:31:54.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.152+0000 7f7e7ec00640 1 -- start start 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.153+0000 7f7e7ec00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7e780cf430 0x7f7e78070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.153+0000 7f7e7ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e78071260 0x7f7e7806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.153+0000 7f7e7ec00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e780717c0 con 0x7f7e7806d090 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.153+0000 7f7e7ec00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7806d4c0 con 0x7f7e780cf430 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.153+0000 7f7e7ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78071aa0 con 0x7f7e78071260 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7e780cf430 0x7f7e78070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7e780cf430 0x7f7e78070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43772/0 (socket says 172.21.15.97:43772) 2024-03-06T07:31:54.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 -- 172.21.15.97:0/330404665 learned_addr learned my addr 172.21.15.97:0/330404665 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:54.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 -- 172.21.15.97:0/330404665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e78071260 msgr2=0x7f7e7806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 --2- 172.21.15.97:0/330404665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e78071260 0x7f7e7806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 -- 172.21.15.97:0/330404665 >> v1:172.21.15.135:6789/0 conn(0x7f7e7806d090 legacy=0x7f7e78165f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.154+0000 7f7e77e00640 1 -- 172.21.15.97:0/330404665 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78071d80 con 0x7f7e780cf430 2024-03-06T07:31:54.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.155+0000 7f7e77e00640 1 --2- 172.21.15.97:0/330404665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e78070d20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e6802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53b2e5c8bd772ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.155+0000 7f7e77400640 1 -- 172.21.15.97:0/330404665 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68035020 con 0x7f7e780cf430 2024-03-06T07:31:54.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.155+0000 7f7e77400640 1 -- 172.21.15.97:0/330404665 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68031640 con 0x7f7e780cf430 2024-03-06T07:31:54.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e77400640 1 -- 172.21.15.97:0/330404665 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6802d040 con 0x7f7e780cf430 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 -- 172.21.15.97:0/330404665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 msgr2=0x7f7e78070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/330404665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e78070d20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e6802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 -- 172.21.15.97:0/330404665 shutdown_connections 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/330404665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e78071260 0x7f7e7806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/330404665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e78070d20 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 -- 172.21.15.97:0/330404665 >> 172.21.15.97:0/330404665 conn(0x7f7e780b1730 msgr2=0x7f7e780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 -- 172.21.15.97:0/330404665 shutdown_connections 2024-03-06T07:31:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.156+0000 7f7e7ec00640 1 -- 172.21.15.97:0/330404665 wait complete. 2024-03-06T07:31:54.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.158+0000 7f7e7ec00640 1 Processor -- start 2024-03-06T07:31:54.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.158+0000 7f7e7ec00640 1 -- start start 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e7806d090 0x7f7e7815e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7ec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e78071260 0x7f7e78146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e7815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78162b70 con 0x7f7e780cf430 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7ec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78072340 con 0x7f7e78071260 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78162cf0 con 0x7f7e7806d090 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e7815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e7815c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43778/0 (socket says 172.21.15.97:43778) 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.159+0000 7f7e7cc00640 1 -- 172.21.15.97:0/1588501451 learned_addr learned my addr 172.21.15.97:0/1588501451 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e6f400640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e78071260 0x7f7e78146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e77e00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e7806d090 0x7f7e7815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e7cc00640 1 -- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e7806d090 msgr2=0x7f7e7815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e7cc00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e7806d090 0x7f7e7815e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e7cc00640 1 -- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e78071260 msgr2=0x7f7e78146d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e7cc00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e78071260 0x7f7e78146d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e7cc00640 1 -- 172.21.15.97:0/1588501451 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7815cda0 con 0x7f7e780cf430 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e6f400640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e78071260 0x7f7e78146d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:54.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e7cc00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e7815c860 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c780 tx=0x7f7e7000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70017070 con 0x7f7e780cf430 2024-03-06T07:31:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.160+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e70003da0 con 0x7f7e780cf430 2024-03-06T07:31:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.161+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7815e6d0 con 0x7f7e780cf430 2024-03-06T07:31:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.161+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e700052b0 con 0x7f7e780cf430 2024-03-06T07:31:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.161+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7815ec90 con 0x7f7e780cf430 2024-03-06T07:31:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.162+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7804f330 con 0x7f7e780cf430 2024-03-06T07:31:54.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.163+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e7001a070 con 0x7f7e780cf430 2024-03-06T07:31:54.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.164+0000 7f7e74c00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5406b4d0 0x7f7e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.164+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e70089890 con 0x7f7e780cf430 2024-03-06T07:31:54.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.164+0000 7f7e77e00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5406b4d0 0x7f7e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.165+0000 7f7e77e00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5406b4d0 0x7f7e5406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7e68036000 tx=0x7f7e6803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.168+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e700586c0 con 0x7f7e780cf430 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.407+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e780b1d60 con 0x7f7e780cf430 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.408+0000 7f7e74c00640 1 -- 172.21.15.97:0/1588501451 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f7e70057fc0 con 0x7f7e780cf430 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:54.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5406b4d0 msgr2=0x7f7e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5406b4d0 0x7f7e5406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7e68036000 tx=0x7f7e6803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 msgr2=0x7f7e7815c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e7815c860 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c780 tx=0x7f7e7000cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 shutdown_connections 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7e5406b4d0 0x7f7e5406d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e780cf430 0x7f7e7815c860 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e78071260 0x7f7e78146d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 --2- 172.21.15.97:0/1588501451 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7e7806d090 0x7f7e7815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 >> 172.21.15.97:0/1588501451 conn(0x7f7e780b1730 msgr2=0x7f7e780afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 shutdown_connections 2024-03-06T07:31:54.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.410+0000 7f7e7ec00640 1 -- 172.21.15.97:0/1588501451 wait complete. 2024-03-06T07:31:54.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:54.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.634+0000 7f4fd1600640 1 Processor -- start 2024-03-06T07:31:54.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.634+0000 7f4fd1600640 1 -- start start 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fd1600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4fcc0d0ca0 0x7f4fcc0cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fd1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc160530 0x7f4fcc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fd1600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fcc06b500 0x7f4fcc0b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fd1600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fcc0b50a0 con 0x7f4fcc06b500 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fd1600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fcc15d4f0 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fd1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc15cc50 con 0x7f4fcc160530 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fcb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fcc06b500 0x7f4fcc0b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fcb400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fcc06b500 0x7f4fcc0b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43792/0 (socket says 172.21.15.97:43792) 2024-03-06T07:31:54.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fcb400640 1 -- 172.21.15.97:0/3023849661 learned_addr learned my addr 172.21.15.97:0/3023849661 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:54.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fc2000640 1 --2- 172.21.15.97:0/3023849661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc160530 0x7f4fcc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.636+0000 7f4fcaa00640 1 --2- 172.21.15.97:0/3023849661 >> v2:172.21.15.135:3300/0 conn(0x7f4fcc0d0ca0 0x7f4fcc0cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.637+0000 7f4fcb400640 1 -- 172.21.15.97:0/3023849661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc160530 msgr2=0x7f4fcc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.637+0000 7f4fcb400640 1 --2- 172.21.15.97:0/3023849661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc160530 0x7f4fcc06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.637+0000 7f4fcb400640 1 -- 172.21.15.97:0/3023849661 >> v2:172.21.15.135:3300/0 conn(0x7f4fcc0d0ca0 msgr2=0x7f4fcc0cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.637+0000 7f4fcb400640 1 --2- 172.21.15.97:0/3023849661 >> v2:172.21.15.135:3300/0 conn(0x7f4fcc0d0ca0 0x7f4fcc0cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.637+0000 7f4fcb400640 1 -- 172.21.15.97:0/3023849661 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc15cf30 con 0x7f4fcc06b500 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.638+0000 7f4fcb400640 1 --2- 172.21.15.97:0/3023849661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc06b500 0x7f4fcc0b5530 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc070eb0 tx=0x7f4fc402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2104d01302078b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.638+0000 7f4fca000640 1 -- 172.21.15.97:0/3023849661 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc402c9e0 con 0x7f4fcc06b500 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.638+0000 7f4fca000640 1 -- 172.21.15.97:0/3023849661 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc402cb80 con 0x7f4fcc06b500 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.638+0000 7f4fca000640 1 -- 172.21.15.97:0/3023849661 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40343f0 con 0x7f4fcc06b500 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 -- 172.21.15.97:0/3023849661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc06b500 msgr2=0x7f4fcc0b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 --2- 172.21.15.97:0/3023849661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc06b500 0x7f4fcc0b5530 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc070eb0 tx=0x7f4fc402be40 comp rx=0 tx=0).stop 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 -- 172.21.15.97:0/3023849661 shutdown_connections 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 --2- 172.21.15.97:0/3023849661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc06b500 0x7f4fcc0b5530 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 --2- 172.21.15.97:0/3023849661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc160530 0x7f4fcc06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 --2- 172.21.15.97:0/3023849661 >> v2:172.21.15.135:3300/0 conn(0x7f4fcc0d0ca0 0x7f4fcc0cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 -- 172.21.15.97:0/3023849661 >> 172.21.15.97:0/3023849661 conn(0x7f4fcc0b1e10 msgr2=0x7f4fcc0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 -- 172.21.15.97:0/3023849661 shutdown_connections 2024-03-06T07:31:54.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.639+0000 7f4fd1600640 1 -- 172.21.15.97:0/3023849661 wait complete. 2024-03-06T07:31:54.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.641+0000 7f4fd1600640 1 Processor -- start 2024-03-06T07:31:54.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.641+0000 7f4fd1600640 1 -- start start 2024-03-06T07:31:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fd1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc06b500 0x7f4fcc0999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fd1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 0x7f4fcc0af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fd1600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fcc160530 0x7f4fcc06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fd1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0d1080 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fd1600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc160990 con 0x7f4fcc160530 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fd1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0b5a70 con 0x7f4fcc06b500 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fc2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 0x7f4fcc0af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fc2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 0x7f4fcc0af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43800/0 (socket says 172.21.15.97:43800) 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fc2000640 1 -- 172.21.15.97:0/648503844 learned_addr learned my addr 172.21.15.97:0/648503844 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fcaa00640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc06b500 0x7f4fcc0999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.642+0000 7f4fcb400640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fcc160530 0x7f4fcc06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc2000640 1 -- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc06b500 msgr2=0x7f4fcc0999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc2000640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc06b500 0x7f4fcc0999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc2000640 1 -- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fcc160530 msgr2=0x7f4fcc06e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc2000640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fcc160530 0x7f4fcc06e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc2000640 1 -- 172.21.15.97:0/648503844 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc0ad0f0 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc2000640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 0x7f4fcc0af470 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400d5f0 tx=0x7f4fb400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb40125f0 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb4019070 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb40129d0 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc1e7f70 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.643+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc1e8530 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.644+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc04f330 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.645+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fb4012b70 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.646+0000 7f4fc3400640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa806b4d0 0x7f4fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.646+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4fb408a670 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.646+0000 7f4fcaa00640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa806b4d0 0x7f4fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.647+0000 7f4fcaa00640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa806b4d0 0x7f4fa806d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc006fd0 tx=0x7f4fbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.651+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fb4059490 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.848+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fcc0b2ae0 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.850+0000 7f4fc3400640 1 -- 172.21.15.97:0/648503844 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f4fb4058d90 con 0x7f4fcc0d0ca0 2024-03-06T07:31:54.853 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:54.853 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":71,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:28.452973+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.30","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:54.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa806b4d0 msgr2=0x7f4fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa806b4d0 0x7f4fa806d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc006fd0 tx=0x7f4fbc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 msgr2=0x7f4fcc0af470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 0x7f4fcc0af470 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400d5f0 tx=0x7f4fb400dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 shutdown_connections 2024-03-06T07:31:54.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa806b4d0 0x7f4fa806d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fcc160530 0x7f4fcc06e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fcc0d0ca0 0x7f4fcc0af470 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 --2- 172.21.15.97:0/648503844 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fcc06b500 0x7f4fcc0999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 >> 172.21.15.97:0/648503844 conn(0x7f4fcc0b1e10 msgr2=0x7f4fcc0cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 shutdown_connections 2024-03-06T07:31:54.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:54.853+0000 7f4fd1600640 1 -- 172.21.15.97:0/648503844 wait complete. 2024-03-06T07:31:54.875 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:54.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.069+0000 7fb5e4200640 1 Processor -- start 2024-03-06T07:31:55.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.069+0000 7fb5e4200640 1 -- start start 2024-03-06T07:31:55.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e4200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb5dc15f9d0 0x7fb5dc15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e4200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5dc070b10 0x7fb5dc070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e4200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc06cef0 con 0x7fb5dc144c00 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e4200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc06d1d0 con 0x7fb5dc070b10 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e4200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc06d4b0 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e2200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5dc070b10 0x7fb5dc070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e2200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5dc070b10 0x7fb5dc070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43816/0 (socket says 172.21.15.97:43816) 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.071+0000 7fb5e2200640 1 -- 172.21.15.97:0/3690824324 learned_addr learned my addr 172.21.15.97:0/3690824324 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.072+0000 7fb5e0e00640 1 --2- 172.21.15.97:0/3690824324 >> v2:172.21.15.135:3300/0 conn(0x7fb5dc15f9d0 0x7fb5dc15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.072+0000 7fb5e2200640 1 -- 172.21.15.97:0/3690824324 >> v2:172.21.15.135:3300/0 conn(0x7fb5dc15f9d0 msgr2=0x7fb5dc15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.072+0000 7fb5e2200640 1 --2- 172.21.15.97:0/3690824324 >> v2:172.21.15.135:3300/0 conn(0x7fb5dc15f9d0 0x7fb5dc15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.072+0000 7fb5e2200640 1 -- 172.21.15.97:0/3690824324 >> v1:172.21.15.135:6789/0 conn(0x7fb5dc144c00 legacy=0x7fb5dc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.072+0000 7fb5e2200640 1 -- 172.21.15.97:0/3690824324 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc06d790 con 0x7fb5dc070b10 2024-03-06T07:31:55.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.072+0000 7fb5e0e00640 1 --2- 172.21.15.97:0/3690824324 >> v2:172.21.15.135:3300/0 conn(0x7fb5dc15f9d0 0x7fb5dc15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.073+0000 7fb5e2200640 1 --2- 172.21.15.97:0/3690824324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc070b10 0x7fb5dc070f20 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0b26f0 tx=0x7fb5d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb939be1a1733776 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.073+0000 7fb5dbe00640 1 -- 172.21.15.97:0/3690824324 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d402ca80 con 0x7fb5dc070b10 2024-03-06T07:31:55.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.073+0000 7fb5dbe00640 1 -- 172.21.15.97:0/3690824324 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d402cc20 con 0x7fb5dc070b10 2024-03-06T07:31:55.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.073+0000 7fb5dbe00640 1 -- 172.21.15.97:0/3690824324 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40343f0 con 0x7fb5dc070b10 2024-03-06T07:31:55.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 -- 172.21.15.97:0/3690824324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc070b10 msgr2=0x7fb5dc070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3690824324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc070b10 0x7fb5dc070f20 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0b26f0 tx=0x7fb5d402bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 -- 172.21.15.97:0/3690824324 shutdown_connections 2024-03-06T07:31:55.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3690824324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc070b10 0x7fb5dc070f20 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3690824324 >> v2:172.21.15.135:3300/0 conn(0x7fb5dc15f9d0 0x7fb5dc15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 -- 172.21.15.97:0/3690824324 >> 172.21.15.97:0/3690824324 conn(0x7fb5dc0b1730 msgr2=0x7fb5dc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 -- 172.21.15.97:0/3690824324 shutdown_connections 2024-03-06T07:31:55.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.074+0000 7fb5e4200640 1 -- 172.21.15.97:0/3690824324 wait complete. 2024-03-06T07:31:55.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.076+0000 7fb5e4200640 1 Processor -- start 2024-03-06T07:31:55.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.076+0000 7fb5e4200640 1 -- start start 2024-03-06T07:31:55.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e4200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5dc070b10 0x7fb5dc15e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e4200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5dc144c00 0x7fb5dc142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e4200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 0x7fb5dc1396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e4200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc06ccd0 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e4200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc06cb50 con 0x7fb5dc070b10 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e4200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc06df90 con 0x7fb5dc144c00 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 0x7fb5dc1396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 0x7fb5dc1396e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43832/0 (socket says 172.21.15.97:43832) 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 -- 172.21.15.97:0/3031389103 learned_addr learned my addr 172.21.15.97:0/3031389103 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e1800640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5dc070b10 0x7fb5dc15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e0e00640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5dc144c00 0x7fb5dc142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 -- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5dc144c00 msgr2=0x7fb5dc142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5dc144c00 0x7fb5dc142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 -- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5dc070b10 msgr2=0x7fb5dc15e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5dc070b10 0x7fb5dc15e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e2200640 1 -- 172.21.15.97:0/3031389103 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc15d3d0 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.077+0000 7fb5e0e00640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5dc144c00 0x7fb5dc142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.078+0000 7fb5e2200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 0x7fb5dc1396e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4036040 tx=0x7fb5d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.078+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4043020 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.078+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d403e450 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.078+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4009510 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.078+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc15d6b0 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.078+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc15dc70 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.079+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc04f330 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.080+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5d4046020 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.081+0000 7fb5d9600640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5c006b4d0 0x7fb5c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.081+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb5d40acc00 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.081+0000 7fb5e1800640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5c006b4d0 0x7fb5c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.082+0000 7fb5e1800640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5c006b4d0 0x7fb5c006d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0004ea0 tx=0x7fb5d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.086+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5d404d030 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.367+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5dc0b4cb0 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.367+0000 7fb5d9600640 1 -- 172.21.15.97:0/3031389103 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fb5d407b9a0 con 0x7fb5dc15f9d0 2024-03-06T07:31:55.369 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:55.370 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:55.370 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:55.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5c006b4d0 msgr2=0x7fb5c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5c006b4d0 0x7fb5c006d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0004ea0 tx=0x7fb5d0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 msgr2=0x7fb5dc1396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 0x7fb5dc1396e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4036040 tx=0x7fb5d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:55.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 shutdown_connections 2024-03-06T07:31:55.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5c006b4d0 0x7fb5c006d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5dc15f9d0 0x7fb5dc1396e0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5dc144c00 0x7fb5dc142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 --2- 172.21.15.97:0/3031389103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5dc070b10 0x7fb5dc15e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 >> 172.21.15.97:0/3031389103 conn(0x7fb5dc0b1730 msgr2=0x7fb5dc06e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 shutdown_connections 2024-03-06T07:31:55.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.370+0000 7fb5e4200640 1 -- 172.21.15.97:0/3031389103 wait complete. 2024-03-06T07:31:55.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.590+0000 7f051fe00640 1 Processor -- start 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.591+0000 7f051fe00640 1 -- start start 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f052015a4b0 0x7f0520071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200b44a0 0x7f05200cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051fe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052006c940 con 0x7f05200cbb60 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051fe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05200ce680 con 0x7f052015a4b0 2024-03-06T07:31:55.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05200cec40 con 0x7f05200b44a0 2024-03-06T07:31:55.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f052015a4b0 0x7f0520071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f052015a4b0 0x7f0520071380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43846/0 (socket says 172.21.15.97:43846) 2024-03-06T07:31:55.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.593+0000 7f051ea00640 1 -- 172.21.15.97:0/4001994977 learned_addr learned my addr 172.21.15.97:0/4001994977 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:55.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.594+0000 7f051f400640 1 --2- 172.21.15.97:0/4001994977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200b44a0 0x7f05200cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.594+0000 7f051ea00640 1 -- 172.21.15.97:0/4001994977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200b44a0 msgr2=0x7f05200cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.594+0000 7f051ea00640 1 --2- 172.21.15.97:0/4001994977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200b44a0 0x7f05200cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.594+0000 7f051ea00640 1 -- 172.21.15.97:0/4001994977 >> v1:172.21.15.135:6789/0 conn(0x7f05200cbb60 legacy=0x7f05200b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.594+0000 7f051ea00640 1 -- 172.21.15.97:0/4001994977 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520071980 con 0x7f052015a4b0 2024-03-06T07:31:55.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.595+0000 7f051ea00640 1 --2- 172.21.15.97:0/4001994977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f052015a4b0 0x7f0520071380 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f051000aa10 tx=0x7f051002dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06fb80f0b7e99b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.595+0000 7f051d600640 1 -- 172.21.15.97:0/4001994977 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051003b070 con 0x7f052015a4b0 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.595+0000 7f051d600640 1 -- 172.21.15.97:0/4001994977 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051002eb50 con 0x7f052015a4b0 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.595+0000 7f051d600640 1 -- 172.21.15.97:0/4001994977 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051000b040 con 0x7f052015a4b0 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 -- 172.21.15.97:0/4001994977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f052015a4b0 msgr2=0x7f0520071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 --2- 172.21.15.97:0/4001994977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f052015a4b0 0x7f0520071380 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f051000aa10 tx=0x7f051002dda0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 -- 172.21.15.97:0/4001994977 shutdown_connections 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 --2- 172.21.15.97:0/4001994977 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200b44a0 0x7f05200cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 --2- 172.21.15.97:0/4001994977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f052015a4b0 0x7f0520071380 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 -- 172.21.15.97:0/4001994977 >> 172.21.15.97:0/4001994977 conn(0x7f052006afc0 msgr2=0x7f052006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 -- 172.21.15.97:0/4001994977 shutdown_connections 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.596+0000 7f051fe00640 1 -- 172.21.15.97:0/4001994977 wait complete. 2024-03-06T07:31:55.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.598+0000 7f051fe00640 1 Processor -- start 2024-03-06T07:31:55.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.598+0000 7f051fe00640 1 -- start start 2024-03-06T07:31:55.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 0x7f05200b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200cbb60 0x7f05201f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f052015a4b0 0x7f052015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05200cbfa0 con 0x7f05200b44a0 2024-03-06T07:31:55.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05200ce8d0 con 0x7f052015a4b0 2024-03-06T07:31:55.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05200cc120 con 0x7f05200cbb60 2024-03-06T07:31:55.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 0x7f05200b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 0x7f05200b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43858/0 (socket says 172.21.15.97:43858) 2024-03-06T07:31:55.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.599+0000 7f051ea00640 1 -- 172.21.15.97:0/1518981855 learned_addr learned my addr 172.21.15.97:0/1518981855 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:55.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051ea00640 1 -- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200cbb60 msgr2=0x7f05201f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051ea00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200cbb60 0x7f05201f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051ea00640 1 -- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f052015a4b0 msgr2=0x7f052015b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051f400640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f052015a4b0 0x7f052015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051ea00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f052015a4b0 0x7f052015b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051ea00640 1 -- 172.21.15.97:0/1518981855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05200ada00 con 0x7f05200b44a0 2024-03-06T07:31:55.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.600+0000 7f051ea00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 0x7f05200b2320 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f051000a1c0 tx=0x7f0510004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.601+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051003b030 con 0x7f05200b44a0 2024-03-06T07:31:55.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.601+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051003d070 con 0x7f05200b44a0 2024-03-06T07:31:55.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.601+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05200adce0 con 0x7f05200b44a0 2024-03-06T07:31:55.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.601+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510034790 con 0x7f05200b44a0 2024-03-06T07:31:55.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.601+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05200ae2a0 con 0x7f05200b44a0 2024-03-06T07:31:55.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.602+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052004f330 con 0x7f05200b44a0 2024-03-06T07:31:55.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.604+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f051003f030 con 0x7f05200b44a0 2024-03-06T07:31:55.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.605+0000 7f0516a00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050006b4d0 0x7f050006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.605+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05100abe30 con 0x7f05200b44a0 2024-03-06T07:31:55.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.605+0000 7f051e000640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050006b4d0 0x7f050006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.606+0000 7f051e000640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050006b4d0 0x7f050006d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f052006cf50 tx=0x7f050c0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.610+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f052006c940 con 0x7f05200b44a0 2024-03-06T07:31:55.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.845+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0520071fd0 con 0x7f05200b44a0 2024-03-06T07:31:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.846+0000 7f0516a00640 1 -- 172.21.15.97:0/1518981855 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f051007ac60 con 0x7f05200b44a0 2024-03-06T07:31:55.848 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:55.849 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:09.132065+0000","modified":"2024-03-06T07:31:12.252322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3151111028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3151111028},{"type":"v1","addr":"172.21.15.97:6837","nonce":3151111028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":5}]} 2024-03-06T07:31:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 43 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050006b4d0 msgr2=0x7f050006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050006b4d0 0x7f050006d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f052006cf50 tx=0x7f050c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 msgr2=0x7f05200b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 0x7f05200b2320 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f051000a1c0 tx=0x7f0510004730 comp rx=0 tx=0).stop 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 shutdown_connections 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f050006b4d0 0x7f050006d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f052015a4b0 0x7f052015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f05200cbb60 0x7f05201f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 --2- 172.21.15.97:0/1518981855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05200b44a0 0x7f05200b2320 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.848+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 >> 172.21.15.97:0/1518981855 conn(0x7f052006afc0 msgr2=0x7f05200cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.849+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 shutdown_connections 2024-03-06T07:31:55.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:55.849+0000 7f051fe00640 1 -- 172.21.15.97:0/1518981855 wait complete. 2024-03-06T07:31:55.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.063+0000 7f5e3b800640 1 Processor -- start 2024-03-06T07:31:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.063+0000 7f5e3b800640 1 -- start start 2024-03-06T07:31:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e3b800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3406cb50 0x7f5e3406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e3b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340b2d10 0x7f5e340b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e3b800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e340717c0 con 0x7f5e340cf430 2024-03-06T07:31:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e3b800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e34071aa0 con 0x7f5e3406cb50 2024-03-06T07:31:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e3b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34071d80 con 0x7f5e340b2d10 2024-03-06T07:31:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e2be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3406cb50 0x7f5e3406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e38e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5e340cf430 0x7f5e34070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45354/0 (socket says 172.21.15.97:45354) 2024-03-06T07:31:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.065+0000 7f5e38e00640 1 -- 172.21.15.97:0/4015223595 learned_addr learned my addr 172.21.15.97:0/4015223595 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:31:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.066+0000 7f5e39800640 1 --2- 172.21.15.97:0/4015223595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340b2d10 0x7f5e340b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.066+0000 7f5e2be00640 1 -- 172.21.15.97:0/4015223595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340b2d10 msgr2=0x7f5e340b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.066+0000 7f5e2be00640 1 --2- 172.21.15.97:0/4015223595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340b2d10 0x7f5e340b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.066+0000 7f5e2be00640 1 -- 172.21.15.97:0/4015223595 >> v1:172.21.15.97:6789/0 conn(0x7f5e340cf430 legacy=0x7f5e34070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.066+0000 7f5e2be00640 1 -- 172.21.15.97:0/4015223595 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e34072060 con 0x7f5e3406cb50 2024-03-06T07:31:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.066+0000 7f5e33e00640 1 -- 172.21.15.97:0/4015223595 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434409630 0 0) 0x7f5e340717c0 con 0x7f5e340cf430 2024-03-06T07:31:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.067+0000 7f5e2be00640 1 --2- 172.21.15.97:0/4015223595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3406cfe0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0097a0 tx=0x7f5e1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2ee1e50327d688 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.067+0000 7f5e33e00640 1 -- 172.21.15.97:0/4015223595 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c035020 con 0x7f5e3406cb50 2024-03-06T07:31:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.067+0000 7f5e33e00640 1 -- 172.21.15.97:0/4015223595 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c031830 con 0x7f5e3406cb50 2024-03-06T07:31:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.067+0000 7f5e33e00640 1 -- 172.21.15.97:0/4015223595 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c031c00 con 0x7f5e3406cb50 2024-03-06T07:31:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 -- 172.21.15.97:0/4015223595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 msgr2=0x7f5e3406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 --2- 172.21.15.97:0/4015223595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3406cfe0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0097a0 tx=0x7f5e1c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 -- 172.21.15.97:0/4015223595 shutdown_connections 2024-03-06T07:31:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 --2- 172.21.15.97:0/4015223595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340b2d10 0x7f5e340b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 --2- 172.21.15.97:0/4015223595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3406cfe0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 -- 172.21.15.97:0/4015223595 >> 172.21.15.97:0/4015223595 conn(0x7f5e340b1730 msgr2=0x7f5e340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 -- 172.21.15.97:0/4015223595 shutdown_connections 2024-03-06T07:31:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.068+0000 7f5e3b800640 1 -- 172.21.15.97:0/4015223595 wait complete. 2024-03-06T07:31:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.070+0000 7f5e3b800640 1 Processor -- start 2024-03-06T07:31:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.070+0000 7f5e3b800640 1 -- start start 2024-03-06T07:31:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e3b800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3414e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e3b800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e340b2d10 0x7f5e3415f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e3b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340cf430 0x7f5e3415d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e3b800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34162b70 con 0x7f5e3406cb50 2024-03-06T07:31:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e3b800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34071430 con 0x7f5e340b2d10 2024-03-06T07:31:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e3b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34162cf0 con 0x7f5e340cf430 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e38e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3414e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e38e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3414e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43876/0 (socket says 172.21.15.97:43876) 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.071+0000 7f5e38e00640 1 -- 172.21.15.97:0/1827497369 learned_addr learned my addr 172.21.15.97:0/1827497369 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e2be00640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e340b2d10 0x7f5e3415f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e38e00640 1 -- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340cf430 msgr2=0x7f5e3415d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e38e00640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340cf430 0x7f5e3415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e38e00640 1 -- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e340b2d10 msgr2=0x7f5e3415f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e38e00640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e340b2d10 0x7f5e3415f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e38e00640 1 -- 172.21.15.97:0/1827497369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3415f960 con 0x7f5e3406cb50 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e38e00640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3414e640 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c430 tx=0x7f5e2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240074e0 con 0x7f5e3406cb50 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e240047e0 con 0x7f5e3406cb50 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24004bd0 con 0x7f5e3406cb50 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3415fc40 con 0x7f5e3406cb50 2024-03-06T07:31:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.072+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3414f390 con 0x7f5e3406cb50 2024-03-06T07:31:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.073+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3404f330 con 0x7f5e3406cb50 2024-03-06T07:31:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.074+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e24011070 con 0x7f5e3406cb50 2024-03-06T07:31:56.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.076+0000 7f5e31600640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1006b4d0 0x7f5e1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.076+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5e24089ed0 con 0x7f5e3406cb50 2024-03-06T07:31:56.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.076+0000 7f5e2be00640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1006b4d0 0x7f5e1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.077+0000 7f5e2be00640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1006b4d0 0x7f5e1006d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c036000 tx=0x7f5e1c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.082+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e24058d00 con 0x7f5e3406cb50 2024-03-06T07:31:56.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.276+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e340b4a40 con 0x7f5e3406cb50 2024-03-06T07:31:56.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.278+0000 7f5e31600640 1 -- 172.21.15.97:0/1827497369 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f5e24058600 con 0x7f5e3406cb50 2024-03-06T07:31:56.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:31:56.281 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":71,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:28.452973+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:06.960932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:08.155371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.30","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.280+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1006b4d0 msgr2=0x7f5e1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.280+0000 7f5e3b800640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1006b4d0 0x7f5e1006d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c036000 tx=0x7f5e1c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.280+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 msgr2=0x7f5e3414e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.280+0000 7f5e3b800640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3414e640 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c430 tx=0x7f5e2400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 shutdown_connections 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e1006b4d0 0x7f5e1006d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e340cf430 0x7f5e3415d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e340b2d10 0x7f5e3415f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 --2- 172.21.15.97:0/1827497369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3406cb50 0x7f5e3414e640 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 >> 172.21.15.97:0/1827497369 conn(0x7f5e340b1730 msgr2=0x7f5e340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 shutdown_connections 2024-03-06T07:31:56.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.281+0000 7f5e3b800640 1 -- 172.21.15.97:0/1827497369 wait complete. 2024-03-06T07:31:56.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:56.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.496+0000 7f4427800640 1 Processor -- start 2024-03-06T07:31:56.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.496+0000 7f4427800640 1 -- start start 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.498+0000 7f4427800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f442015ac90 0x7f442006e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4427800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015b070 0x7f442015ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4427800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f442015ef50 0x7f442015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4427800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44200cc000 con 0x7f442015ac90 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4427800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44200ce360 con 0x7f442015ef50 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4427800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44200ce640 con 0x7f442015b070 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f442015ac90 0x7f442006e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f442015ac90 0x7f442006e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43892/0 (socket says 172.21.15.97:43892) 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 -- 172.21.15.97:0/3448724613 learned_addr learned my addr 172.21.15.97:0/3448724613 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f441fe00640 1 --2- 172.21.15.97:0/3448724613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015b070 0x7f442015ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4425800640 1 --2- 172.21.15.97:0/3448724613 >> v2:172.21.15.135:3300/0 conn(0x7f442015ef50 0x7f442015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 -- 172.21.15.97:0/3448724613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015b070 msgr2=0x7f442015ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 --2- 172.21.15.97:0/3448724613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015b070 0x7f442015ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 -- 172.21.15.97:0/3448724613 >> v2:172.21.15.135:3300/0 conn(0x7f442015ef50 msgr2=0x7f442015f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 --2- 172.21.15.97:0/3448724613 >> v2:172.21.15.135:3300/0 conn(0x7f442015ef50 0x7f442015f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4424e00640 1 -- 172.21.15.97:0/3448724613 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44200ce920 con 0x7f442015ac90 2024-03-06T07:31:56.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.499+0000 7f4425800640 1 --2- 172.21.15.97:0/3448724613 >> v2:172.21.15.135:3300/0 conn(0x7f442015ef50 0x7f442015f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.501+0000 7f4424e00640 1 --2- 172.21.15.97:0/3448724613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ac90 0x7f442006e580 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f44200cd500 tx=0x7f441402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c98d741ba5f7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.501+0000 7f441f400640 1 -- 172.21.15.97:0/3448724613 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44140317d0 con 0x7f442015ac90 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.501+0000 7f441f400640 1 -- 172.21.15.97:0/3448724613 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4414037070 con 0x7f442015ac90 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.501+0000 7f441f400640 1 -- 172.21.15.97:0/3448724613 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414031b90 con 0x7f442015ac90 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 -- 172.21.15.97:0/3448724613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ac90 msgr2=0x7f442006e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 --2- 172.21.15.97:0/3448724613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ac90 0x7f442006e580 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f44200cd500 tx=0x7f441402b870 comp rx=0 tx=0).stop 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 -- 172.21.15.97:0/3448724613 shutdown_connections 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 --2- 172.21.15.97:0/3448724613 >> v2:172.21.15.135:3300/0 conn(0x7f442015ef50 0x7f442015f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 --2- 172.21.15.97:0/3448724613 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015b070 0x7f442015ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 --2- 172.21.15.97:0/3448724613 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ac90 0x7f442006e580 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 -- 172.21.15.97:0/3448724613 >> 172.21.15.97:0/3448724613 conn(0x7f44200cc4f0 msgr2=0x7f44200cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 -- 172.21.15.97:0/3448724613 shutdown_connections 2024-03-06T07:31:56.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.502+0000 7f4427800640 1 -- 172.21.15.97:0/3448724613 wait complete. 2024-03-06T07:31:56.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.505+0000 7f4427800640 1 Processor -- start 2024-03-06T07:31:56.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.505+0000 7f4427800640 1 -- start start 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4427800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015ac90 0x7f4420154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4427800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f442015b070 0x7f442015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4427800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 0x7f442015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4427800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44200cac60 con 0x7f442015ef50 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4427800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44200ceee0 con 0x7f442015b070 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4427800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44200cade0 con 0x7f442015ac90 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4425800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 0x7f442015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4425800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 0x7f442015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43896/0 (socket says 172.21.15.97:43896) 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.506+0000 7f4425800640 1 -- 172.21.15.97:0/120576344 learned_addr learned my addr 172.21.15.97:0/120576344 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4424e00640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015ac90 0x7f4420154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f441fe00640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f442015b070 0x7f442015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4425800640 1 -- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015ac90 msgr2=0x7f4420154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4425800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015ac90 0x7f4420154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4425800640 1 -- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f442015b070 msgr2=0x7f442015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4425800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f442015b070 0x7f442015b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4425800640 1 -- 172.21.15.97:0/120576344 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44201f3cb0 con 0x7f442015ef50 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4425800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 0x7f442015bee0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f441800cca0 tx=0x7f441800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441800f680 con 0x7f442015ef50 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441800f840 con 0x7f442015ef50 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441800fc30 con 0x7f442015ef50 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.507+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44201f3f50 con 0x7f442015ef50 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.508+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44201f4510 con 0x7f442015ef50 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.509+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442004f330 con 0x7f442015ef50 2024-03-06T07:31:56.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.510+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f441800e3e0 con 0x7f442015ef50 2024-03-06T07:31:56.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.510+0000 7f441cc00640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f440006b4d0 0x7f440006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.511+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f441808afe0 con 0x7f442015ef50 2024-03-06T07:31:56.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.511+0000 7f4424e00640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f440006b4d0 0x7f440006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.511+0000 7f4424e00640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f440006b4d0 0x7f440006d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4414036000 tx=0x7f441403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.519+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4418014030 con 0x7f442015ef50 2024-03-06T07:31:56.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.755+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f44200cd1c0 con 0x7f442015ef50 2024-03-06T07:31:56.758 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:31:56.756+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.759+0000 7f441cc00640 1 -- 172.21.15.97:0/120576344 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4418059de0 con 0x7f442015ef50 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:56.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f440006b4d0 msgr2=0x7f440006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f440006b4d0 0x7f440006d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4414036000 tx=0x7f441403d040 comp rx=0 tx=0).stop 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 msgr2=0x7f442015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 0x7f442015bee0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f441800cca0 tx=0x7f441800a650 comp rx=0 tx=0).stop 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 shutdown_connections 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f440006b4d0 0x7f440006d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f442015ef50 0x7f442015bee0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f442015b070 0x7f442015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 --2- 172.21.15.97:0/120576344 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f442015ac90 0x7f4420154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.762+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 >> 172.21.15.97:0/120576344 conn(0x7f44200cc4f0 msgr2=0x7f44200b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.763+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 shutdown_connections 2024-03-06T07:31:56.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.764+0000 7f4427800640 1 -- 172.21.15.97:0/120576344 wait complete. 2024-03-06T07:31:56.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:56.869 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:31:56.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.966+0000 7f5003400640 1 Processor -- start 2024-03-06T07:31:56.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.966+0000 7f5003400640 1 -- start start 2024-03-06T07:31:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.967+0000 7f5003400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc06d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5003400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4ffc163800 0x7f4ffc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5003400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc15cf70 con 0x7f4ffc06e060 2024-03-06T07:31:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5003400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0b3890 con 0x7f4ffc163800 2024-03-06T07:31:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5003400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06c990 con 0x7f4ffc15cb90 2024-03-06T07:31:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5000a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc06d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5000a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc06d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:40546/0 (socket says 172.21.15.97:40546) 2024-03-06T07:31:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f5000a00640 1 -- 172.21.15.97:0/4219711536 learned_addr learned my addr 172.21.15.97:0/4219711536 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.968+0000 7f4ff3e00640 1 --2- 172.21.15.97:0/4219711536 >> v2:172.21.15.135:3300/0 conn(0x7f4ffc163800 0x7f4ffc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.969+0000 7f5000a00640 1 -- 172.21.15.97:0/4219711536 >> v2:172.21.15.135:3300/0 conn(0x7f4ffc163800 msgr2=0x7f4ffc0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.969+0000 7f5000a00640 1 --2- 172.21.15.97:0/4219711536 >> v2:172.21.15.135:3300/0 conn(0x7f4ffc163800 0x7f4ffc0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.969+0000 7f5000a00640 1 -- 172.21.15.97:0/4219711536 >> v1:172.21.15.135:6789/0 conn(0x7f4ffc06e060 legacy=0x7f4ffc0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.969+0000 7f5000a00640 1 -- 172.21.15.97:0/4219711536 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc0b35b0 con 0x7f4ffc15cb90 2024-03-06T07:31:56.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.969+0000 7f4ff3e00640 1 --2- 172.21.15.97:0/4219711536 >> v2:172.21.15.135:3300/0 conn(0x7f4ffc163800 0x7f4ffc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:56.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.971+0000 7f5000a00640 1 --2- 172.21.15.97:0/4219711536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc06d6c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009e50 tx=0x7f4fec02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb362b2155c51ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.971+0000 7f4ffbe00640 1 -- 172.21.15.97:0/4219711536 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec02cb00 con 0x7f4ffc15cb90 2024-03-06T07:31:56.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.971+0000 7f4ffbe00640 1 -- 172.21.15.97:0/4219711536 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec02cca0 con 0x7f4ffc15cb90 2024-03-06T07:31:56.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.971+0000 7f4ffbe00640 1 -- 172.21.15.97:0/4219711536 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec034610 con 0x7f4ffc15cb90 2024-03-06T07:31:56.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 -- 172.21.15.97:0/4219711536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 msgr2=0x7f4ffc06d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 --2- 172.21.15.97:0/4219711536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc06d6c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009e50 tx=0x7f4fec02bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:56.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 -- 172.21.15.97:0/4219711536 shutdown_connections 2024-03-06T07:31:56.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 --2- 172.21.15.97:0/4219711536 >> v2:172.21.15.135:3300/0 conn(0x7f4ffc163800 0x7f4ffc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 --2- 172.21.15.97:0/4219711536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc06d6c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 -- 172.21.15.97:0/4219711536 >> 172.21.15.97:0/4219711536 conn(0x7f4ffc0b1940 msgr2=0x7f4ffc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 -- 172.21.15.97:0/4219711536 shutdown_connections 2024-03-06T07:31:56.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.972+0000 7f5003400640 1 -- 172.21.15.97:0/4219711536 wait complete. 2024-03-06T07:31:56.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.974+0000 7f5003400640 1 Processor -- start 2024-03-06T07:31:56.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.974+0000 7f5003400640 1 -- start start 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.975+0000 7f5003400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ffc06e060 0x7f4ffc155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.975+0000 7f5003400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc09a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.975+0000 7f5003400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 0x7f4ffc0ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.975+0000 7f5003400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06cec0 con 0x7f4ffc163800 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.975+0000 7f5003400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06d230 con 0x7f4ffc06e060 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.975+0000 7f5003400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06dc00 con 0x7f4ffc15cb90 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 0x7f4ffc0ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 0x7f4ffc0ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43916/0 (socket says 172.21.15.97:43916) 2024-03-06T07:31:56.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 -- 172.21.15.97:0/3617513462 learned_addr learned my addr 172.21.15.97:0/3617513462 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:56.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5000a00640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ffc06e060 0x7f4ffc155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 -- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 msgr2=0x7f4ffc09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc09a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 -- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ffc06e060 msgr2=0x7f4ffc155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ffc06e060 0x7f4ffc155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 -- 172.21.15.97:0/3617513462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc09b4f0 con 0x7f4ffc163800 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.976+0000 7f5001400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 0x7f4ffc0ae010 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4001eb0 tx=0x7f4ff400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.977+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4017070 con 0x7f4ffc163800 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.977+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff40052d0 con 0x7f4ffc163800 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.977+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4003810 con 0x7f4ffc163800 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.977+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc09b7d0 con 0x7f4ffc163800 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.977+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc09bd90 con 0x7f4ffc163800 2024-03-06T07:31:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.978+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc04f330 con 0x7f4ffc163800 2024-03-06T07:31:56.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.979+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ff401a070 con 0x7f4ffc163800 2024-03-06T07:31:56.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.980+0000 7f4ff9600640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fdc06b4d0 0x7f4fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.980+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4ff4089620 con 0x7f4ffc163800 2024-03-06T07:31:56.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.980+0000 7f5000a00640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fdc06b4d0 0x7f4fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.981+0000 7f5000a00640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fdc06b4d0 0x7f4fdc06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4fec035000 tx=0x7f4fec030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:56.985+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ff4058410 con 0x7f4ffc163800 2024-03-06T07:31:57.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.207+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4ffc163e30 con 0x7f4ffc163800 2024-03-06T07:31:57.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.763+0000 7f4ff9600640 1 -- 172.21.15.97:0/3617513462 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4ff4057d10 con 0x7f4ffc163800 2024-03-06T07:31:57.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fdc06b4d0 msgr2=0x7f4fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fdc06b4d0 0x7f4fdc06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4fec035000 tx=0x7f4fec030000 comp rx=0 tx=0).stop 2024-03-06T07:31:57.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 msgr2=0x7f4ffc0ae010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 0x7f4ffc0ae010 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4001eb0 tx=0x7f4ff400c890 comp rx=0 tx=0).stop 2024-03-06T07:31:57.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 shutdown_connections 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fdc06b4d0 0x7f4fdc06d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ffc163800 0x7f4ffc0ae010 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4ffc15cb90 0x7f4ffc09a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 --2- 172.21.15.97:0/3617513462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ffc06e060 0x7f4ffc155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 >> 172.21.15.97:0/3617513462 conn(0x7f4ffc0b1940 msgr2=0x7f4ffc0af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 shutdown_connections 2024-03-06T07:31:57.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.767+0000 7f5003400640 1 -- 172.21.15.97:0/3617513462 wait complete. 2024-03-06T07:31:57.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:31:57.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.996+0000 7f2a3f400640 1 Processor -- start 2024-03-06T07:31:57.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.997+0000 7f2a3f400640 1 -- start start 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.998+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a38145750 0x7f2a38072260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.998+0000 7f2a3f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2a380727a0 0x7f2a3806d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.998+0000 7f2a3f400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a380cdbd0 con 0x7f2a380cd450 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.998+0000 7f2a3f400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a380cd980 con 0x7f2a380727a0 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.998+0000 7f2a3f400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380cdeb0 con 0x7f2a38145750 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2a380727a0 0x7f2a3806d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2a380727a0 0x7f2a3806d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43918/0 (socket says 172.21.15.97:43918) 2024-03-06T07:31:58.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 -- 172.21.15.97:0/3961620025 learned_addr learned my addr 172.21.15.97:0/3961620025 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:58.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a3ca00640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a38145750 0x7f2a38072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 -- 172.21.15.97:0/3961620025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a38145750 msgr2=0x7f2a38072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a38145750 0x7f2a38072260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 -- 172.21.15.97:0/3961620025 >> v1:172.21.15.135:6789/0 conn(0x7f2a380cd450 legacy=0x7f2a380ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a37e00640 1 -- 172.21.15.97:0/3961620025 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a380ce190 con 0x7f2a380727a0 2024-03-06T07:31:58.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:57.999+0000 7f2a3ca00640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a38145750 0x7f2a38072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.000+0000 7f2a37e00640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a3806d620 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009870 tx=0x7f2a2402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74418248a09c81f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.001+0000 7f2a37400640 1 -- 172.21.15.97:0/3961620025 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a240317c0 con 0x7f2a380727a0 2024-03-06T07:31:58.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.001+0000 7f2a37400640 1 -- 172.21.15.97:0/3961620025 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a24037070 con 0x7f2a380727a0 2024-03-06T07:31:58.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.001+0000 7f2a37400640 1 -- 172.21.15.97:0/3961620025 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a24031b80 con 0x7f2a380727a0 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.001+0000 7f2a3f400640 1 -- 172.21.15.97:0/3961620025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 msgr2=0x7f2a3806d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.001+0000 7f2a3f400640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a3806d620 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009870 tx=0x7f2a2402b860 comp rx=0 tx=0).stop 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.002+0000 7f2a3f400640 1 -- 172.21.15.97:0/3961620025 shutdown_connections 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.002+0000 7f2a3f400640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a3806d620 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.002+0000 7f2a3f400640 1 --2- 172.21.15.97:0/3961620025 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a38145750 0x7f2a38072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.002+0000 7f2a3f400640 1 -- 172.21.15.97:0/3961620025 >> 172.21.15.97:0/3961620025 conn(0x7f2a380cfa80 msgr2=0x7f2a380cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.002+0000 7f2a3f400640 1 -- 172.21.15.97:0/3961620025 shutdown_connections 2024-03-06T07:31:58.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.002+0000 7f2a3f400640 1 -- 172.21.15.97:0/3961620025 wait complete. 2024-03-06T07:31:58.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.003+0000 7f2a3f400640 1 Processor -- start 2024-03-06T07:31:58.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.003+0000 7f2a3f400640 1 -- start start 2024-03-06T07:31:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.004+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a38138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.004+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a380cd450 0x7f2a38140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.004+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a38145750 0x7f2a38156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.004+0000 7f2a3f400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3815c5c0 con 0x7f2a380727a0 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.004+0000 7f2a3f400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3815c440 con 0x7f2a38145750 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.004+0000 7f2a3f400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3806db60 con 0x7f2a380cd450 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a38138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a38138710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43920/0 (socket says 172.21.15.97:43920) 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 -- 172.21.15.97:0/295503676 learned_addr learned my addr 172.21.15.97:0/295503676 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3d400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a38145750 0x7f2a38156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a37e00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a380cd450 0x7f2a38140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 -- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a380cd450 msgr2=0x7f2a38140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a380cd450 0x7f2a38140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 -- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a38145750 msgr2=0x7f2a38156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a38145750 0x7f2a38156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a3ca00640 1 -- 172.21.15.97:0/295503676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3813fef0 con 0x7f2a380727a0 2024-03-06T07:31:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.005+0000 7f2a37e00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a380cd450 0x7f2a38140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.006+0000 7f2a3ca00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a38138710 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00c7f0 tx=0x7f2a2c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.006+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c015070 con 0x7f2a380727a0 2024-03-06T07:31:58.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.006+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2c007820 con 0x7f2a380727a0 2024-03-06T07:31:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.006+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a381401d0 con 0x7f2a380727a0 2024-03-06T07:31:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.006+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c007bd0 con 0x7f2a380727a0 2024-03-06T07:31:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.006+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a38140790 con 0x7f2a380727a0 2024-03-06T07:31:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.007+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3804f3a0 con 0x7f2a380727a0 2024-03-06T07:31:58.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.008+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a2c018070 con 0x7f2a380727a0 2024-03-06T07:31:58.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.009+0000 7f2a34c00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2a1806b4d0 0x7f2a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.009+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2a2c08a4a0 con 0x7f2a380727a0 2024-03-06T07:31:58.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.009+0000 7f2a37e00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2a1806b4d0 0x7f2a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.010+0000 7f2a37e00640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2a1806b4d0 0x7f2a1806d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2a24036000 tx=0x7f2a24008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.014+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a2c0591f0 con 0x7f2a380727a0 2024-03-06T07:31:58.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.211+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2a380b18b0 con 0x7f2a380727a0 2024-03-06T07:31:58.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.773+0000 7f2a34c00640 1 -- 172.21.15.97:0/295503676 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2a2c058af0 con 0x7f2a380727a0 2024-03-06T07:31:58.776 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:31:58.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2a1806b4d0 msgr2=0x7f2a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2a1806b4d0 0x7f2a1806d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2a24036000 tx=0x7f2a24008030 comp rx=0 tx=0).stop 2024-03-06T07:31:58.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 msgr2=0x7f2a38138710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a38138710 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00c7f0 tx=0x7f2a2c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 shutdown_connections 2024-03-06T07:31:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2a1806b4d0 0x7f2a1806d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a38145750 0x7f2a38156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2a380cd450 0x7f2a38140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 --2- 172.21.15.97:0/295503676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a380727a0 0x7f2a38138710 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.777+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 >> 172.21.15.97:0/295503676 conn(0x7f2a380cfa80 msgr2=0x7f2a38141f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.778+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 shutdown_connections 2024-03-06T07:31:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:58.778+0000 7f2a3f400640 1 -- 172.21.15.97:0/295503676 wait complete. 2024-03-06T07:31:58.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:31:59.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.005+0000 7f195f200640 1 Processor -- start 2024-03-06T07:31:59.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.005+0000 7f195f200640 1 -- start start 2024-03-06T07:31:59.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1960144940 0x7f1960070d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f1960071860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f196006afc0 0x7f196006b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196015ffe0 con 0x7f196006afc0 2024-03-06T07:31:59.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19601608d0 con 0x7f1960144940 2024-03-06T07:31:59.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960072090 con 0x7f1960160470 2024-03-06T07:31:59.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f196006afc0 0x7f196006b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f196006afc0 0x7f196006b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43926/0 (socket says 172.21.15.97:43926) 2024-03-06T07:31:59.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195e800640 1 -- 172.21.15.97:0/4088642841 learned_addr learned my addr 172.21.15.97:0/4088642841 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195de00640 1 --2- 172.21.15.97:0/4088642841 >> v2:172.21.15.135:3300/0 conn(0x7f1960144940 0x7f1960070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f1955400640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f1960071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.007+0000 7f195e800640 1 -- 172.21.15.97:0/4088642841 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 msgr2=0x7f1960071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.008+0000 7f195e800640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f1960071860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.008+0000 7f195e800640 1 -- 172.21.15.97:0/4088642841 >> v2:172.21.15.135:3300/0 conn(0x7f1960144940 msgr2=0x7f1960070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.008+0000 7f195e800640 1 --2- 172.21.15.97:0/4088642841 >> v2:172.21.15.135:3300/0 conn(0x7f1960144940 0x7f1960070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.008+0000 7f195e800640 1 -- 172.21.15.97:0/4088642841 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960072370 con 0x7f196006afc0 2024-03-06T07:31:59.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.008+0000 7f195de00640 1 --2- 172.21.15.97:0/4088642841 >> v2:172.21.15.135:3300/0 conn(0x7f1960144940 0x7f1960070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.008+0000 7f1955400640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f1960071860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.009+0000 7f195e800640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f196006b450 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f19600cd6e0 tx=0x7f195802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ac4b49fc54f425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.009+0000 7f195d400640 1 -- 172.21.15.97:0/4088642841 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195802ca80 con 0x7f196006afc0 2024-03-06T07:31:59.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.009+0000 7f195d400640 1 -- 172.21.15.97:0/4088642841 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195802cc20 con 0x7f196006afc0 2024-03-06T07:31:59.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.009+0000 7f195d400640 1 -- 172.21.15.97:0/4088642841 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19580343f0 con 0x7f196006afc0 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.009+0000 7f195f200640 1 -- 172.21.15.97:0/4088642841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 msgr2=0x7f196006b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.009+0000 7f195f200640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f196006b450 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f19600cd6e0 tx=0x7f195802bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 -- 172.21.15.97:0/4088642841 shutdown_connections 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f196006b450 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 --2- 172.21.15.97:0/4088642841 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f1960071860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 --2- 172.21.15.97:0/4088642841 >> v2:172.21.15.135:3300/0 conn(0x7f1960144940 0x7f1960070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 -- 172.21.15.97:0/4088642841 >> 172.21.15.97:0/4088642841 conn(0x7f19600b18a0 msgr2=0x7f19600b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 -- 172.21.15.97:0/4088642841 shutdown_connections 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.010+0000 7f195f200640 1 -- 172.21.15.97:0/4088642841 wait complete. 2024-03-06T07:31:59.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.011+0000 7f195f200640 1 Processor -- start 2024-03-06T07:31:59.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.011+0000 7f195f200640 1 -- start start 2024-03-06T07:31:59.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.012+0000 7f195f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f19600b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.012+0000 7f195f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1960144940 0x7f19601f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.012+0000 7f195f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f19601f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.012+0000 7f195f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960071da0 con 0x7f196006afc0 2024-03-06T07:31:59.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.012+0000 7f195f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196015fc70 con 0x7f1960144940 2024-03-06T07:31:59.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.012+0000 7f195f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960144d20 con 0x7f1960160470 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f19600b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f19600b2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43942/0 (socket says 172.21.15.97:43942) 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 -- 172.21.15.97:0/267438717 learned_addr learned my addr 172.21.15.97:0/267438717 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 -- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 msgr2=0x7f19601f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195e800640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f19601f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f1955400640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1960144940 0x7f19601f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f19601f5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 -- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1960144940 msgr2=0x7f19601f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1960144940 0x7f19601f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 -- 172.21.15.97:0/267438717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19601436c0 con 0x7f196006afc0 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.013+0000 7f195de00640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f19600b2a20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f195000c7c0 tx=0x7f195000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.014+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950015070 con 0x7f196006afc0 2024-03-06T07:31:59.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.014+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1950007a30 con 0x7f196006afc0 2024-03-06T07:31:59.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.014+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19601439a0 con 0x7f196006afc0 2024-03-06T07:31:59.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.014+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950007e00 con 0x7f196006afc0 2024-03-06T07:31:59.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.014+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19600ac1d0 con 0x7f196006afc0 2024-03-06T07:31:59.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.015+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1960143b70 con 0x7f196006afc0 2024-03-06T07:31:59.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.016+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1950018070 con 0x7f196006afc0 2024-03-06T07:31:59.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.017+0000 7f1956a00640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f193c06b4d0 0x7f193c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.017+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f195008a050 con 0x7f196006afc0 2024-03-06T07:31:59.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.017+0000 7f1955400640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f193c06b4d0 0x7f193c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.018+0000 7f1955400640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f193c06b4d0 0x7f193c06d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1960071870 tx=0x7f1948008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.022+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1950058ff0 con 0x7f196006afc0 2024-03-06T07:31:59.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.221+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f196004f330 con 0x7f196006afc0 2024-03-06T07:31:59.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.783+0000 7f1956a00640 1 -- 172.21.15.97:0/267438717 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7f19500588f0 con 0x7f196006afc0 2024-03-06T07:31:59.786 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:31:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.789+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f193c06b4d0 msgr2=0x7f193c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.789+0000 7f195f200640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f193c06b4d0 0x7f193c06d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1960071870 tx=0x7f1948008040 comp rx=0 tx=0).stop 2024-03-06T07:31:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.789+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 msgr2=0x7f19600b2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.789+0000 7f195f200640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f19600b2a20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f195000c7c0 tx=0x7f195000cc80 comp rx=0 tx=0).stop 2024-03-06T07:31:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 shutdown_connections 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f193c06b4d0 0x7f193c06d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1960160470 0x7f19601f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1960144940 0x7f19601f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 --2- 172.21.15.97:0/267438717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196006afc0 0x7f19600b2a20 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 >> 172.21.15.97:0/267438717 conn(0x7f19600b18a0 msgr2=0x7f19600b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 shutdown_connections 2024-03-06T07:31:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:31:59.790+0000 7f195f200640 1 -- 172.21.15.97:0/267438717 wait complete. 2024-03-06T07:31:59.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:32:00.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.016+0000 7fa9b4000640 1 Processor -- start 2024-03-06T07:32:00.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.016+0000 7fa9b4000640 1 -- start start 2024-03-06T07:32:00.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.017+0000 7fa9b4000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa9ac161b50 0x7fa9ac0d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.017+0000 7fa9b4000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa9ac0d4a00 0x7fa9ac0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.017+0000 7fa9b4000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0cecd0 con 0x7fa9ac0b5490 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.017+0000 7fa9b4000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cef40 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.017+0000 7fa9b4000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cfc80 con 0x7fa9ac161b50 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b0c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa9ac0d4a00 0x7fa9ac0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b2000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa9ac0b5490 0x7fa9ac0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45366/0 (socket says 172.21.15.97:45366) 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b2000640 1 -- 172.21.15.97:0/946137002 learned_addr learned my addr 172.21.15.97:0/946137002 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b1600640 1 --2- 172.21.15.97:0/946137002 >> v2:172.21.15.135:3300/0 conn(0x7fa9ac161b50 0x7fa9ac0d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b0c00640 1 -- 172.21.15.97:0/946137002 >> v2:172.21.15.135:3300/0 conn(0x7fa9ac161b50 msgr2=0x7fa9ac0d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b0c00640 1 --2- 172.21.15.97:0/946137002 >> v2:172.21.15.135:3300/0 conn(0x7fa9ac161b50 0x7fa9ac0d05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b0c00640 1 -- 172.21.15.97:0/946137002 >> v1:172.21.15.97:6789/0 conn(0x7fa9ac0b5490 legacy=0x7fa9ac0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b0c00640 1 -- 172.21.15.97:0/946137002 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac0c7b60 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9b1600640 1 --2- 172.21.15.97:0/946137002 >> v2:172.21.15.135:3300/0 conn(0x7fa9ac161b50 0x7fa9ac0d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.018+0000 7fa9abe00640 1 -- 172.21.15.97:0/946137002 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500607175 0 0) 0x7fa9ac0cecd0 con 0x7fa9ac0b5490 2024-03-06T07:32:00.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.019+0000 7fa9b0c00640 1 --2- 172.21.15.97:0/946137002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0d0ae0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0d3e40 tx=0x7fa99802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c39ae3e0d132cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9abe00640 1 -- 172.21.15.97:0/946137002 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998035020 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9abe00640 1 -- 172.21.15.97:0/946137002 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998031830 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9abe00640 1 -- 172.21.15.97:0/946137002 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998031c00 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 -- 172.21.15.97:0/946137002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 msgr2=0x7fa9ac0d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 --2- 172.21.15.97:0/946137002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0d0ae0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0d3e40 tx=0x7fa99802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 -- 172.21.15.97:0/946137002 shutdown_connections 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 --2- 172.21.15.97:0/946137002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0d0ae0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 --2- 172.21.15.97:0/946137002 >> v2:172.21.15.135:3300/0 conn(0x7fa9ac161b50 0x7fa9ac0d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 -- 172.21.15.97:0/946137002 >> 172.21.15.97:0/946137002 conn(0x7fa9ac0d3e70 msgr2=0x7fa9ac0d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 -- 172.21.15.97:0/946137002 shutdown_connections 2024-03-06T07:32:00.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.020+0000 7fa9b4000640 1 -- 172.21.15.97:0/946137002 wait complete. 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.022+0000 7fa9b4000640 1 Processor -- start 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.022+0000 7fa9b4000640 1 -- start start 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b4000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9ac0b5490 0x7fa9ac1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b4000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b4000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9ac161b50 0x7fa9ac0b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b4000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0d1020 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b4000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0c92a0 con 0x7fa9ac0b5490 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b4000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0d11a0 con 0x7fa9ac161b50 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43960/0 (socket says 172.21.15.97:43960) 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.023+0000 7fa9b0c00640 1 -- 172.21.15.97:0/4134323183 learned_addr learned my addr 172.21.15.97:0/4134323183 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b1600640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9ac0b5490 0x7fa9ac1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b0c00640 1 -- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9ac161b50 msgr2=0x7fa9ac0b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b0c00640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9ac161b50 0x7fa9ac0b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b0c00640 1 -- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9ac0b5490 msgr2=0x7fa9ac1511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b0c00640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9ac0b5490 0x7fa9ac1511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b0c00640 1 -- 172.21.15.97:0/4134323183 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac0cb840 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9b0c00640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0caf40 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa998000c00 tx=0x7fa9980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.024+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998040070 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.025+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998039a20 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.025+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998035040 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.025+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac0cbb20 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.025+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac0ccb00 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.026+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac04f330 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.027+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa99804b060 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.028+0000 7fa9a9600640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.028+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa9980ab890 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.028+0000 7fa9b1600640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.029+0000 7fa9b1600640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa98c06b4d0 0x7fa98c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0a1e20 tx=0x7fa9a40084f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.032+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa99807aaa0 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.229+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fa9ac0d16d0 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.234+0000 7fa9a9600640 1 -- 172.21.15.97:0/4134323183 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v24) v1 ==== 108+0+0 (secure 0 0 0) 0x7fa99807a3a0 con 0x7fa9ac0d4a00 2024-03-06T07:32:00.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa98c06b4d0 msgr2=0x7fa98c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa98c06b4d0 0x7fa98c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0a1e20 tx=0x7fa9a40084f0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 msgr2=0x7fa9ac0caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0caf40 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa998000c00 tx=0x7fa9980046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 shutdown_connections 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9ac161b50 0x7fa9ac0b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9ac0d4a00 0x7fa9ac0caf40 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 --2- 172.21.15.97:0/4134323183 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9ac0b5490 0x7fa9ac1511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 >> 172.21.15.97:0/4134323183 conn(0x7fa9ac0d3e70 msgr2=0x7fa9ac06d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 shutdown_connections 2024-03-06T07:32:00.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.237+0000 7fa9b4000640 1 -- 172.21.15.97:0/4134323183 wait complete. 2024-03-06T07:32:00.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.462+0000 7fdc52800640 1 Processor -- start 2024-03-06T07:32:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.463+0000 7fdc52800640 1 -- start start 2024-03-06T07:32:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.464+0000 7fdc52800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc4c15fc10 0x7fdc4c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.464+0000 7fdc52800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.464+0000 7fdc52800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c0b46c0 con 0x7fdc4c15fff0 2024-03-06T07:32:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.464+0000 7fdc52800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c0b49a0 con 0x7fdc4c15fc10 2024-03-06T07:32:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.464+0000 7fdc52800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c072c90 con 0x7fdc4c0b3f70 2024-03-06T07:32:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.465+0000 7fdc4b400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc4c15fc10 0x7fdc4c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.465+0000 7fdc4b400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc4c15fc10 0x7fdc4c070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:56164/0 (socket says 172.21.15.97:56164) 2024-03-06T07:32:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.465+0000 7fdc4b400640 1 -- 172.21.15.97:0/3860660573 learned_addr learned my addr 172.21.15.97:0/3860660573 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:00.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.465+0000 7fdc4be00640 1 --2- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.466+0000 7fdc4b400640 1 -- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 msgr2=0x7fdc4c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.466+0000 7fdc4b400640 1 --2- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.466+0000 7fdc4b400640 1 -- 172.21.15.97:0/3860660573 >> v1:172.21.15.135:6789/0 conn(0x7fdc4c15fff0 legacy=0x7fdc4c160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.466+0000 7fdc4b400640 1 -- 172.21.15.97:0/3860660573 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c09dd80 con 0x7fdc4c15fc10 2024-03-06T07:32:00.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.467+0000 7fdc4b400640 1 --2- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c070ff0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009870 tx=0x7fdc3c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56fec8f8067e5f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.467+0000 7fdc4a000640 1 -- 172.21.15.97:0/3860660573 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c034070 con 0x7fdc4c15fc10 2024-03-06T07:32:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc4a000640 1 -- 172.21.15.97:0/3860660573 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c030840 con 0x7fdc4c15fc10 2024-03-06T07:32:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc4a000640 1 -- 172.21.15.97:0/3860660573 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c030c10 con 0x7fdc4c15fc10 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc52800640 1 -- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fc10 msgr2=0x7fdc4c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc52800640 1 --2- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c070ff0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009870 tx=0x7fdc3c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc52800640 1 -- 172.21.15.97:0/3860660573 shutdown_connections 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc52800640 1 --2- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc52800640 1 --2- 172.21.15.97:0/3860660573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c070ff0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.468+0000 7fdc52800640 1 -- 172.21.15.97:0/3860660573 >> 172.21.15.97:0/3860660573 conn(0x7fdc4c0b1940 msgr2=0x7fdc4c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.469+0000 7fdc52800640 1 -- 172.21.15.97:0/3860660573 shutdown_connections 2024-03-06T07:32:00.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.469+0000 7fdc52800640 1 -- 172.21.15.97:0/3860660573 wait complete. 2024-03-06T07:32:00.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.471+0000 7fdc52800640 1 Processor -- start 2024-03-06T07:32:00.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.471+0000 7fdc52800640 1 -- start start 2024-03-06T07:32:00.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc52800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c1410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc52800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc52800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fff0 0x7fdc4c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc52800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c15c460 con 0x7fdc4c15fc10 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc52800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c15c2e0 con 0x7fdc4c15fff0 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc52800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c0ce9e0 con 0x7fdc4c0b3f70 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc4aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc4aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c0cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43976/0 (socket says 172.21.15.97:43976) 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.472+0000 7fdc4aa00640 1 -- 172.21.15.97:0/1890882181 learned_addr learned my addr 172.21.15.97:0/1890882181 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4b400640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c1410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4be00640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fff0 0x7fdc4c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4aa00640 1 -- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 msgr2=0x7fdc4c1410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4aa00640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c1410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4aa00640 1 -- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fff0 msgr2=0x7fdc4c0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4aa00640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fff0 0x7fdc4c0cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4aa00640 1 -- 172.21.15.97:0/1890882181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c06c990 con 0x7fdc4c15fc10 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc4aa00640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c0cda90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c730 tx=0x7fdc3400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.473+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34016020 con 0x7fdc4c15fc10 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.474+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc34006d50 con 0x7fdc4c15fc10 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.474+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34003bb0 con 0x7fdc4c15fc10 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.474+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c141600 con 0x7fdc4c15fc10 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.474+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c141b60 con 0x7fdc4c15fc10 2024-03-06T07:32:00.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.475+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c04f330 con 0x7fdc4c15fc10 2024-03-06T07:32:00.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.476+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc3400a850 con 0x7fdc4c15fc10 2024-03-06T07:32:00.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.477+0000 7fdc43400640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc2806b500 0x7fdc2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.477+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdc34089b70 con 0x7fdc4c15fc10 2024-03-06T07:32:00.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.478+0000 7fdc4b400640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc2806b500 0x7fdc2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.478+0000 7fdc4b400640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc2806b500 0x7fdc2806d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c036000 tx=0x7fdc3c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.482+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc34020020 con 0x7fdc4c15fc10 2024-03-06T07:32:00.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.683+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdc4c06ef30 con 0x7fdc4c15fc10 2024-03-06T07:32:00.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.689+0000 7fdc43400640 1 -- 172.21.15.97:0/1890882181 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdc4c06c990 con 0x7fdc4c15fc10 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc2806b500 msgr2=0x7fdc2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc2806b500 0x7fdc2806d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c036000 tx=0x7fdc3c03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 msgr2=0x7fdc4c0cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c0cda90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c730 tx=0x7fdc3400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 shutdown_connections 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc2806b500 0x7fdc2806d9c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc4c15fff0 0x7fdc4c0cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc4c15fc10 0x7fdc4c0cda90 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 --2- 172.21.15.97:0/1890882181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc4c0b3f70 0x7fdc4c1410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.691+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 >> 172.21.15.97:0/1890882181 conn(0x7fdc4c0b1940 msgr2=0x7fdc4c0add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.692+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 shutdown_connections 2024-03-06T07:32:00.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.692+0000 7fdc52800640 1 -- 172.21.15.97:0/1890882181 wait complete. 2024-03-06T07:32:00.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:32:00.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.906+0000 7f6ed5e00640 1 Processor -- start 2024-03-06T07:32:00.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.906+0000 7f6ed5e00640 1 -- start start 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ed5e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed015eff0 0x7f6ed0070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ed5e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6ed006e4d0 0x7f6ed006e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ed5e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed0071d30 con 0x7f6ed00af1c0 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ed5e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed0071260 con 0x7f6ed006e4d0 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ed5e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed015f470 con 0x7f6ed015eff0 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ecf400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6ed006e4d0 0x7f6ed006e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ecea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6ed00af1c0 0x7f6ed006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45380/0 (socket says 172.21.15.97:45380) 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ecea00640 1 -- 172.21.15.97:0/1078123583 learned_addr learned my addr 172.21.15.97:0/1078123583 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.908+0000 7f6ece000640 1 --2- 172.21.15.97:0/1078123583 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed015eff0 0x7f6ed0070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.909+0000 7f6ecf400640 1 -- 172.21.15.97:0/1078123583 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed015eff0 msgr2=0x7f6ed0070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.909+0000 7f6ecf400640 1 --2- 172.21.15.97:0/1078123583 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed015eff0 0x7f6ed0070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.909+0000 7f6ecf400640 1 -- 172.21.15.97:0/1078123583 >> v1:172.21.15.97:6789/0 conn(0x7f6ed00af1c0 legacy=0x7f6ed006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.909+0000 7f6ecf400640 1 -- 172.21.15.97:0/1078123583 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0071490 con 0x7f6ed006e4d0 2024-03-06T07:32:00.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.909+0000 7f6ecd600640 1 -- 172.21.15.97:0/1078123583 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713954215 0 0) 0x7f6ec0001490 con 0x7f6ed00af1c0 2024-03-06T07:32:00.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.910+0000 7f6ecf400640 1 --2- 172.21.15.97:0/1078123583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed006e4d0 0x7f6ed006e960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80097a0 tx=0x7f6ec802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b28cc92fa7b1b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.910+0000 7f6ecd600640 1 -- 172.21.15.97:0/1078123583 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8035020 con 0x7f6ed006e4d0 2024-03-06T07:32:00.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.910+0000 7f6ecd600640 1 -- 172.21.15.97:0/1078123583 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec8031810 con 0x7f6ed006e4d0 2024-03-06T07:32:00.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.910+0000 7f6ecd600640 1 -- 172.21.15.97:0/1078123583 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8031be0 con 0x7f6ed006e4d0 2024-03-06T07:32:00.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 -- 172.21.15.97:0/1078123583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed006e4d0 msgr2=0x7f6ed006e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/1078123583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed006e4d0 0x7f6ed006e960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80097a0 tx=0x7f6ec802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 -- 172.21.15.97:0/1078123583 shutdown_connections 2024-03-06T07:32:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/1078123583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed006e4d0 0x7f6ed006e960 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/1078123583 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed015eff0 0x7f6ed0070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 -- 172.21.15.97:0/1078123583 >> 172.21.15.97:0/1078123583 conn(0x7f6ed00b1730 msgr2=0x7f6ed00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 -- 172.21.15.97:0/1078123583 shutdown_connections 2024-03-06T07:32:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.911+0000 7f6ed5e00640 1 -- 172.21.15.97:0/1078123583 wait complete. 2024-03-06T07:32:00.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.913+0000 7f6ed5e00640 1 Processor -- start 2024-03-06T07:32:00.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.913+0000 7f6ed5e00640 1 -- start start 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.914+0000 7f6ed5e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed006e4d0 0x7f6ed00cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.914+0000 7f6ed5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 0x7f6ed015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.914+0000 7f6ed5e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ed015eff0 0x7f6ed014b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.914+0000 7f6ed5e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00b30b0 con 0x7f6ed00af1c0 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.914+0000 7f6ed5e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0071a50 con 0x7f6ed015eff0 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.914+0000 7f6ed5e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed006d370 con 0x7f6ed006e4d0 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 0x7f6ed015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 0x7f6ed015a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43988/0 (socket says 172.21.15.97:43988) 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 -- 172.21.15.97:0/2071674939 learned_addr learned my addr 172.21.15.97:0/2071674939 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 -- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed006e4d0 msgr2=0x7f6ed00cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed006e4d0 0x7f6ed00cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 -- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ed015eff0 msgr2=0x7f6ed014b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ecf400640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ed015eff0 0x7f6ed014b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ed015eff0 0x7f6ed014b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 -- 172.21.15.97:0/2071674939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed00cdd50 con 0x7f6ed00af1c0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.915+0000 7f6ece000640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 0x7f6ed015a260 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800d5f0 tx=0x7f6eb800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.916+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80125f0 con 0x7f6ed00af1c0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.916+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb8019070 con 0x7f6ed00af1c0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.916+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80129d0 con 0x7f6ed00af1c0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.916+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed00cdff0 con 0x7f6ed00af1c0 2024-03-06T07:32:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.916+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed00ce5b0 con 0x7f6ed00af1c0 2024-03-06T07:32:00.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.917+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed004f330 con 0x7f6ed00af1c0 2024-03-06T07:32:00.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.918+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6eb8012b70 con 0x7f6ed00af1c0 2024-03-06T07:32:00.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.919+0000 7f6ec7400640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6eac06b500 0x7f6eac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.919+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6eb808a260 con 0x7f6ed00af1c0 2024-03-06T07:32:00.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.919+0000 7f6ecea00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6eac06b500 0x7f6eac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.919+0000 7f6ecea00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6eac06b500 0x7f6eac06d9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0007fd0 tx=0x7f6ec0001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:00.923+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6eb8059450 con 0x7f6ed00af1c0 2024-03-06T07:32:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.188+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f6ed0074be0 con 0x7f6ed00af1c0 2024-03-06T07:32:01.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.795+0000 7f6ec7400640 1 -- 172.21.15.97:0/2071674939 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v247) v1 ==== 140+0+0 (secure 0 0 0) 0x7f6eb8058d50 con 0x7f6ed00af1c0 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.798+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6eac06b500 msgr2=0x7f6eac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.798+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6eac06b500 0x7f6eac06d9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0007fd0 tx=0x7f6ec0001690 comp rx=0 tx=0).stop 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.798+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 msgr2=0x7f6ed015a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.798+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 0x7f6ed015a260 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800d5f0 tx=0x7f6eb800dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 shutdown_connections 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6eac06b500 0x7f6eac06d9c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ed015eff0 0x7f6ed014b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ed00af1c0 0x7f6ed015a260 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 --2- 172.21.15.97:0/2071674939 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ed006e4d0 0x7f6ed00cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 >> 172.21.15.97:0/2071674939 conn(0x7f6ed00b1730 msgr2=0x7f6ed015e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 shutdown_connections 2024-03-06T07:32:01.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:01.799+0000 7f6ed5e00640 1 -- 172.21.15.97:0/2071674939 wait complete. 2024-03-06T07:32:01.822 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:32:01.823 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:32:01.823 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:01.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:02.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.032+0000 7fdb63e00640 1 Processor -- start 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.033+0000 7fdb63e00640 1 -- start start 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.034+0000 7fdb63e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdb640ae6b0 0x7fdb640b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.034+0000 7fdb63e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640af300 con 0x7fdb641623d0 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.034+0000 7fdb63e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640aed40 con 0x7fdb640b3a60 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.034+0000 7fdb63e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb640aeaf0 con 0x7fdb640ae6b0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.034+0000 7fdb62a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdb641623d0 0x7fdb640d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45386/0 (socket says 172.21.15.97:45386) 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.034+0000 7fdb62a00640 1 -- 172.21.15.97:0/2780191974 learned_addr learned my addr 172.21.15.97:0/2780191974 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.035+0000 7fdb5a000640 1 --2- 172.21.15.97:0/2780191974 >> v2:172.21.15.135:3300/0 conn(0x7fdb640ae6b0 0x7fdb640b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.035+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242182588 0 0) 0x7fdb640af300 con 0x7fdb641623d0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.035+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48003520 con 0x7fdb641623d0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.035+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242084989 0 0) 0x7fdb48003520 con 0x7fdb641623d0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.035+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 >> v2:172.21.15.135:3300/0 conn(0x7fdb640ae6b0 msgr2=0x7fdb640b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.035+0000 7fdb62000640 1 --2- 172.21.15.97:0/2780191974 >> v2:172.21.15.135:3300/0 conn(0x7fdb640ae6b0 0x7fdb640b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 >> v1:172.21.15.135:6789/0 conn(0x7fdb640b3a60 legacy=0x7fdb640ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb640af020 con 0x7fdb641623d0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdb54001670 con 0x7fdb641623d0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 >> v1:172.21.15.97:6789/0 conn(0x7fdb641623d0 legacy=0x7fdb640d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb48003a00 0x7fdb48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48004800 0x7fdb48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48005210 0x7fdb4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640aeaf0 con 0x7fdb48003a00 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800c160 con 0x7fdb48004800 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800c360 con 0x7fdb48005210 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62a00640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb48003a00 0x7fdb48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb5a000640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48004800 0x7fdb48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb63400640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48005210 0x7fdb4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62a00640 1 -- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48005210 msgr2=0x7fdb4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62a00640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48005210 0x7fdb4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62a00640 1 -- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48004800 msgr2=0x7fdb48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62a00640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48004800 0x7fdb48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.036+0000 7fdb62a00640 1 -- 172.21.15.97:0/2780191974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48003520 con 0x7fdb48003a00 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.037+0000 7fdb63400640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48005210 0x7fdb4800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.037+0000 7fdb5a000640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb48004800 0x7fdb48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.037+0000 7fdb63400640 1 -- 172.21.15.97:0/2780191974 reap_dead start 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.038+0000 7fdb62a00640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb48003a00 0x7fdb48003e20 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c160 tx=0x7fdb48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e519f0c6ffb92ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.038+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54007c60 con 0x7fdb48003a00 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.038+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb54030070 con 0x7fdb48003a00 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.038+0000 7fdb62000640 1 -- 172.21.15.97:0/2780191974 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54005d30 con 0x7fdb48003a00 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 -- 172.21.15.97:0/2780191974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb48003a00 msgr2=0x7fdb48003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb48003a00 0x7fdb48003e20 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c160 tx=0x7fdb48005830 comp rx=0 tx=0).stop 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 -- 172.21.15.97:0/2780191974 shutdown_connections 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2780191974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb48003a00 0x7fdb48003e20 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 -- 172.21.15.97:0/2780191974 >> 172.21.15.97:0/2780191974 conn(0x7fdb6406afc0 msgr2=0x7fdb6406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 -- 172.21.15.97:0/2780191974 shutdown_connections 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.039+0000 7fdb63e00640 1 -- 172.21.15.97:0/2780191974 wait complete. 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.041+0000 7fdb63e00640 1 Processor -- start 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.041+0000 7fdb63e00640 1 -- start start 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.042+0000 7fdb63e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48003a00 0x7fdb64152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.042+0000 7fdb63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 0x7fdb6415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.042+0000 7fdb63e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb6415f240 0x7fdb64162890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.042+0000 7fdb63e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640b3520 con 0x7fdb641646a0 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.042+0000 7fdb63e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6406d5c0 con 0x7fdb6415f240 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.042+0000 7fdb63e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640b36a0 con 0x7fdb48003a00 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 0x7fdb6415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 0x7fdb6415ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44000/0 (socket says 172.21.15.97:44000) 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 -- 172.21.15.97:0/2172178590 learned_addr learned my addr 172.21.15.97:0/2172178590 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 -- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48003a00 msgr2=0x7fdb64152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48003a00 0x7fdb64152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 -- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb6415f240 msgr2=0x7fdb64162890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb63400640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb6415f240 0x7fdb64162890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb6415f240 0x7fdb64162890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 -- 172.21.15.97:0/2172178590 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64163600 con 0x7fdb641646a0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.043+0000 7fdb5a000640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 0x7fdb6415ed00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0114e0 tx=0x7fdb4c0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.044+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c015870 con 0x7fdb641646a0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.044+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c015a10 con 0x7fdb641646a0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.044+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c015e00 con 0x7fdb641646a0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.044+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb641638e0 con 0x7fdb641646a0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.044+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb64163ea0 con 0x7fdb641646a0 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.045+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6415f650 con 0x7fdb641646a0 2024-03-06T07:32:02.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.046+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb4c01d400 con 0x7fdb641646a0 2024-03-06T07:32:02.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.047+0000 7fdb5b400640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb4006b500 0x7fdb4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.047+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb4c08a600 con 0x7fdb641646a0 2024-03-06T07:32:02.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.047+0000 7fdb62a00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb4006b500 0x7fdb4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.048+0000 7fdb62a00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb4006b500 0x7fdb4006d9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdb48004780 tx=0x7fdb54007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.052+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb6415f650 con 0x7fdb641646a0 2024-03-06T07:32:02.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.252+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb6404f330 con 0x7fdb641646a0 2024-03-06T07:32:02.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.254+0000 7fdb5b400640 1 -- 172.21.15.97:0/2172178590 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13009 (secure 0 0 0) 0x7fdb4c019030 con 0x7fdb641646a0 2024-03-06T07:32:02.256 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:02.256 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":74,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:31:59.774987+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/3151111028":"2024-03-07T07:31:56.756966+0000","172.21.15.97:6836/3151111028":"2024-03-07T07:31:56.756966+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:02.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb4006b500 msgr2=0x7fdb4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb4006b500 0x7fdb4006d9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdb48004780 tx=0x7fdb54007000 comp rx=0 tx=0).stop 2024-03-06T07:32:02.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 msgr2=0x7fdb6415ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 0x7fdb6415ed00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0114e0 tx=0x7fdb4c0119a0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 shutdown_connections 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb4006b500 0x7fdb4006d9c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb6415f240 0x7fdb64162890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb641646a0 0x7fdb6415ed00 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 --2- 172.21.15.97:0/2172178590 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb48003a00 0x7fdb64152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 >> 172.21.15.97:0/2172178590 conn(0x7fdb6406afc0 msgr2=0x7fdb640cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 shutdown_connections 2024-03-06T07:32:02.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.257+0000 7fdb63e00640 1 -- 172.21.15.97:0/2172178590 wait complete. 2024-03-06T07:32:02.280 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:02.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:02.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.493+0000 7f089b200640 1 Processor -- start 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.493+0000 7f089b200640 1 -- start start 2024-03-06T07:32:02.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f089b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f089c15e130 0x7f089c0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f089b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c0d25b0 0x7f089c0c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f089b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c0d13e0 con 0x7f089c0cfaf0 2024-03-06T07:32:02.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f089b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089c0cf120 con 0x7f089c15e130 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f089b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0cead0 con 0x7f089c0d25b0 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f0899e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f089c15e130 0x7f089c0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f0899400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f089c0cfaf0 0x7f089c0d1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45398/0 (socket says 172.21.15.97:45398) 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.495+0000 7f0899400640 1 -- 172.21.15.97:0/1993264109 learned_addr learned my addr 172.21.15.97:0/1993264109 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.496+0000 7f089a800640 1 --2- 172.21.15.97:0/1993264109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c0d25b0 0x7f089c0c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.496+0000 7f0899e00640 1 -- 172.21.15.97:0/1993264109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c0d25b0 msgr2=0x7f089c0c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.496+0000 7f0899e00640 1 --2- 172.21.15.97:0/1993264109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c0d25b0 0x7f089c0c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.496+0000 7f0899e00640 1 -- 172.21.15.97:0/1993264109 >> v1:172.21.15.97:6789/0 conn(0x7f089c0cfaf0 legacy=0x7f089c0d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.496+0000 7f0899e00640 1 -- 172.21.15.97:0/1993264109 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c0cedb0 con 0x7f089c15e130 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.496+0000 7f0898a00640 1 -- 172.21.15.97:0/1993264109 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374965711 0 0) 0x7f089c0d13e0 con 0x7f089c0cfaf0 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.497+0000 7f0899e00640 1 --2- 172.21.15.97:0/1993264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c15e130 0x7f089c0cf5b0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097a0 tx=0x7f088c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a2ffd17809f007 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.497+0000 7f0898a00640 1 -- 172.21.15.97:0/1993264109 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c035020 con 0x7f089c15e130 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.497+0000 7f0898a00640 1 -- 172.21.15.97:0/1993264109 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c031830 con 0x7f089c15e130 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.497+0000 7f0898a00640 1 -- 172.21.15.97:0/1993264109 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c031c00 con 0x7f089c15e130 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 -- 172.21.15.97:0/1993264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c15e130 msgr2=0x7f089c0cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 --2- 172.21.15.97:0/1993264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c15e130 0x7f089c0cf5b0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097a0 tx=0x7f088c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 -- 172.21.15.97:0/1993264109 shutdown_connections 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 --2- 172.21.15.97:0/1993264109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c0d25b0 0x7f089c0c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 --2- 172.21.15.97:0/1993264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c15e130 0x7f089c0cf5b0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 -- 172.21.15.97:0/1993264109 >> 172.21.15.97:0/1993264109 conn(0x7f089c09d070 msgr2=0x7f089c0d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 -- 172.21.15.97:0/1993264109 shutdown_connections 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.498+0000 7f089b200640 1 -- 172.21.15.97:0/1993264109 wait complete. 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.500+0000 7f089b200640 1 Processor -- start 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.500+0000 7f089b200640 1 -- start start 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f089b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 0x7f089c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f089b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f089c0d25b0 0x7f089c15e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f089b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c15e130 0x7f089c06b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f089b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0c8030 con 0x7f089c0cfaf0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f089b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0c7eb0 con 0x7f089c0d25b0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f089b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0c81b0 con 0x7f089c15e130 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f0899e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 0x7f089c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f0899e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 0x7f089c15b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44004/0 (socket says 172.21.15.97:44004) 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f0899e00640 1 -- 172.21.15.97:0/1695503173 learned_addr learned my addr 172.21.15.97:0/1695503173 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.501+0000 7f0899400640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f089c0d25b0 0x7f089c15e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f089a800640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c15e130 0x7f089c06b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899e00640 1 -- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c15e130 msgr2=0x7f089c06b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899e00640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c15e130 0x7f089c06b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899e00640 1 -- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f089c0d25b0 msgr2=0x7f089c15e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899e00640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f089c0d25b0 0x7f089c15e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899e00640 1 -- 172.21.15.97:0/1695503173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c15b750 con 0x7f089c0cfaf0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899400640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f089c0d25b0 0x7f089c15e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0899e00640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 0x7f089c15b210 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f088c0098c0 tx=0x7f088c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c041070 con 0x7f089c0cfaf0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c03aa30 con 0x7f089c0cfaf0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c035040 con 0x7f089c0cfaf0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.502+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c15f520 con 0x7f089c0cfaf0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.503+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c15f240 con 0x7f089c0cfaf0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.504+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c04f330 con 0x7f089c0cfaf0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.504+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f088c038020 con 0x7f089c0cfaf0 2024-03-06T07:32:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.506+0000 7f0892000640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f087806b4d0 0x7f087806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.506+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f088c0ab860 con 0x7f089c0cfaf0 2024-03-06T07:32:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.506+0000 7f0899400640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f087806b4d0 0x7f087806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.507+0000 7f0899400640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f087806b4d0 0x7f087806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0884007fd0 tx=0x7f0884009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.511+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f088c07aa70 con 0x7f089c0cfaf0 2024-03-06T07:32:02.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.708+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f089c09d690 con 0x7f089c0cfaf0 2024-03-06T07:32:02.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.709+0000 7f0892000640 1 -- 172.21.15.97:0/1695503173 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f088c07a370 con 0x7f089c0cfaf0 2024-03-06T07:32:02.711 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f087806b4d0 msgr2=0x7f087806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f087806b4d0 0x7f087806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0884007fd0 tx=0x7f0884009040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 msgr2=0x7f089c15b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 0x7f089c15b210 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f088c0098c0 tx=0x7f088c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 shutdown_connections 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f087806b4d0 0x7f087806d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f089c15e130 0x7f089c06b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f089c0d25b0 0x7f089c15e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 --2- 172.21.15.97:0/1695503173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f089c0cfaf0 0x7f089c15b210 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.711+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 >> 172.21.15.97:0/1695503173 conn(0x7f089c09d070 msgr2=0x7f089c0b89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.712+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 shutdown_connections 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.712+0000 7f089b200640 1 -- 172.21.15.97:0/1695503173 wait complete. 2024-03-06T07:32:02.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:32:02.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.941+0000 7f8133200640 1 Processor -- start 2024-03-06T07:32:02.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.941+0000 7f8133200640 1 -- start start 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8133200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f812c0d0c00 0x7f812c0d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8133200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c0b5240 con 0x7f812c15e200 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8133200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c0cf4d0 con 0x7f812c0b2e50 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8133200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c0d2ec0 con 0x7f812c0d0c00 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8131400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f812c0d0c00 0x7f812c0d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8131e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f812c15e200 0x7f812c0d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45412/0 (socket says 172.21.15.97:45412) 2024-03-06T07:32:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8131e00640 1 -- 172.21.15.97:0/4097285116 learned_addr learned my addr 172.21.15.97:0/4097285116 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:02.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8131400640 1 -- 172.21.15.97:0/4097285116 >> v1:172.21.15.135:6789/0 conn(0x7f812c0b2e50 legacy=0x7f812c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8131400640 1 -- 172.21.15.97:0/4097285116 >> v1:172.21.15.97:6789/0 conn(0x7f812c15e200 legacy=0x7f812c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.943+0000 7f8131400640 1 -- 172.21.15.97:0/4097285116 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c0b5500 con 0x7f812c0d0c00 2024-03-06T07:32:02.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8131400640 1 --2- 172.21.15.97:0/4097285116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0d0c00 0x7f812c0d4850 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f811c0097a0 tx=0x7f811c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645dc026fc6f2f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8130a00640 1 -- 172.21.15.97:0/4097285116 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c037070 con 0x7f812c0d0c00 2024-03-06T07:32:02.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8130a00640 1 -- 172.21.15.97:0/4097285116 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c030710 con 0x7f812c0d0c00 2024-03-06T07:32:02.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8130a00640 1 -- 172.21.15.97:0/4097285116 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c030ae0 con 0x7f812c0d0c00 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8133200640 1 -- 172.21.15.97:0/4097285116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0d0c00 msgr2=0x7f812c0d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8133200640 1 --2- 172.21.15.97:0/4097285116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0d0c00 0x7f812c0d4850 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f811c0097a0 tx=0x7f811c02b790 comp rx=0 tx=0).stop 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.945+0000 7f8133200640 1 -- 172.21.15.97:0/4097285116 shutdown_connections 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.946+0000 7f8133200640 1 --2- 172.21.15.97:0/4097285116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0d0c00 0x7f812c0d4850 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.946+0000 7f8133200640 1 -- 172.21.15.97:0/4097285116 >> 172.21.15.97:0/4097285116 conn(0x7f812c0d3d30 msgr2=0x7f812c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.946+0000 7f8133200640 1 -- 172.21.15.97:0/4097285116 shutdown_connections 2024-03-06T07:32:02.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.946+0000 7f8133200640 1 -- 172.21.15.97:0/4097285116 wait complete. 2024-03-06T07:32:02.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.947+0000 7f8133200640 1 Processor -- start 2024-03-06T07:32:02.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.947+0000 7f8133200640 1 -- start start 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.948+0000 7f8133200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c0b2e50 0x7f812c14d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.948+0000 7f8133200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c0d0c00 0x7f812c0cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.948+0000 7f8133200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 0x7f812c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.948+0000 7f8133200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c0cfd60 con 0x7f812c15e200 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.948+0000 7f8133200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c0cfa80 con 0x7f812c0b2e50 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.948+0000 7f8133200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c15e5e0 con 0x7f812c0d0c00 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 0x7f812c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 0x7f812c0c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44020/0 (socket says 172.21.15.97:44020) 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 -- 172.21.15.97:0/381549448 learned_addr learned my addr 172.21.15.97:0/381549448 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8131e00640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c0b2e50 0x7f812c14d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8131400640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c0d0c00 0x7f812c0cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 -- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c0d0c00 msgr2=0x7f812c0cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c0d0c00 0x7f812c0cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 -- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c0b2e50 msgr2=0x7f812c14d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c0b2e50 0x7f812c14d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.949+0000 7f8132800640 1 -- 172.21.15.97:0/381549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c0b2180 con 0x7f812c15e200 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.950+0000 7f8132800640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 0x7f812c0c9150 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f812800c730 tx=0x7f812800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.950+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128016020 con 0x7f812c15e200 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.950+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812800aa50 con 0x7f812c15e200 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.950+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812800ce40 con 0x7f812c15e200 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.950+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c0b2460 con 0x7f812c15e200 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.950+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c0cb340 con 0x7f812c15e200 2024-03-06T07:32:02.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.951+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812c04f330 con 0x7f812c15e200 2024-03-06T07:32:02.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.952+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8128019020 con 0x7f812c15e200 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.953+0000 7f8126000640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f811006b4d0 0x7f811006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.953+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8128089f70 con 0x7f812c15e200 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.953+0000 7f8131e00640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f811006b4d0 0x7f811006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.954+0000 7f8131e00640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f811006b4d0 0x7f811006d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8120000f60 tx=0x7f8120007c60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:02.957+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8128020030 con 0x7f812c15e200 2024-03-06T07:32:03.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.226+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f812c08d4c0 con 0x7f812c15e200 2024-03-06T07:32:03.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.807+0000 7f8126000640 1 -- 172.21.15.97:0/381549448 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v249) v1 ==== 135+0+0 (secure 0 0 0) 0x7f81280212b0 con 0x7f812c15e200 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.810+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f811006b4d0 msgr2=0x7f811006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.810+0000 7f8133200640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f811006b4d0 0x7f811006d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8120000f60 tx=0x7f8120007c60 comp rx=0 tx=0).stop 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.810+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 msgr2=0x7f812c0c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.810+0000 7f8133200640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 0x7f812c0c9150 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f812800c730 tx=0x7f812800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.810+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 shutdown_connections 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f811006b4d0 0x7f811006d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c15e200 0x7f812c0c9150 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c0d0c00 0x7f812c0cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 --2- 172.21.15.97:0/381549448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c0b2e50 0x7f812c14d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 >> 172.21.15.97:0/381549448 conn(0x7f812c0d3d30 msgr2=0x7f812c06d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 shutdown_connections 2024-03-06T07:32:03.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:03.811+0000 7f8133200640 1 -- 172.21.15.97:0/381549448 wait complete. 2024-03-06T07:32:03.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:04.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.033+0000 7fd3d0e00640 1 Processor -- start 2024-03-06T07:32:04.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.033+0000 7fd3d0e00640 1 -- start start 2024-03-06T07:32:04.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3d0e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3cc160700 0x7fd3cc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3d0e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc0b46c0 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3d0e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc0b49a0 con 0x7fd3cc15fc10 2024-03-06T07:32:04.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3d0e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3cc072c90 con 0x7fd3cc160700 2024-03-06T07:32:04.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3ca800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3cc160700 0x7fd3cc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3ca800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3cc160700 0x7fd3cc070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40098/0 (socket says 172.21.15.97:40098) 2024-03-06T07:32:04.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.035+0000 7fd3ca800640 1 -- 172.21.15.97:0/423955924 learned_addr learned my addr 172.21.15.97:0/423955924 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:04.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.036+0000 7fd3ca800640 1 -- 172.21.15.97:0/423955924 >> v1:172.21.15.135:6789/0 conn(0x7fd3cc15fc10 legacy=0x7fd3cc15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.036+0000 7fd3ca800640 1 -- 172.21.15.97:0/423955924 >> v1:172.21.15.97:6789/0 conn(0x7fd3cc0b3f70 legacy=0x7fd3cc0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.036+0000 7fd3ca800640 1 -- 172.21.15.97:0/423955924 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc09dd80 con 0x7fd3cc160700 2024-03-06T07:32:04.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.037+0000 7fd3ca800640 1 --2- 172.21.15.97:0/423955924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc160700 0x7fd3cc070ff0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009e50 tx=0x7fd3c402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2524159a33b7b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.037+0000 7fd3c9400640 1 -- 172.21.15.97:0/423955924 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c402cb10 con 0x7fd3cc160700 2024-03-06T07:32:04.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.037+0000 7fd3c9400640 1 -- 172.21.15.97:0/423955924 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c402ccb0 con 0x7fd3cc160700 2024-03-06T07:32:04.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.037+0000 7fd3c9400640 1 -- 172.21.15.97:0/423955924 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c402ce60 con 0x7fd3cc160700 2024-03-06T07:32:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 -- 172.21.15.97:0/423955924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc160700 msgr2=0x7fd3cc070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/423955924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc160700 0x7fd3cc070ff0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009e50 tx=0x7fd3c402bf70 comp rx=0 tx=0).stop 2024-03-06T07:32:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 -- 172.21.15.97:0/423955924 shutdown_connections 2024-03-06T07:32:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/423955924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc160700 0x7fd3cc070ff0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 -- 172.21.15.97:0/423955924 >> 172.21.15.97:0/423955924 conn(0x7fd3cc0b1940 msgr2=0x7fd3cc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 -- 172.21.15.97:0/423955924 shutdown_connections 2024-03-06T07:32:04.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.038+0000 7fd3d0e00640 1 -- 172.21.15.97:0/423955924 wait complete. 2024-03-06T07:32:04.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.040+0000 7fd3d0e00640 1 Processor -- start 2024-03-06T07:32:04.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.040+0000 7fd3d0e00640 1 -- start start 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3d0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 0x7fd3cc15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3d0e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3cc15fc10 0x7fd3cc0cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3d0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3cc160700 0x7fd3cc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3d0e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc15c460 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3d0e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc15c2e0 con 0x7fd3cc15fc10 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3d0e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0cebb0 con 0x7fd3cc160700 2024-03-06T07:32:04.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 0x7fd3cc15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 0x7fd3cc15b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40106/0 (socket says 172.21.15.97:40106) 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 -- 172.21.15.97:0/4198893712 learned_addr learned my addr 172.21.15.97:0/4198893712 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 -- 172.21.15.97:0/4198893712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3cc160700 msgr2=0x7fd3cc06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3cc160700 0x7fd3cc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 -- 172.21.15.97:0/4198893712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3cc15fc10 msgr2=0x7fd3cc0cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3cc15fc10 0x7fd3cc0cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.041+0000 7fd3c9e00640 1 -- 172.21.15.97:0/4198893712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc0cc010 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.042+0000 7fd3c9e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 0x7fd3cc15b650 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00c460 tx=0x7fd3bc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.042+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc016020 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.042+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc012870 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.042+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc012c60 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.042+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc159e90 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.042+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc15a450 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.043+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3cc0b2a10 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.044+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3bc019020 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.045+0000 7fd3c2000640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3a806b4d0 0x7fd3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.045+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd3bc089cc0 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.045+0000 7fd3c3e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3a806b4d0 0x7fd3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.046+0000 7fd3c3e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3a806b4d0 0x7fd3a806d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000f60 tx=0x7fd3b40079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.050+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3cc0b2a10 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.242+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd3cc04f330 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.249+0000 7fd3c2000640 1 -- 172.21.15.97:0/4198893712 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd3bc090080 con 0x7fd3cc0b3f70 2024-03-06T07:32:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3a806b4d0 msgr2=0x7fd3a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3a806b4d0 0x7fd3a806d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000f60 tx=0x7fd3b40079d0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 msgr2=0x7fd3cc15b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 0x7fd3cc15b650 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00c460 tx=0x7fd3bc00c920 comp rx=0 tx=0).stop 2024-03-06T07:32:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 shutdown_connections 2024-03-06T07:32:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3a806b4d0 0x7fd3a806d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3cc160700 0x7fd3cc06d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3cc15fc10 0x7fd3cc0cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 --2- 172.21.15.97:0/4198893712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3cc0b3f70 0x7fd3cc15b650 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 >> 172.21.15.97:0/4198893712 conn(0x7fd3cc0b1940 msgr2=0x7fd3cc0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 shutdown_connections 2024-03-06T07:32:04.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.252+0000 7fd3d0e00640 1 -- 172.21.15.97:0/4198893712 wait complete. 2024-03-06T07:32:04.271 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:04.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:04.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:04.488 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:04.488 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:04.489 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:04.543 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:04.543 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:32:04.543 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:04.601 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:04.601 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:04.601 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:04.658 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:04.658 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:32:04.658 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:04.713 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:04.714 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:04.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.930+0000 7f593b200640 1 Processor -- start 2024-03-06T07:32:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.930+0000 7f593b200640 1 -- start start 2024-03-06T07:32:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.931+0000 7f593b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5934072680 0x7f59340c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.931+0000 7f593b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f59340cf3e0 0x7f59340ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.931+0000 7f593b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59340ca940 con 0x7f5934144660 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.931+0000 7f593b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59340cabf0 con 0x7f59340cf3e0 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.931+0000 7f593b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59340ce900 con 0x7f5934072680 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.932+0000 7f5939200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f59340cf3e0 0x7f59340ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.932+0000 7f5933400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5934072680 0x7f59340c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.932+0000 7f5939200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f59340cf3e0 0x7f59340ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:43360/0 (socket says 172.21.15.97:43360) 2024-03-06T07:32:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.932+0000 7f5939200640 1 -- 172.21.15.97:0/2263621100 learned_addr learned my addr 172.21.15.97:0/2263621100 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.933+0000 7f5939200640 1 -- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5934072680 msgr2=0x7f59340c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.933+0000 7f5939200640 1 --2- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5934072680 0x7f59340c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.933+0000 7f5939200640 1 -- 172.21.15.97:0/2263621100 >> v1:172.21.15.135:6789/0 conn(0x7f5934144660 legacy=0x7f5934071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.933+0000 7f5939200640 1 -- 172.21.15.97:0/2263621100 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59340cebe0 con 0x7f59340cf3e0 2024-03-06T07:32:04.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.934+0000 7f5939200640 1 --2- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59340cf3e0 0x7f59340ca400 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5924009870 tx=0x7f592402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9a93f4946581ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f5932a00640 1 -- 172.21.15.97:0/2263621100 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924034070 con 0x7f59340cf3e0 2024-03-06T07:32:04.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f5932a00640 1 -- 172.21.15.97:0/2263621100 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5924030840 con 0x7f59340cf3e0 2024-03-06T07:32:04.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f5932a00640 1 -- 172.21.15.97:0/2263621100 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924030c10 con 0x7f59340cf3e0 2024-03-06T07:32:04.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 -- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59340cf3e0 msgr2=0x7f59340ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 --2- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59340cf3e0 0x7f59340ca400 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5924009870 tx=0x7f592402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 -- 172.21.15.97:0/2263621100 shutdown_connections 2024-03-06T07:32:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 --2- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5934072680 0x7f59340c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 --2- 172.21.15.97:0/2263621100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59340cf3e0 0x7f59340ca400 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 -- 172.21.15.97:0/2263621100 >> 172.21.15.97:0/2263621100 conn(0x7f59340d0b90 msgr2=0x7f59340d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 -- 172.21.15.97:0/2263621100 shutdown_connections 2024-03-06T07:32:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.935+0000 7f593b200640 1 -- 172.21.15.97:0/2263621100 wait complete. 2024-03-06T07:32:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.937+0000 7f593b200640 1 Processor -- start 2024-03-06T07:32:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.938+0000 7f593b200640 1 -- start start 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f593b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5934072680 0x7f593415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f593b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59340cf3e0 0x7f593406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f593b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 0x7f593406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f593b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59340cc0f0 con 0x7f5934144660 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f593b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5934144a40 con 0x7f5934072680 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f593b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59340cc270 con 0x7f59340cf3e0 2024-03-06T07:32:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f5939200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 0x7f593406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f5939200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 0x7f593406fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40114/0 (socket says 172.21.15.97:40114) 2024-03-06T07:32:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f5939200640 1 -- 172.21.15.97:0/79290063 learned_addr learned my addr 172.21.15.97:0/79290063 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.939+0000 7f5933e00640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5934072680 0x7f593415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5933400640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59340cf3e0 0x7f593406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5939200640 1 -- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59340cf3e0 msgr2=0x7f593406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5939200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59340cf3e0 0x7f593406f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5939200640 1 -- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5934072680 msgr2=0x7f593415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5939200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5934072680 0x7f593415d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5939200640 1 -- 172.21.15.97:0/79290063 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59340b56d0 con 0x7f5934144660 2024-03-06T07:32:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f5939200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 0x7f593406fac0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5924009870 tx=0x7f5924004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924048070 con 0x7f5934144660 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5924030640 con 0x7f5934144660 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59340b39d0 con 0x7f5934144660 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.940+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924034040 con 0x7f5934144660 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.941+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59340b3f90 con 0x7f5934144660 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.942+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593404f330 con 0x7f5934144660 2024-03-06T07:32:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.942+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f592404b070 con 0x7f5934144660 2024-03-06T07:32:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.943+0000 7f592be00640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f591006b4d0 0x7f591006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.943+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f59240abfc0 con 0x7f5934144660 2024-03-06T07:32:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.944+0000 7f5933e00640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f591006b4d0 0x7f591006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.944+0000 7f5933e00640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f591006b4d0 0x7f591006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f592c007fd0 tx=0x7f592c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:04.948+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f592407b260 con 0x7f5934144660 2024-03-06T07:32:05.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.180+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5934143170 con 0x7f5934144660 2024-03-06T07:32:05.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.181+0000 7f592be00640 1 -- 172.21.15.97:0/79290063 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f592407ab60 con 0x7f5934144660 2024-03-06T07:32:05.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:05.184 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/1472129020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1472129020},{"type":"v1","addr":"172.21.15.135:6834","nonce":1472129020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1609159061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":1609159061},{"type":"v1","addr":"172.21.15.135:6835","nonce":1609159061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/537227296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":537227296},{"type":"v1","addr":"172.21.15.97:6835","nonce":537227296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2246984238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2246984238},{"type":"v1","addr":"172.21.15.97:6837","nonce":2246984238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[]} 2024-03-06T07:32:05.184 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 45 2024-03-06T07:32:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f591006b4d0 msgr2=0x7f591006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f591006b4d0 0x7f591006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f592c007fd0 tx=0x7f592c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 msgr2=0x7f593406fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 0x7f593406fac0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5924009870 tx=0x7f5924004730 comp rx=0 tx=0).stop 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 shutdown_connections 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f591006b4d0 0x7f591006d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5934144660 0x7f593406fac0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59340cf3e0 0x7f593406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 --2- 172.21.15.97:0/79290063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5934072680 0x7f593415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 >> 172.21.15.97:0/79290063 conn(0x7f59340d0b90 msgr2=0x7f59340b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 shutdown_connections 2024-03-06T07:32:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.184+0000 7f593b200640 1 -- 172.21.15.97:0/79290063 wait complete. 2024-03-06T07:32:05.205 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:05.205 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:32:05.205 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:05.205 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:05.205 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:05.206 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:32:05.206 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:05.206 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:05.206 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:05.206 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:32:05.206 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:05.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:05.207 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:05.207 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:32:05.207 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:05.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:05.225 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:05.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:05.226 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:05.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:05.229 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:05.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:05.237 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:05.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:05.272 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:05.273 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:05.278 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:05.278 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:05.284 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:05.284 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:05.290 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:05.290 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:05.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:32:05.433 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:32:05.433 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:32:05.444 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:32:05.446 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:32:05.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.484+0000 7fc916a00640 1 Processor -- start 2024-03-06T07:32:05.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.484+0000 7fc916a00640 1 -- start start 2024-03-06T07:32:05.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.485+0000 7fc916a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc91015b070 0x7fc91006e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.485+0000 7fc916a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100cc000 con 0x7fc91015ac90 2024-03-06T07:32:05.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.485+0000 7fc916a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100ce360 con 0x7fc91015f120 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.485+0000 7fc916a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100ce640 con 0x7fc91015b070 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.485+0000 7fc90fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc91015ac90 0x7fc91015ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41662/0 (socket says 172.21.15.97:41662) 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc90fe00640 1 -- 172.21.15.97:0/1245916264 learned_addr learned my addr 172.21.15.97:0/1245916264 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc907e00640 1 --2- 172.21.15.97:0/1245916264 >> v2:172.21.15.135:3300/0 conn(0x7fc91015b070 0x7fc91006e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381269960 0 0) 0x7fc9100cc000 con 0x7fc91015ac90 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4003520 con 0x7fc91015ac90 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830640419 0 0) 0x7fc9100ce360 con 0x7fc91015f120 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9100cc000 con 0x7fc91015f120 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc907e00640 1 -- 172.21.15.97:0/1245916264 >> v1:172.21.15.135:6789/0 conn(0x7fc91015f120 legacy=0x7fc9100c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc907e00640 1 -- 172.21.15.97:0/1245916264 >> v1:172.21.15.97:6789/0 conn(0x7fc91015ac90 legacy=0x7fc91015ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.486+0000 7fc907e00640 1 -- 172.21.15.97:0/1245916264 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100ce920 con 0x7fc91015b070 2024-03-06T07:32:05.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.487+0000 7fc907e00640 1 --2- 172.21.15.97:0/1245916264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc91006e580 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80097a0 tx=0x7fc8f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d60413b1f36689d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.487+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8035020 con 0x7fc91015b070 2024-03-06T07:32:05.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.487+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8031830 con 0x7fc91015b070 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc90f400640 1 -- 172.21.15.97:0/1245916264 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8031c00 con 0x7fc91015b070 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 -- 172.21.15.97:0/1245916264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 msgr2=0x7fc91006e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 --2- 172.21.15.97:0/1245916264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc91006e580 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80097a0 tx=0x7fc8f802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 -- 172.21.15.97:0/1245916264 shutdown_connections 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 --2- 172.21.15.97:0/1245916264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc91006e580 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 -- 172.21.15.97:0/1245916264 >> 172.21.15.97:0/1245916264 conn(0x7fc9100cc4f0 msgr2=0x7fc9100cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 -- 172.21.15.97:0/1245916264 shutdown_connections 2024-03-06T07:32:05.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.488+0000 7fc916a00640 1 -- 172.21.15.97:0/1245916264 wait complete. 2024-03-06T07:32:05.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.490+0000 7fc916a00640 1 Processor -- start 2024-03-06T07:32:05.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.490+0000 7fc916a00640 1 -- start start 2024-03-06T07:32:05.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc916a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc91015ac90 0x7fc9101561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc916a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc9100b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc916a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 0x7fc9100b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc916a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015f5b0 con 0x7fc91015f120 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc916a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100ceee0 con 0x7fc91015b070 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc916a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015f730 con 0x7fc91015ac90 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc914a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 0x7fc9100b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc914a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 0x7fc9100b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40142/0 (socket says 172.21.15.97:40142) 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc914a00640 1 -- 172.21.15.97:0/2860763384 learned_addr learned my addr 172.21.15.97:0/2860763384 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc90fe00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc91015ac90 0x7fc9101561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.491+0000 7fc907e00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc9100b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc914a00640 1 -- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc91015ac90 msgr2=0x7fc9101561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc914a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc91015ac90 0x7fc9101561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc914a00640 1 -- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 msgr2=0x7fc9100b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc914a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc9100b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc914a00640 1 -- 172.21.15.97:0/2860763384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100c6cc0 con 0x7fc91015f120 2024-03-06T07:32:05.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc914a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 0x7fc9100b0910 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c700 tx=0x7fc90800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908003b70 con 0x7fc91015f120 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc908003d10 con 0x7fc91015f120 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9080052d0 con 0x7fc91015f120 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9100c6fa0 con 0x7fc91015f120 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.492+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9100c7560 con 0x7fc91015f120 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.493+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91004f330 con 0x7fc91015f120 2024-03-06T07:32:05.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.494+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc908005470 con 0x7fc91015f120 2024-03-06T07:32:05.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.495+0000 7fc90cc00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.495+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc908089cd0 con 0x7fc91015f120 2024-03-06T07:32:05.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.496+0000 7fc90fe00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.496+0000 7fc90fe00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc900000f60 tx=0x7fc900007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.499+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc908058760 con 0x7fc91015f120 2024-03-06T07:32:05.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.708+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc9100cd1c0 con 0x7fc91015f120 2024-03-06T07:32:05.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.832+0000 7fc90cc00640 1 -- 172.21.15.97:0/2860763384 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc90805c800 con 0x7fc91015f120 2024-03-06T07:32:05.835 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:32:05.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc8ec06b4d0 msgr2=0x7fc8ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc900000f60 tx=0x7fc900007d30 comp rx=0 tx=0).stop 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 msgr2=0x7fc9100b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 0x7fc9100b0910 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c700 tx=0x7fc90800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 shutdown_connections 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc91015f120 0x7fc9100b0910 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc91015b070 0x7fc9100b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 --2- 172.21.15.97:0/2860763384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc91015ac90 0x7fc9101561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 >> 172.21.15.97:0/2860763384 conn(0x7fc9100cc4f0 msgr2=0x7fc9100ae0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.836+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 shutdown_connections 2024-03-06T07:32:05.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:05.837+0000 7fc916a00640 1 -- 172.21.15.97:0/2860763384 wait complete. 2024-03-06T07:32:05.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:32:06.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.053+0000 7fb1eee00640 1 Processor -- start 2024-03-06T07:32:06.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.053+0000 7fb1eee00640 1 -- start start 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1eee00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb1e8144c00 0x7fb1e806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 0x7fb1e80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1eee00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e806ee70 con 0x7fb1e815f9d0 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1eee00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e806cef0 con 0x7fb1e8144c00 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1eee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e806d1d0 con 0x7fb1e815fe60 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1e7400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb1e815f9d0 0x7fb1e80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41672/0 (socket says 172.21.15.97:41672) 2024-03-06T07:32:06.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1e7400640 1 -- 172.21.15.97:0/878061714 learned_addr learned my addr 172.21.15.97:0/878061714 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1ece00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 0x7fb1e80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1e7e00640 1 --2- 172.21.15.97:0/878061714 >> v2:172.21.15.135:3300/0 conn(0x7fb1e8144c00 0x7fb1e806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.055+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628147031 0 0) 0x7fb1d4001110 con 0x7fb1e815f9d0 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1cc003520 con 0x7fb1e815f9d0 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1029725854 0 0) 0x7fb1cc003520 con 0x7fb1e815f9d0 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 msgr2=0x7fb1e80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 0x7fb1e80ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 >> v2:172.21.15.135:3300/0 conn(0x7fb1e8144c00 msgr2=0x7fb1e806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 --2- 172.21.15.97:0/878061714 >> v2:172.21.15.135:3300/0 conn(0x7fb1e8144c00 0x7fb1e806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e806d4b0 con 0x7fb1e815f9d0 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb1d4001840 con 0x7fb1e815f9d0 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 >> v1:172.21.15.97:6789/0 conn(0x7fb1e815f9d0 legacy=0x7fb1e80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc0039b0 0x7fb1cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 0x7fb1cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e806cef0 con 0x7fb1cc004d40 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e806d1d0 con 0x7fb1cc0039b0 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc00b9c0 con 0x7fb1e815fe60 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.056+0000 7fb1ece00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.057+0000 7fb1e7e00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc0039b0 0x7fb1cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.057+0000 7fb1ece00640 1 -- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 msgr2=0x7fb1cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.057+0000 7fb1ece00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815fe60 0x7fb1cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.057+0000 7fb1ece00640 1 -- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc0039b0 msgr2=0x7fb1cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.057+0000 7fb1ece00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc0039b0 0x7fb1cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.057+0000 7fb1ece00640 1 -- 172.21.15.97:0/878061714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc003520 con 0x7fb1cc004d40 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.058+0000 7fb1ece00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0007390 tx=0x7fb1cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aa27771363a520 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.058+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0039070 con 0x7fb1cc004d40 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.058+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e002ccf0 con 0x7fb1cc004d40 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.058+0000 7fb1e6a00640 1 -- 172.21.15.97:0/878061714 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0009be0 con 0x7fb1cc004d40 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1eee00640 1 -- 172.21.15.97:0/878061714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1cc004d40 msgr2=0x7fb1cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1eee00640 1 --2- 172.21.15.97:0/878061714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0007390 tx=0x7fb1cc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1ece00640 1 -- 172.21.15.97:0/878061714 reap_dead start 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1eee00640 1 -- 172.21.15.97:0/878061714 shutdown_connections 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1eee00640 1 -- 172.21.15.97:0/878061714 >> 172.21.15.97:0/878061714 conn(0x7fb1e80b1730 msgr2=0x7fb1e80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1eee00640 1 -- 172.21.15.97:0/878061714 shutdown_connections 2024-03-06T07:32:06.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.059+0000 7fb1eee00640 1 -- 172.21.15.97:0/878061714 wait complete. 2024-03-06T07:32:06.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.060+0000 7fb1eee00640 1 Processor -- start 2024-03-06T07:32:06.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.061+0000 7fb1eee00640 1 -- start start 2024-03-06T07:32:06.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.061+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc004d40 0x7fb1e8143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.061+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815e710 0x7fb1e813fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.061+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 0x7fb1e813cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1eee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e8160730 con 0x7fb1e813ff90 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1eee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e8160290 con 0x7fb1cc004d40 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1eee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e806cb50 con 0x7fb1e815e710 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 0x7fb1e813cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 0x7fb1e813cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40164/0 (socket says 172.21.15.97:40164) 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 -- 172.21.15.97:0/3775361938 learned_addr learned my addr 172.21.15.97:0/3775361938 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1e7e00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc004d40 0x7fb1e8143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1e7400640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815e710 0x7fb1e813fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 -- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815e710 msgr2=0x7fb1e813fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815e710 0x7fb1e813fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 -- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc004d40 msgr2=0x7fb1e8143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc004d40 0x7fb1e8143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1ece00640 1 -- 172.21.15.97:0/3775361938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8143780 con 0x7fb1e813ff90 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.062+0000 7fb1e7e00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc004d40 0x7fb1e8143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:06.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.063+0000 7fb1ece00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 0x7fb1e813cf10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0002230 tx=0x7fb1e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.063+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0039030 con 0x7fb1e813ff90 2024-03-06T07:32:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.063+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e0043070 con 0x7fb1e813ff90 2024-03-06T07:32:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.063+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0009be0 con 0x7fb1e813ff90 2024-03-06T07:32:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.063+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e815d390 con 0x7fb1e813ff90 2024-03-06T07:32:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.063+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e815d950 con 0x7fb1e813ff90 2024-03-06T07:32:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.064+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e8144390 con 0x7fb1e813ff90 2024-03-06T07:32:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.065+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1e003d7f0 con 0x7fb1e813ff90 2024-03-06T07:32:06.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.066+0000 7fb1dfe00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb1c806b4d0 0x7fb1c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.066+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb1e00ac180 con 0x7fb1e813ff90 2024-03-06T07:32:06.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.066+0000 7fb1e7e00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb1c806b4d0 0x7fb1c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.067+0000 7fb1e7e00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb1c806b4d0 0x7fb1c806d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009a80 tx=0x7fb1d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.070+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1e8144390 con 0x7fb1e813ff90 2024-03-06T07:32:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.336+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb1e804f330 con 0x7fb1e813ff90 2024-03-06T07:32:06.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.338+0000 7fb1dfe00640 1 -- 172.21.15.97:0/3775361938 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb1e00aae20 con 0x7fb1e813ff90 2024-03-06T07:32:06.340 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:06.340 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:32:06.340 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:32:06.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:32:06.341 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:06.342 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.343 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:32:06.344 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.345 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.349 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:32:06.350 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.351 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.352 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:32:06.354 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.340+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb1c806b4d0 msgr2=0x7fb1c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.340+0000 7fb1eee00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb1c806b4d0 0x7fb1c806d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009a80 tx=0x7fb1d8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.340+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 msgr2=0x7fb1e813cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.340+0000 7fb1eee00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 0x7fb1e813cf10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0002230 tx=0x7fb1e0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 shutdown_connections 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb1c806b4d0 0x7fb1c806d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1e813ff90 0x7fb1e813cf10 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb1e815e710 0x7fb1e813fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 --2- 172.21.15.97:0/3775361938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1cc004d40 0x7fb1e8143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 >> 172.21.15.97:0/3775361938 conn(0x7fb1e80b1730 msgr2=0x7fb1e8071670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 shutdown_connections 2024-03-06T07:32:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.341+0000 7fb1eee00640 1 -- 172.21.15.97:0/3775361938 wait complete. 2024-03-06T07:32:06.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:06.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.562+0000 7fc6a3000640 1 Processor -- start 2024-03-06T07:32:06.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.562+0000 7fc6a3000640 1 -- start start 2024-03-06T07:32:06.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.563+0000 7fc6a3000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc69c0af1c0 0x7fc69c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.563+0000 7fc6a3000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c070d20 0x7fc69c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc6a3000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c071d30 con 0x7fc69c06d1a0 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc6a3000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc69c071490 con 0x7fc69c0af1c0 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc6a3000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c071770 con 0x7fc69c070d20 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc69c0af1c0 0x7fc69c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc69c0af1c0 0x7fc69c15eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40168/0 (socket says 172.21.15.97:40168) 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 -- 172.21.15.97:0/1377340695 learned_addr learned my addr 172.21.15.97:0/1377340695 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69b400640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c070d20 0x7fc69c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 -- 172.21.15.97:0/1377340695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c070d20 msgr2=0x7fc69c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c070d20 0x7fc69c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 -- 172.21.15.97:0/1377340695 >> v1:172.21.15.135:6789/0 conn(0x7fc69c06d1a0 legacy=0x7fc69c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69be00640 1 -- 172.21.15.97:0/1377340695 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c071a50 con 0x7fc69c0af1c0 2024-03-06T07:32:06.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.564+0000 7fc69b400640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c070d20 0x7fc69c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:06.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.566+0000 7fc69be00640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c0af1c0 0x7fc69c15eff0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009870 tx=0x7fc68c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f7686787f2853f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.566+0000 7fc69aa00640 1 -- 172.21.15.97:0/1377340695 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c035020 con 0x7fc69c0af1c0 2024-03-06T07:32:06.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.566+0000 7fc69aa00640 1 -- 172.21.15.97:0/1377340695 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c031640 con 0x7fc69c0af1c0 2024-03-06T07:32:06.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.566+0000 7fc69aa00640 1 -- 172.21.15.97:0/1377340695 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c02d040 con 0x7fc69c0af1c0 2024-03-06T07:32:06.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 -- 172.21.15.97:0/1377340695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c0af1c0 msgr2=0x7fc69c15eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c0af1c0 0x7fc69c15eff0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009870 tx=0x7fc68c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 -- 172.21.15.97:0/1377340695 shutdown_connections 2024-03-06T07:32:06.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c070d20 0x7fc69c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 --2- 172.21.15.97:0/1377340695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c0af1c0 0x7fc69c15eff0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 -- 172.21.15.97:0/1377340695 >> 172.21.15.97:0/1377340695 conn(0x7fc69c0b1730 msgr2=0x7fc69c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 -- 172.21.15.97:0/1377340695 shutdown_connections 2024-03-06T07:32:06.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.567+0000 7fc6a3000640 1 -- 172.21.15.97:0/1377340695 wait complete. 2024-03-06T07:32:06.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.569+0000 7fc6a3000640 1 Processor -- start 2024-03-06T07:32:06.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.569+0000 7fc6a3000640 1 -- start start 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc6a3000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc69c06d1a0 0x7fc69c1f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc6a3000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 0x7fc69c1f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc6a3000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c0af1c0 0x7fc69c160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc6a3000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c06ebe0 con 0x7fc69c070d20 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc6a3000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c0711a0 con 0x7fc69c06d1a0 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc6a3000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c06ed60 con 0x7fc69c0af1c0 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc69b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 0x7fc69c1f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc69b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 0x7fc69c1f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40176/0 (socket says 172.21.15.97:40176) 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.570+0000 7fc69b400640 1 -- 172.21.15.97:0/2280096716 learned_addr learned my addr 172.21.15.97:0/2280096716 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69be00640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc69c06d1a0 0x7fc69c1f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc6a1000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c0af1c0 0x7fc69c160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69b400640 1 -- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c0af1c0 msgr2=0x7fc69c160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69b400640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c0af1c0 0x7fc69c160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69b400640 1 -- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc69c06d1a0 msgr2=0x7fc69c1f8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69b400640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc69c06d1a0 0x7fc69c1f8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69b400640 1 -- 172.21.15.97:0/2280096716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c1f5dc0 con 0x7fc69c070d20 2024-03-06T07:32:06.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc6a1000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c0af1c0 0x7fc69c160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc69b400640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 0x7fc69c1f3e80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc68800c820 tx=0x7fc68800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688015070 con 0x7fc69c070d20 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc688007a30 con 0x7fc69c070d20 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688007e00 con 0x7fc69c070d20 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.571+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c1f8c90 con 0x7fc69c070d20 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.572+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c0ce750 con 0x7fc69c070d20 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.573+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c04f330 con 0x7fc69c070d20 2024-03-06T07:32:06.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.573+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc688018070 con 0x7fc69c070d20 2024-03-06T07:32:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.574+0000 7fc693e00640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc67c06b4d0 0x7fc67c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.574+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc688089e00 con 0x7fc69c070d20 2024-03-06T07:32:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.575+0000 7fc69be00640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc67c06b4d0 0x7fc67c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.575+0000 7fc69be00640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc67c06b4d0 0x7fc67c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc68c036000 tx=0x7fc68c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.579+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc69c1f8c90 con 0x7fc69c070d20 2024-03-06T07:32:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.778+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc69c15beb0 con 0x7fc69c070d20 2024-03-06T07:32:06.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.780+0000 7fc693e00640 1 -- 172.21.15.97:0/2280096716 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12880 (secure 0 0 0) 0x7fc69c1f5dc0 con 0x7fc69c070d20 2024-03-06T07:32:06.783 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:06.783 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":78,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:05.847726+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:06.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc67c06b4d0 msgr2=0x7fc67c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc67c06b4d0 0x7fc67c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc68c036000 tx=0x7fc68c03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 msgr2=0x7fc69c1f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 0x7fc69c1f3e80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc68800c820 tx=0x7fc68800cc20 comp rx=0 tx=0).stop 2024-03-06T07:32:06.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 shutdown_connections 2024-03-06T07:32:06.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc67c06b4d0 0x7fc67c06d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc69c0af1c0 0x7fc69c160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c070d20 0x7fc69c1f3e80 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 --2- 172.21.15.97:0/2280096716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc69c06d1a0 0x7fc69c1f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.783+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 >> 172.21.15.97:0/2280096716 conn(0x7fc69c0b1730 msgr2=0x7fc69c0afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.784+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 shutdown_connections 2024-03-06T07:32:06.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:06.784+0000 7fc6a3000640 1 -- 172.21.15.97:0/2280096716 wait complete. 2024-03-06T07:32:06.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:06.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:07.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.009+0000 7fc3ff200640 1 Processor -- start 2024-03-06T07:32:07.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.010+0000 7fc3ff200640 1 -- start start 2024-03-06T07:32:07.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3ff200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc4000c5230 0x7fc4000cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3ff200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc4000b2ea0 0x7fc40015f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3ff200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc400162b30 con 0x7fc40006dbf0 2024-03-06T07:32:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3ff200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc400163700 con 0x7fc4000b2ea0 2024-03-06T07:32:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3ff200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc400162dd0 con 0x7fc4000c5230 2024-03-06T07:32:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3fde00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc40006dbf0 0x7fc4000c4b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41686/0 (socket says 172.21.15.97:41686) 2024-03-06T07:32:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.011+0000 7fc3fde00640 1 -- 172.21.15.97:0/3258576040 learned_addr learned my addr 172.21.15.97:0/3258576040 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fe800640 1 --2- 172.21.15.97:0/3258576040 >> v2:172.21.15.97:3300/0 conn(0x7fc4000b2ea0 0x7fc40015f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fca00640 1 -- 172.21.15.97:0/3258576040 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302828564 0 0) 0x7fc400162b30 con 0x7fc40006dbf0 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fd400640 1 --2- 172.21.15.97:0/3258576040 >> v2:172.21.15.135:3300/0 conn(0x7fc4000c5230 0x7fc4000cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fca00640 1 -- 172.21.15.97:0/3258576040 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e4003520 con 0x7fc40006dbf0 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fe800640 1 -- 172.21.15.97:0/3258576040 >> v2:172.21.15.135:3300/0 conn(0x7fc4000c5230 msgr2=0x7fc4000cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fe800640 1 --2- 172.21.15.97:0/3258576040 >> v2:172.21.15.135:3300/0 conn(0x7fc4000c5230 0x7fc4000cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fe800640 1 -- 172.21.15.97:0/3258576040 >> v1:172.21.15.97:6789/0 conn(0x7fc40006dbf0 legacy=0x7fc4000c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fe800640 1 -- 172.21.15.97:0/3258576040 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4001630b0 con 0x7fc4000b2ea0 2024-03-06T07:32:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.012+0000 7fc3fd400640 1 --2- 172.21.15.97:0/3258576040 >> v2:172.21.15.135:3300/0 conn(0x7fc4000c5230 0x7fc4000cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.013+0000 7fc3fe800640 1 --2- 172.21.15.97:0/3258576040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000b2ea0 0x7fc40015f7e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc400163670 tx=0x7fc3f802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8243be345228aee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.013+0000 7fc3fca00640 1 -- 172.21.15.97:0/3258576040 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8038070 con 0x7fc4000b2ea0 2024-03-06T07:32:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.013+0000 7fc3fca00640 1 -- 172.21.15.97:0/3258576040 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f8031710 con 0x7fc4000b2ea0 2024-03-06T07:32:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.013+0000 7fc3fca00640 1 -- 172.21.15.97:0/3258576040 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8031ae0 con 0x7fc4000b2ea0 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 -- 172.21.15.97:0/3258576040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000b2ea0 msgr2=0x7fc40015f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 --2- 172.21.15.97:0/3258576040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000b2ea0 0x7fc40015f7e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc400163670 tx=0x7fc3f802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 -- 172.21.15.97:0/3258576040 shutdown_connections 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 --2- 172.21.15.97:0/3258576040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000b2ea0 0x7fc40015f7e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 --2- 172.21.15.97:0/3258576040 >> v2:172.21.15.135:3300/0 conn(0x7fc4000c5230 0x7fc4000cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 -- 172.21.15.97:0/3258576040 >> 172.21.15.97:0/3258576040 conn(0x7fc40009a9f0 msgr2=0x7fc40009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 -- 172.21.15.97:0/3258576040 shutdown_connections 2024-03-06T07:32:07.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.014+0000 7fc3ff200640 1 -- 172.21.15.97:0/3258576040 wait complete. 2024-03-06T07:32:07.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.016+0000 7fc3ff200640 1 Processor -- start 2024-03-06T07:32:07.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.016+0000 7fc3ff200640 1 -- start start 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.017+0000 7fc3ff200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 0x7fc400159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.017+0000 7fc3ff200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4000b2ea0 0x7fc4000c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.017+0000 7fc3ff200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4000c5230 0x7fc4000d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.017+0000 7fc3ff200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40015fd20 con 0x7fc40006dbf0 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.017+0000 7fc3ff200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000b3350 con 0x7fc4000c5230 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.017+0000 7fc3ff200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40015fea0 con 0x7fc4000b2ea0 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 0x7fc400159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 0x7fc400159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40190/0 (socket says 172.21.15.97:40190) 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 -- 172.21.15.97:0/96985227 learned_addr learned my addr 172.21.15.97:0/96985227 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 -- 172.21.15.97:0/96985227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4000b2ea0 msgr2=0x7fc4000c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4000b2ea0 0x7fc4000c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 -- 172.21.15.97:0/96985227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4000c5230 msgr2=0x7fc4000d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4000c5230 0x7fc4000d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 -- 172.21.15.97:0/96985227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4000c5ff0 con 0x7fc40006dbf0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3fde00640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 0x7fc400159360 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c460 tx=0x7fc3e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.018+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8017070 con 0x7fc40006dbf0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.019+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e8005ce0 con 0x7fc40006dbf0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.019+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8007290 con 0x7fc40006dbf0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.019+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4000c62d0 con 0x7fc40006dbf0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.019+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4000b4710 con 0x7fc40006dbf0 2024-03-06T07:32:07.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.020+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40004f330 con 0x7fc40006dbf0 2024-03-06T07:32:07.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.021+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3e801a070 con 0x7fc40006dbf0 2024-03-06T07:32:07.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.022+0000 7fc3f6000640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.022+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3e8089230 con 0x7fc40006dbf0 2024-03-06T07:32:07.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.022+0000 7fc3fd400640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.023+0000 7fc3fd400640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc4000c8610 tx=0x7fc3f0008330 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.027+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3e8058400 con 0x7fc40006dbf0 2024-03-06T07:32:07.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.221+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4000bbf40 con 0x7fc40006dbf0 2024-03-06T07:32:07.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.222+0000 7fc3f6000640 1 -- 172.21.15.97:0/96985227 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3e8057d00 con 0x7fc40006dbf0 2024-03-06T07:32:07.224 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:32:07.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3dc06b4d0 msgr2=0x7fc3dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc4000c8610 tx=0x7fc3f0008330 comp rx=0 tx=0).stop 2024-03-06T07:32:07.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 msgr2=0x7fc400159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 0x7fc400159360 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c460 tx=0x7fc3e800c920 comp rx=0 tx=0).stop 2024-03-06T07:32:07.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 shutdown_connections 2024-03-06T07:32:07.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4000c5230 0x7fc4000d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc4000b2ea0 0x7fc4000c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 --2- 172.21.15.97:0/96985227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc40006dbf0 0x7fc400159360 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.224+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 >> 172.21.15.97:0/96985227 conn(0x7fc40009a9f0 msgr2=0x7fc40006fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.225+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 shutdown_connections 2024-03-06T07:32:07.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.225+0000 7fc3ff200640 1 -- 172.21.15.97:0/96985227 wait complete. 2024-03-06T07:32:07.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:32:07.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.450+0000 7fac1be00640 1 Processor -- start 2024-03-06T07:32:07.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.450+0000 7fac1be00640 1 -- start start 2024-03-06T07:32:07.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.451+0000 7fac1be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac1c1561f0 0x7fac1c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.451+0000 7fac1be00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fac1c0aebf0 0x7fac1c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c0cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1be00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c15ce30 con 0x7fac1c1561f0 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1be00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c15c240 con 0x7fac1c0aebf0 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c15c500 con 0x7fac1c0cc800 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac1c1561f0 0x7fac1c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac1c1561f0 0x7fac1c0ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40206/0 (socket says 172.21.15.97:40206) 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 -- 172.21.15.97:0/379974987 learned_addr learned my addr 172.21.15.97:0/379974987 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1a000640 1 --2- 172.21.15.97:0/379974987 >> v2:172.21.15.135:3300/0 conn(0x7fac1c0aebf0 0x7fac1c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 -- 172.21.15.97:0/379974987 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 msgr2=0x7fac1c0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 --2- 172.21.15.97:0/379974987 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c0cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 -- 172.21.15.97:0/379974987 >> v2:172.21.15.135:3300/0 conn(0x7fac1c0aebf0 msgr2=0x7fac1c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 --2- 172.21.15.97:0/379974987 >> v2:172.21.15.135:3300/0 conn(0x7fac1c0aebf0 0x7fac1c0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1aa00640 1 -- 172.21.15.97:0/379974987 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c15c7e0 con 0x7fac1c1561f0 2024-03-06T07:32:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.452+0000 7fac1a000640 1 --2- 172.21.15.97:0/379974987 >> v2:172.21.15.135:3300/0 conn(0x7fac1c0aebf0 0x7fac1c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.453+0000 7fac1aa00640 1 --2- 172.21.15.97:0/379974987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c1561f0 0x7fac1c0ae6b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fac1c068400 tx=0x7fac0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8613d866ed48c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac19600640 1 -- 172.21.15.97:0/379974987 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c035020 con 0x7fac1c1561f0 2024-03-06T07:32:07.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac19600640 1 -- 172.21.15.97:0/379974987 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c031870 con 0x7fac1c1561f0 2024-03-06T07:32:07.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac19600640 1 -- 172.21.15.97:0/379974987 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c031c40 con 0x7fac1c1561f0 2024-03-06T07:32:07.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 -- 172.21.15.97:0/379974987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c1561f0 msgr2=0x7fac1c0ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 --2- 172.21.15.97:0/379974987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c1561f0 0x7fac1c0ae6b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fac1c068400 tx=0x7fac0c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 -- 172.21.15.97:0/379974987 shutdown_connections 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 --2- 172.21.15.97:0/379974987 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c0cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 --2- 172.21.15.97:0/379974987 >> v2:172.21.15.135:3300/0 conn(0x7fac1c0aebf0 0x7fac1c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 --2- 172.21.15.97:0/379974987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c1561f0 0x7fac1c0ae6b0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.454+0000 7fac1be00640 1 -- 172.21.15.97:0/379974987 >> 172.21.15.97:0/379974987 conn(0x7fac1c06afc0 msgr2=0x7fac1c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.455+0000 7fac1be00640 1 -- 172.21.15.97:0/379974987 shutdown_connections 2024-03-06T07:32:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.455+0000 7fac1be00640 1 -- 172.21.15.97:0/379974987 wait complete. 2024-03-06T07:32:07.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.456+0000 7fac1be00640 1 Processor -- start 2024-03-06T07:32:07.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.456+0000 7fac1be00640 1 -- start start 2024-03-06T07:32:07.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.457+0000 7fac1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 0x7fac1c157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.457+0000 7fac1be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.457+0000 7fac1be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac1c1561f0 0x7fac1c158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.457+0000 7fac1be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c1565d0 con 0x7fac1c0aebf0 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.457+0000 7fac1be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0ccc90 con 0x7fac1c1561f0 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.457+0000 7fac1be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c156750 con 0x7fac1c0cc800 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 0x7fac1c157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 0x7fac1c157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40218/0 (socket says 172.21.15.97:40218) 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 -- 172.21.15.97:0/2028419920 learned_addr learned my addr 172.21.15.97:0/2028419920 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1b400640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac1c1561f0 0x7fac1c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1a000640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 -- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 msgr2=0x7fac1c15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c15b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 -- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac1c1561f0 msgr2=0x7fac1c158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac1c1561f0 0x7fac1c158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1aa00640 1 -- 172.21.15.97:0/2028419920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c159040 con 0x7fac1c0aebf0 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.458+0000 7fac1b400640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac1c1561f0 0x7fac1c158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.459+0000 7fac1aa00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 0x7fac1c157600 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000c00 tx=0x7fac0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.459+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c040070 con 0x7fac1c0aebf0 2024-03-06T07:32:07.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.459+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c03a420 con 0x7fac1c0aebf0 2024-03-06T07:32:07.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.459+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c035040 con 0x7fac1c0aebf0 2024-03-06T07:32:07.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.459+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c156b20 con 0x7fac1c0aebf0 2024-03-06T07:32:07.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.459+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c1570e0 con 0x7fac1c0aebf0 2024-03-06T07:32:07.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.460+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c04f330 con 0x7fac1c0aebf0 2024-03-06T07:32:07.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.461+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac0c04b060 con 0x7fac1c0aebf0 2024-03-06T07:32:07.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.462+0000 7fac13400640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fabf806b4d0 0x7fabf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.462+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fac0c0abf10 con 0x7fac1c0aebf0 2024-03-06T07:32:07.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.463+0000 7fac1a000640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fabf806b4d0 0x7fabf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.463+0000 7fac1a000640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fabf806b4d0 0x7fabf806d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0adf50 tx=0x7fac04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.467+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac0c07b040 con 0x7fac1c0aebf0 2024-03-06T07:32:07.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.665+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fac1c0b2b10 con 0x7fac1c0aebf0 2024-03-06T07:32:07.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.666+0000 7fac13400640 1 -- 172.21.15.97:0/2028419920 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7fac0c07a940 con 0x7fac1c0aebf0 2024-03-06T07:32:07.668 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:07.669 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:32:07.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:07.669 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:32:07.669 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:32:07.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:07.669 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.668+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fabf806b4d0 msgr2=0x7fabf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.668+0000 7fac1be00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fabf806b4d0 0x7fabf806d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0adf50 tx=0x7fac04008040 comp rx=0 tx=0).stop 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.668+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 msgr2=0x7fac1c157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.668+0000 7fac1be00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 0x7fac1c157600 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000c00 tx=0x7fac0c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 shutdown_connections 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fabf806b4d0 0x7fabf806d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac1c1561f0 0x7fac1c158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac1c0cc800 0x7fac1c15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 --2- 172.21.15.97:0/2028419920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac1c0aebf0 0x7fac1c157600 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 >> 172.21.15.97:0/2028419920 conn(0x7fac1c06afc0 msgr2=0x7fac1c071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 shutdown_connections 2024-03-06T07:32:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.669+0000 7fac1be00640 1 -- 172.21.15.97:0/2028419920 wait complete. 2024-03-06T07:32:07.689 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:32:07.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:32:07.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:07.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.892+0000 7f843a200640 1 Processor -- start 2024-03-06T07:32:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.893+0000 7f843a200640 1 -- start start 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.894+0000 7f843a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f843406d170 0x7f843406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.894+0000 7f843a200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8434162b40 con 0x7f8434161d80 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.894+0000 7f843a200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843406b060 con 0x7f8434158940 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.894+0000 7f843a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8434162920 con 0x7f843406d170 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f8433e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8434161d80 0x7f8434162210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41690/0 (socket says 172.21.15.97:41690) 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f8433e00640 1 -- 172.21.15.97:0/1228535371 learned_addr learned my addr 172.21.15.97:0/1228535371 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f842aa00640 1 --2- 172.21.15.97:0/1228535371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f843406d170 0x7f843406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f8432a00640 1 -- 172.21.15.97:0/1228535371 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294871112 0 0) 0x7f8434162b40 con 0x7f8434161d80 2024-03-06T07:32:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f8432a00640 1 -- 172.21.15.97:0/1228535371 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8418003520 con 0x7f8434161d80 2024-03-06T07:32:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f8432a00640 1 -- 172.21.15.97:0/1228535371 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800321851 0 0) 0x7f843406b060 con 0x7f8434158940 2024-03-06T07:32:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f842aa00640 1 -- 172.21.15.97:0/1228535371 >> v1:172.21.15.135:6789/0 conn(0x7f8434158940 legacy=0x7f84341417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f842aa00640 1 -- 172.21.15.97:0/1228535371 >> v1:172.21.15.97:6789/0 conn(0x7f8434161d80 legacy=0x7f8434162210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.895+0000 7f842aa00640 1 -- 172.21.15.97:0/1228535371 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843406b340 con 0x7f843406d170 2024-03-06T07:32:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.897+0000 7f842aa00640 1 --2- 172.21.15.97:0/1228535371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f843406d170 0x7f843406d5d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f841c0097c0 tx=0x7f841c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64160015b068b59f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.897+0000 7f8432a00640 1 -- 172.21.15.97:0/1228535371 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c035020 con 0x7f843406d170 2024-03-06T07:32:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.897+0000 7f8432a00640 1 -- 172.21.15.97:0/1228535371 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c031890 con 0x7f843406d170 2024-03-06T07:32:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.897+0000 7f8432a00640 1 -- 172.21.15.97:0/1228535371 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c031c40 con 0x7f843406d170 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 -- 172.21.15.97:0/1228535371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f843406d170 msgr2=0x7f843406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 --2- 172.21.15.97:0/1228535371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f843406d170 0x7f843406d5d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f841c0097c0 tx=0x7f841c02b930 comp rx=0 tx=0).stop 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 -- 172.21.15.97:0/1228535371 shutdown_connections 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 --2- 172.21.15.97:0/1228535371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f843406d170 0x7f843406d5d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 -- 172.21.15.97:0/1228535371 >> 172.21.15.97:0/1228535371 conn(0x7f84340ccee0 msgr2=0x7f84340cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 -- 172.21.15.97:0/1228535371 shutdown_connections 2024-03-06T07:32:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.898+0000 7f843a200640 1 -- 172.21.15.97:0/1228535371 wait complete. 2024-03-06T07:32:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.900+0000 7f843a200640 1 Processor -- start 2024-03-06T07:32:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.900+0000 7f843a200640 1 -- start start 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f843a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 0x7f8434157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f843a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434161d80 0x7f843415f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f843a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8434136260 0x7f84340b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f843a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843406b900 con 0x7f843406d170 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f843a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843406be20 con 0x7f8434161d80 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f843a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843406bfa0 con 0x7f8434136260 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f8433400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 0x7f8434157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f8433400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 0x7f8434157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40222/0 (socket says 172.21.15.97:40222) 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.901+0000 7f8433400640 1 -- 172.21.15.97:0/20573959 learned_addr learned my addr 172.21.15.97:0/20573959 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433e00640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8434136260 0x7f84340b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f842aa00640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434161d80 0x7f843415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433400640 1 -- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8434136260 msgr2=0x7f84340b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433400640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8434136260 0x7f84340b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433400640 1 -- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434161d80 msgr2=0x7f843415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433400640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434161d80 0x7f843415f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433400640 1 -- 172.21.15.97:0/20573959 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434157740 con 0x7f843406d170 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f842aa00640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434161d80 0x7f843415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f8433400640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 0x7f8434157200 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f842400c890 tx=0x7f842400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424017070 con 0x7f843406d170 2024-03-06T07:32:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424003dc0 con 0x7f843406d170 2024-03-06T07:32:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84240052b0 con 0x7f843406d170 2024-03-06T07:32:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.902+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84341f71a0 con 0x7f843406d170 2024-03-06T07:32:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.903+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84341f7760 con 0x7f843406d170 2024-03-06T07:32:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.904+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843404f330 con 0x7f843406d170 2024-03-06T07:32:07.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.905+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f842401a070 con 0x7f843406d170 2024-03-06T07:32:07.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.906+0000 7f842be00640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841006b4d0 0x7f841006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.906+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8424089610 con 0x7f843406d170 2024-03-06T07:32:07.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.906+0000 7f842aa00640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841006b4d0 0x7f841006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.907+0000 7f842aa00640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841006b4d0 0x7f841006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f841c036000 tx=0x7f841c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:07.911+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8434157740 con 0x7f843406d170 2024-03-06T07:32:08.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.118+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f84340684a0 con 0x7f843406d170 2024-03-06T07:32:08.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.857+0000 7f842be00640 1 -- 172.21.15.97:0/20573959 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f84240587f0 con 0x7f843406d170 2024-03-06T07:32:08.860 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841006b4d0 msgr2=0x7f841006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841006b4d0 0x7f841006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f841c036000 tx=0x7f841c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 msgr2=0x7f8434157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 0x7f8434157200 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f842400c890 tx=0x7f842400cd50 comp rx=0 tx=0).stop 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 shutdown_connections 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841006b4d0 0x7f841006d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8434136260 0x7f84340b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434161d80 0x7f843415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 --2- 172.21.15.97:0/20573959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f843406d170 0x7f8434157200 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 >> 172.21.15.97:0/20573959 conn(0x7f84340ccee0 msgr2=0x7f84340b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.861+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 shutdown_connections 2024-03-06T07:32:08.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:08.862+0000 7f843a200640 1 -- 172.21.15.97:0/20573959 wait complete. 2024-03-06T07:32:08.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:32:09.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.094+0000 7f7742a00640 1 Processor -- start 2024-03-06T07:32:09.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.094+0000 7f7742a00640 1 -- start start 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.096+0000 7f7742a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f773c160700 0x7f773c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.096+0000 7f7742a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0b46c0 con 0x7f773c15fc10 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.096+0000 7f7742a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0b49a0 con 0x7f773c0b3f70 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.096+0000 7f7742a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c072c90 con 0x7f773c160700 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.096+0000 7f7740a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f773c160700 0x7f773c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f773be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f773c15fc10 0x7f773c15fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41700/0 (socket says 172.21.15.97:41700) 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f7740a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f773c160700 0x7f773c070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40234/0 (socket says 172.21.15.97:40234) 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f773be00640 1 -- 172.21.15.97:0/1298386583 learned_addr learned my addr 172.21.15.97:0/1298386583 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f7740a00640 1 -- 172.21.15.97:0/1298386583 >> v1:172.21.15.135:6789/0 conn(0x7f773c0b3f70 legacy=0x7f773c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f7740a00640 1 -- 172.21.15.97:0/1298386583 >> v1:172.21.15.97:6789/0 conn(0x7f773c15fc10 legacy=0x7f773c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f7740a00640 1 -- 172.21.15.97:0/1298386583 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c09dd80 con 0x7f773c160700 2024-03-06T07:32:09.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.097+0000 7f773b400640 1 -- 172.21.15.97:0/1298386583 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669130136 0 0) 0x7f773c0b46c0 con 0x7f773c15fc10 2024-03-06T07:32:09.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.098+0000 7f7740a00640 1 --2- 172.21.15.97:0/1298386583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c070ff0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f7734009d90 tx=0x7f773402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1f48c7214b9f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.098+0000 7f773b400640 1 -- 172.21.15.97:0/1298386583 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734039070 con 0x7f773c160700 2024-03-06T07:32:09.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.098+0000 7f773b400640 1 -- 172.21.15.97:0/1298386583 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773402cc60 con 0x7f773c160700 2024-03-06T07:32:09.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.098+0000 7f773b400640 1 -- 172.21.15.97:0/1298386583 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734034660 con 0x7f773c160700 2024-03-06T07:32:09.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 -- 172.21.15.97:0/1298386583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 msgr2=0x7f773c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 --2- 172.21.15.97:0/1298386583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c070ff0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f7734009d90 tx=0x7f773402beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 -- 172.21.15.97:0/1298386583 shutdown_connections 2024-03-06T07:32:09.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 --2- 172.21.15.97:0/1298386583 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c070ff0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 -- 172.21.15.97:0/1298386583 >> 172.21.15.97:0/1298386583 conn(0x7f773c0b1940 msgr2=0x7f773c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 -- 172.21.15.97:0/1298386583 shutdown_connections 2024-03-06T07:32:09.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.099+0000 7f7742a00640 1 -- 172.21.15.97:0/1298386583 wait complete. 2024-03-06T07:32:09.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.101+0000 7f7742a00640 1 Processor -- start 2024-03-06T07:32:09.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.101+0000 7f7742a00640 1 -- start start 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7742a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c0b3f70 0x7f773c158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7742a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15fc10 0x7f773c06e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7742a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7742a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c15c460 con 0x7f773c160700 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7742a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c15c2e0 con 0x7f773c15fc10 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7742a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0cebb0 con 0x7f773c0b3f70 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7740a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7740a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40240/0 (socket says 172.21.15.97:40240) 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.102+0000 7f7740a00640 1 -- 172.21.15.97:0/3491417419 learned_addr learned my addr 172.21.15.97:0/3491417419 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f773be00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c0b3f70 0x7f773c158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7733e00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15fc10 0x7f773c06e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7740a00640 1 -- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c0b3f70 msgr2=0x7f773c158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7740a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c0b3f70 0x7f773c158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7740a00640 1 -- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15fc10 msgr2=0x7f773c06e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7740a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15fc10 0x7f773c06e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7740a00640 1 -- 172.21.15.97:0/3491417419 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c09b5a0 con 0x7f773c160700 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7733e00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15fc10 0x7f773c06e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.103+0000 7f7740a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c157440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7734009d90 tx=0x7f7734004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.104+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734044020 con 0x7f773c160700 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.104+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773403f470 con 0x7f773c160700 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.104+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c09b880 con 0x7f773c160700 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.104+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734039040 con 0x7f773c160700 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.104+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c09be40 con 0x7f773c160700 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.105+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c04f330 con 0x7f773c160700 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.106+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7734047020 con 0x7f773c160700 2024-03-06T07:32:09.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.107+0000 7f7738c00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b4d0 0x7f771806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.107+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f77340acd00 con 0x7f773c160700 2024-03-06T07:32:09.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.107+0000 7f773be00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b4d0 0x7f771806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.108+0000 7f773be00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b4d0 0x7f771806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f772c000f60 tx=0x7f772c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.112+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f773404e030 con 0x7f773c160700 2024-03-06T07:32:09.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.309+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f773c06ef30 con 0x7f773c160700 2024-03-06T07:32:09.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.878+0000 7f7738c00640 1 -- 172.21.15.97:0/3491417419 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7f773407bc30 con 0x7f773c160700 2024-03-06T07:32:09.880 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.882+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b4d0 msgr2=0x7f771806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.882+0000 7f7742a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b4d0 0x7f771806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f772c000f60 tx=0x7f772c007d50 comp rx=0 tx=0).stop 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.882+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 msgr2=0x7f773c157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.882+0000 7f7742a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c157440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7734009d90 tx=0x7f7734004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.883+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 shutdown_connections 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.883+0000 7f7742a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b4d0 0x7f771806d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.883+0000 7f7742a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c160700 0x7f773c157440 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.883+0000 7f7742a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15fc10 0x7f773c06e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.883+0000 7f7742a00640 1 --2- 172.21.15.97:0/3491417419 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c0b3f70 0x7f773c158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.883+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 >> 172.21.15.97:0/3491417419 conn(0x7f773c0b1940 msgr2=0x7f773c0ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.885+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 shutdown_connections 2024-03-06T07:32:09.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:09.886+0000 7f7742a00640 1 -- 172.21.15.97:0/3491417419 wait complete. 2024-03-06T07:32:09.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:32:10.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.105+0000 7f2f12000640 1 Processor -- start 2024-03-06T07:32:10.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.105+0000 7f2f12000640 1 -- start start 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f12000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3530 0x7f2f0c15b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f12000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2f0c0b3910 0x7f2f0c0acf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f12000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c15bcc0 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f12000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c0ce690 con 0x7f2f0c0b3910 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f12000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0ce970 con 0x7f2f0c0b3530 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f0be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2f0c0ad4a0 0x7f2f0c0ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41704/0 (socket says 172.21.15.97:41704) 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f0be00640 1 -- 172.21.15.97:0/3909815269 learned_addr learned my addr 172.21.15.97:0/3909815269 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f0b400640 1 --2- 172.21.15.97:0/3909815269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3530 0x7f2f0c15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.107+0000 7f2f0aa00640 1 --2- 172.21.15.97:0/3909815269 >> v2:172.21.15.135:3300/0 conn(0x7f2f0c0b3910 0x7f2f0c0acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0a000640 1 -- 172.21.15.97:0/3909815269 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836734836 0 0) 0x7f2f04001490 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0a000640 1 -- 172.21.15.97:0/3909815269 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef0003520 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0b400640 1 -- 172.21.15.97:0/3909815269 >> v2:172.21.15.135:3300/0 conn(0x7f2f0c0b3910 msgr2=0x7f2f0c0acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0b400640 1 --2- 172.21.15.97:0/3909815269 >> v2:172.21.15.135:3300/0 conn(0x7f2f0c0b3910 0x7f2f0c0acf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0b400640 1 -- 172.21.15.97:0/3909815269 >> v1:172.21.15.97:6789/0 conn(0x7f2f0c0ad4a0 legacy=0x7f2f0c0ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0a000640 1 -- 172.21.15.97:0/3909815269 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2553591341 0 0) 0x7f2ef0003520 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.108+0000 7f2f0b400640 1 -- 172.21.15.97:0/3909815269 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0cec50 con 0x7f2f0c0b3530 2024-03-06T07:32:10.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.110+0000 7f2f0b400640 1 --2- 172.21.15.97:0/3909815269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3530 0x7f2f0c15b780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0097d0 tx=0x7f2efc02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7f558f0397ee6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.110+0000 7f2f0a000640 1 -- 172.21.15.97:0/3909815269 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc037070 con 0x7f2f0c0b3530 2024-03-06T07:32:10.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.110+0000 7f2f0a000640 1 -- 172.21.15.97:0/3909815269 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc030510 con 0x7f2f0c0b3530 2024-03-06T07:32:10.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.110+0000 7f2f0a000640 1 -- 172.21.15.97:0/3909815269 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04001bc0 con 0x7f2f0c0b3530 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 -- 172.21.15.97:0/3909815269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3530 msgr2=0x7f2f0c15b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 --2- 172.21.15.97:0/3909815269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3530 0x7f2f0c15b780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0097d0 tx=0x7f2efc02b770 comp rx=0 tx=0).stop 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 -- 172.21.15.97:0/3909815269 shutdown_connections 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 --2- 172.21.15.97:0/3909815269 >> v2:172.21.15.135:3300/0 conn(0x7f2f0c0b3910 0x7f2f0c0acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 --2- 172.21.15.97:0/3909815269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3530 0x7f2f0c15b780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 -- 172.21.15.97:0/3909815269 >> 172.21.15.97:0/3909815269 conn(0x7f2f0c0b1940 msgr2=0x7f2f0c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 -- 172.21.15.97:0/3909815269 shutdown_connections 2024-03-06T07:32:10.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.111+0000 7f2f12000640 1 -- 172.21.15.97:0/3909815269 wait complete. 2024-03-06T07:32:10.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.113+0000 7f2f12000640 1 Processor -- start 2024-03-06T07:32:10.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.113+0000 7f2f12000640 1 -- start start 2024-03-06T07:32:10.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f12000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 0x7f2f0c1f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f12000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f0c0b3530 0x7f2f0c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f12000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3910 0x7f2f0c15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f12000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0b3d90 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f12000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0cf210 con 0x7f2f0c0b3530 2024-03-06T07:32:10.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f12000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c15aa30 con 0x7f2f0c0b3910 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f0b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 0x7f2f0c1f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f0b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 0x7f2f0c1f0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40248/0 (socket says 172.21.15.97:40248) 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.114+0000 7f2f0b400640 1 -- 172.21.15.97:0/4222211308 learned_addr learned my addr 172.21.15.97:0/4222211308 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.115+0000 7f2f0b400640 1 -- 172.21.15.97:0/4222211308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3910 msgr2=0x7f2f0c15d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.115+0000 7f2f0b400640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3910 0x7f2f0c15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.115+0000 7f2f0b400640 1 -- 172.21.15.97:0/4222211308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f0c0b3530 msgr2=0x7f2f0c14d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.115+0000 7f2f0b400640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f0c0b3530 0x7f2f0c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.115+0000 7f2f0b400640 1 -- 172.21.15.97:0/4222211308 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c09db70 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.115+0000 7f2f0b400640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 0x7f2f0c1f0240 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2efc000c00 tx=0x7f2efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.116+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc041020 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.116+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc03c870 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.116+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c1f09b0 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.116+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc037040 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.116+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c1f0f70 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.117+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c06f5e0 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.118+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2efc044020 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.119+0000 7f2f03400640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2eec06b4d0 0x7f2eec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.119+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2efc0abde0 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.119+0000 7f2f0aa00640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2eec06b4d0 0x7f2eec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.120+0000 7f2f0aa00640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2eec06b4d0 0x7f2eec06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8006a90 tx=0x7f2ef8006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.123+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f0c06f5e0 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.357+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2f0c14fe60 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.892+0000 7f2f03400640 1 -- 172.21.15.97:0/4222211308 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v48) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2f0c14fe60 con 0x7f2f0c0ad4a0 2024-03-06T07:32:10.894 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:32:10.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2eec06b4d0 msgr2=0x7f2eec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2eec06b4d0 0x7f2eec06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8006a90 tx=0x7f2ef8006a20 comp rx=0 tx=0).stop 2024-03-06T07:32:10.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 msgr2=0x7f2f0c1f0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 0x7f2f0c1f0240 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2efc000c00 tx=0x7f2efc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 shutdown_connections 2024-03-06T07:32:10.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2eec06b4d0 0x7f2eec06d990 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8006a90 tx=0x7f2ef8006a20 comp rx=0 tx=0).stop 2024-03-06T07:32:10.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f0c0b3910 0x7f2f0c15d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f0c0b3530 0x7f2f0c14d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 --2- 172.21.15.97:0/4222211308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f0c0ad4a0 0x7f2f0c1f0240 secure :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2efc000c00 tx=0x7f2efc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.900+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 >> 172.21.15.97:0/4222211308 conn(0x7f2f0c0b1940 msgr2=0x7f2f0c0741a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.902+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 shutdown_connections 2024-03-06T07:32:10.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:10.903+0000 7f2f12000640 1 -- 172.21.15.97:0/4222211308 wait complete. 2024-03-06T07:32:10.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:11.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.104+0000 7f173ea00640 1 Processor -- start 2024-03-06T07:32:11.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.105+0000 7f173ea00640 1 -- start start 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f173ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f173ea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1738071260 0x7f173806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f173ea00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17380717c0 con 0x7f173806d090 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f173ea00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173806d4c0 con 0x7f1738071260 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f173ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1738071aa0 con 0x7f17380cf430 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f1737400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1738071260 0x7f173806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f1737400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1738071260 0x7f173806cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40256/0 (socket says 172.21.15.97:40256) 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.106+0000 7f1737400640 1 -- 172.21.15.97:0/1969665373 learned_addr learned my addr 172.21.15.97:0/1969665373 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.107+0000 7f1737400640 1 -- 172.21.15.97:0/1969665373 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 msgr2=0x7f1738070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.107+0000 7f1737e00640 1 --2- 172.21.15.97:0/1969665373 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.107+0000 7f1737400640 1 --2- 172.21.15.97:0/1969665373 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.107+0000 7f1737400640 1 -- 172.21.15.97:0/1969665373 >> v1:172.21.15.135:6789/0 conn(0x7f173806d090 legacy=0x7f1738165f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.107+0000 7f1737400640 1 -- 172.21.15.97:0/1969665373 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1738071d80 con 0x7f1738071260 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.108+0000 7f1737400640 1 --2- 172.21.15.97:0/1969665373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f173806cb50 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1720009e50 tx=0x7f172002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae04a325acd61907 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.108+0000 7f1736a00640 1 -- 172.21.15.97:0/1969665373 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172002ca70 con 0x7f1738071260 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.108+0000 7f1736a00640 1 -- 172.21.15.97:0/1969665373 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172002cc10 con 0x7f1738071260 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.108+0000 7f1736a00640 1 -- 172.21.15.97:0/1969665373 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17200343f0 con 0x7f1738071260 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 -- 172.21.15.97:0/1969665373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 msgr2=0x7f173806cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 --2- 172.21.15.97:0/1969665373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f173806cb50 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1720009e50 tx=0x7f172002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 -- 172.21.15.97:0/1969665373 shutdown_connections 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 --2- 172.21.15.97:0/1969665373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f173806cb50 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 --2- 172.21.15.97:0/1969665373 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 -- 172.21.15.97:0/1969665373 >> 172.21.15.97:0/1969665373 conn(0x7f17380b1730 msgr2=0x7f17380b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 -- 172.21.15.97:0/1969665373 shutdown_connections 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.109+0000 7f173ea00640 1 -- 172.21.15.97:0/1969665373 wait complete. 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.111+0000 7f173ea00640 1 Processor -- start 2024-03-06T07:32:11.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.111+0000 7f173ea00640 1 -- start start 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f173ea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f173806d090 0x7f173815f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f173ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f17381ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f173ea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738147920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f173ea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1738162b70 con 0x7f1738071260 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f173ea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1738072340 con 0x7f173806d090 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f173ea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1738162cf0 con 0x7f17380cf430 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f17381ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f17381ebd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40272/0 (socket says 172.21.15.97:40272) 2024-03-06T07:32:11.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 -- 172.21.15.97:0/437207901 learned_addr learned my addr 172.21.15.97:0/437207901 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737e00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f173806d090 0x7f173815f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 -- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 msgr2=0x7f1738147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738147920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 -- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f173806d090 msgr2=0x7f173815f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f173806d090 0x7f173815f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.112+0000 7f1737400640 1 -- 172.21.15.97:0/437207901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173815e020 con 0x7f1738071260 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.113+0000 7f1737400640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f17381ebd80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1720009e50 tx=0x7f1720004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.113+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720043020 con 0x7f1738071260 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.113+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172003e450 con 0x7f1738071260 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.113+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173815e300 con 0x7f1738071260 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.113+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172003e840 con 0x7f1738071260 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.113+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173815e8c0 con 0x7f1738071260 2024-03-06T07:32:11.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.114+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1738072550 con 0x7f1738071260 2024-03-06T07:32:11.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.115+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1720046020 con 0x7f1738071260 2024-03-06T07:32:11.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.116+0000 7f172fe00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f171406b4d0 0x7f171406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.116+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17200acca0 con 0x7f1738071260 2024-03-06T07:32:11.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.116+0000 7f1737e00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f171406b4d0 0x7f171406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.117+0000 7f1737e00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f171406b4d0 0x7f171406d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f172800b7c0 tx=0x7f1728008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.120+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1738072550 con 0x7f1738071260 2024-03-06T07:32:11.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.309+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f173804f330 con 0x7f1738071260 2024-03-06T07:32:11.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.312+0000 7f172fe00640 1 -- 172.21.15.97:0/437207901 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f172004d030 con 0x7f1738071260 2024-03-06T07:32:11.313 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:11.314 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":81,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:10.357996+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.314+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f171406b4d0 msgr2=0x7f171406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.314+0000 7f173ea00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f171406b4d0 0x7f171406d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f172800b7c0 tx=0x7f1728008040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.314+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 msgr2=0x7f17381ebd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.314+0000 7f173ea00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f17381ebd80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1720009e50 tx=0x7f1720004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 shutdown_connections 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f171406b4d0 0x7f171406d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f17380cf430 0x7f1738147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1738071260 0x7f17381ebd80 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 --2- 172.21.15.97:0/437207901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f173806d090 0x7f173815f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 >> 172.21.15.97:0/437207901 conn(0x7f17380b1730 msgr2=0x7f17380b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 shutdown_connections 2024-03-06T07:32:11.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.315+0000 7f173ea00640 1 -- 172.21.15.97:0/437207901 wait complete. 2024-03-06T07:32:11.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:11.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.528+0000 7f4fc7c00640 1 Processor -- start 2024-03-06T07:32:11.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.528+0000 7f4fc7c00640 1 -- start start 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc7c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc015f9d0 0x7f4fc015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc7c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fc0070b10 0x7f4fc0070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc7c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc006cef0 con 0x7f4fc0144c00 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc7c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc0160370 con 0x7f4fc0070b10 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc7c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006d1d0 con 0x7f4fc015f9d0 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc5200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4fc0144c00 0x7f4fc006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41710/0 (socket says 172.21.15.97:41710) 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc5200640 1 -- 172.21.15.97:0/2084228689 learned_addr learned my addr 172.21.15.97:0/2084228689 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:11.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fc5c00640 1 --2- 172.21.15.97:0/2084228689 >> v2:172.21.15.97:3300/0 conn(0x7f4fc0070b10 0x7f4fc0070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.530+0000 7f4fbfe00640 1 --2- 172.21.15.97:0/2084228689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc015f9d0 0x7f4fc015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.531+0000 7f4fc5c00640 1 -- 172.21.15.97:0/2084228689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc015f9d0 msgr2=0x7f4fc015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.531+0000 7f4fc5c00640 1 --2- 172.21.15.97:0/2084228689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc015f9d0 0x7f4fc015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.531+0000 7f4fc5c00640 1 -- 172.21.15.97:0/2084228689 >> v1:172.21.15.97:6789/0 conn(0x7f4fc0144c00 legacy=0x7f4fc006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.531+0000 7f4fc5c00640 1 -- 172.21.15.97:0/2084228689 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc006d4b0 con 0x7f4fc0070b10 2024-03-06T07:32:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.531+0000 7f4fbf400640 1 -- 172.21.15.97:0/2084228689 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011272115 0 0) 0x7f4fc006cef0 con 0x7f4fc0144c00 2024-03-06T07:32:11.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.532+0000 7f4fc5c00640 1 --2- 172.21.15.97:0/2084228689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc0070f60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009d90 tx=0x7f4fb802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef7cad7a06e4064 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.532+0000 7f4fbf400640 1 -- 172.21.15.97:0/2084228689 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8039070 con 0x7f4fc0070b10 2024-03-06T07:32:11.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.532+0000 7f4fbf400640 1 -- 172.21.15.97:0/2084228689 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb802cc50 con 0x7f4fc0070b10 2024-03-06T07:32:11.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.532+0000 7f4fbf400640 1 -- 172.21.15.97:0/2084228689 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8034440 con 0x7f4fc0070b10 2024-03-06T07:32:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.532+0000 7f4fc7c00640 1 -- 172.21.15.97:0/2084228689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 msgr2=0x7f4fc0070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.532+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/2084228689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc0070f60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009d90 tx=0x7f4fb802bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.533+0000 7f4fc7c00640 1 -- 172.21.15.97:0/2084228689 shutdown_connections 2024-03-06T07:32:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.533+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/2084228689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc0070f60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.533+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/2084228689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc015f9d0 0x7f4fc015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.533+0000 7f4fc7c00640 1 -- 172.21.15.97:0/2084228689 >> 172.21.15.97:0/2084228689 conn(0x7f4fc00b1730 msgr2=0x7f4fc00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.533+0000 7f4fc7c00640 1 -- 172.21.15.97:0/2084228689 shutdown_connections 2024-03-06T07:32:11.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.533+0000 7f4fc7c00640 1 -- 172.21.15.97:0/2084228689 wait complete. 2024-03-06T07:32:11.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.534+0000 7f4fc7c00640 1 Processor -- start 2024-03-06T07:32:11.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.534+0000 7f4fc7c00640 1 -- start start 2024-03-06T07:32:11.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.535+0000 7f4fc7c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc01426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.535+0000 7f4fc7c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc0144c00 0x7f4fc00b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.535+0000 7f4fc7c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fc015f9d0 0x7f4fc01393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.535+0000 7f4fc7c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006cb50 con 0x7f4fc0070b10 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.535+0000 7f4fc7c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc0160730 con 0x7f4fc015f9d0 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.535+0000 7f4fc7c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006ccd0 con 0x7f4fc0144c00 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc01426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc01426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40302/0 (socket says 172.21.15.97:40302) 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 -- 172.21.15.97:0/3272600776 learned_addr learned my addr 172.21.15.97:0/3272600776 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fbfe00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc0144c00 0x7f4fc00b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 -- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc0144c00 msgr2=0x7f4fc00b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc0144c00 0x7f4fc00b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 -- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fc015f9d0 msgr2=0x7f4fc01393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fc015f9d0 0x7f4fc01393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 -- 172.21.15.97:0/3272600776 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0139920 con 0x7f4fc0070b10 2024-03-06T07:32:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.536+0000 7f4fc5200640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc01426b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400c4d0 tx=0x7f4fb400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.537+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4018070 con 0x7f4fc0070b10 2024-03-06T07:32:11.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.537+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb40074e0 con 0x7f4fc0070b10 2024-03-06T07:32:11.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.537+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4004a20 con 0x7f4fc0070b10 2024-03-06T07:32:11.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.537+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc00b4980 con 0x7f4fc0070b10 2024-03-06T07:32:11.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.537+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc0140e30 con 0x7f4fc0070b10 2024-03-06T07:32:11.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.538+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc004f330 con 0x7f4fc0070b10 2024-03-06T07:32:11.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.539+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fb4004bc0 con 0x7f4fc0070b10 2024-03-06T07:32:11.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.539+0000 7f4fbcc00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa006b500 0x7f4fa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.540+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fb40893c0 con 0x7f4fc0070b10 2024-03-06T07:32:11.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.540+0000 7f4fbfe00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa006b500 0x7f4fa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.540+0000 7f4fbfe00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa006b500 0x7f4fa006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0072810 tx=0x7f4fac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.544+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fc0139920 con 0x7f4fc0070b10 2024-03-06T07:32:11.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.743+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc0072890 con 0x7f4fc0070b10 2024-03-06T07:32:11.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.745+0000 7f4fbcc00640 1 -- 172.21.15.97:0/3272600776 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f4fb4058240 con 0x7f4fc0070b10 2024-03-06T07:32:11.746 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:11.747 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":81,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:10.357996+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:11.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa006b500 msgr2=0x7f4fa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa006b500 0x7f4fa006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0072810 tx=0x7f4fac008040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 msgr2=0x7f4fc01426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc01426b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400c4d0 tx=0x7f4fb400c990 comp rx=0 tx=0).stop 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 shutdown_connections 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4fa006b500 0x7f4fa006d9c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fc015f9d0 0x7f4fc01393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fc0144c00 0x7f4fc00b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 --2- 172.21.15.97:0/3272600776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fc0070b10 0x7f4fc01426b0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 >> 172.21.15.97:0/3272600776 conn(0x7f4fc00b1730 msgr2=0x7f4fc00b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 shutdown_connections 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.748+0000 7f4fc7c00640 1 -- 172.21.15.97:0/3272600776 wait complete. 2024-03-06T07:32:11.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:32:11.843 INFO:tasks.ceph.mds.a.smithi097.stderr:2024-03-06T07:32:11.842+0000 7f41ad000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:11.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.982+0000 7f3b71000640 1 Processor -- start 2024-03-06T07:32:11.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.982+0000 7f3b71000640 1 -- start start 2024-03-06T07:32:11.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b71000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b6c071b10 0x7f3b6c15c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b71000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3b6c071ef0 0x7f3b6c145270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b71000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c0d0030 con 0x7f3b6c1457b0 2024-03-06T07:32:11.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b71000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c0d0310 con 0x7f3b6c071b10 2024-03-06T07:32:11.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b71000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c0d05f0 con 0x7f3b6c071ef0 2024-03-06T07:32:11.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b6c071b10 0x7f3b6c15c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b6c071b10 0x7f3b6c15c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40306/0 (socket says 172.21.15.97:40306) 2024-03-06T07:32:11.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 -- 172.21.15.97:0/2118797388 learned_addr learned my addr 172.21.15.97:0/2118797388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:11.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 -- 172.21.15.97:0/2118797388 >> v2:172.21.15.135:3300/0 conn(0x7f3b6c071ef0 msgr2=0x7f3b6c145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 --2- 172.21.15.97:0/2118797388 >> v2:172.21.15.135:3300/0 conn(0x7f3b6c071ef0 0x7f3b6c145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 -- 172.21.15.97:0/2118797388 >> v1:172.21.15.135:6789/0 conn(0x7f3b6c1457b0 legacy=0x7f3b6c0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.984+0000 7f3b69e00640 1 -- 172.21.15.97:0/2118797388 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c0b3b40 con 0x7f3b6c071b10 2024-03-06T07:32:11.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.985+0000 7f3b69e00640 1 --2- 172.21.15.97:0/2118797388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15c2e0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c075080 tx=0x7f3b5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c586d01718bf60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b69400640 1 -- 172.21.15.97:0/2118797388 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c035020 con 0x7f3b6c071b10 2024-03-06T07:32:11.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b69400640 1 -- 172.21.15.97:0/2118797388 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c031870 con 0x7f3b6c071b10 2024-03-06T07:32:11.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b69400640 1 -- 172.21.15.97:0/2118797388 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c031c40 con 0x7f3b6c071b10 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 -- 172.21.15.97:0/2118797388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 msgr2=0x7f3b6c15c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 --2- 172.21.15.97:0/2118797388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15c2e0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c075080 tx=0x7f3b5c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 -- 172.21.15.97:0/2118797388 shutdown_connections 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 --2- 172.21.15.97:0/2118797388 >> v2:172.21.15.135:3300/0 conn(0x7f3b6c071ef0 0x7f3b6c145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 --2- 172.21.15.97:0/2118797388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15c2e0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 -- 172.21.15.97:0/2118797388 >> 172.21.15.97:0/2118797388 conn(0x7f3b6c0b2840 msgr2=0x7f3b6c0b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 -- 172.21.15.97:0/2118797388 shutdown_connections 2024-03-06T07:32:11.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.986+0000 7f3b71000640 1 -- 172.21.15.97:0/2118797388 wait complete. 2024-03-06T07:32:11.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.988+0000 7f3b71000640 1 Processor -- start 2024-03-06T07:32:11.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.988+0000 7f3b71000640 1 -- start start 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b71000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b71000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b6c071ef0 0x7f3b6c13f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b71000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b6c1457b0 0x7f3b6c15e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b71000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c0cfcc0 con 0x7f3b6c071b10 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b71000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c072380 con 0x7f3b6c071ef0 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b71000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c144e30 con 0x7f3b6c1457b0 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b69e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b69e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40310/0 (socket says 172.21.15.97:40310) 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.989+0000 7f3b69e00640 1 -- 172.21.15.97:0/1056782934 learned_addr learned my addr 172.21.15.97:0/1056782934 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b63e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b6c071ef0 0x7f3b6c13f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b69e00640 1 -- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b6c1457b0 msgr2=0x7f3b6c15e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b69e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b6c1457b0 0x7f3b6c15e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b69e00640 1 -- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b6c071ef0 msgr2=0x7f3b6c13f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b69e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b6c071ef0 0x7f3b6c13f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b69e00640 1 -- 172.21.15.97:0/1056782934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c13fb90 con 0x7f3b6c071b10 2024-03-06T07:32:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b63e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b6c071ef0 0x7f3b6c13f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b69e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15b650 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c000c00 tx=0x7f3b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c03ad60 con 0x7f3b6c071b10 2024-03-06T07:32:11.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c0443f0 con 0x7f3b6c071b10 2024-03-06T07:32:11.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c13fe70 con 0x7f3b6c071b10 2024-03-06T07:32:11.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0447e0 con 0x7f3b6c071b10 2024-03-06T07:32:11.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.990+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c140430 con 0x7f3b6c071b10 2024-03-06T07:32:11.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.991+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c04f330 con 0x7f3b6c071b10 2024-03-06T07:32:11.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.993+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b5c04b060 con 0x7f3b6c071b10 2024-03-06T07:32:11.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.994+0000 7f3b62000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4806b500 0x7f3b4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.995+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b5c0ac460 con 0x7f3b6c071b10 2024-03-06T07:32:11.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.995+0000 7f3b63e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4806b500 0x7f3b4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:11.995+0000 7f3b63e00640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4806b500 0x7f3b4806d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3b54006fd0 tx=0x7f3b54008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.000+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b5c03f070 con 0x7f3b6c071b10 2024-03-06T07:32:12.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.233+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3b6c0b1d10 con 0x7f3b6c071b10 2024-03-06T07:32:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.903+0000 7f3b62000640 1 -- 172.21.15.97:0/1056782934 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3b5c02f320 con 0x7f3b6c071b10 2024-03-06T07:32:12.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4806b500 msgr2=0x7f3b4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4806b500 0x7f3b4806d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3b54006fd0 tx=0x7f3b54008040 comp rx=0 tx=0).stop 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 msgr2=0x7f3b6c15b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15b650 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c000c00 tx=0x7f3b5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 shutdown_connections 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b4806b500 0x7f3b4806d9c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b6c1457b0 0x7f3b6c15e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b6c071ef0 0x7f3b6c13f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 --2- 172.21.15.97:0/1056782934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b6c071b10 0x7f3b6c15b650 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.907+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 >> 172.21.15.97:0/1056782934 conn(0x7f3b6c0b2840 msgr2=0x7f3b6c0afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.908+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 shutdown_connections 2024-03-06T07:32:12.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:12.908+0000 7f3b71000640 1 -- 172.21.15.97:0/1056782934 wait complete. 2024-03-06T07:32:12.928 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:32:12.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:32:13.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.128+0000 7f6a6ce00640 1 Processor -- start 2024-03-06T07:32:13.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.129+0000 7f6a6ce00640 1 -- start start 2024-03-06T07:32:13.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.130+0000 7f6a6ce00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6a6815e3f0 0x7f6a680d2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.130+0000 7f6a6ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 0x7f6a680d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.130+0000 7f6a6ce00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680d4e00 con 0x7f6a680d1060 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.130+0000 7f6a6ce00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680b3ce0 con 0x7f6a6815e3f0 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.130+0000 7f6a6ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b33b0 con 0x7f6a680b5180 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.130+0000 7f6a65400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6a680d1060 0x7f6a680ceac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41726/0 (socket says 172.21.15.97:41726) 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a65400640 1 -- 172.21.15.97:0/4154026159 learned_addr learned my addr 172.21.15.97:0/4154026159 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:13.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a65e00640 1 --2- 172.21.15.97:0/4154026159 >> v2:172.21.15.135:3300/0 conn(0x7f6a6815e3f0 0x7f6a680d2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a66800640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 0x7f6a680d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811474276 0 0) 0x7f6a680d4e00 con 0x7f6a680d1060 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c003520 con 0x7f6a680d1060 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3893445813 0 0) 0x7f6a4c003520 con 0x7f6a680d1060 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 msgr2=0x7f6a680d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 0x7f6a680d2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 >> v2:172.21.15.135:3300/0 conn(0x7f6a6815e3f0 msgr2=0x7f6a680d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 --2- 172.21.15.97:0/4154026159 >> v2:172.21.15.135:3300/0 conn(0x7f6a6815e3f0 0x7f6a680d2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680b3690 con 0x7f6a680d1060 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f6a50001670 con 0x7f6a680d1060 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.131+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 >> v1:172.21.15.97:6789/0 conn(0x7f6a680d1060 legacy=0x7f6a680ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a64a00640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a4c0039b0 0x7f6a4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a64a00640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 0x7f6a4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a64a00640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a4c004d40 0x7f6a4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b3ce0 con 0x7f6a4c004d40 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b33b0 con 0x7f6a4c0039b0 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c00b9c0 con 0x7f6a680b5180 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a66800640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a4c004d40 0x7f6a4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a65e00640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a4c0039b0 0x7f6a4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a65400640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 0x7f6a4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a66800640 1 -- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 msgr2=0x7f6a4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a66800640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a680b5180 0x7f6a4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a66800640 1 -- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a4c0039b0 msgr2=0x7f6a4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a66800640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a4c0039b0 0x7f6a4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.132+0000 7f6a66800640 1 -- 172.21.15.97:0/4154026159 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c003520 con 0x7f6a4c004d40 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.134+0000 7f6a66800640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a4c004d40 0x7f6a4c0051c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009ac0 tx=0x7f6a4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f658598f6b6220ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.134+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60039070 con 0x7f6a4c004d40 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.134+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6002c6b0 con 0x7f6a4c004d40 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.134+0000 7f6a64a00640 1 -- 172.21.15.97:0/4154026159 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60009690 con 0x7f6a4c004d40 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4154026159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a4c004d40 msgr2=0x7f6a4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4154026159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a4c004d40 0x7f6a4c0051c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009ac0 tx=0x7f6a4c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a66800640 1 -- 172.21.15.97:0/4154026159 reap_dead start 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4154026159 shutdown_connections 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4154026159 >> 172.21.15.97:0/4154026159 conn(0x7f6a680d3f20 msgr2=0x7f6a680d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4154026159 shutdown_connections 2024-03-06T07:32:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.135+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4154026159 wait complete. 2024-03-06T07:32:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.137+0000 7f6a6ce00640 1 Processor -- start 2024-03-06T07:32:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.137+0000 7f6a6ce00640 1 -- start start 2024-03-06T07:32:13.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a6ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 0x7f6a680b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a6ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a6809f150 0x7f6a681f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a6ce00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a680a1140 0x7f6a680a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a6ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680d4bc0 con 0x7f6a681f3fd0 2024-03-06T07:32:13.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a6ce00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680d4a40 con 0x7f6a680a1140 2024-03-06T07:32:13.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a6ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680d1d00 con 0x7f6a6809f150 2024-03-06T07:32:13.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 0x7f6a680b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 0x7f6a680b8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40324/0 (socket says 172.21.15.97:40324) 2024-03-06T07:32:13.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.138+0000 7f6a65e00640 1 -- 172.21.15.97:0/4270387355 learned_addr learned my addr 172.21.15.97:0/4270387355 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65400640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a6809f150 0x7f6a681f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a66800640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a680a1140 0x7f6a680a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65e00640 1 -- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a6809f150 msgr2=0x7f6a681f83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65e00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a6809f150 0x7f6a681f83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65e00640 1 -- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a680a1140 msgr2=0x7f6a680a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65e00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a680a1140 0x7f6a680a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65e00640 1 -- 172.21.15.97:0/4270387355 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680b8e50 con 0x7f6a681f3fd0 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a66800640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a680a1140 0x7f6a680a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a65e00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 0x7f6a680b8910 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009fd0 tx=0x7f6a580115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5801a020 con 0x7f6a681f3fd0 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58016470 con 0x7f6a681f3fd0 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58016840 con 0x7f6a681f3fd0 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.139+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6809f630 con 0x7f6a681f3fd0 2024-03-06T07:32:13.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.140+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a681f7bc0 con 0x7f6a681f3fd0 2024-03-06T07:32:13.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.141+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6804f330 con 0x7f6a681f3fd0 2024-03-06T07:32:13.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.142+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a58005d10 con 0x7f6a681f3fd0 2024-03-06T07:32:13.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.143+0000 7f6a5e000640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a4406b4d0 0x7f6a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.143+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a5808b2a0 con 0x7f6a681f3fd0 2024-03-06T07:32:13.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.143+0000 7f6a65400640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a4406b4d0 0x7f6a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.144+0000 7f6a65400640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a4406b4d0 0x7f6a4406d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6a50001eb0 tx=0x7f6a50001c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.147+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a58024030 con 0x7f6a681f3fd0 2024-03-06T07:32:13.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.476+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6a680b9080 con 0x7f6a681f3fd0 2024-03-06T07:32:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.913+0000 7f6a5e000640 1 -- 172.21.15.97:0/4270387355 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6a5805a080 con 0x7f6a681f3fd0 2024-03-06T07:32:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a4406b4d0 msgr2=0x7f6a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a4406b4d0 0x7f6a4406d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6a50001eb0 tx=0x7f6a50001c90 comp rx=0 tx=0).stop 2024-03-06T07:32:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 msgr2=0x7f6a680b8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 0x7f6a680b8910 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009fd0 tx=0x7f6a580115a0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 shutdown_connections 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a4406b4d0 0x7f6a4406d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a680a1140 0x7f6a680a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a6809f150 0x7f6a681f83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 --2- 172.21.15.97:0/4270387355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a681f3fd0 0x7f6a680b8910 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 >> 172.21.15.97:0/4270387355 conn(0x7f6a680d3f20 msgr2=0x7f6a6806c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.917+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 shutdown_connections 2024-03-06T07:32:13.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:13.918+0000 7f6a6ce00640 1 -- 172.21.15.97:0/4270387355 wait complete. 2024-03-06T07:32:13.946 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:32:13.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:14.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.150+0000 7fa9e7000640 1 Processor -- start 2024-03-06T07:32:14.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.151+0000 7fa9e7000640 1 -- start start 2024-03-06T07:32:14.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.152+0000 7fa9e7000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa9e01561f0 0x7fa9e00ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.152+0000 7fa9e7000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa9e00aebf0 0x7fa9e00cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.152+0000 7fa9e7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e00cc800 0x7fa9e00cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.152+0000 7fa9e7000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e015ce30 con 0x7fa9e01561f0 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.152+0000 7fa9e7000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e015c240 con 0x7fa9e00aebf0 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.152+0000 7fa9e7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e015c500 con 0x7fa9e00cc800 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa9e01561f0 0x7fa9e00ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa9e01561f0 0x7fa9e00ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38614/0 (socket says 172.21.15.97:38614) 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 -- 172.21.15.97:0/3555180450 learned_addr learned my addr 172.21.15.97:0/3555180450 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e6600640 1 --2- 172.21.15.97:0/3555180450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e00cc800 0x7fa9e00cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5200640 1 --2- 172.21.15.97:0/3555180450 >> v2:172.21.15.135:3300/0 conn(0x7fa9e00aebf0 0x7fa9e00cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 -- 172.21.15.97:0/3555180450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e00cc800 msgr2=0x7fa9e00cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3555180450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e00cc800 0x7fa9e00cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 -- 172.21.15.97:0/3555180450 >> v2:172.21.15.135:3300/0 conn(0x7fa9e00aebf0 msgr2=0x7fa9e00cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3555180450 >> v2:172.21.15.135:3300/0 conn(0x7fa9e00aebf0 0x7fa9e00cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.153+0000 7fa9e5c00640 1 -- 172.21.15.97:0/3555180450 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e015c7e0 con 0x7fa9e01561f0 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.154+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3555180450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e01561f0 0x7fa9e00ae6b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0068400 tx=0x7fa9d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a6fc6b35705170 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9dfe00640 1 -- 172.21.15.97:0/3555180450 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4035020 con 0x7fa9e01561f0 2024-03-06T07:32:14.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9dfe00640 1 -- 172.21.15.97:0/3555180450 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4031870 con 0x7fa9e01561f0 2024-03-06T07:32:14.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9dfe00640 1 -- 172.21.15.97:0/3555180450 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4031c40 con 0x7fa9e01561f0 2024-03-06T07:32:14.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 -- 172.21.15.97:0/3555180450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e01561f0 msgr2=0x7fa9e00ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3555180450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e01561f0 0x7fa9e00ae6b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0068400 tx=0x7fa9d402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 -- 172.21.15.97:0/3555180450 shutdown_connections 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3555180450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e00cc800 0x7fa9e00cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3555180450 >> v2:172.21.15.135:3300/0 conn(0x7fa9e00aebf0 0x7fa9e00cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3555180450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e01561f0 0x7fa9e00ae6b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 -- 172.21.15.97:0/3555180450 >> 172.21.15.97:0/3555180450 conn(0x7fa9e006afc0 msgr2=0x7fa9e006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 -- 172.21.15.97:0/3555180450 shutdown_connections 2024-03-06T07:32:14.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.155+0000 7fa9e7000640 1 -- 172.21.15.97:0/3555180450 wait complete. 2024-03-06T07:32:14.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.158+0000 7fa9e7000640 1 Processor -- start 2024-03-06T07:32:14.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.158+0000 7fa9e7000640 1 -- start start 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.159+0000 7fa9e7000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9e00aebf0 0x7fa9e01541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.159+0000 7fa9e7000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 0x7fa9e01572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.159+0000 7fa9e7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e01561f0 0x7fa9e0158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.159+0000 7fa9e7000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e01565d0 con 0x7fa9e00cc800 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.159+0000 7fa9e7000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00ccc90 con 0x7fa9e00aebf0 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.159+0000 7fa9e7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0156750 con 0x7fa9e01561f0 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 0x7fa9e01572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 0x7fa9e01572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38622/0 (socket says 172.21.15.97:38622) 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 -- 172.21.15.97:0/3265421192 learned_addr learned my addr 172.21.15.97:0/3265421192 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9e00aebf0 0x7fa9e01541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e6600640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e01561f0 0x7fa9e0158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 -- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e01561f0 msgr2=0x7fa9e0158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e01561f0 0x7fa9e0158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 -- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9e00aebf0 msgr2=0x7fa9e01541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9e00aebf0 0x7fa9e01541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 -- 172.21.15.97:0/3265421192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0157cc0 con 0x7fa9e00cc800 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9e00aebf0 0x7fa9e01541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.160+0000 7fa9e5200640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 0x7fa9e01572e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00d830 tx=0x7fa9cc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.161+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc018030 con 0x7fa9e00cc800 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.161+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc00be10 con 0x7fa9e00cc800 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.161+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e0157fa0 con 0x7fa9e00cc800 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.161+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc0098b0 con 0x7fa9e00cc800 2024-03-06T07:32:14.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.161+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e0158560 con 0x7fa9e00cc800 2024-03-06T07:32:14.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.162+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0158a40 con 0x7fa9e00cc800 2024-03-06T07:32:14.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.163+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9cc01b070 con 0x7fa9e00cc800 2024-03-06T07:32:14.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.164+0000 7fa9dd600640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9c006b4d0 0x7fa9c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.164+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9cc08a4b0 con 0x7fa9e00cc800 2024-03-06T07:32:14.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.164+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9c006b4d0 0x7fa9c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.165+0000 7fa9e5c00640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9c006b4d0 0x7fa9c006d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4036000 tx=0x7fa9d4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.168+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9e0158a40 con 0x7fa9e00cc800 2024-03-06T07:32:14.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.401+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9e00b2b10 con 0x7fa9e00cc800 2024-03-06T07:32:14.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.402+0000 7fa9dd600640 1 -- 172.21.15.97:0/3265421192 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fa9cc059290 con 0x7fa9e00cc800 2024-03-06T07:32:14.403 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:14.403 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:14.403 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:14.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9c006b4d0 msgr2=0x7fa9c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9c006b4d0 0x7fa9c006d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4036000 tx=0x7fa9d4008030 comp rx=0 tx=0).stop 2024-03-06T07:32:14.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 msgr2=0x7fa9e01572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 0x7fa9e01572e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00d830 tx=0x7fa9cc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 shutdown_connections 2024-03-06T07:32:14.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9c006b4d0 0x7fa9c006d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9e01561f0 0x7fa9e0158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e00cc800 0x7fa9e01572e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 --2- 172.21.15.97:0/3265421192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9e00aebf0 0x7fa9e01541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 >> 172.21.15.97:0/3265421192 conn(0x7fa9e006afc0 msgr2=0x7fa9e006e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 shutdown_connections 2024-03-06T07:32:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.404+0000 7fa9e7000640 1 -- 172.21.15.97:0/3265421192 wait complete. 2024-03-06T07:32:14.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:14.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.626+0000 7f6eb5400640 1 Processor -- start 2024-03-06T07:32:14.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.626+0000 7f6eb5400640 1 -- start start 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eb5400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6eb015e0d0 0x7f6eb00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eb5400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eb5400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb00cea60 con 0x7f6eb00d0df0 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eb5400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb00d1260 con 0x7f6eb015e0d0 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eb5400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00d2f60 con 0x7f6eb015f470 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eae800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6eb015e0d0 0x7f6eb00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eae800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6eb015e0d0 0x7f6eb00d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38634/0 (socket says 172.21.15.97:38634) 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6eae800640 1 -- 172.21.15.97:0/1616403616 learned_addr learned my addr 172.21.15.97:0/1616403616 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:14.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.628+0000 7f6ea5e00640 1 --2- 172.21.15.97:0/1616403616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.629+0000 7f6eae800640 1 -- 172.21.15.97:0/1616403616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 msgr2=0x7f6eb00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.629+0000 7f6eae800640 1 --2- 172.21.15.97:0/1616403616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.629+0000 7f6eae800640 1 -- 172.21.15.97:0/1616403616 >> v1:172.21.15.135:6789/0 conn(0x7f6eb00d0df0 legacy=0x7f6eb0160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.629+0000 7f6eae800640 1 -- 172.21.15.97:0/1616403616 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb00d2350 con 0x7f6eb015e0d0 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.630+0000 7f6eae800640 1 --2- 172.21.15.97:0/1616403616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00d1a90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009870 tx=0x7f6ea002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d111f4f99f8fe390 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.630+0000 7f6eade00640 1 -- 172.21.15.97:0/1616403616 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea002f830 con 0x7f6eb015e0d0 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.630+0000 7f6eade00640 1 -- 172.21.15.97:0/1616403616 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea002f9d0 con 0x7f6eb015e0d0 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.630+0000 7f6eade00640 1 -- 172.21.15.97:0/1616403616 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea002fda0 con 0x7f6eb015e0d0 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 -- 172.21.15.97:0/1616403616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 msgr2=0x7f6eb00d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 --2- 172.21.15.97:0/1616403616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00d1a90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009870 tx=0x7f6ea002ba10 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 -- 172.21.15.97:0/1616403616 shutdown_connections 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 --2- 172.21.15.97:0/1616403616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 --2- 172.21.15.97:0/1616403616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00d1a90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 -- 172.21.15.97:0/1616403616 >> 172.21.15.97:0/1616403616 conn(0x7f6eb00d3ec0 msgr2=0x7f6eb00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 -- 172.21.15.97:0/1616403616 shutdown_connections 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.631+0000 7f6eb5400640 1 -- 172.21.15.97:0/1616403616 wait complete. 2024-03-06T07:32:14.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.633+0000 7f6eb5400640 1 Processor -- start 2024-03-06T07:32:14.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.633+0000 7f6eb5400640 1 -- start start 2024-03-06T07:32:14.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eb5400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb00d0df0 0x7f6eb015bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eb5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eb5400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eb5400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00d2bf0 con 0x7f6eb015e0d0 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eb5400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00d2910 con 0x7f6eb00d0df0 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eb5400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb015fb30 con 0x7f6eb015f470 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6ea5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6ea5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38648/0 (socket says 172.21.15.97:38648) 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6ea5e00640 1 -- 172.21.15.97:0/384230350 learned_addr learned my addr 172.21.15.97:0/384230350 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eae800640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb00d0df0 0x7f6eb015bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.634+0000 7f6eaf200640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea5e00640 1 -- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 msgr2=0x7f6eb00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea5e00640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea5e00640 1 -- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb00d0df0 msgr2=0x7f6eb015bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea5e00640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb00d0df0 0x7f6eb015bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea5e00640 1 -- 172.21.15.97:0/384230350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb01f7fa0 con 0x7f6eb015e0d0 2024-03-06T07:32:14.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6eae800640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb00d0df0 0x7f6eb015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:14.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea5e00640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00cbfd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d830 tx=0x7f6e9800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800b660 con 0x7f6eb015e0d0 2024-03-06T07:32:14.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98019070 con 0x7f6eb015e0d0 2024-03-06T07:32:14.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb01f82a0 con 0x7f6eb015e0d0 2024-03-06T07:32:14.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800ba20 con 0x7f6eb015e0d0 2024-03-06T07:32:14.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.635+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb01f8d20 con 0x7f6eb015e0d0 2024-03-06T07:32:14.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.636+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb004f330 con 0x7f6eb015e0d0 2024-03-06T07:32:14.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.638+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e9801e460 con 0x7f6eb015e0d0 2024-03-06T07:32:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.639+0000 7f6ea7400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e8c06b4d0 0x7f6e8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.639+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e9808a920 con 0x7f6eb015e0d0 2024-03-06T07:32:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.640+0000 7f6eae800640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e8c06b4d0 0x7f6e8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.640+0000 7f6eae800640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e8c06b4d0 0x7f6e8c06d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0036000 tx=0x7f6ea0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.644+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6eb01f82a0 con 0x7f6eb015e0d0 2024-03-06T07:32:14.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.835+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eb00ca580 con 0x7f6eb015e0d0 2024-03-06T07:32:14.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.838+0000 7f6ea7400640 1 -- 172.21.15.97:0/384230350 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f6e98059700 con 0x7f6eb015e0d0 2024-03-06T07:32:14.839 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:14.839 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:11.891681+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e8c06b4d0 msgr2=0x7f6e8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e8c06b4d0 0x7f6e8c06d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0036000 tx=0x7f6ea0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 msgr2=0x7f6eb00cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00cbfd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d830 tx=0x7f6e9800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 shutdown_connections 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e8c06b4d0 0x7f6e8c06d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb015f470 0x7f6eb00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb015e0d0 0x7f6eb00cbfd0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 --2- 172.21.15.97:0/384230350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb00d0df0 0x7f6eb015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 >> 172.21.15.97:0/384230350 conn(0x7f6eb00d3ec0 msgr2=0x7f6eb00d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 shutdown_connections 2024-03-06T07:32:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:14.840+0000 7f6eb5400640 1 -- 172.21.15.97:0/384230350 wait complete. 2024-03-06T07:32:14.860 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:32:14.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:15.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.057+0000 7f48cbe00640 1 Processor -- start 2024-03-06T07:32:15.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.057+0000 7f48cbe00640 1 -- start start 2024-03-06T07:32:15.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.059+0000 7f48cbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f48cc15ad60 0x7f48cc0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.059+0000 7f48cbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 0x7f48cc15b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.059+0000 7f48cbe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48cc06bb70 con 0x7f48cc15b680 2024-03-06T07:32:15.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.059+0000 7f48cbe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48cc0b0ec0 con 0x7f48cc15ad60 2024-03-06T07:32:15.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48cbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0b0be0 con 0x7f48cc06c610 2024-03-06T07:32:15.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48cb400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f48cc15b680 0x7f48cc072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59480/0 (socket says 172.21.15.97:59480) 2024-03-06T07:32:15.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48cb400640 1 -- 172.21.15.97:0/2602785816 learned_addr learned my addr 172.21.15.97:0/2602785816 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48caa00640 1 --2- 172.21.15.97:0/2602785816 >> v2:172.21.15.135:3300/0 conn(0x7f48cc15ad60 0x7f48cc0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48ca000640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 0x7f48cc15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164656625 0 0) 0x7f48cc06bb70 con 0x7f48cc15b680 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.060+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b0003520 con 0x7f48cc15b680 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825650558 0 0) 0x7f48b0003520 con 0x7f48cc15b680 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 msgr2=0x7f48cc15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 0x7f48cc15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 >> v2:172.21.15.135:3300/0 conn(0x7f48cc15ad60 msgr2=0x7f48cc0b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 --2- 172.21.15.97:0/2602785816 >> v2:172.21.15.135:3300/0 conn(0x7f48cc15ad60 0x7f48cc0b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc06be10 con 0x7f48cc15b680 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f48c40019f0 con 0x7f48cc15b680 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 >> v1:172.21.15.97:6789/0 conn(0x7f48cc15b680 legacy=0x7f48cc072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 0x7f48b00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48b0003f30 0x7f48b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b0004d40 0x7f48b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0b0ec0 con 0x7f48b0004d40 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0b0be0 con 0x7f48b0003f30 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b000b9c0 con 0x7f48cc06c610 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48cb400640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b0004d40 0x7f48b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48caa00640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 0x7f48b00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.061+0000 7f48ca000640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48b0003f30 0x7f48b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.062+0000 7f48cb400640 1 -- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 msgr2=0x7f48b00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.062+0000 7f48cb400640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48cc06c610 0x7f48b00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.062+0000 7f48cb400640 1 -- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48b0003f30 msgr2=0x7f48b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.062+0000 7f48cb400640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48b0003f30 0x7f48b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.062+0000 7f48cb400640 1 -- 172.21.15.97:0/2602785816 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0003520 con 0x7f48b0004d40 2024-03-06T07:32:15.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.063+0000 7f48cb400640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b0004d40 0x7f48b00051c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f48c40013d0 tx=0x7f48b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefc357d444a8677 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.063+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c400a4e0 con 0x7f48b0004d40 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.063+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c4034650 con 0x7f48b0004d40 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.063+0000 7f48c9600640 1 -- 172.21.15.97:0/2602785816 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c4034a40 con 0x7f48b0004d40 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cbe00640 1 -- 172.21.15.97:0/2602785816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b0004d40 msgr2=0x7f48b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cbe00640 1 --2- 172.21.15.97:0/2602785816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b0004d40 0x7f48b00051c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f48c40013d0 tx=0x7f48b0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cb400640 1 -- 172.21.15.97:0/2602785816 reap_dead start 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cbe00640 1 -- 172.21.15.97:0/2602785816 shutdown_connections 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cbe00640 1 -- 172.21.15.97:0/2602785816 >> 172.21.15.97:0/2602785816 conn(0x7f48cc0b4fe0 msgr2=0x7f48cc0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cbe00640 1 -- 172.21.15.97:0/2602785816 shutdown_connections 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.064+0000 7f48cbe00640 1 -- 172.21.15.97:0/2602785816 wait complete. 2024-03-06T07:32:15.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.065+0000 7f48cbe00640 1 Processor -- start 2024-03-06T07:32:15.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.066+0000 7f48cbe00640 1 -- start start 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.066+0000 7f48cbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48b0004d40 0x7f48cc09db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48cbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 0x7f48cc0ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48cbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48cc09abf0 0x7f48cc14b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48cbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0b0800 con 0x7f48cc13d430 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48cbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc15bb30 con 0x7f48cc09abf0 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48cbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0b0980 con 0x7f48b0004d40 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 0x7f48cc0ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 0x7f48cc0ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38662/0 (socket says 172.21.15.97:38662) 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 -- 172.21.15.97:0/1520716652 learned_addr learned my addr 172.21.15.97:0/1520716652 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:15.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48caa00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48b0004d40 0x7f48cc09db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 -- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48b0004d40 msgr2=0x7f48cc09db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48b0004d40 0x7f48cc09db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48cb400640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48cc09abf0 0x7f48cc14b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 -- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48cc09abf0 msgr2=0x7f48cc14b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48cc09abf0 0x7f48cc14b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.067+0000 7f48ca000640 1 -- 172.21.15.97:0/1520716652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc0aefc0 con 0x7f48cc13d430 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.068+0000 7f48ca000640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 0x7f48cc0ae790 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f48b8011540 tx=0x7f48b8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.068+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8015930 con 0x7f48cc13d430 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.068+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8015ad0 con 0x7f48cc13d430 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.068+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b801d400 con 0x7f48cc13d430 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.068+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc0af2a0 con 0x7f48cc13d430 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.068+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc0aecd0 con 0x7f48cc13d430 2024-03-06T07:32:15.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.069+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc04f330 con 0x7f48cc13d430 2024-03-06T07:32:15.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.070+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48b801d5a0 con 0x7f48cc13d430 2024-03-06T07:32:15.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.071+0000 7f48c2a00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48ac06b500 0x7f48ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.071+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48b808a9f0 con 0x7f48cc13d430 2024-03-06T07:32:15.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.071+0000 7f48caa00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48ac06b500 0x7f48ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.072+0000 7f48caa00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48ac06b500 0x7f48ac06d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0099d0 tx=0x7f48bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.075+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48b8019030 con 0x7f48cc13d430 2024-03-06T07:32:15.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.272+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48cc0af470 con 0x7f48ac06b500 2024-03-06T07:32:15.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.283+0000 7f48c2a00640 1 -- 172.21.15.97:0/1520716652 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252079 (secure 0 0 0) 0x7f48cc0af470 con 0x7f48ac06b500 2024-03-06T07:32:15.288 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:15.290 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48ac06b500 msgr2=0x7f48ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48ac06b500 0x7f48ac06d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0099d0 tx=0x7f48bc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 msgr2=0x7f48cc0ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 0x7f48cc0ae790 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f48b8011540 tx=0x7f48b8011a00 comp rx=0 tx=0).stop 2024-03-06T07:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 shutdown_connections 2024-03-06T07:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48ac06b500 0x7f48ac06d9c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48cc09abf0 0x7f48cc14b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48cc13d430 0x7f48cc0ae790 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 --2- 172.21.15.97:0/1520716652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48b0004d40 0x7f48cc09db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.289+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 >> 172.21.15.97:0/1520716652 conn(0x7f48cc0b4fe0 msgr2=0x7f48cc151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.290+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 shutdown_connections 2024-03-06T07:32:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.290+0000 7f48cbe00640 1 -- 172.21.15.97:0/1520716652 wait complete. 2024-03-06T07:32:15.310 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-03-06T07:32:14.984823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258516,"kb_used_data":9064,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716204,"statfs":{"total":773094113280,"available":772829392896,"internally_reserved":0,"allocated":9281536,"data_stored":4588348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001466"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115641+0000","last_change":"2024-03-06T07:32:10.906307+0000","last_active":"2024-03-06T07:32:12.115641+0000","last_peered":"2024-03-06T07:32:12.115641+0000","last_clean":"2024-03-06T07:32:12.115641+0000","last_became_active":"2024-03-06T07:32:10.906031+0000","last_became_peered":"2024-03-06T07:32:10.906031+0000","last_unstale":"2024-03-06T07:32:12.115641+0000","last_undegraded":"2024-03-06T07:32:12.115641+0000","last_fullsized":"2024-03-06T07:32:12.115641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:17.489151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903664+0000","last_change":"2024-03-06T07:32:10.905159+0000","last_active":"2024-03-06T07:32:11.903664+0000","last_peered":"2024-03-06T07:32:11.903664+0000","last_clean":"2024-03-06T07:32:11.903664+0000","last_became_active":"2024-03-06T07:32:10.904798+0000","last_became_peered":"2024-03-06T07:32:10.904798+0000","last_unstale":"2024-03-06T07:32:11.903664+0000","last_undegraded":"2024-03-06T07:32:11.903664+0000","last_fullsized":"2024-03-06T07:32:11.903664+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:03.467161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114223+0000","last_change":"2024-03-06T07:32:10.911774+0000","last_active":"2024-03-06T07:32:12.114223+0000","last_peered":"2024-03-06T07:32:12.114223+0000","last_clean":"2024-03-06T07:32:12.114223+0000","last_became_active":"2024-03-06T07:32:10.911491+0000","last_became_peered":"2024-03-06T07:32:10.911491+0000","last_unstale":"2024-03-06T07:32:12.114223+0000","last_undegraded":"2024-03-06T07:32:12.114223+0000","last_fullsized":"2024-03-06T07:32:12.114223+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:55.939968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924655+0000","last_change":"2024-03-06T07:32:10.906713+0000","last_active":"2024-03-06T07:32:11.924655+0000","last_peered":"2024-03-06T07:32:11.924655+0000","last_clean":"2024-03-06T07:32:11.924655+0000","last_became_active":"2024-03-06T07:32:10.906407+0000","last_became_peered":"2024-03-06T07:32:10.906407+0000","last_unstale":"2024-03-06T07:32:11.924655+0000","last_undegraded":"2024-03-06T07:32:11.924655+0000","last_fullsized":"2024-03-06T07:32:11.924655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:04.482049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906884+0000","last_change":"2024-03-06T07:32:10.891436+0000","last_active":"2024-03-06T07:32:11.906884+0000","last_peered":"2024-03-06T07:32:11.906884+0000","last_clean":"2024-03-06T07:32:11.906884+0000","last_became_active":"2024-03-06T07:32:10.890765+0000","last_became_peered":"2024-03-06T07:32:10.890765+0000","last_unstale":"2024-03-06T07:32:11.906884+0000","last_undegraded":"2024-03-06T07:32:11.906884+0000","last_fullsized":"2024-03-06T07:32:11.906884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:00.065997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907415+0000","last_change":"2024-03-06T07:32:10.907119+0000","last_active":"2024-03-06T07:32:11.907415+0000","last_peered":"2024-03-06T07:32:11.907415+0000","last_clean":"2024-03-06T07:32:11.907415+0000","last_became_active":"2024-03-06T07:32:10.906793+0000","last_became_peered":"2024-03-06T07:32:10.906793+0000","last_unstale":"2024-03-06T07:32:11.907415+0000","last_undegraded":"2024-03-06T07:32:11.907415+0000","last_fullsized":"2024-03-06T07:32:11.907415+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:52.621173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924612+0000","last_change":"2024-03-06T07:32:10.907152+0000","last_active":"2024-03-06T07:32:11.924612+0000","last_peered":"2024-03-06T07:32:11.924612+0000","last_clean":"2024-03-06T07:32:11.924612+0000","last_became_active":"2024-03-06T07:32:10.906822+0000","last_became_peered":"2024-03-06T07:32:10.906822+0000","last_unstale":"2024-03-06T07:32:11.924612+0000","last_undegraded":"2024-03-06T07:32:11.924612+0000","last_fullsized":"2024-03-06T07:32:11.924612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:29.817303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906966+0000","last_change":"2024-03-06T07:32:10.905884+0000","last_active":"2024-03-06T07:32:11.906966+0000","last_peered":"2024-03-06T07:32:11.906966+0000","last_clean":"2024-03-06T07:32:11.906966+0000","last_became_active":"2024-03-06T07:32:10.903362+0000","last_became_peered":"2024-03-06T07:32:10.903362+0000","last_unstale":"2024-03-06T07:32:11.906966+0000","last_undegraded":"2024-03-06T07:32:11.906966+0000","last_fullsized":"2024-03-06T07:32:11.906966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:15.525677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908051+0000","last_change":"2024-03-06T07:32:10.902379+0000","last_active":"2024-03-06T07:32:11.908051+0000","last_peered":"2024-03-06T07:32:11.908051+0000","last_clean":"2024-03-06T07:32:11.908051+0000","last_became_active":"2024-03-06T07:32:10.901748+0000","last_became_peered":"2024-03-06T07:32:10.901748+0000","last_unstale":"2024-03-06T07:32:11.908051+0000","last_undegraded":"2024-03-06T07:32:11.908051+0000","last_fullsized":"2024-03-06T07:32:11.908051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:27.928075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116362+0000","last_change":"2024-03-06T07:32:10.903489+0000","last_active":"2024-03-06T07:32:12.116362+0000","last_peered":"2024-03-06T07:32:12.116362+0000","last_clean":"2024-03-06T07:32:12.116362+0000","last_became_active":"2024-03-06T07:32:10.903177+0000","last_became_peered":"2024-03-06T07:32:10.903177+0000","last_unstale":"2024-03-06T07:32:12.116362+0000","last_undegraded":"2024-03-06T07:32:12.116362+0000","last_fullsized":"2024-03-06T07:32:12.116362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:12.601104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904266+0000","last_change":"2024-03-06T07:32:10.905087+0000","last_active":"2024-03-06T07:32:11.904266+0000","last_peered":"2024-03-06T07:32:11.904266+0000","last_clean":"2024-03-06T07:32:11.904266+0000","last_became_active":"2024-03-06T07:32:10.904786+0000","last_became_peered":"2024-03-06T07:32:10.904786+0000","last_unstale":"2024-03-06T07:32:11.904266+0000","last_undegraded":"2024-03-06T07:32:11.904266+0000","last_fullsized":"2024-03-06T07:32:11.904266+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:40.611043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115527+0000","last_change":"2024-03-06T07:32:10.903851+0000","last_active":"2024-03-06T07:32:12.115527+0000","last_peered":"2024-03-06T07:32:12.115527+0000","last_clean":"2024-03-06T07:32:12.115527+0000","last_became_active":"2024-03-06T07:32:10.903574+0000","last_became_peered":"2024-03-06T07:32:10.903574+0000","last_unstale":"2024-03-06T07:32:12.115527+0000","last_undegraded":"2024-03-06T07:32:12.115527+0000","last_fullsized":"2024-03-06T07:32:12.115527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:27.928075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113916+0000","last_change":"2024-03-06T07:32:10.910717+0000","last_active":"2024-03-06T07:32:12.113916+0000","last_peered":"2024-03-06T07:32:12.113916+0000","last_clean":"2024-03-06T07:32:12.113916+0000","last_became_active":"2024-03-06T07:32:10.907720+0000","last_became_peered":"2024-03-06T07:32:10.907720+0000","last_unstale":"2024-03-06T07:32:12.113916+0000","last_undegraded":"2024-03-06T07:32:12.113916+0000","last_fullsized":"2024-03-06T07:32:12.113916+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:34.056184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924975+0000","last_change":"2024-03-06T07:32:10.910589+0000","last_active":"2024-03-06T07:32:11.924975+0000","last_peered":"2024-03-06T07:32:11.924975+0000","last_clean":"2024-03-06T07:32:11.924975+0000","last_became_active":"2024-03-06T07:32:10.910325+0000","last_became_peered":"2024-03-06T07:32:10.910325+0000","last_unstale":"2024-03-06T07:32:11.924975+0000","last_undegraded":"2024-03-06T07:32:11.924975+0000","last_fullsized":"2024-03-06T07:32:11.924975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:29.448692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.112832+0000","last_change":"2024-03-06T07:32:10.905760+0000","last_active":"2024-03-06T07:32:12.112832+0000","last_peered":"2024-03-06T07:32:12.112832+0000","last_clean":"2024-03-06T07:32:12.112832+0000","last_became_active":"2024-03-06T07:32:10.905493+0000","last_became_peered":"2024-03-06T07:32:10.905493+0000","last_unstale":"2024-03-06T07:32:12.112832+0000","last_undegraded":"2024-03-06T07:32:12.112832+0000","last_fullsized":"2024-03-06T07:32:12.112832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:03.458405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906531+0000","last_change":"2024-03-06T07:32:10.910181+0000","last_active":"2024-03-06T07:32:11.906531+0000","last_peered":"2024-03-06T07:32:11.906531+0000","last_clean":"2024-03-06T07:32:11.906531+0000","last_became_active":"2024-03-06T07:32:10.909881+0000","last_became_peered":"2024-03-06T07:32:10.909881+0000","last_unstale":"2024-03-06T07:32:11.906531+0000","last_undegraded":"2024-03-06T07:32:11.906531+0000","last_fullsized":"2024-03-06T07:32:11.906531+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:09.470182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908755+0000","last_change":"2024-03-06T07:32:10.897916+0000","last_active":"2024-03-06T07:32:11.908755+0000","last_peered":"2024-03-06T07:32:11.908755+0000","last_clean":"2024-03-06T07:32:11.908755+0000","last_became_active":"2024-03-06T07:32:10.896112+0000","last_became_peered":"2024-03-06T07:32:10.896112+0000","last_unstale":"2024-03-06T07:32:11.908755+0000","last_undegraded":"2024-03-06T07:32:11.908755+0000","last_fullsized":"2024-03-06T07:32:11.908755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:28.186902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.925307+0000","last_change":"2024-03-06T07:32:10.902098+0000","last_active":"2024-03-06T07:32:11.925307+0000","last_peered":"2024-03-06T07:32:11.925307+0000","last_clean":"2024-03-06T07:32:11.925307+0000","last_became_active":"2024-03-06T07:32:10.901826+0000","last_became_peered":"2024-03-06T07:32:10.901826+0000","last_unstale":"2024-03-06T07:32:11.925307+0000","last_undegraded":"2024-03-06T07:32:11.925307+0000","last_fullsized":"2024-03-06T07:32:11.925307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:07.037956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.914056+0000","last_change":"2024-03-06T07:32:10.913902+0000","last_active":"2024-03-06T07:32:11.914056+0000","last_peered":"2024-03-06T07:32:11.914056+0000","last_clean":"2024-03-06T07:32:11.914056+0000","last_became_active":"2024-03-06T07:32:10.913557+0000","last_became_peered":"2024-03-06T07:32:10.913557+0000","last_unstale":"2024-03-06T07:32:11.914056+0000","last_undegraded":"2024-03-06T07:32:11.914056+0000","last_fullsized":"2024-03-06T07:32:11.914056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:27.928075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924897+0000","last_change":"2024-03-06T07:32:10.912261+0000","last_active":"2024-03-06T07:32:11.924897+0000","last_peered":"2024-03-06T07:32:11.924897+0000","last_clean":"2024-03-06T07:32:11.924897+0000","last_became_active":"2024-03-06T07:32:10.911969+0000","last_became_peered":"2024-03-06T07:32:10.911969+0000","last_unstale":"2024-03-06T07:32:11.924897+0000","last_undegraded":"2024-03-06T07:32:11.924897+0000","last_fullsized":"2024-03-06T07:32:11.924897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:58.800571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.925039+0000","last_change":"2024-03-06T07:32:10.905838+0000","last_active":"2024-03-06T07:32:11.925039+0000","last_peered":"2024-03-06T07:32:11.925039+0000","last_clean":"2024-03-06T07:32:11.925039+0000","last_became_active":"2024-03-06T07:32:10.905524+0000","last_became_peered":"2024-03-06T07:32:10.905524+0000","last_unstale":"2024-03-06T07:32:11.925039+0000","last_undegraded":"2024-03-06T07:32:11.925039+0000","last_fullsized":"2024-03-06T07:32:11.925039+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:04.979450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904021+0000","last_change":"2024-03-06T07:32:10.903437+0000","last_active":"2024-03-06T07:32:11.904021+0000","last_peered":"2024-03-06T07:32:11.904021+0000","last_clean":"2024-03-06T07:32:11.904021+0000","last_became_active":"2024-03-06T07:32:10.902775+0000","last_became_peered":"2024-03-06T07:32:10.902775+0000","last_unstale":"2024-03-06T07:32:11.904021+0000","last_undegraded":"2024-03-06T07:32:11.904021+0000","last_fullsized":"2024-03-06T07:32:11.904021+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:00.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115149+0000","last_change":"2024-03-06T07:32:10.906635+0000","last_active":"2024-03-06T07:32:12.115149+0000","last_peered":"2024-03-06T07:32:12.115149+0000","last_clean":"2024-03-06T07:32:12.115149+0000","last_became_active":"2024-03-06T07:32:10.904722+0000","last_became_peered":"2024-03-06T07:32:10.904722+0000","last_unstale":"2024-03-06T07:32:12.115149+0000","last_undegraded":"2024-03-06T07:32:12.115149+0000","last_fullsized":"2024-03-06T07:32:12.115149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:23.206762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924450+0000","last_change":"2024-03-06T07:32:10.911560+0000","last_active":"2024-03-06T07:32:11.924450+0000","last_peered":"2024-03-06T07:32:11.924450+0000","last_clean":"2024-03-06T07:32:11.924450+0000","last_became_active":"2024-03-06T07:32:10.910427+0000","last_became_peered":"2024-03-06T07:32:10.910427+0000","last_unstale":"2024-03-06T07:32:11.924450+0000","last_undegraded":"2024-03-06T07:32:11.924450+0000","last_fullsized":"2024-03-06T07:32:11.924450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:17.154279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903853+0000","last_change":"2024-03-06T07:32:10.903598+0000","last_active":"2024-03-06T07:32:11.903853+0000","last_peered":"2024-03-06T07:32:11.903853+0000","last_clean":"2024-03-06T07:32:11.903853+0000","last_became_active":"2024-03-06T07:32:10.902913+0000","last_became_peered":"2024-03-06T07:32:10.902913+0000","last_unstale":"2024-03-06T07:32:11.903853+0000","last_undegraded":"2024-03-06T07:32:11.903853+0000","last_fullsized":"2024-03-06T07:32:11.903853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:53.829597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115851+0000","last_change":"2024-03-06T07:32:10.917664+0000","last_active":"2024-03-06T07:32:12.115851+0000","last_peered":"2024-03-06T07:32:12.115851+0000","last_clean":"2024-03-06T07:32:12.115851+0000","last_became_active":"2024-03-06T07:32:10.916978+0000","last_became_peered":"2024-03-06T07:32:10.916978+0000","last_unstale":"2024-03-06T07:32:12.115851+0000","last_undegraded":"2024-03-06T07:32:12.115851+0000","last_fullsized":"2024-03-06T07:32:12.115851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:01.928083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908438+0000","last_change":"2024-03-06T07:32:10.896284+0000","last_active":"2024-03-06T07:32:11.908438+0000","last_peered":"2024-03-06T07:32:11.908438+0000","last_clean":"2024-03-06T07:32:11.908438+0000","last_became_active":"2024-03-06T07:32:10.895793+0000","last_became_peered":"2024-03-06T07:32:10.895793+0000","last_unstale":"2024-03-06T07:32:11.908438+0000","last_undegraded":"2024-03-06T07:32:11.908438+0000","last_fullsized":"2024-03-06T07:32:11.908438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:23.720230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115909+0000","last_change":"2024-03-06T07:32:10.907758+0000","last_active":"2024-03-06T07:32:12.115909+0000","last_peered":"2024-03-06T07:32:12.115909+0000","last_clean":"2024-03-06T07:32:12.115909+0000","last_became_active":"2024-03-06T07:32:10.907406+0000","last_became_peered":"2024-03-06T07:32:10.907406+0000","last_unstale":"2024-03-06T07:32:12.115909+0000","last_undegraded":"2024-03-06T07:32:12.115909+0000","last_fullsized":"2024-03-06T07:32:12.115909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:21.535340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903529+0000","last_change":"2024-03-06T07:32:10.902214+0000","last_active":"2024-03-06T07:32:11.903529+0000","last_peered":"2024-03-06T07:32:11.903529+0000","last_clean":"2024-03-06T07:32:11.903529+0000","last_became_active":"2024-03-06T07:32:10.901872+0000","last_became_peered":"2024-03-06T07:32:10.901872+0000","last_unstale":"2024-03-06T07:32:11.903529+0000","last_undegraded":"2024-03-06T07:32:11.903529+0000","last_fullsized":"2024-03-06T07:32:11.903529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:48.555144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904097+0000","last_change":"2024-03-06T07:32:10.899766+0000","last_active":"2024-03-06T07:32:11.904097+0000","last_peered":"2024-03-06T07:32:11.904097+0000","last_clean":"2024-03-06T07:32:11.904097+0000","last_became_active":"2024-03-06T07:32:10.895450+0000","last_became_peered":"2024-03-06T07:32:10.895450+0000","last_unstale":"2024-03-06T07:32:11.904097+0000","last_undegraded":"2024-03-06T07:32:11.904097+0000","last_fullsized":"2024-03-06T07:32:11.904097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:17.110943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906944+0000","last_change":"2024-03-06T07:32:10.902364+0000","last_active":"2024-03-06T07:32:11.906944+0000","last_peered":"2024-03-06T07:32:11.906944+0000","last_clean":"2024-03-06T07:32:11.906944+0000","last_became_active":"2024-03-06T07:32:10.902045+0000","last_became_peered":"2024-03-06T07:32:10.902045+0000","last_unstale":"2024-03-06T07:32:11.906944+0000","last_undegraded":"2024-03-06T07:32:11.906944+0000","last_fullsized":"2024-03-06T07:32:11.906944+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:55.564188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114905+0000","last_change":"2024-03-06T07:32:10.907010+0000","last_active":"2024-03-06T07:32:12.114905+0000","last_peered":"2024-03-06T07:32:12.114905+0000","last_clean":"2024-03-06T07:32:12.114905+0000","last_became_active":"2024-03-06T07:32:10.906688+0000","last_became_peered":"2024-03-06T07:32:10.906688+0000","last_unstale":"2024-03-06T07:32:12.114905+0000","last_undegraded":"2024-03-06T07:32:12.114905+0000","last_fullsized":"2024-03-06T07:32:12.114905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:41.770080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.909015+0000","last_change":"2024-03-06T07:32:10.903032+0000","last_active":"2024-03-06T07:32:11.909015+0000","last_peered":"2024-03-06T07:32:11.909015+0000","last_clean":"2024-03-06T07:32:11.909015+0000","last_became_active":"2024-03-06T07:32:10.901998+0000","last_became_peered":"2024-03-06T07:32:10.901998+0000","last_unstale":"2024-03-06T07:32:11.909015+0000","last_undegraded":"2024-03-06T07:32:11.909015+0000","last_fullsized":"2024-03-06T07:32:11.909015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:25.997977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116309+0000","last_change":"2024-03-06T07:32:09.915655+0000","last_active":"2024-03-06T07:32:12.116309+0000","last_peered":"2024-03-06T07:32:12.116309+0000","last_clean":"2024-03-06T07:32:12.116309+0000","last_became_active":"2024-03-06T07:32:09.914539+0000","last_became_peered":"2024-03-06T07:32:09.914539+0000","last_unstale":"2024-03-06T07:32:12.116309+0000","last_undegraded":"2024-03-06T07:32:12.116309+0000","last_fullsized":"2024-03-06T07:32:12.116309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:08.948200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116065+0000","last_change":"2024-03-06T07:32:10.905398+0000","last_active":"2024-03-06T07:32:12.116065+0000","last_peered":"2024-03-06T07:32:12.116065+0000","last_clean":"2024-03-06T07:32:12.116065+0000","last_became_active":"2024-03-06T07:32:10.904540+0000","last_became_peered":"2024-03-06T07:32:10.904540+0000","last_unstale":"2024-03-06T07:32:12.116065+0000","last_undegraded":"2024-03-06T07:32:12.116065+0000","last_fullsized":"2024-03-06T07:32:12.116065+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:01.458813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.909156+0000","last_change":"2024-03-06T07:32:09.909605+0000","last_active":"2024-03-06T07:32:11.909156+0000","last_peered":"2024-03-06T07:32:11.909156+0000","last_clean":"2024-03-06T07:32:11.909156+0000","last_became_active":"2024-03-06T07:32:09.909320+0000","last_became_peered":"2024-03-06T07:32:09.909320+0000","last_unstale":"2024-03-06T07:32:11.909156+0000","last_undegraded":"2024-03-06T07:32:11.909156+0000","last_fullsized":"2024-03-06T07:32:11.909156+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:22.101485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116367+0000","last_change":"2024-03-06T07:32:10.903612+0000","last_active":"2024-03-06T07:32:12.116367+0000","last_peered":"2024-03-06T07:32:12.116367+0000","last_clean":"2024-03-06T07:32:12.116367+0000","last_became_active":"2024-03-06T07:32:10.903164+0000","last_became_peered":"2024-03-06T07:32:10.903164+0000","last_unstale":"2024-03-06T07:32:12.116367+0000","last_undegraded":"2024-03-06T07:32:12.116367+0000","last_fullsized":"2024-03-06T07:32:12.116367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:45.822921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.914092+0000","last_change":"2024-03-06T07:32:09.907413+0000","last_active":"2024-03-06T07:32:11.914092+0000","last_peered":"2024-03-06T07:32:11.914092+0000","last_clean":"2024-03-06T07:32:11.914092+0000","last_became_active":"2024-03-06T07:32:09.906908+0000","last_became_peered":"2024-03-06T07:32:09.906908+0000","last_unstale":"2024-03-06T07:32:11.914092+0000","last_undegraded":"2024-03-06T07:32:11.914092+0000","last_fullsized":"2024-03-06T07:32:11.914092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:12.783824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115437+0000","last_change":"2024-03-06T07:32:10.916704+0000","last_active":"2024-03-06T07:32:12.115437+0000","last_peered":"2024-03-06T07:32:12.115437+0000","last_clean":"2024-03-06T07:32:12.115437+0000","last_became_active":"2024-03-06T07:32:10.916322+0000","last_became_peered":"2024-03-06T07:32:10.916322+0000","last_unstale":"2024-03-06T07:32:12.115437+0000","last_undegraded":"2024-03-06T07:32:12.115437+0000","last_fullsized":"2024-03-06T07:32:12.115437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:56.042132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913767+0000","last_change":"2024-03-06T07:32:09.900976+0000","last_active":"2024-03-06T07:32:11.913767+0000","last_peered":"2024-03-06T07:32:11.913767+0000","last_clean":"2024-03-06T07:32:11.913767+0000","last_became_active":"2024-03-06T07:32:09.900120+0000","last_became_peered":"2024-03-06T07:32:09.900120+0000","last_unstale":"2024-03-06T07:32:11.913767+0000","last_undegraded":"2024-03-06T07:32:11.913767+0000","last_fullsized":"2024-03-06T07:32:11.913767+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:54.923021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904233+0000","last_change":"2024-03-06T07:32:10.903248+0000","last_active":"2024-03-06T07:32:11.904233+0000","last_peered":"2024-03-06T07:32:11.904233+0000","last_clean":"2024-03-06T07:32:11.904233+0000","last_became_active":"2024-03-06T07:32:10.902339+0000","last_became_peered":"2024-03-06T07:32:10.902339+0000","last_unstale":"2024-03-06T07:32:11.904233+0000","last_undegraded":"2024-03-06T07:32:11.904233+0000","last_fullsized":"2024-03-06T07:32:11.904233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:58.093199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908466+0000","last_change":"2024-03-06T07:32:09.905122+0000","last_active":"2024-03-06T07:32:11.908466+0000","last_peered":"2024-03-06T07:32:11.908466+0000","last_clean":"2024-03-06T07:32:11.908466+0000","last_became_active":"2024-03-06T07:32:09.902443+0000","last_became_peered":"2024-03-06T07:32:09.902443+0000","last_unstale":"2024-03-06T07:32:11.908466+0000","last_undegraded":"2024-03-06T07:32:11.908466+0000","last_fullsized":"2024-03-06T07:32:11.908466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:48.354010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907747+0000","last_change":"2024-03-06T07:32:10.898964+0000","last_active":"2024-03-06T07:32:11.907747+0000","last_peered":"2024-03-06T07:32:11.907747+0000","last_clean":"2024-03-06T07:32:11.907747+0000","last_became_active":"2024-03-06T07:32:10.897563+0000","last_became_peered":"2024-03-06T07:32:10.897563+0000","last_unstale":"2024-03-06T07:32:11.907747+0000","last_undegraded":"2024-03-06T07:32:11.907747+0000","last_fullsized":"2024-03-06T07:32:11.907747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:49.357999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906923+0000","last_change":"2024-03-06T07:32:09.910143+0000","last_active":"2024-03-06T07:32:11.906923+0000","last_peered":"2024-03-06T07:32:11.906923+0000","last_clean":"2024-03-06T07:32:11.906923+0000","last_became_active":"2024-03-06T07:32:09.909790+0000","last_became_peered":"2024-03-06T07:32:09.909790+0000","last_unstale":"2024-03-06T07:32:11.906923+0000","last_undegraded":"2024-03-06T07:32:11.906923+0000","last_fullsized":"2024-03-06T07:32:11.906923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:48.919689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113831+0000","last_change":"2024-03-06T07:32:10.921486+0000","last_active":"2024-03-06T07:32:12.113831+0000","last_peered":"2024-03-06T07:32:12.113831+0000","last_clean":"2024-03-06T07:32:12.113831+0000","last_became_active":"2024-03-06T07:32:10.921156+0000","last_became_peered":"2024-03-06T07:32:10.921156+0000","last_unstale":"2024-03-06T07:32:12.113831+0000","last_undegraded":"2024-03-06T07:32:12.113831+0000","last_fullsized":"2024-03-06T07:32:12.113831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:08.709841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913543+0000","last_change":"2024-03-06T07:32:09.910707+0000","last_active":"2024-03-06T07:32:11.913543+0000","last_peered":"2024-03-06T07:32:11.913543+0000","last_clean":"2024-03-06T07:32:11.913543+0000","last_became_active":"2024-03-06T07:32:09.910401+0000","last_became_peered":"2024-03-06T07:32:09.910401+0000","last_unstale":"2024-03-06T07:32:11.913543+0000","last_undegraded":"2024-03-06T07:32:11.913543+0000","last_fullsized":"2024-03-06T07:32:11.913543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:45.531229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.112912+0000","last_change":"2024-03-06T07:32:10.903420+0000","last_active":"2024-03-06T07:32:12.112912+0000","last_peered":"2024-03-06T07:32:12.112912+0000","last_clean":"2024-03-06T07:32:12.112912+0000","last_became_active":"2024-03-06T07:32:10.903095+0000","last_became_peered":"2024-03-06T07:32:10.903095+0000","last_unstale":"2024-03-06T07:32:12.112912+0000","last_undegraded":"2024-03-06T07:32:12.112912+0000","last_fullsized":"2024-03-06T07:32:12.112912+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:24.419915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904194+0000","last_change":"2024-03-06T07:32:09.911588+0000","last_active":"2024-03-06T07:32:11.904194+0000","last_peered":"2024-03-06T07:32:11.904194+0000","last_clean":"2024-03-06T07:32:11.904194+0000","last_became_active":"2024-03-06T07:32:09.910267+0000","last_became_peered":"2024-03-06T07:32:09.910267+0000","last_unstale":"2024-03-06T07:32:11.904194+0000","last_undegraded":"2024-03-06T07:32:11.904194+0000","last_fullsized":"2024-03-06T07:32:11.904194+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:23.214455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113276+0000","last_change":"2024-03-06T07:32:10.910717+0000","last_active":"2024-03-06T07:32:12.113276+0000","last_peered":"2024-03-06T07:32:12.113276+0000","last_clean":"2024-03-06T07:32:12.113276+0000","last_became_active":"2024-03-06T07:32:10.910383+0000","last_became_peered":"2024-03-06T07:32:10.910383+0000","last_unstale":"2024-03-06T07:32:12.113276+0000","last_undegraded":"2024-03-06T07:32:12.113276+0000","last_fullsized":"2024-03-06T07:32:12.113276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:04.998290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116502+0000","last_change":"2024-03-06T07:32:09.915082+0000","last_active":"2024-03-06T07:32:12.116502+0000","last_peered":"2024-03-06T07:32:12.116502+0000","last_clean":"2024-03-06T07:32:12.116502+0000","last_became_active":"2024-03-06T07:32:09.914223+0000","last_became_peered":"2024-03-06T07:32:09.914223+0000","last_unstale":"2024-03-06T07:32:12.116502+0000","last_undegraded":"2024-03-06T07:32:12.116502+0000","last_fullsized":"2024-03-06T07:32:12.116502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:31.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908205+0000","last_change":"2024-03-06T07:32:10.909941+0000","last_active":"2024-03-06T07:32:11.908205+0000","last_peered":"2024-03-06T07:32:11.908205+0000","last_clean":"2024-03-06T07:32:11.908205+0000","last_became_active":"2024-03-06T07:32:10.909609+0000","last_became_peered":"2024-03-06T07:32:10.909609+0000","last_unstale":"2024-03-06T07:32:11.908205+0000","last_undegraded":"2024-03-06T07:32:11.908205+0000","last_fullsized":"2024-03-06T07:32:11.908205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:17.489151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116303+0000","last_change":"2024-03-06T07:32:09.917195+0000","last_active":"2024-03-06T07:32:12.116303+0000","last_peered":"2024-03-06T07:32:12.116303+0000","last_clean":"2024-03-06T07:32:12.116303+0000","last_became_active":"2024-03-06T07:32:09.916892+0000","last_became_peered":"2024-03-06T07:32:09.916892+0000","last_unstale":"2024-03-06T07:32:12.116303+0000","last_undegraded":"2024-03-06T07:32:12.116303+0000","last_fullsized":"2024-03-06T07:32:12.116303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:50.033779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913894+0000","last_change":"2024-03-06T07:32:10.917530+0000","last_active":"2024-03-06T07:32:11.913894+0000","last_peered":"2024-03-06T07:32:11.913894+0000","last_clean":"2024-03-06T07:32:11.913894+0000","last_became_active":"2024-03-06T07:32:10.913808+0000","last_became_peered":"2024-03-06T07:32:10.913808+0000","last_unstale":"2024-03-06T07:32:11.913894+0000","last_undegraded":"2024-03-06T07:32:11.913894+0000","last_fullsized":"2024-03-06T07:32:11.913894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:41.770080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115074+0000","last_change":"2024-03-06T07:32:09.914071+0000","last_active":"2024-03-06T07:32:12.115074+0000","last_peered":"2024-03-06T07:32:12.115074+0000","last_clean":"2024-03-06T07:32:12.115074+0000","last_became_active":"2024-03-06T07:32:09.913680+0000","last_became_peered":"2024-03-06T07:32:09.913680+0000","last_unstale":"2024-03-06T07:32:12.115074+0000","last_undegraded":"2024-03-06T07:32:12.115074+0000","last_fullsized":"2024-03-06T07:32:12.115074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:39.597163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908735+0000","last_change":"2024-03-06T07:32:10.911895+0000","last_active":"2024-03-06T07:32:11.908735+0000","last_peered":"2024-03-06T07:32:11.908735+0000","last_clean":"2024-03-06T07:32:11.908735+0000","last_became_active":"2024-03-06T07:32:10.909077+0000","last_became_peered":"2024-03-06T07:32:10.909077+0000","last_unstale":"2024-03-06T07:32:11.908735+0000","last_undegraded":"2024-03-06T07:32:11.908735+0000","last_fullsized":"2024-03-06T07:32:11.908735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:59.206364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907657+0000","last_change":"2024-03-06T07:32:09.912589+0000","last_active":"2024-03-06T07:32:11.907657+0000","last_peered":"2024-03-06T07:32:11.907657+0000","last_clean":"2024-03-06T07:32:11.907657+0000","last_became_active":"2024-03-06T07:32:09.910495+0000","last_became_peered":"2024-03-06T07:32:09.910495+0000","last_unstale":"2024-03-06T07:32:11.907657+0000","last_undegraded":"2024-03-06T07:32:11.907657+0000","last_fullsized":"2024-03-06T07:32:11.907657+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:20.461391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913694+0000","last_change":"2024-03-06T07:32:10.912663+0000","last_active":"2024-03-06T07:32:11.913694+0000","last_peered":"2024-03-06T07:32:11.913694+0000","last_clean":"2024-03-06T07:32:11.913694+0000","last_became_active":"2024-03-06T07:32:10.912374+0000","last_became_peered":"2024-03-06T07:32:10.912374+0000","last_unstale":"2024-03-06T07:32:11.913694+0000","last_undegraded":"2024-03-06T07:32:11.913694+0000","last_fullsized":"2024-03-06T07:32:11.913694+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:52.621173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903588+0000","last_change":"2024-03-06T07:32:09.894798+0000","last_active":"2024-03-06T07:32:11.903588+0000","last_peered":"2024-03-06T07:32:11.903588+0000","last_clean":"2024-03-06T07:32:11.903588+0000","last_became_active":"2024-03-06T07:32:09.891974+0000","last_became_peered":"2024-03-06T07:32:09.891974+0000","last_unstale":"2024-03-06T07:32:11.903588+0000","last_undegraded":"2024-03-06T07:32:11.903588+0000","last_fullsized":"2024-03-06T07:32:11.903588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:16.140399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114303+0000","last_change":"2024-03-06T07:32:10.921487+0000","last_active":"2024-03-06T07:32:12.114303+0000","last_peered":"2024-03-06T07:32:12.114303+0000","last_clean":"2024-03-06T07:32:12.114303+0000","last_became_active":"2024-03-06T07:32:10.921132+0000","last_became_peered":"2024-03-06T07:32:10.921132+0000","last_unstale":"2024-03-06T07:32:12.114303+0000","last_undegraded":"2024-03-06T07:32:12.114303+0000","last_fullsized":"2024-03-06T07:32:12.114303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:39.087262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113007+0000","last_change":"2024-03-06T07:32:09.897136+0000","last_active":"2024-03-06T07:32:12.113007+0000","last_peered":"2024-03-06T07:32:12.113007+0000","last_clean":"2024-03-06T07:32:12.113007+0000","last_became_active":"2024-03-06T07:32:09.896787+0000","last_became_peered":"2024-03-06T07:32:09.896787+0000","last_unstale":"2024-03-06T07:32:12.113007+0000","last_undegraded":"2024-03-06T07:32:12.113007+0000","last_fullsized":"2024-03-06T07:32:12.113007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.526095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904002+0000","last_change":"2024-03-06T07:32:10.897713+0000","last_active":"2024-03-06T07:32:11.904002+0000","last_peered":"2024-03-06T07:32:11.904002+0000","last_clean":"2024-03-06T07:32:11.904002+0000","last_became_active":"2024-03-06T07:32:10.897434+0000","last_became_peered":"2024-03-06T07:32:10.897434+0000","last_unstale":"2024-03-06T07:32:11.904002+0000","last_undegraded":"2024-03-06T07:32:11.904002+0000","last_fullsized":"2024-03-06T07:32:11.904002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:53.780886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907111+0000","last_change":"2024-03-06T07:32:09.913543+0000","last_active":"2024-03-06T07:32:11.907111+0000","last_peered":"2024-03-06T07:32:11.907111+0000","last_clean":"2024-03-06T07:32:11.907111+0000","last_became_active":"2024-03-06T07:32:09.913245+0000","last_became_peered":"2024-03-06T07:32:09.913245+0000","last_unstale":"2024-03-06T07:32:11.907111+0000","last_undegraded":"2024-03-06T07:32:11.907111+0000","last_fullsized":"2024-03-06T07:32:11.907111+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:00.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903321+0000","last_change":"2024-03-06T07:32:10.908986+0000","last_active":"2024-03-06T07:32:11.903321+0000","last_peered":"2024-03-06T07:32:11.903321+0000","last_clean":"2024-03-06T07:32:11.903321+0000","last_became_active":"2024-03-06T07:32:10.908630+0000","last_became_peered":"2024-03-06T07:32:10.908630+0000","last_unstale":"2024-03-06T07:32:11.903321+0000","last_undegraded":"2024-03-06T07:32:11.903321+0000","last_fullsized":"2024-03-06T07:32:11.903321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:45.579006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913431+0000","last_change":"2024-03-06T07:32:09.908341+0000","last_active":"2024-03-06T07:32:11.913431+0000","last_peered":"2024-03-06T07:32:11.913431+0000","last_clean":"2024-03-06T07:32:11.913431+0000","last_became_active":"2024-03-06T07:32:09.907997+0000","last_became_peered":"2024-03-06T07:32:09.907997+0000","last_unstale":"2024-03-06T07:32:11.913431+0000","last_undegraded":"2024-03-06T07:32:11.913431+0000","last_fullsized":"2024-03-06T07:32:11.913431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:20.461391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116239+0000","last_change":"2024-03-06T07:32:10.909405+0000","last_active":"2024-03-06T07:32:12.116239+0000","last_peered":"2024-03-06T07:32:12.116239+0000","last_clean":"2024-03-06T07:32:12.116239+0000","last_became_active":"2024-03-06T07:32:10.908470+0000","last_became_peered":"2024-03-06T07:32:10.908470+0000","last_unstale":"2024-03-06T07:32:12.116239+0000","last_undegraded":"2024-03-06T07:32:12.116239+0000","last_fullsized":"2024-03-06T07:32:12.116239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.814981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116746+0000","last_change":"2024-03-06T07:32:09.913931+0000","last_active":"2024-03-06T07:32:12.116746+0000","last_peered":"2024-03-06T07:32:12.116746+0000","last_clean":"2024-03-06T07:32:12.116746+0000","last_became_active":"2024-03-06T07:32:09.911819+0000","last_became_peered":"2024-03-06T07:32:09.911819+0000","last_unstale":"2024-03-06T07:32:12.116746+0000","last_undegraded":"2024-03-06T07:32:12.116746+0000","last_fullsized":"2024-03-06T07:32:12.116746+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:51.121062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116368+0000","last_change":"2024-03-06T07:32:10.905719+0000","last_active":"2024-03-06T07:32:12.116368+0000","last_peered":"2024-03-06T07:32:12.116368+0000","last_clean":"2024-03-06T07:32:12.116368+0000","last_became_active":"2024-03-06T07:32:10.905133+0000","last_became_peered":"2024-03-06T07:32:10.905133+0000","last_unstale":"2024-03-06T07:32:12.116368+0000","last_undegraded":"2024-03-06T07:32:12.116368+0000","last_fullsized":"2024-03-06T07:32:12.116368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:08.709841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.914136+0000","last_change":"2024-03-06T07:32:09.912732+0000","last_active":"2024-03-06T07:32:11.914136+0000","last_peered":"2024-03-06T07:32:11.914136+0000","last_clean":"2024-03-06T07:32:11.914136+0000","last_became_active":"2024-03-06T07:32:09.911917+0000","last_became_peered":"2024-03-06T07:32:09.911917+0000","last_unstale":"2024-03-06T07:32:11.914136+0000","last_undegraded":"2024-03-06T07:32:11.914136+0000","last_fullsized":"2024-03-06T07:32:11.914136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:00.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908934+0000","last_change":"2024-03-06T07:32:10.894290+0000","last_active":"2024-03-06T07:32:11.908934+0000","last_peered":"2024-03-06T07:32:11.908934+0000","last_clean":"2024-03-06T07:32:11.908934+0000","last_became_active":"2024-03-06T07:32:10.893827+0000","last_became_peered":"2024-03-06T07:32:10.893827+0000","last_unstale":"2024-03-06T07:32:11.908934+0000","last_undegraded":"2024-03-06T07:32:11.908934+0000","last_fullsized":"2024-03-06T07:32:11.908934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:00.949687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913933+0000","last_change":"2024-03-06T07:32:09.909723+0000","last_active":"2024-03-06T07:32:11.913933+0000","last_peered":"2024-03-06T07:32:11.913933+0000","last_clean":"2024-03-06T07:32:11.913933+0000","last_became_active":"2024-03-06T07:32:09.909407+0000","last_became_peered":"2024-03-06T07:32:09.909407+0000","last_unstale":"2024-03-06T07:32:11.913933+0000","last_undegraded":"2024-03-06T07:32:11.913933+0000","last_fullsized":"2024-03-06T07:32:11.913933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:05.488616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924750+0000","last_change":"2024-03-06T07:32:10.910025+0000","last_active":"2024-03-06T07:32:11.924750+0000","last_peered":"2024-03-06T07:32:11.924750+0000","last_clean":"2024-03-06T07:32:11.924750+0000","last_became_active":"2024-03-06T07:32:10.907190+0000","last_became_peered":"2024-03-06T07:32:10.907190+0000","last_unstale":"2024-03-06T07:32:11.924750+0000","last_undegraded":"2024-03-06T07:32:11.924750+0000","last_fullsized":"2024-03-06T07:32:11.924750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:48.909907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113024+0000","last_change":"2024-03-06T07:32:09.901839+0000","last_active":"2024-03-06T07:32:12.113024+0000","last_peered":"2024-03-06T07:32:12.113024+0000","last_clean":"2024-03-06T07:32:12.113024+0000","last_became_active":"2024-03-06T07:32:09.900748+0000","last_became_peered":"2024-03-06T07:32:09.900748+0000","last_unstale":"2024-03-06T07:32:12.113024+0000","last_undegraded":"2024-03-06T07:32:12.113024+0000","last_fullsized":"2024-03-06T07:32:12.113024+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:00.914204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907789+0000","last_change":"2024-03-06T07:32:10.900905+0000","last_active":"2024-03-06T07:32:11.907789+0000","last_peered":"2024-03-06T07:32:11.907789+0000","last_clean":"2024-03-06T07:32:11.907789+0000","last_became_active":"2024-03-06T07:32:10.900599+0000","last_became_peered":"2024-03-06T07:32:10.900599+0000","last_unstale":"2024-03-06T07:32:11.907789+0000","last_undegraded":"2024-03-06T07:32:11.907789+0000","last_fullsized":"2024-03-06T07:32:11.907789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:23.206762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924800+0000","last_change":"2024-03-06T07:32:09.903728+0000","last_active":"2024-03-06T07:32:11.924800+0000","last_peered":"2024-03-06T07:32:11.924800+0000","last_clean":"2024-03-06T07:32:11.924800+0000","last_became_active":"2024-03-06T07:32:09.903471+0000","last_became_peered":"2024-03-06T07:32:09.903471+0000","last_unstale":"2024-03-06T07:32:11.924800+0000","last_undegraded":"2024-03-06T07:32:11.924800+0000","last_fullsized":"2024-03-06T07:32:11.924800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:13.158368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924444+0000","last_change":"2024-03-06T07:32:10.900737+0000","last_active":"2024-03-06T07:32:11.924444+0000","last_peered":"2024-03-06T07:32:11.924444+0000","last_clean":"2024-03-06T07:32:11.924444+0000","last_became_active":"2024-03-06T07:32:10.900253+0000","last_became_peered":"2024-03-06T07:32:10.900253+0000","last_unstale":"2024-03-06T07:32:11.924444+0000","last_undegraded":"2024-03-06T07:32:11.924444+0000","last_fullsized":"2024-03-06T07:32:11.924444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.699432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924271+0000","last_change":"2024-03-06T07:32:09.912789+0000","last_active":"2024-03-06T07:32:11.924271+0000","last_peered":"2024-03-06T07:32:11.924271+0000","last_clean":"2024-03-06T07:32:11.924271+0000","last_became_active":"2024-03-06T07:32:09.912514+0000","last_became_peered":"2024-03-06T07:32:09.912514+0000","last_unstale":"2024-03-06T07:32:11.924271+0000","last_undegraded":"2024-03-06T07:32:11.924271+0000","last_fullsized":"2024-03-06T07:32:11.924271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:32.127402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907194+0000","last_change":"2024-03-06T07:32:10.904858+0000","last_active":"2024-03-06T07:32:11.907194+0000","last_peered":"2024-03-06T07:32:11.907194+0000","last_clean":"2024-03-06T07:32:11.907194+0000","last_became_active":"2024-03-06T07:32:10.904518+0000","last_became_peered":"2024-03-06T07:32:10.904518+0000","last_unstale":"2024-03-06T07:32:11.907194+0000","last_undegraded":"2024-03-06T07:32:11.907194+0000","last_fullsized":"2024-03-06T07:32:11.907194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:47.967210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.112667+0000","last_change":"2024-03-06T07:32:09.903109+0000","last_active":"2024-03-06T07:32:12.112667+0000","last_peered":"2024-03-06T07:32:12.112667+0000","last_clean":"2024-03-06T07:32:12.112667+0000","last_became_active":"2024-03-06T07:32:09.900761+0000","last_became_peered":"2024-03-06T07:32:09.900761+0000","last_unstale":"2024-03-06T07:32:12.112667+0000","last_undegraded":"2024-03-06T07:32:12.112667+0000","last_fullsized":"2024-03-06T07:32:12.112667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:55.028253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.909220+0000","last_change":"2024-03-06T07:32:10.902532+0000","last_active":"2024-03-06T07:32:11.909220+0000","last_peered":"2024-03-06T07:32:11.909220+0000","last_clean":"2024-03-06T07:32:11.909220+0000","last_became_active":"2024-03-06T07:32:10.901454+0000","last_became_peered":"2024-03-06T07:32:10.901454+0000","last_unstale":"2024-03-06T07:32:11.909220+0000","last_undegraded":"2024-03-06T07:32:11.909220+0000","last_fullsized":"2024-03-06T07:32:11.909220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:03.712807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903901+0000","last_change":"2024-03-06T07:32:09.905750+0000","last_active":"2024-03-06T07:32:11.903901+0000","last_peered":"2024-03-06T07:32:11.903901+0000","last_clean":"2024-03-06T07:32:11.903901+0000","last_became_active":"2024-03-06T07:32:09.905447+0000","last_became_peered":"2024-03-06T07:32:09.905447+0000","last_unstale":"2024-03-06T07:32:11.903901+0000","last_undegraded":"2024-03-06T07:32:11.903901+0000","last_fullsized":"2024-03-06T07:32:11.903901+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:48.685552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114622+0000","last_change":"2024-03-06T07:32:10.911773+0000","last_active":"2024-03-06T07:32:12.114622+0000","last_peered":"2024-03-06T07:32:12.114622+0000","last_clean":"2024-03-06T07:32:12.114622+0000","last_became_active":"2024-03-06T07:32:10.911251+0000","last_became_peered":"2024-03-06T07:32:10.911251+0000","last_unstale":"2024-03-06T07:32:12.114622+0000","last_undegraded":"2024-03-06T07:32:12.114622+0000","last_fullsized":"2024-03-06T07:32:12.114622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:24.539975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903663+0000","last_change":"2024-03-06T07:32:09.900031+0000","last_active":"2024-03-06T07:32:11.903663+0000","last_peered":"2024-03-06T07:32:11.903663+0000","last_clean":"2024-03-06T07:32:11.903663+0000","last_became_active":"2024-03-06T07:32:09.896182+0000","last_became_peered":"2024-03-06T07:32:09.896182+0000","last_unstale":"2024-03-06T07:32:11.903663+0000","last_undegraded":"2024-03-06T07:32:11.903663+0000","last_fullsized":"2024-03-06T07:32:11.903663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.838640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116821+0000","last_change":"2024-03-06T07:32:10.914152+0000","last_active":"2024-03-06T07:32:12.116821+0000","last_peered":"2024-03-06T07:32:12.116821+0000","last_clean":"2024-03-06T07:32:12.116821+0000","last_became_active":"2024-03-06T07:32:10.913821+0000","last_became_peered":"2024-03-06T07:32:10.913821+0000","last_unstale":"2024-03-06T07:32:12.116821+0000","last_undegraded":"2024-03-06T07:32:12.116821+0000","last_fullsized":"2024-03-06T07:32:12.116821+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:34.056184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115648+0000","last_change":"2024-03-06T07:32:09.912911+0000","last_active":"2024-03-06T07:32:12.115648+0000","last_peered":"2024-03-06T07:32:12.115648+0000","last_clean":"2024-03-06T07:32:12.115648+0000","last_became_active":"2024-03-06T07:32:09.911917+0000","last_became_peered":"2024-03-06T07:32:09.911917+0000","last_unstale":"2024-03-06T07:32:12.115648+0000","last_undegraded":"2024-03-06T07:32:12.115648+0000","last_fullsized":"2024-03-06T07:32:12.115648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.877134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908972+0000","last_change":"2024-03-06T07:32:10.900492+0000","last_active":"2024-03-06T07:32:11.908972+0000","last_peered":"2024-03-06T07:32:11.908972+0000","last_clean":"2024-03-06T07:32:11.908972+0000","last_became_active":"2024-03-06T07:32:10.900174+0000","last_became_peered":"2024-03-06T07:32:10.900174+0000","last_unstale":"2024-03-06T07:32:11.908972+0000","last_undegraded":"2024-03-06T07:32:11.908972+0000","last_fullsized":"2024-03-06T07:32:11.908972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:06.169996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116575+0000","last_change":"2024-03-06T07:32:09.914488+0000","last_active":"2024-03-06T07:32:12.116575+0000","last_peered":"2024-03-06T07:32:12.116575+0000","last_clean":"2024-03-06T07:32:12.116575+0000","last_became_active":"2024-03-06T07:32:09.912735+0000","last_became_peered":"2024-03-06T07:32:09.912735+0000","last_unstale":"2024-03-06T07:32:12.116575+0000","last_undegraded":"2024-03-06T07:32:12.116575+0000","last_fullsized":"2024-03-06T07:32:12.116575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:56.076163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907605+0000","last_change":"2024-03-06T07:32:10.898238+0000","last_active":"2024-03-06T07:32:11.907605+0000","last_peered":"2024-03-06T07:32:11.907605+0000","last_clean":"2024-03-06T07:32:11.907605+0000","last_became_active":"2024-03-06T07:32:10.897914+0000","last_became_peered":"2024-03-06T07:32:10.897914+0000","last_unstale":"2024-03-06T07:32:11.907605+0000","last_undegraded":"2024-03-06T07:32:11.907605+0000","last_fullsized":"2024-03-06T07:32:11.907605+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:41.770080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913914+0000","last_change":"2024-03-06T07:32:09.905630+0000","last_active":"2024-03-06T07:32:11.913914+0000","last_peered":"2024-03-06T07:32:11.913914+0000","last_clean":"2024-03-06T07:32:11.913914+0000","last_became_active":"2024-03-06T07:32:09.905353+0000","last_became_peered":"2024-03-06T07:32:09.905353+0000","last_unstale":"2024-03-06T07:32:11.913914+0000","last_undegraded":"2024-03-06T07:32:11.913914+0000","last_fullsized":"2024-03-06T07:32:11.913914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:05.489460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913942+0000","last_change":"2024-03-06T07:32:10.908036+0000","last_active":"2024-03-06T07:32:11.913942+0000","last_peered":"2024-03-06T07:32:11.913942+0000","last_clean":"2024-03-06T07:32:11.913942+0000","last_became_active":"2024-03-06T07:32:10.907545+0000","last_became_peered":"2024-03-06T07:32:10.907545+0000","last_unstale":"2024-03-06T07:32:11.913942+0000","last_undegraded":"2024-03-06T07:32:11.913942+0000","last_fullsized":"2024-03-06T07:32:11.913942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:23.206762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907108+0000","last_change":"2024-03-06T07:32:09.902786+0000","last_active":"2024-03-06T07:32:11.907108+0000","last_peered":"2024-03-06T07:32:11.907108+0000","last_clean":"2024-03-06T07:32:11.907108+0000","last_became_active":"2024-03-06T07:32:09.902392+0000","last_became_peered":"2024-03-06T07:32:09.902392+0000","last_unstale":"2024-03-06T07:32:11.907108+0000","last_undegraded":"2024-03-06T07:32:11.907108+0000","last_fullsized":"2024-03-06T07:32:11.907108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:30.889092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113956+0000","last_change":"2024-03-06T07:32:10.916279+0000","last_active":"2024-03-06T07:32:12.113956+0000","last_peered":"2024-03-06T07:32:12.113956+0000","last_clean":"2024-03-06T07:32:12.113956+0000","last_became_active":"2024-03-06T07:32:10.915695+0000","last_became_peered":"2024-03-06T07:32:10.915695+0000","last_unstale":"2024-03-06T07:32:12.113956+0000","last_undegraded":"2024-03-06T07:32:12.113956+0000","last_fullsized":"2024-03-06T07:32:12.113956+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:43.352252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115943+0000","last_change":"2024-03-06T07:32:09.910421+0000","last_active":"2024-03-06T07:32:12.115943+0000","last_peered":"2024-03-06T07:32:12.115943+0000","last_clean":"2024-03-06T07:32:12.115943+0000","last_became_active":"2024-03-06T07:32:09.910102+0000","last_became_peered":"2024-03-06T07:32:09.910102+0000","last_unstale":"2024-03-06T07:32:12.115943+0000","last_undegraded":"2024-03-06T07:32:12.115943+0000","last_fullsized":"2024-03-06T07:32:12.115943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:00.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115674+0000","last_change":"2024-03-06T07:32:10.912648+0000","last_active":"2024-03-06T07:32:12.115674+0000","last_peered":"2024-03-06T07:32:12.115674+0000","last_clean":"2024-03-06T07:32:12.115674+0000","last_became_active":"2024-03-06T07:32:10.912354+0000","last_became_peered":"2024-03-06T07:32:10.912354+0000","last_unstale":"2024-03-06T07:32:12.115674+0000","last_undegraded":"2024-03-06T07:32:12.115674+0000","last_fullsized":"2024-03-06T07:32:12.115674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:01.901970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906512+0000","last_change":"2024-03-06T07:32:09.914162+0000","last_active":"2024-03-06T07:32:11.906512+0000","last_peered":"2024-03-06T07:32:11.906512+0000","last_clean":"2024-03-06T07:32:11.906512+0000","last_became_active":"2024-03-06T07:32:09.913255+0000","last_became_peered":"2024-03-06T07:32:09.913255+0000","last_unstale":"2024-03-06T07:32:11.906512+0000","last_undegraded":"2024-03-06T07:32:11.906512+0000","last_fullsized":"2024-03-06T07:32:11.906512+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:02.681729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904617+0000","last_change":"2024-03-06T07:32:10.900210+0000","last_active":"2024-03-06T07:32:11.904617+0000","last_peered":"2024-03-06T07:32:11.904617+0000","last_clean":"2024-03-06T07:32:11.904617+0000","last_became_active":"2024-03-06T07:32:10.899475+0000","last_became_peered":"2024-03-06T07:32:10.899475+0000","last_unstale":"2024-03-06T07:32:11.904617+0000","last_undegraded":"2024-03-06T07:32:11.904617+0000","last_fullsized":"2024-03-06T07:32:11.904617+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:57.090043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115609+0000","last_change":"2024-03-06T07:32:09.915425+0000","last_active":"2024-03-06T07:32:12.115609+0000","last_peered":"2024-03-06T07:32:12.115609+0000","last_clean":"2024-03-06T07:32:12.115609+0000","last_became_active":"2024-03-06T07:32:09.915111+0000","last_became_peered":"2024-03-06T07:32:09.915111+0000","last_unstale":"2024-03-06T07:32:12.115609+0000","last_undegraded":"2024-03-06T07:32:12.115609+0000","last_fullsized":"2024-03-06T07:32:12.115609+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:51.607293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":283,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113985+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:32:12.113985+0000","last_peered":"2024-03-06T07:32:12.113985+0000","last_clean":"2024-03-06T07:32:12.113985+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:32:12.113985+0000","last_undegraded":"2024-03-06T07:32:12.113985+0000","last_fullsized":"2024-03-06T07:32:12.113985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542202,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1015787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542203,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1015808,"data_stored":426043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":426983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542203,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":425349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542202,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1015808,"data_stored":426961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542204,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":425931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1015808,"data_stored":425543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542203,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1015751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:15.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:15.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.599+0000 7fa36b200640 1 Processor -- start 2024-03-06T07:32:15.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.600+0000 7fa36b200640 1 -- start start 2024-03-06T07:32:15.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa36b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa36c161580 0x7fa36c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa36b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d2a70 0x7fa36c0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa36b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa36c161960 con 0x7fa36c0d1e90 2024-03-06T07:32:15.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa36b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36c0cf6f0 con 0x7fa36c161580 2024-03-06T07:32:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa36b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0cf9d0 con 0x7fa36c0d2a70 2024-03-06T07:32:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa369e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa36c161580 0x7fa36c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa369e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa36c161580 0x7fa36c0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38666/0 (socket says 172.21.15.97:38666) 2024-03-06T07:32:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.601+0000 7fa369e00640 1 -- 172.21.15.97:0/3385708529 learned_addr learned my addr 172.21.15.97:0/3385708529 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.602+0000 7fa36a800640 1 --2- 172.21.15.97:0/3385708529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d2a70 0x7fa36c0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.602+0000 7fa369e00640 1 -- 172.21.15.97:0/3385708529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d2a70 msgr2=0x7fa36c0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.602+0000 7fa369e00640 1 --2- 172.21.15.97:0/3385708529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d2a70 0x7fa36c0b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.602+0000 7fa369e00640 1 -- 172.21.15.97:0/3385708529 >> v1:172.21.15.135:6789/0 conn(0x7fa36c0d1e90 legacy=0x7fa36c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.602+0000 7fa369e00640 1 -- 172.21.15.97:0/3385708529 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c0cfcb0 con 0x7fa36c161580 2024-03-06T07:32:15.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.603+0000 7fa369e00640 1 --2- 172.21.15.97:0/3385708529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0d1950 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa36c073740 tx=0x7fa35402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61a293b86ac25ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.603+0000 7fa368a00640 1 -- 172.21.15.97:0/3385708529 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa354035020 con 0x7fa36c161580 2024-03-06T07:32:15.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.603+0000 7fa368a00640 1 -- 172.21.15.97:0/3385708529 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa354031830 con 0x7fa36c161580 2024-03-06T07:32:15.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.603+0000 7fa368a00640 1 -- 172.21.15.97:0/3385708529 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa354031c00 con 0x7fa36c161580 2024-03-06T07:32:15.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 -- 172.21.15.97:0/3385708529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 msgr2=0x7fa36c0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 --2- 172.21.15.97:0/3385708529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0d1950 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa36c073740 tx=0x7fa35402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 -- 172.21.15.97:0/3385708529 shutdown_connections 2024-03-06T07:32:15.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 --2- 172.21.15.97:0/3385708529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d2a70 0x7fa36c0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 --2- 172.21.15.97:0/3385708529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0d1950 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 -- 172.21.15.97:0/3385708529 >> 172.21.15.97:0/3385708529 conn(0x7fa36c0d3d80 msgr2=0x7fa36c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 -- 172.21.15.97:0/3385708529 shutdown_connections 2024-03-06T07:32:15.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.604+0000 7fa36b200640 1 -- 172.21.15.97:0/3385708529 wait complete. 2024-03-06T07:32:15.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.606+0000 7fa36b200640 1 Processor -- start 2024-03-06T07:32:15.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.606+0000 7fa36b200640 1 -- start start 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.607+0000 7fa36b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d1e90 0x7fa36c159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa36c0d2a70 0x7fa36c0b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0cf3a0 con 0x7fa36c161580 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0b39a0 con 0x7fa36c0d2a70 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0b4400 con 0x7fa36c0d1e90 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38674/0 (socket says 172.21.15.97:38674) 2024-03-06T07:32:15.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 -- 172.21.15.97:0/863946327 learned_addr learned my addr 172.21.15.97:0/863946327 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa369e00640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d1e90 0x7fa36c159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa369400640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa36c0d2a70 0x7fa36c0b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 -- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d1e90 msgr2=0x7fa36c159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d1e90 0x7fa36c159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 -- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa36c0d2a70 msgr2=0x7fa36c0b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa36c0d2a70 0x7fa36c0b7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa36a800640 1 -- 172.21.15.97:0/863946327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c0cc6f0 con 0x7fa36c161580 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.608+0000 7fa369400640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa36c0d2a70 0x7fa36c0b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.609+0000 7fa36a800640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0cbea0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c5f0 tx=0x7fa36400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.609+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364015070 con 0x7fa36c161580 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.609+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa364004220 con 0x7fa36c161580 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.609+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c0cc9d0 con 0x7fa36c161580 2024-03-06T07:32:15.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.609+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364003610 con 0x7fa36c161580 2024-03-06T07:32:15.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.609+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c0ccf90 con 0x7fa36c161580 2024-03-06T07:32:15.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.611+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c04f330 con 0x7fa36c161580 2024-03-06T07:32:15.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.612+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3640037b0 con 0x7fa36c161580 2024-03-06T07:32:15.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.612+0000 7fa362000640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa34806b4d0 0x7fa34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.612+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa36408a0b0 con 0x7fa36c161580 2024-03-06T07:32:15.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.613+0000 7fa369e00640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa34806b4d0 0x7fa34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.613+0000 7fa369e00640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa34806b4d0 0x7fa34806d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa354036000 tx=0x7fa35403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.617+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa364058e90 con 0x7fa36c161580 2024-03-06T07:32:15.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.809+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa36c0d3b60 con 0x7fa34806b4d0 2024-03-06T07:32:15.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.816+0000 7fa362000640 1 -- 172.21.15.97:0/863946327 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252079 (secure 0 0 0) 0x7fa36c0d3b60 con 0x7fa34806b4d0 2024-03-06T07:32:15.820 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:15.821 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa34806b4d0 msgr2=0x7fa34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa34806b4d0 0x7fa34806d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa354036000 tx=0x7fa35403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 msgr2=0x7fa36c0cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0cbea0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c5f0 tx=0x7fa36400cab0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 shutdown_connections 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa34806b4d0 0x7fa34806d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa36c161580 0x7fa36c0cbea0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa36c0d2a70 0x7fa36c0b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 --2- 172.21.15.97:0/863946327 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa36c0d1e90 0x7fa36c159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 >> 172.21.15.97:0/863946327 conn(0x7fa36c0d3d80 msgr2=0x7fa36c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 shutdown_connections 2024-03-06T07:32:15.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:15.822+0000 7fa36b200640 1 -- 172.21.15.97:0/863946327 wait complete. 2024-03-06T07:32:15.843 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-03-06T07:32:14.984823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258516,"kb_used_data":9064,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716204,"statfs":{"total":773094113280,"available":772829392896,"internally_reserved":0,"allocated":9281536,"data_stored":4588348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001466"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115641+0000","last_change":"2024-03-06T07:32:10.906307+0000","last_active":"2024-03-06T07:32:12.115641+0000","last_peered":"2024-03-06T07:32:12.115641+0000","last_clean":"2024-03-06T07:32:12.115641+0000","last_became_active":"2024-03-06T07:32:10.906031+0000","last_became_peered":"2024-03-06T07:32:10.906031+0000","last_unstale":"2024-03-06T07:32:12.115641+0000","last_undegraded":"2024-03-06T07:32:12.115641+0000","last_fullsized":"2024-03-06T07:32:12.115641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:17.489151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903664+0000","last_change":"2024-03-06T07:32:10.905159+0000","last_active":"2024-03-06T07:32:11.903664+0000","last_peered":"2024-03-06T07:32:11.903664+0000","last_clean":"2024-03-06T07:32:11.903664+0000","last_became_active":"2024-03-06T07:32:10.904798+0000","last_became_peered":"2024-03-06T07:32:10.904798+0000","last_unstale":"2024-03-06T07:32:11.903664+0000","last_undegraded":"2024-03-06T07:32:11.903664+0000","last_fullsized":"2024-03-06T07:32:11.903664+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:03.467161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114223+0000","last_change":"2024-03-06T07:32:10.911774+0000","last_active":"2024-03-06T07:32:12.114223+0000","last_peered":"2024-03-06T07:32:12.114223+0000","last_clean":"2024-03-06T07:32:12.114223+0000","last_became_active":"2024-03-06T07:32:10.911491+0000","last_became_peered":"2024-03-06T07:32:10.911491+0000","last_unstale":"2024-03-06T07:32:12.114223+0000","last_undegraded":"2024-03-06T07:32:12.114223+0000","last_fullsized":"2024-03-06T07:32:12.114223+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:55.939968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924655+0000","last_change":"2024-03-06T07:32:10.906713+0000","last_active":"2024-03-06T07:32:11.924655+0000","last_peered":"2024-03-06T07:32:11.924655+0000","last_clean":"2024-03-06T07:32:11.924655+0000","last_became_active":"2024-03-06T07:32:10.906407+0000","last_became_peered":"2024-03-06T07:32:10.906407+0000","last_unstale":"2024-03-06T07:32:11.924655+0000","last_undegraded":"2024-03-06T07:32:11.924655+0000","last_fullsized":"2024-03-06T07:32:11.924655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:04.482049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906884+0000","last_change":"2024-03-06T07:32:10.891436+0000","last_active":"2024-03-06T07:32:11.906884+0000","last_peered":"2024-03-06T07:32:11.906884+0000","last_clean":"2024-03-06T07:32:11.906884+0000","last_became_active":"2024-03-06T07:32:10.890765+0000","last_became_peered":"2024-03-06T07:32:10.890765+0000","last_unstale":"2024-03-06T07:32:11.906884+0000","last_undegraded":"2024-03-06T07:32:11.906884+0000","last_fullsized":"2024-03-06T07:32:11.906884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:00.065997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907415+0000","last_change":"2024-03-06T07:32:10.907119+0000","last_active":"2024-03-06T07:32:11.907415+0000","last_peered":"2024-03-06T07:32:11.907415+0000","last_clean":"2024-03-06T07:32:11.907415+0000","last_became_active":"2024-03-06T07:32:10.906793+0000","last_became_peered":"2024-03-06T07:32:10.906793+0000","last_unstale":"2024-03-06T07:32:11.907415+0000","last_undegraded":"2024-03-06T07:32:11.907415+0000","last_fullsized":"2024-03-06T07:32:11.907415+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:52.621173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924612+0000","last_change":"2024-03-06T07:32:10.907152+0000","last_active":"2024-03-06T07:32:11.924612+0000","last_peered":"2024-03-06T07:32:11.924612+0000","last_clean":"2024-03-06T07:32:11.924612+0000","last_became_active":"2024-03-06T07:32:10.906822+0000","last_became_peered":"2024-03-06T07:32:10.906822+0000","last_unstale":"2024-03-06T07:32:11.924612+0000","last_undegraded":"2024-03-06T07:32:11.924612+0000","last_fullsized":"2024-03-06T07:32:11.924612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:29.817303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906966+0000","last_change":"2024-03-06T07:32:10.905884+0000","last_active":"2024-03-06T07:32:11.906966+0000","last_peered":"2024-03-06T07:32:11.906966+0000","last_clean":"2024-03-06T07:32:11.906966+0000","last_became_active":"2024-03-06T07:32:10.903362+0000","last_became_peered":"2024-03-06T07:32:10.903362+0000","last_unstale":"2024-03-06T07:32:11.906966+0000","last_undegraded":"2024-03-06T07:32:11.906966+0000","last_fullsized":"2024-03-06T07:32:11.906966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:15.525677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908051+0000","last_change":"2024-03-06T07:32:10.902379+0000","last_active":"2024-03-06T07:32:11.908051+0000","last_peered":"2024-03-06T07:32:11.908051+0000","last_clean":"2024-03-06T07:32:11.908051+0000","last_became_active":"2024-03-06T07:32:10.901748+0000","last_became_peered":"2024-03-06T07:32:10.901748+0000","last_unstale":"2024-03-06T07:32:11.908051+0000","last_undegraded":"2024-03-06T07:32:11.908051+0000","last_fullsized":"2024-03-06T07:32:11.908051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:27.928075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116362+0000","last_change":"2024-03-06T07:32:10.903489+0000","last_active":"2024-03-06T07:32:12.116362+0000","last_peered":"2024-03-06T07:32:12.116362+0000","last_clean":"2024-03-06T07:32:12.116362+0000","last_became_active":"2024-03-06T07:32:10.903177+0000","last_became_peered":"2024-03-06T07:32:10.903177+0000","last_unstale":"2024-03-06T07:32:12.116362+0000","last_undegraded":"2024-03-06T07:32:12.116362+0000","last_fullsized":"2024-03-06T07:32:12.116362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:12.601104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904266+0000","last_change":"2024-03-06T07:32:10.905087+0000","last_active":"2024-03-06T07:32:11.904266+0000","last_peered":"2024-03-06T07:32:11.904266+0000","last_clean":"2024-03-06T07:32:11.904266+0000","last_became_active":"2024-03-06T07:32:10.904786+0000","last_became_peered":"2024-03-06T07:32:10.904786+0000","last_unstale":"2024-03-06T07:32:11.904266+0000","last_undegraded":"2024-03-06T07:32:11.904266+0000","last_fullsized":"2024-03-06T07:32:11.904266+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:40.611043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115527+0000","last_change":"2024-03-06T07:32:10.903851+0000","last_active":"2024-03-06T07:32:12.115527+0000","last_peered":"2024-03-06T07:32:12.115527+0000","last_clean":"2024-03-06T07:32:12.115527+0000","last_became_active":"2024-03-06T07:32:10.903574+0000","last_became_peered":"2024-03-06T07:32:10.903574+0000","last_unstale":"2024-03-06T07:32:12.115527+0000","last_undegraded":"2024-03-06T07:32:12.115527+0000","last_fullsized":"2024-03-06T07:32:12.115527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:27.928075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113916+0000","last_change":"2024-03-06T07:32:10.910717+0000","last_active":"2024-03-06T07:32:12.113916+0000","last_peered":"2024-03-06T07:32:12.113916+0000","last_clean":"2024-03-06T07:32:12.113916+0000","last_became_active":"2024-03-06T07:32:10.907720+0000","last_became_peered":"2024-03-06T07:32:10.907720+0000","last_unstale":"2024-03-06T07:32:12.113916+0000","last_undegraded":"2024-03-06T07:32:12.113916+0000","last_fullsized":"2024-03-06T07:32:12.113916+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:34.056184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924975+0000","last_change":"2024-03-06T07:32:10.910589+0000","last_active":"2024-03-06T07:32:11.924975+0000","last_peered":"2024-03-06T07:32:11.924975+0000","last_clean":"2024-03-06T07:32:11.924975+0000","last_became_active":"2024-03-06T07:32:10.910325+0000","last_became_peered":"2024-03-06T07:32:10.910325+0000","last_unstale":"2024-03-06T07:32:11.924975+0000","last_undegraded":"2024-03-06T07:32:11.924975+0000","last_fullsized":"2024-03-06T07:32:11.924975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:29.448692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.112832+0000","last_change":"2024-03-06T07:32:10.905760+0000","last_active":"2024-03-06T07:32:12.112832+0000","last_peered":"2024-03-06T07:32:12.112832+0000","last_clean":"2024-03-06T07:32:12.112832+0000","last_became_active":"2024-03-06T07:32:10.905493+0000","last_became_peered":"2024-03-06T07:32:10.905493+0000","last_unstale":"2024-03-06T07:32:12.112832+0000","last_undegraded":"2024-03-06T07:32:12.112832+0000","last_fullsized":"2024-03-06T07:32:12.112832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:03.458405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906531+0000","last_change":"2024-03-06T07:32:10.910181+0000","last_active":"2024-03-06T07:32:11.906531+0000","last_peered":"2024-03-06T07:32:11.906531+0000","last_clean":"2024-03-06T07:32:11.906531+0000","last_became_active":"2024-03-06T07:32:10.909881+0000","last_became_peered":"2024-03-06T07:32:10.909881+0000","last_unstale":"2024-03-06T07:32:11.906531+0000","last_undegraded":"2024-03-06T07:32:11.906531+0000","last_fullsized":"2024-03-06T07:32:11.906531+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:09.470182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908755+0000","last_change":"2024-03-06T07:32:10.897916+0000","last_active":"2024-03-06T07:32:11.908755+0000","last_peered":"2024-03-06T07:32:11.908755+0000","last_clean":"2024-03-06T07:32:11.908755+0000","last_became_active":"2024-03-06T07:32:10.896112+0000","last_became_peered":"2024-03-06T07:32:10.896112+0000","last_unstale":"2024-03-06T07:32:11.908755+0000","last_undegraded":"2024-03-06T07:32:11.908755+0000","last_fullsized":"2024-03-06T07:32:11.908755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:28.186902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.925307+0000","last_change":"2024-03-06T07:32:10.902098+0000","last_active":"2024-03-06T07:32:11.925307+0000","last_peered":"2024-03-06T07:32:11.925307+0000","last_clean":"2024-03-06T07:32:11.925307+0000","last_became_active":"2024-03-06T07:32:10.901826+0000","last_became_peered":"2024-03-06T07:32:10.901826+0000","last_unstale":"2024-03-06T07:32:11.925307+0000","last_undegraded":"2024-03-06T07:32:11.925307+0000","last_fullsized":"2024-03-06T07:32:11.925307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:07.037956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.914056+0000","last_change":"2024-03-06T07:32:10.913902+0000","last_active":"2024-03-06T07:32:11.914056+0000","last_peered":"2024-03-06T07:32:11.914056+0000","last_clean":"2024-03-06T07:32:11.914056+0000","last_became_active":"2024-03-06T07:32:10.913557+0000","last_became_peered":"2024-03-06T07:32:10.913557+0000","last_unstale":"2024-03-06T07:32:11.914056+0000","last_undegraded":"2024-03-06T07:32:11.914056+0000","last_fullsized":"2024-03-06T07:32:11.914056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:27.928075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924897+0000","last_change":"2024-03-06T07:32:10.912261+0000","last_active":"2024-03-06T07:32:11.924897+0000","last_peered":"2024-03-06T07:32:11.924897+0000","last_clean":"2024-03-06T07:32:11.924897+0000","last_became_active":"2024-03-06T07:32:10.911969+0000","last_became_peered":"2024-03-06T07:32:10.911969+0000","last_unstale":"2024-03-06T07:32:11.924897+0000","last_undegraded":"2024-03-06T07:32:11.924897+0000","last_fullsized":"2024-03-06T07:32:11.924897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:58.800571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.925039+0000","last_change":"2024-03-06T07:32:10.905838+0000","last_active":"2024-03-06T07:32:11.925039+0000","last_peered":"2024-03-06T07:32:11.925039+0000","last_clean":"2024-03-06T07:32:11.925039+0000","last_became_active":"2024-03-06T07:32:10.905524+0000","last_became_peered":"2024-03-06T07:32:10.905524+0000","last_unstale":"2024-03-06T07:32:11.925039+0000","last_undegraded":"2024-03-06T07:32:11.925039+0000","last_fullsized":"2024-03-06T07:32:11.925039+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:04.979450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904021+0000","last_change":"2024-03-06T07:32:10.903437+0000","last_active":"2024-03-06T07:32:11.904021+0000","last_peered":"2024-03-06T07:32:11.904021+0000","last_clean":"2024-03-06T07:32:11.904021+0000","last_became_active":"2024-03-06T07:32:10.902775+0000","last_became_peered":"2024-03-06T07:32:10.902775+0000","last_unstale":"2024-03-06T07:32:11.904021+0000","last_undegraded":"2024-03-06T07:32:11.904021+0000","last_fullsized":"2024-03-06T07:32:11.904021+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:00.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115149+0000","last_change":"2024-03-06T07:32:10.906635+0000","last_active":"2024-03-06T07:32:12.115149+0000","last_peered":"2024-03-06T07:32:12.115149+0000","last_clean":"2024-03-06T07:32:12.115149+0000","last_became_active":"2024-03-06T07:32:10.904722+0000","last_became_peered":"2024-03-06T07:32:10.904722+0000","last_unstale":"2024-03-06T07:32:12.115149+0000","last_undegraded":"2024-03-06T07:32:12.115149+0000","last_fullsized":"2024-03-06T07:32:12.115149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:23.206762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924450+0000","last_change":"2024-03-06T07:32:10.911560+0000","last_active":"2024-03-06T07:32:11.924450+0000","last_peered":"2024-03-06T07:32:11.924450+0000","last_clean":"2024-03-06T07:32:11.924450+0000","last_became_active":"2024-03-06T07:32:10.910427+0000","last_became_peered":"2024-03-06T07:32:10.910427+0000","last_unstale":"2024-03-06T07:32:11.924450+0000","last_undegraded":"2024-03-06T07:32:11.924450+0000","last_fullsized":"2024-03-06T07:32:11.924450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:17.154279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903853+0000","last_change":"2024-03-06T07:32:10.903598+0000","last_active":"2024-03-06T07:32:11.903853+0000","last_peered":"2024-03-06T07:32:11.903853+0000","last_clean":"2024-03-06T07:32:11.903853+0000","last_became_active":"2024-03-06T07:32:10.902913+0000","last_became_peered":"2024-03-06T07:32:10.902913+0000","last_unstale":"2024-03-06T07:32:11.903853+0000","last_undegraded":"2024-03-06T07:32:11.903853+0000","last_fullsized":"2024-03-06T07:32:11.903853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:53.829597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115851+0000","last_change":"2024-03-06T07:32:10.917664+0000","last_active":"2024-03-06T07:32:12.115851+0000","last_peered":"2024-03-06T07:32:12.115851+0000","last_clean":"2024-03-06T07:32:12.115851+0000","last_became_active":"2024-03-06T07:32:10.916978+0000","last_became_peered":"2024-03-06T07:32:10.916978+0000","last_unstale":"2024-03-06T07:32:12.115851+0000","last_undegraded":"2024-03-06T07:32:12.115851+0000","last_fullsized":"2024-03-06T07:32:12.115851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:01.928083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908438+0000","last_change":"2024-03-06T07:32:10.896284+0000","last_active":"2024-03-06T07:32:11.908438+0000","last_peered":"2024-03-06T07:32:11.908438+0000","last_clean":"2024-03-06T07:32:11.908438+0000","last_became_active":"2024-03-06T07:32:10.895793+0000","last_became_peered":"2024-03-06T07:32:10.895793+0000","last_unstale":"2024-03-06T07:32:11.908438+0000","last_undegraded":"2024-03-06T07:32:11.908438+0000","last_fullsized":"2024-03-06T07:32:11.908438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:23.720230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115909+0000","last_change":"2024-03-06T07:32:10.907758+0000","last_active":"2024-03-06T07:32:12.115909+0000","last_peered":"2024-03-06T07:32:12.115909+0000","last_clean":"2024-03-06T07:32:12.115909+0000","last_became_active":"2024-03-06T07:32:10.907406+0000","last_became_peered":"2024-03-06T07:32:10.907406+0000","last_unstale":"2024-03-06T07:32:12.115909+0000","last_undegraded":"2024-03-06T07:32:12.115909+0000","last_fullsized":"2024-03-06T07:32:12.115909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:21.535340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903529+0000","last_change":"2024-03-06T07:32:10.902214+0000","last_active":"2024-03-06T07:32:11.903529+0000","last_peered":"2024-03-06T07:32:11.903529+0000","last_clean":"2024-03-06T07:32:11.903529+0000","last_became_active":"2024-03-06T07:32:10.901872+0000","last_became_peered":"2024-03-06T07:32:10.901872+0000","last_unstale":"2024-03-06T07:32:11.903529+0000","last_undegraded":"2024-03-06T07:32:11.903529+0000","last_fullsized":"2024-03-06T07:32:11.903529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:48.555144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904097+0000","last_change":"2024-03-06T07:32:10.899766+0000","last_active":"2024-03-06T07:32:11.904097+0000","last_peered":"2024-03-06T07:32:11.904097+0000","last_clean":"2024-03-06T07:32:11.904097+0000","last_became_active":"2024-03-06T07:32:10.895450+0000","last_became_peered":"2024-03-06T07:32:10.895450+0000","last_unstale":"2024-03-06T07:32:11.904097+0000","last_undegraded":"2024-03-06T07:32:11.904097+0000","last_fullsized":"2024-03-06T07:32:11.904097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:17.110943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906944+0000","last_change":"2024-03-06T07:32:10.902364+0000","last_active":"2024-03-06T07:32:11.906944+0000","last_peered":"2024-03-06T07:32:11.906944+0000","last_clean":"2024-03-06T07:32:11.906944+0000","last_became_active":"2024-03-06T07:32:10.902045+0000","last_became_peered":"2024-03-06T07:32:10.902045+0000","last_unstale":"2024-03-06T07:32:11.906944+0000","last_undegraded":"2024-03-06T07:32:11.906944+0000","last_fullsized":"2024-03-06T07:32:11.906944+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:55.564188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114905+0000","last_change":"2024-03-06T07:32:10.907010+0000","last_active":"2024-03-06T07:32:12.114905+0000","last_peered":"2024-03-06T07:32:12.114905+0000","last_clean":"2024-03-06T07:32:12.114905+0000","last_became_active":"2024-03-06T07:32:10.906688+0000","last_became_peered":"2024-03-06T07:32:10.906688+0000","last_unstale":"2024-03-06T07:32:12.114905+0000","last_undegraded":"2024-03-06T07:32:12.114905+0000","last_fullsized":"2024-03-06T07:32:12.114905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:41.770080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.909015+0000","last_change":"2024-03-06T07:32:10.903032+0000","last_active":"2024-03-06T07:32:11.909015+0000","last_peered":"2024-03-06T07:32:11.909015+0000","last_clean":"2024-03-06T07:32:11.909015+0000","last_became_active":"2024-03-06T07:32:10.901998+0000","last_became_peered":"2024-03-06T07:32:10.901998+0000","last_unstale":"2024-03-06T07:32:11.909015+0000","last_undegraded":"2024-03-06T07:32:11.909015+0000","last_fullsized":"2024-03-06T07:32:11.909015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:25.997977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116309+0000","last_change":"2024-03-06T07:32:09.915655+0000","last_active":"2024-03-06T07:32:12.116309+0000","last_peered":"2024-03-06T07:32:12.116309+0000","last_clean":"2024-03-06T07:32:12.116309+0000","last_became_active":"2024-03-06T07:32:09.914539+0000","last_became_peered":"2024-03-06T07:32:09.914539+0000","last_unstale":"2024-03-06T07:32:12.116309+0000","last_undegraded":"2024-03-06T07:32:12.116309+0000","last_fullsized":"2024-03-06T07:32:12.116309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:08.948200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116065+0000","last_change":"2024-03-06T07:32:10.905398+0000","last_active":"2024-03-06T07:32:12.116065+0000","last_peered":"2024-03-06T07:32:12.116065+0000","last_clean":"2024-03-06T07:32:12.116065+0000","last_became_active":"2024-03-06T07:32:10.904540+0000","last_became_peered":"2024-03-06T07:32:10.904540+0000","last_unstale":"2024-03-06T07:32:12.116065+0000","last_undegraded":"2024-03-06T07:32:12.116065+0000","last_fullsized":"2024-03-06T07:32:12.116065+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:01.458813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.909156+0000","last_change":"2024-03-06T07:32:09.909605+0000","last_active":"2024-03-06T07:32:11.909156+0000","last_peered":"2024-03-06T07:32:11.909156+0000","last_clean":"2024-03-06T07:32:11.909156+0000","last_became_active":"2024-03-06T07:32:09.909320+0000","last_became_peered":"2024-03-06T07:32:09.909320+0000","last_unstale":"2024-03-06T07:32:11.909156+0000","last_undegraded":"2024-03-06T07:32:11.909156+0000","last_fullsized":"2024-03-06T07:32:11.909156+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:22.101485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116367+0000","last_change":"2024-03-06T07:32:10.903612+0000","last_active":"2024-03-06T07:32:12.116367+0000","last_peered":"2024-03-06T07:32:12.116367+0000","last_clean":"2024-03-06T07:32:12.116367+0000","last_became_active":"2024-03-06T07:32:10.903164+0000","last_became_peered":"2024-03-06T07:32:10.903164+0000","last_unstale":"2024-03-06T07:32:12.116367+0000","last_undegraded":"2024-03-06T07:32:12.116367+0000","last_fullsized":"2024-03-06T07:32:12.116367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:45.822921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.914092+0000","last_change":"2024-03-06T07:32:09.907413+0000","last_active":"2024-03-06T07:32:11.914092+0000","last_peered":"2024-03-06T07:32:11.914092+0000","last_clean":"2024-03-06T07:32:11.914092+0000","last_became_active":"2024-03-06T07:32:09.906908+0000","last_became_peered":"2024-03-06T07:32:09.906908+0000","last_unstale":"2024-03-06T07:32:11.914092+0000","last_undegraded":"2024-03-06T07:32:11.914092+0000","last_fullsized":"2024-03-06T07:32:11.914092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:12.783824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115437+0000","last_change":"2024-03-06T07:32:10.916704+0000","last_active":"2024-03-06T07:32:12.115437+0000","last_peered":"2024-03-06T07:32:12.115437+0000","last_clean":"2024-03-06T07:32:12.115437+0000","last_became_active":"2024-03-06T07:32:10.916322+0000","last_became_peered":"2024-03-06T07:32:10.916322+0000","last_unstale":"2024-03-06T07:32:12.115437+0000","last_undegraded":"2024-03-06T07:32:12.115437+0000","last_fullsized":"2024-03-06T07:32:12.115437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:56.042132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913767+0000","last_change":"2024-03-06T07:32:09.900976+0000","last_active":"2024-03-06T07:32:11.913767+0000","last_peered":"2024-03-06T07:32:11.913767+0000","last_clean":"2024-03-06T07:32:11.913767+0000","last_became_active":"2024-03-06T07:32:09.900120+0000","last_became_peered":"2024-03-06T07:32:09.900120+0000","last_unstale":"2024-03-06T07:32:11.913767+0000","last_undegraded":"2024-03-06T07:32:11.913767+0000","last_fullsized":"2024-03-06T07:32:11.913767+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:54.923021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904233+0000","last_change":"2024-03-06T07:32:10.903248+0000","last_active":"2024-03-06T07:32:11.904233+0000","last_peered":"2024-03-06T07:32:11.904233+0000","last_clean":"2024-03-06T07:32:11.904233+0000","last_became_active":"2024-03-06T07:32:10.902339+0000","last_became_peered":"2024-03-06T07:32:10.902339+0000","last_unstale":"2024-03-06T07:32:11.904233+0000","last_undegraded":"2024-03-06T07:32:11.904233+0000","last_fullsized":"2024-03-06T07:32:11.904233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:58.093199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908466+0000","last_change":"2024-03-06T07:32:09.905122+0000","last_active":"2024-03-06T07:32:11.908466+0000","last_peered":"2024-03-06T07:32:11.908466+0000","last_clean":"2024-03-06T07:32:11.908466+0000","last_became_active":"2024-03-06T07:32:09.902443+0000","last_became_peered":"2024-03-06T07:32:09.902443+0000","last_unstale":"2024-03-06T07:32:11.908466+0000","last_undegraded":"2024-03-06T07:32:11.908466+0000","last_fullsized":"2024-03-06T07:32:11.908466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:48.354010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907747+0000","last_change":"2024-03-06T07:32:10.898964+0000","last_active":"2024-03-06T07:32:11.907747+0000","last_peered":"2024-03-06T07:32:11.907747+0000","last_clean":"2024-03-06T07:32:11.907747+0000","last_became_active":"2024-03-06T07:32:10.897563+0000","last_became_peered":"2024-03-06T07:32:10.897563+0000","last_unstale":"2024-03-06T07:32:11.907747+0000","last_undegraded":"2024-03-06T07:32:11.907747+0000","last_fullsized":"2024-03-06T07:32:11.907747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:49.357999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906923+0000","last_change":"2024-03-06T07:32:09.910143+0000","last_active":"2024-03-06T07:32:11.906923+0000","last_peered":"2024-03-06T07:32:11.906923+0000","last_clean":"2024-03-06T07:32:11.906923+0000","last_became_active":"2024-03-06T07:32:09.909790+0000","last_became_peered":"2024-03-06T07:32:09.909790+0000","last_unstale":"2024-03-06T07:32:11.906923+0000","last_undegraded":"2024-03-06T07:32:11.906923+0000","last_fullsized":"2024-03-06T07:32:11.906923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:48.919689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113831+0000","last_change":"2024-03-06T07:32:10.921486+0000","last_active":"2024-03-06T07:32:12.113831+0000","last_peered":"2024-03-06T07:32:12.113831+0000","last_clean":"2024-03-06T07:32:12.113831+0000","last_became_active":"2024-03-06T07:32:10.921156+0000","last_became_peered":"2024-03-06T07:32:10.921156+0000","last_unstale":"2024-03-06T07:32:12.113831+0000","last_undegraded":"2024-03-06T07:32:12.113831+0000","last_fullsized":"2024-03-06T07:32:12.113831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:08.709841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913543+0000","last_change":"2024-03-06T07:32:09.910707+0000","last_active":"2024-03-06T07:32:11.913543+0000","last_peered":"2024-03-06T07:32:11.913543+0000","last_clean":"2024-03-06T07:32:11.913543+0000","last_became_active":"2024-03-06T07:32:09.910401+0000","last_became_peered":"2024-03-06T07:32:09.910401+0000","last_unstale":"2024-03-06T07:32:11.913543+0000","last_undegraded":"2024-03-06T07:32:11.913543+0000","last_fullsized":"2024-03-06T07:32:11.913543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:45.531229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.112912+0000","last_change":"2024-03-06T07:32:10.903420+0000","last_active":"2024-03-06T07:32:12.112912+0000","last_peered":"2024-03-06T07:32:12.112912+0000","last_clean":"2024-03-06T07:32:12.112912+0000","last_became_active":"2024-03-06T07:32:10.903095+0000","last_became_peered":"2024-03-06T07:32:10.903095+0000","last_unstale":"2024-03-06T07:32:12.112912+0000","last_undegraded":"2024-03-06T07:32:12.112912+0000","last_fullsized":"2024-03-06T07:32:12.112912+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:24.419915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904194+0000","last_change":"2024-03-06T07:32:09.911588+0000","last_active":"2024-03-06T07:32:11.904194+0000","last_peered":"2024-03-06T07:32:11.904194+0000","last_clean":"2024-03-06T07:32:11.904194+0000","last_became_active":"2024-03-06T07:32:09.910267+0000","last_became_peered":"2024-03-06T07:32:09.910267+0000","last_unstale":"2024-03-06T07:32:11.904194+0000","last_undegraded":"2024-03-06T07:32:11.904194+0000","last_fullsized":"2024-03-06T07:32:11.904194+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:23.214455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113276+0000","last_change":"2024-03-06T07:32:10.910717+0000","last_active":"2024-03-06T07:32:12.113276+0000","last_peered":"2024-03-06T07:32:12.113276+0000","last_clean":"2024-03-06T07:32:12.113276+0000","last_became_active":"2024-03-06T07:32:10.910383+0000","last_became_peered":"2024-03-06T07:32:10.910383+0000","last_unstale":"2024-03-06T07:32:12.113276+0000","last_undegraded":"2024-03-06T07:32:12.113276+0000","last_fullsized":"2024-03-06T07:32:12.113276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:04.998290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116502+0000","last_change":"2024-03-06T07:32:09.915082+0000","last_active":"2024-03-06T07:32:12.116502+0000","last_peered":"2024-03-06T07:32:12.116502+0000","last_clean":"2024-03-06T07:32:12.116502+0000","last_became_active":"2024-03-06T07:32:09.914223+0000","last_became_peered":"2024-03-06T07:32:09.914223+0000","last_unstale":"2024-03-06T07:32:12.116502+0000","last_undegraded":"2024-03-06T07:32:12.116502+0000","last_fullsized":"2024-03-06T07:32:12.116502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:31.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908205+0000","last_change":"2024-03-06T07:32:10.909941+0000","last_active":"2024-03-06T07:32:11.908205+0000","last_peered":"2024-03-06T07:32:11.908205+0000","last_clean":"2024-03-06T07:32:11.908205+0000","last_became_active":"2024-03-06T07:32:10.909609+0000","last_became_peered":"2024-03-06T07:32:10.909609+0000","last_unstale":"2024-03-06T07:32:11.908205+0000","last_undegraded":"2024-03-06T07:32:11.908205+0000","last_fullsized":"2024-03-06T07:32:11.908205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:17.489151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116303+0000","last_change":"2024-03-06T07:32:09.917195+0000","last_active":"2024-03-06T07:32:12.116303+0000","last_peered":"2024-03-06T07:32:12.116303+0000","last_clean":"2024-03-06T07:32:12.116303+0000","last_became_active":"2024-03-06T07:32:09.916892+0000","last_became_peered":"2024-03-06T07:32:09.916892+0000","last_unstale":"2024-03-06T07:32:12.116303+0000","last_undegraded":"2024-03-06T07:32:12.116303+0000","last_fullsized":"2024-03-06T07:32:12.116303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:50.033779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913894+0000","last_change":"2024-03-06T07:32:10.917530+0000","last_active":"2024-03-06T07:32:11.913894+0000","last_peered":"2024-03-06T07:32:11.913894+0000","last_clean":"2024-03-06T07:32:11.913894+0000","last_became_active":"2024-03-06T07:32:10.913808+0000","last_became_peered":"2024-03-06T07:32:10.913808+0000","last_unstale":"2024-03-06T07:32:11.913894+0000","last_undegraded":"2024-03-06T07:32:11.913894+0000","last_fullsized":"2024-03-06T07:32:11.913894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:41.770080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115074+0000","last_change":"2024-03-06T07:32:09.914071+0000","last_active":"2024-03-06T07:32:12.115074+0000","last_peered":"2024-03-06T07:32:12.115074+0000","last_clean":"2024-03-06T07:32:12.115074+0000","last_became_active":"2024-03-06T07:32:09.913680+0000","last_became_peered":"2024-03-06T07:32:09.913680+0000","last_unstale":"2024-03-06T07:32:12.115074+0000","last_undegraded":"2024-03-06T07:32:12.115074+0000","last_fullsized":"2024-03-06T07:32:12.115074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:39.597163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908735+0000","last_change":"2024-03-06T07:32:10.911895+0000","last_active":"2024-03-06T07:32:11.908735+0000","last_peered":"2024-03-06T07:32:11.908735+0000","last_clean":"2024-03-06T07:32:11.908735+0000","last_became_active":"2024-03-06T07:32:10.909077+0000","last_became_peered":"2024-03-06T07:32:10.909077+0000","last_unstale":"2024-03-06T07:32:11.908735+0000","last_undegraded":"2024-03-06T07:32:11.908735+0000","last_fullsized":"2024-03-06T07:32:11.908735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:59.206364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907657+0000","last_change":"2024-03-06T07:32:09.912589+0000","last_active":"2024-03-06T07:32:11.907657+0000","last_peered":"2024-03-06T07:32:11.907657+0000","last_clean":"2024-03-06T07:32:11.907657+0000","last_became_active":"2024-03-06T07:32:09.910495+0000","last_became_peered":"2024-03-06T07:32:09.910495+0000","last_unstale":"2024-03-06T07:32:11.907657+0000","last_undegraded":"2024-03-06T07:32:11.907657+0000","last_fullsized":"2024-03-06T07:32:11.907657+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:20.461391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913694+0000","last_change":"2024-03-06T07:32:10.912663+0000","last_active":"2024-03-06T07:32:11.913694+0000","last_peered":"2024-03-06T07:32:11.913694+0000","last_clean":"2024-03-06T07:32:11.913694+0000","last_became_active":"2024-03-06T07:32:10.912374+0000","last_became_peered":"2024-03-06T07:32:10.912374+0000","last_unstale":"2024-03-06T07:32:11.913694+0000","last_undegraded":"2024-03-06T07:32:11.913694+0000","last_fullsized":"2024-03-06T07:32:11.913694+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:52.621173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903588+0000","last_change":"2024-03-06T07:32:09.894798+0000","last_active":"2024-03-06T07:32:11.903588+0000","last_peered":"2024-03-06T07:32:11.903588+0000","last_clean":"2024-03-06T07:32:11.903588+0000","last_became_active":"2024-03-06T07:32:09.891974+0000","last_became_peered":"2024-03-06T07:32:09.891974+0000","last_unstale":"2024-03-06T07:32:11.903588+0000","last_undegraded":"2024-03-06T07:32:11.903588+0000","last_fullsized":"2024-03-06T07:32:11.903588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:16.140399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114303+0000","last_change":"2024-03-06T07:32:10.921487+0000","last_active":"2024-03-06T07:32:12.114303+0000","last_peered":"2024-03-06T07:32:12.114303+0000","last_clean":"2024-03-06T07:32:12.114303+0000","last_became_active":"2024-03-06T07:32:10.921132+0000","last_became_peered":"2024-03-06T07:32:10.921132+0000","last_unstale":"2024-03-06T07:32:12.114303+0000","last_undegraded":"2024-03-06T07:32:12.114303+0000","last_fullsized":"2024-03-06T07:32:12.114303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:39.087262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113007+0000","last_change":"2024-03-06T07:32:09.897136+0000","last_active":"2024-03-06T07:32:12.113007+0000","last_peered":"2024-03-06T07:32:12.113007+0000","last_clean":"2024-03-06T07:32:12.113007+0000","last_became_active":"2024-03-06T07:32:09.896787+0000","last_became_peered":"2024-03-06T07:32:09.896787+0000","last_unstale":"2024-03-06T07:32:12.113007+0000","last_undegraded":"2024-03-06T07:32:12.113007+0000","last_fullsized":"2024-03-06T07:32:12.113007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.526095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904002+0000","last_change":"2024-03-06T07:32:10.897713+0000","last_active":"2024-03-06T07:32:11.904002+0000","last_peered":"2024-03-06T07:32:11.904002+0000","last_clean":"2024-03-06T07:32:11.904002+0000","last_became_active":"2024-03-06T07:32:10.897434+0000","last_became_peered":"2024-03-06T07:32:10.897434+0000","last_unstale":"2024-03-06T07:32:11.904002+0000","last_undegraded":"2024-03-06T07:32:11.904002+0000","last_fullsized":"2024-03-06T07:32:11.904002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:53.780886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907111+0000","last_change":"2024-03-06T07:32:09.913543+0000","last_active":"2024-03-06T07:32:11.907111+0000","last_peered":"2024-03-06T07:32:11.907111+0000","last_clean":"2024-03-06T07:32:11.907111+0000","last_became_active":"2024-03-06T07:32:09.913245+0000","last_became_peered":"2024-03-06T07:32:09.913245+0000","last_unstale":"2024-03-06T07:32:11.907111+0000","last_undegraded":"2024-03-06T07:32:11.907111+0000","last_fullsized":"2024-03-06T07:32:11.907111+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:00.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903321+0000","last_change":"2024-03-06T07:32:10.908986+0000","last_active":"2024-03-06T07:32:11.903321+0000","last_peered":"2024-03-06T07:32:11.903321+0000","last_clean":"2024-03-06T07:32:11.903321+0000","last_became_active":"2024-03-06T07:32:10.908630+0000","last_became_peered":"2024-03-06T07:32:10.908630+0000","last_unstale":"2024-03-06T07:32:11.903321+0000","last_undegraded":"2024-03-06T07:32:11.903321+0000","last_fullsized":"2024-03-06T07:32:11.903321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:45.579006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913431+0000","last_change":"2024-03-06T07:32:09.908341+0000","last_active":"2024-03-06T07:32:11.913431+0000","last_peered":"2024-03-06T07:32:11.913431+0000","last_clean":"2024-03-06T07:32:11.913431+0000","last_became_active":"2024-03-06T07:32:09.907997+0000","last_became_peered":"2024-03-06T07:32:09.907997+0000","last_unstale":"2024-03-06T07:32:11.913431+0000","last_undegraded":"2024-03-06T07:32:11.913431+0000","last_fullsized":"2024-03-06T07:32:11.913431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:20.461391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116239+0000","last_change":"2024-03-06T07:32:10.909405+0000","last_active":"2024-03-06T07:32:12.116239+0000","last_peered":"2024-03-06T07:32:12.116239+0000","last_clean":"2024-03-06T07:32:12.116239+0000","last_became_active":"2024-03-06T07:32:10.908470+0000","last_became_peered":"2024-03-06T07:32:10.908470+0000","last_unstale":"2024-03-06T07:32:12.116239+0000","last_undegraded":"2024-03-06T07:32:12.116239+0000","last_fullsized":"2024-03-06T07:32:12.116239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.814981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116746+0000","last_change":"2024-03-06T07:32:09.913931+0000","last_active":"2024-03-06T07:32:12.116746+0000","last_peered":"2024-03-06T07:32:12.116746+0000","last_clean":"2024-03-06T07:32:12.116746+0000","last_became_active":"2024-03-06T07:32:09.911819+0000","last_became_peered":"2024-03-06T07:32:09.911819+0000","last_unstale":"2024-03-06T07:32:12.116746+0000","last_undegraded":"2024-03-06T07:32:12.116746+0000","last_fullsized":"2024-03-06T07:32:12.116746+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:51.121062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116368+0000","last_change":"2024-03-06T07:32:10.905719+0000","last_active":"2024-03-06T07:32:12.116368+0000","last_peered":"2024-03-06T07:32:12.116368+0000","last_clean":"2024-03-06T07:32:12.116368+0000","last_became_active":"2024-03-06T07:32:10.905133+0000","last_became_peered":"2024-03-06T07:32:10.905133+0000","last_unstale":"2024-03-06T07:32:12.116368+0000","last_undegraded":"2024-03-06T07:32:12.116368+0000","last_fullsized":"2024-03-06T07:32:12.116368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:08.709841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.914136+0000","last_change":"2024-03-06T07:32:09.912732+0000","last_active":"2024-03-06T07:32:11.914136+0000","last_peered":"2024-03-06T07:32:11.914136+0000","last_clean":"2024-03-06T07:32:11.914136+0000","last_became_active":"2024-03-06T07:32:09.911917+0000","last_became_peered":"2024-03-06T07:32:09.911917+0000","last_unstale":"2024-03-06T07:32:11.914136+0000","last_undegraded":"2024-03-06T07:32:11.914136+0000","last_fullsized":"2024-03-06T07:32:11.914136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:00.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908934+0000","last_change":"2024-03-06T07:32:10.894290+0000","last_active":"2024-03-06T07:32:11.908934+0000","last_peered":"2024-03-06T07:32:11.908934+0000","last_clean":"2024-03-06T07:32:11.908934+0000","last_became_active":"2024-03-06T07:32:10.893827+0000","last_became_peered":"2024-03-06T07:32:10.893827+0000","last_unstale":"2024-03-06T07:32:11.908934+0000","last_undegraded":"2024-03-06T07:32:11.908934+0000","last_fullsized":"2024-03-06T07:32:11.908934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:00.949687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913933+0000","last_change":"2024-03-06T07:32:09.909723+0000","last_active":"2024-03-06T07:32:11.913933+0000","last_peered":"2024-03-06T07:32:11.913933+0000","last_clean":"2024-03-06T07:32:11.913933+0000","last_became_active":"2024-03-06T07:32:09.909407+0000","last_became_peered":"2024-03-06T07:32:09.909407+0000","last_unstale":"2024-03-06T07:32:11.913933+0000","last_undegraded":"2024-03-06T07:32:11.913933+0000","last_fullsized":"2024-03-06T07:32:11.913933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:05.488616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924750+0000","last_change":"2024-03-06T07:32:10.910025+0000","last_active":"2024-03-06T07:32:11.924750+0000","last_peered":"2024-03-06T07:32:11.924750+0000","last_clean":"2024-03-06T07:32:11.924750+0000","last_became_active":"2024-03-06T07:32:10.907190+0000","last_became_peered":"2024-03-06T07:32:10.907190+0000","last_unstale":"2024-03-06T07:32:11.924750+0000","last_undegraded":"2024-03-06T07:32:11.924750+0000","last_fullsized":"2024-03-06T07:32:11.924750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:48.909907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113024+0000","last_change":"2024-03-06T07:32:09.901839+0000","last_active":"2024-03-06T07:32:12.113024+0000","last_peered":"2024-03-06T07:32:12.113024+0000","last_clean":"2024-03-06T07:32:12.113024+0000","last_became_active":"2024-03-06T07:32:09.900748+0000","last_became_peered":"2024-03-06T07:32:09.900748+0000","last_unstale":"2024-03-06T07:32:12.113024+0000","last_undegraded":"2024-03-06T07:32:12.113024+0000","last_fullsized":"2024-03-06T07:32:12.113024+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:00.914204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907789+0000","last_change":"2024-03-06T07:32:10.900905+0000","last_active":"2024-03-06T07:32:11.907789+0000","last_peered":"2024-03-06T07:32:11.907789+0000","last_clean":"2024-03-06T07:32:11.907789+0000","last_became_active":"2024-03-06T07:32:10.900599+0000","last_became_peered":"2024-03-06T07:32:10.900599+0000","last_unstale":"2024-03-06T07:32:11.907789+0000","last_undegraded":"2024-03-06T07:32:11.907789+0000","last_fullsized":"2024-03-06T07:32:11.907789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:23.206762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924800+0000","last_change":"2024-03-06T07:32:09.903728+0000","last_active":"2024-03-06T07:32:11.924800+0000","last_peered":"2024-03-06T07:32:11.924800+0000","last_clean":"2024-03-06T07:32:11.924800+0000","last_became_active":"2024-03-06T07:32:09.903471+0000","last_became_peered":"2024-03-06T07:32:09.903471+0000","last_unstale":"2024-03-06T07:32:11.924800+0000","last_undegraded":"2024-03-06T07:32:11.924800+0000","last_fullsized":"2024-03-06T07:32:11.924800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:13.158368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924444+0000","last_change":"2024-03-06T07:32:10.900737+0000","last_active":"2024-03-06T07:32:11.924444+0000","last_peered":"2024-03-06T07:32:11.924444+0000","last_clean":"2024-03-06T07:32:11.924444+0000","last_became_active":"2024-03-06T07:32:10.900253+0000","last_became_peered":"2024-03-06T07:32:10.900253+0000","last_unstale":"2024-03-06T07:32:11.924444+0000","last_undegraded":"2024-03-06T07:32:11.924444+0000","last_fullsized":"2024-03-06T07:32:11.924444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.699432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.924271+0000","last_change":"2024-03-06T07:32:09.912789+0000","last_active":"2024-03-06T07:32:11.924271+0000","last_peered":"2024-03-06T07:32:11.924271+0000","last_clean":"2024-03-06T07:32:11.924271+0000","last_became_active":"2024-03-06T07:32:09.912514+0000","last_became_peered":"2024-03-06T07:32:09.912514+0000","last_unstale":"2024-03-06T07:32:11.924271+0000","last_undegraded":"2024-03-06T07:32:11.924271+0000","last_fullsized":"2024-03-06T07:32:11.924271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:32.127402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907194+0000","last_change":"2024-03-06T07:32:10.904858+0000","last_active":"2024-03-06T07:32:11.907194+0000","last_peered":"2024-03-06T07:32:11.907194+0000","last_clean":"2024-03-06T07:32:11.907194+0000","last_became_active":"2024-03-06T07:32:10.904518+0000","last_became_peered":"2024-03-06T07:32:10.904518+0000","last_unstale":"2024-03-06T07:32:11.907194+0000","last_undegraded":"2024-03-06T07:32:11.907194+0000","last_fullsized":"2024-03-06T07:32:11.907194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:47.967210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.112667+0000","last_change":"2024-03-06T07:32:09.903109+0000","last_active":"2024-03-06T07:32:12.112667+0000","last_peered":"2024-03-06T07:32:12.112667+0000","last_clean":"2024-03-06T07:32:12.112667+0000","last_became_active":"2024-03-06T07:32:09.900761+0000","last_became_peered":"2024-03-06T07:32:09.900761+0000","last_unstale":"2024-03-06T07:32:12.112667+0000","last_undegraded":"2024-03-06T07:32:12.112667+0000","last_fullsized":"2024-03-06T07:32:12.112667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:55.028253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.909220+0000","last_change":"2024-03-06T07:32:10.902532+0000","last_active":"2024-03-06T07:32:11.909220+0000","last_peered":"2024-03-06T07:32:11.909220+0000","last_clean":"2024-03-06T07:32:11.909220+0000","last_became_active":"2024-03-06T07:32:10.901454+0000","last_became_peered":"2024-03-06T07:32:10.901454+0000","last_unstale":"2024-03-06T07:32:11.909220+0000","last_undegraded":"2024-03-06T07:32:11.909220+0000","last_fullsized":"2024-03-06T07:32:11.909220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:03.712807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903901+0000","last_change":"2024-03-06T07:32:09.905750+0000","last_active":"2024-03-06T07:32:11.903901+0000","last_peered":"2024-03-06T07:32:11.903901+0000","last_clean":"2024-03-06T07:32:11.903901+0000","last_became_active":"2024-03-06T07:32:09.905447+0000","last_became_peered":"2024-03-06T07:32:09.905447+0000","last_unstale":"2024-03-06T07:32:11.903901+0000","last_undegraded":"2024-03-06T07:32:11.903901+0000","last_fullsized":"2024-03-06T07:32:11.903901+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:48.685552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.114622+0000","last_change":"2024-03-06T07:32:10.911773+0000","last_active":"2024-03-06T07:32:12.114622+0000","last_peered":"2024-03-06T07:32:12.114622+0000","last_clean":"2024-03-06T07:32:12.114622+0000","last_became_active":"2024-03-06T07:32:10.911251+0000","last_became_peered":"2024-03-06T07:32:10.911251+0000","last_unstale":"2024-03-06T07:32:12.114622+0000","last_undegraded":"2024-03-06T07:32:12.114622+0000","last_fullsized":"2024-03-06T07:32:12.114622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:24.539975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.903663+0000","last_change":"2024-03-06T07:32:09.900031+0000","last_active":"2024-03-06T07:32:11.903663+0000","last_peered":"2024-03-06T07:32:11.903663+0000","last_clean":"2024-03-06T07:32:11.903663+0000","last_became_active":"2024-03-06T07:32:09.896182+0000","last_became_peered":"2024-03-06T07:32:09.896182+0000","last_unstale":"2024-03-06T07:32:11.903663+0000","last_undegraded":"2024-03-06T07:32:11.903663+0000","last_fullsized":"2024-03-06T07:32:11.903663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.838640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116821+0000","last_change":"2024-03-06T07:32:10.914152+0000","last_active":"2024-03-06T07:32:12.116821+0000","last_peered":"2024-03-06T07:32:12.116821+0000","last_clean":"2024-03-06T07:32:12.116821+0000","last_became_active":"2024-03-06T07:32:10.913821+0000","last_became_peered":"2024-03-06T07:32:10.913821+0000","last_unstale":"2024-03-06T07:32:12.116821+0000","last_undegraded":"2024-03-06T07:32:12.116821+0000","last_fullsized":"2024-03-06T07:32:12.116821+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:34.056184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115648+0000","last_change":"2024-03-06T07:32:09.912911+0000","last_active":"2024-03-06T07:32:12.115648+0000","last_peered":"2024-03-06T07:32:12.115648+0000","last_clean":"2024-03-06T07:32:12.115648+0000","last_became_active":"2024-03-06T07:32:09.911917+0000","last_became_peered":"2024-03-06T07:32:09.911917+0000","last_unstale":"2024-03-06T07:32:12.115648+0000","last_undegraded":"2024-03-06T07:32:12.115648+0000","last_fullsized":"2024-03-06T07:32:12.115648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.877134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.908972+0000","last_change":"2024-03-06T07:32:10.900492+0000","last_active":"2024-03-06T07:32:11.908972+0000","last_peered":"2024-03-06T07:32:11.908972+0000","last_clean":"2024-03-06T07:32:11.908972+0000","last_became_active":"2024-03-06T07:32:10.900174+0000","last_became_peered":"2024-03-06T07:32:10.900174+0000","last_unstale":"2024-03-06T07:32:11.908972+0000","last_undegraded":"2024-03-06T07:32:11.908972+0000","last_fullsized":"2024-03-06T07:32:11.908972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:06.169996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.116575+0000","last_change":"2024-03-06T07:32:09.914488+0000","last_active":"2024-03-06T07:32:12.116575+0000","last_peered":"2024-03-06T07:32:12.116575+0000","last_clean":"2024-03-06T07:32:12.116575+0000","last_became_active":"2024-03-06T07:32:09.912735+0000","last_became_peered":"2024-03-06T07:32:09.912735+0000","last_unstale":"2024-03-06T07:32:12.116575+0000","last_undegraded":"2024-03-06T07:32:12.116575+0000","last_fullsized":"2024-03-06T07:32:12.116575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:56.076163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907605+0000","last_change":"2024-03-06T07:32:10.898238+0000","last_active":"2024-03-06T07:32:11.907605+0000","last_peered":"2024-03-06T07:32:11.907605+0000","last_clean":"2024-03-06T07:32:11.907605+0000","last_became_active":"2024-03-06T07:32:10.897914+0000","last_became_peered":"2024-03-06T07:32:10.897914+0000","last_unstale":"2024-03-06T07:32:11.907605+0000","last_undegraded":"2024-03-06T07:32:11.907605+0000","last_fullsized":"2024-03-06T07:32:11.907605+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:41.770080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913914+0000","last_change":"2024-03-06T07:32:09.905630+0000","last_active":"2024-03-06T07:32:11.913914+0000","last_peered":"2024-03-06T07:32:11.913914+0000","last_clean":"2024-03-06T07:32:11.913914+0000","last_became_active":"2024-03-06T07:32:09.905353+0000","last_became_peered":"2024-03-06T07:32:09.905353+0000","last_unstale":"2024-03-06T07:32:11.913914+0000","last_undegraded":"2024-03-06T07:32:11.913914+0000","last_fullsized":"2024-03-06T07:32:11.913914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:05.489460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.913942+0000","last_change":"2024-03-06T07:32:10.908036+0000","last_active":"2024-03-06T07:32:11.913942+0000","last_peered":"2024-03-06T07:32:11.913942+0000","last_clean":"2024-03-06T07:32:11.913942+0000","last_became_active":"2024-03-06T07:32:10.907545+0000","last_became_peered":"2024-03-06T07:32:10.907545+0000","last_unstale":"2024-03-06T07:32:11.913942+0000","last_undegraded":"2024-03-06T07:32:11.913942+0000","last_fullsized":"2024-03-06T07:32:11.913942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:23.206762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.907108+0000","last_change":"2024-03-06T07:32:09.902786+0000","last_active":"2024-03-06T07:32:11.907108+0000","last_peered":"2024-03-06T07:32:11.907108+0000","last_clean":"2024-03-06T07:32:11.907108+0000","last_became_active":"2024-03-06T07:32:09.902392+0000","last_became_peered":"2024-03-06T07:32:09.902392+0000","last_unstale":"2024-03-06T07:32:11.907108+0000","last_undegraded":"2024-03-06T07:32:11.907108+0000","last_fullsized":"2024-03-06T07:32:11.907108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:30.889092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113956+0000","last_change":"2024-03-06T07:32:10.916279+0000","last_active":"2024-03-06T07:32:12.113956+0000","last_peered":"2024-03-06T07:32:12.113956+0000","last_clean":"2024-03-06T07:32:12.113956+0000","last_became_active":"2024-03-06T07:32:10.915695+0000","last_became_peered":"2024-03-06T07:32:10.915695+0000","last_unstale":"2024-03-06T07:32:12.113956+0000","last_undegraded":"2024-03-06T07:32:12.113956+0000","last_fullsized":"2024-03-06T07:32:12.113956+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:43.352252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115943+0000","last_change":"2024-03-06T07:32:09.910421+0000","last_active":"2024-03-06T07:32:12.115943+0000","last_peered":"2024-03-06T07:32:12.115943+0000","last_clean":"2024-03-06T07:32:12.115943+0000","last_became_active":"2024-03-06T07:32:09.910102+0000","last_became_peered":"2024-03-06T07:32:09.910102+0000","last_unstale":"2024-03-06T07:32:12.115943+0000","last_undegraded":"2024-03-06T07:32:12.115943+0000","last_fullsized":"2024-03-06T07:32:12.115943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:00.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115674+0000","last_change":"2024-03-06T07:32:10.912648+0000","last_active":"2024-03-06T07:32:12.115674+0000","last_peered":"2024-03-06T07:32:12.115674+0000","last_clean":"2024-03-06T07:32:12.115674+0000","last_became_active":"2024-03-06T07:32:10.912354+0000","last_became_peered":"2024-03-06T07:32:10.912354+0000","last_unstale":"2024-03-06T07:32:12.115674+0000","last_undegraded":"2024-03-06T07:32:12.115674+0000","last_fullsized":"2024-03-06T07:32:12.115674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:01.901970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.906512+0000","last_change":"2024-03-06T07:32:09.914162+0000","last_active":"2024-03-06T07:32:11.906512+0000","last_peered":"2024-03-06T07:32:11.906512+0000","last_clean":"2024-03-06T07:32:11.906512+0000","last_became_active":"2024-03-06T07:32:09.913255+0000","last_became_peered":"2024-03-06T07:32:09.913255+0000","last_unstale":"2024-03-06T07:32:11.906512+0000","last_undegraded":"2024-03-06T07:32:11.906512+0000","last_fullsized":"2024-03-06T07:32:11.906512+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:02.681729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:11.904617+0000","last_change":"2024-03-06T07:32:10.900210+0000","last_active":"2024-03-06T07:32:11.904617+0000","last_peered":"2024-03-06T07:32:11.904617+0000","last_clean":"2024-03-06T07:32:11.904617+0000","last_became_active":"2024-03-06T07:32:10.899475+0000","last_became_peered":"2024-03-06T07:32:10.899475+0000","last_unstale":"2024-03-06T07:32:11.904617+0000","last_undegraded":"2024-03-06T07:32:11.904617+0000","last_fullsized":"2024-03-06T07:32:11.904617+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:09.859915+0000","last_clean_scrub_stamp":"2024-03-06T07:32:09.859915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:57.090043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.115609+0000","last_change":"2024-03-06T07:32:09.915425+0000","last_active":"2024-03-06T07:32:12.115609+0000","last_peered":"2024-03-06T07:32:12.115609+0000","last_clean":"2024-03-06T07:32:12.115609+0000","last_became_active":"2024-03-06T07:32:09.915111+0000","last_became_peered":"2024-03-06T07:32:09.915111+0000","last_unstale":"2024-03-06T07:32:12.115609+0000","last_undegraded":"2024-03-06T07:32:12.115609+0000","last_fullsized":"2024-03-06T07:32:12.115609+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:08.846036+0000","last_clean_scrub_stamp":"2024-03-06T07:32:08.846036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:51.607293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":283,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:32:12.113985+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:32:12.113985+0000","last_peered":"2024-03-06T07:32:12.113985+0000","last_clean":"2024-03-06T07:32:12.113985+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:32:12.113985+0000","last_undegraded":"2024-03-06T07:32:12.113985+0000","last_fullsized":"2024-03-06T07:32:12.113985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542202,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1015787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542203,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1015808,"data_stored":426043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":426983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542203,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":425349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542202,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1015808,"data_stored":426961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542204,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":425931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1015808,"data_stored":425543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542203,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1601536,"data_stored":1015751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:15.846 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:32:15.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:32:16.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.100+0000 7f3596800640 1 Processor -- start 2024-03-06T07:32:16.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.100+0000 7f3596800640 1 -- start start 2024-03-06T07:32:16.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f3596800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f35900d07f0 0x7f35900d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f3596800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f35900d1bf0 0x7f35900cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f3596800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900b3730 con 0x7f359015e450 2024-03-06T07:32:16.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f3596800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35900d4930 con 0x7f35900d07f0 2024-03-06T07:32:16.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f3596800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3590067b20 con 0x7f35900d1bf0 2024-03-06T07:32:16.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f358f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f35900d07f0 0x7f35900d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f358f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f35900d07f0 0x7f35900d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38676/0 (socket says 172.21.15.97:38676) 2024-03-06T07:32:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.102+0000 7f358f400640 1 -- 172.21.15.97:0/3599652459 learned_addr learned my addr 172.21.15.97:0/3599652459 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.103+0000 7f358f400640 1 -- 172.21.15.97:0/3599652459 >> v2:172.21.15.135:3300/0 conn(0x7f35900d1bf0 msgr2=0x7f35900cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.103+0000 7f358f400640 1 --2- 172.21.15.97:0/3599652459 >> v2:172.21.15.135:3300/0 conn(0x7f35900d1bf0 0x7f35900cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.103+0000 7f358f400640 1 -- 172.21.15.97:0/3599652459 >> v1:172.21.15.135:6789/0 conn(0x7f359015e450 legacy=0x7f35900b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.103+0000 7f358f400640 1 -- 172.21.15.97:0/3599652459 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900d1110 con 0x7f35900d07f0 2024-03-06T07:32:16.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.104+0000 7f358f400640 1 --2- 172.21.15.97:0/3599652459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f35900d0bd0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f35900cae50 tx=0x7f358002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f191ad4d6038b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.104+0000 7f358ea00640 1 -- 172.21.15.97:0/3599652459 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358002e070 con 0x7f35900d07f0 2024-03-06T07:32:16.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.104+0000 7f358ea00640 1 -- 172.21.15.97:0/3599652459 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3580037070 con 0x7f35900d07f0 2024-03-06T07:32:16.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.104+0000 7f358ea00640 1 -- 172.21.15.97:0/3599652459 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580031a00 con 0x7f35900d07f0 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 -- 172.21.15.97:0/3599652459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 msgr2=0x7f35900d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 --2- 172.21.15.97:0/3599652459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f35900d0bd0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f35900cae50 tx=0x7f358002b880 comp rx=0 tx=0).stop 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 -- 172.21.15.97:0/3599652459 shutdown_connections 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 --2- 172.21.15.97:0/3599652459 >> v2:172.21.15.135:3300/0 conn(0x7f35900d1bf0 0x7f35900cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 --2- 172.21.15.97:0/3599652459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f35900d0bd0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 -- 172.21.15.97:0/3599652459 >> 172.21.15.97:0/3599652459 conn(0x7f35900d3e10 msgr2=0x7f35900d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 -- 172.21.15.97:0/3599652459 shutdown_connections 2024-03-06T07:32:16.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.105+0000 7f3596800640 1 -- 172.21.15.97:0/3599652459 wait complete. 2024-03-06T07:32:16.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.107+0000 7f3596800640 1 Processor -- start 2024-03-06T07:32:16.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.107+0000 7f3596800640 1 -- start start 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f3596800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f3590150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f3596800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35900d1bf0 0x7f35900b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f3596800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f359015e450 0x7f35900b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f3596800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900b3c80 con 0x7f35900d07f0 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f3596800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900b39a0 con 0x7f35900d1bf0 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f3596800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35900b3f60 con 0x7f359015e450 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.108+0000 7f358f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f3590150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f3590150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38682/0 (socket says 172.21.15.97:38682) 2024-03-06T07:32:16.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 -- 172.21.15.97:0/3782105458 learned_addr learned my addr 172.21.15.97:0/3782105458 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:16.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 -- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f359015e450 msgr2=0x7f35900b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358fe00640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f359015e450 0x7f35900b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f3587e00640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35900d1bf0 0x7f35900b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f359015e450 0x7f35900b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 -- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35900d1bf0 msgr2=0x7f35900b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35900d1bf0 0x7f35900b6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 -- 172.21.15.97:0/3782105458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359014ddc0 con 0x7f35900d07f0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358fe00640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f359015e450 0x7f35900b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.109+0000 7f358f400640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f3590150950 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3580000c00 tx=0x7f35800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.110+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580032880 con 0x7f35900d07f0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.110+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3580032a20 con 0x7f35900d07f0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.110+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580032e10 con 0x7f35900d07f0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.110+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359014e0a0 con 0x7f35900d07f0 2024-03-06T07:32:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.110+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359014e660 con 0x7f35900d07f0 2024-03-06T07:32:16.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.111+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359004f330 con 0x7f35900d07f0 2024-03-06T07:32:16.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.112+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f358004c060 con 0x7f35900d07f0 2024-03-06T07:32:16.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.113+0000 7f3587400640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f356c06b4d0 0x7f356c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.113+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35800aba20 con 0x7f35900d07f0 2024-03-06T07:32:16.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.113+0000 7f3587e00640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f356c06b4d0 0x7f356c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.114+0000 7f3587e00640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f356c06b4d0 0x7f356c06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f35901f70d0 tx=0x7f3578008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.122+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3580040070 con 0x7f35900d07f0 2024-03-06T07:32:16.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.385+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f35900b1870 con 0x7f35900d07f0 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.389+0000 7f3587400640 1 -- 172.21.15.97:0/3782105458 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f358007a8a0 con 0x7f35900d07f0 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:32:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f356c06b4d0 msgr2=0x7f356c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f356c06b4d0 0x7f356c06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f35901f70d0 tx=0x7f3578008040 comp rx=0 tx=0).stop 2024-03-06T07:32:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 msgr2=0x7f3590150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f3590150950 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3580000c00 tx=0x7f35800046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 shutdown_connections 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f356c06b4d0 0x7f356c06d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f359015e450 0x7f35900b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35900d1bf0 0x7f35900b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 --2- 172.21.15.97:0/3782105458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f35900d07f0 0x7f3590150950 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 >> 172.21.15.97:0/3782105458 conn(0x7f35900d3e10 msgr2=0x7f359006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 shutdown_connections 2024-03-06T07:32:16.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.391+0000 7f3596800640 1 -- 172.21.15.97:0/3782105458 wait complete. 2024-03-06T07:32:16.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.605+0000 7f3094600640 1 Processor -- start 2024-03-06T07:32:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.605+0000 7f3094600640 1 -- start start 2024-03-06T07:32:16.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.606+0000 7f3094600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 0x7f308c0b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.606+0000 7f3094600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c0b5e00 con 0x7f308c0cff20 2024-03-06T07:32:16.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f3094600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c0b60e0 con 0x7f308c0b5020 2024-03-06T07:32:16.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f3094600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0b5bc0 con 0x7f308c0b75b0 2024-03-06T07:32:16.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f3091c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f308c0cff20 0x7f308c0b6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59494/0 (socket says 172.21.15.97:59494) 2024-03-06T07:32:16.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f3091c00640 1 -- 172.21.15.97:0/2391471492 learned_addr learned my addr 172.21.15.97:0/2391471492 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f3091200640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 0x7f308c0b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846196827 0 0) 0x7f308c0b5e00 con 0x7f308c0cff20 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.607+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3074003520 con 0x7f308c0cff20 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2076068140 0 0) 0x7f3074003520 con 0x7f308c0cff20 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 msgr2=0x7f308c0b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 0x7f308c0b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 >> v1:172.21.15.135:6789/0 conn(0x7f308c0b5020 legacy=0x7f308c0b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c0b63c0 con 0x7f308c0cff20 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3080001670 con 0x7f308c0cff20 2024-03-06T07:32:16.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 >> v1:172.21.15.97:6789/0 conn(0x7f308c0cff20 legacy=0x7f308c0b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 0x7f30740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3074003f60 0x7f3074004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074004d40 0x7f30740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0b5bc0 con 0x7f3074003f60 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307400bb50 con 0x7f3074004d40 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307400bd50 con 0x7f308c0b75b0 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.608+0000 7f3091200640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3074003f60 0x7f3074004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3091c00640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 0x7f30740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3092600640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074004d40 0x7f30740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3091200640 1 -- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 msgr2=0x7f30740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3091200640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c0b75b0 0x7f30740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3091200640 1 -- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074004d40 msgr2=0x7f30740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3091200640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074004d40 0x7f30740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3091200640 1 -- 172.21.15.97:0/2391471492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3074003520 con 0x7f3074003f60 2024-03-06T07:32:16.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.609+0000 7f3092600640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074004d40 0x7f30740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.610+0000 7f3091200640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3074003f60 0x7f3074004800 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f3078009ac0 tx=0x7f3074005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad194a2efa483d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.610+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078039070 con 0x7f3074003f60 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.610+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307802c6b0 con 0x7f3074003f60 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.610+0000 7f308be00640 1 -- 172.21.15.97:0/2391471492 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078009690 con 0x7f3074003f60 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3094600640 1 -- 172.21.15.97:0/2391471492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3074003f60 msgr2=0x7f3074004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3094600640 1 --2- 172.21.15.97:0/2391471492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3074003f60 0x7f3074004800 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f3078009ac0 tx=0x7f3074005830 comp rx=0 tx=0).stop 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3092600640 1 -- 172.21.15.97:0/2391471492 reap_dead start 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3094600640 1 -- 172.21.15.97:0/2391471492 shutdown_connections 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3094600640 1 -- 172.21.15.97:0/2391471492 >> 172.21.15.97:0/2391471492 conn(0x7f308c0d1c90 msgr2=0x7f308c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3094600640 1 -- 172.21.15.97:0/2391471492 shutdown_connections 2024-03-06T07:32:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.611+0000 7f3094600640 1 -- 172.21.15.97:0/2391471492 wait complete. 2024-03-06T07:32:16.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.613+0000 7f3094600640 1 Processor -- start 2024-03-06T07:32:16.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.613+0000 7f3094600640 1 -- start start 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3094600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074003f60 0x7f308c151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3094600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 0x7f308c0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3094600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c15bed0 0x7f308c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3094600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c161a50 con 0x7f308c15ba50 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3094600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0b6a10 con 0x7f3074003f60 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3094600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0b7a30 con 0x7f308c15bed0 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 0x7f308c0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 0x7f308c0cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38708/0 (socket says 172.21.15.97:38708) 2024-03-06T07:32:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 -- 172.21.15.97:0/451992785 learned_addr learned my addr 172.21.15.97:0/451992785 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3092600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c15bed0 0x7f308c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091c00640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074003f60 0x7f308c151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 -- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c15bed0 msgr2=0x7f308c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c15bed0 0x7f308c0cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 -- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074003f60 msgr2=0x7f308c151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074003f60 0x7f308c151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091200640 1 -- 172.21.15.97:0/451992785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c15ac90 con 0x7f308c15ba50 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.614+0000 7f3091c00640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074003f60 0x7f308c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.615+0000 7f3091200640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 0x7f308c0cab00 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3078000c00 tx=0x7f3078004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.615+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078043020 con 0x7f308c15ba50 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.615+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307802cd90 con 0x7f308c15ba50 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.615+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c15af70 con 0x7f308c15ba50 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.615+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078039040 con 0x7f308c15ba50 2024-03-06T07:32:16.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.615+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c15b530 con 0x7f308c15ba50 2024-03-06T07:32:16.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.616+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c04f330 con 0x7f308c15ba50 2024-03-06T07:32:16.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.617+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3078046020 con 0x7f308c15ba50 2024-03-06T07:32:16.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.618+0000 7f3089600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f306c06b4d0 0x7f306c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.618+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30780abc00 con 0x7f308c15ba50 2024-03-06T07:32:16.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.618+0000 7f3091c00640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f306c06b4d0 0x7f306c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.619+0000 7f3091c00640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f306c06b4d0 0x7f306c06d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3080001eb0 tx=0x7f3080001d80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.623+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f307804d070 con 0x7f308c15ba50 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.857+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f308c0684a0 con 0x7f308c15ba50 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.858+0000 7f3089600640 1 -- 172.21.15.97:0/451992785 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f307807a9e0 con 0x7f308c15ba50 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.860+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f306c06b4d0 msgr2=0x7f306c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.860+0000 7f3094600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f306c06b4d0 0x7f306c06d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3080001eb0 tx=0x7f3080001d80 comp rx=0 tx=0).stop 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.860+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 msgr2=0x7f308c0cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.860+0000 7f3094600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 0x7f308c0cab00 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3078000c00 tx=0x7f3078004730 comp rx=0 tx=0).stop 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 shutdown_connections 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f306c06b4d0 0x7f306c06d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f308c15bed0 0x7f308c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f308c15ba50 0x7f308c0cab00 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 --2- 172.21.15.97:0/451992785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3074003f60 0x7f308c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 >> 172.21.15.97:0/451992785 conn(0x7f308c0d1c90 msgr2=0x7f308c06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 shutdown_connections 2024-03-06T07:32:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:16.861+0000 7f3094600640 1 -- 172.21.15.97:0/451992785 wait complete. 2024-03-06T07:32:17.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:10.358830+0000', 'modified': '2024-03-06T07:32:13.477240+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6741}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6741': {'gid': 6741, 'name': 'a', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3414007063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3414007063}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3414007063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6741, 'qdb_cluster': [6741]} 2024-03-06T07:32:17.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:17.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:32:17.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.495+0000 7f7743000640 1 Processor -- start 2024-03-06T07:32:17.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.496+0000 7f7743000640 1 -- start start 2024-03-06T07:32:17.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.497+0000 7f7743000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f773c15ac90 0x7f773c06e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.497+0000 7f7743000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f773c15b070 0x7f773c15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.497+0000 7f7743000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0cc000 con 0x7f773c15ef50 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f7743000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c0ce360 con 0x7f773c15ac90 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f7743000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c0ce640 con 0x7f773c15b070 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f773be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f773c15ac90 0x7f773c06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f7741000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f773c15ef50 0x7f773c0cac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59508/0 (socket says 172.21.15.97:59508) 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f773be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f773c15ac90 0x7f773c06e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38718/0 (socket says 172.21.15.97:38718) 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f7741000640 1 -- 172.21.15.97:0/2336210846 learned_addr learned my addr 172.21.15.97:0/2336210846 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f7733e00640 1 --2- 172.21.15.97:0/2336210846 >> v2:172.21.15.135:3300/0 conn(0x7f773c15b070 0x7f773c15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f773be00640 1 -- 172.21.15.97:0/2336210846 >> v2:172.21.15.135:3300/0 conn(0x7f773c15b070 msgr2=0x7f773c15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f773be00640 1 --2- 172.21.15.97:0/2336210846 >> v2:172.21.15.135:3300/0 conn(0x7f773c15b070 0x7f773c15ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f773be00640 1 -- 172.21.15.97:0/2336210846 >> v1:172.21.15.97:6789/0 conn(0x7f773c15ef50 legacy=0x7f773c0cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f773be00640 1 -- 172.21.15.97:0/2336210846 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0ce920 con 0x7f773c15ac90 2024-03-06T07:32:17.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.498+0000 7f7733e00640 1 --2- 172.21.15.97:0/2336210846 >> v2:172.21.15.135:3300/0 conn(0x7f773c15b070 0x7f773c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:17.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.499+0000 7f773be00640 1 --2- 172.21.15.97:0/2336210846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ac90 0x7f773c06e580 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f773c075020 tx=0x7f772c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82080597b13305c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.500+0000 7f773b400640 1 -- 172.21.15.97:0/2336210846 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c0315a0 con 0x7f773c15ac90 2024-03-06T07:32:17.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.500+0000 7f773b400640 1 -- 172.21.15.97:0/2336210846 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c037070 con 0x7f773c15ac90 2024-03-06T07:32:17.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.500+0000 7f773b400640 1 -- 172.21.15.97:0/2336210846 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c031960 con 0x7f773c15ac90 2024-03-06T07:32:17.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.500+0000 7f7743000640 1 -- 172.21.15.97:0/2336210846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ac90 msgr2=0x7f773c06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.500+0000 7f7743000640 1 --2- 172.21.15.97:0/2336210846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ac90 0x7f773c06e580 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f773c075020 tx=0x7f772c02b840 comp rx=0 tx=0).stop 2024-03-06T07:32:17.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.501+0000 7f7743000640 1 -- 172.21.15.97:0/2336210846 shutdown_connections 2024-03-06T07:32:17.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.501+0000 7f7743000640 1 --2- 172.21.15.97:0/2336210846 >> v2:172.21.15.135:3300/0 conn(0x7f773c15b070 0x7f773c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.501+0000 7f7743000640 1 --2- 172.21.15.97:0/2336210846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ac90 0x7f773c06e580 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.501+0000 7f7743000640 1 -- 172.21.15.97:0/2336210846 >> 172.21.15.97:0/2336210846 conn(0x7f773c0cc4f0 msgr2=0x7f773c0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.501+0000 7f7743000640 1 -- 172.21.15.97:0/2336210846 shutdown_connections 2024-03-06T07:32:17.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.501+0000 7f7743000640 1 -- 172.21.15.97:0/2336210846 wait complete. 2024-03-06T07:32:17.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.503+0000 7f7743000640 1 Processor -- start 2024-03-06T07:32:17.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.504+0000 7f7743000640 1 -- start start 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7743000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15ac90 0x7f773c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7743000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c15b070 0x7f773c0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7743000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 0x7f773c15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7743000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0c5a20 con 0x7f773c15ef50 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7743000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0ceee0 con 0x7f773c15ac90 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7743000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0c5ba0 con 0x7f773c15b070 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7741000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 0x7f773c15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7741000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 0x7f773c15c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38722/0 (socket says 172.21.15.97:38722) 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.505+0000 7f7741000640 1 -- 172.21.15.97:0/3593344282 learned_addr learned my addr 172.21.15.97:0/3593344282 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f773be00640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15ac90 0x7f773c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7733e00640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c15b070 0x7f773c0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7741000640 1 -- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c15b070 msgr2=0x7f773c0c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7741000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c15b070 0x7f773c0c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7741000640 1 -- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15ac90 msgr2=0x7f773c0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7741000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15ac90 0x7f773c0a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7741000640 1 -- 172.21.15.97:0/3593344282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0a9990 con 0x7f773c15ef50 2024-03-06T07:32:17.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7741000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 0x7f773c15c4c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f773400ca70 tx=0x7f773400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.506+0000 7f7738c00640 1 -- 172.21.15.97:0/3593344282 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734017070 con 0x7f773c15ef50 2024-03-06T07:32:17.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.507+0000 7f7738c00640 1 -- 172.21.15.97:0/3593344282 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773400abd0 con 0x7f773c15ef50 2024-03-06T07:32:17.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.507+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c0c8140 con 0x7f773c15ef50 2024-03-06T07:32:17.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.507+0000 7f7738c00640 1 -- 172.21.15.97:0/3593344282 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734012400 con 0x7f773c15ef50 2024-03-06T07:32:17.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.507+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c0c8700 con 0x7f773c15ef50 2024-03-06T07:32:17.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.509+0000 7f7738c00640 1 -- 172.21.15.97:0/3593344282 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f773401a070 con 0x7f773c15ef50 2024-03-06T07:32:17.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.510+0000 7f7745df6740 1 Processor -- start 2024-03-06T07:32:17.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.510+0000 7f7738c00640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b500 0x7f771806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.510+0000 7f7745df6740 1 -- start start 2024-03-06T07:32:17.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.511+0000 7f7738c00640 1 -- 172.21.15.97:0/3593344282 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7734089d90 con 0x7f773c15ef50 2024-03-06T07:32:17.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.511+0000 7f7733e00640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b500 0x7f771806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.511+0000 7f7733e00640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b500 0x7f771806d9c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f773c0cb740 tx=0x7f7724008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b347b0 0x555b12b07750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12b37540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7745df6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 0x555b12b37a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7745df6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b12a71600 con 0x555b12b07c90 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b12a1a930 con 0x555b12b347b0 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b12a6d6a0 con 0x555b12b34b90 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 0x555b12b37a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 0x555b12b37a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38732/0 (socket says 172.21.15.97:38732) 2024-03-06T07:32:17.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 -- 172.21.15.97:0/4115407945 learned_addr learned my addr 172.21.15.97:0/4115407945 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:17.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 -- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 msgr2=0x555b12b37540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12b37540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 -- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b347b0 msgr2=0x555b12b07750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f773be00640 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b347b0 0x555b12b07750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.512+0000 7f7741000640 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b347b0 0x555b12b07750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.513+0000 7f7741000640 1 -- 172.21.15.97:0/4115407945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b127862d0 con 0x555b12b07c90 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.513+0000 7f773be00640 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b347b0 0x555b12b07750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.513+0000 7f7741000640 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 0x555b12b37a80 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f77340606e0 tx=0x7f7734058220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09623a2ffa60c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.513+0000 7f7730c00640 1 -- 172.21.15.97:0/4115407945 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734054c20 con 0x555b12b07c90 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.513+0000 7f7730c00640 1 -- 172.21.15.97:0/4115407945 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7734060780 con 0x555b12b07c90 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.513+0000 7f7730c00640 1 -- 172.21.15.97:0/4115407945 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734054dc0 con 0x555b12b07c90 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 -- 172.21.15.97:0/4115407945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 msgr2=0x555b12b37a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 0x555b12b37a80 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f77340606e0 tx=0x7f7734058220 comp rx=0 tx=0).stop 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 -- 172.21.15.97:0/4115407945 shutdown_connections 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b07c90 0x555b12b37a80 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12b37540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 --2- 172.21.15.97:0/4115407945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b347b0 0x555b12b07750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 -- 172.21.15.97:0/4115407945 >> 172.21.15.97:0/4115407945 conn(0x555b12b099c0 msgr2=0x555b12b52540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 -- 172.21.15.97:0/4115407945 shutdown_connections 2024-03-06T07:32:17.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.514+0000 7f7745df6740 1 -- 172.21.15.97:0/4115407945 wait complete. 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.515+0000 7f7745df6740 1 -- start start 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.515+0000 7f7745df6740 1 Processor -- start 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.516+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b347b0 0x555b12b27c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.516+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b39880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7745df6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 0x555b12b4be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7745df6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b12b0bf60 con 0x555b12b4b9a0 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b12a2dd90 con 0x555b12b34b90 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b12b0b350 con 0x555b12b347b0 2024-03-06T07:32:17.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7745df6740 10 client.0 _ll_register_callbacks cb 0x555b12a69350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 0x555b12b4be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 0x555b12b4be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38736/0 (socket says 172.21.15.97:38736) 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 -- 172.21.15.97:0/2767109139 learned_addr learned my addr 172.21.15.97:0/2767109139 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f773be00640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b347b0 0x555b12b27c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7733e00640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b39880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 -- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b347b0 msgr2=0x555b12b27c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b347b0 0x555b12b27c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 -- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 msgr2=0x555b12b39880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b39880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7741000640 1 -- 172.21.15.97:0/2767109139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b128506b0 con 0x555b12b4b9a0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.517+0000 7f7733e00640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b39880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7741000640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 0x555b12b4be20 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f7734005c20 tx=0x7f7734088180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43dc442b54cd109 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7729600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7729600640 1 -- 172.21.15.97:0/2767109139 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734086060 con 0x555b12b4b9a0 2024-03-06T07:32:17.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7729600640 1 -- 172.21.15.97:0/2767109139 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7734004b60 con 0x555b12b4b9a0 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7734060780 con 0x555b12b4b9a0 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7729600640 1 -- 172.21.15.97:0/2767109139 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7734004d60 con 0x555b12b4b9a0 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7745df6740 10 client.6885 fetch_fsmap learned FSMap version 51 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x555b12b639f0 con 0x555b12b4b9a0 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.518+0000 7f7729600640 1 -- 172.21.15.97:0/2767109139 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7734060780 con 0x555b12b4b9a0 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7729600640 1 -- 172.21.15.97:0/2767109139 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 51) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7734057a00 con 0x555b12b4b9a0 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7745df6740 10 client.6885 fetch_fsmap finished waiting for FSMap version 51 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7745df6740 10 client.6885 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7745df6740 20 client.6885 populate_metadata read hostname 'smithi097' 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7745df6740 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b62d40 0x555b12b63120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7745df6740 4 client.6885 mds_command: new command op to 6741 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 --> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b12a7eb40 con 0x555b12b62d40 2024-03-06T07:32:17.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.519+0000 7f773be00640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b62d40 0x555b12b63120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.520+0000 7f7729600640 10 client.6885 ms_handle_connect on v2:172.21.15.97:6835/3414007063 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.520+0000 7f773be00640 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b62d40 0x555b12b63120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0434df8045dc764 server_cookie=708a5e21d62c5783 in_seq=0 out_seq=0 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.520+0000 7f7729600640 1 -- 172.21.15.97:0/2767109139 <== mds.0 v2:172.21.15.97:6835/3414007063 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555b12a7eb40 con 0x555b12b62d40 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7729600640 10 client.6885 handle_command_reply: tid=0 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 client.6885 shutdown 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 msgr2=0x555b12b4be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 0x555b12b4be20 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f7734005c20 tx=0x7f7734088180 comp rx=0 tx=0).stop 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 shutdown_connections 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b62d40 0x555b12b63120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b4b9a0 0x555b12b4be20 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b39880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 --2- 172.21.15.97:0/2767109139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b347b0 0x555b12b27c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7729600640 0 client.6885 ms_handle_reset on v2:172.21.15.97:6835/3414007063 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 >> 172.21.15.97:0/2767109139 conn(0x555b12b07c90 msgr2=0x555b12b37530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 shutdown_connections 2024-03-06T07:32:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 1 -- 172.21.15.97:0/2767109139 wait complete. 2024-03-06T07:32:17.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.521+0000 7f7745df6740 20 client.6885 trim_cache size 0 max 16384 2024-03-06T07:32:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.544+0000 7f7745df6740 1 Processor -- start 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.545+0000 7f7745df6740 1 -- start start 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f7745df6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12b0e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b06810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b4b9a0 0x555b12b38630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f7745df6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0c7850 con 0x555b12b347b0 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0c6c50 con 0x555b12b34b90 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0c7250 con 0x555b12b4b9a0 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f773be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12b0e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f773be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12b0e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38750/0 (socket says 172.21.15.97:38750) 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.546+0000 7f773be00640 1 -- 172.21.15.97:0/2649304901 learned_addr learned my addr 172.21.15.97:0/2649304901 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:17.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f7741000640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b4b9a0 0x555b12b38630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f7733e00640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b06810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f773be00640 1 -- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b4b9a0 msgr2=0x555b12b38630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f773be00640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b4b9a0 0x555b12b38630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f773be00640 1 -- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 msgr2=0x555b12b06810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f773be00640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b06810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f773be00640 1 -- 172.21.15.97:0/2649304901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b12b062c0 con 0x555b12b347b0 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f7741000640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b4b9a0 0x555b12b38630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f7733e00640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b06810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.547+0000 7f773be00640 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12b0e770 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f772c034040 tx=0x7f772c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f05647cd9fbb6aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f772be00640 1 -- 172.21.15.97:0/2649304901 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c04f030 con 0x555b12b347b0 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f772be00640 1 -- 172.21.15.97:0/2649304901 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c037040 con 0x555b12b347b0 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f772be00640 1 -- 172.21.15.97:0/2649304901 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c04a170 con 0x555b12b347b0 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 -- 172.21.15.97:0/2649304901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 msgr2=0x555b12b0e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12b0e770 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f772c034040 tx=0x7f772c030000 comp rx=0 tx=0).stop 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 -- 172.21.15.97:0/2649304901 shutdown_connections 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b4b9a0 0x555b12b38630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b34b90 0x555b12b06810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 --2- 172.21.15.97:0/2649304901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12b0e770 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 -- 172.21.15.97:0/2649304901 >> 172.21.15.97:0/2649304901 conn(0x555b12b07c90 msgr2=0x555b12b61ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 -- 172.21.15.97:0/2649304901 shutdown_connections 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.548+0000 7f7745df6740 1 -- 172.21.15.97:0/2649304901 wait complete. 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.549+0000 7f7745df6740 1 -- start start 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.549+0000 7f7745df6740 1 Processor -- start 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12bc1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12bc23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b4b9a0 0x555b12b391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b12b0bf60 con 0x555b12b347b0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b12b0b350 con 0x555b12b4b9a0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b12a2dd90 con 0x555b12b34b90 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f7745df6740 10 client.0 _ll_register_callbacks cb 0x7f773406f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f773be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12bc1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f773be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12bc1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38762/0 (socket says 172.21.15.97:38762) 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.550+0000 7f773be00640 1 -- 172.21.15.97:0/50999206 learned_addr learned my addr 172.21.15.97:0/50999206 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f7733e00640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12bc23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f7741000640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b4b9a0 0x555b12b391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f773be00640 1 -- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 msgr2=0x555b12bc23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f773be00640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12bc23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f773be00640 1 -- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b4b9a0 msgr2=0x555b12b391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f773be00640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b4b9a0 0x555b12b391e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f773be00640 1 -- 172.21.15.97:0/50999206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b12bc6720 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f773be00640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12bc1e60 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f772c02be50 tx=0x7f772c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109810aa9c7ef40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f7729600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f7729600640 1 -- 172.21.15.97:0/50999206 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c031ba0 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f7729600640 1 -- 172.21.15.97:0/50999206 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f772c031d40 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.551+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f772c037040 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7729600640 1 -- 172.21.15.97:0/50999206 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f772c004030 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 10 client.6891 fetch_fsmap learned FSMap version 51 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x555b12b31e00 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7729600640 1 -- 172.21.15.97:0/50999206 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c037040 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7729600640 1 -- 172.21.15.97:0/50999206 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 51) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f772c045040 con 0x555b12b347b0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 10 client.6891 fetch_fsmap finished waiting for FSMap version 51 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 10 client.6891 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 20 client.6891 populate_metadata read hostname 'smithi097' 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b54490 0x555b12b0c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 4 client.6891 mds_command: new command op to 6741 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 --> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555b12a78e90 con 0x555b12b54490 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.552+0000 7f7733e00640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b54490 0x555b12b0c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7733e00640 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b54490 0x555b12b0c450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45e596dfb0934966 server_cookie=6dab1754c066b4a9 in_seq=0 out_seq=0 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7729600640 10 client.6891 ms_handle_connect on v2:172.21.15.97:6835/3414007063 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7729600640 1 -- 172.21.15.97:0/50999206 <== mds.0 v2:172.21.15.97:6835/3414007063 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x555b12a78e90 con 0x555b12b54490 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7729600640 10 client.6891 handle_command_reply: tid=0 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7745df6740 1 client.6891 shutdown 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 msgr2=0x555b12bc1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.553+0000 7f7745df6740 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12bc1e60 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f772c02be50 tx=0x7f772c004730 comp rx=0 tx=0).stop 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 shutdown_connections 2024-03-06T07:32:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:6835/3414007063,v1:172.21.15.97:6837/3414007063] conn(0x555b12b54490 0x555b12b0c450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555b12b4b9a0 0x555b12b391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555b12b34b90 0x555b12bc23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 --2- 172.21.15.97:0/50999206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b12b347b0 0x555b12bc1e60 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7729600640 0 client.6891 ms_handle_reset on v2:172.21.15.97:6835/3414007063 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 >> 172.21.15.97:0/50999206 conn(0x555b12b07c90 msgr2=0x555b12b029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 shutdown_connections 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 1 -- 172.21.15.97:0/50999206 wait complete. 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7745df6740 20 client.6891 trim_cache size 0 max 16384 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 6741, 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 6.6601762410000003, 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 51, 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 82, 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 12.117913659999999 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b500 msgr2=0x7f771806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7743000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b500 0x7f771806d9c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f773c0cb740 tx=0x7f7724008040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 msgr2=0x7f773c15c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.554+0000 7f7743000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 0x7f773c15c4c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f773400ca70 tx=0x7f773400cf30 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 shutdown_connections 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f771806b500 0x7f771806d9c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773c15ef50 0x7f773c15c4c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f773c15b070 0x7f773c0c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 --2- 172.21.15.97:0/3593344282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f773c15ac90 0x7f773c0a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 >> 172.21.15.97:0/3593344282 conn(0x7f773c0cc4f0 msgr2=0x7f773c0c9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 shutdown_connections 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.555+0000 7f7743000640 1 -- 172.21.15.97:0/3593344282 wait complete. 2024-03-06T07:32:17.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:17.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.761+0000 7f2f02000640 1 Processor -- start 2024-03-06T07:32:17.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.762+0000 7f2f02000640 1 -- start start 2024-03-06T07:32:17.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2f02000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc144c00 0x7f2efc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2f02000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2efc15f9d0 0x7f2efc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2f02000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc06ee70 con 0x7f2efc0ce0a0 2024-03-06T07:32:17.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2f02000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc160370 con 0x7f2efc15f9d0 2024-03-06T07:32:17.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2f02000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc06cef0 con 0x7f2efc144c00 2024-03-06T07:32:17.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2ef2a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2efc15f9d0 0x7f2efc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2efbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2efc0ce0a0 0x7f2efc0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59516/0 (socket says 172.21.15.97:59516) 2024-03-06T07:32:17.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.763+0000 7f2efbe00640 1 -- 172.21.15.97:0/3477000388 learned_addr learned my addr 172.21.15.97:0/3477000388 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:17.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.764+0000 7f2efb400640 1 --2- 172.21.15.97:0/3477000388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc144c00 0x7f2efc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.764+0000 7f2ef2a00640 1 -- 172.21.15.97:0/3477000388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc144c00 msgr2=0x7f2efc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.764+0000 7f2ef2a00640 1 --2- 172.21.15.97:0/3477000388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc144c00 0x7f2efc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.764+0000 7f2ef2a00640 1 -- 172.21.15.97:0/3477000388 >> v1:172.21.15.97:6789/0 conn(0x7f2efc0ce0a0 legacy=0x7f2efc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.764+0000 7f2ef2a00640 1 -- 172.21.15.97:0/3477000388 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc06d1d0 con 0x7f2efc15f9d0 2024-03-06T07:32:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.764+0000 7f2efaa00640 1 -- 172.21.15.97:0/3477000388 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768243314 0 0) 0x7f2ef4001490 con 0x7f2efc0ce0a0 2024-03-06T07:32:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.765+0000 7f2ef2a00640 1 --2- 172.21.15.97:0/3477000388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc15f9d0 0x7f2efc15fe30 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40097a0 tx=0x7f2ee402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5ca8cf532b5833 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.765+0000 7f2efaa00640 1 -- 172.21.15.97:0/3477000388 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4035020 con 0x7f2efc15f9d0 2024-03-06T07:32:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.765+0000 7f2efaa00640 1 -- 172.21.15.97:0/3477000388 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4031830 con 0x7f2efc15f9d0 2024-03-06T07:32:17.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.765+0000 7f2efaa00640 1 -- 172.21.15.97:0/3477000388 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4031c00 con 0x7f2efc15f9d0 2024-03-06T07:32:17.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 -- 172.21.15.97:0/3477000388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc15f9d0 msgr2=0x7f2efc15fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 --2- 172.21.15.97:0/3477000388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc15f9d0 0x7f2efc15fe30 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40097a0 tx=0x7f2ee402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 -- 172.21.15.97:0/3477000388 shutdown_connections 2024-03-06T07:32:17.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 --2- 172.21.15.97:0/3477000388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc15f9d0 0x7f2efc15fe30 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 --2- 172.21.15.97:0/3477000388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc144c00 0x7f2efc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 -- 172.21.15.97:0/3477000388 >> 172.21.15.97:0/3477000388 conn(0x7f2efc0b1730 msgr2=0x7f2efc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 -- 172.21.15.97:0/3477000388 shutdown_connections 2024-03-06T07:32:17.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.766+0000 7f2f02000640 1 -- 172.21.15.97:0/3477000388 wait complete. 2024-03-06T07:32:17.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.768+0000 7f2f02000640 1 Processor -- start 2024-03-06T07:32:17.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.768+0000 7f2f02000640 1 -- start start 2024-03-06T07:32:17.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2f02000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2efc0ce0a0 0x7f2efc1f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2f02000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 0x7f2efc1f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2f02000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc15f9d0 0x7f2efc139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2f02000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc160730 con 0x7f2efc144c00 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2f02000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc06d790 con 0x7f2efc0ce0a0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2f02000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc06cb50 con 0x7f2efc15f9d0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 0x7f2efc1f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 0x7f2efc1f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38778/0 (socket says 172.21.15.97:38778) 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 -- 172.21.15.97:0/720182028 learned_addr learned my addr 172.21.15.97:0/720182028 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2efbe00640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc15f9d0 0x7f2efc139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 -- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc15f9d0 msgr2=0x7f2efc139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc15f9d0 0x7f2efc139e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 -- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2efc0ce0a0 msgr2=0x7f2efc1f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2efc0ce0a0 0x7f2efc1f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.770+0000 7f2ef2a00640 1 -- 172.21.15.97:0/720182028 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc1f78e0 con 0x7f2efc144c00 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.771+0000 7f2ef2a00640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 0x7f2efc1f73a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4000c00 tx=0x7f2ee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.771+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4035070 con 0x7f2efc144c00 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.771+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4004230 con 0x7f2efc144c00 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.771+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc13a9a0 con 0x7f2efc144c00 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.771+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee403a6f0 con 0x7f2efc144c00 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.771+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc13af60 con 0x7f2efc144c00 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.772+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc04f330 con 0x7f2efc144c00 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.773+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ee4038020 con 0x7f2efc144c00 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.775+0000 7f2ef3e00640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2ed806b4d0 0x7f2ed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.775+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ee40abf00 con 0x7f2efc144c00 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.775+0000 7f2efb400640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2ed806b4d0 0x7f2ed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.776+0000 7f2efb400640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2ed806b4d0 0x7f2ed806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0098f0 tx=0x7f2eec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:17.779+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ee4040070 con 0x7f2efc144c00 2024-03-06T07:32:18.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.021+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2efc072890 con 0x7f2efc144c00 2024-03-06T07:32:18.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.022+0000 7f2ef3e00640 1 -- 172.21.15.97:0/720182028 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2ee4042e40 con 0x7f2efc144c00 2024-03-06T07:32:18.024 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:18.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2ed806b4d0 msgr2=0x7f2ed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2ed806b4d0 0x7f2ed806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0098f0 tx=0x7f2eec008040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 msgr2=0x7f2efc1f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 0x7f2efc1f73a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4000c00 tx=0x7f2ee40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 shutdown_connections 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2ed806b4d0 0x7f2ed806d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2efc15f9d0 0x7f2efc139e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2efc144c00 0x7f2efc1f73a0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 --2- 172.21.15.97:0/720182028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2efc0ce0a0 0x7f2efc1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 >> 172.21.15.97:0/720182028 conn(0x7f2efc0b1730 msgr2=0x7f2efc0b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 shutdown_connections 2024-03-06T07:32:18.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:18.025+0000 7f2f02000640 1 -- 172.21.15.97:0/720182028 wait complete. 2024-03-06T07:32:18.044 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:18.044 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:18.044 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:32:18.045 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:18.045 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:18.045 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:18.045 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:18.045 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:18.045 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:18.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.045 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:32:18.087 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:18.087 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:18.087 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:18.087 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:18.088 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42337sec preferred_lft 42337sec 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:18.089 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:32:18.090 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.090 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:18.090 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:18.090 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:32:18.090 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:18.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.090 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:18.217 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:32:18.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.221 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:32:18.271 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:32:18.271 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:18.271 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:32:18.271 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:32:18.272 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:18.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.272 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:32:18.272 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:32:18.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:32:18.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:32:18.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:32:18.272 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.453 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:32:18.503 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:18.504 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:18.504 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:18.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:18.578 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:18.625 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:18.626 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:18.626 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:18.711 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:18.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:18.889 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:18.889 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:18.924 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:18.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:18.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:32:18.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:32:18.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:32:18.926 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:18.926 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:18.926 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:18.926 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:18.926 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:18.930 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:18.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:19.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:19.005 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:19.061 INFO:teuthology.orchestra.run.smithi097.stdout:client5120 2024-03-06T07:32:19.062 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:19.062 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:19.103 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:32:19.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:19.204 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:32:19.264 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:19.264 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:19.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:19.310 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:32:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:35+0 records in 2024-03-06T07:32:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:35+0 records out 2024-03-06T07:32:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.302673 s, 121 MB/s 2024-03-06T07:32:19.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:19.711 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:32:19.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:19.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.990+0000 7f9c88e00640 1 Processor -- start 2024-03-06T07:32:19.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.990+0000 7f9c88e00640 1 -- start start 2024-03-06T07:32:19.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.991+0000 7f9c88e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9c840d0ca0 0x7f9c840cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.991+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c840b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.991+0000 7f9c88e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8415d4f0 con 0x7f9c84160530 2024-03-06T07:32:19.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.991+0000 7f9c88e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c840b50a0 con 0x7f9c840d0ca0 2024-03-06T07:32:19.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.991+0000 7f9c88e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8415cc50 con 0x7f9c8406b6d0 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.992+0000 7f9c81400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9c84160530 0x7f9c8406afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59526/0 (socket says 172.21.15.97:59526) 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.992+0000 7f9c81400640 1 -- 172.21.15.97:0/323106439 learned_addr learned my addr 172.21.15.97:0/323106439 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.992+0000 7f9c82800640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c840b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.992+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958724600 0 0) 0x7f9c8415d4f0 con 0x7f9c84160530 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.992+0000 7f9c81e00640 1 --2- 172.21.15.97:0/323106439 >> v2:172.21.15.135:3300/0 conn(0x7f9c840d0ca0 0x7f9c840cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.992+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c68003520 con 0x7f9c84160530 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341057846 0 0) 0x7f9c68003520 con 0x7f9c84160530 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 msgr2=0x7f9c840b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c840b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 >> v2:172.21.15.135:3300/0 conn(0x7f9c840d0ca0 msgr2=0x7f9c840cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 --2- 172.21.15.97:0/323106439 >> v2:172.21.15.135:3300/0 conn(0x7f9c840d0ca0 0x7f9c840cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8415cf30 con 0x7f9c84160530 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9c70001670 con 0x7f9c84160530 2024-03-06T07:32:19.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 >> v1:172.21.15.97:6789/0 conn(0x7f9c84160530 legacy=0x7f9c8406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c680039b0 0x7f9c68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c68004d40 0x7f9c680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840b50a0 con 0x7f9c680039b0 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8415cc50 con 0x7f9c68004d40 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6800b9f0 con 0x7f9c8406b6d0 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c81e00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c680039b0 0x7f9c68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c81400640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c81e00640 1 -- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 msgr2=0x7f9c68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c81e00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c81e00640 1 -- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c68004d40 msgr2=0x7f9c680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.993+0000 7f9c82800640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c68004d40 0x7f9c680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.994+0000 7f9c81e00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c68004d40 0x7f9c680051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.994+0000 7f9c81e00640 1 -- 172.21.15.97:0/323106439 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68003520 con 0x7f9c680039b0 2024-03-06T07:32:19.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.994+0000 7f9c81400640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c8406b6d0 0x7f9c68004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c81e00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c680039b0 0x7f9c68003dd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009ac0 tx=0x7f9c68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50ab841d165cff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74039070 con 0x7f9c680039b0 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7402c710 con 0x7f9c680039b0 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c80a00640 1 -- 172.21.15.97:0/323106439 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74009690 con 0x7f9c680039b0 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c88e00640 1 -- 172.21.15.97:0/323106439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c680039b0 msgr2=0x7f9c68003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c88e00640 1 --2- 172.21.15.97:0/323106439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c680039b0 0x7f9c68003dd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009ac0 tx=0x7f9c68005830 comp rx=0 tx=0).stop 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.995+0000 7f9c82800640 1 -- 172.21.15.97:0/323106439 reap_dead start 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.996+0000 7f9c88e00640 1 -- 172.21.15.97:0/323106439 shutdown_connections 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.996+0000 7f9c88e00640 1 -- 172.21.15.97:0/323106439 >> 172.21.15.97:0/323106439 conn(0x7f9c840b1e10 msgr2=0x7f9c840b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.996+0000 7f9c88e00640 1 -- 172.21.15.97:0/323106439 shutdown_connections 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.996+0000 7f9c88e00640 1 -- 172.21.15.97:0/323106439 wait complete. 2024-03-06T07:32:19.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.997+0000 7f9c88e00640 1 Processor -- start 2024-03-06T07:32:20.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.998+0000 7f9c88e00640 1 -- start start 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.998+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c680039b0 0x7f9c8409a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.998+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 0x7f9c8415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.998+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c8409ddc0 0x7f9c8415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c88e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840d1080 con 0x7f9c8409a7b0 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c88e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84160990 con 0x7f9c8409ddc0 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c88e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840b5530 con 0x7f9c680039b0 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 0x7f9c8415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 0x7f9c8415d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38796/0 (socket says 172.21.15.97:38796) 2024-03-06T07:32:20.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 -- 172.21.15.97:0/2037399588 learned_addr learned my addr 172.21.15.97:0/2037399588 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:20.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 -- 172.21.15.97:0/2037399588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c680039b0 msgr2=0x7f9c8409a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c680039b0 0x7f9c8409a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 -- 172.21.15.97:0/2037399588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c8409ddc0 msgr2=0x7f9c8415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c8409ddc0 0x7f9c8415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 -- 172.21.15.97:0/2037399588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8409cd20 con 0x7f9c8409a7b0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:19.999+0000 7f9c81400640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 0x7f9c8415d860 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000d8b0 tx=0x7f9c7000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.000+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70019070 con 0x7f9c8409a7b0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.000+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c700074e0 con 0x7f9c8409a7b0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.000+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8409d000 con 0x7f9c8409a7b0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.000+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c700049e0 con 0x7f9c8409a7b0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.000+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8409d5c0 con 0x7f9c8409a7b0 2024-03-06T07:32:20.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.001+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c84068fb0 con 0x7f9c8409a7b0 2024-03-06T07:32:20.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.002+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c7001c070 con 0x7f9c8409a7b0 2024-03-06T07:32:20.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.004+0000 7f9c7a000640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.004+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c70089d70 con 0x7f9c8409a7b0 2024-03-06T07:32:20.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.004+0000 7f9c81e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.005+0000 7f9c81e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c74036000 tx=0x7f9c7402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.008+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c84068fb0 con 0x7f9c8409a7b0 2024-03-06T07:32:20.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.322+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c8404f330 con 0x7f9c8409a7b0 2024-03-06T07:32:20.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.323+0000 7f9c7a000640 1 -- 172.21.15.97:0/2037399588 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f9c70058ac0 con 0x7f9c8409a7b0 2024-03-06T07:32:20.325 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:20.325 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:20.326 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:20.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.325+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 msgr2=0x7f9c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.325+0000 7f9c88e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c74036000 tx=0x7f9c7402e000 comp rx=0 tx=0).stop 2024-03-06T07:32:20.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 msgr2=0x7f9c8415d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 0x7f9c8415d860 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000d8b0 tx=0x7f9c7000dd70 comp rx=0 tx=0).stop 2024-03-06T07:32:20.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 shutdown_connections 2024-03-06T07:32:20.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c8409ddc0 0x7f9c8415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c8409a7b0 0x7f9c8415d860 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 --2- 172.21.15.97:0/2037399588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c680039b0 0x7f9c8409a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 >> 172.21.15.97:0/2037399588 conn(0x7f9c840b1e10 msgr2=0x7f9c840cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 shutdown_connections 2024-03-06T07:32:20.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.326+0000 7f9c88e00640 1 -- 172.21.15.97:0/2037399588 wait complete. 2024-03-06T07:32:20.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:32:20.586 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:32:20.586 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:32:20.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:32:20.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.782+0000 7f40c2400640 1 Processor -- start 2024-03-06T07:32:20.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.782+0000 7f40c2400640 1 -- start start 2024-03-06T07:32:20.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40c2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40c2400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f40bc0aebf0 0x7f40bc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40c2400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f40bc0cc800 0x7f40bc0cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40c2400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc15ce30 con 0x7f40bc0aebf0 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40c2400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc15c240 con 0x7f40bc0cc800 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40c2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc15c500 con 0x7f40bc1561f0 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.784+0000 7f40bbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f40bc0aebf0 0x7f40bc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f40bc0aebf0 0x7f40bc0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38808/0 (socket says 172.21.15.97:38808) 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 -- 172.21.15.97:0/1599679123 learned_addr learned my addr 172.21.15.97:0/1599679123 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40c1000640 1 --2- 172.21.15.97:0/1599679123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40c1a00640 1 --2- 172.21.15.97:0/1599679123 >> v2:172.21.15.135:3300/0 conn(0x7f40bc0cc800 0x7f40bc0cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 -- 172.21.15.97:0/1599679123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 msgr2=0x7f40bc0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 --2- 172.21.15.97:0/1599679123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 -- 172.21.15.97:0/1599679123 >> v2:172.21.15.135:3300/0 conn(0x7f40bc0cc800 msgr2=0x7f40bc0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 --2- 172.21.15.97:0/1599679123 >> v2:172.21.15.135:3300/0 conn(0x7f40bc0cc800 0x7f40bc0cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40bbe00640 1 -- 172.21.15.97:0/1599679123 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc15c7e0 con 0x7f40bc0aebf0 2024-03-06T07:32:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.785+0000 7f40c1a00640 1 --2- 172.21.15.97:0/1599679123 >> v2:172.21.15.135:3300/0 conn(0x7f40bc0cc800 0x7f40bc0cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.786+0000 7f40bbe00640 1 --2- 172.21.15.97:0/1599679123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0aebf0 0x7f40bc0cc2c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0cc1f0 tx=0x7f40ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e291ddbd22a58a2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.786+0000 7f40bb400640 1 -- 172.21.15.97:0/1599679123 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac035020 con 0x7f40bc0aebf0 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.786+0000 7f40bb400640 1 -- 172.21.15.97:0/1599679123 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac031870 con 0x7f40bc0aebf0 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.786+0000 7f40bb400640 1 -- 172.21.15.97:0/1599679123 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac031c40 con 0x7f40bc0aebf0 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 -- 172.21.15.97:0/1599679123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0aebf0 msgr2=0x7f40bc0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 --2- 172.21.15.97:0/1599679123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0aebf0 0x7f40bc0cc2c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0cc1f0 tx=0x7f40ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 -- 172.21.15.97:0/1599679123 shutdown_connections 2024-03-06T07:32:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 --2- 172.21.15.97:0/1599679123 >> v2:172.21.15.135:3300/0 conn(0x7f40bc0cc800 0x7f40bc0cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 --2- 172.21.15.97:0/1599679123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0aebf0 0x7f40bc0cc2c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 --2- 172.21.15.97:0/1599679123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 -- 172.21.15.97:0/1599679123 >> 172.21.15.97:0/1599679123 conn(0x7f40bc06afc0 msgr2=0x7f40bc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 -- 172.21.15.97:0/1599679123 shutdown_connections 2024-03-06T07:32:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.787+0000 7f40c2400640 1 -- 172.21.15.97:0/1599679123 wait complete. 2024-03-06T07:32:20.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.789+0000 7f40c2400640 1 Processor -- start 2024-03-06T07:32:20.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.789+0000 7f40c2400640 1 -- start start 2024-03-06T07:32:20.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40c2400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40bc0aebf0 0x7f40bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40c2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 0x7f40bc071e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40c2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc1f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40c2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc1565d0 con 0x7f40bc0cc800 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40c2400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0ccc90 con 0x7f40bc0aebf0 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40c2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc156750 con 0x7f40bc1561f0 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 0x7f40bc071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 0x7f40bc071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38818/0 (socket says 172.21.15.97:38818) 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.790+0000 7f40bbe00640 1 -- 172.21.15.97:0/4029253178 learned_addr learned my addr 172.21.15.97:0/4029253178 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40c1000640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40bc0aebf0 0x7f40bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40c1a00640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc1f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40bbe00640 1 -- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 msgr2=0x7f40bc1f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40bbe00640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc1f3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40bbe00640 1 -- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40bc0aebf0 msgr2=0x7f40bc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40bbe00640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40bc0aebf0 0x7f40bc071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40bbe00640 1 -- 172.21.15.97:0/4029253178 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc157450 con 0x7f40bc0cc800 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40c1000640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40bc0aebf0 0x7f40bc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40bbe00640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 0x7f40bc071e50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f40ac000c00 tx=0x7f40ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac035040 con 0x7f40bc0cc800 2024-03-06T07:32:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac030d00 con 0x7f40bc0cc800 2024-03-06T07:32:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac0456a0 con 0x7f40bc0cc800 2024-03-06T07:32:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.791+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc1f70d0 con 0x7f40bc0cc800 2024-03-06T07:32:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.792+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc1f7690 con 0x7f40bc0cc800 2024-03-06T07:32:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.793+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc04f330 con 0x7f40bc0cc800 2024-03-06T07:32:20.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.794+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40ac038020 con 0x7f40bc0cc800 2024-03-06T07:32:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.795+0000 7f40b8c00640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40a006b4d0 0x7f40a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.795+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40ac0ac490 con 0x7f40bc0cc800 2024-03-06T07:32:20.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.795+0000 7f40c1000640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40a006b4d0 0x7f40a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.796+0000 7f40c1000640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40a006b4d0 0x7f40a006d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f40b0006fd0 tx=0x7f40b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:20.800+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40ac07b3a0 con 0x7f40bc0cc800 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.158+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f40bc0cd770 con 0x7f40bc0cc800 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.159+0000 7f40b8c00640 1 -- 172.21.15.97:0/4029253178 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1653 (secure 0 0 0) 0x7f40ac07aca0 con 0x7f40bc0cc800 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: "total_avail_bytes": 772829392896, 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_bytes": 264720384, 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_bytes": 264720384, 2024-03-06T07:32:21.161 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_ratio": 0.00034241675166413188, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "num_osds": 8, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "num_per_pool_osds": 8, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "stats_by_class": { 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "ssd": { 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "total_avail_bytes": 772829392896, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_bytes": 264720384, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_bytes": 264720384, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "total_used_raw_ratio": 0.00034241675166413188 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "pools": [ 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "name": ".mgr", 2024-03-06T07:32:21.162 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 1, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 590368, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 2, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 1160, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 1187840, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 1.6179307067432092e-06, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367085584384 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephfs_metadata", 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 12, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 2410, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 22, 2024-03-06T07:32:21.163 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 64, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 65536, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 8.9265277836148016e-08, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367085584384 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephfs_data", 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 13, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "stats": { 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "stored": 0, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "objects": 0, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "kb_used": 0, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bytes_used": 0, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "percent_used": 0, 2024-03-06T07:32:21.164 INFO:teuthology.orchestra.run.smithi097.stdout: "max_avail": 367085584384 2024-03-06T07:32:21.165 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:21.165 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:21.165 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:32:21.165 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:32:21.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40a006b4d0 msgr2=0x7f40a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40a006b4d0 0x7f40a006d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f40b0006fd0 tx=0x7f40b0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 msgr2=0x7f40bc071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 0x7f40bc071e50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f40ac000c00 tx=0x7f40ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 shutdown_connections 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40a006b4d0 0x7f40a006d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40bc1561f0 0x7f40bc1f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40bc0cc800 0x7f40bc071e50 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 --2- 172.21.15.97:0/4029253178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40bc0aebf0 0x7f40bc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.161+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 >> 172.21.15.97:0/4029253178 conn(0x7f40bc06afc0 msgr2=0x7f40bc15b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.162+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 shutdown_connections 2024-03-06T07:32:21.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.162+0000 7f40c2400640 1 -- 172.21.15.97:0/4029253178 wait complete. 2024-03-06T07:32:21.183 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:32:21.183 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:21.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:21.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:21.402 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:21.402 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:21.402 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:21.456 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:21.456 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:21.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:21.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.757+0000 7f4c6bc00640 1 Processor -- start 2024-03-06T07:32:21.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.757+0000 7f4c6bc00640 1 -- start start 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c6bc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 0x7f4c6406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c6bc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4c64072340 0x7f4c6406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c6bc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6415b090 con 0x7f4c6415acb0 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c6bc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c640b2a30 con 0x7f4c64072340 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c6bc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b4360 con 0x7f4c640b4680 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c69200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4c6415acb0 0x7f4c640b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59542/0 (socket says 172.21.15.97:59542) 2024-03-06T07:32:21.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.759+0000 7f4c69200640 1 -- 172.21.15.97:0/4244814109 learned_addr learned my addr 172.21.15.97:0/4244814109 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63e00640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 0x7f4c6406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181872986 0 0) 0x7f4c6415b090 con 0x7f4c6415acb0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c69c00640 1 --2- 172.21.15.97:0/4244814109 >> v2:172.21.15.135:3300/0 conn(0x7f4c64072340 0x7f4c6406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c4c003520 con 0x7f4c6415acb0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992123910 0 0) 0x7f4c4c003520 con 0x7f4c6415acb0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 msgr2=0x7f4c6406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 0x7f4c6406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 >> v2:172.21.15.135:3300/0 conn(0x7f4c64072340 msgr2=0x7f4c6406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 --2- 172.21.15.97:0/4244814109 >> v2:172.21.15.135:3300/0 conn(0x7f4c64072340 0x7f4c6406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640b2cf0 con 0x7f4c6415acb0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4c58001670 con 0x7f4c6415acb0 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 >> v1:172.21.15.97:6789/0 conn(0x7f4c6415acb0 legacy=0x7f4c640b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c4c0039b0 0x7f4c4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c004800 0x7f4c4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 0x7f4c4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.760+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b2a30 con 0x7f4c4c0039b0 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b4360 con 0x7f4c4c004800 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c00b9c0 con 0x7f4c640b4680 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c69200640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c4c0039b0 0x7f4c4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c63e00640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c004800 0x7f4c4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c69c00640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 0x7f4c4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c69200640 1 -- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 msgr2=0x7f4c4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.761+0000 7f4c69200640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c640b4680 0x7f4c4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.762+0000 7f4c69200640 1 -- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c004800 msgr2=0x7f4c4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.762+0000 7f4c69200640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c004800 0x7f4c4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.762+0000 7f4c69200640 1 -- 172.21.15.97:0/4244814109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c003520 con 0x7f4c4c0039b0 2024-03-06T07:32:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.763+0000 7f4c69200640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c4c0039b0 0x7f4c4c003dd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4c58007fd0 tx=0x7f4c4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd047147e9446c7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.763+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58007ca0 con 0x7f4c4c0039b0 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.763+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58030070 con 0x7f4c4c0039b0 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.763+0000 7f4c63400640 1 -- 172.21.15.97:0/4244814109 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58005d10 con 0x7f4c4c0039b0 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c6bc00640 1 -- 172.21.15.97:0/4244814109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c4c0039b0 msgr2=0x7f4c4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/4244814109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c4c0039b0 0x7f4c4c003dd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4c58007fd0 tx=0x7f4c4c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c69c00640 1 -- 172.21.15.97:0/4244814109 reap_dead start 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c6bc00640 1 -- 172.21.15.97:0/4244814109 shutdown_connections 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c6bc00640 1 -- 172.21.15.97:0/4244814109 >> 172.21.15.97:0/4244814109 conn(0x7f4c640b1730 msgr2=0x7f4c640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c6bc00640 1 -- 172.21.15.97:0/4244814109 shutdown_connections 2024-03-06T07:32:21.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.764+0000 7f4c6bc00640 1 -- 172.21.15.97:0/4244814109 wait complete. 2024-03-06T07:32:21.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.766+0000 7f4c6bc00640 1 Processor -- start 2024-03-06T07:32:21.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.766+0000 7f4c6bc00640 1 -- start start 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c6bc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c0039b0 0x7f4c640aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c6bc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c64154770 0x7f4c641e5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c6bc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 0x7f4c64158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c6bc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b4010 con 0x7f4c641e5bb0 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c6bc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b32b0 con 0x7f4c4c0039b0 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c6bc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640ccfb0 con 0x7f4c64154770 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c69c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 0x7f4c64158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c69c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 0x7f4c64158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38824/0 (socket says 172.21.15.97:38824) 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.767+0000 7f4c69c00640 1 -- 172.21.15.97:0/1001901216 learned_addr learned my addr 172.21.15.97:0/1001901216 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69200640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c0039b0 0x7f4c640aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c63e00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c64154770 0x7f4c641e5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69c00640 1 -- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c64154770 msgr2=0x7f4c641e5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69c00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c64154770 0x7f4c641e5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69c00640 1 -- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c0039b0 msgr2=0x7f4c640aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69c00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c0039b0 0x7f4c640aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69c00640 1 -- 172.21.15.97:0/1001901216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c641e6050 con 0x7f4c641e5bb0 2024-03-06T07:32:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c69c00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 0x7f4c64158370 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4c64073270 tx=0x7f4c5c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c015430 con 0x7f4c641e5bb0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c01d070 con 0x7f4c641e5bb0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c641e6330 con 0x7f4c641e5bb0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.768+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c015830 con 0x7f4c641e5bb0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.769+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c640ae920 con 0x7f4c641e5bb0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.770+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c640aeb20 con 0x7f4c641e5bb0 2024-03-06T07:32:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.771+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c5c0159d0 con 0x7f4c641e5bb0 2024-03-06T07:32:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.771+0000 7f4c60c00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c4406b4d0 0x7f4c4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.771+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c5c08ac20 con 0x7f4c641e5bb0 2024-03-06T07:32:21.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.772+0000 7f4c69200640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c4406b4d0 0x7f4c4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.772+0000 7f4c69200640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c4406b4d0 0x7f4c4406d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c58034000 tx=0x7f4c58007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.776+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c640aeb20 con 0x7f4c641e5bb0 2024-03-06T07:32:21.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.973+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c6404f330 con 0x7f4c641e5bb0 2024-03-06T07:32:21.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.976+0000 7f4c60c00640 1 -- 172.21.15.97:0/1001901216 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f4c5c010040 con 0x7f4c641e5bb0 2024-03-06T07:32:21.979 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:21.979 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:11.891681+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.979+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c4406b4d0 msgr2=0x7f4c4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.979+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c4406b4d0 0x7f4c4406d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c58034000 tx=0x7f4c58007000 comp rx=0 tx=0).stop 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.979+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 msgr2=0x7f4c64158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.979+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 0x7f4c64158370 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4c64073270 tx=0x7f4c5c00f580 comp rx=0 tx=0).stop 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.979+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 shutdown_connections 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c4406b4d0 0x7f4c4406d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c641e5bb0 0x7f4c64158370 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c64154770 0x7f4c641e5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 --2- 172.21.15.97:0/1001901216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c4c0039b0 0x7f4c640aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 >> 172.21.15.97:0/1001901216 conn(0x7f4c640b1730 msgr2=0x7f4c640af780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 shutdown_connections 2024-03-06T07:32:21.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:21.980+0000 7f4c6bc00640 1 -- 172.21.15.97:0/1001901216 wait complete. 2024-03-06T07:32:22.003 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:08.169299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:09.359746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:22.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.215+0000 7fad2f200640 1 Processor -- start 2024-03-06T07:32:22.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.215+0000 7fad2f200640 1 -- start start 2024-03-06T07:32:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.216+0000 7fad2f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fad2815fc10 0x7fad28070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.216+0000 7fad2f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad280b3f70 0x7fad280ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.216+0000 7fad2f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad280b46c0 con 0x7fad2815fff0 2024-03-06T07:32:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.216+0000 7fad2f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280b49a0 con 0x7fad2815fc10 2024-03-06T07:32:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.216+0000 7fad2f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad28072c90 con 0x7fad280b3f70 2024-03-06T07:32:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad27e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fad2815fc10 0x7fad28070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad1f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fad2815fff0 0x7fad28160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59548/0 (socket says 172.21.15.97:59548) 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad1f400640 1 -- 172.21.15.97:0/268338665 learned_addr learned my addr 172.21.15.97:0/268338665 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad2d200640 1 --2- 172.21.15.97:0/268338665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad280b3f70 0x7fad280ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad27e00640 1 -- 172.21.15.97:0/268338665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad280b3f70 msgr2=0x7fad280ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad27e00640 1 --2- 172.21.15.97:0/268338665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad280b3f70 0x7fad280ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad27e00640 1 -- 172.21.15.97:0/268338665 >> v1:172.21.15.97:6789/0 conn(0x7fad2815fff0 legacy=0x7fad28160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.217+0000 7fad27e00640 1 -- 172.21.15.97:0/268338665 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2809dd80 con 0x7fad2815fc10 2024-03-06T07:32:22.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.218+0000 7fad27e00640 1 --2- 172.21.15.97:0/268338665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad2815fc10 0x7fad28070ff0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad1802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799b7df9b86b1d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad27400640 1 -- 172.21.15.97:0/268338665 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18035020 con 0x7fad2815fc10 2024-03-06T07:32:22.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad27400640 1 -- 172.21.15.97:0/268338665 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad18031810 con 0x7fad2815fc10 2024-03-06T07:32:22.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad27400640 1 -- 172.21.15.97:0/268338665 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18031be0 con 0x7fad2815fc10 2024-03-06T07:32:22.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 -- 172.21.15.97:0/268338665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad2815fc10 msgr2=0x7fad28070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 --2- 172.21.15.97:0/268338665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad2815fc10 0x7fad28070ff0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad1802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 -- 172.21.15.97:0/268338665 shutdown_connections 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 --2- 172.21.15.97:0/268338665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad280b3f70 0x7fad280ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 --2- 172.21.15.97:0/268338665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad2815fc10 0x7fad28070ff0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 -- 172.21.15.97:0/268338665 >> 172.21.15.97:0/268338665 conn(0x7fad280b1940 msgr2=0x7fad280b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 -- 172.21.15.97:0/268338665 shutdown_connections 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.219+0000 7fad2f200640 1 -- 172.21.15.97:0/268338665 wait complete. 2024-03-06T07:32:22.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.222+0000 7fad2f200640 1 Processor -- start 2024-03-06T07:32:22.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.222+0000 7fad2f200640 1 -- start start 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.223+0000 7fad2f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 0x7fad2814a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.223+0000 7fad2f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad2815fc10 0x7fad28157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.223+0000 7fad2f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad2815fff0 0x7fad280cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.223+0000 7fad2f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2815c460 con 0x7fad280b3f70 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.223+0000 7fad2f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2815c2e0 con 0x7fad2815fc10 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.223+0000 7fad2f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad280ce9e0 con 0x7fad2815fff0 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 0x7fad2814a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 0x7fad2814a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38848/0 (socket says 172.21.15.97:38848) 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 -- 172.21.15.97:0/1817277062 learned_addr learned my addr 172.21.15.97:0/1817277062 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 -- 172.21.15.97:0/1817277062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad2815fff0 msgr2=0x7fad280cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad2815fff0 0x7fad280cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 -- 172.21.15.97:0/1817277062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad2815fc10 msgr2=0x7fad28157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad2815fc10 0x7fad28157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 -- 172.21.15.97:0/1817277062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad281583d0 con 0x7fad280b3f70 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.224+0000 7fad27e00640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 0x7fad2814a6a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fad1802bec0 tx=0x7fad180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.225+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18041070 con 0x7fad280b3f70 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.225+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1803ad00 con 0x7fad280b3f70 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.225+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2809cd20 con 0x7fad280b3f70 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.225+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18035040 con 0x7fad280b3f70 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.225+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2809d2e0 con 0x7fad280b3f70 2024-03-06T07:32:22.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.226+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2804f330 con 0x7fad280b3f70 2024-03-06T07:32:22.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.227+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad18038020 con 0x7fad280b3f70 2024-03-06T07:32:22.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.228+0000 7fad24c00640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fad0406b4d0 0x7fad0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.228+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad180abeb0 con 0x7fad280b3f70 2024-03-06T07:32:22.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.228+0000 7fad1f400640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fad0406b4d0 0x7fad0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.229+0000 7fad1f400640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fad0406b4d0 0x7fad0406d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fad10000f60 tx=0x7fad10007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.233+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad1807ad20 con 0x7fad280b3f70 2024-03-06T07:32:22.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.467+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad280b2d40 con 0x7fad280b3f70 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.468+0000 7fad24c00640 1 -- 172.21.15.97:0/1817277062 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad1807a620 con 0x7fad280b3f70 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fad0406b4d0 msgr2=0x7fad0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fad0406b4d0 0x7fad0406d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fad10000f60 tx=0x7fad10007a80 comp rx=0 tx=0).stop 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 msgr2=0x7fad2814a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 0x7fad2814a6a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fad1802bec0 tx=0x7fad180046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 shutdown_connections 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fad0406b4d0 0x7fad0406d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad2815fff0 0x7fad280cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad2815fc10 0x7fad28157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 --2- 172.21.15.97:0/1817277062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fad280b3f70 0x7fad2814a6a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.470+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 >> 172.21.15.97:0/1817277062 conn(0x7fad280b1940 msgr2=0x7fad280ae5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.471+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 shutdown_connections 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.471+0000 7fad2f200640 1 -- 172.21.15.97:0/1817277062 wait complete. 2024-03-06T07:32:22.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:22.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.705+0000 7ffa30000640 1 Processor -- start 2024-03-06T07:32:22.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.706+0000 7ffa30000640 1 -- start start 2024-03-06T07:32:22.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.707+0000 7ffa30000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffa2815fc10 0x7ffa28070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.707+0000 7ffa30000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b46c0 con 0x7ffa2815fff0 2024-03-06T07:32:22.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.707+0000 7ffa30000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b49a0 con 0x7ffa280b3f70 2024-03-06T07:32:22.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.707+0000 7ffa30000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa28072c90 con 0x7ffa2815fc10 2024-03-06T07:32:22.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa2cc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ffa2815fff0 0x7ffa28160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59554/0 (socket says 172.21.15.97:59554) 2024-03-06T07:32:22.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa2cc00640 1 -- 172.21.15.97:0/3068275290 learned_addr learned my addr 172.21.15.97:0/3068275290 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:22.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa2d600640 1 --2- 172.21.15.97:0/3068275290 >> v2:172.21.15.135:3300/0 conn(0x7ffa2815fc10 0x7ffa28070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa27e00640 1 -- 172.21.15.97:0/3068275290 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551925541 0 0) 0x7ffa280b46c0 con 0x7ffa2815fff0 2024-03-06T07:32:22.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa27e00640 1 -- 172.21.15.97:0/3068275290 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa10003520 con 0x7ffa2815fff0 2024-03-06T07:32:22.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa2d600640 1 -- 172.21.15.97:0/3068275290 >> v1:172.21.15.135:6789/0 conn(0x7ffa280b3f70 legacy=0x7ffa280ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa2d600640 1 -- 172.21.15.97:0/3068275290 >> v1:172.21.15.97:6789/0 conn(0x7ffa2815fff0 legacy=0x7ffa28160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.708+0000 7ffa2d600640 1 -- 172.21.15.97:0/3068275290 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2809dd80 con 0x7ffa2815fc10 2024-03-06T07:32:22.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.710+0000 7ffa2d600640 1 --2- 172.21.15.97:0/3068275290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2815fc10 0x7ffa28070ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffa180097a0 tx=0x7ffa1802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eaa8cd88361ddf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.710+0000 7ffa27e00640 1 -- 172.21.15.97:0/3068275290 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18034070 con 0x7ffa2815fc10 2024-03-06T07:32:22.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.710+0000 7ffa27e00640 1 -- 172.21.15.97:0/3068275290 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa18030610 con 0x7ffa2815fc10 2024-03-06T07:32:22.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.710+0000 7ffa27e00640 1 -- 172.21.15.97:0/3068275290 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180309e0 con 0x7ffa2815fc10 2024-03-06T07:32:22.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 -- 172.21.15.97:0/3068275290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2815fc10 msgr2=0x7ffa28070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 --2- 172.21.15.97:0/3068275290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2815fc10 0x7ffa28070ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffa180097a0 tx=0x7ffa1802b960 comp rx=0 tx=0).stop 2024-03-06T07:32:22.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 -- 172.21.15.97:0/3068275290 shutdown_connections 2024-03-06T07:32:22.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 --2- 172.21.15.97:0/3068275290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2815fc10 0x7ffa28070ff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 -- 172.21.15.97:0/3068275290 >> 172.21.15.97:0/3068275290 conn(0x7ffa280b1940 msgr2=0x7ffa280b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 -- 172.21.15.97:0/3068275290 shutdown_connections 2024-03-06T07:32:22.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.711+0000 7ffa30000640 1 -- 172.21.15.97:0/3068275290 wait complete. 2024-03-06T07:32:22.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.713+0000 7ffa30000640 1 Processor -- start 2024-03-06T07:32:22.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.713+0000 7ffa30000640 1 -- start start 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa30000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa280b3f70 0x7ffa28148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa30000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 0x7ffa2809b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa30000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa2815fff0 0x7ffa2815d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa30000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2815c460 con 0x7ffa2815fc10 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa30000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2815c2e0 con 0x7ffa280b3f70 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa30000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280cebb0 con 0x7ffa2815fff0 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa2cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 0x7ffa2809b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa2cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 0x7ffa2809b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38850/0 (socket says 172.21.15.97:38850) 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa2cc00640 1 -- 172.21.15.97:0/1609386908 learned_addr learned my addr 172.21.15.97:0/1609386908 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa2d600640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa280b3f70 0x7ffa28148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.714+0000 7ffa2e000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa2815fff0 0x7ffa2815d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2cc00640 1 -- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa2815fff0 msgr2=0x7ffa2815d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2cc00640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa2815fff0 0x7ffa2815d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2cc00640 1 -- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa280b3f70 msgr2=0x7ffa28148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2cc00640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa280b3f70 0x7ffa28148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2cc00640 1 -- 172.21.15.97:0/1609386908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2809be80 con 0x7ffa2815fc10 2024-03-06T07:32:22.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2e000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa2815fff0 0x7ffa2815d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:22.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa2cc00640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 0x7ffa2809b3b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00cbc0 tx=0x7ffa1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c017070 con 0x7ffa2815fc10 2024-03-06T07:32:22.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c00ad40 con 0x7ffa2815fc10 2024-03-06T07:32:22.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c012620 con 0x7ffa2815fc10 2024-03-06T07:32:22.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2809c160 con 0x7ffa2815fc10 2024-03-06T07:32:22.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.715+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2809c720 con 0x7ffa2815fc10 2024-03-06T07:32:22.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.717+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2804f330 con 0x7ffa2815fc10 2024-03-06T07:32:22.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.718+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa1c01a070 con 0x7ffa2815fc10 2024-03-06T07:32:22.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.719+0000 7ffa25600640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa0c06b4d0 0x7ffa0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.719+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa1c08a080 con 0x7ffa2815fc10 2024-03-06T07:32:22.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.719+0000 7ffa2d600640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa0c06b4d0 0x7ffa0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.720+0000 7ffa2d600640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa0c06b4d0 0x7ffa0c06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ffa18036000 tx=0x7ffa1803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.728+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa1c058f00 con 0x7ffa2815fc10 2024-03-06T07:32:22.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.926+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa2806ef30 con 0x7ffa2815fc10 2024-03-06T07:32:22.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.926+0000 7ffa25600640 1 -- 172.21.15.97:0/1609386908 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa1c058800 con 0x7ffa2815fc10 2024-03-06T07:32:22.929 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:32:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa0c06b4d0 msgr2=0x7ffa0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa0c06b4d0 0x7ffa0c06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ffa18036000 tx=0x7ffa1803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 msgr2=0x7ffa2809b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 0x7ffa2809b3b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00cbc0 tx=0x7ffa1c00a430 comp rx=0 tx=0).stop 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 shutdown_connections 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa0c06b4d0 0x7ffa0c06d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffa2815fff0 0x7ffa2815d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffa2815fc10 0x7ffa2809b3b0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 --2- 172.21.15.97:0/1609386908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa280b3f70 0x7ffa28148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 >> 172.21.15.97:0/1609386908 conn(0x7ffa280b1940 msgr2=0x7ffa281516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 shutdown_connections 2024-03-06T07:32:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:22.929+0000 7ffa30000640 1 -- 172.21.15.97:0/1609386908 wait complete. 2024-03-06T07:32:22.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:23.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.152+0000 7f8d86200640 1 Processor -- start 2024-03-06T07:32:23.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.152+0000 7f8d86200640 1 -- start start 2024-03-06T07:32:23.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.154+0000 7f8d86200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 0x7f8d80070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.154+0000 7f8d86200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d800b3f70 0x7f8d800ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.154+0000 7f8d86200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800b46c0 con 0x7f8d8015fff0 2024-03-06T07:32:23.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.154+0000 7f8d86200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d800b49a0 con 0x7f8d800b3f70 2024-03-06T07:32:23.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.154+0000 7f8d86200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80072c90 con 0x7f8d8015fc10 2024-03-06T07:32:23.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 0x7f8d80070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 0x7f8d80070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:57436/0 (socket says 172.21.15.97:57436) 2024-03-06T07:32:23.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7f400640 1 -- 172.21.15.97:0/835133636 learned_addr learned my addr 172.21.15.97:0/835133636 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:23.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7fe00640 1 --2- 172.21.15.97:0/835133636 >> v2:172.21.15.135:3300/0 conn(0x7f8d800b3f70 0x7f8d800ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7f400640 1 -- 172.21.15.97:0/835133636 >> v2:172.21.15.135:3300/0 conn(0x7f8d800b3f70 msgr2=0x7f8d800ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7f400640 1 --2- 172.21.15.97:0/835133636 >> v2:172.21.15.135:3300/0 conn(0x7f8d800b3f70 0x7f8d800ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.155+0000 7f8d7f400640 1 -- 172.21.15.97:0/835133636 >> v1:172.21.15.135:6789/0 conn(0x7f8d8015fff0 legacy=0x7f8d80160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.156+0000 7f8d7f400640 1 -- 172.21.15.97:0/835133636 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8009dd80 con 0x7f8d8015fc10 2024-03-06T07:32:23.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.156+0000 7f8d7fe00640 1 --2- 172.21.15.97:0/835133636 >> v2:172.21.15.135:3300/0 conn(0x7f8d800b3f70 0x7f8d800ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:23.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.157+0000 7f8d7f400640 1 --2- 172.21.15.97:0/835133636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 0x7f8d80070ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009870 tx=0x7f8d7002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcc4565ac8f4fa3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.157+0000 7f8d7e000640 1 -- 172.21.15.97:0/835133636 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d70034070 con 0x7f8d8015fc10 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.157+0000 7f8d7e000640 1 -- 172.21.15.97:0/835133636 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d70030840 con 0x7f8d8015fc10 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.157+0000 7f8d7e000640 1 -- 172.21.15.97:0/835133636 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d70030c10 con 0x7f8d8015fc10 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 -- 172.21.15.97:0/835133636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 msgr2=0x7f8d80070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 --2- 172.21.15.97:0/835133636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 0x7f8d80070ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009870 tx=0x7f8d7002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 -- 172.21.15.97:0/835133636 shutdown_connections 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 --2- 172.21.15.97:0/835133636 >> v2:172.21.15.135:3300/0 conn(0x7f8d800b3f70 0x7f8d800ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 --2- 172.21.15.97:0/835133636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d8015fc10 0x7f8d80070ff0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 -- 172.21.15.97:0/835133636 >> 172.21.15.97:0/835133636 conn(0x7f8d800b1940 msgr2=0x7f8d800b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 -- 172.21.15.97:0/835133636 shutdown_connections 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.158+0000 7f8d86200640 1 -- 172.21.15.97:0/835133636 wait complete. 2024-03-06T07:32:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.160+0000 7f8d86200640 1 Processor -- start 2024-03-06T07:32:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.160+0000 7f8d86200640 1 -- start start 2024-03-06T07:32:23.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d86200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d800b3f70 0x7f8d80159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d86200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d8015fc10 0x7f8d8015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d86200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 0x7f8d8009cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d86200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8015c460 con 0x7f8d8015fff0 2024-03-06T07:32:23.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d86200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8015c2e0 con 0x7f8d8015fc10 2024-03-06T07:32:23.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d86200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800ce9e0 con 0x7f8d800b3f70 2024-03-06T07:32:23.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d7fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 0x7f8d8009cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d7fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 0x7f8d8009cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38862/0 (socket says 172.21.15.97:38862) 2024-03-06T07:32:23.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d7fe00640 1 -- 172.21.15.97:0/1228484490 learned_addr learned my addr 172.21.15.97:0/1228484490 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:23.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.161+0000 7f8d7f400640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d800b3f70 0x7f8d80159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7ea00640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d8015fc10 0x7f8d8015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7fe00640 1 -- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d800b3f70 msgr2=0x7f8d80159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7fe00640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d800b3f70 0x7f8d80159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7fe00640 1 -- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d8015fc10 msgr2=0x7f8d8015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7fe00640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d8015fc10 0x7f8d8015d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7fe00640 1 -- 172.21.15.97:0/1228484490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d801599a0 con 0x7f8d8015fff0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d7fe00640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 0x7f8d8009cda0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800efc0 tx=0x7f8d78009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78018030 con 0x7f8d8015fff0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7800cde0 con 0x7f8d8015fff0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d80159c80 con 0x7f8d8015fff0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78003bb0 con 0x7f8d8015fff0 2024-03-06T07:32:23.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.162+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8015a520 con 0x7f8d8015fff0 2024-03-06T07:32:23.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.164+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d80159e50 con 0x7f8d8015fff0 2024-03-06T07:32:23.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.165+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d7801b070 con 0x7f8d8015fff0 2024-03-06T07:32:23.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.165+0000 7f8d77400640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d6006b4d0 0x7f8d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.166+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d7808ace0 con 0x7f8d8015fff0 2024-03-06T07:32:23.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.166+0000 7f8d7f400640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d6006b4d0 0x7f8d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.166+0000 7f8d7f400640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d6006b4d0 0x7f8d6006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8d70036000 tx=0x7f8d7003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.171+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d78010040 con 0x7f8d8015fff0 2024-03-06T07:32:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.367+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d8015be40 con 0x7f8d8015fff0 2024-03-06T07:32:23.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.368+0000 7f8d77400640 1 -- 172.21.15.97:0/1228484490 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d8015be40 con 0x7f8d8015fff0 2024-03-06T07:32:23.370 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d6006b4d0 msgr2=0x7f8d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d6006b4d0 0x7f8d6006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8d70036000 tx=0x7f8d7003b040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 msgr2=0x7f8d8009cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 0x7f8d8009cda0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800efc0 tx=0x7f8d78009f90 comp rx=0 tx=0).stop 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 shutdown_connections 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d6006b4d0 0x7f8d6006d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d8015fff0 0x7f8d8009cda0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d8015fc10 0x7f8d8015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 --2- 172.21.15.97:0/1228484490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d800b3f70 0x7f8d80159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 >> 172.21.15.97:0/1228484490 conn(0x7f8d800b1940 msgr2=0x7f8d801571e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 shutdown_connections 2024-03-06T07:32:23.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.371+0000 7f8d86200640 1 -- 172.21.15.97:0/1228484490 wait complete. 2024-03-06T07:32:23.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.595+0000 7faef4400640 1 Processor -- start 2024-03-06T07:32:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.595+0000 7faef4400640 1 -- start start 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.596+0000 7faef4400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faeec0c6460 0x7faeec0c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.596+0000 7faef4400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7faeec0c5960 0x7faeec165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.596+0000 7faef4400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec145b30 con 0x7faeec0c4930 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.596+0000 7faef4400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeec0c68c0 con 0x7faeec0c6460 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.596+0000 7faef4400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeec142ce0 con 0x7faeec0c5960 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faeec0c6460 0x7faeec0c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faeec0c6460 0x7faeec0c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38874/0 (socket says 172.21.15.97:38874) 2024-03-06T07:32:23.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 -- 172.21.15.97:0/1314433560 learned_addr learned my addr 172.21.15.97:0/1314433560 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:23.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef2400640 1 --2- 172.21.15.97:0/1314433560 >> v2:172.21.15.135:3300/0 conn(0x7faeec0c5960 0x7faeec165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 -- 172.21.15.97:0/1314433560 >> v2:172.21.15.135:3300/0 conn(0x7faeec0c5960 msgr2=0x7faeec165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 --2- 172.21.15.97:0/1314433560 >> v2:172.21.15.135:3300/0 conn(0x7faeec0c5960 0x7faeec165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 -- 172.21.15.97:0/1314433560 >> v1:172.21.15.135:6789/0 conn(0x7faeec0c4930 legacy=0x7faeec0c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef1000640 1 -- 172.21.15.97:0/1314433560 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0cbca0 con 0x7faeec0c6460 2024-03-06T07:32:23.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.597+0000 7faef2400640 1 --2- 172.21.15.97:0/1314433560 >> v2:172.21.15.135:3300/0 conn(0x7faeec0c5960 0x7faeec165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:23.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.598+0000 7faef1000640 1 --2- 172.21.15.97:0/1314433560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c6460 0x7faeec0c5420 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7faeec073310 tx=0x7faed802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508e30660bdf88b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faeebe00640 1 -- 172.21.15.97:0/1314433560 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed802e070 con 0x7faeec0c6460 2024-03-06T07:32:23.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faeebe00640 1 -- 172.21.15.97:0/1314433560 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8037070 con 0x7faeec0c6460 2024-03-06T07:32:23.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faeebe00640 1 -- 172.21.15.97:0/1314433560 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8031a00 con 0x7faeec0c6460 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faef4400640 1 -- 172.21.15.97:0/1314433560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c6460 msgr2=0x7faeec0c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faef4400640 1 --2- 172.21.15.97:0/1314433560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c6460 0x7faeec0c5420 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7faeec073310 tx=0x7faed802b880 comp rx=0 tx=0).stop 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faef4400640 1 -- 172.21.15.97:0/1314433560 shutdown_connections 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faef4400640 1 --2- 172.21.15.97:0/1314433560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c6460 0x7faeec0c5420 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faef4400640 1 --2- 172.21.15.97:0/1314433560 >> v2:172.21.15.135:3300/0 conn(0x7faeec0c5960 0x7faeec165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.599+0000 7faef4400640 1 -- 172.21.15.97:0/1314433560 >> 172.21.15.97:0/1314433560 conn(0x7faeec0cc4f0 msgr2=0x7faeec0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.600+0000 7faef4400640 1 -- 172.21.15.97:0/1314433560 shutdown_connections 2024-03-06T07:32:23.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.600+0000 7faef4400640 1 -- 172.21.15.97:0/1314433560 wait complete. 2024-03-06T07:32:23.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.601+0000 7faef4400640 1 Processor -- start 2024-03-06T07:32:23.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.601+0000 7faef4400640 1 -- start start 2024-03-06T07:32:23.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.602+0000 7faef4400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faeec0c4930 0x7faeec1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.602+0000 7faef4400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 0x7faeec13dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.602+0000 7faef4400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faeec0c6460 0x7faeec13e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.602+0000 7faef4400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0ca3b0 con 0x7faeec0c5960 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.602+0000 7faef4400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0ca0d0 con 0x7faeec0c6460 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.602+0000 7faef4400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec06e600 con 0x7faeec0c4930 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 0x7faeec13dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 0x7faeec13dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38878/0 (socket says 172.21.15.97:38878) 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 -- 172.21.15.97:0/2848506826 learned_addr learned my addr 172.21.15.97:0/2848506826 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1a00640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faeec0c4930 0x7faeec1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef2400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faeec0c6460 0x7faeec13e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 -- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faeec0c4930 msgr2=0x7faeec1588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faeec0c4930 0x7faeec1588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 -- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faeec0c6460 msgr2=0x7faeec13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faeec0c6460 0x7faeec13e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1000640 1 -- 172.21.15.97:0/2848506826 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0c8330 con 0x7faeec0c5960 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef1a00640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faeec0c4930 0x7faeec1588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:23.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.603+0000 7faef2400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faeec0c6460 0x7faeec13e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:23.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.604+0000 7faef1000640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 0x7faeec13dcf0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7faed8000c00 tx=0x7faed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.604+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8032880 con 0x7faeec0c5960 2024-03-06T07:32:23.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.604+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8032a20 con 0x7faeec0c5960 2024-03-06T07:32:23.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.604+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8032e10 con 0x7faeec0c5960 2024-03-06T07:32:23.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.604+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec06e290 con 0x7faeec0c5960 2024-03-06T07:32:23.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.604+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec0c7850 con 0x7faeec0c5960 2024-03-06T07:32:23.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.605+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec04f330 con 0x7faeec0c5960 2024-03-06T07:32:23.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.606+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faed804c060 con 0x7faeec0c5960 2024-03-06T07:32:23.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.608+0000 7faee9600640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.608+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faed80abb00 con 0x7faeec0c5960 2024-03-06T07:32:23.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.608+0000 7faef1a00640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.609+0000 7faef1a00640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faecc06b4d0 0x7faecc06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faeec0cabc0 tx=0x7faee0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.612+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faed8040070 con 0x7faeec0c5960 2024-03-06T07:32:23.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.842+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faeec0caf30 con 0x7faeec0c5960 2024-03-06T07:32:23.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.843+0000 7faee9600640 1 -- 172.21.15.97:0/2848506826 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7faeec145b30 con 0x7faeec0c5960 2024-03-06T07:32:23.845 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.845+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faecc06b4d0 msgr2=0x7faecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.845+0000 7faef4400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faecc06b4d0 0x7faecc06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faeec0cabc0 tx=0x7faee0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.845+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 msgr2=0x7faeec13dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.845+0000 7faef4400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 0x7faeec13dcf0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7faed8000c00 tx=0x7faed80046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 shutdown_connections 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faeec0c6460 0x7faeec13e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeec0c5960 0x7faeec13dcf0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 --2- 172.21.15.97:0/2848506826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faeec0c4930 0x7faeec1588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 >> 172.21.15.97:0/2848506826 conn(0x7faeec0cc4f0 msgr2=0x7faeec142f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 shutdown_connections 2024-03-06T07:32:23.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:23.846+0000 7faef4400640 1 -- 172.21.15.97:0/2848506826 wait complete. 2024-03-06T07:32:23.866 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:32:23.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:23.957 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:32:23.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.957 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:24.002 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:32:24.002 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:24.060 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:24.060 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:24.060 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:24.110 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:24.110 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:32:24.110 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:24.163 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:24.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.372+0000 7f46f5200640 1 Processor -- start 2024-03-06T07:32:24.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.372+0000 7f46f5200640 1 -- start start 2024-03-06T07:32:24.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.373+0000 7f46f5200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f46f0144c00 0x7f46f006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46f5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f015f9d0 0x7f46f015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46f5200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46f00ce0a0 0x7f46f00ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46f5200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f0160370 con 0x7f46f00ce0a0 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46f5200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f006ee70 con 0x7f46f0144c00 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46f5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f006cef0 con 0x7f46f015f9d0 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46f00ce0a0 0x7f46f00ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46f00ce0a0 0x7f46f00ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42078/0 (socket says 172.21.15.97:42078) 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 -- 172.21.15.97:0/3392345379 learned_addr learned my addr 172.21.15.97:0/3392345379 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ed400640 1 --2- 172.21.15.97:0/3392345379 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f015f9d0 0x7f46f015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ede00640 1 --2- 172.21.15.97:0/3392345379 >> v2:172.21.15.135:3300/0 conn(0x7f46f0144c00 0x7f46f006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 -- 172.21.15.97:0/3392345379 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f015f9d0 msgr2=0x7f46f015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 --2- 172.21.15.97:0/3392345379 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f015f9d0 0x7f46f015fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 -- 172.21.15.97:0/3392345379 >> v2:172.21.15.135:3300/0 conn(0x7f46f0144c00 msgr2=0x7f46f006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 --2- 172.21.15.97:0/3392345379 >> v2:172.21.15.135:3300/0 conn(0x7f46f0144c00 0x7f46f006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.374+0000 7f46ee800640 1 -- 172.21.15.97:0/3392345379 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f006d1d0 con 0x7f46f00ce0a0 2024-03-06T07:32:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.376+0000 7f46ee800640 1 --2- 172.21.15.97:0/3392345379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f00ce4f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f46f00b25f0 tx=0x7f46e802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f807b0ada4981d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.376+0000 7f46eca00640 1 -- 172.21.15.97:0/3392345379 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8035020 con 0x7f46f00ce0a0 2024-03-06T07:32:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.376+0000 7f46eca00640 1 -- 172.21.15.97:0/3392345379 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e8031880 con 0x7f46f00ce0a0 2024-03-06T07:32:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.376+0000 7f46eca00640 1 -- 172.21.15.97:0/3392345379 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8031c50 con 0x7f46f00ce0a0 2024-03-06T07:32:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.376+0000 7f46f5200640 1 -- 172.21.15.97:0/3392345379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 msgr2=0x7f46f00ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.376+0000 7f46f5200640 1 --2- 172.21.15.97:0/3392345379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f00ce4f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f46f00b25f0 tx=0x7f46e802b920 comp rx=0 tx=0).stop 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 -- 172.21.15.97:0/3392345379 shutdown_connections 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 --2- 172.21.15.97:0/3392345379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f00ce4f0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 --2- 172.21.15.97:0/3392345379 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f015f9d0 0x7f46f015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 --2- 172.21.15.97:0/3392345379 >> v2:172.21.15.135:3300/0 conn(0x7f46f0144c00 0x7f46f006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 -- 172.21.15.97:0/3392345379 >> 172.21.15.97:0/3392345379 conn(0x7f46f00b1730 msgr2=0x7f46f00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 -- 172.21.15.97:0/3392345379 shutdown_connections 2024-03-06T07:32:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.377+0000 7f46f5200640 1 -- 172.21.15.97:0/3392345379 wait complete. 2024-03-06T07:32:24.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.378+0000 7f46f5200640 1 Processor -- start 2024-03-06T07:32:24.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.378+0000 7f46f5200640 1 -- start start 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.379+0000 7f46f5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f013c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.379+0000 7f46f5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f0144c00 0x7f46f015dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.379+0000 7f46f5200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f015f9d0 0x7f46f013ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.379+0000 7f46f5200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f0160730 con 0x7f46f00ce0a0 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.379+0000 7f46f5200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f006d790 con 0x7f46f015f9d0 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.379+0000 7f46f5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f006cb50 con 0x7f46f0144c00 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f013c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f013c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42086/0 (socket says 172.21.15.97:42086) 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 -- 172.21.15.97:0/1732782771 learned_addr learned my addr 172.21.15.97:0/1732782771 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 -- 172.21.15.97:0/1732782771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f0144c00 msgr2=0x7f46f015dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f0144c00 0x7f46f015dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 -- 172.21.15.97:0/1732782771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f015f9d0 msgr2=0x7f46f013ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f015f9d0 0x7f46f013ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 -- 172.21.15.97:0/1732782771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f015e760 con 0x7f46f00ce0a0 2024-03-06T07:32:24.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46ede00640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f013c5a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f46e000ce40 tx=0x7f46e000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0003b90 con 0x7f46f00ce0a0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e0003d30 con 0x7f46f00ce0a0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.380+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0004260 con 0x7f46f00ce0a0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.381+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f015ea40 con 0x7f46f00ce0a0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.381+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f0141c40 con 0x7f46f00ce0a0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.382+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f00cd9a0 con 0x7f46f00ce0a0 2024-03-06T07:32:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.383+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46e0004400 con 0x7f46f00ce0a0 2024-03-06T07:32:24.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.383+0000 7f46e6000640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46cc06b4d0 0x7f46cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.384+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46e008adf0 con 0x7f46f00ce0a0 2024-03-06T07:32:24.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.384+0000 7f46ed400640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46cc06b4d0 0x7f46cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.384+0000 7f46ed400640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46cc06b4d0 0x7f46cc06d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f46d8006fd0 tx=0x7f46d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.388+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46f00cd9a0 con 0x7f46f00ce0a0 2024-03-06T07:32:24.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.623+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46f004f330 con 0x7f46f00ce0a0 2024-03-06T07:32:24.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.624+0000 7f46e6000640 1 -- 172.21.15.97:0/1732782771 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f46e0059c70 con 0x7f46f00ce0a0 2024-03-06T07:32:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:24.627 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:24.627 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46cc06b4d0 msgr2=0x7f46cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46cc06b4d0 0x7f46cc06d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f46d8006fd0 tx=0x7f46d8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 msgr2=0x7f46f013c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f013c5a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f46e000ce40 tx=0x7f46e000a660 comp rx=0 tx=0).stop 2024-03-06T07:32:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 shutdown_connections 2024-03-06T07:32:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46cc06b4d0 0x7f46cc06d990 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46f015f9d0 0x7f46f013ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46f0144c00 0x7f46f015dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 --2- 172.21.15.97:0/1732782771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46f00ce0a0 0x7f46f013c5a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 >> 172.21.15.97:0/1732782771 conn(0x7f46f00b1730 msgr2=0x7f46f00ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 shutdown_connections 2024-03-06T07:32:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.627+0000 7f46f5200640 1 -- 172.21.15.97:0/1732782771 wait complete. 2024-03-06T07:32:24.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:24.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.857+0000 7fadc5600640 1 Processor -- start 2024-03-06T07:32:24.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.857+0000 7fadc5600640 1 -- start start 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadc5600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fadc0144c00 0x7fadc006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadc5600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc015f9d0 0x7fadc015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadc5600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fadc00ce0a0 0x7fadc00ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadc5600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc0160370 con 0x7fadc0144c00 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadc5600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc006ee70 con 0x7fadc00ce0a0 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadc5600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc006cef0 con 0x7fadc015f9d0 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadbea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fadc0144c00 0x7fadc006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadbea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fadc0144c00 0x7fadc006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42098/0 (socket says 172.21.15.97:42098) 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadbea00640 1 -- 172.21.15.97:0/715367919 learned_addr learned my addr 172.21.15.97:0/715367919 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.859+0000 7fadbf400640 1 --2- 172.21.15.97:0/715367919 >> v2:172.21.15.135:3300/0 conn(0x7fadc00ce0a0 0x7fadc00ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.860+0000 7fadbe000640 1 --2- 172.21.15.97:0/715367919 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc015f9d0 0x7fadc015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.860+0000 7fadbea00640 1 -- 172.21.15.97:0/715367919 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc015f9d0 msgr2=0x7fadc015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.860+0000 7fadbea00640 1 --2- 172.21.15.97:0/715367919 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc015f9d0 0x7fadc015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.860+0000 7fadbea00640 1 -- 172.21.15.97:0/715367919 >> v2:172.21.15.135:3300/0 conn(0x7fadc00ce0a0 msgr2=0x7fadc00ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.860+0000 7fadbea00640 1 --2- 172.21.15.97:0/715367919 >> v2:172.21.15.135:3300/0 conn(0x7fadc00ce0a0 0x7fadc00ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.860+0000 7fadbea00640 1 -- 172.21.15.97:0/715367919 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc006d1d0 con 0x7fadc0144c00 2024-03-06T07:32:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.862+0000 7fadbea00640 1 --2- 172.21.15.97:0/715367919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc0144c00 0x7fadc006e930 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fadc00737b0 tx=0x7fadb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcf540835ef0430 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.862+0000 7fadbd600640 1 -- 172.21.15.97:0/715367919 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0035020 con 0x7fadc0144c00 2024-03-06T07:32:24.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.862+0000 7fadbd600640 1 -- 172.21.15.97:0/715367919 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb0031870 con 0x7fadc0144c00 2024-03-06T07:32:24.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.862+0000 7fadbd600640 1 -- 172.21.15.97:0/715367919 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0031c40 con 0x7fadc0144c00 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 -- 172.21.15.97:0/715367919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc0144c00 msgr2=0x7fadc006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 --2- 172.21.15.97:0/715367919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc0144c00 0x7fadc006e930 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fadc00737b0 tx=0x7fadb002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 -- 172.21.15.97:0/715367919 shutdown_connections 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 --2- 172.21.15.97:0/715367919 >> v2:172.21.15.135:3300/0 conn(0x7fadc00ce0a0 0x7fadc00ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 --2- 172.21.15.97:0/715367919 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc015f9d0 0x7fadc015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 --2- 172.21.15.97:0/715367919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc0144c00 0x7fadc006e930 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 -- 172.21.15.97:0/715367919 >> 172.21.15.97:0/715367919 conn(0x7fadc00b1730 msgr2=0x7fadc00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 -- 172.21.15.97:0/715367919 shutdown_connections 2024-03-06T07:32:24.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.863+0000 7fadc5600640 1 -- 172.21.15.97:0/715367919 wait complete. 2024-03-06T07:32:24.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.866+0000 7fadc5600640 1 Processor -- start 2024-03-06T07:32:24.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.866+0000 7fadc5600640 1 -- start start 2024-03-06T07:32:24.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.867+0000 7fadc5600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc00ce0a0 0x7fadc013fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.867+0000 7fadc5600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadc0144c00 0x7fadc0155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.867+0000 7fadc5600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 0x7fadc013e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.867+0000 7fadc5600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0160730 con 0x7fadc015f9d0 2024-03-06T07:32:24.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.867+0000 7fadc5600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc006d790 con 0x7fadc0144c00 2024-03-06T07:32:24.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.867+0000 7fadc5600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc006cb50 con 0x7fadc00ce0a0 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 0x7fadc013e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 0x7fadc013e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42100/0 (socket says 172.21.15.97:42100) 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 -- 172.21.15.97:0/2364629128 learned_addr learned my addr 172.21.15.97:0/2364629128 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbea00640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc00ce0a0 0x7fadc013fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 -- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc00ce0a0 msgr2=0x7fadc013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc00ce0a0 0x7fadc013fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 -- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadc0144c00 msgr2=0x7fadc0155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbe000640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadc0144c00 0x7fadc0155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadc0144c00 0x7fadc0155710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 -- 172.21.15.97:0/2364629128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0155c50 con 0x7fadc015f9d0 2024-03-06T07:32:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.868+0000 7fadbf400640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 0x7fadc013e100 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007d20 tx=0x7fadb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.869+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb8014420 con 0x7fadc015f9d0 2024-03-06T07:32:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.869+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb8019070 con 0x7fadc015f9d0 2024-03-06T07:32:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.869+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb8014820 con 0x7fadc015f9d0 2024-03-06T07:32:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.869+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc013ec70 con 0x7fadc015f9d0 2024-03-06T07:32:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.869+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc013f230 con 0x7fadc015f9d0 2024-03-06T07:32:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.870+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc004f330 con 0x7fadc015f9d0 2024-03-06T07:32:24.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.871+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadb80149c0 con 0x7fadc015f9d0 2024-03-06T07:32:24.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.872+0000 7fadb6a00640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fada006b4d0 0x7fada006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.872+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadb808a9d0 con 0x7fadc015f9d0 2024-03-06T07:32:24.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.872+0000 7fadbea00640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fada006b4d0 0x7fada006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.873+0000 7fadbea00640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fada006b4d0 0x7fada006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fadb0036000 tx=0x7fadb0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:24.876+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadb8059840 con 0x7fadc015f9d0 2024-03-06T07:32:25.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.103+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadc0072890 con 0x7fadc015f9d0 2024-03-06T07:32:25.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.106+0000 7fadb6a00640 1 -- 172.21.15.97:0/2364629128 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16648 (secure 0 0 0) 0x7fadb805f070 con 0x7fadc015f9d0 2024-03-06T07:32:25.108 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:25.108 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:11.891681+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.108+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fada006b4d0 msgr2=0x7fada006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.108+0000 7fadc5600640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fada006b4d0 0x7fada006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fadb0036000 tx=0x7fadb0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.108+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 msgr2=0x7fadc013e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.108+0000 7fadc5600640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 0x7fadc013e100 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007d20 tx=0x7fadb800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 shutdown_connections 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fada006b4d0 0x7fada006d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fadc015f9d0 0x7fadc013e100 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadc0144c00 0x7fadc0155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 --2- 172.21.15.97:0/2364629128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fadc00ce0a0 0x7fadc013fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 >> 172.21.15.97:0/2364629128 conn(0x7fadc00b1730 msgr2=0x7fadc01452e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 shutdown_connections 2024-03-06T07:32:25.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.109+0000 7fadc5600640 1 -- 172.21.15.97:0/2364629128 wait complete. 2024-03-06T07:32:25.130 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:08.169299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:09.359746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:25.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:25.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.328+0000 7fd8fd000640 1 Processor -- start 2024-03-06T07:32:25.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.328+0000 7fd8fd000640 1 -- start start 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8fd000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd8f80b3b40 0x7fd8f80b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8fd000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f806dda0 0x7fd8f806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8fd000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80b04b0 con 0x7fd8f8158770 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8fd000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f815ef20 con 0x7fd8f80b3b40 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8fd000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f815e8d0 con 0x7fd8f806dda0 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8f5e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd8f8158770 0x7fd8f80afa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43170/0 (socket says 172.21.15.97:43170) 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8f5e00640 1 -- 172.21.15.97:0/3874504956 learned_addr learned my addr 172.21.15.97:0/3874504956 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:25.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8ed400640 1 --2- 172.21.15.97:0/3874504956 >> v2:172.21.15.97:3300/0 conn(0x7fd8f80b3b40 0x7fd8f80b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8f6800640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f806dda0 0x7fd8f806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.330+0000 7fd8f5400640 1 -- 172.21.15.97:0/3874504956 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903017508 0 0) 0x7fd8f80b04b0 con 0x7fd8f8158770 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.331+0000 7fd8f5400640 1 -- 172.21.15.97:0/3874504956 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc003520 con 0x7fd8f8158770 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.331+0000 7fd8ed400640 1 -- 172.21.15.97:0/3874504956 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f806dda0 msgr2=0x7fd8f806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.331+0000 7fd8ed400640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f806dda0 0x7fd8f806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.331+0000 7fd8ed400640 1 -- 172.21.15.97:0/3874504956 >> v1:172.21.15.97:6789/0 conn(0x7fd8f8158770 legacy=0x7fd8f80afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.331+0000 7fd8ed400640 1 -- 172.21.15.97:0/3874504956 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f815ebb0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.331+0000 7fd8f6800640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f806dda0 0x7fd8f806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:25.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.332+0000 7fd8ed400640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f80b3fd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ae17c3f87deeb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.332+0000 7fd8f5400640 1 -- 172.21.15.97:0/3874504956 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00315a0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.332+0000 7fd8f5400640 1 -- 172.21.15.97:0/3874504956 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0037070 con 0x7fd8f80b3b40 2024-03-06T07:32:25.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.332+0000 7fd8f5400640 1 -- 172.21.15.97:0/3874504956 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031940 con 0x7fd8f80b3b40 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 -- 172.21.15.97:0/3874504956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 msgr2=0x7fd8f80b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f80b3fd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 -- 172.21.15.97:0/3874504956 shutdown_connections 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f806dda0 0x7fd8f806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 --2- 172.21.15.97:0/3874504956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f80b3fd0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 -- 172.21.15.97:0/3874504956 >> 172.21.15.97:0/3874504956 conn(0x7fd8f80b2840 msgr2=0x7fd8f80b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 -- 172.21.15.97:0/3874504956 shutdown_connections 2024-03-06T07:32:25.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.333+0000 7fd8fd000640 1 -- 172.21.15.97:0/3874504956 wait complete. 2024-03-06T07:32:25.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.335+0000 7fd8fd000640 1 Processor -- start 2024-03-06T07:32:25.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.335+0000 7fd8fd000640 1 -- start start 2024-03-06T07:32:25.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8fd000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8f806dda0 0x7fd8f8151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8fd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8fd000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f8158770 0x7fd8f81543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8fd000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80b0120 con 0x7fd8f80b3b40 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8fd000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80b5d70 con 0x7fd8f806dda0 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8fd000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f815f3b0 con 0x7fd8f8158770 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8ed400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8ed400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f8151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42112/0 (socket says 172.21.15.97:42112) 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8ed400640 1 -- 172.21.15.97:0/2819591259 learned_addr learned my addr 172.21.15.97:0/2819591259 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8f6800640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f8158770 0x7fd8f81543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.336+0000 7fd8f5e00640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8f806dda0 0x7fd8f8151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8ed400640 1 -- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f8158770 msgr2=0x7fd8f81543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8ed400640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f8158770 0x7fd8f81543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8ed400640 1 -- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8f806dda0 msgr2=0x7fd8f8151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8ed400640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8f806dda0 0x7fd8f8151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8ed400640 1 -- 172.21.15.97:0/2819591259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f81550c0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8ed400640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f8151bd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd8e002b880 tx=0x7fd8e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.337+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0032630 con 0x7fd8f80b3b40 2024-03-06T07:32:25.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.338+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e00327f0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.338+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f814f570 con 0x7fd8f80b3b40 2024-03-06T07:32:25.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.338+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0032be0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.338+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f814fb30 con 0x7fd8f80b3b40 2024-03-06T07:32:25.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.339+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f804f330 con 0x7fd8f80b3b40 2024-03-06T07:32:25.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.341+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8e003b3f0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.342+0000 7fd8eea00640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8d406b4d0 0x7fd8d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.342+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8e002f030 con 0x7fd8f80b3b40 2024-03-06T07:32:25.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.342+0000 7fd8f5e00640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8d406b4d0 0x7fd8d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.343+0000 7fd8f5e00640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8d406b4d0 0x7fd8d406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8000c00 tx=0x7fd8e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.346+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8e0040070 con 0x7fd8f80b3b40 2024-03-06T07:32:25.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.602+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8f8157cb0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.603+0000 7fd8eea00640 1 -- 172.21.15.97:0/2819591259 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8e007b0a0 con 0x7fd8f80b3b40 2024-03-06T07:32:25.605 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8d406b4d0 msgr2=0x7fd8d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8d406b4d0 0x7fd8d406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8000c00 tx=0x7fd8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 msgr2=0x7fd8f8151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f8151bd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd8e002b880 tx=0x7fd8e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 shutdown_connections 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd8d406b4d0 0x7fd8d406d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd8f8158770 0x7fd8f81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f80b3b40 0x7fd8f8151bd0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 --2- 172.21.15.97:0/2819591259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8f806dda0 0x7fd8f8151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 >> 172.21.15.97:0/2819591259 conn(0x7fd8f80b2840 msgr2=0x7fd8f815f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 shutdown_connections 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.606+0000 7fd8fd000640 1 -- 172.21.15.97:0/2819591259 wait complete. 2024-03-06T07:32:25.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:25.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.823+0000 7f3b1a800640 1 Processor -- start 2024-03-06T07:32:25.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.823+0000 7f3b1a800640 1 -- start start 2024-03-06T07:32:25.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b1a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b140cf060 0x7f3b140cd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b140cd740 0x7f3b140b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b1a800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b140d20f0 con 0x7f3b1415be60 2024-03-06T07:32:25.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b1a800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b140d23d0 con 0x7f3b140cf060 2024-03-06T07:32:25.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b1a800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140cca00 con 0x7f3b140cd740 2024-03-06T07:32:25.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b18a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b140cf060 0x7f3b140cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b18a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b140cf060 0x7f3b140cd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42116/0 (socket says 172.21.15.97:42116) 2024-03-06T07:32:25.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b18a00640 1 -- 172.21.15.97:0/1660000677 learned_addr learned my addr 172.21.15.97:0/1660000677 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:25.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.825+0000 7f3b19e00640 1 --2- 172.21.15.97:0/1660000677 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b140cd740 0x7f3b140b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.826+0000 7f3b18a00640 1 -- 172.21.15.97:0/1660000677 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b140cd740 msgr2=0x7f3b140b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.826+0000 7f3b18a00640 1 --2- 172.21.15.97:0/1660000677 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b140cd740 0x7f3b140b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.826+0000 7f3b18a00640 1 -- 172.21.15.97:0/1660000677 >> v1:172.21.15.135:6789/0 conn(0x7f3b1415be60 legacy=0x7f3b140d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.826+0000 7f3b18a00640 1 -- 172.21.15.97:0/1660000677 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b140ccce0 con 0x7f3b140cf060 2024-03-06T07:32:25.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.827+0000 7f3b18a00640 1 --2- 172.21.15.97:0/1660000677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cf060 0x7f3b140cd200 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3b000098a0 tx=0x7f3b0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869e7c228357df1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.827+0000 7f3b13e00640 1 -- 172.21.15.97:0/1660000677 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000317c0 con 0x7f3b140cf060 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.827+0000 7f3b13e00640 1 -- 172.21.15.97:0/1660000677 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00037070 con 0x7f3b140cf060 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.827+0000 7f3b13e00640 1 -- 172.21.15.97:0/1660000677 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00031b80 con 0x7f3b140cf060 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.827+0000 7f3b1a800640 1 -- 172.21.15.97:0/1660000677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cf060 msgr2=0x7f3b140cd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.827+0000 7f3b1a800640 1 --2- 172.21.15.97:0/1660000677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cf060 0x7f3b140cd200 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3b000098a0 tx=0x7f3b0002b860 comp rx=0 tx=0).stop 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.828+0000 7f3b1a800640 1 -- 172.21.15.97:0/1660000677 shutdown_connections 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.828+0000 7f3b1a800640 1 --2- 172.21.15.97:0/1660000677 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b140cd740 0x7f3b140b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.828+0000 7f3b1a800640 1 --2- 172.21.15.97:0/1660000677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cf060 0x7f3b140cd200 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.828+0000 7f3b1a800640 1 -- 172.21.15.97:0/1660000677 >> 172.21.15.97:0/1660000677 conn(0x7f3b1409a9f0 msgr2=0x7f3b1409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.828+0000 7f3b1a800640 1 -- 172.21.15.97:0/1660000677 shutdown_connections 2024-03-06T07:32:25.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.828+0000 7f3b1a800640 1 -- 172.21.15.97:0/1660000677 wait complete. 2024-03-06T07:32:25.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.830+0000 7f3b1a800640 1 Processor -- start 2024-03-06T07:32:25.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.830+0000 7f3b1a800640 1 -- start start 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 0x7f3b140c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b140cf060 0x7f3b140c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b1415be60 0x7f3b14142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b1a800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140cc7e0 con 0x7f3b140cd740 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b1a800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140cc660 con 0x7f3b140cf060 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b1a800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140b2000 con 0x7f3b1415be60 2024-03-06T07:32:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b19400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 0x7f3b140c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b19400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 0x7f3b140c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42124/0 (socket says 172.21.15.97:42124) 2024-03-06T07:32:25.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b19400640 1 -- 172.21.15.97:0/3935274655 learned_addr learned my addr 172.21.15.97:0/3935274655 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b19400640 1 -- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b1415be60 msgr2=0x7f3b14142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b19400640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b1415be60 0x7f3b14142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b18a00640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b140cf060 0x7f3b140c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.832+0000 7f3b19400640 1 -- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b140cf060 msgr2=0x7f3b140c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b19400640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b140cf060 0x7f3b140c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b19400640 1 -- 172.21.15.97:0/3935274655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b140c7180 con 0x7f3b140cd740 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b18a00640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b140cf060 0x7f3b140c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b19400640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 0x7f3b140c6940 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400ba50 tx=0x7f3b0400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04019070 con 0x7f3b140cd740 2024-03-06T07:32:25.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0400cc20 con 0x7f3b140cd740 2024-03-06T07:32:25.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04014440 con 0x7f3b140cd740 2024-03-06T07:32:25.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b140c8cc0 con 0x7f3b140cd740 2024-03-06T07:32:25.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.833+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b140c7420 con 0x7f3b140cd740 2024-03-06T07:32:25.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.835+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1404f330 con 0x7f3b140cd740 2024-03-06T07:32:25.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.836+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b0401c070 con 0x7f3b140cd740 2024-03-06T07:32:25.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.837+0000 7f3b11600640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3af406b4d0 0x7f3af406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.837+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b04089e30 con 0x7f3b140cd740 2024-03-06T07:32:25.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.837+0000 7f3b18a00640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3af406b4d0 0x7f3af406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.838+0000 7f3b18a00640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3af406b4d0 0x7f3af406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3b00036000 tx=0x7f3b00008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:25.842+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b040586d0 con 0x7f3b140cd740 2024-03-06T07:32:26.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.041+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3b140d2970 con 0x7f3b140cd740 2024-03-06T07:32:26.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.042+0000 7f3b11600640 1 -- 172.21.15.97:0/3935274655 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3b0405c770 con 0x7f3b140cd740 2024-03-06T07:32:26.044 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3af406b4d0 msgr2=0x7f3af406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3af406b4d0 0x7f3af406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3b00036000 tx=0x7f3b00008030 comp rx=0 tx=0).stop 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 msgr2=0x7f3b140c6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 0x7f3b140c6940 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400ba50 tx=0x7f3b0400bf10 comp rx=0 tx=0).stop 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 shutdown_connections 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3af406b4d0 0x7f3af406d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b1415be60 0x7f3b14142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b140cf060 0x7f3b140c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 --2- 172.21.15.97:0/3935274655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b140cd740 0x7f3b140c6940 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.044+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 >> 172.21.15.97:0/3935274655 conn(0x7f3b1409a9f0 msgr2=0x7f3b140b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.045+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 shutdown_connections 2024-03-06T07:32:26.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.045+0000 7f3b1a800640 1 -- 172.21.15.97:0/3935274655 wait complete. 2024-03-06T07:32:26.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:26.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.264+0000 7fa018000640 1 Processor -- start 2024-03-06T07:32:26.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.264+0000 7fa018000640 1 -- start start 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa018000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa01006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa018000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0100d02a0 0x7fa01006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa018000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0100d1fb0 con 0x7fa01015e360 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa018000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100d2a20 con 0x7fa0100d02a0 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa018000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01009b100 con 0x7fa01015df80 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa016000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0100d02a0 0x7fa01006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa014c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa01015e360 0x7fa0100cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43176/0 (socket says 172.21.15.97:43176) 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.266+0000 7fa014c00640 1 -- 172.21.15.97:0/3833406515 learned_addr learned my addr 172.21.15.97:0/3833406515 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.267+0000 7fa015600640 1 --2- 172.21.15.97:0/3833406515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa01006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.267+0000 7fa016000640 1 -- 172.21.15.97:0/3833406515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 msgr2=0x7fa01006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.267+0000 7fa016000640 1 --2- 172.21.15.97:0/3833406515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa01006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.267+0000 7fa016000640 1 -- 172.21.15.97:0/3833406515 >> v1:172.21.15.97:6789/0 conn(0x7fa01015e360 legacy=0x7fa0100cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.267+0000 7fa016000640 1 -- 172.21.15.97:0/3833406515 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01009b3a0 con 0x7fa0100d02a0 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.267+0000 7fa00fe00640 1 -- 172.21.15.97:0/3833406515 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122982109 0 0) 0x7fa0100d1fb0 con 0x7fa01015e360 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.268+0000 7fa016000640 1 --2- 172.21.15.97:0/3833406515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa01006ee40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa008009d90 tx=0x7fa00802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2530d287bbd8db1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.268+0000 7fa00fe00640 1 -- 172.21.15.97:0/3833406515 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008039070 con 0x7fa0100d02a0 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.268+0000 7fa00fe00640 1 -- 172.21.15.97:0/3833406515 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00802cbc0 con 0x7fa0100d02a0 2024-03-06T07:32:26.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.268+0000 7fa00fe00640 1 -- 172.21.15.97:0/3833406515 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008034440 con 0x7fa0100d02a0 2024-03-06T07:32:26.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 -- 172.21.15.97:0/3833406515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 msgr2=0x7fa01006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 --2- 172.21.15.97:0/3833406515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa01006ee40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa008009d90 tx=0x7fa00802be10 comp rx=0 tx=0).stop 2024-03-06T07:32:26.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 -- 172.21.15.97:0/3833406515 shutdown_connections 2024-03-06T07:32:26.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 --2- 172.21.15.97:0/3833406515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa01006ee40 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 --2- 172.21.15.97:0/3833406515 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa01006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 -- 172.21.15.97:0/3833406515 >> 172.21.15.97:0/3833406515 conn(0x7fa0100b7320 msgr2=0x7fa01009a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 -- 172.21.15.97:0/3833406515 shutdown_connections 2024-03-06T07:32:26.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.269+0000 7fa018000640 1 -- 172.21.15.97:0/3833406515 wait complete. 2024-03-06T07:32:26.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.271+0000 7fa018000640 1 Processor -- start 2024-03-06T07:32:26.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.271+0000 7fa018000640 1 -- start start 2024-03-06T07:32:26.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa018000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa0100ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa018000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa0101593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa018000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa01015e360 0x7fa01015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa018000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01006c610 con 0x7fa0100d02a0 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa018000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100d2500 con 0x7fa01015e360 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa018000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01009b680 con 0x7fa01015df80 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa0100ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa0100ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42142/0 (socket says 172.21.15.97:42142) 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 -- 172.21.15.97:0/3300756768 learned_addr learned my addr 172.21.15.97:0/3300756768 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 -- 172.21.15.97:0/3300756768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 msgr2=0x7fa0101593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa0101593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 -- 172.21.15.97:0/3300756768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa01015e360 msgr2=0x7fa01015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa01015e360 0x7fa01015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.272+0000 7fa015600640 1 -- 172.21.15.97:0/3300756768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100b00b0 con 0x7fa0100d02a0 2024-03-06T07:32:26.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.273+0000 7fa015600640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa0100ceab0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa00400d800 tx=0x7fa00400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.273+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00400b6c0 con 0x7fa0100d02a0 2024-03-06T07:32:26.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.273+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004019070 con 0x7fa0100d02a0 2024-03-06T07:32:26.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.273+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00400ba80 con 0x7fa0100d02a0 2024-03-06T07:32:26.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.273+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0100b0390 con 0x7fa0100d02a0 2024-03-06T07:32:26.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.273+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01015eab0 con 0x7fa0100d02a0 2024-03-06T07:32:26.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.274+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01004f330 con 0x7fa0100d02a0 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.275+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa00400bc20 con 0x7fa0100d02a0 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.276+0000 7fa00d600640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.276+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00408a750 con 0x7fa0100d02a0 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.276+0000 7fa014c00640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.277+0000 7fa014c00640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ff006b4d0 0x7f9ff006d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc007fd0 tx=0x7f9ffc007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.280+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa004059530 con 0x7fa0100d02a0 2024-03-06T07:32:26.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.475+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa010074b70 con 0x7fa0100d02a0 2024-03-06T07:32:26.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.475+0000 7fa00d600640 1 -- 172.21.15.97:0/3300756768 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa004058e30 con 0x7fa0100d02a0 2024-03-06T07:32:26.477 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:32:26.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ff006b4d0 msgr2=0x7f9ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ff006b4d0 0x7f9ff006d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc007fd0 tx=0x7f9ffc007920 comp rx=0 tx=0).stop 2024-03-06T07:32:26.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 msgr2=0x7fa0100ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa0100ceab0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa00400d800 tx=0x7fa00400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 shutdown_connections 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa01015e360 0x7fa01015d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa01015df80 0x7fa0101593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 --2- 172.21.15.97:0/3300756768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0100d02a0 0x7fa0100ceab0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 >> 172.21.15.97:0/3300756768 conn(0x7fa0100b7320 msgr2=0x7fa0100b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 shutdown_connections 2024-03-06T07:32:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.478+0000 7fa018000640 1 -- 172.21.15.97:0/3300756768 wait complete. 2024-03-06T07:32:26.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:26.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.692+0000 7fc2e7e00640 1 Processor -- start 2024-03-06T07:32:26.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.692+0000 7fc2e7e00640 1 -- start start 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.693+0000 7fc2e7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 0x7fc2e006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.693+0000 7fc2e7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc2e015f9d0 0x7fc2e015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.693+0000 7fc2e7e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e006ee70 con 0x7fc2e00ce0a0 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.693+0000 7fc2e7e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e0160370 con 0x7fc2e015f9d0 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.693+0000 7fc2e7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006cef0 con 0x7fc2e0144c00 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.694+0000 7fc2e5e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc2e00ce0a0 0x7fc2e00ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43180/0 (socket says 172.21.15.97:43180) 2024-03-06T07:32:26.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.694+0000 7fc2e5e00640 1 -- 172.21.15.97:0/3742215608 learned_addr learned my addr 172.21.15.97:0/3742215608 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.694+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371381031 0 0) 0x7fc2d8001490 con 0x7fc2e00ce0a0 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.694+0000 7fc2e5400640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 0x7fc2e006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.694+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c8003520 con 0x7fc2e00ce0a0 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.694+0000 7fc2e4a00640 1 --2- 172.21.15.97:0/3742215608 >> v2:172.21.15.135:3300/0 conn(0x7fc2e015f9d0 0x7fc2e015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154277078 0 0) 0x7fc2c8003520 con 0x7fc2e00ce0a0 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 msgr2=0x7fc2e006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 0x7fc2e006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 >> v2:172.21.15.135:3300/0 conn(0x7fc2e015f9d0 msgr2=0x7fc2e015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 --2- 172.21.15.97:0/3742215608 >> v2:172.21.15.135:3300/0 conn(0x7fc2e015f9d0 0x7fc2e015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e006d1d0 con 0x7fc2e00ce0a0 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc2d8001bc0 con 0x7fc2e00ce0a0 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 >> v1:172.21.15.97:6789/0 conn(0x7fc2e00ce0a0 legacy=0x7fc2e00ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2c80039b0 0x7fc2c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2c8004800 0x7fc2c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 0x7fc2c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006cef0 con 0x7fc2c80039b0 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800b9c0 con 0x7fc2c8004800 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800bb40 con 0x7fc2e0144c00 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e5400640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2c80039b0 0x7fc2c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e5400640 1 -- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 msgr2=0x7fc2c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e5400640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2e0144c00 0x7fc2c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e5400640 1 -- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2c8004800 msgr2=0x7fc2c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e4a00640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2c8004800 0x7fc2c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e5400640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2c8004800 0x7fc2c8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.695+0000 7fc2e5400640 1 -- 172.21.15.97:0/3742215608 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c8003520 con 0x7fc2c80039b0 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2e5400640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2c80039b0 0x7fc2c8003dd0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009ac0 tx=0x7fc2c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353a6c71c86268da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4039070 con 0x7fc2c80039b0 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d402c710 con 0x7fc2c80039b0 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2dfe00640 1 -- 172.21.15.97:0/3742215608 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4009690 con 0x7fc2c80039b0 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2e7e00640 1 -- 172.21.15.97:0/3742215608 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2c80039b0 msgr2=0x7fc2c8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/3742215608 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2c80039b0 0x7fc2c8003dd0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009ac0 tx=0x7fc2c8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.697+0000 7fc2e5e00640 1 -- 172.21.15.97:0/3742215608 reap_dead start 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.698+0000 7fc2e7e00640 1 -- 172.21.15.97:0/3742215608 shutdown_connections 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.698+0000 7fc2e7e00640 1 -- 172.21.15.97:0/3742215608 >> 172.21.15.97:0/3742215608 conn(0x7fc2e00b1730 msgr2=0x7fc2e00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.698+0000 7fc2e7e00640 1 -- 172.21.15.97:0/3742215608 shutdown_connections 2024-03-06T07:32:26.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.698+0000 7fc2e7e00640 1 -- 172.21.15.97:0/3742215608 wait complete. 2024-03-06T07:32:26.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.699+0000 7fc2e7e00640 1 Processor -- start 2024-03-06T07:32:26.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.699+0000 7fc2e7e00640 1 -- start start 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.700+0000 7fc2e7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2c80039b0 0x7fc2e00b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.700+0000 7fc2e7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 0x7fc2e00b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.700+0000 7fc2e7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2e00aeaa0 0x7fc2e013be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.700+0000 7fc2e7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e0160730 con 0x7fc2e015c420 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.700+0000 7fc2e7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006d790 con 0x7fc2e00aeaa0 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.700+0000 7fc2e7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006cb50 con 0x7fc2c80039b0 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 0x7fc2e00b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 0x7fc2e00b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42166/0 (socket says 172.21.15.97:42166) 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 -- 172.21.15.97:0/1347863078 learned_addr learned my addr 172.21.15.97:0/1347863078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:26.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e5400640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2c80039b0 0x7fc2e00b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e5e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2e00aeaa0 0x7fc2e013be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 -- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2c80039b0 msgr2=0x7fc2e00b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2c80039b0 0x7fc2e00b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 -- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2e00aeaa0 msgr2=0x7fc2e013be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2e00aeaa0 0x7fc2e013be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.701+0000 7fc2e4a00640 1 -- 172.21.15.97:0/1347863078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e01f4b90 con 0x7fc2e015c420 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.702+0000 7fc2e4a00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 0x7fc2e00b4650 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00e7e0 tx=0x7fc2cc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.702+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc017070 con 0x7fc2e015c420 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.702+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc00c6d0 con 0x7fc2e015c420 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.702+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc00f040 con 0x7fc2e015c420 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.702+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e01f4e50 con 0x7fc2e015c420 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.702+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e00b2fb0 con 0x7fc2e015c420 2024-03-06T07:32:26.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.703+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e004f330 con 0x7fc2e015c420 2024-03-06T07:32:26.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.704+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2cc01a070 con 0x7fc2e015c420 2024-03-06T07:32:26.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.705+0000 7fc2dd600640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2c006b4d0 0x7fc2c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.705+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2cc089d80 con 0x7fc2e015c420 2024-03-06T07:32:26.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.705+0000 7fc2e5400640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2c006b4d0 0x7fc2c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.706+0000 7fc2e5400640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2c006b4d0 0x7fc2c006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4036000 tx=0x7fc2d4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.713+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2cc058b60 con 0x7fc2e015c420 2024-03-06T07:32:26.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.957+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2e0072890 con 0x7fc2e015c420 2024-03-06T07:32:26.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.958+0000 7fc2dd600640 1 -- 172.21.15.97:0/1347863078 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fc2cc058460 con 0x7fc2e015c420 2024-03-06T07:32:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:26.960 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:26.960 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.960+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2c006b4d0 msgr2=0x7fc2c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.960+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2c006b4d0 0x7fc2c006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4036000 tx=0x7fc2d4030000 comp rx=0 tx=0).stop 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.960+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 msgr2=0x7fc2e00b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.960+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 0x7fc2e00b4650 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00e7e0 tx=0x7fc2cc00eca0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 shutdown_connections 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2c006b4d0 0x7fc2c006d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2e00aeaa0 0x7fc2e013be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e015c420 0x7fc2e00b4650 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 --2- 172.21.15.97:0/1347863078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2c80039b0 0x7fc2e00b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 >> 172.21.15.97:0/1347863078 conn(0x7fc2e00b1730 msgr2=0x7fc2e0145250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 shutdown_connections 2024-03-06T07:32:26.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:26.961+0000 7fc2e7e00640 1 -- 172.21.15.97:0/1347863078 wait complete. 2024-03-06T07:32:26.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:27.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.165+0000 7f9c3a400640 1 Processor -- start 2024-03-06T07:32:27.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.166+0000 7f9c3a400640 1 -- start start 2024-03-06T07:32:27.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.168+0000 7f9c3a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c340d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.168+0000 7f9c3a400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9c340d1e90 0x7f9c340d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.168+0000 7f9c3a400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c34161960 con 0x7f9c340d28a0 2024-03-06T07:32:27.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.168+0000 7f9c3a400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340cf6f0 con 0x7f9c340d1e90 2024-03-06T07:32:27.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.168+0000 7f9c3a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340cf9d0 con 0x7f9c34161580 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.169+0000 7f9c39a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9c340d28a0 0x7f9c340d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43190/0 (socket says 172.21.15.97:43190) 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.169+0000 7f9c39a00640 1 -- 172.21.15.97:0/2171690059 learned_addr learned my addr 172.21.15.97:0/2171690059 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.169+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115954154 0 0) 0x7f9c34161960 con 0x7f9c340d28a0 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.169+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1c003520 con 0x7f9c340d28a0 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.169+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2171690059 >> v2:172.21.15.135:3300/0 conn(0x7f9c340d1e90 0x7f9c340d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.169+0000 7f9c39000640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c340d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359543620 0 0) 0x7f9c1c003520 con 0x7f9c340d28a0 2024-03-06T07:32:27.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 msgr2=0x7f9c340d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c340d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 >> v2:172.21.15.135:3300/0 conn(0x7f9c340d1e90 msgr2=0x7f9c340d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 --2- 172.21.15.97:0/2171690059 >> v2:172.21.15.135:3300/0 conn(0x7f9c340d1e90 0x7f9c340d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340cfcb0 con 0x7f9c340d28a0 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9c2c0019f0 con 0x7f9c340d28a0 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c39000640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c340d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 >> v1:172.21.15.97:6789/0 conn(0x7f9c340d28a0 legacy=0x7f9c340d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340cf6f0 con 0x7f9c1c004800 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340cf9d0 con 0x7f9c1c0039b0 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c00b9f0 con 0x7f9c34161580 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33e00640 1 -- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 msgr2=0x7f9c1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c39a00640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.170+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.171+0000 7f9c33e00640 1 -- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c0039b0 msgr2=0x7f9c1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.171+0000 7f9c39000640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.171+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.171+0000 7f9c33e00640 1 -- 172.21.15.97:0/2171690059 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c003520 con 0x7f9c1c004800 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.171+0000 7f9c39a00640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c34161580 0x7f9c1c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.171+0000 7f9c39000640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c0039b0 0x7f9c1c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009ac0 tx=0x7f9c1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6111453d656de671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20039070 con 0x7f9c1c004800 2024-03-06T07:32:27.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2002c710 con 0x7f9c1c004800 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c33400640 1 -- 172.21.15.97:0/2171690059 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20009690 con 0x7f9c1c004800 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c3a400640 1 -- 172.21.15.97:0/2171690059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c1c004800 msgr2=0x7f9c1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2171690059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c1c004800 0x7f9c1c004c80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009ac0 tx=0x7f9c1c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c39a00640 1 -- 172.21.15.97:0/2171690059 reap_dead start 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c3a400640 1 -- 172.21.15.97:0/2171690059 shutdown_connections 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.172+0000 7f9c3a400640 1 -- 172.21.15.97:0/2171690059 >> 172.21.15.97:0/2171690059 conn(0x7f9c340d3d80 msgr2=0x7f9c340d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.173+0000 7f9c3a400640 1 -- 172.21.15.97:0/2171690059 shutdown_connections 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.173+0000 7f9c3a400640 1 -- 172.21.15.97:0/2171690059 wait complete. 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.174+0000 7f9c3a400640 1 Processor -- start 2024-03-06T07:32:27.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.174+0000 7f9c3a400640 1 -- start start 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c3a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c004800 0x7f9c34160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c3a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 0x7f9c34153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c3a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c340c7890 0x7f9c34154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c3a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340cf3a0 con 0x7f9c340c7410 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c3a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340b39a0 con 0x7f9c1c004800 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c3a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340b4ed0 con 0x7f9c340c7890 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.175+0000 7f9c33e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 0x7f9c34153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 0x7f9c34153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42182/0 (socket says 172.21.15.97:42182) 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 -- 172.21.15.97:0/2539168565 learned_addr learned my addr 172.21.15.97:0/2539168565 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 -- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c340c7890 msgr2=0x7f9c34154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c340c7890 0x7f9c34154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 -- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c004800 msgr2=0x7f9c34160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c39000640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c004800 0x7f9c34160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c004800 0x7f9c34160660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 -- 172.21.15.97:0/2539168565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340c9bb0 con 0x7f9c340c7410 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.176+0000 7f9c33e00640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 0x7f9c34153df0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9c20000c00 tx=0x7f9c20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.177+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20039050 con 0x7f9c340c7410 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.177+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20045070 con 0x7f9c340c7410 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.177+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c340c9e90 con 0x7f9c340c7410 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.177+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c200044e0 con 0x7f9c340c7410 2024-03-06T07:32:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.177+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c340ca450 con 0x7f9c340c7410 2024-03-06T07:32:27.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.178+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34068240 con 0x7f9c340c7410 2024-03-06T07:32:27.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.179+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c20003a50 con 0x7f9c340c7410 2024-03-06T07:32:27.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.180+0000 7f9c30c00640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c1406b500 0x7f9c1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.180+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c200ac190 con 0x7f9c340c7410 2024-03-06T07:32:27.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.180+0000 7f9c39000640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c1406b500 0x7f9c1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.181+0000 7f9c39000640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c1406b500 0x7f9c1406d9c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800e890 tx=0x7f9c28008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.184+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c34068240 con 0x7f9c340c7410 2024-03-06T07:32:27.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.378+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c3404f330 con 0x7f9c340c7410 2024-03-06T07:32:27.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.380+0000 7f9c30c00640 1 -- 172.21.15.97:0/2539168565 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f9c2004f020 con 0x7f9c340c7410 2024-03-06T07:32:27.383 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:27.383 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:11.891681+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:27.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c1406b500 msgr2=0x7f9c1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c1406b500 0x7f9c1406d9c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800e890 tx=0x7f9c28008040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 msgr2=0x7f9c34153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 0x7f9c34153df0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9c20000c00 tx=0x7f9c20004730 comp rx=0 tx=0).stop 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 shutdown_connections 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c1406b500 0x7f9c1406d9c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c340c7890 0x7f9c34154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c340c7410 0x7f9c34153df0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 --2- 172.21.15.97:0/2539168565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c1c004800 0x7f9c34160660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 >> 172.21.15.97:0/2539168565 conn(0x7f9c340d3d80 msgr2=0x7f9c3406cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 shutdown_connections 2024-03-06T07:32:27.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.383+0000 7f9c3a400640 1 -- 172.21.15.97:0/2539168565 wait complete. 2024-03-06T07:32:27.403 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:27.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:27.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.603+0000 7f1f2ca00640 1 Processor -- start 2024-03-06T07:32:27.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.603+0000 7f1f2ca00640 1 -- start start 2024-03-06T07:32:27.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.605+0000 7f1f2ca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1f280af1c0 0x7f1f2815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.605+0000 7f1f2ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 0x7f1f2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.605+0000 7f1f2ca00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f28071d30 con 0x7f1f2806d1a0 2024-03-06T07:32:27.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.605+0000 7f1f2ca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f28071490 con 0x7f1f280af1c0 2024-03-06T07:32:27.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.605+0000 7f1f2ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28071770 con 0x7f1f28070d20 2024-03-06T07:32:27.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 0x7f1f2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 0x7f1f2806cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:46348/0 (socket says 172.21.15.97:46348) 2024-03-06T07:32:27.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 -- 172.21.15.97:0/115206702 learned_addr learned my addr 172.21.15.97:0/115206702 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:27.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f25e00640 1 --2- 172.21.15.97:0/115206702 >> v2:172.21.15.135:3300/0 conn(0x7f1f280af1c0 0x7f1f2815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 -- 172.21.15.97:0/115206702 >> v2:172.21.15.135:3300/0 conn(0x7f1f280af1c0 msgr2=0x7f1f2815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 --2- 172.21.15.97:0/115206702 >> v2:172.21.15.135:3300/0 conn(0x7f1f280af1c0 0x7f1f2815eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 -- 172.21.15.97:0/115206702 >> v1:172.21.15.135:6789/0 conn(0x7f1f2806d1a0 legacy=0x7f1f2806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.606+0000 7f1f1fe00640 1 -- 172.21.15.97:0/115206702 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28071a50 con 0x7f1f28070d20 2024-03-06T07:32:27.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.608+0000 7f1f1fe00640 1 --2- 172.21.15.97:0/115206702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 0x7f1f2806cc60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009870 tx=0x7f1f1002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f8ee95d40bbb625 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.608+0000 7f1f25400640 1 -- 172.21.15.97:0/115206702 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10034070 con 0x7f1f28070d20 2024-03-06T07:32:27.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.608+0000 7f1f25400640 1 -- 172.21.15.97:0/115206702 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10030840 con 0x7f1f28070d20 2024-03-06T07:32:27.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.608+0000 7f1f25400640 1 -- 172.21.15.97:0/115206702 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10030c10 con 0x7f1f28070d20 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 -- 172.21.15.97:0/115206702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 msgr2=0x7f1f2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/115206702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 0x7f1f2806cc60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009870 tx=0x7f1f1002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 -- 172.21.15.97:0/115206702 shutdown_connections 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/115206702 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f28070d20 0x7f1f2806cc60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/115206702 >> v2:172.21.15.135:3300/0 conn(0x7f1f280af1c0 0x7f1f2815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 -- 172.21.15.97:0/115206702 >> 172.21.15.97:0/115206702 conn(0x7f1f280b1730 msgr2=0x7f1f280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 -- 172.21.15.97:0/115206702 shutdown_connections 2024-03-06T07:32:27.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.609+0000 7f1f2ca00640 1 -- 172.21.15.97:0/115206702 wait complete. 2024-03-06T07:32:27.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.611+0000 7f1f2ca00640 1 Processor -- start 2024-03-06T07:32:27.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.611+0000 7f1f2ca00640 1 -- start start 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f2ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f2806d1a0 0x7f1f28147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f2ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f28070d20 0x7f1f28144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f2ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 0x7f1f28143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f2ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2806ebe0 con 0x7f1f280af1c0 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f2ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280711a0 con 0x7f1f28070d20 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f2ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2806ed60 con 0x7f1f2806d1a0 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f26800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 0x7f1f28143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f26800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 0x7f1f28143ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42194/0 (socket says 172.21.15.97:42194) 2024-03-06T07:32:27.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f26800640 1 -- 172.21.15.97:0/2970591117 learned_addr learned my addr 172.21.15.97:0/2970591117 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f25e00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f2806d1a0 0x7f1f28147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f1fe00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f28070d20 0x7f1f28144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.613+0000 7f1f26800640 1 -- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f2806d1a0 msgr2=0x7f1f28147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f26800640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f2806d1a0 0x7f1f28147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f26800640 1 -- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f28070d20 msgr2=0x7f1f28144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f26800640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f28070d20 0x7f1f28144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f26800640 1 -- 172.21.15.97:0/2970591117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28144bb0 con 0x7f1f280af1c0 2024-03-06T07:32:27.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f26800640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 0x7f1f28143ba0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000c730 tx=0x7f1f2000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20003b70 con 0x7f1f280af1c0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20003d10 con 0x7f1f280af1c0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000aa90 con 0x7f1f280af1c0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f28144e90 con 0x7f1f280af1c0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.614+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f28145450 con 0x7f1f280af1c0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.616+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2804f330 con 0x7f1f280af1c0 2024-03-06T07:32:27.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.616+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f2000ac30 con 0x7f1f280af1c0 2024-03-06T07:32:27.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.617+0000 7f1f1e000640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1f0406b4d0 0x7f1f0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.617+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f2008a180 con 0x7f1f280af1c0 2024-03-06T07:32:27.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.617+0000 7f1f25e00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1f0406b4d0 0x7f1f0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.618+0000 7f1f25e00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1f0406b4d0 0x7f1f0406d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1f18006fd0 tx=0x7f1f18008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.621+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f28071d30 con 0x7f1f280af1c0 2024-03-06T07:32:27.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.972+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f280b4230 con 0x7f1f280af1c0 2024-03-06T07:32:27.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.973+0000 7f1f1e000640 1 -- 172.21.15.97:0/2970591117 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f1f20058f60 con 0x7f1f280af1c0 2024-03-06T07:32:27.975 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:27.975 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:27.975 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1f0406b4d0 msgr2=0x7f1f0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1f0406b4d0 0x7f1f0406d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1f18006fd0 tx=0x7f1f18008040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 msgr2=0x7f1f28143ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 0x7f1f28143ba0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000c730 tx=0x7f1f2000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 shutdown_connections 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1f0406b4d0 0x7f1f0406d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f280af1c0 0x7f1f28143ba0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f28070d20 0x7f1f28144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 --2- 172.21.15.97:0/2970591117 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f2806d1a0 0x7f1f28147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 >> 172.21.15.97:0/2970591117 conn(0x7f1f280b1730 msgr2=0x7f1f2815beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 shutdown_connections 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:27.975+0000 7f1f2ca00640 1 -- 172.21.15.97:0/2970591117 wait complete. 2024-03-06T07:32:27.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:28.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.171+0000 7f0bd3e00640 1 Processor -- start 2024-03-06T07:32:28.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.171+0000 7f0bd3e00640 1 -- start start 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.172+0000 7f0bd3e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 0x7f0bd40d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.172+0000 7f0bd3e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0bd40b3ec0 0x7f0bd40b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd3e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd4161960 con 0x7f0bd4161580 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd3e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd40d2ce0 con 0x7f0bd40b3ec0 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd3e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40cf6f0 con 0x7f0bd40d2360 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd2a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0bd4161580 0x7f0bd40d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43204/0 (socket says 172.21.15.97:43204) 2024-03-06T07:32:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd2a00640 1 -- 172.21.15.97:0/160178241 learned_addr learned my addr 172.21.15.97:0/160178241 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd3400640 1 --2- 172.21.15.97:0/160178241 >> v2:172.21.15.135:3300/0 conn(0x7f0bd40b3ec0 0x7f0bd40b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd2000640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 0x7f0bd40d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713819115 0 0) 0x7f0bd4161960 con 0x7f0bd4161580 2024-03-06T07:32:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.173+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8003520 con 0x7f0bd4161580 2024-03-06T07:32:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1516755953 0 0) 0x7f0bb8003520 con 0x7f0bd4161580 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 msgr2=0x7f0bd40d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 0x7f0bd40d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 >> v2:172.21.15.135:3300/0 conn(0x7f0bd40b3ec0 msgr2=0x7f0bd40b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 --2- 172.21.15.97:0/160178241 >> v2:172.21.15.135:3300/0 conn(0x7f0bd40b3ec0 0x7f0bd40b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40cf9d0 con 0x7f0bd4161580 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0bc4001670 con 0x7f0bd4161580 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 >> v1:172.21.15.97:6789/0 conn(0x7f0bd4161580 legacy=0x7f0bd40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb80039b0 0x7f0bb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb8004800 0x7f0bb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 0x7f0bb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40d2ce0 con 0x7f0bb8004800 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40cf6f0 con 0x7f0bb80039b0 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb800b9c0 con 0x7f0bd40d2360 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd2000640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb8004800 0x7f0bb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd2a00640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb80039b0 0x7f0bb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.174+0000 7f0bd3400640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 0x7f0bb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.175+0000 7f0bd2000640 1 -- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 msgr2=0x7f0bb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.175+0000 7f0bd2000640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bd40d2360 0x7f0bb80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.175+0000 7f0bd2000640 1 -- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb80039b0 msgr2=0x7f0bb8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.175+0000 7f0bd2000640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb80039b0 0x7f0bb8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.175+0000 7f0bd2000640 1 -- 172.21.15.97:0/160178241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8003520 con 0x7f0bb8004800 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.176+0000 7f0bd2000640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb8004800 0x7f0bb8004c80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009ac0 tx=0x7f0bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b82b355b58deb44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.176+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0039070 con 0x7f0bb8004800 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.176+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc002c6b0 con 0x7f0bb8004800 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.176+0000 7f0bd1600640 1 -- 172.21.15.97:0/160178241 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0009690 con 0x7f0bb8004800 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3e00640 1 -- 172.21.15.97:0/160178241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb8004800 msgr2=0x7f0bb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/160178241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb8004800 0x7f0bb8004c80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009ac0 tx=0x7f0bb8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3400640 1 -- 172.21.15.97:0/160178241 reap_dead start 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3e00640 1 -- 172.21.15.97:0/160178241 shutdown_connections 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3e00640 1 -- 172.21.15.97:0/160178241 >> 172.21.15.97:0/160178241 conn(0x7f0bd40d3d80 msgr2=0x7f0bd40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3e00640 1 -- 172.21.15.97:0/160178241 shutdown_connections 2024-03-06T07:32:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.177+0000 7f0bd3e00640 1 -- 172.21.15.97:0/160178241 wait complete. 2024-03-06T07:32:28.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.178+0000 7f0bd3e00640 1 Processor -- start 2024-03-06T07:32:28.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 -- start start 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bb8004800 0x7f0bd415b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd415b000 0x7f0bd41520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 0x7f0bd415f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b39a0 con 0x7f0bd40b90d0 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b36c0 con 0x7f0bd415b000 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.179+0000 7f0bd3e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40cf3a0 con 0x7f0bb8004800 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 0x7f0bd415f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 0x7f0bd415f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42206/0 (socket says 172.21.15.97:42206) 2024-03-06T07:32:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 -- 172.21.15.97:0/1763007145 learned_addr learned my addr 172.21.15.97:0/1763007145 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 -- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bb8004800 msgr2=0x7f0bd415b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd2a00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bb8004800 0x7f0bd415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd2000640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd415b000 0x7f0bd41520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bb8004800 0x7f0bd415b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 -- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd415b000 msgr2=0x7f0bd41520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd415b000 0x7f0bd41520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 -- 172.21.15.97:0/1763007145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40b8030 con 0x7f0bd40b90d0 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.180+0000 7f0bd3400640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 0x7f0bd415f850 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0bd408dff0 tx=0x7f0bcc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.181+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc01c070 con 0x7f0bd40b90d0 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.181+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc017400 con 0x7f0bd40b90d0 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.181+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0177f0 con 0x7f0bd40b90d0 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.181+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd40b8310 con 0x7f0bd40b90d0 2024-03-06T07:32:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.181+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd40b88d0 con 0x7f0bd40b90d0 2024-03-06T07:32:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.182+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd404f330 con 0x7f0bd40b90d0 2024-03-06T07:32:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.183+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bcc01f070 con 0x7f0bd40b90d0 2024-03-06T07:32:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.184+0000 7f0bcaa00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb406b4d0 0x7f0bb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.184+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0bcc08a2c0 con 0x7f0bd40b90d0 2024-03-06T07:32:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.184+0000 7f0bd2a00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb406b4d0 0x7f0bb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.184+0000 7f0bd2a00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb406b4d0 0x7f0bb406d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40057f0 tx=0x7f0bc4001430 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.188+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bcc012070 con 0x7f0bd40b90d0 2024-03-06T07:32:28.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.481+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd40d3b60 con 0x7f0bd40b90d0 2024-03-06T07:32:28.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.482+0000 7f0bcaa00640 1 -- 172.21.15.97:0/1763007145 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f0bcc058b60 con 0x7f0bd40b90d0 2024-03-06T07:32:28.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:28.485 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:10.358830+0000","modified":"2024-03-06T07:32:13.477240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6741},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6741":{"gid":6741,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3414007063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3414007063},{"type":"v1","addr":"172.21.15.97:6837","nonce":3414007063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6741,"qdb_cluster":[6741]},"id":6}]} 2024-03-06T07:32:28.485 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 51 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb406b4d0 msgr2=0x7f0bb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb406b4d0 0x7f0bb406d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40057f0 tx=0x7f0bc4001430 comp rx=0 tx=0).stop 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 msgr2=0x7f0bd415f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 0x7f0bd415f850 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0bd408dff0 tx=0x7f0bcc00f4a0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 shutdown_connections 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bb406b4d0 0x7f0bb406d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bd40b90d0 0x7f0bd415f850 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bd415b000 0x7f0bd41520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 --2- 172.21.15.97:0/1763007145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bb8004800 0x7f0bd415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 >> 172.21.15.97:0/1763007145 conn(0x7f0bd40d3d80 msgr2=0x7f0bd406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 shutdown_connections 2024-03-06T07:32:28.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.485+0000 7f0bd3e00640 1 -- 172.21.15.97:0/1763007145 wait complete. 2024-03-06T07:32:28.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:28.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.688+0000 7f2272800640 1 Processor -- start 2024-03-06T07:32:28.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.688+0000 7f2272800640 1 -- start start 2024-03-06T07:32:28.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2272800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f226c06b6d0 0x7f226c162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2272800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0cef70 con 0x7f226c0affe0 2024-03-06T07:32:28.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2272800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c15e3b0 con 0x7f226c15dfd0 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2272800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c0b0400 con 0x7f226c06b6d0 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2270a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f226c06b6d0 0x7f226c162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2270a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f226c06b6d0 0x7f226c162700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42220/0 (socket says 172.21.15.97:42220) 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2270a00640 1 -- 172.21.15.97:0/1458143234 learned_addr learned my addr 172.21.15.97:0/1458143234 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.690+0000 7f2271e00640 1 --1- 172.21.15.97:0/1458143234 >> v1:172.21.15.97:6789/0 conn(0x7f226c0affe0 0x7f226c0d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43214/0 (socket says 172.21.15.97:43214) 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.691+0000 7f2270a00640 1 -- 172.21.15.97:0/1458143234 >> v1:172.21.15.135:6789/0 conn(0x7f226c15dfd0 legacy=0x7f226c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.691+0000 7f2270a00640 1 -- 172.21.15.97:0/1458143234 >> v1:172.21.15.97:6789/0 conn(0x7f226c0affe0 legacy=0x7f226c0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.691+0000 7f2270a00640 1 -- 172.21.15.97:0/1458143234 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c161c60 con 0x7f226c06b6d0 2024-03-06T07:32:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.692+0000 7f2270a00640 1 --2- 172.21.15.97:0/1458143234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c06b6d0 0x7f226c162700 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2258009870 tx=0x7f225802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5867ae549c6c4d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.692+0000 7f226be00640 1 -- 172.21.15.97:0/1458143234 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580317d0 con 0x7f226c06b6d0 2024-03-06T07:32:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.692+0000 7f226be00640 1 -- 172.21.15.97:0/1458143234 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2258037070 con 0x7f226c06b6d0 2024-03-06T07:32:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.692+0000 7f226be00640 1 -- 172.21.15.97:0/1458143234 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258031b90 con 0x7f226c06b6d0 2024-03-06T07:32:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 -- 172.21.15.97:0/1458143234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c06b6d0 msgr2=0x7f226c162700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 --2- 172.21.15.97:0/1458143234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c06b6d0 0x7f226c162700 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2258009870 tx=0x7f225802b870 comp rx=0 tx=0).stop 2024-03-06T07:32:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 -- 172.21.15.97:0/1458143234 shutdown_connections 2024-03-06T07:32:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 --2- 172.21.15.97:0/1458143234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c06b6d0 0x7f226c162700 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 -- 172.21.15.97:0/1458143234 >> 172.21.15.97:0/1458143234 conn(0x7f226c06c0d0 msgr2=0x7f226c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 -- 172.21.15.97:0/1458143234 shutdown_connections 2024-03-06T07:32:28.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.693+0000 7f2272800640 1 -- 172.21.15.97:0/1458143234 wait complete. 2024-03-06T07:32:28.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.695+0000 7f2272800640 1 Processor -- start 2024-03-06T07:32:28.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.695+0000 7f2272800640 1 -- start start 2024-03-06T07:32:28.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2272800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f226c06b6d0 0x7f226c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2272800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f226c0affe0 0x7f226c0b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2272800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 0x7f226c160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2272800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c161f00 con 0x7f226c15dfd0 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2272800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c162270 con 0x7f226c06b6d0 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2272800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0ce8d0 con 0x7f226c0affe0 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2271e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 0x7f226c160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2271e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 0x7f226c160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42234/0 (socket says 172.21.15.97:42234) 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.697+0000 7f2271e00640 1 -- 172.21.15.97:0/4007766619 learned_addr learned my addr 172.21.15.97:0/4007766619 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2270a00640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f226c0affe0 0x7f226c0b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271400640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f226c06b6d0 0x7f226c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271e00640 1 -- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f226c0affe0 msgr2=0x7f226c0b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271e00640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f226c0affe0 0x7f226c0b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271e00640 1 -- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f226c06b6d0 msgr2=0x7f226c154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271e00640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f226c06b6d0 0x7f226c154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271e00640 1 -- 172.21.15.97:0/4007766619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c154b70 con 0x7f226c15dfd0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271400640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f226c06b6d0 0x7f226c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.698+0000 7f2271e00640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 0x7f226c160d60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f226400c780 tx=0x7f226400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.699+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264017070 con 0x7f226c15dfd0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.699+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400a640 con 0x7f226c15dfd0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.699+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c0b34d0 con 0x7f226c15dfd0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.699+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226400aa30 con 0x7f226c15dfd0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.699+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c15a440 con 0x7f226c15dfd0 2024-03-06T07:32:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.700+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c04f330 con 0x7f226c15dfd0 2024-03-06T07:32:28.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.702+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f226401a070 con 0x7f226c15dfd0 2024-03-06T07:32:28.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.703+0000 7f2269600640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f224c06b4d0 0x7f224c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.703+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2264089760 con 0x7f226c15dfd0 2024-03-06T07:32:28.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.704+0000 7f2271400640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f224c06b4d0 0x7f224c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.704+0000 7f2271400640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f224c06b4d0 0x7f224c06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2260006fd0 tx=0x7f2260008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.708+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f226c154b70 con 0x7f226c15dfd0 2024-03-06T07:32:28.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.984+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f226c0b5e60 con 0x7f226c15dfd0 2024-03-06T07:32:28.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.985+0000 7f2269600640 1 -- 172.21.15.97:0/4007766619 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16698 (secure 0 0 0) 0x7f2264057f60 con 0x7f226c15dfd0 2024-03-06T07:32:28.988 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:28.988 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":83,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:27.987995+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:08.169299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:09.359746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.1a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f224c06b4d0 msgr2=0x7f224c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f224c06b4d0 0x7f224c06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2260006fd0 tx=0x7f2260008040 comp rx=0 tx=0).stop 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 msgr2=0x7f226c160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 0x7f226c160d60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f226400c780 tx=0x7f226400cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 shutdown_connections 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f224c06b4d0 0x7f224c06d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f226c15dfd0 0x7f226c160d60 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f226c0affe0 0x7f226c0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 --2- 172.21.15.97:0/4007766619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f226c06b6d0 0x7f226c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 >> 172.21.15.97:0/4007766619 conn(0x7f226c06c0d0 msgr2=0x7f226c158840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.988+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 shutdown_connections 2024-03-06T07:32:28.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:28.989+0000 7f2272800640 1 -- 172.21.15.97:0/4007766619 wait complete. 2024-03-06T07:32:29.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:29.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.220+0000 7f4a6fc00640 1 Processor -- start 2024-03-06T07:32:29.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.220+0000 7f4a6fc00640 1 -- start start 2024-03-06T07:32:29.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6fc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4a68145b70 0x7f4a6815c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6fc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4a6815caa0 0x7f4a68162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6fc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a68166ec0 con 0x7f4a680cd5b0 2024-03-06T07:32:29.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6fc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6806cbe0 con 0x7f4a6815caa0 2024-03-06T07:32:29.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6fc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6806cec0 con 0x7f4a68145b70 2024-03-06T07:32:29.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6dc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4a6815caa0 0x7f4a68162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6dc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4a6815caa0 0x7f4a68162fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42244/0 (socket says 172.21.15.97:42244) 2024-03-06T07:32:29.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6dc00640 1 -- 172.21.15.97:0/2723255726 learned_addr learned my addr 172.21.15.97:0/2723255726 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:29.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.222+0000 7f4a6d200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4a680cd5b0 0x7f4a680cd990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43222/0 (socket says 172.21.15.97:43222) 2024-03-06T07:32:29.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.223+0000 7f4a67e00640 1 --2- 172.21.15.97:0/2723255726 >> v2:172.21.15.135:3300/0 conn(0x7f4a68145b70 0x7f4a6815c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.223+0000 7f4a6dc00640 1 -- 172.21.15.97:0/2723255726 >> v2:172.21.15.135:3300/0 conn(0x7f4a68145b70 msgr2=0x7f4a6815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.223+0000 7f4a6dc00640 1 --2- 172.21.15.97:0/2723255726 >> v2:172.21.15.135:3300/0 conn(0x7f4a68145b70 0x7f4a6815c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.223+0000 7f4a6dc00640 1 -- 172.21.15.97:0/2723255726 >> v1:172.21.15.97:6789/0 conn(0x7f4a680cd5b0 legacy=0x7f4a680cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.223+0000 7f4a6dc00640 1 -- 172.21.15.97:0/2723255726 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6806d1a0 con 0x7f4a6815caa0 2024-03-06T07:32:29.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.223+0000 7f4a67400640 1 -- 172.21.15.97:0/2723255726 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757017041 0 0) 0x7f4a68166ec0 con 0x7f4a680cd5b0 2024-03-06T07:32:29.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.224+0000 7f4a6dc00640 1 --2- 172.21.15.97:0/2723255726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a68162fb0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4a681671d0 tx=0x7f4a6002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79745c6126ea5d83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.224+0000 7f4a67400640 1 -- 172.21.15.97:0/2723255726 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6002cb20 con 0x7f4a6815caa0 2024-03-06T07:32:29.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.224+0000 7f4a67400640 1 -- 172.21.15.97:0/2723255726 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6002ccc0 con 0x7f4a6815caa0 2024-03-06T07:32:29.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.224+0000 7f4a67400640 1 -- 172.21.15.97:0/2723255726 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60034610 con 0x7f4a6815caa0 2024-03-06T07:32:29.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 -- 172.21.15.97:0/2723255726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 msgr2=0x7f4a68162fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/2723255726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a68162fb0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4a681671d0 tx=0x7f4a6002bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 -- 172.21.15.97:0/2723255726 shutdown_connections 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/2723255726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a68162fb0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/2723255726 >> v2:172.21.15.135:3300/0 conn(0x7f4a68145b70 0x7f4a6815c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 -- 172.21.15.97:0/2723255726 >> 172.21.15.97:0/2723255726 conn(0x7f4a680cc4f0 msgr2=0x7f4a680cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 -- 172.21.15.97:0/2723255726 shutdown_connections 2024-03-06T07:32:29.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.225+0000 7f4a6fc00640 1 -- 172.21.15.97:0/2723255726 wait complete. 2024-03-06T07:32:29.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.227+0000 7f4a6fc00640 1 Processor -- start 2024-03-06T07:32:29.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.228+0000 7f4a6fc00640 1 -- start start 2024-03-06T07:32:29.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.229+0000 7f4a6fc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a680cd5b0 0x7f4a681445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.229+0000 7f4a6fc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68145b70 0x7f4a6815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.229+0000 7f4a6fc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a680721a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.229+0000 7f4a6fc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a681634f0 con 0x7f4a6815caa0 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.229+0000 7f4a6fc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6815cef0 con 0x7f4a68145b70 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.229+0000 7f4a6fc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68163670 con 0x7f4a680cd5b0 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a680721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a680721a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42250/0 (socket says 172.21.15.97:42250) 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 -- 172.21.15.97:0/4143825886 learned_addr learned my addr 172.21.15.97:0/4143825886 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a67e00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68145b70 0x7f4a6815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6d200640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a680cd5b0 0x7f4a681445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 -- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a680cd5b0 msgr2=0x7f4a681445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a680cd5b0 0x7f4a681445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 -- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68145b70 msgr2=0x7f4a6815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68145b70 0x7f4a6815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 -- 172.21.15.97:0/4143825886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68143550 con 0x7f4a6815caa0 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6d200640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a680cd5b0 0x7f4a681445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.230+0000 7f4a6dc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a680721a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036040 tx=0x7f4a60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.231+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60043020 con 0x7f4a6815caa0 2024-03-06T07:32:29.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.231+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6003e450 con 0x7f4a6815caa0 2024-03-06T07:32:29.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.231+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a68143830 con 0x7f4a6815caa0 2024-03-06T07:32:29.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.231+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60009510 con 0x7f4a6815caa0 2024-03-06T07:32:29.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.231+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a68143df0 con 0x7f4a6815caa0 2024-03-06T07:32:29.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.232+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6804f330 con 0x7f4a6815caa0 2024-03-06T07:32:29.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.233+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a60005d10 con 0x7f4a6815caa0 2024-03-06T07:32:29.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.234+0000 7f4a64c00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a4806b4d0 0x7f4a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.235+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4a600ac960 con 0x7f4a6815caa0 2024-03-06T07:32:29.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.235+0000 7f4a6d200640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a4806b4d0 0x7f4a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.236+0000 7f4a6d200640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a4806b4d0 0x7f4a4806d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c000f60 tx=0x7f4a5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.239+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a6004d070 con 0x7f4a6815caa0 2024-03-06T07:32:29.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.473+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4a6806e8d0 con 0x7f4a6815caa0 2024-03-06T07:32:29.476 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:32:29.474+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:29.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.477+0000 7f4a64c00640 1 -- 172.21.15.97:0/4143825886 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4a6007b720 con 0x7f4a6815caa0 2024-03-06T07:32:29.479 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a4806b4d0 msgr2=0x7f4a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a4806b4d0 0x7f4a4806d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c000f60 tx=0x7f4a5c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 msgr2=0x7f4a680721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a680721a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036040 tx=0x7f4a60004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 shutdown_connections 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4a4806b4d0 0x7f4a4806d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a6815caa0 0x7f4a680721a0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a68145b70 0x7f4a6815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 --2- 172.21.15.97:0/4143825886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4a680cd5b0 0x7f4a681445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.480+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 >> 172.21.15.97:0/4143825886 conn(0x7f4a680cc4f0 msgr2=0x7f4a68147d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.481+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 shutdown_connections 2024-03-06T07:32:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.481+0000 7f4a6fc00640 1 -- 172.21.15.97:0/4143825886 wait complete. 2024-03-06T07:32:29.493 INFO:tasks.ceph.mds.a.smithi097.stderr: -4168> 2024-03-06T07:32:11.842+0000 7f41ad000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:29.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:29.550 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:32:29.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.685+0000 7f4d50e00640 1 Processor -- start 2024-03-06T07:32:29.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.685+0000 7f4d50e00640 1 -- start start 2024-03-06T07:32:29.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.686+0000 7f4d50e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4d4c0aef20 0x7f4d4c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.686+0000 7f4d50e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c15c1f0 con 0x7f4d4c15be10 2024-03-06T07:32:29.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.686+0000 7f4d50e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c06cac0 con 0x7f4d4c06da40 2024-03-06T07:32:29.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.686+0000 7f4d50e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4c0cd200 con 0x7f4d4c0aef20 2024-03-06T07:32:29.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.686+0000 7f4d4b400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4d4c15be10 0x7f4d4c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43246/0 (socket says 172.21.15.97:43246) 2024-03-06T07:32:29.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4b400640 1 -- 172.21.15.97:0/1126780377 learned_addr learned my addr 172.21.15.97:0/1126780377 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:29.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673617676 0 0) 0x7f4d4c15c1f0 con 0x7f4d4c15be10 2024-03-06T07:32:29.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003520 con 0x7f4d4c15be10 2024-03-06T07:32:29.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4be00640 1 --2- 172.21.15.97:0/1126780377 >> v2:172.21.15.135:3300/0 conn(0x7f4d4c0aef20 0x7f4d4c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3576221271 0 0) 0x7f4d30003520 con 0x7f4d4c15be10 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 >> v2:172.21.15.135:3300/0 conn(0x7f4d4c0aef20 msgr2=0x7f4d4c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 --2- 172.21.15.97:0/1126780377 >> v2:172.21.15.135:3300/0 conn(0x7f4d4c0aef20 0x7f4d4c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 >> v1:172.21.15.135:6789/0 conn(0x7f4d4c06da40 legacy=0x7f4d4c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c06cda0 con 0x7f4d4c15be10 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4d3c001670 con 0x7f4d4c15be10 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4be00640 1 --2- 172.21.15.97:0/1126780377 >> v2:172.21.15.135:3300/0 conn(0x7f4d4c0aef20 0x7f4d4c0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 >> v1:172.21.15.97:6789/0 conn(0x7f4d4c15be10 legacy=0x7f4d4c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d300039b0 0x7f4d30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.687+0000 7f4d4aa00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d30004800 0x7f4d30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4aa00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0cd200 con 0x7f4d300053c0 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c280 con 0x7f4d30004800 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c480 con 0x7f4d300039b0 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4be00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4b400640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d300039b0 0x7f4d30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d42a00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d30004800 0x7f4d30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4be00640 1 -- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d300039b0 msgr2=0x7f4d30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4be00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d300039b0 0x7f4d30003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4be00640 1 -- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d30004800 msgr2=0x7f4d30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4be00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d30004800 0x7f4d30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d4be00640 1 -- 172.21.15.97:0/1126780377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30003520 con 0x7f4d300053c0 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.688+0000 7f4d42a00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d30004800 0x7f4d30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.689+0000 7f4d4be00640 1 -- 172.21.15.97:0/1126780377 reap_dead start 2024-03-06T07:32:29.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.689+0000 7f4d4be00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d3000b9c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c490 tx=0x7f4d30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ad956b43a6ffaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44007c10 con 0x7f4d300053c0 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d44033460 con 0x7f4d300053c0 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d4aa00640 1 -- 172.21.15.97:0/1126780377 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44033870 con 0x7f4d300053c0 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 -- 172.21.15.97:0/1126780377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 msgr2=0x7f4d3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d3000b9c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c490 tx=0x7f4d30005830 comp rx=0 tx=0).stop 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 -- 172.21.15.97:0/1126780377 shutdown_connections 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 --2- 172.21.15.97:0/1126780377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d3000b9c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 -- 172.21.15.97:0/1126780377 >> 172.21.15.97:0/1126780377 conn(0x7f4d4c06afc0 msgr2=0x7f4d4c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 -- 172.21.15.97:0/1126780377 shutdown_connections 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.690+0000 7f4d50e00640 1 -- 172.21.15.97:0/1126780377 wait complete. 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.692+0000 7f4d50e00640 1 Processor -- start 2024-03-06T07:32:29.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.692+0000 7f4d50e00640 1 -- start start 2024-03-06T07:32:29.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d4c13f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d50e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c150520 0x7f4d4c14eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d50e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d4c13e780 0x7f4d4c159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d50e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0cda90 con 0x7f4d300053c0 2024-03-06T07:32:29.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d50e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c06d080 con 0x7f4d4c150520 2024-03-06T07:32:29.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d50e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0ccc20 con 0x7f4d4c13e780 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d4b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d4c13f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d4b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d4c13f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42292/0 (socket says 172.21.15.97:42292) 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.693+0000 7f4d4b400640 1 -- 172.21.15.97:0/12193657 learned_addr learned my addr 172.21.15.97:0/12193657 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d42a00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c150520 0x7f4d4c14eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d4b400640 1 -- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d4c13e780 msgr2=0x7f4d4c159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d4b400640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d4c13e780 0x7f4d4c159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d4b400640 1 -- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c150520 msgr2=0x7f4d4c14eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d4b400640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c150520 0x7f4d4c14eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d4b400640 1 -- 172.21.15.97:0/12193657 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c14f480 con 0x7f4d300053c0 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d4b400640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d4c13f0b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00e800 tx=0x7f4d3c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00c8d0 con 0x7f4d300053c0 2024-03-06T07:32:29.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c018070 con 0x7f4d300053c0 2024-03-06T07:32:29.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00ccb0 con 0x7f4d300053c0 2024-03-06T07:32:29.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.694+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c14f760 con 0x7f4d300053c0 2024-03-06T07:32:29.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.695+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c14fd20 con 0x7f4d300053c0 2024-03-06T07:32:29.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.696+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c150210 con 0x7f4d300053c0 2024-03-06T07:32:29.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.696+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d3c01d460 con 0x7f4d300053c0 2024-03-06T07:32:29.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.697+0000 7f4d43e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d2806b4d0 0x7f4d2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.697+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f4d3c08a050 con 0x7f4d300053c0 2024-03-06T07:32:29.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.698+0000 7f4d42a00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d2806b4d0 0x7f4d2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.698+0000 7f4d42a00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d2806b4d0 0x7f4d2806d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0cc3b0 tx=0x7f4d34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.702+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d3c058ef0 con 0x7f4d300053c0 2024-03-06T07:32:29.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:29.936+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4d4c1608c0 con 0x7f4d300053c0 2024-03-06T07:32:30.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.481+0000 7f4d43e00640 1 -- 172.21.15.97:0/12193657 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4d3c0587f0 con 0x7f4d300053c0 2024-03-06T07:32:30.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.485+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d2806b4d0 msgr2=0x7f4d2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.485+0000 7f4d50e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d2806b4d0 0x7f4d2806d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0cc3b0 tx=0x7f4d34009040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.485+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 msgr2=0x7f4d4c13f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.485+0000 7f4d50e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d4c13f0b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00e800 tx=0x7f4d3c00ecc0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 shutdown_connections 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d2806b4d0 0x7f4d2806d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d4c13e780 0x7f4d4c159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c150520 0x7f4d4c14eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 --2- 172.21.15.97:0/12193657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d300053c0 0x7f4d4c13f0b0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 >> 172.21.15.97:0/12193657 conn(0x7f4d4c06afc0 msgr2=0x7f4d4c0cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 shutdown_connections 2024-03-06T07:32:30.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.486+0000 7f4d50e00640 1 -- 172.21.15.97:0/12193657 wait complete. 2024-03-06T07:32:30.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:30.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.717+0000 7f50fd600640 1 Processor -- start 2024-03-06T07:32:30.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.717+0000 7f50fd600640 1 -- start start 2024-03-06T07:32:30.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50fd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f80cd450 0x7f50f806d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50fd600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f50f815c440 0x7f50f806db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50fd600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f8072970 con 0x7f50f8145750 2024-03-06T07:32:30.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50fd600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f80cd980 con 0x7f50f815c440 2024-03-06T07:32:30.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50fd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80cdbd0 con 0x7f50f80cd450 2024-03-06T07:32:30.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50f7400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f50f815c440 0x7f50f806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50f6000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f80cd450 0x7f50f806d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50f6a00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f50f8145750 0x7f50f8072260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:56594/0 (socket says 172.21.15.97:56594) 2024-03-06T07:32:30.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.719+0000 7f50f6a00640 1 -- 172.21.15.97:0/2305641825 learned_addr learned my addr 172.21.15.97:0/2305641825 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:30.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.720+0000 7f50f7400640 1 -- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f80cd450 msgr2=0x7f50f806d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.720+0000 7f50f7400640 1 --2- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f80cd450 0x7f50f806d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.720+0000 7f50f7400640 1 -- 172.21.15.97:0/2305641825 >> v1:172.21.15.135:6789/0 conn(0x7f50f8145750 legacy=0x7f50f8072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.720+0000 7f50f7400640 1 -- 172.21.15.97:0/2305641825 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80cdeb0 con 0x7f50f815c440 2024-03-06T07:32:30.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.720+0000 7f50f6000640 1 --2- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f80cd450 0x7f50f806d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:30.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.722+0000 7f50f7400640 1 --2- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f806db60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009d90 tx=0x7f50f002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=919789d6a158fc63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.722+0000 7f50f5600640 1 -- 172.21.15.97:0/2305641825 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0039070 con 0x7f50f815c440 2024-03-06T07:32:30.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.722+0000 7f50f5600640 1 -- 172.21.15.97:0/2305641825 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f002cbd0 con 0x7f50f815c440 2024-03-06T07:32:30.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.722+0000 7f50f5600640 1 -- 172.21.15.97:0/2305641825 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0034440 con 0x7f50f815c440 2024-03-06T07:32:30.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 -- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 msgr2=0x7f50f806db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 --2- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f806db60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009d90 tx=0x7f50f002be20 comp rx=0 tx=0).stop 2024-03-06T07:32:30.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 -- 172.21.15.97:0/2305641825 shutdown_connections 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 --2- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f80cd450 0x7f50f806d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 --2- 172.21.15.97:0/2305641825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f806db60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 -- 172.21.15.97:0/2305641825 >> 172.21.15.97:0/2305641825 conn(0x7f50f80cfa80 msgr2=0x7f50f80cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 -- 172.21.15.97:0/2305641825 shutdown_connections 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.723+0000 7f50fd600640 1 -- 172.21.15.97:0/2305641825 wait complete. 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.725+0000 7f50fd600640 1 Processor -- start 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.725+0000 7f50fd600640 1 -- start start 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50fd600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 0x7f50f8137570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50fd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f8145750 0x7f50f8142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50fd600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f813eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50fd600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8165ab0 con 0x7f50f80cd450 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50fd600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80ce470 con 0x7f50f815c440 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50fd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8165c30 con 0x7f50f8145750 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 0x7f50f8137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 0x7f50f8137570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42300/0 (socket says 172.21.15.97:42300) 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50f6a00640 1 -- 172.21.15.97:0/1051013283 learned_addr learned my addr 172.21.15.97:0/1051013283 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.726+0000 7f50f7400640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f813eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6000640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f8145750 0x7f50f8142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6a00640 1 -- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f8145750 msgr2=0x7f50f8142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6a00640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f8145750 0x7f50f8142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6a00640 1 -- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 msgr2=0x7f50f813eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6a00640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f813eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6a00640 1 -- 172.21.15.97:0/1051013283 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f813c620 con 0x7f50f80cd450 2024-03-06T07:32:30.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.727+0000 7f50f6a00640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 0x7f50f8137570 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c400 tx=0x7f50e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.728+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8017070 con 0x7f50f80cd450 2024-03-06T07:32:30.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.728+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e8005ce0 con 0x7f50f80cd450 2024-03-06T07:32:30.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.728+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f80cea00 con 0x7f50f80cd450 2024-03-06T07:32:30.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.728+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e80072f0 con 0x7f50f80cd450 2024-03-06T07:32:30.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.728+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f80cefc0 con 0x7f50f80cd450 2024-03-06T07:32:30.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.729+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f804f3a0 con 0x7f50f80cd450 2024-03-06T07:32:30.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.730+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50e801a070 con 0x7f50f80cd450 2024-03-06T07:32:30.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.731+0000 7f50eea00640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f50d806b4d0 0x7f50d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.731+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f50e80895b0 con 0x7f50f80cd450 2024-03-06T07:32:30.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.731+0000 7f50f6000640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f50d806b4d0 0x7f50d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.732+0000 7f50f6000640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f50d806b4d0 0x7f50d806d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009560 tx=0x7f50e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.736+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50e8057d80 con 0x7f50f80cd450 2024-03-06T07:32:30.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:30.932+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f50f80d0110 con 0x7f50f80cd450 2024-03-06T07:32:31.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.484+0000 7f50eea00640 1 -- 172.21.15.97:0/1051013283 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) v1 ==== 181+0+0 (secure 0 0 0) 0x7f50e805e0d0 con 0x7f50f80cd450 2024-03-06T07:32:31.486 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:31.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f50d806b4d0 msgr2=0x7f50d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f50d806b4d0 0x7f50d806d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009560 tx=0x7f50e4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 msgr2=0x7f50f8137570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 0x7f50f8137570 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c400 tx=0x7f50e800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 shutdown_connections 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f50d806b4d0 0x7f50d806d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50f815c440 0x7f50f813eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50f8145750 0x7f50f8142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 --2- 172.21.15.97:0/1051013283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f80cd450 0x7f50f8137570 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.490+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 >> 172.21.15.97:0/1051013283 conn(0x7f50f80cfa80 msgr2=0x7f50f815b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.493+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 shutdown_connections 2024-03-06T07:32:31.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.494+0000 7f50fd600640 1 -- 172.21.15.97:0/1051013283 wait complete. 2024-03-06T07:32:31.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:31.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.717+0000 7f2472600640 1 Processor -- start 2024-03-06T07:32:31.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.717+0000 7f2472600640 1 -- start start 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.718+0000 7f2472600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 0x7f246c15fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.718+0000 7f2472600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c15ff40 0x7f246c06dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f2472600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0cf1e0 con 0x7f246c0ced20 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f2472600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c1603b0 con 0x7f246c15ff40 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f2472600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c070e40 con 0x7f246c145980 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f2462a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f246c0ced20 0x7f246c0712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43248/0 (socket says 172.21.15.97:43248) 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f2462a00640 1 -- 172.21.15.97:0/883164851 learned_addr learned my addr 172.21.15.97:0/883164851 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f246b400640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 0x7f246c15fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f246be00640 1 --2- 172.21.15.97:0/883164851 >> v2:172.21.15.135:3300/0 conn(0x7f246c15ff40 0x7f246c06dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.719+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092258839 0 0) 0x7f246c0cf1e0 con 0x7f246c0ced20 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2450003520 con 0x7f246c0ced20 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626206200 0 0) 0x7f2450003520 con 0x7f246c0ced20 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 msgr2=0x7f246c15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 0x7f246c15fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 >> v2:172.21.15.135:3300/0 conn(0x7f246c15ff40 msgr2=0x7f246c06dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 --2- 172.21.15.97:0/883164851 >> v2:172.21.15.135:3300/0 conn(0x7f246c15ff40 0x7f246c06dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c0cf4c0 con 0x7f246c0ced20 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2454001670 con 0x7f246c0ced20 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 >> v1:172.21.15.97:6789/0 conn(0x7f246c0ced20 legacy=0x7f246c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2450003a20 0x7f2450003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 0x7f2450004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450004d40 0x7f24500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c1603b0 con 0x7f2450003a20 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c070e40 con 0x7f2450004d40 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245000b9c0 con 0x7f246c145980 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.720+0000 7f246b400640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2450003a20 0x7f2450003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.721+0000 7f2462a00640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 0x7f2450004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.721+0000 7f246b400640 1 -- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 msgr2=0x7f2450004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.721+0000 7f246b400640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c145980 0x7f2450004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.721+0000 7f246b400640 1 -- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450004d40 msgr2=0x7f24500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.721+0000 7f246b400640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450004d40 0x7f24500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.721+0000 7f246b400640 1 -- 172.21.15.97:0/883164851 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2450003520 con 0x7f2450003a20 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.722+0000 7f246b400640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2450003a20 0x7f2450003e40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f245c009ac0 tx=0x7f2450005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca65b0da8b14abf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.722+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c039070 con 0x7f2450003a20 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.722+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c02c6b0 con 0x7f2450003a20 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.722+0000 7f246aa00640 1 -- 172.21.15.97:0/883164851 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c009690 con 0x7f2450003a20 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.722+0000 7f2472600640 1 -- 172.21.15.97:0/883164851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2450003a20 msgr2=0x7f2450003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.722+0000 7f2472600640 1 --2- 172.21.15.97:0/883164851 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2450003a20 0x7f2450003e40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f245c009ac0 tx=0x7f2450005830 comp rx=0 tx=0).stop 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.723+0000 7f246be00640 1 -- 172.21.15.97:0/883164851 reap_dead start 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.723+0000 7f2472600640 1 -- 172.21.15.97:0/883164851 shutdown_connections 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.723+0000 7f2472600640 1 -- 172.21.15.97:0/883164851 >> 172.21.15.97:0/883164851 conn(0x7f246c0cfa80 msgr2=0x7f246c0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.723+0000 7f2472600640 1 -- 172.21.15.97:0/883164851 shutdown_connections 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.723+0000 7f2472600640 1 -- 172.21.15.97:0/883164851 wait complete. 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.724+0000 7f2472600640 1 Processor -- start 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 -- start start 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450003a20 0x7f246c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 0x7f246c1e6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c1428e0 0x7f246c1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c160740 con 0x7f246c142460 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c070ad0 con 0x7f2450003a20 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.725+0000 7f2472600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c1608c0 con 0x7f246c1428e0 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 0x7f246c1e6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 0x7f246c1e6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42310/0 (socket says 172.21.15.97:42310) 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 -- 172.21.15.97:0/4226922830 learned_addr learned my addr 172.21.15.97:0/4226922830 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 -- 172.21.15.97:0/4226922830 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c1428e0 msgr2=0x7f246c1f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c1428e0 0x7f246c1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 -- 172.21.15.97:0/4226922830 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450003a20 msgr2=0x7f246c15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450003a20 0x7f246c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 -- 172.21.15.97:0/4226922830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c1e7e00 con 0x7f246c142460 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.726+0000 7f2462a00640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 0x7f246c1e6a40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2454000c00 tx=0x7f245400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.727+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245400c5b0 con 0x7f246c142460 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.727+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454018070 con 0x7f246c142460 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.727+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245400c990 con 0x7f246c142460 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.727+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c1e59a0 con 0x7f246c142460 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.727+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246c1e5f60 con 0x7f246c142460 2024-03-06T07:32:31.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.728+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246c04f330 con 0x7f246c142460 2024-03-06T07:32:31.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.729+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f245400cb30 con 0x7f246c142460 2024-03-06T07:32:31.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.730+0000 7f2463e00640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f244806b550 0x7f244806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.730+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f2454089950 con 0x7f246c142460 2024-03-06T07:32:31.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.730+0000 7f246b400640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f244806b550 0x7f244806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.731+0000 7f246b400640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f244806b550 0x7f244806da10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f245c036000 tx=0x7f245c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.735+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2454022250 con 0x7f246c142460 2024-03-06T07:32:31.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:31.930+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f246c072920 con 0x7f246c142460 2024-03-06T07:32:32.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.497+0000 7f2463e00640 1 -- 172.21.15.97:0/4226922830 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2454089030 con 0x7f246c142460 2024-03-06T07:32:32.500 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:32.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f244806b550 msgr2=0x7f244806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f244806b550 0x7f244806da10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f245c036000 tx=0x7f245c02e000 comp rx=0 tx=0).stop 2024-03-06T07:32:32.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 msgr2=0x7f246c1e6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 0x7f246c1e6a40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2454000c00 tx=0x7f245400eb90 comp rx=0 tx=0).stop 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 shutdown_connections 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f244806b550 0x7f244806da10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f246c1428e0 0x7f246c1f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f246c142460 0x7f246c1e6a40 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.501+0000 7f2472600640 1 --2- 172.21.15.97:0/4226922830 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450003a20 0x7f246c15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.502+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 >> 172.21.15.97:0/4226922830 conn(0x7f246c0cfa80 msgr2=0x7f246c0b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.502+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 shutdown_connections 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.502+0000 7f2472600640 1 -- 172.21.15.97:0/4226922830 wait complete. 2024-03-06T07:32:32.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:32.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.724+0000 7fc68ee00640 1 Processor -- start 2024-03-06T07:32:32.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.724+0000 7fc68ee00640 1 -- start start 2024-03-06T07:32:32.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68ee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 0x7fc6880cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68ee00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6880b3560 con 0x7fc68806dbf0 2024-03-06T07:32:32.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68ee00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6880cc480 con 0x7fc6880cfd90 2024-03-06T07:32:32.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68ee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880d01f0 con 0x7fc688159570 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68e400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc68806dbf0 0x7fc6880ce3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43252/0 (socket says 172.21.15.97:43252) 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68e400640 1 -- 172.21.15.97:0/770817505 learned_addr learned my addr 172.21.15.97:0/770817505 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.726+0000 7fc68da00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 0x7fc6880cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653306325 0 0) 0x7fc6880b3560 con 0x7fc68806dbf0 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc66c003520 con 0x7fc68806dbf0 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570819871 0 0) 0x7fc66c003520 con 0x7fc68806dbf0 2024-03-06T07:32:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 msgr2=0x7fc6880cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 0x7fc6880cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 >> v1:172.21.15.135:6789/0 conn(0x7fc6880cfd90 legacy=0x7fc68806d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6880cc760 con 0x7fc68806dbf0 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc680001670 con 0x7fc68806dbf0 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 >> v1:172.21.15.97:6789/0 conn(0x7fc68806dbf0 legacy=0x7fc6880ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66c0039e0 0x7fc66c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc66c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 0x7fc66c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c00ba70 con 0x7fc66c004800 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c00bbf0 con 0x7fc66c0039e0 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.727+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c00bdf0 con 0x7fc688159570 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68d000640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc66c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68da00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66c0039e0 0x7fc66c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68e400640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 0x7fc66c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68d000640 1 -- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 msgr2=0x7fc66c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68d000640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688159570 0x7fc66c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68d000640 1 -- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66c0039e0 msgr2=0x7fc66c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68d000640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66c0039e0 0x7fc66c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.728+0000 7fc68d000640 1 -- 172.21.15.97:0/770817505 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c003520 con 0x7fc66c004800 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.729+0000 7fc68d000640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc66c004c80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc674001720 tx=0x7fc66c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2a4cfa47a2375f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674001370 con 0x7fc66c004800 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6740052d0 con 0x7fc66c004800 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc687e00640 1 -- 172.21.15.97:0/770817505 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674003830 con 0x7fc66c004800 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc68ee00640 1 -- 172.21.15.97:0/770817505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 msgr2=0x7fc66c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc68ee00640 1 --2- 172.21.15.97:0/770817505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc66c004c80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc674001720 tx=0x7fc66c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc68e400640 1 -- 172.21.15.97:0/770817505 reap_dead start 2024-03-06T07:32:32.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc68ee00640 1 -- 172.21.15.97:0/770817505 shutdown_connections 2024-03-06T07:32:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.730+0000 7fc68ee00640 1 -- 172.21.15.97:0/770817505 >> 172.21.15.97:0/770817505 conn(0x7fc68806afc0 msgr2=0x7fc68806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.731+0000 7fc68ee00640 1 -- 172.21.15.97:0/770817505 shutdown_connections 2024-03-06T07:32:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.731+0000 7fc68ee00640 1 -- 172.21.15.97:0/770817505 wait complete. 2024-03-06T07:32:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.732+0000 7fc68ee00640 1 Processor -- start 2024-03-06T07:32:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.732+0000 7fc68ee00640 1 -- start start 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.733+0000 7fc68ee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc6881655a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.733+0000 7fc68ee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688166530 0x7fc688155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.733+0000 7fc68ee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6880c6790 0x7fc6880c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.733+0000 7fc68ee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880cf9d0 con 0x7fc66c004800 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.733+0000 7fc68ee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880cf850 con 0x7fc6880c6790 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.733+0000 7fc68ee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880cfb50 con 0x7fc688166530 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc6881655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc6881655a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42322/0 (socket says 172.21.15.97:42322) 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 -- 172.21.15.97:0/773348734 learned_addr learned my addr 172.21.15.97:0/773348734 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68d000640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688166530 0x7fc688155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 -- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688166530 msgr2=0x7fc688155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688166530 0x7fc688155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 -- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6880c6790 msgr2=0x7fc6880c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6880c6790 0x7fc6880c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 -- 172.21.15.97:0/773348734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6881669f0 con 0x7fc66c004800 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.734+0000 7fc68da00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc6881655a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc678011e30 tx=0x7fc67800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.735+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc678015630 con 0x7fc66c004800 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.735+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6780157d0 con 0x7fc66c004800 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.735+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc678015ba0 con 0x7fc66c004800 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.735+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc688164ac0 con 0x7fc66c004800 2024-03-06T07:32:32.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.735+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc688165080 con 0x7fc66c004800 2024-03-06T07:32:32.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.736+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68804f330 con 0x7fc66c004800 2024-03-06T07:32:32.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.737+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc67801d400 con 0x7fc66c004800 2024-03-06T07:32:32.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.738+0000 7fc685600640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc66806b500 0x7fc66806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.738+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc67808b040 con 0x7fc66c004800 2024-03-06T07:32:32.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.738+0000 7fc68d000640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc66806b500 0x7fc66806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.739+0000 7fc68d000640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc66806b500 0x7fc66806d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc674033000 tx=0x7fc674009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.742+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc678019030 con 0x7fc66c004800 2024-03-06T07:32:32.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.948+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc6880ce1b0 con 0x7fc66c004800 2024-03-06T07:32:32.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.955+0000 7fc685600640 1 -- 172.21.15.97:0/773348734 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc678010040 con 0x7fc66c004800 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.957+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc66806b500 msgr2=0x7fc66806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.957+0000 7fc68ee00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc66806b500 0x7fc66806d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc674033000 tx=0x7fc674009030 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.957+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 msgr2=0x7fc6881655a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.957+0000 7fc68ee00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc6881655a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc678011e30 tx=0x7fc67800f580 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.957+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 shutdown_connections 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc66806b500 0x7fc66806d9c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6880c6790 0x7fc6880c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc688166530 0x7fc688155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 --2- 172.21.15.97:0/773348734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc66c004800 0x7fc6881655a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 >> 172.21.15.97:0/773348734 conn(0x7fc68806afc0 msgr2=0x7fc6880720f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 shutdown_connections 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:32.958+0000 7fc68ee00640 1 -- 172.21.15.97:0/773348734 wait complete. 2024-03-06T07:32:32.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:32:33.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.175+0000 7f821e400640 1 Processor -- start 2024-03-06T07:32:33.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.175+0000 7f821e400640 1 -- start start 2024-03-06T07:32:33.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f821e400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f82180af1c0 0x7f821815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f821e400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f821806e4d0 0x7f821806e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f821e400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8218071d30 con 0x7f8218070d20 2024-03-06T07:32:33.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f821e400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821806ee80 con 0x7f82180af1c0 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f821e400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82180711d0 con 0x7f821806e4d0 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f8216a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8218070d20 0x7f821806cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43256/0 (socket says 172.21.15.97:43256) 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f8216a00640 1 -- 172.21.15.97:0/2047803210 learned_addr learned my addr 172.21.15.97:0/2047803210 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.177+0000 7f8217400640 1 --2- 172.21.15.97:0/2047803210 >> v2:172.21.15.97:3300/0 conn(0x7f82180af1c0 0x7f821815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8217e00640 1 --2- 172.21.15.97:0/2047803210 >> v2:172.21.15.135:3300/0 conn(0x7f821806e4d0 0x7f821806e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8216000640 1 -- 172.21.15.97:0/2047803210 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470982643 0 0) 0x7f8204001110 con 0x7f8218070d20 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8216000640 1 -- 172.21.15.97:0/2047803210 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81fc003520 con 0x7f8218070d20 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8217400640 1 -- 172.21.15.97:0/2047803210 >> v2:172.21.15.135:3300/0 conn(0x7f821806e4d0 msgr2=0x7f821806e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8217400640 1 --2- 172.21.15.97:0/2047803210 >> v2:172.21.15.135:3300/0 conn(0x7f821806e4d0 0x7f821806e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8217400640 1 -- 172.21.15.97:0/2047803210 >> v1:172.21.15.97:6789/0 conn(0x7f8218070d20 legacy=0x7f821806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8216000640 1 -- 172.21.15.97:0/2047803210 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452999661 0 0) 0x7f81fc003520 con 0x7f8218070d20 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.178+0000 7f8217400640 1 -- 172.21.15.97:0/2047803210 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8218071490 con 0x7f82180af1c0 2024-03-06T07:32:33.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.179+0000 7f8217400640 1 --2- 172.21.15.97:0/2047803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82180af1c0 0x7f821815eff0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f82080097a0 tx=0x7f820802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38048f9b86ced235 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.179+0000 7f8216000640 1 -- 172.21.15.97:0/2047803210 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208037070 con 0x7f82180af1c0 2024-03-06T07:32:33.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.179+0000 7f8216000640 1 -- 172.21.15.97:0/2047803210 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8208030510 con 0x7f82180af1c0 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.179+0000 7f8216000640 1 -- 172.21.15.97:0/2047803210 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204001840 con 0x7f82180af1c0 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 -- 172.21.15.97:0/2047803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82180af1c0 msgr2=0x7f821815eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 --2- 172.21.15.97:0/2047803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82180af1c0 0x7f821815eff0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f82080097a0 tx=0x7f820802b790 comp rx=0 tx=0).stop 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 -- 172.21.15.97:0/2047803210 shutdown_connections 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 --2- 172.21.15.97:0/2047803210 >> v2:172.21.15.135:3300/0 conn(0x7f821806e4d0 0x7f821806e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 --2- 172.21.15.97:0/2047803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82180af1c0 0x7f821815eff0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 -- 172.21.15.97:0/2047803210 >> 172.21.15.97:0/2047803210 conn(0x7f82180b1730 msgr2=0x7f82180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 -- 172.21.15.97:0/2047803210 shutdown_connections 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.180+0000 7f821e400640 1 -- 172.21.15.97:0/2047803210 wait complete. 2024-03-06T07:32:33.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.182+0000 7f821e400640 1 Processor -- start 2024-03-06T07:32:33.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.182+0000 7f821e400640 1 -- start start 2024-03-06T07:32:33.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f821e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 0x7f8218157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f821e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8218070d20 0x7f82181607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f821e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82180af1c0 0x7f8218160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f821e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82180b30b0 con 0x7f821806e4d0 2024-03-06T07:32:33.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f821e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218071a50 con 0x7f8218070d20 2024-03-06T07:32:33.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f821e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821806d370 con 0x7f82180af1c0 2024-03-06T07:32:33.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f8217400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 0x7f8218157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f8217400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 0x7f8218157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42344/0 (socket says 172.21.15.97:42344) 2024-03-06T07:32:33.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.183+0000 7f8217400640 1 -- 172.21.15.97:0/1611560541 learned_addr learned my addr 172.21.15.97:0/1611560541 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f8217400640 1 -- 172.21.15.97:0/1611560541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82180af1c0 msgr2=0x7f8218160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f8217400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82180af1c0 0x7f8218160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f8217400640 1 -- 172.21.15.97:0/1611560541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8218070d20 msgr2=0x7f82181607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f8217400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8218070d20 0x7f82181607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f8217400640 1 -- 172.21.15.97:0/1611560541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180cee10 con 0x7f821806e4d0 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f8217400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 0x7f8218157410 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f82080098c0 tx=0x7f82080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208041020 con 0x7f821806e4d0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820803c930 con 0x7f821806e4d0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208037040 con 0x7f821806e4d0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.184+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82180cf0f0 con 0x7f821806e4d0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.185+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8218156440 con 0x7f821806e4d0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.186+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821804f330 con 0x7f821806e4d0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.187+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8208044020 con 0x7f821806e4d0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.187+0000 7f820f400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f81f806b500 0x7f81f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.188+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f82080ab970 con 0x7f821806e4d0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.188+0000 7f8216a00640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f81f806b500 0x7f81f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.188+0000 7f8216a00640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f81f806b500 0x7f81f806d9c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8204005880 tx=0x7f8204001be0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.192+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f820804b030 con 0x7f821806e4d0 2024-03-06T07:32:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.469+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f821815e830 con 0x7f821806e4d0 2024-03-06T07:32:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.500+0000 7f820f400640 1 -- 172.21.15.97:0/1611560541 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v278) v1 ==== 132+0+0 (secure 0 0 0) 0x7f820807aae0 con 0x7f821806e4d0 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f81f806b500 msgr2=0x7f81f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f81f806b500 0x7f81f806d9c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8204005880 tx=0x7f8204001be0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 msgr2=0x7f8218157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 0x7f8218157410 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f82080098c0 tx=0x7f82080046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 shutdown_connections 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f81f806b500 0x7f81f806d9c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82180af1c0 0x7f8218160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8218070d20 0x7f82181607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 --2- 172.21.15.97:0/1611560541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f821806e4d0 0x7f8218157410 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.504+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 >> 172.21.15.97:0/1611560541 conn(0x7f82180b1730 msgr2=0x7f82180cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.505+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 shutdown_connections 2024-03-06T07:32:33.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.505+0000 7f821e400640 1 -- 172.21.15.97:0/1611560541 wait complete. 2024-03-06T07:32:33.527 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:32:33.527 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:33.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:33.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.733+0000 7f399ba00640 1 Processor -- start 2024-03-06T07:32:33.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.734+0000 7f399ba00640 1 -- start start 2024-03-06T07:32:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.735+0000 7f399ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 0x7f39940ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.735+0000 7f399ba00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399406d190 con 0x7f399415e340 2024-03-06T07:32:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.735+0000 7f399ba00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940cc460 con 0x7f39940d0f30 2024-03-06T07:32:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.735+0000 7f399ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39940cc7d0 con 0x7f39940ce560 2024-03-06T07:32:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.735+0000 7f3993e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f399415e340 0x7f39940ccc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43272/0 (socket says 172.21.15.97:43272) 2024-03-06T07:32:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.735+0000 7f3993e00640 1 -- 172.21.15.97:0/3987895834 learned_addr learned my addr 172.21.15.97:0/3987895834 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871852628 0 0) 0x7f399406d190 con 0x7f399415e340 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3978003520 con 0x7f399415e340 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3999000640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 0x7f39940ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197008157 0 0) 0x7f3978003520 con 0x7f399415e340 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 msgr2=0x7f39940ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 0x7f39940ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 >> v1:172.21.15.135:6789/0 conn(0x7f39940d0f30 legacy=0x7f39940cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399406c0d0 con 0x7f399415e340 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3980001670 con 0x7f399415e340 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 >> v1:172.21.15.97:6789/0 conn(0x7f399415e340 legacy=0x7f39940ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39780039e0 0x7f3978003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 0x7f3978004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3978004d40 0x7f39780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39940cc7d0 con 0x7f3978004d40 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397800bba0 con 0x7f39780039e0 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.736+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397800bda0 con 0x7f39940ce560 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999a00640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3978004d40 0x7f39780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999000640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39780039e0 0x7f3978003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3993e00640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 0x7f3978004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999a00640 1 -- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 msgr2=0x7f3978004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999a00640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940ce560 0x7f3978004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999a00640 1 -- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39780039e0 msgr2=0x7f3978003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999a00640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39780039e0 0x7f3978003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.737+0000 7f3999a00640 1 -- 172.21.15.97:0/3987895834 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978003520 con 0x7f3978004d40 2024-03-06T07:32:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.738+0000 7f3999a00640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3978004d40 0x7f39780051c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f398c001e50 tx=0x7f3978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a1bb25eb864ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.738+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c001a70 con 0x7f3978004d40 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.738+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c003b70 con 0x7f3978004d40 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f3993400640 1 -- 172.21.15.97:0/3987895834 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c0050d0 con 0x7f3978004d40 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f399ba00640 1 -- 172.21.15.97:0/3987895834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3978004d40 msgr2=0x7f39780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f399ba00640 1 --2- 172.21.15.97:0/3987895834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3978004d40 0x7f39780051c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f398c001e50 tx=0x7f3978005830 comp rx=0 tx=0).stop 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f3999a00640 1 -- 172.21.15.97:0/3987895834 reap_dead start 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f399ba00640 1 -- 172.21.15.97:0/3987895834 shutdown_connections 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f399ba00640 1 -- 172.21.15.97:0/3987895834 >> 172.21.15.97:0/3987895834 conn(0x7f39940b3870 msgr2=0x7f39940b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f399ba00640 1 -- 172.21.15.97:0/3987895834 shutdown_connections 2024-03-06T07:32:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.739+0000 7f399ba00640 1 -- 172.21.15.97:0/3987895834 wait complete. 2024-03-06T07:32:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.741+0000 7f399ba00640 1 Processor -- start 2024-03-06T07:32:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.741+0000 7f399ba00640 1 -- start start 2024-03-06T07:32:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f399ba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3978004d40 0x7f399409f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f399ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940b7200 0x7f39941f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f399ba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 0x7f39940b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f399ba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39940cde90 con 0x7f39940b4960 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f399ba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39940cdd10 con 0x7f3978004d40 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f399ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39940ce940 con 0x7f39940b7200 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f3999a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 0x7f39940b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f3999a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 0x7f39940b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42362/0 (socket says 172.21.15.97:42362) 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.742+0000 7f3999a00640 1 -- 172.21.15.97:0/4190441184 learned_addr learned my addr 172.21.15.97:0/4190441184 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3993e00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940b7200 0x7f39941f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999000640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3978004d40 0x7f399409f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999a00640 1 -- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940b7200 msgr2=0x7f39941f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999a00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940b7200 0x7f39941f45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999a00640 1 -- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3978004d40 msgr2=0x7f399409f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999a00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3978004d40 0x7f399409f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999a00640 1 -- 172.21.15.97:0/4190441184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39941f4b30 con 0x7f39940b4960 2024-03-06T07:32:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3999a00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 0x7f39940b4df0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f398c02c060 tx=0x7f398c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c007db0 con 0x7f39940b4960 2024-03-06T07:32:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c003970 con 0x7f39940b4960 2024-03-06T07:32:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.743+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39940b6160 con 0x7f39940b4960 2024-03-06T07:32:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.744+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c006d90 con 0x7f39940b4960 2024-03-06T07:32:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.744+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39940b6720 con 0x7f39940b4960 2024-03-06T07:32:33.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.745+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399404f330 con 0x7f39940b4960 2024-03-06T07:32:33.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.746+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f398c007030 con 0x7f39940b4960 2024-03-06T07:32:33.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.747+0000 7f3990c00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f397406b500 0x7f397406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.747+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f398c0ab250 con 0x7f39940b4960 2024-03-06T07:32:33.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.747+0000 7f3999000640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f397406b500 0x7f397406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.748+0000 7f3999000640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f397406b500 0x7f397406d9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3984006fd0 tx=0x7f3984008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.751+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39941f4b30 con 0x7f39940b4960 2024-03-06T07:32:33.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.949+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39940ce130 con 0x7f39940b4960 2024-03-06T07:32:33.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.951+0000 7f3990c00640 1 -- 172.21.15.97:0/4190441184 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+13009 (secure 0 0 0) 0x7f398c033020 con 0x7f39940b4960 2024-03-06T07:32:33.953 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:33.953 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":87,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:32.495413+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/3414007063":"2024-03-07T07:32:29.475312+0000","172.21.15.97:6835/3414007063":"2024-03-07T07:32:29.475312+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f397406b500 msgr2=0x7f397406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f397406b500 0x7f397406d9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3984006fd0 tx=0x7f3984008040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 msgr2=0x7f39940b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 0x7f39940b4df0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f398c02c060 tx=0x7f398c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 shutdown_connections 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f397406b500 0x7f397406d9c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39940b4960 0x7f39940b4df0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39940b7200 0x7f39941f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 --2- 172.21.15.97:0/4190441184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3978004d40 0x7f399409f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.953+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 >> 172.21.15.97:0/4190441184 conn(0x7f39940b3870 msgr2=0x7f39940d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.954+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 shutdown_connections 2024-03-06T07:32:33.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:33.954+0000 7f399ba00640 1 -- 172.21.15.97:0/4190441184 wait complete. 2024-03-06T07:32:33.976 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:33.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.186+0000 7f0185400640 1 Processor -- start 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.186+0000 7f0185400640 1 -- start start 2024-03-06T07:32:34.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.187+0000 7f0185400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01800b3650 0x7f01800727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.187+0000 7f0185400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f01800cda90 0x7f018006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.187+0000 7f0185400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01800719e0 con 0x7f01800b3df0 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.187+0000 7f0185400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0180072310 con 0x7f01800b3650 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.187+0000 7f0185400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0180071cc0 con 0x7f01800cda90 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01800b3650 0x7f01800727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f01800b3650 0x7f01800727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35144/0 (socket says 172.21.15.97:35144) 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 -- 172.21.15.97:0/4154803210 learned_addr learned my addr 172.21.15.97:0/4154803210 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017e800640 1 --1- 172.21.15.97:0/4154803210 >> v1:172.21.15.97:6789/0 conn(0x7f01800b3df0 0x7f01800b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35394/0 (socket says 172.21.15.97:35394) 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 -- 172.21.15.97:0/4154803210 >> v2:172.21.15.135:3300/0 conn(0x7f01800cda90 msgr2=0x7f018006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017f200640 1 --2- 172.21.15.97:0/4154803210 >> v2:172.21.15.135:3300/0 conn(0x7f01800cda90 0x7f018006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 --2- 172.21.15.97:0/4154803210 >> v2:172.21.15.135:3300/0 conn(0x7f01800cda90 0x7f018006c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 -- 172.21.15.97:0/4154803210 >> v1:172.21.15.97:6789/0 conn(0x7f01800b3df0 legacy=0x7f01800b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017de00640 1 -- 172.21.15.97:0/4154803210 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0180071fa0 con 0x7f01800b3650 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017d400640 1 -- 172.21.15.97:0/4154803210 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639481321 0 0) 0x7f01800719e0 con 0x7f01800b3df0 2024-03-06T07:32:34.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.188+0000 7f017f200640 1 --2- 172.21.15.97:0/4154803210 >> v2:172.21.15.135:3300/0 conn(0x7f01800cda90 0x7f018006c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.189+0000 7f017de00640 1 --2- 172.21.15.97:0/4154803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3650 0x7f01800727a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f018006ed00 tx=0x7f016c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9d61422c4386ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.189+0000 7f017d400640 1 -- 172.21.15.97:0/4154803210 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c034070 con 0x7f01800b3650 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f017d400640 1 -- 172.21.15.97:0/4154803210 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c030860 con 0x7f01800b3650 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f017d400640 1 -- 172.21.15.97:0/4154803210 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c030c30 con 0x7f01800b3650 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 -- 172.21.15.97:0/4154803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3650 msgr2=0x7f01800727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 --2- 172.21.15.97:0/4154803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3650 0x7f01800727a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f018006ed00 tx=0x7f016c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 -- 172.21.15.97:0/4154803210 shutdown_connections 2024-03-06T07:32:34.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 --2- 172.21.15.97:0/4154803210 >> v2:172.21.15.135:3300/0 conn(0x7f01800cda90 0x7f018006c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 --2- 172.21.15.97:0/4154803210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3650 0x7f01800727a0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 -- 172.21.15.97:0/4154803210 >> 172.21.15.97:0/4154803210 conn(0x7f01800b1940 msgr2=0x7f01800b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 -- 172.21.15.97:0/4154803210 shutdown_connections 2024-03-06T07:32:34.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.190+0000 7f0185400640 1 -- 172.21.15.97:0/4154803210 wait complete. 2024-03-06T07:32:34.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.192+0000 7f0185400640 1 Processor -- start 2024-03-06T07:32:34.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.192+0000 7f0185400640 1 -- start start 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f0185400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01800b3650 0x7f01801453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f0185400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 0x7f0180142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f0185400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01800cda90 0x7f018009cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f0185400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018015b3c0 con 0x7f01800b3df0 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f0185400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018015b240 con 0x7f01800b3650 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f0185400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01800cf510 con 0x7f01800cda90 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f017de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 0x7f0180142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f017de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 0x7f0180142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35158/0 (socket says 172.21.15.97:35158) 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.193+0000 7f017de00640 1 -- 172.21.15.97:0/271843546 learned_addr learned my addr 172.21.15.97:0/271843546 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017e800640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01800b3650 0x7f01801453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017f200640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01800cda90 0x7f018009cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017de00640 1 -- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01800cda90 msgr2=0x7f018009cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017de00640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01800cda90 0x7f018009cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017de00640 1 -- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01800b3650 msgr2=0x7f01801453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017de00640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01800b3650 0x7f01801453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017de00640 1 -- 172.21.15.97:0/271843546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0180145900 con 0x7f01800b3df0 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017e800640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01800b3650 0x7f01801453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f017de00640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 0x7f0180142020 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f016c000c00 tx=0x7f016c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.194+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c034040 con 0x7f01800b3df0 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.195+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c03ad90 con 0x7f01800b3df0 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.195+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01801547e0 con 0x7f01800b3df0 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.195+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c004270 con 0x7f01800b3df0 2024-03-06T07:32:34.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.195+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01801506b0 con 0x7f01800b3df0 2024-03-06T07:32:34.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.196+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018004f330 con 0x7f01800b3df0 2024-03-06T07:32:34.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.197+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f016c04b060 con 0x7f01800b3df0 2024-03-06T07:32:34.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.198+0000 7f0176a00640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f016006b4d0 0x7f016006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.198+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f016c0abf90 con 0x7f01800b3df0 2024-03-06T07:32:34.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.198+0000 7f017e800640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f016006b4d0 0x7f016006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.199+0000 7f017e800640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f016006b4d0 0x7f016006d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0170000f60 tx=0x7f0170009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.202+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0180145900 con 0x7f01800b3df0 2024-03-06T07:32:34.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.401+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01800cc0e0 con 0x7f01800b3df0 2024-03-06T07:32:34.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.401+0000 7f0176a00640 1 -- 172.21.15.97:0/271843546 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f016c03f070 con 0x7f01800b3df0 2024-03-06T07:32:34.404 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:32:34.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f016006b4d0 msgr2=0x7f016006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f016006b4d0 0x7f016006d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0170000f60 tx=0x7f0170009040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 msgr2=0x7f0180142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 0x7f0180142020 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f016c000c00 tx=0x7f016c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 shutdown_connections 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f016006b4d0 0x7f016006d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f01800cda90 0x7f018009cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01800b3df0 0x7f0180142020 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 --2- 172.21.15.97:0/271843546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01800b3650 0x7f01801453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 >> 172.21.15.97:0/271843546 conn(0x7f01800b1940 msgr2=0x7f018006e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 shutdown_connections 2024-03-06T07:32:34.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.404+0000 7f0185400640 1 -- 172.21.15.97:0/271843546 wait complete. 2024-03-06T07:32:34.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:32:34.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.626+0000 7f623a400640 1 Processor -- start 2024-03-06T07:32:34.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.626+0000 7f623a400640 1 -- start start 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f623a400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f623406cfe0 0x7f62340b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f623a400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62340717c0 con 0x7f62340cf430 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f623a400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6234071aa0 con 0x7f623406cb50 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f623a400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6234071d80 con 0x7f623406cfe0 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f6233400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f62340cf430 0x7f6234070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35398/0 (socket says 172.21.15.97:35398) 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f6233400640 1 -- 172.21.15.97:0/3793177392 learned_addr learned my addr 172.21.15.97:0/3793177392 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:34.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.628+0000 7f6233e00640 1 --2- 172.21.15.97:0/3793177392 >> v2:172.21.15.97:3300/0 conn(0x7f623406cfe0 0x7f62340b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.629+0000 7f6232000640 1 -- 172.21.15.97:0/3793177392 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973377755 0 0) 0x7f62340717c0 con 0x7f62340cf430 2024-03-06T07:32:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.629+0000 7f6232000640 1 -- 172.21.15.97:0/3793177392 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6218003520 con 0x7f62340cf430 2024-03-06T07:32:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.629+0000 7f6233e00640 1 -- 172.21.15.97:0/3793177392 >> v1:172.21.15.135:6789/0 conn(0x7f623406cb50 legacy=0x7f62340b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.629+0000 7f6233e00640 1 -- 172.21.15.97:0/3793177392 >> v1:172.21.15.97:6789/0 conn(0x7f62340cf430 legacy=0x7f6234070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.629+0000 7f6233e00640 1 -- 172.21.15.97:0/3793177392 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6234072060 con 0x7f623406cfe0 2024-03-06T07:32:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.630+0000 7f6233e00640 1 --2- 172.21.15.97:0/3793177392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f623406cfe0 0x7f62340b3420 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f622c009d90 tx=0x7f622c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2067870c5bc557 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.630+0000 7f6232000640 1 -- 172.21.15.97:0/3793177392 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c039070 con 0x7f623406cfe0 2024-03-06T07:32:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.630+0000 7f6232000640 1 -- 172.21.15.97:0/3793177392 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c02cd00 con 0x7f623406cfe0 2024-03-06T07:32:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.630+0000 7f6232000640 1 -- 172.21.15.97:0/3793177392 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c034640 con 0x7f623406cfe0 2024-03-06T07:32:34.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 -- 172.21.15.97:0/3793177392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f623406cfe0 msgr2=0x7f62340b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 --2- 172.21.15.97:0/3793177392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f623406cfe0 0x7f62340b3420 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f622c009d90 tx=0x7f622c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:34.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 -- 172.21.15.97:0/3793177392 shutdown_connections 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 --2- 172.21.15.97:0/3793177392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f623406cfe0 0x7f62340b3420 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 -- 172.21.15.97:0/3793177392 >> 172.21.15.97:0/3793177392 conn(0x7f62340b1730 msgr2=0x7f62340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 -- 172.21.15.97:0/3793177392 shutdown_connections 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.631+0000 7f623a400640 1 -- 172.21.15.97:0/3793177392 wait complete. 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.633+0000 7f623a400640 1 Processor -- start 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.633+0000 7f623a400640 1 -- start start 2024-03-06T07:32:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f623a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f623406cb50 0x7f6234140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f623a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f623406cfe0 0x7f6234165c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f623a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 0x7f62341661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f623a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6234071430 con 0x7f62340cf430 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f623a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623406d3f0 con 0x7f623406cb50 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f623a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6234162b70 con 0x7f623406cfe0 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 0x7f62341661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 0x7f62341661b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35170/0 (socket says 172.21.15.97:35170) 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 -- 172.21.15.97:0/195412477 learned_addr learned my addr 172.21.15.97:0/195412477 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f623406cb50 0x7f6234140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 -- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f623406cfe0 msgr2=0x7f6234165c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f623406cfe0 0x7f6234165c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 -- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f623406cb50 msgr2=0x7f6234140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f623406cb50 0x7f6234140be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233e00640 1 -- 172.21.15.97:0/195412477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6234141120 con 0x7f62340cf430 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.634+0000 7f6233400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f623406cb50 0x7f6234140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.635+0000 7f6233e00640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 0x7f62341661b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f622c009d90 tx=0x7f622c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.635+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c044020 con 0x7f62340cf430 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.635+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c03a040 con 0x7f62340cf430 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.635+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623415bd30 con 0x7f62340cf430 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.635+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c039050 con 0x7f62340cf430 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.635+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623415c2f0 con 0x7f62340cf430 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.636+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623404f330 con 0x7f62340cf430 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.637+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f622c003bd0 con 0x7f62340cf430 2024-03-06T07:32:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.638+0000 7f622b400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f621406b4d0 0x7f621406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.638+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f622c0ac530 con 0x7f62340cf430 2024-03-06T07:32:34.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.638+0000 7f6233400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f621406b4d0 0x7f621406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.639+0000 7f6233400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f621406b4d0 0x7f621406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f62240011e0 tx=0x7f6224009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.643+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f622c04e060 con 0x7f62340cf430 2024-03-06T07:32:34.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:34.901+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f62341f7d60 con 0x7f62340cf430 2024-03-06T07:32:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.513+0000 7f622b400640 1 -- 172.21.15.97:0/195412477 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v280) v1 ==== 137+0+0 (secure 0 0 0) 0x7f6234141120 con 0x7f62340cf430 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f621406b4d0 msgr2=0x7f621406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f621406b4d0 0x7f621406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f62240011e0 tx=0x7f6224009040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 msgr2=0x7f62341661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 0x7f62341661b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f622c009d90 tx=0x7f622c004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 shutdown_connections 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f621406b4d0 0x7f621406d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62340cf430 0x7f62341661b0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f623406cfe0 0x7f6234165c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 --2- 172.21.15.97:0/195412477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f623406cb50 0x7f6234140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.516+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 >> 172.21.15.97:0/195412477 conn(0x7f62340b1730 msgr2=0x7f62340b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.517+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 shutdown_connections 2024-03-06T07:32:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.517+0000 7f623a400640 1 -- 172.21.15.97:0/195412477 wait complete. 2024-03-06T07:32:35.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:35.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.944+0000 7f3175000640 1 Processor -- start 2024-03-06T07:32:35.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.944+0000 7f3175000640 1 -- start start 2024-03-06T07:32:35.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f3175000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3170158570 0x7f31700ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f3175000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f31700cfde0 0x7f31700cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f3175000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317015b130 con 0x7f3170158950 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f3175000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31700af170 con 0x7f3170158570 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f3175000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317015b3f0 con 0x7f31700cfde0 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f316f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3170158570 0x7f31700ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f316f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3170158570 0x7f31700ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35180/0 (socket says 172.21.15.97:35180) 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.945+0000 7f316f400640 1 -- 172.21.15.97:0/1424721835 learned_addr learned my addr 172.21.15.97:0/1424721835 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.946+0000 7f316f400640 1 -- 172.21.15.97:0/1424721835 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f31700cfde0 msgr2=0x7f31700cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.946+0000 7f316f400640 1 --2- 172.21.15.97:0/1424721835 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f31700cfde0 0x7f31700cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.946+0000 7f316f400640 1 -- 172.21.15.97:0/1424721835 >> v1:172.21.15.135:6789/0 conn(0x7f3170158950 legacy=0x7f31700cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.946+0000 7f316f400640 1 -- 172.21.15.97:0/1424721835 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317015b6d0 con 0x7f3170158570 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.947+0000 7f316f400640 1 --2- 172.21.15.97:0/1424721835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158570 0x7f31700ae6b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3160009870 tx=0x7f316002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a10c55e9dda0d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.947+0000 7f316e000640 1 -- 172.21.15.97:0/1424721835 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31600317c0 con 0x7f3170158570 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.947+0000 7f316e000640 1 -- 172.21.15.97:0/1424721835 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160037070 con 0x7f3170158570 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.947+0000 7f316e000640 1 -- 172.21.15.97:0/1424721835 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160031b80 con 0x7f3170158570 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 -- 172.21.15.97:0/1424721835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158570 msgr2=0x7f31700ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 --2- 172.21.15.97:0/1424721835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158570 0x7f31700ae6b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3160009870 tx=0x7f316002b860 comp rx=0 tx=0).stop 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 -- 172.21.15.97:0/1424721835 shutdown_connections 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 --2- 172.21.15.97:0/1424721835 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f31700cfde0 0x7f31700cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 --2- 172.21.15.97:0/1424721835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158570 0x7f31700ae6b0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 -- 172.21.15.97:0/1424721835 >> 172.21.15.97:0/1424721835 conn(0x7f317006afc0 msgr2=0x7f317006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 -- 172.21.15.97:0/1424721835 shutdown_connections 2024-03-06T07:32:35.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.948+0000 7f3175000640 1 -- 172.21.15.97:0/1424721835 wait complete. 2024-03-06T07:32:35.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.949+0000 7f3175000640 1 Processor -- start 2024-03-06T07:32:35.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 -- start start 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31700cfde0 0x7f3170150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3170158570 0x7f3170150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 0x7f3170146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317015bed0 con 0x7f3170158950 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31700b4950 con 0x7f31700cfde0 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.950+0000 7f3175000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317015c050 con 0x7f3170158570 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 0x7f3170146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 0x7f3170146bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35184/0 (socket says 172.21.15.97:35184) 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 -- 172.21.15.97:0/4046367528 learned_addr learned my addr 172.21.15.97:0/4046367528 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316ea00640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3170158570 0x7f3170150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316f400640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31700cfde0 0x7f3170150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 -- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3170158570 msgr2=0x7f3170150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3170158570 0x7f3170150990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 -- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31700cfde0 msgr2=0x7f3170150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31700cfde0 0x7f3170150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 -- 172.21.15.97:0/4046367528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31700cd220 con 0x7f3170158950 2024-03-06T07:32:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316ea00640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3170158570 0x7f3170150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.951+0000 7f316fe00640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 0x7f3170146bb0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f316800b9a0 tx=0x7f316800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.952+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168019070 con 0x7f3170158950 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.952+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316800cbc0 con 0x7f3170158950 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.952+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31700ccc60 con 0x7f3170158950 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.952+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168014440 con 0x7f3170158950 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.952+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3170158df0 con 0x7f3170158950 2024-03-06T07:32:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.953+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317004f330 con 0x7f3170158950 2024-03-06T07:32:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.954+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f316801c070 con 0x7f3170158950 2024-03-06T07:32:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.955+0000 7f3167400640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.955+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3168089c60 con 0x7f3170158950 2024-03-06T07:32:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.955+0000 7f316f400640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.955+0000 7f316f400640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f315006b4d0 0x7f315006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3160036000 tx=0x7f3160008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:35.959+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3168058dd0 con 0x7f3170158950 2024-03-06T07:32:36.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.154+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f31700cdfb0 con 0x7f3170158950 2024-03-06T07:32:36.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.161+0000 7f3167400640 1 -- 172.21.15.97:0/4046367528 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f31680586d0 con 0x7f3170158950 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f315006b4d0 msgr2=0x7f315006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f315006b4d0 0x7f315006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3160036000 tx=0x7f3160008030 comp rx=0 tx=0).stop 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 msgr2=0x7f3170146bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 0x7f3170146bb0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f316800b9a0 tx=0x7f316800be60 comp rx=0 tx=0).stop 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 shutdown_connections 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170158950 0x7f3170146bb0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3170158570 0x7f3170150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 --2- 172.21.15.97:0/4046367528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31700cfde0 0x7f3170150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 >> 172.21.15.97:0/4046367528 conn(0x7f317006afc0 msgr2=0x7f31700ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 shutdown_connections 2024-03-06T07:32:36.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.163+0000 7f3175000640 1 -- 172.21.15.97:0/4046367528 wait complete. 2024-03-06T07:32:36.184 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:36.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:36.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:36.408 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:36.409 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:36.409 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:36.466 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:36.466 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:32:36.466 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:36.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:36.523 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:36.523 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:36.577 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:36.577 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:32:36.577 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:36.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:36.633 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:36.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:36.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.847+0000 7f9ca4c00640 1 Processor -- start 2024-03-06T07:32:36.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.847+0000 7f9ca4c00640 1 -- start start 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9ca4c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ca015acb0 0x7f9ca00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9ca4c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ca006ec00 0x7f9ca00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9ca4c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca015b090 con 0x7f9ca00b4680 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9ca4c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00b2a30 con 0x7f9ca015acb0 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9ca4c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00b4360 con 0x7f9ca006ec00 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ca015acb0 0x7f9ca00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ca015acb0 0x7f9ca00b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35192/0 (socket says 172.21.15.97:35192) 2024-03-06T07:32:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 -- 172.21.15.97:0/3788090013 learned_addr learned my addr 172.21.15.97:0/3788090013 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 -- 172.21.15.97:0/3788090013 >> v2:172.21.15.135:3300/0 conn(0x7f9ca006ec00 msgr2=0x7f9ca00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 --2- 172.21.15.97:0/3788090013 >> v2:172.21.15.135:3300/0 conn(0x7f9ca006ec00 0x7f9ca00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 -- 172.21.15.97:0/3788090013 >> v1:172.21.15.135:6789/0 conn(0x7f9ca00b4680 legacy=0x7f9ca006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.849+0000 7f9c9de00640 1 -- 172.21.15.97:0/3788090013 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca00b2cf0 con 0x7f9ca015acb0 2024-03-06T07:32:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.850+0000 7f9c9de00640 1 --2- 172.21.15.97:0/3788090013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca015acb0 0x7f9ca00b37d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00b4e60 tx=0x7f9c9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a946ecb239a863e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9c9ca00640 1 -- 172.21.15.97:0/3788090013 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90035020 con 0x7f9ca015acb0 2024-03-06T07:32:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9c9ca00640 1 -- 172.21.15.97:0/3788090013 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c90031870 con 0x7f9ca015acb0 2024-03-06T07:32:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9c9ca00640 1 -- 172.21.15.97:0/3788090013 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90031c40 con 0x7f9ca015acb0 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9ca4c00640 1 -- 172.21.15.97:0/3788090013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca015acb0 msgr2=0x7f9ca00b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/3788090013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca015acb0 0x7f9ca00b37d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00b4e60 tx=0x7f9c9002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9ca4c00640 1 -- 172.21.15.97:0/3788090013 shutdown_connections 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/3788090013 >> v2:172.21.15.135:3300/0 conn(0x7f9ca006ec00 0x7f9ca00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/3788090013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca015acb0 0x7f9ca00b37d0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.851+0000 7f9ca4c00640 1 -- 172.21.15.97:0/3788090013 >> 172.21.15.97:0/3788090013 conn(0x7f9ca00b1730 msgr2=0x7f9ca00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.852+0000 7f9ca4c00640 1 -- 172.21.15.97:0/3788090013 shutdown_connections 2024-03-06T07:32:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.852+0000 7f9ca4c00640 1 -- 172.21.15.97:0/3788090013 wait complete. 2024-03-06T07:32:36.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.853+0000 7f9ca4c00640 1 Processor -- start 2024-03-06T07:32:36.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.853+0000 7f9ca4c00640 1 -- start start 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9ca4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 0x7f9ca014b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9ca4c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca00b4680 0x7f9ca014ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9ca4c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca015acb0 0x7f9ca0149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9ca4c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b3d10 con 0x7f9ca006ec00 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9ca4c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b32b0 con 0x7f9ca00b4680 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9ca4c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b4010 con 0x7f9ca015acb0 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9c9de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 0x7f9ca014b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.854+0000 7f9c9de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 0x7f9ca014b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35194/0 (socket says 172.21.15.97:35194) 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 -- 172.21.15.97:0/1523400732 learned_addr learned my addr 172.21.15.97:0/1523400732 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:36.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 -- 172.21.15.97:0/1523400732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca015acb0 msgr2=0x7f9ca0149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca015acb0 0x7f9ca0149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 -- 172.21.15.97:0/1523400732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca00b4680 msgr2=0x7f9ca014ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca00b4680 0x7f9ca014ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 -- 172.21.15.97:0/1523400732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca014ef50 con 0x7f9ca006ec00 2024-03-06T07:32:36.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c9de00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 0x7f9ca014b280 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9c90000c00 tx=0x7f9c900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.855+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9003ad60 con 0x7f9ca006ec00 2024-03-06T07:32:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.856+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c900443f0 con 0x7f9ca006ec00 2024-03-06T07:32:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.856+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca014b8f0 con 0x7f9ca006ec00 2024-03-06T07:32:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.856+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c900447e0 con 0x7f9ca006ec00 2024-03-06T07:32:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.856+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca014beb0 con 0x7f9ca006ec00 2024-03-06T07:32:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.857+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca004f330 con 0x7f9ca006ec00 2024-03-06T07:32:36.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.858+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c9004b060 con 0x7f9ca006ec00 2024-03-06T07:32:36.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.859+0000 7f9c96000640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.859+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9c900ac0f0 con 0x7f9ca006ec00 2024-03-06T07:32:36.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.859+0000 7f9c9d400640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.860+0000 7f9c9d400640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00b1700 tx=0x7f9c8c007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:36.864+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c9003f070 con 0x7f9ca006ec00 2024-03-06T07:32:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.099+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ca006ce60 con 0x7f9ca006ec00 2024-03-06T07:32:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.100+0000 7f9c96000640 1 -- 172.21.15.97:0/1523400732 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f9c9002f320 con 0x7f9ca006ec00 2024-03-06T07:32:37.102 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:37.102 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5063,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2976745279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2976745279},{"type":"v1","addr":"172.21.15.135:6833","nonce":2976745279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1798000207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1798000207},{"type":"v1","addr":"172.21.15.135:6835","nonce":1798000207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3503570731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3503570731},{"type":"v1","addr":"172.21.15.97:6836","nonce":3503570731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7026,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3896616859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3896616859},{"type":"v1","addr":"172.21.15.97:6837","nonce":3896616859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[]} 2024-03-06T07:32:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 53 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.102+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 msgr2=0x7f9c8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.102+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00b1700 tx=0x7f9c8c007990 comp rx=0 tx=0).stop 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.102+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 msgr2=0x7f9ca014b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.102+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 0x7f9ca014b280 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9c90000c00 tx=0x7f9c900046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 shutdown_connections 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c8006b4d0 0x7f9c8006d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ca015acb0 0x7f9ca0149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca00b4680 0x7f9ca014ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 --2- 172.21.15.97:0/1523400732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ca006ec00 0x7f9ca014b280 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 >> 172.21.15.97:0/1523400732 conn(0x7f9ca00b1730 msgr2=0x7f9ca00cdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 shutdown_connections 2024-03-06T07:32:37.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.103+0000 7f9ca4c00640 1 -- 172.21.15.97:0/1523400732 wait complete. 2024-03-06T07:32:37.125 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:37.125 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:32:37.125 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:37.125 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:37.125 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:37.126 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:32:37.126 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:37.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:37.126 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:37.126 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:32:37.126 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:37.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:37.127 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:37.127 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:32:37.127 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:37.127 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:37.145 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:37.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:37.146 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:37.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:37.147 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:37.148 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:37.153 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:37.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:37.194 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:37.194 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:37.200 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:37.200 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:37.201 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:37.201 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:37.207 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:37.207 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:37.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:32:37.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-06T07:32:37.351 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:32:37.351 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:32:37.352 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:32:37.355 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:32:37.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.394+0000 7fa7aa600640 1 Processor -- start 2024-03-06T07:32:37.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.394+0000 7fa7aa600640 1 -- start start 2024-03-06T07:32:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa7aa600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa7a415e140 0x7fa7a40b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa7aa600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a415e520 con 0x7fa7a40b3960 2024-03-06T07:32:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa7aa600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a40b3e00 con 0x7fa7a406bfd0 2024-03-06T07:32:37.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa7aa600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a40b40e0 con 0x7fa7a415e140 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa7a3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa7a415e140 0x7fa7a40b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa79be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa7a40b3960 0x7fa7a406d2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35406/0 (socket says 172.21.15.97:35406) 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.395+0000 7fa79be00640 1 -- 172.21.15.97:0/3648796236 learned_addr learned my addr 172.21.15.97:0/3648796236 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.396+0000 7fa7a3400640 1 -- 172.21.15.97:0/3648796236 >> v1:172.21.15.135:6789/0 conn(0x7fa7a406bfd0 legacy=0x7fa7a40b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.396+0000 7fa7a3400640 1 -- 172.21.15.97:0/3648796236 >> v1:172.21.15.97:6789/0 conn(0x7fa7a40b3960 legacy=0x7fa7a406d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.396+0000 7fa7a3400640 1 -- 172.21.15.97:0/3648796236 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a40b43c0 con 0x7fa7a415e140 2024-03-06T07:32:37.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.397+0000 7fa7a3400640 1 --2- 172.21.15.97:0/3648796236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40b3420 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa7940097a0 tx=0x7fa79402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ba4de8ffb977af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.397+0000 7fa7a2a00640 1 -- 172.21.15.97:0/3648796236 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa794034070 con 0x7fa7a415e140 2024-03-06T07:32:37.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.397+0000 7fa7a2a00640 1 -- 172.21.15.97:0/3648796236 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa794030610 con 0x7fa7a415e140 2024-03-06T07:32:37.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.397+0000 7fa7a2a00640 1 -- 172.21.15.97:0/3648796236 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7940309e0 con 0x7fa7a415e140 2024-03-06T07:32:37.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 -- 172.21.15.97:0/3648796236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 msgr2=0x7fa7a40b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 --2- 172.21.15.97:0/3648796236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40b3420 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa7940097a0 tx=0x7fa79402b960 comp rx=0 tx=0).stop 2024-03-06T07:32:37.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 -- 172.21.15.97:0/3648796236 shutdown_connections 2024-03-06T07:32:37.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 --2- 172.21.15.97:0/3648796236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40b3420 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 -- 172.21.15.97:0/3648796236 >> 172.21.15.97:0/3648796236 conn(0x7fa7a4071380 msgr2=0x7fa7a4071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 -- 172.21.15.97:0/3648796236 shutdown_connections 2024-03-06T07:32:37.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.398+0000 7fa7aa600640 1 -- 172.21.15.97:0/3648796236 wait complete. 2024-03-06T07:32:37.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.399+0000 7fa7aa600640 1 Processor -- start 2024-03-06T07:32:37.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.400+0000 7fa7aa600640 1 -- start start 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.400+0000 7fa7aa600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a406bfd0 0x7fa7a415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7aa600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40b3960 0x7fa7a4140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7aa600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7aa600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a406d9f0 con 0x7fa7a415e140 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7aa600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a40b4980 con 0x7fa7a40b3960 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7aa600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4161e20 con 0x7fa7a406bfd0 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35230/0 (socket says 172.21.15.97:35230) 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 -- 172.21.15.97:0/510171769 learned_addr learned my addr 172.21.15.97:0/510171769 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa79be00640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40b3960 0x7fa7a4140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3400640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a406bfd0 0x7fa7a415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 -- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a406bfd0 msgr2=0x7fa7a415a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a406bfd0 0x7fa7a415a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 -- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40b3960 msgr2=0x7fa7a4140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40b3960 0x7fa7a4140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.401+0000 7fa7a3e00640 1 -- 172.21.15.97:0/510171769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a40cdff0 con 0x7fa7a415e140 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.402+0000 7fa7a3e00640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40cedb0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00c710 tx=0x7fa79c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.402+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c016020 con 0x7fa7a415e140 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.402+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c006d50 con 0x7fa7a415e140 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.402+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a40ce2d0 con 0x7fa7a415e140 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.402+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c003bb0 con 0x7fa7a415e140 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.402+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a40ce890 con 0x7fa7a415e140 2024-03-06T07:32:37.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.403+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a40cea90 con 0x7fa7a415e140 2024-03-06T07:32:37.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.404+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa79c00a850 con 0x7fa7a415e140 2024-03-06T07:32:37.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.405+0000 7fa79b400640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa78006b4d0 0x7fa78006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.405+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa79c089b00 con 0x7fa7a415e140 2024-03-06T07:32:37.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.405+0000 7fa7a3400640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa78006b4d0 0x7fa78006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.406+0000 7fa7a3400640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa78006b4d0 0x7fa78006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa794036000 tx=0x7fa79403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.409+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7a40cea90 con 0x7fa7a415e140 2024-03-06T07:32:37.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:37.628+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa7a40ccbe0 con 0x7fa7a415e140 2024-03-06T07:32:38.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:38.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.559+0000 7fa79b400640 1 -- 172.21.15.97:0/510171769 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v92) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa79c020020 con 0x7fa7a415e140 2024-03-06T07:32:38.561 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:32:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.563+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa78006b4d0 msgr2=0x7fa78006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.563+0000 7fa7aa600640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa78006b4d0 0x7fa78006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa794036000 tx=0x7fa79403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.563+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 msgr2=0x7fa7a40cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.563+0000 7fa7aa600640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40cedb0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00c710 tx=0x7fa79c00cbd0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 shutdown_connections 2024-03-06T07:32:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa78006b4d0 0x7fa78006d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7a415e140 0x7fa7a40cedb0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a40b3960 0x7fa7a4140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 --2- 172.21.15.97:0/510171769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7a406bfd0 0x7fa7a415a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 >> 172.21.15.97:0/510171769 conn(0x7fa7a4071380 msgr2=0x7fa7a40b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 shutdown_connections 2024-03-06T07:32:38.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.564+0000 7fa7aa600640 1 -- 172.21.15.97:0/510171769 wait complete. 2024-03-06T07:32:38.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:32:38.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.775+0000 7fbf78200640 1 Processor -- start 2024-03-06T07:32:38.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.775+0000 7fbf78200640 1 -- start start 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.776+0000 7fbf78200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf70144c00 0x7fbf7006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.776+0000 7fbf78200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbf7015f9d0 0x7fbf7015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.776+0000 7fbf78200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf700ce0a0 0x7fbf700ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.776+0000 7fbf78200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7006ee70 con 0x7fbf70144c00 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.776+0000 7fbf78200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7006cef0 con 0x7fbf7015f9d0 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.776+0000 7fbf78200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7006d1d0 con 0x7fbf700ce0a0 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf70144c00 0x7fbf7006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf70144c00 0x7fbf7006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35236/0 (socket says 172.21.15.97:35236) 2024-03-06T07:32:38.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 -- 172.21.15.97:0/256586826 learned_addr learned my addr 172.21.15.97:0/256586826 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 -- 172.21.15.97:0/256586826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf700ce0a0 msgr2=0x7fbf700ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 --2- 172.21.15.97:0/256586826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf700ce0a0 0x7fbf700ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 -- 172.21.15.97:0/256586826 >> v2:172.21.15.135:3300/0 conn(0x7fbf7015f9d0 msgr2=0x7fbf7015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 --2- 172.21.15.97:0/256586826 >> v2:172.21.15.135:3300/0 conn(0x7fbf7015f9d0 0x7fbf7015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.777+0000 7fbf75800640 1 -- 172.21.15.97:0/256586826 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7006d4b0 con 0x7fbf70144c00 2024-03-06T07:32:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.779+0000 7fbf75800640 1 --2- 172.21.15.97:0/256586826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf70144c00 0x7fbf7006e930 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbf700b26f0 tx=0x7fbf6402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb0f2408d1f7363 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.779+0000 7fbf6fe00640 1 -- 172.21.15.97:0/256586826 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64035020 con 0x7fbf70144c00 2024-03-06T07:32:38.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.779+0000 7fbf6fe00640 1 -- 172.21.15.97:0/256586826 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6402d040 con 0x7fbf70144c00 2024-03-06T07:32:38.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.779+0000 7fbf6fe00640 1 -- 172.21.15.97:0/256586826 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64031a80 con 0x7fbf70144c00 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 -- 172.21.15.97:0/256586826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf70144c00 msgr2=0x7fbf7006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 --2- 172.21.15.97:0/256586826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf70144c00 0x7fbf7006e930 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbf700b26f0 tx=0x7fbf6402b900 comp rx=0 tx=0).stop 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 -- 172.21.15.97:0/256586826 shutdown_connections 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 --2- 172.21.15.97:0/256586826 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf700ce0a0 0x7fbf700ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 --2- 172.21.15.97:0/256586826 >> v2:172.21.15.135:3300/0 conn(0x7fbf7015f9d0 0x7fbf7015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 --2- 172.21.15.97:0/256586826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf70144c00 0x7fbf7006e930 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 -- 172.21.15.97:0/256586826 >> 172.21.15.97:0/256586826 conn(0x7fbf700b1730 msgr2=0x7fbf700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 -- 172.21.15.97:0/256586826 shutdown_connections 2024-03-06T07:32:38.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.780+0000 7fbf78200640 1 -- 172.21.15.97:0/256586826 wait complete. 2024-03-06T07:32:38.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.782+0000 7fbf78200640 1 Processor -- start 2024-03-06T07:32:38.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.782+0000 7fbf78200640 1 -- start start 2024-03-06T07:32:38.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf78200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 0x7fbf701f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf78200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf70144c00 0x7fbf7013cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf78200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf7015f9d0 0x7fbf7013be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf78200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7006cb50 con 0x7fbf700ce0a0 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf78200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70160730 con 0x7fbf7015f9d0 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf78200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7006ccd0 con 0x7fbf70144c00 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf75800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 0x7fbf701f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf75800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 0x7fbf701f7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35242/0 (socket says 172.21.15.97:35242) 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf75800640 1 -- 172.21.15.97:0/3907390072 learned_addr learned my addr 172.21.15.97:0/3907390072 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf74e00640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf70144c00 0x7fbf7013cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.783+0000 7fbf76200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf7015f9d0 0x7fbf7013be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf75800640 1 -- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf70144c00 msgr2=0x7fbf7013cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf75800640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf70144c00 0x7fbf7013cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf75800640 1 -- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf7015f9d0 msgr2=0x7fbf7013be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf75800640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf7015f9d0 0x7fbf7013be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf75800640 1 -- 172.21.15.97:0/3907390072 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf701f8510 con 0x7fbf700ce0a0 2024-03-06T07:32:38.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf75800640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 0x7fbf701f7950 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fbf6402b940 tx=0x7fbf640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6403ac60 con 0x7fbf700ce0a0 2024-03-06T07:32:38.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6403ae00 con 0x7fbf700ce0a0 2024-03-06T07:32:38.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64044630 con 0x7fbf700ce0a0 2024-03-06T07:32:38.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7013f7c0 con 0x7fbf700ce0a0 2024-03-06T07:32:38.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.784+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7013fd80 con 0x7fbf700ce0a0 2024-03-06T07:32:38.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.786+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7004f330 con 0x7fbf700ce0a0 2024-03-06T07:32:38.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.786+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf64038020 con 0x7fbf700ce0a0 2024-03-06T07:32:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.787+0000 7fbf6d600640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf5006b4d0 0x7fbf5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.787+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbf640abf40 con 0x7fbf700ce0a0 2024-03-06T07:32:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.788+0000 7fbf74e00640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf5006b4d0 0x7fbf5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.788+0000 7fbf74e00640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf5006b4d0 0x7fbf5006d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbf701f7640 tx=0x7fbf5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:38.792+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf6403f070 con 0x7fbf700ce0a0 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.059+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbf7015bd90 con 0x7fbf700ce0a0 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.060+0000 7fbf6d600640 1 -- 172.21.15.97:0/3907390072 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbf6407b100 con 0x7fbf700ce0a0 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.069 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:32:39.076 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:39.076 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:32:39.076 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.076 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:39.076 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:32:39.076 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf5006b4d0 msgr2=0x7fbf5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf5006b4d0 0x7fbf5006d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbf701f7640 tx=0x7fbf5c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 msgr2=0x7fbf701f7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 0x7fbf701f7950 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fbf6402b940 tx=0x7fbf640046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 shutdown_connections 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf5006b4d0 0x7fbf5006d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf7015f9d0 0x7fbf7013be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf70144c00 0x7fbf7013cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 --2- 172.21.15.97:0/3907390072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf700ce0a0 0x7fbf701f7950 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 >> 172.21.15.97:0/3907390072 conn(0x7fbf700b1730 msgr2=0x7fbf700ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 shutdown_connections 2024-03-06T07:32:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.063+0000 7fbf78200640 1 -- 172.21.15.97:0/3907390072 wait complete. 2024-03-06T07:32:39.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:39.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.289+0000 7f967f800640 1 Processor -- start 2024-03-06T07:32:39.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.289+0000 7f967f800640 1 -- start start 2024-03-06T07:32:39.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.291+0000 7f967f800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9678144c00 0x7f967806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.291+0000 7f967f800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967806ee70 con 0x7f967815f9d0 2024-03-06T07:32:39.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.291+0000 7f967f800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967806cef0 con 0x7f967815fe60 2024-03-06T07:32:39.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.291+0000 7f967f800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967806d1d0 con 0x7f9678144c00 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.291+0000 7f966fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f967815f9d0 0x7f96780ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35418/0 (socket says 172.21.15.97:35418) 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.291+0000 7f966fe00640 1 -- 172.21.15.97:0/124833273 learned_addr learned my addr 172.21.15.97:0/124833273 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864578881 0 0) 0x7f9660001110 con 0x7f967815f9d0 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c003520 con 0x7f967815f9d0 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f967ce00640 1 --2- 172.21.15.97:0/124833273 >> v2:172.21.15.135:3300/0 conn(0x7f9678144c00 0x7f967806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4212269723 0 0) 0x7f965c003520 con 0x7f967815f9d0 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 >> v2:172.21.15.135:3300/0 conn(0x7f9678144c00 msgr2=0x7f967806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 --2- 172.21.15.97:0/124833273 >> v2:172.21.15.135:3300/0 conn(0x7f9678144c00 0x7f967806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 >> v1:172.21.15.135:6789/0 conn(0x7f967815fe60 legacy=0x7f96780ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967806d4b0 con 0x7f967815f9d0 2024-03-06T07:32:39.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9660001840 con 0x7f967815f9d0 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 >> v1:172.21.15.97:6789/0 conn(0x7f967815f9d0 legacy=0x7f96780ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f965c0039b0 0x7f965c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f965c004800 0x7f965c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0052e0 0x7f965c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967806d1d0 con 0x7f965c0039b0 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c00c1e0 con 0x7f965c004800 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.292+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c00c3e0 con 0x7f965c0052e0 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967ce00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f965c0039b0 0x7f965c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f966fe00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f965c004800 0x7f965c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967d800640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0052e0 0x7f965c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967ce00640 1 -- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0052e0 msgr2=0x7f965c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967ce00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0052e0 0x7f965c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967ce00640 1 -- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f965c004800 msgr2=0x7f965c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967ce00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f965c004800 0x7f965c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967ce00640 1 -- 172.21.15.97:0/124833273 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c003520 con 0x7f965c0039b0 2024-03-06T07:32:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.293+0000 7f967d800640 1 -- 172.21.15.97:0/124833273 reap_dead start 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.294+0000 7f967ce00640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f965c0039b0 0x7f965c003dd0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f966800a9a0 tx=0x7f965c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4a5f4d88468e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668034070 con 0x7f965c0039b0 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9668030400 con 0x7f965c0039b0 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f9677e00640 1 -- 172.21.15.97:0/124833273 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800a570 con 0x7f965c0039b0 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 -- 172.21.15.97:0/124833273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f965c0039b0 msgr2=0x7f965c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f965c0039b0 0x7f965c003dd0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f966800a9a0 tx=0x7f965c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 -- 172.21.15.97:0/124833273 shutdown_connections 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 --2- 172.21.15.97:0/124833273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f965c0039b0 0x7f965c003dd0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 -- 172.21.15.97:0/124833273 >> 172.21.15.97:0/124833273 conn(0x7f96780b1730 msgr2=0x7f96780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 -- 172.21.15.97:0/124833273 shutdown_connections 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.295+0000 7f967f800640 1 -- 172.21.15.97:0/124833273 wait complete. 2024-03-06T07:32:39.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.297+0000 7f967f800640 1 Processor -- start 2024-03-06T07:32:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.297+0000 7f967f800640 1 -- start start 2024-03-06T07:32:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.298+0000 7f967f800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0039b0 0x7f96781f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.298+0000 7f967f800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 0x7f9678142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.298+0000 7f967f800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f967815d5c0 0x7f967815da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.298+0000 7f967f800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678160730 con 0x7f9678142860 2024-03-06T07:32:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.298+0000 7f967f800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678160270 con 0x7f967815d5c0 2024-03-06T07:32:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.298+0000 7f967f800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967806cb50 con 0x7f965c0039b0 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 0x7f9678142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 0x7f9678142ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35258/0 (socket says 172.21.15.97:35258) 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 -- 172.21.15.97:0/2855064369 learned_addr learned my addr 172.21.15.97:0/2855064369 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f967d800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f967815d5c0 0x7f967815da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f967ce00640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0039b0 0x7f96781f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 -- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0039b0 msgr2=0x7f96781f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0039b0 0x7f96781f48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 -- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f967815d5c0 msgr2=0x7f967815da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f967815d5c0 0x7f967815da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 -- 172.21.15.97:0/2855064369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967815e2f0 con 0x7f9678142860 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f967d800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f967815d5c0 0x7f967815da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.299+0000 7f966fe00640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 0x7f9678142ce0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9660007300 tx=0x7f96600077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.300+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966000e950 con 0x7f9678142860 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.300+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9660018070 con 0x7f9678142860 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.300+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966000ed30 con 0x7f9678142860 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.300+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967815e5d0 con 0x7f9678142860 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.300+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967815eb90 con 0x7f9678142860 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.301+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967804f330 con 0x7f9678142860 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.302+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f966001d460 con 0x7f9678142860 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.303+0000 7f9675600640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.303+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9660089ed0 con 0x7f9678142860 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.304+0000 7f967ce00640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.304+0000 7f967ce00640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f965406b4d0 0x7f965406d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9668036000 tx=0x7f966803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.308+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f967806ee70 con 0x7f9678142860 2024-03-06T07:32:39.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.503+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9678072890 con 0x7f9678142860 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.504+0000 7f9675600640 1 -- 172.21.15.97:0/2855064369 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+12880 (secure 0 0 0) 0x7f9660059090 con 0x7f9678142860 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":92,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:38.556992+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.506+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f965406b4d0 msgr2=0x7f965406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.506+0000 7f967f800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f965406b4d0 0x7f965406d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9668036000 tx=0x7f966803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.506+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 msgr2=0x7f9678142ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.506+0000 7f967f800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 0x7f9678142ce0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9660007300 tx=0x7f96600077c0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 shutdown_connections 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f967815d5c0 0x7f967815da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9678142860 0x7f9678142ce0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 --2- 172.21.15.97:0/2855064369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f965c0039b0 0x7f96781f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 >> 172.21.15.97:0/2855064369 conn(0x7f96780b1730 msgr2=0x7f9678145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 shutdown_connections 2024-03-06T07:32:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.507+0000 7f967f800640 1 -- 172.21.15.97:0/2855064369 wait complete. 2024-03-06T07:32:39.529 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:39.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.734+0000 7f03c5e00640 1 Processor -- start 2024-03-06T07:32:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.734+0000 7f03c5e00640 1 -- start start 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03c5e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f03c00b3530 0x7f03c015b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03c5e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c015bcc0 con 0x7f03c00b3910 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03c5e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c00ae210 con 0x7f03c00ad670 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03c5e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00ce690 con 0x7f03c00b3530 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03bea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f03c00b3530 0x7f03c015b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03bea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f03c00b3530 0x7f03c015b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35260/0 (socket says 172.21.15.97:35260) 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03be000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f03c00b3910 0x7f03c00acf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35428/0 (socket says 172.21.15.97:35428) 2024-03-06T07:32:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.736+0000 7f03bea00640 1 -- 172.21.15.97:0/1252058431 learned_addr learned my addr 172.21.15.97:0/1252058431 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.737+0000 7f03bea00640 1 -- 172.21.15.97:0/1252058431 >> v1:172.21.15.135:6789/0 conn(0x7f03c00ad670 legacy=0x7f03c00adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.737+0000 7f03bea00640 1 -- 172.21.15.97:0/1252058431 >> v1:172.21.15.97:6789/0 conn(0x7f03c00b3910 legacy=0x7f03c00acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.737+0000 7f03bea00640 1 -- 172.21.15.97:0/1252058431 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00ce970 con 0x7f03c00b3530 2024-03-06T07:32:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.737+0000 7f03bd600640 1 -- 172.21.15.97:0/1252058431 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938089310 0 0) 0x7f03a8001490 con 0x7f03c00b3910 2024-03-06T07:32:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.738+0000 7f03bea00640 1 --2- 172.21.15.97:0/1252058431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3530 0x7f03c015b780 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f03b000bb10 tx=0x7f03b002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8049b793ce41da45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.738+0000 7f03bd600640 1 -- 172.21.15.97:0/1252058431 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b003b070 con 0x7f03c00b3530 2024-03-06T07:32:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.738+0000 7f03bd600640 1 -- 172.21.15.97:0/1252058431 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0036440 con 0x7f03c00b3530 2024-03-06T07:32:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.738+0000 7f03bd600640 1 -- 172.21.15.97:0/1252058431 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0036810 con 0x7f03c00b3530 2024-03-06T07:32:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 -- 172.21.15.97:0/1252058431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3530 msgr2=0x7f03c015b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 --2- 172.21.15.97:0/1252058431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3530 0x7f03c015b780 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f03b000bb10 tx=0x7f03b002e170 comp rx=0 tx=0).stop 2024-03-06T07:32:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 -- 172.21.15.97:0/1252058431 shutdown_connections 2024-03-06T07:32:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 --2- 172.21.15.97:0/1252058431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3530 0x7f03c015b780 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 -- 172.21.15.97:0/1252058431 >> 172.21.15.97:0/1252058431 conn(0x7f03c00b1940 msgr2=0x7f03c00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 -- 172.21.15.97:0/1252058431 shutdown_connections 2024-03-06T07:32:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.739+0000 7f03c5e00640 1 -- 172.21.15.97:0/1252058431 wait complete. 2024-03-06T07:32:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.741+0000 7f03c5e00640 1 Processor -- start 2024-03-06T07:32:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.741+0000 7f03c5e00640 1 -- start start 2024-03-06T07:32:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03c5e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c00ad670 0x7f03c01f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03c5e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c00b3530 0x7f03c006e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03c5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 0x7f03c01f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03c5e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00cf210 con 0x7f03c00b3910 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03c5e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00cef30 con 0x7f03c00b3530 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03c5e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00b3d90 con 0x7f03c00ad670 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 0x7f03c01f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 0x7f03c01f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35262/0 (socket says 172.21.15.97:35262) 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bf400640 1 -- 172.21.15.97:0/3366352340 learned_addr learned my addr 172.21.15.97:0/3366352340 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bea00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c00ad670 0x7f03c01f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03be000640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c00b3530 0x7f03c006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bf400640 1 -- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c00ad670 msgr2=0x7f03c01f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bf400640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c00ad670 0x7f03c01f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.742+0000 7f03bf400640 1 -- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c00b3530 msgr2=0x7f03c006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03bf400640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c00b3530 0x7f03c006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03bf400640 1 -- 172.21.15.97:0/3366352340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c006e900 con 0x7f03c00b3910 2024-03-06T07:32:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03bea00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c00ad670 0x7f03c01f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03bf400640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 0x7f03c01f3ed0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c440 tx=0x7f03b800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8011790 con 0x7f03c00b3910 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8017070 con 0x7f03c00b3910 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8011b70 con 0x7f03c00b3910 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c01f02b0 con 0x7f03c00b3910 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.743+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c01efb20 con 0x7f03c00b3910 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.744+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c004f330 con 0x7f03c00b3910 2024-03-06T07:32:39.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.745+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03b8011d10 con 0x7f03c00b3910 2024-03-06T07:32:39.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.746+0000 7f03b7400640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f039c06b500 0x7f039c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.746+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f03b8089bd0 con 0x7f03c00b3910 2024-03-06T07:32:39.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.747+0000 7f03bea00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f039c06b500 0x7f039c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.747+0000 7f03bea00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f039c06b500 0x7f039c06d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f03b0038000 tx=0x7f03b0032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.751+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03b8058e10 con 0x7f03c00b3910 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.948+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03c015ddc0 con 0x7f03c00b3910 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.949+0000 7f03b7400640 1 -- 172.21.15.97:0/3366352340 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03b8058710 con 0x7f03c00b3910 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:32:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f039c06b500 msgr2=0x7f039c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f039c06b500 0x7f039c06d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f03b0038000 tx=0x7f03b0032000 comp rx=0 tx=0).stop 2024-03-06T07:32:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 msgr2=0x7f03c01f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 0x7f03c01f3ed0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c440 tx=0x7f03b800c900 comp rx=0 tx=0).stop 2024-03-06T07:32:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 shutdown_connections 2024-03-06T07:32:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f039c06b500 0x7f039c06d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c00b3910 0x7f03c01f3ed0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c00b3530 0x7f03c006e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 --2- 172.21.15.97:0/3366352340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c00ad670 0x7f03c01f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 >> 172.21.15.97:0/3366352340 conn(0x7f03c00b1940 msgr2=0x7f03c00b06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 shutdown_connections 2024-03-06T07:32:39.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:39.952+0000 7f03c5e00640 1 -- 172.21.15.97:0/3366352340 wait complete. 2024-03-06T07:32:39.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:32:40.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.178+0000 7f6abe400640 1 Processor -- start 2024-03-06T07:32:40.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.179+0000 7f6abe400640 1 -- start start 2024-03-06T07:32:40.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.180+0000 7f6abe400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 0x7f6ab80b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.180+0000 7f6abe400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab80717c0 con 0x7f6ab80cf430 2024-03-06T07:32:40.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.180+0000 7f6abe400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab8071aa0 con 0x7f6ab806cb50 2024-03-06T07:32:40.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.180+0000 7f6abe400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8071d80 con 0x7f6ab806cfe0 2024-03-06T07:32:40.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.180+0000 7f6ab7400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6ab80cf430 0x7f6ab8070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35440/0 (socket says 172.21.15.97:35440) 2024-03-06T07:32:40.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.180+0000 7f6ab7400640 1 -- 172.21.15.97:0/1016105649 learned_addr learned my addr 172.21.15.97:0/1016105649 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 0x7f6ab80b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282595971 0 0) 0x7f6ab80717c0 con 0x7f6ab80cf430 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a9c003520 con 0x7f6ab80cf430 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395630168 0 0) 0x7f6a9c003520 con 0x7f6ab80cf430 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 msgr2=0x7f6ab80b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 0x7f6ab80b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 >> v1:172.21.15.135:6789/0 conn(0x7f6ab806cb50 legacy=0x7f6ab80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8072060 con 0x7f6ab80cf430 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f6aa8001670 con 0x7f6ab80cf430 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 >> v1:172.21.15.97:6789/0 conn(0x7f6ab80cf430 legacy=0x7f6ab8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c0039e0 0x7f6a9c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 0x7f6a9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9c004d40 0x7f6a9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8071d80 con 0x7f6a9c004d40 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c00bb50 con 0x7f6a9c0039e0 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.181+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c00bd20 con 0x7f6ab806cfe0 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9c004d40 0x7f6a9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7400640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c0039e0 0x7f6a9c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 msgr2=0x7f6a9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab6a00640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 0x7f6a9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab806cfe0 0x7f6a9c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c0039e0 msgr2=0x7f6a9c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c0039e0 0x7f6a9c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1016105649 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c003520 con 0x7f6a9c004d40 2024-03-06T07:32:40.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.182+0000 7f6ab7400640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c0039e0 0x7f6a9c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:40.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.183+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9c004d40 0x7f6a9c0051c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0007390 tx=0x7f6a9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615b4bb85c24b74b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.183+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0039070 con 0x7f6a9c004d40 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.183+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab002ccb0 con 0x7f6a9c004d40 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.183+0000 7f6ab6000640 1 -- 172.21.15.97:0/1016105649 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0009be0 con 0x7f6a9c004d40 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6abe400640 1 -- 172.21.15.97:0/1016105649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9c004d40 msgr2=0x7f6a9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6abe400640 1 --2- 172.21.15.97:0/1016105649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9c004d40 0x7f6a9c0051c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0007390 tx=0x7f6a9c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1016105649 reap_dead start 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6abe400640 1 -- 172.21.15.97:0/1016105649 shutdown_connections 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6abe400640 1 -- 172.21.15.97:0/1016105649 >> 172.21.15.97:0/1016105649 conn(0x7f6ab80b1730 msgr2=0x7f6ab80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6abe400640 1 -- 172.21.15.97:0/1016105649 shutdown_connections 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.184+0000 7f6abe400640 1 -- 172.21.15.97:0/1016105649 wait complete. 2024-03-06T07:32:40.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.186+0000 7f6abe400640 1 Processor -- start 2024-03-06T07:32:40.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.186+0000 7f6abe400640 1 -- start start 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6abe400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c004d40 0x7f6ab8144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6abe400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab80b2c10 0x7f6ab80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6abe400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 0x7f6ab815c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6abe400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8071430 con 0x7f6ab815c020 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6abe400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab806d3f0 con 0x7f6a9c004d40 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6abe400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8162b70 con 0x7f6ab80b2c10 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 0x7f6ab815c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 0x7f6ab815c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35274/0 (socket says 172.21.15.97:35274) 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1638973288 learned_addr learned my addr 172.21.15.97:0/1638973288 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab6a00640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab80b2c10 0x7f6ab80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab80b2c10 msgr2=0x7f6ab80b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab80b2c10 0x7f6ab80b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c004d40 msgr2=0x7f6ab8144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c004d40 0x7f6ab8144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.187+0000 7f6ab7e00640 1 -- 172.21.15.97:0/1638973288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8150890 con 0x7f6ab815c020 2024-03-06T07:32:40.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.188+0000 7f6ab7e00640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 0x7f6ab815c4d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0007390 tx=0x7f6ab0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.188+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0039070 con 0x7f6ab815c020 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.188+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab0044070 con 0x7f6ab815c020 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.188+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab815ca10 con 0x7f6ab815c020 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.188+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab002ce60 con 0x7f6ab815c020 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.188+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab814fad0 con 0x7f6ab815c020 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.189+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab804f330 con 0x7f6ab815c020 2024-03-06T07:32:40.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.190+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ab003e830 con 0x7f6ab815c020 2024-03-06T07:32:40.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.191+0000 7f6aaf400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a9806b500 0x7f6a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.191+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ab00abd40 con 0x7f6ab815c020 2024-03-06T07:32:40.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.192+0000 7f6ab7400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a9806b500 0x7f6a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.192+0000 7f6ab7400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a9806b500 0x7f6a9806d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8001eb0 tx=0x7f6aa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.195+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ab007ae70 con 0x7f6ab815c020 2024-03-06T07:32:40.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.392+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6ab80b4a40 con 0x7f6ab815c020 2024-03-06T07:32:40.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.393+0000 7f6aaf400640 1 -- 172.21.15.97:0/1638973288 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v92) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6ab007a770 con 0x7f6ab815c020 2024-03-06T07:32:40.395 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:40.396 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:32:40.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:32:40.396 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:32:40.396 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:32:40.396 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:32:40.396 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:32:40.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a9806b500 msgr2=0x7f6a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a9806b500 0x7f6a9806d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8001eb0 tx=0x7f6aa8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 msgr2=0x7f6ab815c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 0x7f6ab815c4d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0007390 tx=0x7f6ab0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 shutdown_connections 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a9806b500 0x7f6a9806d9c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ab815c020 0x7f6ab815c4d0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6ab80b2c10 0x7f6ab80b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 --2- 172.21.15.97:0/1638973288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a9c004d40 0x7f6ab8144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 >> 172.21.15.97:0/1638973288 conn(0x7f6ab80b1730 msgr2=0x7f6ab80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 shutdown_connections 2024-03-06T07:32:40.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.396+0000 7f6abe400640 1 -- 172.21.15.97:0/1638973288 wait complete. 2024-03-06T07:32:40.417 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:32:40.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:32:40.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.627+0000 7f51e9000640 1 Processor -- start 2024-03-06T07:32:40.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.628+0000 7f51e9000640 1 -- start start 2024-03-06T07:32:40.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.629+0000 7f51e9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f51e4145980 0x7f51e415fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.629+0000 7f51e9000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e4160350 con 0x7f51e415ff40 2024-03-06T07:32:40.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.629+0000 7f51e9000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40cf1e0 con 0x7f51e40ced20 2024-03-06T07:32:40.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.629+0000 7f51e9000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e4070e40 con 0x7f51e4145980 2024-03-06T07:32:40.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f51e4145980 0x7f51e415fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f51e4145980 0x7f51e415fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35284/0 (socket says 172.21.15.97:35284) 2024-03-06T07:32:40.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e1e00640 1 -- 172.21.15.97:0/2348553569 learned_addr learned my addr 172.21.15.97:0/2348553569 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:40.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e1e00640 1 -- 172.21.15.97:0/2348553569 >> v1:172.21.15.135:6789/0 conn(0x7f51e40ced20 legacy=0x7f51e40712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e1e00640 1 -- 172.21.15.97:0/2348553569 >> v1:172.21.15.97:6789/0 conn(0x7f51e415ff40 legacy=0x7f51e40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e0a00640 1 -- 172.21.15.97:0/2348553569 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253873785 0 0) 0x7f51dc001490 con 0x7f51e415ff40 2024-03-06T07:32:40.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.630+0000 7f51e1e00640 1 -- 172.21.15.97:0/2348553569 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e40cf4c0 con 0x7f51e4145980 2024-03-06T07:32:40.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.631+0000 7f51e1e00640 1 --2- 172.21.15.97:0/2348553569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e4145980 0x7f51e415fa00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009870 tx=0x7f51d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97270e5dada6a7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e0a00640 1 -- 172.21.15.97:0/2348553569 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4034070 con 0x7f51e4145980 2024-03-06T07:32:40.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e0a00640 1 -- 172.21.15.97:0/2348553569 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d4030860 con 0x7f51e4145980 2024-03-06T07:32:40.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e0a00640 1 -- 172.21.15.97:0/2348553569 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4030c30 con 0x7f51e4145980 2024-03-06T07:32:40.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 -- 172.21.15.97:0/2348553569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e4145980 msgr2=0x7f51e415fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 --2- 172.21.15.97:0/2348553569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e4145980 0x7f51e415fa00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009870 tx=0x7f51d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 -- 172.21.15.97:0/2348553569 shutdown_connections 2024-03-06T07:32:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 --2- 172.21.15.97:0/2348553569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e4145980 0x7f51e415fa00 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 -- 172.21.15.97:0/2348553569 >> 172.21.15.97:0/2348553569 conn(0x7f51e40cfa80 msgr2=0x7f51e40cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 -- 172.21.15.97:0/2348553569 shutdown_connections 2024-03-06T07:32:40.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.632+0000 7f51e9000640 1 -- 172.21.15.97:0/2348553569 wait complete. 2024-03-06T07:32:40.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.634+0000 7f51e9000640 1 Processor -- start 2024-03-06T07:32:40.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.634+0000 7f51e9000640 1 -- start start 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.635+0000 7f51e9000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e40ced20 0x7f51e41406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.635+0000 7f51e9000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f51e4145980 0x7f51e4141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.635+0000 7f51e9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 0x7f51e4142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.635+0000 7f51e9000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e4160740 con 0x7f51e415ff40 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.635+0000 7f51e9000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e4070ad0 con 0x7f51e40ced20 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.635+0000 7f51e9000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e41608c0 con 0x7f51e4145980 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 0x7f51e4142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 0x7f51e4142030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35300/0 (socket says 172.21.15.97:35300) 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 -- 172.21.15.97:0/1423584668 learned_addr learned my addr 172.21.15.97:0/1423584668 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e1400640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f51e4145980 0x7f51e4141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e1e00640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e40ced20 0x7f51e41406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 -- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f51e4145980 msgr2=0x7f51e4141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f51e4145980 0x7f51e4141af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 -- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e40ced20 msgr2=0x7f51e41406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e40ced20 0x7f51e41406e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 -- 172.21.15.97:0/1423584668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e4142740 con 0x7f51e415ff40 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e1e00640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e40ced20 0x7f51e41406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.636+0000 7f51e2800640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 0x7f51e4142030 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00c780 tx=0x7f51dc00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.637+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc003b70 con 0x7f51e415ff40 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.637+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc003d10 con 0x7f51e415ff40 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.637+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc005270 con 0x7f51e415ff40 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.637+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e4142a20 con 0x7f51e415ff40 2024-03-06T07:32:40.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.637+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e4142fe0 con 0x7f51e415ff40 2024-03-06T07:32:40.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.638+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e404f330 con 0x7f51e415ff40 2024-03-06T07:32:40.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.639+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51dc005410 con 0x7f51e415ff40 2024-03-06T07:32:40.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.640+0000 7f51da000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f51c406b4d0 0x7f51c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.640+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f51dc089e50 con 0x7f51e415ff40 2024-03-06T07:32:40.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.640+0000 7f51e1e00640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f51c406b4d0 0x7f51c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.641+0000 7f51e1e00640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f51c406b4d0 0x7f51c406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f51d4036000 tx=0x7f51d403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.645+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51e4160350 con 0x7f51e415ff40 2024-03-06T07:32:40.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:40.838+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f51e41415a0 con 0x7f51e415ff40 2024-03-06T07:32:41.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.577+0000 7f51da000640 1 -- 172.21.15.97:0/1423584668 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v93) v1 ==== 138+0+0 (secure 0 0 0) 0x7f51dc058f80 con 0x7f51e415ff40 2024-03-06T07:32:41.579 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f51c406b4d0 msgr2=0x7f51c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f51c406b4d0 0x7f51c406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f51d4036000 tx=0x7f51d403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 msgr2=0x7f51e4142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 0x7f51e4142030 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00c780 tx=0x7f51dc00cb80 comp rx=0 tx=0).stop 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 shutdown_connections 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f51c406b4d0 0x7f51c406d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f51e415ff40 0x7f51e4142030 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f51e4145980 0x7f51e4141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.581+0000 7f51e9000640 1 --2- 172.21.15.97:0/1423584668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e40ced20 0x7f51e41406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.582+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 >> 172.21.15.97:0/1423584668 conn(0x7f51e40cfa80 msgr2=0x7f51e40aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.582+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 shutdown_connections 2024-03-06T07:32:41.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.582+0000 7f51e9000640 1 -- 172.21.15.97:0/1423584668 wait complete. 2024-03-06T07:32:41.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:32:41.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.797+0000 7fe18b800640 1 Processor -- start 2024-03-06T07:32:41.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.797+0000 7fe18b800640 1 -- start start 2024-03-06T07:32:41.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe18b800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe1841618e0 0x7fe1840ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe18b800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1840b32f0 con 0x7fe1840b2d80 2024-03-06T07:32:41.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe18b800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1840d0f70 con 0x7fe1840cf6f0 2024-03-06T07:32:41.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe18b800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1840d0d20 con 0x7fe1841618e0 2024-03-06T07:32:41.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe183e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe1840b2d80 0x7fe1840cefe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35456/0 (socket says 172.21.15.97:35456) 2024-03-06T07:32:41.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe183e00640 1 -- 172.21.15.97:0/2940813704 learned_addr learned my addr 172.21.15.97:0/2940813704 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.799+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720214296 0 0) 0x7fe1840b32f0 con 0x7fe1840b2d80 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe188e00640 1 --2- 172.21.15.97:0/2940813704 >> v2:172.21.15.135:3300/0 conn(0x7fe1841618e0 0x7fe1840ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe168003520 con 0x7fe1840b2d80 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580906755 0 0) 0x7fe168003520 con 0x7fe1840b2d80 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 >> v2:172.21.15.135:3300/0 conn(0x7fe1841618e0 msgr2=0x7fe1840ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 --2- 172.21.15.97:0/2940813704 >> v2:172.21.15.135:3300/0 conn(0x7fe1841618e0 0x7fe1840ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 >> v1:172.21.15.135:6789/0 conn(0x7fe1840cf6f0 legacy=0x7fe1840b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1840d1250 con 0x7fe1840b2d80 2024-03-06T07:32:41.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe16c001670 con 0x7fe1840b2d80 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 >> v1:172.21.15.97:6789/0 conn(0x7fe1840b2d80 legacy=0x7fe1840cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1680039b0 0x7fe168003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe168004800 0x7fe168004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe16800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840d0d20 con 0x7fe1680051c0 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16800c110 con 0x7fe168004800 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16800c310 con 0x7fe1680039b0 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.800+0000 7fe189800640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe16800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe183e00640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe168004800 0x7fe168004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe188e00640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1680039b0 0x7fe168003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe189800640 1 -- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1680039b0 msgr2=0x7fe168003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe189800640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1680039b0 0x7fe168003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe189800640 1 -- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe168004800 msgr2=0x7fe168004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe189800640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe168004800 0x7fe168004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe189800640 1 -- 172.21.15.97:0/2940813704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168003520 con 0x7fe1680051c0 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.801+0000 7fe188e00640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1680039b0 0x7fe168003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe189800640 1 -- 172.21.15.97:0/2940813704 reap_dead start 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe189800640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe16800b9c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe17c001720 tx=0x7fe168005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2293ddf3ab0f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c001370 con 0x7fe1680051c0 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c0052d0 con 0x7fe1680051c0 2024-03-06T07:32:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe183400640 1 -- 172.21.15.97:0/2940813704 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c003830 con 0x7fe1680051c0 2024-03-06T07:32:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe18b800640 1 -- 172.21.15.97:0/2940813704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 msgr2=0x7fe16800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.802+0000 7fe18b800640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe16800b9c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe17c001720 tx=0x7fe168005830 comp rx=0 tx=0).stop 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.803+0000 7fe18b800640 1 -- 172.21.15.97:0/2940813704 shutdown_connections 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.803+0000 7fe18b800640 1 --2- 172.21.15.97:0/2940813704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe16800b9c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.803+0000 7fe18b800640 1 -- 172.21.15.97:0/2940813704 >> 172.21.15.97:0/2940813704 conn(0x7fe1840d3e90 msgr2=0x7fe1840d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.803+0000 7fe18b800640 1 -- 172.21.15.97:0/2940813704 shutdown_connections 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.803+0000 7fe18b800640 1 -- 172.21.15.97:0/2940813704 wait complete. 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.804+0000 7fe18b800640 1 Processor -- start 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.804+0000 7fe18b800640 1 -- start start 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.805+0000 7fe18b800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe1840b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.805+0000 7fe18b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1840b7620 0x7fe1840b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.805+0000 7fe18b800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1840c7840 0x7fe1840b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.805+0000 7fe18b800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840ceaa0 con 0x7fe1680051c0 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.805+0000 7fe18b800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe184161cc0 con 0x7fe1840c7840 2024-03-06T07:32:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.805+0000 7fe18b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840cec20 con 0x7fe1840b7620 2024-03-06T07:32:41.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe1840b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe1840b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35324/0 (socket says 172.21.15.97:35324) 2024-03-06T07:32:41.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 -- 172.21.15.97:0/3898280760 learned_addr learned my addr 172.21.15.97:0/3898280760 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 -- 172.21.15.97:0/3898280760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1840b7620 msgr2=0x7fe1840b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1840b7620 0x7fe1840b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 -- 172.21.15.97:0/3898280760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1840c7840 msgr2=0x7fe1840b60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1840c7840 0x7fe1840b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 -- 172.21.15.97:0/3898280760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1840b7ab0 con 0x7fe1680051c0 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe188e00640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe1840b4070 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe174007f70 tx=0x7fe17400f5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174018020 con 0x7fe1680051c0 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1740145f0 con 0x7fe1680051c0 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1840b7d90 con 0x7fe1680051c0 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.806+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1740149c0 con 0x7fe1680051c0 2024-03-06T07:32:41.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.807+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1840b8350 con 0x7fe1680051c0 2024-03-06T07:32:41.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.808+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18404f330 con 0x7fe1680051c0 2024-03-06T07:32:41.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.809+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe17401b020 con 0x7fe1680051c0 2024-03-06T07:32:41.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.810+0000 7fe180c00640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.810+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe17408adf0 con 0x7fe1680051c0 2024-03-06T07:32:41.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.810+0000 7fe183e00640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.811+0000 7fe183e00640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe16006b4d0 0x7fe16006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe16c000f60 tx=0x7fe16c001150 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:41.815+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe174022020 con 0x7fe1680051c0 2024-03-06T07:32:42.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.012+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe1840b1e90 con 0x7fe1680051c0 2024-03-06T07:32:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.585+0000 7fe180c00640 1 -- 172.21.15.97:0/3898280760 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v94) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe174059d20 con 0x7fe1680051c0 2024-03-06T07:32:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:32:42.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.589+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe16006b4d0 msgr2=0x7fe16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.589+0000 7fe18b800640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe16006b4d0 0x7fe16006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe16c000f60 tx=0x7fe16c001150 comp rx=0 tx=0).stop 2024-03-06T07:32:42.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.589+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 msgr2=0x7fe1840b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.589+0000 7fe18b800640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe1840b4070 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe174007f70 tx=0x7fe17400f5e0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.591+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 shutdown_connections 2024-03-06T07:32:42.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.591+0000 7fe18b800640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.591+0000 7fe18b800640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1840c7840 0x7fe1840b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.591+0000 7fe18b800640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1840b7620 0x7fe1840b5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.591+0000 7fe18b800640 1 --2- 172.21.15.97:0/3898280760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1680051c0 0x7fe1840b4070 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.591+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 >> 172.21.15.97:0/3898280760 conn(0x7fe1840d3e90 msgr2=0x7fe1840b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.596+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 shutdown_connections 2024-03-06T07:32:42.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.597+0000 7fe18b800640 1 -- 172.21.15.97:0/3898280760 wait complete. 2024-03-06T07:32:42.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:32:42.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.811+0000 7f44a0800640 1 Processor -- start 2024-03-06T07:32:42.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.812+0000 7f44a0800640 1 -- start start 2024-03-06T07:32:42.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.813+0000 7f44a0800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 0x7f44980b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.813+0000 7f44a0800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44980b3e50 con 0x7f44980b3530 2024-03-06T07:32:42.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.813+0000 7f44a0800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44980ce690 con 0x7f4498162310 2024-03-06T07:32:42.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.813+0000 7f44a0800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44980ce970 con 0x7f44980adf70 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.813+0000 7f449de00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f44980b3530 0x7f449815b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35472/0 (socket says 172.21.15.97:35472) 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449de00640 1 -- 172.21.15.97:0/899261274 learned_addr learned my addr 172.21.15.97:0/899261274 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449d400640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 0x7f44980b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741718041 0 0) 0x7f44980b3e50 con 0x7f44980b3530 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4480004630 con 0x7f44980b3530 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4125423239 0 0) 0x7f4480004630 con 0x7f44980b3530 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 msgr2=0x7f44980b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 0x7f44980b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 >> v1:172.21.15.135:6789/0 conn(0x7f4498162310 legacy=0x7f4498164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.814+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44980cec50 con 0x7f44980b3530 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f448c001670 con 0x7f44980b3530 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 >> v1:172.21.15.97:6789/0 conn(0x7f44980b3530 legacy=0x7f449815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 0x7f4480004af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44800050c0 0x7f4480005910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4480005e50 0x7f44800062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44980ce970 con 0x7f4480005e50 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4480003630 con 0x7f44800050c0 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4480003830 con 0x7f44980adf70 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449e800640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4480005e50 0x7f44800062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449de00640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 0x7f4480004af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449d400640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44800050c0 0x7f4480005910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449e800640 1 -- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 msgr2=0x7f4480004af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449e800640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44980adf70 0x7f4480004af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449e800640 1 -- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44800050c0 msgr2=0x7f4480005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449e800640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44800050c0 0x7f4480005910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.815+0000 7f449e800640 1 -- 172.21.15.97:0/899261274 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4480004630 con 0x7f4480005e50 2024-03-06T07:32:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f449e800640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4480005e50 0x7f44800062d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f4494001e50 tx=0x7f4480006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4338ec118f76088 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494001a70 con 0x7f4480005e50 2024-03-06T07:32:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4494003b70 con 0x7f4480005e50 2024-03-06T07:32:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f449ca00640 1 -- 172.21.15.97:0/899261274 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44940050d0 con 0x7f4480005e50 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f44a0800640 1 -- 172.21.15.97:0/899261274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4480005e50 msgr2=0x7f44800062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f44a0800640 1 --2- 172.21.15.97:0/899261274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4480005e50 0x7f44800062d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f4494001e50 tx=0x7f4480006940 comp rx=0 tx=0).stop 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f449e800640 1 -- 172.21.15.97:0/899261274 reap_dead start 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f44a0800640 1 -- 172.21.15.97:0/899261274 shutdown_connections 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.817+0000 7f44a0800640 1 -- 172.21.15.97:0/899261274 >> 172.21.15.97:0/899261274 conn(0x7f44980b1940 msgr2=0x7f44980b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.818+0000 7f44a0800640 1 -- 172.21.15.97:0/899261274 shutdown_connections 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.818+0000 7f44a0800640 1 -- 172.21.15.97:0/899261274 wait complete. 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.819+0000 7f44a0800640 1 Processor -- start 2024-03-06T07:32:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.819+0000 7f44a0800640 1 -- start start 2024-03-06T07:32:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.820+0000 7f44a0800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4480005e50 0x7f4498156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.820+0000 7f44a0800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4498156570 0x7f4498146d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.820+0000 7f44a0800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 0x7f449814f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.820+0000 7f44a0800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449815aa30 con 0x7f44981472c0 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.820+0000 7f44a0800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44980cf210 con 0x7f4498156570 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.820+0000 7f44a0800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449815abb0 con 0x7f4480005e50 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 0x7f449814f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 0x7f449814f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35350/0 (socket says 172.21.15.97:35350) 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 -- 172.21.15.97:0/2886660885 learned_addr learned my addr 172.21.15.97:0/2886660885 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449de00640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4480005e50 0x7f4498156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449d400640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4498156570 0x7f4498146d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 -- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4480005e50 msgr2=0x7f4498156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4480005e50 0x7f4498156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 -- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4498156570 msgr2=0x7f4498146d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4498156570 0x7f4498146d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 -- 172.21.15.97:0/2886660885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4498159d60 con 0x7f44981472c0 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.821+0000 7f449e800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 0x7f449814f700 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f4494001f70 tx=0x7f44940014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.822+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494007db0 con 0x7f44981472c0 2024-03-06T07:32:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.822+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4494003970 con 0x7f44981472c0 2024-03-06T07:32:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.822+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494006d70 con 0x7f44981472c0 2024-03-06T07:32:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.822+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449815a000 con 0x7f44981472c0 2024-03-06T07:32:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.822+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44980aebf0 con 0x7f44981472c0 2024-03-06T07:32:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.823+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449804f330 con 0x7f44981472c0 2024-03-06T07:32:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.825+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4494041030 con 0x7f44981472c0 2024-03-06T07:32:42.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.825+0000 7f4492000640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f447806b500 0x7f447806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.826+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f44940ab600 con 0x7f44981472c0 2024-03-06T07:32:42.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.826+0000 7f449de00640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f447806b500 0x7f447806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.827+0000 7f449de00640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f447806b500 0x7f447806d9c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f448c001050 tx=0x7f448c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:42.830+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4494033020 con 0x7f44981472c0 2024-03-06T07:32:43.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.066+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f44980ae6e0 con 0x7f44981472c0 2024-03-06T07:32:43.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.626+0000 7f4492000640 1 -- 172.21.15.97:0/2886660885 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v56) v1 ==== 174+0+0 (secure 0 0 0) 0x7f449407a5c0 con 0x7f44981472c0 2024-03-06T07:32:43.631 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:32:43.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.638+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f447806b500 msgr2=0x7f447806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.638+0000 7f44a0800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f447806b500 0x7f447806d9c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f448c001050 tx=0x7f448c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.638+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 msgr2=0x7f449814f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.638+0000 7f44a0800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 0x7f449814f700 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f4494001f70 tx=0x7f44940014a0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.639+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 shutdown_connections 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.639+0000 7f44a0800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f447806b500 0x7f447806d9c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.639+0000 7f44a0800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44981472c0 0x7f449814f700 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.639+0000 7f44a0800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4498156570 0x7f4498146d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.639+0000 7f44a0800640 1 --2- 172.21.15.97:0/2886660885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4480005e50 0x7f4498156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.639+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 >> 172.21.15.97:0/2886660885 conn(0x7f44980b1940 msgr2=0x7f44980cbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.640+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 shutdown_connections 2024-03-06T07:32:43.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.640+0000 7f44a0800640 1 -- 172.21.15.97:0/2886660885 wait complete. 2024-03-06T07:32:43.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:43.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.851+0000 7fcc8d600640 1 Processor -- start 2024-03-06T07:32:43.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.851+0000 7fcc8d600640 1 -- start start 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.852+0000 7fcc8d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880b4680 0x7fcc8806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc8d600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcc88072340 0x7fcc8806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc8d600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8815b090 con 0x7fcc8815acb0 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc8d600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc880b2a30 con 0x7fcc88072340 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc8d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b4360 con 0x7fcc880b4680 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc86a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcc8815acb0 0x7fcc880b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43728/0 (socket says 172.21.15.97:43728) 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc86a00640 1 -- 172.21.15.97:0/3580358886 learned_addr learned my addr 172.21.15.97:0/3580358886 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:43.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc87400640 1 --2- 172.21.15.97:0/3580358886 >> v2:172.21.15.97:3300/0 conn(0x7fcc88072340 0x7fcc8806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc86000640 1 --2- 172.21.15.97:0/3580358886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880b4680 0x7fcc8806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc85600640 1 -- 172.21.15.97:0/3580358886 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344564959 0 0) 0x7fcc8815b090 con 0x7fcc8815acb0 2024-03-06T07:32:43.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc85600640 1 -- 172.21.15.97:0/3580358886 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6c003520 con 0x7fcc8815acb0 2024-03-06T07:32:43.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc87400640 1 -- 172.21.15.97:0/3580358886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880b4680 msgr2=0x7fcc8806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc87400640 1 --2- 172.21.15.97:0/3580358886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880b4680 0x7fcc8806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc87400640 1 -- 172.21.15.97:0/3580358886 >> v1:172.21.15.97:6789/0 conn(0x7fcc8815acb0 legacy=0x7fcc880b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.853+0000 7fcc87400640 1 -- 172.21.15.97:0/3580358886 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc880b2cf0 con 0x7fcc88072340 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc87400640 1 --2- 172.21.15.97:0/3580358886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc8806ea30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc8002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8620d484123f4f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc85600640 1 -- 172.21.15.97:0/3580358886 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80039070 con 0x7fcc88072340 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc85600640 1 -- 172.21.15.97:0/3580358886 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8002cc50 con 0x7fcc88072340 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc85600640 1 -- 172.21.15.97:0/3580358886 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80034440 con 0x7fcc88072340 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc8d600640 1 -- 172.21.15.97:0/3580358886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 msgr2=0x7fcc8806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc8d600640 1 --2- 172.21.15.97:0/3580358886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc8806ea30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc8002bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc8d600640 1 -- 172.21.15.97:0/3580358886 shutdown_connections 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc8d600640 1 --2- 172.21.15.97:0/3580358886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc8806ea30 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.855+0000 7fcc8d600640 1 --2- 172.21.15.97:0/3580358886 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc880b4680 0x7fcc8806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.856+0000 7fcc8d600640 1 -- 172.21.15.97:0/3580358886 >> 172.21.15.97:0/3580358886 conn(0x7fcc880b1730 msgr2=0x7fcc880b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.856+0000 7fcc8d600640 1 -- 172.21.15.97:0/3580358886 shutdown_connections 2024-03-06T07:32:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.856+0000 7fcc8d600640 1 -- 172.21.15.97:0/3580358886 wait complete. 2024-03-06T07:32:43.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.857+0000 7fcc8d600640 1 Processor -- start 2024-03-06T07:32:43.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.857+0000 7fcc8d600640 1 -- start start 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc8d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc881f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc8d600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc880b4680 0x7fcc8814cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc8d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc8815acb0 0x7fcc880b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc8d600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b4010 con 0x7fcc88072340 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc8d600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b32b0 con 0x7fcc880b4680 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc8d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880ccfb0 con 0x7fcc8815acb0 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.858+0000 7fcc86a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc881f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc881f72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33946/0 (socket says 172.21.15.97:33946) 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 -- 172.21.15.97:0/2257876621 learned_addr learned my addr 172.21.15.97:0/2257876621 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 -- 172.21.15.97:0/2257876621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc8815acb0 msgr2=0x7fcc880b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc8815acb0 0x7fcc880b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 -- 172.21.15.97:0/2257876621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc880b4680 msgr2=0x7fcc8814cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc880b4680 0x7fcc8814cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 -- 172.21.15.97:0/2257876621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc881f7970 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc86a00640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc881f72e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c390 tx=0x7fcc7800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.859+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78017070 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.860+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78005ce0 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.860+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc881f7c50 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.860+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc780072f0 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.860+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8815c210 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.861+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8804f330 con 0x7fcc88072340 2024-03-06T07:32:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.862+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc7801a070 con 0x7fcc88072340 2024-03-06T07:32:43.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.862+0000 7fcc7ea00640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc6806b4d0 0x7fcc6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.863+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc78089630 con 0x7fcc88072340 2024-03-06T07:32:43.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.863+0000 7fcc86000640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc6806b4d0 0x7fcc6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.863+0000 7fcc86000640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc6806b4d0 0x7fcc6806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcc880b1670 tx=0x7fcc74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:43.867+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc780584a0 con 0x7fcc88072340 2024-03-06T07:32:44.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-06T07:32:44.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.061+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc8806ce60 con 0x7fcc88072340 2024-03-06T07:32:44.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.064+0000 7fcc7ea00640 1 -- 172.21.15.97:0/2257876621 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fcc78057da0 con 0x7fcc88072340 2024-03-06T07:32:44.067 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:44.067 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":95,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:43.068202+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc6806b4d0 msgr2=0x7fcc6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc6806b4d0 0x7fcc6806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcc880b1670 tx=0x7fcc74008040 comp rx=0 tx=0).stop 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 msgr2=0x7fcc881f72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc881f72e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c390 tx=0x7fcc7800c850 comp rx=0 tx=0).stop 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 shutdown_connections 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcc6806b4d0 0x7fcc6806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc8815acb0 0x7fcc880b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc880b4680 0x7fcc8814cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 --2- 172.21.15.97:0/2257876621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc88072340 0x7fcc881f72e0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 >> 172.21.15.97:0/2257876621 conn(0x7fcc880b1730 msgr2=0x7fcc8815eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 shutdown_connections 2024-03-06T07:32:44.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.067+0000 7fcc8d600640 1 -- 172.21.15.97:0/2257876621 wait complete. 2024-03-06T07:32:44.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:44.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.289+0000 7fbed0e00640 1 Processor -- start 2024-03-06T07:32:44.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.289+0000 7fbed0e00640 1 -- start start 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.290+0000 7fbed0e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbecc1561f0 0x7fbecc0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.290+0000 7fbed0e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbecc0cc9d0 0x7fbecc15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.290+0000 7fbed0e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbecc15c240 con 0x7fbecc0aebf0 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.290+0000 7fbed0e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbecc15ce30 con 0x7fbecc0cc9d0 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.290+0000 7fbed0e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbecc15c500 con 0x7fbecc1561f0 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbecc0cc9d0 0x7fbecc15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbecc0cc9d0 0x7fbecc15f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33950/0 (socket says 172.21.15.97:33950) 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 -- 172.21.15.97:0/1146733603 learned_addr learned my addr 172.21.15.97:0/1146733603 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 -- 172.21.15.97:0/1146733603 >> v2:172.21.15.135:3300/0 conn(0x7fbecc1561f0 msgr2=0x7fbecc0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 --2- 172.21.15.97:0/1146733603 >> v2:172.21.15.135:3300/0 conn(0x7fbecc1561f0 0x7fbecc0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 -- 172.21.15.97:0/1146733603 >> v1:172.21.15.135:6789/0 conn(0x7fbecc0aebf0 legacy=0x7fbecc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.291+0000 7fbecbe00640 1 -- 172.21.15.97:0/1146733603 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc15c7e0 con 0x7fbecc0cc9d0 2024-03-06T07:32:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.292+0000 7fbecbe00640 1 --2- 172.21.15.97:0/1146733603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0cc9d0 0x7fbecc15f830 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0b4db0 tx=0x7fbec402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f69edb511429a785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.292+0000 7fbeca000640 1 -- 172.21.15.97:0/1146733603 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec402ca80 con 0x7fbecc0cc9d0 2024-03-06T07:32:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.292+0000 7fbeca000640 1 -- 172.21.15.97:0/1146733603 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec402cc20 con 0x7fbecc0cc9d0 2024-03-06T07:32:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.292+0000 7fbeca000640 1 -- 172.21.15.97:0/1146733603 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec40343f0 con 0x7fbecc0cc9d0 2024-03-06T07:32:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 -- 172.21.15.97:0/1146733603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0cc9d0 msgr2=0x7fbecc15f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 --2- 172.21.15.97:0/1146733603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0cc9d0 0x7fbecc15f830 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0b4db0 tx=0x7fbec402bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 -- 172.21.15.97:0/1146733603 shutdown_connections 2024-03-06T07:32:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 --2- 172.21.15.97:0/1146733603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0cc9d0 0x7fbecc15f830 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 --2- 172.21.15.97:0/1146733603 >> v2:172.21.15.135:3300/0 conn(0x7fbecc1561f0 0x7fbecc0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 -- 172.21.15.97:0/1146733603 >> 172.21.15.97:0/1146733603 conn(0x7fbecc06afc0 msgr2=0x7fbecc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 -- 172.21.15.97:0/1146733603 shutdown_connections 2024-03-06T07:32:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.293+0000 7fbed0e00640 1 -- 172.21.15.97:0/1146733603 wait complete. 2024-03-06T07:32:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.295+0000 7fbed0e00640 1 Processor -- start 2024-03-06T07:32:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.295+0000 7fbed0e00640 1 -- start start 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbed0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 0x7fbecc06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbed0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbecc0cc9d0 0x7fbecc15ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbed0e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbecc1561f0 0x7fbecc0ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbed0e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc156750 con 0x7fbecc0aebf0 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbed0e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc1565d0 con 0x7fbecc1561f0 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbed0e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc15fd70 con 0x7fbecc0cc9d0 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 0x7fbecc06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 0x7fbecc06dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33962/0 (socket says 172.21.15.97:33962) 2024-03-06T07:32:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 -- 172.21.15.97:0/119133114 learned_addr learned my addr 172.21.15.97:0/119133114 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 -- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbecc0cc9d0 msgr2=0x7fbecc15ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbecc0cc9d0 0x7fbecc15ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 -- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbecc1561f0 msgr2=0x7fbecc0ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecbe00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbecc1561f0 0x7fbecc0ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbecc1561f0 0x7fbecc0ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.296+0000 7fbecb400640 1 -- 172.21.15.97:0/119133114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc158b00 con 0x7fbecc0aebf0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.297+0000 7fbecb400640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 0x7fbecc06dd60 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00b440 tx=0x7fbebc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.297+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc015020 con 0x7fbecc0aebf0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.297+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc011890 con 0x7fbecc0aebf0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.297+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc011c80 con 0x7fbecc0aebf0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.297+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbecc0b1ce0 con 0x7fbecc0aebf0 2024-03-06T07:32:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.297+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc159080 con 0x7fbecc0aebf0 2024-03-06T07:32:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.298+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc04f330 con 0x7fbecc0aebf0 2024-03-06T07:32:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.299+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbebc018020 con 0x7fbecc0aebf0 2024-03-06T07:32:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.300+0000 7fbec3400640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbeac06b4d0 0x7fbeac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.300+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbebc08a2c0 con 0x7fbecc0aebf0 2024-03-06T07:32:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.300+0000 7fbecaa00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbeac06b4d0 0x7fbeac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.301+0000 7fbecaa00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbeac06b4d0 0x7fbeac06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbecc1f5810 tx=0x7fbeb8007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.304+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbebc01f020 con 0x7fbecc0aebf0 2024-03-06T07:32:44.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.498+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbecc0cc780 con 0x7fbecc0aebf0 2024-03-06T07:32:44.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.500+0000 7fbec3400640 1 -- 172.21.15.97:0/119133114 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fbebc00d040 con 0x7fbecc0aebf0 2024-03-06T07:32:44.502 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:44.502 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":95,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:43.068202+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:44.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbeac06b4d0 msgr2=0x7fbeac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbeac06b4d0 0x7fbeac06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbecc1f5810 tx=0x7fbeb8007990 comp rx=0 tx=0).stop 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 msgr2=0x7fbecc06dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 0x7fbecc06dd60 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00b440 tx=0x7fbebc00b900 comp rx=0 tx=0).stop 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 shutdown_connections 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbeac06b4d0 0x7fbeac06d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbecc1561f0 0x7fbecc0ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbecc0cc9d0 0x7fbecc15ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 --2- 172.21.15.97:0/119133114 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbecc0aebf0 0x7fbecc06dd60 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 >> 172.21.15.97:0/119133114 conn(0x7fbecc06afc0 msgr2=0x7fbecc06d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.502+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 shutdown_connections 2024-03-06T07:32:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.503+0000 7fbed0e00640 1 -- 172.21.15.97:0/119133114 wait complete. 2024-03-06T07:32:44.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:32:44.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.719+0000 7fc0c5200640 1 Processor -- start 2024-03-06T07:32:44.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.719+0000 7fc0c5200640 1 -- start start 2024-03-06T07:32:44.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.720+0000 7fc0c5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 0x7fc0c0160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.720+0000 7fc0c5200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc0c0070ff0 0x7fc0c00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.720+0000 7fc0c5200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00b46c0 con 0x7fc0c015fc10 2024-03-06T07:32:44.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.720+0000 7fc0c5200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c0071480 con 0x7fc0c0070ff0 2024-03-06T07:32:44.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.720+0000 7fc0c5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c00b49a0 con 0x7fc0c00b3f70 2024-03-06T07:32:44.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.721+0000 7fc0bd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 0x7fc0c0160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.721+0000 7fc0bd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 0x7fc0c0160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:53586/0 (socket says 172.21.15.97:53586) 2024-03-06T07:32:44.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.721+0000 7fc0bd400640 1 -- 172.21.15.97:0/2756829299 learned_addr learned my addr 172.21.15.97:0/2756829299 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:44.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.721+0000 7fc0be800640 1 --2- 172.21.15.97:0/2756829299 >> v2:172.21.15.135:3300/0 conn(0x7fc0c0070ff0 0x7fc0c00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.722+0000 7fc0bd400640 1 -- 172.21.15.97:0/2756829299 >> v2:172.21.15.135:3300/0 conn(0x7fc0c0070ff0 msgr2=0x7fc0c00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.722+0000 7fc0bd400640 1 --2- 172.21.15.97:0/2756829299 >> v2:172.21.15.135:3300/0 conn(0x7fc0c0070ff0 0x7fc0c00ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.722+0000 7fc0bd400640 1 -- 172.21.15.97:0/2756829299 >> v1:172.21.15.135:6789/0 conn(0x7fc0c015fc10 legacy=0x7fc0c015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.722+0000 7fc0bd400640 1 -- 172.21.15.97:0/2756829299 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0072c90 con 0x7fc0c00b3f70 2024-03-06T07:32:44.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.724+0000 7fc0bd400640 1 --2- 172.21.15.97:0/2756829299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 0x7fc0c0160700 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009870 tx=0x7fc0ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb5c84742089069e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0bca00640 1 -- 172.21.15.97:0/2756829299 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac034070 con 0x7fc0c00b3f70 2024-03-06T07:32:44.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0bca00640 1 -- 172.21.15.97:0/2756829299 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac030840 con 0x7fc0c00b3f70 2024-03-06T07:32:44.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0bca00640 1 -- 172.21.15.97:0/2756829299 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac030c10 con 0x7fc0c00b3f70 2024-03-06T07:32:44.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 -- 172.21.15.97:0/2756829299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 msgr2=0x7fc0c0160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 --2- 172.21.15.97:0/2756829299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 0x7fc0c0160700 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009870 tx=0x7fc0ac02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 -- 172.21.15.97:0/2756829299 shutdown_connections 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 --2- 172.21.15.97:0/2756829299 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c00b3f70 0x7fc0c0160700 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 --2- 172.21.15.97:0/2756829299 >> v2:172.21.15.135:3300/0 conn(0x7fc0c0070ff0 0x7fc0c00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 -- 172.21.15.97:0/2756829299 >> 172.21.15.97:0/2756829299 conn(0x7fc0c00b1940 msgr2=0x7fc0c00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 -- 172.21.15.97:0/2756829299 shutdown_connections 2024-03-06T07:32:44.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.725+0000 7fc0c5200640 1 -- 172.21.15.97:0/2756829299 wait complete. 2024-03-06T07:32:44.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.727+0000 7fc0c5200640 1 Processor -- start 2024-03-06T07:32:44.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.727+0000 7fc0c5200640 1 -- start start 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0c5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 0x7fc0c006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0c5200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0c00b3f70 0x7fc0c0144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0c5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c015fc10 0x7fc0c01f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0c5200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0072a30 con 0x7fc0c0070ff0 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0c5200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c00728b0 con 0x7fc0c00b3f70 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0c5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c015c2e0 con 0x7fc0c015fc10 2024-03-06T07:32:44.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0bde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 0x7fc0c006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0bde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 0x7fc0c006e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33968/0 (socket says 172.21.15.97:33968) 2024-03-06T07:32:44.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.728+0000 7fc0bde00640 1 -- 172.21.15.97:0/898432728 learned_addr learned my addr 172.21.15.97:0/898432728 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bd400640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0c00b3f70 0x7fc0c0144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bde00640 1 -- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c015fc10 msgr2=0x7fc0c01f4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bde00640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c015fc10 0x7fc0c01f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bde00640 1 -- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0c00b3f70 msgr2=0x7fc0c0144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bde00640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0c00b3f70 0x7fc0c0144880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bde00640 1 -- 172.21.15.97:0/898432728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00cb480 con 0x7fc0c0070ff0 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0bde00640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 0x7fc0c006e4e0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c3b0 tx=0x7fc0b000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00052d0 con 0x7fc0c0070ff0 2024-03-06T07:32:44.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b0005490 con 0x7fc0c0070ff0 2024-03-06T07:32:44.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c00cb760 con 0x7fc0c0070ff0 2024-03-06T07:32:44.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.729+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0007260 con 0x7fc0c0070ff0 2024-03-06T07:32:44.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.730+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c009d3b0 con 0x7fc0c0070ff0 2024-03-06T07:32:44.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.731+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c004f330 con 0x7fc0c0070ff0 2024-03-06T07:32:44.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.731+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0b0007400 con 0x7fc0c0070ff0 2024-03-06T07:32:44.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.733+0000 7fc0b6000640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0a006b4d0 0x7fc0a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.733+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0b00895b0 con 0x7fc0c0070ff0 2024-03-06T07:32:44.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.733+0000 7fc0bd400640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0a006b4d0 0x7fc0a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.734+0000 7fc0bd400640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0a006b4d0 0x7fc0a006d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac036000 tx=0x7fc0ac03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:44.737+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0b00584c0 con 0x7fc0c0070ff0 2024-03-06T07:32:45.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.010+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc0c006ef30 con 0x7fc0c0070ff0 2024-03-06T07:32:45.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.649+0000 7fc0b6000640 1 -- 172.21.15.97:0/898432728 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v59) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc0b0057dc0 con 0x7fc0c0070ff0 2024-03-06T07:32:45.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.652+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0a006b4d0 msgr2=0x7fc0a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.652+0000 7fc0c5200640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0a006b4d0 0x7fc0a006d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac036000 tx=0x7fc0ac03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:45.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.652+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 msgr2=0x7fc0c006e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.652+0000 7fc0c5200640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 0x7fc0c006e4e0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c3b0 tx=0x7fc0b000c870 comp rx=0 tx=0).stop 2024-03-06T07:32:45.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 shutdown_connections 2024-03-06T07:32:45.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0a006b4d0 0x7fc0a006d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc0c015fc10 0x7fc0c01f4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0c00b3f70 0x7fc0c0144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 --2- 172.21.15.97:0/898432728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0c0070ff0 0x7fc0c006e4e0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 >> 172.21.15.97:0/898432728 conn(0x7fc0c00b1940 msgr2=0x7fc0c00adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 shutdown_connections 2024-03-06T07:32:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.653+0000 7fc0c5200640 1 -- 172.21.15.97:0/898432728 wait complete. 2024-03-06T07:32:45.679 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:32:45.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:32:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.882+0000 7f107ba00640 1 Processor -- start 2024-03-06T07:32:45.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.882+0000 7f107ba00640 1 -- start start 2024-03-06T07:32:45.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.884+0000 7f107ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1074141850 0x7f107406d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.884+0000 7f107ba00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107415e730 con 0x7f10740cdd70 2024-03-06T07:32:45.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.884+0000 7f107ba00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107415ea10 con 0x7f107415bdc0 2024-03-06T07:32:45.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.884+0000 7f107ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107415ecf0 con 0x7f1074141850 2024-03-06T07:32:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.884+0000 7f1073e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f10740cdd70 0x7f107406d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43740/0 (socket says 172.21.15.97:43740) 2024-03-06T07:32:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.884+0000 7f1073e00640 1 -- 172.21.15.97:0/269299136 learned_addr learned my addr 172.21.15.97:0/269299136 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1079a00640 1 --2- 172.21.15.97:0/269299136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1074141850 0x7f107406d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1073400640 1 -- 172.21.15.97:0/269299136 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541360112 0 0) 0x7f107415e730 con 0x7f10740cdd70 2024-03-06T07:32:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1073400640 1 -- 172.21.15.97:0/269299136 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c003520 con 0x7f10740cdd70 2024-03-06T07:32:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1073400640 1 -- 172.21.15.97:0/269299136 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311200167 0 0) 0x7f107415ea10 con 0x7f107415bdc0 2024-03-06T07:32:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1079a00640 1 -- 172.21.15.97:0/269299136 >> v1:172.21.15.135:6789/0 conn(0x7f107415bdc0 legacy=0x7f10740cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1079a00640 1 -- 172.21.15.97:0/269299136 >> v1:172.21.15.97:6789/0 conn(0x7f10740cdd70 legacy=0x7f107406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.885+0000 7f1079a00640 1 -- 172.21.15.97:0/269299136 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107415efd0 con 0x7f1074141850 2024-03-06T07:32:45.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.887+0000 7f1079a00640 1 --2- 172.21.15.97:0/269299136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1074141850 0x7f107406d880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f106c009dc0 tx=0x7f106c02bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9511ea454211ad0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.887+0000 7f1073400640 1 -- 172.21.15.97:0/269299136 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c039070 con 0x7f1074141850 2024-03-06T07:32:45.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.887+0000 7f1073400640 1 -- 172.21.15.97:0/269299136 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c02c840 con 0x7f1074141850 2024-03-06T07:32:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.887+0000 7f1073400640 1 -- 172.21.15.97:0/269299136 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060001670 con 0x7f1074141850 2024-03-06T07:32:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 -- 172.21.15.97:0/269299136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1074141850 msgr2=0x7f107406d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 --2- 172.21.15.97:0/269299136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1074141850 0x7f107406d880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f106c009dc0 tx=0x7f106c02bd10 comp rx=0 tx=0).stop 2024-03-06T07:32:45.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 -- 172.21.15.97:0/269299136 shutdown_connections 2024-03-06T07:32:45.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 --2- 172.21.15.97:0/269299136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1074141850 0x7f107406d880 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 -- 172.21.15.97:0/269299136 >> 172.21.15.97:0/269299136 conn(0x7f10740b2890 msgr2=0x7f10740ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 -- 172.21.15.97:0/269299136 shutdown_connections 2024-03-06T07:32:45.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.888+0000 7f107ba00640 1 -- 172.21.15.97:0/269299136 wait complete. 2024-03-06T07:32:45.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.890+0000 7f107ba00640 1 Processor -- start 2024-03-06T07:32:45.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.890+0000 7f107ba00640 1 -- start start 2024-03-06T07:32:45.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f107ba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10740cdd70 0x7f107413e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f107ba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 0x7f10740b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f107ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f10740b07a0 0x7f10741f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f107ba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1074143b60 con 0x7f1074141850 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f107ba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10741439e0 con 0x7f10740cdd70 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f107ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1074143ce0 con 0x7f10740b07a0 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f1073e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 0x7f10740b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f1073e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 0x7f10740b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33974/0 (socket says 172.21.15.97:33974) 2024-03-06T07:32:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f1073e00640 1 -- 172.21.15.97:0/3486818038 learned_addr learned my addr 172.21.15.97:0/3486818038 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.891+0000 7f1079000640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10740cdd70 0x7f107413e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1079a00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f10740b07a0 0x7f10741f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1073e00640 1 -- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f10740b07a0 msgr2=0x7f10741f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1073e00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f10740b07a0 0x7f10741f7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1073e00640 1 -- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10740cdd70 msgr2=0x7f107413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1073e00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10740cdd70 0x7f107413e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1073e00640 1 -- 172.21.15.97:0/3486818038 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107413ecf0 con 0x7f1074141850 2024-03-06T07:32:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1073e00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 0x7f10740b0260 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f106000ca70 tx=0x7f106000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060005cc0 con 0x7f1074141850 2024-03-06T07:32:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10741f7bd0 con 0x7f1074141850 2024-03-06T07:32:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1060007070 con 0x7f1074141850 2024-03-06T07:32:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060005e60 con 0x7f1074141850 2024-03-06T07:32:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.892+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1074145eb0 con 0x7f1074141850 2024-03-06T07:32:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.894+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1074145ca0 con 0x7f1074141850 2024-03-06T07:32:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.894+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10600074e0 con 0x7f1074141850 2024-03-06T07:32:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.895+0000 7f1070c00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105406b4d0 0x7f105406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.895+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f106008a130 con 0x7f1074141850 2024-03-06T07:32:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.896+0000 7f1079000640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105406b4d0 0x7f105406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.896+0000 7f1079000640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105406b4d0 0x7f105406d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1068007fd0 tx=0x7f1068009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:45.900+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1060058fc0 con 0x7f1074141850 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.126+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f107406c4b0 con 0x7f1074141850 2024-03-06T07:32:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.659+0000 7f1070c00640 1 -- 172.21.15.97:0/3486818038 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v60) v1 ==== 122+0+0 (secure 0 0 0) 0x7f10600588c0 con 0x7f1074141850 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105406b4d0 msgr2=0x7f105406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105406b4d0 0x7f105406d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1068007fd0 tx=0x7f1068009040 comp rx=0 tx=0).stop 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 msgr2=0x7f10740b0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 0x7f10740b0260 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f106000ca70 tx=0x7f106000cf30 comp rx=0 tx=0).stop 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 shutdown_connections 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105406b4d0 0x7f105406d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f10740b07a0 0x7f10741f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1074141850 0x7f10740b0260 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 --2- 172.21.15.97:0/3486818038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10740cdd70 0x7f107413e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 >> 172.21.15.97:0/3486818038 conn(0x7f10740b2890 msgr2=0x7f1074142900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 shutdown_connections 2024-03-06T07:32:46.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.664+0000 7f107ba00640 1 -- 172.21.15.97:0/3486818038 wait complete. 2024-03-06T07:32:46.686 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:32:46.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:46.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.883+0000 7fcaa1200640 1 Processor -- start 2024-03-06T07:32:46.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.883+0000 7fcaa1200640 1 -- start start 2024-03-06T07:32:46.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fcaa1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fca9c144c00 0x7fca9c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fcaa1200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fca9c15fe60 0x7fca9c0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fcaa1200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c06ee70 con 0x7fca9c15f9d0 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fcaa1200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca9c06cef0 con 0x7fca9c15fe60 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fcaa1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca9c06d1d0 con 0x7fca9c144c00 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fca9c15fe60 0x7fca9c0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fca9c15fe60 0x7fca9c0ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33990/0 (socket says 172.21.15.97:33990) 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 -- 172.21.15.97:0/1082335450 learned_addr learned my addr 172.21.15.97:0/1082335450 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 -- 172.21.15.97:0/1082335450 >> v2:172.21.15.135:3300/0 conn(0x7fca9c144c00 msgr2=0x7fca9c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 --2- 172.21.15.97:0/1082335450 >> v2:172.21.15.135:3300/0 conn(0x7fca9c144c00 0x7fca9c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 -- 172.21.15.97:0/1082335450 >> v1:172.21.15.135:6789/0 conn(0x7fca9c15f9d0 legacy=0x7fca9c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.885+0000 7fca9a800640 1 -- 172.21.15.97:0/1082335450 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c06d4b0 con 0x7fca9c15fe60 2024-03-06T07:32:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.886+0000 7fca9a800640 1 --2- 172.21.15.97:0/1082335450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c15fe60 0x7fca9c0ce7b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fca9c0b25f0 tx=0x7fca9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c809cfa8da69a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fca98a00640 1 -- 172.21.15.97:0/1082335450 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9402ca80 con 0x7fca9c15fe60 2024-03-06T07:32:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fca98a00640 1 -- 172.21.15.97:0/1082335450 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9402cc20 con 0x7fca9c15fe60 2024-03-06T07:32:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fca98a00640 1 -- 172.21.15.97:0/1082335450 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca940343f0 con 0x7fca9c15fe60 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 -- 172.21.15.97:0/1082335450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c15fe60 msgr2=0x7fca9c0ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 --2- 172.21.15.97:0/1082335450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c15fe60 0x7fca9c0ce7b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fca9c0b25f0 tx=0x7fca9402bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 -- 172.21.15.97:0/1082335450 shutdown_connections 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 --2- 172.21.15.97:0/1082335450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c15fe60 0x7fca9c0ce7b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 --2- 172.21.15.97:0/1082335450 >> v2:172.21.15.135:3300/0 conn(0x7fca9c144c00 0x7fca9c06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 -- 172.21.15.97:0/1082335450 >> 172.21.15.97:0/1082335450 conn(0x7fca9c0b1730 msgr2=0x7fca9c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 -- 172.21.15.97:0/1082335450 shutdown_connections 2024-03-06T07:32:46.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.887+0000 7fcaa1200640 1 -- 172.21.15.97:0/1082335450 wait complete. 2024-03-06T07:32:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.889+0000 7fcaa1200640 1 Processor -- start 2024-03-06T07:32:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.889+0000 7fcaa1200640 1 -- start start 2024-03-06T07:32:46.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fcaa1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 0x7fca9c15e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fcaa1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fca9c15f9d0 0x7fca9c13b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fcaa1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fca9c15fe60 0x7fca9c13b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fcaa1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c160730 con 0x7fca9c144c00 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fcaa1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c160270 con 0x7fca9c15f9d0 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fcaa1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c06cb50 con 0x7fca9c15fe60 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fca99e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 0x7fca9c15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fca99e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 0x7fca9c15e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33992/0 (socket says 172.21.15.97:33992) 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.890+0000 7fca99e00640 1 -- 172.21.15.97:0/4003560336 learned_addr learned my addr 172.21.15.97:0/4003560336 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99e00640 1 -- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fca9c15fe60 msgr2=0x7fca9c13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99e00640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fca9c15fe60 0x7fca9c13b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99e00640 1 -- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fca9c15f9d0 msgr2=0x7fca9c13b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99400640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fca9c15f9d0 0x7fca9c13b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99e00640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fca9c15f9d0 0x7fca9c13b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99e00640 1 -- 172.21.15.97:0/4003560336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c13cfd0 con 0x7fca9c144c00 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca99e00640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 0x7fca9c15e930 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00b4d0 tx=0x7fca8c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c014070 con 0x7fca9c144c00 2024-03-06T07:32:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.891+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c010870 con 0x7fca9c144c00 2024-03-06T07:32:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.892+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c010c60 con 0x7fca9c144c00 2024-03-06T07:32:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.892+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9c13d2b0 con 0x7fca9c144c00 2024-03-06T07:32:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.892+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9c15c7c0 con 0x7fca9c144c00 2024-03-06T07:32:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.893+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9c04f330 con 0x7fca9c144c00 2024-03-06T07:32:46.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.894+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca8c017070 con 0x7fca9c144c00 2024-03-06T07:32:46.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.895+0000 7fca92000640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fca7c06b4d0 0x7fca7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.896+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca8c08a020 con 0x7fca9c144c00 2024-03-06T07:32:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.896+0000 7fca99400640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fca7c06b4d0 0x7fca7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.897+0000 7fca99400640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fca7c06b4d0 0x7fca7c06d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fca9c0ae780 tx=0x7fca88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:46.900+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca8c058ea0 con 0x7fca9c144c00 2024-03-06T07:32:47.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.190+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca9c143a30 con 0x7fca9c144c00 2024-03-06T07:32:47.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.191+0000 7fca92000640 1 -- 172.21.15.97:0/4003560336 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fca8c0587a0 con 0x7fca9c144c00 2024-03-06T07:32:47.193 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:47.194 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2473569439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2473569439},{"type":"v1","addr":"172.21.15.135:6833","nonce":2473569439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/807677354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":807677354},{"type":"v1","addr":"172.21.15.135:6835","nonce":807677354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/369774247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":369774247},{"type":"v1","addr":"172.21.15.97:6835","nonce":369774247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:46.128331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/1818345324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1818345324},{"type":"v1","addr":"172.21.15.97:6837","nonce":1818345324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":7}]} 2024-03-06T07:32:47.194 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 60 2024-03-06T07:32:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fca7c06b4d0 msgr2=0x7fca7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fca7c06b4d0 0x7fca7c06d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fca9c0ae780 tx=0x7fca88009040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 msgr2=0x7fca9c15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 0x7fca9c15e930 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00b4d0 tx=0x7fca8c00b990 comp rx=0 tx=0).stop 2024-03-06T07:32:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 shutdown_connections 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fca7c06b4d0 0x7fca7c06d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fca9c15fe60 0x7fca9c13b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fca9c15f9d0 0x7fca9c13b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 --2- 172.21.15.97:0/4003560336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca9c144c00 0x7fca9c15e930 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 >> 172.21.15.97:0/4003560336 conn(0x7fca9c0b1730 msgr2=0x7fca9c142b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 shutdown_connections 2024-03-06T07:32:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.194+0000 7fcaa1200640 1 -- 172.21.15.97:0/4003560336 wait complete. 2024-03-06T07:32:47.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:47.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.407+0000 7f5e35800640 1 Processor -- start 2024-03-06T07:32:47.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.408+0000 7f5e35800640 1 -- start start 2024-03-06T07:32:47.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.409+0000 7f5e35800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3015f9d0 0x7f5e3015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.409+0000 7f5e35800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3006cef0 con 0x7f5e30070b10 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.409+0000 7f5e35800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3006d1d0 con 0x7f5e30144c00 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.409+0000 7f5e35800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3006d4b0 con 0x7f5e3015f9d0 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e26000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3015f9d0 0x7f5e3015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e26000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5e3015f9d0 0x7f5e3015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34006/0 (socket says 172.21.15.97:34006) 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e2f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5e30070b10 0x7f5e30070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43748/0 (socket says 172.21.15.97:43748) 2024-03-06T07:32:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e26000640 1 -- 172.21.15.97:0/2107928695 learned_addr learned my addr 172.21.15.97:0/2107928695 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:47.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e26000640 1 -- 172.21.15.97:0/2107928695 >> v1:172.21.15.135:6789/0 conn(0x7f5e30144c00 legacy=0x7f5e3006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e26000640 1 -- 172.21.15.97:0/2107928695 >> v1:172.21.15.97:6789/0 conn(0x7f5e30070b10 legacy=0x7f5e30070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e26000640 1 -- 172.21.15.97:0/2107928695 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3006d790 con 0x7f5e3015f9d0 2024-03-06T07:32:47.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.410+0000 7f5e2e000640 1 -- 172.21.15.97:0/2107928695 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389278589 0 0) 0x7f5e3006cef0 con 0x7f5e30070b10 2024-03-06T07:32:47.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.411+0000 7f5e26000640 1 --2- 172.21.15.97:0/2107928695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3015f9d0 0x7f5e3015fe60 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009870 tx=0x7f5e1802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5215c87b5fe480ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e2e000640 1 -- 172.21.15.97:0/2107928695 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1802f840 con 0x7f5e3015f9d0 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e2e000640 1 -- 172.21.15.97:0/2107928695 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1802f9e0 con 0x7f5e3015f9d0 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e2e000640 1 -- 172.21.15.97:0/2107928695 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1802fdb0 con 0x7f5e3015f9d0 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 -- 172.21.15.97:0/2107928695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3015f9d0 msgr2=0x7f5e3015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 --2- 172.21.15.97:0/2107928695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3015f9d0 0x7f5e3015fe60 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009870 tx=0x7f5e1802ba20 comp rx=0 tx=0).stop 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 -- 172.21.15.97:0/2107928695 shutdown_connections 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 --2- 172.21.15.97:0/2107928695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e3015f9d0 0x7f5e3015fe60 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 -- 172.21.15.97:0/2107928695 >> 172.21.15.97:0/2107928695 conn(0x7f5e300b1730 msgr2=0x7f5e300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 -- 172.21.15.97:0/2107928695 shutdown_connections 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.412+0000 7f5e35800640 1 -- 172.21.15.97:0/2107928695 wait complete. 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.414+0000 7f5e35800640 1 Processor -- start 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.414+0000 7f5e35800640 1 -- start start 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e35800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e30070b10 0x7f5e301f35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e35800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 0x7f5e30130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e35800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3015f9d0 0x7f5e30131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e35800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3006ccd0 con 0x7f5e30144c00 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e35800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3006cb50 con 0x7f5e30070b10 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e35800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3006df90 con 0x7f5e3015f9d0 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e26000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 0x7f5e30130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.415+0000 7f5e26000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 0x7f5e30130d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34020/0 (socket says 172.21.15.97:34020) 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 -- 172.21.15.97:0/3088022186 learned_addr learned my addr 172.21.15.97:0/3088022186 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:47.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 -- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3015f9d0 msgr2=0x7f5e30131250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3015f9d0 0x7f5e30131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e2ea00640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e30070b10 0x7f5e301f35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 -- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e30070b10 msgr2=0x7f5e301f35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e30070b10 0x7f5e301f35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 -- 172.21.15.97:0/3088022186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30140290 con 0x7f5e30144c00 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.416+0000 7f5e26000640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 0x7f5e30130d10 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5e180090b0 tx=0x7f5e18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.417+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18047070 con 0x7f5e30144c00 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.417+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18037cc0 con 0x7f5e30144c00 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.417+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e30140570 con 0x7f5e30144c00 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.417+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18004270 con 0x7f5e30144c00 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.417+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e30140b30 con 0x7f5e30144c00 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.418+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3004f330 con 0x7f5e30144c00 2024-03-06T07:32:47.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.419+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e1804a070 con 0x7f5e30144c00 2024-03-06T07:32:47.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.420+0000 7f5e27400640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e0c06b4d0 0x7f5e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.421+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e180ac2d0 con 0x7f5e30144c00 2024-03-06T07:32:47.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.421+0000 7f5e2ea00640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e0c06b4d0 0x7f5e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.421+0000 7f5e2ea00640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e0c06b4d0 0x7f5e0c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5e20007fd0 tx=0x7f5e20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.424+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e18033030 con 0x7f5e30144c00 2024-03-06T07:32:47.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.623+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e3015be20 con 0x7f5e30144c00 2024-03-06T07:32:47.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.624+0000 7f5e27400640 1 -- 172.21.15.97:0/3088022186 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f5e1807b140 con 0x7f5e30144c00 2024-03-06T07:32:47.627 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:47.627 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:44.625941+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:47.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e0c06b4d0 msgr2=0x7f5e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e0c06b4d0 0x7f5e0c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5e20007fd0 tx=0x7f5e20009040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 msgr2=0x7f5e30130d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 0x7f5e30130d10 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5e180090b0 tx=0x7f5e18004730 comp rx=0 tx=0).stop 2024-03-06T07:32:47.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 shutdown_connections 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5e0c06b4d0 0x7f5e0c06d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5e3015f9d0 0x7f5e30131250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e30144c00 0x7f5e30130d10 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 --2- 172.21.15.97:0/3088022186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e30070b10 0x7f5e301f35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 >> 172.21.15.97:0/3088022186 conn(0x7f5e300b1730 msgr2=0x7f5e3006e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 shutdown_connections 2024-03-06T07:32:47.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.627+0000 7f5e35800640 1 -- 172.21.15.97:0/3088022186 wait complete. 2024-03-06T07:32:47.646 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:32:47.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:47.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.841+0000 7fb9aae00640 1 Processor -- start 2024-03-06T07:32:47.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.842+0000 7fb9aae00640 1 -- start start 2024-03-06T07:32:47.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.843+0000 7fb9aae00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 0x7fb9a415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.843+0000 7fb9aae00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb9a4070b10 0x7fb9a4070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.843+0000 7fb9aae00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a406cef0 con 0x7fb9a4144c00 2024-03-06T07:32:47.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.843+0000 7fb9aae00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a4160370 con 0x7fb9a4070b10 2024-03-06T07:32:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.843+0000 7fb9aae00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a406d1d0 con 0x7fb9a415f9d0 2024-03-06T07:32:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.844+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 0x7fb9a415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.844+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 0x7fb9a415fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:53656/0 (socket says 172.21.15.97:53656) 2024-03-06T07:32:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.844+0000 7fb9a3400640 1 -- 172.21.15.97:0/1272617729 learned_addr learned my addr 172.21.15.97:0/1272617729 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.844+0000 7fb9a3e00640 1 --1- 172.21.15.97:0/1272617729 >> v1:172.21.15.135:6789/0 conn(0x7fb9a4144c00 0x7fb9a406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:40952/0 (socket says 172.21.15.97:40952) 2024-03-06T07:32:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.844+0000 7fb9a8e00640 1 --2- 172.21.15.97:0/1272617729 >> v2:172.21.15.135:3300/0 conn(0x7fb9a4070b10 0x7fb9a4070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.845+0000 7fb9a3400640 1 -- 172.21.15.97:0/1272617729 >> v2:172.21.15.135:3300/0 conn(0x7fb9a4070b10 msgr2=0x7fb9a4070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.845+0000 7fb9a3400640 1 --2- 172.21.15.97:0/1272617729 >> v2:172.21.15.135:3300/0 conn(0x7fb9a4070b10 0x7fb9a4070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.845+0000 7fb9a3400640 1 -- 172.21.15.97:0/1272617729 >> v1:172.21.15.135:6789/0 conn(0x7fb9a4144c00 legacy=0x7fb9a406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.845+0000 7fb9a3400640 1 -- 172.21.15.97:0/1272617729 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a406d4b0 con 0x7fb9a415f9d0 2024-03-06T07:32:47.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.846+0000 7fb9a3400640 1 --2- 172.21.15.97:0/1272617729 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 0x7fb9a415fe30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb990009870 tx=0x7fb99002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=117a160c0692d70c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.846+0000 7fb9a2a00640 1 -- 172.21.15.97:0/1272617729 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990034070 con 0x7fb9a415f9d0 2024-03-06T07:32:47.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.846+0000 7fb9a2a00640 1 -- 172.21.15.97:0/1272617729 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb990030840 con 0x7fb9a415f9d0 2024-03-06T07:32:47.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9a2a00640 1 -- 172.21.15.97:0/1272617729 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990030c10 con 0x7fb9a415f9d0 2024-03-06T07:32:47.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 -- 172.21.15.97:0/1272617729 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 msgr2=0x7fb9a415fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 --2- 172.21.15.97:0/1272617729 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 0x7fb9a415fe30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb990009870 tx=0x7fb99002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:47.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 -- 172.21.15.97:0/1272617729 shutdown_connections 2024-03-06T07:32:47.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 --2- 172.21.15.97:0/1272617729 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a415f9d0 0x7fb9a415fe30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 --2- 172.21.15.97:0/1272617729 >> v2:172.21.15.135:3300/0 conn(0x7fb9a4070b10 0x7fb9a4070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 -- 172.21.15.97:0/1272617729 >> 172.21.15.97:0/1272617729 conn(0x7fb9a40b1730 msgr2=0x7fb9a40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 -- 172.21.15.97:0/1272617729 shutdown_connections 2024-03-06T07:32:47.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.847+0000 7fb9aae00640 1 -- 172.21.15.97:0/1272617729 wait complete. 2024-03-06T07:32:47.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.849+0000 7fb9aae00640 1 Processor -- start 2024-03-06T07:32:47.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.849+0000 7fb9aae00640 1 -- start start 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9aae00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9a4070b10 0x7fb9a413e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9aae00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a4144c00 0x7fb9a413d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9aae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 0x7fb9a413b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9aae00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a406cb50 con 0x7fb9a415f9d0 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9aae00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4160730 con 0x7fb9a4070b10 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9aae00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a406ccd0 con 0x7fb9a4144c00 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9a8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 0x7fb9a413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.850+0000 7fb9a8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 0x7fb9a413b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34028/0 (socket says 172.21.15.97:34028) 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 -- 172.21.15.97:0/4222325441 learned_addr learned my addr 172.21.15.97:0/4222325441 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a3e00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9a4070b10 0x7fb9a413e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a3400640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a4144c00 0x7fb9a413d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 -- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a4144c00 msgr2=0x7fb9a413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a4144c00 0x7fb9a413d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 -- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9a4070b10 msgr2=0x7fb9a413e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9a4070b10 0x7fb9a413e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 -- 172.21.15.97:0/4222325441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a41f1ef0 con 0x7fb9a415f9d0 2024-03-06T07:32:47.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.851+0000 7fb9a8e00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 0x7fb9a413b480 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00e7f0 tx=0x7fb99c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.852+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c017070 con 0x7fb9a415f9d0 2024-03-06T07:32:47.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.852+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c00ee40 con 0x7fb9a415f9d0 2024-03-06T07:32:47.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.852+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a41f21d0 con 0x7fb9a415f9d0 2024-03-06T07:32:47.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.852+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c00cb80 con 0x7fb9a415f9d0 2024-03-06T07:32:47.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.852+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a41f2790 con 0x7fb9a415f9d0 2024-03-06T07:32:47.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.853+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a40afd20 con 0x7fb9a415f9d0 2024-03-06T07:32:47.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.854+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb99c01a070 con 0x7fb9a415f9d0 2024-03-06T07:32:47.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.855+0000 7fb99be00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb98406b500 0x7fb98406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.855+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb99c08a3a0 con 0x7fb9a415f9d0 2024-03-06T07:32:47.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.855+0000 7fb9a3e00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb98406b500 0x7fb98406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.856+0000 7fb9a3e00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb98406b500 0x7fb98406d9c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb9a413c260 tx=0x7fb994005b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:47.860+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9a40afd20 con 0x7fb9a415f9d0 2024-03-06T07:32:48.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.151+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9a41f1c90 con 0x7fb98406b500 2024-03-06T07:32:48.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.169+0000 7fb99be00640 1 -- 172.21.15.97:0/4222325441 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252013 (secure 0 0 0) 0x7fb994007a60 con 0x7fb98406b500 2024-03-06T07:32:48.175 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb98406b500 msgr2=0x7fb98406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb98406b500 0x7fb98406d9c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb9a413c260 tx=0x7fb994005b70 comp rx=0 tx=0).stop 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 msgr2=0x7fb9a413b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 0x7fb9a413b480 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00e7f0 tx=0x7fb99c00ebf0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 shutdown_connections 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb98406b500 0x7fb98406d9c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a415f9d0 0x7fb9a413b480 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9a4144c00 0x7fb9a413d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 --2- 172.21.15.97:0/4222325441 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9a4070b10 0x7fb9a413e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 >> 172.21.15.97:0/4222325441 conn(0x7fb9a40b1730 msgr2=0x7fb9a40b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.174+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 shutdown_connections 2024-03-06T07:32:48.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.175+0000 7fb9aae00640 1 -- 172.21.15.97:0/4222325441 wait complete. 2024-03-06T07:32:48.202 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-03-06T07:32:46.993823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259744,"kb_used_data":10284,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714976,"statfs":{"total":773094113280,"available":772828135424,"internally_reserved":0,"allocated":10530816,"data_stored":5151446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413191"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662722+0000","last_change":"2024-03-06T07:32:43.653711+0000","last_active":"2024-03-06T07:32:43.662722+0000","last_peered":"2024-03-06T07:32:43.662722+0000","last_clean":"2024-03-06T07:32:43.662722+0000","last_became_active":"2024-03-06T07:32:43.653456+0000","last_became_peered":"2024-03-06T07:32:43.653456+0000","last_unstale":"2024-03-06T07:32:43.662722+0000","last_undegraded":"2024-03-06T07:32:43.662722+0000","last_fullsized":"2024-03-06T07:32:43.662722+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:18.541030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658543+0000","last_change":"2024-03-06T07:32:43.653886+0000","last_active":"2024-03-06T07:32:43.658543+0000","last_peered":"2024-03-06T07:32:43.658543+0000","last_clean":"2024-03-06T07:32:43.658543+0000","last_became_active":"2024-03-06T07:32:43.653566+0000","last_became_peered":"2024-03-06T07:32:43.653566+0000","last_unstale":"2024-03-06T07:32:43.658543+0000","last_undegraded":"2024-03-06T07:32:43.658543+0000","last_fullsized":"2024-03-06T07:32:43.658543+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:54.609852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.663561+0000","last_change":"2024-03-06T07:32:43.657624+0000","last_active":"2024-03-06T07:32:43.663561+0000","last_peered":"2024-03-06T07:32:43.663561+0000","last_clean":"2024-03-06T07:32:43.663561+0000","last_became_active":"2024-03-06T07:32:43.657198+0000","last_became_peered":"2024-03-06T07:32:43.657198+0000","last_unstale":"2024-03-06T07:32:43.663561+0000","last_undegraded":"2024-03-06T07:32:43.663561+0000","last_fullsized":"2024-03-06T07:32:43.663561+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:45.319214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.659651+0000","last_change":"2024-03-06T07:32:43.649794+0000","last_active":"2024-03-06T07:32:43.659651+0000","last_peered":"2024-03-06T07:32:43.659651+0000","last_clean":"2024-03-06T07:32:43.659651+0000","last_became_active":"2024-03-06T07:32:43.649454+0000","last_became_peered":"2024-03-06T07:32:43.649454+0000","last_unstale":"2024-03-06T07:32:43.659651+0000","last_undegraded":"2024-03-06T07:32:43.659651+0000","last_fullsized":"2024-03-06T07:32:43.659651+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:28.102188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657404+0000","last_change":"2024-03-06T07:32:43.647536+0000","last_active":"2024-03-06T07:32:43.657404+0000","last_peered":"2024-03-06T07:32:43.657404+0000","last_clean":"2024-03-06T07:32:43.657404+0000","last_became_active":"2024-03-06T07:32:43.643895+0000","last_became_peered":"2024-03-06T07:32:43.643895+0000","last_unstale":"2024-03-06T07:32:43.657404+0000","last_undegraded":"2024-03-06T07:32:43.657404+0000","last_fullsized":"2024-03-06T07:32:43.657404+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:06.598219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656334+0000","last_change":"2024-03-06T07:32:43.654640+0000","last_active":"2024-03-06T07:32:43.656334+0000","last_peered":"2024-03-06T07:32:43.656334+0000","last_clean":"2024-03-06T07:32:43.656334+0000","last_became_active":"2024-03-06T07:32:43.654321+0000","last_became_peered":"2024-03-06T07:32:43.654321+0000","last_unstale":"2024-03-06T07:32:43.656334+0000","last_undegraded":"2024-03-06T07:32:43.656334+0000","last_fullsized":"2024-03-06T07:32:43.656334+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:06.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654843+0000","last_change":"2024-03-06T07:32:43.653283+0000","last_active":"2024-03-06T07:32:43.654843+0000","last_peered":"2024-03-06T07:32:43.654843+0000","last_clean":"2024-03-06T07:32:43.654843+0000","last_became_active":"2024-03-06T07:32:43.652636+0000","last_became_peered":"2024-03-06T07:32:43.652636+0000","last_unstale":"2024-03-06T07:32:43.654843+0000","last_undegraded":"2024-03-06T07:32:43.654843+0000","last_fullsized":"2024-03-06T07:32:43.654843+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.447329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.644088+0000","last_change":"2024-03-06T07:32:43.638636+0000","last_active":"2024-03-06T07:32:43.644088+0000","last_peered":"2024-03-06T07:32:43.644088+0000","last_clean":"2024-03-06T07:32:43.644088+0000","last_became_active":"2024-03-06T07:32:43.638355+0000","last_became_peered":"2024-03-06T07:32:43.638355+0000","last_unstale":"2024-03-06T07:32:43.644088+0000","last_undegraded":"2024-03-06T07:32:43.644088+0000","last_fullsized":"2024-03-06T07:32:43.644088+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.998526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651259+0000","last_change":"2024-03-06T07:32:43.647311+0000","last_active":"2024-03-06T07:32:43.651259+0000","last_peered":"2024-03-06T07:32:43.651259+0000","last_clean":"2024-03-06T07:32:43.651259+0000","last_became_active":"2024-03-06T07:32:43.647013+0000","last_became_peered":"2024-03-06T07:32:43.647013+0000","last_unstale":"2024-03-06T07:32:43.651259+0000","last_undegraded":"2024-03-06T07:32:43.651259+0000","last_fullsized":"2024-03-06T07:32:43.651259+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:09.989310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657053+0000","last_change":"2024-03-06T07:32:43.645119+0000","last_active":"2024-03-06T07:32:43.657053+0000","last_peered":"2024-03-06T07:32:43.657053+0000","last_clean":"2024-03-06T07:32:43.657053+0000","last_became_active":"2024-03-06T07:32:43.644796+0000","last_became_peered":"2024-03-06T07:32:43.644796+0000","last_unstale":"2024-03-06T07:32:43.657053+0000","last_undegraded":"2024-03-06T07:32:43.657053+0000","last_fullsized":"2024-03-06T07:32:43.657053+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:01.369311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.652391+0000","last_change":"2024-03-06T07:32:43.645762+0000","last_active":"2024-03-06T07:32:43.652391+0000","last_peered":"2024-03-06T07:32:43.652391+0000","last_clean":"2024-03-06T07:32:43.652391+0000","last_became_active":"2024-03-06T07:32:43.644993+0000","last_became_peered":"2024-03-06T07:32:43.644993+0000","last_unstale":"2024-03-06T07:32:43.652391+0000","last_undegraded":"2024-03-06T07:32:43.652391+0000","last_fullsized":"2024-03-06T07:32:43.652391+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.998526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651211+0000","last_change":"2024-03-06T07:32:43.645432+0000","last_active":"2024-03-06T07:32:43.651211+0000","last_peered":"2024-03-06T07:32:43.651211+0000","last_clean":"2024-03-06T07:32:43.651211+0000","last_became_active":"2024-03-06T07:32:43.645140+0000","last_became_peered":"2024-03-06T07:32:43.645140+0000","last_unstale":"2024-03-06T07:32:43.651211+0000","last_undegraded":"2024-03-06T07:32:43.651211+0000","last_fullsized":"2024-03-06T07:32:43.651211+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:51.801166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662432+0000","last_change":"2024-03-06T07:32:43.655963+0000","last_active":"2024-03-06T07:32:43.662432+0000","last_peered":"2024-03-06T07:32:43.662432+0000","last_clean":"2024-03-06T07:32:43.662432+0000","last_became_active":"2024-03-06T07:32:43.654842+0000","last_became_peered":"2024-03-06T07:32:43.654842+0000","last_unstale":"2024-03-06T07:32:43.662432+0000","last_undegraded":"2024-03-06T07:32:43.662432+0000","last_fullsized":"2024-03-06T07:32:43.662432+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:35.984800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.642865+0000","last_change":"2024-03-06T07:32:43.640380+0000","last_active":"2024-03-06T07:32:43.642865+0000","last_peered":"2024-03-06T07:32:43.642865+0000","last_clean":"2024-03-06T07:32:43.642865+0000","last_became_active":"2024-03-06T07:32:43.640033+0000","last_became_peered":"2024-03-06T07:32:43.640033+0000","last_unstale":"2024-03-06T07:32:43.642865+0000","last_undegraded":"2024-03-06T07:32:43.642865+0000","last_fullsized":"2024-03-06T07:32:43.642865+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:39.220605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657760+0000","last_change":"2024-03-06T07:32:43.646577+0000","last_active":"2024-03-06T07:32:43.657760+0000","last_peered":"2024-03-06T07:32:43.657760+0000","last_clean":"2024-03-06T07:32:43.657760+0000","last_became_active":"2024-03-06T07:32:43.646299+0000","last_became_peered":"2024-03-06T07:32:43.646299+0000","last_unstale":"2024-03-06T07:32:43.657760+0000","last_undegraded":"2024-03-06T07:32:43.657760+0000","last_fullsized":"2024-03-06T07:32:43.657760+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.645597+0000","last_change":"2024-03-06T07:32:43.644725+0000","last_active":"2024-03-06T07:32:43.645597+0000","last_peered":"2024-03-06T07:32:43.645597+0000","last_clean":"2024-03-06T07:32:43.645597+0000","last_became_active":"2024-03-06T07:32:43.644331+0000","last_became_peered":"2024-03-06T07:32:43.644331+0000","last_unstale":"2024-03-06T07:32:43.645597+0000","last_undegraded":"2024-03-06T07:32:43.645597+0000","last_fullsized":"2024-03-06T07:32:43.645597+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:40.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654816+0000","last_change":"2024-03-06T07:32:43.647347+0000","last_active":"2024-03-06T07:32:43.654816+0000","last_peered":"2024-03-06T07:32:43.654816+0000","last_clean":"2024-03-06T07:32:43.654816+0000","last_became_active":"2024-03-06T07:32:43.646883+0000","last_became_peered":"2024-03-06T07:32:43.646883+0000","last_unstale":"2024-03-06T07:32:43.654816+0000","last_undegraded":"2024-03-06T07:32:43.654816+0000","last_fullsized":"2024-03-06T07:32:43.654816+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657793+0000","last_change":"2024-03-06T07:32:43.648489+0000","last_active":"2024-03-06T07:32:43.657793+0000","last_peered":"2024-03-06T07:32:43.657793+0000","last_clean":"2024-03-06T07:32:43.657793+0000","last_became_active":"2024-03-06T07:32:43.648205+0000","last_became_peered":"2024-03-06T07:32:43.648205+0000","last_unstale":"2024-03-06T07:32:43.657793+0000","last_undegraded":"2024-03-06T07:32:43.657793+0000","last_fullsized":"2024-03-06T07:32:43.657793+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.447329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654871+0000","last_change":"2024-03-06T07:32:43.647471+0000","last_active":"2024-03-06T07:32:43.654871+0000","last_peered":"2024-03-06T07:32:43.654871+0000","last_clean":"2024-03-06T07:32:43.654871+0000","last_became_active":"2024-03-06T07:32:43.645446+0000","last_became_peered":"2024-03-06T07:32:43.645446+0000","last_unstale":"2024-03-06T07:32:43.654871+0000","last_undegraded":"2024-03-06T07:32:43.654871+0000","last_fullsized":"2024-03-06T07:32:43.654871+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:28.161262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649802+0000","last_change":"2024-03-06T07:32:43.642238+0000","last_active":"2024-03-06T07:32:43.649802+0000","last_peered":"2024-03-06T07:32:43.649802+0000","last_clean":"2024-03-06T07:32:43.649802+0000","last_became_active":"2024-03-06T07:32:43.641966+0000","last_became_peered":"2024-03-06T07:32:43.641966+0000","last_unstale":"2024-03-06T07:32:43.649802+0000","last_undegraded":"2024-03-06T07:32:43.649802+0000","last_fullsized":"2024-03-06T07:32:43.649802+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:09.710140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662970+0000","last_change":"2024-03-06T07:32:43.656383+0000","last_active":"2024-03-06T07:32:43.662970+0000","last_peered":"2024-03-06T07:32:43.662970+0000","last_clean":"2024-03-06T07:32:43.662970+0000","last_became_active":"2024-03-06T07:32:43.656047+0000","last_became_peered":"2024-03-06T07:32:43.656047+0000","last_unstale":"2024-03-06T07:32:43.662970+0000","last_undegraded":"2024-03-06T07:32:43.662970+0000","last_fullsized":"2024-03-06T07:32:43.662970+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:20.685319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658617+0000","last_change":"2024-03-06T07:32:43.654589+0000","last_active":"2024-03-06T07:32:43.658617+0000","last_peered":"2024-03-06T07:32:43.658617+0000","last_clean":"2024-03-06T07:32:43.658617+0000","last_became_active":"2024-03-06T07:32:43.653253+0000","last_became_peered":"2024-03-06T07:32:43.653253+0000","last_unstale":"2024-03-06T07:32:43.658617+0000","last_undegraded":"2024-03-06T07:32:43.658617+0000","last_fullsized":"2024-03-06T07:32:43.658617+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:06.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658532+0000","last_change":"2024-03-06T07:32:43.653795+0000","last_active":"2024-03-06T07:32:43.658532+0000","last_peered":"2024-03-06T07:32:43.658532+0000","last_clean":"2024-03-06T07:32:43.658532+0000","last_became_active":"2024-03-06T07:32:43.653508+0000","last_became_peered":"2024-03-06T07:32:43.653508+0000","last_unstale":"2024-03-06T07:32:43.658532+0000","last_undegraded":"2024-03-06T07:32:43.658532+0000","last_fullsized":"2024-03-06T07:32:43.658532+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653263+0000","last_change":"2024-03-06T07:32:43.646837+0000","last_active":"2024-03-06T07:32:43.653263+0000","last_peered":"2024-03-06T07:32:43.653263+0000","last_clean":"2024-03-06T07:32:43.653263+0000","last_became_active":"2024-03-06T07:32:43.646541+0000","last_became_peered":"2024-03-06T07:32:43.646541+0000","last_unstale":"2024-03-06T07:32:43.653263+0000","last_undegraded":"2024-03-06T07:32:43.653263+0000","last_fullsized":"2024-03-06T07:32:43.653263+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662645+0000","last_change":"2024-03-06T07:32:43.650815+0000","last_active":"2024-03-06T07:32:43.662645+0000","last_peered":"2024-03-06T07:32:43.662645+0000","last_clean":"2024-03-06T07:32:43.662645+0000","last_became_active":"2024-03-06T07:32:43.650458+0000","last_became_peered":"2024-03-06T07:32:43.650458+0000","last_unstale":"2024-03-06T07:32:43.662645+0000","last_undegraded":"2024-03-06T07:32:43.662645+0000","last_fullsized":"2024-03-06T07:32:43.662645+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649322+0000","last_change":"2024-03-06T07:32:43.645026+0000","last_active":"2024-03-06T07:32:43.649322+0000","last_peered":"2024-03-06T07:32:43.649322+0000","last_clean":"2024-03-06T07:32:43.649322+0000","last_became_active":"2024-03-06T07:32:43.644709+0000","last_became_peered":"2024-03-06T07:32:43.644709+0000","last_unstale":"2024-03-06T07:32:43.649322+0000","last_undegraded":"2024-03-06T07:32:43.649322+0000","last_fullsized":"2024-03-06T07:32:43.649322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:51.868663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653299+0000","last_change":"2024-03-06T07:32:43.649074+0000","last_active":"2024-03-06T07:32:43.653299+0000","last_peered":"2024-03-06T07:32:43.653299+0000","last_clean":"2024-03-06T07:32:43.653299+0000","last_became_active":"2024-03-06T07:32:43.648805+0000","last_became_peered":"2024-03-06T07:32:43.648805+0000","last_unstale":"2024-03-06T07:32:43.653299+0000","last_undegraded":"2024-03-06T07:32:43.653299+0000","last_fullsized":"2024-03-06T07:32:43.653299+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656914+0000","last_change":"2024-03-06T07:32:43.646331+0000","last_active":"2024-03-06T07:32:43.656914+0000","last_peered":"2024-03-06T07:32:43.656914+0000","last_clean":"2024-03-06T07:32:43.656914+0000","last_became_active":"2024-03-06T07:32:43.644065+0000","last_became_peered":"2024-03-06T07:32:43.644065+0000","last_unstale":"2024-03-06T07:32:43.656914+0000","last_undegraded":"2024-03-06T07:32:43.656914+0000","last_fullsized":"2024-03-06T07:32:43.656914+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:14.556164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643965+0000","last_change":"2024-03-06T07:32:43.639955+0000","last_active":"2024-03-06T07:32:43.643965+0000","last_peered":"2024-03-06T07:32:43.643965+0000","last_clean":"2024-03-06T07:32:43.643965+0000","last_became_active":"2024-03-06T07:32:43.639019+0000","last_became_peered":"2024-03-06T07:32:43.639019+0000","last_unstale":"2024-03-06T07:32:43.643965+0000","last_undegraded":"2024-03-06T07:32:43.643965+0000","last_fullsized":"2024-03-06T07:32:43.643965+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:06.598219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.642837+0000","last_change":"2024-03-06T07:32:43.640079+0000","last_active":"2024-03-06T07:32:43.642837+0000","last_peered":"2024-03-06T07:32:43.642837+0000","last_clean":"2024-03-06T07:32:43.642837+0000","last_became_active":"2024-03-06T07:32:43.638985+0000","last_became_peered":"2024-03-06T07:32:43.638985+0000","last_unstale":"2024-03-06T07:32:43.642837+0000","last_undegraded":"2024-03-06T07:32:43.642837+0000","last_fullsized":"2024-03-06T07:32:43.642837+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:46.515813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658724+0000","last_change":"2024-03-06T07:32:43.648612+0000","last_active":"2024-03-06T07:32:43.658724+0000","last_peered":"2024-03-06T07:32:43.658724+0000","last_clean":"2024-03-06T07:32:43.658724+0000","last_became_active":"2024-03-06T07:32:43.644145+0000","last_became_peered":"2024-03-06T07:32:43.644145+0000","last_unstale":"2024-03-06T07:32:43.658724+0000","last_undegraded":"2024-03-06T07:32:43.658724+0000","last_fullsized":"2024-03-06T07:32:43.658724+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:06.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.661941+0000","last_change":"2024-03-06T07:32:43.650047+0000","last_active":"2024-03-06T07:32:43.661941+0000","last_peered":"2024-03-06T07:32:43.661941+0000","last_clean":"2024-03-06T07:32:43.661941+0000","last_became_active":"2024-03-06T07:32:43.647039+0000","last_became_peered":"2024-03-06T07:32:43.647039+0000","last_unstale":"2024-03-06T07:32:43.661941+0000","last_undegraded":"2024-03-06T07:32:43.661941+0000","last_fullsized":"2024-03-06T07:32:43.661941+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:54.253449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643297+0000","last_change":"2024-03-06T07:32:43.639196+0000","last_active":"2024-03-06T07:32:43.643297+0000","last_peered":"2024-03-06T07:32:43.643297+0000","last_clean":"2024-03-06T07:32:43.643297+0000","last_became_active":"2024-03-06T07:32:43.638850+0000","last_became_peered":"2024-03-06T07:32:43.638850+0000","last_unstale":"2024-03-06T07:32:43.643297+0000","last_undegraded":"2024-03-06T07:32:43.643297+0000","last_fullsized":"2024-03-06T07:32:43.643297+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:34.032876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"95'4","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666390+0000","last_change":"2024-03-06T07:32:42.621717+0000","last_active":"2024-03-06T07:32:43.666390+0000","last_peered":"2024-03-06T07:32:43.666390+0000","last_clean":"2024-03-06T07:32:43.666390+0000","last_became_active":"2024-03-06T07:32:42.621451+0000","last_became_peered":"2024-03-06T07:32:42.621451+0000","last_unstale":"2024-03-06T07:32:43.666390+0000","last_undegraded":"2024-03-06T07:32:43.666390+0000","last_fullsized":"2024-03-06T07:32:43.666390+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.351890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662385+0000","last_change":"2024-03-06T07:32:43.652942+0000","last_active":"2024-03-06T07:32:43.662385+0000","last_peered":"2024-03-06T07:32:43.662385+0000","last_clean":"2024-03-06T07:32:43.662385+0000","last_became_active":"2024-03-06T07:32:43.652670+0000","last_became_peered":"2024-03-06T07:32:43.652670+0000","last_unstale":"2024-03-06T07:32:43.662385+0000","last_undegraded":"2024-03-06T07:32:43.662385+0000","last_fullsized":"2024-03-06T07:32:43.662385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:45.527402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.663982+0000","last_change":"2024-03-06T07:32:42.622383+0000","last_active":"2024-03-06T07:32:43.663982+0000","last_peered":"2024-03-06T07:32:43.663982+0000","last_clean":"2024-03-06T07:32:43.663982+0000","last_became_active":"2024-03-06T07:32:42.619457+0000","last_became_peered":"2024-03-06T07:32:42.619457+0000","last_unstale":"2024-03-06T07:32:43.663982+0000","last_undegraded":"2024-03-06T07:32:43.663982+0000","last_fullsized":"2024-03-06T07:32:43.663982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:00.357359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657271+0000","last_change":"2024-03-06T07:32:43.647265+0000","last_active":"2024-03-06T07:32:43.657271+0000","last_peered":"2024-03-06T07:32:43.657271+0000","last_clean":"2024-03-06T07:32:43.657271+0000","last_became_active":"2024-03-06T07:32:43.646928+0000","last_became_peered":"2024-03-06T07:32:43.646928+0000","last_unstale":"2024-03-06T07:32:43.657271+0000","last_undegraded":"2024-03-06T07:32:43.657271+0000","last_fullsized":"2024-03-06T07:32:43.657271+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:50.207260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666804+0000","last_change":"2024-03-06T07:32:42.609197+0000","last_active":"2024-03-06T07:32:43.666804+0000","last_peered":"2024-03-06T07:32:43.666804+0000","last_clean":"2024-03-06T07:32:43.666804+0000","last_became_active":"2024-03-06T07:32:42.608652+0000","last_became_peered":"2024-03-06T07:32:42.608652+0000","last_unstale":"2024-03-06T07:32:43.666804+0000","last_undegraded":"2024-03-06T07:32:43.666804+0000","last_fullsized":"2024-03-06T07:32:43.666804+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:39.102351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656735+0000","last_change":"2024-03-06T07:32:43.653009+0000","last_active":"2024-03-06T07:32:43.656735+0000","last_peered":"2024-03-06T07:32:43.656735+0000","last_clean":"2024-03-06T07:32:43.656735+0000","last_became_active":"2024-03-06T07:32:43.652741+0000","last_became_peered":"2024-03-06T07:32:43.652741+0000","last_unstale":"2024-03-06T07:32:43.656735+0000","last_undegraded":"2024-03-06T07:32:43.656735+0000","last_fullsized":"2024-03-06T07:32:43.656735+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:30.110228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666753+0000","last_change":"2024-03-06T07:32:42.617470+0000","last_active":"2024-03-06T07:32:43.666753+0000","last_peered":"2024-03-06T07:32:43.666753+0000","last_clean":"2024-03-06T07:32:43.666753+0000","last_became_active":"2024-03-06T07:32:42.617158+0000","last_became_peered":"2024-03-06T07:32:42.617158+0000","last_unstale":"2024-03-06T07:32:43.666753+0000","last_undegraded":"2024-03-06T07:32:43.666753+0000","last_fullsized":"2024-03-06T07:32:43.666753+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:22.781433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653408+0000","last_change":"2024-03-06T07:32:43.649973+0000","last_active":"2024-03-06T07:32:43.653408+0000","last_peered":"2024-03-06T07:32:43.653408+0000","last_clean":"2024-03-06T07:32:43.653408+0000","last_became_active":"2024-03-06T07:32:43.648156+0000","last_became_peered":"2024-03-06T07:32:43.648156+0000","last_unstale":"2024-03-06T07:32:43.653408+0000","last_undegraded":"2024-03-06T07:32:43.653408+0000","last_fullsized":"2024-03-06T07:32:43.653408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:36.176515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.663183+0000","last_change":"2024-03-06T07:32:42.620568+0000","last_active":"2024-03-06T07:32:43.663183+0000","last_peered":"2024-03-06T07:32:43.663183+0000","last_clean":"2024-03-06T07:32:43.663183+0000","last_became_active":"2024-03-06T07:32:42.620141+0000","last_became_peered":"2024-03-06T07:32:42.620141+0000","last_unstale":"2024-03-06T07:32:43.663183+0000","last_undegraded":"2024-03-06T07:32:43.663183+0000","last_fullsized":"2024-03-06T07:32:43.663183+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:54.912920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653308+0000","last_change":"2024-03-06T07:32:43.650951+0000","last_active":"2024-03-06T07:32:43.653308+0000","last_peered":"2024-03-06T07:32:43.653308+0000","last_clean":"2024-03-06T07:32:43.653308+0000","last_became_active":"2024-03-06T07:32:43.650643+0000","last_became_peered":"2024-03-06T07:32:43.650643+0000","last_unstale":"2024-03-06T07:32:43.653308+0000","last_undegraded":"2024-03-06T07:32:43.653308+0000","last_fullsized":"2024-03-06T07:32:43.653308+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:54.609852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666255+0000","last_change":"2024-03-06T07:32:42.605706+0000","last_active":"2024-03-06T07:32:43.666255+0000","last_peered":"2024-03-06T07:32:43.666255+0000","last_clean":"2024-03-06T07:32:43.666255+0000","last_became_active":"2024-03-06T07:32:42.605367+0000","last_became_peered":"2024-03-06T07:32:42.605367+0000","last_unstale":"2024-03-06T07:32:43.666255+0000","last_undegraded":"2024-03-06T07:32:43.666255+0000","last_fullsized":"2024-03-06T07:32:43.666255+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.113127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658563+0000","last_change":"2024-03-06T07:32:43.647771+0000","last_active":"2024-03-06T07:32:43.658563+0000","last_peered":"2024-03-06T07:32:43.658563+0000","last_clean":"2024-03-06T07:32:43.658563+0000","last_became_active":"2024-03-06T07:32:43.647500+0000","last_became_peered":"2024-03-06T07:32:43.647500+0000","last_unstale":"2024-03-06T07:32:43.658563+0000","last_undegraded":"2024-03-06T07:32:43.658563+0000","last_fullsized":"2024-03-06T07:32:43.658563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:54.609852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.628964+0000","last_change":"2024-03-06T07:32:42.621137+0000","last_active":"2024-03-06T07:32:43.628964+0000","last_peered":"2024-03-06T07:32:43.628964+0000","last_clean":"2024-03-06T07:32:43.628964+0000","last_became_active":"2024-03-06T07:32:42.620847+0000","last_became_peered":"2024-03-06T07:32:42.620847+0000","last_unstale":"2024-03-06T07:32:43.628964+0000","last_undegraded":"2024-03-06T07:32:43.628964+0000","last_fullsized":"2024-03-06T07:32:43.628964+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.351890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654522+0000","last_change":"2024-03-06T07:32:43.652150+0000","last_active":"2024-03-06T07:32:43.654522+0000","last_peered":"2024-03-06T07:32:43.654522+0000","last_clean":"2024-03-06T07:32:43.654522+0000","last_became_active":"2024-03-06T07:32:43.651849+0000","last_became_peered":"2024-03-06T07:32:43.651849+0000","last_unstale":"2024-03-06T07:32:43.654522+0000","last_undegraded":"2024-03-06T07:32:43.654522+0000","last_fullsized":"2024-03-06T07:32:43.654522+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:51.411433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.637768+0000","last_change":"2024-03-06T07:32:42.611257+0000","last_active":"2024-03-06T07:32:43.637768+0000","last_peered":"2024-03-06T07:32:43.637768+0000","last_clean":"2024-03-06T07:32:43.637768+0000","last_became_active":"2024-03-06T07:32:42.609732+0000","last_became_peered":"2024-03-06T07:32:42.609732+0000","last_unstale":"2024-03-06T07:32:43.637768+0000","last_undegraded":"2024-03-06T07:32:43.637768+0000","last_fullsized":"2024-03-06T07:32:43.637768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:19.673368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.633424+0000","last_change":"2024-03-06T07:32:43.629803+0000","last_active":"2024-03-06T07:32:43.633424+0000","last_peered":"2024-03-06T07:32:43.633424+0000","last_clean":"2024-03-06T07:32:43.633424+0000","last_became_active":"2024-03-06T07:32:43.626973+0000","last_became_peered":"2024-03-06T07:32:43.626973+0000","last_unstale":"2024-03-06T07:32:43.633424+0000","last_undegraded":"2024-03-06T07:32:43.633424+0000","last_fullsized":"2024-03-06T07:32:43.633424+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:40.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.638157+0000","last_change":"2024-03-06T07:32:42.616171+0000","last_active":"2024-03-06T07:32:43.638157+0000","last_peered":"2024-03-06T07:32:43.638157+0000","last_clean":"2024-03-06T07:32:43.638157+0000","last_became_active":"2024-03-06T07:32:42.615864+0000","last_became_peered":"2024-03-06T07:32:42.615864+0000","last_unstale":"2024-03-06T07:32:43.638157+0000","last_undegraded":"2024-03-06T07:32:43.638157+0000","last_fullsized":"2024-03-06T07:32:43.638157+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:13.476238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656650+0000","last_change":"2024-03-06T07:32:43.653516+0000","last_active":"2024-03-06T07:32:43.656650+0000","last_peered":"2024-03-06T07:32:43.656650+0000","last_clean":"2024-03-06T07:32:43.656650+0000","last_became_active":"2024-03-06T07:32:43.653237+0000","last_became_peered":"2024-03-06T07:32:43.653237+0000","last_unstale":"2024-03-06T07:32:43.656650+0000","last_undegraded":"2024-03-06T07:32:43.656650+0000","last_fullsized":"2024-03-06T07:32:43.656650+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:55.693582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.639215+0000","last_change":"2024-03-06T07:32:42.614045+0000","last_active":"2024-03-06T07:32:43.639215+0000","last_peered":"2024-03-06T07:32:43.639215+0000","last_clean":"2024-03-06T07:32:43.639215+0000","last_became_active":"2024-03-06T07:32:42.613708+0000","last_became_peered":"2024-03-06T07:32:42.613708+0000","last_unstale":"2024-03-06T07:32:43.639215+0000","last_undegraded":"2024-03-06T07:32:43.639215+0000","last_fullsized":"2024-03-06T07:32:43.639215+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:05.586268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656947+0000","last_change":"2024-03-06T07:32:43.647006+0000","last_active":"2024-03-06T07:32:43.656947+0000","last_peered":"2024-03-06T07:32:43.656947+0000","last_clean":"2024-03-06T07:32:43.656947+0000","last_became_active":"2024-03-06T07:32:43.646746+0000","last_became_peered":"2024-03-06T07:32:43.646746+0000","last_unstale":"2024-03-06T07:32:43.656947+0000","last_undegraded":"2024-03-06T07:32:43.656947+0000","last_fullsized":"2024-03-06T07:32:43.656947+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:40.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"95'3","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.665244+0000","last_change":"2024-03-06T07:32:42.623313+0000","last_active":"2024-03-06T07:32:43.665244+0000","last_peered":"2024-03-06T07:32:43.665244+0000","last_clean":"2024-03-06T07:32:43.665244+0000","last_became_active":"2024-03-06T07:32:42.622510+0000","last_became_peered":"2024-03-06T07:32:42.622510+0000","last_unstale":"2024-03-06T07:32:43.665244+0000","last_undegraded":"2024-03-06T07:32:43.665244+0000","last_fullsized":"2024-03-06T07:32:43.665244+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:13.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.652438+0000","last_change":"2024-03-06T07:32:43.646709+0000","last_active":"2024-03-06T07:32:43.652438+0000","last_peered":"2024-03-06T07:32:43.652438+0000","last_clean":"2024-03-06T07:32:43.652438+0000","last_became_active":"2024-03-06T07:32:43.646402+0000","last_became_peered":"2024-03-06T07:32:43.646402+0000","last_unstale":"2024-03-06T07:32:43.652438+0000","last_undegraded":"2024-03-06T07:32:43.652438+0000","last_fullsized":"2024-03-06T07:32:43.652438+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:51.868663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"95'6","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666971+0000","last_change":"2024-03-06T07:32:42.616997+0000","last_active":"2024-03-06T07:32:43.666971+0000","last_peered":"2024-03-06T07:32:43.666971+0000","last_clean":"2024-03-06T07:32:43.666971+0000","last_became_active":"2024-03-06T07:32:42.616648+0000","last_became_peered":"2024-03-06T07:32:42.616648+0000","last_unstale":"2024-03-06T07:32:43.666971+0000","last_undegraded":"2024-03-06T07:32:43.666971+0000","last_fullsized":"2024-03-06T07:32:43.666971+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:48.033982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.644551+0000","last_change":"2024-03-06T07:32:43.642877+0000","last_active":"2024-03-06T07:32:43.644551+0000","last_peered":"2024-03-06T07:32:43.644551+0000","last_clean":"2024-03-06T07:32:43.644551+0000","last_became_active":"2024-03-06T07:32:43.641469+0000","last_became_peered":"2024-03-06T07:32:43.641469+0000","last_unstale":"2024-03-06T07:32:43.644551+0000","last_undegraded":"2024-03-06T07:32:43.644551+0000","last_fullsized":"2024-03-06T07:32:43.644551+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.667008+0000","last_change":"2024-03-06T07:32:42.619233+0000","last_active":"2024-03-06T07:32:43.667008+0000","last_peered":"2024-03-06T07:32:43.667008+0000","last_clean":"2024-03-06T07:32:43.667008+0000","last_became_active":"2024-03-06T07:32:42.618853+0000","last_became_peered":"2024-03-06T07:32:42.618853+0000","last_unstale":"2024-03-06T07:32:43.667008+0000","last_undegraded":"2024-03-06T07:32:43.667008+0000","last_fullsized":"2024-03-06T07:32:43.667008+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:04.406894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654690+0000","last_change":"2024-03-06T07:32:43.646688+0000","last_active":"2024-03-06T07:32:43.654690+0000","last_peered":"2024-03-06T07:32:43.654690+0000","last_clean":"2024-03-06T07:32:43.654690+0000","last_became_active":"2024-03-06T07:32:43.643233+0000","last_became_peered":"2024-03-06T07:32:43.643233+0000","last_unstale":"2024-03-06T07:32:43.654690+0000","last_undegraded":"2024-03-06T07:32:43.654690+0000","last_fullsized":"2024-03-06T07:32:43.654690+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:00.646184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.623871+0000","last_change":"2024-03-06T07:32:42.608105+0000","last_active":"2024-03-06T07:32:43.623871+0000","last_peered":"2024-03-06T07:32:43.623871+0000","last_clean":"2024-03-06T07:32:43.623871+0000","last_became_active":"2024-03-06T07:32:42.607781+0000","last_became_peered":"2024-03-06T07:32:42.607781+0000","last_unstale":"2024-03-06T07:32:43.623871+0000","last_undegraded":"2024-03-06T07:32:43.623871+0000","last_fullsized":"2024-03-06T07:32:43.623871+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:23.841099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651863+0000","last_change":"2024-03-06T07:32:43.647084+0000","last_active":"2024-03-06T07:32:43.651863+0000","last_peered":"2024-03-06T07:32:43.651863+0000","last_clean":"2024-03-06T07:32:43.651863+0000","last_became_active":"2024-03-06T07:32:43.646775+0000","last_became_peered":"2024-03-06T07:32:43.646775+0000","last_unstale":"2024-03-06T07:32:43.651863+0000","last_undegraded":"2024-03-06T07:32:43.651863+0000","last_fullsized":"2024-03-06T07:32:43.651863+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:39.904147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.618268+0000","last_change":"2024-03-06T07:32:42.621461+0000","last_active":"2024-03-06T07:32:43.618268+0000","last_peered":"2024-03-06T07:32:43.618268+0000","last_clean":"2024-03-06T07:32:43.618268+0000","last_became_active":"2024-03-06T07:32:42.621152+0000","last_became_peered":"2024-03-06T07:32:42.621152+0000","last_unstale":"2024-03-06T07:32:43.618268+0000","last_undegraded":"2024-03-06T07:32:43.618268+0000","last_fullsized":"2024-03-06T07:32:43.618268+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:21.120030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641347+0000","last_change":"2024-03-06T07:32:43.632175+0000","last_active":"2024-03-06T07:32:43.641347+0000","last_peered":"2024-03-06T07:32:43.641347+0000","last_clean":"2024-03-06T07:32:43.641347+0000","last_became_active":"2024-03-06T07:32:43.631712+0000","last_became_peered":"2024-03-06T07:32:43.631712+0000","last_unstale":"2024-03-06T07:32:43.641347+0000","last_undegraded":"2024-03-06T07:32:43.641347+0000","last_fullsized":"2024-03-06T07:32:43.641347+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:54.193380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"95'8","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664380+0000","last_change":"2024-03-06T07:32:42.618791+0000","last_active":"2024-03-06T07:32:43.664380+0000","last_peered":"2024-03-06T07:32:43.664380+0000","last_clean":"2024-03-06T07:32:43.664380+0000","last_became_active":"2024-03-06T07:32:42.618268+0000","last_became_peered":"2024-03-06T07:32:42.618268+0000","last_unstale":"2024-03-06T07:32:43.664380+0000","last_undegraded":"2024-03-06T07:32:43.664380+0000","last_fullsized":"2024-03-06T07:32:43.664380+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:24.327331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662606+0000","last_change":"2024-03-06T07:32:43.649862+0000","last_active":"2024-03-06T07:32:43.662606+0000","last_peered":"2024-03-06T07:32:43.662606+0000","last_clean":"2024-03-06T07:32:43.662606+0000","last_became_active":"2024-03-06T07:32:43.646409+0000","last_became_peered":"2024-03-06T07:32:43.646409+0000","last_unstale":"2024-03-06T07:32:43.662606+0000","last_undegraded":"2024-03-06T07:32:43.662606+0000","last_fullsized":"2024-03-06T07:32:43.662606+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664212+0000","last_change":"2024-03-06T07:32:42.612709+0000","last_active":"2024-03-06T07:32:43.664212+0000","last_peered":"2024-03-06T07:32:43.664212+0000","last_clean":"2024-03-06T07:32:43.664212+0000","last_became_active":"2024-03-06T07:32:42.612310+0000","last_became_peered":"2024-03-06T07:32:42.612310+0000","last_unstale":"2024-03-06T07:32:43.664212+0000","last_undegraded":"2024-03-06T07:32:43.664212+0000","last_fullsized":"2024-03-06T07:32:43.664212+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:18.874144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653367+0000","last_change":"2024-03-06T07:32:43.648936+0000","last_active":"2024-03-06T07:32:43.653367+0000","last_peered":"2024-03-06T07:32:43.653367+0000","last_clean":"2024-03-06T07:32:43.653367+0000","last_became_active":"2024-03-06T07:32:43.646400+0000","last_became_peered":"2024-03-06T07:32:43.646400+0000","last_unstale":"2024-03-06T07:32:43.653367+0000","last_undegraded":"2024-03-06T07:32:43.653367+0000","last_fullsized":"2024-03-06T07:32:43.653367+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:46.261548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664801+0000","last_change":"2024-03-06T07:32:42.619986+0000","last_active":"2024-03-06T07:32:43.664801+0000","last_peered":"2024-03-06T07:32:43.664801+0000","last_clean":"2024-03-06T07:32:43.664801+0000","last_became_active":"2024-03-06T07:32:42.619345+0000","last_became_peered":"2024-03-06T07:32:42.619345+0000","last_unstale":"2024-03-06T07:32:43.664801+0000","last_undegraded":"2024-03-06T07:32:43.664801+0000","last_fullsized":"2024-03-06T07:32:43.664801+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:28.447215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.652415+0000","last_change":"2024-03-06T07:32:43.646022+0000","last_active":"2024-03-06T07:32:43.652415+0000","last_peered":"2024-03-06T07:32:43.652415+0000","last_clean":"2024-03-06T07:32:43.652415+0000","last_became_active":"2024-03-06T07:32:43.645391+0000","last_became_peered":"2024-03-06T07:32:43.645391+0000","last_unstale":"2024-03-06T07:32:43.652415+0000","last_undegraded":"2024-03-06T07:32:43.652415+0000","last_fullsized":"2024-03-06T07:32:43.652415+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:06.598219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.615364+0000","last_change":"2024-03-06T07:32:42.612678+0000","last_active":"2024-03-06T07:32:43.615364+0000","last_peered":"2024-03-06T07:32:43.615364+0000","last_clean":"2024-03-06T07:32:43.615364+0000","last_became_active":"2024-03-06T07:32:42.612254+0000","last_became_peered":"2024-03-06T07:32:42.612254+0000","last_unstale":"2024-03-06T07:32:43.615364+0000","last_undegraded":"2024-03-06T07:32:43.615364+0000","last_fullsized":"2024-03-06T07:32:43.615364+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:22.090112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649704+0000","last_change":"2024-03-06T07:32:43.646511+0000","last_active":"2024-03-06T07:32:43.649704+0000","last_peered":"2024-03-06T07:32:43.649704+0000","last_clean":"2024-03-06T07:32:43.649704+0000","last_became_active":"2024-03-06T07:32:43.646219+0000","last_became_peered":"2024-03-06T07:32:43.646219+0000","last_unstale":"2024-03-06T07:32:43.649704+0000","last_undegraded":"2024-03-06T07:32:43.649704+0000","last_fullsized":"2024-03-06T07:32:43.649704+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:25.565785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666397+0000","last_change":"2024-03-06T07:32:42.613220+0000","last_active":"2024-03-06T07:32:43.666397+0000","last_peered":"2024-03-06T07:32:43.666397+0000","last_clean":"2024-03-06T07:32:43.666397+0000","last_became_active":"2024-03-06T07:32:42.610086+0000","last_became_peered":"2024-03-06T07:32:42.610086+0000","last_unstale":"2024-03-06T07:32:43.666397+0000","last_undegraded":"2024-03-06T07:32:43.666397+0000","last_fullsized":"2024-03-06T07:32:43.666397+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:52.768653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641640+0000","last_change":"2024-03-06T07:32:43.632463+0000","last_active":"2024-03-06T07:32:43.641640+0000","last_peered":"2024-03-06T07:32:43.641640+0000","last_clean":"2024-03-06T07:32:43.641640+0000","last_became_active":"2024-03-06T07:32:43.632163+0000","last_became_peered":"2024-03-06T07:32:43.632163+0000","last_unstale":"2024-03-06T07:32:43.641640+0000","last_undegraded":"2024-03-06T07:32:43.641640+0000","last_fullsized":"2024-03-06T07:32:43.641640+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:39.221449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643464+0000","last_change":"2024-03-06T07:32:42.615804+0000","last_active":"2024-03-06T07:32:43.643464+0000","last_peered":"2024-03-06T07:32:43.643464+0000","last_clean":"2024-03-06T07:32:43.643464+0000","last_became_active":"2024-03-06T07:32:42.615444+0000","last_became_peered":"2024-03-06T07:32:42.615444+0000","last_unstale":"2024-03-06T07:32:43.643464+0000","last_undegraded":"2024-03-06T07:32:43.643464+0000","last_fullsized":"2024-03-06T07:32:43.643464+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:38.521138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.645780+0000","last_change":"2024-03-06T07:32:43.642934+0000","last_active":"2024-03-06T07:32:43.645780+0000","last_peered":"2024-03-06T07:32:43.645780+0000","last_clean":"2024-03-06T07:32:43.645780+0000","last_became_active":"2024-03-06T07:32:43.642579+0000","last_became_peered":"2024-03-06T07:32:43.642579+0000","last_unstale":"2024-03-06T07:32:43.645780+0000","last_undegraded":"2024-03-06T07:32:43.645780+0000","last_fullsized":"2024-03-06T07:32:43.645780+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:28.655010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643143+0000","last_change":"2024-03-06T07:32:42.615569+0000","last_active":"2024-03-06T07:32:43.643143+0000","last_peered":"2024-03-06T07:32:43.643143+0000","last_clean":"2024-03-06T07:32:43.643143+0000","last_became_active":"2024-03-06T07:32:42.615224+0000","last_became_peered":"2024-03-06T07:32:42.615224+0000","last_unstale":"2024-03-06T07:32:43.643143+0000","last_undegraded":"2024-03-06T07:32:43.643143+0000","last_fullsized":"2024-03-06T07:32:43.643143+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:33.634241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649382+0000","last_change":"2024-03-06T07:32:43.640138+0000","last_active":"2024-03-06T07:32:43.649382+0000","last_peered":"2024-03-06T07:32:43.649382+0000","last_clean":"2024-03-06T07:32:43.649382+0000","last_became_active":"2024-03-06T07:32:43.638686+0000","last_became_peered":"2024-03-06T07:32:43.638686+0000","last_unstale":"2024-03-06T07:32:43.649382+0000","last_undegraded":"2024-03-06T07:32:43.649382+0000","last_fullsized":"2024-03-06T07:32:43.649382+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:22.125079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.615731+0000","last_change":"2024-03-06T07:32:42.605795+0000","last_active":"2024-03-06T07:32:43.615731+0000","last_peered":"2024-03-06T07:32:43.615731+0000","last_clean":"2024-03-06T07:32:43.615731+0000","last_became_active":"2024-03-06T07:32:42.605452+0000","last_became_peered":"2024-03-06T07:32:42.605452+0000","last_unstale":"2024-03-06T07:32:43.615731+0000","last_undegraded":"2024-03-06T07:32:43.615731+0000","last_fullsized":"2024-03-06T07:32:43.615731+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:28.447215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658585+0000","last_change":"2024-03-06T07:32:43.654867+0000","last_active":"2024-03-06T07:32:43.658585+0000","last_peered":"2024-03-06T07:32:43.658585+0000","last_clean":"2024-03-06T07:32:43.658585+0000","last_became_active":"2024-03-06T07:32:43.654560+0000","last_became_peered":"2024-03-06T07:32:43.654560+0000","last_unstale":"2024-03-06T07:32:43.658585+0000","last_undegraded":"2024-03-06T07:32:43.658585+0000","last_fullsized":"2024-03-06T07:32:43.658585+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.447329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664980+0000","last_change":"2024-03-06T07:32:42.620054+0000","last_active":"2024-03-06T07:32:43.664980+0000","last_peered":"2024-03-06T07:32:43.664980+0000","last_clean":"2024-03-06T07:32:43.664980+0000","last_became_active":"2024-03-06T07:32:42.618944+0000","last_became_peered":"2024-03-06T07:32:42.618944+0000","last_unstale":"2024-03-06T07:32:43.664980+0000","last_undegraded":"2024-03-06T07:32:43.664980+0000","last_fullsized":"2024-03-06T07:32:43.664980+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:19.440879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662121+0000","last_change":"2024-03-06T07:32:43.647727+0000","last_active":"2024-03-06T07:32:43.662121+0000","last_peered":"2024-03-06T07:32:43.662121+0000","last_clean":"2024-03-06T07:32:43.662121+0000","last_became_active":"2024-03-06T07:32:43.647447+0000","last_became_peered":"2024-03-06T07:32:43.647447+0000","last_unstale":"2024-03-06T07:32:43.662121+0000","last_undegraded":"2024-03-06T07:32:43.662121+0000","last_fullsized":"2024-03-06T07:32:43.662121+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:34.176922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.617018+0000","last_change":"2024-03-06T07:32:42.613251+0000","last_active":"2024-03-06T07:32:43.617018+0000","last_peered":"2024-03-06T07:32:43.617018+0000","last_clean":"2024-03-06T07:32:43.617018+0000","last_became_active":"2024-03-06T07:32:42.610157+0000","last_became_peered":"2024-03-06T07:32:42.610157+0000","last_unstale":"2024-03-06T07:32:43.617018+0000","last_undegraded":"2024-03-06T07:32:43.617018+0000","last_fullsized":"2024-03-06T07:32:43.617018+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:13.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662108+0000","last_change":"2024-03-06T07:32:43.650011+0000","last_active":"2024-03-06T07:32:43.662108+0000","last_peered":"2024-03-06T07:32:43.662108+0000","last_clean":"2024-03-06T07:32:43.662108+0000","last_became_active":"2024-03-06T07:32:43.649662+0000","last_became_peered":"2024-03-06T07:32:43.649662+0000","last_unstale":"2024-03-06T07:32:43.662108+0000","last_undegraded":"2024-03-06T07:32:43.662108+0000","last_fullsized":"2024-03-06T07:32:43.662108+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:34.620080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.616195+0000","last_change":"2024-03-06T07:32:42.616985+0000","last_active":"2024-03-06T07:32:43.616195+0000","last_peered":"2024-03-06T07:32:43.616195+0000","last_clean":"2024-03-06T07:32:43.616195+0000","last_became_active":"2024-03-06T07:32:42.616602+0000","last_became_peered":"2024-03-06T07:32:42.616602+0000","last_unstale":"2024-03-06T07:32:43.616195+0000","last_undegraded":"2024-03-06T07:32:43.616195+0000","last_fullsized":"2024-03-06T07:32:43.616195+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:00.197386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.661505+0000","last_change":"2024-03-06T07:32:43.647486+0000","last_active":"2024-03-06T07:32:43.661505+0000","last_peered":"2024-03-06T07:32:43.661505+0000","last_clean":"2024-03-06T07:32:43.661505+0000","last_became_active":"2024-03-06T07:32:43.647219+0000","last_became_peered":"2024-03-06T07:32:43.647219+0000","last_unstale":"2024-03-06T07:32:43.661505+0000","last_undegraded":"2024-03-06T07:32:43.661505+0000","last_fullsized":"2024-03-06T07:32:43.661505+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.998526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.634461+0000","last_change":"2024-03-06T07:32:42.621703+0000","last_active":"2024-03-06T07:32:43.634461+0000","last_peered":"2024-03-06T07:32:43.634461+0000","last_clean":"2024-03-06T07:32:43.634461+0000","last_became_active":"2024-03-06T07:32:42.621400+0000","last_became_peered":"2024-03-06T07:32:42.621400+0000","last_unstale":"2024-03-06T07:32:43.634461+0000","last_undegraded":"2024-03-06T07:32:43.634461+0000","last_fullsized":"2024-03-06T07:32:43.634461+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:00.197386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641716+0000","last_change":"2024-03-06T07:32:43.633478+0000","last_active":"2024-03-06T07:32:43.641716+0000","last_peered":"2024-03-06T07:32:43.641716+0000","last_clean":"2024-03-06T07:32:43.641716+0000","last_became_active":"2024-03-06T07:32:43.631199+0000","last_became_peered":"2024-03-06T07:32:43.631199+0000","last_unstale":"2024-03-06T07:32:43.641716+0000","last_undegraded":"2024-03-06T07:32:43.641716+0000","last_fullsized":"2024-03-06T07:32:43.641716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:19.263218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.616620+0000","last_change":"2024-03-06T07:32:42.606554+0000","last_active":"2024-03-06T07:32:43.616620+0000","last_peered":"2024-03-06T07:32:43.616620+0000","last_clean":"2024-03-06T07:32:43.616620+0000","last_became_active":"2024-03-06T07:32:42.606302+0000","last_became_peered":"2024-03-06T07:32:42.606302+0000","last_unstale":"2024-03-06T07:32:43.616620+0000","last_undegraded":"2024-03-06T07:32:43.616620+0000","last_fullsized":"2024-03-06T07:32:43.616620+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:00.357359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651040+0000","last_change":"2024-03-06T07:32:43.639952+0000","last_active":"2024-03-06T07:32:43.651040+0000","last_peered":"2024-03-06T07:32:43.651040+0000","last_clean":"2024-03-06T07:32:43.651040+0000","last_became_active":"2024-03-06T07:32:43.639497+0000","last_became_peered":"2024-03-06T07:32:43.639497+0000","last_unstale":"2024-03-06T07:32:43.651040+0000","last_undegraded":"2024-03-06T07:32:43.651040+0000","last_fullsized":"2024-03-06T07:32:43.651040+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:22.957826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.628775+0000","last_change":"2024-03-06T07:32:42.618333+0000","last_active":"2024-03-06T07:32:43.628775+0000","last_peered":"2024-03-06T07:32:43.628775+0000","last_clean":"2024-03-06T07:32:43.628775+0000","last_became_active":"2024-03-06T07:32:42.617948+0000","last_became_peered":"2024-03-06T07:32:42.617948+0000","last_unstale":"2024-03-06T07:32:43.628775+0000","last_undegraded":"2024-03-06T07:32:43.628775+0000","last_fullsized":"2024-03-06T07:32:43.628775+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:22.090112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641023+0000","last_change":"2024-03-06T07:32:43.627715+0000","last_active":"2024-03-06T07:32:43.641023+0000","last_peered":"2024-03-06T07:32:43.641023+0000","last_clean":"2024-03-06T07:32:43.641023+0000","last_became_active":"2024-03-06T07:32:43.626959+0000","last_became_peered":"2024-03-06T07:32:43.626959+0000","last_unstale":"2024-03-06T07:32:43.641023+0000","last_undegraded":"2024-03-06T07:32:43.641023+0000","last_fullsized":"2024-03-06T07:32:43.641023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:34.270634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.640574+0000","last_change":"2024-03-06T07:32:42.604067+0000","last_active":"2024-03-06T07:32:43.640574+0000","last_peered":"2024-03-06T07:32:43.640574+0000","last_clean":"2024-03-06T07:32:43.640574+0000","last_became_active":"2024-03-06T07:32:42.603793+0000","last_became_peered":"2024-03-06T07:32:42.603793+0000","last_unstale":"2024-03-06T07:32:43.640574+0000","last_undegraded":"2024-03-06T07:32:43.640574+0000","last_fullsized":"2024-03-06T07:32:43.640574+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:50.856711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.661683+0000","last_change":"2024-03-06T07:32:43.649207+0000","last_active":"2024-03-06T07:32:43.661683+0000","last_peered":"2024-03-06T07:32:43.661683+0000","last_clean":"2024-03-06T07:32:43.661683+0000","last_became_active":"2024-03-06T07:32:43.647697+0000","last_became_peered":"2024-03-06T07:32:43.647697+0000","last_unstale":"2024-03-06T07:32:43.661683+0000","last_undegraded":"2024-03-06T07:32:43.661683+0000","last_fullsized":"2024-03-06T07:32:43.661683+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:51.868663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.619497+0000","last_change":"2024-03-06T07:32:42.601450+0000","last_active":"2024-03-06T07:32:43.619497+0000","last_peered":"2024-03-06T07:32:43.619497+0000","last_clean":"2024-03-06T07:32:43.619497+0000","last_became_active":"2024-03-06T07:32:42.601159+0000","last_became_peered":"2024-03-06T07:32:42.601159+0000","last_unstale":"2024-03-06T07:32:43.619497+0000","last_undegraded":"2024-03-06T07:32:43.619497+0000","last_fullsized":"2024-03-06T07:32:43.619497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:41.668237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.659494+0000","last_change":"2024-03-06T07:32:43.648279+0000","last_active":"2024-03-06T07:32:43.659494+0000","last_peered":"2024-03-06T07:32:43.659494+0000","last_clean":"2024-03-06T07:32:43.659494+0000","last_became_active":"2024-03-06T07:32:43.647873+0000","last_became_peered":"2024-03-06T07:32:43.647873+0000","last_unstale":"2024-03-06T07:32:43.659494+0000","last_undegraded":"2024-03-06T07:32:43.659494+0000","last_fullsized":"2024-03-06T07:32:43.659494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:48.243786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664096+0000","last_change":"2024-03-06T07:32:42.621747+0000","last_active":"2024-03-06T07:32:43.664096+0000","last_peered":"2024-03-06T07:32:43.664096+0000","last_clean":"2024-03-06T07:32:43.664096+0000","last_became_active":"2024-03-06T07:32:42.621401+0000","last_became_peered":"2024-03-06T07:32:42.621401+0000","last_unstale":"2024-03-06T07:32:43.664096+0000","last_undegraded":"2024-03-06T07:32:43.664096+0000","last_fullsized":"2024-03-06T07:32:43.664096+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.113127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":307,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.635718+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:32:43.635718+0000","last_peered":"2024-03-06T07:32:43.635718+0000","last_clean":"2024-03-06T07:32:43.635718+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:32:43.635718+0000","last_undegraded":"2024-03-06T07:32:43.635718+0000","last_fullsized":"2024-03-06T07:32:43.635718+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542208,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1753088,"data_stored":1083145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":500204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1175552,"data_stored":499622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":500224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1159168,"data_stored":492777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542210,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":492777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":499552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542209,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1753088,"data_stored":1083145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:48.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.461+0000 7f2057400640 1 Processor -- start 2024-03-06T07:32:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.462+0000 7f2057400640 1 -- start start 2024-03-06T07:32:48.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.463+0000 7f2057400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 0x7f205006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.463+0000 7f2057400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205006ee70 con 0x7f205015f9d0 2024-03-06T07:32:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.463+0000 7f2057400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205006cef0 con 0x7f205015fe30 2024-03-06T07:32:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.463+0000 7f2057400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205006d1d0 con 0x7f2050144c00 2024-03-06T07:32:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f205015f9d0 0x7f20500ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43760/0 (socket says 172.21.15.97:43760) 2024-03-06T07:32:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204fe00640 1 -- 172.21.15.97:0/1112683544 learned_addr learned my addr 172.21.15.97:0/1112683544 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f2054a00640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 0x7f205006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429500363 0 0) 0x7f203c001110 con 0x7f205015f9d0 2024-03-06T07:32:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2034003520 con 0x7f205015f9d0 2024-03-06T07:32:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716813256 0 0) 0x7f2034003520 con 0x7f205015f9d0 2024-03-06T07:32:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 msgr2=0x7f205006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 0x7f205006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 >> v1:172.21.15.135:6789/0 conn(0x7f205015fe30 legacy=0x7f20500ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205006d4b0 con 0x7f205015f9d0 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f203c001840 con 0x7f205015f9d0 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.464+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 >> v1:172.21.15.97:6789/0 conn(0x7f205015f9d0 legacy=0x7f20500ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f204f400640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20340039e0 0x7f2034003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f204f400640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2034004800 0x7f2034004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f204f400640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 0x7f20340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205006d1d0 con 0x7f20340039e0 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203400ba70 con 0x7f2034004800 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203400bc20 con 0x7f2050144c00 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f2054a00640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20340039e0 0x7f2034003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f2054a00640 1 -- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 msgr2=0x7f20340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f2054a00640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2050144c00 0x7f20340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f2054a00640 1 -- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2034004800 msgr2=0x7f2034004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f2054a00640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2034004800 0x7f2034004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.465+0000 7f2054a00640 1 -- 172.21.15.97:0/1112683544 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034003520 con 0x7f20340039e0 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.466+0000 7f2054a00640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20340039e0 0x7f2034003dc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f2040009f90 tx=0x7f2034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b833371af40388f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.466+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040039070 con 0x7f20340039e0 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.466+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204002cbd0 con 0x7f20340039e0 2024-03-06T07:32:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.466+0000 7f204f400640 1 -- 172.21.15.97:0/1112683544 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040009c10 con 0x7f20340039e0 2024-03-06T07:32:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2057400640 1 -- 172.21.15.97:0/1112683544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20340039e0 msgr2=0x7f2034003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2057400640 1 --2- 172.21.15.97:0/1112683544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20340039e0 0x7f2034003dc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f2040009f90 tx=0x7f2034005830 comp rx=0 tx=0).stop 2024-03-06T07:32:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2055400640 1 -- 172.21.15.97:0/1112683544 reap_dead start 2024-03-06T07:32:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2057400640 1 -- 172.21.15.97:0/1112683544 shutdown_connections 2024-03-06T07:32:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2057400640 1 -- 172.21.15.97:0/1112683544 >> 172.21.15.97:0/1112683544 conn(0x7f20500b1730 msgr2=0x7f20500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2057400640 1 -- 172.21.15.97:0/1112683544 shutdown_connections 2024-03-06T07:32:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.467+0000 7f2057400640 1 -- 172.21.15.97:0/1112683544 wait complete. 2024-03-06T07:32:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.469+0000 7f2057400640 1 Processor -- start 2024-03-06T07:32:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.469+0000 7f2057400640 1 -- start start 2024-03-06T07:32:48.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f2057400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20340039e0 0x7f205013a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f2057400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 0x7f205015a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f2057400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20500ce730 0x7f205015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f2057400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2050160730 con 0x7f20500ce2d0 2024-03-06T07:32:48.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f2057400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2050160280 con 0x7f20340039e0 2024-03-06T07:32:48.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f2057400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500ceec0 con 0x7f20500ce730 2024-03-06T07:32:48.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f204fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 0x7f205015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f204fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 0x7f205015a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34044/0 (socket says 172.21.15.97:34044) 2024-03-06T07:32:48.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.471+0000 7f204fe00640 1 -- 172.21.15.97:0/925891019 learned_addr learned my addr 172.21.15.97:0/925891019 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:48.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f2054a00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20340039e0 0x7f205013a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f2055400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20500ce730 0x7f205015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204fe00640 1 -- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20500ce730 msgr2=0x7f205015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204fe00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20500ce730 0x7f205015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204fe00640 1 -- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20340039e0 msgr2=0x7f205013a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204fe00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20340039e0 0x7f205013a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204fe00640 1 -- 172.21.15.97:0/925891019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205013a000 con 0x7f20500ce2d0 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f2055400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20500ce730 0x7f205015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:48.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204fe00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 0x7f205015a680 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c7a0 tx=0x7f203c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c018070 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c00ce50 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0047e0 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.472+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205013d340 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.473+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205013d900 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.474+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205004f330 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.475+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f203c004980 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.476+0000 7f204cc00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203006b4d0 0x7f203006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.476+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f203c089750 con 0x7f20500ce2d0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.476+0000 7f2054a00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203006b4d0 0x7f203006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.477+0000 7f2054a00640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203006b4d0 0x7f203006d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2040036000 tx=0x7f204002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.481+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f205006ee70 con 0x7f20500ce2d0 2024-03-06T07:32:48.567 INFO:tasks.ceph.mds.c.smithi097.stderr:2024-03-06T07:32:48.565+0000 7f9d11400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.674+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f205013dad0 con 0x7f203006b4d0 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.692+0000 7f204cc00640 1 -- 172.21.15.97:0/925891019 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252013 (secure 0 0 0) 0x7f205013dad0 con 0x7f203006b4d0 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203006b4d0 msgr2=0x7f203006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203006b4d0 0x7f203006d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2040036000 tx=0x7f204002e000 comp rx=0 tx=0).stop 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 msgr2=0x7f205015a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 0x7f205015a680 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c7a0 tx=0x7f203c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 shutdown_connections 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203006b4d0 0x7f203006d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20500ce730 0x7f205015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20500ce2d0 0x7f205015a680 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 --2- 172.21.15.97:0/925891019 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20340039e0 0x7f205013a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.697+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 >> 172.21.15.97:0/925891019 conn(0x7f20500b1730 msgr2=0x7f20500b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.698+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 shutdown_connections 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.698+0000 7f2057400640 1 -- 172.21.15.97:0/925891019 wait complete. 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-03-06T07:32:46.993823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259744,"kb_used_data":10284,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714976,"statfs":{"total":773094113280,"available":772828135424,"internally_reserved":0,"allocated":10530816,"data_stored":5151446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413191"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662722+0000","last_change":"2024-03-06T07:32:43.653711+0000","last_active":"2024-03-06T07:32:43.662722+0000","last_peered":"2024-03-06T07:32:43.662722+0000","last_clean":"2024-03-06T07:32:43.662722+0000","last_became_active":"2024-03-06T07:32:43.653456+0000","last_became_peered":"2024-03-06T07:32:43.653456+0000","last_unstale":"2024-03-06T07:32:43.662722+0000","last_undegraded":"2024-03-06T07:32:43.662722+0000","last_fullsized":"2024-03-06T07:32:43.662722+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:18.541030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658543+0000","last_change":"2024-03-06T07:32:43.653886+0000","last_active":"2024-03-06T07:32:43.658543+0000","last_peered":"2024-03-06T07:32:43.658543+0000","last_clean":"2024-03-06T07:32:43.658543+0000","last_became_active":"2024-03-06T07:32:43.653566+0000","last_became_peered":"2024-03-06T07:32:43.653566+0000","last_unstale":"2024-03-06T07:32:43.658543+0000","last_undegraded":"2024-03-06T07:32:43.658543+0000","last_fullsized":"2024-03-06T07:32:43.658543+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:54.609852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.663561+0000","last_change":"2024-03-06T07:32:43.657624+0000","last_active":"2024-03-06T07:32:43.663561+0000","last_peered":"2024-03-06T07:32:43.663561+0000","last_clean":"2024-03-06T07:32:43.663561+0000","last_became_active":"2024-03-06T07:32:43.657198+0000","last_became_peered":"2024-03-06T07:32:43.657198+0000","last_unstale":"2024-03-06T07:32:43.663561+0000","last_undegraded":"2024-03-06T07:32:43.663561+0000","last_fullsized":"2024-03-06T07:32:43.663561+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:45.319214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.659651+0000","last_change":"2024-03-06T07:32:43.649794+0000","last_active":"2024-03-06T07:32:43.659651+0000","last_peered":"2024-03-06T07:32:43.659651+0000","last_clean":"2024-03-06T07:32:43.659651+0000","last_became_active":"2024-03-06T07:32:43.649454+0000","last_became_peered":"2024-03-06T07:32:43.649454+0000","last_unstale":"2024-03-06T07:32:43.659651+0000","last_undegraded":"2024-03-06T07:32:43.659651+0000","last_fullsized":"2024-03-06T07:32:43.659651+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:28.102188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657404+0000","last_change":"2024-03-06T07:32:43.647536+0000","last_active":"2024-03-06T07:32:43.657404+0000","last_peered":"2024-03-06T07:32:43.657404+0000","last_clean":"2024-03-06T07:32:43.657404+0000","last_became_active":"2024-03-06T07:32:43.643895+0000","last_became_peered":"2024-03-06T07:32:43.643895+0000","last_unstale":"2024-03-06T07:32:43.657404+0000","last_undegraded":"2024-03-06T07:32:43.657404+0000","last_fullsized":"2024-03-06T07:32:43.657404+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:06.598219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656334+0000","last_change":"2024-03-06T07:32:43.654640+0000","last_active":"2024-03-06T07:32:43.656334+0000","last_peered":"2024-03-06T07:32:43.656334+0000","last_clean":"2024-03-06T07:32:43.656334+0000","last_became_active":"2024-03-06T07:32:43.654321+0000","last_became_peered":"2024-03-06T07:32:43.654321+0000","last_unstale":"2024-03-06T07:32:43.656334+0000","last_undegraded":"2024-03-06T07:32:43.656334+0000","last_fullsized":"2024-03-06T07:32:43.656334+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:06.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654843+0000","last_change":"2024-03-06T07:32:43.653283+0000","last_active":"2024-03-06T07:32:43.654843+0000","last_peered":"2024-03-06T07:32:43.654843+0000","last_clean":"2024-03-06T07:32:43.654843+0000","last_became_active":"2024-03-06T07:32:43.652636+0000","last_became_peered":"2024-03-06T07:32:43.652636+0000","last_unstale":"2024-03-06T07:32:43.654843+0000","last_undegraded":"2024-03-06T07:32:43.654843+0000","last_fullsized":"2024-03-06T07:32:43.654843+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.447329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.644088+0000","last_change":"2024-03-06T07:32:43.638636+0000","last_active":"2024-03-06T07:32:43.644088+0000","last_peered":"2024-03-06T07:32:43.644088+0000","last_clean":"2024-03-06T07:32:43.644088+0000","last_became_active":"2024-03-06T07:32:43.638355+0000","last_became_peered":"2024-03-06T07:32:43.638355+0000","last_unstale":"2024-03-06T07:32:43.644088+0000","last_undegraded":"2024-03-06T07:32:43.644088+0000","last_fullsized":"2024-03-06T07:32:43.644088+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.998526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651259+0000","last_change":"2024-03-06T07:32:43.647311+0000","last_active":"2024-03-06T07:32:43.651259+0000","last_peered":"2024-03-06T07:32:43.651259+0000","last_clean":"2024-03-06T07:32:43.651259+0000","last_became_active":"2024-03-06T07:32:43.647013+0000","last_became_peered":"2024-03-06T07:32:43.647013+0000","last_unstale":"2024-03-06T07:32:43.651259+0000","last_undegraded":"2024-03-06T07:32:43.651259+0000","last_fullsized":"2024-03-06T07:32:43.651259+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:09.989310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657053+0000","last_change":"2024-03-06T07:32:43.645119+0000","last_active":"2024-03-06T07:32:43.657053+0000","last_peered":"2024-03-06T07:32:43.657053+0000","last_clean":"2024-03-06T07:32:43.657053+0000","last_became_active":"2024-03-06T07:32:43.644796+0000","last_became_peered":"2024-03-06T07:32:43.644796+0000","last_unstale":"2024-03-06T07:32:43.657053+0000","last_undegraded":"2024-03-06T07:32:43.657053+0000","last_fullsized":"2024-03-06T07:32:43.657053+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:01.369311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.652391+0000","last_change":"2024-03-06T07:32:43.645762+0000","last_active":"2024-03-06T07:32:43.652391+0000","last_peered":"2024-03-06T07:32:43.652391+0000","last_clean":"2024-03-06T07:32:43.652391+0000","last_became_active":"2024-03-06T07:32:43.644993+0000","last_became_peered":"2024-03-06T07:32:43.644993+0000","last_unstale":"2024-03-06T07:32:43.652391+0000","last_undegraded":"2024-03-06T07:32:43.652391+0000","last_fullsized":"2024-03-06T07:32:43.652391+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.998526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651211+0000","last_change":"2024-03-06T07:32:43.645432+0000","last_active":"2024-03-06T07:32:43.651211+0000","last_peered":"2024-03-06T07:32:43.651211+0000","last_clean":"2024-03-06T07:32:43.651211+0000","last_became_active":"2024-03-06T07:32:43.645140+0000","last_became_peered":"2024-03-06T07:32:43.645140+0000","last_unstale":"2024-03-06T07:32:43.651211+0000","last_undegraded":"2024-03-06T07:32:43.651211+0000","last_fullsized":"2024-03-06T07:32:43.651211+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:51.801166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662432+0000","last_change":"2024-03-06T07:32:43.655963+0000","last_active":"2024-03-06T07:32:43.662432+0000","last_peered":"2024-03-06T07:32:43.662432+0000","last_clean":"2024-03-06T07:32:43.662432+0000","last_became_active":"2024-03-06T07:32:43.654842+0000","last_became_peered":"2024-03-06T07:32:43.654842+0000","last_unstale":"2024-03-06T07:32:43.662432+0000","last_undegraded":"2024-03-06T07:32:43.662432+0000","last_fullsized":"2024-03-06T07:32:43.662432+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:35.984800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.642865+0000","last_change":"2024-03-06T07:32:43.640380+0000","last_active":"2024-03-06T07:32:43.642865+0000","last_peered":"2024-03-06T07:32:43.642865+0000","last_clean":"2024-03-06T07:32:43.642865+0000","last_became_active":"2024-03-06T07:32:43.640033+0000","last_became_peered":"2024-03-06T07:32:43.640033+0000","last_unstale":"2024-03-06T07:32:43.642865+0000","last_undegraded":"2024-03-06T07:32:43.642865+0000","last_fullsized":"2024-03-06T07:32:43.642865+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:39.220605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657760+0000","last_change":"2024-03-06T07:32:43.646577+0000","last_active":"2024-03-06T07:32:43.657760+0000","last_peered":"2024-03-06T07:32:43.657760+0000","last_clean":"2024-03-06T07:32:43.657760+0000","last_became_active":"2024-03-06T07:32:43.646299+0000","last_became_peered":"2024-03-06T07:32:43.646299+0000","last_unstale":"2024-03-06T07:32:43.657760+0000","last_undegraded":"2024-03-06T07:32:43.657760+0000","last_fullsized":"2024-03-06T07:32:43.657760+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.645597+0000","last_change":"2024-03-06T07:32:43.644725+0000","last_active":"2024-03-06T07:32:43.645597+0000","last_peered":"2024-03-06T07:32:43.645597+0000","last_clean":"2024-03-06T07:32:43.645597+0000","last_became_active":"2024-03-06T07:32:43.644331+0000","last_became_peered":"2024-03-06T07:32:43.644331+0000","last_unstale":"2024-03-06T07:32:43.645597+0000","last_undegraded":"2024-03-06T07:32:43.645597+0000","last_fullsized":"2024-03-06T07:32:43.645597+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:40.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654816+0000","last_change":"2024-03-06T07:32:43.647347+0000","last_active":"2024-03-06T07:32:43.654816+0000","last_peered":"2024-03-06T07:32:43.654816+0000","last_clean":"2024-03-06T07:32:43.654816+0000","last_became_active":"2024-03-06T07:32:43.646883+0000","last_became_peered":"2024-03-06T07:32:43.646883+0000","last_unstale":"2024-03-06T07:32:43.654816+0000","last_undegraded":"2024-03-06T07:32:43.654816+0000","last_fullsized":"2024-03-06T07:32:43.654816+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657793+0000","last_change":"2024-03-06T07:32:43.648489+0000","last_active":"2024-03-06T07:32:43.657793+0000","last_peered":"2024-03-06T07:32:43.657793+0000","last_clean":"2024-03-06T07:32:43.657793+0000","last_became_active":"2024-03-06T07:32:43.648205+0000","last_became_peered":"2024-03-06T07:32:43.648205+0000","last_unstale":"2024-03-06T07:32:43.657793+0000","last_undegraded":"2024-03-06T07:32:43.657793+0000","last_fullsized":"2024-03-06T07:32:43.657793+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.447329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654871+0000","last_change":"2024-03-06T07:32:43.647471+0000","last_active":"2024-03-06T07:32:43.654871+0000","last_peered":"2024-03-06T07:32:43.654871+0000","last_clean":"2024-03-06T07:32:43.654871+0000","last_became_active":"2024-03-06T07:32:43.645446+0000","last_became_peered":"2024-03-06T07:32:43.645446+0000","last_unstale":"2024-03-06T07:32:43.654871+0000","last_undegraded":"2024-03-06T07:32:43.654871+0000","last_fullsized":"2024-03-06T07:32:43.654871+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:28.161262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649802+0000","last_change":"2024-03-06T07:32:43.642238+0000","last_active":"2024-03-06T07:32:43.649802+0000","last_peered":"2024-03-06T07:32:43.649802+0000","last_clean":"2024-03-06T07:32:43.649802+0000","last_became_active":"2024-03-06T07:32:43.641966+0000","last_became_peered":"2024-03-06T07:32:43.641966+0000","last_unstale":"2024-03-06T07:32:43.649802+0000","last_undegraded":"2024-03-06T07:32:43.649802+0000","last_fullsized":"2024-03-06T07:32:43.649802+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:09.710140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662970+0000","last_change":"2024-03-06T07:32:43.656383+0000","last_active":"2024-03-06T07:32:43.662970+0000","last_peered":"2024-03-06T07:32:43.662970+0000","last_clean":"2024-03-06T07:32:43.662970+0000","last_became_active":"2024-03-06T07:32:43.656047+0000","last_became_peered":"2024-03-06T07:32:43.656047+0000","last_unstale":"2024-03-06T07:32:43.662970+0000","last_undegraded":"2024-03-06T07:32:43.662970+0000","last_fullsized":"2024-03-06T07:32:43.662970+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:20.685319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658617+0000","last_change":"2024-03-06T07:32:43.654589+0000","last_active":"2024-03-06T07:32:43.658617+0000","last_peered":"2024-03-06T07:32:43.658617+0000","last_clean":"2024-03-06T07:32:43.658617+0000","last_became_active":"2024-03-06T07:32:43.653253+0000","last_became_peered":"2024-03-06T07:32:43.653253+0000","last_unstale":"2024-03-06T07:32:43.658617+0000","last_undegraded":"2024-03-06T07:32:43.658617+0000","last_fullsized":"2024-03-06T07:32:43.658617+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:06.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658532+0000","last_change":"2024-03-06T07:32:43.653795+0000","last_active":"2024-03-06T07:32:43.658532+0000","last_peered":"2024-03-06T07:32:43.658532+0000","last_clean":"2024-03-06T07:32:43.658532+0000","last_became_active":"2024-03-06T07:32:43.653508+0000","last_became_peered":"2024-03-06T07:32:43.653508+0000","last_unstale":"2024-03-06T07:32:43.658532+0000","last_undegraded":"2024-03-06T07:32:43.658532+0000","last_fullsized":"2024-03-06T07:32:43.658532+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653263+0000","last_change":"2024-03-06T07:32:43.646837+0000","last_active":"2024-03-06T07:32:43.653263+0000","last_peered":"2024-03-06T07:32:43.653263+0000","last_clean":"2024-03-06T07:32:43.653263+0000","last_became_active":"2024-03-06T07:32:43.646541+0000","last_became_peered":"2024-03-06T07:32:43.646541+0000","last_unstale":"2024-03-06T07:32:43.653263+0000","last_undegraded":"2024-03-06T07:32:43.653263+0000","last_fullsized":"2024-03-06T07:32:43.653263+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662645+0000","last_change":"2024-03-06T07:32:43.650815+0000","last_active":"2024-03-06T07:32:43.662645+0000","last_peered":"2024-03-06T07:32:43.662645+0000","last_clean":"2024-03-06T07:32:43.662645+0000","last_became_active":"2024-03-06T07:32:43.650458+0000","last_became_peered":"2024-03-06T07:32:43.650458+0000","last_unstale":"2024-03-06T07:32:43.662645+0000","last_undegraded":"2024-03-06T07:32:43.662645+0000","last_fullsized":"2024-03-06T07:32:43.662645+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649322+0000","last_change":"2024-03-06T07:32:43.645026+0000","last_active":"2024-03-06T07:32:43.649322+0000","last_peered":"2024-03-06T07:32:43.649322+0000","last_clean":"2024-03-06T07:32:43.649322+0000","last_became_active":"2024-03-06T07:32:43.644709+0000","last_became_peered":"2024-03-06T07:32:43.644709+0000","last_unstale":"2024-03-06T07:32:43.649322+0000","last_undegraded":"2024-03-06T07:32:43.649322+0000","last_fullsized":"2024-03-06T07:32:43.649322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:51.868663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653299+0000","last_change":"2024-03-06T07:32:43.649074+0000","last_active":"2024-03-06T07:32:43.653299+0000","last_peered":"2024-03-06T07:32:43.653299+0000","last_clean":"2024-03-06T07:32:43.653299+0000","last_became_active":"2024-03-06T07:32:43.648805+0000","last_became_peered":"2024-03-06T07:32:43.648805+0000","last_unstale":"2024-03-06T07:32:43.653299+0000","last_undegraded":"2024-03-06T07:32:43.653299+0000","last_fullsized":"2024-03-06T07:32:43.653299+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:42.561205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656914+0000","last_change":"2024-03-06T07:32:43.646331+0000","last_active":"2024-03-06T07:32:43.656914+0000","last_peered":"2024-03-06T07:32:43.656914+0000","last_clean":"2024-03-06T07:32:43.656914+0000","last_became_active":"2024-03-06T07:32:43.644065+0000","last_became_peered":"2024-03-06T07:32:43.644065+0000","last_unstale":"2024-03-06T07:32:43.656914+0000","last_undegraded":"2024-03-06T07:32:43.656914+0000","last_fullsized":"2024-03-06T07:32:43.656914+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:14.556164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643965+0000","last_change":"2024-03-06T07:32:43.639955+0000","last_active":"2024-03-06T07:32:43.643965+0000","last_peered":"2024-03-06T07:32:43.643965+0000","last_clean":"2024-03-06T07:32:43.643965+0000","last_became_active":"2024-03-06T07:32:43.639019+0000","last_became_peered":"2024-03-06T07:32:43.639019+0000","last_unstale":"2024-03-06T07:32:43.643965+0000","last_undegraded":"2024-03-06T07:32:43.643965+0000","last_fullsized":"2024-03-06T07:32:43.643965+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:06.598219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.642837+0000","last_change":"2024-03-06T07:32:43.640079+0000","last_active":"2024-03-06T07:32:43.642837+0000","last_peered":"2024-03-06T07:32:43.642837+0000","last_clean":"2024-03-06T07:32:43.642837+0000","last_became_active":"2024-03-06T07:32:43.638985+0000","last_became_peered":"2024-03-06T07:32:43.638985+0000","last_unstale":"2024-03-06T07:32:43.642837+0000","last_undegraded":"2024-03-06T07:32:43.642837+0000","last_fullsized":"2024-03-06T07:32:43.642837+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:46.515813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658724+0000","last_change":"2024-03-06T07:32:43.648612+0000","last_active":"2024-03-06T07:32:43.658724+0000","last_peered":"2024-03-06T07:32:43.658724+0000","last_clean":"2024-03-06T07:32:43.658724+0000","last_became_active":"2024-03-06T07:32:43.644145+0000","last_became_peered":"2024-03-06T07:32:43.644145+0000","last_unstale":"2024-03-06T07:32:43.658724+0000","last_undegraded":"2024-03-06T07:32:43.658724+0000","last_fullsized":"2024-03-06T07:32:43.658724+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:06.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.661941+0000","last_change":"2024-03-06T07:32:43.650047+0000","last_active":"2024-03-06T07:32:43.661941+0000","last_peered":"2024-03-06T07:32:43.661941+0000","last_clean":"2024-03-06T07:32:43.661941+0000","last_became_active":"2024-03-06T07:32:43.647039+0000","last_became_peered":"2024-03-06T07:32:43.647039+0000","last_unstale":"2024-03-06T07:32:43.661941+0000","last_undegraded":"2024-03-06T07:32:43.661941+0000","last_fullsized":"2024-03-06T07:32:43.661941+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:54.253449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643297+0000","last_change":"2024-03-06T07:32:43.639196+0000","last_active":"2024-03-06T07:32:43.643297+0000","last_peered":"2024-03-06T07:32:43.643297+0000","last_clean":"2024-03-06T07:32:43.643297+0000","last_became_active":"2024-03-06T07:32:43.638850+0000","last_became_peered":"2024-03-06T07:32:43.638850+0000","last_unstale":"2024-03-06T07:32:43.643297+0000","last_undegraded":"2024-03-06T07:32:43.643297+0000","last_fullsized":"2024-03-06T07:32:43.643297+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:34.032876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"95'4","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666390+0000","last_change":"2024-03-06T07:32:42.621717+0000","last_active":"2024-03-06T07:32:43.666390+0000","last_peered":"2024-03-06T07:32:43.666390+0000","last_clean":"2024-03-06T07:32:43.666390+0000","last_became_active":"2024-03-06T07:32:42.621451+0000","last_became_peered":"2024-03-06T07:32:42.621451+0000","last_unstale":"2024-03-06T07:32:43.666390+0000","last_undegraded":"2024-03-06T07:32:43.666390+0000","last_fullsized":"2024-03-06T07:32:43.666390+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.351890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662385+0000","last_change":"2024-03-06T07:32:43.652942+0000","last_active":"2024-03-06T07:32:43.662385+0000","last_peered":"2024-03-06T07:32:43.662385+0000","last_clean":"2024-03-06T07:32:43.662385+0000","last_became_active":"2024-03-06T07:32:43.652670+0000","last_became_peered":"2024-03-06T07:32:43.652670+0000","last_unstale":"2024-03-06T07:32:43.662385+0000","last_undegraded":"2024-03-06T07:32:43.662385+0000","last_fullsized":"2024-03-06T07:32:43.662385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:45.527402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.663982+0000","last_change":"2024-03-06T07:32:42.622383+0000","last_active":"2024-03-06T07:32:43.663982+0000","last_peered":"2024-03-06T07:32:43.663982+0000","last_clean":"2024-03-06T07:32:43.663982+0000","last_became_active":"2024-03-06T07:32:42.619457+0000","last_became_peered":"2024-03-06T07:32:42.619457+0000","last_unstale":"2024-03-06T07:32:43.663982+0000","last_undegraded":"2024-03-06T07:32:43.663982+0000","last_fullsized":"2024-03-06T07:32:43.663982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:00.357359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.657271+0000","last_change":"2024-03-06T07:32:43.647265+0000","last_active":"2024-03-06T07:32:43.657271+0000","last_peered":"2024-03-06T07:32:43.657271+0000","last_clean":"2024-03-06T07:32:43.657271+0000","last_became_active":"2024-03-06T07:32:43.646928+0000","last_became_peered":"2024-03-06T07:32:43.646928+0000","last_unstale":"2024-03-06T07:32:43.657271+0000","last_undegraded":"2024-03-06T07:32:43.657271+0000","last_fullsized":"2024-03-06T07:32:43.657271+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:50.207260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666804+0000","last_change":"2024-03-06T07:32:42.609197+0000","last_active":"2024-03-06T07:32:43.666804+0000","last_peered":"2024-03-06T07:32:43.666804+0000","last_clean":"2024-03-06T07:32:43.666804+0000","last_became_active":"2024-03-06T07:32:42.608652+0000","last_became_peered":"2024-03-06T07:32:42.608652+0000","last_unstale":"2024-03-06T07:32:43.666804+0000","last_undegraded":"2024-03-06T07:32:43.666804+0000","last_fullsized":"2024-03-06T07:32:43.666804+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:39.102351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656735+0000","last_change":"2024-03-06T07:32:43.653009+0000","last_active":"2024-03-06T07:32:43.656735+0000","last_peered":"2024-03-06T07:32:43.656735+0000","last_clean":"2024-03-06T07:32:43.656735+0000","last_became_active":"2024-03-06T07:32:43.652741+0000","last_became_peered":"2024-03-06T07:32:43.652741+0000","last_unstale":"2024-03-06T07:32:43.656735+0000","last_undegraded":"2024-03-06T07:32:43.656735+0000","last_fullsized":"2024-03-06T07:32:43.656735+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:30.110228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666753+0000","last_change":"2024-03-06T07:32:42.617470+0000","last_active":"2024-03-06T07:32:43.666753+0000","last_peered":"2024-03-06T07:32:43.666753+0000","last_clean":"2024-03-06T07:32:43.666753+0000","last_became_active":"2024-03-06T07:32:42.617158+0000","last_became_peered":"2024-03-06T07:32:42.617158+0000","last_unstale":"2024-03-06T07:32:43.666753+0000","last_undegraded":"2024-03-06T07:32:43.666753+0000","last_fullsized":"2024-03-06T07:32:43.666753+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:22.781433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653408+0000","last_change":"2024-03-06T07:32:43.649973+0000","last_active":"2024-03-06T07:32:43.653408+0000","last_peered":"2024-03-06T07:32:43.653408+0000","last_clean":"2024-03-06T07:32:43.653408+0000","last_became_active":"2024-03-06T07:32:43.648156+0000","last_became_peered":"2024-03-06T07:32:43.648156+0000","last_unstale":"2024-03-06T07:32:43.653408+0000","last_undegraded":"2024-03-06T07:32:43.653408+0000","last_fullsized":"2024-03-06T07:32:43.653408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:36.176515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.663183+0000","last_change":"2024-03-06T07:32:42.620568+0000","last_active":"2024-03-06T07:32:43.663183+0000","last_peered":"2024-03-06T07:32:43.663183+0000","last_clean":"2024-03-06T07:32:43.663183+0000","last_became_active":"2024-03-06T07:32:42.620141+0000","last_became_peered":"2024-03-06T07:32:42.620141+0000","last_unstale":"2024-03-06T07:32:43.663183+0000","last_undegraded":"2024-03-06T07:32:43.663183+0000","last_fullsized":"2024-03-06T07:32:43.663183+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:54.912920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653308+0000","last_change":"2024-03-06T07:32:43.650951+0000","last_active":"2024-03-06T07:32:43.653308+0000","last_peered":"2024-03-06T07:32:43.653308+0000","last_clean":"2024-03-06T07:32:43.653308+0000","last_became_active":"2024-03-06T07:32:43.650643+0000","last_became_peered":"2024-03-06T07:32:43.650643+0000","last_unstale":"2024-03-06T07:32:43.653308+0000","last_undegraded":"2024-03-06T07:32:43.653308+0000","last_fullsized":"2024-03-06T07:32:43.653308+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:54.609852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666255+0000","last_change":"2024-03-06T07:32:42.605706+0000","last_active":"2024-03-06T07:32:43.666255+0000","last_peered":"2024-03-06T07:32:43.666255+0000","last_clean":"2024-03-06T07:32:43.666255+0000","last_became_active":"2024-03-06T07:32:42.605367+0000","last_became_peered":"2024-03-06T07:32:42.605367+0000","last_unstale":"2024-03-06T07:32:43.666255+0000","last_undegraded":"2024-03-06T07:32:43.666255+0000","last_fullsized":"2024-03-06T07:32:43.666255+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.113127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658563+0000","last_change":"2024-03-06T07:32:43.647771+0000","last_active":"2024-03-06T07:32:43.658563+0000","last_peered":"2024-03-06T07:32:43.658563+0000","last_clean":"2024-03-06T07:32:43.658563+0000","last_became_active":"2024-03-06T07:32:43.647500+0000","last_became_peered":"2024-03-06T07:32:43.647500+0000","last_unstale":"2024-03-06T07:32:43.658563+0000","last_undegraded":"2024-03-06T07:32:43.658563+0000","last_fullsized":"2024-03-06T07:32:43.658563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:54.609852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.628964+0000","last_change":"2024-03-06T07:32:42.621137+0000","last_active":"2024-03-06T07:32:43.628964+0000","last_peered":"2024-03-06T07:32:43.628964+0000","last_clean":"2024-03-06T07:32:43.628964+0000","last_became_active":"2024-03-06T07:32:42.620847+0000","last_became_peered":"2024-03-06T07:32:42.620847+0000","last_unstale":"2024-03-06T07:32:43.628964+0000","last_undegraded":"2024-03-06T07:32:43.628964+0000","last_fullsized":"2024-03-06T07:32:43.628964+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.351890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654522+0000","last_change":"2024-03-06T07:32:43.652150+0000","last_active":"2024-03-06T07:32:43.654522+0000","last_peered":"2024-03-06T07:32:43.654522+0000","last_clean":"2024-03-06T07:32:43.654522+0000","last_became_active":"2024-03-06T07:32:43.651849+0000","last_became_peered":"2024-03-06T07:32:43.651849+0000","last_unstale":"2024-03-06T07:32:43.654522+0000","last_undegraded":"2024-03-06T07:32:43.654522+0000","last_fullsized":"2024-03-06T07:32:43.654522+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:51.411433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.637768+0000","last_change":"2024-03-06T07:32:42.611257+0000","last_active":"2024-03-06T07:32:43.637768+0000","last_peered":"2024-03-06T07:32:43.637768+0000","last_clean":"2024-03-06T07:32:43.637768+0000","last_became_active":"2024-03-06T07:32:42.609732+0000","last_became_peered":"2024-03-06T07:32:42.609732+0000","last_unstale":"2024-03-06T07:32:43.637768+0000","last_undegraded":"2024-03-06T07:32:43.637768+0000","last_fullsized":"2024-03-06T07:32:43.637768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:19.673368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.633424+0000","last_change":"2024-03-06T07:32:43.629803+0000","last_active":"2024-03-06T07:32:43.633424+0000","last_peered":"2024-03-06T07:32:43.633424+0000","last_clean":"2024-03-06T07:32:43.633424+0000","last_became_active":"2024-03-06T07:32:43.626973+0000","last_became_peered":"2024-03-06T07:32:43.626973+0000","last_unstale":"2024-03-06T07:32:43.633424+0000","last_undegraded":"2024-03-06T07:32:43.633424+0000","last_fullsized":"2024-03-06T07:32:43.633424+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:40.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.638157+0000","last_change":"2024-03-06T07:32:42.616171+0000","last_active":"2024-03-06T07:32:43.638157+0000","last_peered":"2024-03-06T07:32:43.638157+0000","last_clean":"2024-03-06T07:32:43.638157+0000","last_became_active":"2024-03-06T07:32:42.615864+0000","last_became_peered":"2024-03-06T07:32:42.615864+0000","last_unstale":"2024-03-06T07:32:43.638157+0000","last_undegraded":"2024-03-06T07:32:43.638157+0000","last_fullsized":"2024-03-06T07:32:43.638157+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:13.476238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656650+0000","last_change":"2024-03-06T07:32:43.653516+0000","last_active":"2024-03-06T07:32:43.656650+0000","last_peered":"2024-03-06T07:32:43.656650+0000","last_clean":"2024-03-06T07:32:43.656650+0000","last_became_active":"2024-03-06T07:32:43.653237+0000","last_became_peered":"2024-03-06T07:32:43.653237+0000","last_unstale":"2024-03-06T07:32:43.656650+0000","last_undegraded":"2024-03-06T07:32:43.656650+0000","last_fullsized":"2024-03-06T07:32:43.656650+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:55.693582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.639215+0000","last_change":"2024-03-06T07:32:42.614045+0000","last_active":"2024-03-06T07:32:43.639215+0000","last_peered":"2024-03-06T07:32:43.639215+0000","last_clean":"2024-03-06T07:32:43.639215+0000","last_became_active":"2024-03-06T07:32:42.613708+0000","last_became_peered":"2024-03-06T07:32:42.613708+0000","last_unstale":"2024-03-06T07:32:43.639215+0000","last_undegraded":"2024-03-06T07:32:43.639215+0000","last_fullsized":"2024-03-06T07:32:43.639215+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:05.586268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.656947+0000","last_change":"2024-03-06T07:32:43.647006+0000","last_active":"2024-03-06T07:32:43.656947+0000","last_peered":"2024-03-06T07:32:43.656947+0000","last_clean":"2024-03-06T07:32:43.656947+0000","last_became_active":"2024-03-06T07:32:43.646746+0000","last_became_peered":"2024-03-06T07:32:43.646746+0000","last_unstale":"2024-03-06T07:32:43.656947+0000","last_undegraded":"2024-03-06T07:32:43.656947+0000","last_fullsized":"2024-03-06T07:32:43.656947+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:40.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"95'3","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.665244+0000","last_change":"2024-03-06T07:32:42.623313+0000","last_active":"2024-03-06T07:32:43.665244+0000","last_peered":"2024-03-06T07:32:43.665244+0000","last_clean":"2024-03-06T07:32:43.665244+0000","last_became_active":"2024-03-06T07:32:42.622510+0000","last_became_peered":"2024-03-06T07:32:42.622510+0000","last_unstale":"2024-03-06T07:32:43.665244+0000","last_undegraded":"2024-03-06T07:32:43.665244+0000","last_fullsized":"2024-03-06T07:32:43.665244+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:13.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.652438+0000","last_change":"2024-03-06T07:32:43.646709+0000","last_active":"2024-03-06T07:32:43.652438+0000","last_peered":"2024-03-06T07:32:43.652438+0000","last_clean":"2024-03-06T07:32:43.652438+0000","last_became_active":"2024-03-06T07:32:43.646402+0000","last_became_peered":"2024-03-06T07:32:43.646402+0000","last_unstale":"2024-03-06T07:32:43.652438+0000","last_undegraded":"2024-03-06T07:32:43.652438+0000","last_fullsized":"2024-03-06T07:32:43.652438+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:51.868663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"95'6","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666971+0000","last_change":"2024-03-06T07:32:42.616997+0000","last_active":"2024-03-06T07:32:43.666971+0000","last_peered":"2024-03-06T07:32:43.666971+0000","last_clean":"2024-03-06T07:32:43.666971+0000","last_became_active":"2024-03-06T07:32:42.616648+0000","last_became_peered":"2024-03-06T07:32:42.616648+0000","last_unstale":"2024-03-06T07:32:43.666971+0000","last_undegraded":"2024-03-06T07:32:43.666971+0000","last_fullsized":"2024-03-06T07:32:43.666971+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:48.033982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.644551+0000","last_change":"2024-03-06T07:32:43.642877+0000","last_active":"2024-03-06T07:32:43.644551+0000","last_peered":"2024-03-06T07:32:43.644551+0000","last_clean":"2024-03-06T07:32:43.644551+0000","last_became_active":"2024-03-06T07:32:43.641469+0000","last_became_peered":"2024-03-06T07:32:43.641469+0000","last_unstale":"2024-03-06T07:32:43.644551+0000","last_undegraded":"2024-03-06T07:32:43.644551+0000","last_fullsized":"2024-03-06T07:32:43.644551+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.667008+0000","last_change":"2024-03-06T07:32:42.619233+0000","last_active":"2024-03-06T07:32:43.667008+0000","last_peered":"2024-03-06T07:32:43.667008+0000","last_clean":"2024-03-06T07:32:43.667008+0000","last_became_active":"2024-03-06T07:32:42.618853+0000","last_became_peered":"2024-03-06T07:32:42.618853+0000","last_unstale":"2024-03-06T07:32:43.667008+0000","last_undegraded":"2024-03-06T07:32:43.667008+0000","last_fullsized":"2024-03-06T07:32:43.667008+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:04.406894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.654690+0000","last_change":"2024-03-06T07:32:43.646688+0000","last_active":"2024-03-06T07:32:43.654690+0000","last_peered":"2024-03-06T07:32:43.654690+0000","last_clean":"2024-03-06T07:32:43.654690+0000","last_became_active":"2024-03-06T07:32:43.643233+0000","last_became_peered":"2024-03-06T07:32:43.643233+0000","last_unstale":"2024-03-06T07:32:43.654690+0000","last_undegraded":"2024-03-06T07:32:43.654690+0000","last_fullsized":"2024-03-06T07:32:43.654690+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:00.646184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.623871+0000","last_change":"2024-03-06T07:32:42.608105+0000","last_active":"2024-03-06T07:32:43.623871+0000","last_peered":"2024-03-06T07:32:43.623871+0000","last_clean":"2024-03-06T07:32:43.623871+0000","last_became_active":"2024-03-06T07:32:42.607781+0000","last_became_peered":"2024-03-06T07:32:42.607781+0000","last_unstale":"2024-03-06T07:32:43.623871+0000","last_undegraded":"2024-03-06T07:32:43.623871+0000","last_fullsized":"2024-03-06T07:32:43.623871+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:23.841099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651863+0000","last_change":"2024-03-06T07:32:43.647084+0000","last_active":"2024-03-06T07:32:43.651863+0000","last_peered":"2024-03-06T07:32:43.651863+0000","last_clean":"2024-03-06T07:32:43.651863+0000","last_became_active":"2024-03-06T07:32:43.646775+0000","last_became_peered":"2024-03-06T07:32:43.646775+0000","last_unstale":"2024-03-06T07:32:43.651863+0000","last_undegraded":"2024-03-06T07:32:43.651863+0000","last_fullsized":"2024-03-06T07:32:43.651863+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:39.904147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.618268+0000","last_change":"2024-03-06T07:32:42.621461+0000","last_active":"2024-03-06T07:32:43.618268+0000","last_peered":"2024-03-06T07:32:43.618268+0000","last_clean":"2024-03-06T07:32:43.618268+0000","last_became_active":"2024-03-06T07:32:42.621152+0000","last_became_peered":"2024-03-06T07:32:42.621152+0000","last_unstale":"2024-03-06T07:32:43.618268+0000","last_undegraded":"2024-03-06T07:32:43.618268+0000","last_fullsized":"2024-03-06T07:32:43.618268+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:21.120030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641347+0000","last_change":"2024-03-06T07:32:43.632175+0000","last_active":"2024-03-06T07:32:43.641347+0000","last_peered":"2024-03-06T07:32:43.641347+0000","last_clean":"2024-03-06T07:32:43.641347+0000","last_became_active":"2024-03-06T07:32:43.631712+0000","last_became_peered":"2024-03-06T07:32:43.631712+0000","last_unstale":"2024-03-06T07:32:43.641347+0000","last_undegraded":"2024-03-06T07:32:43.641347+0000","last_fullsized":"2024-03-06T07:32:43.641347+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:54.193380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"95'8","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664380+0000","last_change":"2024-03-06T07:32:42.618791+0000","last_active":"2024-03-06T07:32:43.664380+0000","last_peered":"2024-03-06T07:32:43.664380+0000","last_clean":"2024-03-06T07:32:43.664380+0000","last_became_active":"2024-03-06T07:32:42.618268+0000","last_became_peered":"2024-03-06T07:32:42.618268+0000","last_unstale":"2024-03-06T07:32:43.664380+0000","last_undegraded":"2024-03-06T07:32:43.664380+0000","last_fullsized":"2024-03-06T07:32:43.664380+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:24.327331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662606+0000","last_change":"2024-03-06T07:32:43.649862+0000","last_active":"2024-03-06T07:32:43.662606+0000","last_peered":"2024-03-06T07:32:43.662606+0000","last_clean":"2024-03-06T07:32:43.662606+0000","last_became_active":"2024-03-06T07:32:43.646409+0000","last_became_peered":"2024-03-06T07:32:43.646409+0000","last_unstale":"2024-03-06T07:32:43.662606+0000","last_undegraded":"2024-03-06T07:32:43.662606+0000","last_fullsized":"2024-03-06T07:32:43.662606+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:00.597403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664212+0000","last_change":"2024-03-06T07:32:42.612709+0000","last_active":"2024-03-06T07:32:43.664212+0000","last_peered":"2024-03-06T07:32:43.664212+0000","last_clean":"2024-03-06T07:32:43.664212+0000","last_became_active":"2024-03-06T07:32:42.612310+0000","last_became_peered":"2024-03-06T07:32:42.612310+0000","last_unstale":"2024-03-06T07:32:43.664212+0000","last_undegraded":"2024-03-06T07:32:43.664212+0000","last_fullsized":"2024-03-06T07:32:43.664212+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:18.874144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.653367+0000","last_change":"2024-03-06T07:32:43.648936+0000","last_active":"2024-03-06T07:32:43.653367+0000","last_peered":"2024-03-06T07:32:43.653367+0000","last_clean":"2024-03-06T07:32:43.653367+0000","last_became_active":"2024-03-06T07:32:43.646400+0000","last_became_peered":"2024-03-06T07:32:43.646400+0000","last_unstale":"2024-03-06T07:32:43.653367+0000","last_undegraded":"2024-03-06T07:32:43.653367+0000","last_fullsized":"2024-03-06T07:32:43.653367+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:46.261548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664801+0000","last_change":"2024-03-06T07:32:42.619986+0000","last_active":"2024-03-06T07:32:43.664801+0000","last_peered":"2024-03-06T07:32:43.664801+0000","last_clean":"2024-03-06T07:32:43.664801+0000","last_became_active":"2024-03-06T07:32:42.619345+0000","last_became_peered":"2024-03-06T07:32:42.619345+0000","last_unstale":"2024-03-06T07:32:43.664801+0000","last_undegraded":"2024-03-06T07:32:43.664801+0000","last_fullsized":"2024-03-06T07:32:43.664801+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:28.447215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.652415+0000","last_change":"2024-03-06T07:32:43.646022+0000","last_active":"2024-03-06T07:32:43.652415+0000","last_peered":"2024-03-06T07:32:43.652415+0000","last_clean":"2024-03-06T07:32:43.652415+0000","last_became_active":"2024-03-06T07:32:43.645391+0000","last_became_peered":"2024-03-06T07:32:43.645391+0000","last_unstale":"2024-03-06T07:32:43.652415+0000","last_undegraded":"2024-03-06T07:32:43.652415+0000","last_fullsized":"2024-03-06T07:32:43.652415+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:06.598219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.615364+0000","last_change":"2024-03-06T07:32:42.612678+0000","last_active":"2024-03-06T07:32:43.615364+0000","last_peered":"2024-03-06T07:32:43.615364+0000","last_clean":"2024-03-06T07:32:43.615364+0000","last_became_active":"2024-03-06T07:32:42.612254+0000","last_became_peered":"2024-03-06T07:32:42.612254+0000","last_unstale":"2024-03-06T07:32:43.615364+0000","last_undegraded":"2024-03-06T07:32:43.615364+0000","last_fullsized":"2024-03-06T07:32:43.615364+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:22.090112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649704+0000","last_change":"2024-03-06T07:32:43.646511+0000","last_active":"2024-03-06T07:32:43.649704+0000","last_peered":"2024-03-06T07:32:43.649704+0000","last_clean":"2024-03-06T07:32:43.649704+0000","last_became_active":"2024-03-06T07:32:43.646219+0000","last_became_peered":"2024-03-06T07:32:43.646219+0000","last_unstale":"2024-03-06T07:32:43.649704+0000","last_undegraded":"2024-03-06T07:32:43.649704+0000","last_fullsized":"2024-03-06T07:32:43.649704+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:25.565785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.666397+0000","last_change":"2024-03-06T07:32:42.613220+0000","last_active":"2024-03-06T07:32:43.666397+0000","last_peered":"2024-03-06T07:32:43.666397+0000","last_clean":"2024-03-06T07:32:43.666397+0000","last_became_active":"2024-03-06T07:32:42.610086+0000","last_became_peered":"2024-03-06T07:32:42.610086+0000","last_unstale":"2024-03-06T07:32:43.666397+0000","last_undegraded":"2024-03-06T07:32:43.666397+0000","last_fullsized":"2024-03-06T07:32:43.666397+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:52.768653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641640+0000","last_change":"2024-03-06T07:32:43.632463+0000","last_active":"2024-03-06T07:32:43.641640+0000","last_peered":"2024-03-06T07:32:43.641640+0000","last_clean":"2024-03-06T07:32:43.641640+0000","last_became_active":"2024-03-06T07:32:43.632163+0000","last_became_peered":"2024-03-06T07:32:43.632163+0000","last_unstale":"2024-03-06T07:32:43.641640+0000","last_undegraded":"2024-03-06T07:32:43.641640+0000","last_fullsized":"2024-03-06T07:32:43.641640+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:39.221449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643464+0000","last_change":"2024-03-06T07:32:42.615804+0000","last_active":"2024-03-06T07:32:43.643464+0000","last_peered":"2024-03-06T07:32:43.643464+0000","last_clean":"2024-03-06T07:32:43.643464+0000","last_became_active":"2024-03-06T07:32:42.615444+0000","last_became_peered":"2024-03-06T07:32:42.615444+0000","last_unstale":"2024-03-06T07:32:43.643464+0000","last_undegraded":"2024-03-06T07:32:43.643464+0000","last_fullsized":"2024-03-06T07:32:43.643464+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:38.521138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.645780+0000","last_change":"2024-03-06T07:32:43.642934+0000","last_active":"2024-03-06T07:32:43.645780+0000","last_peered":"2024-03-06T07:32:43.645780+0000","last_clean":"2024-03-06T07:32:43.645780+0000","last_became_active":"2024-03-06T07:32:43.642579+0000","last_became_peered":"2024-03-06T07:32:43.642579+0000","last_unstale":"2024-03-06T07:32:43.645780+0000","last_undegraded":"2024-03-06T07:32:43.645780+0000","last_fullsized":"2024-03-06T07:32:43.645780+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:28.655010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.643143+0000","last_change":"2024-03-06T07:32:42.615569+0000","last_active":"2024-03-06T07:32:43.643143+0000","last_peered":"2024-03-06T07:32:43.643143+0000","last_clean":"2024-03-06T07:32:43.643143+0000","last_became_active":"2024-03-06T07:32:42.615224+0000","last_became_peered":"2024-03-06T07:32:42.615224+0000","last_unstale":"2024-03-06T07:32:43.643143+0000","last_undegraded":"2024-03-06T07:32:43.643143+0000","last_fullsized":"2024-03-06T07:32:43.643143+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:33.634241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.649382+0000","last_change":"2024-03-06T07:32:43.640138+0000","last_active":"2024-03-06T07:32:43.649382+0000","last_peered":"2024-03-06T07:32:43.649382+0000","last_clean":"2024-03-06T07:32:43.649382+0000","last_became_active":"2024-03-06T07:32:43.638686+0000","last_became_peered":"2024-03-06T07:32:43.638686+0000","last_unstale":"2024-03-06T07:32:43.649382+0000","last_undegraded":"2024-03-06T07:32:43.649382+0000","last_fullsized":"2024-03-06T07:32:43.649382+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:22.125079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.615731+0000","last_change":"2024-03-06T07:32:42.605795+0000","last_active":"2024-03-06T07:32:43.615731+0000","last_peered":"2024-03-06T07:32:43.615731+0000","last_clean":"2024-03-06T07:32:43.615731+0000","last_became_active":"2024-03-06T07:32:42.605452+0000","last_became_peered":"2024-03-06T07:32:42.605452+0000","last_unstale":"2024-03-06T07:32:43.615731+0000","last_undegraded":"2024-03-06T07:32:43.615731+0000","last_fullsized":"2024-03-06T07:32:43.615731+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:28.447215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.658585+0000","last_change":"2024-03-06T07:32:43.654867+0000","last_active":"2024-03-06T07:32:43.658585+0000","last_peered":"2024-03-06T07:32:43.658585+0000","last_clean":"2024-03-06T07:32:43.658585+0000","last_became_active":"2024-03-06T07:32:43.654560+0000","last_became_peered":"2024-03-06T07:32:43.654560+0000","last_unstale":"2024-03-06T07:32:43.658585+0000","last_undegraded":"2024-03-06T07:32:43.658585+0000","last_fullsized":"2024-03-06T07:32:43.658585+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.447329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664980+0000","last_change":"2024-03-06T07:32:42.620054+0000","last_active":"2024-03-06T07:32:43.664980+0000","last_peered":"2024-03-06T07:32:43.664980+0000","last_clean":"2024-03-06T07:32:43.664980+0000","last_became_active":"2024-03-06T07:32:42.618944+0000","last_became_peered":"2024-03-06T07:32:42.618944+0000","last_unstale":"2024-03-06T07:32:43.664980+0000","last_undegraded":"2024-03-06T07:32:43.664980+0000","last_fullsized":"2024-03-06T07:32:43.664980+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:19.440879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662121+0000","last_change":"2024-03-06T07:32:43.647727+0000","last_active":"2024-03-06T07:32:43.662121+0000","last_peered":"2024-03-06T07:32:43.662121+0000","last_clean":"2024-03-06T07:32:43.662121+0000","last_became_active":"2024-03-06T07:32:43.647447+0000","last_became_peered":"2024-03-06T07:32:43.647447+0000","last_unstale":"2024-03-06T07:32:43.662121+0000","last_undegraded":"2024-03-06T07:32:43.662121+0000","last_fullsized":"2024-03-06T07:32:43.662121+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:34.176922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.617018+0000","last_change":"2024-03-06T07:32:42.613251+0000","last_active":"2024-03-06T07:32:43.617018+0000","last_peered":"2024-03-06T07:32:43.617018+0000","last_clean":"2024-03-06T07:32:43.617018+0000","last_became_active":"2024-03-06T07:32:42.610157+0000","last_became_peered":"2024-03-06T07:32:42.610157+0000","last_unstale":"2024-03-06T07:32:43.617018+0000","last_undegraded":"2024-03-06T07:32:43.617018+0000","last_fullsized":"2024-03-06T07:32:43.617018+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:13.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.662108+0000","last_change":"2024-03-06T07:32:43.650011+0000","last_active":"2024-03-06T07:32:43.662108+0000","last_peered":"2024-03-06T07:32:43.662108+0000","last_clean":"2024-03-06T07:32:43.662108+0000","last_became_active":"2024-03-06T07:32:43.649662+0000","last_became_peered":"2024-03-06T07:32:43.649662+0000","last_unstale":"2024-03-06T07:32:43.662108+0000","last_undegraded":"2024-03-06T07:32:43.662108+0000","last_fullsized":"2024-03-06T07:32:43.662108+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:34.620080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.616195+0000","last_change":"2024-03-06T07:32:42.616985+0000","last_active":"2024-03-06T07:32:43.616195+0000","last_peered":"2024-03-06T07:32:43.616195+0000","last_clean":"2024-03-06T07:32:43.616195+0000","last_became_active":"2024-03-06T07:32:42.616602+0000","last_became_peered":"2024-03-06T07:32:42.616602+0000","last_unstale":"2024-03-06T07:32:43.616195+0000","last_undegraded":"2024-03-06T07:32:43.616195+0000","last_fullsized":"2024-03-06T07:32:43.616195+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:00.197386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.661505+0000","last_change":"2024-03-06T07:32:43.647486+0000","last_active":"2024-03-06T07:32:43.661505+0000","last_peered":"2024-03-06T07:32:43.661505+0000","last_clean":"2024-03-06T07:32:43.661505+0000","last_became_active":"2024-03-06T07:32:43.647219+0000","last_became_peered":"2024-03-06T07:32:43.647219+0000","last_unstale":"2024-03-06T07:32:43.661505+0000","last_undegraded":"2024-03-06T07:32:43.661505+0000","last_fullsized":"2024-03-06T07:32:43.661505+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.998526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.634461+0000","last_change":"2024-03-06T07:32:42.621703+0000","last_active":"2024-03-06T07:32:43.634461+0000","last_peered":"2024-03-06T07:32:43.634461+0000","last_clean":"2024-03-06T07:32:43.634461+0000","last_became_active":"2024-03-06T07:32:42.621400+0000","last_became_peered":"2024-03-06T07:32:42.621400+0000","last_unstale":"2024-03-06T07:32:43.634461+0000","last_undegraded":"2024-03-06T07:32:43.634461+0000","last_fullsized":"2024-03-06T07:32:43.634461+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:00.197386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641716+0000","last_change":"2024-03-06T07:32:43.633478+0000","last_active":"2024-03-06T07:32:43.641716+0000","last_peered":"2024-03-06T07:32:43.641716+0000","last_clean":"2024-03-06T07:32:43.641716+0000","last_became_active":"2024-03-06T07:32:43.631199+0000","last_became_peered":"2024-03-06T07:32:43.631199+0000","last_unstale":"2024-03-06T07:32:43.641716+0000","last_undegraded":"2024-03-06T07:32:43.641716+0000","last_fullsized":"2024-03-06T07:32:43.641716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:19.263218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.616620+0000","last_change":"2024-03-06T07:32:42.606554+0000","last_active":"2024-03-06T07:32:43.616620+0000","last_peered":"2024-03-06T07:32:43.616620+0000","last_clean":"2024-03-06T07:32:43.616620+0000","last_became_active":"2024-03-06T07:32:42.606302+0000","last_became_peered":"2024-03-06T07:32:42.606302+0000","last_unstale":"2024-03-06T07:32:43.616620+0000","last_undegraded":"2024-03-06T07:32:43.616620+0000","last_fullsized":"2024-03-06T07:32:43.616620+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:00.357359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.651040+0000","last_change":"2024-03-06T07:32:43.639952+0000","last_active":"2024-03-06T07:32:43.651040+0000","last_peered":"2024-03-06T07:32:43.651040+0000","last_clean":"2024-03-06T07:32:43.651040+0000","last_became_active":"2024-03-06T07:32:43.639497+0000","last_became_peered":"2024-03-06T07:32:43.639497+0000","last_unstale":"2024-03-06T07:32:43.651040+0000","last_undegraded":"2024-03-06T07:32:43.651040+0000","last_fullsized":"2024-03-06T07:32:43.651040+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:22.957826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.628775+0000","last_change":"2024-03-06T07:32:42.618333+0000","last_active":"2024-03-06T07:32:43.628775+0000","last_peered":"2024-03-06T07:32:43.628775+0000","last_clean":"2024-03-06T07:32:43.628775+0000","last_became_active":"2024-03-06T07:32:42.617948+0000","last_became_peered":"2024-03-06T07:32:42.617948+0000","last_unstale":"2024-03-06T07:32:43.628775+0000","last_undegraded":"2024-03-06T07:32:43.628775+0000","last_fullsized":"2024-03-06T07:32:43.628775+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:22.090112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.641023+0000","last_change":"2024-03-06T07:32:43.627715+0000","last_active":"2024-03-06T07:32:43.641023+0000","last_peered":"2024-03-06T07:32:43.641023+0000","last_clean":"2024-03-06T07:32:43.641023+0000","last_became_active":"2024-03-06T07:32:43.626959+0000","last_became_peered":"2024-03-06T07:32:43.626959+0000","last_unstale":"2024-03-06T07:32:43.641023+0000","last_undegraded":"2024-03-06T07:32:43.641023+0000","last_fullsized":"2024-03-06T07:32:43.641023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:34.270634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.640574+0000","last_change":"2024-03-06T07:32:42.604067+0000","last_active":"2024-03-06T07:32:43.640574+0000","last_peered":"2024-03-06T07:32:43.640574+0000","last_clean":"2024-03-06T07:32:43.640574+0000","last_became_active":"2024-03-06T07:32:42.603793+0000","last_became_peered":"2024-03-06T07:32:42.603793+0000","last_unstale":"2024-03-06T07:32:43.640574+0000","last_undegraded":"2024-03-06T07:32:43.640574+0000","last_fullsized":"2024-03-06T07:32:43.640574+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:50.856711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.661683+0000","last_change":"2024-03-06T07:32:43.649207+0000","last_active":"2024-03-06T07:32:43.661683+0000","last_peered":"2024-03-06T07:32:43.661683+0000","last_clean":"2024-03-06T07:32:43.661683+0000","last_became_active":"2024-03-06T07:32:43.647697+0000","last_became_peered":"2024-03-06T07:32:43.647697+0000","last_unstale":"2024-03-06T07:32:43.661683+0000","last_undegraded":"2024-03-06T07:32:43.661683+0000","last_fullsized":"2024-03-06T07:32:43.661683+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:51.868663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.619497+0000","last_change":"2024-03-06T07:32:42.601450+0000","last_active":"2024-03-06T07:32:43.619497+0000","last_peered":"2024-03-06T07:32:43.619497+0000","last_clean":"2024-03-06T07:32:43.619497+0000","last_became_active":"2024-03-06T07:32:42.601159+0000","last_became_peered":"2024-03-06T07:32:42.601159+0000","last_unstale":"2024-03-06T07:32:43.619497+0000","last_undegraded":"2024-03-06T07:32:43.619497+0000","last_fullsized":"2024-03-06T07:32:43.619497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:41.668237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.659494+0000","last_change":"2024-03-06T07:32:43.648279+0000","last_active":"2024-03-06T07:32:43.659494+0000","last_peered":"2024-03-06T07:32:43.659494+0000","last_clean":"2024-03-06T07:32:43.659494+0000","last_became_active":"2024-03-06T07:32:43.647873+0000","last_became_peered":"2024-03-06T07:32:43.647873+0000","last_unstale":"2024-03-06T07:32:43.659494+0000","last_undegraded":"2024-03-06T07:32:43.659494+0000","last_fullsized":"2024-03-06T07:32:43.659494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:42.578025+0000","last_clean_scrub_stamp":"2024-03-06T07:32:42.578025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:48.243786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.664096+0000","last_change":"2024-03-06T07:32:42.621747+0000","last_active":"2024-03-06T07:32:43.664096+0000","last_peered":"2024-03-06T07:32:43.664096+0000","last_clean":"2024-03-06T07:32:43.664096+0000","last_became_active":"2024-03-06T07:32:42.621401+0000","last_became_peered":"2024-03-06T07:32:42.621401+0000","last_unstale":"2024-03-06T07:32:43.664096+0000","last_undegraded":"2024-03-06T07:32:43.664096+0000","last_fullsized":"2024-03-06T07:32:43.664096+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:41.566073+0000","last_clean_scrub_stamp":"2024-03-06T07:32:41.566073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.113127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":307,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:32:43.635718+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:32:43.635718+0000","last_peered":"2024-03-06T07:32:43.635718+0000","last_clean":"2024-03-06T07:32:43.635718+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:32:43.635718+0000","last_undegraded":"2024-03-06T07:32:43.635718+0000","last_fullsized":"2024-03-06T07:32:43.635718+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542208,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1753088,"data_stored":1083145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":500204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1175552,"data_stored":499622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":500224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1159168,"data_stored":492777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542210,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":492777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":499552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542209,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1753088,"data_stored":1083145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:48.722 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:32:48.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:32:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.968+0000 7f5415400640 1 Processor -- start 2024-03-06T07:32:48.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.969+0000 7f5415400640 1 -- start start 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.970+0000 7f5415400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f541015af00 0x7f541015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.970+0000 7f5415400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541006dd10 0x7f54100b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.970+0000 7f5415400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541006e190 con 0x7f541006d890 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.970+0000 7f5415400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541006e470 con 0x7f541015af00 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.970+0000 7f5415400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541006e750 con 0x7f541006dd10 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.970+0000 7f540e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f541015af00 0x7f541015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f541015af00 0x7f541015b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34050/0 (socket says 172.21.15.97:34050) 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540e800640 1 -- 172.21.15.97:0/2693686784 learned_addr learned my addr 172.21.15.97:0/2693686784 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540e800640 1 -- 172.21.15.97:0/2693686784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541006dd10 msgr2=0x7f54100b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540f200640 1 --2- 172.21.15.97:0/2693686784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541006dd10 0x7f54100b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540e800640 1 --2- 172.21.15.97:0/2693686784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541006dd10 0x7f54100b4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540e800640 1 -- 172.21.15.97:0/2693686784 >> v1:172.21.15.135:6789/0 conn(0x7f541006d890 legacy=0x7f5410071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.971+0000 7f540e800640 1 -- 172.21.15.97:0/2693686784 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541006ea30 con 0x7f541015af00 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.972+0000 7f540e800640 1 --2- 172.21.15.97:0/2693686784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541015af00 0x7f541015b2e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f5400009870 tx=0x7f540002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730652b4fc28458d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.972+0000 7f540de00640 1 -- 172.21.15.97:0/2693686784 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400035020 con 0x7f541015af00 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f540de00640 1 -- 172.21.15.97:0/2693686784 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5400031640 con 0x7f541015af00 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f540de00640 1 -- 172.21.15.97:0/2693686784 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540002d040 con 0x7f541015af00 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 -- 172.21.15.97:0/2693686784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541015af00 msgr2=0x7f541015b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 --2- 172.21.15.97:0/2693686784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541015af00 0x7f541015b2e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f5400009870 tx=0x7f540002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 -- 172.21.15.97:0/2693686784 shutdown_connections 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 --2- 172.21.15.97:0/2693686784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541006dd10 0x7f54100b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 --2- 172.21.15.97:0/2693686784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541015af00 0x7f541015b2e0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 -- 172.21.15.97:0/2693686784 >> 172.21.15.97:0/2693686784 conn(0x7f54100b2840 msgr2=0x7f54100b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.973+0000 7f5415400640 1 -- 172.21.15.97:0/2693686784 shutdown_connections 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.974+0000 7f5415400640 1 -- 172.21.15.97:0/2693686784 wait complete. 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.975+0000 7f5415400640 1 Processor -- start 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.975+0000 7f5415400640 1 -- start start 2024-03-06T07:32:48.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f5415400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 0x7f5410157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f5415400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f541006dd10 0x7f54101f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f5415400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541015af00 0x7f5410149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f5415400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54100723a0 con 0x7f541006d890 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f5415400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5410072220 con 0x7f541006dd10 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f5415400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54100b45a0 con 0x7f541015af00 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 0x7f5410157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 0x7f5410157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34062/0 (socket says 172.21.15.97:34062) 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 -- 172.21.15.97:0/4257908266 learned_addr learned my addr 172.21.15.97:0/4257908266 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 -- 172.21.15.97:0/4257908266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541015af00 msgr2=0x7f5410149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541015af00 0x7f5410149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 -- 172.21.15.97:0/4257908266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f541006dd10 msgr2=0x7f54101f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f541006dd10 0x7f54101f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.977+0000 7f540e800640 1 -- 172.21.15.97:0/4257908266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54101f4ca0 con 0x7f541006d890 2024-03-06T07:32:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.978+0000 7f540e800640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 0x7f5410157330 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f540002b930 tx=0x7f54000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.978+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400040070 con 0x7f541006d890 2024-03-06T07:32:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.978+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5400039c10 con 0x7f541006d890 2024-03-06T07:32:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.978+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400035040 con 0x7f541006d890 2024-03-06T07:32:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.978+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54101f5240 con 0x7f541006d890 2024-03-06T07:32:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.978+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54101f5800 con 0x7f541006d890 2024-03-06T07:32:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.980+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541004f330 con 0x7f541006d890 2024-03-06T07:32:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.980+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f540004b060 con 0x7f541006d890 2024-03-06T07:32:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.981+0000 7f5406a00640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53ec06b500 0x7f53ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.981+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54000abc50 con 0x7f541006d890 2024-03-06T07:32:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.982+0000 7f5407e00640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53ec06b500 0x7f53ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.982+0000 7f5407e00640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53ec06b500 0x7f53ec06d9c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f541015a8d0 tx=0x7f53f80079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:48.986+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54000b1020 con 0x7f541006d890 2024-03-06T07:32:49.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.255+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f541006bb70 con 0x7f541006d890 2024-03-06T07:32:49.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.260+0000 7f5406a00640 1 -- 172.21.15.97:0/4257908266 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f54101f4ca0 con 0x7f541006d890 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.262+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53ec06b500 msgr2=0x7f53ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.262+0000 7f5415400640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53ec06b500 0x7f53ec06d9c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f541015a8d0 tx=0x7f53f80079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.262+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 msgr2=0x7f5410157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.262+0000 7f5415400640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 0x7f5410157330 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f540002b930 tx=0x7f54000046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 shutdown_connections 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53ec06b500 0x7f53ec06d9c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f541015af00 0x7f5410149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f541006dd10 0x7f54101f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 --2- 172.21.15.97:0/4257908266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f541006d890 0x7f5410157330 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 >> 172.21.15.97:0/4257908266 conn(0x7f54100b2840 msgr2=0x7f54100af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 shutdown_connections 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.263+0000 7f5415400640 1 -- 172.21.15.97:0/4257908266 wait complete. 2024-03-06T07:32:49.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:49.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.483+0000 7f937b000640 1 Processor -- start 2024-03-06T07:32:49.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.483+0000 7f937b000640 1 -- start start 2024-03-06T07:32:49.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.484+0000 7f937b000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f93740d2360 0x7f93740d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.484+0000 7f937b000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374161960 con 0x7f9374161580 2024-03-06T07:32:49.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.484+0000 7f937b000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93740cf6f0 con 0x7f93740b3ec0 2024-03-06T07:32:49.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.484+0000 7f937b000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93740d2ce0 con 0x7f93740d2360 2024-03-06T07:32:49.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9379c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9374161580 0x7f93740d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43772/0 (socket says 172.21.15.97:43772) 2024-03-06T07:32:49.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9379c00640 1 -- 172.21.15.97:0/2268605705 learned_addr learned my addr 172.21.15.97:0/2268605705 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505398178 0 0) 0x7f9374161960 con 0x7f9374161580 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9358003520 con 0x7f9374161580 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9379200640 1 --2- 172.21.15.97:0/2268605705 >> v2:172.21.15.135:3300/0 conn(0x7f93740d2360 0x7f93740d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 74708527 0 0) 0x7f9358003520 con 0x7f9374161580 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 >> v2:172.21.15.135:3300/0 conn(0x7f93740d2360 msgr2=0x7f93740d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 --2- 172.21.15.97:0/2268605705 >> v2:172.21.15.135:3300/0 conn(0x7f93740d2360 0x7f93740d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 >> v1:172.21.15.135:6789/0 conn(0x7f93740b3ec0 legacy=0x7f93740b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.485+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740cf9d0 con 0x7f9374161580 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9364001670 con 0x7f9374161580 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 >> v1:172.21.15.97:6789/0 conn(0x7f9374161580 legacy=0x7f93740d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9358003a00 0x7f9358003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9358004800 0x7f9358004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93580052e0 0x7f935800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740d2ce0 con 0x7f93580052e0 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935800c230 con 0x7f9358004800 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935800c430 con 0x7f9358003a00 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f9379200640 1 --2- 172.21.15.97:0/2268605705 >> v2:172.21.15.135:3300/0 conn(0x7f93740d2360 0x7f93740d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f937a600640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93580052e0 0x7f935800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f937a600640 1 -- 172.21.15.97:0/2268605705 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9358003a00 msgr2=0x7f9358003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f937a600640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9358003a00 0x7f9358003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f937a600640 1 -- 172.21.15.97:0/2268605705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9358004800 msgr2=0x7f9358004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f937a600640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9358004800 0x7f9358004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.486+0000 7f937a600640 1 -- 172.21.15.97:0/2268605705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358003520 con 0x7f93580052e0 2024-03-06T07:32:49.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.487+0000 7f937a600640 1 -- 172.21.15.97:0/2268605705 reap_dead start 2024-03-06T07:32:49.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.487+0000 7f937a600640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93580052e0 0x7f935800b9c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f936c004480 tx=0x7f9358005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348464bd6dd2d072 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c004080 con 0x7f93580052e0 2024-03-06T07:32:49.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c006b50 con 0x7f93580052e0 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f9373e00640 1 -- 172.21.15.97:0/2268605705 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c007020 con 0x7f93580052e0 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 -- 172.21.15.97:0/2268605705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93580052e0 msgr2=0x7f935800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93580052e0 0x7f935800b9c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f936c004480 tx=0x7f9358005830 comp rx=0 tx=0).stop 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 -- 172.21.15.97:0/2268605705 shutdown_connections 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 --2- 172.21.15.97:0/2268605705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93580052e0 0x7f935800b9c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 -- 172.21.15.97:0/2268605705 >> 172.21.15.97:0/2268605705 conn(0x7f93740d3d80 msgr2=0x7f93740d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 -- 172.21.15.97:0/2268605705 shutdown_connections 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.488+0000 7f937b000640 1 -- 172.21.15.97:0/2268605705 wait complete. 2024-03-06T07:32:49.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.490+0000 7f937b000640 1 Processor -- start 2024-03-06T07:32:49.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.490+0000 7f937b000640 1 -- start start 2024-03-06T07:32:49.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937b000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93580052e0 0x7f93741f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93740c9390 0x7f937414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 0x7f93740c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b39a0 con 0x7f93740c7410 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937b000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b36c0 con 0x7f93580052e0 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93740cf3a0 con 0x7f93740c9390 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 0x7f93740c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 0x7f93740c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34074/0 (socket says 172.21.15.97:34074) 2024-03-06T07:32:49.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.491+0000 7f937a600640 1 -- 172.21.15.97:0/2370740958 learned_addr learned my addr 172.21.15.97:0/2370740958 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f9379200640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93740c9390 0x7f937414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f9379c00640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93580052e0 0x7f93741f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f937a600640 1 -- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93740c9390 msgr2=0x7f937414d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f937a600640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93740c9390 0x7f937414d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f937a600640 1 -- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93580052e0 msgr2=0x7f93741f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f937a600640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93580052e0 0x7f93741f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f937a600640 1 -- 172.21.15.97:0/2370740958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740cac30 con 0x7f93740c7410 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.492+0000 7f937a600640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 0x7f93740c7890 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f936c004480 tx=0x7f936c003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.493+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c007df0 con 0x7f93740c7410 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.493+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c041070 con 0x7f93740c7410 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.493+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93741f80e0 con 0x7f93740c7410 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.493+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c004080 con 0x7f93740c7410 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.493+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93740c9e70 con 0x7f93740c7410 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.494+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937404f330 con 0x7f93740c7410 2024-03-06T07:32:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.496+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f936c038840 con 0x7f93740c7410 2024-03-06T07:32:49.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.497+0000 7f9371600640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f935406b4d0 0x7f935406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.497+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f936c0ab460 con 0x7f93740c7410 2024-03-06T07:32:49.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.497+0000 7f9379c00640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f935406b4d0 0x7f935406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.498+0000 7f9379c00640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f935406b4d0 0x7f935406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9364000fd0 tx=0x7f9364009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.502+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f936c034070 con 0x7f93740c7410 2024-03-06T07:32:49.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.813+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93740d3b60 con 0x7f93740c7410 2024-03-06T07:32:49.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.814+0000 7f9371600640 1 -- 172.21.15.97:0/2370740958 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f936c036e20 con 0x7f93740c7410 2024-03-06T07:32:49.817 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:49.817 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2473569439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2473569439},{"type":"v1","addr":"172.21.15.135:6833","nonce":2473569439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/807677354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":807677354},{"type":"v1","addr":"172.21.15.135:6835","nonce":807677354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/369774247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":369774247},{"type":"v1","addr":"172.21.15.97:6835","nonce":369774247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:46.128331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/1818345324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1818345324},{"type":"v1","addr":"172.21.15.97:6837","nonce":1818345324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":7}]} 2024-03-06T07:32:49.817 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 60 2024-03-06T07:32:49.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f935406b4d0 msgr2=0x7f935406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f935406b4d0 0x7f935406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9364000fd0 tx=0x7f9364009040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 msgr2=0x7f93740c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 0x7f93740c7890 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f936c004480 tx=0x7f936c003a70 comp rx=0 tx=0).stop 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 shutdown_connections 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f935406b4d0 0x7f935406d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93740c7410 0x7f93740c7890 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93740c9390 0x7f937414d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 --2- 172.21.15.97:0/2370740958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93580052e0 0x7f93741f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.817+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 >> 172.21.15.97:0/2370740958 conn(0x7f93740d3d80 msgr2=0x7f93740b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.818+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 shutdown_connections 2024-03-06T07:32:49.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:49.818+0000 7f937b000640 1 -- 172.21.15.97:0/2370740958 wait complete. 2024-03-06T07:32:50.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:43.068988+0000', 'modified': '2024-03-06T07:32:46.128331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7113}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7113': {'gid': 7113, 'name': 'c', 'rank': 0, 'incarnation': 56, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/1818345324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1818345324}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1818345324}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7113, 'qdb_cluster': [7113]} 2024-03-06T07:32:50.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:50.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:32:50.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.453+0000 7f2885000640 1 Processor -- start 2024-03-06T07:32:50.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.454+0000 7f2885000640 1 -- start start 2024-03-06T07:32:50.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f2885000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2880161580 0x7f28800d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f2885000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f2885000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f28800d28a0 0x7f28800d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f2885000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2880161960 con 0x7f28800d28a0 2024-03-06T07:32:50.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f2885000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28800cf6f0 con 0x7f2880161580 2024-03-06T07:32:50.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f2885000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800cf9d0 con 0x7f28800d1e90 2024-03-06T07:32:50.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f287fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f28800d28a0 0x7f28800d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f287fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f28800d28a0 0x7f28800d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34088/0 (socket says 172.21.15.97:34088) 2024-03-06T07:32:50.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.455+0000 7f287fe00640 1 -- 172.21.15.97:0/3830413859 learned_addr learned my addr 172.21.15.97:0/3830413859 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f287f400640 1 --2- 172.21.15.97:0/3830413859 >> v2:172.21.15.135:3300/0 conn(0x7f2880161580 0x7f28800d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f2877e00640 1 --2- 172.21.15.97:0/3830413859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f287fe00640 1 -- 172.21.15.97:0/3830413859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 msgr2=0x7f28800d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f287fe00640 1 --2- 172.21.15.97:0/3830413859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f287fe00640 1 -- 172.21.15.97:0/3830413859 >> v2:172.21.15.135:3300/0 conn(0x7f2880161580 msgr2=0x7f28800d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f287fe00640 1 --2- 172.21.15.97:0/3830413859 >> v2:172.21.15.135:3300/0 conn(0x7f2880161580 0x7f28800d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.456+0000 7f287fe00640 1 -- 172.21.15.97:0/3830413859 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800cfcb0 con 0x7f28800d28a0 2024-03-06T07:32:50.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.457+0000 7f287fe00640 1 --2- 172.21.15.97:0/3830413859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28800d06e0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2880073740 tx=0x7f287802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27511331da455ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f287ea00640 1 -- 172.21.15.97:0/3830413859 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287802ca70 con 0x7f28800d28a0 2024-03-06T07:32:50.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f287ea00640 1 -- 172.21.15.97:0/3830413859 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287802cc10 con 0x7f28800d28a0 2024-03-06T07:32:50.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f287ea00640 1 -- 172.21.15.97:0/3830413859 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780343f0 con 0x7f28800d28a0 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 -- 172.21.15.97:0/3830413859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 msgr2=0x7f28800d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 --2- 172.21.15.97:0/3830413859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28800d06e0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2880073740 tx=0x7f287802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 -- 172.21.15.97:0/3830413859 shutdown_connections 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 --2- 172.21.15.97:0/3830413859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28800d06e0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 --2- 172.21.15.97:0/3830413859 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 --2- 172.21.15.97:0/3830413859 >> v2:172.21.15.135:3300/0 conn(0x7f2880161580 0x7f28800d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 -- 172.21.15.97:0/3830413859 >> 172.21.15.97:0/3830413859 conn(0x7f28800d3d80 msgr2=0x7f28800d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 -- 172.21.15.97:0/3830413859 shutdown_connections 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.458+0000 7f2885000640 1 -- 172.21.15.97:0/3830413859 wait complete. 2024-03-06T07:32:50.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.460+0000 7f2885000640 1 Processor -- start 2024-03-06T07:32:50.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.460+0000 7f2885000640 1 -- start start 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2885000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2885000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28801520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2885000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2880161580 0x7f288015f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2885000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800cf3a0 con 0x7f28800d28a0 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2885000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800b39a0 con 0x7f2880161580 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2885000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800d0c20 con 0x7f28800d1e90 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2877e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28801520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2877e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28801520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34094/0 (socket says 172.21.15.97:34094) 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.461+0000 7f2877e00640 1 -- 172.21.15.97:0/2309287139 learned_addr learned my addr 172.21.15.97:0/2309287139 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f287f400640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877e00640 1 -- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 msgr2=0x7f28800c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877e00640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877e00640 1 -- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2880161580 msgr2=0x7f288015f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877e00640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2880161580 0x7f288015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877e00640 1 -- 172.21.15.97:0/2309287139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800ca480 con 0x7f28800d28a0 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877e00640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28801520b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f286800d830 tx=0x7f286800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877400640 1 -- 172.21.15.97:0/2309287139 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286800b6c0 con 0x7f28800d28a0 2024-03-06T07:32:50.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877400640 1 -- 172.21.15.97:0/2309287139 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2868019070 con 0x7f28800d28a0 2024-03-06T07:32:50.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.462+0000 7f2877400640 1 -- 172.21.15.97:0/2309287139 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286800ba80 con 0x7f28800d28a0 2024-03-06T07:32:50.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.463+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28800cc310 con 0x7f28800d28a0 2024-03-06T07:32:50.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.463+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28800cc8d0 con 0x7f28800d28a0 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.466+0000 7f2877400640 1 -- 172.21.15.97:0/2309287139 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f286801e460 con 0x7f28800d28a0 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.466+0000 7f2889273740 1 Processor -- start 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.466+0000 7f2889273740 1 -- start start 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2877400640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f285c06b4d0 0x7f285c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2877400640 1 -- 172.21.15.97:0/2309287139 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f286808a950 con 0x7f28800d28a0 2024-03-06T07:32:50.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f287fe00640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f285c06b4d0 0x7f285c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2889273740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 0x555f0d164930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191d70 0x555f0d194720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d164e70 0x555f0d194c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2889273740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0d0ce600 con 0x555f0d191990 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0d077930 con 0x555f0d164e70 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.467+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0d0ca6a0 con 0x555f0d191d70 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 0x555f0d164930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 0x555f0d164930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34104/0 (socket says 172.21.15.97:34104) 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 -- 172.21.15.97:0/1447920658 learned_addr learned my addr 172.21.15.97:0/1447920658 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287fe00640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f285c06b4d0 0x7f285c06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2878035000 tx=0x7f287802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 -- 172.21.15.97:0/1447920658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191d70 msgr2=0x555f0d194720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191d70 0x555f0d194720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 -- 172.21.15.97:0/1447920658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d164e70 msgr2=0x555f0d194c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d164e70 0x555f0d194c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 -- 172.21.15.97:0/1447920658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f0cde32d0 con 0x555f0d191990 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.468+0000 7f287f400640 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 0x555f0d164930 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2870009910 tx=0x7f2870009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11505a211f8dd5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f286fe00640 1 -- 172.21.15.97:0/1447920658 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870017020 con 0x555f0d191990 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f286fe00640 1 -- 172.21.15.97:0/1447920658 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287000ea10 con 0x555f0d191990 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f286fe00640 1 -- 172.21.15.97:0/1447920658 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000f040 con 0x555f0d191990 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 -- 172.21.15.97:0/1447920658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 msgr2=0x555f0d164930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 0x555f0d164930 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2870009910 tx=0x7f2870009dd0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 -- 172.21.15.97:0/1447920658 shutdown_connections 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d164e70 0x555f0d194c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191d70 0x555f0d194720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 --2- 172.21.15.97:0/1447920658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d191990 0x555f0d164930 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 -- 172.21.15.97:0/1447920658 >> 172.21.15.97:0/1447920658 conn(0x555f0d166ba0 msgr2=0x555f0d1af720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.469+0000 7f2889273740 1 -- 172.21.15.97:0/1447920658 shutdown_connections 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.470+0000 7f2889273740 1 -- 172.21.15.97:0/1447920658 wait complete. 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.470+0000 7f2889273740 1 -- start start 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.470+0000 7f2889273740 1 Processor -- start 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.471+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d18edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.471+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1931c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.471+0000 7f2889273740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d193700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.471+0000 7f2889273740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0d169140 con 0x555f0d1afe70 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.471+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0d08ad90 con 0x555f0d191d70 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.471+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0d168530 con 0x555f0d191990 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f2889273740 10 client.0 _ll_register_callbacks cb 0x555f0d0c6350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d193700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d193700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34110/0 (socket says 172.21.15.97:34110) 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 -- 172.21.15.97:0/3212482971 learned_addr learned my addr 172.21.15.97:0/3212482971 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f2877e00640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287f400640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d18edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 -- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 msgr2=0x555f0d18edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d18edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 -- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 msgr2=0x555f0d1931c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1931c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.472+0000 7f287fe00640 1 -- 172.21.15.97:0/3212482971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f0cead6b0 con 0x555f0d1afe70 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f287fe00640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d193700 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f287802c4e0 tx=0x7f2878004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5da08a803783678 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f286cc00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f286cc00640 1 -- 172.21.15.97:0/3212482971 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878040ba0 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f286cc00640 1 -- 172.21.15.97:0/3212482971 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f287803a040 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f287000ea10 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f286cc00640 1 -- 172.21.15.97:0/3212482971 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2878056050 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f2889273740 10 client.7242 fetch_fsmap learned FSMap version 60 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x555f0d19a530 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f286cc00640 1 -- 172.21.15.97:0/3212482971 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287000ea10 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f286cc00640 1 -- 172.21.15.97:0/3212482971 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2878059020 con 0x555f0d1afe70 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f2889273740 10 client.7242 fetch_fsmap finished waiting for FSMap version 60 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f2889273740 10 client.7242 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.473+0000 7f2889273740 20 client.7242 populate_metadata read hostname 'smithi097' 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.474+0000 7f2889273740 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d195db0 0x555f0d1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.474+0000 7f2889273740 4 client.7242 mds_command: new command op to 7113 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.474+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 --> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f0d0dbb40 con 0x555f0d195db0 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.474+0000 7f287f400640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d195db0 0x555f0d1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.474+0000 7f287f400640 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d195db0 0x555f0d1977c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e3d396932175b9 server_cookie=83af07dda1a0f125 in_seq=0 out_seq=0 2024-03-06T07:32:50.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.474+0000 7f286cc00640 10 client.7242 ms_handle_connect on v2:172.21.15.97:6836/1818345324 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f286cc00640 1 -- 172.21.15.97:0/3212482971 <== mds.0 v2:172.21.15.97:6836/1818345324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555f0d0dbb40 con 0x555f0d195db0 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f286cc00640 10 client.7242 handle_command_reply: tid=0 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f2889273740 1 client.7242 shutdown 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 msgr2=0x555f0d193700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f2889273740 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d193700 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f287802c4e0 tx=0x7f2878004ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 shutdown_connections 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f2889273740 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d195db0 0x555f0d1977c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.475+0000 7f2889273740 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d193700 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f2889273740 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1931c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f2889273740 1 --2- 172.21.15.97:0/3212482971 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d18edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 >> 172.21.15.97:0/3212482971 conn(0x555f0d164e70 msgr2=0x555f0d169670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f286cc00640 0 client.7242 ms_handle_reset on v2:172.21.15.97:6836/1818345324 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 shutdown_connections 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f2889273740 1 -- 172.21.15.97:0/3212482971 wait complete. 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.476+0000 7f2889273740 20 client.7242 trim_cache size 0 max 16384 2024-03-06T07:32:50.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.494+0000 7f2889273740 1 Processor -- start 2024-03-06T07:32:50.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.494+0000 7f2889273740 1 -- start start 2024-03-06T07:32:50.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d1af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d188770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f2889273740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d188cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f2889273740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28801ff400 con 0x555f0d1afe70 2024-03-06T07:32:50.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800cdae0 con 0x555f0d191d70 2024-03-06T07:32:50.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800cb710 con 0x555f0d191990 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f287fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d188cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f287fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d188cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34118/0 (socket says 172.21.15.97:34118) 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f287fe00640 1 -- 172.21.15.97:0/2226627104 learned_addr learned my addr 172.21.15.97:0/2226627104 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.495+0000 7f287f400640 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d1af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287fe00640 1 -- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 msgr2=0x555f0d1af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287fe00640 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d1af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287fe00640 1 -- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 msgr2=0x555f0d188770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287fe00640 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d188770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287fe00640 1 -- 172.21.15.97:0/2226627104 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f0d192500 con 0x555f0d1afe70 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287f400640 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d1af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f287fe00640 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d188cb0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2878000f80 tx=0x7f2878042070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dffd789f77a722 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f286f400640 1 -- 172.21.15.97:0/2226627104 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287805e070 con 0x555f0d1afe70 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f286f400640 1 -- 172.21.15.97:0/2226627104 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287803e5c0 con 0x555f0d1afe70 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.496+0000 7f286f400640 1 -- 172.21.15.97:0/2226627104 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287803e790 con 0x555f0d1afe70 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 -- 172.21.15.97:0/2226627104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 msgr2=0x555f0d188cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d188cb0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2878000f80 tx=0x7f2878042070 comp rx=0 tx=0).stop 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 -- 172.21.15.97:0/2226627104 shutdown_connections 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d188cb0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d188770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 --2- 172.21.15.97:0/2226627104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d1af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 -- 172.21.15.97:0/2226627104 >> 172.21.15.97:0/2226627104 conn(0x555f0d164e70 msgr2=0x555f0d19ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 -- 172.21.15.97:0/2226627104 shutdown_connections 2024-03-06T07:32:50.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.497+0000 7f2889273740 1 -- 172.21.15.97:0/2226627104 wait complete. 2024-03-06T07:32:50.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.498+0000 7f2889273740 1 -- start start 2024-03-06T07:32:50.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.498+0000 7f2889273740 1 Processor -- start 2024-03-06T07:32:50.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d21ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1c2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d169e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0d195c30 con 0x555f0d1afe70 2024-03-06T07:32:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0d19a870 con 0x555f0d191d70 2024-03-06T07:32:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0d196230 con 0x555f0d191990 2024-03-06T07:32:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f2889273740 10 client.0 _ll_register_callbacks cb 0x7f2878009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d169e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d169e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34126/0 (socket says 172.21.15.97:34126) 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 -- 172.21.15.97:0/2178525988 learned_addr learned my addr 172.21.15.97:0/2178525988 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 -- 172.21.15.97:0/2178525988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 msgr2=0x555f0d21ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d21ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 -- 172.21.15.97:0/2178525988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 msgr2=0x555f0d1c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.499+0000 7f287fe00640 1 -- 172.21.15.97:0/2178525988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f0d169bf0 con 0x555f0d1afe70 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f287fe00640 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d169e10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f287802c4e0 tx=0x7f287803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170c60bfff1690cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f286cc00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f286cc00640 1 -- 172.21.15.97:0/2178525988 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287805e040 con 0x555f0d1afe70 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f286cc00640 1 -- 172.21.15.97:0/2178525988 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2878003990 con 0x555f0d1afe70 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f287803e5c0 con 0x555f0d1afe70 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f286cc00640 1 -- 172.21.15.97:0/2178525988 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2878073050 con 0x555f0d1afe70 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f2889273740 10 client.7248 fetch_fsmap learned FSMap version 60 2024-03-06T07:32:50.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x555f0d16a350 con 0x555f0d1afe70 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.500+0000 7f286cc00640 1 -- 172.21.15.97:0/2178525988 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287803e5c0 con 0x555f0d1afe70 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f286cc00640 1 -- 172.21.15.97:0/2178525988 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2878076020 con 0x555f0d1afe70 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f2889273740 10 client.7248 fetch_fsmap finished waiting for FSMap version 60 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f2889273740 10 client.7248 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f2889273740 20 client.7248 populate_metadata read hostname 'smithi097' 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f2889273740 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d1ae240 0x555f0d18ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f2889273740 4 client.7248 mds_command: new command op to 7113 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 --> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555f0d0d5e90 con 0x555f0d1ae240 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f287f400640 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d1ae240 0x555f0d18ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f287f400640 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d1ae240 0x555f0d18ff60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf6d90b006b94822 server_cookie=5e0015af37207a40 in_seq=0 out_seq=0 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.501+0000 7f286cc00640 10 client.7248 ms_handle_connect on v2:172.21.15.97:6836/1818345324 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f286cc00640 1 -- 172.21.15.97:0/2178525988 <== mds.0 v2:172.21.15.97:6836/1818345324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x555f0d0d5e90 con 0x555f0d1ae240 2024-03-06T07:32:50.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f286cc00640 10 client.7248 handle_command_reply: tid=0 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 client.7248 shutdown 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 msgr2=0x555f0d169e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d169e10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f287802c4e0 tx=0x7f287803e000 comp rx=0 tx=0).stop 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 shutdown_connections 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:6836/1818345324,v1:172.21.15.97:6837/1818345324] conn(0x555f0d1ae240 0x555f0d18ff60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555f0d1afe70 0x555f0d169e10 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555f0d191d70 0x555f0d1c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 --2- 172.21.15.97:0/2178525988 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x555f0d191990 0x555f0d21ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 >> 172.21.15.97:0/2178525988 conn(0x555f0d164e70 msgr2=0x555f0d1941f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f286cc00640 0 client.7248 ms_handle_reset on v2:172.21.15.97:6836/1818345324 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 shutdown_connections 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 1 -- 172.21.15.97:0/2178525988 wait complete. 2024-03-06T07:32:50.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.502+0000 7f2889273740 20 client.7248 trim_cache size 0 max 16384 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 7113, 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 6.8720685269999997, 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 60, 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 96, 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 96, 2024-03-06T07:32:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 13.143094945 2024-03-06T07:32:50.511 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f285c06b4d0 msgr2=0x7f285c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f285c06b4d0 0x7f285c06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2878035000 tx=0x7f287802e000 comp rx=0 tx=0).stop 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 msgr2=0x7f28801520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28801520b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f286800d830 tx=0x7f286800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 shutdown_connections 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f285c06b4d0 0x7f285c06d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2880161580 0x7f288015f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28800d28a0 0x7f28801520b0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 --2- 172.21.15.97:0/2309287139 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f28800d1e90 0x7f28800c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 >> 172.21.15.97:0/2309287139 conn(0x7f28800d3d80 msgr2=0x7f28800741a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 shutdown_connections 2024-03-06T07:32:50.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.503+0000 7f2885000640 1 -- 172.21.15.97:0/2309287139 wait complete. 2024-03-06T07:32:50.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.708+0000 7f2ad2a00640 1 Processor -- start 2024-03-06T07:32:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.708+0000 7f2ad2a00640 1 -- start start 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad2a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2acc0cd860 0x7f2acc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad2a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2acc06b500 0x7f2acc06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad2a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc06b900 con 0x7f2acc15e070 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad2a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0b5010 con 0x7f2acc06b500 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad2a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0b52f0 con 0x7f2acc0cd860 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2acc06b500 0x7f2acc06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2acc06b500 0x7f2acc06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34132/0 (socket says 172.21.15.97:34132) 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ad0a00640 1 -- 172.21.15.97:0/4064788397 learned_addr learned my addr 172.21.15.97:0/4064788397 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.710+0000 7f2ac3400640 1 --2- 172.21.15.97:0/4064788397 >> v2:172.21.15.135:3300/0 conn(0x7f2acc0cd860 0x7f2acc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.711+0000 7f2ad0a00640 1 -- 172.21.15.97:0/4064788397 >> v2:172.21.15.135:3300/0 conn(0x7f2acc0cd860 msgr2=0x7f2acc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.711+0000 7f2ad0a00640 1 --2- 172.21.15.97:0/4064788397 >> v2:172.21.15.135:3300/0 conn(0x7f2acc0cd860 0x7f2acc06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.711+0000 7f2ad0a00640 1 -- 172.21.15.97:0/4064788397 >> v1:172.21.15.135:6789/0 conn(0x7f2acc15e070 legacy=0x7f2acc066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.711+0000 7f2ad0a00640 1 -- 172.21.15.97:0/4064788397 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0b55d0 con 0x7f2acc06b500 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.712+0000 7f2ad0a00640 1 --2- 172.21.15.97:0/4064788397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc06b500 0x7f2acc06db00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f2acc06cad0 tx=0x7f2ac402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e3849d6aa2c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.712+0000 7f2acb400640 1 -- 172.21.15.97:0/4064788397 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac402ca80 con 0x7f2acc06b500 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.712+0000 7f2acb400640 1 -- 172.21.15.97:0/4064788397 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac402cc20 con 0x7f2acc06b500 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.712+0000 7f2acb400640 1 -- 172.21.15.97:0/4064788397 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac40343f0 con 0x7f2acc06b500 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 -- 172.21.15.97:0/4064788397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc06b500 msgr2=0x7f2acc06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/4064788397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc06b500 0x7f2acc06db00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f2acc06cad0 tx=0x7f2ac402bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 -- 172.21.15.97:0/4064788397 shutdown_connections 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/4064788397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc06b500 0x7f2acc06db00 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/4064788397 >> v2:172.21.15.135:3300/0 conn(0x7f2acc0cd860 0x7f2acc06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 -- 172.21.15.97:0/4064788397 >> 172.21.15.97:0/4064788397 conn(0x7f2acc0b1e10 msgr2=0x7f2acc0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 -- 172.21.15.97:0/4064788397 shutdown_connections 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.713+0000 7f2ad2a00640 1 -- 172.21.15.97:0/4064788397 wait complete. 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.715+0000 7f2ad2a00640 1 Processor -- start 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.715+0000 7f2ad2a00640 1 -- start start 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad2a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2acc06b500 0x7f2acc099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad2a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2acc0cd860 0x7f2acc0adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 0x7f2acc152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad2a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc15e450 con 0x7f2acc15e070 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad2a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0b5b90 con 0x7f2acc0cd860 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad2a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0b60b0 con 0x7f2acc06b500 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 0x7f2acc152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 0x7f2acc152d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34134/0 (socket says 172.21.15.97:34134) 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2ad0a00640 1 -- 172.21.15.97:0/2224230002 learned_addr learned my addr 172.21.15.97:0/2224230002 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.716+0000 7f2acbe00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2acc06b500 0x7f2acc099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ac3400640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2acc0cd860 0x7f2acc0adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad0a00640 1 -- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2acc06b500 msgr2=0x7f2acc099af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad0a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2acc06b500 0x7f2acc099af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad0a00640 1 -- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2acc0cd860 msgr2=0x7f2acc0adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad0a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2acc0cd860 0x7f2acc0adb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad0a00640 1 -- 172.21.15.97:0/2224230002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0c3f50 con 0x7f2acc15e070 2024-03-06T07:32:50.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ac3400640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2acc0cd860 0x7f2acc0adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad0a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 0x7f2acc152d50 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009f40 tx=0x7f2ac4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4043020 con 0x7f2acc15e070 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac403e450 con 0x7f2acc15e070 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4009510 con 0x7f2acc15e070 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.717+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc0c4230 con 0x7f2acc15e070 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.718+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc0c47f0 con 0x7f2acc15e070 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.719+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc0b1c30 con 0x7f2acc15e070 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.720+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ac4046020 con 0x7f2acc15e070 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.721+0000 7f2ac8c00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.721+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ac40ace00 con 0x7f2acc15e070 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.721+0000 7f2acbe00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.722+0000 7f2acbe00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2acc162cd0 tx=0x7f2ab4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.725+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2acc0b1c30 con 0x7f2acc15e070 2024-03-06T07:32:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.959+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2acc04f330 con 0x7f2acc15e070 2024-03-06T07:32:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.959+0000 7f2ac8c00640 1 -- 172.21.15.97:0/2224230002 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v60) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2ac404d020 con 0x7f2acc15e070 2024-03-06T07:32:50.962 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2aa806b4d0 msgr2=0x7f2aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2acc162cd0 tx=0x7f2ab4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 msgr2=0x7f2acc152d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 0x7f2acc152d50 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009f40 tx=0x7f2ac4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 shutdown_connections 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2acc15e070 0x7f2acc152d50 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2acc0cd860 0x7f2acc0adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 --2- 172.21.15.97:0/2224230002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2acc06b500 0x7f2acc099af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 >> 172.21.15.97:0/2224230002 conn(0x7f2acc0b1e10 msgr2=0x7f2acc06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 shutdown_connections 2024-03-06T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:50.962+0000 7f2ad2a00640 1 -- 172.21.15.97:0/2224230002 wait complete. 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:50.983 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:50.984 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:50.984 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:50.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:50.984 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:32:51.029 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:51.029 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:51.030 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42304sec preferred_lft 42304sec 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:51.032 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:32:51.032 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:51.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.033 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:51.152 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:32:51.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.155 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.199 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:32:51.199 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:32:51.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:32:51.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:32:51.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:32:51.200 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:32:51.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.418 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:32:51.463 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:51.463 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:51.464 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:51.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:32:51.591 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:51.592 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:51.636 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:51.636 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:51.636 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:51.723 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:51.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:51.892 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:51.892 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:51.924 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:51.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:51.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:32:51.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:32:51.924 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:32:51.926 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:51.926 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:51.926 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:51.926 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:51.926 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:51.929 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:51.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:51.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:51.999 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:52.059 INFO:teuthology.orchestra.run.smithi097.stdout:client5249 2024-03-06T07:32:52.059 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:32:52.059 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:52.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:52.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:52.309 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:32:52.309 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:32:52.309 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:52.309 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.310 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:52.315 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:32:52.316 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.320 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_files = 6 2024-03-06T07:32:52.330 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:32:52.330 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:32:52.330 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:52.330 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:32:52.330 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.331 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:52.332 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:32:52.333 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.334 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.336 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.341 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_files = 6 2024-03-06T07:32:52.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:52.384 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:52.544 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:32:52.544 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:32:52.544 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:52.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:32:52.549 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.552 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.553 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:32:52.554 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_files = 6 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.555 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:52.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:52.558 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.564 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:32:52.565 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:32:52.566 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:32:52.566 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_files = 6 2024-03-06T07:32:52.566 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:52.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:52.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.798+0000 7f8d2ba00640 1 Processor -- start 2024-03-06T07:32:52.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.798+0000 7f8d2ba00640 1 -- start start 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.799+0000 7f8d2ba00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8d240b24e0 0x7f8d240b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.799+0000 7f8d2ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d240b2980 0x7f8d24161f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d2ba00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d24162b90 con 0x7f8d241624c0 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d2ba00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d24162930 con 0x7f8d240b24e0 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d2ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24162e70 con 0x7f8d240b2980 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8d240b24e0 0x7f8d240b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8d240b24e0 0x7f8d240b2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34140/0 (socket says 172.21.15.97:34140) 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 -- 172.21.15.97:0/1723069063 learned_addr learned my addr 172.21.15.97:0/1723069063 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d23e00640 1 --2- 172.21.15.97:0/1723069063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d240b2980 0x7f8d24161f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 -- 172.21.15.97:0/1723069063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d240b2980 msgr2=0x7f8d24161f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 --2- 172.21.15.97:0/1723069063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d240b2980 0x7f8d24161f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 -- 172.21.15.97:0/1723069063 >> v1:172.21.15.135:6789/0 conn(0x7f8d241624c0 legacy=0x7f8d2406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.800+0000 7f8d29000640 1 -- 172.21.15.97:0/1723069063 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240d0c50 con 0x7f8d240b24e0 2024-03-06T07:32:52.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.802+0000 7f8d29000640 1 --2- 172.21.15.97:0/1723069063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b2eb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009870 tx=0x7f8d1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1a8202a632e463 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.802+0000 7f8d23400640 1 -- 172.21.15.97:0/1723069063 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14034070 con 0x7f8d240b24e0 2024-03-06T07:32:52.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.802+0000 7f8d23400640 1 -- 172.21.15.97:0/1723069063 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d14030840 con 0x7f8d240b24e0 2024-03-06T07:32:52.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.802+0000 7f8d23400640 1 -- 172.21.15.97:0/1723069063 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14030c10 con 0x7f8d240b24e0 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.802+0000 7f8d2ba00640 1 -- 172.21.15.97:0/1723069063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 msgr2=0x7f8d240b2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.802+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/1723069063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b2eb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009870 tx=0x7f8d1402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.803+0000 7f8d2ba00640 1 -- 172.21.15.97:0/1723069063 shutdown_connections 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.803+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/1723069063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d240b2980 0x7f8d24161f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.803+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/1723069063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b2eb0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.803+0000 7f8d2ba00640 1 -- 172.21.15.97:0/1723069063 >> 172.21.15.97:0/1723069063 conn(0x7f8d240b38e0 msgr2=0x7f8d240b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.803+0000 7f8d2ba00640 1 -- 172.21.15.97:0/1723069063 shutdown_connections 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.803+0000 7f8d2ba00640 1 -- 172.21.15.97:0/1723069063 wait complete. 2024-03-06T07:32:52.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.804+0000 7f8d2ba00640 1 Processor -- start 2024-03-06T07:32:52.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.804+0000 7f8d2ba00640 1 -- start start 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.805+0000 7f8d2ba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.805+0000 7f8d2ba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d240b2980 0x7f8d240ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.805+0000 7f8d2ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d241624c0 0x7f8d24155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.805+0000 7f8d2ba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240b46a0 con 0x7f8d240b24e0 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.805+0000 7f8d2ba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240d1210 con 0x7f8d240b2980 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.805+0000 7f8d2ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240b4820 con 0x7f8d241624c0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34146/0 (socket says 172.21.15.97:34146) 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 -- 172.21.15.97:0/4201242070 learned_addr learned my addr 172.21.15.97:0/4201242070 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29a00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d241624c0 0x7f8d24155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 -- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d241624c0 msgr2=0x7f8d24155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d23e00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d240b2980 0x7f8d240ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d241624c0 0x7f8d24155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 -- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d240b2980 msgr2=0x7f8d240ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d240b2980 0x7f8d240ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 -- 172.21.15.97:0/4201242070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240cd9a0 con 0x7f8d240b24e0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29a00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d241624c0 0x7f8d24155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.806+0000 7f8d29000640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b15d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8d1402b9c0 tx=0x7f8d140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.807+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14034040 con 0x7f8d240b24e0 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.807+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d14039d90 con 0x7f8d240b24e0 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.807+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14043610 con 0x7f8d240b24e0 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.807+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d240cc900 con 0x7f8d240b24e0 2024-03-06T07:32:52.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.807+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d240ccec0 con 0x7f8d240b24e0 2024-03-06T07:32:52.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.808+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d24068050 con 0x7f8d240b24e0 2024-03-06T07:32:52.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.810+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d1404a060 con 0x7f8d240b24e0 2024-03-06T07:32:52.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.811+0000 7f8d20c00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d0006b4d0 0x7f8d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.811+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d140ac300 con 0x7f8d240b24e0 2024-03-06T07:32:52.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.811+0000 7f8d23e00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d0006b4d0 0x7f8d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.812+0000 7f8d23e00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d0006b4d0 0x7f8d0006d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8d240b1310 tx=0x7f8d0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:52.820+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d1403e070 con 0x7f8d240b24e0 2024-03-06T07:32:53.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.017+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d240b6c10 con 0x7f8d240b24e0 2024-03-06T07:32:53.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.018+0000 7f8d20c00640 1 -- 172.21.15.97:0/4201242070 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d1407b0e0 con 0x7f8d240b24e0 2024-03-06T07:32:53.021 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d0006b4d0 msgr2=0x7f8d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d0006b4d0 0x7f8d0006d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8d240b1310 tx=0x7f8d0c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 msgr2=0x7f8d240b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b15d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8d1402b9c0 tx=0x7f8d140046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 shutdown_connections 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d0006b4d0 0x7f8d0006d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d241624c0 0x7f8d24155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d240b2980 0x7f8d240ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 --2- 172.21.15.97:0/4201242070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d240b24e0 0x7f8d240b15d0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 >> 172.21.15.97:0/4201242070 conn(0x7f8d240b38e0 msgr2=0x7f8d240b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 shutdown_connections 2024-03-06T07:32:53.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.021+0000 7f8d2ba00640 1 -- 172.21.15.97:0/4201242070 wait complete. 2024-03-06T07:32:53.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:53.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:32:53.216 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:53.217 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:32:53.219 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:32:53.220 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:32:53.221 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:32:53.222 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:32:53.223 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:32:53.224 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:32:53.225 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_files = 6 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:32:53.226 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:32:53.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:53.228 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:53.229 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:32:53.230 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:32:53.231 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:32:53.232 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:32:53.233 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:53.234 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:53.235 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:32:53.236 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_files = 6 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:53.237 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:32:53.269 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:53.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:53.269 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:53.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:53.270 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:53.270 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:53.270 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:53.270 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:53.287 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:53.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:32:53.288 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:53.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:32:53.290 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:53.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:32:53.292 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:53.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:32:53.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.474+0000 7f386b000640 1 Processor -- start 2024-03-06T07:32:53.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.474+0000 7f386b000640 1 -- start start 2024-03-06T07:32:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f386b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f38640acd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f386b000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38641635a0 con 0x7f386406d400 2024-03-06T07:32:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f386b000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3864163880 con 0x7f386415cc90 2024-03-06T07:32:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f386b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386406cc00 con 0x7f38640d0780 2024-03-06T07:32:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f3863400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f386406d400 0x7f38640d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43784/0 (socket says 172.21.15.97:43784) 2024-03-06T07:32:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f3863400640 1 -- 172.21.15.97:0/1040637098 learned_addr learned my addr 172.21.15.97:0/1040637098 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.476+0000 7f3869000640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f38640acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618357448 0 0) 0x7f38641635a0 con 0x7f386406d400 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3848003520 con 0x7f386406d400 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993566263 0 0) 0x7f3848003520 con 0x7f386406d400 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 msgr2=0x7f38640acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f38640acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 >> v1:172.21.15.135:6789/0 conn(0x7f386415cc90 legacy=0x7f3864162b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386406cee0 con 0x7f386406d400 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3850001670 con 0x7f386406d400 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 >> v1:172.21.15.97:6789/0 conn(0x7f386406d400 legacy=0x7f38640d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38480039e0 0x7f3848003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f3848004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3848004d40 0x7f38480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386406cc00 con 0x7f38480039e0 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384800bad0 con 0x7f3848004d40 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.477+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384800bcd0 con 0x7f38640d0780 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863e00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38480039e0 0x7f3848003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3869000640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3848004d40 0x7f38480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863400640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f3848004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863e00640 1 -- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 msgr2=0x7f3848004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863e00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f3848004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863e00640 1 -- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3848004d40 msgr2=0x7f38480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863e00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3848004d40 0x7f38480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863e00640 1 -- 172.21.15.97:0/1040637098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848003520 con 0x7f38480039e0 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.478+0000 7f3863400640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38640d0780 0x7f3848004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.479+0000 7f3863e00640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38480039e0 0x7f3848003e00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f3854001720 tx=0x7f3848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f335e7b5e32ae838 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.480+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854001370 con 0x7f38480039e0 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.480+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38540052d0 con 0x7f38480039e0 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.480+0000 7f3862a00640 1 -- 172.21.15.97:0/1040637098 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854003830 con 0x7f38480039e0 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.480+0000 7fab22600640 1 Processor -- start 2024-03-06T07:32:53.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.480+0000 7fab22600640 1 -- start start 2024-03-06T07:32:53.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f386b000640 1 -- 172.21.15.97:0/1040637098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38480039e0 msgr2=0x7f3848003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f386b000640 1 --2- 172.21.15.97:0/1040637098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38480039e0 0x7f3848003e00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f3854001720 tx=0x7f3848005830 comp rx=0 tx=0).stop 2024-03-06T07:32:53.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f3869000640 1 -- 172.21.15.97:0/1040637098 reap_dead start 2024-03-06T07:32:53.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f386b000640 1 -- 172.21.15.97:0/1040637098 shutdown_connections 2024-03-06T07:32:53.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f386b000640 1 -- 172.21.15.97:0/1040637098 >> 172.21.15.97:0/1040637098 conn(0x7f386406b360 msgr2=0x7f386406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f386b000640 1 -- 172.21.15.97:0/1040637098 shutdown_connections 2024-03-06T07:32:53.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.481+0000 7f386b000640 1 -- 172.21.15.97:0/1040637098 wait complete. 2024-03-06T07:32:53.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab22600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c0ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab22600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1c155870 con 0x7fab1c15ec10 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab22600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1c0ad870 con 0x7fab1c155490 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab22600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0ad640 con 0x7fab1c0d0780 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab1be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fab1c15ec10 0x7fab1c15f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43792/0 (socket says 172.21.15.97:43792) 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab1be00640 1 -- 172.21.15.97:0/3899075649 learned_addr learned my addr 172.21.15.97:0/3899075649 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.482+0000 7fab1aa00640 1 --2- 172.21.15.97:0/3899075649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c0ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010019898 0 0) 0x7fab1c155870 con 0x7fab1c15ec10 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab00003520 con 0x7fab1c15ec10 2024-03-06T07:32:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481858932 0 0) 0x7fab1c0ad870 con 0x7fab1c155490 2024-03-06T07:32:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7f386b000640 1 Processor -- start 2024-03-06T07:32:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab1c155870 con 0x7fab1c155490 2024-03-06T07:32:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1aa00640 1 -- 172.21.15.97:0/3899075649 >> v1:172.21.15.135:6789/0 conn(0x7fab1c155490 legacy=0x7fab1c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1aa00640 1 -- 172.21.15.97:0/3899075649 >> v1:172.21.15.97:6789/0 conn(0x7fab1c15ec10 legacy=0x7fab1c15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.484+0000 7f386b000640 1 -- start start 2024-03-06T07:32:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.483+0000 7fab1aa00640 1 -- 172.21.15.97:0/3899075649 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c0adb50 con 0x7fab1c0d0780 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.484+0000 7f386b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38480039e0 0x7f3864144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.484+0000 7f386b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 0x7f3864166950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.484+0000 7f386b000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3864153f30 0x7f38641543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.484+0000 7f386b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640b4e60 con 0x7f38641664f0 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f386b000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3864163210 con 0x7f3864153f30 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f386b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640b4fe0 con 0x7f38480039e0 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 0x7f3864166950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 0x7f3864166950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34158/0 (socket says 172.21.15.97:34158) 2024-03-06T07:32:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 -- 172.21.15.97:0/2076163672 learned_addr learned my addr 172.21.15.97:0/2076163672 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7fab1aa00640 1 --2- 172.21.15.97:0/3899075649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c0ae630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fab080097d0 tx=0x7fab0802dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca884ee64bed444 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3869000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3864153f30 0x7f38641543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863e00640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38480039e0 0x7f3864144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 -- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38480039e0 msgr2=0x7f3864144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38480039e0 0x7f3864144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 -- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3864153f30 msgr2=0x7f38641543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3864153f30 0x7f38641543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863400640 1 -- 172.21.15.97:0/2076163672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3864154d60 con 0x7f38641664f0 2024-03-06T07:32:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7f3863e00640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38480039e0 0x7f3864144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.485+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0802f870 con 0x7fab1c0d0780 2024-03-06T07:32:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0802fa10 con 0x7fab1c0d0780 2024-03-06T07:32:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7fab1a000640 1 -- 172.21.15.97:0/3899075649 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0802fdc0 con 0x7fab1c0d0780 2024-03-06T07:32:53.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7f3863400640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 0x7f3864166950 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f385000d800 tx=0x7f385000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850019070 con 0x7f38641664f0 2024-03-06T07:32:53.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3864155040 con 0x7f38641664f0 2024-03-06T07:32:53.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38500074e0 con 0x7f38641664f0 2024-03-06T07:32:53.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38500049e0 con 0x7f38641664f0 2024-03-06T07:32:53.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.486+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3864155600 con 0x7f38641664f0 2024-03-06T07:32:53.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.487+0000 7fab22600640 1 -- 172.21.15.97:0/3899075649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 msgr2=0x7fab1c0ae630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.487+0000 7fab22600640 1 --2- 172.21.15.97:0/3899075649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c0ae630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fab080097d0 tx=0x7fab0802dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.487+0000 7fab22600640 1 -- 172.21.15.97:0/3899075649 shutdown_connections 2024-03-06T07:32:53.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.487+0000 7fab22600640 1 --2- 172.21.15.97:0/3899075649 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c0ae630 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.487+0000 7fab22600640 1 -- 172.21.15.97:0/3899075649 >> 172.21.15.97:0/3899075649 conn(0x7fab1c06b360 msgr2=0x7fab1c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.488+0000 7fab22600640 1 -- 172.21.15.97:0/3899075649 shutdown_connections 2024-03-06T07:32:53.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.488+0000 7fab22600640 1 -- 172.21.15.97:0/3899075649 wait complete. 2024-03-06T07:32:53.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.488+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3864068070 con 0x7f38641664f0 2024-03-06T07:32:53.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.489+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f385001c070 con 0x7f38641664f0 2024-03-06T07:32:53.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.490+0000 7f385be00640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b500 0x7f384406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.490+0000 7fab22600640 1 Processor -- start 2024-03-06T07:32:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.490+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3850089d70 con 0x7f38641664f0 2024-03-06T07:32:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.490+0000 7fab22600640 1 -- start start 2024-03-06T07:32:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.491+0000 7f3863e00640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b500 0x7f384406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.491+0000 7f3863e00640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b500 0x7f384406d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3854033000 tx=0x7f3854009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab22600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c13f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab22600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 0x7fab1c0b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab22600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab1c15ec10 0x7fab1c0b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab22600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0ae110 con 0x7fab1c155490 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab22600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c15f980 con 0x7fab1c15ec10 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab22600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c071960 con 0x7fab1c0d0780 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab1aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 0x7fab1c0b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab1aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 0x7fab1c0b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34170/0 (socket says 172.21.15.97:34170) 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.492+0000 7fab1aa00640 1 -- 172.21.15.97:0/1484713193 learned_addr learned my addr 172.21.15.97:0/1484713193 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1b400640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c13f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1aa00640 1 -- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 msgr2=0x7fab1c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1aa00640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c13f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1be00640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab1c15ec10 0x7fab1c0b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1aa00640 1 -- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab1c15ec10 msgr2=0x7fab1c0b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1aa00640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab1c15ec10 0x7fab1c0b27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1aa00640 1 -- 172.21.15.97:0/1484713193 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c0b2f10 con 0x7fab1c155490 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1b400640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c13f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.493+0000 7fab1be00640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab1c15ec10 0x7fab1c0b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.494+0000 7fab1aa00640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 0x7fab1c0b0a90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fab080098f0 tx=0x7fab080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.495+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08048070 con 0x7fab1c155490 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.495+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08003610 con 0x7fab1c155490 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.495+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080039c0 con 0x7fab1c155490 2024-03-06T07:32:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.495+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c0b31f0 con 0x7fab1c155490 2024-03-06T07:32:53.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.495+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3850058b50 con 0x7f38641664f0 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.495+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1c0b37b0 con 0x7fab1c155490 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.497+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1c068070 con 0x7fab1c155490 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.498+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab08003b60 con 0x7fab1c155490 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.499+0000 7fab13400640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faafc06b4d0 0x7faafc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.499+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab080ad750 con 0x7fab1c155490 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.500+0000 7fab1b400640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faafc06b4d0 0x7faafc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.501+0000 7fab1b400640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faafc06b4d0 0x7faafc06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fab0c001760 tx=0x7fab0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.503+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab08035030 con 0x7fab1c155490 2024-03-06T07:32:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.524+0000 7ff992e00640 1 Processor -- start 2024-03-06T07:32:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.524+0000 7ff992e00640 1 -- start start 2024-03-06T07:32:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.524+0000 7feed5a00640 1 Processor -- start 2024-03-06T07:32:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.524+0000 7feed5a00640 1 -- start start 2024-03-06T07:32:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7ff992e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff98c0b3340 0x7ff98c0d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7ff992e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c06d600 0x7ff98c0d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7ff992e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff98c15e720 0x7ff98c15eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7ff992e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98c15f650 con 0x7ff98c15e720 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7ff992e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98c15f0e0 con 0x7ff98c0b3340 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7ff992e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c15f370 con 0x7ff98c06d600 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.525+0000 7feed5a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feed00b3340 0x7feed00d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7feed5a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed015f370 con 0x7feed006d600 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7feed5a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed006cb20 con 0x7feed015ee30 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7feed5a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed015f650 con 0x7feed00b3340 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff98c15e720 0x7ff98c15eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff98c15e720 0x7ff98c15eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34178/0 (socket says 172.21.15.97:34178) 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 -- 172.21.15.97:0/3398126033 learned_addr learned my addr 172.21.15.97:0/3398126033 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff98be00640 1 --2- 172.21.15.97:0/3398126033 >> v2:172.21.15.135:3300/0 conn(0x7ff98c0b3340 0x7ff98c0d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff98b400640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c06d600 0x7ff98c0d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7feece000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feed006d600 0x7feed015e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43800/0 (socket says 172.21.15.97:43800) 2024-03-06T07:32:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7feece000640 1 -- 172.21.15.97:0/1812740704 learned_addr learned my addr 172.21.15.97:0/1812740704 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7feecea00640 1 --2- 172.21.15.97:0/1812740704 >> v2:172.21.15.97:3300/0 conn(0x7feed00b3340 0x7feed00d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 -- 172.21.15.97:0/3398126033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c06d600 msgr2=0x7ff98c0d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c06d600 0x7ff98c0d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 -- 172.21.15.97:0/3398126033 >> v2:172.21.15.135:3300/0 conn(0x7ff98c0b3340 msgr2=0x7ff98c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 --2- 172.21.15.97:0/3398126033 >> v2:172.21.15.135:3300/0 conn(0x7ff98c0b3340 0x7ff98c0d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.526+0000 7ff990e00640 1 -- 172.21.15.97:0/3398126033 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c06cb20 con 0x7ff98c15e720 2024-03-06T07:32:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.527+0000 7ff98b400640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c06d600 0x7ff98c0d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.527+0000 7feecea00640 1 -- 172.21.15.97:0/1812740704 >> v1:172.21.15.135:6789/0 conn(0x7feed015ee30 legacy=0x7feed00acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.527+0000 7feecd600640 1 -- 172.21.15.97:0/1812740704 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219703606 0 0) 0x7feed015f370 con 0x7feed006d600 2024-03-06T07:32:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.527+0000 7feecea00640 1 -- 172.21.15.97:0/1812740704 >> v1:172.21.15.97:6789/0 conn(0x7feed006d600 legacy=0x7feed015e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.527+0000 7feecea00640 1 -- 172.21.15.97:0/1812740704 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed006ce00 con 0x7feed00b3340 2024-03-06T07:32:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.528+0000 7ff990e00640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c15eba0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff98c06ed00 tx=0x7ff98402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdd51b378cad46c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.528+0000 7ff98aa00640 1 -- 172.21.15.97:0/3398126033 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98402c960 con 0x7ff98c15e720 2024-03-06T07:32:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.528+0000 7ff98aa00640 1 -- 172.21.15.97:0/3398126033 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402cb00 con 0x7ff98c15e720 2024-03-06T07:32:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.528+0000 7ff98aa00640 1 -- 172.21.15.97:0/3398126033 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840343f0 con 0x7ff98c15e720 2024-03-06T07:32:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7feecea00640 1 --2- 172.21.15.97:0/1812740704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed00b3340 0x7feed00d0070 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7feec00097a0 tx=0x7feec002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94df808ce765b402 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7feecd600640 1 -- 172.21.15.97:0/1812740704 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec00315a0 con 0x7feed00b3340 2024-03-06T07:32:53.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 -- 172.21.15.97:0/3398126033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 msgr2=0x7ff98c15eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c15eba0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff98c06ed00 tx=0x7ff98402bdc0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7feecd600640 1 -- 172.21.15.97:0/1812740704 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec0037070 con 0x7feed00b3340 2024-03-06T07:32:53.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7feecd600640 1 -- 172.21.15.97:0/1812740704 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0031960 con 0x7feed00b3340 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 -- 172.21.15.97:0/3398126033 shutdown_connections 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c15eba0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 --2- 172.21.15.97:0/3398126033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c06d600 0x7ff98c0d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 --2- 172.21.15.97:0/3398126033 >> v2:172.21.15.135:3300/0 conn(0x7ff98c0b3340 0x7ff98c0d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 -- 172.21.15.97:0/3398126033 >> 172.21.15.97:0/3398126033 conn(0x7ff98c06b360 msgr2=0x7ff98c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 -- 172.21.15.97:0/3398126033 shutdown_connections 2024-03-06T07:32:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.529+0000 7ff992e00640 1 -- 172.21.15.97:0/3398126033 wait complete. 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 -- 172.21.15.97:0/1812740704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed00b3340 msgr2=0x7feed00d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 --2- 172.21.15.97:0/1812740704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed00b3340 0x7feed00d0070 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7feec00097a0 tx=0x7feec002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 -- 172.21.15.97:0/1812740704 shutdown_connections 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 --2- 172.21.15.97:0/1812740704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed00b3340 0x7feed00d0070 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 -- 172.21.15.97:0/1812740704 >> 172.21.15.97:0/1812740704 conn(0x7feed006b360 msgr2=0x7feed006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 -- 172.21.15.97:0/1812740704 shutdown_connections 2024-03-06T07:32:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.530+0000 7feed5a00640 1 -- 172.21.15.97:0/1812740704 wait complete. 2024-03-06T07:32:53.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.531+0000 7ff992e00640 1 Processor -- start 2024-03-06T07:32:53.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.532+0000 7ff992e00640 1 -- start start 2024-03-06T07:32:53.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.532+0000 7ff992e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff98c06d600 0x7ff98c09c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff992e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c0b3340 0x7ff98c09d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff992e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff992e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c0b3720 con 0x7ff98c15e720 2024-03-06T07:32:53.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff992e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c06d0e0 con 0x7ff98c06d600 2024-03-06T07:32:53.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff992e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c161d10 con 0x7ff98c0b3340 2024-03-06T07:32:53.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7feed5a00640 1 Processor -- start 2024-03-06T07:32:53.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff990e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff990e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c09aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34192/0 (socket says 172.21.15.97:34192) 2024-03-06T07:32:53.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7feed5a00640 1 -- start start 2024-03-06T07:32:53.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff990e00640 1 -- 172.21.15.97:0/957392462 learned_addr learned my addr 172.21.15.97:0/957392462 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff98be00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff98c06d600 0x7ff98c09c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.533+0000 7ff98b400640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c0b3340 0x7ff98c09d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.534+0000 7ff990e00640 1 -- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c0b3340 msgr2=0x7ff98c09d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.534+0000 7ff990e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c0b3340 0x7ff98c09d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.534+0000 7ff990e00640 1 -- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff98c06d600 msgr2=0x7ff98c09c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.534+0000 7ff990e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff98c06d600 0x7ff98c09c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.534+0000 7ff990e00640 1 -- 172.21.15.97:0/957392462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c1f3f00 con 0x7ff98c15e720 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.534+0000 7ff990e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c09aa30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ff984009f40 tx=0x7ff984004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feed5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 0x7feed0162750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984044020 con 0x7ff98c15e720 2024-03-06T07:32:53.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984040670 con 0x7ff98c15e720 2024-03-06T07:32:53.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984009510 con 0x7ff98c15e720 2024-03-06T07:32:53.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feed5a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feed00b3340 0x7feed0160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98c1f41e0 con 0x7ff98c15e720 2024-03-06T07:32:53.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98c1f47a0 con 0x7ff98c15e720 2024-03-06T07:32:53.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feed5a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feed015ee30 0x7feed009d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feed5a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed00d05b0 con 0x7feed006d600 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feed5a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed00b3720 con 0x7feed015ee30 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feed5a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed00d0730 con 0x7feed00b3340 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feecea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 0x7feed0162750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feecea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 0x7feed0162750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34202/0 (socket says 172.21.15.97:34202) 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.535+0000 7feecea00640 1 -- 172.21.15.97:0/4083490534 learned_addr learned my addr 172.21.15.97:0/4083490534 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.536+0000 7feecf400640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feed015ee30 0x7feed009d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.536+0000 7feecea00640 1 -- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feed00b3340 msgr2=0x7feed0160260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.536+0000 7feecea00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feed00b3340 0x7feed0160260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.537+0000 7feecea00640 1 -- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feed015ee30 msgr2=0x7feed009d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.537+0000 7feecea00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feed015ee30 0x7feed009d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.537+0000 7feecea00640 1 -- 172.21.15.97:0/4083490534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0160990 con 0x7feed006d600 2024-03-06T07:32:53.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.537+0000 7feecf400640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feed015ee30 0x7feed009d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:53.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.537+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98c068070 con 0x7ff98c15e720 2024-03-06T07:32:53.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.537+0000 7feecea00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 0x7feed0162750 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7feec00098c0 tx=0x7feec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.538+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0032840 con 0x7feed006d600 2024-03-06T07:32:53.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.538+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec00329e0 con 0x7feed006d600 2024-03-06T07:32:53.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.538+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0032dd0 con 0x7feed006d600 2024-03-06T07:32:53.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.538+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff984047020 con 0x7ff98c15e720 2024-03-06T07:32:53.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.539+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed009abd0 con 0x7feed006d600 2024-03-06T07:32:53.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.539+0000 7ff983e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff96c06b4d0 0x7ff96c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.539+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9840acaf0 con 0x7ff98c15e720 2024-03-06T07:32:53.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.539+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed009b190 con 0x7feed006d600 2024-03-06T07:32:53.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.539+0000 7ff98be00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff96c06b4d0 0x7ff96c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.540+0000 7ff98be00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff96c06b4d0 0x7ff96c06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff98c1405f0 tx=0x7ff97c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.540+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed009b390 con 0x7feed006d600 2024-03-06T07:32:53.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.544+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff98404e030 con 0x7ff98c15e720 2024-03-06T07:32:53.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.546+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feec004c020 con 0x7feed006d600 2024-03-06T07:32:53.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.546+0000 7feec6a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb006b4d0 0x7feeb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.547+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feec002f030 con 0x7feed006d600 2024-03-06T07:32:53.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.547+0000 7feece000640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb006b4d0 0x7feeb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.548+0000 7feece000640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb006b4d0 0x7feeb006d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feebc000c00 tx=0x7feebc007890 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.553+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feed009b390 con 0x7feed006d600 2024-03-06T07:32:53.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.788+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f386415bdb0 con 0x7f38641664f0 2024-03-06T07:32:53.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.790+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fab1c071740 con 0x7fab1c155490 2024-03-06T07:32:53.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.826+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7ff98c06e6a0 con 0x7ff98c15e720 2024-03-06T07:32:53.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:53.830+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7feed0068070 con 0x7feed006d600 2024-03-06T07:32:54.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.725+0000 7f385be00640 1 -- 172.21.15.97:0/2076163672 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5566 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3850058450 con 0x7f38641664f0 2024-03-06T07:32:54.727 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 5566 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b500 msgr2=0x7f384406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b500 0x7f384406d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3854033000 tx=0x7f3854009030 comp rx=0 tx=0).stop 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 msgr2=0x7f3864166950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 0x7f3864166950 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f385000d800 tx=0x7f385000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 shutdown_connections 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b500 0x7f384406d9c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3864153f30 0x7f38641543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38641664f0 0x7f3864166950 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 --2- 172.21.15.97:0/2076163672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38480039e0 0x7f3864144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 >> 172.21.15.97:0/2076163672 conn(0x7f386406b360 msgr2=0x7f386406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 shutdown_connections 2024-03-06T07:32:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.729+0000 7f386b000640 1 -- 172.21.15.97:0/2076163672 wait complete. 2024-03-06T07:32:54.751 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:32:54.749+0000 7f68f6e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:54.755 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:54.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:54.758 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 5195 2024-03-06T07:32:54.759 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 7113 2024-03-06T07:32:54.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.755+0000 7feec6a00640 1 -- 172.21.15.97:0/4083490534 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7113 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7feec0040070 con 0x7feed006d600 2024-03-06T07:32:54.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.756+0000 7fab13400640 1 -- 172.21.15.97:0/1484713193 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5195 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fab0807c5d0 con 0x7fab1c155490 2024-03-06T07:32:54.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.756+0000 7ff983e00640 1 -- 172.21.15.97:0/957392462 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7107 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff98407b970 con 0x7ff98c15e720 2024-03-06T07:32:54.759 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 7107 2024-03-06T07:32:54.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faafc06b4d0 msgr2=0x7faafc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faafc06b4d0 0x7faafc06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fab0c001760 tx=0x7fab0c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:54.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 msgr2=0x7fab1c0b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 0x7fab1c0b0a90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fab080098f0 tx=0x7fab080046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb006b4d0 msgr2=0x7feeb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7feed5a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb006b4d0 0x7feeb006d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feebc000c00 tx=0x7feebc007890 comp rx=0 tx=0).stop 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 msgr2=0x7feed0162750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7feed5a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 0x7feed0162750 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7feec00098c0 tx=0x7feec00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 shutdown_connections 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faafc06b4d0 0x7faafc06d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab1c15ec10 0x7fab1c0b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab1c155490 0x7fab1c0b0a90 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 --2- 172.21.15.97:0/1484713193 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab1c0d0780 0x7fab1c13f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 >> 172.21.15.97:0/1484713193 conn(0x7fab1c06b360 msgr2=0x7fab1c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 shutdown_connections 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.761+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 shutdown_connections 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb006b4d0 0x7feeb006d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feed015ee30 0x7feed009d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feed00b3340 0x7feed0160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 --2- 172.21.15.97:0/4083490534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed006d600 0x7feed0162750 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 >> 172.21.15.97:0/4083490534 conn(0x7feed006b360 msgr2=0x7feed0158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7fab22600640 1 -- 172.21.15.97:0/1484713193 wait complete. 2024-03-06T07:32:54.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 shutdown_connections 2024-03-06T07:32:54.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7feed5a00640 1 -- 172.21.15.97:0/4083490534 wait complete. 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff96c06b4d0 msgr2=0x7ff96c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7ff992e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff96c06b4d0 0x7ff96c06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff98c1405f0 tx=0x7ff97c009190 comp rx=0 tx=0).stop 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 msgr2=0x7ff98c09aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.762+0000 7ff992e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c09aa30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ff984009f40 tx=0x7ff984004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 shutdown_connections 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff96c06b4d0 0x7ff96c06d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff98c15e720 0x7ff98c09aa30 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff98c0b3340 0x7ff98c09d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 --2- 172.21.15.97:0/957392462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff98c06d600 0x7ff98c09c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 >> 172.21.15.97:0/957392462 conn(0x7ff98c06b360 msgr2=0x7ff98c1461f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 shutdown_connections 2024-03-06T07:32:54.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:54.763+0000 7ff992e00640 1 -- 172.21.15.97:0/957392462 wait complete. 2024-03-06T07:32:54.768 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:54.768 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:54.784 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:54.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:54.788 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:54.789 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:54.793 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:54.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:54.794 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:54.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:54.839 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:54.840 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:54.845 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:54.845 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:54.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:54.926 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:32:54.952 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:32:54.990 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:32:54.991 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:32:55.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.023+0000 7f8292c00640 1 Processor -- start 2024-03-06T07:32:55.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.023+0000 7f8292c00640 1 -- start start 2024-03-06T07:32:55.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.025+0000 7f8292c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f828c15cc90 0x7f828c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.025+0000 7f8292c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c1635a0 con 0x7f828c162b00 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.025+0000 7f8292c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c163880 con 0x7f828c0d0780 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.025+0000 7f8292c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c06d940 con 0x7f828c15cc90 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.025+0000 7f8283e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f828c162b00 0x7f828c0d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53268/0 (socket says 172.21.15.97:53268) 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.025+0000 7f8283e00640 1 -- 172.21.15.97:0/3119225439 learned_addr learned my addr 172.21.15.97:0/3119225439 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828be00640 1 --2- 172.21.15.97:0/3119225439 >> v2:172.21.15.135:3300/0 conn(0x7f828c15cc90 0x7f828c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817342905 0 0) 0x7f828c1635a0 con 0x7f828c162b00 2024-03-06T07:32:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8270003520 con 0x7f828c162b00 2024-03-06T07:32:55.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841941461 0 0) 0x7f8270003520 con 0x7f828c162b00 2024-03-06T07:32:55.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 >> v2:172.21.15.135:3300/0 conn(0x7f828c15cc90 msgr2=0x7f828c06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 --2- 172.21.15.97:0/3119225439 >> v2:172.21.15.135:3300/0 conn(0x7f828c15cc90 0x7f828c06d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 >> v1:172.21.15.135:6789/0 conn(0x7f828c0d0780 legacy=0x7f828c0acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c06cc00 con 0x7f828c162b00 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8274001670 con 0x7f828c162b00 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 >> v1:172.21.15.97:6789/0 conn(0x7f828c162b00 legacy=0x7f828c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82700039b0 0x7f8270003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8270004800 0x7f8270004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82700051c0 0x7f827000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c06d940 con 0x7f8270004800 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827000c060 con 0x7f82700051c0 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827000c260 con 0x7f82700039b0 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.026+0000 7f8283e00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8270004800 0x7f8270004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8290c00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82700051c0 0x7f827000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f828be00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82700039b0 0x7f8270003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8283e00640 1 -- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82700039b0 msgr2=0x7f8270003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8283e00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82700039b0 0x7f8270003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8283e00640 1 -- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82700051c0 msgr2=0x7f827000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8283e00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82700051c0 0x7f827000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8283e00640 1 -- 172.21.15.97:0/3119225439 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270003520 con 0x7f8270004800 2024-03-06T07:32:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.027+0000 7f8290c00640 1 -- 172.21.15.97:0/3119225439 reap_dead start 2024-03-06T07:32:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.028+0000 7f8283e00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8270004800 0x7f8270004c80 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f827400c240 tx=0x7f8270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f969b3c0af8d8ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.028+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274007d10 con 0x7f8270004800 2024-03-06T07:32:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.028+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274030070 con 0x7f8270004800 2024-03-06T07:32:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.028+0000 7f828b400640 1 -- 172.21.15.97:0/3119225439 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827402cda0 con 0x7f8270004800 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 -- 172.21.15.97:0/3119225439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8270004800 msgr2=0x7f8270004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8270004800 0x7f8270004c80 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f827400c240 tx=0x7f8270005830 comp rx=0 tx=0).stop 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 -- 172.21.15.97:0/3119225439 shutdown_connections 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 --2- 172.21.15.97:0/3119225439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8270004800 0x7f8270004c80 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 -- 172.21.15.97:0/3119225439 >> 172.21.15.97:0/3119225439 conn(0x7f828c06b360 msgr2=0x7f828c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 -- 172.21.15.97:0/3119225439 shutdown_connections 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.029+0000 7f8292c00640 1 -- 172.21.15.97:0/3119225439 wait complete. 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.031+0000 7f8292c00640 1 Processor -- start 2024-03-06T07:32:55.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.031+0000 7f8292c00640 1 -- start start 2024-03-06T07:32:55.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8292c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8270004800 0x7f828c148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8292c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 0x7f828c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8292c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f828c158a60 0x7f828c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8292c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0b4fe0 con 0x7f828c156a70 2024-03-06T07:32:55.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8292c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0b4e60 con 0x7f8270004800 2024-03-06T07:32:55.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8292c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c06d400 con 0x7f828c158a60 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 0x7f828c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 0x7f828c156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39448/0 (socket says 172.21.15.97:39448) 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 -- 172.21.15.97:0/1588392435 learned_addr learned my addr 172.21.15.97:0/1588392435 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f828be00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8270004800 0x7f828c148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 -- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f828c158a60 msgr2=0x7f828c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f828c158a60 0x7f828c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 -- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8270004800 msgr2=0x7f828c148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8270004800 0x7f828c148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.032+0000 7f8283e00640 1 -- 172.21.15.97:0/1588392435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c159770 con 0x7f828c156a70 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.033+0000 7f8283e00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 0x7f828c156ef0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f827400c240 tx=0x7f827402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.033+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274007d10 con 0x7f828c156a70 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.033+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274004bf0 con 0x7f828c156a70 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.033+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274007070 con 0x7f828c156a70 2024-03-06T07:32:55.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.033+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828c159a50 con 0x7f828c156a70 2024-03-06T07:32:55.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.033+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c09aeb0 con 0x7f828c156a70 2024-03-06T07:32:55.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.035+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c068070 con 0x7f828c156a70 2024-03-06T07:32:55.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.035+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8274048020 con 0x7f828c156a70 2024-03-06T07:32:55.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.036+0000 7f8288c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826806b4d0 0x7f826806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.036+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f82740ab690 con 0x7f828c156a70 2024-03-06T07:32:55.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.036+0000 7f828be00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826806b4d0 0x7f826806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.037+0000 7f828be00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826806b4d0 0x7f826806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f827c0099d0 tx=0x7f827c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.041+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f827407a540 con 0x7f828c156a70 2024-03-06T07:32:55.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.261+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f828c06b7e0 con 0x7f828c156a70 2024-03-06T07:32:55.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.262+0000 7f8288c00640 1 -- 172.21.15.97:0/1588392435 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+1802 (secure 0 0 0) 0x7f8274079e40 con 0x7f828c156a70 2024-03-06T07:32:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:55.265 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:54.750052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:32:55.265 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 62 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826806b4d0 msgr2=0x7f826806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826806b4d0 0x7f826806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f827c0099d0 tx=0x7f827c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 msgr2=0x7f828c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 0x7f828c156ef0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f827400c240 tx=0x7f827402e070 comp rx=0 tx=0).stop 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 shutdown_connections 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826806b4d0 0x7f826806d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f828c158a60 0x7f828c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f828c156a70 0x7f828c156ef0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 --2- 172.21.15.97:0/1588392435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8270004800 0x7f828c148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 >> 172.21.15.97:0/1588392435 conn(0x7f828c06b360 msgr2=0x7f828c0acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 shutdown_connections 2024-03-06T07:32:55.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:55.265+0000 7f8292c00640 1 -- 172.21.15.97:0/1588392435 wait complete. 2024-03-06T07:32:55.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:43.068988+0000', 'modified': '2024-03-06T07:32:54.750052+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 100, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:32:55.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:32:56.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:56.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.898+0000 7fa2c7000640 1 Processor -- start 2024-03-06T07:32:56.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.898+0000 7fa2c7000640 1 -- start start 2024-03-06T07:32:56.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.899+0000 7fa2c7000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa2c01587c0 0x7fa2c006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.899+0000 7fa2c7000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa2c0071ef0 0x7fa2c006c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.899+0000 7fa2c7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c00ad8a0 0x7fa2c0161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.899+0000 7fa2c7000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c00cccd0 con 0x7fa2c01587c0 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.899+0000 7fa2c7000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c015ea60 con 0x7fa2c0071ef0 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.899+0000 7fa2c7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c015ed40 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa2c01587c0 0x7fa2c006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa2c01587c0 0x7fa2c006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39450/0 (socket says 172.21.15.97:39450) 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 -- 172.21.15.97:0/2680836384 learned_addr learned my addr 172.21.15.97:0/2680836384 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2c5000640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c00ad8a0 0x7fa2c0161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 -- 172.21.15.97:0/2680836384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c00ad8a0 msgr2=0x7fa2c0161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c00ad8a0 0x7fa2c0161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 -- 172.21.15.97:0/2680836384 >> v2:172.21.15.135:3300/0 conn(0x7fa2c0071ef0 msgr2=0x7fa2c006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 --2- 172.21.15.97:0/2680836384 >> v2:172.21.15.135:3300/0 conn(0x7fa2c0071ef0 0x7fa2c006c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2bfe00640 1 -- 172.21.15.97:0/2680836384 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c015f020 con 0x7fa2c01587c0 2024-03-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.900+0000 7fa2c5000640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c00ad8a0 0x7fa2c0161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:56.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.901+0000 7fa2bfe00640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c01587c0 0x7fa2c006bf50 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa2c006cd80 tx=0x7fa2b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1154ea5176af41f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2bea00640 1 -- 172.21.15.97:0/2680836384 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b002e070 con 0x7fa2c01587c0 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2bea00640 1 -- 172.21.15.97:0/2680836384 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b0037070 con 0x7fa2c01587c0 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2bea00640 1 -- 172.21.15.97:0/2680836384 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0031a00 con 0x7fa2c01587c0 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 -- 172.21.15.97:0/2680836384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c01587c0 msgr2=0x7fa2c006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c01587c0 0x7fa2c006bf50 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa2c006cd80 tx=0x7fa2b002b880 comp rx=0 tx=0).stop 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 -- 172.21.15.97:0/2680836384 shutdown_connections 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c00ad8a0 0x7fa2c0161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 --2- 172.21.15.97:0/2680836384 >> v2:172.21.15.135:3300/0 conn(0x7fa2c0071ef0 0x7fa2c006c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 --2- 172.21.15.97:0/2680836384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c01587c0 0x7fa2c006bf50 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 -- 172.21.15.97:0/2680836384 >> 172.21.15.97:0/2680836384 conn(0x7fa2c00713d0 msgr2=0x7fa2c00717e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 -- 172.21.15.97:0/2680836384 shutdown_connections 2024-03-06T07:32:56.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.902+0000 7fa2c7000640 1 -- 172.21.15.97:0/2680836384 wait complete. 2024-03-06T07:32:56.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.904+0000 7fa2c7000640 1 Processor -- start 2024-03-06T07:32:56.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.904+0000 7fa2c7000640 1 -- start start 2024-03-06T07:32:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2c7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c0071ef0 0x7fa2c014f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2c7000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 0x7fa2c00b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2c7000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c01587c0 0x7fa2c00b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2c7000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0158ba0 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2c7000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c015e750 con 0x7fa2c01587c0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2c7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c01622d0 con 0x7fa2c0071ef0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.905+0000 7fa2bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 0x7fa2c00b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 0x7fa2c00b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39458/0 (socket says 172.21.15.97:39458) 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 -- 172.21.15.97:0/752495431 learned_addr learned my addr 172.21.15.97:0/752495431 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bfe00640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c0071ef0 0x7fa2c014f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 -- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c0071ef0 msgr2=0x7fa2c014f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c0071ef0 0x7fa2c014f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 -- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c01587c0 msgr2=0x7fa2c00b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2c5000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c01587c0 0x7fa2c00b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c01587c0 0x7fa2c00b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 -- 172.21.15.97:0/752495431 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c006ec30 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.906+0000 7fa2bf400640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 0x7fa2c00b1040 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00b410 tx=0x7fa2ac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.907+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac015020 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.907+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac011850 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.907+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac011c40 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.907+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c006e390 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.907+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c006e950 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.908+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0074cc0 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.909+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2ac018020 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.910+0000 7fa2b7e00640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.910+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa2ac08a230 con 0x7fa2c00ad8a0 2024-03-06T07:32:56.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.910+0000 7fa2bfe00640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.911+0000 7fa2bfe00640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0036000 tx=0x7fa2b003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:56.915+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2ac01f020 con 0x7fa2c00ad8a0 2024-03-06T07:32:57.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.149+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2c0061fd0 con 0x7fa2c00ad8a0 2024-03-06T07:32:57.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.150+0000 7fa2b7e00640 1 -- 172.21.15.97:0/752495431 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fa2ac0172c0 con 0x7fa2c00ad8a0 2024-03-06T07:32:57.152 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:57.152 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":65,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:55.801684+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:32:57.152 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 65 2024-03-06T07:32:57.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2a006b4d0 msgr2=0x7fa2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0036000 tx=0x7fa2b003d040 comp rx=0 tx=0).stop 2024-03-06T07:32:57.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 msgr2=0x7fa2c00b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 0x7fa2c00b1040 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00b410 tx=0x7fa2ac00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 shutdown_connections 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c01587c0 0x7fa2c00b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c00ad8a0 0x7fa2c00b1040 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 --2- 172.21.15.97:0/752495431 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa2c0071ef0 0x7fa2c014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.152+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 >> 172.21.15.97:0/752495431 conn(0x7fa2c00713d0 msgr2=0x7fa2c00b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.153+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 shutdown_connections 2024-03-06T07:32:57.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:57.153+0000 7fa2c7000640 1 -- 172.21.15.97:0/752495431 wait complete. 2024-03-06T07:32:57.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:43.068988+0000', 'modified': '2024-03-06T07:32:55.801684+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 100, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7314}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7314': {'gid': 7314, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.97:6837/1028636781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 1028636781}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1028636781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:32:57.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7314:up:reconnect 2024-03-06T07:32:58.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:58.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.796+0000 7fc991600640 1 Processor -- start 2024-03-06T07:32:58.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.796+0000 7fc991600640 1 -- start start 2024-03-06T07:32:58.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.797+0000 7fc991600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc98c155490 0x7fc98c0ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.797+0000 7fc991600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc991600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc98c15ec10 0x7fc98c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc991600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0ad640 con 0x7fc98c15ec10 2024-03-06T07:32:58.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc991600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c15f980 con 0x7fc98c155490 2024-03-06T07:32:58.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc991600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c155870 con 0x7fc98c0d0070 2024-03-06T07:32:58.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc98b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc98c15ec10 0x7fc98c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc98b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc98c15ec10 0x7fc98c15f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39460/0 (socket says 172.21.15.97:39460) 2024-03-06T07:32:58.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc98b400640 1 -- 172.21.15.97:0/4019164289 learned_addr learned my addr 172.21.15.97:0/4019164289 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:58.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc98a000640 1 --2- 172.21.15.97:0/4019164289 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc98aa00640 1 --2- 172.21.15.97:0/4019164289 >> v2:172.21.15.135:3300/0 conn(0x7fc98c155490 0x7fc98c0ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.798+0000 7fc98b400640 1 -- 172.21.15.97:0/4019164289 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 msgr2=0x7fc98c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.799+0000 7fc98b400640 1 --2- 172.21.15.97:0/4019164289 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.799+0000 7fc98b400640 1 -- 172.21.15.97:0/4019164289 >> v2:172.21.15.135:3300/0 conn(0x7fc98c155490 msgr2=0x7fc98c0ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.799+0000 7fc98b400640 1 --2- 172.21.15.97:0/4019164289 >> v2:172.21.15.135:3300/0 conn(0x7fc98c155490 0x7fc98c0ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.799+0000 7fc98b400640 1 -- 172.21.15.97:0/4019164289 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0ad870 con 0x7fc98c15ec10 2024-03-06T07:32:58.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.799+0000 7fc98aa00640 1 --2- 172.21.15.97:0/4019164289 >> v2:172.21.15.135:3300/0 conn(0x7fc98c155490 0x7fc98c0ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.800+0000 7fc98b400640 1 --2- 172.21.15.97:0/4019164289 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c15f0a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fc98c06ed50 tx=0x7fc97c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c14719d14143e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.804+0000 7fc989600640 1 -- 172.21.15.97:0/4019164289 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c035020 con 0x7fc98c15ec10 2024-03-06T07:32:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.804+0000 7fc989600640 1 -- 172.21.15.97:0/4019164289 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c031880 con 0x7fc98c15ec10 2024-03-06T07:32:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.804+0000 7fc989600640 1 -- 172.21.15.97:0/4019164289 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c031c50 con 0x7fc98c15ec10 2024-03-06T07:32:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 -- 172.21.15.97:0/4019164289 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 msgr2=0x7fc98c15f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 --2- 172.21.15.97:0/4019164289 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c15f0a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fc98c06ed50 tx=0x7fc97c02b920 comp rx=0 tx=0).stop 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 -- 172.21.15.97:0/4019164289 shutdown_connections 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 --2- 172.21.15.97:0/4019164289 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c15f0a0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 --2- 172.21.15.97:0/4019164289 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 --2- 172.21.15.97:0/4019164289 >> v2:172.21.15.135:3300/0 conn(0x7fc98c155490 0x7fc98c0ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 -- 172.21.15.97:0/4019164289 >> 172.21.15.97:0/4019164289 conn(0x7fc98c06b360 msgr2=0x7fc98c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 -- 172.21.15.97:0/4019164289 shutdown_connections 2024-03-06T07:32:58.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.805+0000 7fc991600640 1 -- 172.21.15.97:0/4019164289 wait complete. 2024-03-06T07:32:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.807+0000 7fc991600640 1 Processor -- start 2024-03-06T07:32:58.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.807+0000 7fc991600640 1 -- start start 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.808+0000 7fc991600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.808+0000 7fc991600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc98c155490 0x7fc98c151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.808+0000 7fc991600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.808+0000 7fc991600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0ae110 con 0x7fc98c15ec10 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.808+0000 7fc991600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0ade30 con 0x7fc98c155490 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.808+0000 7fc991600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c15f5e0 con 0x7fc98c0d0070 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39476/0 (socket says 172.21.15.97:39476) 2024-03-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 -- 172.21.15.97:0/3172590924 learned_addr learned my addr 172.21.15.97:0/3172590924 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98aa00640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 -- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 msgr2=0x7fc98c0b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98a000640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc98c155490 0x7fc98c151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 -- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc98c155490 msgr2=0x7fc98c151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc98c155490 0x7fc98c151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98b400640 1 -- 172.21.15.97:0/3172590924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c155b50 con 0x7fc98c15ec10 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.809+0000 7fc98a000640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc98c155490 0x7fc98c151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.810+0000 7fc98b400640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c156670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0090b0 tx=0x7fc97c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.810+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c040070 con 0x7fc98c15ec10 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.810+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c030d10 con 0x7fc98c15ec10 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.810+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c035040 con 0x7fc98c15ec10 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.810+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c155e30 con 0x7fc98c15ec10 2024-03-06T07:32:58.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.810+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c0b26f0 con 0x7fc98c15ec10 2024-03-06T07:32:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.812+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c0b28f0 con 0x7fc98c15ec10 2024-03-06T07:32:58.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.812+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc97c04b060 con 0x7fc98c15ec10 2024-03-06T07:32:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.814+0000 7fc982a00640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc96806b4d0 0x7fc96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.814+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc97c0abf50 con 0x7fc98c15ec10 2024-03-06T07:32:58.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.814+0000 7fc98aa00640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc96806b4d0 0x7fc96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.815+0000 7fc98aa00640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc96806b4d0 0x7fc96806d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc9840075b0 tx=0x7fc984007540 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:58.818+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc98c0b28f0 con 0x7fc98c15ec10 2024-03-06T07:32:59.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.089+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc98c068070 con 0x7fc98c15ec10 2024-03-06T07:32:59.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.090+0000 7fc982a00640 1 -- 172.21.15.97:0/3172590924 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fc97c04b350 con 0x7fc98c15ec10 2024-03-06T07:32:59.092 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:32:59.093 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:32:59.093 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc96806b4d0 msgr2=0x7fc96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc96806b4d0 0x7fc96806d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc9840075b0 tx=0x7fc984007540 comp rx=0 tx=0).stop 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 msgr2=0x7fc98c156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c156670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0090b0 tx=0x7fc97c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 shutdown_connections 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc96806b4d0 0x7fc96806d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc98c15ec10 0x7fc98c156670 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc98c155490 0x7fc98c151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 --2- 172.21.15.97:0/3172590924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc98c0d0070 0x7fc98c0b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 >> 172.21.15.97:0/3172590924 conn(0x7fc98c06b360 msgr2=0x7fc98c158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 shutdown_connections 2024-03-06T07:32:59.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.093+0000 7fc991600640 1 -- 172.21.15.97:0/3172590924 wait complete. 2024-03-06T07:32:59.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:43.068988+0000', 'modified': '2024-03-06T07:32:58.790850+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 100, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7314}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7314': {'gid': 7314, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.97:6837/1028636781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 1028636781}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1028636781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7314, 'qdb_cluster': [7314]} 2024-03-06T07:32:59.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:59.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:32:59.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.727+0000 7f9299a00640 1 Processor -- start 2024-03-06T07:32:59.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.727+0000 7f9299a00640 1 -- start start 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.728+0000 7f9299a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f92940acd60 0x7f92940b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.728+0000 7f9299a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92940b5160 con 0x7f92940d0780 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9299a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92940ae910 con 0x7f929415be90 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9299a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92940b4f30 con 0x7f92940acd60 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9292000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f92940d0780 0x7f929415fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53274/0 (socket says 172.21.15.97:53274) 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9292000640 1 -- 172.21.15.97:0/1524662488 learned_addr learned my addr 172.21.15.97:0/1524662488 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:32:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9293400640 1 --2- 172.21.15.97:0/1524662488 >> v2:172.21.15.97:3300/0 conn(0x7f92940acd60 0x7f92940b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9291600640 1 -- 172.21.15.97:0/1524662488 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272737312 0 0) 0x7f92940b5160 con 0x7f92940d0780 2024-03-06T07:32:59.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9291600640 1 -- 172.21.15.97:0/1524662488 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9278003520 con 0x7f92940d0780 2024-03-06T07:32:59.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9293400640 1 -- 172.21.15.97:0/1524662488 >> v1:172.21.15.135:6789/0 conn(0x7f929415be90 legacy=0x7f92940d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9293400640 1 -- 172.21.15.97:0/1524662488 >> v1:172.21.15.97:6789/0 conn(0x7f92940d0780 legacy=0x7f929415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.729+0000 7f9293400640 1 -- 172.21.15.97:0/1524662488 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940aebf0 con 0x7f92940acd60 2024-03-06T07:32:59.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.731+0000 7f9293400640 1 --2- 172.21.15.97:0/1524662488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940acd60 0x7f92940b49f0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f928c009d90 tx=0x7f928c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eaed1da53ece061 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.731+0000 7f9291600640 1 -- 172.21.15.97:0/1524662488 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c039070 con 0x7f92940acd60 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.731+0000 7f9291600640 1 -- 172.21.15.97:0/1524662488 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c02cc60 con 0x7f92940acd60 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.731+0000 7f9291600640 1 -- 172.21.15.97:0/1524662488 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c034660 con 0x7f92940acd60 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.731+0000 7f9299a00640 1 -- 172.21.15.97:0/1524662488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940acd60 msgr2=0x7f92940b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.731+0000 7f9299a00640 1 --2- 172.21.15.97:0/1524662488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940acd60 0x7f92940b49f0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f928c009d90 tx=0x7f928c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.732+0000 7f9299a00640 1 -- 172.21.15.97:0/1524662488 shutdown_connections 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.732+0000 7f9299a00640 1 --2- 172.21.15.97:0/1524662488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940acd60 0x7f92940b49f0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.732+0000 7f9299a00640 1 -- 172.21.15.97:0/1524662488 >> 172.21.15.97:0/1524662488 conn(0x7f929406b360 msgr2=0x7f929406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.732+0000 7f9299a00640 1 -- 172.21.15.97:0/1524662488 shutdown_connections 2024-03-06T07:32:59.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.732+0000 7f9299a00640 1 -- 172.21.15.97:0/1524662488 wait complete. 2024-03-06T07:32:59.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.733+0000 7f9299a00640 1 Processor -- start 2024-03-06T07:32:59.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.733+0000 7f9299a00640 1 -- start start 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.734+0000 7f9299a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92940acd60 0x7f92940cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.734+0000 7f9299a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 0x7f92940afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.734+0000 7f9299a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f929415be90 0x7f929414f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.734+0000 7f9299a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92940b2c80 con 0x7f92940d0780 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.734+0000 7f9299a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92940af240 con 0x7f92940acd60 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.734+0000 7f9299a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92940b2e00 con 0x7f929415be90 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 0x7f92940afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 0x7f92940afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39504/0 (socket says 172.21.15.97:39504) 2024-03-06T07:32:59.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 -- 172.21.15.97:0/2469265078 learned_addr learned my addr 172.21.15.97:0/2469265078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 -- 172.21.15.97:0/2469265078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f929415be90 msgr2=0x7f929414f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f929415be90 0x7f929414f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 -- 172.21.15.97:0/2469265078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92940acd60 msgr2=0x7f92940cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92940acd60 0x7f92940cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 -- 172.21.15.97:0/2469265078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940cccd0 con 0x7f92940d0780 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.735+0000 7f9292000640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 0x7f92940afb10 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f928000c500 tx=0x7f928000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.736+0000 7f928aa00640 1 -- 172.21.15.97:0/2469265078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280017070 con 0x7f92940d0780 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.736+0000 7f928aa00640 1 -- 172.21.15.97:0/2469265078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9280005ce0 con 0x7f92940d0780 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.736+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929415d040 con 0x7f92940d0780 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.736+0000 7f928aa00640 1 -- 172.21.15.97:0/2469265078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280007330 con 0x7f92940d0780 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.736+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929415d930 con 0x7f92940d0780 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.738+0000 7f928aa00640 1 -- 172.21.15.97:0/2469265078 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f928001a070 con 0x7f92940d0780 2024-03-06T07:32:59.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.740+0000 7f928aa00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f927406b4d0 0x7f927406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.740+0000 7f929c60e740 1 Processor -- start 2024-03-06T07:32:59.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.740+0000 7f928aa00640 1 -- 172.21.15.97:0/2469265078 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9280089670 con 0x7f92940d0780 2024-03-06T07:32:59.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.740+0000 7f929c60e740 1 -- start start 2024-03-06T07:32:59.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.740+0000 7f9292a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f927406b4d0 0x7f927406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f927406b4d0 0x7f927406d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9284000f60 tx=0x7f92840079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f929c60e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 0x55bc86bde750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c0e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86bdec90 0x55bc86c0ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f929c60e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86b48600 con 0x55bc86c0b7b0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86af1930 con 0x55bc86c0bb90 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc86b446a0 con 0x55bc86bdec90 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 0x55bc86bde750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 0x55bc86bde750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39508/0 (socket says 172.21.15.97:39508) 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 -- 172.21.15.97:0/1807173554 learned_addr learned my addr 172.21.15.97:0/1807173554 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292000640 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c0e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 -- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86bdec90 msgr2=0x55bc86c0ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86bdec90 0x55bc86c0ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 -- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 msgr2=0x55bc86c0e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c0e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.741+0000 7f9292a00640 1 -- 172.21.15.97:0/1807173554 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc8685d2d0 con 0x55bc86c0b7b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.742+0000 7f9292a00640 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 0x55bc86bde750 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f9284016870 tx=0x7f928400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d16f5fda8857b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.742+0000 7f927f400640 1 -- 172.21.15.97:0/1807173554 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840162e0 con 0x55bc86c0b7b0 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.742+0000 7f927f400640 1 -- 172.21.15.97:0/1807173554 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9284005110 con 0x55bc86c0b7b0 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.742+0000 7f927f400640 1 -- 172.21.15.97:0/1807173554 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840138b0 con 0x55bc86c0b7b0 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 -- 172.21.15.97:0/1807173554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 msgr2=0x55bc86bde750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 0x55bc86bde750 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f9284016870 tx=0x7f928400f000 comp rx=0 tx=0).stop 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 -- 172.21.15.97:0/1807173554 shutdown_connections 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86bdec90 0x55bc86c0ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c0e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 --2- 172.21.15.97:0/1807173554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0b7b0 0x55bc86bde750 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 -- 172.21.15.97:0/1807173554 >> 172.21.15.97:0/1807173554 conn(0x55bc86be09c0 msgr2=0x55bc86c29540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 -- 172.21.15.97:0/1807173554 shutdown_connections 2024-03-06T07:32:59.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.743+0000 7f929c60e740 1 -- 172.21.15.97:0/1807173554 wait complete. 2024-03-06T07:32:59.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.744+0000 7f929c60e740 1 -- start start 2024-03-06T07:32:59.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.744+0000 7f929c60e740 1 Processor -- start 2024-03-06T07:32:59.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c0b7b0 0x55bc86c10a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c23f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 0x55bc86bfc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86be2f60 con 0x55bc86c3a690 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86b04d90 con 0x55bc86c0bb90 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc86be2350 con 0x55bc86c0b7b0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f929c60e740 10 client.0 _ll_register_callbacks cb 0x55bc86b40350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f9293400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 0x55bc86bfc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f9293400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 0x55bc86bfc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39520/0 (socket says 172.21.15.97:39520) 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.746+0000 7f9293400640 1 -- 172.21.15.97:0/45747064 learned_addr learned my addr 172.21.15.97:0/45747064 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9292a00640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c0b7b0 0x55bc86c10a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9292000640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c23f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9293400640 1 -- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c0b7b0 msgr2=0x55bc86c10a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9293400640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c0b7b0 0x55bc86c10a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9293400640 1 -- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 msgr2=0x55bc86c23f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9293400640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c23f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9293400640 1 -- 172.21.15.97:0/45747064 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc869276b0 con 0x55bc86c3a690 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9293400640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 0x55bc86bfc900 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f928c009eb0 tx=0x7f928c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45a5a6a3e65166a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9273e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9273e00640 1 -- 172.21.15.97:0/45747064 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c039050 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f9273e00640 1 -- 172.21.15.97:0/45747064 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f928c034bf0 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.747+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9284005110 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f9273e00640 1 -- 172.21.15.97:0/45747064 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f928c04d050 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 10 client.7347 fetch_fsmap learned FSMap version 67 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55bc86c11900 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f9273e00640 1 -- 172.21.15.97:0/45747064 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9284005110 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f9273e00640 1 -- 172.21.15.97:0/45747064 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f928c050020 con 0x55bc86c3a690 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 10 client.7347 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 10 client.7347 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 20 client.7347 populate_metadata read hostname 'smithi097' 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c21d20 0x55bc86c3d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 4 client.7347 mds_command: new command op to 7314 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:59.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 --> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc86b55b40 con 0x55bc86c21d20 2024-03-06T07:32:59.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.748+0000 7f9292a00640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c21d20 0x55bc86c3d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.749+0000 7f9292a00640 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c21d20 0x55bc86c3d630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=761552ff0ba7919 server_cookie=5bee91fc27abcd63 in_seq=0 out_seq=0 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.749+0000 7f9273e00640 10 client.7347 ms_handle_connect on v2:172.21.15.97:6835/1028636781 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f9273e00640 1 -- 172.21.15.97:0/45747064 <== mds.0 v2:172.21.15.97:6835/1028636781 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bc86b55b40 con 0x55bc86c21d20 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f9273e00640 10 client.7347 handle_command_reply: tid=0 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 client.7347 shutdown 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 msgr2=0x55bc86bfc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 0x55bc86bfc900 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f928c009eb0 tx=0x7f928c034670 comp rx=0 tx=0).stop 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 shutdown_connections 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c21d20 0x55bc86c3d630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c3a690 0x55bc86bfc900 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0bb90 0x55bc86c23f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 --2- 172.21.15.97:0/45747064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c0b7b0 0x55bc86c10a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 >> 172.21.15.97:0/45747064 conn(0x55bc86bdec90 msgr2=0x55bc86c2ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.750+0000 7f9273e00640 0 client.7347 ms_handle_reset on v2:172.21.15.97:6835/1028636781 2024-03-06T07:32:59.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.751+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 shutdown_connections 2024-03-06T07:32:59.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.751+0000 7f929c60e740 1 -- 172.21.15.97:0/45747064 wait complete. 2024-03-06T07:32:59.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.751+0000 7f929c60e740 20 client.7347 trim_cache size 0 max 16384 2024-03-06T07:32:59.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.769+0000 7f929c60e740 1 Processor -- start 2024-03-06T07:32:59.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.769+0000 7f929c60e740 1 -- start start 2024-03-06T07:32:59.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c43b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f929c60e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c39c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86c20be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f929c60e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9294200540 con 0x55bc86c0bb90 2024-03-06T07:32:59.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92941ff780 con 0x55bc86c0b7b0 2024-03-06T07:32:59.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92940ceff0 con 0x55bc86c3a690 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c39c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c39c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39528/0 (socket says 172.21.15.97:39528) 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 -- 172.21.15.97:0/3220291052 learned_addr learned my addr 172.21.15.97:0/3220291052 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292a00640 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c43b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9293400640 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86c20be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 -- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 msgr2=0x55bc86c20be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86c20be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 -- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 msgr2=0x55bc86c43b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c43b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.770+0000 7f9292000640 1 -- 172.21.15.97:0/3220291052 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc86be4a00 con 0x55bc86c0bb90 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.771+0000 7f9292000640 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c39c70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f928000cec0 tx=0x7f9280001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabc936859f3d208 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.771+0000 7f927ea00640 1 -- 172.21.15.97:0/3220291052 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280086070 con 0x55bc86c0bb90 2024-03-06T07:32:59.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.771+0000 7f927ea00640 1 -- 172.21.15.97:0/3220291052 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800544b0 con 0x55bc86c0bb90 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.771+0000 7f927ea00640 1 -- 172.21.15.97:0/3220291052 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928005a070 con 0x55bc86c0bb90 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.771+0000 7f929c60e740 1 -- 172.21.15.97:0/3220291052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 msgr2=0x55bc86c39c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.771+0000 7f929c60e740 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c39c70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f928000cec0 tx=0x7f9280001150 comp rx=0 tx=0).stop 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 -- 172.21.15.97:0/3220291052 shutdown_connections 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86c20be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c39c70 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 --2- 172.21.15.97:0/3220291052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c43b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 -- 172.21.15.97:0/3220291052 >> 172.21.15.97:0/3220291052 conn(0x55bc86bdec90 msgr2=0x55bc86be5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 -- 172.21.15.97:0/3220291052 shutdown_connections 2024-03-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 -- 172.21.15.97:0/3220291052 wait complete. 2024-03-06T07:32:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.772+0000 7f929c60e740 1 -- start start 2024-03-06T07:32:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 Processor -- start 2024-03-06T07:32:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c98080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c985c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86be22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86c227c0 con 0x55bc86c0bb90 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929415edf0 con 0x55bc86c0b7b0 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.773+0000 7f929c60e740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc86c221a0 con 0x55bc86c3a690 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f929c60e740 10 client.0 _ll_register_callbacks cb 0x7f92940704e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39538/0 (socket says 172.21.15.97:39538) 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 -- 172.21.15.97:0/263695982 learned_addr learned my addr 172.21.15.97:0/263695982 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9293400640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86be22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 -- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 msgr2=0x55bc86be22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292a00640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c98080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86be22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 -- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 msgr2=0x55bc86c98080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c98080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292000640 1 -- 172.21.15.97:0/263695982 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc86c243f0 con 0x55bc86c0bb90 2024-03-06T07:32:59.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9293400640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86be22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.774+0000 7f9292a00640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c98080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9292000640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c985c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f928000cec0 tx=0x7f9280000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0caf6a976e6cc08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9273e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9273e00640 1 -- 172.21.15.97:0/263695982 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280073070 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9273e00640 1 -- 172.21.15.97:0/263695982 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f92800602c0 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92800544b0 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9273e00640 1 -- 172.21.15.97:0/263695982 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9280086040 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 10 client.7353 fetch_fsmap learned FSMap version 67 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55bc86c43bd0 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9273e00640 1 -- 172.21.15.97:0/263695982 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800544b0 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f9273e00640 1 -- 172.21.15.97:0/263695982 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f92800074d0 con 0x55bc86c0bb90 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 10 client.7353 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 10 client.7353 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 20 client.7353 populate_metadata read hostname 'smithi097' 2024-03-06T07:32:59.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c4cc30 0x55bc86c4d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 4 client.7353 mds_command: new command op to 7314 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.775+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 --> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bc86b4fe90 con 0x55bc86c4cc30 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.776+0000 7f9292a00640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c4cc30 0x55bc86c4d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.776+0000 7f9292a00640 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c4cc30 0x55bc86c4d010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78a0139cc851081 server_cookie=49b517707958aa39 in_seq=0 out_seq=0 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.776+0000 7f9273e00640 10 client.7353 ms_handle_connect on v2:172.21.15.97:6835/1028636781 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.776+0000 7f9273e00640 1 -- 172.21.15.97:0/263695982 <== mds.0 v2:172.21.15.97:6835/1028636781 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55bc86b4fe90 con 0x55bc86c4cc30 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.776+0000 7f9273e00640 10 client.7353 handle_command_reply: tid=0 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.776+0000 7f929c60e740 1 client.7353 shutdown 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 msgr2=0x55bc86c985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c985c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f928000cec0 tx=0x7f9280000ed0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 shutdown_connections 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:6835/1028636781,v1:172.21.15.97:6837/1028636781] conn(0x55bc86c4cc30 0x55bc86c4d010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55bc86c3a690 0x55bc86be22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bc86c0bb90 0x55bc86c985c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 --2- 172.21.15.97:0/263695982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bc86c0b7b0 0x55bc86c98080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 >> 172.21.15.97:0/263695982 conn(0x55bc86bdec90 msgr2=0x55bc86bd99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f9273e00640 0 client.7353 ms_handle_reset on v2:172.21.15.97:6835/1028636781 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 shutdown_connections 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 1 -- 172.21.15.97:0/263695982 wait complete. 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.777+0000 7f929c60e740 20 client.7353 trim_cache size 0 max 16384 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 7314, 2024-03-06T07:32:59.783 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 3.9962462580000002, 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 67, 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 100, 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 100, 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 4.7838517100000004 2024-03-06T07:32:59.784 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f927406b4d0 msgr2=0x7f927406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f927406b4d0 0x7f927406d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9284000f60 tx=0x7f92840079d0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 msgr2=0x7f92940afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 0x7f92940afb10 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f928000c500 tx=0x7f928000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 shutdown_connections 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f927406b4d0 0x7f927406d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f929415be90 0x7f929414f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92940d0780 0x7f92940afb10 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 --2- 172.21.15.97:0/2469265078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92940acd60 0x7f92940cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 >> 172.21.15.97:0/2469265078 conn(0x7f929406b360 msgr2=0x7f92940b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 shutdown_connections 2024-03-06T07:32:59.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:32:59.778+0000 7f9299a00640 1 -- 172.21.15.97:0/2469265078 wait complete. 2024-03-06T07:32:59.798 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:32:59.798 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:32:59.798 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:32:59.798 DEBUG:teuthology.orchestra.run.smithi097:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:32:59.798 DEBUG:teuthology.orchestra.run.smithi097:> subdir = "delete_me" 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> size_unit = 1024 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> file_multiplier = 200 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> for i in range(0, file_multiplier): 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> f.write(size * '"'"'x'"'"') 2024-03-06T07:32:59.799 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:33:01.780 INFO:tasks.ceph.mds.c.smithi097.stderr:2024-03-06T07:33:01.778+0000 7fdc29c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:05.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:33:05.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_ops 2024-03-06T07:33:05.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:05.761 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:33:06.028 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:33:06.029 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:33:06.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_files 2024-03-06T07:33:06.267 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:33:06.281 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:33:06.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:06.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.493+0000 7f87abc00640 1 Processor -- start 2024-03-06T07:33:06.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.493+0000 7f87abc00640 1 -- start start 2024-03-06T07:33:06.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.495+0000 7f87abc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 0x7f87a40ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.495+0000 7f87abc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a4155870 con 0x7f87a40d0070 2024-03-06T07:33:06.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.495+0000 7f87abc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a40ad870 con 0x7f87a415f320 2024-03-06T07:33:06.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.495+0000 7f87abc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a40ad640 con 0x7f87a4155490 2024-03-06T07:33:06.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.495+0000 7f87a3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f87a40d0070 0x7f87a415ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51678/0 (socket says 172.21.15.97:51678) 2024-03-06T07:33:06.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.495+0000 7f87a3e00640 1 -- 172.21.15.97:0/773862670 learned_addr learned my addr 172.21.15.97:0/773862670 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:06.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a9200640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 0x7f87a40ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009061392 0 0) 0x7f87a4155870 con 0x7f87a40d0070 2024-03-06T07:33:06.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878c005730 con 0x7f87a40d0070 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2131950717 0 0) 0x7f878c005730 con 0x7f87a40d0070 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 msgr2=0x7f87a40ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 0x7f87a40ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 >> v1:172.21.15.135:6789/0 conn(0x7f87a415f320 legacy=0x7f87a4071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a40adb50 con 0x7f87a40d0070 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8798001670 con 0x7f87a40d0070 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 >> v1:172.21.15.97:6789/0 conn(0x7f87a40d0070 legacy=0x7f87a415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 0x7f878c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f878c006170 0x7f878c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.496+0000 7f87a3400640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006f50 0x7f878c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c00dc80 con 0x7f878c006170 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c00de30 con 0x7f878c006f50 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c00e030 con 0x7f87a4155490 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3e00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f878c006170 0x7f878c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a9200640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 0x7f878c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a9c00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006f50 0x7f878c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3e00640 1 -- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 msgr2=0x7f878c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3e00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a4155490 0x7f878c005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3e00640 1 -- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006f50 msgr2=0x7f878c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3e00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006f50 0x7f878c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.497+0000 7f87a3e00640 1 -- 172.21.15.97:0/773862670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a40adb50 con 0x7f878c006170 2024-03-06T07:33:06.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.498+0000 7f87a9c00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006f50 0x7f878c0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:06.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.499+0000 7f87a3e00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f878c006170 0x7f878c006a10 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8798001050 tx=0x7f878c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6dcb95e0eac1c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.499+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798007d10 con 0x7f878c006170 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.499+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8798030070 con 0x7f878c006170 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.499+0000 7f87a3400640 1 -- 172.21.15.97:0/773862670 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879802cd80 con 0x7f878c006170 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87abc00640 1 -- 172.21.15.97:0/773862670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f878c006170 msgr2=0x7f878c006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87abc00640 1 --2- 172.21.15.97:0/773862670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f878c006170 0x7f878c006a10 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8798001050 tx=0x7f878c007a40 comp rx=0 tx=0).stop 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87a9c00640 1 -- 172.21.15.97:0/773862670 reap_dead start 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87abc00640 1 -- 172.21.15.97:0/773862670 shutdown_connections 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87abc00640 1 -- 172.21.15.97:0/773862670 >> 172.21.15.97:0/773862670 conn(0x7f87a406b360 msgr2=0x7f87a406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87abc00640 1 -- 172.21.15.97:0/773862670 shutdown_connections 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.500+0000 7f87abc00640 1 -- 172.21.15.97:0/773862670 wait complete. 2024-03-06T07:33:06.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.503+0000 7f87abc00640 1 Processor -- start 2024-03-06T07:33:06.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.503+0000 7f87abc00640 1 -- start start 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.504+0000 7f87abc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006170 0x7f87a4155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87abc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a41545f0 0x7f87a41f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87abc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 0x7f87a41f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87abc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a40ae110 con 0x7f87a41562d0 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87abc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a415f980 con 0x7f878c006170 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87abc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a40ae630 con 0x7f87a41545f0 2024-03-06T07:33:06.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 0x7f87a41f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 0x7f87a41f7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48512/0 (socket says 172.21.15.97:48512) 2024-03-06T07:33:06.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a9c00640 1 -- 172.21.15.97:0/1264541520 learned_addr learned my addr 172.21.15.97:0/1264541520 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:06.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a9200640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006170 0x7f87a4155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a3e00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a41545f0 0x7f87a41f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a9c00640 1 -- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a41545f0 msgr2=0x7f87a41f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.505+0000 7f87a9c00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a41545f0 0x7f87a41f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a9c00640 1 -- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006170 msgr2=0x7f87a4155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a9c00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006170 0x7f87a4155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a9c00640 1 -- 172.21.15.97:0/1264541520 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a4154a50 con 0x7f87a41562d0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a9200640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006170 0x7f87a4155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a9c00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 0x7f87a41f7290 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f87a4158fc0 tx=0x7f879c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c006d50 con 0x7f87a41562d0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879c003970 con 0x7f87a41562d0 2024-03-06T07:33:06.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a4155f30 con 0x7f87a41562d0 2024-03-06T07:33:06.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c003d60 con 0x7f87a41562d0 2024-03-06T07:33:06.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.506+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a415ca20 con 0x7f87a41562d0 2024-03-06T07:33:06.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.508+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a406bca0 con 0x7f87a41562d0 2024-03-06T07:33:06.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.510+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f879c005070 con 0x7f87a41562d0 2024-03-06T07:33:06.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.511+0000 7f87a0c00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f878806b500 0x7f878806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.511+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f879c089fa0 con 0x7f87a41562d0 2024-03-06T07:33:06.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.511+0000 7f87a9200640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f878806b500 0x7f878806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.512+0000 7f87a9200640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f878806b500 0x7f878806d9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8794009ad0 tx=0x7f8794008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.520+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87a406bca0 con 0x7f87a41562d0 2024-03-06T07:33:06.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.774+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a41f52f0 con 0x7f87a41562d0 2024-03-06T07:33:06.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.774+0000 7f87a0c00640 1 -- 172.21.15.97:0/1264541520 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f879c058db0 con 0x7f87a41562d0 2024-03-06T07:33:06.777 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:06.777 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:06.777 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f878806b500 msgr2=0x7f878806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f878806b500 0x7f878806d9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8794009ad0 tx=0x7f8794008040 comp rx=0 tx=0).stop 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 msgr2=0x7f87a41f7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 0x7f87a41f7290 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f87a4158fc0 tx=0x7f879c00dc00 comp rx=0 tx=0).stop 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 shutdown_connections 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f878806b500 0x7f878806d9c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41562d0 0x7f87a41f7290 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f87a41545f0 0x7f87a41f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 --2- 172.21.15.97:0/1264541520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f878c006170 0x7f87a4155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.777+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 >> 172.21.15.97:0/1264541520 conn(0x7f87a406b360 msgr2=0x7f87a40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.778+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 shutdown_connections 2024-03-06T07:33:06.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:06.778+0000 7f87abc00640 1 -- 172.21.15.97:0/1264541520 wait complete. 2024-03-06T07:33:06.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:07.019 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":35,"msgr_send_messages":38,"msgr_recv_bytes":12056,"msgr_send_bytes":6008606,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.045643099,"msgr_running_send_time":0.024545573,"msgr_running_recv_time":0.007567114,"msgr_running_fast_dispatch_time":0.010120520,"msgr_send_messages_queue_lat":{"avgcount":37,"sum":0.014731006,"avgtime":0.000398135},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92720},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7123,"msgr_send_messages":13047,"msgr_recv_bytes":2570981,"msgr_send_bytes":8095952,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.543879405,"msgr_running_send_time":1.032704781,"msgr_running_recv_time":0.976165420,"msgr_running_fast_dispatch_time":0.003763178,"msgr_send_messages_queue_lat":{"avgcount":13046,"sum":0.773019114,"avgtime":0.000059253},"msgr_handle_ack_lat":{"avgcount":15582,"sum":0.008931365,"avgtime":0.000000573},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":35,"msgr_send_messages":31,"msgr_recv_bytes":113926,"msgr_send_bytes":4206174,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.027938871,"msgr_running_send_time":0.016404297,"msgr_running_recv_time":0.009689157,"msgr_running_fast_dispatch_time":0.004250567,"msgr_send_messages_queue_lat":{"avgcount":31,"sum":0.012758449,"avgtime":0.000411562},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105920,"msgr_send_encrypted_bytes":6048},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":72,"sum":4.437443479,"avgtime":0.061631159}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000201482,"avgtime":0.000050370}},"mds":{"request":3622,"reply":3621,"reply_latency":{"avgcount":3621,"sum":784.911639606,"avgtime":0.216766539},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3624,"traverse_hit":3624,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3200,"root_rbytes":24550400,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3204,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3467,"handle_client_caps_dirty":2998,"handle_client_cap_release":0,"process_request_cap_release":421},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":413,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":413,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7143,"evex":0,"evtrm":0,"ev":7148,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":9,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":19575831,"rdpos":4195674,"jlat":{"avgcount":13,"sum":1.901862536,"avgtime":0.146297118},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3623,"dn+":3623,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":93504,"heap":248164},"mds_server":{"dispatch_client_request":3623,"dispatch_server_request":0,"handle_client_request":3622,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":412,"sum":0.468255813,"avgtime":0.001136543},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3445,"average_load":3445,"avg_session_uptime":934,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":6132499,"buffer_anon_items":5530,"buffer_meta_bytes":45672,"buffer_meta_items":519,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12315590,"mds_co_items":126206,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":66,"op_send_bytes":14270830,"op_resend":0,"op_reply":66,"op_latency":{"avgcount":66,"sum":0.522991081,"avgtime":0.007924107},"op_inflight":0,"oplen_avg":{"avgcount":66,"sum":119},"op":66,"op_r":36,"op_w":30,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":13,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":13,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":7189,"get_sum":2047335,"get_or_fail_fail":0,"get_or_fail_success":7189,"take":0,"take_sum":0,"put":7188,"put_sum":2047083,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":14272751,"put":66,"put_sum":14272751,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":66,"put":66,"put_sum":66,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":1442314,"max":3758096384,"get_started":0,"get":7143,"get_sum":15379342,"get_or_fail_fail":0,"get_or_fail_success":7143,"take":0,"take_sum":0,"put":13,"put_sum":13937028,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:07.021 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 12056, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 35, "msgr_running_fast_dispatch_time": 0.01012052, "msgr_running_recv_time": 0.007567114, "msgr_running_send_time": 0.024545573, "msgr_running_total_time": 0.045643099, "msgr_send_bytes": 6008606, "msgr_send_encrypted_bytes": 92720, "msgr_send_messages": 38, "msgr_send_messages_queue_lat": { "avgcount": 37, "avgtime": 0.000398135, "sum": 0.014731006 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 15582, "avgtime": 5.73e-07, "sum": 0.008931365 }, "msgr_recv_bytes": 2570981, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7123, "msgr_running_fast_dispatch_time": 0.003763178, "msgr_running_recv_time": 0.97616542, "msgr_running_send_time": 1.032704781, "msgr_running_total_time": 1.543879405, "msgr_send_bytes": 8095952, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 13047, "msgr_send_messages_queue_lat": { "avgcount": 13046, "avgtime": 5.9253e-05, "sum": 0.773019114 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 113926, "msgr_recv_encrypted_bytes": 105920, "msgr_recv_messages": 35, "msgr_running_fast_dispatch_time": 0.004250567, "msgr_running_recv_time": 0.009689157, "msgr_running_send_time": 0.016404297, "msgr_running_total_time": 0.027938871, "msgr_send_bytes": 4206174, "msgr_send_encrypted_bytes": 6048, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 31, "avgtime": 0.000411562, "sum": 0.012758449 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 72, "avgtime": 0.061631159, "sum": 4.437443479 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 5.037e-05, "sum": 0.000201482 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3204, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3467, "handle_client_caps_dirty": 2998, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 421, "q": 0, "reply": 3621, "reply_latency": { "avgcount": 3621, "avgtime": 0.216766539, "sum": 784.911639606 }, "request": 3622, "root_rbytes": 24550400, "root_rfiles": 3200, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3624, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3624, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 413, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 413, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7148, "evadd": 7143, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 13, "avgtime": 0.146297118, "sum": 1.901862536 }, "rdpos": 4195674, "replayed": 5, "seg": 9, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 19575831 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3623, "dn+": 3623, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 93504 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3623, "dispatch_server_request": 0, "handle_client_request": 3622, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 412, "avgtime": 0.001136543, "sum": 0.468255813 } }, "mds_sessions": { "average_load": 3445, "avg_session_uptime": 934, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3445 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 6132499, "buffer_anon_items": 5530, "buffer_meta_bytes": 45672, "buffer_meta_items": 519, "mds_co_bytes": 12315590, "mds_co_items": 126206, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 13, "op": 66, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 66, "avgtime": 0.007924107, "sum": 0.522991081 }, "op_pg": 0, "op_r": 36, "op_reply": 66, "op_resend": 0, "op_rmw": 0, "op_send": 66, "op_send_bytes": 14270830, "op_w": 30, "oplen_avg": { "avgcount": 66, "sum": 119 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 13, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7189, "get_or_fail_fail": 0, "get_or_fail_success": 7189, "get_started": 0, "get_sum": 2047335, "max": 104857600, "put": 7188, "put_sum": 2047083, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 66, "put_sum": 14272751, "take": 66, "take_sum": 14272751, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 66, "put_sum": 66, "take": 66, "take_sum": 66, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 7143, "get_or_fail_fail": 0, "get_or_fail_success": 7143, "get_started": 0, "get_sum": 15379342, "max": 3758096384, "put": 13, "put_sum": 13937028, "take": 0, "take_sum": 0, "val": 1442314, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:07.021 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/413, 0/413 2024-03-06T07:33:08.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:08.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:08.152 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:08.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.259+0000 7f4923200640 1 Processor -- start 2024-03-06T07:33:08.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.260+0000 7f4923200640 1 -- start start 2024-03-06T07:33:08.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4923200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f492415e230 0x7f49240d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4923200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49240d13a0 0x7f49240d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4923200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240b39d0 con 0x7f49240d2940 2024-03-06T07:33:08.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4923200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49240b3780 con 0x7f49240d13a0 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4923200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49240b3cb0 con 0x7f492415e230 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4921400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49240d13a0 0x7f49240d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4921400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49240d13a0 0x7f49240d2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48514/0 (socket says 172.21.15.97:48514) 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4921400640 1 -- 172.21.15.97:0/470564922 learned_addr learned my addr 172.21.15.97:0/470564922 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.261+0000 7f4921e00640 1 --2- 172.21.15.97:0/470564922 >> v2:172.21.15.135:3300/0 conn(0x7f492415e230 0x7f49240d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.262+0000 7f4921400640 1 -- 172.21.15.97:0/470564922 >> v2:172.21.15.135:3300/0 conn(0x7f492415e230 msgr2=0x7f49240d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.262+0000 7f4921400640 1 --2- 172.21.15.97:0/470564922 >> v2:172.21.15.135:3300/0 conn(0x7f492415e230 0x7f49240d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.262+0000 7f4921400640 1 -- 172.21.15.97:0/470564922 >> v1:172.21.15.135:6789/0 conn(0x7f49240d2940 legacy=0x7f49240ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.262+0000 7f4921400640 1 -- 172.21.15.97:0/470564922 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49240b3f90 con 0x7f49240d13a0 2024-03-06T07:33:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.263+0000 7f4921400640 1 --2- 172.21.15.97:0/470564922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49240d13a0 0x7f49240d2400 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f4924073430 tx=0x7f491002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b993abb63594aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.263+0000 7f4920a00640 1 -- 172.21.15.97:0/470564922 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491002f850 con 0x7f49240d13a0 2024-03-06T07:33:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.263+0000 7f4920a00640 1 -- 172.21.15.97:0/470564922 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491002f9f0 con 0x7f49240d13a0 2024-03-06T07:33:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.263+0000 7f4920a00640 1 -- 172.21.15.97:0/470564922 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491002fdc0 con 0x7f49240d13a0 2024-03-06T07:33:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 -- 172.21.15.97:0/470564922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49240d13a0 msgr2=0x7f49240d2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 --2- 172.21.15.97:0/470564922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49240d13a0 0x7f49240d2400 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f4924073430 tx=0x7f491002ba30 comp rx=0 tx=0).stop 2024-03-06T07:33:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 -- 172.21.15.97:0/470564922 shutdown_connections 2024-03-06T07:33:08.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 --2- 172.21.15.97:0/470564922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49240d13a0 0x7f49240d2400 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 --2- 172.21.15.97:0/470564922 >> v2:172.21.15.135:3300/0 conn(0x7f492415e230 0x7f49240d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 -- 172.21.15.97:0/470564922 >> 172.21.15.97:0/470564922 conn(0x7f49240d3e20 msgr2=0x7f49240d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.264+0000 7f4923200640 1 -- 172.21.15.97:0/470564922 shutdown_connections 2024-03-06T07:33:08.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.265+0000 7f4923200640 1 -- 172.21.15.97:0/470564922 wait complete. 2024-03-06T07:33:08.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.266+0000 7f4923200640 1 Processor -- start 2024-03-06T07:33:08.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.267+0000 7f4923200640 1 -- start start 2024-03-06T07:33:08.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4923200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49240d13a0 0x7f49241f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4923200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49240d2940 0x7f49240cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4923200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 0x7f492415c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4923200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49240d1d90 con 0x7f492415e230 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4923200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492415e610 con 0x7f49240d13a0 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4923200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49240d1f10 con 0x7f49240d2940 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4922800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 0x7f492415c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4921e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49240d13a0 0x7f49241f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4922800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 0x7f492415c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48516/0 (socket says 172.21.15.97:48516) 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4922800640 1 -- 172.21.15.97:0/1982162135 learned_addr learned my addr 172.21.15.97:0/1982162135 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.268+0000 7f4921400640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49240d2940 0x7f49240cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f4922800640 1 -- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49240d2940 msgr2=0x7f49240cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f4922800640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49240d2940 0x7f49240cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f4922800640 1 -- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49240d13a0 msgr2=0x7f49241f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f4922800640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49240d13a0 0x7f49241f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f4922800640 1 -- 172.21.15.97:0/1982162135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924160d60 con 0x7f492415e230 2024-03-06T07:33:08.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f4922800640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 0x7f492415c190 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c7d0 tx=0x7f491c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c003b70 con 0x7f492415e230 2024-03-06T07:33:08.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491c003d30 con 0x7f492415e230 2024-03-06T07:33:08.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.269+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c00aa90 con 0x7f492415e230 2024-03-06T07:33:08.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.270+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4924161040 con 0x7f492415e230 2024-03-06T07:33:08.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.270+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4924160980 con 0x7f492415e230 2024-03-06T07:33:08.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.271+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492404f330 con 0x7f492415e230 2024-03-06T07:33:08.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.272+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f491c00ac30 con 0x7f492415e230 2024-03-06T07:33:08.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.273+0000 7f491a000640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f490406b4d0 0x7f490406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.274+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f491c0897e0 con 0x7f492415e230 2024-03-06T07:33:08.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.274+0000 7f4921e00640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f490406b4d0 0x7f490406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.275+0000 7f4921e00640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f490406b4d0 0x7f490406d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f49140079a0 tx=0x7f4914008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.279+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f491c058690 con 0x7f492415e230 2024-03-06T07:33:08.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.601+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49240cf780 con 0x7f492415e230 2024-03-06T07:33:08.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.602+0000 7f491a000640 1 -- 172.21.15.97:0/1982162135 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f491c057f90 con 0x7f492415e230 2024-03-06T07:33:08.603 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:08.603 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:08.603 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f490406b4d0 msgr2=0x7f490406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f490406b4d0 0x7f490406d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f49140079a0 tx=0x7f4914008040 comp rx=0 tx=0).stop 2024-03-06T07:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 msgr2=0x7f492415c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 0x7f492415c190 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c7d0 tx=0x7f491c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 shutdown_connections 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f490406b4d0 0x7f490406d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f492415e230 0x7f492415c190 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49240d2940 0x7f49240cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 --2- 172.21.15.97:0/1982162135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49240d13a0 0x7f49241f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.604+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 >> 172.21.15.97:0/1982162135 conn(0x7f49240d3e20 msgr2=0x7f492406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.605+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 shutdown_connections 2024-03-06T07:33:08.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:08.605+0000 7f4923200640 1 -- 172.21.15.97:0/1982162135 wait complete. 2024-03-06T07:33:08.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:08.899 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":44,"msgr_send_messages":47,"msgr_recv_bytes":24113,"msgr_send_bytes":8866163,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.060747654,"msgr_running_send_time":0.037251932,"msgr_running_recv_time":0.007834386,"msgr_running_fast_dispatch_time":0.012191670,"msgr_send_messages_queue_lat":{"avgcount":46,"sum":0.016358373,"avgtime":0.000355616},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92720},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8075,"msgr_send_messages":15057,"msgr_recv_bytes":2902235,"msgr_send_bytes":8864312,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.799770822,"msgr_running_send_time":1.179270178,"msgr_running_recv_time":1.117768038,"msgr_running_fast_dispatch_time":0.003944503,"msgr_send_messages_queue_lat":{"avgcount":15056,"sum":0.856820052,"avgtime":0.000056908},"msgr_handle_ack_lat":{"avgcount":17976,"sum":0.009693931,"avgtime":0.000000539},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":36,"msgr_send_messages":32,"msgr_recv_bytes":114172,"msgr_send_bytes":4206492,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.028155377,"msgr_running_send_time":0.016503609,"msgr_running_recv_time":0.009751097,"msgr_running_fast_dispatch_time":0.004273580,"msgr_send_messages_queue_lat":{"avgcount":32,"sum":0.012787252,"avgtime":0.000399601},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105920,"msgr_send_encrypted_bytes":6048},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":931,"complete_latency":{"avgcount":74,"sum":4.437774658,"avgtime":0.059969927}},"finisher-PurgeQueue":{"queue_len":99,"complete_latency":{"avgcount":8,"sum":0.000418436,"avgtime":0.000052304}},"mds":{"request":4565,"reply":4564,"reply_latency":{"avgcount":4564,"sum":786.466111012,"avgtime":0.172319480},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4566,"inodes_top":0,"inodes_bottom":4566,"inodes_pin_tail":0,"inodes_pinned":4566,"inodes_expired":0,"inodes_with_caps":2969,"caps":2969,"subtrees":2,"traverse":4567,"traverse_hit":4566,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":24548352,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3734,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3468,"handle_client_caps_dirty":2999,"handle_client_cap_release":2,"process_request_cap_release":1364},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1335,"num_strays_delayed":0,"num_strays_enqueuing":213,"strays_created":1355,"strays_enqueued":20,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8259,"evex":0,"evtrm":0,"ev":8264,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":22912778,"rdpos":4195674,"jlat":{"avgcount":14,"sum":1.950084769,"avgtime":0.139291769},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4566,"dn+":4566,"dn-":0,"cap":2969,"cap+":3202,"cap-":233,"rss":101184,"heap":248164},"mds_server":{"dispatch_client_request":4566,"dispatch_server_request":0,"handle_client_request":4565,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1355,"sum":2.022727219,"avgtime":0.001492787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4304,"average_load":4304,"avg_session_uptime":934,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":6132499,"buffer_anon_items":5530,"buffer_meta_bytes":45672,"buffer_meta_items":519,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12315590,"mds_co_items":126206,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":79,"op_send_bytes":17125573,"op_resend":0,"op_reply":79,"op_latency":{"avgcount":79,"sum":0.607989634,"avgtime":0.007696071},"op_inflight":0,"oplen_avg":{"avgcount":79,"sum":133},"op":79,"op_r":37,"op_w":42,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":9,"osdop_write":16,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":15,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":8151,"get_sum":2304312,"get_or_fail_fail":0,"get_or_fail_success":8151,"take":0,"take_sum":0,"put":8150,"put_sum":2304060,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":79,"take_sum":17137291,"put":79,"put_sum":17137291,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":79,"take_sum":79,"put":79,"put_sum":79,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":9696,"max":3758096384,"get_started":0,"get":233,"get_sum":23533,"get_or_fail_fail":0,"get_or_fail_success":233,"take":0,"take_sum":0,"put":2,"put_sum":13837,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":1941258,"max":3758096384,"get_started":0,"get":8259,"get_sum":18716289,"get_or_fail_fail":0,"get_or_fail_success":8259,"take":0,"take_sum":0,"put":14,"put_sum":16775031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:08.901 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 24113, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 44, "msgr_running_fast_dispatch_time": 0.01219167, "msgr_running_recv_time": 0.007834386, "msgr_running_send_time": 0.037251932, "msgr_running_total_time": 0.060747654, "msgr_send_bytes": 8866163, "msgr_send_encrypted_bytes": 92720, "msgr_send_messages": 47, "msgr_send_messages_queue_lat": { "avgcount": 46, "avgtime": 0.000355616, "sum": 0.016358373 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 17976, "avgtime": 5.39e-07, "sum": 0.009693931 }, "msgr_recv_bytes": 2902235, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 8075, "msgr_running_fast_dispatch_time": 0.003944503, "msgr_running_recv_time": 1.117768038, "msgr_running_send_time": 1.179270178, "msgr_running_total_time": 1.799770822, "msgr_send_bytes": 8864312, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 15057, "msgr_send_messages_queue_lat": { "avgcount": 15056, "avgtime": 5.6908e-05, "sum": 0.856820052 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 114172, "msgr_recv_encrypted_bytes": 105920, "msgr_recv_messages": 36, "msgr_running_fast_dispatch_time": 0.00427358, "msgr_running_recv_time": 0.009751097, "msgr_running_send_time": 0.016503609, "msgr_running_total_time": 0.028155377, "msgr_send_bytes": 4206492, "msgr_send_encrypted_bytes": 6048, "msgr_send_messages": 32, "msgr_send_messages_queue_lat": { "avgcount": 32, "avgtime": 0.000399601, "sum": 0.012787252 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.059969927, "sum": 4.437774658 }, "queue_len": 931 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 8, "avgtime": 5.2304e-05, "sum": 0.000418436 }, "queue_len": 99 }, "mds": { "caps": 2969, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3734, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 3468, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4566, "inodes_bottom": 4566, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4566, "inodes_top": 0, "inodes_with_caps": 2969, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1364, "q": 0, "reply": 4564, "reply_latency": { "avgcount": 4564, "avgtime": 0.17231948, "sum": 786.466111012 }, "request": 4565, "root_rbytes": 24548352, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4567, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4566, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1335, "num_strays_delayed": 0, "num_strays_enqueuing": 213, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1355, "strays_enqueued": 20, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8264, "evadd": 8259, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.139291769, "sum": 1.950084769 }, "rdpos": 4195674, "replayed": 5, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 22912778 }, "mds_mem": { "cap": 2969, "cap+": 3202, "cap-": 233, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4566, "dn+": 4566, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 101184 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4566, "dispatch_server_request": 0, "handle_client_request": 4565, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1355, "avgtime": 0.001492787, "sum": 2.022727219 } }, "mds_sessions": { "average_load": 4304, "avg_session_uptime": 934, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4304 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 6132499, "buffer_anon_items": 5530, "buffer_meta_bytes": 45672, "buffer_meta_items": 519, "mds_co_bytes": 12315590, "mds_co_items": 126206, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 15, "op": 79, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 79, "avgtime": 0.007696071, "sum": 0.607989634 }, "op_pg": 0, "op_r": 37, "op_reply": 79, "op_resend": 0, "op_rmw": 0, "op_send": 79, "op_send_bytes": 17125573, "op_w": 42, "oplen_avg": { "avgcount": 79, "sum": 133 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 15, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 9, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 16, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8151, "get_or_fail_fail": 0, "get_or_fail_success": 8151, "get_started": 0, "get_sum": 2304312, "max": 104857600, "put": 8150, "put_sum": 2304060, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 79, "put_sum": 17137291, "take": 79, "take_sum": 17137291, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 79, "put_sum": 79, "take": 79, "take_sum": 79, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 233, "get_or_fail_fail": 0, "get_or_fail_success": 233, "get_started": 0, "get_sum": 23533, "max": 3758096384, "put": 2, "put_sum": 13837, "take": 0, "take_sum": 0, "val": 9696, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 8259, "get_or_fail_fail": 0, "get_or_fail_success": 8259, "get_started": 0, "get_sum": 18716289, "max": 3758096384, "put": 14, "put_sum": 16775031, "take": 0, "take_sum": 0, "val": 1941258, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:08.901 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1335, 0/1355 2024-03-06T07:33:09.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:10.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.211+0000 7f3fbaa00640 1 Processor -- start 2024-03-06T07:33:10.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.211+0000 7f3fbaa00640 1 -- start start 2024-03-06T07:33:10.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.214+0000 7f3fbaa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 0x7f3fac006f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.214+0000 7f3fbaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3fac04fc00 0x7f3fac0389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fbc000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 0x7f3fac006f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fbc000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 0x7f3fac006f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:47574/0 (socket says 172.21.15.97:47574) 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fb9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3fac04fc00 0x7f3fac0389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fb9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3fac04fc00 0x7f3fac0389a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48526/0 (socket says 172.21.15.97:48526) 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fbaa00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fac0e7b90 con 0x7f3fac006060 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fbaa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac054ad0 con 0x7f3fac04fc00 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.216+0000 7f3fbaa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac038510 con 0x7f3fac048380 2024-03-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.217+0000 7f3fbc000640 1 -- 172.21.15.97:0/1585362202 learned_addr learned my addr 172.21.15.97:0/1585362202 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:10.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.218+0000 7f3fb9600640 1 -- 172.21.15.97:0/1585362202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 msgr2=0x7f3fac006f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.218+0000 7f3fb9600640 1 --2- 172.21.15.97:0/1585362202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 0x7f3fac006f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.218+0000 7f3fb9600640 1 -- 172.21.15.97:0/1585362202 >> v1:172.21.15.135:6789/0 conn(0x7f3fac006060 legacy=0x7f3fac04f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.219+0000 7f3fbc000640 1 --2- 172.21.15.97:0/1585362202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 0x7f3fac006f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-06T07:33:10.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.221+0000 7f3fb3e00640 1 -- 172.21.15.97:0/1585362202 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221165960 0 0) 0x7f3fac0e7b90 con 0x7f3fac006060 2024-03-06T07:33:10.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.221+0000 7f3fb9600640 1 -- 172.21.15.97:0/1585362202 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac007680 con 0x7f3fac04fc00 2024-03-06T07:33:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.225+0000 7f3fb9600640 1 --2- 172.21.15.97:0/1585362202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0389a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4050c30 tx=0x7f3fb40a6b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4887c69171dd2b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.231+0000 7f3fb3e00640 1 -- 172.21.15.97:0/1585362202 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40b4070 con 0x7f3fac04fc00 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.231+0000 7f3fb3e00640 1 -- 172.21.15.97:0/1585362202 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb40a79f0 con 0x7f3fac04fc00 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.231+0000 7f3fb3e00640 1 -- 172.21.15.97:0/1585362202 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40a7e60 con 0x7f3fac04fc00 2024-03-06T07:33:10.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.232+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1585362202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 msgr2=0x7f3fac0389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.232+0000 7f3fbaa00640 1 --2- 172.21.15.97:0/1585362202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0389a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4050c30 tx=0x7f3fb40a6b80 comp rx=0 tx=0).stop 2024-03-06T07:33:10.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.234+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1585362202 shutdown_connections 2024-03-06T07:33:10.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.234+0000 7f3fbaa00640 1 --2- 172.21.15.97:0/1585362202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0389a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.234+0000 7f3fbaa00640 1 --2- 172.21.15.97:0/1585362202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac048380 0x7f3fac006f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.234+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1585362202 >> 172.21.15.97:0/1585362202 conn(0x7f3fac054c90 msgr2=0x7f3fac0550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.241+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1585362202 shutdown_connections 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.241+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1585362202 wait complete. 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.244+0000 7f3fbaa00640 1 Processor -- start 2024-03-06T07:33:10.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.246+0000 7f3fbaa00640 1 -- start start 2024-03-06T07:33:10.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.251+0000 7f3fbaa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac006060 0x7f3fac185310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.251+0000 7f3fbaa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fac048380 0x7f3fac049640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.253+0000 7f3fb8c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fac048380 0x7f3fac049640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.253+0000 7f3fbaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0d8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.253+0000 7f3fbaa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0381a0 con 0x7f3fac04fc00 2024-03-06T07:33:10.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.253+0000 7f3fbaa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac037ec0 con 0x7f3fac048380 2024-03-06T07:33:10.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.253+0000 7f3fbaa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac038ee0 con 0x7f3fac006060 2024-03-06T07:33:10.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.254+0000 7f3fb9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0d8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.255+0000 7f3fb9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0d8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48534/0 (socket says 172.21.15.97:48534) 2024-03-06T07:33:10.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.255+0000 7f3fb9600640 1 -- 172.21.15.97:0/1845897148 learned_addr learned my addr 172.21.15.97:0/1845897148 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:10.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.257+0000 7f3fbc000640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac006060 0x7f3fac185310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.257+0000 7f3fb9600640 1 -- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac006060 msgr2=0x7f3fac185310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.258+0000 7f3fb9600640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac006060 0x7f3fac185310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.258+0000 7f3fb9600640 1 -- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fac048380 msgr2=0x7f3fac049640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.258+0000 7f3fb9600640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fac048380 0x7f3fac049640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.258+0000 7f3fb9600640 1 -- 172.21.15.97:0/1845897148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0dfd40 con 0x7f3fac04fc00 2024-03-06T07:33:10.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.259+0000 7f3fb9600640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0d8c40 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4050c30 tx=0x7f3fb406a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.260+0000 7f3fbc000640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac006060 0x7f3fac185310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.260+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40be070 con 0x7f3fac04fc00 2024-03-06T07:33:10.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.262+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb40afd10 con 0x7f3fac04fc00 2024-03-06T07:33:10.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.266+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1845897148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac0d9620 con 0x7f3fac04fc00 2024-03-06T07:33:10.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.266+0000 7f3fbaa00640 1 -- 172.21.15.97:0/1845897148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac0dffc0 con 0x7f3fac04fc00 2024-03-06T07:33:10.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.266+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40b4040 con 0x7f3fac04fc00 2024-03-06T07:33:10.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.273+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac005560 con 0x7f3fac04fc00 2024-03-06T07:33:10.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.273+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb40c1070 con 0x7f3fac04fc00 2024-03-06T07:33:10.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.275+0000 7f3fb1600640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3f9406b5a0 0x7f3f9406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.275+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3fb4127380 con 0x7f3fac04fc00 2024-03-06T07:33:10.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.283+0000 7f3fbc000640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3f9406b5a0 0x7f3f9406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.285+0000 7f3fbc000640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3f9406b5a0 0x7f3f9406da60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8008930 tx=0x7f3fa8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.299+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fac0dfd40 con 0x7f3fac04fc00 2024-03-06T07:33:10.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.663+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fac03aac0 con 0x7f3fac04fc00 2024-03-06T07:33:10.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.664+0000 7f3fb1600640 1 -- 172.21.15.97:0/1845897148 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f3fb40f5b30 con 0x7f3fac04fc00 2024-03-06T07:33:10.665 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:10.666 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:10.666 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.666+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3f9406b5a0 msgr2=0x7f3f9406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.666+0000 7f3fa6a00640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3f9406b5a0 0x7f3f9406da60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8008930 tx=0x7f3fa8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.666+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 msgr2=0x7f3fac0d8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.666+0000 7f3fa6a00640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0d8c40 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4050c30 tx=0x7f3fb406a7c0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 shutdown_connections 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3f9406b5a0 0x7f3f9406da60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fac04fc00 0x7f3fac0d8c40 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fac048380 0x7f3fac049640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 --2- 172.21.15.97:0/1845897148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fac006060 0x7f3fac185310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 >> 172.21.15.97:0/1845897148 conn(0x7f3fac054c90 msgr2=0x7f3fac03a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 shutdown_connections 2024-03-06T07:33:10.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:10.667+0000 7f3fa6a00640 1 -- 172.21.15.97:0/1845897148 wait complete. 2024-03-06T07:33:10.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:10.908 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":179,"msgr_send_messages":182,"msgr_recv_bytes":94567,"msgr_send_bytes":13208957,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.100261311,"msgr_running_send_time":0.063701838,"msgr_running_recv_time":0.012772936,"msgr_running_fast_dispatch_time":0.018835867,"msgr_send_messages_queue_lat":{"avgcount":181,"sum":0.085303705,"avgtime":0.000471291},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92720},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10440,"msgr_send_messages":17596,"msgr_recv_bytes":3724123,"msgr_send_bytes":9931653,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.107634246,"msgr_running_send_time":1.382363371,"msgr_running_recv_time":1.339893650,"msgr_running_fast_dispatch_time":0.009334891,"msgr_send_messages_queue_lat":{"avgcount":17595,"sum":1.117115884,"avgtime":0.000063490},"msgr_handle_ack_lat":{"avgcount":21709,"sum":0.010392359,"avgtime":0.000000478},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":150,"msgr_send_messages":146,"msgr_recv_bytes":143138,"msgr_send_bytes":4243946,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.041352983,"msgr_running_send_time":0.022391376,"msgr_running_recv_time":0.012989651,"msgr_running_fast_dispatch_time":0.007478271,"msgr_send_messages_queue_lat":{"avgcount":146,"sum":0.022420896,"avgtime":0.000153567},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106192,"msgr_send_encrypted_bytes":6672},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1699,"complete_latency":{"avgcount":79,"sum":6.220862494,"avgtime":0.078745094}},"finisher-PurgeQueue":{"queue_len":53,"complete_latency":{"avgcount":713,"sum":0.531057051,"avgtime":0.000744820}},"mds":{"request":5226,"reply":5226,"reply_latency":{"avgcount":5226,"sum":787.688865507,"avgtime":0.150725002},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5228,"inodes_top":0,"inodes_bottom":5228,"inodes_pin_tail":0,"inodes_pinned":5228,"inodes_expired":0,"inodes_with_caps":2657,"caps":2657,"subtrees":2,"traverse":5228,"traverse_hit":5228,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2041,"root_rbytes":15808512,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":3,"ceph_cap_op_grant":4628,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3962,"handle_client_caps_dirty":3000,"handle_client_cap_release":6,"process_request_cap_release":2025},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1578,"num_strays_delayed":0,"num_strays_enqueuing":16,"strays_created":2017,"strays_enqueued":439,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9903,"evex":0,"evtrm":0,"ev":9908,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26419715,"rdpos":4195674,"jlat":{"avgcount":15,"sum":1.995246711,"avgtime":0.133016447},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":5228,"dn+":5228,"dn-":0,"cap":2657,"cap+":3202,"cap-":545,"rss":113728,"heap":248164},"mds_server":{"dispatch_client_request":5227,"dispatch_server_request":0,"handle_client_request":5226,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2017,"sum":3.245481714,"avgtime":0.001609063},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4887,"average_load":4887,"avg_session_uptime":939,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":28272366,"buffer_anon_items":11338,"buffer_meta_bytes":1848,"buffer_meta_items":21,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16484771,"mds_co_items":221035,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":503,"op_send_bytes":22435635,"op_resend":0,"op_reply":503,"op_latency":{"avgcount":503,"sum":2.143963449,"avgtime":0.004262352},"op_inflight":0,"oplen_avg":{"avgcount":503,"sum":560},"op":503,"op_r":44,"op_w":459,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":16,"osdop_write":25,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":419,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":403,"pq_executed":403,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":272428,"max":104857600,"get_started":0,"get":10765,"get_sum":2990308,"get_or_fail_fail":0,"get_or_fail_success":10765,"take":0,"take_sum":0,"put":9737,"put_sum":2717880,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":503,"take_sum":22483511,"put":503,"put_sum":22483511,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":503,"take_sum":503,"put":503,"put_sum":503,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":455,"get_sum":45955,"get_or_fail_fail":0,"get_or_fail_success":455,"take":0,"take_sum":0,"put":8,"put_sum":45955,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":242799,"max":3758096384,"get_started":0,"get":9903,"get_sum":22223226,"get_or_fail_fail":0,"get_or_fail_success":9903,"take":0,"take_sum":0,"put":17,"put_sum":21980427,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:10.910 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 94567, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 179, "msgr_running_fast_dispatch_time": 0.018835867, "msgr_running_recv_time": 0.012772936, "msgr_running_send_time": 0.063701838, "msgr_running_total_time": 0.100261311, "msgr_send_bytes": 13208957, "msgr_send_encrypted_bytes": 92720, "msgr_send_messages": 182, "msgr_send_messages_queue_lat": { "avgcount": 181, "avgtime": 0.000471291, "sum": 0.085303705 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 21709, "avgtime": 4.78e-07, "sum": 0.010392359 }, "msgr_recv_bytes": 3724123, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10440, "msgr_running_fast_dispatch_time": 0.009334891, "msgr_running_recv_time": 1.33989365, "msgr_running_send_time": 1.382363371, "msgr_running_total_time": 2.107634246, "msgr_send_bytes": 9931653, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 17596, "msgr_send_messages_queue_lat": { "avgcount": 17595, "avgtime": 6.349e-05, "sum": 1.117115884 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 143138, "msgr_recv_encrypted_bytes": 106192, "msgr_recv_messages": 150, "msgr_running_fast_dispatch_time": 0.007478271, "msgr_running_recv_time": 0.012989651, "msgr_running_send_time": 0.022391376, "msgr_running_total_time": 0.041352983, "msgr_send_bytes": 4243946, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 146, "msgr_send_messages_queue_lat": { "avgcount": 146, "avgtime": 0.000153567, "sum": 0.022420896 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 79, "avgtime": 0.078745094, "sum": 6.220862494 }, "queue_len": 1699 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 713, "avgtime": 0.00074482, "sum": 0.531057051 }, "queue_len": 53 }, "mds": { "caps": 2657, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4628, "ceph_cap_op_revoke": 3, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 6, "handle_client_caps": 3962, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5228, "inodes_bottom": 5228, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5228, "inodes_top": 0, "inodes_with_caps": 2657, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2025, "q": 0, "reply": 5226, "reply_latency": { "avgcount": 5226, "avgtime": 0.150725002, "sum": 787.688865507 }, "request": 5226, "root_rbytes": 15808512, "root_rfiles": 2041, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5228, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5228, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1578, "num_strays_delayed": 0, "num_strays_enqueuing": 16, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2017, "strays_enqueued": 439, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9908, "evadd": 9903, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 15, "avgtime": 0.133016447, "sum": 1.995246711 }, "rdpos": 4195674, "replayed": 5, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26419715 }, "mds_mem": { "cap": 2657, "cap+": 3202, "cap-": 545, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5228, "dn+": 5228, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 113728 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5227, "dispatch_server_request": 0, "handle_client_request": 5226, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2017, "avgtime": 0.001609063, "sum": 3.245481714 } }, "mds_sessions": { "average_load": 4887, "avg_session_uptime": 939, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4887 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 28272366, "buffer_anon_items": 11338, "buffer_meta_bytes": 1848, "buffer_meta_items": 21, "mds_co_bytes": 16484771, "mds_co_items": 221035, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 19, "op": 503, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 503, "avgtime": 0.004262352, "sum": 2.143963449 }, "op_pg": 0, "op_r": 44, "op_reply": 503, "op_resend": 0, "op_rmw": 0, "op_send": 503, "op_send_bytes": 22435635, "op_w": 459, "oplen_avg": { "avgcount": 503, "sum": 560 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 419, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 16, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 25, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 403, "pq_executed_ops": 403, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10765, "get_or_fail_fail": 0, "get_or_fail_success": 10765, "get_started": 0, "get_sum": 2990308, "max": 104857600, "put": 9737, "put_sum": 2717880, "take": 0, "take_sum": 0, "val": 272428, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 503, "put_sum": 22483511, "take": 503, "take_sum": 22483511, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 503, "put_sum": 503, "take": 503, "take_sum": 503, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 455, "get_or_fail_fail": 0, "get_or_fail_success": 455, "get_started": 0, "get_sum": 45955, "max": 3758096384, "put": 8, "put_sum": 45955, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 9903, "get_or_fail_fail": 0, "get_or_fail_success": 9903, "get_started": 0, "get_sum": 22223226, "max": 3758096384, "put": 17, "put_sum": 21980427, "take": 0, "take_sum": 0, "val": 242799, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:10.910 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1578, 403/2017 2024-03-06T07:33:11.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:12.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.119+0000 7f8f3fe00640 1 Processor -- start 2024-03-06T07:33:12.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.119+0000 7f8f3fe00640 1 -- start start 2024-03-06T07:33:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.120+0000 7f8f3fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8f380b5020 0x7f8f380d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.120+0000 7f8f3fe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f380d21c0 con 0x7f8f380b44f0 2024-03-06T07:33:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.120+0000 7f8f3fe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f380b48d0 con 0x7f8f380d2cc0 2024-03-06T07:33:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.120+0000 7f8f3fe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f380d0000 con 0x7f8f380b5020 2024-03-06T07:33:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.121+0000 7f8f3d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8f380b44f0 0x7f8f380d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51688/0 (socket says 172.21.15.97:51688) 2024-03-06T07:33:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.121+0000 7f8f3d400640 1 -- 172.21.15.97:0/695353572 learned_addr learned my addr 172.21.15.97:0/695353572 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.121+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739415924 0 0) 0x7f8f380d21c0 con 0x7f8f380b44f0 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.121+0000 7f8f3de00640 1 --2- 172.21.15.97:0/695353572 >> v2:172.21.15.135:3300/0 conn(0x7f8f380b5020 0x7f8f380d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.121+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f20003520 con 0x7f8f380b44f0 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108313298 0 0) 0x7f8f20003520 con 0x7f8f380b44f0 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 >> v2:172.21.15.135:3300/0 conn(0x7f8f380b5020 msgr2=0x7f8f380d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 --2- 172.21.15.97:0/695353572 >> v2:172.21.15.135:3300/0 conn(0x7f8f380b5020 0x7f8f380d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 >> v1:172.21.15.135:6789/0 conn(0x7f8f380d2cc0 legacy=0x7f8f380d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f380d1ee0 con 0x7f8f380b44f0 2024-03-06T07:33:12.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8f2c001670 con 0x7f8f380b44f0 2024-03-06T07:33:12.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 >> v1:172.21.15.97:6789/0 conn(0x7f8f380b44f0 legacy=0x7f8f380d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f200039b0 0x7f8f20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f20004800 0x7f8f20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380d0000 con 0x7f8f200051c0 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2000c160 con 0x7f8f200039b0 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2000c360 con 0x7f8f20004800 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.122+0000 7f8f3de00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3de00640 1 -- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f20004800 msgr2=0x7f8f20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3d400640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f200039b0 0x7f8f20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3ca00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f20004800 0x7f8f20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3de00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f20004800 0x7f8f20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3de00640 1 -- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f200039b0 msgr2=0x7f8f20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3de00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f200039b0 0x7f8f20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3de00640 1 -- 172.21.15.97:0/695353572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f20003520 con 0x7f8f200051c0 2024-03-06T07:33:12.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3ca00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f20004800 0x7f8f20004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:12.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.123+0000 7f8f3d400640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f200039b0 0x7f8f20003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:12.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.124+0000 7f8f3de00640 1 -- 172.21.15.97:0/695353572 reap_dead start 2024-03-06T07:33:12.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.125+0000 7f8f3de00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f2000b9c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009fc0 tx=0x7f8f20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749ab8d2614ee433 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.125+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30009c10 con 0x7f8f200051c0 2024-03-06T07:33:12.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.125+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3002cc70 con 0x7f8f200051c0 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.125+0000 7f8f37e00640 1 -- 172.21.15.97:0/695353572 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30003bd0 con 0x7f8f200051c0 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 -- 172.21.15.97:0/695353572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 msgr2=0x7f8f2000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f2000b9c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009fc0 tx=0x7f8f20005830 comp rx=0 tx=0).stop 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 -- 172.21.15.97:0/695353572 shutdown_connections 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/695353572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f2000b9c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 -- 172.21.15.97:0/695353572 >> 172.21.15.97:0/695353572 conn(0x7f8f380d3fd0 msgr2=0x7f8f380d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 -- 172.21.15.97:0/695353572 shutdown_connections 2024-03-06T07:33:12.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.126+0000 7f8f3fe00640 1 -- 172.21.15.97:0/695353572 wait complete. 2024-03-06T07:33:12.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.128+0000 7f8f3fe00640 1 Processor -- start 2024-03-06T07:33:12.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.128+0000 7f8f3fe00640 1 -- start start 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.129+0000 7f8f3fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f381f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.129+0000 7f8f3fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f380b1610 0x7f8f380b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.129+0000 7f8f3fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f381f54d0 0x7f8f381f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.129+0000 7f8f3fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380cfae0 con 0x7f8f200051c0 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.129+0000 7f8f3fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380b4e40 con 0x7f8f381f54d0 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.129+0000 7f8f3fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380b4ad0 con 0x7f8f380b1610 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f381f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f381f7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48556/0 (socket says 172.21.15.97:48556) 2024-03-06T07:33:12.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3d400640 1 -- 172.21.15.97:0/1083857547 learned_addr learned my addr 172.21.15.97:0/1083857547 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3de00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f381f54d0 0x7f8f381f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3ca00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f380b1610 0x7f8f380b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3d400640 1 -- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f380b1610 msgr2=0x7f8f380b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.130+0000 7f8f3d400640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f380b1610 0x7f8f380b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f3d400640 1 -- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f381f54d0 msgr2=0x7f8f381f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f3d400640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f381f54d0 0x7f8f381f5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f3d400640 1 -- 172.21.15.97:0/1083857547 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f380b1e60 con 0x7f8f200051c0 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f3de00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f381f54d0 0x7f8f381f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:12.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f3d400640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f381f7ee0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00d7e0 tx=0x7f8f2c00dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c019070 con 0x7f8f200051c0 2024-03-06T07:33:12.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0074e0 con 0x7f8f200051c0 2024-03-06T07:33:12.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c004a00 con 0x7f8f200051c0 2024-03-06T07:33:12.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.131+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f380b2140 con 0x7f8f200051c0 2024-03-06T07:33:12.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.132+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f381f4fb0 con 0x7f8f200051c0 2024-03-06T07:33:12.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.133+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f380d35d0 con 0x7f8f200051c0 2024-03-06T07:33:12.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.134+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f2c01c070 con 0x7f8f200051c0 2024-03-06T07:33:12.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.136+0000 7f8f35600640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f1806b4d0 0x7f8f1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.136+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8f2c089e30 con 0x7f8f200051c0 2024-03-06T07:33:12.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.136+0000 7f8f3ca00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f1806b4d0 0x7f8f1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.137+0000 7f8f3ca00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f1806b4d0 0x7f8f1806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8f380b2d40 tx=0x7f8f24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.145+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f380d35d0 con 0x7f8f200051c0 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.463+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f38068070 con 0x7f8f200051c0 2024-03-06T07:33:12.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.464+0000 7f8f35600640 1 -- 172.21.15.97:0/1083857547 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f8f2c058bb0 con 0x7f8f200051c0 2024-03-06T07:33:12.465 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:12.465 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:12.466 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:12.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.466+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f1806b4d0 msgr2=0x7f8f1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.466+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f1806b4d0 0x7f8f1806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8f380b2d40 tx=0x7f8f24009040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.466+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 msgr2=0x7f8f381f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.466+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f381f7ee0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00d7e0 tx=0x7f8f2c00dca0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 shutdown_connections 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8f1806b4d0 0x7f8f1806d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f381f54d0 0x7f8f381f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8f380b1610 0x7f8f380b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 --2- 172.21.15.97:0/1083857547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f200051c0 0x7f8f381f7ee0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 >> 172.21.15.97:0/1083857547 conn(0x7f8f380d3fd0 msgr2=0x7f8f380ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 shutdown_connections 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:12.467+0000 7f8f3fe00640 1 -- 172.21.15.97:0/1083857547 wait complete. 2024-03-06T07:33:12.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:12.743 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":209,"msgr_send_messages":213,"msgr_recv_bytes":102963,"msgr_send_bytes":13221305,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.104087854,"msgr_running_send_time":0.065459347,"msgr_running_recv_time":0.013610370,"msgr_running_fast_dispatch_time":0.019809067,"msgr_send_messages_queue_lat":{"avgcount":212,"sum":0.086779134,"avgtime":0.000409335},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94512},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11009,"msgr_send_messages":19529,"msgr_recv_bytes":3868012,"msgr_send_bytes":10506146,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.298095073,"msgr_running_send_time":1.500884711,"msgr_running_recv_time":1.423235820,"msgr_running_fast_dispatch_time":0.009870264,"msgr_send_messages_queue_lat":{"avgcount":19528,"sum":1.202178145,"avgtime":0.000061561},"msgr_handle_ack_lat":{"avgcount":23487,"sum":0.010977059,"avgtime":0.000000467},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":165,"msgr_send_messages":161,"msgr_recv_bytes":146948,"msgr_send_bytes":4248836,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.042698091,"msgr_running_send_time":0.022888915,"msgr_running_recv_time":0.013378997,"msgr_running_fast_dispatch_time":0.007834787,"msgr_send_messages_queue_lat":{"avgcount":161,"sum":0.022943047,"avgtime":0.000142503},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106192,"msgr_send_encrypted_bytes":6672},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":82,"sum":8.658131685,"avgtime":0.105586971}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":825,"sum":0.592527317,"avgtime":0.000718214}},"mds":{"request":5481,"reply":5480,"reply_latency":{"avgcount":5480,"sum":789.983778440,"avgtime":0.144157623},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4827,"inodes_top":0,"inodes_bottom":4827,"inodes_pin_tail":0,"inodes_pinned":4827,"inodes_expired":0,"inodes_with_caps":1628,"caps":1628,"subtrees":2,"traverse":5483,"traverse_hit":5483,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1609,"root_rbytes":12426240,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":5562,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4902,"handle_client_caps_dirty":3000,"handle_client_cap_release":386,"process_request_cap_release":2280},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1809,"num_strays_delayed":987,"num_strays_enqueuing":0,"strays_created":2272,"strays_enqueued":463,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11051,"evex":0,"evtrm":0,"ev":11056,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28899955,"rdpos":4195674,"jlat":{"avgcount":17,"sum":5.599516469,"avgtime":0.329383321},"replayed":5},"mds_mem":{"ino":2811,"ino+":3214,"ino-":403,"dir":13,"dir+":13,"dir-":0,"dn":5079,"dn+":5482,"dn-":403,"cap":1291,"cap+":3202,"cap-":1911,"rss":113728,"heap":248164},"mds_server":{"dispatch_client_request":5482,"dispatch_server_request":0,"handle_client_request":5481,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2271,"sum":5.540394647,"avgtime":0.002439627},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5008,"average_load":5008,"avg_session_uptime":939,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":28272366,"buffer_anon_items":11338,"buffer_meta_bytes":1848,"buffer_meta_items":21,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16484771,"mds_co_items":221035,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":569,"op_send_bytes":22436623,"op_resend":0,"op_reply":569,"op_latency":{"avgcount":569,"sum":2.360959748,"avgtime":0.004149314},"op_inflight":0,"oplen_avg":{"avgcount":569,"sum":626},"op":569,"op_r":46,"op_w":523,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":18,"osdop_write":27,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":479,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":463,"pq_executed":463,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11379,"get_sum":3091143,"get_or_fail_fail":0,"get_or_fail_success":11379,"take":0,"take_sum":0,"put":11379,"put_sum":3091143,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":569,"take_sum":22485307,"put":569,"put_sum":22485307,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":569,"take_sum":569,"put":569,"put_sum":569,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":463,"get_sum":46763,"get_or_fail_fail":0,"get_or_fail_success":463,"take":0,"take_sum":0,"put":10,"put_sum":46763,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":2726391,"max":3758096384,"get_started":0,"get":11052,"get_sum":24706818,"get_or_fail_fail":0,"get_or_fail_success":11052,"take":0,"take_sum":0,"put":17,"put_sum":21980427,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:12.745 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 102963, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 209, "msgr_running_fast_dispatch_time": 0.019809067, "msgr_running_recv_time": 0.01361037, "msgr_running_send_time": 0.065459347, "msgr_running_total_time": 0.104087854, "msgr_send_bytes": 13221305, "msgr_send_encrypted_bytes": 94512, "msgr_send_messages": 213, "msgr_send_messages_queue_lat": { "avgcount": 212, "avgtime": 0.000409335, "sum": 0.086779134 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 23487, "avgtime": 4.67e-07, "sum": 0.010977059 }, "msgr_recv_bytes": 3868012, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11009, "msgr_running_fast_dispatch_time": 0.009870264, "msgr_running_recv_time": 1.42323582, "msgr_running_send_time": 1.500884711, "msgr_running_total_time": 2.298095073, "msgr_send_bytes": 10506146, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 19529, "msgr_send_messages_queue_lat": { "avgcount": 19528, "avgtime": 6.1561e-05, "sum": 1.202178145 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 146948, "msgr_recv_encrypted_bytes": 106192, "msgr_recv_messages": 165, "msgr_running_fast_dispatch_time": 0.007834787, "msgr_running_recv_time": 0.013378997, "msgr_running_send_time": 0.022888915, "msgr_running_total_time": 0.042698091, "msgr_send_bytes": 4248836, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 161, "msgr_send_messages_queue_lat": { "avgcount": 161, "avgtime": 0.000142503, "sum": 0.022943047 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 82, "avgtime": 0.105586971, "sum": 8.658131685 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 825, "avgtime": 0.000718214, "sum": 0.592527317 }, "queue_len": 0 }, "mds": { "caps": 1628, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5562, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 386, "handle_client_caps": 4902, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4827, "inodes_bottom": 4827, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4827, "inodes_top": 0, "inodes_with_caps": 1628, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2280, "q": 0, "reply": 5480, "reply_latency": { "avgcount": 5480, "avgtime": 0.144157623, "sum": 789.98377844 }, "request": 5481, "root_rbytes": 12426240, "root_rfiles": 1609, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5483, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5483, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1809, "num_strays_delayed": 987, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2272, "strays_enqueued": 463, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11056, "evadd": 11051, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.329383321, "sum": 5.599516469 }, "rdpos": 4195674, "replayed": 5, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28899955 }, "mds_mem": { "cap": 1291, "cap+": 3202, "cap-": 1911, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5079, "dn+": 5482, "dn-": 403, "heap": 248164, "ino": 2811, "ino+": 3214, "ino-": 403, "rss": 113728 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5482, "dispatch_server_request": 0, "handle_client_request": 5481, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2271, "avgtime": 0.002439627, "sum": 5.540394647 } }, "mds_sessions": { "average_load": 5008, "avg_session_uptime": 939, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5008 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 28272366, "buffer_anon_items": 11338, "buffer_meta_bytes": 1848, "buffer_meta_items": 21, "mds_co_bytes": 16484771, "mds_co_items": 221035, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 19, "op": 569, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 569, "avgtime": 0.004149314, "sum": 2.360959748 }, "op_pg": 0, "op_r": 46, "op_reply": 569, "op_resend": 0, "op_rmw": 0, "op_send": 569, "op_send_bytes": 22436623, "op_w": 523, "oplen_avg": { "avgcount": 569, "sum": 626 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 479, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 18, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 27, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 463, "pq_executed_ops": 463, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11379, "get_or_fail_fail": 0, "get_or_fail_success": 11379, "get_started": 0, "get_sum": 3091143, "max": 104857600, "put": 11379, "put_sum": 3091143, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 569, "put_sum": 22485307, "take": 569, "take_sum": 22485307, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 569, "put_sum": 569, "take": 569, "take_sum": 569, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 463, "get_or_fail_fail": 0, "get_or_fail_success": 463, "get_started": 0, "get_sum": 46763, "max": 3758096384, "put": 10, "put_sum": 46763, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 11052, "get_or_fail_fail": 0, "get_or_fail_success": 11052, "get_started": 0, "get_sum": 24706818, "max": 3758096384, "put": 17, "put_sum": 21980427, "take": 0, "take_sum": 0, "val": 2726391, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:12.745 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1809, 463/2272 2024-03-06T07:33:13.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.027+0000 7fbda7e00640 1 Processor -- start 2024-03-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.027+0000 7fbda7e00640 1 -- start start 2024-03-06T07:33:14.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.028+0000 7fbda7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbda815e230 0x7fbda80d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.028+0000 7fbda7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbda80d2b10 0x7fbda80b4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.028+0000 7fbda7e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda80b39d0 con 0x7fbda80d13a0 2024-03-06T07:33:14.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.028+0000 7fbda7e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda80b3780 con 0x7fbda80d2b10 2024-03-06T07:33:14.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.028+0000 7fbda7e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda80b3cb0 con 0x7fbda815e230 2024-03-06T07:33:14.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.028+0000 7fbda7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbda80d2b10 0x7fbda80b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbda80d2b10 0x7fbda80b4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40544/0 (socket says 172.21.15.97:40544) 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda7400640 1 -- 172.21.15.97:0/330526014 learned_addr learned my addr 172.21.15.97:0/330526014 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda6a00640 1 --2- 172.21.15.97:0/330526014 >> v2:172.21.15.135:3300/0 conn(0x7fbda815e230 0x7fbda80d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda7400640 1 -- 172.21.15.97:0/330526014 >> v2:172.21.15.135:3300/0 conn(0x7fbda815e230 msgr2=0x7fbda80d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda7400640 1 --2- 172.21.15.97:0/330526014 >> v2:172.21.15.135:3300/0 conn(0x7fbda815e230 0x7fbda80d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda7400640 1 -- 172.21.15.97:0/330526014 >> v1:172.21.15.135:6789/0 conn(0x7fbda80d13a0 legacy=0x7fbda80d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda7400640 1 -- 172.21.15.97:0/330526014 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80b3f90 con 0x7fbda80d2b10 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.029+0000 7fbda6a00640 1 --2- 172.21.15.97:0/330526014 >> v2:172.21.15.135:3300/0 conn(0x7fbda815e230 0x7fbda80d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:14.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.030+0000 7fbda7400640 1 --2- 172.21.15.97:0/330526014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b4790 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fbda8073430 tx=0x7fbda002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eedbc37d809d409 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.031+0000 7fbda5600640 1 -- 172.21.15.97:0/330526014 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda002cb20 con 0x7fbda80d2b10 2024-03-06T07:33:14.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.031+0000 7fbda5600640 1 -- 172.21.15.97:0/330526014 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda002ccc0 con 0x7fbda80d2b10 2024-03-06T07:33:14.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.031+0000 7fbda5600640 1 -- 172.21.15.97:0/330526014 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0034610 con 0x7fbda80d2b10 2024-03-06T07:33:14.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.031+0000 7fbda7e00640 1 -- 172.21.15.97:0/330526014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 msgr2=0x7fbda80b4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.031+0000 7fbda7e00640 1 --2- 172.21.15.97:0/330526014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b4790 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fbda8073430 tx=0x7fbda002bf80 comp rx=0 tx=0).stop 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.032+0000 7fbda7e00640 1 -- 172.21.15.97:0/330526014 shutdown_connections 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.032+0000 7fbda7e00640 1 --2- 172.21.15.97:0/330526014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b4790 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.032+0000 7fbda7e00640 1 --2- 172.21.15.97:0/330526014 >> v2:172.21.15.135:3300/0 conn(0x7fbda815e230 0x7fbda80d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.032+0000 7fbda7e00640 1 -- 172.21.15.97:0/330526014 >> 172.21.15.97:0/330526014 conn(0x7fbda80d3e20 msgr2=0x7fbda80d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.032+0000 7fbda7e00640 1 -- 172.21.15.97:0/330526014 shutdown_connections 2024-03-06T07:33:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.032+0000 7fbda7e00640 1 -- 172.21.15.97:0/330526014 wait complete. 2024-03-06T07:33:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.034+0000 7fbda7e00640 1 Processor -- start 2024-03-06T07:33:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.034+0000 7fbda7e00640 1 -- start start 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.035+0000 7fbda7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbda80d13a0 0x7fbda81f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.035+0000 7fbda7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.035+0000 7fbda7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda815e230 0x7fbda80b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.035+0000 7fbda7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80d1d90 con 0x7fbda80d2b10 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.035+0000 7fbda7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda815e610 con 0x7fbda815e230 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.035+0000 7fbda7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80d1f10 con 0x7fbda80d13a0 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40560/0 (socket says 172.21.15.97:40560) 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 -- 172.21.15.97:0/1602574376 learned_addr learned my addr 172.21.15.97:0/1602574376 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda7400640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda815e230 0x7fbda80b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6a00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbda80d13a0 0x7fbda81f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 -- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbda80d13a0 msgr2=0x7fbda81f3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbda80d13a0 0x7fbda81f3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 -- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda815e230 msgr2=0x7fbda80b6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda815e230 0x7fbda80b6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda6000640 1 -- 172.21.15.97:0/1602574376 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda81f7ca0 con 0x7fbda80d2b10 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.036+0000 7fbda7400640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda815e230 0x7fbda80b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.037+0000 7fbda6000640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b6a40 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c400 tx=0x7fbd9400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.037+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94011790 con 0x7fbda80d2b10 2024-03-06T07:33:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.037+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd94017070 con 0x7fbda80d2b10 2024-03-06T07:33:14.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.037+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94011bb0 con 0x7fbda80d2b10 2024-03-06T07:33:14.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.038+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda80b75b0 con 0x7fbda80d2b10 2024-03-06T07:33:14.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.038+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda80b7b70 con 0x7fbda80d2b10 2024-03-06T07:33:14.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.041+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd9401c460 con 0x7fbda80d2b10 2024-03-06T07:33:14.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.042+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda804f330 con 0x7fbda80d2b10 2024-03-06T07:33:14.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.043+0000 7fbd9ea00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbd8806b520 0x7fbd8806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.043+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbd94089fe0 con 0x7fbda80d2b10 2024-03-06T07:33:14.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.043+0000 7fbda6a00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbd8806b520 0x7fbd8806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.044+0000 7fbda6a00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbd8806b520 0x7fbd8806d9e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbd9800a950 tx=0x7fbd98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.051+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd94058ee0 con 0x7fbda80d2b10 2024-03-06T07:33:14.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.382+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbda80748c0 con 0x7fbda80d2b10 2024-03-06T07:33:14.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.383+0000 7fbd9ea00640 1 -- 172.21.15.97:0/1602574376 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fbd940587e0 con 0x7fbda80d2b10 2024-03-06T07:33:14.384 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:14.384 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:14.385 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbd8806b520 msgr2=0x7fbd8806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbd8806b520 0x7fbd8806d9e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbd9800a950 tx=0x7fbd98008040 comp rx=0 tx=0).stop 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 msgr2=0x7fbda80b6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b6a40 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c400 tx=0x7fbd9400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 shutdown_connections 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbd8806b520 0x7fbd8806d9e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda815e230 0x7fbda80b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbda80d2b10 0x7fbda80b6a40 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 --2- 172.21.15.97:0/1602574376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbda80d13a0 0x7fbda81f3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.386+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 >> 172.21.15.97:0/1602574376 conn(0x7fbda80d3e20 msgr2=0x7fbda80b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.387+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 shutdown_connections 2024-03-06T07:33:14.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:14.387+0000 7fbda7e00640 1 -- 172.21.15.97:0/1602574376 wait complete. 2024-03-06T07:33:14.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:14.668 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":490,"msgr_send_messages":496,"msgr_recv_bytes":320245,"msgr_send_bytes":13501353,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.145167741,"msgr_running_send_time":0.087056437,"msgr_running_recv_time":0.022045454,"msgr_running_fast_dispatch_time":0.028535965,"msgr_send_messages_queue_lat":{"avgcount":495,"sum":0.117735478,"avgtime":0.000237849},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94512},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12411,"msgr_send_messages":21614,"msgr_recv_bytes":4307858,"msgr_send_bytes":12769216,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.596718484,"msgr_running_send_time":1.681145671,"msgr_running_recv_time":1.543959281,"msgr_running_fast_dispatch_time":0.027103185,"msgr_send_messages_queue_lat":{"avgcount":21613,"sum":1.341479079,"avgtime":0.000062068},"msgr_handle_ack_lat":{"avgcount":25610,"sum":0.011501841,"avgtime":0.000000449},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":458,"msgr_send_messages":456,"msgr_recv_bytes":221388,"msgr_send_bytes":4345304,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.075944842,"msgr_running_send_time":0.036951064,"msgr_running_recv_time":0.021049930,"msgr_running_fast_dispatch_time":0.016768435,"msgr_send_messages_queue_lat":{"avgcount":456,"sum":0.049410378,"avgtime":0.000108356},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106464,"msgr_send_encrypted_bytes":7296},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1578,"complete_latency":{"avgcount":85,"sum":8.659288265,"avgtime":0.101873979}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2628,"sum":1.896831105,"avgtime":0.000721777}},"mds":{"request":6375,"reply":6375,"reply_latency":{"avgcount":6375,"sum":791.581773410,"avgtime":0.124169689},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5859,"inodes_top":0,"inodes_bottom":5859,"inodes_pin_tail":0,"inodes_pinned":5859,"inodes_expired":0,"inodes_with_caps":1121,"caps":1121,"subtrees":2,"traverse":6377,"traverse_hit":6377,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1103,"root_rbytes":8599552,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":5904,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4902,"handle_client_caps_dirty":3000,"handle_client_cap_release":387,"process_request_cap_release":3174},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1259,"num_strays_delayed":174,"num_strays_enqueuing":0,"strays_created":3166,"strays_enqueued":1907,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13913,"evex":0,"evtrm":0,"ev":13918,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":33323133,"rdpos":4195674,"jlat":{"avgcount":18,"sum":5.649417901,"avgtime":0.313856550},"replayed":5},"mds_mem":{"ino":2697,"ino+":3214,"ino-":517,"dir":13,"dir+":13,"dir-":0,"dn":5860,"dn+":6377,"dn-":517,"cap":1121,"cap+":3202,"cap-":2081,"rss":113728,"heap":248164},"mds_server":{"dispatch_client_request":6376,"dispatch_server_request":0,"handle_client_request":6375,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3166,"sum":7.138389617,"avgtime":0.002254702},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5783,"average_load":5783,"avg_session_uptime":939,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":28272366,"buffer_anon_items":11338,"buffer_meta_bytes":1848,"buffer_meta_items":21,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16484771,"mds_co_items":221035,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":1652,"op_send_bytes":25807577,"op_resend":0,"op_reply":1646,"op_latency":{"avgcount":1646,"sum":5.624904109,"avgtime":0.003417317},"op_inflight":5,"oplen_avg":{"avgcount":1652,"sum":1713},"op":1652,"op_r":52,"op_w":1600,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":24,"osdop_write":34,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1547,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":23,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":1524,"pq_executed":1524,"pq_item_in_journal":383},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":13355,"get_sum":3644819,"get_or_fail_fail":0,"get_or_fail_success":13355,"take":0,"take_sum":0,"put":13354,"put_sum":3644566,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1652,"take_sum":26002105,"put":1646,"put_sum":26002105,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1652,"take_sum":1652,"put":1646,"put_sum":1646,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1907,"get_sum":192607,"get_or_fail_fail":0,"get_or_fail_success":1907,"take":0,"take_sum":0,"put":16,"put_sum":192607,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":3963005,"max":3758096384,"get_started":0,"get":13913,"get_sum":29126644,"get_or_fail_fail":0,"get_or_fail_success":13913,"take":0,"take_sum":0,"put":18,"put_sum":25163639,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:14.670 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 320245, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 490, "msgr_running_fast_dispatch_time": 0.028535965, "msgr_running_recv_time": 0.022045454, "msgr_running_send_time": 0.087056437, "msgr_running_total_time": 0.145167741, "msgr_send_bytes": 13501353, "msgr_send_encrypted_bytes": 94512, "msgr_send_messages": 496, "msgr_send_messages_queue_lat": { "avgcount": 495, "avgtime": 0.000237849, "sum": 0.117735478 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 25610, "avgtime": 4.49e-07, "sum": 0.011501841 }, "msgr_recv_bytes": 4307858, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 12411, "msgr_running_fast_dispatch_time": 0.027103185, "msgr_running_recv_time": 1.543959281, "msgr_running_send_time": 1.681145671, "msgr_running_total_time": 2.596718484, "msgr_send_bytes": 12769216, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 21614, "msgr_send_messages_queue_lat": { "avgcount": 21613, "avgtime": 6.2068e-05, "sum": 1.341479079 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 221388, "msgr_recv_encrypted_bytes": 106464, "msgr_recv_messages": 458, "msgr_running_fast_dispatch_time": 0.016768435, "msgr_running_recv_time": 0.02104993, "msgr_running_send_time": 0.036951064, "msgr_running_total_time": 0.075944842, "msgr_send_bytes": 4345304, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 456, "msgr_send_messages_queue_lat": { "avgcount": 456, "avgtime": 0.000108356, "sum": 0.049410378 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 85, "avgtime": 0.101873979, "sum": 8.659288265 }, "queue_len": 1578 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2628, "avgtime": 0.000721777, "sum": 1.896831105 }, "queue_len": 0 }, "mds": { "caps": 1121, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5904, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 387, "handle_client_caps": 4902, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5859, "inodes_bottom": 5859, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5859, "inodes_top": 0, "inodes_with_caps": 1121, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3174, "q": 0, "reply": 6375, "reply_latency": { "avgcount": 6375, "avgtime": 0.124169689, "sum": 791.58177341 }, "request": 6375, "root_rbytes": 8599552, "root_rfiles": 1103, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6377, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6377, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1259, "num_strays_delayed": 174, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3166, "strays_enqueued": 1907, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 13918, "evadd": 13913, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 18, "avgtime": 0.31385655, "sum": 5.649417901 }, "rdpos": 4195674, "replayed": 5, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 33323133 }, "mds_mem": { "cap": 1121, "cap+": 3202, "cap-": 2081, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5860, "dn+": 6377, "dn-": 517, "heap": 248164, "ino": 2697, "ino+": 3214, "ino-": 517, "rss": 113728 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6376, "dispatch_server_request": 0, "handle_client_request": 6375, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3166, "avgtime": 0.002254702, "sum": 7.138389617 } }, "mds_sessions": { "average_load": 5783, "avg_session_uptime": 939, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5783 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 28272366, "buffer_anon_items": 11338, "buffer_meta_bytes": 1848, "buffer_meta_items": 21, "mds_co_bytes": 16484771, "mds_co_items": 221035, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 23, "op": 1652, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 1646, "avgtime": 0.003417317, "sum": 5.624904109 }, "op_pg": 0, "op_r": 52, "op_reply": 1646, "op_resend": 0, "op_rmw": 0, "op_send": 1652, "op_send_bytes": 25807577, "op_w": 1600, "oplen_avg": { "avgcount": 1652, "sum": 1713 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1547, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 34, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1524, "pq_executed_ops": 1524, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 383 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13355, "get_or_fail_fail": 0, "get_or_fail_success": 13355, "get_started": 0, "get_sum": 3644819, "max": 104857600, "put": 13354, "put_sum": 3644566, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1646, "put_sum": 26002105, "take": 1652, "take_sum": 26002105, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1646, "put_sum": 1646, "take": 1652, "take_sum": 1652, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1907, "get_or_fail_fail": 0, "get_or_fail_success": 1907, "get_started": 0, "get_sum": 192607, "max": 3758096384, "put": 16, "put_sum": 192607, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 13913, "get_or_fail_fail": 0, "get_or_fail_success": 13913, "get_started": 0, "get_sum": 29126644, "max": 3758096384, "put": 18, "put_sum": 25163639, "take": 0, "take_sum": 0, "val": 3963005, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:14.670 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/1259, 1524/3166 2024-03-06T07:33:15.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.558+0000 7ff8fea00640 1 Processor -- start 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.558+0000 7ff8fea00640 1 -- start start 2024-03-06T07:33:15.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.559+0000 7ff8fea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8f80cebe0 0x7ff8f80cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.559+0000 7ff8fea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 0x7ff8f8162990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.559+0000 7ff8fea00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f815f070 con 0x7ff8f80701f0 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.559+0000 7ff8fea00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f815f350 con 0x7ff8f80cebe0 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.559+0000 7ff8fea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f815f630 con 0x7ff8f8162520 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.559+0000 7ff8ef400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8f80cebe0 0x7ff8f80cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.560+0000 7ff8ef400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8f80cebe0 0x7ff8f80cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40576/0 (socket says 172.21.15.97:40576) 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.560+0000 7ff8ef400640 1 -- 172.21.15.97:0/1741244885 learned_addr learned my addr 172.21.15.97:0/1741244885 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.560+0000 7ff8ef400640 1 -- 172.21.15.97:0/1741244885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 msgr2=0x7ff8f8162990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.560+0000 7ff8ef400640 1 --2- 172.21.15.97:0/1741244885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 0x7ff8f8162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.560+0000 7ff8ef400640 1 -- 172.21.15.97:0/1741244885 >> v1:172.21.15.135:6789/0 conn(0x7ff8f80701f0 legacy=0x7ff8f80cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.560+0000 7ff8ef400640 1 -- 172.21.15.97:0/1741244885 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f8162000 con 0x7ff8f80cebe0 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.561+0000 7ff8ef400640 1 --2- 172.21.15.97:0/1741244885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f80cf090 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009870 tx=0x7ff8e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197599160954de20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.561+0000 7ff8f7400640 1 -- 172.21.15.97:0/1741244885 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0035020 con 0x7ff8f80cebe0 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.561+0000 7ff8f7400640 1 -- 172.21.15.97:0/1741244885 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e0031640 con 0x7ff8f80cebe0 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.561+0000 7ff8f7400640 1 -- 172.21.15.97:0/1741244885 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e002d040 con 0x7ff8f80cebe0 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 -- 172.21.15.97:0/1741244885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 msgr2=0x7ff8f80cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 --2- 172.21.15.97:0/1741244885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f80cf090 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009870 tx=0x7ff8e002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 -- 172.21.15.97:0/1741244885 shutdown_connections 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 --2- 172.21.15.97:0/1741244885 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 0x7ff8f8162990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 --2- 172.21.15.97:0/1741244885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f80cf090 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 -- 172.21.15.97:0/1741244885 >> 172.21.15.97:0/1741244885 conn(0x7ff8f80d0d30 msgr2=0x7ff8f80d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 -- 172.21.15.97:0/1741244885 shutdown_connections 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.562+0000 7ff8fea00640 1 -- 172.21.15.97:0/1741244885 wait complete. 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.563+0000 7ff8fea00640 1 Processor -- start 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 -- start start 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8f80701f0 0x7ff8f80c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f81f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 0x7ff8f80aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f806b0c0 con 0x7ff8f80cebe0 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f806af40 con 0x7ff8f80701f0 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.564+0000 7ff8fea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f806b240 con 0x7ff8f8162520 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f81f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f81f7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40590/0 (socket says 172.21.15.97:40590) 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 -- 172.21.15.97:0/2813524916 learned_addr learned my addr 172.21.15.97:0/2813524916 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8f7e00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8f80701f0 0x7ff8f80c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 -- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 msgr2=0x7ff8f80aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 0x7ff8f80aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 -- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8f80701f0 msgr2=0x7ff8f80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8f80701f0 0x7ff8f80c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 -- 172.21.15.97:0/2813524916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f80c7f80 con 0x7ff8f80cebe0 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.565+0000 7ff8ef400640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f81f7d50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff8e002b930 tx=0x7ff8e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.566+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e003ac60 con 0x7ff8f80cebe0 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.566+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e003ae20 con 0x7ff8f80cebe0 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.566+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f81f8480 con 0x7ff8f80cebe0 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.566+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0044630 con 0x7ff8f80cebe0 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.566+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f81f8a40 con 0x7ff8f80cebe0 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.567+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f80b6a80 con 0x7ff8f80cebe0 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.569+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8e004b060 con 0x7ff8f80cebe0 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.570+0000 7ff8f4c00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8d406b4d0 0x7ff8d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.570+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff8e00ac0d0 con 0x7ff8f80cebe0 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.570+0000 7ff8f7e00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8d406b4d0 0x7ff8d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.571+0000 7ff8f7e00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8d406b4d0 0x7ff8d406d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80015d0 tx=0x7ff8e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.575+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8f80b6a80 con 0x7ff8f80cebe0 2024-03-06T07:33:15.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:15.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.871+0000 7fdfbb000640 1 Processor -- start 2024-03-06T07:33:15.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.871+0000 7fdfbb000640 1 -- start start 2024-03-06T07:33:15.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfbb000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdfb40d2400 0x7fdfb40d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfbb000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40b39d0 con 0x7fdfb40b4790 2024-03-06T07:33:15.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfbb000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40b3cb0 con 0x7fdfb415e230 2024-03-06T07:33:15.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfbb000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb40b3780 con 0x7fdfb40d2400 2024-03-06T07:33:15.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfba600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdfb40b4790 0x7fdfb40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37186/0 (socket says 172.21.15.97:37186) 2024-03-06T07:33:15.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfba600640 1 -- 172.21.15.97:0/3065664832 learned_addr learned my addr 172.21.15.97:0/3065664832 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:15.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043832706 0 0) 0x7fdfb40b39d0 con 0x7fdfb40b4790 2024-03-06T07:33:15.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.873+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c003520 con 0x7fdfb40b4790 2024-03-06T07:33:15.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb9200640 1 --2- 172.21.15.97:0/3065664832 >> v2:172.21.15.135:3300/0 conn(0x7fdfb40d2400 0x7fdfb40d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3508194079 0 0) 0x7fdf9c003520 con 0x7fdfb40b4790 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 >> v2:172.21.15.135:3300/0 conn(0x7fdfb40d2400 msgr2=0x7fdfb40d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 --2- 172.21.15.97:0/3065664832 >> v2:172.21.15.135:3300/0 conn(0x7fdfb40d2400 0x7fdfb40d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 >> v1:172.21.15.135:6789/0 conn(0x7fdfb415e230 legacy=0x7fdfb40d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb40b3f90 con 0x7fdfb40b4790 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdfac0019f0 con 0x7fdfb40b4790 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 >> v1:172.21.15.97:6789/0 conn(0x7fdfb40b4790 legacy=0x7fdfb40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c003a00 0x7fdf9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf9c004800 0x7fdf9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c005210 0x7fdf9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40b3780 con 0x7fdf9c003a00 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c00c150 con 0x7fdf9c004800 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c00c350 con 0x7fdf9c005210 2024-03-06T07:33:15.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.874+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c003a00 0x7fdf9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfb9200640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf9c004800 0x7fdf9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfba600640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c005210 0x7fdf9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfb9c00640 1 -- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c005210 msgr2=0x7fdf9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c005210 0x7fdf9c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfb9c00640 1 -- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf9c004800 msgr2=0x7fdf9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf9c004800 0x7fdf9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfb9c00640 1 -- 172.21.15.97:0/3065664832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c003520 con 0x7fdf9c003a00 2024-03-06T07:33:15.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.875+0000 7fdfba600640 1 -- 172.21.15.97:0/3065664832 reap_dead start 2024-03-06T07:33:15.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.876+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c003a00 0x7fdf9c003e20 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8001750 tx=0x7fdf9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663cdaa1b179884c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.876+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8001370 con 0x7fdf9c003a00 2024-03-06T07:33:15.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.876+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa80052d0 con 0x7fdf9c003a00 2024-03-06T07:33:15.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.876+0000 7fdfb3e00640 1 -- 172.21.15.97:0/3065664832 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8003830 con 0x7fdf9c003a00 2024-03-06T07:33:15.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.876+0000 7fdfbb000640 1 -- 172.21.15.97:0/3065664832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c003a00 msgr2=0x7fdf9c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.876+0000 7fdfbb000640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c003a00 0x7fdf9c003e20 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8001750 tx=0x7fdf9c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.877+0000 7fdfbb000640 1 -- 172.21.15.97:0/3065664832 shutdown_connections 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.877+0000 7fdfbb000640 1 --2- 172.21.15.97:0/3065664832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c003a00 0x7fdf9c003e20 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.877+0000 7fdfbb000640 1 -- 172.21.15.97:0/3065664832 >> 172.21.15.97:0/3065664832 conn(0x7fdfb40d3e20 msgr2=0x7fdfb40d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.877+0000 7fdfbb000640 1 -- 172.21.15.97:0/3065664832 shutdown_connections 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.877+0000 7fdfbb000640 1 -- 172.21.15.97:0/3065664832 wait complete. 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.879+0000 7fdfbb000640 1 Processor -- start 2024-03-06T07:33:15.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.880+0000 7fdfbb000640 1 -- start start 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfbb000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c003a00 0x7fdfb414e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfbb000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 0x7fdfb4160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfbb000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfb415eca0 0x7fdfb4160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfbb000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40d1d90 con 0x7fdfb415e820 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfbb000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb415e610 con 0x7fdfb415eca0 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfbb000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40d1f10 con 0x7fdf9c003a00 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 0x7fdfb4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 0x7fdfb4160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40608/0 (socket says 172.21.15.97:40608) 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.881+0000 7fdfb9200640 1 -- 172.21.15.97:0/1318837924 learned_addr learned my addr 172.21.15.97:0/1318837924 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c003a00 0x7fdfb414e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfba600640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfb415eca0 0x7fdfb4160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9200640 1 -- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c003a00 msgr2=0x7fdfb414e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9200640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c003a00 0x7fdfb414e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9200640 1 -- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfb415eca0 msgr2=0x7fdfb4160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9200640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfb415eca0 0x7fdfb4160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9200640 1 -- 172.21.15.97:0/1318837924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb41617b0 con 0x7fdfb415e820 2024-03-06T07:33:15.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb9200640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 0x7fdfb4160420 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000f530 tx=0x7fdfa000f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0019020 con 0x7fdfb415e820 2024-03-06T07:33:15.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa0014870 con 0x7fdfb415e820 2024-03-06T07:33:15.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0014c60 con 0x7fdfb415e820 2024-03-06T07:33:15.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.882+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb40cb330 con 0x7fdfb415e820 2024-03-06T07:33:15.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.883+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb41611f0 con 0x7fdfb415e820 2024-03-06T07:33:15.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.883+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8f8068070 con 0x7ff8f80cebe0 2024-03-06T07:33:15.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.884+0000 7ff8f4c00640 1 -- 172.21.15.97:0/2813524916 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7ff8e003f070 con 0x7ff8f80cebe0 2024-03-06T07:33:15.886 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:15.886 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:15.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.884+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb40d3690 con 0x7fdfb415e820 2024-03-06T07:33:15.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.885+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfa001c020 con 0x7fdfb415e820 2024-03-06T07:33:15.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.886+0000 7fdfb1600640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf9406b4d0 0x7fdf9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.886+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdfa008a900 con 0x7fdfb415e820 2024-03-06T07:33:15.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.886+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf9406b4d0 0x7fdf9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8d406b4d0 msgr2=0x7ff8d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8d406b4d0 0x7ff8d406d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80015d0 tx=0x7ff8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 msgr2=0x7ff8f81f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f81f7d50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff8e002b930 tx=0x7ff8e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 shutdown_connections 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff8d406b4d0 0x7ff8d406d990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff8f8162520 0x7ff8f80aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8f80cebe0 0x7ff8f81f7d50 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 --2- 172.21.15.97:0/2813524916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8f80701f0 0x7ff8f80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 >> 172.21.15.97:0/2813524916 conn(0x7ff8f80d0d30 msgr2=0x7ff8f806e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 shutdown_connections 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7ff8fea00640 1 -- 172.21.15.97:0/2813524916 wait complete. 2024-03-06T07:33:15.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.887+0000 7fdfb9c00640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf9406b4d0 0x7fdf9406d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8033000 tx=0x7fdfa8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:15.894+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfb40d3690 con 0x7fdfb415e820 2024-03-06T07:33:15.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:33:16.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.241+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfb4072a40 con 0x7fdfb415e820 2024-03-06T07:33:16.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.241+0000 7fdfb1600640 1 -- 172.21.15.97:0/1318837924 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fdfa0023020 con 0x7fdfb415e820 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.246+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf9406b4d0 msgr2=0x7fdf9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.246+0000 7fdfbb000640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf9406b4d0 0x7fdf9406d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8033000 tx=0x7fdfa8009030 comp rx=0 tx=0).stop 2024-03-06T07:33:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.246+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 msgr2=0x7fdfb4160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.246+0000 7fdfbb000640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 0x7fdfb4160420 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000f530 tx=0x7fdfa000f9f0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.246+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 shutdown_connections 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf9406b4d0 0x7fdf9406d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdfb415eca0 0x7fdfb4160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfb415e820 0x7fdfb4160420 secure :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000f530 tx=0x7fdfa000f9f0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 --2- 172.21.15.97:0/1318837924 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c003a00 0x7fdfb414e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 >> 172.21.15.97:0/1318837924 conn(0x7fdfb40d3e20 msgr2=0x7fdfb40d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 shutdown_connections 2024-03-06T07:33:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:16.247+0000 7fdfbb000640 1 -- 172.21.15.97:0/1318837924 wait complete. 2024-03-06T07:33:16.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:16.346 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:33:16.346 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:33:16.518 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1004,"msgr_send_messages":1008,"msgr_recv_bytes":561330,"msgr_send_bytes":18593599,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.235495856,"msgr_running_send_time":0.136422293,"msgr_running_recv_time":0.033860310,"msgr_running_fast_dispatch_time":0.053067297,"msgr_send_messages_queue_lat":{"avgcount":1007,"sum":0.311426234,"avgtime":0.000309261},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94512},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13213,"msgr_send_messages":24093,"msgr_recv_bytes":4544360,"msgr_send_bytes":13566991,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.798008991,"msgr_running_send_time":1.808585108,"msgr_running_recv_time":1.577709207,"msgr_running_fast_dispatch_time":0.052859713,"msgr_send_messages_queue_lat":{"avgcount":24092,"sum":1.943138866,"avgtime":0.000080654},"msgr_handle_ack_lat":{"avgcount":26571,"sum":0.011783039,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":927,"msgr_send_messages":923,"msgr_recv_bytes":348154,"msgr_send_bytes":4578502,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.137028297,"msgr_running_send_time":0.062915047,"msgr_running_recv_time":0.032863928,"msgr_running_fast_dispatch_time":0.035936811,"msgr_send_messages_queue_lat":{"avgcount":923,"sum":0.288824964,"avgtime":0.000312919},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106464,"msgr_send_encrypted_bytes":7296},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":120,"sum":11.162994772,"avgtime":0.093024956}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5049,"sum":2.204120576,"avgtime":0.000436545}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3991,"inodes_top":0,"inodes_bottom":3991,"inodes_pin_tail":0,"inodes_pinned":3991,"inodes_expired":0,"inodes_with_caps":271,"caps":271,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":392,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":270,"num_strays_delayed":270,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2931,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15000,"evex":15002,"evtrm":15002,"ev":15004,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34088257,"rdpos":4195674,"jlat":{"avgcount":26,"sum":7.254144011,"avgtime":0.279005538},"replayed":5},"mds_mem":{"ino":1164,"ino+":3214,"ino-":2050,"dir":13,"dir+":13,"dir-":0,"dn":4362,"dn+":6412,"dn-":2050,"cap":271,"cap+":3202,"cap-":2931,"rss":118608,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":944,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1218178,"buffer_anon_items":2588,"buffer_meta_bytes":264,"buffer_meta_items":3,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10153434,"mds_co_items":153652,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3387,"op_send_bytes":30922896,"op_resend":0,"op_reply":3387,"op_latency":{"avgcount":3387,"sum":18.979676358,"avgtime":0.005603683},"op_inflight":0,"oplen_avg":{"avgcount":3387,"sum":4036},"op":3387,"op_r":58,"op_w":3329,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":30,"osdop_write":48,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2956,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":2931,"pq_executed":2931,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15140,"get_sum":4088522,"get_or_fail_fail":0,"get_or_fail_success":15140,"take":0,"take_sum":0,"put":15140,"put_sum":4088522,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3387,"take_sum":31220848,"put":3387,"put_sum":31220848,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3387,"take_sum":3387,"put":3387,"put_sum":3387,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2931,"get_sum":296031,"get_or_fail_fail":0,"get_or_fail_success":2931,"take":0,"take_sum":0,"put":22,"put_sum":296031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15000,"get_sum":29891768,"get_or_fail_fail":0,"get_or_fail_success":15000,"take":0,"take_sum":0,"put":26,"put_sum":29891768,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:16.520 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 561330, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1004, "msgr_running_fast_dispatch_time": 0.053067297, "msgr_running_recv_time": 0.03386031, "msgr_running_send_time": 0.136422293, "msgr_running_total_time": 0.235495856, "msgr_send_bytes": 18593599, "msgr_send_encrypted_bytes": 94512, "msgr_send_messages": 1008, "msgr_send_messages_queue_lat": { "avgcount": 1007, "avgtime": 0.000309261, "sum": 0.311426234 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26571, "avgtime": 4.43e-07, "sum": 0.011783039 }, "msgr_recv_bytes": 4544360, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13213, "msgr_running_fast_dispatch_time": 0.052859713, "msgr_running_recv_time": 1.577709207, "msgr_running_send_time": 1.808585108, "msgr_running_total_time": 2.798008991, "msgr_send_bytes": 13566991, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24093, "msgr_send_messages_queue_lat": { "avgcount": 24092, "avgtime": 8.0654e-05, "sum": 1.943138866 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 348154, "msgr_recv_encrypted_bytes": 106464, "msgr_recv_messages": 927, "msgr_running_fast_dispatch_time": 0.035936811, "msgr_running_recv_time": 0.032863928, "msgr_running_send_time": 0.062915047, "msgr_running_total_time": 0.137028297, "msgr_send_bytes": 4578502, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 923, "msgr_send_messages_queue_lat": { "avgcount": 923, "avgtime": 0.000312919, "sum": 0.288824964 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 120, "avgtime": 0.093024956, "sum": 11.162994772 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5049, "avgtime": 0.000436545, "sum": 2.204120576 }, "queue_len": 0 }, "mds": { "caps": 271, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 392, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3991, "inodes_bottom": 3991, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3991, "inodes_top": 0, "inodes_with_caps": 271, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 270, "num_strays_delayed": 270, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2931, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15004, "evadd": 15000, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 26, "avgtime": 0.279005538, "sum": 7.254144011 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34088257 }, "mds_mem": { "cap": 271, "cap+": 3202, "cap-": 2931, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4362, "dn+": 6412, "dn-": 2050, "heap": 248164, "ino": 1164, "ino+": 3214, "ino-": 2050, "rss": 118608 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 944, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1218178, "buffer_anon_items": 2588, "buffer_meta_bytes": 264, "buffer_meta_items": 3, "mds_co_bytes": 10153434, "mds_co_items": 153652, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 53, "op": 3387, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3387, "avgtime": 0.005603683, "sum": 18.979676358 }, "op_pg": 0, "op_r": 58, "op_reply": 3387, "op_resend": 0, "op_rmw": 0, "op_send": 3387, "op_send_bytes": 30922896, "op_w": 3329, "oplen_avg": { "avgcount": 3387, "sum": 4036 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 2956, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2931, "pq_executed_ops": 2931, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15140, "get_or_fail_fail": 0, "get_or_fail_success": 15140, "get_started": 0, "get_sum": 4088522, "max": 104857600, "put": 15140, "put_sum": 4088522, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3387, "put_sum": 31220848, "take": 3387, "take_sum": 31220848, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3387, "put_sum": 3387, "take": 3387, "take_sum": 3387, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2931, "get_or_fail_fail": 0, "get_or_fail_success": 2931, "get_started": 0, "get_sum": 296031, "max": 3758096384, "put": 22, "put_sum": 296031, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15000, "get_or_fail_fail": 0, "get_or_fail_success": 15000, "get_started": 0, "get_sum": 29891768, "max": 3758096384, "put": 26, "put_sum": 29891768, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:16.520 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/270, 2931/3201 2024-03-06T07:33:17.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:17.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.734+0000 7f3acc600640 1 Processor -- start 2024-03-06T07:33:17.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.734+0000 7f3acc600640 1 -- start start 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3acc600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3ac40acd60 0x7f3ac40b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3acc600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac40b5160 con 0x7f3ac415be90 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3acc600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac40ae910 con 0x7f3ac40d0780 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3acc600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac40b4f30 con 0x7f3ac40acd60 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3ac9c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3ac415be90 0x7f3ac40d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37188/0 (socket says 172.21.15.97:37188) 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3ac9c00640 1 -- 172.21.15.97:0/3649587039 learned_addr learned my addr 172.21.15.97:0/3649587039 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.736+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755313971 0 0) 0x7f3ac40b5160 con 0x7f3ac415be90 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac003520 con 0x7f3ac415be90 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3aca600640 1 --2- 172.21.15.97:0/3649587039 >> v2:172.21.15.135:3300/0 conn(0x7f3ac40acd60 0x7f3ac40b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3732792096 0 0) 0x7f3aac003520 con 0x7f3ac415be90 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 >> v2:172.21.15.135:3300/0 conn(0x7f3ac40acd60 msgr2=0x7f3ac40b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/3649587039 >> v2:172.21.15.135:3300/0 conn(0x7f3ac40acd60 0x7f3ac40b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 >> v1:172.21.15.135:6789/0 conn(0x7f3ac40d0780 legacy=0x7f3ac415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40aebf0 con 0x7f3ac415be90 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3ab8001670 con 0x7f3ac415be90 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 >> v1:172.21.15.97:6789/0 conn(0x7f3ac415be90 legacy=0x7f3ac40d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac003a00 0x7f3aac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3aac004800 0x7f3aac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3aac0052e0 0x7f3aac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00c1d0 con 0x7f3aac004800 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00c380 con 0x7f3aac0052e0 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.737+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00c580 con 0x7f3aac003a00 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9200640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3aac004800 0x7f3aac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9c00640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac003a00 0x7f3aac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3aca600640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3aac0052e0 0x7f3aac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9200640 1 -- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac003a00 msgr2=0x7f3aac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9200640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac003a00 0x7f3aac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9200640 1 -- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3aac0052e0 msgr2=0x7f3aac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9200640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3aac0052e0 0x7f3aac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3ac9200640 1 -- 172.21.15.97:0/3649587039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac003520 con 0x7f3aac004800 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.738+0000 7f3aca600640 1 -- 172.21.15.97:0/3649587039 reap_dead start 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.739+0000 7f3ac9200640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3aac004800 0x7f3aac004c80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0001750 tx=0x7f3aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af752e1fae7c34c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.739+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0001370 con 0x7f3aac004800 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.739+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab00052d0 con 0x7f3aac004800 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.739+0000 7f3ac3e00640 1 -- 172.21.15.97:0/3649587039 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0003830 con 0x7f3aac004800 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 -- 172.21.15.97:0/3649587039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3aac004800 msgr2=0x7f3aac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3aac004800 0x7f3aac004c80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0001750 tx=0x7f3aac005830 comp rx=0 tx=0).stop 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 -- 172.21.15.97:0/3649587039 shutdown_connections 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 --2- 172.21.15.97:0/3649587039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3aac004800 0x7f3aac004c80 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 -- 172.21.15.97:0/3649587039 >> 172.21.15.97:0/3649587039 conn(0x7f3ac406b360 msgr2=0x7f3ac406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 -- 172.21.15.97:0/3649587039 shutdown_connections 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.740+0000 7f3acc600640 1 -- 172.21.15.97:0/3649587039 wait complete. 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.742+0000 7f3acc600640 1 Processor -- start 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.742+0000 7f3acc600640 1 -- start start 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3acc600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac004800 0x7f3ac41f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3acc600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 0x7f3ac415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3acc600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40cd2a0 0x7f3ac415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3acc600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40b2c80 con 0x7f3ac40cce20 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3acc600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40af240 con 0x7f3ac40cd2a0 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3acc600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40b2e00 con 0x7f3aac004800 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3ac9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 0x7f3ac415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3ac9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 0x7f3ac415acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40616/0 (socket says 172.21.15.97:40616) 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3ac9200640 1 -- 172.21.15.97:0/2037490324 learned_addr learned my addr 172.21.15.97:0/2037490324 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.743+0000 7f3ac9c00640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac004800 0x7f3ac41f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3aca600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40cd2a0 0x7f3ac415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac9200640 1 -- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac004800 msgr2=0x7f3ac41f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac9200640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac004800 0x7f3ac41f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac9200640 1 -- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40cd2a0 msgr2=0x7f3ac415b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac9200640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40cd2a0 0x7f3ac415b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac9200640 1 -- 172.21.15.97:0/2037490324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40cd750 con 0x7f3ac40cce20 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac9200640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 0x7f3ac415acb0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0005040 tx=0x7f3ab0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0041020 con 0x7f3ac40cce20 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0005480 con 0x7f3ac40cce20 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac40cda30 con 0x7f3ac40cce20 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.744+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0007a30 con 0x7f3ac40cce20 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.745+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac40cdff0 con 0x7f3ac40cce20 2024-03-06T07:33:17.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.746+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac4068070 con 0x7f3ac40cce20 2024-03-06T07:33:17.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.746+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ab0044020 con 0x7f3ac40cce20 2024-03-06T07:33:17.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.748+0000 7f3ac1600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b500 0x7f3aa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.748+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ab00abcc0 con 0x7f3ac40cce20 2024-03-06T07:33:17.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.748+0000 7f3ac9c00640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b500 0x7f3aa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.749+0000 7f3ac9c00640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b500 0x7f3aa406d9c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40b0590 tx=0x7f3ab8001db0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.752+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ab00b2080 con 0x7f3ac40cce20 2024-03-06T07:33:17.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.989+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ac41f4ce0 con 0x7f3ac40cce20 2024-03-06T07:33:17.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.990+0000 7f3ac1600640 1 -- 172.21.15.97:0/2037490324 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f3ab004b030 con 0x7f3ac40cce20 2024-03-06T07:33:17.992 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:17.992 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:17.993 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b500 msgr2=0x7f3aa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b500 0x7f3aa406d9c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40b0590 tx=0x7f3ab8001db0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 msgr2=0x7f3ac415acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 0x7f3ac415acb0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0005040 tx=0x7f3ab0001120 comp rx=0 tx=0).stop 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 shutdown_connections 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa406b500 0x7f3aa406d9c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac40cd2a0 0x7f3ac415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac40cce20 0x7f3ac415acb0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 --2- 172.21.15.97:0/2037490324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3aac004800 0x7f3ac41f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.992+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 >> 172.21.15.97:0/2037490324 conn(0x7f3ac406b360 msgr2=0x7f3ac406c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.993+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 shutdown_connections 2024-03-06T07:33:17.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:17.993+0000 7f3acc600640 1 -- 172.21.15.97:0/2037490324 wait complete. 2024-03-06T07:33:18.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:18.245 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1004,"msgr_send_messages":1009,"msgr_recv_bytes":561330,"msgr_send_bytes":18595391,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.235599808,"msgr_running_send_time":0.136510401,"msgr_running_recv_time":0.033860310,"msgr_running_fast_dispatch_time":0.053067297,"msgr_send_messages_queue_lat":{"avgcount":1008,"sum":0.311471475,"avgtime":0.000308999},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96304},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13215,"msgr_send_messages":24093,"msgr_recv_bytes":4545196,"msgr_send_bytes":13566991,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.798576052,"msgr_running_send_time":1.808862444,"msgr_running_recv_time":1.577826558,"msgr_running_fast_dispatch_time":0.053024605,"msgr_send_messages_queue_lat":{"avgcount":24092,"sum":1.943138866,"avgtime":0.000080654},"msgr_handle_ack_lat":{"avgcount":26573,"sum":0.011783725,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":928,"msgr_send_messages":924,"msgr_recv_bytes":348426,"msgr_send_bytes":4579126,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.137341425,"msgr_running_send_time":0.063082503,"msgr_running_recv_time":0.032926046,"msgr_running_fast_dispatch_time":0.035972456,"msgr_send_messages_queue_lat":{"avgcount":924,"sum":0.288908040,"avgtime":0.000312671},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106736,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":120,"sum":11.162994772,"avgtime":0.093024956}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5049,"sum":2.204120576,"avgtime":0.000436545}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3991,"inodes_top":0,"inodes_bottom":3991,"inodes_pin_tail":0,"inodes_pinned":3991,"inodes_expired":0,"inodes_with_caps":271,"caps":271,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":392,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":270,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2931,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15000,"evex":15002,"evtrm":15002,"ev":15004,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34088257,"rdpos":4195674,"jlat":{"avgcount":26,"sum":7.254144011,"avgtime":0.279005538},"replayed":5},"mds_mem":{"ino":283,"ino+":3214,"ino-":2931,"dir":13,"dir+":13,"dir-":0,"dn":281,"dn+":6412,"dn-":6131,"cap":271,"cap+":3202,"cap-":2931,"rss":118864,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":944,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1218178,"buffer_anon_items":2588,"buffer_meta_bytes":264,"buffer_meta_items":3,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10153434,"mds_co_items":153652,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3387,"op_send_bytes":30922896,"op_resend":0,"op_reply":3387,"op_latency":{"avgcount":3387,"sum":18.979676358,"avgtime":0.005603683},"op_inflight":0,"oplen_avg":{"avgcount":3387,"sum":4036},"op":3387,"op_r":58,"op_w":3329,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":30,"osdop_write":48,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2956,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":2931,"pq_executed":2931,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15143,"get_sum":4089308,"get_or_fail_fail":0,"get_or_fail_success":15143,"take":0,"take_sum":0,"put":15143,"put_sum":4089308,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3387,"take_sum":31220848,"put":3387,"put_sum":31220848,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3387,"take_sum":3387,"put":3387,"put_sum":3387,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2931,"get_sum":296031,"get_or_fail_fail":0,"get_or_fail_success":2931,"take":0,"take_sum":0,"put":22,"put_sum":296031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15000,"get_sum":29891768,"get_or_fail_fail":0,"get_or_fail_success":15000,"take":0,"take_sum":0,"put":26,"put_sum":29891768,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:18.247 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 561330, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1004, "msgr_running_fast_dispatch_time": 0.053067297, "msgr_running_recv_time": 0.03386031, "msgr_running_send_time": 0.136510401, "msgr_running_total_time": 0.235599808, "msgr_send_bytes": 18595391, "msgr_send_encrypted_bytes": 96304, "msgr_send_messages": 1009, "msgr_send_messages_queue_lat": { "avgcount": 1008, "avgtime": 0.000308999, "sum": 0.311471475 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26573, "avgtime": 4.43e-07, "sum": 0.011783725 }, "msgr_recv_bytes": 4545196, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13215, "msgr_running_fast_dispatch_time": 0.053024605, "msgr_running_recv_time": 1.577826558, "msgr_running_send_time": 1.808862444, "msgr_running_total_time": 2.798576052, "msgr_send_bytes": 13566991, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24093, "msgr_send_messages_queue_lat": { "avgcount": 24092, "avgtime": 8.0654e-05, "sum": 1.943138866 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 348426, "msgr_recv_encrypted_bytes": 106736, "msgr_recv_messages": 928, "msgr_running_fast_dispatch_time": 0.035972456, "msgr_running_recv_time": 0.032926046, "msgr_running_send_time": 0.063082503, "msgr_running_total_time": 0.137341425, "msgr_send_bytes": 4579126, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 924, "msgr_send_messages_queue_lat": { "avgcount": 924, "avgtime": 0.000312671, "sum": 0.28890804 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 120, "avgtime": 0.093024956, "sum": 11.162994772 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5049, "avgtime": 0.000436545, "sum": 2.204120576 }, "queue_len": 0 }, "mds": { "caps": 271, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 392, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3991, "inodes_bottom": 3991, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3991, "inodes_top": 0, "inodes_with_caps": 271, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 270, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2931, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15004, "evadd": 15000, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 26, "avgtime": 0.279005538, "sum": 7.254144011 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34088257 }, "mds_mem": { "cap": 271, "cap+": 3202, "cap-": 2931, "dir": 13, "dir+": 13, "dir-": 0, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 283, "ino+": 3214, "ino-": 2931, "rss": 118864 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 944, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1218178, "buffer_anon_items": 2588, "buffer_meta_bytes": 264, "buffer_meta_items": 3, "mds_co_bytes": 10153434, "mds_co_items": 153652, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 53, "op": 3387, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3387, "avgtime": 0.005603683, "sum": 18.979676358 }, "op_pg": 0, "op_r": 58, "op_reply": 3387, "op_resend": 0, "op_rmw": 0, "op_send": 3387, "op_send_bytes": 30922896, "op_w": 3329, "oplen_avg": { "avgcount": 3387, "sum": 4036 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 2956, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2931, "pq_executed_ops": 2931, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15143, "get_or_fail_fail": 0, "get_or_fail_success": 15143, "get_started": 0, "get_sum": 4089308, "max": 104857600, "put": 15143, "put_sum": 4089308, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3387, "put_sum": 31220848, "take": 3387, "take_sum": 31220848, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3387, "put_sum": 3387, "take": 3387, "take_sum": 3387, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2931, "get_or_fail_fail": 0, "get_or_fail_success": 2931, "get_started": 0, "get_sum": 296031, "max": 3758096384, "put": 22, "put_sum": 296031, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15000, "get_or_fail_fail": 0, "get_or_fail_success": 15000, "get_started": 0, "get_sum": 29891768, "max": 3758096384, "put": 26, "put_sum": 29891768, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:18.247 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/270, 2931/3201 2024-03-06T07:33:19.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:19.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.451+0000 7fee42800640 1 Processor -- start 2024-03-06T07:33:19.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.451+0000 7fee42800640 1 -- start start 2024-03-06T07:33:19.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee42800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee3c0b3340 0x7fee3c0d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee42800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c15ee30 0x7fee3c0d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee42800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c15f370 con 0x7fee3c06d600 2024-03-06T07:33:19.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee42800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3c15f650 con 0x7fee3c0b3340 2024-03-06T07:33:19.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee42800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c06cb20 con 0x7fee3c15ee30 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee3b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee3c0b3340 0x7fee3c0d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee3b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee3c0b3340 0x7fee3c0d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40618/0 (socket says 172.21.15.97:40618) 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.453+0000 7fee3b400640 1 -- 172.21.15.97:0/4241581249 learned_addr learned my addr 172.21.15.97:0/4241581249 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.454+0000 7fee3be00640 1 --2- 172.21.15.97:0/4241581249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c15ee30 0x7fee3c0d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.454+0000 7fee3b400640 1 -- 172.21.15.97:0/4241581249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c15ee30 msgr2=0x7fee3c0d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.454+0000 7fee3b400640 1 --2- 172.21.15.97:0/4241581249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c15ee30 0x7fee3c0d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.454+0000 7fee3b400640 1 -- 172.21.15.97:0/4241581249 >> v1:172.21.15.135:6789/0 conn(0x7fee3c06d600 legacy=0x7fee3c15e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.454+0000 7fee3b400640 1 -- 172.21.15.97:0/4241581249 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c06ce00 con 0x7fee3c0b3340 2024-03-06T07:33:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.455+0000 7fee3b400640 1 --2- 172.21.15.97:0/4241581249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c0b3340 0x7fee3c0d0070 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fee3c06ed00 tx=0x7fee2c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16abebf2076f62dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.455+0000 7fee3a000640 1 -- 172.21.15.97:0/4241581249 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c02c930 con 0x7fee3c0b3340 2024-03-06T07:33:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.455+0000 7fee3a000640 1 -- 172.21.15.97:0/4241581249 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c02cad0 con 0x7fee3c0b3340 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.455+0000 7fee3a000640 1 -- 172.21.15.97:0/4241581249 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c0343f0 con 0x7fee3c0b3340 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 -- 172.21.15.97:0/4241581249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c0b3340 msgr2=0x7fee3c0d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 --2- 172.21.15.97:0/4241581249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c0b3340 0x7fee3c0d0070 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fee3c06ed00 tx=0x7fee2c02bd90 comp rx=0 tx=0).stop 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 -- 172.21.15.97:0/4241581249 shutdown_connections 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 --2- 172.21.15.97:0/4241581249 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c15ee30 0x7fee3c0d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 --2- 172.21.15.97:0/4241581249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c0b3340 0x7fee3c0d0070 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 -- 172.21.15.97:0/4241581249 >> 172.21.15.97:0/4241581249 conn(0x7fee3c06b360 msgr2=0x7fee3c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 -- 172.21.15.97:0/4241581249 shutdown_connections 2024-03-06T07:33:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.456+0000 7fee42800640 1 -- 172.21.15.97:0/4241581249 wait complete. 2024-03-06T07:33:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.458+0000 7fee42800640 1 Processor -- start 2024-03-06T07:33:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.458+0000 7fee42800640 1 -- start start 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee42800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c06d600 0x7fee3c1f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee42800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c0b3340 0x7fee3c09cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee42800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 0x7fee3c160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee42800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0acd60 con 0x7fee3c15ee30 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee42800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0b3720 con 0x7fee3c06d600 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee42800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0acee0 con 0x7fee3c0b3340 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 0x7fee3c160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 0x7fee3c160fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40628/0 (socket says 172.21.15.97:40628) 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.459+0000 7fee3be00640 1 -- 172.21.15.97:0/2148000104 learned_addr learned my addr 172.21.15.97:0/2148000104 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3b400640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c06d600 0x7fee3c1f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3be00640 1 -- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c0b3340 msgr2=0x7fee3c09cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3aa00640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c0b3340 0x7fee3c09cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3be00640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c0b3340 0x7fee3c09cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3be00640 1 -- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c06d600 msgr2=0x7fee3c1f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3be00640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c06d600 0x7fee3c1f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3be00640 1 -- 172.21.15.97:0/2148000104 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c09c3a0 con 0x7fee3c15ee30 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3b400640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c06d600 0x7fee3c1f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3aa00640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c0b3340 0x7fee3c09cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee3be00640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 0x7fee3c160fc0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fee3400c4b0 tx=0x7fee3400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34015070 con 0x7fee3c15ee30 2024-03-06T07:33:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee34004220 con 0x7fee3c15ee30 2024-03-06T07:33:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34003610 con 0x7fee3c15ee30 2024-03-06T07:33:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c09b820 con 0x7fee3c15ee30 2024-03-06T07:33:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.460+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c09bde0 con 0x7fee3c15ee30 2024-03-06T07:33:19.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.461+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c068070 con 0x7fee3c15ee30 2024-03-06T07:33:19.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.463+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee340037b0 con 0x7fee3c15ee30 2024-03-06T07:33:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.464+0000 7fee33400640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee1806b4d0 0x7fee1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.464+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fee34089f30 con 0x7fee3c15ee30 2024-03-06T07:33:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.464+0000 7fee3b400640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee1806b4d0 0x7fee1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.464+0000 7fee3b400640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee1806b4d0 0x7fee1806d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fee2c035000 tx=0x7fee2c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.473+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee34058d70 con 0x7fee3c15ee30 2024-03-06T07:33:19.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.703+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee3c072da0 con 0x7fee3c15ee30 2024-03-06T07:33:19.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.704+0000 7fee33400640 1 -- 172.21.15.97:0/2148000104 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fee34058670 con 0x7fee3c15ee30 2024-03-06T07:33:19.707 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:19.707 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:19.707 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:19.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee1806b4d0 msgr2=0x7fee1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee1806b4d0 0x7fee1806d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fee2c035000 tx=0x7fee2c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 msgr2=0x7fee3c160fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 0x7fee3c160fc0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fee3400c4b0 tx=0x7fee3400c970 comp rx=0 tx=0).stop 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 shutdown_connections 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee1806b4d0 0x7fee1806d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee3c15ee30 0x7fee3c160fc0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee3c0b3340 0x7fee3c09cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 --2- 172.21.15.97:0/2148000104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c06d600 0x7fee3c1f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 >> 172.21.15.97:0/2148000104 conn(0x7fee3c06b360 msgr2=0x7fee3c071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 shutdown_connections 2024-03-06T07:33:19.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:19.707+0000 7fee42800640 1 -- 172.21.15.97:0/2148000104 wait complete. 2024-03-06T07:33:19.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:19.932 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1005,"msgr_send_messages":1010,"msgr_recv_bytes":561622,"msgr_send_bytes":18595812,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.235743710,"msgr_running_send_time":0.136575960,"msgr_running_recv_time":0.033890540,"msgr_running_fast_dispatch_time":0.053102332,"msgr_send_messages_queue_lat":{"avgcount":1009,"sum":0.311522416,"avgtime":0.000308743},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96304},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13218,"msgr_send_messages":24093,"msgr_recv_bytes":4548362,"msgr_send_bytes":13566991,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.799420025,"msgr_running_send_time":1.809202735,"msgr_running_recv_time":1.578175669,"msgr_running_fast_dispatch_time":0.053244932,"msgr_send_messages_queue_lat":{"avgcount":24092,"sum":1.943138866,"avgtime":0.000080654},"msgr_handle_ack_lat":{"avgcount":26576,"sum":0.011784280,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":928,"msgr_send_messages":924,"msgr_recv_bytes":348426,"msgr_send_bytes":4579126,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.137341425,"msgr_running_send_time":0.063082503,"msgr_running_recv_time":0.032926046,"msgr_running_fast_dispatch_time":0.035972456,"msgr_send_messages_queue_lat":{"avgcount":924,"sum":0.288908040,"avgtime":0.000312671},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106736,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":121,"sum":11.163005709,"avgtime":0.092256245}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5049,"sum":2.204120576,"avgtime":0.000436545}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3991,"inodes_top":0,"inodes_bottom":3991,"inodes_pin_tail":0,"inodes_pinned":3991,"inodes_expired":0,"inodes_with_caps":271,"caps":271,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":270,"num_strays_delayed":93,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2931,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15000,"evex":15002,"evtrm":15002,"ev":15004,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34088257,"rdpos":4195674,"jlat":{"avgcount":26,"sum":7.254144011,"avgtime":0.279005538},"replayed":5},"mds_mem":{"ino":283,"ino+":3214,"ino-":2931,"dir":13,"dir+":13,"dir-":0,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":118864,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":944,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1218178,"buffer_anon_items":2588,"buffer_meta_bytes":264,"buffer_meta_items":3,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10153434,"mds_co_items":153652,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3388,"op_send_bytes":30922957,"op_resend":0,"op_reply":3388,"op_latency":{"avgcount":3388,"sum":18.982676307,"avgtime":0.005602915},"op_inflight":0,"oplen_avg":{"avgcount":3388,"sum":4038},"op":3388,"op_r":58,"op_w":3330,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":30,"osdop_write":48,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2956,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":54,"omap_rd":48,"omap_del":9},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":2931,"pq_executed":2931,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15147,"get_sum":4092406,"get_or_fail_fail":0,"get_or_fail_success":15147,"take":0,"take_sum":0,"put":15147,"put_sum":4092406,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3388,"take_sum":31220909,"put":3388,"put_sum":31220909,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3388,"take_sum":3388,"put":3388,"put_sum":3388,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2931,"get_sum":296031,"get_or_fail_fail":0,"get_or_fail_success":2931,"take":0,"take_sum":0,"put":22,"put_sum":296031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15000,"get_sum":29891768,"get_or_fail_fail":0,"get_or_fail_success":15000,"take":0,"take_sum":0,"put":26,"put_sum":29891768,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:19.934 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 561622, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1005, "msgr_running_fast_dispatch_time": 0.053102332, "msgr_running_recv_time": 0.03389054, "msgr_running_send_time": 0.13657596, "msgr_running_total_time": 0.23574371, "msgr_send_bytes": 18595812, "msgr_send_encrypted_bytes": 96304, "msgr_send_messages": 1010, "msgr_send_messages_queue_lat": { "avgcount": 1009, "avgtime": 0.000308743, "sum": 0.311522416 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26576, "avgtime": 4.43e-07, "sum": 0.01178428 }, "msgr_recv_bytes": 4548362, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13218, "msgr_running_fast_dispatch_time": 0.053244932, "msgr_running_recv_time": 1.578175669, "msgr_running_send_time": 1.809202735, "msgr_running_total_time": 2.799420025, "msgr_send_bytes": 13566991, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24093, "msgr_send_messages_queue_lat": { "avgcount": 24092, "avgtime": 8.0654e-05, "sum": 1.943138866 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 348426, "msgr_recv_encrypted_bytes": 106736, "msgr_recv_messages": 928, "msgr_running_fast_dispatch_time": 0.035972456, "msgr_running_recv_time": 0.032926046, "msgr_running_send_time": 0.063082503, "msgr_running_total_time": 0.137341425, "msgr_send_bytes": 4579126, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 924, "msgr_send_messages_queue_lat": { "avgcount": 924, "avgtime": 0.000312671, "sum": 0.28890804 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 121, "avgtime": 0.092256245, "sum": 11.163005709 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5049, "avgtime": 0.000436545, "sum": 2.204120576 }, "queue_len": 0 }, "mds": { "caps": 271, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3991, "inodes_bottom": 3991, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3991, "inodes_top": 0, "inodes_with_caps": 271, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 270, "num_strays_delayed": 93, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2931, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15004, "evadd": 15000, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 26, "avgtime": 0.279005538, "sum": 7.254144011 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34088257 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 13, "dir+": 13, "dir-": 0, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 283, "ino+": 3214, "ino-": 2931, "rss": 118864 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 944, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1218178, "buffer_anon_items": 2588, "buffer_meta_bytes": 264, "buffer_meta_items": 3, "mds_co_bytes": 10153434, "mds_co_items": 153652, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 9, "omap_rd": 48, "omap_wr": 54, "op": 3388, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3388, "avgtime": 0.005602915, "sum": 18.982676307 }, "op_pg": 0, "op_r": 58, "op_reply": 3388, "op_resend": 0, "op_rmw": 0, "op_send": 3388, "op_send_bytes": 30922957, "op_w": 3330, "oplen_avg": { "avgcount": 3388, "sum": 4038 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 2956, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2931, "pq_executed_ops": 2931, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15147, "get_or_fail_fail": 0, "get_or_fail_success": 15147, "get_started": 0, "get_sum": 4092406, "max": 104857600, "put": 15147, "put_sum": 4092406, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3388, "put_sum": 31220909, "take": 3388, "take_sum": 31220909, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3388, "put_sum": 3388, "take": 3388, "take_sum": 3388, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2931, "get_or_fail_fail": 0, "get_or_fail_success": 2931, "get_started": 0, "get_sum": 296031, "max": 3758096384, "put": 22, "put_sum": 296031, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15000, "get_or_fail_fail": 0, "get_or_fail_success": 15000, "get_started": 0, "get_sum": 29891768, "max": 3758096384, "put": 26, "put_sum": 29891768, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:19.934 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/270, 2931/3201 2024-03-06T07:33:20.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.142+0000 7f5a98e00640 1 Processor -- start 2024-03-06T07:33:21.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.143+0000 7f5a98e00640 1 -- start start 2024-03-06T07:33:21.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.144+0000 7f5a98e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5a94145920 0x7f5a9406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.144+0000 7f5a98e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a940cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.144+0000 7f5a98e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a941601f0 con 0x7f5a9415faa0 2024-03-06T07:33:21.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.144+0000 7f5a98e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a941604d0 con 0x7f5a94145920 2024-03-06T07:33:21.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.144+0000 7f5a98e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a941607b0 con 0x7f5a940cc490 2024-03-06T07:33:21.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.145+0000 7f5a93e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a940cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.145+0000 7f5a93e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a940cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:54904/0 (socket says 172.21.15.97:54904) 2024-03-06T07:33:21.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.145+0000 7f5a93e00640 1 -- 172.21.15.97:0/831152621 learned_addr learned my addr 172.21.15.97:0/831152621 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:21.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.145+0000 7f5a93400640 1 --2- 172.21.15.97:0/831152621 >> v2:172.21.15.135:3300/0 conn(0x7f5a94145920 0x7f5a9406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.146+0000 7f5a93400640 1 -- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 msgr2=0x7f5a940cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.146+0000 7f5a93400640 1 --2- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a940cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.146+0000 7f5a93400640 1 -- 172.21.15.97:0/831152621 >> v1:172.21.15.135:6789/0 conn(0x7f5a9415faa0 legacy=0x7f5a940cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.146+0000 7f5a93400640 1 -- 172.21.15.97:0/831152621 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a940718a0 con 0x7f5a94145920 2024-03-06T07:33:21.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.147+0000 7f5a93400640 1 --2- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9406c920 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5a840097a0 tx=0x7f5a8402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaee440e55b6dacf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.147+0000 7f5a92a00640 1 -- 172.21.15.97:0/831152621 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84034070 con 0x7f5a94145920 2024-03-06T07:33:21.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.147+0000 7f5a92a00640 1 -- 172.21.15.97:0/831152621 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a84030610 con 0x7f5a94145920 2024-03-06T07:33:21.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.147+0000 7f5a92a00640 1 -- 172.21.15.97:0/831152621 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a840309e0 con 0x7f5a94145920 2024-03-06T07:33:21.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 -- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 msgr2=0x7f5a9406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 --2- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9406c920 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5a840097a0 tx=0x7f5a8402b960 comp rx=0 tx=0).stop 2024-03-06T07:33:21.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 -- 172.21.15.97:0/831152621 shutdown_connections 2024-03-06T07:33:21.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 --2- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a940cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 --2- 172.21.15.97:0/831152621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9406c920 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 -- 172.21.15.97:0/831152621 >> 172.21.15.97:0/831152621 conn(0x7f5a940b1770 msgr2=0x7f5a940b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 -- 172.21.15.97:0/831152621 shutdown_connections 2024-03-06T07:33:21.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.148+0000 7f5a98e00640 1 -- 172.21.15.97:0/831152621 wait complete. 2024-03-06T07:33:21.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.150+0000 7f5a98e00640 1 Processor -- start 2024-03-06T07:33:21.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.150+0000 7f5a98e00640 1 -- start start 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a98e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a941f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a98e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9413a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a98e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 0x7f5a9413c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a98e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a94071f40 con 0x7f5a9415faa0 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a98e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a94071dc0 con 0x7f5a94145920 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a98e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a940720c0 con 0x7f5a940cc490 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a93e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 0x7f5a9413c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a93e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 0x7f5a9413c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40638/0 (socket says 172.21.15.97:40638) 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a93e00640 1 -- 172.21.15.97:0/3162686575 learned_addr learned my addr 172.21.15.97:0/3162686575 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a93400640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a941f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.151+0000 7f5a8be00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9413a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a93e00640 1 -- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 msgr2=0x7f5a941f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a93e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a941f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a93e00640 1 -- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 msgr2=0x7f5a9413a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a93e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9413a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a93e00640 1 -- 172.21.15.97:0/3162686575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9413a820 con 0x7f5a9415faa0 2024-03-06T07:33:21.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a93e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 0x7f5a9413c630 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009d90 tx=0x7f5a8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c00ccc0 con 0x7f5a9415faa0 2024-03-06T07:33:21.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c00ce60 con 0x7f5a9415faa0 2024-03-06T07:33:21.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c003bb0 con 0x7f5a9415faa0 2024-03-06T07:33:21.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.152+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9413ab00 con 0x7f5a9415faa0 2024-03-06T07:33:21.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.153+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9413b0c0 con 0x7f5a9415faa0 2024-03-06T07:33:21.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.154+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9404f330 con 0x7f5a9415faa0 2024-03-06T07:33:21.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.156+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a8c012050 con 0x7f5a9415faa0 2024-03-06T07:33:21.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.157+0000 7f5a8b400640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a7006b4d0 0x7f5a7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.157+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5a8c08a810 con 0x7f5a9415faa0 2024-03-06T07:33:21.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.157+0000 7f5a93400640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a7006b4d0 0x7f5a7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.158+0000 7f5a93400640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a7006b4d0 0x7f5a7006d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5a84036000 tx=0x7f5a8403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.162+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a8c010040 con 0x7f5a9415faa0 2024-03-06T07:33:21.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.399+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a94140110 con 0x7f5a9415faa0 2024-03-06T07:33:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.400+0000 7f5a8b400640 1 -- 172.21.15.97:0/3162686575 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f5a8c059590 con 0x7f5a9415faa0 2024-03-06T07:33:21.402 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:21.402 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a7006b4d0 msgr2=0x7f5a7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a7006b4d0 0x7f5a7006d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5a84036000 tx=0x7f5a8403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 msgr2=0x7f5a9413c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 0x7f5a9413c630 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009d90 tx=0x7f5a8c00c490 comp rx=0 tx=0).stop 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 shutdown_connections 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a7006b4d0 0x7f5a7006d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a9415faa0 0x7f5a9413c630 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a94145920 0x7f5a9413a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 --2- 172.21.15.97:0/3162686575 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a940cc490 0x7f5a941f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 >> 172.21.15.97:0/3162686575 conn(0x7f5a940b1770 msgr2=0x7f5a94072480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 shutdown_connections 2024-03-06T07:33:21.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:21.403+0000 7f5a98e00640 1 -- 172.21.15.97:0/3162686575 wait complete. 2024-03-06T07:33:21.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:21.640 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1041,"msgr_recv_bytes":578871,"msgr_send_bytes":18671522,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.239926093,"msgr_running_send_time":0.138430919,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1040,"sum":0.314455103,"avgtime":0.000302360},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96304},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13255,"msgr_send_messages":24129,"msgr_recv_bytes":4557916,"msgr_send_bytes":13578809,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.803442719,"msgr_running_send_time":1.810798527,"msgr_running_recv_time":1.579152986,"msgr_running_fast_dispatch_time":0.054391740,"msgr_send_messages_queue_lat":{"avgcount":24128,"sum":1.946903093,"avgtime":0.000080690},"msgr_handle_ack_lat":{"avgcount":26577,"sum":0.011784540,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":963,"msgr_send_messages":959,"msgr_recv_bytes":357316,"msgr_send_bytes":4590536,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.140490518,"msgr_running_send_time":0.064300505,"msgr_running_recv_time":0.033729650,"msgr_running_fast_dispatch_time":0.036833690,"msgr_send_messages_queue_lat":{"avgcount":959,"sum":0.290631104,"avgtime":0.000303056},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106736,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":118864,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":949,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":270316,"buffer_anon_items":366,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":952319,"mds_co_items":19249,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15250,"get_sum":4118829,"get_or_fail_fail":0,"get_or_fail_success":15250,"take":0,"take_sum":0,"put":15250,"put_sum":4118829,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:21.642 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.138430919, "msgr_running_total_time": 0.239926093, "msgr_send_bytes": 18671522, "msgr_send_encrypted_bytes": 96304, "msgr_send_messages": 1041, "msgr_send_messages_queue_lat": { "avgcount": 1040, "avgtime": 0.00030236, "sum": 0.314455103 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26577, "avgtime": 4.43e-07, "sum": 0.01178454 }, "msgr_recv_bytes": 4557916, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13255, "msgr_running_fast_dispatch_time": 0.05439174, "msgr_running_recv_time": 1.579152986, "msgr_running_send_time": 1.810798527, "msgr_running_total_time": 2.803442719, "msgr_send_bytes": 13578809, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24129, "msgr_send_messages_queue_lat": { "avgcount": 24128, "avgtime": 8.069e-05, "sum": 1.946903093 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 357316, "msgr_recv_encrypted_bytes": 106736, "msgr_recv_messages": 963, "msgr_running_fast_dispatch_time": 0.03683369, "msgr_running_recv_time": 0.03372965, "msgr_running_send_time": 0.064300505, "msgr_running_total_time": 0.140490518, "msgr_send_bytes": 4590536, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 959, "msgr_send_messages_queue_lat": { "avgcount": 959, "avgtime": 0.000303056, "sum": 0.290631104 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 118864 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 949, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 270316, "buffer_anon_items": 366, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 952319, "mds_co_items": 19249, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15250, "get_or_fail_fail": 0, "get_or_fail_success": 15250, "get_started": 0, "get_sum": 4118829, "max": 104857600, "put": 15250, "put_sum": 4118829, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:21.642 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:22.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.845+0000 7f6f27800640 1 Processor -- start 2024-03-06T07:33:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.845+0000 7f6f27800640 1 -- start start 2024-03-06T07:33:22.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.846+0000 7f6f27800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6f2015cc90 0x7f6f2006d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.846+0000 7f6f27800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6f20162b00 0x7f6f20162f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.846+0000 7f6f27800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f201635a0 con 0x7f6f200d0070 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.846+0000 7f6f27800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2006d940 con 0x7f6f20162b00 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.846+0000 7f6f27800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f20163880 con 0x7f6f2015cc90 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6f20162b00 0x7f6f20162f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6f20162b00 0x7f6f20162f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40646/0 (socket says 172.21.15.97:40646) 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 -- 172.21.15.97:0/2157866196 learned_addr learned my addr 172.21.15.97:0/2157866196 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 -- 172.21.15.97:0/2157866196 >> v2:172.21.15.135:3300/0 conn(0x7f6f2015cc90 msgr2=0x7f6f2006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 --2- 172.21.15.97:0/2157866196 >> v2:172.21.15.135:3300/0 conn(0x7f6f2015cc90 0x7f6f2006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 -- 172.21.15.97:0/2157866196 >> v1:172.21.15.135:6789/0 conn(0x7f6f200d0070 legacy=0x7f6f200acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.847+0000 7f6f1fe00640 1 -- 172.21.15.97:0/2157866196 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2006cc00 con 0x7f6f20162b00 2024-03-06T07:33:22.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.848+0000 7f6f1fe00640 1 --2- 172.21.15.97:0/2157866196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f20162f80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f6f2006ee40 tx=0x7f6f0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd3f38b20e84b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.848+0000 7f6f1f400640 1 -- 172.21.15.97:0/2157866196 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c035020 con 0x7f6f20162b00 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f1f400640 1 -- 172.21.15.97:0/2157866196 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c031870 con 0x7f6f20162b00 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f1f400640 1 -- 172.21.15.97:0/2157866196 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c031c40 con 0x7f6f20162b00 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 -- 172.21.15.97:0/2157866196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 msgr2=0x7f6f20162f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 --2- 172.21.15.97:0/2157866196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f20162f80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f6f2006ee40 tx=0x7f6f0c02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 -- 172.21.15.97:0/2157866196 shutdown_connections 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 --2- 172.21.15.97:0/2157866196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f20162f80 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 --2- 172.21.15.97:0/2157866196 >> v2:172.21.15.135:3300/0 conn(0x7f6f2015cc90 0x7f6f2006d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 -- 172.21.15.97:0/2157866196 >> 172.21.15.97:0/2157866196 conn(0x7f6f2006b360 msgr2=0x7f6f2006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 -- 172.21.15.97:0/2157866196 shutdown_connections 2024-03-06T07:33:22.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.849+0000 7f6f27800640 1 -- 172.21.15.97:0/2157866196 wait complete. 2024-03-06T07:33:22.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.852+0000 7f6f27800640 1 Processor -- start 2024-03-06T07:33:22.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.852+0000 7f6f27800640 1 -- start start 2024-03-06T07:33:22.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f27800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f200d0070 0x7f6f201559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f27800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f2015cc90 0x7f6f201642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f27800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f200b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f27800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f200b4fe0 con 0x7f6f20162b00 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f27800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f200b4e60 con 0x7f6f2015cc90 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f27800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f200d0500 con 0x7f6f200d0070 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f25800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f200b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f25800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f200b0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40654/0 (socket says 172.21.15.97:40654) 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f25800640 1 -- 172.21.15.97:0/3659046310 learned_addr learned my addr 172.21.15.97:0/3659046310 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f1fe00640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f2015cc90 0x7f6f201642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f25800640 1 -- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f200d0070 msgr2=0x7f6f201559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.854+0000 7f6f24e00640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f200d0070 0x7f6f201559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f25800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f200d0070 0x7f6f201559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f25800640 1 -- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f2015cc90 msgr2=0x7f6f201642b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f25800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f2015cc90 0x7f6f201642b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f25800640 1 -- 172.21.15.97:0/3659046310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f201662d0 con 0x7f6f20162b00 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f1fe00640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f2015cc90 0x7f6f201642b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f25800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f200b0fb0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c780 tx=0x7f6f1800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18017070 con 0x7f6f20162b00 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18003da0 con 0x7f6f20162b00 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180052b0 con 0x7f6f20162b00 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f201665b0 con 0x7f6f20162b00 2024-03-06T07:33:22.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.855+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f20166b70 con 0x7f6f20162b00 2024-03-06T07:33:22.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.856+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f20068070 con 0x7f6f20162b00 2024-03-06T07:33:22.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.858+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f1801a070 con 0x7f6f20162b00 2024-03-06T07:33:22.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.859+0000 7f6f1cc00640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f0006b4d0 0x7f6f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.859+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6f18089890 con 0x7f6f20162b00 2024-03-06T07:33:22.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.860+0000 7f6f24e00640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f0006b4d0 0x7f6f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.860+0000 7f6f24e00640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f0006b4d0 0x7f6f0006d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400a810 tx=0x7f6f14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:22.865+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f180586a0 con 0x7f6f20162b00 2024-03-06T07:33:23.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.127+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f2006e6a0 con 0x7f6f20162b00 2024-03-06T07:33:23.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.128+0000 7f6f1cc00640 1 -- 172.21.15.97:0/3659046310 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f6f18057fa0 con 0x7f6f20162b00 2024-03-06T07:33:23.130 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:23.130 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:23.131 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:23.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f0006b4d0 msgr2=0x7f6f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f0006b4d0 0x7f6f0006d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400a810 tx=0x7f6f14008040 comp rx=0 tx=0).stop 2024-03-06T07:33:23.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 msgr2=0x7f6f200b0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f200b0fb0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c780 tx=0x7f6f1800cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:23.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 shutdown_connections 2024-03-06T07:33:23.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f0006b4d0 0x7f6f0006d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f20162b00 0x7f6f200b0fb0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f2015cc90 0x7f6f201642b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 --2- 172.21.15.97:0/3659046310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f200d0070 0x7f6f201559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 >> 172.21.15.97:0/3659046310 conn(0x7f6f2006b360 msgr2=0x7f6f200722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 shutdown_connections 2024-03-06T07:33:23.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:23.131+0000 7f6f27800640 1 -- 172.21.15.97:0/3659046310 wait complete. 2024-03-06T07:33:23.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:23.393 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1042,"msgr_recv_bytes":578871,"msgr_send_bytes":18673314,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240174599,"msgr_running_send_time":0.138653579,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1041,"sum":0.314541425,"avgtime":0.000302153},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":98096},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13257,"msgr_send_messages":24129,"msgr_recv_bytes":4558752,"msgr_send_bytes":13578809,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.804003523,"msgr_running_send_time":1.811050017,"msgr_running_recv_time":1.579273895,"msgr_running_fast_dispatch_time":0.054572264,"msgr_send_messages_queue_lat":{"avgcount":24128,"sum":1.946903093,"avgtime":0.000080690},"msgr_handle_ack_lat":{"avgcount":26579,"sum":0.011784999,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":964,"msgr_send_messages":960,"msgr_recv_bytes":357588,"msgr_send_bytes":4591160,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.140679779,"msgr_running_send_time":0.064396775,"msgr_running_recv_time":0.033770964,"msgr_running_fast_dispatch_time":0.036869351,"msgr_send_messages_queue_lat":{"avgcount":960,"sum":0.290681093,"avgtime":0.000302792},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107008,"msgr_send_encrypted_bytes":8544},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":118864,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":949,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":270316,"buffer_anon_items":366,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":952319,"mds_co_items":19249,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15253,"get_sum":4119615,"get_or_fail_fail":0,"get_or_fail_success":15253,"take":0,"take_sum":0,"put":15253,"put_sum":4119615,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:23.394 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.138653579, "msgr_running_total_time": 0.240174599, "msgr_send_bytes": 18673314, "msgr_send_encrypted_bytes": 98096, "msgr_send_messages": 1042, "msgr_send_messages_queue_lat": { "avgcount": 1041, "avgtime": 0.000302153, "sum": 0.314541425 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26579, "avgtime": 4.43e-07, "sum": 0.011784999 }, "msgr_recv_bytes": 4558752, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13257, "msgr_running_fast_dispatch_time": 0.054572264, "msgr_running_recv_time": 1.579273895, "msgr_running_send_time": 1.811050017, "msgr_running_total_time": 2.804003523, "msgr_send_bytes": 13578809, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24129, "msgr_send_messages_queue_lat": { "avgcount": 24128, "avgtime": 8.069e-05, "sum": 1.946903093 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 357588, "msgr_recv_encrypted_bytes": 107008, "msgr_recv_messages": 964, "msgr_running_fast_dispatch_time": 0.036869351, "msgr_running_recv_time": 0.033770964, "msgr_running_send_time": 0.064396775, "msgr_running_total_time": 0.140679779, "msgr_send_bytes": 4591160, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 960, "msgr_send_messages_queue_lat": { "avgcount": 960, "avgtime": 0.000302792, "sum": 0.290681093 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 118864 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 949, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 270316, "buffer_anon_items": 366, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 952319, "mds_co_items": 19249, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15253, "get_or_fail_fail": 0, "get_or_fail_success": 15253, "get_started": 0, "get_sum": 4119615, "max": 104857600, "put": 15253, "put_sum": 4119615, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:23.395 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:24.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:24.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.604+0000 7fb3fbc00640 1 Processor -- start 2024-03-06T07:33:24.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.604+0000 7fb3fbc00640 1 -- start start 2024-03-06T07:33:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3fbc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb3f40d0500 0x7fb3f4072410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3fbc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f415a9a0 con 0x7fb3f40d0070 2024-03-06T07:33:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3fbc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40710c0 con 0x7fb3f4070ce0 2024-03-06T07:33:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3fbc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f4071370 con 0x7fb3f40d0500 2024-03-06T07:33:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3f9c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb3f40d0500 0x7fb3f4072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3f9c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb3f40d0500 0x7fb3f4072410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46734/0 (socket says 172.21.15.97:46734) 2024-03-06T07:33:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3f3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb3f40d0070 0x7fb3f40acea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44904/0 (socket says 172.21.15.97:44904) 2024-03-06T07:33:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.607+0000 7fb3f9c00640 1 -- 172.21.15.97:0/1197841911 learned_addr learned my addr 172.21.15.97:0/1197841911 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.608+0000 7fb3f9c00640 1 -- 172.21.15.97:0/1197841911 >> v1:172.21.15.135:6789/0 conn(0x7fb3f4070ce0 legacy=0x7fb3f40b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.608+0000 7fb3f9c00640 1 -- 172.21.15.97:0/1197841911 >> v1:172.21.15.97:6789/0 conn(0x7fb3f40d0070 legacy=0x7fb3f40acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.608+0000 7fb3f9c00640 1 -- 172.21.15.97:0/1197841911 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4071650 con 0x7fb3f40d0500 2024-03-06T07:33:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.608+0000 7fb3f3400640 1 -- 172.21.15.97:0/1197841911 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919297150 0 0) 0x7fb3f415a9a0 con 0x7fb3f40d0070 2024-03-06T07:33:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.609+0000 7fb3f9c00640 1 --2- 172.21.15.97:0/1197841911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f4072410 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009e50 tx=0x7fb3ec02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5e3ed043665bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.609+0000 7fb3f3400640 1 -- 172.21.15.97:0/1197841911 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec02cbb0 con 0x7fb3f40d0500 2024-03-06T07:33:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.609+0000 7fb3f3400640 1 -- 172.21.15.97:0/1197841911 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec02cd50 con 0x7fb3f40d0500 2024-03-06T07:33:24.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.609+0000 7fb3f3400640 1 -- 172.21.15.97:0/1197841911 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec034610 con 0x7fb3f40d0500 2024-03-06T07:33:24.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 -- 172.21.15.97:0/1197841911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 msgr2=0x7fb3f4072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/1197841911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f4072410 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009e50 tx=0x7fb3ec02c010 comp rx=0 tx=0).stop 2024-03-06T07:33:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 -- 172.21.15.97:0/1197841911 shutdown_connections 2024-03-06T07:33:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/1197841911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f4072410 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 -- 172.21.15.97:0/1197841911 >> 172.21.15.97:0/1197841911 conn(0x7fb3f406b360 msgr2=0x7fb3f406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 -- 172.21.15.97:0/1197841911 shutdown_connections 2024-03-06T07:33:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.610+0000 7fb3fbc00640 1 -- 172.21.15.97:0/1197841911 wait complete. 2024-03-06T07:33:24.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.612+0000 7fb3fbc00640 1 Processor -- start 2024-03-06T07:33:24.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.612+0000 7fb3fbc00640 1 -- start start 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3fbc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3f4070ce0 0x7fb3f415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3fbc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb3f40d0070 0x7fb3f4158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3fbc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f40b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3fbc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4071ef0 con 0x7fb3f40d0500 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3fbc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4071c10 con 0x7fb3f4070ce0 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3fbc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40ad5b0 con 0x7fb3f40d0070 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3f9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f40b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3f9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f40b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46744/0 (socket says 172.21.15.97:46744) 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.613+0000 7fb3f9c00640 1 -- 172.21.15.97:0/3792004050 learned_addr learned my addr 172.21.15.97:0/3792004050 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f3e00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb3f40d0070 0x7fb3f4158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9200640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3f4070ce0 0x7fb3f415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9c00640 1 -- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb3f40d0070 msgr2=0x7fb3f4158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9c00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb3f40d0070 0x7fb3f4158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9c00640 1 -- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3f4070ce0 msgr2=0x7fb3f415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9c00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3f4070ce0 0x7fb3f415c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9c00640 1 -- 172.21.15.97:0/3792004050 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f414c670 con 0x7fb3f40d0500 2024-03-06T07:33:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9200640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3f4070ce0 0x7fb3f415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f9c00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f40b0910 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009e50 tx=0x7fb3ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec043030 con 0x7fb3f40d0500 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec003990 con 0x7fb3f40d0500 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec009510 con 0x7fb3f40d0500 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.614+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f414c950 con 0x7fb3f40d0500 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.615+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f414cf10 con 0x7fb3f40d0500 2024-03-06T07:33:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.616+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4068070 con 0x7fb3f40d0500 2024-03-06T07:33:24.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.617+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3ec02cbc0 con 0x7fb3f40d0500 2024-03-06T07:33:24.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.617+0000 7fb3f0c00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb3d406b4d0 0x7fb3d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.618+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3ec0ac8b0 con 0x7fb3f40d0500 2024-03-06T07:33:24.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.618+0000 7fb3f9200640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb3d406b4d0 0x7fb3d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.618+0000 7fb3f9200640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb3d406b4d0 0x7fb3d406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40b3120 tx=0x7fb3e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.623+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3ec07b750 con 0x7fb3f40d0500 2024-03-06T07:33:24.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.939+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f4165e80 con 0x7fb3f40d0500 2024-03-06T07:33:24.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.939+0000 7fb3f0c00640 1 -- 172.21.15.97:0/3792004050 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fb3ec07b050 con 0x7fb3f40d0500 2024-03-06T07:33:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:24.942 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:24.942 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb3d406b4d0 msgr2=0x7fb3d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb3d406b4d0 0x7fb3d406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40b3120 tx=0x7fb3e8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 msgr2=0x7fb3f40b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f40b0910 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009e50 tx=0x7fb3ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 shutdown_connections 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb3d406b4d0 0x7fb3d406d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3f40d0500 0x7fb3f40b0910 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb3f40d0070 0x7fb3f4158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 --2- 172.21.15.97:0/3792004050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3f4070ce0 0x7fb3f415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.942+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 >> 172.21.15.97:0/3792004050 conn(0x7fb3f406b360 msgr2=0x7fb3f4157f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.943+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 shutdown_connections 2024-03-06T07:33:24.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:24.943+0000 7fb3fbc00640 1 -- 172.21.15.97:0/3792004050 wait complete. 2024-03-06T07:33:24.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:25.182 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1042,"msgr_recv_bytes":578871,"msgr_send_bytes":18673314,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240174599,"msgr_running_send_time":0.138653579,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1041,"sum":0.314541425,"avgtime":0.000302153},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":98096},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13260,"msgr_send_messages":24130,"msgr_recv_bytes":4559857,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.804800404,"msgr_running_send_time":1.811393903,"msgr_running_recv_time":1.579506714,"msgr_running_fast_dispatch_time":0.054802863,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26583,"sum":0.011786060,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":964,"msgr_send_messages":960,"msgr_recv_bytes":357588,"msgr_send_bytes":4591160,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.140679779,"msgr_running_send_time":0.064396775,"msgr_running_recv_time":0.033770964,"msgr_running_fast_dispatch_time":0.036869351,"msgr_send_messages_queue_lat":{"avgcount":960,"sum":0.290681093,"avgtime":0.000302792},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107008,"msgr_send_encrypted_bytes":8544},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":954,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15256,"get_sum":4120450,"get_or_fail_fail":0,"get_or_fail_success":15256,"take":0,"take_sum":0,"put":15256,"put_sum":4120450,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:25.184 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.138653579, "msgr_running_total_time": 0.240174599, "msgr_send_bytes": 18673314, "msgr_send_encrypted_bytes": 98096, "msgr_send_messages": 1042, "msgr_send_messages_queue_lat": { "avgcount": 1041, "avgtime": 0.000302153, "sum": 0.314541425 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26583, "avgtime": 4.43e-07, "sum": 0.01178606 }, "msgr_recv_bytes": 4559857, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13260, "msgr_running_fast_dispatch_time": 0.054802863, "msgr_running_recv_time": 1.579506714, "msgr_running_send_time": 1.811393903, "msgr_running_total_time": 2.804800404, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 357588, "msgr_recv_encrypted_bytes": 107008, "msgr_recv_messages": 964, "msgr_running_fast_dispatch_time": 0.036869351, "msgr_running_recv_time": 0.033770964, "msgr_running_send_time": 0.064396775, "msgr_running_total_time": 0.140679779, "msgr_send_bytes": 4591160, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 960, "msgr_send_messages_queue_lat": { "avgcount": 960, "avgtime": 0.000302792, "sum": 0.290681093 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 954, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15256, "get_or_fail_fail": 0, "get_or_fail_success": 15256, "get_started": 0, "get_sum": 4120450, "max": 104857600, "put": 15256, "put_sum": 4120450, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:25.184 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:26.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:26.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.403+0000 7ff145200640 1 Processor -- start 2024-03-06T07:33:26.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.403+0000 7ff145200640 1 -- start start 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.405+0000 7ff145200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400b44f0 0x7ff1400d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff145200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff1400d16a0 0x7ff1400d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff145200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400d21c0 con 0x7ff1400b5020 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff145200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1400d0000 con 0x7ff1400d16a0 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff145200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400b4e40 con 0x7ff1400b44f0 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff1400d16a0 0x7ff1400d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff1400b5020 0x7ff1400c7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44912/0 (socket says 172.21.15.97:44912) 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13d400640 1 -- 172.21.15.97:0/577064063 learned_addr learned my addr 172.21.15.97:0/577064063 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff1400d16a0 0x7ff1400d2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46750/0 (socket says 172.21.15.97:46750) 2024-03-06T07:33:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13de00640 1 --2- 172.21.15.97:0/577064063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400b44f0 0x7ff1400d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13e800640 1 -- 172.21.15.97:0/577064063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400b44f0 msgr2=0x7ff1400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13e800640 1 --2- 172.21.15.97:0/577064063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400b44f0 0x7ff1400d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.406+0000 7ff13e800640 1 -- 172.21.15.97:0/577064063 >> v1:172.21.15.97:6789/0 conn(0x7ff1400b5020 legacy=0x7ff1400c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.407+0000 7ff13e800640 1 -- 172.21.15.97:0/577064063 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400b48d0 con 0x7ff1400d16a0 2024-03-06T07:33:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.407+0000 7ff13ca00640 1 -- 172.21.15.97:0/577064063 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577272043 0 0) 0x7ff1400d21c0 con 0x7ff1400b5020 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.408+0000 7ff13e800640 1 --2- 172.21.15.97:0/577064063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400d16a0 0x7ff1400d2af0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7ff138009dc0 tx=0x7ff13802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16906b87ecc6061c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.408+0000 7ff13ca00640 1 -- 172.21.15.97:0/577064063 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138039070 con 0x7ff1400d16a0 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.408+0000 7ff13ca00640 1 -- 172.21.15.97:0/577064063 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13802cbb0 con 0x7ff1400d16a0 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.408+0000 7ff13ca00640 1 -- 172.21.15.97:0/577064063 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138034440 con 0x7ff1400d16a0 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.408+0000 7ff145200640 1 -- 172.21.15.97:0/577064063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400d16a0 msgr2=0x7ff1400d2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.408+0000 7ff145200640 1 --2- 172.21.15.97:0/577064063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400d16a0 0x7ff1400d2af0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7ff138009dc0 tx=0x7ff13802be00 comp rx=0 tx=0).stop 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.409+0000 7ff145200640 1 -- 172.21.15.97:0/577064063 shutdown_connections 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.409+0000 7ff145200640 1 --2- 172.21.15.97:0/577064063 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400d16a0 0x7ff1400d2af0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.409+0000 7ff145200640 1 --2- 172.21.15.97:0/577064063 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400b44f0 0x7ff1400d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.409+0000 7ff145200640 1 -- 172.21.15.97:0/577064063 >> 172.21.15.97:0/577064063 conn(0x7ff1400d3fd0 msgr2=0x7ff1400d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.409+0000 7ff145200640 1 -- 172.21.15.97:0/577064063 shutdown_connections 2024-03-06T07:33:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.409+0000 7ff145200640 1 -- 172.21.15.97:0/577064063 wait complete. 2024-03-06T07:33:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.410+0000 7ff145200640 1 Processor -- start 2024-03-06T07:33:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.411+0000 7ff145200640 1 -- start start 2024-03-06T07:33:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.411+0000 7ff145200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 0x7ff1400b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.411+0000 7ff145200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1400b5020 0x7ff1401523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.411+0000 7ff145200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400d16a0 0x7ff1400b9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff145200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400cfae0 con 0x7ff1400b44f0 2024-03-06T07:33:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff145200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400cf800 con 0x7ff1400b5020 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff145200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400b4ad0 con 0x7ff1400d16a0 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 0x7ff1400b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 0x7ff1400b6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46752/0 (socket says 172.21.15.97:46752) 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 -- 172.21.15.97:0/3065362752 learned_addr learned my addr 172.21.15.97:0/3065362752 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13e800640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400d16a0 0x7ff1400b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13d400640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1400b5020 0x7ff1401523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 -- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400d16a0 msgr2=0x7ff1400b9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400d16a0 0x7ff1400b9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 -- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1400b5020 msgr2=0x7ff1401523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1400b5020 0x7ff1401523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.412+0000 7ff13de00640 1 -- 172.21.15.97:0/3065362752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1401f7ca0 con 0x7ff1400b44f0 2024-03-06T07:33:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.413+0000 7ff13de00640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 0x7ff1400b6bb0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff13000d800 tx=0x7ff13000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.413+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13000b6c0 con 0x7ff1400b44f0 2024-03-06T07:33:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.413+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff130019070 con 0x7ff1400b44f0 2024-03-06T07:33:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.413+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13000ba80 con 0x7ff1400b44f0 2024-03-06T07:33:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.413+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1401f7f80 con 0x7ff1400b44f0 2024-03-06T07:33:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.413+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff140152920 con 0x7ff1400b44f0 2024-03-06T07:33:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.414+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff140068070 con 0x7ff1400b44f0 2024-03-06T07:33:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.415+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff13000bc20 con 0x7ff1400b44f0 2024-03-06T07:33:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.416+0000 7ff136000640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff12006b4d0 0x7ff12006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.416+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff13008a6b0 con 0x7ff1400b44f0 2024-03-06T07:33:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.416+0000 7ff13d400640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff12006b4d0 0x7ff12006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.417+0000 7ff13d400640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff12006b4d0 0x7ff12006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff12c007fd0 tx=0x7ff12c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.421+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff130059560 con 0x7ff1400b44f0 2024-03-06T07:33:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.658+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1400d2390 con 0x7ff1400b44f0 2024-03-06T07:33:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.659+0000 7ff136000640 1 -- 172.21.15.97:0/3065362752 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7ff130058e60 con 0x7ff1400b44f0 2024-03-06T07:33:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:26.661 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.661+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff12006b4d0 msgr2=0x7ff12006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.661+0000 7ff145200640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff12006b4d0 0x7ff12006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff12c007fd0 tx=0x7ff12c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.661+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 msgr2=0x7ff1400b6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.661+0000 7ff145200640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 0x7ff1400b6bb0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff13000d800 tx=0x7ff13000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.661+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 shutdown_connections 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff12006b4d0 0x7ff12006d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff1400d16a0 0x7ff1400b9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1400b5020 0x7ff1401523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 --2- 172.21.15.97:0/3065362752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1400b44f0 0x7ff1400b6bb0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 >> 172.21.15.97:0/3065362752 conn(0x7ff1400d3fd0 msgr2=0x7ff140161330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 shutdown_connections 2024-03-06T07:33:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:26.662+0000 7ff145200640 1 -- 172.21.15.97:0/3065362752 wait complete. 2024-03-06T07:33:26.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:26.967 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1043,"msgr_recv_bytes":578871,"msgr_send_bytes":18675106,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240272932,"msgr_running_send_time":0.138736621,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1042,"sum":0.314582234,"avgtime":0.000301902},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":99888},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13262,"msgr_send_messages":24130,"msgr_recv_bytes":4560693,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.805321369,"msgr_running_send_time":1.811596407,"msgr_running_recv_time":1.579648572,"msgr_running_fast_dispatch_time":0.054971507,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26585,"sum":0.011786463,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":965,"msgr_send_messages":961,"msgr_recv_bytes":357860,"msgr_send_bytes":4591784,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.140946136,"msgr_running_send_time":0.064521720,"msgr_running_recv_time":0.033831776,"msgr_running_fast_dispatch_time":0.036924004,"msgr_send_messages_queue_lat":{"avgcount":961,"sum":0.290796179,"avgtime":0.000302597},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107280,"msgr_send_encrypted_bytes":9168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":954,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15259,"get_sum":4121236,"get_or_fail_fail":0,"get_or_fail_success":15259,"take":0,"take_sum":0,"put":15259,"put_sum":4121236,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:26.968 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.138736621, "msgr_running_total_time": 0.240272932, "msgr_send_bytes": 18675106, "msgr_send_encrypted_bytes": 99888, "msgr_send_messages": 1043, "msgr_send_messages_queue_lat": { "avgcount": 1042, "avgtime": 0.000301902, "sum": 0.314582234 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26585, "avgtime": 4.43e-07, "sum": 0.011786463 }, "msgr_recv_bytes": 4560693, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13262, "msgr_running_fast_dispatch_time": 0.054971507, "msgr_running_recv_time": 1.579648572, "msgr_running_send_time": 1.811596407, "msgr_running_total_time": 2.805321369, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 357860, "msgr_recv_encrypted_bytes": 107280, "msgr_recv_messages": 965, "msgr_running_fast_dispatch_time": 0.036924004, "msgr_running_recv_time": 0.033831776, "msgr_running_send_time": 0.06452172, "msgr_running_total_time": 0.140946136, "msgr_send_bytes": 4591784, "msgr_send_encrypted_bytes": 9168, "msgr_send_messages": 961, "msgr_send_messages_queue_lat": { "avgcount": 961, "avgtime": 0.000302597, "sum": 0.290796179 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 954, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15259, "get_or_fail_fail": 0, "get_or_fail_success": 15259, "get_started": 0, "get_sum": 4121236, "max": 104857600, "put": 15259, "put_sum": 4121236, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:26.968 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:27.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:28.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.171+0000 7fdb75800640 1 Processor -- start 2024-03-06T07:33:28.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.171+0000 7fdb75800640 1 -- start start 2024-03-06T07:33:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb75800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb7015be90 0x7fdb700b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb75800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdb700d0780 0x7fdb7015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb75800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700b5160 con 0x7fdb700acd60 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb75800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb700b4f30 con 0x7fdb700d0780 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb75800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700af240 con 0x7fdb7015be90 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb6f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdb700d0780 0x7fdb7015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb6f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdb700d0780 0x7fdb7015fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46754/0 (socket says 172.21.15.97:46754) 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.173+0000 7fdb6f400640 1 -- 172.21.15.97:0/3275605748 learned_addr learned my addr 172.21.15.97:0/3275605748 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.174+0000 7fdb6ea00640 1 --2- 172.21.15.97:0/3275605748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb7015be90 0x7fdb700b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.174+0000 7fdb6f400640 1 -- 172.21.15.97:0/3275605748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb7015be90 msgr2=0x7fdb700b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.174+0000 7fdb6f400640 1 --2- 172.21.15.97:0/3275605748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb7015be90 0x7fdb700b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.174+0000 7fdb6f400640 1 -- 172.21.15.97:0/3275605748 >> v1:172.21.15.97:6789/0 conn(0x7fdb700acd60 legacy=0x7fdb700d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.174+0000 7fdb6f400640 1 -- 172.21.15.97:0/3275605748 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700ae910 con 0x7fdb700d0780 2024-03-06T07:33:28.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.174+0000 7fdb6d600640 1 -- 172.21.15.97:0/3275605748 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830322589 0 0) 0x7fdb700b5160 con 0x7fdb700acd60 2024-03-06T07:33:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.175+0000 7fdb6f400640 1 --2- 172.21.15.97:0/3275605748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700d0780 0x7fdb7015fb70 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009e50 tx=0x7fdb6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5bada7205ce3224 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.175+0000 7fdb6d600640 1 -- 172.21.15.97:0/3275605748 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6802cb00 con 0x7fdb700d0780 2024-03-06T07:33:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.175+0000 7fdb6d600640 1 -- 172.21.15.97:0/3275605748 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6802cca0 con 0x7fdb700d0780 2024-03-06T07:33:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.175+0000 7fdb6d600640 1 -- 172.21.15.97:0/3275605748 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68034610 con 0x7fdb700d0780 2024-03-06T07:33:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 -- 172.21.15.97:0/3275605748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700d0780 msgr2=0x7fdb7015fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 --2- 172.21.15.97:0/3275605748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700d0780 0x7fdb7015fb70 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009e50 tx=0x7fdb6802bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 -- 172.21.15.97:0/3275605748 shutdown_connections 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 --2- 172.21.15.97:0/3275605748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700d0780 0x7fdb7015fb70 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 --2- 172.21.15.97:0/3275605748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb7015be90 0x7fdb700b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 -- 172.21.15.97:0/3275605748 >> 172.21.15.97:0/3275605748 conn(0x7fdb7006b360 msgr2=0x7fdb7006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 -- 172.21.15.97:0/3275605748 shutdown_connections 2024-03-06T07:33:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.176+0000 7fdb75800640 1 -- 172.21.15.97:0/3275605748 wait complete. 2024-03-06T07:33:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.178+0000 7fdb75800640 1 Processor -- start 2024-03-06T07:33:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.178+0000 7fdb75800640 1 -- start start 2024-03-06T07:33:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb75800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 0x7fdb701e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb75800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb700d0780 0x7fdb701596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb75800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb7015be90 0x7fdb701f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb75800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700b2c80 con 0x7fdb700acd60 2024-03-06T07:33:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb75800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700aeed0 con 0x7fdb7015be90 2024-03-06T07:33:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb75800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700b2e00 con 0x7fdb700d0780 2024-03-06T07:33:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb6ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 0x7fdb701e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb6ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 0x7fdb701e79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46770/0 (socket says 172.21.15.97:46770) 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb6ea00640 1 -- 172.21.15.97:0/2866089337 learned_addr learned my addr 172.21.15.97:0/2866089337 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb6ea00640 1 -- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb700d0780 msgr2=0x7fdb701596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb6e000640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb700d0780 0x7fdb701596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.179+0000 7fdb6f400640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb7015be90 0x7fdb701f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb6ea00640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb700d0780 0x7fdb701596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb6ea00640 1 -- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb7015be90 msgr2=0x7fdb701f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb6ea00640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb7015be90 0x7fdb701f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb6ea00640 1 -- 172.21.15.97:0/2866089337 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7015a030 con 0x7fdb700acd60 2024-03-06T07:33:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb6e000640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb700d0780 0x7fdb701596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb6ea00640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 0x7fdb701e79b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000d620 tx=0x7fdb6000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6000b620 con 0x7fdb700acd60 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb60019070 con 0x7fdb700acd60 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6000ba00 con 0x7fdb700acd60 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7015a310 con 0x7fdb700acd60 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.180+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb700b03d0 con 0x7fdb700acd60 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.182+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb70068070 con 0x7fdb700acd60 2024-03-06T07:33:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.182+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb6000bba0 con 0x7fdb700acd60 2024-03-06T07:33:28.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.183+0000 7fdb66a00640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb5006b4d0 0x7fdb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.184+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdb6008a740 con 0x7fdb700acd60 2024-03-06T07:33:28.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.184+0000 7fdb6e000640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb5006b4d0 0x7fdb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.184+0000 7fdb6e000640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb5006b4d0 0x7fdb5006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdb7015aad0 tx=0x7fdb5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.188+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb600594c0 con 0x7fdb700acd60 2024-03-06T07:33:28.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.422+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb7006b9f0 con 0x7fdb700acd60 2024-03-06T07:33:28.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.422+0000 7fdb66a00640 1 -- 172.21.15.97:0/2866089337 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fdb60058dc0 con 0x7fdb700acd60 2024-03-06T07:33:28.426 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:28.426 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:28.429 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb5006b4d0 msgr2=0x7fdb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb5006b4d0 0x7fdb5006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdb7015aad0 tx=0x7fdb5c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 msgr2=0x7fdb701e79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 0x7fdb701e79b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000d620 tx=0x7fdb6000dae0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 shutdown_connections 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdb5006b4d0 0x7fdb5006d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb7015be90 0x7fdb701f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdb700d0780 0x7fdb701596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 --2- 172.21.15.97:0/2866089337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb700acd60 0x7fdb701e79b0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 >> 172.21.15.97:0/2866089337 conn(0x7fdb7006b360 msgr2=0x7fdb7006cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 shutdown_connections 2024-03-06T07:33:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:28.425+0000 7fdb75800640 1 -- 172.21.15.97:0/2866089337 wait complete. 2024-03-06T07:33:28.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:28.662 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1043,"msgr_recv_bytes":578871,"msgr_send_bytes":18675106,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240272932,"msgr_running_send_time":0.138736621,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1042,"sum":0.314582234,"avgtime":0.000301902},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":99888},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13263,"msgr_send_messages":24130,"msgr_recv_bytes":4561111,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.805577485,"msgr_running_send_time":1.811713739,"msgr_running_recv_time":1.579705049,"msgr_running_fast_dispatch_time":0.055049798,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26586,"sum":0.011786700,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":965,"msgr_send_messages":961,"msgr_recv_bytes":357860,"msgr_send_bytes":4591784,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.141152745,"msgr_running_send_time":0.064642531,"msgr_running_recv_time":0.033866069,"msgr_running_fast_dispatch_time":0.036924004,"msgr_send_messages_queue_lat":{"avgcount":961,"sum":0.290796179,"avgtime":0.000302597},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107280,"msgr_send_encrypted_bytes":9168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":954,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15260,"get_sum":4121564,"get_or_fail_fail":0,"get_or_fail_success":15260,"take":0,"take_sum":0,"put":15260,"put_sum":4121564,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:28.664 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.138736621, "msgr_running_total_time": 0.240272932, "msgr_send_bytes": 18675106, "msgr_send_encrypted_bytes": 99888, "msgr_send_messages": 1043, "msgr_send_messages_queue_lat": { "avgcount": 1042, "avgtime": 0.000301902, "sum": 0.314582234 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26586, "avgtime": 4.43e-07, "sum": 0.0117867 }, "msgr_recv_bytes": 4561111, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13263, "msgr_running_fast_dispatch_time": 0.055049798, "msgr_running_recv_time": 1.579705049, "msgr_running_send_time": 1.811713739, "msgr_running_total_time": 2.805577485, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 357860, "msgr_recv_encrypted_bytes": 107280, "msgr_recv_messages": 965, "msgr_running_fast_dispatch_time": 0.036924004, "msgr_running_recv_time": 0.033866069, "msgr_running_send_time": 0.064642531, "msgr_running_total_time": 0.141152745, "msgr_send_bytes": 4591784, "msgr_send_encrypted_bytes": 9168, "msgr_send_messages": 961, "msgr_send_messages_queue_lat": { "avgcount": 961, "avgtime": 0.000302597, "sum": 0.290796179 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 954, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15260, "get_or_fail_fail": 0, "get_or_fail_success": 15260, "get_started": 0, "get_sum": 4121564, "max": 104857600, "put": 15260, "put_sum": 4121564, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:28.664 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:29.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:29.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.874+0000 7fb5fe600640 1 Processor -- start 2024-03-06T07:33:29.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.874+0000 7fb5fe600640 1 -- start start 2024-03-06T07:33:29.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5fe600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5f80cf090 0x7fb5f815e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5fe600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f815f070 con 0x7fb5f80cebe0 2024-03-06T07:33:29.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5fe600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f815f350 con 0x7fb5f80701f0 2024-03-06T07:33:29.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5fe600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f80cf4c0 con 0x7fb5f80cf090 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5f7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5f80cf090 0x7fb5f815e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5efe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb5f80cebe0 0x7fb5f8162520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44932/0 (socket says 172.21.15.97:44932) 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.876+0000 7fb5efe00640 1 -- 172.21.15.97:0/2456091740 learned_addr learned my addr 172.21.15.97:0/2456091740 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.877+0000 7fb5f7e00640 1 -- 172.21.15.97:0/2456091740 >> v1:172.21.15.135:6789/0 conn(0x7fb5f80701f0 legacy=0x7fb5f80cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.877+0000 7fb5f7e00640 1 -- 172.21.15.97:0/2456091740 >> v1:172.21.15.97:6789/0 conn(0x7fb5f80cebe0 legacy=0x7fb5f8162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.877+0000 7fb5f7e00640 1 -- 172.21.15.97:0/2456091740 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f815f630 con 0x7fb5f80cf090 2024-03-06T07:33:29.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.878+0000 7fb5f7e00640 1 --2- 172.21.15.97:0/2456091740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80cf090 0x7fb5f815e9c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009d90 tx=0x7fb5f002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae84848ac223efbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.878+0000 7fb5f6a00640 1 -- 172.21.15.97:0/2456091740 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0039070 con 0x7fb5f80cf090 2024-03-06T07:33:29.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.878+0000 7fb5f6a00640 1 -- 172.21.15.97:0/2456091740 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f002cc60 con 0x7fb5f80cf090 2024-03-06T07:33:29.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.878+0000 7fb5f6a00640 1 -- 172.21.15.97:0/2456091740 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0034660 con 0x7fb5f80cf090 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 -- 172.21.15.97:0/2456091740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80cf090 msgr2=0x7fb5f815e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 --2- 172.21.15.97:0/2456091740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80cf090 0x7fb5f815e9c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009d90 tx=0x7fb5f002beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 -- 172.21.15.97:0/2456091740 shutdown_connections 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 --2- 172.21.15.97:0/2456091740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80cf090 0x7fb5f815e9c0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 -- 172.21.15.97:0/2456091740 >> 172.21.15.97:0/2456091740 conn(0x7fb5f80d0d30 msgr2=0x7fb5f80d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 -- 172.21.15.97:0/2456091740 shutdown_connections 2024-03-06T07:33:29.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.879+0000 7fb5fe600640 1 -- 172.21.15.97:0/2456091740 wait complete. 2024-03-06T07:33:29.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.880+0000 7fb5fe600640 1 Processor -- start 2024-03-06T07:33:29.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 -- start start 2024-03-06T07:33:29.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 0x7fb5f8154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5f80cebe0 0x7fb5f815e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5f80cf090 0x7fb5f80c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8162c30 con 0x7fb5f80701f0 2024-03-06T07:33:29.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80705d0 con 0x7fb5f80cebe0 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.881+0000 7fb5fe600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8162db0 con 0x7fb5f80cf090 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 0x7fb5f8154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 0x7fb5f8154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46782/0 (socket says 172.21.15.97:46782) 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 -- 172.21.15.97:0/449210521 learned_addr learned my addr 172.21.15.97:0/449210521 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 -- 172.21.15.97:0/449210521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5f80cf090 msgr2=0x7fb5f80c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5f80cf090 0x7fb5f80c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 -- 172.21.15.97:0/449210521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5f80cebe0 msgr2=0x7fb5f815e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5f80cebe0 0x7fb5f815e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 -- 172.21.15.97:0/449210521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f80c98f0 con 0x7fb5f80701f0 2024-03-06T07:33:29.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.882+0000 7fb5f7400640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 0x7fb5f8154b90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c460 tx=0x7fb5e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.883+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8016020 con 0x7fb5f80701f0 2024-03-06T07:33:29.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.883+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e8012870 con 0x7fb5f80701f0 2024-03-06T07:33:29.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.883+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f80c9bd0 con 0x7fb5f80701f0 2024-03-06T07:33:29.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.883+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8012c60 con 0x7fb5f80701f0 2024-03-06T07:33:29.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.883+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f8155930 con 0x7fb5f80701f0 2024-03-06T07:33:29.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.884+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f8068070 con 0x7fb5f80701f0 2024-03-06T07:33:29.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.885+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5e8019020 con 0x7fb5f80701f0 2024-03-06T07:33:29.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.886+0000 7fb5ef400640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5d406b4d0 0x7fb5d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.886+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb5e808a1c0 con 0x7fb5f80701f0 2024-03-06T07:33:29.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.886+0000 7fb5efe00640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5d406b4d0 0x7fb5d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.887+0000 7fb5efe00640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5d406b4d0 0x7fb5d406d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0000f60 tx=0x7fb5e0007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:29.890+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5e8020020 con 0x7fb5f80701f0 2024-03-06T07:33:30.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.127+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5f815ece0 con 0x7fb5f80701f0 2024-03-06T07:33:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.128+0000 7fb5ef400640 1 -- 172.21.15.97:0/449210521 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fb5e80182c0 con 0x7fb5f80701f0 2024-03-06T07:33:30.131 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:30.131 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:30.132 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5d406b4d0 msgr2=0x7fb5d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5d406b4d0 0x7fb5d406d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0000f60 tx=0x7fb5e0007a80 comp rx=0 tx=0).stop 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 msgr2=0x7fb5f8154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 0x7fb5f8154b90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c460 tx=0x7fb5e800c920 comp rx=0 tx=0).stop 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 shutdown_connections 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5d406b4d0 0x7fb5d406d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5f80cf090 0x7fb5f80c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5f80cebe0 0x7fb5f815e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 --2- 172.21.15.97:0/449210521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5f80701f0 0x7fb5f8154b90 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 >> 172.21.15.97:0/449210521 conn(0x7fb5f80d0d30 msgr2=0x7fb5f80d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 shutdown_connections 2024-03-06T07:33:30.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:30.131+0000 7fb5fe600640 1 -- 172.21.15.97:0/449210521 wait complete. 2024-03-06T07:33:30.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:30.369 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1043,"msgr_recv_bytes":578871,"msgr_send_bytes":18675106,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240272932,"msgr_running_send_time":0.138736621,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1042,"sum":0.314582234,"avgtime":0.000301902},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":99888},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13265,"msgr_send_messages":24130,"msgr_recv_bytes":4561947,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.806326044,"msgr_running_send_time":1.812069017,"msgr_running_recv_time":1.579883308,"msgr_running_fast_dispatch_time":0.055254051,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26588,"sum":0.011787227,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":966,"msgr_send_messages":962,"msgr_recv_bytes":358132,"msgr_send_bytes":4592408,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.141465131,"msgr_running_send_time":0.064819968,"msgr_running_recv_time":0.033922631,"msgr_running_fast_dispatch_time":0.036976117,"msgr_send_messages_queue_lat":{"avgcount":962,"sum":0.290912958,"avgtime":0.000302404},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107552,"msgr_send_encrypted_bytes":9792},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":959,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15263,"get_sum":4122350,"get_or_fail_fail":0,"get_or_fail_success":15263,"take":0,"take_sum":0,"put":15263,"put_sum":4122350,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:30.371 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.138736621, "msgr_running_total_time": 0.240272932, "msgr_send_bytes": 18675106, "msgr_send_encrypted_bytes": 99888, "msgr_send_messages": 1043, "msgr_send_messages_queue_lat": { "avgcount": 1042, "avgtime": 0.000301902, "sum": 0.314582234 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26588, "avgtime": 4.43e-07, "sum": 0.011787227 }, "msgr_recv_bytes": 4561947, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13265, "msgr_running_fast_dispatch_time": 0.055254051, "msgr_running_recv_time": 1.579883308, "msgr_running_send_time": 1.812069017, "msgr_running_total_time": 2.806326044, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 358132, "msgr_recv_encrypted_bytes": 107552, "msgr_recv_messages": 966, "msgr_running_fast_dispatch_time": 0.036976117, "msgr_running_recv_time": 0.033922631, "msgr_running_send_time": 0.064819968, "msgr_running_total_time": 0.141465131, "msgr_send_bytes": 4592408, "msgr_send_encrypted_bytes": 9792, "msgr_send_messages": 962, "msgr_send_messages_queue_lat": { "avgcount": 962, "avgtime": 0.000302404, "sum": 0.290912958 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 959, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15263, "get_or_fail_fail": 0, "get_or_fail_success": 15263, "get_started": 0, "get_sum": 4122350, "max": 104857600, "put": 15263, "put_sum": 4122350, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:30.371 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:31.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.588+0000 7f5d8b800640 1 Processor -- start 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.589+0000 7f5d8b800640 1 -- start start 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d8b800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d8415be90 0x7f5d840b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d8b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840d0780 0x7f5d8415fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d8b800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d840b5160 con 0x7f5d840acd60 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d8b800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d840b4f30 con 0x7f5d8415be90 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d8b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840af240 con 0x7f5d840d0780 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d88e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d8415be90 0x7f5d840b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.590+0000 7f5d83e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5d840acd60 0x7f5d840d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44936/0 (socket says 172.21.15.97:44936) 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d83e00640 1 -- 172.21.15.97:0/2823379020 learned_addr learned my addr 172.21.15.97:0/2823379020 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:31.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d89800640 1 --2- 172.21.15.97:0/2823379020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840d0780 0x7f5d8415fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d88e00640 1 -- 172.21.15.97:0/2823379020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840d0780 msgr2=0x7f5d8415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d88e00640 1 --2- 172.21.15.97:0/2823379020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840d0780 0x7f5d8415fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d88e00640 1 -- 172.21.15.97:0/2823379020 >> v1:172.21.15.97:6789/0 conn(0x7f5d840acd60 legacy=0x7f5d840d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d88e00640 1 -- 172.21.15.97:0/2823379020 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d840ae910 con 0x7f5d8415be90 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.591+0000 7f5d83400640 1 -- 172.21.15.97:0/2823379020 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409066640 0 0) 0x7f5d840b5160 con 0x7f5d840acd60 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.592+0000 7f5d88e00640 1 --2- 172.21.15.97:0/2823379020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d8415be90 0x7f5d840b49f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009d90 tx=0x7f5d7402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e64e8b46c5d3e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.592+0000 7f5d83400640 1 -- 172.21.15.97:0/2823379020 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74039070 con 0x7f5d8415be90 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d83400640 1 -- 172.21.15.97:0/2823379020 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7402cc50 con 0x7f5d8415be90 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d83400640 1 -- 172.21.15.97:0/2823379020 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74034440 con 0x7f5d8415be90 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 -- 172.21.15.97:0/2823379020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d8415be90 msgr2=0x7f5d840b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 --2- 172.21.15.97:0/2823379020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d8415be90 0x7f5d840b49f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009d90 tx=0x7f5d7402bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 -- 172.21.15.97:0/2823379020 shutdown_connections 2024-03-06T07:33:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 --2- 172.21.15.97:0/2823379020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840d0780 0x7f5d8415fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 --2- 172.21.15.97:0/2823379020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d8415be90 0x7f5d840b49f0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 -- 172.21.15.97:0/2823379020 >> 172.21.15.97:0/2823379020 conn(0x7f5d8406b360 msgr2=0x7f5d8406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 -- 172.21.15.97:0/2823379020 shutdown_connections 2024-03-06T07:33:31.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.593+0000 7f5d8b800640 1 -- 172.21.15.97:0/2823379020 wait complete. 2024-03-06T07:33:31.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.595+0000 7f5d8b800640 1 Processor -- start 2024-03-06T07:33:31.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.595+0000 7f5d8b800640 1 -- start start 2024-03-06T07:33:31.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d8b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840acd60 0x7f5d841466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d8b800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 0x7f5d840cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d8b800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8415be90 0x7f5d840cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d8b800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840b2c80 con 0x7f5d840d0780 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d8b800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840aeed0 con 0x7f5d8415be90 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d8b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840b2e00 con 0x7f5d840acd60 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d83e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 0x7f5d840cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d83e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 0x7f5d840cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46792/0 (socket says 172.21.15.97:46792) 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d83e00640 1 -- 172.21.15.97:0/3906232382 learned_addr learned my addr 172.21.15.97:0/3906232382 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.596+0000 7f5d88e00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840acd60 0x7f5d841466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d83e00640 1 -- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840acd60 msgr2=0x7f5d841466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d83e00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840acd60 0x7f5d841466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d83e00640 1 -- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8415be90 msgr2=0x7f5d840cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d83e00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8415be90 0x7f5d840cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d83e00640 1 -- 172.21.15.97:0/3906232382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d84146c10 con 0x7f5d840d0780 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d83e00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 0x7f5d840cc1c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c4d0 tx=0x7f5d7000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70018070 con 0x7f5d840d0780 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d70007500 con 0x7f5d840d0780 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70004a20 con 0x7f5d840d0780 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d840ccda0 con 0x7f5d840d0780 2024-03-06T07:33:31.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.597+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d840cd360 con 0x7f5d840d0780 2024-03-06T07:33:31.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.599+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84068070 con 0x7f5d840d0780 2024-03-06T07:33:31.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.600+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d70004bc0 con 0x7f5d840d0780 2024-03-06T07:33:31.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.600+0000 7f5d80c00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.601+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5d700895a0 con 0x7f5d840d0780 2024-03-06T07:33:31.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.601+0000 7f5d88e00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.601+0000 7f5d88e00640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5d74036000 tx=0x7f5d74030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.605+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d84146c10 con 0x7f5d840d0780 2024-03-06T07:33:31.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.849+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d840b2870 con 0x7f5d840d0780 2024-03-06T07:33:31.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.850+0000 7f5d80c00640 1 -- 172.21.15.97:0/3906232382 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f5d70058320 con 0x7f5d840d0780 2024-03-06T07:33:31.852 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:31.853 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:31.853 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:31.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 msgr2=0x7f5d6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5d74036000 tx=0x7f5d74030000 comp rx=0 tx=0).stop 2024-03-06T07:33:31.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 msgr2=0x7f5d840cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 0x7f5d840cc1c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c4d0 tx=0x7f5d7000c990 comp rx=0 tx=0).stop 2024-03-06T07:33:31.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 shutdown_connections 2024-03-06T07:33:31.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d6406b4d0 0x7f5d6406d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d8415be90 0x7f5d840cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d840d0780 0x7f5d840cc1c0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 --2- 172.21.15.97:0/3906232382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d840acd60 0x7f5d841466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 >> 172.21.15.97:0/3906232382 conn(0x7f5d8406b360 msgr2=0x7f5d840cf500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 shutdown_connections 2024-03-06T07:33:31.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:31.853+0000 7f5d8b800640 1 -- 172.21.15.97:0/3906232382 wait complete. 2024-03-06T07:33:31.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:32.094 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1044,"msgr_recv_bytes":578871,"msgr_send_bytes":18676898,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240384037,"msgr_running_send_time":0.138835480,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1043,"sum":0.314626535,"avgtime":0.000301655},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":101680},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13267,"msgr_send_messages":24130,"msgr_recv_bytes":4562783,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.806794527,"msgr_running_send_time":1.812258027,"msgr_running_recv_time":1.579982472,"msgr_running_fast_dispatch_time":0.055427358,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26590,"sum":0.011787691,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":966,"msgr_send_messages":962,"msgr_recv_bytes":358132,"msgr_send_bytes":4592408,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.141465131,"msgr_running_send_time":0.064819968,"msgr_running_recv_time":0.033922631,"msgr_running_fast_dispatch_time":0.036976117,"msgr_send_messages_queue_lat":{"avgcount":962,"sum":0.290912958,"avgtime":0.000302404},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107552,"msgr_send_encrypted_bytes":9792},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":959,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15265,"get_sum":4123006,"get_or_fail_fail":0,"get_or_fail_success":15265,"take":0,"take_sum":0,"put":15265,"put_sum":4123006,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:32.164 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.13883548, "msgr_running_total_time": 0.240384037, "msgr_send_bytes": 18676898, "msgr_send_encrypted_bytes": 101680, "msgr_send_messages": 1044, "msgr_send_messages_queue_lat": { "avgcount": 1043, "avgtime": 0.000301655, "sum": 0.314626535 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26590, "avgtime": 4.43e-07, "sum": 0.011787691 }, "msgr_recv_bytes": 4562783, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13267, "msgr_running_fast_dispatch_time": 0.055427358, "msgr_running_recv_time": 1.579982472, "msgr_running_send_time": 1.812258027, "msgr_running_total_time": 2.806794527, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 358132, "msgr_recv_encrypted_bytes": 107552, "msgr_recv_messages": 966, "msgr_running_fast_dispatch_time": 0.036976117, "msgr_running_recv_time": 0.033922631, "msgr_running_send_time": 0.064819968, "msgr_running_total_time": 0.141465131, "msgr_send_bytes": 4592408, "msgr_send_encrypted_bytes": 9792, "msgr_send_messages": 962, "msgr_send_messages_queue_lat": { "avgcount": 962, "avgtime": 0.000302404, "sum": 0.290912958 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 959, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15265, "get_or_fail_fail": 0, "get_or_fail_success": 15265, "get_started": 0, "get_sum": 4123006, "max": 104857600, "put": 15265, "put_sum": 4123006, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:32.165 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:33.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:33.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.366+0000 7fb996c00640 1 Processor -- start 2024-03-06T07:33:33.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.366+0000 7fb996c00640 1 -- start start 2024-03-06T07:33:33.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.367+0000 7fb996c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb9900d2840 0x7fb9900b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.367+0000 7fb996c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900b39d0 con 0x7fb99015e230 2024-03-06T07:33:33.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.367+0000 7fb996c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900b3cb0 con 0x7fb9900d2400 2024-03-06T07:33:33.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.367+0000 7fb996c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9900b3780 con 0x7fb9900d2840 2024-03-06T07:33:33.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.367+0000 7fb995800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb99015e230 0x7fb9900d0e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44946/0 (socket says 172.21.15.97:44946) 2024-03-06T07:33:33.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.367+0000 7fb995800640 1 -- 172.21.15.97:0/3813249728 learned_addr learned my addr 172.21.15.97:0/3813249728 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257910750 0 0) 0x7fb9900b39d0 con 0x7fb99015e230 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb978003520 con 0x7fb99015e230 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb996200640 1 --2- 172.21.15.97:0/3813249728 >> v2:172.21.15.135:3300/0 conn(0x7fb9900d2840 0x7fb9900b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636740001 0 0) 0x7fb978003520 con 0x7fb99015e230 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 >> v2:172.21.15.135:3300/0 conn(0x7fb9900d2840 msgr2=0x7fb9900b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 --2- 172.21.15.97:0/3813249728 >> v2:172.21.15.135:3300/0 conn(0x7fb9900d2840 0x7fb9900b4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 >> v1:172.21.15.135:6789/0 conn(0x7fb9900d2400 legacy=0x7fb9900b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900b3f90 con 0x7fb99015e230 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb984001670 con 0x7fb99015e230 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb996200640 1 --2- 172.21.15.97:0/3813249728 >> v2:172.21.15.135:3300/0 conn(0x7fb9900d2840 0x7fb9900b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 >> v1:172.21.15.97:6789/0 conn(0x7fb99015e230 legacy=0x7fb9900d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb978003a00 0x7fb978003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb978004800 0x7fb978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9780052e0 0x7fb97800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900b3780 con 0x7fb978004800 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97800c220 con 0x7fb978003a00 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.368+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97800c420 con 0x7fb9780052e0 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb994e00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb978004800 0x7fb978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb995800640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb978003a00 0x7fb978003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb994e00640 1 -- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9780052e0 msgr2=0x7fb97800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb994e00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9780052e0 0x7fb97800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb994e00640 1 -- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb978003a00 msgr2=0x7fb978003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb994e00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb978003a00 0x7fb978003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb994e00640 1 -- 172.21.15.97:0/3813249728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb978003520 con 0x7fb978004800 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.369+0000 7fb996200640 1 -- 172.21.15.97:0/3813249728 reap_dead start 2024-03-06T07:33:33.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.370+0000 7fb994e00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb978004800 0x7fb978004c80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fb97c003d80 tx=0x7fb978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8a5dec8211899e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.370+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c037070 con 0x7fb978004800 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.370+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c032600 con 0x7fb978004800 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.370+0000 7fb98fe00640 1 -- 172.21.15.97:0/3813249728 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c003980 con 0x7fb978004800 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 -- 172.21.15.97:0/3813249728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb978004800 msgr2=0x7fb978004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb978004800 0x7fb978004c80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fb97c003d80 tx=0x7fb978005830 comp rx=0 tx=0).stop 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 -- 172.21.15.97:0/3813249728 shutdown_connections 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 --2- 172.21.15.97:0/3813249728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb978004800 0x7fb978004c80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 -- 172.21.15.97:0/3813249728 >> 172.21.15.97:0/3813249728 conn(0x7fb9900d3e20 msgr2=0x7fb9900d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 -- 172.21.15.97:0/3813249728 shutdown_connections 2024-03-06T07:33:33.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.371+0000 7fb996c00640 1 -- 172.21.15.97:0/3813249728 wait complete. 2024-03-06T07:33:33.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.373+0000 7fb996c00640 1 Processor -- start 2024-03-06T07:33:33.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.373+0000 7fb996c00640 1 -- start start 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb996c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb978004800 0x7fb9900cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb996c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 0x7fb99006c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb996c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb990152e10 0x7fb990153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb996c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900d2cd0 con 0x7fb99015f1a0 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb996c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99015e610 con 0x7fb990152e10 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb996c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900d1d90 con 0x7fb978004800 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.374+0000 7fb994e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 0x7fb99006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 0x7fb99006c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46824/0 (socket says 172.21.15.97:46824) 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 -- 172.21.15.97:0/3184689 learned_addr learned my addr 172.21.15.97:0/3184689 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb995800640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb978004800 0x7fb9900cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 -- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb978004800 msgr2=0x7fb9900cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb978004800 0x7fb9900cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb996200640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb990152e10 0x7fb990153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 -- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb990152e10 msgr2=0x7fb990153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb990152e10 0x7fb990153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 -- 172.21.15.97:0/3184689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99015f910 con 0x7fb99015f1a0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb996200640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb990152e10 0x7fb990153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.375+0000 7fb994e00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 0x7fb99006c330 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fb97c02bba0 tx=0x7fb97c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.376+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c042020 con 0x7fb99015f1a0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.376+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c007470 con 0x7fb99015f1a0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.376+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c037040 con 0x7fb99015f1a0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.376+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99015fbf0 con 0x7fb99015f1a0 2024-03-06T07:33:33.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.376+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9901601b0 con 0x7fb99015f1a0 2024-03-06T07:33:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.377+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99004f330 con 0x7fb99015f1a0 2024-03-06T07:33:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.378+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb97c045020 con 0x7fb99015f1a0 2024-03-06T07:33:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.379+0000 7fb98d600640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb97006b4d0 0x7fb97006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.379+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb97c0abb50 con 0x7fb99015f1a0 2024-03-06T07:33:33.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.379+0000 7fb995800640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb97006b4d0 0x7fb97006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.380+0000 7fb995800640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb97006b4d0 0x7fb97006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb984001c10 tx=0x7fb984000f50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.384+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb97c04c070 con 0x7fb99015f1a0 2024-03-06T07:33:33.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.617+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9900b0c50 con 0x7fb99015f1a0 2024-03-06T07:33:33.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.618+0000 7fb98d600640 1 -- 172.21.15.97:0/3184689 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fb97c07a8d0 con 0x7fb99015f1a0 2024-03-06T07:33:33.620 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:33.620 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:33.620 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:33.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb97006b4d0 msgr2=0x7fb97006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb97006b4d0 0x7fb97006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb984001c10 tx=0x7fb984000f50 comp rx=0 tx=0).stop 2024-03-06T07:33:33.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 msgr2=0x7fb99006c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 0x7fb99006c330 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fb97c02bba0 tx=0x7fb97c0036f0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 shutdown_connections 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb97006b4d0 0x7fb97006d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb990152e10 0x7fb990153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb99015f1a0 0x7fb99006c330 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 --2- 172.21.15.97:0/3184689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb978004800 0x7fb9900cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 >> 172.21.15.97:0/3184689 conn(0x7fb9900d3e20 msgr2=0x7fb9900ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 shutdown_connections 2024-03-06T07:33:33.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:33.620+0000 7fb996c00640 1 -- 172.21.15.97:0/3184689 wait complete. 2024-03-06T07:33:33.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:33.859 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1044,"msgr_recv_bytes":578871,"msgr_send_bytes":18676898,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240384037,"msgr_running_send_time":0.138835480,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1043,"sum":0.314626535,"avgtime":0.000301655},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":101680},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13268,"msgr_send_messages":24130,"msgr_recv_bytes":4563201,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.807072499,"msgr_running_send_time":1.812377779,"msgr_running_recv_time":1.580054388,"msgr_running_fast_dispatch_time":0.055509869,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26591,"sum":0.011787921,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":967,"msgr_send_messages":963,"msgr_recv_bytes":358404,"msgr_send_bytes":4593032,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.141771572,"msgr_running_send_time":0.064993602,"msgr_running_recv_time":0.033978609,"msgr_running_fast_dispatch_time":0.037026517,"msgr_send_messages_queue_lat":{"avgcount":963,"sum":0.291021493,"avgtime":0.000302203},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107824,"msgr_send_encrypted_bytes":10416},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":959,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15267,"get_sum":4123464,"get_or_fail_fail":0,"get_or_fail_success":15267,"take":0,"take_sum":0,"put":15267,"put_sum":4123464,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:33.860 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.13883548, "msgr_running_total_time": 0.240384037, "msgr_send_bytes": 18676898, "msgr_send_encrypted_bytes": 101680, "msgr_send_messages": 1044, "msgr_send_messages_queue_lat": { "avgcount": 1043, "avgtime": 0.000301655, "sum": 0.314626535 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26591, "avgtime": 4.43e-07, "sum": 0.011787921 }, "msgr_recv_bytes": 4563201, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13268, "msgr_running_fast_dispatch_time": 0.055509869, "msgr_running_recv_time": 1.580054388, "msgr_running_send_time": 1.812377779, "msgr_running_total_time": 2.807072499, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 358404, "msgr_recv_encrypted_bytes": 107824, "msgr_recv_messages": 967, "msgr_running_fast_dispatch_time": 0.037026517, "msgr_running_recv_time": 0.033978609, "msgr_running_send_time": 0.064993602, "msgr_running_total_time": 0.141771572, "msgr_send_bytes": 4593032, "msgr_send_encrypted_bytes": 10416, "msgr_send_messages": 963, "msgr_send_messages_queue_lat": { "avgcount": 963, "avgtime": 0.000302203, "sum": 0.291021493 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 959, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15267, "get_or_fail_fail": 0, "get_or_fail_success": 15267, "get_started": 0, "get_sum": 4123464, "max": 104857600, "put": 15267, "put_sum": 4123464, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:33.860 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:34.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:35.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.062+0000 7f6b5e000640 1 Processor -- start 2024-03-06T07:33:35.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.062+0000 7f6b5e000640 1 -- start start 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.063+0000 7f6b5e000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b580b77d0 0x7f6b580d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.063+0000 7f6b5e000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d11f0 0x7f6b580ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.063+0000 7f6b5e000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580cc5f0 con 0x7f6b5806d350 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.063+0000 7f6b5e000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b580d16e0 con 0x7f6b580b77d0 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.063+0000 7f6b5e000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580ccee0 con 0x7f6b580d11f0 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b580b77d0 0x7f6b580d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b580b77d0 0x7f6b580d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48870/0 (socket says 172.21.15.97:48870) 2024-03-06T07:33:35.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 -- 172.21.15.97:0/370996614 learned_addr learned my addr 172.21.15.97:0/370996614 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:35.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b56a00640 1 --2- 172.21.15.97:0/370996614 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d11f0 0x7f6b580ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 -- 172.21.15.97:0/370996614 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d11f0 msgr2=0x7f6b580ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 --2- 172.21.15.97:0/370996614 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d11f0 0x7f6b580ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 -- 172.21.15.97:0/370996614 >> v1:172.21.15.135:6789/0 conn(0x7f6b5806d350 legacy=0x7f6b5815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.064+0000 7f6b57400640 1 -- 172.21.15.97:0/370996614 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580cc890 con 0x7f6b580b77d0 2024-03-06T07:33:35.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.065+0000 7f6b57400640 1 --2- 172.21.15.97:0/370996614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b580b77d0 0x7f6b580d0200 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009870 tx=0x7f6b4802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4739e33f379bed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b56000640 1 -- 172.21.15.97:0/370996614 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b480317c0 con 0x7f6b580b77d0 2024-03-06T07:33:35.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b56000640 1 -- 172.21.15.97:0/370996614 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48037070 con 0x7f6b580b77d0 2024-03-06T07:33:35.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b56000640 1 -- 172.21.15.97:0/370996614 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48031b80 con 0x7f6b580b77d0 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b5e000640 1 -- 172.21.15.97:0/370996614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b580b77d0 msgr2=0x7f6b580d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b5e000640 1 --2- 172.21.15.97:0/370996614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b580b77d0 0x7f6b580d0200 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009870 tx=0x7f6b4802b860 comp rx=0 tx=0).stop 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b5e000640 1 -- 172.21.15.97:0/370996614 shutdown_connections 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b5e000640 1 --2- 172.21.15.97:0/370996614 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d11f0 0x7f6b580ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b5e000640 1 --2- 172.21.15.97:0/370996614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b580b77d0 0x7f6b580d0200 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.066+0000 7f6b5e000640 1 -- 172.21.15.97:0/370996614 >> 172.21.15.97:0/370996614 conn(0x7f6b580d1e00 msgr2=0x7f6b580d2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.067+0000 7f6b5e000640 1 -- 172.21.15.97:0/370996614 shutdown_connections 2024-03-06T07:33:35.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.067+0000 7f6b5e000640 1 -- 172.21.15.97:0/370996614 wait complete. 2024-03-06T07:33:35.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.069+0000 7f6b5e000640 1 Processor -- start 2024-03-06T07:33:35.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.069+0000 7f6b5e000640 1 -- start start 2024-03-06T07:33:35.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b5e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 0x7f6b580c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b5e000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580b77d0 0x7f6b580cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b5e000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580d11f0 0x7f6b5815dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b5e000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580b7d30 con 0x7f6b5806d350 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b5e000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580b7bb0 con 0x7f6b580d11f0 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b5e000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580b4c50 con 0x7f6b580b77d0 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b57400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 0x7f6b580c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b57400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 0x7f6b580c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48884/0 (socket says 172.21.15.97:48884) 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.070+0000 7f6b57400640 1 -- 172.21.15.97:0/1595747503 learned_addr learned my addr 172.21.15.97:0/1595747503 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b56a00640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580b77d0 0x7f6b580cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57400640 1 -- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580b77d0 msgr2=0x7f6b580cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57400640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580b77d0 0x7f6b580cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57400640 1 -- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580d11f0 msgr2=0x7f6b5815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57e00640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580d11f0 0x7f6b5815dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57400640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580d11f0 0x7f6b5815dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57400640 1 -- 172.21.15.97:0/1595747503 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580ca930 con 0x7f6b5806d350 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b56a00640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580b77d0 0x7f6b580cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b57400640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 0x7f6b580c8220 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009870 tx=0x7f6b480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48032650 con 0x7f6b5806d350 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b480327f0 con 0x7f6b5806d350 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.071+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5815e220 con 0x7f6b5806d350 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.072+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48032be0 con 0x7f6b5806d350 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.072+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b581e6ed0 con 0x7f6b5806d350 2024-03-06T07:33:35.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.073+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5806d7b0 con 0x7f6b5806d350 2024-03-06T07:33:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.073+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b4804c060 con 0x7f6b5806d350 2024-03-06T07:33:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.074+0000 7f6b4f400640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3806b500 0x7f6b3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.074+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6b480abd70 con 0x7f6b5806d350 2024-03-06T07:33:35.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.075+0000 7f6b56a00640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3806b500 0x7f6b3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.075+0000 7f6b56a00640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3806b500 0x7f6b3806d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6b580705c0 tx=0x7f6b44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.079+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b5806d7b0 con 0x7f6b5806d350 2024-03-06T07:33:35.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.389+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b580b36e0 con 0x7f6b5806d350 2024-03-06T07:33:35.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.390+0000 7f6b4f400640 1 -- 172.21.15.97:0/1595747503 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f6b580b36e0 con 0x7f6b5806d350 2024-03-06T07:33:35.392 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:35.392 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:35.392 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:35.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3806b500 msgr2=0x7f6b3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3806b500 0x7f6b3806d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6b580705c0 tx=0x7f6b44008040 comp rx=0 tx=0).stop 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 msgr2=0x7f6b580c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 0x7f6b580c8220 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009870 tx=0x7f6b480046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 shutdown_connections 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3806b500 0x7f6b3806d9c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580d11f0 0x7f6b5815dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580b77d0 0x7f6b580cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 --2- 172.21.15.97:0/1595747503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5806d350 0x7f6b580c8220 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.392+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 >> 172.21.15.97:0/1595747503 conn(0x7f6b580d1e00 msgr2=0x7f6b580b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.393+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 shutdown_connections 2024-03-06T07:33:35.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:35.393+0000 7f6b5e000640 1 -- 172.21.15.97:0/1595747503 wait complete. 2024-03-06T07:33:35.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:35.621 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1036,"msgr_send_messages":1044,"msgr_recv_bytes":578871,"msgr_send_bytes":18676898,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240384037,"msgr_running_send_time":0.138835480,"msgr_running_recv_time":0.034886293,"msgr_running_fast_dispatch_time":0.054102973,"msgr_send_messages_queue_lat":{"avgcount":1043,"sum":0.314626535,"avgtime":0.000301655},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":101680},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13270,"msgr_send_messages":24130,"msgr_recv_bytes":4564037,"msgr_send_bytes":13578927,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.807638696,"msgr_running_send_time":1.812641968,"msgr_running_recv_time":1.580181277,"msgr_running_fast_dispatch_time":0.055675176,"msgr_send_messages_queue_lat":{"avgcount":24129,"sum":1.946934927,"avgtime":0.000080688},"msgr_handle_ack_lat":{"avgcount":26593,"sum":0.011788442,"avgtime":0.000000443},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":967,"msgr_send_messages":963,"msgr_recv_bytes":358404,"msgr_send_bytes":4593032,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.141771572,"msgr_running_send_time":0.064993602,"msgr_running_recv_time":0.033978609,"msgr_running_fast_dispatch_time":0.037026517,"msgr_send_messages_queue_lat":{"avgcount":963,"sum":0.291021493,"avgtime":0.000302203},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107824,"msgr_send_encrypted_bytes":10416},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":11.171612955,"avgtime":0.090093652}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6747,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4905,"handle_client_caps_dirty":3000,"handle_client_cap_release":393,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15093,"evex":15002,"evtrm":15002,"ev":96,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34143964,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":178,"cap+":3202,"cap-":3024,"rss":85596,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":964,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3490,"op_send_bytes":30988673,"op_resend":0,"op_reply":3490,"op_latency":{"avgcount":3490,"sum":19.356669931,"avgtime":0.005546323},"op_inflight":0,"oplen_avg":{"avgcount":3490,"sum":4141},"op":3490,"op_r":61,"op_w":3429,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15269,"get_sum":4124120,"get_or_fail_fail":0,"get_or_fail_success":15269,"take":0,"take_sum":0,"put":15269,"put_sum":4124120,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":31296018,"put":3490,"put_sum":31296018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3490,"take_sum":3490,"put":3490,"put_sum":3490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":0,"max":3758096384,"get_started":0,"get":15093,"get_sum":29947475,"get_or_fail_fail":0,"get_or_fail_success":15093,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:35.622 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 578871, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1036, "msgr_running_fast_dispatch_time": 0.054102973, "msgr_running_recv_time": 0.034886293, "msgr_running_send_time": 0.13883548, "msgr_running_total_time": 0.240384037, "msgr_send_bytes": 18676898, "msgr_send_encrypted_bytes": 101680, "msgr_send_messages": 1044, "msgr_send_messages_queue_lat": { "avgcount": 1043, "avgtime": 0.000301655, "sum": 0.314626535 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26593, "avgtime": 4.43e-07, "sum": 0.011788442 }, "msgr_recv_bytes": 4564037, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13270, "msgr_running_fast_dispatch_time": 0.055675176, "msgr_running_recv_time": 1.580181277, "msgr_running_send_time": 1.812641968, "msgr_running_total_time": 2.807638696, "msgr_send_bytes": 13578927, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24130, "msgr_send_messages_queue_lat": { "avgcount": 24129, "avgtime": 8.0688e-05, "sum": 1.946934927 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 358404, "msgr_recv_encrypted_bytes": 107824, "msgr_recv_messages": 967, "msgr_running_fast_dispatch_time": 0.037026517, "msgr_running_recv_time": 0.033978609, "msgr_running_send_time": 0.064993602, "msgr_running_total_time": 0.141771572, "msgr_send_bytes": 4593032, "msgr_send_encrypted_bytes": 10416, "msgr_send_messages": 963, "msgr_send_messages_queue_lat": { "avgcount": 963, "avgtime": 0.000302203, "sum": 0.291021493 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.090093652, "sum": 11.171612955 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6747, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 393, "handle_client_caps": 4905, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 96, "evadd": 15093, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34143964 }, "mds_mem": { "cap": 178, "cap+": 3202, "cap-": 3024, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85596 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 964, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 55, "op": 3490, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3490, "avgtime": 0.005546323, "sum": 19.356669931 }, "op_pg": 0, "op_r": 61, "op_reply": 3490, "op_resend": 0, "op_rmw": 0, "op_send": 3490, "op_send_bytes": 30988673, "op_w": 3429, "oplen_avg": { "avgcount": 3490, "sum": 4141 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15269, "get_or_fail_fail": 0, "get_or_fail_success": 15269, "get_started": 0, "get_sum": 4124120, "max": 104857600, "put": 15269, "put_sum": 4124120, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3490, "put_sum": 31296018, "take": 3490, "take_sum": 31296018, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3490, "put_sum": 3490, "take": 3490, "take_sum": 3490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15093, "get_or_fail_fail": 0, "get_or_fail_success": 15093, "get_started": 0, "get_sum": 29947475, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:35.622 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:36.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:36.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.834+0000 7f1521800640 1 Processor -- start 2024-03-06T07:33:36.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.834+0000 7f1521800640 1 -- start start 2024-03-06T07:33:36.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.835+0000 7f1521800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f151c15be90 0x7f151c0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.835+0000 7f1521800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0acd60 0x7f151c0d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.835+0000 7f1521800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0b5160 con 0x7f151c0d05b0 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.835+0000 7f1521800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c0b4f30 con 0x7f151c15be90 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.835+0000 7f1521800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0af240 con 0x7f151c0acd60 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f151c15be90 0x7f151c0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f151c15be90 0x7f151c0b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48890/0 (socket says 172.21.15.97:48890) 2024-03-06T07:33:36.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 -- 172.21.15.97:0/700920595 learned_addr learned my addr 172.21.15.97:0/700920595 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:36.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 -- 172.21.15.97:0/700920595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0acd60 msgr2=0x7f151c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 --2- 172.21.15.97:0/700920595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0acd60 0x7f151c0d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 -- 172.21.15.97:0/700920595 >> v1:172.21.15.135:6789/0 conn(0x7f151c0d05b0 legacy=0x7f151c06c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.836+0000 7f151aa00640 1 -- 172.21.15.97:0/700920595 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0ae910 con 0x7f151c15be90 2024-03-06T07:33:36.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.837+0000 7f151aa00640 1 --2- 172.21.15.97:0/700920595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c0b49f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f150c009870 tx=0x7f150c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b4a17cf4da5f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.837+0000 7f151a000640 1 -- 172.21.15.97:0/700920595 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c035020 con 0x7f151c15be90 2024-03-06T07:33:36.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.837+0000 7f151a000640 1 -- 172.21.15.97:0/700920595 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c031640 con 0x7f151c15be90 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.837+0000 7f151a000640 1 -- 172.21.15.97:0/700920595 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c02d040 con 0x7f151c15be90 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 -- 172.21.15.97:0/700920595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 msgr2=0x7f151c0b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 --2- 172.21.15.97:0/700920595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c0b49f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f150c009870 tx=0x7f150c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 -- 172.21.15.97:0/700920595 shutdown_connections 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 --2- 172.21.15.97:0/700920595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0acd60 0x7f151c0d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 --2- 172.21.15.97:0/700920595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c0b49f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 -- 172.21.15.97:0/700920595 >> 172.21.15.97:0/700920595 conn(0x7f151c06b360 msgr2=0x7f151c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 -- 172.21.15.97:0/700920595 shutdown_connections 2024-03-06T07:33:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.838+0000 7f1521800640 1 -- 172.21.15.97:0/700920595 wait complete. 2024-03-06T07:33:36.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.840+0000 7f1521800640 1 Processor -- start 2024-03-06T07:33:36.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.840+0000 7f1521800640 1 -- start start 2024-03-06T07:33:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f1521800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f151c0acd60 0x7f151c156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f1521800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0d05b0 0x7f151c15e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f1521800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f1521800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0b2c80 con 0x7f151c15be90 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f1521800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0aeed0 con 0x7f151c0acd60 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f1521800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0b2e00 con 0x7f151c0d05b0 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f151b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f151b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c15ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48902/0 (socket says 172.21.15.97:48902) 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.841+0000 7f151b400640 1 -- 172.21.15.97:0/1040407429 learned_addr learned my addr 172.21.15.97:0/1040407429 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151aa00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f151c0acd60 0x7f151c156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f1513e00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0d05b0 0x7f151c15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151b400640 1 -- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0d05b0 msgr2=0x7f151c15e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151b400640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0d05b0 0x7f151c15e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151b400640 1 -- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f151c0acd60 msgr2=0x7f151c156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151b400640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f151c0acd60 0x7f151c156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151b400640 1 -- 172.21.15.97:0/1040407429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c1f2c30 con 0x7f151c15be90 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f1513e00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0d05b0 0x7f151c15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151aa00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f151c0acd60 0x7f151c156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f151b400640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c15ee80 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f151400c780 tx=0x7f151400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514004b10 con 0x7f151c15be90 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514004cb0 con 0x7f151c15be90 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151400aa90 con 0x7f151c15be90 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.842+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c157f80 con 0x7f151c15be90 2024-03-06T07:33:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.843+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c158540 con 0x7f151c15be90 2024-03-06T07:33:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.844+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c068070 con 0x7f151c15be90 2024-03-06T07:33:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.845+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f151400ac30 con 0x7f151c15be90 2024-03-06T07:33:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.846+0000 7f1512a00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14f806b4d0 0x7f14f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.846+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f151408a4f0 con 0x7f151c15be90 2024-03-06T07:33:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.846+0000 7f151aa00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14f806b4d0 0x7f14f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.847+0000 7f151aa00640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14f806b4d0 0x7f14f806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f150c036000 tx=0x7f150c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:36.851+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1514020070 con 0x7f151c15be90 2024-03-06T07:33:37.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.083+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f151c06d4d0 con 0x7f151c15be90 2024-03-06T07:33:37.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.084+0000 7f1512a00640 1 -- 172.21.15.97:0/1040407429 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f1514059300 con 0x7f151c15be90 2024-03-06T07:33:37.086 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:37.086 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:37.087 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:37.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14f806b4d0 msgr2=0x7f14f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14f806b4d0 0x7f14f806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f150c036000 tx=0x7f150c03c040 comp rx=0 tx=0).stop 2024-03-06T07:33:37.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 msgr2=0x7f151c15ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c15ee80 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f151400c780 tx=0x7f151400cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 shutdown_connections 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f14f806b4d0 0x7f14f806d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f151c15be90 0x7f151c15ee80 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f151c0d05b0 0x7f151c15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 --2- 172.21.15.97:0/1040407429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f151c0acd60 0x7f151c156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 >> 172.21.15.97:0/1040407429 conn(0x7f151c06b360 msgr2=0x7f151c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 shutdown_connections 2024-03-06T07:33:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:37.087+0000 7f1521800640 1 -- 172.21.15.97:0/1040407429 wait complete. 2024-03-06T07:33:37.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:37.340 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1037,"msgr_send_messages":1046,"msgr_recv_bytes":579163,"msgr_send_bytes":18681821,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240673911,"msgr_running_send_time":0.139053783,"msgr_running_recv_time":0.034904319,"msgr_running_fast_dispatch_time":0.054126285,"msgr_send_messages_queue_lat":{"avgcount":1045,"sum":0.314777753,"avgtime":0.000301222},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":103472},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13550,"msgr_send_messages":24236,"msgr_recv_bytes":4644929,"msgr_send_bytes":13616875,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.838780674,"msgr_running_send_time":1.835390443,"msgr_running_recv_time":1.607911949,"msgr_running_fast_dispatch_time":0.055764993,"msgr_send_messages_queue_lat":{"avgcount":24235,"sum":1.965176939,"avgtime":0.000081088},"msgr_handle_ack_lat":{"avgcount":26968,"sum":0.011843081,"avgtime":0.000000439},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":967,"msgr_send_messages":963,"msgr_recv_bytes":358404,"msgr_send_bytes":4593032,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.141771572,"msgr_running_send_time":0.064993602,"msgr_running_recv_time":0.033978609,"msgr_running_fast_dispatch_time":0.037026517,"msgr_send_messages_queue_lat":{"avgcount":963,"sum":0.291021493,"avgtime":0.000302203},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107824,"msgr_send_encrypted_bytes":10416},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":125,"sum":11.171623770,"avgtime":0.089372990}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6853,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5082,"handle_client_caps_dirty":3000,"handle_client_cap_release":494,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15270,"evex":15002,"evtrm":15002,"ev":273,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34497913,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":1,"cap+":3202,"cap-":3201,"rss":85980,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":964,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3491,"op_send_bytes":30991444,"op_resend":0,"op_reply":3491,"op_latency":{"avgcount":3491,"sum":19.358669897,"avgtime":0.005545307},"op_inflight":0,"oplen_avg":{"avgcount":3491,"sum":4143},"op":3491,"op_r":61,"op_w":3430,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":56,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15550,"get_sum":4180014,"get_or_fail_fail":0,"get_or_fail_success":15550,"take":0,"take_sum":0,"put":15550,"put_sum":4180014,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3491,"take_sum":31298789,"put":3491,"put_sum":31298789,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3491,"take_sum":3491,"put":3491,"put_sum":3491,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":353949,"max":3758096384,"get_started":0,"get":15270,"get_sum":30301424,"get_or_fail_fail":0,"get_or_fail_success":15270,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:37.341 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 579163, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1037, "msgr_running_fast_dispatch_time": 0.054126285, "msgr_running_recv_time": 0.034904319, "msgr_running_send_time": 0.139053783, "msgr_running_total_time": 0.240673911, "msgr_send_bytes": 18681821, "msgr_send_encrypted_bytes": 103472, "msgr_send_messages": 1046, "msgr_send_messages_queue_lat": { "avgcount": 1045, "avgtime": 0.000301222, "sum": 0.314777753 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26968, "avgtime": 4.39e-07, "sum": 0.011843081 }, "msgr_recv_bytes": 4644929, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13550, "msgr_running_fast_dispatch_time": 0.055764993, "msgr_running_recv_time": 1.607911949, "msgr_running_send_time": 1.835390443, "msgr_running_total_time": 2.838780674, "msgr_send_bytes": 13616875, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24236, "msgr_send_messages_queue_lat": { "avgcount": 24235, "avgtime": 8.1088e-05, "sum": 1.965176939 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 358404, "msgr_recv_encrypted_bytes": 107824, "msgr_recv_messages": 967, "msgr_running_fast_dispatch_time": 0.037026517, "msgr_running_recv_time": 0.033978609, "msgr_running_send_time": 0.064993602, "msgr_running_total_time": 0.141771572, "msgr_send_bytes": 4593032, "msgr_send_encrypted_bytes": 10416, "msgr_send_messages": 963, "msgr_send_messages_queue_lat": { "avgcount": 963, "avgtime": 0.000302203, "sum": 0.291021493 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 125, "avgtime": 0.08937299, "sum": 11.17162377 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6853, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 494, "handle_client_caps": 5082, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 273, "evadd": 15270, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34497913 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85980 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 964, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 56, "op": 3491, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3491, "avgtime": 0.005545307, "sum": 19.358669897 }, "op_pg": 0, "op_r": 61, "op_reply": 3491, "op_resend": 0, "op_rmw": 0, "op_send": 3491, "op_send_bytes": 30991444, "op_w": 3430, "oplen_avg": { "avgcount": 3491, "sum": 4143 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15550, "get_or_fail_fail": 0, "get_or_fail_success": 15550, "get_started": 0, "get_sum": 4180014, "max": 104857600, "put": 15550, "put_sum": 4180014, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3491, "put_sum": 31298789, "take": 3491, "take_sum": 31298789, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3491, "put_sum": 3491, "take": 3491, "take_sum": 3491, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15270, "get_or_fail_fail": 0, "get_or_fail_success": 15270, "get_started": 0, "get_sum": 30301424, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 353949, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:37.341 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:38.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:38.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:38.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.552+0000 7fc5e7e00640 1 Processor -- start 2024-03-06T07:33:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.552+0000 7fc5e7e00640 1 -- start start 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.553+0000 7fc5e7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e0155490 0x7fc5e00ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.553+0000 7fc5e7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5e00d0070 0x7fc5e00d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.553+0000 7fc5e7e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e0155870 con 0x7fc5e015ec10 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.553+0000 7fc5e7e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00ad640 con 0x7fc5e00d0070 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.553+0000 7fc5e7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e015f980 con 0x7fc5e0155490 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5e00d0070 0x7fc5e00d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5e00d0070 0x7fc5e00d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48906/0 (socket says 172.21.15.97:48906) 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e5e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc5e015ec10 0x7fc5e015f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58542/0 (socket says 172.21.15.97:58542) 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 -- 172.21.15.97:0/2779497723 learned_addr learned my addr 172.21.15.97:0/2779497723 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e5400640 1 --2- 172.21.15.97:0/2779497723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e0155490 0x7fc5e00ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 -- 172.21.15.97:0/2779497723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e0155490 msgr2=0x7fc5e00ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 --2- 172.21.15.97:0/2779497723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e0155490 0x7fc5e00ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 -- 172.21.15.97:0/2779497723 >> v1:172.21.15.97:6789/0 conn(0x7fc5e015ec10 legacy=0x7fc5e015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5dfe00640 1 -- 172.21.15.97:0/2779497723 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885304218 0 0) 0x7fc5e0155870 con 0x7fc5e015ec10 2024-03-06T07:33:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.554+0000 7fc5e4a00640 1 -- 172.21.15.97:0/2779497723 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00ad870 con 0x7fc5e00d0070 2024-03-06T07:33:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.556+0000 7fc5e4a00640 1 --2- 172.21.15.97:0/2779497723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d0070 0x7fc5e00d04f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00bb10 tx=0x7fc5cc02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2109cc9990bc64c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.556+0000 7fc5dfe00640 1 -- 172.21.15.97:0/2779497723 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc03b070 con 0x7fc5e00d0070 2024-03-06T07:33:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.556+0000 7fc5dfe00640 1 -- 172.21.15.97:0/2779497723 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc036440 con 0x7fc5e00d0070 2024-03-06T07:33:38.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.556+0000 7fc5dfe00640 1 -- 172.21.15.97:0/2779497723 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc036810 con 0x7fc5e00d0070 2024-03-06T07:33:38.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 -- 172.21.15.97:0/2779497723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d0070 msgr2=0x7fc5e00d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/2779497723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d0070 0x7fc5e00d04f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00bb10 tx=0x7fc5cc02e150 comp rx=0 tx=0).stop 2024-03-06T07:33:38.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 -- 172.21.15.97:0/2779497723 shutdown_connections 2024-03-06T07:33:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/2779497723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e00d0070 0x7fc5e00d04f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/2779497723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e0155490 0x7fc5e00ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 -- 172.21.15.97:0/2779497723 >> 172.21.15.97:0/2779497723 conn(0x7fc5e006b360 msgr2=0x7fc5e006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 -- 172.21.15.97:0/2779497723 shutdown_connections 2024-03-06T07:33:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.557+0000 7fc5e7e00640 1 -- 172.21.15.97:0/2779497723 wait complete. 2024-03-06T07:33:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.559+0000 7fc5e7e00640 1 Processor -- start 2024-03-06T07:33:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.559+0000 7fc5e7e00640 1 -- start start 2024-03-06T07:33:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d0070 0x7fc5e0149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 0x7fc5e0156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e015ec10 0x7fc5e01572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00ae110 con 0x7fc5e0155490 2024-03-06T07:33:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00ade30 con 0x7fc5e00d0070 2024-03-06T07:33:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e0071960 con 0x7fc5e015ec10 2024-03-06T07:33:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 0x7fc5e0156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 0x7fc5e0156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48920/0 (socket says 172.21.15.97:48920) 2024-03-06T07:33:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.560+0000 7fc5e4a00640 1 -- 172.21.15.97:0/512258532 learned_addr learned my addr 172.21.15.97:0/512258532 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e5e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e015ec10 0x7fc5e01572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e5400640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d0070 0x7fc5e0149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e4a00640 1 -- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e015ec10 msgr2=0x7fc5e01572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e4a00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e015ec10 0x7fc5e01572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e4a00640 1 -- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d0070 msgr2=0x7fc5e0149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e4a00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d0070 0x7fc5e0149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e4a00640 1 -- 172.21.15.97:0/512258532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0157950 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e5e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e015ec10 0x7fc5e01572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e5400640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d0070 0x7fc5e0149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e4a00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 0x7fc5e0156db0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00b2c0 tx=0x7fc5cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc046070 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc004230 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e0157c30 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.561+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc03b070 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.562+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e01581f0 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.563+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e0068070 con 0x7fc5e0155490 2024-03-06T07:33:38.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.564+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5cc049070 con 0x7fc5e0155490 2024-03-06T07:33:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.564+0000 7fc5dd600640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5c006b4d0 0x7fc5c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.565+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc5cc0ac370 con 0x7fc5e0155490 2024-03-06T07:33:38.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.565+0000 7fc5e5400640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5c006b4d0 0x7fc5c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.565+0000 7fc5e5400640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5c006b4d0 0x7fc5c006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4006fd0 tx=0x7fc5d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.571+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5cc07b0f0 con 0x7fc5e0155490 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.809+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e0071740 con 0x7fc5e0155490 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.810+0000 7fc5dd600640 1 -- 172.21.15.97:0/512258532 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fc5cc07a9f0 con 0x7fc5e0155490 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.812+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5c006b4d0 msgr2=0x7fc5c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.812+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5c006b4d0 0x7fc5c006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4006fd0 tx=0x7fc5d4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.812+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 msgr2=0x7fc5e0156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.812+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 0x7fc5e0156db0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00b2c0 tx=0x7fc5cc004730 comp rx=0 tx=0).stop 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 shutdown_connections 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc5c006b4d0 0x7fc5c006d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5e015ec10 0x7fc5e01572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e0155490 0x7fc5e0156db0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 --2- 172.21.15.97:0/512258532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e00d0070 0x7fc5e0149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 >> 172.21.15.97:0/512258532 conn(0x7fc5e006b360 msgr2=0x7fc5e015f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 shutdown_connections 2024-03-06T07:33:38.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:38.813+0000 7fc5e7e00640 1 -- 172.21.15.97:0/512258532 wait complete. 2024-03-06T07:33:38.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:39.057 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1037,"msgr_send_messages":1046,"msgr_recv_bytes":579163,"msgr_send_bytes":18681821,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.240673911,"msgr_running_send_time":0.139053783,"msgr_running_recv_time":0.034904319,"msgr_running_fast_dispatch_time":0.054126285,"msgr_send_messages_queue_lat":{"avgcount":1045,"sum":0.314777753,"avgtime":0.000301222},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":103472},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13552,"msgr_send_messages":24236,"msgr_recv_bytes":4645765,"msgr_send_bytes":13616875,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.839171522,"msgr_running_send_time":1.835562184,"msgr_running_recv_time":1.607998263,"msgr_running_fast_dispatch_time":0.055891056,"msgr_send_messages_queue_lat":{"avgcount":24235,"sum":1.965176939,"avgtime":0.000081088},"msgr_handle_ack_lat":{"avgcount":26970,"sum":0.011843479,"avgtime":0.000000439},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":968,"msgr_send_messages":964,"msgr_recv_bytes":358676,"msgr_send_bytes":4593656,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.142133621,"msgr_running_send_time":0.065176477,"msgr_running_recv_time":0.034058042,"msgr_running_fast_dispatch_time":0.037075233,"msgr_send_messages_queue_lat":{"avgcount":964,"sum":0.291066106,"avgtime":0.000301935},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108096,"msgr_send_encrypted_bytes":11040},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":125,"sum":11.171623770,"avgtime":0.089372990}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5212,"sum":2.218106468,"avgtime":0.000425576}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":178,"caps":178,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6853,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5082,"handle_client_caps_dirty":3000,"handle_client_cap_release":494,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":177,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3024,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15270,"evex":15002,"evtrm":15002,"ev":273,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34497913,"rdpos":4195674,"jlat":{"avgcount":27,"sum":7.260217361,"avgtime":0.268896939},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":1,"cap+":3202,"cap-":3201,"rss":85980,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":964,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":250843,"buffer_anon_items":269,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":734750,"mds_co_items":13587,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3491,"op_send_bytes":30991444,"op_resend":0,"op_reply":3491,"op_latency":{"avgcount":3491,"sum":19.358669897,"avgtime":0.005545307},"op_inflight":0,"oplen_avg":{"avgcount":3491,"sum":4143},"op":3491,"op_r":61,"op_w":3430,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":33,"osdop_write":52,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3049,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":56,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3024,"pq_executed":3024,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15553,"get_sum":4180800,"get_or_fail_fail":0,"get_or_fail_success":15553,"take":0,"take_sum":0,"put":15553,"put_sum":4180800,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3491,"take_sum":31298789,"put":3491,"put_sum":31298789,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3491,"take_sum":3491,"put":3491,"put_sum":3491,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3024,"get_sum":305424,"get_or_fail_fail":0,"get_or_fail_success":3024,"take":0,"take_sum":0,"put":25,"put_sum":305424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":353949,"max":3758096384,"get_started":0,"get":15270,"get_sum":30301424,"get_or_fail_fail":0,"get_or_fail_success":15270,"take":0,"take_sum":0,"put":27,"put_sum":29947475,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:39.058 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 579163, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1037, "msgr_running_fast_dispatch_time": 0.054126285, "msgr_running_recv_time": 0.034904319, "msgr_running_send_time": 0.139053783, "msgr_running_total_time": 0.240673911, "msgr_send_bytes": 18681821, "msgr_send_encrypted_bytes": 103472, "msgr_send_messages": 1046, "msgr_send_messages_queue_lat": { "avgcount": 1045, "avgtime": 0.000301222, "sum": 0.314777753 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26970, "avgtime": 4.39e-07, "sum": 0.011843479 }, "msgr_recv_bytes": 4645765, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13552, "msgr_running_fast_dispatch_time": 0.055891056, "msgr_running_recv_time": 1.607998263, "msgr_running_send_time": 1.835562184, "msgr_running_total_time": 2.839171522, "msgr_send_bytes": 13616875, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24236, "msgr_send_messages_queue_lat": { "avgcount": 24235, "avgtime": 8.1088e-05, "sum": 1.965176939 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 358676, "msgr_recv_encrypted_bytes": 108096, "msgr_recv_messages": 968, "msgr_running_fast_dispatch_time": 0.037075233, "msgr_running_recv_time": 0.034058042, "msgr_running_send_time": 0.065176477, "msgr_running_total_time": 0.142133621, "msgr_send_bytes": 4593656, "msgr_send_encrypted_bytes": 11040, "msgr_send_messages": 964, "msgr_send_messages_queue_lat": { "avgcount": 964, "avgtime": 0.000301935, "sum": 0.291066106 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 125, "avgtime": 0.08937299, "sum": 11.17162377 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5212, "avgtime": 0.000425576, "sum": 2.218106468 }, "queue_len": 0 }, "mds": { "caps": 178, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6853, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 494, "handle_client_caps": 5082, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 178, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 177, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3024, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 273, "evadd": 15270, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 27, "avgtime": 0.268896939, "sum": 7.260217361 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34497913 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85980 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 964, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 250843, "buffer_anon_items": 269, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 734750, "mds_co_items": 13587, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 56, "op": 3491, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3491, "avgtime": 0.005545307, "sum": 19.358669897 }, "op_pg": 0, "op_r": 61, "op_reply": 3491, "op_resend": 0, "op_rmw": 0, "op_send": 3491, "op_send_bytes": 30991444, "op_w": 3430, "oplen_avg": { "avgcount": 3491, "sum": 4143 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3049, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 52, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3024, "pq_executed_ops": 3024, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15553, "get_or_fail_fail": 0, "get_or_fail_success": 15553, "get_started": 0, "get_sum": 4180800, "max": 104857600, "put": 15553, "put_sum": 4180800, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3491, "put_sum": 31298789, "take": 3491, "take_sum": 31298789, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3491, "put_sum": 3491, "take": 3491, "take_sum": 3491, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3024, "get_or_fail_fail": 0, "get_or_fail_success": 3024, "get_started": 0, "get_sum": 305424, "max": 3758096384, "put": 25, "put_sum": 305424, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15270, "get_or_fail_fail": 0, "get_or_fail_success": 15270, "get_started": 0, "get_sum": 30301424, "max": 3758096384, "put": 27, "put_sum": 29947475, "take": 0, "take_sum": 0, "val": 353949, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:39.059 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/177, 3024/3201 2024-03-06T07:33:40.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:40.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.263+0000 7fac77c00640 1 Processor -- start 2024-03-06T07:33:40.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.263+0000 7fac77c00640 1 -- start start 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac77c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac700b3340 0x7fac700d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac77c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac7015ee30 0x7fac700d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac77c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7015f370 con 0x7fac7006d600 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac77c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7015f650 con 0x7fac700b3340 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac77c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7006cb20 con 0x7fac7015ee30 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac75200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac700b3340 0x7fac700d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac75200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac700b3340 0x7fac700d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48922/0 (socket says 172.21.15.97:48922) 2024-03-06T07:33:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac75200640 1 -- 172.21.15.97:0/4014024740 learned_addr learned my addr 172.21.15.97:0/4014024740 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:40.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.265+0000 7fac75c00640 1 --2- 172.21.15.97:0/4014024740 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac7015ee30 0x7fac700d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.266+0000 7fac75200640 1 -- 172.21.15.97:0/4014024740 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac7015ee30 msgr2=0x7fac700d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.266+0000 7fac75200640 1 --2- 172.21.15.97:0/4014024740 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac7015ee30 0x7fac700d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.266+0000 7fac75200640 1 -- 172.21.15.97:0/4014024740 >> v1:172.21.15.135:6789/0 conn(0x7fac7006d600 legacy=0x7fac7015e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.266+0000 7fac75200640 1 -- 172.21.15.97:0/4014024740 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7006ce00 con 0x7fac700b3340 2024-03-06T07:33:40.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.267+0000 7fac75200640 1 --2- 172.21.15.97:0/4014024740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac700b3340 0x7fac700d0070 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fac60009870 tx=0x7fac6002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2ca50e3b603e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.267+0000 7fac6f400640 1 -- 172.21.15.97:0/4014024740 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600317c0 con 0x7fac700b3340 2024-03-06T07:33:40.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.267+0000 7fac6f400640 1 -- 172.21.15.97:0/4014024740 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac60037070 con 0x7fac700b3340 2024-03-06T07:33:40.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.267+0000 7fac6f400640 1 -- 172.21.15.97:0/4014024740 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60031b80 con 0x7fac700b3340 2024-03-06T07:33:40.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 -- 172.21.15.97:0/4014024740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac700b3340 msgr2=0x7fac700d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 --2- 172.21.15.97:0/4014024740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac700b3340 0x7fac700d0070 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fac60009870 tx=0x7fac6002b860 comp rx=0 tx=0).stop 2024-03-06T07:33:40.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 -- 172.21.15.97:0/4014024740 shutdown_connections 2024-03-06T07:33:40.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 --2- 172.21.15.97:0/4014024740 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac7015ee30 0x7fac700d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 --2- 172.21.15.97:0/4014024740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac700b3340 0x7fac700d0070 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 -- 172.21.15.97:0/4014024740 >> 172.21.15.97:0/4014024740 conn(0x7fac7006b360 msgr2=0x7fac7006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 -- 172.21.15.97:0/4014024740 shutdown_connections 2024-03-06T07:33:40.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.268+0000 7fac77c00640 1 -- 172.21.15.97:0/4014024740 wait complete. 2024-03-06T07:33:40.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.270+0000 7fac77c00640 1 Processor -- start 2024-03-06T07:33:40.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.270+0000 7fac77c00640 1 -- start start 2024-03-06T07:33:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac77c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac7006d600 0x7fac701431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac77c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac700b3340 0x7fac70158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac77c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 0x7fac7009d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac77c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac700acd60 con 0x7fac7015ee30 2024-03-06T07:33:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac77c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac700b3720 con 0x7fac7006d600 2024-03-06T07:33:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac77c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac700acee0 con 0x7fac700b3340 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac75c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 0x7fac7009d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac75c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 0x7fac7009d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48930/0 (socket says 172.21.15.97:48930) 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac75c00640 1 -- 172.21.15.97:0/1454484714 learned_addr learned my addr 172.21.15.97:0/1454484714 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac75200640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac7006d600 0x7fac701431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac6fe00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac700b3340 0x7fac70158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac75c00640 1 -- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac700b3340 msgr2=0x7fac70158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.271+0000 7fac75c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac700b3340 0x7fac70158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac75c00640 1 -- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac7006d600 msgr2=0x7fac701431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac75c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac7006d600 0x7fac701431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac75c00640 1 -- 172.21.15.97:0/1454484714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7009ac80 con 0x7fac7015ee30 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac75c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 0x7fac7009d260 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fac68007ca0 tx=0x7fac6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68014420 con 0x7fac7015ee30 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac68019070 con 0x7fac7015ee30 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68014800 con 0x7fac7015ee30 2024-03-06T07:33:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7009b5b0 con 0x7fac7015ee30 2024-03-06T07:33:40.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.272+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac701e8820 con 0x7fac7015ee30 2024-03-06T07:33:40.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.273+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7009b3a0 con 0x7fac7015ee30 2024-03-06T07:33:40.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.275+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac680149a0 con 0x7fac7015ee30 2024-03-06T07:33:40.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.276+0000 7fac6cc00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac5006b4d0 0x7fac5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.276+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fac6808aa40 con 0x7fac7015ee30 2024-03-06T07:33:40.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.276+0000 7fac75200640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac5006b4d0 0x7fac5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.277+0000 7fac75200640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac5006b4d0 0x7fac5006d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fac60036000 tx=0x7fac6003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.280+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac680597c0 con 0x7fac7015ee30 2024-03-06T07:33:40.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.608+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac70068070 con 0x7fac7015ee30 2024-03-06T07:33:40.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.609+0000 7fac6cc00640 1 -- 172.21.15.97:0/1454484714 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fac680590c0 con 0x7fac7015ee30 2024-03-06T07:33:40.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:40.612 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:40.612 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:40.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac5006b4d0 msgr2=0x7fac5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac5006b4d0 0x7fac5006d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fac60036000 tx=0x7fac6003d040 comp rx=0 tx=0).stop 2024-03-06T07:33:40.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 msgr2=0x7fac7009d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 0x7fac7009d260 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fac68007ca0 tx=0x7fac6800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 shutdown_connections 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac5006b4d0 0x7fac5006d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac7015ee30 0x7fac7009d260 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac700b3340 0x7fac70158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 --2- 172.21.15.97:0/1454484714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac7006d600 0x7fac701431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 >> 172.21.15.97:0/1454484714 conn(0x7fac7006b360 msgr2=0x7fac70071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 shutdown_connections 2024-03-06T07:33:40.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:40.612+0000 7fac77c00640 1 -- 172.21.15.97:0/1454484714 wait complete. 2024-03-06T07:33:40.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:40.859 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1086,"msgr_send_messages":1095,"msgr_recv_bytes":600946,"msgr_send_bytes":19061081,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.249258154,"msgr_running_send_time":0.143075874,"msgr_running_recv_time":0.036581595,"msgr_running_fast_dispatch_time":0.056586608,"msgr_send_messages_queue_lat":{"avgcount":1094,"sum":0.317907317,"avgtime":0.000290591},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":103472},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13587,"msgr_send_messages":24269,"msgr_recv_bytes":4654959,"msgr_send_bytes":13627879,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.842794226,"msgr_running_send_time":1.837201329,"msgr_running_recv_time":1.608791017,"msgr_running_fast_dispatch_time":0.056831259,"msgr_send_messages_queue_lat":{"avgcount":24268,"sum":1.966131232,"avgtime":0.000081017},"msgr_handle_ack_lat":{"avgcount":26972,"sum":0.011843986,"avgtime":0.000000439},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":989,"msgr_send_messages":985,"msgr_recv_bytes":364010,"msgr_send_bytes":4600502,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.144301974,"msgr_running_send_time":0.065870065,"msgr_running_recv_time":0.034690642,"msgr_running_fast_dispatch_time":0.037711352,"msgr_send_messages_queue_lat":{"avgcount":985,"sum":0.292105368,"avgtime":0.000296553},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108096,"msgr_send_encrypted_bytes":11040},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":128,"sum":11.249644483,"avgtime":0.087887847}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5388,"sum":2.255421325,"avgtime":0.000418600}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6853,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5082,"handle_client_caps_dirty":3000,"handle_client_cap_release":494,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":84,"num_strays_delayed":84,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3117,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15363,"evex":15002,"evtrm":15002,"ev":366,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34553620,"rdpos":4195674,"jlat":{"avgcount":28,"sum":7.270450210,"avgtime":0.259658936},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":1,"cap+":3202,"cap-":3201,"rss":85980,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":969,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2387107,"buffer_anon_items":802,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1035203,"mds_co_items":18617,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3594,"op_send_bytes":31355056,"op_resend":0,"op_reply":3594,"op_latency":{"avgcount":3594,"sum":19.761663025,"avgtime":0.005498515},"op_inflight":0,"oplen_avg":{"avgcount":3594,"sum":4246},"op":3594,"op_r":64,"op_w":3530,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":36,"osdop_write":56,"osdop_writefull":20,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3142,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":56,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3117,"pq_executed":3117,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15658,"get_sum":4207661,"get_or_fail_fail":0,"get_or_fail_success":15658,"take":0,"take_sum":0,"put":15658,"put_sum":4207661,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3594,"take_sum":31671794,"put":3594,"put_sum":31671794,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3594,"take_sum":3594,"put":3594,"put_sum":3594,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3117,"get_sum":314817,"get_or_fail_fail":0,"get_or_fail_success":3117,"take":0,"take_sum":0,"put":28,"put_sum":314817,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":55707,"max":3758096384,"get_started":0,"get":15363,"get_sum":30357131,"get_or_fail_fail":0,"get_or_fail_success":15363,"take":0,"take_sum":0,"put":28,"put_sum":30301424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:40.860 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 600946, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1086, "msgr_running_fast_dispatch_time": 0.056586608, "msgr_running_recv_time": 0.036581595, "msgr_running_send_time": 0.143075874, "msgr_running_total_time": 0.249258154, "msgr_send_bytes": 19061081, "msgr_send_encrypted_bytes": 103472, "msgr_send_messages": 1095, "msgr_send_messages_queue_lat": { "avgcount": 1094, "avgtime": 0.000290591, "sum": 0.317907317 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26972, "avgtime": 4.39e-07, "sum": 0.011843986 }, "msgr_recv_bytes": 4654959, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13587, "msgr_running_fast_dispatch_time": 0.056831259, "msgr_running_recv_time": 1.608791017, "msgr_running_send_time": 1.837201329, "msgr_running_total_time": 2.842794226, "msgr_send_bytes": 13627879, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24269, "msgr_send_messages_queue_lat": { "avgcount": 24268, "avgtime": 8.1017e-05, "sum": 1.966131232 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 364010, "msgr_recv_encrypted_bytes": 108096, "msgr_recv_messages": 989, "msgr_running_fast_dispatch_time": 0.037711352, "msgr_running_recv_time": 0.034690642, "msgr_running_send_time": 0.065870065, "msgr_running_total_time": 0.144301974, "msgr_send_bytes": 4600502, "msgr_send_encrypted_bytes": 11040, "msgr_send_messages": 985, "msgr_send_messages_queue_lat": { "avgcount": 985, "avgtime": 0.000296553, "sum": 0.292105368 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 128, "avgtime": 0.087887847, "sum": 11.249644483 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5388, "avgtime": 0.0004186, "sum": 2.255421325 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6853, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 494, "handle_client_caps": 5082, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 84, "num_strays_delayed": 84, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3117, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 366, "evadd": 15363, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 28, "avgtime": 0.259658936, "sum": 7.27045021 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34553620 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85980 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 969, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2387107, "buffer_anon_items": 802, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1035203, "mds_co_items": 18617, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 56, "op": 3594, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3594, "avgtime": 0.005498515, "sum": 19.761663025 }, "op_pg": 0, "op_r": 64, "op_reply": 3594, "op_resend": 0, "op_rmw": 0, "op_send": 3594, "op_send_bytes": 31355056, "op_w": 3530, "oplen_avg": { "avgcount": 3594, "sum": 4246 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3142, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 36, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 56, "osdop_writefull": 20, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3117, "pq_executed_ops": 3117, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15658, "get_or_fail_fail": 0, "get_or_fail_success": 15658, "get_started": 0, "get_sum": 4207661, "max": 104857600, "put": 15658, "put_sum": 4207661, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3594, "put_sum": 31671794, "take": 3594, "take_sum": 31671794, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3594, "put_sum": 3594, "take": 3594, "take_sum": 3594, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3117, "get_or_fail_fail": 0, "get_or_fail_success": 3117, "get_started": 0, "get_sum": 314817, "max": 3758096384, "put": 28, "put_sum": 314817, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15363, "get_or_fail_fail": 0, "get_or_fail_success": 15363, "get_started": 0, "get_sum": 30357131, "max": 3758096384, "put": 28, "put_sum": 30301424, "take": 0, "take_sum": 0, "val": 55707, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:40.861 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/84, 3117/3201 2024-03-06T07:33:41.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:42.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.072+0000 7f8db7800640 1 Processor -- start 2024-03-06T07:33:42.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.072+0000 7f8db7800640 1 -- start start 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.074+0000 7f8db7800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8db006c970 0x7f8db0168f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.074+0000 7f8db7800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db00d1a70 con 0x7f8db015e450 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.074+0000 7f8db7800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db015e830 con 0x7f8db00d2490 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.074+0000 7f8db7800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db00d0030 con 0x7f8db006c970 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.074+0000 7f8db4e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8db015e450 0x7f8db00d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58550/0 (socket says 172.21.15.97:58550) 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.074+0000 7f8db4e00640 1 -- 172.21.15.97:0/3649235238 learned_addr learned my addr 172.21.15.97:0/3649235238 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:42.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8db5800640 1 --2- 172.21.15.97:0/3649235238 >> v2:172.21.15.135:3300/0 conn(0x7f8db006c970 0x7f8db0168f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169190085 0 0) 0x7f8db00d1a70 con 0x7f8db015e450 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d98003520 con 0x7f8db015e450 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2645758198 0 0) 0x7f8d98003520 con 0x7f8db015e450 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 >> v2:172.21.15.135:3300/0 conn(0x7f8db006c970 msgr2=0x7f8db0168f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 --2- 172.21.15.97:0/3649235238 >> v2:172.21.15.135:3300/0 conn(0x7f8db006c970 0x7f8db0168f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 >> v1:172.21.15.135:6789/0 conn(0x7f8db00d2490 legacy=0x7f8db006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db006ce40 con 0x7f8db015e450 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8da4001670 con 0x7f8db015e450 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 >> v1:172.21.15.97:6789/0 conn(0x7f8db015e450 legacy=0x7f8db00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d980039b0 0x7f8d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d98004800 0x7f8d98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d980051c0 0x7f8d9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9800c030 con 0x7f8d98004800 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9800c1e0 con 0x7f8d980039b0 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.075+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9800c3e0 con 0x7f8d980051c0 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8dafe00640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d98004800 0x7f8d98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8db4e00640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d980039b0 0x7f8d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8db5800640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d980051c0 0x7f8d9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8dafe00640 1 -- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d980051c0 msgr2=0x7f8d9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8dafe00640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8d980051c0 0x7f8d9800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8dafe00640 1 -- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d980039b0 msgr2=0x7f8d98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8dafe00640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d980039b0 0x7f8d98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8dafe00640 1 -- 172.21.15.97:0/3649235238 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98003520 con 0x7f8d98004800 2024-03-06T07:33:42.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.076+0000 7f8db5800640 1 -- 172.21.15.97:0/3649235238 reap_dead start 2024-03-06T07:33:42.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.077+0000 7f8dafe00640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d98004800 0x7f8d98004c80 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c001720 tx=0x7f8d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc1647d70597a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.077+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c001370 con 0x7f8d98004800 2024-03-06T07:33:42.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.077+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c0052d0 con 0x7f8d98004800 2024-03-06T07:33:42.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.077+0000 7f8daf400640 1 -- 172.21.15.97:0/3649235238 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c003830 con 0x7f8d98004800 2024-03-06T07:33:42.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 -- 172.21.15.97:0/3649235238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d98004800 msgr2=0x7f8d98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d98004800 0x7f8d98004c80 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c001720 tx=0x7f8d98005830 comp rx=0 tx=0).stop 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 -- 172.21.15.97:0/3649235238 shutdown_connections 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 --2- 172.21.15.97:0/3649235238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d98004800 0x7f8d98004c80 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 -- 172.21.15.97:0/3649235238 >> 172.21.15.97:0/3649235238 conn(0x7f8db009aa30 msgr2=0x7f8db009ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 -- 172.21.15.97:0/3649235238 shutdown_connections 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.078+0000 7f8db7800640 1 -- 172.21.15.97:0/3649235238 wait complete. 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.080+0000 7f8db7800640 1 Processor -- start 2024-03-06T07:33:42.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.080+0000 7f8db7800640 1 -- start start 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db7800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d98004800 0x7f8db00c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db7800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db00c69c0 0x7f8db006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db7800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 0x7f8db0070460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db7800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db00d0540 con 0x7f8db0155880 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db7800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db00d0260 con 0x7f8d98004800 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db7800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db006d400 con 0x7f8db00c69c0 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 0x7f8db0070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 0x7f8db0070460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48946/0 (socket says 172.21.15.97:48946) 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.081+0000 7f8db5800640 1 -- 172.21.15.97:0/469022326 learned_addr learned my addr 172.21.15.97:0/469022326 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8dafe00640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db00c69c0 0x7f8db006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db4e00640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d98004800 0x7f8db00c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db5800640 1 -- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db00c69c0 msgr2=0x7f8db006ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db5800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db00c69c0 0x7f8db006ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db5800640 1 -- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d98004800 msgr2=0x7f8db00c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db5800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d98004800 0x7f8db00c6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db5800640 1 -- 172.21.15.97:0/469022326 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db00c8020 con 0x7f8db0155880 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8db5800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 0x7f8db0070460 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009f20 tx=0x7f8da800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8016420 con 0x7f8db0155880 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.082+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da801b070 con 0x7f8db0155880 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.083+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8016800 con 0x7f8db0155880 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.083+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db00c8300 con 0x7f8db0155880 2024-03-06T07:33:42.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.083+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db00b5fd0 con 0x7f8db0155880 2024-03-06T07:33:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.084+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db0068070 con 0x7f8db0155880 2024-03-06T07:33:42.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.085+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8da8005d10 con 0x7f8db0155880 2024-03-06T07:33:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.086+0000 7f8dacc00640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9006b500 0x7f8d9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.086+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8da808a880 con 0x7f8db0155880 2024-03-06T07:33:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.086+0000 7f8db4e00640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9006b500 0x7f8d9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.087+0000 7f8db4e00640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9006b500 0x7f8d9006d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8da4001050 tx=0x7f8da4001110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.091+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8da8024020 con 0x7f8db0155880 2024-03-06T07:33:42.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.327+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8db00ce240 con 0x7f8db0155880 2024-03-06T07:33:42.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.328+0000 7f8dacc00640 1 -- 172.21.15.97:0/469022326 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f8da8059730 con 0x7f8db0155880 2024-03-06T07:33:42.331 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:42.331 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:42.331 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9006b500 msgr2=0x7f8d9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9006b500 0x7f8d9006d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8da4001050 tx=0x7f8da4001110 comp rx=0 tx=0).stop 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 msgr2=0x7f8db0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 0x7f8db0070460 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009f20 tx=0x7f8da800f520 comp rx=0 tx=0).stop 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 shutdown_connections 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9006b500 0x7f8d9006d9c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db0155880 0x7f8db0070460 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db00c69c0 0x7f8db006ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 --2- 172.21.15.97:0/469022326 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d98004800 0x7f8db00c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.331+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 >> 172.21.15.97:0/469022326 conn(0x7f8db009aa30 msgr2=0x7f8db006c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.332+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 shutdown_connections 2024-03-06T07:33:42.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.332+0000 7f8db7800640 1 -- 172.21.15.97:0/469022326 wait complete. 2024-03-06T07:33:42.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:42.582 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1108,"msgr_send_messages":1118,"msgr_recv_bytes":614970,"msgr_send_bytes":19078481,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.253129260,"msgr_running_send_time":0.144815715,"msgr_running_recv_time":0.037542014,"msgr_running_fast_dispatch_time":0.057483999,"msgr_send_messages_queue_lat":{"avgcount":1117,"sum":0.319198555,"avgtime":0.000285764},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":105264},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13632,"msgr_send_messages":24313,"msgr_recv_bytes":4666560,"msgr_send_bytes":13642097,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.849050631,"msgr_running_send_time":1.839640170,"msgr_running_recv_time":1.610272381,"msgr_running_fast_dispatch_time":0.058609440,"msgr_send_messages_queue_lat":{"avgcount":24312,"sum":1.969474971,"avgtime":0.000081008},"msgr_handle_ack_lat":{"avgcount":26975,"sum":0.011844513,"avgtime":0.000000439},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1016,"msgr_send_messages":1012,"msgr_recv_bytes":370886,"msgr_send_bytes":4609602,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.148473128,"msgr_running_send_time":0.068124981,"msgr_running_recv_time":0.035488425,"msgr_running_fast_dispatch_time":0.038570117,"msgr_send_messages_queue_lat":{"avgcount":1012,"sum":0.294088252,"avgtime":0.000290601},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108368,"msgr_send_encrypted_bytes":11664},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":128,"sum":11.249644483,"avgtime":0.087887847}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5535,"sum":2.277417938,"avgtime":0.000411457}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6853,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5082,"handle_client_caps_dirty":3000,"handle_client_cap_release":494,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15447,"evex":15002,"evtrm":15002,"ev":450,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34603936,"rdpos":4195674,"jlat":{"avgcount":28,"sum":7.270450210,"avgtime":0.259658936},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":1,"cap+":3202,"cap-":3201,"rss":85588,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":969,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2387107,"buffer_anon_items":802,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1035203,"mds_co_items":18617,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3685,"op_send_bytes":31363630,"op_resend":0,"op_reply":3685,"op_latency":{"avgcount":3685,"sum":20.097657295,"avgtime":0.005453909},"op_inflight":0,"oplen_avg":{"avgcount":3685,"sum":4337},"op":3685,"op_r":67,"op_w":3618,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":39,"osdop_write":59,"osdop_writefull":21,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":56,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15752,"get_sum":4231650,"get_or_fail_fail":0,"get_or_fail_success":15752,"take":0,"take_sum":0,"put":15752,"put_sum":4231650,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3685,"take_sum":31688852,"put":3685,"put_sum":31688852,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3685,"take_sum":3685,"put":3685,"put_sum":3685,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":31,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":106023,"max":3758096384,"get_started":0,"get":15447,"get_sum":30407447,"get_or_fail_fail":0,"get_or_fail_success":15447,"take":0,"take_sum":0,"put":28,"put_sum":30301424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:42.583 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 614970, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1108, "msgr_running_fast_dispatch_time": 0.057483999, "msgr_running_recv_time": 0.037542014, "msgr_running_send_time": 0.144815715, "msgr_running_total_time": 0.25312926, "msgr_send_bytes": 19078481, "msgr_send_encrypted_bytes": 105264, "msgr_send_messages": 1118, "msgr_send_messages_queue_lat": { "avgcount": 1117, "avgtime": 0.000285764, "sum": 0.319198555 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26975, "avgtime": 4.39e-07, "sum": 0.011844513 }, "msgr_recv_bytes": 4666560, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13632, "msgr_running_fast_dispatch_time": 0.05860944, "msgr_running_recv_time": 1.610272381, "msgr_running_send_time": 1.83964017, "msgr_running_total_time": 2.849050631, "msgr_send_bytes": 13642097, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24313, "msgr_send_messages_queue_lat": { "avgcount": 24312, "avgtime": 8.1008e-05, "sum": 1.969474971 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 370886, "msgr_recv_encrypted_bytes": 108368, "msgr_recv_messages": 1016, "msgr_running_fast_dispatch_time": 0.038570117, "msgr_running_recv_time": 0.035488425, "msgr_running_send_time": 0.068124981, "msgr_running_total_time": 0.148473128, "msgr_send_bytes": 4609602, "msgr_send_encrypted_bytes": 11664, "msgr_send_messages": 1012, "msgr_send_messages_queue_lat": { "avgcount": 1012, "avgtime": 0.000290601, "sum": 0.294088252 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 128, "avgtime": 0.087887847, "sum": 11.249644483 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5535, "avgtime": 0.000411457, "sum": 2.277417938 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6853, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 494, "handle_client_caps": 5082, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 450, "evadd": 15447, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 28, "avgtime": 0.259658936, "sum": 7.27045021 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34603936 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85588 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 969, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2387107, "buffer_anon_items": 802, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1035203, "mds_co_items": 18617, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 56, "op": 3685, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3685, "avgtime": 0.005453909, "sum": 20.097657295 }, "op_pg": 0, "op_r": 67, "op_reply": 3685, "op_resend": 0, "op_rmw": 0, "op_send": 3685, "op_send_bytes": 31363630, "op_w": 3618, "oplen_avg": { "avgcount": 3685, "sum": 4337 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 39, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 59, "osdop_writefull": 21, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15752, "get_or_fail_fail": 0, "get_or_fail_success": 15752, "get_started": 0, "get_sum": 4231650, "max": 104857600, "put": 15752, "put_sum": 4231650, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3685, "put_sum": 31688852, "take": 3685, "take_sum": 31688852, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3685, "put_sum": 3685, "take": 3685, "take_sum": 3685, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 31, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15447, "get_or_fail_fail": 0, "get_or_fail_success": 15447, "get_started": 0, "get_sum": 30407447, "max": 3758096384, "put": 28, "put_sum": 30301424, "take": 0, "take_sum": 0, "val": 106023, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:42.584 INFO:tasks.cephfs.test_strays:Complete purge in 20 seconds 2024-03-06T07:33:42.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.839+0000 7f163b200640 1 Processor -- start 2024-03-06T07:33:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.840+0000 7f163b200640 1 -- start start 2024-03-06T07:33:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.841+0000 7f163b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f163c0d0f80 0x7f163c0d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.841+0000 7f163b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d4980 0x7f163c0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.841+0000 7f163b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163c0b6e50 con 0x7f163c15e410 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.841+0000 7f163b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163c0d4dd0 con 0x7f163c0d0f80 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.841+0000 7f163b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0b7130 con 0x7f163c0d4980 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f163c0d0f80 0x7f163c0d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f163c0d0f80 0x7f163c0d1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48950/0 (socket says 172.21.15.97:48950) 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 -- 172.21.15.97:0/1697739024 learned_addr learned my addr 172.21.15.97:0/1697739024 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1639e00640 1 --1- 172.21.15.97:0/1697739024 >> v1:172.21.15.97:6789/0 conn(0x7f163c15e410 0x7f163c0d0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58562/0 (socket says 172.21.15.97:58562) 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 -- 172.21.15.97:0/1697739024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d4980 msgr2=0x7f163c0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f163a800640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d4980 0x7f163c0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d4980 0x7f163c0b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 -- 172.21.15.97:0/1697739024 >> v1:172.21.15.97:6789/0 conn(0x7f163c15e410 legacy=0x7f163c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1633e00640 1 -- 172.21.15.97:0/1697739024 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c0b7410 con 0x7f163c0d0f80 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f1639400640 1 -- 172.21.15.97:0/1697739024 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22387920 0 0) 0x7f163c0b6e50 con 0x7f163c15e410 2024-03-06T07:33:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.842+0000 7f163a800640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d4980 0x7f163c0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.843+0000 7f1633e00640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d0f80 0x7f163c0d1c40 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1624009870 tx=0x7f162402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328046593d14d979 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f1639400640 1 -- 172.21.15.97:0/1697739024 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624034070 con 0x7f163c0d0f80 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f1639400640 1 -- 172.21.15.97:0/1697739024 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1624030840 con 0x7f163c0d0f80 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f1639400640 1 -- 172.21.15.97:0/1697739024 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624030c10 con 0x7f163c0d0f80 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 -- 172.21.15.97:0/1697739024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d0f80 msgr2=0x7f163c0d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d0f80 0x7f163c0d1c40 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1624009870 tx=0x7f162402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 -- 172.21.15.97:0/1697739024 shutdown_connections 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d4980 0x7f163c0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 --2- 172.21.15.97:0/1697739024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d0f80 0x7f163c0d1c40 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 -- 172.21.15.97:0/1697739024 >> 172.21.15.97:0/1697739024 conn(0x7f163c0d3e60 msgr2=0x7f163c0d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 -- 172.21.15.97:0/1697739024 shutdown_connections 2024-03-06T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.844+0000 7f163b200640 1 -- 172.21.15.97:0/1697739024 wait complete. 2024-03-06T07:33:42.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.846+0000 7f163b200640 1 Processor -- start 2024-03-06T07:33:42.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.846+0000 7f163b200640 1 -- start start 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f163b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d0f80 0x7f163c0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f163b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 0x7f163c157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f163b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c15e410 0x7f163c0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f163b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0b5010 con 0x7f163c0d4980 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f163b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0b79d0 con 0x7f163c15e410 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f163b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0b5190 con 0x7f163c0d0f80 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.847+0000 7f1633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 0x7f163c157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 0x7f163c157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48958/0 (socket says 172.21.15.97:48958) 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 -- 172.21.15.97:0/1977709404 learned_addr learned my addr 172.21.15.97:0/1977709404 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 -- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d0f80 msgr2=0x7f163c0ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1639e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d0f80 0x7f163c0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f163a800640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c15e410 0x7f163c0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d0f80 0x7f163c0ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 -- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c15e410 msgr2=0x7f163c0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c15e410 0x7f163c0c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 -- 172.21.15.97:0/1977709404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c0c7ed0 con 0x7f163c0d4980 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1639e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d0f80 0x7f163c0ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.848+0000 7f1633e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 0x7f163c157980 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f1624000c00 tx=0x7f1624004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.849+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624048070 con 0x7f163c0d4980 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.849+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1624030640 con 0x7f163c0d4980 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.849+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c0c8170 con 0x7f163c0d4980 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.849+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624034040 con 0x7f163c0d4980 2024-03-06T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.849+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c0c8730 con 0x7f163c0d4980 2024-03-06T07:33:42.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.850+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c04f330 con 0x7f163c0d4980 2024-03-06T07:33:42.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.851+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f162404b070 con 0x7f163c0d4980 2024-03-06T07:33:42.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.852+0000 7f1632000640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f161806b4d0 0x7f161806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.852+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f16240ac2c0 con 0x7f163c0d4980 2024-03-06T07:33:42.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.852+0000 7f1639e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f161806b4d0 0x7f161806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.853+0000 7f1639e00640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f161806b4d0 0x7f161806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f162c000f60 tx=0x7f162c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:42.857+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1624030810 con 0x7f163c0d4980 2024-03-06T07:33:43.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.091+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f163c0d2220 con 0x7f163c0d4980 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.092+0000 7f1632000640 1 -- 172.21.15.97:0/1977709404 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f162407b150 con 0x7f163c0d4980 2024-03-06T07:33:43.095 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:43.095 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:43.095 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f161806b4d0 msgr2=0x7f161806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f161806b4d0 0x7f161806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f162c000f60 tx=0x7f162c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 msgr2=0x7f163c157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 0x7f163c157980 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f1624000c00 tx=0x7f1624004730 comp rx=0 tx=0).stop 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 shutdown_connections 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f161806b4d0 0x7f161806d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f163c15e410 0x7f163c0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f163c0d4980 0x7f163c157980 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 --2- 172.21.15.97:0/1977709404 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f163c0d0f80 0x7f163c0ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 >> 172.21.15.97:0/1977709404 conn(0x7f163c0d3e60 msgr2=0x7f163c0b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 shutdown_connections 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.095+0000 7f163b200640 1 -- 172.21.15.97:0/1977709404 wait complete. 2024-03-06T07:33:43.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:43.323 INFO:teuthology.orchestra.run.smithi097.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1108,"msgr_send_messages":1118,"msgr_recv_bytes":614970,"msgr_send_bytes":19078481,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.253129260,"msgr_running_send_time":0.144815715,"msgr_running_recv_time":0.037542014,"msgr_running_fast_dispatch_time":0.057483999,"msgr_send_messages_queue_lat":{"avgcount":1117,"sum":0.319198555,"avgtime":0.000285764},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000235,"avgtime":0.000000117},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":105264},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13633,"msgr_send_messages":24313,"msgr_recv_bytes":4666978,"msgr_send_bytes":13642097,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.849167483,"msgr_running_send_time":1.839688852,"msgr_running_recv_time":1.610301170,"msgr_running_fast_dispatch_time":0.058646414,"msgr_send_messages_queue_lat":{"avgcount":24312,"sum":1.969474971,"avgtime":0.000081008},"msgr_handle_ack_lat":{"avgcount":26976,"sum":0.011844632,"avgtime":0.000000439},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":77808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1016,"msgr_send_messages":1012,"msgr_recv_bytes":370886,"msgr_send_bytes":4609602,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.148473128,"msgr_running_send_time":0.068124981,"msgr_running_recv_time":0.035488425,"msgr_running_fast_dispatch_time":0.038570117,"msgr_send_messages_queue_lat":{"avgcount":1012,"sum":0.294088252,"avgtime":0.000290601},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108368,"msgr_send_encrypted_bytes":11664},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":128,"sum":11.249644483,"avgtime":0.087887847}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5535,"sum":2.277417938,"avgtime":0.000411457}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":792.307080329,"avgtime":0.123604848},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":281,"inodes_top":0,"inodes_bottom":281,"inodes_pin_tail":0,"inodes_pinned":281,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":47,"ceph_cap_op_grant":6853,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5082,"handle_client_caps_dirty":3000,"handle_client_cap_release":494,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15447,"evex":15002,"evtrm":15002,"ev":450,"evexg":0,"evexd":15001,"segadd":16,"segex":17,"segtrm":17,"seg":1,"segmjr":1,"segexg":0,"segexd":16,"expos":34084576,"wrpos":34603936,"rdpos":4195674,"jlat":{"avgcount":28,"sum":7.270450210,"avgtime":0.259658936},"replayed":5},"mds_mem":{"ino":190,"ino+":3214,"ino-":3024,"dir":12,"dir+":13,"dir-":1,"dn":281,"dn+":6412,"dn-":6131,"cap":1,"cap+":3202,"cap-":3201,"rss":85588,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":783.917975772,"avgtime":0.244974367},"req_getattr_latency":{"avgcount":2,"sum":0.189386866,"avgtime":0.094693433},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005707407,"avgtime":0.005707407},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.330313748,"avgtime":0.055052291},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.668007538,"avgtime":0.668007538},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.195688998,"avgtime":0.002248652},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":969,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2387107,"buffer_anon_items":802,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1035203,"mds_co_items":18617,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3685,"op_send_bytes":31363630,"op_resend":0,"op_reply":3685,"op_latency":{"avgcount":3685,"sum":20.097657295,"avgtime":0.005453909},"op_inflight":0,"oplen_avg":{"avgcount":3685,"sum":4337},"op":3685,"op_r":67,"op_w":3618,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":280,"osdop_read":39,"osdop_write":59,"osdop_writefull":21,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":560,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":56,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15753,"get_sum":4231978,"get_or_fail_fail":0,"get_or_fail_success":15753,"take":0,"take_sum":0,"put":15753,"put_sum":4231978,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3685,"take_sum":31688852,"put":3685,"put_sum":31688852,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3685,"take_sum":3685,"put":3685,"put_sum":3685,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":31,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x555dfd006aa0":{"val":106023,"max":3758096384,"get_started":0,"get":15447,"get_sum":30407447,"get_or_fail_fail":0,"get_or_fail_success":15447,"take":0,"take_sum":0,"put":28,"put_sum":30301424,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:43.325 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.17e-07, "sum": 2.35e-07 }, "msgr_recv_bytes": 614970, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1108, "msgr_running_fast_dispatch_time": 0.057483999, "msgr_running_recv_time": 0.037542014, "msgr_running_send_time": 0.144815715, "msgr_running_total_time": 0.25312926, "msgr_send_bytes": 19078481, "msgr_send_encrypted_bytes": 105264, "msgr_send_messages": 1118, "msgr_send_messages_queue_lat": { "avgcount": 1117, "avgtime": 0.000285764, "sum": 0.319198555 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 26976, "avgtime": 4.39e-07, "sum": 0.011844632 }, "msgr_recv_bytes": 4666978, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13633, "msgr_running_fast_dispatch_time": 0.058646414, "msgr_running_recv_time": 1.61030117, "msgr_running_send_time": 1.839688852, "msgr_running_total_time": 2.849167483, "msgr_send_bytes": 13642097, "msgr_send_encrypted_bytes": 77808, "msgr_send_messages": 24313, "msgr_send_messages_queue_lat": { "avgcount": 24312, "avgtime": 8.1008e-05, "sum": 1.969474971 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 370886, "msgr_recv_encrypted_bytes": 108368, "msgr_recv_messages": 1016, "msgr_running_fast_dispatch_time": 0.038570117, "msgr_running_recv_time": 0.035488425, "msgr_running_send_time": 0.068124981, "msgr_running_total_time": 0.148473128, "msgr_send_bytes": 4609602, "msgr_send_encrypted_bytes": 11664, "msgr_send_messages": 1012, "msgr_send_messages_queue_lat": { "avgcount": 1012, "avgtime": 0.000290601, "sum": 0.294088252 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 128, "avgtime": 0.087887847, "sum": 11.249644483 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5535, "avgtime": 0.000411457, "sum": 2.277417938 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6853, "ceph_cap_op_revoke": 47, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 494, "handle_client_caps": 5082, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 281, "inodes_bottom": 281, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 281, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.123604848, "sum": 792.307080329 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 450, "evadd": 15447, "evex": 15002, "evexd": 15001, "evexg": 0, "evlrg": 0, "evtrm": 15002, "expos": 34084576, "jlat": { "avgcount": 28, "avgtime": 0.259658936, "sum": 7.27045021 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 16, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 34603936 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 281, "dn+": 6412, "dn-": 6131, "heap": 248164, "ino": 190, "ino+": 3214, "ino-": 3024, "rss": 85588 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.244974367, "sum": 783.917975772 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.094693433, "sum": 0.189386866 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005707407, "sum": 0.005707407 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.055052291, "sum": 0.330313748 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.668007538, "sum": 0.668007538 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002248652, "sum": 7.195688998 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 969, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2387107, "buffer_anon_items": 802, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1035203, "mds_co_items": 18617, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 56, "op": 3685, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3685, "avgtime": 0.005453909, "sum": 20.097657295 }, "op_pg": 0, "op_r": 67, "op_reply": 3685, "op_resend": 0, "op_rmw": 0, "op_send": 3685, "op_send_bytes": 31363630, "op_w": 3618, "oplen_avg": { "avgcount": 3685, "sum": 4337 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 280, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 39, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 560, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 59, "osdop_writefull": 21, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15753, "get_or_fail_fail": 0, "get_or_fail_success": 15753, "get_started": 0, "get_sum": 4231978, "max": 104857600, "put": 15753, "put_sum": 4231978, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3685, "put_sum": 31688852, "take": 3685, "take_sum": 31688852, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3685, "put_sum": 3685, "take": 3685, "take_sum": 3685, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 31, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x555dfd006aa0": { "get": 15447, "get_or_fail_fail": 0, "get_or_fail_success": 15447, "get_started": 0, "get_sum": 30407447, "max": 3758096384, "put": 28, "put_sum": 30301424, "take": 0, "take_sum": 0, "val": 106023, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:43.325 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:43.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:43.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:43.545 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:43.545 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:33:43.545 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:43.595 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:33:43.595 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:43.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:43.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.889+0000 7f5b21800640 1 Processor -- start 2024-03-06T07:33:43.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.889+0000 7f5b21800640 1 -- start start 2024-03-06T07:33:43.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b21800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c15be90 0x7f5b1c0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b21800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5b1c0d0780 0x7f5b1c15fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b21800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0b5160 con 0x7f5b1c0acd60 2024-03-06T07:33:43.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b21800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1c0b4f30 con 0x7f5b1c0d0780 2024-03-06T07:33:43.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b21800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0af240 con 0x7f5b1c15be90 2024-03-06T07:33:43.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b1b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5b1c0d0780 0x7f5b1c15fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b1b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5b1c0d0780 0x7f5b1c15fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38990/0 (socket says 172.21.15.97:38990) 2024-03-06T07:33:43.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.891+0000 7f5b1b400640 1 -- 172.21.15.97:0/2815534784 learned_addr learned my addr 172.21.15.97:0/2815534784 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:43.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.892+0000 7f5b1aa00640 1 --2- 172.21.15.97:0/2815534784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c15be90 0x7f5b1c0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.892+0000 7f5b1b400640 1 -- 172.21.15.97:0/2815534784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c15be90 msgr2=0x7f5b1c0b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.892+0000 7f5b1b400640 1 --2- 172.21.15.97:0/2815534784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c15be90 0x7f5b1c0b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.892+0000 7f5b1b400640 1 -- 172.21.15.97:0/2815534784 >> v1:172.21.15.135:6789/0 conn(0x7f5b1c0acd60 legacy=0x7f5b1c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.892+0000 7f5b1b400640 1 -- 172.21.15.97:0/2815534784 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c0ae910 con 0x7f5b1c0d0780 2024-03-06T07:33:43.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.893+0000 7f5b1b400640 1 --2- 172.21.15.97:0/2815534784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c15fb70 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009870 tx=0x7f5b1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f98ffb32d148f84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.893+0000 7f5b19600640 1 -- 172.21.15.97:0/2815534784 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14034070 con 0x7f5b1c0d0780 2024-03-06T07:33:43.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.893+0000 7f5b19600640 1 -- 172.21.15.97:0/2815534784 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14030840 con 0x7f5b1c0d0780 2024-03-06T07:33:43.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.893+0000 7f5b19600640 1 -- 172.21.15.97:0/2815534784 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14030c10 con 0x7f5b1c0d0780 2024-03-06T07:33:43.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 -- 172.21.15.97:0/2815534784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c0d0780 msgr2=0x7f5b1c15fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 --2- 172.21.15.97:0/2815534784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c15fb70 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009870 tx=0x7f5b1402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:43.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 -- 172.21.15.97:0/2815534784 shutdown_connections 2024-03-06T07:33:43.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 --2- 172.21.15.97:0/2815534784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c15fb70 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 --2- 172.21.15.97:0/2815534784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c15be90 0x7f5b1c0b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 -- 172.21.15.97:0/2815534784 >> 172.21.15.97:0/2815534784 conn(0x7f5b1c06b360 msgr2=0x7f5b1c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 -- 172.21.15.97:0/2815534784 shutdown_connections 2024-03-06T07:33:43.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.894+0000 7f5b21800640 1 -- 172.21.15.97:0/2815534784 wait complete. 2024-03-06T07:33:43.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.897+0000 7f5b21800640 1 Processor -- start 2024-03-06T07:33:43.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.897+0000 7f5b21800640 1 -- start start 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b21800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c0acd60 0x7f5b1c15b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b21800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b21800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 0x7f5b1c144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b21800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0b2c80 con 0x7f5b1c15be90 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b21800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0aeed0 con 0x7f5b1c0d0780 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b21800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0b2e00 con 0x7f5b1c0acd60 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 0x7f5b1c144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 0x7f5b1c144d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38994/0 (socket says 172.21.15.97:38994) 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1b400640 1 -- 172.21.15.97:0/1242275560 learned_addr learned my addr 172.21.15.97:0/1242275560 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1aa00640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c0acd60 0x7f5b1c15b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1a000640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1b400640 1 -- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c0acd60 msgr2=0x7f5b1c15b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1b400640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c0acd60 0x7f5b1c15b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.899+0000 7f5b1b400640 1 -- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b1c0d0780 msgr2=0x7f5b1c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b1b400640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b1b400640 1 -- 172.21.15.97:0/1242275560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c15a2c0 con 0x7f5b1c15be90 2024-03-06T07:33:43.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b1a000640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:43.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b1b400640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 0x7f5b1c144d00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009870 tx=0x7f5b14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14048070 con 0x7f5b1c15be90 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14030640 con 0x7f5b1c15be90 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c15a5a0 con 0x7f5b1c15be90 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14034040 con 0x7f5b1c15be90 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.900+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c15ab60 con 0x7f5b1c15be90 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.902+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c0adab0 con 0x7f5b1c15be90 2024-03-06T07:33:43.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.902+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b1404b070 con 0x7f5b1c15be90 2024-03-06T07:33:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.904+0000 7f5b12a00640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5af806b4d0 0x7f5af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.904+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5b140ac310 con 0x7f5b1c15be90 2024-03-06T07:33:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.904+0000 7f5b1aa00640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5af806b4d0 0x7f5af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.905+0000 7f5b1aa00640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5af806b4d0 0x7f5af806d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006fd0 tx=0x7f5b0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:43.908+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b1c0adab0 con 0x7f5b1c15be90 2024-03-06T07:33:44.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.105+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b1c068070 con 0x7f5b1c15be90 2024-03-06T07:33:44.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.107+0000 7f5b12a00640 1 -- 172.21.15.97:0/1242275560 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16726 (secure 0 0 0) 0x7f5b1407b000 con 0x7f5b1c15be90 2024-03-06T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":100,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:54.749607+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/1818345324":"2024-03-07T07:32:54.749547+0000","172.21.15.97:6836/1818345324":"2024-03-07T07:32:54.749547+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5af806b4d0 msgr2=0x7f5af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5af806b4d0 0x7f5af806d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006fd0 tx=0x7f5b0c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 msgr2=0x7f5b1c144d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 0x7f5b1c144d00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009870 tx=0x7f5b14004730 comp rx=0 tx=0).stop 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 shutdown_connections 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5af806b4d0 0x7f5af806d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b1c15be90 0x7f5b1c144d00 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b1c0d0780 0x7f5b1c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 --2- 172.21.15.97:0/1242275560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b1c0acd60 0x7f5b1c15b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 >> 172.21.15.97:0/1242275560 conn(0x7f5b1c06b360 msgr2=0x7f5b1c0afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 shutdown_connections 2024-03-06T07:33:44.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.110+0000 7f5b21800640 1 -- 172.21.15.97:0/1242275560 wait complete. 2024-03-06T07:33:44.131 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:40.892840+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:42.065778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:44.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:44.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.335+0000 7fe1a9a00640 1 Processor -- start 2024-03-06T07:33:44.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.335+0000 7fe1a9a00640 1 -- start start 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a9a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe1a40ae630 0x7fe1a415f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a9a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a4155870 con 0x7fe1a4155490 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a9a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a40ad870 con 0x7fe1a40d0780 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a9a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a40aea60 con 0x7fe1a40ae630 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe1a40ae630 0x7fe1a415f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe1a40ae630 0x7fe1a415f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39002/0 (socket says 172.21.15.97:39002) 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a3400640 1 -- 172.21.15.97:0/3539094972 learned_addr learned my addr 172.21.15.97:0/3539094972 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.337+0000 7fe1a2a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe1a4155490 0x7fe1a40d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51478/0 (socket says 172.21.15.97:51478) 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.338+0000 7fe1a3400640 1 -- 172.21.15.97:0/3539094972 >> v1:172.21.15.135:6789/0 conn(0x7fe1a40d0780 legacy=0x7fe1a415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.338+0000 7fe1a3400640 1 -- 172.21.15.97:0/3539094972 >> v1:172.21.15.97:6789/0 conn(0x7fe1a4155490 legacy=0x7fe1a40d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.338+0000 7fe1a3400640 1 -- 172.21.15.97:0/3539094972 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a40adb50 con 0x7fe1a40ae630 2024-03-06T07:33:44.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.339+0000 7fe1a3400640 1 --2- 172.21.15.97:0/3539094972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a415f320 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00d150 tx=0x7fe19c02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e519de9a7c327 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.339+0000 7fe1a2000640 1 -- 172.21.15.97:0/3539094972 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c00b0d0 con 0x7fe1a40ae630 2024-03-06T07:33:44.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.339+0000 7fe1a2000640 1 -- 172.21.15.97:0/3539094972 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c00b270 con 0x7fe1a40ae630 2024-03-06T07:33:44.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.339+0000 7fe1a2000640 1 -- 172.21.15.97:0/3539094972 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c009600 con 0x7fe1a40ae630 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 -- 172.21.15.97:0/3539094972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 msgr2=0x7fe1a415f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/3539094972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a415f320 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00d150 tx=0x7fe19c02e180 comp rx=0 tx=0).stop 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 -- 172.21.15.97:0/3539094972 shutdown_connections 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/3539094972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a415f320 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 -- 172.21.15.97:0/3539094972 >> 172.21.15.97:0/3539094972 conn(0x7fe1a406b360 msgr2=0x7fe1a406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 -- 172.21.15.97:0/3539094972 shutdown_connections 2024-03-06T07:33:44.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.340+0000 7fe1a9a00640 1 -- 172.21.15.97:0/3539094972 wait complete. 2024-03-06T07:33:44.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.342+0000 7fe1a9a00640 1 Processor -- start 2024-03-06T07:33:44.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.342+0000 7fe1a9a00640 1 -- start start 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1a40d0780 0x7fe1a415c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1a4155490 0x7fe1a409b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a9a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a415f980 con 0x7fe1a40ae630 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a9a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40ae110 con 0x7fe1a40d0780 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a9a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4071960 con 0x7fe1a4155490 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a409bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39008/0 (socket says 172.21.15.97:39008) 2024-03-06T07:33:44.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 -- 172.21.15.97:0/612512926 learned_addr learned my addr 172.21.15.97:0/612512926 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:44.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a3400640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1a4155490 0x7fe1a409b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe19be00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1a40d0780 0x7fe1a415c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 -- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1a4155490 msgr2=0x7fe1a409b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1a4155490 0x7fe1a409b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 -- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1a40d0780 msgr2=0x7fe1a415c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1a40d0780 0x7fe1a415c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.343+0000 7fe1a2a00640 1 -- 172.21.15.97:0/612512926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a4159700 con 0x7fe1a40ae630 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.344+0000 7fe1a2a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a409bf20 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fe19400c3c0 tx=0x7fe19400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.344+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1940074e0 con 0x7fe1a40ae630 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.344+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe194007680 con 0x7fe1a40ae630 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.344+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194004a00 con 0x7fe1a40ae630 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.344+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a41599e0 con 0x7fe1a40ae630 2024-03-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.344+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a4159fa0 con 0x7fe1a40ae630 2024-03-06T07:33:44.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.345+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a4068070 con 0x7fe1a40ae630 2024-03-06T07:33:44.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.346+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe19401a020 con 0x7fe1a40ae630 2024-03-06T07:33:44.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.347+0000 7fe19aa00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.347+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe194089a30 con 0x7fe1a40ae630 2024-03-06T07:33:44.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.348+0000 7fe19be00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.348+0000 7fe19be00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe18006b4d0 0x7fe18006d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40710d0 tx=0x7fe18c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.352+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe194010300 con 0x7fe1a40ae630 2024-03-06T07:33:44.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.548+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1a4071740 con 0x7fe1a40ae630 2024-03-06T07:33:44.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.549+0000 7fe19aa00640 1 -- 172.21.15.97:0/612512926 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe194058850 con 0x7fe1a40ae630 2024-03-06T07:33:44.551 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:33:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe18006b4d0 msgr2=0x7fe18006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe18006b4d0 0x7fe18006d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40710d0 tx=0x7fe18c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 msgr2=0x7fe1a409bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a409bf20 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fe19400c3c0 tx=0x7fe19400c880 comp rx=0 tx=0).stop 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 shutdown_connections 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe1a4155490 0x7fe1a409b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe1a40d0780 0x7fe1a415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 --2- 172.21.15.97:0/612512926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1a40ae630 0x7fe1a409bf20 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 >> 172.21.15.97:0/612512926 conn(0x7fe1a406b360 msgr2=0x7fe1a4150790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 shutdown_connections 2024-03-06T07:33:44.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.552+0000 7fe1a9a00640 1 -- 172.21.15.97:0/612512926 wait complete. 2024-03-06T07:33:44.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:44.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.771+0000 7f269f200640 1 Processor -- start 2024-03-06T07:33:44.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.771+0000 7f269f200640 1 -- start start 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f269f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2698070ce0 0x7f26980acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f269f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f26980ad3e0 0x7f269806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f269f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269815a9a0 con 0x7f269806c9f0 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f269f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2698071370 con 0x7f2698070ce0 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f269f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2698071650 con 0x7f26980ad3e0 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f2697e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f269806c9f0 0x7f26980b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51490/0 (socket says 172.21.15.97:51490) 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f2697e00640 1 -- 172.21.15.97:0/2151542706 learned_addr learned my addr 172.21.15.97:0/2151542706 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.774+0000 7f2697400640 1 --2- 172.21.15.97:0/2151542706 >> v2:172.21.15.97:3300/0 conn(0x7f2698070ce0 0x7f26980acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f269d200640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f26980ad3e0 0x7f269806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f2696a00640 1 -- 172.21.15.97:0/2151542706 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755022459 0 0) 0x7f269815a9a0 con 0x7f269806c9f0 2024-03-06T07:33:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f2696a00640 1 -- 172.21.15.97:0/2151542706 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f267c005730 con 0x7f269806c9f0 2024-03-06T07:33:44.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f2697400640 1 -- 172.21.15.97:0/2151542706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f26980ad3e0 msgr2=0x7f269806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f2697400640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f26980ad3e0 0x7f269806cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f2697400640 1 -- 172.21.15.97:0/2151542706 >> v1:172.21.15.97:6789/0 conn(0x7f269806c9f0 legacy=0x7f26980b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f2697400640 1 -- 172.21.15.97:0/2151542706 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2698071930 con 0x7f2698070ce0 2024-03-06T07:33:44.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.775+0000 7f269d200640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f26980ad3e0 0x7f269806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:44.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.777+0000 7f2697400640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980acea0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f26840097a0 tx=0x7f268402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f437aa07178011 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.777+0000 7f2696a00640 1 -- 172.21.15.97:0/2151542706 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684035020 con 0x7f2698070ce0 2024-03-06T07:33:44.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.777+0000 7f2696a00640 1 -- 172.21.15.97:0/2151542706 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2684031810 con 0x7f2698070ce0 2024-03-06T07:33:44.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.777+0000 7f2696a00640 1 -- 172.21.15.97:0/2151542706 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684031bc0 con 0x7f2698070ce0 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 -- 172.21.15.97:0/2151542706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 msgr2=0x7f26980acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980acea0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f26840097a0 tx=0x7f268402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 -- 172.21.15.97:0/2151542706 shutdown_connections 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f26980ad3e0 0x7f269806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 --2- 172.21.15.97:0/2151542706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980acea0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 -- 172.21.15.97:0/2151542706 >> 172.21.15.97:0/2151542706 conn(0x7f269806b360 msgr2=0x7f269806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 -- 172.21.15.97:0/2151542706 shutdown_connections 2024-03-06T07:33:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.778+0000 7f269f200640 1 -- 172.21.15.97:0/2151542706 wait complete. 2024-03-06T07:33:44.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.781+0000 7f269f200640 1 Processor -- start 2024-03-06T07:33:44.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.781+0000 7f269f200640 1 -- start start 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.782+0000 7f269f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f269806c9f0 0x7f26981485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.782+0000 7f269f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.782+0000 7f269f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26980ad3e0 0x7f26980ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.782+0000 7f269f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269806d310 con 0x7f2698070ce0 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.782+0000 7f269f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698071ef0 con 0x7f26980ad3e0 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.782+0000 7f269f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269806d490 con 0x7f269806c9f0 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39030/0 (socket says 172.21.15.97:39030) 2024-03-06T07:33:44.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 -- 172.21.15.97:0/1695402682 learned_addr learned my addr 172.21.15.97:0/1695402682 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:44.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697e00640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f269806c9f0 0x7f26981485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 -- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f269806c9f0 msgr2=0x7f26981485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f269d200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26980ad3e0 0x7f26980ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f269806c9f0 0x7f26981485c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 -- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26980ad3e0 msgr2=0x7f26980ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26980ad3e0 0x7f26980ae680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.783+0000 7f2697400640 1 -- 172.21.15.97:0/1695402682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980ca2f0 con 0x7f2698070ce0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.784+0000 7f2697400640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980c9000 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f26840098c0 tx=0x7f26840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.784+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684041070 con 0x7f2698070ce0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.784+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268403ad00 con 0x7f2698070ce0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.784+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684035040 con 0x7f2698070ce0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.784+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698148b00 con 0x7f2698070ce0 2024-03-06T07:33:44.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.784+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26980c9d30 con 0x7f2698070ce0 2024-03-06T07:33:44.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.785+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2698068070 con 0x7f2698070ce0 2024-03-06T07:33:44.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.786+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f268404c060 con 0x7f2698070ce0 2024-03-06T07:33:44.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.787+0000 7f268fe00640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f267806b4d0 0x7f267806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.787+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f26840abdd0 con 0x7f2698070ce0 2024-03-06T07:33:44.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.787+0000 7f2697e00640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f267806b4d0 0x7f267806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.788+0000 7f2697e00640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f267806b4d0 0x7f267806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2688000f60 tx=0x7f2688009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.792+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f268407abe0 con 0x7f2698070ce0 2024-03-06T07:33:44.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.989+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f26980cc910 con 0x7f2698070ce0 2024-03-06T07:33:44.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.990+0000 7f268fe00640 1 -- 172.21.15.97:0/1695402682 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f268407a4e0 con 0x7f2698070ce0 2024-03-06T07:33:44.992 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f267806b4d0 msgr2=0x7f267806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f267806b4d0 0x7f267806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2688000f60 tx=0x7f2688009040 comp rx=0 tx=0).stop 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 msgr2=0x7f26980c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980c9000 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f26840098c0 tx=0x7f26840046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 shutdown_connections 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f267806b4d0 0x7f267806d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26980ad3e0 0x7f26980ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2698070ce0 0x7f26980c9000 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 --2- 172.21.15.97:0/1695402682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f269806c9f0 0x7f26981485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 >> 172.21.15.97:0/1695402682 conn(0x7f269806b360 msgr2=0x7f26980b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 shutdown_connections 2024-03-06T07:33:44.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:44.993+0000 7f269f200640 1 -- 172.21.15.97:0/1695402682 wait complete. 2024-03-06T07:33:45.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:45.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.213+0000 7ff4fea00640 1 Processor -- start 2024-03-06T07:33:45.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.213+0000 7ff4fea00640 1 -- start start 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4fea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4f815e2d0 0x7ff4f80d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4fea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e6b0 0x7ff4f80d26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4fea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff4f80d2be0 0x7ff4f8161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4fea00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f80cebd0 con 0x7ff4f815e2d0 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4fea00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f80d23e0 con 0x7ff4f80d2be0 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4fea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80d06f0 con 0x7ff4f815e6b0 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4f815e2d0 0x7ff4f80d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4f815e2d0 0x7ff4f80d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39046/0 (socket says 172.21.15.97:39046) 2024-03-06T07:33:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 -- 172.21.15.97:0/792170825 learned_addr learned my addr 172.21.15.97:0/792170825 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:45.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 -- 172.21.15.97:0/792170825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e6b0 msgr2=0x7ff4f80d26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 --2- 172.21.15.97:0/792170825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e6b0 0x7ff4f80d26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 -- 172.21.15.97:0/792170825 >> v2:172.21.15.135:3300/0 conn(0x7ff4f80d2be0 msgr2=0x7ff4f8161fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 --2- 172.21.15.97:0/792170825 >> v2:172.21.15.135:3300/0 conn(0x7ff4f80d2be0 0x7ff4f8161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.215+0000 7ff4f7e00640 1 -- 172.21.15.97:0/792170825 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f80d09d0 con 0x7ff4f815e2d0 2024-03-06T07:33:45.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.217+0000 7ff4f7e00640 1 --2- 172.21.15.97:0/792170825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f815e2d0 0x7ff4f80d1ea0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7ff4f809d830 tx=0x7ff4e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921339810ffe03ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.217+0000 7ff4f7400640 1 -- 172.21.15.97:0/792170825 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8035020 con 0x7ff4f815e2d0 2024-03-06T07:33:45.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4f7400640 1 -- 172.21.15.97:0/792170825 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e8031870 con 0x7ff4f815e2d0 2024-03-06T07:33:45.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4f7400640 1 -- 172.21.15.97:0/792170825 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8031c40 con 0x7ff4f815e2d0 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 -- 172.21.15.97:0/792170825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f815e2d0 msgr2=0x7ff4f80d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 --2- 172.21.15.97:0/792170825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f815e2d0 0x7ff4f80d1ea0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7ff4f809d830 tx=0x7ff4e802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 -- 172.21.15.97:0/792170825 shutdown_connections 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 --2- 172.21.15.97:0/792170825 >> v2:172.21.15.135:3300/0 conn(0x7ff4f80d2be0 0x7ff4f8161fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 --2- 172.21.15.97:0/792170825 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e6b0 0x7ff4f80d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 --2- 172.21.15.97:0/792170825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f815e2d0 0x7ff4f80d1ea0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 -- 172.21.15.97:0/792170825 >> 172.21.15.97:0/792170825 conn(0x7ff4f809d390 msgr2=0x7ff4f80d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 -- 172.21.15.97:0/792170825 shutdown_connections 2024-03-06T07:33:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.218+0000 7ff4fea00640 1 -- 172.21.15.97:0/792170825 wait complete. 2024-03-06T07:33:45.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.220+0000 7ff4fea00640 1 Processor -- start 2024-03-06T07:33:45.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.220+0000 7ff4fea00640 1 -- start start 2024-03-06T07:33:45.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fea00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4f80d2be0 0x7ff4f81526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fea00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e2d0 0x7ff4f80cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 0x7ff4f80cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80d1270 con 0x7ff4f81608f0 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fea00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80d0f90 con 0x7ff4f80d2be0 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fea00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80b3660 con 0x7ff4f815e2d0 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 0x7ff4f80cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 0x7ff4f80cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39050/0 (socket says 172.21.15.97:39050) 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.221+0000 7ff4fca00640 1 -- 172.21.15.97:0/4179904347 learned_addr learned my addr 172.21.15.97:0/4179904347 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4f7e00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4f80d2be0 0x7ff4f81526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4efe00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e2d0 0x7ff4f80cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4fca00640 1 -- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e2d0 msgr2=0x7ff4f80cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4fca00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e2d0 0x7ff4f80cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4fca00640 1 -- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4f80d2be0 msgr2=0x7ff4f81526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4fca00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4f80d2be0 0x7ff4f81526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4fca00640 1 -- 172.21.15.97:0/4179904347 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8160da0 con 0x7ff4f81608f0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4fca00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 0x7ff4f80cbb10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ff4f000b9a0 tx=0x7ff4f000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.222+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0019070 con 0x7ff4f81608f0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.223+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f000cbc0 con 0x7ff4f81608f0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.223+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f8161020 con 0x7ff4f81608f0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.223+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0014440 con 0x7ff4f81608f0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.223+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f81615e0 con 0x7ff4f81608f0 2024-03-06T07:33:45.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.224+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f8161820 con 0x7ff4f81608f0 2024-03-06T07:33:45.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.225+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4f001c070 con 0x7ff4f81608f0 2024-03-06T07:33:45.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.226+0000 7ff4f4c00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4d406b520 0x7ff4d406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.226+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff4f0089f20 con 0x7ff4f81608f0 2024-03-06T07:33:45.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.226+0000 7ff4f7e00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4d406b520 0x7ff4d406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.227+0000 7ff4f7e00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4d406b520 0x7ff4d406d9e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8036000 tx=0x7ff4e803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.231+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4f8161820 con 0x7ff4f81608f0 2024-03-06T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.423+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4f8162e60 con 0x7ff4f81608f0 2024-03-06T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.424+0000 7ff4f4c00640 1 -- 172.21.15.97:0/4179904347 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4f8162e60 con 0x7ff4f81608f0 2024-03-06T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.427+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4d406b520 msgr2=0x7ff4d406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.427+0000 7ff4fea00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4d406b520 0x7ff4d406d9e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8036000 tx=0x7ff4e803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.427+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 msgr2=0x7ff4f80cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.427+0000 7ff4fea00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 0x7ff4f80cbb10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ff4f000b9a0 tx=0x7ff4f000be60 comp rx=0 tx=0).stop 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 shutdown_connections 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4d406b520 0x7ff4d406d9e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f81608f0 0x7ff4f80cbb10 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4f815e2d0 0x7ff4f80cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 --2- 172.21.15.97:0/4179904347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4f80d2be0 0x7ff4f81526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 >> 172.21.15.97:0/4179904347 conn(0x7ff4f809d390 msgr2=0x7ff4f806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 shutdown_connections 2024-03-06T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.428+0000 7ff4fea00640 1 -- 172.21.15.97:0/4179904347 wait complete. 2024-03-06T07:33:45.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.641+0000 7feee0c00640 1 Processor -- start 2024-03-06T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.641+0000 7feee0c00640 1 -- start start 2024-03-06T07:33:45.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.643+0000 7feee0c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feedc0d0cf0 0x7feedc06c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feee0c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc0cf070 con 0x7feedc06cb50 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feee0c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc0cf350 con 0x7feedc1617a0 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feee0c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feedc0b2120 con 0x7feedc0d0cf0 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feeda800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feedc06cb50 0x7feedc0d1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51496/0 (socket says 172.21.15.97:51496) 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feeda800640 1 -- 172.21.15.97:0/1656549194 learned_addr learned my addr 172.21.15.97:0/1656549194 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feed3e00640 1 --2- 172.21.15.97:0/1656549194 >> v2:172.21.15.135:3300/0 conn(0x7feedc0d0cf0 0x7feedc06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428298935 0 0) 0x7feedc0cf070 con 0x7feedc06cb50 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.644+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec0003520 con 0x7feedc06cb50 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873254151 0 0) 0x7feec0003520 con 0x7feedc06cb50 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 >> v2:172.21.15.135:3300/0 conn(0x7feedc0d0cf0 msgr2=0x7feedc06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 --2- 172.21.15.97:0/1656549194 >> v2:172.21.15.135:3300/0 conn(0x7feedc0d0cf0 0x7feedc06c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 >> v1:172.21.15.135:6789/0 conn(0x7feedc1617a0 legacy=0x7feedc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0cf630 con 0x7feedc06cb50 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7feed40019f0 con 0x7feedc06cb50 2024-03-06T07:33:45.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 >> v1:172.21.15.97:6789/0 conn(0x7feedc06cb50 legacy=0x7feedc0d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feec0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feec0004800 0x7feec0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feec0005210 0x7feec000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0b2120 con 0x7feec0003a00 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec000c160 con 0x7feec0004800 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec000c360 con 0x7feec0005210 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.645+0000 7feed9e00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feec0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feed3e00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feec0004800 0x7feec0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feed9e00640 1 -- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feec0005210 msgr2=0x7feec000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feed9e00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feec0005210 0x7feec000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feed9e00640 1 -- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feec0004800 msgr2=0x7feec0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feed9e00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feec0004800 0x7feec0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feed9e00640 1 -- 172.21.15.97:0/1656549194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec0003520 con 0x7feec0003a00 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.646+0000 7feeda800640 1 -- 172.21.15.97:0/1656549194 reap_dead start 2024-03-06T07:33:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.647+0000 7feed9e00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feec0003e20 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7feecc001750 tx=0x7feec0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9089d0884d908e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.647+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc001370 con 0x7feec0003a00 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.647+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc0052d0 con 0x7feec0003a00 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.647+0000 7feed9400640 1 -- 172.21.15.97:0/1656549194 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc003830 con 0x7feec0003a00 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.647+0000 7feee0c00640 1 -- 172.21.15.97:0/1656549194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 msgr2=0x7feec0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.647+0000 7feee0c00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feec0003e20 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7feecc001750 tx=0x7feec0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.648+0000 7feee0c00640 1 -- 172.21.15.97:0/1656549194 shutdown_connections 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.648+0000 7feee0c00640 1 --2- 172.21.15.97:0/1656549194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feec0003e20 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.648+0000 7feee0c00640 1 -- 172.21.15.97:0/1656549194 >> 172.21.15.97:0/1656549194 conn(0x7feedc09aa30 msgr2=0x7feedc09ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.648+0000 7feee0c00640 1 -- 172.21.15.97:0/1656549194 shutdown_connections 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.648+0000 7feee0c00640 1 -- 172.21.15.97:0/1656549194 wait complete. 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.649+0000 7feee0c00640 1 Processor -- start 2024-03-06T07:33:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.650+0000 7feee0c00640 1 -- start start 2024-03-06T07:33:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feee0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feedc1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feee0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feedc14f360 0x7feedc15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feee0c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feedc1f6bd0 0x7feedc159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feee0c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc161d00 con 0x7feec0003a00 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feee0c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc161b80 con 0x7feedc1f6bd0 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feee0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc06c610 con 0x7feedc14f360 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feedc1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feedc1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39076/0 (socket says 172.21.15.97:39076) 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 -- 172.21.15.97:0/1425595656 learned_addr learned my addr 172.21.15.97:0/1425595656 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed3e00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feedc14f360 0x7feedc15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feeda800640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feedc1f6bd0 0x7feedc159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 -- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feedc14f360 msgr2=0x7feedc15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feedc14f360 0x7feedc15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 -- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feedc1f6bd0 msgr2=0x7feedc159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feedc1f6bd0 0x7feedc159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feed9e00640 1 -- 172.21.15.97:0/1425595656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc1f5b30 con 0x7feec0003a00 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.651+0000 7feeda800640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feedc1f6bd0 0x7feedc159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.652+0000 7feed9e00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feedc1590a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7feecc034040 tx=0x7feecc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.652+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc041070 con 0x7feec0003a00 2024-03-06T07:33:45.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.652+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc005480 con 0x7feec0003a00 2024-03-06T07:33:45.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.652+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc007a30 con 0x7feec0003a00 2024-03-06T07:33:45.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.652+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feedc1f5e10 con 0x7feec0003a00 2024-03-06T07:33:45.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.652+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc1f63d0 con 0x7feec0003a00 2024-03-06T07:33:45.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.654+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feedc068070 con 0x7feec0003a00 2024-03-06T07:33:45.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.654+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feecc044070 con 0x7feec0003a00 2024-03-06T07:33:45.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.655+0000 7feed2000640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb806b4d0 0x7feeb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.655+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feecc0abcb0 con 0x7feec0003a00 2024-03-06T07:33:45.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.655+0000 7feed3e00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb806b4d0 0x7feeb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.656+0000 7feed3e00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb806b4d0 0x7feeb806d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7feedc160db0 tx=0x7feec4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.660+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feecc07a510 con 0x7feec0003a00 2024-03-06T07:33:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.869+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feedc0d2d10 con 0x7feec0003a00 2024-03-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.870+0000 7feed2000640 1 -- 172.21.15.97:0/1425595656 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v100) v1 ==== 80+0+122 (secure 0 0 0) 0x7feecc07e5b0 con 0x7feec0003a00 2024-03-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6837/1818345324 2024-03-07T07:32:54.749547+0000 2024-03-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6836/1818345324 2024-03-07T07:32:54.749547+0000 2024-03-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:listed 2 entries 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb806b4d0 msgr2=0x7feeb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb806b4d0 0x7feeb806d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7feedc160db0 tx=0x7feec4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 msgr2=0x7feedc1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feedc1590a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7feecc034040 tx=0x7feecc001120 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 shutdown_connections 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeb806b4d0 0x7feeb806d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feedc1f6bd0 0x7feedc159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feedc14f360 0x7feedc15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 --2- 172.21.15.97:0/1425595656 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec0003a00 0x7feedc1590a0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 >> 172.21.15.97:0/1425595656 conn(0x7feedc09aa30 msgr2=0x7feedc06edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 shutdown_connections 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:45.873+0000 7feee0c00640 1 -- 172.21.15.97:0/1425595656 wait complete. 2024-03-06T07:33:45.892 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:33:45.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:45.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:46.011 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:33:46.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:46.012 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:33:46.057 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:33:46.057 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:46.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:46.098 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:33:46.098 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:46.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:46.135 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:33:46.135 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:46.189 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:46.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.384+0000 7f25a3a00640 1 Processor -- start 2024-03-06T07:33:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.384+0000 7f25a3a00640 1 -- start start 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.385+0000 7f25a3a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f259c15be90 0x7f259c0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.385+0000 7f25a3a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c0acd60 0x7f259c0d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.385+0000 7f25a3a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c0b5160 con 0x7f259c0d05b0 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.385+0000 7f25a3a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0b4f30 con 0x7f259c15be90 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.385+0000 7f25a3a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0af240 con 0x7f259c0acd60 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f259c15be90 0x7f259c0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f259c0d05b0 0x7f259c06c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51506/0 (socket says 172.21.15.97:51506) 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f259c15be90 0x7f259c0b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39078/0 (socket says 172.21.15.97:39078) 2024-03-06T07:33:46.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1a00640 1 -- 172.21.15.97:0/1030948142 learned_addr learned my addr 172.21.15.97:0/1030948142 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f2593e00640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c0acd60 0x7f259c0d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1000640 1 -- 172.21.15.97:0/1030948142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c0acd60 msgr2=0x7f259c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1000640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c0acd60 0x7f259c0d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1000640 1 -- 172.21.15.97:0/1030948142 >> v1:172.21.15.97:6789/0 conn(0x7f259c0d05b0 legacy=0x7f259c06c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f25a1000640 1 -- 172.21.15.97:0/1030948142 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0ae910 con 0x7f259c15be90 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f259be00640 1 -- 172.21.15.97:0/1030948142 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853480360 0 0) 0x7f259c0b5160 con 0x7f259c0d05b0 2024-03-06T07:33:46.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.386+0000 7f2593e00640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c0acd60 0x7f259c0d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:46.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.387+0000 7f25a1000640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c15be90 0x7f259c0b49f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097a0 tx=0x7f258c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505f82001b22ed0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f259be00640 1 -- 172.21.15.97:0/1030948142 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c035020 con 0x7f259c15be90 2024-03-06T07:33:46.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f259be00640 1 -- 172.21.15.97:0/1030948142 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c031810 con 0x7f259c15be90 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f259be00640 1 -- 172.21.15.97:0/1030948142 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c031be0 con 0x7f259c15be90 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 -- 172.21.15.97:0/1030948142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c15be90 msgr2=0x7f259c0b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c15be90 0x7f259c0b49f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097a0 tx=0x7f258c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 -- 172.21.15.97:0/1030948142 shutdown_connections 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c0acd60 0x7f259c0d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 --2- 172.21.15.97:0/1030948142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c15be90 0x7f259c0b49f0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 -- 172.21.15.97:0/1030948142 >> 172.21.15.97:0/1030948142 conn(0x7f259c06b360 msgr2=0x7f259c06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 -- 172.21.15.97:0/1030948142 shutdown_connections 2024-03-06T07:33:46.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.388+0000 7f25a3a00640 1 -- 172.21.15.97:0/1030948142 wait complete. 2024-03-06T07:33:46.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.390+0000 7f25a3a00640 1 Processor -- start 2024-03-06T07:33:46.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.390+0000 7f25a3a00640 1 -- start start 2024-03-06T07:33:46.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a3a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 0x7f259c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a3a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f259c0d05b0 0x7f259c158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a3a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c15be90 0x7f259c159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a3a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0b2c80 con 0x7f259c0acd60 2024-03-06T07:33:46.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a3a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0aeed0 con 0x7f259c0d05b0 2024-03-06T07:33:46.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a3a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0b2e00 con 0x7f259c15be90 2024-03-06T07:33:46.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.391+0000 7f25a1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 0x7f259c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 0x7f259c0b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39086/0 (socket says 172.21.15.97:39086) 2024-03-06T07:33:46.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 -- 172.21.15.97:0/3582434444 learned_addr learned my addr 172.21.15.97:0/3582434444 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 -- 172.21.15.97:0/3582434444 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c15be90 msgr2=0x7f259c159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c15be90 0x7f259c159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 -- 172.21.15.97:0/3582434444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f259c0d05b0 msgr2=0x7f259c158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f259c0d05b0 0x7f259c158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 -- 172.21.15.97:0/3582434444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c158010 con 0x7f259c0acd60 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.392+0000 7f25a1000640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 0x7f259c0b16a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097a0 tx=0x7f258c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.393+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c041070 con 0x7f259c0acd60 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.393+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c03ad00 con 0x7f259c0acd60 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.393+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c1582f0 con 0x7f259c0acd60 2024-03-06T07:33:46.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.393+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c035040 con 0x7f259c0acd60 2024-03-06T07:33:46.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.393+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c1588b0 con 0x7f259c0acd60 2024-03-06T07:33:46.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.394+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c068070 con 0x7f259c0acd60 2024-03-06T07:33:46.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.395+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f258c038020 con 0x7f259c0acd60 2024-03-06T07:33:46.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.396+0000 7f2599600640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f257806b4d0 0x7f257806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.396+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f258c0ac120 con 0x7f259c0acd60 2024-03-06T07:33:46.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.396+0000 7f2593e00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f257806b4d0 0x7f257806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.397+0000 7f2593e00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f257806b4d0 0x7f257806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2584006fd0 tx=0x7f2584008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.400+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f258c07aea0 con 0x7f259c0acd60 2024-03-06T07:33:46.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.646+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259c159850 con 0x7f259c0acd60 2024-03-06T07:33:46.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.647+0000 7f2599600640 1 -- 172.21.15.97:0/3582434444 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f258c07a7a0 con 0x7f259c0acd60 2024-03-06T07:33:46.649 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:46.650 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:46.650 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f257806b4d0 msgr2=0x7f257806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f257806b4d0 0x7f257806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2584006fd0 tx=0x7f2584008040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 msgr2=0x7f259c0b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 0x7f259c0b16a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097a0 tx=0x7f258c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 shutdown_connections 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f257806b4d0 0x7f257806d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f259c15be90 0x7f259c159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f259c0d05b0 0x7f259c158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 --2- 172.21.15.97:0/3582434444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f259c0acd60 0x7f259c0b16a0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.650+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 >> 172.21.15.97:0/3582434444 conn(0x7f259c06b360 msgr2=0x7f259c0c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.651+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 shutdown_connections 2024-03-06T07:33:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.651+0000 7f25a3a00640 1 -- 172.21.15.97:0/3582434444 wait complete. 2024-03-06T07:33:46.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:46.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.885+0000 7fad1b200640 1 Processor -- start 2024-03-06T07:33:46.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.885+0000 7fad1b200640 1 -- start start 2024-03-06T07:33:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad1b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fad140d0780 0x7fad140b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad1b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad140b5160 con 0x7fad140acd60 2024-03-06T07:33:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad1b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad140ae910 con 0x7fad1415be90 2024-03-06T07:33:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad1b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad140b4f30 con 0x7fad140d0780 2024-03-06T07:33:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad19200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fad140acd60 0x7fad1415fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51518/0 (socket says 172.21.15.97:51518) 2024-03-06T07:33:46.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad19200640 1 -- 172.21.15.97:0/3669442403 learned_addr learned my addr 172.21.15.97:0/3669442403 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad13400640 1 --2- 172.21.15.97:0/3669442403 >> v2:172.21.15.135:3300/0 conn(0x7fad140d0780 0x7fad140b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.887+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917215314 0 0) 0x7fad140b5160 con 0x7fad140acd60 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf8003520 con 0x7fad140acd60 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1883238722 0 0) 0x7facf8003520 con 0x7fad140acd60 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 >> v2:172.21.15.135:3300/0 conn(0x7fad140d0780 msgr2=0x7fad140b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 --2- 172.21.15.97:0/3669442403 >> v2:172.21.15.135:3300/0 conn(0x7fad140d0780 0x7fad140b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 >> v1:172.21.15.135:6789/0 conn(0x7fad1415be90 legacy=0x7fad140d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad140aebf0 con 0x7fad140acd60 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fad0c0019f0 con 0x7fad140acd60 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 >> v1:172.21.15.97:6789/0 conn(0x7fad140acd60 legacy=0x7fad1415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7facf8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7facf8004800 0x7facf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facf8005210 0x7facf800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf800c130 con 0x7facf8003a00 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf800c2e0 con 0x7facf8005210 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf800c4e0 con 0x7facf8004800 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.888+0000 7fad13e00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7facf8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad13400640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7facf8004800 0x7facf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad19200640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facf8005210 0x7facf800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad13e00640 1 -- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7facf8004800 msgr2=0x7facf8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad13e00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7facf8004800 0x7facf8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad13e00640 1 -- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facf8005210 msgr2=0x7facf800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad13e00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facf8005210 0x7facf800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad13e00640 1 -- 172.21.15.97:0/3669442403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8003520 con 0x7facf8003a00 2024-03-06T07:33:46.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.889+0000 7fad19200640 1 -- 172.21.15.97:0/3669442403 reap_dead start 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.890+0000 7fad13e00640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7facf8003e20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fad04001720 tx=0x7facf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3a89a4e95d9207 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.890+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04001370 con 0x7facf8003a00 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.890+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040052d0 con 0x7facf8003a00 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.890+0000 7fad12a00640 1 -- 172.21.15.97:0/3669442403 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04003830 con 0x7facf8003a00 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 -- 172.21.15.97:0/3669442403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 msgr2=0x7facf8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7facf8003e20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fad04001720 tx=0x7facf8005830 comp rx=0 tx=0).stop 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 -- 172.21.15.97:0/3669442403 shutdown_connections 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 --2- 172.21.15.97:0/3669442403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7facf8003e20 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 -- 172.21.15.97:0/3669442403 >> 172.21.15.97:0/3669442403 conn(0x7fad1406b360 msgr2=0x7fad1406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 -- 172.21.15.97:0/3669442403 shutdown_connections 2024-03-06T07:33:46.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.891+0000 7fad1b200640 1 -- 172.21.15.97:0/3669442403 wait complete. 2024-03-06T07:33:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.893+0000 7fad1b200640 1 Processor -- start 2024-03-06T07:33:46.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.893+0000 7fad1b200640 1 -- start start 2024-03-06T07:33:46.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad1b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7fad140cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad1b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad14154a80 0x7fad140718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad1b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad141442f0 0x7fad141447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad1b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140b2c80 con 0x7facf8003a00 2024-03-06T07:33:46.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad1b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140af240 con 0x7fad14154a80 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad1b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad140b2e00 con 0x7fad141442f0 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7fad140cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7fad140cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39102/0 (socket says 172.21.15.97:39102) 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 -- 172.21.15.97:0/4044298871 learned_addr learned my addr 172.21.15.97:0/4044298871 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13400640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad14154a80 0x7fad140718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 -- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad141442f0 msgr2=0x7fad141447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad141442f0 0x7fad141447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 -- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad14154a80 msgr2=0x7fad140718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad14154a80 0x7fad140718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.895+0000 7fad13e00640 1 -- 172.21.15.97:0/4044298871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14144ce0 con 0x7facf8003a00 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.896+0000 7fad13e00640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7fad140cc7e0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fad04000c00 tx=0x7fad04001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.896+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04041020 con 0x7facf8003a00 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.896+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad04005480 con 0x7facf8003a00 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.896+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad140cd830 con 0x7facf8003a00 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.896+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04007a30 con 0x7facf8003a00 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.896+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad140ce0d0 con 0x7facf8003a00 2024-03-06T07:33:46.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.897+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad14068070 con 0x7facf8003a00 2024-03-06T07:33:46.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.899+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad04044020 con 0x7facf8003a00 2024-03-06T07:33:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.900+0000 7fad0be00640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7facf406b4d0 0x7facf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.900+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fad040abdf0 con 0x7facf8003a00 2024-03-06T07:33:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.900+0000 7fad13400640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7facf406b4d0 0x7facf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.901+0000 7fad13400640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7facf406b4d0 0x7facf406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad140cebd0 tx=0x7fad00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:46.905+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad0404b030 con 0x7facf8003a00 2024-03-06T07:33:47.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.102+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad140ae410 con 0x7facf8003a00 2024-03-06T07:33:47.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.105+0000 7fad0be00640 1 -- 172.21.15.97:0/4044298871 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16726 (secure 0 0 0) 0x7fad0407ab70 con 0x7facf8003a00 2024-03-06T07:33:47.107 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:47.107 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":100,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:54.749607+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/1818345324":"2024-03-07T07:32:54.749547+0000","172.21.15.97:6836/1818345324":"2024-03-07T07:32:54.749547+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:47.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7facf406b4d0 msgr2=0x7facf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7facf406b4d0 0x7facf406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad140cebd0 tx=0x7fad00008040 comp rx=0 tx=0).stop 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 msgr2=0x7fad140cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7fad140cc7e0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fad04000c00 tx=0x7fad04001120 comp rx=0 tx=0).stop 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 shutdown_connections 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7facf406b4d0 0x7facf406d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fad141442f0 0x7fad141447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad14154a80 0x7fad140718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 --2- 172.21.15.97:0/4044298871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8003a00 0x7fad140cc7e0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 >> 172.21.15.97:0/4044298871 conn(0x7fad1406b360 msgr2=0x7fad1406d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.107+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 shutdown_connections 2024-03-06T07:33:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.108+0000 7fad1b200640 1 -- 172.21.15.97:0/4044298871 wait complete. 2024-03-06T07:33:47.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:40.892840+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:42.065778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:47.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:47.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.335+0000 7f3283e00640 1 Processor -- start 2024-03-06T07:33:47.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.336+0000 7f3283e00640 1 -- start start 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f3283e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3284072980 0x7f32840b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f3283e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32840cc500 0x7f32840cc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f3283e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f32840ccec0 0x7f328406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f3283e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328415b9e0 con 0x7f32840cc500 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f3283e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328415ac20 con 0x7f32840ccec0 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f3283e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328415af00 con 0x7f3284072980 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f327be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32840cc500 0x7f32840cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f327be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32840cc500 0x7f32840cc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39108/0 (socket says 172.21.15.97:39108) 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.337+0000 7f327be00640 1 -- 172.21.15.97:0/462081061 learned_addr learned my addr 172.21.15.97:0/462081061 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f3283400640 1 --2- 172.21.15.97:0/462081061 >> v2:172.21.15.135:3300/0 conn(0x7f32840ccec0 0x7f328406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f3282a00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3284072980 0x7f32840b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f327be00640 1 -- 172.21.15.97:0/462081061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3284072980 msgr2=0x7f32840b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f327be00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3284072980 0x7f32840b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f327be00640 1 -- 172.21.15.97:0/462081061 >> v2:172.21.15.135:3300/0 conn(0x7f32840ccec0 msgr2=0x7f328406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f327be00640 1 --2- 172.21.15.97:0/462081061 >> v2:172.21.15.135:3300/0 conn(0x7f32840ccec0 0x7f328406c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f327be00640 1 -- 172.21.15.97:0/462081061 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328415b1e0 con 0x7f32840cc500 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f3283400640 1 --2- 172.21.15.97:0/462081061 >> v2:172.21.15.135:3300/0 conn(0x7f32840ccec0 0x7f328406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.338+0000 7f3282a00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3284072980 0x7f32840b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.339+0000 7f327be00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32840cc500 0x7f32840cc980 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f32840cd830 tx=0x7f326c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eed9b470726e7bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.339+0000 7f3282000640 1 -- 172.21.15.97:0/462081061 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c038070 con 0x7f32840cc500 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.339+0000 7f3282000640 1 -- 172.21.15.97:0/462081061 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c02f050 con 0x7f32840cc500 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.339+0000 7f3282000640 1 -- 172.21.15.97:0/462081061 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c031970 con 0x7f32840cc500 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 -- 172.21.15.97:0/462081061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32840cc500 msgr2=0x7f32840cc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32840cc500 0x7f32840cc980 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f32840cd830 tx=0x7f326c02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 -- 172.21.15.97:0/462081061 shutdown_connections 2024-03-06T07:33:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 --2- 172.21.15.97:0/462081061 >> v2:172.21.15.135:3300/0 conn(0x7f32840ccec0 0x7f328406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32840cc500 0x7f32840cc980 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 --2- 172.21.15.97:0/462081061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3284072980 0x7f32840b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 -- 172.21.15.97:0/462081061 >> 172.21.15.97:0/462081061 conn(0x7f32840b1770 msgr2=0x7f32840b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 -- 172.21.15.97:0/462081061 shutdown_connections 2024-03-06T07:33:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.340+0000 7f3283e00640 1 -- 172.21.15.97:0/462081061 wait complete. 2024-03-06T07:33:47.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.343+0000 7f3283e00640 1 Processor -- start 2024-03-06T07:33:47.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.343+0000 7f3283e00640 1 -- start start 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.344+0000 7f3283e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 0x7f3284152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.344+0000 7f3283e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32840cc500 0x7f32840c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3283e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32840ccec0 0x7f3284156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3283e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b4a70 con 0x7f3284072980 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3283e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328415a930 con 0x7f32840cc500 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3283e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b4bf0 con 0x7f32840ccec0 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 0x7f3284152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 0x7f3284152e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39118/0 (socket says 172.21.15.97:39118) 2024-03-06T07:33:47.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 -- 172.21.15.97:0/2623199566 learned_addr learned my addr 172.21.15.97:0/2623199566 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:47.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f327be00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32840cc500 0x7f32840c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 -- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32840ccec0 msgr2=0x7f3284156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3283400640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32840ccec0 0x7f3284156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32840ccec0 0x7f3284156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 -- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32840cc500 msgr2=0x7f32840c53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32840cc500 0x7f32840c53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f3282a00640 1 -- 172.21.15.97:0/2623199566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840c5d60 con 0x7f3284072980 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.345+0000 7f327be00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32840cc500 0x7f32840c53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.346+0000 7f3282a00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 0x7f3284152e40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f327400c820 tx=0x7f327400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.346+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274015070 con 0x7f3284072980 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.346+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274007a30 con 0x7f3284072980 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.346+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274007e20 con 0x7f3284072980 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.346+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32840c6040 con 0x7f3284072980 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.346+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32840c6600 con 0x7f3284072980 2024-03-06T07:33:47.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.347+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328404f330 con 0x7f3284072980 2024-03-06T07:33:47.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.348+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3274018070 con 0x7f3284072980 2024-03-06T07:33:47.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.349+0000 7f327aa00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f326006b4d0 0x7f326006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.350+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f327408a190 con 0x7f3284072980 2024-03-06T07:33:47.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.350+0000 7f327be00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f326006b4d0 0x7f326006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.350+0000 7f327be00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f326006b4d0 0x7f326006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f326c035000 tx=0x7f326c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.354+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3274058f10 con 0x7f3284072980 2024-03-06T07:33:47.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.553+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3284072550 con 0x7f3284072980 2024-03-06T07:33:47.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.553+0000 7f327aa00640 1 -- 172.21.15.97:0/2623199566 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3274058810 con 0x7f3284072980 2024-03-06T07:33:47.593 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:33:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f326006b4d0 msgr2=0x7f326006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f326006b4d0 0x7f326006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f326c035000 tx=0x7f326c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 msgr2=0x7f3284152e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 0x7f3284152e40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f327400c820 tx=0x7f327400cce0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 shutdown_connections 2024-03-06T07:33:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f326006b4d0 0x7f326006d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32840ccec0 0x7f3284156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32840cc500 0x7f32840c53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 --2- 172.21.15.97:0/2623199566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3284072980 0x7f3284152e40 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 >> 172.21.15.97:0/2623199566 conn(0x7f32840b1770 msgr2=0x7f32840cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 shutdown_connections 2024-03-06T07:33:47.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.556+0000 7f3283e00640 1 -- 172.21.15.97:0/2623199566 wait complete. 2024-03-06T07:33:47.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:47.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.797+0000 7fdd30a00640 1 Processor -- start 2024-03-06T07:33:47.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.798+0000 7fdd30a00640 1 -- start start 2024-03-06T07:33:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.800+0000 7fdd30a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdd2c15f050 0x7fdd2c0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.800+0000 7fdd30a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c061a10 con 0x7fdd2c0ccc80 2024-03-06T07:33:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.800+0000 7fdd30a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c061cf0 con 0x7fdd2c06af40 2024-03-06T07:33:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.800+0000 7fdd30a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c15f840 con 0x7fdd2c15f050 2024-03-06T07:33:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.800+0000 7fdd2b400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdd2c0ccc80 0x7fdd2c06d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51532/0 (socket says 172.21.15.97:51532) 2024-03-06T07:33:47.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.800+0000 7fdd2b400640 1 -- 172.21.15.97:0/1524506570 learned_addr learned my addr 172.21.15.97:0/1524506570 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:47.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd23e00640 1 --2- 172.21.15.97:0/1524506570 >> v2:172.21.15.135:3300/0 conn(0x7fdd2c15f050 0x7fdd2c0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871256312 0 0) 0x7fdd2c061a10 con 0x7fdd2c0ccc80 2024-03-06T07:33:47.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd10003520 con 0x7fdd2c0ccc80 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1616467996 0 0) 0x7fdd10003520 con 0x7fdd2c0ccc80 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 >> v2:172.21.15.135:3300/0 conn(0x7fdd2c15f050 msgr2=0x7fdd2c0b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd2aa00640 1 --2- 172.21.15.97:0/1524506570 >> v2:172.21.15.135:3300/0 conn(0x7fdd2c15f050 0x7fdd2c0b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.801+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 >> v1:172.21.15.135:6789/0 conn(0x7fdd2c06af40 legacy=0x7fdd2c0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c15f560 con 0x7fdd2c0ccc80 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdd1c001670 con 0x7fdd2c0ccc80 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 >> v1:172.21.15.97:6789/0 conn(0x7fdd2c0ccc80 legacy=0x7fdd2c06d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd100039b0 0x7fdd10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd100052e0 0x7fdd1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c15f840 con 0x7fdd10004800 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1000c1b0 con 0x7fdd100052e0 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1000c3b0 con 0x7fdd100039b0 2024-03-06T07:33:47.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd23e00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2b400640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd100039b0 0x7fdd10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd2be00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd100052e0 0x7fdd1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd23e00640 1 -- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd100039b0 msgr2=0x7fdd10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd23e00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd100039b0 0x7fdd10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd23e00640 1 -- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd100052e0 msgr2=0x7fdd1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd23e00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd100052e0 0x7fdd1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.802+0000 7fdd23e00640 1 -- 172.21.15.97:0/1524506570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd10003520 con 0x7fdd10004800 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.803+0000 7fdd2be00640 1 -- 172.21.15.97:0/1524506570 reap_dead start 2024-03-06T07:33:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.803+0000 7fdd2b400640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd100039b0 0x7fdd10003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.804+0000 7fdd23e00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd10004c80 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c15f490 tx=0x7fdd10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eae3b7b46cf2e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.804+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14039070 con 0x7fdd10004800 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.804+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1402c660 con 0x7fdd10004800 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.804+0000 7fdd2aa00640 1 -- 172.21.15.97:0/1524506570 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14009690 con 0x7fdd10004800 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 -- 172.21.15.97:0/1524506570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 msgr2=0x7fdd10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd10004c80 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c15f490 tx=0x7fdd10005830 comp rx=0 tx=0).stop 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 -- 172.21.15.97:0/1524506570 shutdown_connections 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 --2- 172.21.15.97:0/1524506570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd10004c80 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 -- 172.21.15.97:0/1524506570 >> 172.21.15.97:0/1524506570 conn(0x7fdd2c0725d0 msgr2=0x7fdd2c0729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 -- 172.21.15.97:0/1524506570 shutdown_connections 2024-03-06T07:33:47.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.805+0000 7fdd30a00640 1 -- 172.21.15.97:0/1524506570 wait complete. 2024-03-06T07:33:47.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.808+0000 7fdd30a00640 1 Processor -- start 2024-03-06T07:33:47.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.808+0000 7fdd30a00640 1 -- start start 2024-03-06T07:33:47.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.809+0000 7fdd30a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd2c0cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd30a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd2c0d0890 0x7fdd2c1f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd30a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd2c0cde20 0x7fdd2c1f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd30a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b61b0 con 0x7fdd10004800 2024-03-06T07:33:47.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd30a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b6030 con 0x7fdd2c0cde20 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd30a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c06b3d0 con 0x7fdd2c0d0890 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd2c0cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd2c0cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39140/0 (socket says 172.21.15.97:39140) 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 -- 172.21.15.97:0/2117915189 learned_addr learned my addr 172.21.15.97:0/2117915189 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2be00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd2c0cde20 0x7fdd2c1f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 -- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd2c0d0890 msgr2=0x7fdd2c1f70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd23e00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd2c0d0890 0x7fdd2c1f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd2c0d0890 0x7fdd2c1f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 -- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd2c0cde20 msgr2=0x7fdd2c1f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd2c0cde20 0x7fdd2c1f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2b400640 1 -- 172.21.15.97:0/2117915189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c0ce370 con 0x7fdd10004800 2024-03-06T07:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.810+0000 7fdd2be00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd2c0cde20 0x7fdd2c1f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.811+0000 7fdd2b400640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd2c0cd2d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00d730 tx=0x7fdd1c00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.811+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c019070 con 0x7fdd10004800 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.811+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c0074e0 con 0x7fdd10004800 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.811+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c004a20 con 0x7fdd10004800 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.811+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c0ce650 con 0x7fdd10004800 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.811+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c0cec10 con 0x7fdd10004800 2024-03-06T07:33:47.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.813+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c04f330 con 0x7fdd10004800 2024-03-06T07:33:47.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.813+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd1c01c070 con 0x7fdd10004800 2024-03-06T07:33:47.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.814+0000 7fdd23400640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd0806b4d0 0x7fdd0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.814+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdd1c089c20 con 0x7fdd10004800 2024-03-06T07:33:47.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.814+0000 7fdd23e00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd0806b4d0 0x7fdd0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.815+0000 7fdd23e00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd0806b4d0 0x7fdd0806d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036000 tx=0x7fdd14030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:47.818+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd1c058a40 con 0x7fdd10004800 2024-03-06T07:33:48.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.073+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd2c06e6d0 con 0x7fdd10004800 2024-03-06T07:33:48.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.073+0000 7fdd23400640 1 -- 172.21.15.97:0/2117915189 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd1c058340 con 0x7fdd10004800 2024-03-06T07:33:48.076 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd0806b4d0 msgr2=0x7fdd0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd0806b4d0 0x7fdd0806d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036000 tx=0x7fdd14030000 comp rx=0 tx=0).stop 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 msgr2=0x7fdd2c0cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd2c0cd2d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00d730 tx=0x7fdd1c00dbf0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 shutdown_connections 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd0806b4d0 0x7fdd0806d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd2c0cde20 0x7fdd2c1f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd2c0d0890 0x7fdd2c1f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 --2- 172.21.15.97:0/2117915189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd10004800 0x7fdd2c0cd2d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 >> 172.21.15.97:0/2117915189 conn(0x7fdd2c0725d0 msgr2=0x7fdd2c0ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 shutdown_connections 2024-03-06T07:33:48.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.076+0000 7fdd30a00640 1 -- 172.21.15.97:0/2117915189 wait complete. 2024-03-06T07:33:48.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:48.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.303+0000 7ff812200640 1 Processor -- start 2024-03-06T07:33:48.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.304+0000 7ff812200640 1 -- start start 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff812200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff80c0d2840 0x7ff80c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff812200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0b39d0 con 0x7ff80c15e230 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff812200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0b3cb0 con 0x7ff80c0d2400 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff812200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80c0b3780 con 0x7ff80c0d2840 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff811800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff80c0d2840 0x7ff80c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff811800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff80c0d2840 0x7ff80c0b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39148/0 (socket says 172.21.15.97:39148) 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff811800640 1 -- 172.21.15.97:0/233719363 learned_addr learned my addr 172.21.15.97:0/233719363 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.305+0000 7ff810e00640 1 --1- 172.21.15.97:0/233719363 >> v1:172.21.15.97:6789/0 conn(0x7ff80c15e230 0x7ff80c0d0e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51548/0 (socket says 172.21.15.97:51548) 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.306+0000 7ff811800640 1 -- 172.21.15.97:0/233719363 >> v1:172.21.15.135:6789/0 conn(0x7ff80c0d2400 legacy=0x7ff80c0b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.306+0000 7ff811800640 1 -- 172.21.15.97:0/233719363 >> v1:172.21.15.97:6789/0 conn(0x7ff80c15e230 legacy=0x7ff80c0d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.306+0000 7ff811800640 1 -- 172.21.15.97:0/233719363 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0b3f90 con 0x7ff80c0d2840 2024-03-06T07:33:48.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.307+0000 7ff811800640 1 --2- 172.21.15.97:0/233719363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2840 0x7ff80c0b4ea0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7ff804009e50 tx=0x7ff80402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a4f46156bf4e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.307+0000 7ff80b400640 1 -- 172.21.15.97:0/233719363 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80402cbb0 con 0x7ff80c0d2840 2024-03-06T07:33:48.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.307+0000 7ff80b400640 1 -- 172.21.15.97:0/233719363 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80402cd50 con 0x7ff80c0d2840 2024-03-06T07:33:48.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.307+0000 7ff80b400640 1 -- 172.21.15.97:0/233719363 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804034610 con 0x7ff80c0d2840 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 -- 172.21.15.97:0/233719363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2840 msgr2=0x7ff80c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 --2- 172.21.15.97:0/233719363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2840 0x7ff80c0b4ea0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7ff804009e50 tx=0x7ff80402c010 comp rx=0 tx=0).stop 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 -- 172.21.15.97:0/233719363 shutdown_connections 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 --2- 172.21.15.97:0/233719363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2840 0x7ff80c0b4ea0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 -- 172.21.15.97:0/233719363 >> 172.21.15.97:0/233719363 conn(0x7ff80c0d3e20 msgr2=0x7ff80c0d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 -- 172.21.15.97:0/233719363 shutdown_connections 2024-03-06T07:33:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.308+0000 7ff812200640 1 -- 172.21.15.97:0/233719363 wait complete. 2024-03-06T07:33:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.311+0000 7ff812200640 1 Processor -- start 2024-03-06T07:33:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.311+0000 7ff812200640 1 -- start start 2024-03-06T07:33:48.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.312+0000 7ff812200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 0x7ff80c1f8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.312+0000 7ff812200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff80c0d2840 0x7ff80c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.312+0000 7ff812200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff80c15e230 0x7ff80c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.312+0000 7ff812200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0d2cd0 con 0x7ff80c0d2400 2024-03-06T07:33:48.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.312+0000 7ff812200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c15e610 con 0x7ff80c15e230 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.312+0000 7ff812200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0d1570 con 0x7ff80c0d2840 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 0x7ff80c1f8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 0x7ff80c1f8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39162/0 (socket says 172.21.15.97:39162) 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 -- 172.21.15.97:0/1680159054 learned_addr learned my addr 172.21.15.97:0/1680159054 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff811800640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff80c15e230 0x7ff80c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff80be00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff80c0d2840 0x7ff80c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 -- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff80c0d2840 msgr2=0x7ff80c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff80c0d2840 0x7ff80c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 -- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff80c15e230 msgr2=0x7ff80c15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff80c15e230 0x7ff80c15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 -- 172.21.15.97:0/1680159054 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c1f8580 con 0x7ff80c0d2400 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff811800640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff80c15e230 0x7ff80c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:48.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.313+0000 7ff810e00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 0x7ff80c1f8040 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ff80000c3c0 tx=0x7ff80000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.314+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8000074e0 con 0x7ff80c0d2400 2024-03-06T07:33:48.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.314+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800007680 con 0x7ff80c0d2400 2024-03-06T07:33:48.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.314+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800004a00 con 0x7ff80c0d2400 2024-03-06T07:33:48.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.314+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c0b85d0 con 0x7ff80c0d2400 2024-03-06T07:33:48.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.314+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c0b8b90 con 0x7ff80c0d2400 2024-03-06T07:33:48.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.315+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c04f330 con 0x7ff80c0d2400 2024-03-06T07:33:48.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.316+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff80001a020 con 0x7ff80c0d2400 2024-03-06T07:33:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.317+0000 7ff808c00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff7ec06b4d0 0x7ff7ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.317+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff800089a80 con 0x7ff80c0d2400 2024-03-06T07:33:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.317+0000 7ff80be00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff7ec06b4d0 0x7ff7ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.318+0000 7ff80be00640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff7ec06b4d0 0x7ff7ec06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d4d00 tx=0x7ff7f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.321+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff80008be20 con 0x7ff80c0d2400 2024-03-06T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.516+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff80c0cf610 con 0x7ff80c0d2400 2024-03-06T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.517+0000 7ff808c00640 1 -- 172.21.15.97:0/1680159054 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff800058800 con 0x7ff80c0d2400 2024-03-06T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff7ec06b4d0 msgr2=0x7ff7ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff7ec06b4d0 0x7ff7ec06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d4d00 tx=0x7ff7f8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 msgr2=0x7ff80c1f8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 0x7ff80c1f8040 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ff80000c3c0 tx=0x7ff80000c880 comp rx=0 tx=0).stop 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 shutdown_connections 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff7ec06b4d0 0x7ff7ec06d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff80c15e230 0x7ff80c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff80c0d2840 0x7ff80c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 --2- 172.21.15.97:0/1680159054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff80c0d2400 0x7ff80c1f8040 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 >> 172.21.15.97:0/1680159054 conn(0x7ff80c0d3e20 msgr2=0x7ff80c06cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 shutdown_connections 2024-03-06T07:33:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.520+0000 7ff812200640 1 -- 172.21.15.97:0/1680159054 wait complete. 2024-03-06T07:33:48.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:48.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.730+0000 7f6b5fa00640 1 Processor -- start 2024-03-06T07:33:48.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.731+0000 7f6b5fa00640 1 -- start start 2024-03-06T07:33:48.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.732+0000 7f6b5fa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b5815be90 0x7f6b580b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.732+0000 7f6b5fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b5815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.732+0000 7f6b5fa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580b5160 con 0x7f6b580acd60 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.732+0000 7f6b5fa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b580b4f30 con 0x7f6b5815be90 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.732+0000 7f6b5fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580af240 con 0x7f6b580d0780 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.732+0000 7f6b5d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b5815be90 0x7f6b580b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b5815be90 0x7f6b580b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39166/0 (socket says 172.21.15.97:39166) 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5d000640 1 -- 172.21.15.97:0/4137992446 learned_addr learned my addr 172.21.15.97:0/4137992446 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b4fe00640 1 --1- 172.21.15.97:0/4137992446 >> v1:172.21.15.97:6789/0 conn(0x7f6b580acd60 0x7f6b580d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51550/0 (socket says 172.21.15.97:51550) 2024-03-06T07:33:48.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5da00640 1 --2- 172.21.15.97:0/4137992446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b5815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5d000640 1 -- 172.21.15.97:0/4137992446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 msgr2=0x7f6b5815fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5d000640 1 --2- 172.21.15.97:0/4137992446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b5815fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5d000640 1 -- 172.21.15.97:0/4137992446 >> v1:172.21.15.97:6789/0 conn(0x7f6b580acd60 legacy=0x7f6b580d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b57e00640 1 -- 172.21.15.97:0/4137992446 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593570208 0 0) 0x7f6b580b5160 con 0x7f6b580acd60 2024-03-06T07:33:48.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.733+0000 7f6b5d000640 1 -- 172.21.15.97:0/4137992446 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580ae910 con 0x7f6b5815be90 2024-03-06T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.734+0000 7f6b5d000640 1 --2- 172.21.15.97:0/4137992446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b580b49f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009870 tx=0x7f6b4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f64a892555c762c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.734+0000 7f6b57e00640 1 -- 172.21.15.97:0/4137992446 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48034070 con 0x7f6b5815be90 2024-03-06T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.734+0000 7f6b57e00640 1 -- 172.21.15.97:0/4137992446 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48030840 con 0x7f6b5815be90 2024-03-06T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.734+0000 7f6b57e00640 1 -- 172.21.15.97:0/4137992446 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48030c10 con 0x7f6b5815be90 2024-03-06T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 -- 172.21.15.97:0/4137992446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 msgr2=0x7f6b580b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/4137992446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b580b49f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009870 tx=0x7f6b4802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 -- 172.21.15.97:0/4137992446 shutdown_connections 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/4137992446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b5815fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/4137992446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b580b49f0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 -- 172.21.15.97:0/4137992446 >> 172.21.15.97:0/4137992446 conn(0x7f6b5806b360 msgr2=0x7f6b5806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 -- 172.21.15.97:0/4137992446 shutdown_connections 2024-03-06T07:33:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.735+0000 7f6b5fa00640 1 -- 172.21.15.97:0/4137992446 wait complete. 2024-03-06T07:33:48.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.737+0000 7f6b5fa00640 1 Processor -- start 2024-03-06T07:33:48.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.737+0000 7f6b5fa00640 1 -- start start 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5fa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580acd60 0x7f6b580c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b58156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5fa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b5814a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5fa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580b2c80 con 0x7f6b5815be90 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5fa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580aeed0 con 0x7f6b580acd60 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580b2e00 con 0x7f6b580d0780 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b5814a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b5814a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39176/0 (socket says 172.21.15.97:39176) 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5da00640 1 -- 172.21.15.97:0/715498109 learned_addr learned my addr 172.21.15.97:0/715498109 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b4fe00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b58156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.738+0000 7f6b5d000640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580acd60 0x7f6b580c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5da00640 1 -- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 msgr2=0x7f6b58156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5da00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b58156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5da00640 1 -- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580acd60 msgr2=0x7f6b580c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5da00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580acd60 0x7f6b580c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5da00640 1 -- 172.21.15.97:0/715498109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580c9540 con 0x7f6b5815be90 2024-03-06T07:33:48.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5da00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b5814a660 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000cd50 tx=0x7f6b5000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50003b90 con 0x7f6b5815be90 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b50003d30 con 0x7f6b5815be90 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50004240 con 0x7f6b5815be90 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b58156bd0 con 0x7f6b5815be90 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.739+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5814e8e0 con 0x7f6b5815be90 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.740+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b58068070 con 0x7f6b5815be90 2024-03-06T07:33:48.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.741+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b500043e0 con 0x7f6b5815be90 2024-03-06T07:33:48.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.742+0000 7f6b55600640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3406b4d0 0x7f6b3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.742+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6b5008ac40 con 0x7f6b5815be90 2024-03-06T07:33:48.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.743+0000 7f6b5d000640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3406b4d0 0x7f6b3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.743+0000 7f6b5d000640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3406b4d0 0x7f6b3406d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6b48036000 tx=0x7f6b4803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.747+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b50059a50 con 0x7f6b5815be90 2024-03-06T07:33:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.979+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b58071700 con 0x7f6b5815be90 2024-03-06T07:33:48.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.980+0000 7f6b55600640 1 -- 172.21.15.97:0/715498109 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f6b50059350 con 0x7f6b5815be90 2024-03-06T07:33:48.982 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:48.983 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:48.983 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3406b4d0 msgr2=0x7f6b3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3406b4d0 0x7f6b3406d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6b48036000 tx=0x7f6b4803b040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 msgr2=0x7f6b5814a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b5814a660 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000cd50 tx=0x7f6b5000a660 comp rx=0 tx=0).stop 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 shutdown_connections 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b3406b4d0 0x7f6b3406d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b5815be90 0x7f6b5814a660 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b580d0780 0x7f6b58156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 --2- 172.21.15.97:0/715498109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b580acd60 0x7f6b580c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 >> 172.21.15.97:0/715498109 conn(0x7f6b5806b360 msgr2=0x7f6b58070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 shutdown_connections 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:48.983+0000 7f6b5fa00640 1 -- 172.21.15.97:0/715498109 wait complete. 2024-03-06T07:33:49.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:49.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.204+0000 7f2c89000640 1 Processor -- start 2024-03-06T07:33:49.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.204+0000 7f2c89000640 1 -- start start 2024-03-06T07:33:49.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c89000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c8415e530 0x7f2c840b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c89000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2c840d2960 0x7f2c840ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c89000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c840cec00 0x7f2c8406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c89000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c840b3420 con 0x7f2c840cec00 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c89000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c840b2c00 con 0x7f2c840d2960 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c89000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840d0d30 con 0x7f2c8415e530 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c82800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c840cec00 0x7f2c8406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c82800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c840cec00 0x7f2c8406c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39186/0 (socket says 172.21.15.97:39186) 2024-03-06T07:33:49.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c82800640 1 -- 172.21.15.97:0/2299049975 learned_addr learned my addr 172.21.15.97:0/2299049975 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c81400640 1 --2- 172.21.15.97:0/2299049975 >> v2:172.21.15.135:3300/0 conn(0x7f2c840d2960 0x7f2c840ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.206+0000 7f2c81e00640 1 --2- 172.21.15.97:0/2299049975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c8415e530 0x7f2c840b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.207+0000 7f2c82800640 1 -- 172.21.15.97:0/2299049975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c8415e530 msgr2=0x7f2c840b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.207+0000 7f2c82800640 1 --2- 172.21.15.97:0/2299049975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c8415e530 0x7f2c840b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.207+0000 7f2c82800640 1 -- 172.21.15.97:0/2299049975 >> v2:172.21.15.135:3300/0 conn(0x7f2c840d2960 msgr2=0x7f2c840ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.207+0000 7f2c82800640 1 --2- 172.21.15.97:0/2299049975 >> v2:172.21.15.135:3300/0 conn(0x7f2c840d2960 0x7f2c840ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.207+0000 7f2c82800640 1 -- 172.21.15.97:0/2299049975 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840d1010 con 0x7f2c840cec00 2024-03-06T07:33:49.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.207+0000 7f2c81400640 1 --2- 172.21.15.97:0/2299049975 >> v2:172.21.15.135:3300/0 conn(0x7f2c840d2960 0x7f2c840ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:49.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.208+0000 7f2c82800640 1 --2- 172.21.15.97:0/2299049975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c840cec00 0x7f2c8406c2f0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f2c84072450 tx=0x7f2c7c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613fbdd13a907d69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.209+0000 7f2c80a00640 1 -- 172.21.15.97:0/2299049975 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c02c9e0 con 0x7f2c840cec00 2024-03-06T07:33:49.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.209+0000 7f2c80a00640 1 -- 172.21.15.97:0/2299049975 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c02cb80 con 0x7f2c840cec00 2024-03-06T07:33:49.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.209+0000 7f2c80a00640 1 -- 172.21.15.97:0/2299049975 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c0343f0 con 0x7f2c840cec00 2024-03-06T07:33:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.209+0000 7f2c89000640 1 -- 172.21.15.97:0/2299049975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c840cec00 msgr2=0x7f2c8406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.209+0000 7f2c89000640 1 --2- 172.21.15.97:0/2299049975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c840cec00 0x7f2c8406c2f0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f2c84072450 tx=0x7f2c7c02be40 comp rx=0 tx=0).stop 2024-03-06T07:33:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 -- 172.21.15.97:0/2299049975 shutdown_connections 2024-03-06T07:33:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 --2- 172.21.15.97:0/2299049975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c840cec00 0x7f2c8406c2f0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 --2- 172.21.15.97:0/2299049975 >> v2:172.21.15.135:3300/0 conn(0x7f2c840d2960 0x7f2c840ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 --2- 172.21.15.97:0/2299049975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c8415e530 0x7f2c840b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 -- 172.21.15.97:0/2299049975 >> 172.21.15.97:0/2299049975 conn(0x7f2c840d1e40 msgr2=0x7f2c840d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 -- 172.21.15.97:0/2299049975 shutdown_connections 2024-03-06T07:33:49.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.210+0000 7f2c89000640 1 -- 172.21.15.97:0/2299049975 wait complete. 2024-03-06T07:33:49.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.211+0000 7f2c89000640 1 Processor -- start 2024-03-06T07:33:49.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.211+0000 7f2c89000640 1 -- start start 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.212+0000 7f2c89000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c840cec00 0x7f2c84158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.212+0000 7f2c89000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c840d2960 0x7f2c840b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.212+0000 7f2c89000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 0x7f2c840ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.212+0000 7f2c89000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8415ea90 con 0x7f2c8415e530 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.212+0000 7f2c89000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8415e910 con 0x7f2c840cec00 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.212+0000 7f2c89000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8406c830 con 0x7f2c840d2960 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 0x7f2c840ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 0x7f2c840ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39194/0 (socket says 172.21.15.97:39194) 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 -- 172.21.15.97:0/3941796693 learned_addr learned my addr 172.21.15.97:0/3941796693 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c81e00640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c840cec00 0x7f2c84158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c81400640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c840d2960 0x7f2c840b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 -- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c840d2960 msgr2=0x7f2c840b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c840d2960 0x7f2c840b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 -- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c840cec00 msgr2=0x7f2c84158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c840cec00 0x7f2c84158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 -- 172.21.15.97:0/3941796693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840b4c90 con 0x7f2c8415e530 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c81400640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c840d2960 0x7f2c840b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.213+0000 7f2c82800640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 0x7f2c840ca350 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c036040 tx=0x7f2c7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.214+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c043020 con 0x7f2c8415e530 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.214+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c03e450 con 0x7f2c8415e530 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.214+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c009510 con 0x7f2c8415e530 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.214+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c840b7a30 con 0x7f2c8415e530 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.214+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c840b5210 con 0x7f2c8415e530 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.215+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c84068ea0 con 0x7f2c8415e530 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.216+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c7c046020 con 0x7f2c8415e530 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.217+0000 7f2c7a000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c6406b4d0 0x7f2c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.217+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2c7c0acb10 con 0x7f2c8415e530 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.217+0000 7f2c81e00640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c6406b4d0 0x7f2c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.218+0000 7f2c81e00640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c6406b4d0 0x7f2c6406d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2c74006fd0 tx=0x7f2c74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.222+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c7c04d030 con 0x7f2c8415e530 2024-03-06T07:33:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.420+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c840cd560 con 0x7f2c8415e530 2024-03-06T07:33:49.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.421+0000 7f2c7a000640 1 -- 172.21.15.97:0/3941796693 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16726 (secure 0 0 0) 0x7f2c7c07b920 con 0x7f2c8415e530 2024-03-06T07:33:49.424 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:49.424 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":100,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:54.749607+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/1818345324":"2024-03-07T07:32:54.749547+0000","172.21.15.97:6836/1818345324":"2024-03-07T07:32:54.749547+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:49.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c6406b4d0 msgr2=0x7f2c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c6406b4d0 0x7f2c6406d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2c74006fd0 tx=0x7f2c74008040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 msgr2=0x7f2c840ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 0x7f2c840ca350 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c036040 tx=0x7f2c7c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 shutdown_connections 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c6406b4d0 0x7f2c6406d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c8415e530 0x7f2c840ca350 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c840d2960 0x7f2c840b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 --2- 172.21.15.97:0/3941796693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c840cec00 0x7f2c84158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 >> 172.21.15.97:0/3941796693 conn(0x7f2c840d1e40 msgr2=0x7f2c840d00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.424+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 shutdown_connections 2024-03-06T07:33:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.425+0000 7f2c89000640 1 -- 172.21.15.97:0/3941796693 wait complete. 2024-03-06T07:33:49.445 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:49.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:49.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.639+0000 7f1497200640 1 Processor -- start 2024-03-06T07:33:49.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.640+0000 7f1497200640 1 -- start start 2024-03-06T07:33:49.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.641+0000 7f1497200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 0x7f14980d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.641+0000 7f1497200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14980b39d0 con 0x7f14980d13a0 2024-03-06T07:33:49.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.641+0000 7f1497200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14980b3cb0 con 0x7f14980d2b10 2024-03-06T07:33:49.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.641+0000 7f1497200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14980b3780 con 0x7f149815e230 2024-03-06T07:33:49.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.641+0000 7f148d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f14980d13a0 0x7f14980d2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51566/0 (socket says 172.21.15.97:51566) 2024-03-06T07:33:49.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f148d400640 1 -- 172.21.15.97:0/2465259398 learned_addr learned my addr 172.21.15.97:0/2465259398 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f1495e00640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 0x7f14980d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065524429 0 0) 0x7f14980b39d0 con 0x7f14980d13a0 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c003520 con 0x7f14980d13a0 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291382137 0 0) 0x7f147c003520 con 0x7f14980d13a0 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 msgr2=0x7f14980d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.642+0000 7f1495400640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 0x7f14980d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 >> v1:172.21.15.135:6789/0 conn(0x7f14980d2b10 legacy=0x7f14980b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980b3f90 con 0x7f14980d13a0 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1480001670 con 0x7f14980d13a0 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 >> v1:172.21.15.97:6789/0 conn(0x7f14980d13a0 legacy=0x7f14980d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 0x7f147c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c003f40 0x7f147c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c004d40 0x7f147c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980b3780 con 0x7f147c004d40 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c00ba70 con 0x7f147c003f40 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c00bc40 con 0x7f149815e230 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1496800640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c004d40 0x7f147c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f148d400640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c003f40 0x7f147c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1495e00640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 0x7f147c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1496800640 1 -- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 msgr2=0x7f147c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1496800640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f149815e230 0x7f147c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1496800640 1 -- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c003f40 msgr2=0x7f147c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1496800640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c003f40 0x7f147c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.643+0000 7f1496800640 1 -- 172.21.15.97:0/2465259398 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c003520 con 0x7f147c004d40 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1496800640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c004d40 0x7f147c0051c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f1490001890 tx=0x7f147c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fceb5af58b9a519 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490001430 con 0x7f147c004d40 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1490033620 con 0x7f147c004d40 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1495400640 1 -- 172.21.15.97:0/2465259398 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490033a10 con 0x7f147c004d40 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1497200640 1 -- 172.21.15.97:0/2465259398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c004d40 msgr2=0x7f147c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1497200640 1 --2- 172.21.15.97:0/2465259398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c004d40 0x7f147c0051c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f1490001890 tx=0x7f147c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.645+0000 7f1496800640 1 -- 172.21.15.97:0/2465259398 reap_dead start 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.646+0000 7f1497200640 1 -- 172.21.15.97:0/2465259398 shutdown_connections 2024-03-06T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.646+0000 7f1497200640 1 -- 172.21.15.97:0/2465259398 >> 172.21.15.97:0/2465259398 conn(0x7f14980d3e20 msgr2=0x7f14980d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.646+0000 7f1497200640 1 -- 172.21.15.97:0/2465259398 shutdown_connections 2024-03-06T07:33:49.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.646+0000 7f1497200640 1 -- 172.21.15.97:0/2465259398 wait complete. 2024-03-06T07:33:49.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.647+0000 7f1497200640 1 Processor -- start 2024-03-06T07:33:49.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.648+0000 7f1497200640 1 -- start start 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.648+0000 7f1497200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c004d40 0x7f14980b8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.648+0000 7f1497200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14980b7a60 0x7f14980b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1497200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 0x7f1498160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1497200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980d1d90 con 0x7f14980b8420 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1497200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149815e610 con 0x7f14980b7a60 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1497200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14980d1f10 con 0x7f147c004d40 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 0x7f1498160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 0x7f1498160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39216/0 (socket says 172.21.15.97:39216) 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 -- 172.21.15.97:0/2976506452 learned_addr learned my addr 172.21.15.97:0/2976506452 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1495e00640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c004d40 0x7f14980b8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f148d400640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14980b7a60 0x7f14980b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 -- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c004d40 msgr2=0x7f14980b8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c004d40 0x7f14980b8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 -- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14980b7a60 msgr2=0x7f14980b7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14980b7a60 0x7f14980b7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1496800640 1 -- 172.21.15.97:0/2976506452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1498153690 con 0x7f14980b8420 2024-03-06T07:33:49.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.649+0000 7f1495e00640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c004d40 0x7f14980b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.650+0000 7f1496800640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 0x7f1498160520 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1490001980 tx=0x7f1490001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.650+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490043070 con 0x7f14980b8420 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.650+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1490007030 con 0x7f14980b8420 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.650+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490001430 con 0x7f14980b8420 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.650+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14980b88a0 con 0x7f14980b8420 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.650+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14980cb560 con 0x7f14980b8420 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.651+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149804f330 con 0x7f14980b8420 2024-03-06T07:33:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.652+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1490046070 con 0x7f14980b8420 2024-03-06T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.653+0000 7f148ea00640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f147406b500 0x7f147406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.654+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f14900aaba0 con 0x7f14980b8420 2024-03-06T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.654+0000 7f1495e00640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f147406b500 0x7f147406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.654+0000 7f1495e00640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f147406b500 0x7f147406d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1488007440 tx=0x7f148800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.658+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14900792d0 con 0x7f14980b8420 2024-03-06T07:33:49.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.897+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f149806d310 con 0x7f14980b8420 2024-03-06T07:33:49.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.898+0000 7f148ea00640 1 -- 172.21.15.97:0/2976506452 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f149007d370 con 0x7f14980b8420 2024-03-06T07:33:49.900 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:49.901 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:49.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f147406b500 msgr2=0x7f147406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f147406b500 0x7f147406d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1488007440 tx=0x7f148800b040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 msgr2=0x7f1498160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 0x7f1498160520 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1490001980 tx=0x7f1490001230 comp rx=0 tx=0).stop 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 shutdown_connections 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f147406b500 0x7f147406d9c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14980b8420 0x7f1498160520 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14980b7a60 0x7f14980b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 --2- 172.21.15.97:0/2976506452 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c004d40 0x7f14980b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 >> 172.21.15.97:0/2976506452 conn(0x7f14980d3e20 msgr2=0x7f14980b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 shutdown_connections 2024-03-06T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:49.901+0000 7f1497200640 1 -- 172.21.15.97:0/2976506452 wait complete. 2024-03-06T07:33:49.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:50.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.123+0000 7fe285c00640 1 Processor -- start 2024-03-06T07:33:50.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.123+0000 7fe285c00640 1 -- start start 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.124+0000 7fe285c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe280155490 0x7fe2800ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.124+0000 7fe285c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe2800d0070 0x7fe2800d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe285c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe280155870 con 0x7fe28015ec10 2024-03-06T07:33:50.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe285c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2800ad640 con 0x7fe280155490 2024-03-06T07:33:50.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe285c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28015f980 con 0x7fe2800d0070 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe276000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe2800d0070 0x7fe2800d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe27ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe280155490 0x7fe2800ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe276000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe2800d0070 0x7fe2800d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59146/0 (socket says 172.21.15.97:59146) 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.125+0000 7fe276000640 1 -- 172.21.15.97:0/2945286634 learned_addr learned my addr 172.21.15.97:0/2945286634 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.126+0000 7fe27ea00640 1 -- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe2800d0070 msgr2=0x7fe2800d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.126+0000 7fe27ea00640 1 --2- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe2800d0070 0x7fe2800d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.126+0000 7fe27ea00640 1 -- 172.21.15.97:0/2945286634 >> v1:172.21.15.135:6789/0 conn(0x7fe28015ec10 legacy=0x7fe28015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.126+0000 7fe27ea00640 1 -- 172.21.15.97:0/2945286634 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800ad870 con 0x7fe280155490 2024-03-06T07:33:50.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.127+0000 7fe27ea00640 1 --2- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe280155490 0x7fe2800ae630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe2700097a0 tx=0x7fe27002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=778c1f82c2a2e7bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe27e000640 1 -- 172.21.15.97:0/2945286634 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270035020 con 0x7fe280155490 2024-03-06T07:33:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe27e000640 1 -- 172.21.15.97:0/2945286634 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe270031810 con 0x7fe280155490 2024-03-06T07:33:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe27e000640 1 -- 172.21.15.97:0/2945286634 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270031be0 con 0x7fe280155490 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe285c00640 1 -- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe280155490 msgr2=0x7fe2800ae630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe285c00640 1 --2- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe280155490 0x7fe2800ae630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe2700097a0 tx=0x7fe27002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe285c00640 1 -- 172.21.15.97:0/2945286634 shutdown_connections 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe285c00640 1 --2- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe2800d0070 0x7fe2800d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe285c00640 1 --2- 172.21.15.97:0/2945286634 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe280155490 0x7fe2800ae630 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.128+0000 7fe285c00640 1 -- 172.21.15.97:0/2945286634 >> 172.21.15.97:0/2945286634 conn(0x7fe28006b360 msgr2=0x7fe28006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.129+0000 7fe285c00640 1 -- 172.21.15.97:0/2945286634 shutdown_connections 2024-03-06T07:33:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.129+0000 7fe285c00640 1 -- 172.21.15.97:0/2945286634 wait complete. 2024-03-06T07:33:50.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.130+0000 7fe285c00640 1 Processor -- start 2024-03-06T07:33:50.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.130+0000 7fe285c00640 1 -- start start 2024-03-06T07:33:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.131+0000 7fe285c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe2800d0070 0x7fe28014c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.131+0000 7fe285c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe280155490 0x7fe28009aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.131+0000 7fe285c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 0x7fe28015da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.131+0000 7fe285c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800ae110 con 0x7fe28015ec10 2024-03-06T07:33:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.131+0000 7fe285c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800ade30 con 0x7fe2800d0070 2024-03-06T07:33:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.131+0000 7fe285c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe280071960 con 0x7fe280155490 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 0x7fe28015da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 0x7fe28015da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39226/0 (socket says 172.21.15.97:39226) 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 -- 172.21.15.97:0/1039901109 learned_addr learned my addr 172.21.15.97:0/1039901109 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27ea00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe2800d0070 0x7fe28014c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe276000640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe280155490 0x7fe28009aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 -- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe280155490 msgr2=0x7fe28009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe280155490 0x7fe28009aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 -- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe2800d0070 msgr2=0x7fe28014c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe2800d0070 0x7fe28014c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27f400640 1 -- 172.21.15.97:0/1039901109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28009b480 con 0x7fe28015ec10 2024-03-06T07:33:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.132+0000 7fe27ea00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe2800d0070 0x7fe28014c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.133+0000 7fe27f400640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 0x7fe28015da20 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c730 tx=0x7fe27800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.133+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278003b70 con 0x7fe28015ec10 2024-03-06T07:33:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.133+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe278003d30 con 0x7fe28015ec10 2024-03-06T07:33:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.133+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28009b760 con 0x7fe28015ec10 2024-03-06T07:33:50.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.133+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278005310 con 0x7fe28015ec10 2024-03-06T07:33:50.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.133+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28009c000 con 0x7fe28015ec10 2024-03-06T07:33:50.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.134+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28009b930 con 0x7fe28015ec10 2024-03-06T07:33:50.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.135+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2780054b0 con 0x7fe28015ec10 2024-03-06T07:33:50.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.136+0000 7fe277400640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe25c06b4d0 0x7fe25c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.136+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe27808a0e0 con 0x7fe28015ec10 2024-03-06T07:33:50.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.137+0000 7fe27ea00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe25c06b4d0 0x7fe25c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.137+0000 7fe27ea00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe25c06b4d0 0x7fe25c06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe270036000 tx=0x7fe270008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.141+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe278058f00 con 0x7fe28015ec10 2024-03-06T07:33:50.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.376+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe28009c1d0 con 0x7fe28015ec10 2024-03-06T07:33:50.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.377+0000 7fe277400640 1 -- 172.21.15.97:0/1039901109 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fe278058800 con 0x7fe28015ec10 2024-03-06T07:33:50.379 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:50.379 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:43.068988+0000","modified":"2024-03-06T07:32:58.790850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7314":{"gid":7314,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/1028636781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1028636781},{"type":"v1","addr":"172.21.15.97:6837","nonce":1028636781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7314,"qdb_cluster":[7314]},"id":7}]} 2024-03-06T07:33:50.379 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 67 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe25c06b4d0 msgr2=0x7fe25c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe25c06b4d0 0x7fe25c06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe270036000 tx=0x7fe270008030 comp rx=0 tx=0).stop 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 msgr2=0x7fe28015da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 0x7fe28015da20 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c730 tx=0x7fe27800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 shutdown_connections 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe25c06b4d0 0x7fe25c06d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe28015ec10 0x7fe28015da20 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe280155490 0x7fe28009aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 --2- 172.21.15.97:0/1039901109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe2800d0070 0x7fe28014c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 >> 172.21.15.97:0/1039901109 conn(0x7fe28006b360 msgr2=0x7fe2800b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 shutdown_connections 2024-03-06T07:33:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.379+0000 7fe285c00640 1 -- 172.21.15.97:0/1039901109 wait complete. 2024-03-06T07:33:50.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:50.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.593+0000 7f6656000640 1 Processor -- start 2024-03-06T07:33:50.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.593+0000 7f6656000640 1 -- start start 2024-03-06T07:33:50.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.594+0000 7f6656000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f665015be90 0x7f66500b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.594+0000 7f6656000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f66500d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.594+0000 7f6656000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f66500d05b0 0x7f665006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.594+0000 7f6656000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500b4f30 con 0x7f665015be90 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.594+0000 7f6656000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500af240 con 0x7f66500d05b0 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.594+0000 7f6656000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500b5160 con 0x7f66500acd60 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f665015be90 0x7f66500b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f665015be90 0x7f66500b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39230/0 (socket says 172.21.15.97:39230) 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 -- 172.21.15.97:0/148598186 learned_addr learned my addr 172.21.15.97:0/148598186 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664ea00640 1 --2- 172.21.15.97:0/148598186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f66500d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664fe00640 1 --2- 172.21.15.97:0/148598186 >> v2:172.21.15.135:3300/0 conn(0x7f66500d05b0 0x7f665006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 -- 172.21.15.97:0/148598186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 msgr2=0x7f66500d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 --2- 172.21.15.97:0/148598186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f66500d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 -- 172.21.15.97:0/148598186 >> v2:172.21.15.135:3300/0 conn(0x7f66500d05b0 msgr2=0x7f665006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 --2- 172.21.15.97:0/148598186 >> v2:172.21.15.135:3300/0 conn(0x7f66500d05b0 0x7f665006c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.595+0000 7f664f400640 1 -- 172.21.15.97:0/148598186 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500ae910 con 0x7f665015be90 2024-03-06T07:33:50.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.596+0000 7f664f400640 1 --2- 172.21.15.97:0/148598186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500b49f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f66500af880 tx=0x7f663802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf071e7ddcc6c70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.597+0000 7f664e000640 1 -- 172.21.15.97:0/148598186 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638035020 con 0x7f665015be90 2024-03-06T07:33:50.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.597+0000 7f664e000640 1 -- 172.21.15.97:0/148598186 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6638031870 con 0x7f665015be90 2024-03-06T07:33:50.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.597+0000 7f664e000640 1 -- 172.21.15.97:0/148598186 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638031c40 con 0x7f665015be90 2024-03-06T07:33:50.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 -- 172.21.15.97:0/148598186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 msgr2=0x7f66500b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 --2- 172.21.15.97:0/148598186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500b49f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f66500af880 tx=0x7f663802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 -- 172.21.15.97:0/148598186 shutdown_connections 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 --2- 172.21.15.97:0/148598186 >> v2:172.21.15.135:3300/0 conn(0x7f66500d05b0 0x7f665006c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 --2- 172.21.15.97:0/148598186 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f66500d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 --2- 172.21.15.97:0/148598186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500b49f0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 -- 172.21.15.97:0/148598186 >> 172.21.15.97:0/148598186 conn(0x7f665006b360 msgr2=0x7f665006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 -- 172.21.15.97:0/148598186 shutdown_connections 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.598+0000 7f6656000640 1 -- 172.21.15.97:0/148598186 wait complete. 2024-03-06T07:33:50.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.600+0000 7f6656000640 1 Processor -- start 2024-03-06T07:33:50.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.600+0000 7f6656000640 1 -- start start 2024-03-06T07:33:50.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.600+0000 7f6656000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f6650148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f6656000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66500d05b0 0x7f66500ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f6656000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f6656000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500aeed0 con 0x7f665015be90 2024-03-06T07:33:50.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f6656000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500d09c0 con 0x7f66500d05b0 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f6656000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500b2c80 con 0x7f66500acd60 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39244/0 (socket says 172.21.15.97:39244) 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 -- 172.21.15.97:0/2979114498 learned_addr learned my addr 172.21.15.97:0/2979114498 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664f400640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f6650148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664ea00640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66500d05b0 0x7f66500ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 -- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 msgr2=0x7f6650148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f6650148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 -- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66500d05b0 msgr2=0x7f66500ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66500d05b0 0x7f66500ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664fe00640 1 -- 172.21.15.97:0/2979114498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665014f620 con 0x7f665015be90 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.601+0000 7f664f400640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f6650148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.602+0000 7f664fe00640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500ce820 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6648007d90 tx=0x7f664800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.602+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648014420 con 0x7f665015be90 2024-03-06T07:33:50.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.602+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648019070 con 0x7f665015be90 2024-03-06T07:33:50.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.602+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665014f900 con 0x7f665015be90 2024-03-06T07:33:50.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.602+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648014820 con 0x7f665015be90 2024-03-06T07:33:50.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.602+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665014fec0 con 0x7f665015be90 2024-03-06T07:33:50.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.603+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66500addf0 con 0x7f665015be90 2024-03-06T07:33:50.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.604+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66480149c0 con 0x7f665015be90 2024-03-06T07:33:50.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.605+0000 7f6647400640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.605+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f664808aa70 con 0x7f665015be90 2024-03-06T07:33:50.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.605+0000 7f664f400640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.606+0000 7f664f400640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f662c06b4d0 0x7f662c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6638036000 tx=0x7f663803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.612+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66500addf0 con 0x7f665015be90 2024-03-06T07:33:50.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.806+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6650068070 con 0x7f665015be90 2024-03-06T07:33:50.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.808+0000 7f6647400640 1 -- 172.21.15.97:0/2979114498 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16726 (secure 0 0 0) 0x7f6648059890 con 0x7f665015be90 2024-03-06T07:33:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:50.811 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":100,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:32:54.749607+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:40.892840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:42.065778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/1818345324":"2024-03-07T07:32:54.749547+0000","172.21.15.97:6836/1818345324":"2024-03-07T07:32:54.749547+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:50.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f662c06b4d0 msgr2=0x7f662c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f662c06b4d0 0x7f662c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6638036000 tx=0x7f663803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 msgr2=0x7f66500ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500ce820 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6648007d90 tx=0x7f664800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 shutdown_connections 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665015be90 0x7f66500ce820 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66500d05b0 0x7f66500ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 --2- 172.21.15.97:0/2979114498 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66500acd60 0x7f6650148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 >> 172.21.15.97:0/2979114498 conn(0x7f665006b360 msgr2=0x7f66500ad140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 shutdown_connections 2024-03-06T07:33:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:50.811+0000 7f6656000640 1 -- 172.21.15.97:0/2979114498 wait complete. 2024-03-06T07:33:50.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:33:51.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.028+0000 7f7b46c00640 1 Processor -- start 2024-03-06T07:33:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.029+0000 7f7b46c00640 1 -- start start 2024-03-06T07:33:51.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.030+0000 7f7b46c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b400b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b46c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7b400d0780 0x7f7b4015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b46c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400b5160 con 0x7f7b400acd60 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b46c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b400b4f30 con 0x7f7b400d0780 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b46c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400af240 con 0x7f7b4015be90 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7b400d0780 0x7f7b4015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7b400d0780 0x7f7b4015fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39260/0 (socket says 172.21.15.97:39260) 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b37e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7b400acd60 0x7f7b400d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51574/0 (socket says 172.21.15.97:51574) 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 -- 172.21.15.97:0/508133618 learned_addr learned my addr 172.21.15.97:0/508133618 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b3fe00640 1 --2- 172.21.15.97:0/508133618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b400b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 -- 172.21.15.97:0/508133618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 msgr2=0x7f7b400b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 --2- 172.21.15.97:0/508133618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b400b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 -- 172.21.15.97:0/508133618 >> v1:172.21.15.97:6789/0 conn(0x7f7b400acd60 legacy=0x7f7b400d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b3f400640 1 -- 172.21.15.97:0/508133618 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279781447 0 0) 0x7f7b400b5160 con 0x7f7b400acd60 2024-03-06T07:33:51.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.031+0000 7f7b44c00640 1 -- 172.21.15.97:0/508133618 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400ae910 con 0x7f7b400d0780 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.032+0000 7f7b44c00640 1 --2- 172.21.15.97:0/508133618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4015fb70 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009e50 tx=0x7f7b3802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b778a96847943b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b3f400640 1 -- 172.21.15.97:0/508133618 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3802cb00 con 0x7f7b400d0780 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b3f400640 1 -- 172.21.15.97:0/508133618 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3802cca0 con 0x7f7b400d0780 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b3f400640 1 -- 172.21.15.97:0/508133618 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38034610 con 0x7f7b400d0780 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b46c00640 1 -- 172.21.15.97:0/508133618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 msgr2=0x7f7b4015fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b46c00640 1 --2- 172.21.15.97:0/508133618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4015fb70 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009e50 tx=0x7f7b3802bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b46c00640 1 -- 172.21.15.97:0/508133618 shutdown_connections 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b46c00640 1 --2- 172.21.15.97:0/508133618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4015fb70 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b46c00640 1 --2- 172.21.15.97:0/508133618 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b400b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.033+0000 7f7b46c00640 1 -- 172.21.15.97:0/508133618 >> 172.21.15.97:0/508133618 conn(0x7f7b4006b360 msgr2=0x7f7b4006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.034+0000 7f7b46c00640 1 -- 172.21.15.97:0/508133618 shutdown_connections 2024-03-06T07:33:51.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.034+0000 7f7b46c00640 1 -- 172.21.15.97:0/508133618 wait complete. 2024-03-06T07:33:51.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.035+0000 7f7b46c00640 1 Processor -- start 2024-03-06T07:33:51.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.035+0000 7f7b46c00640 1 -- start start 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b46c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b400acd60 0x7f7b4015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b46c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4014eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b46c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b401f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b46c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400b2c80 con 0x7f7b400d0780 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b46c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400aeed0 con 0x7f7b400acd60 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b46c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400b2e00 con 0x7f7b4015be90 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.036+0000 7f7b37e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4014eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4014eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39276/0 (socket says 172.21.15.97:39276) 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 -- 172.21.15.97:0/744253490 learned_addr learned my addr 172.21.15.97:0/744253490 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b3fe00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b400acd60 0x7f7b4015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 -- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 msgr2=0x7f7b401f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b401f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 -- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b400acd60 msgr2=0x7f7b4015a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b400acd60 0x7f7b4015a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 -- 172.21.15.97:0/744253490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b401f29f0 con 0x7f7b400d0780 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.037+0000 7f7b37e00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4014eaf0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800c4d0 tx=0x7f7b2800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.038+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28018070 con 0x7f7b400d0780 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.038+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28007500 con 0x7f7b400d0780 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.038+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28004a20 con 0x7f7b400d0780 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.038+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b400cf080 con 0x7f7b400d0780 2024-03-06T07:33:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.038+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b40159b00 con 0x7f7b400d0780 2024-03-06T07:33:51.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.039+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b40068070 con 0x7f7b400d0780 2024-03-06T07:33:51.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.041+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b2801b070 con 0x7f7b400d0780 2024-03-06T07:33:51.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.042+0000 7f7b3cc00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.042+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7b28089580 con 0x7f7b400d0780 2024-03-06T07:33:51.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.042+0000 7f7b3fe00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.043+0000 7f7b3fe00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.047+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b280583a0 con 0x7f7b400d0780 2024-03-06T07:33:51.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.273+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b40155d80 con 0x7f7b400d0780 2024-03-06T07:33:51.276 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:33:51.274+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:51.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.277+0000 7f7b3cc00640 1 -- 172.21.15.97:0/744253490 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b28057ca0 con 0x7f7b400d0780 2024-03-06T07:33:51.280 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.280+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b1c06b4d0 msgr2=0x7f7b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30008040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 msgr2=0x7f7b4014eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4014eaf0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800c4d0 tx=0x7f7b2800c990 comp rx=0 tx=0).stop 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 shutdown_connections 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b4015be90 0x7f7b401f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b400d0780 0x7f7b4014eaf0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 --2- 172.21.15.97:0/744253490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b400acd60 0x7f7b4015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 >> 172.21.15.97:0/744253490 conn(0x7f7b4006b360 msgr2=0x7f7b4006d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 shutdown_connections 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.281+0000 7f7b46c00640 1 -- 172.21.15.97:0/744253490 wait complete. 2024-03-06T07:33:51.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:33:51.388 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:33:51.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.489+0000 7f6b1c800640 1 Processor -- start 2024-03-06T07:33:51.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.489+0000 7f6b1c800640 1 -- start start 2024-03-06T07:33:51.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.490+0000 7f6b1c800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b140d0780 0x7f6b140b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.490+0000 7f6b1c800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b140b5160 con 0x7f6b140acd60 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.490+0000 7f6b1c800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b140ae910 con 0x7f6b1415be90 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.490+0000 7f6b1c800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b140b4f30 con 0x7f6b140d0780 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.490+0000 7f6b19400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b140d0780 0x7f6b140b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.491+0000 7f6b1a800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6b140acd60 0x7f6b1415fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51580/0 (socket says 172.21.15.97:51580) 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.491+0000 7f6b19400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6b140d0780 0x7f6b140b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39288/0 (socket says 172.21.15.97:39288) 2024-03-06T07:33:51.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.491+0000 7f6b1a800640 1 -- 172.21.15.97:0/2582086360 learned_addr learned my addr 172.21.15.97:0/2582086360 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:51.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.491+0000 7f6b19400640 1 -- 172.21.15.97:0/2582086360 >> v1:172.21.15.135:6789/0 conn(0x7f6b1415be90 legacy=0x7f6b140d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.491+0000 7f6b19400640 1 -- 172.21.15.97:0/2582086360 >> v1:172.21.15.97:6789/0 conn(0x7f6b140acd60 legacy=0x7f6b1415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.491+0000 7f6b19400640 1 -- 172.21.15.97:0/2582086360 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b140aebf0 con 0x7f6b140d0780 2024-03-06T07:33:51.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.492+0000 7f6b19400640 1 --2- 172.21.15.97:0/2582086360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b140d0780 0x7f6b140b49f0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6b080097a0 tx=0x7f6b0802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65242c5cbf56771 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b18a00640 1 -- 172.21.15.97:0/2582086360 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08037070 con 0x7f6b140d0780 2024-03-06T07:33:51.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b18a00640 1 -- 172.21.15.97:0/2582086360 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08030710 con 0x7f6b140d0780 2024-03-06T07:33:51.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b18a00640 1 -- 172.21.15.97:0/2582086360 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08030ae0 con 0x7f6b140d0780 2024-03-06T07:33:51.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 -- 172.21.15.97:0/2582086360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b140d0780 msgr2=0x7f6b140b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 --2- 172.21.15.97:0/2582086360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b140d0780 0x7f6b140b49f0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6b080097a0 tx=0x7f6b0802b710 comp rx=0 tx=0).stop 2024-03-06T07:33:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 -- 172.21.15.97:0/2582086360 shutdown_connections 2024-03-06T07:33:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 --2- 172.21.15.97:0/2582086360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b140d0780 0x7f6b140b49f0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 -- 172.21.15.97:0/2582086360 >> 172.21.15.97:0/2582086360 conn(0x7f6b1406b360 msgr2=0x7f6b1406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 -- 172.21.15.97:0/2582086360 shutdown_connections 2024-03-06T07:33:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.493+0000 7f6b1c800640 1 -- 172.21.15.97:0/2582086360 wait complete. 2024-03-06T07:33:51.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.495+0000 7f6b1c800640 1 Processor -- start 2024-03-06T07:33:51.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.495+0000 7f6b1c800640 1 -- start start 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1c800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b140acd60 0x7f6b141f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1c800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b140d0780 0x7f6b1409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1c800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 0x7f6b141532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1c800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b140b2c80 con 0x7f6b1415be90 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1c800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b140af240 con 0x7f6b140d0780 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1c800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b140b2e00 con 0x7f6b140acd60 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 0x7f6b141532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 0x7f6b141532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39290/0 (socket says 172.21.15.97:39290) 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.496+0000 7f6b1a800640 1 -- 172.21.15.97:0/1442125490 learned_addr learned my addr 172.21.15.97:0/1442125490 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:51.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b19400640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b140d0780 0x7f6b1409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b19e00640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b140acd60 0x7f6b141f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1a800640 1 -- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b140acd60 msgr2=0x7f6b141f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1a800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b140acd60 0x7f6b141f47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1a800640 1 -- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b140d0780 msgr2=0x7f6b1409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1a800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b140d0780 0x7f6b1409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1a800640 1 -- 172.21.15.97:0/1442125490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b140cd110 con 0x7f6b1415be90 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b19400640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b140d0780 0x7f6b1409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:51.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1a800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 0x7f6b141532f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000c910 tx=0x7f6b1000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10018070 con 0x7f6b1415be90 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1000a700 con 0x7f6b1415be90 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000ab30 con 0x7f6b1415be90 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.497+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b141f5080 con 0x7f6b1415be90 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.498+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b141f5640 con 0x7f6b1415be90 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.499+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b140b5f70 con 0x7f6b1415be90 2024-03-06T07:33:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.500+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b1001b070 con 0x7f6b1415be90 2024-03-06T07:33:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.500+0000 7f6b0e000640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.501+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6b10089a60 con 0x7f6b1415be90 2024-03-06T07:33:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.501+0000 7f6b19e00640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.501+0000 7f6b19e00640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6af806b4d0 0x7f6af806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6b04001500 tx=0x7f6b04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.505+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b140b5f70 con 0x7f6b1415be90 2024-03-06T07:33:51.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:51.740+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6b141f5e70 con 0x7f6b1415be90 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.281+0000 7f6b0e000640 1 -- 172.21.15.97:0/1442125490 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6b140cd110 con 0x7f6b1415be90 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6af806b4d0 msgr2=0x7f6af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6af806b4d0 0x7f6af806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6b04001500 tx=0x7f6b04009040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 msgr2=0x7f6b141532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 0x7f6b141532f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000c910 tx=0x7f6b1000cdd0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 shutdown_connections 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b1415be90 0x7f6b141532f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b140d0780 0x7f6b1409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 --2- 172.21.15.97:0/1442125490 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6b140acd60 0x7f6b141f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.284+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 >> 172.21.15.97:0/1442125490 conn(0x7f6b1406b360 msgr2=0x7f6b14070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.285+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 shutdown_connections 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.285+0000 7f6b1c800640 1 -- 172.21.15.97:0/1442125490 wait complete. 2024-03-06T07:33:52.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:33:52.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.525+0000 7f8e2e200640 1 Processor -- start 2024-03-06T07:33:52.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.525+0000 7f8e2e200640 1 -- start start 2024-03-06T07:33:52.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e2e200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e28156610 0x7f8e280b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e2e200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8e280acd60 0x7f8e280d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e2e200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8e280af2f0 0x7f8e280d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e2e200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e281569f0 con 0x7f8e280acd60 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e2e200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2815c850 con 0x7f8e280af2f0 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e2e200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2815cb30 con 0x7f8e28156610 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e26a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8e280acd60 0x7f8e280d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e26a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8e280acd60 0x7f8e280d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39302/0 (socket says 172.21.15.97:39302) 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e26a00640 1 -- 172.21.15.97:0/548059303 learned_addr learned my addr 172.21.15.97:0/548059303 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e27400640 1 --2- 172.21.15.97:0/548059303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e28156610 0x7f8e280b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.527+0000 7f8e27e00640 1 --2- 172.21.15.97:0/548059303 >> v2:172.21.15.135:3300/0 conn(0x7f8e280af2f0 0x7f8e280d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.528+0000 7f8e26a00640 1 -- 172.21.15.97:0/548059303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e28156610 msgr2=0x7f8e280b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.528+0000 7f8e26a00640 1 --2- 172.21.15.97:0/548059303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e28156610 0x7f8e280b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.528+0000 7f8e26a00640 1 -- 172.21.15.97:0/548059303 >> v2:172.21.15.135:3300/0 conn(0x7f8e280af2f0 msgr2=0x7f8e280d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.528+0000 7f8e26a00640 1 --2- 172.21.15.97:0/548059303 >> v2:172.21.15.135:3300/0 conn(0x7f8e280af2f0 0x7f8e280d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.528+0000 7f8e26a00640 1 -- 172.21.15.97:0/548059303 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2815ce10 con 0x7f8e280acd60 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.528+0000 7f8e27e00640 1 --2- 172.21.15.97:0/548059303 >> v2:172.21.15.135:3300/0 conn(0x7f8e280af2f0 0x7f8e280d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:52.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.530+0000 7f8e26a00640 1 --2- 172.21.15.97:0/548059303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e280d0070 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8e28072e10 tx=0x7f8e1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7676d7670ec1112 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.530+0000 7f8e26000640 1 -- 172.21.15.97:0/548059303 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14035020 con 0x7f8e280acd60 2024-03-06T07:33:52.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.530+0000 7f8e26000640 1 -- 172.21.15.97:0/548059303 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14031870 con 0x7f8e280acd60 2024-03-06T07:33:52.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.530+0000 7f8e26000640 1 -- 172.21.15.97:0/548059303 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14031c40 con 0x7f8e280acd60 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.530+0000 7f8e2e200640 1 -- 172.21.15.97:0/548059303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 msgr2=0x7f8e280d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.530+0000 7f8e2e200640 1 --2- 172.21.15.97:0/548059303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e280d0070 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8e28072e10 tx=0x7f8e1402b910 comp rx=0 tx=0).stop 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 -- 172.21.15.97:0/548059303 shutdown_connections 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 --2- 172.21.15.97:0/548059303 >> v2:172.21.15.135:3300/0 conn(0x7f8e280af2f0 0x7f8e280d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 --2- 172.21.15.97:0/548059303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e280d0070 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 --2- 172.21.15.97:0/548059303 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e28156610 0x7f8e280b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 -- 172.21.15.97:0/548059303 >> 172.21.15.97:0/548059303 conn(0x7f8e2806b360 msgr2=0x7f8e2806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 -- 172.21.15.97:0/548059303 shutdown_connections 2024-03-06T07:33:52.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.531+0000 7f8e2e200640 1 -- 172.21.15.97:0/548059303 wait complete. 2024-03-06T07:33:52.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.532+0000 7f8e2e200640 1 Processor -- start 2024-03-06T07:33:52.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 -- start start 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e2809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e280af2f0 0x7f8e28147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e28156610 0x7f8e28148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2815c540 con 0x7f8e280acd60 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2815d3d0 con 0x7f8e28156610 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.533+0000 7f8e2e200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2806cb30 con 0x7f8e280af2f0 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e2809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e2809aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39310/0 (socket says 172.21.15.97:39310) 2024-03-06T07:33:52.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 -- 172.21.15.97:0/1437197245 learned_addr learned my addr 172.21.15.97:0/1437197245 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 -- 172.21.15.97:0/1437197245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e280af2f0 msgr2=0x7f8e28147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e280af2f0 0x7f8e28147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 -- 172.21.15.97:0/1437197245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e28156610 msgr2=0x7f8e28148010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e28156610 0x7f8e28148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 -- 172.21.15.97:0/1437197245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e28152b80 con 0x7f8e280acd60 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.534+0000 7f8e27400640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e2809aa30 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f8e1800d5f0 tx=0x7f8e1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.535+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180125f0 con 0x7f8e280acd60 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.535+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e18019070 con 0x7f8e280acd60 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.535+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180129d0 con 0x7f8e280acd60 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.535+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e28152e60 con 0x7f8e280acd60 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.535+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e28153420 con 0x7f8e280acd60 2024-03-06T07:33:52.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.536+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e28068070 con 0x7f8e280acd60 2024-03-06T07:33:52.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.537+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e18012b70 con 0x7f8e280acd60 2024-03-06T07:33:52.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.538+0000 7f8e1f400640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e0806b500 0x7f8e0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.538+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f8e1808a620 con 0x7f8e280acd60 2024-03-06T07:33:52.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.538+0000 7f8e26a00640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e0806b500 0x7f8e0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.539+0000 7f8e26a00640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e0806b500 0x7f8e0806d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8e14036000 tx=0x7f8e14030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.543+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e180593f0 con 0x7f8e280acd60 2024-03-06T07:33:52.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:52.741+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e28154e40 con 0x7f8e280acd60 2024-03-06T07:33:53.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.284+0000 7f8e1f400640 1 -- 172.21.15.97:0/1437197245 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8e18058cf0 con 0x7f8e280acd60 2024-03-06T07:33:53.287 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:33:53.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.289+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e0806b500 msgr2=0x7f8e0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.289+0000 7f8e2e200640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e0806b500 0x7f8e0806d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8e14036000 tx=0x7f8e14030000 comp rx=0 tx=0).stop 2024-03-06T07:33:53.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.289+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 msgr2=0x7f8e2809aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.289+0000 7f8e2e200640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e2809aa30 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f8e1800d5f0 tx=0x7f8e1800dab0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.291+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 shutdown_connections 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.291+0000 7f8e2e200640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e0806b500 0x7f8e0806d9c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.291+0000 7f8e2e200640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e28156610 0x7f8e28148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.291+0000 7f8e2e200640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e280af2f0 0x7f8e28147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.291+0000 7f8e2e200640 1 --2- 172.21.15.97:0/1437197245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e280acd60 0x7f8e2809aa30 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.291+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 >> 172.21.15.97:0/1437197245 conn(0x7f8e2806b360 msgr2=0x7f8e2815b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.292+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 shutdown_connections 2024-03-06T07:33:53.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.292+0000 7f8e2e200640 1 -- 172.21.15.97:0/1437197245 wait complete. 2024-03-06T07:33:53.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:33:53.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.517+0000 7f86e1000640 1 Processor -- start 2024-03-06T07:33:53.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.517+0000 7f86e1000640 1 -- start start 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.518+0000 7f86e1000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f86dc15e3b0 0x7f86dc0d1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.518+0000 7f86e1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f86dc0d2d00 0x7f86dc0b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.518+0000 7f86e1000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0d2040 con 0x7f86dc0d4b30 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.518+0000 7f86e1000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc0d1df0 con 0x7f86dc0d2d00 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.518+0000 7f86e1000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc0b3580 con 0x7f86dc15e3b0 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f86dc0d2d00 0x7f86dc0b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f86dc0d2d00 0x7f86dc0b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39320/0 (socket says 172.21.15.97:39320) 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 -- 172.21.15.97:0/3715703938 learned_addr learned my addr 172.21.15.97:0/3715703938 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:53.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86d9e00640 1 --2- 172.21.15.97:0/3715703938 >> v2:172.21.15.135:3300/0 conn(0x7f86dc15e3b0 0x7f86dc0d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 -- 172.21.15.97:0/3715703938 >> v2:172.21.15.135:3300/0 conn(0x7f86dc15e3b0 msgr2=0x7f86dc0d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 --2- 172.21.15.97:0/3715703938 >> v2:172.21.15.135:3300/0 conn(0x7f86dc15e3b0 0x7f86dc0d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 -- 172.21.15.97:0/3715703938 >> v1:172.21.15.97:6789/0 conn(0x7f86dc0d4b30 legacy=0x7f86dc0d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86da800640 1 -- 172.21.15.97:0/3715703938 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0b3860 con 0x7f86dc0d2d00 2024-03-06T07:33:53.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.519+0000 7f86d8a00640 1 -- 172.21.15.97:0/3715703938 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869382081 0 0) 0x7f86dc0d2040 con 0x7f86dc0d4b30 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.520+0000 7f86da800640 1 --2- 172.21.15.97:0/3715703938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc0d2d00 0x7f86dc0b3d80 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0d3fe0 tx=0x7f86d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e200140d5e66968 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86d8a00640 1 -- 172.21.15.97:0/3715703938 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4035020 con 0x7f86dc0d2d00 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86d8a00640 1 -- 172.21.15.97:0/3715703938 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d4031870 con 0x7f86dc0d2d00 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86d8a00640 1 -- 172.21.15.97:0/3715703938 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4031c40 con 0x7f86dc0d2d00 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 -- 172.21.15.97:0/3715703938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc0d2d00 msgr2=0x7f86dc0b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 --2- 172.21.15.97:0/3715703938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc0d2d00 0x7f86dc0b3d80 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0d3fe0 tx=0x7f86d402b910 comp rx=0 tx=0).stop 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 -- 172.21.15.97:0/3715703938 shutdown_connections 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 --2- 172.21.15.97:0/3715703938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc0d2d00 0x7f86dc0b3d80 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 --2- 172.21.15.97:0/3715703938 >> v2:172.21.15.135:3300/0 conn(0x7f86dc15e3b0 0x7f86dc0d1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 -- 172.21.15.97:0/3715703938 >> 172.21.15.97:0/3715703938 conn(0x7f86dc0d4010 msgr2=0x7f86dc0d4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 -- 172.21.15.97:0/3715703938 shutdown_connections 2024-03-06T07:33:53.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.521+0000 7f86e1000640 1 -- 172.21.15.97:0/3715703938 wait complete. 2024-03-06T07:33:53.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.523+0000 7f86e1000640 1 Processor -- start 2024-03-06T07:33:53.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.523+0000 7f86e1000640 1 -- start start 2024-03-06T07:33:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86e1000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86dc0d2d00 0x7f86dc15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86e1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86dc0d4b30 0x7f86dc160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86e1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 0x7f86dc161090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86e1000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc15e790 con 0x7f86dc15e3b0 2024-03-06T07:33:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86e1000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0cff60 con 0x7f86dc0d2d00 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86e1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc15e910 con 0x7f86dc0d4b30 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.524+0000 7f86da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 0x7f86dc161090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 0x7f86dc161090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39334/0 (socket says 172.21.15.97:39334) 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 -- 172.21.15.97:0/3643359024 learned_addr learned my addr 172.21.15.97:0/3643359024 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 -- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86dc0d4b30 msgr2=0x7f86dc160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86d9400640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86dc0d4b30 0x7f86dc160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86d9e00640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86dc0d2d00 0x7f86dc15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86dc0d4b30 0x7f86dc160b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 -- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86dc0d2d00 msgr2=0x7f86dc15c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86dc0d2d00 0x7f86dc15c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 -- 172.21.15.97:0/3643359024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0ccfc0 con 0x7f86dc15e3b0 2024-03-06T07:33:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.525+0000 7f86da800640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 0x7f86dc161090 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009960 tx=0x7f86d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.526+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4041070 con 0x7f86dc15e3b0 2024-03-06T07:33:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.526+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d403ad00 con 0x7f86dc15e3b0 2024-03-06T07:33:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.526+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc0cd280 con 0x7f86dc15e3b0 2024-03-06T07:33:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.526+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4035040 con 0x7f86dc15e3b0 2024-03-06T07:33:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.526+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc0cd840 con 0x7f86dc15e3b0 2024-03-06T07:33:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.527+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0b69d0 con 0x7f86dc15e3b0 2024-03-06T07:33:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.528+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86d4038020 con 0x7f86dc15e3b0 2024-03-06T07:33:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.529+0000 7f86d2000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86b806b4d0 0x7f86b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.529+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f86d40abfa0 con 0x7f86dc15e3b0 2024-03-06T07:33:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.529+0000 7f86d9e00640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86b806b4d0 0x7f86b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.530+0000 7f86d9e00640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86b806b4d0 0x7f86b806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f86cc006fd0 tx=0x7f86cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.533+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86dc0b69d0 con 0x7f86dc15e3b0 2024-03-06T07:33:53.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:53.729+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86dc068ea0 con 0x7f86dc15e3b0 2024-03-06T07:33:54.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.298+0000 7f86d2000640 1 -- 172.21.15.97:0/3643359024 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 169+0+0 (secure 0 0 0) 0x7f86d407af00 con 0x7f86dc15e3b0 2024-03-06T07:33:54.301 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86b806b4d0 msgr2=0x7f86b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86b806b4d0 0x7f86b806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f86cc006fd0 tx=0x7f86cc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 msgr2=0x7f86dc161090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 0x7f86dc161090 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009960 tx=0x7f86d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 shutdown_connections 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f86b806b4d0 0x7f86b806d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86dc15e3b0 0x7f86dc161090 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f86dc0d4b30 0x7f86dc160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 --2- 172.21.15.97:0/3643359024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86dc0d2d00 0x7f86dc15c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.302+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 >> 172.21.15.97:0/3643359024 conn(0x7f86dc0d4010 msgr2=0x7f86dc0b62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.303+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 shutdown_connections 2024-03-06T07:33:54.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:54.303+0000 7f86e1000640 1 -- 172.21.15.97:0/3643359024 wait complete. 2024-03-06T07:33:54.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:54.342 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:54.484 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:33:54.484 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:33:54.484 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:54.484 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:54.485 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:54.486 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:33:54.487 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.488 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:33:54.489 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:33:54.490 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:54.491 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:33:54.494 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.495 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.495 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_files = 6 2024-03-06T07:33:54.495 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.509 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.510 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.511 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:33:54.512 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:33:54.514 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.515 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:33:54.518 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_files = 6 2024-03-06T07:33:54.519 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:54.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:54.561 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:33:54.727 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:33:54.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:33:54.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.730 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:54.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:33:54.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.734 INFO:teuthology.orchestra.run.smithi097.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:33:54.735 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:54.736 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:33:54.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:33:54.738 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:33:54.739 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:33:54.740 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.741 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:33:54.742 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.744 INFO:teuthology.orchestra.run.smithi135.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:54.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:54.778 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:54.938 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-06T07:33:54.939 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-06T07:33:54.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.941 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-06T07:33:54.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2024-03-06T07:33:54.943 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:33:54.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.945 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-06T07:33:54.946 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = "" 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2024-03-06T07:33:54.947 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:54.948 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:54.949 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.950 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.97,172.21.15.135,[v2:172.21.15.135:3301,v1:172.21.15.135:6790]" 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on skipped message = False 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 0b9abceb-2115-4321-96a2-348db0d0fbe4 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.951 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.952 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1/20 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1/20 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 4/10 2024-03-06T07:33:54.953 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.954 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.955 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: mon op complaint time = 120 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: client mount timeout = 600 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: debug client = 20 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = "" 2024-03-06T07:33:54.956 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.a] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.c] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.b] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mds.d] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout:[mds] 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds = 20 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: debug mds balancer = 20 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug frag = True 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: mds debug scatterstat = True 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: mds op complaint time = 180 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: mds verify scatter = True 2024-03-06T07:33:54.957 INFO:teuthology.orchestra.run.smithi135.stdout: osd op complaint time = 180 2024-03-06T07:33:54.958 INFO:teuthology.orchestra.run.smithi135.stdout: rados mon op timeout = 900 2024-03-06T07:33:54.958 INFO:teuthology.orchestra.run.smithi135.stdout: rados osd op timeout = 900 2024-03-06T07:33:54.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:33:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.199+0000 7f03ce800640 1 Processor -- start 2024-03-06T07:33:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.199+0000 7f03ce800640 1 -- start start 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.200+0000 7f03ce800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c80b2c40 0x7f03c815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.200+0000 7f03ce800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f03c806d960 0x7f03c80b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.200+0000 7f03ce800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80b3560 con 0x7f03c806e2f0 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03ce800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c815f700 con 0x7f03c806d960 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03ce800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806d4d0 con 0x7f03c80b2c40 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f03c806d960 0x7f03c80b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f03c806d960 0x7f03c80b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43656/0 (socket says 172.21.15.97:43656) 2024-03-06T07:33:55.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 -- 172.21.15.97:0/3687699527 learned_addr learned my addr 172.21.15.97:0/3687699527 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:55.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03c7400640 1 --2- 172.21.15.97:0/3687699527 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c80b2c40 0x7f03c815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 -- 172.21.15.97:0/3687699527 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c80b2c40 msgr2=0x7f03c815f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 --2- 172.21.15.97:0/3687699527 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c80b2c40 0x7f03c815f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 -- 172.21.15.97:0/3687699527 >> v1:172.21.15.135:6789/0 conn(0x7f03c806e2f0 legacy=0x7f03c8161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.201+0000 7f03bea00640 1 -- 172.21.15.97:0/3687699527 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c806cba0 con 0x7f03c806d960 2024-03-06T07:33:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.202+0000 7f03bea00640 1 --2- 172.21.15.97:0/3687699527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c806d960 0x7f03c80b3020 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009870 tx=0x7f03b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8028dd3ff534538 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.203+0000 7f03c6a00640 1 -- 172.21.15.97:0/3687699527 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0035020 con 0x7f03c806d960 2024-03-06T07:33:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.203+0000 7f03c6a00640 1 -- 172.21.15.97:0/3687699527 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0031640 con 0x7f03c806d960 2024-03-06T07:33:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.203+0000 7f03c6a00640 1 -- 172.21.15.97:0/3687699527 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b002d040 con 0x7f03c806d960 2024-03-06T07:33:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.203+0000 7f03ce800640 1 -- 172.21.15.97:0/3687699527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c806d960 msgr2=0x7f03c80b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.203+0000 7f03ce800640 1 --2- 172.21.15.97:0/3687699527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c806d960 0x7f03c80b3020 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009870 tx=0x7f03b002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.204+0000 7f03ce800640 1 -- 172.21.15.97:0/3687699527 shutdown_connections 2024-03-06T07:33:55.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.204+0000 7f03ce800640 1 --2- 172.21.15.97:0/3687699527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c806d960 0x7f03c80b3020 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.204+0000 7f03ce800640 1 --2- 172.21.15.97:0/3687699527 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c80b2c40 0x7f03c815f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.204+0000 7f03ce800640 1 -- 172.21.15.97:0/3687699527 >> 172.21.15.97:0/3687699527 conn(0x7f03c80b1940 msgr2=0x7f03c80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.204+0000 7f03ce800640 1 -- 172.21.15.97:0/3687699527 shutdown_connections 2024-03-06T07:33:55.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.204+0000 7f03ce800640 1 -- 172.21.15.97:0/3687699527 wait complete. 2024-03-06T07:33:55.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.205+0000 7f03ce800640 1 Processor -- start 2024-03-06T07:33:55.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 -- start start 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c806d960 0x7f03c8154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c806e2f0 0x7f03c8155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 0x7f03c8071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806e750 con 0x7f03c80b2c40 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806d160 con 0x7f03c806d960 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.206+0000 7f03ce800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806e8d0 con 0x7f03c806e2f0 2024-03-06T07:33:55.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 0x7f03c8071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 0x7f03c8071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43664/0 (socket says 172.21.15.97:43664) 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 -- 172.21.15.97:0/4226396814 learned_addr learned my addr 172.21.15.97:0/4226396814 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7400640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c806d960 0x7f03c8154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03bea00640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c806e2f0 0x7f03c8155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 -- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c806e2f0 msgr2=0x7f03c8155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c806e2f0 0x7f03c8155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 -- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c806d960 msgr2=0x7f03c8154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c806d960 0x7f03c8154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 -- 172.21.15.97:0/4226396814 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8154d80 con 0x7f03c80b2c40 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.207+0000 7f03c7e00640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 0x7f03c8071c60 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b410 tx=0x7f03c000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.208+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0015030 con 0x7f03c80b2c40 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.208+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c0004020 con 0x7f03c80b2c40 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.208+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0004410 con 0x7f03c80b2c40 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.208+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c8156530 con 0x7f03c80b2c40 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.208+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8156dd0 con 0x7f03c80b2c40 2024-03-06T07:33:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.209+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c804f330 con 0x7f03c80b2c40 2024-03-06T07:33:55.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.210+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03c0018070 con 0x7f03c80b2c40 2024-03-06T07:33:55.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.211+0000 7f03bfe00640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03a406b4d0 0x7f03a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.211+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f03c0089ab0 con 0x7f03c80b2c40 2024-03-06T07:33:55.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.211+0000 7f03c7400640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03a406b4d0 0x7f03a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.212+0000 7f03c7400640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03a406b4d0 0x7f03a406d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f03c809d9f0 tx=0x7f03b80079a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.215+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03c8154d80 con 0x7f03c80b2c40 2024-03-06T07:33:55.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.415+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f03c80b2530 con 0x7f03c80b2c40 2024-03-06T07:33:55.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.421+0000 7f03bfe00640 1 -- 172.21.15.97:0/4226396814 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f03c0058b30 con 0x7f03c80b2c40 2024-03-06T07:33:55.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03a406b4d0 msgr2=0x7f03a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03a406b4d0 0x7f03a406d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f03c809d9f0 tx=0x7f03b80079a0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 msgr2=0x7f03c8071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 0x7f03c8071c60 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b410 tx=0x7f03c000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 shutdown_connections 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f03a406b4d0 0x7f03a406d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03c80b2c40 0x7f03c8071c60 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03c806e2f0 0x7f03c8155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 --2- 172.21.15.97:0/4226396814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03c806d960 0x7f03c8154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.424+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 >> 172.21.15.97:0/4226396814 conn(0x7f03c80b1940 msgr2=0x7f03c80af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.425+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 shutdown_connections 2024-03-06T07:33:55.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.425+0000 7f03ce800640 1 -- 172.21.15.97:0/4226396814 wait complete. 2024-03-06T07:33:55.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:33:55.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.643+0000 7f6afa800640 1 Processor -- start 2024-03-06T07:33:55.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.643+0000 7f6afa800640 1 -- start start 2024-03-06T07:33:55.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6afa800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6af4157fe0 0x7f6af40b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6afa800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6af406eff0 0x7f6af406e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6afa800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af415e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6afa800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af415ea30 con 0x7f6af4157fe0 2024-03-06T07:33:55.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6afa800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af406eb60 con 0x7f6af406eff0 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6afa800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40b3ef0 con 0x7f6af415e080 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6af3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6af4157fe0 0x7f6af40b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6af3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6af4157fe0 0x7f6af40b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43680/0 (socket says 172.21.15.97:43680) 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.645+0000 7f6af3400640 1 -- 172.21.15.97:0/4245017481 learned_addr learned my addr 172.21.15.97:0/4245017481 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af2a00640 1 --2- 172.21.15.97:0/4245017481 >> v2:172.21.15.135:3300/0 conn(0x7f6af406eff0 0x7f6af406e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af3400640 1 -- 172.21.15.97:0/4245017481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 msgr2=0x7f6af415e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af3e00640 1 --2- 172.21.15.97:0/4245017481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af415e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af3400640 1 --2- 172.21.15.97:0/4245017481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af415e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af3400640 1 -- 172.21.15.97:0/4245017481 >> v2:172.21.15.135:3300/0 conn(0x7f6af406eff0 msgr2=0x7f6af406e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af3400640 1 --2- 172.21.15.97:0/4245017481 >> v2:172.21.15.135:3300/0 conn(0x7f6af406eff0 0x7f6af406e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af3400640 1 -- 172.21.15.97:0/4245017481 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af40b41d0 con 0x7f6af4157fe0 2024-03-06T07:33:55.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.646+0000 7f6af2a00640 1 --2- 172.21.15.97:0/4245017481 >> v2:172.21.15.135:3300/0 conn(0x7f6af406eff0 0x7f6af406e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.647+0000 7f6af3400640 1 --2- 172.21.15.97:0/4245017481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af40b33b0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f6af40d11b0 tx=0x7f6ae402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97546787ef9c9f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.647+0000 7f6af2000640 1 -- 172.21.15.97:0/4245017481 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4035020 con 0x7f6af4157fe0 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.647+0000 7f6af2000640 1 -- 172.21.15.97:0/4245017481 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae402d040 con 0x7f6af4157fe0 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6af2000640 1 -- 172.21.15.97:0/4245017481 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4031a80 con 0x7f6af4157fe0 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 -- 172.21.15.97:0/4245017481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 msgr2=0x7f6af40b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 --2- 172.21.15.97:0/4245017481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af40b33b0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f6af40d11b0 tx=0x7f6ae402b900 comp rx=0 tx=0).stop 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 -- 172.21.15.97:0/4245017481 shutdown_connections 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 --2- 172.21.15.97:0/4245017481 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af415e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 --2- 172.21.15.97:0/4245017481 >> v2:172.21.15.135:3300/0 conn(0x7f6af406eff0 0x7f6af406e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 --2- 172.21.15.97:0/4245017481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af40b33b0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 -- 172.21.15.97:0/4245017481 >> 172.21.15.97:0/4245017481 conn(0x7f6af40d0340 msgr2=0x7f6af40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 -- 172.21.15.97:0/4245017481 shutdown_connections 2024-03-06T07:33:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.648+0000 7f6afa800640 1 -- 172.21.15.97:0/4245017481 wait complete. 2024-03-06T07:33:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.650+0000 7f6afa800640 1 Processor -- start 2024-03-06T07:33:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.650+0000 7f6afa800640 1 -- start start 2024-03-06T07:33:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6afa800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6af406eff0 0x7f6af4158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6afa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af4150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6afa800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af40cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6afa800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af41583c0 con 0x7f6af4157fe0 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6afa800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40b3a40 con 0x7f6af406eff0 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6afa800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af4066d00 con 0x7f6af415e080 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6af2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af4150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6af2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af4150b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43688/0 (socket says 172.21.15.97:43688) 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.651+0000 7f6af2a00640 1 -- 172.21.15.97:0/836154721 learned_addr learned my addr 172.21.15.97:0/836154721 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af3400640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6af406eff0 0x7f6af4158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af3e00640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af40cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af2a00640 1 -- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 msgr2=0x7f6af40cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af2a00640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af40cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af2a00640 1 -- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6af406eff0 msgr2=0x7f6af4158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af2a00640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6af406eff0 0x7f6af4158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af2a00640 1 -- 172.21.15.97:0/836154721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af41523a0 con 0x7f6af4157fe0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af3400640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6af406eff0 0x7f6af4158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6af2a00640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af4150b40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00cf60 tx=0x7f6adc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc01a070 con 0x7f6af4157fe0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc00d840 con 0x7f6af4157fe0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00dc10 con 0x7f6af4157fe0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.652+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af4152680 con 0x7f6af4157fe0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.653+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af4152030 con 0x7f6af4157fe0 2024-03-06T07:33:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.654+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af4074ae0 con 0x7f6af4157fe0 2024-03-06T07:33:55.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.655+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6adc01d070 con 0x7f6af4157fe0 2024-03-06T07:33:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.656+0000 7f6aeb400640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ad006b4d0 0x7f6ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.656+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6adc08a680 con 0x7f6af4157fe0 2024-03-06T07:33:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.656+0000 7f6af3400640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ad006b4d0 0x7f6ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.657+0000 7f6af3400640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ad006b4d0 0x7f6ad006d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4036000 tx=0x7f6ae4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.660+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6adc0591e0 con 0x7f6af4157fe0 2024-03-06T07:33:55.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:55.933+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f6af415ef10 con 0x7f6af4157fe0 2024-03-06T07:33:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.311+0000 7f6aeb400640 1 -- 172.21.15.97:0/836154721 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v348) v1 ==== 134+0+0 (secure 0 0 0) 0x7f6adc05d280 con 0x7f6af4157fe0 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ad006b4d0 msgr2=0x7f6ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ad006b4d0 0x7f6ad006d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4036000 tx=0x7f6ae4008030 comp rx=0 tx=0).stop 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 msgr2=0x7f6af4150b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af4150b40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00cf60 tx=0x7f6adc00cf90 comp rx=0 tx=0).stop 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 shutdown_connections 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ad006b4d0 0x7f6ad006d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6af415e080 0x7f6af40cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6af4157fe0 0x7f6af4150b40 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 --2- 172.21.15.97:0/836154721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6af406eff0 0x7f6af4158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 >> 172.21.15.97:0/836154721 conn(0x7f6af40d0340 msgr2=0x7f6af4154030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 shutdown_connections 2024-03-06T07:33:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.314+0000 7f6afa800640 1 -- 172.21.15.97:0/836154721 wait complete. 2024-03-06T07:33:56.337 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:33:56.337 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:33:56.337 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:33:56.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:56.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.544+0000 7fae2a200640 1 Processor -- start 2024-03-06T07:33:56.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.545+0000 7fae2a200640 1 -- start start 2024-03-06T07:33:56.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.546+0000 7fae2a200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fae24144c00 0x7fae2406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.546+0000 7fae2a200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2406ee70 con 0x7fae2415f9d0 2024-03-06T07:33:56.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.546+0000 7fae2a200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2406cef0 con 0x7fae2415fe60 2024-03-06T07:33:56.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.546+0000 7fae2a200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2406d1d0 con 0x7fae24144c00 2024-03-06T07:33:56.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae1aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fae2415f9d0 0x7fae240ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36766/0 (socket says 172.21.15.97:36766) 2024-03-06T07:33:56.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae1aa00640 1 -- 172.21.15.97:0/2901823570 learned_addr learned my addr 172.21.15.97:0/2901823570 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:33:56.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719698691 0 0) 0x7fae0c001110 con 0x7fae2415f9d0 2024-03-06T07:33:56.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae08003520 con 0x7fae2415f9d0 2024-03-06T07:33:56.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae23400640 1 --2- 172.21.15.97:0/2901823570 >> v2:172.21.15.135:3300/0 conn(0x7fae24144c00 0x7fae2406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350921079 0 0) 0x7fae08003520 con 0x7fae2415f9d0 2024-03-06T07:33:56.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 >> v2:172.21.15.135:3300/0 conn(0x7fae24144c00 msgr2=0x7fae2406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 --2- 172.21.15.97:0/2901823570 >> v2:172.21.15.135:3300/0 conn(0x7fae24144c00 0x7fae2406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 >> v1:172.21.15.135:6789/0 conn(0x7fae2415fe60 legacy=0x7fae240ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2406d4b0 con 0x7fae2415f9d0 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.547+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fae0c001840 con 0x7fae2415f9d0 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 >> v1:172.21.15.97:6789/0 conn(0x7fae2415f9d0 legacy=0x7fae240ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae08003a00 0x7fae08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08004800 0x7fae08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae08005210 0x7fae0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2406d1d0 con 0x7fae08005210 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0800c160 con 0x7fae08003a00 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0800c360 con 0x7fae08004800 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23e00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae08005210 0x7fae0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae1aa00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08004800 0x7fae08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23e00640 1 -- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08004800 msgr2=0x7fae08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23e00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08004800 0x7fae08004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23400640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae08003a00 0x7fae08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23e00640 1 -- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae08003a00 msgr2=0x7fae08003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23e00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae08003a00 0x7fae08003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.548+0000 7fae23e00640 1 -- 172.21.15.97:0/2901823570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae08003520 con 0x7fae08005210 2024-03-06T07:33:56.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.549+0000 7fae23e00640 1 -- 172.21.15.97:0/2901823570 reap_dead start 2024-03-06T07:33:56.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.550+0000 7fae23e00640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae08005210 0x7fae0800b9c0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fae1c001e50 tx=0x7fae08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63236c40c8a3fa14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.550+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c001a70 con 0x7fae08005210 2024-03-06T07:33:56.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.550+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c003b90 con 0x7fae08005210 2024-03-06T07:33:56.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.550+0000 7fae22a00640 1 -- 172.21.15.97:0/2901823570 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0050d0 con 0x7fae08005210 2024-03-06T07:33:56.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.550+0000 7fae2a200640 1 -- 172.21.15.97:0/2901823570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae08005210 msgr2=0x7fae0800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.550+0000 7fae2a200640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae08005210 0x7fae0800b9c0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fae1c001e50 tx=0x7fae08005830 comp rx=0 tx=0).stop 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.551+0000 7fae2a200640 1 -- 172.21.15.97:0/2901823570 shutdown_connections 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.551+0000 7fae2a200640 1 --2- 172.21.15.97:0/2901823570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae08005210 0x7fae0800b9c0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.551+0000 7fae2a200640 1 -- 172.21.15.97:0/2901823570 >> 172.21.15.97:0/2901823570 conn(0x7fae240b1730 msgr2=0x7fae240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.551+0000 7fae2a200640 1 -- 172.21.15.97:0/2901823570 shutdown_connections 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.551+0000 7fae2a200640 1 -- 172.21.15.97:0/2901823570 wait complete. 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.552+0000 7fae2a200640 1 Processor -- start 2024-03-06T07:33:56.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.552+0000 7fae2a200640 1 -- start start 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.553+0000 7fae2a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08005210 0x7fae2413b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.553+0000 7fae2a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae24142e50 0x7fae241432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.553+0000 7fae2a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 0x7fae2415db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.553+0000 7fae2a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae24160730 con 0x7fae2415d6e0 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.553+0000 7fae2a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae24160270 con 0x7fae24142e50 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.553+0000 7fae2a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2406cb50 con 0x7fae08005210 2024-03-06T07:33:56.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 0x7fae2415db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 0x7fae2415db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43706/0 (socket says 172.21.15.97:43706) 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 -- 172.21.15.97:0/2162383619 learned_addr learned my addr 172.21.15.97:0/2162383619 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23400640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08005210 0x7fae2413b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 -- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08005210 msgr2=0x7fae2413b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08005210 0x7fae2413b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 -- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae24142e50 msgr2=0x7fae241432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae24142e50 0x7fae241432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 -- 172.21.15.97:0/2162383619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2413b9d0 con 0x7fae2415d6e0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.554+0000 7fae23e00640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 0x7fae2415db60 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fae1c001f40 tx=0x7fae1c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.555+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c006d50 con 0x7fae2415d6e0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.555+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c039480 con 0x7fae2415d6e0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.555+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c039830 con 0x7fae2415d6e0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.555+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2413c0f0 con 0x7fae2415d6e0 2024-03-06T07:33:56.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.555+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2413c6b0 con 0x7fae2415d6e0 2024-03-06T07:33:56.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.556+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2404f330 con 0x7fae2415d6e0 2024-03-06T07:33:56.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.557+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae1c0399d0 con 0x7fae2415d6e0 2024-03-06T07:33:56.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.558+0000 7fae1be00640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae0006b4d0 0x7fae0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.558+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fae1c0abfe0 con 0x7fae2415d6e0 2024-03-06T07:33:56.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.558+0000 7fae23400640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae0006b4d0 0x7fae0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.559+0000 7fae23400640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae0006b4d0 0x7fae0006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fae14006fd0 tx=0x7fae14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.562+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae2413b9d0 con 0x7fae2415d6e0 2024-03-06T07:33:56.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.757+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae24072890 con 0x7fae2415d6e0 2024-03-06T07:33:56.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.759+0000 7fae1be00640 1 -- 172.21.15.97:0/2162383619 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+13140 (secure 0 0 0) 0x7fae1c033030 con 0x7fae2415d6e0 2024-03-06T07:33:56.762 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:56.762 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":103,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:33:54.296038+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/1028636781":"2024-03-07T07:33:51.275605+0000","172.21.15.97:6835/1028636781":"2024-03-07T07:33:51.275605+0000","172.21.15.97:6837/1818345324":"2024-03-07T07:32:54.749547+0000","172.21.15.97:6836/1818345324":"2024-03-07T07:32:54.749547+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:56.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae0006b4d0 msgr2=0x7fae0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae0006b4d0 0x7fae0006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fae14006fd0 tx=0x7fae14008040 comp rx=0 tx=0).stop 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 msgr2=0x7fae2415db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 0x7fae2415db60 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fae1c001f40 tx=0x7fae1c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 shutdown_connections 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae0006b4d0 0x7fae0006d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae2415d6e0 0x7fae2415db60 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae24142e50 0x7fae241432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 --2- 172.21.15.97:0/2162383619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae08005210 0x7fae2413b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 >> 172.21.15.97:0/2162383619 conn(0x7fae240b1730 msgr2=0x7fae240b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 shutdown_connections 2024-03-06T07:33:56.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.762+0000 7fae2a200640 1 -- 172.21.15.97:0/2162383619 wait complete. 2024-03-06T07:33:56.785 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:56.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:56.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.986+0000 7f61f6a00640 1 Processor -- start 2024-03-06T07:33:56.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.986+0000 7f61f6a00640 1 -- start start 2024-03-06T07:33:56.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.988+0000 7f61f6a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f61f006dcf0 0x7f61f015efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.988+0000 7f61f6a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 0x7f61f006b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.988+0000 7f61f6a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f0071ea0 con 0x7f61f015bb90 2024-03-06T07:33:56.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.988+0000 7f61f6a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f015bf70 con 0x7f61f006dcf0 2024-03-06T07:33:56.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.988+0000 7f61f6a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f015f4e0 con 0x7f61f006afc0 2024-03-06T07:33:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.988+0000 7f61f4a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 0x7f61f006b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61f4a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 0x7f61f006b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:49526/0 (socket says 172.21.15.97:49526) 2024-03-06T07:33:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61f4a00640 1 -- 172.21.15.97:0/1037072302 learned_addr learned my addr 172.21.15.97:0/1037072302 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61ef400640 1 --2- 172.21.15.97:0/1037072302 >> v2:172.21.15.135:3300/0 conn(0x7f61f006dcf0 0x7f61f015efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61f4a00640 1 -- 172.21.15.97:0/1037072302 >> v2:172.21.15.135:3300/0 conn(0x7f61f006dcf0 msgr2=0x7f61f015efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61f4a00640 1 --2- 172.21.15.97:0/1037072302 >> v2:172.21.15.135:3300/0 conn(0x7f61f006dcf0 0x7f61f015efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61f4a00640 1 -- 172.21.15.97:0/1037072302 >> v1:172.21.15.135:6789/0 conn(0x7f61f015bb90 legacy=0x7f61f00cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.989+0000 7f61f4a00640 1 -- 172.21.15.97:0/1037072302 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f006e380 con 0x7f61f006afc0 2024-03-06T07:33:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.991+0000 7f61f4a00640 1 --2- 172.21.15.97:0/1037072302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 0x7f61f006b450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009e80 tx=0x7f61e802bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7318075690b8b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.991+0000 7f61eea00640 1 -- 172.21.15.97:0/1037072302 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e802caf0 con 0x7f61f006afc0 2024-03-06T07:33:56.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.991+0000 7f61eea00640 1 -- 172.21.15.97:0/1037072302 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e802cc90 con 0x7f61f006afc0 2024-03-06T07:33:56.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.991+0000 7f61eea00640 1 -- 172.21.15.97:0/1037072302 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e8034610 con 0x7f61f006afc0 2024-03-06T07:33:56.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 -- 172.21.15.97:0/1037072302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 msgr2=0x7f61f006b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 --2- 172.21.15.97:0/1037072302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 0x7f61f006b450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009e80 tx=0x7f61e802bf50 comp rx=0 tx=0).stop 2024-03-06T07:33:56.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 -- 172.21.15.97:0/1037072302 shutdown_connections 2024-03-06T07:33:56.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 --2- 172.21.15.97:0/1037072302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006afc0 0x7f61f006b450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 --2- 172.21.15.97:0/1037072302 >> v2:172.21.15.135:3300/0 conn(0x7f61f006dcf0 0x7f61f015efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 -- 172.21.15.97:0/1037072302 >> 172.21.15.97:0/1037072302 conn(0x7f61f00b27d0 msgr2=0x7f61f00b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 -- 172.21.15.97:0/1037072302 shutdown_connections 2024-03-06T07:33:56.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.992+0000 7f61f6a00640 1 -- 172.21.15.97:0/1037072302 wait complete. 2024-03-06T07:33:56.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.994+0000 7f61f6a00640 1 Processor -- start 2024-03-06T07:33:56.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.994+0000 7f61f6a00640 1 -- start start 2024-03-06T07:33:56.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 0x7f61f015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f6a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006dcf0 0x7f61f014c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f6a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61f015bb90 0x7f61f014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f6a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f006ec20 con 0x7f61f006afc0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f6a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f006e940 con 0x7f61f015bb90 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f6a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00723a0 con 0x7f61f006dcf0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 0x7f61f015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 0x7f61f015d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43714/0 (socket says 172.21.15.97:43714) 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 -- 172.21.15.97:0/3490603321 learned_addr learned my addr 172.21.15.97:0/3490603321 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61f4a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61f015bb90 0x7f61f014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61ef400640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006dcf0 0x7f61f014c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 -- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006dcf0 msgr2=0x7f61f014c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006dcf0 0x7f61f014c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 -- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61f015bb90 msgr2=0x7f61f014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61f015bb90 0x7f61f014ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.995+0000 7f61efe00640 1 -- 172.21.15.97:0/3490603321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f015cbd0 con 0x7f61f006afc0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.996+0000 7f61efe00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 0x7f61f015d670 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f61e000c400 tx=0x7f61e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.996+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0017070 con 0x7f61f006afc0 2024-03-06T07:33:56.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.996+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e00052d0 con 0x7f61f006afc0 2024-03-06T07:33:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.996+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f00cd7d0 con 0x7f61f006afc0 2024-03-06T07:33:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.996+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0003830 con 0x7f61f006afc0 2024-03-06T07:33:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.996+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f00cdab0 con 0x7f61f006afc0 2024-03-06T07:33:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.997+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f004f330 con 0x7f61f006afc0 2024-03-06T07:33:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.998+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61e001a070 con 0x7f61f006afc0 2024-03-06T07:33:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.999+0000 7f61e7e00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61d006b4d0 0x7f61d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.999+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f61e0089290 con 0x7f61f006afc0 2024-03-06T07:33:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:56.999+0000 7f61ef400640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61d006b4d0 0x7f61d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.000+0000 7f61ef400640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61d006b4d0 0x7f61d006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f61f015b5a0 tx=0x7f61dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.004+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61e00583e0 con 0x7f61f006afc0 2024-03-06T07:33:57.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.201+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61f015e960 con 0x7f61f006afc0 2024-03-06T07:33:57.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.202+0000 7f61e7e00640 1 -- 172.21.15.97:0/3490603321 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61e0087e20 con 0x7f61f006afc0 2024-03-06T07:33:57.204 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.204+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61d006b4d0 msgr2=0x7f61d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.204+0000 7f61f6a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61d006b4d0 0x7f61d006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f61f015b5a0 tx=0x7f61dc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.204+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 msgr2=0x7f61f015d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.204+0000 7f61f6a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 0x7f61f015d670 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f61e000c400 tx=0x7f61e000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 shutdown_connections 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f61d006b4d0 0x7f61d006d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61f015bb90 0x7f61f014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f61f006dcf0 0x7f61f014c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 --2- 172.21.15.97:0/3490603321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f61f006afc0 0x7f61f015d670 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 >> 172.21.15.97:0/3490603321 conn(0x7f61f00b27d0 msgr2=0x7f61f015fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 shutdown_connections 2024-03-06T07:33:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.205+0000 7f61f6a00640 1 -- 172.21.15.97:0/3490603321 wait complete. 2024-03-06T07:33:57.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:33:57.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.425+0000 7f8cb8800640 1 Processor -- start 2024-03-06T07:33:57.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.425+0000 7f8cb8800640 1 -- start start 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb8800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8cb015acb0 0x7f8cb00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb8800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb015b090 con 0x7f8cb00b4680 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb8800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb00b2a30 con 0x7f8cb006ec00 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb8800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb00b4360 con 0x7f8cb015acb0 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8cb015acb0 0x7f8cb00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8cb015acb0 0x7f8cb00b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43716/0 (socket says 172.21.15.97:43716) 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8cb00b4680 0x7f8cb006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36778/0 (socket says 172.21.15.97:36778) 2024-03-06T07:33:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5e00640 1 -- 172.21.15.97:0/2319049005 learned_addr learned my addr 172.21.15.97:0/2319049005 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:57.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5e00640 1 -- 172.21.15.97:0/2319049005 >> v1:172.21.15.135:6789/0 conn(0x7f8cb006ec00 legacy=0x7f8cb00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5e00640 1 -- 172.21.15.97:0/2319049005 >> v1:172.21.15.97:6789/0 conn(0x7f8cb00b4680 legacy=0x7f8cb006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.427+0000 7f8cb5e00640 1 -- 172.21.15.97:0/2319049005 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb00b2cf0 con 0x7f8cb015acb0 2024-03-06T07:33:57.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.428+0000 7f8cb5e00640 1 --2- 172.21.15.97:0/2319049005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb00b37d0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009e50 tx=0x7f8ca802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f44328b8fd3033 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb4a00640 1 -- 172.21.15.97:0/2319049005 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca802cb10 con 0x7f8cb015acb0 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb4a00640 1 -- 172.21.15.97:0/2319049005 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca802ccb0 con 0x7f8cb015acb0 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb4a00640 1 -- 172.21.15.97:0/2319049005 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca802ce60 con 0x7f8cb015acb0 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb8800640 1 -- 172.21.15.97:0/2319049005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 msgr2=0x7f8cb00b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2319049005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb00b37d0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009e50 tx=0x7f8ca802bf70 comp rx=0 tx=0).stop 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb8800640 1 -- 172.21.15.97:0/2319049005 shutdown_connections 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2319049005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb00b37d0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb8800640 1 -- 172.21.15.97:0/2319049005 >> 172.21.15.97:0/2319049005 conn(0x7f8cb00b1730 msgr2=0x7f8cb00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.429+0000 7f8cb8800640 1 -- 172.21.15.97:0/2319049005 shutdown_connections 2024-03-06T07:33:57.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.430+0000 7f8cb8800640 1 -- 172.21.15.97:0/2319049005 wait complete. 2024-03-06T07:33:57.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.431+0000 7f8cb8800640 1 Processor -- start 2024-03-06T07:33:57.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.431+0000 7f8cb8800640 1 -- start start 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb8800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cb006ec00 0x7f8cb0155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb8800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cb00b4680 0x7f8cb00ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb8800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb01460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb8800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b3d10 con 0x7f8cb015acb0 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb8800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b32b0 con 0x7f8cb006ec00 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb8800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b4010 con 0x7f8cb00b4680 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb01460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb01460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43718/0 (socket says 172.21.15.97:43718) 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.432+0000 7f8cb6800640 1 -- 172.21.15.97:0/2882524905 learned_addr learned my addr 172.21.15.97:0/2882524905 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb5e00640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cb006ec00 0x7f8cb0155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb5400640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cb00b4680 0x7f8cb00ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb6800640 1 -- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cb00b4680 msgr2=0x7f8cb00ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb6800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cb00b4680 0x7f8cb00ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb6800640 1 -- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cb006ec00 msgr2=0x7f8cb0155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb6800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cb006ec00 0x7f8cb0155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb6800640 1 -- 172.21.15.97:0/2882524905 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb00c9190 con 0x7f8cb015acb0 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb6800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb01460a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00b410 tx=0x7f8cac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac015030 con 0x7f8cb015acb0 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac004020 con 0x7f8cb015acb0 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb00c94d0 con 0x7f8cb015acb0 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.433+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004410 con 0x7f8cb015acb0 2024-03-06T07:33:57.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.434+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb00c9d10 con 0x7f8cb015acb0 2024-03-06T07:33:57.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.435+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb00c9f10 con 0x7f8cb015acb0 2024-03-06T07:33:57.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.436+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cac018070 con 0x7f8cb015acb0 2024-03-06T07:33:57.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.436+0000 7f8ca6000640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c9406b4d0 0x7f8c9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.437+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8cac089b10 con 0x7f8cb015acb0 2024-03-06T07:33:57.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.437+0000 7f8cb5e00640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c9406b4d0 0x7f8c9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.437+0000 7f8cb5e00640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c9406b4d0 0x7f8c9406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8035000 tx=0x7f8ca8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.441+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cb00c9f10 con 0x7f8cb015acb0 2024-03-06T07:33:57.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:57.688+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f8cb00c96b0 con 0x7f8cb015acb0 2024-03-06T07:33:58.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.323+0000 7f8ca6000640 1 -- 172.21.15.97:0/2882524905 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v350) v1 ==== 145+0+0 (secure 0 0 0) 0x7f8cb00c94d0 con 0x7f8cb015acb0 2024-03-06T07:33:58.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c9406b4d0 msgr2=0x7f8c9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c9406b4d0 0x7f8c9406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8035000 tx=0x7f8ca8030000 comp rx=0 tx=0).stop 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 msgr2=0x7f8cb01460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb01460a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00b410 tx=0x7f8cac00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 shutdown_connections 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c9406b4d0 0x7f8c9406d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cb015acb0 0x7f8cb01460a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8cb00b4680 0x7f8cb00ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 --2- 172.21.15.97:0/2882524905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cb006ec00 0x7f8cb0155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 >> 172.21.15.97:0/2882524905 conn(0x7f8cb00b1730 msgr2=0x7f8cb006d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 shutdown_connections 2024-03-06T07:33:58.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.327+0000 7f8cb8800640 1 -- 172.21.15.97:0/2882524905 wait complete. 2024-03-06T07:33:58.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.555+0000 7f3ac8000640 1 Processor -- start 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.555+0000 7f3ac8000640 1 -- start start 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.556+0000 7f3ac8000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac015fc10 0x7f3ac0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.556+0000 7f3ac8000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3ac015fff0 0x7f3ac0160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.556+0000 7f3ac8000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac00b46c0 con 0x7f3ac00ce4a0 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.556+0000 7f3ac8000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac00b49a0 con 0x7f3ac015fff0 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.556+0000 7f3ac8000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0072c90 con 0x7f3ac015fc10 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3ac015fff0 0x7f3ac0160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3ac015fff0 0x7f3ac0160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43734/0 (socket says 172.21.15.97:43734) 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 -- 172.21.15.97:0/2182588388 learned_addr learned my addr 172.21.15.97:0/2182588388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac6000640 1 --1- 172.21.15.97:0/2182588388 >> v1:172.21.15.97:6789/0 conn(0x7f3ac00ce4a0 0x7f3ac00ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36788/0 (socket says 172.21.15.97:36788) 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac5600640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac015fc10 0x7f3ac0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 -- 172.21.15.97:0/2182588388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac015fc10 msgr2=0x7f3ac0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac015fc10 0x7f3ac0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 -- 172.21.15.97:0/2182588388 >> v1:172.21.15.97:6789/0 conn(0x7f3ac00ce4a0 legacy=0x7f3ac00ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac4c00640 1 -- 172.21.15.97:0/2182588388 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0160990 con 0x7f3ac015fff0 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3abfe00640 1 -- 172.21.15.97:0/2182588388 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511415384 0 0) 0x7f3ac00b46c0 con 0x7f3ac00ce4a0 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.557+0000 7f3ac5600640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac015fc10 0x7f3ac0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.558+0000 7f3ac4c00640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fff0 0x7f3ac0160450 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009870 tx=0x7f3aac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ad19de7a46aaa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3abfe00640 1 -- 172.21.15.97:0/2182588388 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac034070 con 0x7f3ac015fff0 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3abfe00640 1 -- 172.21.15.97:0/2182588388 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac030840 con 0x7f3ac015fff0 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3abfe00640 1 -- 172.21.15.97:0/2182588388 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac030c10 con 0x7f3ac015fff0 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3ac8000640 1 -- 172.21.15.97:0/2182588388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fff0 msgr2=0x7f3ac0160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3ac8000640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fff0 0x7f3ac0160450 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009870 tx=0x7f3aac02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3ac8000640 1 -- 172.21.15.97:0/2182588388 shutdown_connections 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3ac8000640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fff0 0x7f3ac0160450 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3ac8000640 1 --2- 172.21.15.97:0/2182588388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac015fc10 0x7f3ac0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.559+0000 7f3ac8000640 1 -- 172.21.15.97:0/2182588388 >> 172.21.15.97:0/2182588388 conn(0x7f3ac00b1940 msgr2=0x7f3ac00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.560+0000 7f3ac8000640 1 -- 172.21.15.97:0/2182588388 shutdown_connections 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.560+0000 7f3ac8000640 1 -- 172.21.15.97:0/2182588388 wait complete. 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.561+0000 7f3ac8000640 1 Processor -- start 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.561+0000 7f3ac8000640 1 -- start start 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.562+0000 7f3ac8000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac00ce4a0 0x7f3ac015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.562+0000 7f3ac8000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 0x7f3ac0144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.562+0000 7f3ac8000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac015fff0 0x7f3ac009d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.562+0000 7f3ac8000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0072a30 con 0x7f3ac015fc10 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.562+0000 7f3ac8000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00728b0 con 0x7f3ac015fff0 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.562+0000 7f3ac8000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac015c2e0 con 0x7f3ac00ce4a0 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 0x7f3ac0144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 0x7f3ac0144500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43736/0 (socket says 172.21.15.97:43736) 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1765703265 learned_addr learned my addr 172.21.15.97:0/1765703265 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac00ce4a0 msgr2=0x7f3ac015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac5600640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac00ce4a0 0x7f3ac015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac00ce4a0 0x7f3ac015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac015fff0 msgr2=0x7f3ac009d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac6000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac015fff0 0x7f3ac009d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac015fff0 0x7f3ac009d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.563+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1765703265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac009d850 con 0x7f3ac015fc10 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.564+0000 7f3ac4c00640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 0x7f3ac0144500 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3aac000c00 tx=0x7f3aac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.564+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac034040 con 0x7f3ac015fc10 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.564+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac039d90 con 0x7f3ac015fc10 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.564+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac015e1d0 con 0x7f3ac015fc10 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.564+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac043610 con 0x7f3ac015fc10 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.564+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac015e790 con 0x7f3ac015fc10 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.565+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac004f330 con 0x7f3ac015fc10 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.566+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3aac04a060 con 0x7f3ac015fc10 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.567+0000 7f3abd600640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa006b4d0 0x7f3aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.567+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3aac0ac110 con 0x7f3ac015fc10 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.567+0000 7f3ac5600640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa006b4d0 0x7f3aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.568+0000 7f3ac5600640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa006b4d0 0x7f3aa006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8008850 tx=0x7f3ab8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.572+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3aac03e070 con 0x7f3ac015fc10 2024-03-06T07:33:58.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.769+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3ac00b2d40 con 0x7f3ac015fc10 2024-03-06T07:33:58.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.775+0000 7f3abd600640 1 -- 172.21.15.97:0/1765703265 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3aac02d330 con 0x7f3ac015fc10 2024-03-06T07:33:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.777+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa006b4d0 msgr2=0x7f3aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.777+0000 7f3ac8000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa006b4d0 0x7f3aa006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8008850 tx=0x7f3ab8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.777+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 msgr2=0x7f3ac0144500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.777+0000 7f3ac8000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 0x7f3ac0144500 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3aac000c00 tx=0x7f3aac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 shutdown_connections 2024-03-06T07:33:58.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3aa006b4d0 0x7f3aa006d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ac015fff0 0x7f3ac009d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ac015fc10 0x7f3ac0144500 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 --2- 172.21.15.97:0/1765703265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3ac00ce4a0 0x7f3ac015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 >> 172.21.15.97:0/1765703265 conn(0x7f3ac00b1940 msgr2=0x7f3ac00b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 shutdown_connections 2024-03-06T07:33:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:58.778+0000 7f3ac8000640 1 -- 172.21.15.97:0/1765703265 wait complete. 2024-03-06T07:33:58.798 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:58.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:58.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:59.006 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:59.006 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:33:59.006 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:59.059 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:59.060 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:33:59.060 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:59.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:59.116 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:33:59.116 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:59.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:59.173 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:33:59.173 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:59.229 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:59.229 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:33:59.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.445+0000 7f0548a00640 1 Processor -- start 2024-03-06T07:33:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.445+0000 7f0548a00640 1 -- start start 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.446+0000 7f0548a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0544144c00 0x7f054406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.446+0000 7f0548a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f054415f9d0 0x7f054415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.446+0000 7f0548a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054406ee70 con 0x7f05440ce0a0 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0548a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054406cef0 con 0x7f0544144c00 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0548a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054406d1d0 con 0x7f054415f9d0 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0544144c00 0x7f054406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0544144c00 0x7f054406e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43744/0 (socket says 172.21.15.97:43744) 2024-03-06T07:33:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 -- 172.21.15.97:0/539418149 learned_addr learned my addr 172.21.15.97:0/539418149 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 -- 172.21.15.97:0/539418149 >> v2:172.21.15.135:3300/0 conn(0x7f054415f9d0 msgr2=0x7f054415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 --2- 172.21.15.97:0/539418149 >> v2:172.21.15.135:3300/0 conn(0x7f054415f9d0 0x7f054415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 -- 172.21.15.97:0/539418149 >> v1:172.21.15.135:6789/0 conn(0x7f05440ce0a0 legacy=0x7f05440ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.447+0000 7f0541e00640 1 -- 172.21.15.97:0/539418149 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054406d4b0 con 0x7f0544144c00 2024-03-06T07:33:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.448+0000 7f0541e00640 1 --2- 172.21.15.97:0/539418149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f054406e930 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f05440b26f0 tx=0x7f053402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af89759b94ed52b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0540a00640 1 -- 172.21.15.97:0/539418149 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534035020 con 0x7f0544144c00 2024-03-06T07:33:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0540a00640 1 -- 172.21.15.97:0/539418149 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534031870 con 0x7f0544144c00 2024-03-06T07:33:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0540a00640 1 -- 172.21.15.97:0/539418149 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534031c40 con 0x7f0544144c00 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 -- 172.21.15.97:0/539418149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 msgr2=0x7f054406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 --2- 172.21.15.97:0/539418149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f054406e930 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f05440b26f0 tx=0x7f053402b910 comp rx=0 tx=0).stop 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 -- 172.21.15.97:0/539418149 shutdown_connections 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 --2- 172.21.15.97:0/539418149 >> v2:172.21.15.135:3300/0 conn(0x7f054415f9d0 0x7f054415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 --2- 172.21.15.97:0/539418149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f054406e930 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 -- 172.21.15.97:0/539418149 >> 172.21.15.97:0/539418149 conn(0x7f05440b1730 msgr2=0x7f05440b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 -- 172.21.15.97:0/539418149 shutdown_connections 2024-03-06T07:33:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.449+0000 7f0548a00640 1 -- 172.21.15.97:0/539418149 wait complete. 2024-03-06T07:33:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.451+0000 7f0548a00640 1 Processor -- start 2024-03-06T07:33:59.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.452+0000 7f0548a00640 1 -- start start 2024-03-06T07:33:59.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0548a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05440ce0a0 0x7f054415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0548a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f0544134de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0548a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054415f9d0 0x7f05440ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0548a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054406cb50 con 0x7f0544144c00 2024-03-06T07:33:59.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0548a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544160730 con 0x7f05440ce0a0 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0548a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054406ccd0 con 0x7f054415f9d0 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f0544134de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f0544134de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43760/0 (socket says 172.21.15.97:43760) 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541400640 1 -- 172.21.15.97:0/1062933182 learned_addr learned my addr 172.21.15.97:0/1062933182 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0542800640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054415f9d0 0x7f05440ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541e00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05440ce0a0 0x7f054415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541400640 1 -- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054415f9d0 msgr2=0x7f05440ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541400640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054415f9d0 0x7f05440ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.453+0000 7f0541400640 1 -- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05440ce0a0 msgr2=0x7f054415b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f0541400640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05440ce0a0 0x7f054415b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f0541400640 1 -- 172.21.15.97:0/1062933182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054415bcb0 con 0x7f0544144c00 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f0541e00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05440ce0a0 0x7f054415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f0541400640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f0544134de0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f052c00b4a0 tx=0x7f052c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c014070 con 0x7f0544144c00 2024-03-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c010610 con 0x7f0544144c00 2024-03-06T07:33:59.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054415f0b0 con 0x7f0544144c00 2024-03-06T07:33:59.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c010a00 con 0x7f0544144c00 2024-03-06T07:33:59.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.454+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054415f670 con 0x7f0544144c00 2024-03-06T07:33:59.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.455+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054404f330 con 0x7f0544144c00 2024-03-06T07:33:59.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.456+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f052c017070 con 0x7f0544144c00 2024-03-06T07:33:59.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.457+0000 7f053a000640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052006b4d0 0x7f052006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.457+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f052c0899f0 con 0x7f0544144c00 2024-03-06T07:33:59.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.457+0000 7f0541e00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052006b4d0 0x7f052006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.458+0000 7f0541e00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052006b4d0 0x7f052006d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0534036000 tx=0x7f053403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.461+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f054415bcb0 con 0x7f0544144c00 2024-03-06T07:33:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.701+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0544072890 con 0x7f0544144c00 2024-03-06T07:33:59.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.701+0000 7f053a000640 1 -- 172.21.15.97:0/1062933182 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f052c058a70 con 0x7f0544144c00 2024-03-06T07:33:59.704 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:33:59.704 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/708356014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":708356014},{"type":"v1","addr":"172.21.15.135:6835","nonce":708356014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/2207650617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2207650617},{"type":"v1","addr":"172.21.15.135:6833","nonce":2207650617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/3599265582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":3599265582},{"type":"v1","addr":"172.21.15.97:6836","nonce":3599265582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7623,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3226305941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3226305941},{"type":"v1","addr":"172.21.15.97:6837","nonce":3226305941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[]} 2024-03-06T07:33:59.704 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 69 2024-03-06T07:33:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052006b4d0 msgr2=0x7f052006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052006b4d0 0x7f052006d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0534036000 tx=0x7f053403c040 comp rx=0 tx=0).stop 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 msgr2=0x7f0544134de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f0544134de0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f052c00b4a0 tx=0x7f052c00b960 comp rx=0 tx=0).stop 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 shutdown_connections 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052006b4d0 0x7f052006d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054415f9d0 0x7f05440ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0544144c00 0x7f0544134de0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 --2- 172.21.15.97:0/1062933182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05440ce0a0 0x7f054415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 >> 172.21.15.97:0/1062933182 conn(0x7f05440b1730 msgr2=0x7f05440b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 shutdown_connections 2024-03-06T07:33:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.704+0000 7f0548a00640 1 -- 172.21.15.97:0/1062933182 wait complete. 2024-03-06T07:33:59.727 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:33:59.727 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:33:59.728 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:59.728 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:59.728 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:33:59.728 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:33:59.728 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:59.728 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:59.728 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:33:59.729 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:33:59.729 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:33:59.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:59.729 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:33:59.729 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:33:59.729 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:33:59.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:59.744 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:59.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:33:59.748 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:33:59.748 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:33:59.749 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:59.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:33:59.762 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:33:59.762 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:33:59.796 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:33:59.797 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:59.803 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:33:59.803 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:59.809 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:33:59.809 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:59.815 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:33:59.815 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:59.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:33:59.952 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:33:59.954 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:33:59.968 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:33:59.970 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:34:00.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.998+0000 7f5d9b800640 1 Processor -- start 2024-03-06T07:34:00.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.998+0000 7f5d9b800640 1 -- start start 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.999+0000 7f5d9b800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5d9415e360 0x7f5d940cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.999+0000 7f5d9b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 0x7f5d94161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.999+0000 7f5d9b800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d940cf2b0 con 0x7f5d94162270 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.999+0000 7f5d9b800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d940cf590 con 0x7f5d9415e360 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:33:59.999+0000 7f5d9b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940cf870 con 0x7f5d940d12e0 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d98e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5d9415e360 0x7f5d940cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d8be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 0x7f5d94161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d99800640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f5d94162270 0x7f5d9415e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:51674/0 (socket says 172.21.15.97:51674) 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d98e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5d9415e360 0x7f5d940cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:37078/0 (socket says 172.21.15.97:37078) 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d99800640 1 -- 172.21.15.97:0/313377159 learned_addr learned my addr 172.21.15.97:0/313377159 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:00.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d8be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 0x7f5d94161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:49608/0 (socket says 172.21.15.97:49608) 2024-03-06T07:34:00.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d8be00640 1 -- 172.21.15.97:0/313377159 >> v2:172.21.15.135:3300/0 conn(0x7f5d9415e360 msgr2=0x7f5d940cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d8be00640 1 --2- 172.21.15.97:0/313377159 >> v2:172.21.15.135:3300/0 conn(0x7f5d9415e360 0x7f5d940cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d8be00640 1 -- 172.21.15.97:0/313377159 >> v1:172.21.15.135:6789/0 conn(0x7f5d94162270 legacy=0x7f5d9415e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.000+0000 7f5d8be00640 1 -- 172.21.15.97:0/313377159 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940cfb50 con 0x7f5d940d12e0 2024-03-06T07:34:00.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.001+0000 7f5d8be00640 1 --2- 172.21.15.97:0/313377159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 0x7f5d94161d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c0097a0 tx=0x7f5d7c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a418d4a824149cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.001+0000 7f5d93e00640 1 -- 172.21.15.97:0/313377159 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c034070 con 0x7f5d940d12e0 2024-03-06T07:34:00.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d93e00640 1 -- 172.21.15.97:0/313377159 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7c030610 con 0x7f5d940d12e0 2024-03-06T07:34:00.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d93e00640 1 -- 172.21.15.97:0/313377159 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c0309e0 con 0x7f5d940d12e0 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 -- 172.21.15.97:0/313377159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 msgr2=0x7f5d94161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 --2- 172.21.15.97:0/313377159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 0x7f5d94161d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c0097a0 tx=0x7f5d7c02b960 comp rx=0 tx=0).stop 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 -- 172.21.15.97:0/313377159 shutdown_connections 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 --2- 172.21.15.97:0/313377159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d940d12e0 0x7f5d94161d30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 --2- 172.21.15.97:0/313377159 >> v2:172.21.15.135:3300/0 conn(0x7f5d9415e360 0x7f5d940cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 -- 172.21.15.97:0/313377159 >> 172.21.15.97:0/313377159 conn(0x7f5d940d3f50 msgr2=0x7f5d940d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 -- 172.21.15.97:0/313377159 shutdown_connections 2024-03-06T07:34:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.002+0000 7f5d9b800640 1 -- 172.21.15.97:0/313377159 wait complete. 2024-03-06T07:34:00.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.004+0000 7f5d9b800640 1 Processor -- start 2024-03-06T07:34:00.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.004+0000 7f5d9b800640 1 -- start start 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d9b800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 0x7f5d940b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d9b800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d9415e360 0x7f5d94146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d9b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d94162270 0x7f5d94147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d9b800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940d18c0 con 0x7f5d940d12e0 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d9b800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940d1740 con 0x7f5d9415e360 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d9b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940d2ca0 con 0x7f5d94162270 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d98e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 0x7f5d940b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d98e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 0x7f5d940b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43794/0 (socket says 172.21.15.97:43794) 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d98e00640 1 -- 172.21.15.97:0/846776990 learned_addr learned my addr 172.21.15.97:0/846776990 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:00.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d8be00640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d9415e360 0x7f5d94146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.005+0000 7f5d99800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d94162270 0x7f5d94147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d98e00640 1 -- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d94162270 msgr2=0x7f5d94147330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d98e00640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d94162270 0x7f5d94147330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d98e00640 1 -- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d9415e360 msgr2=0x7f5d94146df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d98e00640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d9415e360 0x7f5d94146df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d98e00640 1 -- 172.21.15.97:0/846776990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940b5110 con 0x7f5d940d12e0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d98e00640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 0x7f5d940b6de0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400e9b0 tx=0x7f5d8400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d84017070 con 0x7f5d940d12e0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9415eed0 con 0x7f5d940d12e0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9415f490 con 0x7f5d940d12e0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8400c9a0 con 0x7f5d940d12e0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.006+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8400cd90 con 0x7f5d940d12e0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.008+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c005290 con 0x7f5d940d12e0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.009+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d8401a070 con 0x7f5d940d12e0 2024-03-06T07:34:00.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.010+0000 7f5d91600640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d7006b4d0 0x7f5d7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.010+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f5d84089eb0 con 0x7f5d940d12e0 2024-03-06T07:34:00.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.010+0000 7f5d8be00640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d7006b4d0 0x7f5d7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.011+0000 7f5d8be00640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d7006b4d0 0x7f5d7006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c036000 tx=0x7f5d7c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.014+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d940b5110 con 0x7f5d940d12e0 2024-03-06T07:34:00.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:00.218+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5d6c005580 con 0x7f5d940d12e0 2024-03-06T07:34:01.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.359+0000 7f5d91600640 1 -- 172.21.15.97:0/846776990 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v108) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5d84058970 con 0x7f5d940d12e0 2024-03-06T07:34:01.362 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d7006b4d0 msgr2=0x7f5d7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d7006b4d0 0x7f5d7006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c036000 tx=0x7f5d7c03b040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 msgr2=0x7f5d940b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 0x7f5d940b6de0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400e9b0 tx=0x7f5d8400ee70 comp rx=0 tx=0).stop 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 shutdown_connections 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d7006b4d0 0x7f5d7006d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d94162270 0x7f5d94147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d9415e360 0x7f5d94146df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 --2- 172.21.15.97:0/846776990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d940d12e0 0x7f5d940b6de0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.364+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 >> 172.21.15.97:0/846776990 conn(0x7f5d940d3f50 msgr2=0x7f5d9415c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.365+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 shutdown_connections 2024-03-06T07:34:01.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.365+0000 7f5d9b800640 1 -- 172.21.15.97:0/846776990 wait complete. 2024-03-06T07:34:01.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:34:01.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.586+0000 7fdd7b200640 1 Processor -- start 2024-03-06T07:34:01.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.586+0000 7fdd7b200640 1 -- start start 2024-03-06T07:34:01.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd7b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdd7415eff0 0x7fdd74070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd7b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd74071d30 con 0x7fdd740af1c0 2024-03-06T07:34:01.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd7b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7415f4a0 con 0x7fdd7406e4d0 2024-03-06T07:34:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd7b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd74071260 con 0x7fdd7415eff0 2024-03-06T07:34:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd6b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdd7415eff0 0x7fdd74070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd73e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdd740af1c0 0x7fdd7406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36802/0 (socket says 172.21.15.97:36802) 2024-03-06T07:34:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.588+0000 7fdd73e00640 1 -- 172.21.15.97:0/165106147 learned_addr learned my addr 172.21.15.97:0/165106147 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.589+0000 7fdd6b400640 1 -- 172.21.15.97:0/165106147 >> v1:172.21.15.135:6789/0 conn(0x7fdd7406e4d0 legacy=0x7fdd740b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.589+0000 7fdd6b400640 1 -- 172.21.15.97:0/165106147 >> v1:172.21.15.97:6789/0 conn(0x7fdd740af1c0 legacy=0x7fdd7406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.589+0000 7fdd6b400640 1 -- 172.21.15.97:0/165106147 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74071490 con 0x7fdd7415eff0 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.590+0000 7fdd6b400640 1 --2- 172.21.15.97:0/165106147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7415eff0 0x7fdd74070d20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0097d0 tx=0x7fdd5c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e133ddbaa39f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.590+0000 7fdd73400640 1 -- 172.21.15.97:0/165106147 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c035020 con 0x7fdd7415eff0 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.590+0000 7fdd73400640 1 -- 172.21.15.97:0/165106147 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c031810 con 0x7fdd7415eff0 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.590+0000 7fdd73400640 1 -- 172.21.15.97:0/165106147 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c031be0 con 0x7fdd7415eff0 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 -- 172.21.15.97:0/165106147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7415eff0 msgr2=0x7fdd74070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 --2- 172.21.15.97:0/165106147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7415eff0 0x7fdd74070d20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0097d0 tx=0x7fdd5c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 -- 172.21.15.97:0/165106147 shutdown_connections 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 --2- 172.21.15.97:0/165106147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd7415eff0 0x7fdd74070d20 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 -- 172.21.15.97:0/165106147 >> 172.21.15.97:0/165106147 conn(0x7fdd740b1730 msgr2=0x7fdd740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 -- 172.21.15.97:0/165106147 shutdown_connections 2024-03-06T07:34:01.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.591+0000 7fdd7b200640 1 -- 172.21.15.97:0/165106147 wait complete. 2024-03-06T07:34:01.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.592+0000 7fdd7b200640 1 Processor -- start 2024-03-06T07:34:01.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.593+0000 7fdd7b200640 1 -- start start 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.593+0000 7fdd7b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7406e4d0 0x7fdd74145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd7b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 0x7fdd740ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd7b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd7415eff0 0x7fdd740adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd7b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740b30b0 con 0x7fdd740af1c0 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd7b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74071a50 con 0x7fdd7415eff0 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd7b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7406d370 con 0x7fdd7406e4d0 2024-03-06T07:34:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 0x7fdd740ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 0x7fdd740ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43824/0 (socket says 172.21.15.97:43824) 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 -- 172.21.15.97:0/47357126 learned_addr learned my addr 172.21.15.97:0/47357126 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 -- 172.21.15.97:0/47357126 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7406e4d0 msgr2=0x7fdd74145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7406e4d0 0x7fdd74145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 -- 172.21.15.97:0/47357126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd7415eff0 msgr2=0x7fdd740adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd7415eff0 0x7fdd740adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.594+0000 7fdd6b400640 1 -- 172.21.15.97:0/47357126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74148680 con 0x7fdd740af1c0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.595+0000 7fdd6b400640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 0x7fdd740ad860 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c034040 tx=0x7fdd5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.595+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0453f0 con 0x7fdd740af1c0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.595+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c0455b0 con 0x7fdd740af1c0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.595+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0459a0 con 0x7fdd740af1c0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.595+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd741457a0 con 0x7fdd740af1c0 2024-03-06T07:34:01.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.595+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd74145d60 con 0x7fdd740af1c0 2024-03-06T07:34:01.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.596+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7404f330 con 0x7fdd740af1c0 2024-03-06T07:34:01.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.597+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd5c04c060 con 0x7fdd740af1c0 2024-03-06T07:34:01.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.598+0000 7fdd70c00640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5006b4d0 0x7fdd5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.598+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd5c0abef0 con 0x7fdd740af1c0 2024-03-06T07:34:01.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.598+0000 7fdd73e00640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5006b4d0 0x7fdd5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.599+0000 7fdd73e00640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5006b4d0 0x7fdd5006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdd64000f60 tx=0x7fdd64007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.602+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd5c040070 con 0x7fdd740af1c0 2024-03-06T07:34:01.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.846+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdd7406ebc0 con 0x7fdd740af1c0 2024-03-06T07:34:01.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.847+0000 7fdd70c00640 1 -- 172.21.15.97:0/47357126 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fdd7415f4a0 con 0x7fdd740af1c0 2024-03-06T07:34:01.849 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:01.849 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:34:01.849 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:34:01.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.850 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:34:01.851 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.852 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.853 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.854 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.855 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.856 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:34:01.857 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:34:01.859 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:34:01.860 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.861 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:34:01.862 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:34:01.863 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:34:01.863 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.863 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:01.863 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:34:01.863 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5006b4d0 msgr2=0x7fdd5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5006b4d0 0x7fdd5006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdd64000f60 tx=0x7fdd64007a80 comp rx=0 tx=0).stop 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 msgr2=0x7fdd740ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 0x7fdd740ad860 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c034040 tx=0x7fdd5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 shutdown_connections 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd5006b4d0 0x7fdd5006d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd7415eff0 0x7fdd740adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd740af1c0 0x7fdd740ad860 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 --2- 172.21.15.97:0/47357126 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdd7406e4d0 0x7fdd74145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.849+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 >> 172.21.15.97:0/47357126 conn(0x7fdd740b1730 msgr2=0x7fdd74160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.850+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 shutdown_connections 2024-03-06T07:34:01.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:01.850+0000 7fdd7b200640 1 -- 172.21.15.97:0/47357126 wait complete. 2024-03-06T07:34:01.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:02.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.070+0000 7ff083e00640 1 Processor -- start 2024-03-06T07:34:02.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.071+0000 7ff083e00640 1 -- start start 2024-03-06T07:34:02.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.072+0000 7ff083e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 0x7ff0840cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.072+0000 7ff083e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0840b2b30 con 0x7ff0840cf370 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.072+0000 7ff083e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0840b2e10 con 0x7ff08415a940 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.072+0000 7ff083e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840b28e0 con 0x7ff0840cc2a0 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff07be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff0840cf370 0x7ff08406d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36804/0 (socket says 172.21.15.97:36804) 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff07be00640 1 -- 172.21.15.97:0/1405333511 learned_addr learned my addr 172.21.15.97:0/1405333511 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff083400640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 0x7ff0840cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053225445 0 0) 0x7ff0840b2b30 con 0x7ff0840cf370 2024-03-06T07:34:02.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff068003520 con 0x7ff0840cf370 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2235811233 0 0) 0x7ff068003520 con 0x7ff0840cf370 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 msgr2=0x7ff0840cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 0x7ff0840cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 >> v1:172.21.15.135:6789/0 conn(0x7ff08415a940 legacy=0x7ff0840b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.073+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08406ca50 con 0x7ff0840cf370 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff06c001670 con 0x7ff0840cf370 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 >> v1:172.21.15.97:6789/0 conn(0x7ff0840cf370 legacy=0x7ff08406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0680039e0 0x7ff068003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 0x7ff068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff068004d40 0x7ff0680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840b28e0 con 0x7ff068004d40 2024-03-06T07:34:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06800ba70 con 0x7ff0680039e0 2024-03-06T07:34:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06800bc70 con 0x7ff0840cc2a0 2024-03-06T07:34:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff083400640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff068004d40 0x7ff0680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff07be00640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 0x7ff068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082a00640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0680039e0 0x7ff068003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff083400640 1 -- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 msgr2=0x7ff068004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff083400640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff0840cc2a0 0x7ff068004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff083400640 1 -- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0680039e0 msgr2=0x7ff068003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff083400640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0680039e0 0x7ff068003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff083400640 1 -- 172.21.15.97:0/1405333511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068003520 con 0x7ff068004d40 2024-03-06T07:34:02.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.074+0000 7ff082a00640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0680039e0 0x7ff068003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.075+0000 7ff083400640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff068004d40 0x7ff0680051c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009f90 tx=0x7ff068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19f3d4f346adf67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c039070 con 0x7ff068004d40 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c02cbd0 con 0x7ff068004d40 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff082000640 1 -- 172.21.15.97:0/1405333511 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c009c10 con 0x7ff068004d40 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff083e00640 1 -- 172.21.15.97:0/1405333511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff068004d40 msgr2=0x7ff0680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff083e00640 1 --2- 172.21.15.97:0/1405333511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff068004d40 0x7ff0680051c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009f90 tx=0x7ff068005830 comp rx=0 tx=0).stop 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff083400640 1 -- 172.21.15.97:0/1405333511 reap_dead start 2024-03-06T07:34:02.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff083e00640 1 -- 172.21.15.97:0/1405333511 shutdown_connections 2024-03-06T07:34:02.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff083e00640 1 -- 172.21.15.97:0/1405333511 >> 172.21.15.97:0/1405333511 conn(0x7ff08406afc0 msgr2=0x7ff08406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.076+0000 7ff083e00640 1 -- 172.21.15.97:0/1405333511 shutdown_connections 2024-03-06T07:34:02.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.077+0000 7ff083e00640 1 -- 172.21.15.97:0/1405333511 wait complete. 2024-03-06T07:34:02.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.078+0000 7ff083e00640 1 Processor -- start 2024-03-06T07:34:02.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.078+0000 7ff083e00640 1 -- start start 2024-03-06T07:34:02.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.079+0000 7ff083e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff068004d40 0x7ff084152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.079+0000 7ff083e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff08414e410 0x7ff08414e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.079+0000 7ff083e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 0x7ff0841f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.079+0000 7ff083e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08415ad20 con 0x7ff0841f34e0 2024-03-06T07:34:02.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.079+0000 7ff083e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840b30f0 con 0x7ff08414e410 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.079+0000 7ff083e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08415aea0 con 0x7ff068004d40 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 0x7ff0841f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 0x7ff0841f3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43856/0 (socket says 172.21.15.97:43856) 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 -- 172.21.15.97:0/1749295199 learned_addr learned my addr 172.21.15.97:0/1749295199 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff082a00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff068004d40 0x7ff084152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 -- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff068004d40 msgr2=0x7ff084152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff068004d40 0x7ff084152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 -- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff08414e410 msgr2=0x7ff08414e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff07be00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff08414e410 0x7ff08414e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff08414e410 0x7ff08414e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 -- 172.21.15.97:0/1749295199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08414edd0 con 0x7ff0841f34e0 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.080+0000 7ff083400640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 0x7ff0841f3960 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00e230 tx=0x7ff07c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.081+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c039070 con 0x7ff0841f34e0 2024-03-06T07:34:02.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.081+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c03a040 con 0x7ff0841f34e0 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.081+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c009c10 con 0x7ff0841f34e0 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.081+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08414d870 con 0x7ff0841f34e0 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.081+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff084150ac0 con 0x7ff0841f34e0 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.082+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08404f330 con 0x7ff0841f34e0 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.083+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff07c046070 con 0x7ff0841f34e0 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.084+0000 7ff07aa00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff06006b500 0x7ff06006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.084+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff07c0aba60 con 0x7ff0841f34e0 2024-03-06T07:34:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.084+0000 7ff082a00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff06006b500 0x7ff06006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.085+0000 7ff082a00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff06006b500 0x7ff06006d9c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff0740015b0 tx=0x7ff074009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.088+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff08414d870 con 0x7ff0841f34e0 2024-03-06T07:34:02.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.268+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff08406e180 con 0x7ff0841f34e0 2024-03-06T07:34:02.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.270+0000 7ff07aa00640 1 -- 172.21.15.97:0/1749295199 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+12881 (secure 0 0 0) 0x7ff07c07ab90 con 0x7ff0841f34e0 2024-03-06T07:34:02.272 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:02.272 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":108,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:01.357365+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff06006b500 msgr2=0x7ff06006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff06006b500 0x7ff06006d9c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff0740015b0 tx=0x7ff074009040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 msgr2=0x7ff0841f3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 0x7ff0841f3960 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00e230 tx=0x7ff07c009750 comp rx=0 tx=0).stop 2024-03-06T07:34:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 shutdown_connections 2024-03-06T07:34:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff06006b500 0x7ff06006d9c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff0841f34e0 0x7ff0841f3960 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff08414e410 0x7ff08414e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 --2- 172.21.15.97:0/1749295199 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff068004d40 0x7ff084152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.272+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 >> 172.21.15.97:0/1749295199 conn(0x7ff08406afc0 msgr2=0x7ff0840cdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.273+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 shutdown_connections 2024-03-06T07:34:02.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.273+0000 7ff083e00640 1 -- 172.21.15.97:0/1749295199 wait complete. 2024-03-06T07:34:02.293 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:02.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.496+0000 7f9c8ca00640 1 Processor -- start 2024-03-06T07:34:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.496+0000 7f9c8ca00640 1 -- start start 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c8ca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9c880b3df0 0x7f9c880727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c8ca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9c880b3650 0x7f9c880b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c8ca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c880b4710 con 0x7f9c880cda90 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c8ca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c88072310 con 0x7f9c880b3650 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c8ca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c880719e0 con 0x7f9c880b3df0 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c85400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9c880b3650 0x7f9c880b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c86800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9c880cda90 0x7f9c8806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36806/0 (socket says 172.21.15.97:36806) 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.498+0000 7f9c86800640 1 -- 172.21.15.97:0/711704706 learned_addr learned my addr 172.21.15.97:0/711704706 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.499+0000 7f9c85e00640 1 --2- 172.21.15.97:0/711704706 >> v2:172.21.15.135:3300/0 conn(0x7f9c880b3df0 0x7f9c880727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.499+0000 7f9c85400640 1 -- 172.21.15.97:0/711704706 >> v2:172.21.15.135:3300/0 conn(0x7f9c880b3df0 msgr2=0x7f9c880727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.499+0000 7f9c85400640 1 --2- 172.21.15.97:0/711704706 >> v2:172.21.15.135:3300/0 conn(0x7f9c880b3df0 0x7f9c880727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.499+0000 7f9c85400640 1 -- 172.21.15.97:0/711704706 >> v1:172.21.15.97:6789/0 conn(0x7f9c880cda90 legacy=0x7f9c8806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.499+0000 7f9c85400640 1 -- 172.21.15.97:0/711704706 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88071cc0 con 0x7f9c880b3650 2024-03-06T07:34:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.499+0000 7f9c84a00640 1 -- 172.21.15.97:0/711704706 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677466537 0 0) 0x7f9c880b4710 con 0x7f9c880cda90 2024-03-06T07:34:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.500+0000 7f9c85400640 1 --2- 172.21.15.97:0/711704706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3650 0x7f9c880b41d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9c880b3dc0 tx=0x7f9c7002e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f640bed0507c62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.500+0000 7f9c84a00640 1 -- 172.21.15.97:0/711704706 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7003b070 con 0x7f9c880b3650 2024-03-06T07:34:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.500+0000 7f9c84a00640 1 -- 172.21.15.97:0/711704706 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7002ee50 con 0x7f9c880b3650 2024-03-06T07:34:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.500+0000 7f9c84a00640 1 -- 172.21.15.97:0/711704706 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70036660 con 0x7f9c880b3650 2024-03-06T07:34:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 -- 172.21.15.97:0/711704706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3650 msgr2=0x7f9c880b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/711704706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3650 0x7f9c880b41d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9c880b3dc0 tx=0x7f9c7002e0a0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 -- 172.21.15.97:0/711704706 shutdown_connections 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/711704706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3650 0x7f9c880b41d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/711704706 >> v2:172.21.15.135:3300/0 conn(0x7f9c880b3df0 0x7f9c880727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 -- 172.21.15.97:0/711704706 >> 172.21.15.97:0/711704706 conn(0x7f9c880b1940 msgr2=0x7f9c880b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 -- 172.21.15.97:0/711704706 shutdown_connections 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.501+0000 7f9c8ca00640 1 -- 172.21.15.97:0/711704706 wait complete. 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.503+0000 7f9c8ca00640 1 Processor -- start 2024-03-06T07:34:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.503+0000 7f9c8ca00640 1 -- start start 2024-03-06T07:34:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c8ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c880b3650 0x7f9c8814f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c8ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 0x7f9c88150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c8ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c880cda90 0x7f9c881522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c8ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8815b240 con 0x7f9c880b3df0 2024-03-06T07:34:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c8ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c880b3b30 con 0x7f9c880b3650 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c8ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8815b3c0 con 0x7f9c880cda90 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 0x7f9c88150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 0x7f9c88150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43868/0 (socket says 172.21.15.97:43868) 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 -- 172.21.15.97:0/843529997 learned_addr learned my addr 172.21.15.97:0/843529997 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85e00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c880b3650 0x7f9c8814f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c86800640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c880cda90 0x7f9c881522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 -- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c880cda90 msgr2=0x7f9c881522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c880cda90 0x7f9c881522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 -- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c880b3650 msgr2=0x7f9c8814f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c880b3650 0x7f9c8814f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.504+0000 7f9c85400640 1 -- 172.21.15.97:0/843529997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88152d80 con 0x7f9c880b3df0 2024-03-06T07:34:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.505+0000 7f9c85400640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 0x7f9c88150b20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9c70000c00 tx=0x7f9c70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.505+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70046070 con 0x7f9c880b3df0 2024-03-06T07:34:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.505+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c70004030 con 0x7f9c880b3df0 2024-03-06T07:34:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.505+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c88150040 con 0x7f9c880b3df0 2024-03-06T07:34:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.505+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7003b040 con 0x7f9c880b3df0 2024-03-06T07:34:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.505+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c881e5610 con 0x7f9c880b3df0 2024-03-06T07:34:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.506+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8804f330 con 0x7f9c880b3df0 2024-03-06T07:34:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.507+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c70049070 con 0x7f9c880b3df0 2024-03-06T07:34:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.508+0000 7f9c7e000640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.508+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9c700abc00 con 0x7f9c880b3df0 2024-03-06T07:34:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.509+0000 7f9c85e00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.509+0000 7f9c85e00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9c8815aa40 tx=0x7f9c7800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.513+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c7007ad30 con 0x7f9c880b3df0 2024-03-06T07:34:02.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.691+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c880cc0e0 con 0x7f9c880b3df0 2024-03-06T07:34:02.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.692+0000 7f9c7e000640 1 -- 172.21.15.97:0/843529997 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c7007a630 con 0x7f9c880b3df0 2024-03-06T07:34:02.695 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 msgr2=0x7f9c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9c8815aa40 tx=0x7f9c7800b040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 msgr2=0x7f9c88150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 0x7f9c88150b20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9c70000c00 tx=0x7f9c70004730 comp rx=0 tx=0).stop 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 shutdown_connections 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c6406b4d0 0x7f9c6406d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c880cda90 0x7f9c881522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c880b3df0 0x7f9c88150b20 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 --2- 172.21.15.97:0/843529997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c880b3650 0x7f9c8814f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 >> 172.21.15.97:0/843529997 conn(0x7f9c880b1940 msgr2=0x7f9c8806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 shutdown_connections 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.695+0000 7f9c8ca00640 1 -- 172.21.15.97:0/843529997 wait complete. 2024-03-06T07:34:02.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:34:02.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.914+0000 7f8445600640 1 Processor -- start 2024-03-06T07:34:02.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.914+0000 7f8445600640 1 -- start start 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f8445600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f844015e230 0x7f844006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f8445600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f84400b4320 0x7f84400d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f8445600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f84400ce550 0x7f84400ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f8445600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844015e610 con 0x7f84400ce550 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f8445600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84400cf340 con 0x7f84400b4320 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f8445600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84400cfe60 con 0x7f844015e230 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.915+0000 7f843f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f84400ce550 0x7f84400ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f84400ce550 0x7f84400ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43872/0 (socket says 172.21.15.97:43872) 2024-03-06T07:34:02.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 -- 172.21.15.97:0/4136993255 learned_addr learned my addr 172.21.15.97:0/4136993255 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843ea00640 1 --2- 172.21.15.97:0/4136993255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f844015e230 0x7f844006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843e000640 1 --2- 172.21.15.97:0/4136993255 >> v2:172.21.15.135:3300/0 conn(0x7f84400b4320 0x7f84400d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 -- 172.21.15.97:0/4136993255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f844015e230 msgr2=0x7f844006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 --2- 172.21.15.97:0/4136993255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f844015e230 0x7f844006edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 -- 172.21.15.97:0/4136993255 >> v2:172.21.15.135:3300/0 conn(0x7f84400b4320 msgr2=0x7f84400d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 --2- 172.21.15.97:0/4136993255 >> v2:172.21.15.135:3300/0 conn(0x7f84400b4320 0x7f84400d0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.916+0000 7f843f400640 1 -- 172.21.15.97:0/4136993255 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844006f390 con 0x7f84400ce550 2024-03-06T07:34:02.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.917+0000 7f843f400640 1 --2- 172.21.15.97:0/4136993255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f84400ce9e0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f84400750a0 tx=0x7f843802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53aa6b966492d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.917+0000 7f843d600640 1 -- 172.21.15.97:0/4136993255 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843802e070 con 0x7f84400ce550 2024-03-06T07:34:02.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.917+0000 7f843d600640 1 -- 172.21.15.97:0/4136993255 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8438037070 con 0x7f84400ce550 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.917+0000 7f843d600640 1 -- 172.21.15.97:0/4136993255 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438031a00 con 0x7f84400ce550 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 -- 172.21.15.97:0/4136993255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 msgr2=0x7f84400ce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 --2- 172.21.15.97:0/4136993255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f84400ce9e0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f84400750a0 tx=0x7f843802b880 comp rx=0 tx=0).stop 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 -- 172.21.15.97:0/4136993255 shutdown_connections 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 --2- 172.21.15.97:0/4136993255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f84400ce9e0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 --2- 172.21.15.97:0/4136993255 >> v2:172.21.15.135:3300/0 conn(0x7f84400b4320 0x7f84400d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 --2- 172.21.15.97:0/4136993255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f844015e230 0x7f844006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 -- 172.21.15.97:0/4136993255 >> 172.21.15.97:0/4136993255 conn(0x7f84400b3800 msgr2=0x7f84400b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 -- 172.21.15.97:0/4136993255 shutdown_connections 2024-03-06T07:34:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.918+0000 7f8445600640 1 -- 172.21.15.97:0/4136993255 wait complete. 2024-03-06T07:34:02.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.920+0000 7f8445600640 1 Processor -- start 2024-03-06T07:34:02.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.920+0000 7f8445600640 1 -- start start 2024-03-06T07:34:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f8445600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f84400b4320 0x7f84400c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f8445600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f844014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f8445600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f844015e230 0x7f844006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f8445600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400d06c0 con 0x7f84400ce550 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f8445600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400d03e0 con 0x7f844015e230 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f8445600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84400d1120 con 0x7f84400b4320 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f844014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f844014bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43888/0 (socket says 172.21.15.97:43888) 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 -- 172.21.15.97:0/683139628 learned_addr learned my addr 172.21.15.97:0/683139628 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843f400640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f844015e230 0x7f844006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 -- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f84400b4320 msgr2=0x7f84400c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f84400b4320 0x7f84400c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 -- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f844015e230 msgr2=0x7f844006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f844015e230 0x7f844006c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.921+0000 7f843e000640 1 -- 172.21.15.97:0/683139628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84400c8660 con 0x7f84400ce550 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.922+0000 7f843e000640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f844014bfe0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f842800cb80 tx=0x7f842800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.922+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428015070 con 0x7f84400ce550 2024-03-06T07:34:02.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.922+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8428003b90 con 0x7f84400ce550 2024-03-06T07:34:02.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.922+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84400c5a50 con 0x7f84400ce550 2024-03-06T07:34:02.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.922+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800d040 con 0x7f84400ce550 2024-03-06T07:34:02.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.922+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844014cfe0 con 0x7f84400ce550 2024-03-06T07:34:02.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.923+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844004f330 con 0x7f84400ce550 2024-03-06T07:34:02.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.924+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f842800a8b0 con 0x7f84400ce550 2024-03-06T07:34:02.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.925+0000 7f8436a00640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.925+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8428089e00 con 0x7f84400ce550 2024-03-06T07:34:02.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.925+0000 7f843ea00640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.926+0000 7f843ea00640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841c06b4d0 0x7f841c06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8430006fd0 tx=0x7f8430008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:02.929+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8428058fb0 con 0x7f84400ce550 2024-03-06T07:34:03.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.124+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f84400d1740 con 0x7f84400ce550 2024-03-06T07:34:03.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.124+0000 7f8436a00640 1 -- 172.21.15.97:0/683139628 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v108) v1 ==== 84+0+65 (secure 0 0 0) 0x7f84280588b0 con 0x7f84400ce550 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:03.127 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:34:03.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841c06b4d0 msgr2=0x7f841c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841c06b4d0 0x7f841c06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8430006fd0 tx=0x7f8430008040 comp rx=0 tx=0).stop 2024-03-06T07:34:03.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 msgr2=0x7f844014bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f844014bfe0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f842800cb80 tx=0x7f842800cf80 comp rx=0 tx=0).stop 2024-03-06T07:34:03.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 shutdown_connections 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f844015e230 0x7f844006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84400ce550 0x7f844014bfe0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 --2- 172.21.15.97:0/683139628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f84400b4320 0x7f84400c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 >> 172.21.15.97:0/683139628 conn(0x7f84400b3800 msgr2=0x7f844006cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 shutdown_connections 2024-03-06T07:34:03.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.127+0000 7f8445600640 1 -- 172.21.15.97:0/683139628 wait complete. 2024-03-06T07:34:03.146 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:34:03.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:34:03.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.353+0000 7f8dcfe00640 1 Processor -- start 2024-03-06T07:34:03.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.353+0000 7f8dcfe00640 1 -- start start 2024-03-06T07:34:03.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.355+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc815e060 0x7f8dc815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.355+0000 7f8dcfe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80b3960 con 0x7f8dc80b3580 2024-03-06T07:34:03.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.355+0000 7f8dcfe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80b3bd0 con 0x7f8dc80cf3a0 2024-03-06T07:34:03.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.355+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc806d110 con 0x7f8dc815e060 2024-03-06T07:34:03.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.355+0000 7f8dcd400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8dc80b3580 0x7f8dc806e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36814/0 (socket says 172.21.15.97:36814) 2024-03-06T07:34:03.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.355+0000 7f8dcd400640 1 -- 172.21.15.97:0/1257819157 learned_addr learned my addr 172.21.15.97:0/1257819157 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:03.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dcde00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc815e060 0x7f8dc815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588288382 0 0) 0x7f8dc80b3960 con 0x7f8dc80b3580 2024-03-06T07:34:03.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0003520 con 0x7f8dc80b3580 2024-03-06T07:34:03.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240093805 0 0) 0x7f8db0003520 con 0x7f8dc80b3580 2024-03-06T07:34:03.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc815e060 msgr2=0x7f8dc815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc815e060 0x7f8dc815e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 >> v1:172.21.15.135:6789/0 conn(0x7f8dc80cf3a0 legacy=0x7f8dc806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80b3eb0 con 0x7f8dc80b3580 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8dbc001670 con 0x7f8dc80b3580 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 >> v1:172.21.15.97:6789/0 conn(0x7f8dc80b3580 legacy=0x7f8dc806e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db0003a50 0x7f8db0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db0004800 0x7f8db0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8db000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc806d110 con 0x7f8db00051c0 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db000c0f0 con 0x7f8db0004800 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db000c2f0 con 0x7f8db0003a50 2024-03-06T07:34:03.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.356+0000 7f8dcde00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8db000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcd400640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db0003a50 0x7f8db0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcca00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db0004800 0x7f8db0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcde00640 1 -- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db0003a50 msgr2=0x7f8db0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcde00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db0003a50 0x7f8db0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcde00640 1 -- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db0004800 msgr2=0x7f8db0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcde00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db0004800 0x7f8db0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.357+0000 7f8dcde00640 1 -- 172.21.15.97:0/1257819157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0003520 con 0x7f8db00051c0 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.358+0000 7f8dcde00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8db000b9c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8dc815e440 tx=0x7f8db0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b02f54d9299733 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.358+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000b550 con 0x7f8db00051c0 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.358+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc0034610 con 0x7f8db00051c0 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.358+0000 7f8dc7e00640 1 -- 172.21.15.97:0/1257819157 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00349e0 con 0x7f8db00051c0 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcfe00640 1 -- 172.21.15.97:0/1257819157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 msgr2=0x7f8db000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/1257819157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8db000b9c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8dc815e440 tx=0x7f8db0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcde00640 1 -- 172.21.15.97:0/1257819157 reap_dead start 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcfe00640 1 -- 172.21.15.97:0/1257819157 shutdown_connections 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcfe00640 1 -- 172.21.15.97:0/1257819157 >> 172.21.15.97:0/1257819157 conn(0x7f8dc80cfa80 msgr2=0x7f8dc80cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcfe00640 1 -- 172.21.15.97:0/1257819157 shutdown_connections 2024-03-06T07:34:03.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.359+0000 7f8dcfe00640 1 -- 172.21.15.97:0/1257819157 wait complete. 2024-03-06T07:34:03.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.361+0000 7f8dcfe00640 1 Processor -- start 2024-03-06T07:34:03.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.361+0000 7f8dcfe00640 1 -- start start 2024-03-06T07:34:03.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8dc814ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dc815d360 0x7f8dc815c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc8153140 0x7f8dc81535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80b47e0 con 0x7f8db00051c0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc815ea10 con 0x7f8dc815d360 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80b4470 con 0x7f8dc8153140 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8dc814ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8dc814ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43908/0 (socket says 172.21.15.97:43908) 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 -- 172.21.15.97:0/3886076907 learned_addr learned my addr 172.21.15.97:0/3886076907 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcca00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dc815d360 0x7f8dc815c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 -- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc8153140 msgr2=0x7f8dc81535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc8153140 0x7f8dc81535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 -- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dc815d360 msgr2=0x7f8dc815c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dc815d360 0x7f8dc815c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.362+0000 7f8dcd400640 1 -- 172.21.15.97:0/3886076907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc814f3a0 con 0x7f8db00051c0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.363+0000 7f8dcd400640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8dc814ed70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00e710 tx=0x7f8dbc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.363+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc00ee20 con 0x7f8db00051c0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.363+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dbc018070 con 0x7f8db00051c0 2024-03-06T07:34:03.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.363+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc00ca70 con 0x7f8db00051c0 2024-03-06T07:34:03.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.363+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc814f680 con 0x7f8db00051c0 2024-03-06T07:34:03.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.363+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81506c0 con 0x7f8db00051c0 2024-03-06T07:34:03.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.364+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc804f330 con 0x7f8db00051c0 2024-03-06T07:34:03.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.366+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dbc00cc10 con 0x7f8db00051c0 2024-03-06T07:34:03.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.366+0000 7f8dc5600640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dac06b500 0x7f8dac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.367+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8dbc089c90 con 0x7f8db00051c0 2024-03-06T07:34:03.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.367+0000 7f8dcca00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dac06b500 0x7f8dac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.368+0000 7f8dcca00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dac06b500 0x7f8dac06d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8db8007fd0 tx=0x7f8db8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.372+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8dbc058e50 con 0x7f8db00051c0 2024-03-06T07:34:03.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:03.568+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8dc8154500 con 0x7f8db00051c0 2024-03-06T07:34:04.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.390+0000 7f8dc5600640 1 -- 172.21.15.97:0/3886076907 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v109) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8dbc058750 con 0x7f8db00051c0 2024-03-06T07:34:04.392 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.394+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dac06b500 msgr2=0x7f8dac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.394+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dac06b500 0x7f8dac06d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8db8007fd0 tx=0x7f8db8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.394+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 msgr2=0x7f8dc814ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.394+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8dc814ed70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00e710 tx=0x7f8dbc00ebd0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 shutdown_connections 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8dac06b500 0x7f8dac06d9c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8dc8153140 0x7f8dc81535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dc815d360 0x7f8dc815c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 --2- 172.21.15.97:0/3886076907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db00051c0 0x7f8dc814ed70 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 >> 172.21.15.97:0/3886076907 conn(0x7f8dc80cfa80 msgr2=0x7f8dc80cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.396+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 shutdown_connections 2024-03-06T07:34:04.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.397+0000 7f8dcfe00640 1 -- 172.21.15.97:0/3886076907 wait complete. 2024-03-06T07:34:04.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:34:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.625+0000 7f09bf400640 1 Processor -- start 2024-03-06T07:34:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.625+0000 7f09bf400640 1 -- start start 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.626+0000 7f09bf400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f09b815bdc0 0x7f09b80cd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bf400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f09b80cdba0 0x7f09b806d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bf400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b815e730 con 0x7f09b8141850 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bf400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b80cdff0 con 0x7f09b815bdc0 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bf400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b815ea10 con 0x7f09b80cdba0 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f09b815bdc0 0x7f09b80cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f09b815bdc0 0x7f09b80cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51968/0 (socket says 172.21.15.97:51968) 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 -- 172.21.15.97:0/1646536380 learned_addr learned my addr 172.21.15.97:0/1646536380 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bd400640 1 --2- 172.21.15.97:0/1646536380 >> v2:172.21.15.135:3300/0 conn(0x7f09b80cdba0 0x7f09b806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 -- 172.21.15.97:0/1646536380 >> v2:172.21.15.135:3300/0 conn(0x7f09b80cdba0 msgr2=0x7f09b806d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 --2- 172.21.15.97:0/1646536380 >> v2:172.21.15.135:3300/0 conn(0x7f09b80cdba0 0x7f09b806d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 -- 172.21.15.97:0/1646536380 >> v1:172.21.15.97:6789/0 conn(0x7f09b8141850 legacy=0x7f09b806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.627+0000 7f09bca00640 1 -- 172.21.15.97:0/1646536380 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b815ecf0 con 0x7f09b815bdc0 2024-03-06T07:34:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.629+0000 7f09bca00640 1 --2- 172.21.15.97:0/1646536380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b815bdc0 0x7f09b80cd660 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f09b806c480 tx=0x7f09ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7566f4e19f13ee5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.629+0000 7f09b7400640 1 -- 172.21.15.97:0/1646536380 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac034070 con 0x7f09b815bdc0 2024-03-06T07:34:04.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.629+0000 7f09b7400640 1 -- 172.21.15.97:0/1646536380 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac030860 con 0x7f09b815bdc0 2024-03-06T07:34:04.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.629+0000 7f09b7400640 1 -- 172.21.15.97:0/1646536380 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac030c30 con 0x7f09b815bdc0 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 -- 172.21.15.97:0/1646536380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b815bdc0 msgr2=0x7f09b80cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 --2- 172.21.15.97:0/1646536380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b815bdc0 0x7f09b80cd660 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f09b806c480 tx=0x7f09ac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 -- 172.21.15.97:0/1646536380 shutdown_connections 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 --2- 172.21.15.97:0/1646536380 >> v2:172.21.15.135:3300/0 conn(0x7f09b80cdba0 0x7f09b806d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 --2- 172.21.15.97:0/1646536380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b815bdc0 0x7f09b80cd660 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 -- 172.21.15.97:0/1646536380 >> 172.21.15.97:0/1646536380 conn(0x7f09b80b2890 msgr2=0x7f09b80ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 -- 172.21.15.97:0/1646536380 shutdown_connections 2024-03-06T07:34:04.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.630+0000 7f09bf400640 1 -- 172.21.15.97:0/1646536380 wait complete. 2024-03-06T07:34:04.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.631+0000 7f09bf400640 1 Processor -- start 2024-03-06T07:34:04.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.632+0000 7f09bf400640 1 -- start start 2024-03-06T07:34:04.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.632+0000 7f09bf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 0x7f09b80aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.632+0000 7f09bf400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09b8141850 0x7f09b80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bf400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09b80adda0 0x7f09b8160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bf400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b81439e0 con 0x7f09b80cdba0 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bf400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815f2b0 con 0x7f09b8141850 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bf400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8143b60 con 0x7f09b80adda0 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 0x7f09b80aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 0x7f09b80aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51972/0 (socket says 172.21.15.97:51972) 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 -- 172.21.15.97:0/295745125 learned_addr learned my addr 172.21.15.97:0/295745125 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 -- 172.21.15.97:0/295745125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09b80adda0 msgr2=0x7f09b8160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09b80adda0 0x7f09b8160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 -- 172.21.15.97:0/295745125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09b8141850 msgr2=0x7f09b80ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09b8141850 0x7f09b80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.633+0000 7f09bca00640 1 -- 172.21.15.97:0/295745125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8138310 con 0x7f09b80cdba0 2024-03-06T07:34:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.634+0000 7f09bca00640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 0x7f09b80aef90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f09ac02b9e0 tx=0x7f09ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.634+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac034040 con 0x7f09b80cdba0 2024-03-06T07:34:04.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.634+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac039d90 con 0x7f09b80cdba0 2024-03-06T07:34:04.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.634+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac043610 con 0x7f09b80cdba0 2024-03-06T07:34:04.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.634+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b80b1680 con 0x7f09b80cdba0 2024-03-06T07:34:04.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.634+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b80b0de0 con 0x7f09b80cdba0 2024-03-06T07:34:04.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.635+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b804f330 con 0x7f09b80cdba0 2024-03-06T07:34:04.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.636+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09ac04a060 con 0x7f09b80cdba0 2024-03-06T07:34:04.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.638+0000 7f09b4c00640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f099806b4d0 0x7f099806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.638+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f09ac0ac4e0 con 0x7f09b80cdba0 2024-03-06T07:34:04.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.638+0000 7f09b7e00640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f099806b4d0 0x7f099806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.639+0000 7f09b7e00640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f099806b4d0 0x7f099806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000f60 tx=0x7f09a40079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.642+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09b8138310 con 0x7f09b80cdba0 2024-03-06T07:34:04.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:04.842+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f09b806c4b0 con 0x7f09b80cdba0 2024-03-06T07:34:05.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.405+0000 7f09b4c00640 1 -- 172.21.15.97:0/295745125 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v110) v1 ==== 170+0+0 (secure 0 0 0) 0x7f09ac03e070 con 0x7f09b80cdba0 2024-03-06T07:34:05.408 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:34:05.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.409+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f099806b4d0 msgr2=0x7f099806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.409+0000 7f09bf400640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f099806b4d0 0x7f099806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000f60 tx=0x7f09a40079d0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.409+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 msgr2=0x7f09b80aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.409+0000 7f09bf400640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 0x7f09b80aef90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f09ac02b9e0 tx=0x7f09ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.410+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 shutdown_connections 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.410+0000 7f09bf400640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f099806b4d0 0x7f099806d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.410+0000 7f09bf400640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f09b80adda0 0x7f09b8160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.410+0000 7f09bf400640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09b8141850 0x7f09b80ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.410+0000 7f09bf400640 1 --2- 172.21.15.97:0/295745125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09b80cdba0 0x7f09b80aef90 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.410+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 >> 172.21.15.97:0/295745125 conn(0x7f09b80b2890 msgr2=0x7f09b806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.411+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 shutdown_connections 2024-03-06T07:34:05.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.412+0000 7f09bf400640 1 -- 172.21.15.97:0/295745125 wait complete. 2024-03-06T07:34:05.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:34:05.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.628+0000 7f3867200640 1 Processor -- start 2024-03-06T07:34:05.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.629+0000 7f3867200640 1 -- start start 2024-03-06T07:34:05.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3867200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f386806dbf0 0x7f386816d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3867200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38680b3560 con 0x7f3868159570 2024-03-06T07:34:05.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3867200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38680cc480 con 0x7f38680cff60 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3867200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38680cc760 con 0x7f386806dbf0 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3866800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f386806dbf0 0x7f386816d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3865e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3868159570 0x7f38680cf850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33684/0 (socket says 172.21.15.97:33684) 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3866800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f386806dbf0 0x7f386816d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51986/0 (socket says 172.21.15.97:51986) 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.630+0000 7f3865e00640 1 -- 172.21.15.97:0/3607478162 learned_addr learned my addr 172.21.15.97:0/3607478162 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.631+0000 7f3866800640 1 -- 172.21.15.97:0/3607478162 >> v1:172.21.15.135:6789/0 conn(0x7f38680cff60 legacy=0x7f386806d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.631+0000 7f3866800640 1 -- 172.21.15.97:0/3607478162 >> v1:172.21.15.97:6789/0 conn(0x7f3868159570 legacy=0x7f38680cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.631+0000 7f3866800640 1 -- 172.21.15.97:0/3607478162 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38680cca40 con 0x7f386806dbf0 2024-03-06T07:34:05.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.632+0000 7f3866800640 1 --2- 172.21.15.97:0/3607478162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f386816d270 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f3860009d90 tx=0x7f386002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b935f74aa727a7da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3865400640 1 -- 172.21.15.97:0/3607478162 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860039070 con 0x7f386806dbf0 2024-03-06T07:34:05.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3865400640 1 -- 172.21.15.97:0/3607478162 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386002ccf0 con 0x7f386806dbf0 2024-03-06T07:34:05.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3865400640 1 -- 172.21.15.97:0/3607478162 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860034660 con 0x7f386806dbf0 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 -- 172.21.15.97:0/3607478162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 msgr2=0x7f386816d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 --2- 172.21.15.97:0/3607478162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f386816d270 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f3860009d90 tx=0x7f386002bf40 comp rx=0 tx=0).stop 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 -- 172.21.15.97:0/3607478162 shutdown_connections 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 --2- 172.21.15.97:0/3607478162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f386816d270 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 -- 172.21.15.97:0/3607478162 >> 172.21.15.97:0/3607478162 conn(0x7f386806afc0 msgr2=0x7f386806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 -- 172.21.15.97:0/3607478162 shutdown_connections 2024-03-06T07:34:05.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.633+0000 7f3867200640 1 -- 172.21.15.97:0/3607478162 wait complete. 2024-03-06T07:34:05.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.635+0000 7f3867200640 1 Processor -- start 2024-03-06T07:34:05.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.635+0000 7f3867200640 1 -- start start 2024-03-06T07:34:05.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3867200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f3868162940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3867200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38680cff60 0x7f3868162e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3867200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868159570 0x7f3868153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3867200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386816d930 con 0x7f386806dbf0 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3867200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386816d7b0 con 0x7f3868159570 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3867200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386816dab0 con 0x7f38680cff60 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.636+0000 7f3865e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f3868162940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f3868162940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51992/0 (socket says 172.21.15.97:51992) 2024-03-06T07:34:05.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 -- 172.21.15.97:0/1283975606 learned_addr learned my addr 172.21.15.97:0/1283975606 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f385d400640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38680cff60 0x7f3868162e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 -- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38680cff60 msgr2=0x7f3868162e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38680cff60 0x7f3868162e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 -- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868159570 msgr2=0x7f3868153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3866800640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868159570 0x7f3868153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868159570 0x7f3868153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 -- 172.21.15.97:0/1283975606 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868154bc0 con 0x7f386806dbf0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.637+0000 7f3865e00640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f3868162940 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f385800c480 tx=0x7f385800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.638+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3858017070 con 0x7f386806dbf0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.638+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3858005ce0 con 0x7f386806dbf0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.638+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38580072f0 con 0x7f386806dbf0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.638+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3868154ea0 con 0x7f386806dbf0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.638+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3868155460 con 0x7f386806dbf0 2024-03-06T07:34:05.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.639+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386804f330 con 0x7f386806dbf0 2024-03-06T07:34:05.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.640+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f385801a070 con 0x7f386806dbf0 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.641+0000 7f385ea00640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b4d0 0x7f384406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.641+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f38580893a0 con 0x7f386806dbf0 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.641+0000 7f385d400640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b4d0 0x7f384406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.642+0000 7f385d400640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b4d0 0x7f384406d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f38681fbd90 tx=0x7f3850009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.646+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3858057cc0 con 0x7f386806dbf0 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:05.871+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f38680cdca0 con 0x7f386806dbf0 2024-03-06T07:34:06.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.417+0000 7f385ea00640 1 -- 172.21.15.97:0/1283975606 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f385805bd60 con 0x7f386806dbf0 2024-03-06T07:34:06.423 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:34:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.423+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b4d0 msgr2=0x7f384406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.423+0000 7f3867200640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b4d0 0x7f384406d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f38681fbd90 tx=0x7f3850009040 comp rx=0 tx=0).stop 2024-03-06T07:34:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.423+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 msgr2=0x7f3868162940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.423+0000 7f3867200640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f3868162940 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f385800c480 tx=0x7f385800c940 comp rx=0 tx=0).stop 2024-03-06T07:34:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 shutdown_connections 2024-03-06T07:34:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f384406b4d0 0x7f384406d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868159570 0x7f3868153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f38680cff60 0x7f3868162e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 --2- 172.21.15.97:0/1283975606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f386806dbf0 0x7f3868162940 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 >> 172.21.15.97:0/1283975606 conn(0x7f386806afc0 msgr2=0x7f386806d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 shutdown_connections 2024-03-06T07:34:06.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.424+0000 7f3867200640 1 -- 172.21.15.97:0/1283975606 wait complete. 2024-03-06T07:34:06.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.641+0000 7fc716200640 1 Processor -- start 2024-03-06T07:34:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.641+0000 7fc716200640 1 -- start start 2024-03-06T07:34:06.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.641+0000 7fc716200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7101561f0 0x7fc7100ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.641+0000 7fc716200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71015c240 con 0x7fc7100cc9d0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.641+0000 7fc716200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71015c500 con 0x7fc7100aebf0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.641+0000 7fc716200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71015ce30 con 0x7fc7101561f0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc715800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc7100cc9d0 0x7fc71015f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33698/0 (socket says 172.21.15.97:33698) 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc715800640 1 -- 172.21.15.97:0/2715980691 learned_addr learned my addr 172.21.15.97:0/2715980691 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc714e00640 1 --2- 172.21.15.97:0/2715980691 >> v2:172.21.15.135:3300/0 conn(0x7fc7101561f0 0x7fc7100ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484350989 0 0) 0x7fc71015c240 con 0x7fc7100cc9d0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f4003520 con 0x7fc7100cc9d0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024837331 0 0) 0x7fc6f4003520 con 0x7fc7100cc9d0 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 >> v2:172.21.15.135:3300/0 conn(0x7fc7101561f0 msgr2=0x7fc7100ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 --2- 172.21.15.97:0/2715980691 >> v2:172.21.15.135:3300/0 conn(0x7fc7101561f0 0x7fc7100ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 >> v1:172.21.15.135:6789/0 conn(0x7fc7100aebf0 legacy=0x7fc7100cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71015c7e0 con 0x7fc7100cc9d0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc7080019f0 con 0x7fc7100cc9d0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.642+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 >> v1:172.21.15.97:6789/0 conn(0x7fc7100cc9d0 legacy=0x7fc71015f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc70fe00640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f40039b0 0x7fc6f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc70fe00640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6f40043b0 0x7fc6f4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc70fe00640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6f4005080 0x7fc6f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015ce30 con 0x7fc6f4005080 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f400bfe0 con 0x7fc6f40039b0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f400c1e0 con 0x7fc6f40043b0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc715800640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6f4005080 0x7fc6f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc714e00640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f40039b0 0x7fc6f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc707e00640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6f40043b0 0x7fc6f4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc715800640 1 -- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6f40043b0 msgr2=0x7fc6f4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc715800640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc6f40043b0 0x7fc6f4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc715800640 1 -- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f40039b0 msgr2=0x7fc6f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc715800640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f40039b0 0x7fc6f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.643+0000 7fc715800640 1 -- 172.21.15.97:0/2715980691 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4003520 con 0x7fc6f4005080 2024-03-06T07:34:06.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.644+0000 7fc715800640 1 -- 172.21.15.97:0/2715980691 reap_dead start 2024-03-06T07:34:06.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.644+0000 7fc715800640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6f4005080 0x7fc6f400b9c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc70800a880 tx=0x7fc6f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0381a7d05edc16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800a4a0 con 0x7fc6f4005080 2024-03-06T07:34:06.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc708034650 con 0x7fc6f4005080 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc70fe00640 1 -- 172.21.15.97:0/2715980691 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708034a40 con 0x7fc6f4005080 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 -- 172.21.15.97:0/2715980691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6f4005080 msgr2=0x7fc6f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6f4005080 0x7fc6f400b9c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc70800a880 tx=0x7fc6f4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 -- 172.21.15.97:0/2715980691 shutdown_connections 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 --2- 172.21.15.97:0/2715980691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6f4005080 0x7fc6f400b9c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 -- 172.21.15.97:0/2715980691 >> 172.21.15.97:0/2715980691 conn(0x7fc71006afc0 msgr2=0x7fc71006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 -- 172.21.15.97:0/2715980691 shutdown_connections 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.645+0000 7fc716200640 1 -- 172.21.15.97:0/2715980691 wait complete. 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.647+0000 7fc716200640 1 Processor -- start 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.647+0000 7fc716200640 1 -- start start 2024-03-06T07:34:06.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc716200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f4005080 0x7fc710157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc716200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 0x7fc710158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc716200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc710159340 0x7fc71015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc716200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc710156750 con 0x7fc7101509a0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc716200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101565d0 con 0x7fc6f4005080 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc716200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100ae6b0 con 0x7fc710159340 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.648+0000 7fc707e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 0x7fc710158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 0x7fc710158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52016/0 (socket says 172.21.15.97:52016) 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 -- 172.21.15.97:0/2310955891 learned_addr learned my addr 172.21.15.97:0/2310955891 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc715800640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc710159340 0x7fc71015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc714e00640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f4005080 0x7fc710157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 -- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc710159340 msgr2=0x7fc71015b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc710159340 0x7fc71015b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 -- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f4005080 msgr2=0x7fc710157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f4005080 0x7fc710157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.649+0000 7fc707e00640 1 -- 172.21.15.97:0/2310955891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710157d60 con 0x7fc7101509a0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.650+0000 7fc707e00640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 0x7fc710158e00 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800e5a0 tx=0x7fc6f800e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.650+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80052e0 con 0x7fc7101509a0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.650+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8018070 con 0x7fc7101509a0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.650+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc710158040 con 0x7fc7101509a0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.650+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8003830 con 0x7fc7101509a0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.650+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc710158600 con 0x7fc7101509a0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.651+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc710155d60 con 0x7fc7101509a0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.652+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6f80039d0 con 0x7fc7101509a0 2024-03-06T07:34:06.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.653+0000 7fc70d600640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6ec06b500 0x7fc6ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.653+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6f8089d30 con 0x7fc7101509a0 2024-03-06T07:34:06.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.654+0000 7fc714e00640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6ec06b500 0x7fc6ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.654+0000 7fc714e00640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6ec06b500 0x7fc6ec06d9c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc700009b40 tx=0x7fc700009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.658+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc710155d60 con 0x7fc7101509a0 2024-03-06T07:34:06.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.854+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc71004f330 con 0x7fc7101509a0 2024-03-06T07:34:06.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.857+0000 7fc70d600640 1 -- 172.21.15.97:0/2310955891 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16628 (secure 0 0 0) 0x7fc6f8058c40 con 0x7fc7101509a0 2024-03-06T07:34:06.860 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:06.860 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":111,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:05.873568+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:06.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6ec06b500 msgr2=0x7fc6ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6ec06b500 0x7fc6ec06d9c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc700009b40 tx=0x7fc700009190 comp rx=0 tx=0).stop 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 msgr2=0x7fc710158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 0x7fc710158e00 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800e5a0 tx=0x7fc6f800e9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 shutdown_connections 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6ec06b500 0x7fc6ec06d9c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc710159340 0x7fc71015b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7101509a0 0x7fc710158e00 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 --2- 172.21.15.97:0/2310955891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc6f4005080 0x7fc710157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 >> 172.21.15.97:0/2310955891 conn(0x7fc71006afc0 msgr2=0x7fc710155650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 shutdown_connections 2024-03-06T07:34:06.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:06.860+0000 7fc716200640 1 -- 172.21.15.97:0/2310955891 wait complete. 2024-03-06T07:34:06.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:07.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.087+0000 7f1f0b400640 1 Processor -- start 2024-03-06T07:34:07.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.087+0000 7f1f0b400640 1 -- start start 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f0b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1f040b4680 0x7f1f0406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f0b400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1f04072340 0x7f1f0406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f0b400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0415b090 con 0x7f1f0415acb0 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f0b400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040b2a30 con 0x7f1f040b4680 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f0b400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040b4360 con 0x7f1f04072340 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1f040b4680 0x7f1f0406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1f040b4680 0x7f1f0406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52024/0 (socket says 172.21.15.97:52024) 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 -- 172.21.15.97:0/1760864252 learned_addr learned my addr 172.21.15.97:0/1760864252 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:07.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f09400640 1 --2- 172.21.15.97:0/1760864252 >> v2:172.21.15.135:3300/0 conn(0x7f1f04072340 0x7f1f0406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 -- 172.21.15.97:0/1760864252 >> v2:172.21.15.135:3300/0 conn(0x7f1f04072340 msgr2=0x7f1f0406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 --2- 172.21.15.97:0/1760864252 >> v2:172.21.15.135:3300/0 conn(0x7f1f04072340 0x7f1f0406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 -- 172.21.15.97:0/1760864252 >> v1:172.21.15.135:6789/0 conn(0x7f1f0415acb0 legacy=0x7f1f040b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.089+0000 7f1f03e00640 1 -- 172.21.15.97:0/1760864252 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040b2cf0 con 0x7f1f040b4680 2024-03-06T07:34:07.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.091+0000 7f1f03e00640 1 --2- 172.21.15.97:0/1760864252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040b4680 0x7f1f0406e4f0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f1f040b4e60 tx=0x7f1ef002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c66172df35f9c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f03400640 1 -- 172.21.15.97:0/1760864252 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0035020 con 0x7f1f040b4680 2024-03-06T07:34:07.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f03400640 1 -- 172.21.15.97:0/1760864252 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef0031870 con 0x7f1f040b4680 2024-03-06T07:34:07.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f03400640 1 -- 172.21.15.97:0/1760864252 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0031c40 con 0x7f1f040b4680 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 -- 172.21.15.97:0/1760864252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040b4680 msgr2=0x7f1f0406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 --2- 172.21.15.97:0/1760864252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040b4680 0x7f1f0406e4f0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f1f040b4e60 tx=0x7f1ef002b910 comp rx=0 tx=0).stop 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 -- 172.21.15.97:0/1760864252 shutdown_connections 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 --2- 172.21.15.97:0/1760864252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040b4680 0x7f1f0406e4f0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 --2- 172.21.15.97:0/1760864252 >> v2:172.21.15.135:3300/0 conn(0x7f1f04072340 0x7f1f0406ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 -- 172.21.15.97:0/1760864252 >> 172.21.15.97:0/1760864252 conn(0x7f1f040b1730 msgr2=0x7f1f040b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 -- 172.21.15.97:0/1760864252 shutdown_connections 2024-03-06T07:34:07.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.092+0000 7f1f0b400640 1 -- 172.21.15.97:0/1760864252 wait complete. 2024-03-06T07:34:07.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.094+0000 7f1f0b400640 1 Processor -- start 2024-03-06T07:34:07.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.094+0000 7f1f0b400640 1 -- start start 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f0b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 0x7f1f0415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f0b400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040b4680 0x7f1f04144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f0b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f0415acb0 0x7f1f041447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f0b400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040b4010 con 0x7f1f04072340 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f0b400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040b32b0 con 0x7f1f040b4680 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f0b400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040ccfb0 con 0x7f1f0415acb0 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 0x7f1f0415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 0x7f1f0415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52034/0 (socket says 172.21.15.97:52034) 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 -- 172.21.15.97:0/4070988341 learned_addr learned my addr 172.21.15.97:0/4070988341 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f09400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f0415acb0 0x7f1f041447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f03e00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040b4680 0x7f1f04144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 -- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f0415acb0 msgr2=0x7f1f041447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f0415acb0 0x7f1f041447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 -- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040b4680 msgr2=0x7f1f04144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040b4680 0x7f1f04144290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 -- 172.21.15.97:0/4070988341 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040ce710 con 0x7f1f04072340 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f09400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f0415acb0 0x7f1f041447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f08a00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 0x7f1f0415e9e0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800b420 tx=0x7f1ef800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.095+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8011830 con 0x7f1f04072340 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.097+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8016070 con 0x7f1f04072340 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.097+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8011c10 con 0x7f1f04072340 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.097+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f040ce9f0 con 0x7f1f04072340 2024-03-06T07:34:07.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.097+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f040cefb0 con 0x7f1f04072340 2024-03-06T07:34:07.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.098+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f040cf390 con 0x7f1f04072340 2024-03-06T07:34:07.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.099+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ef801b460 con 0x7f1f04072340 2024-03-06T07:34:07.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.099+0000 7f1f00c00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1ee406b4d0 0x7f1ee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.099+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ef8089d30 con 0x7f1f04072340 2024-03-06T07:34:07.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.099+0000 7f1f03e00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1ee406b4d0 0x7f1ee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.101+0000 7f1f03e00640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1ee406b4d0 0x7f1ee406d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0036000 tx=0x7f1ef0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.104+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ef8020020 con 0x7f1f04072340 2024-03-06T07:34:07.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.299+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f0406ce60 con 0x7f1f04072340 2024-03-06T07:34:07.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.302+0000 7f1f00c00640 1 -- 172.21.15.97:0/4070988341 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16628 (secure 0 0 0) 0x7f1ef8058ba0 con 0x7f1f04072340 2024-03-06T07:34:07.304 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:07.304 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":111,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:05.873568+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:07.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1ee406b4d0 msgr2=0x7f1ee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1ee406b4d0 0x7f1ee406d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0036000 tx=0x7f1ef0008030 comp rx=0 tx=0).stop 2024-03-06T07:34:07.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 msgr2=0x7f1f0415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 0x7f1f0415e9e0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800b420 tx=0x7f1ef800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 shutdown_connections 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1ee406b4d0 0x7f1ee406d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f0415acb0 0x7f1f041447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040b4680 0x7f1f04144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 --2- 172.21.15.97:0/4070988341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f04072340 0x7f1f0415e9e0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 >> 172.21.15.97:0/4070988341 conn(0x7f1f040b1730 msgr2=0x7f1f040af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 shutdown_connections 2024-03-06T07:34:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.305+0000 7f1f0b400640 1 -- 172.21.15.97:0/4070988341 wait complete. 2024-03-06T07:34:07.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:34:07.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.535+0000 7f25f1c00640 1 Processor -- start 2024-03-06T07:34:07.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.535+0000 7f25f1c00640 1 -- start start 2024-03-06T07:34:07.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25f1c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25ec158570 0x7f25ec0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25f1c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec0cfde0 0x7f25ec0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25f1c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec15b130 con 0x7f25ec158950 2024-03-06T07:34:07.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25f1c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec0af170 con 0x7f25ec158570 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25f1c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec15b3f0 con 0x7f25ec0cfde0 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25ebe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25ec158570 0x7f25ec0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25e3400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f25ec158950 0x7f25ec0cf6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33708/0 (socket says 172.21.15.97:33708) 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.536+0000 7f25e3400640 1 -- 172.21.15.97:0/3300293433 learned_addr learned my addr 172.21.15.97:0/3300293433 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.537+0000 7f25f1200640 1 --2- 172.21.15.97:0/3300293433 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec0cfde0 0x7f25ec0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.537+0000 7f25ebe00640 1 -- 172.21.15.97:0/3300293433 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec0cfde0 msgr2=0x7f25ec0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.537+0000 7f25ebe00640 1 --2- 172.21.15.97:0/3300293433 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec0cfde0 0x7f25ec0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.537+0000 7f25ebe00640 1 -- 172.21.15.97:0/3300293433 >> v1:172.21.15.97:6789/0 conn(0x7f25ec158950 legacy=0x7f25ec0cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.537+0000 7f25ebe00640 1 -- 172.21.15.97:0/3300293433 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec15b6d0 con 0x7f25ec158570 2024-03-06T07:34:07.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.538+0000 7f25ebe00640 1 --2- 172.21.15.97:0/3300293433 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec0ae6b0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0097a0 tx=0x7f25dc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211579b50d7d728e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.538+0000 7f25eb400640 1 -- 172.21.15.97:0/3300293433 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc031780 con 0x7f25ec158570 2024-03-06T07:34:07.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.538+0000 7f25eb400640 1 -- 172.21.15.97:0/3300293433 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc037070 con 0x7f25ec158570 2024-03-06T07:34:07.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.538+0000 7f25eb400640 1 -- 172.21.15.97:0/3300293433 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc031b40 con 0x7f25ec158570 2024-03-06T07:34:07.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 -- 172.21.15.97:0/3300293433 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 msgr2=0x7f25ec0ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 --2- 172.21.15.97:0/3300293433 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec0ae6b0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0097a0 tx=0x7f25dc02b820 comp rx=0 tx=0).stop 2024-03-06T07:34:07.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 -- 172.21.15.97:0/3300293433 shutdown_connections 2024-03-06T07:34:07.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 --2- 172.21.15.97:0/3300293433 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec0cfde0 0x7f25ec0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 --2- 172.21.15.97:0/3300293433 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec0ae6b0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 -- 172.21.15.97:0/3300293433 >> 172.21.15.97:0/3300293433 conn(0x7f25ec06afc0 msgr2=0x7f25ec06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 -- 172.21.15.97:0/3300293433 shutdown_connections 2024-03-06T07:34:07.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.539+0000 7f25f1c00640 1 -- 172.21.15.97:0/3300293433 wait complete. 2024-03-06T07:34:07.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.541+0000 7f25f1c00640 1 Processor -- start 2024-03-06T07:34:07.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.541+0000 7f25f1c00640 1 -- start start 2024-03-06T07:34:07.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25ec0cfde0 0x7f25ec144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec158950 0x7f25ec146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec15bed0 con 0x7f25ec158570 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0b4950 con 0x7f25ec0cfde0 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec15c050 con 0x7f25ec158950 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25e3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25e3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52038/0 (socket says 172.21.15.97:52038) 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25e3400640 1 -- 172.21.15.97:0/2676806967 learned_addr learned my addr 172.21.15.97:0/2676806967 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25f1200640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec158950 0x7f25ec146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.542+0000 7f25ebe00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25ec0cfde0 0x7f25ec144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e3400640 1 -- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec158950 msgr2=0x7f25ec146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e3400640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec158950 0x7f25ec146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e3400640 1 -- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25ec0cfde0 msgr2=0x7f25ec144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e3400640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25ec0cfde0 0x7f25ec144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e3400640 1 -- 172.21.15.97:0/2676806967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec145200 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e3400640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec158d50 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c440 tx=0x7f25d400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4017070 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4005ce0 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40072f0 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec145d20 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.543+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec159f30 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.545+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec04f330 con 0x7f25ec158570 2024-03-06T07:34:07.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.545+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25d401a070 con 0x7f25ec158570 2024-03-06T07:34:07.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.546+0000 7f25e8c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25c806b4d0 0x7f25c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.546+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25d4089530 con 0x7f25ec158570 2024-03-06T07:34:07.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.547+0000 7f25ebe00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25c806b4d0 0x7f25c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.547+0000 7f25ebe00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25c806b4d0 0x7f25c806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f25dc036000 tx=0x7f25dc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.551+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25ec145d20 con 0x7f25ec158570 2024-03-06T07:34:07.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:07.786+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f25ec0ced50 con 0x7f25ec158570 2024-03-06T07:34:08.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:08.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:08.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.637+0000 7f25e8c00640 1 -- 172.21.15.97:0/2676806967 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) v1 ==== 118+0+0 (secure 0 0 0) 0x7f25d40583a0 con 0x7f25ec158570 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.640+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25c806b4d0 msgr2=0x7f25c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.640+0000 7f25f1c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25c806b4d0 0x7f25c806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f25dc036000 tx=0x7f25dc03d040 comp rx=0 tx=0).stop 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.640+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 msgr2=0x7f25ec158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.640+0000 7f25f1c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec158d50 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c440 tx=0x7f25d400c900 comp rx=0 tx=0).stop 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 shutdown_connections 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f25c806b4d0 0x7f25c806d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25ec158950 0x7f25ec146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25ec158570 0x7f25ec158d50 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 --2- 172.21.15.97:0/2676806967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25ec0cfde0 0x7f25ec144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 >> 172.21.15.97:0/2676806967 conn(0x7f25ec06afc0 msgr2=0x7f25ec0cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 shutdown_connections 2024-03-06T07:34:08.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.641+0000 7f25f1c00640 1 -- 172.21.15.97:0/2676806967 wait complete. 2024-03-06T07:34:08.661 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:34:08.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:34:08.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.875+0000 7fab5b200640 1 Processor -- start 2024-03-06T07:34:08.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.875+0000 7fab5b200640 1 -- start start 2024-03-06T07:34:08.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.877+0000 7fab5b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fab5c0aef20 0x7fab5c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.877+0000 7fab5b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5c15c1f0 con 0x7fab5c15be10 2024-03-06T07:34:08.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.877+0000 7fab5b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5c06cac0 con 0x7fab5c06da40 2024-03-06T07:34:08.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.877+0000 7fab5b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5c0cd200 con 0x7fab5c0aef20 2024-03-06T07:34:08.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.877+0000 7fab59e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fab5c15be10 0x7fab5c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33724/0 (socket says 172.21.15.97:33724) 2024-03-06T07:34:08.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.877+0000 7fab59e00640 1 -- 172.21.15.97:0/2609520241 learned_addr learned my addr 172.21.15.97:0/2609520241 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353481214 0 0) 0x7fab5c15c1f0 con 0x7fab5c15be10 2024-03-06T07:34:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab40003520 con 0x7fab5c15be10 2024-03-06T07:34:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab5a800640 1 --2- 172.21.15.97:0/2609520241 >> v2:172.21.15.135:3300/0 conn(0x7fab5c0aef20 0x7fab5c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3086188779 0 0) 0x7fab40003520 con 0x7fab5c15be10 2024-03-06T07:34:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 >> v2:172.21.15.135:3300/0 conn(0x7fab5c0aef20 msgr2=0x7fab5c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 --2- 172.21.15.97:0/2609520241 >> v2:172.21.15.135:3300/0 conn(0x7fab5c0aef20 0x7fab5c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 >> v1:172.21.15.135:6789/0 conn(0x7fab5c06da40 legacy=0x7fab5c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5c06cda0 con 0x7fab5c15be10 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fab4c001670 con 0x7fab5c15be10 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 >> v1:172.21.15.97:6789/0 conn(0x7fab5c15be10 legacy=0x7fab5c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400039b0 0x7fab40003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab40004800 0x7fab40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400052e0 0x7fab4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0cd200 con 0x7fab400052e0 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4000c190 con 0x7fab40004800 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.878+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4000c390 con 0x7fab400039b0 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab5a800640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400052e0 0x7fab4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab59e00640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400039b0 0x7fab40003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab51400640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab40004800 0x7fab40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab5a800640 1 -- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400039b0 msgr2=0x7fab40003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab5a800640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400039b0 0x7fab40003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab5a800640 1 -- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab40004800 msgr2=0x7fab40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab5a800640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab40004800 0x7fab40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.879+0000 7fab5a800640 1 -- 172.21.15.97:0/2609520241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40003520 con 0x7fab400052e0 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.880+0000 7fab5a800640 1 -- 172.21.15.97:0/2609520241 reap_dead start 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.880+0000 7fab5a800640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400052e0 0x7fab4000b9c0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fab54007330 tx=0x7fab40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e6e18af82a0d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.880+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54009be0 con 0x7fab400052e0 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.880+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5402ccd0 con 0x7fab400052e0 2024-03-06T07:34:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.880+0000 7fab59400640 1 -- 172.21.15.97:0/2609520241 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54003bd0 con 0x7fab400052e0 2024-03-06T07:34:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 -- 172.21.15.97:0/2609520241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400052e0 msgr2=0x7fab4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400052e0 0x7fab4000b9c0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fab54007330 tx=0x7fab40005830 comp rx=0 tx=0).stop 2024-03-06T07:34:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 -- 172.21.15.97:0/2609520241 shutdown_connections 2024-03-06T07:34:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 --2- 172.21.15.97:0/2609520241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400052e0 0x7fab4000b9c0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 -- 172.21.15.97:0/2609520241 >> 172.21.15.97:0/2609520241 conn(0x7fab5c06afc0 msgr2=0x7fab5c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 -- 172.21.15.97:0/2609520241 shutdown_connections 2024-03-06T07:34:08.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.881+0000 7fab5b200640 1 -- 172.21.15.97:0/2609520241 wait complete. 2024-03-06T07:34:08.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.883+0000 7fab5b200640 1 Processor -- start 2024-03-06T07:34:08.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.883+0000 7fab5b200640 1 -- start start 2024-03-06T07:34:08.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab5b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400052e0 0x7fab5c1421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab5b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 0x7fab5c1526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab5b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab5c152bf0 0x7fab5c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab5b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0cda90 con 0x7fab5c151d60 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab5b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c06d080 con 0x7fab5c152bf0 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab5b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0ccc20 con 0x7fab400052e0 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 0x7fab5c1526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 0x7fab5c1526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52052/0 (socket says 172.21.15.97:52052) 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 -- 172.21.15.97:0/2022143247 learned_addr learned my addr 172.21.15.97:0/2022143247 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab59e00640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400052e0 0x7fab5c1421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 -- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400052e0 msgr2=0x7fab5c1421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400052e0 0x7fab5c1421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 -- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab5c152bf0 msgr2=0x7fab5c14ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab5c152bf0 0x7fab5c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.884+0000 7fab51400640 1 -- 172.21.15.97:0/2022143247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5c150cc0 con 0x7fab5c151d60 2024-03-06T07:34:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.885+0000 7fab51400640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 0x7fab5c1526b0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fab4400e460 tx=0x7fab4400e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.885+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab440135f0 con 0x7fab5c151d60 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.885+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4401a070 con 0x7fab5c151d60 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.885+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab440139d0 con 0x7fab5c151d60 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.885+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5c150fa0 con 0x7fab5c151d60 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.885+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5c151560 con 0x7fab5c151d60 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.886+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5c153010 con 0x7fab5c151d60 2024-03-06T07:34:08.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.887+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab44013b70 con 0x7fab5c151d60 2024-03-06T07:34:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.888+0000 7fab52a00640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab3806b500 0x7fab3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.888+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab4408a2c0 con 0x7fab5c151d60 2024-03-06T07:34:08.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.889+0000 7fab59e00640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab3806b500 0x7fab3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.889+0000 7fab59e00640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab3806b500 0x7fab3806d9c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fab4c000f60 tx=0x7fab4c000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:08.893+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab44024020 con 0x7fab5c151d60 2024-03-06T07:34:09.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.131+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fab5c04f330 con 0x7fab5c151d60 2024-03-06T07:34:09.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.647+0000 7fab52a00640 1 -- 172.21.15.97:0/2022143247 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v74) v1 ==== 122+0+0 (secure 0 0 0) 0x7fab44010300 con 0x7fab5c151d60 2024-03-06T07:34:09.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab3806b500 msgr2=0x7fab3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab3806b500 0x7fab3806d9c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fab4c000f60 tx=0x7fab4c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 msgr2=0x7fab5c1526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 0x7fab5c1526b0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fab4400e460 tx=0x7fab4400e920 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 shutdown_connections 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab3806b500 0x7fab3806d9c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab5c152bf0 0x7fab5c14ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab5c151d60 0x7fab5c1526b0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 --2- 172.21.15.97:0/2022143247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fab400052e0 0x7fab5c1421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 >> 172.21.15.97:0/2022143247 conn(0x7fab5c06afc0 msgr2=0x7fab5c156190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 shutdown_connections 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.651+0000 7fab5b200640 1 -- 172.21.15.97:0/2022143247 wait complete. 2024-03-06T07:34:09.676 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:34:09.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:09.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.885+0000 7ff529400640 1 Processor -- start 2024-03-06T07:34:09.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.885+0000 7ff529400640 1 -- start start 2024-03-06T07:34:09.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff529400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 0x7ff52406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff529400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff524072340 0x7ff52406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff529400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52415b090 con 0x7ff52415acb0 2024-03-06T07:34:09.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff529400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5240b2a30 con 0x7ff524072340 2024-03-06T07:34:09.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff529400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240b4360 con 0x7ff5240b4680 2024-03-06T07:34:09.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff522800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff52415acb0 0x7ff5240b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33728/0 (socket says 172.21.15.97:33728) 2024-03-06T07:34:09.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.886+0000 7ff522800640 1 -- 172.21.15.97:0/815640485 learned_addr learned my addr 172.21.15.97:0/815640485 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff519e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 0x7ff52406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff523200640 1 --2- 172.21.15.97:0/815640485 >> v2:172.21.15.135:3300/0 conn(0x7ff524072340 0x7ff52406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123948002 0 0) 0x7ff52415b090 con 0x7ff52415acb0 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff508003520 con 0x7ff52415acb0 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252820286 0 0) 0x7ff508003520 con 0x7ff52415acb0 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 msgr2=0x7ff52406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 0x7ff52406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 >> v2:172.21.15.135:3300/0 conn(0x7ff524072340 msgr2=0x7ff52406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 --2- 172.21.15.97:0/815640485 >> v2:172.21.15.135:3300/0 conn(0x7ff524072340 0x7ff52406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5240b2cf0 con 0x7ff52415acb0 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff514001670 con 0x7ff52415acb0 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 >> v1:172.21.15.97:6789/0 conn(0x7ff52415acb0 legacy=0x7ff5240b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5080039b0 0x7ff508003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff508004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 0x7ff5080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240b2a30 con 0x7ff508004800 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.887+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240b4360 con 0x7ff5080039b0 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50800baa0 con 0x7ff5240b4680 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff519e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff508004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff522800640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5080039b0 0x7ff508003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff523200640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 0x7ff5080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff519e00640 1 -- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 msgr2=0x7ff5080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff519e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff5240b4680 0x7ff5080051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff519e00640 1 -- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5080039b0 msgr2=0x7ff508003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff519e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5080039b0 0x7ff508003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.888+0000 7ff519e00640 1 -- 172.21.15.97:0/815640485 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508003520 con 0x7ff508004800 2024-03-06T07:34:09.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.889+0000 7ff519e00640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff508004c80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00b5c0 tx=0x7ff508005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6146a7d146d2b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.889+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c00b600 con 0x7ff508004800 2024-03-06T07:34:09.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c02f6f0 con 0x7ff508004800 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff521e00640 1 -- 172.21.15.97:0/815640485 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c02fac0 con 0x7ff508004800 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff529400640 1 -- 172.21.15.97:0/815640485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 msgr2=0x7ff508004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff529400640 1 --2- 172.21.15.97:0/815640485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff508004c80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00b5c0 tx=0x7ff508005830 comp rx=0 tx=0).stop 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff523200640 1 -- 172.21.15.97:0/815640485 reap_dead start 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff529400640 1 -- 172.21.15.97:0/815640485 shutdown_connections 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff529400640 1 -- 172.21.15.97:0/815640485 >> 172.21.15.97:0/815640485 conn(0x7ff5240b1730 msgr2=0x7ff5240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff529400640 1 -- 172.21.15.97:0/815640485 shutdown_connections 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.890+0000 7ff529400640 1 -- 172.21.15.97:0/815640485 wait complete. 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.892+0000 7ff529400640 1 Processor -- start 2024-03-06T07:34:09.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.892+0000 7ff529400640 1 -- start start 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff529400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff524156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff529400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52415d9c0 0x7ff5241548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff529400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff524146f90 0x7ff524147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff529400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240b4010 con 0x7ff508004800 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff529400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240b32b0 con 0x7ff524146f90 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff529400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240ccfb0 con 0x7ff52415d9c0 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff522800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff524156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff522800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff524156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52060/0 (socket says 172.21.15.97:52060) 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.893+0000 7ff522800640 1 -- 172.21.15.97:0/117107087 learned_addr learned my addr 172.21.15.97:0/117107087 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:09.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff522800640 1 -- 172.21.15.97:0/117107087 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52415d9c0 msgr2=0x7ff5241548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff522800640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52415d9c0 0x7ff5241548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff522800640 1 -- 172.21.15.97:0/117107087 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff524146f90 msgr2=0x7ff524147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff522800640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff524146f90 0x7ff524147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff522800640 1 -- 172.21.15.97:0/117107087 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524152dd0 con 0x7ff508004800 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff522800640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff524156ce0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff514001d00 tx=0x7ff51400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400c870 con 0x7ff508004800 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff514018070 con 0x7ff508004800 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400cc50 con 0x7ff508004800 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5241583f0 con 0x7ff508004800 2024-03-06T07:34:09.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.894+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff524157630 con 0x7ff508004800 2024-03-06T07:34:09.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.896+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52404f330 con 0x7ff508004800 2024-03-06T07:34:09.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.896+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff514013420 con 0x7ff508004800 2024-03-06T07:34:09.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.897+0000 7ff51b400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50006b500 0x7ff50006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.897+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff514089e00 con 0x7ff508004800 2024-03-06T07:34:09.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.898+0000 7ff519e00640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50006b500 0x7ff50006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.898+0000 7ff519e00640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50006b500 0x7ff50006d9c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff5080057e0 tx=0x7ff50c034000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:09.902+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff514058c70 con 0x7ff508004800 2024-03-06T07:34:10.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.129+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff52406e900 con 0x7ff508004800 2024-03-06T07:34:10.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.130+0000 7ff51b400640 1 -- 172.21.15.97:0/117107087 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4979 (secure 0 0 0) 0x7ff514058570 con 0x7ff508004800 2024-03-06T07:34:10.131 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:10.131 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1292572481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1292572481},{"type":"v1","addr":"172.21.15.135:6835","nonce":1292572481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7701,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/232111965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":232111965},{"type":"v1","addr":"172.21.15.97:6835","nonce":232111965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2296434777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2296434777},{"type":"v1","addr":"172.21.15.97:6837","nonce":2296434777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:09.132267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7716":{"gid":7716,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6833/322807700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":322807700},{"type":"v1","addr":"172.21.15.135:6833","nonce":322807700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7716,"qdb_cluster":[7716]},"id":8}]} 2024-03-06T07:34:10.131 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 74 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50006b500 msgr2=0x7ff50006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50006b500 0x7ff50006d9c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff5080057e0 tx=0x7ff50c034000 comp rx=0 tx=0).stop 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 msgr2=0x7ff524156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff524156ce0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff514001d00 tx=0x7ff51400ecd0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 shutdown_connections 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50006b500 0x7ff50006d9c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff524146f90 0x7ff524147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52415d9c0 0x7ff5241548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 --2- 172.21.15.97:0/117107087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff508004800 0x7ff524156ce0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 >> 172.21.15.97:0/117107087 conn(0x7ff5240b1730 msgr2=0x7ff5240b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.132+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 shutdown_connections 2024-03-06T07:34:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.133+0000 7ff529400640 1 -- 172.21.15.97:0/117107087 wait complete. 2024-03-06T07:34:10.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:10.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.358+0000 7fa83be00640 1 Processor -- start 2024-03-06T07:34:10.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.358+0000 7fa83be00640 1 -- start start 2024-03-06T07:34:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa83be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 0x7fa834167080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa83be00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa834160b70 con 0x7fa83415f770 2024-03-06T07:34:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa83be00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8340b3040 con 0x7fa8340b4390 2024-03-06T07:34:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa83be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83415fb50 con 0x7fa8340d09b0 2024-03-06T07:34:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa839400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa83415f770 0x7fa83415fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33730/0 (socket says 172.21.15.97:33730) 2024-03-06T07:34:10.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa839400640 1 -- 172.21.15.97:0/1013842908 learned_addr learned my addr 172.21.15.97:0/1013842908 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa839e00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 0x7fa834167080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.360+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331736560 0 0) 0x7fa834160b70 con 0x7fa83415f770 2024-03-06T07:34:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818003520 con 0x7fa83415f770 2024-03-06T07:34:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447215919 0 0) 0x7fa818003520 con 0x7fa83415f770 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 msgr2=0x7fa834167080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 0x7fa834167080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 >> v1:172.21.15.135:6789/0 conn(0x7fa8340b4390 legacy=0x7fa834160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340cd3e0 con 0x7fa83415f770 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa8240019f0 con 0x7fa83415f770 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 >> v1:172.21.15.97:6789/0 conn(0x7fa83415f770 legacy=0x7fa83415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 0x7fa8180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa818003fb0 0x7fa818004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818004d40 0x7fa8180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81800bb80 con 0x7fa818003fb0 2024-03-06T07:34:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81800bd30 con 0x7fa818004d40 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81800bf30 con 0x7fa8340d09b0 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.361+0000 7fa838a00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa818003fb0 0x7fa818004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa839400640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 0x7fa8180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa839e00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818004d40 0x7fa8180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa838a00640 1 -- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 msgr2=0x7fa8180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa838a00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340d09b0 0x7fa8180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa838a00640 1 -- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818004d40 msgr2=0x7fa8180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa838a00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818004d40 0x7fa8180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.362+0000 7fa838a00640 1 -- 172.21.15.97:0/1013842908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa818003520 con 0x7fa818003fb0 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.363+0000 7fa838a00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa818003fb0 0x7fa818004800 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fa820001720 tx=0x7fa818005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83e38e0791a0c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.363+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820001370 con 0x7fa818003fb0 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.363+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8200052d0 con 0x7fa818003fb0 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.363+0000 7fa833e00640 1 -- 172.21.15.97:0/1013842908 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820003830 con 0x7fa818003fb0 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa83be00640 1 -- 172.21.15.97:0/1013842908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa818003fb0 msgr2=0x7fa818004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa83be00640 1 --2- 172.21.15.97:0/1013842908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa818003fb0 0x7fa818004800 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fa820001720 tx=0x7fa818005830 comp rx=0 tx=0).stop 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa839e00640 1 -- 172.21.15.97:0/1013842908 reap_dead start 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa83be00640 1 -- 172.21.15.97:0/1013842908 shutdown_connections 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa83be00640 1 -- 172.21.15.97:0/1013842908 >> 172.21.15.97:0/1013842908 conn(0x7fa8340b3870 msgr2=0x7fa8340b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa83be00640 1 -- 172.21.15.97:0/1013842908 shutdown_connections 2024-03-06T07:34:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.364+0000 7fa83be00640 1 -- 172.21.15.97:0/1013842908 wait complete. 2024-03-06T07:34:10.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.365+0000 7fa83be00640 1 Processor -- start 2024-03-06T07:34:10.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 -- start start 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818003fb0 0x7fa8340c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 0x7fa83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340b5aa0 0x7fa83414eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340b3320 con 0x7fa8341f3cd0 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340b2cd0 con 0x7fa818003fb0 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.366+0000 7fa83be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340b34a0 con 0x7fa8340b5aa0 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 0x7fa83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 0x7fa83406d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52078/0 (socket says 172.21.15.97:52078) 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 -- 172.21.15.97:0/1381430683 learned_addr learned my addr 172.21.15.97:0/1381430683 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa839400640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818003fb0 0x7fa8340c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa839e00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340b5aa0 0x7fa83414eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 -- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340b5aa0 msgr2=0x7fa83414eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340b5aa0 0x7fa83414eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 -- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818003fb0 msgr2=0x7fa8340c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818003fb0 0x7fa8340c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 -- 172.21.15.97:0/1381430683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8341f44e0 con 0x7fa8341f3cd0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa839400640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818003fb0 0x7fa8340c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.367+0000 7fa838a00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 0x7fa83406d990 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa820008fd0 tx=0x7fa820001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.368+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820041070 con 0x7fa8341f3cd0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.368+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa820005480 con 0x7fa8341f3cd0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.368+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8341f47c0 con 0x7fa8341f3cd0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.368+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820007a30 con 0x7fa8341f3cd0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.368+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8340c7350 con 0x7fa8341f3cd0 2024-03-06T07:34:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.369+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8340c7160 con 0x7fa8341f3cd0 2024-03-06T07:34:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.370+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa820044070 con 0x7fa8341f3cd0 2024-03-06T07:34:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.371+0000 7fa831600640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.371+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8200ac190 con 0x7fa8341f3cd0 2024-03-06T07:34:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.371+0000 7fa839400640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.372+0000 7fa839400640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa81406b4d0 0x7fa81406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa824002230 tx=0x7fa8240012b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.375+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa82007b000 con 0x7fa8341f3cd0 2024-03-06T07:34:10.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.571+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8340ce130 con 0x7fa8341f3cd0 2024-03-06T07:34:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.573+0000 7fa831600640 1 -- 172.21.15.97:0/1381430683 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16615 (secure 0 0 0) 0x7fa82007a900 con 0x7fa8341f3cd0 2024-03-06T07:34:10.575 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:10.575 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":112,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:07.398719+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:10.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa81406b4d0 msgr2=0x7fa81406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa81406b4d0 0x7fa81406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa824002230 tx=0x7fa8240012b0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 msgr2=0x7fa83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 0x7fa83406d990 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa820008fd0 tx=0x7fa820001120 comp rx=0 tx=0).stop 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 shutdown_connections 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8340b5aa0 0x7fa83414eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8341f3cd0 0x7fa83406d990 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 --2- 172.21.15.97:0/1381430683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa818003fb0 0x7fa8340c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 >> 172.21.15.97:0/1381430683 conn(0x7fa8340b3870 msgr2=0x7fa83406ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 shutdown_connections 2024-03-06T07:34:10.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.576+0000 7fa83be00640 1 -- 172.21.15.97:0/1381430683 wait complete. 2024-03-06T07:34:10.595 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:34:10.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:10.796 INFO:tasks.ceph.mds.b.smithi135.stderr:2024-03-06T07:34:10.795+0000 7f8994a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:10.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.799+0000 7fa1ed000640 1 Processor -- start 2024-03-06T07:34:10.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.799+0000 7fa1ed000640 1 -- start start 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.800+0000 7fa1ed000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1e81561f0 0x7fa1e80ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.800+0000 7fa1ed000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80cc9d0 0x7fa1e815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.800+0000 7fa1ed000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e815c240 con 0x7fa1e80aebf0 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.800+0000 7fa1ed000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e815ce30 con 0x7fa1e81561f0 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.800+0000 7fa1ed000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e815c500 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.800+0000 7fa1e7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1e81561f0 0x7fa1e80ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1e81561f0 0x7fa1e80ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52084/0 (socket says 172.21.15.97:52084) 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7400640 1 -- 172.21.15.97:0/475714033 learned_addr learned my addr 172.21.15.97:0/475714033 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7e00640 1 --2- 172.21.15.97:0/475714033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80cc9d0 0x7fa1e815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7400640 1 -- 172.21.15.97:0/475714033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80cc9d0 msgr2=0x7fa1e815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7400640 1 --2- 172.21.15.97:0/475714033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80cc9d0 0x7fa1e815f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7400640 1 -- 172.21.15.97:0/475714033 >> v1:172.21.15.135:6789/0 conn(0x7fa1e80aebf0 legacy=0x7fa1e80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.801+0000 7fa1e7400640 1 -- 172.21.15.97:0/475714033 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e815c7e0 con 0x7fa1e81561f0 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.802+0000 7fa1e7400640 1 --2- 172.21.15.97:0/475714033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ae6b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009870 tx=0x7fa1d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0af1d8eaefe84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.802+0000 7fa1e6a00640 1 -- 172.21.15.97:0/475714033 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8035020 con 0x7fa1e81561f0 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.802+0000 7fa1e6a00640 1 -- 172.21.15.97:0/475714033 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d8031640 con 0x7fa1e81561f0 2024-03-06T07:34:10.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.802+0000 7fa1e6a00640 1 -- 172.21.15.97:0/475714033 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d802d040 con 0x7fa1e81561f0 2024-03-06T07:34:10.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 -- 172.21.15.97:0/475714033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e81561f0 msgr2=0x7fa1e80ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 --2- 172.21.15.97:0/475714033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ae6b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009870 tx=0x7fa1d802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 -- 172.21.15.97:0/475714033 shutdown_connections 2024-03-06T07:34:10.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 --2- 172.21.15.97:0/475714033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80cc9d0 0x7fa1e815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 --2- 172.21.15.97:0/475714033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ae6b0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 -- 172.21.15.97:0/475714033 >> 172.21.15.97:0/475714033 conn(0x7fa1e806afc0 msgr2=0x7fa1e806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 -- 172.21.15.97:0/475714033 shutdown_connections 2024-03-06T07:34:10.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.803+0000 7fa1ed000640 1 -- 172.21.15.97:0/475714033 wait complete. 2024-03-06T07:34:10.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.805+0000 7fa1ed000640 1 Processor -- start 2024-03-06T07:34:10.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.805+0000 7fa1ed000640 1 -- start start 2024-03-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1ed000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80aebf0 0x7fa1e81f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1ed000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 0x7fa1e80b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1ed000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1ed000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8156750 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1ed000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81565d0 con 0x7fa1e81561f0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1ed000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e815fd70 con 0x7fa1e80aebf0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 0x7fa1e80b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 0x7fa1e80b05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52092/0 (socket says 172.21.15.97:52092) 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 -- 172.21.15.97:0/3482381920 learned_addr learned my addr 172.21.15.97:0/3482381920 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1e7400640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80aebf0 0x7fa1e81f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1e7e00640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 -- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80aebf0 msgr2=0x7fa1e81f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80aebf0 0x7fa1e81f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 -- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1e81561f0 msgr2=0x7fa1e80ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.806+0000 7fa1dfe00640 1 -- 172.21.15.97:0/3482381920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e81586c0 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.807+0000 7fa1dfe00640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 0x7fa1e80b05d0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000c430 tx=0x7fa1d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.807+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0011790 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.807+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d0017070 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.807+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e81589a0 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.807+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0011bb0 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.807+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e8158f60 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.808+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e804f330 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.809+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1d001c460 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.810+0000 7fa1df400640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa1c406b4d0 0x7fa1c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.810+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1d008a060 con 0x7fa1e80cc9d0 2024-03-06T07:34:10.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.811+0000 7fa1e7400640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa1c406b4d0 0x7fa1c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.811+0000 7fa1e7400640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa1c406b4d0 0x7fa1c406d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8036000 tx=0x7fa1d8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:10.815+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1d0058f70 con 0x7fa1e80cc9d0 2024-03-06T07:34:11.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.004+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1e80b2b10 con 0x7fa1c406b4d0 2024-03-06T07:34:11.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.015+0000 7fa1df400640 1 -- 172.21.15.97:0/3482381920 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250482 (secure 0 0 0) 0x7fa1e80b2b10 con 0x7fa1c406b4d0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:11.021 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:34:11.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa1c406b4d0 msgr2=0x7fa1c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa1c406b4d0 0x7fa1c406d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8036000 tx=0x7fa1d8008030 comp rx=0 tx=0).stop 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 msgr2=0x7fa1e80b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 0x7fa1e80b05d0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000c430 tx=0x7fa1d000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 shutdown_connections 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa1c406b4d0 0x7fa1c406d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1e81561f0 0x7fa1e80ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e80cc9d0 0x7fa1e80b05d0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 --2- 172.21.15.97:0/3482381920 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa1e80aebf0 0x7fa1e81f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 >> 172.21.15.97:0/3482381920 conn(0x7fa1e806afc0 msgr2=0x7fa1e80b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 shutdown_connections 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.021+0000 7fa1ed000640 1 -- 172.21.15.97:0/3482381920 wait complete. 2024-03-06T07:34:11.041 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-06T07:34:09.018209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":168,"acting":168,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":294788,"kb_used_data":12024,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754679932,"statfs":{"total":773094113280,"available":772792250368,"internally_reserved":0,"allocated":12312576,"data_stored":5951165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.624687"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815768+0000","last_change":"2024-03-06T07:34:06.416542+0000","last_active":"2024-03-06T07:34:07.815768+0000","last_peered":"2024-03-06T07:34:07.815768+0000","last_clean":"2024-03-06T07:34:07.815768+0000","last_became_active":"2024-03-06T07:34:06.415731+0000","last_became_peered":"2024-03-06T07:34:06.415731+0000","last_unstale":"2024-03-06T07:34:07.815768+0000","last_undegraded":"2024-03-06T07:34:07.815768+0000","last_fullsized":"2024-03-06T07:34:07.815768+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:44.899228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814610+0000","last_change":"2024-03-06T07:34:06.448884+0000","last_active":"2024-03-06T07:34:07.814610+0000","last_peered":"2024-03-06T07:34:07.814610+0000","last_clean":"2024-03-06T07:34:07.814610+0000","last_became_active":"2024-03-06T07:34:06.448593+0000","last_became_peered":"2024-03-06T07:34:06.448593+0000","last_unstale":"2024-03-06T07:34:07.814610+0000","last_undegraded":"2024-03-06T07:34:07.814610+0000","last_fullsized":"2024-03-06T07:34:07.814610+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:52.689592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814599+0000","last_change":"2024-03-06T07:34:06.444981+0000","last_active":"2024-03-06T07:34:07.814599+0000","last_peered":"2024-03-06T07:34:07.814599+0000","last_clean":"2024-03-06T07:34:07.814599+0000","last_became_active":"2024-03-06T07:34:06.444566+0000","last_became_peered":"2024-03-06T07:34:06.444566+0000","last_unstale":"2024-03-06T07:34:07.814599+0000","last_undegraded":"2024-03-06T07:34:07.814599+0000","last_fullsized":"2024-03-06T07:34:07.814599+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:11.751702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814576+0000","last_change":"2024-03-06T07:34:06.430544+0000","last_active":"2024-03-06T07:34:07.814576+0000","last_peered":"2024-03-06T07:34:07.814576+0000","last_clean":"2024-03-06T07:34:07.814576+0000","last_became_active":"2024-03-06T07:34:06.430134+0000","last_became_peered":"2024-03-06T07:34:06.430134+0000","last_unstale":"2024-03-06T07:34:07.814576+0000","last_undegraded":"2024-03-06T07:34:07.814576+0000","last_fullsized":"2024-03-06T07:34:07.814576+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:58.160771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405287+0000","last_change":"2024-03-06T07:34:06.439755+0000","last_active":"2024-03-06T07:34:07.405287+0000","last_peered":"2024-03-06T07:34:07.405287+0000","last_clean":"2024-03-06T07:34:07.405287+0000","last_became_active":"2024-03-06T07:34:06.438929+0000","last_became_peered":"2024-03-06T07:34:06.438929+0000","last_unstale":"2024-03-06T07:34:07.405287+0000","last_undegraded":"2024-03-06T07:34:07.405287+0000","last_fullsized":"2024-03-06T07:34:07.405287+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:17.785135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412033+0000","last_change":"2024-03-06T07:34:06.427744+0000","last_active":"2024-03-06T07:34:07.412033+0000","last_peered":"2024-03-06T07:34:07.412033+0000","last_clean":"2024-03-06T07:34:07.412033+0000","last_became_active":"2024-03-06T07:34:06.426615+0000","last_became_peered":"2024-03-06T07:34:06.426615+0000","last_unstale":"2024-03-06T07:34:07.412033+0000","last_undegraded":"2024-03-06T07:34:07.412033+0000","last_fullsized":"2024-03-06T07:34:07.412033+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:48.822674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813808+0000","last_change":"2024-03-06T07:34:06.439245+0000","last_active":"2024-03-06T07:34:07.813808+0000","last_peered":"2024-03-06T07:34:07.813808+0000","last_clean":"2024-03-06T07:34:07.813808+0000","last_became_active":"2024-03-06T07:34:06.438968+0000","last_became_peered":"2024-03-06T07:34:06.438968+0000","last_unstale":"2024-03-06T07:34:07.813808+0000","last_undegraded":"2024-03-06T07:34:07.813808+0000","last_fullsized":"2024-03-06T07:34:07.813808+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:49.964864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815767+0000","last_change":"2024-03-06T07:34:06.431196+0000","last_active":"2024-03-06T07:34:07.815767+0000","last_peered":"2024-03-06T07:34:07.815767+0000","last_clean":"2024-03-06T07:34:07.815767+0000","last_became_active":"2024-03-06T07:34:06.430895+0000","last_became_peered":"2024-03-06T07:34:06.430895+0000","last_unstale":"2024-03-06T07:34:07.815767+0000","last_undegraded":"2024-03-06T07:34:07.815767+0000","last_fullsized":"2024-03-06T07:34:07.815767+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:08.661614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815791+0000","last_change":"2024-03-06T07:34:06.424035+0000","last_active":"2024-03-06T07:34:07.815791+0000","last_peered":"2024-03-06T07:34:07.815791+0000","last_clean":"2024-03-06T07:34:07.815791+0000","last_became_active":"2024-03-06T07:34:06.423713+0000","last_became_peered":"2024-03-06T07:34:06.423713+0000","last_unstale":"2024-03-06T07:34:07.815791+0000","last_undegraded":"2024-03-06T07:34:07.815791+0000","last_fullsized":"2024-03-06T07:34:07.815791+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:50.651648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814960+0000","last_change":"2024-03-06T07:34:06.422197+0000","last_active":"2024-03-06T07:34:07.814960+0000","last_peered":"2024-03-06T07:34:07.814960+0000","last_clean":"2024-03-06T07:34:07.814960+0000","last_became_active":"2024-03-06T07:34:06.421762+0000","last_became_peered":"2024-03-06T07:34:06.421762+0000","last_unstale":"2024-03-06T07:34:07.814960+0000","last_undegraded":"2024-03-06T07:34:07.814960+0000","last_fullsized":"2024-03-06T07:34:07.814960+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:40.672706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814042+0000","last_change":"2024-03-06T07:34:06.443659+0000","last_active":"2024-03-06T07:34:07.814042+0000","last_peered":"2024-03-06T07:34:07.814042+0000","last_clean":"2024-03-06T07:34:07.814042+0000","last_became_active":"2024-03-06T07:34:06.443266+0000","last_became_peered":"2024-03-06T07:34:06.443266+0000","last_unstale":"2024-03-06T07:34:07.814042+0000","last_undegraded":"2024-03-06T07:34:07.814042+0000","last_fullsized":"2024-03-06T07:34:07.814042+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:16.681929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813741+0000","last_change":"2024-03-06T07:34:06.444078+0000","last_active":"2024-03-06T07:34:07.813741+0000","last_peered":"2024-03-06T07:34:07.813741+0000","last_clean":"2024-03-06T07:34:07.813741+0000","last_became_active":"2024-03-06T07:34:06.443750+0000","last_became_peered":"2024-03-06T07:34:06.443750+0000","last_unstale":"2024-03-06T07:34:07.813741+0000","last_undegraded":"2024-03-06T07:34:07.813741+0000","last_fullsized":"2024-03-06T07:34:07.813741+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:31.638197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405256+0000","last_change":"2024-03-06T07:34:06.436981+0000","last_active":"2024-03-06T07:34:07.405256+0000","last_peered":"2024-03-06T07:34:07.405256+0000","last_clean":"2024-03-06T07:34:07.405256+0000","last_became_active":"2024-03-06T07:34:06.436269+0000","last_became_peered":"2024-03-06T07:34:06.436269+0000","last_unstale":"2024-03-06T07:34:07.405256+0000","last_undegraded":"2024-03-06T07:34:07.405256+0000","last_fullsized":"2024-03-06T07:34:07.405256+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:54.946568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812086+0000","last_change":"2024-03-06T07:34:06.438420+0000","last_active":"2024-03-06T07:34:07.812086+0000","last_peered":"2024-03-06T07:34:07.812086+0000","last_clean":"2024-03-06T07:34:07.812086+0000","last_became_active":"2024-03-06T07:34:06.438093+0000","last_became_peered":"2024-03-06T07:34:06.438093+0000","last_unstale":"2024-03-06T07:34:07.812086+0000","last_undegraded":"2024-03-06T07:34:07.812086+0000","last_fullsized":"2024-03-06T07:34:07.812086+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:26.571749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447231+0000","last_change":"2024-03-06T07:34:06.442997+0000","last_active":"2024-03-06T07:34:07.447231+0000","last_peered":"2024-03-06T07:34:07.447231+0000","last_clean":"2024-03-06T07:34:07.447231+0000","last_became_active":"2024-03-06T07:34:06.442711+0000","last_became_peered":"2024-03-06T07:34:06.442711+0000","last_unstale":"2024-03-06T07:34:07.447231+0000","last_undegraded":"2024-03-06T07:34:07.447231+0000","last_fullsized":"2024-03-06T07:34:07.447231+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:15.370733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814723+0000","last_change":"2024-03-06T07:34:06.444763+0000","last_active":"2024-03-06T07:34:07.814723+0000","last_peered":"2024-03-06T07:34:07.814723+0000","last_clean":"2024-03-06T07:34:07.814723+0000","last_became_active":"2024-03-06T07:34:06.444422+0000","last_became_peered":"2024-03-06T07:34:06.444422+0000","last_unstale":"2024-03-06T07:34:07.814723+0000","last_undegraded":"2024-03-06T07:34:07.814723+0000","last_fullsized":"2024-03-06T07:34:07.814723+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:47.658866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812998+0000","last_change":"2024-03-06T07:34:06.429677+0000","last_active":"2024-03-06T07:34:07.812998+0000","last_peered":"2024-03-06T07:34:07.812998+0000","last_clean":"2024-03-06T07:34:07.812998+0000","last_became_active":"2024-03-06T07:34:06.428739+0000","last_became_peered":"2024-03-06T07:34:06.428739+0000","last_unstale":"2024-03-06T07:34:07.812998+0000","last_undegraded":"2024-03-06T07:34:07.812998+0000","last_fullsized":"2024-03-06T07:34:07.812998+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:25.554662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813227+0000","last_change":"2024-03-06T07:34:06.440036+0000","last_active":"2024-03-06T07:34:07.813227+0000","last_peered":"2024-03-06T07:34:07.813227+0000","last_clean":"2024-03-06T07:34:07.813227+0000","last_became_active":"2024-03-06T07:34:06.439740+0000","last_became_peered":"2024-03-06T07:34:06.439740+0000","last_unstale":"2024-03-06T07:34:07.813227+0000","last_undegraded":"2024-03-06T07:34:07.813227+0000","last_fullsized":"2024-03-06T07:34:07.813227+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:55.595831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815846+0000","last_change":"2024-03-06T07:34:06.439211+0000","last_active":"2024-03-06T07:34:07.815846+0000","last_peered":"2024-03-06T07:34:07.815846+0000","last_clean":"2024-03-06T07:34:07.815846+0000","last_became_active":"2024-03-06T07:34:06.438886+0000","last_became_peered":"2024-03-06T07:34:06.438886+0000","last_unstale":"2024-03-06T07:34:07.815846+0000","last_undegraded":"2024-03-06T07:34:07.815846+0000","last_fullsized":"2024-03-06T07:34:07.815846+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:10.815713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406363+0000","last_change":"2024-03-06T07:34:06.440484+0000","last_active":"2024-03-06T07:34:07.406363+0000","last_peered":"2024-03-06T07:34:07.406363+0000","last_clean":"2024-03-06T07:34:07.406363+0000","last_became_active":"2024-03-06T07:34:06.440210+0000","last_became_peered":"2024-03-06T07:34:06.440210+0000","last_unstale":"2024-03-06T07:34:07.406363+0000","last_undegraded":"2024-03-06T07:34:07.406363+0000","last_fullsized":"2024-03-06T07:34:07.406363+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:24.819319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813602+0000","last_change":"2024-03-06T07:34:06.428880+0000","last_active":"2024-03-06T07:34:07.813602+0000","last_peered":"2024-03-06T07:34:07.813602+0000","last_clean":"2024-03-06T07:34:07.813602+0000","last_became_active":"2024-03-06T07:34:06.427599+0000","last_became_peered":"2024-03-06T07:34:06.427599+0000","last_unstale":"2024-03-06T07:34:07.813602+0000","last_undegraded":"2024-03-06T07:34:07.813602+0000","last_fullsized":"2024-03-06T07:34:07.813602+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:26.571749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411662+0000","last_change":"2024-03-06T07:34:06.436047+0000","last_active":"2024-03-06T07:34:07.411662+0000","last_peered":"2024-03-06T07:34:07.411662+0000","last_clean":"2024-03-06T07:34:07.411662+0000","last_became_active":"2024-03-06T07:34:06.435697+0000","last_became_peered":"2024-03-06T07:34:06.435697+0000","last_unstale":"2024-03-06T07:34:07.411662+0000","last_undegraded":"2024-03-06T07:34:07.411662+0000","last_fullsized":"2024-03-06T07:34:07.411662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:21.016281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812942+0000","last_change":"2024-03-06T07:34:06.438134+0000","last_active":"2024-03-06T07:34:07.812942+0000","last_peered":"2024-03-06T07:34:07.812942+0000","last_clean":"2024-03-06T07:34:07.812942+0000","last_became_active":"2024-03-06T07:34:06.436496+0000","last_became_peered":"2024-03-06T07:34:06.436496+0000","last_unstale":"2024-03-06T07:34:07.812942+0000","last_undegraded":"2024-03-06T07:34:07.812942+0000","last_fullsized":"2024-03-06T07:34:07.812942+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:48.060635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405175+0000","last_change":"2024-03-06T07:34:06.433877+0000","last_active":"2024-03-06T07:34:07.405175+0000","last_peered":"2024-03-06T07:34:07.405175+0000","last_clean":"2024-03-06T07:34:07.405175+0000","last_became_active":"2024-03-06T07:34:06.433579+0000","last_became_peered":"2024-03-06T07:34:06.433579+0000","last_unstale":"2024-03-06T07:34:07.405175+0000","last_undegraded":"2024-03-06T07:34:07.405175+0000","last_fullsized":"2024-03-06T07:34:07.405175+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:34.209846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406174+0000","last_change":"2024-03-06T07:34:06.436766+0000","last_active":"2024-03-06T07:34:07.406174+0000","last_peered":"2024-03-06T07:34:07.406174+0000","last_clean":"2024-03-06T07:34:07.406174+0000","last_became_active":"2024-03-06T07:34:06.435876+0000","last_became_peered":"2024-03-06T07:34:06.435876+0000","last_unstale":"2024-03-06T07:34:07.406174+0000","last_undegraded":"2024-03-06T07:34:07.406174+0000","last_fullsized":"2024-03-06T07:34:07.406174+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:34.493622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"111'4","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411530+0000","last_change":"2024-03-06T07:34:05.437699+0000","last_active":"2024-03-06T07:34:07.411530+0000","last_peered":"2024-03-06T07:34:07.411530+0000","last_clean":"2024-03-06T07:34:07.411530+0000","last_became_active":"2024-03-06T07:34:05.437396+0000","last_became_peered":"2024-03-06T07:34:05.437396+0000","last_unstale":"2024-03-06T07:34:07.411530+0000","last_undegraded":"2024-03-06T07:34:07.411530+0000","last_fullsized":"2024-03-06T07:34:07.411530+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:33.477137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447590+0000","last_change":"2024-03-06T07:34:06.438312+0000","last_active":"2024-03-06T07:34:07.447590+0000","last_peered":"2024-03-06T07:34:07.447590+0000","last_clean":"2024-03-06T07:34:07.447590+0000","last_became_active":"2024-03-06T07:34:06.437349+0000","last_became_peered":"2024-03-06T07:34:06.437349+0000","last_unstale":"2024-03-06T07:34:07.447590+0000","last_undegraded":"2024-03-06T07:34:07.447590+0000","last_fullsized":"2024-03-06T07:34:07.447590+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:50.851331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814364+0000","last_change":"2024-03-06T07:34:05.441112+0000","last_active":"2024-03-06T07:34:07.814364+0000","last_peered":"2024-03-06T07:34:07.814364+0000","last_clean":"2024-03-06T07:34:07.814364+0000","last_became_active":"2024-03-06T07:34:05.440635+0000","last_became_peered":"2024-03-06T07:34:05.440635+0000","last_unstale":"2024-03-06T07:34:07.814364+0000","last_undegraded":"2024-03-06T07:34:07.814364+0000","last_fullsized":"2024-03-06T07:34:07.814364+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:28.950051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406141+0000","last_change":"2024-03-06T07:34:06.436453+0000","last_active":"2024-03-06T07:34:07.406141+0000","last_peered":"2024-03-06T07:34:07.406141+0000","last_clean":"2024-03-06T07:34:07.406141+0000","last_became_active":"2024-03-06T07:34:06.436199+0000","last_became_peered":"2024-03-06T07:34:06.436199+0000","last_unstale":"2024-03-06T07:34:07.406141+0000","last_undegraded":"2024-03-06T07:34:07.406141+0000","last_fullsized":"2024-03-06T07:34:07.406141+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:36.576401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814975+0000","last_change":"2024-03-06T07:34:05.433032+0000","last_active":"2024-03-06T07:34:07.814975+0000","last_peered":"2024-03-06T07:34:07.814975+0000","last_clean":"2024-03-06T07:34:07.814975+0000","last_became_active":"2024-03-06T07:34:05.432735+0000","last_became_peered":"2024-03-06T07:34:05.432735+0000","last_unstale":"2024-03-06T07:34:07.814975+0000","last_undegraded":"2024-03-06T07:34:07.814975+0000","last_fullsized":"2024-03-06T07:34:07.814975+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:04.579509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814824+0000","last_change":"2024-03-06T07:34:06.445008+0000","last_active":"2024-03-06T07:34:07.814824+0000","last_peered":"2024-03-06T07:34:07.814824+0000","last_clean":"2024-03-06T07:34:07.814824+0000","last_became_active":"2024-03-06T07:34:06.444643+0000","last_became_peered":"2024-03-06T07:34:06.444643+0000","last_unstale":"2024-03-06T07:34:07.814824+0000","last_undegraded":"2024-03-06T07:34:07.814824+0000","last_fullsized":"2024-03-06T07:34:07.814824+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:17.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813704+0000","last_change":"2024-03-06T07:34:05.433936+0000","last_active":"2024-03-06T07:34:07.813704+0000","last_peered":"2024-03-06T07:34:07.813704+0000","last_clean":"2024-03-06T07:34:07.813704+0000","last_became_active":"2024-03-06T07:34:05.433225+0000","last_became_peered":"2024-03-06T07:34:05.433225+0000","last_unstale":"2024-03-06T07:34:07.813704+0000","last_undegraded":"2024-03-06T07:34:07.813704+0000","last_fullsized":"2024-03-06T07:34:07.813704+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:49.850269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814290+0000","last_change":"2024-03-06T07:34:06.432720+0000","last_active":"2024-03-06T07:34:07.814290+0000","last_peered":"2024-03-06T07:34:07.814290+0000","last_clean":"2024-03-06T07:34:07.814290+0000","last_became_active":"2024-03-06T07:34:06.432447+0000","last_became_peered":"2024-03-06T07:34:06.432447+0000","last_unstale":"2024-03-06T07:34:07.814290+0000","last_undegraded":"2024-03-06T07:34:07.814290+0000","last_fullsized":"2024-03-06T07:34:07.814290+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:18.987781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413711+0000","last_change":"2024-03-06T07:34:05.437729+0000","last_active":"2024-03-06T07:34:07.413711+0000","last_peered":"2024-03-06T07:34:07.413711+0000","last_clean":"2024-03-06T07:34:07.413711+0000","last_became_active":"2024-03-06T07:34:05.437353+0000","last_became_peered":"2024-03-06T07:34:05.437353+0000","last_unstale":"2024-03-06T07:34:07.413711+0000","last_undegraded":"2024-03-06T07:34:07.413711+0000","last_fullsized":"2024-03-06T07:34:07.413711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:23.802834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813253+0000","last_change":"2024-03-06T07:34:05.437791+0000","last_active":"2024-03-06T07:34:07.813253+0000","last_peered":"2024-03-06T07:34:07.813253+0000","last_clean":"2024-03-06T07:34:07.813253+0000","last_became_active":"2024-03-06T07:34:05.437497+0000","last_became_peered":"2024-03-06T07:34:05.437497+0000","last_unstale":"2024-03-06T07:34:07.813253+0000","last_undegraded":"2024-03-06T07:34:07.813253+0000","last_fullsized":"2024-03-06T07:34:07.813253+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:49.834846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812013+0000","last_change":"2024-03-06T07:34:06.435648+0000","last_active":"2024-03-06T07:34:07.812013+0000","last_peered":"2024-03-06T07:34:07.812013+0000","last_clean":"2024-03-06T07:34:07.812013+0000","last_became_active":"2024-03-06T07:34:06.435390+0000","last_became_peered":"2024-03-06T07:34:06.435390+0000","last_unstale":"2024-03-06T07:34:07.812013+0000","last_undegraded":"2024-03-06T07:34:07.812013+0000","last_fullsized":"2024-03-06T07:34:07.812013+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:55.595831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:04.375306+0000","last_change":"2024-03-06T07:34:04.375306+0000","last_active":"2024-03-06T07:34:04.375306+0000","last_peered":"2024-03-06T07:34:04.375306+0000","last_clean":"2024-03-06T07:34:04.375306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:04.375306+0000","last_undegraded":"2024-03-06T07:34:04.375306+0000","last_fullsized":"2024-03-06T07:34:04.375306+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411755+0000","last_change":"2024-03-06T07:34:06.427236+0000","last_active":"2024-03-06T07:34:07.411755+0000","last_peered":"2024-03-06T07:34:07.411755+0000","last_clean":"2024-03-06T07:34:07.411755+0000","last_became_active":"2024-03-06T07:34:06.425479+0000","last_became_peered":"2024-03-06T07:34:06.425479+0000","last_unstale":"2024-03-06T07:34:07.411755+0000","last_undegraded":"2024-03-06T07:34:07.411755+0000","last_fullsized":"2024-03-06T07:34:07.411755+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:41.399474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812211+0000","last_change":"2024-03-06T07:34:05.431239+0000","last_active":"2024-03-06T07:34:07.812211+0000","last_peered":"2024-03-06T07:34:07.812211+0000","last_clean":"2024-03-06T07:34:07.812211+0000","last_became_active":"2024-03-06T07:34:05.430738+0000","last_became_peered":"2024-03-06T07:34:05.430738+0000","last_unstale":"2024-03-06T07:34:07.812211+0000","last_undegraded":"2024-03-06T07:34:07.812211+0000","last_fullsized":"2024-03-06T07:34:07.812211+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:14.354248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.414955+0000","last_change":"2024-03-06T07:34:06.431283+0000","last_active":"2024-03-06T07:34:07.414955+0000","last_peered":"2024-03-06T07:34:07.414955+0000","last_clean":"2024-03-06T07:34:07.414955+0000","last_became_active":"2024-03-06T07:34:06.431022+0000","last_became_peered":"2024-03-06T07:34:06.431022+0000","last_unstale":"2024-03-06T07:34:07.414955+0000","last_undegraded":"2024-03-06T07:34:07.414955+0000","last_fullsized":"2024-03-06T07:34:07.414955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:37.885605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813866+0000","last_change":"2024-03-06T07:34:05.434115+0000","last_active":"2024-03-06T07:34:07.813866+0000","last_peered":"2024-03-06T07:34:07.813866+0000","last_clean":"2024-03-06T07:34:07.813866+0000","last_became_active":"2024-03-06T07:34:05.432812+0000","last_became_peered":"2024-03-06T07:34:05.432812+0000","last_unstale":"2024-03-06T07:34:07.813866+0000","last_undegraded":"2024-03-06T07:34:07.813866+0000","last_fullsized":"2024-03-06T07:34:07.813866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:01.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447726+0000","last_change":"2024-03-06T07:34:06.421717+0000","last_active":"2024-03-06T07:34:07.447726+0000","last_peered":"2024-03-06T07:34:07.447726+0000","last_clean":"2024-03-06T07:34:07.447726+0000","last_became_active":"2024-03-06T07:34:06.420868+0000","last_became_peered":"2024-03-06T07:34:06.420868+0000","last_unstale":"2024-03-06T07:34:07.447726+0000","last_undegraded":"2024-03-06T07:34:07.447726+0000","last_fullsized":"2024-03-06T07:34:07.447726+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:44.188808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812562+0000","last_change":"2024-03-06T07:34:05.437594+0000","last_active":"2024-03-06T07:34:07.812562+0000","last_peered":"2024-03-06T07:34:07.812562+0000","last_clean":"2024-03-06T07:34:07.812562+0000","last_became_active":"2024-03-06T07:34:05.436227+0000","last_became_peered":"2024-03-06T07:34:05.436227+0000","last_unstale":"2024-03-06T07:34:07.812562+0000","last_undegraded":"2024-03-06T07:34:07.812562+0000","last_fullsized":"2024-03-06T07:34:07.812562+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.846568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813994+0000","last_change":"2024-03-06T07:34:06.444755+0000","last_active":"2024-03-06T07:34:07.813994+0000","last_peered":"2024-03-06T07:34:07.813994+0000","last_clean":"2024-03-06T07:34:07.813994+0000","last_became_active":"2024-03-06T07:34:06.444479+0000","last_became_peered":"2024-03-06T07:34:06.444479+0000","last_unstale":"2024-03-06T07:34:07.813994+0000","last_undegraded":"2024-03-06T07:34:07.813994+0000","last_fullsized":"2024-03-06T07:34:07.813994+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:48.060635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"111'3","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815224+0000","last_change":"2024-03-06T07:34:05.429203+0000","last_active":"2024-03-06T07:34:07.815224+0000","last_peered":"2024-03-06T07:34:07.815224+0000","last_clean":"2024-03-06T07:34:07.815224+0000","last_became_active":"2024-03-06T07:34:05.428889+0000","last_became_peered":"2024-03-06T07:34:05.428889+0000","last_unstale":"2024-03-06T07:34:07.815224+0000","last_undegraded":"2024-03-06T07:34:07.815224+0000","last_fullsized":"2024-03-06T07:34:07.815224+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:24.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413598+0000","last_change":"2024-03-06T07:34:06.437689+0000","last_active":"2024-03-06T07:34:07.413598+0000","last_peered":"2024-03-06T07:34:07.413598+0000","last_clean":"2024-03-06T07:34:07.413598+0000","last_became_active":"2024-03-06T07:34:06.436768+0000","last_became_peered":"2024-03-06T07:34:06.436768+0000","last_unstale":"2024-03-06T07:34:07.413598+0000","last_undegraded":"2024-03-06T07:34:07.413598+0000","last_fullsized":"2024-03-06T07:34:07.413598+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:53.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"111'6","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815111+0000","last_change":"2024-03-06T07:34:05.435622+0000","last_active":"2024-03-06T07:34:07.815111+0000","last_peered":"2024-03-06T07:34:07.815111+0000","last_clean":"2024-03-06T07:34:07.815111+0000","last_became_active":"2024-03-06T07:34:05.435353+0000","last_became_peered":"2024-03-06T07:34:05.435353+0000","last_unstale":"2024-03-06T07:34:07.815111+0000","last_undegraded":"2024-03-06T07:34:07.815111+0000","last_fullsized":"2024-03-06T07:34:07.815111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.172323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813008+0000","last_change":"2024-03-06T07:34:06.439885+0000","last_active":"2024-03-06T07:34:07.813008+0000","last_peered":"2024-03-06T07:34:07.813008+0000","last_clean":"2024-03-06T07:34:07.813008+0000","last_became_active":"2024-03-06T07:34:06.439512+0000","last_became_peered":"2024-03-06T07:34:06.439512+0000","last_unstale":"2024-03-06T07:34:07.813008+0000","last_undegraded":"2024-03-06T07:34:07.813008+0000","last_fullsized":"2024-03-06T07:34:07.813008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:14.292757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405808+0000","last_change":"2024-03-06T07:34:05.438753+0000","last_active":"2024-03-06T07:34:07.405808+0000","last_peered":"2024-03-06T07:34:07.405808+0000","last_clean":"2024-03-06T07:34:07.405808+0000","last_became_active":"2024-03-06T07:34:05.437002+0000","last_became_peered":"2024-03-06T07:34:05.437002+0000","last_unstale":"2024-03-06T07:34:07.405808+0000","last_undegraded":"2024-03-06T07:34:07.405808+0000","last_fullsized":"2024-03-06T07:34:07.405808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.506880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812940+0000","last_change":"2024-03-06T07:34:06.435548+0000","last_active":"2024-03-06T07:34:07.812940+0000","last_peered":"2024-03-06T07:34:07.812940+0000","last_clean":"2024-03-06T07:34:07.812940+0000","last_became_active":"2024-03-06T07:34:06.435258+0000","last_became_peered":"2024-03-06T07:34:06.435258+0000","last_unstale":"2024-03-06T07:34:07.812940+0000","last_undegraded":"2024-03-06T07:34:07.812940+0000","last_fullsized":"2024-03-06T07:34:07.812940+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:20.152660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.446818+0000","last_change":"2024-03-06T07:34:05.437038+0000","last_active":"2024-03-06T07:34:07.446818+0000","last_peered":"2024-03-06T07:34:07.446818+0000","last_clean":"2024-03-06T07:34:07.446818+0000","last_became_active":"2024-03-06T07:34:05.436728+0000","last_became_peered":"2024-03-06T07:34:05.436728+0000","last_unstale":"2024-03-06T07:34:07.446818+0000","last_undegraded":"2024-03-06T07:34:07.446818+0000","last_fullsized":"2024-03-06T07:34:07.446818+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:38.143183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812746+0000","last_change":"2024-03-06T07:34:06.444516+0000","last_active":"2024-03-06T07:34:07.812746+0000","last_peered":"2024-03-06T07:34:07.812746+0000","last_clean":"2024-03-06T07:34:07.812746+0000","last_became_active":"2024-03-06T07:34:06.443735+0000","last_became_peered":"2024-03-06T07:34:06.443735+0000","last_unstale":"2024-03-06T07:34:07.812746+0000","last_undegraded":"2024-03-06T07:34:07.812746+0000","last_fullsized":"2024-03-06T07:34:07.812746+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:40.729083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"111'8","reported_seq":22,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813740+0000","last_change":"2024-03-06T07:34:05.418144+0000","last_active":"2024-03-06T07:34:07.813740+0000","last_peered":"2024-03-06T07:34:07.813740+0000","last_clean":"2024-03-06T07:34:07.813740+0000","last_became_active":"2024-03-06T07:34:05.417486+0000","last_became_peered":"2024-03-06T07:34:05.417486+0000","last_unstale":"2024-03-06T07:34:07.813740+0000","last_undegraded":"2024-03-06T07:34:07.813740+0000","last_fullsized":"2024-03-06T07:34:07.813740+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:57.782081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814303+0000","last_change":"2024-03-06T07:34:06.422711+0000","last_active":"2024-03-06T07:34:07.814303+0000","last_peered":"2024-03-06T07:34:07.814303+0000","last_clean":"2024-03-06T07:34:07.814303+0000","last_became_active":"2024-03-06T07:34:06.421347+0000","last_became_peered":"2024-03-06T07:34:06.421347+0000","last_unstale":"2024-03-06T07:34:07.814303+0000","last_undegraded":"2024-03-06T07:34:07.814303+0000","last_fullsized":"2024-03-06T07:34:07.814303+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:53.685828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815959+0000","last_change":"2024-03-06T07:34:05.441377+0000","last_active":"2024-03-06T07:34:07.815959+0000","last_peered":"2024-03-06T07:34:07.815959+0000","last_clean":"2024-03-06T07:34:07.815959+0000","last_became_active":"2024-03-06T07:34:05.438203+0000","last_became_peered":"2024-03-06T07:34:05.438203+0000","last_unstale":"2024-03-06T07:34:07.815959+0000","last_undegraded":"2024-03-06T07:34:07.815959+0000","last_fullsized":"2024-03-06T07:34:07.815959+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.846568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405293+0000","last_change":"2024-03-06T07:34:06.445211+0000","last_active":"2024-03-06T07:34:07.405293+0000","last_peered":"2024-03-06T07:34:07.405293+0000","last_clean":"2024-03-06T07:34:07.405293+0000","last_became_active":"2024-03-06T07:34:06.444820+0000","last_became_peered":"2024-03-06T07:34:06.444820+0000","last_unstale":"2024-03-06T07:34:07.405293+0000","last_undegraded":"2024-03-06T07:34:07.405293+0000","last_fullsized":"2024-03-06T07:34:07.405293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:52.841179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814417+0000","last_change":"2024-03-06T07:34:05.431742+0000","last_active":"2024-03-06T07:34:07.814417+0000","last_peered":"2024-03-06T07:34:07.814417+0000","last_clean":"2024-03-06T07:34:07.814417+0000","last_became_active":"2024-03-06T07:34:05.431441+0000","last_became_peered":"2024-03-06T07:34:05.431441+0000","last_unstale":"2024-03-06T07:34:07.814417+0000","last_undegraded":"2024-03-06T07:34:07.814417+0000","last_fullsized":"2024-03-06T07:34:07.814417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:53.533773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814394+0000","last_change":"2024-03-06T07:34:06.438884+0000","last_active":"2024-03-06T07:34:07.814394+0000","last_peered":"2024-03-06T07:34:07.814394+0000","last_clean":"2024-03-06T07:34:07.814394+0000","last_became_active":"2024-03-06T07:34:06.438607+0000","last_became_peered":"2024-03-06T07:34:06.438607+0000","last_unstale":"2024-03-06T07:34:07.814394+0000","last_undegraded":"2024-03-06T07:34:07.814394+0000","last_fullsized":"2024-03-06T07:34:07.814394+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:34.513428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.446979+0000","last_change":"2024-03-06T07:34:05.435463+0000","last_active":"2024-03-06T07:34:07.446979+0000","last_peered":"2024-03-06T07:34:07.446979+0000","last_clean":"2024-03-06T07:34:07.446979+0000","last_became_active":"2024-03-06T07:34:05.435060+0000","last_became_peered":"2024-03-06T07:34:05.435060+0000","last_unstale":"2024-03-06T07:34:07.446979+0000","last_undegraded":"2024-03-06T07:34:07.446979+0000","last_fullsized":"2024-03-06T07:34:07.446979+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:28.950051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814937+0000","last_change":"2024-03-06T07:34:06.438725+0000","last_active":"2024-03-06T07:34:07.814937+0000","last_peered":"2024-03-06T07:34:07.814937+0000","last_clean":"2024-03-06T07:34:07.814937+0000","last_became_active":"2024-03-06T07:34:06.438416+0000","last_became_peered":"2024-03-06T07:34:06.438416+0000","last_unstale":"2024-03-06T07:34:07.814937+0000","last_undegraded":"2024-03-06T07:34:07.814937+0000","last_fullsized":"2024-03-06T07:34:07.814937+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:40.729083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411086+0000","last_change":"2024-03-06T07:34:05.437663+0000","last_active":"2024-03-06T07:34:07.411086+0000","last_peered":"2024-03-06T07:34:07.411086+0000","last_clean":"2024-03-06T07:34:07.411086+0000","last_became_active":"2024-03-06T07:34:05.437248+0000","last_became_peered":"2024-03-06T07:34:05.437248+0000","last_unstale":"2024-03-06T07:34:07.411086+0000","last_undegraded":"2024-03-06T07:34:07.411086+0000","last_fullsized":"2024-03-06T07:34:07.411086+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:35.559916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813835+0000","last_change":"2024-03-06T07:34:06.441881+0000","last_active":"2024-03-06T07:34:07.813835+0000","last_peered":"2024-03-06T07:34:07.813835+0000","last_clean":"2024-03-06T07:34:07.813835+0000","last_became_active":"2024-03-06T07:34:06.441102+0000","last_became_peered":"2024-03-06T07:34:06.441102+0000","last_unstale":"2024-03-06T07:34:07.813835+0000","last_undegraded":"2024-03-06T07:34:07.813835+0000","last_fullsized":"2024-03-06T07:34:07.813835+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:34.429264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405933+0000","last_change":"2024-03-06T07:34:05.424548+0000","last_active":"2024-03-06T07:34:07.405933+0000","last_peered":"2024-03-06T07:34:07.405933+0000","last_clean":"2024-03-06T07:34:07.405933+0000","last_became_active":"2024-03-06T07:34:05.424027+0000","last_became_peered":"2024-03-06T07:34:05.424027+0000","last_unstale":"2024-03-06T07:34:07.405933+0000","last_undegraded":"2024-03-06T07:34:07.405933+0000","last_fullsized":"2024-03-06T07:34:07.405933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:49.850269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813472+0000","last_change":"2024-03-06T07:34:06.438719+0000","last_active":"2024-03-06T07:34:07.813472+0000","last_peered":"2024-03-06T07:34:07.813472+0000","last_clean":"2024-03-06T07:34:07.813472+0000","last_became_active":"2024-03-06T07:34:06.438418+0000","last_became_peered":"2024-03-06T07:34:06.438418+0000","last_unstale":"2024-03-06T07:34:07.813472+0000","last_undegraded":"2024-03-06T07:34:07.813472+0000","last_fullsized":"2024-03-06T07:34:07.813472+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:13.063799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813526+0000","last_change":"2024-03-06T07:34:05.433718+0000","last_active":"2024-03-06T07:34:07.813526+0000","last_peered":"2024-03-06T07:34:07.813526+0000","last_clean":"2024-03-06T07:34:07.813526+0000","last_became_active":"2024-03-06T07:34:05.433428+0000","last_became_peered":"2024-03-06T07:34:05.433428+0000","last_unstale":"2024-03-06T07:34:07.813526+0000","last_undegraded":"2024-03-06T07:34:07.813526+0000","last_fullsized":"2024-03-06T07:34:07.813526+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:31.381706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813944+0000","last_change":"2024-03-06T07:34:06.445204+0000","last_active":"2024-03-06T07:34:07.813944+0000","last_peered":"2024-03-06T07:34:07.813944+0000","last_clean":"2024-03-06T07:34:07.813944+0000","last_became_active":"2024-03-06T07:34:06.443409+0000","last_became_peered":"2024-03-06T07:34:06.443409+0000","last_unstale":"2024-03-06T07:34:07.813944+0000","last_undegraded":"2024-03-06T07:34:07.813944+0000","last_fullsized":"2024-03-06T07:34:07.813944+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:51.065725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406010+0000","last_change":"2024-03-06T07:34:05.430948+0000","last_active":"2024-03-06T07:34:07.406010+0000","last_peered":"2024-03-06T07:34:07.406010+0000","last_clean":"2024-03-06T07:34:07.406010+0000","last_became_active":"2024-03-06T07:34:05.429980+0000","last_became_peered":"2024-03-06T07:34:05.429980+0000","last_unstale":"2024-03-06T07:34:07.406010+0000","last_undegraded":"2024-03-06T07:34:07.406010+0000","last_fullsized":"2024-03-06T07:34:07.406010+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:01.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812706+0000","last_change":"2024-03-06T07:34:06.429605+0000","last_active":"2024-03-06T07:34:07.812706+0000","last_peered":"2024-03-06T07:34:07.812706+0000","last_clean":"2024-03-06T07:34:07.812706+0000","last_became_active":"2024-03-06T07:34:06.427590+0000","last_became_peered":"2024-03-06T07:34:06.427590+0000","last_unstale":"2024-03-06T07:34:07.812706+0000","last_undegraded":"2024-03-06T07:34:07.812706+0000","last_fullsized":"2024-03-06T07:34:07.812706+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:17.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812233+0000","last_change":"2024-03-06T07:34:05.438560+0000","last_active":"2024-03-06T07:34:07.812233+0000","last_peered":"2024-03-06T07:34:07.812233+0000","last_clean":"2024-03-06T07:34:07.812233+0000","last_became_active":"2024-03-06T07:34:05.438214+0000","last_became_peered":"2024-03-06T07:34:05.438214+0000","last_unstale":"2024-03-06T07:34:07.812233+0000","last_undegraded":"2024-03-06T07:34:07.812233+0000","last_fullsized":"2024-03-06T07:34:07.812233+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:49.834846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447269+0000","last_change":"2024-03-06T07:34:06.427612+0000","last_active":"2024-03-06T07:34:07.447269+0000","last_peered":"2024-03-06T07:34:07.447269+0000","last_clean":"2024-03-06T07:34:07.447269+0000","last_became_active":"2024-03-06T07:34:06.427089+0000","last_became_peered":"2024-03-06T07:34:06.427089+0000","last_unstale":"2024-03-06T07:34:07.447269+0000","last_undegraded":"2024-03-06T07:34:07.447269+0000","last_fullsized":"2024-03-06T07:34:07.447269+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:31.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815370+0000","last_change":"2024-03-06T07:34:05.440223+0000","last_active":"2024-03-06T07:34:07.815370+0000","last_peered":"2024-03-06T07:34:07.815370+0000","last_clean":"2024-03-06T07:34:07.815370+0000","last_became_active":"2024-03-06T07:34:05.437938+0000","last_became_peered":"2024-03-06T07:34:05.437938+0000","last_unstale":"2024-03-06T07:34:07.815370+0000","last_undegraded":"2024-03-06T07:34:07.815370+0000","last_fullsized":"2024-03-06T07:34:07.815370+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:02.228169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815947+0000","last_change":"2024-03-06T07:34:06.442205+0000","last_active":"2024-03-06T07:34:07.815947+0000","last_peered":"2024-03-06T07:34:07.815947+0000","last_clean":"2024-03-06T07:34:07.815947+0000","last_became_active":"2024-03-06T07:34:06.441839+0000","last_became_peered":"2024-03-06T07:34:06.441839+0000","last_unstale":"2024-03-06T07:34:07.815947+0000","last_undegraded":"2024-03-06T07:34:07.815947+0000","last_fullsized":"2024-03-06T07:34:07.815947+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:54.121154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.450412+0000","last_change":"2024-03-06T07:34:05.442001+0000","last_active":"2024-03-06T07:34:07.450412+0000","last_peered":"2024-03-06T07:34:07.450412+0000","last_clean":"2024-03-06T07:34:07.450412+0000","last_became_active":"2024-03-06T07:34:05.441051+0000","last_became_peered":"2024-03-06T07:34:05.441051+0000","last_unstale":"2024-03-06T07:34:07.450412+0000","last_undegraded":"2024-03-06T07:34:07.450412+0000","last_fullsized":"2024-03-06T07:34:07.450412+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:50.867356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814910+0000","last_change":"2024-03-06T07:34:06.439451+0000","last_active":"2024-03-06T07:34:07.814910+0000","last_peered":"2024-03-06T07:34:07.814910+0000","last_clean":"2024-03-06T07:34:07.814910+0000","last_became_active":"2024-03-06T07:34:06.439145+0000","last_became_peered":"2024-03-06T07:34:06.439145+0000","last_unstale":"2024-03-06T07:34:07.814910+0000","last_undegraded":"2024-03-06T07:34:07.814910+0000","last_fullsized":"2024-03-06T07:34:07.814910+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:48.714296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815222+0000","last_change":"2024-03-06T07:34:05.438419+0000","last_active":"2024-03-06T07:34:07.815222+0000","last_peered":"2024-03-06T07:34:07.815222+0000","last_clean":"2024-03-06T07:34:07.815222+0000","last_became_active":"2024-03-06T07:34:05.437950+0000","last_became_peered":"2024-03-06T07:34:05.437950+0000","last_unstale":"2024-03-06T07:34:07.815222+0000","last_undegraded":"2024-03-06T07:34:07.815222+0000","last_fullsized":"2024-03-06T07:34:07.815222+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:30.291791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447094+0000","last_change":"2024-03-06T07:34:06.430263+0000","last_active":"2024-03-06T07:34:07.447094+0000","last_peered":"2024-03-06T07:34:07.447094+0000","last_clean":"2024-03-06T07:34:07.447094+0000","last_became_active":"2024-03-06T07:34:06.429695+0000","last_became_peered":"2024-03-06T07:34:06.429695+0000","last_unstale":"2024-03-06T07:34:07.447094+0000","last_undegraded":"2024-03-06T07:34:07.447094+0000","last_fullsized":"2024-03-06T07:34:07.447094+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:25.339129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814448+0000","last_change":"2024-03-06T07:34:05.431901+0000","last_active":"2024-03-06T07:34:07.814448+0000","last_peered":"2024-03-06T07:34:07.814448+0000","last_clean":"2024-03-06T07:34:07.814448+0000","last_became_active":"2024-03-06T07:34:05.431644+0000","last_became_peered":"2024-03-06T07:34:05.431644+0000","last_unstale":"2024-03-06T07:34:07.814448+0000","last_undegraded":"2024-03-06T07:34:07.814448+0000","last_fullsized":"2024-03-06T07:34:07.814448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.172323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413904+0000","last_change":"2024-03-06T07:34:05.438522+0000","last_active":"2024-03-06T07:34:07.413904+0000","last_peered":"2024-03-06T07:34:07.413904+0000","last_clean":"2024-03-06T07:34:07.413904+0000","last_became_active":"2024-03-06T07:34:05.438178+0000","last_became_peered":"2024-03-06T07:34:05.438178+0000","last_unstale":"2024-03-06T07:34:07.413904+0000","last_undegraded":"2024-03-06T07:34:07.413904+0000","last_fullsized":"2024-03-06T07:34:07.413904+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:53.930083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:05.391791+0000","last_change":"2024-03-06T07:34:05.391791+0000","last_active":"2024-03-06T07:34:05.391791+0000","last_peered":"2024-03-06T07:34:05.391791+0000","last_clean":"2024-03-06T07:34:05.391791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:05.391791+0000","last_undegraded":"2024-03-06T07:34:05.391791+0000","last_fullsized":"2024-03-06T07:34:05.391791+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:34:04.375306+0000","last_change":"2024-03-06T07:34:04.375306+0000","last_active":"2024-03-06T07:34:04.375306+0000","last_peered":"2024-03-06T07:34:04.375306+0000","last_clean":"2024-03-06T07:34:04.375306+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:04.375306+0000","last_undegraded":"2024-03-06T07:34:04.375306+0000","last_fullsized":"2024-03-06T07:34:04.375306+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814158+0000","last_change":"2024-03-06T07:34:06.442067+0000","last_active":"2024-03-06T07:34:07.814158+0000","last_peered":"2024-03-06T07:34:07.814158+0000","last_clean":"2024-03-06T07:34:07.814158+0000","last_became_active":"2024-03-06T07:34:06.440841+0000","last_became_peered":"2024-03-06T07:34:06.440841+0000","last_unstale":"2024-03-06T07:34:07.814158+0000","last_undegraded":"2024-03-06T07:34:07.814158+0000","last_fullsized":"2024-03-06T07:34:07.814158+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.034356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813254+0000","last_change":"2024-03-06T07:34:05.424983+0000","last_active":"2024-03-06T07:34:07.813254+0000","last_peered":"2024-03-06T07:34:07.813254+0000","last_clean":"2024-03-06T07:34:07.813254+0000","last_became_active":"2024-03-06T07:34:05.424659+0000","last_became_peered":"2024-03-06T07:34:05.424659+0000","last_unstale":"2024-03-06T07:34:07.813254+0000","last_undegraded":"2024-03-06T07:34:07.813254+0000","last_fullsized":"2024-03-06T07:34:07.813254+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.506880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812927+0000","last_change":"2024-03-06T07:34:06.439910+0000","last_active":"2024-03-06T07:34:07.812927+0000","last_peered":"2024-03-06T07:34:07.812927+0000","last_clean":"2024-03-06T07:34:07.812927+0000","last_became_active":"2024-03-06T07:34:06.439586+0000","last_became_peered":"2024-03-06T07:34:06.439586+0000","last_unstale":"2024-03-06T07:34:07.812927+0000","last_undegraded":"2024-03-06T07:34:07.812927+0000","last_fullsized":"2024-03-06T07:34:07.812927+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:20.170576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812427+0000","last_change":"2024-03-06T07:34:05.441536+0000","last_active":"2024-03-06T07:34:07.812427+0000","last_peered":"2024-03-06T07:34:07.812427+0000","last_clean":"2024-03-06T07:34:07.812427+0000","last_became_active":"2024-03-06T07:34:05.438461+0000","last_became_peered":"2024-03-06T07:34:05.438461+0000","last_unstale":"2024-03-06T07:34:07.812427+0000","last_undegraded":"2024-03-06T07:34:07.812427+0000","last_fullsized":"2024-03-06T07:34:07.812427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:02.228169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814112+0000","last_change":"2024-03-06T07:34:06.443906+0000","last_active":"2024-03-06T07:34:07.814112+0000","last_peered":"2024-03-06T07:34:07.814112+0000","last_clean":"2024-03-06T07:34:07.814112+0000","last_became_active":"2024-03-06T07:34:06.443201+0000","last_became_peered":"2024-03-06T07:34:06.443201+0000","last_unstale":"2024-03-06T07:34:07.814112+0000","last_undegraded":"2024-03-06T07:34:07.814112+0000","last_fullsized":"2024-03-06T07:34:07.814112+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:20.170576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412114+0000","last_change":"2024-03-06T07:34:05.440787+0000","last_active":"2024-03-06T07:34:07.412114+0000","last_peered":"2024-03-06T07:34:07.412114+0000","last_clean":"2024-03-06T07:34:07.412114+0000","last_became_active":"2024-03-06T07:34:05.438433+0000","last_became_peered":"2024-03-06T07:34:05.438433+0000","last_unstale":"2024-03-06T07:34:07.412114+0000","last_undegraded":"2024-03-06T07:34:07.412114+0000","last_fullsized":"2024-03-06T07:34:07.412114+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:33.193361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":336,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814276+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:34:07.814276+0000","last_peered":"2024-03-06T07:34:07.814276+0000","last_clean":"2024-03-06T07:34:07.814276+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:34:07.814276+0000","last_undegraded":"2024-03-06T07:34:07.814276+0000","last_fullsized":"2024-03-06T07:34:07.814276+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":60,"acting":60,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542225,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":1990656,"data_stored":1189831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1392640,"data_stored":598917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36708,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335132,"statfs":{"total":96636764160,"available":96599175168,"internally_reserved":0,"allocated":1396736,"data_stored":599451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1339392,"data_stored":573680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542225,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36716,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335124,"statfs":{"total":96636764160,"available":96599166976,"internally_reserved":0,"allocated":1404928,"data_stored":600529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542227,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":598847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542225,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1396736,"data_stored":598971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542226,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37300,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334540,"statfs":{"total":96636764160,"available":96598568960,"internally_reserved":0,"allocated":2002944,"data_stored":1190939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:11.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:11.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.287+0000 7f29ee400640 1 Processor -- start 2024-03-06T07:34:11.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.287+0000 7f29ee400640 1 -- start start 2024-03-06T07:34:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.288+0000 7f29ee400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 0x7f29e815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29ee400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e806cef0 con 0x7f29e8070b10 2024-03-06T07:34:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29ee400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e806d1d0 con 0x7f29e8144c00 2024-03-06T07:34:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29ee400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8160370 con 0x7f29e815f9d0 2024-03-06T07:34:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29e7e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f29e8070b10 0x7f29e8070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33742/0 (socket says 172.21.15.97:33742) 2024-03-06T07:34:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29e7e00640 1 -- 172.21.15.97:0/3994536940 learned_addr learned my addr 172.21.15.97:0/3994536940 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:11.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29dea00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 0x7f29e815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367141475 0 0) 0x7f29e806cef0 con 0x7f29e8070b10 2024-03-06T07:34:11.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.289+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29cc003520 con 0x7f29e8070b10 2024-03-06T07:34:11.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906059989 0 0) 0x7f29cc003520 con 0x7f29e8070b10 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 msgr2=0x7f29e815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 0x7f29e815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 >> v1:172.21.15.135:6789/0 conn(0x7f29e8144c00 legacy=0x7f29e806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e806d4b0 con 0x7f29e8070b10 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f29e0001670 con 0x7f29e8070b10 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 >> v1:172.21.15.97:6789/0 conn(0x7f29e8070b10 legacy=0x7f29e8070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29cc0039e0 0x7f29cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 0x7f29cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29cc004d40 0x7f29cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc00ba70 con 0x7f29cc0039e0 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc00bc20 con 0x7f29cc004d40 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc00be20 con 0x7f29e815f9d0 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.290+0000 7f29e7400640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29cc0039e0 0x7f29cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29e7e00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29cc004d40 0x7f29cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29dea00640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 0x7f29cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29e7400640 1 -- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 msgr2=0x7f29cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29e7400640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e815f9d0 0x7f29cc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29e7400640 1 -- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29cc004d40 msgr2=0x7f29cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29e7400640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29cc004d40 0x7f29cc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.291+0000 7f29e7400640 1 -- 172.21.15.97:0/3994536940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc003520 con 0x7f29cc0039e0 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.292+0000 7f29e7400640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29cc0039e0 0x7f29cc003e00 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f29d8001e20 tx=0x7f29cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c2e07f8d38ec03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.292+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8001a70 con 0x7f29cc0039e0 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.292+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d8003b70 con 0x7f29cc0039e0 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.292+0000 7f29e6a00640 1 -- 172.21.15.97:0/3994536940 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d80050d0 con 0x7f29cc0039e0 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29ee400640 1 -- 172.21.15.97:0/3994536940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29cc0039e0 msgr2=0x7f29cc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29ee400640 1 --2- 172.21.15.97:0/3994536940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29cc0039e0 0x7f29cc003e00 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f29d8001e20 tx=0x7f29cc005830 comp rx=0 tx=0).stop 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29e7e00640 1 -- 172.21.15.97:0/3994536940 reap_dead start 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29ee400640 1 -- 172.21.15.97:0/3994536940 shutdown_connections 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29ee400640 1 -- 172.21.15.97:0/3994536940 >> 172.21.15.97:0/3994536940 conn(0x7f29e80b1730 msgr2=0x7f29e80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29ee400640 1 -- 172.21.15.97:0/3994536940 shutdown_connections 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.293+0000 7f29ee400640 1 -- 172.21.15.97:0/3994536940 wait complete. 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.295+0000 7f29ee400640 1 Processor -- start 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.295+0000 7f29ee400640 1 -- start start 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29ee400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29cc0039e0 0x7f29e80aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29ee400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e8134320 0x7f29e813f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29ee400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 0x7f29e815a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29ee400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e806cb50 con 0x7f29e815a390 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29ee400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8160730 con 0x7f29e8134320 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29ee400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e806ccd0 con 0x7f29cc0039e0 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 0x7f29e815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 0x7f29e815a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52108/0 (socket says 172.21.15.97:52108) 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 -- 172.21.15.97:0/600060320 learned_addr learned my addr 172.21.15.97:0/600060320 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29cc0039e0 0x7f29e80aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29dea00640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e8134320 0x7f29e813f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 -- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29cc0039e0 msgr2=0x7f29e80aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29cc0039e0 0x7f29e80aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 -- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e8134320 msgr2=0x7f29e813f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e8134320 0x7f29e813f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.296+0000 7f29e7e00640 1 -- 172.21.15.97:0/600060320 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e813f9a0 con 0x7f29e815a390 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.297+0000 7f29e7e00640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 0x7f29e815a810 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000fb0 tx=0x7f29e000d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.297+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0007440 con 0x7f29e815a390 2024-03-06T07:34:11.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.297+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e0007600 con 0x7f29e815a390 2024-03-06T07:34:11.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.297+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0004a20 con 0x7f29e815a390 2024-03-06T07:34:11.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.297+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e813fc80 con 0x7f29e815a390 2024-03-06T07:34:11.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.297+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e8140240 con 0x7f29e815a390 2024-03-06T07:34:11.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.298+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e804f330 con 0x7f29e815a390 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.299+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29e001b020 con 0x7f29e815a390 2024-03-06T07:34:11.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.300+0000 7f29dfe00640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b500 0x7f29c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.301+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29e0089dc0 con 0x7f29e815a390 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.301+0000 7f29e7400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b500 0x7f29c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.301+0000 7f29e7400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b500 0x7f29c406d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f29d8034000 tx=0x7f29d8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.305+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29e0058c30 con 0x7f29e815a390 2024-03-06T07:34:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.498+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29e8072890 con 0x7f29c406b500 2024-03-06T07:34:11.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.513+0000 7f29dfe00640 1 -- 172.21.15.97:0/600060320 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f29e8072890 con 0x7f29c406b500 2024-03-06T07:34:11.518 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:11.519 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b500 msgr2=0x7f29c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b500 0x7f29c406d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f29d8034000 tx=0x7f29d8039040 comp rx=0 tx=0).stop 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 msgr2=0x7f29e815a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 0x7f29e815a810 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000fb0 tx=0x7f29e000d9f0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 shutdown_connections 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b500 0x7f29c406d9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e815a390 0x7f29e815a810 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e8134320 0x7f29e813f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 --2- 172.21.15.97:0/600060320 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29cc0039e0 0x7f29e80aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 >> 172.21.15.97:0/600060320 conn(0x7f29e80b1730 msgr2=0x7f29e81429c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 shutdown_connections 2024-03-06T07:34:11.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.519+0000 7f29ee400640 1 -- 172.21.15.97:0/600060320 wait complete. 2024-03-06T07:34:11.538 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-03-06T07:34:11.018941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294824,"kb_used_data":12072,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754679896,"statfs":{"total":773094113280,"available":772792213504,"internally_reserved":0,"allocated":12361728,"data_stored":5976332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001951"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815768+0000","last_change":"2024-03-06T07:34:06.416542+0000","last_active":"2024-03-06T07:34:07.815768+0000","last_peered":"2024-03-06T07:34:07.815768+0000","last_clean":"2024-03-06T07:34:07.815768+0000","last_became_active":"2024-03-06T07:34:06.415731+0000","last_became_peered":"2024-03-06T07:34:06.415731+0000","last_unstale":"2024-03-06T07:34:07.815768+0000","last_undegraded":"2024-03-06T07:34:07.815768+0000","last_fullsized":"2024-03-06T07:34:07.815768+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:44.899228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413782+0000","last_change":"2024-03-06T07:34:06.444317+0000","last_active":"2024-03-06T07:34:07.413782+0000","last_peered":"2024-03-06T07:34:07.413782+0000","last_clean":"2024-03-06T07:34:07.413782+0000","last_became_active":"2024-03-06T07:34:06.444039+0000","last_became_peered":"2024-03-06T07:34:06.444039+0000","last_unstale":"2024-03-06T07:34:07.413782+0000","last_undegraded":"2024-03-06T07:34:07.413782+0000","last_fullsized":"2024-03-06T07:34:07.413782+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:32.160659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814610+0000","last_change":"2024-03-06T07:34:06.448884+0000","last_active":"2024-03-06T07:34:07.814610+0000","last_peered":"2024-03-06T07:34:07.814610+0000","last_clean":"2024-03-06T07:34:07.814610+0000","last_became_active":"2024-03-06T07:34:06.448593+0000","last_became_peered":"2024-03-06T07:34:06.448593+0000","last_unstale":"2024-03-06T07:34:07.814610+0000","last_undegraded":"2024-03-06T07:34:07.814610+0000","last_fullsized":"2024-03-06T07:34:07.814610+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:52.689592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814599+0000","last_change":"2024-03-06T07:34:06.444981+0000","last_active":"2024-03-06T07:34:07.814599+0000","last_peered":"2024-03-06T07:34:07.814599+0000","last_clean":"2024-03-06T07:34:07.814599+0000","last_became_active":"2024-03-06T07:34:06.444566+0000","last_became_peered":"2024-03-06T07:34:06.444566+0000","last_unstale":"2024-03-06T07:34:07.814599+0000","last_undegraded":"2024-03-06T07:34:07.814599+0000","last_fullsized":"2024-03-06T07:34:07.814599+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:11.751702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814576+0000","last_change":"2024-03-06T07:34:06.430544+0000","last_active":"2024-03-06T07:34:07.814576+0000","last_peered":"2024-03-06T07:34:07.814576+0000","last_clean":"2024-03-06T07:34:07.814576+0000","last_became_active":"2024-03-06T07:34:06.430134+0000","last_became_peered":"2024-03-06T07:34:06.430134+0000","last_unstale":"2024-03-06T07:34:07.814576+0000","last_undegraded":"2024-03-06T07:34:07.814576+0000","last_fullsized":"2024-03-06T07:34:07.814576+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:58.160771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405287+0000","last_change":"2024-03-06T07:34:06.439755+0000","last_active":"2024-03-06T07:34:07.405287+0000","last_peered":"2024-03-06T07:34:07.405287+0000","last_clean":"2024-03-06T07:34:07.405287+0000","last_became_active":"2024-03-06T07:34:06.438929+0000","last_became_peered":"2024-03-06T07:34:06.438929+0000","last_unstale":"2024-03-06T07:34:07.405287+0000","last_undegraded":"2024-03-06T07:34:07.405287+0000","last_fullsized":"2024-03-06T07:34:07.405287+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:17.785135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412033+0000","last_change":"2024-03-06T07:34:06.427744+0000","last_active":"2024-03-06T07:34:07.412033+0000","last_peered":"2024-03-06T07:34:07.412033+0000","last_clean":"2024-03-06T07:34:07.412033+0000","last_became_active":"2024-03-06T07:34:06.426615+0000","last_became_peered":"2024-03-06T07:34:06.426615+0000","last_unstale":"2024-03-06T07:34:07.412033+0000","last_undegraded":"2024-03-06T07:34:07.412033+0000","last_fullsized":"2024-03-06T07:34:07.412033+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:48.822674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813808+0000","last_change":"2024-03-06T07:34:06.439245+0000","last_active":"2024-03-06T07:34:07.813808+0000","last_peered":"2024-03-06T07:34:07.813808+0000","last_clean":"2024-03-06T07:34:07.813808+0000","last_became_active":"2024-03-06T07:34:06.438968+0000","last_became_peered":"2024-03-06T07:34:06.438968+0000","last_unstale":"2024-03-06T07:34:07.813808+0000","last_undegraded":"2024-03-06T07:34:07.813808+0000","last_fullsized":"2024-03-06T07:34:07.813808+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:49.964864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815767+0000","last_change":"2024-03-06T07:34:06.431196+0000","last_active":"2024-03-06T07:34:07.815767+0000","last_peered":"2024-03-06T07:34:07.815767+0000","last_clean":"2024-03-06T07:34:07.815767+0000","last_became_active":"2024-03-06T07:34:06.430895+0000","last_became_peered":"2024-03-06T07:34:06.430895+0000","last_unstale":"2024-03-06T07:34:07.815767+0000","last_undegraded":"2024-03-06T07:34:07.815767+0000","last_fullsized":"2024-03-06T07:34:07.815767+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:08.661614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815791+0000","last_change":"2024-03-06T07:34:06.424035+0000","last_active":"2024-03-06T07:34:07.815791+0000","last_peered":"2024-03-06T07:34:07.815791+0000","last_clean":"2024-03-06T07:34:07.815791+0000","last_became_active":"2024-03-06T07:34:06.423713+0000","last_became_peered":"2024-03-06T07:34:06.423713+0000","last_unstale":"2024-03-06T07:34:07.815791+0000","last_undegraded":"2024-03-06T07:34:07.815791+0000","last_fullsized":"2024-03-06T07:34:07.815791+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:50.651648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413729+0000","last_change":"2024-03-06T07:34:06.445250+0000","last_active":"2024-03-06T07:34:07.413729+0000","last_peered":"2024-03-06T07:34:07.413729+0000","last_clean":"2024-03-06T07:34:07.413729+0000","last_became_active":"2024-03-06T07:34:06.441109+0000","last_became_peered":"2024-03-06T07:34:06.441109+0000","last_unstale":"2024-03-06T07:34:07.413729+0000","last_undegraded":"2024-03-06T07:34:07.413729+0000","last_fullsized":"2024-03-06T07:34:07.413729+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:10.125167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814960+0000","last_change":"2024-03-06T07:34:06.422197+0000","last_active":"2024-03-06T07:34:07.814960+0000","last_peered":"2024-03-06T07:34:07.814960+0000","last_clean":"2024-03-06T07:34:07.814960+0000","last_became_active":"2024-03-06T07:34:06.421762+0000","last_became_peered":"2024-03-06T07:34:06.421762+0000","last_unstale":"2024-03-06T07:34:07.814960+0000","last_undegraded":"2024-03-06T07:34:07.814960+0000","last_fullsized":"2024-03-06T07:34:07.814960+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:40.672706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814042+0000","last_change":"2024-03-06T07:34:06.443659+0000","last_active":"2024-03-06T07:34:07.814042+0000","last_peered":"2024-03-06T07:34:07.814042+0000","last_clean":"2024-03-06T07:34:07.814042+0000","last_became_active":"2024-03-06T07:34:06.443266+0000","last_became_peered":"2024-03-06T07:34:06.443266+0000","last_unstale":"2024-03-06T07:34:07.814042+0000","last_undegraded":"2024-03-06T07:34:07.814042+0000","last_fullsized":"2024-03-06T07:34:07.814042+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:16.681929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813741+0000","last_change":"2024-03-06T07:34:06.444078+0000","last_active":"2024-03-06T07:34:07.813741+0000","last_peered":"2024-03-06T07:34:07.813741+0000","last_clean":"2024-03-06T07:34:07.813741+0000","last_became_active":"2024-03-06T07:34:06.443750+0000","last_became_peered":"2024-03-06T07:34:06.443750+0000","last_unstale":"2024-03-06T07:34:07.813741+0000","last_undegraded":"2024-03-06T07:34:07.813741+0000","last_fullsized":"2024-03-06T07:34:07.813741+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:31.638197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405256+0000","last_change":"2024-03-06T07:34:06.436981+0000","last_active":"2024-03-06T07:34:07.405256+0000","last_peered":"2024-03-06T07:34:07.405256+0000","last_clean":"2024-03-06T07:34:07.405256+0000","last_became_active":"2024-03-06T07:34:06.436269+0000","last_became_peered":"2024-03-06T07:34:06.436269+0000","last_unstale":"2024-03-06T07:34:07.405256+0000","last_undegraded":"2024-03-06T07:34:07.405256+0000","last_fullsized":"2024-03-06T07:34:07.405256+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:54.946568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812086+0000","last_change":"2024-03-06T07:34:06.438420+0000","last_active":"2024-03-06T07:34:07.812086+0000","last_peered":"2024-03-06T07:34:07.812086+0000","last_clean":"2024-03-06T07:34:07.812086+0000","last_became_active":"2024-03-06T07:34:06.438093+0000","last_became_peered":"2024-03-06T07:34:06.438093+0000","last_unstale":"2024-03-06T07:34:07.812086+0000","last_undegraded":"2024-03-06T07:34:07.812086+0000","last_fullsized":"2024-03-06T07:34:07.812086+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:26.571749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.414223+0000","last_change":"2024-03-06T07:34:06.440769+0000","last_active":"2024-03-06T07:34:07.414223+0000","last_peered":"2024-03-06T07:34:07.414223+0000","last_clean":"2024-03-06T07:34:07.414223+0000","last_became_active":"2024-03-06T07:34:06.439137+0000","last_became_peered":"2024-03-06T07:34:06.439137+0000","last_unstale":"2024-03-06T07:34:07.414223+0000","last_undegraded":"2024-03-06T07:34:07.414223+0000","last_fullsized":"2024-03-06T07:34:07.414223+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:26.702951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447231+0000","last_change":"2024-03-06T07:34:06.442997+0000","last_active":"2024-03-06T07:34:07.447231+0000","last_peered":"2024-03-06T07:34:07.447231+0000","last_clean":"2024-03-06T07:34:07.447231+0000","last_became_active":"2024-03-06T07:34:06.442711+0000","last_became_peered":"2024-03-06T07:34:06.442711+0000","last_unstale":"2024-03-06T07:34:07.447231+0000","last_undegraded":"2024-03-06T07:34:07.447231+0000","last_fullsized":"2024-03-06T07:34:07.447231+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:15.370733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814723+0000","last_change":"2024-03-06T07:34:06.444763+0000","last_active":"2024-03-06T07:34:07.814723+0000","last_peered":"2024-03-06T07:34:07.814723+0000","last_clean":"2024-03-06T07:34:07.814723+0000","last_became_active":"2024-03-06T07:34:06.444422+0000","last_became_peered":"2024-03-06T07:34:06.444422+0000","last_unstale":"2024-03-06T07:34:07.814723+0000","last_undegraded":"2024-03-06T07:34:07.814723+0000","last_fullsized":"2024-03-06T07:34:07.814723+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:47.658866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413934+0000","last_change":"2024-03-06T07:34:06.443016+0000","last_active":"2024-03-06T07:34:07.413934+0000","last_peered":"2024-03-06T07:34:07.413934+0000","last_clean":"2024-03-06T07:34:07.413934+0000","last_became_active":"2024-03-06T07:34:06.438171+0000","last_became_peered":"2024-03-06T07:34:06.438171+0000","last_unstale":"2024-03-06T07:34:07.413934+0000","last_undegraded":"2024-03-06T07:34:07.413934+0000","last_fullsized":"2024-03-06T07:34:07.413934+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:02.989354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413070+0000","last_change":"2024-03-06T07:34:06.451264+0000","last_active":"2024-03-06T07:34:07.413070+0000","last_peered":"2024-03-06T07:34:07.413070+0000","last_clean":"2024-03-06T07:34:07.413070+0000","last_became_active":"2024-03-06T07:34:06.447507+0000","last_became_peered":"2024-03-06T07:34:06.447507+0000","last_unstale":"2024-03-06T07:34:07.413070+0000","last_undegraded":"2024-03-06T07:34:07.413070+0000","last_fullsized":"2024-03-06T07:34:07.413070+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:01.780323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812998+0000","last_change":"2024-03-06T07:34:06.429677+0000","last_active":"2024-03-06T07:34:07.812998+0000","last_peered":"2024-03-06T07:34:07.812998+0000","last_clean":"2024-03-06T07:34:07.812998+0000","last_became_active":"2024-03-06T07:34:06.428739+0000","last_became_peered":"2024-03-06T07:34:06.428739+0000","last_unstale":"2024-03-06T07:34:07.812998+0000","last_undegraded":"2024-03-06T07:34:07.812998+0000","last_fullsized":"2024-03-06T07:34:07.812998+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:25.554662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.410857+0000","last_change":"2024-03-06T07:34:06.439437+0000","last_active":"2024-03-06T07:34:07.410857+0000","last_peered":"2024-03-06T07:34:07.410857+0000","last_clean":"2024-03-06T07:34:07.410857+0000","last_became_active":"2024-03-06T07:34:06.439065+0000","last_became_peered":"2024-03-06T07:34:06.439065+0000","last_unstale":"2024-03-06T07:34:07.410857+0000","last_undegraded":"2024-03-06T07:34:07.410857+0000","last_fullsized":"2024-03-06T07:34:07.410857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:32.723872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813227+0000","last_change":"2024-03-06T07:34:06.440036+0000","last_active":"2024-03-06T07:34:07.813227+0000","last_peered":"2024-03-06T07:34:07.813227+0000","last_clean":"2024-03-06T07:34:07.813227+0000","last_became_active":"2024-03-06T07:34:06.439740+0000","last_became_peered":"2024-03-06T07:34:06.439740+0000","last_unstale":"2024-03-06T07:34:07.813227+0000","last_undegraded":"2024-03-06T07:34:07.813227+0000","last_fullsized":"2024-03-06T07:34:07.813227+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:55.595831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.414388+0000","last_change":"2024-03-06T07:34:06.435637+0000","last_active":"2024-03-06T07:34:07.414388+0000","last_peered":"2024-03-06T07:34:07.414388+0000","last_clean":"2024-03-06T07:34:07.414388+0000","last_became_active":"2024-03-06T07:34:06.435349+0000","last_became_peered":"2024-03-06T07:34:06.435349+0000","last_unstale":"2024-03-06T07:34:07.414388+0000","last_undegraded":"2024-03-06T07:34:07.414388+0000","last_fullsized":"2024-03-06T07:34:07.414388+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:41.382870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815846+0000","last_change":"2024-03-06T07:34:06.439211+0000","last_active":"2024-03-06T07:34:07.815846+0000","last_peered":"2024-03-06T07:34:07.815846+0000","last_clean":"2024-03-06T07:34:07.815846+0000","last_became_active":"2024-03-06T07:34:06.438886+0000","last_became_peered":"2024-03-06T07:34:06.438886+0000","last_unstale":"2024-03-06T07:34:07.815846+0000","last_undegraded":"2024-03-06T07:34:07.815846+0000","last_fullsized":"2024-03-06T07:34:07.815846+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:10.815713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406363+0000","last_change":"2024-03-06T07:34:06.440484+0000","last_active":"2024-03-06T07:34:07.406363+0000","last_peered":"2024-03-06T07:34:07.406363+0000","last_clean":"2024-03-06T07:34:07.406363+0000","last_became_active":"2024-03-06T07:34:06.440210+0000","last_became_peered":"2024-03-06T07:34:06.440210+0000","last_unstale":"2024-03-06T07:34:07.406363+0000","last_undegraded":"2024-03-06T07:34:07.406363+0000","last_fullsized":"2024-03-06T07:34:07.406363+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:24.819319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813602+0000","last_change":"2024-03-06T07:34:06.428880+0000","last_active":"2024-03-06T07:34:07.813602+0000","last_peered":"2024-03-06T07:34:07.813602+0000","last_clean":"2024-03-06T07:34:07.813602+0000","last_became_active":"2024-03-06T07:34:06.427599+0000","last_became_peered":"2024-03-06T07:34:06.427599+0000","last_unstale":"2024-03-06T07:34:07.813602+0000","last_undegraded":"2024-03-06T07:34:07.813602+0000","last_fullsized":"2024-03-06T07:34:07.813602+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:26.571749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412698+0000","last_change":"2024-03-06T07:34:06.450330+0000","last_active":"2024-03-06T07:34:07.412698+0000","last_peered":"2024-03-06T07:34:07.412698+0000","last_clean":"2024-03-06T07:34:07.412698+0000","last_became_active":"2024-03-06T07:34:06.448998+0000","last_became_peered":"2024-03-06T07:34:06.448998+0000","last_unstale":"2024-03-06T07:34:07.412698+0000","last_undegraded":"2024-03-06T07:34:07.412698+0000","last_fullsized":"2024-03-06T07:34:07.412698+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:27.703408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411662+0000","last_change":"2024-03-06T07:34:06.436047+0000","last_active":"2024-03-06T07:34:07.411662+0000","last_peered":"2024-03-06T07:34:07.411662+0000","last_clean":"2024-03-06T07:34:07.411662+0000","last_became_active":"2024-03-06T07:34:06.435697+0000","last_became_peered":"2024-03-06T07:34:06.435697+0000","last_unstale":"2024-03-06T07:34:07.411662+0000","last_undegraded":"2024-03-06T07:34:07.411662+0000","last_fullsized":"2024-03-06T07:34:07.411662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:21.016281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812942+0000","last_change":"2024-03-06T07:34:06.438134+0000","last_active":"2024-03-06T07:34:07.812942+0000","last_peered":"2024-03-06T07:34:07.812942+0000","last_clean":"2024-03-06T07:34:07.812942+0000","last_became_active":"2024-03-06T07:34:06.436496+0000","last_became_peered":"2024-03-06T07:34:06.436496+0000","last_unstale":"2024-03-06T07:34:07.812942+0000","last_undegraded":"2024-03-06T07:34:07.812942+0000","last_fullsized":"2024-03-06T07:34:07.812942+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:48.060635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405175+0000","last_change":"2024-03-06T07:34:06.433877+0000","last_active":"2024-03-06T07:34:07.405175+0000","last_peered":"2024-03-06T07:34:07.405175+0000","last_clean":"2024-03-06T07:34:07.405175+0000","last_became_active":"2024-03-06T07:34:06.433579+0000","last_became_peered":"2024-03-06T07:34:06.433579+0000","last_unstale":"2024-03-06T07:34:07.405175+0000","last_undegraded":"2024-03-06T07:34:07.405175+0000","last_fullsized":"2024-03-06T07:34:07.405175+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:34.209846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406174+0000","last_change":"2024-03-06T07:34:06.436766+0000","last_active":"2024-03-06T07:34:07.406174+0000","last_peered":"2024-03-06T07:34:07.406174+0000","last_clean":"2024-03-06T07:34:07.406174+0000","last_became_active":"2024-03-06T07:34:06.435876+0000","last_became_peered":"2024-03-06T07:34:06.435876+0000","last_unstale":"2024-03-06T07:34:07.406174+0000","last_undegraded":"2024-03-06T07:34:07.406174+0000","last_fullsized":"2024-03-06T07:34:07.406174+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:34.493622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"111'4","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411530+0000","last_change":"2024-03-06T07:34:05.437699+0000","last_active":"2024-03-06T07:34:07.411530+0000","last_peered":"2024-03-06T07:34:07.411530+0000","last_clean":"2024-03-06T07:34:07.411530+0000","last_became_active":"2024-03-06T07:34:05.437396+0000","last_became_peered":"2024-03-06T07:34:05.437396+0000","last_unstale":"2024-03-06T07:34:07.411530+0000","last_undegraded":"2024-03-06T07:34:07.411530+0000","last_fullsized":"2024-03-06T07:34:07.411530+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:33.477137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447590+0000","last_change":"2024-03-06T07:34:06.438312+0000","last_active":"2024-03-06T07:34:07.447590+0000","last_peered":"2024-03-06T07:34:07.447590+0000","last_clean":"2024-03-06T07:34:07.447590+0000","last_became_active":"2024-03-06T07:34:06.437349+0000","last_became_peered":"2024-03-06T07:34:06.437349+0000","last_unstale":"2024-03-06T07:34:07.447590+0000","last_undegraded":"2024-03-06T07:34:07.447590+0000","last_fullsized":"2024-03-06T07:34:07.447590+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:50.851331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814364+0000","last_change":"2024-03-06T07:34:05.441112+0000","last_active":"2024-03-06T07:34:07.814364+0000","last_peered":"2024-03-06T07:34:07.814364+0000","last_clean":"2024-03-06T07:34:07.814364+0000","last_became_active":"2024-03-06T07:34:05.440635+0000","last_became_peered":"2024-03-06T07:34:05.440635+0000","last_unstale":"2024-03-06T07:34:07.814364+0000","last_undegraded":"2024-03-06T07:34:07.814364+0000","last_fullsized":"2024-03-06T07:34:07.814364+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:28.950051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406141+0000","last_change":"2024-03-06T07:34:06.436453+0000","last_active":"2024-03-06T07:34:07.406141+0000","last_peered":"2024-03-06T07:34:07.406141+0000","last_clean":"2024-03-06T07:34:07.406141+0000","last_became_active":"2024-03-06T07:34:06.436199+0000","last_became_peered":"2024-03-06T07:34:06.436199+0000","last_unstale":"2024-03-06T07:34:07.406141+0000","last_undegraded":"2024-03-06T07:34:07.406141+0000","last_fullsized":"2024-03-06T07:34:07.406141+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:36.576401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814975+0000","last_change":"2024-03-06T07:34:05.433032+0000","last_active":"2024-03-06T07:34:07.814975+0000","last_peered":"2024-03-06T07:34:07.814975+0000","last_clean":"2024-03-06T07:34:07.814975+0000","last_became_active":"2024-03-06T07:34:05.432735+0000","last_became_peered":"2024-03-06T07:34:05.432735+0000","last_unstale":"2024-03-06T07:34:07.814975+0000","last_undegraded":"2024-03-06T07:34:07.814975+0000","last_fullsized":"2024-03-06T07:34:07.814975+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:04.579509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814824+0000","last_change":"2024-03-06T07:34:06.445008+0000","last_active":"2024-03-06T07:34:07.814824+0000","last_peered":"2024-03-06T07:34:07.814824+0000","last_clean":"2024-03-06T07:34:07.814824+0000","last_became_active":"2024-03-06T07:34:06.444643+0000","last_became_peered":"2024-03-06T07:34:06.444643+0000","last_unstale":"2024-03-06T07:34:07.814824+0000","last_undegraded":"2024-03-06T07:34:07.814824+0000","last_fullsized":"2024-03-06T07:34:07.814824+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:17.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813704+0000","last_change":"2024-03-06T07:34:05.433936+0000","last_active":"2024-03-06T07:34:07.813704+0000","last_peered":"2024-03-06T07:34:07.813704+0000","last_clean":"2024-03-06T07:34:07.813704+0000","last_became_active":"2024-03-06T07:34:05.433225+0000","last_became_peered":"2024-03-06T07:34:05.433225+0000","last_unstale":"2024-03-06T07:34:07.813704+0000","last_undegraded":"2024-03-06T07:34:07.813704+0000","last_fullsized":"2024-03-06T07:34:07.813704+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:49.850269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814290+0000","last_change":"2024-03-06T07:34:06.432720+0000","last_active":"2024-03-06T07:34:07.814290+0000","last_peered":"2024-03-06T07:34:07.814290+0000","last_clean":"2024-03-06T07:34:07.814290+0000","last_became_active":"2024-03-06T07:34:06.432447+0000","last_became_peered":"2024-03-06T07:34:06.432447+0000","last_unstale":"2024-03-06T07:34:07.814290+0000","last_undegraded":"2024-03-06T07:34:07.814290+0000","last_fullsized":"2024-03-06T07:34:07.814290+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:18.987781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413711+0000","last_change":"2024-03-06T07:34:05.437729+0000","last_active":"2024-03-06T07:34:07.413711+0000","last_peered":"2024-03-06T07:34:07.413711+0000","last_clean":"2024-03-06T07:34:07.413711+0000","last_became_active":"2024-03-06T07:34:05.437353+0000","last_became_peered":"2024-03-06T07:34:05.437353+0000","last_unstale":"2024-03-06T07:34:07.413711+0000","last_undegraded":"2024-03-06T07:34:07.413711+0000","last_fullsized":"2024-03-06T07:34:07.413711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:23.802834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412826+0000","last_change":"2024-03-06T07:34:06.440445+0000","last_active":"2024-03-06T07:34:07.412826+0000","last_peered":"2024-03-06T07:34:07.412826+0000","last_clean":"2024-03-06T07:34:07.412826+0000","last_became_active":"2024-03-06T07:34:06.440172+0000","last_became_peered":"2024-03-06T07:34:06.440172+0000","last_unstale":"2024-03-06T07:34:07.412826+0000","last_undegraded":"2024-03-06T07:34:07.412826+0000","last_fullsized":"2024-03-06T07:34:07.412826+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:02.770484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813253+0000","last_change":"2024-03-06T07:34:05.437791+0000","last_active":"2024-03-06T07:34:07.813253+0000","last_peered":"2024-03-06T07:34:07.813253+0000","last_clean":"2024-03-06T07:34:07.813253+0000","last_became_active":"2024-03-06T07:34:05.437497+0000","last_became_peered":"2024-03-06T07:34:05.437497+0000","last_unstale":"2024-03-06T07:34:07.813253+0000","last_undegraded":"2024-03-06T07:34:07.813253+0000","last_fullsized":"2024-03-06T07:34:07.813253+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:49.834846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812013+0000","last_change":"2024-03-06T07:34:06.435648+0000","last_active":"2024-03-06T07:34:07.812013+0000","last_peered":"2024-03-06T07:34:07.812013+0000","last_clean":"2024-03-06T07:34:07.812013+0000","last_became_active":"2024-03-06T07:34:06.435390+0000","last_became_peered":"2024-03-06T07:34:06.435390+0000","last_unstale":"2024-03-06T07:34:07.812013+0000","last_undegraded":"2024-03-06T07:34:07.812013+0000","last_fullsized":"2024-03-06T07:34:07.812013+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:55.595831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411168+0000","last_change":"2024-03-06T07:34:05.434232+0000","last_active":"2024-03-06T07:34:07.411168+0000","last_peered":"2024-03-06T07:34:07.411168+0000","last_clean":"2024-03-06T07:34:07.411168+0000","last_became_active":"2024-03-06T07:34:05.433768+0000","last_became_peered":"2024-03-06T07:34:05.433768+0000","last_unstale":"2024-03-06T07:34:07.411168+0000","last_undegraded":"2024-03-06T07:34:07.411168+0000","last_fullsized":"2024-03-06T07:34:07.411168+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:24.776469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411755+0000","last_change":"2024-03-06T07:34:06.427236+0000","last_active":"2024-03-06T07:34:07.411755+0000","last_peered":"2024-03-06T07:34:07.411755+0000","last_clean":"2024-03-06T07:34:07.411755+0000","last_became_active":"2024-03-06T07:34:06.425479+0000","last_became_peered":"2024-03-06T07:34:06.425479+0000","last_unstale":"2024-03-06T07:34:07.411755+0000","last_undegraded":"2024-03-06T07:34:07.411755+0000","last_fullsized":"2024-03-06T07:34:07.411755+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:41.399474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812211+0000","last_change":"2024-03-06T07:34:05.431239+0000","last_active":"2024-03-06T07:34:07.812211+0000","last_peered":"2024-03-06T07:34:07.812211+0000","last_clean":"2024-03-06T07:34:07.812211+0000","last_became_active":"2024-03-06T07:34:05.430738+0000","last_became_peered":"2024-03-06T07:34:05.430738+0000","last_unstale":"2024-03-06T07:34:07.812211+0000","last_undegraded":"2024-03-06T07:34:07.812211+0000","last_fullsized":"2024-03-06T07:34:07.812211+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:14.354248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.414955+0000","last_change":"2024-03-06T07:34:06.431283+0000","last_active":"2024-03-06T07:34:07.414955+0000","last_peered":"2024-03-06T07:34:07.414955+0000","last_clean":"2024-03-06T07:34:07.414955+0000","last_became_active":"2024-03-06T07:34:06.431022+0000","last_became_peered":"2024-03-06T07:34:06.431022+0000","last_unstale":"2024-03-06T07:34:07.414955+0000","last_undegraded":"2024-03-06T07:34:07.414955+0000","last_fullsized":"2024-03-06T07:34:07.414955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:37.885605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813866+0000","last_change":"2024-03-06T07:34:05.434115+0000","last_active":"2024-03-06T07:34:07.813866+0000","last_peered":"2024-03-06T07:34:07.813866+0000","last_clean":"2024-03-06T07:34:07.813866+0000","last_became_active":"2024-03-06T07:34:05.432812+0000","last_became_peered":"2024-03-06T07:34:05.432812+0000","last_unstale":"2024-03-06T07:34:07.813866+0000","last_undegraded":"2024-03-06T07:34:07.813866+0000","last_fullsized":"2024-03-06T07:34:07.813866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:01.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447726+0000","last_change":"2024-03-06T07:34:06.421717+0000","last_active":"2024-03-06T07:34:07.447726+0000","last_peered":"2024-03-06T07:34:07.447726+0000","last_clean":"2024-03-06T07:34:07.447726+0000","last_became_active":"2024-03-06T07:34:06.420868+0000","last_became_peered":"2024-03-06T07:34:06.420868+0000","last_unstale":"2024-03-06T07:34:07.447726+0000","last_undegraded":"2024-03-06T07:34:07.447726+0000","last_fullsized":"2024-03-06T07:34:07.447726+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:44.188808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812562+0000","last_change":"2024-03-06T07:34:05.437594+0000","last_active":"2024-03-06T07:34:07.812562+0000","last_peered":"2024-03-06T07:34:07.812562+0000","last_clean":"2024-03-06T07:34:07.812562+0000","last_became_active":"2024-03-06T07:34:05.436227+0000","last_became_peered":"2024-03-06T07:34:05.436227+0000","last_unstale":"2024-03-06T07:34:07.812562+0000","last_undegraded":"2024-03-06T07:34:07.812562+0000","last_fullsized":"2024-03-06T07:34:07.812562+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.846568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813994+0000","last_change":"2024-03-06T07:34:06.444755+0000","last_active":"2024-03-06T07:34:07.813994+0000","last_peered":"2024-03-06T07:34:07.813994+0000","last_clean":"2024-03-06T07:34:07.813994+0000","last_became_active":"2024-03-06T07:34:06.444479+0000","last_became_peered":"2024-03-06T07:34:06.444479+0000","last_unstale":"2024-03-06T07:34:07.813994+0000","last_undegraded":"2024-03-06T07:34:07.813994+0000","last_fullsized":"2024-03-06T07:34:07.813994+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:48.060635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"111'3","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815224+0000","last_change":"2024-03-06T07:34:05.429203+0000","last_active":"2024-03-06T07:34:07.815224+0000","last_peered":"2024-03-06T07:34:07.815224+0000","last_clean":"2024-03-06T07:34:07.815224+0000","last_became_active":"2024-03-06T07:34:05.428889+0000","last_became_peered":"2024-03-06T07:34:05.428889+0000","last_unstale":"2024-03-06T07:34:07.815224+0000","last_undegraded":"2024-03-06T07:34:07.815224+0000","last_fullsized":"2024-03-06T07:34:07.815224+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:24.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413598+0000","last_change":"2024-03-06T07:34:06.437689+0000","last_active":"2024-03-06T07:34:07.413598+0000","last_peered":"2024-03-06T07:34:07.413598+0000","last_clean":"2024-03-06T07:34:07.413598+0000","last_became_active":"2024-03-06T07:34:06.436768+0000","last_became_peered":"2024-03-06T07:34:06.436768+0000","last_unstale":"2024-03-06T07:34:07.413598+0000","last_undegraded":"2024-03-06T07:34:07.413598+0000","last_fullsized":"2024-03-06T07:34:07.413598+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:53.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"111'6","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815111+0000","last_change":"2024-03-06T07:34:05.435622+0000","last_active":"2024-03-06T07:34:07.815111+0000","last_peered":"2024-03-06T07:34:07.815111+0000","last_clean":"2024-03-06T07:34:07.815111+0000","last_became_active":"2024-03-06T07:34:05.435353+0000","last_became_peered":"2024-03-06T07:34:05.435353+0000","last_unstale":"2024-03-06T07:34:07.815111+0000","last_undegraded":"2024-03-06T07:34:07.815111+0000","last_fullsized":"2024-03-06T07:34:07.815111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.172323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813008+0000","last_change":"2024-03-06T07:34:06.439885+0000","last_active":"2024-03-06T07:34:07.813008+0000","last_peered":"2024-03-06T07:34:07.813008+0000","last_clean":"2024-03-06T07:34:07.813008+0000","last_became_active":"2024-03-06T07:34:06.439512+0000","last_became_peered":"2024-03-06T07:34:06.439512+0000","last_unstale":"2024-03-06T07:34:07.813008+0000","last_undegraded":"2024-03-06T07:34:07.813008+0000","last_fullsized":"2024-03-06T07:34:07.813008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:14.292757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405808+0000","last_change":"2024-03-06T07:34:05.438753+0000","last_active":"2024-03-06T07:34:07.405808+0000","last_peered":"2024-03-06T07:34:07.405808+0000","last_clean":"2024-03-06T07:34:07.405808+0000","last_became_active":"2024-03-06T07:34:05.437002+0000","last_became_peered":"2024-03-06T07:34:05.437002+0000","last_unstale":"2024-03-06T07:34:07.405808+0000","last_undegraded":"2024-03-06T07:34:07.405808+0000","last_fullsized":"2024-03-06T07:34:07.405808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.506880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812940+0000","last_change":"2024-03-06T07:34:06.435548+0000","last_active":"2024-03-06T07:34:07.812940+0000","last_peered":"2024-03-06T07:34:07.812940+0000","last_clean":"2024-03-06T07:34:07.812940+0000","last_became_active":"2024-03-06T07:34:06.435258+0000","last_became_peered":"2024-03-06T07:34:06.435258+0000","last_unstale":"2024-03-06T07:34:07.812940+0000","last_undegraded":"2024-03-06T07:34:07.812940+0000","last_fullsized":"2024-03-06T07:34:07.812940+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:20.152660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.446818+0000","last_change":"2024-03-06T07:34:05.437038+0000","last_active":"2024-03-06T07:34:07.446818+0000","last_peered":"2024-03-06T07:34:07.446818+0000","last_clean":"2024-03-06T07:34:07.446818+0000","last_became_active":"2024-03-06T07:34:05.436728+0000","last_became_peered":"2024-03-06T07:34:05.436728+0000","last_unstale":"2024-03-06T07:34:07.446818+0000","last_undegraded":"2024-03-06T07:34:07.446818+0000","last_fullsized":"2024-03-06T07:34:07.446818+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:38.143183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812746+0000","last_change":"2024-03-06T07:34:06.444516+0000","last_active":"2024-03-06T07:34:07.812746+0000","last_peered":"2024-03-06T07:34:07.812746+0000","last_clean":"2024-03-06T07:34:07.812746+0000","last_became_active":"2024-03-06T07:34:06.443735+0000","last_became_peered":"2024-03-06T07:34:06.443735+0000","last_unstale":"2024-03-06T07:34:07.812746+0000","last_undegraded":"2024-03-06T07:34:07.812746+0000","last_fullsized":"2024-03-06T07:34:07.812746+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:40.729083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"111'8","reported_seq":22,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813740+0000","last_change":"2024-03-06T07:34:05.418144+0000","last_active":"2024-03-06T07:34:07.813740+0000","last_peered":"2024-03-06T07:34:07.813740+0000","last_clean":"2024-03-06T07:34:07.813740+0000","last_became_active":"2024-03-06T07:34:05.417486+0000","last_became_peered":"2024-03-06T07:34:05.417486+0000","last_unstale":"2024-03-06T07:34:07.813740+0000","last_undegraded":"2024-03-06T07:34:07.813740+0000","last_fullsized":"2024-03-06T07:34:07.813740+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:57.782081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814303+0000","last_change":"2024-03-06T07:34:06.422711+0000","last_active":"2024-03-06T07:34:07.814303+0000","last_peered":"2024-03-06T07:34:07.814303+0000","last_clean":"2024-03-06T07:34:07.814303+0000","last_became_active":"2024-03-06T07:34:06.421347+0000","last_became_peered":"2024-03-06T07:34:06.421347+0000","last_unstale":"2024-03-06T07:34:07.814303+0000","last_undegraded":"2024-03-06T07:34:07.814303+0000","last_fullsized":"2024-03-06T07:34:07.814303+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:53.685828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815959+0000","last_change":"2024-03-06T07:34:05.441377+0000","last_active":"2024-03-06T07:34:07.815959+0000","last_peered":"2024-03-06T07:34:07.815959+0000","last_clean":"2024-03-06T07:34:07.815959+0000","last_became_active":"2024-03-06T07:34:05.438203+0000","last_became_peered":"2024-03-06T07:34:05.438203+0000","last_unstale":"2024-03-06T07:34:07.815959+0000","last_undegraded":"2024-03-06T07:34:07.815959+0000","last_fullsized":"2024-03-06T07:34:07.815959+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.846568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405293+0000","last_change":"2024-03-06T07:34:06.445211+0000","last_active":"2024-03-06T07:34:07.405293+0000","last_peered":"2024-03-06T07:34:07.405293+0000","last_clean":"2024-03-06T07:34:07.405293+0000","last_became_active":"2024-03-06T07:34:06.444820+0000","last_became_peered":"2024-03-06T07:34:06.444820+0000","last_unstale":"2024-03-06T07:34:07.405293+0000","last_undegraded":"2024-03-06T07:34:07.405293+0000","last_fullsized":"2024-03-06T07:34:07.405293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:52.841179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814417+0000","last_change":"2024-03-06T07:34:05.431742+0000","last_active":"2024-03-06T07:34:07.814417+0000","last_peered":"2024-03-06T07:34:07.814417+0000","last_clean":"2024-03-06T07:34:07.814417+0000","last_became_active":"2024-03-06T07:34:05.431441+0000","last_became_peered":"2024-03-06T07:34:05.431441+0000","last_unstale":"2024-03-06T07:34:07.814417+0000","last_undegraded":"2024-03-06T07:34:07.814417+0000","last_fullsized":"2024-03-06T07:34:07.814417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:53.533773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814394+0000","last_change":"2024-03-06T07:34:06.438884+0000","last_active":"2024-03-06T07:34:07.814394+0000","last_peered":"2024-03-06T07:34:07.814394+0000","last_clean":"2024-03-06T07:34:07.814394+0000","last_became_active":"2024-03-06T07:34:06.438607+0000","last_became_peered":"2024-03-06T07:34:06.438607+0000","last_unstale":"2024-03-06T07:34:07.814394+0000","last_undegraded":"2024-03-06T07:34:07.814394+0000","last_fullsized":"2024-03-06T07:34:07.814394+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:34.513428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"111'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.446979+0000","last_change":"2024-03-06T07:34:05.435463+0000","last_active":"2024-03-06T07:34:07.446979+0000","last_peered":"2024-03-06T07:34:07.446979+0000","last_clean":"2024-03-06T07:34:07.446979+0000","last_became_active":"2024-03-06T07:34:05.435060+0000","last_became_peered":"2024-03-06T07:34:05.435060+0000","last_unstale":"2024-03-06T07:34:07.446979+0000","last_undegraded":"2024-03-06T07:34:07.446979+0000","last_fullsized":"2024-03-06T07:34:07.446979+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:28.950051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814937+0000","last_change":"2024-03-06T07:34:06.438725+0000","last_active":"2024-03-06T07:34:07.814937+0000","last_peered":"2024-03-06T07:34:07.814937+0000","last_clean":"2024-03-06T07:34:07.814937+0000","last_became_active":"2024-03-06T07:34:06.438416+0000","last_became_peered":"2024-03-06T07:34:06.438416+0000","last_unstale":"2024-03-06T07:34:07.814937+0000","last_undegraded":"2024-03-06T07:34:07.814937+0000","last_fullsized":"2024-03-06T07:34:07.814937+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:40.729083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411086+0000","last_change":"2024-03-06T07:34:05.437663+0000","last_active":"2024-03-06T07:34:07.411086+0000","last_peered":"2024-03-06T07:34:07.411086+0000","last_clean":"2024-03-06T07:34:07.411086+0000","last_became_active":"2024-03-06T07:34:05.437248+0000","last_became_peered":"2024-03-06T07:34:05.437248+0000","last_unstale":"2024-03-06T07:34:07.411086+0000","last_undegraded":"2024-03-06T07:34:07.411086+0000","last_fullsized":"2024-03-06T07:34:07.411086+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:35.559916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813835+0000","last_change":"2024-03-06T07:34:06.441881+0000","last_active":"2024-03-06T07:34:07.813835+0000","last_peered":"2024-03-06T07:34:07.813835+0000","last_clean":"2024-03-06T07:34:07.813835+0000","last_became_active":"2024-03-06T07:34:06.441102+0000","last_became_peered":"2024-03-06T07:34:06.441102+0000","last_unstale":"2024-03-06T07:34:07.813835+0000","last_undegraded":"2024-03-06T07:34:07.813835+0000","last_fullsized":"2024-03-06T07:34:07.813835+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:34.429264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.405933+0000","last_change":"2024-03-06T07:34:05.424548+0000","last_active":"2024-03-06T07:34:07.405933+0000","last_peered":"2024-03-06T07:34:07.405933+0000","last_clean":"2024-03-06T07:34:07.405933+0000","last_became_active":"2024-03-06T07:34:05.424027+0000","last_became_peered":"2024-03-06T07:34:05.424027+0000","last_unstale":"2024-03-06T07:34:07.405933+0000","last_undegraded":"2024-03-06T07:34:07.405933+0000","last_fullsized":"2024-03-06T07:34:07.405933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:49.850269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813472+0000","last_change":"2024-03-06T07:34:06.438719+0000","last_active":"2024-03-06T07:34:07.813472+0000","last_peered":"2024-03-06T07:34:07.813472+0000","last_clean":"2024-03-06T07:34:07.813472+0000","last_became_active":"2024-03-06T07:34:06.438418+0000","last_became_peered":"2024-03-06T07:34:06.438418+0000","last_unstale":"2024-03-06T07:34:07.813472+0000","last_undegraded":"2024-03-06T07:34:07.813472+0000","last_fullsized":"2024-03-06T07:34:07.813472+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:13.063799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813526+0000","last_change":"2024-03-06T07:34:05.433718+0000","last_active":"2024-03-06T07:34:07.813526+0000","last_peered":"2024-03-06T07:34:07.813526+0000","last_clean":"2024-03-06T07:34:07.813526+0000","last_became_active":"2024-03-06T07:34:05.433428+0000","last_became_peered":"2024-03-06T07:34:05.433428+0000","last_unstale":"2024-03-06T07:34:07.813526+0000","last_undegraded":"2024-03-06T07:34:07.813526+0000","last_fullsized":"2024-03-06T07:34:07.813526+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:31.381706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813944+0000","last_change":"2024-03-06T07:34:06.445204+0000","last_active":"2024-03-06T07:34:07.813944+0000","last_peered":"2024-03-06T07:34:07.813944+0000","last_clean":"2024-03-06T07:34:07.813944+0000","last_became_active":"2024-03-06T07:34:06.443409+0000","last_became_peered":"2024-03-06T07:34:06.443409+0000","last_unstale":"2024-03-06T07:34:07.813944+0000","last_undegraded":"2024-03-06T07:34:07.813944+0000","last_fullsized":"2024-03-06T07:34:07.813944+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:51.065725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.406010+0000","last_change":"2024-03-06T07:34:05.430948+0000","last_active":"2024-03-06T07:34:07.406010+0000","last_peered":"2024-03-06T07:34:07.406010+0000","last_clean":"2024-03-06T07:34:07.406010+0000","last_became_active":"2024-03-06T07:34:05.429980+0000","last_became_peered":"2024-03-06T07:34:05.429980+0000","last_unstale":"2024-03-06T07:34:07.406010+0000","last_undegraded":"2024-03-06T07:34:07.406010+0000","last_fullsized":"2024-03-06T07:34:07.406010+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:01.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812706+0000","last_change":"2024-03-06T07:34:06.429605+0000","last_active":"2024-03-06T07:34:07.812706+0000","last_peered":"2024-03-06T07:34:07.812706+0000","last_clean":"2024-03-06T07:34:07.812706+0000","last_became_active":"2024-03-06T07:34:06.427590+0000","last_became_peered":"2024-03-06T07:34:06.427590+0000","last_unstale":"2024-03-06T07:34:07.812706+0000","last_undegraded":"2024-03-06T07:34:07.812706+0000","last_fullsized":"2024-03-06T07:34:07.812706+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:17.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812233+0000","last_change":"2024-03-06T07:34:05.438560+0000","last_active":"2024-03-06T07:34:07.812233+0000","last_peered":"2024-03-06T07:34:07.812233+0000","last_clean":"2024-03-06T07:34:07.812233+0000","last_became_active":"2024-03-06T07:34:05.438214+0000","last_became_peered":"2024-03-06T07:34:05.438214+0000","last_unstale":"2024-03-06T07:34:07.812233+0000","last_undegraded":"2024-03-06T07:34:07.812233+0000","last_fullsized":"2024-03-06T07:34:07.812233+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:49.834846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447269+0000","last_change":"2024-03-06T07:34:06.427612+0000","last_active":"2024-03-06T07:34:07.447269+0000","last_peered":"2024-03-06T07:34:07.447269+0000","last_clean":"2024-03-06T07:34:07.447269+0000","last_became_active":"2024-03-06T07:34:06.427089+0000","last_became_peered":"2024-03-06T07:34:06.427089+0000","last_unstale":"2024-03-06T07:34:07.447269+0000","last_undegraded":"2024-03-06T07:34:07.447269+0000","last_fullsized":"2024-03-06T07:34:07.447269+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:31.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"111'2","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815370+0000","last_change":"2024-03-06T07:34:05.440223+0000","last_active":"2024-03-06T07:34:07.815370+0000","last_peered":"2024-03-06T07:34:07.815370+0000","last_clean":"2024-03-06T07:34:07.815370+0000","last_became_active":"2024-03-06T07:34:05.437938+0000","last_became_peered":"2024-03-06T07:34:05.437938+0000","last_unstale":"2024-03-06T07:34:07.815370+0000","last_undegraded":"2024-03-06T07:34:07.815370+0000","last_fullsized":"2024-03-06T07:34:07.815370+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:02.228169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815947+0000","last_change":"2024-03-06T07:34:06.442205+0000","last_active":"2024-03-06T07:34:07.815947+0000","last_peered":"2024-03-06T07:34:07.815947+0000","last_clean":"2024-03-06T07:34:07.815947+0000","last_became_active":"2024-03-06T07:34:06.441839+0000","last_became_peered":"2024-03-06T07:34:06.441839+0000","last_unstale":"2024-03-06T07:34:07.815947+0000","last_undegraded":"2024-03-06T07:34:07.815947+0000","last_fullsized":"2024-03-06T07:34:07.815947+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:54.121154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'1","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.450412+0000","last_change":"2024-03-06T07:34:05.442001+0000","last_active":"2024-03-06T07:34:07.450412+0000","last_peered":"2024-03-06T07:34:07.450412+0000","last_clean":"2024-03-06T07:34:07.450412+0000","last_became_active":"2024-03-06T07:34:05.441051+0000","last_became_peered":"2024-03-06T07:34:05.441051+0000","last_unstale":"2024-03-06T07:34:07.450412+0000","last_undegraded":"2024-03-06T07:34:07.450412+0000","last_fullsized":"2024-03-06T07:34:07.450412+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:50.867356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814910+0000","last_change":"2024-03-06T07:34:06.439451+0000","last_active":"2024-03-06T07:34:07.814910+0000","last_peered":"2024-03-06T07:34:07.814910+0000","last_clean":"2024-03-06T07:34:07.814910+0000","last_became_active":"2024-03-06T07:34:06.439145+0000","last_became_peered":"2024-03-06T07:34:06.439145+0000","last_unstale":"2024-03-06T07:34:07.814910+0000","last_undegraded":"2024-03-06T07:34:07.814910+0000","last_fullsized":"2024-03-06T07:34:07.814910+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:48.714296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.815222+0000","last_change":"2024-03-06T07:34:05.438419+0000","last_active":"2024-03-06T07:34:07.815222+0000","last_peered":"2024-03-06T07:34:07.815222+0000","last_clean":"2024-03-06T07:34:07.815222+0000","last_became_active":"2024-03-06T07:34:05.437950+0000","last_became_peered":"2024-03-06T07:34:05.437950+0000","last_unstale":"2024-03-06T07:34:07.815222+0000","last_undegraded":"2024-03-06T07:34:07.815222+0000","last_fullsized":"2024-03-06T07:34:07.815222+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:30.291791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.447094+0000","last_change":"2024-03-06T07:34:06.430263+0000","last_active":"2024-03-06T07:34:07.447094+0000","last_peered":"2024-03-06T07:34:07.447094+0000","last_clean":"2024-03-06T07:34:07.447094+0000","last_became_active":"2024-03-06T07:34:06.429695+0000","last_became_peered":"2024-03-06T07:34:06.429695+0000","last_unstale":"2024-03-06T07:34:07.447094+0000","last_undegraded":"2024-03-06T07:34:07.447094+0000","last_fullsized":"2024-03-06T07:34:07.447094+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:25.339129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814448+0000","last_change":"2024-03-06T07:34:05.431901+0000","last_active":"2024-03-06T07:34:07.814448+0000","last_peered":"2024-03-06T07:34:07.814448+0000","last_clean":"2024-03-06T07:34:07.814448+0000","last_became_active":"2024-03-06T07:34:05.431644+0000","last_became_peered":"2024-03-06T07:34:05.431644+0000","last_unstale":"2024-03-06T07:34:07.814448+0000","last_undegraded":"2024-03-06T07:34:07.814448+0000","last_fullsized":"2024-03-06T07:34:07.814448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.172323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.411537+0000","last_change":"2024-03-06T07:34:06.435827+0000","last_active":"2024-03-06T07:34:07.411537+0000","last_peered":"2024-03-06T07:34:07.411537+0000","last_clean":"2024-03-06T07:34:07.411537+0000","last_became_active":"2024-03-06T07:34:06.435510+0000","last_became_peered":"2024-03-06T07:34:06.435510+0000","last_unstale":"2024-03-06T07:34:07.411537+0000","last_undegraded":"2024-03-06T07:34:07.411537+0000","last_fullsized":"2024-03-06T07:34:07.411537+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:48.748282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.413904+0000","last_change":"2024-03-06T07:34:05.438522+0000","last_active":"2024-03-06T07:34:07.413904+0000","last_peered":"2024-03-06T07:34:07.413904+0000","last_clean":"2024-03-06T07:34:07.413904+0000","last_became_active":"2024-03-06T07:34:05.438178+0000","last_became_peered":"2024-03-06T07:34:05.438178+0000","last_unstale":"2024-03-06T07:34:07.413904+0000","last_undegraded":"2024-03-06T07:34:07.413904+0000","last_fullsized":"2024-03-06T07:34:07.413904+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:53.930083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412914+0000","last_change":"2024-03-06T07:34:06.445115+0000","last_active":"2024-03-06T07:34:07.412914+0000","last_peered":"2024-03-06T07:34:07.412914+0000","last_clean":"2024-03-06T07:34:07.412914+0000","last_became_active":"2024-03-06T07:34:06.442582+0000","last_became_peered":"2024-03-06T07:34:06.442582+0000","last_unstale":"2024-03-06T07:34:07.412914+0000","last_undegraded":"2024-03-06T07:34:07.412914+0000","last_fullsized":"2024-03-06T07:34:07.412914+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:20.264570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412457+0000","last_change":"2024-03-06T07:34:05.430327+0000","last_active":"2024-03-06T07:34:07.412457+0000","last_peered":"2024-03-06T07:34:07.412457+0000","last_clean":"2024-03-06T07:34:07.412457+0000","last_became_active":"2024-03-06T07:34:05.429963+0000","last_became_peered":"2024-03-06T07:34:05.429963+0000","last_unstale":"2024-03-06T07:34:07.412457+0000","last_undegraded":"2024-03-06T07:34:07.412457+0000","last_fullsized":"2024-03-06T07:34:07.412457+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:48.573186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814158+0000","last_change":"2024-03-06T07:34:06.442067+0000","last_active":"2024-03-06T07:34:07.814158+0000","last_peered":"2024-03-06T07:34:07.814158+0000","last_clean":"2024-03-06T07:34:07.814158+0000","last_became_active":"2024-03-06T07:34:06.440841+0000","last_became_peered":"2024-03-06T07:34:06.440841+0000","last_unstale":"2024-03-06T07:34:07.814158+0000","last_undegraded":"2024-03-06T07:34:07.814158+0000","last_fullsized":"2024-03-06T07:34:07.814158+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.034356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.813254+0000","last_change":"2024-03-06T07:34:05.424983+0000","last_active":"2024-03-06T07:34:07.813254+0000","last_peered":"2024-03-06T07:34:07.813254+0000","last_clean":"2024-03-06T07:34:07.813254+0000","last_became_active":"2024-03-06T07:34:05.424659+0000","last_became_peered":"2024-03-06T07:34:05.424659+0000","last_unstale":"2024-03-06T07:34:07.813254+0000","last_undegraded":"2024-03-06T07:34:07.813254+0000","last_fullsized":"2024-03-06T07:34:07.813254+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.506880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812927+0000","last_change":"2024-03-06T07:34:06.439910+0000","last_active":"2024-03-06T07:34:07.812927+0000","last_peered":"2024-03-06T07:34:07.812927+0000","last_clean":"2024-03-06T07:34:07.812927+0000","last_became_active":"2024-03-06T07:34:06.439586+0000","last_became_peered":"2024-03-06T07:34:06.439586+0000","last_unstale":"2024-03-06T07:34:07.812927+0000","last_undegraded":"2024-03-06T07:34:07.812927+0000","last_fullsized":"2024-03-06T07:34:07.812927+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:20.170576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.812427+0000","last_change":"2024-03-06T07:34:05.441536+0000","last_active":"2024-03-06T07:34:07.812427+0000","last_peered":"2024-03-06T07:34:07.812427+0000","last_clean":"2024-03-06T07:34:07.812427+0000","last_became_active":"2024-03-06T07:34:05.438461+0000","last_became_peered":"2024-03-06T07:34:05.438461+0000","last_unstale":"2024-03-06T07:34:07.812427+0000","last_undegraded":"2024-03-06T07:34:07.812427+0000","last_fullsized":"2024-03-06T07:34:07.812427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:02.228169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814112+0000","last_change":"2024-03-06T07:34:06.443906+0000","last_active":"2024-03-06T07:34:07.814112+0000","last_peered":"2024-03-06T07:34:07.814112+0000","last_clean":"2024-03-06T07:34:07.814112+0000","last_became_active":"2024-03-06T07:34:06.443201+0000","last_became_peered":"2024-03-06T07:34:06.443201+0000","last_unstale":"2024-03-06T07:34:07.814112+0000","last_undegraded":"2024-03-06T07:34:07.814112+0000","last_fullsized":"2024-03-06T07:34:07.814112+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.391791+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.391791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:20.170576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.412114+0000","last_change":"2024-03-06T07:34:05.440787+0000","last_active":"2024-03-06T07:34:07.412114+0000","last_peered":"2024-03-06T07:34:07.412114+0000","last_clean":"2024-03-06T07:34:07.412114+0000","last_became_active":"2024-03-06T07:34:05.438433+0000","last_became_peered":"2024-03-06T07:34:05.438433+0000","last_unstale":"2024-03-06T07:34:07.412114+0000","last_undegraded":"2024-03-06T07:34:07.412114+0000","last_fullsized":"2024-03-06T07:34:07.412114+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.375306+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.375306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:33.193361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":336,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.814276+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:34:07.814276+0000","last_peered":"2024-03-06T07:34:07.814276+0000","last_clean":"2024-03-06T07:34:07.814276+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:34:07.814276+0000","last_undegraded":"2024-03-06T07:34:07.814276+0000","last_fullsized":"2024-03-06T07:34:07.814276+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542225,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":1990656,"data_stored":1189831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1392640,"data_stored":598917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36708,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335132,"statfs":{"total":96636764160,"available":96599175168,"internally_reserved":0,"allocated":1396736,"data_stored":599451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":1388544,"data_stored":598847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542225,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36716,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335124,"statfs":{"total":96636764160,"available":96599166976,"internally_reserved":0,"allocated":1404928,"data_stored":600529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542227,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":598847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542225,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1396736,"data_stored":598971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542226,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37300,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334540,"statfs":{"total":96636764160,"available":96598568960,"internally_reserved":0,"allocated":2002944,"data_stored":1190939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:11.542 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:34:11.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:34:11.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.799+0000 7f3970c00640 1 Processor -- start 2024-03-06T07:34:11.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.799+0000 7f3970c00640 1 -- start start 2024-03-06T07:34:11.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.800+0000 7f3970c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f396c145320 0x7f396c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f3970c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f396c06baf0 0x7f396c0cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f3970c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c06c940 con 0x7f396c160420 2024-03-06T07:34:11.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f3970c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c06c6f0 con 0x7f396c06baf0 2024-03-06T07:34:11.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f3970c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c06cc20 con 0x7f396c145320 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f396c06baf0 0x7f396c0cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f396c06baf0 0x7f396c0cdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52112/0 (socket says 172.21.15.97:52112) 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 -- 172.21.15.97:0/2326149654 learned_addr learned my addr 172.21.15.97:0/2326149654 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396b400640 1 --2- 172.21.15.97:0/2326149654 >> v2:172.21.15.135:3300/0 conn(0x7f396c145320 0x7f396c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 -- 172.21.15.97:0/2326149654 >> v2:172.21.15.135:3300/0 conn(0x7f396c145320 msgr2=0x7f396c0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 --2- 172.21.15.97:0/2326149654 >> v2:172.21.15.135:3300/0 conn(0x7f396c145320 0x7f396c0b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 -- 172.21.15.97:0/2326149654 >> v1:172.21.15.97:6789/0 conn(0x7f396c160420 legacy=0x7f396c06b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.801+0000 7f396be00640 1 -- 172.21.15.97:0/2326149654 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c06cf00 con 0x7f396c06baf0 2024-03-06T07:34:11.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.803+0000 7f396be00640 1 --2- 172.21.15.97:0/2326149654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c06baf0 0x7f396c0cdf50 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f396c06c870 tx=0x7f396402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e368b2865264e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.803+0000 7f396a000640 1 -- 172.21.15.97:0/2326149654 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964035020 con 0x7f396c06baf0 2024-03-06T07:34:11.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.803+0000 7f396a000640 1 -- 172.21.15.97:0/2326149654 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3964031870 con 0x7f396c06baf0 2024-03-06T07:34:11.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.803+0000 7f396a000640 1 -- 172.21.15.97:0/2326149654 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964031c40 con 0x7f396c06baf0 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 -- 172.21.15.97:0/2326149654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c06baf0 msgr2=0x7f396c0cdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 --2- 172.21.15.97:0/2326149654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c06baf0 0x7f396c0cdf50 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f396c06c870 tx=0x7f396402b910 comp rx=0 tx=0).stop 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 -- 172.21.15.97:0/2326149654 shutdown_connections 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 --2- 172.21.15.97:0/2326149654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c06baf0 0x7f396c0cdf50 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 --2- 172.21.15.97:0/2326149654 >> v2:172.21.15.135:3300/0 conn(0x7f396c145320 0x7f396c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 -- 172.21.15.97:0/2326149654 >> 172.21.15.97:0/2326149654 conn(0x7f396c0b1a70 msgr2=0x7f396c0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 -- 172.21.15.97:0/2326149654 shutdown_connections 2024-03-06T07:34:11.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.804+0000 7f3970c00640 1 -- 172.21.15.97:0/2326149654 wait complete. 2024-03-06T07:34:11.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.806+0000 7f3970c00640 1 Processor -- start 2024-03-06T07:34:11.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.806+0000 7f3970c00640 1 -- start start 2024-03-06T07:34:11.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f3970c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c06baf0 0x7f396c13a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f3970c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 0x7f396c15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f3970c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f396c140d20 0x7f396c15d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f3970c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0b4de0 con 0x7f396c145320 2024-03-06T07:34:11.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f3970c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c06d4c0 con 0x7f396c06baf0 2024-03-06T07:34:11.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f3970c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c145700 con 0x7f396c140d20 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f396aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 0x7f396c15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f396aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 0x7f396c15bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52126/0 (socket says 172.21.15.97:52126) 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f396aa00640 1 -- 172.21.15.97:0/1063643955 learned_addr learned my addr 172.21.15.97:0/1063643955 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f396be00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f396c140d20 0x7f396c15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.807+0000 7f396b400640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c06baf0 0x7f396c13a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f396aa00640 1 -- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f396c140d20 msgr2=0x7f396c15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f396aa00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f396c140d20 0x7f396c15d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f396aa00640 1 -- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c06baf0 msgr2=0x7f396c13a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f396aa00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c06baf0 0x7f396c13a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f396aa00640 1 -- 172.21.15.97:0/1063643955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c15c6b0 con 0x7f396c145320 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f396aa00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 0x7f396c15bf60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f395800c440 tx=0x7f395800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39580052d0 con 0x7f396c145320 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958005470 con 0x7f396c145320 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958007260 con 0x7f396c145320 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c15c990 con 0x7f396c145320 2024-03-06T07:34:11.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.808+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c15cf50 con 0x7f396c145320 2024-03-06T07:34:11.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.809+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c144e60 con 0x7f396c145320 2024-03-06T07:34:11.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.811+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39580074c0 con 0x7f396c145320 2024-03-06T07:34:11.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.811+0000 7f3963400640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.811+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39580896f0 con 0x7f396c145320 2024-03-06T07:34:11.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.812+0000 7f396b400640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.812+0000 7f396b400640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f394c06b4d0 0x7f394c06d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f396c0692d0 tx=0x7f395c0039b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:11.816+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39580584d0 con 0x7f396c145320 2024-03-06T07:34:12.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.093+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f396c06e6f0 con 0x7f396c145320 2024-03-06T07:34:12.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.098+0000 7f3963400640 1 -- 172.21.15.97:0/1063643955 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3958057dd0 con 0x7f396c145320 2024-03-06T07:34:12.099 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:34:12.099 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:34:12.099 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:34:12.099 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:34:12.099 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:34:12.100 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.100+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f394c06b4d0 msgr2=0x7f394c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.100+0000 7f3970c00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f394c06b4d0 0x7f394c06d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f396c0692d0 tx=0x7f395c0039b0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.100+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 msgr2=0x7f396c15bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.100+0000 7f3970c00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 0x7f396c15bf60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f395800c440 tx=0x7f395800c900 comp rx=0 tx=0).stop 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 shutdown_connections 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f396c140d20 0x7f396c15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c145320 0x7f396c15bf60 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 --2- 172.21.15.97:0/1063643955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c06baf0 0x7f396c13a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 >> 172.21.15.97:0/1063643955 conn(0x7f396c0b1a70 msgr2=0x7f396c0b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 shutdown_connections 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.101+0000 7f3970c00640 1 -- 172.21.15.97:0/1063643955 wait complete. 2024-03-06T07:34:12.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:12.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.330+0000 7f24dac00640 1 Processor -- start 2024-03-06T07:34:12.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.330+0000 7f24dac00640 1 -- start start 2024-03-06T07:34:12.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24dac00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24d415acb0 0x7f24d40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24dac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24dac00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f24d4072340 0x7f24d406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24dac00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d415b090 con 0x7f24d4072340 2024-03-06T07:34:12.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24dac00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d40b2a30 con 0x7f24d415acb0 2024-03-06T07:34:12.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24dac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40b4360 con 0x7f24d40b4680 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24d8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f24d4072340 0x7f24d406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24d8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f24d4072340 0x7f24d406ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52130/0 (socket says 172.21.15.97:52130) 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24d8c00640 1 -- 172.21.15.97:0/2638231821 learned_addr learned my addr 172.21.15.97:0/2638231821 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24cb400640 1 --2- 172.21.15.97:0/2638231821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.332+0000 7f24d3e00640 1 --2- 172.21.15.97:0/2638231821 >> v2:172.21.15.135:3300/0 conn(0x7f24d415acb0 0x7f24d40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.333+0000 7f24d8c00640 1 -- 172.21.15.97:0/2638231821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 msgr2=0x7f24d406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.333+0000 7f24d8c00640 1 --2- 172.21.15.97:0/2638231821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.333+0000 7f24d8c00640 1 -- 172.21.15.97:0/2638231821 >> v2:172.21.15.135:3300/0 conn(0x7f24d415acb0 msgr2=0x7f24d40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.333+0000 7f24d8c00640 1 --2- 172.21.15.97:0/2638231821 >> v2:172.21.15.135:3300/0 conn(0x7f24d415acb0 0x7f24d40b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.333+0000 7f24d8c00640 1 -- 172.21.15.97:0/2638231821 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b2cf0 con 0x7f24d4072340 2024-03-06T07:34:12.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.334+0000 7f24d8c00640 1 --2- 172.21.15.97:0/2638231821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d4072340 0x7f24d406ea30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f24d40b27f0 tx=0x7f24cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11771d530070468f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.334+0000 7f24d3400640 1 -- 172.21.15.97:0/2638231821 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc02ca70 con 0x7f24d4072340 2024-03-06T07:34:12.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.334+0000 7f24d3400640 1 -- 172.21.15.97:0/2638231821 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc02cc10 con 0x7f24d4072340 2024-03-06T07:34:12.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.334+0000 7f24d3400640 1 -- 172.21.15.97:0/2638231821 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc0343f0 con 0x7f24d4072340 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 -- 172.21.15.97:0/2638231821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d4072340 msgr2=0x7f24d406ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 --2- 172.21.15.97:0/2638231821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d4072340 0x7f24d406ea30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f24d40b27f0 tx=0x7f24cc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 -- 172.21.15.97:0/2638231821 shutdown_connections 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 --2- 172.21.15.97:0/2638231821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d4072340 0x7f24d406ea30 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 --2- 172.21.15.97:0/2638231821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 --2- 172.21.15.97:0/2638231821 >> v2:172.21.15.135:3300/0 conn(0x7f24d415acb0 0x7f24d40b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 -- 172.21.15.97:0/2638231821 >> 172.21.15.97:0/2638231821 conn(0x7f24d40b1730 msgr2=0x7f24d40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 -- 172.21.15.97:0/2638231821 shutdown_connections 2024-03-06T07:34:12.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.335+0000 7f24dac00640 1 -- 172.21.15.97:0/2638231821 wait complete. 2024-03-06T07:34:12.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.337+0000 7f24dac00640 1 Processor -- start 2024-03-06T07:34:12.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.337+0000 7f24dac00640 1 -- start start 2024-03-06T07:34:12.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24dac00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24d4072340 0x7f24d41483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24dac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d41570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24dac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 0x7f24d4154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24dac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40b3d10 con 0x7f24d415acb0 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24dac00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40b32b0 con 0x7f24d4072340 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24dac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40b4010 con 0x7f24d40b4680 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 0x7f24d4154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 0x7f24d4154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52134/0 (socket says 172.21.15.97:52134) 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 -- 172.21.15.97:0/1188459800 learned_addr learned my addr 172.21.15.97:0/1188459800 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24cb400640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d41570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d3e00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24d4072340 0x7f24d41483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 -- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 msgr2=0x7f24d41570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d41570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 -- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24d4072340 msgr2=0x7f24d41483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24d4072340 0x7f24d41483a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.338+0000 7f24d8c00640 1 -- 172.21.15.97:0/1188459800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d415b590 con 0x7f24d415acb0 2024-03-06T07:34:12.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24cb400640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d41570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:12.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24d8c00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 0x7f24d4154d80 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009f40 tx=0x7f24cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc043020 con 0x7f24d415acb0 2024-03-06T07:34:12.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc03e450 con 0x7f24d415acb0 2024-03-06T07:34:12.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc009510 con 0x7f24d415acb0 2024-03-06T07:34:12.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d415b870 con 0x7f24d415acb0 2024-03-06T07:34:12.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.339+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d415be30 con 0x7f24d415acb0 2024-03-06T07:34:12.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.340+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d404f330 con 0x7f24d415acb0 2024-03-06T07:34:12.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.341+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24cc046020 con 0x7f24d415acb0 2024-03-06T07:34:12.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.342+0000 7f24d0c00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24b006b4d0 0x7f24b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.342+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24cc0acab0 con 0x7f24d415acb0 2024-03-06T07:34:12.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.342+0000 7f24d3e00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24b006b4d0 0x7f24b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.343+0000 7f24d3e00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24b006b4d0 0x7f24b006d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f24c4006fd0 tx=0x7f24c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.347+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24cc04d030 con 0x7f24d415acb0 2024-03-06T07:34:12.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.581+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24d40b27f0 con 0x7f24d415acb0 2024-03-06T07:34:12.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.582+0000 7f24d0c00640 1 -- 172.21.15.97:0/1188459800 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f24cc07b890 con 0x7f24d415acb0 2024-03-06T07:34:12.583 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:12.583 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1292572481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1292572481},{"type":"v1","addr":"172.21.15.135:6835","nonce":1292572481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7701,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/232111965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":232111965},{"type":"v1","addr":"172.21.15.97:6835","nonce":232111965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2296434777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2296434777},{"type":"v1","addr":"172.21.15.97:6837","nonce":2296434777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:09.132267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7716":{"gid":7716,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6833/322807700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":322807700},{"type":"v1","addr":"172.21.15.135:6833","nonce":322807700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7716,"qdb_cluster":[7716]},"id":8}]} 2024-03-06T07:34:12.583 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 74 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24b006b4d0 msgr2=0x7f24b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24b006b4d0 0x7f24b006d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f24c4006fd0 tx=0x7f24c4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 msgr2=0x7f24d4154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 0x7f24d4154d80 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009f40 tx=0x7f24cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 shutdown_connections 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24b006b4d0 0x7f24b006d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24d415acb0 0x7f24d4154d80 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24d40b4680 0x7f24d41570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 --2- 172.21.15.97:0/1188459800 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24d4072340 0x7f24d41483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.584+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 >> 172.21.15.97:0/1188459800 conn(0x7f24d40b1730 msgr2=0x7f24d415e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.585+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 shutdown_connections 2024-03-06T07:34:12.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:12.585+0000 7f24dac00640 1 -- 172.21.15.97:0/1188459800 wait complete. 2024-03-06T07:34:13.004 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:05.874005+0000', 'modified': '2024-03-06T07:34:09.132267+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7716': {'gid': 7716, 'name': 'b', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.135:6833/322807700', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6832', 'nonce': 322807700}, {'type': 'v1', 'addr': '172.21.15.135:6833', 'nonce': 322807700}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7716, 'qdb_cluster': [7716]} 2024-03-06T07:34:13.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:13.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:34:13.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.221+0000 7f53e6200640 1 Processor -- start 2024-03-06T07:34:13.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.221+0000 7f53e6200640 1 -- start start 2024-03-06T07:34:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.223+0000 7f53e6200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f53e01474c0 0x7f53e00cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.223+0000 7f53e6200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e00ae380 con 0x7f53e01444e0 2024-03-06T07:34:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.223+0000 7f53e6200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e00ae660 con 0x7f53e00c5040 2024-03-06T07:34:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.223+0000 7f53e6200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e00ae940 con 0x7f53e01474c0 2024-03-06T07:34:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.223+0000 7f53df400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f53e01444e0 0x7f53e00c4930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33760/0 (socket says 172.21.15.97:33760) 2024-03-06T07:34:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.223+0000 7f53df400640 1 -- 172.21.15.97:0/3239445182 learned_addr learned my addr 172.21.15.97:0/3239445182 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dfe00640 1 --2- 172.21.15.97:0/3239445182 >> v2:172.21.15.135:3300/0 conn(0x7f53e01474c0 0x7f53e00cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792001253 0 0) 0x7f53e00ae380 con 0x7f53e01444e0 2024-03-06T07:34:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c0003520 con 0x7f53e01444e0 2024-03-06T07:34:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217857963 0 0) 0x7f53c0003520 con 0x7f53e01444e0 2024-03-06T07:34:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 >> v2:172.21.15.135:3300/0 conn(0x7f53e01474c0 msgr2=0x7f53e00cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 --2- 172.21.15.97:0/3239445182 >> v2:172.21.15.135:3300/0 conn(0x7f53e01474c0 0x7f53e00cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 >> v1:172.21.15.135:6789/0 conn(0x7f53e00c5040 legacy=0x7f53e00cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00aec20 con 0x7f53e01444e0 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f53d0001670 con 0x7f53e01444e0 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 >> v1:172.21.15.97:6789/0 conn(0x7f53e01444e0 legacy=0x7f53e00c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00039b0 0x7f53c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c0004800 0x7f53c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00051c0 0x7f53c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00ae940 con 0x7f53c00039b0 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c000c110 con 0x7f53c00051c0 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.224+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c000c310 con 0x7f53c0004800 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53df400640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00039b0 0x7f53c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53df400640 1 -- 172.21.15.97:0/3239445182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c0004800 msgr2=0x7f53c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53df400640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53c0004800 0x7f53c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53df400640 1 -- 172.21.15.97:0/3239445182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00051c0 msgr2=0x7f53c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53df400640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00051c0 0x7f53c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53df400640 1 -- 172.21.15.97:0/3239445182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c0003520 con 0x7f53c00039b0 2024-03-06T07:34:13.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.225+0000 7f53dfe00640 1 -- 172.21.15.97:0/3239445182 reap_dead start 2024-03-06T07:34:13.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.226+0000 7f53df400640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00039b0 0x7f53c0003dd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c240 tx=0x7f53c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7448fb7875c919db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.226+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0007d10 con 0x7f53c00039b0 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.226+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d0030070 con 0x7f53c00039b0 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.226+0000 7f53dea00640 1 -- 172.21.15.97:0/3239445182 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d002cdc0 con 0x7f53c00039b0 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 -- 172.21.15.97:0/3239445182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00039b0 msgr2=0x7f53c0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00039b0 0x7f53c0003dd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c240 tx=0x7f53c0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 -- 172.21.15.97:0/3239445182 shutdown_connections 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 --2- 172.21.15.97:0/3239445182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53c00039b0 0x7f53c0003dd0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 -- 172.21.15.97:0/3239445182 >> 172.21.15.97:0/3239445182 conn(0x7f53e00cc4f0 msgr2=0x7f53e00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 -- 172.21.15.97:0/3239445182 shutdown_connections 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.227+0000 7f53e6200640 1 -- 172.21.15.97:0/3239445182 wait complete. 2024-03-06T07:34:13.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.229+0000 7f53e6200640 1 Processor -- start 2024-03-06T07:34:13.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.229+0000 7f53e6200640 1 -- start start 2024-03-06T07:34:13.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53e6200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00039b0 0x7f53e00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53e6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53e015d810 0x7f53e0146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53e6200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 0x7f53e013e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53e6200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e01448c0 con 0x7f53e0146f40 2024-03-06T07:34:13.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53e6200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0147cb0 con 0x7f53c00039b0 2024-03-06T07:34:13.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53e6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0144a40 con 0x7f53e015d810 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 0x7f53e013e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 0x7f53e013e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52150/0 (socket says 172.21.15.97:52150) 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 -- 172.21.15.97:0/454068426 learned_addr learned my addr 172.21.15.97:0/454068426 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53df400640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00039b0 0x7f53e00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53d7e00640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53e015d810 0x7f53e0146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 -- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53e015d810 msgr2=0x7f53e0146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53e015d810 0x7f53e0146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 -- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00039b0 msgr2=0x7f53e00b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00039b0 0x7f53e00b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53dfe00640 1 -- 172.21.15.97:0/454068426 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00b20b0 con 0x7f53e0146f40 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.230+0000 7f53df400640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00039b0 0x7f53e00b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.231+0000 7f53dfe00640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 0x7f53e013e660 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f53e00707f0 tx=0x7f53d8007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.231+0000 7f53d7400640 1 -- 172.21.15.97:0/454068426 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d800ec50 con 0x7f53e0146f40 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.231+0000 7f53d7400640 1 -- 172.21.15.97:0/454068426 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d8019070 con 0x7f53e0146f40 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.231+0000 7f53d7400640 1 -- 172.21.15.97:0/454068426 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8014420 con 0x7f53e0146f40 2024-03-06T07:34:13.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.231+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e00b1530 con 0x7f53e0146f40 2024-03-06T07:34:13.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.231+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e00b1af0 con 0x7f53e0146f40 2024-03-06T07:34:13.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.233+0000 7f53d7400640 1 -- 172.21.15.97:0/454068426 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53d8005d10 con 0x7f53e0146f40 2024-03-06T07:34:13.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.234+0000 7f53d7400640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53bc06b500 0x7f53bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.234+0000 7f53d7400640 1 -- 172.21.15.97:0/454068426 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53d808a310 con 0x7f53e0146f40 2024-03-06T07:34:13.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.234+0000 7f53df400640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53bc06b500 0x7f53bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.235+0000 7f53e8f77740 1 Processor -- start 2024-03-06T07:34:13.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.235+0000 7f53df400640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53bc06b500 0x7f53bc06d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f53d0034000 tx=0x7f53d0009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.235+0000 7f53e8f77740 1 -- start start 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419d7b0 0x55eb24170750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419db90 0x55eb241a0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 0x55eb241a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53e8f77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb240da600 con 0x55eb24170c90 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb24083930 con 0x55eb2419db90 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb240d66a0 con 0x55eb2419d7b0 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 0x55eb241a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 0x55eb241a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52164/0 (socket says 172.21.15.97:52164) 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.236+0000 7f53dfe00640 1 -- 172.21.15.97:0/3224097595 learned_addr learned my addr 172.21.15.97:0/3224097595 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53df400640 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419d7b0 0x55eb24170750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53d7e00640 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419db90 0x55eb241a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53dfe00640 1 -- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419d7b0 msgr2=0x55eb24170750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53dfe00640 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419d7b0 0x55eb24170750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53dfe00640 1 -- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419db90 msgr2=0x55eb241a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53dfe00640 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419db90 0x55eb241a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53dfe00640 1 -- 172.21.15.97:0/3224097595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb23def2d0 con 0x55eb24170c90 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53dfe00640 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 0x55eb241a0a80 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f53d8060c60 tx=0x7f53d8058b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a0455b31d3ecc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53cf400640 1 -- 172.21.15.97:0/3224097595 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8056b20 con 0x55eb24170c90 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53cf400640 1 -- 172.21.15.97:0/3224097595 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d8056d80 con 0x55eb24170c90 2024-03-06T07:34:13.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.237+0000 7f53cf400640 1 -- 172.21.15.97:0/3224097595 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8058ce0 con 0x55eb24170c90 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 -- 172.21.15.97:0/3224097595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 msgr2=0x55eb241a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 0x55eb241a0a80 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f53d8060c60 tx=0x7f53d8058b40 comp rx=0 tx=0).stop 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 -- 172.21.15.97:0/3224097595 shutdown_connections 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb24170c90 0x55eb241a0a80 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419db90 0x55eb241a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3224097595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419d7b0 0x55eb24170750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 -- 172.21.15.97:0/3224097595 >> 172.21.15.97:0/3224097595 conn(0x55eb241729c0 msgr2=0x55eb241bb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 -- 172.21.15.97:0/3224097595 shutdown_connections 2024-03-06T07:34:13.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.238+0000 7f53e8f77740 1 -- 172.21.15.97:0/3224097595 wait complete. 2024-03-06T07:34:13.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.239+0000 7f53e8f77740 1 -- start start 2024-03-06T07:34:13.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.239+0000 7f53e8f77740 1 Processor -- start 2024-03-06T07:34:13.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 0x55eb241a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419db90 0x55eb241a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb241cb760 0x55eb241a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb24174f60 con 0x55eb2419d7b0 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb24096d90 con 0x55eb241cb760 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb24174350 con 0x55eb2419db90 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53e8f77740 10 client.0 _ll_register_callbacks cb 0x55eb240d2350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53df400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 0x55eb241a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53df400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 0x55eb241a7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52166/0 (socket says 172.21.15.97:52166) 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53df400640 1 -- 172.21.15.97:0/3798833062 learned_addr learned my addr 172.21.15.97:0/3798833062 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53dfe00640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb241cb760 0x55eb241a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53df400640 1 -- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419db90 msgr2=0x55eb241a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.240+0000 7f53d7e00640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419db90 0x55eb241a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53df400640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419db90 0x55eb241a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53df400640 1 -- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb241cb760 msgr2=0x55eb241a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53df400640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb241cb760 0x55eb241a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53df400640 1 -- 172.21.15.97:0/3798833062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb23eb96b0 con 0x55eb2419d7b0 2024-03-06T07:34:13.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53dfe00640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb241cb760 0x55eb241a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53df400640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 0x55eb241a7460 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c360 tx=0x7f53d002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c738f3a0bb24eabc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53c7e00640 1 -- 172.21.15.97:0/3798833062 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d002ce60 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53c7e00640 1 -- 172.21.15.97:0/3798833062 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53d003d070 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53d8056d80 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53c7e00640 1 -- 172.21.15.97:0/3798833062 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53d0007070 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53e8f77740 10 client.7854 fetch_fsmap learned FSMap version 74 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.241+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x55eb241a79a0 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53c7e00640 1 -- 172.21.15.97:0/3798833062 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d8056d80 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53c7e00640 1 -- 172.21.15.97:0/3798833062 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 74) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f53d0007210 con 0x55eb2419d7b0 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53e8f77740 10 client.7854 fetch_fsmap finished waiting for FSMap version 74 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53e8f77740 10 client.7854 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53e8f77740 20 client.7854 populate_metadata read hostname 'smithi097' 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241cf320 0x55eb241cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53e8f77740 4 client.7854 mds_command: new command op to 7716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:13.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 --> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb240e7b40 con 0x55eb241cf320 2024-03-06T07:34:13.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.242+0000 7f53d7e00640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241cf320 0x55eb241cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.243+0000 7f53d7e00640 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241cf320 0x55eb241cf740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56bfa2f523c9d966 server_cookie=f29d6c7d30091e21 in_seq=0 out_seq=0 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.243+0000 7f53c7e00640 10 client.7854 ms_handle_connect on v2:172.21.15.135:6832/322807700 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.245+0000 7f53c7e00640 1 -- 172.21.15.97:0/3798833062 <== mds.0 v2:172.21.15.135:6832/322807700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eb240e7b40 con 0x55eb241cf320 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.245+0000 7f53c7e00640 10 client.7854 handle_command_reply: tid=0 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.245+0000 7f53e8f77740 1 client.7854 shutdown 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.245+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 msgr2=0x55eb241a7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.245+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 0x55eb241a7460 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c360 tx=0x7f53d002e070 comp rx=0 tx=0).stop 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 shutdown_connections 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241cf320 0x55eb241cf740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb241cb760 0x55eb241a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb2419db90 0x55eb241a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 --2- 172.21.15.97:0/3798833062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419d7b0 0x55eb241a7460 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 >> 172.21.15.97:0/3798833062 conn(0x55eb24170c90 msgr2=0x55eb241a0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53c7e00640 0 client.7854 ms_handle_reset on v2:172.21.15.135:6832/322807700 2024-03-06T07:34:13.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 shutdown_connections 2024-03-06T07:34:13.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 1 -- 172.21.15.97:0/3798833062 wait complete. 2024-03-06T07:34:13.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.246+0000 7f53e8f77740 20 client.7854 trim_cache size 0 max 16384 2024-03-06T07:34:13.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.264+0000 7f53e8f77740 1 Processor -- start 2024-03-06T07:34:13.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.264+0000 7f53e8f77740 1 -- start start 2024-03-06T07:34:13.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb241ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb241c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53e8f77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e01ffd20 con 0x55eb2419db90 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00b0c40 con 0x55eb2419d7b0 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0145a40 con 0x55eb241ce700 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb241c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb241c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52176/0 (socket says 172.21.15.97:52176) 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53d7e00640 1 -- 172.21.15.97:0/501330162 learned_addr learned my addr 172.21.15.97:0/501330162 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53dfe00640 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.265+0000 7f53df400640 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb241ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53d7e00640 1 -- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 msgr2=0x55eb241ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53d7e00640 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53d7e00640 1 -- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 msgr2=0x55eb241ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53d7e00640 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb241ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53d7e00640 1 -- 172.21.15.97:0/501330162 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb24171200 con 0x55eb2419db90 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53d7e00640 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb241c9c10 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f53e006a250 tx=0x7f53c8001940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4637def8e3a531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53cea00640 1 -- 172.21.15.97:0/501330162 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c80035f0 con 0x55eb2419db90 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53cea00640 1 -- 172.21.15.97:0/501330162 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c8003790 con 0x55eb2419db90 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.266+0000 7f53cea00640 1 -- 172.21.15.97:0/501330162 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8003b80 con 0x55eb2419db90 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 -- 172.21.15.97:0/501330162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 msgr2=0x55eb241c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb241c9c10 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f53e006a250 tx=0x7f53c8001940 comp rx=0 tx=0).stop 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 -- 172.21.15.97:0/501330162 shutdown_connections 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb241c9c10 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 --2- 172.21.15.97:0/501330162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb241ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 -- 172.21.15.97:0/501330162 >> 172.21.15.97:0/501330162 conn(0x55eb24170c90 msgr2=0x55eb24170570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 -- 172.21.15.97:0/501330162 shutdown_connections 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.267+0000 7f53e8f77740 1 -- 172.21.15.97:0/501330162 wait complete. 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.268+0000 7f53e8f77740 1 -- start start 2024-03-06T07:34:13.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.268+0000 7f53e8f77740 1 Processor -- start 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb2419c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb2419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241bcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb241cbb70 con 0x55eb2419db90 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb241a5280 con 0x55eb2419d7b0 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb241a6510 con 0x55eb241ce700 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53e8f77740 10 client.0 _ll_register_callbacks cb 0x7f53c00036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb2419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb2419c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52180/0 (socket says 172.21.15.97:52180) 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 -- 172.21.15.97:0/2207709793 learned_addr learned my addr 172.21.15.97:0/2207709793 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53df400640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb2419c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53dfe00640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241bcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 -- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 msgr2=0x55eb241bcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241bcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 -- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 msgr2=0x55eb2419c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.269+0000 7f53d7e00640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb2419c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53d7e00640 1 -- 172.21.15.97:0/2207709793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb24173f50 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53dfe00640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241bcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53d7e00640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb2419c5f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4caeed4fae3c564 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53c7e00640 1 -- 172.21.15.97:0/2207709793 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8003940 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53c7e00640 1 -- 172.21.15.97:0/2207709793 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53c8003ae0 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53c8003790 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53c7e00640 1 -- 172.21.15.97:0/2207709793 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53c8032050 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53e8f77740 10 client.7860 fetch_fsmap learned FSMap version 74 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.270+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x55eb241a2980 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53c7e00640 1 -- 172.21.15.97:0/2207709793 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c8003790 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53c7e00640 1 -- 172.21.15.97:0/2207709793 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 74) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f53c8035020 con 0x55eb2419db90 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53e8f77740 10 client.7860 fetch_fsmap finished waiting for FSMap version 74 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53e8f77740 10 client.7860 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53e8f77740 20 client.7860 populate_metadata read hostname 'smithi097' 2024-03-06T07:34:13.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53e8f77740 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241bd320 0x55eb24175ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53e8f77740 4 client.7860 mds_command: new command op to 7716 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:13.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 --> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eb24171dd0 con 0x55eb241bd320 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.271+0000 7f53df400640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241bd320 0x55eb24175ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.273+0000 7f53c7e00640 10 client.7860 ms_handle_connect on v2:172.21.15.135:6832/322807700 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.272+0000 7f53df400640 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241bd320 0x55eb24175ed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74d0d0761f0a642d server_cookie=64f45eaebb1859bd in_seq=0 out_seq=0 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.273+0000 7f53c7e00640 1 -- 172.21.15.97:0/2207709793 <== mds.0 v2:172.21.15.135:6832/322807700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55eb24171dd0 con 0x55eb241bd320 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.273+0000 7f53c7e00640 10 client.7860 handle_command_reply: tid=0 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.273+0000 7f53e8f77740 1 client.7860 shutdown 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 msgr2=0x55eb2419c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb2419c5f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c8001120 comp rx=0 tx=0).stop 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 shutdown_connections 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:6832/322807700,v1:172.21.15.135:6833/322807700] conn(0x55eb241bd320 0x55eb24175ed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55eb241ce700 0x55eb241bcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb2419db90 0x55eb2419c5f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 --2- 172.21.15.97:0/2207709793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eb2419d7b0 0x55eb2419c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53c7e00640 0 client.7860 ms_handle_reset on v2:172.21.15.135:6832/322807700 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 >> 172.21.15.97:0/2207709793 conn(0x55eb24170c90 msgr2=0x55eb241747c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 shutdown_connections 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 1 -- 172.21.15.97:0/2207709793 wait complete. 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e8f77740 20 client.7860 trim_cache size 0 max 16384 2024-03-06T07:34:13.276 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 7716, 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 6.8472259270000002, 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 74, 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 112, 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 112, 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 13.30183385 2024-03-06T07:34:13.277 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53bc06b500 msgr2=0x7f53bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e6200640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53bc06b500 0x7f53bc06d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f53d0034000 tx=0x7f53d0009030 comp rx=0 tx=0).stop 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 msgr2=0x7f53e013e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.274+0000 7f53e6200640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 0x7f53e013e660 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f53e00707f0 tx=0x7f53d8007f20 comp rx=0 tx=0).stop 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 shutdown_connections 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53bc06b500 0x7f53bc06d9c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53e0146f40 0x7f53e013e660 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53e015d810 0x7f53e0146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 --2- 172.21.15.97:0/454068426 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53c00039b0 0x7f53e00b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 >> 172.21.15.97:0/454068426 conn(0x7f53e00cc4f0 msgr2=0x7f53e006cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 shutdown_connections 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.275+0000 7f53e6200640 1 -- 172.21.15.97:0/454068426 wait complete. 2024-03-06T07:34:13.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:34:13.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.478+0000 7f8112800640 1 Processor -- start 2024-03-06T07:34:13.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.478+0000 7f8112800640 1 -- start start 2024-03-06T07:34:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.479+0000 7f8112800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f810c0b40c0 0x7f810c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.479+0000 7f8112800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c161900 con 0x7f810c0ae670 2024-03-06T07:34:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.479+0000 7f8112800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0b49e0 con 0x7f810c1620c0 2024-03-06T07:34:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.479+0000 7f8112800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c15ebd0 con 0x7f810c0b40c0 2024-03-06T07:34:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.480+0000 7f8110a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f810c0ae670 0x7f810c15e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33774/0 (socket says 172.21.15.97:33774) 2024-03-06T07:34:13.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.480+0000 7f8110a00640 1 -- 172.21.15.97:0/1767278061 learned_addr learned my addr 172.21.15.97:0/1767278061 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:13.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.480+0000 7f8111400640 1 --2- 172.21.15.97:0/1767278061 >> v2:172.21.15.135:3300/0 conn(0x7f810c0b40c0 0x7f810c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.480+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934901990 0 0) 0x7f810c161900 con 0x7f810c0ae670 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.480+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f4003520 con 0x7f810c0ae670 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387559073 0 0) 0x7f80f4003520 con 0x7f810c0ae670 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 >> v2:172.21.15.135:3300/0 conn(0x7f810c0b40c0 msgr2=0x7f810c0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 --2- 172.21.15.97:0/1767278061 >> v2:172.21.15.135:3300/0 conn(0x7f810c0b40c0 0x7f810c0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 >> v1:172.21.15.135:6789/0 conn(0x7f810c1620c0 legacy=0x7f810c0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c15eeb0 con 0x7f810c0ae670 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f80f8001670 con 0x7f810c0ae670 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 >> v1:172.21.15.97:6789/0 conn(0x7f810c0ae670 legacy=0x7f810c15e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80f4003a00 0x7f80f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f4004800 0x7f80f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80f40051c0 0x7f80f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c15ebd0 con 0x7f80f40051c0 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f400c0c0 con 0x7f80f4003a00 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f400c2c0 con 0x7f80f4004800 2024-03-06T07:34:13.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f8111e00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80f40051c0 0x7f80f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f8111400640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80f4003a00 0x7f80f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.481+0000 7f8110a00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f4004800 0x7f80f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111e00640 1 -- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f4004800 msgr2=0x7f80f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111e00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f4004800 0x7f80f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111e00640 1 -- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80f4003a00 msgr2=0x7f80f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111e00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80f4003a00 0x7f80f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111e00640 1 -- 172.21.15.97:0/1767278061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f4003520 con 0x7f80f40051c0 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8110a00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f4004800 0x7f80f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111400640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80f4003a00 0x7f80f4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.482+0000 7f8111e00640 1 -- 172.21.15.97:0/1767278061 reap_dead start 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f8111e00640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80f40051c0 0x7f80f400b9c0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8104001e50 tx=0x7f80f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6db24499f1a56b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104001a70 con 0x7f80f40051c0 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8104003b90 con 0x7f80f40051c0 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f810be00640 1 -- 172.21.15.97:0/1767278061 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040050d0 con 0x7f80f40051c0 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f8112800640 1 -- 172.21.15.97:0/1767278061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80f40051c0 msgr2=0x7f80f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f8112800640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80f40051c0 0x7f80f400b9c0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8104001e50 tx=0x7f80f4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:13.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f8112800640 1 -- 172.21.15.97:0/1767278061 shutdown_connections 2024-03-06T07:34:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f8112800640 1 --2- 172.21.15.97:0/1767278061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80f40051c0 0x7f80f400b9c0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.483+0000 7f8112800640 1 -- 172.21.15.97:0/1767278061 >> 172.21.15.97:0/1767278061 conn(0x7f810c06afc0 msgr2=0x7f810c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.484+0000 7f8112800640 1 -- 172.21.15.97:0/1767278061 shutdown_connections 2024-03-06T07:34:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.484+0000 7f8112800640 1 -- 172.21.15.97:0/1767278061 wait complete. 2024-03-06T07:34:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.485+0000 7f8112800640 1 Processor -- start 2024-03-06T07:34:13.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.485+0000 7f8112800640 1 -- start start 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8112800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f40051c0 0x7f810c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8112800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 0x7f810c15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8112800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f810c146e20 0x7f810c1472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8112800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c158660 con 0x7f810c156ef0 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8112800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c161ba0 con 0x7f810c146e20 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8112800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c1587e0 con 0x7f80f40051c0 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.486+0000 7f8110a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 0x7f810c15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 0x7f810c15d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52188/0 (socket says 172.21.15.97:52188) 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 -- 172.21.15.97:0/764587413 learned_addr learned my addr 172.21.15.97:0/764587413 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 -- 172.21.15.97:0/764587413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f40051c0 msgr2=0x7f810c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f40051c0 0x7f810c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 -- 172.21.15.97:0/764587413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f810c146e20 msgr2=0x7f810c1472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f810c146e20 0x7f810c1472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 -- 172.21.15.97:0/764587413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c15a820 con 0x7f810c156ef0 2024-03-06T07:34:13.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8110a00640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 0x7f810c15d1f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f80f800d770 tx=0x7f80f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8019070 con 0x7f810c156ef0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f80074e0 con 0x7f810c156ef0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c15ab00 con 0x7f810c156ef0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.487+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80049e0 con 0x7f810c156ef0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.488+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c15b0c0 con 0x7f810c156ef0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.489+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c04f330 con 0x7f810c156ef0 2024-03-06T07:34:13.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.490+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80f801c070 con 0x7f810c156ef0 2024-03-06T07:34:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.490+0000 7f8109600640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f80ec06b500 0x7f80ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.491+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80f8089c30 con 0x7f810c156ef0 2024-03-06T07:34:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.491+0000 7f8111400640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f80ec06b500 0x7f80ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.491+0000 7f8111400640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f80ec06b500 0x7f80ec06d9c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f81000099e0 tx=0x7f8100008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.495+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80f800de20 con 0x7f810c156ef0 2024-03-06T07:34:13.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.730+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f810c1611a0 con 0x7f810c156ef0 2024-03-06T07:34:13.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.731+0000 7f8109600640 1 -- 172.21.15.97:0/764587413 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7f80f8058ad0 con 0x7f810c156ef0 2024-03-06T07:34:13.732 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f80ec06b500 msgr2=0x7f80ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f80ec06b500 0x7f80ec06d9c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f81000099e0 tx=0x7f8100008040 comp rx=0 tx=0).stop 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 msgr2=0x7f810c15d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 0x7f810c15d1f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f80f800d770 tx=0x7f80f800dc30 comp rx=0 tx=0).stop 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 shutdown_connections 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f80ec06b500 0x7f80ec06d9c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f810c146e20 0x7f810c1472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f810c156ef0 0x7f810c15d1f0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 --2- 172.21.15.97:0/764587413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f80f40051c0 0x7f810c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.733+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 >> 172.21.15.97:0/764587413 conn(0x7f810c06afc0 msgr2=0x7f810c0cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.734+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 shutdown_connections 2024-03-06T07:34:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:13.734+0000 7f8112800640 1 -- 172.21.15.97:0/764587413 wait complete. 2024-03-06T07:34:13.752 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:34:13.753 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:13.753 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:34:13.806 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:34:13.806 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:34:13.806 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42221sec preferred_lft 42221sec 2024-03-06T07:34:13.807 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:13.808 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:34:13.809 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:13.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:13.809 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:34:13.898 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:34:13.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:13.903 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:34:13.953 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:34:13.953 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:34:13.953 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:34:13.954 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:34:13.954 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:34:13.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:13.954 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:34:13.954 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:34:13.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:34:13.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:34:13.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:34:13.954 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:34:14.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.134 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:34:14.185 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:14.185 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:14.186 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:14.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:34:14.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:14.275 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:14.327 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:14.327 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:14.327 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:14.379 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:34:14.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:14.519 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:14.519 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:14.551 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:14.551 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:14.551 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:34:14.551 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:34:14.551 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:34:14.553 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:34:14.553 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:34:14.553 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:34:14.553 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:34:14.553 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:34:14.558 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:34:14.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:14.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.648 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi097.stdout:client7869 2024-03-06T07:34:14.723 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:34:14.723 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:14.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.774 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:34:14.886 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:34:14.886 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:14.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.938 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:34:15.145 INFO:teuthology.orchestra.run.smithi097.stderr:8+0 records in 2024-03-06T07:34:15.145 INFO:teuthology.orchestra.run.smithi097.stderr:8+0 records out 2024-03-06T07:34:15.145 INFO:teuthology.orchestra.run.smithi097.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.150989 s, 55.6 MB/s 2024-03-06T07:34:15.147 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:34:15.147 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:34:15.147 DEBUG:teuthology.orchestra.run.smithi097:> import stat 2024-03-06T07:34:15.147 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:34:15.147 DEBUG:teuthology.orchestra.run.smithi097:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:34:15.147 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:34:15.287 INFO:teuthology.orchestra.run.smithi097.stdout:1099511627777 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:34:15.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.301 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:34:15.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.400 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:34:15.460 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:34:15.460 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:34:15.460 DEBUG:teuthology.orchestra.run.smithi097:> import stat 2024-03-06T07:34:15.460 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:34:15.460 DEBUG:teuthology.orchestra.run.smithi097:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:34:15.461 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:34:15.588 INFO:teuthology.orchestra.run.smithi097.stdout:1099511627777 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:34:15.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:15.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.848+0000 7fd436800640 1 Processor -- start 2024-03-06T07:34:15.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.848+0000 7fd436800640 1 -- start start 2024-03-06T07:34:15.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd436800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd4300b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd436800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 0x7fd4300ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd436800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43015b090 con 0x7fd4300b4680 2024-03-06T07:34:15.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd436800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4300b2a30 con 0x7fd43006ec00 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd436800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4300b4360 con 0x7fd43015acb0 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd42fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 0x7fd4300ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd42fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 0x7fd4300ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:60260/0 (socket says 172.21.15.97:60260) 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd42f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd4300b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.850+0000 7fd42fe00640 1 -- 172.21.15.97:0/290741967 learned_addr learned my addr 172.21.15.97:0/290741967 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.851+0000 7fd42f400640 1 -- 172.21.15.97:0/290741967 >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 msgr2=0x7fd4300ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.851+0000 7fd42f400640 1 --2- 172.21.15.97:0/290741967 >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 0x7fd4300ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.851+0000 7fd42f400640 1 -- 172.21.15.97:0/290741967 >> v1:172.21.15.135:6789/0 conn(0x7fd4300b4680 legacy=0x7fd43006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.851+0000 7fd42f400640 1 -- 172.21.15.97:0/290741967 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4300b2cf0 con 0x7fd43015acb0 2024-03-06T07:34:15.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.851+0000 7fd42fe00640 1 --2- 172.21.15.97:0/290741967 >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 0x7fd4300ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:15.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.852+0000 7fd42f400640 1 --2- 172.21.15.97:0/290741967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd4300b37d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd4200097a0 tx=0x7fd42002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23cf1998d3f7179c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.853+0000 7fd42ea00640 1 -- 172.21.15.97:0/290741967 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd420035020 con 0x7fd43015acb0 2024-03-06T07:34:15.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.853+0000 7fd42ea00640 1 -- 172.21.15.97:0/290741967 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd420031810 con 0x7fd43015acb0 2024-03-06T07:34:15.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.853+0000 7fd42ea00640 1 -- 172.21.15.97:0/290741967 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd420031be0 con 0x7fd43015acb0 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 -- 172.21.15.97:0/290741967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 msgr2=0x7fd4300b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 --2- 172.21.15.97:0/290741967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd4300b37d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd4200097a0 tx=0x7fd42002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 -- 172.21.15.97:0/290741967 shutdown_connections 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 --2- 172.21.15.97:0/290741967 >> v2:172.21.15.135:3300/0 conn(0x7fd43006ec00 0x7fd4300ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 --2- 172.21.15.97:0/290741967 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd4300b37d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 -- 172.21.15.97:0/290741967 >> 172.21.15.97:0/290741967 conn(0x7fd4300b1730 msgr2=0x7fd4300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 -- 172.21.15.97:0/290741967 shutdown_connections 2024-03-06T07:34:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.854+0000 7fd436800640 1 -- 172.21.15.97:0/290741967 wait complete. 2024-03-06T07:34:15.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.856+0000 7fd436800640 1 Processor -- start 2024-03-06T07:34:15.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.856+0000 7fd436800640 1 -- start start 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd436800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd43006ec00 0x7fd4300c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd436800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 0x7fd430146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd436800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd430153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd436800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4300b3d10 con 0x7fd4300b4680 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd436800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4300b32b0 con 0x7fd43006ec00 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd436800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4300b4010 con 0x7fd43015acb0 2024-03-06T07:34:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd427e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 0x7fd430146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd427e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 0x7fd430146380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45758/0 (socket says 172.21.15.97:45758) 2024-03-06T07:34:15.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd427e00640 1 -- 172.21.15.97:0/1777651599 learned_addr learned my addr 172.21.15.97:0/1777651599 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:15.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd42f400640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd43006ec00 0x7fd4300c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.857+0000 7fd42fe00640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd430153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427e00640 1 -- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 msgr2=0x7fd430153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427e00640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd430153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427e00640 1 -- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd43006ec00 msgr2=0x7fd4300c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427e00640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd43006ec00 0x7fd4300c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427e00640 1 -- 172.21.15.97:0/1777651599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4300c9ff0 con 0x7fd4300b4680 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd42f400640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd43006ec00 0x7fd4300c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427e00640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 0x7fd430146380 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fd41800c400 tx=0x7fd41800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4180052d0 con 0x7fd4300b4680 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd418005490 con 0x7fd4300b4680 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4300ca2d0 con 0x7fd4300b4680 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd418007260 con 0x7fd4300b4680 2024-03-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.858+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4301510c0 con 0x7fd4300b4680 2024-03-06T07:34:15.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.859+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43004f330 con 0x7fd4300b4680 2024-03-06T07:34:15.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.860+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd418007400 con 0x7fd4300b4680 2024-03-06T07:34:15.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.861+0000 7fd427400640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd40c06b4d0 0x7fd40c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.861+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4180897f0 con 0x7fd4300b4680 2024-03-06T07:34:15.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.862+0000 7fd42f400640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd40c06b4d0 0x7fd40c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.862+0000 7fd42f400640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd40c06b4d0 0x7fd40c06d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd420036000 tx=0x7fd420008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:15.866+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4180585d0 con 0x7fd4300b4680 2024-03-06T07:34:16.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.106+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4300c9d00 con 0x7fd4300b4680 2024-03-06T07:34:16.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.107+0000 7fd427400640 1 -- 172.21.15.97:0/1777651599 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fd418057ed0 con 0x7fd4300b4680 2024-03-06T07:34:16.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:16.109 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1292572481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1292572481},{"type":"v1","addr":"172.21.15.135:6835","nonce":1292572481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7701,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/232111965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":232111965},{"type":"v1","addr":"172.21.15.97:6835","nonce":232111965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2296434777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2296434777},{"type":"v1","addr":"172.21.15.97:6837","nonce":2296434777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:09.132267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7716":{"gid":7716,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6833/322807700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":322807700},{"type":"v1","addr":"172.21.15.135:6833","nonce":322807700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7716,"qdb_cluster":[7716]},"id":8}]} 2024-03-06T07:34:16.109 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 74 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.109+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd40c06b4d0 msgr2=0x7fd40c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.109+0000 7fd436800640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd40c06b4d0 0x7fd40c06d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd420036000 tx=0x7fd420008030 comp rx=0 tx=0).stop 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 msgr2=0x7fd430146380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 0x7fd430146380 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fd41800c400 tx=0x7fd41800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 shutdown_connections 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd40c06b4d0 0x7fd40c06d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd43015acb0 0x7fd430153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4300b4680 0x7fd430146380 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 --2- 172.21.15.97:0/1777651599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd43006ec00 0x7fd4300c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 >> 172.21.15.97:0/1777651599 conn(0x7fd4300b1730 msgr2=0x7fd4300ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 shutdown_connections 2024-03-06T07:34:16.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.110+0000 7fd436800640 1 -- 172.21.15.97:0/1777651599 wait complete. 2024-03-06T07:34:16.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:34:16.387 INFO:teuthology.orchestra.run.smithi135.stdout:{"message":"","return_code":0} 2024-03-06T07:34:16.388 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:16.388 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:34:16.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.493+0000 7fbbab067f40 1 Processor -- start 2024-03-06T07:34:16.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.493+0000 7fbbab067f40 1 -- start start 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbbab067f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbbab067f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 0x55f7932f6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbbab067f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933cda30 0x55f79345ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbbab067f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f79329bfd0 con 0x55f7933be4f0 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbbab067f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f79329ff30 con 0x55f7933cda30 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbbab067f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f7932f4420 con 0x55f7933c6d40 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.494+0000 7fbba7800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 0x55f7932f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 0x55f7932f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45774/0 (socket says 172.21.15.97:45774) 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 -- 172.21.15.97:0/3485304596 learned_addr learned my addr 172.21.15.97:0/3485304596 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba8c00640 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933cda30 0x55f79345ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba8200640 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 -- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933c6d40 msgr2=0x55f7932f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 -- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933cda30 msgr2=0x55f79345ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933cda30 0x55f79345ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 -- 172.21.15.97:0/3485304596 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7933c5560 con 0x55f7933be4f0 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba7800640 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 0x55f7932f6790 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009730 tx=0x7fbb94009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33507eb1061cfdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba6400640 1 -- 172.21.15.97:0/3485304596 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94019070 con 0x55f7933be4f0 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba6400640 1 -- 172.21.15.97:0/3485304596 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb94007800 con 0x55f7933be4f0 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.495+0000 7fbba6400640 1 -- 172.21.15.97:0/3485304596 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94007bf0 con 0x55f7933be4f0 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 -- 172.21.15.97:0/3485304596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 msgr2=0x55f7932f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 0x55f7932f6790 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009730 tx=0x7fbb94009820 comp rx=0 tx=0).stop 2024-03-06T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 -- 172.21.15.97:0/3485304596 shutdown_connections 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933cda30 0x55f79345ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933be4f0 0x55f7932f6790 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 --2- 172.21.15.97:0/3485304596 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 -- 172.21.15.97:0/3485304596 >> 172.21.15.97:0/3485304596 conn(0x55f793322bf0 msgr2=0x55f7932f5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 -- 172.21.15.97:0/3485304596 shutdown_connections 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.496+0000 7fbbab067f40 1 -- 172.21.15.97:0/3485304596 wait complete. 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.497+0000 7fbbab067f40 1 Processor -- start 2024-03-06T07:34:16.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.497+0000 7fbbab067f40 1 -- start start 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbbab067f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933be4f0 0x55f7933ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbbab067f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbbab067f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933cda30 0x55f7932f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbbab067f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7933d08c0 con 0x55f7933cda30 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbbab067f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7932f5af0 con 0x55f7933c6d40 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbbab067f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f7933d0b40 con 0x55f7933be4f0 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933cda30 0x55f7932f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933cda30 0x55f7932f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45786/0 (socket says 172.21.15.97:45786) 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbba8c00640 1 -- 172.21.15.97:0/2546891620 learned_addr learned my addr 172.21.15.97:0/2546891620 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbba8200640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933be4f0 0x55f7933ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.498+0000 7fbba7800640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba8c00640 1 -- 172.21.15.97:0/2546891620 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933be4f0 msgr2=0x55f7933ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba8c00640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f7933be4f0 0x55f7933ba240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba8c00640 1 -- 172.21.15.97:0/2546891620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933c6d40 msgr2=0x55f7932f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba8c00640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7933c6d40 0x55f7932f6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba8c00640 1 -- 172.21.15.97:0/2546891620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f793465290 con 0x55f7933cda30 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba8c00640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933cda30 0x55f7932f6790 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x55f7932f89e0 tx=0x7fbba000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba5000640 1 -- 172.21.15.97:0/2546891620 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0017070 con 0x55f7933cda30 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba5000640 1 -- 172.21.15.97:0/2546891620 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba000ce00 con 0x55f7933cda30 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f793465570 con 0x55f7933cda30 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbba5000640 1 -- 172.21.15.97:0/2546891620 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0004670 con 0x55f7933cda30 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.499+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f793465b30 con 0x55f7933cda30 2024-03-06T07:34:16.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.501+0000 7fbba5000640 1 -- 172.21.15.97:0/2546891620 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbba001a070 con 0x55f7933cda30 2024-03-06T07:34:16.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.502+0000 7fbba5000640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb88069eb0 0x7fbb8806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.502+0000 7fbba5000640 1 -- 172.21.15.97:0/2546891620 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbba0087eb0 con 0x55f7933cda30 2024-03-06T07:34:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.502+0000 7fbba8200640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb88069eb0 0x7fbb8806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.503+0000 7fbbab067f40 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x55f793461bb0 0x55f793461fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.503+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) v9 -- 0x55f7933bff60 con 0x55f793461bb0 2024-03-06T07:34:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.503+0000 7fbba7800640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x55f793461bb0 0x55f793461fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.503+0000 7fbba8200640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb88069eb0 0x7fbb8806c370 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbb98054000 tx=0x7fbb9804e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.503+0000 7fbba7800640 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x55f793461bb0 0x55f793461fc0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbba7800640 1 -- 172.21.15.97:0/2546891620 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fbb9401c040 con 0x55f793461bb0 2024-03-06T07:34:16.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x55f793461bb0 msgr2=0x55f793461fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x55f793461bb0 0x55f793461fc0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb88069eb0 msgr2=0x7fbb8806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbb88069eb0 0x7fbb8806c370 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbb98054000 tx=0x7fbb9804e000 comp rx=0 tx=0).stop 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933cda30 msgr2=0x55f7932f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 --2- 172.21.15.97:0/2546891620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f7933cda30 0x55f7932f6790 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x55f7932f89e0 tx=0x7fbba000c520 comp rx=0 tx=0).stop 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbba8c00640 1 -- 172.21.15.97:0/2546891620 reap_dead start 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 shutdown_connections 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 >> 172.21.15.97:0/2546891620 conn(0x55f793322bf0 msgr2=0x55f7933cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 shutdown_connections 2024-03-06T07:34:16.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.504+0000 7fbbab067f40 1 -- 172.21.15.97:0/2546891620 wait complete. 2024-03-06T07:34:16.511 INFO:teuthology.orchestra.run.smithi097.stdout:Wfile_adir_1 2024-03-06T07:34:16.511 DEBUG:teuthology.orchestra.run.smithi097:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:34:16.664 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 1099511627777, 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "ancestors": [ 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1099511627776, 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "file_a", 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 8 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1, 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "dir_1", 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 15 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 17, 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi097.stdout: "old_pools": [] 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:16.676 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:34:16.676 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:16.729 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:34:16.729 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:16.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:16.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.992+0000 7f2c38400640 1 Processor -- start 2024-03-06T07:34:16.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.992+0000 7f2c38400640 1 -- start start 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c38400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2c30144c00 0x7f2c3006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c38400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3006ee70 con 0x7f2c3015fe60 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c38400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3006cef0 con 0x7f2c3015f9d0 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c38400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3006d1d0 con 0x7f2c30144c00 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c36400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2c3015fe60 0x7f2c300ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54704/0 (socket says 172.21.15.97:54704) 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c36400640 1 -- 172.21.15.97:0/3208392073 learned_addr learned my addr 172.21.15.97:0/3208392073 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:16.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.994+0000 7f2c35a00640 1 --2- 172.21.15.97:0/3208392073 >> v2:172.21.15.135:3300/0 conn(0x7f2c30144c00 0x7f2c3006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.995+0000 7f2c2fe00640 1 -- 172.21.15.97:0/3208392073 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832500134 0 0) 0x7f2c28001490 con 0x7f2c3015fe60 2024-03-06T07:34:16.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.995+0000 7f2c2fe00640 1 -- 172.21.15.97:0/3208392073 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c18003520 con 0x7f2c3015fe60 2024-03-06T07:34:16.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.995+0000 7f2c35a00640 1 -- 172.21.15.97:0/3208392073 >> v1:172.21.15.135:6789/0 conn(0x7f2c3015f9d0 legacy=0x7f2c300ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.995+0000 7f2c35a00640 1 -- 172.21.15.97:0/3208392073 >> v1:172.21.15.97:6789/0 conn(0x7f2c3015fe60 legacy=0x7f2c300ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.995+0000 7f2c2fe00640 1 -- 172.21.15.97:0/3208392073 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234294297 0 0) 0x7f2c18003520 con 0x7f2c3015fe60 2024-03-06T07:34:16.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.995+0000 7f2c35a00640 1 -- 172.21.15.97:0/3208392073 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3006d4b0 con 0x7f2c30144c00 2024-03-06T07:34:16.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.996+0000 7f2c35a00640 1 --2- 172.21.15.97:0/3208392073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c30144c00 0x7f2c3006e930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2c240097a0 tx=0x7f2c2402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=463ec725ee0f51a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c2fe00640 1 -- 172.21.15.97:0/3208392073 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c24038070 con 0x7f2c30144c00 2024-03-06T07:34:16.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c2fe00640 1 -- 172.21.15.97:0/3208392073 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c24031500 con 0x7f2c30144c00 2024-03-06T07:34:16.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c2fe00640 1 -- 172.21.15.97:0/3208392073 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28001bc0 con 0x7f2c30144c00 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 -- 172.21.15.97:0/3208392073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c30144c00 msgr2=0x7f2c3006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 --2- 172.21.15.97:0/3208392073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c30144c00 0x7f2c3006e930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2c240097a0 tx=0x7f2c2402b820 comp rx=0 tx=0).stop 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 -- 172.21.15.97:0/3208392073 shutdown_connections 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 --2- 172.21.15.97:0/3208392073 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c30144c00 0x7f2c3006e930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 -- 172.21.15.97:0/3208392073 >> 172.21.15.97:0/3208392073 conn(0x7f2c300b1730 msgr2=0x7f2c300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 -- 172.21.15.97:0/3208392073 shutdown_connections 2024-03-06T07:34:17.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.997+0000 7f2c38400640 1 -- 172.21.15.97:0/3208392073 wait complete. 2024-03-06T07:34:17.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.999+0000 7f2c38400640 1 Processor -- start 2024-03-06T07:34:17.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:16.999+0000 7f2c38400640 1 -- start start 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c38400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c30144c00 0x7f2c3013c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c38400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 0x7f2c3013ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c38400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c3015fe60 0x7f2c3013da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c38400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c30160730 con 0x7f2c3015f9d0 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c38400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c30160270 con 0x7f2c3015fe60 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c38400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c300ceec0 con 0x7f2c30144c00 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c35000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 0x7f2c3013ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c35000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 0x7f2c3013ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45792/0 (socket says 172.21.15.97:45792) 2024-03-06T07:34:17.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.000+0000 7f2c35000640 1 -- 172.21.15.97:0/2200600106 learned_addr learned my addr 172.21.15.97:0/2200600106 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35a00640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c30144c00 0x7f2c3013c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c36400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c3015fe60 0x7f2c3013da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35000640 1 -- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c30144c00 msgr2=0x7f2c3013c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35000640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c30144c00 0x7f2c3013c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35000640 1 -- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c3015fe60 msgr2=0x7f2c3013da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35000640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c3015fe60 0x7f2c3013da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35000640 1 -- 172.21.15.97:0/2200600106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3013b8a0 con 0x7f2c3015f9d0 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c36400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c3015fe60 0x7f2c3013da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c35000640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 0x7f2c3013ade0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c450 tx=0x7f2c1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c0052d0 con 0x7f2c3015f9d0 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c005470 con 0x7f2c3015f9d0 2024-03-06T07:34:17.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.001+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c012870 con 0x7f2c3015f9d0 2024-03-06T07:34:17.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.002+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3013bb80 con 0x7f2c3015f9d0 2024-03-06T07:34:17.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.002+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3015da90 con 0x7f2c3015f9d0 2024-03-06T07:34:17.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.003+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c30067830 con 0x7f2c3015f9d0 2024-03-06T07:34:17.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.004+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c1c012a10 con 0x7f2c3015f9d0 2024-03-06T07:34:17.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.005+0000 7f2c2d600640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.005+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c1c089eb0 con 0x7f2c3015f9d0 2024-03-06T07:34:17.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.005+0000 7f2c35a00640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.006+0000 7f2c35a00640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1006b4d0 0x7f2c1006d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2c24035000 tx=0x7f2c24008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.010+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c30067830 con 0x7f2c3015f9d0 2024-03-06T07:34:17.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.206+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c3004f330 con 0x7f2c3015f9d0 2024-03-06T07:34:17.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.208+0000 7f2c2d600640 1 -- 172.21.15.97:0/2200600106 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f2c1c058c90 con 0x7f2c3015f9d0 2024-03-06T07:34:17.211 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:17.211 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":112,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:07.398719+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1006b4d0 msgr2=0x7f2c1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1006b4d0 0x7f2c1006d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2c24035000 tx=0x7f2c24008030 comp rx=0 tx=0).stop 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 msgr2=0x7f2c3013ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 0x7f2c3013ade0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c450 tx=0x7f2c1c00c910 comp rx=0 tx=0).stop 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 shutdown_connections 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c3015fe60 0x7f2c3013da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c3015f9d0 0x7f2c3013ade0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 --2- 172.21.15.97:0/2200600106 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c30144c00 0x7f2c3013c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 >> 172.21.15.97:0/2200600106 conn(0x7f2c300b1730 msgr2=0x7f2c301452e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 shutdown_connections 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.211+0000 7f2c38400640 1 -- 172.21.15.97:0/2200600106 wait complete. 2024-03-06T07:34:17.232 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:03.618857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:04.895815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:17.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:17.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.453+0000 7fdda9c00640 1 Processor -- start 2024-03-06T07:34:17.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.453+0000 7fdda9c00640 1 -- start start 2024-03-06T07:34:17.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.454+0000 7fdda9c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda415be10 0x7fdda40ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.454+0000 7fdda9c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdda40aef20 0x7fdda40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.454+0000 7fdda9c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda415c1f0 con 0x7fdda406d330 2024-03-06T07:34:17.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.454+0000 7fdda9c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda40cd200 con 0x7fdda40aef20 2024-03-06T07:34:17.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.454+0000 7fdda9c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda406cac0 con 0x7fdda415be10 2024-03-06T07:34:17.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.454+0000 7fdd9b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdda40aef20 0x7fdda40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdd9b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdda40aef20 0x7fdda40ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45798/0 (socket says 172.21.15.97:45798) 2024-03-06T07:34:17.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdd9b400640 1 -- 172.21.15.97:0/1103858332 learned_addr learned my addr 172.21.15.97:0/1103858332 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:17.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdda3e00640 1 --2- 172.21.15.97:0/1103858332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda415be10 0x7fdda40ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdd9b400640 1 -- 172.21.15.97:0/1103858332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda415be10 msgr2=0x7fdda40ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdd9b400640 1 --2- 172.21.15.97:0/1103858332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda415be10 0x7fdda40ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdd9b400640 1 -- 172.21.15.97:0/1103858332 >> v1:172.21.15.135:6789/0 conn(0x7fdda406d330 legacy=0x7fdda4071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.455+0000 7fdd9b400640 1 -- 172.21.15.97:0/1103858332 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda406cda0 con 0x7fdda40aef20 2024-03-06T07:34:17.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.456+0000 7fdd9b400640 1 --2- 172.21.15.97:0/1103858332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda40aef20 0x7fdda40ce2b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009870 tx=0x7fdd8c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c497682e4e1fb98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.456+0000 7fdda3400640 1 -- 172.21.15.97:0/1103858332 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c035020 con 0x7fdda40aef20 2024-03-06T07:34:17.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.456+0000 7fdda3400640 1 -- 172.21.15.97:0/1103858332 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c031640 con 0x7fdda40aef20 2024-03-06T07:34:17.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.456+0000 7fdda3400640 1 -- 172.21.15.97:0/1103858332 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c02d040 con 0x7fdda40aef20 2024-03-06T07:34:17.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 -- 172.21.15.97:0/1103858332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda40aef20 msgr2=0x7fdda40ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 --2- 172.21.15.97:0/1103858332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda40aef20 0x7fdda40ce2b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009870 tx=0x7fdd8c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 -- 172.21.15.97:0/1103858332 shutdown_connections 2024-03-06T07:34:17.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 --2- 172.21.15.97:0/1103858332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda40aef20 0x7fdda40ce2b0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 --2- 172.21.15.97:0/1103858332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda415be10 0x7fdda40ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 -- 172.21.15.97:0/1103858332 >> 172.21.15.97:0/1103858332 conn(0x7fdda406afc0 msgr2=0x7fdda406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 -- 172.21.15.97:0/1103858332 shutdown_connections 2024-03-06T07:34:17.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.457+0000 7fdda9c00640 1 -- 172.21.15.97:0/1103858332 wait complete. 2024-03-06T07:34:17.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.459+0000 7fdda9c00640 1 Processor -- start 2024-03-06T07:34:17.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.459+0000 7fdda9c00640 1 -- start start 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 0x7fdda40b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda40aef20 0x7fdda414de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdda415be10 0x7fdda40cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40cda90 con 0x7fdda406d330 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda406d080 con 0x7fdda415be10 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda406d7c0 con 0x7fdda40aef20 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 0x7fdda40b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 0x7fdda40b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45806/0 (socket says 172.21.15.97:45806) 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda3e00640 1 -- 172.21.15.97:0/3945875981 learned_addr learned my addr 172.21.15.97:0/3945875981 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdda9200640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdda415be10 0x7fdda40cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.460+0000 7fdd9b400640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda40aef20 0x7fdda414de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda3e00640 1 -- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda40aef20 msgr2=0x7fdda414de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda3e00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda40aef20 0x7fdda414de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda3e00640 1 -- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdda415be10 msgr2=0x7fdda40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda3e00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdda415be10 0x7fdda40cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda3e00640 1 -- 172.21.15.97:0/3945875981 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda414e3c0 con 0x7fdda406d330 2024-03-06T07:34:17.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda3e00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 0x7fdda40b4170 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400ccf0 tx=0x7fdd9400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94003990 con 0x7fdda406d330 2024-03-06T07:34:17.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd94003b30 con 0x7fdda406d330 2024-03-06T07:34:17.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94004020 con 0x7fdda406d330 2024-03-06T07:34:17.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda414e690 con 0x7fdda406d330 2024-03-06T07:34:17.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.461+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda414ec50 con 0x7fdda406d330 2024-03-06T07:34:17.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.462+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda404f330 con 0x7fdda406d330 2024-03-06T07:34:17.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.463+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd940041c0 con 0x7fdda406d330 2024-03-06T07:34:17.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.464+0000 7fdda0c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd8006b4d0 0x7fdd8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.464+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd9408a830 con 0x7fdda406d330 2024-03-06T07:34:17.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.464+0000 7fdd9b400640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd8006b4d0 0x7fdd8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.465+0000 7fdd9b400640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd8006b4d0 0x7fdd8006d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c036000 tx=0x7fdd8c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.469+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd94014030 con 0x7fdda406d330 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.664+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdda40cff50 con 0x7fdda406d330 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.664+0000 7fdda0c00640 1 -- 172.21.15.97:0/3945875981 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd94059610 con 0x7fdda406d330 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:34:17.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd8006b4d0 msgr2=0x7fdd8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd8006b4d0 0x7fdd8006d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c036000 tx=0x7fdd8c03c040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 msgr2=0x7fdda40b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 0x7fdda40b4170 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400ccf0 tx=0x7fdd9400a5c0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 shutdown_connections 2024-03-06T07:34:17.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdd8006b4d0 0x7fdd8006d990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdda415be10 0x7fdda40cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdda40aef20 0x7fdda414de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 --2- 172.21.15.97:0/3945875981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdda406d330 0x7fdda40b4170 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.667+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 >> 172.21.15.97:0/3945875981 conn(0x7fdda406afc0 msgr2=0x7fdda40afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.668+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 shutdown_connections 2024-03-06T07:34:17.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.668+0000 7fdda9c00640 1 -- 172.21.15.97:0/3945875981 wait complete. 2024-03-06T07:34:17.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:17.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.893+0000 7efe85c00640 1 Processor -- start 2024-03-06T07:34:17.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.894+0000 7efe85c00640 1 -- start start 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.895+0000 7efe85c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efe800b2d80 0x7efe800cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.895+0000 7efe85c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efe800cf6f0 0x7efe800b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.895+0000 7efe85c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe800b32f0 con 0x7efe801618e0 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.895+0000 7efe85c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800d0d20 con 0x7efe800cf6f0 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.895+0000 7efe85c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800d0f70 con 0x7efe800b2d80 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.895+0000 7efe7f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efe800cf6f0 0x7efe800b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe7f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efe800cf6f0 0x7efe800b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45810/0 (socket says 172.21.15.97:45810) 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe7f400640 1 -- 172.21.15.97:0/4053228629 learned_addr learned my addr 172.21.15.97:0/4053228629 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe76000640 1 --2- 172.21.15.97:0/4053228629 >> v2:172.21.15.135:3300/0 conn(0x7efe800b2d80 0x7efe800cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe7f400640 1 -- 172.21.15.97:0/4053228629 >> v2:172.21.15.135:3300/0 conn(0x7efe800b2d80 msgr2=0x7efe800cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe7f400640 1 --2- 172.21.15.97:0/4053228629 >> v2:172.21.15.135:3300/0 conn(0x7efe800b2d80 0x7efe800cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe7f400640 1 -- 172.21.15.97:0/4053228629 >> v1:172.21.15.135:6789/0 conn(0x7efe801618e0 legacy=0x7efe800ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.896+0000 7efe7f400640 1 -- 172.21.15.97:0/4053228629 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800d1250 con 0x7efe800cf6f0 2024-03-06T07:34:17.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.897+0000 7efe7f400640 1 --2- 172.21.15.97:0/4053228629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe800cf6f0 0x7efe800b0c40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7efe8006b8d0 tx=0x7efe7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fe985f6b93a5ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.897+0000 7efe7e000640 1 -- 172.21.15.97:0/4053228629 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78035020 con 0x7efe800cf6f0 2024-03-06T07:34:17.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.897+0000 7efe7e000640 1 -- 172.21.15.97:0/4053228629 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78031870 con 0x7efe800cf6f0 2024-03-06T07:34:17.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.897+0000 7efe7e000640 1 -- 172.21.15.97:0/4053228629 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78031c40 con 0x7efe800cf6f0 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 -- 172.21.15.97:0/4053228629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe800cf6f0 msgr2=0x7efe800b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 --2- 172.21.15.97:0/4053228629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe800cf6f0 0x7efe800b0c40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7efe8006b8d0 tx=0x7efe7802b910 comp rx=0 tx=0).stop 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 -- 172.21.15.97:0/4053228629 shutdown_connections 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 --2- 172.21.15.97:0/4053228629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe800cf6f0 0x7efe800b0c40 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 --2- 172.21.15.97:0/4053228629 >> v2:172.21.15.135:3300/0 conn(0x7efe800b2d80 0x7efe800cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 -- 172.21.15.97:0/4053228629 >> 172.21.15.97:0/4053228629 conn(0x7efe800d3e90 msgr2=0x7efe800d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 -- 172.21.15.97:0/4053228629 shutdown_connections 2024-03-06T07:34:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.898+0000 7efe85c00640 1 -- 172.21.15.97:0/4053228629 wait complete. 2024-03-06T07:34:17.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.900+0000 7efe85c00640 1 Processor -- start 2024-03-06T07:34:17.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.900+0000 7efe85c00640 1 -- start start 2024-03-06T07:34:17.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe85c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe800b2d80 0x7efe800b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe85c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe800cf6f0 0x7efe8006c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe85c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 0x7efe8006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe85c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800d2da0 con 0x7efe801618e0 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe85c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe80161cc0 con 0x7efe800b2d80 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe85c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800b1180 con 0x7efe800cf6f0 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe7f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 0x7efe8006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe7f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 0x7efe8006cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45816/0 (socket says 172.21.15.97:45816) 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe7f400640 1 -- 172.21.15.97:0/1742893369 learned_addr learned my addr 172.21.15.97:0/1742893369 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.901+0000 7efe7ea00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe800b2d80 0x7efe800b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe7f400640 1 -- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe800cf6f0 msgr2=0x7efe8006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe7f400640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe800cf6f0 0x7efe8006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe7f400640 1 -- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe800b2d80 msgr2=0x7efe800b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe7f400640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe800b2d80 0x7efe800b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe7f400640 1 -- 172.21.15.97:0/1742893369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800b6fe0 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe7f400640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 0x7efe8006cdc0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7efe7802b950 tx=0x7efe780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78040070 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78030d00 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78035040 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe800b72c0 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.902+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe800b7880 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.903+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8004f330 con 0x7efe801618e0 2024-03-06T07:34:17.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.904+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe78038020 con 0x7efe801618e0 2024-03-06T07:34:17.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.905+0000 7efe77400640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5c06b500 0x7efe5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.905+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe780ac0d0 con 0x7efe801618e0 2024-03-06T07:34:17.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.905+0000 7efe7ea00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5c06b500 0x7efe5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.906+0000 7efe7ea00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5c06b500 0x7efe5c06d9c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efe70009f20 tx=0x7efe700094d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:17.910+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe7807aeb0 con 0x7efe801618e0 2024-03-06T07:34:18.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.104+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe8015f7b0 con 0x7efe801618e0 2024-03-06T07:34:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.105+0000 7efe77400640 1 -- 172.21.15.97:0/1742893369 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe7807a7b0 con 0x7efe801618e0 2024-03-06T07:34:18.107 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:34:18.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5c06b500 msgr2=0x7efe5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5c06b500 0x7efe5c06d9c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efe70009f20 tx=0x7efe700094d0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 msgr2=0x7efe8006cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 0x7efe8006cdc0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7efe7802b950 tx=0x7efe780046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 shutdown_connections 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5c06b500 0x7efe5c06d9c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe801618e0 0x7efe8006cdc0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe800cf6f0 0x7efe8006c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 --2- 172.21.15.97:0/1742893369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe800b2d80 0x7efe800b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 >> 172.21.15.97:0/1742893369 conn(0x7efe800d3e90 msgr2=0x7efe8015e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 shutdown_connections 2024-03-06T07:34:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.108+0000 7efe85c00640 1 -- 172.21.15.97:0/1742893369 wait complete. 2024-03-06T07:34:18.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.336+0000 7f1f27400640 1 Processor -- start 2024-03-06T07:34:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.336+0000 7f1f27400640 1 -- start start 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.338+0000 7f1f27400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1f2006ec20 0x7f1f200cd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.338+0000 7f1f27400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f20144ab0 con 0x7f1f201446d0 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.338+0000 7f1f27400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200cb6a0 con 0x7f1f200cda50 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.338+0000 7f1f27400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f200cdec0 con 0x7f1f2006ec20 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.338+0000 7f1f24a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f1f201446d0 0x7f1f200cf5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54714/0 (socket says 172.21.15.97:54714) 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.338+0000 7f1f24a00640 1 -- 172.21.15.97:0/2683464847 learned_addr learned my addr 172.21.15.97:0/2683464847 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816555780 0 0) 0x7f1f20144ab0 con 0x7f1f201446d0 2024-03-06T07:34:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f04005730 con 0x7f1f201446d0 2024-03-06T07:34:18.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1fe00640 1 --2- 172.21.15.97:0/2683464847 >> v2:172.21.15.135:3300/0 conn(0x7f1f2006ec20 0x7f1f200cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 151393647 0 0) 0x7f1f04005730 con 0x7f1f201446d0 2024-03-06T07:34:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 >> v2:172.21.15.135:3300/0 conn(0x7f1f2006ec20 msgr2=0x7f1f200cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 --2- 172.21.15.97:0/2683464847 >> v2:172.21.15.135:3300/0 conn(0x7f1f2006ec20 0x7f1f200cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 >> v1:172.21.15.135:6789/0 conn(0x7f1f200cda50 legacy=0x7f1f200cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2006f090 con 0x7f1f201446d0 2024-03-06T07:34:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1f100019f0 con 0x7f1f201446d0 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 >> v1:172.21.15.97:6789/0 conn(0x7f1f201446d0 legacy=0x7f1f200cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f04005c10 0x7f1f04006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f04006a10 0x7f1f04006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040073d0 0x7f1f0400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200cdec0 con 0x7f1f040073d0 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0400e2d0 con 0x7f1f04006a10 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.339+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0400e4d0 con 0x7f1f04005c10 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f25400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040073d0 0x7f1f0400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f1fe00640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f04006a10 0x7f1f04006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f24a00640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f04005c10 0x7f1f04006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f25400640 1 -- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f04005c10 msgr2=0x7f1f04006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f25400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f04005c10 0x7f1f04006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f25400640 1 -- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f04006a10 msgr2=0x7f1f04006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f25400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f04006a10 0x7f1f04006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.340+0000 7f1f25400640 1 -- 172.21.15.97:0/2683464847 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04005730 con 0x7f1f040073d0 2024-03-06T07:34:18.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.341+0000 7f1f25400640 1 -- 172.21.15.97:0/2683464847 reap_dead start 2024-03-06T07:34:18.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.341+0000 7f1f25400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040073d0 0x7f1f0400dbd0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1f18001750 tx=0x7f1f04007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cef4f6ad2d11543 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18001370 con 0x7f1f040073d0 2024-03-06T07:34:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18005310 con 0x7f1f040073d0 2024-03-06T07:34:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f1f400640 1 -- 172.21.15.97:0/2683464847 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18003830 con 0x7f1f040073d0 2024-03-06T07:34:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 -- 172.21.15.97:0/2683464847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040073d0 msgr2=0x7f1f0400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040073d0 0x7f1f0400dbd0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1f18001750 tx=0x7f1f04007a40 comp rx=0 tx=0).stop 2024-03-06T07:34:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 -- 172.21.15.97:0/2683464847 shutdown_connections 2024-03-06T07:34:18.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 --2- 172.21.15.97:0/2683464847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f040073d0 0x7f1f0400dbd0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 -- 172.21.15.97:0/2683464847 >> 172.21.15.97:0/2683464847 conn(0x7f1f200b27d0 msgr2=0x7f1f200b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 -- 172.21.15.97:0/2683464847 shutdown_connections 2024-03-06T07:34:18.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.342+0000 7f1f27400640 1 -- 172.21.15.97:0/2683464847 wait complete. 2024-03-06T07:34:18.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.344+0000 7f1f27400640 1 Processor -- start 2024-03-06T07:34:18.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.344+0000 7f1f27400640 1 -- start start 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f27400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040073d0 0x7f1f201e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f27400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f201f8450 0x7f1f200ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f27400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 0x7f1f201e6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f27400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2006f600 con 0x7f1f200ac280 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f27400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200cb9b0 con 0x7f1f040073d0 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f27400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2006f8e0 con 0x7f1f201f8450 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.345+0000 7f1f25400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 0x7f1f201e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 0x7f1f201e6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45834/0 (socket says 172.21.15.97:45834) 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 -- 172.21.15.97:0/3876787602 learned_addr learned my addr 172.21.15.97:0/3876787602 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f24a00640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040073d0 0x7f1f201e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f1fe00640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f201f8450 0x7f1f200ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 -- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f201f8450 msgr2=0x7f1f200ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f201f8450 0x7f1f200ad580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 -- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040073d0 msgr2=0x7f1f201e9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040073d0 0x7f1f201e9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 -- 172.21.15.97:0/3876787602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200adef0 con 0x7f1f200ac280 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f25400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 0x7f1f201e6660 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f1f18001870 tx=0x7f1f18001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.346+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18001370 con 0x7f1f200ac280 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.347+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f180050d0 con 0x7f1f200ac280 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.347+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18007a30 con 0x7f1f200ac280 2024-03-06T07:34:18.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.347+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f200ab7c0 con 0x7f1f200ac280 2024-03-06T07:34:18.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.347+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f200abd80 con 0x7f1f200ac280 2024-03-06T07:34:18.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.348+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200ae290 con 0x7f1f200ac280 2024-03-06T07:34:18.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.349+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f18007440 con 0x7f1f200ac280 2024-03-06T07:34:18.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.350+0000 7f1f1cc00640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1efc06b4d0 0x7f1efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.350+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f180aaa80 con 0x7f1f200ac280 2024-03-06T07:34:18.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.350+0000 7f1f24a00640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1efc06b4d0 0x7f1efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.351+0000 7f1f24a00640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1efc06b4d0 0x7f1efc06d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1f100013d0 tx=0x7f1f10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.355+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f200cb6a0 con 0x7f1f200ac280 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.550+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1f2004f330 con 0x7f1f200ac280 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.551+0000 7f1f1cc00640 1 -- 172.21.15.97:0/3876787602 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1f18079910 con 0x7f1f200ac280 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:34:18.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1efc06b4d0 msgr2=0x7f1efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1efc06b4d0 0x7f1efc06d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1f100013d0 tx=0x7f1f10009040 comp rx=0 tx=0).stop 2024-03-06T07:34:18.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 msgr2=0x7f1f201e6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 0x7f1f201e6660 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f1f18001870 tx=0x7f1f18001120 comp rx=0 tx=0).stop 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 shutdown_connections 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1efc06b4d0 0x7f1efc06d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f200ac280 0x7f1f201e6660 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1f201f8450 0x7f1f200ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 --2- 172.21.15.97:0/3876787602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f040073d0 0x7f1f201e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 >> 172.21.15.97:0/3876787602 conn(0x7f1f200b27d0 msgr2=0x7f1f200ca370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 shutdown_connections 2024-03-06T07:34:18.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.554+0000 7f1f27400640 1 -- 172.21.15.97:0/3876787602 wait complete. 2024-03-06T07:34:18.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:34:18.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.785+0000 7f2e9ec00640 1 Processor -- start 2024-03-06T07:34:18.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.785+0000 7f2e9ec00640 1 -- start start 2024-03-06T07:34:18.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e9ec00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2e980cd5b0 0x7f2e98145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e9815cc70 0x7f2e98162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e9ec00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e98166ec0 con 0x7f2e980cd990 2024-03-06T07:34:18.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e9ec00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9806cbe0 con 0x7f2e980cd5b0 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9806cec0 con 0x7f2e9815cc70 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e97e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2e980cd5b0 0x7f2e98145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e97e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2e980cd5b0 0x7f2e98145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45844/0 (socket says 172.21.15.97:45844) 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.787+0000 7f2e97e00640 1 -- 172.21.15.97:0/516744879 learned_addr learned my addr 172.21.15.97:0/516744879 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.788+0000 7f2e9cc00640 1 --2- 172.21.15.97:0/516744879 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e9815cc70 0x7f2e98162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.788+0000 7f2e97e00640 1 -- 172.21.15.97:0/516744879 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e9815cc70 msgr2=0x7f2e98162fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.788+0000 7f2e97e00640 1 --2- 172.21.15.97:0/516744879 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e9815cc70 0x7f2e98162fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.788+0000 7f2e97e00640 1 -- 172.21.15.97:0/516744879 >> v1:172.21.15.135:6789/0 conn(0x7f2e980cd990 legacy=0x7f2e9815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.788+0000 7f2e97e00640 1 -- 172.21.15.97:0/516744879 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9806d1a0 con 0x7f2e980cd5b0 2024-03-06T07:34:18.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.789+0000 7f2e97e00640 1 --2- 172.21.15.97:0/516744879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd5b0 0x7f2e98145b70 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009870 tx=0x7f2e8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2d180f565bb445 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e96a00640 1 -- 172.21.15.97:0/516744879 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88035020 con 0x7f2e980cd5b0 2024-03-06T07:34:18.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e96a00640 1 -- 172.21.15.97:0/516744879 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e88031640 con 0x7f2e980cd5b0 2024-03-06T07:34:18.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e96a00640 1 -- 172.21.15.97:0/516744879 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8802d040 con 0x7f2e980cd5b0 2024-03-06T07:34:18.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 -- 172.21.15.97:0/516744879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd5b0 msgr2=0x7f2e98145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/516744879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd5b0 0x7f2e98145b70 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009870 tx=0x7f2e8802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 -- 172.21.15.97:0/516744879 shutdown_connections 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/516744879 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e9815cc70 0x7f2e98162fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/516744879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd5b0 0x7f2e98145b70 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 -- 172.21.15.97:0/516744879 >> 172.21.15.97:0/516744879 conn(0x7f2e980cc4f0 msgr2=0x7f2e980cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 -- 172.21.15.97:0/516744879 shutdown_connections 2024-03-06T07:34:18.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.790+0000 7f2e9ec00640 1 -- 172.21.15.97:0/516744879 wait complete. 2024-03-06T07:34:18.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.792+0000 7f2e9ec00640 1 Processor -- start 2024-03-06T07:34:18.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.792+0000 7f2e9ec00640 1 -- start start 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e980cd5b0 0x7f2e98159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 0x7f2e98141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e9815cc70 0x7f2e981422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e981634f0 con 0x7f2e980cd990 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980cde40 con 0x7f2e9815cc70 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e98163670 con 0x7f2e980cd5b0 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 0x7f2e98141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 0x7f2e98141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45854/0 (socket says 172.21.15.97:45854) 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.793+0000 7f2e97400640 1 -- 172.21.15.97:0/1215864839 learned_addr learned my addr 172.21.15.97:0/1215864839 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97e00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e980cd5b0 0x7f2e98159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e9cc00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e9815cc70 0x7f2e981422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97400640 1 -- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e980cd5b0 msgr2=0x7f2e98159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97400640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e980cd5b0 0x7f2e98159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97400640 1 -- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e9815cc70 msgr2=0x7f2e981422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97400640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e9815cc70 0x7f2e981422a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97400640 1 -- 172.21.15.97:0/1215864839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e981ede60 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e9cc00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e9815cc70 0x7f2e981422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e97400640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 0x7f2e98141d60 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400c430 tx=0x7f2e8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84011790 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e84017070 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84011bb0 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.794+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e981ee140 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.795+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e981ee700 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.796+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9804f330 con 0x7f2e980cd990 2024-03-06T07:34:18.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.797+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e8401c460 con 0x7f2e980cd990 2024-03-06T07:34:18.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.798+0000 7f2e8fe00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e7806b4d0 0x7f2e7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.798+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e8408a0c0 con 0x7f2e980cd990 2024-03-06T07:34:18.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.798+0000 7f2e97e00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e7806b4d0 0x7f2e7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.799+0000 7f2e97e00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e7806b4d0 0x7f2e7806d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2e88036000 tx=0x7f2e8803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:18.802+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e84058ea0 con 0x7f2e980cd990 2024-03-06T07:34:19.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.011+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2e98162570 con 0x7f2e980cd990 2024-03-06T07:34:19.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.011+0000 7f2e8fe00640 1 -- 172.21.15.97:0/1215864839 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v112) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2e840587a0 con 0x7f2e980cd990 2024-03-06T07:34:19.014 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e7806b4d0 msgr2=0x7f2e7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e7806b4d0 0x7f2e7806d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2e88036000 tx=0x7f2e8803c040 comp rx=0 tx=0).stop 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 msgr2=0x7f2e98141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 0x7f2e98141d60 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400c430 tx=0x7f2e8400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 shutdown_connections 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e7806b4d0 0x7f2e7806d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e9815cc70 0x7f2e981422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e980cd990 0x7f2e98141d60 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 --2- 172.21.15.97:0/1215864839 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2e980cd5b0 0x7f2e98159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 >> 172.21.15.97:0/1215864839 conn(0x7f2e980cc4f0 msgr2=0x7f2e98163a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 shutdown_connections 2024-03-06T07:34:19.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.014+0000 7f2e9ec00640 1 -- 172.21.15.97:0/1215864839 wait complete. 2024-03-06T07:34:19.034 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:34:19.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:19.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:19.175 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:34:19.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:19.175 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:19.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:19.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.445+0000 7f88d5400640 1 Processor -- start 2024-03-06T07:34:19.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.445+0000 7f88d5400640 1 -- start start 2024-03-06T07:34:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88d5400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f88d0147310 0x7f88d0072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88d5400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f88d006d390 0x7f88d00ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88d5400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00b39c0 con 0x7f88d006daa0 2024-03-06T07:34:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88d5400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d006df30 con 0x7f88d0147310 2024-03-06T07:34:19.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88d5400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d00ad9e0 con 0x7f88d006d390 2024-03-06T07:34:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88ce800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f88d0147310 0x7f88d0072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88ce800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f88d0147310 0x7f88d0072340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45862/0 (socket says 172.21.15.97:45862) 2024-03-06T07:34:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.447+0000 7f88ce800640 1 -- 172.21.15.97:0/3501572482 learned_addr learned my addr 172.21.15.97:0/3501572482 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.448+0000 7f88cf200640 1 --2- 172.21.15.97:0/3501572482 >> v2:172.21.15.135:3300/0 conn(0x7f88d006d390 0x7f88d00ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.448+0000 7f88ce800640 1 -- 172.21.15.97:0/3501572482 >> v2:172.21.15.135:3300/0 conn(0x7f88d006d390 msgr2=0x7f88d00ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.448+0000 7f88ce800640 1 --2- 172.21.15.97:0/3501572482 >> v2:172.21.15.135:3300/0 conn(0x7f88d006d390 0x7f88d00ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.448+0000 7f88ce800640 1 -- 172.21.15.97:0/3501572482 >> v1:172.21.15.97:6789/0 conn(0x7f88d006daa0 legacy=0x7f88d006cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.448+0000 7f88ce800640 1 -- 172.21.15.97:0/3501572482 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00adcc0 con 0x7f88d0147310 2024-03-06T07:34:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.449+0000 7f88ce800640 1 --2- 172.21.15.97:0/3501572482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d0147310 0x7f88d0072340 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f88d006cb00 tx=0x7f88c002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8134524f1fa7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.449+0000 7f88cde00640 1 -- 172.21.15.97:0/3501572482 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c002cb10 con 0x7f88d0147310 2024-03-06T07:34:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.449+0000 7f88cde00640 1 -- 172.21.15.97:0/3501572482 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c002ccb0 con 0x7f88d0147310 2024-03-06T07:34:19.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.449+0000 7f88cde00640 1 -- 172.21.15.97:0/3501572482 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c002ce60 con 0x7f88d0147310 2024-03-06T07:34:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 -- 172.21.15.97:0/3501572482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d0147310 msgr2=0x7f88d0072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 --2- 172.21.15.97:0/3501572482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d0147310 0x7f88d0072340 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f88d006cb00 tx=0x7f88c002bf70 comp rx=0 tx=0).stop 2024-03-06T07:34:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 -- 172.21.15.97:0/3501572482 shutdown_connections 2024-03-06T07:34:19.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 --2- 172.21.15.97:0/3501572482 >> v2:172.21.15.135:3300/0 conn(0x7f88d006d390 0x7f88d00ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 --2- 172.21.15.97:0/3501572482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d0147310 0x7f88d0072340 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 -- 172.21.15.97:0/3501572482 >> 172.21.15.97:0/3501572482 conn(0x7f88d00b1be0 msgr2=0x7f88d00b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 -- 172.21.15.97:0/3501572482 shutdown_connections 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.450+0000 7f88d5400640 1 -- 172.21.15.97:0/3501572482 wait complete. 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.452+0000 7f88d5400640 1 Processor -- start 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.452+0000 7f88d5400640 1 -- start start 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88d5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 0x7f88d0154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88d5400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88d006daa0 0x7f88d013c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88d5400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88d0147310 0x7f88d0160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88d5400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d01476f0 con 0x7f88d006d390 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88d5400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00ae280 con 0x7f88d0147310 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88d5400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00aece0 con 0x7f88d006daa0 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 0x7f88d0154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 0x7f88d0154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45878/0 (socket says 172.21.15.97:45878) 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 -- 172.21.15.97:0/2446442679 learned_addr learned my addr 172.21.15.97:0/2446442679 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88c5e00640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88d006daa0 0x7f88d013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88cf200640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88d0147310 0x7f88d0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 -- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88d006daa0 msgr2=0x7f88d013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88d006daa0 0x7f88d013c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 -- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88d0147310 msgr2=0x7f88d0160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.453+0000 7f88ce800640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88d0147310 0x7f88d0160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88ce800640 1 -- 172.21.15.97:0/2446442679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d0146000 con 0x7f88d006d390 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88c5e00640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88d006daa0 0x7f88d013c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88cf200640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88d0147310 0x7f88d0160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88ce800640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 0x7f88d0154320 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009f40 tx=0x7f88c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0043020 con 0x7f88d006d390 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c003e450 con 0x7f88d006d390 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d0154860 con 0x7f88d006d390 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c002ce60 con 0x7f88d006d390 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.454+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d0143b90 con 0x7f88d006d390 2024-03-06T07:34:19.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.455+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d004f330 con 0x7f88d006d390 2024-03-06T07:34:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.456+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88c0005d10 con 0x7f88d006d390 2024-03-06T07:34:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.457+0000 7f88c7400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88ac06b4d0 0x7f88ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.457+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88c00accc0 con 0x7f88d006d390 2024-03-06T07:34:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.457+0000 7f88c5e00640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88ac06b4d0 0x7f88ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.458+0000 7f88c5e00640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88ac06b4d0 0x7f88ac06d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f88d01f7270 tx=0x7f88b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.462+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88c00b3080 con 0x7f88d006d390 2024-03-06T07:34:19.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.693+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d006d880 con 0x7f88d006d390 2024-03-06T07:34:19.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.695+0000 7f88c7400640 1 -- 172.21.15.97:0/2446442679 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f88c007baa0 con 0x7f88d006d390 2024-03-06T07:34:19.697 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:19.697 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/1292572481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":1292572481},{"type":"v1","addr":"172.21.15.135:6835","nonce":1292572481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7701,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/232111965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":232111965},{"type":"v1","addr":"172.21.15.97:6835","nonce":232111965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2296434777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2296434777},{"type":"v1","addr":"172.21.15.97:6837","nonce":2296434777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:09.132267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7716":{"gid":7716,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.135:6833/322807700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":322807700},{"type":"v1","addr":"172.21.15.135:6833","nonce":322807700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7716,"qdb_cluster":[7716]},"id":8}]} 2024-03-06T07:34:19.698 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 74 2024-03-06T07:34:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88ac06b4d0 msgr2=0x7f88ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88ac06b4d0 0x7f88ac06d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f88d01f7270 tx=0x7f88b8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 msgr2=0x7f88d0154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 0x7f88d0154320 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009f40 tx=0x7f88c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 shutdown_connections 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88ac06b4d0 0x7f88ac06d990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88d0147310 0x7f88d0160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88d006daa0 0x7f88d013c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 --2- 172.21.15.97:0/2446442679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88d006d390 0x7f88d0154320 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 >> 172.21.15.97:0/2446442679 conn(0x7f88d00b1be0 msgr2=0x7f88d0071630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 shutdown_connections 2024-03-06T07:34:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:19.698+0000 7f88d5400640 1 -- 172.21.15.97:0/2446442679 wait complete. 2024-03-06T07:34:19.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:34:19.961 INFO:teuthology.orchestra.run.smithi135.stdout:{"message":"","return_code":0} 2024-03-06T07:34:19.961 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:19.961 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:19.962 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:19.962 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:19.962 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:19.962 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:19.962 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:19.963 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:34:19.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:19.978 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:34:19.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:34:19.982 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:19.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:34:19.987 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:19.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:34:19.989 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:19.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:34:20.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.169+0000 7fe380a00640 1 Processor -- start 2024-03-06T07:34:20.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.169+0000 7f58ddc00640 1 Processor -- start 2024-03-06T07:34:20.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.170+0000 7fe380a00640 1 -- start start 2024-03-06T07:34:20.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.170+0000 7f58ddc00640 1 -- start start 2024-03-06T07:34:20.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe380a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe380a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe37c06bfd0 0x7fe37c06c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7f58ddc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f58d80cd5b0 0x7f58d8145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe380a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37c15e520 con 0x7fe37c0b3960 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe380a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37c0b3e00 con 0x7fe37c06bfd0 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe380a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0b40e0 con 0x7fe37c15e140 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7f58ddc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f58d815cc70 0x7f58d8162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7f58ddc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d8166ec0 con 0x7f58d80cd990 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7f58ddc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d806cbe0 con 0x7f58d80cd5b0 2024-03-06T07:34:20.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7f58ddc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d806cec0 con 0x7f58d815cc70 2024-03-06T07:34:20.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe379400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe37c0b3960 0x7fe37c06d2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54728/0 (socket says 172.21.15.97:54728) 2024-03-06T07:34:20.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.171+0000 7fe379400640 1 -- 172.21.15.97:0/2677441259 learned_addr learned my addr 172.21.15.97:0/2677441259 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:20.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f58d80cd5b0 0x7f58d8145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f58d80cd5b0 0x7f58d8145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45882/0 (socket says 172.21.15.97:45882) 2024-03-06T07:34:20.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 -- 172.21.15.97:0/1577162677 learned_addr learned my addr 172.21.15.97:0/1577162677 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:20.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7fe37a800640 1 --2- 172.21.15.97:0/2677441259 >> v2:172.21.15.135:3300/0 conn(0x7fe37c06bfd0 0x7fe37c06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7fe379e00640 1 --2- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7fe378a00640 1 -- 172.21.15.97:0/2677441259 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893022810 0 0) 0x7fe37c15e520 con 0x7fe37c0b3960 2024-03-06T07:34:20.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7fe378a00640 1 -- 172.21.15.97:0/2677441259 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe360003520 con 0x7fe37c0b3960 2024-03-06T07:34:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d7400640 1 --2- 172.21.15.97:0/1577162677 >> v2:172.21.15.135:3300/0 conn(0x7f58d815cc70 0x7f58d8162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 -- 172.21.15.97:0/1577162677 >> v2:172.21.15.135:3300/0 conn(0x7f58d815cc70 msgr2=0x7f58d8162fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 --2- 172.21.15.97:0/1577162677 >> v2:172.21.15.135:3300/0 conn(0x7f58d815cc70 0x7f58d8162fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 -- 172.21.15.97:0/1577162677 >> v1:172.21.15.135:6789/0 conn(0x7f58d80cd990 legacy=0x7f58d815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.172+0000 7f58d6a00640 1 -- 172.21.15.97:0/1577162677 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d806d1a0 con 0x7f58d80cd5b0 2024-03-06T07:34:20.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.173+0000 7fe37a800640 1 -- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 msgr2=0x7fe37c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.173+0000 7fe37a800640 1 --2- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c0b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.173+0000 7fe37a800640 1 -- 172.21.15.97:0/2677441259 >> v1:172.21.15.97:6789/0 conn(0x7fe37c0b3960 legacy=0x7fe37c06d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.173+0000 7fe37a800640 1 -- 172.21.15.97:0/2677441259 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37c0b43c0 con 0x7fe37c06bfd0 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7fe37a800640 1 --2- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c06c480 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe374009d90 tx=0x7fe37402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc29bd3bffe9d9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7f58d6a00640 1 --2- 172.21.15.97:0/1577162677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d80cd5b0 0x7f58d8145b70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f58d8162ac0 tx=0x7f58c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24504e31973a975b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7fe378a00640 1 -- 172.21.15.97:0/2677441259 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374039070 con 0x7fe37c06bfd0 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7fe378a00640 1 -- 172.21.15.97:0/2677441259 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37402cc60 con 0x7fe37c06bfd0 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7fe378a00640 1 -- 172.21.15.97:0/2677441259 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374034660 con 0x7fe37c06bfd0 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7f58d5600640 1 -- 172.21.15.97:0/1577162677 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8034070 con 0x7f58d80cd5b0 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7f58d5600640 1 -- 172.21.15.97:0/1577162677 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8030860 con 0x7f58d80cd5b0 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.174+0000 7f58d5600640 1 -- 172.21.15.97:0/1577162677 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8030c30 con 0x7f58d80cd5b0 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7fe380a00640 1 -- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 msgr2=0x7fe37c06c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7fe380a00640 1 --2- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c06c480 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe374009d90 tx=0x7fe37402beb0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7f58ddc00640 1 -- 172.21.15.97:0/1577162677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d80cd5b0 msgr2=0x7f58d8145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7f58ddc00640 1 --2- 172.21.15.97:0/1577162677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d80cd5b0 0x7f58d8145b70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f58d8162ac0 tx=0x7f58c802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7f58ddc00640 1 -- 172.21.15.97:0/1577162677 shutdown_connections 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7f58ddc00640 1 --2- 172.21.15.97:0/1577162677 >> v2:172.21.15.135:3300/0 conn(0x7f58d815cc70 0x7f58d8162fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7f58ddc00640 1 --2- 172.21.15.97:0/1577162677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d80cd5b0 0x7f58d8145b70 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7fe380a00640 1 -- 172.21.15.97:0/2677441259 shutdown_connections 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7fe380a00640 1 --2- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c06c480 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7fe380a00640 1 --2- 172.21.15.97:0/2677441259 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7fe380a00640 1 -- 172.21.15.97:0/2677441259 >> 172.21.15.97:0/2677441259 conn(0x7fe37c071380 msgr2=0x7fe37c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.175+0000 7f58ddc00640 1 -- 172.21.15.97:0/1577162677 >> 172.21.15.97:0/1577162677 conn(0x7f58d80cc4f0 msgr2=0x7f58d80cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7f58ddc00640 1 -- 172.21.15.97:0/1577162677 shutdown_connections 2024-03-06T07:34:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7fe380a00640 1 -- 172.21.15.97:0/2677441259 shutdown_connections 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7fe380a00640 1 -- 172.21.15.97:0/2677441259 wait complete. 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7f58ddc00640 1 -- 172.21.15.97:0/1577162677 wait complete. 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7f3c0e000640 1 Processor -- start 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7f8a6ca00640 1 Processor -- start 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7f3c0e000640 1 -- start start 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.176+0000 7f8a6ca00640 1 -- start start 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.177+0000 7f8a6ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a6815e360 0x7f8a680cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.177+0000 7f8a6ca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a680d12e0 0x7f8a68161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a6ca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8a68162270 0x7f8a6815e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a6ca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a680cf2b0 con 0x7f8a680d12e0 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a6ca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a680cf590 con 0x7f8a68162270 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a6ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680cf870 con 0x7f8a6815e360 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7fe380a00640 1 Processor -- start 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7fe380a00640 1 -- start start 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a65400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a680d12e0 0x7f8a68161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a65400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a680d12e0 0x7f8a68161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45886/0 (socket says 172.21.15.97:45886) 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a65400640 1 -- 172.21.15.97:0/3108297362 learned_addr learned my addr 172.21.15.97:0/3108297362 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a66800640 1 --2- 172.21.15.97:0/3108297362 >> v2:172.21.15.135:3300/0 conn(0x7f8a68162270 0x7f8a6815e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f8a65e00640 1 --2- 172.21.15.97:0/3108297362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a6815e360 0x7f8a680cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f58ddc00640 1 Processor -- start 2024-03-06T07:34:20.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f58ddc00640 1 -- start start 2024-03-06T07:34:20.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f3c0e000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3c0815ac90 0x7f3c0806e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f8a65400640 1 -- 172.21.15.97:0/3108297362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a6815e360 msgr2=0x7f8a680cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f8a65400640 1 --2- 172.21.15.97:0/3108297362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a6815e360 0x7f8a680cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f8a65400640 1 -- 172.21.15.97:0/3108297362 >> v2:172.21.15.135:3300/0 conn(0x7f8a68162270 msgr2=0x7f8a6815e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f8a65400640 1 --2- 172.21.15.97:0/3108297362 >> v2:172.21.15.135:3300/0 conn(0x7f8a68162270 0x7f8a6815e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f8a65400640 1 -- 172.21.15.97:0/3108297362 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a680cfb50 con 0x7f8a680d12e0 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.178+0000 7f3c0e000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3c0815b070 0x7f3c0815ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c0e000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c080cc000 con 0x7f3c0815ef50 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c0e000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080ce360 con 0x7f3c0815ac90 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c0e000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080ce640 con 0x7f3c0815b070 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7fe380a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c13d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c07e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3c0815ef50 0x7f3c080cac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54734/0 (socket says 172.21.15.97:54734) 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c07e00640 1 -- 172.21.15.97:0/2708921041 learned_addr learned my addr 172.21.15.97:0/2708921041 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c06a00640 1 --2- 172.21.15.97:0/2708921041 >> v2:172.21.15.135:3300/0 conn(0x7f3c0815b070 0x7f3c0815ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7fe380a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 0x7fe37c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.179+0000 7f3c07400640 1 --2- 172.21.15.97:0/2708921041 >> v2:172.21.15.97:3300/0 conn(0x7f3c0815ac90 0x7f3c0806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe380a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe380a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c06d9f0 con 0x7fe37c0b3960 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe380a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c0b4980 con 0x7fe37c06bfd0 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe380a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c161e20 con 0x7fe37c15e140 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c13d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 0x7fe37c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f3c06000640 1 -- 172.21.15.97:0/2708921041 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960010298 0 0) 0x7f3c080cc000 con 0x7f3c0815ef50 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 0x7fe37c0ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45900/0 (socket says 172.21.15.97:45900) 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 -- 172.21.15.97:0/4076255628 learned_addr learned my addr 172.21.15.97:0/4076255628 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f3c06a00640 1 -- 172.21.15.97:0/2708921041 >> v2:172.21.15.97:3300/0 conn(0x7f3c0815ac90 msgr2=0x7f3c0806e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f3c06a00640 1 --2- 172.21.15.97:0/2708921041 >> v2:172.21.15.97:3300/0 conn(0x7f3c0815ac90 0x7f3c0806e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f3c06a00640 1 -- 172.21.15.97:0/2708921041 >> v1:172.21.15.97:6789/0 conn(0x7f3c0815ef50 legacy=0x7f3c080cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f3c06a00640 1 -- 172.21.15.97:0/2708921041 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080ce920 con 0x7f3c0815b070 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f58ddc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58d80cd5b0 0x7f58d81621e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f58ddc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58d80cd990 0x7f58d81f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f58ddc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 0x7f58d8146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f58ddc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d81634f0 con 0x7f58d815cc70 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f58ddc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80cde40 con 0x7f58d80cd990 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7f58ddc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d8163670 con 0x7f58d80cd5b0 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe37a800640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 -- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 msgr2=0x7fe37c14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c14fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 -- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 msgr2=0x7fe37c13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c13d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f8a65400640 1 --2- 172.21.15.97:0/3108297362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a680d12e0 0x7f8a68161d30 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f8a680d4f60 tx=0x7f8a5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c494f0e78a57738d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.180+0000 7fe379400640 1 -- 172.21.15.97:0/4076255628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37c13d890 con 0x7fe37c0b3960 2024-03-06T07:34:20.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d6a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58d80cd5b0 0x7f58d81621e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 0x7f58d8146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 0x7f58d8146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45906/0 (socket says 172.21.15.97:45906) 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d6000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58d80cd990 0x7f58d81f9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 -- 172.21.15.97:0/279369619 learned_addr learned my addr 172.21.15.97:0/279369619 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7fe379400640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 0x7fe37c0ce6a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe36800c460 tx=0x7fe36800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368017070 con 0x7fe37c0b3960 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe368005ce0 con 0x7fe37c0b3960 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368007290 con 0x7fe37c0b3960 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f8a64a00640 1 -- 172.21.15.97:0/3108297362 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54034070 con 0x7f8a680d12e0 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f8a64a00640 1 -- 172.21.15.97:0/3108297362 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a54030860 con 0x7f8a680d12e0 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f8a64a00640 1 -- 172.21.15.97:0/3108297362 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54030c30 con 0x7f8a680d12e0 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 -- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58d80cd5b0 msgr2=0x7f58d81621e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58d80cd5b0 0x7f58d81621e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37c1f4120 con 0x7fe37c0b3960 2024-03-06T07:34:20.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 -- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58d80cd990 msgr2=0x7f58d81f9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58d80cd990 0x7f58d81f9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.181+0000 7f58d7400640 1 -- 172.21.15.97:0/279369619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d81fa980 con 0x7f58d815cc70 2024-03-06T07:34:20.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.182+0000 7f58d6000640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58d80cd990 0x7f58d81f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:20.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.182+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37c1f46e0 con 0x7fe37c0b3960 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.182+0000 7f58d7400640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 0x7f58d8146a10 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f58d000cd50 tx=0x7f58d000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.182+0000 7f3c06a00640 1 --2- 172.21.15.97:0/2708921041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815b070 0x7f3c0815ea10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3c08075020 tx=0x7f3bf402e010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2caf14740213f6aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.182+0000 7f8a6ca00640 1 -- 172.21.15.97:0/3108297362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a680d12e0 msgr2=0x7f8a68161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.182+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/3108297362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a680d12e0 0x7f8a68161d30 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f8a680d4f60 tx=0x7f8a5402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f8a6ca00640 1 -- 172.21.15.97:0/3108297362 shutdown_connections 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/3108297362 >> v2:172.21.15.135:3300/0 conn(0x7f8a68162270 0x7f8a6815e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/3108297362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a680d12e0 0x7f8a68161d30 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/3108297362 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a6815e360 0x7f8a680cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f8a6ca00640 1 -- 172.21.15.97:0/3108297362 >> 172.21.15.97:0/3108297362 conn(0x7f8a680d3f50 msgr2=0x7f8a680d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f8a6ca00640 1 -- 172.21.15.97:0/3108297362 shutdown_connections 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0003b90 con 0x7f58d815cc70 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d0003d30 con 0x7f58d815cc70 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0004240 con 0x7f58d815cc70 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d81fac60 con 0x7f58d815cc70 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d81fb220 con 0x7f58d815cc70 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f3c06000640 1 -- 172.21.15.97:0/2708921041 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf403b070 con 0x7f3c0815b070 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f3c06000640 1 -- 172.21.15.97:0/2708921041 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf402edc0 con 0x7f3c0815b070 2024-03-06T07:34:20.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.183+0000 7f3c06000640 1 -- 172.21.15.97:0/2708921041 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4036660 con 0x7f3c0815b070 2024-03-06T07:34:20.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f8a6ca00640 1 -- 172.21.15.97:0/3108297362 wait complete. 2024-03-06T07:34:20.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 -- 172.21.15.97:0/2708921041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815b070 msgr2=0x7f3c0815ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 --2- 172.21.15.97:0/2708921041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815b070 0x7f3c0815ea10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3c08075020 tx=0x7f3bf402e010 comp rx=0 tx=0).stop 2024-03-06T07:34:20.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe37c068010 con 0x7fe37c0b3960 2024-03-06T07:34:20.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 -- 172.21.15.97:0/2708921041 shutdown_connections 2024-03-06T07:34:20.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 --2- 172.21.15.97:0/2708921041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815b070 0x7f3c0815ea10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 --2- 172.21.15.97:0/2708921041 >> v2:172.21.15.97:3300/0 conn(0x7f3c0815ac90 0x7f3c0806e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 -- 172.21.15.97:0/2708921041 >> 172.21.15.97:0/2708921041 conn(0x7f3c080cc4f0 msgr2=0x7f3c080cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe36801a070 con 0x7fe37c0b3960 2024-03-06T07:34:20.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.184+0000 7f3c0e000640 1 -- 172.21.15.97:0/2708921041 shutdown_connections 2024-03-06T07:34:20.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.185+0000 7f3c0e000640 1 -- 172.21.15.97:0/2708921041 wait complete. 2024-03-06T07:34:20.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.185+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d804f330 con 0x7f58d815cc70 2024-03-06T07:34:20.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.186+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58d00043e0 con 0x7f58d815cc70 2024-03-06T07:34:20.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.186+0000 7f8a6ca00640 1 Processor -- start 2024-03-06T07:34:20.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.186+0000 7f8a6ca00640 1 -- start start 2024-03-06T07:34:20.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.186+0000 7fe372000640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe35c06b4d0 0x7fe35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.186+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe368089600 con 0x7fe37c0b3960 2024-03-06T07:34:20.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.186+0000 7f58cea00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58b806b5a0 0x7f58b806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58d008abf0 con 0x7f58d815cc70 2024-03-06T07:34:20.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7fe379e00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe35c06b4d0 0x7fe35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f3c0e000640 1 Processor -- start 2024-03-06T07:34:20.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f58d6a00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58b806b5a0 0x7f58b806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f3c0e000640 1 -- start start 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f8a6ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a680d12e0 0x7f8a680c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f8a6ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 0x7f8a681f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f8a6ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a68162270 0x7f8a6815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f8a6ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680d18c0 con 0x7f8a6815e360 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f8a6ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680d1740 con 0x7f8a68162270 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.187+0000 7f8a6ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6815ec80 con 0x7f8a680d12e0 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f8a65400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 0x7f8a681f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f8a65400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 0x7f8a681f3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45914/0 (socket says 172.21.15.97:45914) 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f8a65400640 1 -- 172.21.15.97:0/864887396 learned_addr learned my addr 172.21.15.97:0/864887396 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:20.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7fe379e00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe35c06b4d0 0x7fe35c06d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009ad0 tx=0x7fe36c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f58d6a00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58b806b5a0 0x7f58b806da60 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f58c8036000 tx=0x7f58c803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f3c0e000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3c0815ac90 0x7f3c08145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f3c0e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 0x7f3c081448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f3c0e000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815ef50 0x7f3c08144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f3c0e000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080c5a20 con 0x7f3c0815b070 2024-03-06T07:34:20.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f3c0e000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080ceee0 con 0x7f3c0815ef50 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f3c0e000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080c5ba0 con 0x7f3c0815ac90 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f8a65e00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a680d12e0 0x7f8a680c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.188+0000 7f8a66800640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a68162270 0x7f8a6815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f8a65400640 1 -- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a680d12e0 msgr2=0x7f8a680c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f8a65400640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a680d12e0 0x7f8a680c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f8a65400640 1 -- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a68162270 msgr2=0x7f8a6815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f8a65400640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a68162270 0x7f8a6815fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f8a65400640 1 -- 172.21.15.97:0/864887396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a68160360 con 0x7f8a6815e360 2024-03-06T07:34:20.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 0x7f3c081448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 0x7f3c081448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45926/0 (socket says 172.21.15.97:45926) 2024-03-06T07:34:20.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 -- 172.21.15.97:0/1972347658 learned_addr learned my addr 172.21.15.97:0/1972347658 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:20.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c07400640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3c0815ac90 0x7f3c08145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f8a65400640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 0x7f8a681f3950 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8a5402b9e0 tx=0x7f8a54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 -- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3c0815ac90 msgr2=0x7f3c08145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3c0815ac90 0x7f3c08145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 -- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815ef50 msgr2=0x7f3c08144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c07e00640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815ef50 0x7f3c08144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815ef50 0x7f3c08144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c06a00640 1 -- 172.21.15.97:0/1972347658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0815dfa0 con 0x7f3c0815b070 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.189+0000 7f3c07e00640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815ef50 0x7f3c08144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:20.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.190+0000 7f3c06a00640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 0x7f3c081448c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400bb30 tx=0x7f3bf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.191+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe37c068010 con 0x7fe37c0b3960 2024-03-06T07:34:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.195+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf403b040 con 0x7f3c0815b070 2024-03-06T07:34:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4044070 con 0x7f3c0815b070 2024-03-06T07:34:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0815e280 con 0x7f3c0815b070 2024-03-06T07:34:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58d0059a50 con 0x7f58d815cc70 2024-03-06T07:34:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf403f660 con 0x7f3c0815b070 2024-03-06T07:34:20.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54048070 con 0x7f8a6815e360 2024-03-06T07:34:20.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a54030660 con 0x7f8a6815e360 2024-03-06T07:34:20.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54034040 con 0x7f8a6815e360 2024-03-06T07:34:20.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a68160620 con 0x7f8a6815e360 2024-03-06T07:34:20.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c080c6db0 con 0x7f3c0815b070 2024-03-06T07:34:20.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.196+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a68160be0 con 0x7f8a6815e360 2024-03-06T07:34:20.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.197+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c080c6fb0 con 0x7f3c0815b070 2024-03-06T07:34:20.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.198+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6804f330 con 0x7f8a6815e360 2024-03-06T07:34:20.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.199+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bf403f800 con 0x7f3c0815b070 2024-03-06T07:34:20.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.199+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a5404b070 con 0x7f8a6815e360 2024-03-06T07:34:20.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.200+0000 7f3bff400640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3be806b4d0 0x7f3be806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.200+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bf40acb00 con 0x7f3c0815b070 2024-03-06T07:34:20.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.200+0000 7f8a5e000640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a4806b5a0 0x7f8a4806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.200+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a540ac300 con 0x7f8a6815e360 2024-03-06T07:34:20.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.200+0000 7f3c07400640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3be806b4d0 0x7f3be806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.201+0000 7f8a65e00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a4806b5a0 0x7f8a4806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.201+0000 7f3c07400640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3be806b4d0 0x7f3be806d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009a30 tx=0x7f3bf8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.201+0000 7f8a65e00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a4806b5a0 0x7f8a4806da60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8a58006fd0 tx=0x7f8a58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.204+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c080c6fb0 con 0x7f3c0815b070 2024-03-06T07:34:20.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.212+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a5407b1f0 con 0x7f8a6815e360 2024-03-06T07:34:20.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.475+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f3c080cd1c0 con 0x7f3c0815b070 2024-03-06T07:34:20.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.486+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f8a6808d9b0 con 0x7f8a6815e360 2024-03-06T07:34:20.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.488+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fe37c0ccec0 con 0x7fe37c0b3960 2024-03-06T07:34:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.494+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f58d806e8d0 con 0x7f58d815cc70 2024-03-06T07:34:20.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.694+0000 7f3bff400640 1 -- 172.21.15.97:0/1972347658 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7701 v75) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3bf407b9f0 con 0x7f3c0815b070 2024-03-06T07:34:20.696 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 7701 2024-03-06T07:34:20.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.697+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3be806b4d0 msgr2=0x7f3be806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.697+0000 7f3c0e000640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3be806b4d0 0x7f3be806d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009a30 tx=0x7f3bf8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.697+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 msgr2=0x7f3c081448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.697+0000 7f3c0e000640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 0x7f3c081448c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400bb30 tx=0x7f3bf40046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.698+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 shutdown_connections 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.698+0000 7f3c0e000640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3be806b4d0 0x7f3be806d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.698+0000 7f3c0e000640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c0815ef50 0x7f3c08144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.698+0000 7f3c0e000640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c0815b070 0x7f3c081448c0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.698+0000 7f3c0e000640 1 --2- 172.21.15.97:0/1972347658 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3c0815ac90 0x7f3c08145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.698+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 >> 172.21.15.97:0/1972347658 conn(0x7f3c080cc4f0 msgr2=0x7f3c08158280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.699+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 shutdown_connections 2024-03-06T07:34:20.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.700+0000 7f3c0e000640 1 -- 172.21.15.97:0/1972347658 wait complete. 2024-03-06T07:34:20.728 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:34:20.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:34:20.736 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:34:20.736 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:34:20.729+0000 7f68f6e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:20.736 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:20.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.737+0000 7f58cea00640 1 -- 172.21.15.97:0/279369619 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7716 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7f58d0059350 con 0x7f58d815cc70 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.737+0000 7f8a5e000640 1 -- 172.21.15.97:0/864887396 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7704 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7f8a5407aaf0 con 0x7f8a6815e360 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 7704 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.738+0000 7fe372000640 1 -- 172.21.15.97:0/4076255628 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5414 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe37c0ccec0 con 0x7fe37c0b3960 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 7716 2024-03-06T07:34:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 5414 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a4806b5a0 msgr2=0x7f8a4806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a4806b5a0 0x7f8a4806da60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8a58006fd0 tx=0x7f8a58008040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 msgr2=0x7f8a681f3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 0x7f8a681f3950 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8a5402b9e0 tx=0x7f8a54004730 comp rx=0 tx=0).stop 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 shutdown_connections 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a4806b5a0 0x7f8a4806da60 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a68162270 0x7f8a6815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a6815e360 0x7f8a681f3950 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 --2- 172.21.15.97:0/864887396 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a680d12e0 0x7f8a680c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 >> 172.21.15.97:0/864887396 conn(0x7f8a680d3f50 msgr2=0x7f8a680b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58b806b5a0 msgr2=0x7f58b806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f58ddc00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58b806b5a0 0x7f58b806da60 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f58c8036000 tx=0x7f58c803b040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 msgr2=0x7f58d8146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.742+0000 7f58ddc00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 0x7f58d8146a10 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f58d000cd50 tx=0x7f58d000a660 comp rx=0 tx=0).stop 2024-03-06T07:34:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 shutdown_connections 2024-03-06T07:34:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f8a6ca00640 1 -- 172.21.15.97:0/864887396 wait complete. 2024-03-06T07:34:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 shutdown_connections 2024-03-06T07:34:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58b806b5a0 0x7f58b806da60 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58d815cc70 0x7f58d8146a10 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58d80cd990 0x7f58d81f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 --2- 172.21.15.97:0/279369619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58d80cd5b0 0x7f58d81621e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 >> 172.21.15.97:0/279369619 conn(0x7f58d80cc4f0 msgr2=0x7f58d80c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe35c06b4d0 msgr2=0x7fe35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe35c06b4d0 0x7fe35c06d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009ad0 tx=0x7fe36c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 msgr2=0x7fe37c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 0x7fe37c0ce6a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe36800c460 tx=0x7fe36800c920 comp rx=0 tx=0).stop 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 shutdown_connections 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe35c06b4d0 0x7fe35c06d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe37c15e140 0x7fe37c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 shutdown_connections 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe37c0b3960 0x7fe37c0ce6a0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 --2- 172.21.15.97:0/4076255628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe37c06bfd0 0x7fe37c13d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 >> 172.21.15.97:0/4076255628 conn(0x7fe37c071380 msgr2=0x7fe37c0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.743+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 shutdown_connections 2024-03-06T07:34:20.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.744+0000 7f58ddc00640 1 -- 172.21.15.97:0/279369619 wait complete. 2024-03-06T07:34:20.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.744+0000 7fe380a00640 1 -- 172.21.15.97:0/4076255628 wait complete. 2024-03-06T07:34:20.765 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:34:20.765 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:34:20.773 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:34:20.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:34:20.774 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:34:20.774 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:20.775 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:34:20.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:34:20.782 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:34:20.782 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:20.819 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:34:20.819 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:20.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:20.884 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:34:20.936 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:34:20.936 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:34:20.961 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:34:20.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.990+0000 7f9714a00640 1 Processor -- start 2024-03-06T07:34:20.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.990+0000 7f9714a00640 1 -- start start 2024-03-06T07:34:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.991+0000 7f9714a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f97100d1cc0 0x7f97100b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.991+0000 7f9714a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100d4c50 con 0x7f97100d24c0 2024-03-06T07:34:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.991+0000 7f9714a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100cfa30 con 0x7f97100b2de0 2024-03-06T07:34:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.991+0000 7f9714a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97100d4a00 con 0x7f97100d1cc0 2024-03-06T07:34:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.991+0000 7f970de00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f97100d24c0 0x7f97100d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:54762/0 (socket says 172.21.15.97:54762) 2024-03-06T07:34:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.991+0000 7f970de00640 1 -- 172.21.15.97:0/1873684329 learned_addr learned my addr 172.21.15.97:0/1873684329 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970e800640 1 --2- 172.21.15.97:0/1873684329 >> v2:172.21.15.135:3300/0 conn(0x7f97100d1cc0 0x7f97100b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970ca00640 1 -- 172.21.15.97:0/1873684329 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665024642 0 0) 0x7f97100d4c50 con 0x7f97100d24c0 2024-03-06T07:34:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970ca00640 1 -- 172.21.15.97:0/1873684329 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f4003520 con 0x7f97100d24c0 2024-03-06T07:34:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970ca00640 1 -- 172.21.15.97:0/1873684329 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2653179472 0 0) 0x7f96f4003520 con 0x7f97100d24c0 2024-03-06T07:34:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970e800640 1 -- 172.21.15.97:0/1873684329 >> v1:172.21.15.135:6789/0 conn(0x7f97100b2de0 legacy=0x7f97100b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970e800640 1 -- 172.21.15.97:0/1873684329 >> v1:172.21.15.97:6789/0 conn(0x7f97100d24c0 legacy=0x7f97100d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.992+0000 7f970e800640 1 -- 172.21.15.97:0/1873684329 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97100cfd10 con 0x7f97100d1cc0 2024-03-06T07:34:20.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.993+0000 7f970e800640 1 --2- 172.21.15.97:0/1873684329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100d1cc0 0x7f97100b11b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9708009d90 tx=0x7f970802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236cfea483d070b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.993+0000 7f970ca00640 1 -- 172.21.15.97:0/1873684329 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9708039070 con 0x7f97100d1cc0 2024-03-06T07:34:20.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.993+0000 7f970ca00640 1 -- 172.21.15.97:0/1873684329 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970802c8b0 con 0x7f97100d1cc0 2024-03-06T07:34:20.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.993+0000 7f970ca00640 1 -- 172.21.15.97:0/1873684329 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700001670 con 0x7f97100d1cc0 2024-03-06T07:34:20.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 -- 172.21.15.97:0/1873684329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100d1cc0 msgr2=0x7f97100b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 --2- 172.21.15.97:0/1873684329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100d1cc0 0x7f97100b11b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9708009d90 tx=0x7f970802bd80 comp rx=0 tx=0).stop 2024-03-06T07:34:20.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 -- 172.21.15.97:0/1873684329 shutdown_connections 2024-03-06T07:34:20.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 --2- 172.21.15.97:0/1873684329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100d1cc0 0x7f97100b11b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 -- 172.21.15.97:0/1873684329 >> 172.21.15.97:0/1873684329 conn(0x7f97100d3ee0 msgr2=0x7f97100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 -- 172.21.15.97:0/1873684329 shutdown_connections 2024-03-06T07:34:20.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.994+0000 7f9714a00640 1 -- 172.21.15.97:0/1873684329 wait complete. 2024-03-06T07:34:20.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.996+0000 7f9714a00640 1 Processor -- start 2024-03-06T07:34:20.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.996+0000 7f9714a00640 1 -- start start 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f9714a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100b2de0 0x7f9710159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f9714a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 0x7f97100ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f9714a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97100d24c0 0x7f97100ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f9714a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100d0080 con 0x7f97100d1cc0 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f9714a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100cf7e0 con 0x7f97100b2de0 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f9714a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97100b16f0 con 0x7f97100d24c0 2024-03-06T07:34:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 0x7f97100ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 0x7f97100ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45968/0 (socket says 172.21.15.97:45968) 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 -- 172.21.15.97:0/30995330 learned_addr learned my addr 172.21.15.97:0/30995330 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970e800640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97100d24c0 0x7f97100ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970de00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100b2de0 0x7f9710159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 -- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97100d24c0 msgr2=0x7f97100ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97100d24c0 0x7f97100ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 -- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100b2de0 msgr2=0x7f9710159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100b2de0 0x7f9710159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970d400640 1 -- 172.21.15.97:0/30995330 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97100b9330 con 0x7f97100d1cc0 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970de00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100b2de0 0x7f9710159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.997+0000 7f970e800640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97100d24c0 0x7f97100ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.998+0000 7f970d400640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 0x7f97100ca3e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c450 tx=0x7f96fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.998+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0052d0 con 0x7f97100d1cc0 2024-03-06T07:34:21.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.998+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc005470 con 0x7f97100d1cc0 2024-03-06T07:34:21.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.998+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc012870 con 0x7f97100d1cc0 2024-03-06T07:34:21.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.998+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97100b9610 con 0x7f97100d1cc0 2024-03-06T07:34:21.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.998+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971015a4b0 con 0x7f97100d1cc0 2024-03-06T07:34:21.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:20.999+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97100b9b30 con 0x7f97100d1cc0 2024-03-06T07:34:21.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.000+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96fc012a10 con 0x7f97100d1cc0 2024-03-06T07:34:21.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.001+0000 7f9706000640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f96f006b500 0x7f96f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.001+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f96fc089ef0 con 0x7f97100d1cc0 2024-03-06T07:34:21.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.001+0000 7f970de00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f96f006b500 0x7f96f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.002+0000 7f970de00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f96f006b500 0x7f96f006d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9700001eb0 tx=0x7f9700009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.005+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96fc058c70 con 0x7f97100d1cc0 2024-03-06T07:34:21.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.223+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f971004f330 con 0x7f97100d1cc0 2024-03-06T07:34:21.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.224+0000 7f9706000640 1 -- 172.21.15.97:0/30995330 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+1802 (secure 0 0 0) 0x7f96fc058570 con 0x7f97100d1cc0 2024-03-06T07:34:21.226 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:21.226 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":76,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:20.730457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:34:21.226 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 76 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f96f006b500 msgr2=0x7f96f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f96f006b500 0x7f96f006d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9700001eb0 tx=0x7f9700009040 comp rx=0 tx=0).stop 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 msgr2=0x7f97100ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 0x7f97100ca3e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c450 tx=0x7f96fc00c910 comp rx=0 tx=0).stop 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 shutdown_connections 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f96f006b500 0x7f96f006d9c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f97100d24c0 0x7f97100ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97100d1cc0 0x7f97100ca3e0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 --2- 172.21.15.97:0/30995330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97100b2de0 0x7f9710159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 >> 172.21.15.97:0/30995330 conn(0x7f97100d3ee0 msgr2=0x7f971006c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.226+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 shutdown_connections 2024-03-06T07:34:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:21.227+0000 7f9714a00640 1 -- 172.21.15.97:0/30995330 wait complete. 2024-03-06T07:34:21.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:05.874005+0000', 'modified': '2024-03-06T07:34:20.730457+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 116, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:21.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:34:22.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.861+0000 7faa31000640 1 Processor -- start 2024-03-06T07:34:22.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.861+0000 7faa31000640 1 -- start start 2024-03-06T07:34:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa31000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa2c0af1c0 0x7faa2c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa31000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7faa2c070d20 0x7faa2c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa31000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2c071d30 con 0x7faa2c06d1a0 2024-03-06T07:34:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa31000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0711d0 con 0x7faa2c0af1c0 2024-03-06T07:34:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa31000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c071490 con 0x7faa2c070d20 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa29e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa2c0af1c0 0x7faa2c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa29e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa2c0af1c0 0x7faa2c15eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45978/0 (socket says 172.21.15.97:45978) 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.863+0000 7faa29e00640 1 -- 172.21.15.97:0/457847635 learned_addr learned my addr 172.21.15.97:0/457847635 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.864+0000 7faa29400640 1 --2- 172.21.15.97:0/457847635 >> v2:172.21.15.135:3300/0 conn(0x7faa2c070d20 0x7faa2c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.864+0000 7faa29e00640 1 -- 172.21.15.97:0/457847635 >> v2:172.21.15.135:3300/0 conn(0x7faa2c070d20 msgr2=0x7faa2c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.864+0000 7faa29e00640 1 --2- 172.21.15.97:0/457847635 >> v2:172.21.15.135:3300/0 conn(0x7faa2c070d20 0x7faa2c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.864+0000 7faa29e00640 1 -- 172.21.15.97:0/457847635 >> v1:172.21.15.135:6789/0 conn(0x7faa2c06d1a0 legacy=0x7faa2c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.864+0000 7faa29e00640 1 -- 172.21.15.97:0/457847635 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c071770 con 0x7faa2c0af1c0 2024-03-06T07:34:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.865+0000 7faa29e00640 1 --2- 172.21.15.97:0/457847635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0af1c0 0x7faa2c15eff0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009e50 tx=0x7faa1c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28840fbd4a787c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa28a00640 1 -- 172.21.15.97:0/457847635 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c02ca80 con 0x7faa2c0af1c0 2024-03-06T07:34:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa28a00640 1 -- 172.21.15.97:0/457847635 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c02cc20 con 0x7faa2c0af1c0 2024-03-06T07:34:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa28a00640 1 -- 172.21.15.97:0/457847635 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c0343f0 con 0x7faa2c0af1c0 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 -- 172.21.15.97:0/457847635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0af1c0 msgr2=0x7faa2c15eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 --2- 172.21.15.97:0/457847635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0af1c0 0x7faa2c15eff0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009e50 tx=0x7faa1c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 -- 172.21.15.97:0/457847635 shutdown_connections 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 --2- 172.21.15.97:0/457847635 >> v2:172.21.15.135:3300/0 conn(0x7faa2c070d20 0x7faa2c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 --2- 172.21.15.97:0/457847635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c0af1c0 0x7faa2c15eff0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 -- 172.21.15.97:0/457847635 >> 172.21.15.97:0/457847635 conn(0x7faa2c0b1730 msgr2=0x7faa2c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 -- 172.21.15.97:0/457847635 shutdown_connections 2024-03-06T07:34:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.866+0000 7faa31000640 1 -- 172.21.15.97:0/457847635 wait complete. 2024-03-06T07:34:22.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.868+0000 7faa31000640 1 Processor -- start 2024-03-06T07:34:22.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.868+0000 7faa31000640 1 -- start start 2024-03-06T07:34:22.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa31000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 0x7faa2c147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa31000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c070d20 0x7faa2c15fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa31000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c0af1c0 0x7faa2c160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa31000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c06ebe0 con 0x7faa2c06d1a0 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa31000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0b30b0 con 0x7faa2c070d20 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa31000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c06ed60 con 0x7faa2c0af1c0 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa29e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 0x7faa2c147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa29e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 0x7faa2c147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45986/0 (socket says 172.21.15.97:45986) 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.869+0000 7faa29e00640 1 -- 172.21.15.97:0/2061500710 learned_addr learned my addr 172.21.15.97:0/2061500710 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa29e00640 1 -- 172.21.15.97:0/2061500710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c0af1c0 msgr2=0x7faa2c160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa29e00640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c0af1c0 0x7faa2c160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa29e00640 1 -- 172.21.15.97:0/2061500710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c070d20 msgr2=0x7faa2c15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa29e00640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c070d20 0x7faa2c15fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa29e00640 1 -- 172.21.15.97:0/2061500710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c15e960 con 0x7faa2c06d1a0 2024-03-06T07:34:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa29e00640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 0x7faa2c147450 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7faa1c036040 tx=0x7faa1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c043020 con 0x7faa2c06d1a0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.870+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c03e450 con 0x7faa2c06d1a0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.871+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c15ec40 con 0x7faa2c06d1a0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.871+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c009510 con 0x7faa2c06d1a0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.871+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c15f200 con 0x7faa2c06d1a0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.872+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c04f330 con 0x7faa2c06d1a0 2024-03-06T07:34:22.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.872+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa1c046020 con 0x7faa2c06d1a0 2024-03-06T07:34:22.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.873+0000 7faa22000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0806b4d0 0x7faa0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.873+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7faa1c0acb50 con 0x7faa2c06d1a0 2024-03-06T07:34:22.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.874+0000 7faa29400640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0806b4d0 0x7faa0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.874+0000 7faa29400640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0806b4d0 0x7faa0806d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa2c160710 tx=0x7faa14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:22.878+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa1c04d030 con 0x7faa2c06d1a0 2024-03-06T07:34:23.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.115+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa2c0ae570 con 0x7faa2c06d1a0 2024-03-06T07:34:23.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.116+0000 7faa22000640 1 -- 172.21.15.97:0/2061500710 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4983 (secure 0 0 0) 0x7faa2c071d30 con 0x7faa2c06d1a0 2024-03-06T07:34:23.118 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:23.118 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":78,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:21.659619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:34:23.118 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 78 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.118+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0806b4d0 msgr2=0x7faa0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.118+0000 7faa31000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0806b4d0 0x7faa0806d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa2c160710 tx=0x7faa14008040 comp rx=0 tx=0).stop 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.118+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 msgr2=0x7faa2c147450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.118+0000 7faa31000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 0x7faa2c147450 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7faa1c036040 tx=0x7faa1c004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 shutdown_connections 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa0806b4d0 0x7faa0806d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa2c0af1c0 0x7faa2c160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa2c070d20 0x7faa2c15fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 --2- 172.21.15.97:0/2061500710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa2c06d1a0 0x7faa2c147450 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 >> 172.21.15.97:0/2061500710 conn(0x7faa2c0b1730 msgr2=0x7faa2c0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 shutdown_connections 2024-03-06T07:34:23.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:23.119+0000 7faa31000640 1 -- 172.21.15.97:0/2061500710 wait complete. 2024-03-06T07:34:23.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:05.874005+0000', 'modified': '2024-03-06T07:34:21.659619+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 116, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7977}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7977': {'gid': 7977, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.97:6837/3308266804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 3308266804}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3308266804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:23.544 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7977:up:reconnect 2024-03-06T07:34:24.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:24.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.763+0000 7f934b200640 1 Processor -- start 2024-03-06T07:34:24.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.763+0000 7f934b200640 1 -- start start 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.764+0000 7f934b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0d0600 0x7f934c0d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f934c0ce840 0x7f934c0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c09d5e0 con 0x7f934c1615c0 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c0d4c00 con 0x7f934c0ce840 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0b40b0 con 0x7f934c0d0600 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f934c0ce840 0x7f934c0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f9349e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f934c1615c0 0x7f934c0d4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57708/0 (socket says 172.21.15.97:57708) 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f9349e00640 1 -- 172.21.15.97:0/960497475 learned_addr learned my addr 172.21.15.97:0/960497475 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934a800640 1 -- 172.21.15.97:0/960497475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0d0600 msgr2=0x7f934c0d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f9341400640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0d0600 0x7f934c0d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934a800640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0d0600 0x7f934c0d0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934a800640 1 -- 172.21.15.97:0/960497475 >> v1:172.21.15.97:6789/0 conn(0x7f934c1615c0 legacy=0x7f934c0d4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f9349400640 1 -- 172.21.15.97:0/960497475 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600809651 0 0) 0x7f934c09d5e0 con 0x7f934c1615c0 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f934a800640 1 -- 172.21.15.97:0/960497475 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c0b3780 con 0x7f934c0ce840 2024-03-06T07:34:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.765+0000 7f9341400640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0d0600 0x7f934c0d0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.767+0000 7f934a800640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c0ce840 0x7f934c0cecb0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f934c0d4d80 tx=0x7f934402b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598256b7920e59f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.767+0000 7f9349400640 1 -- 172.21.15.97:0/960497475 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344031770 con 0x7f934c0ce840 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.767+0000 7f9349400640 1 -- 172.21.15.97:0/960497475 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344037070 con 0x7f934c0ce840 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.767+0000 7f9349400640 1 -- 172.21.15.97:0/960497475 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344031b30 con 0x7f934c0ce840 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.767+0000 7f934b200640 1 -- 172.21.15.97:0/960497475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c0ce840 msgr2=0x7f934c0cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.767+0000 7f934b200640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c0ce840 0x7f934c0cecb0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f934c0d4d80 tx=0x7f934402b810 comp rx=0 tx=0).stop 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.768+0000 7f934b200640 1 -- 172.21.15.97:0/960497475 shutdown_connections 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.768+0000 7f934b200640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c0ce840 0x7f934c0cecb0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.768+0000 7f934b200640 1 --2- 172.21.15.97:0/960497475 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0d0600 0x7f934c0d0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.768+0000 7f934b200640 1 -- 172.21.15.97:0/960497475 >> 172.21.15.97:0/960497475 conn(0x7f934c09d080 msgr2=0x7f934c0d3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.768+0000 7f934b200640 1 -- 172.21.15.97:0/960497475 shutdown_connections 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.768+0000 7f934b200640 1 -- 172.21.15.97:0/960497475 wait complete. 2024-03-06T07:34:24.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.769+0000 7f934b200640 1 Processor -- start 2024-03-06T07:34:24.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.769+0000 7f934b200640 1 -- start start 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.770+0000 7f934b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0ce840 0x7f934c0cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.770+0000 7f934b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c0d0600 0x7f934c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.770+0000 7f934b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 0x7f934c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.770+0000 7f934b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0b4540 con 0x7f934c1615c0 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.770+0000 7f934b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0b3d40 con 0x7f934c0d0600 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.770+0000 7f934b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0b46c0 con 0x7f934c0ce840 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 0x7f934c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 0x7f934c0b8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49412/0 (socket says 172.21.15.97:49412) 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 -- 172.21.15.97:0/2667221271 learned_addr learned my addr 172.21.15.97:0/2667221271 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 -- 172.21.15.97:0/2667221271 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0ce840 msgr2=0x7f934c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0ce840 0x7f934c0cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 -- 172.21.15.97:0/2667221271 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c0d0600 msgr2=0x7f934c15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c0d0600 0x7f934c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 -- 172.21.15.97:0/2667221271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c0c7990 con 0x7f934c1615c0 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f934a800640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 0x7f934c0b8b20 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f9344009890 tx=0x7f93440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.771+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344004230 con 0x7f934c1615c0 2024-03-06T07:34:24.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.772+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93440043d0 con 0x7f934c1615c0 2024-03-06T07:34:24.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.772+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c0ca3e0 con 0x7f934c1615c0 2024-03-06T07:34:24.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.772+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344003850 con 0x7f934c1615c0 2024-03-06T07:34:24.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.772+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c15cce0 con 0x7f934c1615c0 2024-03-06T07:34:24.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.773+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c04f330 con 0x7f934c1615c0 2024-03-06T07:34:24.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.774+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9344031070 con 0x7f934c1615c0 2024-03-06T07:34:24.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.774+0000 7f9342a00640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.774+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f93440abd00 con 0x7f934c1615c0 2024-03-06T07:34:24.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.775+0000 7f9349e00640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.775+0000 7f9349e00640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f932806b4d0 0x7f932806d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f933c005c00 tx=0x7f933c001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:24.783+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9344041070 con 0x7f934c1615c0 2024-03-06T07:34:25.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.014+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f934c0b9bb0 con 0x7f934c1615c0 2024-03-06T07:34:25.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.015+0000 7f9342a00640 1 -- 172.21.15.97:0/2667221271 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f934407ac30 con 0x7f934c1615c0 2024-03-06T07:34:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:25.018 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:34:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:34:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f932806b4d0 msgr2=0x7f932806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f932806b4d0 0x7f932806d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f933c005c00 tx=0x7f933c001690 comp rx=0 tx=0).stop 2024-03-06T07:34:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 msgr2=0x7f934c0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 0x7f934c0b8b20 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f9344009890 tx=0x7f93440046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 shutdown_connections 2024-03-06T07:34:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f934c1615c0 0x7f934c0b8b20 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c0d0600 0x7f934c15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 --2- 172.21.15.97:0/2667221271 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f934c0ce840 0x7f934c0cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 >> 172.21.15.97:0/2667221271 conn(0x7f934c09d080 msgr2=0x7f934c0b5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 shutdown_connections 2024-03-06T07:34:25.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.019+0000 7f934b200640 1 -- 172.21.15.97:0/2667221271 wait complete. 2024-03-06T07:34:25.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:05.874005+0000', 'modified': '2024-03-06T07:34:24.644540+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 116, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7977}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7977': {'gid': 7977, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.97:6837/3308266804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 3308266804}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3308266804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7977, 'qdb_cluster': [7977]} 2024-03-06T07:34:25.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:25.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:34:25.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.646+0000 7ffac7a00640 1 Processor -- start 2024-03-06T07:34:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.647+0000 7ffac7a00640 1 -- start start 2024-03-06T07:34:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.648+0000 7ffac7a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffac00cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.648+0000 7ffac7a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00cf2b0 con 0x7ffac0162440 2024-03-06T07:34:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.648+0000 7ffac7a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00cf590 con 0x7ffac00d12e0 2024-03-06T07:34:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.648+0000 7ffac7a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00cf870 con 0x7ffac015e360 2024-03-06T07:34:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.649+0000 7ffac5a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ffac0162440 0x7ffac00d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57716/0 (socket says 172.21.15.97:57716) 2024-03-06T07:34:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.649+0000 7ffac5a00640 1 -- 172.21.15.97:0/976854153 learned_addr learned my addr 172.21.15.97:0/976854153 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.649+0000 7ffac5000640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffac00cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.649+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685343154 0 0) 0x7ffac00cf2b0 con 0x7ffac0162440 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.649+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4003520 con 0x7ffac0162440 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447990111 0 0) 0x7ffaa4003520 con 0x7ffac0162440 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 msgr2=0x7ffac00cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffac00cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 >> v1:172.21.15.135:6789/0 conn(0x7ffac00d12e0 legacy=0x7ffac0161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00cfb50 con 0x7ffac0162440 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ffab80019f0 con 0x7ffac0162440 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 >> v1:172.21.15.97:6789/0 conn(0x7ffac0162440 legacy=0x7ffac00d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa40039e0 0x7ffaa4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa4004800 0x7ffaa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffaa40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00cf870 con 0x7ffaa40039e0 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa400bb30 con 0x7ffaa4004800 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa400bd30 con 0x7ffac015e360 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5000640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa40039e0 0x7ffaa4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5a00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffaa40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5000640 1 -- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 msgr2=0x7ffaa40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5000640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffaa40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffab7e00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa4004800 0x7ffaa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5000640 1 -- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa4004800 msgr2=0x7ffaa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5000640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa4004800 0x7ffaa4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5000640 1 -- 172.21.15.97:0/976854153 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4003520 con 0x7ffaa40039e0 2024-03-06T07:34:25.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.650+0000 7ffac5a00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac015e360 0x7ffaa40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffac5000640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa40039e0 0x7ffaa4003e00 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009a90 tx=0x7ffaa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c0daf7b9d001e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0039070 con 0x7ffaa40039e0 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab002c680 con 0x7ffaa40039e0 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffabfe00640 1 -- 172.21.15.97:0/976854153 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0009690 con 0x7ffaa40039e0 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffac7a00640 1 -- 172.21.15.97:0/976854153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa40039e0 msgr2=0x7ffaa4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffac7a00640 1 --2- 172.21.15.97:0/976854153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa40039e0 0x7ffaa4003e00 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009a90 tx=0x7ffaa4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.652+0000 7ffac5a00640 1 -- 172.21.15.97:0/976854153 reap_dead start 2024-03-06T07:34:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.653+0000 7ffac7a00640 1 -- 172.21.15.97:0/976854153 shutdown_connections 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.653+0000 7ffac7a00640 1 -- 172.21.15.97:0/976854153 >> 172.21.15.97:0/976854153 conn(0x7ffac00d3f50 msgr2=0x7ffac00d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.653+0000 7ffac7a00640 1 -- 172.21.15.97:0/976854153 shutdown_connections 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.653+0000 7ffac7a00640 1 -- 172.21.15.97:0/976854153 wait complete. 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.654+0000 7ffac7a00640 1 Processor -- start 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 -- start start 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa40039e0 0x7ffac0154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac00cb810 0x7ffac00cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 0x7ffac00b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d18c0 con 0x7ffac00b3b50 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d1740 con 0x7ffaa40039e0 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.655+0000 7ffac7a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00cec10 con 0x7ffac00cb810 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 0x7ffac00b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 0x7ffac00b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49436/0 (socket says 172.21.15.97:49436) 2024-03-06T07:34:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 -- 172.21.15.97:0/985841161 learned_addr learned my addr 172.21.15.97:0/985841161 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffab7e00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac00cb810 0x7ffac00cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 -- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac00cb810 msgr2=0x7ffac00cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5000640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa40039e0 0x7ffac0154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac00cb810 0x7ffac00cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 -- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa40039e0 msgr2=0x7ffac0154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa40039e0 0x7ffac0154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 -- 172.21.15.97:0/985841161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00b5800 con 0x7ffac00b3b50 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffab7e00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac00cb810 0x7ffac00cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.656+0000 7ffac5a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 0x7ffac00b3fd0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ffac00d4f90 tx=0x7ffab800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.657+0000 7ffabd600640 1 -- 172.21.15.97:0/985841161 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab800bb80 con 0x7ffac00b3b50 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.657+0000 7ffabd600640 1 -- 172.21.15.97:0/985841161 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab800bd20 con 0x7ffac00b3b50 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.657+0000 7ffabd600640 1 -- 172.21.15.97:0/985841161 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8007310 con 0x7ffac00b3b50 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.657+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac00b4760 con 0x7ffac00b3b50 2024-03-06T07:34:25.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.657+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac00b4d20 con 0x7ffac00b3b50 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.659+0000 7ffabd600640 1 -- 172.21.15.97:0/985841161 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffab801b020 con 0x7ffac00b3b50 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.660+0000 7ffabd600640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.660+0000 7ffabd600640 1 -- 172.21.15.97:0/985841161 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffab808a4a0 con 0x7ffac00b3b50 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.660+0000 7ffac5000640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.660+0000 7ffaca672740 1 Processor -- start 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.661+0000 7ffaca672740 1 -- start start 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.661+0000 7ffac5000640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ffab0036000 tx=0x7ffab0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725527750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x558725557540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffaca672740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 0x558725557a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffaca672740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x558725491600 con 0x558725527c90 2024-03-06T07:34:25.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872543a930 con 0x558725554b90 2024-03-06T07:34:25.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55872548d6a0 con 0x5587255547b0 2024-03-06T07:34:25.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 0x558725557a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 0x558725557a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49444/0 (socket says 172.21.15.97:49444) 2024-03-06T07:34:25.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 -- 172.21.15.97:0/1296583644 learned_addr learned my addr 172.21.15.97:0/1296583644 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffab7e00640 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x558725557540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5000640 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725527750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 -- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 msgr2=0x558725527750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725527750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 -- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 msgr2=0x558725557540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x558725557540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffac5a00640 1 -- 172.21.15.97:0/1296583644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587251a62d0 con 0x558725527c90 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.662+0000 7ffab7e00640 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x558725557540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.663+0000 7ffac5a00640 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 0x558725557a80 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ffab8055010 tx=0x7ffab800dec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55978c7c4ab3afe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.663+0000 7ffab5600640 1 -- 172.21.15.97:0/1296583644 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab805cd00 con 0x558725527c90 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.663+0000 7ffab5600640 1 -- 172.21.15.97:0/1296583644 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab80612c0 con 0x558725527c90 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.663+0000 7ffab5600640 1 -- 172.21.15.97:0/1296583644 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab801bdc0 con 0x558725527c90 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 -- 172.21.15.97:0/1296583644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 msgr2=0x558725557a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 0x558725557a80 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ffab8055010 tx=0x7ffab800dec0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 -- 172.21.15.97:0/1296583644 shutdown_connections 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725527c90 0x558725557a80 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x558725557540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 --2- 172.21.15.97:0/1296583644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725527750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 -- 172.21.15.97:0/1296583644 >> 172.21.15.97:0/1296583644 conn(0x5587255299c0 msgr2=0x558725572540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 -- 172.21.15.97:0/1296583644 shutdown_connections 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.664+0000 7ffaca672740 1 -- 172.21.15.97:0/1296583644 wait complete. 2024-03-06T07:34:25.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.665+0000 7ffaca672740 1 -- start start 2024-03-06T07:34:25.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.665+0000 7ffaca672740 1 Processor -- start 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725548520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 0x55872555a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x55872556cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872552bf60 con 0x558725554b90 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872544dd90 con 0x55872556cb30 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55872552b350 con 0x5587255547b0 2024-03-06T07:34:25.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffaca672740 10 client.0 _ll_register_callbacks cb 0x558725489350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffab7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 0x55872555a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffab7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 0x55872555a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49458/0 (socket says 172.21.15.97:49458) 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.667+0000 7ffab7e00640 1 -- 172.21.15.97:0/691113802 learned_addr learned my addr 172.21.15.97:0/691113802 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffab7e00640 1 -- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 msgr2=0x558725548520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffab7e00640 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725548520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffab7e00640 1 -- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 msgr2=0x55872556cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffac5a00640 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x55872556cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffab7e00640 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x55872556cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffab7e00640 1 -- 172.21.15.97:0/691113802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587252706b0 con 0x558725554b90 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffab7e00640 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 0x55872555a260 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8001e90 tx=0x7ffaa800f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe1352c1552aae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffaae000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffaae000640 1 -- 172.21.15.97:0/691113802 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8017070 con 0x558725554b90 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffaae000640 1 -- 172.21.15.97:0/691113802 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffaa8013840 con 0x558725554b90 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.668+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffab80612c0 con 0x558725554b90 2024-03-06T07:34:25.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaae000640 1 -- 172.21.15.97:0/691113802 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaa8022050 con 0x558725554b90 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 10 client.8010 fetch_fsmap learned FSMap version 80 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x55872555b3e0 con 0x558725554b90 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaae000640 1 -- 172.21.15.97:0/691113802 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab80612c0 con 0x558725554b90 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaae000640 1 -- 172.21.15.97:0/691113802 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 80) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ffaa8025020 con 0x558725554b90 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 10 client.8010 fetch_fsmap finished waiting for FSMap version 80 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 10 client.8010 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 20 client.8010 populate_metadata read hostname 'smithi097' 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x55872556b2b0 0x55872556b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 4 client.8010 mds_command: new command op to 7977 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:25.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 --> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55872549eb40 con 0x55872556b2b0 2024-03-06T07:34:25.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.669+0000 7ffac5000640 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x55872556b2b0 0x55872556b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.670+0000 7ffac5000640 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x55872556b2b0 0x55872556b690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7883bcb23fa5a6f6 server_cookie=67b5bf7c7b87c21d in_seq=0 out_seq=0 2024-03-06T07:34:25.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.670+0000 7ffaae000640 10 client.8010 ms_handle_connect on v2:172.21.15.97:6836/3308266804 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.671+0000 7ffaae000640 1 -- 172.21.15.97:0/691113802 <== mds.0 v2:172.21.15.97:6836/3308266804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55872549eb40 con 0x55872556b2b0 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.671+0000 7ffaae000640 10 client.8010 handle_command_reply: tid=0 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.671+0000 7ffaca672740 1 client.8010 shutdown 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.671+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 msgr2=0x55872555a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.671+0000 7ffaca672740 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 0x55872555a260 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8001e90 tx=0x7ffaa800f6c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 shutdown_connections 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x55872556b2b0 0x55872556b690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x55872556cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558725554b90 0x55872555a260 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaae000640 0 client.8010 ms_handle_reset on v2:172.21.15.97:6836/3308266804 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 --2- 172.21.15.97:0/691113802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725548520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 >> 172.21.15.97:0/691113802 conn(0x558725527c90 msgr2=0x558725526eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 shutdown_connections 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 1 -- 172.21.15.97:0/691113802 wait complete. 2024-03-06T07:34:25.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.672+0000 7ffaca672740 20 client.8010 trim_cache size 0 max 16384 2024-03-06T07:34:25.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.690+0000 7ffaca672740 1 Processor -- start 2024-03-06T07:34:25.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.690+0000 7ffaca672740 1 -- start start 2024-03-06T07:34:25.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725569090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x5587255dc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffaca672740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 0x558725585e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffaca672740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac0200c30 con 0x55872556cb30 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01ffdf0 con 0x558725554b90 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac015f050 con 0x5587255547b0 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 0x558725585e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 0x558725585e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49460/0 (socket says 172.21.15.97:49460) 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 -- 172.21.15.97:0/1025619246 learned_addr learned my addr 172.21.15.97:0/1025619246 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 -- 172.21.15.97:0/1025619246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 msgr2=0x558725569090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725569090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 -- 172.21.15.97:0/1025619246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 msgr2=0x5587255dc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x5587255dc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.691+0000 7ffac5a00640 1 -- 172.21.15.97:0/1025619246 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558725572d80 con 0x55872556cb30 2024-03-06T07:34:25.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffac5a00640 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 0x558725585e70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ffab8055010 tx=0x7ffab808a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26915237031493c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffab4c00640 1 -- 172.21.15.97:0/1025619246 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8075070 con 0x55872556cb30 2024-03-06T07:34:25.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffab4c00640 1 -- 172.21.15.97:0/1025619246 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab80612c0 con 0x55872556cb30 2024-03-06T07:34:25.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffab4c00640 1 -- 172.21.15.97:0/1025619246 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab801bdc0 con 0x55872556cb30 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 -- 172.21.15.97:0/1025619246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 msgr2=0x558725585e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 0x558725585e70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ffab8055010 tx=0x7ffab808a050 comp rx=0 tx=0).stop 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 -- 172.21.15.97:0/1025619246 shutdown_connections 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55872556cb30 0x558725585e70 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558725554b90 0x5587255dc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 --2- 172.21.15.97:0/1025619246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5587255547b0 0x558725569090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 -- 172.21.15.97:0/1025619246 >> 172.21.15.97:0/1025619246 conn(0x558725527c90 msgr2=0x55872552e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.692+0000 7ffaca672740 1 -- 172.21.15.97:0/1025619246 shutdown_connections 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.693+0000 7ffaca672740 1 -- 172.21.15.97:0/1025619246 wait complete. 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.693+0000 7ffaca672740 1 -- start start 2024-03-06T07:34:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.693+0000 7ffaca672740 1 Processor -- start 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 0x55872552e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x558725554b90 0x558725585e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x5587255863b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872555b100 con 0x5587255547b0 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac015f650 con 0x55872556cb30 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55872556a810 con 0x558725554b90 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffaca672740 10 client.0 _ll_register_callbacks cb 0x7ffaa8007c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.694+0000 7ffac5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 0x55872552e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 0x55872552e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49474/0 (socket says 172.21.15.97:49474) 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 -- 172.21.15.97:0/1194948116 learned_addr learned my addr 172.21.15.97:0/1194948116 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 -- 172.21.15.97:0/1194948116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x558725554b90 msgr2=0x558725585e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x558725554b90 0x558725585e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 -- 172.21.15.97:0/1194948116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 msgr2=0x5587255863b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x5587255863b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 -- 172.21.15.97:0/1194948116 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55872555b5c0 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffac5000640 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 0x55872552e770 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7ffab0037040 tx=0x7ffab000f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683ed4d64914bb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffaae000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffaae000640 1 -- 172.21.15.97:0/1194948116 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0039040 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffaae000640 1 -- 172.21.15.97:0/1194948116 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffab004dd50 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.695+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffab80612c0 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaae000640 1 -- 172.21.15.97:0/1194948116 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab0056050 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 10 client.8016 fetch_fsmap learned FSMap version 80 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x558725557b20 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaae000640 1 -- 172.21.15.97:0/1194948116 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab80612c0 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaae000640 1 -- 172.21.15.97:0/1194948116 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 80) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ffab0059020 con 0x5587255547b0 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 10 client.8016 fetch_fsmap finished waiting for FSMap version 80 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 10 client.8016 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 20 client.8016 populate_metadata read hostname 'smithi097' 2024-03-06T07:34:25.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x5587255688b0 0x558725573790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 4 client.8016 mds_command: new command op to 7977 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 --> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558725498e90 con 0x5587255688b0 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.696+0000 7ffab7e00640 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x5587255688b0 0x558725573790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffab7e00640 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x5587255688b0 0x558725573790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3146f8c20fcfe3ec server_cookie=97dec825e6d3b44f in_seq=0 out_seq=0 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaae000640 10 client.8016 ms_handle_connect on v2:172.21.15.97:6836/3308266804 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaae000640 1 -- 172.21.15.97:0/1194948116 <== mds.0 v2:172.21.15.97:6836/3308266804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x558725498e90 con 0x5587255688b0 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaae000640 10 client.8016 handle_command_reply: tid=0 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 client.8016 shutdown 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 msgr2=0x55872552e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 0x55872552e770 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7ffab0037040 tx=0x7ffab000f180 comp rx=0 tx=0).stop 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 shutdown_connections 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:6836/3308266804,v1:172.21.15.97:6837/3308266804] conn(0x5587255688b0 0x558725573790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55872556cb30 0x5587255863b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x558725554b90 0x558725585e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaae000640 0 client.8016 ms_handle_reset on v2:172.21.15.97:6836/3308266804 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 --2- 172.21.15.97:0/1194948116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587255547b0 0x55872552e770 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 >> 172.21.15.97:0/1194948116 conn(0x558725527c90 msgr2=0x5587255749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 shutdown_connections 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.697+0000 7ffaca672740 1 -- 172.21.15.97:0/1194948116 wait complete. 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffaca672740 20 client.8016 trim_cache size 0 max 16384 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 7977, 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:34:25.703 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 4.058192139, 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 80, 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 116, 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 116, 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 4.7342639269999998 2024-03-06T07:34:25.704 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa9c06b4d0 msgr2=0x7ffa9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ffab0036000 tx=0x7ffab0030000 comp rx=0 tx=0).stop 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 msgr2=0x7ffac00b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 0x7ffac00b3fd0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ffac00d4f90 tx=0x7ffab800b4a0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 shutdown_connections 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffac00b3b50 0x7ffac00b3fd0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ffac00cb810 0x7ffac00cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 --2- 172.21.15.97:0/985841161 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffaa40039e0 0x7ffac0154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 >> 172.21.15.97:0/985841161 conn(0x7ffac00d3f50 msgr2=0x7ffac00d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 shutdown_connections 2024-03-06T07:34:25.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.698+0000 7ffac7a00640 1 -- 172.21.15.97:0/985841161 wait complete. 2024-03-06T07:34:25.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:34:25.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.910+0000 7fdec7e00640 1 Processor -- start 2024-03-06T07:34:25.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.910+0000 7fdec7e00640 1 -- start start 2024-03-06T07:34:25.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.911+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec8162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.911+0000 7fdec7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdec80affe0 0x7fdec80d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.911+0000 7fdec7e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec80cef70 con 0x7fdec815dfd0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.911+0000 7fdec7e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec8162270 con 0x7fdec80affe0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.911+0000 7fdec7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec815e3b0 con 0x7fdec806b6d0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec6a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdec815dfd0 0x7fdec806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57726/0 (socket says 172.21.15.97:57726) 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec6a00640 1 -- 172.21.15.97:0/3145204200 learned_addr learned my addr 172.21.15.97:0/3145204200 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec7400640 1 --2- 172.21.15.97:0/3145204200 >> v2:172.21.15.97:3300/0 conn(0x7fdec80affe0 0x7fdec80d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec6000640 1 --2- 172.21.15.97:0/3145204200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec8162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec5600640 1 -- 172.21.15.97:0/3145204200 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252162435 0 0) 0x7fdec80cef70 con 0x7fdec815dfd0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec5600640 1 -- 172.21.15.97:0/3145204200 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeac003520 con 0x7fdec815dfd0 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec7400640 1 -- 172.21.15.97:0/3145204200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 msgr2=0x7fdec8162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec7400640 1 --2- 172.21.15.97:0/3145204200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec8162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec7400640 1 -- 172.21.15.97:0/3145204200 >> v1:172.21.15.97:6789/0 conn(0x7fdec815dfd0 legacy=0x7fdec806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.912+0000 7fdec7400640 1 -- 172.21.15.97:0/3145204200 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8161c60 con 0x7fdec80affe0 2024-03-06T07:34:25.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.913+0000 7fdec7400640 1 --2- 172.21.15.97:0/3145204200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec80affe0 0x7fdec80d06f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009d90 tx=0x7fdec002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef1b8904355d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec5600640 1 -- 172.21.15.97:0/3145204200 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0039070 con 0x7fdec80affe0 2024-03-06T07:34:25.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec5600640 1 -- 172.21.15.97:0/3145204200 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec002cb30 con 0x7fdec80affe0 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec5600640 1 -- 172.21.15.97:0/3145204200 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0034440 con 0x7fdec80affe0 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 -- 172.21.15.97:0/3145204200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec80affe0 msgr2=0x7fdec80d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 --2- 172.21.15.97:0/3145204200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec80affe0 0x7fdec80d06f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009d90 tx=0x7fdec002bd80 comp rx=0 tx=0).stop 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 -- 172.21.15.97:0/3145204200 shutdown_connections 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 --2- 172.21.15.97:0/3145204200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec80affe0 0x7fdec80d06f0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 --2- 172.21.15.97:0/3145204200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec8162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 -- 172.21.15.97:0/3145204200 >> 172.21.15.97:0/3145204200 conn(0x7fdec806c0d0 msgr2=0x7fdec806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 -- 172.21.15.97:0/3145204200 shutdown_connections 2024-03-06T07:34:25.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.914+0000 7fdec7e00640 1 -- 172.21.15.97:0/3145204200 wait complete. 2024-03-06T07:34:25.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.916+0000 7fdec7e00640 1 Processor -- start 2024-03-06T07:34:25.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.916+0000 7fdec7e00640 1 -- start start 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec81f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdec80affe0 0x7fdec815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 0x7fdec815c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80ce8d0 con 0x7fdec815dfd0 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8161f00 con 0x7fdec80affe0 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80cea50 con 0x7fdec806b6d0 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 0x7fdec815c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 0x7fdec815c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49492/0 (socket says 172.21.15.97:49492) 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.917+0000 7fdec7400640 1 -- 172.21.15.97:0/2828260446 learned_addr learned my addr 172.21.15.97:0/2828260446 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec6a00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec81f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7400640 1 -- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 msgr2=0x7fdec81f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec6000640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdec80affe0 0x7fdec815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7400640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec81f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7400640 1 -- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdec80affe0 msgr2=0x7fdec815c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7400640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdec80affe0 0x7fdec815c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7400640 1 -- 172.21.15.97:0/2828260446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8160ee0 con 0x7fdec815dfd0 2024-03-06T07:34:25.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec6a00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec81f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:25.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7400640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 0x7fdec815c6e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009d90 tx=0x7fdec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0039050 con 0x7fdec815dfd0 2024-03-06T07:34:25.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec0045070 con 0x7fdec815dfd0 2024-03-06T07:34:25.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.918+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec815cd10 con 0x7fdec815dfd0 2024-03-06T07:34:25.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.919+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec003e830 con 0x7fdec815dfd0 2024-03-06T07:34:25.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.919+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec815e870 con 0x7fdec815dfd0 2024-03-06T07:34:25.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.920+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec804f330 con 0x7fdec815dfd0 2024-03-06T07:34:25.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.921+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdec003e9d0 con 0x7fdec815dfd0 2024-03-06T07:34:25.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.922+0000 7fdebea00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdea806b4d0 0x7fdea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.922+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdec00ad040 con 0x7fdec815dfd0 2024-03-06T07:34:25.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.922+0000 7fdec6a00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdea806b4d0 0x7fdea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.922+0000 7fdec6a00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdea806b4d0 0x7fdea806d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8005880 tx=0x7fdeb8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:25.926+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdec007c010 con 0x7fdec815dfd0 2024-03-06T07:34:26.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.164+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdec80b5e60 con 0x7fdec815dfd0 2024-03-06T07:34:26.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.165+0000 7fdebea00640 1 -- 172.21.15.97:0/2828260446 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdec007b910 con 0x7fdec815dfd0 2024-03-06T07:34:26.167 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:34:26.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdea806b4d0 msgr2=0x7fdea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdea806b4d0 0x7fdea806d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8005880 tx=0x7fdeb8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 msgr2=0x7fdec815c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 0x7fdec815c6e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009d90 tx=0x7fdec0004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 shutdown_connections 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdea806b4d0 0x7fdea806d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec815dfd0 0x7fdec815c6e0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdec80affe0 0x7fdec815c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 --2- 172.21.15.97:0/2828260446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdec806b6d0 0x7fdec81f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 >> 172.21.15.97:0/2828260446 conn(0x7fdec806c0d0 msgr2=0x7fdec8159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.167+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 shutdown_connections 2024-03-06T07:34:26.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:26.168+0000 7fdec7e00640 1 -- 172.21.15.97:0/2828260446 wait complete. 2024-03-06T07:34:26.188 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:34:26.188 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:34:26.188 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:34:26.188 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:34:26.188 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:26.188 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:34:26.189 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:34:26.189 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:34:26.189 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:26.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.189 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:34:26.235 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:26.236 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42209sec preferred_lft 42209sec 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:26.237 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:26.238 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:34:26.238 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:26.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.238 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:34:26.320 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:34:26.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.327 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:34:26.377 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:34:26.377 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:34:26.377 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:34:26.377 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:34:26.377 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:34:26.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.378 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:34:26.378 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:34:26.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:34:26.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:34:26.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:34:26.379 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:34:26.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.546 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:34:26.592 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:26.592 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:26.593 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:26.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:34:26.652 INFO:tasks.ceph.mds.c.smithi097.stderr:2024-03-06T07:34:26.650+0000 7f7bbce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:26.691 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:26.691 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:26.743 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:26.743 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:26.743 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:26.787 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:34:26.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:26.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:26.964 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:26.965 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:26.996 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:26.996 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:26.996 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:34:26.996 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:34:26.996 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:34:26.998 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:34:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:34:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:34:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:34:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:34:27.004 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:34:27.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.084 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:34:27.128 INFO:teuthology.orchestra.run.smithi097.stdout:client6016 2024-03-06T07:34:27.128 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:34:27.128 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:27.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.219 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:34:27.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:27.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.463+0000 7fbad1800640 1 Processor -- start 2024-03-06T07:34:27.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.463+0000 7fbad1800640 1 -- start start 2024-03-06T07:34:27.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbad1800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbacc0cec40 0x7fbacc15d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbad1800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbad1800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc06efd0 con 0x7fbacc15de00 2024-03-06T07:34:27.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbad1800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbacc06fc20 con 0x7fbacc0cec40 2024-03-06T07:34:27.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbad1800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc06f2b0 con 0x7fbacc06daf0 2024-03-06T07:34:27.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbacaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbacc0cec40 0x7fbacc15d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbaca000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fbacc15de00 0x7fbacc161ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57742/0 (socket says 172.21.15.97:57742) 2024-03-06T07:34:27.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.466+0000 7fbaca000640 1 -- 172.21.15.97:0/2774511870 learned_addr learned my addr 172.21.15.97:0/2774511870 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.467+0000 7fbacb400640 1 --2- 172.21.15.97:0/2774511870 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.467+0000 7fbacaa00640 1 -- 172.21.15.97:0/2774511870 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 msgr2=0x7fbacc15e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.467+0000 7fbacaa00640 1 --2- 172.21.15.97:0/2774511870 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc15e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.467+0000 7fbacaa00640 1 -- 172.21.15.97:0/2774511870 >> v1:172.21.15.97:6789/0 conn(0x7fbacc15de00 legacy=0x7fbacc161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.467+0000 7fbac9600640 1 -- 172.21.15.97:0/2774511870 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898321986 0 0) 0x7fbacc06efd0 con 0x7fbacc15de00 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.467+0000 7fbacaa00640 1 -- 172.21.15.97:0/2774511870 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc06f8b0 con 0x7fbacc0cec40 2024-03-06T07:34:27.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.468+0000 7fbacaa00640 1 --2- 172.21.15.97:0/2774511870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc0cec40 0x7fbacc15d8c0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fbabc0097a0 tx=0x7fbabc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2f028f87e8374e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.468+0000 7fbac9600640 1 -- 172.21.15.97:0/2774511870 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc035020 con 0x7fbacc0cec40 2024-03-06T07:34:27.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.468+0000 7fbac9600640 1 -- 172.21.15.97:0/2774511870 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc031830 con 0x7fbacc0cec40 2024-03-06T07:34:27.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbac9600640 1 -- 172.21.15.97:0/2774511870 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc031c00 con 0x7fbacc0cec40 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 -- 172.21.15.97:0/2774511870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc0cec40 msgr2=0x7fbacc15d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 --2- 172.21.15.97:0/2774511870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc0cec40 0x7fbacc15d8c0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fbabc0097a0 tx=0x7fbabc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 -- 172.21.15.97:0/2774511870 shutdown_connections 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 --2- 172.21.15.97:0/2774511870 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc15e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 --2- 172.21.15.97:0/2774511870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc0cec40 0x7fbacc15d8c0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 -- 172.21.15.97:0/2774511870 >> 172.21.15.97:0/2774511870 conn(0x7fbacc0d1c50 msgr2=0x7fbacc0d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 -- 172.21.15.97:0/2774511870 shutdown_connections 2024-03-06T07:34:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.469+0000 7fbad1800640 1 -- 172.21.15.97:0/2774511870 wait complete. 2024-03-06T07:34:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.472+0000 7fbad1800640 1 Processor -- start 2024-03-06T07:34:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.472+0000 7fbad1800640 1 -- start start 2024-03-06T07:34:27.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbad1800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc1e9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbad1800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbacc0cec40 0x7fbacc0b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbad1800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 0x7fbacc0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbad1800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc1623b0 con 0x7fbacc15de00 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbad1800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0d19e0 con 0x7fbacc0cec40 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbad1800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc162530 con 0x7fbacc06daf0 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbacb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 0x7fbacc0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbacb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 0x7fbacc0b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49512/0 (socket says 172.21.15.97:49512) 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbacb400640 1 -- 172.21.15.97:0/2322798021 learned_addr learned my addr 172.21.15.97:0/2322798021 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.474+0000 7fbacaa00640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc1e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbaca000640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbacc0cec40 0x7fbacc0b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbacb400640 1 -- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 msgr2=0x7fbacc1e9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbacb400640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc1e9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbacb400640 1 -- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbacc0cec40 msgr2=0x7fbacc0b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbacb400640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbacc0cec40 0x7fbacc0b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbacb400640 1 -- 172.21.15.97:0/2322798021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0b41d0 con 0x7fbacc15de00 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbaca000640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbacc0cec40 0x7fbacc0b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbacb400640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 0x7fbacc0b33b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fbac4007e60 tx=0x7fbac400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4014620 con 0x7fbacc15de00 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4019070 con 0x7fbacc15de00 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc1e87b0 con 0x7fbacc15de00 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4014a00 con 0x7fbacc15de00 2024-03-06T07:34:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.475+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc1e8d70 con 0x7fbacc15de00 2024-03-06T07:34:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.477+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc162b40 con 0x7fbacc15de00 2024-03-06T07:34:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.477+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbac4014ba0 con 0x7fbacc15de00 2024-03-06T07:34:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.479+0000 7fbac2a00640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbaac06b500 0x7fbaac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.479+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbac408ab80 con 0x7fbacc15de00 2024-03-06T07:34:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.479+0000 7fbacaa00640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbaac06b500 0x7fbaac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.480+0000 7fbacaa00640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbaac06b500 0x7fbaac06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbabc036000 tx=0x7fbabc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.483+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbacc162b40 con 0x7fbacc15de00 2024-03-06T07:34:27.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.720+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbacc04f330 con 0x7fbacc15de00 2024-03-06T07:34:27.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.721+0000 7fbac2a00640 1 -- 172.21.15.97:0/2322798021 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fbacc0b41d0 con 0x7fbacc15de00 2024-03-06T07:34:27.724 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:27.724 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:34:27.724 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:34:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbaac06b500 msgr2=0x7fbaac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbaac06b500 0x7fbaac06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbabc036000 tx=0x7fbabc008030 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 msgr2=0x7fbacc0b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 0x7fbacc0b33b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fbac4007e60 tx=0x7fbac400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 shutdown_connections 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbaac06b500 0x7fbaac06d9c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbacc15de00 0x7fbacc0b33b0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbacc0cec40 0x7fbacc0b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 --2- 172.21.15.97:0/2322798021 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbacc06daf0 0x7fbacc1e9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 >> 172.21.15.97:0/2322798021 conn(0x7fbacc0d1c50 msgr2=0x7fbacc0cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 shutdown_connections 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:27.724+0000 7fbad1800640 1 -- 172.21.15.97:0/2322798021 wait complete. 2024-03-06T07:34:27.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:34:27.971 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:34:27.971 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:34:27.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.173+0000 7f2743200640 1 Processor -- start 2024-03-06T07:34:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.173+0000 7f2743200640 1 -- start start 2024-03-06T07:34:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.174+0000 7f2743200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f273c15f9d0 0x7f273c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.174+0000 7f2743200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273c06cef0 con 0x7f273c070b10 2024-03-06T07:34:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.174+0000 7f2743200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273c06d1d0 con 0x7f273c144c00 2024-03-06T07:34:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.174+0000 7f2743200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273c06d4b0 con 0x7f273c15f9d0 2024-03-06T07:34:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.174+0000 7f2741200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f273c070b10 0x7f273c070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57754/0 (socket says 172.21.15.97:57754) 2024-03-06T07:34:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.174+0000 7f2741200640 1 -- 172.21.15.97:0/3949287755 learned_addr learned my addr 172.21.15.97:0/3949287755 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f2733400640 1 --2- 172.21.15.97:0/3949287755 >> v2:172.21.15.135:3300/0 conn(0x7f273c15f9d0 0x7f273c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002356202 0 0) 0x7f273c06cef0 con 0x7f273c070b10 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2720003520 con 0x7f273c070b10 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 575229717 0 0) 0x7f2720003520 con 0x7f273c070b10 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 >> v2:172.21.15.135:3300/0 conn(0x7f273c15f9d0 msgr2=0x7f273c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 --2- 172.21.15.97:0/3949287755 >> v2:172.21.15.135:3300/0 conn(0x7f273c15f9d0 0x7f273c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 >> v1:172.21.15.135:6789/0 conn(0x7f273c144c00 legacy=0x7f273c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c06d790 con 0x7f273c070b10 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2734001670 con 0x7f273c070b10 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 >> v1:172.21.15.97:6789/0 conn(0x7f273c070b10 legacy=0x7f273c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27200039b0 0x7f2720003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f2720004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c06d4b0 con 0x7f2720004800 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272000c060 con 0x7f27200051c0 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.175+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272000c260 con 0x7f27200039b0 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2733400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f2720004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2741200640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f273be00640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27200039b0 0x7f2720003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2733400640 1 -- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27200039b0 msgr2=0x7f2720003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2733400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27200039b0 0x7f2720003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2733400640 1 -- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27200051c0 msgr2=0x7f272000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2733400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2733400640 1 -- 172.21.15.97:0/3949287755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2720003520 con 0x7f2720004800 2024-03-06T07:34:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.176+0000 7f2741200640 1 -- 172.21.15.97:0/3949287755 reap_dead start 2024-03-06T07:34:28.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.178+0000 7f2733400640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f2720004c80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f2724009ac0 tx=0x7f2720005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a888926c9ac7aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.178+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724039070 con 0x7f2720004800 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.178+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272402c6b0 con 0x7f2720004800 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.178+0000 7f273b400640 1 -- 172.21.15.97:0/3949287755 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724009690 con 0x7f2720004800 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 -- 172.21.15.97:0/3949287755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 msgr2=0x7f2720004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f2720004c80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f2724009ac0 tx=0x7f2720005830 comp rx=0 tx=0).stop 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 -- 172.21.15.97:0/3949287755 shutdown_connections 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 --2- 172.21.15.97:0/3949287755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f2720004c80 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 -- 172.21.15.97:0/3949287755 >> 172.21.15.97:0/3949287755 conn(0x7f273c0b1730 msgr2=0x7f273c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 -- 172.21.15.97:0/3949287755 shutdown_connections 2024-03-06T07:34:28.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.179+0000 7f2743200640 1 -- 172.21.15.97:0/3949287755 wait complete. 2024-03-06T07:34:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.181+0000 7f2743200640 1 Processor -- start 2024-03-06T07:34:28.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.181+0000 7f2743200640 1 -- start start 2024-03-06T07:34:28.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.182+0000 7f2743200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f273c15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.182+0000 7f2743200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273c143100 0x7f273c13c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.182+0000 7f2743200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f273c140aa0 0x7f273c140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.182+0000 7f2743200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c06ccd0 con 0x7f2720004800 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.182+0000 7f2743200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c06cb50 con 0x7f273c140aa0 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.182+0000 7f2743200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c06df90 con 0x7f273c143100 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f273c15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f273c15dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49536/0 (socket says 172.21.15.97:49536) 2024-03-06T07:34:28.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 -- 172.21.15.97:0/480165784 learned_addr learned my addr 172.21.15.97:0/480165784 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 -- 172.21.15.97:0/480165784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273c143100 msgr2=0x7f273c13c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273c143100 0x7f273c13c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 -- 172.21.15.97:0/480165784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f273c140aa0 msgr2=0x7f273c140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f273c140aa0 0x7f273c140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 -- 172.21.15.97:0/480165784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c15e3f0 con 0x7f2720004800 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f273be00640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f273c15dd50 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e810 tx=0x7f272c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.183+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c003b80 con 0x7f2720004800 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.184+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c018070 con 0x7f2720004800 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.184+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c15e6d0 con 0x7f2720004800 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.184+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0050d0 con 0x7f2720004800 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.184+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273c15ec90 con 0x7f2720004800 2024-03-06T07:34:28.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.185+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273c04f330 con 0x7f2720004800 2024-03-06T07:34:28.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.186+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f272c005270 con 0x7f2720004800 2024-03-06T07:34:28.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.187+0000 7f2738c00640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271806b4d0 0x7f271806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.187+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f272c08a110 con 0x7f2720004800 2024-03-06T07:34:28.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.187+0000 7f2733400640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271806b4d0 0x7f271806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.187+0000 7f2733400640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271806b4d0 0x7f271806d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2724036000 tx=0x7f272402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.191+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f272c058fc0 con 0x7f2720004800 2024-03-06T07:34:28.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.425+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f273c0ad860 con 0x7f2720004800 2024-03-06T07:34:28.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.426+0000 7f2738c00640 1 -- 172.21.15.97:0/480165784 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f272c0588c0 con 0x7f2720004800 2024-03-06T07:34:28.428 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:28.428 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:34:28.428 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:34:28.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271806b4d0 msgr2=0x7f271806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271806b4d0 0x7f271806d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2724036000 tx=0x7f272402e000 comp rx=0 tx=0).stop 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 msgr2=0x7f273c15dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f273c15dd50 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e810 tx=0x7f272c00ecd0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 shutdown_connections 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271806b4d0 0x7f271806d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f273c140aa0 0x7f273c140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273c143100 0x7f273c13c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 --2- 172.21.15.97:0/480165784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2720004800 0x7f273c15dd50 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 >> 172.21.15.97:0/480165784 conn(0x7f273c0b1730 msgr2=0x7f273c06e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 shutdown_connections 2024-03-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:28.428+0000 7f2743200640 1 -- 172.21.15.97:0/480165784 wait complete. 2024-03-06T07:34:28.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:34:28.718 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:34:28.718 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:34:38.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:38.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:58.720 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:34:58.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.823+0000 7fe9c2635f40 1 Processor -- start 2024-03-06T07:34:58.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.823+0000 7fe9c2635f40 1 -- start start 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9c2635f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 0x55f542fe8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9c2635f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9c2635f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542fcf890 0x55f542fcfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9c2635f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f542eb5fd0 con 0x55f542feb200 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9c2635f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f542f11b70 con 0x55f542fcf890 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9c2635f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f542fec640 con 0x55f542feb5e0 2024-03-06T07:34:58.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 0x55f542fe8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 0x55f542fe8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39758/0 (socket says 172.21.15.97:39758) 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 -- 172.21.15.97:0/1715985040 learned_addr learned my addr 172.21.15.97:0/1715985040 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 -- 172.21.15.97:0/1715985040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 msgr2=0x55f542fe9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 -- 172.21.15.97:0/1715985040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542fcf890 msgr2=0x55f542fcfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542fcf890 0x55f542fcfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.825+0000 7fe9bf000640 1 -- 172.21.15.97:0/1715985040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f542fea9e0 con 0x55f542feb200 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.826+0000 7fe9bf000640 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 0x55f542fe8f20 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000f770 tx=0x7fe9b000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33eee594428a12ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.826+0000 7fe9bd200640 1 -- 172.21.15.97:0/1715985040 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000dad0 con 0x55f542feb200 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.826+0000 7fe9bd200640 1 -- 172.21.15.97:0/1715985040 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b000dc90 con 0x55f542feb200 2024-03-06T07:34:58.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.826+0000 7fe9bd200640 1 -- 172.21.15.97:0/1715985040 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0019400 con 0x55f542feb200 2024-03-06T07:34:58.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 -- 172.21.15.97:0/1715985040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 msgr2=0x55f542fe8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 0x55f542fe8f20 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000f770 tx=0x7fe9b000fc30 comp rx=0 tx=0).stop 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 -- 172.21.15.97:0/1715985040 shutdown_connections 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542fcf890 0x55f542fcfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/1715985040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542feb200 0x55f542fe8f20 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 -- 172.21.15.97:0/1715985040 >> 172.21.15.97:0/1715985040 conn(0x55f542f3cbe0 msgr2=0x55f542fea2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 -- 172.21.15.97:0/1715985040 shutdown_connections 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.827+0000 7fe9c2635f40 1 -- 172.21.15.97:0/1715985040 wait complete. 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.828+0000 7fe9c2635f40 1 Processor -- start 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.828+0000 7fe9c2635f40 1 -- start start 2024-03-06T07:34:58.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9c2635f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 0x55f542fc6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9c2635f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542feb200 0x55f542fe9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9c2635f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9c2635f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f542fbfad0 con 0x55f542fcf890 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9c2635f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f542feac80 con 0x55f542feb200 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9c2635f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f542fec2d0 con 0x55f542feb5e0 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 0x55f542fc6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 0x55f542fc6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39764/0 (socket says 172.21.15.97:39764) 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 learned_addr learned my addr 172.21.15.97:0/3692380035 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542feb200 0x55f542fe9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 msgr2=0x55f542fe7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542feb200 msgr2=0x55f542fe9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f542feb200 0x55f542fe9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f542fc4670 con 0x55f542fcf890 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.829+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f542feb5e0 0x55f542fe7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.830+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 0x55f542fc6440 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000d740 tx=0x55f542f0d0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.830+0000 7fe9b7400640 1 -- 172.21.15.97:0/3692380035 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b001ad80 con 0x55f542fcf890 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.830+0000 7fe9b7400640 1 -- 172.21.15.97:0/3692380035 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b0024030 con 0x55f542fcf890 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.830+0000 7fe9b7400640 1 -- 172.21.15.97:0/3692380035 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0029660 con 0x55f542fcf890 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.830+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f542fcd330 con 0x55f542fcf890 2024-03-06T07:34:58.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.830+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f542fc4bd0 con 0x55f542fcf890 2024-03-06T07:34:58.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.832+0000 7fe9b7400640 1 -- 172.21.15.97:0/3692380035 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9b00283d0 con 0x55f542fcf890 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.833+0000 7fe9b7400640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99c069eb0 0x7fe99c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.833+0000 7fe9b7400640 1 -- 172.21.15.97:0/3692380035 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe9b008d1a0 con 0x55f542fcf890 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.833+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99c069eb0 0x7fe99c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.834+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55f542fe9650 0x55f542fc53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.834+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542fe9650 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.834+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99c069eb0 0x7fe99c06c370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009810 tx=0x7fe9a8002fb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.834+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55f542fe9650 0x55f542fc53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.835+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55f542fe9650 0x55f542fc53d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.836+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 1 ==== osd_op_reply(1 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.836+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fe9b8014040 0x7fe9b8014930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.836+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b8014040 2024-03-06T07:34:58.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.837+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fe9b8014040 0x7fe9b8014930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.837+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fe9b8014040 0x7fe9b8014930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.838+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 1 ==== osd_op_reply(2 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.838+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe9b0019020 0x7fe9b0060ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.838+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b0019020 2024-03-06T07:34:58.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.838+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe9b0019020 0x7fe9b0060ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.839+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe9b0019020 0x7fe9b0060ab0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.839+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 1 ==== osd_op_reply(3 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.839+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7fe9a40028a0 0x7fe9a805d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.839+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9a40028a0 2024-03-06T07:34:58.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.839+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7fe9a40028a0 0x7fe9a805d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.840+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7fe9a40028a0 0x7fe9a805d980 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.840+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== osd_op_reply(4 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.840+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542fe9650 2024-03-06T07:34:58.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.841+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 2 ==== osd_op_reply(5 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.841+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b0019020 2024-03-06T07:34:58.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.842+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 2 ==== osd_op_reply(6 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.842+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9a40028a0 2024-03-06T07:34:58.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.842+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 2 ==== osd_op_reply(7 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.842+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542fe9650 2024-03-06T07:34:58.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.843+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 3 ==== osd_op_reply(8 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.843+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9a40028a0 2024-03-06T07:34:58.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.843+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 3 ==== osd_op_reply(9 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.843+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b0019020 2024-03-06T07:34:58.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.844+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 3 ==== osd_op_reply(10 [pgnls start_epoch 118 out=49b] v112'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.844+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b0019020 2024-03-06T07:34:58.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.844+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 4 ==== osd_op_reply(11 [pgnls start_epoch 118 out=49b] v112'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.844+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9a40028a0 2024-03-06T07:34:58.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.845+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 4 ==== osd_op_reply(12 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.845+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542fe9650 2024-03-06T07:34:58.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.846+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 4 ==== osd_op_reply(13 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.846+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x55f542f109d0 0x7fe9b80138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.846+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542f109d0 2024-03-06T07:34:58.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.846+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x55f542f109d0 0x7fe9b80138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.847+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x55f542f109d0 0x7fe9b80138c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.848+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 1 ==== osd_op_reply(14 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.848+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b8014040 2024-03-06T07:34:58.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.849+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 2 ==== osd_op_reply(15 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.849+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9a40028a0 2024-03-06T07:34:58.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.849+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 5 ==== osd_op_reply(16 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.849+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542f109d0 2024-03-06T07:34:58.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.850+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 2 ==== osd_op_reply(17 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.850+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fe9b005bca0 0x7fe9b008ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.850+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b005bca0 2024-03-06T07:34:58.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.851+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fe9b005bca0 0x7fe9b008ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.852+0000 7fe9be600640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fe9b005bca0 0x7fe9b008ba60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.852+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.5 v2:172.21.15.135:6800/2374976391 1 ==== osd_op_reply(18 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b005bca0 2024-03-06T07:34:58.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.853+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b0019020 2024-03-06T07:34:58.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.853+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 5 ==== osd_op_reply(19 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.853+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542fe9650 2024-03-06T07:34:58.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.854+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 5 ==== osd_op_reply(20 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.854+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x55f542fe9650 2024-03-06T07:34:58.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.855+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 6 ==== osd_op_reply(21 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.855+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55f542fcce50 con 0x7fe9b0019020 2024-03-06T07:34:58.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.855+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 6 ==== osd_op_reply(22 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.855+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9a40028a0 2024-03-06T07:34:58.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.856+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 6 ==== osd_op_reply(23 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.856+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9b8014040 2024-03-06T07:34:58.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.856+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 3 ==== osd_op_reply(24 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.857+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fe9b0063cc0 0x7fe9b006f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.857+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.857+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fe9b0063cc0 0x7fe9b006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.858+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fe9b0063cc0 0x7fe9b006f290 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.859+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.6 v2:172.21.15.135:6824/3958261964 1 ==== osd_op_reply(25 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.859+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9a40028a0 2024-03-06T07:34:58.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.860+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 7 ==== osd_op_reply(26 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.860+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9b005bca0 2024-03-06T07:34:58.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.861+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.5 v2:172.21.15.135:6800/2374976391 2 ==== osd_op_reply(27 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b005bca0 2024-03-06T07:34:58.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.861+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9a40028a0 2024-03-06T07:34:58.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.861+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 8 ==== osd_op_reply(28 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.861+0000 7fe9bfa00640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x55f542f59330 0x7fe9b8022910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.861+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x55f542f59330 2024-03-06T07:34:58.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.862+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x55f542f59330 0x7fe9b8022910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.862+0000 7fe9bf000640 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x55f542f59330 0x7fe9b8022910 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.863+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.2 v2:172.21.15.97:6816/75411953 1 ==== osd_op_reply(29 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f59330 2024-03-06T07:34:58.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.863+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.864+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.6 v2:172.21.15.135:6824/3958261964 2 ==== osd_op_reply(30 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.864+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9a8079040 con 0x7fe9b8014040 2024-03-06T07:34:58.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.865+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 4 ==== osd_op_reply(31 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.865+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.866+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 3 ==== osd_op_reply(32 [pgnls start_epoch 118 out=49b] v112'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.866+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f59330 2024-03-06T07:34:58.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.866+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.2 v2:172.21.15.97:6816/75411953 2 ==== osd_op_reply(33 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f59330 2024-03-06T07:34:58.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.866+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9a40028a0 2024-03-06T07:34:58.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.867+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 9 ==== osd_op_reply(34 [pgnls start_epoch 118 out=49b] v112'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.867+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b0019020 2024-03-06T07:34:58.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.867+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 7 ==== osd_op_reply(35 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.867+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b8014040 2024-03-06T07:34:58.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.868+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 5 ==== osd_op_reply(36 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.868+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b0019020 2024-03-06T07:34:58.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.868+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 8 ==== osd_op_reply(37 [pgnls start_epoch 118 out=49b] v112'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.869+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.870+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 4 ==== osd_op_reply(38 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.870+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b005bca0 2024-03-06T07:34:58.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.871+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.5 v2:172.21.15.135:6800/2374976391 3 ==== osd_op_reply(39 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b005bca0 2024-03-06T07:34:58.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.871+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b005bca0 2024-03-06T07:34:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.872+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.5 v2:172.21.15.135:6800/2374976391 4 ==== osd_op_reply(40 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b005bca0 2024-03-06T07:34:58.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.872+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.873+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.6 v2:172.21.15.135:6824/3958261964 3 ==== osd_op_reply(41 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.873+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b8014040 2024-03-06T07:34:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.873+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 6 ==== osd_op_reply(42 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.873+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b005bca0 2024-03-06T07:34:58.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.874+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.5 v2:172.21.15.135:6800/2374976391 5 ==== osd_op_reply(43 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b005bca0 2024-03-06T07:34:58.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.874+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.876+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 5 ==== osd_op_reply(44 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.876+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542fe9650 2024-03-06T07:34:58.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.877+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 7 ==== osd_op_reply(45 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.877+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542fe9650 2024-03-06T07:34:58.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.877+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 8 ==== osd_op_reply(46 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.878+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542fe9650 2024-03-06T07:34:58.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.878+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 9 ==== osd_op_reply(47 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.878+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542fe9650 2024-03-06T07:34:58.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.879+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 10 ==== osd_op_reply(48 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.879+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.880+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 6 ==== osd_op_reply(49 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.880+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.881+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 7 ==== osd_op_reply(50 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.881+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b0019020 2024-03-06T07:34:58.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.882+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.3 v2:172.21.15.97:6800/3169996968 9 ==== osd_op_reply(51 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b0019020 2024-03-06T07:34:58.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.882+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542fe9650 2024-03-06T07:34:58.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.883+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.7 v2:172.21.15.135:6816/3893802268 11 ==== osd_op_reply(52 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x55f542fe9650 2024-03-06T07:34:58.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.883+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9a40028a0 2024-03-06T07:34:58.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.884+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 10 ==== osd_op_reply(53 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.884+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.885+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 8 ==== osd_op_reply(54 [pgnls start_epoch 118 out=49b] v112'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.885+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.886+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 9 ==== osd_op_reply(55 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.886+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f59330 2024-03-06T07:34:58.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.887+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.2 v2:172.21.15.97:6816/75411953 3 ==== osd_op_reply(56 [pgnls start_epoch 118 out=49b] v112'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f59330 2024-03-06T07:34:58.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.887+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9a40028a0 2024-03-06T07:34:58.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.887+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 11 ==== osd_op_reply(57 [pgnls start_epoch 118 out=81b] v112'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.887+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.888+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 10 ==== osd_op_reply(58 [pgnls start_epoch 118 out=49b] v112'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.888+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.889+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.6 v2:172.21.15.135:6824/3958261964 4 ==== osd_op_reply(59 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.889+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9a40028a0 2024-03-06T07:34:58.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.890+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.0 v2:172.21.15.97:6819/1306424436 12 ==== osd_op_reply(60 [pgnls start_epoch 118 out=49b] v112'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9a40028a0 2024-03-06T07:34:58.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.890+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b8014040 2024-03-06T07:34:58.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.891+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.1 v2:172.21.15.97:6808/2322813614 7 ==== osd_op_reply(61 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b005cd70 con 0x7fe9b8014040 2024-03-06T07:34:58.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.891+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x55f542f109d0 2024-03-06T07:34:58.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.892+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 <== osd.4 v2:172.21.15.135:6808/3314712387 11 ==== osd_op_reply(62 [pgnls start_epoch 118 out=81b] v112'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fe9b005cd70 con 0x55f542f109d0 2024-03-06T07:34:58.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.892+0000 7fe9bf000640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.894+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 <== osd.6 v2:172.21.15.135:6824/3958261964 5 ==== osd_op_reply(63 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9b8011040 con 0x7fe9b0063cc0 2024-03-06T07:34:58.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.894+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7fe9b007e040 con 0x7fe9b005bca0 2024-03-06T07:34:58.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.894+0000 7fe9be600640 1 -- 172.21.15.97:0/3692380035 <== osd.5 v2:172.21.15.135:6800/2374976391 6 ==== osd_op_reply(64 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe9a805a040 con 0x7fe9b005bca0 2024-03-06T07:34:58.897 INFO:teuthology.orchestra.run.smithi097.stdout:10000000001.00000000 2024-03-06T07:34:58.897 INFO:teuthology.orchestra.run.smithi097.stdout:10000000001.00000001 2024-03-06T07:34:58.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7fe9a40028a0 msgr2=0x7fe9a805d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7fe9a40028a0 0x7fe9a805d980 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fe9b8014040 msgr2=0x7fe9b8014930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7fe9b8014040 0x7fe9b8014930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x55f542f59330 msgr2=0x7fe9b8022910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x55f542f59330 0x7fe9b8022910 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe9b0019020 msgr2=0x7fe9b0060ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fe9b0019020 0x7fe9b0060ab0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x55f542f109d0 msgr2=0x7fe9b80138c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x55f542f109d0 0x7fe9b80138c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fe9b005bca0 msgr2=0x7fe9b008ba60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fe9b005bca0 0x7fe9b008ba60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9bfa00640 1 -- 172.21.15.97:0/3692380035 reap_dead start 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fe9b0063cc0 msgr2=0x7fe9b006f290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fe9b0063cc0 0x7fe9b006f290 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55f542fe9650 msgr2=0x55f542fc53d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55f542fe9650 0x55f542fc53d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99c069eb0 msgr2=0x7fe99c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99c069eb0 0x7fe99c06c370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009810 tx=0x7fe9a8002fb0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 msgr2=0x55f542fc6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 0x55f542fc6440 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000d740 tx=0x55f542f0d0b0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 shutdown_connections 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fe9b0063cc0 0x7fe9b006f290 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55f542fe9650 0x55f542fc53d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99c069eb0 0x7fe99c06c370 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 --2- 172.21.15.97:0/3692380035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f542fcf890 0x55f542fc6440 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.895+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 >> 172.21.15.97:0/3692380035 conn(0x55f542f3cbe0 msgr2=0x55f542fd2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.896+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 shutdown_connections 2024-03-06T07:34:58.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:58.896+0000 7fe9c2635f40 1 -- 172.21.15.97:0/3692380035 wait complete. 2024-03-06T07:34:58.903 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:34:58.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:59.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.102+0000 7fa3fa000640 1 Processor -- start 2024-03-06T07:34:59.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.102+0000 7fa3fa000640 1 -- start start 2024-03-06T07:34:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3fa000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa3f406b6d0 0x7fa3f40d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3fa000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f415f480 con 0x7fa3f406e7e0 2024-03-06T07:34:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3fa000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f415e900 con 0x7fa3f406d920 2024-03-06T07:34:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3fa000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f406fae0 con 0x7fa3f406b6d0 2024-03-06T07:34:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa3f406b6d0 0x7fa3f40d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa3f406b6d0 0x7fa3f40d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39770/0 (socket says 172.21.15.97:39770) 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3f3e00640 1 -- 172.21.15.97:0/811314183 learned_addr learned my addr 172.21.15.97:0/811314183 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.104+0000 7fa3f3400640 1 --1- 172.21.15.97:0/811314183 >> v1:172.21.15.97:6789/0 conn(0x7fa3f406e7e0 0x7fa3f406f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52320/0 (socket says 172.21.15.97:52320) 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.105+0000 7fa3f3e00640 1 -- 172.21.15.97:0/811314183 >> v1:172.21.15.135:6789/0 conn(0x7fa3f406d920 legacy=0x7fa3f406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.105+0000 7fa3f3e00640 1 -- 172.21.15.97:0/811314183 >> v1:172.21.15.97:6789/0 conn(0x7fa3f406e7e0 legacy=0x7fa3f406f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.105+0000 7fa3f3e00640 1 -- 172.21.15.97:0/811314183 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f415ebe0 con 0x7fa3f406b6d0 2024-03-06T07:34:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.105+0000 7fa3f2000640 1 -- 172.21.15.97:0/811314183 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009536595 0 0) 0x7fa3f415f480 con 0x7fa3f406e7e0 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.106+0000 7fa3f3e00640 1 --2- 172.21.15.97:0/811314183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406b6d0 0x7fa3f40d0d00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009e50 tx=0x7fa3ec02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b36e4f2553b666 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.106+0000 7fa3f2000640 1 -- 172.21.15.97:0/811314183 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec02cb20 con 0x7fa3f406b6d0 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.106+0000 7fa3f2000640 1 -- 172.21.15.97:0/811314183 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec02ccc0 con 0x7fa3f406b6d0 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.106+0000 7fa3f2000640 1 -- 172.21.15.97:0/811314183 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec034610 con 0x7fa3f406b6d0 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.106+0000 7fa3fa000640 1 -- 172.21.15.97:0/811314183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406b6d0 msgr2=0x7fa3f40d0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.106+0000 7fa3fa000640 1 --2- 172.21.15.97:0/811314183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406b6d0 0x7fa3f40d0d00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009e50 tx=0x7fa3ec02bf80 comp rx=0 tx=0).stop 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.107+0000 7fa3fa000640 1 -- 172.21.15.97:0/811314183 shutdown_connections 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.107+0000 7fa3fa000640 1 --2- 172.21.15.97:0/811314183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406b6d0 0x7fa3f40d0d00 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.107+0000 7fa3fa000640 1 -- 172.21.15.97:0/811314183 >> 172.21.15.97:0/811314183 conn(0x7fa3f406c0d0 msgr2=0x7fa3f406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.107+0000 7fa3fa000640 1 -- 172.21.15.97:0/811314183 shutdown_connections 2024-03-06T07:34:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.107+0000 7fa3fa000640 1 -- 172.21.15.97:0/811314183 wait complete. 2024-03-06T07:34:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.108+0000 7fa3fa000640 1 Processor -- start 2024-03-06T07:34:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.108+0000 7fa3fa000640 1 -- start start 2024-03-06T07:34:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.109+0000 7fa3fa000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3f406b6d0 0x7fa3f41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.109+0000 7fa3fa000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3f406d920 0x7fa3f414c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.109+0000 7fa3fa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 0x7fa3f414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.109+0000 7fa3fa000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f415f1a0 con 0x7fa3f406e7e0 2024-03-06T07:34:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.109+0000 7fa3fa000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f406ebc0 con 0x7fa3f406b6d0 2024-03-06T07:34:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.109+0000 7fa3fa000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f415e4f0 con 0x7fa3f406d920 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 0x7fa3f414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 0x7fa3f414b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39778/0 (socket says 172.21.15.97:39778) 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 -- 172.21.15.97:0/3917538385 learned_addr learned my addr 172.21.15.97:0/3917538385 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 -- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3f406d920 msgr2=0x7fa3f414c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3400640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3f406b6d0 0x7fa3f41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3f406d920 0x7fa3f414c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 -- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3f406b6d0 msgr2=0x7fa3f41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3f406b6d0 0x7fa3f41526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.110+0000 7fa3f3e00640 1 -- 172.21.15.97:0/3917538385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f414e2b0 con 0x7fa3f406e7e0 2024-03-06T07:34:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.111+0000 7fa3f3e00640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 0x7fa3f414b2f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec036040 tx=0x7fa3ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.111+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec043020 con 0x7fa3f406e7e0 2024-03-06T07:34:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.111+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec03e450 con 0x7fa3f406e7e0 2024-03-06T07:34:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.111+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec009510 con 0x7fa3f406e7e0 2024-03-06T07:34:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.111+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f414d2d0 con 0x7fa3f406e7e0 2024-03-06T07:34:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.111+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f414db10 con 0x7fa3f406e7e0 2024-03-06T07:34:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.112+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f414e480 con 0x7fa3f406e7e0 2024-03-06T07:34:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.113+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3ec046020 con 0x7fa3f406e7e0 2024-03-06T07:34:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.114+0000 7fa3eb400640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3d406b4d0 0x7fa3d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.114+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa3ec0acc10 con 0x7fa3f406e7e0 2024-03-06T07:34:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.114+0000 7fa3f3400640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3d406b4d0 0x7fa3d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.115+0000 7fa3f3400640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3d406b4d0 0x7fa3d406d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4000f60 tx=0x7fa3e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.119+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3ec04d030 con 0x7fa3f406e7e0 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.355+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3f404f330 con 0x7fa3f406e7e0 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.355+0000 7fa3eb400640 1 -- 172.21.15.97:0/3917538385 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa3f414d2d0 con 0x7fa3f406e7e0 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3d406b4d0 msgr2=0x7fa3d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3d406b4d0 0x7fa3d406d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4000f60 tx=0x7fa3e4009040 comp rx=0 tx=0).stop 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 msgr2=0x7fa3f414b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 0x7fa3f414b2f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec036040 tx=0x7fa3ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 shutdown_connections 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa3d406b4d0 0x7fa3d406d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3f406e7e0 0x7fa3f414b2f0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa3f406d920 0x7fa3f414c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 --2- 172.21.15.97:0/3917538385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3f406b6d0 0x7fa3f41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.358+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 >> 172.21.15.97:0/3917538385 conn(0x7fa3f406c0d0 msgr2=0x7fa3f4155620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.359+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 shutdown_connections 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.359+0000 7fa3fa000640 1 -- 172.21.15.97:0/3917538385 wait complete. 2024-03-06T07:34:59.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:34:59.607 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:34:59.608 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:34:59.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:59.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.801+0000 7f081b800640 1 Processor -- start 2024-03-06T07:34:59.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.801+0000 7f081b800640 1 -- start start 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.802+0000 7f081b800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0814144c00 0x7f081406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.802+0000 7f081b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081415fe60 0x7f08140ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.802+0000 7f081b800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081406ee70 con 0x7f081415f9d0 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.802+0000 7f081b800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081406cef0 con 0x7f0814144c00 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.802+0000 7f081b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081406d1d0 con 0x7f081415fe60 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0818e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0814144c00 0x7f081406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0813e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f081415f9d0 0x7f08140ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52326/0 (socket says 172.21.15.97:52326) 2024-03-06T07:34:59.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0813e00640 1 -- 172.21.15.97:0/3762121005 learned_addr learned my addr 172.21.15.97:0/3762121005 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:34:59.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0819800640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081415fe60 0x7f08140ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0818e00640 1 -- 172.21.15.97:0/3762121005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081415fe60 msgr2=0x7f08140ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:59.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0818e00640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081415fe60 0x7f08140ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0818e00640 1 -- 172.21.15.97:0/3762121005 >> v1:172.21.15.97:6789/0 conn(0x7f081415f9d0 legacy=0x7f08140ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0818e00640 1 -- 172.21.15.97:0/3762121005 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081406d4b0 con 0x7f0814144c00 2024-03-06T07:34:59.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.803+0000 7f0819800640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081415fe60 0x7f08140ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:59.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.804+0000 7f0818e00640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0814144c00 0x7f081406e930 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0804009d90 tx=0x7f080402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236d968bd1b78e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f0813400640 1 -- 172.21.15.97:0/3762121005 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804039070 con 0x7f0814144c00 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f0813400640 1 -- 172.21.15.97:0/3762121005 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080402cc60 con 0x7f0814144c00 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f0813400640 1 -- 172.21.15.97:0/3762121005 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804034660 con 0x7f0814144c00 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f081b800640 1 -- 172.21.15.97:0/3762121005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0814144c00 msgr2=0x7f081406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f081b800640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0814144c00 0x7f081406e930 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0804009d90 tx=0x7f080402beb0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f081b800640 1 -- 172.21.15.97:0/3762121005 shutdown_connections 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f081b800640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f081415fe60 0x7f08140ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f081b800640 1 --2- 172.21.15.97:0/3762121005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0814144c00 0x7f081406e930 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.805+0000 7f081b800640 1 -- 172.21.15.97:0/3762121005 >> 172.21.15.97:0/3762121005 conn(0x7f08140b1730 msgr2=0x7f08140b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.806+0000 7f081b800640 1 -- 172.21.15.97:0/3762121005 shutdown_connections 2024-03-06T07:34:59.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.806+0000 7f081b800640 1 -- 172.21.15.97:0/3762121005 wait complete. 2024-03-06T07:34:59.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.807+0000 7f081b800640 1 Processor -- start 2024-03-06T07:34:59.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.807+0000 7f081b800640 1 -- start start 2024-03-06T07:34:59.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f081b800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0814144c00 0x7f081415e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f081b800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 0x7f0814143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f081b800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f081415fe60 0x7f081413fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f081b800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814160730 con 0x7f081415f9d0 2024-03-06T07:34:59.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f081b800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814160290 con 0x7f081415fe60 2024-03-06T07:34:59.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f081b800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081406cb50 con 0x7f0814144c00 2024-03-06T07:34:59.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f0813e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 0x7f0814143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f0813e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 0x7f0814143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39800/0 (socket says 172.21.15.97:39800) 2024-03-06T07:34:59.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.808+0000 7f0813e00640 1 -- 172.21.15.97:0/208962878 learned_addr learned my addr 172.21.15.97:0/208962878 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0813e00640 1 -- 172.21.15.97:0/208962878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0814144c00 msgr2=0x7f081415e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0813e00640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0814144c00 0x7f081415e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0813e00640 1 -- 172.21.15.97:0/208962878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f081415fe60 msgr2=0x7f081413fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0813e00640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f081415fe60 0x7f081413fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0813e00640 1 -- 172.21.15.97:0/208962878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081415ece0 con 0x7f081415f9d0 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0813e00640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 0x7f0814143250 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00c430 tx=0x7f07fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc017070 con 0x7f081415f9d0 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc005ce0 con 0x7f081415f9d0 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc0072f0 con 0x7f081415f9d0 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.809+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081415d700 con 0x7f081415f9d0 2024-03-06T07:34:59.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.810+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081415dcc0 con 0x7f081415f9d0 2024-03-06T07:34:59.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.811+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081404f330 con 0x7f081415f9d0 2024-03-06T07:34:59.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.812+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07fc01a070 con 0x7f081415f9d0 2024-03-06T07:34:59.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.812+0000 7f0810c00640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f006b4d0 0x7f07f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.812+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f07fc089750 con 0x7f081415f9d0 2024-03-06T07:34:59.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.813+0000 7f0818e00640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f006b4d0 0x7f07f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.813+0000 7f0818e00640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f006b4d0 0x7f07f006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0804036000 tx=0x7f080402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:34:59.817+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f081406ee70 con 0x7f081415f9d0 2024-03-06T07:35:00.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.102+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0814072600 con 0x7f081415f9d0 2024-03-06T07:35:00.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.104+0000 7f0810c00640 1 -- 172.21.15.97:0/208962878 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f07fc0584a0 con 0x7f081415f9d0 2024-03-06T07:35:00.106 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:00.106 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:00.106 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f006b4d0 msgr2=0x7f07f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f006b4d0 0x7f07f006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0804036000 tx=0x7f080402e000 comp rx=0 tx=0).stop 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 msgr2=0x7f0814143250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 0x7f0814143250 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00c430 tx=0x7f07fc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 shutdown_connections 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f07f006b4d0 0x7f07f006d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f081415fe60 0x7f081413fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081415f9d0 0x7f0814143250 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 --2- 172.21.15.97:0/208962878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0814144c00 0x7f081415e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.106+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 >> 172.21.15.97:0/208962878 conn(0x7f08140b1730 msgr2=0x7f0814071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.107+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 shutdown_connections 2024-03-06T07:35:00.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.107+0000 7f081b800640 1 -- 172.21.15.97:0/208962878 wait complete. 2024-03-06T07:35:00.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:35:00.370 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:35:00.370 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:00.371 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:00.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.456+0000 7f8e5f849f40 1 Processor -- start 2024-03-06T07:35:00.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.456+0000 7f8e5f849f40 1 -- start start 2024-03-06T07:35:00.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5f849f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 0x5581759c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5f849f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581758f3d70 0x5581759af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5f849f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 0x5581759b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5f849f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581758f2fb0 con 0x5581759b17e0 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5f849f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581758f2d10 con 0x5581758f3990 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5f849f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581758f3250 con 0x5581758f3d70 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 0x5581759b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 0x5581759b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39810/0 (socket says 172.21.15.97:39810) 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 -- 172.21.15.97:0/4201271689 learned_addr learned my addr 172.21.15.97:0/4201271689 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 -- 172.21.15.97:0/4201271689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581758f3d70 msgr2=0x5581759af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581758f3d70 0x5581759af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 -- 172.21.15.97:0/4201271689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 msgr2=0x5581759c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 0x5581759c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.457+0000 7f8e5cc00640 1 -- 172.21.15.97:0/4201271689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55817581fad0 con 0x5581759b17e0 2024-03-06T07:35:00.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.458+0000 7f8e5cc00640 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 0x5581759b0f00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400b650 tx=0x7f8e5400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49d29488191c518 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.458+0000 7f8e5ae00640 1 -- 172.21.15.97:0/4201271689 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54014020 con 0x5581759b17e0 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.458+0000 7f8e5ae00640 1 -- 172.21.15.97:0/4201271689 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e54009930 con 0x5581759b17e0 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.458+0000 7f8e5ae00640 1 -- 172.21.15.97:0/4201271689 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54009d20 con 0x5581759b17e0 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.458+0000 7f8e5f849f40 1 -- 172.21.15.97:0/4201271689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 msgr2=0x5581759b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.458+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 0x5581759b0f00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400b650 tx=0x7f8e5400bb10 comp rx=0 tx=0).stop 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 -- 172.21.15.97:0/4201271689 shutdown_connections 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581759b17e0 0x5581759b0f00 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581758f3d70 0x5581759af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/4201271689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 0x5581759c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 -- 172.21.15.97:0/4201271689 >> 172.21.15.97:0/4201271689 conn(0x558175922c50 msgr2=0x5581759b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 -- 172.21.15.97:0/4201271689 shutdown_connections 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 -- 172.21.15.97:0/4201271689 wait complete. 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.459+0000 7f8e5f849f40 1 Processor -- start 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 -- start start 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 0x5581759c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581758f3d70 0x5581759aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581759b17e0 0x5581759aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x558175922380 con 0x5581758f3d70 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581759225f0 con 0x5581758f3990 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.460+0000 7f8e5f849f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581759c7b70 con 0x5581759b17e0 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581758f3d70 0x5581759aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581758f3d70 0x5581759aa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39818/0 (socket says 172.21.15.97:39818) 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 -- 172.21.15.97:0/1649667864 learned_addr learned my addr 172.21.15.97:0/1649667864 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 -- 172.21.15.97:0/1649667864 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581759b17e0 msgr2=0x5581759aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5581759b17e0 0x5581759aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 -- 172.21.15.97:0/1649667864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 msgr2=0x5581759c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e5c200640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 0x5581759c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581758f3990 0x5581759c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 -- 172.21.15.97:0/1649667864 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581759b3220 con 0x5581758f3d70 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.461+0000 7f8e53e00640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581758f3d70 0x5581759aa5f0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009da0 tx=0x7f8e44006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.462+0000 7f8e59a00640 1 -- 172.21.15.97:0/1649667864 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44017070 con 0x5581758f3d70 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.462+0000 7f8e59a00640 1 -- 172.21.15.97:0/1649667864 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e44005e30 con 0x5581758f3d70 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.462+0000 7f8e59a00640 1 -- 172.21.15.97:0/1649667864 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44006e60 con 0x5581758f3d70 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.462+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5581759206f0 con 0x5581758f3d70 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.462+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558175920cb0 con 0x5581758f3d70 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.464+0000 7f8e59a00640 1 -- 172.21.15.97:0/1649667864 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e4401a070 con 0x5581758f3d70 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.464+0000 7f8e59a00640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e38069eb0 0x7f8e3806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.465+0000 7f8e59a00640 1 -- 172.21.15.97:0/1649667864 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8e44087fa0 con 0x5581758f3d70 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.465+0000 7f8e5c200640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e38069eb0 0x7f8e3806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.465+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5581759a4b40 0x558175a63970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.465+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118) v9 -- 0x558175a63eb0 con 0x5581759a4b40 2024-03-06T07:35:00.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.465+0000 7f8e5cc00640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5581759a4b40 0x558175a63970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5c200640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e38069eb0 0x7f8e3806c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c054000 tx=0x7f8e4c04c000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5cc00640 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5581759a4b40 0x558175a63970 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5cc00640 1 -- 172.21.15.97:0/1649667864 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f8e54009ae0 con 0x5581759a4b40 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5581759a4b40 msgr2=0x558175a63970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5581759a4b40 0x558175a63970 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e38069eb0 msgr2=0x7f8e3806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e38069eb0 0x7f8e3806c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c054000 tx=0x7f8e4c04c000 comp rx=0 tx=0).stop 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581758f3d70 msgr2=0x5581759aa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5f849f40 1 --2- 172.21.15.97:0/1649667864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5581758f3d70 0x5581759aa5f0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009da0 tx=0x7f8e44006750 comp rx=0 tx=0).stop 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.466+0000 7f8e5cc00640 1 -- 172.21.15.97:0/1649667864 reap_dead start 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.467+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 shutdown_connections 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.467+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 >> 172.21.15.97:0/1649667864 conn(0x558175922c50 msgr2=0x5581759af090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.467+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 shutdown_connections 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.467+0000 7f8e5f849f40 1 -- 172.21.15.97:0/1649667864 wait complete. 2024-03-06T07:35:00.473 INFO:teuthology.orchestra.run.smithi097.stdout:] 10000000001 stray0 2024-03-06T07:35:00.473 DEBUG:teuthology.orchestra.run.smithi097:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:00.562 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:35:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 1099511627777, 2024-03-06T07:35:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "ancestors": [ 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1536, 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "10000000001", 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 13 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 256, 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "stray0", 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 5 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 17, 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "old_pools": [] 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:35:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:00.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:00.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.770+0000 7fa42b200640 1 Processor -- start 2024-03-06T07:35:00.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.770+0000 7fa42b200640 1 -- start start 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa42b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa4240af820 0x7fa4240b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa42b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4240b3da0 con 0x7fa4240b38f0 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa42b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42406f1f0 con 0x7fa42415e160 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa42b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42406b0f0 con 0x7fa4240af820 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa41b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa4240af820 0x7fa4240b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa429200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa4240b38f0 0x7fa42406b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52342/0 (socket says 172.21.15.97:52342) 2024-03-06T07:35:00.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa429200640 1 -- 172.21.15.97:0/1477591767 learned_addr learned my addr 172.21.15.97:0/1477591767 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:00.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa41b400640 1 -- 172.21.15.97:0/1477591767 >> v1:172.21.15.135:6789/0 conn(0x7fa42415e160 legacy=0x7fa424162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa41b400640 1 -- 172.21.15.97:0/1477591767 >> v1:172.21.15.97:6789/0 conn(0x7fa4240b38f0 legacy=0x7fa42406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.772+0000 7fa41b400640 1 -- 172.21.15.97:0/1477591767 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4240afce0 con 0x7fa4240af820 2024-03-06T07:35:00.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.774+0000 7fa41b400640 1 --2- 172.21.15.97:0/1477591767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240af820 0x7fa4240b33b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0097a0 tx=0x7fa40c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e03d80084e1451 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.774+0000 7fa423400640 1 -- 172.21.15.97:0/1477591767 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0315a0 con 0x7fa4240af820 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.774+0000 7fa423400640 1 -- 172.21.15.97:0/1477591767 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c037070 con 0x7fa4240af820 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.774+0000 7fa423400640 1 -- 172.21.15.97:0/1477591767 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c031960 con 0x7fa4240af820 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 -- 172.21.15.97:0/1477591767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240af820 msgr2=0x7fa4240b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 --2- 172.21.15.97:0/1477591767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240af820 0x7fa4240b33b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0097a0 tx=0x7fa40c02b840 comp rx=0 tx=0).stop 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 -- 172.21.15.97:0/1477591767 shutdown_connections 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 --2- 172.21.15.97:0/1477591767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240af820 0x7fa4240b33b0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 -- 172.21.15.97:0/1477591767 >> 172.21.15.97:0/1477591767 conn(0x7fa4240d0340 msgr2=0x7fa4240d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 -- 172.21.15.97:0/1477591767 shutdown_connections 2024-03-06T07:35:00.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.775+0000 7fa42b200640 1 -- 172.21.15.97:0/1477591767 wait complete. 2024-03-06T07:35:00.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.777+0000 7fa42b200640 1 Processor -- start 2024-03-06T07:35:00.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.777+0000 7fa42b200640 1 -- start start 2024-03-06T07:35:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa42b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4240af820 0x7fa424149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa42b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 0x7fa42414a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa42b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa42415e160 0x7fa4240ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa42b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42406ef10 con 0x7fa4240b38f0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa42b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240b5620 con 0x7fa4240af820 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa42b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42406bc90 con 0x7fa42415e160 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 0x7fa42414a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 0x7fa42414a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39828/0 (socket says 172.21.15.97:39828) 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 -- 172.21.15.97:0/1332965098 learned_addr learned my addr 172.21.15.97:0/1332965098 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa423e00640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4240af820 0x7fa424149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 -- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa42415e160 msgr2=0x7fa4240ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa42415e160 0x7fa4240ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 -- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4240af820 msgr2=0x7fa424149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4240af820 0x7fa424149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa41b400640 1 -- 172.21.15.97:0/1332965098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42414ad90 con 0x7fa4240b38f0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.778+0000 7fa423e00640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4240af820 0x7fa424149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.779+0000 7fa41b400640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 0x7fa42414a850 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa40c000c00 tx=0x7fa40c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.779+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c032840 con 0x7fa4240b38f0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.779+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c0329e0 con 0x7fa4240b38f0 2024-03-06T07:35:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.779+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c032dd0 con 0x7fa4240b38f0 2024-03-06T07:35:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.779+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42415fae0 con 0x7fa4240b38f0 2024-03-06T07:35:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.779+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4241600a0 con 0x7fa4240b38f0 2024-03-06T07:35:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.780+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa424074ae0 con 0x7fa4240b38f0 2024-03-06T07:35:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.781+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa40c04d3f0 con 0x7fa4240b38f0 2024-03-06T07:35:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.782+0000 7fa420c00640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa40006b4d0 0x7fa40006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.782+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa40c02f030 con 0x7fa4240b38f0 2024-03-06T07:35:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.782+0000 7fa423e00640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa40006b4d0 0x7fa40006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.783+0000 7fa423e00640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa40006b4d0 0x7fa40006d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa414007e90 tx=0x7fa4140073a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:00.790+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa40c040070 con 0x7fa4240b38f0 2024-03-06T07:35:01.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.025+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa42404f330 con 0x7fa4240b38f0 2024-03-06T07:35:01.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.026+0000 7fa420c00640 1 -- 172.21.15.97:0/1332965098 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa40c07ad30 con 0x7fa4240b38f0 2024-03-06T07:35:01.028 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:01.028 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:01.029 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.028+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa40006b4d0 msgr2=0x7fa40006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.028+0000 7fa42b200640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa40006b4d0 0x7fa40006d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa414007e90 tx=0x7fa4140073a0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.028+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 msgr2=0x7fa42414a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.028+0000 7fa42b200640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 0x7fa42414a850 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa40c000c00 tx=0x7fa40c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 shutdown_connections 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa40006b4d0 0x7fa40006d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa42415e160 0x7fa4240ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4240b38f0 0x7fa42414a850 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 --2- 172.21.15.97:0/1332965098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4240af820 0x7fa424149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 >> 172.21.15.97:0/1332965098 conn(0x7fa4240d0340 msgr2=0x7fa424157060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 shutdown_connections 2024-03-06T07:35:01.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.029+0000 7fa42b200640 1 -- 172.21.15.97:0/1332965098 wait complete. 2024-03-06T07:35:01.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:35:01.274 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:35:01.274 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:35:01.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:01.274 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:35:01.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:01.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.525+0000 7f7cf4e00640 1 Processor -- start 2024-03-06T07:35:01.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.525+0000 7f7cf4e00640 1 -- start start 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.526+0000 7f7cf4e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7cf00cf150 0x7f7cf00d16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7cf4e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7cf4e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf006ed80 con 0x7f7cf01448a0 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7cf4e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf006f990 con 0x7f7cf00cf150 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7cf4e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf006f060 con 0x7f7cf00b3ad0 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7cf00cf150 0x7f7cf00d16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7cf00cf150 0x7f7cf00d16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39842/0 (socket says 172.21.15.97:39842) 2024-03-06T07:35:01.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 -- 172.21.15.97:0/3412104123 learned_addr learned my addr 172.21.15.97:0/3412104123 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7cee800640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 -- 172.21.15.97:0/3412104123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 msgr2=0x7f7cf00b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf00b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 -- 172.21.15.97:0/3412104123 >> v1:172.21.15.135:6789/0 conn(0x7f7cf01448a0 legacy=0x7f7cf006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7ced400640 1 -- 172.21.15.97:0/3412104123 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf006f340 con 0x7f7cf00cf150 2024-03-06T07:35:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.527+0000 7f7cee800640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf00b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.529+0000 7f7ced400640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf00d16c0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1cd48c7719a32a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.529+0000 7f7ceca00640 1 -- 172.21.15.97:0/3412104123 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc035020 con 0x7f7cf00cf150 2024-03-06T07:35:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.529+0000 7f7ceca00640 1 -- 172.21.15.97:0/3412104123 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc031640 con 0x7f7cf00cf150 2024-03-06T07:35:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.529+0000 7f7ceca00640 1 -- 172.21.15.97:0/3412104123 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc02d040 con 0x7f7cf00cf150 2024-03-06T07:35:01.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 -- 172.21.15.97:0/3412104123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 msgr2=0x7f7cf00d16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf00d16c0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 -- 172.21.15.97:0/3412104123 shutdown_connections 2024-03-06T07:35:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf00b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/3412104123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf00d16c0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 -- 172.21.15.97:0/3412104123 >> 172.21.15.97:0/3412104123 conn(0x7f7cf00d0b40 msgr2=0x7f7cf00d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 -- 172.21.15.97:0/3412104123 shutdown_connections 2024-03-06T07:35:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.530+0000 7f7cf4e00640 1 -- 172.21.15.97:0/3412104123 wait complete. 2024-03-06T07:35:01.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.532+0000 7f7cf4e00640 1 Processor -- start 2024-03-06T07:35:01.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.532+0000 7f7cf4e00640 1 -- start start 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cf4e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cf4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf0158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cf4e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cf01448a0 0x7f7cf00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cf4e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00b4640 con 0x7f7cf00cf150 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cf4e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00b44c0 con 0x7f7cf01448a0 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cf4e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00b47c0 con 0x7f7cf00b3ad0 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7ced400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf0158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7ced400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf0158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39846/0 (socket says 172.21.15.97:39846) 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7ced400640 1 -- 172.21.15.97:0/2808582138 learned_addr learned my addr 172.21.15.97:0/2808582138 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cede00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.533+0000 7f7cee800640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cf01448a0 0x7f7cf00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ced400640 1 -- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 msgr2=0x7f7cf015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ced400640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ced400640 1 -- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cf01448a0 msgr2=0x7f7cf00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ced400640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cf01448a0 0x7f7cf00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ced400640 1 -- 172.21.15.97:0/2808582138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0159140 con 0x7f7cf00cf150 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7cee800640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cf01448a0 0x7f7cf00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ced400640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf0158c00 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc02b930 tx=0x7f7cdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc040070 con 0x7f7cf00cf150 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc039bf0 con 0x7f7cf00cf150 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc035040 con 0x7f7cf00cf150 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf00ca1c0 con 0x7f7cf00cf150 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.534+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf00ca780 con 0x7f7cf00cf150 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.536+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf004f330 con 0x7f7cf00cf150 2024-03-06T07:35:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.536+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cdc04b060 con 0x7f7cf00cf150 2024-03-06T07:35:01.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.537+0000 7f7ce6000640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cd006b4d0 0x7f7cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.537+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7cdc0abe90 con 0x7f7cf00cf150 2024-03-06T07:35:01.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.538+0000 7f7cede00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cd006b4d0 0x7f7cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.538+0000 7f7cede00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cd006b4d0 0x7f7cd006d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009f20 tx=0x7f7ce00094d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.542+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7cdc07abe0 con 0x7f7cf00cf150 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.776+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf00b50b0 con 0x7f7cf00cf150 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.777+0000 7f7ce6000640 1 -- 172.21.15.97:0/2808582138 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f7cdc07a4e0 con 0x7f7cf00cf150 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cd006b4d0 msgr2=0x7f7cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cd006b4d0 0x7f7cd006d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009f20 tx=0x7f7ce00094d0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 msgr2=0x7f7cf0158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf0158c00 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc02b930 tx=0x7f7cdc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 shutdown_connections 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7cd006b4d0 0x7f7cd006d990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7cf01448a0 0x7f7cf00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cf00cf150 0x7f7cf0158c00 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 --2- 172.21.15.97:0/2808582138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7cf00b3ad0 0x7f7cf015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.779+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 >> 172.21.15.97:0/2808582138 conn(0x7f7cf00d0b40 msgr2=0x7f7cf00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.780+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 shutdown_connections 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:01.780+0000 7f7cf4e00640 1 -- 172.21.15.97:0/2808582138 wait complete. 2024-03-06T07:35:01.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:02.041 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:35:02.041 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:35:02.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:02.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.234+0000 7f7448c00640 1 Processor -- start 2024-03-06T07:35:02.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.235+0000 7f7448c00640 1 -- start start 2024-03-06T07:35:02.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7448c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74440d2360 0x7f74440d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7448c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f74440b3ec0 0x7f74440b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7448c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7444161960 con 0x7f7444161580 2024-03-06T07:35:02.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7448c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74440d2ce0 con 0x7f74440b3ec0 2024-03-06T07:35:02.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7448c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74440cf6f0 con 0x7f74440d2360 2024-03-06T07:35:02.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7443400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7444161580 0x7f74440d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52348/0 (socket says 172.21.15.97:52348) 2024-03-06T07:35:02.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7443400640 1 -- 172.21.15.97:0/2582628331 learned_addr learned my addr 172.21.15.97:0/2582628331 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:02.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.237+0000 7f7443e00640 1 --2- 172.21.15.97:0/2582628331 >> v2:172.21.15.97:3300/0 conn(0x7f74440b3ec0 0x7f74440b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f743be00640 1 --2- 172.21.15.97:0/2582628331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74440d2360 0x7f74440d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f7442a00640 1 -- 172.21.15.97:0/2582628331 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275806065 0 0) 0x7f7444161960 con 0x7f7444161580 2024-03-06T07:35:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f7442a00640 1 -- 172.21.15.97:0/2582628331 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7428003520 con 0x7f7444161580 2024-03-06T07:35:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f7443e00640 1 -- 172.21.15.97:0/2582628331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74440d2360 msgr2=0x7f74440d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f7443e00640 1 --2- 172.21.15.97:0/2582628331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74440d2360 0x7f74440d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f7443e00640 1 -- 172.21.15.97:0/2582628331 >> v1:172.21.15.97:6789/0 conn(0x7f7444161580 legacy=0x7f74440d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.238+0000 7f7443e00640 1 -- 172.21.15.97:0/2582628331 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440cf9d0 con 0x7f74440b3ec0 2024-03-06T07:35:02.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.239+0000 7f7443e00640 1 --2- 172.21.15.97:0/2582628331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440b3ec0 0x7f74440b4350 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f743c009d90 tx=0x7f743c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00192400e4db7d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.239+0000 7f7442a00640 1 -- 172.21.15.97:0/2582628331 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c039070 con 0x7f74440b3ec0 2024-03-06T07:35:02.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.239+0000 7f7442a00640 1 -- 172.21.15.97:0/2582628331 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743c02cc60 con 0x7f74440b3ec0 2024-03-06T07:35:02.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.239+0000 7f7442a00640 1 -- 172.21.15.97:0/2582628331 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c034640 con 0x7f74440b3ec0 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 -- 172.21.15.97:0/2582628331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440b3ec0 msgr2=0x7f74440b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 --2- 172.21.15.97:0/2582628331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440b3ec0 0x7f74440b4350 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f743c009d90 tx=0x7f743c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 -- 172.21.15.97:0/2582628331 shutdown_connections 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 --2- 172.21.15.97:0/2582628331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440b3ec0 0x7f74440b4350 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 --2- 172.21.15.97:0/2582628331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74440d2360 0x7f74440d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 -- 172.21.15.97:0/2582628331 >> 172.21.15.97:0/2582628331 conn(0x7f74440d3d80 msgr2=0x7f74440d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 -- 172.21.15.97:0/2582628331 shutdown_connections 2024-03-06T07:35:02.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.240+0000 7f7448c00640 1 -- 172.21.15.97:0/2582628331 wait complete. 2024-03-06T07:35:02.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.242+0000 7f7448c00640 1 Processor -- start 2024-03-06T07:35:02.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.243+0000 7f7448c00640 1 -- start start 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.244+0000 7f7448c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74440b3ec0 0x7f74441f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.244+0000 7f7448c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 0x7f74440b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.244+0000 7f7448c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7444161580 0x7f74440cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.244+0000 7f7448c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440b39a0 con 0x7f74440d2360 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.244+0000 7f7448c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440b36c0 con 0x7f74440b3ec0 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.244+0000 7f7448c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74440cf3a0 con 0x7f7444161580 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 0x7f74440b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 0x7f74440b7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39854/0 (socket says 172.21.15.97:39854) 2024-03-06T07:35:02.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 -- 172.21.15.97:0/3959138173 learned_addr learned my addr 172.21.15.97:0/3959138173 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:02.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 -- 172.21.15.97:0/3959138173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7444161580 msgr2=0x7f74440cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7444161580 0x7f74440cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 -- 172.21.15.97:0/3959138173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74440b3ec0 msgr2=0x7f74441f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74440b3ec0 0x7f74441f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 -- 172.21.15.97:0/3959138173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440b8a50 con 0x7f74440d2360 2024-03-06T07:35:02.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743be00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 0x7f74440b7fe0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f742c00d620 tx=0x7f742c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.245+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00b620 con 0x7f74440d2360 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.246+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c019070 con 0x7f74440d2360 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.246+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74440b8d30 con 0x7f74440d2360 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.246+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00ba00 con 0x7f74440d2360 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.246+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74440b92f0 con 0x7f74440d2360 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.247+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744404f330 con 0x7f74440d2360 2024-03-06T07:35:02.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.248+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f742c00bba0 con 0x7f74440d2360 2024-03-06T07:35:02.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.248+0000 7f743b400640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f742006b4d0 0x7f742006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.249+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f742c08a770 con 0x7f74440d2360 2024-03-06T07:35:02.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.249+0000 7f7443400640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f742006b4d0 0x7f742006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.249+0000 7f7443400640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f742006b4d0 0x7f742006d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7434000f60 tx=0x7f74340078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.257+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f742c0594c0 con 0x7f74440d2360 2024-03-06T07:35:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.505+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74440d3b60 con 0x7f74440d2360 2024-03-06T07:35:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.506+0000 7f743b400640 1 -- 172.21.15.97:0/3959138173 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f742c058dc0 con 0x7f74440d2360 2024-03-06T07:35:02.508 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:02.508 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f742006b4d0 msgr2=0x7f742006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f742006b4d0 0x7f742006d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7434000f60 tx=0x7f74340078f0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 msgr2=0x7f74440b7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 0x7f74440b7fe0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f742c00d620 tx=0x7f742c00dae0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 shutdown_connections 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f742006b4d0 0x7f742006d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7444161580 0x7f74440cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74440d2360 0x7f74440b7fe0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 --2- 172.21.15.97:0/3959138173 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74440b3ec0 0x7f74441f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 >> 172.21.15.97:0/3959138173 conn(0x7f74440d3d80 msgr2=0x7f74441602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 shutdown_connections 2024-03-06T07:35:02.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.509+0000 7f7448c00640 1 -- 172.21.15.97:0/3959138173 wait complete. 2024-03-06T07:35:02.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:35:02.748 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:35:02.749 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:35:02.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.950+0000 7f79dd400640 1 Processor -- start 2024-03-06T07:35:02.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.950+0000 7f79dd400640 1 -- start start 2024-03-06T07:35:02.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.951+0000 7f79dd400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f79d8144c00 0x7f79d806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.951+0000 7f79dd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d815f9d0 0x7f79d815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.951+0000 7f79dd400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d806ee70 con 0x7f79d80ce0a0 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.951+0000 7f79dd400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d8160370 con 0x7f79d8144c00 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.951+0000 7f79dd400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d806cef0 con 0x7f79d815f9d0 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.952+0000 7f79d7200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f79d80ce0a0 0x7f79d80ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52364/0 (socket says 172.21.15.97:52364) 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.952+0000 7f79d7200640 1 -- 172.21.15.97:0/182884578 learned_addr learned my addr 172.21.15.97:0/182884578 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.952+0000 7f79cde00640 1 --2- 172.21.15.97:0/182884578 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d815f9d0 0x7f79d815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.952+0000 7f79d6800640 1 --2- 172.21.15.97:0/182884578 >> v2:172.21.15.135:3300/0 conn(0x7f79d8144c00 0x7f79d806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.952+0000 7f79d5e00640 1 -- 172.21.15.97:0/182884578 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846144597 0 0) 0x7f79d0001110 con 0x7f79d80ce0a0 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.952+0000 7f79d5e00640 1 -- 172.21.15.97:0/182884578 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc003520 con 0x7f79d80ce0a0 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.953+0000 7f79cde00640 1 -- 172.21.15.97:0/182884578 >> v2:172.21.15.135:3300/0 conn(0x7f79d8144c00 msgr2=0x7f79d806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.953+0000 7f79cde00640 1 --2- 172.21.15.97:0/182884578 >> v2:172.21.15.135:3300/0 conn(0x7f79d8144c00 0x7f79d806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.953+0000 7f79cde00640 1 -- 172.21.15.97:0/182884578 >> v1:172.21.15.97:6789/0 conn(0x7f79d80ce0a0 legacy=0x7f79d80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.953+0000 7f79d5e00640 1 -- 172.21.15.97:0/182884578 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754658740 0 0) 0x7f79bc003520 con 0x7f79d80ce0a0 2024-03-06T07:35:02.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.953+0000 7f79cde00640 1 -- 172.21.15.97:0/182884578 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d806d1d0 con 0x7f79d815f9d0 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.954+0000 7f79cde00640 1 --2- 172.21.15.97:0/182884578 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d815f9d0 0x7f79d815fe30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79c00097a0 tx=0x7f79c002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5965132090028b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79d5e00640 1 -- 172.21.15.97:0/182884578 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c0037070 con 0x7f79d815f9d0 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79d5e00640 1 -- 172.21.15.97:0/182884578 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c0030510 con 0x7f79d815f9d0 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79d5e00640 1 -- 172.21.15.97:0/182884578 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0001840 con 0x7f79d815f9d0 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 -- 172.21.15.97:0/182884578 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d815f9d0 msgr2=0x7f79d815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 --2- 172.21.15.97:0/182884578 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d815f9d0 0x7f79d815fe30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79c00097a0 tx=0x7f79c002b790 comp rx=0 tx=0).stop 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 -- 172.21.15.97:0/182884578 shutdown_connections 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 --2- 172.21.15.97:0/182884578 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d815f9d0 0x7f79d815fe30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 --2- 172.21.15.97:0/182884578 >> v2:172.21.15.135:3300/0 conn(0x7f79d8144c00 0x7f79d806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 -- 172.21.15.97:0/182884578 >> 172.21.15.97:0/182884578 conn(0x7f79d80b1730 msgr2=0x7f79d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 -- 172.21.15.97:0/182884578 shutdown_connections 2024-03-06T07:35:02.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.955+0000 7f79dd400640 1 -- 172.21.15.97:0/182884578 wait complete. 2024-03-06T07:35:02.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.957+0000 7f79dd400640 1 Processor -- start 2024-03-06T07:35:02.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.957+0000 7f79dd400640 1 -- start start 2024-03-06T07:35:02.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79dd400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d80ce0a0 0x7f79d815b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79dd400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79d8144c00 0x7f79d80afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79dd400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 0x7f79d80b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79dd400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8160730 con 0x7f79d815f9d0 2024-03-06T07:35:02.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79dd400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d806d790 con 0x7f79d8144c00 2024-03-06T07:35:02.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79dd400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d806cb50 con 0x7f79d80ce0a0 2024-03-06T07:35:02.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79d7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 0x7f79d80b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79d7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 0x7f79d80b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39856/0 (socket says 172.21.15.97:39856) 2024-03-06T07:35:02.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.958+0000 7f79d7200640 1 -- 172.21.15.97:0/3663963747 learned_addr learned my addr 172.21.15.97:0/3663963747 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:02.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79cde00640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79d8144c00 0x7f79d80afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d6800640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d80ce0a0 0x7f79d815b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d7200640 1 -- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d80ce0a0 msgr2=0x7f79d815b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d7200640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d80ce0a0 0x7f79d815b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d7200640 1 -- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79d8144c00 msgr2=0x7f79d80afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d7200640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79d8144c00 0x7f79d80afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d7200640 1 -- 172.21.15.97:0/3663963747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8140810 con 0x7f79d815f9d0 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79d7200640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 0x7f79d80b0240 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c500 tx=0x7f79d000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0005ce0 con 0x7f79d815f9d0 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0007070 con 0x7f79d815f9d0 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.959+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00074a0 con 0x7f79d815f9d0 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.960+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d813b210 con 0x7f79d815f9d0 2024-03-06T07:35:02.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.960+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d813b7d0 con 0x7f79d815f9d0 2024-03-06T07:35:02.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.961+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d806f3c0 con 0x7f79d815f9d0 2024-03-06T07:35:02.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.962+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79d001b400 con 0x7f79d815f9d0 2024-03-06T07:35:02.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.962+0000 7f79cf400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79b406b500 0x7f79b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.963+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f79d0089fe0 con 0x7f79d815f9d0 2024-03-06T07:35:02.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.963+0000 7f79d6800640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79b406b500 0x7f79b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.964+0000 7f79d6800640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79b406b500 0x7f79b406d9c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009fd0 tx=0x7f79c800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:02.967+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79d806f3c0 con 0x7f79d815f9d0 2024-03-06T07:35:03.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.203+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79d804f330 con 0x7f79d815f9d0 2024-03-06T07:35:03.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.204+0000 7f79cf400640 1 -- 172.21.15.97:0/3663963747 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f79d806ee70 con 0x7f79d815f9d0 2024-03-06T07:35:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:03.207 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:03.207 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.206+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79b406b500 msgr2=0x7f79b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.206+0000 7f79dd400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79b406b500 0x7f79b406d9c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009fd0 tx=0x7f79c800b040 comp rx=0 tx=0).stop 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 msgr2=0x7f79d80b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 0x7f79d80b0240 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c500 tx=0x7f79d000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 shutdown_connections 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79b406b500 0x7f79b406d9c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79d815f9d0 0x7f79d80b0240 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79d8144c00 0x7f79d80afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 --2- 172.21.15.97:0/3663963747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79d80ce0a0 0x7f79d815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 >> 172.21.15.97:0/3663963747 conn(0x7f79d80b1730 msgr2=0x7f79d80ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 shutdown_connections 2024-03-06T07:35:03.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.207+0000 7f79dd400640 1 -- 172.21.15.97:0/3663963747 wait complete. 2024-03-06T07:35:03.229 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:35:03.464 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:35:03.464 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:03.464 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:03.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.561+0000 7fee69845f40 1 Processor -- start 2024-03-06T07:35:03.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.562+0000 7fee69845f40 1 -- start start 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee69845f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 0x5639067d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee69845f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639067c9150 0x563906864b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee69845f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 0x563906865070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee69845f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563906744340 con 0x5639066fde40 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee69845f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639066a1fd0 con 0x5639067c9150 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee69845f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639067445e0 con 0x5639067c8d70 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee66c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 0x563906865070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.563+0000 7fee66c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 0x563906865070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39866/0 (socket says 172.21.15.97:39866) 2024-03-06T07:35:03.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 -- 172.21.15.97:0/1396041963 learned_addr learned my addr 172.21.15.97:0/1396041963 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee5fe00640 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639067c9150 0x563906864b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 -- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 msgr2=0x5639067d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 0x5639067d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 -- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639067c9150 msgr2=0x563906864b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639067c9150 0x563906864b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 -- 172.21.15.97:0/1396041963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639067cf770 con 0x5639066fde40 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee5fe00640 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639067c9150 0x563906864b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:03.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee66c00640 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 0x563906865070 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fee6001d0b0 tx=0x7fee60003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4329896eab5690ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee64e00640 1 -- 172.21.15.97:0/1396041963 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60005030 con 0x5639066fde40 2024-03-06T07:35:03.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.564+0000 7fee64e00640 1 -- 172.21.15.97:0/1396041963 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee60062070 con 0x5639066fde40 2024-03-06T07:35:03.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee64e00640 1 -- 172.21.15.97:0/1396041963 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600053f0 con 0x5639066fde40 2024-03-06T07:35:03.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 -- 172.21.15.97:0/1396041963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 msgr2=0x563906865070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 0x563906865070 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fee6001d0b0 tx=0x7fee60003330 comp rx=0 tx=0).stop 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 -- 172.21.15.97:0/1396041963 shutdown_connections 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639066fde40 0x563906865070 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639067c9150 0x563906864b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 --2- 172.21.15.97:0/1396041963 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 0x5639067d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 -- 172.21.15.97:0/1396041963 >> 172.21.15.97:0/1396041963 conn(0x563906728bf0 msgr2=0x563906729f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 -- 172.21.15.97:0/1396041963 shutdown_connections 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.565+0000 7fee69845f40 1 -- 172.21.15.97:0/1396041963 wait complete. 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.566+0000 7fee69845f40 1 Processor -- start 2024-03-06T07:35:03.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.566+0000 7fee69845f40 1 -- start start 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee69845f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639066fde40 0x5639066fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee69845f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 0x5639067bf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee69845f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639067c9150 0x5639067c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee69845f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563906724f50 con 0x5639067c9150 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee69845f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x563906729ad0 con 0x5639066fde40 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee69845f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639067d2a10 con 0x5639067c8d70 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee66c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639067c9150 0x5639067c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee66c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639067c9150 0x5639067c2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39868/0 (socket says 172.21.15.97:39868) 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.567+0000 7fee66c00640 1 -- 172.21.15.97:0/3113751986 learned_addr learned my addr 172.21.15.97:0/3113751986 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee66c00640 1 -- 172.21.15.97:0/3113751986 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 msgr2=0x5639067bf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee66c00640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5639067c8d70 0x5639067bf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee66c00640 1 -- 172.21.15.97:0/3113751986 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639066fde40 msgr2=0x5639066fcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee66c00640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5639066fde40 0x5639066fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee66c00640 1 -- 172.21.15.97:0/3113751986 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639067bf800 con 0x5639067c9150 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee66c00640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639067c9150 0x5639067c2000 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fee60044cf0 tx=0x7fee60005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee5ea00640 1 -- 172.21.15.97:0/3113751986 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60052df0 con 0x5639067c9150 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee5ea00640 1 -- 172.21.15.97:0/3113751986 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6006e070 con 0x5639067c9150 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5639066fca90 con 0x5639067c9150 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.568+0000 7fee5ea00640 1 -- 172.21.15.97:0/3113751986 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60005270 con 0x5639067c9150 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.569+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639067c3df0 con 0x5639067c9150 2024-03-06T07:35:03.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.570+0000 7fee5ea00640 1 -- 172.21.15.97:0/3113751986 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee60005410 con 0x5639067c9150 2024-03-06T07:35:03.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.571+0000 7fee5ea00640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee44069ee0 0x7fee4406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.571+0000 7fee5ea00640 1 -- 172.21.15.97:0/3113751986 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fee600cc690 con 0x5639067c9150 2024-03-06T07:35:03.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.571+0000 7fee66200640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee44069ee0 0x7fee4406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.572+0000 7fee69845f40 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x56390672a270 0x5639067c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.572+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118) v9 -- 0x5639067c6de0 con 0x56390672a270 2024-03-06T07:35:03.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.572+0000 7fee5fe00640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x56390672a270 0x5639067c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.572+0000 7fee66200640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee44069ee0 0x7fee4406c3a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fee58003130 tx=0x7fee58001170 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.572+0000 7fee5fe00640 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x56390672a270 0x5639067c68a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee5fe00640 1 -- 172.21.15.97:0/3113751986 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fee500096b0 con 0x56390672a270 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x56390672a270 msgr2=0x5639067c68a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x56390672a270 0x5639067c68a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee44069ee0 msgr2=0x7fee4406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee44069ee0 0x7fee4406c3a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fee58003130 tx=0x7fee58001170 comp rx=0 tx=0).stop 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639067c9150 msgr2=0x5639067c2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 --2- 172.21.15.97:0/3113751986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639067c9150 0x5639067c2000 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fee60044cf0 tx=0x7fee60005d80 comp rx=0 tx=0).stop 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee66c00640 1 -- 172.21.15.97:0/3113751986 reap_dead start 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 shutdown_connections 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 >> 172.21.15.97:0/3113751986 conn(0x563906728bf0 msgr2=0x5639067c9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 shutdown_connections 2024-03-06T07:35:03.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:03.573+0000 7fee69845f40 1 -- 172.21.15.97:0/3113751986 wait complete. 2024-03-06T07:35:03.579 INFO:teuthology.orchestra.run.smithi097.stdout:W�file_cdir_2# 2024-03-06T07:35:03.580 DEBUG:teuthology.orchestra.run.smithi097:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:03.668 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:35:03.668 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 1099511627777, 2024-03-06T07:35:03.668 INFO:teuthology.orchestra.run.smithi097.stdout: "ancestors": [ 2024-03-06T07:35:03.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:03.668 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1099511628278, 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "file_c", 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 17 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1, 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "dir_2", 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 35 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 17, 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: "old_pools": [] 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:35:03.669 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:03.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:03.673 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:35:03.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:03.726 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:35:03.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:04.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.003+0000 7f7204a00640 1 Processor -- start 2024-03-06T07:35:04.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.003+0000 7f7204a00640 1 -- start start 2024-03-06T07:35:04.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.005+0000 7f7204a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7200144c00 0x7f720006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.005+0000 7f7204a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720006ee70 con 0x7f720015fe60 2024-03-06T07:35:04.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.005+0000 7f7204a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720006cef0 con 0x7f720015f9d0 2024-03-06T07:35:04.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.005+0000 7f7204a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720006d1d0 con 0x7f7200144c00 2024-03-06T07:35:04.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.006+0000 7f71fe800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f720015fe60 0x7f72000ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60856/0 (socket says 172.21.15.97:60856) 2024-03-06T07:35:04.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.006+0000 7f71fe800640 1 -- 172.21.15.97:0/1764013149 learned_addr learned my addr 172.21.15.97:0/1764013149 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:04.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.006+0000 7f71fde00640 1 --2- 172.21.15.97:0/1764013149 >> v2:172.21.15.135:3300/0 conn(0x7f7200144c00 0x7f720006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.006+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067917575 0 0) 0x7f71f8001110 con 0x7f720015fe60 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.006+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e4003520 con 0x7f720015fe60 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032481733 0 0) 0x7f71e4003520 con 0x7f720015fe60 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 >> v2:172.21.15.135:3300/0 conn(0x7f7200144c00 msgr2=0x7f720006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 --2- 172.21.15.97:0/1764013149 >> v2:172.21.15.135:3300/0 conn(0x7f7200144c00 0x7f720006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 >> v1:172.21.15.135:6789/0 conn(0x7f720015f9d0 legacy=0x7f72000ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720006d4b0 con 0x7f720015fe60 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f71f8001840 con 0x7f720015fe60 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 >> v1:172.21.15.97:6789/0 conn(0x7f720015fe60 legacy=0x7f72000ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71e4003a00 0x7f71e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71e4004800 0x7f71e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e40051c0 0x7f71e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720006d1d0 con 0x7f71e4003a00 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e400c0c0 con 0x7f71e4004800 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e400c2c0 con 0x7f71e40051c0 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71fde00640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71e4003a00 0x7f71e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.007+0000 7f71f5400640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71e4004800 0x7f71e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fe800640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e40051c0 0x7f71e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fde00640 1 -- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e40051c0 msgr2=0x7f71e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fde00640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e40051c0 0x7f71e400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fde00640 1 -- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71e4004800 msgr2=0x7f71e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fde00640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71e4004800 0x7f71e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fde00640 1 -- 172.21.15.97:0/1764013149 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4003520 con 0x7f71e4003a00 2024-03-06T07:35:04.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.008+0000 7f71fe800640 1 -- 172.21.15.97:0/1764013149 reap_dead start 2024-03-06T07:35:04.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.009+0000 7f71fde00640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71e4003a00 0x7f71e4003e20 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009fc0 tx=0x7f71e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0cbadab54d263d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.009+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f0039070 con 0x7f71e4003a00 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.009+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f002cc20 con 0x7f71e4003a00 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.009+0000 7f71fd400640 1 -- 172.21.15.97:0/1764013149 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f0009c10 con 0x7f71e4003a00 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 -- 172.21.15.97:0/1764013149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71e4003a00 msgr2=0x7f71e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71e4003a00 0x7f71e4003e20 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009fc0 tx=0x7f71e4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 -- 172.21.15.97:0/1764013149 shutdown_connections 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 --2- 172.21.15.97:0/1764013149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71e4003a00 0x7f71e4003e20 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 -- 172.21.15.97:0/1764013149 >> 172.21.15.97:0/1764013149 conn(0x7f72000b1730 msgr2=0x7f72000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 -- 172.21.15.97:0/1764013149 shutdown_connections 2024-03-06T07:35:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.010+0000 7f7204a00640 1 -- 172.21.15.97:0/1764013149 wait complete. 2024-03-06T07:35:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.012+0000 7f7204a00640 1 Processor -- start 2024-03-06T07:35:04.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.012+0000 7f7204a00640 1 -- start start 2024-03-06T07:35:04.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f7204a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e4003a00 0x7f7200139020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f7204a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f720015dc70 0x7f720015e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f7204a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 0x7f72000b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f7204a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7200160730 con 0x7f72000b2cc0 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f7204a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7200160270 con 0x7f720015dc70 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f7204a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720006cb50 con 0x7f71e4003a00 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 0x7f72000b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 0x7f72000b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37688/0 (socket says 172.21.15.97:37688) 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 -- 172.21.15.97:0/2491000893 learned_addr learned my addr 172.21.15.97:0/2491000893 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71f5400640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f720015dc70 0x7f720015e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 -- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e4003a00 msgr2=0x7f7200139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e4003a00 0x7f7200139020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 -- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f720015dc70 msgr2=0x7f720015e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f720015dc70 0x7f720015e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.014+0000 7f71fe800640 1 -- 172.21.15.97:0/2491000893 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000b3a20 con 0x7f72000b2cc0 2024-03-06T07:35:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.015+0000 7f71fe800640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 0x7f72000b3140 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f72000d0ac0 tx=0x7f71f800d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.015+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8007440 con 0x7f72000b2cc0 2024-03-06T07:35:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.015+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f80075e0 con 0x7f72000b2cc0 2024-03-06T07:35:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.015+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8004a20 con 0x7f72000b2cc0 2024-03-06T07:35:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.015+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72000b3d00 con 0x7f72000b2cc0 2024-03-06T07:35:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.015+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72000b42c0 con 0x7f72000b2cc0 2024-03-06T07:35:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.016+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720004f330 con 0x7f72000b2cc0 2024-03-06T07:35:04.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.017+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71f8004bc0 con 0x7f72000b2cc0 2024-03-06T07:35:04.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.018+0000 7f71f6a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.018+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f71f8089940 con 0x7f72000b2cc0 2024-03-06T07:35:04.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.018+0000 7f71fde00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.019+0000 7f71fde00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f71dc06b4d0 0x7f71dc06d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f71f0036000 tx=0x7f71f002e070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.027+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f720006ee70 con 0x7f72000b2cc0 2024-03-06T07:35:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.260+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7200072890 con 0x7f72000b2cc0 2024-03-06T07:35:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.261+0000 7f71f6a00640 1 -- 172.21.15.97:0/2491000893 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f71f8058730 con 0x7f72000b2cc0 2024-03-06T07:35:04.263 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:04.264 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:04.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.263+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f71dc06b4d0 msgr2=0x7f71dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.263+0000 7f7204a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f71dc06b4d0 0x7f71dc06d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f71f0036000 tx=0x7f71f002e070 comp rx=0 tx=0).stop 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 msgr2=0x7f72000b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 0x7f72000b3140 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f72000d0ac0 tx=0x7f71f800d950 comp rx=0 tx=0).stop 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 shutdown_connections 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72000b2cc0 0x7f72000b3140 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f720015dc70 0x7f720015e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 --2- 172.21.15.97:0/2491000893 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f71e4003a00 0x7f7200139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 >> 172.21.15.97:0/2491000893 conn(0x7f72000b1730 msgr2=0x7f7200143470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 shutdown_connections 2024-03-06T07:35:04.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:04.264+0000 7f7204a00640 1 -- 172.21.15.97:0/2491000893 wait complete. 2024-03-06T07:35:04.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:04.508 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:35:04.508 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:35:04.508 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:35:08.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:08.736 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:09.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:09.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.720+0000 7fa54b000640 1 Processor -- start 2024-03-06T07:35:09.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.721+0000 7fa54b000640 1 -- start start 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.722+0000 7fa54b000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa5440cf370 0x7fa54406d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.722+0000 7fa54b000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5440b2b30 con 0x7fa54415a940 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.722+0000 7fa54b000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5440b2e10 con 0x7fa5440cc2a0 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.723+0000 7fa54b000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5440b28e0 con 0x7fa5440cf370 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.723+0000 7fa549c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa54415a940 0x7fa5440b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60860/0 (socket says 172.21.15.97:60860) 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.723+0000 7fa549c00640 1 -- 172.21.15.97:0/3270338227 learned_addr learned my addr 172.21.15.97:0/3270338227 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.723+0000 7fa549200640 1 --2- 172.21.15.97:0/3270338227 >> v2:172.21.15.135:3300/0 conn(0x7fa5440cf370 0x7fa54406d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.723+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66502701 0 0) 0x7fa5440b2b30 con 0x7fa54415a940 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.723+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa52c003520 con 0x7fa54415a940 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 476686233 0 0) 0x7fa52c003520 con 0x7fa54415a940 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 >> v2:172.21.15.135:3300/0 conn(0x7fa5440cf370 msgr2=0x7fa54406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 --2- 172.21.15.97:0/3270338227 >> v2:172.21.15.135:3300/0 conn(0x7fa5440cf370 0x7fa54406d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 >> v1:172.21.15.135:6789/0 conn(0x7fa5440cc2a0 legacy=0x7fa5440cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54406ca50 con 0x7fa54415a940 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa5380019f0 con 0x7fa54415a940 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 >> v1:172.21.15.97:6789/0 conn(0x7fa54415a940 legacy=0x7fa5440b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c003a00 0x7fa52c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c004800 0x7fa52c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa52c0051c0 0x7fa52c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440b28e0 con 0x7fa52c004800 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c00c0c0 con 0x7fa52c0051c0 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c00c2c0 con 0x7fa52c003a00 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.724+0000 7fa549200640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c004800 0x7fa52c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.725+0000 7fa549200640 1 -- 172.21.15.97:0/3270338227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c003a00 msgr2=0x7fa52c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.725+0000 7fa549200640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c003a00 0x7fa52c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.725+0000 7fa549200640 1 -- 172.21.15.97:0/3270338227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa52c0051c0 msgr2=0x7fa52c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.725+0000 7fa549200640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa52c0051c0 0x7fa52c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.725+0000 7fa549200640 1 -- 172.21.15.97:0/3270338227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c003520 con 0x7fa52c004800 2024-03-06T07:35:09.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.725+0000 7fa54a600640 1 -- 172.21.15.97:0/3270338227 reap_dead start 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.726+0000 7fa549200640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c004800 0x7fa52c004c80 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa530009ac0 tx=0x7fa52c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f542e9d823814e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.726+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530039070 con 0x7fa52c004800 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.726+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53002c710 con 0x7fa52c004800 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.726+0000 7fa543e00640 1 -- 172.21.15.97:0/3270338227 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530009690 con 0x7fa52c004800 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.726+0000 7fa54b000640 1 -- 172.21.15.97:0/3270338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c004800 msgr2=0x7fa52c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.726+0000 7fa54b000640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c004800 0x7fa52c004c80 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa530009ac0 tx=0x7fa52c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.727+0000 7fa54b000640 1 -- 172.21.15.97:0/3270338227 shutdown_connections 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.727+0000 7fa54b000640 1 --2- 172.21.15.97:0/3270338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c004800 0x7fa52c004c80 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.727+0000 7fa54b000640 1 -- 172.21.15.97:0/3270338227 >> 172.21.15.97:0/3270338227 conn(0x7fa54406afc0 msgr2=0x7fa54406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.727+0000 7fa54b000640 1 -- 172.21.15.97:0/3270338227 shutdown_connections 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.727+0000 7fa54b000640 1 -- 172.21.15.97:0/3270338227 wait complete. 2024-03-06T07:35:09.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.729+0000 7fa54b000640 1 Processor -- start 2024-03-06T07:35:09.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.729+0000 7fa54b000640 1 -- start start 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.730+0000 7fa54b000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c004800 0x7fa54415c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.730+0000 7fa54b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 0x7fa54414b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.730+0000 7fa54b000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5441598a0 0x7fa5440ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.730+0000 7fa54b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54415ad20 con 0x7fa544158240 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.730+0000 7fa54b000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440b30f0 con 0x7fa5441598a0 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.730+0000 7fa54b000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54415aea0 con 0x7fa52c004800 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 0x7fa54414b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 0x7fa54414b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37708/0 (socket says 172.21.15.97:37708) 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 -- 172.21.15.97:0/3552423577 learned_addr learned my addr 172.21.15.97:0/3552423577 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549c00640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c004800 0x7fa54415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa54a600640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5441598a0 0x7fa5440ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 -- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c004800 msgr2=0x7fa54415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c004800 0x7fa54415c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 -- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5441598a0 msgr2=0x7fa5440ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5441598a0 0x7fa5440ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 -- 172.21.15.97:0/3552423577 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54414b8b0 con 0x7fa544158240 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549c00640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c004800 0x7fa54415c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa54a600640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5441598a0 0x7fa5440ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.731+0000 7fa549200640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 0x7fa54414b370 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa530000c00 tx=0x7fa530004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.732+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530039050 con 0x7fa544158240 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.732+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530045070 con 0x7fa544158240 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.732+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300044e0 con 0x7fa544158240 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.732+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa544158800 con 0x7fa544158240 2024-03-06T07:35:09.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.732+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa544158dc0 con 0x7fa544158240 2024-03-06T07:35:09.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.733+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54404f330 con 0x7fa544158240 2024-03-06T07:35:09.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.734+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa530003a50 con 0x7fa544158240 2024-03-06T07:35:09.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.735+0000 7fa541600640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52406b500 0x7fa52406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.735+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa5300ac4d0 con 0x7fa544158240 2024-03-06T07:35:09.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.735+0000 7fa549c00640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52406b500 0x7fa52406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.736+0000 7fa549c00640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52406b500 0x7fa52406d9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa538001be0 tx=0x7fa538009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.740+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa54414b8b0 con 0x7fa544158240 2024-03-06T07:35:09.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.974+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa544155fe0 con 0x7fa544158240 2024-03-06T07:35:09.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.975+0000 7fa541600640 1 -- 172.21.15.97:0/3552423577 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa53007b2a0 con 0x7fa544158240 2024-03-06T07:35:09.978 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:09.978 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:09.978 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:09.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52406b500 msgr2=0x7fa52406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52406b500 0x7fa52406d9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa538001be0 tx=0x7fa538009040 comp rx=0 tx=0).stop 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 msgr2=0x7fa54414b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 0x7fa54414b370 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa530000c00 tx=0x7fa530004730 comp rx=0 tx=0).stop 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 shutdown_connections 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52406b500 0x7fa52406d9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5441598a0 0x7fa5440ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa544158240 0x7fa54414b370 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 --2- 172.21.15.97:0/3552423577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa52c004800 0x7fa54415c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 >> 172.21.15.97:0/3552423577 conn(0x7fa54406afc0 msgr2=0x7fa5440ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 shutdown_connections 2024-03-06T07:35:09.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:09.978+0000 7fa54b000640 1 -- 172.21.15.97:0/3552423577 wait complete. 2024-03-06T07:35:10.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:10.229 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:35:10.229 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:35:10.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:10.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.440+0000 7f2453200640 1 Processor -- start 2024-03-06T07:35:10.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.440+0000 7f2453200640 1 -- start start 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2453200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f245415a4b0 0x7f2454071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2453200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f24540cbb60 0x7f24540b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2453200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 0x7f24540cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2453200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24540ce680 con 0x7f24540cbb60 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2453200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24540cec40 con 0x7f245415a4b0 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2453200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245406c940 con 0x7f24540b42d0 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2451400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f24540cbb60 0x7f24540b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2451400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f24540cbb60 0x7f24540b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37710/0 (socket says 172.21.15.97:37710) 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.442+0000 7f2451400640 1 -- 172.21.15.97:0/2741159466 learned_addr learned my addr 172.21.15.97:0/2741159466 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451e00640 1 --2- 172.21.15.97:0/2741159466 >> v2:172.21.15.135:3300/0 conn(0x7f245415a4b0 0x7f2454071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451400640 1 -- 172.21.15.97:0/2741159466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 msgr2=0x7f24540cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451400640 1 --2- 172.21.15.97:0/2741159466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 0x7f24540cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451400640 1 -- 172.21.15.97:0/2741159466 >> v2:172.21.15.135:3300/0 conn(0x7f245415a4b0 msgr2=0x7f2454071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451400640 1 --2- 172.21.15.97:0/2741159466 >> v2:172.21.15.135:3300/0 conn(0x7f245415a4b0 0x7f2454071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451400640 1 -- 172.21.15.97:0/2741159466 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454071980 con 0x7f24540cbb60 2024-03-06T07:35:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.443+0000 7f2451e00640 1 --2- 172.21.15.97:0/2741159466 >> v2:172.21.15.135:3300/0 conn(0x7f245415a4b0 0x7f2454071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.444+0000 7f2451400640 1 --2- 172.21.15.97:0/2741159466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24540b3d90 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2454068400 tx=0x7f244002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693d1cd0316a7851 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.444+0000 7f2450a00640 1 -- 172.21.15.97:0/2741159466 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400317d0 con 0x7f24540cbb60 2024-03-06T07:35:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.444+0000 7f2450a00640 1 -- 172.21.15.97:0/2741159466 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2440037070 con 0x7f24540cbb60 2024-03-06T07:35:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2450a00640 1 -- 172.21.15.97:0/2741159466 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440031b90 con 0x7f24540cbb60 2024-03-06T07:35:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 -- 172.21.15.97:0/2741159466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 msgr2=0x7f24540b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 --2- 172.21.15.97:0/2741159466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24540b3d90 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2454068400 tx=0x7f244002b870 comp rx=0 tx=0).stop 2024-03-06T07:35:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 -- 172.21.15.97:0/2741159466 shutdown_connections 2024-03-06T07:35:10.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 --2- 172.21.15.97:0/2741159466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 0x7f24540cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 --2- 172.21.15.97:0/2741159466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24540b3d90 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 --2- 172.21.15.97:0/2741159466 >> v2:172.21.15.135:3300/0 conn(0x7f245415a4b0 0x7f2454071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 -- 172.21.15.97:0/2741159466 >> 172.21.15.97:0/2741159466 conn(0x7f245406afc0 msgr2=0x7f245406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 -- 172.21.15.97:0/2741159466 shutdown_connections 2024-03-06T07:35:10.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.445+0000 7f2453200640 1 -- 172.21.15.97:0/2741159466 wait complete. 2024-03-06T07:35:10.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.448+0000 7f2453200640 1 Processor -- start 2024-03-06T07:35:10.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.448+0000 7f2453200640 1 -- start start 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2453200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 0x7f2454154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2453200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24541561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2453200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f245415a4b0 0x7f245414f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2453200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24540b4760 con 0x7f24540cbb60 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2453200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24540ce8d0 con 0x7f245415a4b0 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2453200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24540cbfa0 con 0x7f24540b42d0 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24541561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24541561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37726/0 (socket says 172.21.15.97:37726) 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 -- 172.21.15.97:0/2743767576 learned_addr learned my addr 172.21.15.97:0/2743767576 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 -- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 msgr2=0x7f2454154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 0x7f2454154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 -- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f245415a4b0 msgr2=0x7f245414f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2452800640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f245415a4b0 0x7f245414f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f245415a4b0 0x7f245414f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.450+0000 7f2451400640 1 -- 172.21.15.97:0/2743767576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24541549d0 con 0x7f24540cbb60 2024-03-06T07:35:10.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f2452800640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f245415a4b0 0x7f245414f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:10.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f2451400640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24541561d0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2440000c00 tx=0x7f24400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440032870 con 0x7f24540cbb60 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2440032a10 con 0x7f24540cbb60 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2454153c20 con 0x7f24540cbb60 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440032e00 con 0x7f24540cbb60 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.451+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24541506a0 con 0x7f24540cbb60 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.453+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2454154140 con 0x7f24540cbb60 2024-03-06T07:35:10.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.453+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f244004c060 con 0x7f24540cbb60 2024-03-06T07:35:10.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.454+0000 7f244a000640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f243406b4d0 0x7f243406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.454+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f24400abfe0 con 0x7f24540cbb60 2024-03-06T07:35:10.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.455+0000 7f2451e00640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f243406b4d0 0x7f243406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.455+0000 7f2451e00640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f243406b4d0 0x7f243406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2444006fd0 tx=0x7f2444008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.459+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2454154140 con 0x7f24540cbb60 2024-03-06T07:35:10.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.695+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f245404f330 con 0x7f24540cbb60 2024-03-06T07:35:10.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.696+0000 7f244a000640 1 -- 172.21.15.97:0/2743767576 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f2440040070 con 0x7f24540cbb60 2024-03-06T07:35:10.698 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:10.698 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:10.698 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f243406b4d0 msgr2=0x7f243406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f243406b4d0 0x7f243406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2444006fd0 tx=0x7f2444008040 comp rx=0 tx=0).stop 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 msgr2=0x7f24541561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24541561d0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2440000c00 tx=0x7f24400046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 shutdown_connections 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f243406b4d0 0x7f243406d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f245415a4b0 0x7f245414f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24540cbb60 0x7f24541561d0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 --2- 172.21.15.97:0/2743767576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24540b42d0 0x7f2454154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 >> 172.21.15.97:0/2743767576 conn(0x7f245406afc0 msgr2=0x7f2454158000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 shutdown_connections 2024-03-06T07:35:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:10.699+0000 7f2453200640 1 -- 172.21.15.97:0/2743767576 wait complete. 2024-03-06T07:35:10.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:35:10.951 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:35:10.952 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:35:10.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.169+0000 7f3ea2400640 1 Processor -- start 2024-03-06T07:35:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.169+0000 7f3ea2400640 1 -- start start 2024-03-06T07:35:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.170+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 0x7f3e9c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.170+0000 7f3ea2400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3e9c15f9d0 0x7f3e9c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3ea2400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c06ee70 con 0x7f3e9c0ce0a0 2024-03-06T07:35:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3ea2400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c160370 con 0x7f3e9c15f9d0 2024-03-06T07:35:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3ea2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c06cef0 con 0x7f3e9c144c00 2024-03-06T07:35:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3e9be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3e9c0ce0a0 0x7f3e9c0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60876/0 (socket says 172.21.15.97:60876) 2024-03-06T07:35:11.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3e9be00640 1 -- 172.21.15.97:0/4006807395 learned_addr learned my addr 172.21.15.97:0/4006807395 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:11.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3e93e00640 1 --2- 172.21.15.97:0/4006807395 >> v2:172.21.15.135:3300/0 conn(0x7f3e9c15f9d0 0x7f3e9c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3e9b400640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 0x7f3e9c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.171+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799999476 0 0) 0x7f3e94001490 con 0x7f3e9c0ce0a0 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e80003520 con 0x7f3e9c0ce0a0 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708389277 0 0) 0x7f3e80003520 con 0x7f3e9c0ce0a0 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 msgr2=0x7f3e9c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 0x7f3e9c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 >> v2:172.21.15.135:3300/0 conn(0x7f3e9c15f9d0 msgr2=0x7f3e9c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 --2- 172.21.15.97:0/4006807395 >> v2:172.21.15.135:3300/0 conn(0x7f3e9c15f9d0 0x7f3e9c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c06d1d0 con 0x7f3e9c0ce0a0 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3e94001bc0 con 0x7f3e9c0ce0a0 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 >> v1:172.21.15.97:6789/0 conn(0x7f3e9c0ce0a0 legacy=0x7f3e9c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e800039b0 0x7f3e80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e80004800 0x7f3e80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 0x7f3e800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c06cef0 con 0x7f3e80004800 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8000b9c0 con 0x7f3e800039b0 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8000bb40 con 0x7f3e9c144c00 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e93e00640 1 --2- 172.21.15.97:0/4006807395 >> v2:172.21.15.135:3300/0 conn(0x7f3e9c15f9d0 0x7f3e9c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.172+0000 7f3e93e00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e80004800 0x7f3e80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e9b400640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e800039b0 0x7f3e80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e9be00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 0x7f3e800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e93e00640 1 -- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 msgr2=0x7f3e800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e93e00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c144c00 0x7f3e800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e93e00640 1 -- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e800039b0 msgr2=0x7f3e80003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e93e00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e800039b0 0x7f3e80003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.173+0000 7f3e93e00640 1 -- 172.21.15.97:0/4006807395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e80003520 con 0x7f3e80004800 2024-03-06T07:35:11.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.174+0000 7f3e93e00640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e80004800 0x7f3e80004c80 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009c70 tx=0x7f3e80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4ea90e6f42bc29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.174+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84009840 con 0x7f3e80004800 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.174+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e84004020 con 0x7f3e80004800 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.174+0000 7f3e9aa00640 1 -- 172.21.15.97:0/4006807395 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84004410 con 0x7f3e80004800 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3ea2400640 1 -- 172.21.15.97:0/4006807395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e80004800 msgr2=0x7f3e80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3ea2400640 1 --2- 172.21.15.97:0/4006807395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e80004800 0x7f3e80004c80 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009c70 tx=0x7f3e80005830 comp rx=0 tx=0).stop 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3e9be00640 1 -- 172.21.15.97:0/4006807395 reap_dead start 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3ea2400640 1 -- 172.21.15.97:0/4006807395 shutdown_connections 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3ea2400640 1 -- 172.21.15.97:0/4006807395 >> 172.21.15.97:0/4006807395 conn(0x7f3e9c0b1730 msgr2=0x7f3e9c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3ea2400640 1 -- 172.21.15.97:0/4006807395 shutdown_connections 2024-03-06T07:35:11.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.175+0000 7f3ea2400640 1 -- 172.21.15.97:0/4006807395 wait complete. 2024-03-06T07:35:11.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.177+0000 7f3ea2400640 1 Processor -- start 2024-03-06T07:35:11.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.177+0000 7f3ea2400640 1 -- start start 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e80004800 0x7f3e9c1e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 0x7f3e9c1e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c1e5b00 0x7f3e9c15a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3ea2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c160730 con 0x7f3e9c1392b0 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3ea2400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c06d790 con 0x7f3e80004800 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3ea2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c06cb50 con 0x7f3e9c1e5b00 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3e93e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 0x7f3e9c1e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3e93e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 0x7f3e9c1e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37742/0 (socket says 172.21.15.97:37742) 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.178+0000 7f3e93e00640 1 -- 172.21.15.97:0/3018504748 learned_addr learned my addr 172.21.15.97:0/3018504748 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e9b400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e80004800 0x7f3e9c1e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e9be00640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c1e5b00 0x7f3e9c15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e93e00640 1 -- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c1e5b00 msgr2=0x7f3e9c15a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e93e00640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c1e5b00 0x7f3e9c15a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e93e00640 1 -- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e80004800 msgr2=0x7f3e9c1e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e93e00640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e80004800 0x7f3e9c1e70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e93e00640 1 -- 172.21.15.97:0/3018504748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c1e6020 con 0x7f3e9c1392b0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e9be00640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c1e5b00 0x7f3e9c15a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.179+0000 7f3e93e00640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 0x7f3e9c1e55c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009d90 tx=0x7f3e84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.180+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84043020 con 0x7f3e9c1392b0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.180+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8402c960 con 0x7f3e9c1392b0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.180+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8402cd30 con 0x7f3e9c1392b0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.180+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c1e6300 con 0x7f3e9c1392b0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.180+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c1e68c0 con 0x7f3e9c1392b0 2024-03-06T07:35:11.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.181+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c04f330 con 0x7f3e9c1392b0 2024-03-06T07:35:11.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.183+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e84046020 con 0x7f3e9c1392b0 2024-03-06T07:35:11.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.184+0000 7f3e93400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3e7806b4d0 0x7f3e7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.184+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3e840ac4b0 con 0x7f3e9c1392b0 2024-03-06T07:35:11.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.185+0000 7f3e9b400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3e7806b4d0 0x7f3e7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.185+0000 7f3e9b400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3e7806b4d0 0x7f3e7806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009a20 tx=0x7f3e8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.189+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e8404d030 con 0x7f3e9c1392b0 2024-03-06T07:35:11.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.429+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e9c0cf6f0 con 0x7f3e9c1392b0 2024-03-06T07:35:11.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.430+0000 7f3e93400640 1 -- 172.21.15.97:0/3018504748 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f3e8407b200 con 0x7f3e9c1392b0 2024-03-06T07:35:11.431 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:11.432 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:11.432 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:11.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.432+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3e7806b4d0 msgr2=0x7f3e7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.432+0000 7f3ea2400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3e7806b4d0 0x7f3e7806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009a20 tx=0x7f3e8c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:11.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.432+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 msgr2=0x7f3e9c1e55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 0x7f3e9c1e55c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009d90 tx=0x7f3e84004730 comp rx=0 tx=0).stop 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 shutdown_connections 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3e7806b4d0 0x7f3e7806d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3e9c1e5b00 0x7f3e9c15a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3e9c1392b0 0x7f3e9c1e55c0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 --2- 172.21.15.97:0/3018504748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e80004800 0x7f3e9c1e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 >> 172.21.15.97:0/3018504748 conn(0x7f3e9c0b1730 msgr2=0x7f3e9c0af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 shutdown_connections 2024-03-06T07:35:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.433+0000 7f3ea2400640 1 -- 172.21.15.97:0/3018504748 wait complete. 2024-03-06T07:35:11.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:35:11.690 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:35:11.691 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:11.691 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:11.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.781+0000 7fa7cc438f40 1 Processor -- start 2024-03-06T07:35:11.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.781+0000 7fa7cc438f40 1 -- start start 2024-03-06T07:35:11.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7cc438f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 0x5569bf95a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7cc438f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf95a040 0x5569bf886a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7cc438f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a420 0x5569bf964430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7cc438f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569bf887440 con 0x5569bf8ccdd0 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7cc438f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569bf888b90 con 0x5569bf95a040 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7cc438f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569bf886890 con 0x5569bf95a420 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7c8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 0x5569bf95a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7c8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 0x5569bf95a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37756/0 (socket says 172.21.15.97:37756) 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.782+0000 7fa7c8e00640 1 -- 172.21.15.97:0/1648475121 learned_addr learned my addr 172.21.15.97:0/1648475121 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c9800640 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a420 0x5569bf964430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c3e00640 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf95a040 0x5569bf886a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c8e00640 1 -- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a420 msgr2=0x5569bf964430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c8e00640 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a420 0x5569bf964430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c8e00640 1 -- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf95a040 msgr2=0x5569bf886a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c8e00640 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf95a040 0x5569bf886a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c8e00640 1 -- 172.21.15.97:0/1648475121 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569bf7b2ad0 con 0x5569bf8ccdd0 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.783+0000 7fa7c8e00640 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 0x5569bf95a900 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0094f0 tx=0x7fa7bc0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73daf55108d51b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7c2a00640 1 -- 172.21.15.97:0/1648475121 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017070 con 0x5569bf8ccdd0 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7c2a00640 1 -- 172.21.15.97:0/1648475121 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc005530 con 0x5569bf8ccdd0 2024-03-06T07:35:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7c2a00640 1 -- 172.21.15.97:0/1648475121 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc005920 con 0x5569bf8ccdd0 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 -- 172.21.15.97:0/1648475121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 msgr2=0x5569bf95a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 0x5569bf95a900 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0094f0 tx=0x7fa7bc0099b0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 -- 172.21.15.97:0/1648475121 shutdown_connections 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a420 0x5569bf964430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf95a040 0x5569bf886a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/1648475121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf8ccdd0 0x5569bf95a900 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 -- 172.21.15.97:0/1648475121 >> 172.21.15.97:0/1648475121 conn(0x5569bf8b3bf0 msgr2=0x5569bf8cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 -- 172.21.15.97:0/1648475121 shutdown_connections 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.784+0000 7fa7cc438f40 1 -- 172.21.15.97:0/1648475121 wait complete. 2024-03-06T07:35:11.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.785+0000 7fa7cc438f40 1 Processor -- start 2024-03-06T07:35:11.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.785+0000 7fa7cc438f40 1 -- start start 2024-03-06T07:35:11.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.786+0000 7fa7cc438f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf8ccdd0 0x5569bf886a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.786+0000 7fa7cc438f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a040 0x5569bf886310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.786+0000 7fa7cc438f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf95a420 0x5569bf9491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.786+0000 7fa7cc438f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569bf95d470 con 0x5569bf95a420 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.786+0000 7fa7cc438f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569bf8b6430 con 0x5569bf8ccdd0 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.786+0000 7fa7cc438f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569bf954440 con 0x5569bf95a040 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf95a420 0x5569bf9491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf95a420 0x5569bf9491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37760/0 (socket says 172.21.15.97:37760) 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 -- 172.21.15.97:0/334309033 learned_addr learned my addr 172.21.15.97:0/334309033 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c8e00640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf8ccdd0 0x5569bf886a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c3e00640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a040 0x5569bf886310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 -- 172.21.15.97:0/334309033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a040 msgr2=0x5569bf886310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x5569bf95a040 0x5569bf886310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 -- 172.21.15.97:0/334309033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf8ccdd0 msgr2=0x5569bf886a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569bf8ccdd0 0x5569bf886a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 -- 172.21.15.97:0/334309033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569bf9540f0 con 0x5569bf95a420 2024-03-06T07:35:11.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.787+0000 7fa7c9800640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf95a420 0x5569bf9491b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fa7c401ba60 tx=0x7fa7c4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.788+0000 7fa7c1600640 1 -- 172.21.15.97:0/334309033 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4061070 con 0x5569bf95a420 2024-03-06T07:35:11.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.788+0000 7fa7c1600640 1 -- 172.21.15.97:0/334309033 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c4052d50 con 0x5569bf95a420 2024-03-06T07:35:11.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.788+0000 7fa7c1600640 1 -- 172.21.15.97:0/334309033 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4057040 con 0x5569bf95a420 2024-03-06T07:35:11.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.788+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5569bf94d8a0 con 0x5569bf95a420 2024-03-06T07:35:11.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.788+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569bf9496f0 con 0x5569bf95a420 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.790+0000 7fa7c1600640 1 -- 172.21.15.97:0/334309033 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7c4064070 con 0x5569bf95a420 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.790+0000 7fa7c1600640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a8069eb0 0x7fa7a806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.790+0000 7fa7c1600640 1 -- 172.21.15.97:0/334309033 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa7c40cacc0 con 0x5569bf95a420 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.791+0000 7fa7c8e00640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a8069eb0 0x7fa7a806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.791+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5569bf95b6b0 0x5569bf95bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.791+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118) v9 -- 0x5569bf95c060 con 0x5569bf95b6b0 2024-03-06T07:35:11.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.791+0000 7fa7c3e00640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5569bf95b6b0 0x5569bf95bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.791+0000 7fa7c8e00640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a8069eb0 0x7fa7a806c370 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000c00 tx=0x7fa7bc00f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7c3e00640 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5569bf95b6b0 0x5569bf95bb20 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7c3e00640 1 -- 172.21.15.97:0/334309033 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fa7b40075f0 con 0x5569bf95b6b0 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5569bf95b6b0 msgr2=0x5569bf95bb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x5569bf95b6b0 0x5569bf95bb20 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a8069eb0 msgr2=0x7fa7a806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7a8069eb0 0x7fa7a806c370 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000c00 tx=0x7fa7bc00f000 comp rx=0 tx=0).stop 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf95a420 msgr2=0x5569bf9491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 --2- 172.21.15.97:0/334309033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5569bf95a420 0x5569bf9491b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fa7c401ba60 tx=0x7fa7c4003330 comp rx=0 tx=0).stop 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7c9800640 1 -- 172.21.15.97:0/334309033 reap_dead start 2024-03-06T07:35:11.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 shutdown_connections 2024-03-06T07:35:11.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.792+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 >> 172.21.15.97:0/334309033 conn(0x5569bf8b3bf0 msgr2=0x5569bf966910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.793+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 shutdown_connections 2024-03-06T07:35:11.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:11.793+0000 7fa7cc438f40 1 -- 172.21.15.97:0/334309033 wait complete. 2024-03-06T07:35:11.798 INFO:teuthology.orchestra.run.smithi097.stdout:W�file_ddir_2. 2024-03-06T07:35:11.798 DEBUG:teuthology.orchestra.run.smithi097:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 1099511627777, 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "ancestors": [ 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1099511628278, 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "file_d", 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 27 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "dirino": 1, 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "dname": "dir_2", 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 46 2024-03-06T07:35:11.885 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:11.886 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-06T07:35:11.886 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 17, 2024-03-06T07:35:11.886 INFO:teuthology.orchestra.run.smithi097.stdout: "old_pools": [] 2024-03-06T07:35:11.886 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:35:11.886 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:11.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:11.890 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:35:11.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.150+0000 7f6bbbe00640 1 Processor -- start 2024-03-06T07:35:12.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.150+0000 7f6bbbe00640 1 -- start start 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bbbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6bb406cb50 0x7f6bb406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bbbe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb40717c0 con 0x7f6bb40cf430 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bbbe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb4071aa0 con 0x7f6bb40b2d10 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bbbe00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb4071d80 con 0x7f6bb406cb50 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bb8a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6bb406cb50 0x7f6bb406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bb9400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6bb40cf430 0x7f6bb4070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60890/0 (socket says 172.21.15.97:60890) 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.152+0000 7f6bb9400640 1 -- 172.21.15.97:0/1199713353 learned_addr learned my addr 172.21.15.97:0/1199713353 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.153+0000 7f6bb8a00640 1 -- 172.21.15.97:0/1199713353 >> v1:172.21.15.135:6789/0 conn(0x7f6bb40b2d10 legacy=0x7f6bb40b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.153+0000 7f6bb8a00640 1 -- 172.21.15.97:0/1199713353 >> v1:172.21.15.97:6789/0 conn(0x7f6bb40cf430 legacy=0x7f6bb4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.153+0000 7f6bb8a00640 1 -- 172.21.15.97:0/1199713353 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4072060 con 0x7f6bb406cb50 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.155+0000 7f6bb8a00640 1 --2- 172.21.15.97:0/1199713353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb406cfe0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00097a0 tx=0x7f6ba002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab4bbbaf45286c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.155+0000 7f6bb3e00640 1 -- 172.21.15.97:0/1199713353 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0035020 con 0x7f6bb406cb50 2024-03-06T07:35:12.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.155+0000 7f6bb3e00640 1 -- 172.21.15.97:0/1199713353 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba0031830 con 0x7f6bb406cb50 2024-03-06T07:35:12.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.155+0000 7f6bb3e00640 1 -- 172.21.15.97:0/1199713353 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0031c00 con 0x7f6bb406cb50 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.155+0000 7f6bbbe00640 1 -- 172.21.15.97:0/1199713353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 msgr2=0x7f6bb406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.155+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/1199713353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb406cfe0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00097a0 tx=0x7f6ba002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.156+0000 7f6bbbe00640 1 -- 172.21.15.97:0/1199713353 shutdown_connections 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.156+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/1199713353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb406cfe0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.156+0000 7f6bbbe00640 1 -- 172.21.15.97:0/1199713353 >> 172.21.15.97:0/1199713353 conn(0x7f6bb40b1730 msgr2=0x7f6bb40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.156+0000 7f6bbbe00640 1 -- 172.21.15.97:0/1199713353 shutdown_connections 2024-03-06T07:35:12.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.156+0000 7f6bbbe00640 1 -- 172.21.15.97:0/1199713353 wait complete. 2024-03-06T07:35:12.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.158+0000 7f6bbbe00640 1 Processor -- start 2024-03-06T07:35:12.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.158+0000 7f6bbbe00640 1 -- start start 2024-03-06T07:35:12.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bbbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb41fc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bbbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6bb40b2d10 0x7f6bb4147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bbbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6bb40cf430 0x7f6bb41ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bbbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4162b70 con 0x7f6bb406cb50 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bbbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4071430 con 0x7f6bb40b2d10 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bbbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4162cf0 con 0x7f6bb40cf430 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bb9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb41fc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bb9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb41fc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37776/0 (socket says 172.21.15.97:37776) 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.159+0000 7f6bb9400640 1 -- 172.21.15.97:0/694904434 learned_addr learned my addr 172.21.15.97:0/694904434 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb9400640 1 -- 172.21.15.97:0/694904434 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6bb40cf430 msgr2=0x7f6bb41ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb9400640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6bb40cf430 0x7f6bb41ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb9400640 1 -- 172.21.15.97:0/694904434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6bb40b2d10 msgr2=0x7f6bb4147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb9400640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6bb40b2d10 0x7f6bb4147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb9400640 1 -- 172.21.15.97:0/694904434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb41ec5d0 con 0x7f6bb406cb50 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb9400640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb41fc490 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c3f0 tx=0x7f6ba800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8017070 con 0x7f6bb406cb50 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba8005ce0 con 0x7f6bb406cb50 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba80072f0 con 0x7f6bb406cb50 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.160+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb41ec8b0 con 0x7f6bb406cb50 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.161+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb41ece70 con 0x7f6bb406cb50 2024-03-06T07:35:12.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.162+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb404f330 con 0x7f6bb406cb50 2024-03-06T07:35:12.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.163+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ba801a070 con 0x7f6bb406cb50 2024-03-06T07:35:12.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.164+0000 7f6bb1600640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b9406b520 0x7f6b9406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.164+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6ba80896b0 con 0x7f6bb406cb50 2024-03-06T07:35:12.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.164+0000 7f6bb8a00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b9406b520 0x7f6b9406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.165+0000 7f6bb8a00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b9406b520 0x7f6b9406d9e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0036000 tx=0x7f6ba0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.168+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ba8058480 con 0x7f6bb406cb50 2024-03-06T07:35:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.403+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bb40b4a40 con 0x7f6bb406cb50 2024-03-06T07:35:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.403+0000 7f6bb1600640 1 -- 172.21.15.97:0/694904434 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f6ba8057d80 con 0x7f6bb406cb50 2024-03-06T07:35:12.405 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:12.405 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:12.405 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b9406b520 msgr2=0x7f6b9406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b9406b520 0x7f6b9406d9e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0036000 tx=0x7f6ba0030000 comp rx=0 tx=0).stop 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 msgr2=0x7f6bb41fc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb41fc490 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c3f0 tx=0x7f6ba800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 shutdown_connections 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6b9406b520 0x7f6b9406d9e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6bb40cf430 0x7f6bb41ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6bb40b2d10 0x7f6bb4147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 --2- 172.21.15.97:0/694904434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bb406cb50 0x7f6bb41fc490 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 >> 172.21.15.97:0/694904434 conn(0x7f6bb40b1730 msgr2=0x7f6bb40b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.406+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 shutdown_connections 2024-03-06T07:35:12.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:12.407+0000 7f6bbbe00640 1 -- 172.21.15.97:0/694904434 wait complete. 2024-03-06T07:35:12.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:12.641 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:35:12.641 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:35:12.641 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:35:17.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:17.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.851+0000 7fe807600640 1 Processor -- start 2024-03-06T07:35:17.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.851+0000 7fe807600640 1 -- start start 2024-03-06T07:35:17.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.854+0000 7fe807600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe8000b4680 0x7fe80006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.854+0000 7fe807600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80015b090 con 0x7fe800072340 2024-03-06T07:35:17.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.854+0000 7fe807600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8000b2a30 con 0x7fe80015acb0 2024-03-06T07:35:17.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.854+0000 7fe807600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8000b4360 con 0x7fe8000b4680 2024-03-06T07:35:17.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.854+0000 7fe805600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe800072340 0x7fe8000ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36636/0 (socket says 172.21.15.97:36636) 2024-03-06T07:35:17.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.854+0000 7fe805600640 1 -- 172.21.15.97:0/4239865827 learned_addr learned my addr 172.21.15.97:0/4239865827 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7f7e00640 1 --2- 172.21.15.97:0/4239865827 >> v2:172.21.15.135:3300/0 conn(0x7fe8000b4680 0x7fe80006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216670875 0 0) 0x7fe80015b090 con 0x7fe800072340 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e4003520 con 0x7fe800072340 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1005726108 0 0) 0x7fe7e4003520 con 0x7fe800072340 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 >> v2:172.21.15.135:3300/0 conn(0x7fe8000b4680 msgr2=0x7fe80006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 --2- 172.21.15.97:0/4239865827 >> v2:172.21.15.135:3300/0 conn(0x7fe8000b4680 0x7fe80006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 >> v1:172.21.15.135:6789/0 conn(0x7fe80015acb0 legacy=0x7fe8000b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000b2cf0 con 0x7fe800072340 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe7f80019f0 con 0x7fe800072340 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 >> v1:172.21.15.97:6789/0 conn(0x7fe800072340 legacy=0x7fe8000ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7e4003a00 0x7fe7e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7e4004800 0x7fe7e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.855+0000 7fe7ffe00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e40051c0 0x7fe7e400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b4360 con 0x7fe7e4003a00 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400e2d0 con 0x7fe7e4004800 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400e4d0 con 0x7fe7e40051c0 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe804c00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7e4003a00 0x7fe7e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe7f7e00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7e4004800 0x7fe7e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe805600640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e40051c0 0x7fe7e400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe804c00640 1 -- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e40051c0 msgr2=0x7fe7e400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe804c00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e40051c0 0x7fe7e400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe804c00640 1 -- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7e4004800 msgr2=0x7fe7e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe804c00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7e4004800 0x7fe7e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe804c00640 1 -- 172.21.15.97:0/4239865827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4003520 con 0x7fe7e4003a00 2024-03-06T07:35:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe805600640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e40051c0 0x7fe7e400dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.856+0000 7fe805600640 1 -- 172.21.15.97:0/4239865827 reap_dead start 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.857+0000 7fe804c00640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7e4003a00 0x7fe7e4003e20 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001750 tx=0x7fe7e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dc8794c32333b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0001370 con 0x7fe7e4003a00 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f00052d0 con 0x7fe7e4003a00 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe7ffe00640 1 -- 172.21.15.97:0/4239865827 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0003830 con 0x7fe7e4003a00 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe807600640 1 -- 172.21.15.97:0/4239865827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7e4003a00 msgr2=0x7fe7e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe807600640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7e4003a00 0x7fe7e4003e20 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001750 tx=0x7fe7e4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe807600640 1 -- 172.21.15.97:0/4239865827 shutdown_connections 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe807600640 1 --2- 172.21.15.97:0/4239865827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7e4003a00 0x7fe7e4003e20 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe807600640 1 -- 172.21.15.97:0/4239865827 >> 172.21.15.97:0/4239865827 conn(0x7fe8000b1730 msgr2=0x7fe8000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.858+0000 7fe807600640 1 -- 172.21.15.97:0/4239865827 shutdown_connections 2024-03-06T07:35:17.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.859+0000 7fe807600640 1 -- 172.21.15.97:0/4239865827 wait complete. 2024-03-06T07:35:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.860+0000 7fe807600640 1 Processor -- start 2024-03-06T07:35:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.860+0000 7fe807600640 1 -- start start 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.861+0000 7fe807600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e4003a00 0x7fe800144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.861+0000 7fe807600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 0x7fe80015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.861+0000 7fe807600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015c980 0x7fe80015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.861+0000 7fe807600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b4010 con 0x7fe800159b70 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.861+0000 7fe807600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b32b0 con 0x7fe80015c980 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.861+0000 7fe807600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b37d0 con 0x7fe7e4003a00 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 0x7fe80015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 0x7fe80015baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36560/0 (socket says 172.21.15.97:36560) 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 -- 172.21.15.97:0/108985840 learned_addr learned my addr 172.21.15.97:0/108985840 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe805600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015c980 0x7fe80015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 -- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e4003a00 msgr2=0x7fe800144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e4003a00 0x7fe800144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 -- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015c980 msgr2=0x7fe80015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015c980 0x7fe80015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 -- 172.21.15.97:0/108985840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80015c3d0 con 0x7fe800159b70 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe805600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015c980 0x7fe80015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.862+0000 7fe7f7e00640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 0x7fe80015baa0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80116d0 tx=0x7fe7e8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.863+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800f9c0 con 0x7fe800159b70 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.863+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800fb60 con 0x7fe800159b70 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.863+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8015460 con 0x7fe800159b70 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.863+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80015c6b0 con 0x7fe800159b70 2024-03-06T07:35:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.863+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8001572c0 con 0x7fe800159b70 2024-03-06T07:35:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.864+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80004f330 con 0x7fe800159b70 2024-03-06T07:35:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.865+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7e8015600 con 0x7fe800159b70 2024-03-06T07:35:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.866+0000 7fe7fd600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b500 0x7fe7dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.866+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe7e808abe0 con 0x7fe800159b70 2024-03-06T07:35:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.866+0000 7fe804c00640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b500 0x7fe7dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.867+0000 7fe804c00640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b500 0x7fe7dc06d9c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0033000 tx=0x7fe7f002f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:17.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:17.871+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7e8019030 con 0x7fe800159b70 2024-03-06T07:35:18.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.101+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe80006e730 con 0x7fe800159b70 2024-03-06T07:35:18.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.102+0000 7fe7fd600640 1 -- 172.21.15.97:0/108985840 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fe7e80592e0 con 0x7fe800159b70 2024-03-06T07:35:18.104 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:18.104 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:18.104 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:18.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b500 msgr2=0x7fe7dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b500 0x7fe7dc06d9c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0033000 tx=0x7fe7f002f000 comp rx=0 tx=0).stop 2024-03-06T07:35:18.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 msgr2=0x7fe80015baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 0x7fe80015baa0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80116d0 tx=0x7fe7e8011b90 comp rx=0 tx=0).stop 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 shutdown_connections 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7dc06b500 0x7fe7dc06d9c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80015c980 0x7fe80015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe800159b70 0x7fe80015baa0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 --2- 172.21.15.97:0/108985840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7e4003a00 0x7fe800144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.104+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 >> 172.21.15.97:0/108985840 conn(0x7fe8000b1730 msgr2=0x7fe80006d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.105+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 shutdown_connections 2024-03-06T07:35:18.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.105+0000 7fe807600640 1 -- 172.21.15.97:0/108985840 wait complete. 2024-03-06T07:35:18.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:18.342 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:35:18.343 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:35:18.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:18.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.588+0000 7f9f4b200640 1 Processor -- start 2024-03-06T07:35:18.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.588+0000 7f9f4b200640 1 -- start start 2024-03-06T07:35:18.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c06da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9f4c0d0ec0 0x7f9f4c0b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0cf410 con 0x7f9f4c0d0350 2024-03-06T07:35:18.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4c144a10 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c06e050 con 0x7f9f4c144630 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9f4c0d0ec0 0x7f9f4c0b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9f4c0d0ec0 0x7f9f4c0b3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36564/0 (socket says 172.21.15.97:36564) 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f4a800640 1 -- 172.21.15.97:0/2401170436 learned_addr learned my addr 172.21.15.97:0/2401170436 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.590+0000 7f9f49e00640 1 --2- 172.21.15.97:0/2401170436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c06da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.591+0000 7f9f4a800640 1 -- 172.21.15.97:0/2401170436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 msgr2=0x7f9f4c06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.591+0000 7f9f4a800640 1 --2- 172.21.15.97:0/2401170436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c06da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.591+0000 7f9f4a800640 1 -- 172.21.15.97:0/2401170436 >> v1:172.21.15.135:6789/0 conn(0x7f9f4c0d0350 legacy=0x7f9f4c0d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.591+0000 7f9f4a800640 1 -- 172.21.15.97:0/2401170436 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0cf6f0 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.592+0000 7f9f4a800640 1 --2- 172.21.15.97:0/2401170436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c0b3960 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009e50 tx=0x7f9f4402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ea1bbd63de62be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f48a00640 1 -- 172.21.15.97:0/2401170436 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4402c9e0 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f48a00640 1 -- 172.21.15.97:0/2401170436 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4402cb80 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f48a00640 1 -- 172.21.15.97:0/2401170436 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f440343f0 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 -- 172.21.15.97:0/2401170436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 msgr2=0x7f9f4c0b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 --2- 172.21.15.97:0/2401170436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c0b3960 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009e50 tx=0x7f9f4402be40 comp rx=0 tx=0).stop 2024-03-06T07:35:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 -- 172.21.15.97:0/2401170436 shutdown_connections 2024-03-06T07:35:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 --2- 172.21.15.97:0/2401170436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c0b3960 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 --2- 172.21.15.97:0/2401170436 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c06da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 -- 172.21.15.97:0/2401170436 >> 172.21.15.97:0/2401170436 conn(0x7f9f4c06c0d0 msgr2=0x7f9f4c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 -- 172.21.15.97:0/2401170436 shutdown_connections 2024-03-06T07:35:18.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.593+0000 7f9f4b200640 1 -- 172.21.15.97:0/2401170436 wait complete. 2024-03-06T07:35:18.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.595+0000 7f9f4b200640 1 Processor -- start 2024-03-06T07:35:18.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.595+0000 7f9f4b200640 1 -- start start 2024-03-06T07:35:18.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f4b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f4c0d0350 0x7f9f4c1f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f4b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f4b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c1e7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f4b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c06f850 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f4b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0cfd40 con 0x7f9f4c0d0350 2024-03-06T07:35:18.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f4b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0b3ea0 con 0x7f9f4c144630 2024-03-06T07:35:18.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f49400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f49400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c155f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36566/0 (socket says 172.21.15.97:36566) 2024-03-06T07:35:18.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.596+0000 7f9f49400640 1 -- 172.21.15.97:0/3542770687 learned_addr learned my addr 172.21.15.97:0/3542770687 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:18.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49e00640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f4c0d0350 0x7f9f4c1f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f4a800640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c1e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49400640 1 -- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 msgr2=0x7f9f4c1e7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49400640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c1e7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49400640 1 -- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f4c0d0350 msgr2=0x7f9f4c1f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49400640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f4c0d0350 0x7f9f4c1f7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49400640 1 -- 172.21.15.97:0/3542770687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c15d320 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f49400640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c155f10 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800c430 tx=0x7f9f3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38011790 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f38017070 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.597+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4c15d600 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.598+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38011bb0 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.598+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4c15dbc0 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.599+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c04f330 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.600+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f3801c460 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.601+0000 7f9f42000640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f2c06b500 0x7f9f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.601+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9f3808a1d0 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.601+0000 7f9f49e00640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f2c06b500 0x7f9f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.602+0000 7f9f49e00640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f2c06b500 0x7f9f2c06d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c006fd0 tx=0x7f9f3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.605+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f38058f20 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.839+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f4c0b5910 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.841+0000 7f9f42000640 1 -- 172.21.15.97:0/3542770687 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f9f38058820 con 0x7f9f4c0d0ec0 2024-03-06T07:35:18.843 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:18.843 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:18.843 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f2c06b500 msgr2=0x7f9f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f2c06b500 0x7f9f2c06d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c006fd0 tx=0x7f9f3c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 msgr2=0x7f9f4c155f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c155f10 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800c430 tx=0x7f9f3800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 shutdown_connections 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f2c06b500 0x7f9f2c06d9c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f4c144630 0x7f9f4c1e7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f4c0d0ec0 0x7f9f4c155f10 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 --2- 172.21.15.97:0/3542770687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f4c0d0350 0x7f9f4c1f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.843+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 >> 172.21.15.97:0/3542770687 conn(0x7f9f4c06c0d0 msgr2=0x7f9f4c0cccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.844+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 shutdown_connections 2024-03-06T07:35:18.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:18.844+0000 7f9f4b200640 1 -- 172.21.15.97:0/3542770687 wait complete. 2024-03-06T07:35:18.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:35:19.096 INFO:teuthology.orchestra.run.smithi097.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:35:19.096 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:35:19.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:19.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.290+0000 7fa557600640 1 Processor -- start 2024-03-06T07:35:19.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.290+0000 7fa557600640 1 -- start start 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.291+0000 7fa557600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa5500cf430 0x7fa55006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa557600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa55006d920 0x7fa55015dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa557600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5500b35b0 con 0x7fa550151660 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa557600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5500b3ee0 con 0x7fa55006d920 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa557600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5500b3890 con 0x7fa5500cf430 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa555600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa55006d920 0x7fa55015dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa555600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa55006d920 0x7fa55015dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36574/0 (socket says 172.21.15.97:36574) 2024-03-06T07:35:19.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa555600640 1 -- 172.21.15.97:0/2480880250 learned_addr learned my addr 172.21.15.97:0/2480880250 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:19.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.292+0000 7fa547e00640 1 --2- 172.21.15.97:0/2480880250 >> v2:172.21.15.135:3300/0 conn(0x7fa5500cf430 0x7fa55006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.293+0000 7fa555600640 1 -- 172.21.15.97:0/2480880250 >> v2:172.21.15.135:3300/0 conn(0x7fa5500cf430 msgr2=0x7fa55006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.293+0000 7fa555600640 1 --2- 172.21.15.97:0/2480880250 >> v2:172.21.15.135:3300/0 conn(0x7fa5500cf430 0x7fa55006d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.293+0000 7fa555600640 1 -- 172.21.15.97:0/2480880250 >> v1:172.21.15.135:6789/0 conn(0x7fa550151660 legacy=0x7fa550071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.293+0000 7fa555600640 1 -- 172.21.15.97:0/2480880250 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5500b3b70 con 0x7fa55006d920 2024-03-06T07:35:19.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.295+0000 7fa555600640 1 --2- 172.21.15.97:0/2480880250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa55006d920 0x7fa55015dff0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa55006c8e0 tx=0x7fa54802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96636e025a42de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.295+0000 7fa54fe00640 1 -- 172.21.15.97:0/2480880250 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54802c960 con 0x7fa55006d920 2024-03-06T07:35:19.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.295+0000 7fa54fe00640 1 -- 172.21.15.97:0/2480880250 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54802cb00 con 0x7fa55006d920 2024-03-06T07:35:19.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.295+0000 7fa54fe00640 1 -- 172.21.15.97:0/2480880250 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5480343f0 con 0x7fa55006d920 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.295+0000 7fa557600640 1 -- 172.21.15.97:0/2480880250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa55006d920 msgr2=0x7fa55015dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.295+0000 7fa557600640 1 --2- 172.21.15.97:0/2480880250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa55006d920 0x7fa55015dff0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa55006c8e0 tx=0x7fa54802bdc0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.296+0000 7fa557600640 1 -- 172.21.15.97:0/2480880250 shutdown_connections 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.296+0000 7fa557600640 1 --2- 172.21.15.97:0/2480880250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa55006d920 0x7fa55015dff0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.296+0000 7fa557600640 1 --2- 172.21.15.97:0/2480880250 >> v2:172.21.15.135:3300/0 conn(0x7fa5500cf430 0x7fa55006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.296+0000 7fa557600640 1 -- 172.21.15.97:0/2480880250 >> 172.21.15.97:0/2480880250 conn(0x7fa5500b1730 msgr2=0x7fa5500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.296+0000 7fa557600640 1 -- 172.21.15.97:0/2480880250 shutdown_connections 2024-03-06T07:35:19.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.296+0000 7fa557600640 1 -- 172.21.15.97:0/2480880250 wait complete. 2024-03-06T07:35:19.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.297+0000 7fa557600640 1 Processor -- start 2024-03-06T07:35:19.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 -- start start 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa55006d920 0x7fa5501e3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 0x7fa55014fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa550151660 0x7fa5501e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55015e6b0 con 0x7fa5500cf430 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55015e530 con 0x7fa55006d920 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.298+0000 7fa557600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55015e830 con 0x7fa550151660 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 0x7fa55014fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 0x7fa55014fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36588/0 (socket says 172.21.15.97:36588) 2024-03-06T07:35:19.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 -- 172.21.15.97:0/203823369 learned_addr learned my addr 172.21.15.97:0/203823369 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:19.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 -- 172.21.15.97:0/203823369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa550151660 msgr2=0x7fa5501e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa550151660 0x7fa5501e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 -- 172.21.15.97:0/203823369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa55006d920 msgr2=0x7fa5501e3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa55006d920 0x7fa5501e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 -- 172.21.15.97:0/203823369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5501e3df0 con 0x7fa5500cf430 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.299+0000 7fa547e00640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 0x7fa55014fa10 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa53800d5f0 tx=0x7fa53800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.300+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5380125f0 con 0x7fa5500cf430 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.300+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa538019070 con 0x7fa5500cf430 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.300+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5380129d0 con 0x7fa5500cf430 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.300+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5501e40d0 con 0x7fa5500cf430 2024-03-06T07:35:19.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.300+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5501e4690 con 0x7fa5500cf430 2024-03-06T07:35:19.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.301+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55004f330 con 0x7fa5500cf430 2024-03-06T07:35:19.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.302+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa538012b70 con 0x7fa5500cf430 2024-03-06T07:35:19.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.303+0000 7fa54d600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52c06b500 0x7fa52c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.303+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa53808a6f0 con 0x7fa5500cf430 2024-03-06T07:35:19.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.303+0000 7fa554c00640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52c06b500 0x7fa52c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.304+0000 7fa554c00640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52c06b500 0x7fa52c06d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa540003cc0 tx=0x7fa5400079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.308+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa538059440 con 0x7fa5500cf430 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.541+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa55015a4b0 con 0x7fa5500cf430 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.542+0000 7fa54d600640 1 -- 172.21.15.97:0/203823369 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa538058d40 con 0x7fa5500cf430 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52c06b500 msgr2=0x7fa52c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52c06b500 0x7fa52c06d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa540003cc0 tx=0x7fa5400079b0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 msgr2=0x7fa55014fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 0x7fa55014fa10 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa53800d5f0 tx=0x7fa53800dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 shutdown_connections 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa52c06b500 0x7fa52c06d9c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa550151660 0x7fa5501e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5500cf430 0x7fa55014fa10 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 --2- 172.21.15.97:0/203823369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa55006d920 0x7fa5501e3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 >> 172.21.15.97:0/203823369 conn(0x7fa5500b1730 msgr2=0x7fa5500b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 shutdown_connections 2024-03-06T07:35:19.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.545+0000 7fa557600640 1 -- 172.21.15.97:0/203823369 wait complete. 2024-03-06T07:35:19.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-03-06T07:35:19.778 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:35:19.778 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:35:19.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:19.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.975+0000 7f475ce00640 1 Processor -- start 2024-03-06T07:35:19.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.975+0000 7f475ce00640 1 -- start start 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.976+0000 7f475ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f47580d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f475ce00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f47580d18e0 0x7f47580d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f475ce00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475815f030 con 0x7f47580b2a60 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f475ce00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47580cf090 con 0x7f47580d18e0 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f475ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475815f310 con 0x7f47580d0600 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f47580d18e0 0x7f47580d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f47580d18e0 0x7f47580d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36594/0 (socket says 172.21.15.97:36594) 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f474fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f47580b2a60 0x7f47580d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36638/0 (socket says 172.21.15.97:36638) 2024-03-06T07:35:19.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757e00640 1 -- 172.21.15.97:0/2679277010 learned_addr learned my addr 172.21.15.97:0/2679277010 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:19.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757400640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f47580d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757e00640 1 -- 172.21.15.97:0/2679277010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 msgr2=0x7f47580d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757e00640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f47580d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.977+0000 7f4757e00640 1 -- 172.21.15.97:0/2679277010 >> v1:172.21.15.97:6789/0 conn(0x7f47580b2a60 legacy=0x7f47580d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.978+0000 7f4756a00640 1 -- 172.21.15.97:0/2679277010 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076788501 0 0) 0x7f475815f030 con 0x7f47580b2a60 2024-03-06T07:35:19.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.978+0000 7f4757e00640 1 -- 172.21.15.97:0/2679277010 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47580cea40 con 0x7f47580d18e0 2024-03-06T07:35:19.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.978+0000 7f4757400640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f47580d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.979+0000 7f4757e00640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f47580d1d40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4750009e50 tx=0x7f475002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ee12a1d65c07fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f4756a00640 1 -- 172.21.15.97:0/2679277010 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475002cb00 con 0x7f47580d18e0 2024-03-06T07:35:19.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f4756a00640 1 -- 172.21.15.97:0/2679277010 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475002cca0 con 0x7f47580d18e0 2024-03-06T07:35:19.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f4756a00640 1 -- 172.21.15.97:0/2679277010 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750034610 con 0x7f47580d18e0 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f475ce00640 1 -- 172.21.15.97:0/2679277010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 msgr2=0x7f47580d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f475ce00640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f47580d1d40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4750009e50 tx=0x7f475002bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f475ce00640 1 -- 172.21.15.97:0/2679277010 shutdown_connections 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f475ce00640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f47580d1d40 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f475ce00640 1 --2- 172.21.15.97:0/2679277010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f47580d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.980+0000 7f475ce00640 1 -- 172.21.15.97:0/2679277010 >> 172.21.15.97:0/2679277010 conn(0x7f47580d3d10 msgr2=0x7f47580d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.981+0000 7f475ce00640 1 -- 172.21.15.97:0/2679277010 shutdown_connections 2024-03-06T07:35:19.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.981+0000 7f475ce00640 1 -- 172.21.15.97:0/2679277010 wait complete. 2024-03-06T07:35:19.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.982+0000 7f475ce00640 1 Processor -- start 2024-03-06T07:35:19.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.982+0000 7f475ce00640 1 -- start start 2024-03-06T07:35:19.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.983+0000 7f475ce00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47580b2a60 0x7f47580c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.983+0000 7f475ce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f4758150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.983+0000 7f475ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f4758151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.983+0000 7f475ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475815ecf0 con 0x7f47580d18e0 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.983+0000 7f475ce00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580b3250 con 0x7f47580b2a60 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.983+0000 7f475ce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580cf520 con 0x7f47580d0600 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f4758151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f4758151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36600/0 (socket says 172.21.15.97:36600) 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 -- 172.21.15.97:0/2365577817 learned_addr learned my addr 172.21.15.97:0/2365577817 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f474fe00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f4758150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757400640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47580b2a60 0x7f47580c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 -- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 msgr2=0x7f4758150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f4758150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 -- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47580b2a60 msgr2=0x7f47580c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47580b2a60 0x7f47580c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 -- 172.21.15.97:0/2365577817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47580cbfd0 con 0x7f47580d18e0 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757400640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47580b2a60 0x7f47580c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.984+0000 7f4757e00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f4758151220 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4750009e50 tx=0x7f4750004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.985+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750043030 con 0x7f47580d18e0 2024-03-06T07:35:19.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.985+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475002ce50 con 0x7f47580d18e0 2024-03-06T07:35:19.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.985+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475003e450 con 0x7f47580d18e0 2024-03-06T07:35:19.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.985+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47580b4270 con 0x7f47580d18e0 2024-03-06T07:35:19.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.985+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47580b4830 con 0x7f47580d18e0 2024-03-06T07:35:19.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.986+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475804f330 con 0x7f47580d18e0 2024-03-06T07:35:19.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.988+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4750046070 con 0x7f47580d18e0 2024-03-06T07:35:19.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.989+0000 7f474f400640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f473406b4d0 0x7f473406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.989+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f47500ac6d0 con 0x7f47580d18e0 2024-03-06T07:35:19.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.989+0000 7f4757400640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f473406b4d0 0x7f473406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.990+0000 7f4757400640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f473406b4d0 0x7f473406d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f47580b3220 tx=0x7f4748008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:19.993+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f475004d070 con 0x7f47580d18e0 2024-03-06T07:35:20.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.228+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47580d0a70 con 0x7f47580d18e0 2024-03-06T07:35:20.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.229+0000 7f474f400640 1 -- 172.21.15.97:0/2365577817 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f47580cbfd0 con 0x7f47580d18e0 2024-03-06T07:35:20.232 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:20.232 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:20.232 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:20.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f473406b4d0 msgr2=0x7f473406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f473406b4d0 0x7f473406d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f47580b3220 tx=0x7f4748008040 comp rx=0 tx=0).stop 2024-03-06T07:35:20.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 msgr2=0x7f4758151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f4758151220 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4750009e50 tx=0x7f4750004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 shutdown_connections 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f473406b4d0 0x7f473406d990 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f47580d18e0 0x7f4758151220 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f47580d0600 0x7f4758150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 --2- 172.21.15.97:0/2365577817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47580b2a60 0x7f47580c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 >> 172.21.15.97:0/2365577817 conn(0x7f47580d3d10 msgr2=0x7f47580d1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 shutdown_connections 2024-03-06T07:35:20.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.232+0000 7f475ce00640 1 -- 172.21.15.97:0/2365577817 wait complete. 2024-03-06T07:35:20.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-03-06T07:35:20.463 INFO:teuthology.orchestra.run.smithi097.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:35:20.463 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:35:20.463 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:35:20.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.570+0000 7f2e64a35f40 1 Processor -- start 2024-03-06T07:35:20.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.570+0000 7f2e64a35f40 1 -- start start 2024-03-06T07:35:20.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.571+0000 7f2e64a35f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 0x55dd97abc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.571+0000 7f2e64a35f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8cd60 0x55dd97aba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.571+0000 7f2e64a35f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b4b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.571+0000 7f2e64a35f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd97a8bfa0 con 0x55dd97a8c980 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.571+0000 7f2e64a35f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd97a8bd00 con 0x55dd97b4a7d0 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.571+0000 7f2e64a35f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd97a8c240 con 0x55dd97a8cd60 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 0x55dd97abc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 0x55dd97abc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36614/0 (socket says 172.21.15.97:36614) 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 -- 172.21.15.97:0/3073381584 learned_addr learned my addr 172.21.15.97:0/3073381584 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61e00640 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b4b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 -- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8cd60 msgr2=0x55dd97aba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8cd60 0x55dd97aba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 -- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 msgr2=0x55dd97b4b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b4b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.572+0000 7f2e61400640 1 -- 172.21.15.97:0/3073381584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd979b8ad0 con 0x55dd97a8c980 2024-03-06T07:35:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e61400640 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 0x55dd97abc850 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f2e54044770 tx=0x7f2e5404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c86ce609ace531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e5b400640 1 -- 172.21.15.97:0/3073381584 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54057040 con 0x55dd97a8c980 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e5b400640 1 -- 172.21.15.97:0/3073381584 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54052a10 con 0x55dd97a8c980 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e5b400640 1 -- 172.21.15.97:0/3073381584 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54052e00 con 0x55dd97a8c980 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 -- 172.21.15.97:0/3073381584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 msgr2=0x55dd97abc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 0x55dd97abc850 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f2e54044770 tx=0x7f2e5404c000 comp rx=0 tx=0).stop 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 -- 172.21.15.97:0/3073381584 shutdown_connections 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b4b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8cd60 0x55dd97aba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/3073381584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8c980 0x55dd97abc850 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.573+0000 7f2e64a35f40 1 -- 172.21.15.97:0/3073381584 >> 172.21.15.97:0/3073381584 conn(0x55dd97abbc40 msgr2=0x55dd97b4c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.574+0000 7f2e64a35f40 1 -- 172.21.15.97:0/3073381584 shutdown_connections 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.574+0000 7f2e64a35f40 1 -- 172.21.15.97:0/3073381584 wait complete. 2024-03-06T07:35:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.574+0000 7f2e64a35f40 1 Processor -- start 2024-03-06T07:35:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.574+0000 7f2e64a35f40 1 -- start start 2024-03-06T07:35:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.575+0000 7f2e64a35f40 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8c980 0x55dd97b5d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.575+0000 7f2e64a35f40 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 0x55dd97b421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.575+0000 7f2e64a35f40 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b41250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.575+0000 7f2e64a35f40 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd97abec50 con 0x55dd97a8cd60 2024-03-06T07:35:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.575+0000 7f2e64a35f40 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd97b5caa0 con 0x55dd97b4a7d0 2024-03-06T07:35:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.575+0000 7f2e64a35f40 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd97b60b00 con 0x55dd97a8c980 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 0x55dd97b421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 0x55dd97b421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36618/0 (socket says 172.21.15.97:36618) 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 learned_addr learned my addr 172.21.15.97:0/2113511081 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b41250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8c980 msgr2=0x55dd97b5d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55dd97a8c980 0x55dd97b5d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 msgr2=0x55dd97b41250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55dd97b4a7d0 0x55dd97b41250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd97b61ff0 con 0x55dd97a8cd60 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.576+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 0x55dd97b421a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c008490 tx=0x7f2e4c008950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.577+0000 7f2e5a000640 1 -- 172.21.15.97:0/2113511081 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c016070 con 0x55dd97a8cd60 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.577+0000 7f2e5a000640 1 -- 172.21.15.97:0/2113511081 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c0067c0 con 0x55dd97a8cd60 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.577+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dd97b4a3a0 con 0x55dd97a8cd60 2024-03-06T07:35:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.577+0000 7f2e5a000640 1 -- 172.21.15.97:0/2113511081 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c006bb0 con 0x55dd97a8cd60 2024-03-06T07:35:20.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.577+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd97b62530 con 0x55dd97a8cd60 2024-03-06T07:35:20.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.579+0000 7f2e5a000640 1 -- 172.21.15.97:0/2113511081 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e4c019070 con 0x55dd97a8cd60 2024-03-06T07:35:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.580+0000 7f2e5a000640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e40069eb0 0x7f2e4006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.580+0000 7f2e5a000640 1 -- 172.21.15.97:0/2113511081 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2e4c088990 con 0x55dd97a8cd60 2024-03-06T07:35:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.580+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e40069eb0 0x7f2e4006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.580+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55dd97b426e0 0x55dd97c007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.580+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x55dd97b5d570 con 0x55dd97b426e0 2024-03-06T07:35:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.581+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e40069eb0 0x7f2e4006c370 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e54044770 tx=0x7f2e54005ac0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.581+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55dd97b426e0 0x55dd97c007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.582+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55dd97b426e0 0x55dd97c007f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.583+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 1 ==== osd_op_reply(1 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.583+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f2e5c011900 0x7f2e5c010380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.583+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.583+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f2e5c011900 0x7f2e5c010380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.584+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f2e5c011900 0x7f2e5c010380 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.584+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 1 ==== osd_op_reply(2 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.584+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7f2e54067400 0x7f2e54067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.584+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.585+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7f2e54067400 0x7f2e54067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.585+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7f2e54067400 0x7f2e54067860 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.586+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 1 ==== osd_op_reply(3 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.586+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f2e4c01b060 0x7f2e4c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.586+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.586+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f2e4c01b060 0x7f2e4c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.586+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f2e4c01b060 0x7f2e4c055f30 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.587+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== osd_op_reply(4 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.587+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.588+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 2 ==== osd_op_reply(5 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.588+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.588+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 2 ==== osd_op_reply(6 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.588+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.588+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 2 ==== osd_op_reply(7 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.588+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.589+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 3 ==== osd_op_reply(8 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.589+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.590+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 3 ==== osd_op_reply(9 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.590+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.590+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 3 ==== osd_op_reply(10 [pgnls start_epoch 118 out=49b] v112'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.590+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.590+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 4 ==== osd_op_reply(11 [pgnls start_epoch 118 out=49b] v112'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.590+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.591+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 4 ==== osd_op_reply(12 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.591+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.592+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 4 ==== osd_op_reply(13 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.592+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f2e5c00da10 0x7f2e5c016420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.592+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.592+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f2e5c00da10 0x7f2e5c016420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.593+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f2e5c00da10 0x7f2e5c016420 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.594+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 1 ==== osd_op_reply(14 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.594+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.595+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 2 ==== osd_op_reply(15 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.595+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.595+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 5 ==== osd_op_reply(16 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.595+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.596+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 2 ==== osd_op_reply(17 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.596+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7f2e540713f0 0x7f2e54071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.596+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e540713f0 2024-03-06T07:35:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.597+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7f2e540713f0 0x7f2e54071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.598+0000 7f2e60a00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7f2e540713f0 0x7f2e54071850 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.599+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.5 v2:172.21.15.135:6800/2374976391 1 ==== osd_op_reply(18 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e540713f0 2024-03-06T07:35:20.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.599+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.599+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 5 ==== osd_op_reply(19 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.599+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.600+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 5 ==== osd_op_reply(20 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.600+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.601+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 6 ==== osd_op_reply(21 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.601+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.601+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 6 ==== osd_op_reply(22 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.601+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.602+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 6 ==== osd_op_reply(23 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.602+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.602+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 3 ==== osd_op_reply(24 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.602+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7f2e5406f8e0 0x7f2e5405c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.602+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.603+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7f2e5406f8e0 0x7f2e5405c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.604+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7f2e5406f8e0 0x7f2e5405c3e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.605+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.6 v2:172.21.15.135:6824/3958261964 1 ==== osd_op_reply(25 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.605+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.605+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 7 ==== osd_op_reply(26 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.605+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e540713f0 2024-03-06T07:35:20.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.606+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.5 v2:172.21.15.135:6800/2374976391 2 ==== osd_op_reply(27 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e540713f0 2024-03-06T07:35:20.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.606+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.606+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 8 ==== osd_op_reply(28 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.606+0000 7f2e61e00640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7f2e5c03a3f0 0x7f2e5c03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.606+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c03a3f0 2024-03-06T07:35:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.607+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7f2e5c03a3f0 0x7f2e5c03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.607+0000 7f2e61400640 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7f2e5c03a3f0 0x7f2e5c03a850 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.608+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.2 v2:172.21.15.97:6816/75411953 1 ==== osd_op_reply(29 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c03a3f0 2024-03-06T07:35:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.608+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.609+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.6 v2:172.21.15.135:6824/3958261964 2 ==== osd_op_reply(30 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.609+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.609+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 4 ==== osd_op_reply(31 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.609+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.610+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 3 ==== osd_op_reply(32 [pgnls start_epoch 118 out=49b] v112'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.610+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c03a3f0 2024-03-06T07:35:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.610+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.2 v2:172.21.15.97:6816/75411953 2 ==== osd_op_reply(33 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c03a3f0 2024-03-06T07:35:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.610+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.611+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 9 ==== osd_op_reply(34 [pgnls start_epoch 118 out=49b] v112'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.611+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.611+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 7 ==== osd_op_reply(35 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.611+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.612+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 5 ==== osd_op_reply(36 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.612+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.612+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 8 ==== osd_op_reply(37 [pgnls start_epoch 118 out=49b] v112'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.612+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.613+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 4 ==== osd_op_reply(38 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.613+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e540713f0 2024-03-06T07:35:20.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.614+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.5 v2:172.21.15.135:6800/2374976391 3 ==== osd_op_reply(39 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e540713f0 2024-03-06T07:35:20.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.614+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e540713f0 2024-03-06T07:35:20.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.615+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.5 v2:172.21.15.135:6800/2374976391 4 ==== osd_op_reply(40 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e540713f0 2024-03-06T07:35:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.615+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.616+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.6 v2:172.21.15.135:6824/3958261964 3 ==== osd_op_reply(41 [pgnls start_epoch 118 out=49b] v112'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.616+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.617+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 6 ==== osd_op_reply(42 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.617+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e540713f0 2024-03-06T07:35:20.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.618+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.5 v2:172.21.15.135:6800/2374976391 5 ==== osd_op_reply(43 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e540713f0 2024-03-06T07:35:20.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.618+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.619+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 5 ==== osd_op_reply(44 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.619+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.620+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 7 ==== osd_op_reply(45 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.620+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.620+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 8 ==== osd_op_reply(46 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.620+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.621+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 9 ==== osd_op_reply(47 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.621+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.622+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 10 ==== osd_op_reply(48 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.622+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.623+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 6 ==== osd_op_reply(49 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.623+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.623+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 7 ==== osd_op_reply(50 [pgnls start_epoch 118 out=49b] v112'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.623+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e54067400 2024-03-06T07:35:20.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.624+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.3 v2:172.21.15.97:6800/3169996968 9 ==== osd_op_reply(51 [pgnls start_epoch 118 out=49b] v112'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e54067400 2024-03-06T07:35:20.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.624+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x55dd97b426e0 2024-03-06T07:35:20.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.625+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.7 v2:172.21.15.135:6816/3893802268 11 ==== osd_op_reply(52 [pgnls start_epoch 118 out=49b] v112'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x55dd97b426e0 2024-03-06T07:35:20.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.625+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.626+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 10 ==== osd_op_reply(53 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.626+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.627+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 8 ==== osd_op_reply(54 [pgnls start_epoch 118 out=49b] v112'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.627+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.627+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 9 ==== osd_op_reply(55 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.627+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c03a3f0 2024-03-06T07:35:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.628+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.2 v2:172.21.15.97:6816/75411953 3 ==== osd_op_reply(56 [pgnls start_epoch 118 out=49b] v112'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c03a3f0 2024-03-06T07:35:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.628+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.628+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 11 ==== osd_op_reply(57 [pgnls start_epoch 118 out=49b] v118'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.629+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.629+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 10 ==== osd_op_reply(58 [pgnls start_epoch 118 out=49b] v112'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.629+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.630+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.6 v2:172.21.15.135:6824/3958261964 4 ==== osd_op_reply(59 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.630+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e4c01b060 2024-03-06T07:35:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.631+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.0 v2:172.21.15.97:6819/1306424436 12 ==== osd_op_reply(60 [pgnls start_epoch 118 out=49b] v112'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e4c01b060 2024-03-06T07:35:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.631+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c011900 2024-03-06T07:35:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.631+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.1 v2:172.21.15.97:6808/2322813614 7 ==== osd_op_reply(61 [pgnls start_epoch 118 out=49b] v112'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c011900 2024-03-06T07:35:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.631+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5c00da10 2024-03-06T07:35:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.632+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 <== osd.4 v2:172.21.15.135:6808/3314712387 11 ==== osd_op_reply(62 [pgnls start_epoch 118 out=49b] v118'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e54064070 con 0x7f2e5c00da10 2024-03-06T07:35:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.632+0000 7f2e61400640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.633+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 <== osd.6 v2:172.21.15.135:6824/3958261964 5 ==== osd_op_reply(63 [pgnls start_epoch 118 out=49b] v112'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e5c00d6e0 con 0x7f2e5406f8e0 2024-03-06T07:35:20.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.633+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 118 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e118) v9 -- 0x7f2e5c0108c0 con 0x7f2e540713f0 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e60a00640 1 -- 172.21.15.97:0/2113511081 <== osd.5 v2:172.21.15.135:6800/2374976391 6 ==== osd_op_reply(64 [pgnls start_epoch 118 out=49b] v112'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2e4c018030 con 0x7f2e540713f0 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f2e4c01b060 msgr2=0x7f2e4c055f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f2e4c01b060 0x7f2e4c055f30 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f2e5c011900 msgr2=0x7f2e5c010380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f2e5c011900 0x7f2e5c010380 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7f2e5c03a3f0 msgr2=0x7f2e5c03a850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7f2e5c03a3f0 0x7f2e5c03a850 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7f2e54067400 msgr2=0x7f2e54067860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7f2e54067400 0x7f2e54067860 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f2e5c00da10 msgr2=0x7f2e5c016420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7f2e5c00da10 0x7f2e5c016420 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7f2e540713f0 msgr2=0x7f2e54071850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7f2e540713f0 0x7f2e54071850 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e61e00640 1 -- 172.21.15.97:0/2113511081 reap_dead start 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7f2e5406f8e0 msgr2=0x7f2e5405c3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7f2e5406f8e0 0x7f2e5405c3e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55dd97b426e0 msgr2=0x55dd97c007f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.634+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55dd97b426e0 0x55dd97c007f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e40069eb0 msgr2=0x7f2e4006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e40069eb0 0x7f2e4006c370 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e54044770 tx=0x7f2e54005ac0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 msgr2=0x55dd97b421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 0x55dd97b421a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c008490 tx=0x7f2e4c008950 comp rx=0 tx=0).stop 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 shutdown_connections 2024-03-06T07:35:20.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7f2e5406f8e0 0x7f2e5405c3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x55dd97b426e0 0x55dd97c007f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2e40069eb0 0x7f2e4006c370 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 --2- 172.21.15.97:0/2113511081 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55dd97a8cd60 0x55dd97b421a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 >> 172.21.15.97:0/2113511081 conn(0x55dd97abbc40 msgr2=0x55dd97ad7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 shutdown_connections 2024-03-06T07:35:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.635+0000 7f2e64a35f40 1 -- 172.21.15.97:0/2113511081 wait complete. 2024-03-06T07:35:20.642 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:35:20.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.831+0000 7f3de6200640 1 Processor -- start 2024-03-06T07:35:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.831+0000 7f3de6200640 1 -- start start 2024-03-06T07:35:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.832+0000 7f3de6200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3de006cb50 0x7f3de006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.832+0000 7f3de6200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3de00b2d10 0x7f3de00b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.832+0000 7f3de6200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00717c0 con 0x7f3de00cf430 2024-03-06T07:35:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.832+0000 7f3de6200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de0071aa0 con 0x7f3de006cb50 2024-03-06T07:35:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.832+0000 7f3de6200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de0071d80 con 0x7f3de00b2d10 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3de006cb50 0x7f3de006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3de006cb50 0x7f3de006cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36628/0 (socket says 172.21.15.97:36628) 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 -- 172.21.15.97:0/1881483543 learned_addr learned my addr 172.21.15.97:0/1881483543 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddfe00640 1 --2- 172.21.15.97:0/1881483543 >> v2:172.21.15.135:3300/0 conn(0x7f3de00b2d10 0x7f3de00b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 -- 172.21.15.97:0/1881483543 >> v2:172.21.15.135:3300/0 conn(0x7f3de00b2d10 msgr2=0x7f3de00b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 --2- 172.21.15.97:0/1881483543 >> v2:172.21.15.135:3300/0 conn(0x7f3de00b2d10 0x7f3de00b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 -- 172.21.15.97:0/1881483543 >> v1:172.21.15.135:6789/0 conn(0x7f3de00cf430 legacy=0x7f3de0070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddea00640 1 -- 172.21.15.97:0/1881483543 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0072060 con 0x7f3de006cb50 2024-03-06T07:35:20.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.833+0000 7f3ddfe00640 1 --2- 172.21.15.97:0/1881483543 >> v2:172.21.15.135:3300/0 conn(0x7f3de00b2d10 0x7f3de00b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.834+0000 7f3ddea00640 1 --2- 172.21.15.97:0/1881483543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de006cfe0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3de0163aa0 tx=0x7f3dcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439247e56314431a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3dde000640 1 -- 172.21.15.97:0/1881483543 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc035020 con 0x7f3de006cb50 2024-03-06T07:35:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3dde000640 1 -- 172.21.15.97:0/1881483543 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc031870 con 0x7f3de006cb50 2024-03-06T07:35:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3dde000640 1 -- 172.21.15.97:0/1881483543 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc031c40 con 0x7f3de006cb50 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 -- 172.21.15.97:0/1881483543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 msgr2=0x7f3de006cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 --2- 172.21.15.97:0/1881483543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de006cfe0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3de0163aa0 tx=0x7f3dcc02b910 comp rx=0 tx=0).stop 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 -- 172.21.15.97:0/1881483543 shutdown_connections 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 --2- 172.21.15.97:0/1881483543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de006cfe0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 --2- 172.21.15.97:0/1881483543 >> v2:172.21.15.135:3300/0 conn(0x7f3de00b2d10 0x7f3de00b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 -- 172.21.15.97:0/1881483543 >> 172.21.15.97:0/1881483543 conn(0x7f3de00b1730 msgr2=0x7f3de00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 -- 172.21.15.97:0/1881483543 shutdown_connections 2024-03-06T07:35:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.835+0000 7f3de6200640 1 -- 172.21.15.97:0/1881483543 wait complete. 2024-03-06T07:35:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.837+0000 7f3de6200640 1 Processor -- start 2024-03-06T07:35:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.837+0000 7f3de6200640 1 -- start start 2024-03-06T07:35:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3de6200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de0146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3de6200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de00b2d10 0x7f3de0144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3de6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de00cf430 0x7f3de0143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3de6200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de0162b70 con 0x7f3de006cb50 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3de6200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de0071430 con 0x7f3de00b2d10 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3de6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de0162cf0 con 0x7f3de00cf430 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.838+0000 7f3ddf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de0146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de0146cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36644/0 (socket says 172.21.15.97:36644) 2024-03-06T07:35:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 -- 172.21.15.97:0/2436734979 learned_addr learned my addr 172.21.15.97:0/2436734979 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 -- 172.21.15.97:0/2436734979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de00cf430 msgr2=0x7f3de0143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de00cf430 0x7f3de0143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 -- 172.21.15.97:0/2436734979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de00b2d10 msgr2=0x7f3de0144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de00b2d10 0x7f3de0144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 -- 172.21.15.97:0/2436734979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de01446c0 con 0x7f3de006cb50 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3ddf400640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de0146cf0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000ba50 tx=0x7f3dd000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.839+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0019070 con 0x7f3de006cb50 2024-03-06T07:35:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.840+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd000cc10 con 0x7f3de006cb50 2024-03-06T07:35:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.840+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0014440 con 0x7f3de006cb50 2024-03-06T07:35:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.840+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de0143e70 con 0x7f3de006cb50 2024-03-06T07:35:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.840+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de01f91d0 con 0x7f3de006cb50 2024-03-06T07:35:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.841+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de004f330 con 0x7f3de006cb50 2024-03-06T07:35:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.842+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3dd001c070 con 0x7f3de006cb50 2024-03-06T07:35:20.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.842+0000 7f3dd7400640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3dc006b4d0 0x7f3dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.843+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3dd008a080 con 0x7f3de006cb50 2024-03-06T07:35:20.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.843+0000 7f3ddea00640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3dc006b4d0 0x7f3dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.843+0000 7f3ddea00640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3dc006b4d0 0x7f3dc006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc036000 tx=0x7f3dcc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:20.852+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3dd0058e70 con 0x7f3de006cb50 2024-03-06T07:35:21.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.090+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3de006ed10 con 0x7f3de006cb50 2024-03-06T07:35:21.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.091+0000 7f3dd7400640 1 -- 172.21.15.97:0/2436734979 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f3dd0058770 con 0x7f3de006cb50 2024-03-06T07:35:21.093 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:21.093 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:21.094 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:21.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3dc006b4d0 msgr2=0x7f3dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3dc006b4d0 0x7f3dc006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc036000 tx=0x7f3dcc030000 comp rx=0 tx=0).stop 2024-03-06T07:35:21.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 msgr2=0x7f3de0146cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de0146cf0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000ba50 tx=0x7f3dd000bf10 comp rx=0 tx=0).stop 2024-03-06T07:35:21.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 shutdown_connections 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3dc006b4d0 0x7f3dc006d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3de00cf430 0x7f3de0143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de00b2d10 0x7f3de0144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 --2- 172.21.15.97:0/2436734979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de006cb50 0x7f3de0146cf0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 >> 172.21.15.97:0/2436734979 conn(0x7f3de00b1730 msgr2=0x7f3de0160f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 shutdown_connections 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.094+0000 7f3de6200640 1 -- 172.21.15.97:0/2436734979 wait complete. 2024-03-06T07:35:21.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:35:21.346 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:35:21.346 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:35:21.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:21.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.531+0000 7fd5e0600640 1 Processor -- start 2024-03-06T07:35:21.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.531+0000 7fd5e0600640 1 -- start start 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5e0600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd5d815f9d0 0x7fd5d815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5e0600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d806cef0 con 0x7fd5d8070b10 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5e0600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d806d1d0 con 0x7fd5d8144c00 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5e0600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d806d4b0 con 0x7fd5d815f9d0 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5dd200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd5d815f9d0 0x7fd5d815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5de600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd5d8070b10 0x7fd5d8070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36644/0 (socket says 172.21.15.97:36644) 2024-03-06T07:35:21.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.533+0000 7fd5de600640 1 -- 172.21.15.97:0/2184678393 learned_addr learned my addr 172.21.15.97:0/2184678393 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:21.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.534+0000 7fd5dd200640 1 -- 172.21.15.97:0/2184678393 >> v1:172.21.15.135:6789/0 conn(0x7fd5d8144c00 legacy=0x7fd5d806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.534+0000 7fd5dd200640 1 -- 172.21.15.97:0/2184678393 >> v1:172.21.15.97:6789/0 conn(0x7fd5d8070b10 legacy=0x7fd5d8070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.534+0000 7fd5d7e00640 1 -- 172.21.15.97:0/2184678393 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245374573 0 0) 0x7fd5d806cef0 con 0x7fd5d8070b10 2024-03-06T07:35:21.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.534+0000 7fd5dd200640 1 -- 172.21.15.97:0/2184678393 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d806d790 con 0x7fd5d815f9d0 2024-03-06T07:35:21.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.535+0000 7fd5dd200640 1 --2- 172.21.15.97:0/2184678393 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d815fe60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40097a0 tx=0x7fd5c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0a5b0a07972c8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.535+0000 7fd5d7e00640 1 -- 172.21.15.97:0/2184678393 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4035020 con 0x7fd5d815f9d0 2024-03-06T07:35:21.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.535+0000 7fd5d7e00640 1 -- 172.21.15.97:0/2184678393 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c4031830 con 0x7fd5d815f9d0 2024-03-06T07:35:21.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.535+0000 7fd5d7e00640 1 -- 172.21.15.97:0/2184678393 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4031c00 con 0x7fd5d815f9d0 2024-03-06T07:35:21.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 -- 172.21.15.97:0/2184678393 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 msgr2=0x7fd5d815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 --2- 172.21.15.97:0/2184678393 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d815fe60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40097a0 tx=0x7fd5c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 -- 172.21.15.97:0/2184678393 shutdown_connections 2024-03-06T07:35:21.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 --2- 172.21.15.97:0/2184678393 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d815fe60 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 -- 172.21.15.97:0/2184678393 >> 172.21.15.97:0/2184678393 conn(0x7fd5d80b1730 msgr2=0x7fd5d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 -- 172.21.15.97:0/2184678393 shutdown_connections 2024-03-06T07:35:21.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.536+0000 7fd5e0600640 1 -- 172.21.15.97:0/2184678393 wait complete. 2024-03-06T07:35:21.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.538+0000 7fd5e0600640 1 Processor -- start 2024-03-06T07:35:21.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.538+0000 7fd5e0600640 1 -- start start 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5e0600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d8070b10 0x7fd5d81407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5e0600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5d8144c00 0x7fd5d813cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5e0600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d8142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5e0600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d806ccd0 con 0x7fd5d815f9d0 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5e0600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d806cb50 con 0x7fd5d8070b10 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5e0600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d806df90 con 0x7fd5d8144c00 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d8142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d8142190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36654/0 (socket says 172.21.15.97:36654) 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 -- 172.21.15.97:0/1650290631 learned_addr learned my addr 172.21.15.97:0/1650290631 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5ddc00640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d8070b10 0x7fd5d81407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5dd200640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5d8144c00 0x7fd5d813cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 -- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5d8144c00 msgr2=0x7fd5d813cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5d8144c00 0x7fd5d813cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 -- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d8070b10 msgr2=0x7fd5d81407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d8070b10 0x7fd5d81407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.539+0000 7fd5de600640 1 -- 172.21.15.97:0/1650290631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d8143130 con 0x7fd5d815f9d0 2024-03-06T07:35:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5ddc00640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d8070b10 0x7fd5d81407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5dd200640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5d8144c00 0x7fd5d813cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:21.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5de600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d8142190 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000c7a0 tx=0x7fd5d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0018070 con 0x7fd5d815f9d0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d000a640 con 0x7fd5d815f9d0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d000aab0 con 0x7fd5d815f9d0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d8143410 con 0x7fd5d815f9d0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.540+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d81426d0 con 0x7fd5d815f9d0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.542+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d804f330 con 0x7fd5d815f9d0 2024-03-06T07:35:21.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.542+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5d000ac50 con 0x7fd5d815f9d0 2024-03-06T07:35:21.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.543+0000 7fd5d5600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.543+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd5d0089930 con 0x7fd5d815f9d0 2024-03-06T07:35:21.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.543+0000 7fd5ddc00640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.544+0000 7fd5ddc00640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc003eb0 tx=0x7fd5cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.547+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5d0058710 con 0x7fd5d815f9d0 2024-03-06T07:35:21.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.780+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5d8072890 con 0x7fd5d815f9d0 2024-03-06T07:35:21.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.781+0000 7fd5d5600640 1 -- 172.21.15.97:0/1650290631 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fd5d0058010 con 0x7fd5d815f9d0 2024-03-06T07:35:21.784 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:21.784 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:21.784 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:21.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd5b806b4d0 msgr2=0x7fd5b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc003eb0 tx=0x7fd5cc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 msgr2=0x7fd5d8142190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d8142190 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000c7a0 tx=0x7fd5d000cc60 comp rx=0 tx=0).stop 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 shutdown_connections 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5d815f9d0 0x7fd5d8142190 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5d8144c00 0x7fd5d813cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 --2- 172.21.15.97:0/1650290631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d8070b10 0x7fd5d81407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 >> 172.21.15.97:0/1650290631 conn(0x7fd5d80b1730 msgr2=0x7fd5d80b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 shutdown_connections 2024-03-06T07:35:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:21.785+0000 7fd5e0600640 1 -- 172.21.15.97:0/1650290631 wait complete. 2024-03-06T07:35:21.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:22.028 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:35:22.029 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:35:22.029 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:22.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:22.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:22.269 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:22.270 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:35:22.270 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:22.325 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:35:22.325 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:22.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:22.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.626+0000 7f8eaf000640 1 Processor -- start 2024-03-06T07:35:22.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.626+0000 7f8eaf000640 1 -- start start 2024-03-06T07:35:22.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8eaf000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8ea8144c00 0x7f8ea806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8eaf000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8eaf000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ea80ce0a0 0x7f8ea80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8eaf000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea8160370 con 0x7f8ea80ce0a0 2024-03-06T07:35:22.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8eaf000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea806ee70 con 0x7f8ea8144c00 2024-03-06T07:35:22.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8eaf000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea806cef0 con 0x7f8ea815f9d0 2024-03-06T07:35:22.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8ead000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ea80ce0a0 0x7f8ea80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8ead000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8ea80ce0a0 0x7f8ea80ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36668/0 (socket says 172.21.15.97:36668) 2024-03-06T07:35:22.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.628+0000 7f8ead000640 1 -- 172.21.15.97:0/939488514 learned_addr learned my addr 172.21.15.97:0/939488514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8ead000640 1 -- 172.21.15.97:0/939488514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 msgr2=0x7f8ea815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8ea7e00640 1 --2- 172.21.15.97:0/939488514 >> v2:172.21.15.135:3300/0 conn(0x7f8ea8144c00 0x7f8ea806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8e9f400640 1 --2- 172.21.15.97:0/939488514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8ead000640 1 --2- 172.21.15.97:0/939488514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8ead000640 1 -- 172.21.15.97:0/939488514 >> v2:172.21.15.135:3300/0 conn(0x7f8ea8144c00 msgr2=0x7f8ea806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8ead000640 1 --2- 172.21.15.97:0/939488514 >> v2:172.21.15.135:3300/0 conn(0x7f8ea8144c00 0x7f8ea806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.629+0000 7f8ead000640 1 -- 172.21.15.97:0/939488514 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea806d1d0 con 0x7f8ea80ce0a0 2024-03-06T07:35:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.630+0000 7f8ead000640 1 --2- 172.21.15.97:0/939488514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea80ce4f0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80b25f0 tx=0x7f8ea002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824c92f4d957e95f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.630+0000 7f8ea7400640 1 -- 172.21.15.97:0/939488514 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea002ca80 con 0x7f8ea80ce0a0 2024-03-06T07:35:22.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.630+0000 7f8ea7400640 1 -- 172.21.15.97:0/939488514 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea002cc20 con 0x7f8ea80ce0a0 2024-03-06T07:35:22.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.630+0000 7f8ea7400640 1 -- 172.21.15.97:0/939488514 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea00343f0 con 0x7f8ea80ce0a0 2024-03-06T07:35:22.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 -- 172.21.15.97:0/939488514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea80ce0a0 msgr2=0x7f8ea80ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 --2- 172.21.15.97:0/939488514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea80ce4f0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80b25f0 tx=0x7f8ea002bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 -- 172.21.15.97:0/939488514 shutdown_connections 2024-03-06T07:35:22.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 --2- 172.21.15.97:0/939488514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea80ce4f0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 --2- 172.21.15.97:0/939488514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 --2- 172.21.15.97:0/939488514 >> v2:172.21.15.135:3300/0 conn(0x7f8ea8144c00 0x7f8ea806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 -- 172.21.15.97:0/939488514 >> 172.21.15.97:0/939488514 conn(0x7f8ea80b1730 msgr2=0x7f8ea80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 -- 172.21.15.97:0/939488514 shutdown_connections 2024-03-06T07:35:22.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.631+0000 7f8eaf000640 1 -- 172.21.15.97:0/939488514 wait complete. 2024-03-06T07:35:22.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.633+0000 7f8eaf000640 1 Processor -- start 2024-03-06T07:35:22.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 -- start start 2024-03-06T07:35:22.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea813cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 0x7f8ea815eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8160730 con 0x7f8ea8144c00 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea806d790 con 0x7f8ea80ce0a0 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.634+0000 7f8eaf000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea806cb50 con 0x7f8ea815f9d0 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 0x7f8ea815eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 0x7f8ea815eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36682/0 (socket says 172.21.15.97:36682) 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 -- 172.21.15.97:0/1687645247 learned_addr learned my addr 172.21.15.97:0/1687645247 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8ead000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8ea7e00640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 -- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 msgr2=0x7f8ea815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 -- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ea80ce0a0 msgr2=0x7f8ea813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea813cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 -- 172.21.15.97:0/1687645247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea813ba20 con 0x7f8ea8144c00 2024-03-06T07:35:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.635+0000 7f8e9f400640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 0x7f8ea815eba0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000c7f0 tx=0x7f8e9000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.636+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90015070 con 0x7f8ea8144c00 2024-03-06T07:35:22.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.636+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90004240 con 0x7f8ea8144c00 2024-03-06T07:35:22.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.636+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90007620 con 0x7f8ea8144c00 2024-03-06T07:35:22.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.636+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea813bd00 con 0x7f8ea8144c00 2024-03-06T07:35:22.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.636+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea813c2c0 con 0x7f8ea8144c00 2024-03-06T07:35:22.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.637+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea804f330 con 0x7f8ea8144c00 2024-03-06T07:35:22.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.639+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e900079d0 con 0x7f8ea8144c00 2024-03-06T07:35:22.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.640+0000 7f8ea4c00640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e8406b4d0 0x7f8e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.640+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8e9008a7b0 con 0x7f8ea8144c00 2024-03-06T07:35:22.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.640+0000 7f8ea7e00640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e8406b4d0 0x7f8e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.641+0000 7f8ea7e00640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e8406b4d0 0x7f8e8406d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8e98006fd0 tx=0x7f8e98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.645+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e90059500 con 0x7f8ea8144c00 2024-03-06T07:35:22.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.839+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ea80b4cb0 con 0x7f8ea8144c00 2024-03-06T07:35:22.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.842+0000 7f8ea4c00640 1 -- 172.21.15.97:0/1687645247 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f8e90058e00 con 0x7f8ea8144c00 2024-03-06T07:35:22.844 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:22.845 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":118,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:29.791334+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.135:6833/322807700":"2024-03-07T07:34:20.730206+0000","172.21.15.135:6832/322807700":"2024-03-07T07:34:20.730206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.844+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e8406b4d0 msgr2=0x7f8e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.844+0000 7f8eaf000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e8406b4d0 0x7f8e8406d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8e98006fd0 tx=0x7f8e98008040 comp rx=0 tx=0).stop 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.844+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 msgr2=0x7f8ea815eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.844+0000 7f8eaf000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 0x7f8ea815eba0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000c7f0 tx=0x7f8e9000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.844+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 shutdown_connections 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e8406b4d0 0x7f8e8406d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8ea815f9d0 0x7f8ea815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ea8144c00 0x7f8ea815eba0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 --2- 172.21.15.97:0/1687645247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ea80ce0a0 0x7f8ea813cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 >> 172.21.15.97:0/1687645247 conn(0x7f8ea80b1730 msgr2=0x7f8ea80ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 shutdown_connections 2024-03-06T07:35:22.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:22.845+0000 7f8eaf000640 1 -- 172.21.15.97:0/1687645247 wait complete. 2024-03-06T07:35:22.865 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:03.618857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:04.895815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:22.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:23.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.063+0000 7f931a600640 1 Processor -- start 2024-03-06T07:35:23.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.063+0000 7f931a600640 1 -- start start 2024-03-06T07:35:23.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.064+0000 7f931a600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f93140b3f70 0x7f9314160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f931a600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9314070ff0 0x7f93140ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f931a600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9314071440 con 0x7f931415fc10 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f931a600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93140b46c0 con 0x7f9314070ff0 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f931a600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93140b49a0 con 0x7f93140b3f70 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9314070ff0 0x7f93140ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9314070ff0 0x7f93140ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36684/0 (socket says 172.21.15.97:36684) 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 -- 172.21.15.97:0/911708015 learned_addr learned my addr 172.21.15.97:0/911708015 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9312a00640 1 --2- 172.21.15.97:0/911708015 >> v2:172.21.15.135:3300/0 conn(0x7f93140b3f70 0x7f9314160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 -- 172.21.15.97:0/911708015 >> v2:172.21.15.135:3300/0 conn(0x7f93140b3f70 msgr2=0x7f9314160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 --2- 172.21.15.97:0/911708015 >> v2:172.21.15.135:3300/0 conn(0x7f93140b3f70 0x7f9314160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 -- 172.21.15.97:0/911708015 >> v1:172.21.15.135:6789/0 conn(0x7f931415fc10 legacy=0x7f931415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9313e00640 1 -- 172.21.15.97:0/911708015 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314072c90 con 0x7f9314070ff0 2024-03-06T07:35:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.065+0000 7f9312a00640 1 --2- 172.21.15.97:0/911708015 >> v2:172.21.15.135:3300/0 conn(0x7f93140b3f70 0x7f9314160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.066+0000 7f9313e00640 1 --2- 172.21.15.97:0/911708015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9314070ff0 0x7f93140ce4a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f9314074a50 tx=0x7f930c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111ba79418bf7ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.067+0000 7f9312000640 1 -- 172.21.15.97:0/911708015 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c02ca80 con 0x7f9314070ff0 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.067+0000 7f9312000640 1 -- 172.21.15.97:0/911708015 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c02cc20 con 0x7f9314070ff0 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.067+0000 7f9312000640 1 -- 172.21.15.97:0/911708015 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c0343f0 con 0x7f9314070ff0 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.067+0000 7f931a600640 1 -- 172.21.15.97:0/911708015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9314070ff0 msgr2=0x7f93140ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.067+0000 7f931a600640 1 --2- 172.21.15.97:0/911708015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9314070ff0 0x7f93140ce4a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f9314074a50 tx=0x7f930c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.068+0000 7f931a600640 1 -- 172.21.15.97:0/911708015 shutdown_connections 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.068+0000 7f931a600640 1 --2- 172.21.15.97:0/911708015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9314070ff0 0x7f93140ce4a0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.068+0000 7f931a600640 1 --2- 172.21.15.97:0/911708015 >> v2:172.21.15.135:3300/0 conn(0x7f93140b3f70 0x7f9314160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.068+0000 7f931a600640 1 -- 172.21.15.97:0/911708015 >> 172.21.15.97:0/911708015 conn(0x7f93140b1940 msgr2=0x7f93140b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.068+0000 7f931a600640 1 -- 172.21.15.97:0/911708015 shutdown_connections 2024-03-06T07:35:23.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.068+0000 7f931a600640 1 -- 172.21.15.97:0/911708015 wait complete. 2024-03-06T07:35:23.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.069+0000 7f931a600640 1 Processor -- start 2024-03-06T07:35:23.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.069+0000 7f931a600640 1 -- start start 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.070+0000 7f931a600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9314070ff0 0x7f931414c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.070+0000 7f931a600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93140b3f70 0x7f931406e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.070+0000 7f931a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 0x7f93141514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.070+0000 7f931a600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9314072a30 con 0x7f931415fc10 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.070+0000 7f931a600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93140728b0 con 0x7f93140b3f70 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.070+0000 7f931a600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931415c2e0 con 0x7f9314070ff0 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 0x7f93141514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 0x7f93141514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36688/0 (socket says 172.21.15.97:36688) 2024-03-06T07:35:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 -- 172.21.15.97:0/1382641409 learned_addr learned my addr 172.21.15.97:0/1382641409 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:23.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 -- 172.21.15.97:0/1382641409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9314070ff0 msgr2=0x7f931414c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9314070ff0 0x7f931414c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 -- 172.21.15.97:0/1382641409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93140b3f70 msgr2=0x7f931406e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93140b3f70 0x7f931406e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 -- 172.21.15.97:0/1382641409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931406c990 con 0x7f931415fc10 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.071+0000 7f9313e00640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 0x7f93141514b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f930c009f40 tx=0x7f930c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.072+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c043020 con 0x7f931415fc10 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.072+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c03e450 con 0x7f931415fc10 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.072+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c03e840 con 0x7f931415fc10 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.072+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931406cc70 con 0x7f931415fc10 2024-03-06T07:35:23.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.072+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931406d230 con 0x7f931415fc10 2024-03-06T07:35:23.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.073+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93140ad4c0 con 0x7f931415fc10 2024-03-06T07:35:23.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.074+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f930c046020 con 0x7f931415fc10 2024-03-06T07:35:23.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.075+0000 7f930b400640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f92f406b4d0 0x7f92f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.075+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f930c0acf70 con 0x7f931415fc10 2024-03-06T07:35:23.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.076+0000 7f9313400640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f92f406b4d0 0x7f92f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.077+0000 7f9313400640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f92f406b4d0 0x7f92f406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9304006fd0 tx=0x7f9304006bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.079+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93140ad4c0 con 0x7f931415fc10 2024-03-06T07:35:23.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.291+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f931404f330 con 0x7f931415fc10 2024-03-06T07:35:23.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.292+0000 7f930b400640 1 -- 172.21.15.97:0/1382641409 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f930c04d020 con 0x7f931415fc10 2024-03-06T07:35:23.294 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:35:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.294+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f92f406b4d0 msgr2=0x7f92f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.294+0000 7f931a600640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f92f406b4d0 0x7f92f406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9304006fd0 tx=0x7f9304006bc0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 msgr2=0x7f93141514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 0x7f93141514b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f930c009f40 tx=0x7f930c004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 shutdown_connections 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f92f406b4d0 0x7f92f406d990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f931415fc10 0x7f93141514b0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93140b3f70 0x7f931406e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 --2- 172.21.15.97:0/1382641409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9314070ff0 0x7f931414c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 >> 172.21.15.97:0/1382641409 conn(0x7f93140b1940 msgr2=0x7f93140acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 shutdown_connections 2024-03-06T07:35:23.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.295+0000 7f931a600640 1 -- 172.21.15.97:0/1382641409 wait complete. 2024-03-06T07:35:23.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:23.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.542+0000 7f65c9a00640 1 Processor -- start 2024-03-06T07:35:23.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.542+0000 7f65c9a00640 1 -- start start 2024-03-06T07:35:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.543+0000 7f65c9a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 0x7f65c406bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65c9a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c40b2a40 con 0x7f65c406b480 2024-03-06T07:35:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65c9a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c415eb80 con 0x7f65c406afc0 2024-03-06T07:35:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65c9a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40b5e40 con 0x7f65c415e7a0 2024-03-06T07:35:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65bbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f65c406b480 0x7f65c40b5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36652/0 (socket says 172.21.15.97:36652) 2024-03-06T07:35:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65bbe00640 1 -- 172.21.15.97:0/3988968351 learned_addr learned my addr 172.21.15.97:0/3988968351 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65c2a00640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 0x7f65c406bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.544+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547112638 0 0) 0x7f65c40b2a40 con 0x7f65c406b480 2024-03-06T07:35:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a8003520 con 0x7f65c406b480 2024-03-06T07:35:23.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183593179 0 0) 0x7f65a8003520 con 0x7f65c406b480 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 msgr2=0x7f65c406bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 0x7f65c406bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 >> v1:172.21.15.135:6789/0 conn(0x7f65c406afc0 legacy=0x7f65c40b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c40b14d0 con 0x7f65c406b480 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f65ac001670 con 0x7f65c406b480 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 >> v1:172.21.15.97:6789/0 conn(0x7f65c406b480 legacy=0x7f65c40b5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65a80039e0 0x7f65a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 0x7f65a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65a8004d40 0x7f65a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40b5e40 con 0x7f65a8004d40 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800dd90 con 0x7f65a80039e0 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800df90 con 0x7f65c415e7a0 2024-03-06T07:35:23.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.545+0000 7f65c3400640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65a8004d40 0x7f65a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65bbe00640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 0x7f65a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65c2a00640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65a80039e0 0x7f65a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65c3400640 1 -- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 msgr2=0x7f65a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65c3400640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c415e7a0 0x7f65a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65c3400640 1 -- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65a80039e0 msgr2=0x7f65a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65c3400640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65a80039e0 0x7f65a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.546+0000 7f65c3400640 1 -- 172.21.15.97:0/3988968351 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8003520 con 0x7f65a8004d40 2024-03-06T07:35:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.547+0000 7f65c3400640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65a8004d40 0x7f65a80051c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f65bc001e50 tx=0x7f65a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24078d5a478722d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.547+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc001a70 con 0x7f65a8004d40 2024-03-06T07:35:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.547+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc003b70 con 0x7f65a8004d40 2024-03-06T07:35:23.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.547+0000 7f65c2000640 1 -- 172.21.15.97:0/3988968351 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0050d0 con 0x7f65a8004d40 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c9a00640 1 -- 172.21.15.97:0/3988968351 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65a8004d40 msgr2=0x7f65a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c9a00640 1 --2- 172.21.15.97:0/3988968351 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65a8004d40 0x7f65a80051c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f65bc001e50 tx=0x7f65a8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c3400640 1 -- 172.21.15.97:0/3988968351 reap_dead start 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c9a00640 1 -- 172.21.15.97:0/3988968351 shutdown_connections 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c9a00640 1 -- 172.21.15.97:0/3988968351 >> 172.21.15.97:0/3988968351 conn(0x7f65c40b1e80 msgr2=0x7f65c40b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c9a00640 1 -- 172.21.15.97:0/3988968351 shutdown_connections 2024-03-06T07:35:23.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.548+0000 7f65c9a00640 1 -- 172.21.15.97:0/3988968351 wait complete. 2024-03-06T07:35:23.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.550+0000 7f65c9a00640 1 Processor -- start 2024-03-06T07:35:23.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.550+0000 7f65c9a00640 1 -- start start 2024-03-06T07:35:23.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65c9a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c41571e0 0x7f65c40cee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65c9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 0x7f65c40cf390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65c9a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65c4158c30 0x7f65c41590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65c9a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c4073360 con 0x7f65c40cdab0 2024-03-06T07:35:23.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65c9a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40b2cb0 con 0x7f65c4158c30 2024-03-06T07:35:23.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65c9a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40b19f0 con 0x7f65c41571e0 2024-03-06T07:35:23.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 0x7f65c40cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 0x7f65c40cf390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36710/0 (socket says 172.21.15.97:36710) 2024-03-06T07:35:23.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.551+0000 7f65bbe00640 1 -- 172.21.15.97:0/2181588968 learned_addr learned my addr 172.21.15.97:0/2181588968 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:23.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65c2a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c41571e0 0x7f65c40cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65c3400640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65c4158c30 0x7f65c41590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65bbe00640 1 -- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c41571e0 msgr2=0x7f65c40cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65bbe00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c41571e0 0x7f65c40cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65bbe00640 1 -- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65c4158c30 msgr2=0x7f65c41590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65bbe00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65c4158c30 0x7f65c41590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65bbe00640 1 -- 172.21.15.97:0/2181588968 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c40cfa10 con 0x7f65c40cdab0 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65bbe00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 0x7f65c40cf390 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00eb50 tx=0x7f65ac00c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00ec40 con 0x7f65c40cdab0 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac018070 con 0x7f65c40cdab0 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00cbc0 con 0x7f65c40cdab0 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.552+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c40cccf0 con 0x7f65c40cdab0 2024-03-06T07:35:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.553+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c40cd2b0 con 0x7f65c40cdab0 2024-03-06T07:35:23.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.554+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c41595f0 con 0x7f65c40cdab0 2024-03-06T07:35:23.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.554+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65ac01d460 con 0x7f65c40cdab0 2024-03-06T07:35:23.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.555+0000 7f65baa00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65a006b4d0 0x7f65a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.555+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f65ac08a1d0 con 0x7f65c40cdab0 2024-03-06T07:35:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.556+0000 7f65c2a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65a006b4d0 0x7f65a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.556+0000 7f65c2a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65a006b4d0 0x7f65a006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009a10 tx=0x7f65b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.560+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65ac058f20 con 0x7f65c40cdab0 2024-03-06T07:35:23.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.755+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65c404f330 con 0x7f65c40cdab0 2024-03-06T07:35:23.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.756+0000 7f65baa00640 1 -- 172.21.15.97:0/2181588968 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f65ac058820 con 0x7f65c40cdab0 2024-03-06T07:35:23.758 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:35:23.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65a006b4d0 msgr2=0x7f65a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65a006b4d0 0x7f65a006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009a10 tx=0x7f65b4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 msgr2=0x7f65c40cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 0x7f65c40cf390 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00eb50 tx=0x7f65ac00c4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 shutdown_connections 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f65a006b4d0 0x7f65a006d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65c4158c30 0x7f65c41590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65c40cdab0 0x7f65c40cf390 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 --2- 172.21.15.97:0/2181588968 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f65c41571e0 0x7f65c40cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.758+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 >> 172.21.15.97:0/2181588968 conn(0x7f65c40b1e80 msgr2=0x7f65c40ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.759+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 shutdown_connections 2024-03-06T07:35:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.759+0000 7f65c9a00640 1 -- 172.21.15.97:0/2181588968 wait complete. 2024-03-06T07:35:23.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.976+0000 7f43b9800640 1 Processor -- start 2024-03-06T07:35:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.976+0000 7f43b9800640 1 -- start start 2024-03-06T07:35:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.977+0000 7f43b9800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f43b40d07f0 0x7f43b40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.977+0000 7f43b9800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b415e450 0x7f43b40b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.977+0000 7f43b9800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f43b40d1bf0 0x7f43b40b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.977+0000 7f43b9800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40d4930 con 0x7f43b40d1bf0 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.977+0000 7f43b9800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40b3730 con 0x7f43b40d07f0 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.977+0000 7f43b9800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4067b20 con 0x7f43b415e450 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f43b40d1bf0 0x7f43b40b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f43b40d1bf0 0x7f43b40b49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41816/0 (socket says 172.21.15.97:41816) 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 -- 172.21.15.97:0/3778414657 learned_addr learned my addr 172.21.15.97:0/3778414657 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b2a00640 1 --2- 172.21.15.97:0/3778414657 >> v2:172.21.15.135:3300/0 conn(0x7f43b40d07f0 0x7f43b40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43aa000640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b415e450 0x7f43b40b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 -- 172.21.15.97:0/3778414657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b415e450 msgr2=0x7f43b40b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b415e450 0x7f43b40b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 -- 172.21.15.97:0/3778414657 >> v2:172.21.15.135:3300/0 conn(0x7f43b40d07f0 msgr2=0x7f43b40d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 --2- 172.21.15.97:0/3778414657 >> v2:172.21.15.135:3300/0 conn(0x7f43b40d07f0 0x7f43b40d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43b3400640 1 -- 172.21.15.97:0/3778414657 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40d1110 con 0x7f43b40d1bf0 2024-03-06T07:35:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.978+0000 7f43aa000640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b415e450 0x7f43b40b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.979+0000 7f43b3400640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d1bf0 0x7f43b40b49c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f43b408d7b0 tx=0x7f43ac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7533ef2976589d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b2000640 1 -- 172.21.15.97:0/3778414657 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac02c9e0 con 0x7f43b40d1bf0 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b2000640 1 -- 172.21.15.97:0/3778414657 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ac02cb80 con 0x7f43b40d1bf0 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b2000640 1 -- 172.21.15.97:0/3778414657 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac0343f0 con 0x7f43b40d1bf0 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 -- 172.21.15.97:0/3778414657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d1bf0 msgr2=0x7f43b40b49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d1bf0 0x7f43b40b49c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f43b408d7b0 tx=0x7f43ac02be40 comp rx=0 tx=0).stop 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 -- 172.21.15.97:0/3778414657 shutdown_connections 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d1bf0 0x7f43b40b49c0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 --2- 172.21.15.97:0/3778414657 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b415e450 0x7f43b40b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 --2- 172.21.15.97:0/3778414657 >> v2:172.21.15.135:3300/0 conn(0x7f43b40d07f0 0x7f43b40d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 -- 172.21.15.97:0/3778414657 >> 172.21.15.97:0/3778414657 conn(0x7f43b40d3e10 msgr2=0x7f43b40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 -- 172.21.15.97:0/3778414657 shutdown_connections 2024-03-06T07:35:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.980+0000 7f43b9800640 1 -- 172.21.15.97:0/3778414657 wait complete. 2024-03-06T07:35:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.983+0000 7f43b9800640 1 Processor -- start 2024-03-06T07:35:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.983+0000 7f43b9800640 1 -- start start 2024-03-06T07:35:23.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b9800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 0x7f43b41e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b9800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b40d1bf0 0x7f43b40cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b9800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f43b415e450 0x7f43b40c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b9800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40b3c80 con 0x7f43b40d07f0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b9800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40b39a0 con 0x7f43b415e450 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b9800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40b3f60 con 0x7f43b40d1bf0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 0x7f43b41e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 0x7f43b41e9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41822/0 (socket says 172.21.15.97:41822) 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.985+0000 7f43b2a00640 1 -- 172.21.15.97:0/1163945717 learned_addr learned my addr 172.21.15.97:0/1163945717 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43aa000640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b40d1bf0 0x7f43b40cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b3400640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f43b415e450 0x7f43b40c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b2a00640 1 -- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b40d1bf0 msgr2=0x7f43b40cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b2a00640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b40d1bf0 0x7f43b40cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b2a00640 1 -- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f43b415e450 msgr2=0x7f43b40c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b2a00640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f43b415e450 0x7f43b40c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b2a00640 1 -- 172.21.15.97:0/1163945717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b415f4c0 con 0x7f43b40d07f0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b2a00640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 0x7f43b41e9b70 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f43a400d800 tx=0x7f43a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4018020 con 0x7f43b40d07f0 2024-03-06T07:35:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40096a0 con 0x7f43b40d07f0 2024-03-06T07:35:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b415f7a0 con 0x7f43b40d07f0 2024-03-06T07:35:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4009a70 con 0x7f43b40d07f0 2024-03-06T07:35:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.986+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b415fd60 con 0x7f43b40d07f0 2024-03-06T07:35:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.988+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404f330 con 0x7f43b40d07f0 2024-03-06T07:35:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.988+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43a401b020 con 0x7f43b40d07f0 2024-03-06T07:35:23.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.989+0000 7f43ab400640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f439006b500 0x7f439006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.989+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f43a408a4b0 con 0x7f43b40d07f0 2024-03-06T07:35:23.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.990+0000 7f43aa000640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f439006b500 0x7f439006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.990+0000 7f43aa000640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f439006b500 0x7f439006d9c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f43b40a1c70 tx=0x7f439c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:23.997+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43a4022030 con 0x7f43b40d07f0 2024-03-06T07:35:24.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.195+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43b40d0d70 con 0x7f43b40d07f0 2024-03-06T07:35:24.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.196+0000 7f43ab400640 1 -- 172.21.15.97:0/1163945717 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43a4059330 con 0x7f43b40d07f0 2024-03-06T07:35:24.198 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:35:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.198+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f439006b500 msgr2=0x7f439006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.198+0000 7f43b9800640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f439006b500 0x7f439006d9c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f43b40a1c70 tx=0x7f439c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 msgr2=0x7f43b41e9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 0x7f43b41e9b70 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f43a400d800 tx=0x7f43a400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 shutdown_connections 2024-03-06T07:35:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f439006b500 0x7f439006d9c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f43b415e450 0x7f43b40c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f43b40d1bf0 0x7f43b40cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 --2- 172.21.15.97:0/1163945717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f43b40d07f0 0x7f43b41e9b70 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 >> 172.21.15.97:0/1163945717 conn(0x7f43b40d3e10 msgr2=0x7f43b40d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 shutdown_connections 2024-03-06T07:35:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.199+0000 7f43b9800640 1 -- 172.21.15.97:0/1163945717 wait complete. 2024-03-06T07:35:24.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:35:24.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.414+0000 7f37bac00640 1 Processor -- start 2024-03-06T07:35:24.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.414+0000 7f37bac00640 1 -- start start 2024-03-06T07:35:24.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37bac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4071bc0 0x7f37b406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37bac00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f37b4071fa0 0x7f37b415be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37bac00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f37b4151800 0x7f37b4151c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37bac00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b40b5320 con 0x7f37b4071fa0 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37bac00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b406dc60 con 0x7f37b4151800 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37bac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b406df40 con 0x7f37b4071bc0 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.416+0000 7f37b3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f37b4071fa0 0x7f37b415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f37b4071fa0 0x7f37b415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41830/0 (socket says 172.21.15.97:41830) 2024-03-06T07:35:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 -- 172.21.15.97:0/1045368785 learned_addr learned my addr 172.21.15.97:0/1045368785 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 -- 172.21.15.97:0/1045368785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4071bc0 msgr2=0x7f37b406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 --2- 172.21.15.97:0/1045368785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4071bc0 0x7f37b406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b8c00640 1 --2- 172.21.15.97:0/1045368785 >> v2:172.21.15.135:3300/0 conn(0x7f37b4151800 0x7f37b4151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 -- 172.21.15.97:0/1045368785 >> v2:172.21.15.135:3300/0 conn(0x7f37b4151800 msgr2=0x7f37b4151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 --2- 172.21.15.97:0/1045368785 >> v2:172.21.15.135:3300/0 conn(0x7f37b4151800 0x7f37b4151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.417+0000 7f37b3400640 1 -- 172.21.15.97:0/1045368785 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b406e220 con 0x7f37b4071fa0 2024-03-06T07:35:24.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.418+0000 7f37b3400640 1 --2- 172.21.15.97:0/1045368785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b415be80 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f37b406c550 tx=0x7f37a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a935a4855f7487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.418+0000 7f37b2a00640 1 -- 172.21.15.97:0/1045368785 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0035020 con 0x7f37b4071fa0 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37b2a00640 1 -- 172.21.15.97:0/1045368785 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a0031870 con 0x7f37b4071fa0 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37b2a00640 1 -- 172.21.15.97:0/1045368785 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0031c40 con 0x7f37b4071fa0 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 -- 172.21.15.97:0/1045368785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 msgr2=0x7f37b415be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 --2- 172.21.15.97:0/1045368785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b415be80 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f37b406c550 tx=0x7f37a002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 -- 172.21.15.97:0/1045368785 shutdown_connections 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 --2- 172.21.15.97:0/1045368785 >> v2:172.21.15.135:3300/0 conn(0x7f37b4151800 0x7f37b4151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 --2- 172.21.15.97:0/1045368785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b415be80 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 --2- 172.21.15.97:0/1045368785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4071bc0 0x7f37b406d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 -- 172.21.15.97:0/1045368785 >> 172.21.15.97:0/1045368785 conn(0x7f37b40b2620 msgr2=0x7f37b40b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 -- 172.21.15.97:0/1045368785 shutdown_connections 2024-03-06T07:35:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.419+0000 7f37bac00640 1 -- 172.21.15.97:0/1045368785 wait complete. 2024-03-06T07:35:24.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.421+0000 7f37bac00640 1 Processor -- start 2024-03-06T07:35:24.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.421+0000 7f37bac00640 1 -- start start 2024-03-06T07:35:24.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37bac00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37b4071bc0 0x7f37b4150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37bac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b4152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37bac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4151800 0x7f37b4143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37bac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b4152160 con 0x7f37b4071fa0 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37bac00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b406e7e0 con 0x7f37b4071bc0 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37bac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b4143960 con 0x7f37b4151800 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37b3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b4152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.422+0000 7f37b3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b4152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41834/0 (socket says 172.21.15.97:41834) 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 -- 172.21.15.97:0/1770662143 learned_addr learned my addr 172.21.15.97:0/1770662143 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3e00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37b4071bc0 0x7f37b4150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b8c00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4151800 0x7f37b4143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 -- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4151800 msgr2=0x7f37b4143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4151800 0x7f37b4143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 -- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37b4071bc0 msgr2=0x7f37b4150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37b4071bc0 0x7f37b4150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 -- 172.21.15.97:0/1770662143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b40adbd0 con 0x7f37b4071fa0 2024-03-06T07:35:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.423+0000 7f37b3400640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b4152e80 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f37a002b950 tx=0x7f37a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.424+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0040070 con 0x7f37b4071fa0 2024-03-06T07:35:24.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.424+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a003a420 con 0x7f37b4071fa0 2024-03-06T07:35:24.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.424+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0035040 con 0x7f37b4071fa0 2024-03-06T07:35:24.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.424+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b40adeb0 con 0x7f37b4071fa0 2024-03-06T07:35:24.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.424+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b40ae470 con 0x7f37b4071fa0 2024-03-06T07:35:24.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.425+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b404f330 con 0x7f37b4071fa0 2024-03-06T07:35:24.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.426+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37a0038020 con 0x7f37b4071fa0 2024-03-06T07:35:24.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.427+0000 7f37abe00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f379406b4d0 0x7f379406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.427+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f37a00ac2a0 con 0x7f37b4071fa0 2024-03-06T07:35:24.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.427+0000 7f37b3e00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f379406b4d0 0x7f379406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.428+0000 7f37b3e00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f379406b4d0 0x7f379406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f37b4150ce0 tx=0x7f37a4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.431+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37a007b120 con 0x7f37b4071fa0 2024-03-06T07:35:24.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.632+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f37b406c5b0 con 0x7f37b4071fa0 2024-03-06T07:35:24.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.633+0000 7f37abe00640 1 -- 172.21.15.97:0/1770662143 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v118) v1 ==== 80+0+122 (secure 0 0 0) 0x7f37a00ac0c0 con 0x7f37b4071fa0 2024-03-06T07:35:24.635 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.135:6833/322807700 2024-03-07T07:34:20.730206+0000 2024-03-06T07:35:24.635 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.135:6832/322807700 2024-03-07T07:34:20.730206+0000 2024-03-06T07:35:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:listed 2 entries 2024-03-06T07:35:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f379406b4d0 msgr2=0x7f379406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f379406b4d0 0x7f379406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f37b4150ce0 tx=0x7f37a4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 msgr2=0x7f37b4152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b4152e80 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f37a002b950 tx=0x7f37a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 shutdown_connections 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f379406b4d0 0x7f379406d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f37b4151800 0x7f37b4143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4071fa0 0x7f37b4152e80 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 --2- 172.21.15.97:0/1770662143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37b4071bc0 0x7f37b4150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 >> 172.21.15.97:0/1770662143 conn(0x7f37b40b2620 msgr2=0x7f37b415dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 shutdown_connections 2024-03-06T07:35:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:24.635+0000 7f37bac00640 1 -- 172.21.15.97:0/1770662143 wait complete. 2024-03-06T07:35:24.653 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:35:24.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:24.740 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:35:24.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.740 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:35:24.777 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:35:24.777 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:24.834 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:35:24.834 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.881 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:24.882 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:35:24.882 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.937 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:24.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:25.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.133+0000 7f93dae00640 1 Processor -- start 2024-03-06T07:35:25.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.133+0000 7f93dae00640 1 -- start start 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93dae00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f93d40d0c00 0x7f93d40d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93dae00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 0x7f93d40b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93dae00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d40b5240 con 0x7f93d415e200 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93dae00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d40d2ec0 con 0x7f93d40d0c00 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93dae00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40cf4d0 con 0x7f93d40b2e50 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93d9a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f93d415e200 0x7f93d40d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56756/0 (socket says 172.21.15.97:56756) 2024-03-06T07:35:25.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93d9a00640 1 -- 172.21.15.97:0/1507995435 learned_addr learned my addr 172.21.15.97:0/1507995435 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:25.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025630495 0 0) 0x7f93d40b5240 con 0x7f93d415e200 2024-03-06T07:35:25.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93da400640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 0x7f93d40b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93bc003520 con 0x7f93d415e200 2024-03-06T07:35:25.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.136+0000 7f93d9000640 1 --2- 172.21.15.97:0/1507995435 >> v2:172.21.15.135:3300/0 conn(0x7f93d40d0c00 0x7f93d40d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1212300191 0 0) 0x7f93bc003520 con 0x7f93d415e200 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 msgr2=0x7f93d40b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 0x7f93d40b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 >> v2:172.21.15.135:3300/0 conn(0x7f93d40d0c00 msgr2=0x7f93d40d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 --2- 172.21.15.97:0/1507995435 >> v2:172.21.15.135:3300/0 conn(0x7f93d40d0c00 0x7f93d40d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d40b5500 con 0x7f93d415e200 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f93c8001670 con 0x7f93d415e200 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 >> v1:172.21.15.97:6789/0 conn(0x7f93d415e200 legacy=0x7f93d40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93da400640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 0x7f93d40b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93bc0039b0 0x7f93bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93bc004800 0x7f93bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 0x7f93bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40d2ec0 con 0x7f93bc0039b0 2024-03-06T07:35:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40cf4d0 con 0x7f93bc004800 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc00bb50 con 0x7f93d40b2e50 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.137+0000 7f93d9a00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93bc0039b0 0x7f93bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.138+0000 7f93d9a00640 1 -- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 msgr2=0x7f93bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.138+0000 7f93d9a00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93d40b2e50 0x7f93bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.138+0000 7f93d9a00640 1 -- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93bc004800 msgr2=0x7f93bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.138+0000 7f93d9a00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93bc004800 0x7f93bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.138+0000 7f93d9a00640 1 -- 172.21.15.97:0/1507995435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc003520 con 0x7f93bc0039b0 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.139+0000 7f93d9a00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93bc0039b0 0x7f93bc003dd0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f93c8001eb0 tx=0x7f93bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19aa0d41eff0f70e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.139+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8007c80 con 0x7f93bc0039b0 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.139+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8030070 con 0x7f93bc0039b0 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.139+0000 7f93d3e00640 1 -- 172.21.15.97:0/1507995435 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8005cf0 con 0x7f93bc0039b0 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93dae00640 1 -- 172.21.15.97:0/1507995435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93bc0039b0 msgr2=0x7f93bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93dae00640 1 --2- 172.21.15.97:0/1507995435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93bc0039b0 0x7f93bc003dd0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f93c8001eb0 tx=0x7f93bc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93da400640 1 -- 172.21.15.97:0/1507995435 reap_dead start 2024-03-06T07:35:25.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93dae00640 1 -- 172.21.15.97:0/1507995435 shutdown_connections 2024-03-06T07:35:25.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93dae00640 1 -- 172.21.15.97:0/1507995435 >> 172.21.15.97:0/1507995435 conn(0x7f93d40d3d30 msgr2=0x7f93d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93dae00640 1 -- 172.21.15.97:0/1507995435 shutdown_connections 2024-03-06T07:35:25.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.140+0000 7f93dae00640 1 -- 172.21.15.97:0/1507995435 wait complete. 2024-03-06T07:35:25.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.142+0000 7f93dae00640 1 Processor -- start 2024-03-06T07:35:25.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.142+0000 7f93dae00640 1 -- start start 2024-03-06T07:35:25.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93dae00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93bc0039b0 0x7f93d41f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93dae00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93d40b4590 0x7f93d40ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93dae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 0x7f93d40b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93dae00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40cfd60 con 0x7f93d41f3f40 2024-03-06T07:35:25.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93dae00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40cfa80 con 0x7f93d40b4590 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93dae00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d415e5e0 con 0x7f93bc0039b0 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 0x7f93d40b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 0x7f93d40b8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41864/0 (socket says 172.21.15.97:41864) 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 -- 172.21.15.97:0/569079568 learned_addr learned my addr 172.21.15.97:0/569079568 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93d9000640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93d40b4590 0x7f93d40ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 -- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93bc0039b0 msgr2=0x7f93d41f1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93bc0039b0 0x7f93d41f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 -- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93d40b4590 msgr2=0x7f93d40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93d40b4590 0x7f93d40ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.143+0000 7f93da400640 1 -- 172.21.15.97:0/569079568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d41f48b0 con 0x7f93d41f3f40 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.144+0000 7f93da400640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 0x7f93d40b8b20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f93d408e520 tx=0x7f93cc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.144+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc01a070 con 0x7f93d41f3f40 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.144+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc00dd20 con 0x7f93d41f3f40 2024-03-06T07:35:25.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.144+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc015600 con 0x7f93d41f3f40 2024-03-06T07:35:25.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.144+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d41f4b90 con 0x7f93d41f3f40 2024-03-06T07:35:25.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.144+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d40ca710 con 0x7f93d41f3f40 2024-03-06T07:35:25.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.145+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d40c97c0 con 0x7f93d41f3f40 2024-03-06T07:35:25.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.146+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93cc01d070 con 0x7f93d41f3f40 2024-03-06T07:35:25.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.148+0000 7f93d1600640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f93b406b500 0x7f93b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.148+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f93cc089ee0 con 0x7f93d41f3f40 2024-03-06T07:35:25.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.148+0000 7f93d9a00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f93b406b500 0x7f93b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.149+0000 7f93d9a00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f93b406b500 0x7f93b406d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034000 tx=0x7f93c8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.156+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93cc010070 con 0x7f93d41f3f40 2024-03-06T07:35:25.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.392+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93d404f330 con 0x7f93d41f3f40 2024-03-06T07:35:25.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.393+0000 7f93d1600640 1 -- 172.21.15.97:0/569079568 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f93cc058680 con 0x7f93d41f3f40 2024-03-06T07:35:25.396 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:25.396 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:25.396 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:25.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f93b406b500 msgr2=0x7f93b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f93b406b500 0x7f93b406d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034000 tx=0x7f93c8039040 comp rx=0 tx=0).stop 2024-03-06T07:35:25.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 msgr2=0x7f93d40b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 0x7f93d40b8b20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f93d408e520 tx=0x7f93cc00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 shutdown_connections 2024-03-06T07:35:25.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f93b406b500 0x7f93b406d9c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93d41f3f40 0x7f93d40b8b20 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93d40b4590 0x7f93d40ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 --2- 172.21.15.97:0/569079568 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f93bc0039b0 0x7f93d41f1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 >> 172.21.15.97:0/569079568 conn(0x7f93d40d3d30 msgr2=0x7f93d40d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 shutdown_connections 2024-03-06T07:35:25.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.396+0000 7f93dae00640 1 -- 172.21.15.97:0/569079568 wait complete. 2024-03-06T07:35:25.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:25.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.595+0000 7f7d34c00640 1 Processor -- start 2024-03-06T07:35:25.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.596+0000 7f7d34c00640 1 -- start start 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d300d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d34c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7d300b2e50 0x7f7d300d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d34c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d300b5240 con 0x7f7d3015e5e0 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d34c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d300d2ec0 con 0x7f7d300b2e50 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d34c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300cf4d0 con 0x7f7d3015e200 2024-03-06T07:35:25.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d2fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7d3015e5e0 0x7f7d300d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56762/0 (socket says 172.21.15.97:56762) 2024-03-06T07:35:25.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.597+0000 7f7d2fe00640 1 -- 172.21.15.97:0/1671387350 learned_addr learned my addr 172.21.15.97:0/1671387350 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:25.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2f400640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d300d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d26a00640 1 --2- 172.21.15.97:0/1671387350 >> v2:172.21.15.135:3300/0 conn(0x7f7d300b2e50 0x7f7d300d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385594803 0 0) 0x7f7d300b5240 con 0x7f7d3015e5e0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d14003520 con 0x7f7d3015e5e0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1601357609 0 0) 0x7f7d14003520 con 0x7f7d3015e5e0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 msgr2=0x7f7d300d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d300d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 >> v2:172.21.15.135:3300/0 conn(0x7f7d300b2e50 msgr2=0x7f7d300d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 --2- 172.21.15.97:0/1671387350 >> v2:172.21.15.135:3300/0 conn(0x7f7d300b2e50 0x7f7d300d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300b5500 con 0x7f7d3015e5e0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7d280019f0 con 0x7f7d3015e5e0 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d26a00640 1 --2- 172.21.15.97:0/1671387350 >> v2:172.21.15.135:3300/0 conn(0x7f7d300b2e50 0x7f7d300d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 >> v1:172.21.15.97:6789/0 conn(0x7f7d3015e5e0 legacy=0x7f7d300d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d14003fe0 0x7f7d14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14004d40 0x7f7d140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300d2ec0 con 0x7f7d14003fe0 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300cf4d0 con 0x7f7d14004d40 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.598+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1400bac0 con 0x7f7d3015e200 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d26a00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d14003fe0 0x7f7d14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d2f400640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d26a00640 1 -- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 msgr2=0x7f7d140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d2fe00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14004d40 0x7f7d140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d26a00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d26a00640 1 -- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14004d40 msgr2=0x7f7d140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d26a00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14004d40 0x7f7d140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d26a00640 1 -- 172.21.15.97:0/1671387350 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14003520 con 0x7f7d14003fe0 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d2f400640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d3015e200 0x7f7d140039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.599+0000 7f7d2fe00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14004d40 0x7f7d140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.600+0000 7f7d26a00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d14003fe0 0x7f7d14004800 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7d180098a0 tx=0x7f7d14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73c9d03fbe23ee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.600+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800f040 con 0x7f7d14003fe0 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.600+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d18004230 con 0x7f7d14003fe0 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.600+0000 7f7d2ea00640 1 -- 172.21.15.97:0/1671387350 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18003610 con 0x7f7d14003fe0 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d34c00640 1 -- 172.21.15.97:0/1671387350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d14003fe0 msgr2=0x7f7d14004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d34c00640 1 --2- 172.21.15.97:0/1671387350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d14003fe0 0x7f7d14004800 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7d180098a0 tx=0x7f7d14005830 comp rx=0 tx=0).stop 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d2fe00640 1 -- 172.21.15.97:0/1671387350 reap_dead start 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d34c00640 1 -- 172.21.15.97:0/1671387350 shutdown_connections 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d34c00640 1 -- 172.21.15.97:0/1671387350 >> 172.21.15.97:0/1671387350 conn(0x7f7d300d3d30 msgr2=0x7f7d300d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d34c00640 1 -- 172.21.15.97:0/1671387350 shutdown_connections 2024-03-06T07:35:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.601+0000 7f7d34c00640 1 -- 172.21.15.97:0/1671387350 wait complete. 2024-03-06T07:35:25.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.603+0000 7f7d34c00640 1 Processor -- start 2024-03-06T07:35:25.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.603+0000 7f7d34c00640 1 -- start start 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14003fe0 0x7f7d300cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d30155de0 0x7f7d30156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d34c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 0x7f7d300cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d34c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300cfd60 con 0x7f7d300cb0c0 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d34c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300cfa80 con 0x7f7d14003fe0 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d34c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300b4650 con 0x7f7d30155de0 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 0x7f7d300cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d2fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 0x7f7d300cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41888/0 (socket says 172.21.15.97:41888) 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d2fe00640 1 -- 172.21.15.97:0/3432198840 learned_addr learned my addr 172.21.15.97:0/3432198840 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d26a00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d30155de0 0x7f7d30156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.605+0000 7f7d2f400640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14003fe0 0x7f7d300cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2fe00640 1 -- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d30155de0 msgr2=0x7f7d30156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2fe00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d30155de0 0x7f7d30156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2fe00640 1 -- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14003fe0 msgr2=0x7f7d300cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2fe00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14003fe0 0x7f7d300cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2fe00640 1 -- 172.21.15.97:0/3432198840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300c7860 con 0x7f7d300cb0c0 2024-03-06T07:35:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2f400640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14003fe0 0x7f7d300cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d2fe00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 0x7f7d300cb540 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f7d300c7570 tx=0x7f7d2800b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800b8d0 con 0x7f7d300cb0c0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2800ba90 con 0x7f7d300cb0c0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800be80 con 0x7f7d300cb0c0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d300cbe80 con 0x7f7d300cb0c0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.606+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d300cc440 con 0x7f7d300cb0c0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.607+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d300c7a30 con 0x7f7d300cb0c0 2024-03-06T07:35:25.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.608+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d2801b020 con 0x7f7d300cb0c0 2024-03-06T07:35:25.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.610+0000 7f7d27e00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0c06b500 0x7f7d0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.610+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7d2808a360 con 0x7f7d300cb0c0 2024-03-06T07:35:25.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.610+0000 7f7d2f400640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0c06b500 0x7f7d0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.611+0000 7f7d2f400640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0c06b500 0x7f7d0c06d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d20006fd0 tx=0x7f7d20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.613+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d28058a60 con 0x7f7d300cb0c0 2024-03-06T07:35:25.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.856+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d3004f330 con 0x7f7d300cb0c0 2024-03-06T07:35:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.858+0000 7f7d27e00640 1 -- 172.21.15.97:0/3432198840 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f7d2805cb00 con 0x7f7d300cb0c0 2024-03-06T07:35:25.861 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:25.861 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":118,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:29.791334+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.135:6833/322807700":"2024-03-07T07:34:20.730206+0000","172.21.15.135:6832/322807700":"2024-03-07T07:34:20.730206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0c06b500 msgr2=0x7f7d0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0c06b500 0x7f7d0c06d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d20006fd0 tx=0x7f7d20008040 comp rx=0 tx=0).stop 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 msgr2=0x7f7d300cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 0x7f7d300cb540 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f7d300c7570 tx=0x7f7d2800b0d0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 shutdown_connections 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d0c06b500 0x7f7d0c06d9c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d300cb0c0 0x7f7d300cb540 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d30155de0 0x7f7d30156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 --2- 172.21.15.97:0/3432198840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d14003fe0 0x7f7d300cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 >> 172.21.15.97:0/3432198840 conn(0x7f7d300d3d30 msgr2=0x7f7d300d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 shutdown_connections 2024-03-06T07:35:25.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:25.861+0000 7f7d34c00640 1 -- 172.21.15.97:0/3432198840 wait complete. 2024-03-06T07:35:25.889 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:03.618857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:04.895815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:25.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:26.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.097+0000 7fa57fe00640 1 Processor -- start 2024-03-06T07:35:26.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.098+0000 7fa57fe00640 1 -- start start 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.099+0000 7fa57fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 0x7fa5780b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.099+0000 7fa57fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa578162310 0x7fa578164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.099+0000 7fa57fe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5780b3e50 con 0x7fa5780b3530 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.099+0000 7fa57fe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780ce690 con 0x7fa578162310 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.099+0000 7fa57fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780ce970 con 0x7fa5780adf70 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa57d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa5780b3530 0x7fa57815b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56774/0 (socket says 172.21.15.97:56774) 2024-03-06T07:35:26.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa57d400640 1 -- 172.21.15.97:0/128069138 learned_addr learned my addr 172.21.15.97:0/128069138 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa57ca00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 0x7fa5780b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa57de00640 1 --2- 172.21.15.97:0/128069138 >> v2:172.21.15.135:3300/0 conn(0x7fa578162310 0x7fa578164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839125782 0 0) 0x7fa5780b3e50 con 0x7fa5780b3530 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560003520 con 0x7fa5780b3530 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990458280 0 0) 0x7fa560003520 con 0x7fa5780b3530 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 msgr2=0x7fa5780b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.100+0000 7fa577e00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 0x7fa5780b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 >> v2:172.21.15.135:3300/0 conn(0x7fa578162310 msgr2=0x7fa578164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 --2- 172.21.15.97:0/128069138 >> v2:172.21.15.135:3300/0 conn(0x7fa578162310 0x7fa578164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780cec50 con 0x7fa5780b3530 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa56c001670 con 0x7fa5780b3530 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 >> v1:172.21.15.97:6789/0 conn(0x7fa5780b3530 legacy=0x7fa57815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 0x7fa5600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa560003fb0 0x7fa560004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa5600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780ce690 con 0x7fa560004d40 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780ce970 con 0x7fa560003fb0 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56000b9f0 con 0x7fa5780adf70 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57de00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa5600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57d400640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 0x7fa5600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57ca00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa560003fb0 0x7fa560004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57de00640 1 -- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 msgr2=0x7fa5600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57de00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5780adf70 0x7fa5600039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57de00640 1 -- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa560003fb0 msgr2=0x7fa560004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57de00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa560003fb0 0x7fa560004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.101+0000 7fa57de00640 1 -- 172.21.15.97:0/128069138 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560003520 con 0x7fa560004d40 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa57de00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa5600051c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa570009fc0 tx=0x7fa560005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea4902a1157ccdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570039070 con 0x7fa560004d40 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57002cc40 con 0x7fa560004d40 2024-03-06T07:35:26.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa577e00640 1 -- 172.21.15.97:0/128069138 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570009c10 con 0x7fa560004d40 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa57fe00640 1 -- 172.21.15.97:0/128069138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 msgr2=0x7fa5600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa57fe00640 1 --2- 172.21.15.97:0/128069138 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa5600051c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa570009fc0 tx=0x7fa560005830 comp rx=0 tx=0).stop 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa57de00640 1 -- 172.21.15.97:0/128069138 reap_dead start 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa57fe00640 1 -- 172.21.15.97:0/128069138 shutdown_connections 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.103+0000 7fa57fe00640 1 -- 172.21.15.97:0/128069138 >> 172.21.15.97:0/128069138 conn(0x7fa5780b1940 msgr2=0x7fa5780b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.104+0000 7fa57fe00640 1 -- 172.21.15.97:0/128069138 shutdown_connections 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.104+0000 7fa57fe00640 1 -- 172.21.15.97:0/128069138 wait complete. 2024-03-06T07:35:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.105+0000 7fa57fe00640 1 Processor -- start 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.105+0000 7fa57fe00640 1 -- start start 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.106+0000 7fa57fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa57815ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.106+0000 7fa57fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa57815b270 0x7fa57814b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.106+0000 7fa57fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa57814bd90 0x7fa57809af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.106+0000 7fa57fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57815aa30 con 0x7fa560004d40 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.106+0000 7fa57fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780cf210 con 0x7fa57814bd90 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.106+0000 7fa57fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57815abb0 con 0x7fa57815b270 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa57815ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa57815ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41892/0 (socket says 172.21.15.97:41892) 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 -- 172.21.15.97:0/3802161739 learned_addr learned my addr 172.21.15.97:0/3802161739 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 -- 172.21.15.97:0/3802161739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa57815b270 msgr2=0x7fa57814b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa57815b270 0x7fa57814b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 -- 172.21.15.97:0/3802161739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa57814bd90 msgr2=0x7fa57809af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa57814bd90 0x7fa57809af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 -- 172.21.15.97:0/3802161739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57814e3d0 con 0x7fa560004d40 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa57d400640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa57815ad30 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00e8a0 tx=0x7fa56c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.107+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c00c7f0 con 0x7fa560004d40 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.108+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c018070 con 0x7fa560004d40 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.108+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57809bc30 con 0x7fa560004d40 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.108+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c00cbf0 con 0x7fa560004d40 2024-03-06T07:35:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.108+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57809c1f0 con 0x7fa560004d40 2024-03-06T07:35:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.109+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57814c1b0 con 0x7fa560004d40 2024-03-06T07:35:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.110+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa56c013420 con 0x7fa560004d40 2024-03-06T07:35:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.111+0000 7fa575600640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa55806b500 0x7fa55806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.111+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa56c089d60 con 0x7fa560004d40 2024-03-06T07:35:26.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.112+0000 7fa57ca00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa55806b500 0x7fa55806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.112+0000 7fa57ca00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa55806b500 0x7fa55806d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa57814fdc0 tx=0x7fa564008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.116+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa56c058be0 con 0x7fa560004d40 2024-03-06T07:35:26.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.316+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa57804f330 con 0x7fa560004d40 2024-03-06T07:35:26.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.317+0000 7fa575600640 1 -- 172.21.15.97:0/3802161739 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa56c0584e0 con 0x7fa560004d40 2024-03-06T07:35:26.319 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:35:26.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.319+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa55806b500 msgr2=0x7fa55806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.319+0000 7fa57fe00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa55806b500 0x7fa55806d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa57814fdc0 tx=0x7fa564008040 comp rx=0 tx=0).stop 2024-03-06T07:35:26.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.319+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 msgr2=0x7fa57815ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.319+0000 7fa57fe00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa57815ad30 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00e8a0 tx=0x7fa56c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 shutdown_connections 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa55806b500 0x7fa55806d9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa57814bd90 0x7fa57809af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa57815b270 0x7fa57814b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 --2- 172.21.15.97:0/3802161739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa560004d40 0x7fa57815ad30 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 >> 172.21.15.97:0/3802161739 conn(0x7fa5780b1940 msgr2=0x7fa5780cd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 shutdown_connections 2024-03-06T07:35:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.320+0000 7fa57fe00640 1 -- 172.21.15.97:0/3802161739 wait complete. 2024-03-06T07:35:26.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:26.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.557+0000 7f54b1000640 1 Processor -- start 2024-03-06T07:35:26.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.557+0000 7f54b1000640 1 -- start start 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54b1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fc10 0x7f54ac070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54b1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f54ac15fff0 0x7f54ac160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54b1000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0b46c0 con 0x7f54ac0ce4a0 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54b1000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54ac0b49a0 con 0x7f54ac15fff0 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54b1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac072c90 con 0x7f54ac15fc10 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54a9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f54ac15fff0 0x7f54ac160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54aa800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f54ac0ce4a0 0x7f54ac0ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56782/0 (socket says 172.21.15.97:56782) 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54aa800640 1 -- 172.21.15.97:0/3416839322 learned_addr learned my addr 172.21.15.97:0/3416839322 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.559+0000 7f54a9e00640 1 --2- 172.21.15.97:0/3416839322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fc10 0x7f54ac070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.560+0000 7f54a9400640 1 -- 172.21.15.97:0/3416839322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fc10 msgr2=0x7f54ac070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.560+0000 7f54a9400640 1 --2- 172.21.15.97:0/3416839322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fc10 0x7f54ac070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.560+0000 7f54a9400640 1 -- 172.21.15.97:0/3416839322 >> v1:172.21.15.97:6789/0 conn(0x7f54ac0ce4a0 legacy=0x7f54ac0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.560+0000 7f54a9400640 1 -- 172.21.15.97:0/3416839322 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac160990 con 0x7f54ac15fff0 2024-03-06T07:35:26.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.560+0000 7f54a8a00640 1 -- 172.21.15.97:0/3416839322 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715640988 0 0) 0x7f54ac0b46c0 con 0x7f54ac0ce4a0 2024-03-06T07:35:26.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.561+0000 7f54a9400640 1 --2- 172.21.15.97:0/3416839322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fff0 0x7f54ac160450 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f54980097a0 tx=0x7f549802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1ddf6509f70687 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.561+0000 7f54a8a00640 1 -- 172.21.15.97:0/3416839322 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498035020 con 0x7f54ac15fff0 2024-03-06T07:35:26.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.561+0000 7f54a8a00640 1 -- 172.21.15.97:0/3416839322 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498031830 con 0x7f54ac15fff0 2024-03-06T07:35:26.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.561+0000 7f54a8a00640 1 -- 172.21.15.97:0/3416839322 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498031c00 con 0x7f54ac15fff0 2024-03-06T07:35:26.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 -- 172.21.15.97:0/3416839322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fff0 msgr2=0x7f54ac160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 --2- 172.21.15.97:0/3416839322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fff0 0x7f54ac160450 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f54980097a0 tx=0x7f549802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 -- 172.21.15.97:0/3416839322 shutdown_connections 2024-03-06T07:35:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 --2- 172.21.15.97:0/3416839322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fff0 0x7f54ac160450 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 --2- 172.21.15.97:0/3416839322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fc10 0x7f54ac070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 -- 172.21.15.97:0/3416839322 >> 172.21.15.97:0/3416839322 conn(0x7f54ac0b1940 msgr2=0x7f54ac0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 -- 172.21.15.97:0/3416839322 shutdown_connections 2024-03-06T07:35:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.562+0000 7f54b1000640 1 -- 172.21.15.97:0/3416839322 wait complete. 2024-03-06T07:35:26.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.564+0000 7f54b1000640 1 Processor -- start 2024-03-06T07:35:26.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.564+0000 7f54b1000640 1 -- start start 2024-03-06T07:35:26.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54b1000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54ac0ce4a0 0x7f54ac1437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54b1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 0x7f54ac0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54b1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fff0 0x7f54ac0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54b1000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac072a30 con 0x7f54ac15fc10 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54b1000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0728b0 con 0x7f54ac0ce4a0 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54b1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac15c2e0 con 0x7f54ac15fff0 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54a9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 0x7f54ac0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54a9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 0x7f54ac0afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41906/0 (socket says 172.21.15.97:41906) 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54a9400640 1 -- 172.21.15.97:0/3267724762 learned_addr learned my addr 172.21.15.97:0/3267724762 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.565+0000 7f54a9e00640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54ac0ce4a0 0x7f54ac1437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54aa800640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fff0 0x7f54ac0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9400640 1 -- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fff0 msgr2=0x7f54ac0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9400640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fff0 0x7f54ac0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9400640 1 -- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54ac0ce4a0 msgr2=0x7f54ac1437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9400640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54ac0ce4a0 0x7f54ac1437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9400640 1 -- 172.21.15.97:0/3267724762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac15b140 con 0x7f54ac15fc10 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9e00640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54ac0ce4a0 0x7f54ac1437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a9400640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 0x7f54ac0afda0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f54980097a0 tx=0x7f54980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498035040 con 0x7f54ac15fc10 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498004230 con 0x7f54ac15fc10 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.566+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549803a700 con 0x7f54ac15fc10 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.567+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac15b420 con 0x7f54ac15fc10 2024-03-06T07:35:26.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.567+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac15b9e0 con 0x7f54ac15fc10 2024-03-06T07:35:26.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.568+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac04f330 con 0x7f54ac15fc10 2024-03-06T07:35:26.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.569+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5498038020 con 0x7f54ac15fc10 2024-03-06T07:35:26.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.570+0000 7f54a2000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f548c06b4d0 0x7f548c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.570+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f54980abfe0 con 0x7f54ac15fc10 2024-03-06T07:35:26.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.570+0000 7f54a9e00640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f548c06b4d0 0x7f548c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.571+0000 7f54a9e00640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f548c06b4d0 0x7f548c06d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f549c006fd0 tx=0x7f549c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.574+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f549807adc0 con 0x7f54ac15fc10 2024-03-06T07:35:26.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.772+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f54ac06ef30 con 0x7f54ac15fc10 2024-03-06T07:35:26.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.773+0000 7f54a2000640 1 -- 172.21.15.97:0/3267724762 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f549807a6c0 con 0x7f54ac15fc10 2024-03-06T07:35:26.776 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:35:26.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f548c06b4d0 msgr2=0x7f548c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f548c06b4d0 0x7f548c06d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f549c006fd0 tx=0x7f549c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:26.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 msgr2=0x7f54ac0afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 0x7f54ac0afda0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f54980097a0 tx=0x7f54980046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 shutdown_connections 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f548c06b4d0 0x7f548c06d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f54ac15fff0 0x7f54ac0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54ac15fc10 0x7f54ac0afda0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 --2- 172.21.15.97:0/3267724762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f54ac0ce4a0 0x7f54ac1437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 >> 172.21.15.97:0/3267724762 conn(0x7f54ac0b1940 msgr2=0x7f54ac0aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 shutdown_connections 2024-03-06T07:35:26.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:26.776+0000 7f54b1000640 1 -- 172.21.15.97:0/3267724762 wait complete. 2024-03-06T07:35:26.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:27.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.005+0000 7fdf03000640 1 Processor -- start 2024-03-06T07:35:27.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.005+0000 7fdf03000640 1 -- start start 2024-03-06T07:35:27.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdf03000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdefc15e450 0x7fdefc0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdf03000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc0b5490 0x7fdefc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdf03000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc0b3730 con 0x7fdefc0d07f0 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdf03000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdefc0d4930 con 0x7fdefc15e450 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdf03000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc067b20 con 0x7fdefc0b5490 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdef3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdefc15e450 0x7fdefc0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdef3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdefc15e450 0x7fdefc0cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41908/0 (socket says 172.21.15.97:41908) 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.006+0000 7fdef3400640 1 -- 172.21.15.97:0/3218790976 learned_addr learned my addr 172.21.15.97:0/3218790976 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.007+0000 7fdef3400640 1 -- 172.21.15.97:0/3218790976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc0b5490 msgr2=0x7fdefc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.007+0000 7fdef3400640 1 --2- 172.21.15.97:0/3218790976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc0b5490 0x7fdefc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.007+0000 7fdef3400640 1 -- 172.21.15.97:0/3218790976 >> v1:172.21.15.135:6789/0 conn(0x7fdefc0d07f0 legacy=0x7fdefc0d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.007+0000 7fdef3400640 1 -- 172.21.15.97:0/3218790976 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc0d4b80 con 0x7fdefc15e450 2024-03-06T07:35:27.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.008+0000 7fdef3400640 1 --2- 172.21.15.97:0/3218790976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc15e450 0x7fdefc0cea20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009870 tx=0x7fdee402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23649e1d37ff32fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.008+0000 7fdefb400640 1 -- 172.21.15.97:0/3218790976 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee40317c0 con 0x7fdefc15e450 2024-03-06T07:35:27.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.008+0000 7fdefb400640 1 -- 172.21.15.97:0/3218790976 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee4037070 con 0x7fdefc15e450 2024-03-06T07:35:27.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.008+0000 7fdefb400640 1 -- 172.21.15.97:0/3218790976 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4031b80 con 0x7fdefc15e450 2024-03-06T07:35:27.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 -- 172.21.15.97:0/3218790976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc15e450 msgr2=0x7fdefc0cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 --2- 172.21.15.97:0/3218790976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc15e450 0x7fdefc0cea20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009870 tx=0x7fdee402b860 comp rx=0 tx=0).stop 2024-03-06T07:35:27.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 -- 172.21.15.97:0/3218790976 shutdown_connections 2024-03-06T07:35:27.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 --2- 172.21.15.97:0/3218790976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc0b5490 0x7fdefc0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 --2- 172.21.15.97:0/3218790976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc15e450 0x7fdefc0cea20 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 -- 172.21.15.97:0/3218790976 >> 172.21.15.97:0/3218790976 conn(0x7fdefc0d3e10 msgr2=0x7fdefc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 -- 172.21.15.97:0/3218790976 shutdown_connections 2024-03-06T07:35:27.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.009+0000 7fdf03000640 1 -- 172.21.15.97:0/3218790976 wait complete. 2024-03-06T07:35:27.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.012+0000 7fdf03000640 1 Processor -- start 2024-03-06T07:35:27.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.012+0000 7fdf03000640 1 -- start start 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf03000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 0x7fdefc0cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf03000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdefc0d07f0 0x7fdefc0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf03000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc15e450 0x7fdefc154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf03000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0b3f60 con 0x7fdefc0b5490 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf03000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0b3c80 con 0x7fdefc0d07f0 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf03000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0d1bf0 con 0x7fdefc15e450 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 0x7fdefc0cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 0x7fdefc0cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41912/0 (socket says 172.21.15.97:41912) 2024-03-06T07:35:27.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdefbe00640 1 -- 172.21.15.97:0/2247302972 learned_addr learned my addr 172.21.15.97:0/2247302972 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdef3400640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdefc0d07f0 0x7fdefc0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdf01000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc15e450 0x7fdefc154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdefbe00640 1 -- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc15e450 msgr2=0x7fdefc154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.014+0000 7fdefbe00640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc15e450 0x7fdefc154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdefbe00640 1 -- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdefc0d07f0 msgr2=0x7fdefc0caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdefbe00640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdefc0d07f0 0x7fdefc0caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdefbe00640 1 -- 172.21.15.97:0/2247302972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc1f3fd0 con 0x7fdefc0b5490 2024-03-06T07:35:27.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdef3400640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdefc0d07f0 0x7fdefc0caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdefbe00640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 0x7fdefc0cdad0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c7a0 tx=0x7fdeec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec017070 con 0x7fdefc0b5490 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec003da0 con 0x7fdefc0b5490 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0052b0 con 0x7fdefc0b5490 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc0c7d90 con 0x7fdefc0b5490 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.015+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc0c8350 con 0x7fdefc0b5490 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.017+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdefc0ce2d0 con 0x7fdefc0b5490 2024-03-06T07:35:27.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.018+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdeec005450 con 0x7fdefc0b5490 2024-03-06T07:35:27.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.018+0000 7fdef8c00640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.018+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fdeec089820 con 0x7fdefc0b5490 2024-03-06T07:35:27.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.019+0000 7fdef3400640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.019+0000 7fdef3400640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fded806b4d0 0x7fded806d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdee4036000 tx=0x7fdee4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.023+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdefc0ce2d0 con 0x7fdefc0b5490 2024-03-06T07:35:27.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.222+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdefc0d21d0 con 0x7fdefc0b5490 2024-03-06T07:35:27.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.223+0000 7fdef8c00640 1 -- 172.21.15.97:0/2247302972 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdeec058610 con 0x7fdefc0b5490 2024-03-06T07:35:27.225 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fded806b4d0 msgr2=0x7fded806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fded806b4d0 0x7fded806d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdee4036000 tx=0x7fdee4008030 comp rx=0 tx=0).stop 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 msgr2=0x7fdefc0cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 0x7fdefc0cdad0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c7a0 tx=0x7fdeec00cc60 comp rx=0 tx=0).stop 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 shutdown_connections 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdefc15e450 0x7fdefc154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdefc0d07f0 0x7fdefc0caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 --2- 172.21.15.97:0/2247302972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdefc0b5490 0x7fdefc0cdad0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 >> 172.21.15.97:0/2247302972 conn(0x7fdefc0d3e10 msgr2=0x7fdefc06c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 shutdown_connections 2024-03-06T07:35:27.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.226+0000 7fdf03000640 1 -- 172.21.15.97:0/2247302972 wait complete. 2024-03-06T07:35:27.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.447+0000 7f0ae6200640 1 Processor -- start 2024-03-06T07:35:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.448+0000 7f0ae6200640 1 -- start start 2024-03-06T07:35:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.449+0000 7f0ae6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 0x7f0ae00cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.449+0000 7f0ae6200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae00cf140 con 0x7f0ae00d1c20 2024-03-06T07:35:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.449+0000 7f0ae6200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae00cf420 con 0x7f0ae015f110 2024-03-06T07:35:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.449+0000 7f0ae6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00cfd50 con 0x7f0ae00d07b0 2024-03-06T07:35:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adfe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0ae00d1c20 0x7f0ae00d2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56792/0 (socket says 172.21.15.97:56792) 2024-03-06T07:35:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adfe00640 1 -- 172.21.15.97:0/1256390652 learned_addr learned my addr 172.21.15.97:0/1256390652 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:27.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adf400640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 0x7f0ae00cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271931191 0 0) 0x7f0ae00cf140 con 0x7f0ae00d1c20 2024-03-06T07:35:27.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac4003520 con 0x7f0ae00d1c20 2024-03-06T07:35:27.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544704766 0 0) 0x7f0ac4003520 con 0x7f0ae00d1c20 2024-03-06T07:35:27.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 msgr2=0x7f0ae00cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 0x7f0ae00cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 >> v1:172.21.15.135:6789/0 conn(0x7f0ae015f110 legacy=0x7f0ae00d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae00cf700 con 0x7f0ae00d1c20 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0ad80019f0 con 0x7f0ae00d1c20 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.450+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 >> v1:172.21.15.97:6789/0 conn(0x7f0ae00d1c20 legacy=0x7f0ae00d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adea00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 0x7f0ac40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adea00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4004060 0x7f0ac4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adea00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ac40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00cfd50 con 0x7f0ac4004d40 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac400bc00 con 0x7f0ac4004060 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac400be00 con 0x7f0ae00d07b0 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adfe00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ac40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0adf400640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 0x7f0ac40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.451+0000 7f0ad7e00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4004060 0x7f0ac4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.452+0000 7f0adfe00640 1 -- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 msgr2=0x7f0ac40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.452+0000 7f0adfe00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae00d07b0 0x7f0ac40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.452+0000 7f0adfe00640 1 -- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4004060 msgr2=0x7f0ac4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.452+0000 7f0adfe00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4004060 0x7f0ac4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.452+0000 7f0adfe00640 1 -- 172.21.15.97:0/1256390652 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac4003520 con 0x7f0ac4004d40 2024-03-06T07:35:27.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.452+0000 7f0ad7e00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4004060 0x7f0ac4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.453+0000 7f0adfe00640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ac40051c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800a7b0 tx=0x7f0ac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a2884002b67618 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.453+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad800a3d0 con 0x7f0ac4004d40 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.453+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad8034630 con 0x7f0ac4004d40 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.453+0000 7f0adea00640 1 -- 172.21.15.97:0/1256390652 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8034aa0 con 0x7f0ac4004d40 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0ae6200640 1 -- 172.21.15.97:0/1256390652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 msgr2=0x7f0ac40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0ae6200640 1 --2- 172.21.15.97:0/1256390652 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ac40051c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800a7b0 tx=0x7f0ac4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0adfe00640 1 -- 172.21.15.97:0/1256390652 reap_dead start 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0ae6200640 1 -- 172.21.15.97:0/1256390652 shutdown_connections 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0ae6200640 1 -- 172.21.15.97:0/1256390652 >> 172.21.15.97:0/1256390652 conn(0x7f0ae00d3ee0 msgr2=0x7f0ae00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0ae6200640 1 -- 172.21.15.97:0/1256390652 shutdown_connections 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.454+0000 7f0ae6200640 1 -- 172.21.15.97:0/1256390652 wait complete. 2024-03-06T07:35:27.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.455+0000 7f0ae6200640 1 Processor -- start 2024-03-06T07:35:27.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.456+0000 7f0ae6200640 1 -- start start 2024-03-06T07:35:27.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.456+0000 7f0ae6200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ae01f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.456+0000 7f0ae6200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ae00cd6b0 0x7f0ae00cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.456+0000 7f0ae6200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae0158590 0x7f0ae0158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.456+0000 7f0ae6200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae015e390 con 0x7f0ac4004d40 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0ae6200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae015f570 con 0x7f0ae00cd6b0 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0ae6200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae015e510 con 0x7f0ae0158590 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ae01f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ae01f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41940/0 (socket says 172.21.15.97:41940) 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 -- 172.21.15.97:0/3927346789 learned_addr learned my addr 172.21.15.97:0/3927346789 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 -- 172.21.15.97:0/3927346789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae0158590 msgr2=0x7f0ae0158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae0158590 0x7f0ae0158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 -- 172.21.15.97:0/3927346789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ae00cd6b0 msgr2=0x7f0ae00cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ae00cd6b0 0x7f0ae00cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 -- 172.21.15.97:0/3927346789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae01f72b0 con 0x7f0ac4004d40 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.457+0000 7f0adf400640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ae01f6d70 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000b7f0 tx=0x7f0ad000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.458+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0018070 con 0x7f0ac4004d40 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.458+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad00107b0 con 0x7f0ac4004d40 2024-03-06T07:35:27.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.458+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0010b80 con 0x7f0ac4004d40 2024-03-06T07:35:27.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.458+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae0159310 con 0x7f0ac4004d40 2024-03-06T07:35:27.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.458+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae01598d0 con 0x7f0ac4004d40 2024-03-06T07:35:27.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.459+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae004f330 con 0x7f0ac4004d40 2024-03-06T07:35:27.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.460+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ad001b070 con 0x7f0ac4004d40 2024-03-06T07:35:27.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.461+0000 7f0ad7400640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0abc06b500 0x7f0abc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.461+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0ad008a8b0 con 0x7f0ac4004d40 2024-03-06T07:35:27.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.461+0000 7f0ad7e00640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0abc06b500 0x7f0abc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.462+0000 7f0ad7e00640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0abc06b500 0x7f0abc06d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8007fd0 tx=0x7f0ac8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.465+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ad0059600 con 0x7f0ac4004d40 2024-03-06T07:35:27.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.720+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae008d940 con 0x7f0ac4004d40 2024-03-06T07:35:27.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.720+0000 7f0ad7400640 1 -- 172.21.15.97:0/3927346789 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f0ad0058f00 con 0x7f0ac4004d40 2024-03-06T07:35:27.723 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:27.723 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:27.723 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0abc06b500 msgr2=0x7f0abc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0abc06b500 0x7f0abc06d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8007fd0 tx=0x7f0ac8009040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 msgr2=0x7f0ae01f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ae01f6d70 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000b7f0 tx=0x7f0ad000bcb0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 shutdown_connections 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0abc06b500 0x7f0abc06d9c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0ae0158590 0x7f0ae0158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ae00cd6b0 0x7f0ae00cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 --2- 172.21.15.97:0/3927346789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ac4004d40 0x7f0ae01f6d70 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 >> 172.21.15.97:0/3927346789 conn(0x7f0ae00d3ee0 msgr2=0x7f0ae00b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 shutdown_connections 2024-03-06T07:35:27.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.723+0000 7f0ae6200640 1 -- 172.21.15.97:0/3927346789 wait complete. 2024-03-06T07:35:27.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:27.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.948+0000 7fef12800640 1 Processor -- start 2024-03-06T07:35:27.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.948+0000 7fef12800640 1 -- start start 2024-03-06T07:35:27.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef12800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fef0c15e160 0x7fef0c162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef12800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 0x7fef0c06b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef12800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c162b60 con 0x7fef0c0af820 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef12800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0c06b0f0 con 0x7fef0c15e160 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef12800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c0b5620 con 0x7fef0c0b3ac0 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef0aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fef0c0af820 0x7fef0c0b33b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56806/0 (socket says 172.21.15.97:56806) 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef0aa00640 1 -- 172.21.15.97:0/1876041108 learned_addr learned my addr 172.21.15.97:0/1876041108 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef0b400640 1 --2- 172.21.15.97:0/1876041108 >> v2:172.21.15.135:3300/0 conn(0x7fef0c15e160 0x7fef0c162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.950+0000 7fef0be00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 0x7fef0c06b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081500180 0 0) 0x7feef8001110 con 0x7fef0c0af820 2024-03-06T07:35:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef0003520 con 0x7fef0c0af820 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3555426230 0 0) 0x7feef0003520 con 0x7fef0c0af820 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 msgr2=0x7fef0c06b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 0x7fef0c06b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 >> v2:172.21.15.135:3300/0 conn(0x7fef0c15e160 msgr2=0x7fef0c162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 --2- 172.21.15.97:0/1876041108 >> v2:172.21.15.135:3300/0 conn(0x7fef0c15e160 0x7fef0c162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c06f1f0 con 0x7fef0c0af820 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7feef8001840 con 0x7fef0c0af820 2024-03-06T07:35:27.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 >> v1:172.21.15.97:6789/0 conn(0x7fef0c0af820 legacy=0x7fef0c0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 0x7feef00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feef0003f30 0x7feef0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feef0004d40 0x7feef00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c06b0f0 con 0x7feef0004d40 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c0b5620 con 0x7feef0003f30 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef000ba10 con 0x7fef0c0b3ac0 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.951+0000 7fef0be00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feef0004d40 0x7feef00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0aa00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feef0003f30 0x7feef0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0b400640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 0x7feef00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0be00640 1 -- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 msgr2=0x7feef00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0be00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fef0c0b3ac0 0x7feef00039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0be00640 1 -- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feef0003f30 msgr2=0x7feef0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0be00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feef0003f30 0x7feef0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0be00640 1 -- 172.21.15.97:0/1876041108 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0003520 con 0x7feef0004d40 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.952+0000 7fef0aa00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feef0003f30 0x7feef0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.953+0000 7fef0be00640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feef0004d40 0x7feef00051c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fef0400e170 tx=0x7feef0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9e837b391cafc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.953+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04039070 con 0x7feef0004d40 2024-03-06T07:35:27.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.953+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0402cbf0 con 0x7feef0004d40 2024-03-06T07:35:27.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.953+0000 7fef0a000640 1 -- 172.21.15.97:0/1876041108 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04009c10 con 0x7feef0004d40 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef12800640 1 -- 172.21.15.97:0/1876041108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feef0004d40 msgr2=0x7feef00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef12800640 1 --2- 172.21.15.97:0/1876041108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feef0004d40 0x7feef00051c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fef0400e170 tx=0x7feef0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef0be00640 1 -- 172.21.15.97:0/1876041108 reap_dead start 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef12800640 1 -- 172.21.15.97:0/1876041108 shutdown_connections 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef12800640 1 -- 172.21.15.97:0/1876041108 >> 172.21.15.97:0/1876041108 conn(0x7fef0c0d0340 msgr2=0x7fef0c0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef12800640 1 -- 172.21.15.97:0/1876041108 shutdown_connections 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.954+0000 7fef12800640 1 -- 172.21.15.97:0/1876041108 wait complete. 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.956+0000 7fef12800640 1 Processor -- start 2024-03-06T07:35:27.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.956+0000 7fef12800640 1 -- start start 2024-03-06T07:35:27.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef12800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feef0004d40 0x7fef0c1429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef12800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef0c153990 0x7fef0c153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef12800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 0x7fef0c1f7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef12800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c06ef10 con 0x7fef0c1f77d0 2024-03-06T07:35:27.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef12800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c06ec30 con 0x7fef0c153990 2024-03-06T07:35:27.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef12800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c06b580 con 0x7feef0004d40 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 0x7fef0c1f7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 0x7fef0c1f7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41960/0 (socket says 172.21.15.97:41960) 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 -- 172.21.15.97:0/3376337942 learned_addr learned my addr 172.21.15.97:0/3376337942 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 -- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feef0004d40 msgr2=0x7fef0c1429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0b400640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feef0004d40 0x7fef0c1429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0aa00640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef0c153990 0x7fef0c153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feef0004d40 0x7fef0c1429d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 -- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef0c153990 msgr2=0x7fef0c153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef0c153990 0x7fef0c153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0be00640 1 -- 172.21.15.97:0/3376337942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c153620 con 0x7fef0c1f77d0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.957+0000 7fef0aa00640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef0c153990 0x7fef0c153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.958+0000 7fef0be00640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 0x7fef0c1f7c50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fef04009f90 tx=0x7fef04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.958+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04039070 con 0x7fef0c1f77d0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.958+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0403a040 con 0x7fef0c1f77d0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.958+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04009c10 con 0x7fef0c1f77d0 2024-03-06T07:35:27.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.958+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c142f10 con 0x7fef0c1f77d0 2024-03-06T07:35:27.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.958+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c156690 con 0x7fef0c1f77d0 2024-03-06T07:35:27.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.960+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0c074ae0 con 0x7fef0c1f77d0 2024-03-06T07:35:27.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.960+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef04003b90 con 0x7fef0c1f77d0 2024-03-06T07:35:27.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.961+0000 7fef03400640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeec06b500 0x7feeec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.961+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fef040ac3b0 con 0x7fef0c1f77d0 2024-03-06T07:35:27.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.961+0000 7fef0b400640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeec06b500 0x7feeec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.962+0000 7fef0b400640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeec06b500 0x7feeec06d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7feefc009990 tx=0x7feefc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:27.965+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef0c153620 con 0x7fef0c1f77d0 2024-03-06T07:35:28.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.168+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef0c04f330 con 0x7fef0c1f77d0 2024-03-06T07:35:28.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.170+0000 7fef03400640 1 -- 172.21.15.97:0/3376337942 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fef0407aab0 con 0x7fef0c1f77d0 2024-03-06T07:35:28.173 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:28.173 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":118,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:29.791334+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.135:6833/322807700":"2024-03-07T07:34:20.730206+0000","172.21.15.135:6832/322807700":"2024-03-07T07:34:20.730206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.172+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeec06b500 msgr2=0x7feeec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.172+0000 7fef12800640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeec06b500 0x7feeec06d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7feefc009990 tx=0x7feefc008040 comp rx=0 tx=0).stop 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.172+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 msgr2=0x7fef0c1f7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.172+0000 7fef12800640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 0x7fef0c1f7c50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fef04009f90 tx=0x7fef04009750 comp rx=0 tx=0).stop 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 shutdown_connections 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feeec06b500 0x7feeec06d9c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1f77d0 0x7fef0c1f7c50 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef0c153990 0x7fef0c153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 --2- 172.21.15.97:0/3376337942 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feef0004d40 0x7fef0c1429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 >> 172.21.15.97:0/3376337942 conn(0x7fef0c0d0340 msgr2=0x7fef0c161f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 shutdown_connections 2024-03-06T07:35:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.173+0000 7fef12800640 1 -- 172.21.15.97:0/3376337942 wait complete. 2024-03-06T07:35:28.195 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:35:28.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:28.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.401+0000 7ff2fa800640 1 Processor -- start 2024-03-06T07:35:28.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.402+0000 7ff2fa800640 1 -- start start 2024-03-06T07:35:28.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.403+0000 7ff2fa800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 0x7ff2f415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.403+0000 7ff2fa800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff2f40b4390 0x7ff2f4160290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.403+0000 7ff2fa800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f40b3040 con 0x7ff2f41607d0 2024-03-06T07:35:28.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.403+0000 7ff2fa800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f415fb50 con 0x7ff2f40b4390 2024-03-06T07:35:28.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.403+0000 7ff2fa800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40cd3e0 con 0x7ff2f415f770 2024-03-06T07:35:28.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 0x7ff2f415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 0x7ff2f415fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:41546/0 (socket says 172.21.15.97:41546) 2024-03-06T07:35:28.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 -- 172.21.15.97:0/1130319616 learned_addr learned my addr 172.21.15.97:0/1130319616 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:28.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f2a00640 1 --2- 172.21.15.97:0/1130319616 >> v2:172.21.15.135:3300/0 conn(0x7ff2f40b4390 0x7ff2f4160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 -- 172.21.15.97:0/1130319616 >> v2:172.21.15.135:3300/0 conn(0x7ff2f40b4390 msgr2=0x7ff2f4160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 --2- 172.21.15.97:0/1130319616 >> v2:172.21.15.135:3300/0 conn(0x7ff2f40b4390 0x7ff2f4160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 -- 172.21.15.97:0/1130319616 >> v1:172.21.15.135:6789/0 conn(0x7ff2f41607d0 legacy=0x7ff2f40ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.404+0000 7ff2f3400640 1 -- 172.21.15.97:0/1130319616 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f40cd6c0 con 0x7ff2f415f770 2024-03-06T07:35:28.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.406+0000 7ff2f3400640 1 --2- 172.21.15.97:0/1130319616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 0x7ff2f415fd50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009870 tx=0x7ff2e402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6657667a4eb5be7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.406+0000 7ff2f2000640 1 -- 172.21.15.97:0/1130319616 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4035020 con 0x7ff2f415f770 2024-03-06T07:35:28.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.406+0000 7ff2f2000640 1 -- 172.21.15.97:0/1130319616 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e4031640 con 0x7ff2f415f770 2024-03-06T07:35:28.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.406+0000 7ff2f2000640 1 -- 172.21.15.97:0/1130319616 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e402d040 con 0x7ff2f415f770 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 -- 172.21.15.97:0/1130319616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 msgr2=0x7ff2f415fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 --2- 172.21.15.97:0/1130319616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 0x7ff2f415fd50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009870 tx=0x7ff2e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 -- 172.21.15.97:0/1130319616 shutdown_connections 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 --2- 172.21.15.97:0/1130319616 >> v2:172.21.15.135:3300/0 conn(0x7ff2f40b4390 0x7ff2f4160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 --2- 172.21.15.97:0/1130319616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f415f770 0x7ff2f415fd50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 -- 172.21.15.97:0/1130319616 >> 172.21.15.97:0/1130319616 conn(0x7ff2f40b3870 msgr2=0x7ff2f40b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 -- 172.21.15.97:0/1130319616 shutdown_connections 2024-03-06T07:35:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.407+0000 7ff2fa800640 1 -- 172.21.15.97:0/1130319616 wait complete. 2024-03-06T07:35:28.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.409+0000 7ff2fa800640 1 Processor -- start 2024-03-06T07:35:28.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.409+0000 7ff2fa800640 1 -- start start 2024-03-06T07:35:28.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.409+0000 7ff2fa800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f40b4390 0x7ff2f4154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2fa800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f415f770 0x7ff2f41546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2fa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 0x7ff2f40b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2fa800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40b34a0 con 0x7ff2f41607d0 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2fa800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40b3320 con 0x7ff2f415f770 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2fa800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40d09b0 con 0x7ff2f40b4390 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 0x7ff2f40b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 0x7ff2f40b4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41970/0 (socket says 172.21.15.97:41970) 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 -- 172.21.15.97:0/4027478693 learned_addr learned my addr 172.21.15.97:0/4027478693 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3400640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f40b4390 0x7ff2f4154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f2a00640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f415f770 0x7ff2f41546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 -- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f40b4390 msgr2=0x7ff2f4154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f40b4390 0x7ff2f4154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 -- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f415f770 msgr2=0x7ff2f41546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f415f770 0x7ff2f41546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.410+0000 7ff2f3e00640 1 -- 172.21.15.97:0/4027478693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f409cf60 con 0x7ff2f41607d0 2024-03-06T07:35:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.411+0000 7ff2f3e00640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 0x7ff2f40b4b90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00c390 tx=0x7ff2ec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.411+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec017070 con 0x7ff2f41607d0 2024-03-06T07:35:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.411+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec0052d0 con 0x7ff2f41607d0 2024-03-06T07:35:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.411+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec003810 con 0x7ff2f41607d0 2024-03-06T07:35:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.411+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f409d240 con 0x7ff2f41607d0 2024-03-06T07:35:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.411+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f409d800 con 0x7ff2f41607d0 2024-03-06T07:35:28.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.413+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f404f330 con 0x7ff2f41607d0 2024-03-06T07:35:28.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.413+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2ec01a070 con 0x7ff2f41607d0 2024-03-06T07:35:28.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.415+0000 7ff2eb400640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2d006b4d0 0x7ff2d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.415+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff2ec0895a0 con 0x7ff2f41607d0 2024-03-06T07:35:28.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.415+0000 7ff2f3400640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2d006b4d0 0x7ff2d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.415+0000 7ff2f3400640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2d006b4d0 0x7ff2d006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4036000 tx=0x7ff2e403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.419+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2ec058390 con 0x7ff2f41607d0 2024-03-06T07:35:28.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.676+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2f40b7cc0 con 0x7ff2f41607d0 2024-03-06T07:35:28.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.677+0000 7ff2eb400640 1 -- 172.21.15.97:0/4027478693 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7ff2ec057c90 con 0x7ff2f41607d0 2024-03-06T07:35:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:28.680 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:28.681 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:28.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2d006b4d0 msgr2=0x7ff2d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2d006b4d0 0x7ff2d006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4036000 tx=0x7ff2e403c040 comp rx=0 tx=0).stop 2024-03-06T07:35:28.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 msgr2=0x7ff2f40b4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 0x7ff2f40b4b90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00c390 tx=0x7ff2ec00c850 comp rx=0 tx=0).stop 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 shutdown_connections 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2d006b4d0 0x7ff2d006d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f41607d0 0x7ff2f40b4b90 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f415f770 0x7ff2f41546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 --2- 172.21.15.97:0/4027478693 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff2f40b4390 0x7ff2f4154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 >> 172.21.15.97:0/4027478693 conn(0x7ff2f40b3870 msgr2=0x7ff2f40c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 shutdown_connections 2024-03-06T07:35:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.680+0000 7ff2fa800640 1 -- 172.21.15.97:0/4027478693 wait complete. 2024-03-06T07:35:28.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:28.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.903+0000 7ff6d1400640 1 Processor -- start 2024-03-06T07:35:28.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.903+0000 7ff6d1400640 1 -- start start 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.904+0000 7ff6d1400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff6cc0b2a60 0x7ff6cc0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.904+0000 7ff6d1400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 0x7ff6cc0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.904+0000 7ff6d1400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6cc15f030 con 0x7ff6cc0d0600 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.904+0000 7ff6d1400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6cc0cf090 con 0x7ff6cc0b2a60 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.904+0000 7ff6d1400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc15f310 con 0x7ff6cc0d18e0 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.905+0000 7ff6cbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff6cc0d0600 0x7ff6cc0d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56820/0 (socket says 172.21.15.97:56820) 2024-03-06T07:35:28.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.905+0000 7ff6cbe00640 1 -- 172.21.15.97:0/1651403274 learned_addr learned my addr 172.21.15.97:0/1651403274 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.905+0000 7ff6cb400640 1 --2- 172.21.15.97:0/1651403274 >> v2:172.21.15.135:3300/0 conn(0x7ff6cc0b2a60 0x7ff6cc0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.905+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342909424 0 0) 0x7ff6cc15f030 con 0x7ff6cc0d0600 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.905+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b0003520 con 0x7ff6cc0d0600 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.905+0000 7ff6d0a00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 0x7ff6cc0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104764651 0 0) 0x7ff6b0003520 con 0x7ff6cc0d0600 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 msgr2=0x7ff6cc0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 0x7ff6cc0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 >> v2:172.21.15.135:3300/0 conn(0x7ff6cc0b2a60 msgr2=0x7ff6cc0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 --2- 172.21.15.97:0/1651403274 >> v2:172.21.15.135:3300/0 conn(0x7ff6cc0b2a60 0x7ff6cc0d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc0cea40 con 0x7ff6cc0d0600 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff6bc0019f0 con 0x7ff6cc0d0600 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 >> v1:172.21.15.97:6789/0 conn(0x7ff6cc0d0600 legacy=0x7ff6cc0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b00039b0 0x7ff6b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 0x7ff6b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0cf090 con 0x7ff6b0004d40 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc15f310 con 0x7ff6b00039b0 2024-03-06T07:35:28.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b000ba70 con 0x7ff6cc0d18e0 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6d0a00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6cbe00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b00039b0 0x7ff6b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6d0a00640 1 -- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 msgr2=0x7ff6b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6d0a00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc0d18e0 0x7ff6b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6d0a00640 1 -- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b00039b0 msgr2=0x7ff6b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6d0a00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b00039b0 0x7ff6b0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.906+0000 7ff6d0a00640 1 -- 172.21.15.97:0/1651403274 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b0003520 con 0x7ff6b0004d40 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6d0a00640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6b00051c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400ba30 tx=0x7ff6b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a824d3f3731d988d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4036020 con 0x7ff6b0004d40 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c4032680 con 0x7ff6b0004d40 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6caa00640 1 -- 172.21.15.97:0/1651403274 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c400b600 con 0x7ff6b0004d40 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6d1400640 1 -- 172.21.15.97:0/1651403274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 msgr2=0x7ff6b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6d1400640 1 --2- 172.21.15.97:0/1651403274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6b00051c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400ba30 tx=0x7ff6b0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6d0a00640 1 -- 172.21.15.97:0/1651403274 reap_dead start 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.908+0000 7ff6d1400640 1 -- 172.21.15.97:0/1651403274 shutdown_connections 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.909+0000 7ff6d1400640 1 -- 172.21.15.97:0/1651403274 >> 172.21.15.97:0/1651403274 conn(0x7ff6cc0d3d10 msgr2=0x7ff6cc0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.909+0000 7ff6d1400640 1 -- 172.21.15.97:0/1651403274 shutdown_connections 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.909+0000 7ff6d1400640 1 -- 172.21.15.97:0/1651403274 wait complete. 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.910+0000 7ff6d1400640 1 Processor -- start 2024-03-06T07:35:28.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.910+0000 7ff6d1400640 1 -- start start 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.911+0000 7ff6d1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6cc0b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.911+0000 7ff6d1400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc15a020 0x7ff6cc0b8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.911+0000 7ff6d1400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc1f7620 0x7ff6cc1f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.911+0000 7ff6d1400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc15ecf0 con 0x7ff6b0004d40 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.911+0000 7ff6d1400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0b3250 con 0x7ff6cc1f7620 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.911+0000 7ff6d1400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0d1d70 con 0x7ff6cc15a020 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6cc0b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6cc0b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41986/0 (socket says 172.21.15.97:41986) 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 -- 172.21.15.97:0/743519465 learned_addr learned my addr 172.21.15.97:0/743519465 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 -- 172.21.15.97:0/743519465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc15a020 msgr2=0x7ff6cc0b8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc15a020 0x7ff6cc0b8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 -- 172.21.15.97:0/743519465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc1f7620 msgr2=0x7ff6cc1f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc1f7620 0x7ff6cc1f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 -- 172.21.15.97:0/743519465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc0b3ce0 con 0x7ff6b0004d40 2024-03-06T07:35:28.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.912+0000 7ff6cbe00640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6cc0b5ed0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc002150 tx=0x7ff6bc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.913+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc019070 con 0x7ff6b0004d40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.913+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6bc004d60 con 0x7ff6b0004d40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.913+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc00be60 con 0x7ff6b0004d40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.913+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc1f8430 con 0x7ff6b0004d40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.913+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc0b4260 con 0x7ff6b0004d40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.914+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6cc04f330 con 0x7ff6b0004d40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.915+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6bc0072d0 con 0x7ff6b0004d40 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.917+0000 7ff6c3e00640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff6a806b500 0x7ff6a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.917+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff6bc08a2c0 con 0x7ff6b0004d40 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.917+0000 7ff6cb400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff6a806b500 0x7ff6a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.918+0000 7ff6cb400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff6a806b500 0x7ff6a806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4006a90 tx=0x7ff6b4006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:28.920+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6bc059140 con 0x7ff6b0004d40 2024-03-06T07:35:29.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.163+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6cc0b44a0 con 0x7ff6b0004d40 2024-03-06T07:35:29.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.164+0000 7ff6c3e00640 1 -- 172.21.15.97:0/743519465 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4987 (secure 0 0 0) 0x7ff6bc058a40 con 0x7ff6b0004d40 2024-03-06T07:35:29.167 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:29.167 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:05.874005+0000","modified":"2024-03-06T07:34:24.644540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7977":{"gid":7977,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":4,"addr":"172.21.15.97:6837/3308266804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3308266804},{"type":"v1","addr":"172.21.15.97:6837","nonce":3308266804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7977,"qdb_cluster":[7977]},"id":8}]} 2024-03-06T07:35:29.167 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 80 2024-03-06T07:35:29.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff6a806b500 msgr2=0x7ff6a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff6a806b500 0x7ff6a806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4006a90 tx=0x7ff6b4006a20 comp rx=0 tx=0).stop 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 msgr2=0x7ff6cc0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6cc0b5ed0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc002150 tx=0x7ff6bc00b4a0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 shutdown_connections 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff6a806b500 0x7ff6a806d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc1f7620 0x7ff6cc1f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff6cc15a020 0x7ff6cc0b8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 --2- 172.21.15.97:0/743519465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6b0004d40 0x7ff6cc0b5ed0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 >> 172.21.15.97:0/743519465 conn(0x7ff6cc0d3d10 msgr2=0x7ff6cc0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.167+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 shutdown_connections 2024-03-06T07:35:29.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.168+0000 7ff6d1400640 1 -- 172.21.15.97:0/743519465 wait complete. 2024-03-06T07:35:29.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:29.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.381+0000 7f77d8200640 1 Processor -- start 2024-03-06T07:35:29.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.382+0000 7f77d8200640 1 -- start start 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.383+0000 7f77d8200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f77d0144c00 0x7f77d006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.383+0000 7f77d8200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77d015f9d0 0x7f77d015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.383+0000 7f77d8200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d006ee70 con 0x7f77d00ce0a0 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.383+0000 7f77d8200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d006cef0 con 0x7f77d015f9d0 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.383+0000 7f77d8200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d006d1d0 con 0x7f77d0144c00 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77d015f9d0 0x7f77d015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77d015f9d0 0x7f77d015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41992/0 (socket says 172.21.15.97:41992) 2024-03-06T07:35:29.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 -- 172.21.15.97:0/3457880619 learned_addr learned my addr 172.21.15.97:0/3457880619 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:29.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d5800640 1 --2- 172.21.15.97:0/3457880619 >> v2:172.21.15.135:3300/0 conn(0x7f77d0144c00 0x7f77d006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 -- 172.21.15.97:0/3457880619 >> v2:172.21.15.135:3300/0 conn(0x7f77d0144c00 msgr2=0x7f77d006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 --2- 172.21.15.97:0/3457880619 >> v2:172.21.15.135:3300/0 conn(0x7f77d0144c00 0x7f77d006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 -- 172.21.15.97:0/3457880619 >> v1:172.21.15.135:6789/0 conn(0x7f77d00ce0a0 legacy=0x7f77d00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d4e00640 1 -- 172.21.15.97:0/3457880619 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d006d4b0 con 0x7f77d015f9d0 2024-03-06T07:35:29.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.384+0000 7f77d5800640 1 --2- 172.21.15.97:0/3457880619 >> v2:172.21.15.135:3300/0 conn(0x7f77d0144c00 0x7f77d006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:29.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.385+0000 7f77d4e00640 1 --2- 172.21.15.97:0/3457880619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d015f9d0 0x7f77d015fe60 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f77d00b25f0 tx=0x7f77bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba1606b90ff4cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.386+0000 7f77cfe00640 1 -- 172.21.15.97:0/3457880619 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc035020 con 0x7f77d015f9d0 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.386+0000 7f77cfe00640 1 -- 172.21.15.97:0/3457880619 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc031870 con 0x7f77d015f9d0 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.386+0000 7f77cfe00640 1 -- 172.21.15.97:0/3457880619 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc031c40 con 0x7f77d015f9d0 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.386+0000 7f77d8200640 1 -- 172.21.15.97:0/3457880619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d015f9d0 msgr2=0x7f77d015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.386+0000 7f77d8200640 1 --2- 172.21.15.97:0/3457880619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d015f9d0 0x7f77d015fe60 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f77d00b25f0 tx=0x7f77bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.387+0000 7f77d8200640 1 -- 172.21.15.97:0/3457880619 shutdown_connections 2024-03-06T07:35:29.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.387+0000 7f77d8200640 1 --2- 172.21.15.97:0/3457880619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d015f9d0 0x7f77d015fe60 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.387+0000 7f77d8200640 1 --2- 172.21.15.97:0/3457880619 >> v2:172.21.15.135:3300/0 conn(0x7f77d0144c00 0x7f77d006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.387+0000 7f77d8200640 1 -- 172.21.15.97:0/3457880619 >> 172.21.15.97:0/3457880619 conn(0x7f77d00b1730 msgr2=0x7f77d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.387+0000 7f77d8200640 1 -- 172.21.15.97:0/3457880619 shutdown_connections 2024-03-06T07:35:29.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.387+0000 7f77d8200640 1 -- 172.21.15.97:0/3457880619 wait complete. 2024-03-06T07:35:29.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.389+0000 7f77d8200640 1 Processor -- start 2024-03-06T07:35:29.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.390+0000 7f77d8200640 1 -- start start 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d8200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77d00ce0a0 0x7f77d013ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d8200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 0x7f77d013bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d8200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d015f9d0 0x7f77d013c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d8200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d006cb50 con 0x7f77d0144c00 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d8200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d0160730 con 0x7f77d015f9d0 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d8200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d006ccd0 con 0x7f77d00ce0a0 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 0x7f77d013bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 0x7f77d013bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42006/0 (socket says 172.21.15.97:42006) 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.391+0000 7f77d4e00640 1 -- 172.21.15.97:0/2548232794 learned_addr learned my addr 172.21.15.97:0/2548232794 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:29.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d5800640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77d00ce0a0 0x7f77d013ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d6200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d015f9d0 0x7f77d013c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d4e00640 1 -- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77d00ce0a0 msgr2=0x7f77d013ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d4e00640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77d00ce0a0 0x7f77d013ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d4e00640 1 -- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d015f9d0 msgr2=0x7f77d013c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d4e00640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d015f9d0 0x7f77d013c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d4e00640 1 -- 172.21.15.97:0/2548232794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d014cf40 con 0x7f77d0144c00 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.392+0000 7f77d4e00640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 0x7f77d013bff0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f77bc000c00 tx=0x7f77bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.393+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc040070 con 0x7f77d0144c00 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.393+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc03a420 con 0x7f77d0144c00 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.393+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc035040 con 0x7f77d0144c00 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.393+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d014d220 con 0x7f77d0144c00 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.393+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d014d7e0 con 0x7f77d0144c00 2024-03-06T07:35:29.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.394+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d004f330 con 0x7f77d0144c00 2024-03-06T07:35:29.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.396+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77bc04b060 con 0x7f77d0144c00 2024-03-06T07:35:29.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.397+0000 7f77cd600640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f77b006b500 0x7f77b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.397+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f77bc0ac280 con 0x7f77d0144c00 2024-03-06T07:35:29.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.397+0000 7f77d5800640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f77b006b500 0x7f77b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.398+0000 7f77d5800640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f77b006b500 0x7f77b006d9c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f77c4008850 tx=0x7f77c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.402+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77d006ee70 con 0x7f77d0144c00 2024-03-06T07:35:29.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.600+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77d0072890 con 0x7f77d0144c00 2024-03-06T07:35:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.603+0000 7f77cd600640 1 -- 172.21.15.97:0/2548232794 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f77bc07afd0 con 0x7f77d0144c00 2024-03-06T07:35:29.605 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:29.605 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":118,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:34:29.791334+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:03.618857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:04.895815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.135:6833/322807700":"2024-03-07T07:34:20.730206+0000","172.21.15.135:6832/322807700":"2024-03-07T07:34:20.730206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f77b006b500 msgr2=0x7f77b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f77b006b500 0x7f77b006d9c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f77c4008850 tx=0x7f77c4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 msgr2=0x7f77d013bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 0x7f77d013bff0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f77bc000c00 tx=0x7f77bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 shutdown_connections 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f77b006b500 0x7f77b006d9c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d015f9d0 0x7f77d013c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77d0144c00 0x7f77d013bff0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 --2- 172.21.15.97:0/2548232794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f77d00ce0a0 0x7f77d013ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.605+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 >> 172.21.15.97:0/2548232794 conn(0x7f77d00b1730 msgr2=0x7f77d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.606+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 shutdown_connections 2024-03-06T07:35:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.606+0000 7f77d8200640 1 -- 172.21.15.97:0/2548232794 wait complete. 2024-03-06T07:35:29.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:35:29.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.829+0000 7efe78a00640 1 Processor -- start 2024-03-06T07:35:29.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.829+0000 7efe78a00640 1 -- start start 2024-03-06T07:35:29.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.830+0000 7efe78a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efe740c6460 0x7efe740c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.830+0000 7efe78a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe74145b30 con 0x7efe740c5960 2024-03-06T07:35:29.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.830+0000 7efe78a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe74142ce0 con 0x7efe740c4930 2024-03-06T07:35:29.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.830+0000 7efe78a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe740c68c0 con 0x7efe740c6460 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.831+0000 7efe72800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efe740c5960 0x7efe74165b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56836/0 (socket says 172.21.15.97:56836) 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.831+0000 7efe72800640 1 -- 172.21.15.97:0/2404426969 learned_addr learned my addr 172.21.15.97:0/2404426969 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.831+0000 7efe69400640 1 --2- 172.21.15.97:0/2404426969 >> v2:172.21.15.135:3300/0 conn(0x7efe740c6460 0x7efe740c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.831+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513981589 0 0) 0x7efe6c001110 con 0x7efe740c5960 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.831+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58003520 con 0x7efe740c5960 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.831+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874092356 0 0) 0x7efe58003520 con 0x7efe740c5960 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 >> v2:172.21.15.135:3300/0 conn(0x7efe740c6460 msgr2=0x7efe740c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 --2- 172.21.15.97:0/2404426969 >> v2:172.21.15.135:3300/0 conn(0x7efe740c6460 0x7efe740c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 >> v1:172.21.15.135:6789/0 conn(0x7efe740c4930 legacy=0x7efe740c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740cbca0 con 0x7efe740c5960 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7efe6c001840 con 0x7efe740c5960 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 >> v1:172.21.15.97:6789/0 conn(0x7efe740c5960 legacy=0x7efe74165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe58003a00 0x7efe58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58004800 0x7efe58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe58005210 0x7efe5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740c68c0 con 0x7efe58003a00 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5800c160 con 0x7efe58005210 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5800c360 con 0x7efe58004800 2024-03-06T07:35:29.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe71e00640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe58003a00 0x7efe58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe72800640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe58005210 0x7efe5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.832+0000 7efe69400640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58004800 0x7efe58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe71e00640 1 -- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58004800 msgr2=0x7efe58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe71e00640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58004800 0x7efe58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe71e00640 1 -- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe58005210 msgr2=0x7efe5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe71e00640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe58005210 0x7efe5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe71e00640 1 -- 172.21.15.97:0/2404426969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe58003520 con 0x7efe58003a00 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe69400640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58004800 0x7efe58004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe72800640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe58005210 0x7efe5800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.833+0000 7efe72800640 1 -- 172.21.15.97:0/2404426969 reap_dead start 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.834+0000 7efe71e00640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe58003a00 0x7efe58003e20 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7efe64001e20 tx=0x7efe58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c08991ae0edbd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.834+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64001a70 con 0x7efe58003a00 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.834+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe64003b70 con 0x7efe58003a00 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.834+0000 7efe71400640 1 -- 172.21.15.97:0/2404426969 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640050d0 con 0x7efe58003a00 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.834+0000 7efe78a00640 1 -- 172.21.15.97:0/2404426969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe58003a00 msgr2=0x7efe58003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.834+0000 7efe78a00640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe58003a00 0x7efe58003e20 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7efe64001e20 tx=0x7efe58005830 comp rx=0 tx=0).stop 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.835+0000 7efe78a00640 1 -- 172.21.15.97:0/2404426969 shutdown_connections 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.835+0000 7efe78a00640 1 --2- 172.21.15.97:0/2404426969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe58003a00 0x7efe58003e20 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.835+0000 7efe78a00640 1 -- 172.21.15.97:0/2404426969 >> 172.21.15.97:0/2404426969 conn(0x7efe740cc4f0 msgr2=0x7efe740cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.835+0000 7efe78a00640 1 -- 172.21.15.97:0/2404426969 shutdown_connections 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.835+0000 7efe78a00640 1 -- 172.21.15.97:0/2404426969 wait complete. 2024-03-06T07:35:29.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.836+0000 7efe78a00640 1 Processor -- start 2024-03-06T07:35:29.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.837+0000 7efe78a00640 1 -- start start 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe78a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58003a00 0x7efe740cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe78a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 0x7efe741f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe78a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe741409a0 0x7efe74140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe78a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740ca3b0 con 0x7efe7413e270 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe78a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740ca0d0 con 0x7efe741409a0 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe78a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe740cc260 con 0x7efe58003a00 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 0x7efe741f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 0x7efe741f3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42016/0 (socket says 172.21.15.97:42016) 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 -- 172.21.15.97:0/3528429061 learned_addr learned my addr 172.21.15.97:0/3528429061 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe72800640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe741409a0 0x7efe74140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe71e00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58003a00 0x7efe740cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 -- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58003a00 msgr2=0x7efe740cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58003a00 0x7efe740cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 -- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe741409a0 msgr2=0x7efe74140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe741409a0 0x7efe74140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.838+0000 7efe69400640 1 -- 172.21.15.97:0/3528429061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe741f4530 con 0x7efe7413e270 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.839+0000 7efe69400640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 0x7efe741f3ff0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7efe5c0107f0 tx=0x7efe5c010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.839+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00f040 con 0x7efe7413e270 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.839+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c00eb60 con 0x7efe7413e270 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.839+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c01c400 con 0x7efe7413e270 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.839+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe740ce9e0 con 0x7efe7413e270 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.839+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7413fee0 con 0x7efe7413e270 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.840+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe740cb530 con 0x7efe7413e270 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.841+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe5c01c5a0 con 0x7efe7413e270 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.842+0000 7efe6aa00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5006b500 0x7efe5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.842+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5853+0+0 (secure 0 0 0) 0x7efe5c08a6f0 con 0x7efe7413e270 2024-03-06T07:35:29.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.843+0000 7efe71e00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5006b500 0x7efe5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.843+0000 7efe71e00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5006b500 0x7efe5006d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efe64034000 tx=0x7efe64009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:29.847+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe740cb530 con 0x7efe7413e270 2024-03-06T07:35:30.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.082+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efe7404f330 con 0x7efe7413e270 2024-03-06T07:35:30.085 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:35:30.083+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:35:30.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.086+0000 7efe6aa00640 1 -- 172.21.15.97:0/3528429061 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7efe5c018030 con 0x7efe7413e270 2024-03-06T07:35:30.088 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5006b500 msgr2=0x7efe5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5006b500 0x7efe5006d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efe64034000 tx=0x7efe64009030 comp rx=0 tx=0).stop 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 msgr2=0x7efe741f3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 0x7efe741f3ff0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7efe5c0107f0 tx=0x7efe5c010cb0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 shutdown_connections 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efe5006b500 0x7efe5006d9c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efe741409a0 0x7efe74140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe7413e270 0x7efe741f3ff0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 --2- 172.21.15.97:0/3528429061 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efe58003a00 0x7efe740cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.089+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 >> 172.21.15.97:0/3528429061 conn(0x7efe740cc4f0 msgr2=0x7efe74071ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.090+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 shutdown_connections 2024-03-06T07:35:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.090+0000 7efe78a00640 1 -- 172.21.15.97:0/3528429061 wait complete. 2024-03-06T07:35:30.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:35:30.118 INFO:tasks.ceph.mds.c.smithi097.stderr: -8169> 2024-03-06T07:34:26.650+0000 7f7bbce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:30.181 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:35:30.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.302+0000 7fa762200640 1 Processor -- start 2024-03-06T07:35:30.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.303+0000 7fa762200640 1 -- start start 2024-03-06T07:35:30.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.304+0000 7fa762200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa75c0b3df0 0x7fa75c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.304+0000 7fa762200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c0719e0 con 0x7fa75c0b3650 2024-03-06T07:35:30.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.304+0000 7fa762200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c071cc0 con 0x7fa75c0cda90 2024-03-06T07:35:30.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.304+0000 7fa762200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c072310 con 0x7fa75c0b3df0 2024-03-06T07:35:30.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.304+0000 7fa75aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa75c0b3650 0x7fa75c0b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56838/0 (socket says 172.21.15.97:56838) 2024-03-06T07:35:30.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.304+0000 7fa75aa00640 1 -- 172.21.15.97:0/2917930546 learned_addr learned my addr 172.21.15.97:0/2917930546 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:30.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75b400640 1 --2- 172.21.15.97:0/2917930546 >> v2:172.21.15.135:3300/0 conn(0x7fa75c0b3df0 0x7fa75c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171723503 0 0) 0x7fa75c0719e0 con 0x7fa75c0b3650 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa740003520 con 0x7fa75c0b3650 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2659134455 0 0) 0x7fa740003520 con 0x7fa75c0b3650 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 >> v2:172.21.15.135:3300/0 conn(0x7fa75c0b3df0 msgr2=0x7fa75c0727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 --2- 172.21.15.97:0/2917930546 >> v2:172.21.15.135:3300/0 conn(0x7fa75c0b3df0 0x7fa75c0727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 >> v1:172.21.15.135:6789/0 conn(0x7fa75c0cda90 legacy=0x7fa75c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c071fa0 con 0x7fa75c0b3650 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa744001670 con 0x7fa75c0b3650 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 >> v1:172.21.15.97:6789/0 conn(0x7fa75c0b3650 legacy=0x7fa75c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa740003a00 0x7fa740003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa740004800 0x7fa740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa74000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c072310 con 0x7fa740005210 2024-03-06T07:35:30.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74000c140 con 0x7fa740003a00 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.305+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74000c340 con 0x7fa740004800 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75be00640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa74000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75aa00640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa740004800 0x7fa740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75be00640 1 -- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa740004800 msgr2=0x7fa740004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75b400640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa740003a00 0x7fa740003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75be00640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa740004800 0x7fa740004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75be00640 1 -- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa740003a00 msgr2=0x7fa740003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75be00640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa740003a00 0x7fa740003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75be00640 1 -- 172.21.15.97:0/2917930546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa740003520 con 0x7fa740005210 2024-03-06T07:35:30.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.306+0000 7fa75aa00640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa740004800 0x7fa740004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.307+0000 7fa75be00640 1 -- 172.21.15.97:0/2917930546 reap_dead start 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.307+0000 7fa75be00640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa74000b9c0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa754004480 tx=0x7fa740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef13bffe36fb964 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.307+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754034070 con 0x7fa740005210 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.307+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa754030400 con 0x7fa740005210 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.307+0000 7fa75a000640 1 -- 172.21.15.97:0/2917930546 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754004080 con 0x7fa740005210 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 -- 172.21.15.97:0/2917930546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 msgr2=0x7fa74000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa74000b9c0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa754004480 tx=0x7fa740005830 comp rx=0 tx=0).stop 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 -- 172.21.15.97:0/2917930546 shutdown_connections 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 --2- 172.21.15.97:0/2917930546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa74000b9c0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 -- 172.21.15.97:0/2917930546 >> 172.21.15.97:0/2917930546 conn(0x7fa75c0b1940 msgr2=0x7fa75c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 -- 172.21.15.97:0/2917930546 shutdown_connections 2024-03-06T07:35:30.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.308+0000 7fa762200640 1 -- 172.21.15.97:0/2917930546 wait complete. 2024-03-06T07:35:30.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.310+0000 7fa762200640 1 Processor -- start 2024-03-06T07:35:30.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.310+0000 7fa762200640 1 -- start start 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa762200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa75c15f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa762200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa75c14f230 0x7fa75c0ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa762200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa75c14f6b0 0x7fa75c0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa762200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15b3c0 con 0x7fa740005210 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa762200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15b240 con 0x7fa75c14f6b0 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa762200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c0cf510 con 0x7fa75c14f230 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa75b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa75c15f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa75b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa75c15f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42040/0 (socket says 172.21.15.97:42040) 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.311+0000 7fa75b400640 1 -- 172.21.15.97:0/1807856062 learned_addr learned my addr 172.21.15.97:0/1807856062 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75aa00640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa75c14f230 0x7fa75c0ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75b400640 1 -- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa75c14f230 msgr2=0x7fa75c0ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75be00640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa75c14f6b0 0x7fa75c0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75b400640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa75c14f230 0x7fa75c0ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75b400640 1 -- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa75c14f6b0 msgr2=0x7fa75c0ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75b400640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa75c14f6b0 0x7fa75c0ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75b400640 1 -- 172.21.15.97:0/1807856062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c09c570 con 0x7fa740005210 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75aa00640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa75c14f230 0x7fa75c0ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75be00640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa75c14f6b0 0x7fa75c0ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa75b400640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa75c15f510 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00e7e0 tx=0x7fa74c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c017070 con 0x7fa740005210 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c00c6d0 con 0x7fa740005210 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c00f040 con 0x7fa740005210 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c09c850 con 0x7fa740005210 2024-03-06T07:35:30.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.312+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c09ce10 con 0x7fa740005210 2024-03-06T07:35:30.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.313+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c04f330 con 0x7fa740005210 2024-03-06T07:35:30.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.314+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa74c01a070 con 0x7fa740005210 2024-03-06T07:35:30.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.315+0000 7fa753400640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa73806b4d0 0x7fa73806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.315+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa74c089e90 con 0x7fa740005210 2024-03-06T07:35:30.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.316+0000 7fa75aa00640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa73806b4d0 0x7fa73806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.316+0000 7fa75aa00640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa73806b4d0 0x7fa73806d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa744001eb0 tx=0x7fa744009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.320+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa74c058cb0 con 0x7fa740005210 2024-03-06T07:35:30.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:30.551+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa75c0cc0e0 con 0x7fa740005210 2024-03-06T07:35:31.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.090+0000 7fa753400640 1 -- 172.21.15.97:0/1807856062 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa74c0585b0 con 0x7fa740005210 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.093+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa73806b4d0 msgr2=0x7fa73806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.093+0000 7fa762200640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa73806b4d0 0x7fa73806d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa744001eb0 tx=0x7fa744009040 comp rx=0 tx=0).stop 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.093+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 msgr2=0x7fa75c15f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.093+0000 7fa762200640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa75c15f510 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00e7e0 tx=0x7fa74c00eca0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.093+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 shutdown_connections 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa73806b4d0 0x7fa73806d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa75c14f6b0 0x7fa75c0ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa75c14f230 0x7fa75c0ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 --2- 172.21.15.97:0/1807856062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa740005210 0x7fa75c15f510 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 >> 172.21.15.97:0/1807856062 conn(0x7fa75c0b1940 msgr2=0x7fa75c0b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 shutdown_connections 2024-03-06T07:35:31.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.094+0000 7fa762200640 1 -- 172.21.15.97:0/1807856062 wait complete. 2024-03-06T07:35:31.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:35:31.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.372+0000 7f2ba0000640 1 Processor -- start 2024-03-06T07:35:31.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.372+0000 7f2ba0000640 1 -- start start 2024-03-06T07:35:31.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2ba0000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2b9806e150 0x7f2b98168bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2ba0000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b980ceb50 con 0x7f2b980cfe90 2024-03-06T07:35:31.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2ba0000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b980cedf0 con 0x7f2b9806d660 2024-03-06T07:35:31.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2ba0000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b980cf0d0 con 0x7f2b9806e150 2024-03-06T07:35:31.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2b9cc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2b980cfe90 0x7f2b9806da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56840/0 (socket says 172.21.15.97:56840) 2024-03-06T07:35:31.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2b9cc00640 1 -- 172.21.15.97:0/3936653422 learned_addr learned my addr 172.21.15.97:0/3936653422 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:31.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2b9e000640 1 --2- 172.21.15.97:0/3936653422 >> v2:172.21.15.135:3300/0 conn(0x7f2b9806e150 0x7f2b98168bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.374+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028935265 0 0) 0x7f2b980ceb50 con 0x7f2b980cfe90 2024-03-06T07:35:31.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b80003520 con 0x7f2b980cfe90 2024-03-06T07:35:31.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686351979 0 0) 0x7f2b80003520 con 0x7f2b980cfe90 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 >> v2:172.21.15.135:3300/0 conn(0x7f2b9806e150 msgr2=0x7f2b98168bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 --2- 172.21.15.97:0/3936653422 >> v2:172.21.15.135:3300/0 conn(0x7f2b9806e150 0x7f2b98168bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 >> v1:172.21.15.135:6789/0 conn(0x7f2b9806d660 legacy=0x7f2b98071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980cf3b0 con 0x7f2b980cfe90 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2b84001670 con 0x7f2b980cfe90 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 >> v1:172.21.15.97:6789/0 conn(0x7f2b980cfe90 legacy=0x7f2b9806da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b80003a50 0x7f2b80003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b80004800 0x7f2b80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b800052e0 0x7f2b8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980cf0d0 con 0x7f2b80004800 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8000c1b0 con 0x7f2b80003a50 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8000c3b0 con 0x7f2b800052e0 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.375+0000 7f2b9cc00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b80004800 0x7f2b80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9d600640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b80003a50 0x7f2b80003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9cc00640 1 -- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b800052e0 msgr2=0x7f2b8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9cc00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b800052e0 0x7f2b8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9cc00640 1 -- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b80003a50 msgr2=0x7f2b80003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9cc00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b80003a50 0x7f2b80003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9cc00640 1 -- 172.21.15.97:0/3936653422 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b80003520 con 0x7f2b80004800 2024-03-06T07:35:31.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.376+0000 7f2b9e000640 1 -- 172.21.15.97:0/3936653422 reap_dead start 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.377+0000 7f2b9cc00640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b80004800 0x7f2b80004c80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400c210 tx=0x7f2b80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e8b5af2c3db8b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.377+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84007cf0 con 0x7f2b80004800 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.377+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b84030070 con 0x7f2b80004800 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.377+0000 7f2b97e00640 1 -- 172.21.15.97:0/3936653422 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84005d70 con 0x7f2b80004800 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 -- 172.21.15.97:0/3936653422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b80004800 msgr2=0x7f2b80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b80004800 0x7f2b80004c80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400c210 tx=0x7f2b80005830 comp rx=0 tx=0).stop 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 -- 172.21.15.97:0/3936653422 shutdown_connections 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 --2- 172.21.15.97:0/3936653422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b80004800 0x7f2b80004c80 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 -- 172.21.15.97:0/3936653422 >> 172.21.15.97:0/3936653422 conn(0x7f2b980d0340 msgr2=0x7f2b980d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 -- 172.21.15.97:0/3936653422 shutdown_connections 2024-03-06T07:35:31.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.378+0000 7f2ba0000640 1 -- 172.21.15.97:0/3936653422 wait complete. 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.379+0000 7f2ba0000640 1 Processor -- start 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.380+0000 7f2ba0000640 1 -- start start 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.380+0000 7f2ba0000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b80004800 0x7f2b9815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.380+0000 7f2ba0000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b981f7b10 0x7f2b98148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2ba0000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 0x7f2b98160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2ba0000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9806da40 con 0x7f2b98160b30 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2ba0000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980cf970 con 0x7f2b981f7b10 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2ba0000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9806dbc0 con 0x7f2b80004800 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 0x7f2b98160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 0x7f2b98160fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42056/0 (socket says 172.21.15.97:42056) 2024-03-06T07:35:31.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 -- 172.21.15.97:0/636930392 learned_addr learned my addr 172.21.15.97:0/636930392 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9d600640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b80004800 0x7f2b9815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9cc00640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b981f7b10 0x7f2b98148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 -- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b80004800 msgr2=0x7f2b9815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b80004800 0x7f2b9815b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 -- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b981f7b10 msgr2=0x7f2b98148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b981f7b10 0x7f2b98148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.381+0000 7f2b9e000640 1 -- 172.21.15.97:0/636930392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9815a700 con 0x7f2b98160b30 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.382+0000 7f2b9e000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 0x7f2b98160fc0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f2b9806cc80 tx=0x7f2b9000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.382+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000bce0 con 0x7f2b98160b30 2024-03-06T07:35:31.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.382+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9001a070 con 0x7f2b98160b30 2024-03-06T07:35:31.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.382+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90014640 con 0x7f2b98160b30 2024-03-06T07:35:31.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.382+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9815a9e0 con 0x7f2b98160b30 2024-03-06T07:35:31.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.382+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9815afa0 con 0x7f2b98160b30 2024-03-06T07:35:31.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.383+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b980b39a0 con 0x7f2b98160b30 2024-03-06T07:35:31.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.384+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b90005d10 con 0x7f2b98160b30 2024-03-06T07:35:31.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.385+0000 7f2b95600640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b7806b4d0 0x7f2b7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.385+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f2b9008a6c0 con 0x7f2b98160b30 2024-03-06T07:35:31.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.385+0000 7f2b9d600640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b7806b4d0 0x7f2b7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.385+0000 7f2b9d600640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b7806b4d0 0x7f2b7806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c007fd0 tx=0x7f2b8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.389+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b980b39a0 con 0x7f2b98160b30 2024-03-06T07:35:31.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:31.588+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2b9804f3a0 con 0x7f2b98160b30 2024-03-06T07:35:32.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.237+0000 7f2b95600640 1 -- 172.21.15.97:0/636930392 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2b90024020 con 0x7f2b98160b30 2024-03-06T07:35:32.239 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:35:32.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b7806b4d0 msgr2=0x7f2b7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b7806b4d0 0x7f2b7806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c007fd0 tx=0x7f2b8c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:32.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 msgr2=0x7f2b98160fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 0x7f2b98160fc0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f2b9806cc80 tx=0x7f2b9000b480 comp rx=0 tx=0).stop 2024-03-06T07:35:32.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 shutdown_connections 2024-03-06T07:35:32.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b7806b4d0 0x7f2b7806d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b98160b30 0x7f2b98160fc0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b981f7b10 0x7f2b98148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 --2- 172.21.15.97:0/636930392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b80004800 0x7f2b9815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.241+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 >> 172.21.15.97:0/636930392 conn(0x7f2b980d0340 msgr2=0x7f2b980b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.243+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 shutdown_connections 2024-03-06T07:35:32.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.244+0000 7f2ba0000640 1 -- 172.21.15.97:0/636930392 wait complete. 2024-03-06T07:35:32.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:35:32.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.466+0000 7f679c200640 1 Processor -- start 2024-03-06T07:35:32.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.466+0000 7f679c200640 1 -- start start 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f679c200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6794160110 0x7f679406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f679c200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67940ced20 0x7f679406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f679c200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67940cf1e0 con 0x7f6794145980 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f679c200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6794070e40 con 0x7f6794160110 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f679c200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67940cf4c0 con 0x7f67940ced20 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f6798e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6794160110 0x7f679406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f6798e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6794160110 0x7f679406ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42070/0 (socket says 172.21.15.97:42070) 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f6798e00640 1 -- 172.21.15.97:0/2768223307 learned_addr learned my addr 172.21.15.97:0/2768223307 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f6799800640 1 --1- 172.21.15.97:0/2768223307 >> v1:172.21.15.97:6789/0 conn(0x7f6794145980 0x7f679415fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56854/0 (socket says 172.21.15.97:56854) 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.468+0000 7f679a200640 1 --2- 172.21.15.97:0/2768223307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67940ced20 0x7f679406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.469+0000 7f6798e00640 1 -- 172.21.15.97:0/2768223307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67940ced20 msgr2=0x7f679406d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.469+0000 7f6798e00640 1 --2- 172.21.15.97:0/2768223307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67940ced20 0x7f679406d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.469+0000 7f6798e00640 1 -- 172.21.15.97:0/2768223307 >> v1:172.21.15.97:6789/0 conn(0x7f6794145980 legacy=0x7f679415fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.469+0000 7f6798e00640 1 -- 172.21.15.97:0/2768223307 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940cf7a0 con 0x7f6794160110 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.470+0000 7f6798e00640 1 --2- 172.21.15.97:0/2768223307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679406ceb0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6780009870 tx=0x7f678002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbd9f6ff569383e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.470+0000 7f6793e00640 1 -- 172.21.15.97:0/2768223307 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780034070 con 0x7f6794160110 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.470+0000 7f6793e00640 1 -- 172.21.15.97:0/2768223307 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6780030840 con 0x7f6794160110 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.470+0000 7f6793e00640 1 -- 172.21.15.97:0/2768223307 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780030c10 con 0x7f6794160110 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 -- 172.21.15.97:0/2768223307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 msgr2=0x7f679406ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 --2- 172.21.15.97:0/2768223307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679406ceb0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6780009870 tx=0x7f678002b980 comp rx=0 tx=0).stop 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 -- 172.21.15.97:0/2768223307 shutdown_connections 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 --2- 172.21.15.97:0/2768223307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67940ced20 0x7f679406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 --2- 172.21.15.97:0/2768223307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679406ceb0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 -- 172.21.15.97:0/2768223307 >> 172.21.15.97:0/2768223307 conn(0x7f67940cfa80 msgr2=0x7f67940cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 -- 172.21.15.97:0/2768223307 shutdown_connections 2024-03-06T07:35:32.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.471+0000 7f679c200640 1 -- 172.21.15.97:0/2768223307 wait complete. 2024-03-06T07:35:32.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.473+0000 7f679c200640 1 Processor -- start 2024-03-06T07:35:32.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.473+0000 7f679c200640 1 -- start start 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679c200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67940ced20 0x7f6794134060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679c200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6794145980 0x7f67941585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679c200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679415e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679c200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67941608c0 con 0x7f6794160110 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679c200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794160740 con 0x7f67940ced20 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679c200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67940722e0 con 0x7f6794145980 2024-03-06T07:35:32.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679415e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679415e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42072/0 (socket says 172.21.15.97:42072) 2024-03-06T07:35:32.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679a200640 1 -- 172.21.15.97:0/891605111 learned_addr learned my addr 172.21.15.97:0/891605111 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679a200640 1 -- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6794145980 msgr2=0x7f67941585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679a200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6794145980 0x7f67941585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f6799800640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67940ced20 0x7f6794134060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.474+0000 7f679a200640 1 -- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67940ced20 msgr2=0x7f6794134060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f679a200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67940ced20 0x7f6794134060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f679a200640 1 -- 172.21.15.97:0/891605111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6794158b00 con 0x7f6794160110 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f679a200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679415e1c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f678c007d80 tx=0x7f678c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c014420 con 0x7f6794160110 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678c019070 con 0x7f6794160110 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c014800 con 0x7f6794160110 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679413f890 con 0x7f6794160110 2024-03-06T07:35:32.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.475+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679415d9c0 con 0x7f6794160110 2024-03-06T07:35:32.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.476+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679404f330 con 0x7f6794160110 2024-03-06T07:35:32.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.477+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f678c0149a0 con 0x7f6794160110 2024-03-06T07:35:32.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.478+0000 7f6791600640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f677406b4d0 0x7f677406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.478+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5501+0+0 (secure 0 0 0) 0x7f678c08a8c0 con 0x7f6794160110 2024-03-06T07:35:32.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.479+0000 7f6799800640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f677406b4d0 0x7f677406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.479+0000 7f6799800640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f677406b4d0 0x7f677406d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6788000f60 tx=0x7f6788009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.483+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f678c0598a0 con 0x7f6794160110 2024-03-06T07:35:32.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:32.687+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6794072920 con 0x7f6794160110 2024-03-06T07:35:33.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.247+0000 7f6791600640 1 -- 172.21.15.97:0/891605111 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) v1 ==== 169+0+0 (secure 0 0 0) 0x7f678c0591a0 con 0x7f6794160110 2024-03-06T07:35:33.250 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.250+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f677406b4d0 msgr2=0x7f677406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.250+0000 7f679c200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f677406b4d0 0x7f677406d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6788000f60 tx=0x7f6788009040 comp rx=0 tx=0).stop 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.250+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 msgr2=0x7f679415e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.250+0000 7f679c200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679415e1c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f678c007d80 tx=0x7f678c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 shutdown_connections 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f677406b4d0 0x7f677406d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6794160110 0x7f679415e1c0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6794145980 0x7f67941585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 --2- 172.21.15.97:0/891605111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67940ced20 0x7f6794134060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 >> 172.21.15.97:0/891605111 conn(0x7f67940cfa80 msgr2=0x7f6794140160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.251+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 shutdown_connections 2024-03-06T07:35:33.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.252+0000 7f679c200640 1 -- 172.21.15.97:0/891605111 wait complete. 2024-03-06T07:35:33.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:35:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.471+0000 7f59fa400640 1 Processor -- start 2024-03-06T07:35:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.471+0000 7f59fa400640 1 -- start start 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59fa400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f59f40b3650 0x7f59f40727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59fa400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40719e0 con 0x7f59f40b3df0 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59fa400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f4071cc0 con 0x7f59f40cda90 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59fa400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f4072310 con 0x7f59f40b3650 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59f3400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f59f40b3df0 0x7f59f40b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56856/0 (socket says 172.21.15.97:56856) 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59f3400640 1 -- 172.21.15.97:0/727708165 learned_addr learned my addr 172.21.15.97:0/727708165 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212242141 0 0) 0x7f59f40719e0 con 0x7f59f40b3df0 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d8003520 con 0x7f59f40b3df0 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.473+0000 7f59eaa00640 1 --2- 172.21.15.97:0/727708165 >> v2:172.21.15.135:3300/0 conn(0x7f59f40b3650 0x7f59f40727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2949555138 0 0) 0x7f59d8003520 con 0x7f59f40b3df0 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 >> v2:172.21.15.135:3300/0 conn(0x7f59f40b3650 msgr2=0x7f59f40727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 --2- 172.21.15.97:0/727708165 >> v2:172.21.15.135:3300/0 conn(0x7f59f40b3650 0x7f59f40727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 >> v1:172.21.15.135:6789/0 conn(0x7f59f40cda90 legacy=0x7f59f406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4071fa0 con 0x7f59f40b3df0 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f59e4001670 con 0x7f59f40b3df0 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 >> v1:172.21.15.97:6789/0 conn(0x7f59f40b3df0 legacy=0x7f59f40b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80039b0 0x7f59d8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80051c0 0x7f59d800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4072310 con 0x7f59d8004800 2024-03-06T07:35:33.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800e300 con 0x7f59d80039b0 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800e500 con 0x7f59d80051c0 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59eaa00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.474+0000 7f59f3400640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80039b0 0x7f59d8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59eaa00640 1 -- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80051c0 msgr2=0x7f59d800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59eaa00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80051c0 0x7f59d800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59eaa00640 1 -- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80039b0 msgr2=0x7f59d8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59eaa00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80039b0 0x7f59d8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59eaa00640 1 -- 172.21.15.97:0/727708165 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8003520 con 0x7f59d8004800 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59f3400640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80039b0 0x7f59d8003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:33.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.475+0000 7f59f3e00640 1 -- 172.21.15.97:0/727708165 reap_dead start 2024-03-06T07:35:33.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.476+0000 7f59eaa00640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00ba00 tx=0x7f59d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f86c5f9947f022 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.476+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc00b600 con 0x7f59d8004800 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.476+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc038070 con 0x7f59d8004800 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.476+0000 7f59f2a00640 1 -- 172.21.15.97:0/727708165 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc0338a0 con 0x7f59d8004800 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.476+0000 7f59fa400640 1 -- 172.21.15.97:0/727708165 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 msgr2=0x7f59d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.476+0000 7f59fa400640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00ba00 tx=0x7f59d8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.477+0000 7f59fa400640 1 -- 172.21.15.97:0/727708165 shutdown_connections 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.477+0000 7f59fa400640 1 --2- 172.21.15.97:0/727708165 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.477+0000 7f59fa400640 1 -- 172.21.15.97:0/727708165 >> 172.21.15.97:0/727708165 conn(0x7f59f40b1940 msgr2=0x7f59f40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.477+0000 7f59fa400640 1 -- 172.21.15.97:0/727708165 shutdown_connections 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.477+0000 7f59fa400640 1 -- 172.21.15.97:0/727708165 wait complete. 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.478+0000 7f59fa400640 1 Processor -- start 2024-03-06T07:35:33.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.479+0000 7f59fa400640 1 -- start start 2024-03-06T07:35:33.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.479+0000 7f59fa400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59f4145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.479+0000 7f59fa400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59f4145390 0x7f59f4152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59fa400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59f4152b40 0x7f59f409cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59fa400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f415b3c0 con 0x7f59d8004800 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59fa400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f415b240 con 0x7f59f4145390 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59fa400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40cf510 con 0x7f59f4152b40 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59f4145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59f4145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42082/0 (socket says 172.21.15.97:42082) 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 -- 172.21.15.97:0/1933394332 learned_addr learned my addr 172.21.15.97:0/1933394332 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 -- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59f4152b40 msgr2=0x7f59f409cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59f4152b40 0x7f59f409cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59eaa00640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59f4145390 0x7f59f4152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 -- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59f4145390 msgr2=0x7f59f4152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59f4145390 0x7f59f4152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 -- 172.21.15.97:0/1933394332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4153700 con 0x7f59d8004800 2024-03-06T07:35:33.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.480+0000 7f59f3400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59f4145f10 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f59e400d910 tx=0x7f59e400ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.481+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e4019070 con 0x7f59d8004800 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.481+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e40074e0 con 0x7f59d8004800 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.481+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f41539e0 con 0x7f59d8004800 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.481+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e4007690 con 0x7f59d8004800 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.481+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f4153fa0 con 0x7f59d8004800 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.482+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f404f330 con 0x7f59d8004800 2024-03-06T07:35:33.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.483+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59e401c070 con 0x7f59d8004800 2024-03-06T07:35:33.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.484+0000 7f59ebe00640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.484+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f59e4089aa0 con 0x7f59d8004800 2024-03-06T07:35:33.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.484+0000 7f59eaa00640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.485+0000 7f59eaa00640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59d006b500 0x7f59d006d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f59d80057e0 tx=0x7f59dc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.488+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59e4058bb0 con 0x7f59d8004800 2024-03-06T07:35:33.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.693+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f59f40cc0e0 con 0x7f59d8004800 2024-03-06T07:35:33.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.699+0000 7f59ebe00640 1 -- 172.21.15.97:0/1933394332 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f59e40584b0 con 0x7f59d8004800 2024-03-06T07:35:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.701+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59d006b500 msgr2=0x7f59d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.701+0000 7f59fa400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59d006b500 0x7f59d006d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f59d80057e0 tx=0x7f59dc030000 comp rx=0 tx=0).stop 2024-03-06T07:35:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.701+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 msgr2=0x7f59f4145f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.701+0000 7f59fa400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59f4145f10 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f59e400d910 tx=0x7f59e400ddd0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 shutdown_connections 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59f4152b40 0x7f59f409cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59f4145390 0x7f59f4152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 --2- 172.21.15.97:0/1933394332 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d8004800 0x7f59f4145f10 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 >> 172.21.15.97:0/1933394332 conn(0x7f59f40b1940 msgr2=0x7f59f406d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 shutdown_connections 2024-03-06T07:35:33.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.702+0000 7f59fa400640 1 -- 172.21.15.97:0/1933394332 wait complete. 2024-03-06T07:35:33.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:35:33.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.922+0000 7f46dfc00640 1 Processor -- start 2024-03-06T07:35:33.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.922+0000 7f46dfc00640 1 -- start start 2024-03-06T07:35:33.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46dfc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d80d16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46dfc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f46d806e590 0x7f46d80ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46dfc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46d80ce680 0x7f46d80ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46dfc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d806f990 con 0x7f46d80ce680 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46dfc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d806ed80 con 0x7f46d806e590 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46dfc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d806f060 con 0x7f46d81448a0 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46ddc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46d80ce680 0x7f46d80ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46ddc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f46d80ce680 0x7f46d80ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38088/0 (socket says 172.21.15.97:38088) 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.924+0000 7f46ddc00640 1 -- 172.21.15.97:0/1563220006 learned_addr learned my addr 172.21.15.97:0/1563220006 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:33.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46dd200640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d80d16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46d7e00640 1 --2- 172.21.15.97:0/1563220006 >> v2:172.21.15.135:3300/0 conn(0x7f46d806e590 0x7f46d80ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46ddc00640 1 -- 172.21.15.97:0/1563220006 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 msgr2=0x7f46d80d16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46ddc00640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d80d16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46ddc00640 1 -- 172.21.15.97:0/1563220006 >> v2:172.21.15.135:3300/0 conn(0x7f46d806e590 msgr2=0x7f46d80ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46ddc00640 1 --2- 172.21.15.97:0/1563220006 >> v2:172.21.15.135:3300/0 conn(0x7f46d806e590 0x7f46d80ce140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46ddc00640 1 -- 172.21.15.97:0/1563220006 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d806f340 con 0x7f46d80ce680 2024-03-06T07:35:33.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.925+0000 7f46dd200640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d80d16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.926+0000 7f46ddc00640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d80ceb10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f46d8074b00 tx=0x7f46d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d3cc9048eadd76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.926+0000 7f46d7400640 1 -- 172.21.15.97:0/1563220006 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0035020 con 0x7f46d80ce680 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.926+0000 7f46d7400640 1 -- 172.21.15.97:0/1563220006 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0031870 con 0x7f46d80ce680 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.926+0000 7f46d7400640 1 -- 172.21.15.97:0/1563220006 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0031c40 con 0x7f46d80ce680 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 -- 172.21.15.97:0/1563220006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 msgr2=0x7f46d80ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d80ceb10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f46d8074b00 tx=0x7f46d002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:33.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 -- 172.21.15.97:0/1563220006 shutdown_connections 2024-03-06T07:35:33.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d80ceb10 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1563220006 >> v2:172.21.15.135:3300/0 conn(0x7f46d806e590 0x7f46d80ce140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1563220006 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d80d16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 -- 172.21.15.97:0/1563220006 >> 172.21.15.97:0/1563220006 conn(0x7f46d80d0b40 msgr2=0x7f46d80d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 -- 172.21.15.97:0/1563220006 shutdown_connections 2024-03-06T07:35:33.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.927+0000 7f46dfc00640 1 -- 172.21.15.97:0/1563220006 wait complete. 2024-03-06T07:35:33.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.929+0000 7f46dfc00640 1 Processor -- start 2024-03-06T07:35:33.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.929+0000 7f46dfc00640 1 -- start start 2024-03-06T07:35:33.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46dfc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46d806e590 0x7f46d815d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46dfc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d815b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46dfc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d81f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46dfc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80cf050 con 0x7f46d80ce680 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46dfc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d806ea10 con 0x7f46d806e590 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46dfc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80cf1d0 con 0x7f46d81448a0 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d815b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d815b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38102/0 (socket says 172.21.15.97:38102) 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.930+0000 7f46d7e00640 1 -- 172.21.15.97:0/1635780917 learned_addr learned my addr 172.21.15.97:0/1635780917 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46ddc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d81f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46dd200640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46d806e590 0x7f46d815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d7e00640 1 -- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 msgr2=0x7f46d81f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d7e00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d81f68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d7e00640 1 -- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46d806e590 msgr2=0x7f46d815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d7e00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46d806e590 0x7f46d815d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d7e00640 1 -- 172.21.15.97:0/1635780917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d815bca0 con 0x7f46d80ce680 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d7e00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d815b760 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f46c400d800 tx=0x7f46c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400b6c0 con 0x7f46d80ce680 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4019070 con 0x7f46d80ce680 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d81f6df0 con 0x7f46d80ce680 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400ba80 con 0x7f46d80ce680 2024-03-06T07:35:33.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.931+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d815c500 con 0x7f46d80ce680 2024-03-06T07:35:33.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.933+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d804f330 con 0x7f46d80ce680 2024-03-06T07:35:33.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.933+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46c400bc20 con 0x7f46d80ce680 2024-03-06T07:35:33.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.934+0000 7f46d4c00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46b806b4d0 0x7f46b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.934+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f46c408a380 con 0x7f46d80ce680 2024-03-06T07:35:33.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.935+0000 7f46dd200640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46b806b4d0 0x7f46b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.935+0000 7f46dd200640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46b806b4d0 0x7f46b806d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f46c8008e60 tx=0x7f46c80083b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:33.939+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46c4059530 con 0x7f46d80ce680 2024-03-06T07:35:34.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.207+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f46d806ddd0 con 0x7f46d80ce680 2024-03-06T07:35:34.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.250+0000 7f46d4c00640 1 -- 172.21.15.97:0/1635780917 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v424) v1 ==== 142+0+0 (secure 0 0 0) 0x7f46c4058e30 con 0x7f46d80ce680 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.253+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46b806b4d0 msgr2=0x7f46b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.253+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46b806b4d0 0x7f46b806d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f46c8008e60 tx=0x7f46c80083b0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.253+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 msgr2=0x7f46d815b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.253+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d815b760 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f46c400d800 tx=0x7f46c400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.253+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 shutdown_connections 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f46b806b4d0 0x7f46b806d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46d81448a0 0x7f46d81f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46d80ce680 0x7f46d815b760 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 --2- 172.21.15.97:0/1635780917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f46d806e590 0x7f46d815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 >> 172.21.15.97:0/1635780917 conn(0x7f46d80d0b40 msgr2=0x7f46d80b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 shutdown_connections 2024-03-06T07:35:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.254+0000 7f46dfc00640 1 -- 172.21.15.97:0/1635780917 wait complete. 2024-03-06T07:35:34.274 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:35:34.275 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:35:34.275 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:35:34.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:34.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.479+0000 7f531ec00640 1 Processor -- start 2024-03-06T07:35:34.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.480+0000 7f531ec00640 1 -- start start 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.481+0000 7f531ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f53180ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.481+0000 7f531ec00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f53180aebf0 0x7f53180cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.481+0000 7f531ec00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53180cc800 0x7f53180cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.481+0000 7f531ec00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531815ce30 con 0x7f53180cc800 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.481+0000 7f531ec00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531815c240 con 0x7f53180aebf0 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.481+0000 7f531ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531815c500 con 0x7f53181561f0 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53180cc800 0x7f53180cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53180cc800 0x7f53180cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38114/0 (socket says 172.21.15.97:38114) 2024-03-06T07:35:34.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 -- 172.21.15.97:0/2847969598 learned_addr learned my addr 172.21.15.97:0/2847969598 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531d800640 1 --2- 172.21.15.97:0/2847969598 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f53180ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531ce00640 1 --2- 172.21.15.97:0/2847969598 >> v2:172.21.15.135:3300/0 conn(0x7f53180aebf0 0x7f53180cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 -- 172.21.15.97:0/2847969598 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 msgr2=0x7f53180ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 --2- 172.21.15.97:0/2847969598 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f53180ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 -- 172.21.15.97:0/2847969598 >> v2:172.21.15.135:3300/0 conn(0x7f53180aebf0 msgr2=0x7f53180cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 --2- 172.21.15.97:0/2847969598 >> v2:172.21.15.135:3300/0 conn(0x7f53180aebf0 0x7f53180cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.482+0000 7f531e200640 1 -- 172.21.15.97:0/2847969598 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531815c7e0 con 0x7f53180cc800 2024-03-06T07:35:34.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.483+0000 7f531e200640 1 --2- 172.21.15.97:0/2847969598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53180cf800 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f5318068430 tx=0x7f531002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66db7fb40dcdb7c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.484+0000 7f5317e00640 1 -- 172.21.15.97:0/2847969598 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531002ca70 con 0x7f53180cc800 2024-03-06T07:35:34.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.484+0000 7f5317e00640 1 -- 172.21.15.97:0/2847969598 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531002cc10 con 0x7f53180cc800 2024-03-06T07:35:34.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.484+0000 7f5317e00640 1 -- 172.21.15.97:0/2847969598 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100343f0 con 0x7f53180cc800 2024-03-06T07:35:34.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.484+0000 7f531ec00640 1 -- 172.21.15.97:0/2847969598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 msgr2=0x7f53180cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.484+0000 7f531ec00640 1 --2- 172.21.15.97:0/2847969598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53180cf800 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f5318068430 tx=0x7f531002bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.484+0000 7f531ec00640 1 -- 172.21.15.97:0/2847969598 shutdown_connections 2024-03-06T07:35:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.485+0000 7f531ec00640 1 --2- 172.21.15.97:0/2847969598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53180cf800 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.485+0000 7f531ec00640 1 --2- 172.21.15.97:0/2847969598 >> v2:172.21.15.135:3300/0 conn(0x7f53180aebf0 0x7f53180cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.485+0000 7f531ec00640 1 --2- 172.21.15.97:0/2847969598 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f53180ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.485+0000 7f531ec00640 1 -- 172.21.15.97:0/2847969598 >> 172.21.15.97:0/2847969598 conn(0x7f531806afc0 msgr2=0x7f531806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.485+0000 7f531ec00640 1 -- 172.21.15.97:0/2847969598 shutdown_connections 2024-03-06T07:35:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.485+0000 7f531ec00640 1 -- 172.21.15.97:0/2847969598 wait complete. 2024-03-06T07:35:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.486+0000 7f531ec00640 1 Processor -- start 2024-03-06T07:35:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.486+0000 7f531ec00640 1 -- start start 2024-03-06T07:35:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.487+0000 7f531ec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53180aebf0 0x7f53180b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.487+0000 7f531ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53181f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.487+0000 7f531ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f5318145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.487+0000 7f531ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53181565d0 con 0x7f53180cc800 2024-03-06T07:35:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.487+0000 7f531ec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180ccc90 con 0x7f53180aebf0 2024-03-06T07:35:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.487+0000 7f531ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5318156750 con 0x7f53181561f0 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53181f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53181f7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38118/0 (socket says 172.21.15.97:38118) 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 -- 172.21.15.97:0/2405127460 learned_addr learned my addr 172.21.15.97:0/2405127460 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 -- 172.21.15.97:0/2405127460 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 msgr2=0x7f5318145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f5318145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 -- 172.21.15.97:0/2405127460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53180aebf0 msgr2=0x7f53180b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53180aebf0 0x7f53180b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 -- 172.21.15.97:0/2405127460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181f7c90 con 0x7f53180cc800 2024-03-06T07:35:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.488+0000 7f531ce00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53181f7460 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f530400d830 tx=0x7f530400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.489+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304018030 con 0x7f53180cc800 2024-03-06T07:35:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.489+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530400be10 con 0x7f53180cc800 2024-03-06T07:35:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.489+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040098b0 con 0x7f53180cc800 2024-03-06T07:35:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.489+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53181f7f70 con 0x7f53180cc800 2024-03-06T07:35:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.489+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5318156970 con 0x7f53180cc800 2024-03-06T07:35:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.490+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531804f330 con 0x7f53180cc800 2024-03-06T07:35:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.491+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f530401b070 con 0x7f53180cc800 2024-03-06T07:35:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.492+0000 7f5315600640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f52f806b4d0 0x7f52f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.492+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f530408a210 con 0x7f53180cc800 2024-03-06T07:35:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.492+0000 7f531d800640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f52f806b4d0 0x7f52f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.493+0000 7f531d800640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f52f806b4d0 0x7f52f806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f530c0098f0 tx=0x7f530c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.496+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5304059290 con 0x7f53180cc800 2024-03-06T07:35:34.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.693+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53180b2b10 con 0x7f53180cc800 2024-03-06T07:35:34.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.695+0000 7f5315600640 1 -- 172.21.15.97:0/2405127460 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+13148 (secure 0 0 0) 0x7f5304058b90 con 0x7f53180cc800 2024-03-06T07:35:34.698 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:34.698 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":121,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:35:33.238862+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/3308266804":"2024-03-07T07:35:30.084068+0000","172.21.15.97:6836/3308266804":"2024-03-07T07:35:30.084068+0000","172.21.15.135:6833/322807700":"2024-03-07T07:34:20.730206+0000","172.21.15.135:6832/322807700":"2024-03-07T07:34:20.730206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:34.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f52f806b4d0 msgr2=0x7f52f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f52f806b4d0 0x7f52f806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f530c0098f0 tx=0x7f530c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 msgr2=0x7f53181f7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53181f7460 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f530400d830 tx=0x7f530400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 shutdown_connections 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f52f806b4d0 0x7f52f806d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53181561f0 0x7f5318145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53180cc800 0x7f53181f7460 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 --2- 172.21.15.97:0/2405127460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53180aebf0 0x7f53180b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 >> 172.21.15.97:0/2405127460 conn(0x7f531806afc0 msgr2=0x7f5318071b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 shutdown_connections 2024-03-06T07:35:34.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.698+0000 7f531ec00640 1 -- 172.21.15.97:0/2405127460 wait complete. 2024-03-06T07:35:34.719 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:34.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:34.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.930+0000 7fdf85600640 1 Processor -- start 2024-03-06T07:35:34.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.930+0000 7fdf85600640 1 -- start start 2024-03-06T07:35:34.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.931+0000 7fdf85600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf80070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.931+0000 7fdf85600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf8015fff0 0x7fdf80160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf85600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdf800ce4a0 0x7fdf800ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf85600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf800b46c0 con 0x7fdf800ce4a0 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf85600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf800b49a0 con 0x7fdf8015fff0 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf85600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80072c90 con 0x7fdf8015fc10 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdf800ce4a0 0x7fdf800ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdf800ce4a0 0x7fdf800ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38134/0 (socket says 172.21.15.97:38134) 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 -- 172.21.15.97:0/2744633544 learned_addr learned my addr 172.21.15.97:0/2744633544 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7ea00640 1 --2- 172.21.15.97:0/2744633544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf80070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7e000640 1 --2- 172.21.15.97:0/2744633544 >> v2:172.21.15.135:3300/0 conn(0x7fdf8015fff0 0x7fdf80160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 -- 172.21.15.97:0/2744633544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 msgr2=0x7fdf80070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 --2- 172.21.15.97:0/2744633544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf80070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 -- 172.21.15.97:0/2744633544 >> v2:172.21.15.135:3300/0 conn(0x7fdf8015fff0 msgr2=0x7fdf80160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 --2- 172.21.15.97:0/2744633544 >> v2:172.21.15.135:3300/0 conn(0x7fdf8015fff0 0x7fdf80160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7f400640 1 -- 172.21.15.97:0/2744633544 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80160990 con 0x7fdf800ce4a0 2024-03-06T07:35:34.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.932+0000 7fdf7e000640 1 --2- 172.21.15.97:0/2744633544 >> v2:172.21.15.135:3300/0 conn(0x7fdf8015fff0 0x7fdf80160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.934+0000 7fdf7f400640 1 --2- 172.21.15.97:0/2744633544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf800ce4a0 0x7fdf800ce930 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fdf800749e0 tx=0x7fdf7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06e7f57cb1e3dba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.934+0000 7fdf7d600640 1 -- 172.21.15.97:0/2744633544 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7802ca70 con 0x7fdf800ce4a0 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.934+0000 7fdf7d600640 1 -- 172.21.15.97:0/2744633544 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7802cc10 con 0x7fdf800ce4a0 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.934+0000 7fdf7d600640 1 -- 172.21.15.97:0/2744633544 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf780343f0 con 0x7fdf800ce4a0 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.934+0000 7fdf85600640 1 -- 172.21.15.97:0/2744633544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf800ce4a0 msgr2=0x7fdf800ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.934+0000 7fdf85600640 1 --2- 172.21.15.97:0/2744633544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf800ce4a0 0x7fdf800ce930 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fdf800749e0 tx=0x7fdf7802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 -- 172.21.15.97:0/2744633544 shutdown_connections 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 --2- 172.21.15.97:0/2744633544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf800ce4a0 0x7fdf800ce930 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 --2- 172.21.15.97:0/2744633544 >> v2:172.21.15.135:3300/0 conn(0x7fdf8015fff0 0x7fdf80160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 --2- 172.21.15.97:0/2744633544 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf80070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 -- 172.21.15.97:0/2744633544 >> 172.21.15.97:0/2744633544 conn(0x7fdf800b1940 msgr2=0x7fdf800b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 -- 172.21.15.97:0/2744633544 shutdown_connections 2024-03-06T07:35:34.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.935+0000 7fdf85600640 1 -- 172.21.15.97:0/2744633544 wait complete. 2024-03-06T07:35:34.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.937+0000 7fdf85600640 1 Processor -- start 2024-03-06T07:35:34.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.937+0000 7fdf85600640 1 -- start start 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.938+0000 7fdf85600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800ce4a0 0x7fdf8014cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.938+0000 7fdf85600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf8009d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.938+0000 7fdf85600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 0x7fdf801e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.938+0000 7fdf85600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80072a30 con 0x7fdf8015fff0 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.938+0000 7fdf85600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800728b0 con 0x7fdf800ce4a0 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.938+0000 7fdf85600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8015c2e0 con 0x7fdf8015fc10 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 0x7fdf801e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 0x7fdf801e6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38136/0 (socket says 172.21.15.97:38136) 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 -- 172.21.15.97:0/2049006151 learned_addr learned my addr 172.21.15.97:0/2049006151 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7e000640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf8009d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 -- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 msgr2=0x7fdf8009d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf8009d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 -- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800ce4a0 msgr2=0x7fdf8014cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800ce4a0 0x7fdf8014cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 -- 172.21.15.97:0/2049006151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8009c260 con 0x7fdf8015fff0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.939+0000 7fdf7f400640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 0x7fdf801e6b60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fdf78036040 tx=0x7fdf78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.940+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78043020 con 0x7fdf8015fff0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.940+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7803e450 con 0x7fdf8015fff0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.940+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8009c540 con 0x7fdf8015fff0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.940+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78009510 con 0x7fdf8015fff0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.940+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8009cb00 con 0x7fdf8015fff0 2024-03-06T07:35:34.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.941+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8004f330 con 0x7fdf8015fff0 2024-03-06T07:35:34.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.942+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf78046020 con 0x7fdf8015fff0 2024-03-06T07:35:34.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.943+0000 7fdf76a00640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf6006b500 0x7fdf6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.943+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdf780ac590 con 0x7fdf8015fff0 2024-03-06T07:35:34.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.943+0000 7fdf7ea00640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf6006b500 0x7fdf6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.944+0000 7fdf7ea00640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf6006b500 0x7fdf6006d9c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009870 tx=0x7fdf70008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:34.947+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf7804d070 con 0x7fdf8015fff0 2024-03-06T07:35:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.142+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf8006ef30 con 0x7fdf8015fff0 2024-03-06T07:35:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.142+0000 7fdf76a00640 1 -- 172.21.15.97:0/2049006151 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf7807b610 con 0x7fdf8015fff0 2024-03-06T07:35:35.145 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:35:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf6006b500 msgr2=0x7fdf6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf6006b500 0x7fdf6006d9c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009870 tx=0x7fdf70008040 comp rx=0 tx=0).stop 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 msgr2=0x7fdf801e6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 0x7fdf801e6b60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fdf78036040 tx=0x7fdf78004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 shutdown_connections 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf6006b500 0x7fdf6006d9c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf8015fff0 0x7fdf801e6b60 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf8015fc10 0x7fdf8009d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 --2- 172.21.15.97:0/2049006151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800ce4a0 0x7fdf8014cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 >> 172.21.15.97:0/2049006151 conn(0x7fdf800b1940 msgr2=0x7fdf800cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 shutdown_connections 2024-03-06T07:35:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.145+0000 7fdf85600640 1 -- 172.21.15.97:0/2049006151 wait complete. 2024-03-06T07:35:35.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:35:35.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.371+0000 7f961f600640 1 Processor -- start 2024-03-06T07:35:35.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.371+0000 7f961f600640 1 -- start start 2024-03-06T07:35:35.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.373+0000 7f961f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 0x7f961806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.373+0000 7f961f600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f961815fe30 0x7f96180ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.373+0000 7f961f600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961806ee70 con 0x7f961815f9d0 2024-03-06T07:35:35.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.373+0000 7f961f600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961806cef0 con 0x7f961815fe30 2024-03-06T07:35:35.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.373+0000 7f961f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961806d1d0 con 0x7f9618144c00 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.374+0000 7f961d600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f961815fe30 0x7f96180ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.374+0000 7f961cc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 0x7f961806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.374+0000 7f961cc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 0x7f961806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:39798/0 (socket says 172.21.15.97:39798) 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.374+0000 7f961cc00640 1 -- 172.21.15.97:0/49290453 learned_addr learned my addr 172.21.15.97:0/49290453 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.375+0000 7f961cc00640 1 -- 172.21.15.97:0/49290453 >> v2:172.21.15.135:3300/0 conn(0x7f961815fe30 msgr2=0x7f96180ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.375+0000 7f961cc00640 1 --2- 172.21.15.97:0/49290453 >> v2:172.21.15.135:3300/0 conn(0x7f961815fe30 0x7f96180ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.375+0000 7f961cc00640 1 -- 172.21.15.97:0/49290453 >> v1:172.21.15.135:6789/0 conn(0x7f961815f9d0 legacy=0x7f96180ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.375+0000 7f961cc00640 1 -- 172.21.15.97:0/49290453 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961806d4b0 con 0x7f9618144c00 2024-03-06T07:35:35.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.376+0000 7f961cc00640 1 --2- 172.21.15.97:0/49290453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 0x7f961806e930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9608009e50 tx=0x7f960802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81360b78018008c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f9617400640 1 -- 172.21.15.97:0/49290453 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960802cb00 con 0x7f9618144c00 2024-03-06T07:35:35.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f9617400640 1 -- 172.21.15.97:0/49290453 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960802cca0 con 0x7f9618144c00 2024-03-06T07:35:35.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f9617400640 1 -- 172.21.15.97:0/49290453 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608034610 con 0x7f9618144c00 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 -- 172.21.15.97:0/49290453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 msgr2=0x7f961806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 --2- 172.21.15.97:0/49290453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 0x7f961806e930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9608009e50 tx=0x7f960802bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 -- 172.21.15.97:0/49290453 shutdown_connections 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 --2- 172.21.15.97:0/49290453 >> v2:172.21.15.135:3300/0 conn(0x7f961815fe30 0x7f96180ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 --2- 172.21.15.97:0/49290453 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9618144c00 0x7f961806e930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 -- 172.21.15.97:0/49290453 >> 172.21.15.97:0/49290453 conn(0x7f96180b1730 msgr2=0x7f96180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 -- 172.21.15.97:0/49290453 shutdown_connections 2024-03-06T07:35:35.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.377+0000 7f961f600640 1 -- 172.21.15.97:0/49290453 wait complete. 2024-03-06T07:35:35.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.379+0000 7f961f600640 1 Processor -- start 2024-03-06T07:35:35.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.379+0000 7f961f600640 1 -- start start 2024-03-06T07:35:35.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.380+0000 7f961f600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9618144c00 0x7f96181e6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.380+0000 7f961f600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f961815f9d0 0x7f96180ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.380+0000 7f961f600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 0x7f96180ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.380+0000 7f961f600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9618160730 con 0x7f961815fe30 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.380+0000 7f961f600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9618160280 con 0x7f9618144c00 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.380+0000 7f961f600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961806cb50 con 0x7f961815f9d0 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 0x7f96180ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 0x7f96180ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38148/0 (socket says 172.21.15.97:38148) 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 -- 172.21.15.97:0/4004195477 learned_addr learned my addr 172.21.15.97:0/4004195477 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961cc00640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9618144c00 0x7f96181e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f9617e00640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f961815f9d0 0x7f96180ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 -- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f961815f9d0 msgr2=0x7f96180ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f961815f9d0 0x7f96180ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 -- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9618144c00 msgr2=0x7f96181e6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9618144c00 0x7f96181e6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 -- 172.21.15.97:0/4004195477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96181e70d0 con 0x7f961815fe30 2024-03-06T07:35:35.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961cc00640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9618144c00 0x7f96181e6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.381+0000 7f961d600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 0x7f96180ce5a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f961000e7f0 tx=0x7f961000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.382+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610017070 con 0x7f961815fe30 2024-03-06T07:35:35.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.382+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961000ee40 con 0x7f961815fe30 2024-03-06T07:35:35.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.382+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961000cb80 con 0x7f961815fe30 2024-03-06T07:35:35.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.382+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96181e73b0 con 0x7f961815fe30 2024-03-06T07:35:35.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.382+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96181e7970 con 0x7f961815fe30 2024-03-06T07:35:35.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.383+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96181f8380 con 0x7f961815fe30 2024-03-06T07:35:35.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.384+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f961001a070 con 0x7f961815fe30 2024-03-06T07:35:35.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.385+0000 7f9614c00640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.385+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f961008a180 con 0x7f961815fe30 2024-03-06T07:35:35.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.385+0000 7f961cc00640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.386+0000 7f961cc00640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f95f406b4d0 0x7f95f406d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9608035000 tx=0x7f9608030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.390+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9610059290 con 0x7f961815fe30 2024-03-06T07:35:35.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:35.657+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f961804f330 con 0x7f961815fe30 2024-03-06T07:35:36.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.262+0000 7f9614c00640 1 -- 172.21.15.97:0/4004195477 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v426) v1 ==== 143+0+0 (secure 0 0 0) 0x7f9610058b90 con 0x7f961815fe30 2024-03-06T07:35:36.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f95f406b4d0 msgr2=0x7f95f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f95f406b4d0 0x7f95f406d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9608035000 tx=0x7f9608030000 comp rx=0 tx=0).stop 2024-03-06T07:35:36.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 msgr2=0x7f96180ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 0x7f96180ce5a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f961000e7f0 tx=0x7f961000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 shutdown_connections 2024-03-06T07:35:36.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961815fe30 0x7f96180ce5a0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f961815f9d0 0x7f96180ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 --2- 172.21.15.97:0/4004195477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9618144c00 0x7f96181e6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 >> 172.21.15.97:0/4004195477 conn(0x7f96180b1730 msgr2=0x7f96180b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 shutdown_connections 2024-03-06T07:35:36.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.267+0000 7f961f600640 1 -- 172.21.15.97:0/4004195477 wait complete. 2024-03-06T07:35:36.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:35:36.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.500+0000 7f9e85200640 1 Processor -- start 2024-03-06T07:35:36.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.501+0000 7f9e85200640 1 -- start start 2024-03-06T07:35:36.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.502+0000 7f9e85200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 0x7f9e80070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.502+0000 7f9e85200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e800ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.502+0000 7f9e85200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e800b46c0 con 0x7f9e8015fff0 2024-03-06T07:35:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.502+0000 7f9e85200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e800b49a0 con 0x7f9e8015fc10 2024-03-06T07:35:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.502+0000 7f9e85200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e80072c90 con 0x7f9e800b3f70 2024-03-06T07:35:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.503+0000 7f9e7de00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 0x7f9e80070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.503+0000 7f9e7de00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 0x7f9e80070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:35892/0 (socket says 172.21.15.97:35892) 2024-03-06T07:35:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.503+0000 7f9e7de00640 1 -- 172.21.15.97:0/946094033 learned_addr learned my addr 172.21.15.97:0/946094033 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.503+0000 7f9e7e800640 1 --2- 172.21.15.97:0/946094033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e800ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.504+0000 7f9e7e800640 1 -- 172.21.15.97:0/946094033 >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 msgr2=0x7f9e80070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.504+0000 7f9e7e800640 1 --2- 172.21.15.97:0/946094033 >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 0x7f9e80070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.504+0000 7f9e7e800640 1 -- 172.21.15.97:0/946094033 >> v1:172.21.15.135:6789/0 conn(0x7f9e8015fff0 legacy=0x7f9e80160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.504+0000 7f9e7e800640 1 -- 172.21.15.97:0/946094033 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8009dd80 con 0x7f9e800b3f70 2024-03-06T07:35:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.504+0000 7f9e7de00640 1 --2- 172.21.15.97:0/946094033 >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 0x7f9e80070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:36.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.505+0000 7f9e7e800640 1 --2- 172.21.15.97:0/946094033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e800ce4a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9e780097a0 tx=0x7f9e7802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=611363977b92ba77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.506+0000 7f9e7ca00640 1 -- 172.21.15.97:0/946094033 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78035020 con 0x7f9e800b3f70 2024-03-06T07:35:36.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.506+0000 7f9e7ca00640 1 -- 172.21.15.97:0/946094033 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e78031810 con 0x7f9e800b3f70 2024-03-06T07:35:36.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.506+0000 7f9e7ca00640 1 -- 172.21.15.97:0/946094033 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78031be0 con 0x7f9e800b3f70 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.506+0000 7f9e85200640 1 -- 172.21.15.97:0/946094033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 msgr2=0x7f9e800ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.506+0000 7f9e85200640 1 --2- 172.21.15.97:0/946094033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e800ce4a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9e780097a0 tx=0x7f9e7802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.507+0000 7f9e85200640 1 -- 172.21.15.97:0/946094033 shutdown_connections 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.507+0000 7f9e85200640 1 --2- 172.21.15.97:0/946094033 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e800ce4a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.507+0000 7f9e85200640 1 --2- 172.21.15.97:0/946094033 >> v2:172.21.15.135:3300/0 conn(0x7f9e8015fc10 0x7f9e80070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.507+0000 7f9e85200640 1 -- 172.21.15.97:0/946094033 >> 172.21.15.97:0/946094033 conn(0x7f9e800b1940 msgr2=0x7f9e800b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.507+0000 7f9e85200640 1 -- 172.21.15.97:0/946094033 shutdown_connections 2024-03-06T07:35:36.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.507+0000 7f9e85200640 1 -- 172.21.15.97:0/946094033 wait complete. 2024-03-06T07:35:36.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.508+0000 7f9e85200640 1 Processor -- start 2024-03-06T07:35:36.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.509+0000 7f9e85200640 1 -- start start 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.509+0000 7f9e85200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e801525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.509+0000 7f9e85200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 0x7f9e8006d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e85200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e8015fff0 0x7f9e800b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e85200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8015c460 con 0x7f9e8015fc10 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e85200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8015c2e0 con 0x7f9e8015fff0 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e85200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800ce9e0 con 0x7f9e800b3f70 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 0x7f9e8006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 0x7f9e8006d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38154/0 (socket says 172.21.15.97:38154) 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 -- 172.21.15.97:0/4221685035 learned_addr learned my addr 172.21.15.97:0/4221685035 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7e800640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e8015fff0 0x7f9e800b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7de00640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e801525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 -- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 msgr2=0x7f9e801525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e801525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 -- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e8015fff0 msgr2=0x7f9e800b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e8015fff0 0x7f9e800b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.510+0000 7f9e7d400640 1 -- 172.21.15.97:0/4221685035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8006e210 con 0x7f9e8015fc10 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.511+0000 7f9e7d400640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 0x7f9e8006d910 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000c730 tx=0x7f9e7000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.511+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e70016020 con 0x7f9e8015fc10 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.511+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e70006d50 con 0x7f9e8015fc10 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.511+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e70003bb0 con 0x7f9e8015fc10 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.511+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8006e4f0 con 0x7f9e8015fc10 2024-03-06T07:35:36.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.511+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8009ac40 con 0x7f9e8015fc10 2024-03-06T07:35:36.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.512+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8004f330 con 0x7f9e8015fc10 2024-03-06T07:35:36.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.513+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e7000a850 con 0x7f9e8015fc10 2024-03-06T07:35:36.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.514+0000 7f9e76000640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e5c06b4d0 0x7f9e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.514+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9e70089ba0 con 0x7f9e8015fc10 2024-03-06T07:35:36.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.514+0000 7f9e7de00640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e5c06b4d0 0x7f9e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.515+0000 7f9e7de00640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e5c06b4d0 0x7f9e5c06d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009b70 tx=0x7f9e68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.518+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e70020020 con 0x7f9e8015fc10 2024-03-06T07:35:36.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.713+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9e8006ef30 con 0x7f9e8015fc10 2024-03-06T07:35:36.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.721+0000 7f9e76000640 1 -- 172.21.15.97:0/4221685035 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9e70019310 con 0x7f9e8015fc10 2024-03-06T07:35:36.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.723+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e5c06b4d0 msgr2=0x7f9e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.723+0000 7f9e85200640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e5c06b4d0 0x7f9e5c06d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009b70 tx=0x7f9e68008040 comp rx=0 tx=0).stop 2024-03-06T07:35:36.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.723+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 msgr2=0x7f9e8006d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.723+0000 7f9e85200640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 0x7f9e8006d910 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000c730 tx=0x7f9e7000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 shutdown_connections 2024-03-06T07:35:36.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e5c06b4d0 0x7f9e5c06d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e8015fff0 0x7f9e800b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e8015fc10 0x7f9e8006d910 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 --2- 172.21.15.97:0/4221685035 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e800b3f70 0x7f9e801525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 >> 172.21.15.97:0/4221685035 conn(0x7f9e800b1940 msgr2=0x7f9e80158730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 shutdown_connections 2024-03-06T07:35:36.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:36.724+0000 7f9e85200640 1 -- 172.21.15.97:0/4221685035 wait complete. 2024-03-06T07:35:36.745 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:36.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:36.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:36.967 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:36.967 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:35:36.968 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:37.023 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:37.023 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:35:37.023 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:37.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:37.081 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:35:37.081 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:37.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:37.136 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:35:37.136 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:37.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:37.192 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:35:37.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:37.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.411+0000 7f738c200640 1 Processor -- start 2024-03-06T07:35:37.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.411+0000 7f738c200640 1 -- start start 2024-03-06T07:35:37.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.412+0000 7f738c200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f738415f9d0 0x7f738415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.412+0000 7f738c200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7384070b10 0x7f7384070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.412+0000 7f738c200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738406cef0 con 0x7f7384144c00 2024-03-06T07:35:37.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.412+0000 7f738c200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738406d1d0 con 0x7f738415f9d0 2024-03-06T07:35:37.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.412+0000 7f738c200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738406d4b0 con 0x7f7384070b10 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7388e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f738415f9d0 0x7f738415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7389800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7384144c00 0x7f738406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55518/0 (socket says 172.21.15.97:55518) 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7389800640 1 -- 172.21.15.97:0/1005006380 learned_addr learned my addr 172.21.15.97:0/1005006380 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7388e00640 1 -- 172.21.15.97:0/1005006380 >> v2:172.21.15.135:3300/0 conn(0x7f7384070b10 msgr2=0x7f7384070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f738a200640 1 --2- 172.21.15.97:0/1005006380 >> v2:172.21.15.135:3300/0 conn(0x7f7384070b10 0x7f7384070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7388e00640 1 --2- 172.21.15.97:0/1005006380 >> v2:172.21.15.135:3300/0 conn(0x7f7384070b10 0x7f7384070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7388e00640 1 -- 172.21.15.97:0/1005006380 >> v1:172.21.15.97:6789/0 conn(0x7f7384144c00 legacy=0x7f738406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.413+0000 7f7383e00640 1 -- 172.21.15.97:0/1005006380 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658510279 0 0) 0x7f738406cef0 con 0x7f7384144c00 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.414+0000 7f7388e00640 1 -- 172.21.15.97:0/1005006380 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738406d790 con 0x7f738415f9d0 2024-03-06T07:35:37.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.414+0000 7f738a200640 1 --2- 172.21.15.97:0/1005006380 >> v2:172.21.15.135:3300/0 conn(0x7f7384070b10 0x7f7384070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.415+0000 7f7388e00640 1 --2- 172.21.15.97:0/1005006380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f738415f9d0 0x7f738415fe60 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f738406e500 tx=0x7f737002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d7f291b18ee764 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.415+0000 7f7383e00640 1 -- 172.21.15.97:0/1005006380 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370035020 con 0x7f738415f9d0 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.415+0000 7f7383e00640 1 -- 172.21.15.97:0/1005006380 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7370031830 con 0x7f738415f9d0 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.415+0000 7f7383e00640 1 -- 172.21.15.97:0/1005006380 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370031c00 con 0x7f738415f9d0 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.415+0000 7f738c200640 1 -- 172.21.15.97:0/1005006380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f738415f9d0 msgr2=0x7f738415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.415+0000 7f738c200640 1 --2- 172.21.15.97:0/1005006380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f738415f9d0 0x7f738415fe60 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f738406e500 tx=0x7f737002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.416+0000 7f738c200640 1 -- 172.21.15.97:0/1005006380 shutdown_connections 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.416+0000 7f738c200640 1 --2- 172.21.15.97:0/1005006380 >> v2:172.21.15.135:3300/0 conn(0x7f7384070b10 0x7f7384070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.416+0000 7f738c200640 1 --2- 172.21.15.97:0/1005006380 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f738415f9d0 0x7f738415fe60 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.416+0000 7f738c200640 1 -- 172.21.15.97:0/1005006380 >> 172.21.15.97:0/1005006380 conn(0x7f73840b1730 msgr2=0x7f73840b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.416+0000 7f738c200640 1 -- 172.21.15.97:0/1005006380 shutdown_connections 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.416+0000 7f738c200640 1 -- 172.21.15.97:0/1005006380 wait complete. 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.417+0000 7f738c200640 1 Processor -- start 2024-03-06T07:35:37.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.417+0000 7f738c200640 1 -- start start 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.418+0000 7f738c200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 0x7f738415d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.418+0000 7f738c200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7384144c00 0x7f738413d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.418+0000 7f738c200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f738415f9d0 0x7f73840b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.418+0000 7f738c200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738406ccd0 con 0x7f7384070b10 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.418+0000 7f738c200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738406cb50 con 0x7f738415f9d0 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.418+0000 7f738c200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738406df90 con 0x7f7384144c00 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 0x7f738415d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 0x7f738415d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38178/0 (socket says 172.21.15.97:38178) 2024-03-06T07:35:37.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 -- 172.21.15.97:0/598561136 learned_addr learned my addr 172.21.15.97:0/598561136 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 -- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7384144c00 msgr2=0x7f738413d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7384144c00 0x7f738413d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f738a200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f738415f9d0 0x7f73840b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 -- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f738415f9d0 msgr2=0x7f73840b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f738415f9d0 0x7f73840b3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.419+0000 7f7389800640 1 -- 172.21.15.97:0/598561136 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738413d9a0 con 0x7f7384070b10 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.420+0000 7f7389800640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 0x7f738415d960 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f737800c4d0 tx=0x7f737800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.420+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7378018070 con 0x7f7384070b10 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.420+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73780074e0 con 0x7f7384070b10 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.420+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7378004a20 con 0x7f7384070b10 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.420+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738413dc80 con 0x7f7384070b10 2024-03-06T07:35:37.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.420+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738413e240 con 0x7f7384070b10 2024-03-06T07:35:37.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.421+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738404f330 con 0x7f7384070b10 2024-03-06T07:35:37.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.422+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f737801b070 con 0x7f7384070b10 2024-03-06T07:35:37.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.423+0000 7f7381600640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f736406b4d0 0x7f736406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.423+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f73780892f0 con 0x7f7384070b10 2024-03-06T07:35:37.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.423+0000 7f7388e00640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f736406b4d0 0x7f736406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.424+0000 7f7388e00640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f736406b4d0 0x7f736406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7370036000 tx=0x7f7370030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.428+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7378058400 con 0x7f7384070b10 2024-03-06T07:35:37.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.663+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7384072890 con 0x7f7384070b10 2024-03-06T07:35:37.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.664+0000 7f7381600640 1 -- 172.21.15.97:0/598561136 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7378057d00 con 0x7f7384070b10 2024-03-06T07:35:37.666 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:37.666 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/2029862790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":2029862790},{"type":"v1","addr":"172.21.15.135:6834","nonce":2029862790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/4058127513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":4058127513},{"type":"v1","addr":"172.21.15.135:6835","nonce":4058127513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2990314129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2990314129},{"type":"v1","addr":"172.21.15.97:6835","nonce":2990314129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8310,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3975723331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3975723331},{"type":"v1","addr":"172.21.15.97:6837","nonce":3975723331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[]} 2024-03-06T07:35:37.666 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 82 2024-03-06T07:35:37.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f736406b4d0 msgr2=0x7f736406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f736406b4d0 0x7f736406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7370036000 tx=0x7f7370030000 comp rx=0 tx=0).stop 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 msgr2=0x7f738415d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 0x7f738415d960 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f737800c4d0 tx=0x7f737800c990 comp rx=0 tx=0).stop 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 shutdown_connections 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f736406b4d0 0x7f736406d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f738415f9d0 0x7f73840b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7384144c00 0x7f738413d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 --2- 172.21.15.97:0/598561136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7384070b10 0x7f738415d960 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 >> 172.21.15.97:0/598561136 conn(0x7f73840b1730 msgr2=0x7f738415fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.666+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 shutdown_connections 2024-03-06T07:35:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.667+0000 7f738c200640 1 -- 172.21.15.97:0/598561136 wait complete. 2024-03-06T07:35:37.689 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:35:37.689 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:35:37.689 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:35:37.690 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:37.690 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:35:37.690 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:35:37.690 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:35:37.690 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:37.690 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:35:37.690 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:35:37.691 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:35:37.691 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:37.691 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:35:37.691 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:35:37.691 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:35:37.691 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:37.707 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:35:37.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:35:37.708 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:35:37.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:35:37.716 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:35:37.716 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:35:37.718 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:35:37.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:35:37.756 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:35:37.756 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:37.762 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:35:37.762 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:37.765 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:35:37.765 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:37.771 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:35:37.772 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:37.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:35:37.913 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:35:37.913 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:35:37.923 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:35:37.927 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:35:37.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.956+0000 7f9690400640 1 Processor -- start 2024-03-06T07:35:37.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.956+0000 7f9690400640 1 -- start start 2024-03-06T07:35:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f9690400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f968815ac90 0x7f968806e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f9690400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815b070 0x7f968815ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f9690400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96880cc000 con 0x7f968815ef50 2024-03-06T07:35:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f9690400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96880ce360 con 0x7f968815ac90 2024-03-06T07:35:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f9690400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96880ce640 con 0x7f968815b070 2024-03-06T07:35:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f968da00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f968815ac90 0x7f968806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f968da00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f968815ac90 0x7f968806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38220/0 (socket says 172.21.15.97:38220) 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f968e400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f968815ef50 0x7f96880cac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55534/0 (socket says 172.21.15.97:55534) 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.957+0000 7f968da00640 1 -- 172.21.15.97:0/115064707 learned_addr learned my addr 172.21.15.97:0/115064707 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.958+0000 7f968d000640 1 --2- 172.21.15.97:0/115064707 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815b070 0x7f968815ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.958+0000 7f968da00640 1 -- 172.21.15.97:0/115064707 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815b070 msgr2=0x7f968815ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.958+0000 7f968da00640 1 --2- 172.21.15.97:0/115064707 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815b070 0x7f968815ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.958+0000 7f968da00640 1 -- 172.21.15.97:0/115064707 >> v1:172.21.15.97:6789/0 conn(0x7f968815ef50 legacy=0x7f96880cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.958+0000 7f968da00640 1 -- 172.21.15.97:0/115064707 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96880ce920 con 0x7f968815ac90 2024-03-06T07:35:37.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.959+0000 7f968da00640 1 --2- 172.21.15.97:0/115064707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815ac90 0x7f968806e580 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f967c009870 tx=0x7f967c02b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b443acb63cf4165a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.959+0000 7f9687e00640 1 -- 172.21.15.97:0/115064707 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c037070 con 0x7f968815ac90 2024-03-06T07:35:37.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.959+0000 7f9687e00640 1 -- 172.21.15.97:0/115064707 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c030710 con 0x7f968815ac90 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.959+0000 7f9687e00640 1 -- 172.21.15.97:0/115064707 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c030ae0 con 0x7f968815ac90 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 -- 172.21.15.97:0/115064707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815ac90 msgr2=0x7f968806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 --2- 172.21.15.97:0/115064707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815ac90 0x7f968806e580 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f967c009870 tx=0x7f967c02b740 comp rx=0 tx=0).stop 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 -- 172.21.15.97:0/115064707 shutdown_connections 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 --2- 172.21.15.97:0/115064707 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815b070 0x7f968815ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 --2- 172.21.15.97:0/115064707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815ac90 0x7f968806e580 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 -- 172.21.15.97:0/115064707 >> 172.21.15.97:0/115064707 conn(0x7f96880cc4f0 msgr2=0x7f96880cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 -- 172.21.15.97:0/115064707 shutdown_connections 2024-03-06T07:35:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.960+0000 7f9690400640 1 -- 172.21.15.97:0/115064707 wait complete. 2024-03-06T07:35:37.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.962+0000 7f9690400640 1 Processor -- start 2024-03-06T07:35:37.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.962+0000 7f9690400640 1 -- start start 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f9690400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815ac90 0x7f9688159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f9690400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 0x7f96881453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f9690400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f968815ef50 0x7f96881458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f9690400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880c5a20 con 0x7f968815b070 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f9690400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880ceee0 con 0x7f968815ef50 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f9690400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96880c5ba0 con 0x7f968815ac90 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 0x7f96881453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 0x7f96881453a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38236/0 (socket says 172.21.15.97:38236) 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968d000640 1 -- 172.21.15.97:0/489595101 learned_addr learned my addr 172.21.15.97:0/489595101 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968da00640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815ac90 0x7f9688159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968e400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f968815ef50 0x7f96881458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968d000640 1 -- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815ac90 msgr2=0x7f9688159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.963+0000 7f968d000640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815ac90 0x7f9688159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f968d000640 1 -- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f968815ef50 msgr2=0x7f96881458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f968d000640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f968815ef50 0x7f96881458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f968d000640 1 -- 172.21.15.97:0/489595101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9688159ce0 con 0x7f968815b070 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f968e400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f968815ef50 0x7f96881458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f968d000640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 0x7f96881453a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f967400d830 tx=0x7f967400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400b6c0 con 0x7f968815b070 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9674019070 con 0x7f968815b070 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400ba80 con 0x7f968815b070 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96880b3de0 con 0x7f968815b070 2024-03-06T07:35:37.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.964+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96880ae770 con 0x7f968815b070 2024-03-06T07:35:37.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.965+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968804f330 con 0x7f968815b070 2024-03-06T07:35:37.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.966+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f967401e460 con 0x7f968815b070 2024-03-06T07:35:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.968+0000 7f9685600640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f966806b4d0 0x7f966806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.968+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f967408a620 con 0x7f968815b070 2024-03-06T07:35:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.968+0000 7f968da00640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f966806b4d0 0x7f966806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.969+0000 7f968da00640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f966806b4d0 0x7f966806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f967c034000 tx=0x7f967c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:37.972+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9688159ce0 con 0x7f968815b070 2024-03-06T07:35:38.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:38.180+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f96880cd1c0 con 0x7f968815b070 2024-03-06T07:35:38.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:38.894 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:39.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.307+0000 7f9685600640 1 -- 172.21.15.97:0/489595101 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v126) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9674059730 con 0x7f968815b070 2024-03-06T07:35:39.310 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:35:39.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f966806b4d0 msgr2=0x7f966806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f966806b4d0 0x7f966806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f967c034000 tx=0x7f967c008030 comp rx=0 tx=0).stop 2024-03-06T07:35:39.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 msgr2=0x7f96881453a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 0x7f96881453a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f967400d830 tx=0x7f967400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 shutdown_connections 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f966806b4d0 0x7f966806d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f968815ef50 0x7f96881458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f968815b070 0x7f96881453a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 --2- 172.21.15.97:0/489595101 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f968815ac90 0x7f9688159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.311+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 >> 172.21.15.97:0/489595101 conn(0x7f96880cc4f0 msgr2=0x7f96880cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.312+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 shutdown_connections 2024-03-06T07:35:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.312+0000 7f9690400640 1 -- 172.21.15.97:0/489595101 wait complete. 2024-03-06T07:35:39.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:35:39.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.536+0000 7f883d600640 1 Processor -- start 2024-03-06T07:35:39.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.536+0000 7f883d600640 1 -- start start 2024-03-06T07:35:39.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.537+0000 7f883d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 0x7f8838070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.538+0000 7f883d600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8838071d30 con 0x7f88380af1c0 2024-03-06T07:35:39.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.538+0000 7f883d600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883815f470 con 0x7f883806e4d0 2024-03-06T07:35:39.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.538+0000 7f883d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8838071260 con 0x7f883815eff0 2024-03-06T07:35:39.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.538+0000 7f8836a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f88380af1c0 0x7f883806cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55540/0 (socket says 172.21.15.97:55540) 2024-03-06T07:35:39.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.538+0000 7f8836a00640 1 -- 172.21.15.97:0/1823102270 learned_addr learned my addr 172.21.15.97:0/1823102270 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.538+0000 7f8836000640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 0x7f8838070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941836069 0 0) 0x7f8828001110 con 0x7f88380af1c0 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f881c003520 con 0x7f88380af1c0 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995064648 0 0) 0x7f881c003520 con 0x7f88380af1c0 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 msgr2=0x7f8838070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 0x7f8838070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 >> v1:172.21.15.135:6789/0 conn(0x7f883806e4d0 legacy=0x7f88380b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838071490 con 0x7f88380af1c0 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8828001840 con 0x7f88380af1c0 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 >> v1:172.21.15.97:6789/0 conn(0x7f88380af1c0 legacy=0x7f883806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 0x7f881c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f881c003f60 0x7f881c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f881c004d40 0x7f881c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c00ba70 con 0x7f881c004d40 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c00bc20 con 0x7f881c003f60 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881c00be20 con 0x7f883815eff0 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.539+0000 7f8837400640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f881c004d40 0x7f881c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8836000640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f881c003f60 0x7f881c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8836a00640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 0x7f881c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8837400640 1 -- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 msgr2=0x7f881c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8837400640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f883815eff0 0x7f881c0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8837400640 1 -- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f881c003f60 msgr2=0x7f881c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8837400640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f881c003f60 0x7f881c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.540+0000 7f8837400640 1 -- 172.21.15.97:0/1823102270 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c003520 con 0x7f881c004d40 2024-03-06T07:35:39.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.541+0000 7f8837400640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f881c004d40 0x7f881c0051c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8830001e50 tx=0x7f881c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196070bbaf92f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.541+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830001a70 con 0x7f881c004d40 2024-03-06T07:35:39.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.541+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8830003b70 con 0x7f881c004d40 2024-03-06T07:35:39.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.541+0000 7f8835600640 1 -- 172.21.15.97:0/1823102270 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88300050d0 con 0x7f881c004d40 2024-03-06T07:35:39.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f883d600640 1 -- 172.21.15.97:0/1823102270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f881c004d40 msgr2=0x7f881c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f883d600640 1 --2- 172.21.15.97:0/1823102270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f881c004d40 0x7f881c0051c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8830001e50 tx=0x7f881c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f8837400640 1 -- 172.21.15.97:0/1823102270 reap_dead start 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f883d600640 1 -- 172.21.15.97:0/1823102270 shutdown_connections 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f883d600640 1 -- 172.21.15.97:0/1823102270 >> 172.21.15.97:0/1823102270 conn(0x7f88380b1730 msgr2=0x7f88380b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f883d600640 1 -- 172.21.15.97:0/1823102270 shutdown_connections 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.542+0000 7f883d600640 1 -- 172.21.15.97:0/1823102270 wait complete. 2024-03-06T07:35:39.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.544+0000 7f883d600640 1 Processor -- start 2024-03-06T07:35:39.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.544+0000 7f883d600640 1 -- start start 2024-03-06T07:35:39.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f883d600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f881c004d40 0x7f883814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f883d600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88380cd190 0x7f88381e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f883d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 0x7f8838159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f883d600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380b30b0 con 0x7f88381e77e0 2024-03-06T07:35:39.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f883d600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8838071a50 con 0x7f88380cd190 2024-03-06T07:35:39.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f883d600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883806d370 con 0x7f881c004d40 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f8837400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 0x7f8838159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f8837400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 0x7f8838159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38260/0 (socket says 172.21.15.97:38260) 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f8837400640 1 -- 172.21.15.97:0/1808654020 learned_addr learned my addr 172.21.15.97:0/1808654020 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f8836a00640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f881c004d40 0x7f883814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f8836000640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88380cd190 0x7f88381e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.545+0000 7f8837400640 1 -- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f881c004d40 msgr2=0x7f883814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f8837400640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f881c004d40 0x7f883814b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f8837400640 1 -- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88380cd190 msgr2=0x7f88381e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f8837400640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88380cd190 0x7f88381e72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f8837400640 1 -- 172.21.15.97:0/1808654020 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88381e7c80 con 0x7f88381e77e0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f8836a00640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f881c004d40 0x7f883814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f8837400640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 0x7f8838159f60 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8830001f70 tx=0x7f88300014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830007db0 con 0x7f88381e77e0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8830003970 con 0x7f88381e77e0 2024-03-06T07:35:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830006d90 con 0x7f88381e77e0 2024-03-06T07:35:39.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88380cd820 con 0x7f88381e77e0 2024-03-06T07:35:39.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.546+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88380cdde0 con 0x7f88381e77e0 2024-03-06T07:35:39.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.547+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883804f330 con 0x7f88381e77e0 2024-03-06T07:35:39.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.548+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8830041030 con 0x7f88381e77e0 2024-03-06T07:35:39.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.549+0000 7f882ea00640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f881406b4d0 0x7f881406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.549+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f88300aaff0 con 0x7f88381e77e0 2024-03-06T07:35:39.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.550+0000 7f8836a00640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f881406b4d0 0x7f881406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.550+0000 7f8836a00640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f881406b4d0 0x7f881406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88280019b0 tx=0x7f8828009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.554+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8830033020 con 0x7f88381e77e0 2024-03-06T07:35:39.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.818+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f883815fd20 con 0x7f88381e77e0 2024-03-06T07:35:39.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.819+0000 7f882ea00640 1 -- 172.21.15.97:0/1808654020 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f883007a120 con 0x7f88381e77e0 2024-03-06T07:35:39.821 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:35:39.822 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.823 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.824 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.825 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.826 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:35:39.827 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:35:39.829 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:35:39.832 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:35:39.833 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:35:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.821+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f881406b4d0 msgr2=0x7f881406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.821+0000 7f883d600640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f881406b4d0 0x7f881406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88280019b0 tx=0x7f8828009040 comp rx=0 tx=0).stop 2024-03-06T07:35:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.821+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 msgr2=0x7f8838159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.821+0000 7f883d600640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 0x7f8838159f60 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8830001f70 tx=0x7f88300014a0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 shutdown_connections 2024-03-06T07:35:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f881406b4d0 0x7f881406d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88381e77e0 0x7f8838159f60 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88380cd190 0x7f88381e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 --2- 172.21.15.97:0/1808654020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f881c004d40 0x7f883814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 >> 172.21.15.97:0/1808654020 conn(0x7f88380b1730 msgr2=0x7f88380cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 shutdown_connections 2024-03-06T07:35:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:39.822+0000 7f883d600640 1 -- 172.21.15.97:0/1808654020 wait complete. 2024-03-06T07:35:39.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:40.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.049+0000 7f48e2400640 1 Processor -- start 2024-03-06T07:35:40.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.049+0000 7f48e2400640 1 -- start start 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48e2400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f48dc15f9d0 0x7f48dc15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48e2400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f48dc070b10 0x7f48dc070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48e2400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc06cef0 con 0x7f48dc144c00 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48e2400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc06d1d0 con 0x7f48dc070b10 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48e2400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc06d4b0 con 0x7f48dc15f9d0 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48dbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f48dc070b10 0x7f48dc070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48dbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f48dc070b10 0x7f48dc070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38268/0 (socket says 172.21.15.97:38268) 2024-03-06T07:35:40.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48dbe00640 1 -- 172.21.15.97:0/921045620 learned_addr learned my addr 172.21.15.97:0/921045620 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48daa00640 1 --2- 172.21.15.97:0/921045620 >> v2:172.21.15.135:3300/0 conn(0x7f48dc15f9d0 0x7f48dc15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.051+0000 7f48dbe00640 1 -- 172.21.15.97:0/921045620 >> v2:172.21.15.135:3300/0 conn(0x7f48dc15f9d0 msgr2=0x7f48dc15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.052+0000 7f48dbe00640 1 --2- 172.21.15.97:0/921045620 >> v2:172.21.15.135:3300/0 conn(0x7f48dc15f9d0 0x7f48dc15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.052+0000 7f48dbe00640 1 -- 172.21.15.97:0/921045620 >> v1:172.21.15.135:6789/0 conn(0x7f48dc144c00 legacy=0x7f48dc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.052+0000 7f48dbe00640 1 -- 172.21.15.97:0/921045620 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc06d790 con 0x7f48dc070b10 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.052+0000 7f48daa00640 1 --2- 172.21.15.97:0/921045620 >> v2:172.21.15.135:3300/0 conn(0x7f48dc15f9d0 0x7f48dc15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.053+0000 7f48dbe00640 1 --2- 172.21.15.97:0/921045620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc070f20 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0b26f0 tx=0x7f48d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8e3a7ebd0d5db6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.053+0000 7f48da000640 1 -- 172.21.15.97:0/921045620 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d402ca80 con 0x7f48dc070b10 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.053+0000 7f48da000640 1 -- 172.21.15.97:0/921045620 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d402cc20 con 0x7f48dc070b10 2024-03-06T07:35:40.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.053+0000 7f48da000640 1 -- 172.21.15.97:0/921045620 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40343f0 con 0x7f48dc070b10 2024-03-06T07:35:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.053+0000 7f48e2400640 1 -- 172.21.15.97:0/921045620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 msgr2=0x7f48dc070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.053+0000 7f48e2400640 1 --2- 172.21.15.97:0/921045620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc070f20 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0b26f0 tx=0x7f48d402bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.054+0000 7f48e2400640 1 -- 172.21.15.97:0/921045620 shutdown_connections 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.054+0000 7f48e2400640 1 --2- 172.21.15.97:0/921045620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc070f20 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.054+0000 7f48e2400640 1 --2- 172.21.15.97:0/921045620 >> v2:172.21.15.135:3300/0 conn(0x7f48dc15f9d0 0x7f48dc15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.054+0000 7f48e2400640 1 -- 172.21.15.97:0/921045620 >> 172.21.15.97:0/921045620 conn(0x7f48dc0b1730 msgr2=0x7f48dc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.054+0000 7f48e2400640 1 -- 172.21.15.97:0/921045620 shutdown_connections 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.054+0000 7f48e2400640 1 -- 172.21.15.97:0/921045620 wait complete. 2024-03-06T07:35:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.055+0000 7f48e2400640 1 Processor -- start 2024-03-06T07:35:40.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.056+0000 7f48e2400640 1 -- start start 2024-03-06T07:35:40.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.056+0000 7f48e2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.056+0000 7f48e2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48dc144c00 0x7f48dc15e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48e2400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48dc15f9d0 0x7f48dc15eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48e2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc06ccd0 con 0x7f48dc070b10 2024-03-06T07:35:40.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48e2400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc06cb50 con 0x7f48dc15f9d0 2024-03-06T07:35:40.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48e2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc06df90 con 0x7f48dc144c00 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38270/0 (socket says 172.21.15.97:38270) 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 -- 172.21.15.97:0/3604331258 learned_addr learned my addr 172.21.15.97:0/3604331258 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48dbe00640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48dc15f9d0 0x7f48dc15eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48daa00640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48dc144c00 0x7f48dc15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 -- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48dc144c00 msgr2=0x7f48dc15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48dc144c00 0x7f48dc15e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 -- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48dc15f9d0 msgr2=0x7f48dc15eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48dc15f9d0 0x7f48dc15eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48db400640 1 -- 172.21.15.97:0/3604331258 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc13cc00 con 0x7f48dc070b10 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.057+0000 7f48dbe00640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48dc15f9d0 0x7f48dc15eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.058+0000 7f48db400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc153910 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00c410 tx=0x7f48cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.058+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc017070 con 0x7f48dc070b10 2024-03-06T07:35:40.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.058+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc0035f0 con 0x7f48dc070b10 2024-03-06T07:35:40.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.058+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc0039e0 con 0x7f48dc070b10 2024-03-06T07:35:40.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.058+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48dc13cee0 con 0x7f48dc070b10 2024-03-06T07:35:40.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.058+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc13d4a0 con 0x7f48dc070b10 2024-03-06T07:35:40.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.059+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc04f330 con 0x7f48dc070b10 2024-03-06T07:35:40.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.060+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48cc003b80 con 0x7f48dc070b10 2024-03-06T07:35:40.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.061+0000 7f48d3400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48bc06b4d0 0x7f48bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.061+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f48cc089110 con 0x7f48dc070b10 2024-03-06T07:35:40.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.061+0000 7f48daa00640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48bc06b4d0 0x7f48bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.062+0000 7f48daa00640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48bc06b4d0 0x7f48bc06d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f48dc154350 tx=0x7f48c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.065+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48cc058370 con 0x7f48dc070b10 2024-03-06T07:35:40.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.260+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48dc0b14a0 con 0x7f48dc070b10 2024-03-06T07:35:40.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.261+0000 7f48d3400640 1 -- 172.21.15.97:0/3604331258 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+12889 (secure 0 0 0) 0x7f48cc057c70 con 0x7f48dc070b10 2024-03-06T07:35:40.264 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:40.264 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":126,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:35:39.305022+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48bc06b4d0 msgr2=0x7f48bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48bc06b4d0 0x7f48bc06d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f48dc154350 tx=0x7f48c8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 msgr2=0x7f48dc153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc153910 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00c410 tx=0x7f48cc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 shutdown_connections 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f48bc06b4d0 0x7f48bc06d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f48dc15f9d0 0x7f48dc15eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f48dc144c00 0x7f48dc15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 --2- 172.21.15.97:0/3604331258 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48dc070b10 0x7f48dc153910 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.264+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 >> 172.21.15.97:0/3604331258 conn(0x7f48dc0b1730 msgr2=0x7f48dc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.265+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 shutdown_connections 2024-03-06T07:35:40.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.265+0000 7f48e2400640 1 -- 172.21.15.97:0/3604331258 wait complete. 2024-03-06T07:35:40.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:40.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:40.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.496+0000 7f7f20800640 1 Processor -- start 2024-03-06T07:35:40.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.496+0000 7f7f20800640 1 -- start start 2024-03-06T07:35:40.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f20800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1815cb90 0x7f7f1806d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f20800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f1806e060 0x7f7f1806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f20800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1815cf70 con 0x7f7f18163800 2024-03-06T07:35:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f20800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f180b3890 con 0x7f7f1806e060 2024-03-06T07:35:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f20800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1806c990 con 0x7f7f1815cb90 2024-03-06T07:35:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f1e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f1806e060 0x7f7f1806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f1e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f1806e060 0x7f7f1806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38282/0 (socket says 172.21.15.97:38282) 2024-03-06T07:35:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.498+0000 7f7f1e800640 1 -- 172.21.15.97:0/3238744521 learned_addr learned my addr 172.21.15.97:0/3238744521 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.499+0000 7f7f1de00640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1815cb90 0x7f7f1806d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.499+0000 7f7f1e800640 1 -- 172.21.15.97:0/3238744521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1815cb90 msgr2=0x7f7f1806d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.499+0000 7f7f1e800640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1815cb90 0x7f7f1806d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.499+0000 7f7f1e800640 1 -- 172.21.15.97:0/3238744521 >> v1:172.21.15.135:6789/0 conn(0x7f7f18163800 legacy=0x7f7f180b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.499+0000 7f7f1e800640 1 -- 172.21.15.97:0/3238744521 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f180b35b0 con 0x7f7f1806e060 2024-03-06T07:35:40.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.499+0000 7f7f1de00640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1815cb90 0x7f7f1806d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:40.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.500+0000 7f7f1e800640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1806e060 0x7f7f1806e4f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009e50 tx=0x7f7f1402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7113015de9333b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.500+0000 7f7f1ca00640 1 -- 172.21.15.97:0/3238744521 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1402ca70 con 0x7f7f1806e060 2024-03-06T07:35:40.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.500+0000 7f7f1ca00640 1 -- 172.21.15.97:0/3238744521 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1402cc10 con 0x7f7f1806e060 2024-03-06T07:35:40.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.500+0000 7f7f1ca00640 1 -- 172.21.15.97:0/3238744521 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f140343f0 con 0x7f7f1806e060 2024-03-06T07:35:40.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 -- 172.21.15.97:0/3238744521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1806e060 msgr2=0x7f7f1806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1806e060 0x7f7f1806e4f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009e50 tx=0x7f7f1402bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 -- 172.21.15.97:0/3238744521 shutdown_connections 2024-03-06T07:35:40.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1806e060 0x7f7f1806e4f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 --2- 172.21.15.97:0/3238744521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1815cb90 0x7f7f1806d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 -- 172.21.15.97:0/3238744521 >> 172.21.15.97:0/3238744521 conn(0x7f7f180b1940 msgr2=0x7f7f180b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 -- 172.21.15.97:0/3238744521 shutdown_connections 2024-03-06T07:35:40.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.501+0000 7f7f20800640 1 -- 172.21.15.97:0/3238744521 wait complete. 2024-03-06T07:35:40.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.503+0000 7f7f20800640 1 Processor -- start 2024-03-06T07:35:40.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.503+0000 7f7f20800640 1 -- start start 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f20800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1806e060 0x7f7f181eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f20800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 0x7f7f18072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f20800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f18163800 0x7f7f181ecd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f20800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180b3350 con 0x7f7f1815cb90 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f20800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1806d230 con 0x7f7f18163800 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f20800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1806cec0 con 0x7f7f1806e060 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f1d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 0x7f7f18072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f1d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 0x7f7f18072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38294/0 (socket says 172.21.15.97:38294) 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.504+0000 7f7f1d400640 1 -- 172.21.15.97:0/17966508 learned_addr learned my addr 172.21.15.97:0/17966508 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1de00640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1806e060 0x7f7f181eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1e800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f18163800 0x7f7f181ecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1d400640 1 -- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1806e060 msgr2=0x7f7f181eecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1d400640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1806e060 0x7f7f181eecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1d400640 1 -- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f18163800 msgr2=0x7f7f181ecd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1d400640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f18163800 0x7f7f181ecd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1d400640 1 -- 172.21.15.97:0/17966508 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f180715e0 con 0x7f7f1815cb90 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.505+0000 7f7f1d400640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 0x7f7f18072070 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800b3f0 tx=0x7f7f0800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.506+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08015020 con 0x7f7f1815cb90 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.506+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08011830 con 0x7f7f1815cb90 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.506+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1815bca0 con 0x7f7f1815cb90 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.506+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08011c20 con 0x7f7f1815cb90 2024-03-06T07:35:40.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.506+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f18071870 con 0x7f7f1815cb90 2024-03-06T07:35:40.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.507+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1804f330 con 0x7f7f1815cb90 2024-03-06T07:35:40.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.508+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f08018020 con 0x7f7f1815cb90 2024-03-06T07:35:40.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.509+0000 7f7f12000640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.509+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f08089b90 con 0x7f7f1815cb90 2024-03-06T07:35:40.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.509+0000 7f7f1de00640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.510+0000 7f7f1de00640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7efc06b4d0 0x7f7efc06d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00a970 tx=0x7f7f0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.513+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f0801f030 con 0x7f7f1815cb90 2024-03-06T07:35:40.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.711+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f18165a80 con 0x7f7f1815cb90 2024-03-06T07:35:40.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.712+0000 7f7f12000640 1 -- 172.21.15.97:0/17966508 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f08058cc0 con 0x7f7f1815cb90 2024-03-06T07:35:40.714 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:35:40.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7efc06b4d0 msgr2=0x7f7efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7efc06b4d0 0x7f7efc06d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00a970 tx=0x7f7f0c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 msgr2=0x7f7f18072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 0x7f7f18072070 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800b3f0 tx=0x7f7f0800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 shutdown_connections 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f18163800 0x7f7f181ecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f1815cb90 0x7f7f18072070 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 --2- 172.21.15.97:0/17966508 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7f1806e060 0x7f7f181eecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 >> 172.21.15.97:0/17966508 conn(0x7f7f180b1940 msgr2=0x7f7f180cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 shutdown_connections 2024-03-06T07:35:40.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.715+0000 7f7f20800640 1 -- 172.21.15.97:0/17966508 wait complete. 2024-03-06T07:35:40.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:35:40.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.939+0000 7f982da00640 1 Processor -- start 2024-03-06T07:35:40.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.939+0000 7f982da00640 1 -- start start 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f982da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98280cf430 0x7f9828070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f982da00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f982806d260 0x7f98280b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f982da00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98280717c0 con 0x7f9828071260 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f982da00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9828071aa0 con 0x7f982806d260 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f982da00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9828071d80 con 0x7f98280cf430 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f9827400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f982806d260 0x7f98280b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f9827400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f982806d260 0x7f98280b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38308/0 (socket says 172.21.15.97:38308) 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f9827400640 1 -- 172.21.15.97:0/2047223647 learned_addr learned my addr 172.21.15.97:0/2047223647 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.941+0000 7f9826a00640 1 --2- 172.21.15.97:0/2047223647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98280cf430 0x7f9828070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.942+0000 7f9827400640 1 -- 172.21.15.97:0/2047223647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98280cf430 msgr2=0x7f9828070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.942+0000 7f9827400640 1 --2- 172.21.15.97:0/2047223647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98280cf430 0x7f9828070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.942+0000 7f9827400640 1 -- 172.21.15.97:0/2047223647 >> v1:172.21.15.135:6789/0 conn(0x7f9828071260 legacy=0x7f982806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.942+0000 7f9827400640 1 -- 172.21.15.97:0/2047223647 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828072060 con 0x7f982806d260 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.943+0000 7f9827400640 1 --2- 172.21.15.97:0/2047223647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f982806d260 0x7f98280b2d10 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f9820009e50 tx=0x7f982002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b66da9ceda1459 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.943+0000 7f9825600640 1 -- 172.21.15.97:0/2047223647 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982002cb00 con 0x7f982806d260 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.943+0000 7f9825600640 1 -- 172.21.15.97:0/2047223647 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982002cca0 con 0x7f982806d260 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f9825600640 1 -- 172.21.15.97:0/2047223647 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820034610 con 0x7f982806d260 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 -- 172.21.15.97:0/2047223647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f982806d260 msgr2=0x7f98280b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 --2- 172.21.15.97:0/2047223647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f982806d260 0x7f98280b2d10 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f9820009e50 tx=0x7f982002bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 -- 172.21.15.97:0/2047223647 shutdown_connections 2024-03-06T07:35:40.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 --2- 172.21.15.97:0/2047223647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f982806d260 0x7f98280b2d10 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 --2- 172.21.15.97:0/2047223647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98280cf430 0x7f9828070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 -- 172.21.15.97:0/2047223647 >> 172.21.15.97:0/2047223647 conn(0x7f98280b1730 msgr2=0x7f98280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 -- 172.21.15.97:0/2047223647 shutdown_connections 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.944+0000 7f982da00640 1 -- 172.21.15.97:0/2047223647 wait complete. 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.946+0000 7f982da00640 1 Processor -- start 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.946+0000 7f982da00640 1 -- start start 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f982da00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f982806d260 0x7f9828164ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f982da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9828071260 0x7f9828147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f982da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 0x7f98281481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f982da00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828162cf0 con 0x7f98280cf430 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f982da00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828162b70 con 0x7f982806d260 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f982da00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98280b3250 con 0x7f9828071260 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.947+0000 7f9827400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 0x7f98281481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 0x7f98281481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38314/0 (socket says 172.21.15.97:38314) 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 -- 172.21.15.97:0/278979615 learned_addr learned my addr 172.21.15.97:0/278979615 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9826a00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f982806d260 0x7f9828164ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9826000640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9828071260 0x7f9828147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 -- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9828071260 msgr2=0x7f9828147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9828071260 0x7f9828147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 -- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f982806d260 msgr2=0x7f9828164ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f982806d260 0x7f9828164ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 -- 172.21.15.97:0/278979615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982815b090 con 0x7f98280cf430 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9826000640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9828071260 0x7f9828147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.948+0000 7f9827400640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 0x7f98281481a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f9820009e50 tx=0x7f9820004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.949+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820043020 con 0x7f98280cf430 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.949+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982002ce50 con 0x7f98280cf430 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.949+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820009510 con 0x7f98280cf430 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.949+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98281457a0 con 0x7f98280cf430 2024-03-06T07:35:40.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.949+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9828145d60 con 0x7f98280cf430 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.950+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982804f330 con 0x7f98280cf430 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.951+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9820046020 con 0x7f98280cf430 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.952+0000 7f981ea00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f980806b4d0 0x7f980806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.952+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f98200ac540 con 0x7f98280cf430 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.952+0000 7f9826a00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f980806b4d0 0x7f980806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.953+0000 7f9826a00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f980806b4d0 0x7f980806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9818006fd0 tx=0x7f9818008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:40.956+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f982004d030 con 0x7f98280cf430 2024-03-06T07:35:41.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.147+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9828162390 con 0x7f98280cf430 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.148+0000 7f981ea00640 1 -- 172.21.15.97:0/278979615 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v126) v1 ==== 84+0+65 (secure 0 0 0) 0x7f982815b090 con 0x7f98280cf430 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:35:41.150 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f980806b4d0 msgr2=0x7f980806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f980806b4d0 0x7f980806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9818006fd0 tx=0x7f9818008040 comp rx=0 tx=0).stop 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 msgr2=0x7f98281481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 0x7f98281481a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f9820009e50 tx=0x7f9820004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 shutdown_connections 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f980806b4d0 0x7f980806d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98280cf430 0x7f98281481a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9828071260 0x7f9828147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 --2- 172.21.15.97:0/278979615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f982806d260 0x7f9828164ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 >> 172.21.15.97:0/278979615 conn(0x7f98280b1730 msgr2=0x7f98280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.150+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 shutdown_connections 2024-03-06T07:35:41.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.151+0000 7f982da00640 1 -- 172.21.15.97:0/278979615 wait complete. 2024-03-06T07:35:41.171 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:35:41.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:35:41.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.378+0000 7f928a000640 1 Processor -- start 2024-03-06T07:35:41.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.378+0000 7f928a000640 1 -- start start 2024-03-06T07:35:41.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.380+0000 7f928a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 0x7f928415fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.380+0000 7f928a000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840cf1e0 con 0x7f92840ced20 2024-03-06T07:35:41.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.380+0000 7f928a000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840cf4c0 con 0x7f928415ff40 2024-03-06T07:35:41.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.380+0000 7f928a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92841603b0 con 0x7f9284145980 2024-03-06T07:35:41.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.380+0000 7f9282a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f92840ced20 0x7f92840712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55542/0 (socket says 172.21.15.97:55542) 2024-03-06T07:35:41.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.380+0000 7f9282a00640 1 -- 172.21.15.97:0/4156484185 learned_addr learned my addr 172.21.15.97:0/4156484185 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704998133 0 0) 0x7f92840cf1e0 con 0x7f92840ced20 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9268003520 con 0x7f92840ced20 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9283400640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 0x7f928415fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1036858737 0 0) 0x7f9268003520 con 0x7f92840ced20 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 msgr2=0x7f928415fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 0x7f928415fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 >> v1:172.21.15.135:6789/0 conn(0x7f928415ff40 legacy=0x7f92840719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92840cf7a0 con 0x7f92840ced20 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9270001670 con 0x7f92840ced20 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 >> v1:172.21.15.97:6789/0 conn(0x7f92840ced20 legacy=0x7f92840712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f9268003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9268004800 0x7f9268004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 0x7f9268005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92841603b0 con 0x7f9268003a50 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926800bb80 con 0x7f9268004800 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.381+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926800bd80 con 0x7f9284145980 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.382+0000 7f9283400640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f9268003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.382+0000 7f9283400640 1 -- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 msgr2=0x7f9268005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.382+0000 7f9283400640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9284145980 0x7f9268005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.382+0000 7f9283400640 1 -- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9268004800 msgr2=0x7f9268004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.382+0000 7f9283400640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9268004800 0x7f9268004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.382+0000 7f9283400640 1 -- 172.21.15.97:0/4156484185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9268003520 con 0x7f9268003a50 2024-03-06T07:35:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.383+0000 7f9283400640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f9268003e70 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9274007360 tx=0x7f9268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c475abd500f50494 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.383+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274039070 con 0x7f9268003a50 2024-03-06T07:35:41.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.383+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927402ccf0 con 0x7f9268003a50 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.383+0000 7f9282000640 1 -- 172.21.15.97:0/4156484185 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274009be0 con 0x7f9268003a50 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f928a000640 1 -- 172.21.15.97:0/4156484185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 msgr2=0x7f9268003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f928a000640 1 --2- 172.21.15.97:0/4156484185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f9268003e70 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9274007360 tx=0x7f9268005830 comp rx=0 tx=0).stop 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f9283e00640 1 -- 172.21.15.97:0/4156484185 reap_dead start 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f928a000640 1 -- 172.21.15.97:0/4156484185 shutdown_connections 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f928a000640 1 -- 172.21.15.97:0/4156484185 >> 172.21.15.97:0/4156484185 conn(0x7f92840cfa80 msgr2=0x7f92840cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f928a000640 1 -- 172.21.15.97:0/4156484185 shutdown_connections 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.384+0000 7f928a000640 1 -- 172.21.15.97:0/4156484185 wait complete. 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.385+0000 7f928a000640 1 Processor -- start 2024-03-06T07:35:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.385+0000 7f928a000640 1 -- start start 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.386+0000 7f928a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f928413b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.386+0000 7f928a000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f928415b910 0x7f92840ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.386+0000 7f928a000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92840adda0 0x7f92840ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.386+0000 7f928a000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284160740 con 0x7f9268003a50 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.386+0000 7f928a000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284070e40 con 0x7f928415b910 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.386+0000 7f928a000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92841608c0 con 0x7f92840adda0 2024-03-06T07:35:41.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f928413b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f928413b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38332/0 (socket says 172.21.15.97:38332) 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 -- 172.21.15.97:0/1648011976 learned_addr learned my addr 172.21.15.97:0/1648011976 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 -- 172.21.15.97:0/1648011976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92840adda0 msgr2=0x7f92840ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92840adda0 0x7f92840ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 -- 172.21.15.97:0/1648011976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f928415b910 msgr2=0x7f92840ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f928415b910 0x7f92840ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 -- 172.21.15.97:0/1648011976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92841431d0 con 0x7f9268003a50 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f9283400640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f928413b740 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f9274007480 tx=0x7f9274004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274042030 con 0x7f9268003a50 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.387+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927403a040 con 0x7f9268003a50 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.388+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274039070 con 0x7f9268003a50 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.388+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92840b4710 con 0x7f9268003a50 2024-03-06T07:35:41.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.388+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9284143470 con 0x7f9268003a50 2024-03-06T07:35:41.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.389+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92840b4c30 con 0x7f9268003a50 2024-03-06T07:35:41.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.389+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9274045070 con 0x7f9268003a50 2024-03-06T07:35:41.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.390+0000 7f927b400640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f926406b500 0x7f926406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.390+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f92740abaa0 con 0x7f9268003a50 2024-03-06T07:35:41.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.391+0000 7f9282a00640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f926406b500 0x7f926406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.391+0000 7f9282a00640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f926406b500 0x7f926406d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9270000f60 tx=0x7f9270001500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.395+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92840b4c30 con 0x7f9268003a50 2024-03-06T07:35:41.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:41.581+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f928404f330 con 0x7f9268003a50 2024-03-06T07:35:42.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.339+0000 7f927b400640 1 -- 172.21.15.97:0/1648011976 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v127) v1 ==== 138+0+0 (secure 0 0 0) 0x7f927404c070 con 0x7f9268003a50 2024-03-06T07:35:42.341 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:35:42.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.343+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f926406b500 msgr2=0x7f926406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.343+0000 7f928a000640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f926406b500 0x7f926406d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9270000f60 tx=0x7f9270001500 comp rx=0 tx=0).stop 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.343+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 msgr2=0x7f928413b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.343+0000 7f928a000640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f928413b740 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f9274007480 tx=0x7f9274004ab0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 shutdown_connections 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f926406b500 0x7f926406d9c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92840adda0 0x7f92840ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f928415b910 0x7f92840ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 --2- 172.21.15.97:0/1648011976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9268003a50 0x7f928413b740 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 >> 172.21.15.97:0/1648011976 conn(0x7f92840cfa80 msgr2=0x7f92840711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 shutdown_connections 2024-03-06T07:35:42.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.344+0000 7f928a000640 1 -- 172.21.15.97:0/1648011976 wait complete. 2024-03-06T07:35:42.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:35:42.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.577+0000 7fa8d3e00640 1 Processor -- start 2024-03-06T07:35:42.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.577+0000 7fa8d3e00640 1 -- start start 2024-03-06T07:35:42.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8cc0cda90 0x7fa8cc0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d3e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc0719e0 con 0x7fa8cc0b3650 2024-03-06T07:35:42.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d3e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc071cc0 con 0x7fa8cc0b3df0 2024-03-06T07:35:42.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d3e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc072310 con 0x7fa8cc0cda90 2024-03-06T07:35:42.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8cc0cda90 0x7fa8cc0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8cc0cda90 0x7fa8cc0727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38344/0 (socket says 172.21.15.97:38344) 2024-03-06T07:35:42.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d1e00640 1 -- 172.21.15.97:0/1970267957 learned_addr learned my addr 172.21.15.97:0/1970267957 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:42.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.579+0000 7fa8d0a00640 1 --1- 172.21.15.97:0/1970267957 >> v1:172.21.15.97:6789/0 conn(0x7fa8cc0b3650 0x7fa8cc06c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55552/0 (socket says 172.21.15.97:55552) 2024-03-06T07:35:42.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.580+0000 7fa8d1e00640 1 -- 172.21.15.97:0/1970267957 >> v1:172.21.15.135:6789/0 conn(0x7fa8cc0b3df0 legacy=0x7fa8cc0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:42.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.580+0000 7fa8d1e00640 1 -- 172.21.15.97:0/1970267957 >> v1:172.21.15.97:6789/0 conn(0x7fa8cc0b3650 legacy=0x7fa8cc06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.580+0000 7fa8d1e00640 1 -- 172.21.15.97:0/1970267957 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc071fa0 con 0x7fa8cc0cda90 2024-03-06T07:35:42.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.581+0000 7fa8d1e00640 1 --2- 172.21.15.97:0/1970267957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc0727a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009e50 tx=0x7fa8c402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5248c989097eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:42.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.581+0000 7fa8cbe00640 1 -- 172.21.15.97:0/1970267957 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c402cb20 con 0x7fa8cc0cda90 2024-03-06T07:35:42.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.581+0000 7fa8cbe00640 1 -- 172.21.15.97:0/1970267957 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c402ccc0 con 0x7fa8cc0cda90 2024-03-06T07:35:42.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.581+0000 7fa8cbe00640 1 -- 172.21.15.97:0/1970267957 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4034610 con 0x7fa8cc0cda90 2024-03-06T07:35:42.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 -- 172.21.15.97:0/1970267957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 msgr2=0x7fa8cc0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/1970267957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc0727a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009e50 tx=0x7fa8c402bf80 comp rx=0 tx=0).stop 2024-03-06T07:35:42.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 -- 172.21.15.97:0/1970267957 shutdown_connections 2024-03-06T07:35:42.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/1970267957 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc0727a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 -- 172.21.15.97:0/1970267957 >> 172.21.15.97:0/1970267957 conn(0x7fa8cc0b1940 msgr2=0x7fa8cc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:42.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 -- 172.21.15.97:0/1970267957 shutdown_connections 2024-03-06T07:35:42.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.582+0000 7fa8d3e00640 1 -- 172.21.15.97:0/1970267957 wait complete. 2024-03-06T07:35:42.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.584+0000 7fa8d3e00640 1 Processor -- start 2024-03-06T07:35:42.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.584+0000 7fa8d3e00640 1 -- start start 2024-03-06T07:35:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d3e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8cc0b3650 0x7fa8cc152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d3e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8cc0b3df0 0x7fa8cc09be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc15b3c0 con 0x7fa8cc0cda90 2024-03-06T07:35:42.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d3e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc15b240 con 0x7fa8cc0b3df0 2024-03-06T07:35:42.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d3e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0cf510 con 0x7fa8cc0b3650 2024-03-06T07:35:42.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc14e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38346/0 (socket says 172.21.15.97:38346) 2024-03-06T07:35:42.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.585+0000 7fa8d1e00640 1 -- 172.21.15.97:0/182399203 learned_addr learned my addr 172.21.15.97:0/182399203 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d0a00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8cc0b3df0 0x7fa8cc09be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1400640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8cc0b3650 0x7fa8cc152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1e00640 1 -- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8cc0b3650 msgr2=0x7fa8cc152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8cc0b3650 0x7fa8cc152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1e00640 1 -- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8cc0b3df0 msgr2=0x7fa8cc09be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8cc0b3df0 0x7fa8cc09be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1e00640 1 -- 172.21.15.97:0/182399203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc09d530 con 0x7fa8cc0cda90 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d0a00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8cc0b3df0 0x7fa8cc09be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d1e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc14e200 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4036040 tx=0x7fa8c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4043020 con 0x7fa8cc0cda90 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c403e450 con 0x7fa8cc0cda90 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c403e820 con 0x7fa8cc0cda90 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc09d810 con 0x7fa8cc0cda90 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.586+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc14e740 con 0x7fa8cc0cda90 2024-03-06T07:35:42.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.588+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc04f330 con 0x7fa8cc0cda90 2024-03-06T07:35:42.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.589+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8c4046020 con 0x7fa8cc0cda90 2024-03-06T07:35:42.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.590+0000 7fa8c9600640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8ac06b4d0 0x7fa8ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.590+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa8c40ad080 con 0x7fa8cc0cda90 2024-03-06T07:35:42.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.590+0000 7fa8d1400640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8ac06b4d0 0x7fa8ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.591+0000 7fa8d1400640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8ac06b4d0 0x7fa8ac06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc003d90 tx=0x7fa8bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:42.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.594+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8c404d030 con 0x7fa8cc0cda90 2024-03-06T07:35:42.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:42.789+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa8cc0cc0e0 con 0x7fa8cc0cda90 2024-03-06T07:35:43.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.344+0000 7fa8c9600640 1 -- 172.21.15.97:0/182399203 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v128) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa8c407c370 con 0x7fa8cc0cda90 2024-03-06T07:35:43.347 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:35:43.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8ac06b4d0 msgr2=0x7fa8ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8ac06b4d0 0x7fa8ac06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc003d90 tx=0x7fa8bc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:43.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 msgr2=0x7fa8cc14e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc14e200 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4036040 tx=0x7fa8c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 shutdown_connections 2024-03-06T07:35:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa8ac06b4d0 0x7fa8ac06d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8cc0cda90 0x7fa8cc14e200 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8cc0b3df0 0x7fa8cc09be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.348+0000 7fa8d3e00640 1 --2- 172.21.15.97:0/182399203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8cc0b3650 0x7fa8cc152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.349+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 >> 172.21.15.97:0/182399203 conn(0x7fa8cc0b1940 msgr2=0x7fa8cc0b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:43.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.349+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 shutdown_connections 2024-03-06T07:35:43.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.350+0000 7fa8d3e00640 1 -- 172.21.15.97:0/182399203 wait complete. 2024-03-06T07:35:43.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:35:43.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.570+0000 7fbf9d400640 1 Processor -- start 2024-03-06T07:35:43.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.570+0000 7fbf9d400640 1 -- start start 2024-03-06T07:35:43.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.571+0000 7fbf9d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf980b3df0 0x7fbf980727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.571+0000 7fbf9d400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbf980b3650 0x7fbf980b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.571+0000 7fbf9d400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf980b4710 con 0x7fbf980cda90 2024-03-06T07:35:43.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf9d400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf98072310 con 0x7fbf980b3df0 2024-03-06T07:35:43.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf9d400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980719e0 con 0x7fbf980b3650 2024-03-06T07:35:43.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf96800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf980b3df0 0x7fbf980727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf97200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fbf980cda90 0x7fbf9806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55554/0 (socket says 172.21.15.97:55554) 2024-03-06T07:35:43.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf97200640 1 -- 172.21.15.97:0/1693194651 learned_addr learned my addr 172.21.15.97:0/1693194651 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:43.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf8fe00640 1 --2- 172.21.15.97:0/1693194651 >> v2:172.21.15.135:3300/0 conn(0x7fbf980b3650 0x7fbf980b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf96800640 1 -- 172.21.15.97:0/1693194651 >> v2:172.21.15.135:3300/0 conn(0x7fbf980b3650 msgr2=0x7fbf980b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf96800640 1 --2- 172.21.15.97:0/1693194651 >> v2:172.21.15.135:3300/0 conn(0x7fbf980b3650 0x7fbf980b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.572+0000 7fbf96800640 1 -- 172.21.15.97:0/1693194651 >> v1:172.21.15.97:6789/0 conn(0x7fbf980cda90 legacy=0x7fbf9806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.573+0000 7fbf95e00640 1 -- 172.21.15.97:0/1693194651 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628134270 0 0) 0x7fbf980b4710 con 0x7fbf980cda90 2024-03-06T07:35:43.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.573+0000 7fbf96800640 1 -- 172.21.15.97:0/1693194651 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf98071cc0 con 0x7fbf980b3df0 2024-03-06T07:35:43.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.574+0000 7fbf96800640 1 --2- 172.21.15.97:0/1693194651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980b3df0 0x7fbf980727a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fbf980b4c90 tx=0x7fbf8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202e32baf150bdb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.574+0000 7fbf95e00640 1 -- 172.21.15.97:0/1693194651 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88035020 con 0x7fbf980b3df0 2024-03-06T07:35:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.574+0000 7fbf95e00640 1 -- 172.21.15.97:0/1693194651 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf88031830 con 0x7fbf980b3df0 2024-03-06T07:35:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.574+0000 7fbf95e00640 1 -- 172.21.15.97:0/1693194651 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88031c00 con 0x7fbf980b3df0 2024-03-06T07:35:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.574+0000 7fbf9d400640 1 -- 172.21.15.97:0/1693194651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980b3df0 msgr2=0x7fbf980727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.574+0000 7fbf9d400640 1 --2- 172.21.15.97:0/1693194651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980b3df0 0x7fbf980727a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fbf980b4c90 tx=0x7fbf8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.575+0000 7fbf9d400640 1 -- 172.21.15.97:0/1693194651 shutdown_connections 2024-03-06T07:35:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.575+0000 7fbf9d400640 1 --2- 172.21.15.97:0/1693194651 >> v2:172.21.15.135:3300/0 conn(0x7fbf980b3650 0x7fbf980b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.575+0000 7fbf9d400640 1 --2- 172.21.15.97:0/1693194651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980b3df0 0x7fbf980727a0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.575+0000 7fbf9d400640 1 -- 172.21.15.97:0/1693194651 >> 172.21.15.97:0/1693194651 conn(0x7fbf980b1940 msgr2=0x7fbf980b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.575+0000 7fbf9d400640 1 -- 172.21.15.97:0/1693194651 shutdown_connections 2024-03-06T07:35:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.575+0000 7fbf9d400640 1 -- 172.21.15.97:0/1693194651 wait complete. 2024-03-06T07:35:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.576+0000 7fbf9d400640 1 Processor -- start 2024-03-06T07:35:43.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 -- start start 2024-03-06T07:35:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf980b3650 0x7fbf981494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf980b3df0 0x7fbf9809d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 0x7fbf9809d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9815b240 con 0x7fbf980cda90 2024-03-06T07:35:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980b3b30 con 0x7fbf980b3df0 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.577+0000 7fbf9d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9815b3c0 con 0x7fbf980b3650 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 0x7fbf9809d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 0x7fbf9809d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38364/0 (socket says 172.21.15.97:38364) 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 -- 172.21.15.97:0/540540150 learned_addr learned my addr 172.21.15.97:0/540540150 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 -- 172.21.15.97:0/540540150 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf980b3650 msgr2=0x7fbf981494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf980b3650 0x7fbf981494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 -- 172.21.15.97:0/540540150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf980b3df0 msgr2=0x7fbf9809d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf980b3df0 0x7fbf9809d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 -- 172.21.15.97:0/540540150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf98157910 con 0x7fbf980cda90 2024-03-06T07:35:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.578+0000 7fbf97200640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 0x7fbf9809d5c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000c700 tx=0x7fbf9000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.579+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90018070 con 0x7fbf980cda90 2024-03-06T07:35:43.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.579+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9000ce10 con 0x7fbf980cda90 2024-03-06T07:35:43.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.579+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9000a860 con 0x7fbf980cda90 2024-03-06T07:35:43.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.579+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf98149a00 con 0x7fbf980cda90 2024-03-06T07:35:43.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.579+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9809bd50 con 0x7fbf980cda90 2024-03-06T07:35:43.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.580+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9804f330 con 0x7fbf980cda90 2024-03-06T07:35:43.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.581+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf9000aa00 con 0x7fbf980cda90 2024-03-06T07:35:43.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.582+0000 7fbf8ea00640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7406b4d0 0x7fbf7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.582+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fbf90089820 con 0x7fbf980cda90 2024-03-06T07:35:43.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.582+0000 7fbf96800640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7406b4d0 0x7fbf7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.583+0000 7fbf96800640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7406b4d0 0x7fbf7406d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbf88036000 tx=0x7fbf88030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.587+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf90058660 con 0x7fbf980cda90 2024-03-06T07:35:43.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:43.819+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbf980cc0e0 con 0x7fbf980cda90 2024-03-06T07:35:44.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.366+0000 7fbf8ea00640 1 -- 172.21.15.97:0/540540150 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v84) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbf90057f60 con 0x7fbf980cda90 2024-03-06T07:35:44.369 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:35:44.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.370+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7406b4d0 msgr2=0x7fbf7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.370+0000 7fbf9d400640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7406b4d0 0x7fbf7406d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbf88036000 tx=0x7fbf88030000 comp rx=0 tx=0).stop 2024-03-06T07:35:44.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.370+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 msgr2=0x7fbf9809d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.370+0000 7fbf9d400640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 0x7fbf9809d5c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000c700 tx=0x7fbf9000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.371+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 shutdown_connections 2024-03-06T07:35:44.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.371+0000 7fbf9d400640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7406b4d0 0x7fbf7406d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.371+0000 7fbf9d400640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf980cda90 0x7fbf9809d5c0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.371+0000 7fbf9d400640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf980b3df0 0x7fbf9809d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.371+0000 7fbf9d400640 1 --2- 172.21.15.97:0/540540150 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf980b3650 0x7fbf981494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.371+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 >> 172.21.15.97:0/540540150 conn(0x7fbf980b1940 msgr2=0x7fbf980afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:44.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.372+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 shutdown_connections 2024-03-06T07:35:44.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.373+0000 7fbf9d400640 1 -- 172.21.15.97:0/540540150 wait complete. 2024-03-06T07:35:44.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:44.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.587+0000 7f669c200640 1 Processor -- start 2024-03-06T07:35:44.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.588+0000 7f669c200640 1 -- start start 2024-03-06T07:35:44.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f669c200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6694151660 0x7f6694071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f669c200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6694071a60 con 0x7f66940cf430 2024-03-06T07:35:44.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f669c200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66940b35b0 con 0x7f669415dff0 2024-03-06T07:35:44.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f669c200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66940b3ee0 con 0x7f6694151660 2024-03-06T07:35:44.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f6699800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6694151660 0x7f6694071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f6698e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f66940cf430 0x7f669406d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50150/0 (socket says 172.21.15.97:50150) 2024-03-06T07:35:44.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.589+0000 7f6698e00640 1 -- 172.21.15.97:0/3165745030 learned_addr learned my addr 172.21.15.97:0/3165745030 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:44.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.590+0000 7f6699800640 1 -- 172.21.15.97:0/3165745030 >> v1:172.21.15.135:6789/0 conn(0x7f669415dff0 legacy=0x7f66940b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:44.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.590+0000 7f6699800640 1 -- 172.21.15.97:0/3165745030 >> v1:172.21.15.97:6789/0 conn(0x7f66940cf430 legacy=0x7f669406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.590+0000 7f6693e00640 1 -- 172.21.15.97:0/3165745030 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551264873 0 0) 0x7f6694071a60 con 0x7f66940cf430 2024-03-06T07:35:44.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.590+0000 7f6699800640 1 -- 172.21.15.97:0/3165745030 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66940b3890 con 0x7f6694151660 2024-03-06T07:35:44.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.591+0000 7f6699800640 1 --2- 172.21.15.97:0/3165745030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694071520 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f66880097d0 tx=0x7f668802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef855af27a20510 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.591+0000 7f6693e00640 1 -- 172.21.15.97:0/3165745030 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688038070 con 0x7f6694151660 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.591+0000 7f6693e00640 1 -- 172.21.15.97:0/3165745030 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6688031700 con 0x7f6694151660 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.591+0000 7f6693e00640 1 -- 172.21.15.97:0/3165745030 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688031ad0 con 0x7f6694151660 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 -- 172.21.15.97:0/3165745030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 msgr2=0x7f6694071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 --2- 172.21.15.97:0/3165745030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694071520 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f66880097d0 tx=0x7f668802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 -- 172.21.15.97:0/3165745030 shutdown_connections 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 --2- 172.21.15.97:0/3165745030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694071520 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 -- 172.21.15.97:0/3165745030 >> 172.21.15.97:0/3165745030 conn(0x7f66940b1730 msgr2=0x7f66940b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 -- 172.21.15.97:0/3165745030 shutdown_connections 2024-03-06T07:35:44.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.592+0000 7f669c200640 1 -- 172.21.15.97:0/3165745030 wait complete. 2024-03-06T07:35:44.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.593+0000 7f669c200640 1 Processor -- start 2024-03-06T07:35:44.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.593+0000 7f669c200640 1 -- start start 2024-03-06T07:35:44.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.594+0000 7f669c200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66940cf430 0x7f6694153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.594+0000 7f669c200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694168b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.594+0000 7f669c200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f669415dff0 0x7f6694143880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.594+0000 7f669c200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669406daf0 con 0x7f6694151660 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.594+0000 7f669c200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6694071d30 con 0x7f66940cf430 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.594+0000 7f669c200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66940b4a80 con 0x7f669415dff0 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60974/0 (socket says 172.21.15.97:60974) 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 -- 172.21.15.97:0/2343415500 learned_addr learned my addr 172.21.15.97:0/2343415500 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 -- 172.21.15.97:0/2343415500 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f669415dff0 msgr2=0x7f6694143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f669415dff0 0x7f6694143880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 -- 172.21.15.97:0/2343415500 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66940cf430 msgr2=0x7f6694153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66940cf430 0x7f6694153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 -- 172.21.15.97:0/2343415500 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694154040 con 0x7f6694151660 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6698e00640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694168b60 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f668000c500 tx=0x7f668000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.595+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680018070 con 0x7f6694151660 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.596+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680007580 con 0x7f6694151660 2024-03-06T07:35:44.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.596+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6694154320 con 0x7f6694151660 2024-03-06T07:35:44.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.596+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680004a20 con 0x7f6694151660 2024-03-06T07:35:44.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.596+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66941548e0 con 0x7f6694151660 2024-03-06T07:35:44.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.597+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669404f330 con 0x7f6694151660 2024-03-06T07:35:44.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.598+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f668001b070 con 0x7f6694151660 2024-03-06T07:35:44.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.599+0000 7f6691600640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f667406b4d0 0x7f667406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.599+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66800894a0 con 0x7f6694151660 2024-03-06T07:35:44.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.599+0000 7f6699800640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f667406b4d0 0x7f667406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.600+0000 7f6699800640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f667406b4d0 0x7f667406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6688035000 tx=0x7f6688030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.604+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66800583b0 con 0x7f6694151660 2024-03-06T07:35:44.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.792+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f669415bcb0 con 0x7f6694151660 2024-03-06T07:35:44.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.794+0000 7f6691600640 1 -- 172.21.15.97:0/2343415500 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f6680057cb0 con 0x7f6694151660 2024-03-06T07:35:44.796 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:44.797 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":129,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:35:43.820953+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f667406b4d0 msgr2=0x7f667406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f667406b4d0 0x7f667406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6688035000 tx=0x7f6688030000 comp rx=0 tx=0).stop 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 msgr2=0x7f6694168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694168b60 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f668000c500 tx=0x7f668000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 shutdown_connections 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f667406b4d0 0x7f667406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f669415dff0 0x7f6694143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6694151660 0x7f6694168b60 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 --2- 172.21.15.97:0/2343415500 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66940cf430 0x7f6694153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.796+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 >> 172.21.15.97:0/2343415500 conn(0x7f66940b1730 msgr2=0x7f669406de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.797+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 shutdown_connections 2024-03-06T07:35:44.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:44.797+0000 7f669c200640 1 -- 172.21.15.97:0/2343415500 wait complete. 2024-03-06T07:35:44.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:45.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.015+0000 7f8a96400640 1 Processor -- start 2024-03-06T07:35:45.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.015+0000 7f8a96400640 1 -- start start 2024-03-06T07:35:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a96400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a90144c00 0x7f8a9006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a96400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8a9015fe60 0x7f8a900ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a96400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9006ee70 con 0x7f8a9015f9d0 2024-03-06T07:35:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a96400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9006cef0 con 0x7f8a90144c00 2024-03-06T07:35:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a96400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9006d1d0 con 0x7f8a9015fe60 2024-03-06T07:35:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a8f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a90144c00 0x7f8a9006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a8f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8a90144c00 0x7f8a9006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60984/0 (socket says 172.21.15.97:60984) 2024-03-06T07:35:45.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a8f400640 1 -- 172.21.15.97:0/2528242627 learned_addr learned my addr 172.21.15.97:0/2528242627 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:45.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.017+0000 7f8a8fe00640 1 --2- 172.21.15.97:0/2528242627 >> v2:172.21.15.135:3300/0 conn(0x7f8a9015fe60 0x7f8a900ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.018+0000 7f8a8f400640 1 -- 172.21.15.97:0/2528242627 >> v2:172.21.15.135:3300/0 conn(0x7f8a9015fe60 msgr2=0x7f8a900ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.018+0000 7f8a8f400640 1 --2- 172.21.15.97:0/2528242627 >> v2:172.21.15.135:3300/0 conn(0x7f8a9015fe60 0x7f8a900ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.018+0000 7f8a8f400640 1 -- 172.21.15.97:0/2528242627 >> v1:172.21.15.135:6789/0 conn(0x7f8a9015f9d0 legacy=0x7f8a900ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.018+0000 7f8a8f400640 1 -- 172.21.15.97:0/2528242627 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9006d4b0 con 0x7f8a90144c00 2024-03-06T07:35:45.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.019+0000 7f8a8f400640 1 --2- 172.21.15.97:0/2528242627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a90144c00 0x7f8a9006e930 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8a9006e500 tx=0x7f8a8002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7e94f318d2c877 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a8e000640 1 -- 172.21.15.97:0/2528242627 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80034070 con 0x7f8a90144c00 2024-03-06T07:35:45.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a8e000640 1 -- 172.21.15.97:0/2528242627 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a80030860 con 0x7f8a90144c00 2024-03-06T07:35:45.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a8e000640 1 -- 172.21.15.97:0/2528242627 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80030c30 con 0x7f8a90144c00 2024-03-06T07:35:45.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a96400640 1 -- 172.21.15.97:0/2528242627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a90144c00 msgr2=0x7f8a9006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a96400640 1 --2- 172.21.15.97:0/2528242627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a90144c00 0x7f8a9006e930 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8a9006e500 tx=0x7f8a8002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a96400640 1 -- 172.21.15.97:0/2528242627 shutdown_connections 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a96400640 1 --2- 172.21.15.97:0/2528242627 >> v2:172.21.15.135:3300/0 conn(0x7f8a9015fe60 0x7f8a900ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a96400640 1 --2- 172.21.15.97:0/2528242627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a90144c00 0x7f8a9006e930 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.020+0000 7f8a96400640 1 -- 172.21.15.97:0/2528242627 >> 172.21.15.97:0/2528242627 conn(0x7f8a900b1730 msgr2=0x7f8a900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.021+0000 7f8a96400640 1 -- 172.21.15.97:0/2528242627 shutdown_connections 2024-03-06T07:35:45.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.021+0000 7f8a96400640 1 -- 172.21.15.97:0/2528242627 wait complete. 2024-03-06T07:35:45.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.022+0000 7f8a96400640 1 Processor -- start 2024-03-06T07:35:45.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.022+0000 7f8a96400640 1 -- start start 2024-03-06T07:35:45.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a96400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a90144c00 0x7f8a9015cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a96400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 0x7f8a9013b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a96400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a9015fe60 0x7f8a9015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a96400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90160730 con 0x7f8a9015f9d0 2024-03-06T07:35:45.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a96400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90160270 con 0x7f8a90144c00 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a96400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9006cb50 con 0x7f8a9015fe60 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.023+0000 7f8a8ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 0x7f8a9013b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 0x7f8a9013b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60992/0 (socket says 172.21.15.97:60992) 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 -- 172.21.15.97:0/935731792 learned_addr learned my addr 172.21.15.97:0/935731792 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 -- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a9015fe60 msgr2=0x7f8a9015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8fe00640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a9015fe60 0x7f8a9015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8f400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a90144c00 0x7f8a9015cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a9015fe60 0x7f8a9015df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 -- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a90144c00 msgr2=0x7f8a9015cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a90144c00 0x7f8a9015cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 -- 172.21.15.97:0/935731792 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9015d5a0 con 0x7f8a9015f9d0 2024-03-06T07:35:45.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.024+0000 7f8a8ea00640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 0x7f8a9013b390 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00c400 tx=0x7f8a7c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.025+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c0052d0 con 0x7f8a9015f9d0 2024-03-06T07:35:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.025+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c005490 con 0x7f8a9015f9d0 2024-03-06T07:35:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.025+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c007260 con 0x7f8a9015f9d0 2024-03-06T07:35:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.025+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9015d880 con 0x7f8a9015f9d0 2024-03-06T07:35:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.025+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9013c370 con 0x7f8a9015f9d0 2024-03-06T07:35:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.026+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9004f330 con 0x7f8a9015f9d0 2024-03-06T07:35:45.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.027+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a7c007400 con 0x7f8a9015f9d0 2024-03-06T07:35:45.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.027+0000 7f8a87400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a7006b4d0 0x7f8a7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.028+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a7c0896c0 con 0x7f8a9015f9d0 2024-03-06T07:35:45.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.028+0000 7f8a8f400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a7006b4d0 0x7f8a7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.028+0000 7f8a8f400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a7006b4d0 0x7f8a7006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8a80036000 tx=0x7f8a8003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.032+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a9006ee70 con 0x7f8a9015f9d0 2024-03-06T07:35:45.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.212+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a90072890 con 0x7f8a9015f9d0 2024-03-06T07:35:45.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.215+0000 7f8a87400640 1 -- 172.21.15.97:0/935731792 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f8a7c0585d0 con 0x7f8a9015f9d0 2024-03-06T07:35:45.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:45.218 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":129,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:35:43.820953+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:45.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a7006b4d0 msgr2=0x7f8a7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a7006b4d0 0x7f8a7006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8a80036000 tx=0x7f8a8003b040 comp rx=0 tx=0).stop 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 msgr2=0x7f8a9013b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 0x7f8a9013b390 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00c400 tx=0x7f8a7c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 shutdown_connections 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a7006b4d0 0x7f8a7006d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8a9015fe60 0x7f8a9015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a9015f9d0 0x7f8a9013b390 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 --2- 172.21.15.97:0/935731792 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a90144c00 0x7f8a9015cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.218+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 >> 172.21.15.97:0/935731792 conn(0x7f8a900b1730 msgr2=0x7f8a900b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.219+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 shutdown_connections 2024-03-06T07:35:45.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.219+0000 7f8a96400640 1 -- 172.21.15.97:0/935731792 wait complete. 2024-03-06T07:35:45.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:35:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.448+0000 7f7830200640 1 Processor -- start 2024-03-06T07:35:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.448+0000 7f7830200640 1 -- start start 2024-03-06T07:35:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.449+0000 7f7830200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f78280b3650 0x7f78280727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.449+0000 7f7830200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f78280cda90 0x7f782806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.449+0000 7f7830200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78280719e0 con 0x7f78280b3df0 2024-03-06T07:35:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.449+0000 7f7830200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7828072310 con 0x7f78280b3650 2024-03-06T07:35:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.449+0000 7f7830200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7828071cc0 con 0x7f78280cda90 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f78280b3650 0x7f78280727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f78280b3650 0x7f78280727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32772/0 (socket says 172.21.15.97:32772) 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 -- 172.21.15.97:0/2040864698 learned_addr learned my addr 172.21.15.97:0/2040864698 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782e200640 1 --2- 172.21.15.97:0/2040864698 >> v2:172.21.15.135:3300/0 conn(0x7f78280cda90 0x7f782806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 -- 172.21.15.97:0/2040864698 >> v2:172.21.15.135:3300/0 conn(0x7f78280cda90 msgr2=0x7f782806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 --2- 172.21.15.97:0/2040864698 >> v2:172.21.15.135:3300/0 conn(0x7f78280cda90 0x7f782806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 -- 172.21.15.97:0/2040864698 >> v1:172.21.15.135:6789/0 conn(0x7f78280b3df0 legacy=0x7f78280b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.450+0000 7f782ce00640 1 -- 172.21.15.97:0/2040864698 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7828071fa0 con 0x7f78280b3650 2024-03-06T07:35:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.451+0000 7f782ce00640 1 --2- 172.21.15.97:0/2040864698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f78280727a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f782806ed00 tx=0x7f781402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b511446430d872 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7827e00640 1 -- 172.21.15.97:0/2040864698 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781403b070 con 0x7f78280b3650 2024-03-06T07:35:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7827e00640 1 -- 172.21.15.97:0/2040864698 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7814036440 con 0x7f78280b3650 2024-03-06T07:35:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7827e00640 1 -- 172.21.15.97:0/2040864698 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814036810 con 0x7f78280b3650 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7830200640 1 -- 172.21.15.97:0/2040864698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 msgr2=0x7f78280727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7830200640 1 --2- 172.21.15.97:0/2040864698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f78280727a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f782806ed00 tx=0x7f781402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7830200640 1 -- 172.21.15.97:0/2040864698 shutdown_connections 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7830200640 1 --2- 172.21.15.97:0/2040864698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f78280727a0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7830200640 1 --2- 172.21.15.97:0/2040864698 >> v2:172.21.15.135:3300/0 conn(0x7f78280cda90 0x7f782806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.452+0000 7f7830200640 1 -- 172.21.15.97:0/2040864698 >> 172.21.15.97:0/2040864698 conn(0x7f78280b1940 msgr2=0x7f78280b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.453+0000 7f7830200640 1 -- 172.21.15.97:0/2040864698 shutdown_connections 2024-03-06T07:35:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.453+0000 7f7830200640 1 -- 172.21.15.97:0/2040864698 wait complete. 2024-03-06T07:35:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.455+0000 7f7830200640 1 Processor -- start 2024-03-06T07:35:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.455+0000 7f7830200640 1 -- start start 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f7830200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f782809d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f7830200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78280b3df0 0x7f782814f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f7830200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78280cda90 0x7f7828155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f7830200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782815b3c0 con 0x7f78280b3650 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f7830200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782815b240 con 0x7f78280cda90 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f7830200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78280cf510 con 0x7f78280b3df0 2024-03-06T07:35:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f782809d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f782809d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32782/0 (socket says 172.21.15.97:32782) 2024-03-06T07:35:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 -- 172.21.15.97:0/236834535 learned_addr learned my addr 172.21.15.97:0/236834535 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 -- 172.21.15.97:0/236834535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78280b3df0 msgr2=0x7f782814f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78280b3df0 0x7f782814f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 -- 172.21.15.97:0/236834535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78280cda90 msgr2=0x7f7828155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78280cda90 0x7f7828155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.457+0000 7f782d800640 1 -- 172.21.15.97:0/236834535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7828150320 con 0x7f78280b3650 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.458+0000 7f782d800640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f782809d190 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f781c00b420 tx=0x7f781c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.458+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c011830 con 0x7f78280b3650 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.458+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c016070 con 0x7f78280b3650 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.458+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c011c10 con 0x7f78280b3650 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.458+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7828150600 con 0x7f78280b3650 2024-03-06T07:35:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.458+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7828150bc0 con 0x7f78280b3650 2024-03-06T07:35:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.459+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782804f330 con 0x7f78280b3650 2024-03-06T07:35:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.460+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f781c01b460 con 0x7f78280b3650 2024-03-06T07:35:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.461+0000 7f7825600640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f780806b4d0 0x7f780806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.461+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f781c089d30 con 0x7f78280b3650 2024-03-06T07:35:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.462+0000 7f782ce00640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f780806b4d0 0x7f780806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.462+0000 7f782ce00640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f780806b4d0 0x7f780806d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7814038000 tx=0x7f7814030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.470+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f781c020020 con 0x7f78280b3650 2024-03-06T07:35:45.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:45.696+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7828074cc0 con 0x7f78280b3650 2024-03-06T07:35:46.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.406+0000 7f7825600640 1 -- 172.21.15.97:0/236834535 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) v1 ==== 118+0+0 (secure 0 0 0) 0x7f781c058ba0 con 0x7f78280b3650 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f780806b4d0 msgr2=0x7f780806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f780806b4d0 0x7f780806d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7814038000 tx=0x7f7814030000 comp rx=0 tx=0).stop 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 msgr2=0x7f782809d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f782809d190 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f781c00b420 tx=0x7f781c00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 shutdown_connections 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f780806b4d0 0x7f780806d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78280cda90 0x7f7828155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f78280b3df0 0x7f782814f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 --2- 172.21.15.97:0/236834535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78280b3650 0x7f782809d190 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.409+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 >> 172.21.15.97:0/236834535 conn(0x7f78280b1940 msgr2=0x7f78280b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.410+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 shutdown_connections 2024-03-06T07:35:46.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.410+0000 7f7830200640 1 -- 172.21.15.97:0/236834535 wait complete. 2024-03-06T07:35:46.434 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:35:46.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:35:46.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.646+0000 7ff528a00640 1 Processor -- start 2024-03-06T07:35:46.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.646+0000 7ff528a00640 1 -- start start 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff528a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff52415e390 0x7ff524066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff528a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff524072c40 0x7ff5240cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff528a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52406ba70 con 0x7ff52406c200 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff528a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52406e9c0 con 0x7ff52415e390 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff528a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52406ebf0 con 0x7ff524072c40 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff523400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff52415e390 0x7ff524066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff523400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff52415e390 0x7ff524066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32786/0 (socket says 172.21.15.97:32786) 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff522a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff52406c200 0x7ff524072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50158/0 (socket says 172.21.15.97:50158) 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.648+0000 7ff523400640 1 -- 172.21.15.97:0/2833544532 learned_addr learned my addr 172.21.15.97:0/2833544532 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.649+0000 7ff523e00640 1 --2- 172.21.15.97:0/2833544532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff524072c40 0x7ff5240cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.649+0000 7ff523400640 1 -- 172.21.15.97:0/2833544532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff524072c40 msgr2=0x7ff5240cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.649+0000 7ff523400640 1 --2- 172.21.15.97:0/2833544532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff524072c40 0x7ff5240cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.649+0000 7ff522000640 1 -- 172.21.15.97:0/2833544532 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778778541 0 0) 0x7ff52406ba70 con 0x7ff52406c200 2024-03-06T07:35:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.649+0000 7ff523400640 1 -- 172.21.15.97:0/2833544532 >> v1:172.21.15.97:6789/0 conn(0x7ff52406c200 legacy=0x7ff524072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.649+0000 7ff523400640 1 -- 172.21.15.97:0/2833544532 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52406eed0 con 0x7ff52415e390 2024-03-06T07:35:46.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.650+0000 7ff523400640 1 --2- 172.21.15.97:0/2833544532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff52415e390 0x7ff524066d00 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7ff51400bb10 tx=0x7ff51402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364b53bebda1ad34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.650+0000 7ff522000640 1 -- 172.21.15.97:0/2833544532 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51403b070 con 0x7ff52415e390 2024-03-06T07:35:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.650+0000 7ff522000640 1 -- 172.21.15.97:0/2833544532 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff514036440 con 0x7ff52415e390 2024-03-06T07:35:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.650+0000 7ff522000640 1 -- 172.21.15.97:0/2833544532 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514036810 con 0x7ff52415e390 2024-03-06T07:35:46.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 -- 172.21.15.97:0/2833544532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff52415e390 msgr2=0x7ff524066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 --2- 172.21.15.97:0/2833544532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff52415e390 0x7ff524066d00 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7ff51400bb10 tx=0x7ff51402e150 comp rx=0 tx=0).stop 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 -- 172.21.15.97:0/2833544532 shutdown_connections 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 --2- 172.21.15.97:0/2833544532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff524072c40 0x7ff5240cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 --2- 172.21.15.97:0/2833544532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff52415e390 0x7ff524066d00 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 -- 172.21.15.97:0/2833544532 >> 172.21.15.97:0/2833544532 conn(0x7ff5240b1d00 msgr2=0x7ff5240b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 -- 172.21.15.97:0/2833544532 shutdown_connections 2024-03-06T07:35:46.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.651+0000 7ff528a00640 1 -- 172.21.15.97:0/2833544532 wait complete. 2024-03-06T07:35:46.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.654+0000 7ff528a00640 1 Processor -- start 2024-03-06T07:35:46.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.654+0000 7ff528a00640 1 -- start start 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff528a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52406c200 0x7ff5240c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff528a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 0x7ff52414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff528a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff52415e390 0x7ff5240ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff528a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240673c0 con 0x7ff524072c40 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff528a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff524067240 con 0x7ff52415e390 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff528a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240b2a20 con 0x7ff52406c200 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 0x7ff52414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 0x7ff52414f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32798/0 (socket says 172.21.15.97:32798) 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 -- 172.21.15.97:0/3793738999 learned_addr learned my addr 172.21.15.97:0/3793738999 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff523400640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52406c200 0x7ff5240c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff523e00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff52415e390 0x7ff5240ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 -- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52406c200 msgr2=0x7ff5240c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52406c200 0x7ff5240c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 -- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff52415e390 msgr2=0x7ff5240ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff52415e390 0x7ff5240ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff522a00640 1 -- 172.21.15.97:0/3793738999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5240c7cf0 con 0x7ff524072c40 2024-03-06T07:35:46.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff523400640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52406c200 0x7ff5240c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:46.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.656+0000 7ff523e00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff52415e390 0x7ff5240ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:46.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.657+0000 7ff522a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 0x7ff52414f010 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c460 tx=0x7ff51000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.657+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510018070 con 0x7ff524072c40 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.657+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff510007500 con 0x7ff524072c40 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.657+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510004a20 con 0x7ff524072c40 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.657+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5240c7fd0 con 0x7ff524072c40 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.657+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5240c8ab0 con 0x7ff524072c40 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.658+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52404f330 con 0x7ff524072c40 2024-03-06T07:35:46.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.660+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff510004bc0 con 0x7ff524072c40 2024-03-06T07:35:46.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.660+0000 7ff51b400640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50406b4d0 0x7ff50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.660+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff510089580 con 0x7ff524072c40 2024-03-06T07:35:46.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.661+0000 7ff523400640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50406b4d0 0x7ff50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.661+0000 7ff523400640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50406b4d0 0x7ff50406d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff514038000 tx=0x7ff514008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.665+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff510058400 con 0x7ff524072c40 2024-03-06T07:35:46.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:46.923+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff524071400 con 0x7ff524072c40 2024-03-06T07:35:47.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.419+0000 7ff51b400640 1 -- 172.21.15.97:0/3793738999 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff510057d00 con 0x7ff524072c40 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.423+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50406b4d0 msgr2=0x7ff50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.423+0000 7ff528a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50406b4d0 0x7ff50406d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff514038000 tx=0x7ff514008040 comp rx=0 tx=0).stop 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.423+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 msgr2=0x7ff52414f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.423+0000 7ff528a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 0x7ff52414f010 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c460 tx=0x7ff51000c920 comp rx=0 tx=0).stop 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 shutdown_connections 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff50406b4d0 0x7ff50406d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff52415e390 0x7ff5240ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff524072c40 0x7ff52414f010 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 --2- 172.21.15.97:0/3793738999 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff52406c200 0x7ff5240c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 >> 172.21.15.97:0/3793738999 conn(0x7ff5240b1d00 msgr2=0x7ff5240cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 shutdown_connections 2024-03-06T07:35:47.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.424+0000 7ff528a00640 1 -- 172.21.15.97:0/3793738999 wait complete. 2024-03-06T07:35:47.446 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:35:47.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.646+0000 7fa9e3400640 1 Processor -- start 2024-03-06T07:35:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.646+0000 7fa9e3400640 1 -- start start 2024-03-06T07:35:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.648+0000 7fa9e3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.648+0000 7fa9e3400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa9dc070b10 0x7fa9dc070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.648+0000 7fa9e3400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc06cef0 con 0x7fa9dc144c00 2024-03-06T07:35:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.648+0000 7fa9e3400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc160370 con 0x7fa9dc070b10 2024-03-06T07:35:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.648+0000 7fa9e3400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc06d1d0 con 0x7fa9dc15f9d0 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.649+0000 7fa9e1400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa9dc070b10 0x7fa9dc070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.649+0000 7fa9d3e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.649+0000 7fa9e1400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa9dc070b10 0x7fa9dc070f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:40446/0 (socket says 172.21.15.97:40446) 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.649+0000 7fa9e1400640 1 -- 172.21.15.97:0/3630069580 learned_addr learned my addr 172.21.15.97:0/3630069580 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.649+0000 7fa9e1400640 1 -- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 msgr2=0x7fa9dc15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.649+0000 7fa9e1400640 1 --2- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.650+0000 7fa9e1400640 1 -- 172.21.15.97:0/3630069580 >> v1:172.21.15.135:6789/0 conn(0x7fa9dc144c00 legacy=0x7fa9dc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.650+0000 7fa9e1400640 1 -- 172.21.15.97:0/3630069580 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc06d4b0 con 0x7fa9dc070b10 2024-03-06T07:35:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.650+0000 7fa9dbe00640 1 -- 172.21.15.97:0/3630069580 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454743197 0 0) 0x7fa9dc06cef0 con 0x7fa9dc144c00 2024-03-06T07:35:47.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.651+0000 7fa9e1400640 1 --2- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc070b10 0x7fa9dc070f60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009e50 tx=0x7fa9d402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e7cad4fdce55e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.651+0000 7fa9dbe00640 1 -- 172.21.15.97:0/3630069580 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d402cb00 con 0x7fa9dc070b10 2024-03-06T07:35:47.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.651+0000 7fa9dbe00640 1 -- 172.21.15.97:0/3630069580 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d402cca0 con 0x7fa9dc070b10 2024-03-06T07:35:47.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.651+0000 7fa9dbe00640 1 -- 172.21.15.97:0/3630069580 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4034610 con 0x7fa9dc070b10 2024-03-06T07:35:47.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 -- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc070b10 msgr2=0x7fa9dc070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc070b10 0x7fa9dc070f60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009e50 tx=0x7fa9d402bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:47.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 -- 172.21.15.97:0/3630069580 shutdown_connections 2024-03-06T07:35:47.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3630069580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc070b10 0x7fa9dc070f60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 -- 172.21.15.97:0/3630069580 >> 172.21.15.97:0/3630069580 conn(0x7fa9dc0b1730 msgr2=0x7fa9dc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 -- 172.21.15.97:0/3630069580 shutdown_connections 2024-03-06T07:35:47.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.652+0000 7fa9e3400640 1 -- 172.21.15.97:0/3630069580 wait complete. 2024-03-06T07:35:47.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.655+0000 7fa9e3400640 1 Processor -- start 2024-03-06T07:35:47.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.655+0000 7fa9e3400640 1 -- start start 2024-03-06T07:35:47.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.656+0000 7fa9e3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 0x7fa9dc14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e3400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc144c00 0x7fa9dc13e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e3400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc06cb50 con 0x7fa9dc070b10 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e3400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc160730 con 0x7fa9dc144c00 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e3400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc06ccd0 con 0x7fa9dc15f9d0 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 0x7fa9dc14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 0x7fa9dc14e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32814/0 (socket says 172.21.15.97:32814) 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 -- 172.21.15.97:0/3561484483 learned_addr learned my addr 172.21.15.97:0/3561484483 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9d3e00640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc144c00 0x7fa9dc13e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e1400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 -- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 msgr2=0x7fa9dc15a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 -- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc144c00 msgr2=0x7fa9dc13e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc144c00 0x7fa9dc13e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e0a00640 1 -- 172.21.15.97:0/3561484483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc13eb40 con 0x7fa9dc070b10 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.657+0000 7fa9e1400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.658+0000 7fa9e0a00640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 0x7fa9dc14e7a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c460 tx=0x7fa9cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.658+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc005480 con 0x7fa9dc070b10 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.658+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc0035f0 con 0x7fa9dc070b10 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.658+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc139c30 con 0x7fa9dc070b10 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.658+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc003a00 con 0x7fa9dc070b10 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.658+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc13a4d0 con 0x7fa9dc070b10 2024-03-06T07:35:47.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.659+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc04f330 con 0x7fa9dc070b10 2024-03-06T07:35:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.661+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9cc003ba0 con 0x7fa9dc070b10 2024-03-06T07:35:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.662+0000 7fa9d9600640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9b806b670 0x7fa9b806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.662+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9cc089ed0 con 0x7fa9dc070b10 2024-03-06T07:35:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.662+0000 7fa9d3e00640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9b806b670 0x7fa9b806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.663+0000 7fa9d3e00640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9b806b670 0x7fa9b806db30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0b05c0 tx=0x7fa9c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.667+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9cc058d50 con 0x7fa9dc070b10 2024-03-06T07:35:47.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.900+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9dc139e00 con 0x7fa9dc070b10 2024-03-06T07:35:47.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.901+0000 7fa9d9600640 1 -- 172.21.15.97:0/3561484483 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fa9cc058650 con 0x7fa9dc070b10 2024-03-06T07:35:47.903 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:47.903 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1847010599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1847010599},{"type":"v1","addr":"172.21.15.135:6833","nonce":1847010599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6175,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/3157120524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":3157120524},{"type":"v1","addr":"172.21.15.135:6835","nonce":3157120524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:46.925306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:35:47.903 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 88 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9b806b670 msgr2=0x7fa9b806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9b806b670 0x7fa9b806db30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0b05c0 tx=0x7fa9c4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 msgr2=0x7fa9dc14e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 0x7fa9dc14e7a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c460 tx=0x7fa9cc00c920 comp rx=0 tx=0).stop 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 shutdown_connections 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa9b806b670 0x7fa9b806db30 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa9dc15f9d0 0x7fa9dc15a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9dc144c00 0x7fa9dc13e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 --2- 172.21.15.97:0/3561484483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9dc070b10 0x7fa9dc14e7a0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.904+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 >> 172.21.15.97:0/3561484483 conn(0x7fa9dc0b1730 msgr2=0x7fa9dc0b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.905+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 shutdown_connections 2024-03-06T07:35:47.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:47.905+0000 7fa9e3400640 1 -- 172.21.15.97:0/3561484483 wait complete. 2024-03-06T07:35:47.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:48.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.122+0000 7f8bdb600640 1 Processor -- start 2024-03-06T07:35:48.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.123+0000 7f8bdb600640 1 -- start start 2024-03-06T07:35:48.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bdb600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8bd415f9d0 0x7f8bd415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bdb600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bd4070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bdb600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd406cef0 con 0x7f8bd4144c00 2024-03-06T07:35:48.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bdb600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd406d1d0 con 0x7f8bd415f9d0 2024-03-06T07:35:48.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bdb600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd406d4b0 con 0x7f8bd4070b10 2024-03-06T07:35:48.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bd8c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8bd4144c00 0x7f8bd406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50166/0 (socket says 172.21.15.97:50166) 2024-03-06T07:35:48.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.124+0000 7f8bd8c00640 1 -- 172.21.15.97:0/2062507873 learned_addr learned my addr 172.21.15.97:0/2062507873 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd9600640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bd4070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2062507873 >> v2:172.21.15.135:3300/0 conn(0x7f8bd415f9d0 0x7f8bd415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664253296 0 0) 0x7f8bd406cef0 con 0x7f8bd4144c00 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bd4144c00 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519587413 0 0) 0x7f8bb8003520 con 0x7f8bd4144c00 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 msgr2=0x7f8bd4070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bd4070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 >> v2:172.21.15.135:3300/0 conn(0x7f8bd415f9d0 msgr2=0x7f8bd415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 --2- 172.21.15.97:0/2062507873 >> v2:172.21.15.135:3300/0 conn(0x7f8bd415f9d0 0x7f8bd415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd406d790 con 0x7f8bd4144c00 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8bc4001670 con 0x7f8bd4144c00 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 >> v1:172.21.15.97:6789/0 conn(0x7f8bd4144c00 legacy=0x7f8bd406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.125+0000 7f8bd3e00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bb80039b0 0x7f8bb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd3e00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bb8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd3e00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bb8004d40 0x7f8bb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd406d1d0 con 0x7f8bb8004d40 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd406d4b0 con 0x7f8bb80039b0 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb800b9c0 con 0x7f8bd4070b10 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd9600640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bb8004d40 0x7f8bb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd8c00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bb80039b0 0x7f8bb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd9600640 1 -- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 msgr2=0x7f8bb8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bb8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd9600640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bb8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd9600640 1 -- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bb80039b0 msgr2=0x7f8bb8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd9600640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bb80039b0 0x7f8bb8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd9600640 1 -- 172.21.15.97:0/2062507873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8003520 con 0x7f8bb8004d40 2024-03-06T07:35:48.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bd8c00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bb80039b0 0x7f8bb8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:48.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.126+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd4070b10 0x7f8bb8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:48.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.127+0000 7f8bd9600640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bb8004d40 0x7f8bb80051c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009fc0 tx=0x7f8bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb56db75b96bf187 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc039070 con 0x7f8bb8004d40 2024-03-06T07:35:48.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc02cbf0 con 0x7f8bb8004d40 2024-03-06T07:35:48.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bd3e00640 1 -- 172.21.15.97:0/2062507873 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc009c10 con 0x7f8bb8004d40 2024-03-06T07:35:48.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bdb600640 1 -- 172.21.15.97:0/2062507873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bb8004d40 msgr2=0x7f8bb80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2062507873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bb8004d40 0x7f8bb80051c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009fc0 tx=0x7f8bb8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bd9600640 1 -- 172.21.15.97:0/2062507873 reap_dead start 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bdb600640 1 -- 172.21.15.97:0/2062507873 shutdown_connections 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bdb600640 1 -- 172.21.15.97:0/2062507873 >> 172.21.15.97:0/2062507873 conn(0x7f8bd40b1730 msgr2=0x7f8bd40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bdb600640 1 -- 172.21.15.97:0/2062507873 shutdown_connections 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.128+0000 7f8bdb600640 1 -- 172.21.15.97:0/2062507873 wait complete. 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.130+0000 7f8bdb600640 1 Processor -- start 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.130+0000 7f8bdb600640 1 -- start start 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bdb600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 0x7f8bd4140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bdb600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd4141410 0x7f8bd414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bdb600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd413fb60 0x7f8bd414e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bdb600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd406ccd0 con 0x7f8bd413f780 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bdb600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd406cb50 con 0x7f8bd4141410 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bdb600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd406df90 con 0x7f8bd413fb60 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bd8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 0x7f8bd4140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bd8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 0x7f8bd4140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32840/0 (socket says 172.21.15.97:32840) 2024-03-06T07:35:48.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.131+0000 7f8bd8c00640 1 -- 172.21.15.97:0/2903192710 learned_addr learned my addr 172.21.15.97:0/2903192710 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd4141410 0x7f8bd414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd8c00640 1 -- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd413fb60 msgr2=0x7f8bd414e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd9600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd413fb60 0x7f8bd414e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd8c00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd413fb60 0x7f8bd414e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd8c00640 1 -- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd4141410 msgr2=0x7f8bd414df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd8c00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd4141410 0x7f8bd414df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd8c00640 1 -- 172.21.15.97:0/2903192710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd414e9f0 con 0x7f8bd413f780 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd4141410 0x7f8bd414df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd8c00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 0x7f8bd4140ed0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400dae0 tx=0x7f8bc4001700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4019070 con 0x7f8bd413f780 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc4004cb0 con 0x7f8bd413f780 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd41f20f0 con 0x7f8bd413f780 2024-03-06T07:35:48.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.132+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4004e60 con 0x7f8bd413f780 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.133+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd4141c70 con 0x7f8bd413f780 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.134+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd404f330 con 0x7f8bd413f780 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.135+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bc401c070 con 0x7f8bd413f780 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.136+0000 7f8bd1600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bb006b4d0 0x7f8bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.136+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bc4089ef0 con 0x7f8bd413f780 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.136+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bb006b4d0 0x7f8bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.137+0000 7f8bcbe00640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bb006b4d0 0x7f8bb006d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8bd41422d0 tx=0x7f8bbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.141+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bc4058c40 con 0x7f8bd413f780 2024-03-06T07:35:48.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.340+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bd4072890 con 0x7f8bd413f780 2024-03-06T07:35:48.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.341+0000 7f8bd1600640 1 -- 172.21.15.97:0/2903192710 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16667 (secure 0 0 0) 0x7f8bc4058540 con 0x7f8bd413f780 2024-03-06T07:35:48.344 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:48.344 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":130,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:35:45.381910+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:48.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.343+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bb006b4d0 msgr2=0x7f8bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.343+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bb006b4d0 0x7f8bb006d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8bd41422d0 tx=0x7f8bbc008040 comp rx=0 tx=0).stop 2024-03-06T07:35:48.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.343+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 msgr2=0x7f8bd4140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.343+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 0x7f8bd4140ed0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400dae0 tx=0x7f8bc4001700 comp rx=0 tx=0).stop 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 shutdown_connections 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8bb006b4d0 0x7f8bb006d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bd413fb60 0x7f8bd414e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd4141410 0x7f8bd414df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 --2- 172.21.15.97:0/2903192710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bd413f780 0x7f8bd4140ed0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 >> 172.21.15.97:0/2903192710 conn(0x7f8bd40b1730 msgr2=0x7f8bd406e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 shutdown_connections 2024-03-06T07:35:48.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.344+0000 7f8bdb600640 1 -- 172.21.15.97:0/2903192710 wait complete. 2024-03-06T07:35:48.366 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:35:48.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:48.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.567+0000 7f8130c00640 1 Processor -- start 2024-03-06T07:35:48.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.568+0000 7f8130c00640 1 -- start start 2024-03-06T07:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.570+0000 7f8130c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f812c151660 0x7f812c071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.570+0000 7f8130c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f812c0cf430 0x7f812c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.570+0000 7f8130c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c071a60 con 0x7f812c06d920 2024-03-06T07:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.570+0000 7f8130c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c0b3ee0 con 0x7f812c151660 2024-03-06T07:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.570+0000 7f8130c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c0b35b0 con 0x7f812c0cf430 2024-03-06T07:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f812c151660 0x7f812c071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f812c151660 0x7f812c071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32846/0 (socket says 172.21.15.97:32846) 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 -- 172.21.15.97:0/3679862505 learned_addr learned my addr 172.21.15.97:0/3679862505 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f812a800640 1 --1- 172.21.15.97:0/3679862505 >> v1:172.21.15.97:6789/0 conn(0x7f812c06d920 0x7f812c15dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50170/0 (socket says 172.21.15.97:50170) 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8121400640 1 --2- 172.21.15.97:0/3679862505 >> v2:172.21.15.135:3300/0 conn(0x7f812c0cf430 0x7f812c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 -- 172.21.15.97:0/3679862505 >> v2:172.21.15.135:3300/0 conn(0x7f812c0cf430 msgr2=0x7f812c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 --2- 172.21.15.97:0/3679862505 >> v2:172.21.15.135:3300/0 conn(0x7f812c0cf430 0x7f812c06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 -- 172.21.15.97:0/3679862505 >> v1:172.21.15.97:6789/0 conn(0x7f812c06d920 legacy=0x7f812c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.571+0000 7f8129e00640 1 -- 172.21.15.97:0/3679862505 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c0b3890 con 0x7f812c151660 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.572+0000 7f8129e00640 1 --2- 172.21.15.97:0/3679862505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c151660 0x7f812c071520 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f812c06c8e0 tx=0x7f811c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74280b7591722fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8129400640 1 -- 172.21.15.97:0/3679862505 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c02e070 con 0x7f812c151660 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8129400640 1 -- 172.21.15.97:0/3679862505 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c037070 con 0x7f812c151660 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8129400640 1 -- 172.21.15.97:0/3679862505 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c031a00 con 0x7f812c151660 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 -- 172.21.15.97:0/3679862505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c151660 msgr2=0x7f812c071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 --2- 172.21.15.97:0/3679862505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c151660 0x7f812c071520 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f812c06c8e0 tx=0x7f811c02b880 comp rx=0 tx=0).stop 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 -- 172.21.15.97:0/3679862505 shutdown_connections 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 --2- 172.21.15.97:0/3679862505 >> v2:172.21.15.135:3300/0 conn(0x7f812c0cf430 0x7f812c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 --2- 172.21.15.97:0/3679862505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c151660 0x7f812c071520 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 -- 172.21.15.97:0/3679862505 >> 172.21.15.97:0/3679862505 conn(0x7f812c0b1730 msgr2=0x7f812c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 -- 172.21.15.97:0/3679862505 shutdown_connections 2024-03-06T07:35:48.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.573+0000 7f8130c00640 1 -- 172.21.15.97:0/3679862505 wait complete. 2024-03-06T07:35:48.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.575+0000 7f8130c00640 1 Processor -- start 2024-03-06T07:35:48.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.575+0000 7f8130c00640 1 -- start start 2024-03-06T07:35:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8130c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c06d920 0x7f812c14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8130c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 0x7f812c149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8130c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c151660 0x7f812c14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8130c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c15e700 con 0x7f812c0cf430 2024-03-06T07:35:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8130c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c071d30 con 0x7f812c151660 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8130c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c15e880 con 0x7f812c06d920 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 0x7f812c149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 0x7f812c149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32854/0 (socket says 172.21.15.97:32854) 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 -- 172.21.15.97:0/312399952 learned_addr learned my addr 172.21.15.97:0/312399952 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8129e00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c06d920 0x7f812c14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 -- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c06d920 msgr2=0x7f812c14fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c06d920 0x7f812c14fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 -- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c151660 msgr2=0x7f812c14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c151660 0x7f812c14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.577+0000 7f8121400640 1 -- 172.21.15.97:0/312399952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c0ceae0 con 0x7f812c0cf430 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.578+0000 7f8121400640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 0x7f812c149d80 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f811400d5f0 tx=0x7f811400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.578+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81140125f0 con 0x7f812c0cf430 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.578+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8114019070 con 0x7f812c0cf430 2024-03-06T07:35:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.578+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c0cc7e0 con 0x7f812c0cf430 2024-03-06T07:35:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.578+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81140129d0 con 0x7f812c0cf430 2024-03-06T07:35:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.578+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c0cd080 con 0x7f812c0cf430 2024-03-06T07:35:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.579+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812c04f330 con 0x7f812c0cf430 2024-03-06T07:35:48.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.581+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8114012b70 con 0x7f812c0cf430 2024-03-06T07:35:48.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.582+0000 7f8122a00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f810806b500 0x7f810806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.582+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f811408a600 con 0x7f812c0cf430 2024-03-06T07:35:48.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.582+0000 7f8129e00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f810806b500 0x7f810806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.583+0000 7f8129e00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f810806b500 0x7f810806d9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f811c036000 tx=0x7f811c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.587+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8114059480 con 0x7f812c0cf430 2024-03-06T07:35:48.742 INFO:tasks.ceph.mds.a.smithi097.stderr:2024-03-06T07:35:48.740+0000 7fa3d5000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.851+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f812c15a110 con 0x7f810806b500 2024-03-06T07:35:48.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.861+0000 7f8122a00640 1 -- 172.21.15.97:0/312399952 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247090 (secure 0 0 0) 0x7f812c15a110 con 0x7f810806b500 2024-03-06T07:35:48.867 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:48.869 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:35:48.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.867+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f810806b500 msgr2=0x7f810806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.867+0000 7f8130c00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f810806b500 0x7f810806d9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f811c036000 tx=0x7f811c008030 comp rx=0 tx=0).stop 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.867+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 msgr2=0x7f812c149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.867+0000 7f8130c00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 0x7f812c149d80 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f811400d5f0 tx=0x7f811400dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 shutdown_connections 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f810806b500 0x7f810806d9c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f812c151660 0x7f812c14a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812c0cf430 0x7f812c149d80 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 --2- 172.21.15.97:0/312399952 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f812c06d920 0x7f812c14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 >> 172.21.15.97:0/312399952 conn(0x7f812c0b1730 msgr2=0x7f812c06e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 shutdown_connections 2024-03-06T07:35:48.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:48.868+0000 7f8130c00640 1 -- 172.21.15.97:0/312399952 wait complete. 2024-03-06T07:35:48.889 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-03-06T07:35:47.051327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304624,"kb_used_data":13572,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670096,"statfs":{"total":773094113280,"available":772782178304,"internally_reserved":0,"allocated":13897728,"data_stored":6690687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.708634"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353207+0000","last_change":"2024-03-06T07:35:43.353207+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353207+0000","last_undegraded":"2024-03-06T07:35:43.353207+0000","last_fullsized":"2024-03-06T07:35:43.353207+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.361902+0000","last_change":"2024-03-06T07:35:43.361902+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.361902+0000","last_undegraded":"2024-03-06T07:35:43.361902+0000","last_fullsized":"2024-03-06T07:35:43.361902+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.369168+0000","last_change":"2024-03-06T07:35:43.369168+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.369168+0000","last_undegraded":"2024-03-06T07:35:43.369168+0000","last_fullsized":"2024-03-06T07:35:43.369168+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354029+0000","last_change":"2024-03-06T07:35:43.354029+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354029+0000","last_undegraded":"2024-03-06T07:35:43.354029+0000","last_fullsized":"2024-03-06T07:35:43.354029+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.363897+0000","last_change":"2024-03-06T07:35:43.363897+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.363897+0000","last_undegraded":"2024-03-06T07:35:43.363897+0000","last_fullsized":"2024-03-06T07:35:43.363897+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354250+0000","last_change":"2024-03-06T07:35:43.354250+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354250+0000","last_undegraded":"2024-03-06T07:35:43.354250+0000","last_fullsized":"2024-03-06T07:35:43.354250+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353927+0000","last_change":"2024-03-06T07:35:43.353927+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353927+0000","last_undegraded":"2024-03-06T07:35:43.353927+0000","last_fullsized":"2024-03-06T07:35:43.353927+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.366431+0000","last_change":"2024-03-06T07:35:43.366431+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.366431+0000","last_undegraded":"2024-03-06T07:35:43.366431+0000","last_fullsized":"2024-03-06T07:35:43.366431+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.371435+0000","last_change":"2024-03-06T07:35:43.371435+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.371435+0000","last_undegraded":"2024-03-06T07:35:43.371435+0000","last_fullsized":"2024-03-06T07:35:43.371435+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.359154+0000","last_change":"2024-03-06T07:35:43.359154+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.359154+0000","last_undegraded":"2024-03-06T07:35:43.359154+0000","last_fullsized":"2024-03-06T07:35:43.359154+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.362824+0000","last_change":"2024-03-06T07:35:43.362824+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.362824+0000","last_undegraded":"2024-03-06T07:35:43.362824+0000","last_fullsized":"2024-03-06T07:35:43.362824+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.368343+0000","last_change":"2024-03-06T07:35:43.368343+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.368343+0000","last_undegraded":"2024-03-06T07:35:43.368343+0000","last_fullsized":"2024-03-06T07:35:43.368343+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356777+0000","last_change":"2024-03-06T07:35:43.356777+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356777+0000","last_undegraded":"2024-03-06T07:35:43.356777+0000","last_fullsized":"2024-03-06T07:35:43.356777+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.359490+0000","last_change":"2024-03-06T07:35:43.359490+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.359490+0000","last_undegraded":"2024-03-06T07:35:43.359490+0000","last_fullsized":"2024-03-06T07:35:43.359490+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353935+0000","last_change":"2024-03-06T07:35:43.353935+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353935+0000","last_undegraded":"2024-03-06T07:35:43.353935+0000","last_fullsized":"2024-03-06T07:35:43.353935+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.365634+0000","last_change":"2024-03-06T07:35:43.365634+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.365634+0000","last_undegraded":"2024-03-06T07:35:43.365634+0000","last_fullsized":"2024-03-06T07:35:43.365634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356595+0000","last_change":"2024-03-06T07:35:43.356595+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356595+0000","last_undegraded":"2024-03-06T07:35:43.356595+0000","last_fullsized":"2024-03-06T07:35:43.356595+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.371053+0000","last_change":"2024-03-06T07:35:43.371053+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.371053+0000","last_undegraded":"2024-03-06T07:35:43.371053+0000","last_fullsized":"2024-03-06T07:35:43.371053+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.361969+0000","last_change":"2024-03-06T07:35:43.361969+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.361969+0000","last_undegraded":"2024-03-06T07:35:43.361969+0000","last_fullsized":"2024-03-06T07:35:43.361969+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.369725+0000","last_change":"2024-03-06T07:35:43.369725+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.369725+0000","last_undegraded":"2024-03-06T07:35:43.369725+0000","last_fullsized":"2024-03-06T07:35:43.369725+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.369175+0000","last_change":"2024-03-06T07:35:43.369175+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.369175+0000","last_undegraded":"2024-03-06T07:35:43.369175+0000","last_fullsized":"2024-03-06T07:35:43.369175+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.359308+0000","last_change":"2024-03-06T07:35:43.359308+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.359308+0000","last_undegraded":"2024-03-06T07:35:43.359308+0000","last_fullsized":"2024-03-06T07:35:43.359308+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.364512+0000","last_change":"2024-03-06T07:35:43.364512+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.364512+0000","last_undegraded":"2024-03-06T07:35:43.364512+0000","last_fullsized":"2024-03-06T07:35:43.364512+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.361266+0000","last_change":"2024-03-06T07:35:43.361266+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.361266+0000","last_undegraded":"2024-03-06T07:35:43.361266+0000","last_fullsized":"2024-03-06T07:35:43.361266+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.359006+0000","last_change":"2024-03-06T07:35:43.359006+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.359006+0000","last_undegraded":"2024-03-06T07:35:43.359006+0000","last_fullsized":"2024-03-06T07:35:43.359006+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.367137+0000","last_change":"2024-03-06T07:35:43.367137+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.367137+0000","last_undegraded":"2024-03-06T07:35:43.367137+0000","last_fullsized":"2024-03-06T07:35:43.367137+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354301+0000","last_change":"2024-03-06T07:35:43.354301+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354301+0000","last_undegraded":"2024-03-06T07:35:43.354301+0000","last_fullsized":"2024-03-06T07:35:43.354301+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.365324+0000","last_change":"2024-03-06T07:35:43.365324+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.365324+0000","last_undegraded":"2024-03-06T07:35:43.365324+0000","last_fullsized":"2024-03-06T07:35:43.365324+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.355730+0000","last_change":"2024-03-06T07:35:43.355730+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.355730+0000","last_undegraded":"2024-03-06T07:35:43.355730+0000","last_fullsized":"2024-03-06T07:35:43.355730+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353746+0000","last_change":"2024-03-06T07:35:43.353746+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353746+0000","last_undegraded":"2024-03-06T07:35:43.353746+0000","last_fullsized":"2024-03-06T07:35:43.353746+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.365374+0000","last_change":"2024-03-06T07:35:43.365374+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.365374+0000","last_undegraded":"2024-03-06T07:35:43.365374+0000","last_fullsized":"2024-03-06T07:35:43.365374+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353912+0000","last_change":"2024-03-06T07:35:43.353912+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353912+0000","last_undegraded":"2024-03-06T07:35:43.353912+0000","last_fullsized":"2024-03-06T07:35:43.353912+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.366702+0000","last_change":"2024-03-06T07:35:43.366702+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.366702+0000","last_undegraded":"2024-03-06T07:35:43.366702+0000","last_fullsized":"2024-03-06T07:35:43.366702+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.386548+0000","last_change":"2024-03-06T07:35:43.384588+0000","last_active":"2024-03-06T07:35:43.386548+0000","last_peered":"2024-03-06T07:35:43.386548+0000","last_clean":"2024-03-06T07:35:43.386548+0000","last_became_active":"2024-03-06T07:35:43.384278+0000","last_became_peered":"2024-03-06T07:35:43.384278+0000","last_unstale":"2024-03-06T07:35:43.386548+0000","last_undegraded":"2024-03-06T07:35:43.386548+0000","last_fullsized":"2024-03-06T07:35:43.386548+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:06.852904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.366658+0000","last_change":"2024-03-06T07:35:43.366658+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.366658+0000","last_undegraded":"2024-03-06T07:35:43.366658+0000","last_fullsized":"2024-03-06T07:35:43.366658+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.380327+0000","last_change":"2024-03-06T07:35:43.378158+0000","last_active":"2024-03-06T07:35:43.380327+0000","last_peered":"2024-03-06T07:35:43.380327+0000","last_clean":"2024-03-06T07:35:43.380327+0000","last_became_active":"2024-03-06T07:35:43.377863+0000","last_became_peered":"2024-03-06T07:35:43.377863+0000","last_unstale":"2024-03-06T07:35:43.380327+0000","last_undegraded":"2024-03-06T07:35:43.380327+0000","last_fullsized":"2024-03-06T07:35:43.380327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:57.102370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.365793+0000","last_change":"2024-03-06T07:35:43.365793+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.365793+0000","last_undegraded":"2024-03-06T07:35:43.365793+0000","last_fullsized":"2024-03-06T07:35:43.365793+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378402+0000","last_change":"2024-03-06T07:35:43.375835+0000","last_active":"2024-03-06T07:35:43.378402+0000","last_peered":"2024-03-06T07:35:43.378402+0000","last_clean":"2024-03-06T07:35:43.378402+0000","last_became_active":"2024-03-06T07:35:43.375546+0000","last_became_peered":"2024-03-06T07:35:43.375546+0000","last_unstale":"2024-03-06T07:35:43.378402+0000","last_undegraded":"2024-03-06T07:35:43.378402+0000","last_fullsized":"2024-03-06T07:35:43.378402+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:35.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.367811+0000","last_change":"2024-03-06T07:35:43.367811+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.367811+0000","last_undegraded":"2024-03-06T07:35:43.367811+0000","last_fullsized":"2024-03-06T07:35:43.367811+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.372725+0000","last_change":"2024-03-06T07:35:43.368831+0000","last_active":"2024-03-06T07:35:43.372725+0000","last_peered":"2024-03-06T07:35:43.372725+0000","last_clean":"2024-03-06T07:35:43.372725+0000","last_became_active":"2024-03-06T07:35:43.368516+0000","last_became_peered":"2024-03-06T07:35:43.368516+0000","last_unstale":"2024-03-06T07:35:43.372725+0000","last_undegraded":"2024-03-06T07:35:43.372725+0000","last_fullsized":"2024-03-06T07:35:43.372725+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:11.445222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.364677+0000","last_change":"2024-03-06T07:35:43.364677+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.364677+0000","last_undegraded":"2024-03-06T07:35:43.364677+0000","last_fullsized":"2024-03-06T07:35:43.364677+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.376657+0000","last_change":"2024-03-06T07:35:43.373948+0000","last_active":"2024-03-06T07:35:43.376657+0000","last_peered":"2024-03-06T07:35:43.376657+0000","last_clean":"2024-03-06T07:35:43.376657+0000","last_became_active":"2024-03-06T07:35:43.373631+0000","last_became_peered":"2024-03-06T07:35:43.373631+0000","last_unstale":"2024-03-06T07:35:43.376657+0000","last_undegraded":"2024-03-06T07:35:43.376657+0000","last_fullsized":"2024-03-06T07:35:43.376657+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:29.621386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356413+0000","last_change":"2024-03-06T07:35:43.356413+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356413+0000","last_undegraded":"2024-03-06T07:35:43.356413+0000","last_fullsized":"2024-03-06T07:35:43.356413+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.374263+0000","last_change":"2024-03-06T07:35:43.370282+0000","last_active":"2024-03-06T07:35:43.374263+0000","last_peered":"2024-03-06T07:35:43.374263+0000","last_clean":"2024-03-06T07:35:43.374263+0000","last_became_active":"2024-03-06T07:35:43.369383+0000","last_became_peered":"2024-03-06T07:35:43.369383+0000","last_unstale":"2024-03-06T07:35:43.374263+0000","last_undegraded":"2024-03-06T07:35:43.374263+0000","last_fullsized":"2024-03-06T07:35:43.374263+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:30.617999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.365101+0000","last_change":"2024-03-06T07:35:43.365101+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.365101+0000","last_undegraded":"2024-03-06T07:35:43.365101+0000","last_fullsized":"2024-03-06T07:35:43.365101+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.381678+0000","last_change":"2024-03-06T07:35:43.380112+0000","last_active":"2024-03-06T07:35:43.381678+0000","last_peered":"2024-03-06T07:35:43.381678+0000","last_clean":"2024-03-06T07:35:43.381678+0000","last_became_active":"2024-03-06T07:35:43.378931+0000","last_became_peered":"2024-03-06T07:35:43.378931+0000","last_unstale":"2024-03-06T07:35:43.381678+0000","last_undegraded":"2024-03-06T07:35:43.381678+0000","last_fullsized":"2024-03-06T07:35:43.381678+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:12.580678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.360792+0000","last_change":"2024-03-06T07:35:43.360792+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.360792+0000","last_undegraded":"2024-03-06T07:35:43.360792+0000","last_fullsized":"2024-03-06T07:35:43.360792+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378473+0000","last_change":"2024-03-06T07:35:43.376342+0000","last_active":"2024-03-06T07:35:43.378473+0000","last_peered":"2024-03-06T07:35:43.378473+0000","last_clean":"2024-03-06T07:35:43.378473+0000","last_became_active":"2024-03-06T07:35:43.374474+0000","last_became_peered":"2024-03-06T07:35:43.374474+0000","last_unstale":"2024-03-06T07:35:43.378473+0000","last_undegraded":"2024-03-06T07:35:43.378473+0000","last_fullsized":"2024-03-06T07:35:43.378473+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:24.590660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353707+0000","last_change":"2024-03-06T07:35:43.353707+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353707+0000","last_undegraded":"2024-03-06T07:35:43.353707+0000","last_fullsized":"2024-03-06T07:35:43.353707+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.386002+0000","last_change":"2024-03-06T07:35:43.384674+0000","last_active":"2024-03-06T07:35:43.386002+0000","last_peered":"2024-03-06T07:35:43.386002+0000","last_clean":"2024-03-06T07:35:43.386002+0000","last_became_active":"2024-03-06T07:35:43.384373+0000","last_became_peered":"2024-03-06T07:35:43.384373+0000","last_unstale":"2024-03-06T07:35:43.386002+0000","last_undegraded":"2024-03-06T07:35:43.386002+0000","last_fullsized":"2024-03-06T07:35:43.386002+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:48.683496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.375906+0000","last_change":"2024-03-06T07:35:43.375906+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.375906+0000","last_undegraded":"2024-03-06T07:35:43.375906+0000","last_fullsized":"2024-03-06T07:35:43.375906+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.384217+0000","last_change":"2024-03-06T07:35:43.381659+0000","last_active":"2024-03-06T07:35:43.384217+0000","last_peered":"2024-03-06T07:35:43.384217+0000","last_clean":"2024-03-06T07:35:43.384217+0000","last_became_active":"2024-03-06T07:35:43.381344+0000","last_became_peered":"2024-03-06T07:35:43.381344+0000","last_unstale":"2024-03-06T07:35:43.384217+0000","last_undegraded":"2024-03-06T07:35:43.384217+0000","last_fullsized":"2024-03-06T07:35:43.384217+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:41.317784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.364691+0000","last_change":"2024-03-06T07:35:43.364691+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.364691+0000","last_undegraded":"2024-03-06T07:35:43.364691+0000","last_fullsized":"2024-03-06T07:35:43.364691+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.382376+0000","last_change":"2024-03-06T07:35:43.378206+0000","last_active":"2024-03-06T07:35:43.382376+0000","last_peered":"2024-03-06T07:35:43.382376+0000","last_clean":"2024-03-06T07:35:43.382376+0000","last_became_active":"2024-03-06T07:35:43.377916+0000","last_became_peered":"2024-03-06T07:35:43.377916+0000","last_unstale":"2024-03-06T07:35:43.382376+0000","last_undegraded":"2024-03-06T07:35:43.382376+0000","last_fullsized":"2024-03-06T07:35:43.382376+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:49.995592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.367023+0000","last_change":"2024-03-06T07:35:43.367023+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.367023+0000","last_undegraded":"2024-03-06T07:35:43.367023+0000","last_fullsized":"2024-03-06T07:35:43.367023+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.381535+0000","last_change":"2024-03-06T07:35:43.379952+0000","last_active":"2024-03-06T07:35:43.381535+0000","last_peered":"2024-03-06T07:35:43.381535+0000","last_clean":"2024-03-06T07:35:43.381535+0000","last_became_active":"2024-03-06T07:35:43.379541+0000","last_became_peered":"2024-03-06T07:35:43.379541+0000","last_unstale":"2024-03-06T07:35:43.381535+0000","last_undegraded":"2024-03-06T07:35:43.381535+0000","last_fullsized":"2024-03-06T07:35:43.381535+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:03.218749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354637+0000","last_change":"2024-03-06T07:35:43.354637+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354637+0000","last_undegraded":"2024-03-06T07:35:43.354637+0000","last_fullsized":"2024-03-06T07:35:43.354637+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.373018+0000","last_change":"2024-03-06T07:35:43.372351+0000","last_active":"2024-03-06T07:35:43.373018+0000","last_peered":"2024-03-06T07:35:43.373018+0000","last_clean":"2024-03-06T07:35:43.373018+0000","last_became_active":"2024-03-06T07:35:43.372020+0000","last_became_peered":"2024-03-06T07:35:43.372020+0000","last_unstale":"2024-03-06T07:35:43.373018+0000","last_undegraded":"2024-03-06T07:35:43.373018+0000","last_fullsized":"2024-03-06T07:35:43.373018+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:08.569990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.358602+0000","last_change":"2024-03-06T07:35:43.358602+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.358602+0000","last_undegraded":"2024-03-06T07:35:43.358602+0000","last_fullsized":"2024-03-06T07:35:43.358602+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.382771+0000","last_change":"2024-03-06T07:35:43.380748+0000","last_active":"2024-03-06T07:35:43.382771+0000","last_peered":"2024-03-06T07:35:43.382771+0000","last_clean":"2024-03-06T07:35:43.382771+0000","last_became_active":"2024-03-06T07:35:43.380512+0000","last_became_peered":"2024-03-06T07:35:43.380512+0000","last_unstale":"2024-03-06T07:35:43.382771+0000","last_undegraded":"2024-03-06T07:35:43.382771+0000","last_fullsized":"2024-03-06T07:35:43.382771+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:57.084454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.370163+0000","last_change":"2024-03-06T07:35:43.370163+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.370163+0000","last_undegraded":"2024-03-06T07:35:43.370163+0000","last_fullsized":"2024-03-06T07:35:43.370163+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378596+0000","last_change":"2024-03-06T07:35:43.375271+0000","last_active":"2024-03-06T07:35:43.378596+0000","last_peered":"2024-03-06T07:35:43.378596+0000","last_clean":"2024-03-06T07:35:43.378596+0000","last_became_active":"2024-03-06T07:35:43.374378+0000","last_became_peered":"2024-03-06T07:35:43.374378+0000","last_unstale":"2024-03-06T07:35:43.378596+0000","last_undegraded":"2024-03-06T07:35:43.378596+0000","last_fullsized":"2024-03-06T07:35:43.378596+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:06.380459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.366611+0000","last_change":"2024-03-06T07:35:43.366611+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.366611+0000","last_undegraded":"2024-03-06T07:35:43.366611+0000","last_fullsized":"2024-03-06T07:35:43.366611+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.385799+0000","last_change":"2024-03-06T07:35:43.381126+0000","last_active":"2024-03-06T07:35:43.385799+0000","last_peered":"2024-03-06T07:35:43.385799+0000","last_clean":"2024-03-06T07:35:43.385799+0000","last_became_active":"2024-03-06T07:35:43.380688+0000","last_became_peered":"2024-03-06T07:35:43.380688+0000","last_unstale":"2024-03-06T07:35:43.385799+0000","last_undegraded":"2024-03-06T07:35:43.385799+0000","last_fullsized":"2024-03-06T07:35:43.385799+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:55.919575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.375297+0000","last_change":"2024-03-06T07:35:43.375297+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.375297+0000","last_undegraded":"2024-03-06T07:35:43.375297+0000","last_fullsized":"2024-03-06T07:35:43.375297+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.383499+0000","last_change":"2024-03-06T07:35:43.382676+0000","last_active":"2024-03-06T07:35:43.383499+0000","last_peered":"2024-03-06T07:35:43.383499+0000","last_clean":"2024-03-06T07:35:43.383499+0000","last_became_active":"2024-03-06T07:35:43.380914+0000","last_became_peered":"2024-03-06T07:35:43.380914+0000","last_unstale":"2024-03-06T07:35:43.383499+0000","last_undegraded":"2024-03-06T07:35:43.383499+0000","last_fullsized":"2024-03-06T07:35:43.383499+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:24.311442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.369281+0000","last_change":"2024-03-06T07:35:43.369281+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.369281+0000","last_undegraded":"2024-03-06T07:35:43.369281+0000","last_fullsized":"2024-03-06T07:35:43.369281+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.377445+0000","last_change":"2024-03-06T07:35:43.375643+0000","last_active":"2024-03-06T07:35:43.377445+0000","last_peered":"2024-03-06T07:35:43.377445+0000","last_clean":"2024-03-06T07:35:43.377445+0000","last_became_active":"2024-03-06T07:35:43.375368+0000","last_became_peered":"2024-03-06T07:35:43.375368+0000","last_unstale":"2024-03-06T07:35:43.377445+0000","last_undegraded":"2024-03-06T07:35:43.377445+0000","last_fullsized":"2024-03-06T07:35:43.377445+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:43.218935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.360107+0000","last_change":"2024-03-06T07:35:43.360107+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.360107+0000","last_undegraded":"2024-03-06T07:35:43.360107+0000","last_fullsized":"2024-03-06T07:35:43.360107+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.379847+0000","last_change":"2024-03-06T07:35:43.378109+0000","last_active":"2024-03-06T07:35:43.379847+0000","last_peered":"2024-03-06T07:35:43.379847+0000","last_clean":"2024-03-06T07:35:43.379847+0000","last_became_active":"2024-03-06T07:35:43.377689+0000","last_became_peered":"2024-03-06T07:35:43.377689+0000","last_unstale":"2024-03-06T07:35:43.379847+0000","last_undegraded":"2024-03-06T07:35:43.379847+0000","last_fullsized":"2024-03-06T07:35:43.379847+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:24.992428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353230+0000","last_change":"2024-03-06T07:35:43.353230+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353230+0000","last_undegraded":"2024-03-06T07:35:43.353230+0000","last_fullsized":"2024-03-06T07:35:43.353230+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.389082+0000","last_change":"2024-03-06T07:35:43.385621+0000","last_active":"2024-03-06T07:35:43.389082+0000","last_peered":"2024-03-06T07:35:43.389082+0000","last_clean":"2024-03-06T07:35:43.389082+0000","last_became_active":"2024-03-06T07:35:43.383393+0000","last_became_peered":"2024-03-06T07:35:43.383393+0000","last_unstale":"2024-03-06T07:35:43.389082+0000","last_undegraded":"2024-03-06T07:35:43.389082+0000","last_fullsized":"2024-03-06T07:35:43.389082+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:25.646089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354223+0000","last_change":"2024-03-06T07:35:43.354223+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354223+0000","last_undegraded":"2024-03-06T07:35:43.354223+0000","last_fullsized":"2024-03-06T07:35:43.354223+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.384237+0000","last_change":"2024-03-06T07:35:43.381958+0000","last_active":"2024-03-06T07:35:43.384237+0000","last_peered":"2024-03-06T07:35:43.384237+0000","last_clean":"2024-03-06T07:35:43.384237+0000","last_became_active":"2024-03-06T07:35:43.381369+0000","last_became_peered":"2024-03-06T07:35:43.381369+0000","last_unstale":"2024-03-06T07:35:43.384237+0000","last_undegraded":"2024-03-06T07:35:43.384237+0000","last_fullsized":"2024-03-06T07:35:43.384237+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:13.286843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.357195+0000","last_change":"2024-03-06T07:35:43.357195+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.357195+0000","last_undegraded":"2024-03-06T07:35:43.357195+0000","last_fullsized":"2024-03-06T07:35:43.357195+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.384250+0000","last_change":"2024-03-06T07:35:43.381967+0000","last_active":"2024-03-06T07:35:43.384250+0000","last_peered":"2024-03-06T07:35:43.384250+0000","last_clean":"2024-03-06T07:35:43.384250+0000","last_became_active":"2024-03-06T07:35:43.381593+0000","last_became_peered":"2024-03-06T07:35:43.381593+0000","last_unstale":"2024-03-06T07:35:43.384250+0000","last_undegraded":"2024-03-06T07:35:43.384250+0000","last_fullsized":"2024-03-06T07:35:43.384250+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:26.596579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.376407+0000","last_change":"2024-03-06T07:35:43.376407+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.376407+0000","last_undegraded":"2024-03-06T07:35:43.376407+0000","last_fullsized":"2024-03-06T07:35:43.376407+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.379631+0000","last_change":"2024-03-06T07:35:43.378687+0000","last_active":"2024-03-06T07:35:43.379631+0000","last_peered":"2024-03-06T07:35:43.379631+0000","last_clean":"2024-03-06T07:35:43.379631+0000","last_became_active":"2024-03-06T07:35:43.378095+0000","last_became_peered":"2024-03-06T07:35:43.378095+0000","last_unstale":"2024-03-06T07:35:43.379631+0000","last_undegraded":"2024-03-06T07:35:43.379631+0000","last_fullsized":"2024-03-06T07:35:43.379631+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:17.887829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354748+0000","last_change":"2024-03-06T07:35:43.354748+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354748+0000","last_undegraded":"2024-03-06T07:35:43.354748+0000","last_fullsized":"2024-03-06T07:35:43.354748+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.379175+0000","last_change":"2024-03-06T07:35:43.377991+0000","last_active":"2024-03-06T07:35:43.379175+0000","last_peered":"2024-03-06T07:35:43.379175+0000","last_clean":"2024-03-06T07:35:43.379175+0000","last_became_active":"2024-03-06T07:35:43.377732+0000","last_became_peered":"2024-03-06T07:35:43.377732+0000","last_unstale":"2024-03-06T07:35:43.379175+0000","last_undegraded":"2024-03-06T07:35:43.379175+0000","last_fullsized":"2024-03-06T07:35:43.379175+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:26.896658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353857+0000","last_change":"2024-03-06T07:35:43.353857+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353857+0000","last_undegraded":"2024-03-06T07:35:43.353857+0000","last_fullsized":"2024-03-06T07:35:43.353857+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.389197+0000","last_change":"2024-03-06T07:35:43.385136+0000","last_active":"2024-03-06T07:35:43.389197+0000","last_peered":"2024-03-06T07:35:43.389197+0000","last_clean":"2024-03-06T07:35:43.389197+0000","last_became_active":"2024-03-06T07:35:43.384757+0000","last_became_peered":"2024-03-06T07:35:43.384757+0000","last_unstale":"2024-03-06T07:35:43.389197+0000","last_undegraded":"2024-03-06T07:35:43.389197+0000","last_fullsized":"2024-03-06T07:35:43.389197+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:59.974955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356694+0000","last_change":"2024-03-06T07:35:43.356694+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356694+0000","last_undegraded":"2024-03-06T07:35:43.356694+0000","last_fullsized":"2024-03-06T07:35:43.356694+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378843+0000","last_change":"2024-03-06T07:35:43.377777+0000","last_active":"2024-03-06T07:35:43.378843+0000","last_peered":"2024-03-06T07:35:43.378843+0000","last_clean":"2024-03-06T07:35:43.378843+0000","last_became_active":"2024-03-06T07:35:43.374868+0000","last_became_peered":"2024-03-06T07:35:43.374868+0000","last_unstale":"2024-03-06T07:35:43.378843+0000","last_undegraded":"2024-03-06T07:35:43.378843+0000","last_fullsized":"2024-03-06T07:35:43.378843+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:17.660876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354203+0000","last_change":"2024-03-06T07:35:43.354203+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354203+0000","last_undegraded":"2024-03-06T07:35:43.354203+0000","last_fullsized":"2024-03-06T07:35:43.354203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.372173+0000","last_change":"2024-03-06T07:35:43.368644+0000","last_active":"2024-03-06T07:35:43.372173+0000","last_peered":"2024-03-06T07:35:43.372173+0000","last_clean":"2024-03-06T07:35:43.372173+0000","last_became_active":"2024-03-06T07:35:43.367936+0000","last_became_peered":"2024-03-06T07:35:43.367936+0000","last_unstale":"2024-03-06T07:35:43.372173+0000","last_undegraded":"2024-03-06T07:35:43.372173+0000","last_fullsized":"2024-03-06T07:35:43.372173+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:12.463927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.362698+0000","last_change":"2024-03-06T07:35:43.362698+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.362698+0000","last_undegraded":"2024-03-06T07:35:43.362698+0000","last_fullsized":"2024-03-06T07:35:43.362698+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378521+0000","last_change":"2024-03-06T07:35:43.376770+0000","last_active":"2024-03-06T07:35:43.378521+0000","last_peered":"2024-03-06T07:35:43.378521+0000","last_clean":"2024-03-06T07:35:43.378521+0000","last_became_active":"2024-03-06T07:35:43.376483+0000","last_became_peered":"2024-03-06T07:35:43.376483+0000","last_unstale":"2024-03-06T07:35:43.378521+0000","last_undegraded":"2024-03-06T07:35:43.378521+0000","last_fullsized":"2024-03-06T07:35:43.378521+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:54.876124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.366162+0000","last_change":"2024-03-06T07:35:43.366162+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.366162+0000","last_undegraded":"2024-03-06T07:35:43.366162+0000","last_fullsized":"2024-03-06T07:35:43.366162+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.389216+0000","last_change":"2024-03-06T07:35:43.385517+0000","last_active":"2024-03-06T07:35:43.389216+0000","last_peered":"2024-03-06T07:35:43.389216+0000","last_clean":"2024-03-06T07:35:43.389216+0000","last_became_active":"2024-03-06T07:35:43.384210+0000","last_became_peered":"2024-03-06T07:35:43.384210+0000","last_unstale":"2024-03-06T07:35:43.389216+0000","last_undegraded":"2024-03-06T07:35:43.389216+0000","last_fullsized":"2024-03-06T07:35:43.389216+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:21.613773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.364997+0000","last_change":"2024-03-06T07:35:43.364997+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.364997+0000","last_undegraded":"2024-03-06T07:35:43.364997+0000","last_fullsized":"2024-03-06T07:35:43.364997+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.381643+0000","last_change":"2024-03-06T07:35:43.379581+0000","last_active":"2024-03-06T07:35:43.381643+0000","last_peered":"2024-03-06T07:35:43.381643+0000","last_clean":"2024-03-06T07:35:43.381643+0000","last_became_active":"2024-03-06T07:35:43.379237+0000","last_became_peered":"2024-03-06T07:35:43.379237+0000","last_unstale":"2024-03-06T07:35:43.381643+0000","last_undegraded":"2024-03-06T07:35:43.381643+0000","last_fullsized":"2024-03-06T07:35:43.381643+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:11.817793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356880+0000","last_change":"2024-03-06T07:35:43.356880+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356880+0000","last_undegraded":"2024-03-06T07:35:43.356880+0000","last_fullsized":"2024-03-06T07:35:43.356880+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378570+0000","last_change":"2024-03-06T07:35:43.375969+0000","last_active":"2024-03-06T07:35:43.378570+0000","last_peered":"2024-03-06T07:35:43.378570+0000","last_clean":"2024-03-06T07:35:43.378570+0000","last_became_active":"2024-03-06T07:35:43.375710+0000","last_became_peered":"2024-03-06T07:35:43.375710+0000","last_unstale":"2024-03-06T07:35:43.378570+0000","last_undegraded":"2024-03-06T07:35:43.378570+0000","last_fullsized":"2024-03-06T07:35:43.378570+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:47.423584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.373091+0000","last_change":"2024-03-06T07:35:43.373091+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.373091+0000","last_undegraded":"2024-03-06T07:35:43.373091+0000","last_fullsized":"2024-03-06T07:35:43.373091+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.389201+0000","last_change":"2024-03-06T07:35:43.385255+0000","last_active":"2024-03-06T07:35:43.389201+0000","last_peered":"2024-03-06T07:35:43.389201+0000","last_clean":"2024-03-06T07:35:43.389201+0000","last_became_active":"2024-03-06T07:35:43.384945+0000","last_became_peered":"2024-03-06T07:35:43.384945+0000","last_unstale":"2024-03-06T07:35:43.389201+0000","last_undegraded":"2024-03-06T07:35:43.389201+0000","last_fullsized":"2024-03-06T07:35:43.389201+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:21.717053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":365,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.378692+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:35:43.378692+0000","last_peered":"2024-03-06T07:35:43.378692+0000","last_clean":"2024-03-06T07:35:43.378692+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:35:43.378692+0000","last_undegraded":"2024-03-06T07:35:43.378692+0000","last_fullsized":"2024-03-06T07:35:43.378692+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37476,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":2187264,"data_stored":1281465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542245,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":684822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":684822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":691097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542244,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334940,"statfs":{"total":96636764160,"available":96598978560,"internally_reserved":0,"allocated":1593344,"data_stored":691097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334940,"statfs":{"total":96636764160,"available":96598978560,"internally_reserved":0,"allocated":1593344,"data_stored":691097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":684822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41640,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330200,"statfs":{"total":96636764160,"available":96594124800,"internally_reserved":0,"allocated":2187264,"data_stored":1281465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:48.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:49.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.159+0000 7fe5b0600640 1 Processor -- start 2024-03-06T07:35:49.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.160+0000 7fe5b0600640 1 -- start start 2024-03-06T07:35:49.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.161+0000 7fe5b0600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe5a815f470 0x7fe5a80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.161+0000 7fe5b0600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 0x7fe5a8160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.161+0000 7fe5b0600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80cea60 con 0x7fe5a815e0d0 2024-03-06T07:35:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.161+0000 7fe5b0600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a80d2f60 con 0x7fe5a815f470 2024-03-06T07:35:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.161+0000 7fe5b0600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80d1240 con 0x7fe5a80d0df0 2024-03-06T07:35:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.161+0000 7fe5adc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe5a815e0d0 0x7fe5a80d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50184/0 (socket says 172.21.15.97:50184) 2024-03-06T07:35:49.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5adc00640 1 -- 172.21.15.97:0/638836105 learned_addr learned my addr 172.21.15.97:0/638836105 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:49.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650035967 0 0) 0x7fe5a80cea60 con 0x7fe5a815e0d0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe590003520 con 0x7fe5a815e0d0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5ae600640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 0x7fe5a8160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5ad200640 1 --2- 172.21.15.97:0/638836105 >> v2:172.21.15.135:3300/0 conn(0x7fe5a815f470 0x7fe5a80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492055559 0 0) 0x7fe590003520 con 0x7fe5a815e0d0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 msgr2=0x7fe5a8160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 0x7fe5a8160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 >> v2:172.21.15.135:3300/0 conn(0x7fe5a815f470 msgr2=0x7fe5a80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 --2- 172.21.15.97:0/638836105 >> v2:172.21.15.135:3300/0 conn(0x7fe5a815f470 0x7fe5a80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80d2350 con 0x7fe5a815e0d0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe59c001670 con 0x7fe5a815e0d0 2024-03-06T07:35:49.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5ad200640 1 --2- 172.21.15.97:0/638836105 >> v2:172.21.15.135:3300/0 conn(0x7fe5a815f470 0x7fe5a80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.162+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 >> v1:172.21.15.97:6789/0 conn(0x7fe5a815e0d0 legacy=0x7fe5a80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5a7e00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5900039b0 0x7fe590003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5a7e00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 0x7fe590004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5a7e00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe590004d40 0x7fe5900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80d1240 con 0x7fe5900039b0 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59000ba50 con 0x7fe590004d40 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59000bc00 con 0x7fe5a80d0df0 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5adc00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5900039b0 0x7fe590003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5adc00640 1 -- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 msgr2=0x7fe590004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5adc00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80d0df0 0x7fe590004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5adc00640 1 -- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe590004d40 msgr2=0x7fe5900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5adc00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe590004d40 0x7fe5900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.163+0000 7fe5adc00640 1 -- 172.21.15.97:0/638836105 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe590003520 con 0x7fe5900039b0 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.164+0000 7fe5adc00640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5900039b0 0x7fe590003dd0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00c210 tx=0x7fe590005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d15eb489fd7aab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c007d50 con 0x7fe5900039b0 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c030070 con 0x7fe5900039b0 2024-03-06T07:35:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5a7e00640 1 -- 172.21.15.97:0/638836105 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c02cd80 con 0x7fe5900039b0 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5b0600640 1 -- 172.21.15.97:0/638836105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5900039b0 msgr2=0x7fe590003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5b0600640 1 --2- 172.21.15.97:0/638836105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5900039b0 0x7fe590003dd0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00c210 tx=0x7fe590005830 comp rx=0 tx=0).stop 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5ae600640 1 -- 172.21.15.97:0/638836105 reap_dead start 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5b0600640 1 -- 172.21.15.97:0/638836105 shutdown_connections 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5b0600640 1 -- 172.21.15.97:0/638836105 >> 172.21.15.97:0/638836105 conn(0x7fe5a80d3ec0 msgr2=0x7fe5a80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5b0600640 1 -- 172.21.15.97:0/638836105 shutdown_connections 2024-03-06T07:35:49.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.165+0000 7fe5b0600640 1 -- 172.21.15.97:0/638836105 wait complete. 2024-03-06T07:35:49.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.168+0000 7fe5b0600640 1 Processor -- start 2024-03-06T07:35:49.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.169+0000 7fe5b0600640 1 -- start start 2024-03-06T07:35:49.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5b0600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5900039b0 0x7fe5a81523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5b0600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80b6ae0 0x7fe5a80cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5b0600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 0x7fe5a815acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5b0600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80d2bf0 con 0x7fe5a80cda70 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5b0600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80d2910 con 0x7fe5900039b0 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5b0600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a815fb30 con 0x7fe5a80b6ae0 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5ae600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 0x7fe5a815acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5ae600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 0x7fe5a815acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32866/0 (socket says 172.21.15.97:32866) 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5ae600640 1 -- 172.21.15.97:0/890589697 learned_addr learned my addr 172.21.15.97:0/890589697 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5adc00640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5900039b0 0x7fe5a81523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.171+0000 7fe5ad200640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80b6ae0 0x7fe5a80cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5ae600640 1 -- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80b6ae0 msgr2=0x7fe5a80cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5ae600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80b6ae0 0x7fe5a80cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5ae600640 1 -- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5900039b0 msgr2=0x7fe5a81523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5ae600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5900039b0 0x7fe5a81523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5ae600640 1 -- 172.21.15.97:0/890589697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a81f55d0 con 0x7fe5a80cda70 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5ae600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 0x7fe5a815acb0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80d3e90 tx=0x7fe5a000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0018020 con 0x7fe5a80cda70 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a000bab0 con 0x7fe5a80cda70 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0014400 con 0x7fe5a80cda70 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a81f58b0 con 0x7fe5a80cda70 2024-03-06T07:35:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.172+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a81f5e70 con 0x7fe5a80cda70 2024-03-06T07:35:49.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.173+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a804f330 con 0x7fe5a80cda70 2024-03-06T07:35:49.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.174+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5a0005d10 con 0x7fe5a80cda70 2024-03-06T07:35:49.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.175+0000 7fe5a5600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe58806b4d0 0x7fe58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.175+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5a008a820 con 0x7fe5a80cda70 2024-03-06T07:35:49.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.175+0000 7fe5adc00640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe58806b4d0 0x7fe58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.176+0000 7fe5adc00640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe58806b4d0 0x7fe58806d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe59c034000 tx=0x7fe59c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.179+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5a0022030 con 0x7fe5a80cda70 2024-03-06T07:35:49.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.371+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5a81f5380 con 0x7fe58806b4d0 2024-03-06T07:35:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.378+0000 7fe5a5600640 1 -- 172.21.15.97:0/890589697 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251818 (secure 0 0 0) 0x7fe59c0817c0 con 0x7fe58806b4d0 2024-03-06T07:35:49.382 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe58806b4d0 msgr2=0x7fe58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe58806b4d0 0x7fe58806d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe59c034000 tx=0x7fe59c007000 comp rx=0 tx=0).stop 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 msgr2=0x7fe5a815acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 0x7fe5a815acb0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80d3e90 tx=0x7fe5a000b480 comp rx=0 tx=0).stop 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 shutdown_connections 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe58806b4d0 0x7fe58806d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5a80cda70 0x7fe5a815acb0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe5a80b6ae0 0x7fe5a80cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 --2- 172.21.15.97:0/890589697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5900039b0 0x7fe5a81523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 >> 172.21.15.97:0/890589697 conn(0x7fe5a80d3ec0 msgr2=0x7fe5a80b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 shutdown_connections 2024-03-06T07:35:49.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.386+0000 7fe5b0600640 1 -- 172.21.15.97:0/890589697 wait complete. 2024-03-06T07:35:49.408 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-06T07:35:49.051857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304888,"kb_used_data":13836,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754669832,"statfs":{"total":773094113280,"available":772781907968,"internally_reserved":0,"allocated":14168064,"data_stored":6805787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001479"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720739+0000","last_change":"2024-03-06T07:35:44.394272+0000","last_active":"2024-03-06T07:35:45.720739+0000","last_peered":"2024-03-06T07:35:45.720739+0000","last_clean":"2024-03-06T07:35:45.720739+0000","last_became_active":"2024-03-06T07:35:44.393967+0000","last_became_peered":"2024-03-06T07:35:44.393967+0000","last_unstale":"2024-03-06T07:35:45.720739+0000","last_undegraded":"2024-03-06T07:35:45.720739+0000","last_fullsized":"2024-03-06T07:35:45.720739+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:28.599416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394850+0000","last_change":"2024-03-06T07:35:44.392186+0000","last_active":"2024-03-06T07:35:45.394850+0000","last_peered":"2024-03-06T07:35:45.394850+0000","last_clean":"2024-03-06T07:35:45.394850+0000","last_became_active":"2024-03-06T07:35:44.391446+0000","last_became_peered":"2024-03-06T07:35:44.391446+0000","last_unstale":"2024-03-06T07:35:45.394850+0000","last_undegraded":"2024-03-06T07:35:45.394850+0000","last_fullsized":"2024-03-06T07:35:45.394850+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:25.601824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720157+0000","last_change":"2024-03-06T07:35:44.410356+0000","last_active":"2024-03-06T07:35:45.720157+0000","last_peered":"2024-03-06T07:35:45.720157+0000","last_clean":"2024-03-06T07:35:45.720157+0000","last_became_active":"2024-03-06T07:35:44.409633+0000","last_became_peered":"2024-03-06T07:35:44.409633+0000","last_unstale":"2024-03-06T07:35:45.720157+0000","last_undegraded":"2024-03-06T07:35:45.720157+0000","last_fullsized":"2024-03-06T07:35:45.720157+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:12.833768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718970+0000","last_change":"2024-03-06T07:35:44.417975+0000","last_active":"2024-03-06T07:35:45.718970+0000","last_peered":"2024-03-06T07:35:45.718970+0000","last_clean":"2024-03-06T07:35:45.718970+0000","last_became_active":"2024-03-06T07:35:44.417719+0000","last_became_peered":"2024-03-06T07:35:44.417719+0000","last_unstale":"2024-03-06T07:35:45.718970+0000","last_undegraded":"2024-03-06T07:35:45.718970+0000","last_fullsized":"2024-03-06T07:35:45.718970+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:26.008403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721342+0000","last_change":"2024-03-06T07:35:44.415585+0000","last_active":"2024-03-06T07:35:45.721342+0000","last_peered":"2024-03-06T07:35:45.721342+0000","last_clean":"2024-03-06T07:35:45.721342+0000","last_became_active":"2024-03-06T07:35:44.415011+0000","last_became_peered":"2024-03-06T07:35:44.415011+0000","last_unstale":"2024-03-06T07:35:45.721342+0000","last_undegraded":"2024-03-06T07:35:45.721342+0000","last_fullsized":"2024-03-06T07:35:45.721342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354250+0000","last_change":"2024-03-06T07:35:43.354250+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354250+0000","last_undegraded":"2024-03-06T07:35:43.354250+0000","last_fullsized":"2024-03-06T07:35:43.354250+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353927+0000","last_change":"2024-03-06T07:35:43.353927+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353927+0000","last_undegraded":"2024-03-06T07:35:43.353927+0000","last_fullsized":"2024-03-06T07:35:43.353927+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719898+0000","last_change":"2024-03-06T07:35:44.405572+0000","last_active":"2024-03-06T07:35:45.719898+0000","last_peered":"2024-03-06T07:35:45.719898+0000","last_clean":"2024-03-06T07:35:45.719898+0000","last_became_active":"2024-03-06T07:35:44.405225+0000","last_became_peered":"2024-03-06T07:35:44.405225+0000","last_unstale":"2024-03-06T07:35:45.719898+0000","last_undegraded":"2024-03-06T07:35:45.719898+0000","last_fullsized":"2024-03-06T07:35:45.719898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:21.153780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721703+0000","last_change":"2024-03-06T07:35:44.415674+0000","last_active":"2024-03-06T07:35:45.721703+0000","last_peered":"2024-03-06T07:35:45.721703+0000","last_clean":"2024-03-06T07:35:45.721703+0000","last_became_active":"2024-03-06T07:35:44.415198+0000","last_became_peered":"2024-03-06T07:35:44.415198+0000","last_unstale":"2024-03-06T07:35:45.721703+0000","last_undegraded":"2024-03-06T07:35:45.721703+0000","last_fullsized":"2024-03-06T07:35:45.721703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:27.898827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395062+0000","last_change":"2024-03-06T07:35:44.398339+0000","last_active":"2024-03-06T07:35:45.395062+0000","last_peered":"2024-03-06T07:35:45.395062+0000","last_clean":"2024-03-06T07:35:45.395062+0000","last_became_active":"2024-03-06T07:35:44.396519+0000","last_became_peered":"2024-03-06T07:35:44.396519+0000","last_unstale":"2024-03-06T07:35:45.395062+0000","last_undegraded":"2024-03-06T07:35:45.395062+0000","last_fullsized":"2024-03-06T07:35:45.395062+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:48.438390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557893+0000","last_change":"2024-03-06T07:35:44.397455+0000","last_active":"2024-03-06T07:35:45.557893+0000","last_peered":"2024-03-06T07:35:45.557893+0000","last_clean":"2024-03-06T07:35:45.557893+0000","last_became_active":"2024-03-06T07:35:44.397217+0000","last_became_peered":"2024-03-06T07:35:44.397217+0000","last_unstale":"2024-03-06T07:35:45.557893+0000","last_undegraded":"2024-03-06T07:35:45.557893+0000","last_fullsized":"2024-03-06T07:35:45.557893+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:01.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720004+0000","last_change":"2024-03-06T07:35:44.416953+0000","last_active":"2024-03-06T07:35:45.720004+0000","last_peered":"2024-03-06T07:35:45.720004+0000","last_clean":"2024-03-06T07:35:45.720004+0000","last_became_active":"2024-03-06T07:35:44.416659+0000","last_became_peered":"2024-03-06T07:35:44.416659+0000","last_unstale":"2024-03-06T07:35:45.720004+0000","last_undegraded":"2024-03-06T07:35:45.720004+0000","last_fullsized":"2024-03-06T07:35:45.720004+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:31.633974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719022+0000","last_change":"2024-03-06T07:35:44.395498+0000","last_active":"2024-03-06T07:35:45.719022+0000","last_peered":"2024-03-06T07:35:45.719022+0000","last_clean":"2024-03-06T07:35:45.719022+0000","last_became_active":"2024-03-06T07:35:44.395203+0000","last_became_peered":"2024-03-06T07:35:44.395203+0000","last_unstale":"2024-03-06T07:35:45.719022+0000","last_undegraded":"2024-03-06T07:35:45.719022+0000","last_fullsized":"2024-03-06T07:35:45.719022+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:58.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394607+0000","last_change":"2024-03-06T07:35:44.400161+0000","last_active":"2024-03-06T07:35:45.394607+0000","last_peered":"2024-03-06T07:35:45.394607+0000","last_clean":"2024-03-06T07:35:45.394607+0000","last_became_active":"2024-03-06T07:35:44.399845+0000","last_became_peered":"2024-03-06T07:35:44.399845+0000","last_unstale":"2024-03-06T07:35:45.394607+0000","last_undegraded":"2024-03-06T07:35:45.394607+0000","last_fullsized":"2024-03-06T07:35:45.394607+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:33.642819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718604+0000","last_change":"2024-03-06T07:35:44.406110+0000","last_active":"2024-03-06T07:35:45.718604+0000","last_peered":"2024-03-06T07:35:45.718604+0000","last_clean":"2024-03-06T07:35:45.718604+0000","last_became_active":"2024-03-06T07:35:44.405807+0000","last_became_peered":"2024-03-06T07:35:44.405807+0000","last_unstale":"2024-03-06T07:35:45.718604+0000","last_undegraded":"2024-03-06T07:35:45.718604+0000","last_fullsized":"2024-03-06T07:35:45.718604+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:04.519517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719509+0000","last_change":"2024-03-06T07:35:44.395764+0000","last_active":"2024-03-06T07:35:45.719509+0000","last_peered":"2024-03-06T07:35:45.719509+0000","last_clean":"2024-03-06T07:35:45.719509+0000","last_became_active":"2024-03-06T07:35:44.393280+0000","last_became_peered":"2024-03-06T07:35:44.393280+0000","last_unstale":"2024-03-06T07:35:45.719509+0000","last_undegraded":"2024-03-06T07:35:45.719509+0000","last_fullsized":"2024-03-06T07:35:45.719509+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356595+0000","last_change":"2024-03-06T07:35:43.356595+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356595+0000","last_undegraded":"2024-03-06T07:35:43.356595+0000","last_fullsized":"2024-03-06T07:35:43.356595+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720362+0000","last_change":"2024-03-06T07:35:44.395358+0000","last_active":"2024-03-06T07:35:45.720362+0000","last_peered":"2024-03-06T07:35:45.720362+0000","last_clean":"2024-03-06T07:35:45.720362+0000","last_became_active":"2024-03-06T07:35:44.395025+0000","last_became_peered":"2024-03-06T07:35:44.395025+0000","last_unstale":"2024-03-06T07:35:45.720362+0000","last_undegraded":"2024-03-06T07:35:45.720362+0000","last_fullsized":"2024-03-06T07:35:45.720362+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558069+0000","last_change":"2024-03-06T07:35:44.408168+0000","last_active":"2024-03-06T07:35:45.558069+0000","last_peered":"2024-03-06T07:35:45.558069+0000","last_clean":"2024-03-06T07:35:45.558069+0000","last_became_active":"2024-03-06T07:35:44.407536+0000","last_became_peered":"2024-03-06T07:35:44.407536+0000","last_unstale":"2024-03-06T07:35:45.558069+0000","last_undegraded":"2024-03-06T07:35:45.558069+0000","last_fullsized":"2024-03-06T07:35:45.558069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:45.475165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720163+0000","last_change":"2024-03-06T07:35:44.390845+0000","last_active":"2024-03-06T07:35:45.720163+0000","last_peered":"2024-03-06T07:35:45.720163+0000","last_clean":"2024-03-06T07:35:45.720163+0000","last_became_active":"2024-03-06T07:35:44.388466+0000","last_became_peered":"2024-03-06T07:35:44.388466+0000","last_unstale":"2024-03-06T07:35:45.720163+0000","last_undegraded":"2024-03-06T07:35:45.720163+0000","last_fullsized":"2024-03-06T07:35:45.720163+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721001+0000","last_change":"2024-03-06T07:35:44.415326+0000","last_active":"2024-03-06T07:35:45.721001+0000","last_peered":"2024-03-06T07:35:45.721001+0000","last_clean":"2024-03-06T07:35:45.721001+0000","last_became_active":"2024-03-06T07:35:44.414356+0000","last_became_peered":"2024-03-06T07:35:44.414356+0000","last_unstale":"2024-03-06T07:35:45.721001+0000","last_undegraded":"2024-03-06T07:35:45.721001+0000","last_fullsized":"2024-03-06T07:35:45.721001+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557466+0000","last_change":"2024-03-06T07:35:44.394518+0000","last_active":"2024-03-06T07:35:45.557466+0000","last_peered":"2024-03-06T07:35:45.557466+0000","last_clean":"2024-03-06T07:35:45.557466+0000","last_became_active":"2024-03-06T07:35:44.392743+0000","last_became_peered":"2024-03-06T07:35:44.392743+0000","last_unstale":"2024-03-06T07:35:45.557466+0000","last_undegraded":"2024-03-06T07:35:45.557466+0000","last_fullsized":"2024-03-06T07:35:45.557466+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:34.320907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720515+0000","last_change":"2024-03-06T07:35:44.417926+0000","last_active":"2024-03-06T07:35:45.720515+0000","last_peered":"2024-03-06T07:35:45.720515+0000","last_clean":"2024-03-06T07:35:45.720515+0000","last_became_active":"2024-03-06T07:35:44.415414+0000","last_became_peered":"2024-03-06T07:35:44.415414+0000","last_unstale":"2024-03-06T07:35:45.720515+0000","last_undegraded":"2024-03-06T07:35:45.720515+0000","last_fullsized":"2024-03-06T07:35:45.720515+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:14.302817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557430+0000","last_change":"2024-03-06T07:35:44.408475+0000","last_active":"2024-03-06T07:35:45.557430+0000","last_peered":"2024-03-06T07:35:45.557430+0000","last_clean":"2024-03-06T07:35:45.557430+0000","last_became_active":"2024-03-06T07:35:44.408227+0000","last_became_peered":"2024-03-06T07:35:44.408227+0000","last_unstale":"2024-03-06T07:35:45.557430+0000","last_undegraded":"2024-03-06T07:35:45.557430+0000","last_fullsized":"2024-03-06T07:35:45.557430+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395196+0000","last_change":"2024-03-06T07:35:44.416789+0000","last_active":"2024-03-06T07:35:45.395196+0000","last_peered":"2024-03-06T07:35:45.395196+0000","last_clean":"2024-03-06T07:35:45.395196+0000","last_became_active":"2024-03-06T07:35:44.416274+0000","last_became_peered":"2024-03-06T07:35:44.416274+0000","last_unstale":"2024-03-06T07:35:45.395196+0000","last_undegraded":"2024-03-06T07:35:45.395196+0000","last_fullsized":"2024-03-06T07:35:45.395196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:47.395736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720627+0000","last_change":"2024-03-06T07:35:44.399980+0000","last_active":"2024-03-06T07:35:45.720627+0000","last_peered":"2024-03-06T07:35:45.720627+0000","last_clean":"2024-03-06T07:35:45.720627+0000","last_became_active":"2024-03-06T07:35:44.399328+0000","last_became_peered":"2024-03-06T07:35:44.399328+0000","last_unstale":"2024-03-06T07:35:45.720627+0000","last_undegraded":"2024-03-06T07:35:45.720627+0000","last_fullsized":"2024-03-06T07:35:45.720627+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:43.875737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719802+0000","last_change":"2024-03-06T07:35:44.394219+0000","last_active":"2024-03-06T07:35:45.719802+0000","last_peered":"2024-03-06T07:35:45.719802+0000","last_clean":"2024-03-06T07:35:45.719802+0000","last_became_active":"2024-03-06T07:35:44.393893+0000","last_became_peered":"2024-03-06T07:35:44.393893+0000","last_unstale":"2024-03-06T07:35:45.719802+0000","last_undegraded":"2024-03-06T07:35:45.719802+0000","last_fullsized":"2024-03-06T07:35:45.719802+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:18.620474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398830+0000","last_change":"2024-03-06T07:35:44.403178+0000","last_active":"2024-03-06T07:35:45.398830+0000","last_peered":"2024-03-06T07:35:45.398830+0000","last_clean":"2024-03-06T07:35:45.398830+0000","last_became_active":"2024-03-06T07:35:44.402903+0000","last_became_peered":"2024-03-06T07:35:44.402903+0000","last_unstale":"2024-03-06T07:35:45.398830+0000","last_undegraded":"2024-03-06T07:35:45.398830+0000","last_fullsized":"2024-03-06T07:35:45.398830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719226+0000","last_change":"2024-03-06T07:35:44.415700+0000","last_active":"2024-03-06T07:35:45.719226+0000","last_peered":"2024-03-06T07:35:45.719226+0000","last_clean":"2024-03-06T07:35:45.719226+0000","last_became_active":"2024-03-06T07:35:44.415400+0000","last_became_peered":"2024-03-06T07:35:44.415400+0000","last_unstale":"2024-03-06T07:35:45.719226+0000","last_undegraded":"2024-03-06T07:35:45.719226+0000","last_fullsized":"2024-03-06T07:35:45.719226+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:56.935549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719196+0000","last_change":"2024-03-06T07:35:44.406337+0000","last_active":"2024-03-06T07:35:45.719196+0000","last_peered":"2024-03-06T07:35:45.719196+0000","last_clean":"2024-03-06T07:35:45.719196+0000","last_became_active":"2024-03-06T07:35:44.405995+0000","last_became_peered":"2024-03-06T07:35:44.405995+0000","last_unstale":"2024-03-06T07:35:45.719196+0000","last_undegraded":"2024-03-06T07:35:45.719196+0000","last_fullsized":"2024-03-06T07:35:45.719196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:51.011567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719669+0000","last_change":"2024-03-06T07:35:44.415228+0000","last_active":"2024-03-06T07:35:45.719669+0000","last_peered":"2024-03-06T07:35:45.719669+0000","last_clean":"2024-03-06T07:35:45.719669+0000","last_became_active":"2024-03-06T07:35:44.414418+0000","last_became_peered":"2024-03-06T07:35:44.414418+0000","last_unstale":"2024-03-06T07:35:45.719669+0000","last_undegraded":"2024-03-06T07:35:45.719669+0000","last_fullsized":"2024-03-06T07:35:45.719669+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:25.327417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353912+0000","last_change":"2024-03-06T07:35:43.353912+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353912+0000","last_undegraded":"2024-03-06T07:35:43.353912+0000","last_fullsized":"2024-03-06T07:35:43.353912+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720567+0000","last_change":"2024-03-06T07:35:44.408316+0000","last_active":"2024-03-06T07:35:45.720567+0000","last_peered":"2024-03-06T07:35:45.720567+0000","last_clean":"2024-03-06T07:35:45.720567+0000","last_became_active":"2024-03-06T07:35:44.407617+0000","last_became_peered":"2024-03-06T07:35:44.407617+0000","last_unstale":"2024-03-06T07:35:45.720567+0000","last_undegraded":"2024-03-06T07:35:45.720567+0000","last_fullsized":"2024-03-06T07:35:45.720567+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:09.791568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722044+0000","last_change":"2024-03-06T07:35:43.384588+0000","last_active":"2024-03-06T07:35:45.722044+0000","last_peered":"2024-03-06T07:35:45.722044+0000","last_clean":"2024-03-06T07:35:45.722044+0000","last_became_active":"2024-03-06T07:35:43.384278+0000","last_became_peered":"2024-03-06T07:35:43.384278+0000","last_unstale":"2024-03-06T07:35:45.722044+0000","last_undegraded":"2024-03-06T07:35:45.722044+0000","last_fullsized":"2024-03-06T07:35:45.722044+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:30.617999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722113+0000","last_change":"2024-03-06T07:35:44.408075+0000","last_active":"2024-03-06T07:35:45.722113+0000","last_peered":"2024-03-06T07:35:45.722113+0000","last_clean":"2024-03-06T07:35:45.722113+0000","last_became_active":"2024-03-06T07:35:44.407778+0000","last_became_peered":"2024-03-06T07:35:44.407778+0000","last_unstale":"2024-03-06T07:35:45.722113+0000","last_undegraded":"2024-03-06T07:35:45.722113+0000","last_fullsized":"2024-03-06T07:35:45.722113+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:33.972464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399395+0000","last_change":"2024-03-06T07:35:43.378158+0000","last_active":"2024-03-06T07:35:45.399395+0000","last_peered":"2024-03-06T07:35:45.399395+0000","last_clean":"2024-03-06T07:35:45.399395+0000","last_became_active":"2024-03-06T07:35:43.377863+0000","last_became_peered":"2024-03-06T07:35:43.377863+0000","last_unstale":"2024-03-06T07:35:45.399395+0000","last_undegraded":"2024-03-06T07:35:45.399395+0000","last_fullsized":"2024-03-06T07:35:45.399395+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:27.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720318+0000","last_change":"2024-03-06T07:35:44.419353+0000","last_active":"2024-03-06T07:35:45.720318+0000","last_peered":"2024-03-06T07:35:45.720318+0000","last_clean":"2024-03-06T07:35:45.720318+0000","last_became_active":"2024-03-06T07:35:44.418396+0000","last_became_peered":"2024-03-06T07:35:44.418396+0000","last_unstale":"2024-03-06T07:35:45.720318+0000","last_undegraded":"2024-03-06T07:35:45.720318+0000","last_fullsized":"2024-03-06T07:35:45.720318+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:12.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558056+0000","last_change":"2024-03-06T07:35:43.375835+0000","last_active":"2024-03-06T07:35:45.558056+0000","last_peered":"2024-03-06T07:35:45.558056+0000","last_clean":"2024-03-06T07:35:45.558056+0000","last_became_active":"2024-03-06T07:35:43.375546+0000","last_became_peered":"2024-03-06T07:35:43.375546+0000","last_unstale":"2024-03-06T07:35:45.558056+0000","last_undegraded":"2024-03-06T07:35:45.558056+0000","last_fullsized":"2024-03-06T07:35:45.558056+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:44.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721668+0000","last_change":"2024-03-06T07:35:44.411183+0000","last_active":"2024-03-06T07:35:45.721668+0000","last_peered":"2024-03-06T07:35:45.721668+0000","last_clean":"2024-03-06T07:35:45.721668+0000","last_became_active":"2024-03-06T07:35:44.410909+0000","last_became_peered":"2024-03-06T07:35:44.410909+0000","last_unstale":"2024-03-06T07:35:45.721668+0000","last_undegraded":"2024-03-06T07:35:45.721668+0000","last_fullsized":"2024-03-06T07:35:45.721668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394928+0000","last_change":"2024-03-06T07:35:43.368831+0000","last_active":"2024-03-06T07:35:45.394928+0000","last_peered":"2024-03-06T07:35:45.394928+0000","last_clean":"2024-03-06T07:35:45.394928+0000","last_became_active":"2024-03-06T07:35:43.368516+0000","last_became_peered":"2024-03-06T07:35:43.368516+0000","last_unstale":"2024-03-06T07:35:45.394928+0000","last_undegraded":"2024-03-06T07:35:45.394928+0000","last_fullsized":"2024-03-06T07:35:45.394928+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:24.992428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719450+0000","last_change":"2024-03-06T07:35:44.384339+0000","last_active":"2024-03-06T07:35:45.719450+0000","last_peered":"2024-03-06T07:35:45.719450+0000","last_clean":"2024-03-06T07:35:45.719450+0000","last_became_active":"2024-03-06T07:35:44.384031+0000","last_became_peered":"2024-03-06T07:35:44.384031+0000","last_unstale":"2024-03-06T07:35:45.719450+0000","last_undegraded":"2024-03-06T07:35:45.719450+0000","last_fullsized":"2024-03-06T07:35:45.719450+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394361+0000","last_change":"2024-03-06T07:35:43.373948+0000","last_active":"2024-03-06T07:35:45.394361+0000","last_peered":"2024-03-06T07:35:45.394361+0000","last_clean":"2024-03-06T07:35:45.394361+0000","last_became_active":"2024-03-06T07:35:43.373631+0000","last_became_peered":"2024-03-06T07:35:43.373631+0000","last_unstale":"2024-03-06T07:35:45.394361+0000","last_undegraded":"2024-03-06T07:35:45.394361+0000","last_fullsized":"2024-03-06T07:35:45.394361+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:51.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356413+0000","last_change":"2024-03-06T07:35:43.356413+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356413+0000","last_undegraded":"2024-03-06T07:35:43.356413+0000","last_fullsized":"2024-03-06T07:35:43.356413+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719623+0000","last_change":"2024-03-06T07:35:43.370282+0000","last_active":"2024-03-06T07:35:45.719623+0000","last_peered":"2024-03-06T07:35:45.719623+0000","last_clean":"2024-03-06T07:35:45.719623+0000","last_became_active":"2024-03-06T07:35:43.369383+0000","last_became_peered":"2024-03-06T07:35:43.369383+0000","last_unstale":"2024-03-06T07:35:45.719623+0000","last_undegraded":"2024-03-06T07:35:45.719623+0000","last_fullsized":"2024-03-06T07:35:45.719623+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:02.163748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720319+0000","last_change":"2024-03-06T07:35:44.405369+0000","last_active":"2024-03-06T07:35:45.720319+0000","last_peered":"2024-03-06T07:35:45.720319+0000","last_clean":"2024-03-06T07:35:45.720319+0000","last_became_active":"2024-03-06T07:35:44.405046+0000","last_became_peered":"2024-03-06T07:35:44.405046+0000","last_unstale":"2024-03-06T07:35:45.720319+0000","last_undegraded":"2024-03-06T07:35:45.720319+0000","last_fullsized":"2024-03-06T07:35:45.720319+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:27.612553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.381678+0000","last_change":"2024-03-06T07:35:43.380112+0000","last_active":"2024-03-06T07:35:43.381678+0000","last_peered":"2024-03-06T07:35:43.381678+0000","last_clean":"2024-03-06T07:35:43.381678+0000","last_became_active":"2024-03-06T07:35:43.378931+0000","last_became_peered":"2024-03-06T07:35:43.378931+0000","last_unstale":"2024-03-06T07:35:43.381678+0000","last_undegraded":"2024-03-06T07:35:43.381678+0000","last_fullsized":"2024-03-06T07:35:43.381678+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:12.580678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394619+0000","last_change":"2024-03-06T07:35:44.382654+0000","last_active":"2024-03-06T07:35:45.394619+0000","last_peered":"2024-03-06T07:35:45.394619+0000","last_clean":"2024-03-06T07:35:45.394619+0000","last_became_active":"2024-03-06T07:35:44.382389+0000","last_became_peered":"2024-03-06T07:35:44.382389+0000","last_unstale":"2024-03-06T07:35:45.394619+0000","last_undegraded":"2024-03-06T07:35:45.394619+0000","last_fullsized":"2024-03-06T07:35:45.394619+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394363+0000","last_change":"2024-03-06T07:35:43.376342+0000","last_active":"2024-03-06T07:35:45.394363+0000","last_peered":"2024-03-06T07:35:45.394363+0000","last_clean":"2024-03-06T07:35:45.394363+0000","last_became_active":"2024-03-06T07:35:43.374474+0000","last_became_peered":"2024-03-06T07:35:43.374474+0000","last_unstale":"2024-03-06T07:35:45.394363+0000","last_undegraded":"2024-03-06T07:35:45.394363+0000","last_fullsized":"2024-03-06T07:35:45.394363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:03.503542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720355+0000","last_change":"2024-03-06T07:35:44.402234+0000","last_active":"2024-03-06T07:35:45.720355+0000","last_peered":"2024-03-06T07:35:45.720355+0000","last_clean":"2024-03-06T07:35:45.720355+0000","last_became_active":"2024-03-06T07:35:44.401505+0000","last_became_peered":"2024-03-06T07:35:44.401505+0000","last_unstale":"2024-03-06T07:35:45.720355+0000","last_undegraded":"2024-03-06T07:35:45.720355+0000","last_fullsized":"2024-03-06T07:35:45.720355+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:22.846996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401654+0000","last_change":"2024-03-06T07:35:43.384674+0000","last_active":"2024-03-06T07:35:45.401654+0000","last_peered":"2024-03-06T07:35:45.401654+0000","last_clean":"2024-03-06T07:35:45.401654+0000","last_became_active":"2024-03-06T07:35:43.384373+0000","last_became_peered":"2024-03-06T07:35:43.384373+0000","last_unstale":"2024-03-06T07:35:45.401654+0000","last_undegraded":"2024-03-06T07:35:45.401654+0000","last_fullsized":"2024-03-06T07:35:45.401654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:47.747507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400796+0000","last_change":"2024-03-06T07:35:44.400940+0000","last_active":"2024-03-06T07:35:45.400796+0000","last_peered":"2024-03-06T07:35:45.400796+0000","last_clean":"2024-03-06T07:35:45.400796+0000","last_became_active":"2024-03-06T07:35:44.400650+0000","last_became_peered":"2024-03-06T07:35:44.400650+0000","last_unstale":"2024-03-06T07:35:45.400796+0000","last_undegraded":"2024-03-06T07:35:45.400796+0000","last_fullsized":"2024-03-06T07:35:45.400796+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720656+0000","last_change":"2024-03-06T07:35:43.381659+0000","last_active":"2024-03-06T07:35:45.720656+0000","last_peered":"2024-03-06T07:35:45.720656+0000","last_clean":"2024-03-06T07:35:45.720656+0000","last_became_active":"2024-03-06T07:35:43.381344+0000","last_became_peered":"2024-03-06T07:35:43.381344+0000","last_unstale":"2024-03-06T07:35:45.720656+0000","last_undegraded":"2024-03-06T07:35:45.720656+0000","last_fullsized":"2024-03-06T07:35:45.720656+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:48.989771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720762+0000","last_change":"2024-03-06T07:35:44.420113+0000","last_active":"2024-03-06T07:35:45.720762+0000","last_peered":"2024-03-06T07:35:45.720762+0000","last_clean":"2024-03-06T07:35:45.720762+0000","last_became_active":"2024-03-06T07:35:44.418765+0000","last_became_peered":"2024-03-06T07:35:44.418765+0000","last_unstale":"2024-03-06T07:35:45.720762+0000","last_undegraded":"2024-03-06T07:35:45.720762+0000","last_fullsized":"2024-03-06T07:35:45.720762+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"129'3","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398749+0000","last_change":"2024-03-06T07:35:43.378206+0000","last_active":"2024-03-06T07:35:45.398749+0000","last_peered":"2024-03-06T07:35:45.398749+0000","last_clean":"2024-03-06T07:35:45.398749+0000","last_became_active":"2024-03-06T07:35:43.377916+0000","last_became_peered":"2024-03-06T07:35:43.377916+0000","last_unstale":"2024-03-06T07:35:45.398749+0000","last_undegraded":"2024-03-06T07:35:45.398749+0000","last_fullsized":"2024-03-06T07:35:45.398749+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:21.831021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722052+0000","last_change":"2024-03-06T07:35:44.416806+0000","last_active":"2024-03-06T07:35:45.722052+0000","last_peered":"2024-03-06T07:35:45.722052+0000","last_clean":"2024-03-06T07:35:45.722052+0000","last_became_active":"2024-03-06T07:35:44.416506+0000","last_became_peered":"2024-03-06T07:35:44.416506+0000","last_unstale":"2024-03-06T07:35:45.722052+0000","last_undegraded":"2024-03-06T07:35:45.722052+0000","last_fullsized":"2024-03-06T07:35:45.722052+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"129'6","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721134+0000","last_change":"2024-03-06T07:35:43.379952+0000","last_active":"2024-03-06T07:35:45.721134+0000","last_peered":"2024-03-06T07:35:45.721134+0000","last_clean":"2024-03-06T07:35:45.721134+0000","last_became_active":"2024-03-06T07:35:43.379541+0000","last_became_peered":"2024-03-06T07:35:43.379541+0000","last_unstale":"2024-03-06T07:35:45.721134+0000","last_undegraded":"2024-03-06T07:35:45.721134+0000","last_fullsized":"2024-03-06T07:35:45.721134+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.626844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719234+0000","last_change":"2024-03-06T07:35:44.400442+0000","last_active":"2024-03-06T07:35:45.719234+0000","last_peered":"2024-03-06T07:35:45.719234+0000","last_clean":"2024-03-06T07:35:45.719234+0000","last_became_active":"2024-03-06T07:35:44.399865+0000","last_became_peered":"2024-03-06T07:35:44.399865+0000","last_unstale":"2024-03-06T07:35:45.719234+0000","last_undegraded":"2024-03-06T07:35:45.719234+0000","last_fullsized":"2024-03-06T07:35:45.719234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:26.662064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719793+0000","last_change":"2024-03-06T07:35:43.372351+0000","last_active":"2024-03-06T07:35:45.719793+0000","last_peered":"2024-03-06T07:35:45.719793+0000","last_clean":"2024-03-06T07:35:45.719793+0000","last_became_active":"2024-03-06T07:35:43.372020+0000","last_became_peered":"2024-03-06T07:35:43.372020+0000","last_unstale":"2024-03-06T07:35:45.719793+0000","last_undegraded":"2024-03-06T07:35:45.719793+0000","last_fullsized":"2024-03-06T07:35:45.719793+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:11.361058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394255+0000","last_change":"2024-03-06T07:35:44.412709+0000","last_active":"2024-03-06T07:35:45.394255+0000","last_peered":"2024-03-06T07:35:45.394255+0000","last_clean":"2024-03-06T07:35:45.394255+0000","last_became_active":"2024-03-06T07:35:44.411842+0000","last_became_peered":"2024-03-06T07:35:44.411842+0000","last_unstale":"2024-03-06T07:35:45.394255+0000","last_undegraded":"2024-03-06T07:35:45.394255+0000","last_fullsized":"2024-03-06T07:35:45.394255+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:04.234723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394759+0000","last_change":"2024-03-06T07:35:43.380748+0000","last_active":"2024-03-06T07:35:45.394759+0000","last_peered":"2024-03-06T07:35:45.394759+0000","last_clean":"2024-03-06T07:35:45.394759+0000","last_became_active":"2024-03-06T07:35:43.380512+0000","last_became_peered":"2024-03-06T07:35:43.380512+0000","last_unstale":"2024-03-06T07:35:45.394759+0000","last_undegraded":"2024-03-06T07:35:45.394759+0000","last_fullsized":"2024-03-06T07:35:45.394759+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:02.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720212+0000","last_change":"2024-03-06T07:35:44.416049+0000","last_active":"2024-03-06T07:35:45.720212+0000","last_peered":"2024-03-06T07:35:45.720212+0000","last_clean":"2024-03-06T07:35:45.720212+0000","last_became_active":"2024-03-06T07:35:44.415779+0000","last_became_peered":"2024-03-06T07:35:44.415779+0000","last_unstale":"2024-03-06T07:35:45.720212+0000","last_undegraded":"2024-03-06T07:35:45.720212+0000","last_fullsized":"2024-03-06T07:35:45.720212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:42.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":22,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720702+0000","last_change":"2024-03-06T07:35:43.375271+0000","last_active":"2024-03-06T07:35:45.720702+0000","last_peered":"2024-03-06T07:35:45.720702+0000","last_clean":"2024-03-06T07:35:45.720702+0000","last_became_active":"2024-03-06T07:35:43.374378+0000","last_became_peered":"2024-03-06T07:35:43.374378+0000","last_unstale":"2024-03-06T07:35:45.720702+0000","last_undegraded":"2024-03-06T07:35:45.720702+0000","last_fullsized":"2024-03-06T07:35:45.720702+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:47.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721573+0000","last_change":"2024-03-06T07:35:44.403410+0000","last_active":"2024-03-06T07:35:45.721573+0000","last_peered":"2024-03-06T07:35:45.721573+0000","last_clean":"2024-03-06T07:35:45.721573+0000","last_became_active":"2024-03-06T07:35:44.402014+0000","last_became_peered":"2024-03-06T07:35:44.402014+0000","last_unstale":"2024-03-06T07:35:45.721573+0000","last_undegraded":"2024-03-06T07:35:45.721573+0000","last_fullsized":"2024-03-06T07:35:45.721573+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:50.005746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401803+0000","last_change":"2024-03-06T07:35:43.381126+0000","last_active":"2024-03-06T07:35:45.401803+0000","last_peered":"2024-03-06T07:35:45.401803+0000","last_clean":"2024-03-06T07:35:45.401803+0000","last_became_active":"2024-03-06T07:35:43.380688+0000","last_became_peered":"2024-03-06T07:35:43.380688+0000","last_unstale":"2024-03-06T07:35:45.401803+0000","last_undegraded":"2024-03-06T07:35:45.401803+0000","last_fullsized":"2024-03-06T07:35:45.401803+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:31.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401213+0000","last_change":"2024-03-06T07:35:44.412003+0000","last_active":"2024-03-06T07:35:45.401213+0000","last_peered":"2024-03-06T07:35:45.401213+0000","last_clean":"2024-03-06T07:35:45.401213+0000","last_became_active":"2024-03-06T07:35:44.409915+0000","last_became_peered":"2024-03-06T07:35:44.409915+0000","last_unstale":"2024-03-06T07:35:45.401213+0000","last_undegraded":"2024-03-06T07:35:45.401213+0000","last_fullsized":"2024-03-06T07:35:45.401213+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557877+0000","last_change":"2024-03-06T07:35:43.382676+0000","last_active":"2024-03-06T07:35:45.557877+0000","last_peered":"2024-03-06T07:35:45.557877+0000","last_clean":"2024-03-06T07:35:45.557877+0000","last_became_active":"2024-03-06T07:35:43.380914+0000","last_became_peered":"2024-03-06T07:35:43.380914+0000","last_unstale":"2024-03-06T07:35:45.557877+0000","last_undegraded":"2024-03-06T07:35:45.557877+0000","last_fullsized":"2024-03-06T07:35:45.557877+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.893669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720870+0000","last_change":"2024-03-06T07:35:44.400453+0000","last_active":"2024-03-06T07:35:45.720870+0000","last_peered":"2024-03-06T07:35:45.720870+0000","last_clean":"2024-03-06T07:35:45.720870+0000","last_became_active":"2024-03-06T07:35:44.399501+0000","last_became_peered":"2024-03-06T07:35:44.399501+0000","last_unstale":"2024-03-06T07:35:45.720870+0000","last_undegraded":"2024-03-06T07:35:45.720870+0000","last_fullsized":"2024-03-06T07:35:45.720870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.377445+0000","last_change":"2024-03-06T07:35:43.375643+0000","last_active":"2024-03-06T07:35:43.377445+0000","last_peered":"2024-03-06T07:35:43.377445+0000","last_clean":"2024-03-06T07:35:43.377445+0000","last_became_active":"2024-03-06T07:35:43.375368+0000","last_became_peered":"2024-03-06T07:35:43.375368+0000","last_unstale":"2024-03-06T07:35:43.377445+0000","last_undegraded":"2024-03-06T07:35:43.377445+0000","last_fullsized":"2024-03-06T07:35:43.377445+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:43.218935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394708+0000","last_change":"2024-03-06T07:35:44.387752+0000","last_active":"2024-03-06T07:35:45.394708+0000","last_peered":"2024-03-06T07:35:45.394708+0000","last_clean":"2024-03-06T07:35:45.394708+0000","last_became_active":"2024-03-06T07:35:44.386625+0000","last_became_peered":"2024-03-06T07:35:44.386625+0000","last_unstale":"2024-03-06T07:35:45.394708+0000","last_undegraded":"2024-03-06T07:35:45.394708+0000","last_fullsized":"2024-03-06T07:35:45.394708+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398845+0000","last_change":"2024-03-06T07:35:43.378109+0000","last_active":"2024-03-06T07:35:45.398845+0000","last_peered":"2024-03-06T07:35:45.398845+0000","last_clean":"2024-03-06T07:35:45.398845+0000","last_became_active":"2024-03-06T07:35:43.377689+0000","last_became_peered":"2024-03-06T07:35:43.377689+0000","last_unstale":"2024-03-06T07:35:45.398845+0000","last_undegraded":"2024-03-06T07:35:45.398845+0000","last_fullsized":"2024-03-06T07:35:45.398845+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:17.604500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719574+0000","last_change":"2024-03-06T07:35:44.406714+0000","last_active":"2024-03-06T07:35:45.719574+0000","last_peered":"2024-03-06T07:35:45.719574+0000","last_clean":"2024-03-06T07:35:45.719574+0000","last_became_active":"2024-03-06T07:35:44.406421+0000","last_became_peered":"2024-03-06T07:35:44.406421+0000","last_unstale":"2024-03-06T07:35:45.719574+0000","last_undegraded":"2024-03-06T07:35:45.719574+0000","last_fullsized":"2024-03-06T07:35:45.719574+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:49.699470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399284+0000","last_change":"2024-03-06T07:35:43.385621+0000","last_active":"2024-03-06T07:35:45.399284+0000","last_peered":"2024-03-06T07:35:45.399284+0000","last_clean":"2024-03-06T07:35:45.399284+0000","last_became_active":"2024-03-06T07:35:43.383393+0000","last_became_peered":"2024-03-06T07:35:43.383393+0000","last_unstale":"2024-03-06T07:35:45.399284+0000","last_undegraded":"2024-03-06T07:35:45.399284+0000","last_fullsized":"2024-03-06T07:35:45.399284+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:45.593408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354223+0000","last_change":"2024-03-06T07:35:43.354223+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354223+0000","last_undegraded":"2024-03-06T07:35:43.354223+0000","last_fullsized":"2024-03-06T07:35:43.354223+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719668+0000","last_change":"2024-03-06T07:35:43.381958+0000","last_active":"2024-03-06T07:35:45.719668+0000","last_peered":"2024-03-06T07:35:45.719668+0000","last_clean":"2024-03-06T07:35:45.719668+0000","last_became_active":"2024-03-06T07:35:43.381369+0000","last_became_peered":"2024-03-06T07:35:43.381369+0000","last_unstale":"2024-03-06T07:35:45.719668+0000","last_undegraded":"2024-03-06T07:35:45.719668+0000","last_fullsized":"2024-03-06T07:35:45.719668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:26.882853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394343+0000","last_change":"2024-03-06T07:35:44.386726+0000","last_active":"2024-03-06T07:35:45.394343+0000","last_peered":"2024-03-06T07:35:45.394343+0000","last_clean":"2024-03-06T07:35:45.394343+0000","last_became_active":"2024-03-06T07:35:44.385066+0000","last_became_peered":"2024-03-06T07:35:44.385066+0000","last_unstale":"2024-03-06T07:35:45.394343+0000","last_undegraded":"2024-03-06T07:35:45.394343+0000","last_fullsized":"2024-03-06T07:35:45.394343+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:29.921786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719327+0000","last_change":"2024-03-06T07:35:43.381967+0000","last_active":"2024-03-06T07:35:45.719327+0000","last_peered":"2024-03-06T07:35:45.719327+0000","last_clean":"2024-03-06T07:35:45.719327+0000","last_became_active":"2024-03-06T07:35:43.381593+0000","last_became_peered":"2024-03-06T07:35:43.381593+0000","last_unstale":"2024-03-06T07:35:45.719327+0000","last_undegraded":"2024-03-06T07:35:45.719327+0000","last_fullsized":"2024-03-06T07:35:45.719327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:32.956490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719534+0000","last_change":"2024-03-06T07:35:44.416923+0000","last_active":"2024-03-06T07:35:45.719534+0000","last_peered":"2024-03-06T07:35:45.719534+0000","last_clean":"2024-03-06T07:35:45.719534+0000","last_became_active":"2024-03-06T07:35:44.414762+0000","last_became_peered":"2024-03-06T07:35:44.414762+0000","last_unstale":"2024-03-06T07:35:45.719534+0000","last_undegraded":"2024-03-06T07:35:45.719534+0000","last_fullsized":"2024-03-06T07:35:45.719534+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:18.903803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557365+0000","last_change":"2024-03-06T07:35:43.378687+0000","last_active":"2024-03-06T07:35:45.557365+0000","last_peered":"2024-03-06T07:35:45.557365+0000","last_clean":"2024-03-06T07:35:45.557365+0000","last_became_active":"2024-03-06T07:35:43.378095+0000","last_became_peered":"2024-03-06T07:35:43.378095+0000","last_unstale":"2024-03-06T07:35:45.557365+0000","last_undegraded":"2024-03-06T07:35:45.557365+0000","last_fullsized":"2024-03-06T07:35:45.557365+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:11.480478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354748+0000","last_change":"2024-03-06T07:35:43.354748+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354748+0000","last_undegraded":"2024-03-06T07:35:43.354748+0000","last_fullsized":"2024-03-06T07:35:43.354748+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720668+0000","last_change":"2024-03-06T07:35:43.377991+0000","last_active":"2024-03-06T07:35:45.720668+0000","last_peered":"2024-03-06T07:35:45.720668+0000","last_clean":"2024-03-06T07:35:45.720668+0000","last_became_active":"2024-03-06T07:35:43.377732+0000","last_became_peered":"2024-03-06T07:35:43.377732+0000","last_unstale":"2024-03-06T07:35:45.720668+0000","last_undegraded":"2024-03-06T07:35:45.720668+0000","last_fullsized":"2024-03-06T07:35:45.720668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.997519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720542+0000","last_change":"2024-03-06T07:35:44.415821+0000","last_active":"2024-03-06T07:35:45.720542+0000","last_peered":"2024-03-06T07:35:45.720542+0000","last_clean":"2024-03-06T07:35:45.720542+0000","last_became_active":"2024-03-06T07:35:44.415487+0000","last_became_peered":"2024-03-06T07:35:44.415487+0000","last_unstale":"2024-03-06T07:35:45.720542+0000","last_undegraded":"2024-03-06T07:35:45.720542+0000","last_fullsized":"2024-03-06T07:35:45.720542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:46.609382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719944+0000","last_change":"2024-03-06T07:35:43.385136+0000","last_active":"2024-03-06T07:35:45.719944+0000","last_peered":"2024-03-06T07:35:45.719944+0000","last_clean":"2024-03-06T07:35:45.719944+0000","last_became_active":"2024-03-06T07:35:43.384757+0000","last_became_peered":"2024-03-06T07:35:43.384757+0000","last_unstale":"2024-03-06T07:35:45.719944+0000","last_undegraded":"2024-03-06T07:35:45.719944+0000","last_fullsized":"2024-03-06T07:35:45.719944+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:20.137806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394855+0000","last_change":"2024-03-06T07:35:44.417187+0000","last_active":"2024-03-06T07:35:45.394855+0000","last_peered":"2024-03-06T07:35:45.394855+0000","last_clean":"2024-03-06T07:35:45.394855+0000","last_became_active":"2024-03-06T07:35:44.416892+0000","last_became_peered":"2024-03-06T07:35:44.416892+0000","last_unstale":"2024-03-06T07:35:45.394855+0000","last_undegraded":"2024-03-06T07:35:45.394855+0000","last_fullsized":"2024-03-06T07:35:45.394855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394798+0000","last_change":"2024-03-06T07:35:43.377777+0000","last_active":"2024-03-06T07:35:45.394798+0000","last_peered":"2024-03-06T07:35:45.394798+0000","last_clean":"2024-03-06T07:35:45.394798+0000","last_became_active":"2024-03-06T07:35:43.374868+0000","last_became_peered":"2024-03-06T07:35:43.374868+0000","last_unstale":"2024-03-06T07:35:45.394798+0000","last_undegraded":"2024-03-06T07:35:45.394798+0000","last_fullsized":"2024-03-06T07:35:45.394798+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:57.102370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354203+0000","last_change":"2024-03-06T07:35:43.354203+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354203+0000","last_undegraded":"2024-03-06T07:35:43.354203+0000","last_fullsized":"2024-03-06T07:35:43.354203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557609+0000","last_change":"2024-03-06T07:35:43.368644+0000","last_active":"2024-03-06T07:35:45.557609+0000","last_peered":"2024-03-06T07:35:45.557609+0000","last_clean":"2024-03-06T07:35:45.557609+0000","last_became_active":"2024-03-06T07:35:43.367936+0000","last_became_peered":"2024-03-06T07:35:43.367936+0000","last_unstale":"2024-03-06T07:35:45.557609+0000","last_undegraded":"2024-03-06T07:35:45.557609+0000","last_fullsized":"2024-03-06T07:35:45.557609+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:01.783718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720731+0000","last_change":"2024-03-06T07:35:44.417357+0000","last_active":"2024-03-06T07:35:45.720731+0000","last_peered":"2024-03-06T07:35:45.720731+0000","last_clean":"2024-03-06T07:35:45.720731+0000","last_became_active":"2024-03-06T07:35:44.417021+0000","last_became_peered":"2024-03-06T07:35:44.417021+0000","last_unstale":"2024-03-06T07:35:45.720731+0000","last_undegraded":"2024-03-06T07:35:45.720731+0000","last_fullsized":"2024-03-06T07:35:45.720731+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:11.909643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394493+0000","last_change":"2024-03-06T07:35:43.376770+0000","last_active":"2024-03-06T07:35:45.394493+0000","last_peered":"2024-03-06T07:35:45.394493+0000","last_clean":"2024-03-06T07:35:45.394493+0000","last_became_active":"2024-03-06T07:35:43.376483+0000","last_became_peered":"2024-03-06T07:35:43.376483+0000","last_unstale":"2024-03-06T07:35:45.394493+0000","last_undegraded":"2024-03-06T07:35:45.394493+0000","last_fullsized":"2024-03-06T07:35:45.394493+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:32.527625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399036+0000","last_change":"2024-03-06T07:35:44.408484+0000","last_active":"2024-03-06T07:35:45.399036+0000","last_peered":"2024-03-06T07:35:45.399036+0000","last_clean":"2024-03-06T07:35:45.399036+0000","last_became_active":"2024-03-06T07:35:44.407992+0000","last_became_peered":"2024-03-06T07:35:44.407992+0000","last_unstale":"2024-03-06T07:35:45.399036+0000","last_undegraded":"2024-03-06T07:35:45.399036+0000","last_fullsized":"2024-03-06T07:35:45.399036+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720795+0000","last_change":"2024-03-06T07:35:43.385517+0000","last_active":"2024-03-06T07:35:45.720795+0000","last_peered":"2024-03-06T07:35:45.720795+0000","last_clean":"2024-03-06T07:35:45.720795+0000","last_became_active":"2024-03-06T07:35:43.384210+0000","last_became_peered":"2024-03-06T07:35:43.384210+0000","last_unstale":"2024-03-06T07:35:45.720795+0000","last_undegraded":"2024-03-06T07:35:45.720795+0000","last_fullsized":"2024-03-06T07:35:45.720795+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:11.817793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719634+0000","last_change":"2024-03-06T07:35:44.412773+0000","last_active":"2024-03-06T07:35:45.719634+0000","last_peered":"2024-03-06T07:35:45.719634+0000","last_clean":"2024-03-06T07:35:45.719634+0000","last_became_active":"2024-03-06T07:35:44.411292+0000","last_became_peered":"2024-03-06T07:35:44.411292+0000","last_unstale":"2024-03-06T07:35:45.719634+0000","last_undegraded":"2024-03-06T07:35:45.719634+0000","last_fullsized":"2024-03-06T07:35:45.719634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719424+0000","last_change":"2024-03-06T07:35:43.379581+0000","last_active":"2024-03-06T07:35:45.719424+0000","last_peered":"2024-03-06T07:35:45.719424+0000","last_clean":"2024-03-06T07:35:45.719424+0000","last_became_active":"2024-03-06T07:35:43.379237+0000","last_became_peered":"2024-03-06T07:35:43.379237+0000","last_unstale":"2024-03-06T07:35:45.719424+0000","last_undegraded":"2024-03-06T07:35:45.719424+0000","last_fullsized":"2024-03-06T07:35:45.719424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:32.312256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394668+0000","last_change":"2024-03-06T07:35:44.417512+0000","last_active":"2024-03-06T07:35:45.394668+0000","last_peered":"2024-03-06T07:35:45.394668+0000","last_clean":"2024-03-06T07:35:45.394668+0000","last_became_active":"2024-03-06T07:35:44.416206+0000","last_became_peered":"2024-03-06T07:35:44.416206+0000","last_unstale":"2024-03-06T07:35:45.394668+0000","last_undegraded":"2024-03-06T07:35:45.394668+0000","last_fullsized":"2024-03-06T07:35:45.394668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557543+0000","last_change":"2024-03-06T07:35:43.375969+0000","last_active":"2024-03-06T07:35:45.557543+0000","last_peered":"2024-03-06T07:35:45.557543+0000","last_clean":"2024-03-06T07:35:45.557543+0000","last_became_active":"2024-03-06T07:35:43.375710+0000","last_became_peered":"2024-03-06T07:35:43.375710+0000","last_unstale":"2024-03-06T07:35:45.557543+0000","last_undegraded":"2024-03-06T07:35:45.557543+0000","last_fullsized":"2024-03-06T07:35:45.557543+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:38.782993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720514+0000","last_change":"2024-03-06T07:35:44.414475+0000","last_active":"2024-03-06T07:35:45.720514+0000","last_peered":"2024-03-06T07:35:45.720514+0000","last_clean":"2024-03-06T07:35:45.720514+0000","last_became_active":"2024-03-06T07:35:44.414166+0000","last_became_peered":"2024-03-06T07:35:44.414166+0000","last_unstale":"2024-03-06T07:35:45.720514+0000","last_undegraded":"2024-03-06T07:35:45.720514+0000","last_fullsized":"2024-03-06T07:35:45.720514+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:39.798967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721268+0000","last_change":"2024-03-06T07:35:43.385255+0000","last_active":"2024-03-06T07:35:45.721268+0000","last_peered":"2024-03-06T07:35:45.721268+0000","last_clean":"2024-03-06T07:35:45.721268+0000","last_became_active":"2024-03-06T07:35:43.384945+0000","last_became_peered":"2024-03-06T07:35:43.384945+0000","last_unstale":"2024-03-06T07:35:45.721268+0000","last_undegraded":"2024-03-06T07:35:45.721268+0000","last_fullsized":"2024-03-06T07:35:45.721268+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:41.317784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":369,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719242+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:35:45.719242+0000","last_peered":"2024-03-06T07:35:45.719242+0000","last_clean":"2024-03-06T07:35:45.719242+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:35:45.719242+0000","last_undegraded":"2024-03-06T07:35:45.719242+0000","last_fullsized":"2024-03-06T07:35:45.719242+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2236416,"data_stored":1296790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542246,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1630208,"data_stored":705784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":691097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542247,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1626112,"data_stored":704788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1617920,"data_stored":704172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41660,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330180,"statfs":{"total":96636764160,"available":96594104320,"internally_reserved":0,"allocated":2211840,"data_stored":1294540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:49.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.651+0000 7fe55c600640 1 Processor -- start 2024-03-06T07:35:49.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.652+0000 7fe55c600640 1 -- start start 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.653+0000 7fe55c600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe55415eff0 0x7fe554070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.653+0000 7fe55c600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe55406e4d0 0x7fe55406e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.653+0000 7fe55c600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe554071d30 con 0x7fe5540af1c0 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.653+0000 7fe55c600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55406ee80 con 0x7fe55406e4d0 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.653+0000 7fe55c600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe554071260 con 0x7fe55415eff0 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe559c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe5540af1c0 0x7fe55406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50190/0 (socket says 172.21.15.97:50190) 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe559c00640 1 -- 172.21.15.97:0/1096867734 learned_addr learned my addr 172.21.15.97:0/1096867734 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe55a600640 1 --2- 172.21.15.97:0/1096867734 >> v2:172.21.15.97:3300/0 conn(0x7fe55406e4d0 0x7fe55406e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe559200640 1 --2- 172.21.15.97:0/1096867734 >> v2:172.21.15.135:3300/0 conn(0x7fe55415eff0 0x7fe554070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe55a600640 1 -- 172.21.15.97:0/1096867734 >> v2:172.21.15.135:3300/0 conn(0x7fe55415eff0 msgr2=0x7fe554070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe55a600640 1 --2- 172.21.15.97:0/1096867734 >> v2:172.21.15.135:3300/0 conn(0x7fe55415eff0 0x7fe554070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe55a600640 1 -- 172.21.15.97:0/1096867734 >> v1:172.21.15.97:6789/0 conn(0x7fe5540af1c0 legacy=0x7fe55406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe55a600640 1 -- 172.21.15.97:0/1096867734 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55415f4a0 con 0x7fe55406e4d0 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe559200640 1 --2- 172.21.15.97:0/1096867734 >> v2:172.21.15.135:3300/0 conn(0x7fe55415eff0 0x7fe554070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.654+0000 7fe553e00640 1 -- 172.21.15.97:0/1096867734 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408014692 0 0) 0x7fe548001110 con 0x7fe5540af1c0 2024-03-06T07:35:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.655+0000 7fe55a600640 1 --2- 172.21.15.97:0/1096867734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe55406e4d0 0x7fe55406e940 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009d90 tx=0x7fe54c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4994d321ed10bfe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.656+0000 7fe553e00640 1 -- 172.21.15.97:0/1096867734 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c039070 con 0x7fe55406e4d0 2024-03-06T07:35:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.656+0000 7fe553e00640 1 -- 172.21.15.97:0/1096867734 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c02cc60 con 0x7fe55406e4d0 2024-03-06T07:35:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.656+0000 7fe553e00640 1 -- 172.21.15.97:0/1096867734 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c034660 con 0x7fe55406e4d0 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.656+0000 7fe55c600640 1 -- 172.21.15.97:0/1096867734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe55406e4d0 msgr2=0x7fe55406e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.656+0000 7fe55c600640 1 --2- 172.21.15.97:0/1096867734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe55406e4d0 0x7fe55406e940 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009d90 tx=0x7fe54c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.657+0000 7fe55c600640 1 -- 172.21.15.97:0/1096867734 shutdown_connections 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.657+0000 7fe55c600640 1 --2- 172.21.15.97:0/1096867734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe55406e4d0 0x7fe55406e940 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.657+0000 7fe55c600640 1 --2- 172.21.15.97:0/1096867734 >> v2:172.21.15.135:3300/0 conn(0x7fe55415eff0 0x7fe554070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.657+0000 7fe55c600640 1 -- 172.21.15.97:0/1096867734 >> 172.21.15.97:0/1096867734 conn(0x7fe5540b1730 msgr2=0x7fe5540b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.657+0000 7fe55c600640 1 -- 172.21.15.97:0/1096867734 shutdown_connections 2024-03-06T07:35:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.657+0000 7fe55c600640 1 -- 172.21.15.97:0/1096867734 wait complete. 2024-03-06T07:35:49.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.659+0000 7fe55c600640 1 Processor -- start 2024-03-06T07:35:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.660+0000 7fe55c600640 1 -- start start 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe55c600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe55406e4d0 0x7fe554149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe55c600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 0x7fe5541571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe55c600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe55415eff0 0x7fe554157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe55c600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554071a50 con 0x7fe5540af1c0 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe55c600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554071770 con 0x7fe55406e4d0 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe55c600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b30b0 con 0x7fe55415eff0 2024-03-06T07:35:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.661+0000 7fe559200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 0x7fe5541571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 0x7fe5541571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32888/0 (socket says 172.21.15.97:32888) 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 -- 172.21.15.97:0/2620630506 learned_addr learned my addr 172.21.15.97:0/2620630506 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559c00640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe55406e4d0 0x7fe554149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe55a600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe55415eff0 0x7fe554157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 -- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe55415eff0 msgr2=0x7fe554157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe55415eff0 0x7fe554157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 -- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe55406e4d0 msgr2=0x7fe554149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe55406e4d0 0x7fe554149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 -- 172.21.15.97:0/2620630506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe554149d40 con 0x7fe5540af1c0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.662+0000 7fe559200640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 0x7fe5541571c0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe54000c7f0 tx=0x7fe54000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.663+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540015070 con 0x7fe5540af1c0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.663+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5400077c0 con 0x7fe5540af1c0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.663+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540007b70 con 0x7fe5540af1c0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.663+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5540cc7a0 con 0x7fe5540af1c0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.663+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5540cd040 con 0x7fe5540af1c0 2024-03-06T07:35:49.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.664+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55404f330 con 0x7fe5540af1c0 2024-03-06T07:35:49.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.665+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe540018070 con 0x7fe5540af1c0 2024-03-06T07:35:49.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.666+0000 7fe551600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53406b4d0 0x7fe53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.666+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe54008a220 con 0x7fe5540af1c0 2024-03-06T07:35:49.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.666+0000 7fe559c00640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53406b4d0 0x7fe53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.667+0000 7fe559c00640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53406b4d0 0x7fe53406d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe548000f60 tx=0x7fe548007970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.671+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe540059130 con 0x7fe5540af1c0 2024-03-06T07:35:49.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.861+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5540cba10 con 0x7fe53406b4d0 2024-03-06T07:35:49.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.878+0000 7fe551600640 1 -- 172.21.15.97:0/2620630506 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251818 (secure 0 0 0) 0x7fe5540cba10 con 0x7fe53406b4d0 2024-03-06T07:35:49.882 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:49.884 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53406b4d0 msgr2=0x7fe53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53406b4d0 0x7fe53406d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe548000f60 tx=0x7fe548007970 comp rx=0 tx=0).stop 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 msgr2=0x7fe5541571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 0x7fe5541571c0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe54000c7f0 tx=0x7fe54000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 shutdown_connections 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe53406b4d0 0x7fe53406d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe55415eff0 0x7fe554157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5540af1c0 0x7fe5541571c0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 --2- 172.21.15.97:0/2620630506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe55406e4d0 0x7fe554149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 >> 172.21.15.97:0/2620630506 conn(0x7fe5540b1730 msgr2=0x7fe55406db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 shutdown_connections 2024-03-06T07:35:49.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:49.882+0000 7fe55c600640 1 -- 172.21.15.97:0/2620630506 wait complete. 2024-03-06T07:35:49.904 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-06T07:35:49.051857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304888,"kb_used_data":13836,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754669832,"statfs":{"total":773094113280,"available":772781907968,"internally_reserved":0,"allocated":14168064,"data_stored":6805787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001479"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720739+0000","last_change":"2024-03-06T07:35:44.394272+0000","last_active":"2024-03-06T07:35:45.720739+0000","last_peered":"2024-03-06T07:35:45.720739+0000","last_clean":"2024-03-06T07:35:45.720739+0000","last_became_active":"2024-03-06T07:35:44.393967+0000","last_became_peered":"2024-03-06T07:35:44.393967+0000","last_unstale":"2024-03-06T07:35:45.720739+0000","last_undegraded":"2024-03-06T07:35:45.720739+0000","last_fullsized":"2024-03-06T07:35:45.720739+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:28.599416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394850+0000","last_change":"2024-03-06T07:35:44.392186+0000","last_active":"2024-03-06T07:35:45.394850+0000","last_peered":"2024-03-06T07:35:45.394850+0000","last_clean":"2024-03-06T07:35:45.394850+0000","last_became_active":"2024-03-06T07:35:44.391446+0000","last_became_peered":"2024-03-06T07:35:44.391446+0000","last_unstale":"2024-03-06T07:35:45.394850+0000","last_undegraded":"2024-03-06T07:35:45.394850+0000","last_fullsized":"2024-03-06T07:35:45.394850+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:25.601824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720157+0000","last_change":"2024-03-06T07:35:44.410356+0000","last_active":"2024-03-06T07:35:45.720157+0000","last_peered":"2024-03-06T07:35:45.720157+0000","last_clean":"2024-03-06T07:35:45.720157+0000","last_became_active":"2024-03-06T07:35:44.409633+0000","last_became_peered":"2024-03-06T07:35:44.409633+0000","last_unstale":"2024-03-06T07:35:45.720157+0000","last_undegraded":"2024-03-06T07:35:45.720157+0000","last_fullsized":"2024-03-06T07:35:45.720157+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:12.833768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718970+0000","last_change":"2024-03-06T07:35:44.417975+0000","last_active":"2024-03-06T07:35:45.718970+0000","last_peered":"2024-03-06T07:35:45.718970+0000","last_clean":"2024-03-06T07:35:45.718970+0000","last_became_active":"2024-03-06T07:35:44.417719+0000","last_became_peered":"2024-03-06T07:35:44.417719+0000","last_unstale":"2024-03-06T07:35:45.718970+0000","last_undegraded":"2024-03-06T07:35:45.718970+0000","last_fullsized":"2024-03-06T07:35:45.718970+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:26.008403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721342+0000","last_change":"2024-03-06T07:35:44.415585+0000","last_active":"2024-03-06T07:35:45.721342+0000","last_peered":"2024-03-06T07:35:45.721342+0000","last_clean":"2024-03-06T07:35:45.721342+0000","last_became_active":"2024-03-06T07:35:44.415011+0000","last_became_peered":"2024-03-06T07:35:44.415011+0000","last_unstale":"2024-03-06T07:35:45.721342+0000","last_undegraded":"2024-03-06T07:35:45.721342+0000","last_fullsized":"2024-03-06T07:35:45.721342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354250+0000","last_change":"2024-03-06T07:35:43.354250+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354250+0000","last_undegraded":"2024-03-06T07:35:43.354250+0000","last_fullsized":"2024-03-06T07:35:43.354250+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353927+0000","last_change":"2024-03-06T07:35:43.353927+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353927+0000","last_undegraded":"2024-03-06T07:35:43.353927+0000","last_fullsized":"2024-03-06T07:35:43.353927+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719898+0000","last_change":"2024-03-06T07:35:44.405572+0000","last_active":"2024-03-06T07:35:45.719898+0000","last_peered":"2024-03-06T07:35:45.719898+0000","last_clean":"2024-03-06T07:35:45.719898+0000","last_became_active":"2024-03-06T07:35:44.405225+0000","last_became_peered":"2024-03-06T07:35:44.405225+0000","last_unstale":"2024-03-06T07:35:45.719898+0000","last_undegraded":"2024-03-06T07:35:45.719898+0000","last_fullsized":"2024-03-06T07:35:45.719898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:21.153780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721703+0000","last_change":"2024-03-06T07:35:44.415674+0000","last_active":"2024-03-06T07:35:45.721703+0000","last_peered":"2024-03-06T07:35:45.721703+0000","last_clean":"2024-03-06T07:35:45.721703+0000","last_became_active":"2024-03-06T07:35:44.415198+0000","last_became_peered":"2024-03-06T07:35:44.415198+0000","last_unstale":"2024-03-06T07:35:45.721703+0000","last_undegraded":"2024-03-06T07:35:45.721703+0000","last_fullsized":"2024-03-06T07:35:45.721703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:27.898827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395062+0000","last_change":"2024-03-06T07:35:44.398339+0000","last_active":"2024-03-06T07:35:45.395062+0000","last_peered":"2024-03-06T07:35:45.395062+0000","last_clean":"2024-03-06T07:35:45.395062+0000","last_became_active":"2024-03-06T07:35:44.396519+0000","last_became_peered":"2024-03-06T07:35:44.396519+0000","last_unstale":"2024-03-06T07:35:45.395062+0000","last_undegraded":"2024-03-06T07:35:45.395062+0000","last_fullsized":"2024-03-06T07:35:45.395062+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:48.438390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557893+0000","last_change":"2024-03-06T07:35:44.397455+0000","last_active":"2024-03-06T07:35:45.557893+0000","last_peered":"2024-03-06T07:35:45.557893+0000","last_clean":"2024-03-06T07:35:45.557893+0000","last_became_active":"2024-03-06T07:35:44.397217+0000","last_became_peered":"2024-03-06T07:35:44.397217+0000","last_unstale":"2024-03-06T07:35:45.557893+0000","last_undegraded":"2024-03-06T07:35:45.557893+0000","last_fullsized":"2024-03-06T07:35:45.557893+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:01.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720004+0000","last_change":"2024-03-06T07:35:44.416953+0000","last_active":"2024-03-06T07:35:45.720004+0000","last_peered":"2024-03-06T07:35:45.720004+0000","last_clean":"2024-03-06T07:35:45.720004+0000","last_became_active":"2024-03-06T07:35:44.416659+0000","last_became_peered":"2024-03-06T07:35:44.416659+0000","last_unstale":"2024-03-06T07:35:45.720004+0000","last_undegraded":"2024-03-06T07:35:45.720004+0000","last_fullsized":"2024-03-06T07:35:45.720004+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:31.633974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719022+0000","last_change":"2024-03-06T07:35:44.395498+0000","last_active":"2024-03-06T07:35:45.719022+0000","last_peered":"2024-03-06T07:35:45.719022+0000","last_clean":"2024-03-06T07:35:45.719022+0000","last_became_active":"2024-03-06T07:35:44.395203+0000","last_became_peered":"2024-03-06T07:35:44.395203+0000","last_unstale":"2024-03-06T07:35:45.719022+0000","last_undegraded":"2024-03-06T07:35:45.719022+0000","last_fullsized":"2024-03-06T07:35:45.719022+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:58.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394607+0000","last_change":"2024-03-06T07:35:44.400161+0000","last_active":"2024-03-06T07:35:45.394607+0000","last_peered":"2024-03-06T07:35:45.394607+0000","last_clean":"2024-03-06T07:35:45.394607+0000","last_became_active":"2024-03-06T07:35:44.399845+0000","last_became_peered":"2024-03-06T07:35:44.399845+0000","last_unstale":"2024-03-06T07:35:45.394607+0000","last_undegraded":"2024-03-06T07:35:45.394607+0000","last_fullsized":"2024-03-06T07:35:45.394607+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:33.642819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718604+0000","last_change":"2024-03-06T07:35:44.406110+0000","last_active":"2024-03-06T07:35:45.718604+0000","last_peered":"2024-03-06T07:35:45.718604+0000","last_clean":"2024-03-06T07:35:45.718604+0000","last_became_active":"2024-03-06T07:35:44.405807+0000","last_became_peered":"2024-03-06T07:35:44.405807+0000","last_unstale":"2024-03-06T07:35:45.718604+0000","last_undegraded":"2024-03-06T07:35:45.718604+0000","last_fullsized":"2024-03-06T07:35:45.718604+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:04.519517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719509+0000","last_change":"2024-03-06T07:35:44.395764+0000","last_active":"2024-03-06T07:35:45.719509+0000","last_peered":"2024-03-06T07:35:45.719509+0000","last_clean":"2024-03-06T07:35:45.719509+0000","last_became_active":"2024-03-06T07:35:44.393280+0000","last_became_peered":"2024-03-06T07:35:44.393280+0000","last_unstale":"2024-03-06T07:35:45.719509+0000","last_undegraded":"2024-03-06T07:35:45.719509+0000","last_fullsized":"2024-03-06T07:35:45.719509+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356595+0000","last_change":"2024-03-06T07:35:43.356595+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356595+0000","last_undegraded":"2024-03-06T07:35:43.356595+0000","last_fullsized":"2024-03-06T07:35:43.356595+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720362+0000","last_change":"2024-03-06T07:35:44.395358+0000","last_active":"2024-03-06T07:35:45.720362+0000","last_peered":"2024-03-06T07:35:45.720362+0000","last_clean":"2024-03-06T07:35:45.720362+0000","last_became_active":"2024-03-06T07:35:44.395025+0000","last_became_peered":"2024-03-06T07:35:44.395025+0000","last_unstale":"2024-03-06T07:35:45.720362+0000","last_undegraded":"2024-03-06T07:35:45.720362+0000","last_fullsized":"2024-03-06T07:35:45.720362+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558069+0000","last_change":"2024-03-06T07:35:44.408168+0000","last_active":"2024-03-06T07:35:45.558069+0000","last_peered":"2024-03-06T07:35:45.558069+0000","last_clean":"2024-03-06T07:35:45.558069+0000","last_became_active":"2024-03-06T07:35:44.407536+0000","last_became_peered":"2024-03-06T07:35:44.407536+0000","last_unstale":"2024-03-06T07:35:45.558069+0000","last_undegraded":"2024-03-06T07:35:45.558069+0000","last_fullsized":"2024-03-06T07:35:45.558069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:45.475165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720163+0000","last_change":"2024-03-06T07:35:44.390845+0000","last_active":"2024-03-06T07:35:45.720163+0000","last_peered":"2024-03-06T07:35:45.720163+0000","last_clean":"2024-03-06T07:35:45.720163+0000","last_became_active":"2024-03-06T07:35:44.388466+0000","last_became_peered":"2024-03-06T07:35:44.388466+0000","last_unstale":"2024-03-06T07:35:45.720163+0000","last_undegraded":"2024-03-06T07:35:45.720163+0000","last_fullsized":"2024-03-06T07:35:45.720163+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721001+0000","last_change":"2024-03-06T07:35:44.415326+0000","last_active":"2024-03-06T07:35:45.721001+0000","last_peered":"2024-03-06T07:35:45.721001+0000","last_clean":"2024-03-06T07:35:45.721001+0000","last_became_active":"2024-03-06T07:35:44.414356+0000","last_became_peered":"2024-03-06T07:35:44.414356+0000","last_unstale":"2024-03-06T07:35:45.721001+0000","last_undegraded":"2024-03-06T07:35:45.721001+0000","last_fullsized":"2024-03-06T07:35:45.721001+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557466+0000","last_change":"2024-03-06T07:35:44.394518+0000","last_active":"2024-03-06T07:35:45.557466+0000","last_peered":"2024-03-06T07:35:45.557466+0000","last_clean":"2024-03-06T07:35:45.557466+0000","last_became_active":"2024-03-06T07:35:44.392743+0000","last_became_peered":"2024-03-06T07:35:44.392743+0000","last_unstale":"2024-03-06T07:35:45.557466+0000","last_undegraded":"2024-03-06T07:35:45.557466+0000","last_fullsized":"2024-03-06T07:35:45.557466+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:34.320907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720515+0000","last_change":"2024-03-06T07:35:44.417926+0000","last_active":"2024-03-06T07:35:45.720515+0000","last_peered":"2024-03-06T07:35:45.720515+0000","last_clean":"2024-03-06T07:35:45.720515+0000","last_became_active":"2024-03-06T07:35:44.415414+0000","last_became_peered":"2024-03-06T07:35:44.415414+0000","last_unstale":"2024-03-06T07:35:45.720515+0000","last_undegraded":"2024-03-06T07:35:45.720515+0000","last_fullsized":"2024-03-06T07:35:45.720515+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:14.302817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557430+0000","last_change":"2024-03-06T07:35:44.408475+0000","last_active":"2024-03-06T07:35:45.557430+0000","last_peered":"2024-03-06T07:35:45.557430+0000","last_clean":"2024-03-06T07:35:45.557430+0000","last_became_active":"2024-03-06T07:35:44.408227+0000","last_became_peered":"2024-03-06T07:35:44.408227+0000","last_unstale":"2024-03-06T07:35:45.557430+0000","last_undegraded":"2024-03-06T07:35:45.557430+0000","last_fullsized":"2024-03-06T07:35:45.557430+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395196+0000","last_change":"2024-03-06T07:35:44.416789+0000","last_active":"2024-03-06T07:35:45.395196+0000","last_peered":"2024-03-06T07:35:45.395196+0000","last_clean":"2024-03-06T07:35:45.395196+0000","last_became_active":"2024-03-06T07:35:44.416274+0000","last_became_peered":"2024-03-06T07:35:44.416274+0000","last_unstale":"2024-03-06T07:35:45.395196+0000","last_undegraded":"2024-03-06T07:35:45.395196+0000","last_fullsized":"2024-03-06T07:35:45.395196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:47.395736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720627+0000","last_change":"2024-03-06T07:35:44.399980+0000","last_active":"2024-03-06T07:35:45.720627+0000","last_peered":"2024-03-06T07:35:45.720627+0000","last_clean":"2024-03-06T07:35:45.720627+0000","last_became_active":"2024-03-06T07:35:44.399328+0000","last_became_peered":"2024-03-06T07:35:44.399328+0000","last_unstale":"2024-03-06T07:35:45.720627+0000","last_undegraded":"2024-03-06T07:35:45.720627+0000","last_fullsized":"2024-03-06T07:35:45.720627+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:43.875737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719802+0000","last_change":"2024-03-06T07:35:44.394219+0000","last_active":"2024-03-06T07:35:45.719802+0000","last_peered":"2024-03-06T07:35:45.719802+0000","last_clean":"2024-03-06T07:35:45.719802+0000","last_became_active":"2024-03-06T07:35:44.393893+0000","last_became_peered":"2024-03-06T07:35:44.393893+0000","last_unstale":"2024-03-06T07:35:45.719802+0000","last_undegraded":"2024-03-06T07:35:45.719802+0000","last_fullsized":"2024-03-06T07:35:45.719802+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:18.620474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398830+0000","last_change":"2024-03-06T07:35:44.403178+0000","last_active":"2024-03-06T07:35:45.398830+0000","last_peered":"2024-03-06T07:35:45.398830+0000","last_clean":"2024-03-06T07:35:45.398830+0000","last_became_active":"2024-03-06T07:35:44.402903+0000","last_became_peered":"2024-03-06T07:35:44.402903+0000","last_unstale":"2024-03-06T07:35:45.398830+0000","last_undegraded":"2024-03-06T07:35:45.398830+0000","last_fullsized":"2024-03-06T07:35:45.398830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719226+0000","last_change":"2024-03-06T07:35:44.415700+0000","last_active":"2024-03-06T07:35:45.719226+0000","last_peered":"2024-03-06T07:35:45.719226+0000","last_clean":"2024-03-06T07:35:45.719226+0000","last_became_active":"2024-03-06T07:35:44.415400+0000","last_became_peered":"2024-03-06T07:35:44.415400+0000","last_unstale":"2024-03-06T07:35:45.719226+0000","last_undegraded":"2024-03-06T07:35:45.719226+0000","last_fullsized":"2024-03-06T07:35:45.719226+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:56.935549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719196+0000","last_change":"2024-03-06T07:35:44.406337+0000","last_active":"2024-03-06T07:35:45.719196+0000","last_peered":"2024-03-06T07:35:45.719196+0000","last_clean":"2024-03-06T07:35:45.719196+0000","last_became_active":"2024-03-06T07:35:44.405995+0000","last_became_peered":"2024-03-06T07:35:44.405995+0000","last_unstale":"2024-03-06T07:35:45.719196+0000","last_undegraded":"2024-03-06T07:35:45.719196+0000","last_fullsized":"2024-03-06T07:35:45.719196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:51.011567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719669+0000","last_change":"2024-03-06T07:35:44.415228+0000","last_active":"2024-03-06T07:35:45.719669+0000","last_peered":"2024-03-06T07:35:45.719669+0000","last_clean":"2024-03-06T07:35:45.719669+0000","last_became_active":"2024-03-06T07:35:44.414418+0000","last_became_peered":"2024-03-06T07:35:44.414418+0000","last_unstale":"2024-03-06T07:35:45.719669+0000","last_undegraded":"2024-03-06T07:35:45.719669+0000","last_fullsized":"2024-03-06T07:35:45.719669+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:25.327417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353912+0000","last_change":"2024-03-06T07:35:43.353912+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353912+0000","last_undegraded":"2024-03-06T07:35:43.353912+0000","last_fullsized":"2024-03-06T07:35:43.353912+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720567+0000","last_change":"2024-03-06T07:35:44.408316+0000","last_active":"2024-03-06T07:35:45.720567+0000","last_peered":"2024-03-06T07:35:45.720567+0000","last_clean":"2024-03-06T07:35:45.720567+0000","last_became_active":"2024-03-06T07:35:44.407617+0000","last_became_peered":"2024-03-06T07:35:44.407617+0000","last_unstale":"2024-03-06T07:35:45.720567+0000","last_undegraded":"2024-03-06T07:35:45.720567+0000","last_fullsized":"2024-03-06T07:35:45.720567+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:09.791568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722044+0000","last_change":"2024-03-06T07:35:43.384588+0000","last_active":"2024-03-06T07:35:45.722044+0000","last_peered":"2024-03-06T07:35:45.722044+0000","last_clean":"2024-03-06T07:35:45.722044+0000","last_became_active":"2024-03-06T07:35:43.384278+0000","last_became_peered":"2024-03-06T07:35:43.384278+0000","last_unstale":"2024-03-06T07:35:45.722044+0000","last_undegraded":"2024-03-06T07:35:45.722044+0000","last_fullsized":"2024-03-06T07:35:45.722044+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:30.617999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722113+0000","last_change":"2024-03-06T07:35:44.408075+0000","last_active":"2024-03-06T07:35:45.722113+0000","last_peered":"2024-03-06T07:35:45.722113+0000","last_clean":"2024-03-06T07:35:45.722113+0000","last_became_active":"2024-03-06T07:35:44.407778+0000","last_became_peered":"2024-03-06T07:35:44.407778+0000","last_unstale":"2024-03-06T07:35:45.722113+0000","last_undegraded":"2024-03-06T07:35:45.722113+0000","last_fullsized":"2024-03-06T07:35:45.722113+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:33.972464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399395+0000","last_change":"2024-03-06T07:35:43.378158+0000","last_active":"2024-03-06T07:35:45.399395+0000","last_peered":"2024-03-06T07:35:45.399395+0000","last_clean":"2024-03-06T07:35:45.399395+0000","last_became_active":"2024-03-06T07:35:43.377863+0000","last_became_peered":"2024-03-06T07:35:43.377863+0000","last_unstale":"2024-03-06T07:35:45.399395+0000","last_undegraded":"2024-03-06T07:35:45.399395+0000","last_fullsized":"2024-03-06T07:35:45.399395+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:27.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720318+0000","last_change":"2024-03-06T07:35:44.419353+0000","last_active":"2024-03-06T07:35:45.720318+0000","last_peered":"2024-03-06T07:35:45.720318+0000","last_clean":"2024-03-06T07:35:45.720318+0000","last_became_active":"2024-03-06T07:35:44.418396+0000","last_became_peered":"2024-03-06T07:35:44.418396+0000","last_unstale":"2024-03-06T07:35:45.720318+0000","last_undegraded":"2024-03-06T07:35:45.720318+0000","last_fullsized":"2024-03-06T07:35:45.720318+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:12.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558056+0000","last_change":"2024-03-06T07:35:43.375835+0000","last_active":"2024-03-06T07:35:45.558056+0000","last_peered":"2024-03-06T07:35:45.558056+0000","last_clean":"2024-03-06T07:35:45.558056+0000","last_became_active":"2024-03-06T07:35:43.375546+0000","last_became_peered":"2024-03-06T07:35:43.375546+0000","last_unstale":"2024-03-06T07:35:45.558056+0000","last_undegraded":"2024-03-06T07:35:45.558056+0000","last_fullsized":"2024-03-06T07:35:45.558056+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:44.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721668+0000","last_change":"2024-03-06T07:35:44.411183+0000","last_active":"2024-03-06T07:35:45.721668+0000","last_peered":"2024-03-06T07:35:45.721668+0000","last_clean":"2024-03-06T07:35:45.721668+0000","last_became_active":"2024-03-06T07:35:44.410909+0000","last_became_peered":"2024-03-06T07:35:44.410909+0000","last_unstale":"2024-03-06T07:35:45.721668+0000","last_undegraded":"2024-03-06T07:35:45.721668+0000","last_fullsized":"2024-03-06T07:35:45.721668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394928+0000","last_change":"2024-03-06T07:35:43.368831+0000","last_active":"2024-03-06T07:35:45.394928+0000","last_peered":"2024-03-06T07:35:45.394928+0000","last_clean":"2024-03-06T07:35:45.394928+0000","last_became_active":"2024-03-06T07:35:43.368516+0000","last_became_peered":"2024-03-06T07:35:43.368516+0000","last_unstale":"2024-03-06T07:35:45.394928+0000","last_undegraded":"2024-03-06T07:35:45.394928+0000","last_fullsized":"2024-03-06T07:35:45.394928+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:24.992428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719450+0000","last_change":"2024-03-06T07:35:44.384339+0000","last_active":"2024-03-06T07:35:45.719450+0000","last_peered":"2024-03-06T07:35:45.719450+0000","last_clean":"2024-03-06T07:35:45.719450+0000","last_became_active":"2024-03-06T07:35:44.384031+0000","last_became_peered":"2024-03-06T07:35:44.384031+0000","last_unstale":"2024-03-06T07:35:45.719450+0000","last_undegraded":"2024-03-06T07:35:45.719450+0000","last_fullsized":"2024-03-06T07:35:45.719450+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394361+0000","last_change":"2024-03-06T07:35:43.373948+0000","last_active":"2024-03-06T07:35:45.394361+0000","last_peered":"2024-03-06T07:35:45.394361+0000","last_clean":"2024-03-06T07:35:45.394361+0000","last_became_active":"2024-03-06T07:35:43.373631+0000","last_became_peered":"2024-03-06T07:35:43.373631+0000","last_unstale":"2024-03-06T07:35:45.394361+0000","last_undegraded":"2024-03-06T07:35:45.394361+0000","last_fullsized":"2024-03-06T07:35:45.394361+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:51.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356413+0000","last_change":"2024-03-06T07:35:43.356413+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356413+0000","last_undegraded":"2024-03-06T07:35:43.356413+0000","last_fullsized":"2024-03-06T07:35:43.356413+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719623+0000","last_change":"2024-03-06T07:35:43.370282+0000","last_active":"2024-03-06T07:35:45.719623+0000","last_peered":"2024-03-06T07:35:45.719623+0000","last_clean":"2024-03-06T07:35:45.719623+0000","last_became_active":"2024-03-06T07:35:43.369383+0000","last_became_peered":"2024-03-06T07:35:43.369383+0000","last_unstale":"2024-03-06T07:35:45.719623+0000","last_undegraded":"2024-03-06T07:35:45.719623+0000","last_fullsized":"2024-03-06T07:35:45.719623+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:02.163748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720319+0000","last_change":"2024-03-06T07:35:44.405369+0000","last_active":"2024-03-06T07:35:45.720319+0000","last_peered":"2024-03-06T07:35:45.720319+0000","last_clean":"2024-03-06T07:35:45.720319+0000","last_became_active":"2024-03-06T07:35:44.405046+0000","last_became_peered":"2024-03-06T07:35:44.405046+0000","last_unstale":"2024-03-06T07:35:45.720319+0000","last_undegraded":"2024-03-06T07:35:45.720319+0000","last_fullsized":"2024-03-06T07:35:45.720319+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:27.612553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.381678+0000","last_change":"2024-03-06T07:35:43.380112+0000","last_active":"2024-03-06T07:35:43.381678+0000","last_peered":"2024-03-06T07:35:43.381678+0000","last_clean":"2024-03-06T07:35:43.381678+0000","last_became_active":"2024-03-06T07:35:43.378931+0000","last_became_peered":"2024-03-06T07:35:43.378931+0000","last_unstale":"2024-03-06T07:35:43.381678+0000","last_undegraded":"2024-03-06T07:35:43.381678+0000","last_fullsized":"2024-03-06T07:35:43.381678+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:12.580678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394619+0000","last_change":"2024-03-06T07:35:44.382654+0000","last_active":"2024-03-06T07:35:45.394619+0000","last_peered":"2024-03-06T07:35:45.394619+0000","last_clean":"2024-03-06T07:35:45.394619+0000","last_became_active":"2024-03-06T07:35:44.382389+0000","last_became_peered":"2024-03-06T07:35:44.382389+0000","last_unstale":"2024-03-06T07:35:45.394619+0000","last_undegraded":"2024-03-06T07:35:45.394619+0000","last_fullsized":"2024-03-06T07:35:45.394619+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394363+0000","last_change":"2024-03-06T07:35:43.376342+0000","last_active":"2024-03-06T07:35:45.394363+0000","last_peered":"2024-03-06T07:35:45.394363+0000","last_clean":"2024-03-06T07:35:45.394363+0000","last_became_active":"2024-03-06T07:35:43.374474+0000","last_became_peered":"2024-03-06T07:35:43.374474+0000","last_unstale":"2024-03-06T07:35:45.394363+0000","last_undegraded":"2024-03-06T07:35:45.394363+0000","last_fullsized":"2024-03-06T07:35:45.394363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:03.503542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720355+0000","last_change":"2024-03-06T07:35:44.402234+0000","last_active":"2024-03-06T07:35:45.720355+0000","last_peered":"2024-03-06T07:35:45.720355+0000","last_clean":"2024-03-06T07:35:45.720355+0000","last_became_active":"2024-03-06T07:35:44.401505+0000","last_became_peered":"2024-03-06T07:35:44.401505+0000","last_unstale":"2024-03-06T07:35:45.720355+0000","last_undegraded":"2024-03-06T07:35:45.720355+0000","last_fullsized":"2024-03-06T07:35:45.720355+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:22.846996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401654+0000","last_change":"2024-03-06T07:35:43.384674+0000","last_active":"2024-03-06T07:35:45.401654+0000","last_peered":"2024-03-06T07:35:45.401654+0000","last_clean":"2024-03-06T07:35:45.401654+0000","last_became_active":"2024-03-06T07:35:43.384373+0000","last_became_peered":"2024-03-06T07:35:43.384373+0000","last_unstale":"2024-03-06T07:35:45.401654+0000","last_undegraded":"2024-03-06T07:35:45.401654+0000","last_fullsized":"2024-03-06T07:35:45.401654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:47.747507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400796+0000","last_change":"2024-03-06T07:35:44.400940+0000","last_active":"2024-03-06T07:35:45.400796+0000","last_peered":"2024-03-06T07:35:45.400796+0000","last_clean":"2024-03-06T07:35:45.400796+0000","last_became_active":"2024-03-06T07:35:44.400650+0000","last_became_peered":"2024-03-06T07:35:44.400650+0000","last_unstale":"2024-03-06T07:35:45.400796+0000","last_undegraded":"2024-03-06T07:35:45.400796+0000","last_fullsized":"2024-03-06T07:35:45.400796+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720656+0000","last_change":"2024-03-06T07:35:43.381659+0000","last_active":"2024-03-06T07:35:45.720656+0000","last_peered":"2024-03-06T07:35:45.720656+0000","last_clean":"2024-03-06T07:35:45.720656+0000","last_became_active":"2024-03-06T07:35:43.381344+0000","last_became_peered":"2024-03-06T07:35:43.381344+0000","last_unstale":"2024-03-06T07:35:45.720656+0000","last_undegraded":"2024-03-06T07:35:45.720656+0000","last_fullsized":"2024-03-06T07:35:45.720656+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:48.989771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720762+0000","last_change":"2024-03-06T07:35:44.420113+0000","last_active":"2024-03-06T07:35:45.720762+0000","last_peered":"2024-03-06T07:35:45.720762+0000","last_clean":"2024-03-06T07:35:45.720762+0000","last_became_active":"2024-03-06T07:35:44.418765+0000","last_became_peered":"2024-03-06T07:35:44.418765+0000","last_unstale":"2024-03-06T07:35:45.720762+0000","last_undegraded":"2024-03-06T07:35:45.720762+0000","last_fullsized":"2024-03-06T07:35:45.720762+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"129'3","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398749+0000","last_change":"2024-03-06T07:35:43.378206+0000","last_active":"2024-03-06T07:35:45.398749+0000","last_peered":"2024-03-06T07:35:45.398749+0000","last_clean":"2024-03-06T07:35:45.398749+0000","last_became_active":"2024-03-06T07:35:43.377916+0000","last_became_peered":"2024-03-06T07:35:43.377916+0000","last_unstale":"2024-03-06T07:35:45.398749+0000","last_undegraded":"2024-03-06T07:35:45.398749+0000","last_fullsized":"2024-03-06T07:35:45.398749+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:21.831021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722052+0000","last_change":"2024-03-06T07:35:44.416806+0000","last_active":"2024-03-06T07:35:45.722052+0000","last_peered":"2024-03-06T07:35:45.722052+0000","last_clean":"2024-03-06T07:35:45.722052+0000","last_became_active":"2024-03-06T07:35:44.416506+0000","last_became_peered":"2024-03-06T07:35:44.416506+0000","last_unstale":"2024-03-06T07:35:45.722052+0000","last_undegraded":"2024-03-06T07:35:45.722052+0000","last_fullsized":"2024-03-06T07:35:45.722052+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"129'6","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721134+0000","last_change":"2024-03-06T07:35:43.379952+0000","last_active":"2024-03-06T07:35:45.721134+0000","last_peered":"2024-03-06T07:35:45.721134+0000","last_clean":"2024-03-06T07:35:45.721134+0000","last_became_active":"2024-03-06T07:35:43.379541+0000","last_became_peered":"2024-03-06T07:35:43.379541+0000","last_unstale":"2024-03-06T07:35:45.721134+0000","last_undegraded":"2024-03-06T07:35:45.721134+0000","last_fullsized":"2024-03-06T07:35:45.721134+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.626844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719234+0000","last_change":"2024-03-06T07:35:44.400442+0000","last_active":"2024-03-06T07:35:45.719234+0000","last_peered":"2024-03-06T07:35:45.719234+0000","last_clean":"2024-03-06T07:35:45.719234+0000","last_became_active":"2024-03-06T07:35:44.399865+0000","last_became_peered":"2024-03-06T07:35:44.399865+0000","last_unstale":"2024-03-06T07:35:45.719234+0000","last_undegraded":"2024-03-06T07:35:45.719234+0000","last_fullsized":"2024-03-06T07:35:45.719234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:26.662064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719793+0000","last_change":"2024-03-06T07:35:43.372351+0000","last_active":"2024-03-06T07:35:45.719793+0000","last_peered":"2024-03-06T07:35:45.719793+0000","last_clean":"2024-03-06T07:35:45.719793+0000","last_became_active":"2024-03-06T07:35:43.372020+0000","last_became_peered":"2024-03-06T07:35:43.372020+0000","last_unstale":"2024-03-06T07:35:45.719793+0000","last_undegraded":"2024-03-06T07:35:45.719793+0000","last_fullsized":"2024-03-06T07:35:45.719793+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:11.361058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394255+0000","last_change":"2024-03-06T07:35:44.412709+0000","last_active":"2024-03-06T07:35:45.394255+0000","last_peered":"2024-03-06T07:35:45.394255+0000","last_clean":"2024-03-06T07:35:45.394255+0000","last_became_active":"2024-03-06T07:35:44.411842+0000","last_became_peered":"2024-03-06T07:35:44.411842+0000","last_unstale":"2024-03-06T07:35:45.394255+0000","last_undegraded":"2024-03-06T07:35:45.394255+0000","last_fullsized":"2024-03-06T07:35:45.394255+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:04.234723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394759+0000","last_change":"2024-03-06T07:35:43.380748+0000","last_active":"2024-03-06T07:35:45.394759+0000","last_peered":"2024-03-06T07:35:45.394759+0000","last_clean":"2024-03-06T07:35:45.394759+0000","last_became_active":"2024-03-06T07:35:43.380512+0000","last_became_peered":"2024-03-06T07:35:43.380512+0000","last_unstale":"2024-03-06T07:35:45.394759+0000","last_undegraded":"2024-03-06T07:35:45.394759+0000","last_fullsized":"2024-03-06T07:35:45.394759+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:02.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720212+0000","last_change":"2024-03-06T07:35:44.416049+0000","last_active":"2024-03-06T07:35:45.720212+0000","last_peered":"2024-03-06T07:35:45.720212+0000","last_clean":"2024-03-06T07:35:45.720212+0000","last_became_active":"2024-03-06T07:35:44.415779+0000","last_became_peered":"2024-03-06T07:35:44.415779+0000","last_unstale":"2024-03-06T07:35:45.720212+0000","last_undegraded":"2024-03-06T07:35:45.720212+0000","last_fullsized":"2024-03-06T07:35:45.720212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:42.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":22,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720702+0000","last_change":"2024-03-06T07:35:43.375271+0000","last_active":"2024-03-06T07:35:45.720702+0000","last_peered":"2024-03-06T07:35:45.720702+0000","last_clean":"2024-03-06T07:35:45.720702+0000","last_became_active":"2024-03-06T07:35:43.374378+0000","last_became_peered":"2024-03-06T07:35:43.374378+0000","last_unstale":"2024-03-06T07:35:45.720702+0000","last_undegraded":"2024-03-06T07:35:45.720702+0000","last_fullsized":"2024-03-06T07:35:45.720702+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:47.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721573+0000","last_change":"2024-03-06T07:35:44.403410+0000","last_active":"2024-03-06T07:35:45.721573+0000","last_peered":"2024-03-06T07:35:45.721573+0000","last_clean":"2024-03-06T07:35:45.721573+0000","last_became_active":"2024-03-06T07:35:44.402014+0000","last_became_peered":"2024-03-06T07:35:44.402014+0000","last_unstale":"2024-03-06T07:35:45.721573+0000","last_undegraded":"2024-03-06T07:35:45.721573+0000","last_fullsized":"2024-03-06T07:35:45.721573+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:50.005746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401803+0000","last_change":"2024-03-06T07:35:43.381126+0000","last_active":"2024-03-06T07:35:45.401803+0000","last_peered":"2024-03-06T07:35:45.401803+0000","last_clean":"2024-03-06T07:35:45.401803+0000","last_became_active":"2024-03-06T07:35:43.380688+0000","last_became_peered":"2024-03-06T07:35:43.380688+0000","last_unstale":"2024-03-06T07:35:45.401803+0000","last_undegraded":"2024-03-06T07:35:45.401803+0000","last_fullsized":"2024-03-06T07:35:45.401803+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:31.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401213+0000","last_change":"2024-03-06T07:35:44.412003+0000","last_active":"2024-03-06T07:35:45.401213+0000","last_peered":"2024-03-06T07:35:45.401213+0000","last_clean":"2024-03-06T07:35:45.401213+0000","last_became_active":"2024-03-06T07:35:44.409915+0000","last_became_peered":"2024-03-06T07:35:44.409915+0000","last_unstale":"2024-03-06T07:35:45.401213+0000","last_undegraded":"2024-03-06T07:35:45.401213+0000","last_fullsized":"2024-03-06T07:35:45.401213+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557877+0000","last_change":"2024-03-06T07:35:43.382676+0000","last_active":"2024-03-06T07:35:45.557877+0000","last_peered":"2024-03-06T07:35:45.557877+0000","last_clean":"2024-03-06T07:35:45.557877+0000","last_became_active":"2024-03-06T07:35:43.380914+0000","last_became_peered":"2024-03-06T07:35:43.380914+0000","last_unstale":"2024-03-06T07:35:45.557877+0000","last_undegraded":"2024-03-06T07:35:45.557877+0000","last_fullsized":"2024-03-06T07:35:45.557877+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.893669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720870+0000","last_change":"2024-03-06T07:35:44.400453+0000","last_active":"2024-03-06T07:35:45.720870+0000","last_peered":"2024-03-06T07:35:45.720870+0000","last_clean":"2024-03-06T07:35:45.720870+0000","last_became_active":"2024-03-06T07:35:44.399501+0000","last_became_peered":"2024-03-06T07:35:44.399501+0000","last_unstale":"2024-03-06T07:35:45.720870+0000","last_undegraded":"2024-03-06T07:35:45.720870+0000","last_fullsized":"2024-03-06T07:35:45.720870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.377445+0000","last_change":"2024-03-06T07:35:43.375643+0000","last_active":"2024-03-06T07:35:43.377445+0000","last_peered":"2024-03-06T07:35:43.377445+0000","last_clean":"2024-03-06T07:35:43.377445+0000","last_became_active":"2024-03-06T07:35:43.375368+0000","last_became_peered":"2024-03-06T07:35:43.375368+0000","last_unstale":"2024-03-06T07:35:43.377445+0000","last_undegraded":"2024-03-06T07:35:43.377445+0000","last_fullsized":"2024-03-06T07:35:43.377445+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:43.218935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394708+0000","last_change":"2024-03-06T07:35:44.387752+0000","last_active":"2024-03-06T07:35:45.394708+0000","last_peered":"2024-03-06T07:35:45.394708+0000","last_clean":"2024-03-06T07:35:45.394708+0000","last_became_active":"2024-03-06T07:35:44.386625+0000","last_became_peered":"2024-03-06T07:35:44.386625+0000","last_unstale":"2024-03-06T07:35:45.394708+0000","last_undegraded":"2024-03-06T07:35:45.394708+0000","last_fullsized":"2024-03-06T07:35:45.394708+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398845+0000","last_change":"2024-03-06T07:35:43.378109+0000","last_active":"2024-03-06T07:35:45.398845+0000","last_peered":"2024-03-06T07:35:45.398845+0000","last_clean":"2024-03-06T07:35:45.398845+0000","last_became_active":"2024-03-06T07:35:43.377689+0000","last_became_peered":"2024-03-06T07:35:43.377689+0000","last_unstale":"2024-03-06T07:35:45.398845+0000","last_undegraded":"2024-03-06T07:35:45.398845+0000","last_fullsized":"2024-03-06T07:35:45.398845+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:17.604500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719574+0000","last_change":"2024-03-06T07:35:44.406714+0000","last_active":"2024-03-06T07:35:45.719574+0000","last_peered":"2024-03-06T07:35:45.719574+0000","last_clean":"2024-03-06T07:35:45.719574+0000","last_became_active":"2024-03-06T07:35:44.406421+0000","last_became_peered":"2024-03-06T07:35:44.406421+0000","last_unstale":"2024-03-06T07:35:45.719574+0000","last_undegraded":"2024-03-06T07:35:45.719574+0000","last_fullsized":"2024-03-06T07:35:45.719574+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:49.699470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399284+0000","last_change":"2024-03-06T07:35:43.385621+0000","last_active":"2024-03-06T07:35:45.399284+0000","last_peered":"2024-03-06T07:35:45.399284+0000","last_clean":"2024-03-06T07:35:45.399284+0000","last_became_active":"2024-03-06T07:35:43.383393+0000","last_became_peered":"2024-03-06T07:35:43.383393+0000","last_unstale":"2024-03-06T07:35:45.399284+0000","last_undegraded":"2024-03-06T07:35:45.399284+0000","last_fullsized":"2024-03-06T07:35:45.399284+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:45.593408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354223+0000","last_change":"2024-03-06T07:35:43.354223+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354223+0000","last_undegraded":"2024-03-06T07:35:43.354223+0000","last_fullsized":"2024-03-06T07:35:43.354223+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719668+0000","last_change":"2024-03-06T07:35:43.381958+0000","last_active":"2024-03-06T07:35:45.719668+0000","last_peered":"2024-03-06T07:35:45.719668+0000","last_clean":"2024-03-06T07:35:45.719668+0000","last_became_active":"2024-03-06T07:35:43.381369+0000","last_became_peered":"2024-03-06T07:35:43.381369+0000","last_unstale":"2024-03-06T07:35:45.719668+0000","last_undegraded":"2024-03-06T07:35:45.719668+0000","last_fullsized":"2024-03-06T07:35:45.719668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:26.882853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394343+0000","last_change":"2024-03-06T07:35:44.386726+0000","last_active":"2024-03-06T07:35:45.394343+0000","last_peered":"2024-03-06T07:35:45.394343+0000","last_clean":"2024-03-06T07:35:45.394343+0000","last_became_active":"2024-03-06T07:35:44.385066+0000","last_became_peered":"2024-03-06T07:35:44.385066+0000","last_unstale":"2024-03-06T07:35:45.394343+0000","last_undegraded":"2024-03-06T07:35:45.394343+0000","last_fullsized":"2024-03-06T07:35:45.394343+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:29.921786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719327+0000","last_change":"2024-03-06T07:35:43.381967+0000","last_active":"2024-03-06T07:35:45.719327+0000","last_peered":"2024-03-06T07:35:45.719327+0000","last_clean":"2024-03-06T07:35:45.719327+0000","last_became_active":"2024-03-06T07:35:43.381593+0000","last_became_peered":"2024-03-06T07:35:43.381593+0000","last_unstale":"2024-03-06T07:35:45.719327+0000","last_undegraded":"2024-03-06T07:35:45.719327+0000","last_fullsized":"2024-03-06T07:35:45.719327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:32.956490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719534+0000","last_change":"2024-03-06T07:35:44.416923+0000","last_active":"2024-03-06T07:35:45.719534+0000","last_peered":"2024-03-06T07:35:45.719534+0000","last_clean":"2024-03-06T07:35:45.719534+0000","last_became_active":"2024-03-06T07:35:44.414762+0000","last_became_peered":"2024-03-06T07:35:44.414762+0000","last_unstale":"2024-03-06T07:35:45.719534+0000","last_undegraded":"2024-03-06T07:35:45.719534+0000","last_fullsized":"2024-03-06T07:35:45.719534+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:18.903803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557365+0000","last_change":"2024-03-06T07:35:43.378687+0000","last_active":"2024-03-06T07:35:45.557365+0000","last_peered":"2024-03-06T07:35:45.557365+0000","last_clean":"2024-03-06T07:35:45.557365+0000","last_became_active":"2024-03-06T07:35:43.378095+0000","last_became_peered":"2024-03-06T07:35:43.378095+0000","last_unstale":"2024-03-06T07:35:45.557365+0000","last_undegraded":"2024-03-06T07:35:45.557365+0000","last_fullsized":"2024-03-06T07:35:45.557365+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:11.480478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354748+0000","last_change":"2024-03-06T07:35:43.354748+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354748+0000","last_undegraded":"2024-03-06T07:35:43.354748+0000","last_fullsized":"2024-03-06T07:35:43.354748+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720668+0000","last_change":"2024-03-06T07:35:43.377991+0000","last_active":"2024-03-06T07:35:45.720668+0000","last_peered":"2024-03-06T07:35:45.720668+0000","last_clean":"2024-03-06T07:35:45.720668+0000","last_became_active":"2024-03-06T07:35:43.377732+0000","last_became_peered":"2024-03-06T07:35:43.377732+0000","last_unstale":"2024-03-06T07:35:45.720668+0000","last_undegraded":"2024-03-06T07:35:45.720668+0000","last_fullsized":"2024-03-06T07:35:45.720668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.997519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720542+0000","last_change":"2024-03-06T07:35:44.415821+0000","last_active":"2024-03-06T07:35:45.720542+0000","last_peered":"2024-03-06T07:35:45.720542+0000","last_clean":"2024-03-06T07:35:45.720542+0000","last_became_active":"2024-03-06T07:35:44.415487+0000","last_became_peered":"2024-03-06T07:35:44.415487+0000","last_unstale":"2024-03-06T07:35:45.720542+0000","last_undegraded":"2024-03-06T07:35:45.720542+0000","last_fullsized":"2024-03-06T07:35:45.720542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:46.609382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719944+0000","last_change":"2024-03-06T07:35:43.385136+0000","last_active":"2024-03-06T07:35:45.719944+0000","last_peered":"2024-03-06T07:35:45.719944+0000","last_clean":"2024-03-06T07:35:45.719944+0000","last_became_active":"2024-03-06T07:35:43.384757+0000","last_became_peered":"2024-03-06T07:35:43.384757+0000","last_unstale":"2024-03-06T07:35:45.719944+0000","last_undegraded":"2024-03-06T07:35:45.719944+0000","last_fullsized":"2024-03-06T07:35:45.719944+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:20.137806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394855+0000","last_change":"2024-03-06T07:35:44.417187+0000","last_active":"2024-03-06T07:35:45.394855+0000","last_peered":"2024-03-06T07:35:45.394855+0000","last_clean":"2024-03-06T07:35:45.394855+0000","last_became_active":"2024-03-06T07:35:44.416892+0000","last_became_peered":"2024-03-06T07:35:44.416892+0000","last_unstale":"2024-03-06T07:35:45.394855+0000","last_undegraded":"2024-03-06T07:35:45.394855+0000","last_fullsized":"2024-03-06T07:35:45.394855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394798+0000","last_change":"2024-03-06T07:35:43.377777+0000","last_active":"2024-03-06T07:35:45.394798+0000","last_peered":"2024-03-06T07:35:45.394798+0000","last_clean":"2024-03-06T07:35:45.394798+0000","last_became_active":"2024-03-06T07:35:43.374868+0000","last_became_peered":"2024-03-06T07:35:43.374868+0000","last_unstale":"2024-03-06T07:35:45.394798+0000","last_undegraded":"2024-03-06T07:35:45.394798+0000","last_fullsized":"2024-03-06T07:35:45.394798+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:57.102370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354203+0000","last_change":"2024-03-06T07:35:43.354203+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354203+0000","last_undegraded":"2024-03-06T07:35:43.354203+0000","last_fullsized":"2024-03-06T07:35:43.354203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557609+0000","last_change":"2024-03-06T07:35:43.368644+0000","last_active":"2024-03-06T07:35:45.557609+0000","last_peered":"2024-03-06T07:35:45.557609+0000","last_clean":"2024-03-06T07:35:45.557609+0000","last_became_active":"2024-03-06T07:35:43.367936+0000","last_became_peered":"2024-03-06T07:35:43.367936+0000","last_unstale":"2024-03-06T07:35:45.557609+0000","last_undegraded":"2024-03-06T07:35:45.557609+0000","last_fullsized":"2024-03-06T07:35:45.557609+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:01.783718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720731+0000","last_change":"2024-03-06T07:35:44.417357+0000","last_active":"2024-03-06T07:35:45.720731+0000","last_peered":"2024-03-06T07:35:45.720731+0000","last_clean":"2024-03-06T07:35:45.720731+0000","last_became_active":"2024-03-06T07:35:44.417021+0000","last_became_peered":"2024-03-06T07:35:44.417021+0000","last_unstale":"2024-03-06T07:35:45.720731+0000","last_undegraded":"2024-03-06T07:35:45.720731+0000","last_fullsized":"2024-03-06T07:35:45.720731+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:11.909643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394493+0000","last_change":"2024-03-06T07:35:43.376770+0000","last_active":"2024-03-06T07:35:45.394493+0000","last_peered":"2024-03-06T07:35:45.394493+0000","last_clean":"2024-03-06T07:35:45.394493+0000","last_became_active":"2024-03-06T07:35:43.376483+0000","last_became_peered":"2024-03-06T07:35:43.376483+0000","last_unstale":"2024-03-06T07:35:45.394493+0000","last_undegraded":"2024-03-06T07:35:45.394493+0000","last_fullsized":"2024-03-06T07:35:45.394493+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:32.527625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399036+0000","last_change":"2024-03-06T07:35:44.408484+0000","last_active":"2024-03-06T07:35:45.399036+0000","last_peered":"2024-03-06T07:35:45.399036+0000","last_clean":"2024-03-06T07:35:45.399036+0000","last_became_active":"2024-03-06T07:35:44.407992+0000","last_became_peered":"2024-03-06T07:35:44.407992+0000","last_unstale":"2024-03-06T07:35:45.399036+0000","last_undegraded":"2024-03-06T07:35:45.399036+0000","last_fullsized":"2024-03-06T07:35:45.399036+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720795+0000","last_change":"2024-03-06T07:35:43.385517+0000","last_active":"2024-03-06T07:35:45.720795+0000","last_peered":"2024-03-06T07:35:45.720795+0000","last_clean":"2024-03-06T07:35:45.720795+0000","last_became_active":"2024-03-06T07:35:43.384210+0000","last_became_peered":"2024-03-06T07:35:43.384210+0000","last_unstale":"2024-03-06T07:35:45.720795+0000","last_undegraded":"2024-03-06T07:35:45.720795+0000","last_fullsized":"2024-03-06T07:35:45.720795+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:11.817793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719634+0000","last_change":"2024-03-06T07:35:44.412773+0000","last_active":"2024-03-06T07:35:45.719634+0000","last_peered":"2024-03-06T07:35:45.719634+0000","last_clean":"2024-03-06T07:35:45.719634+0000","last_became_active":"2024-03-06T07:35:44.411292+0000","last_became_peered":"2024-03-06T07:35:44.411292+0000","last_unstale":"2024-03-06T07:35:45.719634+0000","last_undegraded":"2024-03-06T07:35:45.719634+0000","last_fullsized":"2024-03-06T07:35:45.719634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719424+0000","last_change":"2024-03-06T07:35:43.379581+0000","last_active":"2024-03-06T07:35:45.719424+0000","last_peered":"2024-03-06T07:35:45.719424+0000","last_clean":"2024-03-06T07:35:45.719424+0000","last_became_active":"2024-03-06T07:35:43.379237+0000","last_became_peered":"2024-03-06T07:35:43.379237+0000","last_unstale":"2024-03-06T07:35:45.719424+0000","last_undegraded":"2024-03-06T07:35:45.719424+0000","last_fullsized":"2024-03-06T07:35:45.719424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:32.312256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394668+0000","last_change":"2024-03-06T07:35:44.417512+0000","last_active":"2024-03-06T07:35:45.394668+0000","last_peered":"2024-03-06T07:35:45.394668+0000","last_clean":"2024-03-06T07:35:45.394668+0000","last_became_active":"2024-03-06T07:35:44.416206+0000","last_became_peered":"2024-03-06T07:35:44.416206+0000","last_unstale":"2024-03-06T07:35:45.394668+0000","last_undegraded":"2024-03-06T07:35:45.394668+0000","last_fullsized":"2024-03-06T07:35:45.394668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557543+0000","last_change":"2024-03-06T07:35:43.375969+0000","last_active":"2024-03-06T07:35:45.557543+0000","last_peered":"2024-03-06T07:35:45.557543+0000","last_clean":"2024-03-06T07:35:45.557543+0000","last_became_active":"2024-03-06T07:35:43.375710+0000","last_became_peered":"2024-03-06T07:35:43.375710+0000","last_unstale":"2024-03-06T07:35:45.557543+0000","last_undegraded":"2024-03-06T07:35:45.557543+0000","last_fullsized":"2024-03-06T07:35:45.557543+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:38.782993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720514+0000","last_change":"2024-03-06T07:35:44.414475+0000","last_active":"2024-03-06T07:35:45.720514+0000","last_peered":"2024-03-06T07:35:45.720514+0000","last_clean":"2024-03-06T07:35:45.720514+0000","last_became_active":"2024-03-06T07:35:44.414166+0000","last_became_peered":"2024-03-06T07:35:44.414166+0000","last_unstale":"2024-03-06T07:35:45.720514+0000","last_undegraded":"2024-03-06T07:35:45.720514+0000","last_fullsized":"2024-03-06T07:35:45.720514+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:39.798967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721268+0000","last_change":"2024-03-06T07:35:43.385255+0000","last_active":"2024-03-06T07:35:45.721268+0000","last_peered":"2024-03-06T07:35:45.721268+0000","last_clean":"2024-03-06T07:35:45.721268+0000","last_became_active":"2024-03-06T07:35:43.384945+0000","last_became_peered":"2024-03-06T07:35:43.384945+0000","last_unstale":"2024-03-06T07:35:45.721268+0000","last_undegraded":"2024-03-06T07:35:45.721268+0000","last_fullsized":"2024-03-06T07:35:45.721268+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:41.317784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":369,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719242+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:35:45.719242+0000","last_peered":"2024-03-06T07:35:45.719242+0000","last_clean":"2024-03-06T07:35:45.719242+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:35:45.719242+0000","last_undegraded":"2024-03-06T07:35:45.719242+0000","last_fullsized":"2024-03-06T07:35:45.719242+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2236416,"data_stored":1296790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542246,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1630208,"data_stored":705784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":691097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542247,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1626112,"data_stored":704788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1617920,"data_stored":704172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41660,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330180,"statfs":{"total":96636764160,"available":96594104320,"internally_reserved":0,"allocated":2211840,"data_stored":1294540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3a is not active+clean 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.354250+0000', 'last_change': '2024-03-06T07:35:43.354250+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.354250+0000', 'last_undegraded': '2024-03-06T07:35:43.354250+0000', 'last_fullsized': '2024-03-06T07:35:43.354250+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.39 is not active+clean 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.353927+0000', 'last_change': '2024-03-06T07:35:43.353927+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.353927+0000', 'last_undegraded': '2024-03-06T07:35:43.353927+0000', 'last_fullsized': '2024-03-06T07:35:43.353927+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2f is not active+clean 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.356595+0000', 'last_change': '2024-03-06T07:35:43.356595+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.356595+0000', 'last_undegraded': '2024-03-06T07:35:43.356595+0000', 'last_fullsized': '2024-03-06T07:35:43.356595+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.20 is not active+clean 2024-03-06T07:35:49.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.353912+0000', 'last_change': '2024-03-06T07:35:43.353912+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.353912+0000', 'last_undegraded': '2024-03-06T07:35:43.353912+0000', 'last_fullsized': '2024-03-06T07:35:43.353912+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1b is not active+clean 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.356413+0000', 'last_change': '2024-03-06T07:35:43.356413+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.356413+0000', 'last_undegraded': '2024-03-06T07:35:43.356413+0000', 'last_fullsized': '2024-03-06T07:35:43.356413+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.b is not active+clean 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.354223+0000', 'last_change': '2024-03-06T07:35:43.354223+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.354223+0000', 'last_undegraded': '2024-03-06T07:35:43.354223+0000', 'last_fullsized': '2024-03-06T07:35:43.354223+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.6 is not active+clean 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.354748+0000', 'last_change': '2024-03-06T07:35:43.354748+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.354748+0000', 'last_undegraded': '2024-03-06T07:35:43.354748+0000', 'last_fullsized': '2024-03-06T07:35:43.354748+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.5 is not active+clean 2024-03-06T07:35:49.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 128, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:35:43.354203+0000', 'last_change': '2024-03-06T07:35:43.354203+0000', 'last_active': '2024-03-06T07:35:43.339559+0000', 'last_peered': '2024-03-06T07:35:43.339559+0000', 'last_clean': '2024-03-06T07:35:43.339559+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:35:43.354203+0000', 'last_undegraded': '2024-03-06T07:35:43.354203+0000', 'last_fullsized': '2024-03-06T07:35:43.354203+0000', 'mapping_epoch': 128, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 128, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'last_clean_scrub_stamp': '2024-03-06T07:35:43.339559+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:35:49.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:35:50.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.140+0000 7ff4d1200640 1 Processor -- start 2024-03-06T07:35:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.140+0000 7ff4d1200640 1 -- start start 2024-03-06T07:35:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.142+0000 7ff4d1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff4cc0af100 0x7ff4cc071e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.142+0000 7ff4d1200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0af530 con 0x7ff4cc06dc80 2024-03-06T07:35:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.142+0000 7ff4d1200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc072530 con 0x7ff4cc1625b0 2024-03-06T07:35:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.142+0000 7ff4d1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4cc162a40 con 0x7ff4cc0af100 2024-03-06T07:35:50.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.142+0000 7ff4c9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff4cc06dc80 0x7ff4cc06e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50196/0 (socket says 172.21.15.97:50196) 2024-03-06T07:35:50.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.142+0000 7ff4c9e00640 1 -- 172.21.15.97:0/2064474602 learned_addr learned my addr 172.21.15.97:0/2064474602 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2064474602 >> v2:172.21.15.135:3300/0 conn(0x7ff4cc0af100 0x7ff4cc071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42245272 0 0) 0x7ff4cc0af530 con 0x7ff4cc06dc80 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b0003520 con 0x7ff4cc06dc80 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359121660 0 0) 0x7ff4b0003520 con 0x7ff4cc06dc80 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 >> v2:172.21.15.135:3300/0 conn(0x7ff4cc0af100 msgr2=0x7ff4cc071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 --2- 172.21.15.97:0/2064474602 >> v2:172.21.15.135:3300/0 conn(0x7ff4cc0af100 0x7ff4cc071e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 >> v1:172.21.15.135:6789/0 conn(0x7ff4cc1625b0 legacy=0x7ff4cc15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc072810 con 0x7ff4cc06dc80 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff4bc001670 con 0x7ff4cc06dc80 2024-03-06T07:35:50.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 >> v1:172.21.15.97:6789/0 conn(0x7ff4cc06dc80 legacy=0x7ff4cc06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00039b0 0x7ff4b0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4b0004800 0x7ff4b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4b00051c0 0x7ff4b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b000c0c0 con 0x7ff4b00051c0 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b000c270 con 0x7ff4b00039b0 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.143+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b000c470 con 0x7ff4b0004800 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4b00051c0 0x7ff4b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4c9e00640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00039b0 0x7ff4b0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4ca800640 1 -- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4b0004800 msgr2=0x7ff4b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4b0004800 0x7ff4b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4ca800640 1 -- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00039b0 msgr2=0x7ff4b0003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00039b0 0x7ff4b0003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.144+0000 7ff4ca800640 1 -- 172.21.15.97:0/2064474602 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0003520 con 0x7ff4b00051c0 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.145+0000 7ff4ca800640 1 -- 172.21.15.97:0/2064474602 reap_dead start 2024-03-06T07:35:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.145+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4b00051c0 0x7ff4b000b9c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4007330 tx=0x7ff4b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cf2594241d486e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.145+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4009be0 con 0x7ff4b00051c0 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.145+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c402ccd0 con 0x7ff4b00051c0 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.145+0000 7ff4c9400640 1 -- 172.21.15.97:0/2064474602 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4003bd0 con 0x7ff4b00051c0 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 -- 172.21.15.97:0/2064474602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4b00051c0 msgr2=0x7ff4b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4b00051c0 0x7ff4b000b9c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4007330 tx=0x7ff4b0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 -- 172.21.15.97:0/2064474602 shutdown_connections 2024-03-06T07:35:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2064474602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4b00051c0 0x7ff4b000b9c0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 -- 172.21.15.97:0/2064474602 >> 172.21.15.97:0/2064474602 conn(0x7ff4cc0cfa30 msgr2=0x7ff4cc0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 -- 172.21.15.97:0/2064474602 shutdown_connections 2024-03-06T07:35:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.146+0000 7ff4d1200640 1 -- 172.21.15.97:0/2064474602 wait complete. 2024-03-06T07:35:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.149+0000 7ff4d1200640 1 Processor -- start 2024-03-06T07:35:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.149+0000 7ff4d1200640 1 -- start start 2024-03-06T07:35:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4d1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00051c0 0x7ff4cc145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4d1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4cc15c440 0x7ff4cc15c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4d1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 0x7ff4cc157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4d1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc162090 con 0x7ff4cc156ed0 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4d1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc161b60 con 0x7ff4b00051c0 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4d1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc071e30 con 0x7ff4cc15c440 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4ca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 0x7ff4cc157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4ca800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 0x7ff4cc157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32908/0 (socket says 172.21.15.97:32908) 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4ca800640 1 -- 172.21.15.97:0/2639683639 learned_addr learned my addr 172.21.15.97:0/2639683639 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4c9e00640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00051c0 0x7ff4cc145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.151+0000 7ff4c1400640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4cc15c440 0x7ff4cc15c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4ca800640 1 -- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4cc15c440 msgr2=0x7ff4cc15c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4cc15c440 0x7ff4cc15c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4ca800640 1 -- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00051c0 msgr2=0x7ff4cc145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00051c0 0x7ff4cc145640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4ca800640 1 -- 172.21.15.97:0/2639683639 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc145b80 con 0x7ff4cc156ed0 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4c1400640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4cc15c440 0x7ff4cc15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4ca800640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 0x7ff4cc157350 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40044a0 tx=0x7ff4c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4042070 con 0x7ff4cc156ed0 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c4003e10 con 0x7ff4cc156ed0 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4039070 con 0x7ff4cc156ed0 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc153440 con 0x7ff4cc156ed0 2024-03-06T07:35:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.152+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc15d1a0 con 0x7ff4cc156ed0 2024-03-06T07:35:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.153+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc04f330 con 0x7ff4cc156ed0 2024-03-06T07:35:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.154+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4c4045070 con 0x7ff4cc156ed0 2024-03-06T07:35:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.156+0000 7ff4c2a00640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.156+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4c40ab750 con 0x7ff4cc156ed0 2024-03-06T07:35:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.156+0000 7ff4c9e00640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.157+0000 7ff4c9e00640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc001050 tx=0x7ff4bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.160+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4cc145b80 con 0x7ff4cc156ed0 2024-03-06T07:35:50.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.429+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff4cc0aeb20 con 0x7ff4cc156ed0 2024-03-06T07:35:50.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.430+0000 7ff4c2a00640 1 -- 172.21.15.97:0/2639683639 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1101 (secure 0 0 0) 0x7ff4c407a530 con 0x7ff4cc156ed0 2024-03-06T07:35:50.433 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:50.433 INFO:teuthology.orchestra.run.smithi097.stdout:{"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":508,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":130,"num_osds":8,"num_up_osds":8,"osd_up_since":1709710053,"num_in_osds":8,"osd_in_since":1709710045,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":89},{"state_name":"creating+peering","count":8}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592708,"bytes_used":312205312,"bytes_avail":772781907968,"bytes_total":773094113280,"inactive_pgs_ratio":0.082474224269390106,"write_bytes_sec":2218,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":88,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"a","status":"up:active","gid":8382}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-03-06T07:35:27.045447+0000","services":{}},"progress_events":{}} 2024-03-06T07:35:50.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 msgr2=0x7ff4a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc001050 tx=0x7ff4bc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 msgr2=0x7ff4cc157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 0x7ff4cc157350 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40044a0 tx=0x7ff4c4004ab0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 shutdown_connections 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4cc156ed0 0x7ff4cc157350 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4cc15c440 0x7ff4cc15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 --2- 172.21.15.97:0/2639683639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4b00051c0 0x7ff4cc145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 >> 172.21.15.97:0/2639683639 conn(0x7ff4cc0cfa30 msgr2=0x7ff4cc0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 shutdown_connections 2024-03-06T07:35:50.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.433+0000 7ff4d1200640 1 -- 172.21.15.97:0/2639683639 wait complete. 2024-03-06T07:35:50.454 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:35:50.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.644+0000 7f8dae400640 1 Processor -- start 2024-03-06T07:35:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.645+0000 7f8dae400640 1 -- start start 2024-03-06T07:35:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.646+0000 7f8dae400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da8144c00 0x7f8da806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.646+0000 7f8dae400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8da815fe30 0x7f8da80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.646+0000 7f8dae400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da806ee70 con 0x7f8da815f9d0 2024-03-06T07:35:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.646+0000 7f8dae400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da806cef0 con 0x7f8da815fe30 2024-03-06T07:35:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.646+0000 7f8dae400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da806d1d0 con 0x7f8da8144c00 2024-03-06T07:35:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.647+0000 7f8da7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8da815fe30 0x7f8da80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.647+0000 7f8da7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da8144c00 0x7f8da806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.647+0000 7f8da7e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8da815fe30 0x7f8da80ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:40564/0 (socket says 172.21.15.97:40564) 2024-03-06T07:35:50.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.647+0000 7f8da7e00640 1 -- 172.21.15.97:0/2499541388 learned_addr learned my addr 172.21.15.97:0/2499541388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:50.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.648+0000 7f8da7e00640 1 -- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da8144c00 msgr2=0x7f8da806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.648+0000 7f8da7e00640 1 --2- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da8144c00 0x7f8da806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.648+0000 7f8da7e00640 1 -- 172.21.15.97:0/2499541388 >> v1:172.21.15.135:6789/0 conn(0x7f8da815f9d0 legacy=0x7f8da80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.648+0000 7f8da7e00640 1 -- 172.21.15.97:0/2499541388 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da806d4b0 con 0x7f8da815fe30 2024-03-06T07:35:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.649+0000 7f8da7e00640 1 --2- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da815fe30 0x7f8da80ce7b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009e50 tx=0x7f8da002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8595b5e0b95ac8f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8da6000640 1 -- 172.21.15.97:0/2499541388 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da002cb00 con 0x7f8da815fe30 2024-03-06T07:35:50.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8da6000640 1 -- 172.21.15.97:0/2499541388 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da002cca0 con 0x7f8da815fe30 2024-03-06T07:35:50.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8da6000640 1 -- 172.21.15.97:0/2499541388 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0034610 con 0x7f8da815fe30 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8dae400640 1 -- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da815fe30 msgr2=0x7f8da80ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8dae400640 1 --2- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da815fe30 0x7f8da80ce7b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009e50 tx=0x7f8da002bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8dae400640 1 -- 172.21.15.97:0/2499541388 shutdown_connections 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8dae400640 1 --2- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da815fe30 0x7f8da80ce7b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8dae400640 1 --2- 172.21.15.97:0/2499541388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da8144c00 0x7f8da806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.650+0000 7f8dae400640 1 -- 172.21.15.97:0/2499541388 >> 172.21.15.97:0/2499541388 conn(0x7f8da80b1730 msgr2=0x7f8da80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.651+0000 7f8dae400640 1 -- 172.21.15.97:0/2499541388 shutdown_connections 2024-03-06T07:35:50.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.651+0000 7f8dae400640 1 -- 172.21.15.97:0/2499541388 wait complete. 2024-03-06T07:35:50.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.652+0000 7f8dae400640 1 Processor -- start 2024-03-06T07:35:50.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.652+0000 7f8dae400640 1 -- start start 2024-03-06T07:35:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.653+0000 7f8dae400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da8144c00 0x7f8da815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.653+0000 7f8dae400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da815f9d0 0x7f8da80ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.653+0000 7f8dae400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 0x7f8da80b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.653+0000 7f8dae400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8160730 con 0x7f8da815fe30 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.653+0000 7f8dae400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8160280 con 0x7f8da8144c00 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.653+0000 7f8dae400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da806cb50 con 0x7f8da815f9d0 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 0x7f8da80b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 0x7f8da80b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32910/0 (socket says 172.21.15.97:32910) 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 -- 172.21.15.97:0/2665211997 learned_addr learned my addr 172.21.15.97:0/2665211997 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da8144c00 0x7f8da815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da6a00640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da815f9d0 0x7f8da80ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 -- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da815f9d0 msgr2=0x7f8da80ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da815f9d0 0x7f8da80ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 -- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da8144c00 msgr2=0x7f8da815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da8144c00 0x7f8da815ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 -- 172.21.15.97:0/2665211997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da81403c0 con 0x7f8da815fe30 2024-03-06T07:35:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da6a00640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da815f9d0 0x7f8da80ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.654+0000 7f8da7e00640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 0x7f8da80b2b30 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009e50 tx=0x7f8da0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.655+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0043020 con 0x7f8da815fe30 2024-03-06T07:35:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.655+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da002ce50 con 0x7f8da815fe30 2024-03-06T07:35:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.655+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da813f320 con 0x7f8da815fe30 2024-03-06T07:35:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.655+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0009510 con 0x7f8da815fe30 2024-03-06T07:35:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.655+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da813f8e0 con 0x7f8da815fe30 2024-03-06T07:35:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.656+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da804f330 con 0x7f8da815fe30 2024-03-06T07:35:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.657+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8da0046020 con 0x7f8da815fe30 2024-03-06T07:35:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.658+0000 7f8d9f400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d8406b4d0 0x7f8d8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.658+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8da00ac870 con 0x7f8da815fe30 2024-03-06T07:35:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.658+0000 7f8da7400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d8406b4d0 0x7f8d8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.659+0000 7f8da7400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d8406b4d0 0x7f8d8406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009a10 tx=0x7f8d98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.663+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8da004d030 con 0x7f8da815fe30 2024-03-06T07:35:50.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.856+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8da8072890 con 0x7f8d8406b4d0 2024-03-06T07:35:50.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.873+0000 7f8d9f400640 1 -- 172.21.15.97:0/2665211997 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251818 (secure 0 0 0) 0x7f8da8072890 con 0x7f8d8406b4d0 2024-03-06T07:35:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:50.879 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.878+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d8406b4d0 msgr2=0x7f8d8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.878+0000 7f8dae400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d8406b4d0 0x7f8d8406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009a10 tx=0x7f8d98008040 comp rx=0 tx=0).stop 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.878+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 msgr2=0x7f8da80b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.878+0000 7f8dae400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 0x7f8da80b2b30 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009e50 tx=0x7f8da0004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 shutdown_connections 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d8406b4d0 0x7f8d8406d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8da815fe30 0x7f8da80b2b30 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8da815f9d0 0x7f8da80ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 --2- 172.21.15.97:0/2665211997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da8144c00 0x7f8da815ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 >> 172.21.15.97:0/2665211997 conn(0x7f8da80b1730 msgr2=0x7f8da8145190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 shutdown_connections 2024-03-06T07:35:50.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:50.879+0000 7f8dae400640 1 -- 172.21.15.97:0/2665211997 wait complete. 2024-03-06T07:35:50.900 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-06T07:35:49.051857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304888,"kb_used_data":13836,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754669832,"statfs":{"total":773094113280,"available":772781907968,"internally_reserved":0,"allocated":14168064,"data_stored":6805787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001479"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720739+0000","last_change":"2024-03-06T07:35:44.394272+0000","last_active":"2024-03-06T07:35:45.720739+0000","last_peered":"2024-03-06T07:35:45.720739+0000","last_clean":"2024-03-06T07:35:45.720739+0000","last_became_active":"2024-03-06T07:35:44.393967+0000","last_became_peered":"2024-03-06T07:35:44.393967+0000","last_unstale":"2024-03-06T07:35:45.720739+0000","last_undegraded":"2024-03-06T07:35:45.720739+0000","last_fullsized":"2024-03-06T07:35:45.720739+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:28.599416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394850+0000","last_change":"2024-03-06T07:35:44.392186+0000","last_active":"2024-03-06T07:35:45.394850+0000","last_peered":"2024-03-06T07:35:45.394850+0000","last_clean":"2024-03-06T07:35:45.394850+0000","last_became_active":"2024-03-06T07:35:44.391446+0000","last_became_peered":"2024-03-06T07:35:44.391446+0000","last_unstale":"2024-03-06T07:35:45.394850+0000","last_undegraded":"2024-03-06T07:35:45.394850+0000","last_fullsized":"2024-03-06T07:35:45.394850+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:25.601824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720157+0000","last_change":"2024-03-06T07:35:44.410356+0000","last_active":"2024-03-06T07:35:45.720157+0000","last_peered":"2024-03-06T07:35:45.720157+0000","last_clean":"2024-03-06T07:35:45.720157+0000","last_became_active":"2024-03-06T07:35:44.409633+0000","last_became_peered":"2024-03-06T07:35:44.409633+0000","last_unstale":"2024-03-06T07:35:45.720157+0000","last_undegraded":"2024-03-06T07:35:45.720157+0000","last_fullsized":"2024-03-06T07:35:45.720157+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:12.833768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718970+0000","last_change":"2024-03-06T07:35:44.417975+0000","last_active":"2024-03-06T07:35:45.718970+0000","last_peered":"2024-03-06T07:35:45.718970+0000","last_clean":"2024-03-06T07:35:45.718970+0000","last_became_active":"2024-03-06T07:35:44.417719+0000","last_became_peered":"2024-03-06T07:35:44.417719+0000","last_unstale":"2024-03-06T07:35:45.718970+0000","last_undegraded":"2024-03-06T07:35:45.718970+0000","last_fullsized":"2024-03-06T07:35:45.718970+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:26.008403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721342+0000","last_change":"2024-03-06T07:35:44.415585+0000","last_active":"2024-03-06T07:35:45.721342+0000","last_peered":"2024-03-06T07:35:45.721342+0000","last_clean":"2024-03-06T07:35:45.721342+0000","last_became_active":"2024-03-06T07:35:44.415011+0000","last_became_peered":"2024-03-06T07:35:44.415011+0000","last_unstale":"2024-03-06T07:35:45.721342+0000","last_undegraded":"2024-03-06T07:35:45.721342+0000","last_fullsized":"2024-03-06T07:35:45.721342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354250+0000","last_change":"2024-03-06T07:35:43.354250+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354250+0000","last_undegraded":"2024-03-06T07:35:43.354250+0000","last_fullsized":"2024-03-06T07:35:43.354250+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353927+0000","last_change":"2024-03-06T07:35:43.353927+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353927+0000","last_undegraded":"2024-03-06T07:35:43.353927+0000","last_fullsized":"2024-03-06T07:35:43.353927+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719898+0000","last_change":"2024-03-06T07:35:44.405572+0000","last_active":"2024-03-06T07:35:45.719898+0000","last_peered":"2024-03-06T07:35:45.719898+0000","last_clean":"2024-03-06T07:35:45.719898+0000","last_became_active":"2024-03-06T07:35:44.405225+0000","last_became_peered":"2024-03-06T07:35:44.405225+0000","last_unstale":"2024-03-06T07:35:45.719898+0000","last_undegraded":"2024-03-06T07:35:45.719898+0000","last_fullsized":"2024-03-06T07:35:45.719898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:21.153780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721703+0000","last_change":"2024-03-06T07:35:44.415674+0000","last_active":"2024-03-06T07:35:45.721703+0000","last_peered":"2024-03-06T07:35:45.721703+0000","last_clean":"2024-03-06T07:35:45.721703+0000","last_became_active":"2024-03-06T07:35:44.415198+0000","last_became_peered":"2024-03-06T07:35:44.415198+0000","last_unstale":"2024-03-06T07:35:45.721703+0000","last_undegraded":"2024-03-06T07:35:45.721703+0000","last_fullsized":"2024-03-06T07:35:45.721703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:27.898827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395062+0000","last_change":"2024-03-06T07:35:44.398339+0000","last_active":"2024-03-06T07:35:45.395062+0000","last_peered":"2024-03-06T07:35:45.395062+0000","last_clean":"2024-03-06T07:35:45.395062+0000","last_became_active":"2024-03-06T07:35:44.396519+0000","last_became_peered":"2024-03-06T07:35:44.396519+0000","last_unstale":"2024-03-06T07:35:45.395062+0000","last_undegraded":"2024-03-06T07:35:45.395062+0000","last_fullsized":"2024-03-06T07:35:45.395062+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:48.438390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557893+0000","last_change":"2024-03-06T07:35:44.397455+0000","last_active":"2024-03-06T07:35:45.557893+0000","last_peered":"2024-03-06T07:35:45.557893+0000","last_clean":"2024-03-06T07:35:45.557893+0000","last_became_active":"2024-03-06T07:35:44.397217+0000","last_became_peered":"2024-03-06T07:35:44.397217+0000","last_unstale":"2024-03-06T07:35:45.557893+0000","last_undegraded":"2024-03-06T07:35:45.557893+0000","last_fullsized":"2024-03-06T07:35:45.557893+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:01.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720004+0000","last_change":"2024-03-06T07:35:44.416953+0000","last_active":"2024-03-06T07:35:45.720004+0000","last_peered":"2024-03-06T07:35:45.720004+0000","last_clean":"2024-03-06T07:35:45.720004+0000","last_became_active":"2024-03-06T07:35:44.416659+0000","last_became_peered":"2024-03-06T07:35:44.416659+0000","last_unstale":"2024-03-06T07:35:45.720004+0000","last_undegraded":"2024-03-06T07:35:45.720004+0000","last_fullsized":"2024-03-06T07:35:45.720004+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:31.633974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719022+0000","last_change":"2024-03-06T07:35:44.395498+0000","last_active":"2024-03-06T07:35:45.719022+0000","last_peered":"2024-03-06T07:35:45.719022+0000","last_clean":"2024-03-06T07:35:45.719022+0000","last_became_active":"2024-03-06T07:35:44.395203+0000","last_became_peered":"2024-03-06T07:35:44.395203+0000","last_unstale":"2024-03-06T07:35:45.719022+0000","last_undegraded":"2024-03-06T07:35:45.719022+0000","last_fullsized":"2024-03-06T07:35:45.719022+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:58.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394607+0000","last_change":"2024-03-06T07:35:44.400161+0000","last_active":"2024-03-06T07:35:45.394607+0000","last_peered":"2024-03-06T07:35:45.394607+0000","last_clean":"2024-03-06T07:35:45.394607+0000","last_became_active":"2024-03-06T07:35:44.399845+0000","last_became_peered":"2024-03-06T07:35:44.399845+0000","last_unstale":"2024-03-06T07:35:45.394607+0000","last_undegraded":"2024-03-06T07:35:45.394607+0000","last_fullsized":"2024-03-06T07:35:45.394607+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:33.642819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718604+0000","last_change":"2024-03-06T07:35:44.406110+0000","last_active":"2024-03-06T07:35:45.718604+0000","last_peered":"2024-03-06T07:35:45.718604+0000","last_clean":"2024-03-06T07:35:45.718604+0000","last_became_active":"2024-03-06T07:35:44.405807+0000","last_became_peered":"2024-03-06T07:35:44.405807+0000","last_unstale":"2024-03-06T07:35:45.718604+0000","last_undegraded":"2024-03-06T07:35:45.718604+0000","last_fullsized":"2024-03-06T07:35:45.718604+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:04.519517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719509+0000","last_change":"2024-03-06T07:35:44.395764+0000","last_active":"2024-03-06T07:35:45.719509+0000","last_peered":"2024-03-06T07:35:45.719509+0000","last_clean":"2024-03-06T07:35:45.719509+0000","last_became_active":"2024-03-06T07:35:44.393280+0000","last_became_peered":"2024-03-06T07:35:44.393280+0000","last_unstale":"2024-03-06T07:35:45.719509+0000","last_undegraded":"2024-03-06T07:35:45.719509+0000","last_fullsized":"2024-03-06T07:35:45.719509+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356595+0000","last_change":"2024-03-06T07:35:43.356595+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356595+0000","last_undegraded":"2024-03-06T07:35:43.356595+0000","last_fullsized":"2024-03-06T07:35:43.356595+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720362+0000","last_change":"2024-03-06T07:35:44.395358+0000","last_active":"2024-03-06T07:35:45.720362+0000","last_peered":"2024-03-06T07:35:45.720362+0000","last_clean":"2024-03-06T07:35:45.720362+0000","last_became_active":"2024-03-06T07:35:44.395025+0000","last_became_peered":"2024-03-06T07:35:44.395025+0000","last_unstale":"2024-03-06T07:35:45.720362+0000","last_undegraded":"2024-03-06T07:35:45.720362+0000","last_fullsized":"2024-03-06T07:35:45.720362+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558069+0000","last_change":"2024-03-06T07:35:44.408168+0000","last_active":"2024-03-06T07:35:45.558069+0000","last_peered":"2024-03-06T07:35:45.558069+0000","last_clean":"2024-03-06T07:35:45.558069+0000","last_became_active":"2024-03-06T07:35:44.407536+0000","last_became_peered":"2024-03-06T07:35:44.407536+0000","last_unstale":"2024-03-06T07:35:45.558069+0000","last_undegraded":"2024-03-06T07:35:45.558069+0000","last_fullsized":"2024-03-06T07:35:45.558069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:45.475165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720163+0000","last_change":"2024-03-06T07:35:44.390845+0000","last_active":"2024-03-06T07:35:45.720163+0000","last_peered":"2024-03-06T07:35:45.720163+0000","last_clean":"2024-03-06T07:35:45.720163+0000","last_became_active":"2024-03-06T07:35:44.388466+0000","last_became_peered":"2024-03-06T07:35:44.388466+0000","last_unstale":"2024-03-06T07:35:45.720163+0000","last_undegraded":"2024-03-06T07:35:45.720163+0000","last_fullsized":"2024-03-06T07:35:45.720163+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721001+0000","last_change":"2024-03-06T07:35:44.415326+0000","last_active":"2024-03-06T07:35:45.721001+0000","last_peered":"2024-03-06T07:35:45.721001+0000","last_clean":"2024-03-06T07:35:45.721001+0000","last_became_active":"2024-03-06T07:35:44.414356+0000","last_became_peered":"2024-03-06T07:35:44.414356+0000","last_unstale":"2024-03-06T07:35:45.721001+0000","last_undegraded":"2024-03-06T07:35:45.721001+0000","last_fullsized":"2024-03-06T07:35:45.721001+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557466+0000","last_change":"2024-03-06T07:35:44.394518+0000","last_active":"2024-03-06T07:35:45.557466+0000","last_peered":"2024-03-06T07:35:45.557466+0000","last_clean":"2024-03-06T07:35:45.557466+0000","last_became_active":"2024-03-06T07:35:44.392743+0000","last_became_peered":"2024-03-06T07:35:44.392743+0000","last_unstale":"2024-03-06T07:35:45.557466+0000","last_undegraded":"2024-03-06T07:35:45.557466+0000","last_fullsized":"2024-03-06T07:35:45.557466+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:34.320907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720515+0000","last_change":"2024-03-06T07:35:44.417926+0000","last_active":"2024-03-06T07:35:45.720515+0000","last_peered":"2024-03-06T07:35:45.720515+0000","last_clean":"2024-03-06T07:35:45.720515+0000","last_became_active":"2024-03-06T07:35:44.415414+0000","last_became_peered":"2024-03-06T07:35:44.415414+0000","last_unstale":"2024-03-06T07:35:45.720515+0000","last_undegraded":"2024-03-06T07:35:45.720515+0000","last_fullsized":"2024-03-06T07:35:45.720515+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:14.302817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557430+0000","last_change":"2024-03-06T07:35:44.408475+0000","last_active":"2024-03-06T07:35:45.557430+0000","last_peered":"2024-03-06T07:35:45.557430+0000","last_clean":"2024-03-06T07:35:45.557430+0000","last_became_active":"2024-03-06T07:35:44.408227+0000","last_became_peered":"2024-03-06T07:35:44.408227+0000","last_unstale":"2024-03-06T07:35:45.557430+0000","last_undegraded":"2024-03-06T07:35:45.557430+0000","last_fullsized":"2024-03-06T07:35:45.557430+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395196+0000","last_change":"2024-03-06T07:35:44.416789+0000","last_active":"2024-03-06T07:35:45.395196+0000","last_peered":"2024-03-06T07:35:45.395196+0000","last_clean":"2024-03-06T07:35:45.395196+0000","last_became_active":"2024-03-06T07:35:44.416274+0000","last_became_peered":"2024-03-06T07:35:44.416274+0000","last_unstale":"2024-03-06T07:35:45.395196+0000","last_undegraded":"2024-03-06T07:35:45.395196+0000","last_fullsized":"2024-03-06T07:35:45.395196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:47.395736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720627+0000","last_change":"2024-03-06T07:35:44.399980+0000","last_active":"2024-03-06T07:35:45.720627+0000","last_peered":"2024-03-06T07:35:45.720627+0000","last_clean":"2024-03-06T07:35:45.720627+0000","last_became_active":"2024-03-06T07:35:44.399328+0000","last_became_peered":"2024-03-06T07:35:44.399328+0000","last_unstale":"2024-03-06T07:35:45.720627+0000","last_undegraded":"2024-03-06T07:35:45.720627+0000","last_fullsized":"2024-03-06T07:35:45.720627+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:43.875737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719802+0000","last_change":"2024-03-06T07:35:44.394219+0000","last_active":"2024-03-06T07:35:45.719802+0000","last_peered":"2024-03-06T07:35:45.719802+0000","last_clean":"2024-03-06T07:35:45.719802+0000","last_became_active":"2024-03-06T07:35:44.393893+0000","last_became_peered":"2024-03-06T07:35:44.393893+0000","last_unstale":"2024-03-06T07:35:45.719802+0000","last_undegraded":"2024-03-06T07:35:45.719802+0000","last_fullsized":"2024-03-06T07:35:45.719802+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:18.620474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398830+0000","last_change":"2024-03-06T07:35:44.403178+0000","last_active":"2024-03-06T07:35:45.398830+0000","last_peered":"2024-03-06T07:35:45.398830+0000","last_clean":"2024-03-06T07:35:45.398830+0000","last_became_active":"2024-03-06T07:35:44.402903+0000","last_became_peered":"2024-03-06T07:35:44.402903+0000","last_unstale":"2024-03-06T07:35:45.398830+0000","last_undegraded":"2024-03-06T07:35:45.398830+0000","last_fullsized":"2024-03-06T07:35:45.398830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719226+0000","last_change":"2024-03-06T07:35:44.415700+0000","last_active":"2024-03-06T07:35:45.719226+0000","last_peered":"2024-03-06T07:35:45.719226+0000","last_clean":"2024-03-06T07:35:45.719226+0000","last_became_active":"2024-03-06T07:35:44.415400+0000","last_became_peered":"2024-03-06T07:35:44.415400+0000","last_unstale":"2024-03-06T07:35:45.719226+0000","last_undegraded":"2024-03-06T07:35:45.719226+0000","last_fullsized":"2024-03-06T07:35:45.719226+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:56.935549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719196+0000","last_change":"2024-03-06T07:35:44.406337+0000","last_active":"2024-03-06T07:35:45.719196+0000","last_peered":"2024-03-06T07:35:45.719196+0000","last_clean":"2024-03-06T07:35:45.719196+0000","last_became_active":"2024-03-06T07:35:44.405995+0000","last_became_peered":"2024-03-06T07:35:44.405995+0000","last_unstale":"2024-03-06T07:35:45.719196+0000","last_undegraded":"2024-03-06T07:35:45.719196+0000","last_fullsized":"2024-03-06T07:35:45.719196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:51.011567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719669+0000","last_change":"2024-03-06T07:35:44.415228+0000","last_active":"2024-03-06T07:35:45.719669+0000","last_peered":"2024-03-06T07:35:45.719669+0000","last_clean":"2024-03-06T07:35:45.719669+0000","last_became_active":"2024-03-06T07:35:44.414418+0000","last_became_peered":"2024-03-06T07:35:44.414418+0000","last_unstale":"2024-03-06T07:35:45.719669+0000","last_undegraded":"2024-03-06T07:35:45.719669+0000","last_fullsized":"2024-03-06T07:35:45.719669+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:25.327417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.353912+0000","last_change":"2024-03-06T07:35:43.353912+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.353912+0000","last_undegraded":"2024-03-06T07:35:43.353912+0000","last_fullsized":"2024-03-06T07:35:43.353912+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720567+0000","last_change":"2024-03-06T07:35:44.408316+0000","last_active":"2024-03-06T07:35:45.720567+0000","last_peered":"2024-03-06T07:35:45.720567+0000","last_clean":"2024-03-06T07:35:45.720567+0000","last_became_active":"2024-03-06T07:35:44.407617+0000","last_became_peered":"2024-03-06T07:35:44.407617+0000","last_unstale":"2024-03-06T07:35:45.720567+0000","last_undegraded":"2024-03-06T07:35:45.720567+0000","last_fullsized":"2024-03-06T07:35:45.720567+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:09.791568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722044+0000","last_change":"2024-03-06T07:35:43.384588+0000","last_active":"2024-03-06T07:35:45.722044+0000","last_peered":"2024-03-06T07:35:45.722044+0000","last_clean":"2024-03-06T07:35:45.722044+0000","last_became_active":"2024-03-06T07:35:43.384278+0000","last_became_peered":"2024-03-06T07:35:43.384278+0000","last_unstale":"2024-03-06T07:35:45.722044+0000","last_undegraded":"2024-03-06T07:35:45.722044+0000","last_fullsized":"2024-03-06T07:35:45.722044+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:30.617999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722113+0000","last_change":"2024-03-06T07:35:44.408075+0000","last_active":"2024-03-06T07:35:45.722113+0000","last_peered":"2024-03-06T07:35:45.722113+0000","last_clean":"2024-03-06T07:35:45.722113+0000","last_became_active":"2024-03-06T07:35:44.407778+0000","last_became_peered":"2024-03-06T07:35:44.407778+0000","last_unstale":"2024-03-06T07:35:45.722113+0000","last_undegraded":"2024-03-06T07:35:45.722113+0000","last_fullsized":"2024-03-06T07:35:45.722113+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:33.972464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399395+0000","last_change":"2024-03-06T07:35:43.378158+0000","last_active":"2024-03-06T07:35:45.399395+0000","last_peered":"2024-03-06T07:35:45.399395+0000","last_clean":"2024-03-06T07:35:45.399395+0000","last_became_active":"2024-03-06T07:35:43.377863+0000","last_became_peered":"2024-03-06T07:35:43.377863+0000","last_unstale":"2024-03-06T07:35:45.399395+0000","last_undegraded":"2024-03-06T07:35:45.399395+0000","last_fullsized":"2024-03-06T07:35:45.399395+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:27.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720318+0000","last_change":"2024-03-06T07:35:44.419353+0000","last_active":"2024-03-06T07:35:45.720318+0000","last_peered":"2024-03-06T07:35:45.720318+0000","last_clean":"2024-03-06T07:35:45.720318+0000","last_became_active":"2024-03-06T07:35:44.418396+0000","last_became_peered":"2024-03-06T07:35:44.418396+0000","last_unstale":"2024-03-06T07:35:45.720318+0000","last_undegraded":"2024-03-06T07:35:45.720318+0000","last_fullsized":"2024-03-06T07:35:45.720318+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:12.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558056+0000","last_change":"2024-03-06T07:35:43.375835+0000","last_active":"2024-03-06T07:35:45.558056+0000","last_peered":"2024-03-06T07:35:45.558056+0000","last_clean":"2024-03-06T07:35:45.558056+0000","last_became_active":"2024-03-06T07:35:43.375546+0000","last_became_peered":"2024-03-06T07:35:43.375546+0000","last_unstale":"2024-03-06T07:35:45.558056+0000","last_undegraded":"2024-03-06T07:35:45.558056+0000","last_fullsized":"2024-03-06T07:35:45.558056+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:44.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721668+0000","last_change":"2024-03-06T07:35:44.411183+0000","last_active":"2024-03-06T07:35:45.721668+0000","last_peered":"2024-03-06T07:35:45.721668+0000","last_clean":"2024-03-06T07:35:45.721668+0000","last_became_active":"2024-03-06T07:35:44.410909+0000","last_became_peered":"2024-03-06T07:35:44.410909+0000","last_unstale":"2024-03-06T07:35:45.721668+0000","last_undegraded":"2024-03-06T07:35:45.721668+0000","last_fullsized":"2024-03-06T07:35:45.721668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394928+0000","last_change":"2024-03-06T07:35:43.368831+0000","last_active":"2024-03-06T07:35:45.394928+0000","last_peered":"2024-03-06T07:35:45.394928+0000","last_clean":"2024-03-06T07:35:45.394928+0000","last_became_active":"2024-03-06T07:35:43.368516+0000","last_became_peered":"2024-03-06T07:35:43.368516+0000","last_unstale":"2024-03-06T07:35:45.394928+0000","last_undegraded":"2024-03-06T07:35:45.394928+0000","last_fullsized":"2024-03-06T07:35:45.394928+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:24.992428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719450+0000","last_change":"2024-03-06T07:35:44.384339+0000","last_active":"2024-03-06T07:35:45.719450+0000","last_peered":"2024-03-06T07:35:45.719450+0000","last_clean":"2024-03-06T07:35:45.719450+0000","last_became_active":"2024-03-06T07:35:44.384031+0000","last_became_peered":"2024-03-06T07:35:44.384031+0000","last_unstale":"2024-03-06T07:35:45.719450+0000","last_undegraded":"2024-03-06T07:35:45.719450+0000","last_fullsized":"2024-03-06T07:35:45.719450+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394361+0000","last_change":"2024-03-06T07:35:43.373948+0000","last_active":"2024-03-06T07:35:45.394361+0000","last_peered":"2024-03-06T07:35:45.394361+0000","last_clean":"2024-03-06T07:35:45.394361+0000","last_became_active":"2024-03-06T07:35:43.373631+0000","last_became_peered":"2024-03-06T07:35:43.373631+0000","last_unstale":"2024-03-06T07:35:45.394361+0000","last_undegraded":"2024-03-06T07:35:45.394361+0000","last_fullsized":"2024-03-06T07:35:45.394361+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:51.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.356413+0000","last_change":"2024-03-06T07:35:43.356413+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.356413+0000","last_undegraded":"2024-03-06T07:35:43.356413+0000","last_fullsized":"2024-03-06T07:35:43.356413+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719623+0000","last_change":"2024-03-06T07:35:43.370282+0000","last_active":"2024-03-06T07:35:45.719623+0000","last_peered":"2024-03-06T07:35:45.719623+0000","last_clean":"2024-03-06T07:35:45.719623+0000","last_became_active":"2024-03-06T07:35:43.369383+0000","last_became_peered":"2024-03-06T07:35:43.369383+0000","last_unstale":"2024-03-06T07:35:45.719623+0000","last_undegraded":"2024-03-06T07:35:45.719623+0000","last_fullsized":"2024-03-06T07:35:45.719623+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:02.163748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720319+0000","last_change":"2024-03-06T07:35:44.405369+0000","last_active":"2024-03-06T07:35:45.720319+0000","last_peered":"2024-03-06T07:35:45.720319+0000","last_clean":"2024-03-06T07:35:45.720319+0000","last_became_active":"2024-03-06T07:35:44.405046+0000","last_became_peered":"2024-03-06T07:35:44.405046+0000","last_unstale":"2024-03-06T07:35:45.720319+0000","last_undegraded":"2024-03-06T07:35:45.720319+0000","last_fullsized":"2024-03-06T07:35:45.720319+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:27.612553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.381678+0000","last_change":"2024-03-06T07:35:43.380112+0000","last_active":"2024-03-06T07:35:43.381678+0000","last_peered":"2024-03-06T07:35:43.381678+0000","last_clean":"2024-03-06T07:35:43.381678+0000","last_became_active":"2024-03-06T07:35:43.378931+0000","last_became_peered":"2024-03-06T07:35:43.378931+0000","last_unstale":"2024-03-06T07:35:43.381678+0000","last_undegraded":"2024-03-06T07:35:43.381678+0000","last_fullsized":"2024-03-06T07:35:43.381678+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:12.580678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394619+0000","last_change":"2024-03-06T07:35:44.382654+0000","last_active":"2024-03-06T07:35:45.394619+0000","last_peered":"2024-03-06T07:35:45.394619+0000","last_clean":"2024-03-06T07:35:45.394619+0000","last_became_active":"2024-03-06T07:35:44.382389+0000","last_became_peered":"2024-03-06T07:35:44.382389+0000","last_unstale":"2024-03-06T07:35:45.394619+0000","last_undegraded":"2024-03-06T07:35:45.394619+0000","last_fullsized":"2024-03-06T07:35:45.394619+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394363+0000","last_change":"2024-03-06T07:35:43.376342+0000","last_active":"2024-03-06T07:35:45.394363+0000","last_peered":"2024-03-06T07:35:45.394363+0000","last_clean":"2024-03-06T07:35:45.394363+0000","last_became_active":"2024-03-06T07:35:43.374474+0000","last_became_peered":"2024-03-06T07:35:43.374474+0000","last_unstale":"2024-03-06T07:35:45.394363+0000","last_undegraded":"2024-03-06T07:35:45.394363+0000","last_fullsized":"2024-03-06T07:35:45.394363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:03.503542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720355+0000","last_change":"2024-03-06T07:35:44.402234+0000","last_active":"2024-03-06T07:35:45.720355+0000","last_peered":"2024-03-06T07:35:45.720355+0000","last_clean":"2024-03-06T07:35:45.720355+0000","last_became_active":"2024-03-06T07:35:44.401505+0000","last_became_peered":"2024-03-06T07:35:44.401505+0000","last_unstale":"2024-03-06T07:35:45.720355+0000","last_undegraded":"2024-03-06T07:35:45.720355+0000","last_fullsized":"2024-03-06T07:35:45.720355+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:22.846996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401654+0000","last_change":"2024-03-06T07:35:43.384674+0000","last_active":"2024-03-06T07:35:45.401654+0000","last_peered":"2024-03-06T07:35:45.401654+0000","last_clean":"2024-03-06T07:35:45.401654+0000","last_became_active":"2024-03-06T07:35:43.384373+0000","last_became_peered":"2024-03-06T07:35:43.384373+0000","last_unstale":"2024-03-06T07:35:45.401654+0000","last_undegraded":"2024-03-06T07:35:45.401654+0000","last_fullsized":"2024-03-06T07:35:45.401654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:47.747507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400796+0000","last_change":"2024-03-06T07:35:44.400940+0000","last_active":"2024-03-06T07:35:45.400796+0000","last_peered":"2024-03-06T07:35:45.400796+0000","last_clean":"2024-03-06T07:35:45.400796+0000","last_became_active":"2024-03-06T07:35:44.400650+0000","last_became_peered":"2024-03-06T07:35:44.400650+0000","last_unstale":"2024-03-06T07:35:45.400796+0000","last_undegraded":"2024-03-06T07:35:45.400796+0000","last_fullsized":"2024-03-06T07:35:45.400796+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720656+0000","last_change":"2024-03-06T07:35:43.381659+0000","last_active":"2024-03-06T07:35:45.720656+0000","last_peered":"2024-03-06T07:35:45.720656+0000","last_clean":"2024-03-06T07:35:45.720656+0000","last_became_active":"2024-03-06T07:35:43.381344+0000","last_became_peered":"2024-03-06T07:35:43.381344+0000","last_unstale":"2024-03-06T07:35:45.720656+0000","last_undegraded":"2024-03-06T07:35:45.720656+0000","last_fullsized":"2024-03-06T07:35:45.720656+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:48.989771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720762+0000","last_change":"2024-03-06T07:35:44.420113+0000","last_active":"2024-03-06T07:35:45.720762+0000","last_peered":"2024-03-06T07:35:45.720762+0000","last_clean":"2024-03-06T07:35:45.720762+0000","last_became_active":"2024-03-06T07:35:44.418765+0000","last_became_peered":"2024-03-06T07:35:44.418765+0000","last_unstale":"2024-03-06T07:35:45.720762+0000","last_undegraded":"2024-03-06T07:35:45.720762+0000","last_fullsized":"2024-03-06T07:35:45.720762+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"129'3","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398749+0000","last_change":"2024-03-06T07:35:43.378206+0000","last_active":"2024-03-06T07:35:45.398749+0000","last_peered":"2024-03-06T07:35:45.398749+0000","last_clean":"2024-03-06T07:35:45.398749+0000","last_became_active":"2024-03-06T07:35:43.377916+0000","last_became_peered":"2024-03-06T07:35:43.377916+0000","last_unstale":"2024-03-06T07:35:45.398749+0000","last_undegraded":"2024-03-06T07:35:45.398749+0000","last_fullsized":"2024-03-06T07:35:45.398749+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:21.831021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722052+0000","last_change":"2024-03-06T07:35:44.416806+0000","last_active":"2024-03-06T07:35:45.722052+0000","last_peered":"2024-03-06T07:35:45.722052+0000","last_clean":"2024-03-06T07:35:45.722052+0000","last_became_active":"2024-03-06T07:35:44.416506+0000","last_became_peered":"2024-03-06T07:35:44.416506+0000","last_unstale":"2024-03-06T07:35:45.722052+0000","last_undegraded":"2024-03-06T07:35:45.722052+0000","last_fullsized":"2024-03-06T07:35:45.722052+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"129'6","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721134+0000","last_change":"2024-03-06T07:35:43.379952+0000","last_active":"2024-03-06T07:35:45.721134+0000","last_peered":"2024-03-06T07:35:45.721134+0000","last_clean":"2024-03-06T07:35:45.721134+0000","last_became_active":"2024-03-06T07:35:43.379541+0000","last_became_peered":"2024-03-06T07:35:43.379541+0000","last_unstale":"2024-03-06T07:35:45.721134+0000","last_undegraded":"2024-03-06T07:35:45.721134+0000","last_fullsized":"2024-03-06T07:35:45.721134+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.626844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719234+0000","last_change":"2024-03-06T07:35:44.400442+0000","last_active":"2024-03-06T07:35:45.719234+0000","last_peered":"2024-03-06T07:35:45.719234+0000","last_clean":"2024-03-06T07:35:45.719234+0000","last_became_active":"2024-03-06T07:35:44.399865+0000","last_became_peered":"2024-03-06T07:35:44.399865+0000","last_unstale":"2024-03-06T07:35:45.719234+0000","last_undegraded":"2024-03-06T07:35:45.719234+0000","last_fullsized":"2024-03-06T07:35:45.719234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:26.662064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719793+0000","last_change":"2024-03-06T07:35:43.372351+0000","last_active":"2024-03-06T07:35:45.719793+0000","last_peered":"2024-03-06T07:35:45.719793+0000","last_clean":"2024-03-06T07:35:45.719793+0000","last_became_active":"2024-03-06T07:35:43.372020+0000","last_became_peered":"2024-03-06T07:35:43.372020+0000","last_unstale":"2024-03-06T07:35:45.719793+0000","last_undegraded":"2024-03-06T07:35:45.719793+0000","last_fullsized":"2024-03-06T07:35:45.719793+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:11.361058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394255+0000","last_change":"2024-03-06T07:35:44.412709+0000","last_active":"2024-03-06T07:35:45.394255+0000","last_peered":"2024-03-06T07:35:45.394255+0000","last_clean":"2024-03-06T07:35:45.394255+0000","last_became_active":"2024-03-06T07:35:44.411842+0000","last_became_peered":"2024-03-06T07:35:44.411842+0000","last_unstale":"2024-03-06T07:35:45.394255+0000","last_undegraded":"2024-03-06T07:35:45.394255+0000","last_fullsized":"2024-03-06T07:35:45.394255+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:04.234723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394759+0000","last_change":"2024-03-06T07:35:43.380748+0000","last_active":"2024-03-06T07:35:45.394759+0000","last_peered":"2024-03-06T07:35:45.394759+0000","last_clean":"2024-03-06T07:35:45.394759+0000","last_became_active":"2024-03-06T07:35:43.380512+0000","last_became_peered":"2024-03-06T07:35:43.380512+0000","last_unstale":"2024-03-06T07:35:45.394759+0000","last_undegraded":"2024-03-06T07:35:45.394759+0000","last_fullsized":"2024-03-06T07:35:45.394759+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:02.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720212+0000","last_change":"2024-03-06T07:35:44.416049+0000","last_active":"2024-03-06T07:35:45.720212+0000","last_peered":"2024-03-06T07:35:45.720212+0000","last_clean":"2024-03-06T07:35:45.720212+0000","last_became_active":"2024-03-06T07:35:44.415779+0000","last_became_peered":"2024-03-06T07:35:44.415779+0000","last_unstale":"2024-03-06T07:35:45.720212+0000","last_undegraded":"2024-03-06T07:35:45.720212+0000","last_fullsized":"2024-03-06T07:35:45.720212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:42.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":22,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720702+0000","last_change":"2024-03-06T07:35:43.375271+0000","last_active":"2024-03-06T07:35:45.720702+0000","last_peered":"2024-03-06T07:35:45.720702+0000","last_clean":"2024-03-06T07:35:45.720702+0000","last_became_active":"2024-03-06T07:35:43.374378+0000","last_became_peered":"2024-03-06T07:35:43.374378+0000","last_unstale":"2024-03-06T07:35:45.720702+0000","last_undegraded":"2024-03-06T07:35:45.720702+0000","last_fullsized":"2024-03-06T07:35:45.720702+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:47.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721573+0000","last_change":"2024-03-06T07:35:44.403410+0000","last_active":"2024-03-06T07:35:45.721573+0000","last_peered":"2024-03-06T07:35:45.721573+0000","last_clean":"2024-03-06T07:35:45.721573+0000","last_became_active":"2024-03-06T07:35:44.402014+0000","last_became_peered":"2024-03-06T07:35:44.402014+0000","last_unstale":"2024-03-06T07:35:45.721573+0000","last_undegraded":"2024-03-06T07:35:45.721573+0000","last_fullsized":"2024-03-06T07:35:45.721573+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:50.005746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401803+0000","last_change":"2024-03-06T07:35:43.381126+0000","last_active":"2024-03-06T07:35:45.401803+0000","last_peered":"2024-03-06T07:35:45.401803+0000","last_clean":"2024-03-06T07:35:45.401803+0000","last_became_active":"2024-03-06T07:35:43.380688+0000","last_became_peered":"2024-03-06T07:35:43.380688+0000","last_unstale":"2024-03-06T07:35:45.401803+0000","last_undegraded":"2024-03-06T07:35:45.401803+0000","last_fullsized":"2024-03-06T07:35:45.401803+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:31.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401213+0000","last_change":"2024-03-06T07:35:44.412003+0000","last_active":"2024-03-06T07:35:45.401213+0000","last_peered":"2024-03-06T07:35:45.401213+0000","last_clean":"2024-03-06T07:35:45.401213+0000","last_became_active":"2024-03-06T07:35:44.409915+0000","last_became_peered":"2024-03-06T07:35:44.409915+0000","last_unstale":"2024-03-06T07:35:45.401213+0000","last_undegraded":"2024-03-06T07:35:45.401213+0000","last_fullsized":"2024-03-06T07:35:45.401213+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557877+0000","last_change":"2024-03-06T07:35:43.382676+0000","last_active":"2024-03-06T07:35:45.557877+0000","last_peered":"2024-03-06T07:35:45.557877+0000","last_clean":"2024-03-06T07:35:45.557877+0000","last_became_active":"2024-03-06T07:35:43.380914+0000","last_became_peered":"2024-03-06T07:35:43.380914+0000","last_unstale":"2024-03-06T07:35:45.557877+0000","last_undegraded":"2024-03-06T07:35:45.557877+0000","last_fullsized":"2024-03-06T07:35:45.557877+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.893669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720870+0000","last_change":"2024-03-06T07:35:44.400453+0000","last_active":"2024-03-06T07:35:45.720870+0000","last_peered":"2024-03-06T07:35:45.720870+0000","last_clean":"2024-03-06T07:35:45.720870+0000","last_became_active":"2024-03-06T07:35:44.399501+0000","last_became_peered":"2024-03-06T07:35:44.399501+0000","last_unstale":"2024-03-06T07:35:45.720870+0000","last_undegraded":"2024-03-06T07:35:45.720870+0000","last_fullsized":"2024-03-06T07:35:45.720870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:35:43.377445+0000","last_change":"2024-03-06T07:35:43.375643+0000","last_active":"2024-03-06T07:35:43.377445+0000","last_peered":"2024-03-06T07:35:43.377445+0000","last_clean":"2024-03-06T07:35:43.377445+0000","last_became_active":"2024-03-06T07:35:43.375368+0000","last_became_peered":"2024-03-06T07:35:43.375368+0000","last_unstale":"2024-03-06T07:35:43.377445+0000","last_undegraded":"2024-03-06T07:35:43.377445+0000","last_fullsized":"2024-03-06T07:35:43.377445+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:43.218935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394708+0000","last_change":"2024-03-06T07:35:44.387752+0000","last_active":"2024-03-06T07:35:45.394708+0000","last_peered":"2024-03-06T07:35:45.394708+0000","last_clean":"2024-03-06T07:35:45.394708+0000","last_became_active":"2024-03-06T07:35:44.386625+0000","last_became_peered":"2024-03-06T07:35:44.386625+0000","last_unstale":"2024-03-06T07:35:45.394708+0000","last_undegraded":"2024-03-06T07:35:45.394708+0000","last_fullsized":"2024-03-06T07:35:45.394708+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398845+0000","last_change":"2024-03-06T07:35:43.378109+0000","last_active":"2024-03-06T07:35:45.398845+0000","last_peered":"2024-03-06T07:35:45.398845+0000","last_clean":"2024-03-06T07:35:45.398845+0000","last_became_active":"2024-03-06T07:35:43.377689+0000","last_became_peered":"2024-03-06T07:35:43.377689+0000","last_unstale":"2024-03-06T07:35:45.398845+0000","last_undegraded":"2024-03-06T07:35:45.398845+0000","last_fullsized":"2024-03-06T07:35:45.398845+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:17.604500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719574+0000","last_change":"2024-03-06T07:35:44.406714+0000","last_active":"2024-03-06T07:35:45.719574+0000","last_peered":"2024-03-06T07:35:45.719574+0000","last_clean":"2024-03-06T07:35:45.719574+0000","last_became_active":"2024-03-06T07:35:44.406421+0000","last_became_peered":"2024-03-06T07:35:44.406421+0000","last_unstale":"2024-03-06T07:35:45.719574+0000","last_undegraded":"2024-03-06T07:35:45.719574+0000","last_fullsized":"2024-03-06T07:35:45.719574+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:49.699470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399284+0000","last_change":"2024-03-06T07:35:43.385621+0000","last_active":"2024-03-06T07:35:45.399284+0000","last_peered":"2024-03-06T07:35:45.399284+0000","last_clean":"2024-03-06T07:35:45.399284+0000","last_became_active":"2024-03-06T07:35:43.383393+0000","last_became_peered":"2024-03-06T07:35:43.383393+0000","last_unstale":"2024-03-06T07:35:45.399284+0000","last_undegraded":"2024-03-06T07:35:45.399284+0000","last_fullsized":"2024-03-06T07:35:45.399284+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:45.593408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354223+0000","last_change":"2024-03-06T07:35:43.354223+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354223+0000","last_undegraded":"2024-03-06T07:35:43.354223+0000","last_fullsized":"2024-03-06T07:35:43.354223+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719668+0000","last_change":"2024-03-06T07:35:43.381958+0000","last_active":"2024-03-06T07:35:45.719668+0000","last_peered":"2024-03-06T07:35:45.719668+0000","last_clean":"2024-03-06T07:35:45.719668+0000","last_became_active":"2024-03-06T07:35:43.381369+0000","last_became_peered":"2024-03-06T07:35:43.381369+0000","last_unstale":"2024-03-06T07:35:45.719668+0000","last_undegraded":"2024-03-06T07:35:45.719668+0000","last_fullsized":"2024-03-06T07:35:45.719668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:26.882853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394343+0000","last_change":"2024-03-06T07:35:44.386726+0000","last_active":"2024-03-06T07:35:45.394343+0000","last_peered":"2024-03-06T07:35:45.394343+0000","last_clean":"2024-03-06T07:35:45.394343+0000","last_became_active":"2024-03-06T07:35:44.385066+0000","last_became_peered":"2024-03-06T07:35:44.385066+0000","last_unstale":"2024-03-06T07:35:45.394343+0000","last_undegraded":"2024-03-06T07:35:45.394343+0000","last_fullsized":"2024-03-06T07:35:45.394343+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:29.921786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719327+0000","last_change":"2024-03-06T07:35:43.381967+0000","last_active":"2024-03-06T07:35:45.719327+0000","last_peered":"2024-03-06T07:35:45.719327+0000","last_clean":"2024-03-06T07:35:45.719327+0000","last_became_active":"2024-03-06T07:35:43.381593+0000","last_became_peered":"2024-03-06T07:35:43.381593+0000","last_unstale":"2024-03-06T07:35:45.719327+0000","last_undegraded":"2024-03-06T07:35:45.719327+0000","last_fullsized":"2024-03-06T07:35:45.719327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:32.956490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719534+0000","last_change":"2024-03-06T07:35:44.416923+0000","last_active":"2024-03-06T07:35:45.719534+0000","last_peered":"2024-03-06T07:35:45.719534+0000","last_clean":"2024-03-06T07:35:45.719534+0000","last_became_active":"2024-03-06T07:35:44.414762+0000","last_became_peered":"2024-03-06T07:35:44.414762+0000","last_unstale":"2024-03-06T07:35:45.719534+0000","last_undegraded":"2024-03-06T07:35:45.719534+0000","last_fullsized":"2024-03-06T07:35:45.719534+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:18.903803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557365+0000","last_change":"2024-03-06T07:35:43.378687+0000","last_active":"2024-03-06T07:35:45.557365+0000","last_peered":"2024-03-06T07:35:45.557365+0000","last_clean":"2024-03-06T07:35:45.557365+0000","last_became_active":"2024-03-06T07:35:43.378095+0000","last_became_peered":"2024-03-06T07:35:43.378095+0000","last_unstale":"2024-03-06T07:35:45.557365+0000","last_undegraded":"2024-03-06T07:35:45.557365+0000","last_fullsized":"2024-03-06T07:35:45.557365+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:11.480478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354748+0000","last_change":"2024-03-06T07:35:43.354748+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354748+0000","last_undegraded":"2024-03-06T07:35:43.354748+0000","last_fullsized":"2024-03-06T07:35:43.354748+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720668+0000","last_change":"2024-03-06T07:35:43.377991+0000","last_active":"2024-03-06T07:35:45.720668+0000","last_peered":"2024-03-06T07:35:45.720668+0000","last_clean":"2024-03-06T07:35:45.720668+0000","last_became_active":"2024-03-06T07:35:43.377732+0000","last_became_peered":"2024-03-06T07:35:43.377732+0000","last_unstale":"2024-03-06T07:35:45.720668+0000","last_undegraded":"2024-03-06T07:35:45.720668+0000","last_fullsized":"2024-03-06T07:35:45.720668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.997519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720542+0000","last_change":"2024-03-06T07:35:44.415821+0000","last_active":"2024-03-06T07:35:45.720542+0000","last_peered":"2024-03-06T07:35:45.720542+0000","last_clean":"2024-03-06T07:35:45.720542+0000","last_became_active":"2024-03-06T07:35:44.415487+0000","last_became_peered":"2024-03-06T07:35:44.415487+0000","last_unstale":"2024-03-06T07:35:45.720542+0000","last_undegraded":"2024-03-06T07:35:45.720542+0000","last_fullsized":"2024-03-06T07:35:45.720542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:46.609382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719944+0000","last_change":"2024-03-06T07:35:43.385136+0000","last_active":"2024-03-06T07:35:45.719944+0000","last_peered":"2024-03-06T07:35:45.719944+0000","last_clean":"2024-03-06T07:35:45.719944+0000","last_became_active":"2024-03-06T07:35:43.384757+0000","last_became_peered":"2024-03-06T07:35:43.384757+0000","last_unstale":"2024-03-06T07:35:45.719944+0000","last_undegraded":"2024-03-06T07:35:45.719944+0000","last_fullsized":"2024-03-06T07:35:45.719944+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:20.137806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394855+0000","last_change":"2024-03-06T07:35:44.417187+0000","last_active":"2024-03-06T07:35:45.394855+0000","last_peered":"2024-03-06T07:35:45.394855+0000","last_clean":"2024-03-06T07:35:45.394855+0000","last_became_active":"2024-03-06T07:35:44.416892+0000","last_became_peered":"2024-03-06T07:35:44.416892+0000","last_unstale":"2024-03-06T07:35:45.394855+0000","last_undegraded":"2024-03-06T07:35:45.394855+0000","last_fullsized":"2024-03-06T07:35:45.394855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394798+0000","last_change":"2024-03-06T07:35:43.377777+0000","last_active":"2024-03-06T07:35:45.394798+0000","last_peered":"2024-03-06T07:35:45.394798+0000","last_clean":"2024-03-06T07:35:45.394798+0000","last_became_active":"2024-03-06T07:35:43.374868+0000","last_became_peered":"2024-03-06T07:35:43.374868+0000","last_unstale":"2024-03-06T07:35:45.394798+0000","last_undegraded":"2024-03-06T07:35:45.394798+0000","last_fullsized":"2024-03-06T07:35:45.394798+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:57.102370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":128,"state":"creating+peering","last_fresh":"2024-03-06T07:35:43.354203+0000","last_change":"2024-03-06T07:35:43.354203+0000","last_active":"2024-03-06T07:35:43.339559+0000","last_peered":"2024-03-06T07:35:43.339559+0000","last_clean":"2024-03-06T07:35:43.339559+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:43.354203+0000","last_undegraded":"2024-03-06T07:35:43.354203+0000","last_fullsized":"2024-03-06T07:35:43.354203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557609+0000","last_change":"2024-03-06T07:35:43.368644+0000","last_active":"2024-03-06T07:35:45.557609+0000","last_peered":"2024-03-06T07:35:45.557609+0000","last_clean":"2024-03-06T07:35:45.557609+0000","last_became_active":"2024-03-06T07:35:43.367936+0000","last_became_peered":"2024-03-06T07:35:43.367936+0000","last_unstale":"2024-03-06T07:35:45.557609+0000","last_undegraded":"2024-03-06T07:35:45.557609+0000","last_fullsized":"2024-03-06T07:35:45.557609+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:01.783718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720731+0000","last_change":"2024-03-06T07:35:44.417357+0000","last_active":"2024-03-06T07:35:45.720731+0000","last_peered":"2024-03-06T07:35:45.720731+0000","last_clean":"2024-03-06T07:35:45.720731+0000","last_became_active":"2024-03-06T07:35:44.417021+0000","last_became_peered":"2024-03-06T07:35:44.417021+0000","last_unstale":"2024-03-06T07:35:45.720731+0000","last_undegraded":"2024-03-06T07:35:45.720731+0000","last_fullsized":"2024-03-06T07:35:45.720731+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:11.909643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394493+0000","last_change":"2024-03-06T07:35:43.376770+0000","last_active":"2024-03-06T07:35:45.394493+0000","last_peered":"2024-03-06T07:35:45.394493+0000","last_clean":"2024-03-06T07:35:45.394493+0000","last_became_active":"2024-03-06T07:35:43.376483+0000","last_became_peered":"2024-03-06T07:35:43.376483+0000","last_unstale":"2024-03-06T07:35:45.394493+0000","last_undegraded":"2024-03-06T07:35:45.394493+0000","last_fullsized":"2024-03-06T07:35:45.394493+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:32.527625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399036+0000","last_change":"2024-03-06T07:35:44.408484+0000","last_active":"2024-03-06T07:35:45.399036+0000","last_peered":"2024-03-06T07:35:45.399036+0000","last_clean":"2024-03-06T07:35:45.399036+0000","last_became_active":"2024-03-06T07:35:44.407992+0000","last_became_peered":"2024-03-06T07:35:44.407992+0000","last_unstale":"2024-03-06T07:35:45.399036+0000","last_undegraded":"2024-03-06T07:35:45.399036+0000","last_fullsized":"2024-03-06T07:35:45.399036+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720795+0000","last_change":"2024-03-06T07:35:43.385517+0000","last_active":"2024-03-06T07:35:45.720795+0000","last_peered":"2024-03-06T07:35:45.720795+0000","last_clean":"2024-03-06T07:35:45.720795+0000","last_became_active":"2024-03-06T07:35:43.384210+0000","last_became_peered":"2024-03-06T07:35:43.384210+0000","last_unstale":"2024-03-06T07:35:45.720795+0000","last_undegraded":"2024-03-06T07:35:45.720795+0000","last_fullsized":"2024-03-06T07:35:45.720795+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:11.817793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719634+0000","last_change":"2024-03-06T07:35:44.412773+0000","last_active":"2024-03-06T07:35:45.719634+0000","last_peered":"2024-03-06T07:35:45.719634+0000","last_clean":"2024-03-06T07:35:45.719634+0000","last_became_active":"2024-03-06T07:35:44.411292+0000","last_became_peered":"2024-03-06T07:35:44.411292+0000","last_unstale":"2024-03-06T07:35:45.719634+0000","last_undegraded":"2024-03-06T07:35:45.719634+0000","last_fullsized":"2024-03-06T07:35:45.719634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719424+0000","last_change":"2024-03-06T07:35:43.379581+0000","last_active":"2024-03-06T07:35:45.719424+0000","last_peered":"2024-03-06T07:35:45.719424+0000","last_clean":"2024-03-06T07:35:45.719424+0000","last_became_active":"2024-03-06T07:35:43.379237+0000","last_became_peered":"2024-03-06T07:35:43.379237+0000","last_unstale":"2024-03-06T07:35:45.719424+0000","last_undegraded":"2024-03-06T07:35:45.719424+0000","last_fullsized":"2024-03-06T07:35:45.719424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:32.312256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394668+0000","last_change":"2024-03-06T07:35:44.417512+0000","last_active":"2024-03-06T07:35:45.394668+0000","last_peered":"2024-03-06T07:35:45.394668+0000","last_clean":"2024-03-06T07:35:45.394668+0000","last_became_active":"2024-03-06T07:35:44.416206+0000","last_became_peered":"2024-03-06T07:35:44.416206+0000","last_unstale":"2024-03-06T07:35:45.394668+0000","last_undegraded":"2024-03-06T07:35:45.394668+0000","last_fullsized":"2024-03-06T07:35:45.394668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557543+0000","last_change":"2024-03-06T07:35:43.375969+0000","last_active":"2024-03-06T07:35:45.557543+0000","last_peered":"2024-03-06T07:35:45.557543+0000","last_clean":"2024-03-06T07:35:45.557543+0000","last_became_active":"2024-03-06T07:35:43.375710+0000","last_became_peered":"2024-03-06T07:35:43.375710+0000","last_unstale":"2024-03-06T07:35:45.557543+0000","last_undegraded":"2024-03-06T07:35:45.557543+0000","last_fullsized":"2024-03-06T07:35:45.557543+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:38.782993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720514+0000","last_change":"2024-03-06T07:35:44.414475+0000","last_active":"2024-03-06T07:35:45.720514+0000","last_peered":"2024-03-06T07:35:45.720514+0000","last_clean":"2024-03-06T07:35:45.720514+0000","last_became_active":"2024-03-06T07:35:44.414166+0000","last_became_peered":"2024-03-06T07:35:44.414166+0000","last_unstale":"2024-03-06T07:35:45.720514+0000","last_undegraded":"2024-03-06T07:35:45.720514+0000","last_fullsized":"2024-03-06T07:35:45.720514+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:39.798967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721268+0000","last_change":"2024-03-06T07:35:43.385255+0000","last_active":"2024-03-06T07:35:45.721268+0000","last_peered":"2024-03-06T07:35:45.721268+0000","last_clean":"2024-03-06T07:35:45.721268+0000","last_became_active":"2024-03-06T07:35:43.384945+0000","last_became_peered":"2024-03-06T07:35:43.384945+0000","last_unstale":"2024-03-06T07:35:45.721268+0000","last_undegraded":"2024-03-06T07:35:45.721268+0000","last_fullsized":"2024-03-06T07:35:45.721268+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:41.317784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":369,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719242+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:35:45.719242+0000","last_peered":"2024-03-06T07:35:45.719242+0000","last_clean":"2024-03-06T07:35:45.719242+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:35:45.719242+0000","last_undegraded":"2024-03-06T07:35:45.719242+0000","last_fullsized":"2024-03-06T07:35:45.719242+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2236416,"data_stored":1296790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542246,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1630208,"data_stored":705784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":691097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542247,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1626112,"data_stored":704788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1617920,"data_stored":704172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41660,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330180,"statfs":{"total":96636764160,"available":96594104320,"internally_reserved":0,"allocated":2211840,"data_stored":1294540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:53.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:54.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.117+0000 7f8947800640 1 Processor -- start 2024-03-06T07:35:54.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.117+0000 7f8947800640 1 -- start start 2024-03-06T07:35:54.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8947800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f894015e160 0x7f8940162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8947800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f89400b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8947800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8940162b60 con 0x7f89400b38f0 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8947800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894006b0f0 con 0x7f894015e160 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8947800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89400b5620 con 0x7f89400af820 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8945800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f89400b38f0 0x7f894006d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42796/0 (socket says 172.21.15.97:42796) 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8945800640 1 -- 172.21.15.97:0/3308459491 learned_addr learned my addr 172.21.15.97:0/3308459491 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.119+0000 7f8944e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f894015e160 0x7f8940162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.120+0000 7f893fe00640 1 --2- 172.21.15.97:0/3308459491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f89400b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.120+0000 7f893f400640 1 -- 172.21.15.97:0/3308459491 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780770155 0 0) 0x7f8938001490 con 0x7f89400b38f0 2024-03-06T07:35:54.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.120+0000 7f8944e00640 1 -- 172.21.15.97:0/3308459491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 msgr2=0x7f89400b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.120+0000 7f8944e00640 1 --2- 172.21.15.97:0/3308459491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f89400b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.120+0000 7f8944e00640 1 -- 172.21.15.97:0/3308459491 >> v1:172.21.15.97:6789/0 conn(0x7f89400b38f0 legacy=0x7f894006d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.120+0000 7f8944e00640 1 -- 172.21.15.97:0/3308459491 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89400b3d70 con 0x7f894015e160 2024-03-06T07:35:54.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.121+0000 7f8944e00640 1 --2- 172.21.15.97:0/3308459491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940162620 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f89340097a0 tx=0x7f893402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1afdca1a2bc85880 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.121+0000 7f893f400640 1 -- 172.21.15.97:0/3308459491 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340315a0 con 0x7f894015e160 2024-03-06T07:35:54.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.121+0000 7f893f400640 1 -- 172.21.15.97:0/3308459491 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934037070 con 0x7f894015e160 2024-03-06T07:35:54.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.121+0000 7f893f400640 1 -- 172.21.15.97:0/3308459491 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934031960 con 0x7f894015e160 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 -- 172.21.15.97:0/3308459491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 msgr2=0x7f8940162620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 --2- 172.21.15.97:0/3308459491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940162620 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f89340097a0 tx=0x7f893402b840 comp rx=0 tx=0).stop 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 -- 172.21.15.97:0/3308459491 shutdown_connections 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 --2- 172.21.15.97:0/3308459491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f89400b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 --2- 172.21.15.97:0/3308459491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940162620 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 -- 172.21.15.97:0/3308459491 >> 172.21.15.97:0/3308459491 conn(0x7f89400d0340 msgr2=0x7f89400d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 -- 172.21.15.97:0/3308459491 shutdown_connections 2024-03-06T07:35:54.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.122+0000 7f8947800640 1 -- 172.21.15.97:0/3308459491 wait complete. 2024-03-06T07:35:54.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.124+0000 7f8947800640 1 Processor -- start 2024-03-06T07:35:54.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.125+0000 7f8947800640 1 -- start start 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8947800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f894015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8947800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89400b38f0 0x7f89400ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8947800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8947800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894006ec30 con 0x7f894015e160 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8947800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89400afce0 con 0x7f89400b38f0 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8947800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894006ef10 con 0x7f89400af820 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8945800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8945800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39556/0 (socket says 172.21.15.97:39556) 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.126+0000 7f8945800640 1 -- 172.21.15.97:0/713374322 learned_addr learned my addr 172.21.15.97:0/713374322 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:54.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8944e00640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f894015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f893fe00640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89400b38f0 0x7f89400ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8945800640 1 -- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 msgr2=0x7f894015ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8945800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f894015ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8945800640 1 -- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89400b38f0 msgr2=0x7f89400ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8945800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89400b38f0 0x7f89400ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8945800640 1 -- 172.21.15.97:0/713374322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8940157870 con 0x7f894015e160 2024-03-06T07:35:54.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f893fe00640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89400b38f0 0x7f89400ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f8945800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940157330 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f893800c7b0 tx=0x7f893800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.127+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938018070 con 0x7f894015e160 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.128+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8938004d60 con 0x7f894015e160 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.128+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8940157b10 con 0x7f894015e160 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.128+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893800a880 con 0x7f894015e160 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.128+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89401580d0 con 0x7f894015e160 2024-03-06T07:35:54.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.129+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89400b4260 con 0x7f894015e160 2024-03-06T07:35:54.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.130+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f893801b070 con 0x7f894015e160 2024-03-06T07:35:54.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.131+0000 7f893cc00640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f892006b4d0 0x7f892006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.131+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89380896b0 con 0x7f894015e160 2024-03-06T07:35:54.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.131+0000 7f8944e00640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f892006b4d0 0x7f892006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.132+0000 7f8944e00640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f892006b4d0 0x7f892006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8934036000 tx=0x7f8934008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.136+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89400b4260 con 0x7f894015e160 2024-03-06T07:35:54.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.347+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89400d11e0 con 0x7f892006b4d0 2024-03-06T07:35:54.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.361+0000 7f893cc00640 1 -- 172.21.15.97:0/713374322 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f89400d11e0 con 0x7f892006b4d0 2024-03-06T07:35:54.366 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f892006b4d0 msgr2=0x7f892006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f892006b4d0 0x7f892006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8934036000 tx=0x7f8934008030 comp rx=0 tx=0).stop 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 msgr2=0x7f8940157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940157330 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f893800c7b0 tx=0x7f893800cc70 comp rx=0 tx=0).stop 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 shutdown_connections 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f892006b4d0 0x7f892006d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894015e160 0x7f8940157330 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89400b38f0 0x7f89400ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 --2- 172.21.15.97:0/713374322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89400af820 0x7f894015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 >> 172.21.15.97:0/713374322 conn(0x7f89400d0340 msgr2=0x7f894006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 shutdown_connections 2024-03-06T07:35:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.366+0000 7f8947800640 1 -- 172.21.15.97:0/713374322 wait complete. 2024-03-06T07:35:54.388 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-03-06T07:35:53.053385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304924,"kb_used_data":13864,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754669796,"statfs":{"total":773094113280,"available":772781871104,"internally_reserved":0,"allocated":14196736,"data_stored":6818932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.682220"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720739+0000","last_change":"2024-03-06T07:35:44.394272+0000","last_active":"2024-03-06T07:35:45.720739+0000","last_peered":"2024-03-06T07:35:45.720739+0000","last_clean":"2024-03-06T07:35:45.720739+0000","last_became_active":"2024-03-06T07:35:44.393967+0000","last_became_peered":"2024-03-06T07:35:44.393967+0000","last_unstale":"2024-03-06T07:35:45.720739+0000","last_undegraded":"2024-03-06T07:35:45.720739+0000","last_fullsized":"2024-03-06T07:35:45.720739+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:28.599416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394850+0000","last_change":"2024-03-06T07:35:44.392186+0000","last_active":"2024-03-06T07:35:45.394850+0000","last_peered":"2024-03-06T07:35:45.394850+0000","last_clean":"2024-03-06T07:35:45.394850+0000","last_became_active":"2024-03-06T07:35:44.391446+0000","last_became_peered":"2024-03-06T07:35:44.391446+0000","last_unstale":"2024-03-06T07:35:45.394850+0000","last_undegraded":"2024-03-06T07:35:45.394850+0000","last_fullsized":"2024-03-06T07:35:45.394850+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:25.601824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720157+0000","last_change":"2024-03-06T07:35:44.410356+0000","last_active":"2024-03-06T07:35:45.720157+0000","last_peered":"2024-03-06T07:35:45.720157+0000","last_clean":"2024-03-06T07:35:45.720157+0000","last_became_active":"2024-03-06T07:35:44.409633+0000","last_became_peered":"2024-03-06T07:35:44.409633+0000","last_unstale":"2024-03-06T07:35:45.720157+0000","last_undegraded":"2024-03-06T07:35:45.720157+0000","last_fullsized":"2024-03-06T07:35:45.720157+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:12.833768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718970+0000","last_change":"2024-03-06T07:35:44.417975+0000","last_active":"2024-03-06T07:35:45.718970+0000","last_peered":"2024-03-06T07:35:45.718970+0000","last_clean":"2024-03-06T07:35:45.718970+0000","last_became_active":"2024-03-06T07:35:44.417719+0000","last_became_peered":"2024-03-06T07:35:44.417719+0000","last_unstale":"2024-03-06T07:35:45.718970+0000","last_undegraded":"2024-03-06T07:35:45.718970+0000","last_fullsized":"2024-03-06T07:35:45.718970+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:26.008403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721342+0000","last_change":"2024-03-06T07:35:44.415585+0000","last_active":"2024-03-06T07:35:45.721342+0000","last_peered":"2024-03-06T07:35:45.721342+0000","last_clean":"2024-03-06T07:35:45.721342+0000","last_became_active":"2024-03-06T07:35:44.415011+0000","last_became_peered":"2024-03-06T07:35:44.415011+0000","last_unstale":"2024-03-06T07:35:45.721342+0000","last_undegraded":"2024-03-06T07:35:45.721342+0000","last_fullsized":"2024-03-06T07:35:45.721342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400614+0000","last_change":"2024-03-06T07:35:44.406778+0000","last_active":"2024-03-06T07:35:45.400614+0000","last_peered":"2024-03-06T07:35:45.400614+0000","last_clean":"2024-03-06T07:35:45.400614+0000","last_became_active":"2024-03-06T07:35:44.406435+0000","last_became_peered":"2024-03-06T07:35:44.406435+0000","last_unstale":"2024-03-06T07:35:45.400614+0000","last_undegraded":"2024-03-06T07:35:45.400614+0000","last_fullsized":"2024-03-06T07:35:45.400614+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:57.456196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399993+0000","last_change":"2024-03-06T07:35:44.413229+0000","last_active":"2024-03-06T07:35:45.399993+0000","last_peered":"2024-03-06T07:35:45.399993+0000","last_clean":"2024-03-06T07:35:45.399993+0000","last_became_active":"2024-03-06T07:35:44.412884+0000","last_became_peered":"2024-03-06T07:35:44.412884+0000","last_unstale":"2024-03-06T07:35:45.399993+0000","last_undegraded":"2024-03-06T07:35:45.399993+0000","last_fullsized":"2024-03-06T07:35:45.399993+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:59.263024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719898+0000","last_change":"2024-03-06T07:35:44.405572+0000","last_active":"2024-03-06T07:35:45.719898+0000","last_peered":"2024-03-06T07:35:45.719898+0000","last_clean":"2024-03-06T07:35:45.719898+0000","last_became_active":"2024-03-06T07:35:44.405225+0000","last_became_peered":"2024-03-06T07:35:44.405225+0000","last_unstale":"2024-03-06T07:35:45.719898+0000","last_undegraded":"2024-03-06T07:35:45.719898+0000","last_fullsized":"2024-03-06T07:35:45.719898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:21.153780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721703+0000","last_change":"2024-03-06T07:35:44.415674+0000","last_active":"2024-03-06T07:35:45.721703+0000","last_peered":"2024-03-06T07:35:45.721703+0000","last_clean":"2024-03-06T07:35:45.721703+0000","last_became_active":"2024-03-06T07:35:44.415198+0000","last_became_peered":"2024-03-06T07:35:44.415198+0000","last_unstale":"2024-03-06T07:35:45.721703+0000","last_undegraded":"2024-03-06T07:35:45.721703+0000","last_fullsized":"2024-03-06T07:35:45.721703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:27.898827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395062+0000","last_change":"2024-03-06T07:35:44.398339+0000","last_active":"2024-03-06T07:35:45.395062+0000","last_peered":"2024-03-06T07:35:45.395062+0000","last_clean":"2024-03-06T07:35:45.395062+0000","last_became_active":"2024-03-06T07:35:44.396519+0000","last_became_peered":"2024-03-06T07:35:44.396519+0000","last_unstale":"2024-03-06T07:35:45.395062+0000","last_undegraded":"2024-03-06T07:35:45.395062+0000","last_fullsized":"2024-03-06T07:35:45.395062+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:48.438390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557893+0000","last_change":"2024-03-06T07:35:44.397455+0000","last_active":"2024-03-06T07:35:45.557893+0000","last_peered":"2024-03-06T07:35:45.557893+0000","last_clean":"2024-03-06T07:35:45.557893+0000","last_became_active":"2024-03-06T07:35:44.397217+0000","last_became_peered":"2024-03-06T07:35:44.397217+0000","last_unstale":"2024-03-06T07:35:45.557893+0000","last_undegraded":"2024-03-06T07:35:45.557893+0000","last_fullsized":"2024-03-06T07:35:45.557893+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:01.527107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720004+0000","last_change":"2024-03-06T07:35:44.416953+0000","last_active":"2024-03-06T07:35:45.720004+0000","last_peered":"2024-03-06T07:35:45.720004+0000","last_clean":"2024-03-06T07:35:45.720004+0000","last_became_active":"2024-03-06T07:35:44.416659+0000","last_became_peered":"2024-03-06T07:35:44.416659+0000","last_unstale":"2024-03-06T07:35:45.720004+0000","last_undegraded":"2024-03-06T07:35:45.720004+0000","last_fullsized":"2024-03-06T07:35:45.720004+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:31.633974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719022+0000","last_change":"2024-03-06T07:35:44.395498+0000","last_active":"2024-03-06T07:35:45.719022+0000","last_peered":"2024-03-06T07:35:45.719022+0000","last_clean":"2024-03-06T07:35:45.719022+0000","last_became_active":"2024-03-06T07:35:44.395203+0000","last_became_peered":"2024-03-06T07:35:44.395203+0000","last_unstale":"2024-03-06T07:35:45.719022+0000","last_undegraded":"2024-03-06T07:35:45.719022+0000","last_fullsized":"2024-03-06T07:35:45.719022+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:58.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394607+0000","last_change":"2024-03-06T07:35:44.400161+0000","last_active":"2024-03-06T07:35:45.394607+0000","last_peered":"2024-03-06T07:35:45.394607+0000","last_clean":"2024-03-06T07:35:45.394607+0000","last_became_active":"2024-03-06T07:35:44.399845+0000","last_became_peered":"2024-03-06T07:35:44.399845+0000","last_unstale":"2024-03-06T07:35:45.394607+0000","last_undegraded":"2024-03-06T07:35:45.394607+0000","last_fullsized":"2024-03-06T07:35:45.394607+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:33.642819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.718604+0000","last_change":"2024-03-06T07:35:44.406110+0000","last_active":"2024-03-06T07:35:45.718604+0000","last_peered":"2024-03-06T07:35:45.718604+0000","last_clean":"2024-03-06T07:35:45.718604+0000","last_became_active":"2024-03-06T07:35:44.405807+0000","last_became_peered":"2024-03-06T07:35:44.405807+0000","last_unstale":"2024-03-06T07:35:45.718604+0000","last_undegraded":"2024-03-06T07:35:45.718604+0000","last_fullsized":"2024-03-06T07:35:45.718604+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:04.519517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719509+0000","last_change":"2024-03-06T07:35:44.395764+0000","last_active":"2024-03-06T07:35:45.719509+0000","last_peered":"2024-03-06T07:35:45.719509+0000","last_clean":"2024-03-06T07:35:45.719509+0000","last_became_active":"2024-03-06T07:35:44.393280+0000","last_became_peered":"2024-03-06T07:35:44.393280+0000","last_unstale":"2024-03-06T07:35:45.719509+0000","last_undegraded":"2024-03-06T07:35:45.719509+0000","last_fullsized":"2024-03-06T07:35:45.719509+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401083+0000","last_change":"2024-03-06T07:35:44.403536+0000","last_active":"2024-03-06T07:35:45.401083+0000","last_peered":"2024-03-06T07:35:45.401083+0000","last_clean":"2024-03-06T07:35:45.401083+0000","last_became_active":"2024-03-06T07:35:44.401326+0000","last_became_peered":"2024-03-06T07:35:44.401326+0000","last_unstale":"2024-03-06T07:35:45.401083+0000","last_undegraded":"2024-03-06T07:35:45.401083+0000","last_fullsized":"2024-03-06T07:35:45.401083+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:10.581432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720362+0000","last_change":"2024-03-06T07:35:44.395358+0000","last_active":"2024-03-06T07:35:45.720362+0000","last_peered":"2024-03-06T07:35:45.720362+0000","last_clean":"2024-03-06T07:35:45.720362+0000","last_became_active":"2024-03-06T07:35:44.395025+0000","last_became_peered":"2024-03-06T07:35:44.395025+0000","last_unstale":"2024-03-06T07:35:45.720362+0000","last_undegraded":"2024-03-06T07:35:45.720362+0000","last_fullsized":"2024-03-06T07:35:45.720362+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558069+0000","last_change":"2024-03-06T07:35:44.408168+0000","last_active":"2024-03-06T07:35:45.558069+0000","last_peered":"2024-03-06T07:35:45.558069+0000","last_clean":"2024-03-06T07:35:45.558069+0000","last_became_active":"2024-03-06T07:35:44.407536+0000","last_became_peered":"2024-03-06T07:35:44.407536+0000","last_unstale":"2024-03-06T07:35:45.558069+0000","last_undegraded":"2024-03-06T07:35:45.558069+0000","last_fullsized":"2024-03-06T07:35:45.558069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:45.475165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720163+0000","last_change":"2024-03-06T07:35:44.390845+0000","last_active":"2024-03-06T07:35:45.720163+0000","last_peered":"2024-03-06T07:35:45.720163+0000","last_clean":"2024-03-06T07:35:45.720163+0000","last_became_active":"2024-03-06T07:35:44.388466+0000","last_became_peered":"2024-03-06T07:35:44.388466+0000","last_unstale":"2024-03-06T07:35:45.720163+0000","last_undegraded":"2024-03-06T07:35:45.720163+0000","last_fullsized":"2024-03-06T07:35:45.720163+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721001+0000","last_change":"2024-03-06T07:35:44.415326+0000","last_active":"2024-03-06T07:35:45.721001+0000","last_peered":"2024-03-06T07:35:45.721001+0000","last_clean":"2024-03-06T07:35:45.721001+0000","last_became_active":"2024-03-06T07:35:44.414356+0000","last_became_peered":"2024-03-06T07:35:44.414356+0000","last_unstale":"2024-03-06T07:35:45.721001+0000","last_undegraded":"2024-03-06T07:35:45.721001+0000","last_fullsized":"2024-03-06T07:35:45.721001+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:45.267184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557466+0000","last_change":"2024-03-06T07:35:44.394518+0000","last_active":"2024-03-06T07:35:45.557466+0000","last_peered":"2024-03-06T07:35:45.557466+0000","last_clean":"2024-03-06T07:35:45.557466+0000","last_became_active":"2024-03-06T07:35:44.392743+0000","last_became_peered":"2024-03-06T07:35:44.392743+0000","last_unstale":"2024-03-06T07:35:45.557466+0000","last_undegraded":"2024-03-06T07:35:45.557466+0000","last_fullsized":"2024-03-06T07:35:45.557466+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:34.320907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720515+0000","last_change":"2024-03-06T07:35:44.417926+0000","last_active":"2024-03-06T07:35:45.720515+0000","last_peered":"2024-03-06T07:35:45.720515+0000","last_clean":"2024-03-06T07:35:45.720515+0000","last_became_active":"2024-03-06T07:35:44.415414+0000","last_became_peered":"2024-03-06T07:35:44.415414+0000","last_unstale":"2024-03-06T07:35:45.720515+0000","last_undegraded":"2024-03-06T07:35:45.720515+0000","last_fullsized":"2024-03-06T07:35:45.720515+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:14.302817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557430+0000","last_change":"2024-03-06T07:35:44.408475+0000","last_active":"2024-03-06T07:35:45.557430+0000","last_peered":"2024-03-06T07:35:45.557430+0000","last_clean":"2024-03-06T07:35:45.557430+0000","last_became_active":"2024-03-06T07:35:44.408227+0000","last_became_peered":"2024-03-06T07:35:44.408227+0000","last_unstale":"2024-03-06T07:35:45.557430+0000","last_undegraded":"2024-03-06T07:35:45.557430+0000","last_fullsized":"2024-03-06T07:35:45.557430+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.395196+0000","last_change":"2024-03-06T07:35:44.416789+0000","last_active":"2024-03-06T07:35:45.395196+0000","last_peered":"2024-03-06T07:35:45.395196+0000","last_clean":"2024-03-06T07:35:45.395196+0000","last_became_active":"2024-03-06T07:35:44.416274+0000","last_became_peered":"2024-03-06T07:35:44.416274+0000","last_unstale":"2024-03-06T07:35:45.395196+0000","last_undegraded":"2024-03-06T07:35:45.395196+0000","last_fullsized":"2024-03-06T07:35:45.395196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:47.395736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720627+0000","last_change":"2024-03-06T07:35:44.399980+0000","last_active":"2024-03-06T07:35:45.720627+0000","last_peered":"2024-03-06T07:35:45.720627+0000","last_clean":"2024-03-06T07:35:45.720627+0000","last_became_active":"2024-03-06T07:35:44.399328+0000","last_became_peered":"2024-03-06T07:35:44.399328+0000","last_unstale":"2024-03-06T07:35:45.720627+0000","last_undegraded":"2024-03-06T07:35:45.720627+0000","last_fullsized":"2024-03-06T07:35:45.720627+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:43.875737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719802+0000","last_change":"2024-03-06T07:35:44.394219+0000","last_active":"2024-03-06T07:35:45.719802+0000","last_peered":"2024-03-06T07:35:45.719802+0000","last_clean":"2024-03-06T07:35:45.719802+0000","last_became_active":"2024-03-06T07:35:44.393893+0000","last_became_peered":"2024-03-06T07:35:44.393893+0000","last_unstale":"2024-03-06T07:35:45.719802+0000","last_undegraded":"2024-03-06T07:35:45.719802+0000","last_fullsized":"2024-03-06T07:35:45.719802+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:18.620474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398830+0000","last_change":"2024-03-06T07:35:44.403178+0000","last_active":"2024-03-06T07:35:45.398830+0000","last_peered":"2024-03-06T07:35:45.398830+0000","last_clean":"2024-03-06T07:35:45.398830+0000","last_became_active":"2024-03-06T07:35:44.402903+0000","last_became_peered":"2024-03-06T07:35:44.402903+0000","last_unstale":"2024-03-06T07:35:45.398830+0000","last_undegraded":"2024-03-06T07:35:45.398830+0000","last_fullsized":"2024-03-06T07:35:45.398830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719226+0000","last_change":"2024-03-06T07:35:44.415700+0000","last_active":"2024-03-06T07:35:45.719226+0000","last_peered":"2024-03-06T07:35:45.719226+0000","last_clean":"2024-03-06T07:35:45.719226+0000","last_became_active":"2024-03-06T07:35:44.415400+0000","last_became_peered":"2024-03-06T07:35:44.415400+0000","last_unstale":"2024-03-06T07:35:45.719226+0000","last_undegraded":"2024-03-06T07:35:45.719226+0000","last_fullsized":"2024-03-06T07:35:45.719226+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:56.935549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719196+0000","last_change":"2024-03-06T07:35:44.406337+0000","last_active":"2024-03-06T07:35:45.719196+0000","last_peered":"2024-03-06T07:35:45.719196+0000","last_clean":"2024-03-06T07:35:45.719196+0000","last_became_active":"2024-03-06T07:35:44.405995+0000","last_became_peered":"2024-03-06T07:35:44.405995+0000","last_unstale":"2024-03-06T07:35:45.719196+0000","last_undegraded":"2024-03-06T07:35:45.719196+0000","last_fullsized":"2024-03-06T07:35:45.719196+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:51.011567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719669+0000","last_change":"2024-03-06T07:35:44.415228+0000","last_active":"2024-03-06T07:35:45.719669+0000","last_peered":"2024-03-06T07:35:45.719669+0000","last_clean":"2024-03-06T07:35:45.719669+0000","last_became_active":"2024-03-06T07:35:44.414418+0000","last_became_peered":"2024-03-06T07:35:44.414418+0000","last_unstale":"2024-03-06T07:35:45.719669+0000","last_undegraded":"2024-03-06T07:35:45.719669+0000","last_fullsized":"2024-03-06T07:35:45.719669+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:25.327417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400043+0000","last_change":"2024-03-06T07:35:44.398139+0000","last_active":"2024-03-06T07:35:45.400043+0000","last_peered":"2024-03-06T07:35:45.400043+0000","last_clean":"2024-03-06T07:35:45.400043+0000","last_became_active":"2024-03-06T07:35:44.397747+0000","last_became_peered":"2024-03-06T07:35:44.397747+0000","last_unstale":"2024-03-06T07:35:45.400043+0000","last_undegraded":"2024-03-06T07:35:45.400043+0000","last_fullsized":"2024-03-06T07:35:45.400043+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:14.059027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720567+0000","last_change":"2024-03-06T07:35:44.408316+0000","last_active":"2024-03-06T07:35:45.720567+0000","last_peered":"2024-03-06T07:35:45.720567+0000","last_clean":"2024-03-06T07:35:45.720567+0000","last_became_active":"2024-03-06T07:35:44.407617+0000","last_became_peered":"2024-03-06T07:35:44.407617+0000","last_unstale":"2024-03-06T07:35:45.720567+0000","last_undegraded":"2024-03-06T07:35:45.720567+0000","last_fullsized":"2024-03-06T07:35:45.720567+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:09.791568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722044+0000","last_change":"2024-03-06T07:35:43.384588+0000","last_active":"2024-03-06T07:35:45.722044+0000","last_peered":"2024-03-06T07:35:45.722044+0000","last_clean":"2024-03-06T07:35:45.722044+0000","last_became_active":"2024-03-06T07:35:43.384278+0000","last_became_peered":"2024-03-06T07:35:43.384278+0000","last_unstale":"2024-03-06T07:35:45.722044+0000","last_undegraded":"2024-03-06T07:35:45.722044+0000","last_fullsized":"2024-03-06T07:35:45.722044+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:30.617999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722113+0000","last_change":"2024-03-06T07:35:44.408075+0000","last_active":"2024-03-06T07:35:45.722113+0000","last_peered":"2024-03-06T07:35:45.722113+0000","last_clean":"2024-03-06T07:35:45.722113+0000","last_became_active":"2024-03-06T07:35:44.407778+0000","last_became_peered":"2024-03-06T07:35:44.407778+0000","last_unstale":"2024-03-06T07:35:45.722113+0000","last_undegraded":"2024-03-06T07:35:45.722113+0000","last_fullsized":"2024-03-06T07:35:45.722113+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:33.972464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399395+0000","last_change":"2024-03-06T07:35:43.378158+0000","last_active":"2024-03-06T07:35:45.399395+0000","last_peered":"2024-03-06T07:35:45.399395+0000","last_clean":"2024-03-06T07:35:45.399395+0000","last_became_active":"2024-03-06T07:35:43.377863+0000","last_became_peered":"2024-03-06T07:35:43.377863+0000","last_unstale":"2024-03-06T07:35:45.399395+0000","last_undegraded":"2024-03-06T07:35:45.399395+0000","last_fullsized":"2024-03-06T07:35:45.399395+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:27.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720318+0000","last_change":"2024-03-06T07:35:44.419353+0000","last_active":"2024-03-06T07:35:45.720318+0000","last_peered":"2024-03-06T07:35:45.720318+0000","last_clean":"2024-03-06T07:35:45.720318+0000","last_became_active":"2024-03-06T07:35:44.418396+0000","last_became_peered":"2024-03-06T07:35:44.418396+0000","last_unstale":"2024-03-06T07:35:45.720318+0000","last_undegraded":"2024-03-06T07:35:45.720318+0000","last_fullsized":"2024-03-06T07:35:45.720318+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:12.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.558056+0000","last_change":"2024-03-06T07:35:43.375835+0000","last_active":"2024-03-06T07:35:45.558056+0000","last_peered":"2024-03-06T07:35:45.558056+0000","last_clean":"2024-03-06T07:35:45.558056+0000","last_became_active":"2024-03-06T07:35:43.375546+0000","last_became_peered":"2024-03-06T07:35:43.375546+0000","last_unstale":"2024-03-06T07:35:45.558056+0000","last_undegraded":"2024-03-06T07:35:45.558056+0000","last_fullsized":"2024-03-06T07:35:45.558056+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:44.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721668+0000","last_change":"2024-03-06T07:35:44.411183+0000","last_active":"2024-03-06T07:35:45.721668+0000","last_peered":"2024-03-06T07:35:45.721668+0000","last_clean":"2024-03-06T07:35:45.721668+0000","last_became_active":"2024-03-06T07:35:44.410909+0000","last_became_peered":"2024-03-06T07:35:44.410909+0000","last_unstale":"2024-03-06T07:35:45.721668+0000","last_undegraded":"2024-03-06T07:35:45.721668+0000","last_fullsized":"2024-03-06T07:35:45.721668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:38.776886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394928+0000","last_change":"2024-03-06T07:35:43.368831+0000","last_active":"2024-03-06T07:35:45.394928+0000","last_peered":"2024-03-06T07:35:45.394928+0000","last_clean":"2024-03-06T07:35:45.394928+0000","last_became_active":"2024-03-06T07:35:43.368516+0000","last_became_peered":"2024-03-06T07:35:43.368516+0000","last_unstale":"2024-03-06T07:35:45.394928+0000","last_undegraded":"2024-03-06T07:35:45.394928+0000","last_fullsized":"2024-03-06T07:35:45.394928+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:24.992428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719450+0000","last_change":"2024-03-06T07:35:44.384339+0000","last_active":"2024-03-06T07:35:45.719450+0000","last_peered":"2024-03-06T07:35:45.719450+0000","last_clean":"2024-03-06T07:35:45.719450+0000","last_became_active":"2024-03-06T07:35:44.384031+0000","last_became_peered":"2024-03-06T07:35:44.384031+0000","last_unstale":"2024-03-06T07:35:45.719450+0000","last_undegraded":"2024-03-06T07:35:45.719450+0000","last_fullsized":"2024-03-06T07:35:45.719450+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394361+0000","last_change":"2024-03-06T07:35:43.373948+0000","last_active":"2024-03-06T07:35:45.394361+0000","last_peered":"2024-03-06T07:35:45.394361+0000","last_clean":"2024-03-06T07:35:45.394361+0000","last_became_active":"2024-03-06T07:35:43.373631+0000","last_became_peered":"2024-03-06T07:35:43.373631+0000","last_unstale":"2024-03-06T07:35:45.394361+0000","last_undegraded":"2024-03-06T07:35:45.394361+0000","last_fullsized":"2024-03-06T07:35:45.394361+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:51.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400827+0000","last_change":"2024-03-06T07:35:44.400502+0000","last_active":"2024-03-06T07:35:45.400827+0000","last_peered":"2024-03-06T07:35:45.400827+0000","last_clean":"2024-03-06T07:35:45.400827+0000","last_became_active":"2024-03-06T07:35:44.400212+0000","last_became_peered":"2024-03-06T07:35:44.400212+0000","last_unstale":"2024-03-06T07:35:45.400827+0000","last_undegraded":"2024-03-06T07:35:45.400827+0000","last_fullsized":"2024-03-06T07:35:45.400827+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:22.680709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719623+0000","last_change":"2024-03-06T07:35:43.370282+0000","last_active":"2024-03-06T07:35:45.719623+0000","last_peered":"2024-03-06T07:35:45.719623+0000","last_clean":"2024-03-06T07:35:45.719623+0000","last_became_active":"2024-03-06T07:35:43.369383+0000","last_became_peered":"2024-03-06T07:35:43.369383+0000","last_unstale":"2024-03-06T07:35:45.719623+0000","last_undegraded":"2024-03-06T07:35:45.719623+0000","last_fullsized":"2024-03-06T07:35:45.719623+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:02.163748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720319+0000","last_change":"2024-03-06T07:35:44.405369+0000","last_active":"2024-03-06T07:35:45.720319+0000","last_peered":"2024-03-06T07:35:45.720319+0000","last_clean":"2024-03-06T07:35:45.720319+0000","last_became_active":"2024-03-06T07:35:44.405046+0000","last_became_peered":"2024-03-06T07:35:44.405046+0000","last_unstale":"2024-03-06T07:35:45.720319+0000","last_undegraded":"2024-03-06T07:35:45.720319+0000","last_fullsized":"2024-03-06T07:35:45.720319+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:27.612553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399915+0000","last_change":"2024-03-06T07:35:43.380112+0000","last_active":"2024-03-06T07:35:45.399915+0000","last_peered":"2024-03-06T07:35:45.399915+0000","last_clean":"2024-03-06T07:35:45.399915+0000","last_became_active":"2024-03-06T07:35:43.378931+0000","last_became_peered":"2024-03-06T07:35:43.378931+0000","last_unstale":"2024-03-06T07:35:45.399915+0000","last_undegraded":"2024-03-06T07:35:45.399915+0000","last_fullsized":"2024-03-06T07:35:45.399915+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:43.684013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394619+0000","last_change":"2024-03-06T07:35:44.382654+0000","last_active":"2024-03-06T07:35:45.394619+0000","last_peered":"2024-03-06T07:35:45.394619+0000","last_clean":"2024-03-06T07:35:45.394619+0000","last_became_active":"2024-03-06T07:35:44.382389+0000","last_became_peered":"2024-03-06T07:35:44.382389+0000","last_unstale":"2024-03-06T07:35:45.394619+0000","last_undegraded":"2024-03-06T07:35:45.394619+0000","last_fullsized":"2024-03-06T07:35:45.394619+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:48.807797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394363+0000","last_change":"2024-03-06T07:35:43.376342+0000","last_active":"2024-03-06T07:35:45.394363+0000","last_peered":"2024-03-06T07:35:45.394363+0000","last_clean":"2024-03-06T07:35:45.394363+0000","last_became_active":"2024-03-06T07:35:43.374474+0000","last_became_peered":"2024-03-06T07:35:43.374474+0000","last_unstale":"2024-03-06T07:35:45.394363+0000","last_undegraded":"2024-03-06T07:35:45.394363+0000","last_fullsized":"2024-03-06T07:35:45.394363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:03.503542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720355+0000","last_change":"2024-03-06T07:35:44.402234+0000","last_active":"2024-03-06T07:35:45.720355+0000","last_peered":"2024-03-06T07:35:45.720355+0000","last_clean":"2024-03-06T07:35:45.720355+0000","last_became_active":"2024-03-06T07:35:44.401505+0000","last_became_peered":"2024-03-06T07:35:44.401505+0000","last_unstale":"2024-03-06T07:35:45.720355+0000","last_undegraded":"2024-03-06T07:35:45.720355+0000","last_fullsized":"2024-03-06T07:35:45.720355+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:22.846996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401654+0000","last_change":"2024-03-06T07:35:43.384674+0000","last_active":"2024-03-06T07:35:45.401654+0000","last_peered":"2024-03-06T07:35:45.401654+0000","last_clean":"2024-03-06T07:35:45.401654+0000","last_became_active":"2024-03-06T07:35:43.384373+0000","last_became_peered":"2024-03-06T07:35:43.384373+0000","last_unstale":"2024-03-06T07:35:45.401654+0000","last_undegraded":"2024-03-06T07:35:45.401654+0000","last_fullsized":"2024-03-06T07:35:45.401654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:47.747507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.400796+0000","last_change":"2024-03-06T07:35:44.400940+0000","last_active":"2024-03-06T07:35:45.400796+0000","last_peered":"2024-03-06T07:35:45.400796+0000","last_clean":"2024-03-06T07:35:45.400796+0000","last_became_active":"2024-03-06T07:35:44.400650+0000","last_became_peered":"2024-03-06T07:35:44.400650+0000","last_unstale":"2024-03-06T07:35:45.400796+0000","last_undegraded":"2024-03-06T07:35:45.400796+0000","last_fullsized":"2024-03-06T07:35:45.400796+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720656+0000","last_change":"2024-03-06T07:35:43.381659+0000","last_active":"2024-03-06T07:35:45.720656+0000","last_peered":"2024-03-06T07:35:45.720656+0000","last_clean":"2024-03-06T07:35:45.720656+0000","last_became_active":"2024-03-06T07:35:43.381344+0000","last_became_peered":"2024-03-06T07:35:43.381344+0000","last_unstale":"2024-03-06T07:35:45.720656+0000","last_undegraded":"2024-03-06T07:35:45.720656+0000","last_fullsized":"2024-03-06T07:35:45.720656+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:48.989771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720762+0000","last_change":"2024-03-06T07:35:44.420113+0000","last_active":"2024-03-06T07:35:45.720762+0000","last_peered":"2024-03-06T07:35:45.720762+0000","last_clean":"2024-03-06T07:35:45.720762+0000","last_became_active":"2024-03-06T07:35:44.418765+0000","last_became_peered":"2024-03-06T07:35:44.418765+0000","last_unstale":"2024-03-06T07:35:45.720762+0000","last_undegraded":"2024-03-06T07:35:45.720762+0000","last_fullsized":"2024-03-06T07:35:45.720762+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:02.799693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"129'3","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398749+0000","last_change":"2024-03-06T07:35:43.378206+0000","last_active":"2024-03-06T07:35:45.398749+0000","last_peered":"2024-03-06T07:35:45.398749+0000","last_clean":"2024-03-06T07:35:45.398749+0000","last_became_active":"2024-03-06T07:35:43.377916+0000","last_became_peered":"2024-03-06T07:35:43.377916+0000","last_unstale":"2024-03-06T07:35:45.398749+0000","last_undegraded":"2024-03-06T07:35:45.398749+0000","last_fullsized":"2024-03-06T07:35:45.398749+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:21.831021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.722052+0000","last_change":"2024-03-06T07:35:44.416806+0000","last_active":"2024-03-06T07:35:45.722052+0000","last_peered":"2024-03-06T07:35:45.722052+0000","last_clean":"2024-03-06T07:35:45.722052+0000","last_became_active":"2024-03-06T07:35:44.416506+0000","last_became_peered":"2024-03-06T07:35:44.416506+0000","last_unstale":"2024-03-06T07:35:45.722052+0000","last_undegraded":"2024-03-06T07:35:45.722052+0000","last_fullsized":"2024-03-06T07:35:45.722052+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.025560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"129'6","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721134+0000","last_change":"2024-03-06T07:35:43.379952+0000","last_active":"2024-03-06T07:35:45.721134+0000","last_peered":"2024-03-06T07:35:45.721134+0000","last_clean":"2024-03-06T07:35:45.721134+0000","last_became_active":"2024-03-06T07:35:43.379541+0000","last_became_peered":"2024-03-06T07:35:43.379541+0000","last_unstale":"2024-03-06T07:35:45.721134+0000","last_undegraded":"2024-03-06T07:35:45.721134+0000","last_fullsized":"2024-03-06T07:35:45.721134+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.626844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719234+0000","last_change":"2024-03-06T07:35:44.400442+0000","last_active":"2024-03-06T07:35:45.719234+0000","last_peered":"2024-03-06T07:35:45.719234+0000","last_clean":"2024-03-06T07:35:45.719234+0000","last_became_active":"2024-03-06T07:35:44.399865+0000","last_became_peered":"2024-03-06T07:35:44.399865+0000","last_unstale":"2024-03-06T07:35:45.719234+0000","last_undegraded":"2024-03-06T07:35:45.719234+0000","last_fullsized":"2024-03-06T07:35:45.719234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:26.662064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719793+0000","last_change":"2024-03-06T07:35:43.372351+0000","last_active":"2024-03-06T07:35:45.719793+0000","last_peered":"2024-03-06T07:35:45.719793+0000","last_clean":"2024-03-06T07:35:45.719793+0000","last_became_active":"2024-03-06T07:35:43.372020+0000","last_became_peered":"2024-03-06T07:35:43.372020+0000","last_unstale":"2024-03-06T07:35:45.719793+0000","last_undegraded":"2024-03-06T07:35:45.719793+0000","last_fullsized":"2024-03-06T07:35:45.719793+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:11.361058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394255+0000","last_change":"2024-03-06T07:35:44.412709+0000","last_active":"2024-03-06T07:35:45.394255+0000","last_peered":"2024-03-06T07:35:45.394255+0000","last_clean":"2024-03-06T07:35:45.394255+0000","last_became_active":"2024-03-06T07:35:44.411842+0000","last_became_peered":"2024-03-06T07:35:44.411842+0000","last_unstale":"2024-03-06T07:35:45.394255+0000","last_undegraded":"2024-03-06T07:35:45.394255+0000","last_fullsized":"2024-03-06T07:35:45.394255+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:04.234723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394759+0000","last_change":"2024-03-06T07:35:43.380748+0000","last_active":"2024-03-06T07:35:45.394759+0000","last_peered":"2024-03-06T07:35:45.394759+0000","last_clean":"2024-03-06T07:35:45.394759+0000","last_became_active":"2024-03-06T07:35:43.380512+0000","last_became_peered":"2024-03-06T07:35:43.380512+0000","last_unstale":"2024-03-06T07:35:45.394759+0000","last_undegraded":"2024-03-06T07:35:45.394759+0000","last_fullsized":"2024-03-06T07:35:45.394759+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:02.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720212+0000","last_change":"2024-03-06T07:35:44.416049+0000","last_active":"2024-03-06T07:35:45.720212+0000","last_peered":"2024-03-06T07:35:45.720212+0000","last_clean":"2024-03-06T07:35:45.720212+0000","last_became_active":"2024-03-06T07:35:44.415779+0000","last_became_peered":"2024-03-06T07:35:44.415779+0000","last_unstale":"2024-03-06T07:35:45.720212+0000","last_undegraded":"2024-03-06T07:35:45.720212+0000","last_fullsized":"2024-03-06T07:35:45.720212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:42.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":22,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720702+0000","last_change":"2024-03-06T07:35:43.375271+0000","last_active":"2024-03-06T07:35:45.720702+0000","last_peered":"2024-03-06T07:35:45.720702+0000","last_clean":"2024-03-06T07:35:45.720702+0000","last_became_active":"2024-03-06T07:35:43.374378+0000","last_became_peered":"2024-03-06T07:35:43.374378+0000","last_unstale":"2024-03-06T07:35:45.720702+0000","last_undegraded":"2024-03-06T07:35:45.720702+0000","last_fullsized":"2024-03-06T07:35:45.720702+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:47.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721573+0000","last_change":"2024-03-06T07:35:44.403410+0000","last_active":"2024-03-06T07:35:45.721573+0000","last_peered":"2024-03-06T07:35:45.721573+0000","last_clean":"2024-03-06T07:35:45.721573+0000","last_became_active":"2024-03-06T07:35:44.402014+0000","last_became_peered":"2024-03-06T07:35:44.402014+0000","last_unstale":"2024-03-06T07:35:45.721573+0000","last_undegraded":"2024-03-06T07:35:45.721573+0000","last_fullsized":"2024-03-06T07:35:45.721573+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:50.005746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401803+0000","last_change":"2024-03-06T07:35:43.381126+0000","last_active":"2024-03-06T07:35:45.401803+0000","last_peered":"2024-03-06T07:35:45.401803+0000","last_clean":"2024-03-06T07:35:45.401803+0000","last_became_active":"2024-03-06T07:35:43.380688+0000","last_became_peered":"2024-03-06T07:35:43.380688+0000","last_unstale":"2024-03-06T07:35:45.401803+0000","last_undegraded":"2024-03-06T07:35:45.401803+0000","last_fullsized":"2024-03-06T07:35:45.401803+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:31.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401213+0000","last_change":"2024-03-06T07:35:44.412003+0000","last_active":"2024-03-06T07:35:45.401213+0000","last_peered":"2024-03-06T07:35:45.401213+0000","last_clean":"2024-03-06T07:35:45.401213+0000","last_became_active":"2024-03-06T07:35:44.409915+0000","last_became_peered":"2024-03-06T07:35:44.409915+0000","last_unstale":"2024-03-06T07:35:45.401213+0000","last_undegraded":"2024-03-06T07:35:45.401213+0000","last_fullsized":"2024-03-06T07:35:45.401213+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557877+0000","last_change":"2024-03-06T07:35:43.382676+0000","last_active":"2024-03-06T07:35:45.557877+0000","last_peered":"2024-03-06T07:35:45.557877+0000","last_clean":"2024-03-06T07:35:45.557877+0000","last_became_active":"2024-03-06T07:35:43.380914+0000","last_became_peered":"2024-03-06T07:35:43.380914+0000","last_unstale":"2024-03-06T07:35:45.557877+0000","last_undegraded":"2024-03-06T07:35:45.557877+0000","last_fullsized":"2024-03-06T07:35:45.557877+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.893669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720870+0000","last_change":"2024-03-06T07:35:44.400453+0000","last_active":"2024-03-06T07:35:45.720870+0000","last_peered":"2024-03-06T07:35:45.720870+0000","last_clean":"2024-03-06T07:35:45.720870+0000","last_became_active":"2024-03-06T07:35:44.399501+0000","last_became_peered":"2024-03-06T07:35:44.399501+0000","last_unstale":"2024-03-06T07:35:45.720870+0000","last_undegraded":"2024-03-06T07:35:45.720870+0000","last_fullsized":"2024-03-06T07:35:45.720870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:06.081717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.402401+0000","last_change":"2024-03-06T07:35:43.375643+0000","last_active":"2024-03-06T07:35:45.402401+0000","last_peered":"2024-03-06T07:35:45.402401+0000","last_clean":"2024-03-06T07:35:45.402401+0000","last_became_active":"2024-03-06T07:35:43.375368+0000","last_became_peered":"2024-03-06T07:35:43.375368+0000","last_unstale":"2024-03-06T07:35:45.402401+0000","last_undegraded":"2024-03-06T07:35:45.402401+0000","last_fullsized":"2024-03-06T07:35:45.402401+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:43.473084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394708+0000","last_change":"2024-03-06T07:35:44.387752+0000","last_active":"2024-03-06T07:35:45.394708+0000","last_peered":"2024-03-06T07:35:45.394708+0000","last_clean":"2024-03-06T07:35:45.394708+0000","last_became_active":"2024-03-06T07:35:44.386625+0000","last_became_peered":"2024-03-06T07:35:44.386625+0000","last_unstale":"2024-03-06T07:35:45.394708+0000","last_undegraded":"2024-03-06T07:35:45.394708+0000","last_fullsized":"2024-03-06T07:35:45.394708+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.316773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.398845+0000","last_change":"2024-03-06T07:35:43.378109+0000","last_active":"2024-03-06T07:35:45.398845+0000","last_peered":"2024-03-06T07:35:45.398845+0000","last_clean":"2024-03-06T07:35:45.398845+0000","last_became_active":"2024-03-06T07:35:43.377689+0000","last_became_peered":"2024-03-06T07:35:43.377689+0000","last_unstale":"2024-03-06T07:35:45.398845+0000","last_undegraded":"2024-03-06T07:35:45.398845+0000","last_fullsized":"2024-03-06T07:35:45.398845+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:17.604500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719574+0000","last_change":"2024-03-06T07:35:44.406714+0000","last_active":"2024-03-06T07:35:45.719574+0000","last_peered":"2024-03-06T07:35:45.719574+0000","last_clean":"2024-03-06T07:35:45.719574+0000","last_became_active":"2024-03-06T07:35:44.406421+0000","last_became_peered":"2024-03-06T07:35:44.406421+0000","last_unstale":"2024-03-06T07:35:45.719574+0000","last_undegraded":"2024-03-06T07:35:45.719574+0000","last_fullsized":"2024-03-06T07:35:45.719574+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:49.699470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399284+0000","last_change":"2024-03-06T07:35:43.385621+0000","last_active":"2024-03-06T07:35:45.399284+0000","last_peered":"2024-03-06T07:35:45.399284+0000","last_clean":"2024-03-06T07:35:45.399284+0000","last_became_active":"2024-03-06T07:35:43.383393+0000","last_became_peered":"2024-03-06T07:35:43.383393+0000","last_unstale":"2024-03-06T07:35:45.399284+0000","last_undegraded":"2024-03-06T07:35:45.399284+0000","last_fullsized":"2024-03-06T07:35:45.399284+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:45.593408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401245+0000","last_change":"2024-03-06T07:35:44.406710+0000","last_active":"2024-03-06T07:35:45.401245+0000","last_peered":"2024-03-06T07:35:45.401245+0000","last_clean":"2024-03-06T07:35:45.401245+0000","last_became_active":"2024-03-06T07:35:44.406086+0000","last_became_peered":"2024-03-06T07:35:44.406086+0000","last_unstale":"2024-03-06T07:35:45.401245+0000","last_undegraded":"2024-03-06T07:35:45.401245+0000","last_fullsized":"2024-03-06T07:35:45.401245+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:52.910160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719668+0000","last_change":"2024-03-06T07:35:43.381958+0000","last_active":"2024-03-06T07:35:45.719668+0000","last_peered":"2024-03-06T07:35:45.719668+0000","last_clean":"2024-03-06T07:35:45.719668+0000","last_became_active":"2024-03-06T07:35:43.381369+0000","last_became_peered":"2024-03-06T07:35:43.381369+0000","last_unstale":"2024-03-06T07:35:45.719668+0000","last_undegraded":"2024-03-06T07:35:45.719668+0000","last_fullsized":"2024-03-06T07:35:45.719668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:29:26.882853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394343+0000","last_change":"2024-03-06T07:35:44.386726+0000","last_active":"2024-03-06T07:35:45.394343+0000","last_peered":"2024-03-06T07:35:45.394343+0000","last_clean":"2024-03-06T07:35:45.394343+0000","last_became_active":"2024-03-06T07:35:44.385066+0000","last_became_peered":"2024-03-06T07:35:44.385066+0000","last_unstale":"2024-03-06T07:35:45.394343+0000","last_undegraded":"2024-03-06T07:35:45.394343+0000","last_fullsized":"2024-03-06T07:35:45.394343+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:29.921786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719327+0000","last_change":"2024-03-06T07:35:43.381967+0000","last_active":"2024-03-06T07:35:45.719327+0000","last_peered":"2024-03-06T07:35:45.719327+0000","last_clean":"2024-03-06T07:35:45.719327+0000","last_became_active":"2024-03-06T07:35:43.381593+0000","last_became_peered":"2024-03-06T07:35:43.381593+0000","last_unstale":"2024-03-06T07:35:45.719327+0000","last_undegraded":"2024-03-06T07:35:45.719327+0000","last_fullsized":"2024-03-06T07:35:45.719327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:32.956490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719534+0000","last_change":"2024-03-06T07:35:44.416923+0000","last_active":"2024-03-06T07:35:45.719534+0000","last_peered":"2024-03-06T07:35:45.719534+0000","last_clean":"2024-03-06T07:35:45.719534+0000","last_became_active":"2024-03-06T07:35:44.414762+0000","last_became_peered":"2024-03-06T07:35:44.414762+0000","last_unstale":"2024-03-06T07:35:45.719534+0000","last_undegraded":"2024-03-06T07:35:45.719534+0000","last_fullsized":"2024-03-06T07:35:45.719534+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:18.903803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557365+0000","last_change":"2024-03-06T07:35:43.378687+0000","last_active":"2024-03-06T07:35:45.557365+0000","last_peered":"2024-03-06T07:35:45.557365+0000","last_clean":"2024-03-06T07:35:45.557365+0000","last_became_active":"2024-03-06T07:35:43.378095+0000","last_became_peered":"2024-03-06T07:35:43.378095+0000","last_unstale":"2024-03-06T07:35:45.557365+0000","last_undegraded":"2024-03-06T07:35:45.557365+0000","last_fullsized":"2024-03-06T07:35:45.557365+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:11.480478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.401813+0000","last_change":"2024-03-06T07:35:44.411248+0000","last_active":"2024-03-06T07:35:45.401813+0000","last_peered":"2024-03-06T07:35:45.401813+0000","last_clean":"2024-03-06T07:35:45.401813+0000","last_became_active":"2024-03-06T07:35:44.410670+0000","last_became_peered":"2024-03-06T07:35:44.410670+0000","last_unstale":"2024-03-06T07:35:45.401813+0000","last_undegraded":"2024-03-06T07:35:45.401813+0000","last_fullsized":"2024-03-06T07:35:45.401813+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:13.430456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720668+0000","last_change":"2024-03-06T07:35:43.377991+0000","last_active":"2024-03-06T07:35:45.720668+0000","last_peered":"2024-03-06T07:35:45.720668+0000","last_clean":"2024-03-06T07:35:45.720668+0000","last_became_active":"2024-03-06T07:35:43.377732+0000","last_became_peered":"2024-03-06T07:35:43.377732+0000","last_unstale":"2024-03-06T07:35:45.720668+0000","last_undegraded":"2024-03-06T07:35:45.720668+0000","last_fullsized":"2024-03-06T07:35:45.720668+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.997519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720542+0000","last_change":"2024-03-06T07:35:44.415821+0000","last_active":"2024-03-06T07:35:45.720542+0000","last_peered":"2024-03-06T07:35:45.720542+0000","last_clean":"2024-03-06T07:35:45.720542+0000","last_became_active":"2024-03-06T07:35:44.415487+0000","last_became_peered":"2024-03-06T07:35:44.415487+0000","last_unstale":"2024-03-06T07:35:45.720542+0000","last_undegraded":"2024-03-06T07:35:45.720542+0000","last_fullsized":"2024-03-06T07:35:45.720542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:46.609382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719944+0000","last_change":"2024-03-06T07:35:43.385136+0000","last_active":"2024-03-06T07:35:45.719944+0000","last_peered":"2024-03-06T07:35:45.719944+0000","last_clean":"2024-03-06T07:35:45.719944+0000","last_became_active":"2024-03-06T07:35:43.384757+0000","last_became_peered":"2024-03-06T07:35:43.384757+0000","last_unstale":"2024-03-06T07:35:45.719944+0000","last_undegraded":"2024-03-06T07:35:45.719944+0000","last_fullsized":"2024-03-06T07:35:45.719944+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:20.137806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394855+0000","last_change":"2024-03-06T07:35:44.417187+0000","last_active":"2024-03-06T07:35:45.394855+0000","last_peered":"2024-03-06T07:35:45.394855+0000","last_clean":"2024-03-06T07:35:45.394855+0000","last_became_active":"2024-03-06T07:35:44.416892+0000","last_became_peered":"2024-03-06T07:35:44.416892+0000","last_unstale":"2024-03-06T07:35:45.394855+0000","last_undegraded":"2024-03-06T07:35:45.394855+0000","last_fullsized":"2024-03-06T07:35:45.394855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:00.097915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394798+0000","last_change":"2024-03-06T07:35:43.377777+0000","last_active":"2024-03-06T07:35:45.394798+0000","last_peered":"2024-03-06T07:35:45.394798+0000","last_clean":"2024-03-06T07:35:45.394798+0000","last_became_active":"2024-03-06T07:35:43.374868+0000","last_became_peered":"2024-03-06T07:35:43.374868+0000","last_unstale":"2024-03-06T07:35:45.394798+0000","last_undegraded":"2024-03-06T07:35:45.394798+0000","last_fullsized":"2024-03-06T07:35:45.394798+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:57.102370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.402497+0000","last_change":"2024-03-06T07:35:44.406506+0000","last_active":"2024-03-06T07:35:45.402497+0000","last_peered":"2024-03-06T07:35:45.402497+0000","last_clean":"2024-03-06T07:35:45.402497+0000","last_became_active":"2024-03-06T07:35:44.406107+0000","last_became_peered":"2024-03-06T07:35:44.406107+0000","last_unstale":"2024-03-06T07:35:45.402497+0000","last_undegraded":"2024-03-06T07:35:45.402497+0000","last_fullsized":"2024-03-06T07:35:45.402497+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:50.460328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557609+0000","last_change":"2024-03-06T07:35:43.368644+0000","last_active":"2024-03-06T07:35:45.557609+0000","last_peered":"2024-03-06T07:35:45.557609+0000","last_clean":"2024-03-06T07:35:45.557609+0000","last_became_active":"2024-03-06T07:35:43.367936+0000","last_became_peered":"2024-03-06T07:35:43.367936+0000","last_unstale":"2024-03-06T07:35:45.557609+0000","last_undegraded":"2024-03-06T07:35:45.557609+0000","last_fullsized":"2024-03-06T07:35:45.557609+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:01.783718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720731+0000","last_change":"2024-03-06T07:35:44.417357+0000","last_active":"2024-03-06T07:35:45.720731+0000","last_peered":"2024-03-06T07:35:45.720731+0000","last_clean":"2024-03-06T07:35:45.720731+0000","last_became_active":"2024-03-06T07:35:44.417021+0000","last_became_peered":"2024-03-06T07:35:44.417021+0000","last_unstale":"2024-03-06T07:35:45.720731+0000","last_undegraded":"2024-03-06T07:35:45.720731+0000","last_fullsized":"2024-03-06T07:35:45.720731+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:11.909643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394493+0000","last_change":"2024-03-06T07:35:43.376770+0000","last_active":"2024-03-06T07:35:45.394493+0000","last_peered":"2024-03-06T07:35:45.394493+0000","last_clean":"2024-03-06T07:35:45.394493+0000","last_became_active":"2024-03-06T07:35:43.376483+0000","last_became_peered":"2024-03-06T07:35:43.376483+0000","last_unstale":"2024-03-06T07:35:45.394493+0000","last_undegraded":"2024-03-06T07:35:45.394493+0000","last_fullsized":"2024-03-06T07:35:45.394493+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:32.527625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.399036+0000","last_change":"2024-03-06T07:35:44.408484+0000","last_active":"2024-03-06T07:35:45.399036+0000","last_peered":"2024-03-06T07:35:45.399036+0000","last_clean":"2024-03-06T07:35:45.399036+0000","last_became_active":"2024-03-06T07:35:44.407992+0000","last_became_peered":"2024-03-06T07:35:44.407992+0000","last_unstale":"2024-03-06T07:35:45.399036+0000","last_undegraded":"2024-03-06T07:35:45.399036+0000","last_fullsized":"2024-03-06T07:35:45.399036+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720795+0000","last_change":"2024-03-06T07:35:43.385517+0000","last_active":"2024-03-06T07:35:45.720795+0000","last_peered":"2024-03-06T07:35:45.720795+0000","last_clean":"2024-03-06T07:35:45.720795+0000","last_became_active":"2024-03-06T07:35:43.384210+0000","last_became_peered":"2024-03-06T07:35:43.384210+0000","last_unstale":"2024-03-06T07:35:45.720795+0000","last_undegraded":"2024-03-06T07:35:45.720795+0000","last_fullsized":"2024-03-06T07:35:45.720795+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:11.817793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719634+0000","last_change":"2024-03-06T07:35:44.412773+0000","last_active":"2024-03-06T07:35:45.719634+0000","last_peered":"2024-03-06T07:35:45.719634+0000","last_clean":"2024-03-06T07:35:45.719634+0000","last_became_active":"2024-03-06T07:35:44.411292+0000","last_became_peered":"2024-03-06T07:35:44.411292+0000","last_unstale":"2024-03-06T07:35:45.719634+0000","last_undegraded":"2024-03-06T07:35:45.719634+0000","last_fullsized":"2024-03-06T07:35:45.719634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:27.529323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719424+0000","last_change":"2024-03-06T07:35:43.379581+0000","last_active":"2024-03-06T07:35:45.719424+0000","last_peered":"2024-03-06T07:35:45.719424+0000","last_clean":"2024-03-06T07:35:45.719424+0000","last_became_active":"2024-03-06T07:35:43.379237+0000","last_became_peered":"2024-03-06T07:35:43.379237+0000","last_unstale":"2024-03-06T07:35:45.719424+0000","last_undegraded":"2024-03-06T07:35:45.719424+0000","last_fullsized":"2024-03-06T07:35:45.719424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:32.312256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.394668+0000","last_change":"2024-03-06T07:35:44.417512+0000","last_active":"2024-03-06T07:35:45.394668+0000","last_peered":"2024-03-06T07:35:45.394668+0000","last_clean":"2024-03-06T07:35:45.394668+0000","last_became_active":"2024-03-06T07:35:44.416206+0000","last_became_peered":"2024-03-06T07:35:44.416206+0000","last_unstale":"2024-03-06T07:35:45.394668+0000","last_undegraded":"2024-03-06T07:35:45.394668+0000","last_fullsized":"2024-03-06T07:35:45.394668+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.959259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.557543+0000","last_change":"2024-03-06T07:35:43.375969+0000","last_active":"2024-03-06T07:35:45.557543+0000","last_peered":"2024-03-06T07:35:45.557543+0000","last_clean":"2024-03-06T07:35:45.557543+0000","last_became_active":"2024-03-06T07:35:43.375710+0000","last_became_peered":"2024-03-06T07:35:43.375710+0000","last_unstale":"2024-03-06T07:35:45.557543+0000","last_undegraded":"2024-03-06T07:35:45.557543+0000","last_fullsized":"2024-03-06T07:35:45.557543+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:38.782993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.720514+0000","last_change":"2024-03-06T07:35:44.414475+0000","last_active":"2024-03-06T07:35:45.720514+0000","last_peered":"2024-03-06T07:35:45.720514+0000","last_clean":"2024-03-06T07:35:45.720514+0000","last_became_active":"2024-03-06T07:35:44.414166+0000","last_became_peered":"2024-03-06T07:35:44.414166+0000","last_unstale":"2024-03-06T07:35:45.720514+0000","last_undegraded":"2024-03-06T07:35:45.720514+0000","last_fullsized":"2024-03-06T07:35:45.720514+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.339559+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.339559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:39.798967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.721268+0000","last_change":"2024-03-06T07:35:43.385255+0000","last_active":"2024-03-06T07:35:45.721268+0000","last_peered":"2024-03-06T07:35:45.721268+0000","last_clean":"2024-03-06T07:35:45.721268+0000","last_became_active":"2024-03-06T07:35:43.384945+0000","last_became_peered":"2024-03-06T07:35:43.384945+0000","last_unstale":"2024-03-06T07:35:45.721268+0000","last_undegraded":"2024-03-06T07:35:45.721268+0000","last_fullsized":"2024-03-06T07:35:45.721268+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:42.323585+0000","last_clean_scrub_stamp":"2024-03-06T07:35:42.323585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:41.317784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":369,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-06T07:35:45.719242+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:35:45.719242+0000","last_peered":"2024-03-06T07:35:45.719242+0000","last_clean":"2024-03-06T07:35:45.719242+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:35:45.719242+0000","last_undegraded":"2024-03-06T07:35:45.719242+0000","last_fullsized":"2024-03-06T07:35:45.719242+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2236416,"data_stored":1296790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542246,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1630208,"data_stored":705784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542246,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41092,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330748,"statfs":{"total":96636764160,"available":96594685952,"internally_reserved":0,"allocated":1622016,"data_stored":704242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542245,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1626112,"data_stored":704332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542247,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":1626112,"data_stored":704788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36928,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334912,"statfs":{"total":96636764160,"available":96598949888,"internally_reserved":0,"allocated":1617920,"data_stored":704172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41660,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330180,"statfs":{"total":96636764160,"available":96594104320,"internally_reserved":0,"allocated":2211840,"data_stored":1294540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:54.392 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:35:54.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:35:54.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.640+0000 7f3d35e00640 1 Processor -- start 2024-03-06T07:35:54.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.640+0000 7f3d35e00640 1 -- start start 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.641+0000 7f3d35e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3d300ce5d0 0x7f3d300cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d35e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d35e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300cf430 con 0x7f3d300ce9b0 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d35e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d300cd670 con 0x7f3d300ce5d0 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d35e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300cfd70 con 0x7f3d3015c100 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3d300ce5d0 0x7f3d300cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3d300ce5d0 0x7f3d300cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39570/0 (socket says 172.21.15.97:39570) 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 -- 172.21.15.97:0/2893108013 learned_addr learned my addr 172.21.15.97:0/2893108013 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d2fe00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 -- 172.21.15.97:0/2893108013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 msgr2=0x7f3d300d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 -- 172.21.15.97:0/2893108013 >> v1:172.21.15.135:6789/0 conn(0x7f3d300ce9b0 legacy=0x7f3d3015f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.642+0000 7f3d34a00640 1 -- 172.21.15.97:0/2893108013 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3015c6f0 con 0x7f3d300ce5d0 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.643+0000 7f3d2fe00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.644+0000 7f3d34a00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce5d0 0x7f3d300cf790 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009870 tx=0x7f3d2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efdc68c11f7149f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.644+0000 7f3d2f400640 1 -- 172.21.15.97:0/2893108013 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24035020 con 0x7f3d300ce5d0 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.644+0000 7f3d2f400640 1 -- 172.21.15.97:0/2893108013 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d24031640 con 0x7f3d300ce5d0 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.644+0000 7f3d2f400640 1 -- 172.21.15.97:0/2893108013 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2402d040 con 0x7f3d300ce5d0 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 -- 172.21.15.97:0/2893108013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce5d0 msgr2=0x7f3d300cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce5d0 0x7f3d300cf790 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009870 tx=0x7f3d2402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 -- 172.21.15.97:0/2893108013 shutdown_connections 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2893108013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce5d0 0x7f3d300cf790 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 -- 172.21.15.97:0/2893108013 >> 172.21.15.97:0/2893108013 conn(0x7f3d300d1bc0 msgr2=0x7f3d300d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 -- 172.21.15.97:0/2893108013 shutdown_connections 2024-03-06T07:35:54.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.645+0000 7f3d35e00640 1 -- 172.21.15.97:0/2893108013 wait complete. 2024-03-06T07:35:54.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.647+0000 7f3d35e00640 1 Processor -- start 2024-03-06T07:35:54.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.647+0000 7f3d35e00640 1 -- start start 2024-03-06T07:35:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d300ce5d0 0x7f3d30155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 0x7f3d300b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300cee40 con 0x7f3d300ce9b0 2024-03-06T07:35:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3015ccb0 con 0x7f3d300ce5d0 2024-03-06T07:35:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3015d1d0 con 0x7f3d3015c100 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 0x7f3d300b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 0x7f3d300b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39582/0 (socket says 172.21.15.97:39582) 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 -- 172.21.15.97:0/2091257265 learned_addr learned my addr 172.21.15.97:0/2091257265 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 -- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 msgr2=0x7f3d300b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35400640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d34a00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d300ce5d0 0x7f3d30155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 -- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d300ce5d0 msgr2=0x7f3d30155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d300ce5d0 0x7f3d30155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d2fe00640 1 -- 172.21.15.97:0/2091257265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30156810 con 0x7f3d300ce9b0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.648+0000 7f3d35400640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.649+0000 7f3d2fe00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 0x7f3d300b6b10 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00c7f0 tx=0x7f3d1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.649+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c015070 con 0x7f3d300ce9b0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.649+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c007820 con 0x7f3d300ce9b0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.649+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c007bd0 con 0x7f3d300ce9b0 2024-03-06T07:35:54.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.649+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d301562b0 con 0x7f3d300ce9b0 2024-03-06T07:35:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.649+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d300b3240 con 0x7f3d300ce9b0 2024-03-06T07:35:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.650+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d30075450 con 0x7f3d300ce9b0 2024-03-06T07:35:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.651+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d1c018070 con 0x7f3d300ce9b0 2024-03-06T07:35:54.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.652+0000 7f3d2cc00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d1006b4d0 0x7f3d1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.652+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d1c08a270 con 0x7f3d300ce9b0 2024-03-06T07:35:54.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.652+0000 7f3d34a00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d1006b4d0 0x7f3d1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.653+0000 7f3d34a00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d1006b4d0 0x7f3d1006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3d24036000 tx=0x7f3d2403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.657+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d30156810 con 0x7f3d300ce9b0 2024-03-06T07:35:54.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.925+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3d30068490 con 0x7f3d300ce9b0 2024-03-06T07:35:54.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.930+0000 7f3d2cc00640 1 -- 172.21.15.97:0/2091257265 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3d1c059180 con 0x7f3d300ce9b0 2024-03-06T07:35:54.932 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:35:54.932 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:35:54.932 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:35:54.932 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:35:54.932 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:54.933 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.932+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d1006b4d0 msgr2=0x7f3d1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.932+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d1006b4d0 0x7f3d1006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3d24036000 tx=0x7f3d2403c040 comp rx=0 tx=0).stop 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.932+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 msgr2=0x7f3d300b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.932+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 0x7f3d300b6b10 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00c7f0 tx=0x7f3d1c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 shutdown_connections 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d1006b4d0 0x7f3d1006d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d3015c100 0x7f3d300b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d300ce9b0 0x7f3d300b6b10 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 --2- 172.21.15.97:0/2091257265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d300ce5d0 0x7f3d30155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 >> 172.21.15.97:0/2091257265 conn(0x7f3d300d1bc0 msgr2=0x7f3d3006db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 shutdown_connections 2024-03-06T07:35:54.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:54.933+0000 7f3d35e00640 1 -- 172.21.15.97:0/2091257265 wait complete. 2024-03-06T07:35:54.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:55.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.173+0000 7f0c51c00640 1 Processor -- start 2024-03-06T07:35:55.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.173+0000 7f0c51c00640 1 -- start start 2024-03-06T07:35:55.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.175+0000 7f0c51c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0c4c15a940 0x7f0c4c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.175+0000 7f0c51c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c0cc2a0 0x7f0c4c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.175+0000 7f0c51c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c0b2b30 con 0x7f0c4c0cf370 2024-03-06T07:35:55.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.175+0000 7f0c51c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4c0b28e0 con 0x7f0c4c15a940 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.175+0000 7f0c51c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0b2e10 con 0x7f0c4c0cc2a0 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0c4c15a940 0x7f0c4c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0c4c15a940 0x7f0c4c0b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39590/0 (socket says 172.21.15.97:39590) 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 -- 172.21.15.97:0/1765128226 learned_addr learned my addr 172.21.15.97:0/1765128226 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c43e00640 1 --1- 172.21.15.97:0/1765128226 >> v1:172.21.15.97:6789/0 conn(0x7f0c4c0cf370 0x7f0c4c06d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42812/0 (socket says 172.21.15.97:42812) 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c51200640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c0cc2a0 0x7f0c4c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 -- 172.21.15.97:0/1765128226 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c0cc2a0 msgr2=0x7f0c4c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c0cc2a0 0x7f0c4c0cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 -- 172.21.15.97:0/1765128226 >> v1:172.21.15.97:6789/0 conn(0x7f0c4c0cf370 legacy=0x7f0c4c06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4be00640 1 -- 172.21.15.97:0/1765128226 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c06ca50 con 0x7f0c4c15a940 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c51200640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c0cc2a0 0x7f0c4c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:55.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.176+0000 7f0c4b400640 1 -- 172.21.15.97:0/1765128226 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638919478 0 0) 0x7f0c4c0b2b30 con 0x7f0c4c0cf370 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.177+0000 7f0c4be00640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c15a940 0x7f0c4c0b45e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00bb10 tx=0x7f0c3c02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc36a3ab9da5d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.177+0000 7f0c4b400640 1 -- 172.21.15.97:0/1765128226 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c03b070 con 0x7f0c4c15a940 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.177+0000 7f0c4b400640 1 -- 172.21.15.97:0/1765128226 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c036440 con 0x7f0c4c15a940 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.177+0000 7f0c4b400640 1 -- 172.21.15.97:0/1765128226 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c036810 con 0x7f0c4c15a940 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 -- 172.21.15.97:0/1765128226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c15a940 msgr2=0x7f0c4c0b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c15a940 0x7f0c4c0b45e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00bb10 tx=0x7f0c3c02e150 comp rx=0 tx=0).stop 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 -- 172.21.15.97:0/1765128226 shutdown_connections 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c0cc2a0 0x7f0c4c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 --2- 172.21.15.97:0/1765128226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c15a940 0x7f0c4c0b45e0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 -- 172.21.15.97:0/1765128226 >> 172.21.15.97:0/1765128226 conn(0x7f0c4c06afc0 msgr2=0x7f0c4c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 -- 172.21.15.97:0/1765128226 shutdown_connections 2024-03-06T07:35:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.178+0000 7f0c51c00640 1 -- 172.21.15.97:0/1765128226 wait complete. 2024-03-06T07:35:55.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.180+0000 7f0c51c00640 1 Processor -- start 2024-03-06T07:35:55.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.180+0000 7f0c51c00640 1 -- start start 2024-03-06T07:35:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c4c0cc2a0 0x7f0c4c1f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 0x7f0c4c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c15a940 0x7f0c4c0cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0b4b20 con 0x7f0c4c0cf370 2024-03-06T07:35:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0b30f0 con 0x7f0c4c0cc2a0 2024-03-06T07:35:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c15ad20 con 0x7f0c4c15a940 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 0x7f0c4c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c43e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 0x7f0c4c158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39592/0 (socket says 172.21.15.97:39592) 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c43e00640 1 -- 172.21.15.97:0/2409110571 learned_addr learned my addr 172.21.15.97:0/2409110571 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c51200640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c15a940 0x7f0c4c0cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c4be00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c4c0cc2a0 0x7f0c4c1f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.181+0000 7f0c43e00640 1 -- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c15a940 msgr2=0x7f0c4c0cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c43e00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c15a940 0x7f0c4c0cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c43e00640 1 -- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c4c0cc2a0 msgr2=0x7f0c4c1f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c43e00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c4c0cc2a0 0x7f0c4c1f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c43e00640 1 -- 172.21.15.97:0/2409110571 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c0aee90 con 0x7f0c4c0cf370 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c43e00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 0x7f0c4c158f00 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c4d0 tx=0x7f0c3400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34018070 con 0x7f0c4c0cf370 2024-03-06T07:35:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34007500 con 0x7f0c4c0cf370 2024-03-06T07:35:55.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34004a20 con 0x7f0c4c0cf370 2024-03-06T07:35:55.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c159850 con 0x7f0c4c0cf370 2024-03-06T07:35:55.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.182+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c0af430 con 0x7f0c4c0cf370 2024-03-06T07:35:55.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.184+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c04f330 con 0x7f0c4c0cf370 2024-03-06T07:35:55.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.185+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c3401b070 con 0x7f0c4c0cf370 2024-03-06T07:35:55.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.186+0000 7f0c48c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0c2806b4d0 0x7f0c2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.186+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c34089420 con 0x7f0c4c0cf370 2024-03-06T07:35:55.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.186+0000 7f0c4be00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0c2806b4d0 0x7f0c2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.187+0000 7f0c4be00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0c2806b4d0 0x7f0c2806d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c038000 tx=0x7f0c3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.191+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c34058330 con 0x7f0c4c0cf370 2024-03-06T07:35:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.430+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c4c06e180 con 0x7f0c4c0cf370 2024-03-06T07:35:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.431+0000 7f0c48c00640 1 -- 172.21.15.97:0/2409110571 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f0c34057c30 con 0x7f0c4c0cf370 2024-03-06T07:35:55.434 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:55.434 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1847010599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1847010599},{"type":"v1","addr":"172.21.15.135:6833","nonce":1847010599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6175,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/3157120524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":3157120524},{"type":"v1","addr":"172.21.15.135:6835","nonce":3157120524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:46.925306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:35:55.434 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 88 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0c2806b4d0 msgr2=0x7f0c2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0c2806b4d0 0x7f0c2806d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c038000 tx=0x7f0c3c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 msgr2=0x7f0c4c158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 0x7f0c4c158f00 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c4d0 tx=0x7f0c3400c990 comp rx=0 tx=0).stop 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 shutdown_connections 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0c2806b4d0 0x7f0c2806d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0c4c15a940 0x7f0c4c0cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4c0cf370 0x7f0c4c158f00 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 --2- 172.21.15.97:0/2409110571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c4c0cc2a0 0x7f0c4c1f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.434+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 >> 172.21.15.97:0/2409110571 conn(0x7f0c4c06afc0 msgr2=0x7f0c4c070990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.435+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 shutdown_connections 2024-03-06T07:35:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:55.435+0000 7f0c51c00640 1 -- 172.21.15.97:0/2409110571 wait complete. 2024-03-06T07:35:55.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:35:46.925306+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8382}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:35:55.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:35:55.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:35:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.068+0000 7fe70b200640 1 Processor -- start 2024-03-06T07:35:56.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.068+0000 7fe70b200640 1 -- start start 2024-03-06T07:35:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe70b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe70415dfa0 0x7fe7040cc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe70b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70415f540 con 0x7fe70415f0f0 2024-03-06T07:35:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe70b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040cd060 con 0x7fe7040ce9b0 2024-03-06T07:35:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe70b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7040cd340 con 0x7fe70415dfa0 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe709e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe70415dfa0 0x7fe7040cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe709400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe70415f0f0 0x7fe7040ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42828/0 (socket says 172.21.15.97:42828) 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.070+0000 7fe709400640 1 -- 172.21.15.97:0/2295226440 learned_addr learned my addr 172.21.15.97:0/2295226440 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.071+0000 7fe709e00640 1 -- 172.21.15.97:0/2295226440 >> v1:172.21.15.135:6789/0 conn(0x7fe7040ce9b0 legacy=0x7fe7040cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.071+0000 7fe709e00640 1 -- 172.21.15.97:0/2295226440 >> v1:172.21.15.97:6789/0 conn(0x7fe70415f0f0 legacy=0x7fe7040ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.071+0000 7fe709e00640 1 -- 172.21.15.97:0/2295226440 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040cd620 con 0x7fe70415dfa0 2024-03-06T07:35:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.071+0000 7fe708a00640 1 -- 172.21.15.97:0/2295226440 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908961815 0 0) 0x7fe70415f540 con 0x7fe70415f0f0 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.072+0000 7fe709e00640 1 --2- 172.21.15.97:0/2295226440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe70415dfa0 0x7fe7040cc980 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80097a0 tx=0x7fe6f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3dda93aebd59ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.072+0000 7fe708a00640 1 -- 172.21.15.97:0/2295226440 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8034070 con 0x7fe70415dfa0 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.072+0000 7fe708a00640 1 -- 172.21.15.97:0/2295226440 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8030610 con 0x7fe70415dfa0 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.072+0000 7fe708a00640 1 -- 172.21.15.97:0/2295226440 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80309e0 con 0x7fe70415dfa0 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.072+0000 7fe70b200640 1 -- 172.21.15.97:0/2295226440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe70415dfa0 msgr2=0x7fe7040cc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.072+0000 7fe70b200640 1 --2- 172.21.15.97:0/2295226440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe70415dfa0 0x7fe7040cc980 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80097a0 tx=0x7fe6f802b960 comp rx=0 tx=0).stop 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.073+0000 7fe70b200640 1 -- 172.21.15.97:0/2295226440 shutdown_connections 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.073+0000 7fe70b200640 1 --2- 172.21.15.97:0/2295226440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe70415dfa0 0x7fe7040cc980 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.073+0000 7fe70b200640 1 -- 172.21.15.97:0/2295226440 >> 172.21.15.97:0/2295226440 conn(0x7fe7040b3620 msgr2=0x7fe7040b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.073+0000 7fe70b200640 1 -- 172.21.15.97:0/2295226440 shutdown_connections 2024-03-06T07:35:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.073+0000 7fe70b200640 1 -- 172.21.15.97:0/2295226440 wait complete. 2024-03-06T07:35:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.074+0000 7fe70b200640 1 Processor -- start 2024-03-06T07:35:56.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.075+0000 7fe70b200640 1 -- start start 2024-03-06T07:35:56.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 0x7fe7040c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe70415dfa0 0x7fe704161570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe70415f0f0 0x7fe704153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cfe00 con 0x7fe7040ce9b0 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cdbe0 con 0x7fe70415f0f0 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cff80 con 0x7fe70415dfa0 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 0x7fe7040c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 0x7fe7040c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39616/0 (socket says 172.21.15.97:39616) 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 -- 172.21.15.97:0/744093633 learned_addr learned my addr 172.21.15.97:0/744093633 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe70a800640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe70415f0f0 0x7fe704153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 -- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe70415dfa0 msgr2=0x7fe704161570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe70415dfa0 0x7fe704161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 -- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe70415f0f0 msgr2=0x7fe704153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe70415f0f0 0x7fe704153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.076+0000 7fe709e00640 1 -- 172.21.15.97:0/744093633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040c9e70 con 0x7fe7040ce9b0 2024-03-06T07:35:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.077+0000 7fe709e00640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 0x7fe7040c9590 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80098c0 tx=0x7fe6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.077+0000 7fe6fe000640 1 -- 172.21.15.97:0/744093633 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8034040 con 0x7fe7040ce9b0 2024-03-06T07:35:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.077+0000 7fe6fe000640 1 -- 172.21.15.97:0/744093633 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8004030 con 0x7fe7040ce9b0 2024-03-06T07:35:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.077+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7040ca150 con 0x7fe7040ce9b0 2024-03-06T07:35:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.077+0000 7fe6fe000640 1 -- 172.21.15.97:0/744093633 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8004420 con 0x7fe7040ce9b0 2024-03-06T07:35:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.077+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7040c9ad0 con 0x7fe7040ce9b0 2024-03-06T07:35:56.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.080+0000 7fe6fe000640 1 -- 172.21.15.97:0/744093633 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6f8037070 con 0x7fe7040ce9b0 2024-03-06T07:35:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.081+0000 7fe70f5a2740 1 Processor -- start 2024-03-06T07:35:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.081+0000 7fe6fe000640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe6e806b4d0 0x7fe6e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.081+0000 7fe70f5a2740 1 -- start start 2024-03-06T07:35:56.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.081+0000 7fe6fe000640 1 -- 172.21.15.97:0/744093633 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6f80abe30 con 0x7fe7040ce9b0 2024-03-06T07:35:56.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.081+0000 7fe70a800640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe6e806b4d0 0x7fe6e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70a800640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe6e806b4d0 0x7fe6e806d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe700007fd0 tx=0x7fe700009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf0a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf3a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf0ae70 0x55e91cf3ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e91ce74600 con 0x55e91cf37d70 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e91ce1d930 con 0x55e91cf37990 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e91ce706a0 con 0x55e91cf0ae70 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.082+0000 7fe709400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf3a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf3a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39622/0 (socket says 172.21.15.97:39622) 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 -- 172.21.15.97:0/3965167497 learned_addr learned my addr 172.21.15.97:0/3965167497 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe70a800640 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf0ae70 0x55e91cf3ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 -- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf0ae70 msgr2=0x55e91cf3ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf0ae70 0x55e91cf3ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 -- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 msgr2=0x55e91cf0a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf0a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 -- 172.21.15.97:0/3965167497 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e91cb892d0 con 0x55e91cf37d70 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe709400640 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf3a720 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4001620 tx=0x7fe6f4001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142d539986e20d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe6f2600640 1 -- 172.21.15.97:0/3965167497 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f40074e0 con 0x55e91cf37d70 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe6f2600640 1 -- 172.21.15.97:0/3965167497 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f4001d30 con 0x55e91cf37d70 2024-03-06T07:35:56.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.083+0000 7fe6f2600640 1 -- 172.21.15.97:0/3965167497 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f40049e0 con 0x55e91cf37d70 2024-03-06T07:35:56.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 -- 172.21.15.97:0/3965167497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 msgr2=0x55e91cf3a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf3a720 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4001620 tx=0x7fe6f4001ae0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 -- 172.21.15.97:0/3965167497 shutdown_connections 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf0ae70 0x55e91cf3ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf3a720 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/3965167497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf0a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 -- 172.21.15.97:0/3965167497 >> 172.21.15.97:0/3965167497 conn(0x55e91cf0cba0 msgr2=0x55e91cf55720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 -- 172.21.15.97:0/3965167497 shutdown_connections 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.084+0000 7fe70f5a2740 1 -- 172.21.15.97:0/3965167497 wait complete. 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.085+0000 7fe70f5a2740 1 -- start start 2024-03-06T07:35:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.086+0000 7fe70f5a2740 1 Processor -- start 2024-03-06T07:35:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.087+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf4cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.087+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf3cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.087+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 0x55e91cf3d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.087+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e91cf0f140 con 0x55e91cf55ee0 2024-03-06T07:35:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.087+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e91ce30d90 con 0x55e91cf37990 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.087+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e91cf0e530 con 0x55e91cf37d70 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70f5a2740 10 client.0 _ll_register_callbacks cb 0x55e91ce6c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 0x55e91cf3d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 0x55e91cf3d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39628/0 (socket says 172.21.15.97:39628) 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 -- 172.21.15.97:0/1149054227 learned_addr learned my addr 172.21.15.97:0/1149054227 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe709e00640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf4cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe709400640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf3cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 -- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 msgr2=0x55e91cf3cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf3cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 -- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 msgr2=0x55e91cf4cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf4cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 -- 172.21.15.97:0/1149054227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e91cc536b0 con 0x55e91cf55ee0 2024-03-06T07:35:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.088+0000 7fe70a800640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 0x55e91cf3d400 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fe70000da50 tx=0x7fe700003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce67c7b1c532f204 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe6e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1149054227 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700026060 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1149054227 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe700012410 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6f4001d30 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1149054227 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe700012610 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe70f5a2740 10 client.8559 fetch_fsmap learned FSMap version 88 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55e91cf676d0 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1149054227 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f4001d30 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1149054227 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 88) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe700020030 con 0x55e91cf55ee0 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe70f5a2740 10 client.8559 fetch_fsmap finished waiting for FSMap version 88 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.089+0000 7fe70f5a2740 10 client.8559 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe70f5a2740 20 client.8559 populate_metadata read hostname 'smithi097' 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf50490 0x55e91cf686d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe70f5a2740 4 client.8559 mds_command: new command op to 8382 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:35:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 --> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e91ce81b40 con 0x55e91cf50490 2024-03-06T07:35:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe709e00640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf50490 0x55e91cf686d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe709e00640 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf50490 0x55e91cf686d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ae35c693e06762 server_cookie=e36c5ec2b1d4a723 in_seq=0 out_seq=0 2024-03-06T07:35:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.090+0000 7fe6e6a00640 10 client.8559 ms_handle_connect on v2:172.21.15.97:6835/3564321877 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.091+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1149054227 <== mds.0 v2:172.21.15.97:6835/3564321877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e91ce81b40 con 0x55e91cf50490 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.091+0000 7fe6e6a00640 10 client.8559 handle_command_reply: tid=0 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 client.8559 shutdown 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 msgr2=0x55e91cf3d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 0x55e91cf3d400 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fe70000da50 tx=0x7fe700003ae0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 shutdown_connections 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf50490 0x55e91cf686d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf55ee0 0x55e91cf3d400 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf3cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1149054227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf37990 0x55e91cf4cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 >> 172.21.15.97:0/1149054227 conn(0x55e91cf0ae70 msgr2=0x55e91cf3a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe6e6a00640 0 client.8559 ms_handle_reset on v2:172.21.15.97:6835/3564321877 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 shutdown_connections 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1149054227 wait complete. 2024-03-06T07:35:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.092+0000 7fe70f5a2740 20 client.8559 trim_cache size 0 max 16384 2024-03-06T07:35:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.110+0000 7fe70f5a2740 1 Processor -- start 2024-03-06T07:35:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.110+0000 7fe70f5a2740 1 -- start start 2024-03-06T07:35:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37990 0x55e91cf50f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 0x55e91cf09d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7041ff250 con 0x55e91cf37d70 2024-03-06T07:35:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7041603e0 con 0x55e91cf55ee0 2024-03-06T07:35:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040b2800 con 0x55e91cf37990 2024-03-06T07:35:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe709400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe709400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf097c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39638/0 (socket says 172.21.15.97:39638) 2024-03-06T07:35:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.111+0000 7fe709400640 1 -- 172.21.15.97:0/1837916648 learned_addr learned my addr 172.21.15.97:0/1837916648 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe709400640 1 -- 172.21.15.97:0/1837916648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37990 msgr2=0x55e91cf50f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe709400640 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37990 0x55e91cf50f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe709400640 1 -- 172.21.15.97:0/1837916648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 msgr2=0x55e91cf09d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe709400640 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 0x55e91cf09d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe709400640 1 -- 172.21.15.97:0/1837916648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e91ce76ae0 con 0x55e91cf37d70 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe709400640 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf097c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c4b0 tx=0x7fe6f4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea1733318a01f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe6f1c00640 1 -- 172.21.15.97:0/1837916648 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4024070 con 0x55e91cf37d70 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe6f1c00640 1 -- 172.21.15.97:0/1837916648 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f40011c0 con 0x55e91cf37d70 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.112+0000 7fe6f1c00640 1 -- 172.21.15.97:0/1837916648 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f40072e0 con 0x55e91cf37d70 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1837916648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 msgr2=0x55e91cf097c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf097c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c4b0 tx=0x7fe6f4006000 comp rx=0 tx=0).stop 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1837916648 shutdown_connections 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 0x55e91cf09d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37d70 0x55e91cf097c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1837916648 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37990 0x55e91cf50f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1837916648 >> 172.21.15.97:0/1837916648 conn(0x55e91cf0ae70 msgr2=0x55e91cf05bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1837916648 shutdown_connections 2024-03-06T07:35:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.113+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1837916648 wait complete. 2024-03-06T07:35:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.114+0000 7fe70f5a2740 1 -- start start 2024-03-06T07:35:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.115+0000 7fe70f5a2740 1 Processor -- start 2024-03-06T07:35:56.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 0x55e91cf27a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf61e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 0x55e91cf54960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e91cf50310 con 0x55e91cf37990 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e91cf67a10 con 0x55e91cf55ee0 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e91cf50910 con 0x55e91cf37d70 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.116+0000 7fe70f5a2740 10 client.0 _ll_register_callbacks cb 0x7fe700004990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 0x55e91cf27a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 0x55e91cf27a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39652/0 (socket says 172.21.15.97:39652) 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 -- 172.21.15.97:0/1481877382 learned_addr learned my addr 172.21.15.97:0/1481877382 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 -- 172.21.15.97:0/1481877382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 msgr2=0x55e91cf61e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf61e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 -- 172.21.15.97:0/1481877382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 msgr2=0x55e91cf54960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 0x55e91cf54960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 -- 172.21.15.97:0/1481877382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e91cf27f40 con 0x55e91cf37990 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe709e00640 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 0x55e91cf27a00 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009890 tx=0x7fe6f807a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a05f0ce8ff2aa21 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe6e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1481877382 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80304d0 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1481877382 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6f8076cb0 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.117+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6f40011c0 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1481877382 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6f80aa0d0 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 10 client.8565 fetch_fsmap learned FSMap version 88 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55e91cf28150 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1481877382 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f40011c0 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1481877382 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 88) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe6f802f050 con 0x55e91cf37990 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 10 client.8565 fetch_fsmap finished waiting for FSMap version 88 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 10 client.8565 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 20 client.8565 populate_metadata read hostname 'smithi097' 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf54ef0 0x55e91cf78530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 4 client.8565 mds_command: new command op to 8382 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:35:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 --> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e91ce7be90 con 0x55e91cf54ef0 2024-03-06T07:35:56.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.118+0000 7fe709400640 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf54ef0 0x55e91cf78530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe709400640 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf54ef0 0x55e91cf78530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff18f0f2008553c1 server_cookie=b23b016e36cc77ce in_seq=0 out_seq=0 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe6e6a00640 10 client.8565 ms_handle_connect on v2:172.21.15.97:6835/3564321877 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe6e6a00640 1 -- 172.21.15.97:0/1481877382 <== mds.0 v2:172.21.15.97:6835/3564321877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55e91ce7be90 con 0x55e91cf54ef0 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe6e6a00640 10 client.8565 handle_command_reply: tid=0 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe70f5a2740 1 client.8565 shutdown 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 msgr2=0x55e91cf27a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.119+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 0x55e91cf27a00 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009890 tx=0x7fe6f807a7a0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 shutdown_connections 2024-03-06T07:35:56.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55e91cf54ef0 0x55e91cf78530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e91cf55ee0 0x55e91cf54960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55e91cf37d70 0x55e91cf61e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 --2- 172.21.15.97:0/1481877382 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e91cf37990 0x55e91cf27a00 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 >> 172.21.15.97:0/1481877382 conn(0x55e91cf0ae70 msgr2=0x55e91cf4c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe6e6a00640 0 client.8565 ms_handle_reset on v2:172.21.15.97:6835/3564321877 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 shutdown_connections 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 1 -- 172.21.15.97:0/1481877382 wait complete. 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.120+0000 7fe70f5a2740 20 client.8565 trim_cache size 0 max 16384 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:35:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 8382, 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 11.750853757, 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 88, 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 130, 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 130, 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 18.204558723000002 2024-03-06T07:35:56.126 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe6e806b4d0 msgr2=0x7fe6e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe6e806b4d0 0x7fe6e806d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe700007fd0 tx=0x7fe700009040 comp rx=0 tx=0).stop 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 msgr2=0x7fe7040c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 0x7fe7040c9590 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80098c0 tx=0x7fe6f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 shutdown_connections 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe6e806b4d0 0x7fe6e806d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe70415f0f0 0x7fe704153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe70415dfa0 0x7fe704161570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 --2- 172.21.15.97:0/744093633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7040ce9b0 0x7fe7040c9590 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 >> 172.21.15.97:0/744093633 conn(0x7fe7040b3620 msgr2=0x7fe7040d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 shutdown_connections 2024-03-06T07:35:56.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.121+0000 7fe70b200640 1 -- 172.21.15.97:0/744093633 wait complete. 2024-03-06T07:35:56.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:35:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.334+0000 7f7d97e00640 1 Processor -- start 2024-03-06T07:35:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.334+0000 7f7d97e00640 1 -- start start 2024-03-06T07:35:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d97e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7d9806b6d0 0x7f7d98162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d97e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d980cef70 con 0x7f7d980affe0 2024-03-06T07:35:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d97e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9815e3b0 con 0x7f7d9815dfd0 2024-03-06T07:35:56.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d97e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d980b0400 con 0x7f7d9806b6d0 2024-03-06T07:35:56.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d96000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7d9806b6d0 0x7f7d98162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d97400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7d980affe0 0x7f7d980d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42830/0 (socket says 172.21.15.97:42830) 2024-03-06T07:35:56.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.337+0000 7f7d97400640 1 -- 172.21.15.97:0/2724680360 learned_addr learned my addr 172.21.15.97:0/2724680360 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:35:56.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.338+0000 7f7d96000640 1 -- 172.21.15.97:0/2724680360 >> v1:172.21.15.135:6789/0 conn(0x7f7d9815dfd0 legacy=0x7f7d9806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.338+0000 7f7d96000640 1 -- 172.21.15.97:0/2724680360 >> v1:172.21.15.97:6789/0 conn(0x7f7d980affe0 legacy=0x7f7d980d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.338+0000 7f7d96000640 1 -- 172.21.15.97:0/2724680360 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98161c60 con 0x7f7d9806b6d0 2024-03-06T07:35:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.338+0000 7f7d95600640 1 -- 172.21.15.97:0/2724680360 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199924010 0 0) 0x7f7d980cef70 con 0x7f7d980affe0 2024-03-06T07:35:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.339+0000 7f7d96000640 1 --2- 172.21.15.97:0/2724680360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9806b6d0 0x7f7d98162700 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f7d840097a0 tx=0x7f7d8402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ec9e043945826b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.339+0000 7f7d95600640 1 -- 172.21.15.97:0/2724680360 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84038070 con 0x7f7d9806b6d0 2024-03-06T07:35:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.339+0000 7f7d95600640 1 -- 172.21.15.97:0/2724680360 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d84031700 con 0x7f7d9806b6d0 2024-03-06T07:35:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.339+0000 7f7d95600640 1 -- 172.21.15.97:0/2724680360 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84031ad0 con 0x7f7d9806b6d0 2024-03-06T07:35:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 -- 172.21.15.97:0/2724680360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9806b6d0 msgr2=0x7f7d98162700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 --2- 172.21.15.97:0/2724680360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9806b6d0 0x7f7d98162700 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f7d840097a0 tx=0x7f7d8402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 -- 172.21.15.97:0/2724680360 shutdown_connections 2024-03-06T07:35:56.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 --2- 172.21.15.97:0/2724680360 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9806b6d0 0x7f7d98162700 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 -- 172.21.15.97:0/2724680360 >> 172.21.15.97:0/2724680360 conn(0x7f7d9806c0d0 msgr2=0x7f7d9806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 -- 172.21.15.97:0/2724680360 shutdown_connections 2024-03-06T07:35:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.340+0000 7f7d97e00640 1 -- 172.21.15.97:0/2724680360 wait complete. 2024-03-06T07:35:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.342+0000 7f7d97e00640 1 Processor -- start 2024-03-06T07:35:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.342+0000 7f7d97e00640 1 -- start start 2024-03-06T07:35:56.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d9806b6d0 0x7f7d98156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d980affe0 0x7f7d980b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 0x7f7d980d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98161f00 con 0x7f7d9815dfd0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98162270 con 0x7f7d980affe0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d980ce8d0 con 0x7f7d9806b6d0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 0x7f7d980d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 0x7f7d980d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39674/0 (socket says 172.21.15.97:39674) 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d97400640 1 -- 172.21.15.97:0/4213790892 learned_addr learned my addr 172.21.15.97:0/4213790892 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.343+0000 7f7d96000640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d980affe0 0x7f7d980b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d96a00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d9806b6d0 0x7f7d98156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97400640 1 -- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d9806b6d0 msgr2=0x7f7d98156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97400640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d9806b6d0 0x7f7d98156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97400640 1 -- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d980affe0 msgr2=0x7f7d980b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97400640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d980affe0 0x7f7d980b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97400640 1 -- 172.21.15.97:0/4213790892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980b1cf0 con 0x7f7d9815dfd0 2024-03-06T07:35:56.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97400640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 0x7f7d980d06f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000c7a0 tx=0x7f7d9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90018070 con 0x7f7d9815dfd0 2024-03-06T07:35:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9000a640 con 0x7f7d9815dfd0 2024-03-06T07:35:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000aab0 con 0x7f7d9815dfd0 2024-03-06T07:35:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.344+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d980b1fd0 con 0x7f7d9815dfd0 2024-03-06T07:35:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.345+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d981570e0 con 0x7f7d9815dfd0 2024-03-06T07:35:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.346+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9804f330 con 0x7f7d9815dfd0 2024-03-06T07:35:56.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.347+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d9000ac50 con 0x7f7d9815dfd0 2024-03-06T07:35:56.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.348+0000 7f7d8f400640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d7806b4d0 0x7f7d7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.348+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d90089930 con 0x7f7d9815dfd0 2024-03-06T07:35:56.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.348+0000 7f7d96a00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d7806b4d0 0x7f7d7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.349+0000 7f7d96a00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d7806b4d0 0x7f7d7806d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7d88006fd0 tx=0x7f7d88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.356+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d90058710 con 0x7f7d9815dfd0 2024-03-06T07:35:56.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.596+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7d980b5e60 con 0x7f7d9815dfd0 2024-03-06T07:35:56.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.597+0000 7f7d8f400640 1 -- 172.21.15.97:0/4213790892 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v88) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7d90058010 con 0x7f7d9815dfd0 2024-03-06T07:35:56.599 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:35:56.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.599+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d7806b4d0 msgr2=0x7f7d7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.599+0000 7f7d97e00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d7806b4d0 0x7f7d7806d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7d88006fd0 tx=0x7f7d88008040 comp rx=0 tx=0).stop 2024-03-06T07:35:56.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 msgr2=0x7f7d980d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 0x7f7d980d06f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000c7a0 tx=0x7f7d9000cc60 comp rx=0 tx=0).stop 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 shutdown_connections 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7d7806b4d0 0x7f7d7806d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d9815dfd0 0x7f7d980d06f0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d980affe0 0x7f7d980b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 --2- 172.21.15.97:0/4213790892 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7d9806b6d0 0x7f7d98156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 >> 172.21.15.97:0/4213790892 conn(0x7f7d9806c0d0 msgr2=0x7f7d980adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 shutdown_connections 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:56.600+0000 7f7d97e00640 1 -- 172.21.15.97:0/4213790892 wait complete. 2024-03-06T07:35:56.622 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:35:56.622 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:35:56.622 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:35:56.623 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:35:56.623 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:56.623 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:35:56.623 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:35:56.623 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:35:56.623 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:56.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:56.623 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:35:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:35:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:35:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:35:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42118sec preferred_lft 42118sec 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:35:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:35:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:56.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:56.676 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:35:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:35:56.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:56.770 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:35:56.821 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:35:56.821 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:35:56.821 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:35:56.821 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:35:56.821 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:35:56.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:56.821 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:35:56.822 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:35:56.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:35:56.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:35:56.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:35:56.822 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:35:56.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:56.989 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:35:57.035 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:35:57.035 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:35:57.036 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:57.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:57.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:57.166 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:57.213 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:57.213 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:57.213 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:57.303 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:35:57.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:57.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:57.430 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:57.430 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:57.468 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:57.468 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:57.468 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:35:57.468 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:35:57.468 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:35:57.472 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:35:57.472 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:35:57.472 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:35:57.472 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:35:57.472 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:35:57.475 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:35:57.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:57.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:57.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:57.555 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:35:57.620 INFO:teuthology.orchestra.run.smithi097.stdout:client6241 2024-03-06T07:35:57.621 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:35:57.621 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:57.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:35:57.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.919+0000 7f705c600640 1 Processor -- start 2024-03-06T07:35:57.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.920+0000 7f705c600640 1 -- start start 2024-03-06T07:35:57.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.921+0000 7f705c600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.921+0000 7f705c600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f705415fff0 0x7f7054160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.921+0000 7f705c600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70540b46c0 con 0x7f70540ce4a0 2024-03-06T07:35:57.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.921+0000 7f705c600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540b49a0 con 0x7f705415fff0 2024-03-06T07:35:57.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.921+0000 7f705c600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7054072c90 con 0x7f705415fc10 2024-03-06T07:35:57.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.922+0000 7f7059200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f705415fff0 0x7f7054160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.922+0000 7f7059c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.922+0000 7f7059200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f705415fff0 0x7f7054160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:33034/0 (socket says 172.21.15.97:33034) 2024-03-06T07:35:57.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.922+0000 7f7059200640 1 -- 172.21.15.97:0/2208521432 learned_addr learned my addr 172.21.15.97:0/2208521432 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:57.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.923+0000 7f7059200640 1 -- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 msgr2=0x7f7054070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:57.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.923+0000 7f7059200640 1 --2- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.923+0000 7f7059200640 1 -- 172.21.15.97:0/2208521432 >> v1:172.21.15.135:6789/0 conn(0x7f70540ce4a0 legacy=0x7f70540ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.923+0000 7f7059200640 1 -- 172.21.15.97:0/2208521432 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7054160990 con 0x7f705415fff0 2024-03-06T07:35:57.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.924+0000 7f7059200640 1 --2- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f7054160450 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7048009870 tx=0x7f704802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e64cf823f53d1fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:57.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f7053e00640 1 -- 172.21.15.97:0/2208521432 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048034070 con 0x7f705415fff0 2024-03-06T07:35:57.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f7053e00640 1 -- 172.21.15.97:0/2208521432 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7048030840 con 0x7f705415fff0 2024-03-06T07:35:57.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f7053e00640 1 -- 172.21.15.97:0/2208521432 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048030c10 con 0x7f705415fff0 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 -- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 msgr2=0x7f7054160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 --2- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f7054160450 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7048009870 tx=0x7f704802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 -- 172.21.15.97:0/2208521432 shutdown_connections 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 --2- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f7054160450 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 --2- 172.21.15.97:0/2208521432 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 -- 172.21.15.97:0/2208521432 >> 172.21.15.97:0/2208521432 conn(0x7f70540b1940 msgr2=0x7f70540b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 -- 172.21.15.97:0/2208521432 shutdown_connections 2024-03-06T07:35:57.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.925+0000 7f705c600640 1 -- 172.21.15.97:0/2208521432 wait complete. 2024-03-06T07:35:57.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.927+0000 7f705c600640 1 Processor -- start 2024-03-06T07:35:57.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.927+0000 7f705c600640 1 -- start start 2024-03-06T07:35:57.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f705c600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 0x7f7054155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f705c600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f705c600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f705409b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f705c600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7054072a30 con 0x7f70540ce4a0 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f705c600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70540728b0 con 0x7f705415fff0 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f705c600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705415c2e0 con 0x7f705415fc10 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.928+0000 7f7059c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 0x7f7054155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 0x7f7054155590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39680/0 (socket says 172.21.15.97:39680) 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 -- 172.21.15.97:0/497718699 learned_addr learned my addr 172.21.15.97:0/497718699 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059200640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 -- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 msgr2=0x7f7054153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f705a600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f705409b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 -- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 msgr2=0x7f705409b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f705409b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 -- 172.21.15.97:0/497718699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7054157200 con 0x7f70540ce4a0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.929+0000 7f7059c00640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 0x7f7054155590 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f704400e9b0 tx=0x7f704400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.930+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044017070 con 0x7f70540ce4a0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.930+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704400c9a0 con 0x7f70540ce4a0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.930+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704400cd90 con 0x7f70540ce4a0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.930+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70541574e0 con 0x7f70540ce4a0 2024-03-06T07:35:57.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.930+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7054157aa0 con 0x7f70540ce4a0 2024-03-06T07:35:57.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.931+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705404f330 con 0x7f70540ce4a0 2024-03-06T07:35:57.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.932+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f704401a070 con 0x7f70540ce4a0 2024-03-06T07:35:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.933+0000 7f7051600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f703806b4d0 0x7f703806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.933+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f704408a1d0 con 0x7f70540ce4a0 2024-03-06T07:35:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.934+0000 7f7059200640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f703806b4d0 0x7f703806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.934+0000 7f7059200640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f703806b4d0 0x7f703806d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7048036000 tx=0x7f704803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:57.938+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70440590c0 con 0x7f70540ce4a0 2024-03-06T07:35:58.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.181+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f705406ef30 con 0x7f70540ce4a0 2024-03-06T07:35:58.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.479+0000 7f7051600640 1 -- 172.21.15.97:0/497718699 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v89) v1 ==== 105+0+0 (secure 0 0 0) 0x7f70440589c0 con 0x7f70540ce4a0 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f703806b4d0 msgr2=0x7f703806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f703806b4d0 0x7f703806d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7048036000 tx=0x7f704803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 msgr2=0x7f7054155590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 0x7f7054155590 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f704400e9b0 tx=0x7f704400ee70 comp rx=0 tx=0).stop 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 shutdown_connections 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f703806b4d0 0x7f703806d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f705415fff0 0x7f705409b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f705415fc10 0x7f7054153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.484+0000 7f705c600640 1 --2- 172.21.15.97:0/497718699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70540ce4a0 0x7f7054155590 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.485+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 >> 172.21.15.97:0/497718699 conn(0x7f70540b1940 msgr2=0x7f70540acd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:58.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.485+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 shutdown_connections 2024-03-06T07:35:58.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.485+0000 7f705c600640 1 -- 172.21.15.97:0/497718699 wait complete. 2024-03-06T07:35:58.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:58.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.693+0000 7fb103200640 1 Processor -- start 2024-03-06T07:35:58.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.694+0000 7fb103200640 1 -- start start 2024-03-06T07:35:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb103200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc15bd90 0x7fb0fc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb103200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb0fc15c170 0x7fb0fc0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb103200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb0fc0ccfb0 0x7fb0fc0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb103200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0b54f0 con 0x7fb0fc0ccfb0 2024-03-06T07:35:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb103200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0d0380 con 0x7fb0fc15c170 2024-03-06T07:35:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb103200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0cd580 con 0x7fb0fc15bd90 2024-03-06T07:35:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb102800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb0fc0ccfb0 0x7fb0fc0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb102800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb0fc0ccfb0 0x7fb0fc0b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39684/0 (socket says 172.21.15.97:39684) 2024-03-06T07:35:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.695+0000 7fb102800640 1 -- 172.21.15.97:0/2660823647 learned_addr learned my addr 172.21.15.97:0/2660823647 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb101e00640 1 --2- 172.21.15.97:0/2660823647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc15bd90 0x7fb0fc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb101400640 1 --2- 172.21.15.97:0/2660823647 >> v2:172.21.15.135:3300/0 conn(0x7fb0fc15c170 0x7fb0fc0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb102800640 1 -- 172.21.15.97:0/2660823647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc15bd90 msgr2=0x7fb0fc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb102800640 1 --2- 172.21.15.97:0/2660823647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc15bd90 0x7fb0fc0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb102800640 1 -- 172.21.15.97:0/2660823647 >> v2:172.21.15.135:3300/0 conn(0x7fb0fc15c170 msgr2=0x7fb0fc0b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb102800640 1 --2- 172.21.15.97:0/2660823647 >> v2:172.21.15.135:3300/0 conn(0x7fb0fc15c170 0x7fb0fc0b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb102800640 1 -- 172.21.15.97:0/2660823647 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc0cd860 con 0x7fb0fc0ccfb0 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.696+0000 7fb101400640 1 --2- 172.21.15.97:0/2660823647 >> v2:172.21.15.135:3300/0 conn(0x7fb0fc15c170 0x7fb0fc0b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.697+0000 7fb102800640 1 --2- 172.21.15.97:0/2660823647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0b5ec0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc070510 tx=0x7fb0f802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f527f93a522b319 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.697+0000 7fb100a00640 1 -- 172.21.15.97:0/2660823647 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f802c9f0 con 0x7fb0fc0ccfb0 2024-03-06T07:35:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.697+0000 7fb100a00640 1 -- 172.21.15.97:0/2660823647 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f802cb90 con 0x7fb0fc0ccfb0 2024-03-06T07:35:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.697+0000 7fb100a00640 1 -- 172.21.15.97:0/2660823647 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80343f0 con 0x7fb0fc0ccfb0 2024-03-06T07:35:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 -- 172.21.15.97:0/2660823647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc0ccfb0 msgr2=0x7fb0fc0b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 --2- 172.21.15.97:0/2660823647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0b5ec0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc070510 tx=0x7fb0f802be50 comp rx=0 tx=0).stop 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 -- 172.21.15.97:0/2660823647 shutdown_connections 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 --2- 172.21.15.97:0/2660823647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0b5ec0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 --2- 172.21.15.97:0/2660823647 >> v2:172.21.15.135:3300/0 conn(0x7fb0fc15c170 0x7fb0fc0b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 --2- 172.21.15.97:0/2660823647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc15bd90 0x7fb0fc0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 -- 172.21.15.97:0/2660823647 >> 172.21.15.97:0/2660823647 conn(0x7fb0fc0725e0 msgr2=0x7fb0fc0729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 -- 172.21.15.97:0/2660823647 shutdown_connections 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.698+0000 7fb103200640 1 -- 172.21.15.97:0/2660823647 wait complete. 2024-03-06T07:35:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.700+0000 7fb103200640 1 Processor -- start 2024-03-06T07:35:58.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.700+0000 7fb103200640 1 -- start start 2024-03-06T07:35:58.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb103200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb103200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 0x7fb0fc15ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb103200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0fc15c170 0x7fb0fc156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb103200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0ce500 con 0x7fb0fc15bd90 2024-03-06T07:35:58.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb103200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0ce380 con 0x7fb0fc15c170 2024-03-06T07:35:58.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb103200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0ce680 con 0x7fb0fc0ccfb0 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 0x7fb0fc15ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 0x7fb0fc15ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39690/0 (socket says 172.21.15.97:39690) 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 -- 172.21.15.97:0/1254963058 learned_addr learned my addr 172.21.15.97:0/1254963058 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101e00640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb102800640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0fc15c170 0x7fb0fc156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 -- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc0ccfb0 msgr2=0x7fb0fc0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 -- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0fc15c170 msgr2=0x7fb0fc156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0fc15c170 0x7fb0fc156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.701+0000 7fb101400640 1 -- 172.21.15.97:0/1254963058 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc158280 con 0x7fb0fc15bd90 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.702+0000 7fb101400640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 0x7fb0fc15ab90 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c820 tx=0x7fb0ec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.702+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec015070 con 0x7fb0fc15bd90 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.702+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ec007a30 con 0x7fb0fc15bd90 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.702+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec007de0 con 0x7fb0fc15bd90 2024-03-06T07:35:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.702+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc158b70 con 0x7fb0fc15bd90 2024-03-06T07:35:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.702+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc158800 con 0x7fb0fc15bd90 2024-03-06T07:35:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.703+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc04f330 con 0x7fb0fc15bd90 2024-03-06T07:35:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.704+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0ec018070 con 0x7fb0fc15bd90 2024-03-06T07:35:58.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.705+0000 7fb0f6000640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb0e006b4d0 0x7fb0e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.705+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0ec08a130 con 0x7fb0fc15bd90 2024-03-06T07:35:58.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.705+0000 7fb101e00640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb0e006b4d0 0x7fb0e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.706+0000 7fb101e00640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb0e006b4d0 0x7fb0e006d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000b7e0 tx=0x7fb0f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.710+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0ec058fb0 con 0x7fb0fc15bd90 2024-03-06T07:35:58.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.943+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0fc0cfa70 con 0x7fb0fc15bd90 2024-03-06T07:35:58.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.944+0000 7fb0f6000640 1 -- 172.21.15.97:0/1254963058 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4998 (secure 0 0 0) 0x7fb0ec0588b0 con 0x7fb0fc15bd90 2024-03-06T07:35:58.947 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:35:58.947 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1847010599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1847010599},{"type":"v1","addr":"172.21.15.135:6833","nonce":1847010599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6175,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/3157120524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":3157120524},{"type":"v1","addr":"172.21.15.135:6835","nonce":3157120524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:58.484681+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:creating","state_seq":1,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:35:58.947 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 90 2024-03-06T07:35:58.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb0e006b4d0 msgr2=0x7fb0e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb0e006b4d0 0x7fb0e006d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000b7e0 tx=0x7fb0f0008040 comp rx=0 tx=0).stop 2024-03-06T07:35:58.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 msgr2=0x7fb0fc15ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 0x7fb0fc15ab90 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c820 tx=0x7fb0ec00cce0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 shutdown_connections 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb0e006b4d0 0x7fb0e006d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0fc15c170 0x7fb0fc156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0fc15bd90 0x7fb0fc15ab90 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 --2- 172.21.15.97:0/1254963058 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0fc0ccfb0 0x7fb0fc0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 >> 172.21.15.97:0/1254963058 conn(0x7fb0fc0725e0 msgr2=0x7fb0fc06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 shutdown_connections 2024-03-06T07:35:58.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:35:58.947+0000 7fb103200640 1 -- 172.21.15.97:0/1254963058 wait complete. 2024-03-06T07:35:58.967 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:36:03.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.182+0000 7f98dc200640 1 Processor -- start 2024-03-06T07:36:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.182+0000 7f98dc200640 1 -- start start 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98dc200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98d415f930 0x7f98d4066f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98dc200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d406ed70 0x7f98d415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98dc200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f98d415c2f0 0x7f98d415c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98dc200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d40ccd90 con 0x7f98d415f930 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98dc200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d40afc00 con 0x7f98d415c2f0 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98dc200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d406f1f0 con 0x7f98d406ed70 2024-03-06T07:36:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98d9800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98d415f930 0x7f98d4066f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98d9800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98d415f930 0x7f98d4066f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54188/0 (socket says 172.21.15.97:54188) 2024-03-06T07:36:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.184+0000 7f98d9800640 1 -- 172.21.15.97:0/785036324 learned_addr learned my addr 172.21.15.97:0/785036324 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98da200640 1 --2- 172.21.15.97:0/785036324 >> v2:172.21.15.135:3300/0 conn(0x7f98d415c2f0 0x7f98d415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d8e00640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d406ed70 0x7f98d415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d9800640 1 -- 172.21.15.97:0/785036324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d406ed70 msgr2=0x7f98d415fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d9800640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d406ed70 0x7f98d415fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d9800640 1 -- 172.21.15.97:0/785036324 >> v2:172.21.15.135:3300/0 conn(0x7f98d415c2f0 msgr2=0x7f98d415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d9800640 1 --2- 172.21.15.97:0/785036324 >> v2:172.21.15.135:3300/0 conn(0x7f98d415c2f0 0x7f98d415c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d9800640 1 -- 172.21.15.97:0/785036324 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d406e5b0 con 0x7f98d415f930 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98da200640 1 --2- 172.21.15.97:0/785036324 >> v2:172.21.15.135:3300/0 conn(0x7f98d415c2f0 0x7f98d415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.185+0000 7f98d8e00640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d406ed70 0x7f98d415fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:04.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.186+0000 7f98d9800640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415f930 0x7f98d4066f10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f98d4071310 tx=0x7f98c802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e1538336e1874e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.186+0000 7f98d3e00640 1 -- 172.21.15.97:0/785036324 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c80317c0 con 0x7f98d415f930 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.186+0000 7f98d3e00640 1 -- 172.21.15.97:0/785036324 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c8037070 con 0x7f98d415f930 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.186+0000 7f98d3e00640 1 -- 172.21.15.97:0/785036324 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c8031b80 con 0x7f98d415f930 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 -- 172.21.15.97:0/785036324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415f930 msgr2=0x7f98d4066f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415f930 0x7f98d4066f10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f98d4071310 tx=0x7f98c802b860 comp rx=0 tx=0).stop 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 -- 172.21.15.97:0/785036324 shutdown_connections 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 --2- 172.21.15.97:0/785036324 >> v2:172.21.15.135:3300/0 conn(0x7f98d415c2f0 0x7f98d415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d406ed70 0x7f98d415fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 --2- 172.21.15.97:0/785036324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415f930 0x7f98d4066f10 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 -- 172.21.15.97:0/785036324 >> 172.21.15.97:0/785036324 conn(0x7f98d40d0340 msgr2=0x7f98d40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 -- 172.21.15.97:0/785036324 shutdown_connections 2024-03-06T07:36:04.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.187+0000 7f98dc200640 1 -- 172.21.15.97:0/785036324 wait complete. 2024-03-06T07:36:04.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.188+0000 7f98dc200640 1 Processor -- start 2024-03-06T07:36:04.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.189+0000 7f98dc200640 1 -- start start 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.189+0000 7f98dc200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d406ed70 0x7f98d40b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.189+0000 7f98dc200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 0x7f98d415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.189+0000 7f98dc200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d415f930 0x7f98d415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98dc200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40cd2c0 con 0x7f98d415c2f0 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98dc200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40ccfe0 con 0x7f98d406ed70 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98dc200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d406ddb0 con 0x7f98d415f930 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 0x7f98d415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 0x7f98d415ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54194/0 (socket says 172.21.15.97:54194) 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 -- 172.21.15.97:0/566954001 learned_addr learned my addr 172.21.15.97:0/566954001 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98da200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d415f930 0x7f98d415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d9800640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d406ed70 0x7f98d40b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 -- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d415f930 msgr2=0x7f98d415b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d415f930 0x7f98d415b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 -- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d406ed70 msgr2=0x7f98d40b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d406ed70 0x7f98d40b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.190+0000 7f98d8e00640 1 -- 172.21.15.97:0/566954001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d41f43b0 con 0x7f98d415c2f0 2024-03-06T07:36:04.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.191+0000 7f98d8e00640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 0x7f98d415ae10 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c7c0 tx=0x7f98c400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.191+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4015070 con 0x7f98d415c2f0 2024-03-06T07:36:04.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.191+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4007a30 con 0x7f98d415c2f0 2024-03-06T07:36:04.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.191+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d41f4690 con 0x7f98d415c2f0 2024-03-06T07:36:04.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.191+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4007e00 con 0x7f98d415c2f0 2024-03-06T07:36:04.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.191+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d40b3830 con 0x7f98d415c2f0 2024-03-06T07:36:04.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.193+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d404f330 con 0x7f98d415c2f0 2024-03-06T07:36:04.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.193+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98c4018070 con 0x7f98d415c2f0 2024-03-06T07:36:04.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.194+0000 7f98d1600640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.195+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98c408a210 con 0x7f98d415c2f0 2024-03-06T07:36:04.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.195+0000 7f98d9800640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.196+0000 7f98d9800640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f98c8036000 tx=0x7f98c803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.199+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98c4058ff0 con 0x7f98d415c2f0 2024-03-06T07:36:04.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.439+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d4067640 con 0x7f98d415c2f0 2024-03-06T07:36:04.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.440+0000 7f98d1600640 1 -- 172.21.15.97:0/566954001 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f98c40588f0 con 0x7f98d415c2f0 2024-03-06T07:36:04.441 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:04.442 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1847010599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1847010599},{"type":"v1","addr":"172.21.15.135:6833","nonce":1847010599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6175,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/3157120524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":3157120524},{"type":"v1","addr":"172.21.15.135:6835","nonce":3157120524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:59.487556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:active","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8379,8382]},"id":9}]} 2024-03-06T07:36:04.442 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 91 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 msgr2=0x7f98b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f98c8036000 tx=0x7f98c803d040 comp rx=0 tx=0).stop 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 msgr2=0x7f98d415ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 0x7f98d415ae10 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c7c0 tx=0x7f98c400cc80 comp rx=0 tx=0).stop 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 shutdown_connections 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98d415f930 0x7f98d415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98d415c2f0 0x7f98d415ae10 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 --2- 172.21.15.97:0/566954001 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d406ed70 0x7f98d40b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.443+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 >> 172.21.15.97:0/566954001 conn(0x7f98d40d0340 msgr2=0x7f98d406b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.444+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 shutdown_connections 2024-03-06T07:36:04.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.444+0000 7f98dc200640 1 -- 172.21.15.97:0/566954001 wait complete. 2024-03-06T07:36:04.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:04.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.649+0000 7f53b2600640 1 Processor -- start 2024-03-06T07:36:04.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.649+0000 7f53b2600640 1 -- start start 2024-03-06T07:36:04.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53b2600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac0cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53b2600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f53ac15f110 0x7f53ac0d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53b2600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ac0d10d0 con 0x7f53ac161c50 2024-03-06T07:36:04.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53b2600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ac0cfd50 con 0x7f53ac15f110 2024-03-06T07:36:04.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53b2600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac0cf140 con 0x7f53ac0d07b0 2024-03-06T07:36:04.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53abe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f53ac161c50 0x7f53ac1620b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49436/0 (socket says 172.21.15.97:49436) 2024-03-06T07:36:04.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53abe00640 1 -- 172.21.15.97:0/1866640522 learned_addr learned my addr 172.21.15.97:0/1866640522 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:04.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53a2a00640 1 --2- 172.21.15.97:0/1866640522 >> v2:172.21.15.135:3300/0 conn(0x7f53ac15f110 0x7f53ac0d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.652+0000 7f53ab400640 1 --2- 172.21.15.97:0/1866640522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac0cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53aaa00640 1 -- 172.21.15.97:0/1866640522 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260327471 0 0) 0x7f53ac0d10d0 con 0x7f53ac161c50 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53aaa00640 1 -- 172.21.15.97:0/1866640522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5390003520 con 0x7f53ac161c50 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53aaa00640 1 -- 172.21.15.97:0/1866640522 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2505049237 0 0) 0x7f5390003520 con 0x7f53ac161c50 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53ab400640 1 -- 172.21.15.97:0/1866640522 >> v2:172.21.15.135:3300/0 conn(0x7f53ac15f110 msgr2=0x7f53ac0d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53ab400640 1 --2- 172.21.15.97:0/1866640522 >> v2:172.21.15.135:3300/0 conn(0x7f53ac15f110 0x7f53ac0d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53ab400640 1 -- 172.21.15.97:0/1866640522 >> v1:172.21.15.97:6789/0 conn(0x7f53ac161c50 legacy=0x7f53ac1620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53ab400640 1 -- 172.21.15.97:0/1866640522 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac0cf420 con 0x7f53ac0d07b0 2024-03-06T07:36:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.653+0000 7f53a2a00640 1 --2- 172.21.15.97:0/1866640522 >> v2:172.21.15.135:3300/0 conn(0x7f53ac15f110 0x7f53ac0d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:04.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.655+0000 7f53ab400640 1 --2- 172.21.15.97:0/1866640522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac0cea80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f539c009d90 tx=0x7f539c02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee2c8be12478587b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53aaa00640 1 -- 172.21.15.97:0/1866640522 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c039070 con 0x7f53ac0d07b0 2024-03-06T07:36:04.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53aaa00640 1 -- 172.21.15.97:0/1866640522 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c02c8a0 con 0x7f53ac0d07b0 2024-03-06T07:36:04.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53aaa00640 1 -- 172.21.15.97:0/1866640522 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4001670 con 0x7f53ac0d07b0 2024-03-06T07:36:04.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 -- 172.21.15.97:0/1866640522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 msgr2=0x7f53ac0cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 --2- 172.21.15.97:0/1866640522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac0cea80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f539c009d90 tx=0x7f539c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 -- 172.21.15.97:0/1866640522 shutdown_connections 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 --2- 172.21.15.97:0/1866640522 >> v2:172.21.15.135:3300/0 conn(0x7f53ac15f110 0x7f53ac0d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 --2- 172.21.15.97:0/1866640522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac0cea80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 -- 172.21.15.97:0/1866640522 >> 172.21.15.97:0/1866640522 conn(0x7f53ac0d3ee0 msgr2=0x7f53ac0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 -- 172.21.15.97:0/1866640522 shutdown_connections 2024-03-06T07:36:04.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.656+0000 7f53b2600640 1 -- 172.21.15.97:0/1866640522 wait complete. 2024-03-06T07:36:04.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.658+0000 7f53b2600640 1 Processor -- start 2024-03-06T07:36:04.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.658+0000 7f53b2600640 1 -- start start 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.659+0000 7f53b2600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.659+0000 7f53b2600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ac15f110 0x7f53ac0cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.659+0000 7f53b2600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 0x7f53ac0cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.659+0000 7f53b2600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac15f570 con 0x7f53ac161c50 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.659+0000 7f53b2600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac0cf9e0 con 0x7f53ac15f110 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.659+0000 7f53b2600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac15e390 con 0x7f53ac0d07b0 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 0x7f53ac0cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 0x7f53ac0cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54208/0 (socket says 172.21.15.97:54208) 2024-03-06T07:36:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 -- 172.21.15.97:0/628159784 learned_addr learned my addr 172.21.15.97:0/628159784 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:04.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53ab400640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53a2a00640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ac15f110 0x7f53ac0cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 -- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 msgr2=0x7f53ac15c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac15c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 -- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ac15f110 msgr2=0x7f53ac0cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ac15f110 0x7f53ac0cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53abe00640 1 -- 172.21.15.97:0/628159784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac15ca20 con 0x7f53ac161c50 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.660+0000 7f53ab400640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.661+0000 7f53abe00640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 0x7f53ac0cb4f0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c6d0 tx=0x7f53a400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.661+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4005ad0 con 0x7f53ac161c50 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.661+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4005c70 con 0x7f53ac161c50 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.661+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac0ca400 con 0x7f53ac161c50 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.661+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4007270 con 0x7f53ac161c50 2024-03-06T07:36:04.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.661+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac0cbe00 con 0x7f53ac161c50 2024-03-06T07:36:04.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.662+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac04f330 con 0x7f53ac161c50 2024-03-06T07:36:04.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.663+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53a4007410 con 0x7f53ac161c50 2024-03-06T07:36:04.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.664+0000 7f53a3e00640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f538806b4d0 0x7f538806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.664+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53a408a0b0 con 0x7f53ac161c50 2024-03-06T07:36:04.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.664+0000 7f53ab400640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f538806b4d0 0x7f538806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.665+0000 7f53ab400640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f538806b4d0 0x7f538806d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f539c036000 tx=0x7f539c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.672+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53a4058e10 con 0x7f53ac161c50 2024-03-06T07:36:04.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.908+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53ac08d940 con 0x7f53ac161c50 2024-03-06T07:36:04.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.909+0000 7f53a3e00640 1 -- 172.21.15.97:0/628159784 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f53a4058710 con 0x7f53ac161c50 2024-03-06T07:36:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:04.910 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1847010599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1847010599},{"type":"v1","addr":"172.21.15.135:6833","nonce":1847010599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6175,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/3157120524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":3157120524},{"type":"v1","addr":"172.21.15.135:6835","nonce":3157120524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:59.487556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:active","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8379,8382]},"id":9}]} 2024-03-06T07:36:04.910 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 91 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.911+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f538806b4d0 msgr2=0x7f538806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.911+0000 7f53b2600640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f538806b4d0 0x7f538806d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f539c036000 tx=0x7f539c030000 comp rx=0 tx=0).stop 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.911+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 msgr2=0x7f53ac0cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.911+0000 7f53b2600640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 0x7f53ac0cb4f0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c6d0 tx=0x7f53a400cb90 comp rx=0 tx=0).stop 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 shutdown_connections 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f538806b4d0 0x7f538806d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ac161c50 0x7f53ac0cb4f0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ac15f110 0x7f53ac0cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 --2- 172.21.15.97:0/628159784 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ac0d07b0 0x7f53ac15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 >> 172.21.15.97:0/628159784 conn(0x7f53ac0d3ee0 msgr2=0x7f53ac0b8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 shutdown_connections 2024-03-06T07:36:04.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:04.912+0000 7f53b2600640 1 -- 172.21.15.97:0/628159784 wait complete. 2024-03-06T07:36:04.931 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-03-06T07:36:04.932 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:36:04.932 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:04.960 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:36:04.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:36:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.159+0000 7f29e9200640 1 Processor -- start 2024-03-06T07:36:05.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.159+0000 7f29e9200640 1 -- start start 2024-03-06T07:36:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.161+0000 7f29e9200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 0x7f29e40cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.161+0000 7f29e9200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40cf2b0 con 0x7f29e40d12e0 2024-03-06T07:36:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.161+0000 7f29e9200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40cf590 con 0x7f29e4162440 2024-03-06T07:36:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.161+0000 7f29e9200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40cf870 con 0x7f29e415e360 2024-03-06T07:36:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.161+0000 7f29e1400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f29e40d12e0 0x7f29e4161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49448/0 (socket says 172.21.15.97:49448) 2024-03-06T07:36:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.161+0000 7f29e1400640 1 -- 172.21.15.97:0/2108731157 learned_addr learned my addr 172.21.15.97:0/2108731157 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e1e00640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 0x7f29e40cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557500362 0 0) 0x7f29e40cf2b0 con 0x7f29e40d12e0 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c8003520 con 0x7f29e40d12e0 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1765375407 0 0) 0x7f29c8003520 con 0x7f29e40d12e0 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 msgr2=0x7f29e40cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 0x7f29e40cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 >> v1:172.21.15.135:6789/0 conn(0x7f29e4162440 legacy=0x7f29e40d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40cfb50 con 0x7f29e40d12e0 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f29d0001670 con 0x7f29e40d12e0 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 >> v1:172.21.15.97:6789/0 conn(0x7f29e40d12e0 legacy=0x7f29e4161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29c80039e0 0x7f29c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 0x7f29c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29c8004d40 0x7f29c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40cf870 con 0x7f29c8004d40 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c800bb00 con 0x7f29c80039e0 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.162+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c800bd00 con 0x7f29e415e360 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e2800640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29c8004d40 0x7f29c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e1400640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 0x7f29c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e1e00640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29c80039e0 0x7f29c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e2800640 1 -- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 msgr2=0x7f29c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e2800640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e415e360 0x7f29c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e2800640 1 -- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29c80039e0 msgr2=0x7f29c8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e2800640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29c80039e0 0x7f29c8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.163+0000 7f29e2800640 1 -- 172.21.15.97:0/2108731157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8003520 con 0x7f29c8004d40 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.165+0000 7f29e2800640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29c8004d40 0x7f29c80051c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f29e40d0670 tx=0x7f29c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15041b28ad162c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.165+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc001a70 con 0x7f29c8004d40 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.165+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29dc003b70 con 0x7f29c8004d40 2024-03-06T07:36:05.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.165+0000 7f29e0a00640 1 -- 172.21.15.97:0/2108731157 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0050d0 con 0x7f29c8004d40 2024-03-06T07:36:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e9200640 1 -- 172.21.15.97:0/2108731157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29c8004d40 msgr2=0x7f29c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e9200640 1 --2- 172.21.15.97:0/2108731157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29c8004d40 0x7f29c80051c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f29e40d0670 tx=0x7f29c8005830 comp rx=0 tx=0).stop 2024-03-06T07:36:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e2800640 1 -- 172.21.15.97:0/2108731157 reap_dead start 2024-03-06T07:36:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e9200640 1 -- 172.21.15.97:0/2108731157 shutdown_connections 2024-03-06T07:36:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e9200640 1 -- 172.21.15.97:0/2108731157 >> 172.21.15.97:0/2108731157 conn(0x7f29e40d3f50 msgr2=0x7f29e40d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e9200640 1 -- 172.21.15.97:0/2108731157 shutdown_connections 2024-03-06T07:36:05.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.166+0000 7f29e9200640 1 -- 172.21.15.97:0/2108731157 wait complete. 2024-03-06T07:36:05.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.168+0000 7f29e9200640 1 Processor -- start 2024-03-06T07:36:05.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.168+0000 7f29e9200640 1 -- start start 2024-03-06T07:36:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e9200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e415fd80 0x7f29e40cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e9200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e40ccb70 0x7f29e4160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 0x7f29e41f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e9200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40d18c0 con 0x7f29e4160fd0 2024-03-06T07:36:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e9200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40d1740 con 0x7f29e415fd80 2024-03-06T07:36:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e9200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40cec10 con 0x7f29e40ccb70 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 0x7f29e41f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 0x7f29e41f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54240/0 (socket says 172.21.15.97:54240) 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.169+0000 7f29e2800640 1 -- 172.21.15.97:0/4226996706 learned_addr learned my addr 172.21.15.97:0/4226996706 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e1400640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e40ccb70 0x7f29e4160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e1e00640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e415fd80 0x7f29e40cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e2800640 1 -- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e40ccb70 msgr2=0x7f29e4160a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e2800640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e40ccb70 0x7f29e4160a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e2800640 1 -- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e415fd80 msgr2=0x7f29e40cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e2800640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e415fd80 0x7f29e40cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e2800640 1 -- 172.21.15.97:0/4226996706 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40cd180 con 0x7f29e4160fd0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.170+0000 7f29e2800640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 0x7f29e41f7c70 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f29e408d370 tx=0x7f29dc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.171+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc007db0 con 0x7f29e4160fd0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.171+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29dc006b50 con 0x7f29e4160fd0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.171+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc03b480 con 0x7f29e4160fd0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.171+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e40cd460 con 0x7f29e4160fd0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.171+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e40cda20 con 0x7f29e4160fd0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.172+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e404f330 con 0x7f29e4160fd0 2024-03-06T07:36:05.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.173+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29dc007030 con 0x7f29e4160fd0 2024-03-06T07:36:05.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.174+0000 7f29da000640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b4d0 0x7f29c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.174+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29dc0ab990 con 0x7f29e4160fd0 2024-03-06T07:36:05.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.175+0000 7f29e1e00640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b4d0 0x7f29c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.175+0000 7f29e1e00640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b4d0 0x7f29c406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f29e41f8d30 tx=0x7f29d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.179+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29dc040070 con 0x7f29e4160fd0 2024-03-06T07:36:05.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.474+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f29e40ceff0 con 0x7f29e4160fd0 2024-03-06T07:36:05.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.692+0000 7f29da000640 1 -- 172.21.15.97:0/4226996706 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6175 v92) v1 ==== 95+0+0 (secure 0 0 0) 0x7f29dc043390 con 0x7f29e4160fd0 2024-03-06T07:36:05.693 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 6175 2024-03-06T07:36:05.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b4d0 msgr2=0x7f29c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b4d0 0x7f29c406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f29e41f8d30 tx=0x7f29d4008040 comp rx=0 tx=0).stop 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 msgr2=0x7f29e41f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 0x7f29e41f7c70 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f29e408d370 tx=0x7f29dc0014a0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 shutdown_connections 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f29c406b4d0 0x7f29c406d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29e4160fd0 0x7f29e41f7c70 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f29e40ccb70 0x7f29e4160a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 --2- 172.21.15.97:0/4226996706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29e415fd80 0x7f29e40cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 >> 172.21.15.97:0/4226996706 conn(0x7f29e40d3f50 msgr2=0x7f29e40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 shutdown_connections 2024-03-06T07:36:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.696+0000 7f29e9200640 1 -- 172.21.15.97:0/4226996706 wait complete. 2024-03-06T07:36:05.719 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:36:05.719 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:05.743 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:36:05.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:36:05.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.993+0000 7ff9e7a00640 1 Processor -- start 2024-03-06T07:36:05.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.993+0000 7ff9e7a00640 1 -- start start 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e7a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9e00c4930 0x7ff9e00c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e7a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4d10 0x7ff9e00c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e7a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e0145b30 con 0x7ff9e00c56b0 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e7a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e0142ce0 con 0x7ff9e00c4930 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e7a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00cbca0 con 0x7ff9e00c4d10 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e5000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9e00c4930 0x7ff9e00c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9dfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4d10 0x7ff9e00c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e5000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9e00c4930 0x7ff9e00c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:48248/0 (socket says 172.21.15.97:48248) 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e5000640 1 -- 172.21.15.97:0/3759401829 learned_addr learned my addr 172.21.15.97:0/3759401829 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.996+0000 7ff9e5a00640 1 --1- 172.21.15.97:0/3759401829 >> v1:172.21.15.135:6789/0 conn(0x7ff9e00c56b0 0x7ff9e006d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:47882/0 (socket says 172.21.15.97:47882) 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.997+0000 7ff9e5000640 1 -- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4d10 msgr2=0x7ff9e00c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.997+0000 7ff9e5000640 1 --2- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4d10 0x7ff9e00c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.997+0000 7ff9e5000640 1 -- 172.21.15.97:0/3759401829 >> v1:172.21.15.135:6789/0 conn(0x7ff9e00c56b0 legacy=0x7ff9e006d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.997+0000 7ff9e5000640 1 -- 172.21.15.97:0/3759401829 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e00cbf80 con 0x7ff9e00c4930 2024-03-06T07:36:05.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.997+0000 7ff9dfe00640 1 --2- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4d10 0x7ff9e00c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:06.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:05.999+0000 7ff9e5000640 1 --2- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c4930 0x7ff9e00c6460 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009870 tx=0x7ff9d402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b3c973a20e76679 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.000+0000 7ff9df400640 1 -- 172.21.15.97:0/3759401829 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4035020 con 0x7ff9e00c4930 2024-03-06T07:36:06.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.000+0000 7ff9df400640 1 -- 172.21.15.97:0/3759401829 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d4031640 con 0x7ff9e00c4930 2024-03-06T07:36:06.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.000+0000 7ff9df400640 1 -- 172.21.15.97:0/3759401829 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d402d040 con 0x7ff9e00c4930 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.000+0000 7ff9e7a00640 1 -- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c4930 msgr2=0x7ff9e00c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.000+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c4930 0x7ff9e00c6460 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009870 tx=0x7ff9d402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.001+0000 7ff9e7a00640 1 -- 172.21.15.97:0/3759401829 shutdown_connections 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.001+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4d10 0x7ff9e00c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.001+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/3759401829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c4930 0x7ff9e00c6460 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.001+0000 7ff9e7a00640 1 -- 172.21.15.97:0/3759401829 >> 172.21.15.97:0/3759401829 conn(0x7ff9e00cc4f0 msgr2=0x7ff9e00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.001+0000 7ff9e7a00640 1 -- 172.21.15.97:0/3759401829 shutdown_connections 2024-03-06T07:36:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.001+0000 7ff9e7a00640 1 -- 172.21.15.97:0/3759401829 wait complete. 2024-03-06T07:36:06.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.002+0000 7ff9e7a00640 1 Processor -- start 2024-03-06T07:36:06.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.003+0000 7ff9e7a00640 1 -- start start 2024-03-06T07:36:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.003+0000 7ff9e7a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4930 0x7ff9e01e0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e7a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 0x7ff9e00c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e7a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c56b0 0x7ff9e01e7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e7a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00cc260 con 0x7ff9e00c4d10 2024-03-06T07:36:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e7a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00ca3b0 con 0x7ff9e00c56b0 2024-03-06T07:36:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e7a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00c5b40 con 0x7ff9e00c4930 2024-03-06T07:36:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 0x7ff9e00c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 0x7ff9e00c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54248/0 (socket says 172.21.15.97:54248) 2024-03-06T07:36:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 -- 172.21.15.97:0/2526929572 learned_addr learned my addr 172.21.15.97:0/2526929572 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 -- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4930 msgr2=0x7ff9e01e0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e5a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c56b0 0x7ff9e01e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e5000640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4930 0x7ff9e01e0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4930 0x7ff9e01e0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 -- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c56b0 msgr2=0x7ff9e01e7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c56b0 0x7ff9e01e7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9dfe00640 1 -- 172.21.15.97:0/2526929572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e01f5260 con 0x7ff9e00c4d10 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.004+0000 7ff9e5a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c56b0 0x7ff9e01e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.005+0000 7ff9dfe00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 0x7ff9e00c8ce0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00ec10 tx=0x7ff9cc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.005+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc017070 con 0x7ff9e00c4d10 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.005+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc00cbd0 con 0x7ff9e00c4d10 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.005+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e01df580 con 0x7ff9e00c4d10 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.005+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc013470 con 0x7ff9e00c4d10 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.005+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e01f57e0 con 0x7ff9e00c4d10 2024-03-06T07:36:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.006+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e004f330 con 0x7ff9e00c4d10 2024-03-06T07:36:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.007+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9cc01a070 con 0x7ff9e00c4d10 2024-03-06T07:36:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.008+0000 7ff9dcc00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9c006b4d0 0x7ff9c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.008+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9cc08a0e0 con 0x7ff9e00c4d10 2024-03-06T07:36:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.008+0000 7ff9e5000640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9c006b4d0 0x7ff9c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.009+0000 7ff9e5000640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9c006b4d0 0x7ff9c006d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4036000 tx=0x7ff9d4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.012+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9cc058f50 con 0x7ff9e00c4d10 2024-03-06T07:36:06.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.282+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7ff9e01443c0 con 0x7ff9e00c4d10 2024-03-06T07:36:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.695+0000 7ff9dcc00640 1 -- 172.21.15.97:0/2526929572 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6172 v93) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff9cc058850 con 0x7ff9e00c4d10 2024-03-06T07:36:06.697 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 6172 2024-03-06T07:36:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.698+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9c006b4d0 msgr2=0x7ff9c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.698+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9c006b4d0 0x7ff9c006d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4036000 tx=0x7ff9d4008030 comp rx=0 tx=0).stop 2024-03-06T07:36:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.698+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 msgr2=0x7ff9e00c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 0x7ff9e00c8ce0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00ec10 tx=0x7ff9cc00c520 comp rx=0 tx=0).stop 2024-03-06T07:36:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 shutdown_connections 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff9c006b4d0 0x7ff9c006d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9e00c56b0 0x7ff9e01e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff9e00c4d10 0x7ff9e00c8ce0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 --2- 172.21.15.97:0/2526929572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9e00c4930 0x7ff9e01e0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 >> 172.21.15.97:0/2526929572 conn(0x7ff9e00cc4f0 msgr2=0x7ff9e01448e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.699+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 shutdown_connections 2024-03-06T07:36:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:06.700+0000 7ff9e7a00640 1 -- 172.21.15.97:0/2526929572 wait complete. 2024-03-06T07:36:06.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:06.719 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:36:06.720 DEBUG:teuthology.orchestra.run.smithi097:> mkdir pin 2024-03-06T07:36:06.720 DEBUG:teuthology.orchestra.run.smithi097:> touch pin/placeholder 2024-03-06T07:36:06.720 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:36:06.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:06.784 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:36:06.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:07.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.039+0000 7f540e800640 1 Processor -- start 2024-03-06T07:36:07.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.040+0000 7f540e800640 1 -- start start 2024-03-06T07:36:07.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.041+0000 7f540e800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f540815fe60 0x7f5408071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.041+0000 7f540e800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540806cef0 con 0x7f540815f9d0 2024-03-06T07:36:07.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.041+0000 7f540e800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540806d1d0 con 0x7f5408144c00 2024-03-06T07:36:07.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.041+0000 7f540e800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540806d4b0 con 0x7f540815fe60 2024-03-06T07:36:07.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.042+0000 7f53ffe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f540815f9d0 0x7f5408070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49452/0 (socket says 172.21.15.97:49452) 2024-03-06T07:36:07.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.042+0000 7f53ffe00640 1 -- 172.21.15.97:0/424749522 learned_addr learned my addr 172.21.15.97:0/424749522 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.042+0000 7f5407e00640 1 --2- 172.21.15.97:0/424749522 >> v2:172.21.15.135:3300/0 conn(0x7f540815fe60 0x7f5408071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.042+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533198271 0 0) 0x7f540806cef0 con 0x7f540815f9d0 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.042+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ec003520 con 0x7f540815f9d0 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257740724 0 0) 0x7f53ec003520 con 0x7f540815f9d0 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 >> v2:172.21.15.135:3300/0 conn(0x7f540815fe60 msgr2=0x7f5408071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 --2- 172.21.15.97:0/424749522 >> v2:172.21.15.135:3300/0 conn(0x7f540815fe60 0x7f5408071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 >> v1:172.21.15.135:6789/0 conn(0x7f5408144c00 legacy=0x7f540806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540806d790 con 0x7f540815f9d0 2024-03-06T07:36:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f53f0001670 con 0x7f540815f9d0 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 >> v1:172.21.15.97:6789/0 conn(0x7f540815f9d0 legacy=0x7f5408070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ec004800 0x7f53ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec0051c0 0x7f53ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540806d4b0 con 0x7f53ec004800 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00c0c0 con 0x7f53ec0051c0 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00c2c0 con 0x7f53ec003a00 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f53ffe00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ec004800 0x7f53ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5407e00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec0051c0 0x7f53ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.043+0000 7f5407400640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.044+0000 7f53ffe00640 1 -- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ec003a00 msgr2=0x7f53ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.044+0000 7f53ffe00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.044+0000 7f53ffe00640 1 -- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec0051c0 msgr2=0x7f53ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.044+0000 7f53ffe00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec0051c0 0x7f53ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.044+0000 7f53ffe00640 1 -- 172.21.15.97:0/424749522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec003520 con 0x7f53ec004800 2024-03-06T07:36:07.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.044+0000 7f5407e00640 1 -- 172.21.15.97:0/424749522 reap_dead start 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.045+0000 7f53ffe00640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ec004800 0x7f53ec004c80 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c190 tx=0x7f53ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01b5a48271e56c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.045+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0007c60 con 0x7f53ec004800 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.045+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f0030070 con 0x7f53ec004800 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.045+0000 7f5406a00640 1 -- 172.21.15.97:0/424749522 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0005d90 con 0x7f53ec004800 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 -- 172.21.15.97:0/424749522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ec004800 msgr2=0x7f53ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ec004800 0x7f53ec004c80 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c190 tx=0x7f53ec005830 comp rx=0 tx=0).stop 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 -- 172.21.15.97:0/424749522 shutdown_connections 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 --2- 172.21.15.97:0/424749522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53ec004800 0x7f53ec004c80 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 -- 172.21.15.97:0/424749522 >> 172.21.15.97:0/424749522 conn(0x7f54080b1730 msgr2=0x7f54080b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 -- 172.21.15.97:0/424749522 shutdown_connections 2024-03-06T07:36:07.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.046+0000 7f540e800640 1 -- 172.21.15.97:0/424749522 wait complete. 2024-03-06T07:36:07.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.049+0000 7f540e800640 1 Processor -- start 2024-03-06T07:36:07.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.049+0000 7f540e800640 1 -- start start 2024-03-06T07:36:07.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.050+0000 7f540e800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec004800 0x7f540815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.050+0000 7f540e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 0x7f540815f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.050+0000 7f540e800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5408141e70 0x7f54081422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.050+0000 7f540e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540806cb50 con 0x7f540815c340 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.050+0000 7f540e800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408160730 con 0x7f53ec004800 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.050+0000 7f540e800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540806ccd0 con 0x7f5408141e70 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 0x7f540815f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 0x7f540815f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54260/0 (socket says 172.21.15.97:54260) 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 -- 172.21.15.97:0/1861746213 learned_addr learned my addr 172.21.15.97:0/1861746213 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f5407400640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec004800 0x7f540815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f5407e00640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5408141e70 0x7f54081422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 -- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5408141e70 msgr2=0x7f54081422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5408141e70 0x7f54081422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 -- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec004800 msgr2=0x7f540815ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec004800 0x7f540815ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.051+0000 7f53ffe00640 1 -- 172.21.15.97:0/1861746213 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540815c8b0 con 0x7f540815c340 2024-03-06T07:36:07.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.052+0000 7f53ffe00640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 0x7f540815f2b0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c190 tx=0x7f53f0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.052+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0045070 con 0x7f540815c340 2024-03-06T07:36:07.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.052+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f00049f0 con 0x7f540815c340 2024-03-06T07:36:07.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.052+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0007c60 con 0x7f540815c340 2024-03-06T07:36:07.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.052+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540815cb90 con 0x7f540815c340 2024-03-06T07:36:07.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.052+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540815d150 con 0x7f540815c340 2024-03-06T07:36:07.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.054+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540804f330 con 0x7f540815c340 2024-03-06T07:36:07.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.054+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53f0007070 con 0x7f540815c340 2024-03-06T07:36:07.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.056+0000 7f53ff400640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53e406b500 0x7f53e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.056+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53f00aae00 con 0x7f540815c340 2024-03-06T07:36:07.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.056+0000 7f5407400640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53e406b500 0x7f53e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.057+0000 7f5407400640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53e406b500 0x7f53e406d9c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f54080686c0 tx=0x7f53f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.062+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53f0079c70 con 0x7f540815c340 2024-03-06T07:36:07.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.338+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5408072890 con 0x7f540815c340 2024-03-06T07:36:07.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.339+0000 7f53ff400640 1 -- 172.21.15.97:0/1861746213 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+3414 (secure 0 0 0) 0x7f53f0079570 con 0x7f540815c340 2024-03-06T07:36:07.340 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:07.341 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:59.487556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:active","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8379,8382]},"id":9}]} 2024-03-06T07:36:07.341 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 93 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.341+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53e406b500 msgr2=0x7f53e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.341+0000 7f540e800640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53e406b500 0x7f53e406d9c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f54080686c0 tx=0x7f53f8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.341+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 msgr2=0x7f540815f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.341+0000 7f540e800640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 0x7f540815f2b0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c190 tx=0x7f53f0001970 comp rx=0 tx=0).stop 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 shutdown_connections 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f53e406b500 0x7f53e406d9c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5408141e70 0x7f54081422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f540815c340 0x7f540815f2b0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 --2- 172.21.15.97:0/1861746213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53ec004800 0x7f540815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 >> 172.21.15.97:0/1861746213 conn(0x7f54080b1730 msgr2=0x7f54080b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 shutdown_connections 2024-03-06T07:36:07.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:07.342+0000 7f540e800640 1 -- 172.21.15.97:0/1861746213 wait complete. 2024-03-06T07:36:07.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:36:07.571 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:36:07.572 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:07.572 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:36:09.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:09.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:09.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:09.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.793+0000 7f94d1400640 1 Processor -- start 2024-03-06T07:36:09.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.793+0000 7f94d1400640 1 -- start start 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94d1400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc144c00 0x7f94cc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94d1400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f94cc15fe30 0x7f94cc0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94d1400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc06ee70 con 0x7f94cc15f9d0 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94d1400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc06cef0 con 0x7f94cc15fe30 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94d1400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc06d1d0 con 0x7f94cc144c00 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94cb200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f94cc15fe30 0x7f94cc0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94c9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f94cc15f9d0 0x7f94cc0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49460/0 (socket says 172.21.15.97:49460) 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.795+0000 7f94c9e00640 1 -- 172.21.15.97:0/3730385793 learned_addr learned my addr 172.21.15.97:0/3730385793 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94ca800640 1 --2- 172.21.15.97:0/3730385793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc144c00 0x7f94cc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94c9400640 1 -- 172.21.15.97:0/3730385793 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590632159 0 0) 0x7f94b4001490 con 0x7f94cc15f9d0 2024-03-06T07:36:09.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94c9400640 1 -- 172.21.15.97:0/3730385793 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac003520 con 0x7f94cc15f9d0 2024-03-06T07:36:09.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94cb200640 1 -- 172.21.15.97:0/3730385793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc144c00 msgr2=0x7f94cc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:09.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94cb200640 1 --2- 172.21.15.97:0/3730385793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc144c00 0x7f94cc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94cb200640 1 -- 172.21.15.97:0/3730385793 >> v1:172.21.15.97:6789/0 conn(0x7f94cc15f9d0 legacy=0x7f94cc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.796+0000 7f94cb200640 1 -- 172.21.15.97:0/3730385793 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc06d4b0 con 0x7f94cc15fe30 2024-03-06T07:36:09.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.797+0000 7f94cb200640 1 --2- 172.21.15.97:0/3730385793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0ce7b0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f94c40097a0 tx=0x7f94c402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7e0a2947a5ad84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94c9400640 1 -- 172.21.15.97:0/3730385793 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4035020 con 0x7f94cc15fe30 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94c9400640 1 -- 172.21.15.97:0/3730385793 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4031810 con 0x7f94cc15fe30 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94c9400640 1 -- 172.21.15.97:0/3730385793 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4031bc0 con 0x7f94cc15fe30 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 -- 172.21.15.97:0/3730385793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 msgr2=0x7f94cc0ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 --2- 172.21.15.97:0/3730385793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0ce7b0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f94c40097a0 tx=0x7f94c402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 -- 172.21.15.97:0/3730385793 shutdown_connections 2024-03-06T07:36:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 --2- 172.21.15.97:0/3730385793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0ce7b0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 --2- 172.21.15.97:0/3730385793 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc144c00 0x7f94cc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 -- 172.21.15.97:0/3730385793 >> 172.21.15.97:0/3730385793 conn(0x7f94cc0b1730 msgr2=0x7f94cc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:09.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 -- 172.21.15.97:0/3730385793 shutdown_connections 2024-03-06T07:36:09.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.798+0000 7f94d1400640 1 -- 172.21.15.97:0/3730385793 wait complete. 2024-03-06T07:36:09.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.800+0000 7f94d1400640 1 Processor -- start 2024-03-06T07:36:09.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.800+0000 7f94d1400640 1 -- start start 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94d1400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94cc144c00 0x7f94cc134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94d1400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc15f9d0 0x7f94cc0c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94d1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94d1400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc160730 con 0x7f94cc15fe30 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94d1400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc160280 con 0x7f94cc144c00 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94d1400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc06cb50 con 0x7f94cc15f9d0 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94cb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94cb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54272/0 (socket says 172.21.15.97:54272) 2024-03-06T07:36:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.801+0000 7f94cb200640 1 -- 172.21.15.97:0/1594081994 learned_addr learned my addr 172.21.15.97:0/1594081994 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94ca800640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94cc144c00 0x7f94cc134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94c9e00640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc15f9d0 0x7f94cc0c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94cb200640 1 -- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc15f9d0 msgr2=0x7f94cc0c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94cb200640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc15f9d0 0x7f94cc0c7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94cb200640 1 -- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94cc144c00 msgr2=0x7f94cc134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94cb200640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94cc144c00 0x7f94cc134300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94cb200640 1 -- 172.21.15.97:0/1594081994 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0c49a0 con 0x7f94cc15fe30 2024-03-06T07:36:09.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94cb200640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0c7750 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f94c402b8f0 tx=0x7f94c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.802+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4041070 con 0x7f94cc15fe30 2024-03-06T07:36:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.803+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4031e40 con 0x7f94cc15fe30 2024-03-06T07:36:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.803+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc15ebd0 con 0x7f94cc15fe30 2024-03-06T07:36:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.803+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4035040 con 0x7f94cc15fe30 2024-03-06T07:36:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.803+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc134f50 con 0x7f94cc15fe30 2024-03-06T07:36:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.804+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc04f330 con 0x7f94cc15fe30 2024-03-06T07:36:09.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.805+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94c4038020 con 0x7f94cc15fe30 2024-03-06T07:36:09.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.806+0000 7f94c2a00640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94a806b500 0x7f94a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.806+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94c40abcb0 con 0x7f94cc15fe30 2024-03-06T07:36:09.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.806+0000 7f94ca800640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94a806b500 0x7f94a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.807+0000 7f94ca800640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94a806b500 0x7f94a806d9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f94cc071d60 tx=0x7f94bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:09.810+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94cc0c49a0 con 0x7f94cc15fe30 2024-03-06T07:36:10.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.047+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94cc072890 con 0x7f94cc15fe30 2024-03-06T07:36:10.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.048+0000 7f94c2a00640 1 -- 172.21.15.97:0/1594081994 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+3414 (secure 0 0 0) 0x7f94c407abc0 con 0x7f94cc15fe30 2024-03-06T07:36:10.049 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:10.049 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:59.487556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:active","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8379,8382]},"id":9}]} 2024-03-06T07:36:10.050 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 93 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.050+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94a806b500 msgr2=0x7f94a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.050+0000 7f94d1400640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94a806b500 0x7f94a806d9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f94cc071d60 tx=0x7f94bc008040 comp rx=0 tx=0).stop 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 msgr2=0x7f94cc0c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0c7750 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f94c402b8f0 tx=0x7f94c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 shutdown_connections 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f94a806b500 0x7f94a806d9c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94cc15fe30 0x7f94cc0c7750 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f94cc15f9d0 0x7f94cc0c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 --2- 172.21.15.97:0/1594081994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94cc144c00 0x7f94cc134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 >> 172.21.15.97:0/1594081994 conn(0x7f94cc0b1730 msgr2=0x7f94cc0ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:10.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 shutdown_connections 2024-03-06T07:36:10.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:10.051+0000 7f94d1400640 1 -- 172.21.15.97:0/1594081994 wait complete. 2024-03-06T07:36:10.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:36:10.280 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:36:10.280 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:10.280 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:36:12.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:12.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.474+0000 7f2413000640 1 Processor -- start 2024-03-06T07:36:12.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.474+0000 7f2413000640 1 -- start start 2024-03-06T07:36:12.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.475+0000 7f2413000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f240c06afc0 0x7f240c160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f2413000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c15e760 0x7f240c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f2413000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c1608c0 con 0x7f240c06b3a0 2024-03-06T07:36:12.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f2413000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c145b00 con 0x7f240c06afc0 2024-03-06T07:36:12.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f2413000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0d02e0 con 0x7f240c15e760 2024-03-06T07:36:12.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f240c06afc0 0x7f240c160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f240c06afc0 0x7f240c160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54276/0 (socket says 172.21.15.97:54276) 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 -- 172.21.15.97:0/3763008745 learned_addr learned my addr 172.21.15.97:0/3763008745 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 -- 172.21.15.97:0/3763008745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c15e760 msgr2=0x7f240c15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 --2- 172.21.15.97:0/3763008745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c15e760 0x7f240c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 -- 172.21.15.97:0/3763008745 >> v1:172.21.15.135:6789/0 conn(0x7f240c06b3a0 legacy=0x7f240c15c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.476+0000 7f240be00640 1 -- 172.21.15.97:0/3763008745 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0cf6d0 con 0x7f240c06afc0 2024-03-06T07:36:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.477+0000 7f240be00640 1 --2- 172.21.15.97:0/3763008745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c06afc0 0x7f240c160380 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f240c062c70 tx=0x7f23fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d13b6ef7e5b17c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f240b400640 1 -- 172.21.15.97:0/3763008745 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc035020 con 0x7f240c06afc0 2024-03-06T07:36:12.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f240b400640 1 -- 172.21.15.97:0/3763008745 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc031640 con 0x7f240c06afc0 2024-03-06T07:36:12.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f240b400640 1 -- 172.21.15.97:0/3763008745 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc02d040 con 0x7f240c06afc0 2024-03-06T07:36:12.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 -- 172.21.15.97:0/3763008745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c06afc0 msgr2=0x7f240c160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 --2- 172.21.15.97:0/3763008745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c06afc0 0x7f240c160380 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f240c062c70 tx=0x7f23fc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 -- 172.21.15.97:0/3763008745 shutdown_connections 2024-03-06T07:36:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 --2- 172.21.15.97:0/3763008745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c15e760 0x7f240c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 --2- 172.21.15.97:0/3763008745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c06afc0 0x7f240c160380 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 -- 172.21.15.97:0/3763008745 >> 172.21.15.97:0/3763008745 conn(0x7f240c0d0b40 msgr2=0x7f240c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.478+0000 7f2413000640 1 -- 172.21.15.97:0/3763008745 shutdown_connections 2024-03-06T07:36:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.479+0000 7f2413000640 1 -- 172.21.15.97:0/3763008745 wait complete. 2024-03-06T07:36:12.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.481+0000 7f2413000640 1 Processor -- start 2024-03-06T07:36:12.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.481+0000 7f2413000640 1 -- start start 2024-03-06T07:36:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2413000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f240c06afc0 0x7f240c140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2413000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c06b3a0 0x7f240c144360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2413000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 0x7f240c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2413000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0cff70 con 0x7f240c15e760 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2413000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0cfc90 con 0x7f240c06afc0 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2413000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c06b820 con 0x7f240c06b3a0 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.482+0000 7f2411000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 0x7f240c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 0x7f240c0a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54284/0 (socket says 172.21.15.97:54284) 2024-03-06T07:36:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 -- 172.21.15.97:0/645136307 learned_addr learned my addr 172.21.15.97:0/645136307 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f240be00640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f240c06afc0 0x7f240c140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2403400640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c06b3a0 0x7f240c144360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 -- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c06b3a0 msgr2=0x7f240c144360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c06b3a0 0x7f240c144360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 -- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f240c06afc0 msgr2=0x7f240c140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f240c06afc0 0x7f240c140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 -- 172.21.15.97:0/645136307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0a9bc0 con 0x7f240c15e760 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.483+0000 7f2411000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 0x7f240c0a9450 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f240400b970 tx=0x7f240400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.484+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404019070 con 0x7f240c15e760 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.484+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240400cb60 con 0x7f240c15e760 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.484+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404014440 con 0x7f240c15e760 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.484+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c0a9ea0 con 0x7f240c15e760 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.484+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c0aa460 con 0x7f240c15e760 2024-03-06T07:36:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.485+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c04f330 con 0x7f240c15e760 2024-03-06T07:36:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.487+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240401c070 con 0x7f240c15e760 2024-03-06T07:36:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.489+0000 7f2408c00640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f23e806b4d0 0x7f23e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.489+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2404089f50 con 0x7f240c15e760 2024-03-06T07:36:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.489+0000 7f240be00640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f23e806b4d0 0x7f23e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.490+0000 7f240be00640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f23e806b4d0 0x7f23e806d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f23fc036000 tx=0x7f23fc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.493+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2404058dc0 con 0x7f240c15e760 2024-03-06T07:36:12.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.717+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f240c06baf0 con 0x7f240c15e760 2024-03-06T07:36:12.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.718+0000 7f2408c00640 1 -- 172.21.15.97:0/645136307 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+3414 (secure 0 0 0) 0x7f24040586c0 con 0x7f240c15e760 2024-03-06T07:36:12.720 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:12.720 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:35:59.487556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:active","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8379,8382]},"id":9}]} 2024-03-06T07:36:12.720 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 93 2024-03-06T07:36:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f23e806b4d0 msgr2=0x7f23e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f23e806b4d0 0x7f23e806d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f23fc036000 tx=0x7f23fc03c040 comp rx=0 tx=0).stop 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 msgr2=0x7f240c0a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 0x7f240c0a9450 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f240400b970 tx=0x7f240400be30 comp rx=0 tx=0).stop 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 shutdown_connections 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f23e806b4d0 0x7f23e806d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f240c15e760 0x7f240c0a9450 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f240c06b3a0 0x7f240c144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 --2- 172.21.15.97:0/645136307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f240c06afc0 0x7f240c140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 >> 172.21.15.97:0/645136307 conn(0x7f240c0d0b40 msgr2=0x7f240c0b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 shutdown_connections 2024-03-06T07:36:12.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:12.720+0000 7f2413000640 1 -- 172.21.15.97:0/645136307 wait complete. 2024-03-06T07:36:12.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:36:12.944 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:36:12.944 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:12.944 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:36:14.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.153+0000 7f9c4b200640 1 Processor -- start 2024-03-06T07:36:15.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.154+0000 7f9c4b200640 1 -- start start 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c4b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9c4415fe60 0x7f9c44071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c4b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4406cef0 con 0x7f9c44144c00 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c4b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4406d1d0 con 0x7f9c4415f9d0 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c4b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4406d4b0 con 0x7f9c4415fe60 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c43e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9c44144c00 0x7f9c4406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39996/0 (socket says 172.21.15.97:39996) 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c43e00640 1 -- 172.21.15.97:0/1626479532 learned_addr learned my addr 172.21.15.97:0/1626479532 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:15.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c49200640 1 --2- 172.21.15.97:0/1626479532 >> v2:172.21.15.135:3300/0 conn(0x7f9c4415fe60 0x7f9c44071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.155+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510128299 0 0) 0x7f9c4406cef0 con 0x7f9c44144c00 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c28003520 con 0x7f9c44144c00 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3369256441 0 0) 0x7f9c28003520 con 0x7f9c44144c00 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 >> v2:172.21.15.135:3300/0 conn(0x7f9c4415fe60 msgr2=0x7f9c44071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 --2- 172.21.15.97:0/1626479532 >> v2:172.21.15.135:3300/0 conn(0x7f9c4415fe60 0x7f9c44071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 >> v1:172.21.15.135:6789/0 conn(0x7f9c4415f9d0 legacy=0x7f9c44070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4406d790 con 0x7f9c44144c00 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9c34001670 con 0x7f9c44144c00 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 >> v1:172.21.15.97:6789/0 conn(0x7f9c44144c00 legacy=0x7f9c4406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c280039b0 0x7f9c28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c28004800 0x7f9c28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c280051c0 0x7f9c2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4406d4b0 con 0x7f9c280039b0 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2800c060 con 0x7f9c280051c0 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2800c260 con 0x7f9c28004800 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.156+0000 7f9c43e00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c280039b0 0x7f9c28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c49200640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c280051c0 0x7f9c2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43400640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c28004800 0x7f9c28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43e00640 1 -- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c28004800 msgr2=0x7f9c28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43e00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c28004800 0x7f9c28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43e00640 1 -- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c280051c0 msgr2=0x7f9c2800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43e00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c280051c0 0x7f9c2800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43e00640 1 -- 172.21.15.97:0/1626479532 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c28003520 con 0x7f9c280039b0 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c49200640 1 -- 172.21.15.97:0/1626479532 reap_dead start 2024-03-06T07:36:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.157+0000 7f9c43400640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c28004800 0x7f9c28004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:15.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.158+0000 7f9c43e00640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c280039b0 0x7f9c28003dd0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c290 tx=0x7f9c28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cddd6fbdd308c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34007d50 con 0x7f9c280039b0 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c34030070 con 0x7f9c280039b0 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c42a00640 1 -- 172.21.15.97:0/1626479532 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34005d10 con 0x7f9c280039b0 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 -- 172.21.15.97:0/1626479532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c280039b0 msgr2=0x7f9c28003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c280039b0 0x7f9c28003dd0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c290 tx=0x7f9c28005830 comp rx=0 tx=0).stop 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 -- 172.21.15.97:0/1626479532 shutdown_connections 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 --2- 172.21.15.97:0/1626479532 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c280039b0 0x7f9c28003dd0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 -- 172.21.15.97:0/1626479532 >> 172.21.15.97:0/1626479532 conn(0x7f9c440b1730 msgr2=0x7f9c440b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 -- 172.21.15.97:0/1626479532 shutdown_connections 2024-03-06T07:36:15.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.159+0000 7f9c4b200640 1 -- 172.21.15.97:0/1626479532 wait complete. 2024-03-06T07:36:15.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.161+0000 7f9c4b200640 1 Processor -- start 2024-03-06T07:36:15.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.161+0000 7f9c4b200640 1 -- start start 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.162+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c280039b0 0x7f9c4415b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.162+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 0x7f9c4415ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.162+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c4415e2b0 0x7f9c4415e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.162+0000 7f9c4b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4406cb50 con 0x7f9c4413f700 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.162+0000 7f9c4b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44160730 con 0x7f9c4415e2b0 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.162+0000 7f9c4b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4406ccd0 con 0x7f9c280039b0 2024-03-06T07:36:15.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 0x7f9c4415ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 0x7f9c4415ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44888/0 (socket says 172.21.15.97:44888) 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 -- 172.21.15.97:0/519717563 learned_addr learned my addr 172.21.15.97:0/519717563 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c49200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c4415e2b0 0x7f9c4415e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43e00640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c280039b0 0x7f9c4415b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 -- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c280039b0 msgr2=0x7f9c4415b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c280039b0 0x7f9c4415b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 -- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c4415e2b0 msgr2=0x7f9c4415e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c4415e2b0 0x7f9c4415e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 -- 172.21.15.97:0/519717563 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4415bb90 con 0x7f9c4413f700 2024-03-06T07:36:15.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.163+0000 7f9c43400640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 0x7f9c4415ca20 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f9c3000d4a0 tx=0x7f9c3000d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.164+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30019070 con 0x7f9c4413f700 2024-03-06T07:36:15.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.164+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c30005480 con 0x7f9c4413f700 2024-03-06T07:36:15.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.164+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4415d250 con 0x7f9c4413f700 2024-03-06T07:36:15.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.164+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30007230 con 0x7f9c4413f700 2024-03-06T07:36:15.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.164+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4415d810 con 0x7f9c4413f700 2024-03-06T07:36:15.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.165+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c441608b0 con 0x7f9c4413f700 2024-03-06T07:36:15.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.167+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c3001c070 con 0x7f9c4413f700 2024-03-06T07:36:15.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.168+0000 7f9c3be00640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c2406b4d0 0x7f9c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.168+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c30089980 con 0x7f9c4413f700 2024-03-06T07:36:15.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.168+0000 7f9c43e00640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c2406b4d0 0x7f9c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.169+0000 7f9c43e00640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c2406b4d0 0x7f9c2406d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9c34034000 tx=0x7f9c34039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.176+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c441608b0 con 0x7f9c4413f700 2024-03-06T07:36:15.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.417+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c4404f330 con 0x7f9c4413f700 2024-03-06T07:36:15.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.417+0000 7f9c3be00640 1 -- 172.21.15.97:0/519717563 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f9c30058760 con 0x7f9c4413f700 2024-03-06T07:36:15.420 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:15.420 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:12.920513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:active","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382,8379]},"id":9}]} 2024-03-06T07:36:15.420 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 94 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.420+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c2406b4d0 msgr2=0x7f9c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.420+0000 7f9c4b200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c2406b4d0 0x7f9c2406d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9c34034000 tx=0x7f9c34039040 comp rx=0 tx=0).stop 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.420+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 msgr2=0x7f9c4415ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.420+0000 7f9c4b200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 0x7f9c4415ca20 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f9c3000d4a0 tx=0x7f9c3000d960 comp rx=0 tx=0).stop 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 shutdown_connections 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9c2406b4d0 0x7f9c2406d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c4415e2b0 0x7f9c4415e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c4413f700 0x7f9c4415ca20 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 --2- 172.21.15.97:0/519717563 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9c280039b0 0x7f9c4415b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 >> 172.21.15.97:0/519717563 conn(0x7f9c440b1730 msgr2=0x7f9c440b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 shutdown_connections 2024-03-06T07:36:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:15.421+0000 7f9c4b200640 1 -- 172.21.15.97:0/519717563 wait complete. 2024-03-06T07:36:15.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:36:15.651 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:36:15.652 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:15.652 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:36:15.652 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:36:15.652 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:15.701 DEBUG:teuthology.orchestra.run.smithi097:> stdin-killer -- python3 -c ' 2024-03-06T07:36:15.701 DEBUG:teuthology.orchestra.run.smithi097:> import fcntl 2024-03-06T07:36:15.701 DEBUG:teuthology.orchestra.run.smithi097:> import sys 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> import time 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> while True: 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> try: 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> if os.read(0, 4096) == b"": 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> break 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> except BlockingIOError: 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> pass 2024-03-06T07:36:15.702 DEBUG:teuthology.orchestra.run.smithi097:> time.sleep(2) 2024-03-06T07:36:15.703 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-06T07:36:15.749 DEBUG:teuthology.orchestra.run.smithi097:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:36:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:36:15.798 INFO:teuthology.orchestra.run.smithi097.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:36:15.892 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:16.798 DEBUG:teuthology.orchestra.run.smithi097:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:36:16.847 INFO:teuthology.orchestra.run.smithi097.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:36:16.847 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-06T07:36:16.847 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 28h/40d Inode: 2199023255552 Links: 2 2024-03-06T07:36:16.847 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi097.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-06 07:36:15.885929693 +0000 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-06 07:36:15.885929693 +0000 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-06 07:36:15.885929693 +0000 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-06 07:36:15.885929693 +0000 2024-03-06T07:36:16.848 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:16.848 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:36:16.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:36:17.163 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:36:17.164 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:36:17.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:36:17.410 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:36:17.410 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:36:17.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:36:17.624 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:36:17.624 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:36:17.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:36:17.855 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:36:17.856 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:36:17.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:36:17.893 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:18.077 INFO:teuthology.orchestra.run.smithi097.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":378,"bytes":65362}},"duration":0.013855864000000001} 2024-03-06T07:36:18.077 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65362, "items": 378 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.013855864, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:36:18.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:36:18.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.334+0000 7fd998c00640 1 Processor -- start 2024-03-06T07:36:18.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.334+0000 7fd998c00640 1 -- start start 2024-03-06T07:36:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.336+0000 7fd998c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 0x7fd994070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.336+0000 7fd998c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 0x7fd994160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.336+0000 7fd998c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940b46c0 con 0x7fd9940ce4a0 2024-03-06T07:36:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.336+0000 7fd998c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9940b49a0 con 0x7fd99415fc10 2024-03-06T07:36:18.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.336+0000 7fd998c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd994072c90 con 0x7fd99415fff0 2024-03-06T07:36:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.337+0000 7fd991e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 0x7fd994070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.337+0000 7fd98be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 0x7fd994160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.337+0000 7fd991e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 0x7fd994070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:56764/0 (socket says 172.21.15.97:56764) 2024-03-06T07:36:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.337+0000 7fd98be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 0x7fd994160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:37496/0 (socket says 172.21.15.97:37496) 2024-03-06T07:36:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.337+0000 7fd991e00640 1 -- 172.21.15.97:0/36285470 learned_addr learned my addr 172.21.15.97:0/36285470 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.337+0000 7fd992800640 1 --1- 172.21.15.97:0/36285470 >> v1:172.21.15.135:6789/0 conn(0x7fd9940ce4a0 0x7fd9940ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:33848/0 (socket says 172.21.15.97:33848) 2024-03-06T07:36:18.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.338+0000 7fd98be00640 1 -- 172.21.15.97:0/36285470 >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 msgr2=0x7fd994070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:18.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.338+0000 7fd98be00640 1 --2- 172.21.15.97:0/36285470 >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 0x7fd994070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.338+0000 7fd98be00640 1 -- 172.21.15.97:0/36285470 >> v1:172.21.15.135:6789/0 conn(0x7fd9940ce4a0 legacy=0x7fd9940ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.338+0000 7fd98be00640 1 -- 172.21.15.97:0/36285470 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994160990 con 0x7fd99415fff0 2024-03-06T07:36:18.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.338+0000 7fd991e00640 1 --2- 172.21.15.97:0/36285470 >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 0x7fd994070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:18.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.339+0000 7fd98be00640 1 --2- 172.21.15.97:0/36285470 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 0x7fd994160450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0097a0 tx=0x7fd97c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70839d26df781dee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.340+0000 7fd991400640 1 -- 172.21.15.97:0/36285470 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c035020 con 0x7fd99415fff0 2024-03-06T07:36:18.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.340+0000 7fd991400640 1 -- 172.21.15.97:0/36285470 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c031830 con 0x7fd99415fff0 2024-03-06T07:36:18.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.340+0000 7fd991400640 1 -- 172.21.15.97:0/36285470 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c031c00 con 0x7fd99415fff0 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.340+0000 7fd998c00640 1 -- 172.21.15.97:0/36285470 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 msgr2=0x7fd994160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.340+0000 7fd998c00640 1 --2- 172.21.15.97:0/36285470 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 0x7fd994160450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0097a0 tx=0x7fd97c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.341+0000 7fd998c00640 1 -- 172.21.15.97:0/36285470 shutdown_connections 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.341+0000 7fd998c00640 1 --2- 172.21.15.97:0/36285470 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd99415fff0 0x7fd994160450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.341+0000 7fd998c00640 1 --2- 172.21.15.97:0/36285470 >> v2:172.21.15.135:3300/0 conn(0x7fd99415fc10 0x7fd994070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.341+0000 7fd998c00640 1 -- 172.21.15.97:0/36285470 >> 172.21.15.97:0/36285470 conn(0x7fd9940b1940 msgr2=0x7fd9940b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.341+0000 7fd998c00640 1 -- 172.21.15.97:0/36285470 shutdown_connections 2024-03-06T07:36:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.341+0000 7fd998c00640 1 -- 172.21.15.97:0/36285470 wait complete. 2024-03-06T07:36:18.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.343+0000 7fd998c00640 1 Processor -- start 2024-03-06T07:36:18.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.344+0000 7fd998c00640 1 -- start start 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.345+0000 7fd998c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9940ce4a0 0x7fd99414e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.345+0000 7fd998c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd99415fc10 0x7fd994156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.345+0000 7fd998c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 0x7fd99409a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.345+0000 7fd998c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd994072a30 con 0x7fd99415fff0 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.345+0000 7fd998c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940728b0 con 0x7fd99415fc10 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.345+0000 7fd998c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99415c2e0 con 0x7fd9940ce4a0 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 0x7fd99409a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 0x7fd99409a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44894/0 (socket says 172.21.15.97:44894) 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 -- 172.21.15.97:0/1442837789 learned_addr learned my addr 172.21.15.97:0/1442837789 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd991e00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9940ce4a0 0x7fd99414e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd98be00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd99415fc10 0x7fd994156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 -- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9940ce4a0 msgr2=0x7fd99414e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9940ce4a0 0x7fd99414e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 -- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd99415fc10 msgr2=0x7fd994156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd99415fc10 0x7fd994156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 -- 172.21.15.97:0/1442837789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994156ba0 con 0x7fd99415fff0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.346+0000 7fd992800640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 0x7fd99409a9f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00c780 tx=0x7fd98c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.347+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c017070 con 0x7fd99415fff0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.347+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c00a640 con 0x7fd99415fff0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.347+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99414f470 con 0x7fd99415fff0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.347+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c00aa30 con 0x7fd99415fff0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.347+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd994156890 con 0x7fd99415fff0 2024-03-06T07:36:18.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.348+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9941596d0 con 0x7fd99415fff0 2024-03-06T07:36:18.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.349+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd98c01a070 con 0x7fd99415fff0 2024-03-06T07:36:18.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.350+0000 7fd98a000640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd97006b4d0 0x7fd97006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.350+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd98c089740 con 0x7fd99415fff0 2024-03-06T07:36:18.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.350+0000 7fd991e00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd97006b4d0 0x7fd97006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.351+0000 7fd991e00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd97006b4d0 0x7fd97006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd984009b70 tx=0x7fd984008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.355+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9941596d0 con 0x7fd99415fff0 2024-03-06T07:36:18.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.648+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fd99415b8f0 con 0x7fd99415fff0 2024-03-06T07:36:18.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.762+0000 7fd98a000640 1 -- 172.21.15.97:0/1442837789 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v95) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd98c057f60 con 0x7fd99415fff0 2024-03-06T07:36:18.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.765+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd97006b4d0 msgr2=0x7fd97006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd97006b4d0 0x7fd97006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd984009b70 tx=0x7fd984008040 comp rx=0 tx=0).stop 2024-03-06T07:36:18.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 msgr2=0x7fd99409a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 0x7fd99409a9f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00c780 tx=0x7fd98c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 shutdown_connections 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd97006b4d0 0x7fd97006d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd99415fff0 0x7fd99409a9f0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd99415fc10 0x7fd994156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 --2- 172.21.15.97:0/1442837789 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9940ce4a0 0x7fd99414e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 >> 172.21.15.97:0/1442837789 conn(0x7fd9940b1940 msgr2=0x7fd9940acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 shutdown_connections 2024-03-06T07:36:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.766+0000 7fd998c00640 1 -- 172.21.15.97:0/1442837789 wait complete. 2024-03-06T07:36:18.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:18.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.981+0000 7f6150a00640 1 Processor -- start 2024-03-06T07:36:18.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.981+0000 7f6150a00640 1 -- start start 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.982+0000 7f6150a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.982+0000 7f6150a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f614c0cc9d0 0x7f614c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.982+0000 7f6150a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c15c240 con 0x7f614c0aebf0 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.982+0000 7f6150a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614c15ce30 con 0x7f614c0cc9d0 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.982+0000 7f6150a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c15c500 con 0x7f614c1561f0 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.983+0000 7f614b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f614c0cc9d0 0x7f614c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.983+0000 7f614a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.983+0000 7f614a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c0ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:37514/0 (socket says 172.21.15.97:37514) 2024-03-06T07:36:18.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.983+0000 7f614a800640 1 -- 172.21.15.97:0/2021343769 learned_addr learned my addr 172.21.15.97:0/2021343769 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:18.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.984+0000 7f614a800640 1 -- 172.21.15.97:0/2021343769 >> v2:172.21.15.135:3300/0 conn(0x7f614c0cc9d0 msgr2=0x7f614c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:18.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.984+0000 7f614a800640 1 --2- 172.21.15.97:0/2021343769 >> v2:172.21.15.135:3300/0 conn(0x7f614c0cc9d0 0x7f614c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.984+0000 7f614a800640 1 -- 172.21.15.97:0/2021343769 >> v1:172.21.15.135:6789/0 conn(0x7f614c0aebf0 legacy=0x7f614c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.984+0000 7f614a800640 1 -- 172.21.15.97:0/2021343769 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c15c7e0 con 0x7f614c1561f0 2024-03-06T07:36:18.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.984+0000 7f614b200640 1 --2- 172.21.15.97:0/2021343769 >> v2:172.21.15.135:3300/0 conn(0x7f614c0cc9d0 0x7f614c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:18.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.985+0000 7f614a800640 1 --2- 172.21.15.97:0/2021343769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c0ae6b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f613c009e50 tx=0x7f613c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b48e73f91688b75f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6149e00640 1 -- 172.21.15.97:0/2021343769 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c02cb00 con 0x7f614c1561f0 2024-03-06T07:36:18.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6149e00640 1 -- 172.21.15.97:0/2021343769 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c02cca0 con 0x7f614c1561f0 2024-03-06T07:36:18.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6149e00640 1 -- 172.21.15.97:0/2021343769 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c034610 con 0x7f614c1561f0 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 -- 172.21.15.97:0/2021343769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 msgr2=0x7f614c0ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 --2- 172.21.15.97:0/2021343769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c0ae6b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f613c009e50 tx=0x7f613c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 -- 172.21.15.97:0/2021343769 shutdown_connections 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 --2- 172.21.15.97:0/2021343769 >> v2:172.21.15.135:3300/0 conn(0x7f614c0cc9d0 0x7f614c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 --2- 172.21.15.97:0/2021343769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c0ae6b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 -- 172.21.15.97:0/2021343769 >> 172.21.15.97:0/2021343769 conn(0x7f614c06afc0 msgr2=0x7f614c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 -- 172.21.15.97:0/2021343769 shutdown_connections 2024-03-06T07:36:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.986+0000 7f6150a00640 1 -- 172.21.15.97:0/2021343769 wait complete. 2024-03-06T07:36:18.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.988+0000 7f6150a00640 1 Processor -- start 2024-03-06T07:36:18.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.988+0000 7f6150a00640 1 -- start start 2024-03-06T07:36:18.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f6150a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 0x7f614c0b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f6150a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f614c0cc9d0 0x7f614c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f6150a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f6150a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c156750 con 0x7f614c0aebf0 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f6150a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1565d0 con 0x7f614c0cc9d0 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f6150a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c15fd70 con 0x7f614c1561f0 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.989+0000 7f614a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 0x7f614c0b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 0x7f614c0b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44904/0 (socket says 172.21.15.97:44904) 2024-03-06T07:36:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 -- 172.21.15.97:0/1187409348 learned_addr learned my addr 172.21.15.97:0/1187409348 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614b200640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f6141e00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f614c0cc9d0 0x7f614c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 -- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 msgr2=0x7f614c15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c15b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 -- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f614c0cc9d0 msgr2=0x7f614c15b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f614c0cc9d0 0x7f614c15b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 -- 172.21.15.97:0/1187409348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c0afb20 con 0x7f614c0aebf0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.990+0000 7f614a800640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 0x7f614c0b0e60 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f613c000f80 tx=0x7f613c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.991+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c043020 con 0x7f614c0aebf0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.991+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c02ce50 con 0x7f614c0aebf0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.991+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c009510 con 0x7f614c0aebf0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.991+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c0afdc0 con 0x7f614c0aebf0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.991+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c0b0380 con 0x7f614c0aebf0 2024-03-06T07:36:18.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.992+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c04f330 con 0x7f614c0aebf0 2024-03-06T07:36:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.993+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f613c046020 con 0x7f614c0aebf0 2024-03-06T07:36:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.994+0000 7f6143400640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f612806b4d0 0x7f612806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.994+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f613c0ac800 con 0x7f614c0aebf0 2024-03-06T07:36:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.994+0000 7f6141e00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f612806b4d0 0x7f612806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.995+0000 7f6141e00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f612806b4d0 0x7f612806d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f614c0b1540 tx=0x7f6134009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:19.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:18.999+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f613c04d030 con 0x7f614c0aebf0 2024-03-06T07:36:19.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.233+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f614c0b2b10 con 0x7f614c0aebf0 2024-03-06T07:36:19.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.233+0000 7f6143400640 1 -- 172.21.15.97:0/1187409348 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f613c07b690 con 0x7f614c0aebf0 2024-03-06T07:36:19.236 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:19.236 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:18.767739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:19.236 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 96 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f612806b4d0 msgr2=0x7f612806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f612806b4d0 0x7f612806d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f614c0b1540 tx=0x7f6134009040 comp rx=0 tx=0).stop 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 msgr2=0x7f614c0b0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 0x7f614c0b0e60 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f613c000f80 tx=0x7f613c004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 shutdown_connections 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f612806b4d0 0x7f612806d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f614c1561f0 0x7f614c15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f614c0cc9d0 0x7f614c15b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 --2- 172.21.15.97:0/1187409348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f614c0aebf0 0x7f614c0b0e60 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 >> 172.21.15.97:0/1187409348 conn(0x7f614c06afc0 msgr2=0x7f614c072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:19.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 shutdown_connections 2024-03-06T07:36:19.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:19.237+0000 7f6150a00640 1 -- 172.21.15.97:0/1187409348 wait complete. 2024-03-06T07:36:19.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:18.767739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:19.460 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:19.896 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:20.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:20.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.679+0000 7fe9bcc00640 1 Processor -- start 2024-03-06T07:36:20.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.679+0000 7fe9bcc00640 1 -- start start 2024-03-06T07:36:20.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9bcc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe9b8144c00 0x7fe9b806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9bcc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe9b815fe60 0x7fe9b80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9bcc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b806ee70 con 0x7fe9b815f9d0 2024-03-06T07:36:20.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9bcc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b806cef0 con 0x7fe9b8144c00 2024-03-06T07:36:20.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9bcc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b806d1d0 con 0x7fe9b815fe60 2024-03-06T07:36:20.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9b5e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe9b8144c00 0x7fe9b806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9ad400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe9b815f9d0 0x7fe9b80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:40010/0 (socket says 172.21.15.97:40010) 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.681+0000 7fe9ad400640 1 -- 172.21.15.97:0/3489885439 learned_addr learned my addr 172.21.15.97:0/3489885439 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.682+0000 7fe9b5e00640 1 -- 172.21.15.97:0/3489885439 >> v2:172.21.15.135:3300/0 conn(0x7fe9b815fe60 msgr2=0x7fe9b80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.682+0000 7fe9b5e00640 1 --2- 172.21.15.97:0/3489885439 >> v2:172.21.15.135:3300/0 conn(0x7fe9b815fe60 0x7fe9b80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.682+0000 7fe9b5e00640 1 -- 172.21.15.97:0/3489885439 >> v1:172.21.15.97:6789/0 conn(0x7fe9b815f9d0 legacy=0x7fe9b80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.682+0000 7fe9b5400640 1 -- 172.21.15.97:0/3489885439 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234282107 0 0) 0x7fe9a0001110 con 0x7fe9b815f9d0 2024-03-06T07:36:20.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.682+0000 7fe9b5e00640 1 -- 172.21.15.97:0/3489885439 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b806d4b0 con 0x7fe9b8144c00 2024-03-06T07:36:20.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.683+0000 7fe9b5e00640 1 --2- 172.21.15.97:0/3489885439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b806e930 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe9b806e500 tx=0x7fe9a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d85162182f0023 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.683+0000 7fe9b5400640 1 -- 172.21.15.97:0/3489885439 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8035020 con 0x7fe9b8144c00 2024-03-06T07:36:20.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.683+0000 7fe9b5400640 1 -- 172.21.15.97:0/3489885439 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a8031830 con 0x7fe9b8144c00 2024-03-06T07:36:20.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.683+0000 7fe9b5400640 1 -- 172.21.15.97:0/3489885439 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8031c00 con 0x7fe9b8144c00 2024-03-06T07:36:20.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 -- 172.21.15.97:0/3489885439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 msgr2=0x7fe9b806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/3489885439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b806e930 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe9b806e500 tx=0x7fe9a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 -- 172.21.15.97:0/3489885439 shutdown_connections 2024-03-06T07:36:20.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/3489885439 >> v2:172.21.15.135:3300/0 conn(0x7fe9b815fe60 0x7fe9b80ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/3489885439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b806e930 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 -- 172.21.15.97:0/3489885439 >> 172.21.15.97:0/3489885439 conn(0x7fe9b80b1730 msgr2=0x7fe9b80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 -- 172.21.15.97:0/3489885439 shutdown_connections 2024-03-06T07:36:20.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.684+0000 7fe9bcc00640 1 -- 172.21.15.97:0/3489885439 wait complete. 2024-03-06T07:36:20.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.687+0000 7fe9bcc00640 1 Processor -- start 2024-03-06T07:36:20.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.687+0000 7fe9bcc00640 1 -- start start 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9bcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b8140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9bcc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9b815f9d0 0x7fe9b813d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9bcc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9b815fe60 0x7fe9b8139170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9bcc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8160730 con 0x7fe9b8144c00 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9bcc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8160270 con 0x7fe9b815fe60 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9bcc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b806cb50 con 0x7fe9b815f9d0 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b8140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b8140060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44918/0 (socket says 172.21.15.97:44918) 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 -- 172.21.15.97:0/2481311978 learned_addr learned my addr 172.21.15.97:0/2481311978 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 -- 172.21.15.97:0/2481311978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9b815f9d0 msgr2=0x7fe9b813d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9b815f9d0 0x7fe9b813d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 -- 172.21.15.97:0/2481311978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9b815fe60 msgr2=0x7fe9b8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9b815fe60 0x7fe9b8139170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.689+0000 7fe9b5e00640 1 -- 172.21.15.97:0/2481311978 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b813ebf0 con 0x7fe9b8144c00 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.690+0000 7fe9b5e00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b8140060 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8000c00 tx=0x7fe9a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.690+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8004230 con 0x7fe9b8144c00 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.690+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a80043d0 con 0x7fe9b8144c00 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.690+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b813e350 con 0x7fe9b8144c00 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.690+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8003850 con 0x7fe9b8144c00 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.690+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b813e910 con 0x7fe9b8144c00 2024-03-06T07:36:20.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.692+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b804f330 con 0x7fe9b8144c00 2024-03-06T07:36:20.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.692+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9a804c060 con 0x7fe9b8144c00 2024-03-06T07:36:20.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.693+0000 7fe9aea00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99406b4d0 0x7fe99406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.693+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9a80abde0 con 0x7fe9b8144c00 2024-03-06T07:36:20.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.693+0000 7fe9ad400640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99406b4d0 0x7fe99406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.694+0000 7fe9ad400640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99406b4d0 0x7fe99406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe9b815bb90 tx=0x7fe9a0001310 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.702+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9a8040070 con 0x7fe9b8144c00 2024-03-06T07:36:20.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.932+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9b806e120 con 0x7fe9b8144c00 2024-03-06T07:36:20.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.933+0000 7fe9aea00640 1 -- 172.21.15.97:0/2481311978 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3412 (secure 0 0 0) 0x7fe9a807abd0 con 0x7fe9b8144c00 2024-03-06T07:36:20.935 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:20.935 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:18.767739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:20.935 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 96 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99406b4d0 msgr2=0x7fe99406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99406b4d0 0x7fe99406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe9b815bb90 tx=0x7fe9a0001310 comp rx=0 tx=0).stop 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 msgr2=0x7fe9b8140060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b8140060 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8000c00 tx=0x7fe9a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 shutdown_connections 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe99406b4d0 0x7fe99406d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9b815fe60 0x7fe9b8139170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.935+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe9b815f9d0 0x7fe9b813d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.936+0000 7fe9bcc00640 1 --2- 172.21.15.97:0/2481311978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9b8144c00 0x7fe9b8140060 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.936+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 >> 172.21.15.97:0/2481311978 conn(0x7fe9b80b1730 msgr2=0x7fe9b8070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.936+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 shutdown_connections 2024-03-06T07:36:20.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:20.936+0000 7fe9bcc00640 1 -- 172.21.15.97:0/2481311978 wait complete. 2024-03-06T07:36:21.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:18.767739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:21.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:21.898 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:22.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:22.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.369+0000 7feb35e00640 1 Processor -- start 2024-03-06T07:36:22.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.369+0000 7feb35e00640 1 -- start start 2024-03-06T07:36:22.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.370+0000 7feb35e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feb30144c00 0x7feb3006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.370+0000 7feb35e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb3015f9d0 0x7feb3015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.370+0000 7feb35e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3006ee70 con 0x7feb300ce0a0 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.370+0000 7feb35e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb30160370 con 0x7feb30144c00 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.370+0000 7feb35e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3006cef0 con 0x7feb3015f9d0 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.371+0000 7feb2ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feb30144c00 0x7feb3006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.371+0000 7feb27e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb3015f9d0 0x7feb3015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.371+0000 7feb2ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7feb30144c00 0x7feb3006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:56828/0 (socket says 172.21.15.97:56828) 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.371+0000 7feb2ea00640 1 -- 172.21.15.97:0/3488672577 learned_addr learned my addr 172.21.15.97:0/3488672577 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:22.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.371+0000 7feb2f400640 1 --1- 172.21.15.97:0/3488672577 >> v1:172.21.15.135:6789/0 conn(0x7feb300ce0a0 0x7feb300ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:33858/0 (socket says 172.21.15.97:33858) 2024-03-06T07:36:22.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.372+0000 7feb2ea00640 1 -- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb3015f9d0 msgr2=0x7feb3015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.372+0000 7feb2ea00640 1 --2- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb3015f9d0 0x7feb3015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.372+0000 7feb2ea00640 1 -- 172.21.15.97:0/3488672577 >> v1:172.21.15.135:6789/0 conn(0x7feb300ce0a0 legacy=0x7feb300ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.372+0000 7feb2ea00640 1 -- 172.21.15.97:0/3488672577 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3006d1d0 con 0x7feb30144c00 2024-03-06T07:36:22.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.372+0000 7feb27e00640 1 --2- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb3015f9d0 0x7feb3015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:22.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.373+0000 7feb2ea00640 1 --2- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb30144c00 0x7feb3006e930 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7feb20009870 tx=0x7feb2002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3160a3d3b9c54b95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.374+0000 7feb2e000640 1 -- 172.21.15.97:0/3488672577 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20034070 con 0x7feb30144c00 2024-03-06T07:36:22.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.374+0000 7feb2e000640 1 -- 172.21.15.97:0/3488672577 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb20030840 con 0x7feb30144c00 2024-03-06T07:36:22.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.374+0000 7feb2e000640 1 -- 172.21.15.97:0/3488672577 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20030c10 con 0x7feb30144c00 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.374+0000 7feb35e00640 1 -- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb30144c00 msgr2=0x7feb3006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.374+0000 7feb35e00640 1 --2- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb30144c00 0x7feb3006e930 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7feb20009870 tx=0x7feb2002b980 comp rx=0 tx=0).stop 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.375+0000 7feb35e00640 1 -- 172.21.15.97:0/3488672577 shutdown_connections 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.375+0000 7feb35e00640 1 --2- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb3015f9d0 0x7feb3015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.375+0000 7feb35e00640 1 --2- 172.21.15.97:0/3488672577 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb30144c00 0x7feb3006e930 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.375+0000 7feb35e00640 1 -- 172.21.15.97:0/3488672577 >> 172.21.15.97:0/3488672577 conn(0x7feb300b1730 msgr2=0x7feb300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.375+0000 7feb35e00640 1 -- 172.21.15.97:0/3488672577 shutdown_connections 2024-03-06T07:36:22.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.375+0000 7feb35e00640 1 -- 172.21.15.97:0/3488672577 wait complete. 2024-03-06T07:36:22.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.376+0000 7feb35e00640 1 Processor -- start 2024-03-06T07:36:22.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.376+0000 7feb35e00640 1 -- start start 2024-03-06T07:36:22.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.377+0000 7feb35e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb300ce0a0 0x7feb301f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.377+0000 7feb35e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb30144c00 0x7feb300cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.377+0000 7feb35e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 0x7feb301f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.377+0000 7feb35e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb30160730 con 0x7feb3015f9d0 2024-03-06T07:36:22.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.377+0000 7feb35e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3006d790 con 0x7feb300ce0a0 2024-03-06T07:36:22.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.377+0000 7feb35e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3006cb50 con 0x7feb30144c00 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 0x7feb301f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 0x7feb301f7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44934/0 (socket says 172.21.15.97:44934) 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 -- 172.21.15.97:0/3290415872 learned_addr learned my addr 172.21.15.97:0/3290415872 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2ea00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb300ce0a0 0x7feb301f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb27e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb30144c00 0x7feb300cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 -- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb30144c00 msgr2=0x7feb300cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb30144c00 0x7feb300cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 -- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb300ce0a0 msgr2=0x7feb301f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb300ce0a0 0x7feb301f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 -- 172.21.15.97:0/3290415872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb301f3f50 con 0x7feb3015f9d0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.378+0000 7feb2f400640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 0x7feb301f7990 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7feb2800c760 tx=0x7feb2800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.379+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28017070 con 0x7feb3015f9d0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.379+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800a640 con 0x7feb3015f9d0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.379+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800a9f0 con 0x7feb3015f9d0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.379+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb301f4230 con 0x7feb3015f9d0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.379+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb301f47f0 con 0x7feb3015f9d0 2024-03-06T07:36:22.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.380+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3004f330 con 0x7feb3015f9d0 2024-03-06T07:36:22.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.381+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb2801a070 con 0x7feb3015f9d0 2024-03-06T07:36:22.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.382+0000 7feb27400640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb0c06b500 0x7feb0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.383+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb280895b0 con 0x7feb3015f9d0 2024-03-06T07:36:22.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.383+0000 7feb2ea00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb0c06b500 0x7feb0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.383+0000 7feb2ea00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb0c06b500 0x7feb0c06d9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7feb20036000 tx=0x7feb2003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.387+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb3006ee70 con 0x7feb3015f9d0 2024-03-06T07:36:22.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.659+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb30072890 con 0x7feb3015f9d0 2024-03-06T07:36:22.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.660+0000 7feb27400640 1 -- 172.21.15.97:0/3290415872 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3412 (secure 0 0 0) 0x7feb28057e70 con 0x7feb3015f9d0 2024-03-06T07:36:22.662 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:22.662 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:18.767739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:22.662 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 96 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb0c06b500 msgr2=0x7feb0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb0c06b500 0x7feb0c06d9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7feb20036000 tx=0x7feb2003b040 comp rx=0 tx=0).stop 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 msgr2=0x7feb301f7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 0x7feb301f7990 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7feb2800c760 tx=0x7feb2800cc20 comp rx=0 tx=0).stop 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 shutdown_connections 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feb0c06b500 0x7feb0c06d9c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb3015f9d0 0x7feb301f7990 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feb30144c00 0x7feb300cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.662+0000 7feb35e00640 1 --2- 172.21.15.97:0/3290415872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb300ce0a0 0x7feb301f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.663+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 >> 172.21.15.97:0/3290415872 conn(0x7feb300b1730 msgr2=0x7feb300ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.663+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 shutdown_connections 2024-03-06T07:36:22.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:22.663+0000 7feb35e00640 1 -- 172.21.15.97:0/3290415872 wait complete. 2024-03-06T07:36:22.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:18.767739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:22.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:23.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:23.900 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.099+0000 7f2804400640 1 Processor -- start 2024-03-06T07:36:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.099+0000 7f2804400640 1 -- start start 2024-03-06T07:36:24.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.100+0000 7f2804400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 0x7f27fc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.101+0000 7f2804400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc06ee70 con 0x7f27fc15f9d0 2024-03-06T07:36:24.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.101+0000 7f2804400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc06cef0 con 0x7f27fc15fe30 2024-03-06T07:36:24.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.101+0000 7f2804400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc06d1d0 con 0x7f27fc144c00 2024-03-06T07:36:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.101+0000 7f2801000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f27fc15f9d0 0x7f27fc0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47842/0 (socket says 172.21.15.97:47842) 2024-03-06T07:36:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.101+0000 7f2801000640 1 -- 172.21.15.97:0/60539860 learned_addr learned my addr 172.21.15.97:0/60539860 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.101+0000 7f2801a00640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 0x7f27fc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562932540 0 0) 0x7f27e8001110 con 0x7f27fc15f9d0 2024-03-06T07:36:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e4003520 con 0x7f27fc15f9d0 2024-03-06T07:36:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2438754346 0 0) 0x7f27e4003520 con 0x7f27fc15f9d0 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 msgr2=0x7f27fc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 0x7f27fc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 >> v1:172.21.15.135:6789/0 conn(0x7f27fc15fe30 legacy=0x7f27fc0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc06d4b0 con 0x7f27fc15f9d0 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f27e8001840 con 0x7f27fc15f9d0 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 >> v1:172.21.15.97:6789/0 conn(0x7f27fc15f9d0 legacy=0x7f27fc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 0x7f27e40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27e4004d40 0x7f27e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc06d1d0 con 0x7f27e4003fb0 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e400bb00 con 0x7f27e4004d40 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.102+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e400bd00 con 0x7f27fc144c00 2024-03-06T07:36:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801000640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801a00640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 0x7f27e40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2802400640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27e4004d40 0x7f27e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801000640 1 -- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 msgr2=0x7f27e40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801000640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc144c00 0x7f27e40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801000640 1 -- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27e4004d40 msgr2=0x7f27e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801000640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27e4004d40 0x7f27e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2801000640 1 -- 172.21.15.97:0/60539860 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e4003520 con 0x7f27e4003fb0 2024-03-06T07:36:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.103+0000 7f2802400640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27e4004d40 0x7f27e40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.104+0000 7f2801000640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27e4004800 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f27e8007fd0 tx=0x7f27e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0dbbf90728ec1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.104+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8007e30 con 0x7f27e4003fb0 2024-03-06T07:36:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.104+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e8030070 con 0x7f27e4003fb0 2024-03-06T07:36:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.104+0000 7f27fbe00640 1 -- 172.21.15.97:0/60539860 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e802cdd0 con 0x7f27e4003fb0 2024-03-06T07:36:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2804400640 1 -- 172.21.15.97:0/60539860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 msgr2=0x7f27e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2804400640 1 --2- 172.21.15.97:0/60539860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27e4004800 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f27e8007fd0 tx=0x7f27e4005830 comp rx=0 tx=0).stop 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2802400640 1 -- 172.21.15.97:0/60539860 reap_dead start 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2804400640 1 -- 172.21.15.97:0/60539860 shutdown_connections 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2804400640 1 -- 172.21.15.97:0/60539860 >> 172.21.15.97:0/60539860 conn(0x7f27fc0b1730 msgr2=0x7f27fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2804400640 1 -- 172.21.15.97:0/60539860 shutdown_connections 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.105+0000 7f2804400640 1 -- 172.21.15.97:0/60539860 wait complete. 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.107+0000 7f2804400640 1 Processor -- start 2024-03-06T07:36:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.107+0000 7f2804400640 1 -- start start 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2804400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27fc13c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2804400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc0b2cc0 0x7f27fc0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2804400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc1e55c0 0x7f27fc1e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2804400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc160730 con 0x7f27e4003fb0 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2804400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc160280 con 0x7f27fc1e55c0 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2804400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc0ceec0 con 0x7f27fc0b2cc0 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2801a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27fc13c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2801a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27fc13c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40018/0 (socket says 172.21.15.97:40018) 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2801a00640 1 -- 172.21.15.97:0/1428800083 learned_addr learned my addr 172.21.15.97:0/1428800083 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.108+0000 7f2801000640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc0b2cc0 0x7f27fc0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801a00640 1 -- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc0b2cc0 msgr2=0x7f27fc0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801a00640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc0b2cc0 0x7f27fc0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2802400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc1e55c0 0x7f27fc1e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801a00640 1 -- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc1e55c0 msgr2=0x7f27fc1e5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801a00640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc1e55c0 0x7f27fc1e5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801a00640 1 -- 172.21.15.97:0/1428800083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0b3a60 con 0x7f27e4003fb0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801000640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc0b2cc0 0x7f27fc0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2801a00640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27fc13c490 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f27f0011600 tx=0x7f27f0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f000f920 con 0x7f27e4003fb0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f000fac0 con 0x7f27e4003fb0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f000fe70 con 0x7f27e4003fb0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc0b3d40 con 0x7f27e4003fb0 2024-03-06T07:36:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.109+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc0b4300 con 0x7f27e4003fb0 2024-03-06T07:36:24.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.110+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc04f330 con 0x7f27e4003fb0 2024-03-06T07:36:24.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.112+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27f0015460 con 0x7f27e4003fb0 2024-03-06T07:36:24.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.112+0000 7f27f9600640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f27dc06b4d0 0x7f27dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:24.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.112+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27f008ac60 con 0x7f27e4003fb0 2024-03-06T07:36:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.113+0000 7f2801000640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f27dc06b4d0 0x7f27dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.113+0000 7f2801000640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f27dc06b4d0 0x7f27dc06d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f27e8034000 tx=0x7f27e8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.117+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27f0019030 con 0x7f27e4003fb0 2024-03-06T07:36:24.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.353+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27fc072890 con 0x7f27e4003fb0 2024-03-06T07:36:24.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.354+0000 7f27f9600640 1 -- 172.21.15.97:0/1428800083 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f27f0059a40 con 0x7f27e4003fb0 2024-03-06T07:36:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:24.356 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:18.767739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:24.357 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 96 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f27dc06b4d0 msgr2=0x7f27dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f27dc06b4d0 0x7f27dc06d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f27e8034000 tx=0x7f27e8039040 comp rx=0 tx=0).stop 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 msgr2=0x7f27fc13c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27fc13c490 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f27f0011600 tx=0x7f27f0011ac0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 shutdown_connections 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f27dc06b4d0 0x7f27dc06d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc1e55c0 0x7f27fc1e5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f27fc0b2cc0 0x7f27fc0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 --2- 172.21.15.97:0/1428800083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27e4003fb0 0x7f27fc13c490 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 >> 172.21.15.97:0/1428800083 conn(0x7f27fc0b1730 msgr2=0x7f27fc142fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:24.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 shutdown_connections 2024-03-06T07:36:24.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:24.357+0000 7f2804400640 1 -- 172.21.15.97:0/1428800083 wait complete. 2024-03-06T07:36:24.578 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:18.767739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:24.578 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:25.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:25.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.855+0000 7f83ad400640 1 Processor -- start 2024-03-06T07:36:25.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.855+0000 7f83ad400640 1 -- start start 2024-03-06T07:36:25.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83ad400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f83a8161580 0x7f83a80d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83ad400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f83a80d1e90 0x7f83a80d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83ad400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d28a0 0x7f83a80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83ad400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a8161960 con 0x7f83a8161580 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83ad400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a80cf6f0 con 0x7f83a80d1e90 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83ad400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80cf9d0 con 0x7f83a80d28a0 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83a7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f83a8161580 0x7f83a80d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83a7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f83a8161580 0x7f83a80d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40024/0 (socket says 172.21.15.97:40024) 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83a7e00640 1 -- 172.21.15.97:0/2095854715 learned_addr learned my addr 172.21.15.97:0/2095854715 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f83aca00640 1 --2- 172.21.15.97:0/2095854715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d28a0 0x7f83a80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.857+0000 7f839f400640 1 --2- 172.21.15.97:0/2095854715 >> v2:172.21.15.135:3300/0 conn(0x7f83a80d1e90 0x7f83a80d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.858+0000 7f83a7e00640 1 -- 172.21.15.97:0/2095854715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d28a0 msgr2=0x7f83a80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.858+0000 7f83a7e00640 1 --2- 172.21.15.97:0/2095854715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d28a0 0x7f83a80d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.858+0000 7f83a7e00640 1 -- 172.21.15.97:0/2095854715 >> v2:172.21.15.135:3300/0 conn(0x7f83a80d1e90 msgr2=0x7f83a80d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:25.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.858+0000 7f83a7e00640 1 --2- 172.21.15.97:0/2095854715 >> v2:172.21.15.135:3300/0 conn(0x7f83a80d1e90 0x7f83a80d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.858+0000 7f83a7e00640 1 -- 172.21.15.97:0/2095854715 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80cfcb0 con 0x7f83a8161580 2024-03-06T07:36:25.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.859+0000 7f83a7e00640 1 --2- 172.21.15.97:0/2095854715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80d1950 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f83a808c9d0 tx=0x7f839802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92c7e168b656542 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.859+0000 7f83a7400640 1 -- 172.21.15.97:0/2095854715 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839802ca70 con 0x7f83a8161580 2024-03-06T07:36:25.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.859+0000 7f83a7400640 1 -- 172.21.15.97:0/2095854715 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839802cc10 con 0x7f83a8161580 2024-03-06T07:36:25.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.859+0000 7f83a7400640 1 -- 172.21.15.97:0/2095854715 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980343f0 con 0x7f83a8161580 2024-03-06T07:36:25.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 -- 172.21.15.97:0/2095854715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 msgr2=0x7f83a80d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 --2- 172.21.15.97:0/2095854715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80d1950 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f83a808c9d0 tx=0x7f839802bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 -- 172.21.15.97:0/2095854715 shutdown_connections 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 --2- 172.21.15.97:0/2095854715 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d28a0 0x7f83a80d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 --2- 172.21.15.97:0/2095854715 >> v2:172.21.15.135:3300/0 conn(0x7f83a80d1e90 0x7f83a80d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 --2- 172.21.15.97:0/2095854715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80d1950 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 -- 172.21.15.97:0/2095854715 >> 172.21.15.97:0/2095854715 conn(0x7f83a80d3d80 msgr2=0x7f83a80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 -- 172.21.15.97:0/2095854715 shutdown_connections 2024-03-06T07:36:25.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.860+0000 7f83ad400640 1 -- 172.21.15.97:0/2095854715 wait complete. 2024-03-06T07:36:25.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.862+0000 7f83ad400640 1 Processor -- start 2024-03-06T07:36:25.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.862+0000 7f83ad400640 1 -- start start 2024-03-06T07:36:25.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83ad400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d1e90 0x7f83a815d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83ad400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a80d28a0 0x7f83a80b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83ad400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83ad400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80cf3a0 con 0x7f83a8161580 2024-03-06T07:36:25.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83ad400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80b39a0 con 0x7f83a80d28a0 2024-03-06T07:36:25.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83ad400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80d0c20 con 0x7f83a80d1e90 2024-03-06T07:36:25.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83aca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83aca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80b6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40038/0 (socket says 172.21.15.97:40038) 2024-03-06T07:36:25.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83aca00640 1 -- 172.21.15.97:0/2858764243 learned_addr learned my addr 172.21.15.97:0/2858764243 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:25.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f839f400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a80d28a0 0x7f83a80b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83a7e00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d1e90 0x7f83a815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.863+0000 7f83aca00640 1 -- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d1e90 msgr2=0x7f83a815d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83aca00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d1e90 0x7f83a815d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83aca00640 1 -- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a80d28a0 msgr2=0x7f83a80b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83aca00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a80d28a0 0x7f83a80b6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83aca00640 1 -- 172.21.15.97:0/2858764243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80c9e10 con 0x7f83a8161580 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83aca00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80b6eb0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f83a000c820 tx=0x7f83a000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0015070 con 0x7f83a8161580 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0004020 con 0x7f83a8161580 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a80c9b30 con 0x7f83a8161580 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0004410 con 0x7f83a8161580 2024-03-06T07:36:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.864+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a80cbbf0 con 0x7f83a8161580 2024-03-06T07:36:25.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.866+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a804f330 con 0x7f83a8161580 2024-03-06T07:36:25.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.866+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83a0007830 con 0x7f83a8161580 2024-03-06T07:36:25.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.868+0000 7f83a4c00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f838406b4d0 0x7f838406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.868+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83a0089f20 con 0x7f83a8161580 2024-03-06T07:36:25.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.868+0000 7f83a7e00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f838406b4d0 0x7f838406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.869+0000 7f83a7e00640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f838406b4d0 0x7f838406d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8398035000 tx=0x7f8398030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:25.872+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83a0058e30 con 0x7f83a8161580 2024-03-06T07:36:25.901 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:26.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.106+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83a80d3b60 con 0x7f83a8161580 2024-03-06T07:36:26.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.107+0000 7f83a4c00640 1 -- 172.21.15.97:0/2858764243 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f83a0058730 con 0x7f83a8161580 2024-03-06T07:36:26.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:18.767739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:26.110 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 96 2024-03-06T07:36:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f838406b4d0 msgr2=0x7f838406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f838406b4d0 0x7f838406d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8398035000 tx=0x7f8398030000 comp rx=0 tx=0).stop 2024-03-06T07:36:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 msgr2=0x7f83a80b6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80b6eb0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f83a000c820 tx=0x7f83a000cce0 comp rx=0 tx=0).stop 2024-03-06T07:36:26.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 shutdown_connections 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f838406b4d0 0x7f838406d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83a8161580 0x7f83a80b6eb0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a80d28a0 0x7f83a80b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 --2- 172.21.15.97:0/2858764243 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f83a80d1e90 0x7f83a815d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 >> 172.21.15.97:0/2858764243 conn(0x7f83a80d3d80 msgr2=0x7f83a80d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 shutdown_connections 2024-03-06T07:36:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:26.110+0000 7f83ad400640 1 -- 172.21.15.97:0/2858764243 wait complete. 2024-03-06T07:36:26.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:18.767739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:26.329 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:27.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:27.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.528+0000 7f9214a00640 1 Processor -- start 2024-03-06T07:36:27.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.528+0000 7f9214a00640 1 -- start start 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.529+0000 7f9214a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9210161580 0x7f92100d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.529+0000 7f9214a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d1e90 0x7f92100d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.529+0000 7f9214a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9210161960 con 0x7f92100d28a0 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.529+0000 7f9214a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100cf6f0 con 0x7f9210161580 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.529+0000 7f9214a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92100cf9d0 con 0x7f92100d1e90 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9210161580 0x7f92100d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9210161580 0x7f92100d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40042/0 (socket says 172.21.15.97:40042) 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 -- 172.21.15.97:0/1712128782 learned_addr learned my addr 172.21.15.97:0/1712128782 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f92100d28a0 0x7f92100d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47856/0 (socket says 172.21.15.97:47856) 2024-03-06T07:36:27.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 -- 172.21.15.97:0/1712128782 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d1e90 msgr2=0x7f92100d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:27.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920ea00640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d1e90 0x7f92100d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d1e90 0x7f92100d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 -- 172.21.15.97:0/1712128782 >> v1:172.21.15.97:6789/0 conn(0x7f92100d28a0 legacy=0x7f92100d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920f400640 1 -- 172.21.15.97:0/1712128782 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100cfcb0 con 0x7f9210161580 2024-03-06T07:36:27.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920e000640 1 -- 172.21.15.97:0/1712128782 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505971849 0 0) 0x7f9210161960 con 0x7f92100d28a0 2024-03-06T07:36:27.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.530+0000 7f920ea00640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d1e90 0x7f92100d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:27.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.531+0000 7f920f400640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9210161580 0x7f92100d1950 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f9200009870 tx=0x7f920002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a076dbd1380b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.532+0000 7f920e000640 1 -- 172.21.15.97:0/1712128782 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200034070 con 0x7f9210161580 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.532+0000 7f920e000640 1 -- 172.21.15.97:0/1712128782 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9200030840 con 0x7f9210161580 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.532+0000 7f920e000640 1 -- 172.21.15.97:0/1712128782 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200030c10 con 0x7f9210161580 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.532+0000 7f9214a00640 1 -- 172.21.15.97:0/1712128782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9210161580 msgr2=0x7f92100d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.532+0000 7f9214a00640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9210161580 0x7f92100d1950 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f9200009870 tx=0x7f920002b980 comp rx=0 tx=0).stop 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.533+0000 7f9214a00640 1 -- 172.21.15.97:0/1712128782 shutdown_connections 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.533+0000 7f9214a00640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d1e90 0x7f92100d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.533+0000 7f9214a00640 1 --2- 172.21.15.97:0/1712128782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9210161580 0x7f92100d1950 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.533+0000 7f9214a00640 1 -- 172.21.15.97:0/1712128782 >> 172.21.15.97:0/1712128782 conn(0x7f92100d3d80 msgr2=0x7f92100d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.533+0000 7f9214a00640 1 -- 172.21.15.97:0/1712128782 shutdown_connections 2024-03-06T07:36:27.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.533+0000 7f9214a00640 1 -- 172.21.15.97:0/1712128782 wait complete. 2024-03-06T07:36:27.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.534+0000 7f9214a00640 1 Processor -- start 2024-03-06T07:36:27.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.535+0000 7f9214a00640 1 -- start start 2024-03-06T07:36:27.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.535+0000 7f9214a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 0x7f92100cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.535+0000 7f9214a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d28a0 0x7f92100b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f9214a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9210161580 0x7f92100b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f9214a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100cf3a0 con 0x7f92100d1e90 2024-03-06T07:36:27.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f9214a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100b39a0 con 0x7f9210161580 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f9214a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92100b4ed0 con 0x7f92100d28a0 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 0x7f92100cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 0x7f92100cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40048/0 (socket says 172.21.15.97:40048) 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 -- 172.21.15.97:0/3020321042 learned_addr learned my addr 172.21.15.97:0/3020321042 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 -- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d28a0 msgr2=0x7f92100b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920ea00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d28a0 0x7f92100b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d28a0 0x7f92100b81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 -- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9210161580 msgr2=0x7f92100b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9210161580 0x7f92100b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.536+0000 7f920f400640 1 -- 172.21.15.97:0/3020321042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100b6e80 con 0x7f92100d1e90 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.537+0000 7f920f400640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 0x7f92100cc130 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f920002b9c0 tx=0x7f92000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.537+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200034040 con 0x7f92100d1e90 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.537+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9200039d90 con 0x7f92100d1e90 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.537+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200043610 con 0x7f92100d1e90 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.537+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92100b7120 con 0x7f92100d1e90 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.537+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92100b76e0 con 0x7f92100d1e90 2024-03-06T07:36:27.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.538+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92100d1cb0 con 0x7f92100d1e90 2024-03-06T07:36:27.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.540+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f920004a060 con 0x7f92100d1e90 2024-03-06T07:36:27.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.541+0000 7f9207400640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91f006b500 0x7f91f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.541+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92000ac400 con 0x7f92100d1e90 2024-03-06T07:36:27.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.541+0000 7f920ea00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91f006b500 0x7f91f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.542+0000 7f920ea00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91f006b500 0x7f91f006d9c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f92101f7700 tx=0x7f91fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.546+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92100d1cb0 con 0x7f92100d1e90 2024-03-06T07:36:27.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.812+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f921004f330 con 0x7f92100d1e90 2024-03-06T07:36:27.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.816+0000 7f9207400640 1 -- 172.21.15.97:0/3020321042 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f920003e070 con 0x7f92100d1e90 2024-03-06T07:36:27.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:27.818 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:18.767739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:27.819 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 96 2024-03-06T07:36:27.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91f006b500 msgr2=0x7f91f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91f006b500 0x7f91f006d9c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f92101f7700 tx=0x7f91fc008040 comp rx=0 tx=0).stop 2024-03-06T07:36:27.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 msgr2=0x7f92100cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 0x7f92100cc130 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f920002b9c0 tx=0x7f92000046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 shutdown_connections 2024-03-06T07:36:27.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91f006b500 0x7f91f006d9c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9210161580 0x7f92100b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f92100d28a0 0x7f92100b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 --2- 172.21.15.97:0/3020321042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92100d1e90 0x7f92100cc130 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 >> 172.21.15.97:0/3020321042 conn(0x7f92100d3d80 msgr2=0x7f921006b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:27.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 shutdown_connections 2024-03-06T07:36:27.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:27.819+0000 7f9214a00640 1 -- 172.21.15.97:0/3020321042 wait complete. 2024-03-06T07:36:27.904 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:28.039 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:18.767739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:28.040 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:29.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:29.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.251+0000 7fd14da00640 1 Processor -- start 2024-03-06T07:36:29.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.252+0000 7fd14da00640 1 -- start start 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14da00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd1481561f0 0x7fd1480ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480aebf0 0x7fd1480cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14da00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd1480cc800 0x7fd1480cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14da00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14815ce30 con 0x7fd1480cc800 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14da00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14815c240 con 0x7fd1481561f0 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14da00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14815c500 con 0x7fd1480aebf0 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.253+0000 7fd14d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd1480cc800 0x7fd1480cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd1480cc800 0x7fd1480cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40062/0 (socket says 172.21.15.97:40062) 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 -- 172.21.15.97:0/2301227200 learned_addr learned my addr 172.21.15.97:0/2301227200 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd147e00640 1 --2- 172.21.15.97:0/2301227200 >> v2:172.21.15.135:3300/0 conn(0x7fd1481561f0 0x7fd1480ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd13f400640 1 --2- 172.21.15.97:0/2301227200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480aebf0 0x7fd1480cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 -- 172.21.15.97:0/2301227200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480aebf0 msgr2=0x7fd1480cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 --2- 172.21.15.97:0/2301227200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480aebf0 0x7fd1480cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 -- 172.21.15.97:0/2301227200 >> v2:172.21.15.135:3300/0 conn(0x7fd1481561f0 msgr2=0x7fd1480ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 --2- 172.21.15.97:0/2301227200 >> v2:172.21.15.135:3300/0 conn(0x7fd1481561f0 0x7fd1480ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.254+0000 7fd14d000640 1 -- 172.21.15.97:0/2301227200 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14815c7e0 con 0x7fd1480cc800 2024-03-06T07:36:29.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.255+0000 7fd14d000640 1 --2- 172.21.15.97:0/2301227200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480cc800 0x7fd1480cf800 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fd1480b4db0 tx=0x7fd14002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c5dfe696c6679c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.255+0000 7fd147400640 1 -- 172.21.15.97:0/2301227200 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14002ca70 con 0x7fd1480cc800 2024-03-06T07:36:29.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.255+0000 7fd147400640 1 -- 172.21.15.97:0/2301227200 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14002cc10 con 0x7fd1480cc800 2024-03-06T07:36:29.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.255+0000 7fd147400640 1 -- 172.21.15.97:0/2301227200 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1400343f0 con 0x7fd1480cc800 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 -- 172.21.15.97:0/2301227200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480cc800 msgr2=0x7fd1480cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 --2- 172.21.15.97:0/2301227200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480cc800 0x7fd1480cf800 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fd1480b4db0 tx=0x7fd14002bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 -- 172.21.15.97:0/2301227200 shutdown_connections 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 --2- 172.21.15.97:0/2301227200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480cc800 0x7fd1480cf800 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 --2- 172.21.15.97:0/2301227200 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480aebf0 0x7fd1480cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 --2- 172.21.15.97:0/2301227200 >> v2:172.21.15.135:3300/0 conn(0x7fd1481561f0 0x7fd1480ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 -- 172.21.15.97:0/2301227200 >> 172.21.15.97:0/2301227200 conn(0x7fd14806afc0 msgr2=0x7fd14806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.256+0000 7fd14da00640 1 -- 172.21.15.97:0/2301227200 shutdown_connections 2024-03-06T07:36:29.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.257+0000 7fd14da00640 1 -- 172.21.15.97:0/2301227200 wait complete. 2024-03-06T07:36:29.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.258+0000 7fd14da00640 1 Processor -- start 2024-03-06T07:36:29.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.258+0000 7fd14da00640 1 -- start start 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 0x7fd1481572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480cc800 0x7fd1480af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14da00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1481561f0 0x7fd1480cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14da00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481565d0 con 0x7fd1480aebf0 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14da00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480ccc90 con 0x7fd1481561f0 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14da00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd148156750 con 0x7fd1480cc800 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd147e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 0x7fd1481572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd147e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 0x7fd1481572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40076/0 (socket says 172.21.15.97:40076) 2024-03-06T07:36:29.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd147e00640 1 -- 172.21.15.97:0/1590570846 learned_addr learned my addr 172.21.15.97:0/1590570846 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd13f400640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480cc800 0x7fd1480af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.260+0000 7fd14d000640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1481561f0 0x7fd1480cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd147e00640 1 -- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480cc800 msgr2=0x7fd1480af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd147e00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480cc800 0x7fd1480af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd147e00640 1 -- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1481561f0 msgr2=0x7fd1480cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd147e00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1481561f0 0x7fd1480cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd147e00640 1 -- 172.21.15.97:0/1590570846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480afdc0 con 0x7fd1480aebf0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd147e00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 0x7fd1481572c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fd13800d800 tx=0x7fd13800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13800b6c0 con 0x7fd1480aebf0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138019070 con 0x7fd1480aebf0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13800ba80 con 0x7fd1480aebf0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1480b00a0 con 0x7fd1480aebf0 2024-03-06T07:36:29.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.261+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1480b0660 con 0x7fd1480aebf0 2024-03-06T07:36:29.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.262+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1481595f0 con 0x7fd1480aebf0 2024-03-06T07:36:29.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.263+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd13800bc20 con 0x7fd1480aebf0 2024-03-06T07:36:29.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.265+0000 7fd144c00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.265+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd13808a820 con 0x7fd1480aebf0 2024-03-06T07:36:29.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.265+0000 7fd13f400640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.266+0000 7fd13f400640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12406b4d0 0x7fd12406d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd130006fd0 tx=0x7fd130008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.269+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd138059530 con 0x7fd1480aebf0 2024-03-06T07:36:29.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.566+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd14804f330 con 0x7fd1480aebf0 2024-03-06T07:36:29.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.567+0000 7fd144c00640 1 -- 172.21.15.97:0/1590570846 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7fd138058e30 con 0x7fd1480aebf0 2024-03-06T07:36:29.569 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:29.569 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:29.569 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12406b4d0 msgr2=0x7fd12406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12406b4d0 0x7fd12406d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd130006fd0 tx=0x7fd130008040 comp rx=0 tx=0).stop 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 msgr2=0x7fd1481572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 0x7fd1481572c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fd13800d800 tx=0x7fd13800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 shutdown_connections 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1481561f0 0x7fd1480cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1480cc800 0x7fd1480af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 --2- 172.21.15.97:0/1590570846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1480aebf0 0x7fd1481572c0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.569+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 >> 172.21.15.97:0/1590570846 conn(0x7fd14806afc0 msgr2=0x7fd14815b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.570+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 shutdown_connections 2024-03-06T07:36:29.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:29.570+0000 7fd14da00640 1 -- 172.21.15.97:0/1590570846 wait complete. 2024-03-06T07:36:29.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:29.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:29.906 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:30.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:30.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.997+0000 7f3943e00640 1 Processor -- start 2024-03-06T07:36:30.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.997+0000 7f3943e00640 1 -- start start 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.998+0000 7f3943e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f393c15e070 0x7f393c066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.998+0000 7f3943e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f393c0cd860 0x7f393c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3943e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c06eb10 con 0x7f393c06b500 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3943e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c06b900 con 0x7f393c15e070 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3943e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c0b5010 con 0x7f393c0cd860 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3941400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f393c15e070 0x7f393c066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3941400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f393c15e070 0x7f393c066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40090/0 (socket says 172.21.15.97:40090) 2024-03-06T07:36:31.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3941400640 1 -- 172.21.15.97:0/406681806 learned_addr learned my addr 172.21.15.97:0/406681806 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:30.999+0000 7f3940a00640 1 --2- 172.21.15.97:0/406681806 >> v2:172.21.15.135:3300/0 conn(0x7f393c0cd860 0x7f393c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.000+0000 7f3941400640 1 -- 172.21.15.97:0/406681806 >> v2:172.21.15.135:3300/0 conn(0x7f393c0cd860 msgr2=0x7f393c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.000+0000 7f3941400640 1 --2- 172.21.15.97:0/406681806 >> v2:172.21.15.135:3300/0 conn(0x7f393c0cd860 0x7f393c06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.000+0000 7f3941400640 1 -- 172.21.15.97:0/406681806 >> v1:172.21.15.97:6789/0 conn(0x7f393c06b500 legacy=0x7f393c161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.000+0000 7f3941400640 1 -- 172.21.15.97:0/406681806 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c0b52f0 con 0x7f393c15e070 2024-03-06T07:36:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.000+0000 7f393be00640 1 -- 172.21.15.97:0/406681806 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766627607 0 0) 0x7f393c06eb10 con 0x7f393c06b500 2024-03-06T07:36:31.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.001+0000 7f3941400640 1 --2- 172.21.15.97:0/406681806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c066d00 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f393c06cad0 tx=0x7f393002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb0be16fa5f1695 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.001+0000 7f393be00640 1 -- 172.21.15.97:0/406681806 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930034070 con 0x7f393c15e070 2024-03-06T07:36:31.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.001+0000 7f393be00640 1 -- 172.21.15.97:0/406681806 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930030860 con 0x7f393c15e070 2024-03-06T07:36:31.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.001+0000 7f393be00640 1 -- 172.21.15.97:0/406681806 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930030c30 con 0x7f393c15e070 2024-03-06T07:36:31.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 -- 172.21.15.97:0/406681806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 msgr2=0x7f393c066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 --2- 172.21.15.97:0/406681806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c066d00 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f393c06cad0 tx=0x7f393002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 -- 172.21.15.97:0/406681806 shutdown_connections 2024-03-06T07:36:31.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 --2- 172.21.15.97:0/406681806 >> v2:172.21.15.135:3300/0 conn(0x7f393c0cd860 0x7f393c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 --2- 172.21.15.97:0/406681806 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c066d00 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 -- 172.21.15.97:0/406681806 >> 172.21.15.97:0/406681806 conn(0x7f393c0b1e10 msgr2=0x7f393c0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:31.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 -- 172.21.15.97:0/406681806 shutdown_connections 2024-03-06T07:36:31.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.002+0000 7f3943e00640 1 -- 172.21.15.97:0/406681806 wait complete. 2024-03-06T07:36:31.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.005+0000 7f3943e00640 1 Processor -- start 2024-03-06T07:36:31.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.005+0000 7f3943e00640 1 -- start start 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3943e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f393c06b500 0x7f393c1f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3943e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0cd860 0x7f393c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3943e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c0d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3943e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0b5b90 con 0x7f393c15e070 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3943e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0b58b0 con 0x7f393c0cd860 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3943e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c15e450 con 0x7f393c06b500 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3941e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c0d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3941e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c0d0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40104/0 (socket says 172.21.15.97:40104) 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3941e00640 1 -- 172.21.15.97:0/2690883128 learned_addr learned my addr 172.21.15.97:0/2690883128 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3940a00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0cd860 0x7f393c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.007+0000 7f3941400640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f393c06b500 0x7f393c1f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3941e00640 1 -- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f393c06b500 msgr2=0x7f393c1f13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3941e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f393c06b500 0x7f393c1f13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3941e00640 1 -- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0cd860 msgr2=0x7f393c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3941e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0cd860 0x7f393c0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3941e00640 1 -- 172.21.15.97:0/2690883128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c1f45a0 con 0x7f393c15e070 2024-03-06T07:36:31.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3940a00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0cd860 0x7f393c0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3941e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c0d0cb0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f393400c780 tx=0x7f393400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934003b70 con 0x7f393c15e070 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3934003d10 con 0x7f393c15e070 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934005270 con 0x7f393c15e070 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c1f4880 con 0x7f393c15e070 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.008+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c1f3e00 con 0x7f393c15e070 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.010+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c15dbd0 con 0x7f393c15e070 2024-03-06T07:36:31.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.010+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3934005410 con 0x7f393c15e070 2024-03-06T07:36:31.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.012+0000 7f3939600640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f391c06b4d0 0x7f391c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.012+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f393408a110 con 0x7f393c15e070 2024-03-06T07:36:31.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.012+0000 7f3941400640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f391c06b4d0 0x7f391c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.013+0000 7f3941400640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f391c06b4d0 0x7f391c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3930036000 tx=0x7f3930008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.016+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f393c15dbd0 con 0x7f393c15e070 2024-03-06T07:36:31.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.412+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f393c072220 con 0x7f393c15e070 2024-03-06T07:36:31.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.413+0000 7f3939600640 1 -- 172.21.15.97:0/2690883128 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f393c072220 con 0x7f393c15e070 2024-03-06T07:36:31.415 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:31.416 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:31.416 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:31.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f391c06b4d0 msgr2=0x7f391c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f391c06b4d0 0x7f391c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3930036000 tx=0x7f3930008030 comp rx=0 tx=0).stop 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 msgr2=0x7f393c0d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c0d0cb0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f393400c780 tx=0x7f393400cb80 comp rx=0 tx=0).stop 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 shutdown_connections 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f391c06b4d0 0x7f391c06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c15e070 0x7f393c0d0cb0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0cd860 0x7f393c0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 --2- 172.21.15.97:0/2690883128 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f393c06b500 0x7f393c1f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 >> 172.21.15.97:0/2690883128 conn(0x7f393c0b1e10 msgr2=0x7f393c0ad4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 shutdown_connections 2024-03-06T07:36:31.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:31.416+0000 7f3943e00640 1 -- 172.21.15.97:0/2690883128 wait complete. 2024-03-06T07:36:31.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:31.638 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:31.908 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:32.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.850+0000 7f6d32000640 1 Processor -- start 2024-03-06T07:36:32.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.850+0000 7f6d32000640 1 -- start start 2024-03-06T07:36:32.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.851+0000 7f6d32000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6d2c0b0300 0x7f6d2c06f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.851+0000 7f6d32000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.851+0000 7f6d32000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c0b6090 con 0x7f6d2c06afc0 2024-03-06T07:36:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.851+0000 7f6d32000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c0d19e0 con 0x7f6d2c0b0300 2024-03-06T07:36:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.851+0000 7f6d32000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0b5db0 con 0x7f6d2c147430 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.852+0000 7f6d2b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6d2c0b0300 0x7f6d2c06f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.852+0000 7f6d2be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6d2c06afc0 0x7f6d2c06d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47870/0 (socket says 172.21.15.97:47870) 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.852+0000 7f6d2b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6d2c0b0300 0x7f6d2c06f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40114/0 (socket says 172.21.15.97:40114) 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.852+0000 7f6d2be00640 1 -- 172.21.15.97:0/3319815484 learned_addr learned my addr 172.21.15.97:0/3319815484 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.852+0000 7f6d23e00640 1 --2- 172.21.15.97:0/3319815484 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.853+0000 7f6d2b400640 1 -- 172.21.15.97:0/3319815484 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 msgr2=0x7f6d2c0d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.853+0000 7f6d2b400640 1 --2- 172.21.15.97:0/3319815484 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0d14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.853+0000 7f6d2b400640 1 -- 172.21.15.97:0/3319815484 >> v1:172.21.15.97:6789/0 conn(0x7f6d2c06afc0 legacy=0x7f6d2c06d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.853+0000 7f6d2b400640 1 -- 172.21.15.97:0/3319815484 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c0d09c0 con 0x7f6d2c0b0300 2024-03-06T07:36:32.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.853+0000 7f6d2aa00640 1 -- 172.21.15.97:0/3319815484 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493019327 0 0) 0x7f6d2c0b6090 con 0x7f6d2c06afc0 2024-03-06T07:36:32.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.854+0000 7f6d2b400640 1 --2- 172.21.15.97:0/3319815484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c06f5e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0097a0 tx=0x7f6d1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1945b4e3fb7b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.854+0000 7f6d2aa00640 1 -- 172.21.15.97:0/3319815484 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c035020 con 0x7f6d2c0b0300 2024-03-06T07:36:32.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.854+0000 7f6d2aa00640 1 -- 172.21.15.97:0/3319815484 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c031830 con 0x7f6d2c0b0300 2024-03-06T07:36:32.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.854+0000 7f6d2aa00640 1 -- 172.21.15.97:0/3319815484 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c031c00 con 0x7f6d2c0b0300 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 -- 172.21.15.97:0/3319815484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c0b0300 msgr2=0x7f6d2c06f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 --2- 172.21.15.97:0/3319815484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c06f5e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0097a0 tx=0x7f6d1c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 -- 172.21.15.97:0/3319815484 shutdown_connections 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 --2- 172.21.15.97:0/3319815484 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0d14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 --2- 172.21.15.97:0/3319815484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c06f5e0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 -- 172.21.15.97:0/3319815484 >> 172.21.15.97:0/3319815484 conn(0x7f6d2c06c0d0 msgr2=0x7f6d2c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 -- 172.21.15.97:0/3319815484 shutdown_connections 2024-03-06T07:36:32.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.855+0000 7f6d32000640 1 -- 172.21.15.97:0/3319815484 wait complete. 2024-03-06T07:36:32.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.857+0000 7f6d32000640 1 Processor -- start 2024-03-06T07:36:32.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.857+0000 7f6d32000640 1 -- start start 2024-03-06T07:36:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d32000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 0x7f6d2c1e9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d32000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c0aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d32000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d32000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0b59d0 con 0x7f6d2c06afc0 2024-03-06T07:36:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d32000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0d0f80 con 0x7f6d2c0b0300 2024-03-06T07:36:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d32000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0b5b50 con 0x7f6d2c147430 2024-03-06T07:36:32.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d2b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 0x7f6d2c1e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d2b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 0x7f6d2c1e9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40130/0 (socket says 172.21.15.97:40130) 2024-03-06T07:36:32.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d2b400640 1 -- 172.21.15.97:0/2992430175 learned_addr learned my addr 172.21.15.97:0/2992430175 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:32.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d2be00640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.859+0000 7f6d23e00640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c0aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d2b400640 1 -- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 msgr2=0x7f6d2c0af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d2b400640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d2b400640 1 -- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d2c0b0300 msgr2=0x7f6d2c0aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d2b400640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c0aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d2b400640 1 -- 172.21.15.97:0/2992430175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c15f4d0 con 0x7f6d2c06afc0 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d2b400640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 0x7f6d2c1e9200 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c02bee0 tx=0x7f6d1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c041070 con 0x7f6d2c06afc0 2024-03-06T07:36:32.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c03b920 con 0x7f6d2c06afc0 2024-03-06T07:36:32.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c15f7b0 con 0x7f6d2c06afc0 2024-03-06T07:36:32.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.860+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c035040 con 0x7f6d2c06afc0 2024-03-06T07:36:32.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.861+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c15fd70 con 0x7f6d2c06afc0 2024-03-06T07:36:32.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.862+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c04f330 con 0x7f6d2c06afc0 2024-03-06T07:36:32.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.863+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d1c04c060 con 0x7f6d2c06afc0 2024-03-06T07:36:32.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.864+0000 7f6d23400640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6d0806b4d0 0x7f6d0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.864+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6d1c0abc60 con 0x7f6d2c06afc0 2024-03-06T07:36:32.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.864+0000 7f6d23e00640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6d0806b4d0 0x7f6d0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.865+0000 7f6d23e00640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6d0806b4d0 0x7f6d0806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:32.868+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d1c07aa10 con 0x7f6d2c06afc0 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.105+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d2c072890 con 0x7f6d2c06afc0 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.106+0000 7f6d23400640 1 -- 172.21.15.97:0/2992430175 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f6d1c07a310 con 0x7f6d2c06afc0 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.108+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6d0806b4d0 msgr2=0x7f6d0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.108+0000 7f6d32000640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6d0806b4d0 0x7f6d0806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14008040 comp rx=0 tx=0).stop 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.108+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 msgr2=0x7f6d2c1e9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 0x7f6d2c1e9200 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c02bee0 tx=0x7f6d1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 shutdown_connections 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6d0806b4d0 0x7f6d0806d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6d2c147430 0x7f6d2c0af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d2c0b0300 0x7f6d2c0aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 --2- 172.21.15.97:0/2992430175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6d2c06afc0 0x7f6d2c1e9200 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 >> 172.21.15.97:0/2992430175 conn(0x7f6d2c06c0d0 msgr2=0x7f6d2c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 shutdown_connections 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:33.109+0000 7f6d32000640 1 -- 172.21.15.97:0/2992430175 wait complete. 2024-03-06T07:36:33.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:33.335 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:33.910 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:34.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:34.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.593+0000 7f685ca00640 1 Processor -- start 2024-03-06T07:36:34.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.593+0000 7f685ca00640 1 -- start start 2024-03-06T07:36:34.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.595+0000 7f685ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 0x7f6858164e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.595+0000 7f685ca00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685806c940 con 0x7f685815a4b0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.595+0000 7f685ca00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6858071980 con 0x7f6858071380 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.595+0000 7f685ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68580ce680 con 0x7f68580b44a0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.595+0000 7f6857400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f685815a4b0 0x7f68580cbb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34980/0 (socket says 172.21.15.97:34980) 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.595+0000 7f6857400640 1 -- 172.21.15.97:0/2252970993 learned_addr learned my addr 172.21.15.97:0/2252970993 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016372956 0 0) 0x7f6848001110 con 0x7f685815a4b0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6857e00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 0x7f6858164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f683c005730 con 0x7f685815a4b0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385295394 0 0) 0x7f683c005730 con 0x7f685815a4b0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 msgr2=0x7f6858164e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 0x7f6858164e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 >> v1:172.21.15.135:6789/0 conn(0x7f6858071380 legacy=0x7f68580b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6858071c60 con 0x7f685815a4b0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f6848001840 con 0x7f685815a4b0 2024-03-06T07:36:34.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 >> v1:172.21.15.97:6789/0 conn(0x7f685815a4b0 legacy=0x7f68580cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 0x7f683c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006220 0x7f683c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.596+0000 7f6856a00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f683c006fa0 0x7f683c007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580ce680 con 0x7f683c006fa0 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c00deb0 con 0x7f683c006220 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683c00e0b0 con 0x7f68580b44a0 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857e00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f683c006fa0 0x7f683c007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f684fe00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006220 0x7f683c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857400640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 0x7f683c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857e00640 1 -- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 msgr2=0x7f683c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857e00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b44a0 0x7f683c005bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857e00640 1 -- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006220 msgr2=0x7f683c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857e00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006220 0x7f683c006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f6857e00640 1 -- 172.21.15.97:0/2252970993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c005730 con 0x7f683c006fa0 2024-03-06T07:36:34.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.597+0000 7f684fe00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006220 0x7f683c006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:34.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.598+0000 7f6857e00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f683c006fa0 0x7f683c007420 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f6850007330 tx=0x7f683c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4acf59cbbb11ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850039070 con 0x7f683c006fa0 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685002cca0 con 0x7f683c006fa0 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f6856a00640 1 -- 172.21.15.97:0/2252970993 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850009be0 con 0x7f683c006fa0 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f685ca00640 1 -- 172.21.15.97:0/2252970993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f683c006fa0 msgr2=0x7f683c007420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f685ca00640 1 --2- 172.21.15.97:0/2252970993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f683c006fa0 0x7f683c007420 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f6850007330 tx=0x7f683c007a40 comp rx=0 tx=0).stop 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f6857e00640 1 -- 172.21.15.97:0/2252970993 reap_dead start 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f685ca00640 1 -- 172.21.15.97:0/2252970993 shutdown_connections 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f685ca00640 1 -- 172.21.15.97:0/2252970993 >> 172.21.15.97:0/2252970993 conn(0x7f685806afc0 msgr2=0x7f685806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f685ca00640 1 -- 172.21.15.97:0/2252970993 shutdown_connections 2024-03-06T07:36:34.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.599+0000 7f685ca00640 1 -- 172.21.15.97:0/2252970993 wait complete. 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.601+0000 7f685ca00640 1 Processor -- start 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.601+0000 7f685ca00640 1 -- start start 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f685ca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006fa0 0x7f6858159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f685ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 0x7f68580add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f685ca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6858146f90 0x7f6858147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f685ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580d00e0 con 0x7f6858151560 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f685ca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580cec40 con 0x7f683c006fa0 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f685ca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68580cc330 con 0x7f6858146f90 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f684fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 0x7f68580add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f684fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 0x7f68580add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46048/0 (socket says 172.21.15.97:46048) 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.602+0000 7f684fe00640 1 -- 172.21.15.97:0/3680267745 learned_addr learned my addr 172.21.15.97:0/3680267745 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:34.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f6857400640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006fa0 0x7f6858159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f6857e00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6858146f90 0x7f6858147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684fe00640 1 -- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6858146f90 msgr2=0x7f6858147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684fe00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6858146f90 0x7f6858147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684fe00640 1 -- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006fa0 msgr2=0x7f6858159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684fe00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006fa0 0x7f6858159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684fe00640 1 -- 172.21.15.97:0/3680267745 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68581519e0 con 0x7f6858151560 2024-03-06T07:36:34.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f6857400640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006fa0 0x7f6858159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684fe00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 0x7f68580add70 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f684000d800 tx=0x7f684000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840005270 con 0x7f6858151560 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6840017070 con 0x7f6858151560 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684000b6b0 con 0x7f6858151560 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.603+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6858151cb0 con 0x7f6858151560 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.604+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6858152270 con 0x7f6858151560 2024-03-06T07:36:34.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.605+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685804f330 con 0x7f6858151560 2024-03-06T07:36:34.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.606+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f684000b850 con 0x7f6858151560 2024-03-06T07:36:34.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.606+0000 7f684f400640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683406b500 0x7f683406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.607+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6840089cf0 con 0x7f6858151560 2024-03-06T07:36:34.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.607+0000 7f6857400640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683406b500 0x7f683406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.608+0000 7f6857400640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683406b500 0x7f683406d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f68480012b0 tx=0x7f6848009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.611+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f684008e070 con 0x7f6858151560 2024-03-06T07:36:34.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.851+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68580690e0 con 0x7f6858151560 2024-03-06T07:36:34.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.852+0000 7f684f400640 1 -- 172.21.15.97:0/3680267745 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f68400102d0 con 0x7f6858151560 2024-03-06T07:36:34.854 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:34.854 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:34.855 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683406b500 msgr2=0x7f683406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683406b500 0x7f683406d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f68480012b0 tx=0x7f6848009040 comp rx=0 tx=0).stop 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 msgr2=0x7f68580add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 0x7f68580add70 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f684000d800 tx=0x7f684000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 shutdown_connections 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683406b500 0x7f683406d9c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6858146f90 0x7f6858147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6858151560 0x7f68580add70 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.854+0000 7f685ca00640 1 --2- 172.21.15.97:0/3680267745 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f683c006fa0 0x7f6858159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.855+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 >> 172.21.15.97:0/3680267745 conn(0x7f685806afc0 msgr2=0x7f68580b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.855+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 shutdown_connections 2024-03-06T07:36:34.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:34.855+0000 7f685ca00640 1 -- 172.21.15.97:0/3680267745 wait complete. 2024-03-06T07:36:35.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:35.078 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:35.912 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:36.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:36.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.288+0000 7fac4f200640 1 Processor -- start 2024-03-06T07:36:36.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.288+0000 7fac4f200640 1 -- start start 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.290+0000 7fac4f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 0x7fac4806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.290+0000 7fac4f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4815b090 con 0x7fac4815acb0 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.290+0000 7fac4f200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480b2a30 con 0x7fac48072340 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.290+0000 7fac4f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac480b4360 con 0x7fac480b4680 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.290+0000 7fac47e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fac4815acb0 0x7fac480b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34988/0 (socket says 172.21.15.97:34988) 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac47e00640 1 -- 172.21.15.97:0/50513278 learned_addr learned my addr 172.21.15.97:0/50513278 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916490160 0 0) 0x7fac4815b090 con 0x7fac4815acb0 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac47400640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 0x7fac4806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2c003520 con 0x7fac4815acb0 2024-03-06T07:36:36.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1875940580 0 0) 0x7fac2c003520 con 0x7fac4815acb0 2024-03-06T07:36:36.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 msgr2=0x7fac4806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 0x7fac4806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 >> v1:172.21.15.135:6789/0 conn(0x7fac48072340 legacy=0x7fac480ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480b2cf0 con 0x7fac4815acb0 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.291+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fac38001670 con 0x7fac4815acb0 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 >> v1:172.21.15.97:6789/0 conn(0x7fac4815acb0 legacy=0x7fac480b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 0x7fac2c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480b4360 con 0x7fac2c003fb0 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c00bb00 con 0x7fac2c004d40 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c00df10 con 0x7fac480b4680 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47400640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47e00640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 0x7fac2c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47400640 1 -- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 msgr2=0x7fac2c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47400640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac480b4680 0x7fac2c0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac4d200640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47400640 1 -- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac2c004d40 msgr2=0x7fac2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47400640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac47400640 1 -- 172.21.15.97:0/50513278 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c003520 con 0x7fac2c003fb0 2024-03-06T07:36:36.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.292+0000 7fac4d200640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac47400640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac2c004800 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fac34009ac0 tx=0x7fac2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d470b325944363bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34039070 con 0x7fac2c003fb0 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3402c600 con 0x7fac2c003fb0 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac46a00640 1 -- 172.21.15.97:0/50513278 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34009690 con 0x7fac2c003fb0 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac4f200640 1 -- 172.21.15.97:0/50513278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 msgr2=0x7fac2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac4f200640 1 --2- 172.21.15.97:0/50513278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac2c004800 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fac34009ac0 tx=0x7fac2c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac4d200640 1 -- 172.21.15.97:0/50513278 reap_dead start 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac4f200640 1 -- 172.21.15.97:0/50513278 shutdown_connections 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.294+0000 7fac4f200640 1 -- 172.21.15.97:0/50513278 >> 172.21.15.97:0/50513278 conn(0x7fac480b1730 msgr2=0x7fac480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.295+0000 7fac4f200640 1 -- 172.21.15.97:0/50513278 shutdown_connections 2024-03-06T07:36:36.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.295+0000 7fac4f200640 1 -- 172.21.15.97:0/50513278 wait complete. 2024-03-06T07:36:36.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.296+0000 7fac4f200640 1 Processor -- start 2024-03-06T07:36:36.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.296+0000 7fac4f200640 1 -- start start 2024-03-06T07:36:36.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac4f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac4814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac4f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac480ad860 0x7fac48159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac4f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac48159a90 0x7fac480affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac4f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480b4010 con 0x7fac2c003fb0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac4f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480b32b0 con 0x7fac480ad860 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac4f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac480cd6c0 con 0x7fac48159a90 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac4814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac47e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac4814a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46066/0 (socket says 172.21.15.97:46066) 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.298+0000 7fac47e00640 1 -- 172.21.15.97:0/1388818099 learned_addr learned my addr 172.21.15.97:0/1388818099 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47400640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac480ad860 0x7fac48159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47e00640 1 -- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac48159a90 msgr2=0x7fac480affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47e00640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac48159a90 0x7fac480affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47e00640 1 -- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac480ad860 msgr2=0x7fac48159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47e00640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac480ad860 0x7fac48159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47e00640 1 -- 172.21.15.97:0/1388818099 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4815c060 con 0x7fac2c003fb0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac47e00640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac4814a9b0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fac38001eb0 tx=0x7fac3800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38019070 con 0x7fac2c003fb0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac380074e0 con 0x7fac2c003fb0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac380049e0 con 0x7fac2c003fb0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac480b0c60 con 0x7fac2c003fb0 2024-03-06T07:36:36.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.299+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac480b1220 con 0x7fac2c003fb0 2024-03-06T07:36:36.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.300+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac480b1460 con 0x7fac2c003fb0 2024-03-06T07:36:36.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.301+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac3801c070 con 0x7fac2c003fb0 2024-03-06T07:36:36.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.302+0000 7fac3fe00640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.302+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac38089e70 con 0x7fac2c003fb0 2024-03-06T07:36:36.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.303+0000 7fac47400640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.303+0000 7fac47400640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fac34036000 tx=0x7fac34030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.311+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac38058c10 con 0x7fac2c003fb0 2024-03-06T07:36:36.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.596+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac4804f330 con 0x7fac2c003fb0 2024-03-06T07:36:36.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.597+0000 7fac3fe00640 1 -- 172.21.15.97:0/1388818099 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7fac38058510 con 0x7fac2c003fb0 2024-03-06T07:36:36.599 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:36.599 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:36.600 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 msgr2=0x7fac2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fac34036000 tx=0x7fac34030000 comp rx=0 tx=0).stop 2024-03-06T07:36:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 msgr2=0x7fac4814a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac4814a9b0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fac38001eb0 tx=0x7fac3800dc70 comp rx=0 tx=0).stop 2024-03-06T07:36:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 shutdown_connections 2024-03-06T07:36:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fac48159a90 0x7fac480affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac480ad860 0x7fac48159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 --2- 172.21.15.97:0/1388818099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac2c003fb0 0x7fac4814a9b0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 >> 172.21.15.97:0/1388818099 conn(0x7fac480b1730 msgr2=0x7fac4815e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:36.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 shutdown_connections 2024-03-06T07:36:36.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:36.600+0000 7fac4f200640 1 -- 172.21.15.97:0/1388818099 wait complete. 2024-03-06T07:36:36.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:36.830 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:37.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:37.914 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:38.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.038+0000 7f8381c00640 1 Processor -- start 2024-03-06T07:36:38.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.038+0000 7f8381c00640 1 -- start start 2024-03-06T07:36:38.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.039+0000 7f8381c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f837c15f9d0 0x7f837c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.039+0000 7f8381c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837c06cef0 con 0x7f837c144c00 2024-03-06T07:36:38.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.039+0000 7f8381c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837c06d1d0 con 0x7f837c070b10 2024-03-06T07:36:38.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.039+0000 7f8381c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837c06d4b0 con 0x7f837c15f9d0 2024-03-06T07:36:38.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f8372000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f837c15f9d0 0x7f837c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f837aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f837c144c00 0x7f837c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34992/0 (socket says 172.21.15.97:34992) 2024-03-06T07:36:38.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f837aa00640 1 -- 172.21.15.97:0/1419759659 learned_addr learned my addr 172.21.15.97:0/1419759659 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f837a000640 1 -- 172.21.15.97:0/1419759659 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197848926 0 0) 0x7f837c06cef0 con 0x7f837c144c00 2024-03-06T07:36:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f837a000640 1 -- 172.21.15.97:0/1419759659 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8360003520 con 0x7f837c144c00 2024-03-06T07:36:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f8372000640 1 -- 172.21.15.97:0/1419759659 >> v1:172.21.15.135:6789/0 conn(0x7f837c070b10 legacy=0x7f837c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f8372000640 1 -- 172.21.15.97:0/1419759659 >> v1:172.21.15.97:6789/0 conn(0x7f837c144c00 legacy=0x7f837c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.040+0000 7f8372000640 1 -- 172.21.15.97:0/1419759659 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c06d790 con 0x7f837c15f9d0 2024-03-06T07:36:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.041+0000 7f8372000640 1 --2- 172.21.15.97:0/1419759659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c15f9d0 0x7f837c15fe60 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f83640097a0 tx=0x7f836402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b298c237ccc773 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f837a000640 1 -- 172.21.15.97:0/1419759659 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364035020 con 0x7f837c15f9d0 2024-03-06T07:36:38.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f837a000640 1 -- 172.21.15.97:0/1419759659 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8364031830 con 0x7f837c15f9d0 2024-03-06T07:36:38.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f837a000640 1 -- 172.21.15.97:0/1419759659 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364031c00 con 0x7f837c15f9d0 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 -- 172.21.15.97:0/1419759659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c15f9d0 msgr2=0x7f837c15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 --2- 172.21.15.97:0/1419759659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c15f9d0 0x7f837c15fe60 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f83640097a0 tx=0x7f836402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 -- 172.21.15.97:0/1419759659 shutdown_connections 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 --2- 172.21.15.97:0/1419759659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c15f9d0 0x7f837c15fe60 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 -- 172.21.15.97:0/1419759659 >> 172.21.15.97:0/1419759659 conn(0x7f837c0b1730 msgr2=0x7f837c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 -- 172.21.15.97:0/1419759659 shutdown_connections 2024-03-06T07:36:38.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.042+0000 7f8381c00640 1 -- 172.21.15.97:0/1419759659 wait complete. 2024-03-06T07:36:38.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.044+0000 7f8381c00640 1 Processor -- start 2024-03-06T07:36:38.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.044+0000 7f8381c00640 1 -- start start 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8381c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f837c070b10 0x7f837c1557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8381c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 0x7f837c15a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8381c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f837c15f9d0 0x7f837c15af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8381c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c06ccd0 con 0x7f837c144c00 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8381c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c06cb50 con 0x7f837c070b10 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8381c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c06df90 con 0x7f837c15f9d0 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8372000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 0x7f837c15a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8372000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 0x7f837c15a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46078/0 (socket says 172.21.15.97:46078) 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.045+0000 7f8372000640 1 -- 172.21.15.97:0/1538396524 learned_addr learned my addr 172.21.15.97:0/1538396524 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f837aa00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f837c070b10 0x7f837c1557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8372000640 1 -- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f837c15f9d0 msgr2=0x7f837c15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8372000640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f837c15f9d0 0x7f837c15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8372000640 1 -- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f837c070b10 msgr2=0x7f837c1557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8372000640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f837c070b10 0x7f837c1557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8372000640 1 -- 172.21.15.97:0/1538396524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c157da0 con 0x7f837c144c00 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8372000640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 0x7f837c15a9d0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f8364000c00 tx=0x7f83640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364040070 con 0x7f837c144c00 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836403a930 con 0x7f837c144c00 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c158080 con 0x7f837c144c00 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.046+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364035040 con 0x7f837c144c00 2024-03-06T07:36:38.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.047+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c13d640 con 0x7f837c144c00 2024-03-06T07:36:38.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.048+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c04f330 con 0x7f837c144c00 2024-03-06T07:36:38.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.049+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f836404b060 con 0x7f837c144c00 2024-03-06T07:36:38.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.049+0000 7f8373400640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f835806b500 0x7f835806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.050+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f83640abac0 con 0x7f837c144c00 2024-03-06T07:36:38.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.050+0000 7f837aa00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f835806b500 0x7f835806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.051+0000 7f837aa00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f835806b500 0x7f835806d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f836c0012e0 tx=0x7f836c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.055+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f836402f2e0 con 0x7f837c144c00 2024-03-06T07:36:38.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.288+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f837c072890 con 0x7f837c144c00 2024-03-06T07:36:38.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.289+0000 7f8373400640 1 -- 172.21.15.97:0/1538396524 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f8364031800 con 0x7f837c144c00 2024-03-06T07:36:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:38.292 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:38.292 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:38.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f835806b500 msgr2=0x7f835806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f835806b500 0x7f835806d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f836c0012e0 tx=0x7f836c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 msgr2=0x7f837c15a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 0x7f837c15a9d0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f8364000c00 tx=0x7f83640046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 shutdown_connections 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f835806b500 0x7f835806d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f837c15f9d0 0x7f837c15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f837c144c00 0x7f837c15a9d0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 --2- 172.21.15.97:0/1538396524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f837c070b10 0x7f837c1557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 >> 172.21.15.97:0/1538396524 conn(0x7f837c0b1730 msgr2=0x7f837c145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 shutdown_connections 2024-03-06T07:36:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:38.292+0000 7f8381c00640 1 -- 172.21.15.97:0/1538396524 wait complete. 2024-03-06T07:36:38.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:38.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:39.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:39.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:39.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:39.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.729+0000 7f14a9600640 1 Processor -- start 2024-03-06T07:36:39.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.730+0000 7f14a9600640 1 -- start start 2024-03-06T07:36:39.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.731+0000 7f14a9600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f14a415e0d0 0x7f14a40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.731+0000 7f14a9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14a4160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.731+0000 7f14a9600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a40cea60 con 0x7f14a415f470 2024-03-06T07:36:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.731+0000 7f14a9600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a40d2f60 con 0x7f14a415e0d0 2024-03-06T07:36:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.731+0000 7f14a9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40d2350 con 0x7f14a40d0fc0 2024-03-06T07:36:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.731+0000 7f14a2000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f14a415f470 0x7f14a40d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35000/0 (socket says 172.21.15.97:35000) 2024-03-06T07:36:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.732+0000 7f14a2000640 1 -- 172.21.15.97:0/2991443064 learned_addr learned my addr 172.21.15.97:0/2991443064 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.732+0000 7f14a3400640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14a4160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.732+0000 7f14a2a00640 1 --2- 172.21.15.97:0/2991443064 >> v2:172.21.15.135:3300/0 conn(0x7f14a415e0d0 0x7f14a40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.732+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103460824 0 0) 0x7f14a40cea60 con 0x7f14a415f470 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.732+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1488003520 con 0x7f14a415f470 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469805439 0 0) 0x7f1488003520 con 0x7f14a415f470 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 msgr2=0x7f14a4160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14a4160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 >> v2:172.21.15.135:3300/0 conn(0x7f14a415e0d0 msgr2=0x7f14a40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 --2- 172.21.15.97:0/2991443064 >> v2:172.21.15.135:3300/0 conn(0x7f14a415e0d0 0x7f14a40d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40d2630 con 0x7f14a415f470 2024-03-06T07:36:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f148c001670 con 0x7f14a415f470 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 >> v1:172.21.15.97:6789/0 conn(0x7f14a415f470 legacy=0x7f14a40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14880039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1488003f80 0x7f1488004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40d2350 con 0x7f1488004d40 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148800b9c0 con 0x7f1488003f80 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148800bbc0 con 0x7f14a40d0fc0 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a3400640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a2a00640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14880039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.733+0000 7f14a2000640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1488003f80 0x7f1488004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.734+0000 7f14a3400640 1 -- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 msgr2=0x7f14880039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.734+0000 7f14a3400640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14880039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.734+0000 7f14a3400640 1 -- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1488003f80 msgr2=0x7f1488004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.734+0000 7f14a3400640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1488003f80 0x7f1488004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.734+0000 7f14a3400640 1 -- 172.21.15.97:0/2991443064 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1488003520 con 0x7f1488004d40 2024-03-06T07:36:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.734+0000 7f14a2a00640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a40d0fc0 0x7f14880039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.735+0000 7f14a3400640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14880051c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f149c009a90 tx=0x7f1488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1145be0bae9c6ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.735+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c039070 con 0x7f1488004d40 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.735+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c02c680 con 0x7f1488004d40 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.735+0000 7f14a1600640 1 -- 172.21.15.97:0/2991443064 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c009690 con 0x7f1488004d40 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a9600640 1 -- 172.21.15.97:0/2991443064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 msgr2=0x7f14880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a9600640 1 --2- 172.21.15.97:0/2991443064 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14880051c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f149c009a90 tx=0x7f1488005830 comp rx=0 tx=0).stop 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a3400640 1 -- 172.21.15.97:0/2991443064 reap_dead start 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a9600640 1 -- 172.21.15.97:0/2991443064 shutdown_connections 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a9600640 1 -- 172.21.15.97:0/2991443064 >> 172.21.15.97:0/2991443064 conn(0x7f14a40d3ec0 msgr2=0x7f14a40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a9600640 1 -- 172.21.15.97:0/2991443064 shutdown_connections 2024-03-06T07:36:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.736+0000 7f14a9600640 1 -- 172.21.15.97:0/2991443064 wait complete. 2024-03-06T07:36:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.738+0000 7f14a9600640 1 Processor -- start 2024-03-06T07:36:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.738+0000 7f14a9600640 1 -- start start 2024-03-06T07:36:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14a4151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a41606c0 0x7f14a40caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a9600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14a4162450 0x7f14a40b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a9600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415fb30 con 0x7f1488004d40 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a9600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40d2bf0 con 0x7f14a4162450 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40d1fd0 con 0x7f14a41606c0 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14a4151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14a4151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46098/0 (socket says 172.21.15.97:46098) 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 -- 172.21.15.97:0/1483066723 learned_addr learned my addr 172.21.15.97:0/1483066723 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a3400640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14a4162450 0x7f14a40b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2000640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a41606c0 0x7f14a40caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 -- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a41606c0 msgr2=0x7f14a40caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a41606c0 0x7f14a40caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 -- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14a4162450 msgr2=0x7f14a40b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14a4162450 0x7f14a40b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.739+0000 7f14a2a00640 1 -- 172.21.15.97:0/1483066723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40c86b0 con 0x7f1488004d40 2024-03-06T07:36:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.740+0000 7f14a2a00640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14a4151ef0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f14940101f0 tx=0x7f14940106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.740+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149401a030 con 0x7f1488004d40 2024-03-06T07:36:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.740+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1494003990 con 0x7f1488004d40 2024-03-06T07:36:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.740+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1494003d60 con 0x7f1488004d40 2024-03-06T07:36:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.740+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a4160b50 con 0x7f1488004d40 2024-03-06T07:36:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.740+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a40c8970 con 0x7f1488004d40 2024-03-06T07:36:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.741+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a404f330 con 0x7f1488004d40 2024-03-06T07:36:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.742+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f149401d070 con 0x7f1488004d40 2024-03-06T07:36:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.743+0000 7f149aa00640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.743+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f149408b120 con 0x7f1488004d40 2024-03-06T07:36:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.743+0000 7f14a2000640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.744+0000 7f14a2000640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f148006b4d0 0x7f148006d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f148c001eb0 tx=0x7f148c000f50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.748+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f149401de40 con 0x7f1488004d40 2024-03-06T07:36:39.916 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:39.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.981+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14a408d920 con 0x7f1488004d40 2024-03-06T07:36:39.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.982+0000 7f149aa00640 1 -- 172.21.15.97:0/1483066723 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f1494059fd0 con 0x7f1488004d40 2024-03-06T07:36:39.985 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:39.985 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:39.985 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f148006b4d0 msgr2=0x7f148006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f148006b4d0 0x7f148006d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f148c001eb0 tx=0x7f148c000f50 comp rx=0 tx=0).stop 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 msgr2=0x7f14a4151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14a4151ef0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f14940101f0 tx=0x7f14940106b0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 shutdown_connections 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14a4162450 0x7f14a40b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f14a41606c0 0x7f14a40caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 --2- 172.21.15.97:0/1483066723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1488004d40 0x7f14a4151ef0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 >> 172.21.15.97:0/1483066723 conn(0x7f14a40d3ec0 msgr2=0x7f14a40b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 shutdown_connections 2024-03-06T07:36:39.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:39.985+0000 7f14a9600640 1 -- 172.21.15.97:0/1483066723 wait complete. 2024-03-06T07:36:40.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:40.206 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:41.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.430+0000 7fb53c400640 1 Processor -- start 2024-03-06T07:36:41.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.430+0000 7fb53c400640 1 -- start start 2024-03-06T07:36:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb53c400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb5340cffd0 0x7fb5340d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb53c400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5340b4390 0x7fb5340d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb53c400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5340cfb00 con 0x7fb5340ce490 2024-03-06T07:36:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb53c400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5340cdc40 con 0x7fb5340b4390 2024-03-06T07:36:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb53c400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5340cd3a0 con 0x7fb5340cffd0 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb539000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5340b4390 0x7fb5340d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb539000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5340b4390 0x7fb5340d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46110/0 (socket says 172.21.15.97:46110) 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.432+0000 7fb539000640 1 -- 172.21.15.97:0/3490836182 learned_addr learned my addr 172.21.15.97:0/3490836182 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.433+0000 7fb539a00640 1 --2- 172.21.15.97:0/3490836182 >> v2:172.21.15.135:3300/0 conn(0x7fb5340cffd0 0x7fb5340d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.433+0000 7fb539000640 1 -- 172.21.15.97:0/3490836182 >> v2:172.21.15.135:3300/0 conn(0x7fb5340cffd0 msgr2=0x7fb5340d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.433+0000 7fb539000640 1 --2- 172.21.15.97:0/3490836182 >> v2:172.21.15.135:3300/0 conn(0x7fb5340cffd0 0x7fb5340d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.433+0000 7fb539000640 1 -- 172.21.15.97:0/3490836182 >> v1:172.21.15.135:6789/0 conn(0x7fb5340ce490 legacy=0x7fb5340ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.433+0000 7fb539000640 1 -- 172.21.15.97:0/3490836182 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5340cd680 con 0x7fb5340b4390 2024-03-06T07:36:41.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.434+0000 7fb539000640 1 --2- 172.21.15.97:0/3490836182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340b4390 0x7fb5340d08f0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb534071cf0 tx=0x7fb52002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea60b59f3d00fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.434+0000 7fb533e00640 1 -- 172.21.15.97:0/3490836182 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520035020 con 0x7fb5340b4390 2024-03-06T07:36:41.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.434+0000 7fb533e00640 1 -- 172.21.15.97:0/3490836182 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520031870 con 0x7fb5340b4390 2024-03-06T07:36:41.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.434+0000 7fb533e00640 1 -- 172.21.15.97:0/3490836182 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520031c40 con 0x7fb5340b4390 2024-03-06T07:36:41.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 -- 172.21.15.97:0/3490836182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340b4390 msgr2=0x7fb5340d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 --2- 172.21.15.97:0/3490836182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340b4390 0x7fb5340d08f0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb534071cf0 tx=0x7fb52002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:41.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 -- 172.21.15.97:0/3490836182 shutdown_connections 2024-03-06T07:36:41.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 --2- 172.21.15.97:0/3490836182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340b4390 0x7fb5340d08f0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 --2- 172.21.15.97:0/3490836182 >> v2:172.21.15.135:3300/0 conn(0x7fb5340cffd0 0x7fb5340d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 -- 172.21.15.97:0/3490836182 >> 172.21.15.97:0/3490836182 conn(0x7fb5340b3870 msgr2=0x7fb5340b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 -- 172.21.15.97:0/3490836182 shutdown_connections 2024-03-06T07:36:41.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.435+0000 7fb53c400640 1 -- 172.21.15.97:0/3490836182 wait complete. 2024-03-06T07:36:41.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.437+0000 7fb53c400640 1 Processor -- start 2024-03-06T07:36:41.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.437+0000 7fb53c400640 1 -- start start 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb53c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5340b4390 0x7fb5341f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb53c400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 0x7fb5340b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb53c400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5340cffd0 0x7fb5340c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb53c400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5340cf130 con 0x7fb5340ce490 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb53c400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5340cefb0 con 0x7fb5340cffd0 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb53c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5340cf2b0 con 0x7fb5340b4390 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb539000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 0x7fb5340b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb539000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 0x7fb5340b69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46114/0 (socket says 172.21.15.97:46114) 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.438+0000 7fb539000640 1 -- 172.21.15.97:0/4167201678 learned_addr learned my addr 172.21.15.97:0/4167201678 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539a00640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5340b4390 0x7fb5341f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb53a400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5340cffd0 0x7fb5340c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539000640 1 -- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5340b4390 msgr2=0x7fb5341f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539000640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5340b4390 0x7fb5341f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539000640 1 -- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5340cffd0 msgr2=0x7fb5340c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539000640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5340cffd0 0x7fb5340c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539000640 1 -- 172.21.15.97:0/4167201678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5341f4c00 con 0x7fb5340ce490 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.439+0000 7fb539000640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 0x7fb5340b69e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fb520000c00 tx=0x7fb5200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.440+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52003ad60 con 0x7fb5340ce490 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.440+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5200443f0 con 0x7fb5340ce490 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.440+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5341f4ee0 con 0x7fb5340ce490 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.440+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200447c0 con 0x7fb5340ce490 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.440+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5340b4b70 con 0x7fb5340ce490 2024-03-06T07:36:41.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.441+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53404f330 con 0x7fb5340ce490 2024-03-06T07:36:41.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.442+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb52004b060 con 0x7fb5340ce490 2024-03-06T07:36:41.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.443+0000 7fb531600640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.444+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb5200ac490 con 0x7fb5340ce490 2024-03-06T07:36:41.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.444+0000 7fb539a00640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.444+0000 7fb539a00640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb51406b4d0 0x7fb51406d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb528006fd0 tx=0x7fb528008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.447+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb52003f070 con 0x7fb5340ce490 2024-03-06T07:36:41.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.683+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5340b14a0 con 0x7fb5340ce490 2024-03-06T07:36:41.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.683+0000 7fb531600640 1 -- 172.21.15.97:0/4167201678 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7fb52002f320 con 0x7fb5340ce490 2024-03-06T07:36:41.686 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:41.686 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:41.686 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb51406b4d0 msgr2=0x7fb51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb51406b4d0 0x7fb51406d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb528006fd0 tx=0x7fb528008040 comp rx=0 tx=0).stop 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 msgr2=0x7fb5340b69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 0x7fb5340b69e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fb520000c00 tx=0x7fb5200046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 shutdown_connections 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5340cffd0 0x7fb5340c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5340ce490 0x7fb5340b69e0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 --2- 172.21.15.97:0/4167201678 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5340b4390 0x7fb5341f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.686+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 >> 172.21.15.97:0/4167201678 conn(0x7fb5340b3870 msgr2=0x7fb5340b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.687+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 shutdown_connections 2024-03-06T07:36:41.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:41.687+0000 7fb53c400640 1 -- 172.21.15.97:0/4167201678 wait complete. 2024-03-06T07:36:41.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:41.907 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:41.918 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:42.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.121+0000 7f57f3c00640 1 Processor -- start 2024-03-06T07:36:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.121+0000 7f57f3c00640 1 -- start start 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f3c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec144c00 0x7f57ec06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f3c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f57ec15f9d0 0x7f57ec15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f3c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f57ec0ce0a0 0x7f57ec0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f3c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec160370 con 0x7f57ec0ce0a0 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f3c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec06ee70 con 0x7f57ec15f9d0 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f3c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec06cef0 con 0x7f57ec144c00 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f1c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f57ec0ce0a0 0x7f57ec0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f1c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f57ec0ce0a0 0x7f57ec0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46128/0 (socket says 172.21.15.97:46128) 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.123+0000 7f57f1c00640 1 -- 172.21.15.97:0/3499854375 learned_addr learned my addr 172.21.15.97:0/3499854375 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57f1200640 1 --2- 172.21.15.97:0/3499854375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec144c00 0x7f57ec06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57ebe00640 1 --2- 172.21.15.97:0/3499854375 >> v2:172.21.15.135:3300/0 conn(0x7f57ec15f9d0 0x7f57ec15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57f1c00640 1 -- 172.21.15.97:0/3499854375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec144c00 msgr2=0x7f57ec06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57f1c00640 1 --2- 172.21.15.97:0/3499854375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec144c00 0x7f57ec06e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57f1c00640 1 -- 172.21.15.97:0/3499854375 >> v2:172.21.15.135:3300/0 conn(0x7f57ec15f9d0 msgr2=0x7f57ec15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57f1c00640 1 --2- 172.21.15.97:0/3499854375 >> v2:172.21.15.135:3300/0 conn(0x7f57ec15f9d0 0x7f57ec15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57f1c00640 1 -- 172.21.15.97:0/3499854375 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec06d1d0 con 0x7f57ec0ce0a0 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.124+0000 7f57ebe00640 1 --2- 172.21.15.97:0/3499854375 >> v2:172.21.15.135:3300/0 conn(0x7f57ec15f9d0 0x7f57ec15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.125+0000 7f57f1c00640 1 --2- 172.21.15.97:0/3499854375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec0ce0a0 0x7f57ec0ce4f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0b27f0 tx=0x7f57e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59008a3080e02569 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.125+0000 7f57eb400640 1 -- 172.21.15.97:0/3499854375 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e402ca70 con 0x7f57ec0ce0a0 2024-03-06T07:36:43.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.125+0000 7f57eb400640 1 -- 172.21.15.97:0/3499854375 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e402cc10 con 0x7f57ec0ce0a0 2024-03-06T07:36:43.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.125+0000 7f57eb400640 1 -- 172.21.15.97:0/3499854375 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e40343f0 con 0x7f57ec0ce0a0 2024-03-06T07:36:43.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 -- 172.21.15.97:0/3499854375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec0ce0a0 msgr2=0x7f57ec0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 --2- 172.21.15.97:0/3499854375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec0ce0a0 0x7f57ec0ce4f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0b27f0 tx=0x7f57e402bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 -- 172.21.15.97:0/3499854375 shutdown_connections 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 --2- 172.21.15.97:0/3499854375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec0ce0a0 0x7f57ec0ce4f0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 --2- 172.21.15.97:0/3499854375 >> v2:172.21.15.135:3300/0 conn(0x7f57ec15f9d0 0x7f57ec15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 --2- 172.21.15.97:0/3499854375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec144c00 0x7f57ec06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 -- 172.21.15.97:0/3499854375 >> 172.21.15.97:0/3499854375 conn(0x7f57ec0b1730 msgr2=0x7f57ec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 -- 172.21.15.97:0/3499854375 shutdown_connections 2024-03-06T07:36:43.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.126+0000 7f57f3c00640 1 -- 172.21.15.97:0/3499854375 wait complete. 2024-03-06T07:36:43.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.129+0000 7f57f3c00640 1 Processor -- start 2024-03-06T07:36:43.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.130+0000 7f57f3c00640 1 -- start start 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.131+0000 7f57f3c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec0ce0a0 0x7f57ec1f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.131+0000 7f57f3c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 0x7f57ec139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.131+0000 7f57f3c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57ec15f9d0 0x7f57ec15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.131+0000 7f57f3c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec160730 con 0x7f57ec144c00 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.131+0000 7f57f3c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec06d790 con 0x7f57ec15f9d0 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.131+0000 7f57f3c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec06cb50 con 0x7f57ec0ce0a0 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 0x7f57ec139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 0x7f57ec139b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46130/0 (socket says 172.21.15.97:46130) 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 -- 172.21.15.97:0/266158514 learned_addr learned my addr 172.21.15.97:0/266158514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57f1200640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec0ce0a0 0x7f57ec1f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 -- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec0ce0a0 msgr2=0x7f57ec1f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57f1c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57ec15f9d0 0x7f57ec15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec0ce0a0 0x7f57ec1f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 -- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57ec15f9d0 msgr2=0x7f57ec15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57ec15f9d0 0x7f57ec15ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 -- 172.21.15.97:0/266158514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec1f76a0 con 0x7f57ec144c00 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.132+0000 7f57ebe00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 0x7f57ec139b90 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f57d800c820 tx=0x7f57d800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.133+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8015070 con 0x7f57ec144c00 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.133+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d8007a30 con 0x7f57ec144c00 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.133+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8007e00 con 0x7f57ec144c00 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.133+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec1f7980 con 0x7f57ec144c00 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.133+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec1f7180 con 0x7f57ec144c00 2024-03-06T07:36:43.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.134+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ec04f330 con 0x7f57ec144c00 2024-03-06T07:36:43.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.135+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57d8018070 con 0x7f57ec144c00 2024-03-06T07:36:43.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.136+0000 7f57e8c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f57cc06b4d0 0x7f57cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.136+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f57d808a140 con 0x7f57ec144c00 2024-03-06T07:36:43.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.136+0000 7f57f1200640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f57cc06b4d0 0x7f57cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.137+0000 7f57f1200640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f57cc06b4d0 0x7f57cc06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f57e000b7e0 tx=0x7f57e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.146+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57d8059010 con 0x7f57ec144c00 2024-03-06T07:36:43.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.379+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57ec0cf410 con 0x7f57ec144c00 2024-03-06T07:36:43.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.380+0000 7f57e8c00640 1 -- 172.21.15.97:0/266158514 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f57d8058910 con 0x7f57ec144c00 2024-03-06T07:36:43.382 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:43.382 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:43.382 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:43.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f57cc06b4d0 msgr2=0x7f57cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f57cc06b4d0 0x7f57cc06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f57e000b7e0 tx=0x7f57e0008040 comp rx=0 tx=0).stop 2024-03-06T07:36:43.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 msgr2=0x7f57ec139b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 0x7f57ec139b90 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f57d800c820 tx=0x7f57d800cc20 comp rx=0 tx=0).stop 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 shutdown_connections 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f57cc06b4d0 0x7f57cc06d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57ec15f9d0 0x7f57ec15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57ec144c00 0x7f57ec139b90 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 --2- 172.21.15.97:0/266158514 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f57ec0ce0a0 0x7f57ec1f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 >> 172.21.15.97:0/266158514 conn(0x7f57ec0b1730 msgr2=0x7f57ec070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 shutdown_connections 2024-03-06T07:36:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:43.382+0000 7f57f3c00640 1 -- 172.21.15.97:0/266158514 wait complete. 2024-03-06T07:36:43.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:43.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:43.920 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:44.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.813+0000 7f9e31e00640 1 Processor -- start 2024-03-06T07:36:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.814+0000 7f9e31e00640 1 -- start start 2024-03-06T07:36:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.815+0000 7f9e31e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 0x7f9e2c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.815+0000 7f9e31e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2c06cef0 con 0x7f9e2c144c00 2024-03-06T07:36:44.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.815+0000 7f9e31e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2c06d1d0 con 0x7f9e2c15f9d0 2024-03-06T07:36:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.815+0000 7f9e31e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c06d4b0 con 0x7f9e2c15fe60 2024-03-06T07:36:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9e2c144c00 0x7f9e2c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53032/0 (socket says 172.21.15.97:53032) 2024-03-06T07:36:44.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2aa00640 1 -- 172.21.15.97:0/968413644 learned_addr learned my addr 172.21.15.97:0/968413644 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182030780 0 0) 0x7f9e2c06cef0 con 0x7f9e2c144c00 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2b400640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 0x7f9e2c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e10003520 con 0x7f9e2c144c00 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526178221 0 0) 0x7f9e10003520 con 0x7f9e2c144c00 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 msgr2=0x7f9e2c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 0x7f9e2c071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 >> v1:172.21.15.135:6789/0 conn(0x7f9e2c15f9d0 legacy=0x7f9e2c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c06d790 con 0x7f9e2c144c00 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.816+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9e1c0019f0 con 0x7f9e2c144c00 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 >> v1:172.21.15.97:6789/0 conn(0x7f9e2c144c00 legacy=0x7f9e2c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e100039e0 0x7f9e10003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 0x7f9e10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c06d4b0 con 0x7f9e10004d40 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1000baa0 con 0x7f9e100039e0 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1000bca0 con 0x7f9e2c15fe60 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2b400640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e2aa00640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e100039e0 0x7f9e10003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.817+0000 7f9e22000640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 0x7f9e10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.818+0000 7f9e2b400640 1 -- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 msgr2=0x7f9e10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.818+0000 7f9e2b400640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c15fe60 0x7f9e10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.818+0000 7f9e2b400640 1 -- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e100039e0 msgr2=0x7f9e10003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.818+0000 7f9e2b400640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e100039e0 0x7f9e10003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.818+0000 7f9e2b400640 1 -- 172.21.15.97:0/968413644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10003520 con 0x7f9e10004d40 2024-03-06T07:36:44.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.818+0000 7f9e2aa00640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e100039e0 0x7f9e10003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:44.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e2b400640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e100051c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009a90 tx=0x7f9e10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04812f8d68eaf95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24039070 con 0x7f9e10004d40 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2402c680 con 0x7f9e10004d40 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e2a000640 1 -- 172.21.15.97:0/968413644 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24009690 con 0x7f9e10004d40 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e31e00640 1 -- 172.21.15.97:0/968413644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 msgr2=0x7f9e100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e31e00640 1 --2- 172.21.15.97:0/968413644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e100051c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009a90 tx=0x7f9e10005830 comp rx=0 tx=0).stop 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e2b400640 1 -- 172.21.15.97:0/968413644 reap_dead start 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e31e00640 1 -- 172.21.15.97:0/968413644 shutdown_connections 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.819+0000 7f9e31e00640 1 -- 172.21.15.97:0/968413644 >> 172.21.15.97:0/968413644 conn(0x7f9e2c0b1730 msgr2=0x7f9e2c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.820+0000 7f9e31e00640 1 -- 172.21.15.97:0/968413644 shutdown_connections 2024-03-06T07:36:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.820+0000 7f9e31e00640 1 -- 172.21.15.97:0/968413644 wait complete. 2024-03-06T07:36:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.821+0000 7f9e31e00640 1 Processor -- start 2024-03-06T07:36:44.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 -- start start 2024-03-06T07:36:44.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e2c13dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e2c15b4d0 0x7f9e2c15add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c13f7d0 0x7f9e2c13fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c06cb50 con 0x7f9e10004d40 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c160730 con 0x7f9e2c15b4d0 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.822+0000 7f9e31e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c06ccd0 con 0x7f9e2c13f7d0 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e2c13dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e2c13dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53828/0 (socket says 172.21.15.97:53828) 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 -- 172.21.15.97:0/2128643321 learned_addr learned my addr 172.21.15.97:0/2128643321 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e22000640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e2c15b4d0 0x7f9e2c15add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 -- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c13f7d0 msgr2=0x7f9e2c13fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c13f7d0 0x7f9e2c13fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 -- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e2c15b4d0 msgr2=0x7f9e2c15add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e2c15b4d0 0x7f9e2c15add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.823+0000 7f9e2aa00640 1 -- 172.21.15.97:0/2128643321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c13d560 con 0x7f9e10004d40 2024-03-06T07:36:44.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.824+0000 7f9e2aa00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e2c13dcc0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c001580 tx=0x7f9e1c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.824+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c00ee30 con 0x7f9e10004d40 2024-03-06T07:36:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.824+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c018070 con 0x7f9e10004d40 2024-03-06T07:36:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.824+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c13d840 con 0x7f9e10004d40 2024-03-06T07:36:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.824+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c013640 con 0x7f9e10004d40 2024-03-06T07:36:44.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.824+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c1e6a90 con 0x7f9e10004d40 2024-03-06T07:36:44.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.826+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c04f330 con 0x7f9e10004d40 2024-03-06T07:36:44.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.827+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e1c0137e0 con 0x7f9e10004d40 2024-03-06T07:36:44.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.828+0000 7f9e23e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e0806b4d0 0x7f9e0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.828+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e1c08aa60 con 0x7f9e10004d40 2024-03-06T07:36:44.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.828+0000 7f9e22000640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e0806b4d0 0x7f9e0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.829+0000 7f9e22000640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e0806b4d0 0x7f9e0806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9e14007fd0 tx=0x7f9e14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:44.833+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e1c059800 con 0x7f9e10004d40 2024-03-06T07:36:45.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.067+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2c072890 con 0x7f9e10004d40 2024-03-06T07:36:45.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.067+0000 7f9e23e00640 1 -- 172.21.15.97:0/2128643321 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f9e1c059100 con 0x7f9e10004d40 2024-03-06T07:36:45.070 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:45.070 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:45.070 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e0806b4d0 msgr2=0x7f9e0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e0806b4d0 0x7f9e0806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9e14007fd0 tx=0x7f9e14009040 comp rx=0 tx=0).stop 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 msgr2=0x7f9e2c13dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e2c13dcc0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c001580 tx=0x7f9e1c00e530 comp rx=0 tx=0).stop 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 shutdown_connections 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e0806b4d0 0x7f9e0806d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e2c13f7d0 0x7f9e2c13fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e2c15b4d0 0x7f9e2c15add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 --2- 172.21.15.97:0/2128643321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e10004d40 0x7f9e2c13dcc0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 >> 172.21.15.97:0/2128643321 conn(0x7f9e2c0b1730 msgr2=0x7f9e2c0ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:45.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 shutdown_connections 2024-03-06T07:36:45.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:45.070+0000 7f9e31e00640 1 -- 172.21.15.97:0/2128643321 wait complete. 2024-03-06T07:36:45.291 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:45.291 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:45.922 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:46.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:46.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.502+0000 7f6fa3200640 1 Processor -- start 2024-03-06T07:36:46.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.502+0000 7f6fa3200640 1 -- start start 2024-03-06T07:36:46.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.504+0000 7f6fa3200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6f9c0b4680 0x7f6f9c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.504+0000 7f6fa3200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9c15b090 con 0x7f6f9c15acb0 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.504+0000 7f6fa3200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9c0b2a30 con 0x7f6f9c072340 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.504+0000 7f6fa3200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9c0b4360 con 0x7f6f9c0b4680 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.504+0000 7f6f9b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6f9c0b4680 0x7f6f9c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6f9c0b4680 0x7f6f9c06e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53844/0 (socket says 172.21.15.97:53844) 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f6f9c15acb0 0x7f6f9c0b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53038/0 (socket says 172.21.15.97:53038) 2024-03-06T07:36:46.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9b400640 1 -- 172.21.15.97:0/208827269 learned_addr learned my addr 172.21.15.97:0/208827269 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:46.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9b400640 1 -- 172.21.15.97:0/208827269 >> v1:172.21.15.135:6789/0 conn(0x7f6f9c072340 legacy=0x7f6f9c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:46.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9b400640 1 -- 172.21.15.97:0/208827269 >> v1:172.21.15.97:6789/0 conn(0x7f6f9c15acb0 legacy=0x7f6f9c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9b400640 1 -- 172.21.15.97:0/208827269 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c0b2cf0 con 0x7f6f9c0b4680 2024-03-06T07:36:46.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.505+0000 7f6f9aa00640 1 -- 172.21.15.97:0/208827269 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858707398 0 0) 0x7f6f9c15b090 con 0x7f6f9c15acb0 2024-03-06T07:36:46.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.506+0000 7f6f9b400640 1 --2- 172.21.15.97:0/208827269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c06e4f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009870 tx=0x7f6f8802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69353310a2ad738 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.506+0000 7f6f9aa00640 1 -- 172.21.15.97:0/208827269 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88034070 con 0x7f6f9c0b4680 2024-03-06T07:36:46.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.506+0000 7f6f9aa00640 1 -- 172.21.15.97:0/208827269 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f88030860 con 0x7f6f9c0b4680 2024-03-06T07:36:46.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.506+0000 7f6f9aa00640 1 -- 172.21.15.97:0/208827269 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88030c30 con 0x7f6f9c0b4680 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 -- 172.21.15.97:0/208827269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 msgr2=0x7f6f9c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 --2- 172.21.15.97:0/208827269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c06e4f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009870 tx=0x7f6f8802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 -- 172.21.15.97:0/208827269 shutdown_connections 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 --2- 172.21.15.97:0/208827269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c06e4f0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 -- 172.21.15.97:0/208827269 >> 172.21.15.97:0/208827269 conn(0x7f6f9c0b1730 msgr2=0x7f6f9c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 -- 172.21.15.97:0/208827269 shutdown_connections 2024-03-06T07:36:46.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.507+0000 7f6fa3200640 1 -- 172.21.15.97:0/208827269 wait complete. 2024-03-06T07:36:46.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.509+0000 7f6fa3200640 1 Processor -- start 2024-03-06T07:36:46.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.509+0000 7f6fa3200640 1 -- start start 2024-03-06T07:36:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6fa3200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f9c072340 0x7f6f9c15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6fa3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6fa3200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f9c15acb0 0x7f6f9c1515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6fa3200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c0b4010 con 0x7f6f9c0b4680 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6fa3200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c0b32b0 con 0x7f6f9c072340 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6fa3200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c06ea30 con 0x7f6f9c15acb0 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6f9b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6f9b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c0ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53848/0 (socket says 172.21.15.97:53848) 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.510+0000 7f6f9b400640 1 -- 172.21.15.97:0/734916036 learned_addr learned my addr 172.21.15.97:0/734916036 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9be00640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f9c072340 0x7f6f9c15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6fa1200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f9c15acb0 0x7f6f9c1515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9b400640 1 -- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f9c15acb0 msgr2=0x7f6f9c1515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9b400640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f9c15acb0 0x7f6f9c1515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9b400640 1 -- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f9c072340 msgr2=0x7f6f9c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9b400640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f9c072340 0x7f6f9c15dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9b400640 1 -- 172.21.15.97:0/734916036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c0cd6e0 con 0x7f6f9c0b4680 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f9b400640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c0ccf70 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009870 tx=0x7f6f88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88048070 con 0x7f6f9c0b4680 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f88030660 con 0x7f6f9c0b4680 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c0cd9c0 con 0x7f6f9c0b4680 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.511+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88034040 con 0x7f6f9c0b4680 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.512+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c0cdf80 con 0x7f6f9c0b4680 2024-03-06T07:36:46.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.513+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c04f330 con 0x7f6f9c0b4680 2024-03-06T07:36:46.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.514+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f8804b070 con 0x7f6f9c0b4680 2024-03-06T07:36:46.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.514+0000 7f6f93e00640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f7c06b4d0 0x7f6f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.515+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6f880ac350 con 0x7f6f9c0b4680 2024-03-06T07:36:46.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.515+0000 7f6f9be00640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f7c06b4d0 0x7f6f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.515+0000 7f6f9be00640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f7c06b4d0 0x7f6f7c06d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c000f60 tx=0x7f6f8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.521+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f8807b180 con 0x7f6f9c0b4680 2024-03-06T07:36:46.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.830+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f9c06ce60 con 0x7f6f9c0b4680 2024-03-06T07:36:46.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.831+0000 7f6f93e00640 1 -- 172.21.15.97:0/734916036 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f6f8807aa80 con 0x7f6f9c0b4680 2024-03-06T07:36:46.834 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:46.834 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:46.834 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f7c06b4d0 msgr2=0x7f6f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f7c06b4d0 0x7f6f7c06d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c000f60 tx=0x7f6f8c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 msgr2=0x7f6f9c0ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c0ccf70 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009870 tx=0x7f6f88004730 comp rx=0 tx=0).stop 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 shutdown_connections 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6f7c06b4d0 0x7f6f7c06d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f9c15acb0 0x7f6f9c1515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f9c0b4680 0x7f6f9c0ccf70 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 --2- 172.21.15.97:0/734916036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f9c072340 0x7f6f9c15dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 >> 172.21.15.97:0/734916036 conn(0x7f6f9c0b1730 msgr2=0x7f6f9c0b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 shutdown_connections 2024-03-06T07:36:46.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:46.834+0000 7f6fa3200640 1 -- 172.21.15.97:0/734916036 wait complete. 2024-03-06T07:36:47.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:47.062 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:47.924 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:48.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:48.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.275+0000 7f1d45000640 1 Processor -- start 2024-03-06T07:36:48.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.275+0000 7f1d45000640 1 -- start start 2024-03-06T07:36:48.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.276+0000 7f1d45000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1d4015dfd0 0x7f1d40162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.276+0000 7f1d45000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d400cef70 con 0x7f1d4006b6d0 2024-03-06T07:36:48.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.276+0000 7f1d45000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4015e3b0 con 0x7f1d400affe0 2024-03-06T07:36:48.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.276+0000 7f1d45000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d40162270 con 0x7f1d4015dfd0 2024-03-06T07:36:48.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.277+0000 7f1d3fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f1d4006b6d0 0x7f1d400d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53044/0 (socket says 172.21.15.97:53044) 2024-03-06T07:36:48.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.277+0000 7f1d3fe00640 1 -- 172.21.15.97:0/1347457418 learned_addr learned my addr 172.21.15.97:0/1347457418 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.277+0000 7f1d3f400640 1 --2- 172.21.15.97:0/1347457418 >> v2:172.21.15.135:3300/0 conn(0x7f1d4015dfd0 0x7f1d40162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.277+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165124404 0 0) 0x7f1d400cef70 con 0x7f1d4006b6d0 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.277+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d24003520 con 0x7f1d4006b6d0 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486417421 0 0) 0x7f1d24003520 con 0x7f1d4006b6d0 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 >> v2:172.21.15.135:3300/0 conn(0x7f1d4015dfd0 msgr2=0x7f1d40162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 --2- 172.21.15.97:0/1347457418 >> v2:172.21.15.135:3300/0 conn(0x7f1d4015dfd0 0x7f1d40162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 >> v1:172.21.15.135:6789/0 conn(0x7f1d400affe0 legacy=0x7f1d4006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40161c60 con 0x7f1d4006b6d0 2024-03-06T07:36:48.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1d380019f0 con 0x7f1d4006b6d0 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 >> v1:172.21.15.97:6789/0 conn(0x7f1d4006b6d0 legacy=0x7f1d400d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d240039b0 0x7f1d24003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d24004800 0x7f1d24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d40162270 con 0x7f1d240052e0 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2400c1b0 con 0x7f1d24004800 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2400c3b0 con 0x7f1d240039b0 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.278+0000 7f1d3fe00640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3f400640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d240039b0 0x7f1d24003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3ea00640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d24004800 0x7f1d24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3fe00640 1 -- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d240039b0 msgr2=0x7f1d24003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3fe00640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d240039b0 0x7f1d24003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3fe00640 1 -- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d24004800 msgr2=0x7f1d24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3fe00640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d24004800 0x7f1d24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.279+0000 7f1d3fe00640 1 -- 172.21.15.97:0/1347457418 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d24003520 con 0x7f1d240052e0 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.280+0000 7f1d3fe00640 1 -- 172.21.15.97:0/1347457418 reap_dead start 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.280+0000 7f1d3fe00640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d2400b9c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800a850 tx=0x7f1d24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82286a50ce183dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800a4a0 con 0x7f1d240052e0 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d38034650 con 0x7f1d240052e0 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d3e000640 1 -- 172.21.15.97:0/1347457418 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38034a40 con 0x7f1d240052e0 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d45000640 1 -- 172.21.15.97:0/1347457418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 msgr2=0x7f1d2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d45000640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d2400b9c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800a850 tx=0x7f1d24005830 comp rx=0 tx=0).stop 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d45000640 1 -- 172.21.15.97:0/1347457418 shutdown_connections 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d45000640 1 --2- 172.21.15.97:0/1347457418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d2400b9c0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.281+0000 7f1d45000640 1 -- 172.21.15.97:0/1347457418 >> 172.21.15.97:0/1347457418 conn(0x7f1d4006c0d0 msgr2=0x7f1d4006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:48.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.282+0000 7f1d45000640 1 -- 172.21.15.97:0/1347457418 shutdown_connections 2024-03-06T07:36:48.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.282+0000 7f1d45000640 1 -- 172.21.15.97:0/1347457418 wait complete. 2024-03-06T07:36:48.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.283+0000 7f1d45000640 1 Processor -- start 2024-03-06T07:36:48.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.283+0000 7f1d45000640 1 -- start start 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.284+0000 7f1d45000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d40154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.284+0000 7f1d45000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d40152520 0x7f1d40145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.284+0000 7f1d45000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d400d06f0 0x7f1d400d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.284+0000 7f1d45000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400ce8d0 con 0x7f1d240052e0 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.284+0000 7f1d45000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d40161f00 con 0x7f1d40152520 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.284+0000 7f1d45000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400cea50 con 0x7f1d400d06f0 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d40154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d40154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53880/0 (socket says 172.21.15.97:53880) 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 -- 172.21.15.97:0/2048782218 learned_addr learned my addr 172.21.15.97:0/2048782218 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3ea00640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d40152520 0x7f1d40145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3fe00640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d400d06f0 0x7f1d400d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 -- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d400d06f0 msgr2=0x7f1d400d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d400d06f0 0x7f1d400d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 -- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d40152520 msgr2=0x7f1d40145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d40152520 0x7f1d40145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 -- 172.21.15.97:0/2048782218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40155780 con 0x7f1d240052e0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3ea00640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d40152520 0x7f1d40145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.285+0000 7f1d3f400640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d40154f30 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f1d30011540 tx=0x7f1d30011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.286+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30015930 con 0x7f1d240052e0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.286+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d30015ad0 con 0x7f1d240052e0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.286+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3001d400 con 0x7f1d240052e0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.286+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d401f4160 con 0x7f1d240052e0 2024-03-06T07:36:48.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.286+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d401f4720 con 0x7f1d240052e0 2024-03-06T07:36:48.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.287+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d401f3f80 con 0x7f1d240052e0 2024-03-06T07:36:48.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.288+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d3001d5a0 con 0x7f1d240052e0 2024-03-06T07:36:48.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.289+0000 7f1d37400640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d2006b4d0 0x7f1d2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.289+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1d3008aae0 con 0x7f1d240052e0 2024-03-06T07:36:48.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.289+0000 7f1d3ea00640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d2006b4d0 0x7f1d2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.290+0000 7f1d3ea00640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d2006b4d0 0x7f1d2006d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c000c00 tx=0x7f1d2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.294+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d401f3f80 con 0x7f1d240052e0 2024-03-06T07:36:48.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.535+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d4004f330 con 0x7f1d240052e0 2024-03-06T07:36:48.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.536+0000 7f1d37400640 1 -- 172.21.15.97:0/2048782218 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f1d30019030 con 0x7f1d240052e0 2024-03-06T07:36:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:48.539 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:48.539 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d2006b4d0 msgr2=0x7f1d2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d2006b4d0 0x7f1d2006d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c000c00 tx=0x7f1d2c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 msgr2=0x7f1d40154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d40154f30 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f1d30011540 tx=0x7f1d30011a00 comp rx=0 tx=0).stop 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 shutdown_connections 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d2006b4d0 0x7f1d2006d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1d400d06f0 0x7f1d400d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d40152520 0x7f1d40145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 --2- 172.21.15.97:0/2048782218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d240052e0 0x7f1d40154f30 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 >> 172.21.15.97:0/2048782218 conn(0x7f1d4006c0d0 msgr2=0x7f1d40162530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:48.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 shutdown_connections 2024-03-06T07:36:48.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:48.539+0000 7f1d45000640 1 -- 172.21.15.97:0/2048782218 wait complete. 2024-03-06T07:36:48.761 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:48.762 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:49.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.966+0000 7f681a800640 1 Processor -- start 2024-03-06T07:36:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.967+0000 7f681a800640 1 -- start start 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.968+0000 7f681a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f681415e0d0 0x7f68140d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.968+0000 7f681a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415f470 0x7f68140d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.968+0000 7f681a800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f68140d0df0 0x7f6814160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.968+0000 7f681a800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68140d1260 con 0x7f681415e0d0 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.968+0000 7f681a800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68140d2f60 con 0x7f68140d0df0 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.968+0000 7f681a800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68140cea60 con 0x7f681415f470 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f681415e0d0 0x7f68140d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f681415e0d0 0x7f68140d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53894/0 (socket says 172.21.15.97:53894) 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 -- 172.21.15.97:0/2024568148 learned_addr learned my addr 172.21.15.97:0/2024568148 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6812a00640 1 --2- 172.21.15.97:0/2024568148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415f470 0x7f68140d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813e00640 1 --2- 172.21.15.97:0/2024568148 >> v2:172.21.15.135:3300/0 conn(0x7f68140d0df0 0x7f6814160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 -- 172.21.15.97:0/2024568148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415f470 msgr2=0x7f68140d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 --2- 172.21.15.97:0/2024568148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415f470 0x7f68140d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 -- 172.21.15.97:0/2024568148 >> v2:172.21.15.135:3300/0 conn(0x7f68140d0df0 msgr2=0x7f6814160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 --2- 172.21.15.97:0/2024568148 >> v2:172.21.15.135:3300/0 conn(0x7f68140d0df0 0x7f6814160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.969+0000 7f6813400640 1 -- 172.21.15.97:0/2024568148 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140d2350 con 0x7f681415e0d0 2024-03-06T07:36:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.970+0000 7f6813400640 1 --2- 172.21.15.97:0/2024568148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f681415e0d0 0x7f68140d1a90 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f681408cde0 tx=0x7f680402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fcc0ce33bb7773 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f6812000640 1 -- 172.21.15.97:0/2024568148 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804034070 con 0x7f681415e0d0 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f6812000640 1 -- 172.21.15.97:0/2024568148 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6804030860 con 0x7f681415e0d0 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f6812000640 1 -- 172.21.15.97:0/2024568148 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804030c30 con 0x7f681415e0d0 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 -- 172.21.15.97:0/2024568148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f681415e0d0 msgr2=0x7f68140d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 --2- 172.21.15.97:0/2024568148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f681415e0d0 0x7f68140d1a90 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f681408cde0 tx=0x7f680402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 -- 172.21.15.97:0/2024568148 shutdown_connections 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 --2- 172.21.15.97:0/2024568148 >> v2:172.21.15.135:3300/0 conn(0x7f68140d0df0 0x7f6814160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 --2- 172.21.15.97:0/2024568148 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415f470 0x7f68140d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 --2- 172.21.15.97:0/2024568148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f681415e0d0 0x7f68140d1a90 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 -- 172.21.15.97:0/2024568148 >> 172.21.15.97:0/2024568148 conn(0x7f68140d3ec0 msgr2=0x7f68140d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.971+0000 7f681a800640 1 -- 172.21.15.97:0/2024568148 shutdown_connections 2024-03-06T07:36:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.972+0000 7f681a800640 1 -- 172.21.15.97:0/2024568148 wait complete. 2024-03-06T07:36:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.973+0000 7f681a800640 1 Processor -- start 2024-03-06T07:36:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.973+0000 7f681a800640 1 -- start start 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f681a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 0x7f68140b9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f681a800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415e0d0 0x7f681414e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f681a800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681415f470 0x7f68140c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f681a800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140d2bf0 con 0x7f68140d0df0 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f681a800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140d2910 con 0x7f681415f470 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f681a800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681415fb30 con 0x7f681415e0d0 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.974+0000 7f6813400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 0x7f68140b9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 0x7f68140b9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53900/0 (socket says 172.21.15.97:53900) 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 -- 172.21.15.97:0/648952686 learned_addr learned my addr 172.21.15.97:0/648952686 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813e00640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681415f470 0x7f68140c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6812a00640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415e0d0 0x7f681414e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 -- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415e0d0 msgr2=0x7f681414e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415e0d0 0x7f681414e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 -- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681415f470 msgr2=0x7f68140c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681415f470 0x7f68140c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 -- 172.21.15.97:0/648952686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140caf90 con 0x7f68140d0df0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.975+0000 7f6813400640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 0x7f68140b9960 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f6804000c00 tx=0x7f6804004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.976+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804048070 con 0x7f68140d0df0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.976+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6804030660 con 0x7f68140d0df0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.976+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804034040 con 0x7f68140d0df0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.976+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68140cb230 con 0x7f68140d0df0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.976+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68140cb7f0 con 0x7f68140d0df0 2024-03-06T07:36:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.977+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681404f330 con 0x7f68140d0df0 2024-03-06T07:36:49.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.980+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f680404b070 con 0x7f68140d0df0 2024-03-06T07:36:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.981+0000 7f680b400640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b4d0 0x7f67f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.981+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f68040ac680 con 0x7f68140d0df0 2024-03-06T07:36:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.981+0000 7f6812a00640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b4d0 0x7f67f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.982+0000 7f6812a00640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b4d0 0x7f67f406d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f68140cf120 tx=0x7f6800008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:49.984+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f680407b5c0 con 0x7f68140d0df0 2024-03-06T07:36:50.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.223+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f681408d920 con 0x7f68140d0df0 2024-03-06T07:36:50.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.224+0000 7f680b400640 1 -- 172.21.15.97:0/648952686 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f6804030830 con 0x7f68140d0df0 2024-03-06T07:36:50.226 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:50.226 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:50.227 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:50.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b4d0 msgr2=0x7f67f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b4d0 0x7f67f406d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f68140cf120 tx=0x7f6800008040 comp rx=0 tx=0).stop 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 msgr2=0x7f68140b9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 0x7f68140b9960 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f6804000c00 tx=0x7f6804004730 comp rx=0 tx=0).stop 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 shutdown_connections 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b4d0 0x7f67f406d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681415f470 0x7f68140c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f681415e0d0 0x7f681414e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 --2- 172.21.15.97:0/648952686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68140d0df0 0x7f68140b9960 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 >> 172.21.15.97:0/648952686 conn(0x7f68140d3ec0 msgr2=0x7f68140d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.227+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 shutdown_connections 2024-03-06T07:36:50.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:50.228+0000 7f681a800640 1 -- 172.21.15.97:0/648952686 wait complete. 2024-03-06T07:36:50.448 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:50.449 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:51.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:51.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.668+0000 7f4f06000640 1 Processor -- start 2024-03-06T07:36:51.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.668+0000 7f4f06000640 1 -- start start 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.669+0000 7f4f06000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4f000d0fc0 0x7f4f00160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.669+0000 7f4f06000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f000cea60 con 0x7f4f0015f470 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.669+0000 7f4f06000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f000d2350 con 0x7f4f0015e0d0 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.669+0000 7f4f06000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f000d2f60 con 0x7f4f000d0fc0 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4effe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4f000d0fc0 0x7f4f00160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4efea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4f0015f470 0x7f4f000d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53058/0 (socket says 172.21.15.97:53058) 2024-03-06T07:36:51.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4efea00640 1 -- 172.21.15.97:0/2952461266 learned_addr learned my addr 172.21.15.97:0/2952461266 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4effe00640 1 -- 172.21.15.97:0/2952461266 >> v1:172.21.15.135:6789/0 conn(0x7f4f0015e0d0 legacy=0x7f4f000d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4effe00640 1 -- 172.21.15.97:0/2952461266 >> v1:172.21.15.97:6789/0 conn(0x7f4f0015f470 legacy=0x7f4f000d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4effe00640 1 -- 172.21.15.97:0/2952461266 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000d2630 con 0x7f4f000d0fc0 2024-03-06T07:36:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.670+0000 7f4efe000640 1 -- 172.21.15.97:0/2952461266 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388641711 0 0) 0x7f4f000cea60 con 0x7f4f0015f470 2024-03-06T07:36:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.671+0000 7f4effe00640 1 --2- 172.21.15.97:0/2952461266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f000d0fc0 0x7f4f00160dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f4f000caf20 tx=0x7f4ef802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb5b95d6a631eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4efe000640 1 -- 172.21.15.97:0/2952461266 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8039070 con 0x7f4f000d0fc0 2024-03-06T07:36:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4efe000640 1 -- 172.21.15.97:0/2952461266 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef802ccd0 con 0x7f4f000d0fc0 2024-03-06T07:36:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4efe000640 1 -- 172.21.15.97:0/2952461266 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8034660 con 0x7f4f000d0fc0 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4f06000640 1 -- 172.21.15.97:0/2952461266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f000d0fc0 msgr2=0x7f4f00160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4f06000640 1 --2- 172.21.15.97:0/2952461266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f000d0fc0 0x7f4f00160dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f4f000caf20 tx=0x7f4ef802bf20 comp rx=0 tx=0).stop 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4f06000640 1 -- 172.21.15.97:0/2952461266 shutdown_connections 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4f06000640 1 --2- 172.21.15.97:0/2952461266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f000d0fc0 0x7f4f00160dc0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.672+0000 7f4f06000640 1 -- 172.21.15.97:0/2952461266 >> 172.21.15.97:0/2952461266 conn(0x7f4f000d3ec0 msgr2=0x7f4f000d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.673+0000 7f4f06000640 1 -- 172.21.15.97:0/2952461266 shutdown_connections 2024-03-06T07:36:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.673+0000 7f4f06000640 1 -- 172.21.15.97:0/2952461266 wait complete. 2024-03-06T07:36:51.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.675+0000 7f4f06000640 1 Processor -- start 2024-03-06T07:36:51.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.675+0000 7f4f06000640 1 -- start start 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4f06000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f000d0fc0 0x7f4f000b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4f06000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 0x7f4f00155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4f06000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4f0015f470 0x7f4f000b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4f06000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000cff10 con 0x7f4f0015e0d0 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4f06000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000d2bf0 con 0x7f4f000d0fc0 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4f06000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0015fb30 con 0x7f4f0015f470 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 0x7f4f00155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 0x7f4f00155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53922/0 (socket says 172.21.15.97:53922) 2024-03-06T07:36:51.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 -- 172.21.15.97:0/3930337184 learned_addr learned my addr 172.21.15.97:0/3930337184 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4eff400640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f000d0fc0 0x7f4f000b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4effe00640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4f0015f470 0x7f4f000b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 -- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4f0015f470 msgr2=0x7f4f000b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4f0015f470 0x7f4f000b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 -- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f000d0fc0 msgr2=0x7f4f000b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f000d0fc0 0x7f4f000b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4efea00640 1 -- 172.21.15.97:0/3930337184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000b7a30 con 0x7f4f0015e0d0 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.676+0000 7f4eff400640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f000d0fc0 0x7f4f000b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.677+0000 7f4efea00640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 0x7f4f00155720 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800c430 tx=0x7f4ee800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.677+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80074e0 con 0x7f4f0015e0d0 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.677+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee80047e0 con 0x7f4f0015e0d0 2024-03-06T07:36:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.677+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8004bd0 con 0x7f4f0015e0d0 2024-03-06T07:36:51.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.677+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f000ccc40 con 0x7f4f0015e0d0 2024-03-06T07:36:51.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.677+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f000cd200 con 0x7f4f0015e0d0 2024-03-06T07:36:51.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.678+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0004f330 con 0x7f4f0015e0d0 2024-03-06T07:36:51.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.680+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ee8011070 con 0x7f4f0015e0d0 2024-03-06T07:36:51.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.681+0000 7f4ef7400640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4edc06b4d0 0x7f4edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.681+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ee8089f10 con 0x7f4f0015e0d0 2024-03-06T07:36:51.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.681+0000 7f4eff400640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4edc06b4d0 0x7f4edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.682+0000 7f4eff400640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4edc06b4d0 0x7f4edc06d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0003eb0 tx=0x7f4ef0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.686+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f000b7a30 con 0x7f4f0015e0d0 2024-03-06T07:36:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.917+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f00159520 con 0x7f4f0015e0d0 2024-03-06T07:36:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.919+0000 7f4ef7400640 1 -- 172.21.15.97:0/3930337184 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f4ee8058d40 con 0x7f4f0015e0d0 2024-03-06T07:36:51.921 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:51.921 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:51.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4edc06b4d0 msgr2=0x7f4edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4edc06b4d0 0x7f4edc06d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0003eb0 tx=0x7f4ef0009040 comp rx=0 tx=0).stop 2024-03-06T07:36:51.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 msgr2=0x7f4f00155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 0x7f4f00155720 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800c430 tx=0x7f4ee800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 shutdown_connections 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4edc06b4d0 0x7f4edc06d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4f0015f470 0x7f4f000b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4f0015e0d0 0x7f4f00155720 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 --2- 172.21.15.97:0/3930337184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f000d0fc0 0x7f4f000b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 >> 172.21.15.97:0/3930337184 conn(0x7f4f000d3ec0 msgr2=0x7f4f000b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 shutdown_connections 2024-03-06T07:36:51.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:51.921+0000 7f4f06000640 1 -- 172.21.15.97:0/3930337184 wait complete. 2024-03-06T07:36:51.927 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:52.143 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:52.143 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:53.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:53.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.356+0000 7feaa0c00640 1 Processor -- start 2024-03-06T07:36:53.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.356+0000 7feaa0c00640 1 -- start start 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.357+0000 7feaa0c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fea9c0d0600 0x7fea9c0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.358+0000 7feaa0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea9c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.358+0000 7feaa0c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9c15f030 con 0x7fea9c0b2a60 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.358+0000 7feaa0c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9c0cf090 con 0x7fea9c0d0600 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.358+0000 7feaa0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c15f310 con 0x7fea9c0d18e0 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.358+0000 7fea93e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fea9c0b2a60 0x7fea9c0d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53068/0 (socket says 172.21.15.97:53068) 2024-03-06T07:36:53.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.358+0000 7fea93e00640 1 -- 172.21.15.97:0/2186242483 learned_addr learned my addr 172.21.15.97:0/2186242483 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9b400640 1 --2- 172.21.15.97:0/2186242483 >> v2:172.21.15.135:3300/0 conn(0x7fea9c0d0600 0x7fea9c0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9be00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea9c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543739072 0 0) 0x7fea9c15f030 con 0x7fea9c0b2a60 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea80003520 con 0x7fea9c0b2a60 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021960695 0 0) 0x7fea80003520 con 0x7fea9c0b2a60 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 msgr2=0x7fea9c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea9c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 >> v2:172.21.15.135:3300/0 conn(0x7fea9c0d0600 msgr2=0x7fea9c0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 --2- 172.21.15.97:0/2186242483 >> v2:172.21.15.135:3300/0 conn(0x7fea9c0d0600 0x7fea9c0d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c0cea40 con 0x7fea9c0b2a60 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fea84001670 con 0x7fea9c0b2a60 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9be00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea9c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 >> v1:172.21.15.97:6789/0 conn(0x7fea9c0b2a60 legacy=0x7fea9c0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea80003ef0 0x7fea80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80004d40 0x7fea800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0cf090 con 0x7fea80003ef0 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c15f310 con 0x7fea80004d40 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.359+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea8000b9f0 con 0x7fea9c0d18e0 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea93e00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea80003ef0 0x7fea80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea9b400640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea9be00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80004d40 0x7fea800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea93e00640 1 -- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 msgr2=0x7fea800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea93e00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0d18e0 0x7fea800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea93e00640 1 -- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80004d40 msgr2=0x7fea800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea93e00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80004d40 0x7fea800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea93e00640 1 -- 172.21.15.97:0/2186242483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea80003520 con 0x7fea80003ef0 2024-03-06T07:36:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.360+0000 7fea9be00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80004d40 0x7fea800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.362+0000 7fea93e00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea80003ef0 0x7fea80004800 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fea8400c290 tx=0x7fea80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1002f1a9a283e992 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.362+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84007d50 con 0x7fea80003ef0 2024-03-06T07:36:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.362+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea84030070 con 0x7fea80003ef0 2024-03-06T07:36:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.362+0000 7fea9aa00640 1 -- 172.21.15.97:0/2186242483 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8402cdc0 con 0x7fea80003ef0 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.362+0000 7feaa0c00640 1 -- 172.21.15.97:0/2186242483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea80003ef0 msgr2=0x7fea80004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.362+0000 7feaa0c00640 1 --2- 172.21.15.97:0/2186242483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea80003ef0 0x7fea80004800 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fea8400c290 tx=0x7fea80005830 comp rx=0 tx=0).stop 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.363+0000 7fea9be00640 1 -- 172.21.15.97:0/2186242483 reap_dead start 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.363+0000 7feaa0c00640 1 -- 172.21.15.97:0/2186242483 shutdown_connections 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.363+0000 7feaa0c00640 1 -- 172.21.15.97:0/2186242483 >> 172.21.15.97:0/2186242483 conn(0x7fea9c0d3d10 msgr2=0x7fea9c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.363+0000 7feaa0c00640 1 -- 172.21.15.97:0/2186242483 shutdown_connections 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.363+0000 7feaa0c00640 1 -- 172.21.15.97:0/2186242483 wait complete. 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.364+0000 7feaa0c00640 1 Processor -- start 2024-03-06T07:36:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 -- start start 2024-03-06T07:36:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80003ef0 0x7fea9c0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 0x7fea9c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0c8200 0x7fea9c0b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c15ecf0 con 0x7fea9c0c7840 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0b3250 con 0x7fea80003ef0 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.365+0000 7feaa0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0d1d70 con 0x7fea9c0c8200 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 0x7fea9c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 0x7fea9c0c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53950/0 (socket says 172.21.15.97:53950) 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 -- 172.21.15.97:0/3166351247 learned_addr learned my addr 172.21.15.97:0/3166351247 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea9b400640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80003ef0 0x7fea9c0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea9be00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0c8200 0x7fea9c0b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 -- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0c8200 msgr2=0x7fea9c0b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0c8200 0x7fea9c0b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 -- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80003ef0 msgr2=0x7fea9c0c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80003ef0 0x7fea9c0c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea93e00640 1 -- 172.21.15.97:0/3166351247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c0c8680 con 0x7fea9c0c7840 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.366+0000 7fea9b400640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80003ef0 0x7fea9c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:53.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.367+0000 7fea93e00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 0x7fea9c0c7cc0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fea8400c3b0 tx=0x7fea9c0ce780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.367+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84045070 con 0x7fea9c0c7840 2024-03-06T07:36:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.367+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea84004bf0 con 0x7fea9c0c7840 2024-03-06T07:36:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.367+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9c0b4950 con 0x7fea9c0c7840 2024-03-06T07:36:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.367+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84001c80 con 0x7fea9c0c7840 2024-03-06T07:36:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.367+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9c0b5910 con 0x7fea9c0c7840 2024-03-06T07:36:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.368+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9c04f330 con 0x7fea9c0c7840 2024-03-06T07:36:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.369+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea84007070 con 0x7fea9c0c7840 2024-03-06T07:36:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.370+0000 7fea93400640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea7806b500 0x7fea7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.370+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea840ab2c0 con 0x7fea9c0c7840 2024-03-06T07:36:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.370+0000 7fea9b400640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea7806b500 0x7fea7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.371+0000 7fea9b400640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea7806b500 0x7fea7806d9c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009a80 tx=0x7fea8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:53.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.374+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea84079b20 con 0x7fea9c0c7840 2024-03-06T07:36:53.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.610+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea9c08d7e0 con 0x7fea9c0c7840 2024-03-06T07:36:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.611+0000 7fea93400640 1 -- 172.21.15.97:0/3166351247 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7fea8407dbc0 con 0x7fea9c0c7840 2024-03-06T07:36:53.613 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:53.613 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.613+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea7806b500 msgr2=0x7fea7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.613+0000 7feaa0c00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea7806b500 0x7fea7806d9c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009a80 tx=0x7fea8c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.613+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 msgr2=0x7fea9c0c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.613+0000 7feaa0c00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 0x7fea9c0c7cc0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fea8400c3b0 tx=0x7fea9c0ce780 comp rx=0 tx=0).stop 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 shutdown_connections 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea7806b500 0x7fea7806d9c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea9c0c8200 0x7fea9c0b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea9c0c7840 0x7fea9c0c7cc0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 --2- 172.21.15.97:0/3166351247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea80003ef0 0x7fea9c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 >> 172.21.15.97:0/3166351247 conn(0x7fea9c0d3d10 msgr2=0x7fea9c15d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:53.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 shutdown_connections 2024-03-06T07:36:53.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:53.614+0000 7feaa0c00640 1 -- 172.21.15.97:0/3166351247 wait complete. 2024-03-06T07:36:53.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:53.835 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:53.930 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:54.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:55.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.045+0000 7fe787200640 1 Processor -- start 2024-03-06T07:36:55.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.045+0000 7fe787200640 1 -- start start 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe787200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe78806afc0 0x7fe78806b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe787200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78815fed0 con 0x7fe78815f8f0 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe787200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7881601b0 con 0x7fe788156320 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe787200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe788071dc0 con 0x7fe78806afc0 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe786800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe78806afc0 0x7fe78806b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe785400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe78815f8f0 0x7fe7880ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52672/0 (socket says 172.21.15.97:52672) 2024-03-06T07:36:55.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe785400640 1 -- 172.21.15.97:0/4205737723 learned_addr learned my addr 172.21.15.97:0/4205737723 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:55.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe786800640 1 -- 172.21.15.97:0/4205737723 >> v1:172.21.15.135:6789/0 conn(0x7fe788156320 legacy=0x7fe78815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:55.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe786800640 1 -- 172.21.15.97:0/4205737723 >> v1:172.21.15.97:6789/0 conn(0x7fe78815f8f0 legacy=0x7fe7880ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.047+0000 7fe786800640 1 -- 172.21.15.97:0/4205737723 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe788160490 con 0x7fe78806afc0 2024-03-06T07:36:55.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.049+0000 7fe786800640 1 --2- 172.21.15.97:0/4205737723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe78806afc0 0x7fe78806b400 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fe788071f40 tx=0x7fe77802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add99f921d9295b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.049+0000 7fe784a00640 1 -- 172.21.15.97:0/4205737723 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778039070 con 0x7fe78806afc0 2024-03-06T07:36:55.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.049+0000 7fe784a00640 1 -- 172.21.15.97:0/4205737723 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7780316f0 con 0x7fe78806afc0 2024-03-06T07:36:55.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.049+0000 7fe784a00640 1 -- 172.21.15.97:0/4205737723 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778031ac0 con 0x7fe78806afc0 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.049+0000 7fe787200640 1 -- 172.21.15.97:0/4205737723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe78806afc0 msgr2=0x7fe78806b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.049+0000 7fe787200640 1 --2- 172.21.15.97:0/4205737723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe78806afc0 0x7fe78806b400 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fe788071f40 tx=0x7fe77802b790 comp rx=0 tx=0).stop 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.050+0000 7fe787200640 1 -- 172.21.15.97:0/4205737723 shutdown_connections 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.050+0000 7fe787200640 1 --2- 172.21.15.97:0/4205737723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe78806afc0 0x7fe78806b400 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.050+0000 7fe787200640 1 -- 172.21.15.97:0/4205737723 >> 172.21.15.97:0/4205737723 conn(0x7fe78806c0d0 msgr2=0x7fe78806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.050+0000 7fe787200640 1 -- 172.21.15.97:0/4205737723 shutdown_connections 2024-03-06T07:36:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.050+0000 7fe787200640 1 -- 172.21.15.97:0/4205737723 wait complete. 2024-03-06T07:36:55.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.051+0000 7fe787200640 1 Processor -- start 2024-03-06T07:36:55.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.051+0000 7fe787200640 1 -- start start 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.052+0000 7fe787200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe78806afc0 0x7fe78813a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.052+0000 7fe787200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 0x7fe78815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.052+0000 7fe787200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe78815f8f0 0x7fe78815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.052+0000 7fe787200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe788156700 con 0x7fe788156320 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.052+0000 7fe787200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78806b940 con 0x7fe78806afc0 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.052+0000 7fe787200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe788072250 con 0x7fe78815f8f0 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 0x7fe78815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 0x7fe78815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44776/0 (socket says 172.21.15.97:44776) 2024-03-06T07:36:55.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 -- 172.21.15.97:0/4221916672 learned_addr learned my addr 172.21.15.97:0/4221916672 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785e00640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe78806afc0 0x7fe78813a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 -- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe78815f8f0 msgr2=0x7fe78815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe78815f8f0 0x7fe78815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 -- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe78806afc0 msgr2=0x7fe78813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe78806afc0 0x7fe78813a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 -- 172.21.15.97:0/4221916672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78814c270 con 0x7fe788156320 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785e00640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe78806afc0 0x7fe78813a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.053+0000 7fe785400640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 0x7fe78815c090 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c3c0 tx=0x7fe77400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.054+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe774017070 con 0x7fe788156320 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.054+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe774005ce0 con 0x7fe788156320 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.054+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78814c550 con 0x7fe788156320 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.054+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7740072f0 con 0x7fe788156320 2024-03-06T07:36:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.054+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78814cb10 con 0x7fe788156320 2024-03-06T07:36:55.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.055+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe78804f330 con 0x7fe788156320 2024-03-06T07:36:55.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.056+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe77401a070 con 0x7fe788156320 2024-03-06T07:36:55.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.057+0000 7fe77e000640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe76806b4d0 0x7fe76806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.057+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe774089750 con 0x7fe788156320 2024-03-06T07:36:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.058+0000 7fe785e00640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe76806b4d0 0x7fe76806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.058+0000 7fe785e00640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe76806b4d0 0x7fe76806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe780004520 tx=0x7fe780009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.061+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe774058460 con 0x7fe788156320 2024-03-06T07:36:55.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.296+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7880cda80 con 0x7fe788156320 2024-03-06T07:36:55.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.297+0000 7fe77e000640 1 -- 172.21.15.97:0/4221916672 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7fe774057d60 con 0x7fe788156320 2024-03-06T07:36:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:55.299 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:55.299 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.299+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe76806b4d0 msgr2=0x7fe76806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.299+0000 7fe787200640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe76806b4d0 0x7fe76806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe780004520 tx=0x7fe780009040 comp rx=0 tx=0).stop 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.299+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 msgr2=0x7fe78815c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.299+0000 7fe787200640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 0x7fe78815c090 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c3c0 tx=0x7fe77400c880 comp rx=0 tx=0).stop 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 shutdown_connections 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe76806b4d0 0x7fe76806d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe78815f8f0 0x7fe78815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe788156320 0x7fe78815c090 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 --2- 172.21.15.97:0/4221916672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe78806afc0 0x7fe78813a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 >> 172.21.15.97:0/4221916672 conn(0x7fe78806c0d0 msgr2=0x7fe7880cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 shutdown_connections 2024-03-06T07:36:55.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:55.300+0000 7fe787200640 1 -- 172.21.15.97:0/4221916672 wait complete. 2024-03-06T07:36:55.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:55.526 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:55.932 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:56.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.738+0000 7f1484400640 1 Processor -- start 2024-03-06T07:36:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.738+0000 7f1484400640 1 -- start start 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1484400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1484400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f147c15f470 0x7f147c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1484400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c0cea60 con 0x7f147c0d0df0 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1484400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147c0d1260 con 0x7f147c15f470 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1484400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0d2f60 con 0x7f147c15e0d0 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1481000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f147c15f470 0x7f147c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1481000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f147c15f470 0x7f147c0d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44790/0 (socket says 172.21.15.97:44790) 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1482400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f147c0d0df0 0x7f147c160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52674/0 (socket says 172.21.15.97:52674) 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1481000640 1 -- 172.21.15.97:0/1353347975 learned_addr learned my addr 172.21.15.97:0/1353347975 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.740+0000 7f1481a00640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.741+0000 7f1481000640 1 -- 172.21.15.97:0/1353347975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 msgr2=0x7f147c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.741+0000 7f1481000640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.741+0000 7f1481000640 1 -- 172.21.15.97:0/1353347975 >> v1:172.21.15.97:6789/0 conn(0x7f147c0d0df0 legacy=0x7f147c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.741+0000 7f1481000640 1 -- 172.21.15.97:0/1353347975 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0d2350 con 0x7f147c15f470 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.741+0000 7f1481a00640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.741+0000 7f147be00640 1 -- 172.21.15.97:0/1353347975 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327298127 0 0) 0x7f147c0cea60 con 0x7f147c0d0df0 2024-03-06T07:36:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.742+0000 7f1481000640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c15f470 0x7f147c0d08b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f1468009870 tx=0x7f146802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e909bd3f58b3a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.742+0000 7f147be00640 1 -- 172.21.15.97:0/1353347975 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146802f840 con 0x7f147c15f470 2024-03-06T07:36:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.742+0000 7f147be00640 1 -- 172.21.15.97:0/1353347975 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146802f9e0 con 0x7f147c15f470 2024-03-06T07:36:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.742+0000 7f147be00640 1 -- 172.21.15.97:0/1353347975 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146802fdb0 con 0x7f147c15f470 2024-03-06T07:36:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 -- 172.21.15.97:0/1353347975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c15f470 msgr2=0x7f147c0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c15f470 0x7f147c0d08b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f1468009870 tx=0x7f146802ba20 comp rx=0 tx=0).stop 2024-03-06T07:36:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 -- 172.21.15.97:0/1353347975 shutdown_connections 2024-03-06T07:36:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c15f470 0x7f147c0d08b0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 --2- 172.21.15.97:0/1353347975 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 -- 172.21.15.97:0/1353347975 >> 172.21.15.97:0/1353347975 conn(0x7f147c0d3ec0 msgr2=0x7f147c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 -- 172.21.15.97:0/1353347975 shutdown_connections 2024-03-06T07:36:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.743+0000 7f1484400640 1 -- 172.21.15.97:0/1353347975 wait complete. 2024-03-06T07:36:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.745+0000 7f1484400640 1 Processor -- start 2024-03-06T07:36:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.745+0000 7f1484400640 1 -- start start 2024-03-06T07:36:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1484400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 0x7f147c0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1484400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1484400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c15f470 0x7f147c0b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1484400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0d2bf0 con 0x7f147c0d0df0 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1484400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0d2910 con 0x7f147c15f470 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1484400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c15fb30 con 0x7f147c15e0d0 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1481a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 0x7f147c0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1481a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 0x7f147c0cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44804/0 (socket says 172.21.15.97:44804) 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1481a00640 1 -- 172.21.15.97:0/4109949045 learned_addr learned my addr 172.21.15.97:0/4109949045 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.747+0000 7f1481000640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1482400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c15f470 0x7f147c0b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481a00640 1 -- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 msgr2=0x7f147c0b7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481a00640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0b7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481a00640 1 -- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c15f470 msgr2=0x7f147c0b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481a00640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c15f470 0x7f147c0b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481a00640 1 -- 172.21.15.97:0/4109949045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0b8790 con 0x7f147c0d0df0 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1482400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c15f470 0x7f147c0b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481000640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1481a00640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 0x7f147c0cb2e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f147000c7f0 tx=0x7f147000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470015070 con 0x7f147c0d0df0 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1470007820 con 0x7f147c0d0df0 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470007bd0 con 0x7f147c0d0df0 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.748+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c0b8a70 con 0x7f147c0d0df0 2024-03-06T07:36:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.749+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c0b9030 con 0x7f147c0d0df0 2024-03-06T07:36:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.750+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c04f330 con 0x7f147c0d0df0 2024-03-06T07:36:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.751+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1470018070 con 0x7f147c0d0df0 2024-03-06T07:36:56.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.752+0000 7f1479600640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f145c06b4d0 0x7f145c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.752+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f147008a380 con 0x7f147c0d0df0 2024-03-06T07:36:56.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.752+0000 7f1481000640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f145c06b4d0 0x7f145c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.753+0000 7f1481000640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f145c06b4d0 0x7f145c06d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1468036000 tx=0x7f1468039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.757+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14700591b0 con 0x7f147c0d0df0 2024-03-06T07:36:56.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.992+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f147c08d920 con 0x7f147c0d0df0 2024-03-06T07:36:56.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.993+0000 7f1479600640 1 -- 172.21.15.97:0/4109949045 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3413 (secure 0 0 0) 0x7f1470058ab0 con 0x7f147c0d0df0 2024-03-06T07:36:56.996 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:56.996 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:27.920501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8382,"mds_1":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"c","rank":1,"incarnation":90,"state":"up:stopping","state_seq":7,"addr":"172.21.15.97:6836/440933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":440933929},{"type":"v1","addr":"172.21.15.97:6836","nonce":440933929}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:56.996 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 97 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.996+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f145c06b4d0 msgr2=0x7f145c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.996+0000 7f1484400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f145c06b4d0 0x7f145c06d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1468036000 tx=0x7f1468039040 comp rx=0 tx=0).stop 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.996+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 msgr2=0x7f147c0cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.996+0000 7f1484400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 0x7f147c0cb2e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f147000c7f0 tx=0x7f147000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 shutdown_connections 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f145c06b4d0 0x7f145c06d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f147c15f470 0x7f147c0b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f147c15e0d0 0x7f147c0b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 --2- 172.21.15.97:0/4109949045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f147c0d0df0 0x7f147c0cb2e0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 >> 172.21.15.97:0/4109949045 conn(0x7f147c0d3ec0 msgr2=0x7f147c0cef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 shutdown_connections 2024-03-06T07:36:57.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:56.997+0000 7f1484400640 1 -- 172.21.15.97:0/4109949045 wait complete. 2024-03-06T07:36:57.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:27.920501+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8382, 'mds_1': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'c', 'rank': 1, 'incarnation': 90, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.97:6836/440933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 440933929}, {'type': 'v1', 'addr': '172.21.15.97:6836', 'nonce': 440933929}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:57.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8379:up:stopping 2024-03-06T07:36:57.934 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:58.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:58.264 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:36:58.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.410+0000 7fe012200640 1 Processor -- start 2024-03-06T07:36:58.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.410+0000 7fe012200640 1 -- start start 2024-03-06T07:36:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe012200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fe00c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe012200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c06cef0 con 0x7fe00c15f9d0 2024-03-06T07:36:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe012200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c06d1d0 con 0x7fe00c144c00 2024-03-06T07:36:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe012200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c06d4b0 con 0x7fe00c15fe60 2024-03-06T07:36:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe00aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe00c15f9d0 0x7fe00c070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52688/0 (socket says 172.21.15.97:52688) 2024-03-06T07:36:58.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe00aa00640 1 -- 172.21.15.97:0/3431995510 learned_addr learned my addr 172.21.15.97:0/3431995510 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.412+0000 7fe00be00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fe00c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506779624 0 0) 0x7fe00c06cef0 con 0x7fe00c15f9d0 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff0003520 con 0x7fe00c15f9d0 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811613581 0 0) 0x7fdff0003520 con 0x7fe00c15f9d0 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 msgr2=0x7fe00c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fe00c071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 >> v1:172.21.15.135:6789/0 conn(0x7fe00c144c00 legacy=0x7fe00c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c06d790 con 0x7fe00c15f9d0 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdff8001670 con 0x7fe00c15f9d0 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 >> v1:172.21.15.97:6789/0 conn(0x7fe00c15f9d0 legacy=0x7fe00c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff00039e0 0x7fdff0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdff0004800 0x7fdff0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fdff00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c06d4b0 con 0x7fdff0004800 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff000ba70 con 0x7fdff00039e0 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff000bc70 con 0x7fe00c15fe60 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.413+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdff0004800 0x7fdff0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00be00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fdff00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00aa00640 1 -- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 msgr2=0x7fdff00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fdff00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00b400640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff00039e0 0x7fdff0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00aa00640 1 -- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff00039e0 msgr2=0x7fdff0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff00039e0 0x7fdff0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00aa00640 1 -- 172.21.15.97:0/3431995510 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0003520 con 0x7fdff0004800 2024-03-06T07:36:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00be00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c15fe60 0x7fdff00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.414+0000 7fe00b400640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff00039e0 0x7fdff0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:58.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.415+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdff0004800 0x7fdff0004c80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fdff8001050 tx=0x7fdff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741cc927cb435ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.415+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8007d30 con 0x7fdff0004800 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.415+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8030070 con 0x7fdff0004800 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.415+0000 7fe00a000640 1 -- 172.21.15.97:0/3431995510 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8005d30 con 0x7fdff0004800 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe012200640 1 -- 172.21.15.97:0/3431995510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdff0004800 msgr2=0x7fdff0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe012200640 1 --2- 172.21.15.97:0/3431995510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdff0004800 0x7fdff0004c80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fdff8001050 tx=0x7fdff0005830 comp rx=0 tx=0).stop 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe00be00640 1 -- 172.21.15.97:0/3431995510 reap_dead start 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe012200640 1 -- 172.21.15.97:0/3431995510 shutdown_connections 2024-03-06T07:36:58.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe012200640 1 -- 172.21.15.97:0/3431995510 >> 172.21.15.97:0/3431995510 conn(0x7fe00c0b1730 msgr2=0x7fe00c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe012200640 1 -- 172.21.15.97:0/3431995510 shutdown_connections 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.416+0000 7fe012200640 1 -- 172.21.15.97:0/3431995510 wait complete. 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.418+0000 7fe012200640 1 Processor -- start 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.418+0000 7fe012200640 1 -- start start 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe012200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff0004800 0x7fe00c1f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe012200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 0x7fe00c0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe012200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c0b32e0 0x7fe00c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe012200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c06cb50 con 0x7fe00c0b4570 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe012200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c160730 con 0x7fdff0004800 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe012200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c06ccd0 con 0x7fe00c0b32e0 2024-03-06T07:36:58.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe00aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 0x7fe00c0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe00aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 0x7fe00c0b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44834/0 (socket says 172.21.15.97:44834) 2024-03-06T07:36:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe00aa00640 1 -- 172.21.15.97:0/3948452375 learned_addr learned my addr 172.21.15.97:0/3948452375 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe00b400640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff0004800 0x7fe00c1f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.419+0000 7fe00aa00640 1 -- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c0b32e0 msgr2=0x7fe00c15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c0b32e0 0x7fe00c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe00aa00640 1 -- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff0004800 msgr2=0x7fe00c1f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff0004800 0x7fe00c1f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe00aa00640 1 -- 172.21.15.97:0/3948452375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c138eb0 con 0x7fe00c0b4570 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe00b400640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff0004800 0x7fe00c1f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe00aa00640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 0x7fe00c0b2da0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fdff800c350 tx=0x7fdff8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8045070 con 0x7fe00c0b4570 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8004bf0 con 0x7fe00c0b4570 2024-03-06T07:36:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8007d30 con 0x7fe00c0b4570 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c139150 con 0x7fe00c0b4570 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.420+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c0b3940 con 0x7fe00c0b4570 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.421+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c04f330 con 0x7fe00c0b4570 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.422+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdff8007070 con 0x7fe00c0b4570 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.423+0000 7fe003400640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfec06b4d0 0x7fdfec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.423+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdff80ab380 con 0x7fe00c0b4570 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.424+0000 7fe00b400640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfec06b4d0 0x7fdfec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.424+0000 7fe00b400640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfec06b4d0 0x7fdfec06d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe00c072810 tx=0x7fdffc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.428+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdff807a2a0 con 0x7fe00c0b4570 2024-03-06T07:36:58.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.663+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe00c0b1120 con 0x7fe00c0b4570 2024-03-06T07:36:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.664+0000 7fe003400640 1 -- 172.21.15.97:0/3948452375 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2609 (secure 0 0 0) 0x7fdff8079ba0 con 0x7fe00c0b4570 2024-03-06T07:36:58.666 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:36:58.666 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:57.921337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:36:58.666 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 98 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfec06b4d0 msgr2=0x7fdfec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfec06b4d0 0x7fdfec06d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe00c072810 tx=0x7fdffc009040 comp rx=0 tx=0).stop 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 msgr2=0x7fe00c0b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 0x7fe00c0b2da0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fdff800c350 tx=0x7fdff8000fe0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 shutdown_connections 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdfec06b4d0 0x7fdfec06d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe00c0b32e0 0x7fe00c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe00c0b4570 0x7fe00c0b2da0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 --2- 172.21.15.97:0/3948452375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdff0004800 0x7fe00c1f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.666+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 >> 172.21.15.97:0/3948452375 conn(0x7fe00c0b1730 msgr2=0x7fe00c0afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:58.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.667+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 shutdown_connections 2024-03-06T07:36:58.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:58.667+0000 7fe012200640 1 -- 172.21.15.97:0/3948452375 wait complete. 2024-03-06T07:36:58.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:43.821565+0000', 'modified': '2024-03-06T07:36:57.921337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8382}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8382': {'gid': 8382, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6837/3564321877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 3564321877}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3564321877}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8382, 'qdb_cluster': [8382]} 2024-03-06T07:36:58.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:36:58.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:36:59.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.103+0000 7f7b82600640 1 Processor -- start 2024-03-06T07:36:59.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.103+0000 7f7b82600640 1 -- start start 2024-03-06T07:36:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b82600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 0x7f7b7c0d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b82600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b7c0ce970 0x7f7b7c0cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b82600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0cf3a0 con 0x7f7b7c0cedd0 2024-03-06T07:36:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b82600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7c06cae0 con 0x7f7b7c0ce970 2024-03-06T07:36:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b82600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c06d0a0 con 0x7f7b7c0ce590 2024-03-06T07:36:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b7be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7b7c0cedd0 0x7f7b7c0cc9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52702/0 (socket says 172.21.15.97:52702) 2024-03-06T07:36:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.105+0000 7f7b7be00640 1 -- 172.21.15.97:0/2764310521 learned_addr learned my addr 172.21.15.97:0/2764310521 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:36:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b72a00640 1 --2- 172.21.15.97:0/2764310521 >> v2:172.21.15.135:3300/0 conn(0x7f7b7c0ce970 0x7f7b7c0cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 0x7f7b7c0d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935588214 0 0) 0x7f7b7c0cf3a0 con 0x7f7b7c0cedd0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b60003520 con 0x7f7b7c0cedd0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632764432 0 0) 0x7f7b60003520 con 0x7f7b7c0cedd0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 msgr2=0x7f7b7c0d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 0x7f7b7c0d1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 >> v2:172.21.15.135:3300/0 conn(0x7f7b7c0ce970 msgr2=0x7f7b7c0cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 --2- 172.21.15.97:0/2764310521 >> v2:172.21.15.135:3300/0 conn(0x7f7b7c0ce970 0x7f7b7c0cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c06cd30 con 0x7f7b7c0cedd0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7b74001670 con 0x7f7b7c0cedd0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 >> v1:172.21.15.97:6789/0 conn(0x7f7b7c0cedd0 legacy=0x7f7b7c0cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 0x7f7b60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b60004d40 0x7f7b600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c06cae0 con 0x7f7b600039b0 2024-03-06T07:36:59.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c06d0a0 con 0x7f7b60004d40 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.106+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6000b9c0 con 0x7f7b7c0ce590 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.107+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.107+0000 7f7b7b400640 1 -- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 msgr2=0x7f7b60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.107+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c0ce590 0x7f7b60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.107+0000 7f7b7b400640 1 -- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b60004d40 msgr2=0x7f7b600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.107+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b60004d40 0x7f7b600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.107+0000 7f7b7b400640 1 -- 172.21.15.97:0/2764310521 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b60003520 con 0x7f7b600039b0 2024-03-06T07:36:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.108+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b60003dd0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009fc0 tx=0x7f7b60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc48e03a57fe7fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.108+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c039070 con 0x7f7b600039b0 2024-03-06T07:36:59.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.108+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c02cc60 con 0x7f7b600039b0 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.108+0000 7f7b7aa00640 1 -- 172.21.15.97:0/2764310521 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c009c10 con 0x7f7b600039b0 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b82600640 1 -- 172.21.15.97:0/2764310521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 msgr2=0x7f7b60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b82600640 1 --2- 172.21.15.97:0/2764310521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b60003dd0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009fc0 tx=0x7f7b60005830 comp rx=0 tx=0).stop 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b7be00640 1 -- 172.21.15.97:0/2764310521 reap_dead start 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b82600640 1 -- 172.21.15.97:0/2764310521 shutdown_connections 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b82600640 1 -- 172.21.15.97:0/2764310521 >> 172.21.15.97:0/2764310521 conn(0x7f7b7c0d1d10 msgr2=0x7f7b7c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b82600640 1 -- 172.21.15.97:0/2764310521 shutdown_connections 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.109+0000 7f7b82600640 1 -- 172.21.15.97:0/2764310521 wait complete. 2024-03-06T07:36:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.111+0000 7f7b82600640 1 Processor -- start 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.111+0000 7f7b82600640 1 -- start start 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b82600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b7c154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b82600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c1f3fb0 0x7f7b7c0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b82600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b7c0cb010 0x7f7b7c0aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b82600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c161b00 con 0x7f7b600039b0 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b82600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c161980 con 0x7f7b7c0cb010 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b82600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c0d1660 con 0x7f7b7c1f3fb0 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b7c154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b7c154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44854/0 (socket says 172.21.15.97:44854) 2024-03-06T07:36:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 -- 172.21.15.97:0/1145135641 learned_addr learned my addr 172.21.15.97:0/1145135641 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 -- 172.21.15.97:0/1145135641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c1f3fb0 msgr2=0x7f7b7c0caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c1f3fb0 0x7f7b7c0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 -- 172.21.15.97:0/1145135641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b7c0cb010 msgr2=0x7f7b7c0aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b7c0cb010 0x7f7b7c0aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.112+0000 7f7b7b400640 1 -- 172.21.15.97:0/1145135641 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c09be20 con 0x7f7b600039b0 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.113+0000 7f7b7b400640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b7c154240 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00e230 tx=0x7f7b6c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.113+0000 7f7b73e00640 1 -- 172.21.15.97:0/1145135641 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c039050 con 0x7f7b600039b0 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.113+0000 7f7b73e00640 1 -- 172.21.15.97:0/1145135641 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c03a040 con 0x7f7b600039b0 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.113+0000 7f7b73e00640 1 -- 172.21.15.97:0/1145135641 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c034e20 con 0x7f7b600039b0 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.113+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c09c100 con 0x7f7b600039b0 2024-03-06T07:36:59.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.113+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c09c6c0 con 0x7f7b600039b0 2024-03-06T07:36:59.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.115+0000 7f7b73e00640 1 -- 172.21.15.97:0/1145135641 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b6c047030 con 0x7f7b600039b0 2024-03-06T07:36:59.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.116+0000 7f7b73e00640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b5806b500 0x7f7b5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.116+0000 7f7b73e00640 1 -- 172.21.15.97:0/1145135641 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b6c0ac550 con 0x7f7b600039b0 2024-03-06T07:36:59.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.117+0000 7f7b72a00640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b5806b500 0x7f7b5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.117+0000 7f7b85227740 1 Processor -- start 2024-03-06T07:36:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.117+0000 7f7b72a00640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b5806b500 0x7f7b5806d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7b640093d0 tx=0x7f7b64009300 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.117+0000 7f7b85227740 1 -- start start 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.118+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08a7b0 0x55f1eb05d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.118+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb08d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.118+0000 7f7b85227740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 0x55f1eb08da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.118+0000 7f7b85227740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eafc7600 con 0x55f1eb05dc90 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.118+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eaf70930 con 0x55f1eb08ab90 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.118+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1eafc36a0 con 0x55f1eb08a7b0 2024-03-06T07:36:59.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 0x55f1eb08da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 0x55f1eb08da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44856/0 (socket says 172.21.15.97:44856) 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 -- 172.21.15.97:0/388056696 learned_addr learned my addr 172.21.15.97:0/388056696 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b72a00640 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb08d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7b400640 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08a7b0 0x55f1eb05d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 -- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08a7b0 msgr2=0x55f1eb05d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08a7b0 0x55f1eb05d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 -- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 msgr2=0x55f1eb08d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb08d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 -- 172.21.15.97:0/388056696 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1eacdc2d0 con 0x55f1eb05dc90 2024-03-06T07:36:59.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.119+0000 7f7b7be00640 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 0x55f1eb08da80 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0cf1b0 tx=0x7f7b74007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9add28c6efce08eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b6be00640 1 -- 172.21.15.97:0/388056696 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74004bf0 con 0x55f1eb05dc90 2024-03-06T07:36:59.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b6be00640 1 -- 172.21.15.97:0/388056696 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b74004d90 con 0x55f1eb05dc90 2024-03-06T07:36:59.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b6be00640 1 -- 172.21.15.97:0/388056696 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74007270 con 0x55f1eb05dc90 2024-03-06T07:36:59.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 -- 172.21.15.97:0/388056696 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 msgr2=0x55f1eb08da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 0x55f1eb08da80 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0cf1b0 tx=0x7f7b74007d20 comp rx=0 tx=0).stop 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 -- 172.21.15.97:0/388056696 shutdown_connections 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb05dc90 0x55f1eb08da80 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb08d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 --2- 172.21.15.97:0/388056696 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08a7b0 0x55f1eb05d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 -- 172.21.15.97:0/388056696 >> 172.21.15.97:0/388056696 conn(0x55f1eb05f9c0 msgr2=0x55f1eb0a8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 -- 172.21.15.97:0/388056696 shutdown_connections 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.120+0000 7f7b85227740 1 -- 172.21.15.97:0/388056696 wait complete. 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.121+0000 7f7b85227740 1 -- start start 2024-03-06T07:36:59.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.121+0000 7f7b85227740 1 Processor -- start 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb07bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb0b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb090870 0x55f1eb0a82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb061f60 con 0x55f1eb08a7b0 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eaf83d90 con 0x55f1eb090870 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb061350 con 0x55f1eb08ab90 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b85227740 10 client.0 _ll_register_callbacks cb 0x55f1eafbf350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb07bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.122+0000 7f7b7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb07bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44872/0 (socket says 172.21.15.97:44872) 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 -- 172.21.15.97:0/2266485896 learned_addr learned my addr 172.21.15.97:0/2266485896 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:59.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 -- 172.21.15.97:0/2266485896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 msgr2=0x55f1eb0b99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb0b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 -- 172.21.15.97:0/2266485896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb090870 msgr2=0x55f1eb0a82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb090870 0x55f1eb0a82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 -- 172.21.15.97:0/2266485896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1eada66b0 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb07bb00 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c082e60 tx=0x7f7b6c0810b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff575657807193d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b68c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b68c00640 1 -- 172.21.15.97:0/2266485896 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c092070 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b68c00640 1 -- 172.21.15.97:0/2266485896 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b6c0772f0 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.123+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b74004d90 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b68c00640 1 -- 172.21.15.97:0/2266485896 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b6c07ac80 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 10 client.8820 fetch_fsmap learned FSMap version 98 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x55f1eb090350 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b68c00640 1 -- 172.21.15.97:0/2266485896 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b74004d90 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b68c00640 1 -- 172.21.15.97:0/2266485896 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 98) v1 ==== 1597+0+0 (secure 0 0 0) 0x7f7b6c082f00 con 0x55f1eb08a7b0 2024-03-06T07:36:59.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 10 client.8820 fetch_fsmap finished waiting for FSMap version 98 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 10 client.8820 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 20 client.8820 populate_metadata read hostname 'smithi097' 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0a87e0 0x55f1eb0a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 4 client.8820 mds_command: new command op to 8382 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 --> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1eafd4b40 con 0x55f1eb0a87e0 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.124+0000 7f7b72a00640 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0a87e0 0x55f1eb0a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.125+0000 7f7b72a00640 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0a87e0 0x55f1eb0a65b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b73b0188a8da55be server_cookie=e116519cfc3aa767 in_seq=0 out_seq=0 2024-03-06T07:36:59.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.125+0000 7f7b68c00640 10 client.8820 ms_handle_connect on v2:172.21.15.97:6835/3564321877 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b68c00640 1 -- 172.21.15.97:0/2266485896 <== mds.0 v2:172.21.15.97:6835/3564321877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f1eafd4b40 con 0x55f1eb0a87e0 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b68c00640 10 client.8820 handle_command_reply: tid=0 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 client.8820 shutdown 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 msgr2=0x55f1eb07bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb07bb00 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c082e60 tx=0x7f7b6c0810b0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 shutdown_connections 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0a87e0 0x55f1eb0a65b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb090870 0x55f1eb0a82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb0b99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 --2- 172.21.15.97:0/2266485896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb07bb00 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 >> 172.21.15.97:0/2266485896 conn(0x55f1eb05dc90 msgr2=0x55f1eb08d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b68c00640 0 client.8820 ms_handle_reset on v2:172.21.15.97:6835/3564321877 2024-03-06T07:36:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 shutdown_connections 2024-03-06T07:36:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.126+0000 7f7b85227740 1 -- 172.21.15.97:0/2266485896 wait complete. 2024-03-06T07:36:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.127+0000 7f7b85227740 20 client.8820 trim_cache size 0 max 16384 2024-03-06T07:36:59.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.163+0000 7f7b85227740 1 Processor -- start 2024-03-06T07:36:59.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.163+0000 7f7b85227740 1 -- start start 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.164+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0aae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.164+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb07b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.164+0000 7f7b85227740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 0x55f1eb111c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.164+0000 7f7b85227740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c1ff870 con 0x55f1eb090870 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.164+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb061f60 con 0x55f1eb08a7b0 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.164+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb07b580 con 0x55f1eb08ab90 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 0x55f1eb111c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 0x55f1eb111c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44884/0 (socket says 172.21.15.97:44884) 2024-03-06T07:36:59.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 -- 172.21.15.97:0/2963593159 learned_addr learned my addr 172.21.15.97:0/2963593159 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b72a00640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb07b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 -- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 msgr2=0x55f1eb07b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb07b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 -- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08a7b0 msgr2=0x55f1eb0aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0aae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7be00640 1 -- 172.21.15.97:0/2963593159 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1eb0cc520 con 0x55f1eb090870 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b7b400640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.165+0000 7f7b72a00640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb07b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.166+0000 7f7b7be00640 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 0x55f1eb111c30 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f7b74007d60 tx=0x7f7b7400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db23ca8ea9032b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.166+0000 7f7b6b400640 1 -- 172.21.15.97:0/2963593159 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74004cf0 con 0x55f1eb090870 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.166+0000 7f7b6b400640 1 -- 172.21.15.97:0/2963593159 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b74017030 con 0x55f1eb090870 2024-03-06T07:36:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.166+0000 7f7b6b400640 1 -- 172.21.15.97:0/2963593159 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74007230 con 0x55f1eb090870 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 -- 172.21.15.97:0/2963593159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 msgr2=0x55f1eb111c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 0x55f1eb111c30 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f7b74007d60 tx=0x7f7b7400f000 comp rx=0 tx=0).stop 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 -- 172.21.15.97:0/2963593159 shutdown_connections 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb090870 0x55f1eb111c30 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb08ab90 0x55f1eb07b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 --2- 172.21.15.97:0/2963593159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 -- 172.21.15.97:0/2963593159 >> 172.21.15.97:0/2963593159 conn(0x55f1eb05dc90 msgr2=0x55f1eb089980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 -- 172.21.15.97:0/2963593159 shutdown_connections 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.167+0000 7f7b85227740 1 -- 172.21.15.97:0/2963593159 wait complete. 2024-03-06T07:36:59.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.168+0000 7f7b85227740 1 -- start start 2024-03-06T07:36:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.168+0000 7f7b85227740 1 Processor -- start 2024-03-06T07:36:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.169+0000 7f7b85227740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.169+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb0a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.169+0000 7f7b85227740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb090870 0x55f1eb0a2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.169+0000 7f7b85227740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb064bb0 con 0x55f1eb08a7b0 2024-03-06T07:36:59.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.169+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb0b5bf0 con 0x55f1eb08ab90 2024-03-06T07:36:59.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.169+0000 7f7b85227740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1eb0aa9a0 con 0x55f1eb090870 2024-03-06T07:36:59.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.170+0000 7f7b85227740 10 client.0 _ll_register_callbacks cb 0x7f7b6c009390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.170+0000 7f7b7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.170+0000 7f7b7b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0fe8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44890/0 (socket says 172.21.15.97:44890) 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.170+0000 7f7b7b400640 1 -- 172.21.15.97:0/3685830898 learned_addr learned my addr 172.21.15.97:0/3685830898 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.170+0000 7f7b72a00640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb0a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.170+0000 7f7b7be00640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb090870 0x55f1eb0a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7b400640 1 -- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb090870 msgr2=0x55f1eb0a2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7b400640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb090870 0x55f1eb0a2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7b400640 1 -- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 msgr2=0x55f1eb0a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7b400640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb0a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7b400640 1 -- 172.21.15.97:0/3685830898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1eb0aabf0 con 0x55f1eb08a7b0 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7be00640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb090870 0x55f1eb0a2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:59.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b7b400640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0fe8b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c081010 tx=0x7f7b6c0aa900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dd721003637d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.171+0000 7f7b68c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b68c00640 1 -- 172.21.15.97:0/3685830898 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c09e060 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b68c00640 1 -- 172.21.15.97:0/3685830898 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b6c099b50 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b74017030 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b68c00640 1 -- 172.21.15.97:0/3685830898 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b6c0a4050 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 10 client.8826 fetch_fsmap learned FSMap version 99 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) v3 -- 0x55f1eb0aabf0 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b68c00640 1 -- 172.21.15.97:0/3685830898 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b74017030 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b68c00640 1 -- 172.21.15.97:0/3685830898 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 99) v1 ==== 2172+0+0 (secure 0 0 0) 0x7f7b6c099070 con 0x55f1eb08a7b0 2024-03-06T07:36:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 10 client.8826 fetch_fsmap finished waiting for FSMap version 99 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 10 client.8826 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 20 client.8826 populate_metadata read hostname 'smithi097' 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0ba6d0 0x55f1eb0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 4 client.8826 mds_command: new command op to 8382 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.172+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 --> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f1eafcee90 con 0x55f1eb0ba6d0 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.173+0000 7f7b72a00640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0ba6d0 0x55f1eb0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.173+0000 7f7b72a00640 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0ba6d0 0x55f1eb0b4c00 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d8006ddaaa79e8 server_cookie=c3b6669be93e94b0 in_seq=0 out_seq=0 2024-03-06T07:36:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.173+0000 7f7b68c00640 10 client.8826 ms_handle_connect on v2:172.21.15.97:6835/3564321877 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.173+0000 7f7b68c00640 1 -- 172.21.15.97:0/3685830898 <== mds.0 v2:172.21.15.97:6835/3564321877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55f1eafcee90 con 0x55f1eb0ba6d0 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.173+0000 7f7b68c00640 10 client.8826 handle_command_reply: tid=0 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 client.8826 shutdown 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 msgr2=0x55f1eb0fe8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0fe8b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c081010 tx=0x7f7b6c0aa900 comp rx=0 tx=0).stop 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 shutdown_connections 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:6835/3564321877,v1:172.21.15.97:6837/3564321877] conn(0x55f1eb0ba6d0 0x55f1eb0b4c00 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x55f1eb090870 0x55f1eb0a2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f1eb08ab90 0x55f1eb0a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 --2- 172.21.15.97:0/3685830898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f1eb08a7b0 0x55f1eb0fe8b0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 >> 172.21.15.97:0/3685830898 conn(0x55f1eb05dc90 msgr2=0x55f1eb089980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b68c00640 0 client.8826 ms_handle_reset on v2:172.21.15.97:6835/3564321877 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 shutdown_connections 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 1 -- 172.21.15.97:0/3685830898 wait complete. 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.174+0000 7f7b85227740 20 client.8826 trim_cache size 0 max 16384 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 8382, 2024-03-06T07:36:59.178 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 74.805036788999999, 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 98, 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 134, 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 134, 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 81.258741368000003 2024-03-06T07:36:59.179 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b5806b500 msgr2=0x7f7b5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b5806b500 0x7f7b5806d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7b640093d0 tx=0x7f7b64009300 comp rx=0 tx=0).stop 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 msgr2=0x7f7b7c154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b7c154240 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00e230 tx=0x7f7b6c009750 comp rx=0 tx=0).stop 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 shutdown_connections 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b5806b500 0x7f7b5806d9c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b7c0cb010 0x7f7b7c0aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b7c1f3fb0 0x7f7b7c0caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 --2- 172.21.15.97:0/1145135641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b600039b0 0x7f7b7c154240 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 >> 172.21.15.97:0/1145135641 conn(0x7f7b7c0d1d10 msgr2=0x7f7b7c0cd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 shutdown_connections 2024-03-06T07:36:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59.175+0000 7f7b82600640 1 -- 172.21.15.97:0/1145135641 wait complete. 2024-03-06T07:36:59.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:36:59.936 INFO:teuthology.orchestra.run.smithi097.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:36:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:36:59.961 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:36:59.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:37:00.116 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:37:00.234 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:37:00.234 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:00.235 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:00.289 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:00.289 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:00.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:00.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.578+0000 7f9934800640 1 Processor -- start 2024-03-06T07:37:00.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.578+0000 7f9934800640 1 -- start start 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.579+0000 7f9934800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f992c06f1b0 0x7f992c06f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.579+0000 7f9934800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f992c0cb460 0x7f992c0cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.579+0000 7f9934800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c071df0 con 0x7f992c06afc0 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.579+0000 7f9934800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c071ba0 con 0x7f992c0cb460 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.579+0000 7f9934800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c072a00 con 0x7f992c06f1b0 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.580+0000 7f9931e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f992c06afc0 0x7f992c06b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52718/0 (socket says 172.21.15.97:52718) 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.580+0000 7f9931e00640 1 -- 172.21.15.97:0/1605912068 learned_addr learned my addr 172.21.15.97:0/1605912068 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.580+0000 7f9932800640 1 --2- 172.21.15.97:0/1605912068 >> v2:172.21.15.97:3300/0 conn(0x7f992c0cb460 0x7f992c0cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.580+0000 7f9930a00640 1 -- 172.21.15.97:0/1605912068 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069145317 0 0) 0x7f992c071df0 con 0x7f992c06afc0 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.580+0000 7f9930a00640 1 -- 172.21.15.97:0/1605912068 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9914003520 con 0x7f992c06afc0 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.580+0000 7f9931400640 1 --2- 172.21.15.97:0/1605912068 >> v2:172.21.15.135:3300/0 conn(0x7f992c06f1b0 0x7f992c06f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.581+0000 7f9932800640 1 -- 172.21.15.97:0/1605912068 >> v2:172.21.15.135:3300/0 conn(0x7f992c06f1b0 msgr2=0x7f992c06f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.581+0000 7f9932800640 1 --2- 172.21.15.97:0/1605912068 >> v2:172.21.15.135:3300/0 conn(0x7f992c06f1b0 0x7f992c06f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.581+0000 7f9932800640 1 -- 172.21.15.97:0/1605912068 >> v1:172.21.15.97:6789/0 conn(0x7f992c06afc0 legacy=0x7f992c06b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.581+0000 7f9932800640 1 -- 172.21.15.97:0/1605912068 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c0720d0 con 0x7f992c0cb460 2024-03-06T07:37:00.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.582+0000 7f9932800640 1 --2- 172.21.15.97:0/1605912068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0cb8f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f992c0d0b70 tx=0x7f992802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d3b45ac28084ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.582+0000 7f9930a00640 1 -- 172.21.15.97:0/1605912068 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928039070 con 0x7f992c0cb460 2024-03-06T07:37:00.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.582+0000 7f9930a00640 1 -- 172.21.15.97:0/1605912068 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992802cc70 con 0x7f992c0cb460 2024-03-06T07:37:00.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.582+0000 7f9930a00640 1 -- 172.21.15.97:0/1605912068 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928034660 con 0x7f992c0cb460 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.582+0000 7f9934800640 1 -- 172.21.15.97:0/1605912068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 msgr2=0x7f992c0cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.582+0000 7f9934800640 1 --2- 172.21.15.97:0/1605912068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0cb8f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f992c0d0b70 tx=0x7f992802bec0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.583+0000 7f9934800640 1 -- 172.21.15.97:0/1605912068 shutdown_connections 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.583+0000 7f9934800640 1 --2- 172.21.15.97:0/1605912068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0cb8f0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.583+0000 7f9934800640 1 --2- 172.21.15.97:0/1605912068 >> v2:172.21.15.135:3300/0 conn(0x7f992c06f1b0 0x7f992c06f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.583+0000 7f9934800640 1 -- 172.21.15.97:0/1605912068 >> 172.21.15.97:0/1605912068 conn(0x7f992c0d0ba0 msgr2=0x7f992c0d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.583+0000 7f9934800640 1 -- 172.21.15.97:0/1605912068 shutdown_connections 2024-03-06T07:37:00.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.583+0000 7f9934800640 1 -- 172.21.15.97:0/1605912068 wait complete. 2024-03-06T07:37:00.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.584+0000 7f9934800640 1 Processor -- start 2024-03-06T07:37:00.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.585+0000 7f9934800640 1 -- start start 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9934800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f992c06afc0 0x7f992c1411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9934800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f992c06f1b0 0x7f992c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9934800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9934800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c1459c0 con 0x7f992c0cb460 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9934800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c072690 con 0x7f992c06f1b0 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9934800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992c145b40 con 0x7f992c06afc0 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44910/0 (socket says 172.21.15.97:44910) 2024-03-06T07:37:00.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 -- 172.21.15.97:0/2999867010 learned_addr learned my addr 172.21.15.97:0/2999867010 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:00.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9931e00640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f992c06afc0 0x7f992c1411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9931400640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f992c06f1b0 0x7f992c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 -- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f992c06afc0 msgr2=0x7f992c1411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f992c06afc0 0x7f992c1411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 -- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f992c06f1b0 msgr2=0x7f992c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f992c06f1b0 0x7f992c0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9932800640 1 -- 172.21.15.97:0/2999867010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c0c7a50 con 0x7f992c0cb460 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.586+0000 7f9931400640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f992c06f1b0 0x7f992c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.587+0000 7f9932800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0c9690 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f9928009e80 tx=0x7f9928004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.587+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928039050 con 0x7f992c0cb460 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.587+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9928045070 con 0x7f992c0cb460 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.587+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c1f39d0 con 0x7f992c0cb460 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.587+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928009450 con 0x7f992c0cb460 2024-03-06T07:37:00.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.587+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c1f3f90 con 0x7f992c0cb460 2024-03-06T07:37:00.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.588+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c0b1d60 con 0x7f992c0cb460 2024-03-06T07:37:00.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.589+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f992803e850 con 0x7f992c0cb460 2024-03-06T07:37:00.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.591+0000 7f9926000640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f991006b4d0 0x7f991006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.591+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f99280ad2b0 con 0x7f992c0cb460 2024-03-06T07:37:00.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.591+0000 7f9931e00640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f991006b4d0 0x7f991006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.592+0000 7f9931e00640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f991006b4d0 0x7f991006d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9920000f60 tx=0x7f9920009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.594+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f992c0b1d60 con 0x7f992c0cb460 2024-03-06T07:37:00.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.942+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f992c04f330 con 0x7f992c0cb460 2024-03-06T07:37:00.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.944+0000 7f9926000640 1 -- 172.21.15.97:0/2999867010 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16709 (secure 0 0 0) 0x7f992802ce20 con 0x7f992c0cb460 2024-03-06T07:37:00.946 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:00.947 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":134,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:36:29.869182+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.946+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f991006b4d0 msgr2=0x7f991006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.946+0000 7f9934800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f991006b4d0 0x7f991006d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9920000f60 tx=0x7f9920009040 comp rx=0 tx=0).stop 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.946+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 msgr2=0x7f992c0c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.946+0000 7f9934800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0c9690 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f9928009e80 tx=0x7f9928004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 shutdown_connections 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f991006b4d0 0x7f991006d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f992c0cb460 0x7f992c0c9690 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f992c06f1b0 0x7f992c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 --2- 172.21.15.97:0/2999867010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f992c06afc0 0x7f992c1411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 >> 172.21.15.97:0/2999867010 conn(0x7f992c0d0ba0 msgr2=0x7f992c142590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 shutdown_connections 2024-03-06T07:37:00.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:00.947+0000 7f9934800640 1 -- 172.21.15.97:0/2999867010 wait complete. 2024-03-06T07:37:00.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:41.634478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:42.840855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:37:00.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:37:01.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.172+0000 7ff999e00640 1 Processor -- start 2024-03-06T07:37:01.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.173+0000 7ff999e00640 1 -- start start 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff999e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff99415fc20 0x7ff9941629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff999e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9940ce3a0 0x7ff9940ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff999e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9940ccc40 con 0x7ff9940b3310 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff999e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9940d05c0 con 0x7ff9940ce3a0 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff999e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940d08a0 con 0x7ff99415fc20 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff992a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff9940b3310 0x7ff9940cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52720/0 (socket says 172.21.15.97:52720) 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff992a00640 1 -- 172.21.15.97:0/2861862089 learned_addr learned my addr 172.21.15.97:0/2861862089 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff98a000640 1 --2- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff99415fc20 0x7ff9941629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.175+0000 7ff993400640 1 --2- 172.21.15.97:0/2861862089 >> v2:172.21.15.135:3300/0 conn(0x7ff9940ce3a0 0x7ff9940ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.176+0000 7ff993400640 1 -- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff99415fc20 msgr2=0x7ff9941629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.176+0000 7ff993400640 1 --2- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff99415fc20 0x7ff9941629e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.176+0000 7ff993400640 1 -- 172.21.15.97:0/2861862089 >> v1:172.21.15.97:6789/0 conn(0x7ff9940b3310 legacy=0x7ff9940cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.176+0000 7ff993400640 1 -- 172.21.15.97:0/2861862089 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940d0b80 con 0x7ff9940ce3a0 2024-03-06T07:37:01.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.178+0000 7ff993400640 1 --2- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940ce3a0 0x7ff9940ce830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0097a0 tx=0x7ff98c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101bf1a774bce318 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.181+0000 7ff992000640 1 -- 172.21.15.97:0/2861862089 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c031780 con 0x7ff9940ce3a0 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.181+0000 7ff992000640 1 -- 172.21.15.97:0/2861862089 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c037070 con 0x7ff9940ce3a0 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.181+0000 7ff992000640 1 -- 172.21.15.97:0/2861862089 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c031b40 con 0x7ff9940ce3a0 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 -- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940ce3a0 msgr2=0x7ff9940ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 --2- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940ce3a0 0x7ff9940ce830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0097a0 tx=0x7ff98c02b820 comp rx=0 tx=0).stop 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 -- 172.21.15.97:0/2861862089 shutdown_connections 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 --2- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940ce3a0 0x7ff9940ce830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 --2- 172.21.15.97:0/2861862089 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff99415fc20 0x7ff9941629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 -- 172.21.15.97:0/2861862089 >> 172.21.15.97:0/2861862089 conn(0x7ff9940b3790 msgr2=0x7ff9940b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 -- 172.21.15.97:0/2861862089 shutdown_connections 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.182+0000 7ff999e00640 1 -- 172.21.15.97:0/2861862089 wait complete. 2024-03-06T07:37:01.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.185+0000 7ff999e00640 1 Processor -- start 2024-03-06T07:37:01.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.185+0000 7ff999e00640 1 -- start start 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff999e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940b3310 0x7ff99415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff999e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9940ce3a0 0x7ff9940caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff999e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 0x7ff9940b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff999e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cd2a0 con 0x7ff99415fc20 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff999e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cd120 con 0x7ff9940b3310 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff999e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cd420 con 0x7ff9940ce3a0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff993400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 0x7ff9940b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff993400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 0x7ff9940b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44920/0 (socket says 172.21.15.97:44920) 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.186+0000 7ff993400640 1 -- 172.21.15.97:0/605115589 learned_addr learned my addr 172.21.15.97:0/605115589 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff98a000640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9940ce3a0 0x7ff9940caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff992a00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940b3310 0x7ff99415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff993400640 1 -- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9940ce3a0 msgr2=0x7ff9940caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff993400640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9940ce3a0 0x7ff9940caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff993400640 1 -- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940b3310 msgr2=0x7ff99415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff993400640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940b3310 0x7ff99415cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff993400640 1 -- 172.21.15.97:0/605115589 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940c9de0 con 0x7ff99415fc20 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff993400640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 0x7ff9940b4d70 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0098c0 tx=0x7ff98c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.187+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c032610 con 0x7ff99415fc20 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.188+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0327b0 con 0x7ff99415fc20 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.188+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9940ca0c0 con 0x7ff99415fc20 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.188+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c032ba0 con 0x7ff99415fc20 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.188+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9940c8420 con 0x7ff99415fc20 2024-03-06T07:37:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.189+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99404f330 con 0x7ff99415fc20 2024-03-06T07:37:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.190+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff98c0453f0 con 0x7ff99415fc20 2024-03-06T07:37:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.191+0000 7ff98be00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff97006b500 0x7ff97006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.191+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff98c0abf40 con 0x7ff99415fc20 2024-03-06T07:37:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.192+0000 7ff992a00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff97006b500 0x7ff97006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.192+0000 7ff992a00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff97006b500 0x7ff97006d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff9840012e0 tx=0x7ff984009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.195+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff98c040070 con 0x7ff99415fc20 2024-03-06T07:37:01.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.393+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9940b7be0 con 0x7ff99415fc20 2024-03-06T07:37:01.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.394+0000 7ff98be00640 1 -- 172.21.15.97:0/605115589 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff98c02f050 con 0x7ff99415fc20 2024-03-06T07:37:01.396 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff97006b500 msgr2=0x7ff97006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff97006b500 0x7ff97006d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff9840012e0 tx=0x7ff984009040 comp rx=0 tx=0).stop 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 msgr2=0x7ff9940b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 0x7ff9940b4d70 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0098c0 tx=0x7ff98c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 shutdown_connections 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff97006b500 0x7ff97006d9c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff99415fc20 0x7ff9940b4d70 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff9940ce3a0 0x7ff9940caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.396+0000 7ff999e00640 1 --2- 172.21.15.97:0/605115589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9940b3310 0x7ff99415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.397+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 >> 172.21.15.97:0/605115589 conn(0x7ff9940b3790 msgr2=0x7ff99406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.397+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 shutdown_connections 2024-03-06T07:37:01.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.397+0000 7ff999e00640 1 -- 172.21.15.97:0/605115589 wait complete. 2024-03-06T07:37:01.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:37:01.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.624+0000 7f209bc00640 1 Processor -- start 2024-03-06T07:37:01.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.625+0000 7f209bc00640 1 -- start start 2024-03-06T07:37:01.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.626+0000 7f209bc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f20940b3f70 0x7f2094160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.626+0000 7f209bc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094071440 con 0x7f209415fc10 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.626+0000 7f209bc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20940b46c0 con 0x7f2094070ff0 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.626+0000 7f209bc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20940b49a0 con 0x7f20940b3f70 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.626+0000 7f2099200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f209415fc10 0x7f209415fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52722/0 (socket says 172.21.15.97:52722) 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.626+0000 7f2099200640 1 -- 172.21.15.97:0/2113012986 learned_addr learned my addr 172.21.15.97:0/2113012986 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.627+0000 7f2093e00640 1 --2- 172.21.15.97:0/2113012986 >> v2:172.21.15.97:3300/0 conn(0x7f20940b3f70 0x7f2094160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.627+0000 7f2093400640 1 -- 172.21.15.97:0/2113012986 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244898291 0 0) 0x7f2094071440 con 0x7f209415fc10 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.627+0000 7f2093400640 1 -- 172.21.15.97:0/2113012986 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f207c003520 con 0x7f209415fc10 2024-03-06T07:37:01.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.627+0000 7f2093e00640 1 -- 172.21.15.97:0/2113012986 >> v1:172.21.15.135:6789/0 conn(0x7f2094070ff0 legacy=0x7f20940ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.627+0000 7f2093e00640 1 -- 172.21.15.97:0/2113012986 >> v1:172.21.15.97:6789/0 conn(0x7f209415fc10 legacy=0x7f209415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.627+0000 7f2093e00640 1 -- 172.21.15.97:0/2113012986 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094072c90 con 0x7f20940b3f70 2024-03-06T07:37:01.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.628+0000 7f2093e00640 1 --2- 172.21.15.97:0/2113012986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f2094160700 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f20800097a0 tx=0x7f208002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8a08a4ee54a439 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.628+0000 7f2093400640 1 -- 172.21.15.97:0/2113012986 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080035020 con 0x7f20940b3f70 2024-03-06T07:37:01.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f2093400640 1 -- 172.21.15.97:0/2113012986 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2080031830 con 0x7f20940b3f70 2024-03-06T07:37:01.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f2093400640 1 -- 172.21.15.97:0/2113012986 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080031be0 con 0x7f20940b3f70 2024-03-06T07:37:01.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 -- 172.21.15.97:0/2113012986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 msgr2=0x7f2094160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 --2- 172.21.15.97:0/2113012986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f2094160700 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f20800097a0 tx=0x7f208002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 -- 172.21.15.97:0/2113012986 shutdown_connections 2024-03-06T07:37:01.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 --2- 172.21.15.97:0/2113012986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f2094160700 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 -- 172.21.15.97:0/2113012986 >> 172.21.15.97:0/2113012986 conn(0x7f20940b1940 msgr2=0x7f20940b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 -- 172.21.15.97:0/2113012986 shutdown_connections 2024-03-06T07:37:01.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.629+0000 7f209bc00640 1 -- 172.21.15.97:0/2113012986 wait complete. 2024-03-06T07:37:01.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.631+0000 7f209bc00640 1 Processor -- start 2024-03-06T07:37:01.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.631+0000 7f209bc00640 1 -- start start 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f209bc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2094070ff0 0x7f209415afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f209bc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f209409d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f209bc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f209415fc10 0x7f209413e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f209bc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2094072a30 con 0x7f20940b3f70 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f209bc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20940728b0 con 0x7f209415fc10 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f209bc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209415c2e0 con 0x7f2094070ff0 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f2093e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f209409d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f2093e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f209409d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44946/0 (socket says 172.21.15.97:44946) 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.632+0000 7f2093e00640 1 -- 172.21.15.97:0/3243440605 learned_addr learned my addr 172.21.15.97:0/3243440605 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:01.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2099c00640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f209415fc10 0x7f209413e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2099200640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2094070ff0 0x7f209415afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2093e00640 1 -- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2094070ff0 msgr2=0x7f209415afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2093e00640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2094070ff0 0x7f209415afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2093e00640 1 -- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f209415fc10 msgr2=0x7f209413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2093e00640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f209415fc10 0x7f209413e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2093e00640 1 -- 172.21.15.97:0/3243440605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209415c9a0 con 0x7f20940b3f70 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2099c00640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f209415fc10 0x7f209413e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:01.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2093e00640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f209409d380 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f208002b910 tx=0x7f20800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080041070 con 0x7f20940b3f70 2024-03-06T07:37:01.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208003b840 con 0x7f20940b3f70 2024-03-06T07:37:01.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080035040 con 0x7f20940b3f70 2024-03-06T07:37:01.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.633+0000 7f209bc00640 1 -- 172.21.15.97:0/3243440605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209415a4c0 con 0x7f20940b3f70 2024-03-06T07:37:01.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.634+0000 7f209bc00640 1 -- 172.21.15.97:0/3243440605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209415c5e0 con 0x7f20940b3f70 2024-03-06T07:37:01.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.635+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209415a2e0 con 0x7f20940b3f70 2024-03-06T07:37:01.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.636+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f208004c060 con 0x7f20940b3f70 2024-03-06T07:37:01.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.637+0000 7f2090c00640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f207406b4d0 0x7f207406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.637+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f20800abd30 con 0x7f20940b3f70 2024-03-06T07:37:01.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.637+0000 7f2099200640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f207406b4d0 0x7f207406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.638+0000 7f2099200640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f207406b4d0 0x7f207406d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2088000f60 tx=0x7f2088009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.646+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f209415a2e0 con 0x7f20940b3f70 2024-03-06T07:37:01.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.833+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f209404f330 con 0x7f20940b3f70 2024-03-06T07:37:01.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.833+0000 7f2090c00640 1 -- 172.21.15.97:0/3243440605 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2080031800 con 0x7f20940b3f70 2024-03-06T07:37:01.835 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:37:01.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f207406b4d0 msgr2=0x7f207406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f207406b4d0 0x7f207406d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2088000f60 tx=0x7f2088009040 comp rx=0 tx=0).stop 2024-03-06T07:37:01.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 msgr2=0x7f209409d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f209409d380 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f208002b910 tx=0x7f20800046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 shutdown_connections 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f207406b4d0 0x7f207406d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f209415fc10 0x7f209413e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20940b3f70 0x7f209409d380 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 --2- 172.21.15.97:0/3243440605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2094070ff0 0x7f209415afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 >> 172.21.15.97:0/3243440605 conn(0x7f20940b1940 msgr2=0x7f2094151270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 shutdown_connections 2024-03-06T07:37:01.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:01.836+0000 7f2086000640 1 -- 172.21.15.97:0/3243440605 wait complete. 2024-03-06T07:37:01.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:37:02.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.057+0000 7f0f31200640 1 Processor -- start 2024-03-06T07:37:02.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.057+0000 7f0f31200640 1 -- start start 2024-03-06T07:37:02.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f31200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0f2c066d00 0x7f0f2c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f31200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0f2c067140 0x7f0f2c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f31200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c0b0f40 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f31200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b5020 con 0x7f0f2c067140 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f31200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b5300 con 0x7f0f2c066d00 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f2a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0f2c067140 0x7f0f2c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f2a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0f2c067140 0x7f0f2c06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44956/0 (socket says 172.21.15.97:44956) 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f2a800640 1 -- 172.21.15.97:0/2219811443 learned_addr learned my addr 172.21.15.97:0/2219811443 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f2a800640 1 -- 172.21.15.97:0/2219811443 >> v2:172.21.15.135:3300/0 conn(0x7f0f2c066d00 msgr2=0x7f0f2c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.059+0000 7f0f21400640 1 --2- 172.21.15.97:0/2219811443 >> v2:172.21.15.135:3300/0 conn(0x7f0f2c066d00 0x7f0f2c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.060+0000 7f0f2a800640 1 --2- 172.21.15.97:0/2219811443 >> v2:172.21.15.135:3300/0 conn(0x7f0f2c066d00 0x7f0f2c06afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.060+0000 7f0f2a800640 1 -- 172.21.15.97:0/2219811443 >> v1:172.21.15.135:6789/0 conn(0x7f0f2c0cd2b0 legacy=0x7f0f2c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.060+0000 7f0f2a800640 1 -- 172.21.15.97:0/2219811443 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0b1220 con 0x7f0f2c067140 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.061+0000 7f0f2a800640 1 --2- 172.21.15.97:0/2219811443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c067140 0x7f0f2c06b500 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c070650 tx=0x7f0f2402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01fb16100f09a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.061+0000 7f0f29400640 1 -- 172.21.15.97:0/2219811443 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2402cb20 con 0x7f0f2c067140 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.061+0000 7f0f29400640 1 -- 172.21.15.97:0/2219811443 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2402ccc0 con 0x7f0f2c067140 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.061+0000 7f0f29400640 1 -- 172.21.15.97:0/2219811443 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24034610 con 0x7f0f2c067140 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 -- 172.21.15.97:0/2219811443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c067140 msgr2=0x7f0f2c06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 --2- 172.21.15.97:0/2219811443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c067140 0x7f0f2c06b500 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c070650 tx=0x7f0f2402bf80 comp rx=0 tx=0).stop 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 -- 172.21.15.97:0/2219811443 shutdown_connections 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 --2- 172.21.15.97:0/2219811443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c067140 0x7f0f2c06b500 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 --2- 172.21.15.97:0/2219811443 >> v2:172.21.15.135:3300/0 conn(0x7f0f2c066d00 0x7f0f2c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 -- 172.21.15.97:0/2219811443 >> 172.21.15.97:0/2219811443 conn(0x7f0f2c0b1e80 msgr2=0x7f0f2c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 -- 172.21.15.97:0/2219811443 shutdown_connections 2024-03-06T07:37:02.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.062+0000 7f0f31200640 1 -- 172.21.15.97:0/2219811443 wait complete. 2024-03-06T07:37:02.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.064+0000 7f0f31200640 1 Processor -- start 2024-03-06T07:37:02.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.064+0000 7f0f31200640 1 -- start start 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f31200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f2c066d00 0x7f0f2c15ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f31200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f2c067140 0x7f0f2c0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f31200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 0x7f0f2c14dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f31200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b2b30 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f31200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0cd690 con 0x7f0f2c067140 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f31200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b2cb0 con 0x7f0f2c066d00 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f2a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 0x7f0f2c14dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f2a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 0x7f0f2c14dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44968/0 (socket says 172.21.15.97:44968) 2024-03-06T07:37:02.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.065+0000 7f0f2a800640 1 -- 172.21.15.97:0/2767200381 learned_addr learned my addr 172.21.15.97:0/2767200381 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f21400640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f2c067140 0x7f0f2c0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f29e00640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f2c066d00 0x7f0f2c15ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f2a800640 1 -- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f2c066d00 msgr2=0x7f0f2c15ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f2a800640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f2c066d00 0x7f0f2c15ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f2a800640 1 -- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f2c067140 msgr2=0x7f0f2c0c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f2a800640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f2c067140 0x7f0f2c0c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f2a800640 1 -- 172.21.15.97:0/2767200381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0c78b0 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f21400640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f2c067140 0x7f0f2c0c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f2a800640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 0x7f0f2c14dc40 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f0f24036040 tx=0x7f0f24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24043020 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2403e450 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2403e860 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.066+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c0c7b90 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.067+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c0c8150 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.068+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c0c8480 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.070+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f24046020 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.071+0000 7f0f22a00640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f0806b4d0 0x7f0f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.071+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f240acc00 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.071+0000 7f0f29e00640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f0806b4d0 0x7f0f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.072+0000 7f0f29e00640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f0806b4d0 0x7f0f0806d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006fd0 tx=0x7f0f1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.076+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f2404d030 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.272+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f2c04f330 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.273+0000 7f0f22a00640 1 -- 172.21.15.97:0/2767200381 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f2407b9a0 con 0x7f0f2c0cd2b0 2024-03-06T07:37:02.275 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f0806b4d0 msgr2=0x7f0f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f0806b4d0 0x7f0f0806d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006fd0 tx=0x7f0f1c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 msgr2=0x7f0f2c14dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 0x7f0f2c14dc40 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f0f24036040 tx=0x7f0f24004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 shutdown_connections 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0f0806b4d0 0x7f0f0806d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f2c0cd2b0 0x7f0f2c14dc40 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f2c067140 0x7f0f2c0c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 --2- 172.21.15.97:0/2767200381 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0f2c066d00 0x7f0f2c15ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 >> 172.21.15.97:0/2767200381 conn(0x7f0f2c0b1e80 msgr2=0x7f0f2c0d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.275+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 shutdown_connections 2024-03-06T07:37:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.276+0000 7f0f31200640 1 -- 172.21.15.97:0/2767200381 wait complete. 2024-03-06T07:37:02.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:37:02.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.494+0000 7f00b1600640 1 Processor -- start 2024-03-06T07:37:02.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.494+0000 7f00b1600640 1 -- start start 2024-03-06T07:37:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00b1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00b1600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac161960 con 0x7f00ac0d2a70 2024-03-06T07:37:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00b1600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0cf6f0 con 0x7f00ac0d1e90 2024-03-06T07:37:02.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00b1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0cf9d0 con 0x7f00ac161580 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00b0c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f00ac0d2a70 0x7f00ac0b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52732/0 (socket says 172.21.15.97:52732) 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00b0c00640 1 -- 172.21.15.97:0/1327148794 learned_addr learned my addr 172.21.15.97:0/1327148794 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.496+0000 7f00abe00640 1 --2- 172.21.15.97:0/1327148794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.497+0000 7f00aaa00640 1 -- 172.21.15.97:0/1327148794 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585197464 0 0) 0x7f00ac161960 con 0x7f00ac0d2a70 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.497+0000 7f00aaa00640 1 -- 172.21.15.97:0/1327148794 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0090003520 con 0x7f00ac0d2a70 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.497+0000 7f00aaa00640 1 -- 172.21.15.97:0/1327148794 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568311697 0 0) 0x7f00ac0cf6f0 con 0x7f00ac0d1e90 2024-03-06T07:37:02.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.497+0000 7f00abe00640 1 -- 172.21.15.97:0/1327148794 >> v1:172.21.15.135:6789/0 conn(0x7f00ac0d1e90 legacy=0x7f00ac0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.497+0000 7f00abe00640 1 -- 172.21.15.97:0/1327148794 >> v1:172.21.15.97:6789/0 conn(0x7f00ac0d2a70 legacy=0x7f00ac0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.497+0000 7f00abe00640 1 -- 172.21.15.97:0/1327148794 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0cfcb0 con 0x7f00ac161580 2024-03-06T07:37:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.499+0000 7f00abe00640 1 --2- 172.21.15.97:0/1327148794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0d1950 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f009c0097d0 tx=0x7f009c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bcb75bc66103e9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.499+0000 7f00aaa00640 1 -- 172.21.15.97:0/1327148794 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c035020 con 0x7f00ac161580 2024-03-06T07:37:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.499+0000 7f00aaa00640 1 -- 172.21.15.97:0/1327148794 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c031890 con 0x7f00ac161580 2024-03-06T07:37:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.499+0000 7f00aaa00640 1 -- 172.21.15.97:0/1327148794 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c031c40 con 0x7f00ac161580 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 -- 172.21.15.97:0/1327148794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 msgr2=0x7f00ac0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 --2- 172.21.15.97:0/1327148794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0d1950 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f009c0097d0 tx=0x7f009c02b930 comp rx=0 tx=0).stop 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 -- 172.21.15.97:0/1327148794 shutdown_connections 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 --2- 172.21.15.97:0/1327148794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0d1950 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 -- 172.21.15.97:0/1327148794 >> 172.21.15.97:0/1327148794 conn(0x7f00ac0d3d80 msgr2=0x7f00ac0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 -- 172.21.15.97:0/1327148794 shutdown_connections 2024-03-06T07:37:02.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.500+0000 7f00b1600640 1 -- 172.21.15.97:0/1327148794 wait complete. 2024-03-06T07:37:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.502+0000 7f00b1600640 1 Processor -- start 2024-03-06T07:37:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.502+0000 7f00b1600640 1 -- start start 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 0x7f00ac0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b1600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ac0d2a70 0x7f00ac0b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0cf3a0 con 0x7f00ac0d1e90 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b1600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0b39a0 con 0x7f00ac0d2a70 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0b45d0 con 0x7f00ac161580 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 0x7f00ac0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 0x7f00ac0cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44984/0 (socket says 172.21.15.97:44984) 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00abe00640 1 -- 172.21.15.97:0/3248380990 learned_addr learned my addr 172.21.15.97:0/3248380990 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00b0c00640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.503+0000 7f00ab400640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ac0d2a70 0x7f00ac0b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00abe00640 1 -- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 msgr2=0x7f00ac0b83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00abe00640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0b83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00abe00640 1 -- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ac0d2a70 msgr2=0x7f00ac0b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00abe00640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ac0d2a70 0x7f00ac0b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00abe00640 1 -- 172.21.15.97:0/3248380990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0cd540 con 0x7f00ac0d1e90 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00b0c00640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00ab400640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ac0d2a70 0x7f00ac0b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00abe00640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 0x7f00ac0cb010 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f009c02b970 tx=0x7f009c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c035040 con 0x7f00ac0d1e90 2024-03-06T07:37:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c030d20 con 0x7f00ac0d1e90 2024-03-06T07:37:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac0cd820 con 0x7f00ac0d1e90 2024-03-06T07:37:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.504+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0456c0 con 0x7f00ac0d1e90 2024-03-06T07:37:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.505+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0b96d0 con 0x7f00ac0d1e90 2024-03-06T07:37:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.506+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac04f330 con 0x7f00ac0d1e90 2024-03-06T07:37:02.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.507+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f009c038020 con 0x7f00ac0d1e90 2024-03-06T07:37:02.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.507+0000 7f00a3e00640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.508+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f009c0ac540 con 0x7f00ac0d1e90 2024-03-06T07:37:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.508+0000 7f00ab400640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.508+0000 7f00ab400640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f008c06b4d0 0x7f008c06d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0b8d80 tx=0x7f0098009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.512+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f009c02d040 con 0x7f00ac0d1e90 2024-03-06T07:37:02.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.721+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f00ac0d3b60 con 0x7f00ac0d1e90 2024-03-06T07:37:02.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.722+0000 7f00a3e00640 1 -- 172.21.15.97:0/3248380990 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v134) v1 ==== 80+0+0 (secure 0 0 0) 0x7f009c07b410 con 0x7f00ac0d1e90 2024-03-06T07:37:02.724 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.724+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f008c06b4d0 msgr2=0x7f008c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.724+0000 7f00b1600640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f008c06b4d0 0x7f008c06d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0b8d80 tx=0x7f0098009040 comp rx=0 tx=0).stop 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.724+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 msgr2=0x7f00ac0cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.724+0000 7f00b1600640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 0x7f00ac0cb010 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f009c02b970 tx=0x7f009c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 shutdown_connections 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00ac161580 0x7f00ac0b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ac0d2a70 0x7f00ac0b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 --2- 172.21.15.97:0/3248380990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac0d1e90 0x7f00ac0cb010 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 >> 172.21.15.97:0/3248380990 conn(0x7f00ac0d3d80 msgr2=0x7f00ac0d4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 shutdown_connections 2024-03-06T07:37:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:02.725+0000 7f00b1600640 1 -- 172.21.15.97:0/3248380990 wait complete. 2024-03-06T07:37:02.746 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:37:02.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:02.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:02.883 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:02.884 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:37:02.935 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:37:02.935 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:02.974 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:37:02.974 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:02.974 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:03.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:37:03.012 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:37:03.012 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:03.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:37:03.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:03.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.256+0000 7f74e4e00640 1 Processor -- start 2024-03-06T07:37:03.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.256+0000 7f74e4e00640 1 -- start start 2024-03-06T07:37:03.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74e4e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 0x7f74e00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74e4e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f74e015f470 0x7f74e00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74e4e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e00cea60 con 0x7f74e00d0df0 2024-03-06T07:37:03.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74e4e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e00d1260 con 0x7f74e015f470 2024-03-06T07:37:03.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74e4e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00d2f60 con 0x7f74e015e0d0 2024-03-06T07:37:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74de800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f74e00d0df0 0x7f74e0160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52746/0 (socket says 172.21.15.97:52746) 2024-03-06T07:37:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.257+0000 7f74de800640 1 -- 172.21.15.97:0/2737892120 learned_addr learned my addr 172.21.15.97:0/2737892120 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dde00640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 0x7f74e00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74d7e00640 1 --2- 172.21.15.97:0/2737892120 >> v2:172.21.15.135:3300/0 conn(0x7f74e015f470 0x7f74e00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106210599 0 0) 0x7f74e00cea60 con 0x7f74e00d0df0 2024-03-06T07:37:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c4003520 con 0x7f74e00d0df0 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852890065 0 0) 0x7f74c4003520 con 0x7f74e00d0df0 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 msgr2=0x7f74e00d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 0x7f74e00d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 >> v2:172.21.15.135:3300/0 conn(0x7f74e015f470 msgr2=0x7f74e00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 --2- 172.21.15.97:0/2737892120 >> v2:172.21.15.135:3300/0 conn(0x7f74e015f470 0x7f74e00d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00d2350 con 0x7f74e00d0df0 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f74d80019f0 con 0x7f74e00d0df0 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74d7e00640 1 --2- 172.21.15.97:0/2737892120 >> v2:172.21.15.135:3300/0 conn(0x7f74e015f470 0x7f74e00d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 >> v1:172.21.15.97:6789/0 conn(0x7f74e00d0df0 legacy=0x7f74e0160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 0x7f74c40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74c4004030 0x7f74c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74c4004d40 0x7f74c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00d1260 con 0x7f74c4004d40 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c400bb30 con 0x7f74c4004030 2024-03-06T07:37:03.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.258+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c400bd30 con 0x7f74e015e0d0 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74de800640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74c4004d40 0x7f74c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74dde00640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 0x7f74c40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74d7e00640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74c4004030 0x7f74c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74de800640 1 -- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 msgr2=0x7f74c40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74de800640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e015e0d0 0x7f74c40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74de800640 1 -- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74c4004030 msgr2=0x7f74c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74de800640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74c4004030 0x7f74c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.259+0000 7f74de800640 1 -- 172.21.15.97:0/2737892120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4003520 con 0x7f74c4004d40 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.260+0000 7f74de800640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74c4004d40 0x7f74c40051c0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f74d800a7d0 tx=0x7f74c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81426e47dba4c74c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d800a3f0 con 0x7f74c4004d40 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8034630 con 0x7f74c4004d40 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74dd400640 1 -- 172.21.15.97:0/2737892120 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d802ce60 con 0x7f74c4004d40 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74e4e00640 1 -- 172.21.15.97:0/2737892120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74c4004d40 msgr2=0x7f74c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74e4e00640 1 --2- 172.21.15.97:0/2737892120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74c4004d40 0x7f74c40051c0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f74d800a7d0 tx=0x7f74c4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74de800640 1 -- 172.21.15.97:0/2737892120 reap_dead start 2024-03-06T07:37:03.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74e4e00640 1 -- 172.21.15.97:0/2737892120 shutdown_connections 2024-03-06T07:37:03.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74e4e00640 1 -- 172.21.15.97:0/2737892120 >> 172.21.15.97:0/2737892120 conn(0x7f74e00d3ec0 msgr2=0x7f74e00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:03.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74e4e00640 1 -- 172.21.15.97:0/2737892120 shutdown_connections 2024-03-06T07:37:03.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.261+0000 7f74e4e00640 1 -- 172.21.15.97:0/2737892120 wait complete. 2024-03-06T07:37:03.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.263+0000 7f74e4e00640 1 Processor -- start 2024-03-06T07:37:03.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.263+0000 7f74e4e00640 1 -- start start 2024-03-06T07:37:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74e4e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e0151200 0x7f74e00c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74e4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 0x7f74e00cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74e4e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74e00cb040 0x7f74e00ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74e4e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00d2bf0 con 0x7f74e01515e0 2024-03-06T07:37:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74e4e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00d2910 con 0x7f74e00cb040 2024-03-06T07:37:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74e4e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e015fb30 con 0x7f74e0151200 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.264+0000 7f74d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 0x7f74e00cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 0x7f74e00cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45008/0 (socket says 172.21.15.97:45008) 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 -- 172.21.15.97:0/335402665 learned_addr learned my addr 172.21.15.97:0/335402665 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74dde00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e0151200 0x7f74e00c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74de800640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74e00cb040 0x7f74e00ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 -- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e0151200 msgr2=0x7f74e00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e0151200 0x7f74e00c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 -- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74e00cb040 msgr2=0x7f74e00ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74e00cb040 0x7f74e00ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 -- 172.21.15.97:0/335402665 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00cd6e0 con 0x7f74e01515e0 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74dde00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e0151200 0x7f74e00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:03.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.265+0000 7f74d7e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 0x7f74e00cca10 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f74c8010860 tx=0x7f74c8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:03.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.266+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c801a020 con 0x7f74e01515e0 2024-03-06T07:37:03.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.266+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c800eb60 con 0x7f74e01515e0 2024-03-06T07:37:03.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.266+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c800f040 con 0x7f74e01515e0 2024-03-06T07:37:03.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.266+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e00cd980 con 0x7f74e01515e0 2024-03-06T07:37:03.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.266+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e00cdf40 con 0x7f74e01515e0 2024-03-06T07:37:03.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.267+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e004f330 con 0x7f74e01515e0 2024-03-06T07:37:03.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.269+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74c801d020 con 0x7f74e01515e0 2024-03-06T07:37:03.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.270+0000 7f74d6000640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74bc06b4d0 0x7f74bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.270+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f74c808aca0 con 0x7f74e01515e0 2024-03-06T07:37:03.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.270+0000 7f74dde00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74bc06b4d0 0x7f74bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.271+0000 7f74dde00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74bc06b4d0 0x7f74bc06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f74d0006fd0 tx=0x7f74d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:03.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.275+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74c8024020 con 0x7f74e01515e0 2024-03-06T07:37:03.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.515+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74e008d920 con 0x7f74e01515e0 2024-03-06T07:37:03.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.516+0000 7f74d6000640 1 -- 172.21.15.97:0/335402665 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3399 (secure 0 0 0) 0x7f74c8059a40 con 0x7f74e01515e0 2024-03-06T07:37:03.517 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/2007524244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2007524244},{"type":"v1","addr":"172.21.15.97:6836","nonce":2007524244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:57.921337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:37:03.518 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 99 2024-03-06T07:37:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74bc06b4d0 msgr2=0x7f74bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74bc06b4d0 0x7f74bc06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f74d0006fd0 tx=0x7f74d0008040 comp rx=0 tx=0).stop 2024-03-06T07:37:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 msgr2=0x7f74e00cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 0x7f74e00cca10 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f74c8010860 tx=0x7f74c8010d20 comp rx=0 tx=0).stop 2024-03-06T07:37:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 shutdown_connections 2024-03-06T07:37:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74bc06b4d0 0x7f74bc06d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74e00cb040 0x7f74e00ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74e01515e0 0x7f74e00cca10 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 --2- 172.21.15.97:0/335402665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74e0151200 0x7f74e00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 >> 172.21.15.97:0/335402665 conn(0x7f74e00d3ec0 msgr2=0x7f74e00b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 shutdown_connections 2024-03-06T07:37:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.519+0000 7f74e4e00640 1 -- 172.21.15.97:0/335402665 wait complete. 2024-03-06T07:37:03.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:03.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.738+0000 7fa0c1600640 1 Processor -- start 2024-03-06T07:37:03.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.738+0000 7fa0c1600640 1 -- start start 2024-03-06T07:37:03.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.739+0000 7fa0c1600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0bc0d24c0 0x7fa0bc0d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.739+0000 7fa0c1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0c1600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0bc0d2de0 con 0x7fa0bc0d07f0 2024-03-06T07:37:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0c1600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc0d4a00 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0c1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc0cf7e0 con 0x7fa0bc0b2de0 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0baa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0bc0d24c0 0x7fa0bc0d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0baa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0bc0d24c0 0x7fa0bc0d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45010/0 (socket says 172.21.15.97:45010) 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0baa00640 1 -- 172.21.15.97:0/2207732429 learned_addr learned my addr 172.21.15.97:0/2207732429 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.740+0000 7fa0b3e00640 1 --2- 172.21.15.97:0/2207732429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.741+0000 7fa0baa00640 1 -- 172.21.15.97:0/2207732429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 msgr2=0x7fa0bc0d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.741+0000 7fa0baa00640 1 --2- 172.21.15.97:0/2207732429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0d28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.741+0000 7fa0baa00640 1 -- 172.21.15.97:0/2207732429 >> v1:172.21.15.135:6789/0 conn(0x7fa0bc0d07f0 legacy=0x7fa0bc0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.741+0000 7fa0baa00640 1 -- 172.21.15.97:0/2207732429 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0d4c50 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.742+0000 7fa0baa00640 1 --2- 172.21.15.97:0/2207732429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc0d1cc0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009870 tx=0x7fa0ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b786cf870ad609 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.743+0000 7fa0ba000640 1 -- 172.21.15.97:0/2207732429 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac035020 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.743+0000 7fa0ba000640 1 -- 172.21.15.97:0/2207732429 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac031640 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.743+0000 7fa0ba000640 1 -- 172.21.15.97:0/2207732429 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac02d040 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 -- 172.21.15.97:0/2207732429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 msgr2=0x7fa0bc0d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2207732429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc0d1cc0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009870 tx=0x7fa0ac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 -- 172.21.15.97:0/2207732429 shutdown_connections 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2207732429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0d28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2207732429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc0d1cc0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 -- 172.21.15.97:0/2207732429 >> 172.21.15.97:0/2207732429 conn(0x7fa0bc0d3ee0 msgr2=0x7fa0bc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 -- 172.21.15.97:0/2207732429 shutdown_connections 2024-03-06T07:37:03.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.744+0000 7fa0c1600640 1 -- 172.21.15.97:0/2207732429 wait complete. 2024-03-06T07:37:03.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.746+0000 7fa0c1600640 1 Processor -- start 2024-03-06T07:37:03.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.746+0000 7fa0c1600640 1 -- start start 2024-03-06T07:37:03.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.747+0000 7fa0c1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.747+0000 7fa0c1600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0bc0d07f0 0x7fa0bc0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.747+0000 7fa0c1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc1593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.747+0000 7fa0c1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc0cfd10 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.747+0000 7fa0c1600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc0d0080 con 0x7fa0bc0d07f0 2024-03-06T07:37:03.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.747+0000 7fa0c1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc15e5b0 con 0x7fa0bc0b2de0 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc1593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc1593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45014/0 (socket says 172.21.15.97:45014) 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 -- 172.21.15.97:0/2197541016 learned_addr learned my addr 172.21.15.97:0/2197541016 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0b3e00640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0bc0d07f0 0x7fa0bc0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0baa00640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 -- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 msgr2=0x7fa0bc0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0cc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 -- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0bc0d07f0 msgr2=0x7fa0bc0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0bc0d07f0 0x7fa0bc0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 -- 172.21.15.97:0/2197541016 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0ccad0 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.748+0000 7fa0bb400640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc1593d0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c7a0 tx=0x7fa0b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:03.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.749+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4003b70 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.749+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4003d30 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.749+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400aa90 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.749+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc0ccdb0 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.749+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc0cd370 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.750+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc04f330 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.751+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0b400ac30 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.752+0000 7fa0b2a00640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa09806b4d0 0x7fa09806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:03.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.753+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0b408a1e0 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.753+0000 7fa0baa00640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa09806b4d0 0x7fa09806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:03.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.753+0000 7fa0baa00640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa09806b4d0 0x7fa09806d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac036000 tx=0x7fa0ac008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:03.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.757+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0b4059010 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.953+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0bc0d22a0 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.955+0000 7fa0b2a00640 1 -- 172.21.15.97:0/2197541016 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16709 (secure 0 0 0) 0x7fa0b4058910 con 0x7fa0bc0d24c0 2024-03-06T07:37:03.957 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:03.957 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":134,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:36:29.869182+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa09806b4d0 msgr2=0x7fa09806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa09806b4d0 0x7fa09806d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac036000 tx=0x7fa0ac008030 comp rx=0 tx=0).stop 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 msgr2=0x7fa0bc1593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc1593d0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c7a0 tx=0x7fa0b400cc60 comp rx=0 tx=0).stop 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 shutdown_connections 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa09806b4d0 0x7fa09806d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0bc0d24c0 0x7fa0bc1593d0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0bc0d07f0 0x7fa0bc0cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 --2- 172.21.15.97:0/2197541016 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0bc0b2de0 0x7fa0bc0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 >> 172.21.15.97:0/2197541016 conn(0x7fa0bc0d3ee0 msgr2=0x7fa0bc06cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:03.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 shutdown_connections 2024-03-06T07:37:03.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:03.958+0000 7fa0c1600640 1 -- 172.21.15.97:0/2197541016 wait complete. 2024-03-06T07:37:03.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:41.634478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:42.840855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:37:03.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:37:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.173+0000 7fd217000640 1 Processor -- start 2024-03-06T07:37:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.173+0000 7fd217000640 1 -- start start 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd217000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd21006d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd217000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd2100ad860 0x7fd2100adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd217000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2100ae200 con 0x7fd2100cfcb0 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd217000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21006d970 con 0x7fd2100ad860 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd217000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21015f3b0 con 0x7fd21006d050 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd215200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd2100ad860 0x7fd2100adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd216600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd2100cfcb0 0x7fd2100d0110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49160/0 (socket says 172.21.15.97:49160) 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd215200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd2100ad860 0x7fd2100adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59448/0 (socket says 172.21.15.97:59448) 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd216600640 1 -- 172.21.15.97:0/2548262877 learned_addr learned my addr 172.21.15.97:0/2548262877 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.175+0000 7fd215c00640 1 --2- 172.21.15.97:0/2548262877 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd21006d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.176+0000 7fd215200640 1 -- 172.21.15.97:0/2548262877 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 msgr2=0x7fd21006d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.176+0000 7fd215200640 1 --2- 172.21.15.97:0/2548262877 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd21006d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.176+0000 7fd215200640 1 -- 172.21.15.97:0/2548262877 >> v1:172.21.15.97:6789/0 conn(0x7fd2100cfcb0 legacy=0x7fd2100d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.176+0000 7fd20fe00640 1 -- 172.21.15.97:0/2548262877 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734457835 0 0) 0x7fd208001110 con 0x7fd2100cfcb0 2024-03-06T07:37:04.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.176+0000 7fd215200640 1 -- 172.21.15.97:0/2548262877 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2101436f0 con 0x7fd2100ad860 2024-03-06T07:37:04.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.177+0000 7fd215200640 1 --2- 172.21.15.97:0/2548262877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100ad860 0x7fd2100adcc0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0097a0 tx=0x7fd1fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6433ec6f8a5edf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd20fe00640 1 -- 172.21.15.97:0/2548262877 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc035020 con 0x7fd2100ad860 2024-03-06T07:37:04.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd20fe00640 1 -- 172.21.15.97:0/2548262877 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc031830 con 0x7fd2100ad860 2024-03-06T07:37:04.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd20fe00640 1 -- 172.21.15.97:0/2548262877 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc031c00 con 0x7fd2100ad860 2024-03-06T07:37:04.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd217000640 1 -- 172.21.15.97:0/2548262877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100ad860 msgr2=0x7fd2100adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd217000640 1 --2- 172.21.15.97:0/2548262877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100ad860 0x7fd2100adcc0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0097a0 tx=0x7fd1fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd217000640 1 -- 172.21.15.97:0/2548262877 shutdown_connections 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd217000640 1 --2- 172.21.15.97:0/2548262877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100ad860 0x7fd2100adcc0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd217000640 1 --2- 172.21.15.97:0/2548262877 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd21006d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.178+0000 7fd217000640 1 -- 172.21.15.97:0/2548262877 >> 172.21.15.97:0/2548262877 conn(0x7fd2100cc850 msgr2=0x7fd2100ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.179+0000 7fd217000640 1 -- 172.21.15.97:0/2548262877 shutdown_connections 2024-03-06T07:37:04.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.179+0000 7fd217000640 1 -- 172.21.15.97:0/2548262877 wait complete. 2024-03-06T07:37:04.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.180+0000 7fd217000640 1 Processor -- start 2024-03-06T07:37:04.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.180+0000 7fd217000640 1 -- start start 2024-03-06T07:37:04.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.181+0000 7fd217000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd210145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.181+0000 7fd217000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2100ad860 0x7fd2101608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.181+0000 7fd217000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 0x7fd2101f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.181+0000 7fd217000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2101444b0 con 0x7fd2100cfcb0 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.181+0000 7fd217000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210143cb0 con 0x7fd2100ad860 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.181+0000 7fd217000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21015e5f0 con 0x7fd21006d050 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 0x7fd2101f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 0x7fd2101f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59462/0 (socket says 172.21.15.97:59462) 2024-03-06T07:37:04.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 -- 172.21.15.97:0/1984253574 learned_addr learned my addr 172.21.15.97:0/1984253574 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 -- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 msgr2=0x7fd210145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd215c00640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd210145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd215200640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2100ad860 0x7fd2101608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd210145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 -- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2100ad860 msgr2=0x7fd2101608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2100ad860 0x7fd2101608e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 -- 172.21.15.97:0/1984253574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2101457a0 con 0x7fd2100cfcb0 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.182+0000 7fd216600640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 0x7fd2101f3b40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c780 tx=0x7fd20800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.183+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208018070 con 0x7fd2100cfcb0 2024-03-06T07:37:04.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.183+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2080070d0 con 0x7fd2100cfcb0 2024-03-06T07:37:04.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.183+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208007500 con 0x7fd2100cfcb0 2024-03-06T07:37:04.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.183+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd210145a80 con 0x7fd2100cfcb0 2024-03-06T07:37:04.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.183+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd210146040 con 0x7fd2100cfcb0 2024-03-06T07:37:04.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.184+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21004f330 con 0x7fd2100cfcb0 2024-03-06T07:37:04.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.185+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd20801b070 con 0x7fd2100cfcb0 2024-03-06T07:37:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.186+0000 7fd20d600640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd1f006b4d0 0x7fd1f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.186+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd2080894e0 con 0x7fd2100cfcb0 2024-03-06T07:37:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.186+0000 7fd215c00640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd1f006b4d0 0x7fd1f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.187+0000 7fd215c00640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd1f006b4d0 0x7fd1f006d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd204009e90 tx=0x7fd20400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.191+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2100ae200 con 0x7fd2100cfcb0 2024-03-06T07:37:04.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.392+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd21006e370 con 0x7fd2100cfcb0 2024-03-06T07:37:04.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.393+0000 7fd20d600640 1 -- 172.21.15.97:0/1984253574 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd208058310 con 0x7fd2100cfcb0 2024-03-06T07:37:04.394 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd1f006b4d0 msgr2=0x7fd1f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd1f006b4d0 0x7fd1f006d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd204009e90 tx=0x7fd20400b040 comp rx=0 tx=0).stop 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 msgr2=0x7fd2101f3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 0x7fd2101f3b40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c780 tx=0x7fd20800cc40 comp rx=0 tx=0).stop 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 shutdown_connections 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd1f006b4d0 0x7fd1f006d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2100cfcb0 0x7fd2101f3b40 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2100ad860 0x7fd2101608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.395+0000 7fd217000640 1 --2- 172.21.15.97:0/1984253574 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd21006d050 0x7fd210145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.396+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 >> 172.21.15.97:0/1984253574 conn(0x7fd2100cc850 msgr2=0x7fd21015ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.396+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 shutdown_connections 2024-03-06T07:37:04.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.396+0000 7fd217000640 1 -- 172.21.15.97:0/1984253574 wait complete. 2024-03-06T07:37:04.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:37:04.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.618+0000 7efdcce00640 1 Processor -- start 2024-03-06T07:37:04.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.618+0000 7efdcce00640 1 -- start start 2024-03-06T07:37:04.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.619+0000 7efdcce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdcce00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efdc80b3650 0x7efdc80b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdcce00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efdc80cda90 0x7efdc806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdcce00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc8072310 con 0x7efdc80cda90 2024-03-06T07:37:04.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdcce00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc80b4710 con 0x7efdc80b3650 2024-03-06T07:37:04.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdcce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80719e0 con 0x7efdc80b3df0 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdc6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efdc80cda90 0x7efdc806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdc6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efdc80cda90 0x7efdc806c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59474/0 (socket says 172.21.15.97:59474) 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdc6800640 1 -- 172.21.15.97:0/3373216003 learned_addr learned my addr 172.21.15.97:0/3373216003 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdc5e00640 1 --2- 172.21.15.97:0/3373216003 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.620+0000 7efdbd400640 1 --2- 172.21.15.97:0/3373216003 >> v2:172.21.15.135:3300/0 conn(0x7efdc80b3650 0x7efdc80b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.621+0000 7efdc6800640 1 -- 172.21.15.97:0/3373216003 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 msgr2=0x7efdc80727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.621+0000 7efdc6800640 1 --2- 172.21.15.97:0/3373216003 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc80727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.621+0000 7efdc6800640 1 -- 172.21.15.97:0/3373216003 >> v2:172.21.15.135:3300/0 conn(0x7efdc80b3650 msgr2=0x7efdc80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.621+0000 7efdc6800640 1 --2- 172.21.15.97:0/3373216003 >> v2:172.21.15.135:3300/0 conn(0x7efdc80b3650 0x7efdc80b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.621+0000 7efdc6800640 1 -- 172.21.15.97:0/3373216003 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8071cc0 con 0x7efdc80cda90 2024-03-06T07:37:04.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.622+0000 7efdc6800640 1 --2- 172.21.15.97:0/3373216003 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc806c990 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7efdc80b2a00 tx=0x7efdc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62df834174c82319 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.622+0000 7efdc5400640 1 -- 172.21.15.97:0/3373216003 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc002ca70 con 0x7efdc80cda90 2024-03-06T07:37:04.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.622+0000 7efdc5400640 1 -- 172.21.15.97:0/3373216003 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc002cc10 con 0x7efdc80cda90 2024-03-06T07:37:04.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.622+0000 7efdc5400640 1 -- 172.21.15.97:0/3373216003 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00343f0 con 0x7efdc80cda90 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 -- 172.21.15.97:0/3373216003 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 msgr2=0x7efdc806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 --2- 172.21.15.97:0/3373216003 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc806c990 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7efdc80b2a00 tx=0x7efdc002bed0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 -- 172.21.15.97:0/3373216003 shutdown_connections 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 --2- 172.21.15.97:0/3373216003 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc806c990 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 --2- 172.21.15.97:0/3373216003 >> v2:172.21.15.135:3300/0 conn(0x7efdc80b3650 0x7efdc80b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 --2- 172.21.15.97:0/3373216003 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc80727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 -- 172.21.15.97:0/3373216003 >> 172.21.15.97:0/3373216003 conn(0x7efdc80b1940 msgr2=0x7efdc80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 -- 172.21.15.97:0/3373216003 shutdown_connections 2024-03-06T07:37:04.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.623+0000 7efdcce00640 1 -- 172.21.15.97:0/3373216003 wait complete. 2024-03-06T07:37:04.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.625+0000 7efdcce00640 1 Processor -- start 2024-03-06T07:37:04.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.625+0000 7efdcce00640 1 -- start start 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdcce00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdc80b3650 0x7efdc8154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdcce00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc809cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdcce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc809d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdcce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc815b240 con 0x7efdc80cda90 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdcce00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80b3b30 con 0x7efdc80b3650 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdcce00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc815b3c0 con 0x7efdc80b3df0 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc809d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc809d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59476/0 (socket says 172.21.15.97:59476) 2024-03-06T07:37:04.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 -- 172.21.15.97:0/1263909509 learned_addr learned my addr 172.21.15.97:0/1263909509 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:04.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 -- 172.21.15.97:0/1263909509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 msgr2=0x7efdc809cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc809cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 -- 172.21.15.97:0/1263909509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdc80b3650 msgr2=0x7efdc8154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdc80b3650 0x7efdc8154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.626+0000 7efdc6800640 1 -- 172.21.15.97:0/1263909509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80cb330 con 0x7efdc80cda90 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.627+0000 7efdc6800640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc809d100 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009f40 tx=0x7efdc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.627+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0043020 con 0x7efdc80cda90 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.627+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc003e450 con 0x7efdc80cda90 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.627+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0009510 con 0x7efdc80cda90 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.627+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc80cb610 con 0x7efdc80cda90 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.627+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc80cbbd0 con 0x7efdc80cda90 2024-03-06T07:37:04.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.628+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc804f330 con 0x7efdc80cda90 2024-03-06T07:37:04.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.629+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdc0046020 con 0x7efdc80cda90 2024-03-06T07:37:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.631+0000 7efdbea00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efda406b500 0x7efda406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.631+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efdc00acd50 con 0x7efdc80cda90 2024-03-06T07:37:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.631+0000 7efdc5e00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efda406b500 0x7efda406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.632+0000 7efdc5e00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efda406b500 0x7efda406d9c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006fd0 tx=0x7efdb8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.635+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdc004d020 con 0x7efdc80cda90 2024-03-06T07:37:04.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.831+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efdc8155dd0 con 0x7efdc80cda90 2024-03-06T07:37:04.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.832+0000 7efdbea00640 1 -- 172.21.15.97:0/1263909509 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7efdc007bb80 con 0x7efdc80cda90 2024-03-06T07:37:04.834 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.834+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efda406b500 msgr2=0x7efda406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.834+0000 7efdcce00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efda406b500 0x7efda406d9c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006fd0 tx=0x7efdb8008040 comp rx=0 tx=0).stop 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.834+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 msgr2=0x7efdc809d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.834+0000 7efdcce00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc809d100 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009f40 tx=0x7efdc0004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 shutdown_connections 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efda406b500 0x7efda406d9c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdc80cda90 0x7efdc809d100 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efdc80b3df0 0x7efdc809cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 --2- 172.21.15.97:0/1263909509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdc80b3650 0x7efdc8154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 >> 172.21.15.97:0/1263909509 conn(0x7efdc80b1940 msgr2=0x7efdc80b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 shutdown_connections 2024-03-06T07:37:04.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:04.835+0000 7efdcce00640 1 -- 172.21.15.97:0/1263909509 wait complete. 2024-03-06T07:37:04.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:37:05.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.070+0000 7f9cebe00640 1 Processor -- start 2024-03-06T07:37:05.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.071+0000 7f9cebe00640 1 -- start start 2024-03-06T07:37:05.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.073+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 0x7f9cec0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.073+0000 7f9cebe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0d13e0 con 0x7f9cec0cfaf0 2024-03-06T07:37:05.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.073+0000 7f9cebe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0cead0 con 0x7f9cec0d25b0 2024-03-06T07:37:05.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.073+0000 7f9cebe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0cf120 con 0x7f9cec15e130 2024-03-06T07:37:05.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.074+0000 7f9cea000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9cec0cfaf0 0x7f9cec0d1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49162/0 (socket says 172.21.15.97:49162) 2024-03-06T07:37:05.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.074+0000 7f9cea000640 1 -- 172.21.15.97:0/2187947156 learned_addr learned my addr 172.21.15.97:0/2187947156 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.074+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 0x7f9cec0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.074+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654672375 0 0) 0x7f9cec0d13e0 con 0x7f9cec0cfaf0 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.074+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0003520 con 0x7f9cec0cfaf0 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107042386 0 0) 0x7f9cd0003520 con 0x7f9cec0cfaf0 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 msgr2=0x7f9cec0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 0x7f9cec0cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 >> v1:172.21.15.135:6789/0 conn(0x7f9cec0d25b0 legacy=0x7f9cec0c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0cedb0 con 0x7f9cec0cfaf0 2024-03-06T07:37:05.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9cd4001670 con 0x7f9cec0cfaf0 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 >> v1:172.21.15.97:6789/0 conn(0x7f9cec0cfaf0 legacy=0x7f9cec0d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cd0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cd0004800 0x7f9cd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 0x7f9cd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0cf120 con 0x7f9cd00039e0 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd000baa0 con 0x7f9cd0004800 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd000bca0 con 0x7f9cec15e130 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cd0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9cea000640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cd0004800 0x7f9cd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.075+0000 7f9ceb400640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 0x7f9cd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.076+0000 7f9ceaa00640 1 -- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 msgr2=0x7f9cd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.076+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15e130 0x7f9cd00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.076+0000 7f9ceaa00640 1 -- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cd0004800 msgr2=0x7f9cd0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.076+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cd0004800 0x7f9cd0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.076+0000 7f9ceaa00640 1 -- 172.21.15.97:0/2187947156 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd0003520 con 0x7f9cd00039e0 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.077+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cd0003e00 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009f90 tx=0x7f9cd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18d98bd481ae930 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.077+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc039070 con 0x7f9cd00039e0 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.077+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc02cbd0 con 0x7f9cd00039e0 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.077+0000 7f9ce9600640 1 -- 172.21.15.97:0/2187947156 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc009c10 con 0x7f9cd00039e0 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.077+0000 7f9cebe00640 1 -- 172.21.15.97:0/2187947156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 msgr2=0x7f9cd0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.077+0000 7f9cebe00640 1 --2- 172.21.15.97:0/2187947156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cd0003e00 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009f90 tx=0x7f9cd0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.078+0000 7f9ceb400640 1 -- 172.21.15.97:0/2187947156 reap_dead start 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.078+0000 7f9cebe00640 1 -- 172.21.15.97:0/2187947156 shutdown_connections 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.078+0000 7f9cebe00640 1 -- 172.21.15.97:0/2187947156 >> 172.21.15.97:0/2187947156 conn(0x7f9cec09d070 msgr2=0x7f9cec0d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.078+0000 7f9cebe00640 1 -- 172.21.15.97:0/2187947156 shutdown_connections 2024-03-06T07:37:05.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.078+0000 7f9cebe00640 1 -- 172.21.15.97:0/2187947156 wait complete. 2024-03-06T07:37:05.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.079+0000 7f9cebe00640 1 Processor -- start 2024-03-06T07:37:05.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.080+0000 7f9cebe00640 1 -- start start 2024-03-06T07:37:05.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.080+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cec157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.080+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15a910 0x7f9cec1f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cec1f3fc0 0x7f9cec0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9cebe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0c8200 con 0x7f9cd00039e0 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9cebe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0c8080 con 0x7f9cec1f3fc0 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9cebe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0cf5b0 con 0x7f9cec15a910 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cec157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cec157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59482/0 (socket says 172.21.15.97:59482) 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 -- 172.21.15.97:0/3147887560 learned_addr learned my addr 172.21.15.97:0/3147887560 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 -- 172.21.15.97:0/3147887560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15a910 msgr2=0x7f9cec1f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15a910 0x7f9cec1f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 -- 172.21.15.97:0/3147887560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cec1f3fc0 msgr2=0x7f9cec0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cec1f3fc0 0x7f9cec0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 -- 172.21.15.97:0/3147887560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec1f44b0 con 0x7f9cd00039e0 2024-03-06T07:37:05.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.081+0000 7f9ceaa00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cec157580 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc037040 tx=0x7f9cdc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.082+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc039050 con 0x7f9cd00039e0 2024-03-06T07:37:05.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.082+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc03a040 con 0x7f9cd00039e0 2024-03-06T07:37:05.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.082+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc003990 con 0x7f9cd00039e0 2024-03-06T07:37:05.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.082+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec1f4790 con 0x7f9cd00039e0 2024-03-06T07:37:05.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.082+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec0ccaf0 con 0x7f9cd00039e0 2024-03-06T07:37:05.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.083+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec04f330 con 0x7f9cd00039e0 2024-03-06T07:37:05.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.084+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cdc047030 con 0x7f9cd00039e0 2024-03-06T07:37:05.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.085+0000 7f9ce2a00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9cc806b500 0x7f9cc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.085+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9cdc0ac790 con 0x7f9cd00039e0 2024-03-06T07:37:05.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.085+0000 7f9cea000640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9cc806b500 0x7f9cc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.086+0000 7f9cea000640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9cc806b500 0x7f9cc806d9c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4001050 tx=0x7f9cd4000f60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.094+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cdc04e020 con 0x7f9cd00039e0 2024-03-06T07:37:05.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.290+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9cec09d690 con 0x7f9cd00039e0 2024-03-06T07:37:05.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.291+0000 7f9ce2a00640 1 -- 172.21.15.97:0/3147887560 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cdc07b5c0 con 0x7f9cd00039e0 2024-03-06T07:37:05.292 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.294+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9cc806b500 msgr2=0x7f9cc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.294+0000 7f9cebe00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9cc806b500 0x7f9cc806d9c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4001050 tx=0x7f9cd4000f60 comp rx=0 tx=0).stop 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.294+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 msgr2=0x7f9cec157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.294+0000 7f9cebe00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cec157580 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc037040 tx=0x7f9cdc009750 comp rx=0 tx=0).stop 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 shutdown_connections 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9cc806b500 0x7f9cc806d9c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cec1f3fc0 0x7f9cec0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9cec15a910 0x7f9cec1f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 --2- 172.21.15.97:0/3147887560 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cd00039e0 0x7f9cec157580 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 >> 172.21.15.97:0/3147887560 conn(0x7f9cec09d070 msgr2=0x7f9cec0c7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 shutdown_connections 2024-03-06T07:37:05.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.295+0000 7f9cebe00640 1 -- 172.21.15.97:0/3147887560 wait complete. 2024-03-06T07:37:05.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:05.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.520+0000 7f7997400640 1 Processor -- start 2024-03-06T07:37:05.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.520+0000 7f7997400640 1 -- start start 2024-03-06T07:37:05.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7997400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7990144c00 0x7f799006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7997400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f799015f9d0 0x7f799015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7997400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79900ce0a0 0x7f79900ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7997400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799006ee70 con 0x7f799015f9d0 2024-03-06T07:37:05.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7997400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799006cef0 con 0x7f7990144c00 2024-03-06T07:37:05.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7997400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799006d1d0 con 0x7f79900ce0a0 2024-03-06T07:37:05.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7987e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f799015f9d0 0x7f799015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7987e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f799015f9d0 0x7f799015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59494/0 (socket says 172.21.15.97:59494) 2024-03-06T07:37:05.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.522+0000 7f7987e00640 1 -- 172.21.15.97:0/3051932747 learned_addr learned my addr 172.21.15.97:0/3051932747 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7987e00640 1 -- 172.21.15.97:0/3051932747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79900ce0a0 msgr2=0x7f79900ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7994a00640 1 --2- 172.21.15.97:0/3051932747 >> v2:172.21.15.135:3300/0 conn(0x7f7990144c00 0x7f799006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7995400640 1 --2- 172.21.15.97:0/3051932747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79900ce0a0 0x7f79900ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7987e00640 1 --2- 172.21.15.97:0/3051932747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79900ce0a0 0x7f79900ce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7987e00640 1 -- 172.21.15.97:0/3051932747 >> v2:172.21.15.135:3300/0 conn(0x7f7990144c00 msgr2=0x7f799006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7987e00640 1 --2- 172.21.15.97:0/3051932747 >> v2:172.21.15.135:3300/0 conn(0x7f7990144c00 0x7f799006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7987e00640 1 -- 172.21.15.97:0/3051932747 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799006d4b0 con 0x7f799015f9d0 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.523+0000 7f7994a00640 1 --2- 172.21.15.97:0/3051932747 >> v2:172.21.15.135:3300/0 conn(0x7f7990144c00 0x7f799006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:05.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.524+0000 7f7987e00640 1 --2- 172.21.15.97:0/3051932747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f799015f9d0 0x7f799015fe60 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f79900b25f0 tx=0x7f797802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe0f61c14fb9258 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.524+0000 7f798fe00640 1 -- 172.21.15.97:0/3051932747 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978035020 con 0x7f799015f9d0 2024-03-06T07:37:05.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.524+0000 7f798fe00640 1 -- 172.21.15.97:0/3051932747 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978031870 con 0x7f799015f9d0 2024-03-06T07:37:05.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.524+0000 7f798fe00640 1 -- 172.21.15.97:0/3051932747 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978031c40 con 0x7f799015f9d0 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 -- 172.21.15.97:0/3051932747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f799015f9d0 msgr2=0x7f799015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 --2- 172.21.15.97:0/3051932747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f799015f9d0 0x7f799015fe60 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f79900b25f0 tx=0x7f797802b910 comp rx=0 tx=0).stop 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 -- 172.21.15.97:0/3051932747 shutdown_connections 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 --2- 172.21.15.97:0/3051932747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79900ce0a0 0x7f79900ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 --2- 172.21.15.97:0/3051932747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f799015f9d0 0x7f799015fe60 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 --2- 172.21.15.97:0/3051932747 >> v2:172.21.15.135:3300/0 conn(0x7f7990144c00 0x7f799006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 -- 172.21.15.97:0/3051932747 >> 172.21.15.97:0/3051932747 conn(0x7f79900b1730 msgr2=0x7f79900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 -- 172.21.15.97:0/3051932747 shutdown_connections 2024-03-06T07:37:05.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.525+0000 7f7997400640 1 -- 172.21.15.97:0/3051932747 wait complete. 2024-03-06T07:37:05.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.527+0000 7f7997400640 1 Processor -- start 2024-03-06T07:37:05.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.527+0000 7f7997400640 1 -- start start 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7997400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79900ce0a0 0x7f7990138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7997400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 0x7f799013bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7997400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f799015f9d0 0x7f79901e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7997400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799006cb50 con 0x7f7990144c00 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7997400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7990160730 con 0x7f79900ce0a0 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7997400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799006ccd0 con 0x7f799015f9d0 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7987e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 0x7f799013bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.528+0000 7f7987e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 0x7f799013bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59498/0 (socket says 172.21.15.97:59498) 2024-03-06T07:37:05.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 -- 172.21.15.97:0/2749698181 learned_addr learned my addr 172.21.15.97:0/2749698181 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7994a00640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79900ce0a0 0x7f7990138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7995400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f799015f9d0 0x7f79901e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 -- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f799015f9d0 msgr2=0x7f79901e77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f799015f9d0 0x7f79901e77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 -- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79900ce0a0 msgr2=0x7f7990138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79900ce0a0 0x7f7990138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 -- 172.21.15.97:0/2749698181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79901394d0 con 0x7f7990144c00 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f7987e00640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 0x7f799013bcf0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f797802b950 tx=0x7f79780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.529+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978040070 con 0x7f7990144c00 2024-03-06T07:37:05.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.530+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797803a420 con 0x7f7990144c00 2024-03-06T07:37:05.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.530+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978035040 con 0x7f7990144c00 2024-03-06T07:37:05.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.530+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799013d710 con 0x7f7990144c00 2024-03-06T07:37:05.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.530+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799013dcd0 con 0x7f7990144c00 2024-03-06T07:37:05.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.531+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799004f330 con 0x7f7990144c00 2024-03-06T07:37:05.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.532+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f797804b060 con 0x7f7990144c00 2024-03-06T07:37:05.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.533+0000 7f798d600640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f796c06b4d0 0x7f796c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.533+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f79780ac1c0 con 0x7f7990144c00 2024-03-06T07:37:05.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.533+0000 7f7994a00640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f796c06b4d0 0x7f796c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.534+0000 7f7994a00640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f796c06b4d0 0x7f796c06d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f79901e8570 tx=0x7f7980008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.537+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f797807b010 con 0x7f7990144c00 2024-03-06T07:37:05.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.764+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7990072890 con 0x7f7990144c00 2024-03-06T07:37:05.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.765+0000 7f798d600640 1 -- 172.21.15.97:0/2749698181 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3399 (secure 0 0 0) 0x7f797807a910 con 0x7f7990144c00 2024-03-06T07:37:05.766 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:05.766 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/2007524244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2007524244},{"type":"v1","addr":"172.21.15.97:6836","nonce":2007524244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:57.921337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:37:05.767 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 99 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f796c06b4d0 msgr2=0x7f796c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f796c06b4d0 0x7f796c06d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f79901e8570 tx=0x7f7980008040 comp rx=0 tx=0).stop 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 msgr2=0x7f799013bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 0x7f799013bcf0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f797802b950 tx=0x7f79780046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 shutdown_connections 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f796c06b4d0 0x7f796c06d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f799015f9d0 0x7f79901e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7990144c00 0x7f799013bcf0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 --2- 172.21.15.97:0/2749698181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79900ce0a0 0x7f7990138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.767+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 >> 172.21.15.97:0/2749698181 conn(0x7f79900b1730 msgr2=0x7f79900b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.768+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 shutdown_connections 2024-03-06T07:37:05.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.768+0000 7f7997400640 1 -- 172.21.15.97:0/2749698181 wait complete. 2024-03-06T07:37:05.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:05.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.975+0000 7fe860400640 1 Processor -- start 2024-03-06T07:37:05.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.975+0000 7fe860400640 1 -- start start 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.976+0000 7fe860400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe85815fe60 0x7fe858071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.976+0000 7fe860400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85806cef0 con 0x7fe85815f9d0 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.976+0000 7fe860400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85806d1d0 con 0x7fe858144c00 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.976+0000 7fe860400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85806d4b0 con 0x7fe85815fe60 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85e400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe85815fe60 0x7fe858071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85e400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe85815fe60 0x7fe858071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59510/0 (socket says 172.21.15.97:59510) 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85e400640 1 -- 172.21.15.97:0/3556396232 learned_addr learned my addr 172.21.15.97:0/3556396232 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85d000640 1 --1- 172.21.15.97:0/3556396232 >> v1:172.21.15.97:6789/0 conn(0x7fe85815f9d0 0x7fe858070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49168/0 (socket says 172.21.15.97:49168) 2024-03-06T07:37:05.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85e400640 1 -- 172.21.15.97:0/3556396232 >> v1:172.21.15.135:6789/0 conn(0x7fe858144c00 legacy=0x7fe85806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85e400640 1 -- 172.21.15.97:0/3556396232 >> v1:172.21.15.97:6789/0 conn(0x7fe85815f9d0 legacy=0x7fe858070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.977+0000 7fe85e400640 1 -- 172.21.15.97:0/3556396232 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85806d790 con 0x7fe85815fe60 2024-03-06T07:37:05.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.978+0000 7fe85e400640 1 --2- 172.21.15.97:0/3556396232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe85815fe60 0x7fe858071220 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fe850009e50 tx=0x7fe85002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc47b5eaded5495f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.978+0000 7fe857e00640 1 -- 172.21.15.97:0/3556396232 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85002cb10 con 0x7fe85815fe60 2024-03-06T07:37:05.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.978+0000 7fe857e00640 1 -- 172.21.15.97:0/3556396232 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85002ccb0 con 0x7fe85815fe60 2024-03-06T07:37:05.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.978+0000 7fe857e00640 1 -- 172.21.15.97:0/3556396232 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85002ce60 con 0x7fe85815fe60 2024-03-06T07:37:05.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 -- 172.21.15.97:0/3556396232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe85815fe60 msgr2=0x7fe858071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 --2- 172.21.15.97:0/3556396232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe85815fe60 0x7fe858071220 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fe850009e50 tx=0x7fe85002bf70 comp rx=0 tx=0).stop 2024-03-06T07:37:05.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 -- 172.21.15.97:0/3556396232 shutdown_connections 2024-03-06T07:37:05.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 --2- 172.21.15.97:0/3556396232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe85815fe60 0x7fe858071220 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 -- 172.21.15.97:0/3556396232 >> 172.21.15.97:0/3556396232 conn(0x7fe8580b1730 msgr2=0x7fe8580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 -- 172.21.15.97:0/3556396232 shutdown_connections 2024-03-06T07:37:05.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.979+0000 7fe860400640 1 -- 172.21.15.97:0/3556396232 wait complete. 2024-03-06T07:37:05.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.981+0000 7fe860400640 1 Processor -- start 2024-03-06T07:37:05.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.981+0000 7fe860400640 1 -- start start 2024-03-06T07:37:05.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe860400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 0x7fe85815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe860400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe85815f9d0 0x7fe85815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe860400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe85815fe60 0x7fe858138b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe860400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85806cb50 con 0x7fe858144c00 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe860400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe858160730 con 0x7fe85815f9d0 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe860400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85806ccd0 con 0x7fe85815fe60 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 0x7fe85815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 0x7fe85815cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59512/0 (socket says 172.21.15.97:59512) 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 -- 172.21.15.97:0/3876741506 learned_addr learned my addr 172.21.15.97:0/3876741506 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85d000640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe85815f9d0 0x7fe85815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85e400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe85815fe60 0x7fe858138b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 -- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe85815fe60 msgr2=0x7fe858138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe85815fe60 0x7fe858138b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 -- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe85815f9d0 msgr2=0x7fe85815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe85815f9d0 0x7fe85815ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.982+0000 7fe85da00640 1 -- 172.21.15.97:0/3876741506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85815f160 con 0x7fe858144c00 2024-03-06T07:37:05.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.983+0000 7fe85da00640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 0x7fe85815cf60 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00b410 tx=0x7fe84c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.983+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c015020 con 0x7fe858144c00 2024-03-06T07:37:05.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.983+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c011830 con 0x7fe858144c00 2024-03-06T07:37:05.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.983+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c011c20 con 0x7fe858144c00 2024-03-06T07:37:05.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.983+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85815d620 con 0x7fe858144c00 2024-03-06T07:37:05.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.983+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe858138050 con 0x7fe858144c00 2024-03-06T07:37:05.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.985+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85804f330 con 0x7fe858144c00 2024-03-06T07:37:05.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.985+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe84c018020 con 0x7fe858144c00 2024-03-06T07:37:05.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.986+0000 7fe855600640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe83806b4d0 0x7fe83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.986+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe84c08a0f0 con 0x7fe858144c00 2024-03-06T07:37:05.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.986+0000 7fe85d000640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe83806b4d0 0x7fe83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.987+0000 7fe85d000640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe83806b4d0 0x7fe83806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe858071be0 tx=0x7fe844009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:05.990+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe84c01f020 con 0x7fe858144c00 2024-03-06T07:37:06.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.174+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe858139070 con 0x7fe858144c00 2024-03-06T07:37:06.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.176+0000 7fe855600640 1 -- 172.21.15.97:0/3876741506 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16709 (secure 0 0 0) 0x7fe84c0172c0 con 0x7fe858144c00 2024-03-06T07:37:06.178 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:06.178 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":134,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:36:29.869182+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:06.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe83806b4d0 msgr2=0x7fe83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe83806b4d0 0x7fe83806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe858071be0 tx=0x7fe844009040 comp rx=0 tx=0).stop 2024-03-06T07:37:06.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 msgr2=0x7fe85815cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 0x7fe85815cf60 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00b410 tx=0x7fe84c00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 shutdown_connections 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe83806b4d0 0x7fe83806d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe85815fe60 0x7fe858138b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe85815f9d0 0x7fe85815ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 --2- 172.21.15.97:0/3876741506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe858144c00 0x7fe85815cf60 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.178+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 >> 172.21.15.97:0/3876741506 conn(0x7fe8580b1730 msgr2=0x7fe858070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.179+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 shutdown_connections 2024-03-06T07:37:06.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.179+0000 7fe860400640 1 -- 172.21.15.97:0/3876741506 wait complete. 2024-03-06T07:37:06.196 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:37:06.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:06.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.382+0000 7fb7bbe00640 1 Processor -- start 2024-03-06T07:37:06.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.382+0000 7fb7bbe00640 1 -- start start 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7bbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 0x7fb7bc162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7bbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb7bc06b6d0 0x7fb7bc0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7bbe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0cef70 con 0x7fb7bc0affe0 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7bbe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7bc162270 con 0x7fb7bc06b6d0 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7bbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc15e3b0 con 0x7fb7bc15dfd0 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7b3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb7bc0affe0 0x7fb7bc06afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49184/0 (socket says 172.21.15.97:49184) 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.384+0000 7fb7b3e00640 1 -- 172.21.15.97:0/2971098307 learned_addr learned my addr 172.21.15.97:0/2971098307 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7bb400640 1 --2- 172.21.15.97:0/2971098307 >> v2:172.21.15.135:3300/0 conn(0x7fb7bc06b6d0 0x7fb7bc0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 0x7fb7bc162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516902258 0 0) 0x7fb7bc0cef70 con 0x7fb7bc0affe0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a0003520 con 0x7fb7bc0affe0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721893387 0 0) 0x7fb7a0003520 con 0x7fb7bc0affe0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 msgr2=0x7fb7bc162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 0x7fb7bc162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 >> v2:172.21.15.135:3300/0 conn(0x7fb7bc06b6d0 msgr2=0x7fb7bc0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 --2- 172.21.15.97:0/2971098307 >> v2:172.21.15.135:3300/0 conn(0x7fb7bc06b6d0 0x7fb7bc0d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc161c60 con 0x7fb7bc0affe0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb7a4001670 con 0x7fb7bc0affe0 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 >> v1:172.21.15.97:6789/0 conn(0x7fb7bc0affe0 legacy=0x7fb7bc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7a00039b0 0x7fb7a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 0x7fb7a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a0004d40 0x7fb7a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc162270 con 0x7fb7a00039b0 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc15e3b0 con 0x7fb7a0004d40 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.385+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a000ba40 con 0x7fb7bc15dfd0 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.386+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7a00039b0 0x7fb7a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.386+0000 7fb7baa00640 1 -- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 msgr2=0x7fb7a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.386+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc15dfd0 0x7fb7a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.386+0000 7fb7baa00640 1 -- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a0004d40 msgr2=0x7fb7a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.386+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a0004d40 0x7fb7a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.386+0000 7fb7baa00640 1 -- 172.21.15.97:0/2971098307 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a0003520 con 0x7fb7a00039b0 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.387+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7a00039b0 0x7fb7a0003dd0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009ac0 tx=0x7fb7a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f375109147550a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.387+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac039070 con 0x7fb7a00039b0 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.387+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ac02c710 con 0x7fb7a00039b0 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.387+0000 7fb7ba000640 1 -- 172.21.15.97:0/2971098307 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac009690 con 0x7fb7a00039b0 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2971098307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7a00039b0 msgr2=0x7fb7a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2971098307 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7a00039b0 0x7fb7a0003dd0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009ac0 tx=0x7fb7a0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:06.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bb400640 1 -- 172.21.15.97:0/2971098307 reap_dead start 2024-03-06T07:37:06.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2971098307 shutdown_connections 2024-03-06T07:37:06.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2971098307 >> 172.21.15.97:0/2971098307 conn(0x7fb7bc06c0d0 msgr2=0x7fb7bc06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2971098307 shutdown_connections 2024-03-06T07:37:06.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.388+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2971098307 wait complete. 2024-03-06T07:37:06.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.391+0000 7fb7bbe00640 1 Processor -- start 2024-03-06T07:37:06.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.391+0000 7fb7bbe00640 1 -- start start 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.392+0000 7fb7bbe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a00039b0 0x7fb7bc1f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.392+0000 7fb7bbe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc146110 0x7fb7bc153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.392+0000 7fb7bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 0x7fb7bc0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.392+0000 7fb7bbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc0ce8d0 con 0x7fb7bc153f60 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.392+0000 7fb7bbe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc161f00 con 0x7fb7a00039b0 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bbe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc0cea50 con 0x7fb7bc146110 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 0x7fb7bc0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 0x7fb7bc0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59540/0 (socket says 172.21.15.97:59540) 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 -- 172.21.15.97:0/2222277408 learned_addr learned my addr 172.21.15.97:0/2222277408 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 -- 172.21.15.97:0/2222277408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc146110 msgr2=0x7fb7bc153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc146110 0x7fb7bc153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 -- 172.21.15.97:0/2222277408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a00039b0 msgr2=0x7fb7bc1f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a00039b0 0x7fb7bc1f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 -- 172.21.15.97:0/2222277408 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc142080 con 0x7fb7bc153f60 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.393+0000 7fb7bb400640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 0x7fb7bc0b46c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc0cde70 tx=0x7fb7b400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.394+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400bae0 con 0x7fb7bc153f60 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.394+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b401a070 con 0x7fb7bc153f60 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.394+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc142320 con 0x7fb7bc153f60 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.394+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4014400 con 0x7fb7bc153f60 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.394+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc1428e0 con 0x7fb7bc153f60 2024-03-06T07:37:06.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.395+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc04f330 con 0x7fb7bc153f60 2024-03-06T07:37:06.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.396+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7b4005d10 con 0x7fb7bc153f60 2024-03-06T07:37:06.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.398+0000 7fb7b2a00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb79806b550 0x7fb79806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.398+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb7b408a170 con 0x7fb7bc153f60 2024-03-06T07:37:06.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.398+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb79806b550 0x7fb79806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.399+0000 7fb7baa00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb79806b550 0x7fb79806da10 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac036000 tx=0x7fb7ac02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.402+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7b4006000 con 0x7fb7bc153f60 2024-03-06T07:37:06.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.695+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7bc0b5e60 con 0x7fb7bc153f60 2024-03-06T07:37:06.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.696+0000 7fb7b2a00640 1 -- 172.21.15.97:0/2222277408 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3399 (secure 0 0 0) 0x7fb7b4059010 con 0x7fb7bc153f60 2024-03-06T07:37:06.697 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:06.697 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/2007524244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2007524244},{"type":"v1","addr":"172.21.15.97:6836","nonce":2007524244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:57.921337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:37:06.698 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 99 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb79806b550 msgr2=0x7fb79806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb79806b550 0x7fb79806da10 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac036000 tx=0x7fb7ac02e000 comp rx=0 tx=0).stop 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 msgr2=0x7fb7bc0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 0x7fb7bc0b46c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc0cde70 tx=0x7fb7b400b480 comp rx=0 tx=0).stop 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 shutdown_connections 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb79806b550 0x7fb79806da10 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc153f60 0x7fb7bc0b46c0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7bc146110 0x7fb7bc153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 --2- 172.21.15.97:0/2222277408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7a00039b0 0x7fb7bc1f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.698+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 >> 172.21.15.97:0/2222277408 conn(0x7fb7bc06c0d0 msgr2=0x7fb7bc0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.699+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 shutdown_connections 2024-03-06T07:37:06.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.699+0000 7fb7bbe00640 1 -- 172.21.15.97:0/2222277408 wait complete. 2024-03-06T07:37:06.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:06.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.909+0000 7fa053800640 1 Processor -- start 2024-03-06T07:37:06.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.909+0000 7fa053800640 1 -- start start 2024-03-06T07:37:06.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.911+0000 7fa053800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa04c144c00 0x7fa04c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.911+0000 7fa053800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.911+0000 7fa053800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c06ee70 con 0x7fa04c15f9d0 2024-03-06T07:37:06.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.911+0000 7fa053800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c06cef0 con 0x7fa04c144c00 2024-03-06T07:37:06.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.911+0000 7fa053800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c06d1d0 con 0x7fa04c15fe60 2024-03-06T07:37:06.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.912+0000 7fa051800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.912+0000 7fa050e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa04c144c00 0x7fa04c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.912+0000 7fa04be00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa04c15f9d0 0x7fa04c0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:51648/0 (socket says 172.21.15.97:51648) 2024-03-06T07:37:06.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.912+0000 7fa04be00640 1 -- 172.21.15.97:0/3265849832 learned_addr learned my addr 172.21.15.97:0/3265849832 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.913+0000 7fa050e00640 1 -- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 msgr2=0x7fa04c0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.913+0000 7fa050e00640 1 --2- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c0ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.913+0000 7fa050e00640 1 -- 172.21.15.97:0/3265849832 >> v1:172.21.15.135:6789/0 conn(0x7fa04c15f9d0 legacy=0x7fa04c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.913+0000 7fa04b400640 1 -- 172.21.15.97:0/3265849832 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618845995 0 0) 0x7fa038001110 con 0x7fa04c15f9d0 2024-03-06T07:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.913+0000 7fa050e00640 1 -- 172.21.15.97:0/3265849832 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c06d4b0 con 0x7fa04c144c00 2024-03-06T07:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.913+0000 7fa051800640 1 --2- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:37:06.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.916+0000 7fa050e00640 1 --2- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c06e930 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa0400097a0 tx=0x7fa04002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d56f64efb300cc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.916+0000 7fa04b400640 1 -- 172.21.15.97:0/3265849832 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040035020 con 0x7fa04c144c00 2024-03-06T07:37:06.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.916+0000 7fa04b400640 1 -- 172.21.15.97:0/3265849832 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040031830 con 0x7fa04c144c00 2024-03-06T07:37:06.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.916+0000 7fa04b400640 1 -- 172.21.15.97:0/3265849832 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040031c00 con 0x7fa04c144c00 2024-03-06T07:37:06.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 -- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 msgr2=0x7fa04c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 --2- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c06e930 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa0400097a0 tx=0x7fa04002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 -- 172.21.15.97:0/3265849832 shutdown_connections 2024-03-06T07:37:06.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 --2- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 --2- 172.21.15.97:0/3265849832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c06e930 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 -- 172.21.15.97:0/3265849832 >> 172.21.15.97:0/3265849832 conn(0x7fa04c0b1730 msgr2=0x7fa04c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 -- 172.21.15.97:0/3265849832 shutdown_connections 2024-03-06T07:37:06.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.917+0000 7fa053800640 1 -- 172.21.15.97:0/3265849832 wait complete. 2024-03-06T07:37:06.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.919+0000 7fa053800640 1 Processor -- start 2024-03-06T07:37:06.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.919+0000 7fa053800640 1 -- start start 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa053800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c1f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa053800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 0x7fa04c143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa053800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c1f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa053800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c160730 con 0x7fa04c15f9d0 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa053800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c160290 con 0x7fa04c144c00 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa053800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c06cb50 con 0x7fa04c15fe60 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa04be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 0x7fa04c143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.920+0000 7fa04be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 0x7fa04c143080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59552/0 (socket says 172.21.15.97:59552) 2024-03-06T07:37:06.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 -- 172.21.15.97:0/2705223552 learned_addr learned my addr 172.21.15.97:0/2705223552 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa050e00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c1f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa051800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c1f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 -- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 msgr2=0x7fa04c1f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c1f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 -- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 msgr2=0x7fa04c1f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c1f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 -- 172.21.15.97:0/2705223552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c153d10 con 0x7fa04c15f9d0 2024-03-06T07:37:06.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa051800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.921+0000 7fa04be00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 0x7fa04c143080 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa03800c7b0 tx=0x7fa03800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.922+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa038018070 con 0x7fa04c15f9d0 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.922+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0380072d0 con 0x7fa04c15f9d0 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.922+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c1f4c90 con 0x7fa04c15f9d0 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.922+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0380047e0 con 0x7fa04c15f9d0 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.922+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c1f7ce0 con 0x7fa04c15f9d0 2024-03-06T07:37:06.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.923+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c04f330 con 0x7fa04c15f9d0 2024-03-06T07:37:06.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.924+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa03801b070 con 0x7fa04c15f9d0 2024-03-06T07:37:06.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.925+0000 7fa048c00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa02c06b4d0 0x7fa02c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.925+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa038089490 con 0x7fa04c15f9d0 2024-03-06T07:37:06.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.925+0000 7fa050e00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa02c06b4d0 0x7fa02c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.926+0000 7fa050e00640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa02c06b4d0 0x7fa02c06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa040036000 tx=0x7fa04003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:06.930+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa04c06ee70 con 0x7fa04c15f9d0 2024-03-06T07:37:07.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.150+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa04c072890 con 0x7fa04c15f9d0 2024-03-06T07:37:07.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.151+0000 7fa048c00640 1 -- 172.21.15.97:0/2705223552 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3399 (secure 0 0 0) 0x7fa038058360 con 0x7fa04c15f9d0 2024-03-06T07:37:07.153 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:07.153 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/2007524244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2007524244},{"type":"v1","addr":"172.21.15.97:6836","nonce":2007524244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:43.821565+0000","modified":"2024-03-06T07:36:57.921337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8382},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8382":{"gid":8382,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6837/3564321877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":3564321877},{"type":"v1","addr":"172.21.15.97:6837","nonce":3564321877}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8382,"qdb_cluster":[8382]},"id":9}]} 2024-03-06T07:37:07.153 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 99 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa02c06b4d0 msgr2=0x7fa02c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa02c06b4d0 0x7fa02c06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa040036000 tx=0x7fa04003c040 comp rx=0 tx=0).stop 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 msgr2=0x7fa04c143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 0x7fa04c143080 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa03800c7b0 tx=0x7fa03800cc70 comp rx=0 tx=0).stop 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 shutdown_connections 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa02c06b4d0 0x7fa02c06d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa04c15fe60 0x7fa04c1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa04c15f9d0 0x7fa04c143080 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 --2- 172.21.15.97:0/2705223552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa04c144c00 0x7fa04c1f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 >> 172.21.15.97:0/2705223552 conn(0x7fa04c0b1730 msgr2=0x7fa04c1451b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.154+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 shutdown_connections 2024-03-06T07:37:07.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.155+0000 7fa053800640 1 -- 172.21.15.97:0/2705223552 wait complete. 2024-03-06T07:37:07.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:07.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.367+0000 7f5d25e00640 1 Processor -- start 2024-03-06T07:37:07.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.368+0000 7f5d25e00640 1 -- start start 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d25e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5d200cdd80 0x7f5d201627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d25e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d2006c2e0 0x7f5d2006cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d25e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d200d0010 con 0x7f5d2006d020 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d25e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2006d470 con 0x7f5d2006c2e0 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d25e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d200d0ec0 con 0x7f5d200cdd80 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d16000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d2006c2e0 0x7f5d2006cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d1f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5d2006d020 0x7f5d2015d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49192/0 (socket says 172.21.15.97:49192) 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.369+0000 7f5d1f400640 1 -- 172.21.15.97:0/1475486719 learned_addr learned my addr 172.21.15.97:0/1475486719 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d1ea00640 1 --2- 172.21.15.97:0/1475486719 >> v2:172.21.15.135:3300/0 conn(0x7f5d200cdd80 0x7f5d201627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d16000640 1 -- 172.21.15.97:0/1475486719 >> v2:172.21.15.135:3300/0 conn(0x7f5d200cdd80 msgr2=0x7f5d201627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d16000640 1 --2- 172.21.15.97:0/1475486719 >> v2:172.21.15.135:3300/0 conn(0x7f5d200cdd80 0x7f5d201627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d16000640 1 -- 172.21.15.97:0/1475486719 >> v1:172.21.15.97:6789/0 conn(0x7f5d2006d020 legacy=0x7f5d2015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d16000640 1 -- 172.21.15.97:0/1475486719 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d200d02b0 con 0x7f5d2006c2e0 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d1e000640 1 -- 172.21.15.97:0/1475486719 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911631505 0 0) 0x7f5d200d0010 con 0x7f5d2006d020 2024-03-06T07:37:07.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.370+0000 7f5d1ea00640 1 --2- 172.21.15.97:0/1475486719 >> v2:172.21.15.135:3300/0 conn(0x7f5d200cdd80 0x7f5d201627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:07.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.371+0000 7f5d16000640 1 --2- 172.21.15.97:0/1475486719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006c2e0 0x7f5d2006cae0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f5d2006e3d0 tx=0x7f5d0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e279010d62ab546 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d1e000640 1 -- 172.21.15.97:0/1475486719 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08035020 con 0x7f5d2006c2e0 2024-03-06T07:37:07.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d1e000640 1 -- 172.21.15.97:0/1475486719 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08031830 con 0x7f5d2006c2e0 2024-03-06T07:37:07.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d1e000640 1 -- 172.21.15.97:0/1475486719 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08031c00 con 0x7f5d2006c2e0 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 -- 172.21.15.97:0/1475486719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006c2e0 msgr2=0x7f5d2006cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1475486719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006c2e0 0x7f5d2006cae0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f5d2006e3d0 tx=0x7f5d0802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 -- 172.21.15.97:0/1475486719 shutdown_connections 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1475486719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006c2e0 0x7f5d2006cae0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1475486719 >> v2:172.21.15.135:3300/0 conn(0x7f5d200cdd80 0x7f5d201627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 -- 172.21.15.97:0/1475486719 >> 172.21.15.97:0/1475486719 conn(0x7f5d200b3870 msgr2=0x7f5d200b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 -- 172.21.15.97:0/1475486719 shutdown_connections 2024-03-06T07:37:07.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.372+0000 7f5d25e00640 1 -- 172.21.15.97:0/1475486719 wait complete. 2024-03-06T07:37:07.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.374+0000 7f5d25e00640 1 Processor -- start 2024-03-06T07:37:07.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.374+0000 7f5d25e00640 1 -- start start 2024-03-06T07:37:07.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d25e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2006c2e0 0x7f5d201d6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d25e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 0x7f5d200cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d25e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d200cdd80 0x7f5d2009bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d25e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200d0b50 con 0x7f5d2006d020 2024-03-06T07:37:07.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d25e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200d0870 con 0x7f5d2006c2e0 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d25e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200ce160 con 0x7f5d200cdd80 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d16000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 0x7f5d200cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d16000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 0x7f5d200cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59574/0 (socket says 172.21.15.97:59574) 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.375+0000 7f5d16000640 1 -- 172.21.15.97:0/1419989831 learned_addr learned my addr 172.21.15.97:0/1419989831 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d16000640 1 -- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d200cdd80 msgr2=0x7f5d2009bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d1ea00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2006c2e0 0x7f5d201d6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d16000640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d200cdd80 0x7f5d2009bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d16000640 1 -- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2006c2e0 msgr2=0x7f5d201d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d16000640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2006c2e0 0x7f5d201d6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d16000640 1 -- 172.21.15.97:0/1419989831 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d201d7430 con 0x7f5d2006d020 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d16000640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 0x7f5d200cb270 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f5d08000c00 tx=0x7f5d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08040070 con 0x7f5d2006d020 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08039a20 con 0x7f5d2006d020 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08035040 con 0x7f5d2006d020 2024-03-06T07:37:07.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.376+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d201d7710 con 0x7f5d2006d020 2024-03-06T07:37:07.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.377+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d201d7cd0 con 0x7f5d2006d020 2024-03-06T07:37:07.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.378+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2006bbc0 con 0x7f5d2006d020 2024-03-06T07:37:07.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.379+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d0804b060 con 0x7f5d2006d020 2024-03-06T07:37:07.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.379+0000 7f5d17e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.380+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5d080abc60 con 0x7f5d2006d020 2024-03-06T07:37:07.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.380+0000 7f5d1ea00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.380+0000 7f5d1ea00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5d201e5f40 tx=0x7f5d100084f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.389+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d0807aaa0 con 0x7f5d2006d020 2024-03-06T07:37:07.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.570+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d200d1570 con 0x7f5d2006d020 2024-03-06T07:37:07.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.573+0000 7f5d17e00640 1 -- 172.21.15.97:0/1419989831 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16709 (secure 0 0 0) 0x7f5d0807a3a0 con 0x7f5d2006d020 2024-03-06T07:37:07.574 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:07.574 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":134,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:36:29.869182+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:41.634478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:42.840855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.22","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:07.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cfc06b4d0 msgr2=0x7f5cfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5d201e5f40 tx=0x7f5d100084f0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 msgr2=0x7f5d200cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 0x7f5d200cb270 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f5d08000c00 tx=0x7f5d080046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 shutdown_connections 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d200cdd80 0x7f5d2009bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d2006d020 0x7f5d200cb270 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 --2- 172.21.15.97:0/1419989831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d2006c2e0 0x7f5d201d6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 >> 172.21.15.97:0/1419989831 conn(0x7f5d200b3870 msgr2=0x7f5d200b4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 shutdown_connections 2024-03-06T07:37:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.575+0000 7f5d25e00640 1 -- 172.21.15.97:0/1419989831 wait complete. 2024-03-06T07:37:07.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:37:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.794+0000 7fb2eac00640 1 Processor -- start 2024-03-06T07:37:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.794+0000 7fb2eac00640 1 -- start start 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2eac00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb2e406afc0 0x7fb2e406b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2eac00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb2e40cacb0 0x7fb2e40cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2eac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e40cc5c0 0x7fb2e4161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2eac00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e406b8e0 con 0x7fb2e40cacb0 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2eac00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e40cd320 con 0x7fb2e406afc0 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2eac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e40cca80 con 0x7fb2e40cc5c0 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb2e40cacb0 0x7fb2e40cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb2e40cacb0 0x7fb2e40cc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59582/0 (socket says 172.21.15.97:59582) 2024-03-06T07:37:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 -- 172.21.15.97:0/2431742292 learned_addr learned my addr 172.21.15.97:0/2431742292 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 -- 172.21.15.97:0/2431742292 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e40cc5c0 msgr2=0x7fb2e4161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 --2- 172.21.15.97:0/2431742292 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e40cc5c0 0x7fb2e4161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 -- 172.21.15.97:0/2431742292 >> v2:172.21.15.135:3300/0 conn(0x7fb2e406afc0 msgr2=0x7fb2e406b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3e00640 1 --2- 172.21.15.97:0/2431742292 >> v2:172.21.15.135:3300/0 conn(0x7fb2e406afc0 0x7fb2e406b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.796+0000 7fb2e3400640 1 --2- 172.21.15.97:0/2431742292 >> v2:172.21.15.135:3300/0 conn(0x7fb2e406afc0 0x7fb2e406b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.797+0000 7fb2e3400640 1 -- 172.21.15.97:0/2431742292 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e40ccd60 con 0x7fb2e40cacb0 2024-03-06T07:37:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.797+0000 7fb2e3e00640 1 --2- 172.21.15.97:0/2431742292 >> v2:172.21.15.135:3300/0 conn(0x7fb2e406afc0 0x7fb2e406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.797+0000 7fb2e3400640 1 --2- 172.21.15.97:0/2431742292 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e40cc080 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40676a0 tx=0x7fb2d002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3846ef92931c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.798+0000 7fb2e2a00640 1 -- 172.21.15.97:0/2431742292 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00317d0 con 0x7fb2e40cacb0 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.798+0000 7fb2e2a00640 1 -- 172.21.15.97:0/2431742292 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d0037070 con 0x7fb2e40cacb0 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.798+0000 7fb2e2a00640 1 -- 172.21.15.97:0/2431742292 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0031b90 con 0x7fb2e40cacb0 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.798+0000 7fb2eac00640 1 -- 172.21.15.97:0/2431742292 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 msgr2=0x7fb2e40cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.798+0000 7fb2eac00640 1 --2- 172.21.15.97:0/2431742292 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e40cc080 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40676a0 tx=0x7fb2d002b870 comp rx=0 tx=0).stop 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.798+0000 7fb2eac00640 1 -- 172.21.15.97:0/2431742292 shutdown_connections 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.799+0000 7fb2eac00640 1 --2- 172.21.15.97:0/2431742292 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e40cc5c0 0x7fb2e4161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.799+0000 7fb2eac00640 1 --2- 172.21.15.97:0/2431742292 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e40cc080 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.799+0000 7fb2eac00640 1 --2- 172.21.15.97:0/2431742292 >> v2:172.21.15.135:3300/0 conn(0x7fb2e406afc0 0x7fb2e406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.799+0000 7fb2eac00640 1 -- 172.21.15.97:0/2431742292 >> 172.21.15.97:0/2431742292 conn(0x7fb2e40d0b90 msgr2=0x7fb2e40d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.799+0000 7fb2eac00640 1 -- 172.21.15.97:0/2431742292 shutdown_connections 2024-03-06T07:37:07.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.799+0000 7fb2eac00640 1 -- 172.21.15.97:0/2431742292 wait complete. 2024-03-06T07:37:07.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.800+0000 7fb2eac00640 1 Processor -- start 2024-03-06T07:37:07.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.800+0000 7fb2eac00640 1 -- start start 2024-03-06T07:37:07.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2eac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e406afc0 0x7fb2e4141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2eac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e415e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2eac00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e40cc5c0 0x7fb2e415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2eac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e4144b00 con 0x7fb2e40cacb0 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2eac00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e4144980 con 0x7fb2e40cc5c0 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2eac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e4162170 con 0x7fb2e406afc0 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.801+0000 7fb2e3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e415e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e415e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59588/0 (socket says 172.21.15.97:59588) 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 -- 172.21.15.97:0/3142786619 learned_addr learned my addr 172.21.15.97:0/3142786619 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 -- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e406afc0 msgr2=0x7fb2e4141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e406afc0 0x7fb2e4141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 -- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e40cc5c0 msgr2=0x7fb2e415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e8c00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e40cc5c0 0x7fb2e415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e40cc5c0 0x7fb2e415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 -- 172.21.15.97:0/3142786619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e415eef0 con 0x7fb2e40cacb0 2024-03-06T07:37:07.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.802+0000 7fb2e3400640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e415e530 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000c00 tx=0x7fb2d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.803+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0032870 con 0x7fb2e40cacb0 2024-03-06T07:37:07.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.803+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d0032a30 con 0x7fb2e40cacb0 2024-03-06T07:37:07.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.803+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e415f1d0 con 0x7fb2e40cacb0 2024-03-06T07:37:07.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.803+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0032e20 con 0x7fb2e40cacb0 2024-03-06T07:37:07.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.803+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e40b6ae0 con 0x7fb2e40cacb0 2024-03-06T07:37:07.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.804+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e415f6f0 con 0x7fb2e40cacb0 2024-03-06T07:37:07.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.805+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d004c060 con 0x7fb2e40cacb0 2024-03-06T07:37:07.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.806+0000 7fb2dbe00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2c406b4d0 0x7fb2c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.806+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb2d00abe30 con 0x7fb2e40cacb0 2024-03-06T07:37:07.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.806+0000 7fb2e3e00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2c406b4d0 0x7fb2c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.807+0000 7fb2e3e00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2c406b4d0 0x7fb2c406d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4006fd0 tx=0x7fb2d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:07.810+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2e415f6f0 con 0x7fb2e40cacb0 2024-03-06T07:37:08.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.065+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb2e40b3ab0 con 0x7fb2e40cacb0 2024-03-06T07:37:08.067 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:37:08.066+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:37:08.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.069+0000 7fb2dbe00640 1 -- 172.21.15.97:0/3142786619 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb2e40b3ab0 con 0x7fb2e40cacb0 2024-03-06T07:37:08.070 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2c406b4d0 msgr2=0x7fb2c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2c406b4d0 0x7fb2c406d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4006fd0 tx=0x7fb2d4008040 comp rx=0 tx=0).stop 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 msgr2=0x7fb2e415e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e415e530 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000c00 tx=0x7fb2d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 shutdown_connections 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb2c406b4d0 0x7fb2c406d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e40cc5c0 0x7fb2e415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e40cacb0 0x7fb2e415e530 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 --2- 172.21.15.97:0/3142786619 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2e406afc0 0x7fb2e4141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.073+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 >> 172.21.15.97:0/3142786619 conn(0x7fb2e40d0b90 msgr2=0x7fb2e40b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:08.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.074+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 shutdown_connections 2024-03-06T07:37:08.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.075+0000 7fb2eac00640 1 -- 172.21.15.97:0/3142786619 wait complete. 2024-03-06T07:37:08.088 INFO:tasks.ceph.mds.a.smithi097.stderr: -9258> 2024-03-06T07:35:48.740+0000 7fa3d5000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:37:08.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:37:08.161 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:37:08.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.280+0000 7f0552000640 1 Processor -- start 2024-03-06T07:37:08.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.280+0000 7f0552000640 1 -- start start 2024-03-06T07:37:08.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.281+0000 7f0552000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 0x7f054c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.281+0000 7f0552000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c15b130 con 0x7f054c158950 2024-03-06T07:37:08.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.281+0000 7f0552000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c0af170 con 0x7f054c0cfde0 2024-03-06T07:37:08.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.281+0000 7f0552000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c15b3f0 con 0x7f054c158570 2024-03-06T07:37:08.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f054c158950 0x7f054c0cf6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49208/0 (socket says 172.21.15.97:49208) 2024-03-06T07:37:08.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054be00640 1 -- 172.21.15.97:0/3635015534 learned_addr learned my addr 172.21.15.97:0/3635015534 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f0550c00640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 0x7f054c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898065564 0 0) 0x7f054c15b130 con 0x7f054c158950 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0534003520 con 0x7f054c158950 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079905165 0 0) 0x7f0534003520 con 0x7f054c158950 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 msgr2=0x7f054c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.282+0000 7f054b400640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 0x7f054c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 >> v1:172.21.15.135:6789/0 conn(0x7f054c0cfde0 legacy=0x7f054c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c15b6d0 con 0x7f054c158950 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0538001670 con 0x7f054c158950 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 >> v1:172.21.15.97:6789/0 conn(0x7f054c158950 legacy=0x7f054c0cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05340039e0 0x7f0534003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0534004800 0x7f0534004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 0x7f0534005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c15b3f0 con 0x7f05340039e0 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053400baf0 con 0x7f0534004800 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053400df00 con 0x7f054c158570 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f0550c00640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05340039e0 0x7f0534003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f0550c00640 1 -- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 msgr2=0x7f0534005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f0550c00640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c158570 0x7f0534005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f0550c00640 1 -- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0534004800 msgr2=0x7f0534004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f0550c00640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0534004800 0x7f0534004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.283+0000 7f0550c00640 1 -- 172.21.15.97:0/3635015534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534003520 con 0x7f05340039e0 2024-03-06T07:37:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.284+0000 7f0550c00640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05340039e0 0x7f0534003e00 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f0540009ac0 tx=0x7f0534005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a05e42e89b10c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540039070 con 0x7f05340039e0 2024-03-06T07:37:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054002c710 con 0x7f05340039e0 2024-03-06T07:37:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f054b400640 1 -- 172.21.15.97:0/3635015534 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540009690 con 0x7f05340039e0 2024-03-06T07:37:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f0552000640 1 -- 172.21.15.97:0/3635015534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05340039e0 msgr2=0x7f0534003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f0552000640 1 --2- 172.21.15.97:0/3635015534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05340039e0 0x7f0534003e00 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f0540009ac0 tx=0x7f0534005830 comp rx=0 tx=0).stop 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f0551600640 1 -- 172.21.15.97:0/3635015534 reap_dead start 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f0552000640 1 -- 172.21.15.97:0/3635015534 shutdown_connections 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.285+0000 7f0552000640 1 -- 172.21.15.97:0/3635015534 >> 172.21.15.97:0/3635015534 conn(0x7f054c06afc0 msgr2=0x7f054c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.286+0000 7f0552000640 1 -- 172.21.15.97:0/3635015534 shutdown_connections 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.286+0000 7f0552000640 1 -- 172.21.15.97:0/3635015534 wait complete. 2024-03-06T07:37:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.287+0000 7f0552000640 1 Processor -- start 2024-03-06T07:37:08.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.287+0000 7f0552000640 1 -- start start 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f0552000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05340039e0 0x7f054c14ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f0552000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 0x7f054c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f0552000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c14b430 0x7f054c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f0552000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c15bed0 con 0x7f054c154e20 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f0552000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0b4950 con 0x7f05340039e0 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f0552000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c15c050 con 0x7f054c14b430 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.288+0000 7f054be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 0x7f054c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 0x7f054c15cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59616/0 (socket says 172.21.15.97:59616) 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 -- 172.21.15.97:0/897521312 learned_addr learned my addr 172.21.15.97:0/897521312 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f0550c00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05340039e0 0x7f054c14ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:08.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 -- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c14b430 msgr2=0x7f054c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c14b430 0x7f054c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 -- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05340039e0 msgr2=0x7f054c14ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05340039e0 0x7f054c14ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 -- 172.21.15.97:0/897521312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c15ac80 con 0x7f054c154e20 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f054be00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 0x7f054c15cac0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f053800c750 tx=0x7f053800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538018070 con 0x7f054c154e20 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05380074e0 con 0x7f054c154e20 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.289+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053800ce60 con 0x7f054c154e20 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.290+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c15a9a0 con 0x7f054c154e20 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.290+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c154620 con 0x7f054c154e20 2024-03-06T07:37:08.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.291+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c0b0ea0 con 0x7f054c154e20 2024-03-06T07:37:08.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.292+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0538004a00 con 0x7f054c154e20 2024-03-06T07:37:08.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.293+0000 7f0548c00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052c06b500 0x7f052c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:08.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.293+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0538089760 con 0x7f054c154e20 2024-03-06T07:37:08.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.293+0000 7f0550c00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052c06b500 0x7f052c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:08.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.294+0000 7f0550c00640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052c06b500 0x7f052c06d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0540036000 tx=0x7f0540030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:08.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.297+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f054c0b0ea0 con 0x7f054c154e20 2024-03-06T07:37:08.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:08.530+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f054c04f330 con 0x7f054c154e20 2024-03-06T07:37:09.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.072+0000 7f0548c00640 1 -- 172.21.15.97:0/897521312 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v101) v1 ==== 104+0+0 (secure 0 0 0) 0x7f053805ce20 con 0x7f054c154e20 2024-03-06T07:37:09.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052c06b500 msgr2=0x7f052c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052c06b500 0x7f052c06d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0540036000 tx=0x7f0540030000 comp rx=0 tx=0).stop 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 msgr2=0x7f054c15cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 0x7f054c15cac0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f053800c750 tx=0x7f053800cc10 comp rx=0 tx=0).stop 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 shutdown_connections 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f052c06b500 0x7f052c06d9c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f054c14b430 0x7f054c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f054c154e20 0x7f054c15cac0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 --2- 172.21.15.97:0/897521312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05340039e0 0x7f054c14ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 >> 172.21.15.97:0/897521312 conn(0x7f054c06afc0 msgr2=0x7f054c155340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.076+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 shutdown_connections 2024-03-06T07:37:09.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.077+0000 7f0552000640 1 -- 172.21.15.97:0/897521312 wait complete. 2024-03-06T07:37:09.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:37:09.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:09.304 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:09.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.307+0000 7f8e9e600640 1 Processor -- start 2024-03-06T07:37:09.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.307+0000 7f8e9e600640 1 -- start start 2024-03-06T07:37:09.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.308+0000 7f8e9e600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980af540 0x7f8e98066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.308+0000 7f8e9e600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8e980b4fe0 0x7f8e980b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.308+0000 7f8e9e600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980cd880 con 0x7f8e9815e2c0 2024-03-06T07:37:09.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.308+0000 7f8e9e600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980b08d0 con 0x7f8e980b4fe0 2024-03-06T07:37:09.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.308+0000 7f8e9e600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980cdb60 con 0x7f8e980af540 2024-03-06T07:37:09.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.309+0000 7f8e97e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8e980b4fe0 0x7f8e980b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.309+0000 7f8e8fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980af540 0x7f8e98066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.309+0000 7f8e97e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8e980b4fe0 0x7f8e980b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:56902/0 (socket says 172.21.15.97:56902) 2024-03-06T07:37:09.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.309+0000 7f8e97e00640 1 -- 172.21.15.97:0/2688138246 learned_addr learned my addr 172.21.15.97:0/2688138246 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:09.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.310+0000 7f8e97e00640 1 -- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980af540 msgr2=0x7f8e98066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.310+0000 7f8e97e00640 1 --2- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980af540 0x7f8e98066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.310+0000 7f8e97e00640 1 -- 172.21.15.97:0/2688138246 >> v1:172.21.15.135:6789/0 conn(0x7f8e9815e2c0 legacy=0x7f8e980b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.310+0000 7f8e97e00640 1 -- 172.21.15.97:0/2688138246 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e980b0560 con 0x7f8e980b4fe0 2024-03-06T07:37:09.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.311+0000 7f8e97e00640 1 --2- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e980b4fe0 0x7f8e980b5470 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009e50 tx=0x7f8e9002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2edea1777927605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.311+0000 7f8e96a00640 1 -- 172.21.15.97:0/2688138246 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9002c9e0 con 0x7f8e980b4fe0 2024-03-06T07:37:09.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.311+0000 7f8e96a00640 1 -- 172.21.15.97:0/2688138246 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9002cb80 con 0x7f8e980b4fe0 2024-03-06T07:37:09.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.311+0000 7f8e96a00640 1 -- 172.21.15.97:0/2688138246 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e900343f0 con 0x7f8e980b4fe0 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 -- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e980b4fe0 msgr2=0x7f8e980b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 --2- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e980b4fe0 0x7f8e980b5470 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009e50 tx=0x7f8e9002be40 comp rx=0 tx=0).stop 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 -- 172.21.15.97:0/2688138246 shutdown_connections 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 --2- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980af540 0x7f8e98066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 --2- 172.21.15.97:0/2688138246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e980b4fe0 0x7f8e980b5470 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 -- 172.21.15.97:0/2688138246 >> 172.21.15.97:0/2688138246 conn(0x7f8e980b1e80 msgr2=0x7f8e980b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:09.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 -- 172.21.15.97:0/2688138246 shutdown_connections 2024-03-06T07:37:09.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.312+0000 7f8e9e600640 1 -- 172.21.15.97:0/2688138246 wait complete. 2024-03-06T07:37:09.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.314+0000 7f8e9e600640 1 Processor -- start 2024-03-06T07:37:09.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.314+0000 7f8e9e600640 1 -- start start 2024-03-06T07:37:09.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e9e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 0x7f8e980c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e9e600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980b4fe0 0x7f8e980ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e9e600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e9815e2c0 0x7f8e981f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e9e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980b0180 con 0x7f8e980af540 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e9e600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9815e6a0 con 0x7f8e9815e2c0 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e9e600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980b0300 con 0x7f8e980b4fe0 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.315+0000 7f8e97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 0x7f8e980c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 0x7f8e980c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59622/0 (socket says 172.21.15.97:59622) 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 -- 172.21.15.97:0/4117946511 learned_addr learned my addr 172.21.15.97:0/4117946511 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e8fe00640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980b4fe0 0x7f8e980ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97e00640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e9815e2c0 0x7f8e981f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 -- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980b4fe0 msgr2=0x7f8e980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980b4fe0 0x7f8e980ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 -- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e9815e2c0 msgr2=0x7f8e981f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e9815e2c0 0x7f8e981f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 -- 172.21.15.97:0/4117946511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e980c4d30 con 0x7f8e980af540 2024-03-06T07:37:09.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97e00640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e9815e2c0 0x7f8e981f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.316+0000 7f8e97400640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 0x7f8e980c7400 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c3c0 tx=0x7f8e8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.317+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e880052d0 con 0x7f8e980af540 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.317+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e88005490 con 0x7f8e980af540 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.317+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88007260 con 0x7f8e980af540 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.317+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e980c7b00 con 0x7f8e980af540 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.317+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e980c4250 con 0x7f8e980af540 2024-03-06T07:37:09.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.318+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9804f3a0 con 0x7f8e980af540 2024-03-06T07:37:09.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.319+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e88007400 con 0x7f8e980af540 2024-03-06T07:37:09.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.320+0000 7f8e8f400640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e7406b500 0x7f8e7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.320+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8e88089640 con 0x7f8e980af540 2024-03-06T07:37:09.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.320+0000 7f8e8fe00640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e7406b500 0x7f8e7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.321+0000 7f8e8fe00640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e7406b500 0x7f8e7406d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8e980b5ff0 tx=0x7f8e80009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.325+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e88057dd0 con 0x7f8e980af540 2024-03-06T07:37:09.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:09.526+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e98074540 con 0x7f8e980af540 2024-03-06T07:37:10.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.250+0000 7f8e8f400640 1 -- 172.21.15.97:0/4117946511 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v136) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8e8805be70 con 0x7f8e980af540 2024-03-06T07:37:10.251 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.255+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e7406b500 msgr2=0x7f8e7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.255+0000 7f8e9e600640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e7406b500 0x7f8e7406d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8e980b5ff0 tx=0x7f8e80009190 comp rx=0 tx=0).stop 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.255+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 msgr2=0x7f8e980c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.255+0000 7f8e9e600640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 0x7f8e980c7400 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c3c0 tx=0x7f8e8800c880 comp rx=0 tx=0).stop 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.256+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 shutdown_connections 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.256+0000 7f8e9e600640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8e7406b500 0x7f8e7406d9c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.256+0000 7f8e9e600640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e9815e2c0 0x7f8e981f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.256+0000 7f8e9e600640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8e980b4fe0 0x7f8e980ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.256+0000 7f8e9e600640 1 --2- 172.21.15.97:0/4117946511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e980af540 0x7f8e980c7400 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.256+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 >> 172.21.15.97:0/4117946511 conn(0x7f8e980b1e80 msgr2=0x7f8e9806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.257+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 shutdown_connections 2024-03-06T07:37:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.257+0000 7f8e9e600640 1 -- 172.21.15.97:0/4117946511 wait complete. 2024-03-06T07:37:10.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:37:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.485+0000 7fd2eb000640 1 Processor -- start 2024-03-06T07:37:10.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.485+0000 7fd2eb000640 1 -- start start 2024-03-06T07:37:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2eb000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd2e415e170 0x7fd2e40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2eb000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd2e40b4b60 0x7fd2e4161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2eb000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e40b3690 con 0x7fd2e406e570 2024-03-06T07:37:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2eb000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e406ea70 con 0x7fd2e415e170 2024-03-06T07:37:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2eb000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e40b3970 con 0x7fd2e40b4b60 2024-03-06T07:37:10.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2e3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd2e415e170 0x7fd2e40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2e3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd2e415e170 0x7fd2e40ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59630/0 (socket says 172.21.15.97:59630) 2024-03-06T07:37:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2e3e00640 1 -- 172.21.15.97:0/2003918618 learned_addr learned my addr 172.21.15.97:0/2003918618 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2e9000640 1 --2- 172.21.15.97:0/2003918618 >> v2:172.21.15.135:3300/0 conn(0x7fd2e40b4b60 0x7fd2e4161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2e3e00640 1 -- 172.21.15.97:0/2003918618 >> v2:172.21.15.135:3300/0 conn(0x7fd2e40b4b60 msgr2=0x7fd2e4161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.487+0000 7fd2e3e00640 1 --2- 172.21.15.97:0/2003918618 >> v2:172.21.15.135:3300/0 conn(0x7fd2e40b4b60 0x7fd2e4161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.488+0000 7fd2e3e00640 1 -- 172.21.15.97:0/2003918618 >> v1:172.21.15.135:6789/0 conn(0x7fd2e406e570 legacy=0x7fd2e40b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.488+0000 7fd2e3e00640 1 -- 172.21.15.97:0/2003918618 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e40b3c50 con 0x7fd2e415e170 2024-03-06T07:37:10.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.488+0000 7fd2e9000640 1 --2- 172.21.15.97:0/2003918618 >> v2:172.21.15.135:3300/0 conn(0x7fd2e40b4b60 0x7fd2e4161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:10.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.489+0000 7fd2e3e00640 1 --2- 172.21.15.97:0/2003918618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e415e170 0x7fd2e40ad860 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40d0aa0 tx=0x7fd2d402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6393213312866020 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.489+0000 7fd2e2a00640 1 -- 172.21.15.97:0/2003918618 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d40317d0 con 0x7fd2e415e170 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.489+0000 7fd2e2a00640 1 -- 172.21.15.97:0/2003918618 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4037070 con 0x7fd2e415e170 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.489+0000 7fd2e2a00640 1 -- 172.21.15.97:0/2003918618 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4031b90 con 0x7fd2e415e170 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.489+0000 7fd2eb000640 1 -- 172.21.15.97:0/2003918618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e415e170 msgr2=0x7fd2e40ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.489+0000 7fd2eb000640 1 --2- 172.21.15.97:0/2003918618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e415e170 0x7fd2e40ad860 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40d0aa0 tx=0x7fd2d402b870 comp rx=0 tx=0).stop 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.490+0000 7fd2eb000640 1 -- 172.21.15.97:0/2003918618 shutdown_connections 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.490+0000 7fd2eb000640 1 --2- 172.21.15.97:0/2003918618 >> v2:172.21.15.135:3300/0 conn(0x7fd2e40b4b60 0x7fd2e4161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.490+0000 7fd2eb000640 1 --2- 172.21.15.97:0/2003918618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e415e170 0x7fd2e40ad860 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.490+0000 7fd2eb000640 1 -- 172.21.15.97:0/2003918618 >> 172.21.15.97:0/2003918618 conn(0x7fd2e40cfa80 msgr2=0x7fd2e40cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.490+0000 7fd2eb000640 1 -- 172.21.15.97:0/2003918618 shutdown_connections 2024-03-06T07:37:10.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.490+0000 7fd2eb000640 1 -- 172.21.15.97:0/2003918618 wait complete. 2024-03-06T07:37:10.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.491+0000 7fd2eb000640 1 Processor -- start 2024-03-06T07:37:10.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 -- start start 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 0x7fd2e41e6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e40b4b60 0x7fd2e40af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2e415e170 0x7fd2e415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e40adda0 con 0x7fd2e406e570 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e406df80 con 0x7fd2e40b4b60 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.492+0000 7fd2eb000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e40adf20 con 0x7fd2e415e170 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 0x7fd2e41e6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 0x7fd2e41e6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59642/0 (socket says 172.21.15.97:59642) 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 -- 172.21.15.97:0/1588493115 learned_addr learned my addr 172.21.15.97:0/1588493115 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 -- 172.21.15.97:0/1588493115 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2e415e170 msgr2=0x7fd2e415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2e415e170 0x7fd2e415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 -- 172.21.15.97:0/1588493115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e40b4b60 msgr2=0x7fd2e40af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e40b4b60 0x7fd2e40af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 -- 172.21.15.97:0/1588493115 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e41446b0 con 0x7fd2e406e570 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.493+0000 7fd2e3e00640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 0x7fd2e41e6ad0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fd2d402b8b0 tx=0x7fd2d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.494+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4032870 con 0x7fd2e406e570 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.494+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4032a10 con 0x7fd2e406e570 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.494+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4032e00 con 0x7fd2e406e570 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.494+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e4144990 con 0x7fd2e406e570 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.494+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e4145230 con 0x7fd2e406e570 2024-03-06T07:37:10.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.495+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e404f330 con 0x7fd2e406e570 2024-03-06T07:37:10.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.496+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2d404c060 con 0x7fd2e406e570 2024-03-06T07:37:10.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.497+0000 7fd2dbe00640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2c406b4d0 0x7fd2c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.497+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fd2d40abda0 con 0x7fd2e406e570 2024-03-06T07:37:10.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.497+0000 7fd2e3400640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2c406b4d0 0x7fd2c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.498+0000 7fd2e3400640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2c406b4d0 0x7fd2c406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd2e415d890 tx=0x7fd2d0006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.501+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2d4040070 con 0x7fd2e406e570 2024-03-06T07:37:10.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:10.699+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2e40b2e30 con 0x7fd2e406e570 2024-03-06T07:37:11.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.261+0000 7fd2dbe00640 1 -- 172.21.15.97:0/1588493115 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v137) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd2d407aca0 con 0x7fd2e406e570 2024-03-06T07:37:11.263 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.265+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2c406b4d0 msgr2=0x7fd2c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.265+0000 7fd2eb000640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2c406b4d0 0x7fd2c406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd2e415d890 tx=0x7fd2d0006ba0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.265+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 msgr2=0x7fd2e41e6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.265+0000 7fd2eb000640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 0x7fd2e41e6ad0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fd2d402b8b0 tx=0x7fd2d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 shutdown_connections 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2c406b4d0 0x7fd2c406d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd2e415e170 0x7fd2e415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e40b4b60 0x7fd2e40af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 --2- 172.21.15.97:0/1588493115 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd2e406e570 0x7fd2e41e6ad0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 >> 172.21.15.97:0/1588493115 conn(0x7fd2e40cfa80 msgr2=0x7fd2e415a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.266+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 shutdown_connections 2024-03-06T07:37:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.268+0000 7fd2eb000640 1 -- 172.21.15.97:0/1588493115 wait complete. 2024-03-06T07:37:11.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:37:11.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.487+0000 7fc7c0e00640 1 Processor -- start 2024-03-06T07:37:11.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.487+0000 7fc7c0e00640 1 -- start start 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.488+0000 7fc7c0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 0x7fc7bc0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7c0e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7bc0aef20 0x7fc7bc0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7c0e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc15c1f0 con 0x7fc7bc15be10 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7c0e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cd200 con 0x7fc7bc0aef20 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7c0e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc06cac0 con 0x7fc7bc06da40 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7bb400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc7bc15be10 0x7fc7bc06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49214/0 (socket says 172.21.15.97:49214) 2024-03-06T07:37:11.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7bb400640 1 -- 172.21.15.97:0/867966669 learned_addr learned my addr 172.21.15.97:0/867966669 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/867966669 >> v2:172.21.15.135:3300/0 conn(0x7fc7bc0aef20 0x7fc7bc0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7b3e00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 0x7fc7bc0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335136830 0 0) 0x7fc7bc15c1f0 con 0x7fc7bc15be10 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.489+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0003520 con 0x7fc7bc15be10 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699228294 0 0) 0x7fc7a0003520 con 0x7fc7bc15be10 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 msgr2=0x7fc7bc0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 0x7fc7bc0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 >> v2:172.21.15.135:3300/0 conn(0x7fc7bc0aef20 msgr2=0x7fc7bc0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 --2- 172.21.15.97:0/867966669 >> v2:172.21.15.135:3300/0 conn(0x7fc7bc0aef20 0x7fc7bc0ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc06cda0 con 0x7fc7bc15be10 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc7ac001670 con 0x7fc7bc15be10 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 >> v1:172.21.15.97:6789/0 conn(0x7fc7bc15be10 legacy=0x7fc7bc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 0x7fc7a00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a0003f80 0x7fc7a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7a0004d40 0x7fc7a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cd200 con 0x7fc7a0004d40 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc06cac0 con 0x7fc7a0003f80 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a000ba70 con 0x7fc7bc06da40 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7a0004d40 0x7fc7a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.490+0000 7fc7bb400640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 0x7fc7a00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.491+0000 7fc7b3e00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a0003f80 0x7fc7a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.491+0000 7fc7bbe00640 1 -- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 msgr2=0x7fc7a00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.491+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7bc06da40 0x7fc7a00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.491+0000 7fc7bbe00640 1 -- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a0003f80 msgr2=0x7fc7a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.491+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a0003f80 0x7fc7a0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.491+0000 7fc7bbe00640 1 -- 172.21.15.97:0/867966669 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0003520 con 0x7fc7a0004d40 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7a0004d40 0x7fc7a00051c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4007360 tx=0x7fc7a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9a5c2af5ca5f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4039070 con 0x7fc7a0004d40 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b402ccf0 con 0x7fc7a0004d40 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7baa00640 1 -- 172.21.15.97:0/867966669 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4009be0 con 0x7fc7a0004d40 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7c0e00640 1 -- 172.21.15.97:0/867966669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7a0004d40 msgr2=0x7fc7a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/867966669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7a0004d40 0x7fc7a00051c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4007360 tx=0x7fc7a0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.492+0000 7fc7bbe00640 1 -- 172.21.15.97:0/867966669 reap_dead start 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.493+0000 7fc7c0e00640 1 -- 172.21.15.97:0/867966669 shutdown_connections 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.493+0000 7fc7c0e00640 1 -- 172.21.15.97:0/867966669 >> 172.21.15.97:0/867966669 conn(0x7fc7bc06afc0 msgr2=0x7fc7bc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.493+0000 7fc7c0e00640 1 -- 172.21.15.97:0/867966669 shutdown_connections 2024-03-06T07:37:11.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.493+0000 7fc7c0e00640 1 -- 172.21.15.97:0/867966669 wait complete. 2024-03-06T07:37:11.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.494+0000 7fc7c0e00640 1 Processor -- start 2024-03-06T07:37:11.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 -- start start 2024-03-06T07:37:11.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7a0004d40 0x7fc7bc14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7bc14eb10 0x7fc7bc0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 0x7fc7bc15e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cda90 con 0x7fc7bc0adda0 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc06d080 con 0x7fc7bc14eb10 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.495+0000 7fc7c0e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0ccc20 con 0x7fc7a0004d40 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 0x7fc7bc15e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 0x7fc7bc15e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59654/0 (socket says 172.21.15.97:59654) 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 -- 172.21.15.97:0/156946945 learned_addr learned my addr 172.21.15.97:0/156946945 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7b3e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7bc14eb10 0x7fc7bc0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bb400640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7a0004d40 0x7fc7bc14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 -- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7a0004d40 msgr2=0x7fc7bc14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7a0004d40 0x7fc7bc14c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 -- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7bc14eb10 msgr2=0x7fc7bc0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7bc14eb10 0x7fc7bc0ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 -- 172.21.15.97:0/156946945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc14ef90 con 0x7fc7bc0adda0 2024-03-06T07:37:11.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.496+0000 7fc7bbe00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 0x7fc7bc15e580 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4007450 tx=0x7fc7b4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.497+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4039070 con 0x7fc7bc0adda0 2024-03-06T07:37:11.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.497+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b403a040 con 0x7fc7bc0adda0 2024-03-06T07:37:11.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.497+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4009be0 con 0x7fc7bc0adda0 2024-03-06T07:37:11.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.497+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc0ce3e0 con 0x7fc7bc0adda0 2024-03-06T07:37:11.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.497+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc0ce9a0 con 0x7fc7bc0adda0 2024-03-06T07:37:11.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.498+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc04f330 con 0x7fc7bc0adda0 2024-03-06T07:37:11.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.499+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7b4046030 con 0x7fc7bc0adda0 2024-03-06T07:37:11.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.500+0000 7fc7b3400640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79806b4d0 0x7fc79806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.500+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc7b40ab6f0 con 0x7fc7bc0adda0 2024-03-06T07:37:11.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.500+0000 7fc7bb400640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79806b4d0 0x7fc79806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.501+0000 7fc7bb400640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79806b4d0 0x7fc79806d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac000c00 tx=0x7fc7ac000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.504+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7b404d060 con 0x7fc7bc0adda0 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.700+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc7bc071be0 con 0x7fc7bc0adda0 2024-03-06T07:37:11.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.706+0000 7fc7b3400640 1 -- 172.21.15.97:0/156946945 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc7b407a860 con 0x7fc7bc0adda0 2024-03-06T07:37:11.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79806b4d0 msgr2=0x7fc79806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79806b4d0 0x7fc79806d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac000c00 tx=0x7fc7ac000ed0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 msgr2=0x7fc7bc15e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 0x7fc7bc15e580 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4007450 tx=0x7fc7b4004ab0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 shutdown_connections 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc79806b4d0 0x7fc79806d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7bc0adda0 0x7fc7bc15e580 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7bc14eb10 0x7fc7bc0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 --2- 172.21.15.97:0/156946945 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7a0004d40 0x7fc7bc14c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 >> 172.21.15.97:0/156946945 conn(0x7fc7bc06afc0 msgr2=0x7fc7bc0af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 shutdown_connections 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.709+0000 7fc7c0e00640 1 -- 172.21.15.97:0/156946945 wait complete. 2024-03-06T07:37:11.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:37:11.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.935+0000 7f278d800640 1 Processor -- start 2024-03-06T07:37:11.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.935+0000 7f278d800640 1 -- start start 2024-03-06T07:37:11.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f278d800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f278815acb0 0x7f27880b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f278d800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f27880b4680 0x7f278806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f278d800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278815b090 con 0x7f2788072340 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f278d800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880b2a30 con 0x7f27880b4680 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f278d800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880b4360 con 0x7f278815acb0 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f277e000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f27880b4680 0x7f278806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f277e000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f27880b4680 0x7f278806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59668/0 (socket says 172.21.15.97:59668) 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f277e000640 1 -- 172.21.15.97:0/2823473648 learned_addr learned my addr 172.21.15.97:0/2823473648 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.937+0000 7f2786a00640 1 --2- 172.21.15.97:0/2823473648 >> v2:172.21.15.135:3300/0 conn(0x7f278815acb0 0x7f27880b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.938+0000 7f277e000640 1 -- 172.21.15.97:0/2823473648 >> v2:172.21.15.135:3300/0 conn(0x7f278815acb0 msgr2=0x7f27880b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.938+0000 7f277e000640 1 --2- 172.21.15.97:0/2823473648 >> v2:172.21.15.135:3300/0 conn(0x7f278815acb0 0x7f27880b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.938+0000 7f277e000640 1 -- 172.21.15.97:0/2823473648 >> v1:172.21.15.135:6789/0 conn(0x7f2788072340 legacy=0x7f27880ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.938+0000 7f277e000640 1 -- 172.21.15.97:0/2823473648 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880b2cf0 con 0x7f27880b4680 2024-03-06T07:37:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.939+0000 7f277e000640 1 --2- 172.21.15.97:0/2823473648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27880b4680 0x7f278806e4f0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f27880b27f0 tx=0x7f277002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f355128e5f1a146d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.939+0000 7f2786000640 1 -- 172.21.15.97:0/2823473648 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770035020 con 0x7f27880b4680 2024-03-06T07:37:11.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.939+0000 7f2786000640 1 -- 172.21.15.97:0/2823473648 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770031870 con 0x7f27880b4680 2024-03-06T07:37:11.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.939+0000 7f2786000640 1 -- 172.21.15.97:0/2823473648 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770031c40 con 0x7f27880b4680 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.939+0000 7f278d800640 1 -- 172.21.15.97:0/2823473648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27880b4680 msgr2=0x7f278806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.939+0000 7f278d800640 1 --2- 172.21.15.97:0/2823473648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27880b4680 0x7f278806e4f0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f27880b27f0 tx=0x7f277002b910 comp rx=0 tx=0).stop 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.940+0000 7f278d800640 1 -- 172.21.15.97:0/2823473648 shutdown_connections 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.940+0000 7f278d800640 1 --2- 172.21.15.97:0/2823473648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27880b4680 0x7f278806e4f0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.940+0000 7f278d800640 1 --2- 172.21.15.97:0/2823473648 >> v2:172.21.15.135:3300/0 conn(0x7f278815acb0 0x7f27880b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.940+0000 7f278d800640 1 -- 172.21.15.97:0/2823473648 >> 172.21.15.97:0/2823473648 conn(0x7f27880b1730 msgr2=0x7f27880b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.940+0000 7f278d800640 1 -- 172.21.15.97:0/2823473648 shutdown_connections 2024-03-06T07:37:11.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.940+0000 7f278d800640 1 -- 172.21.15.97:0/2823473648 wait complete. 2024-03-06T07:37:11.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.941+0000 7f278d800640 1 Processor -- start 2024-03-06T07:37:11.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.942+0000 7f278d800640 1 -- start start 2024-03-06T07:37:11.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.942+0000 7f278d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 0x7f27881f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f278d800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27880b4680 0x7f2788167680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f278d800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f278815acb0 0x7f2788148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f278d800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880b3d10 con 0x7f2788072340 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f278d800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880b32b0 con 0x7f27880b4680 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f278d800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27880b4010 con 0x7f278815acb0 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 0x7f27881f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 0x7f27881f84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59676/0 (socket says 172.21.15.97:59676) 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 -- 172.21.15.97:0/772043887 learned_addr learned my addr 172.21.15.97:0/772043887 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2787400640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f278815acb0 0x7f2788148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 -- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f278815acb0 msgr2=0x7f2788148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f278815acb0 0x7f2788148310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 -- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27880b4680 msgr2=0x7f2788167680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f277e000640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27880b4680 0x7f2788167680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27880b4680 0x7f2788167680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2786a00640 1 -- 172.21.15.97:0/772043887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788167e20 con 0x7f2788072340 2024-03-06T07:37:11.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.943+0000 7f2787400640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f278815acb0 0x7f2788148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:11.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.944+0000 7f2786a00640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 0x7f27881f84b0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f277800d830 tx=0x7f277800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.944+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800b6c0 con 0x7f2788072340 2024-03-06T07:37:11.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.944+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778019070 con 0x7f2788072340 2024-03-06T07:37:11.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.944+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27880cdf90 con 0x7f2788072340 2024-03-06T07:37:11.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.944+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800ba80 con 0x7f2788072340 2024-03-06T07:37:11.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.944+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27880ce550 con 0x7f2788072340 2024-03-06T07:37:11.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.945+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278804f330 con 0x7f2788072340 2024-03-06T07:37:11.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.946+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f277801e460 con 0x7f2788072340 2024-03-06T07:37:11.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.947+0000 7f277f400640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f276406b4d0 0x7f276406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.947+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f277808a520 con 0x7f2788072340 2024-03-06T07:37:11.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.947+0000 7f277e000640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f276406b4d0 0x7f276406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.948+0000 7f277e000640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f276406b4d0 0x7f276406d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2770036000 tx=0x7f2770008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:11.952+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2778059730 con 0x7f2788072340 2024-03-06T07:37:12.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.220+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f2788166960 con 0x7f2788072340 2024-03-06T07:37:12.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.264+0000 7f277f400640 1 -- 172.21.15.97:0/772043887 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v506) v1 ==== 140+0+0 (secure 0 0 0) 0x7f2778059030 con 0x7f2788072340 2024-03-06T07:37:12.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f276406b4d0 msgr2=0x7f276406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f276406b4d0 0x7f276406d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2770036000 tx=0x7f2770008030 comp rx=0 tx=0).stop 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 msgr2=0x7f27881f84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 0x7f27881f84b0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f277800d830 tx=0x7f277800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 shutdown_connections 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f276406b4d0 0x7f276406d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f278815acb0 0x7f2788148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27880b4680 0x7f2788167680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 --2- 172.21.15.97:0/772043887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2788072340 0x7f27881f84b0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 >> 172.21.15.97:0/772043887 conn(0x7f27880b1730 msgr2=0x7f278814dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 shutdown_connections 2024-03-06T07:37:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.267+0000 7f278d800640 1 -- 172.21.15.97:0/772043887 wait complete. 2024-03-06T07:37:12.287 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ok 2024-03-06T07:37:12.287 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:37:12.287 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:37:12.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:12.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.479+0000 7fd363400640 1 Processor -- start 2024-03-06T07:37:12.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.479+0000 7fd363400640 1 -- start start 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd363400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd35c15f9d0 0x7fd35c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd363400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c070b10 0x7fd35c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd363400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c06cef0 con 0x7fd35c144c00 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd363400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35c06d1d0 con 0x7fd35c15f9d0 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd363400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c06d4b0 con 0x7fd35c070b10 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd35be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd35c15f9d0 0x7fd35c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd360a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd35c144c00 0x7fd35c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49222/0 (socket says 172.21.15.97:49222) 2024-03-06T07:37:12.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd35be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd35c15f9d0 0x7fd35c15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59678/0 (socket says 172.21.15.97:59678) 2024-03-06T07:37:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd360a00640 1 -- 172.21.15.97:0/2955214334 learned_addr learned my addr 172.21.15.97:0/2955214334 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd35be00640 1 -- 172.21.15.97:0/2955214334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c070b10 msgr2=0x7fd35c070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd35be00640 1 --2- 172.21.15.97:0/2955214334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c070b10 0x7fd35c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd35be00640 1 -- 172.21.15.97:0/2955214334 >> v1:172.21.15.97:6789/0 conn(0x7fd35c144c00 legacy=0x7fd35c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.482+0000 7fd35be00640 1 -- 172.21.15.97:0/2955214334 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c06d790 con 0x7fd35c15f9d0 2024-03-06T07:37:12.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.481+0000 7fd35b400640 1 -- 172.21.15.97:0/2955214334 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258596201 0 0) 0x7fd35c06cef0 con 0x7fd35c144c00 2024-03-06T07:37:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd35be00640 1 --2- 172.21.15.97:0/2955214334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c15fe60 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fd3480097a0 tx=0x7fd34802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2050bf98a99ed02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd35b400640 1 -- 172.21.15.97:0/2955214334 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348035020 con 0x7fd35c15f9d0 2024-03-06T07:37:12.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd35b400640 1 -- 172.21.15.97:0/2955214334 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348031830 con 0x7fd35c15f9d0 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd35b400640 1 -- 172.21.15.97:0/2955214334 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348031c00 con 0x7fd35c15f9d0 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd363400640 1 -- 172.21.15.97:0/2955214334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 msgr2=0x7fd35c15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd363400640 1 --2- 172.21.15.97:0/2955214334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c15fe60 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fd3480097a0 tx=0x7fd34802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd363400640 1 -- 172.21.15.97:0/2955214334 shutdown_connections 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd363400640 1 --2- 172.21.15.97:0/2955214334 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c070b10 0x7fd35c070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd363400640 1 --2- 172.21.15.97:0/2955214334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c15fe60 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.483+0000 7fd363400640 1 -- 172.21.15.97:0/2955214334 >> 172.21.15.97:0/2955214334 conn(0x7fd35c0b1730 msgr2=0x7fd35c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.484+0000 7fd363400640 1 -- 172.21.15.97:0/2955214334 shutdown_connections 2024-03-06T07:37:12.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.484+0000 7fd363400640 1 -- 172.21.15.97:0/2955214334 wait complete. 2024-03-06T07:37:12.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.485+0000 7fd363400640 1 Processor -- start 2024-03-06T07:37:12.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.485+0000 7fd363400640 1 -- start start 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd363400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c070b10 0x7fd35c143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd363400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c144c00 0x7fd35c0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd363400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c1e7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd363400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c06ccd0 con 0x7fd35c15f9d0 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd363400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c06cb50 con 0x7fd35c070b10 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd363400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c06df90 con 0x7fd35c144c00 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd361400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c1e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd361400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c1e7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59682/0 (socket says 172.21.15.97:59682) 2024-03-06T07:37:12.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.486+0000 7fd361400640 1 -- 172.21.15.97:0/1424577744 learned_addr learned my addr 172.21.15.97:0/1424577744 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd361400640 1 -- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c144c00 msgr2=0x7fd35c0cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd35be00640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c144c00 0x7fd35c0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd360a00640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c070b10 0x7fd35c143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd361400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c144c00 0x7fd35c0cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd361400640 1 -- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c070b10 msgr2=0x7fd35c143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd361400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c070b10 0x7fd35c143500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd361400640 1 -- 172.21.15.97:0/1424577744 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c1e75b0 con 0x7fd35c15f9d0 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd361400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c1e7070 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fd35400b9a0 tx=0x7fd35400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd354019070 con 0x7fd35c15f9d0 2024-03-06T07:37:12.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35400cbc0 con 0x7fd35c15f9d0 2024-03-06T07:37:12.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd354014440 con 0x7fd35c15f9d0 2024-03-06T07:37:12.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c1e62b0 con 0x7fd35c15f9d0 2024-03-06T07:37:12.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.487+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1e6870 con 0x7fd35c15f9d0 2024-03-06T07:37:12.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.489+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c04f330 con 0x7fd35c15f9d0 2024-03-06T07:37:12.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.490+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd35401c070 con 0x7fd35c15f9d0 2024-03-06T07:37:12.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.491+0000 7fd358c00640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.491+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd354089db0 con 0x7fd35c15f9d0 2024-03-06T07:37:12.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.491+0000 7fd360a00640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.492+0000 7fd360a00640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd350007fd0 tx=0x7fd350009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.495+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd35408e070 con 0x7fd35c15f9d0 2024-03-06T07:37:12.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.689+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd35c072890 con 0x7fd35c15f9d0 2024-03-06T07:37:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.691+0000 7fd358c00640 1 -- 172.21.15.97:0/1424577744 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+13018 (secure 0 0 0) 0x7fd354058e00 con 0x7fd35c15f9d0 2024-03-06T07:37:12.693 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:12.693 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":137,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:11.250763+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6837/3564321877":"2024-03-07T07:37:08.066201+0000","172.21.15.97:6835/3564321877":"2024-03-07T07:37:08.066201+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:12.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 msgr2=0x7fd33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd350007fd0 tx=0x7fd350009040 comp rx=0 tx=0).stop 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 msgr2=0x7fd35c1e7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c1e7070 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fd35400b9a0 tx=0x7fd35400be60 comp rx=0 tx=0).stop 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 shutdown_connections 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd35c15f9d0 0x7fd35c1e7070 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd35c144c00 0x7fd35c0cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 --2- 172.21.15.97:0/1424577744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c070b10 0x7fd35c143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 >> 172.21.15.97:0/1424577744 conn(0x7fd35c0b1730 msgr2=0x7fd35c0ae200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 shutdown_connections 2024-03-06T07:37:12.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.693+0000 7fd363400640 1 -- 172.21.15.97:0/1424577744 wait complete. 2024-03-06T07:37:12.715 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:37:12.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:37:12.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.917+0000 7f8aad000640 1 Processor -- start 2024-03-06T07:37:12.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.917+0000 7f8aad000640 1 -- start start 2024-03-06T07:37:12.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.918+0000 7f8aad000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa80ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.918+0000 7f8aad000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8aa80aef20 0x7f8aa80ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.918+0000 7f8aad000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa815c1f0 con 0x7f8aa806d330 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.918+0000 7f8aad000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa80cd200 con 0x7f8aa80aef20 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.918+0000 7f8aad000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806cac0 con 0x7f8aa815be10 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8aa80aef20 0x7f8aa80ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8aa80aef20 0x7f8aa80ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59686/0 (socket says 172.21.15.97:59686) 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 -- 172.21.15.97:0/467702352 learned_addr learned my addr 172.21.15.97:0/467702352 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa7e00640 1 --1- 172.21.15.97:0/467702352 >> v1:172.21.15.97:6789/0 conn(0x7f8aa806d330 0x7f8aa8071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49230/0 (socket says 172.21.15.97:49230) 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa7400640 1 --2- 172.21.15.97:0/467702352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa80ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 -- 172.21.15.97:0/467702352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 msgr2=0x7f8aa80ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 --2- 172.21.15.97:0/467702352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa80ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 -- 172.21.15.97:0/467702352 >> v1:172.21.15.97:6789/0 conn(0x7f8aa806d330 legacy=0x7f8aa8071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.919+0000 7f8aa6a00640 1 -- 172.21.15.97:0/467702352 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa806cda0 con 0x7f8aa80aef20 2024-03-06T07:37:12.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.920+0000 7f8aa6a00640 1 --2- 172.21.15.97:0/467702352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa80aef20 0x7f8aa80ce2b0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009870 tx=0x7f8a9402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7379997a9c08ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aa6000640 1 -- 172.21.15.97:0/467702352 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94034070 con 0x7f8aa80aef20 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aa6000640 1 -- 172.21.15.97:0/467702352 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a94030840 con 0x7f8aa80aef20 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aa6000640 1 -- 172.21.15.97:0/467702352 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94030c10 con 0x7f8aa80aef20 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 -- 172.21.15.97:0/467702352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa80aef20 msgr2=0x7f8aa80ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 --2- 172.21.15.97:0/467702352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa80aef20 0x7f8aa80ce2b0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009870 tx=0x7f8a9402b980 comp rx=0 tx=0).stop 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 -- 172.21.15.97:0/467702352 shutdown_connections 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 --2- 172.21.15.97:0/467702352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa80aef20 0x7f8aa80ce2b0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 --2- 172.21.15.97:0/467702352 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa80ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 -- 172.21.15.97:0/467702352 >> 172.21.15.97:0/467702352 conn(0x7f8aa806afc0 msgr2=0x7f8aa806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 -- 172.21.15.97:0/467702352 shutdown_connections 2024-03-06T07:37:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.921+0000 7f8aad000640 1 -- 172.21.15.97:0/467702352 wait complete. 2024-03-06T07:37:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.923+0000 7f8aad000640 1 Processor -- start 2024-03-06T07:37:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.923+0000 7f8aad000640 1 -- start start 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aad000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 0x7f8aa8157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aad000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa80aef20 0x7f8aa8140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aad000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa814ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aad000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa80cda90 con 0x7f8aa806d330 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aad000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806d080 con 0x7f8aa80aef20 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aad000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806d7c0 con 0x7f8aa815be10 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.924+0000 7f8aa7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 0x7f8aa8157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 0x7f8aa8157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59692/0 (socket says 172.21.15.97:59692) 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 -- 172.21.15.97:0/2020093779 learned_addr learned my addr 172.21.15.97:0/2020093779 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7e00640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa814ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 -- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 msgr2=0x7f8aa814ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa6a00640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa80aef20 0x7f8aa8140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa814ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 -- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa80aef20 msgr2=0x7f8aa8140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa80aef20 0x7f8aa8140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 -- 172.21.15.97:0/2020093779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8150940 con 0x7f8aa806d330 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8aa7400640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 0x7f8aa8157990 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c820 tx=0x7f8a9800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.925+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98015070 con 0x7f8aa806d330 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.926+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a98004020 con 0x7f8aa806d330 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.926+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a980043f0 con 0x7f8aa806d330 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.926+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa8158030 con 0x7f8aa806d330 2024-03-06T07:37:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.926+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa81585f0 con 0x7f8aa806d330 2024-03-06T07:37:12.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.927+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa804f330 con 0x7f8aa806d330 2024-03-06T07:37:12.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.928+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a98007830 con 0x7f8aa806d330 2024-03-06T07:37:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.929+0000 7f8a9f400640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a8806b4d0 0x7f8a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.929+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8a98089c90 con 0x7f8aa806d330 2024-03-06T07:37:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.929+0000 7f8aa6a00640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a8806b4d0 0x7f8a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.930+0000 7f8aa6a00640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a8806b4d0 0x7f8a8806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8a94036000 tx=0x7f8a9403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:12.933+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a98058ea0 con 0x7f8aa806d330 2024-03-06T07:37:13.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.115+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8aa80af7a0 con 0x7f8aa806d330 2024-03-06T07:37:13.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.116+0000 7f8a9f400640 1 -- 172.21.15.97:0/2020093779 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v137) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a980587a0 con 0x7f8aa806d330 2024-03-06T07:37:13.118 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:37:13.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a8806b4d0 msgr2=0x7f8a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a8806b4d0 0x7f8a8806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8a94036000 tx=0x7f8a9403b040 comp rx=0 tx=0).stop 2024-03-06T07:37:13.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 msgr2=0x7f8aa8157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 0x7f8aa8157990 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c820 tx=0x7f8a9800cce0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 shutdown_connections 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8a8806b4d0 0x7f8a8806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8aa815be10 0x7f8aa814ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa80aef20 0x7f8aa8140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 --2- 172.21.15.97:0/2020093779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8aa806d330 0x7f8aa8157990 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.118+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 >> 172.21.15.97:0/2020093779 conn(0x7f8aa806afc0 msgr2=0x7f8aa8156310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.119+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 shutdown_connections 2024-03-06T07:37:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.119+0000 7f8aad000640 1 -- 172.21.15.97:0/2020093779 wait complete. 2024-03-06T07:37:13.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:37:13.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.338+0000 7f4df3400640 1 Processor -- start 2024-03-06T07:37:13.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.338+0000 7f4df3400640 1 -- start start 2024-03-06T07:37:13.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df3400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4dec15fe60 0x7f4dec071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df3400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dec06cef0 con 0x7f4dec144c00 2024-03-06T07:37:13.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df3400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dec06d1d0 con 0x7f4dec15f9d0 2024-03-06T07:37:13.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df3400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dec06d4b0 con 0x7f4dec15fe60 2024-03-06T07:37:13.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df0a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4dec144c00 0x7f4dec06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49244/0 (socket says 172.21.15.97:49244) 2024-03-06T07:37:13.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df0a00640 1 -- 172.21.15.97:0/2764284626 learned_addr learned my addr 172.21.15.97:0/2764284626 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:13.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4df1400640 1 --2- 172.21.15.97:0/2764284626 >> v2:172.21.15.135:3300/0 conn(0x7f4dec15fe60 0x7f4dec071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350900155 0 0) 0x7f4dec06cef0 con 0x7f4dec144c00 2024-03-06T07:37:13.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.340+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd4003520 con 0x7f4dec144c00 2024-03-06T07:37:13.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3924317907 0 0) 0x7f4dd4003520 con 0x7f4dec144c00 2024-03-06T07:37:13.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 >> v2:172.21.15.135:3300/0 conn(0x7f4dec15fe60 msgr2=0x7f4dec071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 --2- 172.21.15.97:0/2764284626 >> v2:172.21.15.135:3300/0 conn(0x7f4dec15fe60 0x7f4dec071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 >> v1:172.21.15.135:6789/0 conn(0x7f4dec15f9d0 legacy=0x7f4dec070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec06d790 con 0x7f4dec144c00 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f4de0001670 con 0x7f4dec144c00 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4df1400640 1 --2- 172.21.15.97:0/2764284626 >> v2:172.21.15.135:3300/0 conn(0x7f4dec15fe60 0x7f4dec071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 >> v1:172.21.15.97:6789/0 conn(0x7f4dec144c00 legacy=0x7f4dec06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dd40039b0 0x7f4dd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec06d4b0 con 0x7f4dd40039b0 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd400c110 con 0x7f4dd4004800 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd400c310 con 0x7f4dd40051c0 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4df0a00640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dd40039b0 0x7f4dd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4debe00640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.341+0000 7f4df0a00640 1 -- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dd40051c0 msgr2=0x7f4dd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4df1400640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4df0a00640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4df0a00640 1 -- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd4004800 msgr2=0x7f4dd4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4df0a00640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4df0a00640 1 -- 172.21.15.97:0/2764284626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd4003520 con 0x7f4dd40039b0 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4debe00640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:13.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.342+0000 7f4df1400640 1 -- 172.21.15.97:0/2764284626 reap_dead start 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4df0a00640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dd40039b0 0x7f4dd4003dd0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f4de000c210 tx=0x7f4dd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622ec4fa50528b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de0007d30 con 0x7f4dd40039b0 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de0030070 con 0x7f4dd40039b0 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4deb400640 1 -- 172.21.15.97:0/2764284626 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de0005d70 con 0x7f4dd40039b0 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4df3400640 1 -- 172.21.15.97:0/2764284626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dd40039b0 msgr2=0x7f4dd4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4df3400640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dd40039b0 0x7f4dd4003dd0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f4de000c210 tx=0x7f4dd4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4df3400640 1 -- 172.21.15.97:0/2764284626 shutdown_connections 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4df3400640 1 --2- 172.21.15.97:0/2764284626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dd40039b0 0x7f4dd4003dd0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.343+0000 7f4df3400640 1 -- 172.21.15.97:0/2764284626 >> 172.21.15.97:0/2764284626 conn(0x7f4dec0b1730 msgr2=0x7f4dec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:13.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.344+0000 7f4df3400640 1 -- 172.21.15.97:0/2764284626 shutdown_connections 2024-03-06T07:37:13.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.344+0000 7f4df3400640 1 -- 172.21.15.97:0/2764284626 wait complete. 2024-03-06T07:37:13.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.345+0000 7f4df3400640 1 Processor -- start 2024-03-06T07:37:13.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.345+0000 7f4df3400640 1 -- start start 2024-03-06T07:37:13.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4df3400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd40039b0 0x7f4dec1e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4df3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 0x7f4dec15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4df3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dec15ddc0 0x7f4dec15e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4df3400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec06cb50 con 0x7f4dec15b240 2024-03-06T07:37:13.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4df3400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec160730 con 0x7f4dd40039b0 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4df3400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec06ccd0 con 0x7f4dec15ddc0 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.346+0000 7f4debe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 0x7f4dec15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 0x7f4dec15b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59700/0 (socket says 172.21.15.97:59700) 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 -- 172.21.15.97:0/859515429 learned_addr learned my addr 172.21.15.97:0/859515429 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4df0a00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd40039b0 0x7f4dec1e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 -- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dec15ddc0 msgr2=0x7f4dec15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dec15ddc0 0x7f4dec15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 -- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd40039b0 msgr2=0x7f4dec1e86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd40039b0 0x7f4dec1e86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 -- 172.21.15.97:0/859515429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec15eb40 con 0x7f4dec15b240 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4debe00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 0x7f4dec15b6c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800d460 tx=0x7f4dd800d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8018070 con 0x7f4dec15b240 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd8003610 con 0x7f4dec15b240 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8003a20 con 0x7f4dec15b240 2024-03-06T07:37:13.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.347+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dec130150 con 0x7f4dec15b240 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.348+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dec15e780 con 0x7f4dec15b240 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.349+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dec12ff40 con 0x7f4dec15b240 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.350+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dd801b070 con 0x7f4dec15b240 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.350+0000 7f4de8c00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.351+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4dd8089590 con 0x7f4dec15b240 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.351+0000 7f4df0a00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.351+0000 7f4df0a00640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4de0034000 tx=0x7f4de0039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.355+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dd80580c0 con 0x7f4dec15b240 2024-03-06T07:37:13.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:13.601+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7f4dec04f330 con 0x7f4dec15b240 2024-03-06T07:37:14.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.277+0000 7f4de8c00640 1 -- 172.21.15.97:0/859515429 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v508) v1 ==== 144+0+0 (secure 0 0 0) 0x7f4dd805c160 con 0x7f4dec15b240 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.280+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4dcc06b4d0 msgr2=0x7f4dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.280+0000 7f4df3400640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4de0034000 tx=0x7f4de0039040 comp rx=0 tx=0).stop 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.280+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 msgr2=0x7f4dec15b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.280+0000 7f4df3400640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 0x7f4dec15b6c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800d460 tx=0x7f4dd800d920 comp rx=0 tx=0).stop 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 shutdown_connections 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dec15ddc0 0x7f4dec15e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dec15b240 0x7f4dec15b6c0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 --2- 172.21.15.97:0/859515429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dd40039b0 0x7f4dec1e86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 >> 172.21.15.97:0/859515429 conn(0x7f4dec0b1730 msgr2=0x7f4dec0ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 shutdown_connections 2024-03-06T07:37:14.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.281+0000 7f4df3400640 1 -- 172.21.15.97:0/859515429 wait complete. 2024-03-06T07:37:14.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:37:14.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.513+0000 7f73e7400640 1 Processor -- start 2024-03-06T07:37:14.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.514+0000 7f73e7400640 1 -- start start 2024-03-06T07:37:14.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.515+0000 7f73e7400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f73e015bfe0 0x7f73e00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.515+0000 7f73e7400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00cf8a0 con 0x7f73e00cec90 2024-03-06T07:37:14.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.515+0000 7f73e7400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00cfb40 con 0x7f73e00cd3c0 2024-03-06T07:37:14.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.515+0000 7f73e7400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00cfe20 con 0x7f73e015bfe0 2024-03-06T07:37:14.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.515+0000 7f73e5400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f73e00cec90 0x7f73e006edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48128/0 (socket says 172.21.15.97:48128) 2024-03-06T07:37:14.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.515+0000 7f73e5400640 1 -- 172.21.15.97:0/4160126646 learned_addr learned my addr 172.21.15.97:0/4160126646 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:14.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73e4a00640 1 --2- 172.21.15.97:0/4160126646 >> v2:172.21.15.135:3300/0 conn(0x7f73e015bfe0 0x7f73e00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730185900 0 0) 0x7f73e00cf8a0 con 0x7f73e00cec90 2024-03-06T07:37:14.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c4003520 con 0x7f73e00cec90 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463236443 0 0) 0x7f73c4003520 con 0x7f73e00cec90 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 >> v2:172.21.15.135:3300/0 conn(0x7f73e015bfe0 msgr2=0x7f73e00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 --2- 172.21.15.97:0/4160126646 >> v2:172.21.15.135:3300/0 conn(0x7f73e015bfe0 0x7f73e00d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 >> v1:172.21.15.135:6789/0 conn(0x7f73e00cd3c0 legacy=0x7f73e00ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00d0100 con 0x7f73e00cec90 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f73d80019f0 con 0x7f73e00cec90 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 >> v1:172.21.15.97:6789/0 conn(0x7f73e00cec90 legacy=0x7f73e006edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73c40039b0 0x7f73c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73c4004800 0x7f73c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c40051c0 0x7f73c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00cfe20 con 0x7f73c4004800 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c400c070 con 0x7f73c40051c0 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.516+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c400c270 con 0x7f73c40039b0 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73dfe00640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73c4004800 0x7f73c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73dfe00640 1 -- 172.21.15.97:0/4160126646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73c40039b0 msgr2=0x7f73c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73dfe00640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73c40039b0 0x7f73c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73dfe00640 1 -- 172.21.15.97:0/4160126646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c40051c0 msgr2=0x7f73c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73dfe00640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c40051c0 0x7f73c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73dfe00640 1 -- 172.21.15.97:0/4160126646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c4003520 con 0x7f73c4004800 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.517+0000 7f73e5400640 1 -- 172.21.15.97:0/4160126646 reap_dead start 2024-03-06T07:37:14.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.518+0000 7f73dfe00640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73c4004800 0x7f73c4004c80 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f73cc001750 tx=0x7f73c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cae35b36fd6bf75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.518+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc001370 con 0x7f73c4004800 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.518+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc0052d0 con 0x7f73c4004800 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.518+0000 7f73df400640 1 -- 172.21.15.97:0/4160126646 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc003830 con 0x7f73c4004800 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 -- 172.21.15.97:0/4160126646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73c4004800 msgr2=0x7f73c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73c4004800 0x7f73c4004c80 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f73cc001750 tx=0x7f73c4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 -- 172.21.15.97:0/4160126646 shutdown_connections 2024-03-06T07:37:14.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 --2- 172.21.15.97:0/4160126646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73c4004800 0x7f73c4004c80 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 -- 172.21.15.97:0/4160126646 >> 172.21.15.97:0/4160126646 conn(0x7f73e00b3800 msgr2=0x7f73e00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 -- 172.21.15.97:0/4160126646 shutdown_connections 2024-03-06T07:37:14.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.519+0000 7f73e7400640 1 -- 172.21.15.97:0/4160126646 wait complete. 2024-03-06T07:37:14.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.521+0000 7f73e7400640 1 Processor -- start 2024-03-06T07:37:14.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.521+0000 7f73e7400640 1 -- start start 2024-03-06T07:37:14.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73e7400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c4004800 0x7f73e01555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 0x7f73e00afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73e7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73e01f2620 0x7f73e01f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73e7400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e015c3c0 con 0x7f73e00af910 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73e7400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00b4540 con 0x7f73c4004800 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73e7400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e015c540 con 0x7f73e01f2620 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 0x7f73e00afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 0x7f73e00afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55974/0 (socket says 172.21.15.97:55974) 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 -- 172.21.15.97:0/3127073294 learned_addr learned my addr 172.21.15.97:0/3127073294 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 -- 172.21.15.97:0/3127073294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73e01f2620 msgr2=0x7f73e01f2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73e01f2620 0x7f73e01f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 -- 172.21.15.97:0/3127073294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c4004800 msgr2=0x7f73e01555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c4004800 0x7f73e01555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.522+0000 7f73dfe00640 1 -- 172.21.15.97:0/3127073294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e01f3380 con 0x7f73e00af910 2024-03-06T07:37:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.523+0000 7f73dfe00640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 0x7f73e00afd90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f73cc034040 tx=0x7f73cc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.523+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc041070 con 0x7f73e00af910 2024-03-06T07:37:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.523+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc005480 con 0x7f73e00af910 2024-03-06T07:37:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.523+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e01f3960 con 0x7f73e00af910 2024-03-06T07:37:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.523+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc007a30 con 0x7f73e00af910 2024-03-06T07:37:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.523+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e01f3f20 con 0x7f73e00af910 2024-03-06T07:37:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.524+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e00b02d0 con 0x7f73e00af910 2024-03-06T07:37:14.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.525+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73cc044070 con 0x7f73e00af910 2024-03-06T07:37:14.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.526+0000 7f73dcc00640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73c006b4d0 0x7f73c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.526+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f73cc0aba70 con 0x7f73e00af910 2024-03-06T07:37:14.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.526+0000 7f73e4a00640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73c006b4d0 0x7f73c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.527+0000 7f73e4a00640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73c006b4d0 0x7f73c006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006a90 tx=0x7f73d0006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.530+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73cc07ab50 con 0x7f73e00af910 2024-03-06T07:37:14.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.729+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f73e004f330 con 0x7f73e00af910 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.736+0000 7f73dcc00640 1 -- 172.21.15.97:0/3127073294 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f73cc07a450 con 0x7f73e00af910 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73c006b4d0 msgr2=0x7f73c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73c006b4d0 0x7f73c006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006a90 tx=0x7f73d0006a20 comp rx=0 tx=0).stop 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 msgr2=0x7f73e00afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 0x7f73e00afd90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f73cc034040 tx=0x7f73cc001120 comp rx=0 tx=0).stop 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 shutdown_connections 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73c006b4d0 0x7f73c006d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73e01f2620 0x7f73e01f2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73e00af910 0x7f73e00afd90 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 --2- 172.21.15.97:0/3127073294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73c4004800 0x7f73e01555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 >> 172.21.15.97:0/3127073294 conn(0x7f73e00b3800 msgr2=0x7f73e0070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 shutdown_connections 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:14.739+0000 7f73e7400640 1 -- 172.21.15.97:0/3127073294 wait complete. 2024-03-06T07:37:14.760 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:37:14.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:37:14.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:37:14.983 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:37:14.983 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:14.983 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:15.040 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:37:15.041 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:37:15.041 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:15.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:37:15.098 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:15.098 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:15.151 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:37:15.151 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:37:15.151 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:15.207 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:37:15.208 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:37:15.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.421+0000 7fc11b200640 1 Processor -- start 2024-03-06T07:37:15.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.421+0000 7fc11b200640 1 -- start start 2024-03-06T07:37:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11b200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc11c15e0b0 0x7fc11c0b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c06e780 con 0x7fc11c15dcd0 2024-03-06T07:37:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c06ee50 con 0x7fc11c0b40e0 2024-03-06T07:37:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11b200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c06d770 con 0x7fc11c15e0b0 2024-03-06T07:37:15.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc11c15e0b0 0x7fc11c0b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc11c15e0b0 0x7fc11c0b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55976/0 (socket says 172.21.15.97:55976) 2024-03-06T07:37:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11a800640 1 -- 172.21.15.97:0/3435055276 learned_addr learned my addr 172.21.15.97:0/3435055276 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:15.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc119e00640 1 --1- 172.21.15.97:0/3435055276 >> v1:172.21.15.97:6789/0 conn(0x7fc11c15dcd0 0x7fc11c0b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48134/0 (socket says 172.21.15.97:48134) 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11a800640 1 -- 172.21.15.97:0/3435055276 >> v1:172.21.15.135:6789/0 conn(0x7fc11c0b40e0 legacy=0x7fc11c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc11a800640 1 -- 172.21.15.97:0/3435055276 >> v1:172.21.15.97:6789/0 conn(0x7fc11c15dcd0 legacy=0x7fc11c0b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.424+0000 7fc11a800640 1 -- 172.21.15.97:0/3435055276 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c06d9c0 con 0x7fc11c15e0b0 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.423+0000 7fc118a00640 1 -- 172.21.15.97:0/3435055276 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461841683 0 0) 0x7fc11c06e780 con 0x7fc11c15dcd0 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.425+0000 7fc11a800640 1 --2- 172.21.15.97:0/3435055276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c15e0b0 0x7fc11c0b5860 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc114009e50 tx=0x7fc11402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1352b012e600f3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.425+0000 7fc118a00640 1 -- 172.21.15.97:0/3435055276 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11402ca80 con 0x7fc11c15e0b0 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.425+0000 7fc118a00640 1 -- 172.21.15.97:0/3435055276 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11402cc20 con 0x7fc11c15e0b0 2024-03-06T07:37:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.425+0000 7fc118a00640 1 -- 172.21.15.97:0/3435055276 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1140343f0 con 0x7fc11c15e0b0 2024-03-06T07:37:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.425+0000 7fc11b200640 1 -- 172.21.15.97:0/3435055276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c15e0b0 msgr2=0x7fc11c0b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.425+0000 7fc11b200640 1 --2- 172.21.15.97:0/3435055276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c15e0b0 0x7fc11c0b5860 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc114009e50 tx=0x7fc11402bee0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.426+0000 7fc11b200640 1 -- 172.21.15.97:0/3435055276 shutdown_connections 2024-03-06T07:37:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.426+0000 7fc11b200640 1 --2- 172.21.15.97:0/3435055276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c15e0b0 0x7fc11c0b5860 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.426+0000 7fc11b200640 1 -- 172.21.15.97:0/3435055276 >> 172.21.15.97:0/3435055276 conn(0x7fc11c06c0d0 msgr2=0x7fc11c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.426+0000 7fc11b200640 1 -- 172.21.15.97:0/3435055276 shutdown_connections 2024-03-06T07:37:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.426+0000 7fc11b200640 1 -- 172.21.15.97:0/3435055276 wait complete. 2024-03-06T07:37:15.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.427+0000 7fc11b200640 1 Processor -- start 2024-03-06T07:37:15.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.427+0000 7fc11b200640 1 -- start start 2024-03-06T07:37:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.428+0000 7fc11b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 0x7fc11c1f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.428+0000 7fc11b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc11c15dcd0 0x7fc11c158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.428+0000 7fc11b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc11c15e0b0 0x7fc11c153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.428+0000 7fc11b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c06e260 con 0x7fc11c0b40e0 2024-03-06T07:37:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.428+0000 7fc11b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c06df80 con 0x7fc11c15e0b0 2024-03-06T07:37:15.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.428+0000 7fc11b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c06f290 con 0x7fc11c15dcd0 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 0x7fc11c1f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 0x7fc11c1f35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55992/0 (socket says 172.21.15.97:55992) 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 -- 172.21.15.97:0/94334215 learned_addr learned my addr 172.21.15.97:0/94334215 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 -- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc11c15dcd0 msgr2=0x7fc11c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc11a800640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc11c15e0b0 0x7fc11c153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119400640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc11c15dcd0 0x7fc11c158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc11c15dcd0 0x7fc11c158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 -- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc11c15e0b0 msgr2=0x7fc11c153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc11c15e0b0 0x7fc11c153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 -- 172.21.15.97:0/94334215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c153eb0 con 0x7fc11c0b40e0 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc11a800640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc11c15e0b0 0x7fc11c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.429+0000 7fc119e00640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 0x7fc11c1f35a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00c4d0 tx=0x7fc10c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:15.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.430+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c018070 con 0x7fc11c0b40e0 2024-03-06T07:37:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.430+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c007500 con 0x7fc11c0b40e0 2024-03-06T07:37:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.430+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c004a20 con 0x7fc11c0b40e0 2024-03-06T07:37:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.430+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c1593a0 con 0x7fc11c0b40e0 2024-03-06T07:37:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.430+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c159960 con 0x7fc11c0b40e0 2024-03-06T07:37:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.431+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c04f330 con 0x7fc11c0b40e0 2024-03-06T07:37:15.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.432+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc10c004bc0 con 0x7fc11c0b40e0 2024-03-06T07:37:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.433+0000 7fc112000640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0fc06b500 0x7fc0fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.433+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc10c089140 con 0x7fc11c0b40e0 2024-03-06T07:37:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.433+0000 7fc119400640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0fc06b500 0x7fc0fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.434+0000 7fc119400640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0fc06b500 0x7fc0fc06d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0b1f60 tx=0x7fc108009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.437+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc10c0582b0 con 0x7fc11c0b40e0 2024-03-06T07:37:15.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.669+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc11c0cfb00 con 0x7fc11c0b40e0 2024-03-06T07:37:15.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.669+0000 7fc112000640 1 -- 172.21.15.97:0/94334215 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2119 (secure 0 0 0) 0x7fc10c057bb0 con 0x7fc11c0b40e0 2024-03-06T07:37:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:15.672 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":101,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6836/2007524244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2007524244},{"type":"v1","addr":"172.21.15.97:6836","nonce":2007524244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1901939669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1901939669},{"type":"v1","addr":"172.21.15.97:6837","nonce":1901939669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[]} 2024-03-06T07:37:15.672 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 101 2024-03-06T07:37:15.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0fc06b500 msgr2=0x7fc0fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0fc06b500 0x7fc0fc06d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0b1f60 tx=0x7fc108009040 comp rx=0 tx=0).stop 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 msgr2=0x7fc11c1f35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 0x7fc11c1f35a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00c4d0 tx=0x7fc10c00c990 comp rx=0 tx=0).stop 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 shutdown_connections 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc0fc06b500 0x7fc0fc06d9c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc11c15e0b0 0x7fc11c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc11c15dcd0 0x7fc11c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 --2- 172.21.15.97:0/94334215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc11c0b40e0 0x7fc11c1f35a0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 >> 172.21.15.97:0/94334215 conn(0x7fc11c06c0d0 msgr2=0x7fc11c15b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 shutdown_connections 2024-03-06T07:37:15.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.672+0000 7fc11b200640 1 -- 172.21.15.97:0/94334215 wait complete. 2024-03-06T07:37:15.692 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:37:15.692 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:37:15.692 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:37:15.692 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:37:15.692 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:37:15.692 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:37:15.693 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:37:15.693 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:37:15.693 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:37:15.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:37:15.694 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:37:15.694 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:37:15.701 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:37:15.701 DEBUG:teuthology.parallel:result is None 2024-03-06T07:37:15.707 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:37:15.707 DEBUG:teuthology.parallel:result is None 2024-03-06T07:37:15.710 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:37:15.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:37:15.714 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:37:15.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:37:15.761 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:37:15.761 DEBUG:teuthology.parallel:result is None 2024-03-06T07:37:15.767 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:37:15.767 DEBUG:teuthology.parallel:result is None 2024-03-06T07:37:15.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:37:15.868 INFO:tasks.ceph.mds.d.smithi135.stdout:starting mds.d at 2024-03-06T07:37:15.868 INFO:tasks.ceph.mds.b.smithi135.stdout:starting mds.b at 2024-03-06T07:37:15.910 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:37:15.914 INFO:tasks.ceph.mds.a.smithi097.stdout:starting mds.a at 2024-03-06T07:37:15.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.938+0000 7fd4ca400640 1 Processor -- start 2024-03-06T07:37:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.939+0000 7fd4ca400640 1 -- start start 2024-03-06T07:37:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4ca400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c40cd5b0 0x7fd4c4145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4ca400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd4c40cd990 0x7fd4c415c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4ca400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c4166ec0 con 0x7fd4c415caa0 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4ca400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c406cbe0 con 0x7fd4c40cd990 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4ca400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c406cec0 con 0x7fd4c40cd5b0 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4c3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd4c415caa0 0x7fd4c4162fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48138/0 (socket says 172.21.15.97:48138) 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4c3e00640 1 -- 172.21.15.97:0/1059716719 learned_addr learned my addr 172.21.15.97:0/1059716719 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4c3400640 1 --2- 172.21.15.97:0/1059716719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c40cd5b0 0x7fd4c4145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.940+0000 7fd4c2a00640 1 --2- 172.21.15.97:0/1059716719 >> v2:172.21.15.135:3300/0 conn(0x7fd4c40cd990 0x7fd4c415c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c2000640 1 -- 172.21.15.97:0/1059716719 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074953574 0 0) 0x7fd4c4166ec0 con 0x7fd4c415caa0 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c2000640 1 -- 172.21.15.97:0/1059716719 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a8003520 con 0x7fd4c415caa0 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c3400640 1 -- 172.21.15.97:0/1059716719 >> v2:172.21.15.135:3300/0 conn(0x7fd4c40cd990 msgr2=0x7fd4c415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c3400640 1 --2- 172.21.15.97:0/1059716719 >> v2:172.21.15.135:3300/0 conn(0x7fd4c40cd990 0x7fd4c415c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c3400640 1 -- 172.21.15.97:0/1059716719 >> v1:172.21.15.97:6789/0 conn(0x7fd4c415caa0 legacy=0x7fd4c4162fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c3400640 1 -- 172.21.15.97:0/1059716719 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c406d1a0 con 0x7fd4c40cd5b0 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.941+0000 7fd4c2a00640 1 --2- 172.21.15.97:0/1059716719 >> v2:172.21.15.135:3300/0 conn(0x7fd4c40cd990 0x7fd4c415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.942+0000 7fd4c3400640 1 --2- 172.21.15.97:0/1059716719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c40cd5b0 0x7fd4c4145b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40097a0 tx=0x7fd4b402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e9de54905f9f957 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.942+0000 7fd4c2000640 1 -- 172.21.15.97:0/1059716719 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4035020 con 0x7fd4c40cd5b0 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4c2000640 1 -- 172.21.15.97:0/1059716719 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b4031810 con 0x7fd4c40cd5b0 2024-03-06T07:37:15.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4c2000640 1 -- 172.21.15.97:0/1059716719 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4031bc0 con 0x7fd4c40cd5b0 2024-03-06T07:37:15.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 -- 172.21.15.97:0/1059716719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c40cd5b0 msgr2=0x7fd4c4145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1059716719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c40cd5b0 0x7fd4c4145b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40097a0 tx=0x7fd4b402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 -- 172.21.15.97:0/1059716719 shutdown_connections 2024-03-06T07:37:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1059716719 >> v2:172.21.15.135:3300/0 conn(0x7fd4c40cd990 0x7fd4c415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1059716719 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c40cd5b0 0x7fd4c4145b70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 -- 172.21.15.97:0/1059716719 >> 172.21.15.97:0/1059716719 conn(0x7fd4c40cc4f0 msgr2=0x7fd4c40cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 -- 172.21.15.97:0/1059716719 shutdown_connections 2024-03-06T07:37:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.943+0000 7fd4ca400640 1 -- 172.21.15.97:0/1059716719 wait complete. 2024-03-06T07:37:15.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.945+0000 7fd4ca400640 1 Processor -- start 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.945+0000 7fd4ca400640 1 -- start start 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4ca400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4c40cd5b0 0x7fd4c41482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4ca400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 0x7fd4c40b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4ca400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c415caa0 0x7fd4c41427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4ca400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40cde10 con 0x7fd4c40cd990 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4ca400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c415cef0 con 0x7fd4c40cd5b0 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4ca400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c41636c0 con 0x7fd4c415caa0 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4c2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 0x7fd4c40b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4c2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 0x7fd4c40b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56046/0 (socket says 172.21.15.97:56046) 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4c2a00640 1 -- 172.21.15.97:0/1205551492 learned_addr learned my addr 172.21.15.97:0/1205551492 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:15.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4c3400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4c40cd5b0 0x7fd4c41482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.946+0000 7fd4c3e00640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c415caa0 0x7fd4c41427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c2a00640 1 -- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c415caa0 msgr2=0x7fd4c41427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c2a00640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c415caa0 0x7fd4c41427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c2a00640 1 -- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4c40cd5b0 msgr2=0x7fd4c41482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c2a00640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4c40cd5b0 0x7fd4c41482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c2a00640 1 -- 172.21.15.97:0/1205551492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c4148810 con 0x7fd4c40cd990 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c3e00640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c415caa0 0x7fd4c41427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4c2a00640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 0x7fd4c40b1130 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000ebe0 tx=0x7fd4b000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0017070 con 0x7fd4c40cd990 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b000cc30 con 0x7fd4c40cd990 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0013470 con 0x7fd4c40cd990 2024-03-06T07:37:15.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c40b31d0 con 0x7fd4c40cd990 2024-03-06T07:37:15.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.947+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c4146f30 con 0x7fd4c40cd990 2024-03-06T07:37:15.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.948+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c404f330 con 0x7fd4c40cd990 2024-03-06T07:37:15.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.949+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4b001a070 con 0x7fd4c40cd990 2024-03-06T07:37:15.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.950+0000 7fd4bb400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:15.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.950+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd4b0089de0 con 0x7fd4c40cd990 2024-03-06T07:37:15.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.951+0000 7fd4c3400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:15.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.951+0000 7fd4c3400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd4a406b4d0 0x7fd4a406d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4036000 tx=0x7fd4b403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:15.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:15.955+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4b0058900 con 0x7fd4c40cd990 2024-03-06T07:37:16.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:16.146+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd4c406e8d0 con 0x7fd4c40cd990 2024-03-06T07:37:17.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.299+0000 7fd4bb400640 1 -- 172.21.15.97:0/1205551492 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd4b005c9a0 con 0x7fd4c40cd990 2024-03-06T07:37:17.302 INFO:teuthology.orchestra.run.smithi097.stderr: removed all blocklist entries 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.303+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd4a406b4d0 msgr2=0x7fd4a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.303+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd4a406b4d0 0x7fd4a406d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4036000 tx=0x7fd4b403c040 comp rx=0 tx=0).stop 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.303+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 msgr2=0x7fd4c40b1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.303+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 0x7fd4c40b1130 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000ebe0 tx=0x7fd4b000c520 comp rx=0 tx=0).stop 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 shutdown_connections 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4c415caa0 0x7fd4c41427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c40cd990 0x7fd4c40b1130 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 --2- 172.21.15.97:0/1205551492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4c40cd5b0 0x7fd4c41482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 >> 172.21.15.97:0/1205551492 conn(0x7fd4c40cc4f0 msgr2=0x7fd4c4162b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:17.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 shutdown_connections 2024-03-06T07:37:17.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.304+0000 7fd4ca400640 1 -- 172.21.15.97:0/1205551492 wait complete. 2024-03-06T07:37:17.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:37:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.522+0000 7f0442000640 1 Processor -- start 2024-03-06T07:37:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.523+0000 7f0442000640 1 -- start start 2024-03-06T07:37:17.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.524+0000 7f0442000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f043c0af1c0 0x7f043c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.524+0000 7f0442000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c070d20 0x7f043c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.524+0000 7f0442000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c071d30 con 0x7f043c06d1a0 2024-03-06T07:37:17.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.524+0000 7f0442000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c071490 con 0x7f043c0af1c0 2024-03-06T07:37:17.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.524+0000 7f0442000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c071770 con 0x7f043c070d20 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.525+0000 7f043b400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f043c0af1c0 0x7f043c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.525+0000 7f0432a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c070d20 0x7f043c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.525+0000 7f0432a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c070d20 0x7f043c06cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:46122/0 (socket says 172.21.15.97:46122) 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.525+0000 7f0432a00640 1 -- 172.21.15.97:0/2137029654 learned_addr learned my addr 172.21.15.97:0/2137029654 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.526+0000 7f043b400640 1 -- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c070d20 msgr2=0x7f043c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.526+0000 7f043b400640 1 --2- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c070d20 0x7f043c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.526+0000 7f043b400640 1 -- 172.21.15.97:0/2137029654 >> v1:172.21.15.135:6789/0 conn(0x7f043c06d1a0 legacy=0x7f043c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.526+0000 7f043b400640 1 -- 172.21.15.97:0/2137029654 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c071a50 con 0x7f043c0af1c0 2024-03-06T07:37:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.527+0000 7f043b400640 1 --2- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0af1c0 0x7f043c15eff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f042c009d90 tx=0x7f042c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=348a725208884b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.527+0000 7f043aa00640 1 -- 172.21.15.97:0/2137029654 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c039070 con 0x7f043c0af1c0 2024-03-06T07:37:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.527+0000 7f043aa00640 1 -- 172.21.15.97:0/2137029654 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c02cc60 con 0x7f043c0af1c0 2024-03-06T07:37:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.527+0000 7f043aa00640 1 -- 172.21.15.97:0/2137029654 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c034660 con 0x7f043c0af1c0 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 -- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0af1c0 msgr2=0x7f043c15eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 --2- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0af1c0 0x7f043c15eff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f042c009d90 tx=0x7f042c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 -- 172.21.15.97:0/2137029654 shutdown_connections 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 --2- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c070d20 0x7f043c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 --2- 172.21.15.97:0/2137029654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c0af1c0 0x7f043c15eff0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 -- 172.21.15.97:0/2137029654 >> 172.21.15.97:0/2137029654 conn(0x7f043c0b1730 msgr2=0x7f043c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 -- 172.21.15.97:0/2137029654 shutdown_connections 2024-03-06T07:37:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.528+0000 7f0442000640 1 -- 172.21.15.97:0/2137029654 wait complete. 2024-03-06T07:37:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.530+0000 7f0442000640 1 Processor -- start 2024-03-06T07:37:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.530+0000 7f0442000640 1 -- start start 2024-03-06T07:37:17.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.531+0000 7f0442000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c06d1a0 0x7f043c14b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.531+0000 7f0442000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 0x7f043c1e6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0442000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0af1c0 0x7f043c1e7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0442000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c06ebe0 con 0x7f043c070d20 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0442000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0711a0 con 0x7f043c06d1a0 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0442000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c06ed60 con 0x7f043c0af1c0 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 0x7f043c1e6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 0x7f043c1e6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56048/0 (socket says 172.21.15.97:56048) 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 -- 172.21.15.97:0/1969093365 learned_addr learned my addr 172.21.15.97:0/1969093365 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f043b400640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c06d1a0 0x7f043c14b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 -- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0af1c0 msgr2=0x7f043c1e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f043be00640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0af1c0 0x7f043c1e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0af1c0 0x7f043c1e7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 -- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c06d1a0 msgr2=0x7f043c14b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c06d1a0 0x7f043c14b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f0432a00640 1 -- 172.21.15.97:0/1969093365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0cd120 con 0x7f043c070d20 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.532+0000 7f043b400640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c06d1a0 0x7f043c14b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.533+0000 7f0432a00640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 0x7f043c1e6fd0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f042400e8b0 tx=0x7f042400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.533+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424017070 con 0x7f043c070d20 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.533+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042400c940 con 0x7f043c070d20 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.533+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042400cd10 con 0x7f043c070d20 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.533+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c0cd400 con 0x7f043c070d20 2024-03-06T07:37:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.533+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c0cd9c0 con 0x7f043c070d20 2024-03-06T07:37:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.534+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c04f330 con 0x7f043c070d20 2024-03-06T07:37:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.535+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f042401a070 con 0x7f043c070d20 2024-03-06T07:37:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.536+0000 7f0433e00640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f041806b4d0 0x7f041806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.536+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f042408a1e0 con 0x7f043c070d20 2024-03-06T07:37:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.537+0000 7f043b400640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f041806b4d0 0x7f041806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.537+0000 7f043b400640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f041806b4d0 0x7f041806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f042c036000 tx=0x7f042c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.543+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04240593a0 con 0x7f043c070d20 2024-03-06T07:37:17.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.806+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f043c074be0 con 0x7f043c070d20 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.807+0000 7f0433e00640 1 -- 172.21.15.97:0/1969093365 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0424058ca0 con 0x7f043c070d20 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_dump": [ 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.a", 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlSUMSIBAAq91jLYGT+QNqjNqZ87D3eA==", 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.b", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlvnjxBRAAZcQqxCELZ5/sPhzGz95EFg==", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.c", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlxhlJMRAA2aldiNTWOSU42Ipc5LcxFQ==", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mds.d", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhl6m1vFhAAxNNPnsoG9ofZ1Sb1zhyQ1g==", 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.812 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.0", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC2GuhlQudWChAAZX8A0FZPyMKq9xhq4swlsQ==", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.1", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC5GuhlwMR1IxAACRY1tzxcXrc4E7bKsBAHJw==", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.2", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC8GuhlxgigLBAAUyi9F1kXn/AyprRQCdrLcA==", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.814 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.3", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQC/GuhlwwhuNxAAQmMbXQbMUv8R0T49vnEMCg==", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.4", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDGGuhlbgVrJRAA+bUk+meJTnCv9Trlm/NELQ==", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.815 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.5", 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDKGuhlSvVCAxAAaIo45jOR26qF3XE5XDP9gw==", 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.6", 2024-03-06T07:37:17.816 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDNGuhlKHUkDRAANyB3t6fnD6n47HRSyJY2tQ==", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "osd.7", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQDQGuhlC3TWJBAAyT2h5dS4EP7Eojhxtq+Btg==", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow profile osd", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile osd", 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.0", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ==", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow r", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.1", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlaEEdORAAROwjGDgTDnvI78zngLwZGA==", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:37:17.818 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.2", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCxGuhlBiDkLhAAmGNNjI3FDo03iwwf7oFAtA==", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.3", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCyGuhlabj+BxAAauCjU11J5d90Q9k3RHS/ag==", 2024-03-06T07:37:17.819 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow r", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow rw", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow rwx" 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "client.admin", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCvGuhlfQ2dExAAad+I0Q9vf/cvs2cTvr6cbA==", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": "allow *", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow *", 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.x", 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhlggODFBAAXAU7t/BjcF9Wjv3SrYN+ag==", 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "entity": "mgr.y", 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "AQCwGuhllhquCBAAY6cU5412NE7uWr8L/AtDXQ==", 2024-03-06T07:37:17.821 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": { 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": "allow *", 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": "allow profile mgr", 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": "allow *" 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-06T07:37:17.822 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f041806b4d0 msgr2=0x7f041806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f041806b4d0 0x7f041806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f042c036000 tx=0x7f042c030000 comp rx=0 tx=0).stop 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 msgr2=0x7f043c1e6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 0x7f043c1e6fd0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f042400e8b0 tx=0x7f042400ed70 comp rx=0 tx=0).stop 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 shutdown_connections 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f041806b4d0 0x7f041806d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f043c0af1c0 0x7f043c1e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f043c070d20 0x7f043c1e6fd0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 --2- 172.21.15.97:0/1969093365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f043c06d1a0 0x7f043c14b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 >> 172.21.15.97:0/1969093365 conn(0x7f043c0b1730 msgr2=0x7f043c15c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 shutdown_connections 2024-03-06T07:37:17.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:17.810+0000 7f0442000640 1 -- 172.21.15.97:0/1969093365 wait complete. 2024-03-06T07:37:17.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:18.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.025+0000 7f0043600640 1 Processor -- start 2024-03-06T07:37:18.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.025+0000 7f0043600640 1 -- start start 2024-03-06T07:37:18.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0043600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 0x7f003c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0043600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f003c070b10 0x7f003c070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0043600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c06cef0 con 0x7f003c144c00 2024-03-06T07:37:18.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0043600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c160370 con 0x7f003c070b10 2024-03-06T07:37:18.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0043600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c06d1d0 con 0x7f003c15f9d0 2024-03-06T07:37:18.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0040c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f003c144c00 0x7f003c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48140/0 (socket says 172.21.15.97:48140) 2024-03-06T07:37:18.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.027+0000 7f0040c00640 1 -- 172.21.15.97:0/1654308265 learned_addr learned my addr 172.21.15.97:0/1654308265 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:18.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932819187 0 0) 0x7f003c06cef0 con 0x7f003c144c00 2024-03-06T07:37:18.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020003520 con 0x7f003c144c00 2024-03-06T07:37:18.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f0033e00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 0x7f003c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f0041600640 1 --2- 172.21.15.97:0/1654308265 >> v2:172.21.15.135:3300/0 conn(0x7f003c070b10 0x7f003c070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781180655 0 0) 0x7f0020003520 con 0x7f003c144c00 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 msgr2=0x7f003c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 0x7f003c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 >> v2:172.21.15.135:3300/0 conn(0x7f003c070b10 msgr2=0x7f003c070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 --2- 172.21.15.97:0/1654308265 >> v2:172.21.15.135:3300/0 conn(0x7f003c070b10 0x7f003c070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c06d4b0 con 0x7f003c144c00 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f002c001670 con 0x7f003c144c00 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 >> v1:172.21.15.97:6789/0 conn(0x7f003c144c00 legacy=0x7f003c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00200039b0 0x7f0020003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f0020004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 0x7f00200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c06d1d0 con 0x7f00200039b0 2024-03-06T07:37:18.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002000ba70 con 0x7f0020004800 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002000bc70 con 0x7f003c15f9d0 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.028+0000 7f0040c00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00200039b0 0x7f0020003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0033e00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f0020004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0041600640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 0x7f00200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0040c00640 1 -- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 msgr2=0x7f00200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0040c00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f003c15f9d0 0x7f00200051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0040c00640 1 -- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 msgr2=0x7f0020004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0040c00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f0020004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.029+0000 7f0040c00640 1 -- 172.21.15.97:0/1654308265 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020003520 con 0x7f00200039b0 2024-03-06T07:37:18.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.030+0000 7f0040c00640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00200039b0 0x7f0020003dd0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c160 tx=0x7f0020005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f844101297fa573 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.030+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c007c80 con 0x7f00200039b0 2024-03-06T07:37:18.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.030+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c030070 con 0x7f00200039b0 2024-03-06T07:37:18.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.030+0000 7f003be00640 1 -- 172.21.15.97:0/1654308265 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c005cf0 con 0x7f00200039b0 2024-03-06T07:37:18.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.031+0000 7f0043600640 1 -- 172.21.15.97:0/1654308265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00200039b0 msgr2=0x7f0020003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.031+0000 7f0043600640 1 --2- 172.21.15.97:0/1654308265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00200039b0 0x7f0020003dd0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c160 tx=0x7f0020005830 comp rx=0 tx=0).stop 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.031+0000 7f0041600640 1 -- 172.21.15.97:0/1654308265 reap_dead start 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.031+0000 7f0043600640 1 -- 172.21.15.97:0/1654308265 shutdown_connections 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.031+0000 7f0043600640 1 -- 172.21.15.97:0/1654308265 >> 172.21.15.97:0/1654308265 conn(0x7f003c0b1730 msgr2=0x7f003c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.032+0000 7f0043600640 1 -- 172.21.15.97:0/1654308265 shutdown_connections 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.032+0000 7f0043600640 1 -- 172.21.15.97:0/1654308265 wait complete. 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.033+0000 7f0043600640 1 Processor -- start 2024-03-06T07:37:18.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.034+0000 7f0043600640 1 -- start start 2024-03-06T07:37:18.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.034+0000 7f0043600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f003c0aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.034+0000 7f0043600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 0x7f003c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.034+0000 7f0043600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f003c1f7240 0x7f003c15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.034+0000 7f0043600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c06cb50 con 0x7f003c1f6e00 2024-03-06T07:37:18.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.034+0000 7f0043600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c160730 con 0x7f003c1f7240 2024-03-06T07:37:18.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0043600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c06ccd0 con 0x7f00200039b0 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 0x7f003c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 0x7f003c15a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56070/0 (socket says 172.21.15.97:56070) 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 -- 172.21.15.97:0/2258292694 learned_addr learned my addr 172.21.15.97:0/2258292694 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0040c00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f003c0aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 -- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 msgr2=0x7f003c0aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f003c0aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0041600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f003c1f7240 0x7f003c15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 -- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f003c1f7240 msgr2=0x7f003c15a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f003c1f7240 0x7f003c15a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 -- 172.21.15.97:0/2258292694 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c15cf00 con 0x7f003c1f6e00 2024-03-06T07:37:18.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0040c00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f003c0aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.035+0000 7f0033e00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 0x7f003c15a200 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f002400f580 tx=0x7f002400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.036+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024019020 con 0x7f003c1f6e00 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.036+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024014870 con 0x7f003c1f6e00 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.036+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024014c60 con 0x7f003c1f6e00 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.036+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c15e730 con 0x7f003c1f6e00 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.036+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c15ecf0 con 0x7f003c1f6e00 2024-03-06T07:37:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.037+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c1f4160 con 0x7f003c1f6e00 2024-03-06T07:37:18.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.038+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f002401c020 con 0x7f003c1f6e00 2024-03-06T07:37:18.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.039+0000 7f0039600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b4d0 0x7f001806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.039+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f002408a7c0 con 0x7f003c1f6e00 2024-03-06T07:37:18.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.039+0000 7f0040c00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b4d0 0x7f001806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.040+0000 7f0040c00640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b4d0 0x7f001806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f002c034000 tx=0x7f002c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.043+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0024023020 con 0x7f003c1f6e00 2024-03-06T07:37:18.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.238+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f003c04f330 con 0x7f003c1f6e00 2024-03-06T07:37:18.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.239+0000 7f0039600640 1 -- 172.21.15.97:0/2258292694 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+12889 (secure 0 0 0) 0x7f0024026350 con 0x7f003c1f6e00 2024-03-06T07:37:18.242 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:18.242 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":140,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:17.296107+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:18.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b4d0 msgr2=0x7f001806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b4d0 0x7f001806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f002c034000 tx=0x7f002c007000 comp rx=0 tx=0).stop 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 msgr2=0x7f003c15a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 0x7f003c15a200 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f002400f580 tx=0x7f002400fa40 comp rx=0 tx=0).stop 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 shutdown_connections 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b4d0 0x7f001806d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f003c1f7240 0x7f003c15a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003c1f6e00 0x7f003c15a200 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 --2- 172.21.15.97:0/2258292694 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f003c0aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 >> 172.21.15.97:0/2258292694 conn(0x7f003c0b1730 msgr2=0x7f003c0afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 shutdown_connections 2024-03-06T07:37:18.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.242+0000 7f0043600640 1 -- 172.21.15.97:0/2258292694 wait complete. 2024-03-06T07:37:18.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:37:18.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:37:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.468+0000 7fbec1800640 1 Processor -- start 2024-03-06T07:37:18.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.468+0000 7fbec1800640 1 -- start start 2024-03-06T07:37:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbec1800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbebc1551a0 0x7fbebc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbec1800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc0aee20 0x7fbebc0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbec1800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0cfb30 con 0x7fbebc154dc0 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbec1800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc0af290 con 0x7fbebc1551a0 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbec1800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc15e540 con 0x7fbebc0aee20 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbeb3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbebc1551a0 0x7fbebc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbebbe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fbebc154dc0 0x7fbebc161b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48152/0 (socket says 172.21.15.97:48152) 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.470+0000 7fbebbe00640 1 -- 172.21.15.97:0/3459294331 learned_addr learned my addr 172.21.15.97:0/3459294331 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.471+0000 7fbec0e00640 1 --2- 172.21.15.97:0/3459294331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc0aee20 0x7fbebc0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.471+0000 7fbeb3400640 1 -- 172.21.15.97:0/3459294331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc0aee20 msgr2=0x7fbebc0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.471+0000 7fbeb3400640 1 --2- 172.21.15.97:0/3459294331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc0aee20 0x7fbebc0cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.471+0000 7fbeb3400640 1 -- 172.21.15.97:0/3459294331 >> v1:172.21.15.97:6789/0 conn(0x7fbebc154dc0 legacy=0x7fbebc161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.471+0000 7fbeb3400640 1 -- 172.21.15.97:0/3459294331 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0cfe10 con 0x7fbebc1551a0 2024-03-06T07:37:18.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.472+0000 7fbeb3400640 1 --2- 172.21.15.97:0/3459294331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc1551a0 0x7fbebc0ae8e0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fbea40097a0 tx=0x7fbea402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887a747e1d07d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.472+0000 7fbebb400640 1 -- 172.21.15.97:0/3459294331 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4035020 con 0x7fbebc1551a0 2024-03-06T07:37:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.472+0000 7fbebb400640 1 -- 172.21.15.97:0/3459294331 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea4031810 con 0x7fbebc1551a0 2024-03-06T07:37:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.472+0000 7fbebb400640 1 -- 172.21.15.97:0/3459294331 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4031be0 con 0x7fbebc1551a0 2024-03-06T07:37:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 -- 172.21.15.97:0/3459294331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc1551a0 msgr2=0x7fbebc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 --2- 172.21.15.97:0/3459294331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc1551a0 0x7fbebc0ae8e0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fbea40097a0 tx=0x7fbea402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 -- 172.21.15.97:0/3459294331 shutdown_connections 2024-03-06T07:37:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 --2- 172.21.15.97:0/3459294331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc0aee20 0x7fbebc0cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 --2- 172.21.15.97:0/3459294331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc1551a0 0x7fbebc0ae8e0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 -- 172.21.15.97:0/3459294331 >> 172.21.15.97:0/3459294331 conn(0x7fbebc06afc0 msgr2=0x7fbebc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 -- 172.21.15.97:0/3459294331 shutdown_connections 2024-03-06T07:37:18.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.473+0000 7fbec1800640 1 -- 172.21.15.97:0/3459294331 wait complete. 2024-03-06T07:37:18.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.475+0000 7fbec1800640 1 Processor -- start 2024-03-06T07:37:18.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.475+0000 7fbec1800640 1 -- start start 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbec1800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbebc0aee20 0x7fbebc1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbec1800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 0x7fbebc1f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbec1800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc1551a0 0x7fbebc15fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbec1800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0cf820 con 0x7fbebc154dc0 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbec1800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0d03d0 con 0x7fbebc0aee20 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbec1800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0cf100 con 0x7fbebc1551a0 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 0x7fbebc1f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 0x7fbebc1f4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56094/0 (socket says 172.21.15.97:56094) 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.476+0000 7fbeb3400640 1 -- 172.21.15.97:0/592471311 learned_addr learned my addr 172.21.15.97:0/592471311 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb3400640 1 -- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc1551a0 msgr2=0x7fbebc15fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb3400640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc1551a0 0x7fbebc15fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbebbe00640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbebc0aee20 0x7fbebc1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb3400640 1 -- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbebc0aee20 msgr2=0x7fbebc1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb3400640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbebc0aee20 0x7fbebc1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb3400640 1 -- 172.21.15.97:0/592471311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc155cb0 con 0x7fbebc154dc0 2024-03-06T07:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbebbe00640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbebc0aee20 0x7fbebc1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb3400640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 0x7fbebc1f4a00 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098c0 tx=0x7fbea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4040070 con 0x7fbebc154dc0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea403e070 con 0x7fbebc154dc0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc155f90 con 0x7fbebc154dc0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4035040 con 0x7fbebc154dc0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.477+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc156550 con 0x7fbebc154dc0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.479+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc04f330 con 0x7fbebc154dc0 2024-03-06T07:37:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.480+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbea4038020 con 0x7fbebc154dc0 2024-03-06T07:37:18.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.480+0000 7fbeb8c00640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe9806b4d0 0x7fbe9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.481+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbea40ab6e0 con 0x7fbebc154dc0 2024-03-06T07:37:18.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.481+0000 7fbebbe00640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe9806b4d0 0x7fbe9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.481+0000 7fbebbe00640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe9806b4d0 0x7fbe9806d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbeac000f60 tx=0x7fbeac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.485+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbea407a940 con 0x7fbebc154dc0 2024-03-06T07:37:18.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.682+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbebc0add40 con 0x7fbebc154dc0 2024-03-06T07:37:18.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.682+0000 7fbeb8c00640 1 -- 172.21.15.97:0/592471311 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbea407a240 con 0x7fbebc154dc0 2024-03-06T07:37:18.685 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:37:18.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe9806b4d0 msgr2=0x7fbe9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe9806b4d0 0x7fbe9806d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbeac000f60 tx=0x7fbeac009040 comp rx=0 tx=0).stop 2024-03-06T07:37:18.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 msgr2=0x7fbebc1f4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 0x7fbebc1f4a00 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098c0 tx=0x7fbea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 shutdown_connections 2024-03-06T07:37:18.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe9806b4d0 0x7fbe9806d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbebc1551a0 0x7fbebc15fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbebc154dc0 0x7fbebc1f4a00 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 --2- 172.21.15.97:0/592471311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbebc0aee20 0x7fbebc1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 >> 172.21.15.97:0/592471311 conn(0x7fbebc06afc0 msgr2=0x7fbebc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 shutdown_connections 2024-03-06T07:37:18.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.685+0000 7fbec1800640 1 -- 172.21.15.97:0/592471311 wait complete. 2024-03-06T07:37:18.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:37:18.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.901+0000 7f89d7200640 1 Processor -- start 2024-03-06T07:37:18.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.901+0000 7f89d7200640 1 -- start start 2024-03-06T07:37:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89d7200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f89d00af1c0 0x7f89d015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89d7200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d0071d30 con 0x7f89d006e4d0 2024-03-06T07:37:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89d7200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d00711d0 con 0x7f89d0070d20 2024-03-06T07:37:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89d7200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d0071490 con 0x7f89d00af1c0 2024-03-06T07:37:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89d5200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f89d006e4d0 0x7f89d00b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48156/0 (socket says 172.21.15.97:48156) 2024-03-06T07:37:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89d5200640 1 -- 172.21.15.97:0/2226268446 learned_addr learned my addr 172.21.15.97:0/2226268446 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.903+0000 7f89cfe00640 1 --2- 172.21.15.97:0/2226268446 >> v2:172.21.15.135:3300/0 conn(0x7f89d00af1c0 0x7f89d015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850361588 0 0) 0x7f89c8001490 con 0x7f89d006e4d0 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b0003520 con 0x7f89d006e4d0 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701885779 0 0) 0x7f89b0003520 con 0x7f89d006e4d0 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 >> v2:172.21.15.135:3300/0 conn(0x7f89d00af1c0 msgr2=0x7f89d015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 --2- 172.21.15.97:0/2226268446 >> v2:172.21.15.135:3300/0 conn(0x7f89d00af1c0 0x7f89d015eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 >> v1:172.21.15.135:6789/0 conn(0x7f89d0070d20 legacy=0x7f89d006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0071770 con 0x7f89d006e4d0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f89c8001bc0 con 0x7f89d006e4d0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 >> v1:172.21.15.97:6789/0 conn(0x7f89d006e4d0 legacy=0x7f89d00b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b00039b0 0x7f89b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89b0004800 0x7f89b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89b00051c0 0x7f89b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d0071490 con 0x7f89b0004800 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b000c090 con 0x7f89b00039b0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b000c290 con 0x7f89b00051c0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.904+0000 7f89c7400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89b0004800 0x7f89b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89cfe00640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b00039b0 0x7f89b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89c7400640 1 -- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89b00051c0 msgr2=0x7f89b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89d5200640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89b00051c0 0x7f89b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89c7400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89b00051c0 0x7f89b000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89c7400640 1 -- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b00039b0 msgr2=0x7f89b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89c7400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b00039b0 0x7f89b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89d5200640 1 -- 172.21.15.97:0/2226268446 reap_dead start 2024-03-06T07:37:18.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.905+0000 7f89c7400640 1 -- 172.21.15.97:0/2226268446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b0003520 con 0x7f89b0004800 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.906+0000 7f89c7400640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89b0004800 0x7f89b0004c80 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f89b8001750 tx=0x7f89b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dda46a1533e760d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.906+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8001370 con 0x7f89b0004800 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.906+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b80052d0 con 0x7f89b0004800 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.906+0000 7f89cf400640 1 -- 172.21.15.97:0/2226268446 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8003830 con 0x7f89b0004800 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 -- 172.21.15.97:0/2226268446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89b0004800 msgr2=0x7f89b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89b0004800 0x7f89b0004c80 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f89b8001750 tx=0x7f89b0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 -- 172.21.15.97:0/2226268446 shutdown_connections 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 --2- 172.21.15.97:0/2226268446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89b0004800 0x7f89b0004c80 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 -- 172.21.15.97:0/2226268446 >> 172.21.15.97:0/2226268446 conn(0x7f89d00b1730 msgr2=0x7f89d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 -- 172.21.15.97:0/2226268446 shutdown_connections 2024-03-06T07:37:18.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.907+0000 7f89d7200640 1 -- 172.21.15.97:0/2226268446 wait complete. 2024-03-06T07:37:18.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.909+0000 7f89d7200640 1 Processor -- start 2024-03-06T07:37:18.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.909+0000 7f89d7200640 1 -- start start 2024-03-06T07:37:18.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d7200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b0004800 0x7f89d0142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 0x7f89d0152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d7200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d0152f60 0x7f89d00cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d7200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d015eff0 con 0x7f89d01524f0 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d7200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00b30b0 con 0x7f89b0004800 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d7200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d015f170 con 0x7f89d0152f60 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 0x7f89d0152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 0x7f89d0152a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56116/0 (socket says 172.21.15.97:56116) 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 -- 172.21.15.97:0/612269573 learned_addr learned my addr 172.21.15.97:0/612269573 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89cfe00640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b0004800 0x7f89d0142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89d5200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d0152f60 0x7f89d00cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 -- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d0152f60 msgr2=0x7f89d00cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d0152f60 0x7f89d00cfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 -- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b0004800 msgr2=0x7f89d0142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.910+0000 7f89c7400640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b0004800 0x7f89d0142240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89c7400640 1 -- 172.21.15.97:0/612269573 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0142780 con 0x7f89d01524f0 2024-03-06T07:37:18.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89c7400640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 0x7f89d0152a20 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f89b8005040 tx=0x7f89b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8041070 con 0x7f89d01524f0 2024-03-06T07:37:18.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b8005480 con 0x7f89d01524f0 2024-03-06T07:37:18.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8007a30 con 0x7f89d01524f0 2024-03-06T07:37:18.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d0154860 con 0x7f89d01524f0 2024-03-06T07:37:18.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.911+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d0154e20 con 0x7f89d01524f0 2024-03-06T07:37:18.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.913+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d004f330 con 0x7f89d01524f0 2024-03-06T07:37:18.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.913+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89b8044070 con 0x7f89d01524f0 2024-03-06T07:37:18.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.914+0000 7f89ccc00640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89ac06b4d0 0x7f89ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.914+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f89b80ab9d0 con 0x7f89d01524f0 2024-03-06T07:37:18.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.915+0000 7f89cfe00640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89ac06b4d0 0x7f89ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.915+0000 7f89cfe00640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89ac06b4d0 0x7f89ac06d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f89d00686c0 tx=0x7f89c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:18.919+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89b807aba0 con 0x7f89d01524f0 2024-03-06T07:37:19.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.103+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f89d004f330 con 0x7f89d01524f0 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.104+0000 7f89ccc00640 1 -- 172.21.15.97:0/612269573 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v140) v1 ==== 84+0+65 (secure 0 0 0) 0x7f89b807a4a0 con 0x7f89d01524f0 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-06T07:37:19.106 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-06T07:37:19.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89ac06b4d0 msgr2=0x7f89ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89ac06b4d0 0x7f89ac06d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f89d00686c0 tx=0x7f89c0008040 comp rx=0 tx=0).stop 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 msgr2=0x7f89d0152a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 0x7f89d0152a20 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f89b8005040 tx=0x7f89b8001120 comp rx=0 tx=0).stop 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 shutdown_connections 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89ac06b4d0 0x7f89ac06d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d0152f60 0x7f89d00cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d01524f0 0x7f89d0152a20 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 --2- 172.21.15.97:0/612269573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b0004800 0x7f89d0142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 >> 172.21.15.97:0/612269573 conn(0x7f89d00b1730 msgr2=0x7f89d00b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 shutdown_connections 2024-03-06T07:37:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.106+0000 7f89d7200640 1 -- 172.21.15.97:0/612269573 wait complete. 2024-03-06T07:37:19.126 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:37:19.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:37:19.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.318+0000 7fee9be00640 1 Processor -- start 2024-03-06T07:37:19.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.319+0000 7fee9be00640 1 -- start start 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee9c0d18f0 0x7fee9c0d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9be00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c0d1080 con 0x7fee9c0b5320 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9be00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c0cfd60 con 0x7fee9c0d0c70 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9be00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0d2990 con 0x7fee9c0d18f0 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee9c0d18f0 0x7fee9c0d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fee9c0b5320 0x7fee9c0cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48166/0 (socket says 172.21.15.97:48166) 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.320+0000 7fee9aa00640 1 -- 172.21.15.97:0/656033640 learned_addr learned my addr 172.21.15.97:0/656033640 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.321+0000 7fee9a000640 1 -- 172.21.15.97:0/656033640 >> v1:172.21.15.135:6789/0 conn(0x7fee9c0d0c70 legacy=0x7fee9c0b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:19.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.321+0000 7fee9a000640 1 -- 172.21.15.97:0/656033640 >> v1:172.21.15.97:6789/0 conn(0x7fee9c0b5320 legacy=0x7fee9c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.321+0000 7fee9a000640 1 -- 172.21.15.97:0/656033640 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0d1d80 con 0x7fee9c0d18f0 2024-03-06T07:37:19.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.321+0000 7fee99600640 1 -- 172.21.15.97:0/656033640 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874881560 0 0) 0x7fee8c001110 con 0x7fee9c0b5320 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.322+0000 7fee9a000640 1 --2- 172.21.15.97:0/656033640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d18f0 0x7fee9c0d0730 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fee84009d90 tx=0x7fee8402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36f109bf39e0a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.322+0000 7fee99600640 1 -- 172.21.15.97:0/656033640 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84039070 con 0x7fee9c0d18f0 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.322+0000 7fee99600640 1 -- 172.21.15.97:0/656033640 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8402cc60 con 0x7fee9c0d18f0 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.322+0000 7fee99600640 1 -- 172.21.15.97:0/656033640 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84034660 con 0x7fee9c0d18f0 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 -- 172.21.15.97:0/656033640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d18f0 msgr2=0x7fee9c0d0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 --2- 172.21.15.97:0/656033640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d18f0 0x7fee9c0d0730 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fee84009d90 tx=0x7fee8402beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 -- 172.21.15.97:0/656033640 shutdown_connections 2024-03-06T07:37:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 --2- 172.21.15.97:0/656033640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d18f0 0x7fee9c0d0730 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 -- 172.21.15.97:0/656033640 >> 172.21.15.97:0/656033640 conn(0x7fee9c0d3d20 msgr2=0x7fee9c0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:19.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 -- 172.21.15.97:0/656033640 shutdown_connections 2024-03-06T07:37:19.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.323+0000 7fee9be00640 1 -- 172.21.15.97:0/656033640 wait complete. 2024-03-06T07:37:19.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.324+0000 7fee9be00640 1 Processor -- start 2024-03-06T07:37:19.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.324+0000 7fee9be00640 1 -- start start 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee9c0b5320 0x7fee9c1f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 0x7fee9c1f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee9c06c530 0x7fee9c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0d2620 con 0x7fee9c0d0c70 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0d2340 con 0x7fee9c0b5320 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0b7280 con 0x7fee9c06c530 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.325+0000 7fee9a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 0x7fee9c1f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 0x7fee9c1f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56138/0 (socket says 172.21.15.97:56138) 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 -- 172.21.15.97:0/1312803780 learned_addr learned my addr 172.21.15.97:0/1312803780 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:19.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 -- 172.21.15.97:0/1312803780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee9c06c530 msgr2=0x7fee9c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee9c06c530 0x7fee9c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 -- 172.21.15.97:0/1312803780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee9c0b5320 msgr2=0x7fee9c1f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee9c0b5320 0x7fee9c1f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 -- 172.21.15.97:0/1312803780 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0cbae0 con 0x7fee9c0d0c70 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9a000640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 0x7fee9c1f70a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fee84009d90 tx=0x7fee84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84044020 con 0x7fee9c0d0c70 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8403f670 con 0x7fee9c0d0c70 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c0cbdc0 con 0x7fee9c0d0c70 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.326+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84039040 con 0x7fee9c0d0c70 2024-03-06T07:37:19.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.327+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c0cc380 con 0x7fee9c0d0c70 2024-03-06T07:37:19.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.328+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c04f330 con 0x7fee9c0d0c70 2024-03-06T07:37:19.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.329+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee84047020 con 0x7fee9c0d0c70 2024-03-06T07:37:19.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.329+0000 7fee92a00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee7806b520 0x7fee7806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.329+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fee840acac0 con 0x7fee9c0d0c70 2024-03-06T07:37:19.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.330+0000 7fee9aa00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee7806b520 0x7fee7806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.330+0000 7fee9aa00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee7806b520 0x7fee7806d9e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0cacf0 tx=0x7fee8c001310 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.334+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee8404e030 con 0x7fee9c0d0c70 2024-03-06T07:37:19.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:19.528+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fee9c162810 con 0x7fee9c0d0c70 2024-03-06T07:37:20.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.328+0000 7fee92a00640 1 -- 172.21.15.97:0/1312803780 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v141) v1 ==== 138+0+0 (secure 0 0 0) 0x7fee8407bd00 con 0x7fee9c0d0c70 2024-03-06T07:37:20.331 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-06T07:37:20.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.333+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee7806b520 msgr2=0x7fee7806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.333+0000 7fee9be00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee7806b520 0x7fee7806d9e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0cacf0 tx=0x7fee8c001310 comp rx=0 tx=0).stop 2024-03-06T07:37:20.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.333+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 msgr2=0x7fee9c1f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.333+0000 7fee9be00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 0x7fee9c1f70a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fee84009d90 tx=0x7fee84004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.334+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 shutdown_connections 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.334+0000 7fee9be00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fee7806b520 0x7fee7806d9e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.334+0000 7fee9be00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fee9c06c530 0x7fee9c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.334+0000 7fee9be00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee9c0d0c70 0x7fee9c1f70a0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.334+0000 7fee9be00640 1 --2- 172.21.15.97:0/1312803780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee9c0b5320 0x7fee9c1f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.334+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 >> 172.21.15.97:0/1312803780 conn(0x7fee9c0d3d20 msgr2=0x7fee9c161560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.335+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 shutdown_connections 2024-03-06T07:37:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.335+0000 7fee9be00640 1 -- 172.21.15.97:0/1312803780 wait complete. 2024-03-06T07:37:20.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:37:20.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.570+0000 7f4fb5000640 1 Processor -- start 2024-03-06T07:37:20.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.570+0000 7f4fb5000640 1 -- start start 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fb5000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fb00b3f70 0x7f4fb0160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fb5000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4fb0070ff0 0x7f4fb00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fb5000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb0071440 con 0x7f4fb015fc10 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fb5000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb00b46c0 con 0x7f4fb00b3f70 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fb5000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb00b49a0 con 0x7f4fb0070ff0 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fad400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fb00b3f70 0x7f4fb0160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fad400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4fb00b3f70 0x7f4fb0160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56150/0 (socket says 172.21.15.97:56150) 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fad400640 1 -- 172.21.15.97:0/1180719353 learned_addr learned my addr 172.21.15.97:0/1180719353 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.572+0000 7f4fade00640 1 --1- 172.21.15.97:0/1180719353 >> v1:172.21.15.97:6789/0 conn(0x7f4fb015fc10 0x7f4fb015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48172/0 (socket says 172.21.15.97:48172) 2024-03-06T07:37:20.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4fae800640 1 --2- 172.21.15.97:0/1180719353 >> v2:172.21.15.135:3300/0 conn(0x7f4fb0070ff0 0x7f4fb00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4fad400640 1 -- 172.21.15.97:0/1180719353 >> v2:172.21.15.135:3300/0 conn(0x7f4fb0070ff0 msgr2=0x7f4fb00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4fad400640 1 --2- 172.21.15.97:0/1180719353 >> v2:172.21.15.135:3300/0 conn(0x7f4fb0070ff0 0x7f4fb00ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4fad400640 1 -- 172.21.15.97:0/1180719353 >> v1:172.21.15.97:6789/0 conn(0x7f4fb015fc10 legacy=0x7f4fb015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4fad400640 1 -- 172.21.15.97:0/1180719353 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb0072c90 con 0x7f4fb00b3f70 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4fae800640 1 --2- 172.21.15.97:0/1180719353 >> v2:172.21.15.135:3300/0 conn(0x7f4fb0070ff0 0x7f4fb00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.573+0000 7f4faca00640 1 -- 172.21.15.97:0/1180719353 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226075237 0 0) 0x7f4fb0071440 con 0x7f4fb015fc10 2024-03-06T07:37:20.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.574+0000 7f4fad400640 1 --2- 172.21.15.97:0/1180719353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb0160700 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cf6a0 tx=0x7f4f9c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7da83087642388 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4faca00640 1 -- 172.21.15.97:0/1180719353 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c034070 con 0x7f4fb00b3f70 2024-03-06T07:37:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4faca00640 1 -- 172.21.15.97:0/1180719353 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c030860 con 0x7f4fb00b3f70 2024-03-06T07:37:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4faca00640 1 -- 172.21.15.97:0/1180719353 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c030c30 con 0x7f4fb00b3f70 2024-03-06T07:37:20.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4fb5000640 1 -- 172.21.15.97:0/1180719353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 msgr2=0x7f4fb0160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4fb5000640 1 --2- 172.21.15.97:0/1180719353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb0160700 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cf6a0 tx=0x7f4f9c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4fb5000640 1 -- 172.21.15.97:0/1180719353 shutdown_connections 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4fb5000640 1 --2- 172.21.15.97:0/1180719353 >> v2:172.21.15.135:3300/0 conn(0x7f4fb0070ff0 0x7f4fb00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4fb5000640 1 --2- 172.21.15.97:0/1180719353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb0160700 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.575+0000 7f4fb5000640 1 -- 172.21.15.97:0/1180719353 >> 172.21.15.97:0/1180719353 conn(0x7f4fb00b1940 msgr2=0x7f4fb00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.576+0000 7f4fb5000640 1 -- 172.21.15.97:0/1180719353 shutdown_connections 2024-03-06T07:37:20.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.576+0000 7f4fb5000640 1 -- 172.21.15.97:0/1180719353 wait complete. 2024-03-06T07:37:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.577+0000 7f4fb5000640 1 Processor -- start 2024-03-06T07:37:20.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.577+0000 7f4fb5000640 1 -- start start 2024-03-06T07:37:20.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.578+0000 7f4fb5000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fb0070ff0 0x7f4fb009a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:20.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.578+0000 7f4fb5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.578+0000 7f4fb5000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fb015fc10 0x7f4fb009b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.578+0000 7f4fb5000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb0072a30 con 0x7f4fb00b3f70 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.578+0000 7f4fb5000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00728b0 con 0x7f4fb0070ff0 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.578+0000 7f4fb5000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb015c2e0 con 0x7f4fb015fc10 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb014e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56154/0 (socket says 172.21.15.97:56154) 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 -- 172.21.15.97:0/3509558502 learned_addr learned my addr 172.21.15.97:0/3509558502 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fade00640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fb0070ff0 0x7f4fb009a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fae800640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fb015fc10 0x7f4fb009b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 -- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fb015fc10 msgr2=0x7f4fb009b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fb015fc10 0x7f4fb009b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 -- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fb0070ff0 msgr2=0x7f4fb009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fb0070ff0 0x7f4fb009a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fad400640 1 -- 172.21.15.97:0/3509558502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb009af30 con 0x7f4fb00b3f70 2024-03-06T07:37:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.579+0000 7f4fade00640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fb0070ff0 0x7f4fb009a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.580+0000 7f4fad400640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb014e320 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c02b9e0 tx=0x7f4f9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.580+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c048070 con 0x7f4fb00b3f70 2024-03-06T07:37:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.580+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c030660 con 0x7f4fb00b3f70 2024-03-06T07:37:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.580+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb014e860 con 0x7f4fb00b3f70 2024-03-06T07:37:20.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.580+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c034040 con 0x7f4fb00b3f70 2024-03-06T07:37:20.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.580+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb0155900 con 0x7f4fb00b3f70 2024-03-06T07:37:20.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.581+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb004f330 con 0x7f4fb00b3f70 2024-03-06T07:37:20.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.583+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f9c04b070 con 0x7f4fb00b3f70 2024-03-06T07:37:20.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.583+0000 7f4fa6000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4f9006b4d0 0x7f4f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:20.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.583+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f4f9c0ac790 con 0x7f4fb00b3f70 2024-03-06T07:37:20.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.584+0000 7f4fade00640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4f9006b4d0 0x7f4f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:20.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.584+0000 7f4fade00640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4f9006b4d0 0x7f4f9006d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0000f60 tx=0x7f4fa0007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.588+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f9c052020 con 0x7f4fb00b3f70 2024-03-06T07:37:20.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:20.785+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4fb006ef30 con 0x7f4fb00b3f70 2024-03-06T07:37:21.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.335+0000 7f4fa6000640 1 -- 172.21.15.97:0/3509558502 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v142) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4f9c07b860 con 0x7f4fb00b3f70 2024-03-06T07:37:21.338 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.340+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4f9006b4d0 msgr2=0x7f4f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.340+0000 7f4fb5000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4f9006b4d0 0x7f4f9006d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0000f60 tx=0x7f4fa0007ca0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.340+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 msgr2=0x7f4fb014e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.340+0000 7f4fb5000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb014e320 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c02b9e0 tx=0x7f4f9c004730 comp rx=0 tx=0).stop 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.341+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 shutdown_connections 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.341+0000 7f4fb5000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4f9006b4d0 0x7f4f9006d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.341+0000 7f4fb5000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4fb015fc10 0x7f4fb009b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.341+0000 7f4fb5000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fb00b3f70 0x7f4fb014e320 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.342+0000 7f4fb5000640 1 --2- 172.21.15.97:0/3509558502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4fb0070ff0 0x7f4fb009a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.342+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 >> 172.21.15.97:0/3509558502 conn(0x7f4fb00b1940 msgr2=0x7f4fb00acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:21.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.342+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 shutdown_connections 2024-03-06T07:37:21.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.343+0000 7f4fb5000640 1 -- 172.21.15.97:0/3509558502 wait complete. 2024-03-06T07:37:21.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:37:21.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.555+0000 7fcb4f200640 1 Processor -- start 2024-03-06T07:37:21.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.556+0000 7fcb4f200640 1 -- start start 2024-03-06T07:37:21.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.557+0000 7fcb4f200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcb5015e000 0x7fcb500ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.557+0000 7fcb4f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500cf020 con 0x7fcb500d1b90 2024-03-06T07:37:21.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.557+0000 7fcb4f200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500cf300 con 0x7fcb500d40b0 2024-03-06T07:37:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.557+0000 7fcb4f200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb500cedd0 con 0x7fcb5015e000 2024-03-06T07:37:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.557+0000 7fcb45400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcb500d1b90 0x7fcb500d39a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48176/0 (socket says 172.21.15.97:48176) 2024-03-06T07:37:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.557+0000 7fcb45400640 1 -- 172.21.15.97:0/243498212 learned_addr learned my addr 172.21.15.97:0/243498212 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4de00640 1 --2- 172.21.15.97:0/243498212 >> v2:172.21.15.135:3300/0 conn(0x7fcb5015e000 0x7fcb500ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623591585 0 0) 0x7fcb500cf020 con 0x7fcb500d1b90 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb34003520 con 0x7fcb500d1b90 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015721563 0 0) 0x7fcb34003520 con 0x7fcb500d1b90 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 >> v2:172.21.15.135:3300/0 conn(0x7fcb5015e000 msgr2=0x7fcb500ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 --2- 172.21.15.97:0/243498212 >> v2:172.21.15.135:3300/0 conn(0x7fcb5015e000 0x7fcb500ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 >> v1:172.21.15.135:6789/0 conn(0x7fcb500d40b0 legacy=0x7fcb500b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb500cf5e0 con 0x7fcb500d1b90 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fcb38001670 con 0x7fcb500d1b90 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 >> v1:172.21.15.97:6789/0 conn(0x7fcb500d1b90 legacy=0x7fcb500d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb340039b0 0x7fcb34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.558+0000 7fcb4d400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb340051c0 0x7fcb3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500cedd0 con 0x7fcb34004800 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3400c060 con 0x7fcb340039b0 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3400c260 con 0x7fcb340051c0 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb45400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb4de00640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb340039b0 0x7fcb34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb45400640 1 -- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb340051c0 msgr2=0x7fcb3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb45400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb340051c0 0x7fcb3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb45400640 1 -- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb340039b0 msgr2=0x7fcb34003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb45400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb340039b0 0x7fcb34003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb45400640 1 -- 172.21.15.97:0/243498212 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb34003520 con 0x7fcb34004800 2024-03-06T07:37:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.559+0000 7fcb4e800640 1 -- 172.21.15.97:0/243498212 reap_dead start 2024-03-06T07:37:21.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.560+0000 7fcb45400640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb34004c80 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c240 tx=0x7fcb34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f226d6d25f2b9adf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.560+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38007cf0 con 0x7fcb34004800 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.560+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38030070 con 0x7fcb34004800 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.560+0000 7fcb4d400640 1 -- 172.21.15.97:0/243498212 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38005d70 con 0x7fcb34004800 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 -- 172.21.15.97:0/243498212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 msgr2=0x7fcb34004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb34004c80 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c240 tx=0x7fcb34005830 comp rx=0 tx=0).stop 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 -- 172.21.15.97:0/243498212 shutdown_connections 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 --2- 172.21.15.97:0/243498212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb34004c80 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 -- 172.21.15.97:0/243498212 >> 172.21.15.97:0/243498212 conn(0x7fcb5008d740 msgr2=0x7fcb5009d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 -- 172.21.15.97:0/243498212 shutdown_connections 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.561+0000 7fcb4f200640 1 -- 172.21.15.97:0/243498212 wait complete. 2024-03-06T07:37:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.563+0000 7fcb4f200640 1 Processor -- start 2024-03-06T07:37:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.563+0000 7fcb4f200640 1 -- start start 2024-03-06T07:37:21.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb500b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb501f6fb0 0x7fcb50158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb5014edc0 0x7fcb5014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500d2010 con 0x7fcb34004800 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500cfba0 con 0x7fcb5014edc0 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500ce890 con 0x7fcb501f6fb0 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb500b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb500b6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56172/0 (socket says 172.21.15.97:56172) 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.564+0000 7fcb4de00640 1 -- 172.21.15.97:0/234731392 learned_addr learned my addr 172.21.15.97:0/234731392 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4e800640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb5014edc0 0x7fcb5014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb45400640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb501f6fb0 0x7fcb50158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4de00640 1 -- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb501f6fb0 msgr2=0x7fcb50158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4de00640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb501f6fb0 0x7fcb50158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4de00640 1 -- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb5014edc0 msgr2=0x7fcb5014f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4de00640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb5014edc0 0x7fcb5014f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4de00640 1 -- 172.21.15.97:0/234731392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb500b70d0 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4de00640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb500b6960 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fcb40011700 tx=0x7fcb40011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4000f9c0 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4000fb60 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb500b73b0 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.565+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40015460 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.566+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb50159350 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.567+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb500b7e40 con 0x7fcb34004800 2024-03-06T07:37:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.568+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb40015600 con 0x7fcb34004800 2024-03-06T07:37:21.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.568+0000 7fcb46a00640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.569+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcb4008abe0 con 0x7fcb34004800 2024-03-06T07:37:21.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.569+0000 7fcb45400640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.569+0000 7fcb45400640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcb38034000 tx=0x7fcb38039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.573+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb500b7e40 con 0x7fcb34004800 2024-03-06T07:37:21.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:21.807+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcb5004f330 con 0x7fcb34004800 2024-03-06T07:37:22.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.377+0000 7fcb46a00640 1 -- 172.21.15.97:0/234731392 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v103) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcb40019030 con 0x7fcb34004800 2024-03-06T07:37:22.384 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 20 and data pool 21 2024-03-06T07:37:22.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb2c06b4d0 msgr2=0x7fcb2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcb38034000 tx=0x7fcb38039040 comp rx=0 tx=0).stop 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 msgr2=0x7fcb500b6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb500b6960 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fcb40011700 tx=0x7fcb40011bc0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 shutdown_connections 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb5014edc0 0x7fcb5014f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcb501f6fb0 0x7fcb50158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 --2- 172.21.15.97:0/234731392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb34004800 0x7fcb500b6960 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.390+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 >> 172.21.15.97:0/234731392 conn(0x7fcb5008d740 msgr2=0x7fcb5015b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:22.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.392+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 shutdown_connections 2024-03-06T07:37:22.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.392+0000 7fcb4f200640 1 -- 172.21.15.97:0/234731392 wait complete. 2024-03-06T07:37:22.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:22.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.597+0000 7fa752e00640 1 Processor -- start 2024-03-06T07:37:22.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.597+0000 7fa752e00640 1 -- start start 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.598+0000 7fa752e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa74c15eff0 0x7fa74c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.598+0000 7fa752e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa74c06e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.598+0000 7fa752e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74c071d30 con 0x7fa74c0af1c0 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.598+0000 7fa752e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c071260 con 0x7fa74c15eff0 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.598+0000 7fa752e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c15f4a0 con 0x7fa74c06e4d0 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa74be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa74c0af1c0 0x7fa74c06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48190/0 (socket says 172.21.15.97:48190) 2024-03-06T07:37:22.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa74be00640 1 -- 172.21.15.97:0/16732764 learned_addr learned my addr 172.21.15.97:0/16732764 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa750e00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa74c06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053075085 0 0) 0x7fa73c001110 con 0x7fa74c0af1c0 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa743e00640 1 --2- 172.21.15.97:0/16732764 >> v2:172.21.15.135:3300/0 conn(0x7fa74c15eff0 0x7fa74c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa730003520 con 0x7fa74c0af1c0 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.599+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427470608 0 0) 0x7fa730003520 con 0x7fa74c0af1c0 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 msgr2=0x7fa74c06e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa74c06e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 >> v2:172.21.15.135:3300/0 conn(0x7fa74c15eff0 msgr2=0x7fa74c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 --2- 172.21.15.97:0/16732764 >> v2:172.21.15.135:3300/0 conn(0x7fa74c15eff0 0x7fa74c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c071490 con 0x7fa74c0af1c0 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa73c001840 con 0x7fa74c0af1c0 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 >> v1:172.21.15.97:6789/0 conn(0x7fa74c0af1c0 legacy=0x7fa74c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa730003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa730004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730004d40 0x7fa7300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c15f4a0 con 0x7fa7300039b0 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73000b9c0 con 0x7fa730004d40 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73000bb40 con 0x7fa74c06e4d0 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74be00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa730003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa743e00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa730004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74be00640 1 -- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 msgr2=0x7fa730004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:22.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa750e00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730004d40 0x7fa7300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74be00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa730004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74be00640 1 -- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730004d40 msgr2=0x7fa7300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:22.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74be00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730004d40 0x7fa7300051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.600+0000 7fa74be00640 1 -- 172.21.15.97:0/16732764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa730003520 con 0x7fa7300039b0 2024-03-06T07:37:22.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.601+0000 7fa743e00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c06e4d0 0x7fa730004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:22.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa74be00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa730003dd0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c4f0 tx=0x7fa730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bf83cfffd2f4d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:22.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c00c0e0 con 0x7fa7300039b0 2024-03-06T07:37:22.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c030070 con 0x7fa7300039b0 2024-03-06T07:37:22.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa74b400640 1 -- 172.21.15.97:0/16732764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c0058d0 con 0x7fa7300039b0 2024-03-06T07:37:22.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa752e00640 1 -- 172.21.15.97:0/16732764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 msgr2=0x7fa730003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa752e00640 1 --2- 172.21.15.97:0/16732764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa730003dd0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c4f0 tx=0x7fa730005830 comp rx=0 tx=0).stop 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.602+0000 7fa750e00640 1 -- 172.21.15.97:0/16732764 reap_dead start 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.603+0000 7fa752e00640 1 -- 172.21.15.97:0/16732764 shutdown_connections 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.603+0000 7fa752e00640 1 -- 172.21.15.97:0/16732764 >> 172.21.15.97:0/16732764 conn(0x7fa74c0b1730 msgr2=0x7fa74c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.603+0000 7fa752e00640 1 -- 172.21.15.97:0/16732764 shutdown_connections 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.603+0000 7fa752e00640 1 -- 172.21.15.97:0/16732764 wait complete. 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.604+0000 7fa752e00640 1 Processor -- start 2024-03-06T07:37:22.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.604+0000 7fa752e00640 1 -- start start 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.605+0000 7fa752e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa74c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.605+0000 7fa752e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c1e7c70 0x7fa74c1f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.605+0000 7fa752e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0afff0 0x7fa74c1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.605+0000 7fa752e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c0b30b0 con 0x7fa7300039b0 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.605+0000 7fa752e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c071a50 con 0x7fa74c0afff0 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.605+0000 7fa752e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c06d370 con 0x7fa74c1e7c70 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa74c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa74c0cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56190/0 (socket says 172.21.15.97:56190) 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 -- 172.21.15.97:0/4294541069 learned_addr learned my addr 172.21.15.97:0/4294541069 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa750e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0afff0 0x7fa74c1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 -- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c1e7c70 msgr2=0x7fa74c1f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa743e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c1e7c70 0x7fa74c1f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c1e7c70 0x7fa74c1f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 -- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0afff0 msgr2=0x7fa74c1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0afff0 0x7fa74c1581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 -- 172.21.15.97:0/4294541069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0ccd30 con 0x7fa7300039b0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa750e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0afff0 0x7fa74c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.606+0000 7fa74be00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa74c0cc430 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c4f0 tx=0x7fa73c001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.607+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c045070 con 0x7fa7300039b0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.607+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c000ed0 con 0x7fa7300039b0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.607+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c004be0 con 0x7fa7300039b0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.607+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c0cd010 con 0x7fa7300039b0 2024-03-06T07:37:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.607+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c0ce6d0 con 0x7fa7300039b0 2024-03-06T07:37:22.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.608+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c04f330 con 0x7fa7300039b0 2024-03-06T07:37:22.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.609+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa73c048070 con 0x7fa7300039b0 2024-03-06T07:37:22.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.610+0000 7fa748c00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa72806b4d0 0x7fa72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.610+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa73c0ab4c0 con 0x7fa7300039b0 2024-03-06T07:37:22.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.610+0000 7fa743e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa72806b4d0 0x7fa72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.611+0000 7fa743e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa72806b4d0 0x7fa72806d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0b4680 tx=0x7fa734008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:22.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.615+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa74c071d30 con 0x7fa7300039b0 2024-03-06T07:37:22.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.804+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa74c06ec40 con 0x7fa7300039b0 2024-03-06T07:37:22.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.807+0000 7fa748c00640 1 -- 172.21.15.97:0/4294541069 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16680 (secure 0 0 0) 0x7fa73c07a2a0 con 0x7fa7300039b0 2024-03-06T07:37:22.810 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:22.810 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":143,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:21.809318+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa72806b4d0 msgr2=0x7fa72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa72806b4d0 0x7fa72806d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0b4680 tx=0x7fa734008040 comp rx=0 tx=0).stop 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 msgr2=0x7fa74c0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa74c0cc430 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c4f0 tx=0x7fa73c001c10 comp rx=0 tx=0).stop 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 shutdown_connections 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa72806b4d0 0x7fa72806d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0afff0 0x7fa74c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa74c1e7c70 0x7fa74c1f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 --2- 172.21.15.97:0/4294541069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7300039b0 0x7fa74c0cc430 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 >> 172.21.15.97:0/4294541069 conn(0x7fa74c0b1730 msgr2=0x7fa74c1602e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.810+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 shutdown_connections 2024-03-06T07:37:22.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:22.811+0000 7fa752e00640 1 -- 172.21.15.97:0/4294541069 wait complete. 2024-03-06T07:37:22.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.028+0000 7f49e6800640 1 Processor -- start 2024-03-06T07:37:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.028+0000 7f49e6800640 1 -- start start 2024-03-06T07:37:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49e6800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f49e00b4680 0x7f49e006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49e6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f49e0072340 0x7f49e006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49e6800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e015b090 con 0x7f49e015acb0 2024-03-06T07:37:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49e6800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e00b2a30 con 0x7f49e0072340 2024-03-06T07:37:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49e6800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e00b4360 con 0x7f49e00b4680 2024-03-06T07:37:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49df400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f49e015acb0 0x7f49e00b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48198/0 (socket says 172.21.15.97:48198) 2024-03-06T07:37:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49df400640 1 -- 172.21.15.97:0/778989995 learned_addr learned my addr 172.21.15.97:0/778989995 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.030+0000 7f49dfe00640 1 --2- 172.21.15.97:0/778989995 >> v2:172.21.15.97:3300/0 conn(0x7f49e0072340 0x7f49e006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49dea00640 1 -- 172.21.15.97:0/778989995 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892225168 0 0) 0x7f49e015b090 con 0x7f49e015acb0 2024-03-06T07:37:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49d6a00640 1 --2- 172.21.15.97:0/778989995 >> v2:172.21.15.135:3300/0 conn(0x7f49e00b4680 0x7f49e006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49dea00640 1 -- 172.21.15.97:0/778989995 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c4003520 con 0x7f49e015acb0 2024-03-06T07:37:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49dfe00640 1 -- 172.21.15.97:0/778989995 >> v2:172.21.15.135:3300/0 conn(0x7f49e00b4680 msgr2=0x7f49e006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49dfe00640 1 --2- 172.21.15.97:0/778989995 >> v2:172.21.15.135:3300/0 conn(0x7f49e00b4680 0x7f49e006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49dfe00640 1 -- 172.21.15.97:0/778989995 >> v1:172.21.15.97:6789/0 conn(0x7f49e015acb0 legacy=0x7f49e00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.031+0000 7f49dfe00640 1 -- 172.21.15.97:0/778989995 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e00b2cf0 con 0x7f49e0072340 2024-03-06T07:37:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.032+0000 7f49dfe00640 1 --2- 172.21.15.97:0/778989995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e0072340 0x7f49e006ea30 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f49e0068560 tx=0x7f49d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2786aea75ae855 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.032+0000 7f49dea00640 1 -- 172.21.15.97:0/778989995 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8035020 con 0x7f49e0072340 2024-03-06T07:37:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.032+0000 7f49dea00640 1 -- 172.21.15.97:0/778989995 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d8031830 con 0x7f49e0072340 2024-03-06T07:37:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.032+0000 7f49dea00640 1 -- 172.21.15.97:0/778989995 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8031c00 con 0x7f49e0072340 2024-03-06T07:37:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 -- 172.21.15.97:0/778989995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e0072340 msgr2=0x7f49e006ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 --2- 172.21.15.97:0/778989995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e0072340 0x7f49e006ea30 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f49e0068560 tx=0x7f49d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 -- 172.21.15.97:0/778989995 shutdown_connections 2024-03-06T07:37:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 --2- 172.21.15.97:0/778989995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e0072340 0x7f49e006ea30 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 --2- 172.21.15.97:0/778989995 >> v2:172.21.15.135:3300/0 conn(0x7f49e00b4680 0x7f49e006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 -- 172.21.15.97:0/778989995 >> 172.21.15.97:0/778989995 conn(0x7f49e00b1730 msgr2=0x7f49e00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 -- 172.21.15.97:0/778989995 shutdown_connections 2024-03-06T07:37:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.033+0000 7f49e6800640 1 -- 172.21.15.97:0/778989995 wait complete. 2024-03-06T07:37:23.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.035+0000 7f49e6800640 1 Processor -- start 2024-03-06T07:37:23.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.035+0000 7f49e6800640 1 -- start start 2024-03-06T07:37:23.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.036+0000 7f49e6800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49e0072340 0x7f49e01f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.036+0000 7f49e6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 0x7f49e015b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.036+0000 7f49e6800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49e015acb0 0x7f49e0147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.036+0000 7f49e6800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00b4010 con 0x7f49e00b4680 2024-03-06T07:37:23.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.036+0000 7f49e6800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00b32b0 con 0x7f49e015acb0 2024-03-06T07:37:23.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.036+0000 7f49e6800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00ccfb0 con 0x7f49e0072340 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 0x7f49e015b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 0x7f49e015b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56202/0 (socket says 172.21.15.97:56202) 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 -- 172.21.15.97:0/1245986797 learned_addr learned my addr 172.21.15.97:0/1245986797 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49df400640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49e0072340 0x7f49e01f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 -- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49e0072340 msgr2=0x7f49e01f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49dfe00640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49e015acb0 0x7f49e0147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49e0072340 0x7f49e01f3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 -- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49e015acb0 msgr2=0x7f49e0147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49e015acb0 0x7f49e0147d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.037+0000 7f49d6a00640 1 -- 172.21.15.97:0/1245986797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e0158e50 con 0x7f49e00b4680 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.038+0000 7f49d6a00640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 0x7f49e015b090 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d800 tx=0x7f49c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.038+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c800b6c0 con 0x7f49e00b4680 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.038+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c8019070 con 0x7f49e00b4680 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.038+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e0159130 con 0x7f49e00b4680 2024-03-06T07:37:23.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.038+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c800ba80 con 0x7f49e00b4680 2024-03-06T07:37:23.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.038+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e01596f0 con 0x7f49e00b4680 2024-03-06T07:37:23.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.039+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e004f330 con 0x7f49e00b4680 2024-03-06T07:37:23.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.040+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49c800bc20 con 0x7f49e00b4680 2024-03-06T07:37:23.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.041+0000 7f49d7e00640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f49bc06b4d0 0x7f49bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.041+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49c808a6f0 con 0x7f49e00b4680 2024-03-06T07:37:23.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.041+0000 7f49df400640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f49bc06b4d0 0x7f49bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.041+0000 7f49df400640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f49bc06b4d0 0x7f49bc06d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f49d00012e0 tx=0x7f49d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.046+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49c801c360 con 0x7f49e00b4680 2024-03-06T07:37:23.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.245+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49e006ce60 con 0x7f49e00b4680 2024-03-06T07:37:23.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.248+0000 7f49d7e00640 1 -- 172.21.15.97:0/1245986797 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f49c8059560 con 0x7f49e00b4680 2024-03-06T07:37:23.251 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:23.251 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":143,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:21.809318+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f49bc06b4d0 msgr2=0x7f49bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f49bc06b4d0 0x7f49bc06d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f49d00012e0 tx=0x7f49d0009040 comp rx=0 tx=0).stop 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 msgr2=0x7f49e015b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 0x7f49e015b090 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d800 tx=0x7f49c800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 shutdown_connections 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f49bc06b4d0 0x7f49bc06d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49e015acb0 0x7f49e0147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49e00b4680 0x7f49e015b090 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 --2- 172.21.15.97:0/1245986797 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f49e0072340 0x7f49e01f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.251+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 >> 172.21.15.97:0/1245986797 conn(0x7f49e00b1730 msgr2=0x7f49e015edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.252+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 shutdown_connections 2024-03-06T07:37:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.252+0000 7f49e6800640 1 -- 172.21.15.97:0/1245986797 wait complete. 2024-03-06T07:37:23.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:37:23.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.502+0000 7f6c1e400640 1 Processor -- start 2024-03-06T07:37:23.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.502+0000 7f6c1e400640 1 -- start start 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.503+0000 7f6c1e400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6c1815fc10 0x7f6c18070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c1e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 0x7f6c180ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c1e400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c180b46c0 con 0x7f6c1815fff0 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c1e400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c180b49a0 con 0x7f6c1815fc10 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c1e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18072c90 con 0x7f6c180b3f70 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c17e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 0x7f6c180ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c17400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6c1815fc10 0x7f6c18070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c17e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 0x7f6c180ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:46258/0 (socket says 172.21.15.97:46258) 2024-03-06T07:37:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.504+0000 7f6c17e00640 1 -- 172.21.15.97:0/1474934573 learned_addr learned my addr 172.21.15.97:0/1474934573 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.505+0000 7f6c17e00640 1 -- 172.21.15.97:0/1474934573 >> v2:172.21.15.135:3300/0 conn(0x7f6c1815fc10 msgr2=0x7f6c18070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.505+0000 7f6c17e00640 1 --2- 172.21.15.97:0/1474934573 >> v2:172.21.15.135:3300/0 conn(0x7f6c1815fc10 0x7f6c18070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.505+0000 7f6c17e00640 1 -- 172.21.15.97:0/1474934573 >> v1:172.21.15.135:6789/0 conn(0x7f6c1815fff0 legacy=0x7f6c18160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.505+0000 7f6c17e00640 1 -- 172.21.15.97:0/1474934573 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1809dd80 con 0x7f6c180b3f70 2024-03-06T07:37:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.506+0000 7f6c17e00640 1 --2- 172.21.15.97:0/1474934573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 0x7f6c180ce4a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009870 tx=0x7f6c1002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0a7a34659060ede server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.507+0000 7f6c16a00640 1 -- 172.21.15.97:0/1474934573 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10034070 con 0x7f6c180b3f70 2024-03-06T07:37:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.507+0000 7f6c16a00640 1 -- 172.21.15.97:0/1474934573 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c10030840 con 0x7f6c180b3f70 2024-03-06T07:37:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.507+0000 7f6c16a00640 1 -- 172.21.15.97:0/1474934573 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10030c10 con 0x7f6c180b3f70 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.507+0000 7f6c1e400640 1 -- 172.21.15.97:0/1474934573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 msgr2=0x7f6c180ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.507+0000 7f6c1e400640 1 --2- 172.21.15.97:0/1474934573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 0x7f6c180ce4a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009870 tx=0x7f6c1002b980 comp rx=0 tx=0).stop 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.508+0000 7f6c1e400640 1 -- 172.21.15.97:0/1474934573 shutdown_connections 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.508+0000 7f6c1e400640 1 --2- 172.21.15.97:0/1474934573 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c180b3f70 0x7f6c180ce4a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.508+0000 7f6c1e400640 1 --2- 172.21.15.97:0/1474934573 >> v2:172.21.15.135:3300/0 conn(0x7f6c1815fc10 0x7f6c18070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.508+0000 7f6c1e400640 1 -- 172.21.15.97:0/1474934573 >> 172.21.15.97:0/1474934573 conn(0x7f6c180b1940 msgr2=0x7f6c180b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.508+0000 7f6c1e400640 1 -- 172.21.15.97:0/1474934573 shutdown_connections 2024-03-06T07:37:23.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.508+0000 7f6c1e400640 1 -- 172.21.15.97:0/1474934573 wait complete. 2024-03-06T07:37:23.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.510+0000 7f6c1e400640 1 Processor -- start 2024-03-06T07:37:23.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.511+0000 7f6c1e400640 1 -- start start 2024-03-06T07:37:23.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.512+0000 7f6c1e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 0x7f6c1809c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.512+0000 7f6c1e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c1815fc10 0x7f6c180cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.512+0000 7f6c1e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c1815fff0 0x7f6c18154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.512+0000 7f6c1e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1815c460 con 0x7f6c180b3f70 2024-03-06T07:37:23.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.512+0000 7f6c1e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1815c2e0 con 0x7f6c1815fc10 2024-03-06T07:37:23.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.512+0000 7f6c1e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c180ce9e0 con 0x7f6c1815fff0 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 0x7f6c1809c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 0x7f6c1809c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56204/0 (socket says 172.21.15.97:56204) 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 -- 172.21.15.97:0/3585834371 learned_addr learned my addr 172.21.15.97:0/3585834371 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17e00640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c1815fff0 0x7f6c18154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c0ea00640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c1815fc10 0x7f6c180cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 -- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c1815fff0 msgr2=0x7f6c18154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c1815fff0 0x7f6c18154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 -- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c1815fc10 msgr2=0x7f6c180cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c1815fc10 0x7f6c180cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 -- 172.21.15.97:0/3585834371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180cb520 con 0x7f6c180b3f70 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17e00640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c1815fff0 0x7f6c18154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:23.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.513+0000 7f6c17400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 0x7f6c1809c5a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800efc0 tx=0x7f6c0800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.514+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08018020 con 0x7f6c180b3f70 2024-03-06T07:37:23.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.514+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c08014480 con 0x7f6c180b3f70 2024-03-06T07:37:23.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.514+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1809bac0 con 0x7f6c180b3f70 2024-03-06T07:37:23.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.514+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08014850 con 0x7f6c180b3f70 2024-03-06T07:37:23.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.514+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1809cae0 con 0x7f6c180b3f70 2024-03-06T07:37:23.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.515+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c18159590 con 0x7f6c180b3f70 2024-03-06T07:37:23.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.516+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c08005d10 con 0x7f6c180b3f70 2024-03-06T07:37:23.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.518+0000 7f6c0fe00640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6bf406b4d0 0x7f6bf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.518+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c0808b100 con 0x7f6c180b3f70 2024-03-06T07:37:23.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.518+0000 7f6c0ea00640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6bf406b4d0 0x7f6bf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.519+0000 7f6c0ea00640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6bf406b4d0 0x7f6bf406d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6c00007fd0 tx=0x7f6c00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.522+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c18159590 con 0x7f6c180b3f70 2024-03-06T07:37:23.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:23.755+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6c1804f330 con 0x7f6c180b3f70 2024-03-06T07:37:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.391+0000 7f6c0fe00640 1 -- 172.21.15.97:0/3585834371 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v106) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6c08022030 con 0x7f6c180b3f70 2024-03-06T07:37:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.395+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6bf406b4d0 msgr2=0x7f6bf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.395+0000 7f6c1e400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6bf406b4d0 0x7f6bf406d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6c00007fd0 tx=0x7f6c00009040 comp rx=0 tx=0).stop 2024-03-06T07:37:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 msgr2=0x7f6c1809c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 0x7f6c1809c5a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800efc0 tx=0x7f6c0800c490 comp rx=0 tx=0).stop 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 shutdown_connections 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6bf406b4d0 0x7f6bf406d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6c1815fff0 0x7f6c18154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c1815fc10 0x7f6c180cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 --2- 172.21.15.97:0/3585834371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c180b3f70 0x7f6c1809c5a0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 >> 172.21.15.97:0/3585834371 conn(0x7f6c180b1940 msgr2=0x7f6c180acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 shutdown_connections 2024-03-06T07:37:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.396+0000 7f6c1e400640 1 -- 172.21.15.97:0/3585834371 wait complete. 2024-03-06T07:37:24.418 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:37:24.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:37:24.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.625+0000 7f91f7000640 1 Processor -- start 2024-03-06T07:37:24.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.626+0000 7f91f7000640 1 -- start start 2024-03-06T07:37:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91f7000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f91f0161b50 0x7f91f00d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91f7000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00cecd0 con 0x7f91f00b5490 2024-03-06T07:37:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91f7000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00cfc80 con 0x7f91f00d4a00 2024-03-06T07:37:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91f7000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00cef40 con 0x7f91f0161b50 2024-03-06T07:37:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91efe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f91f0161b50 0x7f91f00d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91f5000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f91f00b5490 0x7f91f00c8080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49342/0 (socket says 172.21.15.97:49342) 2024-03-06T07:37:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.627+0000 7f91f5000640 1 -- 172.21.15.97:0/3940743684 learned_addr learned my addr 172.21.15.97:0/3940743684 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.628+0000 7f91efe00640 1 -- 172.21.15.97:0/3940743684 >> v1:172.21.15.135:6789/0 conn(0x7f91f00d4a00 legacy=0x7f91f00d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.628+0000 7f91efe00640 1 -- 172.21.15.97:0/3940743684 >> v1:172.21.15.97:6789/0 conn(0x7f91f00b5490 legacy=0x7f91f00c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.628+0000 7f91efe00640 1 -- 172.21.15.97:0/3940743684 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00c7b60 con 0x7f91f0161b50 2024-03-06T07:37:24.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.628+0000 7f91eea00640 1 -- 172.21.15.97:0/3940743684 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670432505 0 0) 0x7f91f00cecd0 con 0x7f91f00b5490 2024-03-06T07:37:24.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.629+0000 7f91efe00640 1 --2- 172.21.15.97:0/3940743684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f0161b50 0x7f91f00d05a0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009d90 tx=0x7f91e002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca455891f49eb88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.629+0000 7f91eea00640 1 -- 172.21.15.97:0/3940743684 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0039070 con 0x7f91f0161b50 2024-03-06T07:37:24.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.629+0000 7f91eea00640 1 -- 172.21.15.97:0/3940743684 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e002cc60 con 0x7f91f0161b50 2024-03-06T07:37:24.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.629+0000 7f91eea00640 1 -- 172.21.15.97:0/3940743684 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0034660 con 0x7f91f0161b50 2024-03-06T07:37:24.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 -- 172.21.15.97:0/3940743684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f0161b50 msgr2=0x7f91f00d05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 --2- 172.21.15.97:0/3940743684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f0161b50 0x7f91f00d05a0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009d90 tx=0x7f91e002beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 -- 172.21.15.97:0/3940743684 shutdown_connections 2024-03-06T07:37:24.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 --2- 172.21.15.97:0/3940743684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f0161b50 0x7f91f00d05a0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 -- 172.21.15.97:0/3940743684 >> 172.21.15.97:0/3940743684 conn(0x7f91f00d3e70 msgr2=0x7f91f00d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:24.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 -- 172.21.15.97:0/3940743684 shutdown_connections 2024-03-06T07:37:24.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.630+0000 7f91f7000640 1 -- 172.21.15.97:0/3940743684 wait complete. 2024-03-06T07:37:24.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.632+0000 7f91f7000640 1 Processor -- start 2024-03-06T07:37:24.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.632+0000 7f91f7000640 1 -- start start 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91f7000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 0x7f91f015e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91f7000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00d4a00 0x7f91f00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91f7000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f91f0161b50 0x7f91f00cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91f7000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00c8910 con 0x7f91f00b5490 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91f7000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00c8790 con 0x7f91f00d4a00 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91f7000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00c8a90 con 0x7f91f0161b50 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 0x7f91f015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 0x7f91f015e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38240/0 (socket says 172.21.15.97:38240) 2024-03-06T07:37:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.633+0000 7f91efe00640 1 -- 172.21.15.97:0/1653661722 learned_addr learned my addr 172.21.15.97:0/1653661722 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91ef400640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00d4a00 0x7f91f00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91efe00640 1 -- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f91f0161b50 msgr2=0x7f91f00cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91efe00640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f91f0161b50 0x7f91f00cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91efe00640 1 -- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00d4a00 msgr2=0x7f91f00caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91efe00640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00d4a00 0x7f91f00caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91efe00640 1 -- 172.21.15.97:0/1653661722 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f015ee60 con 0x7f91f00b5490 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91ef400640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00d4a00 0x7f91f00caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91efe00640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 0x7f91f015e880 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009d90 tx=0x7f91e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0044020 con 0x7f91f00b5490 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e003f670 con 0x7f91f00b5490 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f015f140 con 0x7f91f00b5490 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.634+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0039040 con 0x7f91f00b5490 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.635+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f015f700 con 0x7f91f00b5490 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.636+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f004f330 con 0x7f91f00b5490 2024-03-06T07:37:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.637+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91e0047020 con 0x7f91f00b5490 2024-03-06T07:37:24.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.637+0000 7f91e7e00640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91cc06b500 0x7f91cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.637+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91e00acc40 con 0x7f91f00b5490 2024-03-06T07:37:24.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.638+0000 7f91ef400640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91cc06b500 0x7f91cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.638+0000 7f91ef400640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91cc06b500 0x7f91cc06d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f91f014da00 tx=0x7f91d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.642+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91e004e070 con 0x7f91f00b5490 2024-03-06T07:37:24.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:24.902+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f91f015a170 con 0x7f91f00b5490 2024-03-06T07:37:25.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.401+0000 7f91e7e00640 1 -- 172.21.15.97:0/1653661722 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v107) v1 ==== 122+0+0 (secure 0 0 0) 0x7f91e007bb50 con 0x7f91f00b5490 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91cc06b500 msgr2=0x7f91cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91cc06b500 0x7f91cc06d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f91f014da00 tx=0x7f91d8009040 comp rx=0 tx=0).stop 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 msgr2=0x7f91f015e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 0x7f91f015e880 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009d90 tx=0x7f91e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 shutdown_connections 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f91cc06b500 0x7f91cc06d9c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f91f0161b50 0x7f91f00cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00d4a00 0x7f91f00caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 --2- 172.21.15.97:0/1653661722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91f00b5490 0x7f91f015e880 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.404+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 >> 172.21.15.97:0/1653661722 conn(0x7f91f00d3e70 msgr2=0x7f91f00c8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:25.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.405+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 shutdown_connections 2024-03-06T07:37:25.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.405+0000 7f91f7000640 1 -- 172.21.15.97:0/1653661722 wait complete. 2024-03-06T07:37:25.428 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:37:25.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:25.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.638+0000 7f7b6a400640 1 Processor -- start 2024-03-06T07:37:25.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.638+0000 7f7b6a400640 1 -- start start 2024-03-06T07:37:25.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b6a400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b64161580 0x7f7b640d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b6a400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7b640d1e90 0x7f7b640d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b6a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b6a400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b64161960 con 0x7f7b640d1e90 2024-03-06T07:37:25.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b6a400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b640cf6f0 con 0x7f7b64161580 2024-03-06T07:37:25.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b6a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640cf9d0 con 0x7f7b640d28a0 2024-03-06T07:37:25.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7b640d1e90 0x7f7b640d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7b640d1e90 0x7f7b640d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38242/0 (socket says 172.21.15.97:38242) 2024-03-06T07:37:25.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 -- 172.21.15.97:0/525986896 learned_addr learned my addr 172.21.15.97:0/525986896 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:25.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b69000640 1 --2- 172.21.15.97:0/525986896 >> v2:172.21.15.135:3300/0 conn(0x7f7b64161580 0x7f7b640d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b69a00640 1 --2- 172.21.15.97:0/525986896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 -- 172.21.15.97:0/525986896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 msgr2=0x7f7b640d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 --2- 172.21.15.97:0/525986896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 -- 172.21.15.97:0/525986896 >> v2:172.21.15.135:3300/0 conn(0x7f7b64161580 msgr2=0x7f7b640d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 --2- 172.21.15.97:0/525986896 >> v2:172.21.15.135:3300/0 conn(0x7f7b64161580 0x7f7b640d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.640+0000 7f7b63e00640 1 -- 172.21.15.97:0/525986896 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b640cfcb0 con 0x7f7b640d1e90 2024-03-06T07:37:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.642+0000 7f7b63e00640 1 --2- 172.21.15.97:0/525986896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b640d1e90 0x7f7b640d2360 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f7b64073740 tx=0x7f7b5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3649f7118ad01a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.642+0000 7f7b63400640 1 -- 172.21.15.97:0/525986896 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50035020 con 0x7f7b640d1e90 2024-03-06T07:37:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.642+0000 7f7b63400640 1 -- 172.21.15.97:0/525986896 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b50031870 con 0x7f7b640d1e90 2024-03-06T07:37:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.642+0000 7f7b63400640 1 -- 172.21.15.97:0/525986896 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50031c40 con 0x7f7b640d1e90 2024-03-06T07:37:25.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.642+0000 7f7b6a400640 1 -- 172.21.15.97:0/525986896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b640d1e90 msgr2=0x7f7b640d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.642+0000 7f7b6a400640 1 --2- 172.21.15.97:0/525986896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b640d1e90 0x7f7b640d2360 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f7b64073740 tx=0x7f7b5002b910 comp rx=0 tx=0).stop 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 -- 172.21.15.97:0/525986896 shutdown_connections 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 --2- 172.21.15.97:0/525986896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 --2- 172.21.15.97:0/525986896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b640d1e90 0x7f7b640d2360 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 --2- 172.21.15.97:0/525986896 >> v2:172.21.15.135:3300/0 conn(0x7f7b64161580 0x7f7b640d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 -- 172.21.15.97:0/525986896 >> 172.21.15.97:0/525986896 conn(0x7f7b640d3d80 msgr2=0x7f7b640d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 -- 172.21.15.97:0/525986896 shutdown_connections 2024-03-06T07:37:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.643+0000 7f7b6a400640 1 -- 172.21.15.97:0/525986896 wait complete. 2024-03-06T07:37:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.644+0000 7f7b6a400640 1 Processor -- start 2024-03-06T07:37:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 -- start start 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b640d1e90 0x7f7b641582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 0x7f7b640b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640cf3a0 con 0x7f7b64161580 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640b39a0 con 0x7f7b640d1e90 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.645+0000 7f7b6a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640d0c20 con 0x7f7b640d28a0 2024-03-06T07:37:25.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 0x7f7b640b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 0x7f7b640b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38246/0 (socket says 172.21.15.97:38246) 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 -- 172.21.15.97:0/953420311 learned_addr learned my addr 172.21.15.97:0/953420311 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69000640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b640d1e90 0x7f7b641582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b63e00640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 -- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 msgr2=0x7f7b640b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 -- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b640d1e90 msgr2=0x7f7b641582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b640d1e90 0x7f7b641582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.646+0000 7f7b69a00640 1 -- 172.21.15.97:0/953420311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6415bf90 con 0x7f7b64161580 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.647+0000 7f7b69a00640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 0x7f7b640b47f0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00cd70 tx=0x7f7b5c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.647+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c003b90 con 0x7f7b64161580 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.647+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c003d30 con 0x7f7b64161580 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.647+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b640cbc30 con 0x7f7b64161580 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.647+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c004240 con 0x7f7b64161580 2024-03-06T07:37:25.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.647+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b640cc540 con 0x7f7b64161580 2024-03-06T07:37:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.648+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6404f330 con 0x7f7b64161580 2024-03-06T07:37:25.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.649+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b5c0043e0 con 0x7f7b64161580 2024-03-06T07:37:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.651+0000 7f7b60c00640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b4406b4d0 0x7f7b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.651+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b5c08acd0 con 0x7f7b64161580 2024-03-06T07:37:25.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.651+0000 7f7b69000640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b4406b4d0 0x7f7b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.652+0000 7f7b69000640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b4406b4d0 0x7f7b4406d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7b58006fd0 tx=0x7f7b58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:25.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.654+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b5c059b40 con 0x7f7b64161580 2024-03-06T07:37:25.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.890+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b640d3b60 con 0x7f7b64161580 2024-03-06T07:37:25.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.892+0000 7f7b60c00640 1 -- 172.21.15.97:0/953420311 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4990 (secure 0 0 0) 0x7f7b5c059440 con 0x7f7b64161580 2024-03-06T07:37:25.894 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:25.894 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5840,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2881704649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":2881704649},{"type":"v1","addr":"172.21.15.135:6835","nonce":2881704649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6448,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/281989301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":281989301},{"type":"v1","addr":"172.21.15.135:6834","nonce":281989301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:24.903864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9024},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:37:25.895 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 107 2024-03-06T07:37:25.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b4406b4d0 msgr2=0x7f7b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b4406b4d0 0x7f7b4406d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7b58006fd0 tx=0x7f7b58008040 comp rx=0 tx=0).stop 2024-03-06T07:37:25.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 msgr2=0x7f7b640b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 0x7f7b640b47f0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00cd70 tx=0x7f7b5c00a660 comp rx=0 tx=0).stop 2024-03-06T07:37:25.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 shutdown_connections 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7b4406b4d0 0x7f7b4406d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7b64161580 0x7f7b640b47f0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7b640d28a0 0x7f7b640b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 --2- 172.21.15.97:0/953420311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b640d1e90 0x7f7b641582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 >> 172.21.15.97:0/953420311 conn(0x7f7b640d3d80 msgr2=0x7f7b6406cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 shutdown_connections 2024-03-06T07:37:25.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:25.895+0000 7f7b6a400640 1 -- 172.21.15.97:0/953420311 wait complete. 2024-03-06T07:37:25.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:26.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.166+0000 7fcd05000640 1 Processor -- start 2024-03-06T07:37:26.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.167+0000 7fcd05000640 1 -- start start 2024-03-06T07:37:26.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fcd05000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcd001623d0 0x7fcd000ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fcd05000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 0x7fcd000b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fcd05000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd000af300 con 0x7fcd000b3a60 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fcd05000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd000aed40 con 0x7fcd001623d0 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fcd05000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000af020 con 0x7fcd000d0140 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fccffe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcd000b3a60 0x7fcd000ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49354/0 (socket says 172.21.15.97:49354) 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.168+0000 7fccffe00640 1 -- 172.21.15.97:0/3920107908 learned_addr learned my addr 172.21.15.97:0/3920107908 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccff400640 1 --2- 172.21.15.97:0/3920107908 >> v2:172.21.15.135:3300/0 conn(0x7fcd001623d0 0x7fcd000ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccf6a00640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 0x7fcd000b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025036299 0 0) 0x7fcd000af300 con 0x7fcd000b3a60 2024-03-06T07:37:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce4003520 con 0x7fcd000b3a60 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108995737 0 0) 0x7fcce4003520 con 0x7fcd000b3a60 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 msgr2=0x7fcd000b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 0x7fcd000b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 >> v2:172.21.15.135:3300/0 conn(0x7fcd001623d0 msgr2=0x7fcd000ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 --2- 172.21.15.97:0/3920107908 >> v2:172.21.15.135:3300/0 conn(0x7fcd001623d0 0x7fcd000ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0006d2e0 con 0x7fcd000b3a60 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fccf80019f0 con 0x7fcd000b3a60 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.169+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 >> v1:172.21.15.97:6789/0 conn(0x7fcd000b3a60 legacy=0x7fcd000ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccfea00640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcce40039b0 0x7fcce4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccfea00640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce4004800 0x7fcce4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccfea00640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 0x7fcce40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000aed40 con 0x7fcce40039b0 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000af020 con 0x7fcce4004800 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400ba10 con 0x7fcd000d0140 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccff400640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcce40039b0 0x7fcce4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccff400640 1 -- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 msgr2=0x7fcce40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccff400640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd000d0140 0x7fcce40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccff400640 1 -- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce4004800 msgr2=0x7fcce4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccff400640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce4004800 0x7fcce4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.170+0000 7fccff400640 1 -- 172.21.15.97:0/3920107908 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4003520 con 0x7fcce40039b0 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.171+0000 7fccff400640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcce40039b0 0x7fcce4003dd0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009af0 tx=0x7fcce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342dd61209cbd006 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0039070 con 0x7fcce40039b0 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf002c710 con 0x7fcce40039b0 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fccfea00640 1 -- 172.21.15.97:0/3920107908 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0009690 con 0x7fcce40039b0 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fcd05000640 1 -- 172.21.15.97:0/3920107908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcce40039b0 msgr2=0x7fcce4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fcd05000640 1 --2- 172.21.15.97:0/3920107908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcce40039b0 0x7fcce4003dd0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009af0 tx=0x7fcce4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fccffe00640 1 -- 172.21.15.97:0/3920107908 reap_dead start 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fcd05000640 1 -- 172.21.15.97:0/3920107908 shutdown_connections 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fcd05000640 1 -- 172.21.15.97:0/3920107908 >> 172.21.15.97:0/3920107908 conn(0x7fcd0006afc0 msgr2=0x7fcd0006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fcd05000640 1 -- 172.21.15.97:0/3920107908 shutdown_connections 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.172+0000 7fcd05000640 1 -- 172.21.15.97:0/3920107908 wait complete. 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.174+0000 7fcd05000640 1 Processor -- start 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.174+0000 7fcd05000640 1 -- start start 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fcd05000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce40039b0 0x7fcd001631c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fcd05000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd00164c30 0x7fcd0015feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fcd05000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 0x7fcd0015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fcd05000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000ce940 con 0x7fcd0015e220 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fcd05000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000d05a0 con 0x7fcce40039b0 2024-03-06T07:37:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fcd05000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000ceac0 con 0x7fcd00164c30 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fccffe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 0x7fcd0015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fccffe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 0x7fcd0015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38264/0 (socket says 172.21.15.97:38264) 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.175+0000 7fccffe00640 1 -- 172.21.15.97:0/3809534027 learned_addr learned my addr 172.21.15.97:0/3809534027 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccff400640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce40039b0 0x7fcd001631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccf6a00640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd00164c30 0x7fcd0015feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccffe00640 1 -- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd00164c30 msgr2=0x7fcd0015feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccffe00640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd00164c30 0x7fcd0015feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccffe00640 1 -- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce40039b0 msgr2=0x7fcd001631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccffe00640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce40039b0 0x7fcd001631c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccffe00640 1 -- 172.21.15.97:0/3809534027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000a9cd0 con 0x7fcd0015e220 2024-03-06T07:37:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccf6a00640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd00164c30 0x7fcd0015feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:26.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccffe00640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 0x7fcd0015e680 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fcd000cdaf0 tx=0x7fccf800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf800a610 con 0x7fcd0015e220 2024-03-06T07:37:26.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf800a7d0 con 0x7fcd0015e220 2024-03-06T07:37:26.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd00163930 con 0x7fcd0015e220 2024-03-06T07:37:26.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.176+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf800abc0 con 0x7fcd0015e220 2024-03-06T07:37:26.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.177+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd00163ef0 con 0x7fcd0015e220 2024-03-06T07:37:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.178+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0004f330 con 0x7fcd0015e220 2024-03-06T07:37:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.179+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccf801c3f0 con 0x7fcd0015e220 2024-03-06T07:37:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.179+0000 7fccf7e00640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccdc06b500 0x7fccdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.180+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccf8089ac0 con 0x7fcd0015e220 2024-03-06T07:37:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.180+0000 7fccff400640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccdc06b500 0x7fccdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.181+0000 7fccff400640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccdc06b500 0x7fccdc06d9c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fccf0036000 tx=0x7fccf0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.184+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccf80588a0 con 0x7fcd0015e220 2024-03-06T07:37:26.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.380+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0006d020 con 0x7fcd0015e220 2024-03-06T07:37:26.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.383+0000 7fccf7e00640 1 -- 172.21.15.97:0/3809534027 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16667 (secure 0 0 0) 0x7fccf80581a0 con 0x7fcd0015e220 2024-03-06T07:37:26.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:26.386 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":144,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:23.377199+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.385+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccdc06b500 msgr2=0x7fccdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.385+0000 7fcd05000640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccdc06b500 0x7fccdc06d9c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fccf0036000 tx=0x7fccf0030000 comp rx=0 tx=0).stop 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.385+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 msgr2=0x7fcd0015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.385+0000 7fcd05000640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 0x7fcd0015e680 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fcd000cdaf0 tx=0x7fccf800cc10 comp rx=0 tx=0).stop 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 shutdown_connections 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fccdc06b500 0x7fccdc06d9c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0015e220 0x7fcd0015e680 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcd00164c30 0x7fcd0015feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 --2- 172.21.15.97:0/3809534027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcce40039b0 0x7fcd001631c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 >> 172.21.15.97:0/3809534027 conn(0x7fcd0006afc0 msgr2=0x7fcd000afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 shutdown_connections 2024-03-06T07:37:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.386+0000 7fcd05000640 1 -- 172.21.15.97:0/3809534027 wait complete. 2024-03-06T07:37:26.408 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:37:26.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.603+0000 7f89dc400640 1 Processor -- start 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.603+0000 7f89dc400640 1 -- start start 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.604+0000 7f89dc400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f89d415f9d0 0x7f89d415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89dc400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f89d4070b10 0x7f89d4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89dc400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d406cef0 con 0x7f89d4144c00 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89dc400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d406d1d0 con 0x7f89d415f9d0 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89dc400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d406d4b0 con 0x7f89d4070b10 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f89d415f9d0 0x7f89d415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f89d415f9d0 0x7f89d415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38280/0 (socket says 172.21.15.97:38280) 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f89d4144c00 0x7f89d406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49358/0 (socket says 172.21.15.97:49358) 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9000640 1 -- 172.21.15.97:0/2585699183 learned_addr learned my addr 172.21.15.97:0/2585699183 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89da400640 1 --2- 172.21.15.97:0/2585699183 >> v2:172.21.15.135:3300/0 conn(0x7f89d4070b10 0x7f89d4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9000640 1 -- 172.21.15.97:0/2585699183 >> v2:172.21.15.135:3300/0 conn(0x7f89d4070b10 msgr2=0x7f89d4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9000640 1 --2- 172.21.15.97:0/2585699183 >> v2:172.21.15.135:3300/0 conn(0x7f89d4070b10 0x7f89d4070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.605+0000 7f89d9000640 1 -- 172.21.15.97:0/2585699183 >> v1:172.21.15.97:6789/0 conn(0x7f89d4144c00 legacy=0x7f89d406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.606+0000 7f89d9000640 1 -- 172.21.15.97:0/2585699183 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d406d790 con 0x7f89d415f9d0 2024-03-06T07:37:26.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.606+0000 7f89d3e00640 1 -- 172.21.15.97:0/2585699183 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504627496 0 0) 0x7f89d406cef0 con 0x7f89d4144c00 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.607+0000 7f89d9000640 1 --2- 172.21.15.97:0/2585699183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d415f9d0 0x7f89d415fe60 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f89d406e500 tx=0x7f89c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af8ae048cf1054e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.607+0000 7f89d3e00640 1 -- 172.21.15.97:0/2585699183 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0034070 con 0x7f89d415f9d0 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.607+0000 7f89d3e00640 1 -- 172.21.15.97:0/2585699183 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c0030860 con 0x7f89d415f9d0 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.607+0000 7f89d3e00640 1 -- 172.21.15.97:0/2585699183 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0030c30 con 0x7f89d415f9d0 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 -- 172.21.15.97:0/2585699183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d415f9d0 msgr2=0x7f89d415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 --2- 172.21.15.97:0/2585699183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d415f9d0 0x7f89d415fe60 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f89d406e500 tx=0x7f89c002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 -- 172.21.15.97:0/2585699183 shutdown_connections 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 --2- 172.21.15.97:0/2585699183 >> v2:172.21.15.135:3300/0 conn(0x7f89d4070b10 0x7f89d4070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 --2- 172.21.15.97:0/2585699183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d415f9d0 0x7f89d415fe60 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 -- 172.21.15.97:0/2585699183 >> 172.21.15.97:0/2585699183 conn(0x7f89d40b1730 msgr2=0x7f89d40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 -- 172.21.15.97:0/2585699183 shutdown_connections 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.608+0000 7f89dc400640 1 -- 172.21.15.97:0/2585699183 wait complete. 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.609+0000 7f89dc400640 1 Processor -- start 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.610+0000 7f89dc400640 1 -- start start 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.610+0000 7f89dc400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d4070b10 0x7f89d415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.610+0000 7f89dc400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 0x7f89d40a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.610+0000 7f89dc400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89d415f9d0 0x7f89d413db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89dc400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d406ccd0 con 0x7f89d4144c00 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89dc400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d406cb50 con 0x7f89d415f9d0 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89dc400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d406df90 con 0x7f89d4070b10 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 0x7f89d40a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 0x7f89d40a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38286/0 (socket says 172.21.15.97:38286) 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 -- 172.21.15.97:0/483842263 learned_addr learned my addr 172.21.15.97:0/483842263 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9a00640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d4070b10 0x7f89d415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 -- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d4070b10 msgr2=0x7f89d415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89da400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89d415f9d0 0x7f89d413db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d4070b10 0x7f89d415b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 -- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89d415f9d0 msgr2=0x7f89d413db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89d415f9d0 0x7f89d413db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.611+0000 7f89d9000640 1 -- 172.21.15.97:0/483842263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d4071bd0 con 0x7f89d4144c00 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.612+0000 7f89d9000640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 0x7f89d40a9450 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f89c0000c00 tx=0x7f89c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.612+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0034040 con 0x7f89d4144c00 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.612+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c003ab90 con 0x7f89d4144c00 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.612+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0004030 con 0x7f89d4144c00 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.612+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d413d580 con 0x7f89d4144c00 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.612+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d413c670 con 0x7f89d4144c00 2024-03-06T07:37:26.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.613+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d404f330 con 0x7f89d4144c00 2024-03-06T07:37:26.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.615+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89c0037070 con 0x7f89d4144c00 2024-03-06T07:37:26.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.616+0000 7f89d1600640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89b406b4d0 0x7f89b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.616+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89c00ac1d0 con 0x7f89d4144c00 2024-03-06T07:37:26.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.616+0000 7f89d9a00640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89b406b4d0 0x7f89b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.617+0000 7f89d9a00640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89b406b4d0 0x7f89b406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f89c8007fd0 tx=0x7f89c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.620+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89c003f070 con 0x7f89d4144c00 2024-03-06T07:37:26.757 INFO:tasks.ceph.mds.c.smithi097.stderr:2024-03-06T07:37:26.755+0000 7f0c4a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:37:26.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.848+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89d4072890 con 0x7f89b406b4d0 2024-03-06T07:37:26.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.858+0000 7f89d1600640 1 -- 172.21.15.97:0/483842263 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f89d4072890 con 0x7f89b406b4d0 2024-03-06T07:37:26.864 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:26.865 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:37:26.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89b406b4d0 msgr2=0x7f89b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89b406b4d0 0x7f89b406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f89c8007fd0 tx=0x7f89c8009040 comp rx=0 tx=0).stop 2024-03-06T07:37:26.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 msgr2=0x7f89d40a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 0x7f89d40a9450 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f89c0000c00 tx=0x7f89c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 shutdown_connections 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f89b406b4d0 0x7f89b406d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89d415f9d0 0x7f89d413db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89d4144c00 0x7f89d40a9450 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 --2- 172.21.15.97:0/483842263 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f89d4070b10 0x7f89d415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.863+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 >> 172.21.15.97:0/483842263 conn(0x7f89d40b1730 msgr2=0x7f89d40cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.864+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 shutdown_connections 2024-03-06T07:37:26.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:26.864+0000 7f89dc400640 1 -- 172.21.15.97:0/483842263 wait complete. 2024-03-06T07:37:26.890 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-03-06T07:37:25.084404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310444,"kb_used_data":15224,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664276,"statfs":{"total":773094113280,"available":772776218624,"internally_reserved":0,"allocated":15589376,"data_stored":7451372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.752196"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400157+0000","last_change":"2024-03-06T07:37:22.383332+0000","last_active":"2024-03-06T07:37:23.400157+0000","last_peered":"2024-03-06T07:37:23.400157+0000","last_clean":"2024-03-06T07:37:23.400157+0000","last_became_active":"2024-03-06T07:37:22.382414+0000","last_became_peered":"2024-03-06T07:37:22.382414+0000","last_unstale":"2024-03-06T07:37:23.400157+0000","last_undegraded":"2024-03-06T07:37:23.400157+0000","last_fullsized":"2024-03-06T07:37:23.400157+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:40.838978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462385+0000","last_change":"2024-03-06T07:37:22.400356+0000","last_active":"2024-03-06T07:37:23.462385+0000","last_peered":"2024-03-06T07:37:23.462385+0000","last_clean":"2024-03-06T07:37:23.462385+0000","last_became_active":"2024-03-06T07:37:22.400055+0000","last_became_peered":"2024-03-06T07:37:22.400055+0000","last_unstale":"2024-03-06T07:37:23.462385+0000","last_undegraded":"2024-03-06T07:37:23.462385+0000","last_fullsized":"2024-03-06T07:37:23.462385+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:46.584284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464352+0000","last_change":"2024-03-06T07:37:22.395370+0000","last_active":"2024-03-06T07:37:23.464352+0000","last_peered":"2024-03-06T07:37:23.464352+0000","last_clean":"2024-03-06T07:37:23.464352+0000","last_became_active":"2024-03-06T07:37:22.394936+0000","last_became_peered":"2024-03-06T07:37:22.394936+0000","last_unstale":"2024-03-06T07:37:23.464352+0000","last_undegraded":"2024-03-06T07:37:23.464352+0000","last_fullsized":"2024-03-06T07:37:23.464352+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:55.000031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398992+0000","last_change":"2024-03-06T07:37:22.399940+0000","last_active":"2024-03-06T07:37:23.398992+0000","last_peered":"2024-03-06T07:37:23.398992+0000","last_clean":"2024-03-06T07:37:23.398992+0000","last_became_active":"2024-03-06T07:37:22.399611+0000","last_became_peered":"2024-03-06T07:37:22.399611+0000","last_unstale":"2024-03-06T07:37:23.398992+0000","last_undegraded":"2024-03-06T07:37:23.398992+0000","last_fullsized":"2024-03-06T07:37:23.398992+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:46.193975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398275+0000","last_change":"2024-03-06T07:37:22.397393+0000","last_active":"2024-03-06T07:37:23.398275+0000","last_peered":"2024-03-06T07:37:23.398275+0000","last_clean":"2024-03-06T07:37:23.398275+0000","last_became_active":"2024-03-06T07:37:22.397108+0000","last_became_peered":"2024-03-06T07:37:22.397108+0000","last_unstale":"2024-03-06T07:37:23.398275+0000","last_undegraded":"2024-03-06T07:37:23.398275+0000","last_fullsized":"2024-03-06T07:37:23.398275+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:51.600779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399474+0000","last_change":"2024-03-06T07:37:22.394968+0000","last_active":"2024-03-06T07:37:23.399474+0000","last_peered":"2024-03-06T07:37:23.399474+0000","last_clean":"2024-03-06T07:37:23.399474+0000","last_became_active":"2024-03-06T07:37:22.394621+0000","last_became_peered":"2024-03-06T07:37:22.394621+0000","last_unstale":"2024-03-06T07:37:23.399474+0000","last_undegraded":"2024-03-06T07:37:23.399474+0000","last_fullsized":"2024-03-06T07:37:23.399474+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:01.705472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394571+0000","last_change":"2024-03-06T07:37:22.410318+0000","last_active":"2024-03-06T07:37:23.394571+0000","last_peered":"2024-03-06T07:37:23.394571+0000","last_clean":"2024-03-06T07:37:23.394571+0000","last_became_active":"2024-03-06T07:37:22.409584+0000","last_became_peered":"2024-03-06T07:37:22.409584+0000","last_unstale":"2024-03-06T07:37:23.394571+0000","last_undegraded":"2024-03-06T07:37:23.394571+0000","last_fullsized":"2024-03-06T07:37:23.394571+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:11.979155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399026+0000","last_change":"2024-03-06T07:37:22.391801+0000","last_active":"2024-03-06T07:37:23.399026+0000","last_peered":"2024-03-06T07:37:23.399026+0000","last_clean":"2024-03-06T07:37:23.399026+0000","last_became_active":"2024-03-06T07:37:22.391522+0000","last_became_peered":"2024-03-06T07:37:22.391522+0000","last_unstale":"2024-03-06T07:37:23.399026+0000","last_undegraded":"2024-03-06T07:37:23.399026+0000","last_fullsized":"2024-03-06T07:37:23.399026+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:47.121952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462791+0000","last_change":"2024-03-06T07:37:22.386445+0000","last_active":"2024-03-06T07:37:23.462791+0000","last_peered":"2024-03-06T07:37:23.462791+0000","last_clean":"2024-03-06T07:37:23.462791+0000","last_became_active":"2024-03-06T07:37:22.384570+0000","last_became_peered":"2024-03-06T07:37:22.384570+0000","last_unstale":"2024-03-06T07:37:23.462791+0000","last_undegraded":"2024-03-06T07:37:23.462791+0000","last_fullsized":"2024-03-06T07:37:23.462791+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.954146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405797+0000","last_change":"2024-03-06T07:37:22.403983+0000","last_active":"2024-03-06T07:37:23.405797+0000","last_peered":"2024-03-06T07:37:23.405797+0000","last_clean":"2024-03-06T07:37:23.405797+0000","last_became_active":"2024-03-06T07:37:22.403283+0000","last_became_peered":"2024-03-06T07:37:22.403283+0000","last_unstale":"2024-03-06T07:37:23.405797+0000","last_undegraded":"2024-03-06T07:37:23.405797+0000","last_fullsized":"2024-03-06T07:37:23.405797+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:32.623856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399440+0000","last_change":"2024-03-06T07:37:22.390213+0000","last_active":"2024-03-06T07:37:23.399440+0000","last_peered":"2024-03-06T07:37:23.399440+0000","last_clean":"2024-03-06T07:37:23.399440+0000","last_became_active":"2024-03-06T07:37:22.389709+0000","last_became_peered":"2024-03-06T07:37:22.389709+0000","last_unstale":"2024-03-06T07:37:23.399440+0000","last_undegraded":"2024-03-06T07:37:23.399440+0000","last_fullsized":"2024-03-06T07:37:23.399440+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:52.221971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405134+0000","last_change":"2024-03-06T07:37:22.392961+0000","last_active":"2024-03-06T07:37:23.405134+0000","last_peered":"2024-03-06T07:37:23.405134+0000","last_clean":"2024-03-06T07:37:23.405134+0000","last_became_active":"2024-03-06T07:37:22.392507+0000","last_became_peered":"2024-03-06T07:37:22.392507+0000","last_unstale":"2024-03-06T07:37:23.405134+0000","last_undegraded":"2024-03-06T07:37:23.405134+0000","last_fullsized":"2024-03-06T07:37:23.405134+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:17.676813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398658+0000","last_change":"2024-03-06T07:37:22.397145+0000","last_active":"2024-03-06T07:37:23.398658+0000","last_peered":"2024-03-06T07:37:23.398658+0000","last_clean":"2024-03-06T07:37:23.398658+0000","last_became_active":"2024-03-06T07:37:22.396761+0000","last_became_peered":"2024-03-06T07:37:22.396761+0000","last_unstale":"2024-03-06T07:37:23.398658+0000","last_undegraded":"2024-03-06T07:37:23.398658+0000","last_fullsized":"2024-03-06T07:37:23.398658+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:11.759303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397445+0000","last_change":"2024-03-06T07:37:22.409468+0000","last_active":"2024-03-06T07:37:23.397445+0000","last_peered":"2024-03-06T07:37:23.397445+0000","last_clean":"2024-03-06T07:37:23.397445+0000","last_became_active":"2024-03-06T07:37:22.408685+0000","last_became_peered":"2024-03-06T07:37:22.408685+0000","last_unstale":"2024-03-06T07:37:23.397445+0000","last_undegraded":"2024-03-06T07:37:23.397445+0000","last_fullsized":"2024-03-06T07:37:23.397445+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:00.774528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.404679+0000","last_change":"2024-03-06T07:37:22.389604+0000","last_active":"2024-03-06T07:37:23.404679+0000","last_peered":"2024-03-06T07:37:23.404679+0000","last_clean":"2024-03-06T07:37:23.404679+0000","last_became_active":"2024-03-06T07:37:22.389135+0000","last_became_peered":"2024-03-06T07:37:22.389135+0000","last_unstale":"2024-03-06T07:37:23.404679+0000","last_undegraded":"2024-03-06T07:37:23.404679+0000","last_fullsized":"2024-03-06T07:37:23.404679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:58.167076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460346+0000","last_change":"2024-03-06T07:37:22.384427+0000","last_active":"2024-03-06T07:37:23.460346+0000","last_peered":"2024-03-06T07:37:23.460346+0000","last_clean":"2024-03-06T07:37:23.460346+0000","last_became_active":"2024-03-06T07:37:22.384148+0000","last_became_peered":"2024-03-06T07:37:22.384148+0000","last_unstale":"2024-03-06T07:37:23.460346+0000","last_undegraded":"2024-03-06T07:37:23.460346+0000","last_fullsized":"2024-03-06T07:37:23.460346+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:21.865440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406118+0000","last_change":"2024-03-06T07:37:22.402878+0000","last_active":"2024-03-06T07:37:23.406118+0000","last_peered":"2024-03-06T07:37:23.406118+0000","last_clean":"2024-03-06T07:37:23.406118+0000","last_became_active":"2024-03-06T07:37:22.397630+0000","last_became_peered":"2024-03-06T07:37:22.397630+0000","last_unstale":"2024-03-06T07:37:23.406118+0000","last_undegraded":"2024-03-06T07:37:23.406118+0000","last_fullsized":"2024-03-06T07:37:23.406118+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:48.107097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464532+0000","last_change":"2024-03-06T07:37:22.404191+0000","last_active":"2024-03-06T07:37:23.464532+0000","last_peered":"2024-03-06T07:37:23.464532+0000","last_clean":"2024-03-06T07:37:23.464532+0000","last_became_active":"2024-03-06T07:37:22.403870+0000","last_became_peered":"2024-03-06T07:37:22.403870+0000","last_unstale":"2024-03-06T07:37:23.464532+0000","last_undegraded":"2024-03-06T07:37:23.464532+0000","last_fullsized":"2024-03-06T07:37:23.464532+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:43.656249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399922+0000","last_change":"2024-03-06T07:37:22.386594+0000","last_active":"2024-03-06T07:37:23.399922+0000","last_peered":"2024-03-06T07:37:23.399922+0000","last_clean":"2024-03-06T07:37:23.399922+0000","last_became_active":"2024-03-06T07:37:22.386176+0000","last_became_peered":"2024-03-06T07:37:22.386176+0000","last_unstale":"2024-03-06T07:37:23.399922+0000","last_undegraded":"2024-03-06T07:37:23.399922+0000","last_fullsized":"2024-03-06T07:37:23.399922+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:58.164860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460929+0000","last_change":"2024-03-06T07:37:22.393011+0000","last_active":"2024-03-06T07:37:23.460929+0000","last_peered":"2024-03-06T07:37:23.460929+0000","last_clean":"2024-03-06T07:37:23.460929+0000","last_became_active":"2024-03-06T07:37:22.386815+0000","last_became_peered":"2024-03-06T07:37:22.386815+0000","last_unstale":"2024-03-06T07:37:23.460929+0000","last_undegraded":"2024-03-06T07:37:23.460929+0000","last_fullsized":"2024-03-06T07:37:23.460929+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:23.464868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405456+0000","last_change":"2024-03-06T07:37:22.409729+0000","last_active":"2024-03-06T07:37:23.405456+0000","last_peered":"2024-03-06T07:37:23.405456+0000","last_clean":"2024-03-06T07:37:23.405456+0000","last_became_active":"2024-03-06T07:37:22.409269+0000","last_became_peered":"2024-03-06T07:37:22.409269+0000","last_unstale":"2024-03-06T07:37:23.405456+0000","last_undegraded":"2024-03-06T07:37:23.405456+0000","last_fullsized":"2024-03-06T07:37:23.405456+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:36:26.241249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397630+0000","last_change":"2024-03-06T07:37:22.404312+0000","last_active":"2024-03-06T07:37:23.397630+0000","last_peered":"2024-03-06T07:37:23.397630+0000","last_clean":"2024-03-06T07:37:23.397630+0000","last_became_active":"2024-03-06T07:37:22.403787+0000","last_became_peered":"2024-03-06T07:37:22.403787+0000","last_unstale":"2024-03-06T07:37:23.397630+0000","last_undegraded":"2024-03-06T07:37:23.397630+0000","last_fullsized":"2024-03-06T07:37:23.397630+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:45.033772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398401+0000","last_change":"2024-03-06T07:37:22.408249+0000","last_active":"2024-03-06T07:37:23.398401+0000","last_peered":"2024-03-06T07:37:23.398401+0000","last_clean":"2024-03-06T07:37:23.398401+0000","last_became_active":"2024-03-06T07:37:22.407921+0000","last_became_peered":"2024-03-06T07:37:22.407921+0000","last_unstale":"2024-03-06T07:37:23.398401+0000","last_undegraded":"2024-03-06T07:37:23.398401+0000","last_fullsized":"2024-03-06T07:37:23.398401+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:42.436707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.404953+0000","last_change":"2024-03-06T07:37:22.396679+0000","last_active":"2024-03-06T07:37:23.404953+0000","last_peered":"2024-03-06T07:37:23.404953+0000","last_clean":"2024-03-06T07:37:23.404953+0000","last_became_active":"2024-03-06T07:37:22.392969+0000","last_became_peered":"2024-03-06T07:37:22.392969+0000","last_unstale":"2024-03-06T07:37:23.404953+0000","last_undegraded":"2024-03-06T07:37:23.404953+0000","last_fullsized":"2024-03-06T07:37:23.404953+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:29.190943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460954+0000","last_change":"2024-03-06T07:37:22.390025+0000","last_active":"2024-03-06T07:37:23.460954+0000","last_peered":"2024-03-06T07:37:23.460954+0000","last_clean":"2024-03-06T07:37:23.460954+0000","last_became_active":"2024-03-06T07:37:22.389757+0000","last_became_peered":"2024-03-06T07:37:22.389757+0000","last_unstale":"2024-03-06T07:37:23.460954+0000","last_undegraded":"2024-03-06T07:37:23.460954+0000","last_fullsized":"2024-03-06T07:37:23.460954+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:51.600779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400085+0000","last_change":"2024-03-06T07:37:22.386833+0000","last_active":"2024-03-06T07:37:23.400085+0000","last_peered":"2024-03-06T07:37:23.400085+0000","last_clean":"2024-03-06T07:37:23.400085+0000","last_became_active":"2024-03-06T07:37:22.385573+0000","last_became_peered":"2024-03-06T07:37:22.385573+0000","last_unstale":"2024-03-06T07:37:23.400085+0000","last_undegraded":"2024-03-06T07:37:23.400085+0000","last_fullsized":"2024-03-06T07:37:23.400085+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:16.796463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461027+0000","last_change":"2024-03-06T07:37:22.394540+0000","last_active":"2024-03-06T07:37:23.461027+0000","last_peered":"2024-03-06T07:37:23.461027+0000","last_clean":"2024-03-06T07:37:23.461027+0000","last_became_active":"2024-03-06T07:37:22.394242+0000","last_became_peered":"2024-03-06T07:37:22.394242+0000","last_unstale":"2024-03-06T07:37:23.461027+0000","last_undegraded":"2024-03-06T07:37:23.461027+0000","last_fullsized":"2024-03-06T07:37:23.461027+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.331051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394425+0000","last_change":"2024-03-06T07:37:22.396145+0000","last_active":"2024-03-06T07:37:23.394425+0000","last_peered":"2024-03-06T07:37:23.394425+0000","last_clean":"2024-03-06T07:37:23.394425+0000","last_became_active":"2024-03-06T07:37:22.395856+0000","last_became_peered":"2024-03-06T07:37:22.395856+0000","last_unstale":"2024-03-06T07:37:23.394425+0000","last_undegraded":"2024-03-06T07:37:23.394425+0000","last_fullsized":"2024-03-06T07:37:23.394425+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:27.224804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399112+0000","last_change":"2024-03-06T07:37:22.395992+0000","last_active":"2024-03-06T07:37:23.399112+0000","last_peered":"2024-03-06T07:37:23.399112+0000","last_clean":"2024-03-06T07:37:23.399112+0000","last_became_active":"2024-03-06T07:37:22.395649+0000","last_became_peered":"2024-03-06T07:37:22.395649+0000","last_unstale":"2024-03-06T07:37:23.399112+0000","last_undegraded":"2024-03-06T07:37:23.399112+0000","last_fullsized":"2024-03-06T07:37:23.399112+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:26.141228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397848+0000","last_change":"2024-03-06T07:37:22.399174+0000","last_active":"2024-03-06T07:37:23.397848+0000","last_peered":"2024-03-06T07:37:23.397848+0000","last_clean":"2024-03-06T07:37:23.397848+0000","last_became_active":"2024-03-06T07:37:22.397414+0000","last_became_peered":"2024-03-06T07:37:22.397414+0000","last_unstale":"2024-03-06T07:37:23.397848+0000","last_undegraded":"2024-03-06T07:37:23.397848+0000","last_fullsized":"2024-03-06T07:37:23.397848+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:16.801579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461873+0000","last_change":"2024-03-06T07:37:22.398035+0000","last_active":"2024-03-06T07:37:23.461873+0000","last_peered":"2024-03-06T07:37:23.461873+0000","last_clean":"2024-03-06T07:37:23.461873+0000","last_became_active":"2024-03-06T07:37:22.397812+0000","last_became_peered":"2024-03-06T07:37:22.397812+0000","last_unstale":"2024-03-06T07:37:23.461873+0000","last_undegraded":"2024-03-06T07:37:23.461873+0000","last_fullsized":"2024-03-06T07:37:23.461873+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:52.355345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460439+0000","last_change":"2024-03-06T07:37:22.399262+0000","last_active":"2024-03-06T07:37:23.460439+0000","last_peered":"2024-03-06T07:37:23.460439+0000","last_clean":"2024-03-06T07:37:23.460439+0000","last_became_active":"2024-03-06T07:37:22.397780+0000","last_became_peered":"2024-03-06T07:37:22.397780+0000","last_unstale":"2024-03-06T07:37:23.460439+0000","last_undegraded":"2024-03-06T07:37:23.460439+0000","last_fullsized":"2024-03-06T07:37:23.460439+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.071420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405857+0000","last_change":"2024-03-06T07:37:22.396378+0000","last_active":"2024-03-06T07:37:23.405857+0000","last_peered":"2024-03-06T07:37:23.405857+0000","last_clean":"2024-03-06T07:37:23.405857+0000","last_became_active":"2024-03-06T07:37:22.395832+0000","last_became_peered":"2024-03-06T07:37:22.395832+0000","last_unstale":"2024-03-06T07:37:23.405857+0000","last_undegraded":"2024-03-06T07:37:23.405857+0000","last_fullsized":"2024-03-06T07:37:23.405857+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.403390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"143'4","reported_seq":18,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405948+0000","last_change":"2024-03-06T07:37:21.370711+0000","last_active":"2024-03-06T07:37:23.405948+0000","last_peered":"2024-03-06T07:37:23.405948+0000","last_clean":"2024-03-06T07:37:23.405948+0000","last_became_active":"2024-03-06T07:37:21.370383+0000","last_became_peered":"2024-03-06T07:37:21.370383+0000","last_unstale":"2024-03-06T07:37:23.405948+0000","last_undegraded":"2024-03-06T07:37:23.405948+0000","last_fullsized":"2024-03-06T07:37:23.405948+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:47.441553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462751+0000","last_change":"2024-03-06T07:37:22.402996+0000","last_active":"2024-03-06T07:37:23.462751+0000","last_peered":"2024-03-06T07:37:23.462751+0000","last_clean":"2024-03-06T07:37:23.462751+0000","last_became_active":"2024-03-06T07:37:22.401815+0000","last_became_peered":"2024-03-06T07:37:22.401815+0000","last_unstale":"2024-03-06T07:37:23.462751+0000","last_undegraded":"2024-03-06T07:37:23.462751+0000","last_fullsized":"2024-03-06T07:37:23.462751+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:17.676813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462486+0000","last_change":"2024-03-06T07:37:21.369463+0000","last_active":"2024-03-06T07:37:23.462486+0000","last_peered":"2024-03-06T07:37:23.462486+0000","last_clean":"2024-03-06T07:37:23.462486+0000","last_became_active":"2024-03-06T07:37:21.369196+0000","last_became_peered":"2024-03-06T07:37:21.369196+0000","last_unstale":"2024-03-06T07:37:23.462486+0000","last_undegraded":"2024-03-06T07:37:23.462486+0000","last_fullsized":"2024-03-06T07:37:23.462486+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:33.070997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399109+0000","last_change":"2024-03-06T07:37:22.397450+0000","last_active":"2024-03-06T07:37:23.399109+0000","last_peered":"2024-03-06T07:37:23.399109+0000","last_clean":"2024-03-06T07:37:23.399109+0000","last_became_active":"2024-03-06T07:37:22.397121+0000","last_became_peered":"2024-03-06T07:37:22.397121+0000","last_unstale":"2024-03-06T07:37:23.399109+0000","last_undegraded":"2024-03-06T07:37:23.399109+0000","last_fullsized":"2024-03-06T07:37:23.399109+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:06.238541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399112+0000","last_change":"2024-03-06T07:37:21.363721+0000","last_active":"2024-03-06T07:37:23.399112+0000","last_peered":"2024-03-06T07:37:23.399112+0000","last_clean":"2024-03-06T07:37:23.399112+0000","last_became_active":"2024-03-06T07:37:21.362783+0000","last_became_peered":"2024-03-06T07:37:21.362783+0000","last_unstale":"2024-03-06T07:37:23.399112+0000","last_undegraded":"2024-03-06T07:37:23.399112+0000","last_fullsized":"2024-03-06T07:37:23.399112+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:24.658794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405257+0000","last_change":"2024-03-06T07:37:22.392998+0000","last_active":"2024-03-06T07:37:23.405257+0000","last_peered":"2024-03-06T07:37:23.405257+0000","last_clean":"2024-03-06T07:37:23.405257+0000","last_became_active":"2024-03-06T07:37:22.392721+0000","last_became_peered":"2024-03-06T07:37:22.392721+0000","last_unstale":"2024-03-06T07:37:23.405257+0000","last_undegraded":"2024-03-06T07:37:23.405257+0000","last_fullsized":"2024-03-06T07:37:23.405257+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.954146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399249+0000","last_change":"2024-03-06T07:37:21.364047+0000","last_active":"2024-03-06T07:37:23.399249+0000","last_peered":"2024-03-06T07:37:23.399249+0000","last_clean":"2024-03-06T07:37:23.399249+0000","last_became_active":"2024-03-06T07:37:21.363758+0000","last_became_peered":"2024-03-06T07:37:21.363758+0000","last_unstale":"2024-03-06T07:37:23.399249+0000","last_undegraded":"2024-03-06T07:37:23.399249+0000","last_fullsized":"2024-03-06T07:37:23.399249+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:15.462083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461999+0000","last_change":"2024-03-06T07:37:22.398997+0000","last_active":"2024-03-06T07:37:23.461999+0000","last_peered":"2024-03-06T07:37:23.461999+0000","last_clean":"2024-03-06T07:37:23.461999+0000","last_became_active":"2024-03-06T07:37:22.398617+0000","last_became_peered":"2024-03-06T07:37:22.398617+0000","last_unstale":"2024-03-06T07:37:23.461999+0000","last_undegraded":"2024-03-06T07:37:23.461999+0000","last_fullsized":"2024-03-06T07:37:23.461999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:58.167076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460590+0000","last_change":"2024-03-06T07:37:21.360910+0000","last_active":"2024-03-06T07:37:23.460590+0000","last_peered":"2024-03-06T07:37:23.460590+0000","last_clean":"2024-03-06T07:37:23.460590+0000","last_became_active":"2024-03-06T07:37:21.360647+0000","last_became_peered":"2024-03-06T07:37:21.360647+0000","last_unstale":"2024-03-06T07:37:23.460590+0000","last_undegraded":"2024-03-06T07:37:23.460590+0000","last_fullsized":"2024-03-06T07:37:23.460590+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:25.413969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397783+0000","last_change":"2024-03-06T07:37:22.399270+0000","last_active":"2024-03-06T07:37:23.397783+0000","last_peered":"2024-03-06T07:37:23.397783+0000","last_clean":"2024-03-06T07:37:23.397783+0000","last_became_active":"2024-03-06T07:37:22.397037+0000","last_became_peered":"2024-03-06T07:37:22.397037+0000","last_unstale":"2024-03-06T07:37:23.397783+0000","last_undegraded":"2024-03-06T07:37:23.397783+0000","last_fullsized":"2024-03-06T07:37:23.397783+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:08.439418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398492+0000","last_change":"2024-03-06T07:37:21.374109+0000","last_active":"2024-03-06T07:37:23.398492+0000","last_peered":"2024-03-06T07:37:23.398492+0000","last_clean":"2024-03-06T07:37:23.398492+0000","last_became_active":"2024-03-06T07:37:21.373826+0000","last_became_peered":"2024-03-06T07:37:21.373826+0000","last_unstale":"2024-03-06T07:37:23.398492+0000","last_undegraded":"2024-03-06T07:37:23.398492+0000","last_fullsized":"2024-03-06T07:37:23.398492+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:22.241595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461814+0000","last_change":"2024-03-06T07:37:22.393473+0000","last_active":"2024-03-06T07:37:23.461814+0000","last_peered":"2024-03-06T07:37:23.461814+0000","last_clean":"2024-03-06T07:37:23.461814+0000","last_became_active":"2024-03-06T07:37:22.393198+0000","last_became_peered":"2024-03-06T07:37:22.393198+0000","last_unstale":"2024-03-06T07:37:23.461814+0000","last_undegraded":"2024-03-06T07:37:23.461814+0000","last_fullsized":"2024-03-06T07:37:23.461814+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:58.117494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460575+0000","last_change":"2024-03-06T07:37:21.372994+0000","last_active":"2024-03-06T07:37:23.460575+0000","last_peered":"2024-03-06T07:37:23.460575+0000","last_clean":"2024-03-06T07:37:23.460575+0000","last_became_active":"2024-03-06T07:37:21.372042+0000","last_became_peered":"2024-03-06T07:37:21.372042+0000","last_unstale":"2024-03-06T07:37:23.460575+0000","last_undegraded":"2024-03-06T07:37:23.460575+0000","last_fullsized":"2024-03-06T07:37:23.460575+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:55.878213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398649+0000","last_change":"2024-03-06T07:37:22.393948+0000","last_active":"2024-03-06T07:37:23.398649+0000","last_peered":"2024-03-06T07:37:23.398649+0000","last_clean":"2024-03-06T07:37:23.398649+0000","last_became_active":"2024-03-06T07:37:22.393528+0000","last_became_peered":"2024-03-06T07:37:22.393528+0000","last_unstale":"2024-03-06T07:37:23.398649+0000","last_undegraded":"2024-03-06T07:37:23.398649+0000","last_fullsized":"2024-03-06T07:37:23.398649+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.331051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.396724+0000","last_change":"2024-03-06T07:37:21.369842+0000","last_active":"2024-03-06T07:37:23.396724+0000","last_peered":"2024-03-06T07:37:23.396724+0000","last_clean":"2024-03-06T07:37:23.396724+0000","last_became_active":"2024-03-06T07:37:21.369508+0000","last_became_peered":"2024-03-06T07:37:21.369508+0000","last_unstale":"2024-03-06T07:37:23.396724+0000","last_undegraded":"2024-03-06T07:37:23.396724+0000","last_fullsized":"2024-03-06T07:37:23.396724+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:51.109279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399950+0000","last_change":"2024-03-06T07:37:22.406741+0000","last_active":"2024-03-06T07:37:23.399950+0000","last_peered":"2024-03-06T07:37:23.399950+0000","last_clean":"2024-03-06T07:37:23.399950+0000","last_became_active":"2024-03-06T07:37:22.406430+0000","last_became_peered":"2024-03-06T07:37:22.406430+0000","last_unstale":"2024-03-06T07:37:23.399950+0000","last_undegraded":"2024-03-06T07:37:23.399950+0000","last_fullsized":"2024-03-06T07:37:23.399950+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:42.722561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462845+0000","last_change":"2024-03-06T07:37:21.361888+0000","last_active":"2024-03-06T07:37:23.462845+0000","last_peered":"2024-03-06T07:37:23.462845+0000","last_clean":"2024-03-06T07:37:23.462845+0000","last_became_active":"2024-03-06T07:37:21.361309+0000","last_became_peered":"2024-03-06T07:37:21.361309+0000","last_unstale":"2024-03-06T07:37:23.462845+0000","last_undegraded":"2024-03-06T07:37:23.462845+0000","last_fullsized":"2024-03-06T07:37:23.462845+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:44.982476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405877+0000","last_change":"2024-03-06T07:37:22.404220+0000","last_active":"2024-03-06T07:37:23.405877+0000","last_peered":"2024-03-06T07:37:23.405877+0000","last_clean":"2024-03-06T07:37:23.405877+0000","last_became_active":"2024-03-06T07:37:22.403909+0000","last_became_peered":"2024-03-06T07:37:22.403909+0000","last_unstale":"2024-03-06T07:37:23.405877+0000","last_undegraded":"2024-03-06T07:37:23.405877+0000","last_fullsized":"2024-03-06T07:37:23.405877+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:32.350202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406063+0000","last_change":"2024-03-06T07:37:21.374894+0000","last_active":"2024-03-06T07:37:23.406063+0000","last_peered":"2024-03-06T07:37:23.406063+0000","last_clean":"2024-03-06T07:37:23.406063+0000","last_became_active":"2024-03-06T07:37:21.374628+0000","last_became_peered":"2024-03-06T07:37:21.374628+0000","last_unstale":"2024-03-06T07:37:23.406063+0000","last_undegraded":"2024-03-06T07:37:23.406063+0000","last_fullsized":"2024-03-06T07:37:23.406063+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:54.464276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460871+0000","last_change":"2024-03-06T07:37:22.403021+0000","last_active":"2024-03-06T07:37:23.460871+0000","last_peered":"2024-03-06T07:37:23.460871+0000","last_clean":"2024-03-06T07:37:23.460871+0000","last_became_active":"2024-03-06T07:37:22.402748+0000","last_became_peered":"2024-03-06T07:37:22.402748+0000","last_unstale":"2024-03-06T07:37:23.460871+0000","last_undegraded":"2024-03-06T07:37:23.460871+0000","last_fullsized":"2024-03-06T07:37:23.460871+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:39.516810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"143'6","reported_seq":20,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394666+0000","last_change":"2024-03-06T07:37:21.365460+0000","last_active":"2024-03-06T07:37:23.394666+0000","last_peered":"2024-03-06T07:37:23.394666+0000","last_clean":"2024-03-06T07:37:23.394666+0000","last_became_active":"2024-03-06T07:37:21.365184+0000","last_became_peered":"2024-03-06T07:37:21.365184+0000","last_unstale":"2024-03-06T07:37:23.394666+0000","last_undegraded":"2024-03-06T07:37:23.394666+0000","last_fullsized":"2024-03-06T07:37:23.394666+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:47.441553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463908+0000","last_change":"2024-03-06T07:37:22.395416+0000","last_active":"2024-03-06T07:37:23.463908+0000","last_peered":"2024-03-06T07:37:23.463908+0000","last_clean":"2024-03-06T07:37:23.463908+0000","last_became_active":"2024-03-06T07:37:22.394404+0000","last_became_peered":"2024-03-06T07:37:22.394404+0000","last_unstale":"2024-03-06T07:37:23.463908+0000","last_undegraded":"2024-03-06T07:37:23.463908+0000","last_fullsized":"2024-03-06T07:37:23.463908+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:41.520623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463818+0000","last_change":"2024-03-06T07:37:21.374508+0000","last_active":"2024-03-06T07:37:23.463818+0000","last_peered":"2024-03-06T07:37:23.463818+0000","last_clean":"2024-03-06T07:37:23.463818+0000","last_became_active":"2024-03-06T07:37:21.374098+0000","last_became_peered":"2024-03-06T07:37:21.374098+0000","last_unstale":"2024-03-06T07:37:23.463818+0000","last_undegraded":"2024-03-06T07:37:23.463818+0000","last_fullsized":"2024-03-06T07:37:23.463818+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:20.363553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461963+0000","last_change":"2024-03-06T07:37:22.388870+0000","last_active":"2024-03-06T07:37:23.461963+0000","last_peered":"2024-03-06T07:37:23.461963+0000","last_clean":"2024-03-06T07:37:23.461963+0000","last_became_active":"2024-03-06T07:37:22.386400+0000","last_became_peered":"2024-03-06T07:37:22.386400+0000","last_unstale":"2024-03-06T07:37:23.461963+0000","last_undegraded":"2024-03-06T07:37:23.461963+0000","last_fullsized":"2024-03-06T07:37:23.461963+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:51.117104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464133+0000","last_change":"2024-03-06T07:37:21.376580+0000","last_active":"2024-03-06T07:37:23.464133+0000","last_peered":"2024-03-06T07:37:23.464133+0000","last_clean":"2024-03-06T07:37:23.464133+0000","last_became_active":"2024-03-06T07:37:21.375739+0000","last_became_peered":"2024-03-06T07:37:21.375739+0000","last_unstale":"2024-03-06T07:37:23.464133+0000","last_undegraded":"2024-03-06T07:37:23.464133+0000","last_fullsized":"2024-03-06T07:37:23.464133+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:57.149568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397656+0000","last_change":"2024-03-06T07:37:22.399075+0000","last_active":"2024-03-06T07:37:23.397656+0000","last_peered":"2024-03-06T07:37:23.397656+0000","last_clean":"2024-03-06T07:37:23.397656+0000","last_became_active":"2024-03-06T07:37:22.398773+0000","last_became_peered":"2024-03-06T07:37:22.398773+0000","last_unstale":"2024-03-06T07:37:23.397656+0000","last_undegraded":"2024-03-06T07:37:23.397656+0000","last_fullsized":"2024-03-06T07:37:23.397656+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:23.464868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398898+0000","last_change":"2024-03-06T07:37:21.366115+0000","last_active":"2024-03-06T07:37:23.398898+0000","last_peered":"2024-03-06T07:37:23.398898+0000","last_clean":"2024-03-06T07:37:23.398898+0000","last_became_active":"2024-03-06T07:37:21.360739+0000","last_became_peered":"2024-03-06T07:37:21.360739+0000","last_unstale":"2024-03-06T07:37:23.398898+0000","last_undegraded":"2024-03-06T07:37:23.398898+0000","last_fullsized":"2024-03-06T07:37:23.398898+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:11.295318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463497+0000","last_change":"2024-03-06T07:37:22.391541+0000","last_active":"2024-03-06T07:37:23.463497+0000","last_peered":"2024-03-06T07:37:23.463497+0000","last_clean":"2024-03-06T07:37:23.463497+0000","last_became_active":"2024-03-06T07:37:22.391266+0000","last_became_peered":"2024-03-06T07:37:22.391266+0000","last_unstale":"2024-03-06T07:37:23.463497+0000","last_undegraded":"2024-03-06T07:37:23.463497+0000","last_fullsized":"2024-03-06T07:37:23.463497+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:36.812503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"143'8","reported_seq":22,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460487+0000","last_change":"2024-03-06T07:37:21.367352+0000","last_active":"2024-03-06T07:37:23.460487+0000","last_peered":"2024-03-06T07:37:23.460487+0000","last_clean":"2024-03-06T07:37:23.460487+0000","last_became_active":"2024-03-06T07:37:21.367107+0000","last_became_peered":"2024-03-06T07:37:21.367107+0000","last_unstale":"2024-03-06T07:37:23.460487+0000","last_undegraded":"2024-03-06T07:37:23.460487+0000","last_fullsized":"2024-03-06T07:37:23.460487+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:34:02.301827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406244+0000","last_change":"2024-03-06T07:37:22.384018+0000","last_active":"2024-03-06T07:37:23.406244+0000","last_peered":"2024-03-06T07:37:23.406244+0000","last_clean":"2024-03-06T07:37:23.406244+0000","last_became_active":"2024-03-06T07:37:22.383729+0000","last_became_peered":"2024-03-06T07:37:22.383729+0000","last_unstale":"2024-03-06T07:37:23.406244+0000","last_undegraded":"2024-03-06T07:37:23.406244+0000","last_fullsized":"2024-03-06T07:37:23.406244+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:25.594731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399833+0000","last_change":"2024-03-06T07:37:21.375503+0000","last_active":"2024-03-06T07:37:23.399833+0000","last_peered":"2024-03-06T07:37:23.399833+0000","last_clean":"2024-03-06T07:37:23.399833+0000","last_became_active":"2024-03-06T07:37:21.375171+0000","last_became_peered":"2024-03-06T07:37:21.375171+0000","last_unstale":"2024-03-06T07:37:23.399833+0000","last_undegraded":"2024-03-06T07:37:23.399833+0000","last_fullsized":"2024-03-06T07:37:23.399833+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.774103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461058+0000","last_change":"2024-03-06T07:37:22.396545+0000","last_active":"2024-03-06T07:37:23.461058+0000","last_peered":"2024-03-06T07:37:23.461058+0000","last_clean":"2024-03-06T07:37:23.461058+0000","last_became_active":"2024-03-06T07:37:22.394046+0000","last_became_peered":"2024-03-06T07:37:22.394046+0000","last_unstale":"2024-03-06T07:37:23.461058+0000","last_undegraded":"2024-03-06T07:37:23.461058+0000","last_fullsized":"2024-03-06T07:37:23.461058+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:11.759303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462594+0000","last_change":"2024-03-06T07:37:21.373115+0000","last_active":"2024-03-06T07:37:23.462594+0000","last_peered":"2024-03-06T07:37:23.462594+0000","last_clean":"2024-03-06T07:37:23.462594+0000","last_became_active":"2024-03-06T07:37:21.372851+0000","last_became_peered":"2024-03-06T07:37:21.372851+0000","last_unstale":"2024-03-06T07:37:23.462594+0000","last_undegraded":"2024-03-06T07:37:23.462594+0000","last_fullsized":"2024-03-06T07:37:23.462594+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:47.441553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464313+0000","last_change":"2024-03-06T07:37:22.404472+0000","last_active":"2024-03-06T07:37:23.464313+0000","last_peered":"2024-03-06T07:37:23.464313+0000","last_clean":"2024-03-06T07:37:23.464313+0000","last_became_active":"2024-03-06T07:37:22.402498+0000","last_became_peered":"2024-03-06T07:37:22.402498+0000","last_unstale":"2024-03-06T07:37:23.464313+0000","last_undegraded":"2024-03-06T07:37:23.464313+0000","last_fullsized":"2024-03-06T07:37:23.464313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:18.419563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461003+0000","last_change":"2024-03-06T07:37:21.354195+0000","last_active":"2024-03-06T07:37:23.461003+0000","last_peered":"2024-03-06T07:37:23.461003+0000","last_clean":"2024-03-06T07:37:23.461003+0000","last_became_active":"2024-03-06T07:37:21.353866+0000","last_became_peered":"2024-03-06T07:37:21.353866+0000","last_unstale":"2024-03-06T07:37:23.461003+0000","last_undegraded":"2024-03-06T07:37:23.461003+0000","last_fullsized":"2024-03-06T07:37:23.461003+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:49.470863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399805+0000","last_change":"2024-03-06T07:37:22.394693+0000","last_active":"2024-03-06T07:37:23.399805+0000","last_peered":"2024-03-06T07:37:23.399805+0000","last_clean":"2024-03-06T07:37:23.399805+0000","last_became_active":"2024-03-06T07:37:22.394425+0000","last_became_peered":"2024-03-06T07:37:22.394425+0000","last_unstale":"2024-03-06T07:37:23.399805+0000","last_undegraded":"2024-03-06T07:37:23.399805+0000","last_fullsized":"2024-03-06T07:37:23.399805+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:21.378845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462203+0000","last_change":"2024-03-06T07:37:21.364866+0000","last_active":"2024-03-06T07:37:23.462203+0000","last_peered":"2024-03-06T07:37:23.462203+0000","last_clean":"2024-03-06T07:37:23.462203+0000","last_became_active":"2024-03-06T07:37:21.364556+0000","last_became_peered":"2024-03-06T07:37:21.364556+0000","last_unstale":"2024-03-06T07:37:23.462203+0000","last_undegraded":"2024-03-06T07:37:23.462203+0000","last_fullsized":"2024-03-06T07:37:23.462203+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:54.464276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398048+0000","last_change":"2024-03-06T07:37:22.397431+0000","last_active":"2024-03-06T07:37:23.398048+0000","last_peered":"2024-03-06T07:37:23.398048+0000","last_clean":"2024-03-06T07:37:23.398048+0000","last_became_active":"2024-03-06T07:37:22.397099+0000","last_became_peered":"2024-03-06T07:37:22.397099+0000","last_unstale":"2024-03-06T07:37:23.398048+0000","last_undegraded":"2024-03-06T07:37:23.398048+0000","last_fullsized":"2024-03-06T07:37:23.398048+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:21.173058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394402+0000","last_change":"2024-03-06T07:37:21.372528+0000","last_active":"2024-03-06T07:37:23.394402+0000","last_peered":"2024-03-06T07:37:23.394402+0000","last_clean":"2024-03-06T07:37:23.394402+0000","last_became_active":"2024-03-06T07:37:21.372238+0000","last_became_peered":"2024-03-06T07:37:21.372238+0000","last_unstale":"2024-03-06T07:37:23.394402+0000","last_undegraded":"2024-03-06T07:37:23.394402+0000","last_fullsized":"2024-03-06T07:37:23.394402+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:54.464276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394803+0000","last_change":"2024-03-06T07:37:22.401310+0000","last_active":"2024-03-06T07:37:23.394803+0000","last_peered":"2024-03-06T07:37:23.394803+0000","last_clean":"2024-03-06T07:37:23.394803+0000","last_became_active":"2024-03-06T07:37:22.401025+0000","last_became_peered":"2024-03-06T07:37:22.401025+0000","last_unstale":"2024-03-06T07:37:23.394803+0000","last_undegraded":"2024-03-06T07:37:23.394803+0000","last_fullsized":"2024-03-06T07:37:23.394803+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:52.355345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405371+0000","last_change":"2024-03-06T07:37:21.370098+0000","last_active":"2024-03-06T07:37:23.405371+0000","last_peered":"2024-03-06T07:37:23.405371+0000","last_clean":"2024-03-06T07:37:23.405371+0000","last_became_active":"2024-03-06T07:37:21.369792+0000","last_became_peered":"2024-03-06T07:37:21.369792+0000","last_unstale":"2024-03-06T07:37:23.405371+0000","last_undegraded":"2024-03-06T07:37:23.405371+0000","last_fullsized":"2024-03-06T07:37:23.405371+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:49.986528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.396740+0000","last_change":"2024-03-06T07:37:22.408331+0000","last_active":"2024-03-06T07:37:23.396740+0000","last_peered":"2024-03-06T07:37:23.396740+0000","last_clean":"2024-03-06T07:37:23.396740+0000","last_became_active":"2024-03-06T07:37:22.407996+0000","last_became_peered":"2024-03-06T07:37:22.407996+0000","last_unstale":"2024-03-06T07:37:23.396740+0000","last_undegraded":"2024-03-06T07:37:23.396740+0000","last_fullsized":"2024-03-06T07:37:23.396740+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:00.033313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405004+0000","last_change":"2024-03-06T07:37:21.371516+0000","last_active":"2024-03-06T07:37:23.405004+0000","last_peered":"2024-03-06T07:37:23.405004+0000","last_clean":"2024-03-06T07:37:23.405004+0000","last_became_active":"2024-03-06T07:37:21.370796+0000","last_became_peered":"2024-03-06T07:37:21.370796+0000","last_unstale":"2024-03-06T07:37:23.405004+0000","last_undegraded":"2024-03-06T07:37:23.405004+0000","last_fullsized":"2024-03-06T07:37:23.405004+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:47.675808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394734+0000","last_change":"2024-03-06T07:37:22.409916+0000","last_active":"2024-03-06T07:37:23.394734+0000","last_peered":"2024-03-06T07:37:23.394734+0000","last_clean":"2024-03-06T07:37:23.394734+0000","last_became_active":"2024-03-06T07:37:22.409181+0000","last_became_peered":"2024-03-06T07:37:22.409181+0000","last_unstale":"2024-03-06T07:37:23.394734+0000","last_undegraded":"2024-03-06T07:37:23.394734+0000","last_fullsized":"2024-03-06T07:37:23.394734+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:58.117494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462985+0000","last_change":"2024-03-06T07:37:21.367483+0000","last_active":"2024-03-06T07:37:23.462985+0000","last_peered":"2024-03-06T07:37:23.462985+0000","last_clean":"2024-03-06T07:37:23.462985+0000","last_became_active":"2024-03-06T07:37:21.366939+0000","last_became_peered":"2024-03-06T07:37:21.366939+0000","last_unstale":"2024-03-06T07:37:23.462985+0000","last_undegraded":"2024-03-06T07:37:23.462985+0000","last_fullsized":"2024-03-06T07:37:23.462985+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:47.211106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400238+0000","last_change":"2024-03-06T07:37:22.406359+0000","last_active":"2024-03-06T07:37:23.400238+0000","last_peered":"2024-03-06T07:37:23.400238+0000","last_clean":"2024-03-06T07:37:23.400238+0000","last_became_active":"2024-03-06T07:37:22.405754+0000","last_became_peered":"2024-03-06T07:37:22.405754+0000","last_unstale":"2024-03-06T07:37:23.400238+0000","last_undegraded":"2024-03-06T07:37:23.400238+0000","last_fullsized":"2024-03-06T07:37:23.400238+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:05.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406233+0000","last_change":"2024-03-06T07:37:21.375388+0000","last_active":"2024-03-06T07:37:23.406233+0000","last_peered":"2024-03-06T07:37:23.406233+0000","last_clean":"2024-03-06T07:37:23.406233+0000","last_became_active":"2024-03-06T07:37:21.375114+0000","last_became_peered":"2024-03-06T07:37:21.375114+0000","last_unstale":"2024-03-06T07:37:23.406233+0000","last_undegraded":"2024-03-06T07:37:23.406233+0000","last_fullsized":"2024-03-06T07:37:23.406233+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:44.982476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464902+0000","last_change":"2024-03-06T07:37:22.394822+0000","last_active":"2024-03-06T07:37:23.464902+0000","last_peered":"2024-03-06T07:37:23.464902+0000","last_clean":"2024-03-06T07:37:23.464902+0000","last_became_active":"2024-03-06T07:37:22.394527+0000","last_became_peered":"2024-03-06T07:37:22.394527+0000","last_unstale":"2024-03-06T07:37:23.464902+0000","last_undegraded":"2024-03-06T07:37:23.464902+0000","last_fullsized":"2024-03-06T07:37:23.464902+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:48.691100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"144'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.402824+0000","last_change":"2024-03-06T07:37:21.369209+0000","last_active":"2024-03-06T07:37:23.402824+0000","last_peered":"2024-03-06T07:37:23.402824+0000","last_clean":"2024-03-06T07:37:23.402824+0000","last_became_active":"2024-03-06T07:37:21.368820+0000","last_became_peered":"2024-03-06T07:37:21.368820+0000","last_unstale":"2024-03-06T07:37:23.402824+0000","last_undegraded":"2024-03-06T07:37:23.402824+0000","last_fullsized":"2024-03-06T07:37:23.402824+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:50.585486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462038+0000","last_change":"2024-03-06T07:37:22.400181+0000","last_active":"2024-03-06T07:37:23.462038+0000","last_peered":"2024-03-06T07:37:23.462038+0000","last_clean":"2024-03-06T07:37:23.462038+0000","last_became_active":"2024-03-06T07:37:22.399887+0000","last_became_peered":"2024-03-06T07:37:22.399887+0000","last_unstale":"2024-03-06T07:37:23.462038+0000","last_undegraded":"2024-03-06T07:37:23.462038+0000","last_fullsized":"2024-03-06T07:37:23.462038+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:29.190943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400026+0000","last_change":"2024-03-06T07:37:21.355184+0000","last_active":"2024-03-06T07:37:23.400026+0000","last_peered":"2024-03-06T07:37:23.400026+0000","last_clean":"2024-03-06T07:37:23.400026+0000","last_became_active":"2024-03-06T07:37:21.354882+0000","last_became_peered":"2024-03-06T07:37:21.354882+0000","last_unstale":"2024-03-06T07:37:23.400026+0000","last_undegraded":"2024-03-06T07:37:23.400026+0000","last_fullsized":"2024-03-06T07:37:23.400026+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:22.449576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399756+0000","last_change":"2024-03-06T07:37:22.396365+0000","last_active":"2024-03-06T07:37:23.399756+0000","last_peered":"2024-03-06T07:37:23.399756+0000","last_clean":"2024-03-06T07:37:23.399756+0000","last_became_active":"2024-03-06T07:37:22.396080+0000","last_became_peered":"2024-03-06T07:37:22.396080+0000","last_unstale":"2024-03-06T07:37:23.399756+0000","last_undegraded":"2024-03-06T07:37:23.399756+0000","last_fullsized":"2024-03-06T07:37:23.399756+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:48.226399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399226+0000","last_change":"2024-03-06T07:37:21.353350+0000","last_active":"2024-03-06T07:37:23.399226+0000","last_peered":"2024-03-06T07:37:23.399226+0000","last_clean":"2024-03-06T07:37:23.399226+0000","last_became_active":"2024-03-06T07:37:21.352640+0000","last_became_peered":"2024-03-06T07:37:21.352640+0000","last_unstale":"2024-03-06T07:37:23.399226+0000","last_undegraded":"2024-03-06T07:37:23.399226+0000","last_fullsized":"2024-03-06T07:37:23.399226+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:38.501518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460551+0000","last_change":"2024-03-06T07:37:22.403004+0000","last_active":"2024-03-06T07:37:23.460551+0000","last_peered":"2024-03-06T07:37:23.460551+0000","last_clean":"2024-03-06T07:37:23.460551+0000","last_became_active":"2024-03-06T07:37:22.402049+0000","last_became_peered":"2024-03-06T07:37:22.402049+0000","last_unstale":"2024-03-06T07:37:23.460551+0000","last_undegraded":"2024-03-06T07:37:23.460551+0000","last_fullsized":"2024-03-06T07:37:23.460551+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:12.310610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394502+0000","last_change":"2024-03-06T07:37:21.369250+0000","last_active":"2024-03-06T07:37:23.394502+0000","last_peered":"2024-03-06T07:37:23.394502+0000","last_clean":"2024-03-06T07:37:23.394502+0000","last_became_active":"2024-03-06T07:37:21.368493+0000","last_became_peered":"2024-03-06T07:37:21.368493+0000","last_unstale":"2024-03-06T07:37:23.394502+0000","last_undegraded":"2024-03-06T07:37:23.394502+0000","last_fullsized":"2024-03-06T07:37:23.394502+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:31.480272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399589+0000","last_change":"2024-03-06T07:37:22.396371+0000","last_active":"2024-03-06T07:37:23.399589+0000","last_peered":"2024-03-06T07:37:23.399589+0000","last_clean":"2024-03-06T07:37:23.399589+0000","last_became_active":"2024-03-06T07:37:22.395650+0000","last_became_peered":"2024-03-06T07:37:22.395650+0000","last_unstale":"2024-03-06T07:37:23.399589+0000","last_undegraded":"2024-03-06T07:37:23.399589+0000","last_fullsized":"2024-03-06T07:37:23.399589+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:16.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.404803+0000","last_change":"2024-03-06T07:37:21.374738+0000","last_active":"2024-03-06T07:37:23.404803+0000","last_peered":"2024-03-06T07:37:23.404803+0000","last_clean":"2024-03-06T07:37:23.404803+0000","last_became_active":"2024-03-06T07:37:21.374324+0000","last_became_peered":"2024-03-06T07:37:21.374324+0000","last_unstale":"2024-03-06T07:37:23.404803+0000","last_undegraded":"2024-03-06T07:37:23.404803+0000","last_fullsized":"2024-03-06T07:37:23.404803+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:33.070997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405053+0000","last_change":"2024-03-06T07:37:22.385530+0000","last_active":"2024-03-06T07:37:23.405053+0000","last_peered":"2024-03-06T07:37:23.405053+0000","last_clean":"2024-03-06T07:37:23.405053+0000","last_became_active":"2024-03-06T07:37:22.385134+0000","last_became_peered":"2024-03-06T07:37:22.385134+0000","last_unstale":"2024-03-06T07:37:23.405053+0000","last_undegraded":"2024-03-06T07:37:23.405053+0000","last_fullsized":"2024-03-06T07:37:23.405053+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:46.584284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463358+0000","last_change":"2024-03-06T07:37:21.372688+0000","last_active":"2024-03-06T07:37:23.463358+0000","last_peered":"2024-03-06T07:37:23.463358+0000","last_clean":"2024-03-06T07:37:23.463358+0000","last_became_active":"2024-03-06T07:37:21.372275+0000","last_became_peered":"2024-03-06T07:37:21.372275+0000","last_unstale":"2024-03-06T07:37:23.463358+0000","last_undegraded":"2024-03-06T07:37:23.463358+0000","last_fullsized":"2024-03-06T07:37:23.463358+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:00.690180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399339+0000","last_change":"2024-03-06T07:37:22.404380+0000","last_active":"2024-03-06T07:37:23.399339+0000","last_peered":"2024-03-06T07:37:23.399339+0000","last_clean":"2024-03-06T07:37:23.399339+0000","last_became_active":"2024-03-06T07:37:22.404068+0000","last_became_peered":"2024-03-06T07:37:22.404068+0000","last_unstale":"2024-03-06T07:37:23.399339+0000","last_undegraded":"2024-03-06T07:37:23.399339+0000","last_fullsized":"2024-03-06T07:37:23.399339+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:21.095007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461848+0000","last_change":"2024-03-06T07:37:21.369739+0000","last_active":"2024-03-06T07:37:23.461848+0000","last_peered":"2024-03-06T07:37:23.461848+0000","last_clean":"2024-03-06T07:37:23.461848+0000","last_became_active":"2024-03-06T07:37:21.369469+0000","last_became_peered":"2024-03-06T07:37:21.369469+0000","last_unstale":"2024-03-06T07:37:23.461848+0000","last_undegraded":"2024-03-06T07:37:23.461848+0000","last_fullsized":"2024-03-06T07:37:23.461848+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:49.986528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394619+0000","last_change":"2024-03-06T07:37:22.410356+0000","last_active":"2024-03-06T07:37:23.394619+0000","last_peered":"2024-03-06T07:37:23.394619+0000","last_clean":"2024-03-06T07:37:23.394619+0000","last_became_active":"2024-03-06T07:37:22.410028+0000","last_became_peered":"2024-03-06T07:37:22.410028+0000","last_unstale":"2024-03-06T07:37:23.394619+0000","last_undegraded":"2024-03-06T07:37:23.394619+0000","last_fullsized":"2024-03-06T07:37:23.394619+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:24.219967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"143'3","reported_seq":17,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464453+0000","last_change":"2024-03-06T07:37:21.371926+0000","last_active":"2024-03-06T07:37:23.464453+0000","last_peered":"2024-03-06T07:37:23.464453+0000","last_clean":"2024-03-06T07:37:23.464453+0000","last_became_active":"2024-03-06T07:37:21.371654+0000","last_became_peered":"2024-03-06T07:37:21.371654+0000","last_unstale":"2024-03-06T07:37:23.464453+0000","last_undegraded":"2024-03-06T07:37:23.464453+0000","last_fullsized":"2024-03-06T07:37:23.464453+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:51.206679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":395,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405127+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:37:23.405127+0000","last_peered":"2024-03-06T07:37:23.405127+0000","last_clean":"2024-03-06T07:37:23.405127+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:37:23.405127+0000","last_undegraded":"2024-03-06T07:37:23.405127+0000","last_fullsized":"2024-03-06T07:37:23.405127+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542264,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37688,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334152,"statfs":{"total":96636764160,"available":96598171648,"internally_reserved":0,"allocated":2400256,"data_stored":1377397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542265,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1798144,"data_stored":780223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37104,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334736,"statfs":{"total":96636764160,"available":96598769664,"internally_reserved":0,"allocated":1802240,"data_stored":786399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":786377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1789952,"data_stored":780167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37104,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334736,"statfs":{"total":96636764160,"available":96598769664,"internally_reserved":0,"allocated":1802240,"data_stored":781759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1802240,"data_stored":786959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41844,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329996,"statfs":{"total":96636764160,"available":96593915904,"internally_reserved":0,"allocated":2396160,"data_stored":1372091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:37:26.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:37:27.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.121+0000 7f8dbb200640 1 Processor -- start 2024-03-06T07:37:27.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.122+0000 7f8dbb200640 1 -- start start 2024-03-06T07:37:27.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.123+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db4161580 0x7f8db40d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.123+0000 7f8dbb200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8db40d1e90 0x7f8db40d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.123+0000 7f8dbb200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db4161960 con 0x7f8db40d28a0 2024-03-06T07:37:27.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.123+0000 7f8dbb200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db40cf6f0 con 0x7f8db40d1e90 2024-03-06T07:37:27.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.123+0000 7f8dbb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40cf9d0 con 0x7f8db4161580 2024-03-06T07:37:27.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.124+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db4161580 0x7f8db40d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.124+0000 7f8db9400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8db40d1e90 0x7f8db40d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.124+0000 7f8db9400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8db40d1e90 0x7f8db40d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:50518/0 (socket says 172.21.15.97:50518) 2024-03-06T07:37:27.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.124+0000 7f8db9400640 1 -- 172.21.15.97:0/331383925 learned_addr learned my addr 172.21.15.97:0/331383925 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:27.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.125+0000 7f8db9e00640 1 -- 172.21.15.97:0/331383925 >> v2:172.21.15.135:3300/0 conn(0x7f8db40d1e90 msgr2=0x7f8db40d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:27.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.125+0000 7f8db9e00640 1 --2- 172.21.15.97:0/331383925 >> v2:172.21.15.135:3300/0 conn(0x7f8db40d1e90 0x7f8db40d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.125+0000 7f8db9e00640 1 -- 172.21.15.97:0/331383925 >> v1:172.21.15.135:6789/0 conn(0x7f8db40d28a0 legacy=0x7f8db40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.125+0000 7f8db9e00640 1 -- 172.21.15.97:0/331383925 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40cfcb0 con 0x7f8db4161580 2024-03-06T07:37:27.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.126+0000 7f8db9e00640 1 --2- 172.21.15.97:0/331383925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db4161580 0x7f8db40d1950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8da80097a0 tx=0x7f8da802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d352e0b67f0fa80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.126+0000 7f8db8a00640 1 -- 172.21.15.97:0/331383925 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8035020 con 0x7f8db4161580 2024-03-06T07:37:27.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8db8a00640 1 -- 172.21.15.97:0/331383925 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da8031810 con 0x7f8db4161580 2024-03-06T07:37:27.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8db8a00640 1 -- 172.21.15.97:0/331383925 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8031be0 con 0x7f8db4161580 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 -- 172.21.15.97:0/331383925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db4161580 msgr2=0x7f8db40d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 --2- 172.21.15.97:0/331383925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db4161580 0x7f8db40d1950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8da80097a0 tx=0x7f8da802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 -- 172.21.15.97:0/331383925 shutdown_connections 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 --2- 172.21.15.97:0/331383925 >> v2:172.21.15.135:3300/0 conn(0x7f8db40d1e90 0x7f8db40d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 --2- 172.21.15.97:0/331383925 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db4161580 0x7f8db40d1950 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 -- 172.21.15.97:0/331383925 >> 172.21.15.97:0/331383925 conn(0x7f8db40d3d80 msgr2=0x7f8db40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 -- 172.21.15.97:0/331383925 shutdown_connections 2024-03-06T07:37:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.127+0000 7f8dbb200640 1 -- 172.21.15.97:0/331383925 wait complete. 2024-03-06T07:37:27.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.129+0000 7f8dbb200640 1 Processor -- start 2024-03-06T07:37:27.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.129+0000 7f8dbb200640 1 -- start start 2024-03-06T07:37:27.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 0x7f8db40cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db40d28a0 0x7f8db40cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db4161580 0x7f8db4152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dbb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40cf3a0 con 0x7f8db40d1e90 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dbb200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40b39a0 con 0x7f8db4161580 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dbb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40b4ed0 con 0x7f8db40d28a0 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 0x7f8db40cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 0x7f8db40cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38292/0 (socket says 172.21.15.97:38292) 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8db9e00640 1 -- 172.21.15.97:0/3574002078 learned_addr learned my addr 172.21.15.97:0/3574002078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8db9400640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db40d28a0 0x7f8db40cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.130+0000 7f8dba800640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db4161580 0x7f8db4152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8db9e00640 1 -- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db40d28a0 msgr2=0x7f8db40cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8db9e00640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db40d28a0 0x7f8db40cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8db9e00640 1 -- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db4161580 msgr2=0x7f8db4152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8db9e00640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db4161580 0x7f8db4152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8db9e00640 1 -- 172.21.15.97:0/3574002078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40cd0e0 con 0x7f8db40d1e90 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8dba800640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db4161580 0x7f8db4152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8db9e00640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 0x7f8db40cb8a0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f8da8034040 tx=0x7f8da80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8040070 con 0x7f8db40d1e90 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da8039cf0 con 0x7f8db40d1e90 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8035040 con 0x7f8db40d1e90 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db40cd3c0 con 0x7f8db40d1e90 2024-03-06T07:37:27.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.131+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db40ccdf0 con 0x7f8db40d1e90 2024-03-06T07:37:27.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.132+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db41610c0 con 0x7f8db40d1e90 2024-03-06T07:37:27.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.133+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8da8038020 con 0x7f8db40d1e90 2024-03-06T07:37:27.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.134+0000 7f8dae000640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9806b4d0 0x7f8d9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.134+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8da80abd90 con 0x7f8db40d1e90 2024-03-06T07:37:27.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.135+0000 7f8db9400640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9806b4d0 0x7f8d9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.135+0000 7f8db9400640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9806b4d0 0x7f8d9806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8db40cd750 tx=0x7f8da4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.139+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8da807ac00 con 0x7f8db40d1e90 2024-03-06T07:37:27.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.330+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8db404f330 con 0x7f8d9806b4d0 2024-03-06T07:37:27.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.343+0000 7f8dae000640 1 -- 172.21.15.97:0/3574002078 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f8db404f330 con 0x7f8d9806b4d0 2024-03-06T07:37:27.349 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:27.351 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:37:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9806b4d0 msgr2=0x7f8d9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9806b4d0 0x7f8d9806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8db40cd750 tx=0x7f8da4008040 comp rx=0 tx=0).stop 2024-03-06T07:37:27.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 msgr2=0x7f8db40cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 0x7f8db40cb8a0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f8da8034040 tx=0x7f8da80046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 shutdown_connections 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8d9806b4d0 0x7f8d9806d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8db4161580 0x7f8db4152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8db40d28a0 0x7f8db40cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 --2- 172.21.15.97:0/3574002078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8db40d1e90 0x7f8db40cb8a0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 >> 172.21.15.97:0/3574002078 conn(0x7f8db40d3d80 msgr2=0x7f8db415e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 shutdown_connections 2024-03-06T07:37:27.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.350+0000 7f8dbb200640 1 -- 172.21.15.97:0/3574002078 wait complete. 2024-03-06T07:37:27.377 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-03-06T07:37:27.085266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310444,"kb_used_data":15224,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664276,"statfs":{"total":773094113280,"available":772776218624,"internally_reserved":0,"allocated":15589376,"data_stored":7451372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001934"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400157+0000","last_change":"2024-03-06T07:37:22.383332+0000","last_active":"2024-03-06T07:37:23.400157+0000","last_peered":"2024-03-06T07:37:23.400157+0000","last_clean":"2024-03-06T07:37:23.400157+0000","last_became_active":"2024-03-06T07:37:22.382414+0000","last_became_peered":"2024-03-06T07:37:22.382414+0000","last_unstale":"2024-03-06T07:37:23.400157+0000","last_undegraded":"2024-03-06T07:37:23.400157+0000","last_fullsized":"2024-03-06T07:37:23.400157+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:40.838978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462385+0000","last_change":"2024-03-06T07:37:22.400356+0000","last_active":"2024-03-06T07:37:23.462385+0000","last_peered":"2024-03-06T07:37:23.462385+0000","last_clean":"2024-03-06T07:37:23.462385+0000","last_became_active":"2024-03-06T07:37:22.400055+0000","last_became_peered":"2024-03-06T07:37:22.400055+0000","last_unstale":"2024-03-06T07:37:23.462385+0000","last_undegraded":"2024-03-06T07:37:23.462385+0000","last_fullsized":"2024-03-06T07:37:23.462385+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:46.584284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464352+0000","last_change":"2024-03-06T07:37:22.395370+0000","last_active":"2024-03-06T07:37:23.464352+0000","last_peered":"2024-03-06T07:37:23.464352+0000","last_clean":"2024-03-06T07:37:23.464352+0000","last_became_active":"2024-03-06T07:37:22.394936+0000","last_became_peered":"2024-03-06T07:37:22.394936+0000","last_unstale":"2024-03-06T07:37:23.464352+0000","last_undegraded":"2024-03-06T07:37:23.464352+0000","last_fullsized":"2024-03-06T07:37:23.464352+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:55.000031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398992+0000","last_change":"2024-03-06T07:37:22.399940+0000","last_active":"2024-03-06T07:37:23.398992+0000","last_peered":"2024-03-06T07:37:23.398992+0000","last_clean":"2024-03-06T07:37:23.398992+0000","last_became_active":"2024-03-06T07:37:22.399611+0000","last_became_peered":"2024-03-06T07:37:22.399611+0000","last_unstale":"2024-03-06T07:37:23.398992+0000","last_undegraded":"2024-03-06T07:37:23.398992+0000","last_fullsized":"2024-03-06T07:37:23.398992+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:46.193975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398275+0000","last_change":"2024-03-06T07:37:22.397393+0000","last_active":"2024-03-06T07:37:23.398275+0000","last_peered":"2024-03-06T07:37:23.398275+0000","last_clean":"2024-03-06T07:37:23.398275+0000","last_became_active":"2024-03-06T07:37:22.397108+0000","last_became_peered":"2024-03-06T07:37:22.397108+0000","last_unstale":"2024-03-06T07:37:23.398275+0000","last_undegraded":"2024-03-06T07:37:23.398275+0000","last_fullsized":"2024-03-06T07:37:23.398275+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:51.600779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399474+0000","last_change":"2024-03-06T07:37:22.394968+0000","last_active":"2024-03-06T07:37:23.399474+0000","last_peered":"2024-03-06T07:37:23.399474+0000","last_clean":"2024-03-06T07:37:23.399474+0000","last_became_active":"2024-03-06T07:37:22.394621+0000","last_became_peered":"2024-03-06T07:37:22.394621+0000","last_unstale":"2024-03-06T07:37:23.399474+0000","last_undegraded":"2024-03-06T07:37:23.399474+0000","last_fullsized":"2024-03-06T07:37:23.399474+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:01.705472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394571+0000","last_change":"2024-03-06T07:37:22.410318+0000","last_active":"2024-03-06T07:37:23.394571+0000","last_peered":"2024-03-06T07:37:23.394571+0000","last_clean":"2024-03-06T07:37:23.394571+0000","last_became_active":"2024-03-06T07:37:22.409584+0000","last_became_peered":"2024-03-06T07:37:22.409584+0000","last_unstale":"2024-03-06T07:37:23.394571+0000","last_undegraded":"2024-03-06T07:37:23.394571+0000","last_fullsized":"2024-03-06T07:37:23.394571+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:11.979155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399026+0000","last_change":"2024-03-06T07:37:22.391801+0000","last_active":"2024-03-06T07:37:23.399026+0000","last_peered":"2024-03-06T07:37:23.399026+0000","last_clean":"2024-03-06T07:37:23.399026+0000","last_became_active":"2024-03-06T07:37:22.391522+0000","last_became_peered":"2024-03-06T07:37:22.391522+0000","last_unstale":"2024-03-06T07:37:23.399026+0000","last_undegraded":"2024-03-06T07:37:23.399026+0000","last_fullsized":"2024-03-06T07:37:23.399026+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:47.121952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462791+0000","last_change":"2024-03-06T07:37:22.386445+0000","last_active":"2024-03-06T07:37:23.462791+0000","last_peered":"2024-03-06T07:37:23.462791+0000","last_clean":"2024-03-06T07:37:23.462791+0000","last_became_active":"2024-03-06T07:37:22.384570+0000","last_became_peered":"2024-03-06T07:37:22.384570+0000","last_unstale":"2024-03-06T07:37:23.462791+0000","last_undegraded":"2024-03-06T07:37:23.462791+0000","last_fullsized":"2024-03-06T07:37:23.462791+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.954146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405797+0000","last_change":"2024-03-06T07:37:22.403983+0000","last_active":"2024-03-06T07:37:23.405797+0000","last_peered":"2024-03-06T07:37:23.405797+0000","last_clean":"2024-03-06T07:37:23.405797+0000","last_became_active":"2024-03-06T07:37:22.403283+0000","last_became_peered":"2024-03-06T07:37:22.403283+0000","last_unstale":"2024-03-06T07:37:23.405797+0000","last_undegraded":"2024-03-06T07:37:23.405797+0000","last_fullsized":"2024-03-06T07:37:23.405797+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:32.623856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399440+0000","last_change":"2024-03-06T07:37:22.390213+0000","last_active":"2024-03-06T07:37:23.399440+0000","last_peered":"2024-03-06T07:37:23.399440+0000","last_clean":"2024-03-06T07:37:23.399440+0000","last_became_active":"2024-03-06T07:37:22.389709+0000","last_became_peered":"2024-03-06T07:37:22.389709+0000","last_unstale":"2024-03-06T07:37:23.399440+0000","last_undegraded":"2024-03-06T07:37:23.399440+0000","last_fullsized":"2024-03-06T07:37:23.399440+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:52.221971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405134+0000","last_change":"2024-03-06T07:37:22.392961+0000","last_active":"2024-03-06T07:37:23.405134+0000","last_peered":"2024-03-06T07:37:23.405134+0000","last_clean":"2024-03-06T07:37:23.405134+0000","last_became_active":"2024-03-06T07:37:22.392507+0000","last_became_peered":"2024-03-06T07:37:22.392507+0000","last_unstale":"2024-03-06T07:37:23.405134+0000","last_undegraded":"2024-03-06T07:37:23.405134+0000","last_fullsized":"2024-03-06T07:37:23.405134+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:17.676813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398658+0000","last_change":"2024-03-06T07:37:22.397145+0000","last_active":"2024-03-06T07:37:23.398658+0000","last_peered":"2024-03-06T07:37:23.398658+0000","last_clean":"2024-03-06T07:37:23.398658+0000","last_became_active":"2024-03-06T07:37:22.396761+0000","last_became_peered":"2024-03-06T07:37:22.396761+0000","last_unstale":"2024-03-06T07:37:23.398658+0000","last_undegraded":"2024-03-06T07:37:23.398658+0000","last_fullsized":"2024-03-06T07:37:23.398658+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:11.759303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397445+0000","last_change":"2024-03-06T07:37:22.409468+0000","last_active":"2024-03-06T07:37:23.397445+0000","last_peered":"2024-03-06T07:37:23.397445+0000","last_clean":"2024-03-06T07:37:23.397445+0000","last_became_active":"2024-03-06T07:37:22.408685+0000","last_became_peered":"2024-03-06T07:37:22.408685+0000","last_unstale":"2024-03-06T07:37:23.397445+0000","last_undegraded":"2024-03-06T07:37:23.397445+0000","last_fullsized":"2024-03-06T07:37:23.397445+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:00.774528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.404679+0000","last_change":"2024-03-06T07:37:22.389604+0000","last_active":"2024-03-06T07:37:23.404679+0000","last_peered":"2024-03-06T07:37:23.404679+0000","last_clean":"2024-03-06T07:37:23.404679+0000","last_became_active":"2024-03-06T07:37:22.389135+0000","last_became_peered":"2024-03-06T07:37:22.389135+0000","last_unstale":"2024-03-06T07:37:23.404679+0000","last_undegraded":"2024-03-06T07:37:23.404679+0000","last_fullsized":"2024-03-06T07:37:23.404679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:58.167076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460346+0000","last_change":"2024-03-06T07:37:22.384427+0000","last_active":"2024-03-06T07:37:23.460346+0000","last_peered":"2024-03-06T07:37:23.460346+0000","last_clean":"2024-03-06T07:37:23.460346+0000","last_became_active":"2024-03-06T07:37:22.384148+0000","last_became_peered":"2024-03-06T07:37:22.384148+0000","last_unstale":"2024-03-06T07:37:23.460346+0000","last_undegraded":"2024-03-06T07:37:23.460346+0000","last_fullsized":"2024-03-06T07:37:23.460346+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:21.865440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406118+0000","last_change":"2024-03-06T07:37:22.402878+0000","last_active":"2024-03-06T07:37:23.406118+0000","last_peered":"2024-03-06T07:37:23.406118+0000","last_clean":"2024-03-06T07:37:23.406118+0000","last_became_active":"2024-03-06T07:37:22.397630+0000","last_became_peered":"2024-03-06T07:37:22.397630+0000","last_unstale":"2024-03-06T07:37:23.406118+0000","last_undegraded":"2024-03-06T07:37:23.406118+0000","last_fullsized":"2024-03-06T07:37:23.406118+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:48.107097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464532+0000","last_change":"2024-03-06T07:37:22.404191+0000","last_active":"2024-03-06T07:37:23.464532+0000","last_peered":"2024-03-06T07:37:23.464532+0000","last_clean":"2024-03-06T07:37:23.464532+0000","last_became_active":"2024-03-06T07:37:22.403870+0000","last_became_peered":"2024-03-06T07:37:22.403870+0000","last_unstale":"2024-03-06T07:37:23.464532+0000","last_undegraded":"2024-03-06T07:37:23.464532+0000","last_fullsized":"2024-03-06T07:37:23.464532+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:43.656249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399922+0000","last_change":"2024-03-06T07:37:22.386594+0000","last_active":"2024-03-06T07:37:23.399922+0000","last_peered":"2024-03-06T07:37:23.399922+0000","last_clean":"2024-03-06T07:37:23.399922+0000","last_became_active":"2024-03-06T07:37:22.386176+0000","last_became_peered":"2024-03-06T07:37:22.386176+0000","last_unstale":"2024-03-06T07:37:23.399922+0000","last_undegraded":"2024-03-06T07:37:23.399922+0000","last_fullsized":"2024-03-06T07:37:23.399922+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:58.164860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460929+0000","last_change":"2024-03-06T07:37:22.393011+0000","last_active":"2024-03-06T07:37:23.460929+0000","last_peered":"2024-03-06T07:37:23.460929+0000","last_clean":"2024-03-06T07:37:23.460929+0000","last_became_active":"2024-03-06T07:37:22.386815+0000","last_became_peered":"2024-03-06T07:37:22.386815+0000","last_unstale":"2024-03-06T07:37:23.460929+0000","last_undegraded":"2024-03-06T07:37:23.460929+0000","last_fullsized":"2024-03-06T07:37:23.460929+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:23.464868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405456+0000","last_change":"2024-03-06T07:37:22.409729+0000","last_active":"2024-03-06T07:37:23.405456+0000","last_peered":"2024-03-06T07:37:23.405456+0000","last_clean":"2024-03-06T07:37:23.405456+0000","last_became_active":"2024-03-06T07:37:22.409269+0000","last_became_peered":"2024-03-06T07:37:22.409269+0000","last_unstale":"2024-03-06T07:37:23.405456+0000","last_undegraded":"2024-03-06T07:37:23.405456+0000","last_fullsized":"2024-03-06T07:37:23.405456+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:36:26.241249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397630+0000","last_change":"2024-03-06T07:37:22.404312+0000","last_active":"2024-03-06T07:37:23.397630+0000","last_peered":"2024-03-06T07:37:23.397630+0000","last_clean":"2024-03-06T07:37:23.397630+0000","last_became_active":"2024-03-06T07:37:22.403787+0000","last_became_peered":"2024-03-06T07:37:22.403787+0000","last_unstale":"2024-03-06T07:37:23.397630+0000","last_undegraded":"2024-03-06T07:37:23.397630+0000","last_fullsized":"2024-03-06T07:37:23.397630+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:45.033772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398401+0000","last_change":"2024-03-06T07:37:22.408249+0000","last_active":"2024-03-06T07:37:23.398401+0000","last_peered":"2024-03-06T07:37:23.398401+0000","last_clean":"2024-03-06T07:37:23.398401+0000","last_became_active":"2024-03-06T07:37:22.407921+0000","last_became_peered":"2024-03-06T07:37:22.407921+0000","last_unstale":"2024-03-06T07:37:23.398401+0000","last_undegraded":"2024-03-06T07:37:23.398401+0000","last_fullsized":"2024-03-06T07:37:23.398401+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:42.436707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.404953+0000","last_change":"2024-03-06T07:37:22.396679+0000","last_active":"2024-03-06T07:37:23.404953+0000","last_peered":"2024-03-06T07:37:23.404953+0000","last_clean":"2024-03-06T07:37:23.404953+0000","last_became_active":"2024-03-06T07:37:22.392969+0000","last_became_peered":"2024-03-06T07:37:22.392969+0000","last_unstale":"2024-03-06T07:37:23.404953+0000","last_undegraded":"2024-03-06T07:37:23.404953+0000","last_fullsized":"2024-03-06T07:37:23.404953+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:29.190943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460954+0000","last_change":"2024-03-06T07:37:22.390025+0000","last_active":"2024-03-06T07:37:23.460954+0000","last_peered":"2024-03-06T07:37:23.460954+0000","last_clean":"2024-03-06T07:37:23.460954+0000","last_became_active":"2024-03-06T07:37:22.389757+0000","last_became_peered":"2024-03-06T07:37:22.389757+0000","last_unstale":"2024-03-06T07:37:23.460954+0000","last_undegraded":"2024-03-06T07:37:23.460954+0000","last_fullsized":"2024-03-06T07:37:23.460954+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:51.600779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400085+0000","last_change":"2024-03-06T07:37:22.386833+0000","last_active":"2024-03-06T07:37:23.400085+0000","last_peered":"2024-03-06T07:37:23.400085+0000","last_clean":"2024-03-06T07:37:23.400085+0000","last_became_active":"2024-03-06T07:37:22.385573+0000","last_became_peered":"2024-03-06T07:37:22.385573+0000","last_unstale":"2024-03-06T07:37:23.400085+0000","last_undegraded":"2024-03-06T07:37:23.400085+0000","last_fullsized":"2024-03-06T07:37:23.400085+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:16.796463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461027+0000","last_change":"2024-03-06T07:37:22.394540+0000","last_active":"2024-03-06T07:37:23.461027+0000","last_peered":"2024-03-06T07:37:23.461027+0000","last_clean":"2024-03-06T07:37:23.461027+0000","last_became_active":"2024-03-06T07:37:22.394242+0000","last_became_peered":"2024-03-06T07:37:22.394242+0000","last_unstale":"2024-03-06T07:37:23.461027+0000","last_undegraded":"2024-03-06T07:37:23.461027+0000","last_fullsized":"2024-03-06T07:37:23.461027+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.331051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394425+0000","last_change":"2024-03-06T07:37:22.396145+0000","last_active":"2024-03-06T07:37:23.394425+0000","last_peered":"2024-03-06T07:37:23.394425+0000","last_clean":"2024-03-06T07:37:23.394425+0000","last_became_active":"2024-03-06T07:37:22.395856+0000","last_became_peered":"2024-03-06T07:37:22.395856+0000","last_unstale":"2024-03-06T07:37:23.394425+0000","last_undegraded":"2024-03-06T07:37:23.394425+0000","last_fullsized":"2024-03-06T07:37:23.394425+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:27.224804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399112+0000","last_change":"2024-03-06T07:37:22.395992+0000","last_active":"2024-03-06T07:37:23.399112+0000","last_peered":"2024-03-06T07:37:23.399112+0000","last_clean":"2024-03-06T07:37:23.399112+0000","last_became_active":"2024-03-06T07:37:22.395649+0000","last_became_peered":"2024-03-06T07:37:22.395649+0000","last_unstale":"2024-03-06T07:37:23.399112+0000","last_undegraded":"2024-03-06T07:37:23.399112+0000","last_fullsized":"2024-03-06T07:37:23.399112+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:26.141228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397848+0000","last_change":"2024-03-06T07:37:22.399174+0000","last_active":"2024-03-06T07:37:23.397848+0000","last_peered":"2024-03-06T07:37:23.397848+0000","last_clean":"2024-03-06T07:37:23.397848+0000","last_became_active":"2024-03-06T07:37:22.397414+0000","last_became_peered":"2024-03-06T07:37:22.397414+0000","last_unstale":"2024-03-06T07:37:23.397848+0000","last_undegraded":"2024-03-06T07:37:23.397848+0000","last_fullsized":"2024-03-06T07:37:23.397848+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:16.801579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461873+0000","last_change":"2024-03-06T07:37:22.398035+0000","last_active":"2024-03-06T07:37:23.461873+0000","last_peered":"2024-03-06T07:37:23.461873+0000","last_clean":"2024-03-06T07:37:23.461873+0000","last_became_active":"2024-03-06T07:37:22.397812+0000","last_became_peered":"2024-03-06T07:37:22.397812+0000","last_unstale":"2024-03-06T07:37:23.461873+0000","last_undegraded":"2024-03-06T07:37:23.461873+0000","last_fullsized":"2024-03-06T07:37:23.461873+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:52.355345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460439+0000","last_change":"2024-03-06T07:37:22.399262+0000","last_active":"2024-03-06T07:37:23.460439+0000","last_peered":"2024-03-06T07:37:23.460439+0000","last_clean":"2024-03-06T07:37:23.460439+0000","last_became_active":"2024-03-06T07:37:22.397780+0000","last_became_peered":"2024-03-06T07:37:22.397780+0000","last_unstale":"2024-03-06T07:37:23.460439+0000","last_undegraded":"2024-03-06T07:37:23.460439+0000","last_fullsized":"2024-03-06T07:37:23.460439+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.071420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405857+0000","last_change":"2024-03-06T07:37:22.396378+0000","last_active":"2024-03-06T07:37:23.405857+0000","last_peered":"2024-03-06T07:37:23.405857+0000","last_clean":"2024-03-06T07:37:23.405857+0000","last_became_active":"2024-03-06T07:37:22.395832+0000","last_became_peered":"2024-03-06T07:37:22.395832+0000","last_unstale":"2024-03-06T07:37:23.405857+0000","last_undegraded":"2024-03-06T07:37:23.405857+0000","last_fullsized":"2024-03-06T07:37:23.405857+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.403390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"143'4","reported_seq":18,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405948+0000","last_change":"2024-03-06T07:37:21.370711+0000","last_active":"2024-03-06T07:37:23.405948+0000","last_peered":"2024-03-06T07:37:23.405948+0000","last_clean":"2024-03-06T07:37:23.405948+0000","last_became_active":"2024-03-06T07:37:21.370383+0000","last_became_peered":"2024-03-06T07:37:21.370383+0000","last_unstale":"2024-03-06T07:37:23.405948+0000","last_undegraded":"2024-03-06T07:37:23.405948+0000","last_fullsized":"2024-03-06T07:37:23.405948+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:47.441553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462751+0000","last_change":"2024-03-06T07:37:22.402996+0000","last_active":"2024-03-06T07:37:23.462751+0000","last_peered":"2024-03-06T07:37:23.462751+0000","last_clean":"2024-03-06T07:37:23.462751+0000","last_became_active":"2024-03-06T07:37:22.401815+0000","last_became_peered":"2024-03-06T07:37:22.401815+0000","last_unstale":"2024-03-06T07:37:23.462751+0000","last_undegraded":"2024-03-06T07:37:23.462751+0000","last_fullsized":"2024-03-06T07:37:23.462751+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:17.676813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462486+0000","last_change":"2024-03-06T07:37:21.369463+0000","last_active":"2024-03-06T07:37:23.462486+0000","last_peered":"2024-03-06T07:37:23.462486+0000","last_clean":"2024-03-06T07:37:23.462486+0000","last_became_active":"2024-03-06T07:37:21.369196+0000","last_became_peered":"2024-03-06T07:37:21.369196+0000","last_unstale":"2024-03-06T07:37:23.462486+0000","last_undegraded":"2024-03-06T07:37:23.462486+0000","last_fullsized":"2024-03-06T07:37:23.462486+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:33.070997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399109+0000","last_change":"2024-03-06T07:37:22.397450+0000","last_active":"2024-03-06T07:37:23.399109+0000","last_peered":"2024-03-06T07:37:23.399109+0000","last_clean":"2024-03-06T07:37:23.399109+0000","last_became_active":"2024-03-06T07:37:22.397121+0000","last_became_peered":"2024-03-06T07:37:22.397121+0000","last_unstale":"2024-03-06T07:37:23.399109+0000","last_undegraded":"2024-03-06T07:37:23.399109+0000","last_fullsized":"2024-03-06T07:37:23.399109+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:06.238541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399112+0000","last_change":"2024-03-06T07:37:21.363721+0000","last_active":"2024-03-06T07:37:23.399112+0000","last_peered":"2024-03-06T07:37:23.399112+0000","last_clean":"2024-03-06T07:37:23.399112+0000","last_became_active":"2024-03-06T07:37:21.362783+0000","last_became_peered":"2024-03-06T07:37:21.362783+0000","last_unstale":"2024-03-06T07:37:23.399112+0000","last_undegraded":"2024-03-06T07:37:23.399112+0000","last_fullsized":"2024-03-06T07:37:23.399112+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:24.658794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405257+0000","last_change":"2024-03-06T07:37:22.392998+0000","last_active":"2024-03-06T07:37:23.405257+0000","last_peered":"2024-03-06T07:37:23.405257+0000","last_clean":"2024-03-06T07:37:23.405257+0000","last_became_active":"2024-03-06T07:37:22.392721+0000","last_became_peered":"2024-03-06T07:37:22.392721+0000","last_unstale":"2024-03-06T07:37:23.405257+0000","last_undegraded":"2024-03-06T07:37:23.405257+0000","last_fullsized":"2024-03-06T07:37:23.405257+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.954146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399249+0000","last_change":"2024-03-06T07:37:21.364047+0000","last_active":"2024-03-06T07:37:23.399249+0000","last_peered":"2024-03-06T07:37:23.399249+0000","last_clean":"2024-03-06T07:37:23.399249+0000","last_became_active":"2024-03-06T07:37:21.363758+0000","last_became_peered":"2024-03-06T07:37:21.363758+0000","last_unstale":"2024-03-06T07:37:23.399249+0000","last_undegraded":"2024-03-06T07:37:23.399249+0000","last_fullsized":"2024-03-06T07:37:23.399249+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:15.462083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461999+0000","last_change":"2024-03-06T07:37:22.398997+0000","last_active":"2024-03-06T07:37:23.461999+0000","last_peered":"2024-03-06T07:37:23.461999+0000","last_clean":"2024-03-06T07:37:23.461999+0000","last_became_active":"2024-03-06T07:37:22.398617+0000","last_became_peered":"2024-03-06T07:37:22.398617+0000","last_unstale":"2024-03-06T07:37:23.461999+0000","last_undegraded":"2024-03-06T07:37:23.461999+0000","last_fullsized":"2024-03-06T07:37:23.461999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:58.167076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460590+0000","last_change":"2024-03-06T07:37:21.360910+0000","last_active":"2024-03-06T07:37:23.460590+0000","last_peered":"2024-03-06T07:37:23.460590+0000","last_clean":"2024-03-06T07:37:23.460590+0000","last_became_active":"2024-03-06T07:37:21.360647+0000","last_became_peered":"2024-03-06T07:37:21.360647+0000","last_unstale":"2024-03-06T07:37:23.460590+0000","last_undegraded":"2024-03-06T07:37:23.460590+0000","last_fullsized":"2024-03-06T07:37:23.460590+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:25.413969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397783+0000","last_change":"2024-03-06T07:37:22.399270+0000","last_active":"2024-03-06T07:37:23.397783+0000","last_peered":"2024-03-06T07:37:23.397783+0000","last_clean":"2024-03-06T07:37:23.397783+0000","last_became_active":"2024-03-06T07:37:22.397037+0000","last_became_peered":"2024-03-06T07:37:22.397037+0000","last_unstale":"2024-03-06T07:37:23.397783+0000","last_undegraded":"2024-03-06T07:37:23.397783+0000","last_fullsized":"2024-03-06T07:37:23.397783+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:08.439418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398492+0000","last_change":"2024-03-06T07:37:21.374109+0000","last_active":"2024-03-06T07:37:23.398492+0000","last_peered":"2024-03-06T07:37:23.398492+0000","last_clean":"2024-03-06T07:37:23.398492+0000","last_became_active":"2024-03-06T07:37:21.373826+0000","last_became_peered":"2024-03-06T07:37:21.373826+0000","last_unstale":"2024-03-06T07:37:23.398492+0000","last_undegraded":"2024-03-06T07:37:23.398492+0000","last_fullsized":"2024-03-06T07:37:23.398492+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:22.241595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461814+0000","last_change":"2024-03-06T07:37:22.393473+0000","last_active":"2024-03-06T07:37:23.461814+0000","last_peered":"2024-03-06T07:37:23.461814+0000","last_clean":"2024-03-06T07:37:23.461814+0000","last_became_active":"2024-03-06T07:37:22.393198+0000","last_became_peered":"2024-03-06T07:37:22.393198+0000","last_unstale":"2024-03-06T07:37:23.461814+0000","last_undegraded":"2024-03-06T07:37:23.461814+0000","last_fullsized":"2024-03-06T07:37:23.461814+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:58.117494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460575+0000","last_change":"2024-03-06T07:37:21.372994+0000","last_active":"2024-03-06T07:37:23.460575+0000","last_peered":"2024-03-06T07:37:23.460575+0000","last_clean":"2024-03-06T07:37:23.460575+0000","last_became_active":"2024-03-06T07:37:21.372042+0000","last_became_peered":"2024-03-06T07:37:21.372042+0000","last_unstale":"2024-03-06T07:37:23.460575+0000","last_undegraded":"2024-03-06T07:37:23.460575+0000","last_fullsized":"2024-03-06T07:37:23.460575+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:55.878213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398649+0000","last_change":"2024-03-06T07:37:22.393948+0000","last_active":"2024-03-06T07:37:23.398649+0000","last_peered":"2024-03-06T07:37:23.398649+0000","last_clean":"2024-03-06T07:37:23.398649+0000","last_became_active":"2024-03-06T07:37:22.393528+0000","last_became_peered":"2024-03-06T07:37:22.393528+0000","last_unstale":"2024-03-06T07:37:23.398649+0000","last_undegraded":"2024-03-06T07:37:23.398649+0000","last_fullsized":"2024-03-06T07:37:23.398649+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.331051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.396724+0000","last_change":"2024-03-06T07:37:21.369842+0000","last_active":"2024-03-06T07:37:23.396724+0000","last_peered":"2024-03-06T07:37:23.396724+0000","last_clean":"2024-03-06T07:37:23.396724+0000","last_became_active":"2024-03-06T07:37:21.369508+0000","last_became_peered":"2024-03-06T07:37:21.369508+0000","last_unstale":"2024-03-06T07:37:23.396724+0000","last_undegraded":"2024-03-06T07:37:23.396724+0000","last_fullsized":"2024-03-06T07:37:23.396724+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:51.109279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399950+0000","last_change":"2024-03-06T07:37:22.406741+0000","last_active":"2024-03-06T07:37:23.399950+0000","last_peered":"2024-03-06T07:37:23.399950+0000","last_clean":"2024-03-06T07:37:23.399950+0000","last_became_active":"2024-03-06T07:37:22.406430+0000","last_became_peered":"2024-03-06T07:37:22.406430+0000","last_unstale":"2024-03-06T07:37:23.399950+0000","last_undegraded":"2024-03-06T07:37:23.399950+0000","last_fullsized":"2024-03-06T07:37:23.399950+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:42.722561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462845+0000","last_change":"2024-03-06T07:37:21.361888+0000","last_active":"2024-03-06T07:37:23.462845+0000","last_peered":"2024-03-06T07:37:23.462845+0000","last_clean":"2024-03-06T07:37:23.462845+0000","last_became_active":"2024-03-06T07:37:21.361309+0000","last_became_peered":"2024-03-06T07:37:21.361309+0000","last_unstale":"2024-03-06T07:37:23.462845+0000","last_undegraded":"2024-03-06T07:37:23.462845+0000","last_fullsized":"2024-03-06T07:37:23.462845+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:44.982476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405877+0000","last_change":"2024-03-06T07:37:22.404220+0000","last_active":"2024-03-06T07:37:23.405877+0000","last_peered":"2024-03-06T07:37:23.405877+0000","last_clean":"2024-03-06T07:37:23.405877+0000","last_became_active":"2024-03-06T07:37:22.403909+0000","last_became_peered":"2024-03-06T07:37:22.403909+0000","last_unstale":"2024-03-06T07:37:23.405877+0000","last_undegraded":"2024-03-06T07:37:23.405877+0000","last_fullsized":"2024-03-06T07:37:23.405877+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:32:32.350202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406063+0000","last_change":"2024-03-06T07:37:21.374894+0000","last_active":"2024-03-06T07:37:23.406063+0000","last_peered":"2024-03-06T07:37:23.406063+0000","last_clean":"2024-03-06T07:37:23.406063+0000","last_became_active":"2024-03-06T07:37:21.374628+0000","last_became_peered":"2024-03-06T07:37:21.374628+0000","last_unstale":"2024-03-06T07:37:23.406063+0000","last_undegraded":"2024-03-06T07:37:23.406063+0000","last_fullsized":"2024-03-06T07:37:23.406063+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:54.464276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460871+0000","last_change":"2024-03-06T07:37:22.403021+0000","last_active":"2024-03-06T07:37:23.460871+0000","last_peered":"2024-03-06T07:37:23.460871+0000","last_clean":"2024-03-06T07:37:23.460871+0000","last_became_active":"2024-03-06T07:37:22.402748+0000","last_became_peered":"2024-03-06T07:37:22.402748+0000","last_unstale":"2024-03-06T07:37:23.460871+0000","last_undegraded":"2024-03-06T07:37:23.460871+0000","last_fullsized":"2024-03-06T07:37:23.460871+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:39.516810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"143'6","reported_seq":20,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394666+0000","last_change":"2024-03-06T07:37:21.365460+0000","last_active":"2024-03-06T07:37:23.394666+0000","last_peered":"2024-03-06T07:37:23.394666+0000","last_clean":"2024-03-06T07:37:23.394666+0000","last_became_active":"2024-03-06T07:37:21.365184+0000","last_became_peered":"2024-03-06T07:37:21.365184+0000","last_unstale":"2024-03-06T07:37:23.394666+0000","last_undegraded":"2024-03-06T07:37:23.394666+0000","last_fullsized":"2024-03-06T07:37:23.394666+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:47.441553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463908+0000","last_change":"2024-03-06T07:37:22.395416+0000","last_active":"2024-03-06T07:37:23.463908+0000","last_peered":"2024-03-06T07:37:23.463908+0000","last_clean":"2024-03-06T07:37:23.463908+0000","last_became_active":"2024-03-06T07:37:22.394404+0000","last_became_peered":"2024-03-06T07:37:22.394404+0000","last_unstale":"2024-03-06T07:37:23.463908+0000","last_undegraded":"2024-03-06T07:37:23.463908+0000","last_fullsized":"2024-03-06T07:37:23.463908+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:41.520623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463818+0000","last_change":"2024-03-06T07:37:21.374508+0000","last_active":"2024-03-06T07:37:23.463818+0000","last_peered":"2024-03-06T07:37:23.463818+0000","last_clean":"2024-03-06T07:37:23.463818+0000","last_became_active":"2024-03-06T07:37:21.374098+0000","last_became_peered":"2024-03-06T07:37:21.374098+0000","last_unstale":"2024-03-06T07:37:23.463818+0000","last_undegraded":"2024-03-06T07:37:23.463818+0000","last_fullsized":"2024-03-06T07:37:23.463818+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:20.363553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461963+0000","last_change":"2024-03-06T07:37:22.388870+0000","last_active":"2024-03-06T07:37:23.461963+0000","last_peered":"2024-03-06T07:37:23.461963+0000","last_clean":"2024-03-06T07:37:23.461963+0000","last_became_active":"2024-03-06T07:37:22.386400+0000","last_became_peered":"2024-03-06T07:37:22.386400+0000","last_unstale":"2024-03-06T07:37:23.461963+0000","last_undegraded":"2024-03-06T07:37:23.461963+0000","last_fullsized":"2024-03-06T07:37:23.461963+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:51.117104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464133+0000","last_change":"2024-03-06T07:37:21.376580+0000","last_active":"2024-03-06T07:37:23.464133+0000","last_peered":"2024-03-06T07:37:23.464133+0000","last_clean":"2024-03-06T07:37:23.464133+0000","last_became_active":"2024-03-06T07:37:21.375739+0000","last_became_peered":"2024-03-06T07:37:21.375739+0000","last_unstale":"2024-03-06T07:37:23.464133+0000","last_undegraded":"2024-03-06T07:37:23.464133+0000","last_fullsized":"2024-03-06T07:37:23.464133+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:57.149568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.397656+0000","last_change":"2024-03-06T07:37:22.399075+0000","last_active":"2024-03-06T07:37:23.397656+0000","last_peered":"2024-03-06T07:37:23.397656+0000","last_clean":"2024-03-06T07:37:23.397656+0000","last_became_active":"2024-03-06T07:37:22.398773+0000","last_became_peered":"2024-03-06T07:37:22.398773+0000","last_unstale":"2024-03-06T07:37:23.397656+0000","last_undegraded":"2024-03-06T07:37:23.397656+0000","last_fullsized":"2024-03-06T07:37:23.397656+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:23.464868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398898+0000","last_change":"2024-03-06T07:37:21.366115+0000","last_active":"2024-03-06T07:37:23.398898+0000","last_peered":"2024-03-06T07:37:23.398898+0000","last_clean":"2024-03-06T07:37:23.398898+0000","last_became_active":"2024-03-06T07:37:21.360739+0000","last_became_peered":"2024-03-06T07:37:21.360739+0000","last_unstale":"2024-03-06T07:37:23.398898+0000","last_undegraded":"2024-03-06T07:37:23.398898+0000","last_fullsized":"2024-03-06T07:37:23.398898+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:11.295318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463497+0000","last_change":"2024-03-06T07:37:22.391541+0000","last_active":"2024-03-06T07:37:23.463497+0000","last_peered":"2024-03-06T07:37:23.463497+0000","last_clean":"2024-03-06T07:37:23.463497+0000","last_became_active":"2024-03-06T07:37:22.391266+0000","last_became_peered":"2024-03-06T07:37:22.391266+0000","last_unstale":"2024-03-06T07:37:23.463497+0000","last_undegraded":"2024-03-06T07:37:23.463497+0000","last_fullsized":"2024-03-06T07:37:23.463497+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:36.812503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"143'8","reported_seq":22,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460487+0000","last_change":"2024-03-06T07:37:21.367352+0000","last_active":"2024-03-06T07:37:23.460487+0000","last_peered":"2024-03-06T07:37:23.460487+0000","last_clean":"2024-03-06T07:37:23.460487+0000","last_became_active":"2024-03-06T07:37:21.367107+0000","last_became_peered":"2024-03-06T07:37:21.367107+0000","last_unstale":"2024-03-06T07:37:23.460487+0000","last_undegraded":"2024-03-06T07:37:23.460487+0000","last_fullsized":"2024-03-06T07:37:23.460487+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:34:02.301827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406244+0000","last_change":"2024-03-06T07:37:22.384018+0000","last_active":"2024-03-06T07:37:23.406244+0000","last_peered":"2024-03-06T07:37:23.406244+0000","last_clean":"2024-03-06T07:37:23.406244+0000","last_became_active":"2024-03-06T07:37:22.383729+0000","last_became_peered":"2024-03-06T07:37:22.383729+0000","last_unstale":"2024-03-06T07:37:23.406244+0000","last_undegraded":"2024-03-06T07:37:23.406244+0000","last_fullsized":"2024-03-06T07:37:23.406244+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:25.594731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399833+0000","last_change":"2024-03-06T07:37:21.375503+0000","last_active":"2024-03-06T07:37:23.399833+0000","last_peered":"2024-03-06T07:37:23.399833+0000","last_clean":"2024-03-06T07:37:23.399833+0000","last_became_active":"2024-03-06T07:37:21.375171+0000","last_became_peered":"2024-03-06T07:37:21.375171+0000","last_unstale":"2024-03-06T07:37:23.399833+0000","last_undegraded":"2024-03-06T07:37:23.399833+0000","last_fullsized":"2024-03-06T07:37:23.399833+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.774103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461058+0000","last_change":"2024-03-06T07:37:22.396545+0000","last_active":"2024-03-06T07:37:23.461058+0000","last_peered":"2024-03-06T07:37:23.461058+0000","last_clean":"2024-03-06T07:37:23.461058+0000","last_became_active":"2024-03-06T07:37:22.394046+0000","last_became_peered":"2024-03-06T07:37:22.394046+0000","last_unstale":"2024-03-06T07:37:23.461058+0000","last_undegraded":"2024-03-06T07:37:23.461058+0000","last_fullsized":"2024-03-06T07:37:23.461058+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:11.759303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462594+0000","last_change":"2024-03-06T07:37:21.373115+0000","last_active":"2024-03-06T07:37:23.462594+0000","last_peered":"2024-03-06T07:37:23.462594+0000","last_clean":"2024-03-06T07:37:23.462594+0000","last_became_active":"2024-03-06T07:37:21.372851+0000","last_became_peered":"2024-03-06T07:37:21.372851+0000","last_unstale":"2024-03-06T07:37:23.462594+0000","last_undegraded":"2024-03-06T07:37:23.462594+0000","last_fullsized":"2024-03-06T07:37:23.462594+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:47.441553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464313+0000","last_change":"2024-03-06T07:37:22.404472+0000","last_active":"2024-03-06T07:37:23.464313+0000","last_peered":"2024-03-06T07:37:23.464313+0000","last_clean":"2024-03-06T07:37:23.464313+0000","last_became_active":"2024-03-06T07:37:22.402498+0000","last_became_peered":"2024-03-06T07:37:22.402498+0000","last_unstale":"2024-03-06T07:37:23.464313+0000","last_undegraded":"2024-03-06T07:37:23.464313+0000","last_fullsized":"2024-03-06T07:37:23.464313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:18.419563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"143'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461003+0000","last_change":"2024-03-06T07:37:21.354195+0000","last_active":"2024-03-06T07:37:23.461003+0000","last_peered":"2024-03-06T07:37:23.461003+0000","last_clean":"2024-03-06T07:37:23.461003+0000","last_became_active":"2024-03-06T07:37:21.353866+0000","last_became_peered":"2024-03-06T07:37:21.353866+0000","last_unstale":"2024-03-06T07:37:23.461003+0000","last_undegraded":"2024-03-06T07:37:23.461003+0000","last_fullsized":"2024-03-06T07:37:23.461003+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:49.470863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399805+0000","last_change":"2024-03-06T07:37:22.394693+0000","last_active":"2024-03-06T07:37:23.399805+0000","last_peered":"2024-03-06T07:37:23.399805+0000","last_clean":"2024-03-06T07:37:23.399805+0000","last_became_active":"2024-03-06T07:37:22.394425+0000","last_became_peered":"2024-03-06T07:37:22.394425+0000","last_unstale":"2024-03-06T07:37:23.399805+0000","last_undegraded":"2024-03-06T07:37:23.399805+0000","last_fullsized":"2024-03-06T07:37:23.399805+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:21.378845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462203+0000","last_change":"2024-03-06T07:37:21.364866+0000","last_active":"2024-03-06T07:37:23.462203+0000","last_peered":"2024-03-06T07:37:23.462203+0000","last_clean":"2024-03-06T07:37:23.462203+0000","last_became_active":"2024-03-06T07:37:21.364556+0000","last_became_peered":"2024-03-06T07:37:21.364556+0000","last_unstale":"2024-03-06T07:37:23.462203+0000","last_undegraded":"2024-03-06T07:37:23.462203+0000","last_fullsized":"2024-03-06T07:37:23.462203+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:54.464276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.398048+0000","last_change":"2024-03-06T07:37:22.397431+0000","last_active":"2024-03-06T07:37:23.398048+0000","last_peered":"2024-03-06T07:37:23.398048+0000","last_clean":"2024-03-06T07:37:23.398048+0000","last_became_active":"2024-03-06T07:37:22.397099+0000","last_became_peered":"2024-03-06T07:37:22.397099+0000","last_unstale":"2024-03-06T07:37:23.398048+0000","last_undegraded":"2024-03-06T07:37:23.398048+0000","last_fullsized":"2024-03-06T07:37:23.398048+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:21.173058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394402+0000","last_change":"2024-03-06T07:37:21.372528+0000","last_active":"2024-03-06T07:37:23.394402+0000","last_peered":"2024-03-06T07:37:23.394402+0000","last_clean":"2024-03-06T07:37:23.394402+0000","last_became_active":"2024-03-06T07:37:21.372238+0000","last_became_peered":"2024-03-06T07:37:21.372238+0000","last_unstale":"2024-03-06T07:37:23.394402+0000","last_undegraded":"2024-03-06T07:37:23.394402+0000","last_fullsized":"2024-03-06T07:37:23.394402+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:54.464276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394803+0000","last_change":"2024-03-06T07:37:22.401310+0000","last_active":"2024-03-06T07:37:23.394803+0000","last_peered":"2024-03-06T07:37:23.394803+0000","last_clean":"2024-03-06T07:37:23.394803+0000","last_became_active":"2024-03-06T07:37:22.401025+0000","last_became_peered":"2024-03-06T07:37:22.401025+0000","last_unstale":"2024-03-06T07:37:23.394803+0000","last_undegraded":"2024-03-06T07:37:23.394803+0000","last_fullsized":"2024-03-06T07:37:23.394803+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:52.355345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405371+0000","last_change":"2024-03-06T07:37:21.370098+0000","last_active":"2024-03-06T07:37:23.405371+0000","last_peered":"2024-03-06T07:37:23.405371+0000","last_clean":"2024-03-06T07:37:23.405371+0000","last_became_active":"2024-03-06T07:37:21.369792+0000","last_became_peered":"2024-03-06T07:37:21.369792+0000","last_unstale":"2024-03-06T07:37:23.405371+0000","last_undegraded":"2024-03-06T07:37:23.405371+0000","last_fullsized":"2024-03-06T07:37:23.405371+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:49.986528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.396740+0000","last_change":"2024-03-06T07:37:22.408331+0000","last_active":"2024-03-06T07:37:23.396740+0000","last_peered":"2024-03-06T07:37:23.396740+0000","last_clean":"2024-03-06T07:37:23.396740+0000","last_became_active":"2024-03-06T07:37:22.407996+0000","last_became_peered":"2024-03-06T07:37:22.407996+0000","last_unstale":"2024-03-06T07:37:23.396740+0000","last_undegraded":"2024-03-06T07:37:23.396740+0000","last_fullsized":"2024-03-06T07:37:23.396740+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:00.033313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405004+0000","last_change":"2024-03-06T07:37:21.371516+0000","last_active":"2024-03-06T07:37:23.405004+0000","last_peered":"2024-03-06T07:37:23.405004+0000","last_clean":"2024-03-06T07:37:23.405004+0000","last_became_active":"2024-03-06T07:37:21.370796+0000","last_became_peered":"2024-03-06T07:37:21.370796+0000","last_unstale":"2024-03-06T07:37:23.405004+0000","last_undegraded":"2024-03-06T07:37:23.405004+0000","last_fullsized":"2024-03-06T07:37:23.405004+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:47.675808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394734+0000","last_change":"2024-03-06T07:37:22.409916+0000","last_active":"2024-03-06T07:37:23.394734+0000","last_peered":"2024-03-06T07:37:23.394734+0000","last_clean":"2024-03-06T07:37:23.394734+0000","last_became_active":"2024-03-06T07:37:22.409181+0000","last_became_peered":"2024-03-06T07:37:22.409181+0000","last_unstale":"2024-03-06T07:37:23.394734+0000","last_undegraded":"2024-03-06T07:37:23.394734+0000","last_fullsized":"2024-03-06T07:37:23.394734+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:58.117494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462985+0000","last_change":"2024-03-06T07:37:21.367483+0000","last_active":"2024-03-06T07:37:23.462985+0000","last_peered":"2024-03-06T07:37:23.462985+0000","last_clean":"2024-03-06T07:37:23.462985+0000","last_became_active":"2024-03-06T07:37:21.366939+0000","last_became_peered":"2024-03-06T07:37:21.366939+0000","last_unstale":"2024-03-06T07:37:23.462985+0000","last_undegraded":"2024-03-06T07:37:23.462985+0000","last_fullsized":"2024-03-06T07:37:23.462985+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:47.211106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400238+0000","last_change":"2024-03-06T07:37:22.406359+0000","last_active":"2024-03-06T07:37:23.400238+0000","last_peered":"2024-03-06T07:37:23.400238+0000","last_clean":"2024-03-06T07:37:23.400238+0000","last_became_active":"2024-03-06T07:37:22.405754+0000","last_became_peered":"2024-03-06T07:37:22.405754+0000","last_unstale":"2024-03-06T07:37:23.400238+0000","last_undegraded":"2024-03-06T07:37:23.400238+0000","last_fullsized":"2024-03-06T07:37:23.400238+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:05.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"143'2","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.406233+0000","last_change":"2024-03-06T07:37:21.375388+0000","last_active":"2024-03-06T07:37:23.406233+0000","last_peered":"2024-03-06T07:37:23.406233+0000","last_clean":"2024-03-06T07:37:23.406233+0000","last_became_active":"2024-03-06T07:37:21.375114+0000","last_became_peered":"2024-03-06T07:37:21.375114+0000","last_unstale":"2024-03-06T07:37:23.406233+0000","last_undegraded":"2024-03-06T07:37:23.406233+0000","last_fullsized":"2024-03-06T07:37:23.406233+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:44.982476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464902+0000","last_change":"2024-03-06T07:37:22.394822+0000","last_active":"2024-03-06T07:37:23.464902+0000","last_peered":"2024-03-06T07:37:23.464902+0000","last_clean":"2024-03-06T07:37:23.464902+0000","last_became_active":"2024-03-06T07:37:22.394527+0000","last_became_peered":"2024-03-06T07:37:22.394527+0000","last_unstale":"2024-03-06T07:37:23.464902+0000","last_undegraded":"2024-03-06T07:37:23.464902+0000","last_fullsized":"2024-03-06T07:37:23.464902+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:48.691100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"144'1","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.402824+0000","last_change":"2024-03-06T07:37:21.369209+0000","last_active":"2024-03-06T07:37:23.402824+0000","last_peered":"2024-03-06T07:37:23.402824+0000","last_clean":"2024-03-06T07:37:23.402824+0000","last_became_active":"2024-03-06T07:37:21.368820+0000","last_became_peered":"2024-03-06T07:37:21.368820+0000","last_unstale":"2024-03-06T07:37:23.402824+0000","last_undegraded":"2024-03-06T07:37:23.402824+0000","last_fullsized":"2024-03-06T07:37:23.402824+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:50.585486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.462038+0000","last_change":"2024-03-06T07:37:22.400181+0000","last_active":"2024-03-06T07:37:23.462038+0000","last_peered":"2024-03-06T07:37:23.462038+0000","last_clean":"2024-03-06T07:37:23.462038+0000","last_became_active":"2024-03-06T07:37:22.399887+0000","last_became_peered":"2024-03-06T07:37:22.399887+0000","last_unstale":"2024-03-06T07:37:23.462038+0000","last_undegraded":"2024-03-06T07:37:23.462038+0000","last_fullsized":"2024-03-06T07:37:23.462038+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:29.190943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.400026+0000","last_change":"2024-03-06T07:37:21.355184+0000","last_active":"2024-03-06T07:37:23.400026+0000","last_peered":"2024-03-06T07:37:23.400026+0000","last_clean":"2024-03-06T07:37:23.400026+0000","last_became_active":"2024-03-06T07:37:21.354882+0000","last_became_peered":"2024-03-06T07:37:21.354882+0000","last_unstale":"2024-03-06T07:37:23.400026+0000","last_undegraded":"2024-03-06T07:37:23.400026+0000","last_fullsized":"2024-03-06T07:37:23.400026+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:22.449576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399756+0000","last_change":"2024-03-06T07:37:22.396365+0000","last_active":"2024-03-06T07:37:23.399756+0000","last_peered":"2024-03-06T07:37:23.399756+0000","last_clean":"2024-03-06T07:37:23.399756+0000","last_became_active":"2024-03-06T07:37:22.396080+0000","last_became_peered":"2024-03-06T07:37:22.396080+0000","last_unstale":"2024-03-06T07:37:23.399756+0000","last_undegraded":"2024-03-06T07:37:23.399756+0000","last_fullsized":"2024-03-06T07:37:23.399756+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:48.226399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399226+0000","last_change":"2024-03-06T07:37:21.353350+0000","last_active":"2024-03-06T07:37:23.399226+0000","last_peered":"2024-03-06T07:37:23.399226+0000","last_clean":"2024-03-06T07:37:23.399226+0000","last_became_active":"2024-03-06T07:37:21.352640+0000","last_became_peered":"2024-03-06T07:37:21.352640+0000","last_unstale":"2024-03-06T07:37:23.399226+0000","last_undegraded":"2024-03-06T07:37:23.399226+0000","last_fullsized":"2024-03-06T07:37:23.399226+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:38.501518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.460551+0000","last_change":"2024-03-06T07:37:22.403004+0000","last_active":"2024-03-06T07:37:23.460551+0000","last_peered":"2024-03-06T07:37:23.460551+0000","last_clean":"2024-03-06T07:37:23.460551+0000","last_became_active":"2024-03-06T07:37:22.402049+0000","last_became_peered":"2024-03-06T07:37:22.402049+0000","last_unstale":"2024-03-06T07:37:23.460551+0000","last_undegraded":"2024-03-06T07:37:23.460551+0000","last_fullsized":"2024-03-06T07:37:23.460551+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:12.310610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394502+0000","last_change":"2024-03-06T07:37:21.369250+0000","last_active":"2024-03-06T07:37:23.394502+0000","last_peered":"2024-03-06T07:37:23.394502+0000","last_clean":"2024-03-06T07:37:23.394502+0000","last_became_active":"2024-03-06T07:37:21.368493+0000","last_became_peered":"2024-03-06T07:37:21.368493+0000","last_unstale":"2024-03-06T07:37:23.394502+0000","last_undegraded":"2024-03-06T07:37:23.394502+0000","last_fullsized":"2024-03-06T07:37:23.394502+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:31.480272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399589+0000","last_change":"2024-03-06T07:37:22.396371+0000","last_active":"2024-03-06T07:37:23.399589+0000","last_peered":"2024-03-06T07:37:23.399589+0000","last_clean":"2024-03-06T07:37:23.399589+0000","last_became_active":"2024-03-06T07:37:22.395650+0000","last_became_peered":"2024-03-06T07:37:22.395650+0000","last_unstale":"2024-03-06T07:37:23.399589+0000","last_undegraded":"2024-03-06T07:37:23.399589+0000","last_fullsized":"2024-03-06T07:37:23.399589+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:16.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.404803+0000","last_change":"2024-03-06T07:37:21.374738+0000","last_active":"2024-03-06T07:37:23.404803+0000","last_peered":"2024-03-06T07:37:23.404803+0000","last_clean":"2024-03-06T07:37:23.404803+0000","last_became_active":"2024-03-06T07:37:21.374324+0000","last_became_peered":"2024-03-06T07:37:21.374324+0000","last_unstale":"2024-03-06T07:37:23.404803+0000","last_undegraded":"2024-03-06T07:37:23.404803+0000","last_fullsized":"2024-03-06T07:37:23.404803+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:33.070997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405053+0000","last_change":"2024-03-06T07:37:22.385530+0000","last_active":"2024-03-06T07:37:23.405053+0000","last_peered":"2024-03-06T07:37:23.405053+0000","last_clean":"2024-03-06T07:37:23.405053+0000","last_became_active":"2024-03-06T07:37:22.385134+0000","last_became_peered":"2024-03-06T07:37:22.385134+0000","last_unstale":"2024-03-06T07:37:23.405053+0000","last_undegraded":"2024-03-06T07:37:23.405053+0000","last_fullsized":"2024-03-06T07:37:23.405053+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:46.584284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.463358+0000","last_change":"2024-03-06T07:37:21.372688+0000","last_active":"2024-03-06T07:37:23.463358+0000","last_peered":"2024-03-06T07:37:23.463358+0000","last_clean":"2024-03-06T07:37:23.463358+0000","last_became_active":"2024-03-06T07:37:21.372275+0000","last_became_peered":"2024-03-06T07:37:21.372275+0000","last_unstale":"2024-03-06T07:37:23.463358+0000","last_undegraded":"2024-03-06T07:37:23.463358+0000","last_fullsized":"2024-03-06T07:37:23.463358+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:00.690180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.399339+0000","last_change":"2024-03-06T07:37:22.404380+0000","last_active":"2024-03-06T07:37:23.399339+0000","last_peered":"2024-03-06T07:37:23.399339+0000","last_clean":"2024-03-06T07:37:23.399339+0000","last_became_active":"2024-03-06T07:37:22.404068+0000","last_became_peered":"2024-03-06T07:37:22.404068+0000","last_unstale":"2024-03-06T07:37:23.399339+0000","last_undegraded":"2024-03-06T07:37:23.399339+0000","last_fullsized":"2024-03-06T07:37:23.399339+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:21.095007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.461848+0000","last_change":"2024-03-06T07:37:21.369739+0000","last_active":"2024-03-06T07:37:23.461848+0000","last_peered":"2024-03-06T07:37:23.461848+0000","last_clean":"2024-03-06T07:37:23.461848+0000","last_became_active":"2024-03-06T07:37:21.369469+0000","last_became_peered":"2024-03-06T07:37:21.369469+0000","last_unstale":"2024-03-06T07:37:23.461848+0000","last_undegraded":"2024-03-06T07:37:23.461848+0000","last_fullsized":"2024-03-06T07:37:23.461848+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:49.986528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.394619+0000","last_change":"2024-03-06T07:37:22.410356+0000","last_active":"2024-03-06T07:37:23.394619+0000","last_peered":"2024-03-06T07:37:23.394619+0000","last_clean":"2024-03-06T07:37:23.394619+0000","last_became_active":"2024-03-06T07:37:22.410028+0000","last_became_peered":"2024-03-06T07:37:22.410028+0000","last_unstale":"2024-03-06T07:37:23.394619+0000","last_undegraded":"2024-03-06T07:37:23.394619+0000","last_fullsized":"2024-03-06T07:37:23.394619+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:21.329262+0000","last_clean_scrub_stamp":"2024-03-06T07:37:21.329262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:24.219967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"143'3","reported_seq":17,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.464453+0000","last_change":"2024-03-06T07:37:21.371926+0000","last_active":"2024-03-06T07:37:23.464453+0000","last_peered":"2024-03-06T07:37:23.464453+0000","last_clean":"2024-03-06T07:37:23.464453+0000","last_became_active":"2024-03-06T07:37:21.371654+0000","last_became_peered":"2024-03-06T07:37:21.371654+0000","last_unstale":"2024-03-06T07:37:23.464453+0000","last_undegraded":"2024-03-06T07:37:23.464453+0000","last_fullsized":"2024-03-06T07:37:23.464453+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:37:20.313970+0000","last_clean_scrub_stamp":"2024-03-06T07:37:20.313970+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:51.206679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":395,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:37:23.405127+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:37:23.405127+0000","last_peered":"2024-03-06T07:37:23.405127+0000","last_clean":"2024-03-06T07:37:23.405127+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:37:23.405127+0000","last_undegraded":"2024-03-06T07:37:23.405127+0000","last_fullsized":"2024-03-06T07:37:23.405127+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542264,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37688,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334152,"statfs":{"total":96636764160,"available":96598171648,"internally_reserved":0,"allocated":2400256,"data_stored":1377397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542265,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1798144,"data_stored":780223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37104,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334736,"statfs":{"total":96636764160,"available":96598769664,"internally_reserved":0,"allocated":1802240,"data_stored":786399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":786377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1789952,"data_stored":780167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37104,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334736,"statfs":{"total":96636764160,"available":96598769664,"internally_reserved":0,"allocated":1802240,"data_stored":781759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1802240,"data_stored":786959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41844,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329996,"statfs":{"total":96636764160,"available":96593915904,"internally_reserved":0,"allocated":2396160,"data_stored":1372091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:37:27.381 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:37:27.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:37:27.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.626+0000 7f4019800640 1 Processor -- start 2024-03-06T07:37:27.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.627+0000 7f4019800640 1 -- start start 2024-03-06T07:37:27.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.628+0000 7f4019800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f401415b780 0x7f40140b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.628+0000 7f4019800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f4014165330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.628+0000 7f4019800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40140ce690 con 0x7f40140b3a70 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.628+0000 7f4019800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140ce970 con 0x7f401415b780 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.628+0000 7f4019800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140cec50 con 0x7f40140ce1a0 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4012a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f401415b780 0x7f40140b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f400a000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f40140b3a70 0x7f40140cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49368/0 (socket says 172.21.15.97:49368) 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f400a000640 1 -- 172.21.15.97:0/3488606424 learned_addr learned my addr 172.21.15.97:0/3488606424 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4013400640 1 --2- 172.21.15.97:0/3488606424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f4014165330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4012a00640 1 -- 172.21.15.97:0/3488606424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 msgr2=0x7f4014165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4012a00640 1 --2- 172.21.15.97:0/3488606424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f4014165330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4012a00640 1 -- 172.21.15.97:0/3488606424 >> v1:172.21.15.97:6789/0 conn(0x7f40140b3a70 legacy=0x7f40140cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4012a00640 1 -- 172.21.15.97:0/3488606424 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140cef30 con 0x7f401415b780 2024-03-06T07:37:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.629+0000 7f4012000640 1 -- 172.21.15.97:0/3488606424 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682455439 0 0) 0x7f40140ce690 con 0x7f40140b3a70 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.630+0000 7f4012a00640 1 --2- 172.21.15.97:0/3488606424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f401415b780 0x7f40140b3530 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f40040097a0 tx=0x7f400402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e3bdac4be6aad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4012000640 1 -- 172.21.15.97:0/3488606424 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004035020 con 0x7f401415b780 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4012000640 1 -- 172.21.15.97:0/3488606424 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4004031830 con 0x7f401415b780 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4012000640 1 -- 172.21.15.97:0/3488606424 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004031c00 con 0x7f401415b780 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4019800640 1 -- 172.21.15.97:0/3488606424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f401415b780 msgr2=0x7f40140b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4019800640 1 --2- 172.21.15.97:0/3488606424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f401415b780 0x7f40140b3530 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f40040097a0 tx=0x7f400402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4019800640 1 -- 172.21.15.97:0/3488606424 shutdown_connections 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4019800640 1 --2- 172.21.15.97:0/3488606424 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f4014165330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4019800640 1 --2- 172.21.15.97:0/3488606424 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f401415b780 0x7f40140b3530 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.631+0000 7f4019800640 1 -- 172.21.15.97:0/3488606424 >> 172.21.15.97:0/3488606424 conn(0x7f40140b1940 msgr2=0x7f40140b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.632+0000 7f4019800640 1 -- 172.21.15.97:0/3488606424 shutdown_connections 2024-03-06T07:37:27.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.632+0000 7f4019800640 1 -- 172.21.15.97:0/3488606424 wait complete. 2024-03-06T07:37:27.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.633+0000 7f4019800640 1 Processor -- start 2024-03-06T07:37:27.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.633+0000 7f4019800640 1 -- start start 2024-03-06T07:37:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4019800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 0x7f401414f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4019800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f401415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4019800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f401415b780 0x7f401415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4019800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401415abb0 con 0x7f40140b3a70 2024-03-06T07:37:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4019800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401415aa30 con 0x7f401415b780 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4019800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140af120 con 0x7f40140ce1a0 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.634+0000 7f4012a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 0x7f401414f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 0x7f401414f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38306/0 (socket says 172.21.15.97:38306) 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 -- 172.21.15.97:0/944266151 learned_addr learned my addr 172.21.15.97:0/944266151 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 -- 172.21.15.97:0/944266151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 msgr2=0x7f401415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f401415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 -- 172.21.15.97:0/944266151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f401415b780 msgr2=0x7f401415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f401415b780 0x7f401415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 -- 172.21.15.97:0/944266151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40141533b0 con 0x7f40140b3a70 2024-03-06T07:37:27.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.635+0000 7f4012a00640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 0x7f401414f2a0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4004034040 tx=0x7f40040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.636+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004040070 con 0x7f40140b3a70 2024-03-06T07:37:27.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.636+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40040399a0 con 0x7f40140b3a70 2024-03-06T07:37:27.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.636+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40141f73d0 con 0x7f40140b3a70 2024-03-06T07:37:27.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.636+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004035040 con 0x7f40140b3a70 2024-03-06T07:37:27.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.636+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401415cbc0 con 0x7f40140b3a70 2024-03-06T07:37:27.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.637+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401404f330 con 0x7f40140b3a70 2024-03-06T07:37:27.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.638+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4004038020 con 0x7f40140b3a70 2024-03-06T07:37:27.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.639+0000 7f400b400640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ff006b4d0 0x7f3ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.639+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40040abbe0 con 0x7f40140b3a70 2024-03-06T07:37:27.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.639+0000 7f400a000640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ff006b4d0 0x7f3ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.640+0000 7f400a000640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ff006b4d0 0x7f3ff006d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc001560 tx=0x7f3ffc007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.645+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f400407a9d0 con 0x7f40140b3a70 2024-03-06T07:37:27.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.912+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f40140b1e50 con 0x7f40140b3a70 2024-03-06T07:37:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.917+0000 7f400b400640 1 -- 172.21.15.97:0/944266151 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f400407a2d0 con 0x7f40140b3a70 2024-03-06T07:37:27.919 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-06T07:37:27.920 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxGuhlV4xNJBAAB5JhnthWY+0cbrFGVtroSQ== 2024-03-06T07:37:27.920 INFO:teuthology.orchestra.run.smithi097.stdout: caps mds = "allow" 2024-03-06T07:37:27.920 INFO:teuthology.orchestra.run.smithi097.stdout: caps mon = "allow r" 2024-03-06T07:37:27.920 INFO:teuthology.orchestra.run.smithi097.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:37:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:updated caps for client.0 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ff006b4d0 msgr2=0x7f3ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ff006b4d0 0x7f3ff006d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc001560 tx=0x7f3ffc007cc0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 msgr2=0x7f401414f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 0x7f401414f2a0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4004034040 tx=0x7f40040046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 shutdown_connections 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3ff006b4d0 0x7f3ff006d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f401415b780 0x7f401415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f40140ce1a0 0x7f401415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 --2- 172.21.15.97:0/944266151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f40140b3a70 0x7f401414f2a0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.920+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 >> 172.21.15.97:0/944266151 conn(0x7f40140b1940 msgr2=0x7f40140cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.921+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 shutdown_connections 2024-03-06T07:37:27.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:27.921+0000 7f4019800640 1 -- 172.21.15.97:0/944266151 wait complete. 2024-03-06T07:37:27.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:28.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.151+0000 7f6f1b400640 1 Processor -- start 2024-03-06T07:37:28.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.151+0000 7f6f1b400640 1 -- start start 2024-03-06T07:37:28.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.152+0000 7f6f1b400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 0x7f6f1406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.152+0000 7f6f1b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.152+0000 7f6f1b400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1406ee70 con 0x7f6f140ce0a0 2024-03-06T07:37:28.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.152+0000 7f6f1b400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f14160370 con 0x7f6f14144c00 2024-03-06T07:37:28.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.152+0000 7f6f1b400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1406cef0 con 0x7f6f1415f9d0 2024-03-06T07:37:28.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.153+0000 7f6f18a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 0x7f6f1406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.153+0000 7f6f13e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.153+0000 7f6f18a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 0x7f6f1406e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:50534/0 (socket says 172.21.15.97:50534) 2024-03-06T07:37:28.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.153+0000 7f6f13e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59536/0 (socket says 172.21.15.97:59536) 2024-03-06T07:37:28.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.153+0000 7f6f18a00640 1 -- 172.21.15.97:0/1016293767 learned_addr learned my addr 172.21.15.97:0/1016293767 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:28.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.154+0000 7f6f13e00640 1 -- 172.21.15.97:0/1016293767 >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 msgr2=0x7f6f1406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:28.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.154+0000 7f6f13e00640 1 --2- 172.21.15.97:0/1016293767 >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 0x7f6f1406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.154+0000 7f6f13e00640 1 -- 172.21.15.97:0/1016293767 >> v1:172.21.15.135:6789/0 conn(0x7f6f140ce0a0 legacy=0x7f6f140ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.154+0000 7f6f13e00640 1 -- 172.21.15.97:0/1016293767 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1406d1d0 con 0x7f6f1415f9d0 2024-03-06T07:37:28.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.154+0000 7f6f18a00640 1 --2- 172.21.15.97:0/1016293767 >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 0x7f6f1406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:37:28.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.155+0000 7f6f13e00640 1 --2- 172.21.15.97:0/1016293767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415fe30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f000097a0 tx=0x7f6f0002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d427da9db88ad95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f13400640 1 -- 172.21.15.97:0/1016293767 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00035020 con 0x7f6f1415f9d0 2024-03-06T07:37:28.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f13400640 1 -- 172.21.15.97:0/1016293767 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00031810 con 0x7f6f1415f9d0 2024-03-06T07:37:28.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f13400640 1 -- 172.21.15.97:0/1016293767 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00031be0 con 0x7f6f1415f9d0 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 -- 172.21.15.97:0/1016293767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 msgr2=0x7f6f1415fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 --2- 172.21.15.97:0/1016293767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415fe30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f000097a0 tx=0x7f6f0002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 -- 172.21.15.97:0/1016293767 shutdown_connections 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 --2- 172.21.15.97:0/1016293767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415fe30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 --2- 172.21.15.97:0/1016293767 >> v2:172.21.15.135:3300/0 conn(0x7f6f14144c00 0x7f6f1406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 -- 172.21.15.97:0/1016293767 >> 172.21.15.97:0/1016293767 conn(0x7f6f140b1730 msgr2=0x7f6f140b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 -- 172.21.15.97:0/1016293767 shutdown_connections 2024-03-06T07:37:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.156+0000 7f6f1b400640 1 -- 172.21.15.97:0/1016293767 wait complete. 2024-03-06T07:37:28.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.159+0000 7f6f1b400640 1 Processor -- start 2024-03-06T07:37:28.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.159+0000 7f6f1b400640 1 -- start start 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f1b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 0x7f6f1413c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f1b400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f14144c00 0x7f6f14140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f1b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f1b400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f14160730 con 0x7f6f140ce0a0 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f1b400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1406d790 con 0x7f6f14144c00 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f1b400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1406cb50 con 0x7f6f1415f9d0 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 0x7f6f1413c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 0x7f6f1413c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38314/0 (socket says 172.21.15.97:38314) 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 -- 172.21.15.97:0/2130353302 learned_addr learned my addr 172.21.15.97:0/2130353302 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f13e00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f14144c00 0x7f6f14140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f19400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 -- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 msgr2=0x7f6f1415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 -- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f14144c00 msgr2=0x7f6f14140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f14144c00 0x7f6f14140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f18a00640 1 -- 172.21.15.97:0/2130353302 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f14153d70 con 0x7f6f140ce0a0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.160+0000 7f6f13e00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f14144c00 0x7f6f14140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.161+0000 7f6f18a00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 0x7f6f1413c0e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800e9d0 tx=0x7f6f0800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.161+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08017070 con 0x7f6f140ce0a0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.161+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0800ca60 con 0x7f6f140ce0a0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.161+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f14140070 con 0x7f6f140ce0a0 2024-03-06T07:37:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.161+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0800ce10 con 0x7f6f140ce0a0 2024-03-06T07:37:28.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.161+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1413c620 con 0x7f6f140ce0a0 2024-03-06T07:37:28.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.162+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1413cf00 con 0x7f6f140ce0a0 2024-03-06T07:37:28.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.163+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f0801a070 con 0x7f6f140ce0a0 2024-03-06T07:37:28.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.165+0000 7f6f10c00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ef406b4d0 0x7f6ef406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.165+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f0808a530 con 0x7f6f140ce0a0 2024-03-06T07:37:28.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.165+0000 7f6f13e00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ef406b4d0 0x7f6ef406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.166+0000 7f6f13e00640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ef406b4d0 0x7f6ef406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f00008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.169+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f080593a0 con 0x7f6f140ce0a0 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.392+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f1404f330 con 0x7f6f140ce0a0 2024-03-06T07:37:28.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.393+0000 7f6f10c00640 1 -- 172.21.15.97:0/2130353302 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4990 (secure 0 0 0) 0x7f6f08058ca0 con 0x7f6f140ce0a0 2024-03-06T07:37:28.395 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:28.395 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5840,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2881704649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":2881704649},{"type":"v1","addr":"172.21.15.135:6835","nonce":2881704649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6448,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/281989301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":281989301},{"type":"v1","addr":"172.21.15.135:6834","nonce":281989301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:24.903864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9024},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:37:28.396 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 107 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ef406b4d0 msgr2=0x7f6ef406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ef406b4d0 0x7f6ef406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f00008030 comp rx=0 tx=0).stop 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 msgr2=0x7f6f1413c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 0x7f6f1413c0e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800e9d0 tx=0x7f6f0800ee90 comp rx=0 tx=0).stop 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 shutdown_connections 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6ef406b4d0 0x7f6ef406d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6f1415f9d0 0x7f6f1415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6f14144c00 0x7f6f14140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 --2- 172.21.15.97:0/2130353302 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6f140ce0a0 0x7f6f1413c0e0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.395+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 >> 172.21.15.97:0/2130353302 conn(0x7f6f140b1730 msgr2=0x7f6f140ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.396+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 shutdown_connections 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:28.396+0000 7f6f1b400640 1 -- 172.21.15.97:0/2130353302 wait complete. 2024-03-06T07:37:28.817 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:37:24.903864+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9024}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:37:28.817 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:37:28.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:37:29.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.027+0000 7ff622800640 1 Processor -- start 2024-03-06T07:37:29.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.028+0000 7ff622800640 1 -- start start 2024-03-06T07:37:29.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.029+0000 7ff622800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 0x7ff61c0d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.029+0000 7ff622800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0b5240 con 0x7ff61c0d0c00 2024-03-06T07:37:29.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.029+0000 7ff622800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0cf4d0 con 0x7ff61c15e200 2024-03-06T07:37:29.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.029+0000 7ff622800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0d2ec0 con 0x7ff61c0b2e50 2024-03-06T07:37:29.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.029+0000 7ff620a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff61c0d0c00 0x7ff61c0b3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49384/0 (socket says 172.21.15.97:49384) 2024-03-06T07:37:29.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.029+0000 7ff620a00640 1 -- 172.21.15.97:0/2602996084 learned_addr learned my addr 172.21.15.97:0/2602996084 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff621e00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 0x7ff61c0d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036285009 0 0) 0x7ff61c0b5240 con 0x7ff61c0d0c00 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604003520 con 0x7ff61c0d0c00 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172097215 0 0) 0x7ff604003520 con 0x7ff61c0d0c00 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 msgr2=0x7ff61c0d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 0x7ff61c0d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 >> v1:172.21.15.135:6789/0 conn(0x7ff61c15e200 legacy=0x7ff61c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0b5500 con 0x7ff61c0d0c00 2024-03-06T07:37:29.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.030+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff608001670 con 0x7ff61c0d0c00 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 >> v1:172.21.15.97:6789/0 conn(0x7ff61c0d0c00 legacy=0x7ff61c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6040039e0 0x7ff604003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff604004390 0x7ff604004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 0x7ff6040050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0d2ec0 con 0x7ff604004390 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60400b9f0 con 0x7ff6040039e0 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60400bbf0 con 0x7ff61c0b2e50 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff620a00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff604004390 0x7ff604004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff621400640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6040039e0 0x7ff604003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff621e00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 0x7ff6040050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.031+0000 7ff620a00640 1 -- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 msgr2=0x7ff6040050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.032+0000 7ff620a00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c0b2e50 0x7ff6040050c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.032+0000 7ff620a00640 1 -- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6040039e0 msgr2=0x7ff604003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.032+0000 7ff620a00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6040039e0 0x7ff604003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.032+0000 7ff620a00640 1 -- 172.21.15.97:0/2602996084 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff604003520 con 0x7ff604004390 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.033+0000 7ff620a00640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff604004390 0x7ff604004860 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c190 tx=0x7ff604005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6127dd387bbb87bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.033+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608007cc0 con 0x7ff604004390 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.033+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608030070 con 0x7ff604004390 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.033+0000 7ff61be00640 1 -- 172.21.15.97:0/2602996084 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608005d10 con 0x7ff604004390 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff622800640 1 -- 172.21.15.97:0/2602996084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff604004390 msgr2=0x7ff604004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff622800640 1 --2- 172.21.15.97:0/2602996084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff604004390 0x7ff604004860 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c190 tx=0x7ff604005830 comp rx=0 tx=0).stop 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff621e00640 1 -- 172.21.15.97:0/2602996084 reap_dead start 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff622800640 1 -- 172.21.15.97:0/2602996084 shutdown_connections 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff622800640 1 -- 172.21.15.97:0/2602996084 >> 172.21.15.97:0/2602996084 conn(0x7ff61c0d3d30 msgr2=0x7ff61c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff622800640 1 -- 172.21.15.97:0/2602996084 shutdown_connections 2024-03-06T07:37:29.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.034+0000 7ff622800640 1 -- 172.21.15.97:0/2602996084 wait complete. 2024-03-06T07:37:29.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.036+0000 7ff622800640 1 Processor -- start 2024-03-06T07:37:29.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.036+0000 7ff622800640 1 -- start start 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff622800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff604004390 0x7ff61c160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff622800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 0x7ff61c06b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff622800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c06beb0 0x7ff61c15eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff622800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0cfd60 con 0x7ff61c0c7860 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff622800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0cfa80 con 0x7ff604004390 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff622800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c15e5e0 con 0x7ff61c06beb0 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff620a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 0x7ff61c06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff620a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 0x7ff61c06b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38338/0 (socket says 172.21.15.97:38338) 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff620a00640 1 -- 172.21.15.97:0/2983176794 learned_addr learned my addr 172.21.15.97:0/2983176794 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff621400640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff604004390 0x7ff61c160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.037+0000 7ff621e00640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c06beb0 0x7ff61c15eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff620a00640 1 -- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c06beb0 msgr2=0x7ff61c15eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff620a00640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c06beb0 0x7ff61c15eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff620a00640 1 -- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff604004390 msgr2=0x7ff61c160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff620a00640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff604004390 0x7ff61c160630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff620a00640 1 -- 172.21.15.97:0/2983176794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c160ca0 con 0x7ff61c0c7860 2024-03-06T07:37:29.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff621400640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff604004390 0x7ff61c160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:29.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff620a00640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 0x7ff61c06b970 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c190 tx=0x7ff6080019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff619600640 1 -- 172.21.15.97:0/2983176794 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608007cc0 con 0x7ff61c0c7860 2024-03-06T07:37:29.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff619600640 1 -- 172.21.15.97:0/2983176794 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60802cbd0 con 0x7ff61c0c7860 2024-03-06T07:37:29.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff619600640 1 -- 172.21.15.97:0/2983176794 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608005e20 con 0x7ff61c0c7860 2024-03-06T07:37:29.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.038+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c160f80 con 0x7ff61c0c7860 2024-03-06T07:37:29.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.039+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c161540 con 0x7ff61c0c7860 2024-03-06T07:37:29.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.041+0000 7ff619600640 1 -- 172.21.15.97:0/2983176794 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff608047060 con 0x7ff61c0c7860 2024-03-06T07:37:29.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.041+0000 7ff619600640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.041+0000 7ff619600640 1 -- 172.21.15.97:0/2983176794 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6080ab4a0 con 0x7ff61c0c7860 2024-03-06T07:37:29.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.042+0000 7ff621400640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.042+0000 7ff621400640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0ce500 tx=0x7ff610009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.042+0000 7ff626beb740 1 Processor -- start 2024-03-06T07:37:29.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 -- start start 2024-03-06T07:37:29.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b752e0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 0x561b75310720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b752e0e70 0x561b75310c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b7524a600 con 0x561b7530d990 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b751f3930 con 0x561b752e0e70 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.043+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b752466a0 con 0x561b7530dd70 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b752e0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b752e0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38350/0 (socket says 172.21.15.97:38350) 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 -- 172.21.15.97:0/1670515245 learned_addr learned my addr 172.21.15.97:0/1670515245 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 -- 172.21.15.97:0/1670515245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 msgr2=0x561b75310720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 0x561b75310720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 -- 172.21.15.97:0/1670515245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b752e0e70 msgr2=0x561b75310c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b752e0e70 0x561b75310c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 -- 172.21.15.97:0/1670515245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b74f5f2d0 con 0x561b7530d990 2024-03-06T07:37:29.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff621400640 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b752e0930 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7ff610001760 tx=0x7ff610013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869c6d9df551df4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.044+0000 7ff60dc00640 1 -- 172.21.15.97:0/1670515245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610027070 con 0x561b7530d990 2024-03-06T07:37:29.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff60dc00640 1 -- 172.21.15.97:0/1670515245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610014900 con 0x561b7530d990 2024-03-06T07:37:29.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff60dc00640 1 -- 172.21.15.97:0/1670515245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610014cb0 con 0x561b7530d990 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 -- 172.21.15.97:0/1670515245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 msgr2=0x561b752e0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b752e0930 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7ff610001760 tx=0x7ff610013000 comp rx=0 tx=0).stop 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 -- 172.21.15.97:0/1670515245 shutdown_connections 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b752e0e70 0x561b75310c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 0x561b75310720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 --2- 172.21.15.97:0/1670515245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b752e0930 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 -- 172.21.15.97:0/1670515245 >> 172.21.15.97:0/1670515245 conn(0x561b752e2ba0 msgr2=0x561b7532b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.045+0000 7ff626beb740 1 -- 172.21.15.97:0/1670515245 shutdown_connections 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.046+0000 7ff626beb740 1 -- 172.21.15.97:0/1670515245 wait complete. 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.046+0000 7ff626beb740 1 -- start start 2024-03-06T07:37:29.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.046+0000 7ff626beb740 1 Processor -- start 2024-03-06T07:37:29.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.047+0000 7ff626beb740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b75305d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.047+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 0x561b7533c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.047+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b75308d80 0x561b75309200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.047+0000 7ff626beb740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b752e5140 con 0x561b7530d990 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.047+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b75206d90 con 0x561b75308d80 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.047+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b752e4530 con 0x561b7530dd70 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff626beb740 10 client.0 _ll_register_callbacks cb 0x561b75242350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b75305d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b75305d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38356/0 (socket says 172.21.15.97:38356) 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 -- 172.21.15.97:0/3475232692 learned_addr learned my addr 172.21.15.97:0/3475232692 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 -- 172.21.15.97:0/3475232692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 msgr2=0x561b7533c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 0x561b7533c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 -- 172.21.15.97:0/3475232692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b75308d80 msgr2=0x561b75309200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b75308d80 0x561b75309200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 -- 172.21.15.97:0/3475232692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b750296b0 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff621400640 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b75305d50 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7ff6100147d0 tx=0x7ff610001320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce8ebc2cbf9b125 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.048+0000 7ff602000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff602000640 1 -- 172.21.15.97:0/3475232692 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610032020 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff602000640 1 -- 172.21.15.97:0/3475232692 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff610022a20 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff610014900 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff602000640 1 -- 172.21.15.97:0/3475232692 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff61003c050 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 10 client.9156 fetch_fsmap learned FSMap version 107 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x561b75306930 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff602000640 1 -- 172.21.15.97:0/3475232692 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610014900 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff602000640 1 -- 172.21.15.97:0/3475232692 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 107) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff61003f020 con 0x561b7530d990 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 10 client.9156 fetch_fsmap finished waiting for FSMap version 107 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 10 client.9156 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 20 client.9156 populate_metadata read hostname 'smithi097' 2024-03-06T07:37:29.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b7530f9e0 0x561b7530fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 4 client.9156 mds_command: new command op to 9024 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:37:29.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.049+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 --> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b75257b40 con 0x561b7530f9e0 2024-03-06T07:37:29.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.050+0000 7ff620a00640 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b7530f9e0 0x561b7530fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.050+0000 7ff620a00640 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b7530f9e0 0x561b7530fdc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=523100685560f29d server_cookie=7e56445ea9ee2a4c in_seq=0 out_seq=0 2024-03-06T07:37:29.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.050+0000 7ff602000640 10 client.9156 ms_handle_connect on v2:172.21.15.97:6834/4190861064 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff602000640 1 -- 172.21.15.97:0/3475232692 <== mds.0 v2:172.21.15.97:6834/4190861064 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561b75257b40 con 0x561b7530f9e0 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff602000640 10 client.9156 handle_command_reply: tid=0 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 client.9156 shutdown 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 msgr2=0x561b75305d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b75305d50 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7ff6100147d0 tx=0x7ff610001320 comp rx=0 tx=0).stop 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 shutdown_connections 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b7530f9e0 0x561b7530fdc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b75308d80 0x561b75309200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b7530dd70 0x561b7533c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 --2- 172.21.15.97:0/3475232692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530d990 0x561b75305d50 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff602000640 0 client.9156 ms_handle_reset on v2:172.21.15.97:6834/4190861064 2024-03-06T07:37:29.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.051+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 >> 172.21.15.97:0/3475232692 conn(0x561b752e0e70 msgr2=0x561b7532ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.052+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 shutdown_connections 2024-03-06T07:37:29.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.052+0000 7ff626beb740 1 -- 172.21.15.97:0/3475232692 wait complete. 2024-03-06T07:37:29.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.052+0000 7ff626beb740 20 client.9156 trim_cache size 0 max 16384 2024-03-06T07:37:29.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.069+0000 7ff626beb740 1 Processor -- start 2024-03-06T07:37:29.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 -- start start 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75310d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7532e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7533b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c200770 con 0x561b7530dd70 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c1ff8d0 con 0x561b7530d990 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.070+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c1f4be0 con 0x561b75308d80 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7533b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7533b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38362/0 (socket says 172.21.15.97:38362) 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 -- 172.21.15.97:0/827467218 learned_addr learned my addr 172.21.15.97:0/827467218 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621400640 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75310d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 -- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 msgr2=0x561b75310d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75310d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 -- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 msgr2=0x561b7532e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff620a00640 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7532e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7532e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.071+0000 7ff621e00640 1 -- 172.21.15.97:0/827467218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b7533edd0 con 0x561b7530dd70 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff621e00640 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7533b260 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7ff61c08e520 tx=0x7ff614003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712c9d9d8b8ee45c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff60d200640 1 -- 172.21.15.97:0/827467218 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614016610 con 0x561b7530dd70 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff60d200640 1 -- 172.21.15.97:0/827467218 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61401b070 con 0x561b7530dd70 2024-03-06T07:37:29.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff60d200640 1 -- 172.21.15.97:0/827467218 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6140169f0 con 0x561b7530dd70 2024-03-06T07:37:29.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 -- 172.21.15.97:0/827467218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 msgr2=0x561b7533b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7533b260 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7ff61c08e520 tx=0x7ff614003ed0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 -- 172.21.15.97:0/827467218 shutdown_connections 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7533b260 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7532e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 --2- 172.21.15.97:0/827467218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75310d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.072+0000 7ff626beb740 1 -- 172.21.15.97:0/827467218 >> 172.21.15.97:0/827467218 conn(0x561b752e0e70 msgr2=0x561b752e7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.073+0000 7ff626beb740 1 -- 172.21.15.97:0/827467218 shutdown_connections 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.073+0000 7ff626beb740 1 -- 172.21.15.97:0/827467218 wait complete. 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.073+0000 7ff626beb740 1 -- start start 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.073+0000 7ff626beb740 1 Processor -- start 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75307ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7530bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7539b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b75306c10 con 0x561b7530dd70 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c1f7f60 con 0x561b7530d990 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b75307370 con 0x561b75308d80 2024-03-06T07:37:29.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.074+0000 7ff626beb740 10 client.0 _ll_register_callbacks cb 0x7ff610025070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7539b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7539b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38366/0 (socket says 172.21.15.97:38366) 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 -- 172.21.15.97:0/754855871 learned_addr learned my addr 172.21.15.97:0/754855871 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621400640 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75307ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 -- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 msgr2=0x561b75307ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75307ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 -- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 msgr2=0x561b7530bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7530bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 -- 172.21.15.97:0/754855871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b75306000 con 0x561b7530dd70 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff621e00640 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7539b7f0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x561b752e8120 tx=0x7ff614009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c3814cd3af3951 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.075+0000 7ff602000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff602000640 1 -- 172.21.15.97:0/754855871 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614016d70 con 0x561b7530dd70 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff602000640 1 -- 172.21.15.97:0/754855871 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff614029470 con 0x561b7530dd70 2024-03-06T07:37:29.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff61401b070 con 0x561b7530dd70 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff602000640 1 -- 172.21.15.97:0/754855871 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff614031050 con 0x561b7530dd70 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 10 client.9162 fetch_fsmap learned FSMap version 107 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x561b7530c250 con 0x561b7530dd70 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff602000640 1 -- 172.21.15.97:0/754855871 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61401b070 con 0x561b7530dd70 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff602000640 1 -- 172.21.15.97:0/754855871 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 107) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff614034020 con 0x561b7530dd70 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 10 client.9162 fetch_fsmap finished waiting for FSMap version 107 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 10 client.9162 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 20 client.9162 populate_metadata read hostname 'smithi097' 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b752e4b00 0x561b7539f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 4 client.9162 mds_command: new command op to 9024 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.076+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 --> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561b75251e90 con 0x561b752e4b00 2024-03-06T07:37:29.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff621400640 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b752e4b00 0x561b7539f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff621400640 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b752e4b00 0x561b7539f460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfc83b3fe635b269 server_cookie=5e39f5dc65924e7d in_seq=0 out_seq=0 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff602000640 10 client.9162 ms_handle_connect on v2:172.21.15.97:6834/4190861064 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff602000640 1 -- 172.21.15.97:0/754855871 <== mds.0 v2:172.21.15.97:6834/4190861064 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x561b75251e90 con 0x561b752e4b00 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff602000640 10 client.9162 handle_command_reply: tid=0 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff626beb740 1 client.9162 shutdown 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 msgr2=0x561b7539b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff626beb740 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7539b7f0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x561b752e8120 tx=0x7ff614009750 comp rx=0 tx=0).stop 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.077+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 shutdown_connections 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561b7530dd70 0x561b7539b7f0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.97:6834/4190861064,v1:172.21.15.97:6835/4190861064] conn(0x561b752e4b00 0x561b7539f460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7530d990 0x561b7530bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 --2- 172.21.15.97:0/754855871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x561b75308d80 0x561b75307ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 >> 172.21.15.97:0/754855871 conn(0x561b752e0e70 msgr2=0x561b752e7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff602000640 0 client.9162 ms_handle_reset on v2:172.21.15.97:6834/4190861064 2024-03-06T07:37:29.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 shutdown_connections 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 1 -- 172.21.15.97:0/754855871 wait complete. 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.078+0000 7ff626beb740 20 client.9162 trim_cache size 0 max 16384 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "0b9abceb-2115-4321-96a2-348db0d0fbe4", 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 9024, 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 6.698421272, 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 107, 2024-03-06T07:37:29.084 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 144, 2024-03-06T07:37:29.085 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 144, 2024-03-06T07:37:29.085 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 13.165251634000001 2024-03-06T07:37:29.085 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:37:29.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff5fc06b4d0 msgr2=0x7ff5fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0ce500 tx=0x7ff610009040 comp rx=0 tx=0).stop 2024-03-06T07:37:29.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 msgr2=0x7ff61c06b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 0x7ff61c06b970 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c190 tx=0x7ff6080019c0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 shutdown_connections 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff61c06beb0 0x7ff61c15eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff61c0c7860 0x7ff61c06b970 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 --2- 172.21.15.97:0/2983176794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff604004390 0x7ff61c160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 >> 172.21.15.97:0/2983176794 conn(0x7ff61c0d3d30 msgr2=0x7ff61c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 shutdown_connections 2024-03-06T07:37:29.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.079+0000 7ff622800640 1 -- 172.21.15.97:0/2983176794 wait complete. 2024-03-06T07:37:29.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:37:29.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.294+0000 7f9107e00640 1 Processor -- start 2024-03-06T07:37:29.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.294+0000 7f9107e00640 1 -- start start 2024-03-06T07:37:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.295+0000 7f9107e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9108067000 0x7f91080d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.295+0000 7f9107e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f910806bd20 0x7f910806c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.295+0000 7f9107e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91080b58c0 con 0x7f910815e0d0 2024-03-06T07:37:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.295+0000 7f9107e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91080d0220 con 0x7f9108067000 2024-03-06T07:37:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.295+0000 7f9107e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91080d0500 con 0x7f910806bd20 2024-03-06T07:37:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9108067000 0x7f91080d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9108067000 0x7f91080d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38376/0 (socket says 172.21.15.97:38376) 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 -- 172.21.15.97:0/4180406077 learned_addr learned my addr 172.21.15.97:0/4180406077 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f9107400640 1 --2- 172.21.15.97:0/4180406077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f910806bd20 0x7f910806c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 -- 172.21.15.97:0/4180406077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f910806bd20 msgr2=0x7f910806c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 --2- 172.21.15.97:0/4180406077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f910806bd20 0x7f910806c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 -- 172.21.15.97:0/4180406077 >> v1:172.21.15.135:6789/0 conn(0x7f910815e0d0 legacy=0x7f91080b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.296+0000 7f90ffe00640 1 -- 172.21.15.97:0/4180406077 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91080d07e0 con 0x7f9108067000 2024-03-06T07:37:29.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.297+0000 7f90ffe00640 1 --2- 172.21.15.97:0/4180406077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9108067000 0x7f91080d0d00 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009870 tx=0x7f90f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15677e57259161c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.297+0000 7f9106000640 1 -- 172.21.15.97:0/4180406077 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0035020 con 0x7f9108067000 2024-03-06T07:37:29.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9106000640 1 -- 172.21.15.97:0/4180406077 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f0031640 con 0x7f9108067000 2024-03-06T07:37:29.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9106000640 1 -- 172.21.15.97:0/4180406077 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f002d040 con 0x7f9108067000 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 -- 172.21.15.97:0/4180406077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9108067000 msgr2=0x7f91080d0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 --2- 172.21.15.97:0/4180406077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9108067000 0x7f91080d0d00 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009870 tx=0x7f90f002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 -- 172.21.15.97:0/4180406077 shutdown_connections 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 --2- 172.21.15.97:0/4180406077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f910806bd20 0x7f910806c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 --2- 172.21.15.97:0/4180406077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9108067000 0x7f91080d0d00 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 -- 172.21.15.97:0/4180406077 >> 172.21.15.97:0/4180406077 conn(0x7f91080725e0 msgr2=0x7f91080729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 -- 172.21.15.97:0/4180406077 shutdown_connections 2024-03-06T07:37:29.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.298+0000 7f9107e00640 1 -- 172.21.15.97:0/4180406077 wait complete. 2024-03-06T07:37:29.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.300+0000 7f9107e00640 1 Processor -- start 2024-03-06T07:37:29.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.300+0000 7f9107e00640 1 -- start start 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f9107e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9108067000 0x7f91080b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f9107e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 0x7f91080b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f9107e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910815e0d0 0x7f9108159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f9107e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910815e630 con 0x7f910806bd20 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f9107e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910815e4b0 con 0x7f910815e0d0 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f9107e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91080b5ba0 con 0x7f9108067000 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f90ffe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 0x7f91080b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f90ffe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 0x7f91080b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38384/0 (socket says 172.21.15.97:38384) 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.301+0000 7f90ffe00640 1 -- 172.21.15.97:0/1396714823 learned_addr learned my addr 172.21.15.97:0/1396714823 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f9107400640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910815e0d0 0x7f9108159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f9106a00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9108067000 0x7f91080b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90ffe00640 1 -- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9108067000 msgr2=0x7f91080b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90ffe00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9108067000 0x7f91080b07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90ffe00640 1 -- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910815e0d0 msgr2=0x7f9108159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90ffe00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910815e0d0 0x7f9108159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90ffe00640 1 -- 172.21.15.97:0/1396714823 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910815e7b0 con 0x7f910806bd20 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f9107400640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910815e0d0 0x7f9108159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90ffe00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 0x7f91080b2700 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f90f0000c00 tx=0x7f90f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0040070 con 0x7f910806bd20 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f00399f0 con 0x7f910806bd20 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9108160640 con 0x7f910806bd20 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.302+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0035040 con 0x7f910806bd20 2024-03-06T07:37:29.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.303+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9108159fa0 con 0x7f910806bd20 2024-03-06T07:37:29.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.304+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910804f330 con 0x7f910806bd20 2024-03-06T07:37:29.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.305+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90f004b060 con 0x7f910806bd20 2024-03-06T07:37:29.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.305+0000 7f90fea00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f90e406b4d0 0x7f90e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.306+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90f00abaf0 con 0x7f910806bd20 2024-03-06T07:37:29.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.306+0000 7f9106a00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f90e406b4d0 0x7f90e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.306+0000 7f9106a00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f90e406b4d0 0x7f90e406d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f90f8006fd0 tx=0x7f90f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.310+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90f007aa00 con 0x7f910806bd20 2024-03-06T07:37:29.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.555+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f91080b2c40 con 0x7f910806bd20 2024-03-06T07:37:29.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.556+0000 7f90fea00640 1 -- 172.21.15.97:0/1396714823 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v107) v1 ==== 53+0+73 (secure 0 0 0) 0x7f90f007a300 con 0x7f910806bd20 2024-03-06T07:37:29.558 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f90e406b4d0 msgr2=0x7f90e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f90e406b4d0 0x7f90e406d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f90f8006fd0 tx=0x7f90f8008040 comp rx=0 tx=0).stop 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 msgr2=0x7f91080b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 0x7f91080b2700 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f90f0000c00 tx=0x7f90f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 shutdown_connections 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f90e406b4d0 0x7f90e406d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910815e0d0 0x7f9108159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f910806bd20 0x7f91080b2700 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 --2- 172.21.15.97:0/1396714823 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9108067000 0x7f91080b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 >> 172.21.15.97:0/1396714823 conn(0x7f91080725e0 msgr2=0x7f91080729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 shutdown_connections 2024-03-06T07:37:29.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:29.559+0000 7f9107e00640 1 -- 172.21.15.97:0/1396714823 wait complete. 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:37:29.581 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:29.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:29.581 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:29.634 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 42025sec preferred_lft 42025sec 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:29.635 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:37:29.636 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:29.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:29.637 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:37:29.718 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:37:29.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:29.722 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:37:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:37:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:37:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:37:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:37:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:37:29.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:29.769 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:37:29.769 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:37:29.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:37:29.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:37:29.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:37:29.769 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:29.988 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:37:30.036 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:37:30.036 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:37:30.037 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:30.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:30.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:37:30.124 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:37:30.125 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:30.177 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:30.177 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:37:30.177 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:30.221 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:37:30.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:30.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:37:30.395 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:37:30.395 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:37:30.427 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:37:30.427 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:37:30.427 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:37:30.427 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:37:30.427 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:37:30.430 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:37:30.430 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:37:30.430 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:37:30.430 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:37:30.430 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:37:30.434 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:37:30.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:30.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:30.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:30.506 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:37:30.564 INFO:teuthology.orchestra.run.smithi097.stdout:client6541 2024-03-06T07:37:30.564 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:30.564 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:30.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:37:30.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.858+0000 7fcddb800640 1 Processor -- start 2024-03-06T07:37:30.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.858+0000 7fcddb800640 1 -- start start 2024-03-06T07:37:30.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.859+0000 7fcddb800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdd4070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.859+0000 7fcddb800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd40b46c0 con 0x7fcdd415fff0 2024-03-06T07:37:30.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcddb800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd40b49a0 con 0x7fcdd40b3f70 2024-03-06T07:37:30.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcddb800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4072c90 con 0x7fcdd415fc10 2024-03-06T07:37:30.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcdd3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcdd415fff0 0x7fcdd4160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49396/0 (socket says 172.21.15.97:49396) 2024-03-06T07:37:30.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcdd3e00640 1 -- 172.21.15.97:0/4024062175 learned_addr learned my addr 172.21.15.97:0/4024062175 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdd4070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415909945 0 0) 0x7fcdd40b46c0 con 0x7fcdd415fff0 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.860+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb8003520 con 0x7fcdd415fff0 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588620000 0 0) 0x7fcdb8003520 con 0x7fcdd415fff0 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 msgr2=0x7fcdd4070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdd4070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 >> v1:172.21.15.135:6789/0 conn(0x7fcdd40b3f70 legacy=0x7fcdd40ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd409dd80 con 0x7fcdd415fff0 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fcdbc001670 con 0x7fcdd415fff0 2024-03-06T07:37:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 >> v1:172.21.15.97:6789/0 conn(0x7fcdd415fff0 legacy=0x7fcdd4160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdb8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdb8004800 0x7fcdb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4072c90 con 0x7fcdb80039e0 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800baa0 con 0x7fcdb8004800 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800bca0 con 0x7fcdd415fc10 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.861+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdb8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd3e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdb8004800 0x7fcdb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd9800640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd8e00640 1 -- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 msgr2=0x7fcdb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd8e00640 1 -- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdb8004800 msgr2=0x7fcdb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdb8004800 0x7fcdb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd8e00640 1 -- 172.21.15.97:0/4024062175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8003520 con 0x7fcdb80039e0 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd9800640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd415fc10 0x7fcdb80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.862+0000 7fcdd3e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdb8004800 0x7fcdb8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:30.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.863+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdb8003e00 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009f90 tx=0x7fcdb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7745a3e51d0151c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.863+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4039070 con 0x7fcdb80039e0 2024-03-06T07:37:30.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.863+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc402cba0 con 0x7fcdb80039e0 2024-03-06T07:37:30.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.863+0000 7fcdd3400640 1 -- 172.21.15.97:0/4024062175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4009c10 con 0x7fcdb80039e0 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcddb800640 1 -- 172.21.15.97:0/4024062175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 msgr2=0x7fcdb8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcddb800640 1 --2- 172.21.15.97:0/4024062175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdb8003e00 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009f90 tx=0x7fcdb8005830 comp rx=0 tx=0).stop 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcdd9800640 1 -- 172.21.15.97:0/4024062175 reap_dead start 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcddb800640 1 -- 172.21.15.97:0/4024062175 shutdown_connections 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcddb800640 1 -- 172.21.15.97:0/4024062175 >> 172.21.15.97:0/4024062175 conn(0x7fcdd40b1940 msgr2=0x7fcdd40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcddb800640 1 -- 172.21.15.97:0/4024062175 shutdown_connections 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.864+0000 7fcddb800640 1 -- 172.21.15.97:0/4024062175 wait complete. 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.865+0000 7fcddb800640 1 Processor -- start 2024-03-06T07:37:30.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.866+0000 7fcddb800640 1 -- start start 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.866+0000 7fcddb800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdd4143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcddb800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdd41f4220 0x7fcdd409a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcddb800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd406d480 0x7fcdd406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcddb800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd415c460 con 0x7fcdb80039e0 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcddb800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd415c2e0 con 0x7fcdd41f4220 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcddb800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4070ff0 con 0x7fcdd406d480 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdd4143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdd4143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38402/0 (socket says 172.21.15.97:38402) 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 -- 172.21.15.97:0/2162312495 learned_addr learned my addr 172.21.15.97:0/2162312495 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 -- 172.21.15.97:0/2162312495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd406d480 msgr2=0x7fcdd406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd406d480 0x7fcdd406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 -- 172.21.15.97:0/2162312495 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdd41f4220 msgr2=0x7fcdd409a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdd41f4220 0x7fcdd409a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.867+0000 7fcdd8e00640 1 -- 172.21.15.97:0/2162312495 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd409b360 con 0x7fcdb80039e0 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.868+0000 7fcdd8e00640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdd4143310 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fcdc402c610 tx=0x7fcdc4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.868+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4042020 con 0x7fcdb80039e0 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.868+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc403a040 con 0x7fcdb80039e0 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.868+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd409b640 con 0x7fcdb80039e0 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.868+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4039070 con 0x7fcdb80039e0 2024-03-06T07:37:30.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.868+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd409c600 con 0x7fcdb80039e0 2024-03-06T07:37:30.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.869+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd404f330 con 0x7fcdb80039e0 2024-03-06T07:37:30.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.870+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdc4005d10 con 0x7fcdb80039e0 2024-03-06T07:37:30.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.871+0000 7fcdd0c00640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcdb006b4d0 0x7fcdb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.871+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcdc40ac6d0 con 0x7fcdb80039e0 2024-03-06T07:37:30.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.871+0000 7fcdd3e00640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcdb006b4d0 0x7fcdb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.872+0000 7fcdd3e00640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcdb006b4d0 0x7fcdb006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4146530 tx=0x7fcdbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:30.876+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdc404c030 con 0x7fcdb80039e0 2024-03-06T07:37:31.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.114+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fcdd406ef30 con 0x7fcdb80039e0 2024-03-06T07:37:31.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.453+0000 7fcdd0c00640 1 -- 172.21.15.97:0/2162312495 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v108) v1 ==== 105+0+0 (secure 0 0 0) 0x7fcdc407b5a0 con 0x7fcdb80039e0 2024-03-06T07:37:31.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcdb006b4d0 msgr2=0x7fcdb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcdb006b4d0 0x7fcdb006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4146530 tx=0x7fcdbc009040 comp rx=0 tx=0).stop 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 msgr2=0x7fcdd4143310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdd4143310 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fcdc402c610 tx=0x7fcdc4009750 comp rx=0 tx=0).stop 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 shutdown_connections 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcdb006b4d0 0x7fcdb006d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcdd406d480 0x7fcdd406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdd41f4220 0x7fcdd409a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 --2- 172.21.15.97:0/2162312495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdb80039e0 0x7fcdd4143310 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 >> 172.21.15.97:0/2162312495 conn(0x7fcdd40b1940 msgr2=0x7fcdd40ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.456+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 shutdown_connections 2024-03-06T07:37:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.457+0000 7fcddb800640 1 -- 172.21.15.97:0/2162312495 wait complete. 2024-03-06T07:37:31.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:31.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.670+0000 7f5d75200640 1 Processor -- start 2024-03-06T07:37:31.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.671+0000 7f5d75200640 1 -- start start 2024-03-06T07:37:31.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d75200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5d700d24c0 0x7f5d700d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d75200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d700b2de0 0x7f5d700d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d75200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700d07f0 0x7f5d700d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d75200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d700d4a00 con 0x7f5d700b2de0 2024-03-06T07:37:31.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d75200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d700cf7e0 con 0x7f5d700d24c0 2024-03-06T07:37:31.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d75200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700d0080 con 0x7f5d700d07f0 2024-03-06T07:37:31.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d65400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d700b2de0 0x7f5d700d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d65400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5d700b2de0 0x7f5d700d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38410/0 (socket says 172.21.15.97:38410) 2024-03-06T07:37:31.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.672+0000 7f5d65400640 1 -- 172.21.15.97:0/1423852311 learned_addr learned my addr 172.21.15.97:0/1423852311 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d6de00640 1 --2- 172.21.15.97:0/1423852311 >> v2:172.21.15.135:3300/0 conn(0x7f5d700d24c0 0x7f5d700d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d6e800640 1 --2- 172.21.15.97:0/1423852311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700d07f0 0x7f5d700d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d65400640 1 -- 172.21.15.97:0/1423852311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700d07f0 msgr2=0x7f5d700d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d65400640 1 --2- 172.21.15.97:0/1423852311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700d07f0 0x7f5d700d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d65400640 1 -- 172.21.15.97:0/1423852311 >> v2:172.21.15.135:3300/0 conn(0x7f5d700d24c0 msgr2=0x7f5d700d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d65400640 1 --2- 172.21.15.97:0/1423852311 >> v2:172.21.15.135:3300/0 conn(0x7f5d700d24c0 0x7f5d700d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.673+0000 7f5d65400640 1 -- 172.21.15.97:0/1423852311 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700d2de0 con 0x7f5d700b2de0 2024-03-06T07:37:31.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.675+0000 7f5d65400640 1 --2- 172.21.15.97:0/1423852311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700b2de0 0x7f5d700d28a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f5d7008d300 tx=0x7f5d5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ebb4b3419bd276 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.675+0000 7f5d6d400640 1 -- 172.21.15.97:0/1423852311 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5802e070 con 0x7f5d700b2de0 2024-03-06T07:37:31.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.675+0000 7f5d6d400640 1 -- 172.21.15.97:0/1423852311 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d58037070 con 0x7f5d700b2de0 2024-03-06T07:37:31.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.675+0000 7f5d6d400640 1 -- 172.21.15.97:0/1423852311 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58031a00 con 0x7f5d700b2de0 2024-03-06T07:37:31.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 -- 172.21.15.97:0/1423852311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700b2de0 msgr2=0x7f5d700d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 --2- 172.21.15.97:0/1423852311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700b2de0 0x7f5d700d28a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f5d7008d300 tx=0x7f5d5802b880 comp rx=0 tx=0).stop 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 -- 172.21.15.97:0/1423852311 shutdown_connections 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 --2- 172.21.15.97:0/1423852311 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700d07f0 0x7f5d700d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 --2- 172.21.15.97:0/1423852311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700b2de0 0x7f5d700d28a0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 --2- 172.21.15.97:0/1423852311 >> v2:172.21.15.135:3300/0 conn(0x7f5d700d24c0 0x7f5d700d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 -- 172.21.15.97:0/1423852311 >> 172.21.15.97:0/1423852311 conn(0x7f5d700d3ee0 msgr2=0x7f5d700d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 -- 172.21.15.97:0/1423852311 shutdown_connections 2024-03-06T07:37:31.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.676+0000 7f5d75200640 1 -- 172.21.15.97:0/1423852311 wait complete. 2024-03-06T07:37:31.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.679+0000 7f5d75200640 1 Processor -- start 2024-03-06T07:37:31.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.679+0000 7f5d75200640 1 -- start start 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d75200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700b2de0 0x7f5d7015e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d75200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d700d07f0 0x7f5d7015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d75200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 0x7f5d700cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d75200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700cfd10 con 0x7f5d700d24c0 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d75200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700cfa30 con 0x7f5d700d07f0 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d75200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7015e5b0 con 0x7f5d700b2de0 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d6e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 0x7f5d700cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d6e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 0x7f5d700cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38418/0 (socket says 172.21.15.97:38418) 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d6e800640 1 -- 172.21.15.97:0/4051722760 learned_addr learned my addr 172.21.15.97:0/4051722760 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d6de00640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700b2de0 0x7f5d7015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.681+0000 7f5d65400640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d700d07f0 0x7f5d7015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d6e800640 1 -- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700b2de0 msgr2=0x7f5d7015e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d6e800640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700b2de0 0x7f5d7015e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d6e800640 1 -- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d700d07f0 msgr2=0x7f5d7015edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d6e800640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d700d07f0 0x7f5d7015edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d6e800640 1 -- 172.21.15.97:0/4051722760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700c9470 con 0x7f5d700d24c0 2024-03-06T07:37:31.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d6e800640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 0x7f5d700cb1f0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d600 tx=0x7f5d6800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d68018020 con 0x7f5d700d24c0 2024-03-06T07:37:31.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d68014620 con 0x7f5d700d24c0 2024-03-06T07:37:31.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d680149f0 con 0x7f5d700d24c0 2024-03-06T07:37:31.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.682+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d700c9750 con 0x7f5d700d24c0 2024-03-06T07:37:31.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.683+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d700c9d10 con 0x7f5d700d24c0 2024-03-06T07:37:31.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.684+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7004f330 con 0x7f5d700d24c0 2024-03-06T07:37:31.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.685+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d6801b020 con 0x7f5d700d24c0 2024-03-06T07:37:31.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.685+0000 7f5d66a00640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d4c06b500 0x7f5d4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.686+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5d6808a4b0 con 0x7f5d700d24c0 2024-03-06T07:37:31.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.686+0000 7f5d6de00640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d4c06b500 0x7f5d4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.687+0000 7f5d6de00640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d4c06b500 0x7f5d4c06d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009a60 tx=0x7f5d60003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.690+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d68022030 con 0x7f5d700d24c0 2024-03-06T07:37:31.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.939+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d700d22a0 con 0x7f5d700d24c0 2024-03-06T07:37:31.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.940+0000 7f5d66a00640 1 -- 172.21.15.97:0/4051722760 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5006 (secure 0 0 0) 0x7f5d68059270 con 0x7f5d700d24c0 2024-03-06T07:37:31.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:31.943 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":109,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5840,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2881704649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":2881704649},{"type":"v1","addr":"172.21.15.135:6835","nonce":2881704649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6448,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/281989301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":281989301},{"type":"v1","addr":"172.21.15.135:6834","nonce":281989301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:31.458964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:creating","state_seq":1,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:37:31.943 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 109 2024-03-06T07:37:31.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d4c06b500 msgr2=0x7f5d4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d4c06b500 0x7f5d4c06d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009a60 tx=0x7f5d60003990 comp rx=0 tx=0).stop 2024-03-06T07:37:31.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 msgr2=0x7f5d700cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 0x7f5d700cb1f0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d600 tx=0x7f5d6800dac0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 shutdown_connections 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5d4c06b500 0x7f5d4c06d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d700d24c0 0x7f5d700cb1f0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d700d07f0 0x7f5d7015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 --2- 172.21.15.97:0/4051722760 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5d700b2de0 0x7f5d7015e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 >> 172.21.15.97:0/4051722760 conn(0x7f5d700d3ee0 msgr2=0x7f5d700d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 shutdown_connections 2024-03-06T07:37:31.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:31.943+0000 7f5d75200640 1 -- 172.21.15.97:0/4051722760 wait complete. 2024-03-06T07:37:31.964 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:37:36.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:37.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.175+0000 7fa86a400640 1 Processor -- start 2024-03-06T07:37:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.175+0000 7fa86a400640 1 -- start start 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa86a400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa864144660 0x7fa864071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa86a400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8640ca5d0 0x7fa8640724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa86a400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8640cabf0 con 0x7fa8640cf3e0 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa86a400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8640ca9b0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa86a400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8640ce900 con 0x7fa864144660 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa863e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8640ca5d0 0x7fa8640724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa863e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa8640ca5d0 0x7fa8640724b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51270/0 (socket says 172.21.15.97:51270) 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa863e00640 1 -- 172.21.15.97:0/3795156159 learned_addr learned my addr 172.21.15.97:0/3795156159 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.178+0000 7fa862a00640 1 --1- 172.21.15.97:0/3795156159 >> v1:172.21.15.97:6789/0 conn(0x7fa8640cf3e0 0x7fa8640c9ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58918/0 (socket says 172.21.15.97:58918) 2024-03-06T07:37:37.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.179+0000 7fa863400640 1 --2- 172.21.15.97:0/3795156159 >> v2:172.21.15.135:3300/0 conn(0x7fa864144660 0x7fa864071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.179+0000 7fa863e00640 1 -- 172.21.15.97:0/3795156159 >> v2:172.21.15.135:3300/0 conn(0x7fa864144660 msgr2=0x7fa864071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.179+0000 7fa863e00640 1 --2- 172.21.15.97:0/3795156159 >> v2:172.21.15.135:3300/0 conn(0x7fa864144660 0x7fa864071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.179+0000 7fa863e00640 1 -- 172.21.15.97:0/3795156159 >> v1:172.21.15.97:6789/0 conn(0x7fa8640cf3e0 legacy=0x7fa8640c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.179+0000 7fa863e00640 1 -- 172.21.15.97:0/3795156159 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8640cebe0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.180+0000 7fa863e00640 1 --2- 172.21.15.97:0/3795156159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa8640724b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fa8640d1850 tx=0x7fa85c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be3301b714aaac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.180+0000 7fa862000640 1 -- 172.21.15.97:0/3795156159 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c02cb10 con 0x7fa8640ca5d0 2024-03-06T07:37:37.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.180+0000 7fa862000640 1 -- 172.21.15.97:0/3795156159 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85c02ccb0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa862000640 1 -- 172.21.15.97:0/3795156159 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c02ce60 con 0x7fa8640ca5d0 2024-03-06T07:37:37.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa86a400640 1 -- 172.21.15.97:0/3795156159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 msgr2=0x7fa8640724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa86a400640 1 --2- 172.21.15.97:0/3795156159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa8640724b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fa8640d1850 tx=0x7fa85c02bf70 comp rx=0 tx=0).stop 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa86a400640 1 -- 172.21.15.97:0/3795156159 shutdown_connections 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa86a400640 1 --2- 172.21.15.97:0/3795156159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa8640724b0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa86a400640 1 --2- 172.21.15.97:0/3795156159 >> v2:172.21.15.135:3300/0 conn(0x7fa864144660 0x7fa864071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.181+0000 7fa86a400640 1 -- 172.21.15.97:0/3795156159 >> 172.21.15.97:0/3795156159 conn(0x7fa8640d0b90 msgr2=0x7fa8640d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.182+0000 7fa86a400640 1 -- 172.21.15.97:0/3795156159 shutdown_connections 2024-03-06T07:37:37.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.182+0000 7fa86a400640 1 -- 172.21.15.97:0/3795156159 wait complete. 2024-03-06T07:37:37.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.183+0000 7fa86a400640 1 Processor -- start 2024-03-06T07:37:37.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.183+0000 7fa86a400640 1 -- start start 2024-03-06T07:37:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa86a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa86413f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa86a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8640cf3e0 0x7fa8641f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa86a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa864144660 0x7fa8641f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa86a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640cc0f0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa86a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa864144a40 con 0x7fa864144660 2024-03-06T07:37:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa86a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640cc270 con 0x7fa8640cf3e0 2024-03-06T07:37:37.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa863400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa86413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa863400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa86413f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51284/0 (socket says 172.21.15.97:51284) 2024-03-06T07:37:37.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.185+0000 7fa863400640 1 -- 172.21.15.97:0/74826463 learned_addr learned my addr 172.21.15.97:0/74826463 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa863400640 1 -- 172.21.15.97:0/74826463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8640cf3e0 msgr2=0x7fa8641f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa863400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8640cf3e0 0x7fa8641f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa863400640 1 -- 172.21.15.97:0/74826463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa864144660 msgr2=0x7fa8641f7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa863400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa864144660 0x7fa8641f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa863400640 1 -- 172.21.15.97:0/74826463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8641f3860 con 0x7fa8640ca5d0 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa863400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa86413f530 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fa85400d5f0 tx=0x7fa85400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8540125f0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa854019070 con 0x7fa8640ca5d0 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8540129d0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.186+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8641f7a10 con 0x7fa8640ca5d0 2024-03-06T07:37:37.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.187+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8640b39d0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.188+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86404f330 con 0x7fa8640ca5d0 2024-03-06T07:37:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.189+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa854012b70 con 0x7fa8640ca5d0 2024-03-06T07:37:37.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.190+0000 7fa85b400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa84406b4d0 0x7fa84406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.190+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa85408a6b0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.190+0000 7fa862a00640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa84406b4d0 0x7fa84406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.191+0000 7fa862a00640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa84406b4d0 0x7fa84406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa850000f60 tx=0x7fa850007a00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.194+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa854059470 con 0x7fa8640ca5d0 2024-03-06T07:37:37.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.486+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8641438c0 con 0x7fa8640ca5d0 2024-03-06T07:37:37.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.487+0000 7fa85b400640 1 -- 172.21.15.97:0/74826463 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5009 (secure 0 0 0) 0x7fa854058d70 con 0x7fa8640ca5d0 2024-03-06T07:37:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:37.490 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":110,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5840,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2881704649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":2881704649},{"type":"v1","addr":"172.21.15.135:6835","nonce":2881704649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6448,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/281989301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":281989301},{"type":"v1","addr":"172.21.15.135:6834","nonce":281989301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:32.462565+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[6463,9024]},"id":10}]} 2024-03-06T07:37:37.490 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 110 2024-03-06T07:37:37.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa84406b4d0 msgr2=0x7fa84406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa84406b4d0 0x7fa84406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa850000f60 tx=0x7fa850007a00 comp rx=0 tx=0).stop 2024-03-06T07:37:37.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 msgr2=0x7fa86413f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa86413f530 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fa85400d5f0 tx=0x7fa85400dab0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 shutdown_connections 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa84406b4d0 0x7fa84406d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa864144660 0x7fa8641f7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa8640cf3e0 0x7fa8641f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 --2- 172.21.15.97:0/74826463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8640ca5d0 0x7fa86413f530 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 >> 172.21.15.97:0/74826463 conn(0x7fa8640d0b90 msgr2=0x7fa8640cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 shutdown_connections 2024-03-06T07:37:37.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.491+0000 7fa86a400640 1 -- 172.21.15.97:0/74826463 wait complete. 2024-03-06T07:37:37.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:37.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.706+0000 7fe503800640 1 Processor -- start 2024-03-06T07:37:37.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.707+0000 7fe503800640 1 -- start start 2024-03-06T07:37:37.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.708+0000 7fe503800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 0x7fe4fc0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.708+0000 7fe503800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc15b090 con 0x7fe4fc15acb0 2024-03-06T07:37:37.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.708+0000 7fe503800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc0b2a30 con 0x7fe4fc0b4680 2024-03-06T07:37:37.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.708+0000 7fe503800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0b4360 con 0x7fe4fc06ec00 2024-03-06T07:37:37.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe500e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe4fc15acb0 0x7fe4fc0b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58920/0 (socket says 172.21.15.97:58920) 2024-03-06T07:37:37.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe500e00640 1 -- 172.21.15.97:0/3630967116 learned_addr learned my addr 172.21.15.97:0/3630967116 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380841531 0 0) 0x7fe4fc15b090 con 0x7fe4fc15acb0 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe501800640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 0x7fe4fc0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4005730 con 0x7fe4fc15acb0 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594581994 0 0) 0x7fe4e4005730 con 0x7fe4fc15acb0 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 msgr2=0x7fe4fc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 0x7fe4fc0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 >> v1:172.21.15.135:6789/0 conn(0x7fe4fc0b4680 legacy=0x7fe4fc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc0b2cf0 con 0x7fe4fc15acb0 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.709+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe4f0001670 con 0x7fe4fc15acb0 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 >> v1:172.21.15.97:6789/0 conn(0x7fe4fc15acb0 legacy=0x7fe4fc0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 0x7fe4e4005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4e4006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4e4006f50 0x7fe4e40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0b4360 con 0x7fe4e4006220 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e400dd90 con 0x7fe4e4006f50 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e400df90 con 0x7fe4fc06ec00 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4e4006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe500e00640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 0x7fe4e4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.710+0000 7fe501800640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4e4006f50 0x7fe4e40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.711+0000 7fe4fbe00640 1 -- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 msgr2=0x7fe4e4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.711+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc06ec00 0x7fe4e4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.711+0000 7fe4fbe00640 1 -- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4e4006f50 msgr2=0x7fe4e40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.711+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4e4006f50 0x7fe4e40073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.711+0000 7fe4fbe00640 1 -- 172.21.15.97:0/3630967116 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4005730 con 0x7fe4e4006220 2024-03-06T07:37:37.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.712+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4e4006a10 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8001750 tx=0x7fe4e4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de27a1a7ff3126ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.713+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8001370 con 0x7fe4e4006220 2024-03-06T07:37:37.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.713+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e80052d0 con 0x7fe4e4006220 2024-03-06T07:37:37.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.713+0000 7fe4fb400640 1 -- 172.21.15.97:0/3630967116 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8003830 con 0x7fe4e4006220 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe503800640 1 -- 172.21.15.97:0/3630967116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 msgr2=0x7fe4e4006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe503800640 1 --2- 172.21.15.97:0/3630967116 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4e4006a10 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8001750 tx=0x7fe4e4007a40 comp rx=0 tx=0).stop 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe501800640 1 -- 172.21.15.97:0/3630967116 reap_dead start 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe503800640 1 -- 172.21.15.97:0/3630967116 shutdown_connections 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe503800640 1 -- 172.21.15.97:0/3630967116 >> 172.21.15.97:0/3630967116 conn(0x7fe4fc0b1730 msgr2=0x7fe4fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe503800640 1 -- 172.21.15.97:0/3630967116 shutdown_connections 2024-03-06T07:37:37.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.714+0000 7fe503800640 1 -- 172.21.15.97:0/3630967116 wait complete. 2024-03-06T07:37:37.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.715+0000 7fe503800640 1 Processor -- start 2024-03-06T07:37:37.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.716+0000 7fe503800640 1 -- start start 2024-03-06T07:37:37.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.716+0000 7fe503800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4fc1f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe503800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc15b090 0x7fe4fc1f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe503800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4fc15c5f0 0x7fe4fc15caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe503800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0b4010 con 0x7fe4e4006220 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe503800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0b32b0 con 0x7fe4fc15c5f0 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe503800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc072340 con 0x7fe4fc15b090 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4fc1f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4fc1f48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51302/0 (socket says 172.21.15.97:51302) 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 -- 172.21.15.97:0/639809737 learned_addr learned my addr 172.21.15.97:0/639809737 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe501800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4fc15c5f0 0x7fe4fc15caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 -- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc15b090 msgr2=0x7fe4fc1f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc15b090 0x7fe4fc1f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc15b090 0x7fe4fc1f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 -- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4fc15c5f0 msgr2=0x7fe4fc15caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4fc15c5f0 0x7fe4fc15caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe500e00640 1 -- 172.21.15.97:0/639809737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc1f4df0 con 0x7fe4e4006220 2024-03-06T07:37:37.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.717+0000 7fe501800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4fc15c5f0 0x7fe4fc15caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:37.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.718+0000 7fe500e00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4fc1f48b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000e830 tx=0x7fe4f000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.718+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000c870 con 0x7fe4e4006220 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.718+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0018070 con 0x7fe4e4006220 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.718+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc15b550 con 0x7fe4e4006220 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.718+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000cc50 con 0x7fe4e4006220 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.718+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc15bb10 con 0x7fe4e4006220 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.719+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc04f330 con 0x7fe4e4006220 2024-03-06T07:37:37.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.720+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4f001d460 con 0x7fe4e4006220 2024-03-06T07:37:37.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.722+0000 7fe4f8c00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.722+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe4f008a100 con 0x7fe4e4006220 2024-03-06T07:37:37.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.722+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.723+0000 7fe4fbe00640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8033000 tx=0x7fe4e8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.726+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4fc1f4df0 con 0x7fe4e4006220 2024-03-06T07:37:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.969+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4fc06ce60 con 0x7fe4e4006220 2024-03-06T07:37:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.970+0000 7fe4f8c00640 1 -- 172.21.15.97:0/639809737 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5009 (secure 0 0 0) 0x7fe4f0058ec0 con 0x7fe4e4006220 2024-03-06T07:37:37.972 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:37.972 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":110,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5840,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/2881704649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6833","nonce":2881704649},{"type":"v1","addr":"172.21.15.135:6835","nonce":2881704649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6448,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6834/281989301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":281989301},{"type":"v1","addr":"172.21.15.135:6834","nonce":281989301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:32.462565+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[6463,9024]},"id":10}]} 2024-03-06T07:37:37.973 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 110 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.973+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe4dc06b500 msgr2=0x7fe4dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.973+0000 7fe503800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8033000 tx=0x7fe4e8009030 comp rx=0 tx=0).stop 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.973+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 msgr2=0x7fe4fc1f48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.973+0000 7fe503800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4fc1f48b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000e830 tx=0x7fe4f000ec30 comp rx=0 tx=0).stop 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 shutdown_connections 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4fc15c5f0 0x7fe4fc15caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe4fc15b090 0x7fe4fc1f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 --2- 172.21.15.97:0/639809737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4e4006220 0x7fe4fc1f48b0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 >> 172.21.15.97:0/639809737 conn(0x7fe4fc0b1730 msgr2=0x7fe4fc0d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 shutdown_connections 2024-03-06T07:37:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:37.974+0000 7fe503800640 1 -- 172.21.15.97:0/639809737 wait complete. 2024-03-06T07:37:38.000 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and a 2024-03-06T07:37:38.001 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:37:38.001 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:37:38.022 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:37:38.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:37:38.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.220+0000 7f64ee800640 1 Processor -- start 2024-03-06T07:37:38.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.220+0000 7f64ee800640 1 -- start start 2024-03-06T07:37:38.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.222+0000 7f64ee800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 0x7f64e80b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ee800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e806d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ee800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80b2b30 con 0x7f64e80cc2a0 2024-03-06T07:37:38.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ee800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e80b28e0 con 0x7f64e815a940 2024-03-06T07:37:38.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ee800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b2e10 con 0x7f64e80cf370 2024-03-06T07:37:38.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ed400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 0x7f64e80b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64eca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ed400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 0x7f64e80b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:43550/0 (socket says 172.21.15.97:43550) 2024-03-06T07:37:38.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.223+0000 7f64ed400640 1 -- 172.21.15.97:0/2908362407 learned_addr learned my addr 172.21.15.97:0/2908362407 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:38.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.224+0000 7f64eca00640 1 -- 172.21.15.97:0/2908362407 >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 msgr2=0x7f64e80b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:38.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.224+0000 7f64eca00640 1 --2- 172.21.15.97:0/2908362407 >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 0x7f64e80b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.224+0000 7f64eca00640 1 -- 172.21.15.97:0/2908362407 >> v1:172.21.15.135:6789/0 conn(0x7f64e80cc2a0 legacy=0x7f64e80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.224+0000 7f64eca00640 1 -- 172.21.15.97:0/2908362407 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e806ca50 con 0x7f64e80cf370 2024-03-06T07:37:38.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.224+0000 7f64ed400640 1 --2- 172.21.15.97:0/2908362407 >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 0x7f64e80b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:37:38.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.225+0000 7f64eca00640 1 --2- 172.21.15.97:0/2908362407 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e806d250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097a0 tx=0x7f64d402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2017e2177f99c3dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.226+0000 7f64e7e00640 1 -- 172.21.15.97:0/2908362407 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4035020 con 0x7f64e80cf370 2024-03-06T07:37:38.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.226+0000 7f64e7e00640 1 -- 172.21.15.97:0/2908362407 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d4031810 con 0x7f64e80cf370 2024-03-06T07:37:38.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.226+0000 7f64e7e00640 1 -- 172.21.15.97:0/2908362407 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4031be0 con 0x7f64e80cf370 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 -- 172.21.15.97:0/2908362407 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 msgr2=0x7f64e806d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 --2- 172.21.15.97:0/2908362407 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e806d250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097a0 tx=0x7f64d402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 -- 172.21.15.97:0/2908362407 shutdown_connections 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 --2- 172.21.15.97:0/2908362407 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e806d250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 --2- 172.21.15.97:0/2908362407 >> v2:172.21.15.135:3300/0 conn(0x7f64e815a940 0x7f64e80b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 -- 172.21.15.97:0/2908362407 >> 172.21.15.97:0/2908362407 conn(0x7f64e806afc0 msgr2=0x7f64e806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 -- 172.21.15.97:0/2908362407 shutdown_connections 2024-03-06T07:37:38.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.227+0000 7f64ee800640 1 -- 172.21.15.97:0/2908362407 wait complete. 2024-03-06T07:37:38.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.230+0000 7f64ee800640 1 Processor -- start 2024-03-06T07:37:38.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.230+0000 7f64ee800640 1 -- start start 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.231+0000 7f64ee800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 0x7f64e815d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.231+0000 7f64ee800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e814c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.231+0000 7f64ee800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64e815a940 0x7f64e80ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.231+0000 7f64ee800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b4b20 con 0x7f64e80cc2a0 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.231+0000 7f64ee800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b30f0 con 0x7f64e815a940 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.231+0000 7f64ee800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e806d790 con 0x7f64e80cf370 2024-03-06T07:37:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 0x7f64e815d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 0x7f64e815d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51314/0 (socket says 172.21.15.97:51314) 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 -- 172.21.15.97:0/573544628 learned_addr learned my addr 172.21.15.97:0/573544628 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64eca00640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e814c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ede00640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64e815a940 0x7f64e80ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 -- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 msgr2=0x7f64e814c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e814c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 -- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64e815a940 msgr2=0x7f64e80ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64e815a940 0x7f64e80ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.232+0000 7f64ed400640 1 -- 172.21.15.97:0/573544628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80ab980 con 0x7f64e80cc2a0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.233+0000 7f64ed400640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 0x7f64e815d010 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00e0f0 tx=0x7f64dc00e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.233+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc00b310 con 0x7f64e80cc2a0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.233+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc009600 con 0x7f64e80cc2a0 2024-03-06T07:37:38.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.233+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0099b0 con 0x7f64e80cc2a0 2024-03-06T07:37:38.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.233+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e80aa8e0 con 0x7f64e80cc2a0 2024-03-06T07:37:38.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.233+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e80aaea0 con 0x7f64e80cc2a0 2024-03-06T07:37:38.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.234+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e80ab5c0 con 0x7f64e80cc2a0 2024-03-06T07:37:38.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.236+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64dc005d10 con 0x7f64e80cc2a0 2024-03-06T07:37:38.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.237+0000 7f64e5600640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64c806b500 0x7f64c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.238+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f64dc08cb40 con 0x7f64e80cc2a0 2024-03-06T07:37:38.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.238+0000 7f64eca00640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64c806b500 0x7f64c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.238+0000 7f64eca00640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64c806b500 0x7f64c806d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f64d4036000 tx=0x7f64d4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.242+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64dc090030 con 0x7f64e80cc2a0 2024-03-06T07:37:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:38.555+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f64e804f330 con 0x7f64e80cc2a0 2024-03-06T07:37:39.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:39.438 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:39.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.558+0000 7f64e5600640 1 -- 172.21.15.97:0/573544628 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6448 v111) v1 ==== 95+0+0 (secure 0 0 0) 0x7f64dc024020 con 0x7f64e80cc2a0 2024-03-06T07:37:39.560 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 6448 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.561+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64c806b500 msgr2=0x7f64c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.561+0000 7f64ee800640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64c806b500 0x7f64c806d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f64d4036000 tx=0x7f64d4008030 comp rx=0 tx=0).stop 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.561+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 msgr2=0x7f64e815d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.561+0000 7f64ee800640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 0x7f64e815d010 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00e0f0 tx=0x7f64dc00e5b0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 shutdown_connections 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f64c806b500 0x7f64c806d9c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64e815a940 0x7f64e80ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64e80cf370 0x7f64e814c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 --2- 172.21.15.97:0/573544628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f64e80cc2a0 0x7f64e815d010 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 >> 172.21.15.97:0/573544628 conn(0x7f64e806afc0 msgr2=0x7f64e815e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 shutdown_connections 2024-03-06T07:37:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.562+0000 7f64ee800640 1 -- 172.21.15.97:0/573544628 wait complete. 2024-03-06T07:37:39.589 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:37:39.589 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:37:39.612 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:37:39.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:37:39.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.822+0000 7fdfa3c00640 1 Processor -- start 2024-03-06T07:37:39.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.823+0000 7fdfa3c00640 1 -- start start 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.825+0000 7fdfa3c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf9c06db80 0x7fdf9c0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.825+0000 7fdfa3c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 0x7fdf9c0ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.825+0000 7fdfa3c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c15b9e0 con 0x7fdf9c1581d0 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.825+0000 7fdfa3c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cbff0 con 0x7fdf9c06db80 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.825+0000 7fdfa3c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cc220 con 0x7fdf9c06e470 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.826+0000 7fdfa1200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdf9c1581d0 0x7fdf9c06d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58932/0 (socket says 172.21.15.97:58932) 2024-03-06T07:37:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.826+0000 7fdfa1200640 1 -- 172.21.15.97:0/1629230463 learned_addr learned my addr 172.21.15.97:0/1629230463 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:39.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.826+0000 7fdf9be00640 1 --2- 172.21.15.97:0/1629230463 >> v2:172.21.15.135:3300/0 conn(0x7fdf9c06db80 0x7fdf9c0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.826+0000 7fdfa1c00640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 0x7fdf9c0ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.826+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694355636 0 0) 0x7fdf9c15b9e0 con 0x7fdf9c1581d0 2024-03-06T07:37:39.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.826+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf84003520 con 0x7fdf9c1581d0 2024-03-06T07:37:39.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28791812 0 0) 0x7fdf84003520 con 0x7fdf9c1581d0 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 msgr2=0x7fdf9c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 0x7fdf9c0ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 >> v2:172.21.15.135:3300/0 conn(0x7fdf9c06db80 msgr2=0x7fdf9c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 --2- 172.21.15.97:0/1629230463 >> v2:172.21.15.135:3300/0 conn(0x7fdf9c06db80 0x7fdf9c0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c15ac20 con 0x7fdf9c1581d0 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fdf90001670 con 0x7fdf9c1581d0 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 >> v1:172.21.15.97:6789/0 conn(0x7fdf9c1581d0 legacy=0x7fdf9c06d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf840039b0 0x7fdf84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf84004800 0x7fdf84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 0x7fdf840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cc220 con 0x7fdf840039b0 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8400b9c0 con 0x7fdf84004800 2024-03-06T07:37:39.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8400bb40 con 0x7fdf9c06e470 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdfa1200640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf840039b0 0x7fdf84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdfa1200640 1 -- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 msgr2=0x7fdf840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdfa1200640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c06e470 0x7fdf840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdf9be00640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf84004800 0x7fdf84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.827+0000 7fdfa1200640 1 -- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf84004800 msgr2=0x7fdf84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.828+0000 7fdfa1200640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf84004800 0x7fdf84004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.828+0000 7fdfa1200640 1 -- 172.21.15.97:0/1629230463 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84003520 con 0x7fdf840039b0 2024-03-06T07:37:39.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdfa1200640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf840039b0 0x7fdf84003dd0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c190 tx=0x7fdf84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e81f76b732e3a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:39.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90007c60 con 0x7fdf840039b0 2024-03-06T07:37:39.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf90030070 con 0x7fdf840039b0 2024-03-06T07:37:39.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdf9b400640 1 -- 172.21.15.97:0/1629230463 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90005d30 con 0x7fdf840039b0 2024-03-06T07:37:39.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdfa3c00640 1 -- 172.21.15.97:0/1629230463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf840039b0 msgr2=0x7fdf84003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/1629230463 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf840039b0 0x7fdf84003dd0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c190 tx=0x7fdf84005830 comp rx=0 tx=0).stop 2024-03-06T07:37:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.829+0000 7fdfa1c00640 1 -- 172.21.15.97:0/1629230463 reap_dead start 2024-03-06T07:37:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.830+0000 7fdfa3c00640 1 -- 172.21.15.97:0/1629230463 shutdown_connections 2024-03-06T07:37:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.830+0000 7fdfa3c00640 1 -- 172.21.15.97:0/1629230463 >> 172.21.15.97:0/1629230463 conn(0x7fdf9c0cc4f0 msgr2=0x7fdf9c0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.830+0000 7fdfa3c00640 1 -- 172.21.15.97:0/1629230463 shutdown_connections 2024-03-06T07:37:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.830+0000 7fdfa3c00640 1 -- 172.21.15.97:0/1629230463 wait complete. 2024-03-06T07:37:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.831+0000 7fdfa3c00640 1 Processor -- start 2024-03-06T07:37:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.832+0000 7fdfa3c00640 1 -- start start 2024-03-06T07:37:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.832+0000 7fdfa3c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf840039b0 0x7fdf9c0c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.832+0000 7fdfa3c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c1480d0 0x7fdf9c150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.832+0000 7fdfa3c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 0x7fdf9c153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa3c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c15b4c0 con 0x7fdf9c153730 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa3c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c15b1e0 con 0x7fdf840039b0 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa3c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c06cbe0 con 0x7fdf9c1480d0 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 0x7fdf9c153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 0x7fdf9c153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51326/0 (socket says 172.21.15.97:51326) 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 -- 172.21.15.97:0/2988108023 learned_addr learned my addr 172.21.15.97:0/2988108023 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdf9be00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c1480d0 0x7fdf9c150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1200640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf840039b0 0x7fdf9c0c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 -- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c1480d0 msgr2=0x7fdf9c150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c1480d0 0x7fdf9c150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 -- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf840039b0 msgr2=0x7fdf9c0c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf840039b0 0x7fdf9c0c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.833+0000 7fdfa1c00640 1 -- 172.21.15.97:0/2988108023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c14bbe0 con 0x7fdf9c153730 2024-03-06T07:37:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.834+0000 7fdfa1c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 0x7fdf9c153bb0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c06ec10 tx=0x7fdf9400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.834+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94018020 con 0x7fdf9c153730 2024-03-06T07:37:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.834+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9400bb70 con 0x7fdf9c153730 2024-03-06T07:37:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.834+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c071590 con 0x7fdf9c153730 2024-03-06T07:37:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.834+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94014400 con 0x7fdf9c153730 2024-03-06T07:37:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.834+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c0c7090 con 0x7fdf9c153730 2024-03-06T07:37:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.836+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c04f330 con 0x7fdf9c153730 2024-03-06T07:37:39.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.837+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf94005d10 con 0x7fdf9c153730 2024-03-06T07:37:39.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.838+0000 7fdf98c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf7c06b500 0x7fdf7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.838+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdf9408a8b0 con 0x7fdf9c153730 2024-03-06T07:37:39.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.838+0000 7fdfa1200640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf7c06b500 0x7fdf7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.839+0000 7fdfa1200640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf7c06b500 0x7fdf7c06d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdf90034000 tx=0x7fdf90007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:39.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:39.843+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf94022030 con 0x7fdf9c153730 2024-03-06T07:37:40.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.114+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fdf9c0cd1c0 con 0x7fdf9c153730 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.563+0000 7fdf98c00640 1 -- 172.21.15.97:0/2988108023 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5840 v112) v1 ==== 95+0+0 (secure 0 0 0) 0x7fdf94059700 con 0x7fdf9c153730 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:failed mds gid 5840 2024-03-06T07:37:40.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf7c06b500 msgr2=0x7fdf7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf7c06b500 0x7fdf7c06d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdf90034000 tx=0x7fdf90007000 comp rx=0 tx=0).stop 2024-03-06T07:37:40.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 msgr2=0x7fdf9c153bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 0x7fdf9c153bb0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c06ec10 tx=0x7fdf9400ef30 comp rx=0 tx=0).stop 2024-03-06T07:37:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 shutdown_connections 2024-03-06T07:37:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdf7c06b500 0x7fdf7c06d9c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf9c153730 0x7fdf9c153bb0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdf9c1480d0 0x7fdf9c150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 --2- 172.21.15.97:0/2988108023 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf840039b0 0x7fdf9c0c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.566+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 >> 172.21.15.97:0/2988108023 conn(0x7fdf9c0cc4f0 msgr2=0x7fdf9c158d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:40.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.567+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 shutdown_connections 2024-03-06T07:37:40.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.567+0000 7fdfa3c00640 1 -- 172.21.15.97:0/2988108023 wait complete. 2024-03-06T07:37:40.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:40.590 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:37:40.590 DEBUG:teuthology.orchestra.run.smithi097:> mkdir dir_1 dir_2 2024-03-06T07:37:40.590 DEBUG:teuthology.orchestra.run.smithi097:> touch dir_1/original 2024-03-06T07:37:40.591 DEBUG:teuthology.orchestra.run.smithi097:> ln dir_1/original dir_2/linkto 2024-03-06T07:37:40.591 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:37:40.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:40.660 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 dir_1) 2024-03-06T07:37:40.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:40.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.946+0000 7fea4a600640 1 Processor -- start 2024-03-06T07:37:40.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.946+0000 7fea4a600640 1 -- start start 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.947+0000 7fea4a600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea440d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.947+0000 7fea4a600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440cea60 con 0x7fea440d0df0 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.947+0000 7fea4a600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440d2350 con 0x7fea4415e0d0 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.947+0000 7fea4a600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea440d1260 con 0x7fea4415f470 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.948+0000 7fea43e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fea440d0df0 0x7fea44160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58938/0 (socket says 172.21.15.97:58938) 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.948+0000 7fea43e00640 1 -- 172.21.15.97:0/2731422588 learned_addr learned my addr 172.21.15.97:0/2731422588 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:40.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.948+0000 7fea3be00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea440d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.948+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332708896 0 0) 0x7fea440cea60 con 0x7fea440d0df0 2024-03-06T07:37:40.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.948+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea28003520 con 0x7fea440d0df0 2024-03-06T07:37:40.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421312527 0 0) 0x7fea28003520 con 0x7fea440d0df0 2024-03-06T07:37:40.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 msgr2=0x7fea440d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea440d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 >> v1:172.21.15.135:6789/0 conn(0x7fea4415e0d0 legacy=0x7fea440d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440d2630 con 0x7fea440d0df0 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fea3c0019f0 con 0x7fea440d0df0 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 >> v1:172.21.15.97:6789/0 conn(0x7fea440d0df0 legacy=0x7fea44160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea280039e0 0x7fea28003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440d1260 con 0x7fea28004800 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2800bb00 con 0x7fea280039e0 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2800bd00 con 0x7fea4415f470 2024-03-06T07:37:40.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea3be00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.949+0000 7fea43400640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea280039e0 0x7fea28003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea43e00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea3be00640 1 -- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 msgr2=0x7fea280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea3be00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea3be00640 1 -- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea280039e0 msgr2=0x7fea28003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea3be00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea280039e0 0x7fea28003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea3be00640 1 -- 172.21.15.97:0/2731422588 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28003520 con 0x7fea28004800 2024-03-06T07:37:40.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.950+0000 7fea43e00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea4415f470 0x7fea280051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:40.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.951+0000 7fea3be00640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea28004c80 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009a90 tx=0x7fea28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae68df021493165 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.951+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c039070 con 0x7fea28004800 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.951+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c02c660 con 0x7fea28004800 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.951+0000 7fea42a00640 1 -- 172.21.15.97:0/2731422588 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c009690 con 0x7fea28004800 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea4a600640 1 -- 172.21.15.97:0/2731422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 msgr2=0x7fea28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea4a600640 1 --2- 172.21.15.97:0/2731422588 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea28004c80 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009a90 tx=0x7fea28005830 comp rx=0 tx=0).stop 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea43e00640 1 -- 172.21.15.97:0/2731422588 reap_dead start 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea4a600640 1 -- 172.21.15.97:0/2731422588 shutdown_connections 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea4a600640 1 -- 172.21.15.97:0/2731422588 >> 172.21.15.97:0/2731422588 conn(0x7fea440d3ec0 msgr2=0x7fea440d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea4a600640 1 -- 172.21.15.97:0/2731422588 shutdown_connections 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.952+0000 7fea4a600640 1 -- 172.21.15.97:0/2731422588 wait complete. 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.953+0000 7fea4a600640 1 Processor -- start 2024-03-06T07:37:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.954+0000 7fea4a600640 1 -- start start 2024-03-06T07:37:40.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.954+0000 7fea4a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea440cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.954+0000 7fea4a600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440cb910 0x7fea44155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea4a600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440ca280 0x7fea44155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea4a600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440d2bf0 con 0x7fea28004800 2024-03-06T07:37:40.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea4a600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440cff10 con 0x7fea440cb910 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea4a600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4415fb30 con 0x7fea440ca280 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea440cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea440cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51344/0 (socket says 172.21.15.97:51344) 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 -- 172.21.15.97:0/2936242039 learned_addr learned my addr 172.21.15.97:0/2936242039 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 -- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440ca280 msgr2=0x7fea44155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43e00640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440ca280 0x7fea44155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea3be00640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440cb910 0x7fea44155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440ca280 0x7fea44155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 -- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440cb910 msgr2=0x7fea44155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440cb910 0x7fea44155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea43400640 1 -- 172.21.15.97:0/2936242039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440c9500 con 0x7fea28004800 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.955+0000 7fea3be00640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440cb910 0x7fea44155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.956+0000 7fea43400640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea440cd770 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fea3400e580 tx=0x7fea3400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.956+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea340052e0 con 0x7fea28004800 2024-03-06T07:37:40.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.956+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34018070 con 0x7fea28004800 2024-03-06T07:37:40.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.956+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea440c97a0 con 0x7fea28004800 2024-03-06T07:37:40.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.956+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34003810 con 0x7fea28004800 2024-03-06T07:37:40.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.956+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea440c9d60 con 0x7fea28004800 2024-03-06T07:37:40.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.957+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4404f330 con 0x7fea28004800 2024-03-06T07:37:40.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.959+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea340039b0 con 0x7fea28004800 2024-03-06T07:37:40.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.961+0000 7fea3b400640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2006b500 0x7fea2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.961+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fea34089e90 con 0x7fea28004800 2024-03-06T07:37:40.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.961+0000 7fea3be00640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2006b500 0x7fea2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.962+0000 7fea3be00640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2006b500 0x7fea2006d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fea2c036000 tx=0x7fea2c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:40.969+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea34058c60 con 0x7fea28004800 2024-03-06T07:37:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.238+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea4408d920 con 0x7fea28004800 2024-03-06T07:37:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.238+0000 7fea3b400640 1 -- 172.21.15.97:0/2936242039 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fea3408bd70 con 0x7fea28004800 2024-03-06T07:37:41.241 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:41.241 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":112,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:32.462565+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[6463,9024]},"id":10}]} 2024-03-06T07:37:41.241 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 112 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2006b500 msgr2=0x7fea2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2006b500 0x7fea2006d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fea2c036000 tx=0x7fea2c030000 comp rx=0 tx=0).stop 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 msgr2=0x7fea440cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea440cd770 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fea3400e580 tx=0x7fea3400ea40 comp rx=0 tx=0).stop 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 shutdown_connections 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fea2006b500 0x7fea2006d9c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fea440ca280 0x7fea44155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea440cb910 0x7fea44155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 --2- 172.21.15.97:0/2936242039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea28004800 0x7fea440cd770 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 >> 172.21.15.97:0/2936242039 conn(0x7fea440d3ec0 msgr2=0x7fea440b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 shutdown_connections 2024-03-06T07:37:41.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:41.241+0000 7fea4a600640 1 -- 172.21.15.97:0/2936242039 wait complete. 2024-03-06T07:37:41.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:37:41.470 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:37:41.470 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:41.470 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:37:43.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:43.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.671+0000 7ff693200640 1 Processor -- start 2024-03-06T07:37:43.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.671+0000 7ff693200640 1 -- start start 2024-03-06T07:37:43.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff693200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff693200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff68c0b4680 0x7ff68c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff693200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff68c072340 0x7ff68c06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff693200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c15b090 con 0x7ff68c072340 2024-03-06T07:37:43.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff693200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c0b2a30 con 0x7ff68c0b4680 2024-03-06T07:37:43.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff693200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0b4360 con 0x7ff68c15acb0 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff691200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff68c072340 0x7ff68c06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff691200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff68c072340 0x7ff68c06ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51350/0 (socket says 172.21.15.97:51350) 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff691200640 1 -- 172.21.15.97:0/2334222369 learned_addr learned my addr 172.21.15.97:0/2334222369 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.673+0000 7ff68b400640 1 --2- 172.21.15.97:0/2334222369 >> v2:172.21.15.135:3300/0 conn(0x7ff68c0b4680 0x7ff68c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff68be00640 1 --2- 172.21.15.97:0/2334222369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff691200640 1 -- 172.21.15.97:0/2334222369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 msgr2=0x7ff68c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff691200640 1 --2- 172.21.15.97:0/2334222369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff691200640 1 -- 172.21.15.97:0/2334222369 >> v2:172.21.15.135:3300/0 conn(0x7ff68c0b4680 msgr2=0x7ff68c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff691200640 1 --2- 172.21.15.97:0/2334222369 >> v2:172.21.15.135:3300/0 conn(0x7ff68c0b4680 0x7ff68c06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff691200640 1 -- 172.21.15.97:0/2334222369 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c0b2cf0 con 0x7ff68c072340 2024-03-06T07:37:43.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.674+0000 7ff68b400640 1 --2- 172.21.15.97:0/2334222369 >> v2:172.21.15.135:3300/0 conn(0x7ff68c0b4680 0x7ff68c06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.675+0000 7ff691200640 1 --2- 172.21.15.97:0/2334222369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c072340 0x7ff68c06ea30 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7ff68c06f020 tx=0x7ff68402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de7f082551ead8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.675+0000 7ff68aa00640 1 -- 172.21.15.97:0/2334222369 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68402ca80 con 0x7ff68c072340 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.675+0000 7ff68aa00640 1 -- 172.21.15.97:0/2334222369 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68402cc20 con 0x7ff68c072340 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.675+0000 7ff68aa00640 1 -- 172.21.15.97:0/2334222369 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6840343f0 con 0x7ff68c072340 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 -- 172.21.15.97:0/2334222369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c072340 msgr2=0x7ff68c06ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 --2- 172.21.15.97:0/2334222369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c072340 0x7ff68c06ea30 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7ff68c06f020 tx=0x7ff68402bee0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 -- 172.21.15.97:0/2334222369 shutdown_connections 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 --2- 172.21.15.97:0/2334222369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c072340 0x7ff68c06ea30 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 --2- 172.21.15.97:0/2334222369 >> v2:172.21.15.135:3300/0 conn(0x7ff68c0b4680 0x7ff68c06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 --2- 172.21.15.97:0/2334222369 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 -- 172.21.15.97:0/2334222369 >> 172.21.15.97:0/2334222369 conn(0x7ff68c0b1730 msgr2=0x7ff68c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:43.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 -- 172.21.15.97:0/2334222369 shutdown_connections 2024-03-06T07:37:43.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.676+0000 7ff693200640 1 -- 172.21.15.97:0/2334222369 wait complete. 2024-03-06T07:37:43.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.679+0000 7ff693200640 1 Processor -- start 2024-03-06T07:37:43.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.679+0000 7ff693200640 1 -- start start 2024-03-06T07:37:43.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff693200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff68c072340 0x7ff68c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff693200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 0x7ff68c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff693200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff693200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0b3d10 con 0x7ff68c0b4680 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff693200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0b32b0 con 0x7ff68c072340 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff693200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0b4010 con 0x7ff68c15acb0 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff68b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 0x7ff68c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff68b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 0x7ff68c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51352/0 (socket says 172.21.15.97:51352) 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff68b400640 1 -- 172.21.15.97:0/2246934713 learned_addr learned my addr 172.21.15.97:0/2246934713 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff68be00640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff68c072340 0x7ff68c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.681+0000 7ff691200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff68b400640 1 -- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 msgr2=0x7ff68c0ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff68b400640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff68b400640 1 -- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff68c072340 msgr2=0x7ff68c0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff68b400640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff68c072340 0x7ff68c0ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff68b400640 1 -- 172.21.15.97:0/2246934713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c14bab0 con 0x7ff68c0b4680 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff68b400640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 0x7ff68c0ae3f0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7ff67800d800 tx=0x7ff67800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678018070 con 0x7ff68c0b4680 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff678004030 con 0x7ff68c0b4680 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c14a730 con 0x7ff68c0b4680 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678004400 con 0x7ff68c0b4680 2024-03-06T07:37:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.682+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c14acf0 con 0x7ff68c0b4680 2024-03-06T07:37:43.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.684+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c04f330 con 0x7ff68c0b4680 2024-03-06T07:37:43.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.685+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6780096a0 con 0x7ff68c0b4680 2024-03-06T07:37:43.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.686+0000 7ff683e00640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff66c06b4d0 0x7ff66c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.686+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff67808a620 con 0x7ff68c0b4680 2024-03-06T07:37:43.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.686+0000 7ff68be00640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff66c06b4d0 0x7ff66c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.687+0000 7ff68be00640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff66c06b4d0 0x7ff66c06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff67c006fd0 tx=0x7ff67c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.690+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff68c14bab0 con 0x7ff68c0b4680 2024-03-06T07:37:43.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.923+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff68c06ce60 con 0x7ff68c0b4680 2024-03-06T07:37:43.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.924+0000 7ff683e00640 1 -- 172.21.15.97:0/2246934713 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+3423 (secure 0 0 0) 0x7ff6780593e0 con 0x7ff68c0b4680 2024-03-06T07:37:43.926 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:43.926 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":112,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:32.462565+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[6463,9024]},"id":10}]} 2024-03-06T07:37:43.927 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 112 2024-03-06T07:37:43.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.926+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff66c06b4d0 msgr2=0x7ff66c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.926+0000 7ff693200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff66c06b4d0 0x7ff66c06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff67c006fd0 tx=0x7ff67c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:43.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.926+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 msgr2=0x7ff68c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.926+0000 7ff693200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 0x7ff68c0ae3f0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7ff67800d800 tx=0x7ff67800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 shutdown_connections 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff66c06b4d0 0x7ff66c06d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff68c15acb0 0x7ff68c0ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff68c0b4680 0x7ff68c0ae3f0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 --2- 172.21.15.97:0/2246934713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff68c072340 0x7ff68c0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 >> 172.21.15.97:0/2246934713 conn(0x7ff68c0b1730 msgr2=0x7ff68c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 shutdown_connections 2024-03-06T07:37:43.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:43.927+0000 7ff693200640 1 -- 172.21.15.97:0/2246934713 wait complete. 2024-03-06T07:37:43.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:37:44.165 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:37:44.166 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:44.166 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:37:46.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:46.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.367+0000 7f20a9400640 1 Processor -- start 2024-03-06T07:37:46.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.367+0000 7f20a9400640 1 -- start start 2024-03-06T07:37:46.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a9400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f20a4144c00 0x7f20a406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a9400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a406ee70 con 0x7f20a415fe60 2024-03-06T07:37:46.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a9400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a406cef0 con 0x7f20a415f9d0 2024-03-06T07:37:46.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a9400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a406d1d0 con 0x7f20a4144c00 2024-03-06T07:37:46.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a3200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f20a415fe60 0x7f20a40ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45628/0 (socket says 172.21.15.97:45628) 2024-03-06T07:37:46.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a3200640 1 -- 172.21.15.97:0/2649086004 learned_addr learned my addr 172.21.15.97:0/2649086004 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:46.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a2800640 1 --2- 172.21.15.97:0/2649086004 >> v2:172.21.15.135:3300/0 conn(0x7f20a4144c00 0x7f20a406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.369+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743344483 0 0) 0x7f209c001490 con 0x7f20a415fe60 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2088003520 con 0x7f20a415fe60 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3195338616 0 0) 0x7f2088003520 con 0x7f20a415fe60 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 >> v2:172.21.15.135:3300/0 conn(0x7f20a4144c00 msgr2=0x7f20a406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 --2- 172.21.15.97:0/2649086004 >> v2:172.21.15.135:3300/0 conn(0x7f20a4144c00 0x7f20a406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 >> v1:172.21.15.135:6789/0 conn(0x7f20a415f9d0 legacy=0x7f20a40ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a406d4b0 con 0x7f20a415fe60 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f209c001bc0 con 0x7f20a415fe60 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 >> v1:172.21.15.97:6789/0 conn(0x7f20a415fe60 legacy=0x7f20a40ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2088003a00 0x7f2088003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2088004800 0x7f2088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20880051c0 0x7f208800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a406d1d0 con 0x7f20880051c0 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208800c090 con 0x7f2088004800 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208800c290 con 0x7f2088003a00 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.370+0000 7f20a3200640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20880051c0 0x7f208800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a2800640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2088003a00 0x7f2088003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a1e00640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2088004800 0x7f2088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a3200640 1 -- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2088003a00 msgr2=0x7f2088003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a3200640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2088003a00 0x7f2088003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a3200640 1 -- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2088004800 msgr2=0x7f2088004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a3200640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2088004800 0x7f2088004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.371+0000 7f20a3200640 1 -- 172.21.15.97:0/2649086004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088003520 con 0x7f20880051c0 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.372+0000 7f20a3200640 1 -- 172.21.15.97:0/2649086004 reap_dead start 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.372+0000 7f20a3200640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20880051c0 0x7f208800b9c0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f209c00a9a0 tx=0x7f2088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b5d3178f9769a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.372+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00a5c0 con 0x7f20880051c0 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.372+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c034670 con 0x7f20880051c0 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.372+0000 7f20a1400640 1 -- 172.21.15.97:0/2649086004 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c034a60 con 0x7f20880051c0 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 -- 172.21.15.97:0/2649086004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20880051c0 msgr2=0x7f208800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20880051c0 0x7f208800b9c0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f209c00a9a0 tx=0x7f2088005830 comp rx=0 tx=0).stop 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 -- 172.21.15.97:0/2649086004 shutdown_connections 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 --2- 172.21.15.97:0/2649086004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20880051c0 0x7f208800b9c0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 -- 172.21.15.97:0/2649086004 >> 172.21.15.97:0/2649086004 conn(0x7f20a40b1730 msgr2=0x7f20a40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:46.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 -- 172.21.15.97:0/2649086004 shutdown_connections 2024-03-06T07:37:46.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.373+0000 7f20a9400640 1 -- 172.21.15.97:0/2649086004 wait complete. 2024-03-06T07:37:46.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.376+0000 7f20a9400640 1 Processor -- start 2024-03-06T07:37:46.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.376+0000 7f20a9400640 1 -- start start 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a9400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20880051c0 0x7f20a40aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 0x7f20a40ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a9400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20a415b240 0x7f20a415b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a9400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4160730 con 0x7f20a4132fe0 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a9400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4160270 con 0x7f20880051c0 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a9400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a406cb50 con 0x7f20a415b240 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 0x7f20a40ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 0x7f20a40ad030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40800/0 (socket says 172.21.15.97:40800) 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 -- 172.21.15.97:0/2712517821 learned_addr learned my addr 172.21.15.97:0/2712517821 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a3200640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20a415b240 0x7f20a415b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a2800640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20880051c0 0x7f20a40aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 -- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20a415b240 msgr2=0x7f20a415b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20a415b240 0x7f20a415b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 -- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20880051c0 msgr2=0x7f20a40aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20880051c0 0x7f20a40aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.378+0000 7f20a1e00640 1 -- 172.21.15.97:0/2712517821 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4135ed0 con 0x7f20a4132fe0 2024-03-06T07:37:46.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.379+0000 7f20a1e00640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 0x7f20a40ad030 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f209000e460 tx=0x7f209000e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.379+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900135f0 con 0x7f20a4132fe0 2024-03-06T07:37:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.379+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209001a070 con 0x7f20a4132fe0 2024-03-06T07:37:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.379+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900139d0 con 0x7f20a4132fe0 2024-03-06T07:37:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.379+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a4133580 con 0x7f20a4132fe0 2024-03-06T07:37:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.379+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a4133b40 con 0x7f20a4132fe0 2024-03-06T07:37:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.381+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a404f330 con 0x7f20a4132fe0 2024-03-06T07:37:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.382+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2090013b70 con 0x7f20a4132fe0 2024-03-06T07:37:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.383+0000 7f209aa00640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f208406b4d0 0x7f208406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.383+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2090089ec0 con 0x7f20a4132fe0 2024-03-06T07:37:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.384+0000 7f20a2800640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f208406b4d0 0x7f208406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.384+0000 7f20a2800640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f208406b4d0 0x7f208406d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f20a40afbb0 tx=0x7f2094008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:46.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.391+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2090023020 con 0x7f20a4132fe0 2024-03-06T07:37:46.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.697+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a4072890 con 0x7f20a4132fe0 2024-03-06T07:37:46.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.698+0000 7f209aa00640 1 -- 172.21.15.97:0/2712517821 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f20900586c0 con 0x7f20a4132fe0 2024-03-06T07:37:46.700 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:46.700 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":113,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:45.917742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024,6463]},"id":10}]} 2024-03-06T07:37:46.700 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 113 2024-03-06T07:37:46.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f208406b4d0 msgr2=0x7f208406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f208406b4d0 0x7f208406d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f20a40afbb0 tx=0x7f2094008040 comp rx=0 tx=0).stop 2024-03-06T07:37:46.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 msgr2=0x7f20a40ad030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 0x7f20a40ad030 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f209000e460 tx=0x7f209000e920 comp rx=0 tx=0).stop 2024-03-06T07:37:46.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 shutdown_connections 2024-03-06T07:37:46.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f208406b4d0 0x7f208406d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20a415b240 0x7f20a415b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20a4132fe0 0x7f20a40ad030 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 --2- 172.21.15.97:0/2712517821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20880051c0 0x7f20a40aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 >> 172.21.15.97:0/2712517821 conn(0x7f20a40b1730 msgr2=0x7f20a40cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:46.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 shutdown_connections 2024-03-06T07:37:46.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:46.700+0000 7f20a9400640 1 -- 172.21.15.97:0/2712517821 wait complete. 2024-03-06T07:37:46.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:37:46.974 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:37:46.975 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:46.975 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2024-03-06T07:37:46.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:46.975 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 dir_2) 2024-03-06T07:37:47.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:47.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.252+0000 7f003e400640 1 Processor -- start 2024-03-06T07:37:47.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.252+0000 7f003e400640 1 -- start start 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f003e400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f00380d2360 0x7f00380d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f003e400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0038161960 con 0x7f0038161580 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f003e400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00380cf6f0 con 0x7f00380b3ec0 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f003e400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00380d2ce0 con 0x7f00380d2360 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f003d000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0038161580 0x7f00380d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45644/0 (socket says 172.21.15.97:45644) 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f003d000640 1 -- 172.21.15.97:0/1684064713 learned_addr learned my addr 172.21.15.97:0/1684064713 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:47.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f0037e00640 1 --2- 172.21.15.97:0/1684064713 >> v2:172.21.15.135:3300/0 conn(0x7f00380d2360 0x7f00380d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227485906 0 0) 0x7f0038161960 con 0x7f0038161580 2024-03-06T07:37:47.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.254+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020003520 con 0x7f0038161580 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2656533932 0 0) 0x7f0020003520 con 0x7f0038161580 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 >> v2:172.21.15.135:3300/0 conn(0x7f00380d2360 msgr2=0x7f00380d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 --2- 172.21.15.97:0/1684064713 >> v2:172.21.15.135:3300/0 conn(0x7f00380d2360 0x7f00380d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 >> v1:172.21.15.135:6789/0 conn(0x7f00380b3ec0 legacy=0x7f00380b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00380cf9d0 con 0x7f0038161580 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f002c001670 con 0x7f0038161580 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 >> v1:172.21.15.97:6789/0 conn(0x7f0038161580 legacy=0x7f00380d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f0020003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0020004800 0x7f0020004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00200051c0 0x7f002000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00380d2ce0 con 0x7f0020004800 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002000c060 con 0x7f00200051c0 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002000c260 con 0x7f00200039b0 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.255+0000 7f0037e00640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0020004800 0x7f0020004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f003d000640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f0020003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f003da00640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00200051c0 0x7f002000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f0037e00640 1 -- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 msgr2=0x7f0020003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f0037e00640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f0020003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f0037e00640 1 -- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00200051c0 msgr2=0x7f002000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f0037e00640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00200051c0 0x7f002000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f0037e00640 1 -- 172.21.15.97:0/1684064713 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020003520 con 0x7f0020004800 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f003d000640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00200039b0 0x7f0020003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f003da00640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00200051c0 0x7f002000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:47.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.256+0000 7f003da00640 1 -- 172.21.15.97:0/1684064713 reap_dead start 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.257+0000 7f0037e00640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0020004800 0x7f0020004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f0024009ac0 tx=0x7f0020005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7493ba5f4eeb2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.257+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024039070 con 0x7f0020004800 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.257+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002402c6b0 con 0x7f0020004800 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.257+0000 7f0037400640 1 -- 172.21.15.97:0/1684064713 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024009690 con 0x7f0020004800 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 -- 172.21.15.97:0/1684064713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0020004800 msgr2=0x7f0020004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0020004800 0x7f0020004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f0024009ac0 tx=0x7f0020005830 comp rx=0 tx=0).stop 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 -- 172.21.15.97:0/1684064713 shutdown_connections 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 --2- 172.21.15.97:0/1684064713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0020004800 0x7f0020004c80 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 -- 172.21.15.97:0/1684064713 >> 172.21.15.97:0/1684064713 conn(0x7f00380d3d80 msgr2=0x7f00380d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 -- 172.21.15.97:0/1684064713 shutdown_connections 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.258+0000 7f003e400640 1 -- 172.21.15.97:0/1684064713 wait complete. 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.260+0000 7f003e400640 1 Processor -- start 2024-03-06T07:37:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.260+0000 7f003e400640 1 -- start start 2024-03-06T07:37:47.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f00380ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 0x7f00380cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00381e9a00 0x7f00381e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00380b39a0 con 0x7f003815b070 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00380b36c0 con 0x7f0020004800 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00380cf3a0 con 0x7f00381e9a00 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f0037e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 0x7f00380cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f0037e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 0x7f00380cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40822/0 (socket says 172.21.15.97:40822) 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f0037e00640 1 -- 172.21.15.97:0/1663041215 learned_addr learned my addr 172.21.15.97:0/1663041215 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003d000640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f00380ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f003da00640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00381e9a00 0x7f00381e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f0037e00640 1 -- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00381e9a00 msgr2=0x7f00381e9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.261+0000 7f0037e00640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00381e9a00 0x7f00381e9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0037e00640 1 -- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 msgr2=0x7f00380ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0037e00640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f00380ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0037e00640 1 -- 172.21.15.97:0/1663041215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00380cd4a0 con 0x7f003815b070 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0037e00640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 0x7f00380cb460 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f0024009ac0 tx=0x7f0024004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024039050 con 0x7f003815b070 2024-03-06T07:37:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024045070 con 0x7f003815b070 2024-03-06T07:37:47.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240044e0 con 0x7f003815b070 2024-03-06T07:37:47.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00380cd760 con 0x7f003815b070 2024-03-06T07:37:47.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.262+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00380cdd20 con 0x7f003815b070 2024-03-06T07:37:47.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.264+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00381ea3c0 con 0x7f003815b070 2024-03-06T07:37:47.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.264+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0024003a50 con 0x7f003815b070 2024-03-06T07:37:47.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.265+0000 7f0034c00640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b500 0x7f001806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.265+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f00240abd30 con 0x7f003815b070 2024-03-06T07:37:47.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.265+0000 7f003d000640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b500 0x7f001806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.266+0000 7f003d000640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b500 0x7f001806d9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0038069720 tx=0x7f002c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.270+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f002404e020 con 0x7f003815b070 2024-03-06T07:37:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.505+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f003804f330 con 0x7f003815b070 2024-03-06T07:37:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.506+0000 7f0034c00640 1 -- 172.21.15.97:0/1663041215 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f002407ac20 con 0x7f003815b070 2024-03-06T07:37:47.508 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:47.508 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":113,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:45.917742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024,6463]},"id":10}]} 2024-03-06T07:37:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 113 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b500 msgr2=0x7f001806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b500 0x7f001806d9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0038069720 tx=0x7f002c009040 comp rx=0 tx=0).stop 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 msgr2=0x7f00380cb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 0x7f00380cb460 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f0024009ac0 tx=0x7f0024004730 comp rx=0 tx=0).stop 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 shutdown_connections 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f001806b500 0x7f001806d9c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00381e9a00 0x7f00381e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f003815b070 0x7f00380cb460 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 --2- 172.21.15.97:0/1663041215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0020004800 0x7f00380ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 >> 172.21.15.97:0/1663041215 conn(0x7f00380d3d80 msgr2=0x7f003806c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:47.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.508+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 shutdown_connections 2024-03-06T07:37:47.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:47.509+0000 7f003e400640 1 -- 172.21.15.97:0/1663041215 wait complete. 2024-03-06T07:37:47.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:37:47.735 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"14","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:37:47.736 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:37:47.736 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:37:49.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.946+0000 7fd01e600640 1 Processor -- start 2024-03-06T07:37:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.946+0000 7fd01e600640 1 -- start start 2024-03-06T07:37:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd01e600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd01815f9d0 0x7fd01815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd01e600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd018070b10 0x7fd018070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd01e600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01806cef0 con 0x7fd018144c00 2024-03-06T07:37:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd01e600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01806d1d0 con 0x7fd018070b10 2024-03-06T07:37:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd01e600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01806d4b0 con 0x7fd01815f9d0 2024-03-06T07:37:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd017e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd018070b10 0x7fd018070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd017e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd018070b10 0x7fd018070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40826/0 (socket says 172.21.15.97:40826) 2024-03-06T07:37:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd017e00640 1 -- 172.21.15.97:0/3034574131 learned_addr learned my addr 172.21.15.97:0/3034574131 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd016a00640 1 --2- 172.21.15.97:0/3034574131 >> v2:172.21.15.135:3300/0 conn(0x7fd01815f9d0 0x7fd01815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd017e00640 1 -- 172.21.15.97:0/3034574131 >> v2:172.21.15.135:3300/0 conn(0x7fd01815f9d0 msgr2=0x7fd01815fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.948+0000 7fd017e00640 1 --2- 172.21.15.97:0/3034574131 >> v2:172.21.15.135:3300/0 conn(0x7fd01815f9d0 0x7fd01815fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.949+0000 7fd017e00640 1 -- 172.21.15.97:0/3034574131 >> v1:172.21.15.135:6789/0 conn(0x7fd018144c00 legacy=0x7fd01806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.949+0000 7fd017e00640 1 -- 172.21.15.97:0/3034574131 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01806d790 con 0x7fd018070b10 2024-03-06T07:37:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.950+0000 7fd017e00640 1 --2- 172.21.15.97:0/3034574131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018070b10 0x7fd018070f20 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fd0180b26f0 tx=0x7fd01002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8ac90664fd205e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.950+0000 7fd016000640 1 -- 172.21.15.97:0/3034574131 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01002ca80 con 0x7fd018070b10 2024-03-06T07:37:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.950+0000 7fd016000640 1 -- 172.21.15.97:0/3034574131 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01002cc20 con 0x7fd018070b10 2024-03-06T07:37:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.950+0000 7fd016000640 1 -- 172.21.15.97:0/3034574131 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0100343f0 con 0x7fd018070b10 2024-03-06T07:37:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 -- 172.21.15.97:0/3034574131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018070b10 msgr2=0x7fd018070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 --2- 172.21.15.97:0/3034574131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018070b10 0x7fd018070f20 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fd0180b26f0 tx=0x7fd01002bee0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 -- 172.21.15.97:0/3034574131 shutdown_connections 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 --2- 172.21.15.97:0/3034574131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018070b10 0x7fd018070f20 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 --2- 172.21.15.97:0/3034574131 >> v2:172.21.15.135:3300/0 conn(0x7fd01815f9d0 0x7fd01815fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 -- 172.21.15.97:0/3034574131 >> 172.21.15.97:0/3034574131 conn(0x7fd0180b1730 msgr2=0x7fd0180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 -- 172.21.15.97:0/3034574131 shutdown_connections 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.951+0000 7fd01e600640 1 -- 172.21.15.97:0/3034574131 wait complete. 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.952+0000 7fd01e600640 1 Processor -- start 2024-03-06T07:37:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.953+0000 7fd01e600640 1 -- start start 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.953+0000 7fd01e600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd018070b10 0x7fd018139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.953+0000 7fd01e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 0x7fd01815d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd01e600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd01815f9d0 0x7fd01815de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd01e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01806ccd0 con 0x7fd018144c00 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd01e600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01806cb50 con 0x7fd01815f9d0 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd01e600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01806df90 con 0x7fd018070b10 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 0x7fd01815d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 0x7fd01815d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40838/0 (socket says 172.21.15.97:40838) 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 -- 172.21.15.97:0/2054168357 learned_addr learned my addr 172.21.15.97:0/2054168357 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 -- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd018070b10 msgr2=0x7fd018139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd018070b10 0x7fd018139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd017e00640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd01815f9d0 0x7fd01815de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 -- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd01815f9d0 msgr2=0x7fd01815de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd01815f9d0 0x7fd01815de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.954+0000 7fd016a00640 1 -- 172.21.15.97:0/2054168357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01814d320 con 0x7fd018144c00 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.955+0000 7fd016a00640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 0x7fd01815d930 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fd00400d5f0 tx=0x7fd00400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.955+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040125f0 con 0x7fd018144c00 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.955+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd004019070 con 0x7fd018144c00 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.955+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040129d0 con 0x7fd018144c00 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.955+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01814d600 con 0x7fd018144c00 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.955+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01814dbc0 con 0x7fd018144c00 2024-03-06T07:37:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.957+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01804f330 con 0x7fd018144c00 2024-03-06T07:37:49.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.957+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd004012b70 con 0x7fd018144c00 2024-03-06T07:37:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.958+0000 7fd00f400640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcff806b500 0x7fcff806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.958+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd00408a680 con 0x7fd018144c00 2024-03-06T07:37:49.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.958+0000 7fd017400640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcff806b500 0x7fcff806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.959+0000 7fd017400640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcff806b500 0x7fcff806d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd008003cc0 tx=0x7fd0080079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:49.967+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd004059450 con 0x7fd018144c00 2024-03-06T07:37:50.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.251+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd018072890 con 0x7fd018144c00 2024-03-06T07:37:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.252+0000 7fd00f400640 1 -- 172.21.15.97:0/2054168357 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+3424 (secure 0 0 0) 0x7fd004058d50 con 0x7fd018144c00 2024-03-06T07:37:50.254 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:50.254 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":113,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:45.917742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024,6463]},"id":10}]} 2024-03-06T07:37:50.255 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 113 2024-03-06T07:37:50.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcff806b500 msgr2=0x7fcff806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcff806b500 0x7fcff806d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd008003cc0 tx=0x7fd0080079b0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 msgr2=0x7fd01815d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 0x7fd01815d930 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fd00400d5f0 tx=0x7fd00400dab0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 shutdown_connections 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcff806b500 0x7fcff806d9c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd01815f9d0 0x7fd01815de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd018144c00 0x7fd01815d930 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 --2- 172.21.15.97:0/2054168357 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd018070b10 0x7fd018139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 >> 172.21.15.97:0/2054168357 conn(0x7fd0180b1730 msgr2=0x7fd018142bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 shutdown_connections 2024-03-06T07:37:50.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:50.256+0000 7fd01e600640 1 -- 172.21.15.97:0/2054168357 wait complete. 2024-03-06T07:37:50.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:37:50.471 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"14","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:37:50.472 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:37:50.472 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:37:52.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:52.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.687+0000 7f5a36000640 1 Processor -- start 2024-03-06T07:37:52.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.687+0000 7f5a36000640 1 -- start start 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.690+0000 7f5a36000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5a300cec60 0x7f5a300b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.690+0000 7f5a36000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300cf340 con 0x7f5a3006edb0 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.690+0000 7f5a36000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300cfe60 con 0x7f5a3015e230 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.690+0000 7f5a36000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3015e610 con 0x7f5a300cec60 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.690+0000 7f5a26a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5a3006edb0 0x7f5a300ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45660/0 (socket says 172.21.15.97:45660) 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.690+0000 7f5a26a00640 1 -- 172.21.15.97:0/2843843368 learned_addr learned my addr 172.21.15.97:0/2843843368 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2fe00640 1 --2- 172.21.15.97:0/2843843368 >> v2:172.21.15.135:3300/0 conn(0x7f5a300cec60 0x7f5a300b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579370388 0 0) 0x7f5a300cf340 con 0x7f5a3006edb0 2024-03-06T07:37:52.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a14003520 con 0x7f5a3006edb0 2024-03-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2739953750 0 0) 0x7f5a14003520 con 0x7f5a3006edb0 2024-03-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 >> v2:172.21.15.135:3300/0 conn(0x7f5a300cec60 msgr2=0x7f5a300b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 --2- 172.21.15.97:0/2843843368 >> v2:172.21.15.135:3300/0 conn(0x7f5a300cec60 0x7f5a300b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 >> v1:172.21.15.135:6789/0 conn(0x7f5a3015e230 legacy=0x7f5a300d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3006f390 con 0x7f5a3006edb0 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f5a18001670 con 0x7f5a3006edb0 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 >> v1:172.21.15.97:6789/0 conn(0x7f5a3006edb0 legacy=0x7f5a300ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a140039b0 0x7f5a14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a14004800 0x7f5a14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a140051c0 0x7f5a1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3015e610 con 0x7f5a140051c0 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1400c160 con 0x7f5a14004800 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.691+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1400c360 con 0x7f5a140039b0 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2fe00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a140051c0 0x7f5a1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a26a00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a14004800 0x7f5a14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2f400640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a140039b0 0x7f5a14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2fe00640 1 -- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a140039b0 msgr2=0x7f5a14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2fe00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a140039b0 0x7f5a14003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2fe00640 1 -- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a14004800 msgr2=0x7f5a14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2fe00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a14004800 0x7f5a14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a2fe00640 1 -- 172.21.15.97:0/2843843368 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14003520 con 0x7f5a140051c0 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.692+0000 7f5a26a00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a14004800 0x7f5a14004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.693+0000 7f5a2fe00640 1 -- 172.21.15.97:0/2843843368 reap_dead start 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.693+0000 7f5a2fe00640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a140051c0 0x7f5a1400b9c0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009a90 tx=0x7f5a14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396e7e1fa47b9376 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28009690 con 0x7f5a140051c0 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28004020 con 0x7f5a140051c0 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a2ea00640 1 -- 172.21.15.97:0/2843843368 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28004410 con 0x7f5a140051c0 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 -- 172.21.15.97:0/2843843368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a140051c0 msgr2=0x7f5a1400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a140051c0 0x7f5a1400b9c0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009a90 tx=0x7f5a14005830 comp rx=0 tx=0).stop 2024-03-06T07:37:52.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 -- 172.21.15.97:0/2843843368 shutdown_connections 2024-03-06T07:37:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 --2- 172.21.15.97:0/2843843368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a140051c0 0x7f5a1400b9c0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 -- 172.21.15.97:0/2843843368 >> 172.21.15.97:0/2843843368 conn(0x7f5a300b3800 msgr2=0x7f5a300b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 -- 172.21.15.97:0/2843843368 shutdown_connections 2024-03-06T07:37:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.694+0000 7f5a36000640 1 -- 172.21.15.97:0/2843843368 wait complete. 2024-03-06T07:37:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.697+0000 7f5a36000640 1 Processor -- start 2024-03-06T07:37:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.697+0000 7f5a36000640 1 -- start start 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a36000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a140051c0 0x7f5a300b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a36000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 0x7f5a300b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a36000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a3014b090 0x7f5a300c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a36000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300d06c0 con 0x7f5a3015b4a0 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a36000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300d03e0 con 0x7f5a140051c0 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a36000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300b4320 con 0x7f5a3014b090 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a26a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 0x7f5a300b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a26a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 0x7f5a300b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40844/0 (socket says 172.21.15.97:40844) 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.699+0000 7f5a26a00640 1 -- 172.21.15.97:0/2277645202 learned_addr learned my addr 172.21.15.97:0/2277645202 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:52.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a2f400640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a140051c0 0x7f5a300b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a2fe00640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a3014b090 0x7f5a300c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a26a00640 1 -- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a3014b090 msgr2=0x7f5a300c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a26a00640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a3014b090 0x7f5a300c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a26a00640 1 -- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a140051c0 msgr2=0x7f5a300b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a26a00640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a140051c0 0x7f5a300b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a26a00640 1 -- 172.21.15.97:0/2277645202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300b7770 con 0x7f5a3015b4a0 2024-03-06T07:37:52.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a2f400640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a140051c0 0x7f5a300b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.700+0000 7f5a26a00640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 0x7f5a300b7230 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800d770 tx=0x7f5a1800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.701+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18019070 con 0x7f5a3015b4a0 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.701+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a180074e0 con 0x7f5a3015b4a0 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.701+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18004a00 con 0x7f5a3015b4a0 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.701+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a300b6190 con 0x7f5a3015b4a0 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.701+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a300b6750 con 0x7f5a3015b4a0 2024-03-06T07:37:52.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.702+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3004f330 con 0x7f5a3015b4a0 2024-03-06T07:37:52.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.704+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a18004ba0 con 0x7f5a3015b4a0 2024-03-06T07:37:52.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.704+0000 7f5a27e00640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a0c06b500 0x7f5a0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.705+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5a18089c00 con 0x7f5a3015b4a0 2024-03-06T07:37:52.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.705+0000 7f5a2f400640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a0c06b500 0x7f5a0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.706+0000 7f5a2f400640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a0c06b500 0x7f5a0c06d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5a20007fd0 tx=0x7f5a20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.710+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a18090080 con 0x7f5a3015b4a0 2024-03-06T07:37:52.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.962+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a300d1740 con 0x7f5a3015b4a0 2024-03-06T07:37:52.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.963+0000 7f5a27e00640 1 -- 172.21.15.97:0/2277645202 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f5a18058a60 con 0x7f5a3015b4a0 2024-03-06T07:37:52.966 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:52.966 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":113,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:45.917742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024,6463]},"id":10}]} 2024-03-06T07:37:52.966 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 113 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a0c06b500 msgr2=0x7f5a0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a0c06b500 0x7f5a0c06d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5a20007fd0 tx=0x7f5a20009040 comp rx=0 tx=0).stop 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 msgr2=0x7f5a300b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 0x7f5a300b7230 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800d770 tx=0x7f5a1800dc30 comp rx=0 tx=0).stop 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 shutdown_connections 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5a0c06b500 0x7f5a0c06d9c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5a3014b090 0x7f5a300c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a3015b4a0 0x7f5a300b7230 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 --2- 172.21.15.97:0/2277645202 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a140051c0 0x7f5a300b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.966+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 >> 172.21.15.97:0/2277645202 conn(0x7f5a300b3800 msgr2=0x7f5a300af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.967+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 shutdown_connections 2024-03-06T07:37:52.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:52.967+0000 7f5a36000640 1 -- 172.21.15.97:0/2277645202 wait complete. 2024-03-06T07:37:52.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:37:53.188 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:37:53.188 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:37:53.189 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:37:55.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:55.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.447+0000 7f5238400640 1 Processor -- start 2024-03-06T07:37:55.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.447+0000 7f5238400640 1 -- start start 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.449+0000 7f5238400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f523006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.449+0000 7f5238400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5230072340 0x7f523006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.449+0000 7f5238400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523015b090 con 0x7f523015acb0 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.449+0000 7f5238400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52300b2a30 con 0x7f5230072340 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.449+0000 7f5238400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52300b4360 con 0x7f52300b4680 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.449+0000 7f5235a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f523015acb0 0x7f52300b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44432/0 (socket says 172.21.15.97:44432) 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5235a00640 1 -- 172.21.15.97:0/3031278245 learned_addr learned my addr 172.21.15.97:0/3031278245 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:55.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5236400640 1 --2- 172.21.15.97:0/3031278245 >> v2:172.21.15.97:3300/0 conn(0x7f5230072340 0x7f523006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5235000640 1 --2- 172.21.15.97:0/3031278245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f523006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f522fe00640 1 -- 172.21.15.97:0/3031278245 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202763321 0 0) 0x7f523015b090 con 0x7f523015acb0 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f522fe00640 1 -- 172.21.15.97:0/3031278245 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5218003520 con 0x7f523015acb0 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5236400640 1 -- 172.21.15.97:0/3031278245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 msgr2=0x7f523006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5236400640 1 --2- 172.21.15.97:0/3031278245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f523006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5236400640 1 -- 172.21.15.97:0/3031278245 >> v1:172.21.15.97:6789/0 conn(0x7f523015acb0 legacy=0x7f52300b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.450+0000 7f5236400640 1 -- 172.21.15.97:0/3031278245 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52300b2cf0 con 0x7f5230072340 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.451+0000 7f5236400640 1 --2- 172.21.15.97:0/3031278245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230072340 0x7f523006ea30 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5228009d90 tx=0x7f522802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4ad308759f0dc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f522fe00640 1 -- 172.21.15.97:0/3031278245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228039070 con 0x7f5230072340 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f522fe00640 1 -- 172.21.15.97:0/3031278245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522802cc60 con 0x7f5230072340 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f522fe00640 1 -- 172.21.15.97:0/3031278245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228034640 con 0x7f5230072340 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 -- 172.21.15.97:0/3031278245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230072340 msgr2=0x7f523006ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 --2- 172.21.15.97:0/3031278245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230072340 0x7f523006ea30 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5228009d90 tx=0x7f522802beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 -- 172.21.15.97:0/3031278245 shutdown_connections 2024-03-06T07:37:55.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 --2- 172.21.15.97:0/3031278245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230072340 0x7f523006ea30 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 --2- 172.21.15.97:0/3031278245 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f523006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 -- 172.21.15.97:0/3031278245 >> 172.21.15.97:0/3031278245 conn(0x7f52300b1730 msgr2=0x7f52300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:55.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 -- 172.21.15.97:0/3031278245 shutdown_connections 2024-03-06T07:37:55.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.452+0000 7f5238400640 1 -- 172.21.15.97:0/3031278245 wait complete. 2024-03-06T07:37:55.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.455+0000 7f5238400640 1 Processor -- start 2024-03-06T07:37:55.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.455+0000 7f5238400640 1 -- start start 2024-03-06T07:37:55.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5238400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5230072340 0x7f52301f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5238400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f52301524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5238400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 0x7f523015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5238400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52300b4010 con 0x7f523015acb0 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5238400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52300b32b0 con 0x7f5230072340 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5238400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52300ccfb0 con 0x7f52300b4680 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5236400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 0x7f523015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5236400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 0x7f523015baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60444/0 (socket says 172.21.15.97:60444) 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.457+0000 7f5236400640 1 -- 172.21.15.97:0/1378555735 learned_addr learned my addr 172.21.15.97:0/1378555735 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5235000640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f52301524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5235a00640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5230072340 0x7f52301f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5236400640 1 -- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 msgr2=0x7f52301524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5236400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f52301524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5236400640 1 -- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5230072340 msgr2=0x7f52301f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5236400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5230072340 0x7f52301f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5236400640 1 -- 172.21.15.97:0/1378555735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523015c3a0 con 0x7f523015acb0 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5235a00640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5230072340 0x7f52301f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f5236400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 0x7f523015baa0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5228009d90 tx=0x7f5228004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228039050 con 0x7f523015acb0 2024-03-06T07:37:55.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.458+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5228045070 con 0x7f523015acb0 2024-03-06T07:37:55.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.459+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523015c680 con 0x7f523015acb0 2024-03-06T07:37:55.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.459+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522803e830 con 0x7f523015acb0 2024-03-06T07:37:55.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.459+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523015cc40 con 0x7f523015acb0 2024-03-06T07:37:55.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.460+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523004f330 con 0x7f523015acb0 2024-03-06T07:37:55.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.461+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f522803e9d0 con 0x7f523015acb0 2024-03-06T07:37:55.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.461+0000 7f522d600640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f521006b4d0 0x7f521006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.462+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f52280ad1e0 con 0x7f523015acb0 2024-03-06T07:37:55.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.462+0000 7f5235a00640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f521006b4d0 0x7f521006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.462+0000 7f5235a00640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f521006b4d0 0x7f521006d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5224005880 tx=0x7f5224009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.466+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f522807c1f0 con 0x7f523015acb0 2024-03-06T07:37:55.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.736+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f523006ce60 con 0x7f523015acb0 2024-03-06T07:37:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.736+0000 7f522d600640 1 -- 172.21.15.97:0/1378555735 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f522807baf0 con 0x7f523015acb0 2024-03-06T07:37:55.739 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:55.739 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":113,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:37:45.917742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024,6463]},"id":10}]} 2024-03-06T07:37:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 113 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.738+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f521006b4d0 msgr2=0x7f521006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.738+0000 7f5238400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f521006b4d0 0x7f521006d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5224005880 tx=0x7f5224009040 comp rx=0 tx=0).stop 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.738+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 msgr2=0x7f523015baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.738+0000 7f5238400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 0x7f523015baa0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5228009d90 tx=0x7f5228004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 shutdown_connections 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f521006b4d0 0x7f521006d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523015acb0 0x7f523015baa0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f52300b4680 0x7f52301524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 --2- 172.21.15.97:0/1378555735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5230072340 0x7f52301f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 >> 172.21.15.97:0/1378555735 conn(0x7f52300b1730 msgr2=0x7f52300b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 shutdown_connections 2024-03-06T07:37:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:55.739+0000 7f5238400640 1 -- 172.21.15.97:0/1378555735 wait complete. 2024-03-06T07:37:55.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:37:55.988 INFO:teuthology.orchestra.run.smithi097.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-06T07:37:55.989 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:55.989 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2024-03-06T07:37:55.989 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:55.989 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:56.036 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:37:56.037 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:56.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:37:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.346+0000 7f53bb200640 1 Processor -- start 2024-03-06T07:37:56.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.346+0000 7f53bb200640 1 -- start start 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53bb200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53bc15e200 0x7f53bc0d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53bb200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0b5240 con 0x7f53bc0d0c00 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53bb200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0cf4d0 con 0x7f53bc0b2e50 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53bb200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0d2ec0 con 0x7f53bc15e200 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53b9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53bc15e200 0x7f53bc0d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53b9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f53bc15e200 0x7f53bc0d4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60456/0 (socket says 172.21.15.97:60456) 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53b9e00640 1 -- 172.21.15.97:0/2981402498 learned_addr learned my addr 172.21.15.97:0/2981402498 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53ba800640 1 --1- 172.21.15.97:0/2981402498 >> v1:172.21.15.97:6789/0 conn(0x7f53bc0d0c00 0x7f53bc0b3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44438/0 (socket says 172.21.15.97:44438) 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53b9e00640 1 -- 172.21.15.97:0/2981402498 >> v1:172.21.15.135:6789/0 conn(0x7f53bc0b2e50 legacy=0x7f53bc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:56.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.348+0000 7f53b9e00640 1 -- 172.21.15.97:0/2981402498 >> v1:172.21.15.97:6789/0 conn(0x7f53bc0d0c00 legacy=0x7f53bc0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.349+0000 7f53b9e00640 1 -- 172.21.15.97:0/2981402498 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0b5500 con 0x7f53bc15e200 2024-03-06T07:37:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.350+0000 7f53b9e00640 1 --2- 172.21.15.97:0/2981402498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc15e200 0x7f53bc0d4850 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009870 tx=0x7f53ac02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68be39f99edd72a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.350+0000 7f53b9400640 1 -- 172.21.15.97:0/2981402498 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac02f840 con 0x7f53bc15e200 2024-03-06T07:37:56.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53b9400640 1 -- 172.21.15.97:0/2981402498 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac02f9e0 con 0x7f53bc15e200 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53b9400640 1 -- 172.21.15.97:0/2981402498 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac02fdb0 con 0x7f53bc15e200 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 -- 172.21.15.97:0/2981402498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc15e200 msgr2=0x7f53bc0d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 --2- 172.21.15.97:0/2981402498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc15e200 0x7f53bc0d4850 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009870 tx=0x7f53ac02ba20 comp rx=0 tx=0).stop 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 -- 172.21.15.97:0/2981402498 shutdown_connections 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 --2- 172.21.15.97:0/2981402498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc15e200 0x7f53bc0d4850 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 -- 172.21.15.97:0/2981402498 >> 172.21.15.97:0/2981402498 conn(0x7f53bc0d3d30 msgr2=0x7f53bc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 -- 172.21.15.97:0/2981402498 shutdown_connections 2024-03-06T07:37:56.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.351+0000 7f53bb200640 1 -- 172.21.15.97:0/2981402498 wait complete. 2024-03-06T07:37:56.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.353+0000 7f53bb200640 1 Processor -- start 2024-03-06T07:37:56.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.353+0000 7f53bb200640 1 -- start start 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53bb200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53bc0b2e50 0x7f53bc0cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53bb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 0x7f53bc0b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53bb200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53bc15e200 0x7f53bc0b9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53bb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0cfd60 con 0x7f53bc0d0c00 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53bb200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0cfa80 con 0x7f53bc0b2e50 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53bb200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc15e5e0 con 0x7f53bc15e200 2024-03-06T07:37:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53b1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 0x7f53bc0b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53b1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 0x7f53bc0b90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60468/0 (socket says 172.21.15.97:60468) 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.354+0000 7f53b1400640 1 -- 172.21.15.97:0/1597362140 learned_addr learned my addr 172.21.15.97:0/1597362140 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53ba800640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53bc15e200 0x7f53bc0b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b9e00640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53bc0b2e50 0x7f53bc0cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b1400640 1 -- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53bc15e200 msgr2=0x7f53bc0b9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b1400640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53bc15e200 0x7f53bc0b9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b1400640 1 -- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53bc0b2e50 msgr2=0x7f53bc0cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b1400640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53bc0b2e50 0x7f53bc0cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b1400640 1 -- 172.21.15.97:0/1597362140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc15acd0 con 0x7f53bc0d0c00 2024-03-06T07:37:56.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53ba800640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53bc15e200 0x7f53bc0b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b1400640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 0x7f53bc0b90e0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c400 tx=0x7f53a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4011790 con 0x7f53bc0d0c00 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4017070 con 0x7f53bc0d0c00 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4011bb0 con 0x7f53bc0d0c00 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.355+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc0cc7c0 con 0x7f53bc0d0c00 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.356+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc0cb210 con 0x7f53bc0d0c00 2024-03-06T07:37:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.357+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc04f330 con 0x7f53bc0d0c00 2024-03-06T07:37:56.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.359+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53a401c460 con 0x7f53bc0d0c00 2024-03-06T07:37:56.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.360+0000 7f53b2a00640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.360+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f53a408a080 con 0x7f53bc0d0c00 2024-03-06T07:37:56.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.360+0000 7f53b9e00640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.361+0000 7f53b9e00640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f539806b4d0 0x7f539806d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f53ac036000 tx=0x7f53ac008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.368+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53a4058ed0 con 0x7f53bc0d0c00 2024-03-06T07:37:56.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.584+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53bc08d4c0 con 0x7f53bc0d0c00 2024-03-06T07:37:56.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.586+0000 7f53b2a00640 1 -- 172.21.15.97:0/1597362140 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f53a40587d0 con 0x7f53bc0d0c00 2024-03-06T07:37:56.589 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:37:56.590 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":148,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:40.116487+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.39","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f539806b4d0 msgr2=0x7f539806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f539806b4d0 0x7f539806d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f53ac036000 tx=0x7f53ac008030 comp rx=0 tx=0).stop 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 msgr2=0x7f53bc0b90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 0x7f53bc0b90e0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c400 tx=0x7f53a400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 shutdown_connections 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f53bc15e200 0x7f53bc0b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53bc0d0c00 0x7f53bc0b90e0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 --2- 172.21.15.97:0/1597362140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53bc0b2e50 0x7f53bc0cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 >> 172.21.15.97:0/1597362140 conn(0x7f53bc0d3d30 msgr2=0x7f53bc0d24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:56.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.589+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 shutdown_connections 2024-03-06T07:37:56.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.590+0000 7f53bb200640 1 -- 172.21.15.97:0/1597362140 wait complete. 2024-03-06T07:37:56.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:37:19.579712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:37:20.835626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:37:56.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:37:56.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.817+0000 7f6437000640 1 Processor -- start 2024-03-06T07:37:56.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.817+0000 7f6437000640 1 -- start start 2024-03-06T07:37:56.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.819+0000 7f6437000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f643015c330 0x7f64300d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.820+0000 7f6437000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f64300cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.820+0000 7f6437000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64300ce1e0 con 0x7f64300b43b0 2024-03-06T07:37:56.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.820+0000 7f6437000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64300cedf0 con 0x7f643015c330 2024-03-06T07:37:56.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.820+0000 7f6437000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64300ce4c0 con 0x7f64301605b0 2024-03-06T07:37:56.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.820+0000 7f6435000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f64300cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.820+0000 7f642fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f643015c330 0x7f64300d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.821+0000 7f6435000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f64300cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:50112/0 (socket says 172.21.15.97:50112) 2024-03-06T07:37:56.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.821+0000 7f6435000640 1 -- 172.21.15.97:0/1675286046 learned_addr learned my addr 172.21.15.97:0/1675286046 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:56.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.821+0000 7f642fe00640 1 -- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 msgr2=0x7f64300cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:56.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.821+0000 7f642fe00640 1 --2- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f64300cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.821+0000 7f642fe00640 1 -- 172.21.15.97:0/1675286046 >> v1:172.21.15.135:6789/0 conn(0x7f64300b43b0 legacy=0x7f643015fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.821+0000 7f642fe00640 1 -- 172.21.15.97:0/1675286046 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64300ce7a0 con 0x7f643015c330 2024-03-06T07:37:56.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.823+0000 7f642fe00640 1 --2- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f643015c330 0x7f64300d0570 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f64200097a0 tx=0x7f642002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a43ad739954975 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.824+0000 7f642ea00640 1 -- 172.21.15.97:0/1675286046 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420035020 con 0x7f643015c330 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.824+0000 7f642ea00640 1 -- 172.21.15.97:0/1675286046 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6420031830 con 0x7f643015c330 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.824+0000 7f642ea00640 1 -- 172.21.15.97:0/1675286046 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420031c00 con 0x7f643015c330 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.824+0000 7f6437000640 1 -- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f643015c330 msgr2=0x7f64300d0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 --2- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f643015c330 0x7f64300d0570 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f64200097a0 tx=0x7f642002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 -- 172.21.15.97:0/1675286046 shutdown_connections 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 --2- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f64300cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 --2- 172.21.15.97:0/1675286046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f643015c330 0x7f64300d0570 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 -- 172.21.15.97:0/1675286046 >> 172.21.15.97:0/1675286046 conn(0x7f64300b3790 msgr2=0x7f64300b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:56.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 -- 172.21.15.97:0/1675286046 shutdown_connections 2024-03-06T07:37:56.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.825+0000 7f6437000640 1 -- 172.21.15.97:0/1675286046 wait complete. 2024-03-06T07:37:56.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.826+0000 7f6437000640 1 Processor -- start 2024-03-06T07:37:56.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.827+0000 7f6437000640 1 -- start start 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.827+0000 7f6437000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64300b43b0 0x7f64300c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.827+0000 7f6437000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 0x7f64300b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f6437000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f643015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f6437000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64300cf280 con 0x7f643015c330 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f6437000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643015d720 con 0x7f64300b43b0 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f6437000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64300c6b10 con 0x7f64301605b0 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 0x7f64300b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 0x7f64300b6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60472/0 (socket says 172.21.15.97:60472) 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 -- 172.21.15.97:0/2769947442 learned_addr learned my addr 172.21.15.97:0/2769947442 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f6435000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f643015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642fe00640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64300b43b0 0x7f64300c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 -- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 msgr2=0x7f643015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f643015ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 -- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64300b43b0 msgr2=0x7f64300c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64300b43b0 0x7f64300c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.828+0000 7f642f400640 1 -- 172.21.15.97:0/2769947442 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64300c7370 con 0x7f643015c330 2024-03-06T07:37:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.829+0000 7f642f400640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 0x7f64300b6960 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c400 tx=0x7f641c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.829+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c0052d0 con 0x7f643015c330 2024-03-06T07:37:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.829+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64300c6db0 con 0x7f643015c330 2024-03-06T07:37:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.829+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c005490 con 0x7f643015c330 2024-03-06T07:37:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.829+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c007260 con 0x7f643015c330 2024-03-06T07:37:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.829+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64300b4b50 con 0x7f643015c330 2024-03-06T07:37:56.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.830+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6430071430 con 0x7f643015c330 2024-03-06T07:37:56.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.833+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f641c007400 con 0x7f643015c330 2024-03-06T07:37:56.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.834+0000 7f6427e00640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f641006b4d0 0x7f641006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.834+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f641c0898a0 con 0x7f643015c330 2024-03-06T07:37:56.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.834+0000 7f642fe00640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f641006b4d0 0x7f641006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.835+0000 7f642fe00640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f641006b4d0 0x7f641006d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6420036000 tx=0x7f6420008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:56.839+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6430071430 con 0x7f643015c330 2024-03-06T07:37:57.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.036+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64300b4840 con 0x7f643015c330 2024-03-06T07:37:57.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.037+0000 7f6427e00640 1 -- 172.21.15.97:0/2769947442 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64300b4840 con 0x7f643015c330 2024-03-06T07:37:57.039 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f641006b4d0 msgr2=0x7f641006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f641006b4d0 0x7f641006d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6420036000 tx=0x7f6420008030 comp rx=0 tx=0).stop 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 msgr2=0x7f64300b6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 0x7f64300b6960 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c400 tx=0x7f641c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 shutdown_connections 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f641006b4d0 0x7f641006d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f64301605b0 0x7f643015ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f643015c330 0x7f64300b6960 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 --2- 172.21.15.97:0/2769947442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64300b43b0 0x7f64300c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.039+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 >> 172.21.15.97:0/2769947442 conn(0x7f64300b3790 msgr2=0x7f64300cd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.040+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 shutdown_connections 2024-03-06T07:37:57.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.040+0000 7f6437000640 1 -- 172.21.15.97:0/2769947442 wait complete. 2024-03-06T07:37:57.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:37:57.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.319+0000 7fd597e00640 1 Processor -- start 2024-03-06T07:37:57.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.319+0000 7fd597e00640 1 -- start start 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd598161630 0x7fd5980cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd5980ce0e0 0x7fd5980ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5980d0210 con 0x7fd59806ebe0 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5980cec50 con 0x7fd598161630 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5980d0820 con 0x7fd5980ce0e0 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd596a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd598161630 0x7fd5980cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd59806ebe0 0x7fd59806b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44440/0 (socket says 172.21.15.97:44440) 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd597400640 1 -- 172.21.15.97:0/3151081158 learned_addr learned my addr 172.21.15.97:0/3151081158 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.321+0000 7fd596000640 1 --2- 172.21.15.97:0/3151081158 >> v2:172.21.15.135:3300/0 conn(0x7fd5980ce0e0 0x7fd5980ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.322+0000 7fd596a00640 1 -- 172.21.15.97:0/3151081158 >> v2:172.21.15.135:3300/0 conn(0x7fd5980ce0e0 msgr2=0x7fd5980ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.322+0000 7fd596a00640 1 --2- 172.21.15.97:0/3151081158 >> v2:172.21.15.135:3300/0 conn(0x7fd5980ce0e0 0x7fd5980ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.322+0000 7fd596a00640 1 -- 172.21.15.97:0/3151081158 >> v1:172.21.15.97:6789/0 conn(0x7fd59806ebe0 legacy=0x7fd59806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.322+0000 7fd596a00640 1 -- 172.21.15.97:0/3151081158 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598161a10 con 0x7fd598161630 2024-03-06T07:37:57.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.322+0000 7fd595600640 1 -- 172.21.15.97:0/3151081158 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831845829 0 0) 0x7fd5980d0210 con 0x7fd59806ebe0 2024-03-06T07:37:57.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.323+0000 7fd596a00640 1 --2- 172.21.15.97:0/3151081158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd5980cb190 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fd598074d90 tx=0x7fd58802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77950a97d5ac679 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd595600640 1 -- 172.21.15.97:0/3151081158 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588035020 con 0x7fd598161630 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd595600640 1 -- 172.21.15.97:0/3151081158 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd588031830 con 0x7fd598161630 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd595600640 1 -- 172.21.15.97:0/3151081158 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588031c00 con 0x7fd598161630 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 -- 172.21.15.97:0/3151081158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 msgr2=0x7fd5980cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 --2- 172.21.15.97:0/3151081158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd5980cb190 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fd598074d90 tx=0x7fd58802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 -- 172.21.15.97:0/3151081158 shutdown_connections 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 --2- 172.21.15.97:0/3151081158 >> v2:172.21.15.135:3300/0 conn(0x7fd5980ce0e0 0x7fd5980ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 --2- 172.21.15.97:0/3151081158 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd5980cb190 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 -- 172.21.15.97:0/3151081158 >> 172.21.15.97:0/3151081158 conn(0x7fd5980cd5c0 msgr2=0x7fd5980cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.324+0000 7fd597e00640 1 -- 172.21.15.97:0/3151081158 shutdown_connections 2024-03-06T07:37:57.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.325+0000 7fd597e00640 1 -- 172.21.15.97:0/3151081158 wait complete. 2024-03-06T07:37:57.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.327+0000 7fd597e00640 1 Processor -- start 2024-03-06T07:37:57.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.327+0000 7fd597e00640 1 -- start start 2024-03-06T07:37:57.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59806ebe0 0x7fd5980c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5980ce0e0 0x7fd59815f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd59815fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598072780 con 0x7fd598161630 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980d04b0 con 0x7fd59806ebe0 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59806f010 con 0x7fd5980ce0e0 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd59815fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd59815fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60480/0 (socket says 172.21.15.97:60480) 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.329+0000 7fd597400640 1 -- 172.21.15.97:0/2977426896 learned_addr learned my addr 172.21.15.97:0/2977426896 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd596a00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59806ebe0 0x7fd5980c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd596000640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5980ce0e0 0x7fd59815f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597400640 1 -- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5980ce0e0 msgr2=0x7fd59815f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597400640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5980ce0e0 0x7fd59815f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597400640 1 -- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59806ebe0 msgr2=0x7fd5980c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597400640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59806ebe0 0x7fd5980c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597400640 1 -- 172.21.15.97:0/2977426896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980c89d0 con 0x7fd598161630 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd596a00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59806ebe0 0x7fd5980c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd596000640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5980ce0e0 0x7fd59815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597400640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd59815fae0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c7a0 tx=0x7fd59000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590018070 con 0x7fd598161630 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59000a640 con 0x7fd598161630 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000aa70 con 0x7fd598161630 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.330+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5980c8cb0 con 0x7fd598161630 2024-03-06T07:37:57.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.331+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5980c9270 con 0x7fd598161630 2024-03-06T07:37:57.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.332+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59804f330 con 0x7fd598161630 2024-03-06T07:37:57.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.333+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd59001b070 con 0x7fd598161630 2024-03-06T07:37:57.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.333+0000 7fd58ea00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.334+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd590089840 con 0x7fd598161630 2024-03-06T07:37:57.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.334+0000 7fd596a00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.335+0000 7fd596a00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd588036000 tx=0x7fd588008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.338+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd590058690 con 0x7fd598161630 2024-03-06T07:37:57.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.537+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd598160e50 con 0x7fd598161630 2024-03-06T07:37:57.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.537+0000 7fd58ea00640 1 -- 172.21.15.97:0/2977426896 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd590057f90 con 0x7fd598161630 2024-03-06T07:37:57.540 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:37:57.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 msgr2=0x7fd57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd588036000 tx=0x7fd588008030 comp rx=0 tx=0).stop 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 msgr2=0x7fd59815fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd59815fae0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c7a0 tx=0x7fd59000cc60 comp rx=0 tx=0).stop 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 shutdown_connections 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd598161630 0x7fd59815fae0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd5980ce0e0 0x7fd59815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 --2- 172.21.15.97:0/2977426896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd59806ebe0 0x7fd5980c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 >> 172.21.15.97:0/2977426896 conn(0x7fd5980cd5c0 msgr2=0x7fd5980717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 shutdown_connections 2024-03-06T07:37:57.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.541+0000 7fd597e00640 1 -- 172.21.15.97:0/2977426896 wait complete. 2024-03-06T07:37:57.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:37:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.801+0000 7f8bc7e00640 1 Processor -- start 2024-03-06T07:37:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.801+0000 7f8bc7e00640 1 -- start start 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8bc815e000 0x7f8bc80ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc7e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80cf020 con 0x7f8bc80d40b0 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc7e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80cf300 con 0x7f8bc80d1b90 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc7e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80cedd0 con 0x7f8bc815e000 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8bc815e000 0x7f8bc80ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc7400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8bc80d40b0 0x7f8bc80b3de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44448/0 (socket says 172.21.15.97:44448) 2024-03-06T07:37:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.803+0000 7f8bc7400640 1 -- 172.21.15.97:0/3552858747 learned_addr learned my addr 172.21.15.97:0/3552858747 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.804+0000 7f8bc6a00640 1 -- 172.21.15.97:0/3552858747 >> v1:172.21.15.135:6789/0 conn(0x7f8bc80d1b90 legacy=0x7f8bc80d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.804+0000 7f8bc6a00640 1 -- 172.21.15.97:0/3552858747 >> v1:172.21.15.97:6789/0 conn(0x7f8bc80d40b0 legacy=0x7f8bc80b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.804+0000 7f8bc6a00640 1 -- 172.21.15.97:0/3552858747 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80cf5e0 con 0x7f8bc815e000 2024-03-06T07:37:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.805+0000 7f8bc6a00640 1 --2- 172.21.15.97:0/3552858747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc815e000 0x7f8bc80ce890 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80097a0 tx=0x7f8bb802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9748b00c6d1bafd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.805+0000 7f8bc6000640 1 -- 172.21.15.97:0/3552858747 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8037070 con 0x7f8bc815e000 2024-03-06T07:37:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.805+0000 7f8bc6000640 1 -- 172.21.15.97:0/3552858747 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb8030710 con 0x7f8bc815e000 2024-03-06T07:37:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.805+0000 7f8bc6000640 1 -- 172.21.15.97:0/3552858747 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8030ae0 con 0x7f8bc815e000 2024-03-06T07:37:57.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3552858747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc815e000 msgr2=0x7f8bc80ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3552858747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc815e000 0x7f8bc80ce890 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80097a0 tx=0x7f8bb802b710 comp rx=0 tx=0).stop 2024-03-06T07:37:57.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3552858747 shutdown_connections 2024-03-06T07:37:57.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3552858747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc815e000 0x7f8bc80ce890 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3552858747 >> 172.21.15.97:0/3552858747 conn(0x7f8bc808d740 msgr2=0x7f8bc809d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3552858747 shutdown_connections 2024-03-06T07:37:57.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.806+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3552858747 wait complete. 2024-03-06T07:37:57.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.808+0000 7f8bc7e00640 1 Processor -- start 2024-03-06T07:37:57.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.808+0000 7f8bc7e00640 1 -- start start 2024-03-06T07:37:57.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc7e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bc80d1b90 0x7f8bc81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 0x7f8bc80c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc7e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bc815e000 0x7f8bc80c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80d2010 con 0x7f8bc80d40b0 2024-03-06T07:37:57.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc7e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80cfba0 con 0x7f8bc815e000 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc7e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80b44f0 con 0x7f8bc80d1b90 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 0x7f8bc80c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 0x7f8bc80c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60494/0 (socket says 172.21.15.97:60494) 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 -- 172.21.15.97:0/3568908518 learned_addr learned my addr 172.21.15.97:0/3568908518 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bc6a00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bc80d1b90 0x7f8bc81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 -- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bc80d1b90 msgr2=0x7f8bc81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bc80d1b90 0x7f8bc81553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 -- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bc815e000 msgr2=0x7f8bc80c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bc815e000 0x7f8bc80c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.809+0000 7f8bbfe00640 1 -- 172.21.15.97:0/3568908518 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80b6b40 con 0x7f8bc80d40b0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.810+0000 7f8bbfe00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 0x7f8bc80c8e80 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000c430 tx=0x7f8bb000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.810+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0011790 con 0x7f8bc80d40b0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.810+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb0017070 con 0x7f8bc80d40b0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.810+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0011bb0 con 0x7f8bc80d40b0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.810+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc80b6e20 con 0x7f8bc80d40b0 2024-03-06T07:37:57.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.810+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc80b7900 con 0x7f8bc80d40b0 2024-03-06T07:37:57.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.811+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc804f330 con 0x7f8bc80d40b0 2024-03-06T07:37:57.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.812+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bb001c460 con 0x7f8bc80d40b0 2024-03-06T07:37:57.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.813+0000 7f8bbea00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8ba406b4d0 0x7f8ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.813+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8bb008a080 con 0x7f8bc80d40b0 2024-03-06T07:37:57.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.814+0000 7f8bc6a00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8ba406b4d0 0x7f8ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.814+0000 7f8bc6a00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8ba406b4d0 0x7f8ba406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8034000 tx=0x7f8bb8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:57.819+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bb0058f70 con 0x7f8bc80d40b0 2024-03-06T07:37:58.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.015+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8bc80b4950 con 0x7f8bc80d40b0 2024-03-06T07:37:58.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.016+0000 7f8bbea00640 1 -- 172.21.15.97:0/3568908518 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8bb0058870 con 0x7f8bc80d40b0 2024-03-06T07:37:58.018 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8ba406b4d0 msgr2=0x7f8ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8ba406b4d0 0x7f8ba406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8034000 tx=0x7f8bb8008030 comp rx=0 tx=0).stop 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 msgr2=0x7f8bc80c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 0x7f8bc80c8e80 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000c430 tx=0x7f8bb000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 shutdown_connections 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8ba406b4d0 0x7f8ba406d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bc815e000 0x7f8bc80c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bc80d40b0 0x7f8bc80c8e80 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 --2- 172.21.15.97:0/3568908518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8bc80d1b90 0x7f8bc81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.018+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 >> 172.21.15.97:0/3568908518 conn(0x7f8bc808d740 msgr2=0x7f8bc806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:58.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.019+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 shutdown_connections 2024-03-06T07:37:58.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.019+0000 7f8bc7e00640 1 -- 172.21.15.97:0/3568908518 wait complete. 2024-03-06T07:37:58.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:37:58.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.221+0000 7f50b7c00640 1 Processor -- start 2024-03-06T07:37:58.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.221+0000 7f50b7c00640 1 -- start start 2024-03-06T07:37:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.222+0000 7f50b7c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f50b00d2830 0x7f50b00b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.222+0000 7f50b7c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00b32c0 con 0x7f50b015f510 2024-03-06T07:37:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.222+0000 7f50b7c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b015f8f0 con 0x7f50b00af640 2024-03-06T07:37:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.222+0000 7f50b7c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b00aeb60 con 0x7f50b00d2830 2024-03-06T07:37:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.223+0000 7f50b5200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f50b015f510 0x7f50b00b1410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44458/0 (socket says 172.21.15.97:44458) 2024-03-06T07:37:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.223+0000 7f50b5200640 1 -- 172.21.15.97:0/727540592 learned_addr learned my addr 172.21.15.97:0/727540592 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:58.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.223+0000 7f50afe00640 1 --2- 172.21.15.97:0/727540592 >> v2:172.21.15.135:3300/0 conn(0x7f50b00d2830 0x7f50b00b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.223+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981206992 0 0) 0x7f50b00b32c0 con 0x7f50b015f510 2024-03-06T07:37:58.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.223+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098003520 con 0x7f50b015f510 2024-03-06T07:37:58.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 181390107 0 0) 0x7f5098003520 con 0x7f50b015f510 2024-03-06T07:37:58.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 >> v2:172.21.15.135:3300/0 conn(0x7f50b00d2830 msgr2=0x7f50b00b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:58.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 --2- 172.21.15.97:0/727540592 >> v2:172.21.15.135:3300/0 conn(0x7f50b00d2830 0x7f50b00b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 >> v1:172.21.15.135:6789/0 conn(0x7f50b00af640 legacy=0x7f50b00afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00aee40 con 0x7f50b015f510 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f50a4001690 con 0x7f50b015f510 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 >> v1:172.21.15.97:6789/0 conn(0x7f50b015f510 legacy=0x7f50b00b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5098004800 0x7f5098004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980051c0 0x7f509800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00aeb60 con 0x7f50980039b0 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509800c060 con 0x7f50980051c0 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509800c260 con 0x7f5098004800 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50b5200640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.224+0000 7f50b5c00640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980051c0 0x7f509800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50afe00640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5098004800 0x7f5098004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50b5200640 1 -- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5098004800 msgr2=0x7f5098004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50b5200640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5098004800 0x7f5098004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50b5200640 1 -- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980051c0 msgr2=0x7f509800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:58.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50b5200640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980051c0 0x7f509800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50b5200640 1 -- 172.21.15.97:0/727540592 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00aee40 con 0x7f50980039b0 2024-03-06T07:37:58.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.225+0000 7f50b5c00640 1 -- 172.21.15.97:0/727540592 reap_dead start 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.226+0000 7f50b5200640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f50b00b3490 tx=0x7f5098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa4626523e07f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.226+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4007d30 con 0x7f50980039b0 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.226+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4030070 con 0x7f50980039b0 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.226+0000 7f50af400640 1 -- 172.21.15.97:0/727540592 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a402cd40 con 0x7f50980039b0 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 -- 172.21.15.97:0/727540592 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50980039b0 msgr2=0x7f5098003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f50b00b3490 tx=0x7f5098005830 comp rx=0 tx=0).stop 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 -- 172.21.15.97:0/727540592 shutdown_connections 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 --2- 172.21.15.97:0/727540592 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 -- 172.21.15.97:0/727540592 >> 172.21.15.97:0/727540592 conn(0x7f50b00d1d10 msgr2=0x7f50b00d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:58.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 -- 172.21.15.97:0/727540592 shutdown_connections 2024-03-06T07:37:58.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.227+0000 7f50b7c00640 1 -- 172.21.15.97:0/727540592 wait complete. 2024-03-06T07:37:58.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.229+0000 7f50b7c00640 1 Processor -- start 2024-03-06T07:37:58.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.229+0000 7f50b7c00640 1 -- start start 2024-03-06T07:37:58.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b7c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980039b0 0x7f50b00cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b7c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50b00c8de0 0x7f50b01f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b7c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 0x7f50b00c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b7c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00b0fc0 con 0x7f50b00ca5f0 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b7c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00b3560 con 0x7f50980039b0 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b7c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00b1140 con 0x7f50b00c8de0 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 0x7f50b00c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 0x7f50b00c6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60514/0 (socket says 172.21.15.97:60514) 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5c00640 1 -- 172.21.15.97:0/1920909698 learned_addr learned my addr 172.21.15.97:0/1920909698 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5c00640 1 -- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50b00c8de0 msgr2=0x7f50b01f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50b00c8de0 0x7f50b01f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5c00640 1 -- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980039b0 msgr2=0x7f50b00cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.230+0000 7f50b5200640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980039b0 0x7f50b00cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50b5c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980039b0 0x7f50b00cf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50b5c00640 1 -- 172.21.15.97:0/1920909698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b01f4140 con 0x7f50b00ca5f0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50b5c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 0x7f50b00c6fa0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f50b00723c0 tx=0x7f50a800e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8019070 con 0x7f50b00ca5f0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a8006d60 con 0x7f50b00ca5f0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8003b90 con 0x7f50b00ca5f0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b00c9550 con 0x7f50b00ca5f0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.231+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b00c9b10 con 0x7f50b00ca5f0 2024-03-06T07:37:58.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.233+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b00b2f80 con 0x7f50b00ca5f0 2024-03-06T07:37:58.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.234+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50a801c070 con 0x7f50b00ca5f0 2024-03-06T07:37:58.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.235+0000 7f50acc00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.235+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f50a80896e0 con 0x7f50b00ca5f0 2024-03-06T07:37:58.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.235+0000 7f50b5200640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.236+0000 7f50b5200640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f50a4034000 tx=0x7f50a402f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:58.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.243+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50b00b2f80 con 0x7f50b00ca5f0 2024-03-06T07:37:58.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.496+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f50b004f330 con 0x7f50b00ca5f0 2024-03-06T07:37:58.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.497+0000 7f50acc00640 1 -- 172.21.15.97:0/1920909698 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v148) v1 ==== 80+0+0 (secure 0 0 0) 0x7f50b01f4140 con 0x7f50b00ca5f0 2024-03-06T07:37:58.499 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:37:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f509006b4d0 msgr2=0x7f509006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f50a4034000 tx=0x7f50a402f000 comp rx=0 tx=0).stop 2024-03-06T07:37:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 msgr2=0x7f50b00c6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 0x7f50b00c6fa0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f50b00723c0 tx=0x7f50a800e9e0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 shutdown_connections 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50b00ca5f0 0x7f50b00c6fa0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50b00c8de0 0x7f50b01f3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 --2- 172.21.15.97:0/1920909698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50980039b0 0x7f50b00cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 >> 172.21.15.97:0/1920909698 conn(0x7f50b00d1d10 msgr2=0x7f50b00b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 shutdown_connections 2024-03-06T07:37:58.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:58.499+0000 7f50b7c00640 1 -- 172.21.15.97:0/1920909698 wait complete. 2024-03-06T07:37:58.518 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:37:58.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:58.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:58.624 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:37:58.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:58.625 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:37:58.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:37:58.881 INFO:teuthology.orchestra.run.smithi097.stdout:{"message":"","return_code":0} 2024-03-06T07:37:58.881 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:37:58.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json cache drop 2024-03-06T07:37:59.108 INFO:teuthology.orchestra.run.smithi097.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":285,"bytes":58994}},"duration":0.0088563119999999999} 2024-03-06T07:37:59.109 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 58994, "items": 285 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.008856312, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:37:59.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:37:59.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.287+0000 7f5cfe200640 1 Processor -- start 2024-03-06T07:37:59.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.287+0000 7f5cfe200640 1 -- start start 2024-03-06T07:37:59.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfe200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfe200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5cf815e4b0 0x7f5cf80af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfe200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf80b1440 con 0x7f5cf80b5db0 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfe200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf806b500 con 0x7f5cf815e4b0 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfe200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf806c0d0 con 0x7f5cf815e0d0 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cf7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5cf815e4b0 0x7f5cf80af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfd800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5cf80b5db0 0x7f5cf8072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44462/0 (socket says 172.21.15.97:44462) 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfd800640 1 -- 172.21.15.97:0/106908212 learned_addr learned my addr 172.21.15.97:0/106908212 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.289+0000 7f5cfce00640 1 --2- 172.21.15.97:0/106908212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.290+0000 7f5cf7e00640 1 -- 172.21.15.97:0/106908212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 msgr2=0x7f5cf806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.290+0000 7f5cf7e00640 1 --2- 172.21.15.97:0/106908212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.290+0000 7f5cf7e00640 1 -- 172.21.15.97:0/106908212 >> v1:172.21.15.97:6789/0 conn(0x7f5cf80b5db0 legacy=0x7f5cf8072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.290+0000 7f5cf7e00640 1 -- 172.21.15.97:0/106908212 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf80b16e0 con 0x7f5cf815e4b0 2024-03-06T07:37:59.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.290+0000 7f5cf7400640 1 -- 172.21.15.97:0/106908212 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734723336 0 0) 0x7f5cf80b1440 con 0x7f5cf80b5db0 2024-03-06T07:37:59.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.291+0000 7f5cf7e00640 1 --2- 172.21.15.97:0/106908212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf80af240 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40097a0 tx=0x7f5ce402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486ba30c838dd7ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.291+0000 7f5cf7400640 1 -- 172.21.15.97:0/106908212 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4034070 con 0x7f5cf815e4b0 2024-03-06T07:37:59.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cf7400640 1 -- 172.21.15.97:0/106908212 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce4030800 con 0x7f5cf815e4b0 2024-03-06T07:37:59.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cf7400640 1 -- 172.21.15.97:0/106908212 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4030bd0 con 0x7f5cf815e4b0 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cfe200640 1 -- 172.21.15.97:0/106908212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 msgr2=0x7f5cf80af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cfe200640 1 --2- 172.21.15.97:0/106908212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf80af240 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40097a0 tx=0x7f5ce402b940 comp rx=0 tx=0).stop 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cfe200640 1 -- 172.21.15.97:0/106908212 shutdown_connections 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cfe200640 1 --2- 172.21.15.97:0/106908212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf80af240 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cfe200640 1 --2- 172.21.15.97:0/106908212 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.292+0000 7f5cfe200640 1 -- 172.21.15.97:0/106908212 >> 172.21.15.97:0/106908212 conn(0x7f5cf80b4fe0 msgr2=0x7f5cf80b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.293+0000 7f5cfe200640 1 -- 172.21.15.97:0/106908212 shutdown_connections 2024-03-06T07:37:59.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.293+0000 7f5cfe200640 1 -- 172.21.15.97:0/106908212 wait complete. 2024-03-06T07:37:59.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.294+0000 7f5cfe200640 1 Processor -- start 2024-03-06T07:37:59.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.294+0000 7f5cfe200640 1 -- start start 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.295+0000 7f5cfe200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cf80b5db0 0x7f5cf81528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.295+0000 7f5cfe200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf8152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.295+0000 7f5cfe200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf8154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.295+0000 7f5cfe200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf806c320 con 0x7f5cf815e4b0 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.295+0000 7f5cfe200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf806c690 con 0x7f5cf80b5db0 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.295+0000 7f5cfe200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80ae170 con 0x7f5cf815e0d0 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf8154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf8154950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60532/0 (socket says 172.21.15.97:60532) 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 -- 172.21.15.97:0/936822882 learned_addr learned my addr 172.21.15.97:0/936822882 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfce00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cf80b5db0 0x7f5cf81528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cf7e00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf8152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 -- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 msgr2=0x7f5cf8152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf8152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 -- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cf80b5db0 msgr2=0x7f5cf81528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cf80b5db0 0x7f5cf81528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 -- 172.21.15.97:0/936822882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8156010 con 0x7f5cf815e4b0 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cf7e00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf8152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfce00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cf80b5db0 0x7f5cf81528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.296+0000 7f5cfd800640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf8154950 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c780 tx=0x7f5cf000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.297+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0018070 con 0x7f5cf815e4b0 2024-03-06T07:37:59.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.297+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf000a640 con 0x7f5cf815e4b0 2024-03-06T07:37:59.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.297+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000aab0 con 0x7f5cf815e4b0 2024-03-06T07:37:59.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.297+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf8155710 con 0x7f5cf815e4b0 2024-03-06T07:37:59.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.297+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf8099bf0 con 0x7f5cf815e4b0 2024-03-06T07:37:59.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.298+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf804f330 con 0x7f5cf815e4b0 2024-03-06T07:37:59.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.299+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cf000ac50 con 0x7f5cf815e4b0 2024-03-06T07:37:59.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.300+0000 7f5cf4c00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cd806b500 0x7f5cd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.300+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5cf0089870 con 0x7f5cf815e4b0 2024-03-06T07:37:59.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.300+0000 7f5cfce00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cd806b500 0x7f5cd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.301+0000 7f5cfce00640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cd806b500 0x7f5cd806d9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5cec006fd0 tx=0x7f5cec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.304+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cf0058630 con 0x7f5cf815e4b0 2024-03-06T07:37:59.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.545+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5cf8155c30 con 0x7f5cf815e4b0 2024-03-06T07:37:59.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.545+0000 7f5cf4c00640 1 -- 172.21.15.97:0/936822882 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v113) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5cf0057f30 con 0x7f5cf815e4b0 2024-03-06T07:37:59.547 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:37:59.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cd806b500 msgr2=0x7f5cd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cd806b500 0x7f5cd806d9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5cec006fd0 tx=0x7f5cec008040 comp rx=0 tx=0).stop 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 msgr2=0x7f5cf8154950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf8154950 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c780 tx=0x7f5cf000cc40 comp rx=0 tx=0).stop 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 shutdown_connections 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5cd806b500 0x7f5cd806d9c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5cf815e4b0 0x7f5cf8154950 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5cf815e0d0 0x7f5cf8152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 --2- 172.21.15.97:0/936822882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cf80b5db0 0x7f5cf81528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 >> 172.21.15.97:0/936822882 conn(0x7f5cf80b4fe0 msgr2=0x7f5cf815cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 shutdown_connections 2024-03-06T07:37:59.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:37:59.548+0000 7f5cfe200640 1 -- 172.21.15.97:0/936822882 wait complete. 2024-03-06T07:37:59.573 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:37:59.574 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:59.574 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 41995sec preferred_lft 41995sec 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link proto kernel_ll 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:37:59.620 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d002:6fff:fe5b:7325/64 scope link proto kernel_ll 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 2e:74:1a:3b:7c:e3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::2c74:1aff:fe3b:7ce3/64 scope link proto kernel_ll 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether da:dc:e2:b3:81:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::d8dc:e2ff:feb3:8183/64 scope link proto kernel_ll 2024-03-06T07:37:59.621 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:59.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:59.622 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:37:59.744 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-06T07:37:59.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:59.750 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-06T07:37:59.796 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-06T07:37:59.796 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:37:59.796 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:37:59.796 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:37:59.796 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:37:59.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:59.796 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-06T07:37:59.796 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-06T07:37:59.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:37:59.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:37:59.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:37:59.797 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-06T07:38:00.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:00.001 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:38:00.048 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:38:00.048 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:38:00.048 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:38:00.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:00.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-06T07:38:00.128 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:38:00.128 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:38:00.171 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:38:00.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:38:00.172 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:38:00.259 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:38:00.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:00.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:38:00.435 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:38:00.435 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:38:00.466 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:38:00.466 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:38:00.466 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.135:3300,172.21.15.135:3301" 2024-03-06T07:38:00.466 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with new device syntax: 0@0b9abceb-2115-4321-96a2-348db0d0fbe4.cephfs=/ 2024-03-06T07:38:00.467 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.97:3300/172.21.15.135:3300/172.21.15.135:3301" will pass to kernel 2024-03-06T07:38:00.470 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:38:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:38:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:38:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:38:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:38:00.474 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:38:00.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:00.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:38:00.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:00.556 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:38:00.609 INFO:teuthology.orchestra.run.smithi097.stdout:client5957 2024-03-06T07:38:00.609 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:38:00.609 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:38:00.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:00.650 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/original) 2024-03-06T07:38:00.747 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:38:00.748 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:38:00.797 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:38:00.797 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:38:00.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:38:01.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.082+0000 7f4c19800640 1 Processor -- start 2024-03-06T07:38:01.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.082+0000 7f4c19800640 1 -- start start 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.084+0000 7f4c19800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4c14161580 0x7f4c140d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.084+0000 7f4c19800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c140d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c19800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c14161960 con 0x7f4c140d28a0 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c19800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c140cf6f0 con 0x7f4c14161580 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c19800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140cf9d0 con 0x7f4c140d1e90 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c13e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4c14161580 0x7f4c140d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c13e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4c14161580 0x7f4c140d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60542/0 (socket says 172.21.15.97:60542) 2024-03-06T07:38:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c13e00640 1 -- 172.21.15.97:0/956221571 learned_addr learned my addr 172.21.15.97:0/956221571 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:01.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.085+0000 7f4c13400640 1 --2- 172.21.15.97:0/956221571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c140d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.086+0000 7f4c13e00640 1 -- 172.21.15.97:0/956221571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 msgr2=0x7f4c140d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.086+0000 7f4c13e00640 1 --2- 172.21.15.97:0/956221571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c140d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.086+0000 7f4c13e00640 1 -- 172.21.15.97:0/956221571 >> v1:172.21.15.135:6789/0 conn(0x7f4c140d28a0 legacy=0x7f4c140d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.086+0000 7f4c13e00640 1 -- 172.21.15.97:0/956221571 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c140cfcb0 con 0x7f4c14161580 2024-03-06T07:38:01.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.087+0000 7f4c13e00640 1 --2- 172.21.15.97:0/956221571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c14161580 0x7f4c140d1950 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009870 tx=0x7f4c0402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2f1be536e64771 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.087+0000 7f4c12a00640 1 -- 172.21.15.97:0/956221571 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04034070 con 0x7f4c14161580 2024-03-06T07:38:01.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.087+0000 7f4c12a00640 1 -- 172.21.15.97:0/956221571 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04030840 con 0x7f4c14161580 2024-03-06T07:38:01.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.087+0000 7f4c12a00640 1 -- 172.21.15.97:0/956221571 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04030c10 con 0x7f4c14161580 2024-03-06T07:38:01.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 -- 172.21.15.97:0/956221571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c14161580 msgr2=0x7f4c140d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 --2- 172.21.15.97:0/956221571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c14161580 0x7f4c140d1950 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009870 tx=0x7f4c0402b980 comp rx=0 tx=0).stop 2024-03-06T07:38:01.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 -- 172.21.15.97:0/956221571 shutdown_connections 2024-03-06T07:38:01.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 --2- 172.21.15.97:0/956221571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c140d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 --2- 172.21.15.97:0/956221571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c14161580 0x7f4c140d1950 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 -- 172.21.15.97:0/956221571 >> 172.21.15.97:0/956221571 conn(0x7f4c140d3d80 msgr2=0x7f4c140d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 -- 172.21.15.97:0/956221571 shutdown_connections 2024-03-06T07:38:01.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.088+0000 7f4c19800640 1 -- 172.21.15.97:0/956221571 wait complete. 2024-03-06T07:38:01.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.091+0000 7f4c19800640 1 Processor -- start 2024-03-06T07:38:01.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.092+0000 7f4c19800640 1 -- start start 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c19800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c1414bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c19800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 0x7f4c1415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c19800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c14161580 0x7f4c1415d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c19800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140cf3a0 con 0x7f4c140d28a0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c19800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140b39a0 con 0x7f4c14161580 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c19800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140b4ed0 con 0x7f4c140d1e90 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c13400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 0x7f4c1415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c13400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 0x7f4c1415d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60552/0 (socket says 172.21.15.97:60552) 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c13400640 1 -- 172.21.15.97:0/3745395878 learned_addr learned my addr 172.21.15.97:0/3745395878 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.094+0000 7f4c13e00640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c1414bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c18e00640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c14161580 0x7f4c1415d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c13400640 1 -- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 msgr2=0x7f4c1414bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c13400640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c1414bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c13400640 1 -- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c14161580 msgr2=0x7f4c1415d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c13400640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c14161580 0x7f4c1415d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c13400640 1 -- 172.21.15.97:0/3745395878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c140b9450 con 0x7f4c140d28a0 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.095+0000 7f4c13400640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 0x7f4c1415d060 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000e930 tx=0x7f4c0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.096+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00017070 con 0x7f4c140d28a0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.096+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0000ca00 con 0x7f4c140d28a0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.096+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0000cdf0 con 0x7f4c140d28a0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.096+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c140b9730 con 0x7f4c140d28a0 2024-03-06T07:38:01.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.096+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1414c750 con 0x7f4c140d28a0 2024-03-06T07:38:01.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.097+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1414c950 con 0x7f4c140d28a0 2024-03-06T07:38:01.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.098+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c0001a070 con 0x7f4c140d28a0 2024-03-06T07:38:01.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.100+0000 7f4c0be00640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bf406b4d0 0x7f4bf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.100+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4c0008a560 con 0x7f4c140d28a0 2024-03-06T07:38:01.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.100+0000 7f4c13e00640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bf406b4d0 0x7f4bf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.101+0000 7f4c13e00640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bf406b4d0 0x7f4bf406d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4c04036000 tx=0x7f4c0403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.104+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c1414c950 con 0x7f4c140d28a0 2024-03-06T07:38:01.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.304+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c1404f330 con 0x7f4c140d28a0 2024-03-06T07:38:01.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.305+0000 7f4c0be00640 1 -- 172.21.15.97:0/3745395878 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f4c00058e30 con 0x7f4c140d28a0 2024-03-06T07:38:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:01.308 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":148,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:40.116487+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.39","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:38:01.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.308+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bf406b4d0 msgr2=0x7f4bf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.308+0000 7f4c19800640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bf406b4d0 0x7f4bf406d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4c04036000 tx=0x7f4c0403b040 comp rx=0 tx=0).stop 2024-03-06T07:38:01.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.308+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 msgr2=0x7f4c1415d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.308+0000 7f4c19800640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 0x7f4c1415d060 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000e930 tx=0x7f4c0000edf0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 shutdown_connections 2024-03-06T07:38:01.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4bf406b4d0 0x7f4bf406d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c14161580 0x7f4c1415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c140d28a0 0x7f4c1415d060 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 --2- 172.21.15.97:0/3745395878 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4c140d1e90 0x7f4c1414bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 >> 172.21.15.97:0/3745395878 conn(0x7f4c140d3d80 msgr2=0x7f4c1415e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 shutdown_connections 2024-03-06T07:38:01.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.309+0000 7f4c19800640 1 -- 172.21.15.97:0/3745395878 wait complete. 2024-03-06T07:38:01.327 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:37:19.579712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:37:20.835626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:38:01.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:38:01.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.527+0000 7f4cb9a00640 1 Processor -- start 2024-03-06T07:38:01.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.527+0000 7f4cb9a00640 1 -- start start 2024-03-06T07:38:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.528+0000 7f4cb9a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4cb40b3f70 0x7f4cb4160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.528+0000 7f4cb9a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4cb4070ff0 0x7f4cb40ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.528+0000 7f4cb9a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb4071440 con 0x7f4cb415fc10 2024-03-06T07:38:01.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.528+0000 7f4cb9a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40b46c0 con 0x7f4cb4070ff0 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.528+0000 7f4cb9a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40b49a0 con 0x7f4cb40b3f70 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4cb4070ff0 0x7f4cb40ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4cb4070ff0 0x7f4cb40ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60554/0 (socket says 172.21.15.97:60554) 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 -- 172.21.15.97:0/1789056906 learned_addr learned my addr 172.21.15.97:0/1789056906 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb2000640 1 --2- 172.21.15.97:0/1789056906 >> v2:172.21.15.135:3300/0 conn(0x7f4cb40b3f70 0x7f4cb4160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 -- 172.21.15.97:0/1789056906 >> v2:172.21.15.135:3300/0 conn(0x7f4cb40b3f70 msgr2=0x7f4cb4160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 --2- 172.21.15.97:0/1789056906 >> v2:172.21.15.135:3300/0 conn(0x7f4cb40b3f70 0x7f4cb4160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 -- 172.21.15.97:0/1789056906 >> v1:172.21.15.135:6789/0 conn(0x7f4cb415fc10 legacy=0x7f4cb415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb3400640 1 -- 172.21.15.97:0/1789056906 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4072c90 con 0x7f4cb4070ff0 2024-03-06T07:38:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.529+0000 7f4cb2000640 1 --2- 172.21.15.97:0/1789056906 >> v2:172.21.15.135:3300/0 conn(0x7f4cb40b3f70 0x7f4cb4160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:01.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.530+0000 7f4cb3400640 1 --2- 172.21.15.97:0/1789056906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb40ce4a0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4074a50 tx=0x7f4cac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e97a5217170696 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb1600640 1 -- 172.21.15.97:0/1789056906 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac02ca80 con 0x7f4cb4070ff0 2024-03-06T07:38:01.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb1600640 1 -- 172.21.15.97:0/1789056906 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac02cc20 con 0x7f4cb4070ff0 2024-03-06T07:38:01.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb1600640 1 -- 172.21.15.97:0/1789056906 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0343f0 con 0x7f4cb4070ff0 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb9a00640 1 -- 172.21.15.97:0/1789056906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 msgr2=0x7f4cb40ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/1789056906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb40ce4a0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4074a50 tx=0x7f4cac02bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb9a00640 1 -- 172.21.15.97:0/1789056906 shutdown_connections 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/1789056906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb40ce4a0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/1789056906 >> v2:172.21.15.135:3300/0 conn(0x7f4cb40b3f70 0x7f4cb4160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.531+0000 7f4cb9a00640 1 -- 172.21.15.97:0/1789056906 >> 172.21.15.97:0/1789056906 conn(0x7f4cb40b1940 msgr2=0x7f4cb40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.532+0000 7f4cb9a00640 1 -- 172.21.15.97:0/1789056906 shutdown_connections 2024-03-06T07:38:01.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.532+0000 7f4cb9a00640 1 -- 172.21.15.97:0/1789056906 wait complete. 2024-03-06T07:38:01.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.533+0000 7f4cb9a00640 1 Processor -- start 2024-03-06T07:38:01.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.533+0000 7f4cb9a00640 1 -- start start 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.534+0000 7f4cb9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb4152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.534+0000 7f4cb9a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb40b3f70 0x7f4cb41573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.534+0000 7f4cb9a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cb415fc10 0x7f4cb41578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.534+0000 7f4cb9a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4072a30 con 0x7f4cb4070ff0 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.534+0000 7f4cb9a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb40728b0 con 0x7f4cb40b3f70 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.534+0000 7f4cb9a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb415c2e0 con 0x7f4cb415fc10 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb4152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb4152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60568/0 (socket says 172.21.15.97:60568) 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 -- 172.21.15.97:0/178093783 learned_addr learned my addr 172.21.15.97:0/178093783 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb3400640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cb415fc10 0x7f4cb41578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2000640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb40b3f70 0x7f4cb41573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 -- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cb415fc10 msgr2=0x7f4cb41578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cb415fc10 0x7f4cb41578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 -- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb40b3f70 msgr2=0x7f4cb41573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb40b3f70 0x7f4cb41573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb2a00640 1 -- 172.21.15.97:0/178093783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4159b20 con 0x7f4cb4070ff0 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.535+0000 7f4cb3400640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cb415fc10 0x7f4cb41578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.536+0000 7f4cb2a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb4152260 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400b3e0 tx=0x7f4ca400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.536+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4015020 con 0x7f4cb4070ff0 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.536+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4011850 con 0x7f4cb4070ff0 2024-03-06T07:38:01.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.536+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4011c40 con 0x7f4cb4070ff0 2024-03-06T07:38:01.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.536+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4159e00 con 0x7f4cb4070ff0 2024-03-06T07:38:01.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.536+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb415a3c0 con 0x7f4cb4070ff0 2024-03-06T07:38:01.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.537+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb41598b0 con 0x7f4cb4070ff0 2024-03-06T07:38:01.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.538+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ca4018020 con 0x7f4cb4070ff0 2024-03-06T07:38:01.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.539+0000 7f4caaa00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c9406b4d0 0x7f4c9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.539+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4ca408a140 con 0x7f4cb4070ff0 2024-03-06T07:38:01.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.539+0000 7f4cb2000640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c9406b4d0 0x7f4c9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.540+0000 7f4cb2000640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c9406b4d0 0x7f4c9406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4cb415f7e0 tx=0x7f4ca0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.544+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ca401f020 con 0x7f4cb4070ff0 2024-03-06T07:38:01.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.742+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cb404f330 con 0x7f4cb4070ff0 2024-03-06T07:38:01.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.742+0000 7f4caaa00640 1 -- 172.21.15.97:0/178093783 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ca40172c0 con 0x7f4cb4070ff0 2024-03-06T07:38:01.744 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:38:01.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c9406b4d0 msgr2=0x7f4c9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c9406b4d0 0x7f4c9406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4cb415f7e0 tx=0x7f4ca0008040 comp rx=0 tx=0).stop 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 msgr2=0x7f4cb4152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb4152260 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400b3e0 tx=0x7f4ca400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 shutdown_connections 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4c9406b4d0 0x7f4c9406d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4cb415fc10 0x7f4cb41578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cb40b3f70 0x7f4cb41573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 --2- 172.21.15.97:0/178093783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4070ff0 0x7f4cb4152260 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 >> 172.21.15.97:0/178093783 conn(0x7f4cb40b1940 msgr2=0x7f4cb4158590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 shutdown_connections 2024-03-06T07:38:01.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:01.745+0000 7f4cb9a00640 1 -- 172.21.15.97:0/178093783 wait complete. 2024-03-06T07:38:01.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:38:02.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.010+0000 7f410a200640 1 Processor -- start 2024-03-06T07:38:02.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.010+0000 7f410a200640 1 -- start start 2024-03-06T07:38:02.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.011+0000 7f410a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f410415df80 0x7f410406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.011+0000 7f410a200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f410406b500 0x7f4104162320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.011+0000 7f410a200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41040b5c90 con 0x7f41040b01a0 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.011+0000 7f410a200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4104072a00 con 0x7f410406b500 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.011+0000 7f410a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41040d1820 con 0x7f410415df80 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4109800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f410406b500 0x7f4104162320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4103e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f41040b01a0 0x7f4104161c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44472/0 (socket says 172.21.15.97:44472) 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4103e00640 1 -- 172.21.15.97:0/3999038409 learned_addr learned my addr 172.21.15.97:0/3999038409 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4108e00640 1 --2- 172.21.15.97:0/3999038409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f410415df80 0x7f410406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4109800640 1 -- 172.21.15.97:0/3999038409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f410415df80 msgr2=0x7f410406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4109800640 1 --2- 172.21.15.97:0/3999038409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f410415df80 0x7f410406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4109800640 1 -- 172.21.15.97:0/3999038409 >> v1:172.21.15.97:6789/0 conn(0x7f41040b01a0 legacy=0x7f4104161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4109800640 1 -- 172.21.15.97:0/3999038409 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040d1540 con 0x7f410406b500 2024-03-06T07:38:02.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.012+0000 7f4103400640 1 -- 172.21.15.97:0/3999038409 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920301696 0 0) 0x7f41040b5c90 con 0x7f41040b01a0 2024-03-06T07:38:02.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.014+0000 7f4109800640 1 --2- 172.21.15.97:0/3999038409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410406b500 0x7f4104162320 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009d90 tx=0x7f40fc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713b9489b83d08a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.014+0000 7f4103400640 1 -- 172.21.15.97:0/3999038409 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc039070 con 0x7f410406b500 2024-03-06T07:38:02.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.014+0000 7f4103400640 1 -- 172.21.15.97:0/3999038409 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc02cc50 con 0x7f410406b500 2024-03-06T07:38:02.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.014+0000 7f4103400640 1 -- 172.21.15.97:0/3999038409 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc034440 con 0x7f410406b500 2024-03-06T07:38:02.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 -- 172.21.15.97:0/3999038409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410406b500 msgr2=0x7f4104162320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 --2- 172.21.15.97:0/3999038409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410406b500 0x7f4104162320 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009d90 tx=0x7f40fc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 -- 172.21.15.97:0/3999038409 shutdown_connections 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 --2- 172.21.15.97:0/3999038409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410406b500 0x7f4104162320 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 --2- 172.21.15.97:0/3999038409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f410415df80 0x7f410406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 -- 172.21.15.97:0/3999038409 >> 172.21.15.97:0/3999038409 conn(0x7f41040d0a20 msgr2=0x7f41040d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 -- 172.21.15.97:0/3999038409 shutdown_connections 2024-03-06T07:38:02.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.015+0000 7f410a200640 1 -- 172.21.15.97:0/3999038409 wait complete. 2024-03-06T07:38:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.017+0000 7f410a200640 1 Processor -- start 2024-03-06T07:38:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.018+0000 7f410a200640 1 -- start start 2024-03-06T07:38:02.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.019+0000 7f410a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f410406b500 0x7f4104155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.019+0000 7f410a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f41040b01a0 0x7f41040ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.019+0000 7f410a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 0x7f410414b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.019+0000 7f410a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040b5770 con 0x7f410415df80 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.019+0000 7f410a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040b5490 con 0x7f410406b500 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.019+0000 7f410a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410415e360 con 0x7f41040b01a0 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 0x7f410414b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 0x7f410414b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60578/0 (socket says 172.21.15.97:60578) 2024-03-06T07:38:02.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 -- 172.21.15.97:0/2557866748 learned_addr learned my addr 172.21.15.97:0/2557866748 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:02.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4108e00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f410406b500 0x7f4104155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4103e00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f41040b01a0 0x7f41040ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 -- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f41040b01a0 msgr2=0x7f41040ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f41040b01a0 0x7f41040ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 -- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f410406b500 msgr2=0x7f4104155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f410406b500 0x7f4104155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 -- 172.21.15.97:0/2557866748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410406f180 con 0x7f410415df80 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4103e00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f41040b01a0 0x7f41040ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4108e00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f410406b500 0x7f4104155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.020+0000 7f4109800640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 0x7f410414b440 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009d90 tx=0x7f40fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.021+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc044020 con 0x7f410415df80 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.021+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc03a040 con 0x7f410415df80 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.021+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc039050 con 0x7f410415df80 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.021+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104158750 con 0x7f410415df80 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.021+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4104158470 con 0x7f410415df80 2024-03-06T07:38:02.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.022+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410404f330 con 0x7f410415df80 2024-03-06T07:38:02.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.023+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40fc003990 con 0x7f410415df80 2024-03-06T07:38:02.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.025+0000 7f4100c00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40e406b4d0 0x7f40e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.025+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f40fc0ac680 con 0x7f410415df80 2024-03-06T07:38:02.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.025+0000 7f4108e00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40e406b4d0 0x7f40e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.026+0000 7f4108e00640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40e406b4d0 0x7f40e406d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f40f8006fd0 tx=0x7f40f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.029+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40fc04e060 con 0x7f410415df80 2024-03-06T07:38:02.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.308+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f41040702f0 con 0x7f410415df80 2024-03-06T07:38:02.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.308+0000 7f4100c00640 1 -- 172.21.15.97:0/2557866748 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40fc07b4d0 con 0x7f410415df80 2024-03-06T07:38:02.310 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:38:02.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40e406b4d0 msgr2=0x7f40e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40e406b4d0 0x7f40e406d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f40f8006fd0 tx=0x7f40f8008040 comp rx=0 tx=0).stop 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 msgr2=0x7f410414b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 0x7f410414b440 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009d90 tx=0x7f40fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 shutdown_connections 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f40e406b4d0 0x7f40e406d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f410415df80 0x7f410414b440 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f41040b01a0 0x7f41040ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 --2- 172.21.15.97:0/2557866748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f410406b500 0x7f4104155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 >> 172.21.15.97:0/2557866748 conn(0x7f41040d0a20 msgr2=0x7f4104156390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 shutdown_connections 2024-03-06T07:38:02.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.311+0000 7f410a200640 1 -- 172.21.15.97:0/2557866748 wait complete. 2024-03-06T07:38:02.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:38:02.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.551+0000 7fcbc6400640 1 Processor -- start 2024-03-06T07:38:02.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.551+0000 7fcbc6400640 1 -- start start 2024-03-06T07:38:02.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.552+0000 7fcbc6400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbc6400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcbc015fff0 0x7fcbc0160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbc6400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00b46c0 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbc6400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc00b49a0 con 0x7fcbc015fff0 2024-03-06T07:38:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbc6400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0072c90 con 0x7fcbc015fc10 2024-03-06T07:38:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbb7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcbc015fff0 0x7fcbc0160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbb7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcbc015fff0 0x7fcbc0160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60582/0 (socket says 172.21.15.97:60582) 2024-03-06T07:38:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbb7e00640 1 -- 172.21.15.97:0/2920829188 learned_addr learned my addr 172.21.15.97:0/2920829188 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.553+0000 7fcbbf400640 1 --2- 172.21.15.97:0/2920829188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.554+0000 7fcbb7e00640 1 -- 172.21.15.97:0/2920829188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 msgr2=0x7fcbc0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.554+0000 7fcbb7e00640 1 --2- 172.21.15.97:0/2920829188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.554+0000 7fcbb7e00640 1 -- 172.21.15.97:0/2920829188 >> v1:172.21.15.135:6789/0 conn(0x7fcbc00ce4a0 legacy=0x7fcbc00ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.554+0000 7fcbb7e00640 1 -- 172.21.15.97:0/2920829188 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc0160990 con 0x7fcbc015fff0 2024-03-06T07:38:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.555+0000 7fcbb7e00640 1 --2- 172.21.15.97:0/2920829188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc015fff0 0x7fcbc0160450 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009870 tx=0x7fcba802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2623ccfb5877cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.555+0000 7fcbbea00640 1 -- 172.21.15.97:0/2920829188 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8035020 con 0x7fcbc015fff0 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.555+0000 7fcbbea00640 1 -- 172.21.15.97:0/2920829188 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba8031640 con 0x7fcbc015fff0 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.555+0000 7fcbbea00640 1 -- 172.21.15.97:0/2920829188 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba802d040 con 0x7fcbc015fff0 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 -- 172.21.15.97:0/2920829188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc015fff0 msgr2=0x7fcbc0160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 --2- 172.21.15.97:0/2920829188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc015fff0 0x7fcbc0160450 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009870 tx=0x7fcba802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 -- 172.21.15.97:0/2920829188 shutdown_connections 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 --2- 172.21.15.97:0/2920829188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc015fff0 0x7fcbc0160450 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 --2- 172.21.15.97:0/2920829188 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 -- 172.21.15.97:0/2920829188 >> 172.21.15.97:0/2920829188 conn(0x7fcbc00b1940 msgr2=0x7fcbc00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 -- 172.21.15.97:0/2920829188 shutdown_connections 2024-03-06T07:38:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.556+0000 7fcbc6400640 1 -- 172.21.15.97:0/2920829188 wait complete. 2024-03-06T07:38:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.558+0000 7fcbc6400640 1 Processor -- start 2024-03-06T07:38:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.558+0000 7fcbc6400640 1 -- start start 2024-03-06T07:38:02.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbc6400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 0x7fcbc014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbc6400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbc6400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc015fff0 0x7fcbc00b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbc6400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0072a30 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbc6400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00728b0 con 0x7fcbc015fff0 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbc6400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc015c2e0 con 0x7fcbc015fc10 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbbf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 0x7fcbc014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbbf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 0x7fcbc014e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60586/0 (socket says 172.21.15.97:60586) 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.559+0000 7fcbbf400640 1 -- 172.21.15.97:0/3024905336 learned_addr learned my addr 172.21.15.97:0/3024905336 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbbf400640 1 -- 172.21.15.97:0/3024905336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 msgr2=0x7fcbc0150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbbf400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbbf400640 1 -- 172.21.15.97:0/3024905336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc015fff0 msgr2=0x7fcbc00b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbbf400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc015fff0 0x7fcbc00b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbbf400640 1 -- 172.21.15.97:0/3024905336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc014f0c0 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbbf400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 0x7fcbc014e770 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000d800 tx=0x7fcbb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000b6c0 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb0019070 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000ba80 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.560+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc014f3a0 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.561+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc014f960 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.562+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc00b2a10 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.563+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbb000bc20 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.563+0000 7fcbb7400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.563+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcbb008a6f0 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.564+0000 7fcbb7e00640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.564+0000 7fcbb7e00640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcba8036000 tx=0x7fcba8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.573+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbc00b2a10 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.769+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcbc004f330 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.769+0000 7fcbb7400640 1 -- 172.21.15.97:0/3024905336 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbb0059540 con 0x7fcbc00ce4a0 2024-03-06T07:38:02.771 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb9c06b4d0 msgr2=0x7fcb9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcba8036000 tx=0x7fcba8030000 comp rx=0 tx=0).stop 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 msgr2=0x7fcbc014e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 0x7fcbc014e770 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000d800 tx=0x7fcbb000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 shutdown_connections 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc015fff0 0x7fcbc00b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcbc015fc10 0x7fcbc0150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 --2- 172.21.15.97:0/3024905336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbc00ce4a0 0x7fcbc014e770 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 >> 172.21.15.97:0/3024905336 conn(0x7fcbc00b1940 msgr2=0x7fcbc00acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 shutdown_connections 2024-03-06T07:38:02.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:02.772+0000 7fcbc6400640 1 -- 172.21.15.97:0/3024905336 wait complete. 2024-03-06T07:38:02.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:38:03.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.035+0000 7f76ada00640 1 Processor -- start 2024-03-06T07:38:03.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.035+0000 7f76ada00640 1 -- start start 2024-03-06T07:38:03.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.037+0000 7f76ada00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f76a80d27a0 0x7f76a80b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.037+0000 7f76ada00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8161960 con 0x7f76a80d2360 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.037+0000 7f76ada00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a80cf6f0 con 0x7f76a8161580 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.037+0000 7f76ada00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a80cf9d0 con 0x7f76a80d27a0 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.037+0000 7f769f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f76a80d2360 0x7f76a80b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44486/0 (socket says 172.21.15.97:44486) 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.037+0000 7f769f400640 1 -- 172.21.15.97:0/1562277155 learned_addr learned my addr 172.21.15.97:0/1562277155 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76ad000640 1 --2- 172.21.15.97:0/1562277155 >> v2:172.21.15.135:3300/0 conn(0x7f76a80d27a0 0x7f76a80b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463556557 0 0) 0x7f76a8161960 con 0x7f76a80d2360 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c003520 con 0x7f76a80d2360 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355807723 0 0) 0x7f768c003520 con 0x7f76a80d2360 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 >> v2:172.21.15.135:3300/0 conn(0x7f76a80d27a0 msgr2=0x7f76a80b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 --2- 172.21.15.97:0/1562277155 >> v2:172.21.15.135:3300/0 conn(0x7f76a80d27a0 0x7f76a80b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 >> v1:172.21.15.135:6789/0 conn(0x7f76a8161580 legacy=0x7f76a80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a80cfcb0 con 0x7f76a80d2360 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7690001670 con 0x7f76a80d2360 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 >> v1:172.21.15.97:6789/0 conn(0x7f76a80d2360 legacy=0x7f76a80b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c0039b0 0x7f768c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f768c004800 0x7f768c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768c0051c0 0x7f768c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80cf9d0 con 0x7f768c004800 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c00c060 con 0x7f768c0051c0 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.038+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c00c260 con 0x7f768c0039b0 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f769f400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f768c004800 0x7f768c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f76a7e00640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c0039b0 0x7f768c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f769f400640 1 -- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c0039b0 msgr2=0x7f768c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f769f400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c0039b0 0x7f768c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f769f400640 1 -- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768c0051c0 msgr2=0x7f768c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f769f400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768c0051c0 0x7f768c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f769f400640 1 -- 172.21.15.97:0/1562277155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c003520 con 0x7f768c004800 2024-03-06T07:38:03.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f76ad000640 1 -- 172.21.15.97:0/1562277155 reap_dead start 2024-03-06T07:38:03.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.039+0000 7f76a7e00640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c0039b0 0x7f768c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:03.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.040+0000 7f769f400640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f768c004800 0x7f768c004c80 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f769000c240 tx=0x7f768c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a81159fd761cf26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.040+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690007d10 con 0x7f768c004800 2024-03-06T07:38:03.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.040+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7690030070 con 0x7f768c004800 2024-03-06T07:38:03.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.040+0000 7f76a7400640 1 -- 172.21.15.97:0/1562277155 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769002cda0 con 0x7f768c004800 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 -- 172.21.15.97:0/1562277155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f768c004800 msgr2=0x7f768c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f768c004800 0x7f768c004c80 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f769000c240 tx=0x7f768c005830 comp rx=0 tx=0).stop 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 -- 172.21.15.97:0/1562277155 shutdown_connections 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 --2- 172.21.15.97:0/1562277155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f768c004800 0x7f768c004c80 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 -- 172.21.15.97:0/1562277155 >> 172.21.15.97:0/1562277155 conn(0x7f76a80d3d80 msgr2=0x7f76a80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 -- 172.21.15.97:0/1562277155 shutdown_connections 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.041+0000 7f76ada00640 1 -- 172.21.15.97:0/1562277155 wait complete. 2024-03-06T07:38:03.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.043+0000 7f76ada00640 1 Processor -- start 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.043+0000 7f76ada00640 1 -- start start 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.044+0000 7f76ada00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c004800 0x7f76a806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.044+0000 7f76ada00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 0x7f76a80b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.044+0000 7f76ada00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76a80b6eb0 0x7f76a80b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.044+0000 7f76ada00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80d2bf0 con 0x7f76a81561b0 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.044+0000 7f76ada00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80b39a0 con 0x7f76a80b6eb0 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.044+0000 7f76ada00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80cf3a0 con 0x7f768c004800 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 0x7f76a80b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 0x7f76a80b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60602/0 (socket says 172.21.15.97:60602) 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 -- 172.21.15.97:0/1610733690 learned_addr learned my addr 172.21.15.97:0/1610733690 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:03.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f76a7e00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c004800 0x7f76a806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f76ad000640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76a80b6eb0 0x7f76a80b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 -- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c004800 msgr2=0x7f76a806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c004800 0x7f76a806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 -- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76a80b6eb0 msgr2=0x7f76a80b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76a80b6eb0 0x7f76a80b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f769f400640 1 -- 172.21.15.97:0/1610733690 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a806d530 con 0x7f76a81561b0 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.045+0000 7f76ad000640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76a80b6eb0 0x7f76a80b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.046+0000 7f769f400640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 0x7f76a80b6970 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f769000c360 tx=0x7f769002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.046+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690007d10 con 0x7f76a81561b0 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.046+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7690004bf0 con 0x7f76a81561b0 2024-03-06T07:38:03.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.046+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a80c7760 con 0x7f76a81561b0 2024-03-06T07:38:03.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.046+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690007070 con 0x7f76a81561b0 2024-03-06T07:38:03.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.046+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a80c7d20 con 0x7f76a81561b0 2024-03-06T07:38:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.048+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a804f330 con 0x7f76a81561b0 2024-03-06T07:38:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.048+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7690048020 con 0x7f76a81561b0 2024-03-06T07:38:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.049+0000 7f76a4c00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f768406b500 0x7f768406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.050+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f76900ab740 con 0x7f76a81561b0 2024-03-06T07:38:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.050+0000 7f76a7e00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f768406b500 0x7f768406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.051+0000 7f76a7e00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f768406b500 0x7f768406d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7698007fd0 tx=0x7f7698009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.054+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f769007a500 con 0x7f76a81561b0 2024-03-06T07:38:03.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.269+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f76a80b7290 con 0x7f76a81561b0 2024-03-06T07:38:03.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.270+0000 7f76a4c00640 1 -- 172.21.15.97:0/1610733690 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v148) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7690034e80 con 0x7f76a81561b0 2024-03-06T07:38:03.272 INFO:teuthology.orchestra.run.smithi097.stderr:listed 0 entries 2024-03-06T07:38:03.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f768406b500 msgr2=0x7f768406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f768406b500 0x7f768406d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7698007fd0 tx=0x7f7698009040 comp rx=0 tx=0).stop 2024-03-06T07:38:03.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 msgr2=0x7f76a80b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 0x7f76a80b6970 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f769000c360 tx=0x7f769002e070 comp rx=0 tx=0).stop 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 shutdown_connections 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f768406b500 0x7f768406d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76a80b6eb0 0x7f76a80b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76a81561b0 0x7f76a80b6970 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 --2- 172.21.15.97:0/1610733690 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f768c004800 0x7f76a806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 >> 172.21.15.97:0/1610733690 conn(0x7f76a80d3d80 msgr2=0x7f76a80c6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 shutdown_connections 2024-03-06T07:38:03.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.273+0000 7f76ada00640 1 -- 172.21.15.97:0/1610733690 wait complete. 2024-03-06T07:38:03.293 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:38:03.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:03.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:38:03.381 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:38:03.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:38:03.381 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:38:03.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:38:03.699 INFO:teuthology.orchestra.run.smithi097.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:38:03.699 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:38:03.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:38:03.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.905+0000 7f6843e00640 1 Processor -- start 2024-03-06T07:38:03.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.905+0000 7f6843e00640 1 -- start start 2024-03-06T07:38:03.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.906+0000 7f6843e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f68440d1ea0 0x7f68440d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.906+0000 7f6843e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68440c7970 0x7f68440c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440d13e0 con 0x7f684415e130 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440d2840 con 0x7f68440c7970 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440cf120 con 0x7f68440d1ea0 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68440c7970 0x7f68440c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68440c7970 0x7f68440c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35008/0 (socket says 172.21.15.97:35008) 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 -- 172.21.15.97:0/2396561788 learned_addr learned my addr 172.21.15.97:0/2396561788 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6842000640 1 --2- 172.21.15.97:0/2396561788 >> v2:172.21.15.135:3300/0 conn(0x7f68440d1ea0 0x7f68440d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 -- 172.21.15.97:0/2396561788 >> v2:172.21.15.135:3300/0 conn(0x7f68440d1ea0 msgr2=0x7f68440d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 --2- 172.21.15.97:0/2396561788 >> v2:172.21.15.135:3300/0 conn(0x7f68440d1ea0 0x7f68440d2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 -- 172.21.15.97:0/2396561788 >> v1:172.21.15.135:6789/0 conn(0x7f684415e130 legacy=0x7f68440cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:03.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.907+0000 7f6843400640 1 -- 172.21.15.97:0/2396561788 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68440cead0 con 0x7f68440c7970 2024-03-06T07:38:03.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.908+0000 7f6843400640 1 --2- 172.21.15.97:0/2396561788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68440c7970 0x7f68440c7e00 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f68440b9ed0 tx=0x7f683c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf54b80708d5132 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.909+0000 7f6841600640 1 -- 172.21.15.97:0/2396561788 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02ca80 con 0x7f68440c7970 2024-03-06T07:38:03.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.909+0000 7f6841600640 1 -- 172.21.15.97:0/2396561788 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c02cc20 con 0x7f68440c7970 2024-03-06T07:38:03.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.909+0000 7f6841600640 1 -- 172.21.15.97:0/2396561788 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0343f0 con 0x7f68440c7970 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 -- 172.21.15.97:0/2396561788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68440c7970 msgr2=0x7f68440c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 --2- 172.21.15.97:0/2396561788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68440c7970 0x7f68440c7e00 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f68440b9ed0 tx=0x7f683c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 -- 172.21.15.97:0/2396561788 shutdown_connections 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 --2- 172.21.15.97:0/2396561788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68440c7970 0x7f68440c7e00 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 --2- 172.21.15.97:0/2396561788 >> v2:172.21.15.135:3300/0 conn(0x7f68440d1ea0 0x7f68440d2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 -- 172.21.15.97:0/2396561788 >> 172.21.15.97:0/2396561788 conn(0x7f684409d070 msgr2=0x7f68440d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 -- 172.21.15.97:0/2396561788 shutdown_connections 2024-03-06T07:38:03.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.910+0000 7f6843e00640 1 -- 172.21.15.97:0/2396561788 wait complete. 2024-03-06T07:38:03.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.911+0000 7f6843e00640 1 Processor -- start 2024-03-06T07:38:03.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.912+0000 7f6843e00640 1 -- start start 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.912+0000 7f6843e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68440c7970 0x7f684414c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68440d1ea0 0x7f68440b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 0x7f68440ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68440cfcc0 con 0x7f684415e130 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68440c8340 con 0x7f68440c7970 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68440d16c0 con 0x7f68440d1ea0 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 0x7f68440ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 0x7f68440ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35016/0 (socket says 172.21.15.97:35016) 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843400640 1 -- 172.21.15.97:0/1034022926 learned_addr learned my addr 172.21.15.97:0/1034022926 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6842a00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68440c7970 0x7f684414c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6842000640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68440d1ea0 0x7f68440b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843400640 1 -- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68440d1ea0 msgr2=0x7f68440b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.913+0000 7f6843400640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68440d1ea0 0x7f68440b5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f6843400640 1 -- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68440c7970 msgr2=0x7f684414c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f6843400640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68440c7970 0x7f684414c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f6843400640 1 -- 172.21.15.97:0/1034022926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684415cf10 con 0x7f684415e130 2024-03-06T07:38:03.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f6843400640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 0x7f68440ccfa0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f683c009f40 tx=0x7f683c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c043020 con 0x7f684415e130 2024-03-06T07:38:03.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c03e450 con 0x7f684415e130 2024-03-06T07:38:03.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684415f770 con 0x7f684415e130 2024-03-06T07:38:03.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.914+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c03e820 con 0x7f684415e130 2024-03-06T07:38:03.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.915+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684415c670 con 0x7f684415e130 2024-03-06T07:38:03.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.916+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684404f330 con 0x7f684415e130 2024-03-06T07:38:03.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.917+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f683c046020 con 0x7f684415e130 2024-03-06T07:38:03.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.918+0000 7f683b400640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f682406b4d0 0x7f682406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.918+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f683c0acf50 con 0x7f684415e130 2024-03-06T07:38:03.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.918+0000 7f6842a00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f682406b4d0 0x7f682406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.919+0000 7f6842a00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f682406b4d0 0x7f682406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6834007fd0 tx=0x7f6834009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:03.922+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f683c04d020 con 0x7f684415e130 2024-03-06T07:38:04.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.167+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f684409d690 con 0x7f684415e130 2024-03-06T07:38:04.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.690+0000 7f683b400640 1 -- 172.21.15.97:0/1034022926 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v114) v1 ==== 105+0+0 (secure 0 0 0) 0x7f684415cf10 con 0x7f684415e130 2024-03-06T07:38:04.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f682406b4d0 msgr2=0x7f682406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f682406b4d0 0x7f682406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6834007fd0 tx=0x7f6834009040 comp rx=0 tx=0).stop 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 msgr2=0x7f68440ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 0x7f68440ccfa0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f683c009f40 tx=0x7f683c004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 shutdown_connections 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f682406b4d0 0x7f682406d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f684415e130 0x7f68440ccfa0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68440d1ea0 0x7f68440b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 --2- 172.21.15.97:0/1034022926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68440c7970 0x7f684414c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 >> 172.21.15.97:0/1034022926 conn(0x7f684409d070 msgr2=0x7f6844162310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 shutdown_connections 2024-03-06T07:38:04.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.694+0000 7f6843e00640 1 -- 172.21.15.97:0/1034022926 wait complete. 2024-03-06T07:38:04.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:04.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.927+0000 7f1ba8000640 1 Processor -- start 2024-03-06T07:38:04.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.927+0000 7f1ba8000640 1 -- start start 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.930+0000 7f1ba8000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1ba0144c00 0x7f1ba006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.930+0000 7f1ba8000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1ba015f9d0 0x7f1ba015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.930+0000 7f1ba8000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba006ee70 con 0x7f1ba00ce0a0 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.930+0000 7f1ba8000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba006cef0 con 0x7f1ba015f9d0 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.930+0000 7f1ba8000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba006d1d0 con 0x7f1ba0144c00 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.930+0000 7f1ba4c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1ba015f9d0 0x7f1ba015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba4c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1ba015f9d0 0x7f1ba015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35030/0 (socket says 172.21.15.97:35030) 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba4c00640 1 -- 172.21.15.97:0/1413361926 learned_addr learned my addr 172.21.15.97:0/1413361926 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba5600640 1 --2- 172.21.15.97:0/1413361926 >> v2:172.21.15.135:3300/0 conn(0x7f1ba0144c00 0x7f1ba006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba4c00640 1 -- 172.21.15.97:0/1413361926 >> v2:172.21.15.135:3300/0 conn(0x7f1ba0144c00 msgr2=0x7f1ba006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba4c00640 1 --2- 172.21.15.97:0/1413361926 >> v2:172.21.15.135:3300/0 conn(0x7f1ba0144c00 0x7f1ba006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba4c00640 1 -- 172.21.15.97:0/1413361926 >> v1:172.21.15.135:6789/0 conn(0x7f1ba00ce0a0 legacy=0x7f1ba00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.931+0000 7f1ba4c00640 1 -- 172.21.15.97:0/1413361926 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba006d4b0 con 0x7f1ba015f9d0 2024-03-06T07:38:04.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.932+0000 7f1ba4c00640 1 --2- 172.21.15.97:0/1413361926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba015f9d0 0x7f1ba015fe60 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00b25f0 tx=0x7f1b8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6128ca4ef5525218 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.932+0000 7f1b9fe00640 1 -- 172.21.15.97:0/1413361926 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c035020 con 0x7f1ba015f9d0 2024-03-06T07:38:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.932+0000 7f1b9fe00640 1 -- 172.21.15.97:0/1413361926 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c031870 con 0x7f1ba015f9d0 2024-03-06T07:38:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1b9fe00640 1 -- 172.21.15.97:0/1413361926 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c031c40 con 0x7f1ba015f9d0 2024-03-06T07:38:04.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 -- 172.21.15.97:0/1413361926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba015f9d0 msgr2=0x7f1ba015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 --2- 172.21.15.97:0/1413361926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba015f9d0 0x7f1ba015fe60 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00b25f0 tx=0x7f1b8c02b910 comp rx=0 tx=0).stop 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 -- 172.21.15.97:0/1413361926 shutdown_connections 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 --2- 172.21.15.97:0/1413361926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba015f9d0 0x7f1ba015fe60 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 --2- 172.21.15.97:0/1413361926 >> v2:172.21.15.135:3300/0 conn(0x7f1ba0144c00 0x7f1ba006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 -- 172.21.15.97:0/1413361926 >> 172.21.15.97:0/1413361926 conn(0x7f1ba00b1730 msgr2=0x7f1ba00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 -- 172.21.15.97:0/1413361926 shutdown_connections 2024-03-06T07:38:04.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.933+0000 7f1ba8000640 1 -- 172.21.15.97:0/1413361926 wait complete. 2024-03-06T07:38:04.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.935+0000 7f1ba8000640 1 Processor -- start 2024-03-06T07:38:04.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.935+0000 7f1ba8000640 1 -- start start 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba8000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1ba00ce0a0 0x7f1ba01392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba8000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 0x7f1ba013d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba8000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba015f9d0 0x7f1ba0142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba8000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba006cb50 con 0x7f1ba0144c00 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba8000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba0160730 con 0x7f1ba015f9d0 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba8000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba006ccd0 con 0x7f1ba00ce0a0 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 0x7f1ba013d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 0x7f1ba013d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35038/0 (socket says 172.21.15.97:35038) 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba4c00640 1 -- 172.21.15.97:0/2018185335 learned_addr learned my addr 172.21.15.97:0/2018185335 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.936+0000 7f1ba5600640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1ba00ce0a0 0x7f1ba01392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba6000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba015f9d0 0x7f1ba0142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba4c00640 1 -- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1ba00ce0a0 msgr2=0x7f1ba01392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba4c00640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1ba00ce0a0 0x7f1ba01392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba4c00640 1 -- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba015f9d0 msgr2=0x7f1ba0142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba4c00640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba015f9d0 0x7f1ba0142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba4c00640 1 -- 172.21.15.97:0/2018185335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba014f540 con 0x7f1ba0144c00 2024-03-06T07:38:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba4c00640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 0x7f1ba013d4c0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c000c00 tx=0x7f1b8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c035040 con 0x7f1ba0144c00 2024-03-06T07:38:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c045480 con 0x7f1ba0144c00 2024-03-06T07:38:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.937+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba014f7e0 con 0x7f1ba0144c00 2024-03-06T07:38:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.938+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c045870 con 0x7f1ba0144c00 2024-03-06T07:38:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.938+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba014fda0 con 0x7f1ba0144c00 2024-03-06T07:38:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.939+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba004f330 con 0x7f1ba0144c00 2024-03-06T07:38:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.940+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b8c038020 con 0x7f1ba0144c00 2024-03-06T07:38:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.941+0000 7f1b9d600640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1b8006b4d0 0x7f1b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.941+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1b8c0ac7a0 con 0x7f1ba0144c00 2024-03-06T07:38:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.942+0000 7f1ba5600640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1b8006b4d0 0x7f1b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.942+0000 7f1ba5600640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1b8006b4d0 0x7f1b8006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400a970 tx=0x7f1b94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:04.946+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ba006ee70 con 0x7f1ba0144c00 2024-03-06T07:38:05.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.181+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba0072890 con 0x7f1ba0144c00 2024-03-06T07:38:05.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.182+0000 7f1b9d600640 1 -- 172.21.15.97:0/2018185335 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f1b8c07b5f0 con 0x7f1ba0144c00 2024-03-06T07:38:05.183 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:05.183 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:05.183 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1b8006b4d0 msgr2=0x7f1b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1b8006b4d0 0x7f1b8006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400a970 tx=0x7f1b94008040 comp rx=0 tx=0).stop 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 msgr2=0x7f1ba013d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 0x7f1ba013d4c0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c000c00 tx=0x7f1b8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 shutdown_connections 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1b8006b4d0 0x7f1b8006d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba015f9d0 0x7f1ba0142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ba0144c00 0x7f1ba013d4c0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 --2- 172.21.15.97:0/2018185335 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1ba00ce0a0 0x7f1ba01392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 >> 172.21.15.97:0/2018185335 conn(0x7f1ba00b1730 msgr2=0x7f1ba0145370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.184+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 shutdown_connections 2024-03-06T07:38:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:05.185+0000 7f1ba8000640 1 -- 172.21.15.97:0/2018185335 wait complete. 2024-03-06T07:38:05.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:05.406 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:06.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:06.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.630+0000 7f7c03200640 1 Processor -- start 2024-03-06T07:38:06.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.630+0000 7f7c03200640 1 -- start start 2024-03-06T07:38:06.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.631+0000 7f7c03200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7c040d2360 0x7f7c040d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.631+0000 7f7c03200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7c040b3ec0 0x7f7c040b4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.631+0000 7f7c03200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c04161960 con 0x7f7c04161580 2024-03-06T07:38:06.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.631+0000 7f7c03200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c040d2ce0 con 0x7f7c040d2360 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.631+0000 7f7c03200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c040cf6f0 con 0x7f7c040b3ec0 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7c040d2360 0x7f7c040d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7c040d2360 0x7f7c040d27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35054/0 (socket says 172.21.15.97:35054) 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 -- 172.21.15.97:0/4232739325 learned_addr learned my addr 172.21.15.97:0/4232739325 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7c02800640 1 --2- 172.21.15.97:0/4232739325 >> v2:172.21.15.135:3300/0 conn(0x7f7c040b3ec0 0x7f7c040b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 -- 172.21.15.97:0/4232739325 >> v2:172.21.15.135:3300/0 conn(0x7f7c040b3ec0 msgr2=0x7f7c040b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 --2- 172.21.15.97:0/4232739325 >> v2:172.21.15.135:3300/0 conn(0x7f7c040b3ec0 0x7f7c040b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 -- 172.21.15.97:0/4232739325 >> v1:172.21.15.135:6789/0 conn(0x7f7c04161580 legacy=0x7f7c040d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.632+0000 7f7bf9400640 1 -- 172.21.15.97:0/4232739325 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c040cf9d0 con 0x7f7c040d2360 2024-03-06T07:38:06.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.634+0000 7f7bf9400640 1 --2- 172.21.15.97:0/4232739325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c040d2360 0x7f7c040d27a0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7c040d2e60 tx=0x7f7bec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5132b6026c07745 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.634+0000 7f7c01400640 1 -- 172.21.15.97:0/4232739325 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec035020 con 0x7f7c040d2360 2024-03-06T07:38:06.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.634+0000 7f7c01400640 1 -- 172.21.15.97:0/4232739325 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec031870 con 0x7f7c040d2360 2024-03-06T07:38:06.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.634+0000 7f7c01400640 1 -- 172.21.15.97:0/4232739325 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec031c40 con 0x7f7c040d2360 2024-03-06T07:38:06.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 -- 172.21.15.97:0/4232739325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c040d2360 msgr2=0x7f7c040d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 --2- 172.21.15.97:0/4232739325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c040d2360 0x7f7c040d27a0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7c040d2e60 tx=0x7f7bec02b910 comp rx=0 tx=0).stop 2024-03-06T07:38:06.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 -- 172.21.15.97:0/4232739325 shutdown_connections 2024-03-06T07:38:06.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 --2- 172.21.15.97:0/4232739325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c040d2360 0x7f7c040d27a0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 --2- 172.21.15.97:0/4232739325 >> v2:172.21.15.135:3300/0 conn(0x7f7c040b3ec0 0x7f7c040b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 -- 172.21.15.97:0/4232739325 >> 172.21.15.97:0/4232739325 conn(0x7f7c040d3d80 msgr2=0x7f7c040d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:06.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 -- 172.21.15.97:0/4232739325 shutdown_connections 2024-03-06T07:38:06.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.635+0000 7f7c03200640 1 -- 172.21.15.97:0/4232739325 wait complete. 2024-03-06T07:38:06.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.637+0000 7f7c03200640 1 Processor -- start 2024-03-06T07:38:06.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.637+0000 7f7c03200640 1 -- start start 2024-03-06T07:38:06.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c03200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c040b3ec0 0x7f7c040cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c03200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c040d2360 0x7f7c040b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c03200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 0x7f7c040b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c03200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c040b39a0 con 0x7f7c04161580 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c03200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c040b36c0 con 0x7f7c040b3ec0 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c03200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c040cf3a0 con 0x7f7c040d2360 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 0x7f7c040b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 0x7f7c040b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35070/0 (socket says 172.21.15.97:35070) 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 -- 172.21.15.97:0/2887350417 learned_addr learned my addr 172.21.15.97:0/2887350417 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7bf9400640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c040d2360 0x7f7c040b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 -- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c040d2360 msgr2=0x7f7c040b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c01e00640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c040b3ec0 0x7f7c040cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c040d2360 0x7f7c040b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 -- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c040b3ec0 msgr2=0x7f7c040cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c040b3ec0 0x7f7c040cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c02800640 1 -- 172.21.15.97:0/2887350417 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c040cb0a0 con 0x7f7c04161580 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.638+0000 7f7c01e00640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c040b3ec0 0x7f7c040cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:06.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.639+0000 7f7c02800640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 0x7f7c040b21e0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00cc30 tx=0x7f7bfc00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.639+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00ce40 con 0x7f7c04161580 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.639+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc003990 con 0x7f7c04161580 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.639+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc003d40 con 0x7f7c04161580 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.639+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c040cb380 con 0x7f7c04161580 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.639+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c04153440 con 0x7f7c04161580 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.640+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0404f330 con 0x7f7c04161580 2024-03-06T07:38:06.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.641+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bfc003ee0 con 0x7f7c04161580 2024-03-06T07:38:06.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.643+0000 7f7bfaa00640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7be006b500 0x7f7be006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.643+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7bfc08a790 con 0x7f7c04161580 2024-03-06T07:38:06.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.643+0000 7f7c01e00640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7be006b500 0x7f7be006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.644+0000 7f7c01e00640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7be006b500 0x7f7be006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400a830 tx=0x7f7bf4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.647+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bfc014030 con 0x7f7c04161580 2024-03-06T07:38:06.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.880+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c040d3b60 con 0x7f7c04161580 2024-03-06T07:38:06.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.881+0000 7f7bfaa00640 1 -- 172.21.15.97:0/2887350417 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f7bfc0594c0 con 0x7f7c04161580 2024-03-06T07:38:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:06.882 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:06.883 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.883+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7be006b500 msgr2=0x7f7be006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7be006b500 0x7f7be006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400a830 tx=0x7f7bf4008040 comp rx=0 tx=0).stop 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 msgr2=0x7f7c040b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 0x7f7c040b21e0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00cc30 tx=0x7f7bfc00a5c0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 shutdown_connections 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7be006b500 0x7f7be006d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c04161580 0x7f7c040b21e0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c040d2360 0x7f7c040b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 --2- 172.21.15.97:0/2887350417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c040b3ec0 0x7f7c040cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 >> 172.21.15.97:0/2887350417 conn(0x7f7c040d3d80 msgr2=0x7f7c0415e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:06.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 shutdown_connections 2024-03-06T07:38:06.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:06.884+0000 7f7c03200640 1 -- 172.21.15.97:0/2887350417 wait complete. 2024-03-06T07:38:07.104 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:07.104 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:08.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:08.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.315+0000 7f2509400640 1 Processor -- start 2024-03-06T07:38:08.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.316+0000 7f2509400640 1 -- start start 2024-03-06T07:38:08.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.317+0000 7f2509400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2504144c00 0x7f250406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.317+0000 7f2509400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250406ee70 con 0x7f250415f9d0 2024-03-06T07:38:08.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.317+0000 7f2509400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250406cef0 con 0x7f250415fe60 2024-03-06T07:38:08.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.317+0000 7f2509400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250406d1d0 con 0x7f2504144c00 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f24f9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f250415f9d0 0x7f25040ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52010/0 (socket says 172.21.15.97:52010) 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f24f9e00640 1 -- 172.21.15.97:0/715191501 learned_addr learned my addr 172.21.15.97:0/715191501 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2502800640 1 --2- 172.21.15.97:0/715191501 >> v2:172.21.15.135:3300/0 conn(0x7f2504144c00 0x7f250406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172161988 0 0) 0x7f24ec001110 con 0x7f250415f9d0 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e8003520 con 0x7f250415f9d0 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209497712 0 0) 0x7f24e8003520 con 0x7f250415f9d0 2024-03-06T07:38:08.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 >> v2:172.21.15.135:3300/0 conn(0x7f2504144c00 msgr2=0x7f250406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 --2- 172.21.15.97:0/715191501 >> v2:172.21.15.135:3300/0 conn(0x7f2504144c00 0x7f250406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 >> v1:172.21.15.135:6789/0 conn(0x7f250415fe60 legacy=0x7f25040ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250406d4b0 con 0x7f250415f9d0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f24ec001840 con 0x7f250415f9d0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 >> v1:172.21.15.97:6789/0 conn(0x7f250415f9d0 legacy=0x7f25040ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.318+0000 7f2501e00640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2501e00640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250406d1d0 con 0x7f24e80051c0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e800c060 con 0x7f24e80039b0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e800c260 con 0x7f24e8004800 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2503200640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2502800640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f24f9e00640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2503200640 1 -- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24e8004800 msgr2=0x7f24e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:08.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2503200640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2503200640 1 -- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24e80039b0 msgr2=0x7f24e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:08.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2503200640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2503200640 1 -- 172.21.15.97:0/715191501 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e8003520 con 0x7f24e80051c0 2024-03-06T07:38:08.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.319+0000 7f2502800640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:08.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.320+0000 7f2503200640 1 -- 172.21.15.97:0/715191501 reap_dead start 2024-03-06T07:38:08.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.320+0000 7f2503200640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f24e800b9c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f24fc001750 tx=0x7f24e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92af57202046f141 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:08.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc001370 con 0x7f24e80051c0 2024-03-06T07:38:08.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc0052f0 con 0x7f24e80051c0 2024-03-06T07:38:08.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2501e00640 1 -- 172.21.15.97:0/715191501 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc003830 con 0x7f24e80051c0 2024-03-06T07:38:08.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 -- 172.21.15.97:0/715191501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 msgr2=0x7f24e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f24e800b9c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f24fc001750 tx=0x7f24e8005830 comp rx=0 tx=0).stop 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 -- 172.21.15.97:0/715191501 shutdown_connections 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 --2- 172.21.15.97:0/715191501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 -- 172.21.15.97:0/715191501 >> 172.21.15.97:0/715191501 conn(0x7f25040b1730 msgr2=0x7f25040b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 -- 172.21.15.97:0/715191501 shutdown_connections 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.321+0000 7f2509400640 1 -- 172.21.15.97:0/715191501 wait complete. 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.323+0000 7f2509400640 1 Processor -- start 2024-03-06T07:38:08.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.323+0000 7f2509400640 1 -- start start 2024-03-06T07:38:08.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2509400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f25041e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2509400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25040b4470 0x7f250415d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2509400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25040af010 0x7f2504138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2509400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504160730 con 0x7f24e80051c0 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2509400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504160270 con 0x7f25040af010 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2509400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250406cb50 con 0x7f25040b4470 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2502800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f25041e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2502800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f25041e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35078/0 (socket says 172.21.15.97:35078) 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.324+0000 7f2502800640 1 -- 172.21.15.97:0/2598830169 learned_addr learned my addr 172.21.15.97:0/2598830169 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2503200640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25040af010 0x7f2504138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f24f9e00640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25040b4470 0x7f250415d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2502800640 1 -- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25040b4470 msgr2=0x7f250415d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2502800640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25040b4470 0x7f250415d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2502800640 1 -- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25040af010 msgr2=0x7f2504138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2502800640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25040af010 0x7f2504138fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2502800640 1 -- 172.21.15.97:0/2598830169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25040b4a10 con 0x7f24e80051c0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2502800640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f25041e55c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f24f400f350 tx=0x7f24f400f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4003b90 con 0x7f24e80051c0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f4003d30 con 0x7f24e80051c0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4004220 con 0x7f24e80051c0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25040ae530 con 0x7f24e80051c0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.325+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25040aeaf0 con 0x7f24e80051c0 2024-03-06T07:38:08.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.327+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250404f330 con 0x7f24e80051c0 2024-03-06T07:38:08.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.327+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24f4005d10 con 0x7f24e80051c0 2024-03-06T07:38:08.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.328+0000 7f24fb400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24e006b4d0 0x7f24e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.328+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f24f408a980 con 0x7f24e80051c0 2024-03-06T07:38:08.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.329+0000 7f24f9e00640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24e006b4d0 0x7f24e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.329+0000 7f24f9e00640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24e006b4d0 0x7f24e006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0012b0 tx=0x7f24ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:08.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.333+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24f4012070 con 0x7f24e80051c0 2024-03-06T07:38:08.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.567+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2504072890 con 0x7f24e80051c0 2024-03-06T07:38:08.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.568+0000 7f24fb400640 1 -- 172.21.15.97:0/2598830169 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f24f4059740 con 0x7f24e80051c0 2024-03-06T07:38:08.569 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:08.569 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:08.570 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24e006b4d0 msgr2=0x7f24e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24e006b4d0 0x7f24e006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0012b0 tx=0x7f24ec009040 comp rx=0 tx=0).stop 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 msgr2=0x7f25041e55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f25041e55c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f24f400f350 tx=0x7f24f400f810 comp rx=0 tx=0).stop 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 shutdown_connections 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f24e006b4d0 0x7f24e006d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25040af010 0x7f2504138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f25040b4470 0x7f250415d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 --2- 172.21.15.97:0/2598830169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e80051c0 0x7f25041e55c0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.570+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 >> 172.21.15.97:0/2598830169 conn(0x7f25040b1730 msgr2=0x7f25040b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:08.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.571+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 shutdown_connections 2024-03-06T07:38:08.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:08.571+0000 7f2509400640 1 -- 172.21.15.97:0/2598830169 wait complete. 2024-03-06T07:38:08.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:08.792 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:09.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:09.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:09.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:10.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.009+0000 7f1080600640 1 Processor -- start 2024-03-06T07:38:10.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.010+0000 7f1080600640 1 -- start start 2024-03-06T07:38:10.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f1080600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1078144c00 0x7f107806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f1080600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f107815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f1080600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f10780ce0a0 0x7f10780ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f1080600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1078160370 con 0x7f1078144c00 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f1080600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107806ee70 con 0x7f10780ce0a0 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f1080600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107806cef0 con 0x7f107815f9d0 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f107dc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1078144c00 0x7f107806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f107dc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1078144c00 0x7f107806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35092/0 (socket says 172.21.15.97:35092) 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.011+0000 7f107dc00640 1 -- 172.21.15.97:0/2672305900 learned_addr learned my addr 172.21.15.97:0/2672305900 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107e600640 1 --2- 172.21.15.97:0/2672305900 >> v2:172.21.15.135:3300/0 conn(0x7f10780ce0a0 0x7f10780ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107d200640 1 --2- 172.21.15.97:0/2672305900 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f107815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107dc00640 1 -- 172.21.15.97:0/2672305900 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 msgr2=0x7f107815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107dc00640 1 --2- 172.21.15.97:0/2672305900 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f107815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107dc00640 1 -- 172.21.15.97:0/2672305900 >> v2:172.21.15.135:3300/0 conn(0x7f10780ce0a0 msgr2=0x7f10780ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107dc00640 1 --2- 172.21.15.97:0/2672305900 >> v2:172.21.15.135:3300/0 conn(0x7f10780ce0a0 0x7f10780ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107dc00640 1 -- 172.21.15.97:0/2672305900 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107806d1d0 con 0x7f1078144c00 2024-03-06T07:38:10.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.012+0000 7f107e600640 1 --2- 172.21.15.97:0/2672305900 >> v2:172.21.15.135:3300/0 conn(0x7f10780ce0a0 0x7f10780ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.013+0000 7f107dc00640 1 --2- 172.21.15.97:0/2672305900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1078144c00 0x7f107806e930 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f10780737b0 tx=0x7f106c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ab826f33568ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.013+0000 7f1077e00640 1 -- 172.21.15.97:0/2672305900 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c035020 con 0x7f1078144c00 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.013+0000 7f1077e00640 1 -- 172.21.15.97:0/2672305900 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c031870 con 0x7f1078144c00 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.013+0000 7f1077e00640 1 -- 172.21.15.97:0/2672305900 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c031c40 con 0x7f1078144c00 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 -- 172.21.15.97:0/2672305900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1078144c00 msgr2=0x7f107806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 --2- 172.21.15.97:0/2672305900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1078144c00 0x7f107806e930 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f10780737b0 tx=0x7f106c02b910 comp rx=0 tx=0).stop 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 -- 172.21.15.97:0/2672305900 shutdown_connections 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 --2- 172.21.15.97:0/2672305900 >> v2:172.21.15.135:3300/0 conn(0x7f10780ce0a0 0x7f10780ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 --2- 172.21.15.97:0/2672305900 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f107815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 --2- 172.21.15.97:0/2672305900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1078144c00 0x7f107806e930 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.014+0000 7f1080600640 1 -- 172.21.15.97:0/2672305900 >> 172.21.15.97:0/2672305900 conn(0x7f10780b1730 msgr2=0x7f10780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.015+0000 7f1080600640 1 -- 172.21.15.97:0/2672305900 shutdown_connections 2024-03-06T07:38:10.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.015+0000 7f1080600640 1 -- 172.21.15.97:0/2672305900 wait complete. 2024-03-06T07:38:10.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.018+0000 7f1080600640 1 Processor -- start 2024-03-06T07:38:10.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.018+0000 7f1080600640 1 -- start start 2024-03-06T07:38:10.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.020+0000 7f1080600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 0x7f107815a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.020+0000 7f1080600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1078144c00 0x7f10780aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.020+0000 7f1080600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f10780af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.020+0000 7f1080600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078160730 con 0x7f10780ce0a0 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.020+0000 7f1080600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107806d790 con 0x7f1078144c00 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.020+0000 7f1080600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107806cb50 con 0x7f107815f9d0 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 0x7f107815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 0x7f107815a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35096/0 (socket says 172.21.15.97:35096) 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 -- 172.21.15.97:0/2757263754 learned_addr learned my addr 172.21.15.97:0/2757263754 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 -- 172.21.15.97:0/2757263754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 msgr2=0x7f10780af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f10780af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 -- 172.21.15.97:0/2757263754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1078144c00 msgr2=0x7f10780aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1078144c00 0x7f10780aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 -- 172.21.15.97:0/2757263754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10781e7830 con 0x7f10780ce0a0 2024-03-06T07:38:10.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.021+0000 7f107dc00640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 0x7f107815a910 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f106c02b950 tx=0x7f106c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.022+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c040070 con 0x7f10780ce0a0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.022+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c03a420 con 0x7f10780ce0a0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.022+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c035040 con 0x7f10780ce0a0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.022+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10781e7b10 con 0x7f10780ce0a0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.022+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10781e80d0 con 0x7f10780ce0a0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.023+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107804f330 con 0x7f10780ce0a0 2024-03-06T07:38:10.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.024+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f106c04b060 con 0x7f10780ce0a0 2024-03-06T07:38:10.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.025+0000 7f1075600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105806b4d0 0x7f105806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.025+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f106c0ac170 con 0x7f10780ce0a0 2024-03-06T07:38:10.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.025+0000 7f107d200640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105806b4d0 0x7f105806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.026+0000 7f107d200640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105806b4d0 0x7f105806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1064006fd0 tx=0x7f1064008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:10.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.029+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f106c07b060 con 0x7f10780ce0a0 2024-03-06T07:38:10.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.266+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1078072890 con 0x7f10780ce0a0 2024-03-06T07:38:10.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.267+0000 7f1075600640 1 -- 172.21.15.97:0/2757263754 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f106c07a960 con 0x7f10780ce0a0 2024-03-06T07:38:10.270 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:10.270 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:10.270 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:10.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105806b4d0 msgr2=0x7f105806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105806b4d0 0x7f105806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1064006fd0 tx=0x7f1064008040 comp rx=0 tx=0).stop 2024-03-06T07:38:10.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 msgr2=0x7f107815a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 0x7f107815a910 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f106c02b950 tx=0x7f106c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 shutdown_connections 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f105806b4d0 0x7f105806d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f107815f9d0 0x7f10780af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1078144c00 0x7f10780aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 --2- 172.21.15.97:0/2757263754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10780ce0a0 0x7f107815a910 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 >> 172.21.15.97:0/2757263754 conn(0x7f10780b1730 msgr2=0x7f1078143600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 shutdown_connections 2024-03-06T07:38:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:10.270+0000 7f1080600640 1 -- 172.21.15.97:0/2757263754 wait complete. 2024-03-06T07:38:10.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:10.491 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:11.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:11.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.700+0000 7ff400a00640 1 Processor -- start 2024-03-06T07:38:11.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.700+0000 7ff400a00640 1 -- start start 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.701+0000 7ff400a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff3fc15f9d0 0x7ff3fc15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.701+0000 7ff400a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 0x7ff3fc070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff400a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc06cef0 con 0x7ff3fc144c00 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff400a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc06d1d0 con 0x7ff3fc15f9d0 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff400a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc06d4b0 con 0x7ff3fc070b10 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff3f9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff3fc144c00 0x7ff3fc06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52018/0 (socket says 172.21.15.97:52018) 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff3f9e00640 1 -- 172.21.15.97:0/3225568284 learned_addr learned my addr 172.21.15.97:0/3225568284 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456252293 0 0) 0x7ff3fc06cef0 con 0x7ff3fc144c00 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0003520 con 0x7ff3fc144c00 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff3fa800640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 0x7ff3fc070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.702+0000 7ff3f9400640 1 --2- 172.21.15.97:0/3225568284 >> v2:172.21.15.135:3300/0 conn(0x7ff3fc15f9d0 0x7ff3fc15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868970717 0 0) 0x7ff3e0003520 con 0x7ff3fc144c00 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 msgr2=0x7ff3fc070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 0x7ff3fc070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 >> v2:172.21.15.135:3300/0 conn(0x7ff3fc15f9d0 msgr2=0x7ff3fc15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 --2- 172.21.15.97:0/3225568284 >> v2:172.21.15.135:3300/0 conn(0x7ff3fc15f9d0 0x7ff3fc15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc06d790 con 0x7ff3fc144c00 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff3ec001670 con 0x7ff3fc144c00 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 >> v1:172.21.15.97:6789/0 conn(0x7ff3fc144c00 legacy=0x7ff3fc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 0x7ff3e00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e0003fe0 0x7ff3e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0004d40 0x7ff3e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc06d1d0 con 0x7ff3e0004d40 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc06d4b0 con 0x7ff3e0003fe0 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e000bac0 con 0x7ff3fc070b10 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3fa800640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0004d40 0x7ff3e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f9e00640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 0x7ff3e00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.703+0000 7ff3f9400640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e0003fe0 0x7ff3e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.704+0000 7ff3fa800640 1 -- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 msgr2=0x7ff3e00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:11.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.704+0000 7ff3fa800640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3fc070b10 0x7ff3e00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.704+0000 7ff3fa800640 1 -- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e0003fe0 msgr2=0x7ff3e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.704+0000 7ff3fa800640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e0003fe0 0x7ff3e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.704+0000 7ff3fa800640 1 -- 172.21.15.97:0/3225568284 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0003520 con 0x7ff3e0004d40 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.705+0000 7ff3fa800640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0004d40 0x7ff3e00051c0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4007360 tx=0x7ff3e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a551d10ca7a289e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.705+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4039070 con 0x7ff3e0004d40 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.705+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f402ccd0 con 0x7ff3e0004d40 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.705+0000 7ff3f8a00640 1 -- 172.21.15.97:0/3225568284 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4009be0 con 0x7ff3e0004d40 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff400a00640 1 -- 172.21.15.97:0/3225568284 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0004d40 msgr2=0x7ff3e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff400a00640 1 --2- 172.21.15.97:0/3225568284 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3e0004d40 0x7ff3e00051c0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4007360 tx=0x7ff3e0005830 comp rx=0 tx=0).stop 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff3fa800640 1 -- 172.21.15.97:0/3225568284 reap_dead start 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff400a00640 1 -- 172.21.15.97:0/3225568284 shutdown_connections 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff400a00640 1 -- 172.21.15.97:0/3225568284 >> 172.21.15.97:0/3225568284 conn(0x7ff3fc0b1730 msgr2=0x7ff3fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff400a00640 1 -- 172.21.15.97:0/3225568284 shutdown_connections 2024-03-06T07:38:11.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.706+0000 7ff400a00640 1 -- 172.21.15.97:0/3225568284 wait complete. 2024-03-06T07:38:11.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.708+0000 7ff400a00640 1 Processor -- start 2024-03-06T07:38:11.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.708+0000 7ff400a00640 1 -- start start 2024-03-06T07:38:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff400a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e0004d40 0x7ff3fc15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff400a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3fc0b3a30 0x7ff3fc0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff400a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 0x7ff3fc1f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff400a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc06ccd0 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff400a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc06cb50 con 0x7ff3fc0b3a30 2024-03-06T07:38:11.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff400a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc06df90 con 0x7ff3e0004d40 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 0x7ff3fc1f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 0x7ff3fc1f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35104/0 (socket says 172.21.15.97:35104) 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 -- 172.21.15.97:0/1012493781 learned_addr learned my addr 172.21.15.97:0/1012493781 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3f9400640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3fc0b3a30 0x7ff3fc0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3f9e00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e0004d40 0x7ff3fc15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 -- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e0004d40 msgr2=0x7ff3fc15a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e0004d40 0x7ff3fc15a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 -- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3fc0b3a30 msgr2=0x7ff3fc0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3fc0b3a30 0x7ff3fc0b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3fa800640 1 -- 172.21.15.97:0/1012493781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0b4850 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.709+0000 7ff3f9e00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e0004d40 0x7ff3fc15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.710+0000 7ff3fa800640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 0x7ff3fc1f7410 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4037040 tx=0x7ff3f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.710+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4039030 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.710+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4043070 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.710+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc15ae30 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.710+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4009be0 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.710+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc142950 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.712+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc04f330 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.712+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3f403d7d0 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.713+0000 7ff3f2000640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3dc06b500 0x7ff3dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.713+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff3f40ac5e0 con 0x7ff3fc0b43d0 2024-03-06T07:38:11.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.713+0000 7ff3f9e00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3dc06b500 0x7ff3dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.714+0000 7ff3f9e00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3dc06b500 0x7ff3dc06d9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec001050 tx=0x7ff3ec000fe0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:11.718+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3f407b310 con 0x7ff3fc0b43d0 2024-03-06T07:38:12.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.053+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3fc0724b0 con 0x7ff3fc0b43d0 2024-03-06T07:38:12.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.053+0000 7ff3f2000640 1 -- 172.21.15.97:0/1012493781 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7ff3f407ac10 con 0x7ff3fc0b43d0 2024-03-06T07:38:12.056 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:12.056 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:12.056 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3dc06b500 msgr2=0x7ff3dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3dc06b500 0x7ff3dc06d9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec001050 tx=0x7ff3ec000fe0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 msgr2=0x7ff3fc1f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 0x7ff3fc1f7410 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4037040 tx=0x7ff3f4004ab0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 shutdown_connections 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff3dc06b500 0x7ff3dc06d9c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3fc0b43d0 0x7ff3fc1f7410 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3fc0b3a30 0x7ff3fc0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 --2- 172.21.15.97:0/1012493781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3e0004d40 0x7ff3fc15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.057+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 >> 172.21.15.97:0/1012493781 conn(0x7ff3fc0b1730 msgr2=0x7ff3fc06e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.058+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 shutdown_connections 2024-03-06T07:38:12.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:12.058+0000 7ff400a00640 1 -- 172.21.15.97:0/1012493781 wait complete. 2024-03-06T07:38:12.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:12.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:13.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:13.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.497+0000 7f74ab200640 1 Processor -- start 2024-03-06T07:38:13.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.497+0000 7f74ab200640 1 -- start start 2024-03-06T07:38:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.498+0000 7f74ab200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f74ac0ceb40 0x7f74ac0cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.498+0000 7f74ab200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0cf000 con 0x7f74ac161760 2024-03-06T07:38:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74ab200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0cd570 con 0x7f74ac161380 2024-03-06T07:38:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74ab200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0cd850 con 0x7f74ac0ceb40 2024-03-06T07:38:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74aa800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f74ac161760 0x7f74ac0cfe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52022/0 (socket says 172.21.15.97:52022) 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74aa800640 1 -- 172.21.15.97:0/4212863699 learned_addr learned my addr 172.21.15.97:0/4212863699 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74a9400640 1 --2- 172.21.15.97:0/4212863699 >> v2:172.21.15.135:3300/0 conn(0x7f74ac0ceb40 0x7f74ac0cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898871641 0 0) 0x7f74ac0cf000 con 0x7f74ac161760 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.499+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7490003520 con 0x7f74ac161760 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028418569 0 0) 0x7f7490003520 con 0x7f74ac161760 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 >> v2:172.21.15.135:3300/0 conn(0x7f74ac0ceb40 msgr2=0x7f74ac0cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 --2- 172.21.15.97:0/4212863699 >> v2:172.21.15.135:3300/0 conn(0x7f74ac0ceb40 0x7f74ac0cf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 >> v1:172.21.15.135:6789/0 conn(0x7f74ac161380 legacy=0x7f74ac0ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0cdb30 con 0x7f74ac161760 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f74a40019f0 con 0x7f74ac161760 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 >> v1:172.21.15.97:6789/0 conn(0x7f74ac161760 legacy=0x7f74ac0cfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74900039b0 0x7f7490003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7490004800 0x7f7490004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74900051c0 0x7f749000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cd850 con 0x7f7490004800 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749000c160 con 0x7f74900039b0 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749000c360 con 0x7f74900051c0 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a9400640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7490004800 0x7f7490004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.500+0000 7f74a9e00640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74900039b0 0x7f7490003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74aa800640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74900051c0 0x7f749000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74a9400640 1 -- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74900051c0 msgr2=0x7f749000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74a9400640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74900051c0 0x7f749000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74a9400640 1 -- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74900039b0 msgr2=0x7f7490003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74a9400640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74900039b0 0x7f7490003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74a9400640 1 -- 172.21.15.97:0/4212863699 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490003520 con 0x7f7490004800 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74aa800640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74900051c0 0x7f749000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.501+0000 7f74aa800640 1 -- 172.21.15.97:0/4212863699 reap_dead start 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.502+0000 7f74a9400640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7490004800 0x7f7490004c80 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f7498009ac0 tx=0x7f7490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200692361eb889c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.502+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498039070 con 0x7f7490004800 2024-03-06T07:38:13.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.502+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749802c6b0 con 0x7f7490004800 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.502+0000 7f74a8a00640 1 -- 172.21.15.97:0/4212863699 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498009690 con 0x7f7490004800 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 -- 172.21.15.97:0/4212863699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7490004800 msgr2=0x7f7490004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7490004800 0x7f7490004c80 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f7498009ac0 tx=0x7f7490005830 comp rx=0 tx=0).stop 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 -- 172.21.15.97:0/4212863699 shutdown_connections 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 --2- 172.21.15.97:0/4212863699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7490004800 0x7f7490004c80 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 -- 172.21.15.97:0/4212863699 >> 172.21.15.97:0/4212863699 conn(0x7f74ac0b3690 msgr2=0x7f74ac0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 -- 172.21.15.97:0/4212863699 shutdown_connections 2024-03-06T07:38:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.503+0000 7f74ab200640 1 -- 172.21.15.97:0/4212863699 wait complete. 2024-03-06T07:38:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.504+0000 7f74ab200640 1 Processor -- start 2024-03-06T07:38:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.505+0000 7f74ab200640 1 -- start start 2024-03-06T07:38:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.505+0000 7f74ab200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7490004800 0x7f74ac0c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74ab200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 0x7f74ac1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74ab200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74ac1f3f70 0x7f74ac1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74ab200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0b2820 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74ab200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0b26a0 con 0x7f74ac1f3f70 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74ab200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0b29a0 con 0x7f7490004800 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 0x7f74ac1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 0x7f74ac1f3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35122/0 (socket says 172.21.15.97:35122) 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 -- 172.21.15.97:0/1340530582 learned_addr learned my addr 172.21.15.97:0/1340530582 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9e00640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7490004800 0x7f74ac0c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74aa800640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74ac1f3f70 0x7f74ac1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 -- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7490004800 msgr2=0x7f74ac0c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7490004800 0x7f74ac0c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 -- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74ac1f3f70 msgr2=0x7f74ac1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74ac1f3f70 0x7f74ac1f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9400640 1 -- 172.21.15.97:0/1340530582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0c80e0 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74aa800640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74ac1f3f70 0x7f74ac1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.506+0000 7f74a9e00640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7490004800 0x7f74ac0c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.507+0000 7f74a9400640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 0x7f74ac1f3a30 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f7498037040 tx=0x7f7498004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.507+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498042020 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.507+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749803a040 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.507+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498039070 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.507+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac0c83c0 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.507+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac0afa60 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.508+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac04f330 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.509+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7498045020 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.510+0000 7f74a2000640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f748c06b500 0x7f748c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.510+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f74980aba20 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.511+0000 7f74a9e00640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f748c06b500 0x7f748c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.511+0000 7f74a9e00640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f748c06b500 0x7f748c06d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f749c000c00 tx=0x7f749c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.518+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f749804c030 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.754+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74ac0b7ac0 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.755+0000 7f74a2000640 1 -- 172.21.15.97:0/1340530582 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f749807a7e0 con 0x7f74ac0c5ae0 2024-03-06T07:38:13.758 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:13.758 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:13.758 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f748c06b500 msgr2=0x7f748c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f748c06b500 0x7f748c06d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f749c000c00 tx=0x7f749c009040 comp rx=0 tx=0).stop 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 msgr2=0x7f74ac1f3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 0x7f74ac1f3a30 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f7498037040 tx=0x7f7498004730 comp rx=0 tx=0).stop 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 shutdown_connections 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f748c06b500 0x7f748c06d9c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74ac1f3f70 0x7f74ac1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74ac0c5ae0 0x7f74ac1f3a30 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 --2- 172.21.15.97:0/1340530582 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7490004800 0x7f74ac0c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 >> 172.21.15.97:0/1340530582 conn(0x7f74ac0b3690 msgr2=0x7f74ac06ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 shutdown_connections 2024-03-06T07:38:13.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:13.758+0000 7f74ab200640 1 -- 172.21.15.97:0/1340530582 wait complete. 2024-03-06T07:38:13.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:13.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:14.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:15.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.192+0000 7f124f400640 1 Processor -- start 2024-03-06T07:38:15.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.192+0000 7f124f400640 1 -- start start 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.193+0000 7f124f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f124815acb0 0x7f12480b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124f400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f12480b4680 0x7f124806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1248072340 0x7f124806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124f400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f124815b090 con 0x7f1248072340 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124f400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12480b2a30 con 0x7f12480b4680 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124f400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12480b4360 con 0x7f124815acb0 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1248072340 0x7f124806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1248072340 0x7f124806ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49614/0 (socket says 172.21.15.97:49614) 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124d400640 1 -- 172.21.15.97:0/2568841535 learned_addr learned my addr 172.21.15.97:0/2568841535 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:15.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124ca00640 1 --2- 172.21.15.97:0/2568841535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f124815acb0 0x7f12480b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f1247e00640 1 --2- 172.21.15.97:0/2568841535 >> v2:172.21.15.135:3300/0 conn(0x7f12480b4680 0x7f124806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124d400640 1 -- 172.21.15.97:0/2568841535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f124815acb0 msgr2=0x7f12480b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.194+0000 7f124d400640 1 --2- 172.21.15.97:0/2568841535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f124815acb0 0x7f12480b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.195+0000 7f124d400640 1 -- 172.21.15.97:0/2568841535 >> v2:172.21.15.135:3300/0 conn(0x7f12480b4680 msgr2=0x7f124806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.195+0000 7f124d400640 1 --2- 172.21.15.97:0/2568841535 >> v2:172.21.15.135:3300/0 conn(0x7f12480b4680 0x7f124806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.195+0000 7f124d400640 1 -- 172.21.15.97:0/2568841535 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12480b2cf0 con 0x7f1248072340 2024-03-06T07:38:15.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.195+0000 7f1247e00640 1 --2- 172.21.15.97:0/2568841535 >> v2:172.21.15.135:3300/0 conn(0x7f12480b4680 0x7f124806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:15.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.196+0000 7f124d400640 1 --2- 172.21.15.97:0/2568841535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1248072340 0x7f124806ea30 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f124806f020 tx=0x7f124002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d56c408e166abf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:15.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.196+0000 7f1247400640 1 -- 172.21.15.97:0/2568841535 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124002ca70 con 0x7f1248072340 2024-03-06T07:38:15.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.196+0000 7f1247400640 1 -- 172.21.15.97:0/2568841535 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124002cc10 con 0x7f1248072340 2024-03-06T07:38:15.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.196+0000 7f1247400640 1 -- 172.21.15.97:0/2568841535 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12400343f0 con 0x7f1248072340 2024-03-06T07:38:15.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 -- 172.21.15.97:0/2568841535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1248072340 msgr2=0x7f124806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 --2- 172.21.15.97:0/2568841535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1248072340 0x7f124806ea30 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f124806f020 tx=0x7f124002bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 -- 172.21.15.97:0/2568841535 shutdown_connections 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 --2- 172.21.15.97:0/2568841535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1248072340 0x7f124806ea30 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 --2- 172.21.15.97:0/2568841535 >> v2:172.21.15.135:3300/0 conn(0x7f12480b4680 0x7f124806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 --2- 172.21.15.97:0/2568841535 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f124815acb0 0x7f12480b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 -- 172.21.15.97:0/2568841535 >> 172.21.15.97:0/2568841535 conn(0x7f12480b1730 msgr2=0x7f12480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 -- 172.21.15.97:0/2568841535 shutdown_connections 2024-03-06T07:38:15.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.197+0000 7f124f400640 1 -- 172.21.15.97:0/2568841535 wait complete. 2024-03-06T07:38:15.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.199+0000 7f124f400640 1 Processor -- start 2024-03-06T07:38:15.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.199+0000 7f124f400640 1 -- start start 2024-03-06T07:38:15.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124f400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1248072340 0x7f124814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 0x7f124815b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124f400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f124815acb0 0x7f124806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124f400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12480b3d10 con 0x7f12480b4680 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124f400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12480b32b0 con 0x7f124815acb0 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124f400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12480b4010 con 0x7f1248072340 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 0x7f124815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 0x7f124815b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49628/0 (socket says 172.21.15.97:49628) 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 -- 172.21.15.97:0/3076101692 learned_addr learned my addr 172.21.15.97:0/3076101692 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124ca00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1248072340 0x7f124814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124d400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f124815acb0 0x7f124806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 -- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1248072340 msgr2=0x7f124814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1248072340 0x7f124814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 -- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f124815acb0 msgr2=0x7f124806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f124815acb0 0x7f124806d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f1247e00640 1 -- 172.21.15.97:0/3076101692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124814ab30 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.200+0000 7f124ca00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1248072340 0x7f124814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.201+0000 7f1247e00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 0x7f124815b260 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f123400c7f0 tx=0x7f123400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.201+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234015070 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.201+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1234007820 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.201+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124815ba90 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.201+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234007c10 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.201+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124815c050 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.202+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124804f330 con 0x7f12480b4680 2024-03-06T07:38:15.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.203+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1234018070 con 0x7f12480b4680 2024-03-06T07:38:15.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.205+0000 7f1244c00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f122806b4d0 0x7f122806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.205+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f123408a450 con 0x7f12480b4680 2024-03-06T07:38:15.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.205+0000 7f124ca00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f122806b4d0 0x7f122806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.206+0000 7f124ca00640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f122806b4d0 0x7f122806d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f123c006fd0 tx=0x7f123c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:15.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.209+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1234059190 con 0x7f12480b4680 2024-03-06T07:38:15.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.445+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f124815f560 con 0x7f12480b4680 2024-03-06T07:38:15.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.446+0000 7f1244c00640 1 -- 172.21.15.97:0/3076101692 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f1234058a90 con 0x7f12480b4680 2024-03-06T07:38:15.448 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:15.449 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:15.449 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:15.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f122806b4d0 msgr2=0x7f122806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f122806b4d0 0x7f122806d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f123c006fd0 tx=0x7f123c008040 comp rx=0 tx=0).stop 2024-03-06T07:38:15.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 msgr2=0x7f124815b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 0x7f124815b260 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f123400c7f0 tx=0x7f123400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 shutdown_connections 2024-03-06T07:38:15.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f122806b4d0 0x7f122806d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f124815acb0 0x7f124806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12480b4680 0x7f124815b260 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 --2- 172.21.15.97:0/3076101692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1248072340 0x7f124814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 >> 172.21.15.97:0/3076101692 conn(0x7f12480b1730 msgr2=0x7f124815e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 shutdown_connections 2024-03-06T07:38:15.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:15.449+0000 7f124f400640 1 -- 172.21.15.97:0/3076101692 wait complete. 2024-03-06T07:38:15.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:15.670 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:16.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:16.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.876+0000 7fae50000640 1 Processor -- start 2024-03-06T07:38:16.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.876+0000 7fae50000640 1 -- start start 2024-03-06T07:38:16.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae50000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fae480af1c0 0x7fae4815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae50000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fae48070d20 0x7fae4806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae50000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae48071d30 con 0x7fae4806d1a0 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae50000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae480711d0 con 0x7fae480af1c0 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae50000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae48071490 con 0x7fae48070d20 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae4d600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fae480af1c0 0x7fae4815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae4e000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fae4806d1a0 0x7fae4806e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51054/0 (socket says 172.21.15.97:51054) 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.878+0000 7fae4e000640 1 -- 172.21.15.97:0/373963225 learned_addr learned my addr 172.21.15.97:0/373963225 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:16.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.879+0000 7fae4cc00640 1 --2- 172.21.15.97:0/373963225 >> v2:172.21.15.135:3300/0 conn(0x7fae48070d20 0x7fae4806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.879+0000 7fae4d600640 1 -- 172.21.15.97:0/373963225 >> v2:172.21.15.135:3300/0 conn(0x7fae48070d20 msgr2=0x7fae4806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:16.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.879+0000 7fae4d600640 1 --2- 172.21.15.97:0/373963225 >> v2:172.21.15.135:3300/0 conn(0x7fae48070d20 0x7fae4806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.879+0000 7fae4d600640 1 -- 172.21.15.97:0/373963225 >> v1:172.21.15.97:6789/0 conn(0x7fae4806d1a0 legacy=0x7fae4806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.879+0000 7fae4d600640 1 -- 172.21.15.97:0/373963225 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae48071770 con 0x7fae480af1c0 2024-03-06T07:38:16.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.880+0000 7fae4d600640 1 --2- 172.21.15.97:0/373963225 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae480af1c0 0x7fae4815eff0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0097a0 tx=0x7fae3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedeaed7bfdc011 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.880+0000 7fae47e00640 1 -- 172.21.15.97:0/373963225 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c035020 con 0x7fae480af1c0 2024-03-06T07:38:16.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.880+0000 7fae47e00640 1 -- 172.21.15.97:0/373963225 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c031830 con 0x7fae480af1c0 2024-03-06T07:38:16.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.880+0000 7fae47e00640 1 -- 172.21.15.97:0/373963225 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c031c00 con 0x7fae480af1c0 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 -- 172.21.15.97:0/373963225 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae480af1c0 msgr2=0x7fae4815eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 --2- 172.21.15.97:0/373963225 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae480af1c0 0x7fae4815eff0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0097a0 tx=0x7fae3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 -- 172.21.15.97:0/373963225 shutdown_connections 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 --2- 172.21.15.97:0/373963225 >> v2:172.21.15.135:3300/0 conn(0x7fae48070d20 0x7fae4806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 --2- 172.21.15.97:0/373963225 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae480af1c0 0x7fae4815eff0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 -- 172.21.15.97:0/373963225 >> 172.21.15.97:0/373963225 conn(0x7fae480b1730 msgr2=0x7fae480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 -- 172.21.15.97:0/373963225 shutdown_connections 2024-03-06T07:38:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.881+0000 7fae50000640 1 -- 172.21.15.97:0/373963225 wait complete. 2024-03-06T07:38:16.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.883+0000 7fae50000640 1 Processor -- start 2024-03-06T07:38:16.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.883+0000 7fae50000640 1 -- start start 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae50000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae4806d1a0 0x7fae48155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae50000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 0x7fae4815f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae50000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae480af1c0 0x7fae481527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae50000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4806ebe0 con 0x7fae48070d20 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae50000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae480b30b0 con 0x7fae480af1c0 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae50000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4806ed60 con 0x7fae4806d1a0 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae4cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 0x7fae4815f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae4cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 0x7fae4815f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49646/0 (socket says 172.21.15.97:49646) 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae4cc00640 1 -- 172.21.15.97:0/2823448811 learned_addr learned my addr 172.21.15.97:0/2823448811 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.884+0000 7fae4d600640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae4806d1a0 0x7fae48155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4e000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae480af1c0 0x7fae481527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4cc00640 1 -- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae4806d1a0 msgr2=0x7fae48155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4cc00640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae4806d1a0 0x7fae48155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4cc00640 1 -- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae480af1c0 msgr2=0x7fae481527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4cc00640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae480af1c0 0x7fae481527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4cc00640 1 -- 172.21.15.97:0/2823448811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4815fb10 con 0x7fae48070d20 2024-03-06T07:38:16.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4d600640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae4806d1a0 0x7fae48155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:16.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae4cc00640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 0x7fae4815f5d0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fae3400d800 tx=0x7fae3400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3400b6c0 con 0x7fae48070d20 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34019070 con 0x7fae48070d20 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3400ba80 con 0x7fae48070d20 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4815fdb0 con 0x7fae48070d20 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.885+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae48160370 con 0x7fae48070d20 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.887+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4804f330 con 0x7fae48070d20 2024-03-06T07:38:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.887+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae3400bc20 con 0x7fae48070d20 2024-03-06T07:38:16.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.888+0000 7fae45600640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae2806b4d0 0x7fae2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.888+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fae3408a6e0 con 0x7fae48070d20 2024-03-06T07:38:16.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.889+0000 7fae4d600640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae2806b4d0 0x7fae2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.889+0000 7fae4d600640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae2806b4d0 0x7fae2806d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fae3c036000 tx=0x7fae3c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:16.898+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae34059530 con 0x7fae48070d20 2024-03-06T07:38:17.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.138+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae48074be0 con 0x7fae48070d20 2024-03-06T07:38:17.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.139+0000 7fae45600640 1 -- 172.21.15.97:0/2823448811 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7fae34058e30 con 0x7fae48070d20 2024-03-06T07:38:17.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:17.141 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:17.142 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:17.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.141+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae2806b4d0 msgr2=0x7fae2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.141+0000 7fae50000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae2806b4d0 0x7fae2806d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fae3c036000 tx=0x7fae3c008030 comp rx=0 tx=0).stop 2024-03-06T07:38:17.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 msgr2=0x7fae4815f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 0x7fae4815f5d0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fae3400d800 tx=0x7fae3400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 shutdown_connections 2024-03-06T07:38:17.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fae2806b4d0 0x7fae2806d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae480af1c0 0x7fae481527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae48070d20 0x7fae4815f5d0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 --2- 172.21.15.97:0/2823448811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fae4806d1a0 0x7fae48155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 >> 172.21.15.97:0/2823448811 conn(0x7fae480b1730 msgr2=0x7fae480b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 shutdown_connections 2024-03-06T07:38:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:17.142+0000 7fae50000640 1 -- 172.21.15.97:0/2823448811 wait complete. 2024-03-06T07:38:17.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:17.365 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:18.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:18.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.570+0000 7f69fb000640 1 Processor -- start 2024-03-06T07:38:18.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.570+0000 7f69fb000640 1 -- start start 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69fb000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f69f415e0d0 0x7f69f40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69fb000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415f470 0x7f69f40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69fb000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40cea60 con 0x7f69f40d0df0 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69fb000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40d1260 con 0x7f69f415e0d0 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69fb000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40d2f60 con 0x7f69f415f470 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f69f415e0d0 0x7f69f40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69f3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f69f415e0d0 0x7f69f40d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49648/0 (socket says 172.21.15.97:49648) 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.572+0000 7f69f3e00640 1 -- 172.21.15.97:0/2276338554 learned_addr learned my addr 172.21.15.97:0/2276338554 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.573+0000 7f69eb400640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415f470 0x7f69f40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.573+0000 7f69f3e00640 1 -- 172.21.15.97:0/2276338554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415f470 msgr2=0x7f69f40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.573+0000 7f69f3e00640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415f470 0x7f69f40d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.573+0000 7f69f3e00640 1 -- 172.21.15.97:0/2276338554 >> v1:172.21.15.97:6789/0 conn(0x7f69f40d0df0 legacy=0x7f69f4160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.573+0000 7f69f3e00640 1 -- 172.21.15.97:0/2276338554 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f40d2350 con 0x7f69f415e0d0 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.573+0000 7f69eb400640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415f470 0x7f69f40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.574+0000 7f69f3e00640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f415e0d0 0x7f69f40d1a90 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009870 tx=0x7f69e402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5325ae26d103a829 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.574+0000 7f69f3400640 1 -- 172.21.15.97:0/2276338554 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e402f830 con 0x7f69f415e0d0 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.574+0000 7f69f3400640 1 -- 172.21.15.97:0/2276338554 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e402f9d0 con 0x7f69f415e0d0 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.574+0000 7f69f3400640 1 -- 172.21.15.97:0/2276338554 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e402fda0 con 0x7f69f415e0d0 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 -- 172.21.15.97:0/2276338554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f415e0d0 msgr2=0x7f69f40d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f415e0d0 0x7f69f40d1a90 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009870 tx=0x7f69e402ba10 comp rx=0 tx=0).stop 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 -- 172.21.15.97:0/2276338554 shutdown_connections 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415f470 0x7f69f40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 --2- 172.21.15.97:0/2276338554 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f415e0d0 0x7f69f40d1a90 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 -- 172.21.15.97:0/2276338554 >> 172.21.15.97:0/2276338554 conn(0x7f69f40d3ec0 msgr2=0x7f69f40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 -- 172.21.15.97:0/2276338554 shutdown_connections 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.575+0000 7f69fb000640 1 -- 172.21.15.97:0/2276338554 wait complete. 2024-03-06T07:38:18.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.578+0000 7f69fb000640 1 Processor -- start 2024-03-06T07:38:18.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.578+0000 7f69fb000640 1 -- start start 2024-03-06T07:38:18.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.579+0000 7f69fb000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 0x7f69f40b9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.579+0000 7f69fb000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415e0d0 0x7f69f406bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.579+0000 7f69fb000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f415f470 0x7f69f4151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.579+0000 7f69fb000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40d2bf0 con 0x7f69f40d0df0 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.579+0000 7f69fb000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40d2910 con 0x7f69f415f470 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.579+0000 7f69fb000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f415fb30 con 0x7f69f415e0d0 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 0x7f69f40b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 0x7f69f40b9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49652/0 (socket says 172.21.15.97:49652) 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 -- 172.21.15.97:0/4217857229 learned_addr learned my addr 172.21.15.97:0/4217857229 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69eb400640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415e0d0 0x7f69f406bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f9000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f415f470 0x7f69f4151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 -- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415e0d0 msgr2=0x7f69f406bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415e0d0 0x7f69f406bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 -- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f415f470 msgr2=0x7f69f4151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f415f470 0x7f69f4151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.580+0000 7f69f3e00640 1 -- 172.21.15.97:0/4217857229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f40c78c0 con 0x7f69f40d0df0 2024-03-06T07:38:18.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.581+0000 7f69f3e00640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 0x7f69f40b9370 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f69e402ba50 tx=0x7f69e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.581+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4047070 con 0x7f69f40d0df0 2024-03-06T07:38:18.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.581+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e402f800 con 0x7f69f40d0df0 2024-03-06T07:38:18.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.581+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e40426a0 con 0x7f69f40d0df0 2024-03-06T07:38:18.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.581+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f40c7ba0 con 0x7f69f40d0df0 2024-03-06T07:38:18.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.581+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f40c8160 con 0x7f69f40d0df0 2024-03-06T07:38:18.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.582+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f404f330 con 0x7f69f40d0df0 2024-03-06T07:38:18.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.583+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69e404a070 con 0x7f69f40d0df0 2024-03-06T07:38:18.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.584+0000 7f69f0c00640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69d006b4d0 0x7f69d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.584+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f69e40ac5c0 con 0x7f69f40d0df0 2024-03-06T07:38:18.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.584+0000 7f69eb400640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69d006b4d0 0x7f69d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.585+0000 7f69eb400640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69d006b4d0 0x7f69d006d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f69f40b60b0 tx=0x7f69dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.589+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69e4033030 con 0x7f69f40d0df0 2024-03-06T07:38:18.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.823+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69f408d920 con 0x7f69f40d0df0 2024-03-06T07:38:18.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.824+0000 7f69f0c00640 1 -- 172.21.15.97:0/4217857229 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f69e407fe20 con 0x7f69f40d0df0 2024-03-06T07:38:18.826 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:18.826 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:18.827 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69d006b4d0 msgr2=0x7f69d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69d006b4d0 0x7f69d006d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f69f40b60b0 tx=0x7f69dc008040 comp rx=0 tx=0).stop 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 msgr2=0x7f69f40b9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 0x7f69f40b9370 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f69e402ba50 tx=0x7f69e4004730 comp rx=0 tx=0).stop 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 shutdown_connections 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f69d006b4d0 0x7f69d006d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69f415f470 0x7f69f4151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f69f415e0d0 0x7f69f406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 --2- 172.21.15.97:0/4217857229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69f40d0df0 0x7f69f40b9370 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.826+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 >> 172.21.15.97:0/4217857229 conn(0x7f69f40d3ec0 msgr2=0x7f69f40b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.827+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 shutdown_connections 2024-03-06T07:38:18.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:18.827+0000 7f69fb000640 1 -- 172.21.15.97:0/4217857229 wait complete. 2024-03-06T07:38:19.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:19.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:20.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:20.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.260+0000 7f9eeca00640 1 Processor -- start 2024-03-06T07:38:20.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.260+0000 7f9eeca00640 1 -- start start 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9eeca00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ee80d2360 0x7f9ee80d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9eeca00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ee80b3ec0 0x7f9ee80b4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9eeca00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee8161960 con 0x7f9ee8161580 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9eeca00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee80d2ce0 con 0x7f9ee80b3ec0 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9eeca00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee80cf6f0 con 0x7f9ee80d2360 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9ee7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ee80b3ec0 0x7f9ee80b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9ee7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ee80b3ec0 0x7f9ee80b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49658/0 (socket says 172.21.15.97:49658) 2024-03-06T07:38:20.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.262+0000 7f9ee7e00640 1 -- 172.21.15.97:0/2262036131 learned_addr learned my addr 172.21.15.97:0/2262036131 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:20.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.263+0000 7f9ee7e00640 1 -- 172.21.15.97:0/2262036131 >> v2:172.21.15.135:3300/0 conn(0x7f9ee80d2360 msgr2=0x7f9ee80d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:20.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.263+0000 7f9ee7e00640 1 --2- 172.21.15.97:0/2262036131 >> v2:172.21.15.135:3300/0 conn(0x7f9ee80d2360 0x7f9ee80d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.263+0000 7f9ee7e00640 1 -- 172.21.15.97:0/2262036131 >> v1:172.21.15.135:6789/0 conn(0x7f9ee8161580 legacy=0x7f9ee80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:20.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.263+0000 7f9ee7e00640 1 -- 172.21.15.97:0/2262036131 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee80cf9d0 con 0x7f9ee80b3ec0 2024-03-06T07:38:20.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.264+0000 7f9ee7e00640 1 --2- 172.21.15.97:0/2262036131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80b4310 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80d2e60 tx=0x7f9ee002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89b6ff5873c5bba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:20.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.264+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2262036131 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee002ca80 con 0x7f9ee80b3ec0 2024-03-06T07:38:20.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.264+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2262036131 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee002cc20 con 0x7f9ee80b3ec0 2024-03-06T07:38:20.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.264+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2262036131 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00343f0 con 0x7f9ee80b3ec0 2024-03-06T07:38:20.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.264+0000 7f9eeca00640 1 -- 172.21.15.97:0/2262036131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee80b3ec0 msgr2=0x7f9ee80b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.264+0000 7f9eeca00640 1 --2- 172.21.15.97:0/2262036131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80b4310 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80d2e60 tx=0x7f9ee002bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.265+0000 7f9eeca00640 1 -- 172.21.15.97:0/2262036131 shutdown_connections 2024-03-06T07:38:20.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.265+0000 7f9eeca00640 1 --2- 172.21.15.97:0/2262036131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80b4310 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.265+0000 7f9eeca00640 1 --2- 172.21.15.97:0/2262036131 >> v2:172.21.15.135:3300/0 conn(0x7f9ee80d2360 0x7f9ee80d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.265+0000 7f9eeca00640 1 -- 172.21.15.97:0/2262036131 >> 172.21.15.97:0/2262036131 conn(0x7f9ee80d3d80 msgr2=0x7f9ee80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:20.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.265+0000 7f9eeca00640 1 -- 172.21.15.97:0/2262036131 shutdown_connections 2024-03-06T07:38:20.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.265+0000 7f9eeca00640 1 -- 172.21.15.97:0/2262036131 wait complete. 2024-03-06T07:38:20.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.267+0000 7f9eeca00640 1 Processor -- start 2024-03-06T07:38:20.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.267+0000 7f9eeca00640 1 -- start start 2024-03-06T07:38:20.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9eeca00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9eeca00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ee80d2360 0x7f9ee806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9eeca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 0x7f9ee8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9eeca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80b39a0 con 0x7f9ee8161580 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9eeca00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80b36c0 con 0x7f9ee80b3ec0 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9eeca00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80cf3a0 con 0x7f9ee80d2360 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 0x7f9ee8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 0x7f9ee8155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49668/0 (socket says 172.21.15.97:49668) 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 -- 172.21.15.97:0/680502683 learned_addr learned my addr 172.21.15.97:0/680502683 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7400640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 -- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ee80d2360 msgr2=0x7f9ee806c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ee80d2360 0x7f9ee806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 -- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ee80b3ec0 msgr2=0x7f9ee80cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.269+0000 7f9ee7e00640 1 -- 172.21.15.97:0/680502683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee80caef0 con 0x7f9ee8161580 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.270+0000 7f9ee7e00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 0x7f9ee8155ca0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009f10 tx=0x7f9ee0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.270+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0043020 con 0x7f9ee8161580 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.270+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee003e450 con 0x7f9ee8161580 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.270+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee003e820 con 0x7f9ee8161580 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.270+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee80cb1d0 con 0x7f9ee8161580 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.270+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee80cb790 con 0x7f9ee8161580 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.271+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee804f330 con 0x7f9ee8161580 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.272+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ee0046020 con 0x7f9ee8161580 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.274+0000 7f9edf400640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ec406b4d0 0x7f9ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.274+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9ee00acc30 con 0x7f9ee8161580 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.274+0000 7f9ee7400640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ec406b4d0 0x7f9ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.275+0000 7f9ee7400640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ec406b4d0 0x7f9ec406d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806ce40 tx=0x7f9ed8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.278+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ee004d030 con 0x7f9ee8161580 2024-03-06T07:38:20.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.511+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ee80d3b60 con 0x7f9ee8161580 2024-03-06T07:38:20.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.512+0000 7f9edf400640 1 -- 172.21.15.97:0/680502683 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f9ee007b9f0 con 0x7f9ee8161580 2024-03-06T07:38:20.514 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:20.514 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:20.515 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.514+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ec406b4d0 msgr2=0x7f9ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.514+0000 7f9eeca00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ec406b4d0 0x7f9ec406d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806ce40 tx=0x7f9ed8009040 comp rx=0 tx=0).stop 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.514+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 msgr2=0x7f9ee8155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.514+0000 7f9eeca00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 0x7f9ee8155ca0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009f10 tx=0x7f9ee0004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 shutdown_connections 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9ec406b4d0 0x7f9ec406d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee8161580 0x7f9ee8155ca0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ee80d2360 0x7f9ee806c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 --2- 172.21.15.97:0/680502683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ee80b3ec0 0x7f9ee80cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 >> 172.21.15.97:0/680502683 conn(0x7f9ee80d3d80 msgr2=0x7f9ee80d0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:20.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 shutdown_connections 2024-03-06T07:38:20.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:20.515+0000 7f9eeca00640 1 -- 172.21.15.97:0/680502683 wait complete. 2024-03-06T07:38:20.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:20.736 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:21.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:21.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.944+0000 7fbccfe00640 1 Processor -- start 2024-03-06T07:38:21.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.944+0000 7fbccfe00640 1 -- start start 2024-03-06T07:38:21.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.945+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd00d0c00 0x7fbcd00d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.945+0000 7fbccfe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbcd00b2e50 0x7fbcd00b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.945+0000 7fbccfe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd00b5240 con 0x7fbcd015e200 2024-03-06T07:38:21.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.945+0000 7fbccfe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd00d2ec0 con 0x7fbcd00b2e50 2024-03-06T07:38:21.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.945+0000 7fbccfe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd00cf4d0 con 0x7fbcd00d0c00 2024-03-06T07:38:21.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.946+0000 7fbccf400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbcd00b2e50 0x7fbcd00b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.946+0000 7fbcce000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd00d0c00 0x7fbcd00d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.946+0000 7fbccf400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbcd00b2e50 0x7fbcd00b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:42646/0 (socket says 172.21.15.97:42646) 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.946+0000 7fbccf400640 1 -- 172.21.15.97:0/2240617524 learned_addr learned my addr 172.21.15.97:0/2240617524 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.947+0000 7fbccf400640 1 -- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd00d0c00 msgr2=0x7fbcd00d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.947+0000 7fbccf400640 1 --2- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd00d0c00 0x7fbcd00d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.947+0000 7fbccf400640 1 -- 172.21.15.97:0/2240617524 >> v1:172.21.15.135:6789/0 conn(0x7fbcd015e200 legacy=0x7fbcd00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.947+0000 7fbccf400640 1 -- 172.21.15.97:0/2240617524 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd00b5500 con 0x7fbcd00b2e50 2024-03-06T07:38:21.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.947+0000 7fbcce000640 1 --2- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd00d0c00 0x7fbcd00d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:38:21.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.948+0000 7fbccf400640 1 --2- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00b2e50 0x7fbcd00b3640 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009e50 tx=0x7fbcc802bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded91d0704a0afbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.949+0000 7fbccd600640 1 -- 172.21.15.97:0/2240617524 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc802cb90 con 0x7fbcd00b2e50 2024-03-06T07:38:21.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.949+0000 7fbccd600640 1 -- 172.21.15.97:0/2240617524 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc802cd30 con 0x7fbcd00b2e50 2024-03-06T07:38:21.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.949+0000 7fbccd600640 1 -- 172.21.15.97:0/2240617524 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8034610 con 0x7fbcd00b2e50 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.949+0000 7fbccfe00640 1 -- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00b2e50 msgr2=0x7fbcd00b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.949+0000 7fbccfe00640 1 --2- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00b2e50 0x7fbcd00b3640 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009e50 tx=0x7fbcc802bff0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.950+0000 7fbccfe00640 1 -- 172.21.15.97:0/2240617524 shutdown_connections 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.950+0000 7fbccfe00640 1 --2- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd00d0c00 0x7fbcd00d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.950+0000 7fbccfe00640 1 --2- 172.21.15.97:0/2240617524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00b2e50 0x7fbcd00b3640 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.950+0000 7fbccfe00640 1 -- 172.21.15.97:0/2240617524 >> 172.21.15.97:0/2240617524 conn(0x7fbcd00d3d30 msgr2=0x7fbcd00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:21.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.950+0000 7fbccfe00640 1 -- 172.21.15.97:0/2240617524 shutdown_connections 2024-03-06T07:38:21.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.950+0000 7fbccfe00640 1 -- 172.21.15.97:0/2240617524 wait complete. 2024-03-06T07:38:21.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.952+0000 7fbccfe00640 1 Processor -- start 2024-03-06T07:38:21.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.953+0000 7fbccfe00640 1 -- start start 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.954+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 0x7fbcd0151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.954+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00d0c00 0x7fbcd00ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.954+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd015e200 0x7fbcd00ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.954+0000 7fbccfe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd00cfd60 con 0x7fbcd00b2e50 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.954+0000 7fbccfe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd00cfa80 con 0x7fbcd00d0c00 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.954+0000 7fbccfe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd015e5e0 con 0x7fbcd015e200 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 0x7fbcd0151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 0x7fbcd0151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49680/0 (socket says 172.21.15.97:49680) 2024-03-06T07:38:21.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 -- 172.21.15.97:0/3873583257 learned_addr learned my addr 172.21.15.97:0/3873583257 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:21.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 -- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd015e200 msgr2=0x7fbcd00ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccf400640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd015e200 0x7fbcd00ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbcce000640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00d0c00 0x7fbcd00ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd015e200 0x7fbcd00ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 -- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00d0c00 msgr2=0x7fbcd00ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00d0c00 0x7fbcd00ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 -- 172.21.15.97:0/3873583257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd01f7240 con 0x7fbcd00b2e50 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.955+0000 7fbccea00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 0x7fbcd0151ef0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000c3f0 tx=0x7fbcc000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.956+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc00052d0 con 0x7fbcd00b2e50 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.956+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc0005490 con 0x7fbcd00b2e50 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.956+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0007260 con 0x7fbcd00b2e50 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.956+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd01f7520 con 0x7fbcd00b2e50 2024-03-06T07:38:21.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.956+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd01f7ae0 con 0x7fbcd00b2e50 2024-03-06T07:38:21.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.957+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd004f330 con 0x7fbcd00b2e50 2024-03-06T07:38:21.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.958+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcc0007400 con 0x7fbcd00b2e50 2024-03-06T07:38:21.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.959+0000 7fbcc6a00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcb006b4d0 0x7fbcb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.959+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbcc0089710 con 0x7fbcd00b2e50 2024-03-06T07:38:21.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.960+0000 7fbcce000640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcb006b4d0 0x7fbcb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.960+0000 7fbcce000640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcb006b4d0 0x7fbcb006d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00c7a10 tx=0x7fbcbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:21.964+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcc0058560 con 0x7fbcd00b2e50 2024-03-06T07:38:22.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.199+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcd00d4560 con 0x7fbcd00b2e50 2024-03-06T07:38:22.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.200+0000 7fbcc6a00640 1 -- 172.21.15.97:0/3873583257 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7fbcc0057e60 con 0x7fbcd00b2e50 2024-03-06T07:38:22.202 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:22.202 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:22.202 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:22.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcb006b4d0 msgr2=0x7fbcb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:22.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcb006b4d0 0x7fbcb006d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00c7a10 tx=0x7fbcbc008040 comp rx=0 tx=0).stop 2024-03-06T07:38:22.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 msgr2=0x7fbcd0151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:22.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 0x7fbcd0151ef0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000c3f0 tx=0x7fbcc000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 shutdown_connections 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbcb006b4d0 0x7fbcb006d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbcd015e200 0x7fbcd00ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd00d0c00 0x7fbcd00ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 --2- 172.21.15.97:0/3873583257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbcd00b2e50 0x7fbcd0151ef0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 >> 172.21.15.97:0/3873583257 conn(0x7fbcd00d3d30 msgr2=0x7fbcd006c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:22.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.203+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 shutdown_connections 2024-03-06T07:38:22.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:22.204+0000 7fbccfe00640 1 -- 172.21.15.97:0/3873583257 wait complete. 2024-03-06T07:38:22.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:22.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:23.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:23.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.642+0000 7f9b3c000640 1 Processor -- start 2024-03-06T07:38:23.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.642+0000 7f9b3c000640 1 -- start start 2024-03-06T07:38:23.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.644+0000 7f9b3c000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 0x7f9b3415e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.644+0000 7f9b3c000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b340b35b0 con 0x7f9b34151660 2024-03-06T07:38:23.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.644+0000 7f9b3c000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b340b3890 con 0x7f9b340cf430 2024-03-06T07:38:23.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.644+0000 7f9b3c000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340b3ee0 con 0x7f9b3415dff0 2024-03-06T07:38:23.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.645+0000 7f9b39600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9b34151660 0x7f9b34071520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51076/0 (socket says 172.21.15.97:51076) 2024-03-06T07:38:23.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.645+0000 7f9b39600640 1 -- 172.21.15.97:0/2939306966 learned_addr learned my addr 172.21.15.97:0/2939306966 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.645+0000 7f9b3a000640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 0x7f9b3415e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.645+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632948426 0 0) 0x7f9b340b35b0 con 0x7f9b34151660 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.645+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c003440 con 0x7f9b34151660 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215735550 0 0) 0x7f9b1c003440 con 0x7f9b34151660 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 msgr2=0x7f9b3415e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 0x7f9b3415e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 >> v1:172.21.15.135:6789/0 conn(0x7f9b340cf430 legacy=0x7f9b3406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b340b3b70 con 0x7f9b34151660 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9b28001670 con 0x7f9b34151660 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 >> v1:172.21.15.97:6789/0 conn(0x7f9b34151660 legacy=0x7f9b34071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 0x7f9b1c003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b1c004050 0x7f9b1c00a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c00b090 0x7f9b1c00d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340b3ee0 con 0x7f9b1c00b090 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c00de00 con 0x7f9b1c004050 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c00e000 con 0x7f9b3415dff0 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b3a000640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c00b090 0x7f9b1c00d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b39600640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 0x7f9b1c003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.646+0000 7f9b38c00640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b1c004050 0x7f9b1c00a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.647+0000 7f9b3a000640 1 -- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 msgr2=0x7f9b1c003ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.647+0000 7f9b3a000640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b3415dff0 0x7f9b1c003ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.647+0000 7f9b3a000640 1 -- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b1c004050 msgr2=0x7f9b1c00a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.647+0000 7f9b3a000640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b1c004050 0x7f9b1c00a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.647+0000 7f9b3a000640 1 -- 172.21.15.97:0/2939306966 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c003440 con 0x7f9b1c00b090 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b3a000640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c00b090 0x7f9b1c00d520 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009fc0 tx=0x7f9b1c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121b304ca5936fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c039070 con 0x7f9b1c00b090 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c02cc20 con 0x7f9b1c00b090 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b33e00640 1 -- 172.21.15.97:0/2939306966 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c009c10 con 0x7f9b1c00b090 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b3c000640 1 -- 172.21.15.97:0/2939306966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c00b090 msgr2=0x7f9b1c00d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b3c000640 1 --2- 172.21.15.97:0/2939306966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c00b090 0x7f9b1c00d520 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009fc0 tx=0x7f9b1c003730 comp rx=0 tx=0).stop 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.648+0000 7f9b3a000640 1 -- 172.21.15.97:0/2939306966 reap_dead start 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.649+0000 7f9b3c000640 1 -- 172.21.15.97:0/2939306966 shutdown_connections 2024-03-06T07:38:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.649+0000 7f9b3c000640 1 -- 172.21.15.97:0/2939306966 >> 172.21.15.97:0/2939306966 conn(0x7f9b340b1730 msgr2=0x7f9b340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.649+0000 7f9b3c000640 1 -- 172.21.15.97:0/2939306966 shutdown_connections 2024-03-06T07:38:23.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.649+0000 7f9b3c000640 1 -- 172.21.15.97:0/2939306966 wait complete. 2024-03-06T07:38:23.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.650+0000 7f9b3c000640 1 Processor -- start 2024-03-06T07:38:23.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.651+0000 7f9b3c000640 1 -- start start 2024-03-06T07:38:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.651+0000 7f9b3c000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b1c00b090 0x7f9b3415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.651+0000 7f9b3c000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b340ce820 0x7f9b3414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.651+0000 7f9b3c000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 0x7f9b34153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3c000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3415e480 con 0x7f9b3414e6b0 2024-03-06T07:38:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3c000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3406daf0 con 0x7f9b340ce820 2024-03-06T07:38:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3c000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3415e600 con 0x7f9b1c00b090 2024-03-06T07:38:23.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 0x7f9b34153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 0x7f9b34153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49694/0 (socket says 172.21.15.97:49694) 2024-03-06T07:38:23.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 -- 172.21.15.97:0/63765233 learned_addr learned my addr 172.21.15.97:0/63765233 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:23.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b38c00640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b340ce820 0x7f9b3414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 -- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b1c00b090 msgr2=0x7f9b3415d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b1c00b090 0x7f9b3415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 -- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b340ce820 msgr2=0x7f9b3414e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b340ce820 0x7f9b3414e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.652+0000 7f9b3a000640 1 -- 172.21.15.97:0/63765233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3414eb30 con 0x7f9b3414e6b0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.653+0000 7f9b3a000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 0x7f9b34153d10 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c02c690 tx=0x7f9b2c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.653+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c045070 con 0x7f9b3414e6b0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.653+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c03a040 con 0x7f9b3414e6b0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.653+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c039070 con 0x7f9b3414e6b0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.653+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3414eda0 con 0x7f9b3414e6b0 2024-03-06T07:38:23.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.653+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3414f360 con 0x7f9b3414e6b0 2024-03-06T07:38:23.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.654+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3404f330 con 0x7f9b3414e6b0 2024-03-06T07:38:23.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.656+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b2c048070 con 0x7f9b3414e6b0 2024-03-06T07:38:23.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.657+0000 7f9b31600640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9b1406b4d0 0x7f9b1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.657+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9b2c0abb00 con 0x7f9b3414e6b0 2024-03-06T07:38:23.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.657+0000 7f9b39600640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9b1406b4d0 0x7f9b1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.658+0000 7f9b39600640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9b1406b4d0 0x7f9b1406d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9b28001eb0 tx=0x7f9b28001d80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.662+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b2c07a3a0 con 0x7f9b3414e6b0 2024-03-06T07:38:23.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.927+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b340716a0 con 0x7f9b3414e6b0 2024-03-06T07:38:23.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.928+0000 7f9b31600640 1 -- 172.21.15.97:0/63765233 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f9b2c07e440 con 0x7f9b3414e6b0 2024-03-06T07:38:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:23.931 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:23.931 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.930+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9b1406b4d0 msgr2=0x7f9b1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.930+0000 7f9b3c000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9b1406b4d0 0x7f9b1406d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9b28001eb0 tx=0x7f9b28001d80 comp rx=0 tx=0).stop 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.930+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 msgr2=0x7f9b34153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.930+0000 7f9b3c000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 0x7f9b34153d10 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c02c690 tx=0x7f9b2c009750 comp rx=0 tx=0).stop 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 shutdown_connections 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9b1406b4d0 0x7f9b1406d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b3414e6b0 0x7f9b34153d10 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b340ce820 0x7f9b3414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 --2- 172.21.15.97:0/63765233 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9b1c00b090 0x7f9b3415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 >> 172.21.15.97:0/63765233 conn(0x7f9b340b1730 msgr2=0x7f9b340b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 shutdown_connections 2024-03-06T07:38:23.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:23.931+0000 7f9b3c000640 1 -- 172.21.15.97:0/63765233 wait complete. 2024-03-06T07:38:24.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:24.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:25.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.352+0000 7fd005800640 1 Processor -- start 2024-03-06T07:38:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.352+0000 7fd005800640 1 -- start start 2024-03-06T07:38:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fd005800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd0000b4390 0x7fd000160460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fd005800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd0000cf9a0 0x7fd0000cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fd005800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0000b3040 con 0x7fd00015f770 2024-03-06T07:38:25.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fd005800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00015fb50 con 0x7fd0000cf9a0 2024-03-06T07:38:25.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fd005800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0000cd3e0 con 0x7fd0000b4390 2024-03-06T07:38:25.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fcfff400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd0000cf9a0 0x7fd0000cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fcfff400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd0000cf9a0 0x7fd0000cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38680/0 (socket says 172.21.15.97:38680) 2024-03-06T07:38:25.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.354+0000 7fcfff400640 1 -- 172.21.15.97:0/3638591078 learned_addr learned my addr 172.21.15.97:0/3638591078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:25.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.355+0000 7fcff6000640 1 --2- 172.21.15.97:0/3638591078 >> v2:172.21.15.135:3300/0 conn(0x7fd0000b4390 0x7fd000160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.355+0000 7fcfff400640 1 -- 172.21.15.97:0/3638591078 >> v2:172.21.15.135:3300/0 conn(0x7fd0000b4390 msgr2=0x7fd000160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:25.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.355+0000 7fcfff400640 1 --2- 172.21.15.97:0/3638591078 >> v2:172.21.15.135:3300/0 conn(0x7fd0000b4390 0x7fd000160460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.355+0000 7fcfff400640 1 -- 172.21.15.97:0/3638591078 >> v1:172.21.15.135:6789/0 conn(0x7fd00015f770 legacy=0x7fd00015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:25.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.355+0000 7fcfff400640 1 -- 172.21.15.97:0/3638591078 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0000cd6c0 con 0x7fd0000cf9a0 2024-03-06T07:38:25.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.356+0000 7fcfff400640 1 --2- 172.21.15.97:0/3638591078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd0000cfdd0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fd000160e10 tx=0x7fcff802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfcba38dc59256af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.356+0000 7fcffe000640 1 -- 172.21.15.97:0/3638591078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff802e070 con 0x7fd0000cf9a0 2024-03-06T07:38:25.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.356+0000 7fcffe000640 1 -- 172.21.15.97:0/3638591078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff8037070 con 0x7fd0000cf9a0 2024-03-06T07:38:25.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fcffe000640 1 -- 172.21.15.97:0/3638591078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8031a00 con 0x7fd0000cf9a0 2024-03-06T07:38:25.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 -- 172.21.15.97:0/3638591078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 msgr2=0x7fd0000cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 --2- 172.21.15.97:0/3638591078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd0000cfdd0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fd000160e10 tx=0x7fcff802b880 comp rx=0 tx=0).stop 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 -- 172.21.15.97:0/3638591078 shutdown_connections 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 --2- 172.21.15.97:0/3638591078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd0000cfdd0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 --2- 172.21.15.97:0/3638591078 >> v2:172.21.15.135:3300/0 conn(0x7fd0000b4390 0x7fd000160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 -- 172.21.15.97:0/3638591078 >> 172.21.15.97:0/3638591078 conn(0x7fd0000b3870 msgr2=0x7fd0000b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 -- 172.21.15.97:0/3638591078 shutdown_connections 2024-03-06T07:38:25.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.357+0000 7fd005800640 1 -- 172.21.15.97:0/3638591078 wait complete. 2024-03-06T07:38:25.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.359+0000 7fd005800640 1 Processor -- start 2024-03-06T07:38:25.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.359+0000 7fd005800640 1 -- start start 2024-03-06T07:38:25.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fd005800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0000b4390 0x7fd0000b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fd005800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd000158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fd005800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd00015f770 0x7fd000158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fd005800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000b34a0 con 0x7fd0000cf9a0 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fd005800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000b3320 con 0x7fd00015f770 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fd005800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001609a0 con 0x7fd0000b4390 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fcff6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd000158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fcff6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd000158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38688/0 (socket says 172.21.15.97:38688) 2024-03-06T07:38:25.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.360+0000 7fcff6000640 1 -- 172.21.15.97:0/3352967822 learned_addr learned my addr 172.21.15.97:0/3352967822 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.361+0000 7fcff6000640 1 -- 172.21.15.97:0/3352967822 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0000b4390 msgr2=0x7fd0000b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.361+0000 7fcff6000640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0000b4390 0x7fd0000b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.361+0000 7fcff6000640 1 -- 172.21.15.97:0/3352967822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd00015f770 msgr2=0x7fd000158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.361+0000 7fcff6000640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd00015f770 0x7fd000158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.361+0000 7fcff6000640 1 -- 172.21.15.97:0/3352967822 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00006cfe0 con 0x7fd0000cf9a0 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.361+0000 7fcff6000640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd000158110 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c820 tx=0x7fcfe800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.362+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8015070 con 0x7fd0000cf9a0 2024-03-06T07:38:25.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.362+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe8004220 con 0x7fd0000cf9a0 2024-03-06T07:38:25.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.362+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe800ce10 con 0x7fd0000cf9a0 2024-03-06T07:38:25.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.362+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00006d280 con 0x7fd0000cf9a0 2024-03-06T07:38:25.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.362+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd000157630 con 0x7fd0000cf9a0 2024-03-06T07:38:25.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.364+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00004f330 con 0x7fd0000cf9a0 2024-03-06T07:38:25.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.364+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfe8007830 con 0x7fd0000cf9a0 2024-03-06T07:38:25.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.366+0000 7fcff7400640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.366+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcfe808a080 con 0x7fd0000cf9a0 2024-03-06T07:38:25.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.366+0000 7fcffea00640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.367+0000 7fcffea00640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcff000abb0 tx=0x7fcff0009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.370+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfe8058ed0 con 0x7fd0000cf9a0 2024-03-06T07:38:25.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.598+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0000ce130 con 0x7fd0000cf9a0 2024-03-06T07:38:25.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.599+0000 7fcff7400640 1 -- 172.21.15.97:0/3352967822 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7fcfe80587d0 con 0x7fd0000cf9a0 2024-03-06T07:38:25.603 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:25.603 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:25.604 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:25.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.601+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfdc06b500 msgr2=0x7fcfdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.601+0000 7fd005800640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcff000abb0 tx=0x7fcff0009250 comp rx=0 tx=0).stop 2024-03-06T07:38:25.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.601+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 msgr2=0x7fd000158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.601+0000 7fd005800640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd000158110 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c820 tx=0x7fcfe800cce0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 shutdown_connections 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd00015f770 0x7fd000158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd0000cf9a0 0x7fd000158110 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 --2- 172.21.15.97:0/3352967822 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd0000b4390 0x7fd0000b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 >> 172.21.15.97:0/3352967822 conn(0x7fd0000b3870 msgr2=0x7fd00015cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 shutdown_connections 2024-03-06T07:38:25.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:25.602+0000 7fd005800640 1 -- 172.21.15.97:0/3352967822 wait complete. 2024-03-06T07:38:25.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:25.824 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:26.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:27.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.037+0000 7f006f200640 1 Processor -- start 2024-03-06T07:38:27.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.038+0000 7f006f200640 1 -- start start 2024-03-06T07:38:27.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.040+0000 7f006f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f00700d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.040+0000 7f006f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f00700d18e0 0x7f00700d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.040+0000 7f006f200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007015f030 con 0x7f00700d0600 2024-03-06T07:38:27.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.040+0000 7f006f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00700cf090 con 0x7f00700d18e0 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.040+0000 7f006f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007015f310 con 0x7f00700b2a60 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.040+0000 7f006e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f00700d18e0 0x7f00700d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f006e800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f00700d18e0 0x7f00700d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38694/0 (socket says 172.21.15.97:38694) 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f006e800640 1 -- 172.21.15.97:0/1256791978 learned_addr learned my addr 172.21.15.97:0/1256791978 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f0065400640 1 --2- 172.21.15.97:0/1256791978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f00700d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f006e800640 1 -- 172.21.15.97:0/1256791978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 msgr2=0x7f00700d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f006e800640 1 --2- 172.21.15.97:0/1256791978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f00700d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f006e800640 1 -- 172.21.15.97:0/1256791978 >> v1:172.21.15.135:6789/0 conn(0x7f00700d0600 legacy=0x7f00700d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.041+0000 7f006e800640 1 -- 172.21.15.97:0/1256791978 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700cea40 con 0x7f00700d18e0 2024-03-06T07:38:27.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.042+0000 7f006e800640 1 --2- 172.21.15.97:0/1256791978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700d1d40 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f0068009e50 tx=0x7f006802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f9f72503bd2e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:27.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.042+0000 7f006d400640 1 -- 172.21.15.97:0/1256791978 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006802ca70 con 0x7f00700d18e0 2024-03-06T07:38:27.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.042+0000 7f006d400640 1 -- 172.21.15.97:0/1256791978 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006802cc10 con 0x7f00700d18e0 2024-03-06T07:38:27.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.042+0000 7f006d400640 1 -- 172.21.15.97:0/1256791978 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00680343f0 con 0x7f00700d18e0 2024-03-06T07:38:27.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.043+0000 7f006f200640 1 -- 172.21.15.97:0/1256791978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 msgr2=0x7f00700d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.043+0000 7f006f200640 1 --2- 172.21.15.97:0/1256791978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700d1d40 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f0068009e50 tx=0x7f006802bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.044+0000 7f006f200640 1 -- 172.21.15.97:0/1256791978 shutdown_connections 2024-03-06T07:38:27.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.044+0000 7f006f200640 1 --2- 172.21.15.97:0/1256791978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700d1d40 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.044+0000 7f006f200640 1 --2- 172.21.15.97:0/1256791978 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f00700d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.044+0000 7f006f200640 1 -- 172.21.15.97:0/1256791978 >> 172.21.15.97:0/1256791978 conn(0x7f00700d3d10 msgr2=0x7f00700d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:27.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.044+0000 7f006f200640 1 -- 172.21.15.97:0/1256791978 shutdown_connections 2024-03-06T07:38:27.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.044+0000 7f006f200640 1 -- 172.21.15.97:0/1256791978 wait complete. 2024-03-06T07:38:27.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.046+0000 7f006f200640 1 Processor -- start 2024-03-06T07:38:27.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.046+0000 7f006f200640 1 -- start start 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f0070160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00700d0600 0x7f00700c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007015ecf0 con 0x7f00700d18e0 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00700b3250 con 0x7f00700d0600 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00700cf520 con 0x7f00700b2a60 2024-03-06T07:38:27.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700b91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38708/0 (socket says 172.21.15.97:38708) 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.048+0000 7f006e800640 1 -- 172.21.15.97:0/2008307771 learned_addr learned my addr 172.21.15.97:0/2008307771 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006de00640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f0070160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f0065400640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00700d0600 0x7f00700c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006e800640 1 -- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 msgr2=0x7f0070160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006e800640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f0070160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006e800640 1 -- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00700d0600 msgr2=0x7f00700c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006e800640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00700d0600 0x7f00700c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006e800640 1 -- 172.21.15.97:0/2008307771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700c8670 con 0x7f00700d18e0 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006e800640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700b91b0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0068036040 tx=0x7f0068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068043020 con 0x7f00700d18e0 2024-03-06T07:38:27.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006803e450 con 0x7f00700d18e0 2024-03-06T07:38:27.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00700c8950 con 0x7f00700d18e0 2024-03-06T07:38:27.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.049+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068009510 con 0x7f00700d18e0 2024-03-06T07:38:27.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.050+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00700c8f10 con 0x7f00700d18e0 2024-03-06T07:38:27.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.051+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007004f330 con 0x7f00700d18e0 2024-03-06T07:38:27.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.052+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0068046020 con 0x7f00700d18e0 2024-03-06T07:38:27.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.053+0000 7f0066a00640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f004c06b4d0 0x7f004c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.053+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f00680ac9d0 con 0x7f00700d18e0 2024-03-06T07:38:27.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.053+0000 7f006de00640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f004c06b4d0 0x7f004c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.054+0000 7f006de00640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f004c06b4d0 0x7f004c06d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f00700d28b0 tx=0x7f0060009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:27.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.058+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f006804d030 con 0x7f00700d18e0 2024-03-06T07:38:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.298+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f007008d4d0 con 0x7f00700d18e0 2024-03-06T07:38:27.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.299+0000 7f0066a00640 1 -- 172.21.15.97:0/2008307771 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f006807b820 con 0x7f00700d18e0 2024-03-06T07:38:27.302 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:27.302 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:27.302 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f004c06b4d0 msgr2=0x7f004c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f004c06b4d0 0x7f004c06d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f00700d28b0 tx=0x7f0060009040 comp rx=0 tx=0).stop 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 msgr2=0x7f00700b91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700b91b0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0068036040 tx=0x7f0068004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 shutdown_connections 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f004c06b4d0 0x7f004c06d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00700d18e0 0x7f00700b91b0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00700d0600 0x7f00700c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 --2- 172.21.15.97:0/2008307771 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f00700b2a60 0x7f0070160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 >> 172.21.15.97:0/2008307771 conn(0x7f00700d3d10 msgr2=0x7f007015dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 shutdown_connections 2024-03-06T07:38:27.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:27.302+0000 7f006f200640 1 -- 172.21.15.97:0/2008307771 wait complete. 2024-03-06T07:38:27.528 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:27.528 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:28.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:28.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.740+0000 7f9a2d000640 1 Processor -- start 2024-03-06T07:38:28.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.740+0000 7f9a2d000640 1 -- start start 2024-03-06T07:38:28.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.742+0000 7f9a2d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9a28147310 0x7f9a28072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.742+0000 7f9a2d000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280b39c0 con 0x7f9a2806daa0 2024-03-06T07:38:28.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.742+0000 7f9a2d000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280ad9e0 con 0x7f9a2806d390 2024-03-06T07:38:28.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.742+0000 7f9a2d000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2806df30 con 0x7f9a28147310 2024-03-06T07:38:28.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.742+0000 7f9a1d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9a2806daa0 0x7f9a2806cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55158/0 (socket says 172.21.15.97:55158) 2024-03-06T07:38:28.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.742+0000 7f9a1d400640 1 -- 172.21.15.97:0/617051799 learned_addr learned my addr 172.21.15.97:0/617051799 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25e00640 1 --2- 172.21.15.97:0/617051799 >> v2:172.21.15.135:3300/0 conn(0x7f9a28147310 0x7f9a28072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306628094 0 0) 0x7f9a10001110 con 0x7f9a2806daa0 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c003520 con 0x7f9a2806daa0 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745431109 0 0) 0x7f9a0c003520 con 0x7f9a2806daa0 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 >> v2:172.21.15.135:3300/0 conn(0x7f9a28147310 msgr2=0x7f9a28072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 --2- 172.21.15.97:0/617051799 >> v2:172.21.15.135:3300/0 conn(0x7f9a28147310 0x7f9a28072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 >> v1:172.21.15.135:6789/0 conn(0x7f9a2806d390 legacy=0x7f9a280ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a280adcc0 con 0x7f9a2806daa0 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9a10001840 con 0x7f9a2806daa0 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 >> v1:172.21.15.97:6789/0 conn(0x7f9a2806daa0 legacy=0x7f9a2806cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a0c0039b0 0x7f9a0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.743+0000 7f9a25400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a25400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a0c0051c0 0x7f9a0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c00c030 con 0x7f9a0c004800 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c00c1e0 con 0x7f9a0c0051c0 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c00c3e0 con 0x7f9a0c0039b0 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a1d400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a25e00640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a0c0039b0 0x7f9a0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a1d400640 1 -- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a0c0039b0 msgr2=0x7f9a0c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a1d400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a0c0039b0 0x7f9a0c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a1d400640 1 -- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a0c0051c0 msgr2=0x7f9a0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a26800640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a0c0051c0 0x7f9a0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a1d400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a0c0051c0 0x7f9a0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a1d400640 1 -- 172.21.15.97:0/617051799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c003520 con 0x7f9a0c004800 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a26800640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a0c0051c0 0x7f9a0c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:28.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.744+0000 7f9a26800640 1 -- 172.21.15.97:0/617051799 reap_dead start 2024-03-06T07:38:28.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.745+0000 7f9a1d400640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a0c004c80 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c4c0 tx=0x7f9a0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59626bb2b3911d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.745+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000c0e0 con 0x7f9a0c004800 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10030070 con 0x7f9a0c004800 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a25400640 1 -- 172.21.15.97:0/617051799 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a100058d0 con 0x7f9a0c004800 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 -- 172.21.15.97:0/617051799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 msgr2=0x7f9a0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a0c004c80 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c4c0 tx=0x7f9a0c005830 comp rx=0 tx=0).stop 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 -- 172.21.15.97:0/617051799 shutdown_connections 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 --2- 172.21.15.97:0/617051799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a0c004c80 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 -- 172.21.15.97:0/617051799 >> 172.21.15.97:0/617051799 conn(0x7f9a280b1be0 msgr2=0x7f9a280b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 -- 172.21.15.97:0/617051799 shutdown_connections 2024-03-06T07:38:28.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.746+0000 7f9a2d000640 1 -- 172.21.15.97:0/617051799 wait complete. 2024-03-06T07:38:28.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.748+0000 7f9a2d000640 1 Processor -- start 2024-03-06T07:38:28.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.748+0000 7f9a2d000640 1 -- start start 2024-03-06T07:38:28.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a2d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a2813e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a2d000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a2815d8c0 0x7f9a280b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a2d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a2815d310 0x7f9a280b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a2d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a281476f0 con 0x7f9a0c004800 2024-03-06T07:38:28.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a2d000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280ae280 con 0x7f9a2815d8c0 2024-03-06T07:38:28.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a2d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28072340 con 0x7f9a2815d310 2024-03-06T07:38:28.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a25e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a2813e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a25e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a2813e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38720/0 (socket says 172.21.15.97:38720) 2024-03-06T07:38:28.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.749+0000 7f9a25e00640 1 -- 172.21.15.97:0/3939983533 learned_addr learned my addr 172.21.15.97:0/3939983533 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:28.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a1d400640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a2815d8c0 0x7f9a280b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a26800640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a2815d310 0x7f9a280b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a25e00640 1 -- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a2815d310 msgr2=0x7f9a280b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a25e00640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a2815d310 0x7f9a280b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a25e00640 1 -- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a2815d8c0 msgr2=0x7f9a280b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a25e00640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a2815d8c0 0x7f9a280b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a25e00640 1 -- 172.21.15.97:0/3939983533 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2813ec80 con 0x7f9a0c004800 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a1d400640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a2815d8c0 0x7f9a280b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a25e00640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a2813e330 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800f400 tx=0x7f9a1800f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18016620 con 0x7f9a0c004800 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1801b070 con 0x7f9a0c004800 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18016a20 con 0x7f9a0c004800 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.750+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2813ef60 con 0x7f9a0c004800 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.751+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2813f520 con 0x7f9a0c004800 2024-03-06T07:38:28.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.752+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2804f330 con 0x7f9a0c004800 2024-03-06T07:38:28.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.752+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a18005d10 con 0x7f9a0c004800 2024-03-06T07:38:28.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.754+0000 7f9a1ea00640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a0406b4d0 0x7f9a0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.754+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a1808af40 con 0x7f9a0c004800 2024-03-06T07:38:28.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.754+0000 7f9a1d400640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a0406b4d0 0x7f9a0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.755+0000 7f9a1d400640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a0406b4d0 0x7f9a0406d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9a10034000 tx=0x7f9a10039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.758+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a18025020 con 0x7f9a0c004800 2024-03-06T07:38:28.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.997+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a2806cd10 con 0x7f9a0c004800 2024-03-06T07:38:29.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:28.997+0000 7f9a1ea00640 1 -- 172.21.15.97:0/3939983533 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f9a18059e30 con 0x7f9a0c004800 2024-03-06T07:38:29.000 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:29.000 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:29.000 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a0406b4d0 msgr2=0x7f9a0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a0406b4d0 0x7f9a0406d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9a10034000 tx=0x7f9a10039040 comp rx=0 tx=0).stop 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 msgr2=0x7f9a2813e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a2813e330 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800f400 tx=0x7f9a1800f8c0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 shutdown_connections 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9a0406b4d0 0x7f9a0406d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9a2815d310 0x7f9a280b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a2815d8c0 0x7f9a280b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.000+0000 7f9a2d000640 1 --2- 172.21.15.97:0/3939983533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a0c004800 0x7f9a2813e330 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.001+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 >> 172.21.15.97:0/3939983533 conn(0x7f9a280b1be0 msgr2=0x7f9a2815e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:29.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.001+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 shutdown_connections 2024-03-06T07:38:29.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:29.001+0000 7f9a2d000640 1 -- 172.21.15.97:0/3939983533 wait complete. 2024-03-06T07:38:29.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:29.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:30.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:30.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.450+0000 7fd366000640 1 Processor -- start 2024-03-06T07:38:30.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.451+0000 7fd366000640 1 -- start start 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd366000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d24c0 0x7fd3600d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd366000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3600b2de0 0x7fd3600d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd366000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3600d2de0 con 0x7fd3600d07f0 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd366000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3600d4a00 con 0x7fd3600b2de0 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd366000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600cf7e0 con 0x7fd3600d24c0 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd356a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3600b2de0 0x7fd3600d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd35fe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd3600d07f0 0x7fd3600d0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55172/0 (socket says 172.21.15.97:55172) 2024-03-06T07:38:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.452+0000 7fd35fe00640 1 -- 172.21.15.97:0/2656535194 learned_addr learned my addr 172.21.15.97:0/2656535194 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.453+0000 7fd35f400640 1 --2- 172.21.15.97:0/2656535194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d24c0 0x7fd3600d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.453+0000 7fd356a00640 1 -- 172.21.15.97:0/2656535194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d24c0 msgr2=0x7fd3600d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.453+0000 7fd356a00640 1 --2- 172.21.15.97:0/2656535194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d24c0 0x7fd3600d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.453+0000 7fd356a00640 1 -- 172.21.15.97:0/2656535194 >> v1:172.21.15.97:6789/0 conn(0x7fd3600d07f0 legacy=0x7fd3600d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.453+0000 7fd356a00640 1 -- 172.21.15.97:0/2656535194 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3600d4c50 con 0x7fd3600b2de0 2024-03-06T07:38:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.453+0000 7fd35ea00640 1 -- 172.21.15.97:0/2656535194 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631374695 0 0) 0x7fd3600d2de0 con 0x7fd3600d07f0 2024-03-06T07:38:30.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.454+0000 7fd356a00640 1 --2- 172.21.15.97:0/2656535194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600b2de0 0x7fd3600d28a0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fd3480097a0 tx=0x7fd34802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2528c5859add1281 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.454+0000 7fd35ea00640 1 -- 172.21.15.97:0/2656535194 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348031780 con 0x7fd3600b2de0 2024-03-06T07:38:30.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.454+0000 7fd35ea00640 1 -- 172.21.15.97:0/2656535194 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348037070 con 0x7fd3600b2de0 2024-03-06T07:38:30.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.454+0000 7fd35ea00640 1 -- 172.21.15.97:0/2656535194 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348031b40 con 0x7fd3600b2de0 2024-03-06T07:38:30.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 -- 172.21.15.97:0/2656535194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600b2de0 msgr2=0x7fd3600d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 --2- 172.21.15.97:0/2656535194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600b2de0 0x7fd3600d28a0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fd3480097a0 tx=0x7fd34802b820 comp rx=0 tx=0).stop 2024-03-06T07:38:30.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 -- 172.21.15.97:0/2656535194 shutdown_connections 2024-03-06T07:38:30.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 --2- 172.21.15.97:0/2656535194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600b2de0 0x7fd3600d28a0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 --2- 172.21.15.97:0/2656535194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d24c0 0x7fd3600d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 -- 172.21.15.97:0/2656535194 >> 172.21.15.97:0/2656535194 conn(0x7fd3600d3ee0 msgr2=0x7fd3600d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 -- 172.21.15.97:0/2656535194 shutdown_connections 2024-03-06T07:38:30.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.455+0000 7fd366000640 1 -- 172.21.15.97:0/2656535194 wait complete. 2024-03-06T07:38:30.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.458+0000 7fd366000640 1 Processor -- start 2024-03-06T07:38:30.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.458+0000 7fd366000640 1 -- start start 2024-03-06T07:38:30.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.459+0000 7fd366000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3600b2de0 0x7fd3601547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd366000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d07f0 0x7fd3600c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd366000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 0x7fd36015cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd366000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600cfd10 con 0x7fd3600d24c0 2024-03-06T07:38:30.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd366000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600d0080 con 0x7fd3600b2de0 2024-03-06T07:38:30.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd366000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36015e5b0 con 0x7fd3600d07f0 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 0x7fd36015cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 0x7fd36015cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38730/0 (socket says 172.21.15.97:38730) 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 -- 172.21.15.97:0/1923238786 learned_addr learned my addr 172.21.15.97:0/1923238786 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd356a00640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d07f0 0x7fd3600c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35f400640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3600b2de0 0x7fd3601547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 -- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d07f0 msgr2=0x7fd3600c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d07f0 0x7fd3600c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 -- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3600b2de0 msgr2=0x7fd3601547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3600b2de0 0x7fd3601547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35fe00640 1 -- 172.21.15.97:0/1923238786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36015b1b0 con 0x7fd3600d24c0 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.460+0000 7fd35f400640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3600b2de0 0x7fd3601547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.461+0000 7fd35fe00640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 0x7fd36015cee0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fd35800cc00 tx=0x7fd35800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.461+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35800aca0 con 0x7fd3600d24c0 2024-03-06T07:38:30.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.461+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35800ae40 con 0x7fd3600d24c0 2024-03-06T07:38:30.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.461+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36015b490 con 0x7fd3600d24c0 2024-03-06T07:38:30.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.461+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358012600 con 0x7fd3600d24c0 2024-03-06T07:38:30.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.461+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36015ba50 con 0x7fd3600d24c0 2024-03-06T07:38:30.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.462+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36004f330 con 0x7fd3600d24c0 2024-03-06T07:38:30.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.463+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3580127a0 con 0x7fd3600d24c0 2024-03-06T07:38:30.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.465+0000 7fd357e00640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.465+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd35808aa20 con 0x7fd3600d24c0 2024-03-06T07:38:30.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.465+0000 7fd35f400640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.466+0000 7fd35f400640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd350006fd0 tx=0x7fd350008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.469+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3580597e0 con 0x7fd3600d24c0 2024-03-06T07:38:30.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.700+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3600d22a0 con 0x7fd3600d24c0 2024-03-06T07:38:30.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.700+0000 7fd357e00640 1 -- 172.21.15.97:0/1923238786 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7fd3580590e0 con 0x7fd3600d24c0 2024-03-06T07:38:30.703 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:30.703 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:30.703 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:30.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 msgr2=0x7fd33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd350006fd0 tx=0x7fd350008040 comp rx=0 tx=0).stop 2024-03-06T07:38:30.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 msgr2=0x7fd36015cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 0x7fd36015cee0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fd35800cc00 tx=0x7fd35800a060 comp rx=0 tx=0).stop 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 shutdown_connections 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3600d24c0 0x7fd36015cee0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3600d07f0 0x7fd3600c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 --2- 172.21.15.97:0/1923238786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3600b2de0 0x7fd3601547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.703+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 >> 172.21.15.97:0/1923238786 conn(0x7fd3600d3ee0 msgr2=0x7fd3600d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.704+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 shutdown_connections 2024-03-06T07:38:30.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:30.704+0000 7fd366000640 1 -- 172.21.15.97:0/1923238786 wait complete. 2024-03-06T07:38:30.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:30.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:31.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:32.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.138+0000 7f578d000640 1 Processor -- start 2024-03-06T07:38:32.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.138+0000 7f578d000640 1 -- start start 2024-03-06T07:38:32.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f578d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f57880cf430 0x7f578806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f578d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f578806d920 0x7f578815dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f578d000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880b35b0 con 0x7f5788151660 2024-03-06T07:38:32.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f578d000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57880b3ee0 con 0x7f578806d920 2024-03-06T07:38:32.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f578d000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57880b3890 con 0x7f57880cf430 2024-03-06T07:38:32.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f5785e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5788151660 0x7f5788071520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55178/0 (socket says 172.21.15.97:55178) 2024-03-06T07:38:32.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f5785e00640 1 -- 172.21.15.97:0/1879704171 learned_addr learned my addr 172.21.15.97:0/1879704171 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:32.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.141+0000 7f5786800640 1 --2- 172.21.15.97:0/1879704171 >> v2:172.21.15.97:3300/0 conn(0x7f578806d920 0x7f578815dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.142+0000 7f5786800640 1 -- 172.21.15.97:0/1879704171 >> v2:172.21.15.135:3300/0 conn(0x7f57880cf430 msgr2=0x7f578806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.142+0000 7f5786800640 1 --2- 172.21.15.97:0/1879704171 >> v2:172.21.15.135:3300/0 conn(0x7f57880cf430 0x7f578806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.142+0000 7f5786800640 1 -- 172.21.15.97:0/1879704171 >> v1:172.21.15.97:6789/0 conn(0x7f5788151660 legacy=0x7f5788071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.142+0000 7f5786800640 1 -- 172.21.15.97:0/1879704171 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57880b3b70 con 0x7f578806d920 2024-03-06T07:38:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.142+0000 7f5784a00640 1 -- 172.21.15.97:0/1879704171 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983122444 0 0) 0x7f57880b35b0 con 0x7f5788151660 2024-03-06T07:38:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.143+0000 7f5786800640 1 --2- 172.21.15.97:0/1879704171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f578815dff0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f578806c8e0 tx=0x7f578002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22963658dd65f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:32.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.143+0000 7f5784a00640 1 -- 172.21.15.97:0/1879704171 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780039070 con 0x7f578806d920 2024-03-06T07:38:32.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.143+0000 7f5784a00640 1 -- 172.21.15.97:0/1879704171 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578002cb60 con 0x7f578806d920 2024-03-06T07:38:32.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.143+0000 7f5784a00640 1 -- 172.21.15.97:0/1879704171 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780034440 con 0x7f578806d920 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 -- 172.21.15.97:0/1879704171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 msgr2=0x7f578815dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 --2- 172.21.15.97:0/1879704171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f578815dff0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f578806c8e0 tx=0x7f578002bdb0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 -- 172.21.15.97:0/1879704171 shutdown_connections 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 --2- 172.21.15.97:0/1879704171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f578815dff0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 --2- 172.21.15.97:0/1879704171 >> v2:172.21.15.135:3300/0 conn(0x7f57880cf430 0x7f578806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 -- 172.21.15.97:0/1879704171 >> 172.21.15.97:0/1879704171 conn(0x7f57880b1730 msgr2=0x7f57880b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 -- 172.21.15.97:0/1879704171 shutdown_connections 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.144+0000 7f578d000640 1 -- 172.21.15.97:0/1879704171 wait complete. 2024-03-06T07:38:32.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.146+0000 7f578d000640 1 Processor -- start 2024-03-06T07:38:32.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.146+0000 7f578d000640 1 -- start start 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f578d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f57881450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f578d000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57880cf430 0x7f57881f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f578d000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5788151660 0x7f5788146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f578d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578815e6b0 con 0x7f578806d920 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f578d000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578815e530 con 0x7f57880cf430 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f578d000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578815e830 con 0x7f5788151660 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f57881450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f57881450e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38744/0 (socket says 172.21.15.97:38744) 2024-03-06T07:38:32.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 -- 172.21.15.97:0/914233194 learned_addr learned my addr 172.21.15.97:0/914233194 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5786800640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5788151660 0x7f5788146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 -- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5788151660 msgr2=0x7f5788146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785400640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57880cf430 0x7f57881f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5788151660 0x7f5788146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 -- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57880cf430 msgr2=0x7f57881f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57880cf430 0x7f57881f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.147+0000 7f5785e00640 1 -- 172.21.15.97:0/914233194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57881f4680 con 0x7f578806d920 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.148+0000 7f5785e00640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f57881450e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f577800c4d0 tx=0x7f577800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.148+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778018070 con 0x7f578806d920 2024-03-06T07:38:32.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.148+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5778007500 con 0x7f578806d920 2024-03-06T07:38:32.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.148+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778004a20 con 0x7f578806d920 2024-03-06T07:38:32.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.148+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57881f4960 con 0x7f578806d920 2024-03-06T07:38:32.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.148+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5788145d60 con 0x7f578806d920 2024-03-06T07:38:32.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.149+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5788145f60 con 0x7f578806d920 2024-03-06T07:38:32.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.150+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f577801b070 con 0x7f578806d920 2024-03-06T07:38:32.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.151+0000 7f577e000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f576806b4d0 0x7f576806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.151+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5778089620 con 0x7f578806d920 2024-03-06T07:38:32.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.151+0000 7f5785400640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f576806b4d0 0x7f576806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.152+0000 7f5785400640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f576806b4d0 0x7f576806d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f57880b4860 tx=0x7f5774008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:32.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.156+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5788145f60 con 0x7f578806d920 2024-03-06T07:38:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.387+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f578804f330 con 0x7f578806d920 2024-03-06T07:38:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.388+0000 7f577e000640 1 -- 172.21.15.97:0/914233194 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f57780583e0 con 0x7f578806d920 2024-03-06T07:38:32.391 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:32.391 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:32.391 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.390+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f576806b4d0 msgr2=0x7f576806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.390+0000 7f578d000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f576806b4d0 0x7f576806d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f57880b4860 tx=0x7f5774008040 comp rx=0 tx=0).stop 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.390+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 msgr2=0x7f57881450e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.390+0000 7f578d000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f57881450e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f577800c4d0 tx=0x7f577800c990 comp rx=0 tx=0).stop 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 shutdown_connections 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f576806b4d0 0x7f576806d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5788151660 0x7f5788146280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57880cf430 0x7f57881f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 --2- 172.21.15.97:0/914233194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f578806d920 0x7f57881450e0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 >> 172.21.15.97:0/914233194 conn(0x7f57880b1730 msgr2=0x7f578806dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 shutdown_connections 2024-03-06T07:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:32.391+0000 7f578d000640 1 -- 172.21.15.97:0/914233194 wait complete. 2024-03-06T07:38:32.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:32.612 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:33.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:33.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.813+0000 7f2f2aa00640 1 Processor -- start 2024-03-06T07:38:33.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.814+0000 7f2f2aa00640 1 -- start start 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.815+0000 7f2f2aa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2f2415f9d0 0x7f2f2415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.815+0000 7f2f2aa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2406cef0 con 0x7f2f24070b10 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.815+0000 7f2f2aa00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2406d1d0 con 0x7f2f24144c00 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.815+0000 7f2f2aa00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2406d4b0 con 0x7f2f2415f9d0 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.816+0000 7f2f28a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2f24070b10 0x7f2f24070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55180/0 (socket says 172.21.15.97:55180) 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.816+0000 7f2f28a00640 1 -- 172.21.15.97:0/3449803669 learned_addr learned my addr 172.21.15.97:0/3449803669 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.816+0000 7f2f1b400640 1 --2- 172.21.15.97:0/3449803669 >> v2:172.21.15.135:3300/0 conn(0x7f2f2415f9d0 0x7f2f2415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.816+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820061657 0 0) 0x7f2f2406cef0 con 0x7f2f24070b10 2024-03-06T07:38:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.816+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f08003520 con 0x7f2f24070b10 2024-03-06T07:38:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.816+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3176222297 0 0) 0x7f2f08003520 con 0x7f2f24070b10 2024-03-06T07:38:33.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 >> v2:172.21.15.135:3300/0 conn(0x7f2f2415f9d0 msgr2=0x7f2f2415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 --2- 172.21.15.97:0/3449803669 >> v2:172.21.15.135:3300/0 conn(0x7f2f2415f9d0 0x7f2f2415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 >> v1:172.21.15.135:6789/0 conn(0x7f2f24144c00 legacy=0x7f2f2406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2406d790 con 0x7f2f24070b10 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2f1c0019f0 con 0x7f2f24070b10 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 >> v1:172.21.15.97:6789/0 conn(0x7f2f24070b10 legacy=0x7f2f24070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f080039b0 0x7f2f08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f080051c0 0x7f2f0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2406d4b0 con 0x7f2f08004800 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0800c060 con 0x7f2f080039b0 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0800c260 con 0x7f2f080051c0 2024-03-06T07:38:33.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f1b400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f23e00640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f080039b0 0x7f2f08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.817+0000 7f2f28a00640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f080051c0 0x7f2f0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.818+0000 7f2f1b400640 1 -- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f080051c0 msgr2=0x7f2f0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.818+0000 7f2f1b400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f080051c0 0x7f2f0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.818+0000 7f2f1b400640 1 -- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f080039b0 msgr2=0x7f2f08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.818+0000 7f2f1b400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f080039b0 0x7f2f08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.818+0000 7f2f1b400640 1 -- 172.21.15.97:0/3449803669 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f08003520 con 0x7f2f08004800 2024-03-06T07:38:33.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.818+0000 7f2f28a00640 1 -- 172.21.15.97:0/3449803669 reap_dead start 2024-03-06T07:38:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.819+0000 7f2f1b400640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f08004c80 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009ac0 tx=0x7f2f08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94f446b80f08b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c039070 con 0x7f2f08004800 2024-03-06T07:38:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c02c6d0 con 0x7f2f08004800 2024-03-06T07:38:33.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f23400640 1 -- 172.21.15.97:0/3449803669 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c009690 con 0x7f2f08004800 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 -- 172.21.15.97:0/3449803669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 msgr2=0x7f2f08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f08004c80 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009ac0 tx=0x7f2f08005830 comp rx=0 tx=0).stop 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 -- 172.21.15.97:0/3449803669 shutdown_connections 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/3449803669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f08004c80 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 -- 172.21.15.97:0/3449803669 >> 172.21.15.97:0/3449803669 conn(0x7f2f240b1730 msgr2=0x7f2f240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 -- 172.21.15.97:0/3449803669 shutdown_connections 2024-03-06T07:38:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.820+0000 7f2f2aa00640 1 -- 172.21.15.97:0/3449803669 wait complete. 2024-03-06T07:38:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.823+0000 7f2f2aa00640 1 Processor -- start 2024-03-06T07:38:33.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.823+0000 7f2f2aa00640 1 -- start start 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.824+0000 7f2f2aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f2415b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.824+0000 7f2f2aa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f241e5f70 0x7f2f2415e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.824+0000 7f2f2aa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f2415ea90 0x7f2f241315a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.824+0000 7f2f2aa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2406ccd0 con 0x7f2f08004800 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.824+0000 7f2f2aa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2406cb50 con 0x7f2f2415ea90 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.824+0000 7f2f2aa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2406df90 con 0x7f2f241e5f70 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f2415b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f2415b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38758/0 (socket says 172.21.15.97:38758) 2024-03-06T07:38:33.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 -- 172.21.15.97:0/1197301151 learned_addr learned my addr 172.21.15.97:0/1197301151 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 -- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f241e5f70 msgr2=0x7f2f2415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f1b400640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f241e5f70 0x7f2f2415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f241e5f70 0x7f2f2415e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 -- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f2415ea90 msgr2=0x7f2f241315a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f2415ea90 0x7f2f241315a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 -- 172.21.15.97:0/1197301151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2415bdd0 con 0x7f2f08004800 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.825+0000 7f2f23e00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f2415b890 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400f5f0 tx=0x7f2f1400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.826+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f140052e0 con 0x7f2f08004800 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.826+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f14018070 con 0x7f2f08004800 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.826+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14003810 con 0x7f2f08004800 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.826+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f24130500 con 0x7f2f08004800 2024-03-06T07:38:33.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.826+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f24130ac0 con 0x7f2f08004800 2024-03-06T07:38:33.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.827+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2404f330 con 0x7f2f08004800 2024-03-06T07:38:33.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.829+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f140039b0 con 0x7f2f08004800 2024-03-06T07:38:33.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.830+0000 7f2f20c00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f0006b4d0 0x7f2f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.830+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2f1408a0e0 con 0x7f2f08004800 2024-03-06T07:38:33.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.831+0000 7f2f1b400640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f0006b4d0 0x7f2f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.831+0000 7f2f1b400640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f0006b4d0 0x7f2f0006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c036000 tx=0x7f2f0c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:33.836+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f2415bdd0 con 0x7f2f08004800 2024-03-06T07:38:34.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.129+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f24145250 con 0x7f2f08004800 2024-03-06T07:38:34.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.130+0000 7f2f20c00640 1 -- 172.21.15.97:0/1197301151 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7f2f14058e10 con 0x7f2f08004800 2024-03-06T07:38:34.133 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:34.133 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:34.133 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:34.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f0006b4d0 msgr2=0x7f2f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f0006b4d0 0x7f2f0006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c036000 tx=0x7f2f0c030000 comp rx=0 tx=0).stop 2024-03-06T07:38:34.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 msgr2=0x7f2f2415b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f2415b890 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400f5f0 tx=0x7f2f1400fab0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 shutdown_connections 2024-03-06T07:38:34.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f0006b4d0 0x7f2f0006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f2415ea90 0x7f2f241315a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f241e5f70 0x7f2f2415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 --2- 172.21.15.97:0/1197301151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f08004800 0x7f2f2415b890 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.132+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 >> 172.21.15.97:0/1197301151 conn(0x7f2f240b1730 msgr2=0x7f2f240afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:34.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.133+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 shutdown_connections 2024-03-06T07:38:34.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:34.133+0000 7f2f2aa00640 1 -- 172.21.15.97:0/1197301151 wait complete. 2024-03-06T07:38:34.353 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:34.353 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:35.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:35.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.582+0000 7fe8f5200640 1 Processor -- start 2024-03-06T07:38:35.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.582+0000 7fe8f5200640 1 -- start start 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.583+0000 7fe8f5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 0x7fe8f015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8f5200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe8f0070b10 0x7fe8f0070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8f5200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f006cef0 con 0x7fe8f0144c00 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8f5200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f0160370 con 0x7fe8f0070b10 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8f5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f006d1d0 con 0x7fe8f015f9d0 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8ede00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe8f0144c00 0x7fe8f006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57992/0 (socket says 172.21.15.97:57992) 2024-03-06T07:38:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8ede00640 1 -- 172.21.15.97:0/1562130162 learned_addr learned my addr 172.21.15.97:0/1562130162 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8ee800640 1 --2- 172.21.15.97:0/1562130162 >> v2:172.21.15.135:3300/0 conn(0x7fe8f0070b10 0x7fe8f0070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8ed400640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 0x7fe8f015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418822671 0 0) 0x7fe8f006cef0 con 0x7fe8f0144c00 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.584+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d4003520 con 0x7fe8f0144c00 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340913865 0 0) 0x7fe8d4003520 con 0x7fe8f0144c00 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 msgr2=0x7fe8f015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 0x7fe8f015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 >> v2:172.21.15.135:3300/0 conn(0x7fe8f0070b10 msgr2=0x7fe8f0070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 --2- 172.21.15.97:0/1562130162 >> v2:172.21.15.135:3300/0 conn(0x7fe8f0070b10 0x7fe8f0070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f006d4b0 con 0x7fe8f0144c00 2024-03-06T07:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe8e0001670 con 0x7fe8f0144c00 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 >> v1:172.21.15.97:6789/0 conn(0x7fe8f0144c00 legacy=0x7fe8f006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 0x7fe8d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8d4003f80 0x7fe8d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8d4004d40 0x7fe8d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f006d1d0 con 0x7fe8d4004d40 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d400b9c0 con 0x7fe8d4003f80 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d400bb40 con 0x7fe8f015f9d0 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8ee800640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8d4004d40 0x7fe8d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.585+0000 7fe8ed400640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8d4003f80 0x7fe8d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ede00640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 0x7fe8d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ee800640 1 -- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 msgr2=0x7fe8d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ee800640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f015f9d0 0x7fe8d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ee800640 1 -- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8d4003f80 msgr2=0x7fe8d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ee800640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8d4003f80 0x7fe8d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ee800640 1 -- 172.21.15.97:0/1562130162 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d4003520 con 0x7fe8d4004d40 2024-03-06T07:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.586+0000 7fe8ed400640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8d4003f80 0x7fe8d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:35.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.587+0000 7fe8ee800640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8d4004d40 0x7fe8d40051c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800e170 tx=0x7fe8d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca759938b3816d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.587+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8039070 con 0x7fe8d4004d40 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.587+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e802cc20 con 0x7fe8d4004d40 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.587+0000 7fe8eca00640 1 -- 172.21.15.97:0/1562130162 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8009c10 con 0x7fe8d4004d40 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8f5200640 1 -- 172.21.15.97:0/1562130162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8d4004d40 msgr2=0x7fe8d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1562130162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8d4004d40 0x7fe8d40051c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800e170 tx=0x7fe8d4005830 comp rx=0 tx=0).stop 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8ee800640 1 -- 172.21.15.97:0/1562130162 reap_dead start 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8f5200640 1 -- 172.21.15.97:0/1562130162 shutdown_connections 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8f5200640 1 -- 172.21.15.97:0/1562130162 >> 172.21.15.97:0/1562130162 conn(0x7fe8f00b1730 msgr2=0x7fe8f00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8f5200640 1 -- 172.21.15.97:0/1562130162 shutdown_connections 2024-03-06T07:38:35.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.588+0000 7fe8f5200640 1 -- 172.21.15.97:0/1562130162 wait complete. 2024-03-06T07:38:35.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.591+0000 7fe8f5200640 1 Processor -- start 2024-03-06T07:38:35.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.591+0000 7fe8f5200640 1 -- start start 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8f5200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8f015b3c0 0x7fe8f015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8f5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 0x7fe8f014c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8f5200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f014cb50 0x7fe8f00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8f5200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f006cb50 con 0x7fe8f015d8d0 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8f5200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0160730 con 0x7fe8f015b3c0 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8f5200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f006ccd0 con 0x7fe8f014cb50 2024-03-06T07:38:35.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 0x7fe8f014c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 0x7fe8f014c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50440/0 (socket says 172.21.15.97:50440) 2024-03-06T07:38:35.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 -- 172.21.15.97:0/1171925349 learned_addr learned my addr 172.21.15.97:0/1171925349 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ee800640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f014cb50 0x7fe8f00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 -- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f014cb50 msgr2=0x7fe8f00ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f014cb50 0x7fe8f00ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 -- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8f015b3c0 msgr2=0x7fe8f015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ede00640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8f015b3c0 0x7fe8f015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8f015b3c0 0x7fe8f015d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.593+0000 7fe8ed400640 1 -- 172.21.15.97:0/1171925349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f014d360 con 0x7fe8f015d8d0 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.594+0000 7fe8ed400640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 0x7fe8f014c610 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc011540 tx=0x7fe8dc011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.594+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc015870 con 0x7fe8f015d8d0 2024-03-06T07:38:35.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.594+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc015a10 con 0x7fe8f015d8d0 2024-03-06T07:38:35.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.594+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc015dc0 con 0x7fe8f015d8d0 2024-03-06T07:38:35.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.594+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f014a720 con 0x7fe8f015d8d0 2024-03-06T07:38:35.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.594+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f014cfd0 con 0x7fe8f015d8d0 2024-03-06T07:38:35.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.595+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f014a510 con 0x7fe8f015d8d0 2024-03-06T07:38:35.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.596+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8dc00f460 con 0x7fe8f015d8d0 2024-03-06T07:38:35.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.597+0000 7fe8e6000640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe8d006b4d0 0x7fe8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.597+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe8dc08ac00 con 0x7fe8f015d8d0 2024-03-06T07:38:35.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.597+0000 7fe8ede00640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe8d006b4d0 0x7fe8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.598+0000 7fe8ede00640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe8d006b4d0 0x7fe8d006d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0001050 tx=0x7fe8e0001430 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.602+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8dc019030 con 0x7fe8f015d8d0 2024-03-06T07:38:35.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.835+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f004f330 con 0x7fe8f015d8d0 2024-03-06T07:38:35.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.836+0000 7fe8e6000640 1 -- 172.21.15.97:0/1171925349 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3421 (secure 0 0 0) 0x7fe8dc0599c0 con 0x7fe8f015d8d0 2024-03-06T07:38:35.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:35.838 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:04.694904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:35.838 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 115 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe8d006b4d0 msgr2=0x7fe8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe8d006b4d0 0x7fe8d006d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0001050 tx=0x7fe8e0001430 comp rx=0 tx=0).stop 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 msgr2=0x7fe8f014c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 0x7fe8f014c610 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc011540 tx=0x7fe8dc011a00 comp rx=0 tx=0).stop 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 shutdown_connections 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe8d006b4d0 0x7fe8d006d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8f014cb50 0x7fe8f00ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8f015d8d0 0x7fe8f014c610 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 --2- 172.21.15.97:0/1171925349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8f015b3c0 0x7fe8f015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.838+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 >> 172.21.15.97:0/1171925349 conn(0x7fe8f00b1730 msgr2=0x7fe8f00afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.839+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 shutdown_connections 2024-03-06T07:38:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:35.839+0000 7fe8f5200640 1 -- 172.21.15.97:0/1171925349 wait complete. 2024-03-06T07:38:36.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:04.694904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:36.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:37.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.283+0000 7fc5aa000640 1 Processor -- start 2024-03-06T07:38:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.283+0000 7fc5aa000640 1 -- start start 2024-03-06T07:38:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.284+0000 7fc5aa000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5a415a940 0x7fc5a40b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.284+0000 7fc5aa000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc5a40cc2a0 0x7fc5a40cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5aa000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a40b2b30 con 0x7fc5a40cf370 2024-03-06T07:38:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5aa000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a40b28e0 con 0x7fc5a415a940 2024-03-06T07:38:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5aa000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a40b2e10 con 0x7fc5a40cc2a0 2024-03-06T07:38:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5a415a940 0x7fc5a40b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5a415a940 0x7fc5a40b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50444/0 (socket says 172.21.15.97:50444) 2024-03-06T07:38:37.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 -- 172.21.15.97:0/3929308630 learned_addr learned my addr 172.21.15.97:0/3929308630 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:37.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 -- 172.21.15.97:0/3929308630 >> v2:172.21.15.135:3300/0 conn(0x7fc5a40cc2a0 msgr2=0x7fc5a40cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 --2- 172.21.15.97:0/3929308630 >> v2:172.21.15.135:3300/0 conn(0x7fc5a40cc2a0 0x7fc5a40cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 -- 172.21.15.97:0/3929308630 >> v1:172.21.15.135:6789/0 conn(0x7fc5a40cf370 legacy=0x7fc5a406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.285+0000 7fc5a8c00640 1 -- 172.21.15.97:0/3929308630 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a406ca50 con 0x7fc5a415a940 2024-03-06T07:38:37.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.286+0000 7fc5a8c00640 1 --2- 172.21.15.97:0/3929308630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a415a940 0x7fc5a40b45e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fc5a406ec20 tx=0x7fc59402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197a443f6e4494a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5a3400640 1 -- 172.21.15.97:0/3929308630 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594035020 con 0x7fc5a415a940 2024-03-06T07:38:37.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5a3400640 1 -- 172.21.15.97:0/3929308630 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594031870 con 0x7fc5a415a940 2024-03-06T07:38:37.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5a3400640 1 -- 172.21.15.97:0/3929308630 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594031c40 con 0x7fc5a415a940 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 -- 172.21.15.97:0/3929308630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a415a940 msgr2=0x7fc5a40b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3929308630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a415a940 0x7fc5a40b45e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fc5a406ec20 tx=0x7fc59402b910 comp rx=0 tx=0).stop 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 -- 172.21.15.97:0/3929308630 shutdown_connections 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3929308630 >> v2:172.21.15.135:3300/0 conn(0x7fc5a40cc2a0 0x7fc5a40cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3929308630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a415a940 0x7fc5a40b45e0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 -- 172.21.15.97:0/3929308630 >> 172.21.15.97:0/3929308630 conn(0x7fc5a406afc0 msgr2=0x7fc5a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.287+0000 7fc5aa000640 1 -- 172.21.15.97:0/3929308630 shutdown_connections 2024-03-06T07:38:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.288+0000 7fc5aa000640 1 -- 172.21.15.97:0/3929308630 wait complete. 2024-03-06T07:38:37.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.291+0000 7fc5aa000640 1 Processor -- start 2024-03-06T07:38:37.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.291+0000 7fc5aa000640 1 -- start start 2024-03-06T07:38:37.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5aa000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5a40cc2a0 0x7fc5a414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 0x7fc5a40aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5aa000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5a415a940 0x7fc5a40af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5aa000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a40b4b20 con 0x7fc5a40cf370 2024-03-06T07:38:37.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5aa000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a40b30f0 con 0x7fc5a415a940 2024-03-06T07:38:37.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5aa000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a415ad20 con 0x7fc5a40cc2a0 2024-03-06T07:38:37.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 0x7fc5a40aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 0x7fc5a40aee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50454/0 (socket says 172.21.15.97:50454) 2024-03-06T07:38:37.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.293+0000 7fc5a3e00640 1 -- 172.21.15.97:0/3905889176 learned_addr learned my addr 172.21.15.97:0/3905889176 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a8c00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5a40cc2a0 0x7fc5a414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a9600640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5a415a940 0x7fc5a40af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a3e00640 1 -- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5a40cc2a0 msgr2=0x7fc5a414ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a3e00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5a40cc2a0 0x7fc5a414ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a3e00640 1 -- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5a415a940 msgr2=0x7fc5a40af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a3e00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5a415a940 0x7fc5a40af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a3e00640 1 -- 172.21.15.97:0/3905889176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4148680 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a3e00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 0x7fc5a40aee90 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fc59000c7a0 tx=0x7fc59000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590017070 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc590003da0 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5900052b0 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a4150740 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.294+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a4150d00 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.295+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a404f330 con 0x7fc5a40cf370 2024-03-06T07:38:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.296+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc590005450 con 0x7fc5a40cf370 2024-03-06T07:38:37.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.298+0000 7fc5a0c00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc58406b4d0 0x7fc58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.298+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc590089850 con 0x7fc5a40cf370 2024-03-06T07:38:37.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.298+0000 7fc5a8c00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc58406b4d0 0x7fc58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.299+0000 7fc5a8c00640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc58406b4d0 0x7fc58406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc594036000 tx=0x7fc59403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.303+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5a4148680 con 0x7fc5a40cf370 2024-03-06T07:38:37.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.553+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5a40ae7d0 con 0x7fc5a40cf370 2024-03-06T07:38:37.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.554+0000 7fc5a0c00640 1 -- 172.21.15.97:0/3905889176 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc590058610 con 0x7fc5a40cf370 2024-03-06T07:38:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:37.556 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:37.557 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.556+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc58406b4d0 msgr2=0x7fc58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.556+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc58406b4d0 0x7fc58406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc594036000 tx=0x7fc59403c040 comp rx=0 tx=0).stop 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.556+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 msgr2=0x7fc5a40aee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.556+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 0x7fc5a40aee90 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fc59000c7a0 tx=0x7fc59000cc60 comp rx=0 tx=0).stop 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 shutdown_connections 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc58406b4d0 0x7fc58406d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5a415a940 0x7fc5a40af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5a40cf370 0x7fc5a40aee90 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 --2- 172.21.15.97:0/3905889176 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc5a40cc2a0 0x7fc5a414ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 >> 172.21.15.97:0/3905889176 conn(0x7fc5a406afc0 msgr2=0x7fc5a40b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 shutdown_connections 2024-03-06T07:38:37.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:37.557+0000 7fc5aa000640 1 -- 172.21.15.97:0/3905889176 wait complete. 2024-03-06T07:38:37.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:37.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:38.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:38.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.987+0000 7f362e400640 1 Processor -- start 2024-03-06T07:38:38.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.988+0000 7f362e400640 1 -- start start 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.989+0000 7f362e400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f362815f9d0 0x7f362815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.989+0000 7f362e400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362806cef0 con 0x7f3628144c00 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.989+0000 7f362e400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362806d1d0 con 0x7f3628070b10 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.989+0000 7f362e400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362806d4b0 con 0x7f362815f9d0 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f361fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f362815f9d0 0x7f362815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f3627400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3628144c00 0x7f362806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57998/0 (socket says 172.21.15.97:57998) 2024-03-06T07:38:38.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f3627400640 1 -- 172.21.15.97:0/3835368274 learned_addr learned my addr 172.21.15.97:0/3835368274 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:38.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f361fe00640 1 -- 172.21.15.97:0/3835368274 >> v1:172.21.15.135:6789/0 conn(0x7f3628070b10 legacy=0x7f3628070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:38.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f361fe00640 1 -- 172.21.15.97:0/3835368274 >> v1:172.21.15.97:6789/0 conn(0x7f3628144c00 legacy=0x7f362806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f361fe00640 1 -- 172.21.15.97:0/3835368274 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362806d790 con 0x7f362815f9d0 2024-03-06T07:38:38.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.990+0000 7f3626a00640 1 -- 172.21.15.97:0/3835368274 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225843773 0 0) 0x7f362806cef0 con 0x7f3628144c00 2024-03-06T07:38:38.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.991+0000 7f361fe00640 1 --2- 172.21.15.97:0/3835368274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f362815f9d0 0x7f362815fe60 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f36100097a0 tx=0x7f361002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5471d1f72759739d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:38.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.992+0000 7f3626a00640 1 -- 172.21.15.97:0/3835368274 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3610035020 con 0x7f362815f9d0 2024-03-06T07:38:38.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.992+0000 7f3626a00640 1 -- 172.21.15.97:0/3835368274 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3610031830 con 0x7f362815f9d0 2024-03-06T07:38:38.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.992+0000 7f3626a00640 1 -- 172.21.15.97:0/3835368274 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3610031c00 con 0x7f362815f9d0 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.992+0000 7f362e400640 1 -- 172.21.15.97:0/3835368274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f362815f9d0 msgr2=0x7f362815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.992+0000 7f362e400640 1 --2- 172.21.15.97:0/3835368274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f362815f9d0 0x7f362815fe60 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f36100097a0 tx=0x7f361002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.993+0000 7f362e400640 1 -- 172.21.15.97:0/3835368274 shutdown_connections 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.993+0000 7f362e400640 1 --2- 172.21.15.97:0/3835368274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f362815f9d0 0x7f362815fe60 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.993+0000 7f362e400640 1 -- 172.21.15.97:0/3835368274 >> 172.21.15.97:0/3835368274 conn(0x7f36280b1730 msgr2=0x7f36280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.993+0000 7f362e400640 1 -- 172.21.15.97:0/3835368274 shutdown_connections 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.993+0000 7f362e400640 1 -- 172.21.15.97:0/3835368274 wait complete. 2024-03-06T07:38:38.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.994+0000 7f362e400640 1 Processor -- start 2024-03-06T07:38:38.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.994+0000 7f362e400640 1 -- start start 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.995+0000 7f362e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3628070b10 0x7f362815add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.995+0000 7f362e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 0x7f36280ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.995+0000 7f362e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f362815f9d0 0x7f36281e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.995+0000 7f362e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362806ccd0 con 0x7f3628144c00 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.995+0000 7f362e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362806cb50 con 0x7f362815f9d0 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.995+0000 7f362e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362806df90 con 0x7f3628070b10 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 0x7f36280ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 0x7f36280ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50478/0 (socket says 172.21.15.97:50478) 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 -- 172.21.15.97:0/3938278799 learned_addr learned my addr 172.21.15.97:0/3938278799 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f3627400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3628070b10 0x7f362815add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 -- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3628070b10 msgr2=0x7f362815add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f3627e00640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f362815f9d0 0x7f36281e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3628070b10 0x7f362815add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 -- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f362815f9d0 msgr2=0x7f36281e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f362815f9d0 0x7f36281e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f361fe00640 1 -- 172.21.15.97:0/3938278799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36281e5c60 con 0x7f3628144c00 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.996+0000 7f3627e00640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f362815f9d0 0x7f36281e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.997+0000 7f361fe00640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 0x7f36280ad860 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f36100097a0 tx=0x7f36100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.997+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3610004230 con 0x7f3628144c00 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.997+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36100043f0 con 0x7f3628144c00 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.997+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36280b4660 con 0x7f3628144c00 2024-03-06T07:38:39.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.997+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3610003850 con 0x7f3628144c00 2024-03-06T07:38:39.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.997+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36280b36c0 con 0x7f3628144c00 2024-03-06T07:38:39.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.999+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362804f330 con 0x7f3628144c00 2024-03-06T07:38:39.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:38.999+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f361004c060 con 0x7f3628144c00 2024-03-06T07:38:39.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.000+0000 7f361f400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f360406b4d0 0x7f360406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.000+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f36100abd20 con 0x7f3628144c00 2024-03-06T07:38:39.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.000+0000 7f3627400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f360406b4d0 0x7f360406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.001+0000 7f3627400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f360406b4d0 0x7f360406d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3618000f60 tx=0x7f3618009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.009+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36100b0070 con 0x7f3628144c00 2024-03-06T07:38:39.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.239+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36280721b0 con 0x7f3628144c00 2024-03-06T07:38:39.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.240+0000 7f361f400640 1 -- 172.21.15.97:0/3938278799 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f3610040070 con 0x7f3628144c00 2024-03-06T07:38:39.242 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:39.243 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:39.243 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.242+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f360406b4d0 msgr2=0x7f360406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.242+0000 7f362e400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f360406b4d0 0x7f360406d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3618000f60 tx=0x7f3618009040 comp rx=0 tx=0).stop 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.242+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 msgr2=0x7f36280ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.242+0000 7f362e400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 0x7f36280ad860 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f36100097a0 tx=0x7f36100046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 shutdown_connections 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f360406b4d0 0x7f360406d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f362815f9d0 0x7f36281e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3628144c00 0x7f36280ad860 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 --2- 172.21.15.97:0/3938278799 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3628070b10 0x7f362815add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 >> 172.21.15.97:0/3938278799 conn(0x7f36280b1730 msgr2=0x7f3628145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:39.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 shutdown_connections 2024-03-06T07:38:39.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:39.243+0000 7f362e400640 1 -- 172.21.15.97:0/3938278799 wait complete. 2024-03-06T07:38:39.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:39.471 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:39.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:39.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:40.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:40.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.686+0000 7f969c000640 1 Processor -- start 2024-03-06T07:38:40.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.686+0000 7f969c000640 1 -- start start 2024-03-06T07:38:40.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f969c000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f96940d09b0 0x7f9694167080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f969c000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9694160b70 con 0x7f96940b4390 2024-03-06T07:38:40.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f969c000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96940b3040 con 0x7f969415f770 2024-03-06T07:38:40.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f969c000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969415fb50 con 0x7f96940d09b0 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f969a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f96940d09b0 0x7f9694167080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f9698c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f96940b4390 0x7f9694160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58002/0 (socket says 172.21.15.97:58002) 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.688+0000 7f9698c00640 1 -- 172.21.15.97:0/3861933432 learned_addr learned my addr 172.21.15.97:0/3861933432 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.689+0000 7f969a000640 1 -- 172.21.15.97:0/3861933432 >> v1:172.21.15.135:6789/0 conn(0x7f969415f770 legacy=0x7f969415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.689+0000 7f969a000640 1 -- 172.21.15.97:0/3861933432 >> v1:172.21.15.97:6789/0 conn(0x7f96940b4390 legacy=0x7f9694160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.689+0000 7f969a000640 1 -- 172.21.15.97:0/3861933432 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96940cd3e0 con 0x7f96940d09b0 2024-03-06T07:38:40.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.689+0000 7f9693e00640 1 -- 172.21.15.97:0/3861933432 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152555831 0 0) 0x7f9694160b70 con 0x7f96940b4390 2024-03-06T07:38:40.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.690+0000 7f969a000640 1 --2- 172.21.15.97:0/3861933432 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96940d09b0 0x7f9694167080 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f968c009d90 tx=0x7f968c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b68c6d36a0a0e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.690+0000 7f9693e00640 1 -- 172.21.15.97:0/3861933432 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c039070 con 0x7f96940d09b0 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f9693e00640 1 -- 172.21.15.97:0/3861933432 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c02cbd0 con 0x7f96940d09b0 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f9693e00640 1 -- 172.21.15.97:0/3861933432 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c034440 con 0x7f96940d09b0 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 -- 172.21.15.97:0/3861933432 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96940d09b0 msgr2=0x7f9694167080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 --2- 172.21.15.97:0/3861933432 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96940d09b0 0x7f9694167080 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f968c009d90 tx=0x7f968c02be20 comp rx=0 tx=0).stop 2024-03-06T07:38:40.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 -- 172.21.15.97:0/3861933432 shutdown_connections 2024-03-06T07:38:40.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 --2- 172.21.15.97:0/3861933432 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96940d09b0 0x7f9694167080 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 -- 172.21.15.97:0/3861933432 >> 172.21.15.97:0/3861933432 conn(0x7f96940b3870 msgr2=0x7f96940b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:40.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 -- 172.21.15.97:0/3861933432 shutdown_connections 2024-03-06T07:38:40.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.691+0000 7f969c000640 1 -- 172.21.15.97:0/3861933432 wait complete. 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.693+0000 7f969c000640 1 Processor -- start 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.693+0000 7f969c000640 1 -- start start 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969c000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96940b4390 0x7f969415a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969c000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96940d09b0 0x7f9694150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969c000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 0x7f96940cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969c000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96940b3320 con 0x7f969415f770 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969c000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96940b2cd0 con 0x7f96940d09b0 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969c000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96940b34a0 con 0x7f96940b4390 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 0x7f96940cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 0x7f96940cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50496/0 (socket says 172.21.15.97:50496) 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.694+0000 7f969a000640 1 -- 172.21.15.97:0/342972801 learned_addr learned my addr 172.21.15.97:0/342972801 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f9699600640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96940b4390 0x7f969415a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f9698c00640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96940d09b0 0x7f9694150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f969a000640 1 -- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96940b4390 msgr2=0x7f969415a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f969a000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96940b4390 0x7f969415a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f969a000640 1 -- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96940d09b0 msgr2=0x7f9694150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f969a000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96940d09b0 0x7f9694150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f969a000640 1 -- 172.21.15.97:0/342972801 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9694150560 con 0x7f969415f770 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f9698c00640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96940d09b0 0x7f9694150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f969a000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 0x7f96940cf9a0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f968c009eb0 tx=0x7f968c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c044030 con 0x7f969415f770 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c02c9d0 con 0x7f969415f770 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.695+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c039040 con 0x7f969415f770 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.696+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96940b5ff0 con 0x7f969415f770 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.696+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96940b65b0 con 0x7f969415f770 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.697+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969404f330 con 0x7f969415f770 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.698+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f968c03f470 con 0x7f969415f770 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.699+0000 7f9691600640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f967406b4d0 0x7f967406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.699+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f968c0acc20 con 0x7f969415f770 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.699+0000 7f9699600640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f967406b4d0 0x7f967406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.700+0000 7f9699600640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f967406b4d0 0x7f967406d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9688006fd0 tx=0x7f9688008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.703+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f968c07ba80 con 0x7f969415f770 2024-03-06T07:38:40.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.938+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96940b7cc0 con 0x7f969415f770 2024-03-06T07:38:40.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.939+0000 7f9691600640 1 -- 172.21.15.97:0/342972801 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f968c07b380 con 0x7f969415f770 2024-03-06T07:38:40.944 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:40.945 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:40.945 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:40.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f967406b4d0 msgr2=0x7f967406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f967406b4d0 0x7f967406d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9688006fd0 tx=0x7f9688008040 comp rx=0 tx=0).stop 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 msgr2=0x7f96940cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 0x7f96940cf9a0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f968c009eb0 tx=0x7f968c004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 shutdown_connections 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f967406b4d0 0x7f967406d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f969415f770 0x7f96940cf9a0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96940d09b0 0x7f9694150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 --2- 172.21.15.97:0/342972801 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f96940b4390 0x7f969415a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 >> 172.21.15.97:0/342972801 conn(0x7f96940b3870 msgr2=0x7f96940d2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 shutdown_connections 2024-03-06T07:38:40.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:40.942+0000 7f969c000640 1 -- 172.21.15.97:0/342972801 wait complete. 2024-03-06T07:38:41.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:41.171 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:42.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:42.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.379+0000 7f9fb6c00640 1 Processor -- start 2024-03-06T07:38:42.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.380+0000 7f9fb6c00640 1 -- start start 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.381+0000 7f9fb6c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 0x7f9fb015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.381+0000 7f9fb6c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb006cef0 con 0x7f9fb0070b10 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.381+0000 7f9fb6c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb006d1d0 con 0x7f9fb0144c00 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.381+0000 7f9fb6c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0160370 con 0x7f9fb015f9d0 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9fb4c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9fb0070b10 0x7f9fb0070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58014/0 (socket says 172.21.15.97:58014) 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9fb4c00640 1 -- 172.21.15.97:0/3302933235 learned_addr learned my addr 172.21.15.97:0/3302933235 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9faf400640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 0x7f9fb015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482774469 0 0) 0x7f9fb006cef0 con 0x7f9fb0070b10 2024-03-06T07:38:42.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f90003520 con 0x7f9fb0070b10 2024-03-06T07:38:42.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549810053 0 0) 0x7f9f90003520 con 0x7f9fb0070b10 2024-03-06T07:38:42.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.382+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 msgr2=0x7f9fb015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 0x7f9fb015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 >> v1:172.21.15.135:6789/0 conn(0x7f9fb0144c00 legacy=0x7f9fb006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb006d4b0 con 0x7f9fb0070b10 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9fa8001670 con 0x7f9fb0070b10 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 >> v1:172.21.15.97:6789/0 conn(0x7f9fb0070b10 legacy=0x7f9fb0070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 0x7f9f900039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f90003f60 0x7f9f90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f90004d40 0x7f9f900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9000ba70 con 0x7f9f90004d40 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9000bc20 con 0x7f9f90003f60 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9000be20 con 0x7f9fb015f9d0 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f90004d40 0x7f9f900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fafe00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 0x7f9f900039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9faf400640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f90003f60 0x7f9f90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fb4c00640 1 -- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 msgr2=0x7f9f900039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9fb015f9d0 0x7f9f900039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fb4c00640 1 -- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f90003f60 msgr2=0x7f9f90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f90003f60 0x7f9f90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.383+0000 7f9fb4c00640 1 -- 172.21.15.97:0/3302933235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f90003520 con 0x7f9f90004d40 2024-03-06T07:38:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.384+0000 7f9faf400640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f90003f60 0x7f9f90004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:42.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.385+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f90004d40 0x7f9f900051c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800c210 tx=0x7f9f90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c25ab9a69d2ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.385+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8007d30 con 0x7f9f90004d40 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.385+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa8030070 con 0x7f9f90004d40 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.385+0000 7f9faea00640 1 -- 172.21.15.97:0/3302933235 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa802cda0 con 0x7f9f90004d40 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb6c00640 1 -- 172.21.15.97:0/3302933235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f90004d40 msgr2=0x7f9f900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/3302933235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f90004d40 0x7f9f900051c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800c210 tx=0x7f9f90005830 comp rx=0 tx=0).stop 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb4c00640 1 -- 172.21.15.97:0/3302933235 reap_dead start 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb6c00640 1 -- 172.21.15.97:0/3302933235 shutdown_connections 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb6c00640 1 -- 172.21.15.97:0/3302933235 >> 172.21.15.97:0/3302933235 conn(0x7f9fb00b1730 msgr2=0x7f9fb00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb6c00640 1 -- 172.21.15.97:0/3302933235 shutdown_connections 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.386+0000 7f9fb6c00640 1 -- 172.21.15.97:0/3302933235 wait complete. 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.388+0000 7f9fb6c00640 1 Processor -- start 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.388+0000 7f9fb6c00640 1 -- start start 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb6c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f90004d40 0x7f9fb013d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb6c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb01e7020 0x7f9fb013d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb6c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 0x7f9fb015b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb6c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb006cb50 con 0x7f9fb015ae40 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb6c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0160730 con 0x7f9fb01e7020 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb6c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb006ccd0 con 0x7f9f90004d40 2024-03-06T07:38:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 0x7f9fb015b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 0x7f9fb015b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50518/0 (socket says 172.21.15.97:50518) 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fb4c00640 1 -- 172.21.15.97:0/2000641179 learned_addr learned my addr 172.21.15.97:0/2000641179 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.389+0000 7f9fafe00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f90004d40 0x7f9fb013d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9faf400640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb01e7020 0x7f9fb013d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fb4c00640 1 -- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f90004d40 msgr2=0x7f9fb013d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f90004d40 0x7f9fb013d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fb4c00640 1 -- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb01e7020 msgr2=0x7f9fb013d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb01e7020 0x7f9fb013d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fb4c00640 1 -- 172.21.15.97:0/2000641179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb01e75e0 con 0x7f9fb015ae40 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fafe00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f90004d40 0x7f9fb013d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.390+0000 7f9fb4c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 0x7f9fb015b2d0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8001970 tx=0x7f9fa802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.391+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8004bf0 con 0x7f9fb015ae40 2024-03-06T07:38:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.391+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa8004d90 con 0x7f9fb015ae40 2024-03-06T07:38:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.391+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8007290 con 0x7f9fb015ae40 2024-03-06T07:38:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.391+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb01e78c0 con 0x7f9fb015ae40 2024-03-06T07:38:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.391+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb01e7e80 con 0x7f9fb015ae40 2024-03-06T07:38:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.392+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb004f330 con 0x7f9fb015ae40 2024-03-06T07:38:42.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.394+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fa8007430 con 0x7f9fb015ae40 2024-03-06T07:38:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.395+0000 7f9fa7e00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f8c06f910 0x7f9f8c071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.395+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9fa80aaab0 con 0x7f9fb015ae40 2024-03-06T07:38:42.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.395+0000 7f9fafe00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f8c06f910 0x7f9f8c071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.396+0000 7f9fafe00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f8c06f910 0x7f9f8c071dd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9fb01e81e0 tx=0x7f9fa0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.404+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fa80799a0 con 0x7f9fb015ae40 2024-03-06T07:38:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.674+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fb004f330 con 0x7f9fb015ae40 2024-03-06T07:38:42.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.675+0000 7f9fa7e00640 1 -- 172.21.15.97:0/2000641179 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f9fa80792a0 con 0x7f9fb015ae40 2024-03-06T07:38:42.677 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:42.677 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:42.677 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.677+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f8c06f910 msgr2=0x7f9f8c071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.677+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f8c06f910 0x7f9f8c071dd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9fb01e81e0 tx=0x7f9fa0009040 comp rx=0 tx=0).stop 2024-03-06T07:38:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.677+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 msgr2=0x7f9fb015b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.677+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 0x7f9fb015b2d0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8001970 tx=0x7f9fa802e070 comp rx=0 tx=0).stop 2024-03-06T07:38:42.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.677+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 shutdown_connections 2024-03-06T07:38:42.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.677+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f8c06f910 0x7f9f8c071dd0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.678+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9fb015ae40 0x7f9fb015b2d0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.678+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9fb01e7020 0x7f9fb013d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.678+0000 7f9fb6c00640 1 --2- 172.21.15.97:0/2000641179 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f90004d40 0x7f9fb013d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.678+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 >> 172.21.15.97:0/2000641179 conn(0x7f9fb00b1730 msgr2=0x7f9fb006e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.678+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 shutdown_connections 2024-03-06T07:38:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:42.678+0000 7f9fb6c00640 1 -- 172.21.15.97:0/2000641179 wait complete. 2024-03-06T07:38:42.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:42.899 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:43.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:44.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.140+0000 7f7434800640 1 Processor -- start 2024-03-06T07:38:44.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.140+0000 7f7434800640 1 -- start start 2024-03-06T07:38:44.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.141+0000 7f7434800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f742c144c00 0x7f742c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.141+0000 7f7434800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c06ee70 con 0x7f742c15f9d0 2024-03-06T07:38:44.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.141+0000 7f7434800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c06cef0 con 0x7f742c15fe60 2024-03-06T07:38:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.141+0000 7f7434800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c06d1d0 con 0x7f742c144c00 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.142+0000 7f7431400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f742c15f9d0 0x7f742c0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:40444/0 (socket says 172.21.15.97:40444) 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.142+0000 7f7431400640 1 -- 172.21.15.97:0/614284811 learned_addr learned my addr 172.21.15.97:0/614284811 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.142+0000 7f7431e00640 1 --2- 172.21.15.97:0/614284811 >> v2:172.21.15.135:3300/0 conn(0x7f742c144c00 0x7f742c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.142+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983996669 0 0) 0x7f741c001110 con 0x7f742c15f9d0 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.142+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7418003520 con 0x7f742c15f9d0 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 997071986 0 0) 0x7f7418003520 con 0x7f742c15f9d0 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 >> v2:172.21.15.135:3300/0 conn(0x7f742c144c00 msgr2=0x7f742c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 --2- 172.21.15.97:0/614284811 >> v2:172.21.15.135:3300/0 conn(0x7f742c144c00 0x7f742c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 >> v1:172.21.15.135:6789/0 conn(0x7f742c15fe60 legacy=0x7f742c0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c06d4b0 con 0x7f742c15f9d0 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f741c001840 con 0x7f742c15f9d0 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 >> v1:172.21.15.97:6789/0 conn(0x7f742c15f9d0 legacy=0x7f742c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74180039b0 0x7f7418003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7418004800 0x7f7418004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74180051c0 0x7f741800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c06d1d0 con 0x7f74180051c0 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741800c060 con 0x7f74180039b0 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741800c260 con 0x7f7418004800 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7432800640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74180051c0 0x7f741800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.143+0000 7f7431e00640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74180039b0 0x7f7418003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.144+0000 7f7431400640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7418004800 0x7f7418004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.144+0000 7f7432800640 1 -- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7418004800 msgr2=0x7f7418004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.144+0000 7f7432800640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7418004800 0x7f7418004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.144+0000 7f7432800640 1 -- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74180039b0 msgr2=0x7f7418003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.144+0000 7f7432800640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74180039b0 0x7f7418003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.144+0000 7f7432800640 1 -- 172.21.15.97:0/614284811 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418003520 con 0x7f74180051c0 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7432800640 1 -- 172.21.15.97:0/614284811 reap_dead start 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7432800640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74180051c0 0x7f741800b9c0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f7428001750 tx=0x7f7418005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e46cbebfaf438d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428001370 con 0x7f74180051c0 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74280052f0 con 0x7f74180051c0 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7430a00640 1 -- 172.21.15.97:0/614284811 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428003830 con 0x7f74180051c0 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7434800640 1 -- 172.21.15.97:0/614284811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74180051c0 msgr2=0x7f741800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.145+0000 7f7434800640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74180051c0 0x7f741800b9c0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f7428001750 tx=0x7f7418005830 comp rx=0 tx=0).stop 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.146+0000 7f7434800640 1 -- 172.21.15.97:0/614284811 shutdown_connections 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.146+0000 7f7434800640 1 --2- 172.21.15.97:0/614284811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74180051c0 0x7f741800b9c0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.146+0000 7f7434800640 1 -- 172.21.15.97:0/614284811 >> 172.21.15.97:0/614284811 conn(0x7f742c0b1730 msgr2=0x7f742c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.146+0000 7f7434800640 1 -- 172.21.15.97:0/614284811 shutdown_connections 2024-03-06T07:38:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.146+0000 7f7434800640 1 -- 172.21.15.97:0/614284811 wait complete. 2024-03-06T07:38:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.149+0000 7f7434800640 1 Processor -- start 2024-03-06T07:38:44.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.149+0000 7f7434800640 1 -- start start 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.150+0000 7f7434800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74180051c0 0x7f742c1f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.150+0000 7f7434800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 0x7f742c0b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.150+0000 7f7434800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742c138c10 0x7f742c139090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.150+0000 7f7434800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c160730 con 0x7f742c15d4b0 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.150+0000 7f7434800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c160270 con 0x7f742c138c10 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.150+0000 7f7434800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c06cb50 con 0x7f74180051c0 2024-03-06T07:38:44.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 0x7f742c0b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 0x7f742c0b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50334/0 (socket says 172.21.15.97:50334) 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 -- 172.21.15.97:0/2102637479 learned_addr learned my addr 172.21.15.97:0/2102637479 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 -- 172.21.15.97:0/2102637479 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74180051c0 msgr2=0x7f742c1f68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74180051c0 0x7f742c1f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 -- 172.21.15.97:0/2102637479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742c138c10 msgr2=0x7f742c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742c138c10 0x7f742c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 -- 172.21.15.97:0/2102637479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0b3e90 con 0x7f742c15d4b0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.151+0000 7f7431400640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 0x7f742c0b34e0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f741c00e7b0 tx=0x7f741c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.152+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c00c870 con 0x7f742c15d4b0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.152+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c018070 con 0x7f742c15d4b0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.152+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c00cc30 con 0x7f742c15d4b0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.152+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c0b4170 con 0x7f742c15d4b0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.152+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c0b4730 con 0x7f742c15d4b0 2024-03-06T07:38:44.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.153+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c04f330 con 0x7f742c15d4b0 2024-03-06T07:38:44.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.154+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f741c01d460 con 0x7f742c15d4b0 2024-03-06T07:38:44.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.155+0000 7f7422000640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f741006b4d0 0x7f741006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.155+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f741c08a1c0 con 0x7f742c15d4b0 2024-03-06T07:38:44.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.155+0000 7f7431e00640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f741006b4d0 0x7f741006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.156+0000 7f7431e00640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f741006b4d0 0x7f741006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7424009fa0 tx=0x7f7424009750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.164+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f742c06ee70 con 0x7f742c15d4b0 2024-03-06T07:38:44.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.395+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f742c072890 con 0x7f742c15d4b0 2024-03-06T07:38:44.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.396+0000 7f7422000640 1 -- 172.21.15.97:0/2102637479 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f741c058f90 con 0x7f742c15d4b0 2024-03-06T07:38:44.399 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:44.399 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:44.399 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f741006b4d0 msgr2=0x7f741006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f741006b4d0 0x7f741006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7424009fa0 tx=0x7f7424009750 comp rx=0 tx=0).stop 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 msgr2=0x7f742c0b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 0x7f742c0b34e0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f741c00e7b0 tx=0x7f741c00ec70 comp rx=0 tx=0).stop 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 shutdown_connections 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f741006b4d0 0x7f741006d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f742c138c10 0x7f742c139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f742c15d4b0 0x7f742c0b34e0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 --2- 172.21.15.97:0/2102637479 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74180051c0 0x7f742c1f68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 >> 172.21.15.97:0/2102637479 conn(0x7f742c0b1730 msgr2=0x7f742c0af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 shutdown_connections 2024-03-06T07:38:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:44.399+0000 7f7434800640 1 -- 172.21.15.97:0/2102637479 wait complete. 2024-03-06T07:38:44.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:44.622 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:45.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:45.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.838+0000 7fa6e5800640 1 Processor -- start 2024-03-06T07:38:45.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.838+0000 7fa6e5800640 1 -- start start 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6e5800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6e0144c00 0x7fa6e006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6e5800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 0x7fa6e00ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6e5800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e006ee70 con 0x7fa6e015f9d0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6e5800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e006cef0 con 0x7fa6e0144c00 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6e5800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006d1d0 con 0x7fa6e015fe60 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6df400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 0x7fa6e00ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6dea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6e0144c00 0x7fa6e006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6dea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6e0144c00 0x7fa6e006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:38466/0 (socket says 172.21.15.97:38466) 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6df400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 0x7fa6e00ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:34114/0 (socket says 172.21.15.97:34114) 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6dea00640 1 -- 172.21.15.97:0/2688260510 learned_addr learned my addr 172.21.15.97:0/2688260510 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.840+0000 7fa6de000640 1 --1- 172.21.15.97:0/2688260510 >> v1:172.21.15.135:6789/0 conn(0x7fa6e015f9d0 0x7fa6e00ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:54782/0 (socket says 172.21.15.97:54782) 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.841+0000 7fa6dea00640 1 -- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 msgr2=0x7fa6e00ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.841+0000 7fa6dea00640 1 --2- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 0x7fa6e00ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.841+0000 7fa6dea00640 1 -- 172.21.15.97:0/2688260510 >> v1:172.21.15.135:6789/0 conn(0x7fa6e015f9d0 legacy=0x7fa6e00ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.841+0000 7fa6dea00640 1 -- 172.21.15.97:0/2688260510 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e006d4b0 con 0x7fa6e0144c00 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.841+0000 7fa6df400640 1 --2- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 0x7fa6e00ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:38:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.844+0000 7fa6dea00640 1 --2- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e006e930 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009870 tx=0x7fa6d002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=728f05e7cd5f2ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.844+0000 7fa6dd600640 1 -- 172.21.15.97:0/2688260510 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0034070 con 0x7fa6e0144c00 2024-03-06T07:38:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.844+0000 7fa6dd600640 1 -- 172.21.15.97:0/2688260510 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d0030840 con 0x7fa6e0144c00 2024-03-06T07:38:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.844+0000 7fa6dd600640 1 -- 172.21.15.97:0/2688260510 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0030c10 con 0x7fa6e0144c00 2024-03-06T07:38:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 -- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 msgr2=0x7fa6e006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 --2- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e006e930 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009870 tx=0x7fa6d002b980 comp rx=0 tx=0).stop 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 -- 172.21.15.97:0/2688260510 shutdown_connections 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 --2- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015fe60 0x7fa6e00ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 --2- 172.21.15.97:0/2688260510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e006e930 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 -- 172.21.15.97:0/2688260510 >> 172.21.15.97:0/2688260510 conn(0x7fa6e00b1730 msgr2=0x7fa6e00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 -- 172.21.15.97:0/2688260510 shutdown_connections 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.845+0000 7fa6e5800640 1 -- 172.21.15.97:0/2688260510 wait complete. 2024-03-06T07:38:45.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.847+0000 7fa6e5800640 1 Processor -- start 2024-03-06T07:38:45.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.847+0000 7fa6e5800640 1 -- start start 2024-03-06T07:38:45.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6e5800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e00b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6e5800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015f9d0 0x7fa6e015b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6e5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 0x7fa6e015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6e5800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0160730 con 0x7fa6e015fe60 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6e5800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0160290 con 0x7fa6e0144c00 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6e5800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006cb50 con 0x7fa6e015f9d0 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6df400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 0x7fa6e015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6df400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 0x7fa6e015b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50340/0 (socket says 172.21.15.97:50340) 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.848+0000 7fa6df400640 1 -- 172.21.15.97:0/1150875955 learned_addr learned my addr 172.21.15.97:0/1150875955 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6dea00640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e00b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6de000640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015f9d0 0x7fa6e015b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6df400640 1 -- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015f9d0 msgr2=0x7fa6e015b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6df400640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015f9d0 0x7fa6e015b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6df400640 1 -- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 msgr2=0x7fa6e00b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6df400640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e00b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6df400640 1 -- 172.21.15.97:0/1150875955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e01e8dd0 con 0x7fa6e015fe60 2024-03-06T07:38:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.849+0000 7fa6df400640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 0x7fa6e015b8d0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800efc0 tx=0x7fa6d8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.850+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8018030 con 0x7fa6e015fe60 2024-03-06T07:38:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.850+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d800cde0 con 0x7fa6e015fe60 2024-03-06T07:38:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.850+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e01e90b0 con 0x7fa6e015fe60 2024-03-06T07:38:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.850+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8003bb0 con 0x7fa6e015fe60 2024-03-06T07:38:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.850+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01e9670 con 0x7fa6e015fe60 2024-03-06T07:38:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.851+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e004f330 con 0x7fa6e015fe60 2024-03-06T07:38:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.852+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6d801b070 con 0x7fa6e015fe60 2024-03-06T07:38:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.853+0000 7fa6d6a00640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa6c006b4d0 0x7fa6c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.853+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa6d808acb0 con 0x7fa6e015fe60 2024-03-06T07:38:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.853+0000 7fa6dea00640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa6c006b4d0 0x7fa6c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.854+0000 7fa6dea00640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa6c006b4d0 0x7fa6c006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0036000 tx=0x7fa6d003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:45.857+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6d8010040 con 0x7fa6e015fe60 2024-03-06T07:38:46.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.095+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e00ceba0 con 0x7fa6e015fe60 2024-03-06T07:38:46.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.096+0000 7fa6d6a00640 1 -- 172.21.15.97:0/1150875955 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fa6d8059b10 con 0x7fa6e015fe60 2024-03-06T07:38:46.098 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:46.098 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa6c006b4d0 msgr2=0x7fa6c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa6c006b4d0 0x7fa6c006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0036000 tx=0x7fa6d003b040 comp rx=0 tx=0).stop 2024-03-06T07:38:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 msgr2=0x7fa6e015b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 0x7fa6e015b8d0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800efc0 tx=0x7fa6d8009f90 comp rx=0 tx=0).stop 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 shutdown_connections 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa6c006b4d0 0x7fa6c006d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6e015fe60 0x7fa6e015b8d0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa6e015f9d0 0x7fa6e015b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 --2- 172.21.15.97:0/1150875955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e0144c00 0x7fa6e00b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 >> 172.21.15.97:0/1150875955 conn(0x7fa6e00b1730 msgr2=0x7fa6e0070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 shutdown_connections 2024-03-06T07:38:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:46.099+0000 7fa6e5800640 1 -- 172.21.15.97:0/1150875955 wait complete. 2024-03-06T07:38:46.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:46.320 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:47.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:47.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.528+0000 7fd305400640 1 Processor -- start 2024-03-06T07:38:47.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.528+0000 7fd305400640 1 -- start start 2024-03-06T07:38:47.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.529+0000 7fd305400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd30015acb0 0x7fd3000b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.529+0000 7fd305400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30015b090 con 0x7fd3000b4680 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.529+0000 7fd305400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000b2a30 con 0x7fd30006ec00 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.529+0000 7fd305400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3000b4360 con 0x7fd30015acb0 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.529+0000 7fd2fe800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd30015acb0 0x7fd3000b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fe800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd30015acb0 0x7fd3000b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50352/0 (socket says 172.21.15.97:50352) 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fe800640 1 -- 172.21.15.97:0/3227534779 learned_addr learned my addr 172.21.15.97:0/3227534779 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fde00640 1 --1- 172.21.15.97:0/3227534779 >> v1:172.21.15.97:6789/0 conn(0x7fd3000b4680 0x7fd30006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:40458/0 (socket says 172.21.15.97:40458) 2024-03-06T07:38:47.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fe800640 1 -- 172.21.15.97:0/3227534779 >> v1:172.21.15.135:6789/0 conn(0x7fd30006ec00 legacy=0x7fd3000ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:47.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fe800640 1 -- 172.21.15.97:0/3227534779 >> v1:172.21.15.97:6789/0 conn(0x7fd3000b4680 legacy=0x7fd30006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fe800640 1 -- 172.21.15.97:0/3227534779 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000b2cf0 con 0x7fd30015acb0 2024-03-06T07:38:47.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.530+0000 7fd2fd400640 1 -- 172.21.15.97:0/3227534779 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417172840 0 0) 0x7fd30015b090 con 0x7fd3000b4680 2024-03-06T07:38:47.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.531+0000 7fd2fe800640 1 --2- 172.21.15.97:0/3227534779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30015acb0 0x7fd3000b37d0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009870 tx=0x7fd2f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b36a29c9302dd5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.531+0000 7fd2fd400640 1 -- 172.21.15.97:0/3227534779 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0034070 con 0x7fd30015acb0 2024-03-06T07:38:47.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd2fd400640 1 -- 172.21.15.97:0/3227534779 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0030860 con 0x7fd30015acb0 2024-03-06T07:38:47.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd2fd400640 1 -- 172.21.15.97:0/3227534779 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0030c30 con 0x7fd30015acb0 2024-03-06T07:38:47.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 -- 172.21.15.97:0/3227534779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30015acb0 msgr2=0x7fd3000b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 --2- 172.21.15.97:0/3227534779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30015acb0 0x7fd3000b37d0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009870 tx=0x7fd2f002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 -- 172.21.15.97:0/3227534779 shutdown_connections 2024-03-06T07:38:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 --2- 172.21.15.97:0/3227534779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30015acb0 0x7fd3000b37d0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 -- 172.21.15.97:0/3227534779 >> 172.21.15.97:0/3227534779 conn(0x7fd3000b1730 msgr2=0x7fd3000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 -- 172.21.15.97:0/3227534779 shutdown_connections 2024-03-06T07:38:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.532+0000 7fd305400640 1 -- 172.21.15.97:0/3227534779 wait complete. 2024-03-06T07:38:47.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.534+0000 7fd305400640 1 Processor -- start 2024-03-06T07:38:47.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.534+0000 7fd305400640 1 -- start start 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd305400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 0x7fd30013fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd305400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3000b4680 0x7fd3001400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd305400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd30015acb0 0x7fd300142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd305400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000b3d10 con 0x7fd30006ec00 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd305400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000b32b0 con 0x7fd3000b4680 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd305400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000b4010 con 0x7fd30015acb0 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd2fe800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 0x7fd30013fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd2fe800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 0x7fd30013fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50366/0 (socket says 172.21.15.97:50366) 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd2fe800640 1 -- 172.21.15.97:0/3204658092 learned_addr learned my addr 172.21.15.97:0/3204658092 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd2fde00640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3000b4680 0x7fd3001400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd2ff200640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd30015acb0 0x7fd300142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.535+0000 7fd2fe800640 1 -- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd30015acb0 msgr2=0x7fd300142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2fe800640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd30015acb0 0x7fd300142be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2fe800640 1 -- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3000b4680 msgr2=0x7fd3001400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2fe800640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3000b4680 0x7fd3001400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2fe800640 1 -- 172.21.15.97:0/3204658092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3001405e0 con 0x7fd30006ec00 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2fde00640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3000b4680 0x7fd3001400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2ff200640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd30015acb0 0x7fd300142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:47.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2fe800640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 0x7fd30013fb60 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009870 tx=0x7fd2f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0034040 con 0x7fd30006ec00 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f003ab90 con 0x7fd30006ec00 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0004030 con 0x7fd30006ec00 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3000c49a0 con 0x7fd30006ec00 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.536+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3000c4f60 con 0x7fd30006ec00 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.538+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30004f330 con 0x7fd30006ec00 2024-03-06T07:38:47.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.538+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2f004b060 con 0x7fd30006ec00 2024-03-06T07:38:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.539+0000 7fd2f6a00640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.539+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd2f00ac320 con 0x7fd30006ec00 2024-03-06T07:38:47.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.539+0000 7fd2fde00640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.540+0000 7fd2fde00640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd3000cbc50 tx=0x7fd2ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.544+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2f003f070 con 0x7fd30006ec00 2024-03-06T07:38:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.800+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd30006e600 con 0x7fd30006ec00 2024-03-06T07:38:47.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.801+0000 7fd2f6a00640 1 -- 172.21.15.97:0/3204658092 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fd2f0038e90 con 0x7fd30006ec00 2024-03-06T07:38:47.803 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:47.803 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:47.804 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2e006b4d0 msgr2=0x7fd2e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd3000cbc50 tx=0x7fd2ec009040 comp rx=0 tx=0).stop 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 msgr2=0x7fd30013fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 0x7fd30013fb60 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009870 tx=0x7fd2f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 shutdown_connections 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd30015acb0 0x7fd300142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3000b4680 0x7fd3001400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 --2- 172.21.15.97:0/3204658092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd30006ec00 0x7fd30013fb60 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 >> 172.21.15.97:0/3204658092 conn(0x7fd3000b1730 msgr2=0x7fd300154360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 shutdown_connections 2024-03-06T07:38:47.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:47.804+0000 7fd305400640 1 -- 172.21.15.97:0/3204658092 wait complete. 2024-03-06T07:38:48.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:48.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:49.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:49.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.252+0000 7ff868c00640 1 Processor -- start 2024-03-06T07:38:49.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.252+0000 7ff868c00640 1 -- start start 2024-03-06T07:38:49.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff868c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415e0d0 0x7ff8640d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff868c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8640d0fc0 0x7ff864160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff868c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640cea60 con 0x7ff86415f470 2024-03-06T07:38:49.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff868c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8640d2f60 con 0x7ff8640d0fc0 2024-03-06T07:38:49.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff868c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640d2350 con 0x7ff86415e0d0 2024-03-06T07:38:49.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff862800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8640d0fc0 0x7ff864160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff862800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff8640d0fc0 0x7ff864160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50376/0 (socket says 172.21.15.97:50376) 2024-03-06T07:38:49.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.254+0000 7ff862800640 1 -- 172.21.15.97:0/271184376 learned_addr learned my addr 172.21.15.97:0/271184376 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:49.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.255+0000 7ff861e00640 1 --2- 172.21.15.97:0/271184376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415e0d0 0x7ff8640d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.255+0000 7ff862800640 1 -- 172.21.15.97:0/271184376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415e0d0 msgr2=0x7ff8640d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.255+0000 7ff862800640 1 --2- 172.21.15.97:0/271184376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415e0d0 0x7ff8640d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.255+0000 7ff862800640 1 -- 172.21.15.97:0/271184376 >> v1:172.21.15.135:6789/0 conn(0x7ff86415f470 legacy=0x7ff8640d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.255+0000 7ff862800640 1 -- 172.21.15.97:0/271184376 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640d2630 con 0x7ff8640d0fc0 2024-03-06T07:38:49.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.256+0000 7ff862800640 1 --2- 172.21.15.97:0/271184376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff864160dc0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009e50 tx=0x7ff85c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0e4eb28e6ad4c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.256+0000 7ff860a00640 1 -- 172.21.15.97:0/271184376 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c02cb00 con 0x7ff8640d0fc0 2024-03-06T07:38:49.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.256+0000 7ff860a00640 1 -- 172.21.15.97:0/271184376 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85c02cca0 con 0x7ff8640d0fc0 2024-03-06T07:38:49.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.256+0000 7ff860a00640 1 -- 172.21.15.97:0/271184376 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c034610 con 0x7ff8640d0fc0 2024-03-06T07:38:49.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 -- 172.21.15.97:0/271184376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 msgr2=0x7ff864160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 --2- 172.21.15.97:0/271184376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff864160dc0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009e50 tx=0x7ff85c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 -- 172.21.15.97:0/271184376 shutdown_connections 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 --2- 172.21.15.97:0/271184376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff864160dc0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 --2- 172.21.15.97:0/271184376 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415e0d0 0x7ff8640d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 -- 172.21.15.97:0/271184376 >> 172.21.15.97:0/271184376 conn(0x7ff8640d3ec0 msgr2=0x7ff8640d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 -- 172.21.15.97:0/271184376 shutdown_connections 2024-03-06T07:38:49.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.257+0000 7ff868c00640 1 -- 172.21.15.97:0/271184376 wait complete. 2024-03-06T07:38:49.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.259+0000 7ff868c00640 1 Processor -- start 2024-03-06T07:38:49.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.259+0000 7ff868c00640 1 -- start start 2024-03-06T07:38:49.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff868c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff86415c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff868c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff86415e0d0 0x7ff8640c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff868c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415f470 0x7ff8640b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff868c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86415fb30 con 0x7ff8640d0fc0 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff868c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640d2bf0 con 0x7ff86415e0d0 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff868c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640d1fd0 con 0x7ff86415f470 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff861e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff86415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff861e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff86415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50386/0 (socket says 172.21.15.97:50386) 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff861e00640 1 -- 172.21.15.97:0/575755621 learned_addr learned my addr 172.21.15.97:0/575755621 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff861400640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff86415e0d0 0x7ff8640c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff862800640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415f470 0x7ff8640b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.260+0000 7ff861e00640 1 -- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415f470 msgr2=0x7ff8640b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff861e00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415f470 0x7ff8640b1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff861e00640 1 -- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff86415e0d0 msgr2=0x7ff8640c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff861e00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff86415e0d0 0x7ff8640c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff861e00640 1 -- 172.21.15.97:0/575755621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640c7fa0 con 0x7ff8640d0fc0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff861400640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff86415e0d0 0x7ff8640c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff861e00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff86415c570 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff85400d800 tx=0x7ff85400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85400b660 con 0x7ff8640d0fc0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854019070 con 0x7ff8640d0fc0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85400ba20 con 0x7ff8640d0fc0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8641f7d70 con 0x7ff8640d0fc0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.261+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8641f8330 con 0x7ff8640d0fc0 2024-03-06T07:38:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.262+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86404f330 con 0x7ff8640d0fc0 2024-03-06T07:38:49.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.264+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff85400bbc0 con 0x7ff8640d0fc0 2024-03-06T07:38:49.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.265+0000 7ff85a000640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff84406b4d0 0x7ff84406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.265+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff85408a750 con 0x7ff8640d0fc0 2024-03-06T07:38:49.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.265+0000 7ff861400640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff84406b4d0 0x7ff84406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.266+0000 7ff861400640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff84406b4d0 0x7ff84406d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff8640b2030 tx=0x7ff850009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.269+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff85401c360 con 0x7ff8640d0fc0 2024-03-06T07:38:49.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.510+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8640b3b30 con 0x7ff8640d0fc0 2024-03-06T07:38:49.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.511+0000 7ff85a000640 1 -- 172.21.15.97:0/575755621 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7ff854059510 con 0x7ff8640d0fc0 2024-03-06T07:38:49.513 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:49.513 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:49.513 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:49.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff84406b4d0 msgr2=0x7ff84406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff84406b4d0 0x7ff84406d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff8640b2030 tx=0x7ff850009040 comp rx=0 tx=0).stop 2024-03-06T07:38:49.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 msgr2=0x7ff86415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff86415c570 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff85400d800 tx=0x7ff85400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 shutdown_connections 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff84406b4d0 0x7ff84406d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff86415f470 0x7ff8640b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff86415e0d0 0x7ff8640c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 --2- 172.21.15.97:0/575755621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff8640d0fc0 0x7ff86415c570 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 >> 172.21.15.97:0/575755621 conn(0x7ff8640d3ec0 msgr2=0x7ff86415e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 shutdown_connections 2024-03-06T07:38:49.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:49.514+0000 7ff868c00640 1 -- 172.21.15.97:0/575755621 wait complete. 2024-03-06T07:38:49.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:49.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:50.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:50.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.947+0000 7f5020e00640 1 Processor -- start 2024-03-06T07:38:50.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.947+0000 7f5020e00640 1 -- start start 2024-03-06T07:38:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5020e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5020e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f501c15f9d0 0x7f501c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5020e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f501c0ce0a0 0x7f501c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5020e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c160370 con 0x7f501c0ce0a0 2024-03-06T07:38:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5020e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c06ee70 con 0x7f501c15f9d0 2024-03-06T07:38:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5020e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c06cef0 con 0x7f501c144c00 2024-03-06T07:38:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f501a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f501c0ce0a0 0x7f501c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f501a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f501c0ce0a0 0x7f501c0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50390/0 (socket says 172.21.15.97:50390) 2024-03-06T07:38:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f501a800640 1 -- 172.21.15.97:0/3574408629 learned_addr learned my addr 172.21.15.97:0/3574408629 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5011400640 1 --2- 172.21.15.97:0/3574408629 >> v2:172.21.15.135:3300/0 conn(0x7f501c15f9d0 0x7f501c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f5019e00640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.949+0000 7f501a800640 1 -- 172.21.15.97:0/3574408629 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 msgr2=0x7f501c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.950+0000 7f501a800640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.950+0000 7f501a800640 1 -- 172.21.15.97:0/3574408629 >> v2:172.21.15.135:3300/0 conn(0x7f501c15f9d0 msgr2=0x7f501c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.950+0000 7f501a800640 1 --2- 172.21.15.97:0/3574408629 >> v2:172.21.15.135:3300/0 conn(0x7f501c15f9d0 0x7f501c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.950+0000 7f501a800640 1 -- 172.21.15.97:0/3574408629 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c06d1d0 con 0x7f501c0ce0a0 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.950+0000 7f5011400640 1 --2- 172.21.15.97:0/3574408629 >> v2:172.21.15.135:3300/0 conn(0x7f501c15f9d0 0x7f501c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.950+0000 7f5019e00640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.951+0000 7f501a800640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0ce4f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f501c0b27f0 tx=0x7f501402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47fbaf18999acf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.951+0000 7f5019400640 1 -- 172.21.15.97:0/3574408629 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014035020 con 0x7f501c0ce0a0 2024-03-06T07:38:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.951+0000 7f5019400640 1 -- 172.21.15.97:0/3574408629 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501402d040 con 0x7f501c0ce0a0 2024-03-06T07:38:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.951+0000 7f5019400640 1 -- 172.21.15.97:0/3574408629 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014031a80 con 0x7f501c0ce0a0 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 -- 172.21.15.97:0/3574408629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 msgr2=0x7f501c0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0ce4f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f501c0b27f0 tx=0x7f501402b900 comp rx=0 tx=0).stop 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 -- 172.21.15.97:0/3574408629 shutdown_connections 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0ce4f0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 --2- 172.21.15.97:0/3574408629 >> v2:172.21.15.135:3300/0 conn(0x7f501c15f9d0 0x7f501c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 --2- 172.21.15.97:0/3574408629 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 -- 172.21.15.97:0/3574408629 >> 172.21.15.97:0/3574408629 conn(0x7f501c0b1730 msgr2=0x7f501c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 -- 172.21.15.97:0/3574408629 shutdown_connections 2024-03-06T07:38:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.952+0000 7f5020e00640 1 -- 172.21.15.97:0/3574408629 wait complete. 2024-03-06T07:38:50.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.954+0000 7f5020e00640 1 Processor -- start 2024-03-06T07:38:50.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.954+0000 7f5020e00640 1 -- start start 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5020e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5020e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5020e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f501c15f9d0 0x7f501c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5020e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c160730 con 0x7f501c0ce0a0 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5020e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c06d790 con 0x7f501c15f9d0 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5020e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c06cb50 con 0x7f501c144c00 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5019e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5019e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50396/0 (socket says 172.21.15.97:50396) 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5019e00640 1 -- 172.21.15.97:0/3613284774 learned_addr learned my addr 172.21.15.97:0/3613284774 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f501a800640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f501c15f9d0 0x7f501c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5019e00640 1 -- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 msgr2=0x7f501c139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.955+0000 7f5011400640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5019e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c139450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5019e00640 1 -- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f501c15f9d0 msgr2=0x7f501c071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5019e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f501c15f9d0 0x7f501c071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5019e00640 1 -- 172.21.15.97:0/3613284774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c1e55c0 con 0x7f501c0ce0a0 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f501a800640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f501c15f9d0 0x7f501c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5011400640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:50.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5019e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0b46f0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f500c00cee0 tx=0x7f500c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c01a070 con 0x7f501c0ce0a0 2024-03-06T07:38:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c00d840 con 0x7f501c0ce0a0 2024-03-06T07:38:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c1e58a0 con 0x7f501c0ce0a0 2024-03-06T07:38:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c00dc10 con 0x7f501c0ce0a0 2024-03-06T07:38:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.956+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c0b3060 con 0x7f501c0ce0a0 2024-03-06T07:38:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.957+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c04f330 con 0x7f501c0ce0a0 2024-03-06T07:38:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.958+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f500c01d070 con 0x7f501c0ce0a0 2024-03-06T07:38:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.959+0000 7f5012a00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.959+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f500c08aa30 con 0x7f501c0ce0a0 2024-03-06T07:38:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.959+0000 7f5011400640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.960+0000 7f5011400640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ff806b4d0 0x7f4ff806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:50.964+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f500c0591a0 con 0x7f501c0ce0a0 2024-03-06T07:38:51.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.226+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f501c15f7b0 con 0x7f501c0ce0a0 2024-03-06T07:38:51.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.227+0000 7f5012a00640 1 -- 172.21.15.97:0/3613284774 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f500c05d240 con 0x7f501c0ce0a0 2024-03-06T07:38:51.229 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:51.229 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:51.229 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ff806b4d0 msgr2=0x7f4ff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ff806b4d0 0x7f4ff806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).stop 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 msgr2=0x7f501c0b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0b46f0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f500c00cee0 tx=0x7f500c00a710 comp rx=0 tx=0).stop 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 shutdown_connections 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f501c15f9d0 0x7f501c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f501c144c00 0x7f501c139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 --2- 172.21.15.97:0/3613284774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f501c0ce0a0 0x7f501c0b46f0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 >> 172.21.15.97:0/3613284774 conn(0x7f501c0b1730 msgr2=0x7f501c145250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:51.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 shutdown_connections 2024-03-06T07:38:51.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:51.230+0000 7f5020e00640 1 -- 172.21.15.97:0/3613284774 wait complete. 2024-03-06T07:38:51.452 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:51.452 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:52.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.660+0000 7f3cd5600640 1 Processor -- start 2024-03-06T07:38:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.660+0000 7f3cd5600640 1 -- start start 2024-03-06T07:38:52.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3cd5600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3cd015e230 0x7f3cd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3cd5600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00cf340 con 0x7f3cd00ce550 2024-03-06T07:38:52.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3cd5600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00cfe60 con 0x7f3cd00b4320 2024-03-06T07:38:52.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3cd5600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd015e610 con 0x7f3cd015e230 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3ccea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3cd015e230 0x7f3cd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3ccea00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3cd015e230 0x7f3cd006edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50404/0 (socket says 172.21.15.97:50404) 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3ccea00640 1 -- 172.21.15.97:0/2245593348 learned_addr learned my addr 172.21.15.97:0/2245593348 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.662+0000 7f3ccf400640 1 --1- 172.21.15.97:0/2245593348 >> v1:172.21.15.97:6789/0 conn(0x7f3cd00ce550 0x7f3cd00cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:40464/0 (socket says 172.21.15.97:40464) 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.663+0000 7f3ccea00640 1 -- 172.21.15.97:0/2245593348 >> v1:172.21.15.135:6789/0 conn(0x7f3cd00b4320 legacy=0x7f3cd00d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.663+0000 7f3ccea00640 1 -- 172.21.15.97:0/2245593348 >> v1:172.21.15.97:6789/0 conn(0x7f3cd00ce550 legacy=0x7f3cd00cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.663+0000 7f3ccea00640 1 -- 172.21.15.97:0/2245593348 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd006f390 con 0x7f3cd015e230 2024-03-06T07:38:52.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.663+0000 7f3ccd600640 1 -- 172.21.15.97:0/2245593348 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704719422 0 0) 0x7f3cd00cf340 con 0x7f3cd00ce550 2024-03-06T07:38:52.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.664+0000 7f3ccea00640 1 --2- 172.21.15.97:0/2245593348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd015e230 0x7f3cd006edb0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009e50 tx=0x7f3cc002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18ec1b7406756ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.664+0000 7f3ccd600640 1 -- 172.21.15.97:0/2245593348 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc002ca80 con 0x7f3cd015e230 2024-03-06T07:38:52.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3ccd600640 1 -- 172.21.15.97:0/2245593348 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc002cc20 con 0x7f3cd015e230 2024-03-06T07:38:52.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3ccd600640 1 -- 172.21.15.97:0/2245593348 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00343f0 con 0x7f3cd015e230 2024-03-06T07:38:52.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 -- 172.21.15.97:0/2245593348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd015e230 msgr2=0x7f3cd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 --2- 172.21.15.97:0/2245593348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd015e230 0x7f3cd006edb0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009e50 tx=0x7f3cc002bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 -- 172.21.15.97:0/2245593348 shutdown_connections 2024-03-06T07:38:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 --2- 172.21.15.97:0/2245593348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd015e230 0x7f3cd006edb0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 -- 172.21.15.97:0/2245593348 >> 172.21.15.97:0/2245593348 conn(0x7f3cd00b3800 msgr2=0x7f3cd00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 -- 172.21.15.97:0/2245593348 shutdown_connections 2024-03-06T07:38:52.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.665+0000 7f3cd5600640 1 -- 172.21.15.97:0/2245593348 wait complete. 2024-03-06T07:38:52.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.667+0000 7f3cd5600640 1 Processor -- start 2024-03-06T07:38:52.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.667+0000 7f3cd5600640 1 -- start start 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cd5600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd00b4320 0x7f3cd015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cd5600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 0x7f3cd00c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cd5600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd015e230 0x7f3cd00c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cd5600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00d06c0 con 0x7f3cd00ce550 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cd5600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00d03e0 con 0x7f3cd015e230 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cd5600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00cca30 con 0x7f3cd00b4320 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 0x7f3cd00c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 0x7f3cd00c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50410/0 (socket says 172.21.15.97:50410) 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.668+0000 7f3cce000640 1 -- 172.21.15.97:0/4081257328 learned_addr learned my addr 172.21.15.97:0/4081257328 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:52.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3ccea00640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd00b4320 0x7f3cd015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cce000640 1 -- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd00b4320 msgr2=0x7f3cd015a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cce000640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd00b4320 0x7f3cd015a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cce000640 1 -- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd015e230 msgr2=0x7f3cd00c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cce000640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd015e230 0x7f3cd00c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cce000640 1 -- 172.21.15.97:0/4081257328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00b5cb0 con 0x7f3cd00ce550 2024-03-06T07:38:52.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cce000640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 0x7f3cd00c8da0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c460 tx=0x7f3cbc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc011790 con 0x7f3cd00ce550 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc017070 con 0x7f3cd00ce550 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd00b5f90 con 0x7f3cd00ce550 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.669+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc011bb0 con 0x7f3cd00ce550 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.670+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd00b6550 con 0x7f3cd00ce550 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.671+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd004f330 con 0x7f3cd00ce550 2024-03-06T07:38:52.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.671+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cbc01c460 con 0x7f3cd00ce550 2024-03-06T07:38:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.672+0000 7f3cc6a00640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b500 0x7f3cb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.672+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3cbc089fe0 con 0x7f3cd00ce550 2024-03-06T07:38:52.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.673+0000 7f3ccea00640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b500 0x7f3cb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.673+0000 7f3ccea00640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b500 0x7f3cb006d9c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0035000 tx=0x7f3cc0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.677+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cbc058e90 con 0x7f3cd00ce550 2024-03-06T07:38:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.915+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cd006b470 con 0x7f3cd00ce550 2024-03-06T07:38:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.915+0000 7f3cc6a00640 1 -- 172.21.15.97:0/4081257328 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f3cbc058790 con 0x7f3cd00ce550 2024-03-06T07:38:52.918 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:52.918 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b500 msgr2=0x7f3cb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b500 0x7f3cb006d9c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0035000 tx=0x7f3cc0030000 comp rx=0 tx=0).stop 2024-03-06T07:38:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 msgr2=0x7f3cd00c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 0x7f3cd00c8da0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c460 tx=0x7f3cbc00c920 comp rx=0 tx=0).stop 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 shutdown_connections 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3cb006b500 0x7f3cb006d9c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3cd015e230 0x7f3cd00c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd00ce550 0x7f3cd00c8da0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 --2- 172.21.15.97:0/4081257328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3cd00b4320 0x7f3cd015a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 >> 172.21.15.97:0/4081257328 conn(0x7f3cd00b3800 msgr2=0x7f3cd00b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 shutdown_connections 2024-03-06T07:38:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:52.918+0000 7f3cd5600640 1 -- 172.21.15.97:0/4081257328 wait complete. 2024-03-06T07:38:53.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:53.140 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:54.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:54.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.353+0000 7f10a3000640 1 Processor -- start 2024-03-06T07:38:54.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.354+0000 7f10a3000640 1 -- start start 2024-03-06T07:38:54.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a3000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f109c15e1d0 0x7f109c06db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a3000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f109c0aec50 0x7f109c167b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a3000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109c06e160 con 0x7f109c06afc0 2024-03-06T07:38:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a3000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f109c06ed70 con 0x7f109c0aec50 2024-03-06T07:38:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a3000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f109c06e440 con 0x7f109c15e1d0 2024-03-06T07:38:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f109c0aec50 0x7f109c167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f109c0aec50 0x7f109c167b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60114/0 (socket says 172.21.15.97:60114) 2024-03-06T07:38:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.355+0000 7f10a1000640 1 -- 172.21.15.97:0/3470132856 learned_addr learned my addr 172.21.15.97:0/3470132856 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:54.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.356+0000 7f1093400640 1 --2- 172.21.15.97:0/3470132856 >> v2:172.21.15.135:3300/0 conn(0x7f109c15e1d0 0x7f109c06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.356+0000 7f10a1000640 1 -- 172.21.15.97:0/3470132856 >> v2:172.21.15.135:3300/0 conn(0x7f109c15e1d0 msgr2=0x7f109c06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:54.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.356+0000 7f10a1000640 1 --2- 172.21.15.97:0/3470132856 >> v2:172.21.15.135:3300/0 conn(0x7f109c15e1d0 0x7f109c06db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.356+0000 7f10a1000640 1 -- 172.21.15.97:0/3470132856 >> v1:172.21.15.97:6789/0 conn(0x7f109c06afc0 legacy=0x7f109c06b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.356+0000 7f10a1000640 1 -- 172.21.15.97:0/3470132856 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c06e720 con 0x7f109c0aec50 2024-03-06T07:38:54.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.358+0000 7f10a1000640 1 --2- 172.21.15.97:0/3470132856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c0aec50 0x7f109c167b10 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f109c071b10 tx=0x7f109402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48a04a6d04993a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.358+0000 7f109b400640 1 -- 172.21.15.97:0/3470132856 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109402cb10 con 0x7f109c0aec50 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.358+0000 7f109b400640 1 -- 172.21.15.97:0/3470132856 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109402ccb0 con 0x7f109c0aec50 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.358+0000 7f109b400640 1 -- 172.21.15.97:0/3470132856 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109402ce60 con 0x7f109c0aec50 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.358+0000 7f10a3000640 1 -- 172.21.15.97:0/3470132856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c0aec50 msgr2=0x7f109c167b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 --2- 172.21.15.97:0/3470132856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c0aec50 0x7f109c167b10 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f109c071b10 tx=0x7f109402bf70 comp rx=0 tx=0).stop 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 -- 172.21.15.97:0/3470132856 shutdown_connections 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 --2- 172.21.15.97:0/3470132856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c0aec50 0x7f109c167b10 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 --2- 172.21.15.97:0/3470132856 >> v2:172.21.15.135:3300/0 conn(0x7f109c15e1d0 0x7f109c06db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 -- 172.21.15.97:0/3470132856 >> 172.21.15.97:0/3470132856 conn(0x7f109c0d0b40 msgr2=0x7f109c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 -- 172.21.15.97:0/3470132856 shutdown_connections 2024-03-06T07:38:54.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.359+0000 7f10a3000640 1 -- 172.21.15.97:0/3470132856 wait complete. 2024-03-06T07:38:54.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.360+0000 7f10a3000640 1 Processor -- start 2024-03-06T07:38:54.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.361+0000 7f10a3000640 1 -- start start 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a3000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f109c06afc0 0x7f109c14dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a3000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0aec50 0x7f109c160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a3000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 0x7f109c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a3000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109c0d1630 con 0x7f109c15e1d0 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a3000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109c0704b0 con 0x7f109c0aec50 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a3000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f109c0d17b0 con 0x7f109c06afc0 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 0x7f109c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 0x7f109c15e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60130/0 (socket says 172.21.15.97:60130) 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 -- 172.21.15.97:0/3115214679 learned_addr learned my addr 172.21.15.97:0/3115214679 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f1093400640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0aec50 0x7f109c160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f109be00640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f109c06afc0 0x7f109c14dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 -- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f109c06afc0 msgr2=0x7f109c14dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f109c06afc0 0x7f109c14dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 -- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0aec50 msgr2=0x7f109c160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0aec50 0x7f109c160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f10a1000640 1 -- 172.21.15.97:0/3115214679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c14e550 con 0x7f109c15e1d0 2024-03-06T07:38:54.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f109be00640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f109c06afc0 0x7f109c14dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:54.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.362+0000 7f1093400640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0aec50 0x7f109c160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:54.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.363+0000 7f10a1000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 0x7f109c15e880 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f1094009f40 tx=0x7f1094004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.363+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094043020 con 0x7f109c15e1d0 2024-03-06T07:38:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.363+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109403e450 con 0x7f109c15e1d0 2024-03-06T07:38:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.363+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109402ce60 con 0x7f109c15e1d0 2024-03-06T07:38:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.363+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109c160f30 con 0x7f109c15e1d0 2024-03-06T07:38:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.363+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f109c15af60 con 0x7f109c15e1d0 2024-03-06T07:38:54.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.364+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109c04f330 con 0x7f109c15e1d0 2024-03-06T07:38:54.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.366+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1094046020 con 0x7f109c15e1d0 2024-03-06T07:38:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.367+0000 7f1098c00640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f107806b4d0 0x7f107806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.367+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f10940ac990 con 0x7f109c15e1d0 2024-03-06T07:38:54.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.367+0000 7f109be00640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f107806b4d0 0x7f107806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.368+0000 7f109be00640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f107806b4d0 0x7f107806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f108c001680 tx=0x7f108c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.372+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f109404d030 con 0x7f109c15e1d0 2024-03-06T07:38:54.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.608+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f109c072db0 con 0x7f109c15e1d0 2024-03-06T07:38:54.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.608+0000 7f1098c00640 1 -- 172.21.15.97:0/3115214679 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f109407b900 con 0x7f109c15e1d0 2024-03-06T07:38:54.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:54.611 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:54.611 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f107806b4d0 msgr2=0x7f107806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f107806b4d0 0x7f107806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f108c001680 tx=0x7f108c009040 comp rx=0 tx=0).stop 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 msgr2=0x7f109c15e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 0x7f109c15e880 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f1094009f40 tx=0x7f1094004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 shutdown_connections 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f107806b4d0 0x7f107806d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f109c15e1d0 0x7f109c15e880 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0aec50 0x7f109c160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 --2- 172.21.15.97:0/3115214679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f109c06afc0 0x7f109c14dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 >> 172.21.15.97:0/3115214679 conn(0x7f109c0d0b40 msgr2=0x7f109c0b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 shutdown_connections 2024-03-06T07:38:54.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:54.611+0000 7f10a3000640 1 -- 172.21.15.97:0/3115214679 wait complete. 2024-03-06T07:38:54.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:54.835 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:55.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:56.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.053+0000 7efed8e00640 1 Processor -- start 2024-03-06T07:38:56.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.053+0000 7efed8e00640 1 -- start start 2024-03-06T07:38:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.054+0000 7efed8e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efed40d0600 0x7efed40d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.054+0000 7efed8e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed415f030 con 0x7efed40d18e0 2024-03-06T07:38:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.054+0000 7efed8e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed415f310 con 0x7efed40b2a60 2024-03-06T07:38:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.054+0000 7efed8e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40cf090 con 0x7efed40d0600 2024-03-06T07:38:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.054+0000 7efed3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efed40d18e0 0x7efed40cf520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37812/0 (socket says 172.21.15.97:37812) 2024-03-06T07:38:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.054+0000 7efed3e00640 1 -- 172.21.15.97:0/74243834 learned_addr learned my addr 172.21.15.97:0/74243834 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:56.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed3400640 1 --2- 172.21.15.97:0/74243834 >> v2:172.21.15.135:3300/0 conn(0x7efed40d0600 0x7efed40d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080937445 0 0) 0x7efed415f030 con 0x7efed40d18e0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8003520 con 0x7efed40d18e0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402172444 0 0) 0x7efeb8003520 con 0x7efed40d18e0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 >> v2:172.21.15.135:3300/0 conn(0x7efed40d0600 msgr2=0x7efed40d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 --2- 172.21.15.97:0/74243834 >> v2:172.21.15.135:3300/0 conn(0x7efed40d0600 0x7efed40d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 >> v1:172.21.15.135:6789/0 conn(0x7efed40b2a60 legacy=0x7efed40d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40cea40 con 0x7efed40d18e0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7efecc0019f0 con 0x7efed40d18e0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 >> v1:172.21.15.97:6789/0 conn(0x7efed40d18e0 legacy=0x7efed40cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efeb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efeb8004800 0x7efeb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efeb80051c0 0x7efeb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40cf090 con 0x7efeb80039b0 2024-03-06T07:38:56.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb800c060 con 0x7efeb8004800 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.055+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb800c260 con 0x7efeb80051c0 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3400640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efeb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efecbe00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efeb8004800 0x7efeb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3400640 1 -- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efeb80051c0 msgr2=0x7efeb800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3400640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efeb80051c0 0x7efeb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3400640 1 -- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efeb8004800 msgr2=0x7efeb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3400640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efeb8004800 0x7efeb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3400640 1 -- 172.21.15.97:0/74243834 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8003520 con 0x7efeb80039b0 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efecbe00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efeb8004800 0x7efeb8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:56.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.056+0000 7efed3e00640 1 -- 172.21.15.97:0/74243834 reap_dead start 2024-03-06T07:38:56.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.057+0000 7efed3400640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efeb8003dd0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7efec4009ac0 tx=0x7efeb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3bb626637919a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.057+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4039070 con 0x7efeb80039b0 2024-03-06T07:38:56.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.057+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec402c6b0 con 0x7efeb80039b0 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.057+0000 7efed2a00640 1 -- 172.21.15.97:0/74243834 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4009690 con 0x7efeb80039b0 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 -- 172.21.15.97:0/74243834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 msgr2=0x7efeb8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efeb8003dd0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7efec4009ac0 tx=0x7efeb8005830 comp rx=0 tx=0).stop 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 -- 172.21.15.97:0/74243834 shutdown_connections 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 --2- 172.21.15.97:0/74243834 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efeb8003dd0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 -- 172.21.15.97:0/74243834 >> 172.21.15.97:0/74243834 conn(0x7efed40d3d10 msgr2=0x7efed40d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 -- 172.21.15.97:0/74243834 shutdown_connections 2024-03-06T07:38:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.058+0000 7efed8e00640 1 -- 172.21.15.97:0/74243834 wait complete. 2024-03-06T07:38:56.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.060+0000 7efed8e00640 1 Processor -- start 2024-03-06T07:38:56.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.060+0000 7efed8e00640 1 -- start start 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efed41f8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed8e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed40b6930 0x7efed4158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed8e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efed40b74d0 0x7efed40b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed8e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed415ecf0 con 0x7efeb80039b0 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed8e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40b3250 con 0x7efed40b6930 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed8e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40d1d70 con 0x7efed40b74d0 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efed41f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efed41f8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60140/0 (socket says 172.21.15.97:60140) 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed3400640 1 -- 172.21.15.97:0/3434586401 learned_addr learned my addr 172.21.15.97:0/3434586401 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efed3e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efed40b74d0 0x7efed40b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.061+0000 7efecbe00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed40b6930 0x7efed4158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed3400640 1 -- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efed40b74d0 msgr2=0x7efed40b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed3400640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efed40b74d0 0x7efed40b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed3400640 1 -- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed40b6930 msgr2=0x7efed4158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed3400640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed40b6930 0x7efed4158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed3400640 1 -- 172.21.15.97:0/3434586401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40b6250 con 0x7efeb80039b0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed3400640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efed41f8560 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7efec4000c00 tx=0x7efec4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4042020 con 0x7efeb80039b0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec403a040 con 0x7efeb80039b0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4039070 con 0x7efeb80039b0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.062+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed40b5c90 con 0x7efeb80039b0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.063+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed415f780 con 0x7efeb80039b0 2024-03-06T07:38:56.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.064+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed404f330 con 0x7efeb80039b0 2024-03-06T07:38:56.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.065+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efec4045020 con 0x7efeb80039b0 2024-03-06T07:38:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.066+0000 7efecb400640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efeb006b500 0x7efeb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.066+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efec40ab4c0 con 0x7efeb80039b0 2024-03-06T07:38:56.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.066+0000 7efecbe00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efeb006b500 0x7efeb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.067+0000 7efecbe00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efeb006b500 0x7efeb006d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efed40b2300 tx=0x7efebc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.070+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efec404c070 con 0x7efeb80039b0 2024-03-06T07:38:56.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.307+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efed408d7e0 con 0x7efeb80039b0 2024-03-06T07:38:56.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.308+0000 7efecb400640 1 -- 172.21.15.97:0/3434586401 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7efec407a3b0 con 0x7efeb80039b0 2024-03-06T07:38:56.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:56.310 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:56.310 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:56.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efeb006b500 msgr2=0x7efeb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efeb006b500 0x7efeb006d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efed40b2300 tx=0x7efebc009040 comp rx=0 tx=0).stop 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 msgr2=0x7efed41f8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efed41f8560 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7efec4000c00 tx=0x7efec4004730 comp rx=0 tx=0).stop 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 shutdown_connections 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7efeb006b500 0x7efeb006d9c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7efed40b74d0 0x7efed40b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed40b6930 0x7efed4158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 --2- 172.21.15.97:0/3434586401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efeb80039b0 0x7efed41f8560 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 >> 172.21.15.97:0/3434586401 conn(0x7efed40d3d10 msgr2=0x7efed40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.310+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 shutdown_connections 2024-03-06T07:38:56.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:56.311+0000 7efed8e00640 1 -- 172.21.15.97:0/3434586401 wait complete. 2024-03-06T07:38:56.533 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:56.533 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:57.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:57.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.748+0000 7f692ba00640 1 Processor -- start 2024-03-06T07:38:57.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.748+0000 7f692ba00640 1 -- start start 2024-03-06T07:38:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f692ba00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f692415e0d0 0x7f69240d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f692ba00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f692415f470 0x7f69240d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f692ba00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69240d1200 con 0x7f69240d0df0 2024-03-06T07:38:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f692ba00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69240d2f60 con 0x7f692415f470 2024-03-06T07:38:57.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f692ba00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69240cea60 con 0x7f692415e0d0 2024-03-06T07:38:57.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f6923e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f692415f470 0x7f69240d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f6923e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f692415f470 0x7f69240d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60152/0 (socket says 172.21.15.97:60152) 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.750+0000 7f6923e00640 1 -- 172.21.15.97:0/4111390646 learned_addr learned my addr 172.21.15.97:0/4111390646 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6929000640 1 --2- 172.21.15.97:0/4111390646 >> v2:172.21.15.135:3300/0 conn(0x7f692415e0d0 0x7f69240d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6923e00640 1 -- 172.21.15.97:0/4111390646 >> v2:172.21.15.135:3300/0 conn(0x7f692415e0d0 msgr2=0x7f69240d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6923e00640 1 --2- 172.21.15.97:0/4111390646 >> v2:172.21.15.135:3300/0 conn(0x7f692415e0d0 0x7f69240d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6923e00640 1 -- 172.21.15.97:0/4111390646 >> v1:172.21.15.97:6789/0 conn(0x7f69240d0df0 legacy=0x7f6924160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6923e00640 1 -- 172.21.15.97:0/4111390646 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69240d2350 con 0x7f692415f470 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6923400640 1 -- 172.21.15.97:0/4111390646 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266603326 0 0) 0x7f691c001490 con 0x7f69240d0df0 2024-03-06T07:38:57.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.751+0000 7f6929000640 1 --2- 172.21.15.97:0/4111390646 >> v2:172.21.15.135:3300/0 conn(0x7f692415e0d0 0x7f69240d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.752+0000 7f6923e00640 1 --2- 172.21.15.97:0/4111390646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f692415f470 0x7f69240d08b0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f69240b6a80 tx=0x7f691002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3328ffc8c83e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.752+0000 7f6923400640 1 -- 172.21.15.97:0/4111390646 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691002f850 con 0x7f692415f470 2024-03-06T07:38:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.752+0000 7f6923400640 1 -- 172.21.15.97:0/4111390646 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691002f9f0 con 0x7f692415f470 2024-03-06T07:38:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.752+0000 7f6923400640 1 -- 172.21.15.97:0/4111390646 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691002fdc0 con 0x7f692415f470 2024-03-06T07:38:57.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 -- 172.21.15.97:0/4111390646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f692415f470 msgr2=0x7f69240d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 --2- 172.21.15.97:0/4111390646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f692415f470 0x7f69240d08b0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f69240b6a80 tx=0x7f691002ba30 comp rx=0 tx=0).stop 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 -- 172.21.15.97:0/4111390646 shutdown_connections 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 --2- 172.21.15.97:0/4111390646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f692415f470 0x7f69240d08b0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 --2- 172.21.15.97:0/4111390646 >> v2:172.21.15.135:3300/0 conn(0x7f692415e0d0 0x7f69240d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 -- 172.21.15.97:0/4111390646 >> 172.21.15.97:0/4111390646 conn(0x7f69240d3ec0 msgr2=0x7f69240d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 -- 172.21.15.97:0/4111390646 shutdown_connections 2024-03-06T07:38:57.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.753+0000 7f692ba00640 1 -- 172.21.15.97:0/4111390646 wait complete. 2024-03-06T07:38:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.754+0000 7f692ba00640 1 Processor -- start 2024-03-06T07:38:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.755+0000 7f692ba00640 1 -- start start 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.755+0000 7f692ba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 0x7f69241526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.755+0000 7f692ba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f692415e0d0 0x7f69240c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.755+0000 7f692ba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f692415f470 0x7f69240b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f692ba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69240d2bf0 con 0x7f69240d0df0 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f692ba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69240d2910 con 0x7f692415f470 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f692ba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692415fb30 con 0x7f692415e0d0 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 0x7f69241526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 0x7f69241526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60156/0 (socket says 172.21.15.97:60156) 2024-03-06T07:38:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 -- 172.21.15.97:0/3591755015 learned_addr learned my addr 172.21.15.97:0/3591755015 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6923e00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f692415e0d0 0x7f69240c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 -- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f692415e0d0 msgr2=0x7f69240c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f692415e0d0 0x7f69240c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 -- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f692415f470 msgr2=0x7f69240b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929a00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f692415f470 0x7f69240b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f692415f470 0x7f69240b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929000640 1 -- 172.21.15.97:0/3591755015 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69240cb490 con 0x7f69240d0df0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.756+0000 7f6929a00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f692415f470 0x7f69240b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.757+0000 7f6929000640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 0x7f69241526d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f691800c7f0 tx=0x7f691800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.757+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918015070 con 0x7f69240d0df0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.757+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918007a30 con 0x7f69240d0df0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.757+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918007e00 con 0x7f69240d0df0 2024-03-06T07:38:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.757+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69240cb770 con 0x7f69240d0df0 2024-03-06T07:38:57.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.757+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69240cbd30 con 0x7f69240d0df0 2024-03-06T07:38:57.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.758+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692404f330 con 0x7f69240d0df0 2024-03-06T07:38:57.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.759+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6918018070 con 0x7f69240d0df0 2024-03-06T07:38:57.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.760+0000 7f6920c00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f690406b4d0 0x7f690406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.760+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f691808a1e0 con 0x7f69240d0df0 2024-03-06T07:38:57.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.760+0000 7f6923e00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f690406b4d0 0x7f690406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.761+0000 7f6923e00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f690406b4d0 0x7f690406d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6910036000 tx=0x7f6910039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:57.765+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6918059030 con 0x7f69240d0df0 2024-03-06T07:38:58.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.001+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f692415d9f0 con 0x7f69240d0df0 2024-03-06T07:38:58.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.003+0000 7f6920c00640 1 -- 172.21.15.97:0/3591755015 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f6918058930 con 0x7f69240d0df0 2024-03-06T07:38:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:58.005 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:58.005 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f690406b4d0 msgr2=0x7f690406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f690406b4d0 0x7f690406d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6910036000 tx=0x7f6910039040 comp rx=0 tx=0).stop 2024-03-06T07:38:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 msgr2=0x7f69241526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 0x7f69241526d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f691800c7f0 tx=0x7f691800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 shutdown_connections 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f690406b4d0 0x7f690406d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f692415f470 0x7f69240b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f692415e0d0 0x7f69240c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 --2- 172.21.15.97:0/3591755015 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69240d0df0 0x7f69241526d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.005+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 >> 172.21.15.97:0/3591755015 conn(0x7f69240d3ec0 msgr2=0x7f69240b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.006+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 shutdown_connections 2024-03-06T07:38:58.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:58.006+0000 7f692ba00640 1 -- 172.21.15.97:0/3591755015 wait complete. 2024-03-06T07:38:58.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:58.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:38:59.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:59.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.440+0000 7f5fe3000640 1 Processor -- start 2024-03-06T07:38:59.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.440+0000 7f5fe3000640 1 -- start start 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.441+0000 7f5fe3000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5fdc1561f0 0x7f5fdc0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.441+0000 7f5fe3000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5fdc0aebf0 0x7f5fdc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.441+0000 7f5fe3000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc15c240 con 0x7f5fdc0cc800 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.441+0000 7f5fe3000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc15ce30 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.441+0000 7f5fe3000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc15c500 con 0x7f5fdc1561f0 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5fdc0aebf0 0x7f5fdc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe2600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5fdc0cc800 0x7f5fdc0cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37830/0 (socket says 172.21.15.97:37830) 2024-03-06T07:38:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe2600640 1 -- 172.21.15.97:0/1462945683 learned_addr learned my addr 172.21.15.97:0/1462945683 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:38:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1c00640 1 --2- 172.21.15.97:0/1462945683 >> v2:172.21.15.135:3300/0 conn(0x7f5fdc1561f0 0x7f5fdc0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1200640 1 -- 172.21.15.97:0/1462945683 >> v2:172.21.15.135:3300/0 conn(0x7f5fdc1561f0 msgr2=0x7f5fdc0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1200640 1 --2- 172.21.15.97:0/1462945683 >> v2:172.21.15.135:3300/0 conn(0x7f5fdc1561f0 0x7f5fdc0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1200640 1 -- 172.21.15.97:0/1462945683 >> v1:172.21.15.97:6789/0 conn(0x7f5fdc0cc800 legacy=0x7f5fdc0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1200640 1 -- 172.21.15.97:0/1462945683 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc15c7e0 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.442+0000 7f5fe1c00640 1 --2- 172.21.15.97:0/1462945683 >> v2:172.21.15.135:3300/0 conn(0x7f5fdc1561f0 0x7f5fdc0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:59.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.443+0000 7f5fe1200640 1 --2- 172.21.15.97:0/1462945683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc0cc2c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0b4db0 tx=0x7f5fc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b22863082a5f901 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.444+0000 7f5fdbe00640 1 -- 172.21.15.97:0/1462945683 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8035020 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.444+0000 7f5fdbe00640 1 -- 172.21.15.97:0/1462945683 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc8031830 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.444+0000 7f5fdbe00640 1 -- 172.21.15.97:0/1462945683 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8031c00 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.444+0000 7f5fe3000640 1 -- 172.21.15.97:0/1462945683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 msgr2=0x7f5fdc0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.444+0000 7f5fe3000640 1 --2- 172.21.15.97:0/1462945683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc0cc2c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0b4db0 tx=0x7f5fc802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.445+0000 7f5fe3000640 1 -- 172.21.15.97:0/1462945683 shutdown_connections 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.445+0000 7f5fe3000640 1 --2- 172.21.15.97:0/1462945683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc0cc2c0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.445+0000 7f5fe3000640 1 --2- 172.21.15.97:0/1462945683 >> v2:172.21.15.135:3300/0 conn(0x7f5fdc1561f0 0x7f5fdc0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.445+0000 7f5fe3000640 1 -- 172.21.15.97:0/1462945683 >> 172.21.15.97:0/1462945683 conn(0x7f5fdc06afc0 msgr2=0x7f5fdc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:59.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.445+0000 7f5fe3000640 1 -- 172.21.15.97:0/1462945683 shutdown_connections 2024-03-06T07:38:59.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.445+0000 7f5fe3000640 1 -- 172.21.15.97:0/1462945683 wait complete. 2024-03-06T07:38:59.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.447+0000 7f5fe3000640 1 Processor -- start 2024-03-06T07:38:59.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.447+0000 7f5fe3000640 1 -- start start 2024-03-06T07:38:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe3000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe3000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0cc800 0x7f5fdc0b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe3000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5fdc1561f0 0x7f5fdc0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe3000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1565d0 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe3000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0ccc50 con 0x7f5fdc0cc800 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe3000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc156750 con 0x7f5fdc1561f0 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc14d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60182/0 (socket says 172.21.15.97:60182) 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 -- 172.21.15.97:0/2865127065 learned_addr learned my addr 172.21.15.97:0/2865127065 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1200640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0cc800 0x7f5fdc0b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 -- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5fdc1561f0 msgr2=0x7f5fdc0b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5fdc1561f0 0x7f5fdc0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 -- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0cc800 msgr2=0x7f5fdc0b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0cc800 0x7f5fdc0b0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.448+0000 7f5fe1c00640 1 -- 172.21.15.97:0/2865127065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc1f2370 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.449+0000 7f5fe1c00640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc14d6b0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000ce00 tx=0x7f5fd000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.449+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd000f890 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.449+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd000fa30 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.449+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc1f2650 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.449+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd000fe00 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.449+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc1f2c10 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.450+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc04f330 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.451+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fd0018400 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.452+0000 7f5fd9600640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5fbc06b4d0 0x7f5fbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.452+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5fd008b0a0 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.452+0000 7f5fe1200640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5fbc06b4d0 0x7f5fbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.453+0000 7f5fe1200640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5fbc06b4d0 0x7f5fbc06d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8036000 tx=0x7f5fc8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.456+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fd0014030 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.687+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fdc1559a0 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.688+0000 7f5fd9600640 1 -- 172.21.15.97:0/2865127065 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f5fd000ae20 con 0x7f5fdc0aebf0 2024-03-06T07:38:59.691 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:38:59.691 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:38:59.691 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5fbc06b4d0 msgr2=0x7f5fbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5fbc06b4d0 0x7f5fbc06d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8036000 tx=0x7f5fc8008030 comp rx=0 tx=0).stop 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 msgr2=0x7f5fdc14d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc14d6b0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000ce00 tx=0x7f5fd000a650 comp rx=0 tx=0).stop 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 shutdown_connections 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5fbc06b4d0 0x7f5fbc06d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5fdc1561f0 0x7f5fdc0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0cc800 0x7f5fdc0b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 --2- 172.21.15.97:0/2865127065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5fdc0aebf0 0x7f5fdc14d6b0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 >> 172.21.15.97:0/2865127065 conn(0x7f5fdc06afc0 msgr2=0x7f5fdc06d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:59.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 shutdown_connections 2024-03-06T07:38:59.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:38:59.691+0000 7f5fe3000640 1 -- 172.21.15.97:0/2865127065 wait complete. 2024-03-06T07:38:59.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:38:59.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:00.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:01.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.126+0000 7fc7e1000640 1 Processor -- start 2024-03-06T07:39:01.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.127+0000 7fc7e1000640 1 -- start start 2024-03-06T07:39:01.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7e1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7dc066d00 0x7fc7dc0af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7e1000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7dc15bf10 0x7fc7dc1649d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7e1000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0b3f60 con 0x7fc7dc15b420 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7e1000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc0afd50 con 0x7fc7dc066d00 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7e1000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc0b4200 con 0x7fc7dc15bf10 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7d9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7dc066d00 0x7fc7dc0af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7d9400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc7dc066d00 0x7fc7dc0af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60196/0 (socket says 172.21.15.97:60196) 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7d9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc7dc15b420 0x7fc7dc15b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37832/0 (socket says 172.21.15.97:37832) 2024-03-06T07:39:01.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.128+0000 7fc7d9400640 1 -- 172.21.15.97:0/344486625 learned_addr learned my addr 172.21.15.97:0/344486625 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7da800640 1 --2- 172.21.15.97:0/344486625 >> v2:172.21.15.135:3300/0 conn(0x7fc7dc15bf10 0x7fc7dc1649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7d9400640 1 -- 172.21.15.97:0/344486625 >> v2:172.21.15.135:3300/0 conn(0x7fc7dc15bf10 msgr2=0x7fc7dc1649d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7d9400640 1 --2- 172.21.15.97:0/344486625 >> v2:172.21.15.135:3300/0 conn(0x7fc7dc15bf10 0x7fc7dc1649d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7d9400640 1 -- 172.21.15.97:0/344486625 >> v1:172.21.15.97:6789/0 conn(0x7fc7dc15b420 legacy=0x7fc7dc15b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7d9400640 1 -- 172.21.15.97:0/344486625 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0b44e0 con 0x7fc7dc066d00 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7da800640 1 --2- 172.21.15.97:0/344486625 >> v2:172.21.15.135:3300/0 conn(0x7fc7dc15bf10 0x7fc7dc1649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:01.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.129+0000 7fc7d8a00640 1 -- 172.21.15.97:0/344486625 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743524064 0 0) 0x7fc7dc0b3f60 con 0x7fc7dc15b420 2024-03-06T07:39:01.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.130+0000 7fc7d9400640 1 --2- 172.21.15.97:0/344486625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0af6c0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009870 tx=0x7fc7c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8b1c3d05aab4b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7d8a00640 1 -- 172.21.15.97:0/344486625 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4035020 con 0x7fc7dc066d00 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7d8a00640 1 -- 172.21.15.97:0/344486625 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c4031870 con 0x7fc7dc066d00 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7d8a00640 1 -- 172.21.15.97:0/344486625 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4031c40 con 0x7fc7dc066d00 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 -- 172.21.15.97:0/344486625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 msgr2=0x7fc7dc0af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 --2- 172.21.15.97:0/344486625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0af6c0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009870 tx=0x7fc7c402b910 comp rx=0 tx=0).stop 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 -- 172.21.15.97:0/344486625 shutdown_connections 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 --2- 172.21.15.97:0/344486625 >> v2:172.21.15.135:3300/0 conn(0x7fc7dc15bf10 0x7fc7dc1649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 --2- 172.21.15.97:0/344486625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0af6c0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 -- 172.21.15.97:0/344486625 >> 172.21.15.97:0/344486625 conn(0x7fc7dc0d0340 msgr2=0x7fc7dc0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 -- 172.21.15.97:0/344486625 shutdown_connections 2024-03-06T07:39:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.131+0000 7fc7e1000640 1 -- 172.21.15.97:0/344486625 wait complete. 2024-03-06T07:39:01.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.133+0000 7fc7e1000640 1 Processor -- start 2024-03-06T07:39:01.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.133+0000 7fc7e1000640 1 -- start start 2024-03-06T07:39:01.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7e1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7e1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7dc15b420 0x7fc7dc0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7e1000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7dc15bf10 0x7fc7dc0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7e1000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc15b1a0 con 0x7fc7dc066d00 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7e1000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc06c8a0 con 0x7fc7dc15bf10 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7e1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc06e3b0 con 0x7fc7dc15b420 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7d9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7d9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60206/0 (socket says 172.21.15.97:60206) 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.135+0000 7fc7d9e00640 1 -- 172.21.15.97:0/970953378 learned_addr learned my addr 172.21.15.97:0/970953378 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d9e00640 1 -- 172.21.15.97:0/970953378 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7dc15b420 msgr2=0x7fc7dc0ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d9e00640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7dc15b420 0x7fc7dc0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d9e00640 1 -- 172.21.15.97:0/970953378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7dc15bf10 msgr2=0x7fc7dc0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d9e00640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7dc15bf10 0x7fc7dc0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d9e00640 1 -- 172.21.15.97:0/970953378 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc13e690 con 0x7fc7dc066d00 2024-03-06T07:39:01.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d9e00640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0cbf20 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c4d0 tx=0x7fc7cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc018070 con 0x7fc7dc066d00 2024-03-06T07:39:01.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc007500 con 0x7fc7dc066d00 2024-03-06T07:39:01.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc13e970 con 0x7fc7dc066d00 2024-03-06T07:39:01.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc004a20 con 0x7fc7dc066d00 2024-03-06T07:39:01.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.136+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc153ab0 con 0x7fc7dc066d00 2024-03-06T07:39:01.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.137+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc074ae0 con 0x7fc7dc066d00 2024-03-06T07:39:01.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.139+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7cc01b070 con 0x7fc7dc066d00 2024-03-06T07:39:01.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.140+0000 7fc7d2000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7b806b500 0x7fc7b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.140+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc7cc089550 con 0x7fc7dc066d00 2024-03-06T07:39:01.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.141+0000 7fc7d9400640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7b806b500 0x7fc7b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.141+0000 7fc7d9400640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7b806b500 0x7fc7b806d9c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4036000 tx=0x7fc7c4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.145+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7cc0583a0 con 0x7fc7dc066d00 2024-03-06T07:39:01.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.383+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7dc04f330 con 0x7fc7dc066d00 2024-03-06T07:39:01.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.384+0000 7fc7d2000640 1 -- 172.21.15.97:0/970953378 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc7cc006e20 con 0x7fc7dc066d00 2024-03-06T07:39:01.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:01.385 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:01.386 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:01.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7b806b500 msgr2=0x7fc7b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7b806b500 0x7fc7b806d9c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4036000 tx=0x7fc7c4030000 comp rx=0 tx=0).stop 2024-03-06T07:39:01.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 msgr2=0x7fc7dc0cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0cbf20 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c4d0 tx=0x7fc7cc00c990 comp rx=0 tx=0).stop 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 shutdown_connections 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc7b806b500 0x7fc7b806d9c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7dc15bf10 0x7fc7dc0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7dc15b420 0x7fc7dc0ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 --2- 172.21.15.97:0/970953378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7dc066d00 0x7fc7dc0cbf20 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 >> 172.21.15.97:0/970953378 conn(0x7fc7dc0d0340 msgr2=0x7fc7dc071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 shutdown_connections 2024-03-06T07:39:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:01.387+0000 7fc7e1000640 1 -- 172.21.15.97:0/970953378 wait complete. 2024-03-06T07:39:01.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:01.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:02.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:02.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.826+0000 7f34d0e00640 1 Processor -- start 2024-03-06T07:39:02.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.826+0000 7f34d0e00640 1 -- start start 2024-03-06T07:39:02.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.827+0000 7f34d0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 0x7f34cc071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.827+0000 7f34d0e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc06cef0 con 0x7f34cc15f9d0 2024-03-06T07:39:02.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.827+0000 7f34d0e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc06d1d0 con 0x7f34cc144c00 2024-03-06T07:39:02.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.827+0000 7f34d0e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc06d4b0 con 0x7f34cc15fe60 2024-03-06T07:39:02.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.827+0000 7f34c1400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f34cc15f9d0 0x7f34cc070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37836/0 (socket says 172.21.15.97:37836) 2024-03-06T07:39:02.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.827+0000 7f34c1400640 1 -- 172.21.15.97:0/3390676485 learned_addr learned my addr 172.21.15.97:0/3390676485 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:02.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34ca800640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 0x7f34cc071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440905836 0 0) 0x7f34cc06cef0 con 0x7f34cc15f9d0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b0003520 con 0x7f34cc15f9d0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773612986 0 0) 0x7f34b0003520 con 0x7f34cc15f9d0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 msgr2=0x7f34cc071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 0x7f34cc071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 >> v1:172.21.15.135:6789/0 conn(0x7f34cc144c00 legacy=0x7f34cc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc06d790 con 0x7f34cc15f9d0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f34b4001670 con 0x7f34cc15f9d0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 >> v1:172.21.15.97:6789/0 conn(0x7f34cc15f9d0 legacy=0x7f34cc070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34b00039e0 0x7f34b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 0x7f34b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b0004d40 0x7f34b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.828+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc06d4b0 con 0x7f34b00039e0 2024-03-06T07:39:02.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b000ba70 con 0x7f34b0004d40 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0010080 con 0x7f34cc15fe60 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34b00039e0 0x7f34b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34ca800640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b0004d40 0x7f34b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c1400640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 0x7f34b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9e00640 1 -- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 msgr2=0x7f34b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15fe60 0x7f34b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9e00640 1 -- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b0004d40 msgr2=0x7f34b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b0004d40 0x7f34b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34c9e00640 1 -- 172.21.15.97:0/3390676485 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b0003520 con 0x7f34b00039e0 2024-03-06T07:39:02.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.829+0000 7f34ca800640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b0004d40 0x7f34b00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:02.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.830+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34b00039e0 0x7f34b0003e00 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f34bc001e20 tx=0x7f34b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1120ca9abb27f361 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:02.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc001a70 con 0x7f34b00039e0 2024-03-06T07:39:02.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc003b70 con 0x7f34b00039e0 2024-03-06T07:39:02.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34c9400640 1 -- 172.21.15.97:0/3390676485 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc0050d0 con 0x7f34b00039e0 2024-03-06T07:39:02.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34d0e00640 1 -- 172.21.15.97:0/3390676485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34b00039e0 msgr2=0x7f34b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3390676485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34b00039e0 0x7f34b0003e00 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f34bc001e20 tx=0x7f34b0005830 comp rx=0 tx=0).stop 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34ca800640 1 -- 172.21.15.97:0/3390676485 reap_dead start 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34d0e00640 1 -- 172.21.15.97:0/3390676485 shutdown_connections 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34d0e00640 1 -- 172.21.15.97:0/3390676485 >> 172.21.15.97:0/3390676485 conn(0x7f34cc0b1730 msgr2=0x7f34cc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34d0e00640 1 -- 172.21.15.97:0/3390676485 shutdown_connections 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.831+0000 7f34d0e00640 1 -- 172.21.15.97:0/3390676485 wait complete. 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.833+0000 7f34d0e00640 1 Processor -- start 2024-03-06T07:39:02.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.833+0000 7f34d0e00640 1 -- start start 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34d0e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b00039e0 0x7f34cc15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34d0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 0x7f34cc142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34d0e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15c340 0x7f34cc15c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34d0e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc06cb50 con 0x7f34cc15eaa0 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34d0e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc160730 con 0x7f34b00039e0 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34d0e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc06ccd0 con 0x7f34cc15c340 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.834+0000 7f34c1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 0x7f34cc142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 0x7f34cc142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60216/0 (socket says 172.21.15.97:60216) 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 -- 172.21.15.97:0/3903830204 learned_addr learned my addr 172.21.15.97:0/3903830204 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b00039e0 0x7f34cc15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34ca800640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15c340 0x7f34cc15c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 -- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15c340 msgr2=0x7f34cc15c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15c340 0x7f34cc15c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 -- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b00039e0 msgr2=0x7f34cc15a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b00039e0 0x7f34cc15a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 -- 172.21.15.97:0/3903830204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0b3150 con 0x7f34cc15eaa0 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b00039e0 0x7f34cc15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.835+0000 7f34c1400640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 0x7f34cc142d60 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e780 tx=0x7f34b400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.836+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b400ee90 con 0x7f34cc15eaa0 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.836+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b4018070 con 0x7f34cc15eaa0 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.836+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b400cab0 con 0x7f34cc15eaa0 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.836+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc0b3430 con 0x7f34cc15eaa0 2024-03-06T07:39:02.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.836+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc0b39f0 con 0x7f34cc15eaa0 2024-03-06T07:39:02.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.837+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc04f330 con 0x7f34cc15eaa0 2024-03-06T07:39:02.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.839+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34b400cc50 con 0x7f34cc15eaa0 2024-03-06T07:39:02.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.840+0000 7f34c2a00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f34a806b6a0 0x7f34a806db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.840+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f34b408a160 con 0x7f34cc15eaa0 2024-03-06T07:39:02.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.840+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f34a806b6a0 0x7f34a806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.841+0000 7f34c9e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f34a806b6a0 0x7f34a806db60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f34bc034000 tx=0x7f34bc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:02.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:02.845+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34b4058f20 con 0x7f34cc15eaa0 2024-03-06T07:39:03.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.074+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34cc072890 con 0x7f34cc15eaa0 2024-03-06T07:39:03.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.075+0000 7f34c2a00640 1 -- 172.21.15.97:0/3903830204 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f34b4058820 con 0x7f34cc15eaa0 2024-03-06T07:39:03.076 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:03.076 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:03.076 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:03.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f34a806b6a0 msgr2=0x7f34a806db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f34a806b6a0 0x7f34a806db60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f34bc034000 tx=0x7f34bc009030 comp rx=0 tx=0).stop 2024-03-06T07:39:03.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 msgr2=0x7f34cc142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 0x7f34cc142d60 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e780 tx=0x7f34b400ec40 comp rx=0 tx=0).stop 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 shutdown_connections 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f34a806b6a0 0x7f34a806db60 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f34cc15c340 0x7f34cc15c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34cc15eaa0 0x7f34cc142d60 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 --2- 172.21.15.97:0/3903830204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34b00039e0 0x7f34cc15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.077+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 >> 172.21.15.97:0/3903830204 conn(0x7f34cc0b1730 msgr2=0x7f34cc0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.078+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 shutdown_connections 2024-03-06T07:39:03.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:03.078+0000 7f34d0e00640 1 -- 172.21.15.97:0/3903830204 wait complete. 2024-03-06T07:39:03.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:03.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:04.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:04.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.498+0000 7f681e800640 1 Processor -- start 2024-03-06T07:39:04.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.499+0000 7f681e800640 1 -- start start 2024-03-06T07:39:04.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f681e800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f681815e230 0x7f681806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f681e800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 0x7f68180d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f681e800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180cf340 con 0x7f68180ce550 2024-03-06T07:39:04.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f681e800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681815e610 con 0x7f681815e230 2024-03-06T07:39:04.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f681e800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cfe60 con 0x7f68180b4320 2024-03-06T07:39:04.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f6817e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f68180ce550 0x7f68180cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55906/0 (socket says 172.21.15.97:55906) 2024-03-06T07:39:04.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.500+0000 7f6817e00640 1 -- 172.21.15.97:0/785031711 learned_addr learned my addr 172.21.15.97:0/785031711 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:04.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6817400640 1 --2- 172.21.15.97:0/785031711 >> v2:172.21.15.135:3300/0 conn(0x7f681815e230 0x7f681806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f680fe00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 0x7f68180d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60669882 0 0) 0x7f68180cf340 con 0x7f68180ce550 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc003520 con 0x7f68180ce550 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3789596507 0 0) 0x7f67fc003520 con 0x7f68180ce550 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 msgr2=0x7f68180d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 0x7f68180d0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 >> v2:172.21.15.135:3300/0 conn(0x7f681815e230 msgr2=0x7f681806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 --2- 172.21.15.97:0/785031711 >> v2:172.21.15.135:3300/0 conn(0x7f681815e230 0x7f681806edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681806f390 con 0x7f68180ce550 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f68100019f0 con 0x7f68180ce550 2024-03-06T07:39:04.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 >> v1:172.21.15.97:6789/0 conn(0x7f68180ce550 legacy=0x7f68180cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67fc0039b0 0x7f67fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67fc004800 0x7f67fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 0x7f67fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681815e610 con 0x7f67fc004800 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cfe60 con 0x7f67fc0039b0 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.501+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc00b9c0 con 0x7f68180b4320 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f680fe00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67fc004800 0x7f67fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f6817e00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 0x7f67fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f6817400640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67fc0039b0 0x7f67fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f680fe00640 1 -- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 msgr2=0x7f67fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f680fe00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68180b4320 0x7f67fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f680fe00640 1 -- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67fc0039b0 msgr2=0x7f67fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f680fe00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67fc0039b0 0x7f67fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f680fe00640 1 -- 172.21.15.97:0/785031711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc003520 con 0x7f67fc004800 2024-03-06T07:39:04.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.502+0000 7f6817400640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67fc0039b0 0x7f67fc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:04.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.503+0000 7f680fe00640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67fc004800 0x7f67fc004c80 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f6800009af0 tx=0x7f67fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a51a109e101078a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800039070 con 0x7f67fc004800 2024-03-06T07:39:04.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680002c6b0 con 0x7f67fc004800 2024-03-06T07:39:04.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f6816a00640 1 -- 172.21.15.97:0/785031711 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800009690 con 0x7f67fc004800 2024-03-06T07:39:04.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f681e800640 1 -- 172.21.15.97:0/785031711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67fc004800 msgr2=0x7f67fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f681e800640 1 --2- 172.21.15.97:0/785031711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67fc004800 0x7f67fc004c80 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f6800009af0 tx=0x7f67fc005830 comp rx=0 tx=0).stop 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f6817e00640 1 -- 172.21.15.97:0/785031711 reap_dead start 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f681e800640 1 -- 172.21.15.97:0/785031711 shutdown_connections 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f681e800640 1 -- 172.21.15.97:0/785031711 >> 172.21.15.97:0/785031711 conn(0x7f68180b3800 msgr2=0x7f68180b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f681e800640 1 -- 172.21.15.97:0/785031711 shutdown_connections 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.504+0000 7f681e800640 1 -- 172.21.15.97:0/785031711 wait complete. 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.506+0000 7f681e800640 1 Processor -- start 2024-03-06T07:39:04.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.506+0000 7f681e800640 1 -- start start 2024-03-06T07:39:04.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.507+0000 7f681e800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67fc004800 0x7f681814f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.507+0000 7f681e800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681814ff40 0x7f6818151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.507+0000 7f681e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 0x7f68180c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.507+0000 7f681e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180d06c0 con 0x7f6818152510 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.507+0000 7f681e800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180d03e0 con 0x7f681814ff40 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.507+0000 7f681e800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cca30 con 0x7f67fc004800 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 0x7f68180c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 0x7f68180c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44468/0 (socket says 172.21.15.97:44468) 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 -- 172.21.15.97:0/837177774 learned_addr learned my addr 172.21.15.97:0/837177774 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817400640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67fc004800 0x7f681814f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f680fe00640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681814ff40 0x7f6818151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 -- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67fc004800 msgr2=0x7f681814f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67fc004800 0x7f681814f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 -- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681814ff40 msgr2=0x7f6818151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681814ff40 0x7f6818151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 -- 172.21.15.97:0/837177774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68180c7060 con 0x7f6818152510 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f680fe00640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681814ff40 0x7f6818151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.508+0000 7f6817e00640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 0x7f68180c62c0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f6810002230 tx=0x7f6810007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.509+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681000eb00 con 0x7f6818152510 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.509+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681000ecc0 con 0x7f6818152510 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.509+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810013420 con 0x7f6818152510 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.509+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68180b60f0 con 0x7f6818152510 2024-03-06T07:39:04.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.509+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681814fb40 con 0x7f6818152510 2024-03-06T07:39:04.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.510+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681804f330 con 0x7f6818152510 2024-03-06T07:39:04.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.511+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f681001a020 con 0x7f6818152510 2024-03-06T07:39:04.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.512+0000 7f680f400640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b500 0x7f67f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.512+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6810089cd0 con 0x7f6818152510 2024-03-06T07:39:04.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.513+0000 7f6817400640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b500 0x7f67f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.513+0000 7f6817400640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b500 0x7f67f406d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6808009b20 tx=0x7f6808008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.517+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6810021020 con 0x7f6818152510 2024-03-06T07:39:04.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.740+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68180cf6f0 con 0x7f6818152510 2024-03-06T07:39:04.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.741+0000 7f680f400640 1 -- 172.21.15.97:0/837177774 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f6810058a90 con 0x7f6818152510 2024-03-06T07:39:04.743 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:04.743 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:04.743 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:04.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.745+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b500 msgr2=0x7f67f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.745+0000 7f681e800640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b500 0x7f67f406d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6808009b20 tx=0x7f6808008040 comp rx=0 tx=0).stop 2024-03-06T07:39:04.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.745+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 msgr2=0x7f68180c62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.745+0000 7f681e800640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 0x7f68180c62c0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f6810002230 tx=0x7f6810007bc0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 shutdown_connections 2024-03-06T07:39:04.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f67f406b500 0x7f67f406d9c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6818152510 0x7f68180c62c0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f681814ff40 0x7f6818151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 --2- 172.21.15.97:0/837177774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f67fc004800 0x7f681814f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 >> 172.21.15.97:0/837177774 conn(0x7f68180b3800 msgr2=0x7f681815d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:04.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 shutdown_connections 2024-03-06T07:39:04.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:04.746+0000 7f681e800640 1 -- 172.21.15.97:0/837177774 wait complete. 2024-03-06T07:39:04.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:04.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:05.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:06.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.181+0000 7fd694e00640 1 Processor -- start 2024-03-06T07:39:06.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.181+0000 7fd694e00640 1 -- start start 2024-03-06T07:39:06.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.182+0000 7fd694e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 0x7fd6900b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.182+0000 7fd694e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd69006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd694e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69015b090 con 0x7fd690072340 2024-03-06T07:39:06.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd694e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6900b2a30 con 0x7fd69015acb0 2024-03-06T07:39:06.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd694e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b4360 con 0x7fd6900b4680 2024-03-06T07:39:06.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd68d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd69006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd68de00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 0x7fd6900b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd68d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd69006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:53524/0 (socket says 172.21.15.97:53524) 2024-03-06T07:39:06.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd68d400640 1 -- 172.21.15.97:0/2351386607 learned_addr learned my addr 172.21.15.97:0/2351386607 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:06.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.183+0000 7fd68de00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 0x7fd6900b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:40212/0 (socket says 172.21.15.97:40212) 2024-03-06T07:39:06.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.184+0000 7fd68d400640 1 -- 172.21.15.97:0/2351386607 >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 msgr2=0x7fd6900b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:06.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.184+0000 7fd68d400640 1 --2- 172.21.15.97:0/2351386607 >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 0x7fd6900b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.184+0000 7fd68d400640 1 -- 172.21.15.97:0/2351386607 >> v1:172.21.15.135:6789/0 conn(0x7fd690072340 legacy=0x7fd6900ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.184+0000 7fd68d400640 1 -- 172.21.15.97:0/2351386607 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6900b2cf0 con 0x7fd6900b4680 2024-03-06T07:39:06.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.184+0000 7fd68de00640 1 --2- 172.21.15.97:0/2351386607 >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 0x7fd6900b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:06.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.185+0000 7fd68d400640 1 --2- 172.21.15.97:0/2351386607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd69006e4f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009870 tx=0x7fd67c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbcb8931b00db066 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.186+0000 7fd68ca00640 1 -- 172.21.15.97:0/2351386607 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c034070 con 0x7fd6900b4680 2024-03-06T07:39:06.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.186+0000 7fd68ca00640 1 -- 172.21.15.97:0/2351386607 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c030840 con 0x7fd6900b4680 2024-03-06T07:39:06.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.186+0000 7fd68ca00640 1 -- 172.21.15.97:0/2351386607 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c030c10 con 0x7fd6900b4680 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.186+0000 7fd694e00640 1 -- 172.21.15.97:0/2351386607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 msgr2=0x7fd69006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.186+0000 7fd694e00640 1 --2- 172.21.15.97:0/2351386607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd69006e4f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009870 tx=0x7fd67c02b980 comp rx=0 tx=0).stop 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.186+0000 7fd694e00640 1 -- 172.21.15.97:0/2351386607 shutdown_connections 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.187+0000 7fd694e00640 1 --2- 172.21.15.97:0/2351386607 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd69006e4f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.187+0000 7fd694e00640 1 --2- 172.21.15.97:0/2351386607 >> v2:172.21.15.135:3300/0 conn(0x7fd69015acb0 0x7fd6900b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.187+0000 7fd694e00640 1 -- 172.21.15.97:0/2351386607 >> 172.21.15.97:0/2351386607 conn(0x7fd6900b1730 msgr2=0x7fd6900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.187+0000 7fd694e00640 1 -- 172.21.15.97:0/2351386607 shutdown_connections 2024-03-06T07:39:06.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.187+0000 7fd694e00640 1 -- 172.21.15.97:0/2351386607 wait complete. 2024-03-06T07:39:06.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.188+0000 7fd694e00640 1 Processor -- start 2024-03-06T07:39:06.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.188+0000 7fd694e00640 1 -- start start 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.189+0000 7fd694e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 0x7fd6900ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.189+0000 7fd694e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd6900ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.189+0000 7fd694e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd69015acb0 0x7fd6900cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.189+0000 7fd694e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b3d10 con 0x7fd690072340 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.189+0000 7fd694e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b32b0 con 0x7fd69015acb0 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.189+0000 7fd694e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b4010 con 0x7fd6900b4680 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 0x7fd6900ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 0x7fd6900ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44470/0 (socket says 172.21.15.97:44470) 2024-03-06T07:39:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 -- 172.21.15.97:0/3217110295 learned_addr learned my addr 172.21.15.97:0/3217110295 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:06.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68d400640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd6900ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 -- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 msgr2=0x7fd6900ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd6900ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 -- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd69015acb0 msgr2=0x7fd6900cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd69015acb0 0x7fd6900cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 -- 172.21.15.97:0/3217110295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6900ae1b0 con 0x7fd690072340 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.190+0000 7fd68de00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 0x7fd6900ccfb0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fd68000ed40 tx=0x7fd68000d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.191+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680010640 con 0x7fd690072340 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.191+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6900ae4f0 con 0x7fd690072340 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.191+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6900ae770 con 0x7fd690072340 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.191+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6800107e0 con 0x7fd690072340 2024-03-06T07:39:06.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.191+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680010b90 con 0x7fd690072340 2024-03-06T07:39:06.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.192+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd668005290 con 0x7fd690072340 2024-03-06T07:39:06.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.193+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd68000f3d0 con 0x7fd690072340 2024-03-06T07:39:06.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.194+0000 7fd686000640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd67006b4d0 0x7fd67006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.194+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd68008acd0 con 0x7fd690072340 2024-03-06T07:39:06.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.194+0000 7fd68d400640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd67006b4d0 0x7fd67006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.195+0000 7fd68d400640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd67006b4d0 0x7fd67006d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd67c036000 tx=0x7fd67c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.203+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6900ae4f0 con 0x7fd690072340 2024-03-06T07:39:06.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.442+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd668005850 con 0x7fd690072340 2024-03-06T07:39:06.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.443+0000 7fd686000640 1 -- 172.21.15.97:0/3217110295 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fd680014030 con 0x7fd690072340 2024-03-06T07:39:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:06.445 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:06.445 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:06.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd67006b4d0 msgr2=0x7fd67006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd67006b4d0 0x7fd67006d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd67c036000 tx=0x7fd67c03b040 comp rx=0 tx=0).stop 2024-03-06T07:39:06.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 msgr2=0x7fd6900ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 0x7fd6900ccfb0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fd68000ed40 tx=0x7fd68000d720 comp rx=0 tx=0).stop 2024-03-06T07:39:06.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 shutdown_connections 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd67006b4d0 0x7fd67006d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd69015acb0 0x7fd6900cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6900b4680 0x7fd6900ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 --2- 172.21.15.97:0/3217110295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd690072340 0x7fd6900ccfb0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 >> 172.21.15.97:0/3217110295 conn(0x7fd6900b1730 msgr2=0x7fd6900af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 shutdown_connections 2024-03-06T07:39:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:06.446+0000 7fd694e00640 1 -- 172.21.15.97:0/3217110295 wait complete. 2024-03-06T07:39:06.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:06.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:07.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:07.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.883+0000 7f5639000640 1 Processor -- start 2024-03-06T07:39:07.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.884+0000 7f5639000640 1 -- start start 2024-03-06T07:39:07.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.885+0000 7f5639000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 0x7f5634071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.885+0000 7f5639000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563406cef0 con 0x7f563415f9d0 2024-03-06T07:39:07.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.885+0000 7f5639000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563406d1d0 con 0x7f5634144c00 2024-03-06T07:39:07.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.885+0000 7f5639000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563406d4b0 con 0x7f563415fe60 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.886+0000 7f5629400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f563415f9d0 0x7f5634070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55908/0 (socket says 172.21.15.97:55908) 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.886+0000 7f5629400640 1 -- 172.21.15.97:0/1588024728 learned_addr learned my addr 172.21.15.97:0/1588024728 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.886+0000 7f5632800640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 0x7f5634071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.886+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124178094 0 0) 0x7f563406cef0 con 0x7f563415f9d0 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.886+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5618003520 con 0x7f563415f9d0 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167616964 0 0) 0x7f5618003520 con 0x7f563415f9d0 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 msgr2=0x7f5634071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 0x7f5634071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 >> v1:172.21.15.135:6789/0 conn(0x7f5634144c00 legacy=0x7f563406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563406d790 con 0x7f563415f9d0 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f561c001670 con 0x7f563415f9d0 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 >> v1:172.21.15.97:6789/0 conn(0x7f563415f9d0 legacy=0x7f5634070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56180039e0 0x7f5618003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 0x7f5618004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563406d4b0 con 0x7f5618004d40 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561800baa0 con 0x7f56180039e0 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561800bca0 con 0x7f563415fe60 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5632800640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.887+0000 7f5631e00640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56180039e0 0x7f5618003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.888+0000 7f5629400640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 0x7f5618004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.888+0000 7f5632800640 1 -- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 msgr2=0x7f5618004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.888+0000 7f5632800640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415fe60 0x7f5618004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.888+0000 7f5632800640 1 -- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56180039e0 msgr2=0x7f5618003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.888+0000 7f5632800640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56180039e0 0x7f5618003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.888+0000 7f5632800640 1 -- 172.21.15.97:0/1588024728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5618003520 con 0x7f5618004d40 2024-03-06T07:39:07.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.889+0000 7f5632800640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56180051c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f562c009f90 tx=0x7f5618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23788340ec03ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:07.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.889+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c039070 con 0x7f5618004d40 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.889+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c02cbd0 con 0x7f5618004d40 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.889+0000 7f5631400640 1 -- 172.21.15.97:0/1588024728 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c009c10 con 0x7f5618004d40 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5639000640 1 -- 172.21.15.97:0/1588024728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 msgr2=0x7f56180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5639000640 1 --2- 172.21.15.97:0/1588024728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56180051c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f562c009f90 tx=0x7f5618005830 comp rx=0 tx=0).stop 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5632800640 1 -- 172.21.15.97:0/1588024728 reap_dead start 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5639000640 1 -- 172.21.15.97:0/1588024728 shutdown_connections 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5639000640 1 -- 172.21.15.97:0/1588024728 >> 172.21.15.97:0/1588024728 conn(0x7f56340b1730 msgr2=0x7f56340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5639000640 1 -- 172.21.15.97:0/1588024728 shutdown_connections 2024-03-06T07:39:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.890+0000 7f5639000640 1 -- 172.21.15.97:0/1588024728 wait complete. 2024-03-06T07:39:07.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.893+0000 7f5639000640 1 Processor -- start 2024-03-06T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.893+0000 7f5639000640 1 -- start start 2024-03-06T07:39:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.894+0000 7f5639000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56341e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.894+0000 7f5639000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56341e6f40 0x7f56341e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.894+0000 7f5639000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415e870 0x7f563413c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.894+0000 7f5639000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563406cb50 con 0x7f5618004d40 2024-03-06T07:39:07.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.894+0000 7f5639000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5634160730 con 0x7f56341e6f40 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.894+0000 7f5639000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563406ccd0 con 0x7f563415e870 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56341e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56341e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44496/0 (socket says 172.21.15.97:44496) 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 -- 172.21.15.97:0/457922927 learned_addr learned my addr 172.21.15.97:0/457922927 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5632800640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415e870 0x7f563413c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5629400640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56341e6f40 0x7f56341e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 -- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415e870 msgr2=0x7f563413c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415e870 0x7f563413c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 -- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56341e6f40 msgr2=0x7f56341e9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56341e6f40 0x7f56341e9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 -- 172.21.15.97:0/457922927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56341e5ea0 con 0x7f5618004d40 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5632800640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415e870 0x7f563413c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:07.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.895+0000 7f5631e00640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56341e55c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f562400e550 tx=0x7f562400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.896+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240052e0 con 0x7f5618004d40 2024-03-06T07:39:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.896+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5624018070 con 0x7f5618004d40 2024-03-06T07:39:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.896+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56341e6180 con 0x7f5618004d40 2024-03-06T07:39:07.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.896+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5624003810 con 0x7f5618004d40 2024-03-06T07:39:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.896+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56341e6740 con 0x7f5618004d40 2024-03-06T07:39:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.897+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563404f330 con 0x7f5618004d40 2024-03-06T07:39:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.898+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56240039b0 con 0x7f5618004d40 2024-03-06T07:39:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.900+0000 7f562aa00640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f561006b4d0 0x7f561006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.900+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5624089fa0 con 0x7f5618004d40 2024-03-06T07:39:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.900+0000 7f5629400640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f561006b4d0 0x7f561006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.901+0000 7f5629400640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f561006b4d0 0x7f561006d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f561c001eb0 tx=0x7f561c000f50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:07.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:07.904+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5624058cd0 con 0x7f5618004d40 2024-03-06T07:39:08.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.138+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56341e6b40 con 0x7f5618004d40 2024-03-06T07:39:08.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.138+0000 7f562aa00640 1 -- 172.21.15.97:0/457922927 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f56240585d0 con 0x7f5618004d40 2024-03-06T07:39:08.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:08.141 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:08.141 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:08.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f561006b4d0 msgr2=0x7f561006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f561006b4d0 0x7f561006d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f561c001eb0 tx=0x7f561c000f50 comp rx=0 tx=0).stop 2024-03-06T07:39:08.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 msgr2=0x7f56341e55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56341e55c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f562400e550 tx=0x7f562400ea10 comp rx=0 tx=0).stop 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 shutdown_connections 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f561006b4d0 0x7f561006d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f563415e870 0x7f563413c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56341e6f40 0x7f56341e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 --2- 172.21.15.97:0/457922927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618004d40 0x7f56341e55c0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 >> 172.21.15.97:0/457922927 conn(0x7f56340b1730 msgr2=0x7f56340ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 shutdown_connections 2024-03-06T07:39:08.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:08.141+0000 7f5639000640 1 -- 172.21.15.97:0/457922927 wait complete. 2024-03-06T07:39:08.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:08.362 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:09.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:09.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.577+0000 7fbe9d400640 1 Processor -- start 2024-03-06T07:39:09.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.577+0000 7fbe9d400640 1 -- start start 2024-03-06T07:39:09.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe9d400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbe98161580 0x7fbe980d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe9d400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbe980d1e90 0x7fbe980d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe9d400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe98161960 con 0x7fbe980d28a0 2024-03-06T07:39:09.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe9d400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980d2d10 con 0x7fbe980d1e90 2024-03-06T07:39:09.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe9d400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980cf6f0 con 0x7fbe98161580 2024-03-06T07:39:09.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe8fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbe980d1e90 0x7fbe980d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe8fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbe980d1e90 0x7fbe980d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44506/0 (socket says 172.21.15.97:44506) 2024-03-06T07:39:09.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe8fe00640 1 -- 172.21.15.97:0/3059941129 learned_addr learned my addr 172.21.15.97:0/3059941129 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:09.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe8fe00640 1 -- 172.21.15.97:0/3059941129 >> v2:172.21.15.135:3300/0 conn(0x7fbe98161580 msgr2=0x7fbe980d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:09.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.579+0000 7fbe8fe00640 1 --2- 172.21.15.97:0/3059941129 >> v2:172.21.15.135:3300/0 conn(0x7fbe98161580 0x7fbe980d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.580+0000 7fbe8fe00640 1 -- 172.21.15.97:0/3059941129 >> v1:172.21.15.135:6789/0 conn(0x7fbe980d28a0 legacy=0x7fbe980d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:09.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.580+0000 7fbe8fe00640 1 -- 172.21.15.97:0/3059941129 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980cf9d0 con 0x7fbe980d1e90 2024-03-06T07:39:09.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.581+0000 7fbe8fe00640 1 --2- 172.21.15.97:0/3059941129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d1e90 0x7fbe980d2360 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fbe98073740 tx=0x7fbe8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c41a774e3acfe1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:09.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.581+0000 7fbe97400640 1 -- 172.21.15.97:0/3059941129 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80035020 con 0x7fbe980d1e90 2024-03-06T07:39:09.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.581+0000 7fbe97400640 1 -- 172.21.15.97:0/3059941129 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe80031870 con 0x7fbe980d1e90 2024-03-06T07:39:09.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.581+0000 7fbe97400640 1 -- 172.21.15.97:0/3059941129 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80031c40 con 0x7fbe980d1e90 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 -- 172.21.15.97:0/3059941129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d1e90 msgr2=0x7fbe980d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 --2- 172.21.15.97:0/3059941129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d1e90 0x7fbe980d2360 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fbe98073740 tx=0x7fbe8002b910 comp rx=0 tx=0).stop 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 -- 172.21.15.97:0/3059941129 shutdown_connections 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 --2- 172.21.15.97:0/3059941129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d1e90 0x7fbe980d2360 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 --2- 172.21.15.97:0/3059941129 >> v2:172.21.15.135:3300/0 conn(0x7fbe98161580 0x7fbe980d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 -- 172.21.15.97:0/3059941129 >> 172.21.15.97:0/3059941129 conn(0x7fbe980d3d80 msgr2=0x7fbe980d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 -- 172.21.15.97:0/3059941129 shutdown_connections 2024-03-06T07:39:09.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.582+0000 7fbe9d400640 1 -- 172.21.15.97:0/3059941129 wait complete. 2024-03-06T07:39:09.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.583+0000 7fbe9d400640 1 Processor -- start 2024-03-06T07:39:09.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.584+0000 7fbe9d400640 1 -- start start 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.584+0000 7fbe9d400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980d1e90 0x7fbe98150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe9d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 0x7fbe980cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe9d400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbe98161580 0x7fbe9815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe9d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980b39a0 con 0x7fbe980d28a0 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe9d400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980b36c0 con 0x7fbe980d1e90 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe9d400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980cf3a0 con 0x7fbe98161580 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 0x7fbe980cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 0x7fbe980cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44522/0 (socket says 172.21.15.97:44522) 2024-03-06T07:39:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 -- 172.21.15.97:0/2404806972 learned_addr learned my addr 172.21.15.97:0/2404806972 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 -- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbe98161580 msgr2=0x7fbe9815c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe97e00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980d1e90 0x7fbe98150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe9ca00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbe98161580 0x7fbe9815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbe98161580 0x7fbe9815c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 -- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980d1e90 msgr2=0x7fbe98150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980d1e90 0x7fbe98150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.585+0000 7fbe8fe00640 1 -- 172.21.15.97:0/2404806972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980cbef0 con 0x7fbe980d28a0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.586+0000 7fbe8fe00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 0x7fbe980cb860 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fbe8002b950 tx=0x7fbe800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.586+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8003ad60 con 0x7fbe980d28a0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.586+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe800443f0 con 0x7fbe980d28a0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.586+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe980cc1d0 con 0x7fbe980d28a0 2024-03-06T07:39:09.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.586+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe800447e0 con 0x7fbe980d28a0 2024-03-06T07:39:09.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.586+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe980b8ed0 con 0x7fbe980d28a0 2024-03-06T07:39:09.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.587+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9804f330 con 0x7fbe980d28a0 2024-03-06T07:39:09.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.588+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe80038020 con 0x7fbe980d28a0 2024-03-06T07:39:09.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.590+0000 7fbe94c00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe7406b4d0 0x7fbe7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.590+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbe800ac560 con 0x7fbe980d28a0 2024-03-06T07:39:09.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.590+0000 7fbe97e00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe7406b4d0 0x7fbe7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.591+0000 7fbe97e00640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe7406b4d0 0x7fbe7406d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800a810 tx=0x7fbe88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:09.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.594+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe8003f070 con 0x7fbe980d28a0 2024-03-06T07:39:09.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.832+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe980d3b60 con 0x7fbe980d28a0 2024-03-06T07:39:09.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.833+0000 7fbe94c00640 1 -- 172.21.15.97:0/2404806972 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fbe8007b320 con 0x7fbe980d28a0 2024-03-06T07:39:09.835 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:09.835 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:09.835 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.835+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe7406b4d0 msgr2=0x7fbe7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.835+0000 7fbe9d400640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe7406b4d0 0x7fbe7406d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800a810 tx=0x7fbe88008040 comp rx=0 tx=0).stop 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.835+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 msgr2=0x7fbe980cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.835+0000 7fbe9d400640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 0x7fbe980cb860 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fbe8002b950 tx=0x7fbe800046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 shutdown_connections 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbe7406b4d0 0x7fbe7406d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbe98161580 0x7fbe9815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbe980d28a0 0x7fbe980cb860 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 --2- 172.21.15.97:0/2404806972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980d1e90 0x7fbe98150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 >> 172.21.15.97:0/2404806972 conn(0x7fbe980d3d80 msgr2=0x7fbe9806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:09.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 shutdown_connections 2024-03-06T07:39:09.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:09.836+0000 7fbe9d400640 1 -- 172.21.15.97:0/2404806972 wait complete. 2024-03-06T07:39:09.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:09.877 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:10.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:10.059 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:11.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:11.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.266+0000 7fa5f8e00640 1 Processor -- start 2024-03-06T07:39:11.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.266+0000 7fa5f8e00640 1 -- start start 2024-03-06T07:39:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f8e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa5f415acb0 0x7fa5f40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f8e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa5f40b4680 0x7fa5f406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f8e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f415b090 con 0x7fa5f4072340 2024-03-06T07:39:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f8e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40b2a30 con 0x7fa5f415acb0 2024-03-06T07:39:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f8e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40b4360 con 0x7fa5f40b4680 2024-03-06T07:39:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa5f415acb0 0x7fa5f40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f2800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa5f4072340 0x7fa5f40ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55912/0 (socket says 172.21.15.97:55912) 2024-03-06T07:39:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f1e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa5f415acb0 0x7fa5f40b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44524/0 (socket says 172.21.15.97:44524) 2024-03-06T07:39:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.268+0000 7fa5f2800640 1 -- 172.21.15.97:0/422672953 learned_addr learned my addr 172.21.15.97:0/422672953 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.269+0000 7fa5f1e00640 1 -- 172.21.15.97:0/422672953 >> v2:172.21.15.135:3300/0 conn(0x7fa5f40b4680 msgr2=0x7fa5f406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.269+0000 7fa5f1e00640 1 --2- 172.21.15.97:0/422672953 >> v2:172.21.15.135:3300/0 conn(0x7fa5f40b4680 0x7fa5f406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.269+0000 7fa5f1e00640 1 -- 172.21.15.97:0/422672953 >> v1:172.21.15.97:6789/0 conn(0x7fa5f4072340 legacy=0x7fa5f40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.269+0000 7fa5f1e00640 1 -- 172.21.15.97:0/422672953 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40b2cf0 con 0x7fa5f415acb0 2024-03-06T07:39:11.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.270+0000 7fa5f1e00640 1 --2- 172.21.15.97:0/422672953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f415acb0 0x7fa5f40b37d0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4068560 tx=0x7fa5e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5256c19fb229f1a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.270+0000 7fa5f1400640 1 -- 172.21.15.97:0/422672953 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4035020 con 0x7fa5f415acb0 2024-03-06T07:39:11.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.270+0000 7fa5f1400640 1 -- 172.21.15.97:0/422672953 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4031830 con 0x7fa5f415acb0 2024-03-06T07:39:11.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.270+0000 7fa5f1400640 1 -- 172.21.15.97:0/422672953 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4031c00 con 0x7fa5f415acb0 2024-03-06T07:39:11.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 -- 172.21.15.97:0/422672953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f415acb0 msgr2=0x7fa5f40b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/422672953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f415acb0 0x7fa5f40b37d0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4068560 tx=0x7fa5e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 -- 172.21.15.97:0/422672953 shutdown_connections 2024-03-06T07:39:11.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/422672953 >> v2:172.21.15.135:3300/0 conn(0x7fa5f40b4680 0x7fa5f406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/422672953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f415acb0 0x7fa5f40b37d0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 -- 172.21.15.97:0/422672953 >> 172.21.15.97:0/422672953 conn(0x7fa5f40b1730 msgr2=0x7fa5f40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:11.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 -- 172.21.15.97:0/422672953 shutdown_connections 2024-03-06T07:39:11.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.271+0000 7fa5f8e00640 1 -- 172.21.15.97:0/422672953 wait complete. 2024-03-06T07:39:11.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.273+0000 7fa5f8e00640 1 Processor -- start 2024-03-06T07:39:11.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.273+0000 7fa5f8e00640 1 -- start start 2024-03-06T07:39:11.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 0x7fa5f4144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f8e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f40b4680 0x7fa5f4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f8e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f415acb0 0x7fa5f4144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f8e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40b3d10 con 0x7fa5f4072340 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f8e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40b32b0 con 0x7fa5f40b4680 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f8e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40b4010 con 0x7fa5f415acb0 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 0x7fa5f4144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 0x7fa5f4144400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44528/0 (socket says 172.21.15.97:44528) 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 -- 172.21.15.97:0/2269169802 learned_addr learned my addr 172.21.15.97:0/2269169802 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 -- 172.21.15.97:0/2269169802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f415acb0 msgr2=0x7fa5f4144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f415acb0 0x7fa5f4144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 -- 172.21.15.97:0/2269169802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f40b4680 msgr2=0x7fa5f4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f40b4680 0x7fa5f4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.274+0000 7fa5f1e00640 1 -- 172.21.15.97:0/2269169802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40c8ee0 con 0x7fa5f4072340 2024-03-06T07:39:11.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.275+0000 7fa5f1e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 0x7fa5f4144400 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009890 tx=0x7fa5e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.275+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4041070 con 0x7fa5f4072340 2024-03-06T07:39:11.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.275+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e403a950 con 0x7fa5f4072340 2024-03-06T07:39:11.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.275+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f40c91c0 con 0x7fa5f4072340 2024-03-06T07:39:11.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.275+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4035040 con 0x7fa5f4072340 2024-03-06T07:39:11.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.275+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f40c9780 con 0x7fa5f4072340 2024-03-06T07:39:11.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.277+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f404f330 con 0x7fa5f4072340 2024-03-06T07:39:11.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.278+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5e4038020 con 0x7fa5f4072340 2024-03-06T07:39:11.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.279+0000 7fa5eaa00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.279+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa5e40abc50 con 0x7fa5f4072340 2024-03-06T07:39:11.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.279+0000 7fa5e9400640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.280+0000 7fa5e9400640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d006b4d0 0x7fa5d006d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa5f40b16e0 tx=0x7fa5dc006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.288+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5e40b1020 con 0x7fa5f4072340 2024-03-06T07:39:11.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.621+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5f415f450 con 0x7fa5f4072340 2024-03-06T07:39:11.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.622+0000 7fa5eaa00640 1 -- 172.21.15.97:0/2269169802 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fa5e407ab40 con 0x7fa5f4072340 2024-03-06T07:39:11.624 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:11.624 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:11.624 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d006b4d0 msgr2=0x7fa5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d006b4d0 0x7fa5d006d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa5f40b16e0 tx=0x7fa5dc006bf0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 msgr2=0x7fa5f4144400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 0x7fa5f4144400 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009890 tx=0x7fa5e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 shutdown_connections 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa5f415acb0 0x7fa5f4144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5f40b4680 0x7fa5f4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 --2- 172.21.15.97:0/2269169802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f4072340 0x7fa5f4144400 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.624+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 >> 172.21.15.97:0/2269169802 conn(0x7fa5f40b1730 msgr2=0x7fa5f40cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:11.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.625+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 shutdown_connections 2024-03-06T07:39:11.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:11.625+0000 7fa5f8e00640 1 -- 172.21.15.97:0/2269169802 wait complete. 2024-03-06T07:39:11.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:11.846 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:12.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:13.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.059+0000 7f3fe9800640 1 Processor -- start 2024-03-06T07:39:13.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.060+0000 7f3fe9800640 1 -- start start 2024-03-06T07:39:13.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.062+0000 7f3fe9800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3fe415f9d0 0x7f3fe415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.062+0000 7f3fe9800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe406cef0 con 0x7f3fe4144c00 2024-03-06T07:39:13.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.062+0000 7f3fe9800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe406d1d0 con 0x7f3fe4070b10 2024-03-06T07:39:13.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.062+0000 7f3fe9800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe406d4b0 con 0x7f3fe415f9d0 2024-03-06T07:39:13.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.063+0000 7f3fe2a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3fe4144c00 0x7f3fe406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55928/0 (socket says 172.21.15.97:55928) 2024-03-06T07:39:13.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.063+0000 7f3fe2a00640 1 -- 172.21.15.97:0/1375396624 learned_addr learned my addr 172.21.15.97:0/1375396624 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.063+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440870337 0 0) 0x7f3fe406cef0 con 0x7f3fe4144c00 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.063+0000 7f3fda000640 1 --2- 172.21.15.97:0/1375396624 >> v2:172.21.15.135:3300/0 conn(0x7f3fe415f9d0 0x7f3fe415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.063+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc8003520 con 0x7f3fe4144c00 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.063+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 916706730 0 0) 0x7f3fc8003520 con 0x7f3fe4144c00 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 >> v2:172.21.15.135:3300/0 conn(0x7f3fe415f9d0 msgr2=0x7f3fe415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 --2- 172.21.15.97:0/1375396624 >> v2:172.21.15.135:3300/0 conn(0x7f3fe415f9d0 0x7f3fe415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 >> v1:172.21.15.135:6789/0 conn(0x7f3fe4070b10 legacy=0x7f3fe4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe406d790 con 0x7f3fe4144c00 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3fd40019f0 con 0x7f3fe4144c00 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fda000640 1 --2- 172.21.15.97:0/1375396624 >> v2:172.21.15.135:3300/0 conn(0x7f3fe415f9d0 0x7f3fe415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 >> v1:172.21.15.97:6789/0 conn(0x7f3fe4144c00 legacy=0x7f3fe406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fc80039b0 0x7f3fc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc8004800 0x7f3fc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fc80051c0 0x7f3fc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe406d4b0 con 0x7f3fc80039b0 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc800c110 con 0x7f3fc8004800 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc800c310 con 0x7f3fc80051c0 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fc80039b0 0x7f3fc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fda000640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc8004800 0x7f3fc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe3400640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fc80051c0 0x7f3fc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2a00640 1 -- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fc80051c0 msgr2=0x7f3fc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fc80051c0 0x7f3fc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2a00640 1 -- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc8004800 msgr2=0x7f3fc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc8004800 0x7f3fc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.064+0000 7f3fe2a00640 1 -- 172.21.15.97:0/1375396624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8003520 con 0x7f3fc80039b0 2024-03-06T07:39:13.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.065+0000 7f3fe3400640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fc80051c0 0x7f3fc800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:13.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.065+0000 7f3fe3400640 1 -- 172.21.15.97:0/1375396624 reap_dead start 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fc80039b0 0x7f3fc8003dd0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400a850 tx=0x7f3fc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d40bb6454ad4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400a4a0 con 0x7f3fc80039b0 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd4034650 con 0x7f3fc80039b0 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe2000640 1 -- 172.21.15.97:0/1375396624 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4034a80 con 0x7f3fc80039b0 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe9800640 1 -- 172.21.15.97:0/1375396624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fc80039b0 msgr2=0x7f3fc8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fc80039b0 0x7f3fc8003dd0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400a850 tx=0x7f3fc8005830 comp rx=0 tx=0).stop 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe9800640 1 -- 172.21.15.97:0/1375396624 shutdown_connections 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1375396624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fc80039b0 0x7f3fc8003dd0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.066+0000 7f3fe9800640 1 -- 172.21.15.97:0/1375396624 >> 172.21.15.97:0/1375396624 conn(0x7f3fe40b1730 msgr2=0x7f3fe40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.067+0000 7f3fe9800640 1 -- 172.21.15.97:0/1375396624 shutdown_connections 2024-03-06T07:39:13.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.067+0000 7f3fe9800640 1 -- 172.21.15.97:0/1375396624 wait complete. 2024-03-06T07:39:13.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.068+0000 7f3fe9800640 1 Processor -- start 2024-03-06T07:39:13.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.068+0000 7f3fe9800640 1 -- start start 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe9800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc80039b0 0x7f3fe40b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe9800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe415dd20 0x7f3fe40aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe9800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 0x7f3fe41f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe9800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe406ccd0 con 0x7f3fe415d390 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe9800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe406cb50 con 0x7f3fc80039b0 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe9800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe406df90 con 0x7f3fe415dd20 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 0x7f3fe41f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 0x7f3fe41f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44556/0 (socket says 172.21.15.97:44556) 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.070+0000 7f3fe3400640 1 -- 172.21.15.97:0/1909974409 learned_addr learned my addr 172.21.15.97:0/1909974409 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fda000640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe415dd20 0x7f3fe40aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc80039b0 0x7f3fe40b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe3400640 1 -- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe415dd20 msgr2=0x7f3fe40aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe3400640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe415dd20 0x7f3fe40aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe3400640 1 -- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc80039b0 msgr2=0x7f3fe40b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe3400640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc80039b0 0x7f3fe40b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe3400640 1 -- 172.21.15.97:0/1909974409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40b2bd0 con 0x7f3fe415d390 2024-03-06T07:39:13.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe3400640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 0x7f3fe41f7580 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40d0ac0 tx=0x7f3fdc00d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc018070 con 0x7f3fe415d390 2024-03-06T07:39:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc0035f0 con 0x7f3fe415d390 2024-03-06T07:39:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe40b2eb0 con 0x7f3fe415d390 2024-03-06T07:39:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.071+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0039e0 con 0x7f3fe415d390 2024-03-06T07:39:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.072+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe40b3470 con 0x7f3fe415d390 2024-03-06T07:39:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.073+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe40b3670 con 0x7f3fe415d390 2024-03-06T07:39:13.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.074+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fdc01b070 con 0x7f3fe415d390 2024-03-06T07:39:13.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.075+0000 7f3fdb400640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc006b4d0 0x7f3fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.075+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3fdc0896b0 con 0x7f3fe415d390 2024-03-06T07:39:13.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.075+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc006b4d0 0x7f3fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.076+0000 7f3fe2a00640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc006b4d0 0x7f3fc006d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4034000 tx=0x7f3fd402f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.080+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fe40b3670 con 0x7f3fe415d390 2024-03-06T07:39:13.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.315+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe404f330 con 0x7f3fe415d390 2024-03-06T07:39:13.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.316+0000 7f3fdb400640 1 -- 172.21.15.97:0/1909974409 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f3fdc058470 con 0x7f3fe415d390 2024-03-06T07:39:13.318 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:13.318 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:13.318 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.318+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc006b4d0 msgr2=0x7f3fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.318+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc006b4d0 0x7f3fc006d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4034000 tx=0x7f3fd402f000 comp rx=0 tx=0).stop 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.318+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 msgr2=0x7f3fe41f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.318+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 0x7f3fe41f7580 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40d0ac0 tx=0x7f3fdc00d810 comp rx=0 tx=0).stop 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 shutdown_connections 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3fc006b4d0 0x7f3fc006d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fe415d390 0x7f3fe41f7580 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3fe415dd20 0x7f3fe40aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 --2- 172.21.15.97:0/1909974409 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fc80039b0 0x7f3fe40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 >> 172.21.15.97:0/1909974409 conn(0x7f3fe40b1730 msgr2=0x7f3fe406e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 shutdown_connections 2024-03-06T07:39:13.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:13.319+0000 7f3fe9800640 1 -- 172.21.15.97:0/1909974409 wait complete. 2024-03-06T07:39:13.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:13.543 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:14.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:14.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.747+0000 7f3168a00640 1 Processor -- start 2024-03-06T07:39:14.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.747+0000 7f3168a00640 1 -- start start 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.749+0000 7f3168a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f31640af820 0x7f31640b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.749+0000 7f3168a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31640b3da0 con 0x7f316415e160 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.749+0000 7f3168a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316406f1f0 con 0x7f31640b38f0 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.749+0000 7f3168a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316406b0f0 con 0x7f31640af820 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.749+0000 7f3161e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f316415e160 0x7f3164162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60992/0 (socket says 172.21.15.97:60992) 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.749+0000 7f3161e00640 1 -- 172.21.15.97:0/612948675 learned_addr learned my addr 172.21.15.97:0/612948675 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3159400640 1 --2- 172.21.15.97:0/612948675 >> v2:172.21.15.135:3300/0 conn(0x7f31640af820 0x7f31640b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200928398 0 0) 0x7f3154001110 con 0x7f316415e160 2024-03-06T07:39:14.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3148003520 con 0x7f316415e160 2024-03-06T07:39:14.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3497126509 0 0) 0x7f3148003520 con 0x7f316415e160 2024-03-06T07:39:14.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 >> v2:172.21.15.135:3300/0 conn(0x7f31640af820 msgr2=0x7f31640b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 --2- 172.21.15.97:0/612948675 >> v2:172.21.15.135:3300/0 conn(0x7f31640af820 0x7f31640b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 >> v1:172.21.15.135:6789/0 conn(0x7f31640b38f0 legacy=0x7f316406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640afce0 con 0x7f316415e160 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3154001840 con 0x7f316415e160 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 >> v1:172.21.15.97:6789/0 conn(0x7f316415e160 legacy=0x7f3164162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31480039b0 0x7f3148003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3148004800 0x7f3148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f314800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316406b0f0 con 0x7f31480051c0 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314800c160 con 0x7f31480039b0 2024-03-06T07:39:14.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.750+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314800c360 con 0x7f3148004800 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3162800640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f314800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3161e00640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31480039b0 0x7f3148003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3159400640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3148004800 0x7f3148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3162800640 1 -- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3148004800 msgr2=0x7f3148004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3162800640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3148004800 0x7f3148004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3162800640 1 -- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31480039b0 msgr2=0x7f3148003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3162800640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31480039b0 0x7f3148003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3162800640 1 -- 172.21.15.97:0/612948675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3148003520 con 0x7f31480051c0 2024-03-06T07:39:14.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.751+0000 7f3161e00640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31480039b0 0x7f3148003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.753+0000 7f3162800640 1 -- 172.21.15.97:0/612948675 reap_dead start 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.753+0000 7f3162800640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f314800b9c0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f315c001e50 tx=0x7f3148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9217a90456568a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.753+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c001a70 con 0x7f31480051c0 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.753+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c003bb0 con 0x7f31480051c0 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.753+0000 7f3161400640 1 -- 172.21.15.97:0/612948675 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c0050d0 con 0x7f31480051c0 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 -- 172.21.15.97:0/612948675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 msgr2=0x7f314800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f314800b9c0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f315c001e50 tx=0x7f3148005830 comp rx=0 tx=0).stop 2024-03-06T07:39:14.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 -- 172.21.15.97:0/612948675 shutdown_connections 2024-03-06T07:39:14.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 --2- 172.21.15.97:0/612948675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f314800b9c0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 -- 172.21.15.97:0/612948675 >> 172.21.15.97:0/612948675 conn(0x7f31640d0340 msgr2=0x7f31640d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:14.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 -- 172.21.15.97:0/612948675 shutdown_connections 2024-03-06T07:39:14.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.754+0000 7f3168a00640 1 -- 172.21.15.97:0/612948675 wait complete. 2024-03-06T07:39:14.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.756+0000 7f3168a00640 1 Processor -- start 2024-03-06T07:39:14.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.756+0000 7f3168a00640 1 -- start start 2024-03-06T07:39:14.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3168a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f3164145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3168a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641e7540 0x7f31641e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3168a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3164157b20 0x7f3164157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3168a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316406ef10 con 0x7f31480051c0 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3168a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b5620 con 0x7f31641e7540 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3168a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b33b0 con 0x7f3164157b20 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3161e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f3164145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3161e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f3164145740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51750/0 (socket says 172.21.15.97:51750) 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3161e00640 1 -- 172.21.15.97:0/3282990922 learned_addr learned my addr 172.21.15.97:0/3282990922 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.757+0000 7f3159400640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641e7540 0x7f31641e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3162800640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3164157b20 0x7f3164157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3161e00640 1 -- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3164157b20 msgr2=0x7f3164157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3161e00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3164157b20 0x7f3164157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3161e00640 1 -- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641e7540 msgr2=0x7f31641e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3161e00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641e7540 0x7f31641e79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3161e00640 1 -- 172.21.15.97:0/3282990922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3164142b60 con 0x7f31480051c0 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3159400640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641e7540 0x7f31641e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3162800640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3164157b20 0x7f3164157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3161e00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f3164145740 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f3154000ff0 tx=0x7f315400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400ee60 con 0x7f31480051c0 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3154018070 con 0x7f31480051c0 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400caa0 con 0x7f31480051c0 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3164158850 con 0x7f31480051c0 2024-03-06T07:39:14.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.758+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3164158e10 con 0x7f31480051c0 2024-03-06T07:39:14.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.760+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3164074ae0 con 0x7f31480051c0 2024-03-06T07:39:14.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.761+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f315400cc40 con 0x7f31480051c0 2024-03-06T07:39:14.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.761+0000 7f315aa00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f314006b4d0 0x7f314006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.762+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f315408a040 con 0x7f31480051c0 2024-03-06T07:39:14.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.762+0000 7f3159400640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f314006b4d0 0x7f314006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.762+0000 7f3159400640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f314006b4d0 0x7f314006d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f31641f7360 tx=0x7f314c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:14.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:14.766+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3164142b60 con 0x7f31480051c0 2024-03-06T07:39:15.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.001+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f316404f330 con 0x7f31480051c0 2024-03-06T07:39:15.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.002+0000 7f315aa00640 1 -- 172.21.15.97:0/3282990922 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f31640b3da0 con 0x7f31480051c0 2024-03-06T07:39:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:15.005 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:15.005 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f314006b4d0 msgr2=0x7f314006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f314006b4d0 0x7f314006d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f31641f7360 tx=0x7f314c008040 comp rx=0 tx=0).stop 2024-03-06T07:39:15.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 msgr2=0x7f3164145740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f3164145740 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f3154000ff0 tx=0x7f315400ec10 comp rx=0 tx=0).stop 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 shutdown_connections 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f314006b4d0 0x7f314006d990 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3164157b20 0x7f3164157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641e7540 0x7f31641e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 --2- 172.21.15.97:0/3282990922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31480051c0 0x7f3164145740 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 >> 172.21.15.97:0/3282990922 conn(0x7f31640d0340 msgr2=0x7f316406be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 shutdown_connections 2024-03-06T07:39:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:15.005+0000 7f3168a00640 1 -- 172.21.15.97:0/3282990922 wait complete. 2024-03-06T07:39:15.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:15.226 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:16.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:16.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.436+0000 7fe7c0000640 1 Processor -- start 2024-03-06T07:39:16.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.436+0000 7fe7c0000640 1 -- start start 2024-03-06T07:39:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7c0000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe7b815e0d0 0x7fe7b80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7c0000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe7b80d0fc0 0x7fe7b8160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7c0000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b80cea60 con 0x7fe7b815f470 2024-03-06T07:39:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7c0000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b80d2f60 con 0x7fe7b80d0fc0 2024-03-06T07:39:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7c0000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b80d2350 con 0x7fe7b815e0d0 2024-03-06T07:39:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7be000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe7b80d0fc0 0x7fe7b8160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7be000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe7b80d0fc0 0x7fe7b8160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51758/0 (socket says 172.21.15.97:51758) 2024-03-06T07:39:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.439+0000 7fe7be000640 1 -- 172.21.15.97:0/3663422982 learned_addr learned my addr 172.21.15.97:0/3663422982 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.440+0000 7fe7bd600640 1 --2- 172.21.15.97:0/3663422982 >> v2:172.21.15.135:3300/0 conn(0x7fe7b815e0d0 0x7fe7b80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.440+0000 7fe7be000640 1 -- 172.21.15.97:0/3663422982 >> v2:172.21.15.135:3300/0 conn(0x7fe7b815e0d0 msgr2=0x7fe7b80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.440+0000 7fe7be000640 1 --2- 172.21.15.97:0/3663422982 >> v2:172.21.15.135:3300/0 conn(0x7fe7b815e0d0 0x7fe7b80d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.440+0000 7fe7be000640 1 -- 172.21.15.97:0/3663422982 >> v1:172.21.15.135:6789/0 conn(0x7fe7b815f470 legacy=0x7fe7b80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.440+0000 7fe7be000640 1 -- 172.21.15.97:0/3663422982 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b80d2630 con 0x7fe7b80d0fc0 2024-03-06T07:39:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.441+0000 7fe7be000640 1 --2- 172.21.15.97:0/3663422982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b8160dc0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80b6a80 tx=0x7fe7b002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9edaaf3fd31a626 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.442+0000 7fe7b7e00640 1 -- 172.21.15.97:0/3663422982 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b002cb20 con 0x7fe7b80d0fc0 2024-03-06T07:39:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.442+0000 7fe7b7e00640 1 -- 172.21.15.97:0/3663422982 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b002ccc0 con 0x7fe7b80d0fc0 2024-03-06T07:39:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.442+0000 7fe7b7e00640 1 -- 172.21.15.97:0/3663422982 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0034610 con 0x7fe7b80d0fc0 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 -- 172.21.15.97:0/3663422982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b80d0fc0 msgr2=0x7fe7b8160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 --2- 172.21.15.97:0/3663422982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b8160dc0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80b6a80 tx=0x7fe7b002bf80 comp rx=0 tx=0).stop 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 -- 172.21.15.97:0/3663422982 shutdown_connections 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 --2- 172.21.15.97:0/3663422982 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b8160dc0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 --2- 172.21.15.97:0/3663422982 >> v2:172.21.15.135:3300/0 conn(0x7fe7b815e0d0 0x7fe7b80d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 -- 172.21.15.97:0/3663422982 >> 172.21.15.97:0/3663422982 conn(0x7fe7b80d3ec0 msgr2=0x7fe7b80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 -- 172.21.15.97:0/3663422982 shutdown_connections 2024-03-06T07:39:16.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.443+0000 7fe7c0000640 1 -- 172.21.15.97:0/3663422982 wait complete. 2024-03-06T07:39:16.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.445+0000 7fe7c0000640 1 Processor -- start 2024-03-06T07:39:16.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.445+0000 7fe7c0000640 1 -- start start 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7c0000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b80b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7c0000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7b815e0d0 0x7fe7b80caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7c0000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 0x7fe7b80cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7c0000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815fb30 con 0x7fe7b815f470 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7c0000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80d2bf0 con 0x7fe7b80d0fc0 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7c0000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80d1fd0 con 0x7fe7b815e0d0 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 0x7fe7b80cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 0x7fe7b80cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51774/0 (socket says 172.21.15.97:51774) 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 -- 172.21.15.97:0/824598172 learned_addr learned my addr 172.21.15.97:0/824598172 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7bd600640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b80b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7bcc00640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7b815e0d0 0x7fe7b80caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 -- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7b815e0d0 msgr2=0x7fe7b80caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7b815e0d0 0x7fe7b80caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 -- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7b80d0fc0 msgr2=0x7fe7b80b7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b80b7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.446+0000 7fe7be000640 1 -- 172.21.15.97:0/824598172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b81f3e90 con 0x7fe7b815f470 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7bd600640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b80b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7be000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 0x7fe7b80cb4d0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0036040 tx=0x7fe7b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0043020 con 0x7fe7b815f470 2024-03-06T07:39:16.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b003e450 con 0x7fe7b815f470 2024-03-06T07:39:16.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b80b81d0 con 0x7fe7b815f470 2024-03-06T07:39:16.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0009510 con 0x7fe7b815f470 2024-03-06T07:39:16.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.447+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b80b8790 con 0x7fe7b815f470 2024-03-06T07:39:16.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.449+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b804f330 con 0x7fe7b815f470 2024-03-06T07:39:16.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.450+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7b0046020 con 0x7fe7b815f470 2024-03-06T07:39:16.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.451+0000 7fe7b5600640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79806b4d0 0x7fe79806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.451+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe7b00acbf0 con 0x7fe7b815f470 2024-03-06T07:39:16.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.451+0000 7fe7bd600640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79806b4d0 0x7fe79806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.452+0000 7fe7bd600640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79806b4d0 0x7fe79806d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:16.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.459+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7b004d030 con 0x7fe7b815f470 2024-03-06T07:39:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.690+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7b808d920 con 0x7fe7b815f470 2024-03-06T07:39:16.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.691+0000 7fe7b5600640 1 -- 172.21.15.97:0/824598172 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe7b007b920 con 0x7fe7b815f470 2024-03-06T07:39:16.693 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:16.694 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:16.694 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79806b4d0 msgr2=0x7fe79806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79806b4d0 0x7fe79806d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac008040 comp rx=0 tx=0).stop 2024-03-06T07:39:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 msgr2=0x7fe7b80cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 0x7fe7b80cb4d0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0036040 tx=0x7fe7b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 shutdown_connections 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79806b4d0 0x7fe79806d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7b815f470 0x7fe7b80cb4d0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7b815e0d0 0x7fe7b80caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 --2- 172.21.15.97:0/824598172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7b80d0fc0 0x7fe7b80b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 >> 172.21.15.97:0/824598172 conn(0x7fe7b80d3ec0 msgr2=0x7fe7b806bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 shutdown_connections 2024-03-06T07:39:16.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:16.694+0000 7fe7c0000640 1 -- 172.21.15.97:0/824598172 wait complete. 2024-03-06T07:39:16.915 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:16.916 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:17.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:18.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.137+0000 7f1287800640 1 Processor -- start 2024-03-06T07:39:18.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.137+0000 7f1287800640 1 -- start start 2024-03-06T07:39:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.138+0000 7f1287800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f128015f8d0 0x7f12800cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.138+0000 7f1287800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1280070de0 con 0x7f128015f4f0 2024-03-06T07:39:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.138+0000 7f1287800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800cd0a0 con 0x7f12800ccc40 2024-03-06T07:39:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.138+0000 7f1287800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1280160a20 con 0x7f128015f8d0 2024-03-06T07:39:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.139+0000 7f1284e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f128015f4f0 0x7f12800706d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60996/0 (socket says 172.21.15.97:60996) 2024-03-06T07:39:18.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.139+0000 7f1284e00640 1 -- 172.21.15.97:0/4057075749 learned_addr learned my addr 172.21.15.97:0/4057075749 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:18.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.139+0000 7f1277e00640 1 --2- 172.21.15.97:0/4057075749 >> v2:172.21.15.135:3300/0 conn(0x7f128015f8d0 0x7f12800cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.139+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515234529 0 0) 0x7f1280070de0 con 0x7f128015f4f0 2024-03-06T07:39:18.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.139+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1264003520 con 0x7f128015f4f0 2024-03-06T07:39:18.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299925545 0 0) 0x7f1264003520 con 0x7f128015f4f0 2024-03-06T07:39:18.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 >> v2:172.21.15.135:3300/0 conn(0x7f128015f8d0 msgr2=0x7f12800cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 --2- 172.21.15.97:0/4057075749 >> v2:172.21.15.135:3300/0 conn(0x7f128015f8d0 0x7f12800cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 >> v1:172.21.15.135:6789/0 conn(0x7f12800ccc40 legacy=0x7f128015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128006ef50 con 0x7f128015f4f0 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1270001670 con 0x7f128015f4f0 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 >> v1:172.21.15.97:6789/0 conn(0x7f128015f4f0 legacy=0x7f12800706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1264004800 0x7f1264004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f126400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280160a20 con 0x7f12640051c0 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126400c0c0 con 0x7f1264004800 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126400c2c0 con 0x7f1264003a00 2024-03-06T07:39:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f1285800640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f126400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.140+0000 7f1277e00640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1264004800 0x7f1264004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1285800640 1 -- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1264003a00 msgr2=0x7f1264003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1284e00640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1285800640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1285800640 1 -- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1264004800 msgr2=0x7f1264004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1285800640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1264004800 0x7f1264004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1285800640 1 -- 172.21.15.97:0/4057075749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264003520 con 0x7f12640051c0 2024-03-06T07:39:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.141+0000 7f1284e00640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.142+0000 7f1285800640 1 -- 172.21.15.97:0/4057075749 reap_dead start 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.142+0000 7f1285800640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f126400b9c0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f1278001e50 tx=0x7f1264005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748dbeebb9adb405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278001a70 con 0x7f12640051c0 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1278003bb0 con 0x7f12640051c0 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f127fe00640 1 -- 172.21.15.97:0/4057075749 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12780050d0 con 0x7f12640051c0 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 -- 172.21.15.97:0/4057075749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 msgr2=0x7f126400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f126400b9c0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f1278001e50 tx=0x7f1264005830 comp rx=0 tx=0).stop 2024-03-06T07:39:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 -- 172.21.15.97:0/4057075749 shutdown_connections 2024-03-06T07:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 --2- 172.21.15.97:0/4057075749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f126400b9c0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 -- 172.21.15.97:0/4057075749 >> 172.21.15.97:0/4057075749 conn(0x7f12800b3800 msgr2=0x7f12800b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 -- 172.21.15.97:0/4057075749 shutdown_connections 2024-03-06T07:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.143+0000 7f1287800640 1 -- 172.21.15.97:0/4057075749 wait complete. 2024-03-06T07:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.146+0000 7f1287800640 1 Processor -- start 2024-03-06T07:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.146+0000 7f1287800640 1 -- start start 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1287800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f128006da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1287800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f128015d890 0x7f128006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1287800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1280156d70 0x7f128015ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1287800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800cc880 con 0x7f12640051c0 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1287800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800cc700 con 0x7f1280156d70 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1287800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12800cca00 con 0x7f128015d890 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1284e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f128006da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1284e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f128006da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51786/0 (socket says 172.21.15.97:51786) 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.148+0000 7f1284e00640 1 -- 172.21.15.97:0/2011650009 learned_addr learned my addr 172.21.15.97:0/2011650009 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1284e00640 1 -- 172.21.15.97:0/2011650009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f128015d890 msgr2=0x7f128006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1284e00640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f128015d890 0x7f128006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1284e00640 1 -- 172.21.15.97:0/2011650009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1280156d70 msgr2=0x7f128015ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1284e00640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1280156d70 0x7f128015ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1284e00640 1 -- 172.21.15.97:0/2011650009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128006c9d0 con 0x7f12640051c0 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1284e00640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f128006da70 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f127000d800 tx=0x7f127000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270019070 con 0x7f12640051c0 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12700074e0 con 0x7f12640051c0 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128006ccb0 con 0x7f12640051c0 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12700049e0 con 0x7f12640051c0 2024-03-06T07:39:18.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.149+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128006d270 con 0x7f12640051c0 2024-03-06T07:39:18.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.151+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128004f330 con 0x7f12640051c0 2024-03-06T07:39:18.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.151+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f127001c070 con 0x7f12640051c0 2024-03-06T07:39:18.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.153+0000 7f127d600640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f125c06b500 0x7f125c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.153+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1270089d50 con 0x7f12640051c0 2024-03-06T07:39:18.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.153+0000 7f1277e00640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f125c06b500 0x7f125c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.154+0000 7f1277e00640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f125c06b500 0x7f125c06d9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f128015a820 tx=0x7f1268008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.157+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1270058b10 con 0x7f12640051c0 2024-03-06T07:39:18.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.435+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12800cff00 con 0x7f12640051c0 2024-03-06T07:39:18.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.436+0000 7f127d600640 1 -- 172.21.15.97:0/2011650009 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f1270058410 con 0x7f12640051c0 2024-03-06T07:39:18.439 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:18.439 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:18.439 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:18.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f125c06b500 msgr2=0x7f125c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f125c06b500 0x7f125c06d9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f128015a820 tx=0x7f1268008040 comp rx=0 tx=0).stop 2024-03-06T07:39:18.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 msgr2=0x7f128006da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f128006da70 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f127000d800 tx=0x7f127000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 shutdown_connections 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f125c06b500 0x7f125c06d9c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1280156d70 0x7f128015ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f128015d890 0x7f128006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 --2- 172.21.15.97:0/2011650009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12640051c0 0x7f128006da70 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 >> 172.21.15.97:0/2011650009 conn(0x7f12800b3800 msgr2=0x7f128015fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 shutdown_connections 2024-03-06T07:39:18.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:18.439+0000 7f1287800640 1 -- 172.21.15.97:0/2011650009 wait complete. 2024-03-06T07:39:18.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:18.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:19.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:19.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.869+0000 7f9f43200640 1 Processor -- start 2024-03-06T07:39:19.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.869+0000 7f9f43200640 1 -- start start 2024-03-06T07:39:19.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.871+0000 7f9f43200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 0x7f9f3c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.871+0000 7f9f43200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c06cef0 con 0x7f9f3c144c00 2024-03-06T07:39:19.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.871+0000 7f9f43200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c06d1d0 con 0x7f9f3c15f9d0 2024-03-06T07:39:19.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.871+0000 7f9f43200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06d4b0 con 0x7f9f3c15fe60 2024-03-06T07:39:19.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.871+0000 7f9f3be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9f3c144c00 0x7f9f3c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:32776/0 (socket says 172.21.15.97:32776) 2024-03-06T07:39:19.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.871+0000 7f9f3be00640 1 -- 172.21.15.97:0/2288406294 learned_addr learned my addr 172.21.15.97:0/2288406294 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429673275 0 0) 0x7f9f3c06cef0 con 0x7f9f3c144c00 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20003520 con 0x7f9f3c144c00 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f41200640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 0x7f9f3c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432880581 0 0) 0x7f9f20003520 con 0x7f9f3c144c00 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 msgr2=0x7f9f3c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 0x7f9f3c071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 >> v1:172.21.15.135:6789/0 conn(0x7f9f3c15f9d0 legacy=0x7f9f3c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c06d790 con 0x7f9f3c144c00 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9f2c0019f0 con 0x7f9f3c144c00 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 >> v1:172.21.15.97:6789/0 conn(0x7f9f3c144c00 legacy=0x7f9f3c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f200039e0 0x7f9f20003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 0x7f9f20005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06d4b0 con 0x7f9f200039e0 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2000baf0 con 0x7f9f20004800 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.872+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2000bcf0 con 0x7f9f3c15fe60 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f200039e0 0x7f9f20003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f3be00640 1 -- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 msgr2=0x7f9f20005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f3c15fe60 0x7f9f20005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f3be00640 1 -- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f20004800 msgr2=0x7f9f20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f33400640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.873+0000 7f9f3be00640 1 -- 172.21.15.97:0/2288406294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20003520 con 0x7f9f200039e0 2024-03-06T07:39:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.874+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f200039e0 0x7f9f20003e00 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00a8d0 tx=0x7f9f20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5773fbda2168bb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.874+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c00a4a0 con 0x7f9f200039e0 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.874+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c034650 con 0x7f9f200039e0 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.874+0000 7f9f3b400640 1 -- 172.21.15.97:0/2288406294 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c034a40 con 0x7f9f200039e0 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f43200640 1 -- 172.21.15.97:0/2288406294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f200039e0 msgr2=0x7f9f20003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f43200640 1 --2- 172.21.15.97:0/2288406294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f200039e0 0x7f9f20003e00 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00a8d0 tx=0x7f9f20005830 comp rx=0 tx=0).stop 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f41200640 1 -- 172.21.15.97:0/2288406294 reap_dead start 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f43200640 1 -- 172.21.15.97:0/2288406294 shutdown_connections 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f43200640 1 -- 172.21.15.97:0/2288406294 >> 172.21.15.97:0/2288406294 conn(0x7f9f3c0b1730 msgr2=0x7f9f3c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f43200640 1 -- 172.21.15.97:0/2288406294 shutdown_connections 2024-03-06T07:39:19.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.875+0000 7f9f43200640 1 -- 172.21.15.97:0/2288406294 wait complete. 2024-03-06T07:39:19.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.878+0000 7f9f43200640 1 Processor -- start 2024-03-06T07:39:19.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.878+0000 7f9f43200640 1 -- start start 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f43200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f200039e0 0x7f9f3c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f43200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 0x7f9f3c15d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f43200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f3c13cdd0 0x7f9f3c15de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f43200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06cb50 con 0x7f9f3c14f930 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f43200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c160730 con 0x7f9f3c13cdd0 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f43200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06ccd0 con 0x7f9f200039e0 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f33400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 0x7f9f3c15d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f33400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 0x7f9f3c15d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51806/0 (socket says 172.21.15.97:51806) 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.880+0000 7f9f33400640 1 -- 172.21.15.97:0/2717567203 learned_addr learned my addr 172.21.15.97:0/2717567203 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f200039e0 0x7f9f3c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f41200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f3c13cdd0 0x7f9f3c15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f33400640 1 -- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f200039e0 msgr2=0x7f9f3c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f33400640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f200039e0 0x7f9f3c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f33400640 1 -- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f3c13cdd0 msgr2=0x7f9f3c15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f33400640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f3c13cdd0 0x7f9f3c15de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f33400640 1 -- 172.21.15.97:0/2717567203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c1f1d40 con 0x7f9f3c14f930 2024-03-06T07:39:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f33400640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 0x7f9f3c15d910 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400d5d0 tx=0x7f9f2400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.881+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24005480 con 0x7f9f3c14f930 2024-03-06T07:39:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.882+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f24017070 con 0x7f9f3c14f930 2024-03-06T07:39:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.882+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c1f2000 con 0x7f9f3c14f930 2024-03-06T07:39:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.882+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24003810 con 0x7f9f3c14f930 2024-03-06T07:39:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.882+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c1f25c0 con 0x7f9f3c14f930 2024-03-06T07:39:19.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.883+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c04f330 con 0x7f9f3c14f930 2024-03-06T07:39:19.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.884+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f240039b0 con 0x7f9f3c14f930 2024-03-06T07:39:19.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.885+0000 7f9f38c00640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f1806b500 0x7f9f1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.885+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9f2408a090 con 0x7f9f3c14f930 2024-03-06T07:39:19.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.885+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f1806b500 0x7f9f1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.886+0000 7f9f3be00640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f1806b500 0x7f9f1806d9c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c034000 tx=0x7f9f2c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:19.889+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f24058e50 con 0x7f9f3c14f930 2024-03-06T07:39:20.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.122+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f3c072890 con 0x7f9f3c14f930 2024-03-06T07:39:20.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.123+0000 7f9f38c00640 1 -- 172.21.15.97:0/2717567203 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f9f24058750 con 0x7f9f3c14f930 2024-03-06T07:39:20.125 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:20.125 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:20.125 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f1806b500 msgr2=0x7f9f1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f1806b500 0x7f9f1806d9c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c034000 tx=0x7f9f2c009030 comp rx=0 tx=0).stop 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 msgr2=0x7f9f3c15d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 0x7f9f3c15d910 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400d5d0 tx=0x7f9f2400da90 comp rx=0 tx=0).stop 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 shutdown_connections 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9f1806b500 0x7f9f1806d9c0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f3c13cdd0 0x7f9f3c15de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f3c14f930 0x7f9f3c15d910 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 --2- 172.21.15.97:0/2717567203 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9f200039e0 0x7f9f3c13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.125+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 >> 172.21.15.97:0/2717567203 conn(0x7f9f3c0b1730 msgr2=0x7f9f3c0b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.126+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 shutdown_connections 2024-03-06T07:39:20.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:20.126+0000 7f9f43200640 1 -- 172.21.15.97:0/2717567203 wait complete. 2024-03-06T07:39:20.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:20.348 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:21.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:21.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.555+0000 7f13dee00640 1 Processor -- start 2024-03-06T07:39:21.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.555+0000 7f13dee00640 1 -- start start 2024-03-06T07:39:21.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13dee00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f13d815e0d0 0x7f13d80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13dee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d8160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13dee00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80cea60 con 0x7f13d815f470 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13dee00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80d2f60 con 0x7f13d815e0d0 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13dee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80d2350 con 0x7f13d80d0fc0 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13d7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f13d815e0d0 0x7f13d80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13d7e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f13d815e0d0 0x7f13d80d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51820/0 (socket says 172.21.15.97:51820) 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.557+0000 7f13d7e00640 1 -- 172.21.15.97:0/3396480091 learned_addr learned my addr 172.21.15.97:0/3396480091 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:21.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.558+0000 7f13dce00640 1 --2- 172.21.15.97:0/3396480091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d8160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.558+0000 7f13d7e00640 1 -- 172.21.15.97:0/3396480091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 msgr2=0x7f13d8160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.558+0000 7f13d7e00640 1 --2- 172.21.15.97:0/3396480091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d8160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.558+0000 7f13d7e00640 1 -- 172.21.15.97:0/3396480091 >> v1:172.21.15.135:6789/0 conn(0x7f13d815f470 legacy=0x7f13d80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.558+0000 7f13d7e00640 1 -- 172.21.15.97:0/3396480091 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80d2630 con 0x7f13d815e0d0 2024-03-06T07:39:21.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.559+0000 7f13d7e00640 1 --2- 172.21.15.97:0/3396480091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815e0d0 0x7f13d80d1a90 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009870 tx=0x7f13c802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a25fb657e9ff850 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.560+0000 7f13d7400640 1 -- 172.21.15.97:0/3396480091 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c802f840 con 0x7f13d815e0d0 2024-03-06T07:39:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.560+0000 7f13d7400640 1 -- 172.21.15.97:0/3396480091 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c802f9e0 con 0x7f13d815e0d0 2024-03-06T07:39:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.560+0000 7f13d7400640 1 -- 172.21.15.97:0/3396480091 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c802fdb0 con 0x7f13d815e0d0 2024-03-06T07:39:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.560+0000 7f13dee00640 1 -- 172.21.15.97:0/3396480091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815e0d0 msgr2=0x7f13d80d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.560+0000 7f13dee00640 1 --2- 172.21.15.97:0/3396480091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815e0d0 0x7f13d80d1a90 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009870 tx=0x7f13c802ba20 comp rx=0 tx=0).stop 2024-03-06T07:39:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.561+0000 7f13dee00640 1 -- 172.21.15.97:0/3396480091 shutdown_connections 2024-03-06T07:39:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.561+0000 7f13dee00640 1 --2- 172.21.15.97:0/3396480091 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d8160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.561+0000 7f13dee00640 1 --2- 172.21.15.97:0/3396480091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815e0d0 0x7f13d80d1a90 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.561+0000 7f13dee00640 1 -- 172.21.15.97:0/3396480091 >> 172.21.15.97:0/3396480091 conn(0x7f13d80d3ec0 msgr2=0x7f13d80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.561+0000 7f13dee00640 1 -- 172.21.15.97:0/3396480091 shutdown_connections 2024-03-06T07:39:21.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.561+0000 7f13dee00640 1 -- 172.21.15.97:0/3396480091 wait complete. 2024-03-06T07:39:21.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.562+0000 7f13dee00640 1 Processor -- start 2024-03-06T07:39:21.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 -- start start 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d80cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f13d815e0d0 0x7f13d81f8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 0x7f13d80c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d815fb30 con 0x7f13d815f470 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80d2bf0 con 0x7f13d815e0d0 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.563+0000 7f13dee00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80d1fd0 con 0x7f13d80d0fc0 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 0x7f13d80c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 0x7f13d80c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51832/0 (socket says 172.21.15.97:51832) 2024-03-06T07:39:21.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 -- 172.21.15.97:0/2480324172 learned_addr learned my addr 172.21.15.97:0/2480324172 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13d7e00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d80cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13cf400640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f13d815e0d0 0x7f13d81f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 -- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 msgr2=0x7f13d80cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d80cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 -- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f13d815e0d0 msgr2=0x7f13d81f8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f13d815e0d0 0x7f13d81f8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13dce00640 1 -- 172.21.15.97:0/2480324172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d81f8ac0 con 0x7f13d815f470 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.564+0000 7f13cf400640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f13d815e0d0 0x7f13d81f8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.565+0000 7f13dce00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 0x7f13d80c9e20 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f13d0007e30 tx=0x7f13d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.565+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0014620 con 0x7f13d815f470 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.565+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0019070 con 0x7f13d815f470 2024-03-06T07:39:21.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.565+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0014a00 con 0x7f13d815f470 2024-03-06T07:39:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.565+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d80cdf50 con 0x7f13d815f470 2024-03-06T07:39:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.565+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d80cb970 con 0x7f13d815f470 2024-03-06T07:39:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.566+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d804f330 con 0x7f13d815f470 2024-03-06T07:39:21.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.567+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13d0014ba0 con 0x7f13d815f470 2024-03-06T07:39:21.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.568+0000 7f13d4c00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f13b406b4d0 0x7f13b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.568+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f13d008ab10 con 0x7f13d815f470 2024-03-06T07:39:21.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.568+0000 7f13d7e00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f13b406b4d0 0x7f13b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.569+0000 7f13d7e00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f13b406b4d0 0x7f13b406d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f13c8036000 tx=0x7f13c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:21.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.573+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13d81f8ac0 con 0x7f13d815f470 2024-03-06T07:39:21.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.805+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13d808d920 con 0x7f13d815f470 2024-03-06T07:39:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.806+0000 7f13d4c00640 1 -- 172.21.15.97:0/2480324172 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f13d0059960 con 0x7f13d815f470 2024-03-06T07:39:21.808 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:21.809 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:21.809 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f13b406b4d0 msgr2=0x7f13b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f13b406b4d0 0x7f13b406d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f13c8036000 tx=0x7f13c8008030 comp rx=0 tx=0).stop 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 msgr2=0x7f13d80c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 0x7f13d80c9e20 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f13d0007e30 tx=0x7f13d000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 shutdown_connections 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f13b406b4d0 0x7f13b406d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13d815f470 0x7f13d80c9e20 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f13d815e0d0 0x7f13d81f8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 --2- 172.21.15.97:0/2480324172 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f13d80d0fc0 0x7f13d80cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 >> 172.21.15.97:0/2480324172 conn(0x7f13d80d3ec0 msgr2=0x7f13d80b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:21.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 shutdown_connections 2024-03-06T07:39:21.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:21.809+0000 7f13dee00640 1 -- 172.21.15.97:0/2480324172 wait complete. 2024-03-06T07:39:22.030 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:22.030 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:23.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:23.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.241+0000 7fd004400640 1 Processor -- start 2024-03-06T07:39:23.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.241+0000 7fd004400640 1 -- start start 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.243+0000 7fd004400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc144c00 0x7fcffc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.243+0000 7fd004400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcffc15f9d0 0x7fcffc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.243+0000 7fd004400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc06ee70 con 0x7fcffc0ce0a0 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.243+0000 7fd004400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcffc160370 con 0x7fcffc15f9d0 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.243+0000 7fd004400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc06cef0 con 0x7fcffc144c00 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.243+0000 7fd001000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcffc15f9d0 0x7fcffc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcffc15f9d0 0x7fcffc15fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51834/0 (socket says 172.21.15.97:51834) 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001000640 1 -- 172.21.15.97:0/4127602279 learned_addr learned my addr 172.21.15.97:0/4127602279 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd002400640 1 --1- 172.21.15.97:0/4127602279 >> v1:172.21.15.97:6789/0 conn(0x7fcffc0ce0a0 0x7fcffc0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:32784/0 (socket says 172.21.15.97:32784) 2024-03-06T07:39:23.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001a00640 1 --2- 172.21.15.97:0/4127602279 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc144c00 0x7fcffc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001000640 1 -- 172.21.15.97:0/4127602279 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc144c00 msgr2=0x7fcffc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:23.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001000640 1 --2- 172.21.15.97:0/4127602279 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc144c00 0x7fcffc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001000640 1 -- 172.21.15.97:0/4127602279 >> v1:172.21.15.97:6789/0 conn(0x7fcffc0ce0a0 legacy=0x7fcffc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.244+0000 7fd001000640 1 -- 172.21.15.97:0/4127602279 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc06d1d0 con 0x7fcffc15f9d0 2024-03-06T07:39:23.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.245+0000 7fd001000640 1 --2- 172.21.15.97:0/4127602279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc15f9d0 0x7fcffc15fe30 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009870 tx=0x7fcff002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ca5674eebb9331 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:23.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.245+0000 7fcffbe00640 1 -- 172.21.15.97:0/4127602279 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0034070 con 0x7fcffc15f9d0 2024-03-06T07:39:23.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fcffbe00640 1 -- 172.21.15.97:0/4127602279 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0030840 con 0x7fcffc15f9d0 2024-03-06T07:39:23.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fcffbe00640 1 -- 172.21.15.97:0/4127602279 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0030c10 con 0x7fcffc15f9d0 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 -- 172.21.15.97:0/4127602279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc15f9d0 msgr2=0x7fcffc15fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 --2- 172.21.15.97:0/4127602279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc15f9d0 0x7fcffc15fe30 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009870 tx=0x7fcff002b980 comp rx=0 tx=0).stop 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 -- 172.21.15.97:0/4127602279 shutdown_connections 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 --2- 172.21.15.97:0/4127602279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc15f9d0 0x7fcffc15fe30 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 --2- 172.21.15.97:0/4127602279 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc144c00 0x7fcffc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 -- 172.21.15.97:0/4127602279 >> 172.21.15.97:0/4127602279 conn(0x7fcffc0b1730 msgr2=0x7fcffc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 -- 172.21.15.97:0/4127602279 shutdown_connections 2024-03-06T07:39:23.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.246+0000 7fd004400640 1 -- 172.21.15.97:0/4127602279 wait complete. 2024-03-06T07:39:23.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.248+0000 7fd004400640 1 Processor -- start 2024-03-06T07:39:23.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.248+0000 7fd004400640 1 -- start start 2024-03-06T07:39:23.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd004400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcffc0ce0a0 0x7fcffc1e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd004400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 0x7fcffc15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd004400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc15f9d0 0x7fcffc1e84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd004400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc160730 con 0x7fcffc144c00 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd004400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc06d790 con 0x7fcffc0ce0a0 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd004400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc06cb50 con 0x7fcffc15f9d0 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd001000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 0x7fcffc15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd001000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 0x7fcffc15b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51850/0 (socket says 172.21.15.97:51850) 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.249+0000 7fd001000640 1 -- 172.21.15.97:0/1776104874 learned_addr learned my addr 172.21.15.97:0/1776104874 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001a00640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcffc0ce0a0 0x7fcffc1e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd002400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc15f9d0 0x7fcffc1e84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001000640 1 -- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc15f9d0 msgr2=0x7fcffc1e84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001000640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc15f9d0 0x7fcffc1e84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001000640 1 -- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcffc0ce0a0 msgr2=0x7fcffc1e5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001000640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcffc0ce0a0 0x7fcffc1e5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001000640 1 -- 172.21.15.97:0/1776104874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc12ffe0 con 0x7fcffc144c00 2024-03-06T07:39:23.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.250+0000 7fd001000640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 0x7fcffc15b650 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fcff002b9c0 tx=0x7fcff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.251+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0034040 con 0x7fcffc144c00 2024-03-06T07:39:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.251+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0030640 con 0x7fcffc144c00 2024-03-06T07:39:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.251+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc130280 con 0x7fcffc144c00 2024-03-06T07:39:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.251+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00423f0 con 0x7fcffc144c00 2024-03-06T07:39:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.251+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc130840 con 0x7fcffc144c00 2024-03-06T07:39:23.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.252+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc072160 con 0x7fcffc144c00 2024-03-06T07:39:23.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.253+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff0037070 con 0x7fcffc144c00 2024-03-06T07:39:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.254+0000 7fcff9600640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfe006b4d0 0x7fcfe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.254+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcff00ac560 con 0x7fcffc144c00 2024-03-06T07:39:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.254+0000 7fd001a00640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfe006b4d0 0x7fcfe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.255+0000 7fd001a00640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfe006b4d0 0x7fcfe006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcfec006fd0 tx=0x7fcfec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:23.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.259+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcffc072160 con 0x7fcffc144c00 2024-03-06T07:39:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.505+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcffc04f330 con 0x7fcffc144c00 2024-03-06T07:39:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.506+0000 7fcff9600640 1 -- 172.21.15.97:0/1776104874 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fcff007b440 con 0x7fcffc144c00 2024-03-06T07:39:23.508 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:23.508 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfe006b4d0 msgr2=0x7fcfe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfe006b4d0 0x7fcfe006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcfec006fd0 tx=0x7fcfec008040 comp rx=0 tx=0).stop 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 msgr2=0x7fcffc15b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 0x7fcffc15b650 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fcff002b9c0 tx=0x7fcff00046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 shutdown_connections 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcfe006b4d0 0x7fcfe006d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcffc15f9d0 0x7fcffc1e84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcffc144c00 0x7fcffc15b650 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 --2- 172.21.15.97:0/1776104874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcffc0ce0a0 0x7fcffc1e5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.508+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 >> 172.21.15.97:0/1776104874 conn(0x7fcffc0b1730 msgr2=0x7fcffc145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.509+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 shutdown_connections 2024-03-06T07:39:23.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:23.509+0000 7fd004400640 1 -- 172.21.15.97:0/1776104874 wait complete. 2024-03-06T07:39:23.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:23.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:24.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:24.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.943+0000 7f65bc600640 1 Processor -- start 2024-03-06T07:39:24.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.943+0000 7f65bc600640 1 -- start start 2024-03-06T07:39:24.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65bc600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f65b415f470 0x7f65b40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65bc600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40d1200 con 0x7f65b40d0df0 2024-03-06T07:39:24.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65bc600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40cea60 con 0x7f65b415e0d0 2024-03-06T07:39:24.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65bc600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b40d2f60 con 0x7f65b415f470 2024-03-06T07:39:24.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65ba600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f65b40d0df0 0x7f65b4160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59248/0 (socket says 172.21.15.97:59248) 2024-03-06T07:39:24.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65ba600640 1 -- 172.21.15.97:0/2484315152 learned_addr learned my addr 172.21.15.97:0/2484315152 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.945+0000 7f65b9200640 1 --2- 172.21.15.97:0/2484315152 >> v2:172.21.15.135:3300/0 conn(0x7f65b415f470 0x7f65b40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997217082 0 0) 0x7f65ac001490 con 0x7f65b40d0df0 2024-03-06T07:39:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659c003520 con 0x7f65b40d0df0 2024-03-06T07:39:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128925345 0 0) 0x7f659c003520 con 0x7f65b40d0df0 2024-03-06T07:39:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 >> v2:172.21.15.135:3300/0 conn(0x7f65b415f470 msgr2=0x7f65b40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 --2- 172.21.15.97:0/2484315152 >> v2:172.21.15.135:3300/0 conn(0x7f65b415f470 0x7f65b40d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 >> v1:172.21.15.135:6789/0 conn(0x7f65b415e0d0 legacy=0x7f65b40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40d2350 con 0x7f65b40d0df0 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f65ac001bc0 con 0x7f65b40d0df0 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 >> v1:172.21.15.97:6789/0 conn(0x7f65b40d0df0 legacy=0x7f65b4160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f659c0039b0 0x7f659c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c004800 0x7f659c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f659c0051c0 0x7f659c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c00c040 con 0x7f659c0039b0 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c00c1f0 con 0x7f659c0051c0 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.946+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c00c3f0 con 0x7f659c004800 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9c00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f659c0039b0 0x7f659c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9c00640 1 -- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c004800 msgr2=0x7f659c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:24.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9200640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c004800 0x7f659c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65ba600640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f659c0051c0 0x7f659c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9c00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c004800 0x7f659c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9c00640 1 -- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f659c0051c0 msgr2=0x7f659c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9c00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f659c0051c0 0x7f659c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65b9c00640 1 -- 172.21.15.97:0/2484315152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c003520 con 0x7f659c0039b0 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65ba600640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f659c0051c0 0x7f659c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:24.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.947+0000 7f65ba600640 1 -- 172.21.15.97:0/2484315152 reap_dead start 2024-03-06T07:39:24.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.948+0000 7f65b9c00640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f659c0039b0 0x7f659c003dd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f65a8001720 tx=0x7f659c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec1791344e30fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8001370 con 0x7f659c0039b0 2024-03-06T07:39:24.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a80052d0 con 0x7f659c0039b0 2024-03-06T07:39:24.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65b3e00640 1 -- 172.21.15.97:0/2484315152 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8003830 con 0x7f659c0039b0 2024-03-06T07:39:24.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 -- 172.21.15.97:0/2484315152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f659c0039b0 msgr2=0x7f659c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f659c0039b0 0x7f659c003dd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f65a8001720 tx=0x7f659c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 -- 172.21.15.97:0/2484315152 shutdown_connections 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 --2- 172.21.15.97:0/2484315152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f659c0039b0 0x7f659c003dd0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 -- 172.21.15.97:0/2484315152 >> 172.21.15.97:0/2484315152 conn(0x7f65b40d3ec0 msgr2=0x7f65b40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 -- 172.21.15.97:0/2484315152 shutdown_connections 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.949+0000 7f65bc600640 1 -- 172.21.15.97:0/2484315152 wait complete. 2024-03-06T07:39:24.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.951+0000 7f65bc600640 1 Processor -- start 2024-03-06T07:39:24.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.951+0000 7f65bc600640 1 -- start start 2024-03-06T07:39:24.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65bc600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c0039b0 0x7f65b40ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65bc600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65b414de70 0x7f65b4151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65bc600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 0x7f65b40b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65bc600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40d2bf0 con 0x7f65b40b4730 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65bc600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40d2910 con 0x7f65b414de70 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65bc600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b415fb30 con 0x7f659c0039b0 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65ba600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 0x7f65b40b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.952+0000 7f65ba600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 0x7f65b40b4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56322/0 (socket says 172.21.15.97:56322) 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 -- 172.21.15.97:0/4152142338 learned_addr learned my addr 172.21.15.97:0/4152142338 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65b9200640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65b414de70 0x7f65b4151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65b9c00640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c0039b0 0x7f65b40ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 -- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c0039b0 msgr2=0x7f65b40ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c0039b0 0x7f65b40ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 -- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65b414de70 msgr2=0x7f65b4151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65b414de70 0x7f65b4151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 -- 172.21.15.97:0/4152142338 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b414edc0 con 0x7f65b40b4730 2024-03-06T07:39:24.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.953+0000 7f65ba600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 0x7f65b40b4b90 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f65b408d000 tx=0x7f65ac00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.954+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00bdb0 con 0x7f65b40b4730 2024-03-06T07:39:24.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.954+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac004b60 con 0x7f65b40b4730 2024-03-06T07:39:24.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.954+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac007070 con 0x7f65b40b4730 2024-03-06T07:39:24.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.954+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b414e520 con 0x7f65b40b4730 2024-03-06T07:39:24.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.954+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b414eae0 con 0x7f65b40b4730 2024-03-06T07:39:24.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.955+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b404f330 con 0x7f65b40b4730 2024-03-06T07:39:24.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.956+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65ac01b020 con 0x7f65b40b4730 2024-03-06T07:39:24.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.957+0000 7f65b1600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f659406b4d0 0x7f659406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.957+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f65ac08a400 con 0x7f65b40b4730 2024-03-06T07:39:24.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.957+0000 7f65b9c00640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f659406b4d0 0x7f659406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.958+0000 7f65b9c00640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f659406b4d0 0x7f659406d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f65a8033000 tx=0x7f65a8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:24.961+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65b40d1200 con 0x7f65b40b4730 2024-03-06T07:39:25.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.202+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b415d9f0 con 0x7f65b40b4730 2024-03-06T07:39:25.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.203+0000 7f65b1600640 1 -- 172.21.15.97:0/4152142338 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f65ac0591c0 con 0x7f65b40b4730 2024-03-06T07:39:25.205 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:25.205 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:25.206 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.205+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f659406b4d0 msgr2=0x7f659406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.205+0000 7f65bc600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f659406b4d0 0x7f659406d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f65a8033000 tx=0x7f65a8009030 comp rx=0 tx=0).stop 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.205+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 msgr2=0x7f65b40b4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.205+0000 7f65bc600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 0x7f65b40b4b90 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f65b408d000 tx=0x7f65ac00b4a0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 shutdown_connections 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f659406b4d0 0x7f659406d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f65b40b4730 0x7f65b40b4b90 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65b414de70 0x7f65b4151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 --2- 172.21.15.97:0/4152142338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f659c0039b0 0x7f65b40ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 >> 172.21.15.97:0/4152142338 conn(0x7f65b40d3ec0 msgr2=0x7f65b40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:25.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 shutdown_connections 2024-03-06T07:39:25.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:25.206+0000 7f65bc600640 1 -- 172.21.15.97:0/4152142338 wait complete. 2024-03-06T07:39:25.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:25.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:26.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:26.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.643+0000 7fa7dd000640 1 Processor -- start 2024-03-06T07:39:26.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.643+0000 7fa7dd000640 1 -- start start 2024-03-06T07:39:26.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.645+0000 7fa7dd000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa7d8144c00 0x7fa7d806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.645+0000 7fa7dd000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.645+0000 7fa7dd000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d806ee70 con 0x7fa7d80ce0a0 2024-03-06T07:39:26.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.645+0000 7fa7dd000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d8160370 con 0x7fa7d8144c00 2024-03-06T07:39:26.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.645+0000 7fa7dd000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d806cef0 con 0x7fa7d815f9d0 2024-03-06T07:39:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.646+0000 7fa7cfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.646+0000 7fa7d5e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa7d8144c00 0x7fa7d806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.646+0000 7fa7d5e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa7d8144c00 0x7fa7d806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:52616/0 (socket says 172.21.15.97:52616) 2024-03-06T07:39:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.646+0000 7fa7cfe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59446/0 (socket says 172.21.15.97:59446) 2024-03-06T07:39:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.646+0000 7fa7d5e00640 1 -- 172.21.15.97:0/2775249893 learned_addr learned my addr 172.21.15.97:0/2775249893 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:26.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.647+0000 7fa7d5e00640 1 -- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 msgr2=0x7fa7d815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.647+0000 7fa7d5e00640 1 --2- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.647+0000 7fa7d5e00640 1 -- 172.21.15.97:0/2775249893 >> v1:172.21.15.135:6789/0 conn(0x7fa7d80ce0a0 legacy=0x7fa7d80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.647+0000 7fa7d5e00640 1 -- 172.21.15.97:0/2775249893 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d806d1d0 con 0x7fa7d8144c00 2024-03-06T07:39:26.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.647+0000 7fa7d5400640 1 -- 172.21.15.97:0/2775249893 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624989039 0 0) 0x7fa7d0001490 con 0x7fa7d80ce0a0 2024-03-06T07:39:26.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.648+0000 7fa7d5e00640 1 --2- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d8144c00 0x7fa7d806e930 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009870 tx=0x7fa7c802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b2e0744691e9fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.649+0000 7fa7d5400640 1 -- 172.21.15.97:0/2775249893 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8034070 con 0x7fa7d8144c00 2024-03-06T07:39:26.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.649+0000 7fa7d5400640 1 -- 172.21.15.97:0/2775249893 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c8030840 con 0x7fa7d8144c00 2024-03-06T07:39:26.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.649+0000 7fa7d5400640 1 -- 172.21.15.97:0/2775249893 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8030c10 con 0x7fa7d8144c00 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 -- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d8144c00 msgr2=0x7fa7d806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 --2- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d8144c00 0x7fa7d806e930 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009870 tx=0x7fa7c802b980 comp rx=0 tx=0).stop 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 -- 172.21.15.97:0/2775249893 shutdown_connections 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 --2- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 --2- 172.21.15.97:0/2775249893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d8144c00 0x7fa7d806e930 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 -- 172.21.15.97:0/2775249893 >> 172.21.15.97:0/2775249893 conn(0x7fa7d80b1730 msgr2=0x7fa7d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 -- 172.21.15.97:0/2775249893 shutdown_connections 2024-03-06T07:39:26.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.650+0000 7fa7dd000640 1 -- 172.21.15.97:0/2775249893 wait complete. 2024-03-06T07:39:26.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.652+0000 7fa7dd000640 1 Processor -- start 2024-03-06T07:39:26.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.652+0000 7fa7dd000640 1 -- start start 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.653+0000 7fa7dd000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d80ce0a0 0x7fa7d80b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.653+0000 7fa7dd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 0x7fa7d815edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.653+0000 7fa7dd000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.653+0000 7fa7dd000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d8160730 con 0x7fa7d8144c00 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.653+0000 7fa7dd000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d806d790 con 0x7fa7d80ce0a0 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.653+0000 7fa7dd000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d806cb50 con 0x7fa7d815f9d0 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 0x7fa7d815edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 0x7fa7d815edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56324/0 (socket says 172.21.15.97:56324) 2024-03-06T07:39:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 -- 172.21.15.97:0/3231388712 learned_addr learned my addr 172.21.15.97:0/3231388712 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7d5e00640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d80ce0a0 0x7fa7d80b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7d6800640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 -- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 msgr2=0x7fa7d815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 -- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d80ce0a0 msgr2=0x7fa7d80b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d80ce0a0 0x7fa7d80b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.654+0000 7fa7cfe00640 1 -- 172.21.15.97:0/3231388712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d815dff0 con 0x7fa7d8144c00 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.655+0000 7fa7cfe00640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 0x7fa7d815edb0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000e9b0 tx=0x7fa7c000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.655+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0017070 con 0x7fa7d8144c00 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.655+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c000c9a0 con 0x7fa7d8144c00 2024-03-06T07:39:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.655+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c000cd90 con 0x7fa7d8144c00 2024-03-06T07:39:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.655+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d815e2d0 con 0x7fa7d8144c00 2024-03-06T07:39:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.655+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d815dca0 con 0x7fa7d8144c00 2024-03-06T07:39:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.656+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d804f330 con 0x7fa7d8144c00 2024-03-06T07:39:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.657+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7c001a070 con 0x7fa7d8144c00 2024-03-06T07:39:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.659+0000 7fa7ce000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7b406b500 0x7fa7b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.659+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa7c008a5c0 con 0x7fa7d8144c00 2024-03-06T07:39:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.659+0000 7fa7d5e00640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7b406b500 0x7fa7b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.660+0000 7fa7d5e00640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7b406b500 0x7fa7b406d9c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8036000 tx=0x7fa7c803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.663+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7c0059410 con 0x7fa7d8144c00 2024-03-06T07:39:26.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.896+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7d8072890 con 0x7fa7d8144c00 2024-03-06T07:39:26.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.897+0000 7fa7ce000640 1 -- 172.21.15.97:0/3231388712 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fa7c0058d10 con 0x7fa7d8144c00 2024-03-06T07:39:26.899 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:26.900 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:26.900 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:26.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7b406b500 msgr2=0x7fa7b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7b406b500 0x7fa7b406d9c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8036000 tx=0x7fa7c803b040 comp rx=0 tx=0).stop 2024-03-06T07:39:26.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 msgr2=0x7fa7d815edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 0x7fa7d815edb0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000e9b0 tx=0x7fa7c000ee70 comp rx=0 tx=0).stop 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 shutdown_connections 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa7b406b500 0x7fa7b406d9c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa7d815f9d0 0x7fa7d815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7d8144c00 0x7fa7d815edb0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 --2- 172.21.15.97:0/3231388712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7d80ce0a0 0x7fa7d80b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.899+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 >> 172.21.15.97:0/3231388712 conn(0x7fa7d80b1730 msgr2=0x7fa7d80ced80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.900+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 shutdown_connections 2024-03-06T07:39:26.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:26.900+0000 7fa7dd000640 1 -- 172.21.15.97:0/3231388712 wait complete. 2024-03-06T07:39:27.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:27.126 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:28.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:28.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.360+0000 7fe0f2400640 1 Processor -- start 2024-03-06T07:39:28.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.360+0000 7fe0f2400640 1 -- start start 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.361+0000 7fe0f2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 0x7fe0ec0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f2400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe0ec0aebf0 0x7fe0ec0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f2400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec15c240 con 0x7fe0ec0cc800 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f2400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec15ce30 con 0x7fe0ec0aebf0 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec15c500 con 0x7fe0ec1561f0 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f1a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe0ec0cc800 0x7fe0ec0cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59256/0 (socket says 172.21.15.97:59256) 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f1a00640 1 -- 172.21.15.97:0/2856163435 learned_addr learned my addr 172.21.15.97:0/2856163435 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0f1000640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 0x7fe0ec0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.362+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012641432 0 0) 0x7fe0ec15c240 con 0x7fe0ec0cc800 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d0003520 con 0x7fe0ec0cc800 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2856163435 >> v2:172.21.15.135:3300/0 conn(0x7fe0ec0aebf0 0x7fe0ec0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711789308 0 0) 0x7fe0d0003520 con 0x7fe0ec0cc800 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 msgr2=0x7fe0ec0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 0x7fe0ec0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 >> v2:172.21.15.135:3300/0 conn(0x7fe0ec0aebf0 msgr2=0x7fe0ec0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 --2- 172.21.15.97:0/2856163435 >> v2:172.21.15.135:3300/0 conn(0x7fe0ec0aebf0 0x7fe0ec0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec15c7e0 con 0x7fe0ec0cc800 2024-03-06T07:39:28.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe0e4001670 con 0x7fe0ec0cc800 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 >> v1:172.21.15.97:6789/0 conn(0x7fe0ec0cc800 legacy=0x7fe0ec0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 0x7fe0d00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d0004d40 0x7fe0d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec15ce30 con 0x7fe0d0003f80 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec15c500 con 0x7fe0d0004d40 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d000ba70 con 0x7fe0ec1561f0 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.363+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0ebe00640 1 -- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 msgr2=0x7fe0d00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec1561f0 0x7fe0d00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0ebe00640 1 -- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d0004d40 msgr2=0x7fe0d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:28.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0f1a00640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d0004d40 0x7fe0d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d0004d40 0x7fe0d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0ebe00640 1 -- 172.21.15.97:0/2856163435 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0003520 con 0x7fe0d0003f80 2024-03-06T07:39:28.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.364+0000 7fe0f1a00640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d0004d40 0x7fe0d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:28.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.365+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0d0004800 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400ba30 tx=0x7fe0d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8874c21331442b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.365+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4036020 con 0x7fe0d0003f80 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.365+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d4032680 con 0x7fe0d0003f80 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.365+0000 7fe0eb400640 1 -- 172.21.15.97:0/2856163435 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400b600 con 0x7fe0d0003f80 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f2400640 1 -- 172.21.15.97:0/2856163435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 msgr2=0x7fe0d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2856163435 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0d0004800 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400ba30 tx=0x7fe0d0005830 comp rx=0 tx=0).stop 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f1a00640 1 -- 172.21.15.97:0/2856163435 reap_dead start 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f2400640 1 -- 172.21.15.97:0/2856163435 shutdown_connections 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f2400640 1 -- 172.21.15.97:0/2856163435 >> 172.21.15.97:0/2856163435 conn(0x7fe0ec06afc0 msgr2=0x7fe0ec06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f2400640 1 -- 172.21.15.97:0/2856163435 shutdown_connections 2024-03-06T07:39:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.366+0000 7fe0f2400640 1 -- 172.21.15.97:0/2856163435 wait complete. 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.368+0000 7fe0f2400640 1 Processor -- start 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.368+0000 7fe0f2400640 1 -- start start 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0ec1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f2400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec146400 0x7fe0ec15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f2400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0ec1582e0 0x7fe0ec15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec1565d0 con 0x7fe0d0003f80 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f2400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0ccc90 con 0x7fe0ec1582e0 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f2400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec156750 con 0x7fe0ec146400 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0ec1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0ec1519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56348/0 (socket says 172.21.15.97:56348) 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f1000640 1 -- 172.21.15.97:0/2600104511 learned_addr learned my addr 172.21.15.97:0/2600104511 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.369+0000 7fe0f1000640 1 -- 172.21.15.97:0/2600104511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec146400 msgr2=0x7fe0ec15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f1000640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec146400 0x7fe0ec15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f1000640 1 -- 172.21.15.97:0/2600104511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0ec1582e0 msgr2=0x7fe0ec15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f1000640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0ec1582e0 0x7fe0ec15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f1000640 1 -- 172.21.15.97:0/2600104511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec151ef0 con 0x7fe0d0003f80 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f1000640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0ec1519b0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00ef10 tx=0x7fe0dc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc018020 con 0x7fe0d0003f80 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc014400 con 0x7fe0d0003f80 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0147d0 con 0x7fe0d0003f80 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec1521c0 con 0x7fe0d0003f80 2024-03-06T07:39:28.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.370+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec152780 con 0x7fe0d0003f80 2024-03-06T07:39:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.371+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec04f330 con 0x7fe0d0003f80 2024-03-06T07:39:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.372+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0dc005d10 con 0x7fe0d0003f80 2024-03-06T07:39:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.374+0000 7fe0e8c00640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0c806b4d0 0x7fe0c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.374+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe0dc08abb0 con 0x7fe0d0003f80 2024-03-06T07:39:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.374+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0c806b4d0 0x7fe0c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.375+0000 7fe0ebe00640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0c806b4d0 0x7fe0c806d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400ba00 tx=0x7fe0d4032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:28.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.379+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0dc022030 con 0x7fe0d0003f80 2024-03-06T07:39:28.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.644+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0ec0b2b10 con 0x7fe0d0003f80 2024-03-06T07:39:28.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.645+0000 7fe0e8c00640 1 -- 172.21.15.97:0/2600104511 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe0dc059970 con 0x7fe0d0003f80 2024-03-06T07:39:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:28.648 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0c806b4d0 msgr2=0x7fe0c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0c806b4d0 0x7fe0c806d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400ba00 tx=0x7fe0d4032000 comp rx=0 tx=0).stop 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 msgr2=0x7fe0ec1519b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0ec1519b0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00ef10 tx=0x7fe0dc00b480 comp rx=0 tx=0).stop 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 shutdown_connections 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0c806b4d0 0x7fe0c806d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0ec1582e0 0x7fe0ec15b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0ec146400 0x7fe0ec15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 --2- 172.21.15.97:0/2600104511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0d0003f80 0x7fe0ec1519b0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 >> 172.21.15.97:0/2600104511 conn(0x7fe0ec06afc0 msgr2=0x7fe0ec06d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:28.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 shutdown_connections 2024-03-06T07:39:28.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:28.648+0000 7fe0f2400640 1 -- 172.21.15.97:0/2600104511 wait complete. 2024-03-06T07:39:28.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:28.869 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:29.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:30.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.079+0000 7ff370e00640 1 Processor -- start 2024-03-06T07:39:30.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.080+0000 7ff370e00640 1 -- start start 2024-03-06T07:39:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.081+0000 7ff370e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff36c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.081+0000 7ff370e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c071d30 con 0x7ff36c0af1c0 2024-03-06T07:39:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.081+0000 7ff370e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c15f470 con 0x7ff36c06e4d0 2024-03-06T07:39:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.081+0000 7ff370e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c071260 con 0x7ff36c15eff0 2024-03-06T07:39:30.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.081+0000 7ff369e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff36c0af1c0 0x7ff36c06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59270/0 (socket says 172.21.15.97:59270) 2024-03-06T07:39:30.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.081+0000 7ff369e00640 1 -- 172.21.15.97:0/2977227739 learned_addr learned my addr 172.21.15.97:0/2977227739 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff361400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff36c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704233878 0 0) 0x7ff35c001110 con 0x7ff36c0af1c0 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff350003520 con 0x7ff36c0af1c0 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 122389575 0 0) 0x7ff350003520 con 0x7ff36c0af1c0 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 msgr2=0x7ff36c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff36c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 >> v1:172.21.15.135:6789/0 conn(0x7ff36c06e4d0 legacy=0x7ff36c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c071490 con 0x7ff36c0af1c0 2024-03-06T07:39:30.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff35c001840 con 0x7ff36c0af1c0 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 >> v1:172.21.15.97:6789/0 conn(0x7ff36c0af1c0 legacy=0x7ff36c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff3500039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.082+0000 7ff369400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff350004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff369400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff350004d40 0x7ff3500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35000ba70 con 0x7ff350003f60 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35000bc20 con 0x7ff350004d40 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35000be20 con 0x7ff36c15eff0 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff361400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff350004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff369e00640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff3500039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff36a800640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff350004d40 0x7ff3500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff361400640 1 -- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 msgr2=0x7ff3500039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff361400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff3500039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff361400640 1 -- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff350004d40 msgr2=0x7ff3500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:30.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff361400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff350004d40 0x7ff3500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff361400640 1 -- 172.21.15.97:0/2977227739 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350003520 con 0x7ff350003f60 2024-03-06T07:39:30.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.083+0000 7ff369e00640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c15eff0 0x7ff3500039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.085+0000 7ff361400640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff350004800 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7ff354009ac0 tx=0x7ff350005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411f37ed4ca5bae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.085+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354039070 con 0x7ff350003f60 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.085+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35402c6b0 con 0x7ff350003f60 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.085+0000 7ff369400640 1 -- 172.21.15.97:0/2977227739 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354009690 con 0x7ff350003f60 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff370e00640 1 -- 172.21.15.97:0/2977227739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 msgr2=0x7ff350004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff370e00640 1 --2- 172.21.15.97:0/2977227739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff350004800 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7ff354009ac0 tx=0x7ff350005830 comp rx=0 tx=0).stop 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff36a800640 1 -- 172.21.15.97:0/2977227739 reap_dead start 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff370e00640 1 -- 172.21.15.97:0/2977227739 shutdown_connections 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff370e00640 1 -- 172.21.15.97:0/2977227739 >> 172.21.15.97:0/2977227739 conn(0x7ff36c0b1730 msgr2=0x7ff36c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff370e00640 1 -- 172.21.15.97:0/2977227739 shutdown_connections 2024-03-06T07:39:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.086+0000 7ff370e00640 1 -- 172.21.15.97:0/2977227739 wait complete. 2024-03-06T07:39:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.088+0000 7ff370e00640 1 Processor -- start 2024-03-06T07:39:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.088+0000 7ff370e00640 1 -- start start 2024-03-06T07:39:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff370e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff36c0afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff370e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c14f0a0 0x7ff36c14f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff370e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff36c0cc5f0 0x7ff36c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff370e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c0b30b0 con 0x7ff350003f60 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff370e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c071a50 con 0x7ff36c0cc5f0 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff370e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c06d370 con 0x7ff36c14f0a0 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff369e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff36c0afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff369e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff36c0afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56364/0 (socket says 172.21.15.97:56364) 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.089+0000 7ff369e00640 1 -- 172.21.15.97:0/3572181747 learned_addr learned my addr 172.21.15.97:0/3572181747 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff36a800640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff36c0cc5f0 0x7ff36c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff361400640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c14f0a0 0x7ff36c14f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff369e00640 1 -- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c14f0a0 msgr2=0x7ff36c14f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff369e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c14f0a0 0x7ff36c14f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff369e00640 1 -- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff36c0cc5f0 msgr2=0x7ff36c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff369e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff36c0cc5f0 0x7ff36c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff369e00640 1 -- 172.21.15.97:0/3572181747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c0cdb70 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff36a800640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff36c0cc5f0 0x7ff36c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff369e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff36c0afff0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00d800 tx=0x7ff35c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c019070 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c0074e0 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c14fa60 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0049e0 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.090+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c0cde10 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.091+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36c04f330 con 0x7ff350003f60 2024-03-06T07:39:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.092+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff35c01c070 con 0x7ff350003f60 2024-03-06T07:39:30.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.093+0000 7ff362a00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff34806b4d0 0x7ff34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.093+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff35c089c00 con 0x7ff350003f60 2024-03-06T07:39:30.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.093+0000 7ff361400640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff34806b4d0 0x7ff34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.094+0000 7ff361400640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff34806b4d0 0x7ff34806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff354036000 tx=0x7ff354030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.098+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff36c071d30 con 0x7ff350003f60 2024-03-06T07:39:30.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.326+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff36c06eac0 con 0x7ff350003f60 2024-03-06T07:39:30.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.327+0000 7ff362a00640 1 -- 172.21.15.97:0/3572181747 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7ff35c0583f0 con 0x7ff350003f60 2024-03-06T07:39:30.329 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:30.329 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:30.330 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff34806b4d0 msgr2=0x7ff34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff34806b4d0 0x7ff34806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff354036000 tx=0x7ff354030000 comp rx=0 tx=0).stop 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 msgr2=0x7ff36c0afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff36c0afff0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00d800 tx=0x7ff35c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 shutdown_connections 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff34806b4d0 0x7ff34806d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff36c0cc5f0 0x7ff36c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff36c14f0a0 0x7ff36c14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 --2- 172.21.15.97:0/3572181747 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff350003f60 0x7ff36c0afff0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.329+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 >> 172.21.15.97:0/3572181747 conn(0x7ff36c0b1730 msgr2=0x7ff36c0af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:30.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.330+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 shutdown_connections 2024-03-06T07:39:30.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:30.330+0000 7ff370e00640 1 -- 172.21.15.97:0/3572181747 wait complete. 2024-03-06T07:39:30.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:30.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:31.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:31.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.767+0000 7f7bc6000640 1 Processor -- start 2024-03-06T07:39:31.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.767+0000 7f7bc6000640 1 -- start start 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.768+0000 7f7bc6000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 0x7f7bc00b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.768+0000 7f7bc6000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7bc0072670 0x7f7bc0072ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.768+0000 7f7bc6000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc0066f50 con 0x7f7bc015be20 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bc6000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc0073000 con 0x7f7bc0072670 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bc6000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0066d00 con 0x7f7bc00af5a0 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bc4c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7bc015be20 0x7f7bc00b16a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59286/0 (socket says 172.21.15.97:59286) 2024-03-06T07:39:31.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bc4c00640 1 -- 172.21.15.97:0/2730463721 learned_addr learned my addr 172.21.15.97:0/2730463721 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380141912 0 0) 0x7f7bc0066f50 con 0x7f7bc015be20 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bc5600640 1 --2- 172.21.15.97:0/2730463721 >> v2:172.21.15.135:3300/0 conn(0x7f7bc0072670 0x7f7bc0072ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba8003520 con 0x7f7bc015be20 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.769+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 0x7f7bc00b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4120275692 0 0) 0x7f7ba8003520 con 0x7f7bc015be20 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 msgr2=0x7f7bc00b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 0x7f7bc00b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 >> v2:172.21.15.135:3300/0 conn(0x7f7bc0072670 msgr2=0x7f7bc0072ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 --2- 172.21.15.97:0/2730463721 >> v2:172.21.15.135:3300/0 conn(0x7f7bc0072670 0x7f7bc0072ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0067230 con 0x7f7bc015be20 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f7bb4001670 con 0x7f7bc015be20 2024-03-06T07:39:31.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 >> v1:172.21.15.97:6789/0 conn(0x7f7bc015be20 legacy=0x7f7bc00b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ba80039b0 0x7f7ba8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ba8004800 0x7f7ba8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 0x7f7ba80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0073000 con 0x7f7ba8004800 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0066d00 con 0x7f7ba80039b0 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba800b9f0 con 0x7f7bc00af5a0 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ba8004800 0x7f7ba8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bc4c00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ba80039b0 0x7f7ba8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.770+0000 7f7bc5600640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 0x7f7ba80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.771+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 msgr2=0x7f7ba80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.771+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7bc00af5a0 0x7f7ba80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.771+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ba80039b0 msgr2=0x7f7ba8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.771+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ba80039b0 0x7f7ba8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.771+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2730463721 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8003520 con 0x7f7ba8004800 2024-03-06T07:39:31.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.771+0000 7f7bc4c00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ba80039b0 0x7f7ba8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.772+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ba8004800 0x7f7ba8004c80 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00ba00 tx=0x7f7ba8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d959d6ddd215d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac036020 con 0x7f7ba8004800 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac032620 con 0x7f7ba8004800 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bbf400640 1 -- 172.21.15.97:0/2730463721 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac00b600 con 0x7f7ba8004800 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bc6000640 1 -- 172.21.15.97:0/2730463721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ba8004800 msgr2=0x7f7ba8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2730463721 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ba8004800 0x7f7ba8004c80 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00ba00 tx=0x7f7ba8005830 comp rx=0 tx=0).stop 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bc5600640 1 -- 172.21.15.97:0/2730463721 reap_dead start 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bc6000640 1 -- 172.21.15.97:0/2730463721 shutdown_connections 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bc6000640 1 -- 172.21.15.97:0/2730463721 >> 172.21.15.97:0/2730463721 conn(0x7f7bc00b4fe0 msgr2=0x7f7bc00b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.773+0000 7f7bc6000640 1 -- 172.21.15.97:0/2730463721 shutdown_connections 2024-03-06T07:39:31.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.774+0000 7f7bc6000640 1 -- 172.21.15.97:0/2730463721 wait complete. 2024-03-06T07:39:31.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.775+0000 7f7bc6000640 1 Processor -- start 2024-03-06T07:39:31.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.775+0000 7f7bc6000640 1 -- start start 2024-03-06T07:39:31.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.776+0000 7f7bc6000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7ba8004800 0x7f7bc0152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.776+0000 7f7bc6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 0x7f7bc01f4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.776+0000 7f7bc6000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bc009c170 0x7f7bc009c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.776+0000 7f7bc6000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00b0b50 con 0x7f7bc009ac80 2024-03-06T07:39:31.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.776+0000 7f7bc6000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00b1180 con 0x7f7bc009c170 2024-03-06T07:39:31.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.776+0000 7f7bc6000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00b5db0 con 0x7f7ba8004800 2024-03-06T07:39:31.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 0x7f7bc01f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 0x7f7bc01f4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56386/0 (socket says 172.21.15.97:56386) 2024-03-06T07:39:31.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2869888074 learned_addr learned my addr 172.21.15.97:0/2869888074 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bc4c00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7ba8004800 0x7f7bc0152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bc5600640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bc009c170 0x7f7bc009c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7ba8004800 msgr2=0x7f7bc0152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7ba8004800 0x7f7bc0152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bc009c170 msgr2=0x7f7bc009c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bc009c170 0x7f7bc009c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bbfe00640 1 -- 172.21.15.97:0/2869888074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc009b420 con 0x7f7bc009ac80 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.777+0000 7f7bc5600640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bc009c170 0x7f7bc009c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.778+0000 7f7bbfe00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 0x7f7bc01f4550 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00bb20 tx=0x7f7bac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.778+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac041070 con 0x7f7bc009ac80 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.778+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac032e10 con 0x7f7bc009ac80 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.778+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac036040 con 0x7f7bc009ac80 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.778+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc009b700 con 0x7f7bc009ac80 2024-03-06T07:39:31.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.778+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc014ff30 con 0x7f7bc009ac80 2024-03-06T07:39:31.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.779+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc015b680 con 0x7f7bc009ac80 2024-03-06T07:39:31.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.780+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bac039020 con 0x7f7bc009ac80 2024-03-06T07:39:31.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.781+0000 7f7bbcc00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ba006b500 0x7f7ba006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.781+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7bac0abce0 con 0x7f7bc009ac80 2024-03-06T07:39:31.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.781+0000 7f7bc4c00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ba006b500 0x7f7ba006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.782+0000 7f7bc4c00640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ba006b500 0x7f7ba006d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4001810 tx=0x7f7bb4000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:31.786+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bc015b680 con 0x7f7bc009ac80 2024-03-06T07:39:32.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.021+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bc004f330 con 0x7f7bc009ac80 2024-03-06T07:39:32.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.021+0000 7f7bbcc00640 1 -- 172.21.15.97:0/2869888074 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f7bac07aaa0 con 0x7f7bc009ac80 2024-03-06T07:39:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:32.024 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:32.024 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:32.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ba006b500 msgr2=0x7f7ba006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ba006b500 0x7f7ba006d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4001810 tx=0x7f7bb4000ed0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 msgr2=0x7f7bc01f4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 0x7f7bc01f4550 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00bb20 tx=0x7f7bac004730 comp rx=0 tx=0).stop 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 shutdown_connections 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7ba006b500 0x7f7ba006d9c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7bc009c170 0x7f7bc009c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bc009ac80 0x7f7bc01f4550 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 --2- 172.21.15.97:0/2869888074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7ba8004800 0x7f7bc0152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 >> 172.21.15.97:0/2869888074 conn(0x7f7bc00b4fe0 msgr2=0x7f7bc006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 shutdown_connections 2024-03-06T07:39:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:32.024+0000 7f7bc6000640 1 -- 172.21.15.97:0/2869888074 wait complete. 2024-03-06T07:39:32.247 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:32.247 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:33.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:33.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.457+0000 7fd678c00640 1 Processor -- start 2024-03-06T07:39:33.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.457+0000 7fd678c00640 1 -- start start 2024-03-06T07:39:33.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd678c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd6741561f0 0x7fd6740ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd678c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67415c240 con 0x7fd6740aebf0 2024-03-06T07:39:33.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd678c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67415c500 con 0x7fd6740cc9d0 2024-03-06T07:39:33.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd678c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67415ce30 con 0x7fd6741561f0 2024-03-06T07:39:33.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd66be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd6740aebf0 0x7fd6740cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59294/0 (socket says 172.21.15.97:59294) 2024-03-06T07:39:33.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd66be00640 1 -- 172.21.15.97:0/2802033612 learned_addr learned my addr 172.21.15.97:0/2802033612 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:33.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.459+0000 7fd673400640 1 --2- 172.21.15.97:0/2802033612 >> v2:172.21.15.97:3300/0 conn(0x7fd6741561f0 0x7fd6740ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.460+0000 7fd672a00640 1 -- 172.21.15.97:0/2802033612 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395836463 0 0) 0x7fd67415c240 con 0x7fd6740aebf0 2024-03-06T07:39:33.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.460+0000 7fd672a00640 1 -- 172.21.15.97:0/2802033612 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd658003520 con 0x7fd6740aebf0 2024-03-06T07:39:33.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.460+0000 7fd673400640 1 -- 172.21.15.97:0/2802033612 >> v1:172.21.15.135:6789/0 conn(0x7fd6740cc9d0 legacy=0x7fd67415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:33.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.460+0000 7fd673400640 1 -- 172.21.15.97:0/2802033612 >> v1:172.21.15.97:6789/0 conn(0x7fd6740aebf0 legacy=0x7fd6740cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.460+0000 7fd673400640 1 -- 172.21.15.97:0/2802033612 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67415c7e0 con 0x7fd6741561f0 2024-03-06T07:39:33.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.461+0000 7fd673400640 1 --2- 172.21.15.97:0/2802033612 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd6740ae6b0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd6640097a0 tx=0x7fd66402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a48d4e13dff056d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:33.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.461+0000 7fd672a00640 1 -- 172.21.15.97:0/2802033612 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664034070 con 0x7fd6741561f0 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.461+0000 7fd672a00640 1 -- 172.21.15.97:0/2802033612 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664030610 con 0x7fd6741561f0 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.461+0000 7fd672a00640 1 -- 172.21.15.97:0/2802033612 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6640309e0 con 0x7fd6741561f0 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 -- 172.21.15.97:0/2802033612 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 msgr2=0x7fd6740ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 --2- 172.21.15.97:0/2802033612 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd6740ae6b0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd6640097a0 tx=0x7fd66402b960 comp rx=0 tx=0).stop 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 -- 172.21.15.97:0/2802033612 shutdown_connections 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 --2- 172.21.15.97:0/2802033612 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd6740ae6b0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 -- 172.21.15.97:0/2802033612 >> 172.21.15.97:0/2802033612 conn(0x7fd67406afc0 msgr2=0x7fd67406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 -- 172.21.15.97:0/2802033612 shutdown_connections 2024-03-06T07:39:33.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.462+0000 7fd678c00640 1 -- 172.21.15.97:0/2802033612 wait complete. 2024-03-06T07:39:33.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.464+0000 7fd678c00640 1 Processor -- start 2024-03-06T07:39:33.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.464+0000 7fd678c00640 1 -- start start 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd678c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6740aebf0 0x7fd6740ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd678c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6740cc9d0 0x7fd67414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd678c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd67415e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd678c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6741565d0 con 0x7fd6741561f0 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd678c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67406e360 con 0x7fd6740cc9d0 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd678c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd674156750 con 0x7fd6740aebf0 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd67415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd67415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56410/0 (socket says 172.21.15.97:56410) 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 -- 172.21.15.97:0/1375505154 learned_addr learned my addr 172.21.15.97:0/1375505154 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:33.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673400640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6740aebf0 0x7fd6740ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd66be00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6740cc9d0 0x7fd67414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 -- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6740aebf0 msgr2=0x7fd6740ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6740aebf0 0x7fd6740ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 -- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6740cc9d0 msgr2=0x7fd67414d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6740cc9d0 0x7fd67414d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673e00640 1 -- 172.21.15.97:0/1375505154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6741f80f0 con 0x7fd6741561f0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.466+0000 7fd673400640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6740aebf0 0x7fd6740ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.467+0000 7fd673e00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd67415e430 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c7a0 tx=0x7fd66c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.467+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c003b70 con 0x7fd6741561f0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.467+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c003d30 con 0x7fd6741561f0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.467+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00aa90 con 0x7fd6741561f0 2024-03-06T07:39:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.467+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6740cd500 con 0x7fd6741561f0 2024-03-06T07:39:33.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.467+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6740cdac0 con 0x7fd6741561f0 2024-03-06T07:39:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.468+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67404f330 con 0x7fd6741561f0 2024-03-06T07:39:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.469+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd66c00ac30 con 0x7fd6741561f0 2024-03-06T07:39:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.470+0000 7fd66b400640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd65006b4d0 0x7fd65006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.470+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd66c08a0e0 con 0x7fd6741561f0 2024-03-06T07:39:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.471+0000 7fd673400640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd65006b4d0 0x7fd65006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.471+0000 7fd673400640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd65006b4d0 0x7fd65006d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd664036000 tx=0x7fd664008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:33.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.475+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd66c058fd0 con 0x7fd6741561f0 2024-03-06T07:39:33.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.709+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd67406ce70 con 0x7fd6741561f0 2024-03-06T07:39:33.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.710+0000 7fd66b400640 1 -- 172.21.15.97:0/1375505154 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fd66c0588d0 con 0x7fd6741561f0 2024-03-06T07:39:33.712 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:33.713 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:33.713 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd65006b4d0 msgr2=0x7fd65006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd65006b4d0 0x7fd65006d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd664036000 tx=0x7fd664008030 comp rx=0 tx=0).stop 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 msgr2=0x7fd67415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd67415e430 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c7a0 tx=0x7fd66c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 shutdown_connections 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd65006b4d0 0x7fd65006d990 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6741561f0 0x7fd67415e430 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6740cc9d0 0x7fd67414d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 --2- 172.21.15.97:0/1375505154 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6740aebf0 0x7fd6740ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 >> 172.21.15.97:0/1375505154 conn(0x7fd67406afc0 msgr2=0x7fd67415b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:33.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 shutdown_connections 2024-03-06T07:39:33.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:33.713+0000 7fd678c00640 1 -- 172.21.15.97:0/1375505154 wait complete. 2024-03-06T07:39:33.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:33.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:34.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:35.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.141+0000 7f79c8e00640 1 Processor -- start 2024-03-06T07:39:35.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.141+0000 7f79c8e00640 1 -- start start 2024-03-06T07:39:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.142+0000 7f79c8e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 0x7f79c40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.142+0000 7f79c8e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f79c415f470 0x7f79c40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79c8e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c40cea60 con 0x7f79c40d0df0 2024-03-06T07:39:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79c8e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c40d1260 con 0x7f79c415f470 2024-03-06T07:39:35.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79c8e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c40d2f60 con 0x7f79c415e0d0 2024-03-06T07:39:35.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79bbe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f79c415f470 0x7f79c40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79c1e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 0x7f79c40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79c1e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 0x7f79c40d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59116/0 (socket says 172.21.15.97:59116) 2024-03-06T07:39:35.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.143+0000 7f79c1e00640 1 -- 172.21.15.97:0/736493751 learned_addr learned my addr 172.21.15.97:0/736493751 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.144+0000 7f79c1e00640 1 -- 172.21.15.97:0/736493751 >> v2:172.21.15.135:3300/0 conn(0x7f79c415f470 msgr2=0x7f79c40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.144+0000 7f79c1e00640 1 --2- 172.21.15.97:0/736493751 >> v2:172.21.15.135:3300/0 conn(0x7f79c415f470 0x7f79c40d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.144+0000 7f79c1e00640 1 -- 172.21.15.97:0/736493751 >> v1:172.21.15.135:6789/0 conn(0x7f79c40d0df0 legacy=0x7f79c4160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.144+0000 7f79c1e00640 1 -- 172.21.15.97:0/736493751 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c40d2350 con 0x7f79c415e0d0 2024-03-06T07:39:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.144+0000 7f79bbe00640 1 --2- 172.21.15.97:0/736493751 >> v2:172.21.15.135:3300/0 conn(0x7f79c415f470 0x7f79c40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:35.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.146+0000 7f79c1e00640 1 --2- 172.21.15.97:0/736493751 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 0x7f79c40d1a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009870 tx=0x7f79b402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d51392669ba8e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.146+0000 7f79c1400640 1 -- 172.21.15.97:0/736493751 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b402f830 con 0x7f79c415e0d0 2024-03-06T07:39:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.146+0000 7f79c1400640 1 -- 172.21.15.97:0/736493751 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b402f9d0 con 0x7f79c415e0d0 2024-03-06T07:39:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.146+0000 7f79c1400640 1 -- 172.21.15.97:0/736493751 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b402fda0 con 0x7f79c415e0d0 2024-03-06T07:39:35.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 -- 172.21.15.97:0/736493751 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 msgr2=0x7f79c40d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 --2- 172.21.15.97:0/736493751 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 0x7f79c40d1a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009870 tx=0x7f79b402ba10 comp rx=0 tx=0).stop 2024-03-06T07:39:35.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 -- 172.21.15.97:0/736493751 shutdown_connections 2024-03-06T07:39:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 --2- 172.21.15.97:0/736493751 >> v2:172.21.15.135:3300/0 conn(0x7f79c415f470 0x7f79c40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 --2- 172.21.15.97:0/736493751 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415e0d0 0x7f79c40d1a90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 -- 172.21.15.97:0/736493751 >> 172.21.15.97:0/736493751 conn(0x7f79c40d3ec0 msgr2=0x7f79c40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 -- 172.21.15.97:0/736493751 shutdown_connections 2024-03-06T07:39:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.147+0000 7f79c8e00640 1 -- 172.21.15.97:0/736493751 wait complete. 2024-03-06T07:39:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.148+0000 7f79c8e00640 1 Processor -- start 2024-03-06T07:39:35.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.149+0000 7f79c8e00640 1 -- start start 2024-03-06T07:39:35.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.150+0000 7f79c8e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79c40d0df0 0x7f79c40cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.150+0000 7f79c8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 0x7f79c40b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.150+0000 7f79c8e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415f470 0x7f79c40cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.150+0000 7f79c8e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c40d2bf0 con 0x7f79c415e0d0 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.150+0000 7f79c8e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c40d2910 con 0x7f79c40d0df0 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.150+0000 7f79c8e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c415fb30 con 0x7f79c415f470 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 0x7f79c40b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 0x7f79c40b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48008/0 (socket says 172.21.15.97:48008) 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 -- 172.21.15.97:0/1615267654 learned_addr learned my addr 172.21.15.97:0/1615267654 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79c2800640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415f470 0x7f79c40cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79c1e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79c40d0df0 0x7f79c40cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 -- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415f470 msgr2=0x7f79c40cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415f470 0x7f79c40cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 -- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79c40d0df0 msgr2=0x7f79c40cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79c40d0df0 0x7f79c40cbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 -- 172.21.15.97:0/1615267654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c40b6fc0 con 0x7f79c415e0d0 2024-03-06T07:39:35.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.151+0000 7f79bbe00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 0x7f79c40b6a80 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00ebe0 tx=0x7f79ac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.152+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac017070 con 0x7f79c415e0d0 2024-03-06T07:39:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.152+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac00cc30 con 0x7f79c415e0d0 2024-03-06T07:39:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.152+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c40b9b90 con 0x7f79c415e0d0 2024-03-06T07:39:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.152+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac013470 con 0x7f79c415e0d0 2024-03-06T07:39:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.152+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c40cb410 con 0x7f79c415e0d0 2024-03-06T07:39:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.153+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c404f330 con 0x7f79c415e0d0 2024-03-06T07:39:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.155+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79ac01a070 con 0x7f79c415e0d0 2024-03-06T07:39:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.156+0000 7f79ba000640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79a006b4d0 0x7f79a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.156+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f79ac08a170 con 0x7f79c415e0d0 2024-03-06T07:39:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.156+0000 7f79c1e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79a006b4d0 0x7f79a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.157+0000 7f79c1e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79a006b4d0 0x7f79a006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f79b4036000 tx=0x7f79b4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:35.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.161+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79ac058fc0 con 0x7f79c415e0d0 2024-03-06T07:39:35.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.398+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79c408d920 con 0x7f79c415e0d0 2024-03-06T07:39:35.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.399+0000 7f79ba000640 1 -- 172.21.15.97:0/1615267654 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f79ac0588c0 con 0x7f79c415e0d0 2024-03-06T07:39:35.401 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:35.401 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:35.402 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:35.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.401+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79a006b4d0 msgr2=0x7f79a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.401+0000 7f79c8e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79a006b4d0 0x7f79a006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f79b4036000 tx=0x7f79b4008030 comp rx=0 tx=0).stop 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 msgr2=0x7f79c40b6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 0x7f79c40b6a80 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00ebe0 tx=0x7f79ac00c520 comp rx=0 tx=0).stop 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 shutdown_connections 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f79a006b4d0 0x7f79a006d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f79c415f470 0x7f79c40cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79c415e0d0 0x7f79c40b6a80 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 --2- 172.21.15.97:0/1615267654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79c40d0df0 0x7f79c40cbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 >> 172.21.15.97:0/1615267654 conn(0x7f79c40d3ec0 msgr2=0x7f79c40b4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:35.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 shutdown_connections 2024-03-06T07:39:35.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:35.402+0000 7f79c8e00640 1 -- 172.21.15.97:0/1615267654 wait complete. 2024-03-06T07:39:35.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:35.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:36.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:36.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.839+0000 7f32aaa00640 1 Processor -- start 2024-03-06T07:39:36.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.840+0000 7f32aaa00640 1 -- start start 2024-03-06T07:39:36.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32aaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32a415a940 0x7f32a40b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32aaa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a40cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32aaa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40b2b30 con 0x7f32a40cf370 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32aaa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a40b28e0 con 0x7f32a415a940 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32aaa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40b2e10 con 0x7f32a40cc2a0 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32a9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32a415a940 0x7f32a40b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32a9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f32a415a940 0x7f32a40b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48014/0 (socket says 172.21.15.97:48014) 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32a9600640 1 -- 172.21.15.97:0/3447534875 learned_addr learned my addr 172.21.15.97:0/3447534875 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.841+0000 7f32a8c00640 1 --1- 172.21.15.97:0/3447534875 >> v1:172.21.15.97:6789/0 conn(0x7f32a40cf370 0x7f32a406d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48632/0 (socket says 172.21.15.97:48632) 2024-03-06T07:39:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.842+0000 7f32aa000640 1 --2- 172.21.15.97:0/3447534875 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a40cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.842+0000 7f32a9600640 1 -- 172.21.15.97:0/3447534875 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 msgr2=0x7f32a40cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.842+0000 7f32a9600640 1 --2- 172.21.15.97:0/3447534875 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a40cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.842+0000 7f32a9600640 1 -- 172.21.15.97:0/3447534875 >> v1:172.21.15.97:6789/0 conn(0x7f32a40cf370 legacy=0x7f32a406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.842+0000 7f32a9600640 1 -- 172.21.15.97:0/3447534875 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a406ca50 con 0x7f32a415a940 2024-03-06T07:39:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.842+0000 7f32a3e00640 1 -- 172.21.15.97:0/3447534875 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298963238 0 0) 0x7f32a40b2b30 con 0x7f32a40cf370 2024-03-06T07:39:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.843+0000 7f32a9600640 1 --2- 172.21.15.97:0/3447534875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a40b45e0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f329800bb10 tx=0x7f329802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96873a28b5dc30cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32a3e00640 1 -- 172.21.15.97:0/3447534875 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329803b070 con 0x7f32a415a940 2024-03-06T07:39:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32a3e00640 1 -- 172.21.15.97:0/3447534875 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3298036440 con 0x7f32a415a940 2024-03-06T07:39:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32a3e00640 1 -- 172.21.15.97:0/3447534875 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298036810 con 0x7f32a415a940 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32aaa00640 1 -- 172.21.15.97:0/3447534875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 msgr2=0x7f32a40b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32aaa00640 1 --2- 172.21.15.97:0/3447534875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a40b45e0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f329800bb10 tx=0x7f329802e150 comp rx=0 tx=0).stop 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32aaa00640 1 -- 172.21.15.97:0/3447534875 shutdown_connections 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32aaa00640 1 --2- 172.21.15.97:0/3447534875 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a40cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32aaa00640 1 --2- 172.21.15.97:0/3447534875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a40b45e0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.844+0000 7f32aaa00640 1 -- 172.21.15.97:0/3447534875 >> 172.21.15.97:0/3447534875 conn(0x7f32a406afc0 msgr2=0x7f32a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.845+0000 7f32aaa00640 1 -- 172.21.15.97:0/3447534875 shutdown_connections 2024-03-06T07:39:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.845+0000 7f32aaa00640 1 -- 172.21.15.97:0/3447534875 wait complete. 2024-03-06T07:39:36.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.847+0000 7f32aaa00640 1 Processor -- start 2024-03-06T07:39:36.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.847+0000 7f32aaa00640 1 -- start start 2024-03-06T07:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aaa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a41f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aaa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32a40cf370 0x7f32a4153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a4156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aaa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40b4b20 con 0x7f32a415a940 2024-03-06T07:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aaa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40b30f0 con 0x7f32a40cf370 2024-03-06T07:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aaa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a415ad20 con 0x7f32a40cc2a0 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a4156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a4156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48022/0 (socket says 172.21.15.97:48022) 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.848+0000 7f32aa000640 1 -- 172.21.15.97:0/1007935781 learned_addr learned my addr 172.21.15.97:0/1007935781 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32a8c00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32a40cf370 0x7f32a4153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32a9600640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a41f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32aa000640 1 -- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 msgr2=0x7f32a41f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32aa000640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a41f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32aa000640 1 -- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32a40cf370 msgr2=0x7f32a4153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32aa000640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32a40cf370 0x7f32a4153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32aa000640 1 -- 172.21.15.97:0/1007935781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a40af7c0 con 0x7f32a415a940 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32aa000640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a4156e90 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f329c00cc30 tx=0x7f329c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.849+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c00ce40 con 0x7f32a415a940 2024-03-06T07:39:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.850+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c003990 con 0x7f32a415a940 2024-03-06T07:39:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.850+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c003d40 con 0x7f32a415a940 2024-03-06T07:39:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.850+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a4156920 con 0x7f32a415a940 2024-03-06T07:39:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.850+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a41f3a90 con 0x7f32a415a940 2024-03-06T07:39:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.851+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a404f330 con 0x7f32a415a940 2024-03-06T07:39:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.852+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f329c003ee0 con 0x7f32a415a940 2024-03-06T07:39:36.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.853+0000 7f32a1600640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f328406b4d0 0x7f328406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.853+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f329c08a750 con 0x7f32a415a940 2024-03-06T07:39:36.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.853+0000 7f32a9600640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f328406b4d0 0x7f328406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.854+0000 7f32a9600640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f328406b4d0 0x7f328406d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3298038000 tx=0x7f3298008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:36.857+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32a40af7c0 con 0x7f32a415a940 2024-03-06T07:39:37.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.184+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32a406b440 con 0x7f32a415a940 2024-03-06T07:39:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.185+0000 7f32a1600640 1 -- 172.21.15.97:0/1007935781 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f329c014030 con 0x7f32a415a940 2024-03-06T07:39:37.187 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:37.187 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:37.187 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:37.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.187+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f328406b4d0 msgr2=0x7f328406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:37.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.187+0000 7f32aaa00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f328406b4d0 0x7f328406d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3298038000 tx=0x7f3298008040 comp rx=0 tx=0).stop 2024-03-06T07:39:37.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.187+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 msgr2=0x7f32a4156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.187+0000 7f32aaa00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a4156e90 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f329c00cc30 tx=0x7f329c00a5c0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 shutdown_connections 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f328406b4d0 0x7f328406d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32a415a940 0x7f32a4156e90 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32a40cf370 0x7f32a4153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 --2- 172.21.15.97:0/1007935781 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f32a40cc2a0 0x7f32a41f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 >> 172.21.15.97:0/1007935781 conn(0x7f32a406afc0 msgr2=0x7f32a4153e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 shutdown_connections 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:37.188+0000 7f32aaa00640 1 -- 172.21.15.97:0/1007935781 wait complete. 2024-03-06T07:39:37.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:37.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:38.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:38.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.612+0000 7fe80c600640 1 Processor -- start 2024-03-06T07:39:38.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.613+0000 7fe80c600640 1 -- start start 2024-03-06T07:39:38.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.614+0000 7fe80c600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80415bff0 0x7fe8040d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.614+0000 7fe80c600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe80415fac0 0x7fe80415ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.614+0000 7fe80c600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80415c3d0 con 0x7fe8041606d0 2024-03-06T07:39:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.614+0000 7fe80c600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe804160470 con 0x7fe80415fac0 2024-03-06T07:39:38.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.614+0000 7fe80c600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040d0bc0 con 0x7fe80415bff0 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe809c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80415bff0 0x7fe8040d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe80a600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe80415fac0 0x7fe80415ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe809c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80415bff0 0x7fe8040d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59142/0 (socket says 172.21.15.97:59142) 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe809c00640 1 -- 172.21.15.97:0/2866499338 learned_addr learned my addr 172.21.15.97:0/2866499338 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe80a600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe80415fac0 0x7fe80415ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:37120/0 (socket says 172.21.15.97:37120) 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe80a600640 1 -- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80415bff0 msgr2=0x7fe8040d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe80a600640 1 --2- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80415bff0 0x7fe8040d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe80a600640 1 -- 172.21.15.97:0/2866499338 >> v1:172.21.15.135:6789/0 conn(0x7fe8041606d0 legacy=0x7fe8040cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.615+0000 7fe80a600640 1 -- 172.21.15.97:0/2866499338 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8040cdb50 con 0x7fe80415fac0 2024-03-06T07:39:38.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.617+0000 7fe80a600640 1 --2- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe80415ff30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009d90 tx=0x7fe7fc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f095179b56312e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.618+0000 7fe803e00640 1 -- 172.21.15.97:0/2866499338 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc039070 con 0x7fe80415fac0 2024-03-06T07:39:38.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.618+0000 7fe803e00640 1 -- 172.21.15.97:0/2866499338 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc02cc50 con 0x7fe80415fac0 2024-03-06T07:39:38.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.618+0000 7fe803e00640 1 -- 172.21.15.97:0/2866499338 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc034440 con 0x7fe80415fac0 2024-03-06T07:39:38.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 -- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 msgr2=0x7fe80415ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 --2- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe80415ff30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009d90 tx=0x7fe7fc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 -- 172.21.15.97:0/2866499338 shutdown_connections 2024-03-06T07:39:38.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 --2- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe80415ff30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 --2- 172.21.15.97:0/2866499338 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe80415bff0 0x7fe8040d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 -- 172.21.15.97:0/2866499338 >> 172.21.15.97:0/2866499338 conn(0x7fe8040d1c50 msgr2=0x7fe8040d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:38.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 -- 172.21.15.97:0/2866499338 shutdown_connections 2024-03-06T07:39:38.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.619+0000 7fe80c600640 1 -- 172.21.15.97:0/2866499338 wait complete. 2024-03-06T07:39:38.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.621+0000 7fe80c600640 1 Processor -- start 2024-03-06T07:39:38.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.621+0000 7fe80c600640 1 -- start start 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.622+0000 7fe80c600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 0x7fe80415f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe80c600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe8040cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe80c600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8041606d0 0x7fe80415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe80c600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040cfe80 con 0x7fe80415bff0 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe80c600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040cfba0 con 0x7fe80415fac0 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe80c600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040d0160 con 0x7fe8041606d0 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 0x7fe80415f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 0x7fe80415f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48038/0 (socket says 172.21.15.97:48038) 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 -- 172.21.15.97:0/2617788926 learned_addr learned my addr 172.21.15.97:0/2617788926 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809200640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe8040cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe80a600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8041606d0 0x7fe80415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 -- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8041606d0 msgr2=0x7fe80415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8041606d0 0x7fe80415d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 -- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 msgr2=0x7fe8040cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe8040cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809c00640 1 -- 172.21.15.97:0/2617788926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8040cb330 con 0x7fe80415bff0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.623+0000 7fe809200640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe8040cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.624+0000 7fe809c00640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 0x7fe80415f380 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e870 tx=0x7fe7f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.624+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8017070 con 0x7fe80415bff0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.624+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f800c8e0 con 0x7fe80415bff0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.624+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f800ccd0 con 0x7fe80415bff0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.624+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8040cb610 con 0x7fe80415bff0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.624+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8041f1b20 con 0x7fe80415bff0 2024-03-06T07:39:38.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.625+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80404f330 con 0x7fe80415bff0 2024-03-06T07:39:38.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.626+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7f801a070 con 0x7fe80415bff0 2024-03-06T07:39:38.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.628+0000 7fe801600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7e406b4d0 0x7fe7e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.628+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe7f808a570 con 0x7fe80415bff0 2024-03-06T07:39:38.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.628+0000 7fe809200640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7e406b4d0 0x7fe7e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.629+0000 7fe809200640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7e406b4d0 0x7fe7e406d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe8041e5a40 tx=0x7fe7f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.632+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7f80593f0 con 0x7fe80415bff0 2024-03-06T07:39:38.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.870+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8040b7bd0 con 0x7fe80415bff0 2024-03-06T07:39:38.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.871+0000 7fe801600640 1 -- 172.21.15.97:0/2617788926 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe7f8058cf0 con 0x7fe80415bff0 2024-03-06T07:39:38.873 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:38.873 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:38.874 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:38.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7e406b4d0 msgr2=0x7fe7e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7e406b4d0 0x7fe7e406d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe8041e5a40 tx=0x7fe7f0009040 comp rx=0 tx=0).stop 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 msgr2=0x7fe80415f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 0x7fe80415f380 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e870 tx=0x7fe7f800ed30 comp rx=0 tx=0).stop 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 shutdown_connections 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe7e406b4d0 0x7fe7e406d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe8041606d0 0x7fe80415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe80415fac0 0x7fe8040cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 --2- 172.21.15.97:0/2617788926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe80415bff0 0x7fe80415f380 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 >> 172.21.15.97:0/2617788926 conn(0x7fe8040d1c50 msgr2=0x7fe8040af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 shutdown_connections 2024-03-06T07:39:38.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:38.874+0000 7fe80c600640 1 -- 172.21.15.97:0/2617788926 wait complete. 2024-03-06T07:39:39.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:39.095 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:40.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:40.009 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:40.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:40.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.300+0000 7fb168000640 1 Processor -- start 2024-03-06T07:39:40.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.301+0000 7fb168000640 1 -- start start 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb168000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 0x7fb160071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb168000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb16015dff0 0x7fb16015e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb168000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb160071a60 con 0x7fb1600cf430 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb168000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1600b3ee0 con 0x7fb16015dff0 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb168000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600b35b0 con 0x7fb160151660 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb164c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb1600cf430 0x7fb16006d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48636/0 (socket says 172.21.15.97:48636) 2024-03-06T07:39:40.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb164c00640 1 -- 172.21.15.97:0/360456576 learned_addr learned my addr 172.21.15.97:0/360456576 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:40.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb166000640 1 --2- 172.21.15.97:0/360456576 >> v2:172.21.15.135:3300/0 conn(0x7fb16015dff0 0x7fb16015e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.303+0000 7fb165600640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 0x7fb160071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346609009 0 0) 0x7fb160071a60 con 0x7fb1600cf430 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb148005730 con 0x7fb1600cf430 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433778877 0 0) 0x7fb148005730 con 0x7fb1600cf430 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 msgr2=0x7fb160071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 0x7fb160071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 >> v2:172.21.15.135:3300/0 conn(0x7fb16015dff0 msgr2=0x7fb16015e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 --2- 172.21.15.97:0/360456576 >> v2:172.21.15.135:3300/0 conn(0x7fb16015dff0 0x7fb16015e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1600b3890 con 0x7fb1600cf430 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb14c001670 con 0x7fb1600cf430 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 >> v1:172.21.15.97:6789/0 conn(0x7fb1600cf430 legacy=0x7fb16006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb148005bc0 0x7fb148005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb148006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 0x7fb1480073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600b3ee0 con 0x7fb148006a10 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600b35b0 con 0x7fb148005bc0 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.304+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14800dc50 con 0x7fb160151660 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb164c00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb148006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb165600640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb148005bc0 0x7fb148005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb166000640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 0x7fb1480073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb164c00640 1 -- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 msgr2=0x7fb1480073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb164c00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151660 0x7fb1480073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb164c00640 1 -- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb148005bc0 msgr2=0x7fb148005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb164c00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb148005bc0 0x7fb148005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.305+0000 7fb164c00640 1 -- 172.21.15.97:0/360456576 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148005730 con 0x7fb148006a10 2024-03-06T07:39:40.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.306+0000 7fb164c00640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb148006e90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb14c001050 tx=0x7fb148007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e785c3966169e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:40.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.306+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c007d70 con 0x7fb148006a10 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.306+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c030070 con 0x7fb148006a10 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.306+0000 7fb15fe00640 1 -- 172.21.15.97:0/360456576 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c02cd80 con 0x7fb148006a10 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb168000640 1 -- 172.21.15.97:0/360456576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 msgr2=0x7fb148006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb168000640 1 --2- 172.21.15.97:0/360456576 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb148006e90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb14c001050 tx=0x7fb148007a40 comp rx=0 tx=0).stop 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb166000640 1 -- 172.21.15.97:0/360456576 reap_dead start 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb168000640 1 -- 172.21.15.97:0/360456576 shutdown_connections 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb168000640 1 -- 172.21.15.97:0/360456576 >> 172.21.15.97:0/360456576 conn(0x7fb1600b1730 msgr2=0x7fb1600b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb168000640 1 -- 172.21.15.97:0/360456576 shutdown_connections 2024-03-06T07:39:40.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.307+0000 7fb168000640 1 -- 172.21.15.97:0/360456576 wait complete. 2024-03-06T07:39:40.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.310+0000 7fb168000640 1 Processor -- start 2024-03-06T07:39:40.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.310+0000 7fb168000640 1 -- start start 2024-03-06T07:39:40.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb168000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb160151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb168000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151f80 0x7fb1601efca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb168000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1601f01e0 0x7fb1601f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb168000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16006daf0 con 0x7fb148006a10 2024-03-06T07:39:40.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb168000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb160071d30 con 0x7fb1601f01e0 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb168000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb160071520 con 0x7fb160151f80 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb160151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb160151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48060/0 (socket says 172.21.15.97:48060) 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 -- 172.21.15.97:0/956383096 learned_addr learned my addr 172.21.15.97:0/956383096 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 -- 172.21.15.97:0/956383096 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151f80 msgr2=0x7fb1601efca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151f80 0x7fb1601efca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 -- 172.21.15.97:0/956383096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1601f01e0 msgr2=0x7fb1601f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1601f01e0 0x7fb1601f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.312+0000 7fb165600640 1 -- 172.21.15.97:0/956383096 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1601548e0 con 0x7fb148006a10 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.313+0000 7fb165600640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb160151a40 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fb154011540 tx=0x7fb154011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.313+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154015930 con 0x7fb148006a10 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.313+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb154015ad0 con 0x7fb148006a10 2024-03-06T07:39:40.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.313+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1601524e0 con 0x7fb148006a10 2024-03-06T07:39:40.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.313+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15401d400 con 0x7fb148006a10 2024-03-06T07:39:40.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.313+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb160152aa0 con 0x7fb148006a10 2024-03-06T07:39:40.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.314+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16004f330 con 0x7fb148006a10 2024-03-06T07:39:40.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.315+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb15401d5a0 con 0x7fb148006a10 2024-03-06T07:39:40.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.316+0000 7fb15d600640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb14006b500 0x7fb14006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.316+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb15408a9e0 con 0x7fb148006a10 2024-03-06T07:39:40.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.316+0000 7fb164c00640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb14006b500 0x7fb14006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.317+0000 7fb164c00640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb14006b500 0x7fb14006d9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb14c034000 tx=0x7fb14c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:40.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.325+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1601548e0 con 0x7fb148006a10 2024-03-06T07:39:40.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.558+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1600cc6d0 con 0x7fb148006a10 2024-03-06T07:39:40.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.559+0000 7fb15d600640 1 -- 172.21.15.97:0/956383096 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fb15405f070 con 0x7fb148006a10 2024-03-06T07:39:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:40.562 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:40.562 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb14006b500 msgr2=0x7fb14006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb14006b500 0x7fb14006d9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb14c034000 tx=0x7fb14c02e000 comp rx=0 tx=0).stop 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 msgr2=0x7fb160151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb160151a40 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fb154011540 tx=0x7fb154011a00 comp rx=0 tx=0).stop 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 shutdown_connections 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb14006b500 0x7fb14006d9c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1601f01e0 0x7fb1601f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb160151f80 0x7fb1601efca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 --2- 172.21.15.97:0/956383096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb148006a10 0x7fb160151a40 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 >> 172.21.15.97:0/956383096 conn(0x7fb1600b1730 msgr2=0x7fb1600b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 shutdown_connections 2024-03-06T07:39:40.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:40.562+0000 7fb168000640 1 -- 172.21.15.97:0/956383096 wait complete. 2024-03-06T07:39:40.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:40.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:41.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:42.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:41.998+0000 7f601fa00640 1 Processor -- start 2024-03-06T07:39:42.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:41.999+0000 7f601fa00640 1 -- start start 2024-03-06T07:39:42.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.000+0000 7f601fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6018144c00 0x7f601806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.000+0000 7f601fa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f601815f9d0 0x7f601815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.000+0000 7f601fa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601806ee70 con 0x7f60180ce0a0 2024-03-06T07:39:42.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.000+0000 7f601fa00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6018160370 con 0x7f601815f9d0 2024-03-06T07:39:42.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.000+0000 7f601fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601806cef0 con 0x7f6018144c00 2024-03-06T07:39:42.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f601da00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f60180ce0a0 0x7f60180ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48646/0 (socket says 172.21.15.97:48646) 2024-03-06T07:39:42.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f601da00640 1 -- 172.21.15.97:0/4035302530 learned_addr learned my addr 172.21.15.97:0/4035302530 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f600fe00640 1 --2- 172.21.15.97:0/4035302530 >> v2:172.21.15.135:3300/0 conn(0x7f601815f9d0 0x7f601815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977025670 0 0) 0x7f6010001490 con 0x7f60180ce0a0 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f601d000640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6018144c00 0x7f601806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc003520 con 0x7f60180ce0a0 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774689146 0 0) 0x7f5ffc003520 con 0x7f60180ce0a0 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6018144c00 msgr2=0x7f601806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6018144c00 0x7f601806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 >> v2:172.21.15.135:3300/0 conn(0x7f601815f9d0 msgr2=0x7f601815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 --2- 172.21.15.97:0/4035302530 >> v2:172.21.15.135:3300/0 conn(0x7f601815f9d0 0x7f601815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601806d1d0 con 0x7f60180ce0a0 2024-03-06T07:39:42.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.001+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f6010001bc0 con 0x7f60180ce0a0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 >> v1:172.21.15.97:6789/0 conn(0x7f60180ce0a0 legacy=0x7f60180ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc0039b0 0x7f5ffc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ffc0043a0 0x7f5ffc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ffc004fa0 0x7f5ffc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601806cef0 con 0x7f5ffc004fa0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc00bf00 con 0x7f5ffc0043a0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc00c0b0 con 0x7f5ffc0039b0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f601da00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ffc004fa0 0x7f5ffc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f600fe00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ffc0043a0 0x7f5ffc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f601d000640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc0039b0 0x7f5ffc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.002+0000 7f601da00640 1 -- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc0039b0 msgr2=0x7f5ffc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.003+0000 7f601da00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc0039b0 0x7f5ffc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.003+0000 7f601da00640 1 -- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ffc0043a0 msgr2=0x7f5ffc004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.003+0000 7f601da00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ffc0043a0 0x7f5ffc004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.003+0000 7f601da00640 1 -- 172.21.15.97:0/4035302530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc003520 con 0x7f5ffc004fa0 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.004+0000 7f601da00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ffc004fa0 0x7f5ffc00b9c0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f601000aa70 tx=0x7f5ffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a932a3eed289292f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.004+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010038070 con 0x7f5ffc004fa0 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.004+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60100072f0 con 0x7f5ffc004fa0 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.004+0000 7f6017e00640 1 -- 172.21.15.97:0/4035302530 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601000a690 con 0x7f5ffc004fa0 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601fa00640 1 -- 172.21.15.97:0/4035302530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ffc004fa0 msgr2=0x7f5ffc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601fa00640 1 --2- 172.21.15.97:0/4035302530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ffc004fa0 0x7f5ffc00b9c0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f601000aa70 tx=0x7f5ffc005830 comp rx=0 tx=0).stop 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601da00640 1 -- 172.21.15.97:0/4035302530 reap_dead start 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601fa00640 1 -- 172.21.15.97:0/4035302530 shutdown_connections 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601fa00640 1 -- 172.21.15.97:0/4035302530 >> 172.21.15.97:0/4035302530 conn(0x7f60180b1730 msgr2=0x7f60180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601fa00640 1 -- 172.21.15.97:0/4035302530 shutdown_connections 2024-03-06T07:39:42.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.005+0000 7f601fa00640 1 -- 172.21.15.97:0/4035302530 wait complete. 2024-03-06T07:39:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.007+0000 7f601fa00640 1 Processor -- start 2024-03-06T07:39:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.007+0000 7f601fa00640 1 -- start start 2024-03-06T07:39:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc004fa0 0x7f601815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601fa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 0x7f601815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601fa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f601815cd00 0x7f6018138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601fa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6018160730 con 0x7f601815c340 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601fa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601806d790 con 0x7f601815cd00 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601806cb50 con 0x7f5ffc004fa0 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 0x7f601815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 0x7f601815c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48086/0 (socket says 172.21.15.97:48086) 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 -- 172.21.15.97:0/3820436802 learned_addr learned my addr 172.21.15.97:0/3820436802 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601d000640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc004fa0 0x7f601815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f601da00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f601815cd00 0x7f6018138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 -- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc004fa0 msgr2=0x7f601815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc004fa0 0x7f601815dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 -- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f601815cd00 msgr2=0x7f6018138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f601815cd00 0x7f6018138710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.008+0000 7f600fe00640 1 -- 172.21.15.97:0/3820436802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601815d180 con 0x7f601815c340 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.009+0000 7f600fe00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 0x7f601815c7c0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f60000115e0 tx=0x7f6000011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.009+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000004220 con 0x7f601815c340 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.009+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60000043c0 con 0x7f601815c340 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.009+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6018137950 con 0x7f601815c340 2024-03-06T07:39:42.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.009+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600000f930 con 0x7f601815c340 2024-03-06T07:39:42.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.009+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6018137f10 con 0x7f601815c340 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.010+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60180b2040 con 0x7f601815c340 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.011+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f600000fad0 con 0x7f601815c340 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.013+0000 7f6015600640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5ff406b500 0x7f5ff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.013+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f600008a860 con 0x7f601815c340 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.013+0000 7f601d000640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5ff406b500 0x7f5ff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.014+0000 7f601d000640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5ff406b500 0x7f5ff406d9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6008006fd0 tx=0x7f6008008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:42.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.018+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60180b2040 con 0x7f601815c340 2024-03-06T07:39:42.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.346+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6018072890 con 0x7f601815c340 2024-03-06T07:39:42.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.347+0000 7f6015600640 1 -- 172.21.15.97:0/3820436802 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f6000019030 con 0x7f601815c340 2024-03-06T07:39:42.349 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:42.349 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:42.349 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:42.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5ff406b500 msgr2=0x7f5ff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5ff406b500 0x7f5ff406d9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6008006fd0 tx=0x7f6008008040 comp rx=0 tx=0).stop 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 msgr2=0x7f601815c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 0x7f601815c7c0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f60000115e0 tx=0x7f6000011aa0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 shutdown_connections 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5ff406b500 0x7f5ff406d9c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f601815cd00 0x7f6018138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f601815c340 0x7f601815c7c0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 --2- 172.21.15.97:0/3820436802 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5ffc004fa0 0x7f601815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 >> 172.21.15.97:0/3820436802 conn(0x7f60180b1730 msgr2=0x7f6018070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 shutdown_connections 2024-03-06T07:39:42.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:42.349+0000 7f601fa00640 1 -- 172.21.15.97:0/3820436802 wait complete. 2024-03-06T07:39:42.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:42.570 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:43.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:43.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.781+0000 7fe6a1c00640 1 Processor -- start 2024-03-06T07:39:43.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.781+0000 7fe6a1c00640 1 -- start start 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.782+0000 7fe6a1c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 0x7fe69c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.782+0000 7fe6a1c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe69c070b10 0x7fe69c070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.782+0000 7fe6a1c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c06cef0 con 0x7fe69c144c00 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.782+0000 7fe6a1c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69c160370 con 0x7fe69c070b10 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.782+0000 7fe6a1c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c06d1d0 con 0x7fe69c15f9d0 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.783+0000 7fe69aa00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe69c144c00 0x7fe69c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:48662/0 (socket says 172.21.15.97:48662) 2024-03-06T07:39:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.783+0000 7fe69aa00640 1 -- 172.21.15.97:0/1911194764 learned_addr learned my addr 172.21.15.97:0/1911194764 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:43.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.783+0000 7fe69a000640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 0x7fe69c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.783+0000 7fe69b400640 1 --2- 172.21.15.97:0/1911194764 >> v2:172.21.15.135:3300/0 conn(0x7fe69c070b10 0x7fe69c070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.783+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790350810 0 0) 0x7fe69c06cef0 con 0x7fe69c144c00 2024-03-06T07:39:43.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.783+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe680003520 con 0x7fe69c144c00 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790665238 0 0) 0x7fe680003520 con 0x7fe69c144c00 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 msgr2=0x7fe69c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 0x7fe69c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 >> v2:172.21.15.135:3300/0 conn(0x7fe69c070b10 msgr2=0x7fe69c070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 --2- 172.21.15.97:0/1911194764 >> v2:172.21.15.135:3300/0 conn(0x7fe69c070b10 0x7fe69c070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c06d4b0 con 0x7fe69c144c00 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe68c001670 con 0x7fe69c144c00 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 >> v1:172.21.15.97:6789/0 conn(0x7fe69c144c00 legacy=0x7fe69c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6800039b0 0x7fe680003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe680004800 0x7fe680004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 0x7fe6800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c06d1d0 con 0x7fe6800039b0 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68000b9c0 con 0x7fe680004800 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68000bbc0 con 0x7fe69c15f9d0 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe69b400640 1 --2- 172.21.15.97:0/1911194764 >> v2:172.21.15.135:3300/0 conn(0x7fe69c070b10 0x7fe69c070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe69aa00640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6800039b0 0x7fe680003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe69aa00640 1 -- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 msgr2=0x7fe6800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.784+0000 7fe69aa00640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c15f9d0 0x7fe6800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.785+0000 7fe69aa00640 1 -- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe680004800 msgr2=0x7fe680004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.785+0000 7fe69aa00640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe680004800 0x7fe680004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.785+0000 7fe69aa00640 1 -- 172.21.15.97:0/1911194764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680003520 con 0x7fe6800039b0 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe69aa00640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6800039b0 0x7fe680003dd0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fe68c001050 tx=0x7fe680005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cefbc93cacd0096 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c007ca0 con 0x7fe6800039b0 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c030070 con 0x7fe6800039b0 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe699600640 1 -- 172.21.15.97:0/1911194764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c005d10 con 0x7fe6800039b0 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe6a1c00640 1 -- 172.21.15.97:0/1911194764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6800039b0 msgr2=0x7fe680003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/1911194764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6800039b0 0x7fe680003dd0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fe68c001050 tx=0x7fe680005830 comp rx=0 tx=0).stop 2024-03-06T07:39:43.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe69b400640 1 -- 172.21.15.97:0/1911194764 reap_dead start 2024-03-06T07:39:43.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe6a1c00640 1 -- 172.21.15.97:0/1911194764 shutdown_connections 2024-03-06T07:39:43.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.786+0000 7fe6a1c00640 1 -- 172.21.15.97:0/1911194764 >> 172.21.15.97:0/1911194764 conn(0x7fe69c0b1730 msgr2=0x7fe69c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.787+0000 7fe6a1c00640 1 -- 172.21.15.97:0/1911194764 shutdown_connections 2024-03-06T07:39:43.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.787+0000 7fe6a1c00640 1 -- 172.21.15.97:0/1911194764 wait complete. 2024-03-06T07:39:43.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.788+0000 7fe6a1c00640 1 Processor -- start 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.788+0000 7fe6a1c00640 1 -- start start 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.789+0000 7fe6a1c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6800039b0 0x7fe69c1f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.789+0000 7fe6a1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 0x7fe69c13c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.789+0000 7fe6a1c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c142fb0 0x7fe69c13c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.789+0000 7fe6a1c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c06cb50 con 0x7fe69c142b30 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.789+0000 7fe6a1c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c160730 con 0x7fe6800039b0 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.789+0000 7fe6a1c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c06ccd0 con 0x7fe69c142fb0 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 0x7fe69c13c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 0x7fe69c13c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48110/0 (socket says 172.21.15.97:48110) 2024-03-06T07:39:43.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 -- 172.21.15.97:0/349748904 learned_addr learned my addr 172.21.15.97:0/349748904 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69aa00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6800039b0 0x7fe69c1f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 -- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c142fb0 msgr2=0x7fe69c13c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c142fb0 0x7fe69c13c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 -- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6800039b0 msgr2=0x7fe69c1f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6800039b0 0x7fe69c1f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 -- 172.21.15.97:0/349748904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c14f080 con 0x7fe69c142b30 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.790+0000 7fe69a000640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 0x7fe69c13c110 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fe68800e7d0 tx=0x7fe68800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.791+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688017070 con 0x7fe69c142b30 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.791+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68800c940 con 0x7fe69c142b30 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.791+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68800cd10 con 0x7fe69c142b30 2024-03-06T07:39:43.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.791+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c14f360 con 0x7fe69c142b30 2024-03-06T07:39:43.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.791+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c14f920 con 0x7fe69c142b30 2024-03-06T07:39:43.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.792+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c04f330 con 0x7fe69c142b30 2024-03-06T07:39:43.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.793+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe68801a070 con 0x7fe69c142b30 2024-03-06T07:39:43.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.794+0000 7fe692a00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe67c06b4d0 0x7fe67c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.794+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe68808a4c0 con 0x7fe69c142b30 2024-03-06T07:39:43.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.794+0000 7fe69aa00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe67c06b4d0 0x7fe67c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.795+0000 7fe69aa00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe67c06b4d0 0x7fe67c06d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe68c034000 tx=0x7fe68c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:43.799+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe688059310 con 0x7fe69c142b30 2024-03-06T07:39:44.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.030+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe69c072890 con 0x7fe69c142b30 2024-03-06T07:39:44.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.031+0000 7fe692a00640 1 -- 172.21.15.97:0/349748904 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe688058c10 con 0x7fe69c142b30 2024-03-06T07:39:44.034 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:44.034 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:44.034 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe67c06b4d0 msgr2=0x7fe67c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe67c06b4d0 0x7fe67c06d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe68c034000 tx=0x7fe68c007000 comp rx=0 tx=0).stop 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 msgr2=0x7fe69c13c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 0x7fe69c13c110 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fe68800e7d0 tx=0x7fe68800ec90 comp rx=0 tx=0).stop 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 shutdown_connections 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe67c06b4d0 0x7fe67c06d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe69c142fb0 0x7fe69c13c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe69c142b30 0x7fe69c13c110 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 --2- 172.21.15.97:0/349748904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe6800039b0 0x7fe69c1f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 >> 172.21.15.97:0/349748904 conn(0x7fe69c0b1730 msgr2=0x7fe69c0aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 shutdown_connections 2024-03-06T07:39:44.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:44.034+0000 7fe6a1c00640 1 -- 172.21.15.97:0/349748904 wait complete. 2024-03-06T07:39:44.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:44.257 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:45.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.474+0000 7fb065000640 1 Processor -- start 2024-03-06T07:39:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.474+0000 7fb065000640 1 -- start start 2024-03-06T07:39:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.475+0000 7fb065000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb060147310 0x7fb060072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb065000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0600b39c0 con 0x7fb06006daa0 2024-03-06T07:39:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb065000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0600ad9e0 con 0x7fb06006d390 2024-03-06T07:39:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb065000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06006df30 con 0x7fb060147310 2024-03-06T07:39:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb05d400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb06006daa0 0x7fb06006cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51532/0 (socket says 172.21.15.97:51532) 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb05d400640 1 -- 172.21.15.97:0/3576167796 learned_addr learned my addr 172.21.15.97:0/3576167796 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb05de00640 1 --2- 172.21.15.97:0/3576167796 >> v2:172.21.15.135:3300/0 conn(0x7fb060147310 0x7fb060072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188954889 0 0) 0x7fb04c001110 con 0x7fb06006daa0 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.476+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb044003520 con 0x7fb06006daa0 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156090561 0 0) 0x7fb044003520 con 0x7fb06006daa0 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 >> v2:172.21.15.135:3300/0 conn(0x7fb060147310 msgr2=0x7fb060072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 --2- 172.21.15.97:0/3576167796 >> v2:172.21.15.135:3300/0 conn(0x7fb060147310 0x7fb060072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 >> v1:172.21.15.135:6789/0 conn(0x7fb06006d390 legacy=0x7fb0600ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0600adcc0 con 0x7fb06006daa0 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb04c001840 con 0x7fb06006daa0 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 >> v1:172.21.15.97:6789/0 conn(0x7fb06006daa0 legacy=0x7fb06006cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0440039b0 0x7fb044003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb044004800 0x7fb044004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0440051c0 0x7fb04400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04400c030 con 0x7fb0440039b0 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04400c1e0 con 0x7fb044004800 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04400c3e0 con 0x7fb0440051c0 2024-03-06T07:39:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.477+0000 7fb05de00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0440039b0 0x7fb044003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.478+0000 7fb05de00640 1 -- 172.21.15.97:0/3576167796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0440051c0 msgr2=0x7fb04400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.478+0000 7fb05de00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0440051c0 0x7fb04400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.478+0000 7fb05de00640 1 -- 172.21.15.97:0/3576167796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb044004800 msgr2=0x7fb044004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.478+0000 7fb05de00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb044004800 0x7fb044004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.478+0000 7fb05de00640 1 -- 172.21.15.97:0/3576167796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044003520 con 0x7fb0440039b0 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.478+0000 7fb05e800640 1 -- 172.21.15.97:0/3576167796 reap_dead start 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.479+0000 7fb05de00640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0440039b0 0x7fb044003dd0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb050009ac0 tx=0x7fb044005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9aa34bda433f918 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.479+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050039070 con 0x7fb0440039b0 2024-03-06T07:39:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.479+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05002c6d0 con 0x7fb0440039b0 2024-03-06T07:39:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.479+0000 7fb05ca00640 1 -- 172.21.15.97:0/3576167796 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050009690 con 0x7fb0440039b0 2024-03-06T07:39:45.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 -- 172.21.15.97:0/3576167796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0440039b0 msgr2=0x7fb044003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0440039b0 0x7fb044003dd0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb050009ac0 tx=0x7fb044005830 comp rx=0 tx=0).stop 2024-03-06T07:39:45.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 -- 172.21.15.97:0/3576167796 shutdown_connections 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 --2- 172.21.15.97:0/3576167796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0440039b0 0x7fb044003dd0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 -- 172.21.15.97:0/3576167796 >> 172.21.15.97:0/3576167796 conn(0x7fb0600b1be0 msgr2=0x7fb0600b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 -- 172.21.15.97:0/3576167796 shutdown_connections 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.480+0000 7fb065000640 1 -- 172.21.15.97:0/3576167796 wait complete. 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.482+0000 7fb065000640 1 Processor -- start 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.482+0000 7fb065000640 1 -- start start 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb065000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0440039b0 0x7fb06015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb065000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 0x7fb0601465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb065000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0600b4710 0x7fb0600b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb065000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0601476f0 con 0x7fb060146140 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb065000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600ae280 con 0x7fb0440039b0 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb065000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb060072340 con 0x7fb0600b4710 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 0x7fb0601465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 0x7fb0601465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48082/0 (socket says 172.21.15.97:48082) 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 -- 172.21.15.97:0/3072715388 learned_addr learned my addr 172.21.15.97:0/3072715388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 -- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0600b4710 msgr2=0x7fb0600b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0600b4710 0x7fb0600b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 -- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0440039b0 msgr2=0x7fb06015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05de00640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0440039b0 0x7fb06015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0440039b0 0x7fb06015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.483+0000 7fb05d400640 1 -- 172.21.15.97:0/3072715388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06013d740 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.484+0000 7fb05d400640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 0x7fb0601465c0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0012b0 tx=0x7fb04c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.484+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c018070 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.484+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c00ce50 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.484+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c0047e0 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.484+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06013dec0 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.484+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06013e480 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.485+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06004f330 con 0x7fb060146140 2024-03-06T07:39:45.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.486+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb04c004980 con 0x7fb060146140 2024-03-06T07:39:45.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.487+0000 7fb056000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb04006b4d0 0x7fb04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.487+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb04c089680 con 0x7fb060146140 2024-03-06T07:39:45.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.487+0000 7fb05de00640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb04006b4d0 0x7fb04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.488+0000 7fb05de00640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb04006b4d0 0x7fb04006d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb050036000 tx=0x7fb050030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:45.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.492+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0600b39c0 con 0x7fb060146140 2024-03-06T07:39:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.723+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0600b3000 con 0x7fb060146140 2024-03-06T07:39:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.724+0000 7fb056000640 1 -- 172.21.15.97:0/3072715388 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fb04c058440 con 0x7fb060146140 2024-03-06T07:39:45.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:45.726 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:45.727 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb04006b4d0 msgr2=0x7fb04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb04006b4d0 0x7fb04006d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb050036000 tx=0x7fb050030000 comp rx=0 tx=0).stop 2024-03-06T07:39:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 msgr2=0x7fb0601465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 0x7fb0601465c0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0012b0 tx=0x7fb04c00cc00 comp rx=0 tx=0).stop 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 shutdown_connections 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb04006b4d0 0x7fb04006d990 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb0600b4710 0x7fb0600b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb060146140 0x7fb0601465c0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 --2- 172.21.15.97:0/3072715388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0440039b0 0x7fb06015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 >> 172.21.15.97:0/3072715388 conn(0x7fb0600b1be0 msgr2=0x7fb06015eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 shutdown_connections 2024-03-06T07:39:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:45.726+0000 7fb065000640 1 -- 172.21.15.97:0/3072715388 wait complete. 2024-03-06T07:39:45.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:45.956 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:46.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.173+0000 7f0b26c00640 1 Processor -- start 2024-03-06T07:39:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.173+0000 7f0b26c00640 1 -- start start 2024-03-06T07:39:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.174+0000 7f0b26c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0b20161580 0x7f0b200d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.174+0000 7f0b26c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0b200d2a70 0x7f0b200b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.174+0000 7f0b26c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b20161960 con 0x7f0b200d1e90 2024-03-06T07:39:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.174+0000 7f0b26c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b200cf6f0 con 0x7f0b200d2a70 2024-03-06T07:39:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.174+0000 7f0b26c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b200cf9d0 con 0x7f0b20161580 2024-03-06T07:39:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.175+0000 7f0b24e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0b200d1e90 0x7f0b200d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51548/0 (socket says 172.21.15.97:51548) 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.175+0000 7f0b24e00640 1 -- 172.21.15.97:0/2324972016 learned_addr learned my addr 172.21.15.97:0/2324972016 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.175+0000 7f0b26200640 1 --2- 172.21.15.97:0/2324972016 >> v2:172.21.15.97:3300/0 conn(0x7f0b200d2a70 0x7f0b200b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.175+0000 7f0b25800640 1 --2- 172.21.15.97:0/2324972016 >> v2:172.21.15.135:3300/0 conn(0x7f0b20161580 0x7f0b200d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.175+0000 7f0b1fe00640 1 -- 172.21.15.97:0/2324972016 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220116878 0 0) 0x7f0b20161960 con 0x7f0b200d1e90 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.175+0000 7f0b1fe00640 1 -- 172.21.15.97:0/2324972016 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08003520 con 0x7f0b200d1e90 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.176+0000 7f0b26200640 1 -- 172.21.15.97:0/2324972016 >> v2:172.21.15.135:3300/0 conn(0x7f0b20161580 msgr2=0x7f0b200d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.176+0000 7f0b26200640 1 --2- 172.21.15.97:0/2324972016 >> v2:172.21.15.135:3300/0 conn(0x7f0b20161580 0x7f0b200d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.176+0000 7f0b26200640 1 -- 172.21.15.97:0/2324972016 >> v1:172.21.15.97:6789/0 conn(0x7f0b200d1e90 legacy=0x7f0b200d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.176+0000 7f0b26200640 1 -- 172.21.15.97:0/2324972016 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200cfcb0 con 0x7f0b200d2a70 2024-03-06T07:39:47.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.176+0000 7f0b25800640 1 --2- 172.21.15.97:0/2324972016 >> v2:172.21.15.135:3300/0 conn(0x7f0b20161580 0x7f0b200d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.178+0000 7f0b26200640 1 --2- 172.21.15.97:0/2324972016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b200b3ec0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f0b20073740 tx=0x7f0b1802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dc6f6bccbff05c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.178+0000 7f0b1fe00640 1 -- 172.21.15.97:0/2324972016 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18039070 con 0x7f0b200d2a70 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.178+0000 7f0b1fe00640 1 -- 172.21.15.97:0/2324972016 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1802cc80 con 0x7f0b200d2a70 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.178+0000 7f0b1fe00640 1 -- 172.21.15.97:0/2324972016 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18034640 con 0x7f0b200d2a70 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 -- 172.21.15.97:0/2324972016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 msgr2=0x7f0b200b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 --2- 172.21.15.97:0/2324972016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b200b3ec0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f0b20073740 tx=0x7f0b1802bed0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 -- 172.21.15.97:0/2324972016 shutdown_connections 2024-03-06T07:39:47.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 --2- 172.21.15.97:0/2324972016 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b200b3ec0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 --2- 172.21.15.97:0/2324972016 >> v2:172.21.15.135:3300/0 conn(0x7f0b20161580 0x7f0b200d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 -- 172.21.15.97:0/2324972016 >> 172.21.15.97:0/2324972016 conn(0x7f0b200d3d80 msgr2=0x7f0b200d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:47.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 -- 172.21.15.97:0/2324972016 shutdown_connections 2024-03-06T07:39:47.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.179+0000 7f0b26c00640 1 -- 172.21.15.97:0/2324972016 wait complete. 2024-03-06T07:39:47.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.181+0000 7f0b26c00640 1 Processor -- start 2024-03-06T07:39:47.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.181+0000 7f0b26c00640 1 -- start start 2024-03-06T07:39:47.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b26c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b200d1e90 0x7f0b201f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:47.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b26c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b201f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:47.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b26c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0b20161580 0x7f0b200cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:47.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b26c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b200cf3a0 con 0x7f0b200d2a70 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b26c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b200b39a0 con 0x7f0b200d1e90 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b26c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b200b4400 con 0x7f0b20161580 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b24e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b201f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b24e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b201f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48102/0 (socket says 172.21.15.97:48102) 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b24e00640 1 -- 172.21.15.97:0/3721385815 learned_addr learned my addr 172.21.15.97:0/3721385815 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.182+0000 7f0b25800640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b200d1e90 0x7f0b201f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b26200640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0b20161580 0x7f0b200cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b24e00640 1 -- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0b20161580 msgr2=0x7f0b200cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b24e00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0b20161580 0x7f0b200cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b24e00640 1 -- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b200d1e90 msgr2=0x7f0b201f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b24e00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b200d1e90 0x7f0b201f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b24e00640 1 -- 172.21.15.97:0/3721385815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b201f7180 con 0x7f0b200d2a70 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b26200640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0b20161580 0x7f0b200cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b24e00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b201f6c40 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c430 tx=0x7f0b0c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c017070 con 0x7f0b200d2a70 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c005ce0 con 0x7f0b200d2a70 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b200b7bf0 con 0x7f0b200d2a70 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0072f0 con 0x7f0b200d2a70 2024-03-06T07:39:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.183+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b200b81b0 con 0x7f0b200d2a70 2024-03-06T07:39:47.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.185+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2004f330 con 0x7f0b200d2a70 2024-03-06T07:39:47.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.186+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b0c01a070 con 0x7f0b200d2a70 2024-03-06T07:39:47.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.186+0000 7f0b1d600640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b0006b4d0 0x7f0b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:47.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.187+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0b0c089670 con 0x7f0b200d2a70 2024-03-06T07:39:47.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.187+0000 7f0b25800640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b0006b4d0 0x7f0b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:47.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.188+0000 7f0b25800640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b0006b4d0 0x7f0b0006d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f0b14006fd0 tx=0x7f0b14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:47.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.191+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b0c058560 con 0x7f0b200d2a70 2024-03-06T07:39:47.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.434+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b200d3b60 con 0x7f0b200d2a70 2024-03-06T07:39:47.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.435+0000 7f0b1d600640 1 -- 172.21.15.97:0/3721385815 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f0b0c057e60 con 0x7f0b200d2a70 2024-03-06T07:39:47.437 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:47.438 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:47.438 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:47.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.438+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b0006b4d0 msgr2=0x7f0b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.438+0000 7f0b26c00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b0006b4d0 0x7f0b0006d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f0b14006fd0 tx=0x7f0b14008040 comp rx=0 tx=0).stop 2024-03-06T07:39:47.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.438+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 msgr2=0x7f0b201f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.438+0000 7f0b26c00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b201f6c40 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c430 tx=0x7f0b0c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 shutdown_connections 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0b0006b4d0 0x7f0b0006d990 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0b20161580 0x7f0b200cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b200d2a70 0x7f0b201f6c40 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 --2- 172.21.15.97:0/3721385815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b200d1e90 0x7f0b201f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 >> 172.21.15.97:0/3721385815 conn(0x7f0b200d3d80 msgr2=0x7f0b200d08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 shutdown_connections 2024-03-06T07:39:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:47.439+0000 7f0b26c00640 1 -- 172.21.15.97:0/3721385815 wait complete. 2024-03-06T07:39:47.660 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:47.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:48.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:48.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.873+0000 7fa0ea000640 1 Processor -- start 2024-03-06T07:39:48.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.873+0000 7fa0ea000640 1 -- start start 2024-03-06T07:39:48.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0ea000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0e40aedc0 0x7fa0e40cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0ea000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa0e40cc800 0x7fa0e415f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0ea000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e415c240 con 0x7fa0e41561f0 2024-03-06T07:39:48.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0ea000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e415ce30 con 0x7fa0e40aedc0 2024-03-06T07:39:48.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0ea000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e415c500 con 0x7fa0e40cc800 2024-03-06T07:39:48.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0e3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0e40aedc0 0x7fa0e40cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0e3e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa0e40aedc0 0x7fa0e40cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48108/0 (socket says 172.21.15.97:48108) 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0e3e00640 1 -- 172.21.15.97:0/2995409376 learned_addr learned my addr 172.21.15.97:0/2995409376 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0e8c00640 1 --1- 172.21.15.97:0/2995409376 >> v1:172.21.15.97:6789/0 conn(0x7fa0e41561f0 0x7fa0e40ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51558/0 (socket says 172.21.15.97:51558) 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.875+0000 7fa0e9600640 1 --2- 172.21.15.97:0/2995409376 >> v2:172.21.15.135:3300/0 conn(0x7fa0e40cc800 0x7fa0e415f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.876+0000 7fa0e3e00640 1 -- 172.21.15.97:0/2995409376 >> v2:172.21.15.135:3300/0 conn(0x7fa0e40cc800 msgr2=0x7fa0e415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.876+0000 7fa0e3e00640 1 --2- 172.21.15.97:0/2995409376 >> v2:172.21.15.135:3300/0 conn(0x7fa0e40cc800 0x7fa0e415f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.876+0000 7fa0e3e00640 1 -- 172.21.15.97:0/2995409376 >> v1:172.21.15.97:6789/0 conn(0x7fa0e41561f0 legacy=0x7fa0e40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.876+0000 7fa0e3e00640 1 -- 172.21.15.97:0/2995409376 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e415c7e0 con 0x7fa0e40aedc0 2024-03-06T07:39:48.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.876+0000 7fa0e9600640 1 --2- 172.21.15.97:0/2995409376 >> v2:172.21.15.135:3300/0 conn(0x7fa0e40cc800 0x7fa0e415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:48.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.877+0000 7fa0e3e00640 1 --2- 172.21.15.97:0/2995409376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e40aedc0 0x7fa0e40cc2c0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40cc1f0 tx=0x7fa0d002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a333e24ebaa251 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:48.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.878+0000 7fa0e3400640 1 -- 172.21.15.97:0/2995409376 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0034070 con 0x7fa0e40aedc0 2024-03-06T07:39:48.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.878+0000 7fa0e3400640 1 -- 172.21.15.97:0/2995409376 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d0030880 con 0x7fa0e40aedc0 2024-03-06T07:39:48.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.878+0000 7fa0e3400640 1 -- 172.21.15.97:0/2995409376 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0030c50 con 0x7fa0e40aedc0 2024-03-06T07:39:48.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.878+0000 7fa0ea000640 1 -- 172.21.15.97:0/2995409376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e40aedc0 msgr2=0x7fa0e40cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.878+0000 7fa0ea000640 1 --2- 172.21.15.97:0/2995409376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e40aedc0 0x7fa0e40cc2c0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40cc1f0 tx=0x7fa0d002b9c0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.879+0000 7fa0ea000640 1 -- 172.21.15.97:0/2995409376 shutdown_connections 2024-03-06T07:39:48.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.879+0000 7fa0ea000640 1 --2- 172.21.15.97:0/2995409376 >> v2:172.21.15.135:3300/0 conn(0x7fa0e40cc800 0x7fa0e415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.879+0000 7fa0ea000640 1 --2- 172.21.15.97:0/2995409376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e40aedc0 0x7fa0e40cc2c0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.879+0000 7fa0ea000640 1 -- 172.21.15.97:0/2995409376 >> 172.21.15.97:0/2995409376 conn(0x7fa0e406afc0 msgr2=0x7fa0e406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:48.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.879+0000 7fa0ea000640 1 -- 172.21.15.97:0/2995409376 shutdown_connections 2024-03-06T07:39:48.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.879+0000 7fa0ea000640 1 -- 172.21.15.97:0/2995409376 wait complete. 2024-03-06T07:39:48.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.880+0000 7fa0ea000640 1 Processor -- start 2024-03-06T07:39:48.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.880+0000 7fa0ea000640 1 -- start start 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.881+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0e40aedc0 0x7fa0e4150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.881+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0e40cc800 0x7fa0e414cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.881+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 0x7fa0e414de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.881+0000 7fa0ea000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e41565d0 con 0x7fa0e41561f0 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.881+0000 7fa0ea000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40ccc10 con 0x7fa0e40cc800 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.881+0000 7fa0ea000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4156750 con 0x7fa0e40aedc0 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 0x7fa0e414de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 0x7fa0e414de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48124/0 (socket says 172.21.15.97:48124) 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 -- 172.21.15.97:0/220010147 learned_addr learned my addr 172.21.15.97:0/220010147 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e8c00640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0e40aedc0 0x7fa0e4150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e3e00640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0e40cc800 0x7fa0e414cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 -- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0e40aedc0 msgr2=0x7fa0e4150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0e40aedc0 0x7fa0e4150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 -- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0e40cc800 msgr2=0x7fa0e414cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0e40cc800 0x7fa0e414cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e9600640 1 -- 172.21.15.97:0/220010147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e414e6f0 con 0x7fa0e41561f0 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.882+0000 7fa0e8c00640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0e40aedc0 0x7fa0e4150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.883+0000 7fa0e9600640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 0x7fa0e414de10 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00cef0 tx=0x7fa0dc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.883+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc01a070 con 0x7fa0e41561f0 2024-03-06T07:39:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.883+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc00d860 con 0x7fa0e41561f0 2024-03-06T07:39:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.883+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00dc50 con 0x7fa0e41561f0 2024-03-06T07:39:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.883+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e414e9d0 con 0x7fa0e41561f0 2024-03-06T07:39:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.883+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e414ef90 con 0x7fa0e41561f0 2024-03-06T07:39:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.885+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e404f330 con 0x7fa0e41561f0 2024-03-06T07:39:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.885+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0dc01d070 con 0x7fa0e41561f0 2024-03-06T07:39:48.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.886+0000 7fa0e0c00640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa0c406b4d0 0x7fa0c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.886+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa0dc08ab00 con 0x7fa0e41561f0 2024-03-06T07:39:48.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.886+0000 7fa0e8c00640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa0c406b4d0 0x7fa0c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.887+0000 7fa0e8c00640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa0c406b4d0 0x7fa0c406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8007fd0 tx=0x7fa0d8007d80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:48.890+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0dc059950 con 0x7fa0e41561f0 2024-03-06T07:39:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.129+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0e406ce70 con 0x7fa0e41561f0 2024-03-06T07:39:49.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.129+0000 7fa0e0c00640 1 -- 172.21.15.97:0/220010147 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fa0dc08ae20 con 0x7fa0e41561f0 2024-03-06T07:39:49.132 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:49.132 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa0c406b4d0 msgr2=0x7fa0c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa0c406b4d0 0x7fa0c406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8007fd0 tx=0x7fa0d8007d80 comp rx=0 tx=0).stop 2024-03-06T07:39:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 msgr2=0x7fa0e414de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 0x7fa0e414de10 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00cef0 tx=0x7fa0dc00a710 comp rx=0 tx=0).stop 2024-03-06T07:39:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 shutdown_connections 2024-03-06T07:39:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa0c406b4d0 0x7fa0c406d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0e41561f0 0x7fa0e414de10 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0e40cc800 0x7fa0e414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 --2- 172.21.15.97:0/220010147 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa0e40aedc0 0x7fa0e4150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 >> 172.21.15.97:0/220010147 conn(0x7fa0e406afc0 msgr2=0x7fa0e40b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:49.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 shutdown_connections 2024-03-06T07:39:49.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:49.133+0000 7fa0ea000640 1 -- 172.21.15.97:0/220010147 wait complete. 2024-03-06T07:39:49.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:49.354 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:50.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:50.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.567+0000 7fe7c4e00640 1 Processor -- start 2024-03-06T07:39:50.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.567+0000 7fe7c4e00640 1 -- start start 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7c4e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe7c0157fe0 0x7fe7c00b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7c4e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 0x7fe7c006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7c4e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c00b3ef0 con 0x7fe7c006eff0 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7c4e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c006eb60 con 0x7fe7c0157fe0 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7c4e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c00b41d0 con 0x7fe7c015e790 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe7c0157fe0 0x7fe7c00b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe7c0157fe0 0x7fe7c00b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48138/0 (socket says 172.21.15.97:48138) 2024-03-06T07:39:50.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 -- 172.21.15.97:0/573279313 learned_addr learned my addr 172.21.15.97:0/573279313 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:50.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 -- 172.21.15.97:0/573279313 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 msgr2=0x7fe7c006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:50.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 --2- 172.21.15.97:0/573279313 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 0x7fe7c006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 -- 172.21.15.97:0/573279313 >> v1:172.21.15.135:6789/0 conn(0x7fe7c006eff0 legacy=0x7fe7c015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:50.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.570+0000 7fe7bde00640 1 -- 172.21.15.97:0/573279313 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c00b44b0 con 0x7fe7c0157fe0 2024-03-06T07:39:50.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.571+0000 7fe7bde00640 1 --2- 172.21.15.97:0/573279313 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c0157fe0 0x7fe7c00b33b0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000af60 tx=0x7fe7b002cff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853a7d87f1273584 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.572+0000 7fe7bd400640 1 -- 172.21.15.97:0/573279313 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b002d990 con 0x7fe7c0157fe0 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.572+0000 7fe7bd400640 1 -- 172.21.15.97:0/573279313 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b002db30 con 0x7fe7c0157fe0 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.572+0000 7fe7bd400640 1 -- 172.21.15.97:0/573279313 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0008500 con 0x7fe7c0157fe0 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.572+0000 7fe7c4e00640 1 -- 172.21.15.97:0/573279313 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c0157fe0 msgr2=0x7fe7c00b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.572+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/573279313 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c0157fe0 0x7fe7c00b33b0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000af60 tx=0x7fe7b002cff0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.573+0000 7fe7c4e00640 1 -- 172.21.15.97:0/573279313 shutdown_connections 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.573+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/573279313 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 0x7fe7c006e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.573+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/573279313 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c0157fe0 0x7fe7c00b33b0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.573+0000 7fe7c4e00640 1 -- 172.21.15.97:0/573279313 >> 172.21.15.97:0/573279313 conn(0x7fe7c00d0340 msgr2=0x7fe7c00d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.573+0000 7fe7c4e00640 1 -- 172.21.15.97:0/573279313 shutdown_connections 2024-03-06T07:39:50.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.573+0000 7fe7c4e00640 1 -- 172.21.15.97:0/573279313 wait complete. 2024-03-06T07:39:50.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.574+0000 7fe7c4e00640 1 Processor -- start 2024-03-06T07:39:50.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.575+0000 7fe7c4e00640 1 -- start start 2024-03-06T07:39:50.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.575+0000 7fe7c4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 0x7fe7c014bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.575+0000 7fe7c4e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7c0157fe0 0x7fe7c014c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.575+0000 7fe7c4e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 0x7fe7c00b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7c4e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c0066d00 con 0x7fe7c006eff0 2024-03-06T07:39:50.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7c4e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c01583c0 con 0x7fe7c0157fe0 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7c4e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c0066e80 con 0x7fe7c015e790 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 0x7fe7c014bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 0x7fe7c014bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48152/0 (socket says 172.21.15.97:48152) 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 -- 172.21.15.97:0/2412022730 learned_addr learned my addr 172.21.15.97:0/2412022730 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 -- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 msgr2=0x7fe7c00b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 0x7fe7c00b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 -- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7c0157fe0 msgr2=0x7fe7c014c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7b5400640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7c0157fe0 0x7fe7c014c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7c0157fe0 0x7fe7c014c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.576+0000 7fe7bde00640 1 -- 172.21.15.97:0/2412022730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c014e940 con 0x7fe7c006eff0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.577+0000 7fe7bde00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 0x7fe7c014bf40 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000af60 tx=0x7fe7b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.577+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0044020 con 0x7fe7c006eff0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.577+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b0040690 con 0x7fe7c006eff0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.577+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c014ebb0 con 0x7fe7c006eff0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.577+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0040a80 con 0x7fe7c006eff0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.577+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c01589d0 con 0x7fe7c006eff0 2024-03-06T07:39:50.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.578+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c0074ae0 con 0x7fe7c006eff0 2024-03-06T07:39:50.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.579+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7b0047020 con 0x7fe7c006eff0 2024-03-06T07:39:50.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.580+0000 7fe7b6a00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.580+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe7b00ad480 con 0x7fe7c006eff0 2024-03-06T07:39:50.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.580+0000 7fe7b5400640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.581+0000 7fe7b5400640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79c06b4d0 0x7fe79c06d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe7c014f7f0 tx=0x7fe7a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.585+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7b004e030 con 0x7fe7c006eff0 2024-03-06T07:39:50.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.816+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7c015da50 con 0x7fe7c006eff0 2024-03-06T07:39:50.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.817+0000 7fe7b6a00640 1 -- 172.21.15.97:0/2412022730 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe7b007c400 con 0x7fe7c006eff0 2024-03-06T07:39:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:50.819 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":116,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:38:36.706910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:50.820 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 116 2024-03-06T07:39:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79c06b4d0 msgr2=0x7fe79c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79c06b4d0 0x7fe79c06d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe7c014f7f0 tx=0x7fe7a8009040 comp rx=0 tx=0).stop 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 msgr2=0x7fe7c014bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 0x7fe7c014bf40 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000af60 tx=0x7fe7b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 shutdown_connections 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe7c015e790 0x7fe7c00b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe7c0157fe0 0x7fe7c014c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 --2- 172.21.15.97:0/2412022730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c006eff0 0x7fe7c014bf40 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 >> 172.21.15.97:0/2412022730 conn(0x7fe7c00d0340 msgr2=0x7fe7c00d0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 shutdown_connections 2024-03-06T07:39:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:50.820+0000 7fe7c4e00640 1 -- 172.21.15.97:0/2412022730 wait complete. 2024-03-06T07:39:51.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:38:36.706910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:51.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:52.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.251+0000 7fc56a200640 1 Processor -- start 2024-03-06T07:39:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.251+0000 7fc56a200640 1 -- start start 2024-03-06T07:39:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc56a200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc56406d390 0x7fc564072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc56a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc5640ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc56a200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640b39c0 con 0x7fc564147310 2024-03-06T07:39:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc56a200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56406df30 con 0x7fc56406d390 2024-03-06T07:39:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc56a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640ad9e0 con 0x7fc56406daa0 2024-03-06T07:39:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc562a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc56406d390 0x7fc564072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc563e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc5640ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc562a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc56406d390 0x7fc564072340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:60546/0 (socket says 172.21.15.97:60546) 2024-03-06T07:39:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.253+0000 7fc562a00640 1 -- 172.21.15.97:0/1249987466 learned_addr learned my addr 172.21.15.97:0/1249987466 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.254+0000 7fc562a00640 1 -- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 msgr2=0x7fc5640ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.254+0000 7fc562a00640 1 --2- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc5640ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.254+0000 7fc562a00640 1 -- 172.21.15.97:0/1249987466 >> v1:172.21.15.135:6789/0 conn(0x7fc564147310 legacy=0x7fc56406cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.254+0000 7fc562a00640 1 -- 172.21.15.97:0/1249987466 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640adcc0 con 0x7fc56406d390 2024-03-06T07:39:52.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.254+0000 7fc563e00640 1 --2- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc5640ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:52.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.255+0000 7fc562a00640 1 --2- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc564072340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009870 tx=0x7fc54c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd89bd23e59d7bba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:52.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.256+0000 7fc562000640 1 -- 172.21.15.97:0/1249987466 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c034070 con 0x7fc56406d390 2024-03-06T07:39:52.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.256+0000 7fc562000640 1 -- 172.21.15.97:0/1249987466 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c030840 con 0x7fc56406d390 2024-03-06T07:39:52.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.256+0000 7fc562000640 1 -- 172.21.15.97:0/1249987466 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c030c10 con 0x7fc56406d390 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 -- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 msgr2=0x7fc564072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 --2- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc564072340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009870 tx=0x7fc54c02b980 comp rx=0 tx=0).stop 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 -- 172.21.15.97:0/1249987466 shutdown_connections 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 --2- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc5640ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 --2- 172.21.15.97:0/1249987466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc564072340 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 -- 172.21.15.97:0/1249987466 >> 172.21.15.97:0/1249987466 conn(0x7fc5640b1be0 msgr2=0x7fc5640b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:52.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 -- 172.21.15.97:0/1249987466 shutdown_connections 2024-03-06T07:39:52.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.257+0000 7fc56a200640 1 -- 172.21.15.97:0/1249987466 wait complete. 2024-03-06T07:39:52.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.260+0000 7fc56a200640 1 Processor -- start 2024-03-06T07:39:52.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.260+0000 7fc56a200640 1 -- start start 2024-03-06T07:39:52.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc56a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc56415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc56a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc564160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc56a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 0x7fc5641e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc56a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5641476f0 con 0x7fc564147310 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc56a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640ae280 con 0x7fc56406d390 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc56a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640aece0 con 0x7fc56406daa0 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc563e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 0x7fc5641e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc563e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 0x7fc5641e9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48160/0 (socket says 172.21.15.97:48160) 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc563e00640 1 -- 172.21.15.97:0/1021069414 learned_addr learned my addr 172.21.15.97:0/1021069414 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc562a00640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc564160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.262+0000 7fc563400640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc56415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc563e00640 1 -- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 msgr2=0x7fc564160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc563e00640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc564160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc563e00640 1 -- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 msgr2=0x7fc56415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc563e00640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc56415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc563e00640 1 -- 172.21.15.97:0/1021069414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56415ed80 con 0x7fc564147310 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc563e00640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 0x7fc5641e9030 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00efc0 tx=0x7fc55c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c018030 con 0x7fc564147310 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c00cde0 con 0x7fc564147310 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c003bb0 con 0x7fc564147310 2024-03-06T07:39:52.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56415f060 con 0x7fc564147310 2024-03-06T07:39:52.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.263+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56415fb40 con 0x7fc564147310 2024-03-06T07:39:52.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.265+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56404f330 con 0x7fc564147310 2024-03-06T07:39:52.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.266+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc55c01b070 con 0x7fc564147310 2024-03-06T07:39:52.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.267+0000 7fc55b400640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc54006b500 0x7fc54006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.267+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc55c08ac80 con 0x7fc564147310 2024-03-06T07:39:52.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.267+0000 7fc563400640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc54006b500 0x7fc54006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.268+0000 7fc563400640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc54006b500 0x7fc54006d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc554007fd0 tx=0x7fc554009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:52.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.271+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc55c010040 con 0x7fc564147310 2024-03-06T07:39:52.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.499+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5640b3000 con 0x7fc564147310 2024-03-06T07:39:52.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.500+0000 7fc55b400640 1 -- 172.21.15.97:0/1021069414 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fc55c012340 con 0x7fc564147310 2024-03-06T07:39:52.502 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:52.502 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:52.503 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc54006b500 msgr2=0x7fc54006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc54006b500 0x7fc54006d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc554007fd0 tx=0x7fc554009040 comp rx=0 tx=0).stop 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 msgr2=0x7fc5641e9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 0x7fc5641e9030 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00efc0 tx=0x7fc55c009f90 comp rx=0 tx=0).stop 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 shutdown_connections 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc54006b500 0x7fc54006d9c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc564147310 0x7fc5641e9030 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc56406daa0 0x7fc564160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 --2- 172.21.15.97:0/1021069414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc56406d390 0x7fc56415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 >> 172.21.15.97:0/1021069414 conn(0x7fc5640b1be0 msgr2=0x7fc5640b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:52.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 shutdown_connections 2024-03-06T07:39:52.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:52.503+0000 7fc56a200640 1 -- 172.21.15.97:0/1021069414 wait complete. 2024-03-06T07:39:52.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:52.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:53.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:53.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.934+0000 7f4d57000640 1 Processor -- start 2024-03-06T07:39:53.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.934+0000 7f4d57000640 1 -- start start 2024-03-06T07:39:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d57000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4d50144c00 0x7f4d5006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d57000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d5015f9d0 0x7f4d5015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d57000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5006ee70 con 0x7f4d500ce0a0 2024-03-06T07:39:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d57000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d50160370 con 0x7f4d50144c00 2024-03-06T07:39:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d57000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5006cef0 con 0x7f4d5015f9d0 2024-03-06T07:39:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d4fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4d50144c00 0x7f4d5006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d55000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4d500ce0a0 0x7f4d500ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42054/0 (socket says 172.21.15.97:42054) 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d4fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4d50144c00 0x7f4d5006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46066/0 (socket says 172.21.15.97:46066) 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d55000640 1 -- 172.21.15.97:0/1987541083 learned_addr learned my addr 172.21.15.97:0/1987541083 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.936+0000 7f4d4f400640 1 --2- 172.21.15.97:0/1987541083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d5015f9d0 0x7f4d5015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.937+0000 7f4d4fe00640 1 -- 172.21.15.97:0/1987541083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d5015f9d0 msgr2=0x7f4d5015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.937+0000 7f4d4fe00640 1 --2- 172.21.15.97:0/1987541083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d5015f9d0 0x7f4d5015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.937+0000 7f4d4fe00640 1 -- 172.21.15.97:0/1987541083 >> v1:172.21.15.97:6789/0 conn(0x7f4d500ce0a0 legacy=0x7f4d500ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.937+0000 7f4d4fe00640 1 -- 172.21.15.97:0/1987541083 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5006d1d0 con 0x7f4d50144c00 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.938+0000 7f4d4fe00640 1 --2- 172.21.15.97:0/1987541083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d50144c00 0x7f4d5006e930 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f4d400097a0 tx=0x7f4d4002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5704c06695d570fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.938+0000 7f4d4ea00640 1 -- 172.21.15.97:0/1987541083 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40035020 con 0x7f4d50144c00 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.938+0000 7f4d4ea00640 1 -- 172.21.15.97:0/1987541083 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d40031810 con 0x7f4d50144c00 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.938+0000 7f4d4ea00640 1 -- 172.21.15.97:0/1987541083 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40031be0 con 0x7f4d50144c00 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.938+0000 7f4d57000640 1 -- 172.21.15.97:0/1987541083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d50144c00 msgr2=0x7f4d5006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.938+0000 7f4d57000640 1 --2- 172.21.15.97:0/1987541083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d50144c00 0x7f4d5006e930 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f4d400097a0 tx=0x7f4d4002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.939+0000 7f4d57000640 1 -- 172.21.15.97:0/1987541083 shutdown_connections 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.939+0000 7f4d57000640 1 --2- 172.21.15.97:0/1987541083 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d5015f9d0 0x7f4d5015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.939+0000 7f4d57000640 1 --2- 172.21.15.97:0/1987541083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d50144c00 0x7f4d5006e930 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.939+0000 7f4d57000640 1 -- 172.21.15.97:0/1987541083 >> 172.21.15.97:0/1987541083 conn(0x7f4d500b1730 msgr2=0x7f4d500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.939+0000 7f4d57000640 1 -- 172.21.15.97:0/1987541083 shutdown_connections 2024-03-06T07:39:53.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.939+0000 7f4d57000640 1 -- 172.21.15.97:0/1987541083 wait complete. 2024-03-06T07:39:53.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.940+0000 7f4d57000640 1 Processor -- start 2024-03-06T07:39:53.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.941+0000 7f4d57000640 1 -- start start 2024-03-06T07:39:53.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.941+0000 7f4d57000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d500ce0a0 0x7f4d5015aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d57000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d50144c00 0x7f4d500b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d57000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 0x7f4d500b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d57000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d50160730 con 0x7f4d5015f9d0 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d57000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5006d790 con 0x7f4d50144c00 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d57000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5006cb50 con 0x7f4d500ce0a0 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 0x7f4d500b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 0x7f4d500b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46076/0 (socket says 172.21.15.97:46076) 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 -- 172.21.15.97:0/2376927297 learned_addr learned my addr 172.21.15.97:0/2376927297 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d4fe00640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d500ce0a0 0x7f4d5015aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d4f400640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d50144c00 0x7f4d500b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 -- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d500ce0a0 msgr2=0x7f4d5015aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d500ce0a0 0x7f4d5015aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 -- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d50144c00 msgr2=0x7f4d500b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:53.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d50144c00 0x7f4d500b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:53.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d55000640 1 -- 172.21.15.97:0/2376927297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d501e8330 con 0x7f4d5015f9d0 2024-03-06T07:39:53.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.942+0000 7f4d4f400640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d50144c00 0x7f4d500b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:53.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.943+0000 7f4d55000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 0x7f4d500b38d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800ca80 tx=0x7f4d4800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:53.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.943+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48017070 con 0x7f4d5015f9d0 2024-03-06T07:39:53.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.943+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4800ac30 con 0x7f4d5015f9d0 2024-03-06T07:39:53.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.943+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d501e8610 con 0x7f4d5015f9d0 2024-03-06T07:39:53.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.943+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48012400 con 0x7f4d5015f9d0 2024-03-06T07:39:53.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.943+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d501e8bd0 con 0x7f4d5015f9d0 2024-03-06T07:39:53.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.945+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5004f330 con 0x7f4d5015f9d0 2024-03-06T07:39:53.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.945+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d4801a070 con 0x7f4d5015f9d0 2024-03-06T07:39:53.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.946+0000 7f4d47e00640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d3006b4d0 0x7f4d3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:53.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.946+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4d48089e20 con 0x7f4d5015f9d0 2024-03-06T07:39:53.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.946+0000 7f4d4fe00640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d3006b4d0 0x7f4d3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:53.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.947+0000 7f4d4fe00640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d3006b4d0 0x7f4d3006d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4d40036000 tx=0x7f4d40008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:53.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:53.951+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d5006ee70 con 0x7f4d5015f9d0 2024-03-06T07:39:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.289+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d50072890 con 0x7f4d5015f9d0 2024-03-06T07:39:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.290+0000 7f4d47e00640 1 -- 172.21.15.97:0/2376927297 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f4d48058620 con 0x7f4d5015f9d0 2024-03-06T07:39:54.292 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:54.292 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.292+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d3006b4d0 msgr2=0x7f4d3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.292+0000 7f4d57000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d3006b4d0 0x7f4d3006d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4d40036000 tx=0x7f4d40008030 comp rx=0 tx=0).stop 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.292+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 msgr2=0x7f4d500b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.292+0000 7f4d57000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 0x7f4d500b38d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800ca80 tx=0x7f4d4800cf40 comp rx=0 tx=0).stop 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 shutdown_connections 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d3006b4d0 0x7f4d3006d990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d5015f9d0 0x7f4d500b38d0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d50144c00 0x7f4d500b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 --2- 172.21.15.97:0/2376927297 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d500ce0a0 0x7f4d5015aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 >> 172.21.15.97:0/2376927297 conn(0x7f4d500b1730 msgr2=0x7f4d50142d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 shutdown_connections 2024-03-06T07:39:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:54.293+0000 7f4d57000640 1 -- 172.21.15.97:0/2376927297 wait complete. 2024-03-06T07:39:54.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:54.515 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:55.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:55.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.727+0000 7f3a15400640 1 Processor -- start 2024-03-06T07:39:55.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.728+0000 7f3a15400640 1 -- start start 2024-03-06T07:39:55.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a15400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3a1015e070 0x7f3a10066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a15400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f3a1006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a15400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1006eb10 con 0x7f3a1006b500 2024-03-06T07:39:55.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a15400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1006b970 con 0x7f3a1015e070 2024-03-06T07:39:55.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a15400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b5010 con 0x7f3a100cd860 2024-03-06T07:39:55.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a0f200640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3a1006b500 0x7f3a10161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42064/0 (socket says 172.21.15.97:42064) 2024-03-06T07:39:55.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.729+0000 7f3a0f200640 1 -- 172.21.15.97:0/1417584774 learned_addr learned my addr 172.21.15.97:0/1417584774 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0de00640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f3a1006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836432578 0 0) 0x7f3a1006eb10 con 0x7f3a1006b500 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f4003520 con 0x7f3a1006b500 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0e800640 1 --2- 172.21.15.97:0/1417584774 >> v2:172.21.15.135:3300/0 conn(0x7f3a1015e070 0x7f3a10066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351443691 0 0) 0x7f39f4003520 con 0x7f3a1006b500 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 msgr2=0x7f3a1006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f3a1006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 >> v2:172.21.15.135:3300/0 conn(0x7f3a1015e070 msgr2=0x7f3a10066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 --2- 172.21.15.97:0/1417584774 >> v2:172.21.15.135:3300/0 conn(0x7f3a1015e070 0x7f3a10066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100b52f0 con 0x7f3a1006b500 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3a080019f0 con 0x7f3a1006b500 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 >> v1:172.21.15.97:6789/0 conn(0x7f3a1006b500 legacy=0x7f3a10161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39f4004800 0x7f39f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f39f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1006b970 con 0x7f39f4004800 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b5010 con 0x7f39f40039b0 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.730+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f400ba70 con 0x7f3a100cd860 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0de00640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39f4004800 0x7f39f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0e800640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0f200640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f39f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0de00640 1 -- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 msgr2=0x7f39f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0de00640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f39f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0de00640 1 -- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39f40039b0 msgr2=0x7f39f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0de00640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0de00640 1 -- 172.21.15.97:0/1417584774 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4003520 con 0x7f39f4004800 2024-03-06T07:39:55.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.731+0000 7f3a0f200640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a100cd860 0x7f39f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.732+0000 7f3a0de00640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39f4004800 0x7f39f4004c80 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f39f800ba00 tx=0x7f39f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44bfb14dfaa1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f800b600 con 0x7f39f4004800 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f8038070 con 0x7f39f4004800 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a0d400640 1 -- 172.21.15.97:0/1417584774 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f80338a0 con 0x7f39f4004800 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a15400640 1 -- 172.21.15.97:0/1417584774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39f4004800 msgr2=0x7f39f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a15400640 1 --2- 172.21.15.97:0/1417584774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39f4004800 0x7f39f4004c80 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f39f800ba00 tx=0x7f39f4005830 comp rx=0 tx=0).stop 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a0f200640 1 -- 172.21.15.97:0/1417584774 reap_dead start 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a15400640 1 -- 172.21.15.97:0/1417584774 shutdown_connections 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a15400640 1 -- 172.21.15.97:0/1417584774 >> 172.21.15.97:0/1417584774 conn(0x7f3a100b1e10 msgr2=0x7f3a100b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a15400640 1 -- 172.21.15.97:0/1417584774 shutdown_connections 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.733+0000 7f3a15400640 1 -- 172.21.15.97:0/1417584774 wait complete. 2024-03-06T07:39:55.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.735+0000 7f3a15400640 1 Processor -- start 2024-03-06T07:39:55.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.735+0000 7f3a15400640 1 -- start start 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a15400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39f4004800 0x7f3a100998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a15400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 0x7f3a100cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a15400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a100cf980 0x7f3a100c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a15400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b5b90 con 0x7f3a10155da0 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a15400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b58b0 con 0x7f3a100cf980 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a15400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1015e450 con 0x7f39f4004800 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a0de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 0x7f3a100cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a0de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 0x7f3a100cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46102/0 (socket says 172.21.15.97:46102) 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.736+0000 7f3a0de00640 1 -- 172.21.15.97:0/828569457 learned_addr learned my addr 172.21.15.97:0/828569457 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0e800640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39f4004800 0x7f3a100998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0f200640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a100cf980 0x7f3a100c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0de00640 1 -- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39f4004800 msgr2=0x7f3a100998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0de00640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39f4004800 0x7f3a100998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0de00640 1 -- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a100cf980 msgr2=0x7f3a100c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0de00640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a100cf980 0x7f3a100c4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0de00640 1 -- 172.21.15.97:0/828569457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100c3f70 con 0x7f3a10155da0 2024-03-06T07:39:55.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a0de00640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 0x7f3a100cf440 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f39f802bf30 tx=0x7f39f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8042070 con 0x7f3a10155da0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f8004020 con 0x7f3a10155da0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a100d0740 con 0x7f3a10155da0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.737+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8004410 con 0x7f3a10155da0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.738+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a100c4210 con 0x7f3a10155da0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.739+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1004f330 con 0x7f3a10155da0 2024-03-06T07:39:55.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.740+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39f8031050 con 0x7f3a10155da0 2024-03-06T07:39:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.740+0000 7f3a06a00640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39ec06b500 0x7f39ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.741+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f39f80ac240 con 0x7f3a10155da0 2024-03-06T07:39:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.741+0000 7f3a0e800640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39ec06b500 0x7f39ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.741+0000 7f3a0e800640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39ec06b500 0x7f39ec06d9c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3a101f76e0 tx=0x7f3a00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:55.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:55.749+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39f807af70 con 0x7f3a10155da0 2024-03-06T07:39:56.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.028+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a100cfd60 con 0x7f3a10155da0 2024-03-06T07:39:56.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.029+0000 7f3a06a00640 1 -- 172.21.15.97:0/828569457 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f39f807a870 con 0x7f3a10155da0 2024-03-06T07:39:56.031 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:56.032 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:56.032 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39ec06b500 msgr2=0x7f39ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39ec06b500 0x7f39ec06d9c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3a101f76e0 tx=0x7f3a00008040 comp rx=0 tx=0).stop 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 msgr2=0x7f3a100cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 0x7f3a100cf440 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f39f802bf30 tx=0x7f39f8004730 comp rx=0 tx=0).stop 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 shutdown_connections 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39ec06b500 0x7f39ec06d9c0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a100cf980 0x7f3a100c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a10155da0 0x7f3a100cf440 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 --2- 172.21.15.97:0/828569457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f39f4004800 0x7f3a100998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 >> 172.21.15.97:0/828569457 conn(0x7f3a100b1e10 msgr2=0x7f3a100ad810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 shutdown_connections 2024-03-06T07:39:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:56.032+0000 7f3a15400640 1 -- 172.21.15.97:0/828569457 wait complete. 2024-03-06T07:39:56.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:56.254 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:57.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:57.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.467+0000 7fb23d200640 1 Processor -- start 2024-03-06T07:39:57.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.467+0000 7fb23d200640 1 -- start start 2024-03-06T07:39:57.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.468+0000 7fb23d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb238151660 0x7fb238071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.468+0000 7fb23d200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb23815dff0 0x7fb23815e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.468+0000 7fb23d200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb238071a60 con 0x7fb2380cf430 2024-03-06T07:39:57.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.468+0000 7fb23d200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2380b3ee0 con 0x7fb23815dff0 2024-03-06T07:39:57.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.468+0000 7fb23d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380b35b0 con 0x7fb238151660 2024-03-06T07:39:57.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb23815dff0 0x7fb23815e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb235e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb238151660 0x7fb238071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb23815dff0 0x7fb23815e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:38440/0 (socket says 172.21.15.97:38440) 2024-03-06T07:39:57.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 -- 172.21.15.97:0/3787693428 learned_addr learned my addr 172.21.15.97:0/3787693428 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:57.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 -- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb238151660 msgr2=0x7fb238071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:57.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 --2- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb238151660 0x7fb238071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 -- 172.21.15.97:0/3787693428 >> v1:172.21.15.135:6789/0 conn(0x7fb2380cf430 legacy=0x7fb23806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.469+0000 7fb236800640 1 -- 172.21.15.97:0/3787693428 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2380b3890 con 0x7fb23815dff0 2024-03-06T07:39:57.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.471+0000 7fb236800640 1 --2- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb23815e480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb23000aa10 tx=0x7fb23002de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=477fb8260627fcec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:57.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.471+0000 7fb235400640 1 -- 172.21.15.97:0/3787693428 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23003b070 con 0x7fb23815dff0 2024-03-06T07:39:57.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.471+0000 7fb235400640 1 -- 172.21.15.97:0/3787693428 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23002ebf0 con 0x7fb23815dff0 2024-03-06T07:39:57.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.471+0000 7fb235400640 1 -- 172.21.15.97:0/3787693428 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23000b040 con 0x7fb23815dff0 2024-03-06T07:39:57.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 -- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 msgr2=0x7fb23815e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 --2- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb23815e480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb23000aa10 tx=0x7fb23002de40 comp rx=0 tx=0).stop 2024-03-06T07:39:57.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 -- 172.21.15.97:0/3787693428 shutdown_connections 2024-03-06T07:39:57.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 --2- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb23815e480 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 --2- 172.21.15.97:0/3787693428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb238151660 0x7fb238071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 -- 172.21.15.97:0/3787693428 >> 172.21.15.97:0/3787693428 conn(0x7fb2380b1730 msgr2=0x7fb2380b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:57.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 -- 172.21.15.97:0/3787693428 shutdown_connections 2024-03-06T07:39:57.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.472+0000 7fb23d200640 1 -- 172.21.15.97:0/3787693428 wait complete. 2024-03-06T07:39:57.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.474+0000 7fb23d200640 1 Processor -- start 2024-03-06T07:39:57.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.474+0000 7fb23d200640 1 -- start start 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb23d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2380cf430 0x7fb23813cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb23d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 0x7fb2381f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb23d200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb238151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb23d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23806daf0 con 0x7fb238151660 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb23d200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb238071d30 con 0x7fb23815dff0 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb23d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23806cb50 con 0x7fb2380cf430 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.475+0000 7fb22d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 0x7fb2381f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 0x7fb2381f8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46114/0 (socket says 172.21.15.97:46114) 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 -- 172.21.15.97:0/2040677979 learned_addr learned my addr 172.21.15.97:0/2040677979 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb236800640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb238151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb235e00640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2380cf430 0x7fb23813cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 -- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2380cf430 msgr2=0x7fb23813cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2380cf430 0x7fb23813cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 -- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 msgr2=0x7fb238151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb238151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 -- 172.21.15.97:0/2040677979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23814fa80 con 0x7fb238151660 2024-03-06T07:39:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb236800640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb238151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.476+0000 7fb22d400640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 0x7fb2381f8910 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c400 tx=0x7fb22000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.477+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2200052d0 con 0x7fb238151660 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.477+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb220005490 con 0x7fb238151660 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.477+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220007260 con 0x7fb238151660 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.477+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23813db40 con 0x7fb238151660 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.477+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23813e100 con 0x7fb238151660 2024-03-06T07:39:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.478+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23804f330 con 0x7fb238151660 2024-03-06T07:39:57.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.479+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb220007400 con 0x7fb238151660 2024-03-06T07:39:57.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.480+0000 7fb22ea00640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb21406b500 0x7fb21406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.480+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb220089710 con 0x7fb238151660 2024-03-06T07:39:57.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.480+0000 7fb235e00640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb21406b500 0x7fb21406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.481+0000 7fb235e00640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb21406b500 0x7fb21406d9c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb2280074a0 tx=0x7fb22800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:57.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.485+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb220058560 con 0x7fb238151660 2024-03-06T07:39:57.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.723+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb238159690 con 0x7fb238151660 2024-03-06T07:39:57.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.724+0000 7fb22ea00640 1 -- 172.21.15.97:0/2040677979 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fb220057e60 con 0x7fb238151660 2024-03-06T07:39:57.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:57.727 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:57.727 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb21406b500 msgr2=0x7fb21406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb21406b500 0x7fb21406d9c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb2280074a0 tx=0x7fb22800b040 comp rx=0 tx=0).stop 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 msgr2=0x7fb2381f8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 0x7fb2381f8910 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c400 tx=0x7fb22000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 shutdown_connections 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb21406b500 0x7fb21406d9c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb23815dff0 0x7fb238151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb238151660 0x7fb2381f8910 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 --2- 172.21.15.97:0/2040677979 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb2380cf430 0x7fb23813cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.727+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 >> 172.21.15.97:0/2040677979 conn(0x7fb2380b1730 msgr2=0x7fb238159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.728+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 shutdown_connections 2024-03-06T07:39:57.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:57.728+0000 7fb23d200640 1 -- 172.21.15.97:0/2040677979 wait complete. 2024-03-06T07:39:57.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:57.951 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:39:58.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.168+0000 7f3a21400640 1 Processor -- start 2024-03-06T07:39:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.168+0000 7f3a21400640 1 -- start start 2024-03-06T07:39:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.170+0000 7f3a21400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a1c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.170+0000 7f3a21400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c071d30 con 0x7f3a1c070d20 2024-03-06T07:39:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.170+0000 7f3a21400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c071490 con 0x7f3a1c06e4d0 2024-03-06T07:39:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.170+0000 7f3a21400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c071770 con 0x7f3a1c0af1c0 2024-03-06T07:39:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.170+0000 7f3a19e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3a1c070d20 0x7f3a1c06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42070/0 (socket says 172.21.15.97:42070) 2024-03-06T07:39:59.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.170+0000 7f3a19e00640 1 -- 172.21.15.97:0/1322622509 learned_addr learned my addr 172.21.15.97:0/1322622509 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a1a800640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a1c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516620204 0 0) 0x7f3a08001110 con 0x7f3a1c070d20 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a00003520 con 0x7f3a1c070d20 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681169217 0 0) 0x7f3a00003520 con 0x7f3a1c070d20 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 msgr2=0x7f3a1c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a1c15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 >> v1:172.21.15.135:6789/0 conn(0x7f3a1c06e4d0 legacy=0x7f3a1c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c071a50 con 0x7f3a1c070d20 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f3a08001840 con 0x7f3a1c070d20 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 >> v1:172.21.15.97:6789/0 conn(0x7f3a1c070d20 legacy=0x7f3a1c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a000039e0 0x7f3a00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a00004d40 0x7f3a000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c071770 con 0x7f3a00004d40 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0000ba70 con 0x7f3a000039e0 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.171+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0000bc70 con 0x7f3a1c0af1c0 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1b200640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a00004d40 0x7f3a000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1a800640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a000039e0 0x7f3a00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a19e00640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1b200640 1 -- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 msgr2=0x7f3a00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1b200640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1b200640 1 -- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a000039e0 msgr2=0x7f3a00003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1b200640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a000039e0 0x7f3a00003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1b200640 1 -- 172.21.15.97:0/1322622509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00003520 con 0x7f3a00004d40 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a19e00640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a1c0af1c0 0x7f3a00004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:59.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.172+0000 7f3a1a800640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a000039e0 0x7f3a00003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.173+0000 7f3a1b200640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a00004d40 0x7f3a000051c0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f3a14001e20 tx=0x7f3a00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8422d72a065c670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.173+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a14001a70 con 0x7f3a00004d40 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.173+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a14003b70 con 0x7f3a00004d40 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.173+0000 7f3a19400640 1 -- 172.21.15.97:0/1322622509 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a140050d0 con 0x7f3a00004d40 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a21400640 1 -- 172.21.15.97:0/1322622509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a00004d40 msgr2=0x7f3a000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a21400640 1 --2- 172.21.15.97:0/1322622509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a00004d40 0x7f3a000051c0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f3a14001e20 tx=0x7f3a00005830 comp rx=0 tx=0).stop 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a1b200640 1 -- 172.21.15.97:0/1322622509 reap_dead start 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a21400640 1 -- 172.21.15.97:0/1322622509 shutdown_connections 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a21400640 1 -- 172.21.15.97:0/1322622509 >> 172.21.15.97:0/1322622509 conn(0x7f3a1c0b1730 msgr2=0x7f3a1c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a21400640 1 -- 172.21.15.97:0/1322622509 shutdown_connections 2024-03-06T07:39:59.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.174+0000 7f3a21400640 1 -- 172.21.15.97:0/1322622509 wait complete. 2024-03-06T07:39:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.176+0000 7f3a21400640 1 Processor -- start 2024-03-06T07:39:59.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.176+0000 7f3a21400640 1 -- start start 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a21400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a00004d40 0x7f3a1c14c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a21400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 0x7f3a1c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a21400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a1c0ce890 0x7f3a1c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a21400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c06d370 con 0x7f3a1c1597f0 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a21400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0711a0 con 0x7f3a1c0ce890 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a21400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b3ce0 con 0x7f3a00004d40 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a19e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 0x7f3a1c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a19e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 0x7f3a1c0cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46132/0 (socket says 172.21.15.97:46132) 2024-03-06T07:39:59.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a19e00640 1 -- 172.21.15.97:0/3316010231 learned_addr learned my addr 172.21.15.97:0/3316010231 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a1a800640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a00004d40 0x7f3a1c14c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.177+0000 7f3a1b200640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a1c0ce890 0x7f3a1c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a19e00640 1 -- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a00004d40 msgr2=0x7f3a1c14c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a19e00640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a00004d40 0x7f3a1c14c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a19e00640 1 -- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a1c0ce890 msgr2=0x7f3a1c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a19e00640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a1c0ce890 0x7f3a1c0ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a19e00640 1 -- 172.21.15.97:0/3316010231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c0ccd30 con 0x7f3a1c1597f0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a19e00640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 0x7f3a1c0cc430 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800d940 tx=0x7f3a0800de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08019070 con 0x7f3a1c1597f0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a080074e0 con 0x7f3a1c1597f0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080049e0 con 0x7f3a1c1597f0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.178+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c0cd010 con 0x7f3a1c1597f0 2024-03-06T07:39:59.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.179+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c0cd5d0 con 0x7f3a1c1597f0 2024-03-06T07:39:59.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.180+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1c04f330 con 0x7f3a1c1597f0 2024-03-06T07:39:59.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.181+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a08004b80 con 0x7f3a1c1597f0 2024-03-06T07:39:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.181+0000 7f3a12a00640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39fc06b4d0 0x7f39fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.182+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3a08089e50 con 0x7f3a1c1597f0 2024-03-06T07:39:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.182+0000 7f3a1a800640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39fc06b4d0 0x7f39fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.182+0000 7f3a1a800640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39fc06b4d0 0x7f39fc06d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009aa0 tx=0x7f3a0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:59.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.190+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a1c071d30 con 0x7f3a1c1597f0 2024-03-06T07:39:59.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.430+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a1c15bee0 con 0x7f3a1c1597f0 2024-03-06T07:39:59.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.431+0000 7f3a12a00640 1 -- 172.21.15.97:0/3316010231 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f3a08058c10 con 0x7f3a1c1597f0 2024-03-06T07:39:59.432 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:39:59.432 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:39:59.432 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39fc06b4d0 msgr2=0x7f39fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39fc06b4d0 0x7f39fc06d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009aa0 tx=0x7f3a0c008040 comp rx=0 tx=0).stop 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 msgr2=0x7f3a1c0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 0x7f3a1c0cc430 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800d940 tx=0x7f3a0800de00 comp rx=0 tx=0).stop 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 shutdown_connections 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f39fc06b4d0 0x7f39fc06d990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a1c0ce890 0x7f3a1c0ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a1c1597f0 0x7f3a1c0cc430 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.433+0000 7f3a21400640 1 --2- 172.21.15.97:0/3316010231 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3a00004d40 0x7f3a1c14c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.434+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 >> 172.21.15.97:0/3316010231 conn(0x7f3a1c0b1730 msgr2=0x7f3a1c0b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.434+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 shutdown_connections 2024-03-06T07:39:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:39:59.434+0000 7f3a21400640 1 -- 172.21.15.97:0/3316010231 wait complete. 2024-03-06T07:39:59.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:39:59.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:00.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:00.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.874+0000 7f4dc4000640 1 Processor -- start 2024-03-06T07:40:00.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.874+0000 7f4dc4000640 1 -- start start 2024-03-06T07:40:00.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.875+0000 7f4dc4000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4dbc15f9d0 0x7f4dbc15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.875+0000 7f4dc4000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4dbc070b10 0x7f4dbc070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.875+0000 7f4dc4000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc06cef0 con 0x7f4dbc144c00 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.875+0000 7f4dc4000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc06d1d0 con 0x7f4dbc15f9d0 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.875+0000 7f4dc4000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc06d4b0 con 0x7f4dbc070b10 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4dbc15f9d0 0x7f4dbc15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4dbc15f9d0 0x7f4dbc15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46148/0 (socket says 172.21.15.97:46148) 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 -- 172.21.15.97:0/252741713 learned_addr learned my addr 172.21.15.97:0/252741713 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc2000640 1 --2- 172.21.15.97:0/252741713 >> v2:172.21.15.135:3300/0 conn(0x7f4dbc070b10 0x7f4dbc070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 -- 172.21.15.97:0/252741713 >> v2:172.21.15.135:3300/0 conn(0x7f4dbc070b10 msgr2=0x7f4dbc070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 --2- 172.21.15.97:0/252741713 >> v2:172.21.15.135:3300/0 conn(0x7f4dbc070b10 0x7f4dbc070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 -- 172.21.15.97:0/252741713 >> v1:172.21.15.135:6789/0 conn(0x7f4dbc144c00 legacy=0x7f4dbc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:00.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.876+0000 7f4dc0c00640 1 -- 172.21.15.97:0/252741713 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc06d790 con 0x7f4dbc15f9d0 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.877+0000 7f4dc0c00640 1 --2- 172.21.15.97:0/252741713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc15fe60 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc06e500 tx=0x7f4da802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c489909535ff54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.878+0000 7f4dbbe00640 1 -- 172.21.15.97:0/252741713 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8035020 con 0x7f4dbc15f9d0 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.878+0000 7f4dbbe00640 1 -- 172.21.15.97:0/252741713 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da8031870 con 0x7f4dbc15f9d0 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.878+0000 7f4dbbe00640 1 -- 172.21.15.97:0/252741713 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8031c40 con 0x7f4dbc15f9d0 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.878+0000 7f4dc4000640 1 -- 172.21.15.97:0/252741713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc15f9d0 msgr2=0x7f4dbc15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.878+0000 7f4dc4000640 1 --2- 172.21.15.97:0/252741713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc15fe60 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc06e500 tx=0x7f4da802b910 comp rx=0 tx=0).stop 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.879+0000 7f4dc4000640 1 -- 172.21.15.97:0/252741713 shutdown_connections 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.879+0000 7f4dc4000640 1 --2- 172.21.15.97:0/252741713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc15fe60 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.879+0000 7f4dc4000640 1 --2- 172.21.15.97:0/252741713 >> v2:172.21.15.135:3300/0 conn(0x7f4dbc070b10 0x7f4dbc070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.879+0000 7f4dc4000640 1 -- 172.21.15.97:0/252741713 >> 172.21.15.97:0/252741713 conn(0x7f4dbc0b1730 msgr2=0x7f4dbc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.879+0000 7f4dc4000640 1 -- 172.21.15.97:0/252741713 shutdown_connections 2024-03-06T07:40:00.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.879+0000 7f4dc4000640 1 -- 172.21.15.97:0/252741713 wait complete. 2024-03-06T07:40:00.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.881+0000 7f4dc4000640 1 Processor -- start 2024-03-06T07:40:00.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.881+0000 7f4dc4000640 1 -- start start 2024-03-06T07:40:00.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc4000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc070b10 0x7f4dbc0b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc4000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 0x7f4dbc153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc4000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc0ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc4000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc06ccd0 con 0x7f4dbc144c00 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc4000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc06cb50 con 0x7f4dbc15f9d0 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc4000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc06df90 con 0x7f4dbc070b10 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 0x7f4dbc153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 0x7f4dbc153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46164/0 (socket says 172.21.15.97:46164) 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.882+0000 7f4dc0c00640 1 -- 172.21.15.97:0/2087953210 learned_addr learned my addr 172.21.15.97:0/2087953210 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc1600640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc070b10 0x7f4dbc0b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc2000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc0ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc0c00640 1 -- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc070b10 msgr2=0x7f4dbc0b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc0c00640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc070b10 0x7f4dbc0b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc0c00640 1 -- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15f9d0 msgr2=0x7f4dbc0ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc0c00640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc0ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc0c00640 1 -- 172.21.15.97:0/2087953210 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc153ba0 con 0x7f4dbc144c00 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc0c00640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 0x7f4dbc153660 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f4da802b950 tx=0x7f4da80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8040070 con 0x7f4dbc144c00 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da8030d00 con 0x7f4dbc144c00 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8035040 con 0x7f4dbc144c00 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.883+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc0b3cb0 con 0x7f4dbc144c00 2024-03-06T07:40:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.884+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc13b3e0 con 0x7f4dbc144c00 2024-03-06T07:40:00.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.885+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc0b41d0 con 0x7f4dbc144c00 2024-03-06T07:40:00.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.886+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4da804b060 con 0x7f4dbc144c00 2024-03-06T07:40:00.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.887+0000 7f4db9600640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.887+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4da80abfe0 con 0x7f4dbc144c00 2024-03-06T07:40:00.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.887+0000 7f4dc1600640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.888+0000 7f4dc1600640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4db0003d90 tx=0x7f4db0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:00.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:00.891+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4da807adc0 con 0x7f4dbc144c00 2024-03-06T07:40:01.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.165+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dbc04f330 con 0x7f4dbc144c00 2024-03-06T07:40:01.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.166+0000 7f4db9600640 1 -- 172.21.15.97:0/2087953210 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f4da807a6c0 con 0x7f4dbc144c00 2024-03-06T07:40:01.167 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:01.167 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:01.167 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9c06b4d0 msgr2=0x7f4d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4db0003d90 tx=0x7f4db0009040 comp rx=0 tx=0).stop 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 msgr2=0x7f4dbc153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 0x7f4dbc153660 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f4da802b950 tx=0x7f4da80046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 shutdown_connections 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dbc15f9d0 0x7f4dbc0ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4dbc144c00 0x7f4dbc153660 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 --2- 172.21.15.97:0/2087953210 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4dbc070b10 0x7f4dbc0b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 >> 172.21.15.97:0/2087953210 conn(0x7f4dbc0b1730 msgr2=0x7f4dbc0b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.168+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 shutdown_connections 2024-03-06T07:40:01.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:01.169+0000 7f4dc4000640 1 -- 172.21.15.97:0/2087953210 wait complete. 2024-03-06T07:40:01.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:01.389 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:02.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:02.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.598+0000 7f66aaa00640 1 Processor -- start 2024-03-06T07:40:02.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.598+0000 7f66aaa00640 1 -- start start 2024-03-06T07:40:02.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.599+0000 7f66aaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f66a415fe60 0x7f66a4071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.599+0000 7f66aaa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a406cef0 con 0x7f66a4144c00 2024-03-06T07:40:02.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.599+0000 7f66aaa00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a406d1d0 con 0x7f66a415f9d0 2024-03-06T07:40:02.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.599+0000 7f66aaa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a406d4b0 con 0x7f66a415fe60 2024-03-06T07:40:02.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a3e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f66a4144c00 0x7f66a406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42084/0 (socket says 172.21.15.97:42084) 2024-03-06T07:40:02.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a3e00640 1 -- 172.21.15.97:0/4292860366 learned_addr learned my addr 172.21.15.97:0/4292860366 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:02.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a8a00640 1 --2- 172.21.15.97:0/4292860366 >> v2:172.21.15.97:3300/0 conn(0x7f66a415fe60 0x7f66a4071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a3400640 1 -- 172.21.15.97:0/4292860366 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026166275 0 0) 0x7f66a406cef0 con 0x7f66a4144c00 2024-03-06T07:40:02.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a8a00640 1 -- 172.21.15.97:0/4292860366 >> v1:172.21.15.135:6789/0 conn(0x7f66a415f9d0 legacy=0x7f66a4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:02.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a8a00640 1 -- 172.21.15.97:0/4292860366 >> v1:172.21.15.97:6789/0 conn(0x7f66a4144c00 legacy=0x7f66a406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.600+0000 7f66a8a00640 1 -- 172.21.15.97:0/4292860366 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a406d790 con 0x7f66a415fe60 2024-03-06T07:40:02.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.601+0000 7f66a8a00640 1 --2- 172.21.15.97:0/4292860366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a415fe60 0x7f66a4071220 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f669c009d90 tx=0x7f669c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347f2cb6c0a54b0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:02.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66a3400640 1 -- 172.21.15.97:0/4292860366 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c039070 con 0x7f66a415fe60 2024-03-06T07:40:02.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66a3400640 1 -- 172.21.15.97:0/4292860366 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c02cc60 con 0x7f66a415fe60 2024-03-06T07:40:02.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66a3400640 1 -- 172.21.15.97:0/4292860366 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c034660 con 0x7f66a415fe60 2024-03-06T07:40:02.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 -- 172.21.15.97:0/4292860366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a415fe60 msgr2=0x7f66a4071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 --2- 172.21.15.97:0/4292860366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a415fe60 0x7f66a4071220 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f669c009d90 tx=0x7f669c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 -- 172.21.15.97:0/4292860366 shutdown_connections 2024-03-06T07:40:02.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 --2- 172.21.15.97:0/4292860366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a415fe60 0x7f66a4071220 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 -- 172.21.15.97:0/4292860366 >> 172.21.15.97:0/4292860366 conn(0x7f66a40b1730 msgr2=0x7f66a40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:02.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 -- 172.21.15.97:0/4292860366 shutdown_connections 2024-03-06T07:40:02.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.602+0000 7f66aaa00640 1 -- 172.21.15.97:0/4292860366 wait complete. 2024-03-06T07:40:02.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.605+0000 7f66aaa00640 1 Processor -- start 2024-03-06T07:40:02.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.605+0000 7f66aaa00640 1 -- start start 2024-03-06T07:40:02.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66aaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 0x7f66a4140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66aaa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66a415f9d0 0x7f66a4139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66aaa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66a415fe60 0x7f66a41e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66aaa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a406cb50 con 0x7f66a4144c00 2024-03-06T07:40:02.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66aaa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4160730 con 0x7f66a415f9d0 2024-03-06T07:40:02.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66aaa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a406ccd0 con 0x7f66a415fe60 2024-03-06T07:40:02.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 0x7f66a4140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 0x7f66a4140260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46186/0 (socket says 172.21.15.97:46186) 2024-03-06T07:40:02.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66a3e00640 1 -- 172.21.15.97:0/1996245068 learned_addr learned my addr 172.21.15.97:0/1996245068 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:02.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f669be00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66a415f9d0 0x7f66a4139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.607+0000 7f66a8a00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66a415fe60 0x7f66a41e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a3e00640 1 -- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66a415fe60 msgr2=0x7f66a41e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a3e00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66a415fe60 0x7f66a41e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a3e00640 1 -- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66a415f9d0 msgr2=0x7f66a4139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a3e00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66a415f9d0 0x7f66a4139690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a3e00640 1 -- 172.21.15.97:0/1996245068 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a41e5b00 con 0x7f66a4144c00 2024-03-06T07:40:02.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a3e00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 0x7f66a4140260 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f669400c460 tx=0x7f669400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:02.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694018070 con 0x7f66a4144c00 2024-03-06T07:40:02.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6694007500 con 0x7f66a4144c00 2024-03-06T07:40:02.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694004a20 con 0x7f66a4144c00 2024-03-06T07:40:02.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a41e5da0 con 0x7f66a4144c00 2024-03-06T07:40:02.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.608+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a41e6360 con 0x7f66a4144c00 2024-03-06T07:40:02.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.609+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a404f330 con 0x7f66a4144c00 2024-03-06T07:40:02.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.610+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6694004bc0 con 0x7f66a4144c00 2024-03-06T07:40:02.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.612+0000 7f66a0c00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f668006b4d0 0x7f668006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.612+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6694089540 con 0x7f66a4144c00 2024-03-06T07:40:02.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.612+0000 7f669be00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f668006b4d0 0x7f668006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.613+0000 7f669be00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f668006b4d0 0x7f668006d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f66a415e680 tx=0x7f668c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:02.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.617+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6694058390 con 0x7f66a4144c00 2024-03-06T07:40:02.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.918+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66a4072890 con 0x7f66a4144c00 2024-03-06T07:40:02.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.918+0000 7f66a0c00640 1 -- 172.21.15.97:0/1996245068 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f6694006e20 con 0x7f66a4144c00 2024-03-06T07:40:02.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:02.921 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:02.921 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f668006b4d0 msgr2=0x7f668006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f668006b4d0 0x7f668006d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f66a415e680 tx=0x7f668c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 msgr2=0x7f66a4140260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 0x7f66a4140260 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f669400c460 tx=0x7f669400c920 comp rx=0 tx=0).stop 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 shutdown_connections 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f668006b4d0 0x7f668006d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f66a415fe60 0x7f66a41e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66a415f9d0 0x7f66a4139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 --2- 172.21.15.97:0/1996245068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4144c00 0x7f66a4140260 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.921+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 >> 172.21.15.97:0/1996245068 conn(0x7f66a40b1730 msgr2=0x7f66a40b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.922+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 shutdown_connections 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:02.922+0000 7f66aaa00640 1 -- 172.21.15.97:0/1996245068 wait complete. 2024-03-06T07:40:03.143 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:03.144 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:04.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:04.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.356+0000 7f4d7d800640 1 Processor -- start 2024-03-06T07:40:04.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.356+0000 7f4d7d800640 1 -- start start 2024-03-06T07:40:04.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.357+0000 7f4d7d800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f4d7815acb0 0x7f4d780b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.357+0000 7f4d7d800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d7806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.357+0000 7f4d7d800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4d78072340 0x7f4d7806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.357+0000 7f4d7d800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d7815b090 con 0x7f4d78072340 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.357+0000 7f4d7d800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d780b2a30 con 0x7f4d7815acb0 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.357+0000 7f4d7d800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780b4360 con 0x7f4d780b4680 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4d78072340 0x7f4d7806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4d78072340 0x7f4d7806ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55570/0 (socket says 172.21.15.97:55570) 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 -- 172.21.15.97:0/1544262413 learned_addr learned my addr 172.21.15.97:0/1544262413 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d76a00640 1 --2- 172.21.15.97:0/1544262413 >> v2:172.21.15.135:3300/0 conn(0x7f4d7815acb0 0x7f4d780b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d76000640 1 --2- 172.21.15.97:0/1544262413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d7806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 -- 172.21.15.97:0/1544262413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 msgr2=0x7f4d7806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 --2- 172.21.15.97:0/1544262413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d7806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 -- 172.21.15.97:0/1544262413 >> v2:172.21.15.135:3300/0 conn(0x7f4d7815acb0 msgr2=0x7f4d780b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:04.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 --2- 172.21.15.97:0/1544262413 >> v2:172.21.15.135:3300/0 conn(0x7f4d7815acb0 0x7f4d780b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.358+0000 7f4d77400640 1 -- 172.21.15.97:0/1544262413 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780b2cf0 con 0x7f4d78072340 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.359+0000 7f4d77400640 1 --2- 172.21.15.97:0/1544262413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d7806ea30 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f4d780b27f0 tx=0x7f4d7002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacb74a9fb867a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.360+0000 7f4d75600640 1 -- 172.21.15.97:0/1544262413 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7002ec80 con 0x7f4d78072340 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.360+0000 7f4d75600640 1 -- 172.21.15.97:0/1544262413 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7002ee20 con 0x7f4d78072340 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.360+0000 7f4d75600640 1 -- 172.21.15.97:0/1544262413 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70036610 con 0x7f4d78072340 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.360+0000 7f4d7d800640 1 -- 172.21.15.97:0/1544262413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 msgr2=0x7f4d7806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.360+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1544262413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d7806ea30 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f4d780b27f0 tx=0x7f4d7002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 -- 172.21.15.97:0/1544262413 shutdown_connections 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1544262413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d7806ea30 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1544262413 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d7806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1544262413 >> v2:172.21.15.135:3300/0 conn(0x7f4d7815acb0 0x7f4d780b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 -- 172.21.15.97:0/1544262413 >> 172.21.15.97:0/1544262413 conn(0x7f4d780b1730 msgr2=0x7f4d780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:04.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 -- 172.21.15.97:0/1544262413 shutdown_connections 2024-03-06T07:40:04.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.361+0000 7f4d7d800640 1 -- 172.21.15.97:0/1544262413 wait complete. 2024-03-06T07:40:04.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.363+0000 7f4d7d800640 1 Processor -- start 2024-03-06T07:40:04.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.364+0000 7f4d7d800640 1 -- start start 2024-03-06T07:40:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d7d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d780ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d7d800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d780cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d7d800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d7815acb0 0x7f4d78153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d7d800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780b3d10 con 0x7f4d78072340 2024-03-06T07:40:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d7d800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780b32b0 con 0x7f4d7815acb0 2024-03-06T07:40:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d7d800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780b4010 con 0x7f4d780b4680 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.365+0000 7f4d76a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d780ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d780ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55572/0 (socket says 172.21.15.97:55572) 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 -- 172.21.15.97:0/1181266252 learned_addr learned my addr 172.21.15.97:0/1181266252 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76000640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d780cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 -- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 msgr2=0x7f4d780cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d780cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 -- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d7815acb0 msgr2=0x7f4d78153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d7815acb0 0x7f4d78153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 -- 172.21.15.97:0/1181266252 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780cdec0 con 0x7f4d78072340 2024-03-06T07:40:04.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d76a00640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d780ccf70 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800d800 tx=0x7f4d6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:04.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.366+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6800b6c0 con 0x7f4d78072340 2024-03-06T07:40:04.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.367+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d68019070 con 0x7f4d78072340 2024-03-06T07:40:04.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.367+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d78154090 con 0x7f4d78072340 2024-03-06T07:40:04.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.367+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6800ba80 con 0x7f4d78072340 2024-03-06T07:40:04.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.367+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d78153350 con 0x7f4d78072340 2024-03-06T07:40:04.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.368+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d7804f330 con 0x7f4d78072340 2024-03-06T07:40:04.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.369+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d6800bc20 con 0x7f4d78072340 2024-03-06T07:40:04.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.370+0000 7f4d6ea00640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d5806b4d0 0x7f4d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.370+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4d6808a6d0 con 0x7f4d78072340 2024-03-06T07:40:04.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.370+0000 7f4d76000640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d5806b4d0 0x7f4d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.371+0000 7f4d76000640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d5806b4d0 0x7f4d5806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4d780cd4b0 tx=0x7f4d64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.375+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d780cdec0 con 0x7f4d78072340 2024-03-06T07:40:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.654+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d7806ce60 con 0x7f4d78072340 2024-03-06T07:40:04.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.655+0000 7f4d6ea00640 1 -- 172.21.15.97:0/1181266252 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f4d68059530 con 0x7f4d78072340 2024-03-06T07:40:04.656 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:04.657 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:04.657 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:04.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d5806b4d0 msgr2=0x7f4d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d5806b4d0 0x7f4d5806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4d780cd4b0 tx=0x7f4d64008040 comp rx=0 tx=0).stop 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 msgr2=0x7f4d780ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d780ccf70 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800d800 tx=0x7f4d6800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 shutdown_connections 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f4d5806b4d0 0x7f4d5806d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d7815acb0 0x7f4d78153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f4d780b4680 0x7f4d780cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 --2- 172.21.15.97:0/1181266252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4d78072340 0x7f4d780ccf70 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 >> 172.21.15.97:0/1181266252 conn(0x7f4d780b1730 msgr2=0x7f4d780af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 shutdown_connections 2024-03-06T07:40:04.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:04.659+0000 7f4d7d800640 1 -- 172.21.15.97:0/1181266252 wait complete. 2024-03-06T07:40:04.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:04.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:05.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:06.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.091+0000 7f04b4000640 1 Processor -- start 2024-03-06T07:40:06.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.092+0000 7f04b4000640 1 -- start start 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.093+0000 7f04b4000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f04ac144c00 0x7f04ac06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.093+0000 7f04b4000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 0x7f04ac15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.093+0000 7f04b4000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac06ee70 con 0x7f04ac0ce0a0 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.093+0000 7f04b4000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac160370 con 0x7f04ac144c00 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.093+0000 7f04b4000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac06cef0 con 0x7f04ac15f9d0 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.094+0000 7f04b0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 0x7f04ac15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.094+0000 7f04b1600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f04ac144c00 0x7f04ac06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.094+0000 7f04b0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 0x7f04ac15fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:42544/0 (socket says 172.21.15.97:42544) 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.094+0000 7f04b0c00640 1 -- 172.21.15.97:0/2490948744 learned_addr learned my addr 172.21.15.97:0/2490948744 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.095+0000 7f04b0c00640 1 -- 172.21.15.97:0/2490948744 >> v2:172.21.15.135:3300/0 conn(0x7f04ac144c00 msgr2=0x7f04ac06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.095+0000 7f04b0c00640 1 --2- 172.21.15.97:0/2490948744 >> v2:172.21.15.135:3300/0 conn(0x7f04ac144c00 0x7f04ac06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.095+0000 7f04b0c00640 1 -- 172.21.15.97:0/2490948744 >> v1:172.21.15.135:6789/0 conn(0x7f04ac0ce0a0 legacy=0x7f04ac0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.095+0000 7f04b0c00640 1 -- 172.21.15.97:0/2490948744 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac06d1d0 con 0x7f04ac15f9d0 2024-03-06T07:40:06.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.097+0000 7f04b0c00640 1 --2- 172.21.15.97:0/2490948744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 0x7f04ac15fe30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0498009870 tx=0x7f049802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c7a109931f207c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04abe00640 1 -- 172.21.15.97:0/2490948744 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498034070 con 0x7f04ac15f9d0 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04abe00640 1 -- 172.21.15.97:0/2490948744 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0498030840 con 0x7f04ac15f9d0 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04abe00640 1 -- 172.21.15.97:0/2490948744 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498030c10 con 0x7f04ac15f9d0 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 -- 172.21.15.97:0/2490948744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 msgr2=0x7f04ac15fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 --2- 172.21.15.97:0/2490948744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 0x7f04ac15fe30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0498009870 tx=0x7f049802b980 comp rx=0 tx=0).stop 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 -- 172.21.15.97:0/2490948744 shutdown_connections 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 --2- 172.21.15.97:0/2490948744 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac15f9d0 0x7f04ac15fe30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 --2- 172.21.15.97:0/2490948744 >> v2:172.21.15.135:3300/0 conn(0x7f04ac144c00 0x7f04ac06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 -- 172.21.15.97:0/2490948744 >> 172.21.15.97:0/2490948744 conn(0x7f04ac0b1730 msgr2=0x7f04ac0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.098+0000 7f04b4000640 1 -- 172.21.15.97:0/2490948744 shutdown_connections 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.099+0000 7f04b4000640 1 -- 172.21.15.97:0/2490948744 wait complete. 2024-03-06T07:40:06.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.101+0000 7f04b4000640 1 Processor -- start 2024-03-06T07:40:06.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.101+0000 7f04b4000640 1 -- start start 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b4000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ac0ce0a0 0x7f04ac13cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b4000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac144c00 0x7f04ac1e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b4000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 0x7f04ac1392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b4000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac160730 con 0x7f04ac15f9d0 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b4000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac06d790 con 0x7f04ac0ce0a0 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b4000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac06cb50 con 0x7f04ac144c00 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 0x7f04ac1392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 0x7f04ac1392b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55576/0 (socket says 172.21.15.97:55576) 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.102+0000 7f04b2000640 1 -- 172.21.15.97:0/1509592593 learned_addr learned my addr 172.21.15.97:0/1509592593 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b0c00640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac144c00 0x7f04ac1e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b1600640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ac0ce0a0 0x7f04ac13cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b2000640 1 -- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac144c00 msgr2=0x7f04ac1e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b2000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac144c00 0x7f04ac1e7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b2000640 1 -- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ac0ce0a0 msgr2=0x7f04ac13cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b2000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ac0ce0a0 0x7f04ac13cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b2000640 1 -- 172.21.15.97:0/1509592593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac1e6c90 con 0x7f04ac15f9d0 2024-03-06T07:40:06.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b2000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 0x7f04ac1392b0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c6b0 tx=0x7f04a400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:06.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4003b70 con 0x7f04ac15f9d0 2024-03-06T07:40:06.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4003d30 con 0x7f04ac15f9d0 2024-03-06T07:40:06.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.103+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac1e6f70 con 0x7f04ac15f9d0 2024-03-06T07:40:06.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.104+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4005310 con 0x7f04ac15f9d0 2024-03-06T07:40:06.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.104+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac1e7530 con 0x7f04ac15f9d0 2024-03-06T07:40:06.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.105+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ac04f330 con 0x7f04ac15f9d0 2024-03-06T07:40:06.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.106+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04a40054b0 con 0x7f04ac15f9d0 2024-03-06T07:40:06.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.107+0000 7f04a9600640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f048c06b4d0 0x7f048c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.108+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f04a408a120 con 0x7f04ac15f9d0 2024-03-06T07:40:06.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.108+0000 7f04b1600640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f048c06b4d0 0x7f048c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.109+0000 7f04b1600640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f048c06b4d0 0x7f048c06d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009ad0 tx=0x7f04a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:06.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.112+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04ac06ee70 con 0x7f04ac15f9d0 2024-03-06T07:40:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.343+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04ac1454a0 con 0x7f04ac15f9d0 2024-03-06T07:40:06.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.344+0000 7f04a9600640 1 -- 172.21.15.97:0/1509592593 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f04a4058ee0 con 0x7f04ac15f9d0 2024-03-06T07:40:06.347 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:06.347 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:06.347 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:06.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f048c06b4d0 msgr2=0x7f048c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f048c06b4d0 0x7f048c06d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009ad0 tx=0x7f04a0008040 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 msgr2=0x7f04ac1392b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 0x7f04ac1392b0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c6b0 tx=0x7f04a400cb70 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 shutdown_connections 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f048c06b4d0 0x7f048c06d990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04ac15f9d0 0x7f04ac1392b0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f04ac144c00 0x7f04ac1e7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 --2- 172.21.15.97:0/1509592593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ac0ce0a0 0x7f04ac13cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 >> 172.21.15.97:0/1509592593 conn(0x7f04ac0b1730 msgr2=0x7f04ac0cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 shutdown_connections 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:06.347+0000 7f04b4000640 1 -- 172.21.15.97:0/1509592593 wait complete. 2024-03-06T07:40:06.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:06.570 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:07.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:07.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.785+0000 7faa41200640 1 Processor -- start 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.785+0000 7faa41200640 1 -- start start 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.787+0000 7faa41200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7faa3c06b6d0 0x7faa3c162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.787+0000 7faa41200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c0cef70 con 0x7faa3c0affe0 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.787+0000 7faa41200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c15e3b0 con 0x7faa3c15dfd0 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.787+0000 7faa41200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3c0b0400 con 0x7faa3c06b6d0 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.787+0000 7faa3be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7faa3c0affe0 0x7faa3c0d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53706/0 (socket says 172.21.15.97:53706) 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3be00640 1 -- 172.21.15.97:0/3408551428 learned_addr learned my addr 172.21.15.97:0/3408551428 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa32a00640 1 --2- 172.21.15.97:0/3408551428 >> v2:172.21.15.135:3300/0 conn(0x7faa3c06b6d0 0x7faa3c162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704262210 0 0) 0x7faa3c0cef70 con 0x7faa3c0affe0 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa20003520 con 0x7faa3c0affe0 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 390013671 0 0) 0x7faa20003520 con 0x7faa3c0affe0 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 >> v2:172.21.15.135:3300/0 conn(0x7faa3c06b6d0 msgr2=0x7faa3c162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 --2- 172.21.15.97:0/3408551428 >> v2:172.21.15.135:3300/0 conn(0x7faa3c06b6d0 0x7faa3c162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 >> v1:172.21.15.135:6789/0 conn(0x7faa3c15dfd0 legacy=0x7faa3c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c161c60 con 0x7faa3c0affe0 2024-03-06T07:40:07.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.788+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7faa340019f0 con 0x7faa3c0affe0 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 >> v1:172.21.15.97:6789/0 conn(0x7faa3c0affe0 legacy=0x7faa3c0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa200039b0 0x7faa20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa20004800 0x7faa20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa200052e0 0x7faa2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0b0400 con 0x7faa20004800 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2000c1b0 con 0x7faa200039b0 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2000c3b0 con 0x7faa200052e0 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa32a00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa20004800 0x7faa20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3b400640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa200039b0 0x7faa20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3be00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa200052e0 0x7faa2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa32a00640 1 -- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa200052e0 msgr2=0x7faa2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa32a00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa200052e0 0x7faa2000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa32a00640 1 -- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa200039b0 msgr2=0x7faa20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa32a00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa200039b0 0x7faa20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa32a00640 1 -- 172.21.15.97:0/3408551428 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa20003520 con 0x7faa20004800 2024-03-06T07:40:07.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.789+0000 7faa3be00640 1 -- 172.21.15.97:0/3408551428 reap_dead start 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.791+0000 7faa32a00640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa20004800 0x7faa20004c80 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7faa24009a90 tx=0x7faa20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323bf856356a81af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.791+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24039070 con 0x7faa20004800 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.791+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2402c660 con 0x7faa20004800 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.791+0000 7faa3aa00640 1 -- 172.21.15.97:0/3408551428 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24009690 con 0x7faa20004800 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.791+0000 7faa41200640 1 -- 172.21.15.97:0/3408551428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa20004800 msgr2=0x7faa20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.791+0000 7faa41200640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa20004800 0x7faa20004c80 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7faa24009a90 tx=0x7faa20005830 comp rx=0 tx=0).stop 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.792+0000 7faa41200640 1 -- 172.21.15.97:0/3408551428 shutdown_connections 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.792+0000 7faa41200640 1 --2- 172.21.15.97:0/3408551428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa20004800 0x7faa20004c80 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.792+0000 7faa41200640 1 -- 172.21.15.97:0/3408551428 >> 172.21.15.97:0/3408551428 conn(0x7faa3c06c0d0 msgr2=0x7faa3c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.792+0000 7faa41200640 1 -- 172.21.15.97:0/3408551428 shutdown_connections 2024-03-06T07:40:07.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.792+0000 7faa41200640 1 -- 172.21.15.97:0/3408551428 wait complete. 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.793+0000 7faa41200640 1 Processor -- start 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.793+0000 7faa41200640 1 -- start start 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.794+0000 7faa41200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa20004800 0x7faa3c156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.794+0000 7faa41200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 0x7faa3c0b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.794+0000 7faa41200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa3c0b2b80 0x7faa3c145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.794+0000 7faa41200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c161f00 con 0x7faa3c142b30 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.794+0000 7faa41200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c162270 con 0x7faa3c0b2b80 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.794+0000 7faa41200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0ce8d0 con 0x7faa20004800 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 0x7faa3c0b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 0x7faa3c0b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55584/0 (socket says 172.21.15.97:55584) 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 -- 172.21.15.97:0/1991658004 learned_addr learned my addr 172.21.15.97:0/1991658004 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 -- 172.21.15.97:0/1991658004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa20004800 msgr2=0x7faa3c156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa20004800 0x7faa3c156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 -- 172.21.15.97:0/1991658004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa3c0b2b80 msgr2=0x7faa3c145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa3c0b2b80 0x7faa3c145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 -- 172.21.15.97:0/1991658004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c1560b0 con 0x7faa3c142b30 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa32a00640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 0x7faa3c0b2640 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7faa24037040 tx=0x7faa24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.795+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24039050 con 0x7faa3c142b30 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.796+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa24045070 con 0x7faa3c142b30 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.796+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa240044e0 con 0x7faa3c142b30 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.796+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3c156320 con 0x7faa3c142b30 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.796+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3c1462e0 con 0x7faa3c142b30 2024-03-06T07:40:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.797+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3c0b2fd0 con 0x7faa3c142b30 2024-03-06T07:40:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.799+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa24003830 con 0x7faa3c142b30 2024-03-06T07:40:07.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.799+0000 7faa33e00640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa1806b4d0 0x7faa1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.800+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faa240abdb0 con 0x7faa3c142b30 2024-03-06T07:40:07.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.800+0000 7faa3b400640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa1806b4d0 0x7faa1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.800+0000 7faa3b400640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa1806b4d0 0x7faa1806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7faa2c007fd0 tx=0x7faa2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:07.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:07.805+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa2404e020 con 0x7faa3c142b30 2024-03-06T07:40:08.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.051+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa3c04f330 con 0x7faa3c142b30 2024-03-06T07:40:08.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.052+0000 7faa33e00640 1 -- 172.21.15.97:0/1991658004 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7faa2407ac00 con 0x7faa3c142b30 2024-03-06T07:40:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:08.054 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:08.054 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa1806b4d0 msgr2=0x7faa1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa1806b4d0 0x7faa1806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7faa2c007fd0 tx=0x7faa2c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 msgr2=0x7faa3c0b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 0x7faa3c0b2640 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7faa24037040 tx=0x7faa24004730 comp rx=0 tx=0).stop 2024-03-06T07:40:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 shutdown_connections 2024-03-06T07:40:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7faa1806b4d0 0x7faa1806d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa3c0b2b80 0x7faa3c145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa3c142b30 0x7faa3c0b2640 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 --2- 172.21.15.97:0/1991658004 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faa20004800 0x7faa3c156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.055+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 >> 172.21.15.97:0/1991658004 conn(0x7faa3c06c0d0 msgr2=0x7faa3c06edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:08.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.056+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 shutdown_connections 2024-03-06T07:40:08.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:08.056+0000 7faa41200640 1 -- 172.21.15.97:0/1991658004 wait complete. 2024-03-06T07:40:08.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:08.278 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:09.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:09.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.492+0000 7fd17ac00640 1 Processor -- start 2024-03-06T07:40:09.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.492+0000 7fd17ac00640 1 -- start start 2024-03-06T07:40:09.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.494+0000 7fd17ac00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd1740d0fc0 0x7fd174160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.494+0000 7fd17ac00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740cea60 con 0x7fd17415e0d0 2024-03-06T07:40:09.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.494+0000 7fd17ac00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740d2350 con 0x7fd17415f470 2024-03-06T07:40:09.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.494+0000 7fd17ac00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1740d2f60 con 0x7fd1740d0fc0 2024-03-06T07:40:09.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd178c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd1740d0fc0 0x7fd174160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd178c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd1740d0fc0 0x7fd174160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55592/0 (socket says 172.21.15.97:55592) 2024-03-06T07:40:09.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd173e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd17415e0d0 0x7fd1740d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53708/0 (socket says 172.21.15.97:53708) 2024-03-06T07:40:09.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd178c00640 1 -- 172.21.15.97:0/3578321140 learned_addr learned my addr 172.21.15.97:0/3578321140 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:09.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd178c00640 1 -- 172.21.15.97:0/3578321140 >> v1:172.21.15.135:6789/0 conn(0x7fd17415f470 legacy=0x7fd1740d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:09.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd178c00640 1 -- 172.21.15.97:0/3578321140 >> v1:172.21.15.97:6789/0 conn(0x7fd17415e0d0 legacy=0x7fd1740d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.495+0000 7fd178c00640 1 -- 172.21.15.97:0/3578321140 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740d2630 con 0x7fd1740d0fc0 2024-03-06T07:40:09.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.496+0000 7fd178c00640 1 --2- 172.21.15.97:0/3578321140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1740d0fc0 0x7fd174160dc0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fd1740caf20 tx=0x7fd16c02bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275ec3b7df8c2539 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:09.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd173400640 1 -- 172.21.15.97:0/3578321140 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c02cb60 con 0x7fd1740d0fc0 2024-03-06T07:40:09.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd173400640 1 -- 172.21.15.97:0/3578321140 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c02cd00 con 0x7fd1740d0fc0 2024-03-06T07:40:09.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd173400640 1 -- 172.21.15.97:0/3578321140 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c034610 con 0x7fd1740d0fc0 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 -- 172.21.15.97:0/3578321140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1740d0fc0 msgr2=0x7fd174160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3578321140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1740d0fc0 0x7fd174160dc0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fd1740caf20 tx=0x7fd16c02bfc0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 -- 172.21.15.97:0/3578321140 shutdown_connections 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3578321140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1740d0fc0 0x7fd174160dc0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 -- 172.21.15.97:0/3578321140 >> 172.21.15.97:0/3578321140 conn(0x7fd1740d3ec0 msgr2=0x7fd1740d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 -- 172.21.15.97:0/3578321140 shutdown_connections 2024-03-06T07:40:09.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.497+0000 7fd17ac00640 1 -- 172.21.15.97:0/3578321140 wait complete. 2024-03-06T07:40:09.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.499+0000 7fd17ac00640 1 Processor -- start 2024-03-06T07:40:09.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.499+0000 7fd17ac00640 1 -- start start 2024-03-06T07:40:09.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd17ac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1740d0fc0 0x7fd17414f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd17ac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 0x7fd1740cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd17ac00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd17415f470 0x7fd1740c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd17ac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740cff10 con 0x7fd17415e0d0 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd17ac00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740d2bf0 con 0x7fd17415f470 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd17ac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17415fb30 con 0x7fd1740d0fc0 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd16b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 0x7fd1740cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd16b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 0x7fd1740cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55604/0 (socket says 172.21.15.97:55604) 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.500+0000 7fd16b400640 1 -- 172.21.15.97:0/3950800266 learned_addr learned my addr 172.21.15.97:0/3950800266 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd173e00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1740d0fc0 0x7fd17414f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd178c00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd17415f470 0x7fd1740c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd16b400640 1 -- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1740d0fc0 msgr2=0x7fd17414f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd16b400640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1740d0fc0 0x7fd17414f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd16b400640 1 -- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd17415f470 msgr2=0x7fd1740c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd16b400640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd17415f470 0x7fd1740c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd16b400640 1 -- 172.21.15.97:0/3950800266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740ccfa0 con 0x7fd17415e0d0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd16b400640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 0x7fd1740cc4e0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00b420 tx=0x7fd15c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c011830 con 0x7fd17415e0d0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c016070 con 0x7fd17415e0d0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1740cd280 con 0x7fd17415e0d0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c011c10 con 0x7fd17415e0d0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.501+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1740cd840 con 0x7fd17415e0d0 2024-03-06T07:40:09.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.503+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1740d36b0 con 0x7fd17415e0d0 2024-03-06T07:40:09.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.505+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd15c01b460 con 0x7fd17415e0d0 2024-03-06T07:40:09.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.506+0000 7fd170c00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd15006b4d0 0x7fd15006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.506+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd15c089d20 con 0x7fd17415e0d0 2024-03-06T07:40:09.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.506+0000 7fd173e00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd15006b4d0 0x7fd15006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.507+0000 7fd173e00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd15006b4d0 0x7fd15006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd164000f60 tx=0x7fd164009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:09.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.510+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1740d36b0 con 0x7fd17415e0d0 2024-03-06T07:40:09.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.744+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1740ca790 con 0x7fd17415e0d0 2024-03-06T07:40:09.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.745+0000 7fd170c00640 1 -- 172.21.15.97:0/3950800266 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fd1740ca790 con 0x7fd17415e0d0 2024-03-06T07:40:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:09.747 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:09.748 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:09.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.747+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd15006b4d0 msgr2=0x7fd15006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.747+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd15006b4d0 0x7fd15006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd164000f60 tx=0x7fd164009040 comp rx=0 tx=0).stop 2024-03-06T07:40:09.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.747+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 msgr2=0x7fd1740cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.747+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 0x7fd1740cc4e0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00b420 tx=0x7fd15c00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 shutdown_connections 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd15006b4d0 0x7fd15006d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd17415f470 0x7fd1740c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd17415e0d0 0x7fd1740cc4e0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 --2- 172.21.15.97:0/3950800266 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1740d0fc0 0x7fd17414f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 >> 172.21.15.97:0/3950800266 conn(0x7fd1740d3ec0 msgr2=0x7fd1740b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 shutdown_connections 2024-03-06T07:40:09.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:09.748+0000 7fd17ac00640 1 -- 172.21.15.97:0/3950800266 wait complete. 2024-03-06T07:40:09.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:09.973 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:10.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:10.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:10.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:11.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.152+0000 7fe0cba00640 1 Processor -- start 2024-03-06T07:40:11.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.152+0000 7fe0cba00640 1 -- start start 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.154+0000 7fe0cba00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe0c40b4680 0x7fe0c406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.154+0000 7fe0cba00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c415b090 con 0x7fe0c415acb0 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.154+0000 7fe0cba00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40b2a30 con 0x7fe0c4072340 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.154+0000 7fe0cba00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c40b4360 con 0x7fe0c40b4680 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0bbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe0c40b4680 0x7fe0c406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0bbe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe0c40b4680 0x7fe0c406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55606/0 (socket says 172.21.15.97:55606) 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0c9000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe0c415acb0 0x7fe0c40b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53714/0 (socket says 172.21.15.97:53714) 2024-03-06T07:40:11.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0bbe00640 1 -- 172.21.15.97:0/2839624438 learned_addr learned my addr 172.21.15.97:0/2839624438 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:11.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0bbe00640 1 -- 172.21.15.97:0/2839624438 >> v1:172.21.15.135:6789/0 conn(0x7fe0c4072340 legacy=0x7fe0c40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:11.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0bbe00640 1 -- 172.21.15.97:0/2839624438 >> v1:172.21.15.97:6789/0 conn(0x7fe0c415acb0 legacy=0x7fe0c40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.155+0000 7fe0bbe00640 1 -- 172.21.15.97:0/2839624438 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40b2cf0 con 0x7fe0c40b4680 2024-03-06T07:40:11.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.156+0000 7fe0bbe00640 1 --2- 172.21.15.97:0/2839624438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c40b4680 0x7fe0c406e4f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009870 tx=0x7fe0ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8686340c8f6fc6a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:11.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.156+0000 7fe0c3e00640 1 -- 172.21.15.97:0/2839624438 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac034070 con 0x7fe0c40b4680 2024-03-06T07:40:11.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0c3e00640 1 -- 172.21.15.97:0/2839624438 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac030860 con 0x7fe0c40b4680 2024-03-06T07:40:11.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0c3e00640 1 -- 172.21.15.97:0/2839624438 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac030c30 con 0x7fe0c40b4680 2024-03-06T07:40:11.159 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 -- 172.21.15.97:0/2839624438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c40b4680 msgr2=0x7fe0c406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 --2- 172.21.15.97:0/2839624438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c40b4680 0x7fe0c406e4f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009870 tx=0x7fe0ac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 -- 172.21.15.97:0/2839624438 shutdown_connections 2024-03-06T07:40:11.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 --2- 172.21.15.97:0/2839624438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c40b4680 0x7fe0c406e4f0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 -- 172.21.15.97:0/2839624438 >> 172.21.15.97:0/2839624438 conn(0x7fe0c40b1730 msgr2=0x7fe0c40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:11.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 -- 172.21.15.97:0/2839624438 shutdown_connections 2024-03-06T07:40:11.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.157+0000 7fe0cba00640 1 -- 172.21.15.97:0/2839624438 wait complete. 2024-03-06T07:40:11.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.159+0000 7fe0cba00640 1 Processor -- start 2024-03-06T07:40:11.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.159+0000 7fe0cba00640 1 -- start start 2024-03-06T07:40:11.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0cba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 0x7fe0c40ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0cba00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0c40b4680 0x7fe0c41e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0cba00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0c415acb0 0x7fe0c415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0cba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40b4010 con 0x7fe0c4072340 2024-03-06T07:40:11.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0cba00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40b32b0 con 0x7fe0c415acb0 2024-03-06T07:40:11.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0cba00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c406ea30 con 0x7fe0c40b4680 2024-03-06T07:40:11.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0c9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 0x7fe0c40ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0c9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 0x7fe0c40ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55620/0 (socket says 172.21.15.97:55620) 2024-03-06T07:40:11.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0c9000640 1 -- 172.21.15.97:0/163684668 learned_addr learned my addr 172.21.15.97:0/163684668 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:11.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0bbe00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0c40b4680 0x7fe0c41e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0c9000640 1 -- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0c40b4680 msgr2=0x7fe0c41e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0c9a00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0c415acb0 0x7fe0c415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.160+0000 7fe0c9000640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0c40b4680 0x7fe0c41e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c9000640 1 -- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0c415acb0 msgr2=0x7fe0c415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c9000640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0c415acb0 0x7fe0c415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c9000640 1 -- 172.21.15.97:0/163684668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c41428b0 con 0x7fe0c4072340 2024-03-06T07:40:11.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c9a00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0c415acb0 0x7fe0c415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c9000640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 0x7fe0c40ce750 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400c3f0 tx=0x7fe0b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b40074e0 con 0x7fe0c4072340 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b4007680 con 0x7fe0c4072340 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4004a00 con 0x7fe0c4072340 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c4142b90 con 0x7fe0c4072340 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.161+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c4143150 con 0x7fe0c4072340 2024-03-06T07:40:11.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.162+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c404f330 con 0x7fe0c4072340 2024-03-06T07:40:11.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.164+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0b401a020 con 0x7fe0c4072340 2024-03-06T07:40:11.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.166+0000 7fe0c1600640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0a006b4d0 0x7fe0a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.166+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe0b4089a20 con 0x7fe0c4072340 2024-03-06T07:40:11.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.166+0000 7fe0bbe00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0a006b4d0 0x7fe0a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.167+0000 7fe0bbe00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0a006b4d0 0x7fe0a006d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac036000 tx=0x7fe0ac03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.173+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0b4010300 con 0x7fe0c4072340 2024-03-06T07:40:11.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.429+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0c406f4a0 con 0x7fe0c4072340 2024-03-06T07:40:11.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.429+0000 7fe0c1600640 1 -- 172.21.15.97:0/163684668 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fe0b40587e0 con 0x7fe0c4072340 2024-03-06T07:40:11.432 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:11.432 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:11.432 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:11.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0a006b4d0 msgr2=0x7fe0a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0a006b4d0 0x7fe0a006d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac036000 tx=0x7fe0ac03b040 comp rx=0 tx=0).stop 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 msgr2=0x7fe0c40ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 0x7fe0c40ce750 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400c3f0 tx=0x7fe0b400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 shutdown_connections 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe0a006b4d0 0x7fe0a006d990 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0c415acb0 0x7fe0c415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe0c40b4680 0x7fe0c41e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 --2- 172.21.15.97:0/163684668 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0c4072340 0x7fe0c40ce750 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 >> 172.21.15.97:0/163684668 conn(0x7fe0c40b1730 msgr2=0x7fe0c406da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 shutdown_connections 2024-03-06T07:40:11.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:11.432+0000 7fe0cba00640 1 -- 172.21.15.97:0/163684668 wait complete. 2024-03-06T07:40:11.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:11.659 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:12.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:12.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.865+0000 7f1353a00640 1 Processor -- start 2024-03-06T07:40:12.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.865+0000 7f1353a00640 1 -- start start 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.866+0000 7f1353a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f134c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.866+0000 7f1353a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f134c15f9d0 0x7f134c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.866+0000 7f1353a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c06ee70 con 0x7f134c0ce0a0 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.866+0000 7f1353a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c160370 con 0x7f134c15f9d0 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.866+0000 7f1353a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06cef0 con 0x7f134c144c00 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.867+0000 7f1351a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f134c0ce0a0 0x7f134c0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53722/0 (socket says 172.21.15.97:53722) 2024-03-06T07:40:12.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.867+0000 7f1351a00640 1 -- 172.21.15.97:0/3876687791 learned_addr learned my addr 172.21.15.97:0/3876687791 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.867+0000 7f134be00640 1 --2- 172.21.15.97:0/3876687791 >> v2:172.21.15.135:3300/0 conn(0x7f134c15f9d0 0x7f134c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.867+0000 7f1351000640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f134c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.867+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532745407 0 0) 0x7f1344001110 con 0x7f134c0ce0a0 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.867+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1330003520 con 0x7f134c0ce0a0 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2435245280 0 0) 0x7f1330003520 con 0x7f134c0ce0a0 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 msgr2=0x7f134c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f134c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 >> v2:172.21.15.135:3300/0 conn(0x7f134c15f9d0 msgr2=0x7f134c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:12.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 --2- 172.21.15.97:0/3876687791 >> v2:172.21.15.135:3300/0 conn(0x7f134c15f9d0 0x7f134c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c06d1d0 con 0x7f134c0ce0a0 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f1344001840 con 0x7f134c0ce0a0 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134be00640 1 --2- 172.21.15.97:0/3876687791 >> v2:172.21.15.135:3300/0 conn(0x7f134c15f9d0 0x7f134c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 >> v1:172.21.15.97:6789/0 conn(0x7f134c0ce0a0 legacy=0x7f134c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f13300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1330003f30 0x7f1330004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330004d40 0x7f13300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06cef0 con 0x7f1330003f30 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133000b9c0 con 0x7f1330004d40 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133000bb90 con 0x7f134c144c00 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134be00640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1330003f30 0x7f1330004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f1351000640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f13300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134be00640 1 -- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 msgr2=0x7f13300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.868+0000 7f134be00640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f13300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.869+0000 7f134be00640 1 -- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330004d40 msgr2=0x7f13300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.869+0000 7f134be00640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330004d40 0x7f13300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.869+0000 7f134be00640 1 -- 172.21.15.97:0/3876687791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1330003520 con 0x7f1330003f30 2024-03-06T07:40:12.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.869+0000 7f1351000640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c144c00 0x7f13300039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f134be00640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1330003f30 0x7f1330004800 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f13380073e0 tx=0x7f1330005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874cdc67f1034464 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338039070 con 0x7f1330003f30 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133802cdc0 con 0x7f1330003f30 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f134b400640 1 -- 172.21.15.97:0/3876687791 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338009c80 con 0x7f1330003f30 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f1353a00640 1 -- 172.21.15.97:0/3876687791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1330003f30 msgr2=0x7f1330004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f1353a00640 1 --2- 172.21.15.97:0/3876687791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1330003f30 0x7f1330004800 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f13380073e0 tx=0x7f1330005830 comp rx=0 tx=0).stop 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.870+0000 7f1351a00640 1 -- 172.21.15.97:0/3876687791 reap_dead start 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.871+0000 7f1353a00640 1 -- 172.21.15.97:0/3876687791 shutdown_connections 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.871+0000 7f1353a00640 1 -- 172.21.15.97:0/3876687791 >> 172.21.15.97:0/3876687791 conn(0x7f134c0b1730 msgr2=0x7f134c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:12.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.871+0000 7f1353a00640 1 -- 172.21.15.97:0/3876687791 shutdown_connections 2024-03-06T07:40:12.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.871+0000 7f1353a00640 1 -- 172.21.15.97:0/3876687791 wait complete. 2024-03-06T07:40:12.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.873+0000 7f1353a00640 1 Processor -- start 2024-03-06T07:40:12.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.873+0000 7f1353a00640 1 -- start start 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f1353a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330003f30 0x7f134c13ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f1353a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 0x7f134c1f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f1353a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c15ab10 0x7f134c15afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f1353a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c160730 con 0x7f134c0b2dd0 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f1353a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06d790 con 0x7f1330003f30 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f1353a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06cb50 con 0x7f134c15ab10 2024-03-06T07:40:12.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 0x7f134c1f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 0x7f134c1f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55634/0 (socket says 172.21.15.97:55634) 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 -- 172.21.15.97:0/2128551843 learned_addr learned my addr 172.21.15.97:0/2128551843 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 -- 172.21.15.97:0/2128551843 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c15ab10 msgr2=0x7f134c15afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c15ab10 0x7f134c15afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 -- 172.21.15.97:0/2128551843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330003f30 msgr2=0x7f134c13ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330003f30 0x7f134c13ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.875+0000 7f134be00640 1 -- 172.21.15.97:0/2128551843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c1f7460 con 0x7f134c0b2dd0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.876+0000 7f134be00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 0x7f134c1f6f20 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f1338037040 tx=0x7f1338004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.876+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338042020 con 0x7f134c0b2dd0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.876+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133803a040 con 0x7f134c0b2dd0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.876+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338039070 con 0x7f134c0b2dd0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.876+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c0b3270 con 0x7f134c0b2dd0 2024-03-06T07:40:12.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.876+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c0b3830 con 0x7f134c0b2dd0 2024-03-06T07:40:12.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.877+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c04f330 con 0x7f134c0b2dd0 2024-03-06T07:40:12.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.878+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1338045020 con 0x7f134c0b2dd0 2024-03-06T07:40:12.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.879+0000 7f1348c00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f132c06b500 0x7f132c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.879+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f13380ac080 con 0x7f134c0b2dd0 2024-03-06T07:40:12.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.879+0000 7f1351000640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f132c06b500 0x7f132c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.880+0000 7f1351000640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f132c06b500 0x7f132c06d9c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f133c009910 tx=0x7f133c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:12.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:12.889+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f133804c030 con 0x7f134c0b2dd0 2024-03-06T07:40:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.132+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f134c072890 con 0x7f134c0b2dd0 2024-03-06T07:40:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.132+0000 7f1348c00640 1 -- 172.21.15.97:0/2128551843 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f133807af60 con 0x7f134c0b2dd0 2024-03-06T07:40:13.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:13.135 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:13.135 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:13.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f132c06b500 msgr2=0x7f132c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f132c06b500 0x7f132c06d9c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f133c009910 tx=0x7f133c008040 comp rx=0 tx=0).stop 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 msgr2=0x7f134c1f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 0x7f134c1f6f20 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f1338037040 tx=0x7f1338004ab0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 shutdown_connections 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f132c06b500 0x7f132c06d9c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f134c15ab10 0x7f134c15afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f134c0b2dd0 0x7f134c1f6f20 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 --2- 172.21.15.97:0/2128551843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1330003f30 0x7f134c13ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 >> 172.21.15.97:0/2128551843 conn(0x7f134c0b1730 msgr2=0x7f134c145250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 shutdown_connections 2024-03-06T07:40:13.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:13.135+0000 7f1353a00640 1 -- 172.21.15.97:0/2128551843 wait complete. 2024-03-06T07:40:13.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:13.364 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:14.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:14.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.612+0000 7f3d73e00640 1 Processor -- start 2024-03-06T07:40:14.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.612+0000 7f3d73e00640 1 -- start start 2024-03-06T07:40:14.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d73e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3d740b40c0 0x7f3d740b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d73e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3d741620c0 0x7f3d740b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d73e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d74161900 con 0x7f3d740ae670 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d73e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d740b49e0 con 0x7f3d741620c0 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d73e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7415ebd0 con 0x7f3d740b40c0 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d72000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3d740ae670 0x7f3d7415e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:40996/0 (socket says 172.21.15.97:40996) 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d72000640 1 -- 172.21.15.97:0/2722663385 learned_addr learned my addr 172.21.15.97:0/2722663385 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.614+0000 7f3d73400640 1 --2- 172.21.15.97:0/2722663385 >> v2:172.21.15.97:3300/0 conn(0x7f3d741620c0 0x7f3d740b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.615+0000 7f3d72a00640 1 --2- 172.21.15.97:0/2722663385 >> v2:172.21.15.135:3300/0 conn(0x7f3d740b40c0 0x7f3d740b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.615+0000 7f3d73400640 1 -- 172.21.15.97:0/2722663385 >> v2:172.21.15.135:3300/0 conn(0x7f3d740b40c0 msgr2=0x7f3d740b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:14.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.615+0000 7f3d73400640 1 --2- 172.21.15.97:0/2722663385 >> v2:172.21.15.135:3300/0 conn(0x7f3d740b40c0 0x7f3d740b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.615+0000 7f3d73400640 1 -- 172.21.15.97:0/2722663385 >> v1:172.21.15.97:6789/0 conn(0x7f3d740ae670 legacy=0x7f3d7415e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.615+0000 7f3d71600640 1 -- 172.21.15.97:0/2722663385 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695473155 0 0) 0x7f3d74161900 con 0x7f3d740ae670 2024-03-06T07:40:14.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.615+0000 7f3d73400640 1 -- 172.21.15.97:0/2722663385 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7415eeb0 con 0x7f3d741620c0 2024-03-06T07:40:14.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.616+0000 7f3d73400640 1 --2- 172.21.15.97:0/2722663385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d741620c0 0x7f3d740b28e0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f3d740ad980 tx=0x7f3d6c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5076d0118ed272b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:14.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.616+0000 7f3d71600640 1 -- 172.21.15.97:0/2722663385 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c039070 con 0x7f3d741620c0 2024-03-06T07:40:14.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.616+0000 7f3d71600640 1 -- 172.21.15.97:0/2722663385 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c02cc70 con 0x7f3d741620c0 2024-03-06T07:40:14.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.616+0000 7f3d71600640 1 -- 172.21.15.97:0/2722663385 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c034660 con 0x7f3d741620c0 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 -- 172.21.15.97:0/2722663385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d741620c0 msgr2=0x7f3d740b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2722663385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d741620c0 0x7f3d740b28e0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f3d740ad980 tx=0x7f3d6c02bec0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 -- 172.21.15.97:0/2722663385 shutdown_connections 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2722663385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d741620c0 0x7f3d740b28e0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2722663385 >> v2:172.21.15.135:3300/0 conn(0x7f3d740b40c0 0x7f3d740b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 -- 172.21.15.97:0/2722663385 >> 172.21.15.97:0/2722663385 conn(0x7f3d7406afc0 msgr2=0x7f3d7406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 -- 172.21.15.97:0/2722663385 shutdown_connections 2024-03-06T07:40:14.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.617+0000 7f3d73e00640 1 -- 172.21.15.97:0/2722663385 wait complete. 2024-03-06T07:40:14.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.619+0000 7f3d73e00640 1 Processor -- start 2024-03-06T07:40:14.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.619+0000 7f3d73e00640 1 -- start start 2024-03-06T07:40:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d740ae670 0x7f3d7414a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 0x7f3d7415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d741620c0 0x7f3d741f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d73e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74158660 con 0x7f3d740b40c0 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d73e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74161ba0 con 0x7f3d740ae670 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d73e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741587e0 con 0x7f3d741620c0 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d72000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 0x7f3d7415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d72000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 0x7f3d7415b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34208/0 (socket says 172.21.15.97:34208) 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d72000640 1 -- 172.21.15.97:0/2495074595 learned_addr learned my addr 172.21.15.97:0/2495074595 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.620+0000 7f3d72a00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d740ae670 0x7f3d7414a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d73400640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d741620c0 0x7f3d741f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d72000640 1 -- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d741620c0 msgr2=0x7f3d741f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d72000640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d741620c0 0x7f3d741f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d72000640 1 -- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d740ae670 msgr2=0x7f3d7414a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d72000640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d740ae670 0x7f3d7414a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d72000640 1 -- 172.21.15.97:0/2495074595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7415fab0 con 0x7f3d740b40c0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d72000640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 0x7f3d7415b1e0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000c410 tx=0x7f3d6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d600074e0 con 0x7f3d740b40c0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d600047e0 con 0x7f3d740b40c0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7415fd90 con 0x7f3d740b40c0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60004bd0 con 0x7f3d740b40c0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.621+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d74160350 con 0x7f3d740b40c0 2024-03-06T07:40:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.622+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7404f330 con 0x7f3d740b40c0 2024-03-06T07:40:14.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.624+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d60011070 con 0x7f3d740b40c0 2024-03-06T07:40:14.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.625+0000 7f3d6aa00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d5406b4d0 0x7f3d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.625+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3d60089e10 con 0x7f3d740b40c0 2024-03-06T07:40:14.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.626+0000 7f3d72a00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d5406b4d0 0x7f3d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.626+0000 7f3d72a00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d5406b4d0 0x7f3d5406d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3d64006fd0 tx=0x7f3d64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:14.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.630+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d60058ce0 con 0x7f3d740b40c0 2024-03-06T07:40:14.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.865+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d740adf40 con 0x7f3d740b40c0 2024-03-06T07:40:14.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.865+0000 7f3d6aa00640 1 -- 172.21.15.97:0/2495074595 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f3d600585e0 con 0x7f3d740b40c0 2024-03-06T07:40:14.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:14.868 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:14.868 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:14.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d5406b4d0 msgr2=0x7f3d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d5406b4d0 0x7f3d5406d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3d64006fd0 tx=0x7f3d64008040 comp rx=0 tx=0).stop 2024-03-06T07:40:14.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 msgr2=0x7f3d7415b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 0x7f3d7415b1e0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000c410 tx=0x7f3d6000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 shutdown_connections 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3d5406b4d0 0x7f3d5406d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3d741620c0 0x7f3d741f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d740b40c0 0x7f3d7415b1e0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 --2- 172.21.15.97:0/2495074595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d740ae670 0x7f3d7414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 >> 172.21.15.97:0/2495074595 conn(0x7f3d7406afc0 msgr2=0x7f3d740af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 shutdown_connections 2024-03-06T07:40:14.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:14.868+0000 7f3d73e00640 1 -- 172.21.15.97:0/2495074595 wait complete. 2024-03-06T07:40:15.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:15.097 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:16.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:16.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.300+0000 7fa82be00640 1 Processor -- start 2024-03-06T07:40:16.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.300+0000 7fa82be00640 1 -- start start 2024-03-06T07:40:16.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa82be00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa824144c00 0x7fa82406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa82be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa82415f9d0 0x7fa82415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa82be00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82406ee70 con 0x7fa8240ce0a0 2024-03-06T07:40:16.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa82be00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82406cef0 con 0x7fa82415f9d0 2024-03-06T07:40:16.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa82be00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82406d1d0 con 0x7fa824144c00 2024-03-06T07:40:16.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa828a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa82415f9d0 0x7fa82415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa829e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa8240ce0a0 0x7fa8240ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41008/0 (socket says 172.21.15.97:41008) 2024-03-06T07:40:16.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.302+0000 7fa829e00640 1 -- 172.21.15.97:0/1857392895 learned_addr learned my addr 172.21.15.97:0/1857392895 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:16.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.303+0000 7fa829400640 1 --2- 172.21.15.97:0/1857392895 >> v2:172.21.15.135:3300/0 conn(0x7fa824144c00 0x7fa82406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.303+0000 7fa828a00640 1 -- 172.21.15.97:0/1857392895 >> v2:172.21.15.135:3300/0 conn(0x7fa824144c00 msgr2=0x7fa82406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:16.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.303+0000 7fa828a00640 1 --2- 172.21.15.97:0/1857392895 >> v2:172.21.15.135:3300/0 conn(0x7fa824144c00 0x7fa82406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.303+0000 7fa828a00640 1 -- 172.21.15.97:0/1857392895 >> v1:172.21.15.97:6789/0 conn(0x7fa8240ce0a0 legacy=0x7fa8240ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.303+0000 7fa828a00640 1 -- 172.21.15.97:0/1857392895 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82406d4b0 con 0x7fa82415f9d0 2024-03-06T07:40:16.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.303+0000 7fa823e00640 1 -- 172.21.15.97:0/1857392895 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371884839 0 0) 0x7fa81c001490 con 0x7fa8240ce0a0 2024-03-06T07:40:16.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.305+0000 7fa828a00640 1 --2- 172.21.15.97:0/1857392895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa82415fe60 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fa8240b25f0 tx=0x7fa81002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90135eca8f1d655d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:16.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.305+0000 7fa823e00640 1 -- 172.21.15.97:0/1857392895 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810035020 con 0x7fa82415f9d0 2024-03-06T07:40:16.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.305+0000 7fa823e00640 1 -- 172.21.15.97:0/1857392895 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810031830 con 0x7fa82415f9d0 2024-03-06T07:40:16.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.305+0000 7fa823e00640 1 -- 172.21.15.97:0/1857392895 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810031c00 con 0x7fa82415f9d0 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.305+0000 7fa82be00640 1 -- 172.21.15.97:0/1857392895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 msgr2=0x7fa82415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.305+0000 7fa82be00640 1 --2- 172.21.15.97:0/1857392895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa82415fe60 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fa8240b25f0 tx=0x7fa81002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.306+0000 7fa82be00640 1 -- 172.21.15.97:0/1857392895 shutdown_connections 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.306+0000 7fa82be00640 1 --2- 172.21.15.97:0/1857392895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa82415fe60 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.306+0000 7fa82be00640 1 --2- 172.21.15.97:0/1857392895 >> v2:172.21.15.135:3300/0 conn(0x7fa824144c00 0x7fa82406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.306+0000 7fa82be00640 1 -- 172.21.15.97:0/1857392895 >> 172.21.15.97:0/1857392895 conn(0x7fa8240b1730 msgr2=0x7fa8240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.306+0000 7fa82be00640 1 -- 172.21.15.97:0/1857392895 shutdown_connections 2024-03-06T07:40:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.306+0000 7fa82be00640 1 -- 172.21.15.97:0/1857392895 wait complete. 2024-03-06T07:40:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.307+0000 7fa82be00640 1 Processor -- start 2024-03-06T07:40:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.307+0000 7fa82be00640 1 -- start start 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.308+0000 7fa82be00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8240ce0a0 0x7fa82413d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.308+0000 7fa82be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa824144c00 0x7fa82413fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.308+0000 7fa82be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa8241e74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.308+0000 7fa82be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82406cb50 con 0x7fa82415f9d0 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.308+0000 7fa82be00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa824160730 con 0x7fa8240ce0a0 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.308+0000 7fa82be00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82406ccd0 con 0x7fa824144c00 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa8241e74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa8241e74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34218/0 (socket says 172.21.15.97:34218) 2024-03-06T07:40:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 -- 172.21.15.97:0/280179888 learned_addr learned my addr 172.21.15.97:0/280179888 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829400640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8240ce0a0 0x7fa82413d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa828a00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa824144c00 0x7fa82413fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 -- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa824144c00 msgr2=0x7fa82413fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa824144c00 0x7fa82413fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 -- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8240ce0a0 msgr2=0x7fa82413d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8240ce0a0 0x7fa82413d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa829e00640 1 -- 172.21.15.97:0/280179888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82413de00 con 0x7fa82415f9d0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.309+0000 7fa828a00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa824144c00 0x7fa82413fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.310+0000 7fa829e00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa8241e74b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00c700 tx=0x7fa81c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.310+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c018070 con 0x7fa82415f9d0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.310+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c00ce10 con 0x7fa82415f9d0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.310+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00a880 con 0x7fa82415f9d0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.310+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82413c930 con 0x7fa82415f9d0 2024-03-06T07:40:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.310+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8241402f0 con 0x7fa82415f9d0 2024-03-06T07:40:16.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.311+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82413ce50 con 0x7fa82415f9d0 2024-03-06T07:40:16.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.312+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa81c00aa20 con 0x7fa82415f9d0 2024-03-06T07:40:16.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.313+0000 7fa821600640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.313+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa81c089910 con 0x7fa82415f9d0 2024-03-06T07:40:16.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.313+0000 7fa829400640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.314+0000 7fa829400640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80406b4d0 0x7fa80406d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa818006fd0 tx=0x7fa818008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:16.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.318+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa82406ee70 con 0x7fa82415f9d0 2024-03-06T07:40:16.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.609+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa82404f330 con 0x7fa82415f9d0 2024-03-06T07:40:16.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.610+0000 7fa821600640 1 -- 172.21.15.97:0/280179888 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fa81c058640 con 0x7fa82415f9d0 2024-03-06T07:40:16.613 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:16.613 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:16.613 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80406b4d0 msgr2=0x7fa80406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80406b4d0 0x7fa80406d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa818006fd0 tx=0x7fa818008040 comp rx=0 tx=0).stop 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 msgr2=0x7fa8241e74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa8241e74b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00c700 tx=0x7fa81c00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 shutdown_connections 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa82415f9d0 0x7fa8241e74b0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa824144c00 0x7fa82413fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 --2- 172.21.15.97:0/280179888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8240ce0a0 0x7fa82413d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 >> 172.21.15.97:0/280179888 conn(0x7fa8240b1730 msgr2=0x7fa8240affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:16.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 shutdown_connections 2024-03-06T07:40:16.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:16.613+0000 7fa82be00640 1 -- 172.21.15.97:0/280179888 wait complete. 2024-03-06T07:40:16.833 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:16.833 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:17.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:18.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.057+0000 7fd151200640 1 Processor -- start 2024-03-06T07:40:18.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.057+0000 7fd151200640 1 -- start start 2024-03-06T07:40:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.059+0000 7fd151200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd14c15f8d0 0x7fd14c0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.059+0000 7fd151200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd14c0ccc40 0x7fd14c15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.059+0000 7fd151200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14c070de0 con 0x7fd14c15f4f0 2024-03-06T07:40:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.059+0000 7fd151200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14c0cd0a0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.059+0000 7fd151200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14c160a20 con 0x7fd14c0ccc40 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd14c15f8d0 0x7fd14c0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd14c15f8d0 0x7fd14c0cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34222/0 (socket says 172.21.15.97:34222) 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 -- 172.21.15.97:0/3095579825 learned_addr learned my addr 172.21.15.97:0/3095579825 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd14a800640 1 --2- 172.21.15.97:0/3095579825 >> v2:172.21.15.135:3300/0 conn(0x7fd14c0ccc40 0x7fd14c15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 -- 172.21.15.97:0/3095579825 >> v2:172.21.15.135:3300/0 conn(0x7fd14c0ccc40 msgr2=0x7fd14c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 --2- 172.21.15.97:0/3095579825 >> v2:172.21.15.135:3300/0 conn(0x7fd14c0ccc40 0x7fd14c15fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 -- 172.21.15.97:0/3095579825 >> v1:172.21.15.135:6789/0 conn(0x7fd14c15f4f0 legacy=0x7fd14c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.060+0000 7fd149400640 1 -- 172.21.15.97:0/3095579825 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c06ef50 con 0x7fd14c15f8d0 2024-03-06T07:40:18.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.062+0000 7fd149400640 1 --2- 172.21.15.97:0/3095579825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c0cc700 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b1420 tx=0x7fd13802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904d15f128850a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:18.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.062+0000 7fd148a00640 1 -- 172.21.15.97:0/3095579825 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138035020 con 0x7fd14c15f8d0 2024-03-06T07:40:18.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.062+0000 7fd148a00640 1 -- 172.21.15.97:0/3095579825 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138031870 con 0x7fd14c15f8d0 2024-03-06T07:40:18.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.062+0000 7fd148a00640 1 -- 172.21.15.97:0/3095579825 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138031c40 con 0x7fd14c15f8d0 2024-03-06T07:40:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 -- 172.21.15.97:0/3095579825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 msgr2=0x7fd14c0cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 --2- 172.21.15.97:0/3095579825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c0cc700 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b1420 tx=0x7fd13802b910 comp rx=0 tx=0).stop 2024-03-06T07:40:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 -- 172.21.15.97:0/3095579825 shutdown_connections 2024-03-06T07:40:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 --2- 172.21.15.97:0/3095579825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c0cc700 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 --2- 172.21.15.97:0/3095579825 >> v2:172.21.15.135:3300/0 conn(0x7fd14c0ccc40 0x7fd14c15fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 -- 172.21.15.97:0/3095579825 >> 172.21.15.97:0/3095579825 conn(0x7fd14c0b3800 msgr2=0x7fd14c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:18.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 -- 172.21.15.97:0/3095579825 shutdown_connections 2024-03-06T07:40:18.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.063+0000 7fd151200640 1 -- 172.21.15.97:0/3095579825 wait complete. 2024-03-06T07:40:18.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.065+0000 7fd151200640 1 Processor -- start 2024-03-06T07:40:18.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.065+0000 7fd151200640 1 -- start start 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd151200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd14c0ccc40 0x7fd14c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd151200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd14c15f4f0 0x7fd14c1f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd151200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c1f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd151200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c160410 con 0x7fd14c15f8d0 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd151200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c160290 con 0x7fd14c15f4f0 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd151200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c160590 con 0x7fd14c0ccc40 2024-03-06T07:40:18.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd14a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c1f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd14a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c1f4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34226/0 (socket says 172.21.15.97:34226) 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd14a800640 1 -- 172.21.15.97:0/3828434224 learned_addr learned my addr 172.21.15.97:0/3828434224 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd14a800640 1 -- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd14c0ccc40 msgr2=0x7fd14c0b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd149e00640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd14c0ccc40 0x7fd14c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.066+0000 7fd149400640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd14c15f4f0 0x7fd14c1f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd14a800640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd14c0ccc40 0x7fd14c0b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd14a800640 1 -- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd14c15f4f0 msgr2=0x7fd14c1f4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd14a800640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd14c15f4f0 0x7fd14c1f4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd14a800640 1 -- 172.21.15.97:0/3828434224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c06cdc0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd149400640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd14c15f4f0 0x7fd14c1f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd149e00640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd14c0ccc40 0x7fd14c0b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd14a800640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c1f4b50 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fd144007e60 tx=0x7fd14400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144014420 con 0x7fd14c15f8d0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd144019070 con 0x7fd14c15f8d0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144014820 con 0x7fd14c15f8d0 2024-03-06T07:40:18.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c06d0a0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.067+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c06d660 con 0x7fd14c15f8d0 2024-03-06T07:40:18.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.068+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c0cc9d0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.070+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1440149c0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.071+0000 7fd142000640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12c06b4d0 0x7fd12c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.071+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd14408a9f0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.072+0000 7fd149e00640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12c06b4d0 0x7fd12c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.072+0000 7fd149e00640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12c06b4d0 0x7fd12c06d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b2fd0 tx=0x7fd13c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:18.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.077+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd14c0cc9d0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.310+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd14c04f330 con 0x7fd14c15f8d0 2024-03-06T07:40:18.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.311+0000 7fd142000640 1 -- 172.21.15.97:0/3828434224 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fd1440598e0 con 0x7fd14c15f8d0 2024-03-06T07:40:18.314 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:18.314 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:18.314 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:18.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12c06b4d0 msgr2=0x7fd12c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12c06b4d0 0x7fd12c06d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b2fd0 tx=0x7fd13c008040 comp rx=0 tx=0).stop 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 msgr2=0x7fd14c1f4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c1f4b50 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fd144007e60 tx=0x7fd14400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 shutdown_connections 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd12c06b4d0 0x7fd12c06d990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14c15f8d0 0x7fd14c1f4b50 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd14c15f4f0 0x7fd14c1f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 --2- 172.21.15.97:0/3828434224 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd14c0ccc40 0x7fd14c0b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 >> 172.21.15.97:0/3828434224 conn(0x7fd14c0b3800 msgr2=0x7fd14c06f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 shutdown_connections 2024-03-06T07:40:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:18.314+0000 7fd151200640 1 -- 172.21.15.97:0/3828434224 wait complete. 2024-03-06T07:40:18.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:18.541 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:19.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:19.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.766+0000 7fb79aa00640 1 Processor -- start 2024-03-06T07:40:19.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.767+0000 7fb79aa00640 1 -- start start 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.768+0000 7fb79aa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb7941561f0 0x7fb7940ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.768+0000 7fb79aa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79415c240 con 0x7fb7940aebf0 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.768+0000 7fb79aa00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79415c500 con 0x7fb7940cc9d0 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.768+0000 7fb79aa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79415ce30 con 0x7fb7941561f0 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.769+0000 7fb799600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb7941561f0 0x7fb7940ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.769+0000 7fb799600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb7941561f0 0x7fb7940ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34232/0 (socket says 172.21.15.97:34232) 2024-03-06T07:40:19.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.769+0000 7fb799600640 1 -- 172.21.15.97:0/6106771 learned_addr learned my addr 172.21.15.97:0/6106771 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:19.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.769+0000 7fb799600640 1 -- 172.21.15.97:0/6106771 >> v1:172.21.15.135:6789/0 conn(0x7fb7940cc9d0 legacy=0x7fb79415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:19.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.769+0000 7fb799600640 1 -- 172.21.15.97:0/6106771 >> v1:172.21.15.97:6789/0 conn(0x7fb7940aebf0 legacy=0x7fb7940cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.769+0000 7fb799600640 1 -- 172.21.15.97:0/6106771 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79415c7e0 con 0x7fb7941561f0 2024-03-06T07:40:19.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.770+0000 7fb799600640 1 --2- 172.21.15.97:0/6106771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7940ae6b0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fb788009870 tx=0x7fb78802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8642dbd10856c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:19.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.771+0000 7fb793e00640 1 -- 172.21.15.97:0/6106771 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788034070 con 0x7fb7941561f0 2024-03-06T07:40:19.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.771+0000 7fb793e00640 1 -- 172.21.15.97:0/6106771 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb788030860 con 0x7fb7941561f0 2024-03-06T07:40:19.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.771+0000 7fb793e00640 1 -- 172.21.15.97:0/6106771 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788030c30 con 0x7fb7941561f0 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 -- 172.21.15.97:0/6106771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 msgr2=0x7fb7940ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 --2- 172.21.15.97:0/6106771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7940ae6b0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fb788009870 tx=0x7fb78802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 -- 172.21.15.97:0/6106771 shutdown_connections 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 --2- 172.21.15.97:0/6106771 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7940ae6b0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 -- 172.21.15.97:0/6106771 >> 172.21.15.97:0/6106771 conn(0x7fb79406afc0 msgr2=0x7fb79406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 -- 172.21.15.97:0/6106771 shutdown_connections 2024-03-06T07:40:19.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.772+0000 7fb79aa00640 1 -- 172.21.15.97:0/6106771 wait complete. 2024-03-06T07:40:19.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.774+0000 7fb79aa00640 1 Processor -- start 2024-03-06T07:40:19.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.774+0000 7fb79aa00640 1 -- start start 2024-03-06T07:40:19.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.775+0000 7fb79aa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7940aebf0 0x7fb794158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.775+0000 7fb79aa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7940cc9d0 0x7fb794156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.775+0000 7fb79aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7941e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.775+0000 7fb79aa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7941565d0 con 0x7fb7941561f0 2024-03-06T07:40:19.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.775+0000 7fb79aa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79406e360 con 0x7fb7940aebf0 2024-03-06T07:40:19.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.775+0000 7fb79aa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb794156750 con 0x7fb7940cc9d0 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7941e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7941e54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34248/0 (socket says 172.21.15.97:34248) 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 -- 172.21.15.97:0/194458457 learned_addr learned my addr 172.21.15.97:0/194458457 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb799600640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7940aebf0 0x7fb794158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb798c00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7940cc9d0 0x7fb794156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 -- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7940cc9d0 msgr2=0x7fb794156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7940cc9d0 0x7fb794156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 -- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7940aebf0 msgr2=0x7fb794158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7940aebf0 0x7fb794158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.776+0000 7fb79a000640 1 -- 172.21.15.97:0/194458457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb794157270 con 0x7fb7941561f0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.777+0000 7fb79a000640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7941e54f0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00c7b0 tx=0x7fb78c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.777+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c004b60 con 0x7fb7941561f0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.777+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c004d00 con 0x7fb7941561f0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.777+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb794157550 con 0x7fb7941561f0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.777+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c00aa90 con 0x7fb7941561f0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.777+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb794157b10 con 0x7fb7941561f0 2024-03-06T07:40:19.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.778+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79404f330 con 0x7fb7941561f0 2024-03-06T07:40:19.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.779+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb78c00ac30 con 0x7fb7941561f0 2024-03-06T07:40:19.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.780+0000 7fb791600640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb77406b4d0 0x7fb77406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.780+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb78c08a4b0 con 0x7fb7941561f0 2024-03-06T07:40:19.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.780+0000 7fb799600640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb77406b4d0 0x7fb77406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.781+0000 7fb799600640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb77406b4d0 0x7fb77406d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb788036000 tx=0x7fb78803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:19.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:19.789+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb78c020070 con 0x7fb7941561f0 2024-03-06T07:40:20.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.046+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7940b2b10 con 0x7fb7941561f0 2024-03-06T07:40:20.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.046+0000 7fb791600640 1 -- 172.21.15.97:0/194458457 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fb78c059300 con 0x7fb7941561f0 2024-03-06T07:40:20.049 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:20.049 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:20.049 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:20.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb77406b4d0 msgr2=0x7fb77406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:20.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb77406b4d0 0x7fb77406d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb788036000 tx=0x7fb78803b040 comp rx=0 tx=0).stop 2024-03-06T07:40:20.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 msgr2=0x7fb7941e54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:20.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7941e54f0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00c7b0 tx=0x7fb78c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:40:20.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 shutdown_connections 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb77406b4d0 0x7fb77406d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7941561f0 0x7fb7941e54f0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb7940cc9d0 0x7fb794156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 --2- 172.21.15.97:0/194458457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7940aebf0 0x7fb794158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 >> 172.21.15.97:0/194458457 conn(0x7fb79406afc0 msgr2=0x7fb79415a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 shutdown_connections 2024-03-06T07:40:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:20.049+0000 7fb79aa00640 1 -- 172.21.15.97:0/194458457 wait complete. 2024-03-06T07:40:20.270 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:20.270 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:21.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:21.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.487+0000 7f1dace00640 1 Processor -- start 2024-03-06T07:40:21.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.487+0000 7f1dace00640 1 -- start start 2024-03-06T07:40:21.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.488+0000 7f1dace00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da80d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.488+0000 7f1dace00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1da80d1e90 0x7f1da80d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.488+0000 7f1dace00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da8161960 con 0x7f1da80d28a0 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.489+0000 7f1dace00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da80cf6f0 con 0x7f1da80d1e90 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.489+0000 7f1dace00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80cf9d0 con 0x7f1da8161580 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.489+0000 7f1da7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da80d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.489+0000 7f1d9ea00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1da80d1e90 0x7f1da80d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.489+0000 7f1da7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da80d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:59796/0 (socket says 172.21.15.97:59796) 2024-03-06T07:40:21.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.489+0000 7f1da7400640 1 -- 172.21.15.97:0/264181571 learned_addr learned my addr 172.21.15.97:0/264181571 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:21.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.490+0000 7f1da7400640 1 -- 172.21.15.97:0/264181571 >> v2:172.21.15.135:3300/0 conn(0x7f1da80d1e90 msgr2=0x7f1da80d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:21.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.490+0000 7f1da7400640 1 --2- 172.21.15.97:0/264181571 >> v2:172.21.15.135:3300/0 conn(0x7f1da80d1e90 0x7f1da80d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.490+0000 7f1da7400640 1 -- 172.21.15.97:0/264181571 >> v1:172.21.15.135:6789/0 conn(0x7f1da80d28a0 legacy=0x7f1da80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.490+0000 7f1da7400640 1 -- 172.21.15.97:0/264181571 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da80cfcb0 con 0x7f1da8161580 2024-03-06T07:40:21.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.490+0000 7f1d9ea00640 1 --2- 172.21.15.97:0/264181571 >> v2:172.21.15.135:3300/0 conn(0x7f1da80d1e90 0x7f1da80d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:40:21.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.491+0000 7f1da7400640 1 --2- 172.21.15.97:0/264181571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da80d1950 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009e50 tx=0x7f1d9802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6939e8cf0ce4428 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.492+0000 7f1da6a00640 1 -- 172.21.15.97:0/264181571 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9802cb00 con 0x7f1da8161580 2024-03-06T07:40:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.492+0000 7f1da6a00640 1 -- 172.21.15.97:0/264181571 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9802cca0 con 0x7f1da8161580 2024-03-06T07:40:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.492+0000 7f1da6a00640 1 -- 172.21.15.97:0/264181571 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98034610 con 0x7f1da8161580 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.492+0000 7f1dace00640 1 -- 172.21.15.97:0/264181571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 msgr2=0x7f1da80d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.492+0000 7f1dace00640 1 --2- 172.21.15.97:0/264181571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da80d1950 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009e50 tx=0x7f1d9802bf60 comp rx=0 tx=0).stop 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.493+0000 7f1dace00640 1 -- 172.21.15.97:0/264181571 shutdown_connections 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.493+0000 7f1dace00640 1 --2- 172.21.15.97:0/264181571 >> v2:172.21.15.135:3300/0 conn(0x7f1da80d1e90 0x7f1da80d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.493+0000 7f1dace00640 1 --2- 172.21.15.97:0/264181571 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da80d1950 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.493+0000 7f1dace00640 1 -- 172.21.15.97:0/264181571 >> 172.21.15.97:0/264181571 conn(0x7f1da80d3d80 msgr2=0x7f1da80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.493+0000 7f1dace00640 1 -- 172.21.15.97:0/264181571 shutdown_connections 2024-03-06T07:40:21.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.493+0000 7f1dace00640 1 -- 172.21.15.97:0/264181571 wait complete. 2024-03-06T07:40:21.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.494+0000 7f1dace00640 1 Processor -- start 2024-03-06T07:40:21.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.494+0000 7f1dace00640 1 -- start start 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.495+0000 7f1dace00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1da80d1e90 0x7f1da80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.495+0000 7f1dace00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 0x7f1da80cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.495+0000 7f1dace00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da814b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.495+0000 7f1dace00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80cf3a0 con 0x7f1da80d28a0 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.495+0000 7f1dace00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80b39a0 con 0x7f1da80d1e90 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.495+0000 7f1dace00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80b4ed0 con 0x7f1da8161580 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 0x7f1da80cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 0x7f1da80cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34252/0 (socket says 172.21.15.97:34252) 2024-03-06T07:40:21.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 -- 172.21.15.97:0/3210424785 learned_addr learned my addr 172.21.15.97:0/3210424785 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:21.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1da7e00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da814b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1da7400640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1da80d1e90 0x7f1da80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 -- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 msgr2=0x7f1da814b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da814b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 -- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1da80d1e90 msgr2=0x7f1da80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1da80d1e90 0x7f1da80cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.496+0000 7f1d9ea00640 1 -- 172.21.15.97:0/3210424785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da80b7d20 con 0x7f1da80d28a0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.497+0000 7f1d9ea00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 0x7f1da80cc680 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000ebe0 tx=0x7f1d9000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.497+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90017070 con 0x7f1da80d28a0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.497+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9000cc30 con 0x7f1da80d28a0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.497+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90013470 con 0x7f1da80d28a0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.497+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da80ccbc0 con 0x7f1da80d28a0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.497+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da80cd170 con 0x7f1da80d28a0 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.498+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da804f330 con 0x7f1da80d28a0 2024-03-06T07:40:21.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.499+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d9001a070 con 0x7f1da80d28a0 2024-03-06T07:40:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.500+0000 7f1d9fe00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d8406b4d0 0x7f1d8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.500+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1d9008a120 con 0x7f1da80d28a0 2024-03-06T07:40:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.500+0000 7f1da7400640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d8406b4d0 0x7f1d8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.501+0000 7f1da7400640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d8406b4d0 0x7f1d8406d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1d98035000 tx=0x7f1d98030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.504+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d90059010 con 0x7f1da80d28a0 2024-03-06T07:40:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.740+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1da80d3b60 con 0x7f1da80d28a0 2024-03-06T07:40:21.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.741+0000 7f1d9fe00640 1 -- 172.21.15.97:0/3210424785 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f1d90058910 con 0x7f1da80d28a0 2024-03-06T07:40:21.743 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:21.743 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:21.743 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.743+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d8406b4d0 msgr2=0x7f1d8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.743+0000 7f1dace00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d8406b4d0 0x7f1d8406d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1d98035000 tx=0x7f1d98030000 comp rx=0 tx=0).stop 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.743+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 msgr2=0x7f1da80cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.743+0000 7f1dace00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 0x7f1da80cc680 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000ebe0 tx=0x7f1d9000c520 comp rx=0 tx=0).stop 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 shutdown_connections 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f1d8406b4d0 0x7f1d8406d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f1da8161580 0x7f1da814b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1da80d28a0 0x7f1da80cc680 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 --2- 172.21.15.97:0/3210424785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1da80d1e90 0x7f1da80cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 >> 172.21.15.97:0/3210424785 conn(0x7f1da80d3d80 msgr2=0x7f1da815e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:21.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 shutdown_connections 2024-03-06T07:40:21.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:21.744+0000 7f1dace00640 1 -- 172.21.15.97:0/3210424785 wait complete. 2024-03-06T07:40:21.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:21.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:22.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:23.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.189+0000 7f5571e00640 1 Processor -- start 2024-03-06T07:40:23.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.189+0000 7f5571e00640 1 -- start start 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f556c15dfd0 0x7f556c162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c0affe0 0x7f556c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f556c06b500 0x7f556c0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c162270 con 0x7f556c06b500 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0cef70 con 0x7f556c15dfd0 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c15e3b0 con 0x7f556c0affe0 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f556c06b500 0x7f556c0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f556c06b500 0x7f556c0d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34260/0 (socket says 172.21.15.97:34260) 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5571400640 1 -- 172.21.15.97:0/373599518 learned_addr learned my addr 172.21.15.97:0/373599518 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f556be00640 1 --2- 172.21.15.97:0/373599518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c0affe0 0x7f556c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.191+0000 7f5570a00640 1 --2- 172.21.15.97:0/373599518 >> v2:172.21.15.135:3300/0 conn(0x7f556c15dfd0 0x7f556c162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.192+0000 7f5571400640 1 -- 172.21.15.97:0/373599518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c0affe0 msgr2=0x7f556c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:23.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.192+0000 7f5571400640 1 --2- 172.21.15.97:0/373599518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c0affe0 0x7f556c06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.192+0000 7f5571400640 1 -- 172.21.15.97:0/373599518 >> v2:172.21.15.135:3300/0 conn(0x7f556c15dfd0 msgr2=0x7f556c162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:23.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.192+0000 7f5571400640 1 --2- 172.21.15.97:0/373599518 >> v2:172.21.15.135:3300/0 conn(0x7f556c15dfd0 0x7f556c162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.192+0000 7f5571400640 1 -- 172.21.15.97:0/373599518 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c161c60 con 0x7f556c06b500 2024-03-06T07:40:23.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.193+0000 7f5571400640 1 --2- 172.21.15.97:0/373599518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c06b500 0x7f556c0d06f0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f556c0d1a70 tx=0x7f556402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b823de5f98a89b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:23.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f556b400640 1 -- 172.21.15.97:0/373599518 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556402c950 con 0x7f556c06b500 2024-03-06T07:40:23.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f556b400640 1 -- 172.21.15.97:0/373599518 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556402caf0 con 0x7f556c06b500 2024-03-06T07:40:23.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f556b400640 1 -- 172.21.15.97:0/373599518 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55640343f0 con 0x7f556c06b500 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f5571e00640 1 -- 172.21.15.97:0/373599518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c06b500 msgr2=0x7f556c0d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f5571e00640 1 --2- 172.21.15.97:0/373599518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c06b500 0x7f556c0d06f0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f556c0d1a70 tx=0x7f556402bdb0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f5571e00640 1 -- 172.21.15.97:0/373599518 shutdown_connections 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.194+0000 7f5571e00640 1 --2- 172.21.15.97:0/373599518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c06b500 0x7f556c0d06f0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.195+0000 7f5571e00640 1 --2- 172.21.15.97:0/373599518 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c0affe0 0x7f556c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.195+0000 7f5571e00640 1 --2- 172.21.15.97:0/373599518 >> v2:172.21.15.135:3300/0 conn(0x7f556c15dfd0 0x7f556c162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.195+0000 7f5571e00640 1 -- 172.21.15.97:0/373599518 >> 172.21.15.97:0/373599518 conn(0x7f556c06c0d0 msgr2=0x7f556c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:23.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.195+0000 7f5571e00640 1 -- 172.21.15.97:0/373599518 shutdown_connections 2024-03-06T07:40:23.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.195+0000 7f5571e00640 1 -- 172.21.15.97:0/373599518 wait complete. 2024-03-06T07:40:23.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.198+0000 7f5571e00640 1 Processor -- start 2024-03-06T07:40:23.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.198+0000 7f5571e00640 1 -- start start 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.200+0000 7f5571e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c06b500 0x7f556c160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.200+0000 7f5571e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556c0affe0 0x7f556c1e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.200+0000 7f5571e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 0x7f556c1e9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.200+0000 7f5571e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0ce8d0 con 0x7f556c15dfd0 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.200+0000 7f5571e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c161f00 con 0x7f556c0affe0 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.200+0000 7f5571e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0cea50 con 0x7f556c06b500 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 0x7f556c1e9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 0x7f556c1e9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34274/0 (socket says 172.21.15.97:34274) 2024-03-06T07:40:23.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 -- 172.21.15.97:0/3562865363 learned_addr learned my addr 172.21.15.97:0/3562865363 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:23.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f556be00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556c0affe0 0x7f556c1e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5570a00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c06b500 0x7f556c160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 -- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c06b500 msgr2=0x7f556c160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c06b500 0x7f556c160d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 -- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556c0affe0 msgr2=0x7f556c1e98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556c0affe0 0x7f556c1e98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.201+0000 7f5571400640 1 -- 172.21.15.97:0/3562865363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c15ff20 con 0x7f556c15dfd0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.202+0000 7f5571400640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 0x7f556c1e9de0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f5564036040 tx=0x7f5564004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.202+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564043020 con 0x7f556c15dfd0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.202+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556403e450 con 0x7f556c15dfd0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.202+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556403e840 con 0x7f556c15dfd0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.202+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1529f0 con 0x7f556c15dfd0 2024-03-06T07:40:23.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.202+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c15f680 con 0x7f556c15dfd0 2024-03-06T07:40:23.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.204+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c04f330 con 0x7f556c15dfd0 2024-03-06T07:40:23.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.204+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5564046020 con 0x7f556c15dfd0 2024-03-06T07:40:23.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.205+0000 7f5568c00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f554c06b4d0 0x7f554c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.205+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f55640acb50 con 0x7f556c15dfd0 2024-03-06T07:40:23.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.205+0000 7f5570a00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f554c06b4d0 0x7f554c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.206+0000 7f5570a00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f554c06b4d0 0x7f554c06d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5560006fd0 tx=0x7f5560008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:23.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.210+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f556404d030 con 0x7f556c15dfd0 2024-03-06T07:40:23.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.502+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f556c06c630 con 0x7f556c15dfd0 2024-03-06T07:40:23.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.503+0000 7f5568c00640 1 -- 172.21.15.97:0/3562865363 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f556c1529f0 con 0x7f556c15dfd0 2024-03-06T07:40:23.506 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:23.506 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:23.506 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f554c06b4d0 msgr2=0x7f554c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f554c06b4d0 0x7f554c06d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5560006fd0 tx=0x7f5560008040 comp rx=0 tx=0).stop 2024-03-06T07:40:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 msgr2=0x7f556c1e9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 0x7f556c1e9de0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f5564036040 tx=0x7f5564004a40 comp rx=0 tx=0).stop 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 shutdown_connections 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f554c06b4d0 0x7f554c06d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c15dfd0 0x7f556c1e9de0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f556c0affe0 0x7f556c1e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 --2- 172.21.15.97:0/3562865363 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f556c06b500 0x7f556c160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 >> 172.21.15.97:0/3562865363 conn(0x7f556c06c0d0 msgr2=0x7f556c159140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 shutdown_connections 2024-03-06T07:40:23.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:23.506+0000 7f5571e00640 1 -- 172.21.15.97:0/3562865363 wait complete. 2024-03-06T07:40:23.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:23.736 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:24.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:24.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.967+0000 7f1209600640 1 Processor -- start 2024-03-06T07:40:24.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.967+0000 7f1209600640 1 -- start start 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.968+0000 7f1209600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f120415acb0 0x7f12040b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.968+0000 7f1209600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12040b4680 0x7f120406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.968+0000 7f1209600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120415b090 con 0x7f1204072340 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.968+0000 7f1209600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040b2a30 con 0x7f120415acb0 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.968+0000 7f1209600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b4360 con 0x7f12040b4680 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.968+0000 7f1202a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f120415acb0 0x7f12040b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f1202a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f120415acb0 0x7f12040b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33832/0 (socket says 172.21.15.97:33832) 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f1202a00640 1 -- 172.21.15.97:0/1018856196 learned_addr learned my addr 172.21.15.97:0/1018856196 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:24.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f11fa000640 1 --2- 172.21.15.97:0/1018856196 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12040b4680 0x7f120406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:24.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f1202a00640 1 -- 172.21.15.97:0/1018856196 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12040b4680 msgr2=0x7f120406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:24.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f1202a00640 1 --2- 172.21.15.97:0/1018856196 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12040b4680 0x7f120406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:24.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f1202a00640 1 -- 172.21.15.97:0/1018856196 >> v1:172.21.15.135:6789/0 conn(0x7f1204072340 legacy=0x7f12040ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:24.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.969+0000 7f1202a00640 1 -- 172.21.15.97:0/1018856196 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040b2cf0 con 0x7f120415acb0 2024-03-06T07:40:24.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.971+0000 7f1202a00640 1 --2- 172.21.15.97:0/1018856196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f120415acb0 0x7f12040b37d0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009870 tx=0x7f11f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bffeca52e2a59e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:24.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.971+0000 7f1202000640 1 -- 172.21.15.97:0/1018856196 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4035020 con 0x7f120415acb0 2024-03-06T07:40:24.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.971+0000 7f1202000640 1 -- 172.21.15.97:0/1018856196 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4031640 con 0x7f120415acb0 2024-03-06T07:40:24.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.971+0000 7f1202000640 1 -- 172.21.15.97:0/1018856196 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f402d040 con 0x7f120415acb0 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 -- 172.21.15.97:0/1018856196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f120415acb0 msgr2=0x7f12040b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 --2- 172.21.15.97:0/1018856196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f120415acb0 0x7f12040b37d0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009870 tx=0x7f11f402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 -- 172.21.15.97:0/1018856196 shutdown_connections 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 --2- 172.21.15.97:0/1018856196 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f12040b4680 0x7f120406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 --2- 172.21.15.97:0/1018856196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f120415acb0 0x7f12040b37d0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 -- 172.21.15.97:0/1018856196 >> 172.21.15.97:0/1018856196 conn(0x7f12040b1730 msgr2=0x7f12040b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 -- 172.21.15.97:0/1018856196 shutdown_connections 2024-03-06T07:40:24.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.972+0000 7f1209600640 1 -- 172.21.15.97:0/1018856196 wait complete. 2024-03-06T07:40:24.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.974+0000 7f1209600640 1 Processor -- start 2024-03-06T07:40:24.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.974+0000 7f1209600640 1 -- start start 2024-03-06T07:40:24.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.975+0000 7f1209600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 0x7f1204141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:24.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.975+0000 7f1209600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12040b4680 0x7f12040caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:24.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.975+0000 7f1209600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f120415acb0 0x7f12040cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:24.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1209600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b3d10 con 0x7f1204072340 2024-03-06T07:40:24.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1209600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b32b0 con 0x7f12040b4680 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1209600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b4010 con 0x7f120415acb0 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 0x7f1204141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 0x7f1204141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33846/0 (socket says 172.21.15.97:33846) 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 -- 172.21.15.97:0/3158861077 learned_addr learned my addr 172.21.15.97:0/3158861077 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f11fa000640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12040b4680 0x7f12040caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1203400640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f120415acb0 0x7f12040cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 -- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f120415acb0 msgr2=0x7f12040cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f120415acb0 0x7f12040cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 -- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12040b4680 msgr2=0x7f12040caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12040b4680 0x7f12040caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1202a00640 1 -- 172.21.15.97:0/3158861077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204143db0 con 0x7f1204072340 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f11fa000640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12040b4680 0x7f12040caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:24.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.976+0000 7f1203400640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f120415acb0 0x7f12040cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.977+0000 7f1202a00640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 0x7f1204141330 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f11f402b930 tx=0x7f11f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.977+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4040070 con 0x7f1204072340 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.977+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f40399f0 con 0x7f1204072340 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.977+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4035040 con 0x7f1204072340 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.977+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1204144050 con 0x7f1204072340 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.977+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1204144610 con 0x7f1204072340 2024-03-06T07:40:24.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.979+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120404f330 con 0x7f1204072340 2024-03-06T07:40:24.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.980+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11f404b060 con 0x7f1204072340 2024-03-06T07:40:24.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.981+0000 7f11fb400640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11e006b4d0 0x7f11e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:24.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.981+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f11f40abc90 con 0x7f1204072340 2024-03-06T07:40:24.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.981+0000 7f11fa000640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11e006b4d0 0x7f11e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:24.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.982+0000 7f11fa000640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11e006b4d0 0x7f11e006d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f120415f490 tx=0x7f11ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:24.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:24.986+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11f407aa50 con 0x7f1204072340 2024-03-06T07:40:25.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.220+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120415f050 con 0x7f1204072340 2024-03-06T07:40:25.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.221+0000 7f11fb400640 1 -- 172.21.15.97:0/3158861077 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f11f407a350 con 0x7f1204072340 2024-03-06T07:40:25.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:25.223 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:25.224 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:25.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.223+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11e006b4d0 msgr2=0x7f11e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11e006b4d0 0x7f11e006d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f120415f490 tx=0x7f11ec008040 comp rx=0 tx=0).stop 2024-03-06T07:40:25.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 msgr2=0x7f1204141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 0x7f1204141330 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f11f402b930 tx=0x7f11f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 shutdown_connections 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f11e006b4d0 0x7f11e006d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f120415acb0 0x7f12040cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12040b4680 0x7f12040caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 --2- 172.21.15.97:0/3158861077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1204072340 0x7f1204141330 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 >> 172.21.15.97:0/3158861077 conn(0x7f12040b1730 msgr2=0x7f120415e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 shutdown_connections 2024-03-06T07:40:25.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:25.224+0000 7f1209600640 1 -- 172.21.15.97:0/3158861077 wait complete. 2024-03-06T07:40:25.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:25.450 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:26.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.656+0000 7f8c83a00640 1 Processor -- start 2024-03-06T07:40:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.656+0000 7f8c83a00640 1 -- start start 2024-03-06T07:40:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.658+0000 7f8c83a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8c7c15f9d0 0x7f8c7c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.658+0000 7f8c83a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c070b10 0x7f8c7c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.658+0000 7f8c83a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c06cef0 con 0x7f8c7c144c00 2024-03-06T07:40:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.658+0000 7f8c83a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c06d1d0 con 0x7f8c7c15f9d0 2024-03-06T07:40:26.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.658+0000 7f8c83a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c06d4b0 con 0x7f8c7c070b10 2024-03-06T07:40:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.659+0000 7f8c81a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c070b10 0x7f8c7c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.659+0000 7f8c73e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8c7c15f9d0 0x7f8c7c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.659+0000 7f8c73e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8c7c15f9d0 0x7f8c7c15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:33680/0 (socket says 172.21.15.97:33680) 2024-03-06T07:40:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.659+0000 7f8c73e00640 1 -- 172.21.15.97:0/3490206141 learned_addr learned my addr 172.21.15.97:0/3490206141 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.659+0000 7f8c81a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c070b10 0x7f8c7c070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:57972/0 (socket says 172.21.15.97:57972) 2024-03-06T07:40:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.660+0000 7f8c73e00640 1 -- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c070b10 msgr2=0x7f8c7c070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.660+0000 7f8c73e00640 1 --2- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c070b10 0x7f8c7c070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.660+0000 7f8c73e00640 1 -- 172.21.15.97:0/3490206141 >> v1:172.21.15.135:6789/0 conn(0x7f8c7c144c00 legacy=0x7f8c7c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.660+0000 7f8c73e00640 1 -- 172.21.15.97:0/3490206141 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c06d790 con 0x7f8c7c15f9d0 2024-03-06T07:40:26.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.661+0000 7f8c73e00640 1 --2- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c15f9d0 0x7f8c7c15fe60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009870 tx=0x7f8c6402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abea659fa3c4bc23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c7be00640 1 -- 172.21.15.97:0/3490206141 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64034070 con 0x7f8c7c15f9d0 2024-03-06T07:40:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c7be00640 1 -- 172.21.15.97:0/3490206141 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64030840 con 0x7f8c7c15f9d0 2024-03-06T07:40:26.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c7be00640 1 -- 172.21.15.97:0/3490206141 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64030c10 con 0x7f8c7c15f9d0 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c83a00640 1 -- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c15f9d0 msgr2=0x7f8c7c15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c15f9d0 0x7f8c7c15fe60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009870 tx=0x7f8c6402b980 comp rx=0 tx=0).stop 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c83a00640 1 -- 172.21.15.97:0/3490206141 shutdown_connections 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c070b10 0x7f8c7c070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3490206141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c15f9d0 0x7f8c7c15fe60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.662+0000 7f8c83a00640 1 -- 172.21.15.97:0/3490206141 >> 172.21.15.97:0/3490206141 conn(0x7f8c7c0b1730 msgr2=0x7f8c7c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.663+0000 7f8c83a00640 1 -- 172.21.15.97:0/3490206141 shutdown_connections 2024-03-06T07:40:26.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.663+0000 7f8c83a00640 1 -- 172.21.15.97:0/3490206141 wait complete. 2024-03-06T07:40:26.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.665+0000 7f8c83a00640 1 Processor -- start 2024-03-06T07:40:26.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.665+0000 7f8c83a00640 1 -- start start 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c83a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c070b10 0x7f8c7c13f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c83a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 0x7f8c7c13d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c83a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c15f9d0 0x7f8c7c0b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c83a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c06ccd0 con 0x7f8c7c144c00 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c83a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c06cb50 con 0x7f8c7c070b10 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c83a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c06df90 con 0x7f8c7c15f9d0 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c73e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 0x7f8c7c13d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c73e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 0x7f8c7c13d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33848/0 (socket says 172.21.15.97:33848) 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c73e00640 1 -- 172.21.15.97:0/3500666145 learned_addr learned my addr 172.21.15.97:0/3500666145 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c81000640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c070b10 0x7f8c7c13f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c81a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c15f9d0 0x7f8c7c0b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c73e00640 1 -- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c15f9d0 msgr2=0x7f8c7c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.667+0000 7f8c73e00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c15f9d0 0x7f8c7c0b0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c73e00640 1 -- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c070b10 msgr2=0x7f8c7c13f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c73e00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c070b10 0x7f8c7c13f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c73e00640 1 -- 172.21.15.97:0/3500666145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c13fe20 con 0x7f8c7c144c00 2024-03-06T07:40:26.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c73e00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 0x7f8c7c13d2a0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f8c64000c00 tx=0x7f8c64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64048070 con 0x7f8c7c144c00 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64030640 con 0x7f8c7c144c00 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c140100 con 0x7f8c7c144c00 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64034040 con 0x7f8c7c144c00 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.668+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c13a3d0 con 0x7f8c7c144c00 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.670+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c13a1c0 con 0x7f8c7c144c00 2024-03-06T07:40:26.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.671+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c6404b070 con 0x7f8c7c144c00 2024-03-06T07:40:26.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.672+0000 7f8c79600640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c5806b4d0 0x7f8c5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.672+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8c640ac240 con 0x7f8c7c144c00 2024-03-06T07:40:26.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.672+0000 7f8c81000640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c5806b4d0 0x7f8c5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.673+0000 7f8c81000640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c5806b4d0 0x7f8c5806d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c0015a0 tx=0x7f8c6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:26.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.676+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c6407b130 con 0x7f8c7c144c00 2024-03-06T07:40:26.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.911+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c04f330 con 0x7f8c7c144c00 2024-03-06T07:40:26.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.912+0000 7f8c79600640 1 -- 172.21.15.97:0/3500666145 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f8c6407aa30 con 0x7f8c7c144c00 2024-03-06T07:40:26.915 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:26.915 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:26.916 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:26.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c5806b4d0 msgr2=0x7f8c5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c5806b4d0 0x7f8c5806d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c0015a0 tx=0x7f8c6c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 msgr2=0x7f8c7c13d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 0x7f8c7c13d2a0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f8c64000c00 tx=0x7f8c64004730 comp rx=0 tx=0).stop 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 shutdown_connections 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c5806b4d0 0x7f8c5806d990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c7c15f9d0 0x7f8c7c0b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c144c00 0x7f8c7c13d2a0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 --2- 172.21.15.97:0/3500666145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c7c070b10 0x7f8c7c13f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 >> 172.21.15.97:0/3500666145 conn(0x7f8c7c0b1730 msgr2=0x7f8c7c0af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:26.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.915+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 shutdown_connections 2024-03-06T07:40:26.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:26.916+0000 7f8c83a00640 1 -- 172.21.15.97:0/3500666145 wait complete. 2024-03-06T07:40:27.137 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:27.137 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:28.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.367+0000 7fd376800640 1 Processor -- start 2024-03-06T07:40:28.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.367+0000 7fd376800640 1 -- start start 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.368+0000 7fd376800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3700af1c0 0x7fd37015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd376800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd37006e4d0 0x7fd37006e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd376800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370071d30 con 0x7fd370070d20 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd376800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3700711d0 con 0x7fd3700af1c0 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd376800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd370071490 con 0x7fd37006e4d0 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3700af1c0 0x7fd37015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd3700af1c0 0x7fd37015eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33852/0 (socket says 172.21.15.97:33852) 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 -- 172.21.15.97:0/1496726010 learned_addr learned my addr 172.21.15.97:0/1496726010 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36ea00640 1 --1- 172.21.15.97:0/1496726010 >> v1:172.21.15.97:6789/0 conn(0x7fd370070d20 0x7fd37006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55346/0 (socket says 172.21.15.97:55346) 2024-03-06T07:40:28.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 -- 172.21.15.97:0/1496726010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd37006e4d0 msgr2=0x7fd37006e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:28.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36fe00640 1 --2- 172.21.15.97:0/1496726010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd37006e4d0 0x7fd37006e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 --2- 172.21.15.97:0/1496726010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd37006e4d0 0x7fd37006e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 -- 172.21.15.97:0/1496726010 >> v1:172.21.15.97:6789/0 conn(0x7fd370070d20 legacy=0x7fd37006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36f400640 1 -- 172.21.15.97:0/1496726010 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd370071770 con 0x7fd3700af1c0 2024-03-06T07:40:28.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.369+0000 7fd36e000640 1 -- 172.21.15.97:0/1496726010 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29227634 0 0) 0x7fd35c001110 con 0x7fd370070d20 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.371+0000 7fd36f400640 1 --2- 172.21.15.97:0/1496726010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3700af1c0 0x7fd37015eff0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fd3600098a0 tx=0x7fd36002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6c879976c4233f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.371+0000 7fd36e000640 1 -- 172.21.15.97:0/1496726010 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360034070 con 0x7fd3700af1c0 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.371+0000 7fd36e000640 1 -- 172.21.15.97:0/1496726010 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360030840 con 0x7fd3700af1c0 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.371+0000 7fd36e000640 1 -- 172.21.15.97:0/1496726010 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360030c10 con 0x7fd3700af1c0 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.371+0000 7fd376800640 1 -- 172.21.15.97:0/1496726010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3700af1c0 msgr2=0x7fd37015eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.371+0000 7fd376800640 1 --2- 172.21.15.97:0/1496726010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3700af1c0 0x7fd37015eff0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fd3600098a0 tx=0x7fd36002b980 comp rx=0 tx=0).stop 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.372+0000 7fd376800640 1 -- 172.21.15.97:0/1496726010 shutdown_connections 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.372+0000 7fd376800640 1 --2- 172.21.15.97:0/1496726010 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd37006e4d0 0x7fd37006e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.372+0000 7fd376800640 1 --2- 172.21.15.97:0/1496726010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3700af1c0 0x7fd37015eff0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.372+0000 7fd376800640 1 -- 172.21.15.97:0/1496726010 >> 172.21.15.97:0/1496726010 conn(0x7fd3700b1730 msgr2=0x7fd3700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.372+0000 7fd376800640 1 -- 172.21.15.97:0/1496726010 shutdown_connections 2024-03-06T07:40:28.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.372+0000 7fd376800640 1 -- 172.21.15.97:0/1496726010 wait complete. 2024-03-06T07:40:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.373+0000 7fd376800640 1 Processor -- start 2024-03-06T07:40:28.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.374+0000 7fd376800640 1 -- start start 2024-03-06T07:40:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.374+0000 7fd376800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 0x7fd3701e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd376800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd370070d20 0x7fd37014a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd376800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3700af1c0 0x7fd37014ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd376800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37006d370 con 0x7fd37006e4d0 2024-03-06T07:40:28.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd376800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700b30b0 con 0x7fd370070d20 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd376800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700b35d0 con 0x7fd3700af1c0 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 0x7fd3701e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 0x7fd3701e7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33864/0 (socket says 172.21.15.97:33864) 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 -- 172.21.15.97:0/1577325175 learned_addr learned my addr 172.21.15.97:0/1577325175 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 -- 172.21.15.97:0/1577325175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3700af1c0 msgr2=0x7fd37014ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3700af1c0 0x7fd37014ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 -- 172.21.15.97:0/1577325175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd370070d20 msgr2=0x7fd37014a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd370070d20 0x7fd37014a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.375+0000 7fd36f400640 1 -- 172.21.15.97:0/1577325175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3701f41c0 con 0x7fd37006e4d0 2024-03-06T07:40:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.376+0000 7fd36f400640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 0x7fd3701e7680 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fd3600090b0 tx=0x7fd3600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.376+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360034040 con 0x7fd37006e4d0 2024-03-06T07:40:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.376+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36003ad90 con 0x7fd37006e4d0 2024-03-06T07:40:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.376+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3701e6ac0 con 0x7fd37006e4d0 2024-03-06T07:40:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.376+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360004270 con 0x7fd37006e4d0 2024-03-06T07:40:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.376+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37014b670 con 0x7fd37006e4d0 2024-03-06T07:40:28.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.377+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37004f330 con 0x7fd37006e4d0 2024-03-06T07:40:28.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.378+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd36004b060 con 0x7fd37006e4d0 2024-03-06T07:40:28.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.379+0000 7fd367400640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35006b4d0 0x7fd35006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.380+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd3600ac300 con 0x7fd37006e4d0 2024-03-06T07:40:28.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.380+0000 7fd36ea00640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35006b4d0 0x7fd35006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.380+0000 7fd36ea00640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35006b4d0 0x7fd35006d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd3700af9f0 tx=0x7fd35c001310 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:28.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.384+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3701f41c0 con 0x7fd37006e4d0 2024-03-06T07:40:28.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.642+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3700b1340 con 0x7fd37006e4d0 2024-03-06T07:40:28.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.643+0000 7fd367400640 1 -- 172.21.15.97:0/1577325175 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fd36003f070 con 0x7fd37006e4d0 2024-03-06T07:40:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:28.645 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:28.646 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.645+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35006b4d0 msgr2=0x7fd35006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.645+0000 7fd376800640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35006b4d0 0x7fd35006d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd3700af9f0 tx=0x7fd35c001310 comp rx=0 tx=0).stop 2024-03-06T07:40:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.645+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 msgr2=0x7fd3701e7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.645+0000 7fd376800640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 0x7fd3701e7680 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fd3600090b0 tx=0x7fd3600046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 shutdown_connections 2024-03-06T07:40:28.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd35006b4d0 0x7fd35006d990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3700af1c0 0x7fd37014ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd370070d20 0x7fd37014a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 --2- 172.21.15.97:0/1577325175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd37006e4d0 0x7fd3701e7680 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 >> 172.21.15.97:0/1577325175 conn(0x7fd3700b1730 msgr2=0x7fd3700b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:28.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 shutdown_connections 2024-03-06T07:40:28.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:28.646+0000 7fd376800640 1 -- 172.21.15.97:0/1577325175 wait complete. 2024-03-06T07:40:28.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:28.867 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:29.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:30.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.086+0000 7f73d1200640 1 Processor -- start 2024-03-06T07:40:30.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.086+0000 7f73d1200640 1 -- start start 2024-03-06T07:40:30.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.087+0000 7f73d1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.087+0000 7f73d1200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f73cc15f9d0 0x7f73cc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.087+0000 7f73d1200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73cc0ce0a0 0x7f73cc0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.087+0000 7f73d1200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc160370 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.087+0000 7f73d1200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc06ee70 con 0x7f73cc15f9d0 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.087+0000 7f73d1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc06cef0 con 0x7f73cc144c00 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73cc0ce0a0 0x7f73cc0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73cc0ce0a0 0x7f73cc0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33874/0 (socket says 172.21.15.97:33874) 2024-03-06T07:40:30.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 -- 172.21.15.97:0/968672716 learned_addr learned my addr 172.21.15.97:0/968672716 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73c9e00640 1 --2- 172.21.15.97:0/968672716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73c1400640 1 --2- 172.21.15.97:0/968672716 >> v2:172.21.15.135:3300/0 conn(0x7f73cc15f9d0 0x7f73cc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 -- 172.21.15.97:0/968672716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 msgr2=0x7f73cc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 --2- 172.21.15.97:0/968672716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc06e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 -- 172.21.15.97:0/968672716 >> v2:172.21.15.135:3300/0 conn(0x7f73cc15f9d0 msgr2=0x7f73cc15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 --2- 172.21.15.97:0/968672716 >> v2:172.21.15.135:3300/0 conn(0x7f73cc15f9d0 0x7f73cc15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73ca800640 1 -- 172.21.15.97:0/968672716 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc06d1d0 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.088+0000 7f73c1400640 1 --2- 172.21.15.97:0/968672716 >> v2:172.21.15.135:3300/0 conn(0x7f73cc15f9d0 0x7f73cc15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.089+0000 7f73ca800640 1 --2- 172.21.15.97:0/968672716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc0ce4f0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0b27f0 tx=0x7f73c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f706f869f94c11d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.089+0000 7f73c9400640 1 -- 172.21.15.97:0/968672716 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c402ca80 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73c9400640 1 -- 172.21.15.97:0/968672716 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c402cc20 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73c9400640 1 -- 172.21.15.97:0/968672716 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40343f0 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 -- 172.21.15.97:0/968672716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 msgr2=0x7f73cc0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 --2- 172.21.15.97:0/968672716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc0ce4f0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0b27f0 tx=0x7f73c402bee0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 -- 172.21.15.97:0/968672716 shutdown_connections 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 --2- 172.21.15.97:0/968672716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc0ce4f0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 --2- 172.21.15.97:0/968672716 >> v2:172.21.15.135:3300/0 conn(0x7f73cc15f9d0 0x7f73cc15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 --2- 172.21.15.97:0/968672716 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 -- 172.21.15.97:0/968672716 >> 172.21.15.97:0/968672716 conn(0x7f73cc0b1730 msgr2=0x7f73cc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 -- 172.21.15.97:0/968672716 shutdown_connections 2024-03-06T07:40:30.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.090+0000 7f73d1200640 1 -- 172.21.15.97:0/968672716 wait complete. 2024-03-06T07:40:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.092+0000 7f73d1200640 1 Processor -- start 2024-03-06T07:40:30.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.093+0000 7f73d1200640 1 -- start start 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73d1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc13c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73d1200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc1392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73d1200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73cc15f9d0 0x7f73cc1e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73d1200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc160730 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73d1200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc06d790 con 0x7f73cc15f9d0 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73d1200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc06cb50 con 0x7f73cc144c00 2024-03-06T07:40:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73c9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc13c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73c9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc13c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33876/0 (socket says 172.21.15.97:33876) 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.094+0000 7f73c9e00640 1 -- 172.21.15.97:0/1829331218 learned_addr learned my addr 172.21.15.97:0/1829331218 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c1400640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc1392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73ca800640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73cc15f9d0 0x7f73cc1e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c9e00640 1 -- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 msgr2=0x7f73cc1392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c9e00640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc1392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c9e00640 1 -- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73cc15f9d0 msgr2=0x7f73cc1e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c9e00640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73cc15f9d0 0x7f73cc1e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c9e00640 1 -- 172.21.15.97:0/1829331218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc1e5b00 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73ca800640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73cc15f9d0 0x7f73cc1e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:30.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c9e00640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc13c020 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00c7f0 tx=0x7f73bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc015070 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc004220 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.095+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc007620 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.096+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc1e5da0 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.096+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc1e6360 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.097+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc04f330 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.098+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73bc0079d0 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.099+0000 7f73c2a00640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.099+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f73bc08a3b0 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.099+0000 7f73c1400640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.100+0000 7f73c1400640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73a806b4d0 0x7f73a806d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f73cc15ba60 tx=0x7f73b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:30.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.103+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73bc059180 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.341+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73cc072890 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.342+0000 7f73c2a00640 1 -- 172.21.15.97:0/1829331218 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f73bc058a80 con 0x7f73cc0ce0a0 2024-03-06T07:40:30.345 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:30.345 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:30.345 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73a806b4d0 msgr2=0x7f73a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73a806b4d0 0x7f73a806d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f73cc15ba60 tx=0x7f73b4008040 comp rx=0 tx=0).stop 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 msgr2=0x7f73cc13c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc13c020 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00c7f0 tx=0x7f73bc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 shutdown_connections 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73cc15f9d0 0x7f73cc1e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f73cc144c00 0x7f73cc1392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 --2- 172.21.15.97:0/1829331218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc0ce0a0 0x7f73cc13c020 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.345+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 >> 172.21.15.97:0/1829331218 conn(0x7f73cc0b1730 msgr2=0x7f73cc070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.346+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 shutdown_connections 2024-03-06T07:40:30.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:30.346+0000 7f73d1200640 1 -- 172.21.15.97:0/1829331218 wait complete. 2024-03-06T07:40:30.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:30.574 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:31.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:31.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.785+0000 7fde35c00640 1 Processor -- start 2024-03-06T07:40:31.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.785+0000 7fde35c00640 1 -- start start 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.786+0000 7fde35c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fde30161580 0x7fde300d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.786+0000 7fde35c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fde300d1e90 0x7fde300d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.786+0000 7fde35c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde30161960 con 0x7fde300d28a0 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.786+0000 7fde35c00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde300d2d10 con 0x7fde30161580 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.786+0000 7fde35c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde300cf6f0 con 0x7fde300d1e90 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.786+0000 7fde2fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fde30161580 0x7fde300d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fde30161580 0x7fde300d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33890/0 (socket says 172.21.15.97:33890) 2024-03-06T07:40:31.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2fe00640 1 -- 172.21.15.97:0/852873334 learned_addr learned my addr 172.21.15.97:0/852873334 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2f400640 1 --2- 172.21.15.97:0/852873334 >> v2:172.21.15.135:3300/0 conn(0x7fde300d1e90 0x7fde300d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2fe00640 1 -- 172.21.15.97:0/852873334 >> v2:172.21.15.135:3300/0 conn(0x7fde300d1e90 msgr2=0x7fde300d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2fe00640 1 --2- 172.21.15.97:0/852873334 >> v2:172.21.15.135:3300/0 conn(0x7fde300d1e90 0x7fde300d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2fe00640 1 -- 172.21.15.97:0/852873334 >> v1:172.21.15.135:6789/0 conn(0x7fde300d28a0 legacy=0x7fde300d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.787+0000 7fde2fe00640 1 -- 172.21.15.97:0/852873334 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde300cf9d0 con 0x7fde30161580 2024-03-06T07:40:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.788+0000 7fde2fe00640 1 --2- 172.21.15.97:0/852873334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde300d1950 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fde3008c9d0 tx=0x7fde2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a84f41f135bc0ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.788+0000 7fde2ea00640 1 -- 172.21.15.97:0/852873334 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2002ca80 con 0x7fde30161580 2024-03-06T07:40:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.788+0000 7fde2ea00640 1 -- 172.21.15.97:0/852873334 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2002cc20 con 0x7fde30161580 2024-03-06T07:40:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.788+0000 7fde2ea00640 1 -- 172.21.15.97:0/852873334 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde200343f0 con 0x7fde30161580 2024-03-06T07:40:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 -- 172.21.15.97:0/852873334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 msgr2=0x7fde300d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 --2- 172.21.15.97:0/852873334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde300d1950 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fde3008c9d0 tx=0x7fde2002bee0 comp rx=0 tx=0).stop 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 -- 172.21.15.97:0/852873334 shutdown_connections 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 --2- 172.21.15.97:0/852873334 >> v2:172.21.15.135:3300/0 conn(0x7fde300d1e90 0x7fde300d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 --2- 172.21.15.97:0/852873334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde300d1950 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 -- 172.21.15.97:0/852873334 >> 172.21.15.97:0/852873334 conn(0x7fde300d3d80 msgr2=0x7fde300d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 -- 172.21.15.97:0/852873334 shutdown_connections 2024-03-06T07:40:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.789+0000 7fde35c00640 1 -- 172.21.15.97:0/852873334 wait complete. 2024-03-06T07:40:31.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.791+0000 7fde35c00640 1 Processor -- start 2024-03-06T07:40:31.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.791+0000 7fde35c00640 1 -- start start 2024-03-06T07:40:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fde300d1e90 0x7fde300b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde300d28a0 0x7fde3015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde3015d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde300b39a0 con 0x7fde30161580 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde300b36c0 con 0x7fde300d28a0 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde300cf3a0 con 0x7fde300d1e90 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde3015d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde3015d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33900/0 (socket says 172.21.15.97:33900) 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde35200640 1 -- 172.21.15.97:0/1481793627 learned_addr learned my addr 172.21.15.97:0/1481793627 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.792+0000 7fde2f400640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde300d28a0 0x7fde3015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde2fe00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fde300d1e90 0x7fde300b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35200640 1 -- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fde300d1e90 msgr2=0x7fde300b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35200640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fde300d1e90 0x7fde300b72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35200640 1 -- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde300d28a0 msgr2=0x7fde3015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35200640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde300d28a0 0x7fde3015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:31.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35200640 1 -- 172.21.15.97:0/1481793627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde300caef0 con 0x7fde30161580 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35200640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde3015d1f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fde2800b490 tx=0x7fde2800b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde28015020 con 0x7fde30161580 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde28011850 con 0x7fde30161580 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde28011c40 con 0x7fde30161580 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.793+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde300cb1d0 con 0x7fde30161580 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.794+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde300b7830 con 0x7fde30161580 2024-03-06T07:40:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.795+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde300d18b0 con 0x7fde30161580 2024-03-06T07:40:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.796+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde28018020 con 0x7fde30161580 2024-03-06T07:40:31.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.796+0000 7fde27e00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fde1006b4d0 0x7fde1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:31.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.797+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fde2808a0f0 con 0x7fde30161580 2024-03-06T07:40:31.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.797+0000 7fde2fe00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fde1006b4d0 0x7fde1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:31.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.798+0000 7fde2fe00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fde1006b4d0 0x7fde1006d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fde20035000 tx=0x7fde20030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:31.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:31.802+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde2801f020 con 0x7fde30161580 2024-03-06T07:40:32.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.106+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3004f330 con 0x7fde30161580 2024-03-06T07:40:32.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.107+0000 7fde27e00640 1 -- 172.21.15.97:0/1481793627 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fde280172c0 con 0x7fde30161580 2024-03-06T07:40:32.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:32.109 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:32.110 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:32.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.109+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fde1006b4d0 msgr2=0x7fde1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.109+0000 7fde35c00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fde1006b4d0 0x7fde1006d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fde20035000 tx=0x7fde20030000 comp rx=0 tx=0).stop 2024-03-06T07:40:32.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 msgr2=0x7fde3015d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde3015d1f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fde2800b490 tx=0x7fde2800b950 comp rx=0 tx=0).stop 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 shutdown_connections 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fde1006b4d0 0x7fde1006d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde30161580 0x7fde3015d1f0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde300d28a0 0x7fde3015ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 --2- 172.21.15.97:0/1481793627 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fde300d1e90 0x7fde300b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 >> 172.21.15.97:0/1481793627 conn(0x7fde300d3d80 msgr2=0x7fde300b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 shutdown_connections 2024-03-06T07:40:32.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:32.110+0000 7fde35c00640 1 -- 172.21.15.97:0/1481793627 wait complete. 2024-03-06T07:40:32.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:32.334 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:33.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:33.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.543+0000 7f58ed800640 1 Processor -- start 2024-03-06T07:40:33.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.543+0000 7f58ed800640 1 -- start start 2024-03-06T07:40:33.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.544+0000 7f58ed800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.544+0000 7f58ed800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f58e815f9d0 0x7f58e815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58ed800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e806ee70 con 0x7f58e80ce0a0 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58ed800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e8160370 con 0x7f58e815f9d0 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58ed800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e806cef0 con 0x7f58e8144c00 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58dfe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f58e815f9d0 0x7f58e815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58e6a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58dfe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f58e815f9d0 0x7f58e815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:33732/0 (socket says 172.21.15.97:33732) 2024-03-06T07:40:33.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.545+0000 7f58dfe00640 1 -- 172.21.15.97:0/2646883616 learned_addr learned my addr 172.21.15.97:0/2646883616 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:33.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.546+0000 7f58dfe00640 1 -- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 msgr2=0x7f58e806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:33.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.546+0000 7f58dfe00640 1 --2- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.546+0000 7f58dfe00640 1 -- 172.21.15.97:0/2646883616 >> v1:172.21.15.135:6789/0 conn(0x7f58e80ce0a0 legacy=0x7f58e80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.546+0000 7f58dfe00640 1 -- 172.21.15.97:0/2646883616 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e806d1d0 con 0x7f58e815f9d0 2024-03-06T07:40:33.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.546+0000 7f58e6a00640 1 --2- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:40:33.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.547+0000 7f58dfe00640 1 --2- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e815fe30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009870 tx=0x7f58d002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e61b09225980c7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:33.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.548+0000 7f58e6000640 1 -- 172.21.15.97:0/2646883616 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0034070 con 0x7f58e815f9d0 2024-03-06T07:40:33.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.548+0000 7f58e6000640 1 -- 172.21.15.97:0/2646883616 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d0030840 con 0x7f58e815f9d0 2024-03-06T07:40:33.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.548+0000 7f58e6000640 1 -- 172.21.15.97:0/2646883616 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0030c10 con 0x7f58e815f9d0 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.548+0000 7f58ed800640 1 -- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 msgr2=0x7f58e815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.548+0000 7f58ed800640 1 --2- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e815fe30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009870 tx=0x7f58d002b980 comp rx=0 tx=0).stop 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.549+0000 7f58ed800640 1 -- 172.21.15.97:0/2646883616 shutdown_connections 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.549+0000 7f58ed800640 1 --2- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e815fe30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.549+0000 7f58ed800640 1 --2- 172.21.15.97:0/2646883616 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.549+0000 7f58ed800640 1 -- 172.21.15.97:0/2646883616 >> 172.21.15.97:0/2646883616 conn(0x7f58e80b1730 msgr2=0x7f58e80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.549+0000 7f58ed800640 1 -- 172.21.15.97:0/2646883616 shutdown_connections 2024-03-06T07:40:33.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.549+0000 7f58ed800640 1 -- 172.21.15.97:0/2646883616 wait complete. 2024-03-06T07:40:33.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.551+0000 7f58ed800640 1 Processor -- start 2024-03-06T07:40:33.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.552+0000 7f58ed800640 1 -- start start 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.553+0000 7f58ed800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 0x7f58e8150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.553+0000 7f58ed800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e8071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.553+0000 7f58ed800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e813bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.553+0000 7f58ed800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e8160730 con 0x7f58e80ce0a0 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.553+0000 7f58ed800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e806d790 con 0x7f58e815f9d0 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.553+0000 7f58ed800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e806cb50 con 0x7f58e8144c00 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 0x7f58e8150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 0x7f58e8150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33906/0 (socket says 172.21.15.97:33906) 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 -- 172.21.15.97:0/1281754772 learned_addr learned my addr 172.21.15.97:0/1281754772 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:33.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58dfe00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e8071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 -- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 msgr2=0x7f58e8071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e8071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e7400640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e813bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 -- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 msgr2=0x7f58e813bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e813bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 -- 172.21.15.97:0/1281754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80a9810 con 0x7f58e80ce0a0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.554+0000 7f58e6a00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 0x7f58e8150a90 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f58d800ebe0 tx=0x7f58d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.555+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8017070 con 0x7f58e80ce0a0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.555+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d800cc30 con 0x7f58e80ce0a0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.555+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8013470 con 0x7f58e80ce0a0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.555+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e814e110 con 0x7f58e80ce0a0 2024-03-06T07:40:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.555+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e814e6d0 con 0x7f58e80ce0a0 2024-03-06T07:40:33.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.556+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e80b2950 con 0x7f58e80ce0a0 2024-03-06T07:40:33.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.557+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58d801a070 con 0x7f58e80ce0a0 2024-03-06T07:40:33.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.558+0000 7f58dea00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58c406b4d0 0x7f58c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.558+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f58d808a530 con 0x7f58e80ce0a0 2024-03-06T07:40:33.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.558+0000 7f58dfe00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58c406b4d0 0x7f58c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.559+0000 7f58dfe00640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58c406b4d0 0x7f58c406d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f58d0036000 tx=0x7f58d003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.563+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58e80b2950 con 0x7f58e80ce0a0 2024-03-06T07:40:33.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.795+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e804f330 con 0x7f58e80ce0a0 2024-03-06T07:40:33.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.796+0000 7f58dea00640 1 -- 172.21.15.97:0/1281754772 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f58d8059380 con 0x7f58e80ce0a0 2024-03-06T07:40:33.799 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:33.799 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:33.799 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58c406b4d0 msgr2=0x7f58c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58c406b4d0 0x7f58c406d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f58d0036000 tx=0x7f58d003b040 comp rx=0 tx=0).stop 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 msgr2=0x7f58e8150a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 0x7f58e8150a90 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f58d800ebe0 tx=0x7f58d800c520 comp rx=0 tx=0).stop 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 shutdown_connections 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f58c406b4d0 0x7f58c406d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58e815f9d0 0x7f58e813bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58e8144c00 0x7f58e8071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 --2- 172.21.15.97:0/1281754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e80ce0a0 0x7f58e8150a90 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 >> 172.21.15.97:0/1281754772 conn(0x7f58e80b1730 msgr2=0x7f58e81429e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:33.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 shutdown_connections 2024-03-06T07:40:33.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:33.799+0000 7f58ed800640 1 -- 172.21.15.97:0/1281754772 wait complete. 2024-03-06T07:40:34.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:34.021 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:35.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:35.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.233+0000 7f9854600640 1 Processor -- start 2024-03-06T07:40:35.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.233+0000 7f9854600640 1 -- start start 2024-03-06T07:40:35.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.234+0000 7f9854600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f984c15f470 0x7f984c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.234+0000 7f9854600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0d1200 con 0x7f984c15e0d0 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9854600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0cea60 con 0x7f984c0d0df0 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9854600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984c0d2f60 con 0x7f984c15f470 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9851200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f984c15f470 0x7f984c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9851c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f984c15e0d0 0x7f984c0d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42062/0 (socket says 172.21.15.97:42062) 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9851c00640 1 -- 172.21.15.97:0/2035644056 learned_addr learned my addr 172.21.15.97:0/2035644056 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9851200640 1 -- 172.21.15.97:0/2035644056 >> v1:172.21.15.135:6789/0 conn(0x7f984c0d0df0 legacy=0x7f984c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9851200640 1 -- 172.21.15.97:0/2035644056 >> v1:172.21.15.97:6789/0 conn(0x7f984c15e0d0 legacy=0x7f984c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.235+0000 7f9851200640 1 -- 172.21.15.97:0/2035644056 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c0d2350 con 0x7f984c15f470 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.236+0000 7f9851200640 1 --2- 172.21.15.97:0/2035644056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c15f470 0x7f984c0d08b0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f984c0c77f0 tx=0x7f983802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968a6e435a72cff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f984be00640 1 -- 172.21.15.97:0/2035644056 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838034070 con 0x7f984c15f470 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f984be00640 1 -- 172.21.15.97:0/2035644056 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9838030610 con 0x7f984c15f470 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f984be00640 1 -- 172.21.15.97:0/2035644056 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98380309e0 con 0x7f984c15f470 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f9854600640 1 -- 172.21.15.97:0/2035644056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c15f470 msgr2=0x7f984c0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f9854600640 1 --2- 172.21.15.97:0/2035644056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c15f470 0x7f984c0d08b0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f984c0c77f0 tx=0x7f983802b960 comp rx=0 tx=0).stop 2024-03-06T07:40:35.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f9854600640 1 -- 172.21.15.97:0/2035644056 shutdown_connections 2024-03-06T07:40:35.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f9854600640 1 --2- 172.21.15.97:0/2035644056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c15f470 0x7f984c0d08b0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.237+0000 7f9854600640 1 -- 172.21.15.97:0/2035644056 >> 172.21.15.97:0/2035644056 conn(0x7f984c0d3ec0 msgr2=0x7f984c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:35.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.238+0000 7f9854600640 1 -- 172.21.15.97:0/2035644056 shutdown_connections 2024-03-06T07:40:35.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.238+0000 7f9854600640 1 -- 172.21.15.97:0/2035644056 wait complete. 2024-03-06T07:40:35.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.239+0000 7f9854600640 1 Processor -- start 2024-03-06T07:40:35.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.239+0000 7f9854600640 1 -- start start 2024-03-06T07:40:35.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.240+0000 7f9854600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 0x7f984c158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.240+0000 7f9854600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f984c15e0d0 0x7f984c158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.240+0000 7f9854600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f984c15f470 0x7f984c1f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.240+0000 7f9854600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c0d2bf0 con 0x7f984c0d0df0 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.240+0000 7f9854600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c0d2910 con 0x7f984c15e0d0 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.240+0000 7f9854600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c0cff10 con 0x7f984c15f470 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 0x7f984c158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 0x7f984c158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45184/0 (socket says 172.21.15.97:45184) 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 -- 172.21.15.97:0/2212461471 learned_addr learned my addr 172.21.15.97:0/2212461471 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851200640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f984c15e0d0 0x7f984c158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9852600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f984c15f470 0x7f984c1f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 -- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f984c15f470 msgr2=0x7f984c1f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f984c15f470 0x7f984c1f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 -- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f984c15e0d0 msgr2=0x7f984c158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f984c15e0d0 0x7f984c158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851c00640 1 -- 172.21.15.97:0/2212461471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c0c8560 con 0x7f984c0d0df0 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.241+0000 7f9851200640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f984c15e0d0 0x7f984c158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.242+0000 7f9851c00640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 0x7f984c158700 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f984000c400 tx=0x7f984000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.242+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840017070 con 0x7f984c0d0df0 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.242+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9840005ce0 con 0x7f984c0d0df0 2024-03-06T07:40:35.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.242+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984c0c79e0 con 0x7f984c0d0df0 2024-03-06T07:40:35.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.242+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98400072f0 con 0x7f984c0d0df0 2024-03-06T07:40:35.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.242+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c0c7fa0 con 0x7f984c0d0df0 2024-03-06T07:40:35.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.243+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c04f330 con 0x7f984c0d0df0 2024-03-06T07:40:35.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.244+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f984001a070 con 0x7f984c0d0df0 2024-03-06T07:40:35.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.245+0000 7f9849600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f982c06b4d0 0x7f982c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.245+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9840089630 con 0x7f984c0d0df0 2024-03-06T07:40:35.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.245+0000 7f9851200640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f982c06b4d0 0x7f982c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.246+0000 7f9851200640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f982c06b4d0 0x7f982c06d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9838036000 tx=0x7f9838008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:35.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.253+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f984c0d1200 con 0x7f984c0d0df0 2024-03-06T07:40:35.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.488+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f984c08d920 con 0x7f984c0d0df0 2024-03-06T07:40:35.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.489+0000 7f9849600640 1 -- 172.21.15.97:0/2212461471 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f9840058490 con 0x7f984c0d0df0 2024-03-06T07:40:35.492 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:35.492 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:35.492 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f982c06b4d0 msgr2=0x7f982c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f982c06b4d0 0x7f982c06d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9838036000 tx=0x7f9838008030 comp rx=0 tx=0).stop 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 msgr2=0x7f984c158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 0x7f984c158700 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f984000c400 tx=0x7f984000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 shutdown_connections 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f982c06b4d0 0x7f982c06d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f984c15f470 0x7f984c1f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f984c15e0d0 0x7f984c158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 --2- 172.21.15.97:0/2212461471 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f984c0d0df0 0x7f984c158700 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 >> 172.21.15.97:0/2212461471 conn(0x7f984c0d3ec0 msgr2=0x7f984c0cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 shutdown_connections 2024-03-06T07:40:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:35.492+0000 7f9854600640 1 -- 172.21.15.97:0/2212461471 wait complete. 2024-03-06T07:40:35.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:35.713 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:36.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:36.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.934+0000 7f2b8ec00640 1 Processor -- start 2024-03-06T07:40:36.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.934+0000 7f2b8ec00640 1 -- start start 2024-03-06T07:40:36.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.935+0000 7f2b8ec00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2b8815f9d0 0x7f2b8815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:36.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.935+0000 7f2b8ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b88070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.935+0000 7f2b8ec00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8806cef0 con 0x7f2b88144c00 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.935+0000 7f2b8ec00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8806d1d0 con 0x7f2b8815f9d0 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.935+0000 7f2b8ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8806d4b0 con 0x7f2b88070b10 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2b8815f9d0 0x7f2b8815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2b8815f9d0 0x7f2b8815fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45198/0 (socket says 172.21.15.97:45198) 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 -- 172.21.15.97:0/1624758340 learned_addr learned my addr 172.21.15.97:0/1624758340 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b8cc00640 1 --2- 172.21.15.97:0/1624758340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b88070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 -- 172.21.15.97:0/1624758340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 msgr2=0x7f2b88070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 --2- 172.21.15.97:0/1624758340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b88070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:36.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 -- 172.21.15.97:0/1624758340 >> v1:172.21.15.135:6789/0 conn(0x7f2b88144c00 legacy=0x7f2b8806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:36.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.936+0000 7f2b87400640 1 -- 172.21.15.97:0/1624758340 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8806d790 con 0x7f2b8815f9d0 2024-03-06T07:40:36.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.938+0000 7f2b87400640 1 --2- 172.21.15.97:0/1624758340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b8815f9d0 0x7f2b8815fe60 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009870 tx=0x7f2b7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e82597fe57cbcb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:36.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.938+0000 7f2b86a00640 1 -- 172.21.15.97:0/1624758340 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70035020 con 0x7f2b8815f9d0 2024-03-06T07:40:36.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.938+0000 7f2b86a00640 1 -- 172.21.15.97:0/1624758340 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70031640 con 0x7f2b8815f9d0 2024-03-06T07:40:36.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.938+0000 7f2b86a00640 1 -- 172.21.15.97:0/1624758340 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7002d040 con 0x7f2b8815f9d0 2024-03-06T07:40:36.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 -- 172.21.15.97:0/1624758340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b8815f9d0 msgr2=0x7f2b8815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:36.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/1624758340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b8815f9d0 0x7f2b8815fe60 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009870 tx=0x7f2b7002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:40:36.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 -- 172.21.15.97:0/1624758340 shutdown_connections 2024-03-06T07:40:36.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/1624758340 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b88070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:36.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/1624758340 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b8815f9d0 0x7f2b8815fe60 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:36.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 -- 172.21.15.97:0/1624758340 >> 172.21.15.97:0/1624758340 conn(0x7f2b880b1730 msgr2=0x7f2b880b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:36.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 -- 172.21.15.97:0/1624758340 shutdown_connections 2024-03-06T07:40:36.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.939+0000 7f2b8ec00640 1 -- 172.21.15.97:0/1624758340 wait complete. 2024-03-06T07:40:36.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.941+0000 7f2b8ec00640 1 Processor -- start 2024-03-06T07:40:36.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.941+0000 7f2b8ec00640 1 -- start start 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b8ec00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b8815b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b8ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 0x7f2b8815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b8ec00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b8815f9d0 0x7f2b881e9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b8ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8806ccd0 con 0x7f2b88144c00 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b8ec00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8806cb50 con 0x7f2b8815f9d0 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b8ec00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8806df90 con 0x7f2b88070b10 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b87400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 0x7f2b8815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:36.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b87400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 0x7f2b8815eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45204/0 (socket says 172.21.15.97:45204) 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b87400640 1 -- 172.21.15.97:0/3164438914 learned_addr learned my addr 172.21.15.97:0/3164438914 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.942+0000 7f2b87e00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b8815b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b8cc00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b8815f9d0 0x7f2b881e9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87400640 1 -- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 msgr2=0x7f2b8815b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87400640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b8815b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87400640 1 -- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b8815f9d0 msgr2=0x7f2b881e9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87400640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b8815f9d0 0x7f2b881e9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87400640 1 -- 172.21.15.97:0/3164438914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b881e7810 con 0x7f2b88144c00 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87e00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b8815b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b87400640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 0x7f2b8815eeb0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f2b7002b930 tx=0x7f2b700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:36.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70040070 con 0x7f2b88144c00 2024-03-06T07:40:36.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70039c10 con 0x7f2b88144c00 2024-03-06T07:40:36.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70035040 con 0x7f2b88144c00 2024-03-06T07:40:36.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b881e7ab0 con 0x7f2b88144c00 2024-03-06T07:40:36.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.943+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b881e8070 con 0x7f2b88144c00 2024-03-06T07:40:36.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.945+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b88160550 con 0x7f2b88144c00 2024-03-06T07:40:36.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.945+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b7004b060 con 0x7f2b88144c00 2024-03-06T07:40:36.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.946+0000 7f2b7fe00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b6406b4d0 0x7f2b6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:36.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.947+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2b700abd20 con 0x7f2b88144c00 2024-03-06T07:40:36.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.947+0000 7f2b87e00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b6406b4d0 0x7f2b6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:36.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.948+0000 7f2b87e00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b6406b4d0 0x7f2b6406d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2b78003eb0 tx=0x7f2b78009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:36.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:36.951+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b88160550 con 0x7f2b88144c00 2024-03-06T07:40:37.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.191+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b880cc800 con 0x7f2b88144c00 2024-03-06T07:40:37.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.192+0000 7f2b7fe00640 1 -- 172.21.15.97:0/3164438914 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f2b7007ab80 con 0x7f2b88144c00 2024-03-06T07:40:37.195 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:37.195 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:37.195 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b6406b4d0 msgr2=0x7f2b6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b6406b4d0 0x7f2b6406d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2b78003eb0 tx=0x7f2b78009040 comp rx=0 tx=0).stop 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 msgr2=0x7f2b8815eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 0x7f2b8815eeb0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f2b7002b930 tx=0x7f2b700046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 shutdown_connections 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2b6406b4d0 0x7f2b6406d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b8815f9d0 0x7f2b881e9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b88144c00 0x7f2b8815eeb0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 --2- 172.21.15.97:0/3164438914 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2b88070b10 0x7f2b8815b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 >> 172.21.15.97:0/3164438914 conn(0x7f2b880b1730 msgr2=0x7f2b88070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:37.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 shutdown_connections 2024-03-06T07:40:37.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:37.195+0000 7f2b8ec00640 1 -- 172.21.15.97:0/3164438914 wait complete. 2024-03-06T07:40:37.422 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:37.422 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:38.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:38.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.641+0000 7f9792800640 1 Processor -- start 2024-03-06T07:40:38.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.641+0000 7f9792800640 1 -- start start 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f9792800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f978c144c00 0x7f978c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f9792800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f9792800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f978c0ce0a0 0x7f978c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f9792800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c160370 con 0x7f978c0ce0a0 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f9792800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c06ee70 con 0x7f978c144c00 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f9792800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c06cef0 con 0x7f978c15f9d0 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f978be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f978c0ce0a0 0x7f978c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f978be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f978c0ce0a0 0x7f978c0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45212/0 (socket says 172.21.15.97:45212) 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f978be00640 1 -- 172.21.15.97:0/1223723821 learned_addr learned my addr 172.21.15.97:0/1223723821 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:38.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f978aa00640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.643+0000 7f978b400640 1 --2- 172.21.15.97:0/1223723821 >> v2:172.21.15.135:3300/0 conn(0x7f978c144c00 0x7f978c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.644+0000 7f978be00640 1 -- 172.21.15.97:0/1223723821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 msgr2=0x7f978c15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.644+0000 7f978be00640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.644+0000 7f978be00640 1 -- 172.21.15.97:0/1223723821 >> v2:172.21.15.135:3300/0 conn(0x7f978c144c00 msgr2=0x7f978c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.644+0000 7f978be00640 1 --2- 172.21.15.97:0/1223723821 >> v2:172.21.15.135:3300/0 conn(0x7f978c144c00 0x7f978c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.644+0000 7f978be00640 1 -- 172.21.15.97:0/1223723821 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c06d1d0 con 0x7f978c0ce0a0 2024-03-06T07:40:38.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.644+0000 7f978aa00640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:38.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.645+0000 7f978be00640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c0ce4f0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f978c0b25f0 tx=0x7f978402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259f61c36f1bc28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:38.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.645+0000 7f978a000640 1 -- 172.21.15.97:0/1223723821 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978402ca70 con 0x7f978c0ce0a0 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.645+0000 7f978a000640 1 -- 172.21.15.97:0/1223723821 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978402cc10 con 0x7f978c0ce0a0 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.645+0000 7f978a000640 1 -- 172.21.15.97:0/1223723821 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840343f0 con 0x7f978c0ce0a0 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 -- 172.21.15.97:0/1223723821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 msgr2=0x7f978c0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c0ce4f0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f978c0b25f0 tx=0x7f978402bed0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 -- 172.21.15.97:0/1223723821 shutdown_connections 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c0ce4f0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 --2- 172.21.15.97:0/1223723821 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c15fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 --2- 172.21.15.97:0/1223723821 >> v2:172.21.15.135:3300/0 conn(0x7f978c144c00 0x7f978c06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 -- 172.21.15.97:0/1223723821 >> 172.21.15.97:0/1223723821 conn(0x7f978c0b1730 msgr2=0x7f978c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 -- 172.21.15.97:0/1223723821 shutdown_connections 2024-03-06T07:40:38.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.646+0000 7f9792800640 1 -- 172.21.15.97:0/1223723821 wait complete. 2024-03-06T07:40:38.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.648+0000 7f9792800640 1 Processor -- start 2024-03-06T07:40:38.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.648+0000 7f9792800640 1 -- start start 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.648+0000 7f9792800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c1e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f9792800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f978c144c00 0x7f978c1e8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f9792800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c1e8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f9792800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c160730 con 0x7f978c0ce0a0 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f9792800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c06d790 con 0x7f978c144c00 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f9792800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c06cb50 con 0x7f978c15f9d0 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c1e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c1e60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45216/0 (socket says 172.21.15.97:45216) 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 -- 172.21.15.97:0/3244440280 learned_addr learned my addr 172.21.15.97:0/3244440280 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978aa00640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f978c144c00 0x7f978c1e8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 -- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 msgr2=0x7f978c1e8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c1e8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 -- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f978c144c00 msgr2=0x7f978c1e8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f978c144c00 0x7f978c1e8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.649+0000 7f978b400640 1 -- 172.21.15.97:0/3244440280 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c15d430 con 0x7f978c0ce0a0 2024-03-06T07:40:38.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.650+0000 7f978b400640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c1e60c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f977c00d800 tx=0x7f977c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.650+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c018020 con 0x7f978c0ce0a0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.650+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c0096a0 con 0x7f978c0ce0a0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.650+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c009a70 con 0x7f978c0ce0a0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.650+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c140df0 con 0x7f978c0ce0a0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.650+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c1e78d0 con 0x7f978c0ce0a0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.651+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c04f330 con 0x7f978c0ce0a0 2024-03-06T07:40:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.652+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f977c01b020 con 0x7f978c0ce0a0 2024-03-06T07:40:38.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.653+0000 7f9783400640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f976c06b4d0 0x7f976c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.653+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f977c08a750 con 0x7f978c0ce0a0 2024-03-06T07:40:38.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.653+0000 7f978aa00640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f976c06b4d0 0x7f976c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.654+0000 7f978aa00640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f976c06b4d0 0x7f976c06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f978c15f010 tx=0x7f9778008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:38.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.657+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f977c022020 con 0x7f978c0ce0a0 2024-03-06T07:40:38.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.956+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f978c072890 con 0x7f978c0ce0a0 2024-03-06T07:40:38.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.956+0000 7f9783400640 1 -- 172.21.15.97:0/3244440280 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f977c058f60 con 0x7f978c0ce0a0 2024-03-06T07:40:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:38.959 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:38.959 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:38.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f976c06b4d0 msgr2=0x7f976c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f976c06b4d0 0x7f976c06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f978c15f010 tx=0x7f9778008040 comp rx=0 tx=0).stop 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 msgr2=0x7f978c1e60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c1e60c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f977c00d800 tx=0x7f977c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 shutdown_connections 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f976c06b4d0 0x7f976c06d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f978c15f9d0 0x7f978c1e8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f978c144c00 0x7f978c1e8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 --2- 172.21.15.97:0/3244440280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f978c0ce0a0 0x7f978c1e60c0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 >> 172.21.15.97:0/3244440280 conn(0x7f978c0b1730 msgr2=0x7f978c145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 shutdown_connections 2024-03-06T07:40:38.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:38.959+0000 7f9792800640 1 -- 172.21.15.97:0/3244440280 wait complete. 2024-03-06T07:40:39.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:39.184 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:40.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:40.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.404+0000 7fdc8e000640 1 Processor -- start 2024-03-06T07:40:40.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.404+0000 7fdc8e000640 1 -- start start 2024-03-06T07:40:40.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.406+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc8815acb0 0x7fdc880b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.406+0000 7fdc8e000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc8806ec00 0x7fdc880ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.406+0000 7fdc8e000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8815b090 con 0x7fdc880b4680 2024-03-06T07:40:40.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.406+0000 7fdc8e000640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc880b2a30 con 0x7fdc8806ec00 2024-03-06T07:40:40.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.406+0000 7fdc8e000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880b4360 con 0x7fdc8815acb0 2024-03-06T07:40:40.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.407+0000 7fdc87e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc8806ec00 0x7fdc880ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.407+0000 7fdc87400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc8815acb0 0x7fdc880b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.407+0000 7fdc87e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc8806ec00 0x7fdc880ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:45130/0 (socket says 172.21.15.97:45130) 2024-03-06T07:40:40.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.407+0000 7fdc87e00640 1 -- 172.21.15.97:0/67461450 learned_addr learned my addr 172.21.15.97:0/67461450 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:40.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.408+0000 7fdc87400640 1 -- 172.21.15.97:0/67461450 >> v2:172.21.15.135:3300/0 conn(0x7fdc8806ec00 msgr2=0x7fdc880ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:40.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.408+0000 7fdc87400640 1 --2- 172.21.15.97:0/67461450 >> v2:172.21.15.135:3300/0 conn(0x7fdc8806ec00 0x7fdc880ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.408+0000 7fdc87400640 1 -- 172.21.15.97:0/67461450 >> v1:172.21.15.135:6789/0 conn(0x7fdc880b4680 legacy=0x7fdc8806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.408+0000 7fdc87400640 1 -- 172.21.15.97:0/67461450 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc880b2cf0 con 0x7fdc8815acb0 2024-03-06T07:40:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.409+0000 7fdc87400640 1 --2- 172.21.15.97:0/67461450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc8815acb0 0x7fdc880b37d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdc780097a0 tx=0x7fdc7802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60118d764077f0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:40.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.410+0000 7fdc86a00640 1 -- 172.21.15.97:0/67461450 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78035020 con 0x7fdc8815acb0 2024-03-06T07:40:40.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.410+0000 7fdc86a00640 1 -- 172.21.15.97:0/67461450 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc78031830 con 0x7fdc8815acb0 2024-03-06T07:40:40.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.410+0000 7fdc86a00640 1 -- 172.21.15.97:0/67461450 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78031c00 con 0x7fdc8815acb0 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 -- 172.21.15.97:0/67461450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc8815acb0 msgr2=0x7fdc880b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 --2- 172.21.15.97:0/67461450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc8815acb0 0x7fdc880b37d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdc780097a0 tx=0x7fdc7802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 -- 172.21.15.97:0/67461450 shutdown_connections 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 --2- 172.21.15.97:0/67461450 >> v2:172.21.15.135:3300/0 conn(0x7fdc8806ec00 0x7fdc880ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 --2- 172.21.15.97:0/67461450 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc8815acb0 0x7fdc880b37d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 -- 172.21.15.97:0/67461450 >> 172.21.15.97:0/67461450 conn(0x7fdc880b1730 msgr2=0x7fdc880b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 -- 172.21.15.97:0/67461450 shutdown_connections 2024-03-06T07:40:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.411+0000 7fdc8e000640 1 -- 172.21.15.97:0/67461450 wait complete. 2024-03-06T07:40:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.413+0000 7fdc8e000640 1 Processor -- start 2024-03-06T07:40:40.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.414+0000 7fdc8e000640 1 -- start start 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.415+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 0x7fdc881e5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.415+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc880b4680 0x7fdc8806d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.415+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc8815acb0 0x7fdc8806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.415+0000 7fdc8e000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880b3d10 con 0x7fdc8806ec00 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.415+0000 7fdc8e000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880b32b0 con 0x7fdc8815acb0 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.415+0000 7fdc8e000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880b4010 con 0x7fdc880b4680 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 0x7fdc881e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 0x7fdc881e5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45230/0 (socket says 172.21.15.97:45230) 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 -- 172.21.15.97:0/3622102791 learned_addr learned my addr 172.21.15.97:0/3622102791 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87e00640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc8815acb0 0x7fdc8806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 -- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc880b4680 msgr2=0x7fdc8806d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc7fe00640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc880b4680 0x7fdc8806d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc880b4680 0x7fdc8806d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 -- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc8815acb0 msgr2=0x7fdc8806dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc8815acb0 0x7fdc8806dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 -- 172.21.15.97:0/3622102791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8815a520 con 0x7fdc8806ec00 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87e00640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc8815acb0 0x7fdc8806dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.416+0000 7fdc87400640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 0x7fdc881e5940 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fdc78000c00 tx=0x7fdc780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.417+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78040070 con 0x7fdc8806ec00 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.417+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7803a850 con 0x7fdc8806ec00 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.417+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78035040 con 0x7fdc8806ec00 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.417+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc881599a0 con 0x7fdc8806ec00 2024-03-06T07:40:40.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.417+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc88159f60 con 0x7fdc8806ec00 2024-03-06T07:40:40.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.418+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc88140cc0 con 0x7fdc8806ec00 2024-03-06T07:40:40.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.419+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc7804b060 con 0x7fdc8806ec00 2024-03-06T07:40:40.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.420+0000 7fdc7f400640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.420+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdc780abc20 con 0x7fdc8806ec00 2024-03-06T07:40:40.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.420+0000 7fdc7fe00640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.421+0000 7fdc7fe00640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc6406b4d0 0x7fdc6406d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fdc881e7160 tx=0x7fdc70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:40.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.424+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc88140cc0 con 0x7fdc8806ec00 2024-03-06T07:40:40.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:40.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:40.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.687+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc8804f330 con 0x7fdc8806ec00 2024-03-06T07:40:40.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.688+0000 7fdc7f400640 1 -- 172.21.15.97:0/3622102791 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fdc78031800 con 0x7fdc8806ec00 2024-03-06T07:40:40.690 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:40.690 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:40.690 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc6406b4d0 msgr2=0x7fdc6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc6406b4d0 0x7fdc6406d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fdc881e7160 tx=0x7fdc70009040 comp rx=0 tx=0).stop 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 msgr2=0x7fdc881e5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 0x7fdc881e5940 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fdc78000c00 tx=0x7fdc780046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 shutdown_connections 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc8815acb0 0x7fdc8806dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdc880b4680 0x7fdc8806d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 --2- 172.21.15.97:0/3622102791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc8806ec00 0x7fdc881e5940 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.690+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 >> 172.21.15.97:0/3622102791 conn(0x7fdc880b1730 msgr2=0x7fdc8815b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.691+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 shutdown_connections 2024-03-06T07:40:40.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:40.691+0000 7fdc8e000640 1 -- 172.21.15.97:0/3622102791 wait complete. 2024-03-06T07:40:40.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:40.912 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:41.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:42.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.128+0000 7f2d19e00640 1 Processor -- start 2024-03-06T07:40:42.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.128+0000 7f2d19e00640 1 -- start start 2024-03-06T07:40:42.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.131+0000 7f2d19e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2d1406cb50 0x7f2d1406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.131+0000 7f2d19e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d140717c0 con 0x7f2d140cf430 2024-03-06T07:40:42.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.131+0000 7f2d19e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d14071aa0 con 0x7f2d140b2d10 2024-03-06T07:40:42.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.131+0000 7f2d19e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d14071d80 con 0x7f2d1406cb50 2024-03-06T07:40:42.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.131+0000 7f2d12a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2d140cf430 0x7f2d14070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42072/0 (socket says 172.21.15.97:42072) 2024-03-06T07:40:42.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.131+0000 7f2d12a00640 1 -- 172.21.15.97:0/1876636390 learned_addr learned my addr 172.21.15.97:0/1876636390 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:42.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d12000640 1 --2- 172.21.15.97:0/1876636390 >> v2:172.21.15.135:3300/0 conn(0x7f2d1406cb50 0x7f2d1406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638450711 0 0) 0x7f2d140717c0 con 0x7f2d140cf430 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf8003520 con 0x7f2d140cf430 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403514808 0 0) 0x7f2cf8003520 con 0x7f2d140cf430 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 >> v2:172.21.15.135:3300/0 conn(0x7f2d1406cb50 msgr2=0x7f2d1406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 --2- 172.21.15.97:0/1876636390 >> v2:172.21.15.135:3300/0 conn(0x7f2d1406cb50 0x7f2d1406cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 >> v1:172.21.15.135:6789/0 conn(0x7f2d140b2d10 legacy=0x7f2d140b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14072060 con 0x7f2d140cf430 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2d04001670 con 0x7f2d140cf430 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 >> v1:172.21.15.97:6789/0 conn(0x7f2d140cf430 legacy=0x7f2d14070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80039b0 0x7f2cf8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2cf8004800 0x7f2cf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2cf80051c0 0x7f2cf800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14071d80 con 0x7f2cf80051c0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf800c0f0 con 0x7f2cf80039b0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.132+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf800c2f0 con 0x7f2cf8004800 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d13400640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2cf80051c0 0x7f2cf800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d12a00640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80039b0 0x7f2cf8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d13400640 1 -- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2cf8004800 msgr2=0x7f2cf8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d13400640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2cf8004800 0x7f2cf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d13400640 1 -- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80039b0 msgr2=0x7f2cf8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d13400640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80039b0 0x7f2cf8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.133+0000 7f2d13400640 1 -- 172.21.15.97:0/1876636390 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8003520 con 0x7f2cf80051c0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.134+0000 7f2d13400640 1 -- 172.21.15.97:0/1876636390 reap_dead start 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.134+0000 7f2d13400640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2cf80051c0 0x7f2cf800b9c0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c004480 tx=0x7f2cf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323556dfcee40ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.134+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c034070 con 0x7f2cf80051c0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0c006b50 con 0x7f2cf80051c0 2024-03-06T07:40:42.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d11600640 1 -- 172.21.15.97:0/1876636390 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c004080 con 0x7f2cf80051c0 2024-03-06T07:40:42.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 -- 172.21.15.97:0/1876636390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2cf80051c0 msgr2=0x7f2cf800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2cf80051c0 0x7f2cf800b9c0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c004480 tx=0x7f2cf8005830 comp rx=0 tx=0).stop 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 -- 172.21.15.97:0/1876636390 shutdown_connections 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 --2- 172.21.15.97:0/1876636390 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2cf80051c0 0x7f2cf800b9c0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 -- 172.21.15.97:0/1876636390 >> 172.21.15.97:0/1876636390 conn(0x7f2d140b1730 msgr2=0x7f2d140b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 -- 172.21.15.97:0/1876636390 shutdown_connections 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.135+0000 7f2d19e00640 1 -- 172.21.15.97:0/1876636390 wait complete. 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.137+0000 7f2d19e00640 1 Processor -- start 2024-03-06T07:40:42.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.137+0000 7f2d19e00640 1 -- start start 2024-03-06T07:40:42.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d19e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80051c0 0x7f2d14150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d19e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 0x7f2d14152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d19e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2d141532f0 0x7f2d14151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d19e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14162cf0 con 0x7f2d14146dd0 2024-03-06T07:40:42.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d19e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14162b70 con 0x7f2cf80051c0 2024-03-06T07:40:42.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d19e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1406cfe0 con 0x7f2d141532f0 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d12000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 0x7f2d14152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d12000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 0x7f2d14152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45240/0 (socket says 172.21.15.97:45240) 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.138+0000 7f2d12000640 1 -- 172.21.15.97:0/854860688 learned_addr learned my addr 172.21.15.97:0/854860688 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12a00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80051c0 0x7f2d14150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d13400640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2d141532f0 0x7f2d14151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12000640 1 -- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2d141532f0 msgr2=0x7f2d14151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12000640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2d141532f0 0x7f2d14151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12000640 1 -- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80051c0 msgr2=0x7f2d14150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12000640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80051c0 0x7f2d14150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12000640 1 -- 172.21.15.97:0/854860688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d141663a0 con 0x7f2d14146dd0 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12a00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80051c0 0x7f2d14150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d13400640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2d141532f0 0x7f2d14151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d12000640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 0x7f2d14152db0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000d800 tx=0x7f2d0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.139+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00018020 con 0x7f2d14146dd0 2024-03-06T07:40:42.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.140+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0000b6c0 con 0x7f2d14146dd0 2024-03-06T07:40:42.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.140+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d14165380 con 0x7f2d14146dd0 2024-03-06T07:40:42.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.140+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000ba90 con 0x7f2d14146dd0 2024-03-06T07:40:42.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.140+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d141648a0 con 0x7f2d14146dd0 2024-03-06T07:40:42.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.141+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d14153700 con 0x7f2d14146dd0 2024-03-06T07:40:42.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.142+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d0001b020 con 0x7f2d14146dd0 2024-03-06T07:40:42.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.143+0000 7f2d0b400640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2cf406b4d0 0x7f2cf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.143+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2d0008a390 con 0x7f2d14146dd0 2024-03-06T07:40:42.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.143+0000 7f2d12a00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2cf406b4d0 0x7f2cf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.144+0000 7f2d12a00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2cf406b4d0 0x7f2cf406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2d140cf190 tx=0x7f2d04000f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.148+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d14153700 con 0x7f2d14146dd0 2024-03-06T07:40:42.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.383+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d1404f330 con 0x7f2d14146dd0 2024-03-06T07:40:42.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.384+0000 7f2d0b400640 1 -- 172.21.15.97:0/854860688 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f2d00022030 con 0x7f2d14146dd0 2024-03-06T07:40:42.387 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:42.387 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2cf406b4d0 msgr2=0x7f2cf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2cf406b4d0 0x7f2cf406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2d140cf190 tx=0x7f2d04000f10 comp rx=0 tx=0).stop 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 msgr2=0x7f2d14152db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 0x7f2d14152db0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000d800 tx=0x7f2d0000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 shutdown_connections 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2cf406b4d0 0x7f2cf406d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2d141532f0 0x7f2d14151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d14146dd0 0x7f2d14152db0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 --2- 172.21.15.97:0/854860688 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf80051c0 0x7f2d14150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 >> 172.21.15.97:0/854860688 conn(0x7f2d140b1730 msgr2=0x7f2d14070db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 shutdown_connections 2024-03-06T07:40:42.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:42.387+0000 7f2d19e00640 1 -- 172.21.15.97:0/854860688 wait complete. 2024-03-06T07:40:42.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:42.616 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:43.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:43.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.824+0000 7fe606400640 1 Processor -- start 2024-03-06T07:40:43.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.824+0000 7fe606400640 1 -- start start 2024-03-06T07:40:43.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe606400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe60015acb0 0x7fe6000b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:43.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe606400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60015b090 con 0x7fe60006ec00 2024-03-06T07:40:43.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe606400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000b2a30 con 0x7fe6000b4680 2024-03-06T07:40:43.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe606400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6000b4360 con 0x7fe60015acb0 2024-03-06T07:40:43.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe5ff400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe60015acb0 0x7fe6000b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe5ff400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe60015acb0 0x7fe6000b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45242/0 (socket says 172.21.15.97:45242) 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe5ff400640 1 -- 172.21.15.97:0/1956043045 learned_addr learned my addr 172.21.15.97:0/1956043045 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.826+0000 7fe5ffe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe60006ec00 0x7fe6000ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42088/0 (socket says 172.21.15.97:42088) 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.827+0000 7fe5ff400640 1 -- 172.21.15.97:0/1956043045 >> v1:172.21.15.135:6789/0 conn(0x7fe6000b4680 legacy=0x7fe60006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.827+0000 7fe5ff400640 1 -- 172.21.15.97:0/1956043045 >> v1:172.21.15.97:6789/0 conn(0x7fe60006ec00 legacy=0x7fe6000ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.827+0000 7fe5fe000640 1 -- 172.21.15.97:0/1956043045 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102102748 0 0) 0x7fe60015b090 con 0x7fe60006ec00 2024-03-06T07:40:43.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.827+0000 7fe5ff400640 1 -- 172.21.15.97:0/1956043045 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000b2cf0 con 0x7fe60015acb0 2024-03-06T07:40:43.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.828+0000 7fe5ff400640 1 --2- 172.21.15.97:0/1956043045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60015acb0 0x7fe6000b37d0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009870 tx=0x7fe5f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8959d6aaacd3a537 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:43.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.828+0000 7fe5fe000640 1 -- 172.21.15.97:0/1956043045 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0034070 con 0x7fe60015acb0 2024-03-06T07:40:43.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.828+0000 7fe5fe000640 1 -- 172.21.15.97:0/1956043045 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0030860 con 0x7fe60015acb0 2024-03-06T07:40:43.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.828+0000 7fe5fe000640 1 -- 172.21.15.97:0/1956043045 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0030c30 con 0x7fe60015acb0 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.828+0000 7fe606400640 1 -- 172.21.15.97:0/1956043045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60015acb0 msgr2=0x7fe6000b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.828+0000 7fe606400640 1 --2- 172.21.15.97:0/1956043045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60015acb0 0x7fe6000b37d0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009870 tx=0x7fe5f002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.829+0000 7fe606400640 1 -- 172.21.15.97:0/1956043045 shutdown_connections 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.829+0000 7fe606400640 1 --2- 172.21.15.97:0/1956043045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60015acb0 0x7fe6000b37d0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.829+0000 7fe606400640 1 -- 172.21.15.97:0/1956043045 >> 172.21.15.97:0/1956043045 conn(0x7fe6000b1730 msgr2=0x7fe6000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.829+0000 7fe606400640 1 -- 172.21.15.97:0/1956043045 shutdown_connections 2024-03-06T07:40:43.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.829+0000 7fe606400640 1 -- 172.21.15.97:0/1956043045 wait complete. 2024-03-06T07:40:43.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.830+0000 7fe606400640 1 Processor -- start 2024-03-06T07:40:43.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.830+0000 7fe606400640 1 -- start start 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.831+0000 7fe606400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 0x7fe60015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.831+0000 7fe606400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe6000b4680 0x7fe6000ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.831+0000 7fe606400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe60015acb0 0x7fe6001484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.831+0000 7fe606400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b3d10 con 0x7fe60006ec00 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.831+0000 7fe606400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b32b0 con 0x7fe60015acb0 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.831+0000 7fe606400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b4010 con 0x7fe6000b4680 2024-03-06T07:40:43.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 0x7fe60015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 0x7fe60015b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60414/0 (socket says 172.21.15.97:60414) 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 -- 172.21.15.97:0/1955281815 learned_addr learned my addr 172.21.15.97:0/1955281815 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ffe00640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe60015acb0 0x7fe6001484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5fea00640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe6000b4680 0x7fe6000ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 -- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe6000b4680 msgr2=0x7fe6000ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe6000b4680 0x7fe6000ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 -- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe60015acb0 msgr2=0x7fe6001484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe60015acb0 0x7fe6001484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 -- 172.21.15.97:0/1955281815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000ca610 con 0x7fe60006ec00 2024-03-06T07:40:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ffe00640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe60015acb0 0x7fe6001484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:43.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.832+0000 7fe5ff400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 0x7fe60015b450 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0000c00 tx=0x7fe5f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:43.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.833+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0048070 con 0x7fe60006ec00 2024-03-06T07:40:43.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.833+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0030660 con 0x7fe60006ec00 2024-03-06T07:40:43.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.833+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0034040 con 0x7fe60006ec00 2024-03-06T07:40:43.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.833+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe600148a10 con 0x7fe60006ec00 2024-03-06T07:40:43.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.833+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6000cab90 con 0x7fe60006ec00 2024-03-06T07:40:43.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.834+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60004f330 con 0x7fe60006ec00 2024-03-06T07:40:43.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.835+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5f0037070 con 0x7fe60006ec00 2024-03-06T07:40:43.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.836+0000 7fe5f7400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:43.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.836+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe5f00ac1d0 con 0x7fe60006ec00 2024-03-06T07:40:43.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.836+0000 7fe5fea00640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:43.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.837+0000 7fe5fea00640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe5e006b4d0 0x7fe5e006d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe60006e0e0 tx=0x7fe5ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:43.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:43.841+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5f007b140 con 0x7fe60006ec00 2024-03-06T07:40:44.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.078+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe60006ce60 con 0x7fe60006ec00 2024-03-06T07:40:44.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.079+0000 7fe5f7400640 1 -- 172.21.15.97:0/1955281815 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fe5f007aa40 con 0x7fe60006ec00 2024-03-06T07:40:44.081 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:44.081 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:44.081 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.082+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe5e006b4d0 msgr2=0x7fe5e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.082+0000 7fe606400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe5e006b4d0 0x7fe5e006d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe60006e0e0 tx=0x7fe5ec009040 comp rx=0 tx=0).stop 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 msgr2=0x7fe60015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 0x7fe60015b450 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0000c00 tx=0x7fe5f0004730 comp rx=0 tx=0).stop 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 shutdown_connections 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe60015acb0 0x7fe6001484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe6000b4680 0x7fe6000ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 --2- 172.21.15.97:0/1955281815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe60006ec00 0x7fe60015b450 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 >> 172.21.15.97:0/1955281815 conn(0x7fe6000b1730 msgr2=0x7fe600155e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 shutdown_connections 2024-03-06T07:40:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:44.083+0000 7fe606400640 1 -- 172.21.15.97:0/1955281815 wait complete. 2024-03-06T07:40:44.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:44.305 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:45.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:45.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.517+0000 7f9df0c00640 1 Processor -- start 2024-03-06T07:40:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.517+0000 7f9df0c00640 1 -- start start 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9df0c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9dec0b4680 0x7f9dec06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9df0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 0x7f9dec06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9df0c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec15b090 con 0x7f9dec15acb0 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9df0c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dec0b2a30 con 0x7f9dec0b4680 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9df0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec0b4360 con 0x7f9dec072340 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9de9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9dec15acb0 0x7f9dec0b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43014/0 (socket says 172.21.15.97:43014) 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.519+0000 7f9de9e00640 1 -- 172.21.15.97:0/3534863324 learned_addr learned my addr 172.21.15.97:0/3534863324 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9dea800640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 0x7f9dec06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de9400640 1 --2- 172.21.15.97:0/3534863324 >> v2:172.21.15.135:3300/0 conn(0x7f9dec0b4680 0x7f9dec06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64055207 0 0) 0x7f9dec15b090 con 0x7f9dec15acb0 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd0003520 con 0x7f9dec15acb0 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225128856 0 0) 0x7f9dd0003520 con 0x7f9dec15acb0 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 msgr2=0x7f9dec06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 0x7f9dec06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 >> v2:172.21.15.135:3300/0 conn(0x7f9dec0b4680 msgr2=0x7f9dec06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 --2- 172.21.15.97:0/3534863324 >> v2:172.21.15.135:3300/0 conn(0x7f9dec0b4680 0x7f9dec06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec0b2cf0 con 0x7f9dec15acb0 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f9ddc001670 con 0x7f9dec15acb0 2024-03-06T07:40:45.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 >> v1:172.21.15.97:6789/0 conn(0x7f9dec15acb0 legacy=0x7f9dec0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd00039b0 0x7f9dd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 0x7f9dd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec0b2a30 con 0x7f9dd0004d40 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec0b4360 con 0x7f9dd00039b0 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.520+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd000dbd0 con 0x7f9dec072340 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9dea800640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9de9e00640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd00039b0 0x7f9dd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9de9400640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 0x7f9dd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9dea800640 1 -- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 msgr2=0x7f9dd0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:45.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9dea800640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec072340 0x7f9dd0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9dea800640 1 -- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd00039b0 msgr2=0x7f9dd0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9dea800640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd00039b0 0x7f9dd0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.521+0000 7f9dea800640 1 -- 172.21.15.97:0/3534863324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0003520 con 0x7f9dd0004d40 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.522+0000 7f9dea800640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dd00051c0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009fc0 tx=0x7f9dd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5db92e12ddd6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4039070 con 0x7f9dd0004d40 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de402cc20 con 0x7f9dd0004d40 2024-03-06T07:40:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9de8a00640 1 -- 172.21.15.97:0/3534863324 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4009c10 con 0x7f9dd0004d40 2024-03-06T07:40:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9df0c00640 1 -- 172.21.15.97:0/3534863324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 msgr2=0x7f9dd00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9df0c00640 1 --2- 172.21.15.97:0/3534863324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dd00051c0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009fc0 tx=0x7f9dd0005830 comp rx=0 tx=0).stop 2024-03-06T07:40:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9dea800640 1 -- 172.21.15.97:0/3534863324 reap_dead start 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9df0c00640 1 -- 172.21.15.97:0/3534863324 shutdown_connections 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9df0c00640 1 -- 172.21.15.97:0/3534863324 >> 172.21.15.97:0/3534863324 conn(0x7f9dec0b1730 msgr2=0x7f9dec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.523+0000 7f9df0c00640 1 -- 172.21.15.97:0/3534863324 shutdown_connections 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.524+0000 7f9df0c00640 1 -- 172.21.15.97:0/3534863324 wait complete. 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.525+0000 7f9df0c00640 1 Processor -- start 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.525+0000 7f9df0c00640 1 -- start start 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.526+0000 7f9df0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dec149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.526+0000 7f9df0c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dec14dda0 0x7f9dec14e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.526+0000 7f9df0c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec14bf50 0x7f9dec14c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.526+0000 7f9df0c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec0b4010 con 0x7f9dd0004d40 2024-03-06T07:40:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.526+0000 7f9df0c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec0b32b0 con 0x7f9dec14dda0 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.526+0000 7f9df0c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec06ea30 con 0x7f9dec14bf50 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dec149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dec149870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60430/0 (socket says 172.21.15.97:60430) 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 -- 172.21.15.97:0/1148404660 learned_addr learned my addr 172.21.15.97:0/1148404660 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 -- 172.21.15.97:0/1148404660 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec14bf50 msgr2=0x7f9dec14c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec14bf50 0x7f9dec14c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 -- 172.21.15.97:0/1148404660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dec14dda0 msgr2=0x7f9dec14e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dec14dda0 0x7f9dec14e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 -- 172.21.15.97:0/1148404660 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec14c940 con 0x7f9dd0004d40 2024-03-06T07:40:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de9e00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dec149870 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00e870 tx=0x7f9ddc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc00c7f0 con 0x7f9dd0004d40 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.527+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ddc018070 con 0x7f9dd0004d40 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.528+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc00cbd0 con 0x7f9dd0004d40 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.528+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dec0ce1e0 con 0x7f9dd0004d40 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.528+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec0ce7a0 con 0x7f9dd0004d40 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.529+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dec04f330 con 0x7f9dd0004d40 2024-03-06T07:40:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.530+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ddc01d460 con 0x7f9dd0004d40 2024-03-06T07:40:45.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.531+0000 7f9de2000640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9dcc06b500 0x7f9dcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.531+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9ddc08a1e0 con 0x7f9dd0004d40 2024-03-06T07:40:45.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.531+0000 7f9de9400640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9dcc06b500 0x7f9dcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.532+0000 7f9de9400640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9dcc06b500 0x7f9dcc06d9c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9dec14ae30 tx=0x7f9dd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.535+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dec14c940 con 0x7f9dd0004d40 2024-03-06T07:40:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.768+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dec0ce9a0 con 0x7f9dd0004d40 2024-03-06T07:40:45.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.769+0000 7f9de2000640 1 -- 172.21.15.97:0/1148404660 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f9ddc058fa0 con 0x7f9dd0004d40 2024-03-06T07:40:45.771 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:45.771 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:45.771 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:45.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9dcc06b500 msgr2=0x7f9dcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9dcc06b500 0x7f9dcc06d9c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9dec14ae30 tx=0x7f9dd8008040 comp rx=0 tx=0).stop 2024-03-06T07:40:45.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 msgr2=0x7f9dec149870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dec149870 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00e870 tx=0x7f9ddc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:40:45.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 shutdown_connections 2024-03-06T07:40:45.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9dcc06b500 0x7f9dcc06d9c0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9dec14bf50 0x7f9dec14c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dec14dda0 0x7f9dec14e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 --2- 172.21.15.97:0/1148404660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9dd0004d40 0x7f9dec149870 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 >> 172.21.15.97:0/1148404660 conn(0x7f9dec0b1730 msgr2=0x7f9dec071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:45.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.771+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 shutdown_connections 2024-03-06T07:40:45.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:45.772+0000 7f9df0c00640 1 -- 172.21.15.97:0/1148404660 wait complete. 2024-03-06T07:40:45.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:45.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:46.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:47.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.216+0000 7f5b62e00640 1 Processor -- start 2024-03-06T07:40:47.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.216+0000 7f5b62e00640 1 -- start start 2024-03-06T07:40:47.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b62e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5b5c0d4830 0x7f5b5c0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b62e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c15f030 con 0x7f5b5c0d0600 2024-03-06T07:40:47.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b62e00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c15f310 con 0x7f5b5c0b2a60 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b62e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5c0d4cc0 con 0x7f5b5c0d4830 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b61a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5b5c0d0600 0x7f5b5c0d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43022/0 (socket says 172.21.15.97:43022) 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b61a00640 1 -- 172.21.15.97:0/410925871 learned_addr learned my addr 172.21.15.97:0/410925871 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.218+0000 7f5b62400640 1 --2- 172.21.15.97:0/410925871 >> v2:172.21.15.97:3300/0 conn(0x7f5b5c0d4830 0x7f5b5c0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.219+0000 7f5b5be00640 1 -- 172.21.15.97:0/410925871 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857777334 0 0) 0x7f5b5c15f030 con 0x7f5b5c0d0600 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.219+0000 7f5b5be00640 1 -- 172.21.15.97:0/410925871 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b44003520 con 0x7f5b5c0d0600 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.219+0000 7f5b62400640 1 -- 172.21.15.97:0/410925871 >> v1:172.21.15.135:6789/0 conn(0x7f5b5c0b2a60 legacy=0x7f5b5c0d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.219+0000 7f5b62400640 1 -- 172.21.15.97:0/410925871 >> v1:172.21.15.97:6789/0 conn(0x7f5b5c0d0600 legacy=0x7f5b5c0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.219+0000 7f5b62400640 1 -- 172.21.15.97:0/410925871 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0cea40 con 0x7f5b5c0d4830 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.220+0000 7f5b62400640 1 --2- 172.21.15.97:0/410925871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0cf520 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f5b540097a0 tx=0x7f5b5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613022d3b9bdf792 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.220+0000 7f5b5be00640 1 -- 172.21.15.97:0/410925871 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54035020 con 0x7f5b5c0d4830 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.220+0000 7f5b5be00640 1 -- 172.21.15.97:0/410925871 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54031830 con 0x7f5b5c0d4830 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.220+0000 7f5b5be00640 1 -- 172.21.15.97:0/410925871 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54031c00 con 0x7f5b5c0d4830 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 -- 172.21.15.97:0/410925871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d4830 msgr2=0x7f5b5c0cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 --2- 172.21.15.97:0/410925871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0cf520 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f5b540097a0 tx=0x7f5b5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 -- 172.21.15.97:0/410925871 shutdown_connections 2024-03-06T07:40:47.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 --2- 172.21.15.97:0/410925871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0cf520 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 -- 172.21.15.97:0/410925871 >> 172.21.15.97:0/410925871 conn(0x7f5b5c0d3d10 msgr2=0x7f5b5c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:47.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 -- 172.21.15.97:0/410925871 shutdown_connections 2024-03-06T07:40:47.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.221+0000 7f5b62e00640 1 -- 172.21.15.97:0/410925871 wait complete. 2024-03-06T07:40:47.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.222+0000 7f5b62e00640 1 Processor -- start 2024-03-06T07:40:47.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.223+0000 7f5b62e00640 1 -- start start 2024-03-06T07:40:47.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.223+0000 7f5b62e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b5c0b2a60 0x7f5b5c14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.223+0000 7f5b62e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 0x7f5b5c0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b62e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b62e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0b3250 con 0x7f5b5c0d0600 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b62e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0cf090 con 0x7f5b5c0d4830 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b62e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c15ecf0 con 0x7f5b5c0b2a60 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 0x7f5b5c0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 0x7f5b5c0b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60448/0 (socket says 172.21.15.97:60448) 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 -- 172.21.15.97:0/810037391 learned_addr learned my addr 172.21.15.97:0/810037391 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b62400640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61a00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b5c0b2a60 0x7f5b5c14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 -- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b5c0b2a60 msgr2=0x7f5b5c14e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b5c0b2a60 0x7f5b5c14e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 -- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b5c0d4830 msgr2=0x7f5b5c0b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61000640 1 -- 172.21.15.97:0/810037391 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0c9d30 con 0x7f5b5c0d0600 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.224+0000 7f5b61a00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b5c0b2a60 0x7f5b5c14e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.225+0000 7f5b61000640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 0x7f5b5c0b33d0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000c7c0 tx=0x7f5b5000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.225+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50016020 con 0x7f5b5c0d0600 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.225+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5000aa50 con 0x7f5b5c0d0600 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.225+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50012420 con 0x7f5b5c0d0600 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.225+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c0b7530 con 0x7f5b5c0d0600 2024-03-06T07:40:47.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.225+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c0b7af0 con 0x7f5b5c0d0600 2024-03-06T07:40:47.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.227+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c04f330 con 0x7f5b5c0d0600 2024-03-06T07:40:47.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.227+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b50019020 con 0x7f5b5c0d0600 2024-03-06T07:40:47.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.228+0000 7f5b59600640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b4006b500 0x7f5b4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.228+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5b5008a0e0 con 0x7f5b5c0d0600 2024-03-06T07:40:47.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.229+0000 7f5b61a00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b4006b500 0x7f5b4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.230+0000 7f5b61a00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b4006b500 0x7f5b4006d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c005880 tx=0x7f5b4c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:47.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.233+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b50020070 con 0x7f5b5c0d0600 2024-03-06T07:40:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.482+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b5c0b7cc0 con 0x7f5b5c0d0600 2024-03-06T07:40:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.483+0000 7f5b59600640 1 -- 172.21.15.97:0/810037391 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f5b50058e10 con 0x7f5b5c0d0600 2024-03-06T07:40:47.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:47.486 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b4006b500 msgr2=0x7f5b4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b4006b500 0x7f5b4006d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c005880 tx=0x7f5b4c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 msgr2=0x7f5b5c0b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 0x7f5b5c0b33d0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000c7c0 tx=0x7f5b5000cc80 comp rx=0 tx=0).stop 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 shutdown_connections 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5b4006b500 0x7f5b4006d9c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b5c0d4830 0x7f5b5c0b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b5c0d0600 0x7f5b5c0b33d0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 --2- 172.21.15.97:0/810037391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b5c0b2a60 0x7f5b5c14e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 >> 172.21.15.97:0/810037391 conn(0x7f5b5c0d3d10 msgr2=0x7f5b5c0b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:47.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.485+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 shutdown_connections 2024-03-06T07:40:47.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:47.486+0000 7f5b62e00640 1 -- 172.21.15.97:0/810037391 wait complete. 2024-03-06T07:40:47.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:47.713 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:48.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:48.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.921+0000 7f63d4800640 1 Processor -- start 2024-03-06T07:40:48.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.922+0000 7f63d4800640 1 -- start start 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d4800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f63cc151660 0x7f63cc071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d4800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63cc15dff0 0x7f63cc15e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d4800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc071a60 con 0x7f63cc0cf430 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d4800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc0b3ee0 con 0x7f63cc15dff0 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d4800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc0b35b0 con 0x7f63cc151660 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d2800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63cc15dff0 0x7f63cc15e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d1400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f63cc0cf430 0x7f63cc06d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43026/0 (socket says 172.21.15.97:43026) 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d2800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63cc15dff0 0x7f63cc15e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60458/0 (socket says 172.21.15.97:60458) 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d1400640 1 -- 172.21.15.97:0/3648582095 learned_addr learned my addr 172.21.15.97:0/3648582095 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.925+0000 7f63d1e00640 1 --2- 172.21.15.97:0/3648582095 >> v2:172.21.15.135:3300/0 conn(0x7f63cc151660 0x7f63cc071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.926+0000 7f63d2800640 1 -- 172.21.15.97:0/3648582095 >> v2:172.21.15.135:3300/0 conn(0x7f63cc151660 msgr2=0x7f63cc071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:48.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.926+0000 7f63d2800640 1 --2- 172.21.15.97:0/3648582095 >> v2:172.21.15.135:3300/0 conn(0x7f63cc151660 0x7f63cc071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:48.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.926+0000 7f63d2800640 1 -- 172.21.15.97:0/3648582095 >> v1:172.21.15.97:6789/0 conn(0x7f63cc0cf430 legacy=0x7f63cc06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:48.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.926+0000 7f63d2800640 1 -- 172.21.15.97:0/3648582095 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0b3890 con 0x7f63cc15dff0 2024-03-06T07:40:48.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.926+0000 7f63d0a00640 1 -- 172.21.15.97:0/3648582095 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603944701 0 0) 0x7f63cc071a60 con 0x7f63cc0cf430 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.927+0000 7f63d2800640 1 --2- 172.21.15.97:0/3648582095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc15dff0 0x7f63cc15e480 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009d90 tx=0x7f63c802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c83b5c2e439fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.927+0000 7f63d0a00640 1 -- 172.21.15.97:0/3648582095 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8039070 con 0x7f63cc15dff0 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.927+0000 7f63d0a00640 1 -- 172.21.15.97:0/3648582095 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c802cb40 con 0x7f63cc15dff0 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.927+0000 7f63d0a00640 1 -- 172.21.15.97:0/3648582095 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8034440 con 0x7f63cc15dff0 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 -- 172.21.15.97:0/3648582095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc15dff0 msgr2=0x7f63cc15e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 --2- 172.21.15.97:0/3648582095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc15dff0 0x7f63cc15e480 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009d90 tx=0x7f63c802bd90 comp rx=0 tx=0).stop 2024-03-06T07:40:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 -- 172.21.15.97:0/3648582095 shutdown_connections 2024-03-06T07:40:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 --2- 172.21.15.97:0/3648582095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc15dff0 0x7f63cc15e480 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 --2- 172.21.15.97:0/3648582095 >> v2:172.21.15.135:3300/0 conn(0x7f63cc151660 0x7f63cc071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 -- 172.21.15.97:0/3648582095 >> 172.21.15.97:0/3648582095 conn(0x7f63cc0b1730 msgr2=0x7f63cc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 -- 172.21.15.97:0/3648582095 shutdown_connections 2024-03-06T07:40:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.928+0000 7f63d4800640 1 -- 172.21.15.97:0/3648582095 wait complete. 2024-03-06T07:40:48.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.930+0000 7f63d4800640 1 Processor -- start 2024-03-06T07:40:48.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.930+0000 7f63d4800640 1 -- start start 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d4800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 0x7f63cc155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d4800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63cc151660 0x7f63cc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d4800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63cc15dff0 0x7f63cc148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d4800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc06daf0 con 0x7f63cc0cf430 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d4800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc071d30 con 0x7f63cc151660 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d4800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc06cb50 con 0x7f63cc15dff0 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 0x7f63cc155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 0x7f63cc155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60466/0 (socket says 172.21.15.97:60466) 2024-03-06T07:40:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 -- 172.21.15.97:0/4013710976 learned_addr learned my addr 172.21.15.97:0/4013710976 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:48.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 -- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63cc15dff0 msgr2=0x7f63cc148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63cc15dff0 0x7f63cc148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 -- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63cc151660 msgr2=0x7f63cc0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1400640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63cc151660 0x7f63cc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63cc151660 0x7f63cc0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.931+0000 7f63d1e00640 1 -- 172.21.15.97:0/4013710976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc155570 con 0x7f63cc0cf430 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.932+0000 7f63d1e00640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 0x7f63cc155030 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f63c000d800 tx=0x7f63c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.932+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000b6c0 con 0x7f63cc0cf430 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.932+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0019070 con 0x7f63cc0cf430 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.932+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000ba80 con 0x7f63cc0cf430 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.932+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc14f7c0 con 0x7f63cc0cf430 2024-03-06T07:40:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.932+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc14fd80 con 0x7f63cc0cf430 2024-03-06T07:40:48.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.933+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc04f330 con 0x7f63cc0cf430 2024-03-06T07:40:48.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.934+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63c000bc20 con 0x7f63cc0cf430 2024-03-06T07:40:48.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.935+0000 7f63c6000640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63b006b500 0x7f63b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:48.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.935+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f63c008a6e0 con 0x7f63cc0cf430 2024-03-06T07:40:48.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.935+0000 7f63d1400640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63b006b500 0x7f63b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:48.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.936+0000 7f63d1400640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63b006b500 0x7f63b006d9c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f63bc007fd0 tx=0x7f63bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:48.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:48.944+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63c0059530 con 0x7f63cc0cf430 2024-03-06T07:40:49.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.179+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63cc14a240 con 0x7f63cc0cf430 2024-03-06T07:40:49.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.180+0000 7f63c6000640 1 -- 172.21.15.97:0/4013710976 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f63c0058e30 con 0x7f63cc0cf430 2024-03-06T07:40:49.182 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:49.182 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:49.182 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.182+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63b006b500 msgr2=0x7f63b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.182+0000 7f63d4800640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63b006b500 0x7f63b006d9c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f63bc007fd0 tx=0x7f63bc009040 comp rx=0 tx=0).stop 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.182+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 msgr2=0x7f63cc155030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.182+0000 7f63d4800640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 0x7f63cc155030 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f63c000d800 tx=0x7f63c000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 shutdown_connections 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f63b006b500 0x7f63b006d9c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f63cc15dff0 0x7f63cc148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63cc151660 0x7f63cc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 --2- 172.21.15.97:0/4013710976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc0cf430 0x7f63cc155030 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 >> 172.21.15.97:0/4013710976 conn(0x7f63cc0b1730 msgr2=0x7f63cc14a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:49.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 shutdown_connections 2024-03-06T07:40:49.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:49.183+0000 7f63d4800640 1 -- 172.21.15.97:0/4013710976 wait complete. 2024-03-06T07:40:49.404 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:49.404 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:50.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:50.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.611+0000 7fc2f7600640 1 Processor -- start 2024-03-06T07:40:50.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.611+0000 7fc2f7600640 1 -- start start 2024-03-06T07:40:50.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.612+0000 7fc2f7600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc2f00d0ca0 0x7fc2f00cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.612+0000 7fc2f7600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f015d4f0 con 0x7fc2f0160530 2024-03-06T07:40:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.612+0000 7fc2f7600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f015cc50 con 0x7fc2f006b6d0 2024-03-06T07:40:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.613+0000 7fc2f7600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f00b50a0 con 0x7fc2f00d0ca0 2024-03-06T07:40:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.613+0000 7fc2efe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc2f0160530 0x7fc2f006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43030/0 (socket says 172.21.15.97:43030) 2024-03-06T07:40:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.613+0000 7fc2efe00640 1 -- 172.21.15.97:0/502741909 learned_addr learned my addr 172.21.15.97:0/502741909 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.613+0000 7fc2f4c00640 1 --2- 172.21.15.97:0/502741909 >> v2:172.21.15.135:3300/0 conn(0x7fc2f00d0ca0 0x7fc2f00cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.613+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812070052 0 0) 0x7fc2f015d4f0 con 0x7fc2f0160530 2024-03-06T07:40:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d8003520 con 0x7fc2f0160530 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2955294659 0 0) 0x7fc2d8003520 con 0x7fc2f0160530 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 >> v2:172.21.15.135:3300/0 conn(0x7fc2f00d0ca0 msgr2=0x7fc2f00cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 --2- 172.21.15.97:0/502741909 >> v2:172.21.15.135:3300/0 conn(0x7fc2f00d0ca0 0x7fc2f00cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 >> v1:172.21.15.135:6789/0 conn(0x7fc2f006b6d0 legacy=0x7fc2f00b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f015cf30 con 0x7fc2f0160530 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fc2dc001670 con 0x7fc2f0160530 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 >> v1:172.21.15.97:6789/0 conn(0x7fc2f0160530 legacy=0x7fc2f006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2d8003a00 0x7fc2d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004800 0x7fc2d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d8005210 0x7fc2d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00b50a0 con 0x7fc2d8005210 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d800c160 con 0x7fc2d8004800 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d800c360 con 0x7fc2d8003a00 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.614+0000 7fc2f5600640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d8005210 0x7fc2d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2efe00640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004800 0x7fc2d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2f4c00640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2d8003a00 0x7fc2d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2f5600640 1 -- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2d8003a00 msgr2=0x7fc2d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2f5600640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2d8003a00 0x7fc2d8003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2f5600640 1 -- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004800 msgr2=0x7fc2d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2f5600640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8004800 0x7fc2d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.615+0000 7fc2f5600640 1 -- 172.21.15.97:0/502741909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8003520 con 0x7fc2d8005210 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.616+0000 7fc2f5600640 1 -- 172.21.15.97:0/502741909 reap_dead start 2024-03-06T07:40:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.616+0000 7fc2f5600640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d8005210 0x7fc2d800b9c0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00712c0 tx=0x7fc2d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b81146ce0c5028e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.616+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8001a70 con 0x7fc2d8005210 2024-03-06T07:40:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.616+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8003b70 con 0x7fc2d8005210 2024-03-06T07:40:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.616+0000 7fc2ef400640 1 -- 172.21.15.97:0/502741909 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80050d0 con 0x7fc2d8005210 2024-03-06T07:40:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 -- 172.21.15.97:0/502741909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d8005210 msgr2=0x7fc2d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d8005210 0x7fc2d800b9c0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00712c0 tx=0x7fc2d8005830 comp rx=0 tx=0).stop 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 -- 172.21.15.97:0/502741909 shutdown_connections 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 --2- 172.21.15.97:0/502741909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2d8005210 0x7fc2d800b9c0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 -- 172.21.15.97:0/502741909 >> 172.21.15.97:0/502741909 conn(0x7fc2f00b1e10 msgr2=0x7fc2f00b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 -- 172.21.15.97:0/502741909 shutdown_connections 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.617+0000 7fc2f7600640 1 -- 172.21.15.97:0/502741909 wait complete. 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.619+0000 7fc2f7600640 1 Processor -- start 2024-03-06T07:40:50.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.619+0000 7fc2f7600640 1 -- start start 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f7600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8005210 0x7fc2f00c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f7600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 0x7fc2f00af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f7600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f014e300 0x7fc2f01e6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f7600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00d1080 con 0x7fc2f014dea0 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f7600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0160990 con 0x7fc2d8005210 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f7600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00cfe50 con 0x7fc2f014e300 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 0x7fc2f00af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 0x7fc2f00af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60474/0 (socket says 172.21.15.97:60474) 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2efe00640 1 -- 172.21.15.97:0/2953403675 learned_addr learned my addr 172.21.15.97:0/2953403675 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f4c00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8005210 0x7fc2f00c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.621+0000 7fc2f5600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f014e300 0x7fc2f01e6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.622+0000 7fc2efe00640 1 -- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f014e300 msgr2=0x7fc2f01e6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.622+0000 7fc2efe00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f014e300 0x7fc2f01e6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.622+0000 7fc2efe00640 1 -- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8005210 msgr2=0x7fc2f00c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.622+0000 7fc2efe00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8005210 0x7fc2f00c2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.622+0000 7fc2efe00640 1 -- 172.21.15.97:0/2953403675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f00c4220 con 0x7fc2f014dea0 2024-03-06T07:40:50.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.622+0000 7fc2efe00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 0x7fc2f00af200 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e730 tx=0x7fc2dc00eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.623+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc00c660 con 0x7fc2f014dea0 2024-03-06T07:40:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.623+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc018070 con 0x7fc2f014dea0 2024-03-06T07:40:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.623+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc00ca40 con 0x7fc2f014dea0 2024-03-06T07:40:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.623+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f00c4500 con 0x7fc2f014dea0 2024-03-06T07:40:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.623+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f00c4ac0 con 0x7fc2f014dea0 2024-03-06T07:40:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.624+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f004f330 con 0x7fc2f014dea0 2024-03-06T07:40:50.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.625+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2dc00cbe0 con 0x7fc2f014dea0 2024-03-06T07:40:50.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.626+0000 7fc2ecc00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b4d0 0x7fc2d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.626+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc2dc08a0a0 con 0x7fc2f014dea0 2024-03-06T07:40:50.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.626+0000 7fc2f4c00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b4d0 0x7fc2d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.627+0000 7fc2f4c00640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b4d0 0x7fc2d006d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009a10 tx=0x7fc2e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.630+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2dc058e70 con 0x7fc2f014dea0 2024-03-06T07:40:50.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.864+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2f00715f0 con 0x7fc2f014dea0 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.865+0000 7fc2ecc00640 1 -- 172.21.15.97:0/2953403675 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fc2dc058770 con 0x7fc2f014dea0 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b4d0 msgr2=0x7fc2d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b4d0 0x7fc2d006d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009a10 tx=0x7fc2e4008040 comp rx=0 tx=0).stop 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 msgr2=0x7fc2f00af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 0x7fc2f00af200 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e730 tx=0x7fc2dc00eb30 comp rx=0 tx=0).stop 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 shutdown_connections 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc2d006b4d0 0x7fc2d006d990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc2f014e300 0x7fc2f01e6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2f014dea0 0x7fc2f00af200 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 --2- 172.21.15.97:0/2953403675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2d8005210 0x7fc2f00c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 >> 172.21.15.97:0/2953403675 conn(0x7fc2f00b1e10 msgr2=0x7fc2f006d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.867+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 shutdown_connections 2024-03-06T07:40:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:50.868+0000 7fc2f7600640 1 -- 172.21.15.97:0/2953403675 wait complete. 2024-03-06T07:40:51.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:51.090 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:52.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:52.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.305+0000 7f30a1600640 1 Processor -- start 2024-03-06T07:40:52.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.305+0000 7f30a1600640 1 -- start start 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.306+0000 7f30a1600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f309c0d24c0 0x7f309c0d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.306+0000 7f30a1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 0x7f309c0d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.306+0000 7f30a1600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c0d2de0 con 0x7f309c0d07f0 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.306+0000 7f30a1600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0d4a00 con 0x7f309c0d24c0 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.306+0000 7f30a1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0cf7e0 con 0x7f309c0b2de0 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.307+0000 7f309b400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f309c0d07f0 0x7f309c0d0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:43042/0 (socket says 172.21.15.97:43042) 2024-03-06T07:40:52.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.307+0000 7f309b400640 1 -- 172.21.15.97:0/2325513371 learned_addr learned my addr 172.21.15.97:0/2325513371 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:52.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.307+0000 7f309aa00640 1 --2- 172.21.15.97:0/2325513371 >> v2:172.21.15.135:3300/0 conn(0x7f309c0d24c0 0x7f309c0d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.307+0000 7f309a000640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 0x7f309c0d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.307+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758777897 0 0) 0x7f309c0d2de0 con 0x7f309c0d07f0 2024-03-06T07:40:52.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.307+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3080003520 con 0x7f309c0d07f0 2024-03-06T07:40:52.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440533020 0 0) 0x7f3080003520 con 0x7f309c0d07f0 2024-03-06T07:40:52.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 msgr2=0x7f309c0d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 0x7f309c0d28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 >> v2:172.21.15.135:3300/0 conn(0x7f309c0d24c0 msgr2=0x7f309c0d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 --2- 172.21.15.97:0/2325513371 >> v2:172.21.15.135:3300/0 conn(0x7f309c0d24c0 0x7f309c0d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0d4c50 con 0x7f309c0d07f0 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f30940019f0 con 0x7f309c0d07f0 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 >> v1:172.21.15.97:6789/0 conn(0x7f309c0d07f0 legacy=0x7f309c0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f3080003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3080004800 0x7f3080004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 0x7f30800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0d4a00 con 0x7f30800039b0 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0cf7e0 con 0x7f3080004800 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308000b9c0 con 0x7f309c0b2de0 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.308+0000 7f309aa00640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f3080003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309b400640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 0x7f30800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309a000640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3080004800 0x7f3080004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309aa00640 1 -- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 msgr2=0x7f30800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309aa00640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c0b2de0 0x7f30800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309aa00640 1 -- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3080004800 msgr2=0x7f3080004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309aa00640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3080004800 0x7f3080004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309aa00640 1 -- 172.21.15.97:0/2325513371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3080003520 con 0x7f30800039b0 2024-03-06T07:40:52.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.309+0000 7f309a000640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3080004800 0x7f3080004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.310+0000 7f309aa00640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f3080003dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f308c009af0 tx=0x7f3080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca85ec6b92b4b9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c039070 con 0x7f30800039b0 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c02c710 con 0x7f30800039b0 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f3099600640 1 -- 172.21.15.97:0/2325513371 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c009690 con 0x7f30800039b0 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f30a1600640 1 -- 172.21.15.97:0/2325513371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 msgr2=0x7f3080003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f30a1600640 1 --2- 172.21.15.97:0/2325513371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f3080003dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f308c009af0 tx=0x7f3080005830 comp rx=0 tx=0).stop 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f309b400640 1 -- 172.21.15.97:0/2325513371 reap_dead start 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f30a1600640 1 -- 172.21.15.97:0/2325513371 shutdown_connections 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.311+0000 7f30a1600640 1 -- 172.21.15.97:0/2325513371 >> 172.21.15.97:0/2325513371 conn(0x7f309c0d3ee0 msgr2=0x7f309c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.312+0000 7f30a1600640 1 -- 172.21.15.97:0/2325513371 shutdown_connections 2024-03-06T07:40:52.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.312+0000 7f30a1600640 1 -- 172.21.15.97:0/2325513371 wait complete. 2024-03-06T07:40:52.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.313+0000 7f30a1600640 1 Processor -- start 2024-03-06T07:40:52.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.313+0000 7f30a1600640 1 -- start start 2024-03-06T07:40:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.314+0000 7f30a1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f309c152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.314+0000 7f30a1600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c15b600 0x7f309c15e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.314+0000 7f30a1600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c0c9be0 0x7f309c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.314+0000 7f30a1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0cfd10 con 0x7f30800039b0 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.314+0000 7f30a1600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0d0080 con 0x7f309c0c9be0 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.314+0000 7f30a1600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c15e5b0 con 0x7f309c15b600 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f309c152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f309c152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60486/0 (socket says 172.21.15.97:60486) 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 -- 172.21.15.97:0/3595049949 learned_addr learned my addr 172.21.15.97:0/3595049949 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309b400640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c0c9be0 0x7f309c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309a000640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c15b600 0x7f309c15e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 -- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c15b600 msgr2=0x7f309c15e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c15b600 0x7f309c15e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 -- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c0c9be0 msgr2=0x7f309c0ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c0c9be0 0x7f309c0ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.315+0000 7f309aa00640 1 -- 172.21.15.97:0/3595049949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0b6b40 con 0x7f30800039b0 2024-03-06T07:40:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.316+0000 7f309aa00640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f309c152360 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f308c037040 tx=0x7f308c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.316+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c039050 con 0x7f30800039b0 2024-03-06T07:40:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.316+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c045070 con 0x7f30800039b0 2024-03-06T07:40:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.316+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c1528a0 con 0x7f30800039b0 2024-03-06T07:40:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.316+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0044e0 con 0x7f30800039b0 2024-03-06T07:40:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.316+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c0b70c0 con 0x7f30800039b0 2024-03-06T07:40:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.317+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c04f330 con 0x7f30800039b0 2024-03-06T07:40:52.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.318+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f308c003a50 con 0x7f30800039b0 2024-03-06T07:40:52.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.319+0000 7f3092a00640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f307c06b4d0 0x7f307c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.320+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f308c0abdc0 con 0x7f30800039b0 2024-03-06T07:40:52.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.320+0000 7f309a000640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f307c06b4d0 0x7f307c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.320+0000 7f309a000640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f307c06b4d0 0x7f307c06d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f309c15ba90 tx=0x7f3088008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:52.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.324+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f308c04e020 con 0x7f30800039b0 2024-03-06T07:40:52.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.580+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f309c0d22a0 con 0x7f30800039b0 2024-03-06T07:40:52.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.581+0000 7f3092a00640 1 -- 172.21.15.97:0/3595049949 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f308c07aca0 con 0x7f30800039b0 2024-03-06T07:40:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:52.583 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:52.583 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:52.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f307c06b4d0 msgr2=0x7f307c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f307c06b4d0 0x7f307c06d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f309c15ba90 tx=0x7f3088008040 comp rx=0 tx=0).stop 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 msgr2=0x7f309c152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f309c152360 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f308c037040 tx=0x7f308c004730 comp rx=0 tx=0).stop 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 shutdown_connections 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f307c06b4d0 0x7f307c06d990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c0c9be0 0x7f309c0ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f309c15b600 0x7f309c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 --2- 172.21.15.97:0/3595049949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30800039b0 0x7f309c152360 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 >> 172.21.15.97:0/3595049949 conn(0x7f309c0d3ee0 msgr2=0x7f309c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 shutdown_connections 2024-03-06T07:40:52.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:52.583+0000 7f30a1600640 1 -- 172.21.15.97:0/3595049949 wait complete. 2024-03-06T07:40:52.804 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:52.804 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:53.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:54.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.018+0000 7f273a400640 1 Processor -- start 2024-03-06T07:40:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.018+0000 7f273a400640 1 -- start start 2024-03-06T07:40:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f273a400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2734147310 0x7f2734072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f273a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 0x7f27340ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f273a400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27340b39c0 con 0x7f273406daa0 2024-03-06T07:40:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f273a400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273406df30 con 0x7f2734147310 2024-03-06T07:40:54.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f273a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340ad9e0 con 0x7f273406d390 2024-03-06T07:40:54.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f272be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f273406daa0 0x7f273406cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45262/0 (socket says 172.21.15.97:45262) 2024-03-06T07:40:54.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.020+0000 7f272be00640 1 -- 172.21.15.97:0/680843732 learned_addr learned my addr 172.21.15.97:0/680843732 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2733400640 1 --2- 172.21.15.97:0/680843732 >> v2:172.21.15.135:3300/0 conn(0x7f2734147310 0x7f2734072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2733e00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 0x7f27340ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314659040 0 0) 0x7f271c001110 con 0x7f273406daa0 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2718003520 con 0x7f273406daa0 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3001939210 0 0) 0x7f2718003520 con 0x7f273406daa0 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 msgr2=0x7f27340ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 0x7f27340ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 >> v2:172.21.15.135:3300/0 conn(0x7f2734147310 msgr2=0x7f2734072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 --2- 172.21.15.97:0/680843732 >> v2:172.21.15.135:3300/0 conn(0x7f2734147310 0x7f2734072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340adcc0 con 0x7f273406daa0 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f271c001840 con 0x7f273406daa0 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 >> v1:172.21.15.97:6789/0 conn(0x7f273406daa0 legacy=0x7f273406cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 0x7f27180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2718003f80 0x7f2718004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718004d40 0x7f27180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340ad9e0 con 0x7f2718003f80 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271800b9c0 con 0x7f2718004d40 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.021+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271800bb40 con 0x7f273406d390 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f272be00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2718003f80 0x7f2718004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f2733e00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718004d40 0x7f27180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f2733400640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 0x7f27180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f272be00640 1 -- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 msgr2=0x7f27180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f272be00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273406d390 0x7f27180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f272be00640 1 -- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718004d40 msgr2=0x7f27180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f272be00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718004d40 0x7f27180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.022+0000 7f272be00640 1 -- 172.21.15.97:0/680843732 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2718003520 con 0x7f2718003f80 2024-03-06T07:40:54.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.024+0000 7f272be00640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2718003f80 0x7f2718004800 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f271c00c320 tx=0x7f2718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0210495af1116d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.024+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c007e60 con 0x7f2718003f80 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.024+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c030070 con 0x7f2718003f80 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.024+0000 7f2732a00640 1 -- 172.21.15.97:0/680843732 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c02ce30 con 0x7f2718003f80 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f273a400640 1 -- 172.21.15.97:0/680843732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2718003f80 msgr2=0x7f2718004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f273a400640 1 --2- 172.21.15.97:0/680843732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2718003f80 0x7f2718004800 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f271c00c320 tx=0x7f2718005830 comp rx=0 tx=0).stop 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f2733e00640 1 -- 172.21.15.97:0/680843732 reap_dead start 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f273a400640 1 -- 172.21.15.97:0/680843732 shutdown_connections 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f273a400640 1 -- 172.21.15.97:0/680843732 >> 172.21.15.97:0/680843732 conn(0x7f27340b1be0 msgr2=0x7f27340b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f273a400640 1 -- 172.21.15.97:0/680843732 shutdown_connections 2024-03-06T07:40:54.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.025+0000 7f273a400640 1 -- 172.21.15.97:0/680843732 wait complete. 2024-03-06T07:40:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.027+0000 7f273a400640 1 Processor -- start 2024-03-06T07:40:54.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.027+0000 7f273a400640 1 -- start start 2024-03-06T07:40:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f273a400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718003f80 0x7f273413c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f273a400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273413fe40 0x7f273413ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f273a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 0x7f273413e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f273a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27341476f0 con 0x7f27341402c0 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f273a400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340ae280 con 0x7f2718003f80 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f273a400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340ae7a0 con 0x7f273413fe40 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 0x7f273413e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 0x7f273413e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34196/0 (socket says 172.21.15.97:34196) 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 -- 172.21.15.97:0/543597565 learned_addr learned my addr 172.21.15.97:0/543597565 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718003f80 0x7f273413c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f272be00640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273413fe40 0x7f273413ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 -- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273413fe40 msgr2=0x7f273413ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273413fe40 0x7f273413ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 -- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718003f80 msgr2=0x7f273413c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718003f80 0x7f273413c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.028+0000 7f2733e00640 1 -- 172.21.15.97:0/543597565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273415e620 con 0x7f27341402c0 2024-03-06T07:40:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f272be00640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273413fe40 0x7f273413ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f2733e00640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 0x7f273413e410 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f27340cfe20 tx=0x7f272c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c011e40 con 0x7f27341402c0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c01b070 con 0x7f27341402c0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c003990 con 0x7f27341402c0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273415e900 con 0x7f27341402c0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.029+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273415eec0 con 0x7f27341402c0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.030+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273404f330 con 0x7f27341402c0 2024-03-06T07:40:54.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.031+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f272c005d10 con 0x7f27341402c0 2024-03-06T07:40:54.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.033+0000 7f272b400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.033+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f272c08a7d0 con 0x7f27341402c0 2024-03-06T07:40:54.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.033+0000 7f2733400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.034+0000 7f2733400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271006b500 0x7f271006d9c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2724009a10 tx=0x7f2724008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:54.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.037+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f272c025020 con 0x7f27341402c0 2024-03-06T07:40:54.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.271+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27340b3000 con 0x7f27341402c0 2024-03-06T07:40:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.272+0000 7f272b400640 1 -- 172.21.15.97:0/543597565 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f272c059590 con 0x7f27341402c0 2024-03-06T07:40:54.275 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:54.275 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271006b500 msgr2=0x7f271006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271006b500 0x7f271006d9c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2724009a10 tx=0x7f2724008040 comp rx=0 tx=0).stop 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 msgr2=0x7f273413e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 0x7f273413e410 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f27340cfe20 tx=0x7f272c0115a0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 shutdown_connections 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27341402c0 0x7f273413e410 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f273413fe40 0x7f273413ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 --2- 172.21.15.97:0/543597565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2718003f80 0x7f273413c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 >> 172.21.15.97:0/543597565 conn(0x7f27340b1be0 msgr2=0x7f2734140830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 shutdown_connections 2024-03-06T07:40:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:54.275+0000 7f273a400640 1 -- 172.21.15.97:0/543597565 wait complete. 2024-03-06T07:40:54.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:54.501 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:55.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:55.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 Processor -- start 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 -- start start 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7c2015f9d0 0x7f7c2015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7c20070b10 0x7f7c20070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2006cef0 con 0x7f7c20144c00 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2006d1d0 con 0x7f7c20070b10 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c27600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2006d4b0 con 0x7f7c2015f9d0 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c25600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7c20070b10 0x7f7c20070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c24c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f7c20144c00 0x7f7c2006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45278/0 (socket says 172.21.15.97:45278) 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c24c00640 1 -- 172.21.15.97:0/1411309667 learned_addr learned my addr 172.21.15.97:0/1411309667 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:55.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.713+0000 7f7c25600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7c20070b10 0x7f7c20070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34198/0 (socket says 172.21.15.97:34198) 2024-03-06T07:40:55.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.714+0000 7f7c1fe00640 1 --2- 172.21.15.97:0/1411309667 >> v2:172.21.15.135:3300/0 conn(0x7f7c2015f9d0 0x7f7c2015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:55.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.714+0000 7f7c25600640 1 -- 172.21.15.97:0/1411309667 >> v2:172.21.15.135:3300/0 conn(0x7f7c2015f9d0 msgr2=0x7f7c2015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:55.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.714+0000 7f7c25600640 1 --2- 172.21.15.97:0/1411309667 >> v2:172.21.15.135:3300/0 conn(0x7f7c2015f9d0 0x7f7c2015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.714+0000 7f7c25600640 1 -- 172.21.15.97:0/1411309667 >> v1:172.21.15.97:6789/0 conn(0x7f7c20144c00 legacy=0x7f7c2006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:55.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.714+0000 7f7c25600640 1 -- 172.21.15.97:0/1411309667 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2006d790 con 0x7f7c20070b10 2024-03-06T07:40:55.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.714+0000 7f7c1f400640 1 -- 172.21.15.97:0/1411309667 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308950279 0 0) 0x7f7c2006cef0 con 0x7f7c20144c00 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.715+0000 7f7c25600640 1 --2- 172.21.15.97:0/1411309667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c20070b10 0x7f7c20070f20 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f7c200b26f0 tx=0x7f7c1802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644fcb3cd4126420 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.715+0000 7f7c1f400640 1 -- 172.21.15.97:0/1411309667 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18039070 con 0x7f7c20070b10 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.716+0000 7f7c1f400640 1 -- 172.21.15.97:0/1411309667 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1802cc80 con 0x7f7c20070b10 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.716+0000 7f7c1f400640 1 -- 172.21.15.97:0/1411309667 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18034660 con 0x7f7c20070b10 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 -- 172.21.15.97:0/1411309667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c20070b10 msgr2=0x7f7c20070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 --2- 172.21.15.97:0/1411309667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c20070b10 0x7f7c20070f20 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f7c200b26f0 tx=0x7f7c1802bed0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 -- 172.21.15.97:0/1411309667 shutdown_connections 2024-03-06T07:40:55.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 --2- 172.21.15.97:0/1411309667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c20070b10 0x7f7c20070f20 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 --2- 172.21.15.97:0/1411309667 >> v2:172.21.15.135:3300/0 conn(0x7f7c2015f9d0 0x7f7c2015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 -- 172.21.15.97:0/1411309667 >> 172.21.15.97:0/1411309667 conn(0x7f7c200b1730 msgr2=0x7f7c200b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:55.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 -- 172.21.15.97:0/1411309667 shutdown_connections 2024-03-06T07:40:55.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.717+0000 7f7c27600640 1 -- 172.21.15.97:0/1411309667 wait complete. 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.718+0000 7f7c27600640 1 Processor -- start 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.719+0000 7f7c27600640 1 -- start start 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.719+0000 7f7c27600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c20070b10 0x7f7c2014d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.719+0000 7f7c27600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c20144c00 0x7f7c2014d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c27600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 0x7f7c2014dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c27600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2006ccd0 con 0x7f7c2015f9d0 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c27600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2006cb50 con 0x7f7c20070b10 2024-03-06T07:40:55.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c27600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2006df90 con 0x7f7c20144c00 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 0x7f7c2014dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 0x7f7c2014dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34212/0 (socket says 172.21.15.97:34212) 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 -- 172.21.15.97:0/679507257 learned_addr learned my addr 172.21.15.97:0/679507257 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c1fe00640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c20144c00 0x7f7c2014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 -- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c20144c00 msgr2=0x7f7c2014d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c20144c00 0x7f7c2014d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 -- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c20070b10 msgr2=0x7f7c2014d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c20070b10 0x7f7c2014d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.720+0000 7f7c25600640 1 -- 172.21.15.97:0/679507257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2013a730 con 0x7f7c2015f9d0 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.721+0000 7f7c25600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 0x7f7c2014dc70 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f7c18037040 tx=0x7f7c18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:55.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.721+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18044020 con 0x7f7c2015f9d0 2024-03-06T07:40:55.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.721+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1803a040 con 0x7f7c2015f9d0 2024-03-06T07:40:55.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.721+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18039050 con 0x7f7c2015f9d0 2024-03-06T07:40:55.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.721+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2013a170 con 0x7f7c2015f9d0 2024-03-06T07:40:55.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.721+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2013d000 con 0x7f7c2015f9d0 2024-03-06T07:40:55.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.722+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2004f330 con 0x7f7c2015f9d0 2024-03-06T07:40:55.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.723+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c18003b90 con 0x7f7c2015f9d0 2024-03-06T07:40:55.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.724+0000 7f7c1cc00640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7c0006b4d0 0x7f7c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:55.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.724+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7c180acb60 con 0x7f7c2015f9d0 2024-03-06T07:40:55.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.724+0000 7f7c24c00640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7c0006b4d0 0x7f7c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:55.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.725+0000 7f7c24c00640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7c0006b4d0 0x7f7c0006d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7c14007fd0 tx=0x7f7c14007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:55.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.729+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c1804e020 con 0x7f7c2015f9d0 2024-03-06T07:40:55.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.965+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c2015f690 con 0x7f7c2015f9d0 2024-03-06T07:40:55.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.965+0000 7f7c1cc00640 1 -- 172.21.15.97:0/679507257 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f7c1807b9b0 con 0x7f7c2015f9d0 2024-03-06T07:40:55.968 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:55.968 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:55.969 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7c0006b4d0 msgr2=0x7f7c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7c0006b4d0 0x7f7c0006d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7c14007fd0 tx=0x7f7c14007920 comp rx=0 tx=0).stop 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 msgr2=0x7f7c2014dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 0x7f7c2014dc70 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f7c18037040 tx=0x7f7c18004a40 comp rx=0 tx=0).stop 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 shutdown_connections 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f7c0006b4d0 0x7f7c0006d990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c2015f9d0 0x7f7c2014dc70 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f7c20144c00 0x7f7c2014d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 --2- 172.21.15.97:0/679507257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c20070b10 0x7f7c2014d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 >> 172.21.15.97:0/679507257 conn(0x7f7c200b1730 msgr2=0x7f7c201431a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 shutdown_connections 2024-03-06T07:40:55.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:55.968+0000 7f7c27600640 1 -- 172.21.15.97:0/679507257 wait complete. 2024-03-06T07:40:56.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:56.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:57.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.414+0000 7f445b600640 1 Processor -- start 2024-03-06T07:40:57.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.414+0000 7f445b600640 1 -- start start 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f445b600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445415f470 0x7f44540d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f445b600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f44540d0df0 0x7f4454160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f445b600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540cea60 con 0x7f445415e0d0 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f445b600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44540d1260 con 0x7f44540d0df0 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f445b600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44540d2f60 con 0x7f445415f470 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f4459600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f44540d0df0 0x7f4454160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f4458c00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f445415e0d0 0x7f44540d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45288/0 (socket says 172.21.15.97:45288) 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f4458c00640 1 -- 172.21.15.97:0/1127045731 learned_addr learned my addr 172.21.15.97:0/1127045731 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.416+0000 7f4459600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f44540d0df0 0x7f4454160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34214/0 (socket says 172.21.15.97:34214) 2024-03-06T07:40:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.417+0000 7f444be00640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445415f470 0x7f44540d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.417+0000 7f4459600640 1 -- 172.21.15.97:0/1127045731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445415f470 msgr2=0x7f44540d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.417+0000 7f4459600640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445415f470 0x7f44540d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.417+0000 7f4459600640 1 -- 172.21.15.97:0/1127045731 >> v1:172.21.15.97:6789/0 conn(0x7f445415e0d0 legacy=0x7f44540d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.417+0000 7f4459600640 1 -- 172.21.15.97:0/1127045731 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540d2350 con 0x7f44540d0df0 2024-03-06T07:40:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.417+0000 7f444be00640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445415f470 0x7f44540d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.418+0000 7f4459600640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44540d0df0 0x7f4454160dc0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f444c009d90 tx=0x7f444c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00271c520ef73ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:57.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.418+0000 7f4453e00640 1 -- 172.21.15.97:0/1127045731 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c039070 con 0x7f44540d0df0 2024-03-06T07:40:57.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.418+0000 7f4453e00640 1 -- 172.21.15.97:0/1127045731 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c02cce0 con 0x7f44540d0df0 2024-03-06T07:40:57.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.418+0000 7f4453e00640 1 -- 172.21.15.97:0/1127045731 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c034660 con 0x7f44540d0df0 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 -- 172.21.15.97:0/1127045731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44540d0df0 msgr2=0x7f4454160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44540d0df0 0x7f4454160dc0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f444c009d90 tx=0x7f444c02bf30 comp rx=0 tx=0).stop 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 -- 172.21.15.97:0/1127045731 shutdown_connections 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44540d0df0 0x7f4454160dc0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 --2- 172.21.15.97:0/1127045731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f445415f470 0x7f44540d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 -- 172.21.15.97:0/1127045731 >> 172.21.15.97:0/1127045731 conn(0x7f44540d3ec0 msgr2=0x7f44540d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 -- 172.21.15.97:0/1127045731 shutdown_connections 2024-03-06T07:40:57.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.419+0000 7f445b600640 1 -- 172.21.15.97:0/1127045731 wait complete. 2024-03-06T07:40:57.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.421+0000 7f445b600640 1 Processor -- start 2024-03-06T07:40:57.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.421+0000 7f445b600640 1 -- start start 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f445b600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44540d0df0 0x7f44541f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f445b600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445415e0d0 0x7f4454154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f445b600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 0x7f44540c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f445b600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44540d2bf0 con 0x7f445415f470 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f445b600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44540d2910 con 0x7f445415e0d0 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f445b600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445415fb30 con 0x7f44540d0df0 2024-03-06T07:40:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f4459600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 0x7f44540c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f4459600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 0x7f44540c90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34220/0 (socket says 172.21.15.97:34220) 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.422+0000 7f4459600640 1 -- 172.21.15.97:0/3751962472 learned_addr learned my addr 172.21.15.97:0/3751962472 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f444be00640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445415e0d0 0x7f4454154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4458c00640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44540d0df0 0x7f44541f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4459600640 1 -- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44540d0df0 msgr2=0x7f44541f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4459600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44540d0df0 0x7f44541f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4459600640 1 -- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445415e0d0 msgr2=0x7f4454154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4459600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445415e0d0 0x7f4454154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4459600640 1 -- 172.21.15.97:0/3751962472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540c9a40 con 0x7f445415f470 2024-03-06T07:40:57.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4459600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 0x7f44540c90f0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f444c009d90 tx=0x7f444c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.423+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c039050 con 0x7f445415f470 2024-03-06T07:40:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.424+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c045070 con 0x7f445415f470 2024-03-06T07:40:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.424+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c03e830 con 0x7f445415f470 2024-03-06T07:40:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.424+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44540c9d20 con 0x7f445415f470 2024-03-06T07:40:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.424+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44541554f0 con 0x7f445415f470 2024-03-06T07:40:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.425+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445404f330 con 0x7f445415f470 2024-03-06T07:40:57.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.426+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f444c03e9d0 con 0x7f445415f470 2024-03-06T07:40:57.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.427+0000 7f4451600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443006b4d0 0x7f443006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:57.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.427+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f444c0acff0 con 0x7f445415f470 2024-03-06T07:40:57.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.427+0000 7f4458c00640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443006b4d0 0x7f443006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:57.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.428+0000 7f4458c00640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443006b4d0 0x7f443006d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f44540d08d0 tx=0x7f4444009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:57.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.432+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f444c07bea0 con 0x7f445415f470 2024-03-06T07:40:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.664+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f445408d610 con 0x7f445415f470 2024-03-06T07:40:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.665+0000 7f4451600640 1 -- 172.21.15.97:0/3751962472 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f444c07b7a0 con 0x7f445415f470 2024-03-06T07:40:57.666 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:57.666 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443006b4d0 msgr2=0x7f443006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443006b4d0 0x7f443006d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f44540d08d0 tx=0x7f4444009040 comp rx=0 tx=0).stop 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 msgr2=0x7f44540c90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 0x7f44540c90f0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f444c009d90 tx=0x7f444c004a40 comp rx=0 tx=0).stop 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 shutdown_connections 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f443006b4d0 0x7f443006d990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f445415f470 0x7f44540c90f0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445415e0d0 0x7f4454154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 --2- 172.21.15.97:0/3751962472 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44540d0df0 0x7f44541f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.667+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 >> 172.21.15.97:0/3751962472 conn(0x7f44540d3ec0 msgr2=0x7f44540b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.668+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 shutdown_connections 2024-03-06T07:40:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:57.668+0000 7f445b600640 1 -- 172.21.15.97:0/3751962472 wait complete. 2024-03-06T07:40:57.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:57.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:40:58.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.094+0000 7fa060c00640 1 Processor -- start 2024-03-06T07:40:59.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.094+0000 7fa060c00640 1 -- start start 2024-03-06T07:40:59.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.096+0000 7fa060c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa05c0b40c0 0x7fa05c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.096+0000 7fa060c00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c161900 con 0x7fa05c1620c0 2024-03-06T07:40:59.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.096+0000 7fa060c00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c0b49e0 con 0x7fa05c0ae670 2024-03-06T07:40:59.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.096+0000 7fa060c00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c15ebd0 con 0x7fa05c0b40c0 2024-03-06T07:40:59.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.096+0000 7fa05be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa05c1620c0 0x7fa05c0b28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45296/0 (socket says 172.21.15.97:45296) 2024-03-06T07:40:59.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.096+0000 7fa05be00640 1 -- 172.21.15.97:0/2004032605 learned_addr learned my addr 172.21.15.97:0/2004032605 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:40:59.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05b400640 1 --2- 172.21.15.97:0/2004032605 >> v2:172.21.15.135:3300/0 conn(0x7fa05c0b40c0 0x7fa05c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861605899 0 0) 0x7fa05c161900 con 0x7fa05c1620c0 2024-03-06T07:40:59.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa03c003520 con 0x7fa05c1620c0 2024-03-06T07:40:59.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1204550527 0 0) 0x7fa03c003520 con 0x7fa05c1620c0 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 >> v2:172.21.15.135:3300/0 conn(0x7fa05c0b40c0 msgr2=0x7fa05c0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 --2- 172.21.15.97:0/2004032605 >> v2:172.21.15.135:3300/0 conn(0x7fa05c0b40c0 0x7fa05c0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 >> v1:172.21.15.135:6789/0 conn(0x7fa05c0ae670 legacy=0x7fa05c15e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c15eeb0 con 0x7fa05c1620c0 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa0540019f0 con 0x7fa05c1620c0 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 >> v1:172.21.15.97:6789/0 conn(0x7fa05c1620c0 legacy=0x7fa05c0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa03c0039b0 0x7fa03c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c004800 0x7fa03c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa03c0051c0 0x7fa03c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c15ebd0 con 0x7fa03c0051c0 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c00c060 con 0x7fa03c004800 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.097+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c00c260 con 0x7fa03c0039b0 2024-03-06T07:40:59.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05be00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa03c0051c0 0x7fa03c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05b400640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa03c0039b0 0x7fa03c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa053e00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c004800 0x7fa03c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05be00640 1 -- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa03c0039b0 msgr2=0x7fa03c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05be00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa03c0039b0 0x7fa03c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05be00640 1 -- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c004800 msgr2=0x7fa03c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05be00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c004800 0x7fa03c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa05be00640 1 -- 172.21.15.97:0/2004032605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c003520 con 0x7fa03c0051c0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.098+0000 7fa053e00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c004800 0x7fa03c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.099+0000 7fa05be00640 1 -- 172.21.15.97:0/2004032605 reap_dead start 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.099+0000 7fa05be00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa03c0051c0 0x7fa03c00b9c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fa05400a850 tx=0x7fa03c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0b6e65becf8eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.099+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400a4a0 con 0x7fa03c0051c0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.099+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa054034650 con 0x7fa03c0051c0 2024-03-06T07:40:59.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.099+0000 7fa05aa00640 1 -- 172.21.15.97:0/2004032605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054034a40 con 0x7fa03c0051c0 2024-03-06T07:40:59.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 -- 172.21.15.97:0/2004032605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa03c0051c0 msgr2=0x7fa03c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa03c0051c0 0x7fa03c00b9c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fa05400a850 tx=0x7fa03c005830 comp rx=0 tx=0).stop 2024-03-06T07:40:59.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 -- 172.21.15.97:0/2004032605 shutdown_connections 2024-03-06T07:40:59.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 --2- 172.21.15.97:0/2004032605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa03c0051c0 0x7fa03c00b9c0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 -- 172.21.15.97:0/2004032605 >> 172.21.15.97:0/2004032605 conn(0x7fa05c06afc0 msgr2=0x7fa05c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:59.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 -- 172.21.15.97:0/2004032605 shutdown_connections 2024-03-06T07:40:59.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.100+0000 7fa060c00640 1 -- 172.21.15.97:0/2004032605 wait complete. 2024-03-06T07:40:59.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.102+0000 7fa060c00640 1 Processor -- start 2024-03-06T07:40:59.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.102+0000 7fa060c00640 1 -- start start 2024-03-06T07:40:59.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa060c00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c0051c0 0x7fa05c15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa060c00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa05c14fd40 0x7fa05c0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa060c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 0x7fa05c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa060c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c158660 con 0x7fa05c0b2d30 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa060c00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c161ba0 con 0x7fa03c0051c0 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa060c00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c1587e0 con 0x7fa05c14fd40 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa05be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 0x7fa05c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa05be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 0x7fa05c1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34240/0 (socket says 172.21.15.97:34240) 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa05be00640 1 -- 172.21.15.97:0/2718824030 learned_addr learned my addr 172.21.15.97:0/2718824030 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.103+0000 7fa05b400640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c0051c0 0x7fa05c15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa053e00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa05c14fd40 0x7fa05c0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa05be00640 1 -- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa05c14fd40 msgr2=0x7fa05c0b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa05be00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa05c14fd40 0x7fa05c0b15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa05be00640 1 -- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c0051c0 msgr2=0x7fa05c15b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa05be00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c0051c0 0x7fa05c15b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa05be00640 1 -- 172.21.15.97:0/2718824030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c1595e0 con 0x7fa05c0b2d30 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa05be00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 0x7fa05c1552d0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa05400a850 tx=0x7fa054001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054045070 con 0x7fa05c0b2d30 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05403a040 con 0x7fa05c0b2d30 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c15b6e0 con 0x7fa05c0b2d30 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.104+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054004d60 con 0x7fa05c0b2d30 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.105+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c0b35d0 con 0x7fa05c0b2d30 2024-03-06T07:40:59.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.106+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c0b3810 con 0x7fa05c0b2d30 2024-03-06T07:40:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.107+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa054048070 con 0x7fa05c0b2d30 2024-03-06T07:40:59.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.107+0000 7fa053400640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.108+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa0540abd50 con 0x7fa05c0b2d30 2024-03-06T07:40:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.108+0000 7fa05b400640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.108+0000 7fa05b400640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa044009a10 tx=0x7fa044008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:59.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.112+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa05c0b3810 con 0x7fa05c0b2d30 2024-03-06T07:40:59.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.346+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa05c04f330 con 0x7fa05c0b2d30 2024-03-06T07:40:59.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.346+0000 7fa053400640 1 -- 172.21.15.97:0/2718824030 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fa05407a550 con 0x7fa05c0b2d30 2024-03-06T07:40:59.348 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:40:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:40:59.348 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:40:59.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa03806b4d0 msgr2=0x7fa03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa044009a10 tx=0x7fa044008040 comp rx=0 tx=0).stop 2024-03-06T07:40:59.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 msgr2=0x7fa05c1552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 0x7fa05c1552d0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa05400a850 tx=0x7fa054001dc0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 shutdown_connections 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c0b2d30 0x7fa05c1552d0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa05c14fd40 0x7fa05c0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 --2- 172.21.15.97:0/2718824030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa03c0051c0 0x7fa05c15b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 >> 172.21.15.97:0/2718824030 conn(0x7fa05c06afc0 msgr2=0x7fa05c161100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 shutdown_connections 2024-03-06T07:40:59.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:40:59.349+0000 7fa060c00640 1 -- 172.21.15.97:0/2718824030 wait complete. 2024-03-06T07:40:59.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:40:59.570 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:00.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:00.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.777+0000 7f6860400640 1 Processor -- start 2024-03-06T07:41:00.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.777+0000 7f6860400640 1 -- start start 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f6860400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f685815e160 0x7f6858162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f6860400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68580af820 0x7f68580b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f6860400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b38f0 0x7f685806d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f6860400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685806b0f0 con 0x7f68580af820 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f6860400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68580b5620 con 0x7f685815e160 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f6860400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6858162b60 con 0x7f68580b38f0 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f685d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68580af820 0x7f68580b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f685d000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f68580af820 0x7f68580b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34250/0 (socket says 172.21.15.97:34250) 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.779+0000 7f685d000640 1 -- 172.21.15.97:0/3617259949 learned_addr learned my addr 172.21.15.97:0/3617259949 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685da00640 1 --2- 172.21.15.97:0/3617259949 >> v2:172.21.15.135:3300/0 conn(0x7f685815e160 0x7f6858162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685e400640 1 --2- 172.21.15.97:0/3617259949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b38f0 0x7f685806d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685d000640 1 -- 172.21.15.97:0/3617259949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b38f0 msgr2=0x7f685806d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685d000640 1 --2- 172.21.15.97:0/3617259949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b38f0 0x7f685806d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685d000640 1 -- 172.21.15.97:0/3617259949 >> v2:172.21.15.135:3300/0 conn(0x7f685815e160 msgr2=0x7f6858162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685d000640 1 --2- 172.21.15.97:0/3617259949 >> v2:172.21.15.135:3300/0 conn(0x7f685815e160 0x7f6858162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.780+0000 7f685d000640 1 -- 172.21.15.97:0/3617259949 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580b3d70 con 0x7f68580af820 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.781+0000 7f685d000640 1 --2- 172.21.15.97:0/3617259949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68580af820 0x7f68580b33b0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f68580d0310 tx=0x7f684402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce8ee564d4efe10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.781+0000 7f6857e00640 1 -- 172.21.15.97:0/3617259949 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684402e070 con 0x7f68580af820 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.781+0000 7f6857e00640 1 -- 172.21.15.97:0/3617259949 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844037070 con 0x7f68580af820 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.781+0000 7f6857e00640 1 -- 172.21.15.97:0/3617259949 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844031a00 con 0x7f68580af820 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 -- 172.21.15.97:0/3617259949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68580af820 msgr2=0x7f68580b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 --2- 172.21.15.97:0/3617259949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68580af820 0x7f68580b33b0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f68580d0310 tx=0x7f684402b880 comp rx=0 tx=0).stop 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 -- 172.21.15.97:0/3617259949 shutdown_connections 2024-03-06T07:41:00.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 --2- 172.21.15.97:0/3617259949 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580b38f0 0x7f685806d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 --2- 172.21.15.97:0/3617259949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68580af820 0x7f68580b33b0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 --2- 172.21.15.97:0/3617259949 >> v2:172.21.15.135:3300/0 conn(0x7f685815e160 0x7f6858162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 -- 172.21.15.97:0/3617259949 >> 172.21.15.97:0/3617259949 conn(0x7f68580d0340 msgr2=0x7f68580d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 -- 172.21.15.97:0/3617259949 shutdown_connections 2024-03-06T07:41:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.782+0000 7f6860400640 1 -- 172.21.15.97:0/3617259949 wait complete. 2024-03-06T07:41:00.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.785+0000 7f6860400640 1 Processor -- start 2024-03-06T07:41:00.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.785+0000 7f6860400640 1 -- start start 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.786+0000 7f6860400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580af820 0x7f68580ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.786+0000 7f6860400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68580b38f0 0x7f68580ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.786+0000 7f6860400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 0x7f68581609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.786+0000 7f6860400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685806ec30 con 0x7f685815e160 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.786+0000 7f6860400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580afce0 con 0x7f68580b38f0 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.786+0000 7f6860400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685806ef10 con 0x7f68580af820 2024-03-06T07:41:00.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 0x7f68581609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 0x7f68581609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34256/0 (socket says 172.21.15.97:34256) 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 -- 172.21.15.97:0/4113882717 learned_addr learned my addr 172.21.15.97:0/4113882717 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685d000640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68580b38f0 0x7f68580ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685da00640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580af820 0x7f68580ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 -- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580af820 msgr2=0x7f68580ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580af820 0x7f68580ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 -- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68580b38f0 msgr2=0x7f68580ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68580b38f0 0x7f68580ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 -- 172.21.15.97:0/4113882717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580cd250 con 0x7f685815e160 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.787+0000 7f685e400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 0x7f68581609b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f6850007d70 tx=0x7f685000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.788+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850014420 con 0x7f685815e160 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.788+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6850019070 con 0x7f685815e160 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.788+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850014820 con 0x7f685815e160 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.788+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68580cd530 con 0x7f685815e160 2024-03-06T07:41:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.788+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68580cdaf0 con 0x7f685815e160 2024-03-06T07:41:00.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.789+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6858074ae0 con 0x7f685815e160 2024-03-06T07:41:00.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.790+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68500149c0 con 0x7f685815e160 2024-03-06T07:41:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.791+0000 7f6855600640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683806b4d0 0x7f683806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.791+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f685008a9f0 con 0x7f685815e160 2024-03-06T07:41:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.791+0000 7f685da00640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683806b4d0 0x7f683806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.792+0000 7f685da00640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683806b4d0 0x7f683806d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f684c006fd0 tx=0x7f684c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:00.796+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6850059840 con 0x7f685815e160 2024-03-06T07:41:01.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.035+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f685804f330 con 0x7f685815e160 2024-03-06T07:41:01.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.036+0000 7f6855600640 1 -- 172.21.15.97:0/4113882717 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f685005f070 con 0x7f685815e160 2024-03-06T07:41:01.038 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:01.038 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:01.038 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.039+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683806b4d0 msgr2=0x7f683806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.039+0000 7f6860400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683806b4d0 0x7f683806d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f684c006fd0 tx=0x7f684c008040 comp rx=0 tx=0).stop 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.039+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 msgr2=0x7f68581609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.039+0000 7f6860400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 0x7f68581609b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f6850007d70 tx=0x7f685000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 shutdown_connections 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f683806b4d0 0x7f683806d990 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f685815e160 0x7f68581609b0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68580b38f0 0x7f68580ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 --2- 172.21.15.97:0/4113882717 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f68580af820 0x7f68580ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 >> 172.21.15.97:0/4113882717 conn(0x7f68580d0340 msgr2=0x7f68580ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:01.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 shutdown_connections 2024-03-06T07:41:01.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:01.040+0000 7f6860400640 1 -- 172.21.15.97:0/4113882717 wait complete. 2024-03-06T07:41:01.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:01.262 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:02.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:02.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.474+0000 7f566f200640 1 Processor -- start 2024-03-06T07:41:02.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.474+0000 7f566f200640 1 -- start start 2024-03-06T07:41:02.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.475+0000 7f566f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5668144660 0x7f5668071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566f200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f56680ca5d0 0x7f56680724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566f200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56680cabf0 con 0x7f56680cf3e0 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566f200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56680ce900 con 0x7f56680ca5d0 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56680cebe0 con 0x7f5668144660 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566d200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f56680ca5d0 0x7f56680724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f565f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f56680cf3e0 0x7f56680c9ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45310/0 (socket says 172.21.15.97:45310) 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f565f400640 1 -- 172.21.15.97:0/185998078 learned_addr learned my addr 172.21.15.97:0/185998078 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f5667e00640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5668144660 0x7f5668071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566d200640 1 -- 172.21.15.97:0/185998078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5668144660 msgr2=0x7f5668071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566d200640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5668144660 0x7f5668071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566d200640 1 -- 172.21.15.97:0/185998078 >> v1:172.21.15.97:6789/0 conn(0x7f56680cf3e0 legacy=0x7f56680c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f566d200640 1 -- 172.21.15.97:0/185998078 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56680ceec0 con 0x7f56680ca5d0 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.476+0000 7f5667e00640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5668144660 0x7f5668071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.477+0000 7f566d200640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f56680724b0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f566002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcbf28cda15a758 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f5667400640 1 -- 172.21.15.97:0/185998078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660035020 con 0x7f56680ca5d0 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f5667400640 1 -- 172.21.15.97:0/185998078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660031810 con 0x7f56680ca5d0 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f5667400640 1 -- 172.21.15.97:0/185998078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660031be0 con 0x7f56680ca5d0 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f566f200640 1 -- 172.21.15.97:0/185998078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 msgr2=0x7f56680724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f566f200640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f56680724b0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f566002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f566f200640 1 -- 172.21.15.97:0/185998078 shutdown_connections 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f566f200640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f56680724b0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f566f200640 1 --2- 172.21.15.97:0/185998078 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5668144660 0x7f5668071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.478+0000 7f566f200640 1 -- 172.21.15.97:0/185998078 >> 172.21.15.97:0/185998078 conn(0x7f56680d0b90 msgr2=0x7f56680d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.479+0000 7f566f200640 1 -- 172.21.15.97:0/185998078 shutdown_connections 2024-03-06T07:41:02.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.479+0000 7f566f200640 1 -- 172.21.15.97:0/185998078 wait complete. 2024-03-06T07:41:02.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.480+0000 7f566f200640 1 Processor -- start 2024-03-06T07:41:02.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.480+0000 7f566f200640 1 -- start start 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f566f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f566815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f566f200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f56680cf3e0 0x7f566815e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f566f200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5668144660 0x7f56680b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f566f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56680cc270 con 0x7f56680ca5d0 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f566f200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56680cc0f0 con 0x7f5668144660 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f566f200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56680cc3f0 con 0x7f56680cf3e0 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.481+0000 7f5667e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f566815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f566815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34276/0 (socket says 172.21.15.97:34276) 2024-03-06T07:41:02.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 -- 172.21.15.97:0/308456569 learned_addr learned my addr 172.21.15.97:0/308456569 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 -- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f56680cf3e0 msgr2=0x7f566815e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f56680cf3e0 0x7f566815e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 -- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5668144660 msgr2=0x7f56680b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f566d200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5668144660 0x7f56680b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5668144660 0x7f56680b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 -- 172.21.15.97:0/308456569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56681f6fe0 con 0x7f56680ca5d0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.482+0000 7f5667e00640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f566815a8b0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f565800c820 tx=0x7f565800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.483+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658015070 con 0x7f56680ca5d0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.483+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5658007a30 con 0x7f56680ca5d0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.483+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658007de0 con 0x7f56680ca5d0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.483+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56681f72c0 con 0x7f56680ca5d0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.483+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56681f7880 con 0x7f56680ca5d0 2024-03-06T07:41:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.484+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566804f330 con 0x7f56680ca5d0 2024-03-06T07:41:02.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.485+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5658018070 con 0x7f56680ca5d0 2024-03-06T07:41:02.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.486+0000 7f5664c00640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564406b4d0 0x7f564406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:02.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.487+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f565808a170 con 0x7f56680ca5d0 2024-03-06T07:41:02.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.487+0000 7f565f400640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564406b4d0 0x7f564406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:02.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.487+0000 7f565f400640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564406b4d0 0x7f564406d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f56500012e0 tx=0x7f5650001630 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:02.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.492+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5658058f30 con 0x7f56680ca5d0 2024-03-06T07:41:02.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.877+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56681626e0 con 0x7f56680ca5d0 2024-03-06T07:41:02.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.878+0000 7f5664c00640 1 -- 172.21.15.97:0/308456569 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f5658058830 con 0x7f56680ca5d0 2024-03-06T07:41:02.879 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:02.880 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:02.880 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564406b4d0 msgr2=0x7f564406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564406b4d0 0x7f564406d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f56500012e0 tx=0x7f5650001630 comp rx=0 tx=0).stop 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 msgr2=0x7f566815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f566815a8b0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f565800c820 tx=0x7f565800cce0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 shutdown_connections 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564406b4d0 0x7f564406d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5668144660 0x7f56680b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f56680cf3e0 0x7f566815e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 --2- 172.21.15.97:0/308456569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56680ca5d0 0x7f566815a8b0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.880+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 >> 172.21.15.97:0/308456569 conn(0x7f56680d0b90 msgr2=0x7f566806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:02.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.881+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 shutdown_connections 2024-03-06T07:41:02.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:02.881+0000 7f566f200640 1 -- 172.21.15.97:0/308456569 wait complete. 2024-03-06T07:41:03.103 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:03.103 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:04.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:04.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.310+0000 7f214da00640 1 Processor -- start 2024-03-06T07:41:04.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.311+0000 7f214da00640 1 -- start start 2024-03-06T07:41:04.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f214da00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f214815acb0 0x7f21480b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f214da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 0x7f21480ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f214da00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214815b090 con 0x7f21480b4680 2024-03-06T07:41:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f214da00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480b2a30 con 0x7f214815acb0 2024-03-06T07:41:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f214da00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480b4360 con 0x7f214806ec00 2024-03-06T07:41:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f213e000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f21480b4680 0x7f214806e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45168/0 (socket says 172.21.15.97:45168) 2024-03-06T07:41:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.312+0000 7f213e000640 1 -- 172.21.15.97:0/1432305855 learned_addr learned my addr 172.21.15.97:0/1432305855 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2147400640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 0x7f21480ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146a00640 1 --2- 172.21.15.97:0/1432305855 >> v2:172.21.15.135:3300/0 conn(0x7f214815acb0 0x7f21480b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548767233 0 0) 0x7f214815b090 con 0x7f21480b4680 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003520 con 0x7f21480b4680 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 932592057 0 0) 0x7f212c003520 con 0x7f21480b4680 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 msgr2=0x7f21480ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 0x7f21480ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 >> v2:172.21.15.135:3300/0 conn(0x7f214815acb0 msgr2=0x7f21480b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 --2- 172.21.15.97:0/1432305855 >> v2:172.21.15.135:3300/0 conn(0x7f214815acb0 0x7f21480b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480b2cf0 con 0x7f21480b4680 2024-03-06T07:41:04.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2130001670 con 0x7f21480b4680 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 >> v1:172.21.15.97:6789/0 conn(0x7f21480b4680 legacy=0x7f214806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.313+0000 7f2146000640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f212c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146000640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f212c004800 0x7f212c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146000640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 0x7f212c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480b2a30 con 0x7f212c0039b0 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480b4360 con 0x7f212c004800 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c00b9c0 con 0x7f214806ec00 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146a00640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f212c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f213e000640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f212c004800 0x7f212c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146a00640 1 -- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 msgr2=0x7f212c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146a00640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f214806ec00 0x7f212c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146a00640 1 -- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f212c004800 msgr2=0x7f212c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:04.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146a00640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f212c004800 0x7f212c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.314+0000 7f2146a00640 1 -- 172.21.15.97:0/1432305855 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c003520 con 0x7f212c0039b0 2024-03-06T07:41:04.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.315+0000 7f2146a00640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f212c003dd0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f2138009fc0 tx=0x7f212c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b38513cafdc1d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:04.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.315+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138039070 con 0x7f212c0039b0 2024-03-06T07:41:04.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213802cc60 con 0x7f212c0039b0 2024-03-06T07:41:04.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f2146000640 1 -- 172.21.15.97:0/1432305855 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138009c10 con 0x7f212c0039b0 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f214da00640 1 -- 172.21.15.97:0/1432305855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 msgr2=0x7f212c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f214da00640 1 --2- 172.21.15.97:0/1432305855 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f212c003dd0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f2138009fc0 tx=0x7f212c005830 comp rx=0 tx=0).stop 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f2147400640 1 -- 172.21.15.97:0/1432305855 reap_dead start 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f214da00640 1 -- 172.21.15.97:0/1432305855 shutdown_connections 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f214da00640 1 -- 172.21.15.97:0/1432305855 >> 172.21.15.97:0/1432305855 conn(0x7f21480b1730 msgr2=0x7f21480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f214da00640 1 -- 172.21.15.97:0/1432305855 shutdown_connections 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.316+0000 7f214da00640 1 -- 172.21.15.97:0/1432305855 wait complete. 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.318+0000 7f214da00640 1 Processor -- start 2024-03-06T07:41:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.318+0000 7f214da00640 1 -- start start 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f214da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f214815b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f214da00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21480cef50 0x7f214815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f214da00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21481459f0 0x7f21481f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f214da00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480b3d10 con 0x7f212c0039b0 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f214da00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480b32b0 con 0x7f21481459f0 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f214da00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480b4010 con 0x7f21480cef50 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f2146a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f214815b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f2146a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f214815b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60102/0 (socket says 172.21.15.97:60102) 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.319+0000 7f2146a00640 1 -- 172.21.15.97:0/582141883 learned_addr learned my addr 172.21.15.97:0/582141883 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f213e000640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21480cef50 0x7f214815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2147400640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21481459f0 0x7f21481f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2146a00640 1 -- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21480cef50 msgr2=0x7f214815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2146a00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21480cef50 0x7f214815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2146a00640 1 -- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21481459f0 msgr2=0x7f21481f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2146a00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21481459f0 0x7f21481f5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2146a00640 1 -- 172.21.15.97:0/582141883 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214815d970 con 0x7f212c0039b0 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f2146a00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f214815b090 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f213800e230 tx=0x7f2138009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.320+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138039050 con 0x7f212c0039b0 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.321+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213803a040 con 0x7f212c0039b0 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.321+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138009c10 con 0x7f212c0039b0 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.321+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214815dc10 con 0x7f212c0039b0 2024-03-06T07:41:04.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.321+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214815def0 con 0x7f212c0039b0 2024-03-06T07:41:04.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.322+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214815bb10 con 0x7f212c0039b0 2024-03-06T07:41:04.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.323+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2138047030 con 0x7f212c0039b0 2024-03-06T07:41:04.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.324+0000 7f213f400640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f212406b4d0 0x7f212406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:04.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.324+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f21380ac420 con 0x7f212c0039b0 2024-03-06T07:41:04.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.324+0000 7f213e000640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f212406b4d0 0x7f212406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:04.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.325+0000 7f213e000640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f212406b4d0 0x7f212406d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2148145f10 tx=0x7f2130009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:04.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.333+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f214815bb10 con 0x7f212c0039b0 2024-03-06T07:41:04.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.568+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f214804f330 con 0x7f212c0039b0 2024-03-06T07:41:04.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.569+0000 7f213f400640 1 -- 172.21.15.97:0/582141883 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f213804e060 con 0x7f212c0039b0 2024-03-06T07:41:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:04.571 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:04.572 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:04.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f212406b4d0 msgr2=0x7f212406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:04.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f212406b4d0 0x7f212406d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2148145f10 tx=0x7f2130009040 comp rx=0 tx=0).stop 2024-03-06T07:41:04.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 msgr2=0x7f214815b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:04.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f214815b090 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f213800e230 tx=0x7f2138009750 comp rx=0 tx=0).stop 2024-03-06T07:41:04.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 shutdown_connections 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f212406b4d0 0x7f212406d990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21481459f0 0x7f21481f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f21480cef50 0x7f214815b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 --2- 172.21.15.97:0/582141883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f212c0039b0 0x7f214815b090 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.572+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 >> 172.21.15.97:0/582141883 conn(0x7f21480b1730 msgr2=0x7f21480b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.573+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 shutdown_connections 2024-03-06T07:41:04.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:04.573+0000 7f214da00640 1 -- 172.21.15.97:0/582141883 wait complete. 2024-03-06T07:41:04.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:04.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:05.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:06.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.001+0000 7f2f72a00640 1 Processor -- start 2024-03-06T07:41:06.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.001+0000 7f2f72a00640 1 -- start start 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f72a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 0x7f2f6c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f72a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c15b090 con 0x7f2f6c15acb0 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f72a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c0b2a30 con 0x7f2f6c0b4680 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f72a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b4360 con 0x7f2f6c06ec00 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f6be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2f6c15acb0 0x7f2f6c0b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45180/0 (socket says 172.21.15.97:45180) 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f6be00640 1 -- 172.21.15.97:0/1201277675 learned_addr learned my addr 172.21.15.97:0/1201277675 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908930615 0 0) 0x7f2f6c15b090 con 0x7f2f6c15acb0 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f50003520 con 0x7f2f6c15acb0 2024-03-06T07:41:06.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.003+0000 7f2f70a00640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 0x7f2f6c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349973800 0 0) 0x7f2f50003520 con 0x7f2f6c15acb0 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 msgr2=0x7f2f6c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 0x7f2f6c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 >> v1:172.21.15.135:6789/0 conn(0x7f2f6c0b4680 legacy=0x7f2f6c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c0b2cf0 con 0x7f2f6c15acb0 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2f5c0019f0 con 0x7f2f6c15acb0 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 >> v1:172.21.15.97:6789/0 conn(0x7f2f6c15acb0 legacy=0x7f2f6c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f500039e0 0x7f2f50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 0x7f2f50005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b4360 con 0x7f2f50004800 2024-03-06T07:41:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5000bb80 con 0x7f2f500039e0 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5000bd80 con 0x7f2f6c06ec00 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f63400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.004+0000 7f2f6be00640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f500039e0 0x7f2f50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.005+0000 7f2f70a00640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 0x7f2f50005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.005+0000 7f2f63400640 1 -- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 msgr2=0x7f2f50005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.005+0000 7f2f63400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f6c06ec00 0x7f2f50005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.005+0000 7f2f63400640 1 -- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f500039e0 msgr2=0x7f2f50003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.005+0000 7f2f63400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f500039e0 0x7f2f50003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.005+0000 7f2f63400640 1 -- 172.21.15.97:0/1201277675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50003520 con 0x7f2f50004800 2024-03-06T07:41:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.006+0000 7f2f63400640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f2f54001750 tx=0x7f2f50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3887656fa04239e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.006+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54001370 con 0x7f2f50004800 2024-03-06T07:41:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.006+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f540052d0 con 0x7f2f50004800 2024-03-06T07:41:06.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.006+0000 7f2f6b400640 1 -- 172.21.15.97:0/1201277675 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54003830 con 0x7f2f50004800 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f72a00640 1 -- 172.21.15.97:0/1201277675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 msgr2=0x7f2f50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1201277675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f50004800 0x7f2f50004c80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f2f54001750 tx=0x7f2f50005830 comp rx=0 tx=0).stop 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f70a00640 1 -- 172.21.15.97:0/1201277675 reap_dead start 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f72a00640 1 -- 172.21.15.97:0/1201277675 shutdown_connections 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f72a00640 1 -- 172.21.15.97:0/1201277675 >> 172.21.15.97:0/1201277675 conn(0x7f2f6c0b1730 msgr2=0x7f2f6c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f72a00640 1 -- 172.21.15.97:0/1201277675 shutdown_connections 2024-03-06T07:41:06.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.007+0000 7f2f72a00640 1 -- 172.21.15.97:0/1201277675 wait complete. 2024-03-06T07:41:06.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.008+0000 7f2f72a00640 1 Processor -- start 2024-03-06T07:41:06.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.009+0000 7f2f72a00640 1 -- start start 2024-03-06T07:41:06.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.009+0000 7f2f72a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c1e61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f72a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1413d0 0x7f2f6c1e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f72a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 0x7f2f6c15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f72a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b4010 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f72a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b32b0 con 0x7f2f6c1413d0 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f72a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c072340 con 0x7f2f50004800 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 0x7f2f6c15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 0x7f2f6c15d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60122/0 (socket says 172.21.15.97:60122) 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 -- 172.21.15.97:0/1854625969 learned_addr learned my addr 172.21.15.97:0/1854625969 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f63400640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1413d0 0x7f2f6c1e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 -- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 msgr2=0x7f2f6c1e61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c1e61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 -- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1413d0 msgr2=0x7f2f6c1e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1413d0 0x7f2f6c1e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f70a00640 1 -- 172.21.15.97:0/1854625969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c1e5120 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.010+0000 7f2f63400640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1413d0 0x7f2f6c1e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.011+0000 7f2f70a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 0x7f2f6c15d0a0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0cf970 tx=0x7f2f6400f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.011+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64004440 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.011+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64019070 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.011+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c1e5400 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.011+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64003850 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.011+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c1e59c0 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.012+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c04f330 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.013+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f64012070 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.014+0000 7f2f68c00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4806b500 0x7f2f4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:06.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.015+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2f64089c30 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.015+0000 7f2f6be00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4806b500 0x7f2f4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:06.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.016+0000 7f2f6be00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4806b500 0x7f2f4806d9c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c005c30 tx=0x7f2f5c001250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:06.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.019+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f64058960 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.256+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f6c06ce60 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.257+0000 7f2f68c00640 1 -- 172.21.15.97:0/1854625969 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f2f64058260 con 0x7f2f6c1e4b60 2024-03-06T07:41:06.259 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:06.259 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:06.259 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.259+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4806b500 msgr2=0x7f2f4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.259+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4806b500 0x7f2f4806d9c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c005c30 tx=0x7f2f5c001250 comp rx=0 tx=0).stop 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.259+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 msgr2=0x7f2f6c15d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.259+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 0x7f2f6c15d0a0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0cf970 tx=0x7f2f6400f920 comp rx=0 tx=0).stop 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 shutdown_connections 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2f4806b500 0x7f2f4806d9c0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f6c1e4b60 0x7f2f6c15d0a0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f6c1413d0 0x7f2f6c1e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 --2- 172.21.15.97:0/1854625969 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2f50004800 0x7f2f6c1e61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 >> 172.21.15.97:0/1854625969 conn(0x7f2f6c0b1730 msgr2=0x7f2f6c071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:06.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 shutdown_connections 2024-03-06T07:41:06.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:06.260+0000 7f2f72a00640 1 -- 172.21.15.97:0/1854625969 wait complete. 2024-03-06T07:41:06.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:06.480 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:07.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:07.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.695+0000 7fa230800640 1 Processor -- start 2024-03-06T07:41:07.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.695+0000 7fa230800640 1 -- start start 2024-03-06T07:41:07.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.696+0000 7fa230800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa2280cec60 0x7fa2280b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.696+0000 7fa230800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2280cf340 con 0x7fa22815e230 2024-03-06T07:41:07.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.696+0000 7fa230800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2280cfe60 con 0x7fa22806edb0 2024-03-06T07:41:07.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.696+0000 7fa230800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22815e610 con 0x7fa2280cec60 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.697+0000 7fa22de00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa22815e230 0x7fa2280d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45182/0 (socket says 172.21.15.97:45182) 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.697+0000 7fa22de00640 1 -- 172.21.15.97:0/1092207385 learned_addr learned my addr 172.21.15.97:0/1092207385 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.697+0000 7fa22e800640 1 --2- 172.21.15.97:0/1092207385 >> v2:172.21.15.135:3300/0 conn(0x7fa2280cec60 0x7fa2280b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.697+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294693331 0 0) 0x7fa2280cf340 con 0x7fa22815e230 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.697+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa210003520 con 0x7fa22815e230 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853695220 0 0) 0x7fa210003520 con 0x7fa22815e230 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 >> v2:172.21.15.135:3300/0 conn(0x7fa2280cec60 msgr2=0x7fa2280b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 --2- 172.21.15.97:0/1092207385 >> v2:172.21.15.135:3300/0 conn(0x7fa2280cec60 0x7fa2280b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 >> v1:172.21.15.135:6789/0 conn(0x7fa22806edb0 legacy=0x7fa2280ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22806f390 con 0x7fa22815e230 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fa21c001670 con 0x7fa22815e230 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 >> v1:172.21.15.97:6789/0 conn(0x7fa22815e230 legacy=0x7fa2280d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa210003a00 0x7fa210003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa210004800 0x7fa210004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa21000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22815e610 con 0x7fa210005210 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21000c160 con 0x7fa210004800 2024-03-06T07:41:07.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21000c360 con 0x7fa210003a00 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22e800640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa21000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22d400640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa210004800 0x7fa210004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22e800640 1 -- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa210003a00 msgr2=0x7fa210003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.698+0000 7fa22de00640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa210003a00 0x7fa210003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.699+0000 7fa22e800640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa210003a00 0x7fa210003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.699+0000 7fa22e800640 1 -- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa210004800 msgr2=0x7fa210004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.699+0000 7fa22e800640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa210004800 0x7fa210004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.699+0000 7fa22e800640 1 -- 172.21.15.97:0/1092207385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa210003520 con 0x7fa210005210 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.699+0000 7fa22de00640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa210003a00 0x7fa210003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:07.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.700+0000 7fa22e800640 1 -- 172.21.15.97:0/1092207385 reap_dead start 2024-03-06T07:41:07.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.700+0000 7fa22e800640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa21000b9c0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa228072720 tx=0x7fa210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116d3b56eef0ffad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:07.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.700+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224009c10 con 0x7fa210005210 2024-03-06T07:41:07.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.700+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22402cbd0 con 0x7fa210005210 2024-03-06T07:41:07.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.700+0000 7fa22ca00640 1 -- 172.21.15.97:0/1092207385 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224003990 con 0x7fa210005210 2024-03-06T07:41:07.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 -- 172.21.15.97:0/1092207385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 msgr2=0x7fa21000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa21000b9c0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa228072720 tx=0x7fa210005830 comp rx=0 tx=0).stop 2024-03-06T07:41:07.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 -- 172.21.15.97:0/1092207385 shutdown_connections 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 --2- 172.21.15.97:0/1092207385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa21000b9c0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 -- 172.21.15.97:0/1092207385 >> 172.21.15.97:0/1092207385 conn(0x7fa2280b3800 msgr2=0x7fa2280b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 -- 172.21.15.97:0/1092207385 shutdown_connections 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.701+0000 7fa230800640 1 -- 172.21.15.97:0/1092207385 wait complete. 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.703+0000 7fa230800640 1 Processor -- start 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.703+0000 7fa230800640 1 -- start start 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa230800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa2280b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa230800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa228155b50 0x7fa2280c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa230800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2280af550 0x7fa2280afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa230800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280d06c0 con 0x7fa210005210 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa230800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280d03e0 con 0x7fa2280af550 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa230800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280b4320 con 0x7fa228155b50 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa22de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa2280b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa22de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa2280b7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60138/0 (socket says 172.21.15.97:60138) 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.705+0000 7fa22de00640 1 -- 172.21.15.97:0/3201181897 learned_addr learned my addr 172.21.15.97:0/3201181897 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22de00640 1 -- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa228155b50 msgr2=0x7fa2280c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22de00640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa228155b50 0x7fa2280c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22de00640 1 -- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2280af550 msgr2=0x7fa2280afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22e800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2280af550 0x7fa2280afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22de00640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2280af550 0x7fa2280afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22de00640 1 -- 172.21.15.97:0/3201181897 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2280c7050 con 0x7fa210005210 2024-03-06T07:41:07.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa22de00640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa2280b7340 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00d7e0 tx=0x7fa21c00dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:07.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c019070 con 0x7fa210005210 2024-03-06T07:41:07.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c0074e0 con 0x7fa210005210 2024-03-06T07:41:07.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c004a00 con 0x7fa210005210 2024-03-06T07:41:07.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.706+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2280c7330 con 0x7fa210005210 2024-03-06T07:41:07.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.707+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2280b4b40 con 0x7fa210005210 2024-03-06T07:41:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.708+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2280c8a60 con 0x7fa210005210 2024-03-06T07:41:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.709+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa21c004ba0 con 0x7fa210005210 2024-03-06T07:41:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.710+0000 7fa222000640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa20c06b500 0x7fa20c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.710+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa21c089c70 con 0x7fa210005210 2024-03-06T07:41:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.710+0000 7fa22d400640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa20c06b500 0x7fa20c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.711+0000 7fa22d400640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa20c06b500 0x7fa20c06d9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa228068490 tx=0x7fa218009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:07.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.714+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2280c8a60 con 0x7fa210005210 2024-03-06T07:41:07.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.963+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa22804f330 con 0x7fa210005210 2024-03-06T07:41:07.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.964+0000 7fa222000640 1 -- 172.21.15.97:0/3201181897 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fa21c058ac0 con 0x7fa210005210 2024-03-06T07:41:07.967 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:07.967 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:07.967 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa20c06b500 msgr2=0x7fa20c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa20c06b500 0x7fa20c06d9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa228068490 tx=0x7fa218009040 comp rx=0 tx=0).stop 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 msgr2=0x7fa2280b7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa2280b7340 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00d7e0 tx=0x7fa21c00dca0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 shutdown_connections 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fa20c06b500 0x7fa20c06d9c0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2280af550 0x7fa2280afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fa228155b50 0x7fa2280c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 --2- 172.21.15.97:0/3201181897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa210005210 0x7fa2280b7340 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:07.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 >> 172.21.15.97:0/3201181897 conn(0x7fa2280b3800 msgr2=0x7fa22815cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:07.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 shutdown_connections 2024-03-06T07:41:07.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:07.967+0000 7fa230800640 1 -- 172.21.15.97:0/3201181897 wait complete. 2024-03-06T07:41:08.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:08.190 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:09.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:09.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.401+0000 7fcc0c000640 1 Processor -- start 2024-03-06T07:41:09.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.401+0000 7fcc0c000640 1 -- start start 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc0c000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcc0415f9d0 0x7fcc0415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc0c000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0406cef0 con 0x7fcc04070b10 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc0c000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0406d1d0 con 0x7fcc04144c00 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc0c000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0406d4b0 con 0x7fcc0415f9d0 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc08c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcc0415f9d0 0x7fcc0415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc0a000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcc04070b10 0x7fcc04070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:45192/0 (socket says 172.21.15.97:45192) 2024-03-06T07:41:09.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc0a000640 1 -- 172.21.15.97:0/1677460461 learned_addr learned my addr 172.21.15.97:0/1677460461 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:09.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.403+0000 7fcc03e00640 1 -- 172.21.15.97:0/1677460461 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241472499 0 0) 0x7fcc0406cef0 con 0x7fcc04070b10 2024-03-06T07:41:09.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.404+0000 7fcc03e00640 1 -- 172.21.15.97:0/1677460461 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbec003520 con 0x7fcc04070b10 2024-03-06T07:41:09.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.404+0000 7fcc08c00640 1 -- 172.21.15.97:0/1677460461 >> v1:172.21.15.135:6789/0 conn(0x7fcc04144c00 legacy=0x7fcc0406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:09.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.404+0000 7fcc08c00640 1 -- 172.21.15.97:0/1677460461 >> v1:172.21.15.97:6789/0 conn(0x7fcc04070b10 legacy=0x7fcc04070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:09.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.404+0000 7fcc08c00640 1 -- 172.21.15.97:0/1677460461 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0406d790 con 0x7fcc0415f9d0 2024-03-06T07:41:09.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.405+0000 7fcc08c00640 1 --2- 172.21.15.97:0/1677460461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0415f9d0 0x7fcc0415fe60 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00097a0 tx=0x7fcbf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d8e418176c9d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:09.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc03e00640 1 -- 172.21.15.97:0/1677460461 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0035020 con 0x7fcc0415f9d0 2024-03-06T07:41:09.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc03e00640 1 -- 172.21.15.97:0/1677460461 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf0031830 con 0x7fcc0415f9d0 2024-03-06T07:41:09.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc03e00640 1 -- 172.21.15.97:0/1677460461 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0031c00 con 0x7fcc0415f9d0 2024-03-06T07:41:09.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 -- 172.21.15.97:0/1677460461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0415f9d0 msgr2=0x7fcc0415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:09.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1677460461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0415f9d0 0x7fcc0415fe60 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00097a0 tx=0x7fcbf002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 -- 172.21.15.97:0/1677460461 shutdown_connections 2024-03-06T07:41:09.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1677460461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0415f9d0 0x7fcc0415fe60 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 -- 172.21.15.97:0/1677460461 >> 172.21.15.97:0/1677460461 conn(0x7fcc040b1730 msgr2=0x7fcc040b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:09.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 -- 172.21.15.97:0/1677460461 shutdown_connections 2024-03-06T07:41:09.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.406+0000 7fcc0c000640 1 -- 172.21.15.97:0/1677460461 wait complete. 2024-03-06T07:41:09.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.408+0000 7fcc0c000640 1 Processor -- start 2024-03-06T07:41:09.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.408+0000 7fcc0c000640 1 -- start start 2024-03-06T07:41:09.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc0c000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 0x7fcc0415e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:09.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc0c000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc04144c00 0x7fcc0413c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:09.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc0c000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc0415f9d0 0x7fcc0413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:09.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc0c000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406ccd0 con 0x7fcc04070b10 2024-03-06T07:41:09.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc0c000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406cb50 con 0x7fcc0415f9d0 2024-03-06T07:41:09.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc0c000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406df90 con 0x7fcc04144c00 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc09600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 0x7fcc0415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc09600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 0x7fcc0415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60158/0 (socket says 172.21.15.97:60158) 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.409+0000 7fcc09600640 1 -- 172.21.15.97:0/1700110039 learned_addr learned my addr 172.21.15.97:0/1700110039 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc09600640 1 -- 172.21.15.97:0/1700110039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc04144c00 msgr2=0x7fcc0413c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc09600640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc04144c00 0x7fcc0413c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc09600640 1 -- 172.21.15.97:0/1700110039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc0415f9d0 msgr2=0x7fcc0413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc09600640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc0415f9d0 0x7fcc0413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc09600640 1 -- 172.21.15.97:0/1700110039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0415ee00 con 0x7fcc04070b10 2024-03-06T07:41:09.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc09600640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 0x7fcc0415e820 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800c430 tx=0x7fcbf800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:09.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8011790 con 0x7fcc04070b10 2024-03-06T07:41:09.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf8017070 con 0x7fcc04070b10 2024-03-06T07:41:09.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8011bb0 con 0x7fcc04070b10 2024-03-06T07:41:09.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.410+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0415f0e0 con 0x7fcc04070b10 2024-03-06T07:41:09.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.411+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0415f6a0 con 0x7fcc04070b10 2024-03-06T07:41:09.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.412+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0404f330 con 0x7fcc04070b10 2024-03-06T07:41:09.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.412+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbf801c460 con 0x7fcc04070b10 2024-03-06T07:41:09.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.413+0000 7fcc01600640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcbe406b4d0 0x7fcbe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:09.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.413+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcbf808a1a0 con 0x7fcc04070b10 2024-03-06T07:41:09.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.413+0000 7fcc08c00640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcbe406b4d0 0x7fcbe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:09.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.414+0000 7fcc08c00640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcbe406b4d0 0x7fcbe406d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0036000 tx=0x7fcbf0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:09.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.418+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbf8058f60 con 0x7fcc04070b10 2024-03-06T07:41:09.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.656+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc040721b0 con 0x7fcc04070b10 2024-03-06T07:41:09.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.657+0000 7fcc01600640 1 -- 172.21.15.97:0/1700110039 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fcbf8058860 con 0x7fcc04070b10 2024-03-06T07:41:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:09.660 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:09.660 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:09.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcbe406b4d0 msgr2=0x7fcbe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcbe406b4d0 0x7fcbe406d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0036000 tx=0x7fcbf0030000 comp rx=0 tx=0).stop 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 msgr2=0x7fcc0415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 0x7fcc0415e820 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800c430 tx=0x7fcbf800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 shutdown_connections 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fcbe406b4d0 0x7fcbe406d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc0415f9d0 0x7fcc0413fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fcc04144c00 0x7fcc0413c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 --2- 172.21.15.97:0/1700110039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc04070b10 0x7fcc0415e820 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 >> 172.21.15.97:0/1700110039 conn(0x7fcc040b1730 msgr2=0x7fcc04145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 shutdown_connections 2024-03-06T07:41:09.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:09.660+0000 7fcc0c000640 1 -- 172.21.15.97:0/1700110039 wait complete. 2024-03-06T07:41:09.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:09.882 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:10.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:10.658 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:10.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:11.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.090+0000 7fed12e00640 1 Processor -- start 2024-03-06T07:41:11.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.091+0000 7fed12e00640 1 -- start start 2024-03-06T07:41:11.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.092+0000 7fed12e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fed0c161580 0x7fed0c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.092+0000 7fed12e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c0d1e90 0x7fed0c0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.092+0000 7fed12e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c161960 con 0x7fed0c0d28a0 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.092+0000 7fed12e00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c0cf6f0 con 0x7fed0c161580 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.092+0000 7fed12e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0cf9d0 con 0x7fed0c0d1e90 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.092+0000 7fed11a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fed0c161580 0x7fed0c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fed0c161580 0x7fed0c0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60168/0 (socket says 172.21.15.97:60168) 2024-03-06T07:41:11.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11a00640 1 -- 172.21.15.97:0/2881360022 learned_addr learned my addr 172.21.15.97:0/2881360022 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11000640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c0d1e90 0x7fed0c0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11a00640 1 -- 172.21.15.97:0/2881360022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c0d1e90 msgr2=0x7fed0c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11a00640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c0d1e90 0x7fed0c0d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11a00640 1 -- 172.21.15.97:0/2881360022 >> v1:172.21.15.97:6789/0 conn(0x7fed0c0d28a0 legacy=0x7fed0c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11a00640 1 -- 172.21.15.97:0/2881360022 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c0cfcb0 con 0x7fed0c161580 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed0be00640 1 -- 172.21.15.97:0/2881360022 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255055524 0 0) 0x7fed0c161960 con 0x7fed0c0d28a0 2024-03-06T07:41:11.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.093+0000 7fed11000640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c0d1e90 0x7fed0c0d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:11.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.095+0000 7fed11a00640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c161580 0x7fed0c0d1950 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fed00009870 tx=0x7fed0002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35c0537cf479d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:11.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.095+0000 7fed0be00640 1 -- 172.21.15.97:0/2881360022 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00034070 con 0x7fed0c161580 2024-03-06T07:41:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.095+0000 7fed0be00640 1 -- 172.21.15.97:0/2881360022 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00030840 con 0x7fed0c161580 2024-03-06T07:41:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.095+0000 7fed0be00640 1 -- 172.21.15.97:0/2881360022 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00030c10 con 0x7fed0c161580 2024-03-06T07:41:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 -- 172.21.15.97:0/2881360022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c161580 msgr2=0x7fed0c0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c161580 0x7fed0c0d1950 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fed00009870 tx=0x7fed0002b980 comp rx=0 tx=0).stop 2024-03-06T07:41:11.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 -- 172.21.15.97:0/2881360022 shutdown_connections 2024-03-06T07:41:11.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c0d1e90 0x7fed0c0d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 --2- 172.21.15.97:0/2881360022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c161580 0x7fed0c0d1950 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 -- 172.21.15.97:0/2881360022 >> 172.21.15.97:0/2881360022 conn(0x7fed0c0d3d80 msgr2=0x7fed0c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:11.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 -- 172.21.15.97:0/2881360022 shutdown_connections 2024-03-06T07:41:11.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.096+0000 7fed12e00640 1 -- 172.21.15.97:0/2881360022 wait complete. 2024-03-06T07:41:11.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.098+0000 7fed12e00640 1 Processor -- start 2024-03-06T07:41:11.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.098+0000 7fed12e00640 1 -- start start 2024-03-06T07:41:11.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed12e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 0x7fed0c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:11.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed12e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed0c0d28a0 0x7fed0c0b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:11.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed12e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c161580 0x7fed0c0b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:11.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed12e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0cf3a0 con 0x7fed0c0d1e90 2024-03-06T07:41:11.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed12e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0b39a0 con 0x7fed0c0d28a0 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed12e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0b4ed0 con 0x7fed0c161580 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed11a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 0x7fed0c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed11a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 0x7fed0c0cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60170/0 (socket says 172.21.15.97:60170) 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.099+0000 7fed11a00640 1 -- 172.21.15.97:0/3111957636 learned_addr learned my addr 172.21.15.97:0/3111957636 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed11a00640 1 -- 172.21.15.97:0/3111957636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c161580 msgr2=0x7fed0c0b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed11a00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c161580 0x7fed0c0b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed11a00640 1 -- 172.21.15.97:0/3111957636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed0c0d28a0 msgr2=0x7fed0c0b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed11a00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed0c0d28a0 0x7fed0c0b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed11a00640 1 -- 172.21.15.97:0/3111957636 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c0b6e80 con 0x7fed0c0d1e90 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed11a00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 0x7fed0c0cc130 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fed0002b9c0 tx=0x7fed000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00034040 con 0x7fed0c0d1e90 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00039d90 con 0x7fed0c0d1e90 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00043610 con 0x7fed0c0d1e90 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c0b7120 con 0x7fed0c0d1e90 2024-03-06T07:41:11.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.100+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c0b76e0 con 0x7fed0c0d1e90 2024-03-06T07:41:11.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.102+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c0d1cb0 con 0x7fed0c0d1e90 2024-03-06T07:41:11.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.102+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed0004a060 con 0x7fed0c0d1e90 2024-03-06T07:41:11.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.103+0000 7fed09600640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fecec06b500 0x7fecec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:11.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.103+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fed000ac420 con 0x7fed0c0d1e90 2024-03-06T07:41:11.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.104+0000 7fed11000640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fecec06b500 0x7fecec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:11.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.104+0000 7fed11000640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fecec06b500 0x7fecec06d9c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006fd0 tx=0x7fecf8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:11.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.108+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed0c0d1cb0 con 0x7fed0c0d1e90 2024-03-06T07:41:11.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.342+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c04f330 con 0x7fed0c0d1e90 2024-03-06T07:41:11.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.343+0000 7fed09600640 1 -- 172.21.15.97:0/3111957636 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fed0003e070 con 0x7fed0c0d1e90 2024-03-06T07:41:11.345 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:11.345 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:11.346 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fecec06b500 msgr2=0x7fecec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fecec06b500 0x7fecec06d9c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006fd0 tx=0x7fecf8008040 comp rx=0 tx=0).stop 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 msgr2=0x7fed0c0cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 0x7fed0c0cc130 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fed0002b9c0 tx=0x7fed000046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 shutdown_connections 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fecec06b500 0x7fecec06d9c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fed0c161580 0x7fed0c0b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed0c0d28a0 0x7fed0c0b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 --2- 172.21.15.97:0/3111957636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed0c0d1e90 0x7fed0c0cc130 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:11.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.345+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 >> 172.21.15.97:0/3111957636 conn(0x7fed0c0d3d80 msgr2=0x7fed0c06b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:11.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.346+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 shutdown_connections 2024-03-06T07:41:11.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:11.346+0000 7fed12e00640 1 -- 172.21.15.97:0/3111957636 wait complete. 2024-03-06T07:41:11.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:11.570 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:12.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:12.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.782+0000 7f3b4e600640 1 Processor -- start 2024-03-06T07:41:12.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.782+0000 7f3b4e600640 1 -- start start 2024-03-06T07:41:12.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.783+0000 7f3b4e600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3b4815f470 0x7f3b480d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.783+0000 7f3b4e600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b480d0df0 0x7f3b48160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.783+0000 7f3b4e600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b480d1200 con 0x7f3b4815e0d0 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.783+0000 7f3b4e600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b480d2f60 con 0x7f3b480d0df0 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.783+0000 7f3b4e600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b480cea60 con 0x7f3b4815f470 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b480d0df0 0x7f3b48160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b480d0df0 0x7f3b48160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60182/0 (socket says 172.21.15.97:60182) 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 -- 172.21.15.97:0/58659493 learned_addr learned my addr 172.21.15.97:0/58659493 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 -- 172.21.15.97:0/58659493 >> v2:172.21.15.135:3300/0 conn(0x7f3b4815f470 msgr2=0x7f3b480d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 --2- 172.21.15.97:0/58659493 >> v2:172.21.15.135:3300/0 conn(0x7f3b4815f470 0x7f3b480d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:12.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 -- 172.21.15.97:0/58659493 >> v1:172.21.15.135:6789/0 conn(0x7f3b4815e0d0 legacy=0x7f3b480d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:12.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.784+0000 7f3b47e00640 1 -- 172.21.15.97:0/58659493 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480d2350 con 0x7f3b480d0df0 2024-03-06T07:41:12.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.785+0000 7f3b47e00640 1 --2- 172.21.15.97:0/58659493 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b480d0df0 0x7f3b48160dc0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f3b480c77f0 tx=0x7f3b4002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1ca27bfd6dcfe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:12.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b46a00640 1 -- 172.21.15.97:0/58659493 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4002cb20 con 0x7f3b480d0df0 2024-03-06T07:41:12.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b46a00640 1 -- 172.21.15.97:0/58659493 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4002ccc0 con 0x7f3b480d0df0 2024-03-06T07:41:12.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b46a00640 1 -- 172.21.15.97:0/58659493 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40034610 con 0x7f3b480d0df0 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 -- 172.21.15.97:0/58659493 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b480d0df0 msgr2=0x7f3b48160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 --2- 172.21.15.97:0/58659493 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b480d0df0 0x7f3b48160dc0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f3b480c77f0 tx=0x7f3b4002bf80 comp rx=0 tx=0).stop 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 -- 172.21.15.97:0/58659493 shutdown_connections 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 --2- 172.21.15.97:0/58659493 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b480d0df0 0x7f3b48160dc0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 --2- 172.21.15.97:0/58659493 >> v2:172.21.15.135:3300/0 conn(0x7f3b4815f470 0x7f3b480d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 -- 172.21.15.97:0/58659493 >> 172.21.15.97:0/58659493 conn(0x7f3b480d3ec0 msgr2=0x7f3b480d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 -- 172.21.15.97:0/58659493 shutdown_connections 2024-03-06T07:41:12.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.786+0000 7f3b4e600640 1 -- 172.21.15.97:0/58659493 wait complete. 2024-03-06T07:41:12.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.788+0000 7f3b4e600640 1 Processor -- start 2024-03-06T07:41:12.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.788+0000 7f3b4e600640 1 -- start start 2024-03-06T07:41:12.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b4e600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b480d0df0 0x7f3b481f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:12.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b4e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 0x7f3b480cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:12.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b4e600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b4815f470 0x7f3b480cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b4e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480d2bf0 con 0x7f3b4815e0d0 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b4e600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480d2910 con 0x7f3b4815f470 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b4e600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4815fb30 con 0x7f3b480d0df0 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.789+0000 7f3b3ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 0x7f3b480cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 0x7f3b480cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60198/0 (socket says 172.21.15.97:60198) 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 -- 172.21.15.97:0/619499042 learned_addr learned my addr 172.21.15.97:0/619499042 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 -- 172.21.15.97:0/619499042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b480d0df0 msgr2=0x7f3b481f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b480d0df0 0x7f3b481f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 -- 172.21.15.97:0/619499042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b4815f470 msgr2=0x7f3b480cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b4815f470 0x7f3b480cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 -- 172.21.15.97:0/619499042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b481f7d40 con 0x7f3b4815e0d0 2024-03-06T07:41:12.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3ea00640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 0x7f3b480cd160 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000b4a0 tx=0x7f3b3000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:12.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30014070 con 0x7f3b4815e0d0 2024-03-06T07:41:12.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30010610 con 0x7f3b4815e0d0 2024-03-06T07:41:12.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.790+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30010a00 con 0x7f3b4815e0d0 2024-03-06T07:41:12.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.791+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b481f8020 con 0x7f3b4815e0d0 2024-03-06T07:41:12.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.791+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b481f52a0 con 0x7f3b4815e0d0 2024-03-06T07:41:12.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.792+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4804f330 con 0x7f3b4815e0d0 2024-03-06T07:41:12.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.793+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b30017070 con 0x7f3b4815e0d0 2024-03-06T07:41:12.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.794+0000 7f3b3fe00640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2406b4d0 0x7f3b2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:12.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.794+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3b30089c10 con 0x7f3b4815e0d0 2024-03-06T07:41:12.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.794+0000 7f3b47400640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2406b4d0 0x7f3b2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:12.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.795+0000 7f3b47400640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2406b4d0 0x7f3b2406d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3b38001560 tx=0x7f3b38007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:12.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:12.799+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b30058a70 con 0x7f3b4815e0d0 2024-03-06T07:41:13.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.033+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b4808d920 con 0x7f3b4815e0d0 2024-03-06T07:41:13.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.034+0000 7f3b3fe00640 1 -- 172.21.15.97:0/619499042 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f3b30058370 con 0x7f3b4815e0d0 2024-03-06T07:41:13.036 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:13.036 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:13.036 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2406b4d0 msgr2=0x7f3b2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2406b4d0 0x7f3b2406d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3b38001560 tx=0x7f3b38007990 comp rx=0 tx=0).stop 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 msgr2=0x7f3b480cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 0x7f3b480cd160 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000b4a0 tx=0x7f3b3000b960 comp rx=0 tx=0).stop 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 shutdown_connections 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f3b2406b4d0 0x7f3b2406d990 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b4815f470 0x7f3b480cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4815e0d0 0x7f3b480cd160 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 --2- 172.21.15.97:0/619499042 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f3b480d0df0 0x7f3b481f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.036+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 >> 172.21.15.97:0/619499042 conn(0x7f3b480d3ec0 msgr2=0x7f3b480b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:13.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.037+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 shutdown_connections 2024-03-06T07:41:13.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:13.037+0000 7f3b4e600640 1 -- 172.21.15.97:0/619499042 wait complete. 2024-03-06T07:41:13.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:13.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:14.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:14.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.474+0000 7f6a99a00640 1 Processor -- start 2024-03-06T07:41:14.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.474+0000 7f6a99a00640 1 -- start start 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a99a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6a9415f9d0 0x7f6a9415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a99a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9406cef0 con 0x7f6a94070b10 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a99a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9406d1d0 con 0x7f6a94144c00 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a99a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9406d4b0 con 0x7f6a9415f9d0 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a8a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6a9415f9d0 0x7f6a9415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a8a000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6a9415f9d0 0x7f6a9415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55714/0 (socket says 172.21.15.97:55714) 2024-03-06T07:41:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a8a000640 1 -- 172.21.15.97:0/3951937986 learned_addr learned my addr 172.21.15.97:0/3951937986 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:14.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.476+0000 7f6a93400640 1 --1- 172.21.15.97:0/3951937986 >> v1:172.21.15.97:6789/0 conn(0x7f6a94070b10 0x7f6a94070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39614/0 (socket says 172.21.15.97:39614) 2024-03-06T07:41:14.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.477+0000 7f6a8a000640 1 -- 172.21.15.97:0/3951937986 >> v1:172.21.15.135:6789/0 conn(0x7f6a94144c00 legacy=0x7f6a9406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:14.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.477+0000 7f6a8a000640 1 -- 172.21.15.97:0/3951937986 >> v1:172.21.15.97:6789/0 conn(0x7f6a94070b10 legacy=0x7f6a94070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:14.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.477+0000 7f6a8a000640 1 -- 172.21.15.97:0/3951937986 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9406d790 con 0x7f6a9415f9d0 2024-03-06T07:41:14.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.478+0000 7f6a8a000640 1 --2- 172.21.15.97:0/3951937986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a9415fe60 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009870 tx=0x7f6a7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7c11d36cfe1b36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:14.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.479+0000 7f6a92000640 1 -- 172.21.15.97:0/3951937986 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c034070 con 0x7f6a9415f9d0 2024-03-06T07:41:14.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.479+0000 7f6a92000640 1 -- 172.21.15.97:0/3951937986 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c030860 con 0x7f6a9415f9d0 2024-03-06T07:41:14.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.479+0000 7f6a92000640 1 -- 172.21.15.97:0/3951937986 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c030c30 con 0x7f6a9415f9d0 2024-03-06T07:41:14.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.479+0000 7f6a99a00640 1 -- 172.21.15.97:0/3951937986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 msgr2=0x7f6a9415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:14.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.479+0000 7f6a99a00640 1 --2- 172.21.15.97:0/3951937986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a9415fe60 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009870 tx=0x7f6a7c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.480+0000 7f6a99a00640 1 -- 172.21.15.97:0/3951937986 shutdown_connections 2024-03-06T07:41:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.480+0000 7f6a99a00640 1 --2- 172.21.15.97:0/3951937986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a9415fe60 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.480+0000 7f6a99a00640 1 -- 172.21.15.97:0/3951937986 >> 172.21.15.97:0/3951937986 conn(0x7f6a940b1730 msgr2=0x7f6a940b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.480+0000 7f6a99a00640 1 -- 172.21.15.97:0/3951937986 shutdown_connections 2024-03-06T07:41:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.480+0000 7f6a99a00640 1 -- 172.21.15.97:0/3951937986 wait complete. 2024-03-06T07:41:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.481+0000 7f6a99a00640 1 Processor -- start 2024-03-06T07:41:14.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.481+0000 7f6a99a00640 1 -- start start 2024-03-06T07:41:14.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.482+0000 7f6a99a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a94070b10 0x7f6a94136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:14.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.482+0000 7f6a99a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a94144c00 0x7f6a941e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.482+0000 7f6a99a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a941e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.482+0000 7f6a99a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9406ccd0 con 0x7f6a9415f9d0 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.482+0000 7f6a99a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9406cb50 con 0x7f6a94144c00 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.482+0000 7f6a99a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9406df90 con 0x7f6a94070b10 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a941e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a941e8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55730/0 (socket says 172.21.15.97:55730) 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 -- 172.21.15.97:0/4020606673 learned_addr learned my addr 172.21.15.97:0/4020606673 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a8a000640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a94144c00 0x7f6a941e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a92a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a94070b10 0x7f6a94136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:14.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 -- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a94070b10 msgr2=0x7f6a94136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a94070b10 0x7f6a94136190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 -- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a94144c00 msgr2=0x7f6a941e8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a94144c00 0x7f6a941e8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a93400640 1 -- 172.21.15.97:0/4020606673 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9415f060 con 0x7f6a9415f9d0 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.483+0000 7f6a8a000640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a94144c00 0x7f6a941e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.484+0000 7f6a93400640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a941e8fa0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c780 tx=0x7f6a8c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.484+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c017070 con 0x7f6a9415f9d0 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.484+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c00a640 con 0x7f6a9415f9d0 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.484+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a941e6500 con 0x7f6a9415f9d0 2024-03-06T07:41:14.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.484+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c00aa30 con 0x7f6a9415f9d0 2024-03-06T07:41:14.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.484+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a941e6ac0 con 0x7f6a9415f9d0 2024-03-06T07:41:14.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.485+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a941e6310 con 0x7f6a9415f9d0 2024-03-06T07:41:14.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.486+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a8c01a070 con 0x7f6a9415f9d0 2024-03-06T07:41:14.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.487+0000 7f6a8b400640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a7006b4d0 0x7f6a7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:14.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.487+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6a8c089750 con 0x7f6a9415f9d0 2024-03-06T07:41:14.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.487+0000 7f6a92a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a7006b4d0 0x7f6a7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:14.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.488+0000 7f6a92a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a7006b4d0 0x7f6a7006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6a84003d90 tx=0x7f6a84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:14.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.491+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a8c057f60 con 0x7f6a9415f9d0 2024-03-06T07:41:14.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.723+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a9404f330 con 0x7f6a9415f9d0 2024-03-06T07:41:14.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.724+0000 7f6a8b400640 1 -- 172.21.15.97:0/4020606673 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f6a8c05c000 con 0x7f6a9415f9d0 2024-03-06T07:41:14.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:14.726 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:14.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.726+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a7006b4d0 msgr2=0x7f6a7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:14.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.726+0000 7f6a99a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a7006b4d0 0x7f6a7006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6a84003d90 tx=0x7f6a84009040 comp rx=0 tx=0).stop 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.726+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 msgr2=0x7f6a941e8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.726+0000 7f6a99a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a941e8fa0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c780 tx=0x7f6a8c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 shutdown_connections 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6a7006b4d0 0x7f6a7006d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a9415f9d0 0x7f6a941e8fa0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a94144c00 0x7f6a941e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 --2- 172.21.15.97:0/4020606673 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6a94070b10 0x7f6a94136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 >> 172.21.15.97:0/4020606673 conn(0x7f6a940b1730 msgr2=0x7f6a940ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 shutdown_connections 2024-03-06T07:41:14.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:14.727+0000 7f6a99a00640 1 -- 172.21.15.97:0/4020606673 wait complete. 2024-03-06T07:41:14.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:14.948 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:15.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:16.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.169+0000 7feee7a00640 1 Processor -- start 2024-03-06T07:41:16.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.169+0000 7feee7a00640 1 -- start start 2024-03-06T07:41:16.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.172+0000 7feee7a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee00b4680 0x7feee006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:16.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.172+0000 7feee7a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feee0072340 0x7feee006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.172+0000 7feee7a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee015b090 con 0x7feee015acb0 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.172+0000 7feee7a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee00b2a30 con 0x7feee0072340 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.172+0000 7feee7a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee00b4360 con 0x7feee00b4680 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.172+0000 7feee5a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feee0072340 0x7feee006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feee5a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feee0072340 0x7feee006ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55736/0 (socket says 172.21.15.97:55736) 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feee5a00640 1 -- 172.21.15.97:0/805211391 learned_addr learned my addr 172.21.15.97:0/805211391 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feed7e00640 1 --2- 172.21.15.97:0/805211391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee00b4680 0x7feee006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feee5a00640 1 -- 172.21.15.97:0/805211391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee00b4680 msgr2=0x7feee006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:16.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feee5a00640 1 --2- 172.21.15.97:0/805211391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee00b4680 0x7feee006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feee5a00640 1 -- 172.21.15.97:0/805211391 >> v1:172.21.15.135:6789/0 conn(0x7feee015acb0 legacy=0x7feee00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:16.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.173+0000 7feee5a00640 1 -- 172.21.15.97:0/805211391 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee00b2cf0 con 0x7feee0072340 2024-03-06T07:41:16.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.174+0000 7feee5a00640 1 --2- 172.21.15.97:0/805211391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee006ea30 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7feed8009e50 tx=0x7feed802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fb3db42d349fce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:16.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.174+0000 7feedfe00640 1 -- 172.21.15.97:0/805211391 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed802ca70 con 0x7feee0072340 2024-03-06T07:41:16.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.174+0000 7feedfe00640 1 -- 172.21.15.97:0/805211391 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed802cc10 con 0x7feee0072340 2024-03-06T07:41:16.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.174+0000 7feedfe00640 1 -- 172.21.15.97:0/805211391 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80343f0 con 0x7feee0072340 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 -- 172.21.15.97:0/805211391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 msgr2=0x7feee006ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 --2- 172.21.15.97:0/805211391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee006ea30 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7feed8009e50 tx=0x7feed802bed0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 -- 172.21.15.97:0/805211391 shutdown_connections 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 --2- 172.21.15.97:0/805211391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee006ea30 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 --2- 172.21.15.97:0/805211391 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee00b4680 0x7feee006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 -- 172.21.15.97:0/805211391 >> 172.21.15.97:0/805211391 conn(0x7feee00b1730 msgr2=0x7feee00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 -- 172.21.15.97:0/805211391 shutdown_connections 2024-03-06T07:41:16.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.175+0000 7feee7a00640 1 -- 172.21.15.97:0/805211391 wait complete. 2024-03-06T07:41:16.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.178+0000 7feee7a00640 1 Processor -- start 2024-03-06T07:41:16.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.178+0000 7feee7a00640 1 -- start start 2024-03-06T07:41:16.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee7a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee0155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee7a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feee00b4680 0x7feee00a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee7a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee015acb0 0x7feee00a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee7a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00b4010 con 0x7feee0072340 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee7a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00b32b0 con 0x7feee00b4680 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee7a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee00ccfb0 con 0x7feee015acb0 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee0155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee0155e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55746/0 (socket says 172.21.15.97:55746) 2024-03-06T07:41:16.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 -- 172.21.15.97:0/43927286 learned_addr learned my addr 172.21.15.97:0/43927286 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 -- 172.21.15.97:0/43927286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee015acb0 msgr2=0x7feee00a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee015acb0 0x7feee00a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 -- 172.21.15.97:0/43927286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feee00b4680 msgr2=0x7feee00a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feee00b4680 0x7feee00a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.180+0000 7feee5000640 1 -- 172.21.15.97:0/43927286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee01e5c60 con 0x7feee0072340 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.181+0000 7feee5000640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee0155e80 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7feed000b3f0 tx=0x7feed000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.181+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0015020 con 0x7feee0072340 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.181+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed0011830 con 0x7feee0072340 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.181+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0011c20 con 0x7feee0072340 2024-03-06T07:41:16.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.181+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee01e5f00 con 0x7feee0072340 2024-03-06T07:41:16.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.181+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee01e64c0 con 0x7feee0072340 2024-03-06T07:41:16.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.182+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee004f330 con 0x7feee0072340 2024-03-06T07:41:16.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.184+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feed0018020 con 0x7feee0072340 2024-03-06T07:41:16.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.185+0000 7feedd600640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feebc06b4d0 0x7feebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:16.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.185+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feed0089d90 con 0x7feee0072340 2024-03-06T07:41:16.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.185+0000 7feed7e00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feebc06b4d0 0x7feebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:16.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.186+0000 7feed7e00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feebc06b4d0 0x7feebc06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7feec8006fd0 tx=0x7feec8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:16.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.190+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feed001f030 con 0x7feee0072340 2024-03-06T07:41:16.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.423+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feee006ce60 con 0x7feee0072340 2024-03-06T07:41:16.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.424+0000 7feedd600640 1 -- 172.21.15.97:0/43927286 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7feed0058c80 con 0x7feee0072340 2024-03-06T07:41:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:16.426 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:16.427 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feebc06b4d0 msgr2=0x7feebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feebc06b4d0 0x7feebc06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7feec8006fd0 tx=0x7feec8008040 comp rx=0 tx=0).stop 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 msgr2=0x7feee0155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee0155e80 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7feed000b3f0 tx=0x7feed000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 shutdown_connections 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7feebc06b4d0 0x7feebc06d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7feee015acb0 0x7feee00a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feee00b4680 0x7feee00a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 --2- 172.21.15.97:0/43927286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feee0072340 0x7feee0155e80 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:16.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 >> 172.21.15.97:0/43927286 conn(0x7feee00b1730 msgr2=0x7feee00b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:16.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 shutdown_connections 2024-03-06T07:41:16.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:16.426+0000 7feee7a00640 1 -- 172.21.15.97:0/43927286 wait complete. 2024-03-06T07:41:16.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:16.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:17.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.859+0000 7f506fe00640 1 Processor -- start 2024-03-06T07:41:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.859+0000 7f506fe00640 1 -- start start 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.860+0000 7f506fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 0x7f5068071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.860+0000 7f506fe00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f50680cf3e0 0x7f50680724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.860+0000 7f506fe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50680cabf0 con 0x7f50680c9ec0 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.860+0000 7f506fe00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50680ce900 con 0x7f50680cf3e0 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.860+0000 7f506fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50680cebe0 con 0x7f5068144660 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.861+0000 7f506de00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f50680c9ec0 0x7f50680ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39628/0 (socket says 172.21.15.97:39628) 2024-03-06T07:41:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.861+0000 7f506de00640 1 -- 172.21.15.97:0/4229526074 learned_addr learned my addr 172.21.15.97:0/4229526074 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.861+0000 7f506ca00640 1 --2- 172.21.15.97:0/4229526074 >> v2:172.21.15.135:3300/0 conn(0x7f50680cf3e0 0x7f50680724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.861+0000 7f506d400640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 0x7f5068071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.861+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517264166 0 0) 0x7f50680cabf0 con 0x7f50680c9ec0 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.861+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5050003520 con 0x7f50680c9ec0 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1882907949 0 0) 0x7f5050003520 con 0x7f50680c9ec0 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 msgr2=0x7f5068071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 0x7f5068071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 >> v2:172.21.15.135:3300/0 conn(0x7f50680cf3e0 msgr2=0x7f50680724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 --2- 172.21.15.97:0/4229526074 >> v2:172.21.15.135:3300/0 conn(0x7f50680cf3e0 0x7f50680724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50680ceec0 con 0x7f50680c9ec0 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f50600019f0 con 0x7f50680c9ec0 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 >> v1:172.21.15.97:6789/0 conn(0x7f50680c9ec0 legacy=0x7f50680ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50500039b0 0x7f5050003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5050004800 0x7f5050004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 0x7f50500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50680ce900 con 0x7f50500039b0 2024-03-06T07:41:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50680cebe0 con 0x7f5050004800 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505000b9c0 con 0x7f5068144660 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.862+0000 7f506d400640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50500039b0 0x7f5050003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506de00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 0x7f50500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506ca00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5050004800 0x7f5050004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506d400640 1 -- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 msgr2=0x7f50500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506d400640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5068144660 0x7f50500051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506d400640 1 -- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5050004800 msgr2=0x7f5050004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506d400640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5050004800 0x7f5050004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.863+0000 7f506d400640 1 -- 172.21.15.97:0/4229526074 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050003520 con 0x7f50500039b0 2024-03-06T07:41:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.864+0000 7f506d400640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50500039b0 0x7f5050003dd0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f505c009ac0 tx=0x7f5050005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b1e13629aa583d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.865+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c039070 con 0x7f50500039b0 2024-03-06T07:41:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.865+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c02c6b0 con 0x7f50500039b0 2024-03-06T07:41:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.865+0000 7f5067e00640 1 -- 172.21.15.97:0/4229526074 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c009690 con 0x7f50500039b0 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.865+0000 7f506fe00640 1 -- 172.21.15.97:0/4229526074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50500039b0 msgr2=0x7f5050003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.865+0000 7f506fe00640 1 --2- 172.21.15.97:0/4229526074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50500039b0 0x7f5050003dd0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f505c009ac0 tx=0x7f5050005830 comp rx=0 tx=0).stop 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.865+0000 7f506de00640 1 -- 172.21.15.97:0/4229526074 reap_dead start 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.866+0000 7f506fe00640 1 -- 172.21.15.97:0/4229526074 shutdown_connections 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.866+0000 7f506fe00640 1 -- 172.21.15.97:0/4229526074 >> 172.21.15.97:0/4229526074 conn(0x7f50680d0b90 msgr2=0x7f50680d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.866+0000 7f506fe00640 1 -- 172.21.15.97:0/4229526074 shutdown_connections 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.866+0000 7f506fe00640 1 -- 172.21.15.97:0/4229526074 wait complete. 2024-03-06T07:41:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.867+0000 7f506fe00640 1 Processor -- start 2024-03-06T07:41:17.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.868+0000 7f506fe00640 1 -- start start 2024-03-06T07:41:17.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506fe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50500039b0 0x7f506815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 0x7f506813a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506fe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f506813ad40 0x7f506806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50680720f0 con 0x7f506813a190 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506fe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068071f70 con 0x7f506813ad40 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506fe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5068072270 con 0x7f50500039b0 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 0x7f506813a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 0x7f506813a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55754/0 (socket says 172.21.15.97:55754) 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 -- 172.21.15.97:0/3886516082 learned_addr learned my addr 172.21.15.97:0/3886516082 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506d400640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50500039b0 0x7f506815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506de00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f506813ad40 0x7f506806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 -- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50500039b0 msgr2=0x7f506815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50500039b0 0x7f506815d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 -- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f506813ad40 msgr2=0x7f506806f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:17.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f506813ad40 0x7f506806f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:17.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506ca00640 1 -- 172.21.15.97:0/3886516082 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50681393d0 con 0x7f506813a190 2024-03-06T07:41:17.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.869+0000 7f506d400640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50500039b0 0x7f506815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:17.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.870+0000 7f506ca00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 0x7f506813a800 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f50540108e0 tx=0x7f5054010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.870+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505400f040 con 0x7f506813a190 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.870+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505400eb60 con 0x7f506813a190 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.870+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50681396b0 con 0x7f506813a190 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.870+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505401c400 con 0x7f506813a190 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.870+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506815dc90 con 0x7f506813a190 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.871+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506813b120 con 0x7f506813a190 2024-03-06T07:41:17.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.872+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f505401c5a0 con 0x7f506813a190 2024-03-06T07:41:17.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.873+0000 7f5065600640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504806b500 0x7f504806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:17.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.873+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f505408a8c0 con 0x7f506813a190 2024-03-06T07:41:17.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.874+0000 7f506d400640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504806b500 0x7f504806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:17.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.875+0000 7f506d400640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504806b500 0x7f504806d9c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f505c036000 tx=0x7f505c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:17.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:17.879+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5054018030 con 0x7f506813a190 2024-03-06T07:41:18.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.106+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f506804f330 con 0x7f506813a190 2024-03-06T07:41:18.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.107+0000 7f5065600640 1 -- 172.21.15.97:0/3886516082 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f5054059720 con 0x7f506813a190 2024-03-06T07:41:18.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:18.109 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:18.110 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504806b500 msgr2=0x7f504806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504806b500 0x7f504806d9c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f505c036000 tx=0x7f505c030000 comp rx=0 tx=0).stop 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 msgr2=0x7f506813a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 0x7f506813a800 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f50540108e0 tx=0x7f5054010da0 comp rx=0 tx=0).stop 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 shutdown_connections 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f504806b500 0x7f504806d9c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f506813ad40 0x7f506806f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f506813a190 0x7f506813a800 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 --2- 172.21.15.97:0/3886516082 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f50500039b0 0x7f506815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 >> 172.21.15.97:0/3886516082 conn(0x7f50680d0b90 msgr2=0x7f50680b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 shutdown_connections 2024-03-06T07:41:18.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:18.109+0000 7f506fe00640 1 -- 172.21.15.97:0/3886516082 wait complete. 2024-03-06T07:41:18.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:18.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:19.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:19.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.541+0000 7fce41600640 1 Processor -- start 2024-03-06T07:41:19.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.541+0000 7fce41600640 1 -- start start 2024-03-06T07:41:19.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.543+0000 7fce41600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:19.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.543+0000 7fce41600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fce3c0cc2a0 0x7fce3c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:19.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.543+0000 7fce41600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c0b2b30 con 0x7fce3c0cf370 2024-03-06T07:41:19.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.543+0000 7fce41600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3c0b28e0 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.543+0000 7fce41600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0b2e10 con 0x7fce3c15a940 2024-03-06T07:41:19.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce3be00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:19.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fce3c0cc2a0 0x7fce3c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:19.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fce3c0cc2a0 0x7fce3c0cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:52810/0 (socket says 172.21.15.97:52810) 2024-03-06T07:41:19.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 -- 172.21.15.97:0/1563443643 learned_addr learned my addr 172.21.15.97:0/1563443643 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:19.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 -- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 msgr2=0x7fce3c0b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 --2- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c0b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 -- 172.21.15.97:0/1563443643 >> v1:172.21.15.135:6789/0 conn(0x7fce3c0cf370 legacy=0x7fce3c06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:19.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.544+0000 7fce40c00640 1 -- 172.21.15.97:0/1563443643 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c06ca50 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.546+0000 7fce40c00640 1 --2- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c0cc750 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fce34009e50 tx=0x7fce3402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b13df7c1f1bbd54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:19.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.546+0000 7fce3b400640 1 -- 172.21.15.97:0/1563443643 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3402cb00 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce3b400640 1 -- 172.21.15.97:0/1563443643 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3402cca0 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce3b400640 1 -- 172.21.15.97:0/1563443643 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34034610 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 -- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cc2a0 msgr2=0x7fce3c0cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:19.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 --2- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c0cc750 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fce34009e50 tx=0x7fce3402bf60 comp rx=0 tx=0).stop 2024-03-06T07:41:19.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 -- 172.21.15.97:0/1563443643 shutdown_connections 2024-03-06T07:41:19.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 --2- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c0cc750 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 --2- 172.21.15.97:0/1563443643 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c0b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 -- 172.21.15.97:0/1563443643 >> 172.21.15.97:0/1563443643 conn(0x7fce3c06afc0 msgr2=0x7fce3c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:19.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 -- 172.21.15.97:0/1563443643 shutdown_connections 2024-03-06T07:41:19.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.547+0000 7fce41600640 1 -- 172.21.15.97:0/1563443643 wait complete. 2024-03-06T07:41:19.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.549+0000 7fce41600640 1 Processor -- start 2024-03-06T07:41:19.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.549+0000 7fce41600640 1 -- start start 2024-03-06T07:41:19.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce41600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:19.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce41600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cf370 0x7fce3c14fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:19.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce41600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:19.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce41600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0b4b20 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce41600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0b30f0 con 0x7fce3c0cf370 2024-03-06T07:41:19.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce41600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c15ad20 con 0x7fce3c15a940 2024-03-06T07:41:19.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:19.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c15b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55758/0 (socket says 172.21.15.97:55758) 2024-03-06T07:41:19.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce3be00640 1 -- 172.21.15.97:0/2296002711 learned_addr learned my addr 172.21.15.97:0/2296002711 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.551+0000 7fce33400640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cf370 0x7fce3c14fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce40c00640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce3be00640 1 -- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 msgr2=0x7fce3c1500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce3be00640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c1500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce3be00640 1 -- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cf370 msgr2=0x7fce3c14fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce3be00640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cf370 0x7fce3c14fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce3be00640 1 -- 172.21.15.97:0/2296002711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c158500 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce40c00640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c1500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce3be00640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c15b100 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fce2400e8a0 tx=0x7fce2400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24017070 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2400c940 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2400cd30 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.552+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c1587e0 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.553+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c158da0 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.554+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c04f330 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.555+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce2401a070 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.555+0000 7fce38c00640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:19.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.555+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce2408a090 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.556+0000 7fce33400640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:19.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.556+0000 7fce33400640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fce1806b4d0 0x7fce1806d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fce3c1f7660 tx=0x7fce2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:19.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.560+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce24058ef0 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.805+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce3c06e180 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.805+0000 7fce38c00640 1 -- 172.21.15.97:0/2296002711 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fce240587f0 con 0x7fce3c0cc2a0 2024-03-06T07:41:19.808 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:19.808 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:19.808 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fce1806b4d0 msgr2=0x7fce1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fce1806b4d0 0x7fce1806d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fce3c1f7660 tx=0x7fce2c009040 comp rx=0 tx=0).stop 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 msgr2=0x7fce3c15b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c15b100 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fce2400e8a0 tx=0x7fce2400ed60 comp rx=0 tx=0).stop 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 shutdown_connections 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fce3c15a940 0x7fce3c1500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce3c0cf370 0x7fce3c14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 --2- 172.21.15.97:0/2296002711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce3c0cc2a0 0x7fce3c15b100 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 >> 172.21.15.97:0/2296002711 conn(0x7fce3c06afc0 msgr2=0x7fce3c15e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 shutdown_connections 2024-03-06T07:41:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:19.808+0000 7fce41600640 1 -- 172.21.15.97:0/2296002711 wait complete. 2024-03-06T07:41:20.030 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:20.030 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:21.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:21.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.242+0000 7f98e1000640 1 Processor -- start 2024-03-06T07:41:21.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.242+0000 7f98e1000640 1 -- start start 2024-03-06T07:41:21.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98e1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98dc15e230 0x7f98dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98e1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98e1000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0cf340 con 0x7f98dc0b4320 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98e1000640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98dc15e610 con 0x7f98dc15e230 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98e1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0cfe60 con 0x7f98dc0ce550 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98d9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98dc15e230 0x7f98dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98d9e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f98dc15e230 0x7f98dc06edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55768/0 (socket says 172.21.15.97:55768) 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.244+0000 7f98d9e00640 1 -- 172.21.15.97:0/788147665 learned_addr learned my addr 172.21.15.97:0/788147665 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:21.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.245+0000 7f98da800640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:21.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.245+0000 7f98d9e00640 1 -- 172.21.15.97:0/788147665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 msgr2=0x7f98dc0ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:21.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.245+0000 7f98d9e00640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.245+0000 7f98d9e00640 1 -- 172.21.15.97:0/788147665 >> v1:172.21.15.135:6789/0 conn(0x7f98dc0b4320 legacy=0x7f98dc0d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:21.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.245+0000 7f98d9e00640 1 -- 172.21.15.97:0/788147665 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc06f390 con 0x7f98dc15e230 2024-03-06T07:41:21.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.245+0000 7f98da800640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:21.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.246+0000 7f98d9e00640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc06edb0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009870 tx=0x7f98cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1101ed9ce97051 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:21.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.246+0000 7f98d9400640 1 -- 172.21.15.97:0/788147665 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0317c0 con 0x7f98dc15e230 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.246+0000 7f98d9400640 1 -- 172.21.15.97:0/788147665 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc037070 con 0x7f98dc15e230 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.246+0000 7f98d9400640 1 -- 172.21.15.97:0/788147665 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc031b80 con 0x7f98dc15e230 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 -- 172.21.15.97:0/788147665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 msgr2=0x7f98dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc06edb0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009870 tx=0x7f98cc02b860 comp rx=0 tx=0).stop 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 -- 172.21.15.97:0/788147665 shutdown_connections 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 --2- 172.21.15.97:0/788147665 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc06edb0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 -- 172.21.15.97:0/788147665 >> 172.21.15.97:0/788147665 conn(0x7f98dc0b3800 msgr2=0x7f98dc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:21.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 -- 172.21.15.97:0/788147665 shutdown_connections 2024-03-06T07:41:21.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.247+0000 7f98e1000640 1 -- 172.21.15.97:0/788147665 wait complete. 2024-03-06T07:41:21.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.249+0000 7f98e1000640 1 Processor -- start 2024-03-06T07:41:21.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.249+0000 7f98e1000640 1 -- start start 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98e1000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98dc0b4320 0x7f98dc0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98e1000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98e1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc0b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98e1000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0d06c0 con 0x7f98dc15e230 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98e1000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0d03e0 con 0x7f98dc0b4320 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98e1000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0cc320 con 0x7f98dc0ce550 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc0b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc0b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55770/0 (socket says 172.21.15.97:55770) 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98da800640 1 -- 172.21.15.97:0/1275568331 learned_addr learned my addr 172.21.15.97:0/1275568331 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98d9e00640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98dc0b4320 0x7f98dc0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:21.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.250+0000 7f98d1400640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98da800640 1 -- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 msgr2=0x7f98dc0b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98da800640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98da800640 1 -- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98dc0b4320 msgr2=0x7f98dc0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98da800640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98dc0b4320 0x7f98dc0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98da800640 1 -- 172.21.15.97:0/1275568331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0cb6c0 con 0x7f98dc15e230 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98da800640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc0b4e30 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f98d400cef0 tx=0x7f98d400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d401a070 con 0x7f98dc15e230 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d400d860 con 0x7f98dc15e230 2024-03-06T07:41:21.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.251+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc0c9650 con 0x7f98dc15e230 2024-03-06T07:41:21.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.252+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d400dc50 con 0x7f98dc15e230 2024-03-06T07:41:21.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.252+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc0c9c10 con 0x7f98dc15e230 2024-03-06T07:41:21.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.253+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc04f330 con 0x7f98dc15e230 2024-03-06T07:41:21.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.254+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98d401d070 con 0x7f98dc15e230 2024-03-06T07:41:21.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.255+0000 7f98d2a00640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:21.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.255+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f98d408a9f0 con 0x7f98dc15e230 2024-03-06T07:41:21.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.255+0000 7f98d9e00640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:21.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.256+0000 7f98d9e00640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f98cc036000 tx=0x7f98cc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:21.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.260+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98d40592c0 con 0x7f98dc15e230 2024-03-06T07:41:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.505+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98dc0d1740 con 0x7f98dc15e230 2024-03-06T07:41:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.506+0000 7f98d2a00640 1 -- 172.21.15.97:0/1275568331 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f98d405d360 con 0x7f98dc15e230 2024-03-06T07:41:21.509 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:21.509 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.509+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 msgr2=0x7f98b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.509+0000 7f98e1000640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f98cc036000 tx=0x7f98cc03d040 comp rx=0 tx=0).stop 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.509+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 msgr2=0x7f98dc0b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.509+0000 7f98e1000640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc0b4e30 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f98d400cef0 tx=0x7f98d400a710 comp rx=0 tx=0).stop 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 shutdown_connections 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f98dc15e230 0x7f98dc0b4e30 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f98dc0ce550 0x7f98dc0b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 --2- 172.21.15.97:0/1275568331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98dc0b4320 0x7f98dc0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 >> 172.21.15.97:0/1275568331 conn(0x7f98dc0b3800 msgr2=0x7f98dc15d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 shutdown_connections 2024-03-06T07:41:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:21.510+0000 7f98e1000640 1 -- 172.21.15.97:0/1275568331 wait complete. 2024-03-06T07:41:21.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:21.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:22.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:22.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.950+0000 7ff4ce600640 1 Processor -- start 2024-03-06T07:41:22.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.950+0000 7ff4ce600640 1 -- start start 2024-03-06T07:41:22.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4ce600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c8144c00 0x7ff4c806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:22.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4ce600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4c815f9d0 0x7ff4c815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4ce600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c806ee70 con 0x7ff4c80ce0a0 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4ce600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c8160370 con 0x7ff4c815f9d0 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4ce600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c806cef0 con 0x7ff4c8144c00 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4c6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4c815f9d0 0x7ff4c815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4c6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4c815f9d0 0x7ff4c815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55786/0 (socket says 172.21.15.97:55786) 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4c6a00640 1 -- 172.21.15.97:0/3693127495 learned_addr learned my addr 172.21.15.97:0/3693127495 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.952+0000 7ff4c7e00640 1 --1- 172.21.15.97:0/3693127495 >> v1:172.21.15.97:6789/0 conn(0x7ff4c80ce0a0 0x7ff4c80ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39630/0 (socket says 172.21.15.97:39630) 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.953+0000 7ff4c7400640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c8144c00 0x7ff4c806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:22.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.953+0000 7ff4c6a00640 1 -- 172.21.15.97:0/3693127495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c8144c00 msgr2=0x7ff4c806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:22.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.953+0000 7ff4c6a00640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c8144c00 0x7ff4c806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:22.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.953+0000 7ff4c6a00640 1 -- 172.21.15.97:0/3693127495 >> v1:172.21.15.97:6789/0 conn(0x7ff4c80ce0a0 legacy=0x7ff4c80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:22.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.953+0000 7ff4c6a00640 1 -- 172.21.15.97:0/3693127495 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c806d1d0 con 0x7ff4c815f9d0 2024-03-06T07:41:22.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.953+0000 7ff4c7400640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c8144c00 0x7ff4c806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:22.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.954+0000 7ff4c6a00640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c815f9d0 0x7ff4c815fe30 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009870 tx=0x7ff4b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8973db1404d7af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:22.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.954+0000 7ff4c6000640 1 -- 172.21.15.97:0/3693127495 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4034070 con 0x7ff4c815f9d0 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.954+0000 7ff4c6000640 1 -- 172.21.15.97:0/3693127495 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4030840 con 0x7ff4c815f9d0 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.954+0000 7ff4c6000640 1 -- 172.21.15.97:0/3693127495 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4030c10 con 0x7ff4c815f9d0 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 -- 172.21.15.97:0/3693127495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c815f9d0 msgr2=0x7ff4c815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c815f9d0 0x7ff4c815fe30 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009870 tx=0x7ff4b402b980 comp rx=0 tx=0).stop 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 -- 172.21.15.97:0/3693127495 shutdown_connections 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c815f9d0 0x7ff4c815fe30 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 --2- 172.21.15.97:0/3693127495 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c8144c00 0x7ff4c806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 -- 172.21.15.97:0/3693127495 >> 172.21.15.97:0/3693127495 conn(0x7ff4c80b1730 msgr2=0x7ff4c80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 -- 172.21.15.97:0/3693127495 shutdown_connections 2024-03-06T07:41:22.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.955+0000 7ff4ce600640 1 -- 172.21.15.97:0/3693127495 wait complete. 2024-03-06T07:41:22.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.957+0000 7ff4ce600640 1 Processor -- start 2024-03-06T07:41:22.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.957+0000 7ff4ce600640 1 -- start start 2024-03-06T07:41:22.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4ce600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c80ce0a0 0x7ff4c815e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:22.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4ce600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 0x7ff4c815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:22.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4ce600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c815f9d0 0x7ff4c813d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4ce600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8160730 con 0x7ff4c8144c00 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4ce600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c806d790 con 0x7ff4c80ce0a0 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4ce600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c806cb50 con 0x7ff4c815f9d0 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4c6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 0x7ff4c815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4c6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 0x7ff4c815ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55788/0 (socket says 172.21.15.97:55788) 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4c6a00640 1 -- 172.21.15.97:0/1502265955 learned_addr learned my addr 172.21.15.97:0/1502265955 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4c7400640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c80ce0a0 0x7ff4c815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.958+0000 7ff4c7e00640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c815f9d0 0x7ff4c813d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c6a00640 1 -- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c815f9d0 msgr2=0x7ff4c813d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c6a00640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c815f9d0 0x7ff4c813d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c6a00640 1 -- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c80ce0a0 msgr2=0x7ff4c815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c6a00640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c80ce0a0 0x7ff4c815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c6a00640 1 -- 172.21.15.97:0/1502265955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c81e6ae0 con 0x7ff4c8144c00 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c7e00640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c815f9d0 0x7ff4c813d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c7400640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c80ce0a0 0x7ff4c815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4c6a00640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 0x7ff4c815ab20 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40090b0 tx=0x7ff4b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4034040 con 0x7ff4c8144c00 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4039d10 con 0x7ff4c8144c00 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4042630 con 0x7ff4c8144c00 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.959+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c815b060 con 0x7ff4c8144c00 2024-03-06T07:41:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.960+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c813cfd0 con 0x7ff4c8144c00 2024-03-06T07:41:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.961+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c804f330 con 0x7ff4c8144c00 2024-03-06T07:41:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.962+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4b4037070 con 0x7ff4c8144c00 2024-03-06T07:41:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.963+0000 7ff4bf400640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.963+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff4b40ac650 con 0x7ff4c8144c00 2024-03-06T07:41:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.963+0000 7ff4c7400640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.964+0000 7ff4c7400640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8006fd0 tx=0x7ff4b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:22.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:22.967+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4b407b4a0 con 0x7ff4c8144c00 2024-03-06T07:41:23.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.204+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4c8072890 con 0x7ff4c8144c00 2024-03-06T07:41:23.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.205+0000 7ff4bf400640 1 -- 172.21.15.97:0/1502265955 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7ff4b407ada0 con 0x7ff4c8144c00 2024-03-06T07:41:23.208 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:23.208 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:23.208 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:23.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 msgr2=0x7ff4a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:23.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8006fd0 tx=0x7ff4b8008040 comp rx=0 tx=0).stop 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 msgr2=0x7ff4c815ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 0x7ff4c815ab20 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40090b0 tx=0x7ff4b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 shutdown_connections 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff4c815f9d0 0x7ff4c813d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4c8144c00 0x7ff4c815ab20 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 --2- 172.21.15.97:0/1502265955 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c80ce0a0 0x7ff4c815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 >> 172.21.15.97:0/1502265955 conn(0x7ff4c80b1730 msgr2=0x7ff4c80b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 shutdown_connections 2024-03-06T07:41:23.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:23.208+0000 7ff4ce600640 1 -- 172.21.15.97:0/1502265955 wait complete. 2024-03-06T07:41:23.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:23.429 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:24.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:24.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.641+0000 7f5727400640 1 Processor -- start 2024-03-06T07:41:24.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.641+0000 7f5727400640 1 -- start start 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.642+0000 7f5727400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5720144c00 0x7f572006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.642+0000 7f5727400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f572015f9d0 0x7f572015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.642+0000 7f5727400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572006ee70 con 0x7f57200ce0a0 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.642+0000 7f5727400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572006cef0 con 0x7f572015f9d0 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.642+0000 7f5727400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572006d1d0 con 0x7f5720144c00 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f572015f9d0 0x7f572015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f572015f9d0 0x7f572015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33692/0 (socket says 172.21.15.97:33692) 2024-03-06T07:41:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 -- 172.21.15.97:0/694146571 learned_addr learned my addr 172.21.15.97:0/694146571 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f5724a00640 1 --2- 172.21.15.97:0/694146571 >> v2:172.21.15.135:3300/0 conn(0x7f5720144c00 0x7f572006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 -- 172.21.15.97:0/694146571 >> v2:172.21.15.135:3300/0 conn(0x7f5720144c00 msgr2=0x7f572006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 --2- 172.21.15.97:0/694146571 >> v2:172.21.15.135:3300/0 conn(0x7f5720144c00 0x7f572006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 -- 172.21.15.97:0/694146571 >> v1:172.21.15.135:6789/0 conn(0x7f57200ce0a0 legacy=0x7f57200ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.643+0000 7f571fe00640 1 -- 172.21.15.97:0/694146571 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572006d4b0 con 0x7f572015f9d0 2024-03-06T07:41:24.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.644+0000 7f571fe00640 1 --2- 172.21.15.97:0/694146571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f572015f9d0 0x7f572015fe60 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f57200b25f0 tx=0x7f570802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102fa1bf8d8a121c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:24.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.645+0000 7f571f400640 1 -- 172.21.15.97:0/694146571 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708035020 con 0x7f572015f9d0 2024-03-06T07:41:24.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.645+0000 7f571f400640 1 -- 172.21.15.97:0/694146571 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708031870 con 0x7f572015f9d0 2024-03-06T07:41:24.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.645+0000 7f571f400640 1 -- 172.21.15.97:0/694146571 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708031c40 con 0x7f572015f9d0 2024-03-06T07:41:24.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 -- 172.21.15.97:0/694146571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f572015f9d0 msgr2=0x7f572015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:24.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 --2- 172.21.15.97:0/694146571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f572015f9d0 0x7f572015fe60 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f57200b25f0 tx=0x7f570802b910 comp rx=0 tx=0).stop 2024-03-06T07:41:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 -- 172.21.15.97:0/694146571 shutdown_connections 2024-03-06T07:41:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 --2- 172.21.15.97:0/694146571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f572015f9d0 0x7f572015fe60 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 --2- 172.21.15.97:0/694146571 >> v2:172.21.15.135:3300/0 conn(0x7f5720144c00 0x7f572006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 -- 172.21.15.97:0/694146571 >> 172.21.15.97:0/694146571 conn(0x7f57200b1730 msgr2=0x7f57200b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 -- 172.21.15.97:0/694146571 shutdown_connections 2024-03-06T07:41:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.646+0000 7f5727400640 1 -- 172.21.15.97:0/694146571 wait complete. 2024-03-06T07:41:24.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.649+0000 7f5727400640 1 Processor -- start 2024-03-06T07:41:24.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.649+0000 7f5727400640 1 -- start start 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.650+0000 7f5727400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 0x7f5720138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.650+0000 7f5727400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5720144c00 0x7f572013c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.650+0000 7f5727400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572015f9d0 0x7f5720141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.650+0000 7f5727400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572006cb50 con 0x7f57200ce0a0 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.650+0000 7f5727400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5720160730 con 0x7f572015f9d0 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.650+0000 7f5727400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572006ccd0 con 0x7f5720144c00 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 0x7f5720138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 0x7f5720138d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33698/0 (socket says 172.21.15.97:33698) 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 -- 172.21.15.97:0/1879003723 learned_addr learned my addr 172.21.15.97:0/1879003723 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5725400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572015f9d0 0x7f5720141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 -- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5720144c00 msgr2=0x7f572013c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5720144c00 0x7f572013c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 -- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572015f9d0 msgr2=0x7f5720141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572015f9d0 0x7f5720141970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 -- 172.21.15.97:0/1879003723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57201f41d0 con 0x7f57200ce0a0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.651+0000 7f5724a00640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 0x7f5720138d60 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f571000d830 tx=0x7f571000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.652+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000b660 con 0x7f57200ce0a0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.652+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710019070 con 0x7f57200ce0a0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.652+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000ba20 con 0x7f57200ce0a0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.652+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572013a590 con 0x7f57200ce0a0 2024-03-06T07:41:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.652+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57200b2bd0 con 0x7f57200ce0a0 2024-03-06T07:41:24.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.653+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572004f330 con 0x7f57200ce0a0 2024-03-06T07:41:24.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.654+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f571001e460 con 0x7f57200ce0a0 2024-03-06T07:41:24.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.655+0000 7f571cc00640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f56fc06b4d0 0x7f56fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:24.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.655+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f571008a8a0 con 0x7f57200ce0a0 2024-03-06T07:41:24.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.655+0000 7f571fe00640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f56fc06b4d0 0x7f56fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:24.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.656+0000 7f571fe00640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f56fc06b4d0 0x7f56fc06d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5708036000 tx=0x7f5708030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:24.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.660+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5710059700 con 0x7f57200ce0a0 2024-03-06T07:41:24.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.941+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5720072890 con 0x7f57200ce0a0 2024-03-06T07:41:24.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.942+0000 7f571cc00640 1 -- 172.21.15.97:0/1879003723 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f5710059000 con 0x7f57200ce0a0 2024-03-06T07:41:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:24.944 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:24.944 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f56fc06b4d0 msgr2=0x7f56fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f56fc06b4d0 0x7f56fc06d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5708036000 tx=0x7f5708030000 comp rx=0 tx=0).stop 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 msgr2=0x7f5720138d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 0x7f5720138d60 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f571000d830 tx=0x7f571000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 shutdown_connections 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f56fc06b4d0 0x7f56fc06d990 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572015f9d0 0x7f5720141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5720144c00 0x7f572013c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 --2- 172.21.15.97:0/1879003723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57200ce0a0 0x7f5720138d60 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:24.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.945+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 >> 172.21.15.97:0/1879003723 conn(0x7f57200b1730 msgr2=0x7f5720145370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.946+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 shutdown_connections 2024-03-06T07:41:24.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:24.946+0000 7f5727400640 1 -- 172.21.15.97:0/1879003723 wait complete. 2024-03-06T07:41:25.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:25.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:26.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:26.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.402+0000 7f0bfd600640 1 Processor -- start 2024-03-06T07:41:26.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.402+0000 7f0bfd600640 1 -- start start 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bfd600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0bf80d07b0 0x7f0bf80cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bfd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf80d20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bfd600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf80cf140 con 0x7f0bf815f110 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bfd600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80cfd50 con 0x7f0bf80d07b0 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bfd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80cf420 con 0x7f0bf80d1c20 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bf6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0bf80d07b0 0x7f0bf80cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:26.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bf6a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0bf80d07b0 0x7f0bf80cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33710/0 (socket says 172.21.15.97:33710) 2024-03-06T07:41:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bf6000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0bf815f110 0x7f0bf80d0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50092/0 (socket says 172.21.15.97:50092) 2024-03-06T07:41:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.405+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1377002666 learned_addr learned my addr 172.21.15.97:0/1377002666 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf7400640 1 --2- 172.21.15.97:0/1377002666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf80d20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf5600640 1 -- 172.21.15.97:0/1377002666 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249828037 0 0) 0x7f0bf80cf140 con 0x7f0bf815f110 2024-03-06T07:41:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf5600640 1 -- 172.21.15.97:0/1377002666 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bdc003520 con 0x7f0bf815f110 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1377002666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 msgr2=0x7f0bf80d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf6a00640 1 --2- 172.21.15.97:0/1377002666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf80d20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1377002666 >> v1:172.21.15.97:6789/0 conn(0x7f0bf815f110 legacy=0x7f0bf80d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.406+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1377002666 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf80cf700 con 0x7f0bf80d07b0 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.407+0000 7f0bf6a00640 1 --2- 172.21.15.97:0/1377002666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80cea80 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009870 tx=0x7f0be802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab27b90ecd64b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.407+0000 7f0bf5600640 1 -- 172.21.15.97:0/1377002666 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8034070 con 0x7f0bf80d07b0 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.407+0000 7f0bf5600640 1 -- 172.21.15.97:0/1377002666 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be8030840 con 0x7f0bf80d07b0 2024-03-06T07:41:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.407+0000 7f0bf5600640 1 -- 172.21.15.97:0/1377002666 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8030bf0 con 0x7f0bf80d07b0 2024-03-06T07:41:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 -- 172.21.15.97:0/1377002666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 msgr2=0x7f0bf80cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1377002666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80cea80 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009870 tx=0x7f0be802b980 comp rx=0 tx=0).stop 2024-03-06T07:41:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 -- 172.21.15.97:0/1377002666 shutdown_connections 2024-03-06T07:41:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1377002666 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf80d20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1377002666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80cea80 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 -- 172.21.15.97:0/1377002666 >> 172.21.15.97:0/1377002666 conn(0x7f0bf80d3ee0 msgr2=0x7f0bf80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 -- 172.21.15.97:0/1377002666 shutdown_connections 2024-03-06T07:41:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.408+0000 7f0bfd600640 1 -- 172.21.15.97:0/1377002666 wait complete. 2024-03-06T07:41:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.410+0000 7f0bfd600640 1 Processor -- start 2024-03-06T07:41:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.410+0000 7f0bfd600640 1 -- start start 2024-03-06T07:41:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bfd600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bfd600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf81f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bfd600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bf815f110 0x7f0bf80c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:26.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bfd600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815e390 con 0x7f0bf80d07b0 2024-03-06T07:41:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bfd600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815f570 con 0x7f0bf815f110 2024-03-06T07:41:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bfd600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815e510 con 0x7f0bf80d1c20 2024-03-06T07:41:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bf6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bf6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33712/0 (socket says 172.21.15.97:33712) 2024-03-06T07:41:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.411+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1181752749 learned_addr learned my addr 172.21.15.97:0/1181752749 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6000640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf81f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf7400640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bf815f110 0x7f0bf80c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 msgr2=0x7f0bf81f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6a00640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf81f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bf815f110 msgr2=0x7f0bf80c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6a00640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bf815f110 0x7f0bf80c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6a00640 1 -- 172.21.15.97:0/1181752749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf81f4a20 con 0x7f0bf80d07b0 2024-03-06T07:41:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf7400640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bf815f110 0x7f0bf80c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0bf6a00640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80ca090 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f0be8000c00 tx=0x7f0be8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.412+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8048070 con 0x7f0bf80d07b0 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.413+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be8030640 con 0x7f0bf80d07b0 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.413+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8034040 con 0x7f0bf80d07b0 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.413+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf81f4cc0 con 0x7f0bf80d07b0 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.413+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf81f5280 con 0x7f0bf80d07b0 2024-03-06T07:41:26.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.414+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf804f330 con 0x7f0bf80d07b0 2024-03-06T07:41:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.415+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0be804b070 con 0x7f0bf80d07b0 2024-03-06T07:41:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.415+0000 7f0beea00640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bd806b4d0 0x7f0bd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.416+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0be80ac260 con 0x7f0bf80d07b0 2024-03-06T07:41:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.416+0000 7f0bf6000640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bd806b4d0 0x7f0bd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.417+0000 7f0bf6000640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bd806b4d0 0x7f0bd806d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80b84d0 tx=0x7f0be4007b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:26.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.420+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0be807b140 con 0x7f0bf80d07b0 2024-03-06T07:41:26.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.655+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf806cc50 con 0x7f0bf80d07b0 2024-03-06T07:41:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.656+0000 7f0beea00640 1 -- 172.21.15.97:0/1181752749 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f0be807aa40 con 0x7f0bf80d07b0 2024-03-06T07:41:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:26.659 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:26.659 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bd806b4d0 msgr2=0x7f0bd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bd806b4d0 0x7f0bd806d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80b84d0 tx=0x7f0be4007b10 comp rx=0 tx=0).stop 2024-03-06T07:41:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 msgr2=0x7f0bf80ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80ca090 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f0be8000c00 tx=0x7f0be8004730 comp rx=0 tx=0).stop 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 shutdown_connections 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0bd806b4d0 0x7f0bd806d990 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bf815f110 0x7f0bf80c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0bf80d1c20 0x7f0bf81f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 --2- 172.21.15.97:0/1181752749 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bf80d07b0 0x7f0bf80ca090 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 >> 172.21.15.97:0/1181752749 conn(0x7f0bf80d3ee0 msgr2=0x7f0bf815d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 shutdown_connections 2024-03-06T07:41:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:26.659+0000 7f0bfd600640 1 -- 172.21.15.97:0/1181752749 wait complete. 2024-03-06T07:41:26.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:26.881 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:27.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:28.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.092+0000 7f0f02e00640 1 Processor -- start 2024-03-06T07:41:28.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.092+0000 7f0f02e00640 1 -- start start 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f02e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0efc161580 0x7f0efc0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f02e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0efc0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f02e00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc161960 con 0x7f0efc0d1e90 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f02e00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc0cf6f0 con 0x7f0efc161580 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f02e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0cf9d0 con 0x7f0efc0d2a70 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f01000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0efc0d1e90 0x7f0efc0d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50098/0 (socket says 172.21.15.97:50098) 2024-03-06T07:41:28.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f01000640 1 -- 172.21.15.97:0/285922295 learned_addr learned my addr 172.21.15.97:0/285922295 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:28.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f01a00640 1 --2- 172.21.15.97:0/285922295 >> v2:172.21.15.135:3300/0 conn(0x7f0efc161580 0x7f0efc0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.094+0000 7f0f02400640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0efc0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864961437 0 0) 0x7f0efc161960 con 0x7f0efc0d1e90 2024-03-06T07:41:28.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee4003520 con 0x7f0efc0d1e90 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354540517 0 0) 0x7f0ee4003520 con 0x7f0efc0d1e90 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 msgr2=0x7f0efc0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0efc0b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 >> v2:172.21.15.135:3300/0 conn(0x7f0efc161580 msgr2=0x7f0efc0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 --2- 172.21.15.97:0/285922295 >> v2:172.21.15.135:3300/0 conn(0x7f0efc161580 0x7f0efc0d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc0cfcb0 con 0x7f0efc0d1e90 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f0ee8001690 con 0x7f0efc0d1e90 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 >> v1:172.21.15.97:6789/0 conn(0x7f0efc0d1e90 legacy=0x7f0efc0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ee40039b0 0x7f0ee4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0ee4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0ee40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0cf6f0 con 0x7f0ee4004800 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0cf9d0 con 0x7f0ee40039b0 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.095+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee400b9c0 con 0x7f0efc0d2a70 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01000640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0ee4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01a00640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ee40039b0 0x7f0ee4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f02400640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0ee40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01000640 1 -- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 msgr2=0x7f0ee40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01000640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0ee40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01000640 1 -- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ee40039b0 msgr2=0x7f0ee4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01000640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ee40039b0 0x7f0ee4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f01000640 1 -- 172.21.15.97:0/285922295 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4003520 con 0x7f0ee4004800 2024-03-06T07:41:28.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.096+0000 7f0f02400640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0d2a70 0x7f0ee40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0f01000640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0ee4004c80 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800c210 tx=0x7f0ee4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c77509148fcd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8007d60 con 0x7f0ee4004800 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee8030070 con 0x7f0ee4004800 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0efbe00640 1 -- 172.21.15.97:0/285922295 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee802cd20 con 0x7f0ee4004800 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0f02e00640 1 -- 172.21.15.97:0/285922295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 msgr2=0x7f0ee4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0f02e00640 1 --2- 172.21.15.97:0/285922295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0ee4004c80 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800c210 tx=0x7f0ee4005830 comp rx=0 tx=0).stop 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.098+0000 7f0f02400640 1 -- 172.21.15.97:0/285922295 reap_dead start 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.099+0000 7f0f02e00640 1 -- 172.21.15.97:0/285922295 shutdown_connections 2024-03-06T07:41:28.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.099+0000 7f0f02e00640 1 -- 172.21.15.97:0/285922295 >> 172.21.15.97:0/285922295 conn(0x7f0efc0d3d80 msgr2=0x7f0efc0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:28.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.099+0000 7f0f02e00640 1 -- 172.21.15.97:0/285922295 shutdown_connections 2024-03-06T07:41:28.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.099+0000 7f0f02e00640 1 -- 172.21.15.97:0/285922295 wait complete. 2024-03-06T07:41:28.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.100+0000 7f0f02e00640 1 Processor -- start 2024-03-06T07:41:28.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 -- start start 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0efc0cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0efc0c7410 0x7f0efc0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0ca8c0 0x7f0efc14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0cf3a0 con 0x7f0ee4004800 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0b39a0 con 0x7f0efc0c7410 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.101+0000 7f0f02e00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0b3ec0 con 0x7f0efc0ca8c0 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0efc0cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0efc0cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33732/0 (socket says 172.21.15.97:33732) 2024-03-06T07:41:28.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 -- 172.21.15.97:0/2845432361 learned_addr learned my addr 172.21.15.97:0/2845432361 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 -- 172.21.15.97:0/2845432361 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0ca8c0 msgr2=0x7f0efc14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0ca8c0 0x7f0efc14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 -- 172.21.15.97:0/2845432361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0efc0c7410 msgr2=0x7f0efc0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0efc0c7410 0x7f0efc0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 -- 172.21.15.97:0/2845432361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc0c92e0 con 0x7f0ee4004800 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.102+0000 7f0f01a00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0efc0cce60 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0011660 tx=0x7f0ef0011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.103+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000f8e0 con 0x7f0ee4004800 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.103+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef000fa80 con 0x7f0ee4004800 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.103+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000fe30 con 0x7f0ee4004800 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.103+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0efc0c95c0 con 0x7f0ee4004800 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.103+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc0c9b80 con 0x7f0ee4004800 2024-03-06T07:41:28.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.104+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc04f330 con 0x7f0ee4004800 2024-03-06T07:41:28.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.106+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ef0015460 con 0x7f0ee4004800 2024-03-06T07:41:28.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.107+0000 7f0ef9600640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0edc06b4d0 0x7f0edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:28.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.107+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0ef008ab40 con 0x7f0ee4004800 2024-03-06T07:41:28.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.108+0000 7f0f01000640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0edc06b4d0 0x7f0edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:28.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.108+0000 7f0f01000640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0edc06b4d0 0x7f0edc06d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8034000 tx=0x7f0ee802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:28.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.112+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ef0019030 con 0x7f0ee4004800 2024-03-06T07:41:28.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.345+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0efc0d3b60 con 0x7f0ee4004800 2024-03-06T07:41:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.346+0000 7f0ef9600640 1 -- 172.21.15.97:0/2845432361 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f0ef0059900 con 0x7f0ee4004800 2024-03-06T07:41:28.348 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:28.349 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0edc06b4d0 msgr2=0x7f0edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0edc06b4d0 0x7f0edc06d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8034000 tx=0x7f0ee802e000 comp rx=0 tx=0).stop 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 msgr2=0x7f0efc0cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0efc0cce60 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0011660 tx=0x7f0ef0011b20 comp rx=0 tx=0).stop 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 shutdown_connections 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f0edc06b4d0 0x7f0edc06d990 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0efc0ca8c0 0x7f0efc14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0efc0c7410 0x7f0efc0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 --2- 172.21.15.97:0/2845432361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0ee4004800 0x7f0efc0cce60 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.349+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 >> 172.21.15.97:0/2845432361 conn(0x7f0efc0d3d80 msgr2=0x7f0efc15f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:28.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.350+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 shutdown_connections 2024-03-06T07:41:28.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:28.350+0000 7f0f02e00640 1 -- 172.21.15.97:0/2845432361 wait complete. 2024-03-06T07:41:28.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:28.574 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:29.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:29.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.782+0000 7fbfa3a00640 1 Processor -- start 2024-03-06T07:41:29.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.782+0000 7fbfa3a00640 1 -- start start 2024-03-06T07:41:29.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbfa3a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf9c06ec00 0x7fbf9c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:29.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbfa3a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c15b090 con 0x7fbf9c0b4680 2024-03-06T07:41:29.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbfa3a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0b2a30 con 0x7fbf9c15acb0 2024-03-06T07:41:29.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbfa3a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b4360 con 0x7fbf9c06ec00 2024-03-06T07:41:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbfa1a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fbf9c06ec00 0x7fbf9c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbf9be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fbf9c0b4680 0x7fbf9c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50106/0 (socket says 172.21.15.97:50106) 2024-03-06T07:41:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.784+0000 7fbf9be00640 1 -- 172.21.15.97:0/1869996132 learned_addr learned my addr 172.21.15.97:0/1869996132 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.785+0000 7fbfa1a00640 1 -- 172.21.15.97:0/1869996132 >> v1:172.21.15.135:6789/0 conn(0x7fbf9c15acb0 legacy=0x7fbf9c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.785+0000 7fbfa1a00640 1 -- 172.21.15.97:0/1869996132 >> v1:172.21.15.97:6789/0 conn(0x7fbf9c0b4680 legacy=0x7fbf9c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.785+0000 7fbfa1a00640 1 -- 172.21.15.97:0/1869996132 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0b2cf0 con 0x7fbf9c06ec00 2024-03-06T07:41:29.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.787+0000 7fbfa1a00640 1 --2- 172.21.15.97:0/1869996132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c06ec00 0x7fbf9c0ccfb0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fbf940097a0 tx=0x7fbf9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f47313278e4d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:29.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.787+0000 7fbf9b400640 1 -- 172.21.15.97:0/1869996132 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94035020 con 0x7fbf9c06ec00 2024-03-06T07:41:29.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.787+0000 7fbf9b400640 1 -- 172.21.15.97:0/1869996132 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94031830 con 0x7fbf9c06ec00 2024-03-06T07:41:29.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.787+0000 7fbf9b400640 1 -- 172.21.15.97:0/1869996132 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94031c00 con 0x7fbf9c06ec00 2024-03-06T07:41:29.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.787+0000 7fbfa3a00640 1 -- 172.21.15.97:0/1869996132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c06ec00 msgr2=0x7fbf9c0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:29.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.788+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/1869996132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c06ec00 0x7fbf9c0ccfb0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fbf940097a0 tx=0x7fbf9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:41:29.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.788+0000 7fbfa3a00640 1 -- 172.21.15.97:0/1869996132 shutdown_connections 2024-03-06T07:41:29.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.788+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/1869996132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c06ec00 0x7fbf9c0ccfb0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:29.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.788+0000 7fbfa3a00640 1 -- 172.21.15.97:0/1869996132 >> 172.21.15.97:0/1869996132 conn(0x7fbf9c0b1730 msgr2=0x7fbf9c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:29.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.788+0000 7fbfa3a00640 1 -- 172.21.15.97:0/1869996132 shutdown_connections 2024-03-06T07:41:29.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.788+0000 7fbfa3a00640 1 -- 172.21.15.97:0/1869996132 wait complete. 2024-03-06T07:41:29.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.790+0000 7fbfa3a00640 1 Processor -- start 2024-03-06T07:41:29.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.790+0000 7fbfa3a00640 1 -- start start 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa3a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf9c06ec00 0x7fbf9c1484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa3a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf9c0b4680 0x7fbf9c148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa3a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 0x7fbf9c15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa3a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b4010 con 0x7fbf9c15acb0 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa3a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b32b0 con 0x7fbf9c0b4680 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa3a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c072340 con 0x7fbf9c06ec00 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa1a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 0x7fbf9c15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa1a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 0x7fbf9c15bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33750/0 (socket says 172.21.15.97:33750) 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.791+0000 7fbfa1a00640 1 -- 172.21.15.97:0/573963682 learned_addr learned my addr 172.21.15.97:0/573963682 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbf9be00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf9c0b4680 0x7fbf9c148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1000640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf9c06ec00 0x7fbf9c1484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1a00640 1 -- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf9c06ec00 msgr2=0x7fbf9c1484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf9c06ec00 0x7fbf9c1484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:29.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1a00640 1 -- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf9c0b4680 msgr2=0x7fbf9c148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf9c0b4680 0x7fbf9c148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1a00640 1 -- 172.21.15.97:0/573963682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c159c70 con 0x7fbf9c15acb0 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbf9be00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf9c0b4680 0x7fbf9c148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1000640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf9c06ec00 0x7fbf9c1484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa1a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 0x7fbf9c15bce0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fbf9402bee0 tx=0x7fbf940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94041070 con 0x7fbf9c15acb0 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9403a8d0 con 0x7fbf9c15acb0 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c1596b0 con 0x7fbf9c15acb0 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.792+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94035040 con 0x7fbf9c15acb0 2024-03-06T07:41:29.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.793+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c15c220 con 0x7fbf9c15acb0 2024-03-06T07:41:29.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.794+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c04f330 con 0x7fbf9c15acb0 2024-03-06T07:41:29.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.795+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf94038020 con 0x7fbf9c15acb0 2024-03-06T07:41:29.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.795+0000 7fbf98c00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7c06b4d0 0x7fbf7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:29.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.796+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbf940abc50 con 0x7fbf9c15acb0 2024-03-06T07:41:29.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.796+0000 7fbfa1000640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7c06b4d0 0x7fbf7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:29.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.797+0000 7fbfa1000640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7c06b4d0 0x7fbf7c06d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c007e90 tx=0x7fbf8c0073a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:29.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:29.800+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf9407aaa0 con 0x7fbf9c15acb0 2024-03-06T07:41:30.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.048+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf9c06ce60 con 0x7fbf9c15acb0 2024-03-06T07:41:30.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.049+0000 7fbf98c00640 1 -- 172.21.15.97:0/573963682 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7fbf9407a3a0 con 0x7fbf9c15acb0 2024-03-06T07:41:30.051 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:30.052 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:30.052 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7c06b4d0 msgr2=0x7fbf7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7c06b4d0 0x7fbf7c06d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c007e90 tx=0x7fbf8c0073a0 comp rx=0 tx=0).stop 2024-03-06T07:41:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 msgr2=0x7fbf9c15bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 0x7fbf9c15bce0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fbf9402bee0 tx=0x7fbf940046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 shutdown_connections 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fbf7c06b4d0 0x7fbf7c06d990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf9c15acb0 0x7fbf9c15bce0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf9c0b4680 0x7fbf9c148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 --2- 172.21.15.97:0/573963682 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fbf9c06ec00 0x7fbf9c1484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 >> 172.21.15.97:0/573963682 conn(0x7fbf9c0b1730 msgr2=0x7fbf9c0af7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 shutdown_connections 2024-03-06T07:41:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:30.052+0000 7fbfa3a00640 1 -- 172.21.15.97:0/573963682 wait complete. 2024-03-06T07:41:30.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:37:21.810170+0000', 'modified': '2024-03-06T07:39:50.920314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9024, 'mds_1': 6463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6463': {'gid': 6463, 'name': 'a', 'rank': 1, 'incarnation': 109, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.97:6837/1171184135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 1171184135}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 1171184135}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:38:36.706910+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9024': {'gid': 9024, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.97:6835/4190861064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 4190861064}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 4190861064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9024, 'qdb_cluster': [9024]} 2024-03-06T07:41:30.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6463:up:stopping 2024-03-06T07:41:31.275 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:37:21.810170+0000", "damaged": [], "data_pools": [ 21 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6463": { "addr": "172.21.15.97:6837/1171184135", "addrs": { "addrvec": [ { "addr": "172.21.15.97:6836", "nonce": 1171184135, "type": "v2" }, { "addr": "172.21.15.97:6837", "nonce": 1171184135, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6463, "incarnation": 109, "join_fscid": -1, "laggy_since": "2024-03-06T07:38:36.706910+0000", "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_9024": { "addr": "172.21.15.97:6835/4190861064", "addrs": { "addrvec": [ { "addr": "172.21.15.97:6834", "nonce": 4190861064, "type": "v2" }, { "addr": "172.21.15.97:6835", "nonce": 4190861064, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9024, "incarnation": 103, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 20, "modified": "2024-03-06T07:39:50.920314+0000", "qdb_cluster": [ 9024 ], "qdb_leader": 9024, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9024, "mds_1": 6463 } } } ], "standbys": [] } 2024-03-06T07:41:31.276 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:41:31.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:41:31.406 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:41:31.528 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:41:31.528 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:41:31.529 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:31.581 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:41:31.581 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:41:31.581 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:31.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:41:31.638 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:41:31.638 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:31.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:41:31.692 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:41:31.693 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:31.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:41:31.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:31.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.952+0000 7fa000400640 1 Processor -- start 2024-03-06T07:41:31.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.953+0000 7fa000400640 1 -- start start 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7fa000400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7fa000400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ff80b4680 0x7f9ff806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7fa000400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ff8072340 0x7f9ff806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7fa000400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff815b090 con 0x7f9ff8072340 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7fa000400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff80b2a30 con 0x7f9ff80b4680 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7fa000400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80b4360 con 0x7f9ff815acb0 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7f9ffe400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ff8072340 0x7f9ff806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7f9ffe400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ff8072340 0x7f9ff806ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33754/0 (socket says 172.21.15.97:33754) 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.954+0000 7f9ffe400640 1 -- 172.21.15.97:0/3398917349 learned_addr learned my addr 172.21.15.97:0/3398917349 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:31.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffda00640 1 --2- 172.21.15.97:0/3398917349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffd000640 1 --2- 172.21.15.97:0/3398917349 >> v2:172.21.15.135:3300/0 conn(0x7f9ff80b4680 0x7f9ff806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffe400640 1 -- 172.21.15.97:0/3398917349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 msgr2=0x7f9ff80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:31.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffe400640 1 --2- 172.21.15.97:0/3398917349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffe400640 1 -- 172.21.15.97:0/3398917349 >> v2:172.21.15.135:3300/0 conn(0x7f9ff80b4680 msgr2=0x7f9ff806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:31.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffe400640 1 --2- 172.21.15.97:0/3398917349 >> v2:172.21.15.135:3300/0 conn(0x7f9ff80b4680 0x7f9ff806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.955+0000 7f9ffe400640 1 -- 172.21.15.97:0/3398917349 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff80b2cf0 con 0x7f9ff8072340 2024-03-06T07:41:31.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.956+0000 7f9ffe400640 1 --2- 172.21.15.97:0/3398917349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff8072340 0x7f9ff806ea30 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806f020 tx=0x7f9ff002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05745d621f973ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:31.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.956+0000 7f9ff7e00640 1 -- 172.21.15.97:0/3398917349 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff002ca70 con 0x7f9ff8072340 2024-03-06T07:41:31.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.956+0000 7f9ff7e00640 1 -- 172.21.15.97:0/3398917349 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff002cc10 con 0x7f9ff8072340 2024-03-06T07:41:31.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.956+0000 7f9ff7e00640 1 -- 172.21.15.97:0/3398917349 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00343f0 con 0x7f9ff8072340 2024-03-06T07:41:31.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 -- 172.21.15.97:0/3398917349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff8072340 msgr2=0x7f9ff806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:31.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 --2- 172.21.15.97:0/3398917349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff8072340 0x7f9ff806ea30 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806f020 tx=0x7f9ff002bed0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 -- 172.21.15.97:0/3398917349 shutdown_connections 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 --2- 172.21.15.97:0/3398917349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff8072340 0x7f9ff806ea30 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 --2- 172.21.15.97:0/3398917349 >> v2:172.21.15.135:3300/0 conn(0x7f9ff80b4680 0x7f9ff806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 --2- 172.21.15.97:0/3398917349 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 -- 172.21.15.97:0/3398917349 >> 172.21.15.97:0/3398917349 conn(0x7f9ff80b1730 msgr2=0x7f9ff80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 -- 172.21.15.97:0/3398917349 shutdown_connections 2024-03-06T07:41:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.957+0000 7fa000400640 1 -- 172.21.15.97:0/3398917349 wait complete. 2024-03-06T07:41:31.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.959+0000 7fa000400640 1 Processor -- start 2024-03-06T07:41:31.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.959+0000 7fa000400640 1 -- start start 2024-03-06T07:41:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7fa000400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ff8072340 0x7f9ff81492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7fa000400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 0x7f9ff80ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7fa000400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7fa000400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80b3d10 con 0x7f9ff80b4680 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7fa000400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80b32b0 con 0x7f9ff8072340 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7fa000400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80b4010 con 0x7f9ff815acb0 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7f9ffd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 0x7f9ff80ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7f9ffd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 0x7f9ff80ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33764/0 (socket says 172.21.15.97:33764) 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7f9ffd000640 1 -- 172.21.15.97:0/3303193491 learned_addr learned my addr 172.21.15.97:0/3303193491 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7f9ffda00640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ff8072340 0x7f9ff81492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.960+0000 7f9ffe400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffd000640 1 -- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 msgr2=0x7f9ff80cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffd000640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffd000640 1 -- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ff8072340 msgr2=0x7f9ff81492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffd000640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ff8072340 0x7f9ff81492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffd000640 1 -- 172.21.15.97:0/3303193491 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff8150e80 con 0x7f9ff80b4680 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffda00640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ff8072340 0x7f9ff81492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ffd000640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 0x7f9ff80ae270 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400d800 tx=0x7f9fe400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4018030 con 0x7f9ff80b4680 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe4009640 con 0x7f9ff80b4680 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4009810 con 0x7f9ff80b4680 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff80aecc0 con 0x7f9ff80b4680 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.961+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff81504f0 con 0x7f9ff80b4680 2024-03-06T07:41:31.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.962+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff804f330 con 0x7f9ff80b4680 2024-03-06T07:41:31.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.963+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fe401b070 con 0x7f9ff80b4680 2024-03-06T07:41:31.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.964+0000 7f9ff5600640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fd806b4d0 0x7f9fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:31.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.964+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9fe408a4d0 con 0x7f9ff80b4680 2024-03-06T07:41:31.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.965+0000 7f9ffda00640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fd806b4d0 0x7f9fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:31.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.965+0000 7f9ffda00640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fd806b4d0 0x7f9fd806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9fec006fd0 tx=0x7f9fec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:31.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:31.969+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fe4059290 con 0x7f9ff80b4680 2024-03-06T07:41:32.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.201+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ff806ce60 con 0x7f9ff80b4680 2024-03-06T07:41:32.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.202+0000 7f9ff5600640 1 -- 172.21.15.97:0/3303193491 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f9fe4058b90 con 0x7f9ff80b4680 2024-03-06T07:41:32.205 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:32.205 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:32.205 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fd806b4d0 msgr2=0x7f9fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fd806b4d0 0x7f9fd806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9fec006fd0 tx=0x7f9fec008040 comp rx=0 tx=0).stop 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 msgr2=0x7f9ff80ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 0x7f9ff80ae270 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400d800 tx=0x7f9fe400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 shutdown_connections 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9fd806b4d0 0x7f9fd806d990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9ff815acb0 0x7f9ff80cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ff80b4680 0x7f9ff80ae270 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 --2- 172.21.15.97:0/3303193491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ff8072340 0x7f9ff81492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 >> 172.21.15.97:0/3303193491 conn(0x7f9ff80b1730 msgr2=0x7f9ff806d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:32.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 shutdown_connections 2024-03-06T07:41:32.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.205+0000 7fa000400640 1 -- 172.21.15.97:0/3303193491 wait complete. 2024-03-06T07:41:32.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:41:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.420+0000 7fb87b200640 1 Processor -- start 2024-03-06T07:41:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.421+0000 7fb87b200640 1 -- start start 2024-03-06T07:41:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.422+0000 7fb87b200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb87c161580 0x7fb87c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.422+0000 7fb87b200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c161960 con 0x7fb87c0d2a70 2024-03-06T07:41:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.422+0000 7fb87b200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0cf6f0 con 0x7fb87c0d1e90 2024-03-06T07:41:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.422+0000 7fb87b200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c0cf9d0 con 0x7fb87c161580 2024-03-06T07:41:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.422+0000 7fb87a800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb87c0d2a70 0x7fb87c0b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50116/0 (socket says 172.21.15.97:50116) 2024-03-06T07:41:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.422+0000 7fb87a800640 1 -- 172.21.15.97:0/2011532655 learned_addr learned my addr 172.21.15.97:0/2011532655 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879e00640 1 --2- 172.21.15.97:0/2011532655 >> v2:172.21.15.135:3300/0 conn(0x7fb87c161580 0x7fb87c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292715724 0 0) 0x7fb87c161960 con 0x7fb87c0d2a70 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860003520 con 0x7fb87c0d2a70 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1747027096 0 0) 0x7fb860003520 con 0x7fb87c0d2a70 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 >> v2:172.21.15.135:3300/0 conn(0x7fb87c161580 msgr2=0x7fb87c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 --2- 172.21.15.97:0/2011532655 >> v2:172.21.15.135:3300/0 conn(0x7fb87c161580 0x7fb87c0d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 >> v1:172.21.15.135:6789/0 conn(0x7fb87c0d1e90 legacy=0x7fb87c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0cfcb0 con 0x7fb87c0d2a70 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fb8740019f0 con 0x7fb87c0d2a70 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 >> v1:172.21.15.97:6789/0 conn(0x7fb87c0d2a70 legacy=0x7fb87c0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8600039b0 0x7fb860003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0cf9d0 con 0x7fb8600051c0 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86000c060 con 0x7fb860004800 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.423+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86000c260 con 0x7fb8600039b0 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb87a800640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb873e00640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb87a800640 1 -- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8600039b0 msgr2=0x7fb860003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb87a800640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb8600039b0 0x7fb860003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb87a800640 1 -- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb860004800 msgr2=0x7fb860004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb87a800640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb87a800640 1 -- 172.21.15.97:0/2011532655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860003520 con 0x7fb8600051c0 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.424+0000 7fb873e00640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.425+0000 7fb87a800640 1 -- 172.21.15.97:0/2011532655 reap_dead start 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.425+0000 7fb87a800640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fb87400a830 tx=0x7fb860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e429aa39462a1ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.425+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87400a480 con 0x7fb8600051c0 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874034630 con 0x7fb8600051c0 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb879400640 1 -- 172.21.15.97:0/2011532655 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874034a00 con 0x7fb8600051c0 2024-03-06T07:41:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 -- 172.21.15.97:0/2011532655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 msgr2=0x7fb86000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fb87400a830 tx=0x7fb860005830 comp rx=0 tx=0).stop 2024-03-06T07:41:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 -- 172.21.15.97:0/2011532655 shutdown_connections 2024-03-06T07:41:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 --2- 172.21.15.97:0/2011532655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 -- 172.21.15.97:0/2011532655 >> 172.21.15.97:0/2011532655 conn(0x7fb87c0d3d80 msgr2=0x7fb87c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 -- 172.21.15.97:0/2011532655 shutdown_connections 2024-03-06T07:41:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.426+0000 7fb87b200640 1 -- 172.21.15.97:0/2011532655 wait complete. 2024-03-06T07:41:32.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.429+0000 7fb87b200640 1 Processor -- start 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.429+0000 7fb87b200640 1 -- start start 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb87b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb87c0b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb87b200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb87c14d8d0 0x7fb87c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb87b200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb87c14dd30 0x7fb87c0c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb87b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0cf3a0 con 0x7fb8600051c0 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb87b200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0b39a0 con 0x7fb87c14dd30 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb87b200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0d1950 con 0x7fb87c14d8d0 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb879e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb87c0b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb879e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb87c0b8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33780/0 (socket says 172.21.15.97:33780) 2024-03-06T07:41:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.431+0000 7fb879e00640 1 -- 172.21.15.97:0/517637421 learned_addr learned my addr 172.21.15.97:0/517637421 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb873e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb87c14d8d0 0x7fb87c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb879e00640 1 -- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb87c14d8d0 msgr2=0x7fb87c0c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb87a800640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb87c14dd30 0x7fb87c0c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb879e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb87c14d8d0 0x7fb87c0c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb879e00640 1 -- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb87c14dd30 msgr2=0x7fb87c0c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb879e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb87c14dd30 0x7fb87c0c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb879e00640 1 -- 172.21.15.97:0/517637421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0b7c10 con 0x7fb8600051c0 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb873e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb87c14d8d0 0x7fb87c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb87a800640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb87c14dd30 0x7fb87c0c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb879e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb87c0b8cb0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00e7a0 tx=0x7fb86c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c017070 con 0x7fb8600051c0 2024-03-06T07:41:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c00c940 con 0x7fb8600051c0 2024-03-06T07:41:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c00cd10 con 0x7fb8600051c0 2024-03-06T07:41:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c0b7ef0 con 0x7fb8600051c0 2024-03-06T07:41:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.432+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0b84b0 con 0x7fb8600051c0 2024-03-06T07:41:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.434+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c04f330 con 0x7fb8600051c0 2024-03-06T07:41:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.434+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb86c01a070 con 0x7fb8600051c0 2024-03-06T07:41:32.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.436+0000 7fb872000640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.436+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb86c08a540 con 0x7fb8600051c0 2024-03-06T07:41:32.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.436+0000 7fb873e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.437+0000 7fb873e00640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb85806b4d0 0x7fb85806d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0cb7b0 tx=0x7fb864009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:32.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.440+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb86c059300 con 0x7fb8600051c0 2024-03-06T07:41:32.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.753+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb87c0d3b60 con 0x7fb8600051c0 2024-03-06T07:41:32.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.755+0000 7fb872000640 1 -- 172.21.15.97:0/517637421 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16717 (secure 0 0 0) 0x7fb86c058c00 con 0x7fb8600051c0 2024-03-06T07:41:32.758 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:32.758 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":148,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:40.116487+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.39","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:41:32.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb85806b4d0 msgr2=0x7fb85806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb85806b4d0 0x7fb85806d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0cb7b0 tx=0x7fb864009040 comp rx=0 tx=0).stop 2024-03-06T07:41:32.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 msgr2=0x7fb87c0b8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb87c0b8cb0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00e7a0 tx=0x7fb86c00ec60 comp rx=0 tx=0).stop 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 shutdown_connections 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb87c14dd30 0x7fb87c0c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb87c14d8d0 0x7fb87c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 --2- 172.21.15.97:0/517637421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb8600051c0 0x7fb87c0b8cb0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.757+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 >> 172.21.15.97:0/517637421 conn(0x7fb87c0d3d80 msgr2=0x7fb87c0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.758+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 shutdown_connections 2024-03-06T07:41:32.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.758+0000 7fb87b200640 1 -- 172.21.15.97:0/517637421 wait complete. 2024-03-06T07:41:32.778 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:34.971154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:37:19.579712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:37:20.835626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:41:32.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:41:32.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.982+0000 7f5671200640 1 Processor -- start 2024-03-06T07:41:32.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.982+0000 7f5671200640 1 -- start start 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f5671200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f566c0cc560 0x7f566c06b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f5671200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f566c0b29a0 0x7f566c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f5671200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c0b0f30 con 0x7f566c072670 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f5671200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c0b1cf0 con 0x7f566c0b29a0 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f5671200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c0b1210 con 0x7f566c0cc560 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f566a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f566c0b29a0 0x7f566c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f566a800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f566c0b29a0 0x7f566c0ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33782/0 (socket says 172.21.15.97:33782) 2024-03-06T07:41:32.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f566a800640 1 -- 172.21.15.97:0/249899537 learned_addr learned my addr 172.21.15.97:0/249899537 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:32.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.984+0000 7f5669e00640 1 --2- 172.21.15.97:0/249899537 >> v2:172.21.15.135:3300/0 conn(0x7f566c0cc560 0x7f566c06b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.985+0000 7f566a800640 1 -- 172.21.15.97:0/249899537 >> v2:172.21.15.135:3300/0 conn(0x7f566c0cc560 msgr2=0x7f566c06b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:32.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.985+0000 7f566a800640 1 --2- 172.21.15.97:0/249899537 >> v2:172.21.15.135:3300/0 conn(0x7f566c0cc560 0x7f566c06b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.985+0000 7f566a800640 1 -- 172.21.15.97:0/249899537 >> v1:172.21.15.135:6789/0 conn(0x7f566c072670 legacy=0x7f566c06ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:32.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.985+0000 7f566a800640 1 -- 172.21.15.97:0/249899537 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c0b14f0 con 0x7f566c0b29a0 2024-03-06T07:41:32.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.986+0000 7f566a800640 1 --2- 172.21.15.97:0/249899537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c0b29a0 0x7f566c0ccbd0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f566c072580 tx=0x7f566402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d22c1b5352c3e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:32.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.986+0000 7f5669400640 1 -- 172.21.15.97:0/249899537 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566402ea90 con 0x7f566c0b29a0 2024-03-06T07:41:32.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.986+0000 7f5669400640 1 -- 172.21.15.97:0/249899537 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566402ec30 con 0x7f566c0b29a0 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.986+0000 7f5669400640 1 -- 172.21.15.97:0/249899537 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664009600 con 0x7f566c0b29a0 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 -- 172.21.15.97:0/249899537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c0b29a0 msgr2=0x7f566c0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 --2- 172.21.15.97:0/249899537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c0b29a0 0x7f566c0ccbd0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f566c072580 tx=0x7f566402e0f0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 -- 172.21.15.97:0/249899537 shutdown_connections 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 --2- 172.21.15.97:0/249899537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c0b29a0 0x7f566c0ccbd0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 --2- 172.21.15.97:0/249899537 >> v2:172.21.15.135:3300/0 conn(0x7f566c0cc560 0x7f566c06b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 -- 172.21.15.97:0/249899537 >> 172.21.15.97:0/249899537 conn(0x7f566c0b1e80 msgr2=0x7f566c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 -- 172.21.15.97:0/249899537 shutdown_connections 2024-03-06T07:41:32.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.987+0000 7f5671200640 1 -- 172.21.15.97:0/249899537 wait complete. 2024-03-06T07:41:32.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.988+0000 7f5671200640 1 Processor -- start 2024-03-06T07:41:32.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.989+0000 7f5671200640 1 -- start start 2024-03-06T07:41:32.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5671200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 0x7f566c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5671200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f566c0b29a0 0x7f566c0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5671200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f566c0cc560 0x7f566c0c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5671200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0b0b50 con 0x7f566c072670 2024-03-06T07:41:32.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5671200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c06c190 con 0x7f566c0cc560 2024-03-06T07:41:32.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5671200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0b0cd0 con 0x7f566c0b29a0 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.990+0000 7f5669e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 0x7f566c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 0x7f566c15a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33786/0 (socket says 172.21.15.97:33786) 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 -- 172.21.15.97:0/892643037 learned_addr learned my addr 172.21.15.97:0/892643037 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f566a800640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f566c0cc560 0x7f566c0c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5663e00640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f566c0b29a0 0x7f566c0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 -- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f566c0b29a0 msgr2=0x7f566c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f566c0b29a0 0x7f566c0ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 -- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f566c0cc560 msgr2=0x7f566c0c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f566c0cc560 0x7f566c0c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 -- 172.21.15.97:0/892643037 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c1f40d0 con 0x7f566c072670 2024-03-06T07:41:32.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f566a800640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f566c0cc560 0x7f566c0c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.991+0000 7f5669e00640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 0x7f566c15a030 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f565c00d5f0 tx=0x7f565c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.992+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c0125f0 con 0x7f566c072670 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.992+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c019070 con 0x7f566c072670 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.992+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c0129d0 con 0x7f566c072670 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.992+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c1f43b0 con 0x7f566c072670 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.992+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c099ac0 con 0x7f566c072670 2024-03-06T07:41:32.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.993+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c1f4580 con 0x7f566c072670 2024-03-06T07:41:32.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.994+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f565c012b70 con 0x7f566c072670 2024-03-06T07:41:32.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.995+0000 7f5662000640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564806b4d0 0x7f564806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:32.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.995+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f565c08a6b0 con 0x7f566c072670 2024-03-06T07:41:32.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.995+0000 7f5663e00640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564806b4d0 0x7f564806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:32.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.996+0000 7f5663e00640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564806b4d0 0x7f564806d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f566c0ca520 tx=0x7f5654009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:32.999+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f565c059470 con 0x7f566c072670 2024-03-06T07:41:33.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.196+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f566c04f330 con 0x7f566c072670 2024-03-06T07:41:33.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.197+0000 7f5662000640 1 -- 172.21.15.97:0/892643037 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7f565c058d70 con 0x7f566c072670 2024-03-06T07:41:33.199 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564806b4d0 msgr2=0x7f564806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564806b4d0 0x7f564806d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f566c0ca520 tx=0x7f5654009040 comp rx=0 tx=0).stop 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 msgr2=0x7f566c15a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 0x7f566c15a030 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f565c00d5f0 tx=0x7f565c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 shutdown_connections 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f564806b4d0 0x7f564806d990 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f566c0cc560 0x7f566c0c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f566c0b29a0 0x7f566c0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 --2- 172.21.15.97:0/892643037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f566c072670 0x7f566c15a030 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 >> 172.21.15.97:0/892643037 conn(0x7f566c0b1e80 msgr2=0x7f566c0b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 shutdown_connections 2024-03-06T07:41:33.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.199+0000 7f5671200640 1 -- 172.21.15.97:0/892643037 wait complete. 2024-03-06T07:41:33.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:41:33.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.488+0000 7f58aa400640 1 Processor -- start 2024-03-06T07:41:33.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.488+0000 7f58aa400640 1 -- start start 2024-03-06T07:41:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58aa400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f58a406da40 0x7f58a40ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58aa400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58aa400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a415c1f0 con 0x7f58a415be10 2024-03-06T07:41:33.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58aa400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a40cd200 con 0x7f58a406da40 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58aa400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a406cac0 con 0x7f58a40aef20 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f589be00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f58a406da40 0x7f58a40ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58a9000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f58a415be10 0x7f58a406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50132/0 (socket says 172.21.15.97:50132) 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.490+0000 7f58a9000640 1 -- 172.21.15.97:0/2045822858 learned_addr learned my addr 172.21.15.97:0/2045822858 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.491+0000 7f58a9a00640 1 --2- 172.21.15.97:0/2045822858 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.491+0000 7f589be00640 1 -- 172.21.15.97:0/2045822858 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 msgr2=0x7f58a40ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.491+0000 7f589be00640 1 --2- 172.21.15.97:0/2045822858 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.491+0000 7f589be00640 1 -- 172.21.15.97:0/2045822858 >> v1:172.21.15.97:6789/0 conn(0x7f58a415be10 legacy=0x7f58a406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.491+0000 7f589be00640 1 -- 172.21.15.97:0/2045822858 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a406cda0 con 0x7f58a406da40 2024-03-06T07:41:33.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.491+0000 7f58a3e00640 1 -- 172.21.15.97:0/2045822858 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063679593 0 0) 0x7f58a415c1f0 con 0x7f58a415be10 2024-03-06T07:41:33.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.492+0000 7f589be00640 1 --2- 172.21.15.97:0/2045822858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a406da40 0x7f58a40ccc20 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f588c0097a0 tx=0x7f588c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75facf8af2b5e071 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.492+0000 7f58a3e00640 1 -- 172.21.15.97:0/2045822858 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c035020 con 0x7f58a406da40 2024-03-06T07:41:33.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.492+0000 7f58a3e00640 1 -- 172.21.15.97:0/2045822858 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588c031830 con 0x7f58a406da40 2024-03-06T07:41:33.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.492+0000 7f58a3e00640 1 -- 172.21.15.97:0/2045822858 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c031c00 con 0x7f58a406da40 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 -- 172.21.15.97:0/2045822858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a406da40 msgr2=0x7f58a40ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 --2- 172.21.15.97:0/2045822858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a406da40 0x7f58a40ccc20 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f588c0097a0 tx=0x7f588c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 -- 172.21.15.97:0/2045822858 shutdown_connections 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 --2- 172.21.15.97:0/2045822858 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 --2- 172.21.15.97:0/2045822858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a406da40 0x7f58a40ccc20 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 -- 172.21.15.97:0/2045822858 >> 172.21.15.97:0/2045822858 conn(0x7f58a406afc0 msgr2=0x7f58a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 -- 172.21.15.97:0/2045822858 shutdown_connections 2024-03-06T07:41:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.493+0000 7f58aa400640 1 -- 172.21.15.97:0/2045822858 wait complete. 2024-03-06T07:41:33.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.495+0000 7f58aa400640 1 Processor -- start 2024-03-06T07:41:33.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.495+0000 7f58aa400640 1 -- start start 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.496+0000 7f58aa400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58a406da40 0x7f58a41f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.496+0000 7f58aa400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.496+0000 7f58aa400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 0x7f58a40addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.496+0000 7f58aa400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40cda90 con 0x7f58a415be10 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.496+0000 7f58aa400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a406d080 con 0x7f58a406da40 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.496+0000 7f58aa400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4071ab0 con 0x7f58a40aef20 2024-03-06T07:41:33.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 0x7f58a40addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 0x7f58a40addd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33798/0 (socket says 172.21.15.97:33798) 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 -- 172.21.15.97:0/3234620938 learned_addr learned my addr 172.21.15.97:0/3234620938 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f589be00640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9000640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58a406da40 0x7f58a41f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 -- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 msgr2=0x7f58a40ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 -- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58a406da40 msgr2=0x7f58a41f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58a406da40 0x7f58a41f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9a00640 1 -- 172.21.15.97:0/3234620938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4144d80 con 0x7f58a415be10 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.497+0000 7f58a9000640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58a406da40 0x7f58a41f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.498+0000 7f58a9a00640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 0x7f58a40addd0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f589c00d800 tx=0x7f589c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.498+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00b6c0 con 0x7f58a415be10 2024-03-06T07:41:33.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.498+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c019070 con 0x7f58a415be10 2024-03-06T07:41:33.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.498+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a4145060 con 0x7f58a415be10 2024-03-06T07:41:33.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.498+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00bac0 con 0x7f58a415be10 2024-03-06T07:41:33.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.498+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a4145620 con 0x7f58a415be10 2024-03-06T07:41:33.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.499+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a404f330 con 0x7f58a415be10 2024-03-06T07:41:33.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.500+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f589c00bc60 con 0x7f58a415be10 2024-03-06T07:41:33.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.501+0000 7f58a1600640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f588006b4d0 0x7f588006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.501+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f589c08a6b0 con 0x7f58a415be10 2024-03-06T07:41:33.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.501+0000 7f58a9000640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f588006b4d0 0x7f588006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.502+0000 7f58a9000640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f588006b4d0 0x7f588006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5894007fd0 tx=0x7f5894009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.509+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f589c059510 con 0x7f58a415be10 2024-03-06T07:41:33.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.708+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f58a40ccd10 con 0x7f58a415be10 2024-03-06T07:41:33.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.709+0000 7f58a1600640 1 -- 172.21.15.97:0/3234620938 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+11 (secure 0 0 0) 0x7f589c058e10 con 0x7f58a415be10 2024-03-06T07:41:33.712 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-06T07:41:33.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f588006b4d0 msgr2=0x7f588006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f588006b4d0 0x7f588006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5894007fd0 tx=0x7f5894009040 comp rx=0 tx=0).stop 2024-03-06T07:41:33.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 msgr2=0x7f58a40addd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 0x7f58a40addd0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f589c00d800 tx=0x7f589c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 shutdown_connections 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f588006b4d0 0x7f588006d990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58a415be10 0x7f58a40addd0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f58a40aef20 0x7f58a40ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 --2- 172.21.15.97:0/3234620938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f58a406da40 0x7f58a41f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 >> 172.21.15.97:0/3234620938 conn(0x7f58a406afc0 msgr2=0x7f58a4159400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 shutdown_connections 2024-03-06T07:41:33.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.712+0000 7f58aa400640 1 -- 172.21.15.97:0/3234620938 wait complete. 2024-03-06T07:41:33.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:41:33.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.962+0000 7f44b7200640 1 Processor -- start 2024-03-06T07:41:33.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.962+0000 7f44b7200640 1 -- start start 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.963+0000 7f44b7200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44b80d17f0 0x7f44b80d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.963+0000 7f44b7200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b80cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.963+0000 7f44b7200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f44b80ceaf0 0x7f44b80cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.963+0000 7f44b7200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b8159c80 con 0x7f44b80ceaf0 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.963+0000 7f44b7200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b80cf460 con 0x7f44b80d17f0 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.963+0000 7f44b7200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80d0560 con 0x7f44b80cc650 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f44b80ceaf0 0x7f44b80cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f44b80ceaf0 0x7f44b80cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47170/0 (socket says 172.21.15.97:47170) 2024-03-06T07:41:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 -- 172.21.15.97:0/2102078269 learned_addr learned my addr 172.21.15.97:0/2102078269 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b5e00640 1 --2- 172.21.15.97:0/2102078269 >> v2:172.21.15.135:3300/0 conn(0x7f44b80d17f0 0x7f44b80d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b5400640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b80cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 -- 172.21.15.97:0/2102078269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 msgr2=0x7f44b80cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b80cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 -- 172.21.15.97:0/2102078269 >> v2:172.21.15.135:3300/0 conn(0x7f44b80d17f0 msgr2=0x7f44b80d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 --2- 172.21.15.97:0/2102078269 >> v2:172.21.15.135:3300/0 conn(0x7f44b80d17f0 0x7f44b80d1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b6800640 1 -- 172.21.15.97:0/2102078269 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80d0840 con 0x7f44b80ceaf0 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b5e00640 1 --2- 172.21.15.97:0/2102078269 >> v2:172.21.15.135:3300/0 conn(0x7f44b80d17f0 0x7f44b80d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:33.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.964+0000 7f44b5400640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b80cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.965+0000 7f44b6800640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80ceaf0 0x7f44b80cef20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f44b806e9e0 tx=0x7f44b002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f0166ea0e250a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.966+0000 7f44b4a00640 1 -- 172.21.15.97:0/2102078269 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b002c9e0 con 0x7f44b80ceaf0 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.966+0000 7f44b4a00640 1 -- 172.21.15.97:0/2102078269 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b002cb80 con 0x7f44b80ceaf0 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.966+0000 7f44b4a00640 1 -- 172.21.15.97:0/2102078269 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b00343f0 con 0x7f44b80ceaf0 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.966+0000 7f44b7200640 1 -- 172.21.15.97:0/2102078269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80ceaf0 msgr2=0x7f44b80cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.966+0000 7f44b7200640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80ceaf0 0x7f44b80cef20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f44b806e9e0 tx=0x7f44b002be40 comp rx=0 tx=0).stop 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.966+0000 7f44b7200640 1 -- 172.21.15.97:0/2102078269 shutdown_connections 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.967+0000 7f44b7200640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80ceaf0 0x7f44b80cef20 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.967+0000 7f44b7200640 1 --2- 172.21.15.97:0/2102078269 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b80cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.967+0000 7f44b7200640 1 --2- 172.21.15.97:0/2102078269 >> v2:172.21.15.135:3300/0 conn(0x7f44b80d17f0 0x7f44b80d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.967+0000 7f44b7200640 1 -- 172.21.15.97:0/2102078269 >> 172.21.15.97:0/2102078269 conn(0x7f44b809a9f0 msgr2=0x7f44b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.967+0000 7f44b7200640 1 -- 172.21.15.97:0/2102078269 shutdown_connections 2024-03-06T07:41:33.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.967+0000 7f44b7200640 1 -- 172.21.15.97:0/2102078269 wait complete. 2024-03-06T07:41:33.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.968+0000 7f44b7200640 1 Processor -- start 2024-03-06T07:41:33.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 -- start start 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44b80ceaf0 0x7f44b806d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 0x7f44b806d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80cf7f0 con 0x7f44b80d17f0 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80d2110 con 0x7f44b80ceaf0 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.969+0000 7f44b7200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80cf970 con 0x7f44b80cc650 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 0x7f44b806d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 0x7f44b806d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47184/0 (socket says 172.21.15.97:47184) 2024-03-06T07:41:33.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 -- 172.21.15.97:0/415088827 learned_addr learned my addr 172.21.15.97:0/415088827 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b5400640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44b80ceaf0 0x7f44b806d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b5e00640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 -- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 msgr2=0x7f44b814f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b814f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 -- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44b80ceaf0 msgr2=0x7f44b806d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44b80ceaf0 0x7f44b806d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b6800640 1 -- 172.21.15.97:0/415088827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b814f5f0 con 0x7f44b80d17f0 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.970+0000 7f44b5400640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44b80ceaf0 0x7f44b806d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.971+0000 7f44b6800640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 0x7f44b806d660 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009f40 tx=0x7f44b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.971+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0043020 con 0x7f44b80d17f0 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.971+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b003e450 con 0x7f44b80d17f0 2024-03-06T07:41:33.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.971+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b80c8db0 con 0x7f44b80d17f0 2024-03-06T07:41:33.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.971+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0009510 con 0x7f44b80d17f0 2024-03-06T07:41:33.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.971+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b80c9370 con 0x7f44b80d17f0 2024-03-06T07:41:33.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.973+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b804f330 con 0x7f44b80d17f0 2024-03-06T07:41:33.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.973+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44b0046020 con 0x7f44b80d17f0 2024-03-06T07:41:33.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.974+0000 7f44ae000640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f449406b4d0 0x7f449406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:33.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.974+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f44b00acba0 con 0x7f44b80d17f0 2024-03-06T07:41:33.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.974+0000 7f44b5e00640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f449406b4d0 0x7f449406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:33.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.975+0000 7f44b5e00640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f449406b4d0 0x7f449406d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f44a000b840 tx=0x7f44a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:33.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:33.979+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44b004d030 con 0x7f44b80d17f0 2024-03-06T07:41:34.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.274+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44b806ecd0 con 0x7f44b80d17f0 2024-03-06T07:41:34.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.275+0000 7f44ae000640 1 -- 172.21.15.97:0/415088827 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44b814f5f0 con 0x7f44b80d17f0 2024-03-06T07:41:34.277 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f449406b4d0 msgr2=0x7f449406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f449406b4d0 0x7f449406d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f44a000b840 tx=0x7f44a0008040 comp rx=0 tx=0).stop 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 msgr2=0x7f44b806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 0x7f44b806d660 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009f40 tx=0x7f44b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 shutdown_connections 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f449406b4d0 0x7f449406d990 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44b80d17f0 0x7f44b806d660 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44b80ceaf0 0x7f44b806d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 --2- 172.21.15.97:0/415088827 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f44b80cc650 0x7f44b814f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 >> 172.21.15.97:0/415088827 conn(0x7f44b809a9f0 msgr2=0x7f44b8160080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:34.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.277+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 shutdown_connections 2024-03-06T07:41:34.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.278+0000 7f44b7200640 1 -- 172.21.15.97:0/415088827 wait complete. 2024-03-06T07:41:34.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.484+0000 7f2c46800640 1 Processor -- start 2024-03-06T07:41:34.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.484+0000 7f2c46800640 1 -- start start 2024-03-06T07:41:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.486+0000 7f2c46800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c40168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.486+0000 7f2c46800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4006eb10 con 0x7f2c400cd860 2024-03-06T07:41:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.486+0000 7f2c46800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c400b5010 con 0x7f2c4015e070 2024-03-06T07:41:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.486+0000 7f2c46800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400b52f0 con 0x7f2c4006b6d0 2024-03-06T07:41:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.486+0000 7f2c37e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2c400cd860 0x7f2c4006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37062/0 (socket says 172.21.15.97:37062) 2024-03-06T07:41:34.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.486+0000 7f2c37e00640 1 -- 172.21.15.97:0/4223708692 learned_addr learned my addr 172.21.15.97:0/4223708692 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c40168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941738998 0 0) 0x7f2c4006eb10 con 0x7f2c400cd860 2024-03-06T07:41:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c24003520 con 0x7f2c400cd860 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965383513 0 0) 0x7f2c24003520 con 0x7f2c400cd860 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 msgr2=0x7f2c40168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c40168fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 >> v1:172.21.15.135:6789/0 conn(0x7f2c4015e070 legacy=0x7f2c40066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400b55d0 con 0x7f2c400cd860 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2c28001690 con 0x7f2c400cd860 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c40168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 >> v1:172.21.15.97:6789/0 conn(0x7f2c400cd860 legacy=0x7f2c4006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c240039e0 0x7f2c24003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400b52f0 con 0x7f2c24004d40 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2400bb20 con 0x7f2c240039e0 2024-03-06T07:41:34.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.487+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2400bd20 con 0x7f2c4006b6d0 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3f400640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c240039e0 0x7f2c24003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c37e00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3fe00640 1 -- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 msgr2=0x7f2c24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4006b6d0 0x7f2c24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3fe00640 1 -- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c240039e0 msgr2=0x7f2c24003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c240039e0 0x7f2c24003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.488+0000 7f2c3fe00640 1 -- 172.21.15.97:0/4223708692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c24003520 con 0x7f2c24004d40 2024-03-06T07:41:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.489+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c240051c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f2c380073b0 tx=0x7f2c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e422e11dcbe027 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.489+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38039070 con 0x7f2c24004d40 2024-03-06T07:41:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.489+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3802cd30 con 0x7f2c24004d40 2024-03-06T07:41:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.489+0000 7f2c3ea00640 1 -- 172.21.15.97:0/4223708692 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38009c80 con 0x7f2c24004d40 2024-03-06T07:41:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c46800640 1 -- 172.21.15.97:0/4223708692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 msgr2=0x7f2c240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c46800640 1 --2- 172.21.15.97:0/4223708692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c240051c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f2c380073b0 tx=0x7f2c24005830 comp rx=0 tx=0).stop 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c3fe00640 1 -- 172.21.15.97:0/4223708692 reap_dead start 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c46800640 1 -- 172.21.15.97:0/4223708692 shutdown_connections 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c46800640 1 -- 172.21.15.97:0/4223708692 >> 172.21.15.97:0/4223708692 conn(0x7f2c400b1e10 msgr2=0x7f2c400b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c46800640 1 -- 172.21.15.97:0/4223708692 shutdown_connections 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.490+0000 7f2c46800640 1 -- 172.21.15.97:0/4223708692 wait complete. 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.492+0000 7f2c46800640 1 Processor -- start 2024-03-06T07:41:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.492+0000 7f2c46800640 1 -- start start 2024-03-06T07:41:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c46800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c400d0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c46800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c400ce010 0x7f2c4015a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c46800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4015ae90 0x7f2c4015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c46800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4015e450 con 0x7f2c24004d40 2024-03-06T07:41:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c46800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400b5b90 con 0x7f2c400ce010 2024-03-06T07:41:34.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c46800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400b60b0 con 0x7f2c4015ae90 2024-03-06T07:41:34.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c3f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c400d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c3f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c400d0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47212/0 (socket says 172.21.15.97:47212) 2024-03-06T07:41:34.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.493+0000 7f2c3f400640 1 -- 172.21.15.97:0/4266915160 learned_addr learned my addr 172.21.15.97:0/4266915160 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3f400640 1 -- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4015ae90 msgr2=0x7f2c4015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3fe00640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4015ae90 0x7f2c4015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3f400640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4015ae90 0x7f2c4015b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c37e00640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c400ce010 0x7f2c4015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3f400640 1 -- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c400ce010 msgr2=0x7f2c4015a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3f400640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c400ce010 0x7f2c4015a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3f400640 1 -- 172.21.15.97:0/4266915160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c401f4fe0 con 0x7f2c24004d40 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c3f400640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c400d0560 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000e5b0 tx=0x7f2c3000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:34.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300052e0 con 0x7f2c24004d40 2024-03-06T07:41:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c30018070 con 0x7f2c24004d40 2024-03-06T07:41:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c400d0bb0 con 0x7f2c24004d40 2024-03-06T07:41:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30003810 con 0x7f2c24004d40 2024-03-06T07:41:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.494+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c400ce470 con 0x7f2c24004d40 2024-03-06T07:41:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.495+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4015d8d0 con 0x7f2c24004d40 2024-03-06T07:41:34.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.496+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c300039b0 con 0x7f2c24004d40 2024-03-06T07:41:34.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.497+0000 7f2c37400640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.497+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2c30089f40 con 0x7f2c24004d40 2024-03-06T07:41:34.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.497+0000 7f2c37e00640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.498+0000 7f2c37e00640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2c28001d10 tx=0x7f2c280014f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:34.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.506+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c4015d8d0 con 0x7f2c24004d40 2024-03-06T07:41:34.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.737+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4004f330 con 0x7f2c24004d40 2024-03-06T07:41:34.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.738+0000 7f2c37400640 1 -- 172.21.15.97:0/4266915160 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f2c3000ce60 con 0x7f2c24004d40 2024-03-06T07:41:34.741 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:34.741 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:34.741 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1c06b4d0 msgr2=0x7f2c1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2c28001d10 tx=0x7f2c280014f0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 msgr2=0x7f2c400d0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c400d0560 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000e5b0 tx=0x7f2c3000ea70 comp rx=0 tx=0).stop 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 shutdown_connections 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4015ae90 0x7f2c4015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c400ce010 0x7f2c4015a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 --2- 172.21.15.97:0/4266915160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c24004d40 0x7f2c400d0560 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 >> 172.21.15.97:0/4266915160 conn(0x7f2c400b1e10 msgr2=0x7f2c4015bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:34.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 shutdown_connections 2024-03-06T07:41:34.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.741+0000 7f2c46800640 1 -- 172.21.15.97:0/4266915160 wait complete. 2024-03-06T07:41:34.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:41:34.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.964+0000 7ff7abe00640 1 Processor -- start 2024-03-06T07:41:34.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.964+0000 7ff7abe00640 1 -- start start 2024-03-06T07:41:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.966+0000 7ff7abe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 0x7ff7a415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.966+0000 7ff7abe00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a40b3040 con 0x7ff7a40cf9a0 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.966+0000 7ff7abe00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a40cd3e0 con 0x7ff7a40b4390 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.966+0000 7ff7abe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a415fb50 con 0x7ff7a415f770 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.967+0000 7ff7a9e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff7a40cf9a0 0x7ff7a40cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37070/0 (socket says 172.21.15.97:37070) 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.967+0000 7ff7a9e00640 1 -- 172.21.15.97:0/2598282655 learned_addr learned my addr 172.21.15.97:0/2598282655 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.967+0000 7ff7a9400640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 0x7ff7a415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.967+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430931187 0 0) 0x7ff7a40b3040 con 0x7ff7a40cf9a0 2024-03-06T07:41:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.967+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78c003520 con 0x7ff7a40cf9a0 2024-03-06T07:41:34.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2588929214 0 0) 0x7ff78c003520 con 0x7ff7a40cf9a0 2024-03-06T07:41:34.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 msgr2=0x7ff7a415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:34.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 0x7ff7a415fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 >> v1:172.21.15.135:6789/0 conn(0x7ff7a40b4390 legacy=0x7ff7a4160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40cd6c0 con 0x7ff7a40cf9a0 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff79c0019f0 con 0x7ff7a40cf9a0 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 >> v1:172.21.15.97:6789/0 conn(0x7ff7a40cf9a0 legacy=0x7ff7a40cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 0x7ff78c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c003f60 0x7ff78c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff78c004d40 0x7ff78c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c00bb50 con 0x7ff78c004d40 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c00bcd0 con 0x7ff78c003f60 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c00bed0 con 0x7ff7a415f770 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff78c004d40 0x7ff78c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a8a00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c003f60 0x7ff78c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.968+0000 7ff7a9400640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 0x7ff78c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.969+0000 7ff7a9e00640 1 -- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 msgr2=0x7ff78c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.969+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff7a415f770 0x7ff78c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.969+0000 7ff7a9e00640 1 -- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c003f60 msgr2=0x7ff78c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.969+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c003f60 0x7ff78c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.969+0000 7ff7a9e00640 1 -- 172.21.15.97:0/2598282655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c003520 con 0x7ff78c004d40 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff78c004d40 0x7ff78c0051c0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00a850 tx=0x7ff78c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd424b1a219c1b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c00a4a0 con 0x7ff78c004d40 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c034650 con 0x7ff78c004d40 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7a3e00640 1 -- 172.21.15.97:0/2598282655 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c034a40 con 0x7ff78c004d40 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7abe00640 1 -- 172.21.15.97:0/2598282655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff78c004d40 msgr2=0x7ff78c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7abe00640 1 --2- 172.21.15.97:0/2598282655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff78c004d40 0x7ff78c0051c0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00a850 tx=0x7ff78c005830 comp rx=0 tx=0).stop 2024-03-06T07:41:34.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.970+0000 7ff7a9e00640 1 -- 172.21.15.97:0/2598282655 reap_dead start 2024-03-06T07:41:34.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.971+0000 7ff7abe00640 1 -- 172.21.15.97:0/2598282655 shutdown_connections 2024-03-06T07:41:34.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.971+0000 7ff7abe00640 1 -- 172.21.15.97:0/2598282655 >> 172.21.15.97:0/2598282655 conn(0x7ff7a40b3870 msgr2=0x7ff7a40b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:34.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.971+0000 7ff7abe00640 1 -- 172.21.15.97:0/2598282655 shutdown_connections 2024-03-06T07:41:34.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.971+0000 7ff7abe00640 1 -- 172.21.15.97:0/2598282655 wait complete. 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.972+0000 7ff7abe00640 1 Processor -- start 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.972+0000 7ff7abe00640 1 -- start start 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.973+0000 7ff7abe00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff78c004d40 0x7ff7a41f8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.973+0000 7ff7abe00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7a40b5fb0 0x7ff7a4159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.973+0000 7ff7abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 0x7ff7a40b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.973+0000 7ff7abe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40b34a0 con 0x7ff7a41597c0 2024-03-06T07:41:34.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.973+0000 7ff7abe00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40b3320 con 0x7ff7a40b5fb0 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.973+0000 7ff7abe00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a41609a0 con 0x7ff78c004d40 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 0x7ff7a40b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 0x7ff7a40b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47238/0 (socket says 172.21.15.97:47238) 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 -- 172.21.15.97:0/753684157 learned_addr learned my addr 172.21.15.97:0/753684157 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9400640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff78c004d40 0x7ff7a41f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a8a00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7a40b5fb0 0x7ff7a4159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 -- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff78c004d40 msgr2=0x7ff7a41f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff78c004d40 0x7ff7a41f8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 -- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7a40b5fb0 msgr2=0x7ff7a4159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:34.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7a40b5fb0 0x7ff7a4159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.974+0000 7ff7a9e00640 1 -- 172.21.15.97:0/753684157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40b7a80 con 0x7ff7a41597c0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.975+0000 7ff7a9e00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 0x7ff7a40b6bc0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00a970 tx=0x7ff79c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.975+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c034800 con 0x7ff7a41597c0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.975+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c044070 con 0x7ff7a41597c0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.975+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c034ba0 con 0x7ff7a41597c0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.975+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a41f7ac0 con 0x7ff7a41597c0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.975+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a41f7d60 con 0x7ff7a41597c0 2024-03-06T07:41:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.976+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a415f1b0 con 0x7ff7a41597c0 2024-03-06T07:41:34.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.977+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff79c049460 con 0x7ff7a41597c0 2024-03-06T07:41:34.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.979+0000 7ff7a1600640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:34.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.979+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff79c0ac550 con 0x7ff7a41597c0 2024-03-06T07:41:34.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.979+0000 7ff7a9400640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:34.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.980+0000 7ff7a9400640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff78406b4d0 0x7ff78406d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff7980099f0 tx=0x7ff798008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:34.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:34.983+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7a415f1b0 con 0x7ff7a41597c0 2024-03-06T07:41:35.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.180+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7a404f330 con 0x7ff7a41597c0 2024-03-06T07:41:35.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.181+0000 7ff7a1600640 1 -- 172.21.15.97:0/753684157 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16717 (secure 0 0 0) 0x7ff79c00a540 con 0x7ff7a41597c0 2024-03-06T07:41:35.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:35.184 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":148,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:40.116487+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.39","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:41:35.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff78406b4d0 msgr2=0x7ff78406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff78406b4d0 0x7ff78406d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff7980099f0 tx=0x7ff798008040 comp rx=0 tx=0).stop 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 msgr2=0x7ff7a40b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 0x7ff7a40b6bc0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00a970 tx=0x7ff79c001dc0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 shutdown_connections 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7a41597c0 0x7ff7a40b6bc0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7a40b5fb0 0x7ff7a4159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 --2- 172.21.15.97:0/753684157 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff78c004d40 0x7ff7a41f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 >> 172.21.15.97:0/753684157 conn(0x7ff7a40b3870 msgr2=0x7ff7a40d1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 shutdown_connections 2024-03-06T07:41:35.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.184+0000 7ff7abe00640 1 -- 172.21.15.97:0/753684157 wait complete. 2024-03-06T07:41:35.204 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:41:35.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:35.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.405+0000 7f0389600640 1 Processor -- start 2024-03-06T07:41:35.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.405+0000 7f0389600640 1 -- start start 2024-03-06T07:41:35.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0389600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03840d0ca0 0x7f03840cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0389600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0384160530 0x7f038406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0389600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038415d4f0 con 0x7f038406b500 2024-03-06T07:41:35.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0389600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03840b50a0 con 0x7f0384160530 2024-03-06T07:41:35.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0389600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038415cc50 con 0x7f03840d0ca0 2024-03-06T07:41:35.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0382000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0384160530 0x7f038406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0382000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0384160530 0x7f038406afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47246/0 (socket says 172.21.15.97:47246) 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0382000640 1 -- 172.21.15.97:0/3459802768 learned_addr learned my addr 172.21.15.97:0/3459802768 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.407+0000 7f0382a00640 1 --2- 172.21.15.97:0/3459802768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03840d0ca0 0x7f03840cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.408+0000 7f0382000640 1 -- 172.21.15.97:0/3459802768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03840d0ca0 msgr2=0x7f03840cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.408+0000 7f0382000640 1 --2- 172.21.15.97:0/3459802768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03840d0ca0 0x7f03840cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.408+0000 7f0382000640 1 -- 172.21.15.97:0/3459802768 >> v1:172.21.15.135:6789/0 conn(0x7f038406b500 legacy=0x7f03840b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:35.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.408+0000 7f0382000640 1 -- 172.21.15.97:0/3459802768 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038415cf30 con 0x7f0384160530 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.409+0000 7f0382000640 1 --2- 172.21.15.97:0/3459802768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0384160530 0x7f038406afc0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f0370009870 tx=0x7f037002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e36db976e7a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.409+0000 7f0381600640 1 -- 172.21.15.97:0/3459802768 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03700317c0 con 0x7f0384160530 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.409+0000 7f0381600640 1 -- 172.21.15.97:0/3459802768 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0370037070 con 0x7f0384160530 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.409+0000 7f0381600640 1 -- 172.21.15.97:0/3459802768 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0370031b80 con 0x7f0384160530 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 -- 172.21.15.97:0/3459802768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0384160530 msgr2=0x7f038406afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 --2- 172.21.15.97:0/3459802768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0384160530 0x7f038406afc0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f0370009870 tx=0x7f037002b860 comp rx=0 tx=0).stop 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 -- 172.21.15.97:0/3459802768 shutdown_connections 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 --2- 172.21.15.97:0/3459802768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0384160530 0x7f038406afc0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 --2- 172.21.15.97:0/3459802768 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f03840d0ca0 0x7f03840cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 -- 172.21.15.97:0/3459802768 >> 172.21.15.97:0/3459802768 conn(0x7f03840b1e10 msgr2=0x7f03840b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:35.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 -- 172.21.15.97:0/3459802768 shutdown_connections 2024-03-06T07:41:35.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.410+0000 7f0389600640 1 -- 172.21.15.97:0/3459802768 wait complete. 2024-03-06T07:41:35.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.412+0000 7f0389600640 1 Processor -- start 2024-03-06T07:41:35.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.412+0000 7f0389600640 1 -- start start 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0389600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 0x7f03840cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0389600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03840d0ca0 0x7f03840cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0389600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0384160530 0x7f038406d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0389600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03840d1080 con 0x7f038406b500 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0389600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0384160990 con 0x7f03840d0ca0 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0389600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03840b5c40 con 0x7f0384160530 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 0x7f03840cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 0x7f03840cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47252/0 (socket says 172.21.15.97:47252) 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 -- 172.21.15.97:0/2482969218 learned_addr learned my addr 172.21.15.97:0/2482969218 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382000640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03840d0ca0 0x7f03840cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0383400640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0384160530 0x7f038406d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 -- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0384160530 msgr2=0x7f038406d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0384160530 0x7f038406d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 -- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03840d0ca0 msgr2=0x7f03840cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03840d0ca0 0x7f03840cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.413+0000 7f0382a00640 1 -- 172.21.15.97:0/2482969218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038406e4f0 con 0x7f038406b500 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.414+0000 7f0382a00640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 0x7f03840cb780 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f037400d830 tx=0x7f037400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.414+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400b6c0 con 0x7f038406b500 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.414+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0374019070 con 0x7f038406b500 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.414+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400ba80 con 0x7f038406b500 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.414+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038406e7d0 con 0x7f038406b500 2024-03-06T07:41:35.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.414+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038406ed90 con 0x7f038406b500 2024-03-06T07:41:35.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.415+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038404f330 con 0x7f038406b500 2024-03-06T07:41:35.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.416+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f037401e460 con 0x7f038406b500 2024-03-06T07:41:35.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.417+0000 7f037aa00640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f036406b4d0 0x7f036406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.417+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f037408aa00 con 0x7f038406b500 2024-03-06T07:41:35.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.417+0000 7f0382000640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f036406b4d0 0x7f036406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.418+0000 7f0382000640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f036406b4d0 0x7f036406d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0370036000 tx=0x7f037003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:35.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.422+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0374059730 con 0x7f038406b500 2024-03-06T07:41:35.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.663+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03840b2ae0 con 0x7f038406b500 2024-03-06T07:41:35.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.664+0000 7f037aa00640 1 -- 172.21.15.97:0/2482969218 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f0374059030 con 0x7f038406b500 2024-03-06T07:41:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:35.667 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:35.667 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f036406b4d0 msgr2=0x7f036406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f036406b4d0 0x7f036406d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0370036000 tx=0x7f037003d040 comp rx=0 tx=0).stop 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 msgr2=0x7f03840cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 0x7f03840cb780 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f037400d830 tx=0x7f037400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 shutdown_connections 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f036406b4d0 0x7f036406d990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f0384160530 0x7f038406d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f03840d0ca0 0x7f03840cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 --2- 172.21.15.97:0/2482969218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f038406b500 0x7f03840cb780 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 >> 172.21.15.97:0/2482969218 conn(0x7f03840b1e10 msgr2=0x7f038415fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 shutdown_connections 2024-03-06T07:41:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.667+0000 7f0389600640 1 -- 172.21.15.97:0/2482969218 wait complete. 2024-03-06T07:41:35.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:41:35.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.893+0000 7f496e400640 1 Processor -- start 2024-03-06T07:41:35.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.893+0000 7f496e400640 1 -- start start 2024-03-06T07:41:35.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.894+0000 7f496e400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4968144c00 0x7f496806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.894+0000 7f496e400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f496815fe60 0x7f49680ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.894+0000 7f496e400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496806ee70 con 0x7f496815f9d0 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f496e400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496806cef0 con 0x7f4968144c00 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f496e400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496806d1d0 con 0x7f496815fe60 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4968144c00 0x7f496806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f4968144c00 0x7f496806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47262/0 (socket says 172.21.15.97:47262) 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 -- 172.21.15.97:0/2904611524 learned_addr learned my addr 172.21.15.97:0/2904611524 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 -- 172.21.15.97:0/2904611524 >> v2:172.21.15.135:3300/0 conn(0x7f496815fe60 msgr2=0x7f49680ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 --2- 172.21.15.97:0/2904611524 >> v2:172.21.15.135:3300/0 conn(0x7f496815fe60 0x7f49680ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 -- 172.21.15.97:0/2904611524 >> v1:172.21.15.97:6789/0 conn(0x7f496815f9d0 legacy=0x7f49680ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.895+0000 7f4967400640 1 -- 172.21.15.97:0/2904611524 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496806d4b0 con 0x7f4968144c00 2024-03-06T07:41:35.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.896+0000 7f4967400640 1 --2- 172.21.15.97:0/2904611524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4968144c00 0x7f496806e930 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f496806e500 tx=0x7f495002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f3fa6aa5ad3948 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:35.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f4966000640 1 -- 172.21.15.97:0/2904611524 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950034070 con 0x7f4968144c00 2024-03-06T07:41:35.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f4966000640 1 -- 172.21.15.97:0/2904611524 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4950030870 con 0x7f4968144c00 2024-03-06T07:41:35.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f4966000640 1 -- 172.21.15.97:0/2904611524 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950030c40 con 0x7f4968144c00 2024-03-06T07:41:35.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f496e400640 1 -- 172.21.15.97:0/2904611524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4968144c00 msgr2=0x7f496806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f496e400640 1 --2- 172.21.15.97:0/2904611524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4968144c00 0x7f496806e930 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f496806e500 tx=0x7f495002b9b0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f496e400640 1 -- 172.21.15.97:0/2904611524 shutdown_connections 2024-03-06T07:41:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f496e400640 1 --2- 172.21.15.97:0/2904611524 >> v2:172.21.15.135:3300/0 conn(0x7f496815fe60 0x7f49680ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f496e400640 1 --2- 172.21.15.97:0/2904611524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4968144c00 0x7f496806e930 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.897+0000 7f496e400640 1 -- 172.21.15.97:0/2904611524 >> 172.21.15.97:0/2904611524 conn(0x7f49680b1730 msgr2=0x7f49680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.898+0000 7f496e400640 1 -- 172.21.15.97:0/2904611524 shutdown_connections 2024-03-06T07:41:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.898+0000 7f496e400640 1 -- 172.21.15.97:0/2904611524 wait complete. 2024-03-06T07:41:35.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.900+0000 7f496e400640 1 Processor -- start 2024-03-06T07:41:35.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.900+0000 7f496e400640 1 -- start start 2024-03-06T07:41:35.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f496e400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4968144c00 0x7f49681393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f496e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 0x7f496813c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f496e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f496815fe60 0x7f496813c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f496e400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968160730 con 0x7f496815f9d0 2024-03-06T07:41:35.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f496e400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968160270 con 0x7f4968144c00 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f496e400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496806cb50 con 0x7f496815fe60 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f4966a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 0x7f496813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f4966a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 0x7f496813c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47276/0 (socket says 172.21.15.97:47276) 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f4966a00640 1 -- 172.21.15.97:0/2478602511 learned_addr learned my addr 172.21.15.97:0/2478602511 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.902+0000 7f4967400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4968144c00 0x7f49681393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4967e00640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f496815fe60 0x7f496813c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4966a00640 1 -- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f496815fe60 msgr2=0x7f496813c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4966a00640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f496815fe60 0x7f496813c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4966a00640 1 -- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4968144c00 msgr2=0x7f49681393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4966a00640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4968144c00 0x7f49681393e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:35.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4966a00640 1 -- 172.21.15.97:0/2478602511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4968139920 con 0x7f496815f9d0 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4967e00640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f496815fe60 0x7f496813c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f4966a00640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 0x7f496813c190 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f495800c790 tx=0x7f495800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4958016020 con 0x7f496815f9d0 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4958006d50 con 0x7f496815f9d0 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4958003bb0 con 0x7f496815f9d0 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.903+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496813ee20 con 0x7f496815f9d0 2024-03-06T07:41:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.904+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49681e5a40 con 0x7f496815f9d0 2024-03-06T07:41:35.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.905+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4968143a60 con 0x7f496815f9d0 2024-03-06T07:41:35.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.905+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f495800a850 con 0x7f496815f9d0 2024-03-06T07:41:35.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.906+0000 7f495f400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f494406b4d0 0x7f494406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:35.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.906+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4958089f30 con 0x7f496815f9d0 2024-03-06T07:41:35.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.907+0000 7f4967400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f494406b4d0 0x7f494406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:35.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.907+0000 7f4967400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f494406b4d0 0x7f494406d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4950036000 tx=0x7f495003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:35.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:35.911+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4968143a60 con 0x7f496815f9d0 2024-03-06T07:41:36.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.148+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f496804f330 con 0x7f496815f9d0 2024-03-06T07:41:36.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.149+0000 7f495f400640 1 -- 172.21.15.97:0/2478602511 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3468 (secure 0 0 0) 0x7f4958020020 con 0x7f496815f9d0 2024-03-06T07:41:36.151 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:36.151 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:37:21.810170+0000","modified":"2024-03-06T07:39:50.920314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9024,"mds_1":6463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6463":{"gid":6463,"name":"a","rank":1,"incarnation":109,"state":"up:stopping","state_seq":5,"addr":"172.21.15.97:6837/1171184135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":1171184135},{"type":"v1","addr":"172.21.15.97:6837","nonce":1171184135}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:38:36.706910+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9024":{"gid":9024,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.97:6835/4190861064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":4190861064},{"type":"v1","addr":"172.21.15.97:6835","nonce":4190861064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9024,"qdb_cluster":[9024]},"id":10}]} 2024-03-06T07:41:36.151 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 117 2024-03-06T07:41:36.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.151+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f494406b4d0 msgr2=0x7f494406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.151+0000 7f496e400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f494406b4d0 0x7f494406d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4950036000 tx=0x7f495003b040 comp rx=0 tx=0).stop 2024-03-06T07:41:36.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 msgr2=0x7f496813c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 0x7f496813c190 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f495800c790 tx=0x7f495800cc50 comp rx=0 tx=0).stop 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 shutdown_connections 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f494406b4d0 0x7f494406d990 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f496815fe60 0x7f496813c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f496815f9d0 0x7f496813c190 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 --2- 172.21.15.97:0/2478602511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4968144c00 0x7f49681393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 >> 172.21.15.97:0/2478602511 conn(0x7f49680b1730 msgr2=0x7f4968070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 shutdown_connections 2024-03-06T07:41:36.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.152+0000 7f496e400640 1 -- 172.21.15.97:0/2478602511 wait complete. 2024-03-06T07:41:36.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:41:36.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.386+0000 7f24a3400640 1 Processor -- start 2024-03-06T07:41:36.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.386+0000 7f24a3400640 1 -- start start 2024-03-06T07:41:36.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f249c144660 0x7f249c071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c0cf3e0 0x7f249c0724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a3400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0cabf0 con 0x7f249c0c9ec0 2024-03-06T07:41:36.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a3400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249c0ce900 con 0x7f249c144660 2024-03-06T07:41:36.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a3400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0cebe0 con 0x7f249c0cf3e0 2024-03-06T07:41:36.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f249c144660 0x7f249c071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a0a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f249c144660 0x7f249c071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47292/0 (socket says 172.21.15.97:47292) 2024-03-06T07:41:36.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a1400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f249c0c9ec0 0x7f249c0ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37094/0 (socket says 172.21.15.97:37094) 2024-03-06T07:41:36.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a0a00640 1 -- 172.21.15.97:0/1357856661 learned_addr learned my addr 172.21.15.97:0/1357856661 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:36.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f249be00640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c0cf3e0 0x7f249c0724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a0a00640 1 -- 172.21.15.97:0/1357856661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c0cf3e0 msgr2=0x7f249c0724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a0a00640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c0cf3e0 0x7f249c0724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.388+0000 7f24a0a00640 1 -- 172.21.15.97:0/1357856661 >> v1:172.21.15.97:6789/0 conn(0x7f249c0c9ec0 legacy=0x7f249c0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.389+0000 7f24a0a00640 1 -- 172.21.15.97:0/1357856661 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c0ceec0 con 0x7f249c144660 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.389+0000 7f249be00640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c0cf3e0 0x7f249c0724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.389+0000 7f249b400640 1 -- 172.21.15.97:0/1357856661 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215571717 0 0) 0x7f249c0cabf0 con 0x7f249c0c9ec0 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.390+0000 7f24a0a00640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c144660 0x7f249c071f70 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f2490009870 tx=0x7f249002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeeb4533749599e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.390+0000 7f249b400640 1 -- 172.21.15.97:0/1357856661 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490034070 con 0x7f249c144660 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.390+0000 7f249b400640 1 -- 172.21.15.97:0/1357856661 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490030840 con 0x7f249c144660 2024-03-06T07:41:36.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.390+0000 7f249b400640 1 -- 172.21.15.97:0/1357856661 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490030c10 con 0x7f249c144660 2024-03-06T07:41:36.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 -- 172.21.15.97:0/1357856661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c144660 msgr2=0x7f249c071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c144660 0x7f249c071f70 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f2490009870 tx=0x7f249002b980 comp rx=0 tx=0).stop 2024-03-06T07:41:36.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 -- 172.21.15.97:0/1357856661 shutdown_connections 2024-03-06T07:41:36.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c0cf3e0 0x7f249c0724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 --2- 172.21.15.97:0/1357856661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c144660 0x7f249c071f70 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 -- 172.21.15.97:0/1357856661 >> 172.21.15.97:0/1357856661 conn(0x7f249c0d0b90 msgr2=0x7f249c0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 -- 172.21.15.97:0/1357856661 shutdown_connections 2024-03-06T07:41:36.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.391+0000 7f24a3400640 1 -- 172.21.15.97:0/1357856661 wait complete. 2024-03-06T07:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.393+0000 7f24a3400640 1 Processor -- start 2024-03-06T07:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.393+0000 7f24a3400640 1 -- start start 2024-03-06T07:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 0x7f249c1f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a3400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f249c0cf3e0 0x7f249c1f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a3400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c144660 0x7f249c0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a3400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0cc270 con 0x7f249c0c9ec0 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a3400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0cc0f0 con 0x7f249c0cf3e0 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a3400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0cc3f0 con 0x7f249c144660 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 0x7f249c1f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 0x7f249c1f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47306/0 (socket says 172.21.15.97:47306) 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a0a00640 1 -- 172.21.15.97:0/3211555731 learned_addr learned my addr 172.21.15.97:0/3211555731 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f249be00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f249c0cf3e0 0x7f249c1f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.394+0000 7f24a1400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c144660 0x7f249c0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a0a00640 1 -- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c144660 msgr2=0x7f249c0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a0a00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c144660 0x7f249c0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a0a00640 1 -- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f249c0cf3e0 msgr2=0x7f249c1f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a0a00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f249c0cf3e0 0x7f249c1f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a0a00640 1 -- 172.21.15.97:0/3211555731 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c1f78e0 con 0x7f249c0c9ec0 2024-03-06T07:41:36.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a1400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c144660 0x7f249c0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:36.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f249be00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f249c0cf3e0 0x7f249c1f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:36.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a0a00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 0x7f249c1f2fd0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f24900090b0 tx=0x7f24900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:36.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490034040 con 0x7f249c0c9ec0 2024-03-06T07:41:36.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249003ad90 con 0x7f249c0c9ec0 2024-03-06T07:41:36.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c1f7bc0 con 0x7f249c0c9ec0 2024-03-06T07:41:36.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.395+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490004270 con 0x7f249c0c9ec0 2024-03-06T07:41:36.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.396+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c152a60 con 0x7f249c0c9ec0 2024-03-06T07:41:36.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.397+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c0a9450 con 0x7f249c0c9ec0 2024-03-06T07:41:36.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.398+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f249004b060 con 0x7f249c0c9ec0 2024-03-06T07:41:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.400+0000 7f2498c00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f247c06b4d0 0x7f247c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.400+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f24900ac1d0 con 0x7f249c0c9ec0 2024-03-06T07:41:36.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.400+0000 7f249be00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f247c06b4d0 0x7f247c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.401+0000 7f249be00640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f247c06b4d0 0x7f247c06d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2488006fd0 tx=0x7f2488008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:36.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.405+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f249003f070 con 0x7f249c0c9ec0 2024-03-06T07:41:36.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.602+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f249c04f330 con 0x7f249c0c9ec0 2024-03-06T07:41:36.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.603+0000 7f2498c00640 1 -- 172.21.15.97:0/3211555731 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f249007b150 con 0x7f249c0c9ec0 2024-03-06T07:41:36.606 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:36.606 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":148,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:37:40.116487+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:37:19.579712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:37:20.835626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.39","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f247c06b4d0 msgr2=0x7f247c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f247c06b4d0 0x7f247c06d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2488006fd0 tx=0x7f2488008040 comp rx=0 tx=0).stop 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 msgr2=0x7f249c1f2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 0x7f249c1f2fd0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f24900090b0 tx=0x7f24900046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 shutdown_connections 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f247c06b4d0 0x7f247c06d990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f249c144660 0x7f249c0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f249c0cf3e0 0x7f249c1f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 --2- 172.21.15.97:0/3211555731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f249c0c9ec0 0x7f249c1f2fd0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 >> 172.21.15.97:0/3211555731 conn(0x7f249c0d0b90 msgr2=0x7f249c06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:36.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 shutdown_connections 2024-03-06T07:41:36.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.606+0000 7f24a3400640 1 -- 172.21.15.97:0/3211555731 wait complete. 2024-03-06T07:41:36.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:41:36.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.837+0000 7f8296000640 1 Processor -- start 2024-03-06T07:41:36.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.837+0000 7f8296000640 1 -- start start 2024-03-06T07:41:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.839+0000 7f8296000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 0x7f82900d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.839+0000 7f8296000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8290161960 con 0x7f82900d1e90 2024-03-06T07:41:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.839+0000 7f8296000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82900cf6f0 con 0x7f82900d2a70 2024-03-06T07:41:36.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.839+0000 7f8296000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82900cf9d0 con 0x7f8290161580 2024-03-06T07:41:36.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.839+0000 7f8287e00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f82900d1e90 0x7f82900d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37098/0 (socket says 172.21.15.97:37098) 2024-03-06T07:41:36.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.839+0000 7f8287e00640 1 -- 172.21.15.97:0/2486242255 learned_addr learned my addr 172.21.15.97:0/2486242255 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:36.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f8294c00640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 0x7f82900d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852664879 0 0) 0x7f8290161960 con 0x7f82900d1e90 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8274003520 con 0x7f82900d1e90 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553821667 0 0) 0x7f8274003520 con 0x7f82900d1e90 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 msgr2=0x7f82900d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 0x7f82900d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 >> v1:172.21.15.135:6789/0 conn(0x7f82900d2a70 legacy=0x7f82900b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82900cfcb0 con 0x7f82900d1e90 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f8278001670 con 0x7f82900d1e90 2024-03-06T07:41:36.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 >> v1:172.21.15.97:6789/0 conn(0x7f82900d1e90 legacy=0x7f82900d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82740039e0 0x7f8274003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 0x7f8274004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f82740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82900cf9d0 con 0x7f8274004d40 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827400ba70 con 0x7f82740039e0 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.840+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827400bc70 con 0x7f8290161580 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8295600640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f82740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8294c00640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82740039e0 0x7f8274003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8295600640 1 -- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 msgr2=0x7f8274004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8295600640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8290161580 0x7f8274004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8295600640 1 -- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82740039e0 msgr2=0x7f8274003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8295600640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82740039e0 0x7f8274003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.841+0000 7f8295600640 1 -- 172.21.15.97:0/2486242255 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274003520 con 0x7f8274004d40 2024-03-06T07:41:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.842+0000 7f8295600640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f82740051c0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f8288001e20 tx=0x7f8274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601c399a39a41f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.842+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288001a70 con 0x7f8274004d40 2024-03-06T07:41:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8288003b70 con 0x7f8274004d40 2024-03-06T07:41:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f828fe00640 1 -- 172.21.15.97:0/2486242255 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82880050d0 con 0x7f8274004d40 2024-03-06T07:41:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8296000640 1 -- 172.21.15.97:0/2486242255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 msgr2=0x7f82740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8296000640 1 --2- 172.21.15.97:0/2486242255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f82740051c0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f8288001e20 tx=0x7f8274005830 comp rx=0 tx=0).stop 2024-03-06T07:41:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8295600640 1 -- 172.21.15.97:0/2486242255 reap_dead start 2024-03-06T07:41:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8296000640 1 -- 172.21.15.97:0/2486242255 shutdown_connections 2024-03-06T07:41:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8296000640 1 -- 172.21.15.97:0/2486242255 >> 172.21.15.97:0/2486242255 conn(0x7f82900d3d80 msgr2=0x7f82900d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8296000640 1 -- 172.21.15.97:0/2486242255 shutdown_connections 2024-03-06T07:41:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.843+0000 7f8296000640 1 -- 172.21.15.97:0/2486242255 wait complete. 2024-03-06T07:41:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.845+0000 7f8296000640 1 Processor -- start 2024-03-06T07:41:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.845+0000 7f8296000640 1 -- start start 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8296000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f829014c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8296000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82900cbef0 0x7f82900c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8296000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82900c9720 0x7f82900c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8296000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82900cf3a0 con 0x7f8274004d40 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8296000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82900b39a0 con 0x7f82900cbef0 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8296000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82900d1950 con 0x7f82900c9720 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8294c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f829014c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8294c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f829014c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47314/0 (socket says 172.21.15.97:47314) 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.846+0000 7f8294c00640 1 -- 172.21.15.97:0/4031610631 learned_addr learned my addr 172.21.15.97:0/4031610631 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f8294c00640 1 -- 172.21.15.97:0/4031610631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82900c9720 msgr2=0x7f82900c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f8294c00640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82900c9720 0x7f82900c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f8294c00640 1 -- 172.21.15.97:0/4031610631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82900cbef0 msgr2=0x7f82900c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f8294c00640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82900cbef0 0x7f82900c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f8294c00640 1 -- 172.21.15.97:0/4031610631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82900c8e20 con 0x7f8274004d40 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f8294c00640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f829014c130 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f8280010890 tx=0x7f8280010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.847+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828000f040 con 0x7f8274004d40 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.848+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828000eae0 con 0x7f8274004d40 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.848+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82900c7d80 con 0x7f8274004d40 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.848+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828001c400 con 0x7f8274004d40 2024-03-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.848+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82900c8340 con 0x7f8274004d40 2024-03-06T07:41:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.849+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82900c8a60 con 0x7f8274004d40 2024-03-06T07:41:36.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.850+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f828001c5a0 con 0x7f8274004d40 2024-03-06T07:41:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.851+0000 7f828d600640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826c06b500 0x7f826c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.851+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f828008a790 con 0x7f8274004d40 2024-03-06T07:41:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.851+0000 7f8287e00640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826c06b500 0x7f826c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:36.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.852+0000 7f8287e00640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826c06b500 0x7f826c06d9c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f82900cc310 tx=0x7f8278001520 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:36.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:36.860+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8280018030 con 0x7f8274004d40 2024-03-06T07:41:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.096+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f829004f330 con 0x7f8274004d40 2024-03-06T07:41:37.100 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-06T07:41:37.097+0000 7f68fa000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:41:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.101+0000 7f828d600640 1 -- 172.21.15.97:0/4031610631 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) v1 ==== 161+0+0 (secure 0 0 0) 0x7f82800595f0 con 0x7f8274004d40 2024-03-06T07:41:37.103 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826c06b500 msgr2=0x7f826c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826c06b500 0x7f826c06d9c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f82900cc310 tx=0x7f8278001520 comp rx=0 tx=0).stop 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 msgr2=0x7f829014c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f829014c130 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f8280010890 tx=0x7f8280010d50 comp rx=0 tx=0).stop 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 shutdown_connections 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f826c06b500 0x7f826c06d9c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f82900c9720 0x7f82900c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82900cbef0 0x7f82900c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 --2- 172.21.15.97:0/4031610631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8274004d40 0x7f829014c130 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.104+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 >> 172.21.15.97:0/4031610631 conn(0x7f82900d3d80 msgr2=0x7f82900ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.105+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 shutdown_connections 2024-03-06T07:41:37.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.105+0000 7f8296000640 1 -- 172.21.15.97:0/4031610631 wait complete. 2024-03-06T07:41:37.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:41:37.206 INFO:tasks.ceph.mds.c.smithi097.stdout:starting mds.c at 2024-03-06T07:41:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.317+0000 7ff398800640 1 Processor -- start 2024-03-06T07:41:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.317+0000 7ff398800640 1 -- start start 2024-03-06T07:41:37.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.318+0000 7ff398800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff3900b3650 0x7ff3900727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:37.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.318+0000 7ff398800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 0x7ff39006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:37.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.318+0000 7ff398800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3900719e0 con 0x7ff3900b3df0 2024-03-06T07:41:37.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.318+0000 7ff398800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff390072310 con 0x7ff3900b3650 2024-03-06T07:41:37.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.318+0000 7ff398800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff390071cc0 con 0x7ff3900cda90 2024-03-06T07:41:37.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.319+0000 7ff396800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 0x7ff39006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:37.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.319+0000 7ff395400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff3900b3650 0x7ff3900727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:37.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.319+0000 7ff396800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 0x7ff39006c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:54174/0 (socket says 172.21.15.97:54174) 2024-03-06T07:41:37.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.319+0000 7ff396800640 1 -- 172.21.15.97:0/940974792 learned_addr learned my addr 172.21.15.97:0/940974792 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:37.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.320+0000 7ff396800640 1 -- 172.21.15.97:0/940974792 >> v2:172.21.15.135:3300/0 conn(0x7ff3900b3650 msgr2=0x7ff3900727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:37.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.320+0000 7ff396800640 1 --2- 172.21.15.97:0/940974792 >> v2:172.21.15.135:3300/0 conn(0x7ff3900b3650 0x7ff3900727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.320+0000 7ff396800640 1 -- 172.21.15.97:0/940974792 >> v1:172.21.15.135:6789/0 conn(0x7ff3900b3df0 legacy=0x7ff3900b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:37.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.320+0000 7ff396800640 1 -- 172.21.15.97:0/940974792 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390071fa0 con 0x7ff3900cda90 2024-03-06T07:41:37.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.320+0000 7ff395400640 1 --2- 172.21.15.97:0/940974792 >> v2:172.21.15.135:3300/0 conn(0x7ff3900b3650 0x7ff3900727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:41:37.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.321+0000 7ff396800640 1 --2- 172.21.15.97:0/940974792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 0x7ff39006c990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009e50 tx=0x7ff38c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb1ea9146765be44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff394a00640 1 -- 172.21.15.97:0/940974792 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c02cb00 con 0x7ff3900cda90 2024-03-06T07:41:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff394a00640 1 -- 172.21.15.97:0/940974792 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c02cca0 con 0x7ff3900cda90 2024-03-06T07:41:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff394a00640 1 -- 172.21.15.97:0/940974792 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c034610 con 0x7ff3900cda90 2024-03-06T07:41:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff398800640 1 -- 172.21.15.97:0/940974792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 msgr2=0x7ff39006c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff398800640 1 --2- 172.21.15.97:0/940974792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 0x7ff39006c990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009e50 tx=0x7ff38c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:41:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff398800640 1 -- 172.21.15.97:0/940974792 shutdown_connections 2024-03-06T07:41:37.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff398800640 1 --2- 172.21.15.97:0/940974792 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900cda90 0x7ff39006c990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff398800640 1 --2- 172.21.15.97:0/940974792 >> v2:172.21.15.135:3300/0 conn(0x7ff3900b3650 0x7ff3900727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.322+0000 7ff398800640 1 -- 172.21.15.97:0/940974792 >> 172.21.15.97:0/940974792 conn(0x7ff3900b1940 msgr2=0x7ff3900b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:37.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.323+0000 7ff398800640 1 -- 172.21.15.97:0/940974792 shutdown_connections 2024-03-06T07:41:37.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.323+0000 7ff398800640 1 -- 172.21.15.97:0/940974792 wait complete. 2024-03-06T07:41:37.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.324+0000 7ff398800640 1 Processor -- start 2024-03-06T07:41:37.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.324+0000 7ff398800640 1 -- start start 2024-03-06T07:41:37.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.325+0000 7ff398800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 0x7ff390144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:37.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.325+0000 7ff398800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900b3df0 0x7ff39009d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:37.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.325+0000 7ff398800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3900cda90 0x7ff3900ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:37.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.325+0000 7ff398800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39015b3c0 con 0x7ff3900b3650 2024-03-06T07:41:37.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.325+0000 7ff398800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39015b240 con 0x7ff3900cda90 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.325+0000 7ff398800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3900cf510 con 0x7ff3900b3df0 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 0x7ff390144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 0x7ff390144470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47330/0 (socket says 172.21.15.97:47330) 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 -- 172.21.15.97:0/1234600871 learned_addr learned my addr 172.21.15.97:0/1234600871 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 -- 172.21.15.97:0/1234600871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900b3df0 msgr2=0x7ff39009d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900b3df0 0x7ff39009d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 -- 172.21.15.97:0/1234600871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3900cda90 msgr2=0x7ff3900ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3900cda90 0x7ff3900ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 -- 172.21.15.97:0/1234600871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3901449b0 con 0x7ff3900b3650 2024-03-06T07:41:37.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.326+0000 7ff395e00640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 0x7ff390144470 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7ff38400c400 tx=0x7ff38400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:37.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.327+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3840052d0 con 0x7ff3900b3650 2024-03-06T07:41:37.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.327+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff384005490 con 0x7ff3900b3650 2024-03-06T07:41:37.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.327+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384007260 con 0x7ff3900b3650 2024-03-06T07:41:37.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.327+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3900c93a0 con 0x7ff3900b3650 2024-03-06T07:41:37.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.327+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3900c9960 con 0x7ff3900b3650 2024-03-06T07:41:37.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.328+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39004f330 con 0x7ff3900b3650 2024-03-06T07:41:37.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.329+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff384007400 con 0x7ff3900b3650 2024-03-06T07:41:37.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.330+0000 7ff38a000640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff37406b4d0 0x7ff37406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:37.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.330+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5910+0+0 (secure 0 0 0) 0x7ff3840898c0 con 0x7ff3900b3650 2024-03-06T07:41:37.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.330+0000 7ff395400640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff37406b4d0 0x7ff37406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:37.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.331+0000 7ff395400640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff37406b4d0 0x7ff37406d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff380006e80 tx=0x7ff380006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:37.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.335+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3840585d0 con 0x7ff3900b3650 2024-03-06T07:41:37.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:37.577+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff39006dde0 con 0x7ff3900b3650 2024-03-06T07:41:38.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.111+0000 7ff38a000640 1 -- 172.21.15.97:0/1234600871 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff384057ed0 con 0x7ff3900b3650 2024-03-06T07:41:38.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.115+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff37406b4d0 msgr2=0x7ff37406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.115+0000 7ff398800640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff37406b4d0 0x7ff37406d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff380006e80 tx=0x7ff380006dd0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.115+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 msgr2=0x7ff390144470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.115+0000 7ff398800640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 0x7ff390144470 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7ff38400c400 tx=0x7ff38400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.115+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 shutdown_connections 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff37406b4d0 0x7ff37406d990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3900cda90 0x7ff3900ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff3900b3df0 0x7ff39009d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 --2- 172.21.15.97:0/1234600871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3900b3650 0x7ff390144470 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 >> 172.21.15.97:0/1234600871 conn(0x7ff3900b1940 msgr2=0x7ff3900b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 shutdown_connections 2024-03-06T07:41:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.116+0000 7ff398800640 1 -- 172.21.15.97:0/1234600871 wait complete. 2024-03-06T07:41:38.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:41:38.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.340+0000 7fe423000640 1 Processor -- start 2024-03-06T07:41:38.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.341+0000 7fe423000640 1 -- start start 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.342+0000 7fe423000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe41c0d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.342+0000 7fe423000640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c0d4d90 con 0x7fe41c0cfc30 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.342+0000 7fe423000640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c0d4ad0 con 0x7fe41c0d0880 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.342+0000 7fe423000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b4c60 con 0x7fe41c0d0c60 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.342+0000 7fe41b400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe41c0cfc30 0x7fe41c0d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37100/0 (socket says 172.21.15.97:37100) 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.342+0000 7fe41b400640 1 -- 172.21.15.97:0/1002381769 learned_addr learned my addr 172.21.15.97:0/1002381769 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:38.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe421000640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe41c0d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483416917 0 0) 0x7fe41c0d4d90 con 0x7fe41c0cfc30 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe400003520 con 0x7fe41c0cfc30 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355000472 0 0) 0x7fe400003520 con 0x7fe41c0cfc30 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 msgr2=0x7fe41c0d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe41c0d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 >> v1:172.21.15.135:6789/0 conn(0x7fe41c0d0880 legacy=0x7fe41c0b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0cf710 con 0x7fe41c0cfc30 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fe408001670 con 0x7fe41c0cfc30 2024-03-06T07:41:38.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe421000640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe41c0d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 >> v1:172.21.15.97:6789/0 conn(0x7fe41c0cfc30 legacy=0x7fe41c0d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe400003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe400004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.343+0000 7fe41aa00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe400004d40 0x7fe4000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b4c60 con 0x7fe400003a50 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40000bbd0 con 0x7fe400004d40 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40000bdd0 con 0x7fe41c0d0c60 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41be00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe400003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe421000640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe400004d40 0x7fe4000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41be00640 1 -- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 msgr2=0x7fe400004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:38.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41b400640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe400004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41be00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0d0c60 0x7fe400004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41be00640 1 -- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe400004d40 msgr2=0x7fe4000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:38.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41be00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe400004d40 0x7fe4000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe41be00640 1 -- 172.21.15.97:0/1002381769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400003520 con 0x7fe400003a50 2024-03-06T07:41:38.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.344+0000 7fe421000640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe400004d40 0x7fe4000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:38.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.345+0000 7fe41be00640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe400003e70 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fe40c001720 tx=0x7fe400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c954430051494c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:38.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c001370 con 0x7fe400003a50 2024-03-06T07:41:38.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40c0052d0 con 0x7fe400003a50 2024-03-06T07:41:38.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe41aa00640 1 -- 172.21.15.97:0/1002381769 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c003830 con 0x7fe400003a50 2024-03-06T07:41:38.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe423000640 1 -- 172.21.15.97:0/1002381769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 msgr2=0x7fe400003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe423000640 1 --2- 172.21.15.97:0/1002381769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe400003e70 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fe40c001720 tx=0x7fe400005830 comp rx=0 tx=0).stop 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe421000640 1 -- 172.21.15.97:0/1002381769 reap_dead start 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe423000640 1 -- 172.21.15.97:0/1002381769 shutdown_connections 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe423000640 1 -- 172.21.15.97:0/1002381769 >> 172.21.15.97:0/1002381769 conn(0x7fe41c0d3e30 msgr2=0x7fe41c0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe423000640 1 -- 172.21.15.97:0/1002381769 shutdown_connections 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.346+0000 7fe423000640 1 -- 172.21.15.97:0/1002381769 wait complete. 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.348+0000 7fe423000640 1 Processor -- start 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.348+0000 7fe423000640 1 -- start start 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe423000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe41c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe423000640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe41c153750 0x7fe41c153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe423000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0cc5a0 0x7fe41c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe423000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b5200 con 0x7fe400003a50 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe423000640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b5080 con 0x7fe41c153750 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe423000640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b5380 con 0x7fe41c0cc5a0 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe41be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe41c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.349+0000 7fe41be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe41c0b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47342/0 (socket says 172.21.15.97:47342) 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 -- 172.21.15.97:0/560184861 learned_addr learned my addr 172.21.15.97:0/560184861 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41b400640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe41c153750 0x7fe41c153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe421000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0cc5a0 0x7fe41c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 -- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0cc5a0 msgr2=0x7fe41c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0cc5a0 0x7fe41c0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 -- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe41c153750 msgr2=0x7fe41c153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe41c153750 0x7fe41c153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 -- 172.21.15.97:0/560184861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0b1300 con 0x7fe400003a50 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.350+0000 7fe41be00640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe41c0b20c0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fe40c001840 tx=0x7fe40c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.351+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c041070 con 0x7fe400003a50 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.351+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40c005480 con 0x7fe400003a50 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.351+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c007a30 con 0x7fe400003a50 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.351+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c0b15e0 con 0x7fe400003a50 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.351+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c0b1ba0 con 0x7fe400003a50 2024-03-06T07:41:38.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.352+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c04f3a0 con 0x7fe400003a50 2024-03-06T07:41:38.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.353+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe40c044070 con 0x7fe400003a50 2024-03-06T07:41:38.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.354+0000 7fe413e00640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe3fc06b4d0 0x7fe3fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:38.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.354+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fe40c0abe70 con 0x7fe400003a50 2024-03-06T07:41:38.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.354+0000 7fe41b400640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe3fc06b4d0 0x7fe3fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:38.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.355+0000 7fe41b400640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe3fc06b4d0 0x7fe3fc06d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe41c160090 tx=0x7fe408001780 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:38.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.359+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe40c07ad30 con 0x7fe400003a50 2024-03-06T07:41:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:38.558+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe41c0d2b90 con 0x7fe400003a50 2024-03-06T07:41:39.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.115+0000 7fe413e00640 1 -- 172.21.15.97:0/560184861 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v150) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe40c07a630 con 0x7fe400003a50 2024-03-06T07:41:39.118 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:41:39.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe3fc06b4d0 msgr2=0x7fe3fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe3fc06b4d0 0x7fe3fc06d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe41c160090 tx=0x7fe408001780 comp rx=0 tx=0).stop 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 msgr2=0x7fe41c0b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe41c0b20c0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fe40c001840 tx=0x7fe40c001120 comp rx=0 tx=0).stop 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 shutdown_connections 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe3fc06b4d0 0x7fe3fc06d990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe41c0cc5a0 0x7fe41c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe41c153750 0x7fe41c153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 --2- 172.21.15.97:0/560184861 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe400003a50 0x7fe41c0b20c0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 >> 172.21.15.97:0/560184861 conn(0x7fe41c0d3e30 msgr2=0x7fe41c0d19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:39.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 shutdown_connections 2024-03-06T07:41:39.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.120+0000 7fe423000640 1 -- 172.21.15.97:0/560184861 wait complete. 2024-03-06T07:41:39.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:41:39.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.358+0000 7fb55cc00640 1 Processor -- start 2024-03-06T07:41:39.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.358+0000 7fb55cc00640 1 -- start start 2024-03-06T07:41:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.359+0000 7fb55cc00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb55815adf0 0x7fb55806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.359+0000 7fb55cc00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb55806e8e0 0x7fb55815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.359+0000 7fb55cc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815f020 0x7fb5580716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.359+0000 7fb55cc00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55815b1d0 con 0x7fb55815adf0 2024-03-06T07:41:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.359+0000 7fb55cc00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580b2bd0 con 0x7fb55806e8e0 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.359+0000 7fb55cc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580b2eb0 con 0x7fb55815f020 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb55815adf0 0x7fb55806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb55815adf0 0x7fb55806e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47344/0 (socket says 172.21.15.97:47344) 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 -- 172.21.15.97:0/3313341572 learned_addr learned my addr 172.21.15.97:0/3313341572 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 -- 172.21.15.97:0/3313341572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815f020 msgr2=0x7fb5580716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 --2- 172.21.15.97:0/3313341572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815f020 0x7fb5580716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 -- 172.21.15.97:0/3313341572 >> v2:172.21.15.135:3300/0 conn(0x7fb55806e8e0 msgr2=0x7fb55815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555400640 1 --2- 172.21.15.97:0/3313341572 >> v2:172.21.15.135:3300/0 conn(0x7fb55806e8e0 0x7fb55815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 --2- 172.21.15.97:0/3313341572 >> v2:172.21.15.135:3300/0 conn(0x7fb55806e8e0 0x7fb55815eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555e00640 1 -- 172.21.15.97:0/3313341572 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580b3190 con 0x7fb55815adf0 2024-03-06T07:41:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.360+0000 7fb555400640 1 --2- 172.21.15.97:0/3313341572 >> v2:172.21.15.135:3300/0 conn(0x7fb55806e8e0 0x7fb55815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.361+0000 7fb555e00640 1 --2- 172.21.15.97:0/3313341572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55815adf0 0x7fb55806e3a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fb558068ce0 tx=0x7fb54002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd7f826b86b192f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb554a00640 1 -- 172.21.15.97:0/3313341572 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54003b070 con 0x7fb55815adf0 2024-03-06T07:41:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb554a00640 1 -- 172.21.15.97:0/3313341572 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540036440 con 0x7fb55815adf0 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb554a00640 1 -- 172.21.15.97:0/3313341572 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540036810 con 0x7fb55815adf0 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 -- 172.21.15.97:0/3313341572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55815adf0 msgr2=0x7fb55806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3313341572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55815adf0 0x7fb55806e3a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fb558068ce0 tx=0x7fb54002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 -- 172.21.15.97:0/3313341572 shutdown_connections 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3313341572 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815f020 0x7fb5580716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3313341572 >> v2:172.21.15.135:3300/0 conn(0x7fb55806e8e0 0x7fb55815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3313341572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55815adf0 0x7fb55806e3a0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 -- 172.21.15.97:0/3313341572 >> 172.21.15.97:0/3313341572 conn(0x7fb5580cfa80 msgr2=0x7fb5580cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 -- 172.21.15.97:0/3313341572 shutdown_connections 2024-03-06T07:41:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.362+0000 7fb55cc00640 1 -- 172.21.15.97:0/3313341572 wait complete. 2024-03-06T07:41:39.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.364+0000 7fb55cc00640 1 Processor -- start 2024-03-06T07:41:39.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.364+0000 7fb55cc00640 1 -- start start 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb55cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 0x7fb558143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb55cc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815adf0 0x7fb55815c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb55cc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb55815f020 0x7fb55815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb55cc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580b3990 con 0x7fb55806e8e0 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb55cc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815f850 con 0x7fb55815f020 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb55cc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815f4b0 con 0x7fb55815adf0 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb555e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 0x7fb558143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb555e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 0x7fb558143350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47358/0 (socket says 172.21.15.97:47358) 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.365+0000 7fb555e00640 1 -- 172.21.15.97:0/3788205464 learned_addr learned my addr 172.21.15.97:0/3788205464 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb555e00640 1 -- 172.21.15.97:0/3788205464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815adf0 msgr2=0x7fb55815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb555e00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815adf0 0x7fb55815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb555e00640 1 -- 172.21.15.97:0/3788205464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb55815f020 msgr2=0x7fb55815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb555e00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb55815f020 0x7fb55815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb555e00640 1 -- 172.21.15.97:0/3788205464 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558157fa0 con 0x7fb55806e8e0 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb555e00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 0x7fb558143350 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb54002e120 tx=0x7fb5400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54003b040 con 0x7fb55806e8e0 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540046070 con 0x7fb55806e8e0 2024-03-06T07:41:39.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.366+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55814fbf0 con 0x7fb55806e8e0 2024-03-06T07:41:39.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.367+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540034bb0 con 0x7fb55806e8e0 2024-03-06T07:41:39.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.367+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb558143890 con 0x7fb55806e8e0 2024-03-06T07:41:39.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.368+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55804f330 con 0x7fb55806e8e0 2024-03-06T07:41:39.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.369+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb54004b460 con 0x7fb55806e8e0 2024-03-06T07:41:39.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.369+0000 7fb54e000640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb53406b4d0 0x7fb53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:39.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.369+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5472+0+0 (secure 0 0 0) 0x7fb5400af010 con 0x7fb55806e8e0 2024-03-06T07:41:39.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.370+0000 7fb555400640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb53406b4d0 0x7fb53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:39.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.370+0000 7fb555400640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb53406b4d0 0x7fb53406d990 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb55814e830 tx=0x7fb548008330 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:39.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.374+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb540049360 con 0x7fb55806e8e0 2024-03-06T07:41:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:39.567+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb55806cd10 con 0x7fb55806e8e0 2024-03-06T07:41:40.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.125+0000 7fb54e000640 1 -- 172.21.15.97:0/3788205464 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v151) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb54007df80 con 0x7fb55806e8e0 2024-03-06T07:41:40.128 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.129+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb53406b4d0 msgr2=0x7fb53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.129+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb53406b4d0 0x7fb53406d990 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb55814e830 tx=0x7fb548008330 comp rx=0 tx=0).stop 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.129+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 msgr2=0x7fb558143350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.129+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 0x7fb558143350 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb54002e120 tx=0x7fb5400046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.130+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 shutdown_connections 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.130+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb53406b4d0 0x7fb53406d990 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.130+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb55815f020 0x7fb55815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.130+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb55815adf0 0x7fb55815c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.130+0000 7fb55cc00640 1 --2- 172.21.15.97:0/3788205464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb55806e8e0 0x7fb558143350 secure :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb54002e120 tx=0x7fb5400046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.131+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 >> 172.21.15.97:0/3788205464 conn(0x7fb5580cfa80 msgr2=0x7fb5580cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:40.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.132+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 shutdown_connections 2024-03-06T07:41:40.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.132+0000 7fb55cc00640 1 -- 172.21.15.97:0/3788205464 wait complete. 2024-03-06T07:41:40.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:41:40.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.356+0000 7f47a4400640 1 Processor -- start 2024-03-06T07:41:40.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.356+0000 7f47a4400640 1 -- start start 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a4400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f479c0b3650 0x7f479c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a4400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0cda90 0x7f479c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a4400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479c0719e0 con 0x7f479c0b3df0 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a4400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c072310 con 0x7f479c0b3650 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a4400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c071cc0 con 0x7f479c0cda90 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f479c0b3650 0x7f479c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a1000640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f479c0b3650 0x7f479c0727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47370/0 (socket says 172.21.15.97:47370) 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a1000640 1 -- 172.21.15.97:0/766518791 learned_addr learned my addr 172.21.15.97:0/766518791 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.358+0000 7f47a2400640 1 --2- 172.21.15.97:0/766518791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0cda90 0x7f479c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.359+0000 7f47a1000640 1 -- 172.21.15.97:0/766518791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0cda90 msgr2=0x7f479c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:40.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.359+0000 7f47a1000640 1 --2- 172.21.15.97:0/766518791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0cda90 0x7f479c06c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.359+0000 7f47a1000640 1 -- 172.21.15.97:0/766518791 >> v1:172.21.15.97:6789/0 conn(0x7f479c0b3df0 legacy=0x7f479c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.359+0000 7f47a1000640 1 -- 172.21.15.97:0/766518791 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c071fa0 con 0x7f479c0b3650 2024-03-06T07:41:40.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.359+0000 7f479be00640 1 -- 172.21.15.97:0/766518791 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607295851 0 0) 0x7f479c0719e0 con 0x7f479c0b3df0 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a1000640 1 --2- 172.21.15.97:0/766518791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3650 0x7f479c0727a0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f4788009870 tx=0x7f478802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1d0b9a515940f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f479be00640 1 -- 172.21.15.97:0/766518791 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788034070 con 0x7f479c0b3650 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f479be00640 1 -- 172.21.15.97:0/766518791 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788030840 con 0x7f479c0b3650 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f479be00640 1 -- 172.21.15.97:0/766518791 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788030c10 con 0x7f479c0b3650 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a4400640 1 -- 172.21.15.97:0/766518791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3650 msgr2=0x7f479c0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a4400640 1 --2- 172.21.15.97:0/766518791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3650 0x7f479c0727a0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f4788009870 tx=0x7f478802b980 comp rx=0 tx=0).stop 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a4400640 1 -- 172.21.15.97:0/766518791 shutdown_connections 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a4400640 1 --2- 172.21.15.97:0/766518791 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0cda90 0x7f479c06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a4400640 1 --2- 172.21.15.97:0/766518791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3650 0x7f479c0727a0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.360+0000 7f47a4400640 1 -- 172.21.15.97:0/766518791 >> 172.21.15.97:0/766518791 conn(0x7f479c0b1940 msgr2=0x7f479c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.361+0000 7f47a4400640 1 -- 172.21.15.97:0/766518791 shutdown_connections 2024-03-06T07:41:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.361+0000 7f47a4400640 1 -- 172.21.15.97:0/766518791 wait complete. 2024-03-06T07:41:40.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.362+0000 7f47a4400640 1 Processor -- start 2024-03-06T07:41:40.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.362+0000 7f47a4400640 1 -- start start 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a4400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0b3650 0x7f479c0aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a4400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 0x7f479c146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a4400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f479c0cda90 0x7f479c147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a4400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c15b3c0 con 0x7f479c0b3df0 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a4400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c15b240 con 0x7f479c0cda90 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a4400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0cf510 con 0x7f479c0b3650 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 0x7f479c146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a1000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 0x7f479c146b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47372/0 (socket says 172.21.15.97:47372) 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.363+0000 7f47a1000640 1 -- 172.21.15.97:0/2959046818 learned_addr learned my addr 172.21.15.97:0/2959046818 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a1000640 1 -- 172.21.15.97:0/2959046818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0b3650 msgr2=0x7f479c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a1000640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0b3650 0x7f479c0aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a1000640 1 -- 172.21.15.97:0/2959046818 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f479c0cda90 msgr2=0x7f479c147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a1000640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f479c0cda90 0x7f479c147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a1000640 1 -- 172.21.15.97:0/2959046818 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0addc0 con 0x7f479c0b3df0 2024-03-06T07:41:40.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a1000640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 0x7f479c146b40 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f47880097c0 tx=0x7f47880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788034040 con 0x7f479c0b3df0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788039d90 con 0x7f479c0b3df0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c0af0d0 con 0x7f479c0b3df0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.364+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788043630 con 0x7f479c0b3df0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.365+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c147990 con 0x7f479c0b3df0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.366+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c04f330 con 0x7f479c0b3df0 2024-03-06T07:41:40.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.366+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f478804a060 con 0x7f479c0b3df0 2024-03-06T07:41:40.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.367+0000 7f4799600640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f477c06b4d0 0x7f477c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.367+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f47880ac1f0 con 0x7f479c0b3df0 2024-03-06T07:41:40.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.368+0000 7f47a1a00640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f477c06b4d0 0x7f477c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.368+0000 7f47a1a00640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f477c06b4d0 0x7f477c06d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4790001560 tx=0x7f4790007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:40.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.372+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f478803e070 con 0x7f479c0b3df0 2024-03-06T07:41:40.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.567+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f479c0cc0e0 con 0x7f479c0b3df0 2024-03-06T07:41:40.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.572+0000 7f4799600640 1 -- 172.21.15.97:0/2959046818 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f478802d330 con 0x7f479c0b3df0 2024-03-06T07:41:40.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f477c06b4d0 msgr2=0x7f477c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f477c06b4d0 0x7f477c06d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4790001560 tx=0x7f4790007930 comp rx=0 tx=0).stop 2024-03-06T07:41:40.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 msgr2=0x7f479c146b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 0x7f479c146b40 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f47880097c0 tx=0x7f47880046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 shutdown_connections 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f477c06b4d0 0x7f477c06d990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f479c0cda90 0x7f479c147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f479c0b3df0 0x7f479c146b40 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 --2- 172.21.15.97:0/2959046818 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f479c0b3650 0x7f479c0aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 >> 172.21.15.97:0/2959046818 conn(0x7f479c0b1940 msgr2=0x7f479c06d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 shutdown_connections 2024-03-06T07:41:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.575+0000 7f47a4400640 1 -- 172.21.15.97:0/2959046818 wait complete. 2024-03-06T07:41:40.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:41:40.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:40.797 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:40.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.804+0000 7f59e0800640 1 Processor -- start 2024-03-06T07:41:40.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.804+0000 7f59e0800640 1 -- start start 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59e0800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f59d815e230 0x7f59d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59e0800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d80d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59e0800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f59d80ce550 0x7f59d80ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59e0800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d815e610 con 0x7f59d80ce550 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59e0800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80cf340 con 0x7f59d815e230 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59e0800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80cfe60 con 0x7f59d80b4320 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59de800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f59d80ce550 0x7f59d80ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59de800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f59d80ce550 0x7f59d80ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47376/0 (socket says 172.21.15.97:47376) 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.805+0000 7f59de800640 1 -- 172.21.15.97:0/299836913 learned_addr learned my addr 172.21.15.97:0/299836913 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59dd400640 1 --2- 172.21.15.97:0/299836913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d80d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59dde00640 1 --2- 172.21.15.97:0/299836913 >> v2:172.21.15.135:3300/0 conn(0x7f59d815e230 0x7f59d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59de800640 1 -- 172.21.15.97:0/299836913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 msgr2=0x7f59d80d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59de800640 1 --2- 172.21.15.97:0/299836913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d80d0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59de800640 1 -- 172.21.15.97:0/299836913 >> v2:172.21.15.135:3300/0 conn(0x7f59d815e230 msgr2=0x7f59d806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59de800640 1 --2- 172.21.15.97:0/299836913 >> v2:172.21.15.135:3300/0 conn(0x7f59d815e230 0x7f59d806edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.806+0000 7f59de800640 1 -- 172.21.15.97:0/299836913 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d806f390 con 0x7f59d80ce550 2024-03-06T07:41:40.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.808+0000 7f59de800640 1 --2- 172.21.15.97:0/299836913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d80ce550 0x7f59d80ce9e0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f59d8072390 tx=0x7f59d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b660a03a2c856b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:40.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.808+0000 7f59dca00640 1 -- 172.21.15.97:0/299836913 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d402c9e0 con 0x7f59d80ce550 2024-03-06T07:41:40.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.808+0000 7f59dca00640 1 -- 172.21.15.97:0/299836913 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d402cb80 con 0x7f59d80ce550 2024-03-06T07:41:40.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.808+0000 7f59dca00640 1 -- 172.21.15.97:0/299836913 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40343f0 con 0x7f59d80ce550 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 -- 172.21.15.97:0/299836913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d80ce550 msgr2=0x7f59d80ce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 --2- 172.21.15.97:0/299836913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d80ce550 0x7f59d80ce9e0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f59d8072390 tx=0x7f59d402be40 comp rx=0 tx=0).stop 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 -- 172.21.15.97:0/299836913 shutdown_connections 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 --2- 172.21.15.97:0/299836913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d80ce550 0x7f59d80ce9e0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 --2- 172.21.15.97:0/299836913 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d80d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 --2- 172.21.15.97:0/299836913 >> v2:172.21.15.135:3300/0 conn(0x7f59d815e230 0x7f59d806edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 -- 172.21.15.97:0/299836913 >> 172.21.15.97:0/299836913 conn(0x7f59d80b3800 msgr2=0x7f59d80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:40.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 -- 172.21.15.97:0/299836913 shutdown_connections 2024-03-06T07:41:40.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.809+0000 7f59e0800640 1 -- 172.21.15.97:0/299836913 wait complete. 2024-03-06T07:41:40.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.811+0000 7f59e0800640 1 Processor -- start 2024-03-06T07:41:40.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.811+0000 7f59e0800640 1 -- start start 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59e0800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d8156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59e0800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80ce550 0x7f59d80b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59e0800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 0x7f59d81580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59e0800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80d06c0 con 0x7f59d815e230 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59e0800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80d03e0 con 0x7f59d80ce550 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59e0800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80d1120 con 0x7f59d80b4320 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59de800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 0x7f59d81580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59de800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 0x7f59d81580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47390/0 (socket says 172.21.15.97:47390) 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59de800640 1 -- 172.21.15.97:0/3741171002 learned_addr learned my addr 172.21.15.97:0/3741171002 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59dde00640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d8156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.812+0000 7f59dd400640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80ce550 0x7f59d80b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59de800640 1 -- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 msgr2=0x7f59d8156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59de800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d8156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59de800640 1 -- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80ce550 msgr2=0x7f59d80b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:40.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59de800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80ce550 0x7f59d80b6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:40.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59de800640 1 -- 172.21.15.97:0/3741171002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8156880 con 0x7f59d815e230 2024-03-06T07:41:40.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59de800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 0x7f59d81580d0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036040 tx=0x7f59d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4043020 con 0x7f59d815e230 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d403e450 con 0x7f59d815e230 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d8158b90 con 0x7f59d815e230 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4009510 con 0x7f59d815e230 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.813+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d80b5b30 con 0x7f59d815e230 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.815+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d804f330 con 0x7f59d815e230 2024-03-06T07:41:40.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.816+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59d4046020 con 0x7f59d815e230 2024-03-06T07:41:40.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.816+0000 7f59d2000640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59bc06b4d0 0x7f59bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:40.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.817+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f59d40ac700 con 0x7f59d815e230 2024-03-06T07:41:40.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.817+0000 7f59dde00640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59bc06b4d0 0x7f59bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:40.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.818+0000 7f59dde00640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59bc06b4d0 0x7f59bc06d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f59d81f75a0 tx=0x7f59cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:40.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:40.821+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59d404d030 con 0x7f59d815e230 2024-03-06T07:41:41.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.099+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7f59d80d1740 con 0x7f59d815e230 2024-03-06T07:41:41.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.128+0000 7f59d2000640 1 -- 172.21.15.97:0/3741171002 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v726) v1 ==== 141+0+0 (secure 0 0 0) 0x7f59d407b810 con 0x7f59d815e230 2024-03-06T07:41:41.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59bc06b4d0 msgr2=0x7f59bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:41.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59bc06b4d0 0x7f59bc06d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f59d81f75a0 tx=0x7f59cc008040 comp rx=0 tx=0).stop 2024-03-06T07:41:41.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 msgr2=0x7f59d81580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:41.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 0x7f59d81580d0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036040 tx=0x7f59d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 shutdown_connections 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f59bc06b4d0 0x7f59bc06d990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d815e230 0x7f59d81580d0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59d80ce550 0x7f59d80b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 --2- 172.21.15.97:0/3741171002 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f59d80b4320 0x7f59d8156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.132+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 >> 172.21.15.97:0/3741171002 conn(0x7f59d80b3800 msgr2=0x7f59d806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.133+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 shutdown_connections 2024-03-06T07:41:41.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:41.133+0000 7f59e0800640 1 -- 172.21.15.97:0/3741171002 wait complete. 2024-03-06T07:41:41.155 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays ... ERROR 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner: 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:41:41.156 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:Ran 9 tests in 800.945s 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:41:41.157 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:41:41.158 INFO:tasks.cephfs_test_runner: 2024-03-06T07:41:41.158 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:41:41.343 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a03aeca5d3a44ccab5acc938c0e009a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:41:41.344 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:41:41.353 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-06T07:41:41.367 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-06T07:41:41.368 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:41:41.368 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:41.430 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:41:41.430 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-06T07:41:41.430 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:41.484 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053427,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:41:41.484 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:41:41.484 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:41.539 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:41:41.539 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-06T07:41:41.539 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:41:41.588 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:41:41.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:41.588 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:41:41.644 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:41:41.644 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:41:41.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:41.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:41:41.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:41.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:41:41.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:41.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-03-06T07:41:41.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:41.946 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-06T07:41:41.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:41.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-03-06T07:41:42.071 INFO:teuthology.orchestra.run.smithi097.stderr:Cannot find device "ceph-brx" 2024-03-06T07:41:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:41:42.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:42.075 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-06T07:41:42.132 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:41:42.132 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:41:42.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:42.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:41:42.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:42.229 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:41:42.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:42.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link delete ceph-brx 2024-03-06T07:41:42.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:42.413 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-06T07:41:42.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:42.461 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link delete ceph-brx 2024-03-06T07:41:42.602 INFO:teuthology.orchestra.run.smithi135.stderr:Cannot find device "ceph-brx" 2024-03-06T07:41:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:41:42.607 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:41:42.617 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:41:42.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:41:42.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.830+0000 7f2c51400640 1 Processor -- start 2024-03-06T07:41:42.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.830+0000 7f2c51400640 1 -- start start 2024-03-06T07:41:42.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c51400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:42.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c51400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c4c15e0b0 0x7f2c4c06f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:42.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c51400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4c06e780 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c51400640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4c06d770 con 0x7f2c4c15e0b0 2024-03-06T07:41:42.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c51400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c06ee50 con 0x7f2c4c15dcd0 2024-03-06T07:41:42.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c4b400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c4c15e0b0 0x7f2c4c06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c50a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2c4c0b39d0 0x7f2c4c161b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37122/0 (socket says 172.21.15.97:37122) 2024-03-06T07:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.832+0000 7f2c50a00640 1 -- 172.21.15.97:0/2590253031 learned_addr learned my addr 172.21.15.97:0/2590253031 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.833+0000 7f2c4be00640 1 --2- 172.21.15.97:0/2590253031 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.833+0000 7f2c4b400640 1 -- 172.21.15.97:0/2590253031 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15dcd0 msgr2=0x7f2c4c0b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.833+0000 7f2c4b400640 1 --2- 172.21.15.97:0/2590253031 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.833+0000 7f2c4b400640 1 -- 172.21.15.97:0/2590253031 >> v1:172.21.15.97:6789/0 conn(0x7f2c4c0b39d0 legacy=0x7f2c4c161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:42.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.833+0000 7f2c4b400640 1 -- 172.21.15.97:0/2590253031 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c06d9c0 con 0x7f2c4c15e0b0 2024-03-06T07:41:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.834+0000 7f2c4b400640 1 --2- 172.21.15.97:0/2590253031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c15e0b0 0x7f2c4c06f290 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009790 tx=0x7f2c3802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9961058b7be465b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.834+0000 7f2c4aa00640 1 -- 172.21.15.97:0/2590253031 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38031780 con 0x7f2c4c15e0b0 2024-03-06T07:41:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.834+0000 7f2c4aa00640 1 -- 172.21.15.97:0/2590253031 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c38037070 con 0x7f2c4c15e0b0 2024-03-06T07:41:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.834+0000 7f2c4aa00640 1 -- 172.21.15.97:0/2590253031 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38031b40 con 0x7f2c4c15e0b0 2024-03-06T07:41:42.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 -- 172.21.15.97:0/2590253031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c15e0b0 msgr2=0x7f2c4c06f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 --2- 172.21.15.97:0/2590253031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c15e0b0 0x7f2c4c06f290 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009790 tx=0x7f2c3802b820 comp rx=0 tx=0).stop 2024-03-06T07:41:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 -- 172.21.15.97:0/2590253031 shutdown_connections 2024-03-06T07:41:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 --2- 172.21.15.97:0/2590253031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c15e0b0 0x7f2c4c06f290 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 --2- 172.21.15.97:0/2590253031 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 -- 172.21.15.97:0/2590253031 >> 172.21.15.97:0/2590253031 conn(0x7f2c4c06c0d0 msgr2=0x7f2c4c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 -- 172.21.15.97:0/2590253031 shutdown_connections 2024-03-06T07:41:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.835+0000 7f2c51400640 1 -- 172.21.15.97:0/2590253031 wait complete. 2024-03-06T07:41:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.837+0000 7f2c51400640 1 Processor -- start 2024-03-06T07:41:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.837+0000 7f2c51400640 1 -- start start 2024-03-06T07:41:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c51400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 0x7f2c4c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c51400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c51400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15e0b0 0x7f2c4c1f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c51400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c06e260 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c51400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c06df80 con 0x7f2c4c15dcd0 2024-03-06T07:41:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c51400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c0d16f0 con 0x7f2c4c15e0b0 2024-03-06T07:41:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 0x7f2c4c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 0x7f2c4c15c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47412/0 (socket says 172.21.15.97:47412) 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 -- 172.21.15.97:0/3171708536 learned_addr learned my addr 172.21.15.97:0/3171708536 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4b400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 -- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15e0b0 msgr2=0x7f2c4c1f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15e0b0 0x7f2c4c1f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 -- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c4c15dcd0 msgr2=0x7f2c4c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.838+0000 7f2c4be00640 1 -- 172.21.15.97:0/3171708536 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c1f45d0 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.839+0000 7f2c4be00640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 0x7f2c4c15c4f0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00d5f0 tx=0x7f2c3c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.839+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0125f0 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.839+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c019070 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.839+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0129d0 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.839+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c1f48b0 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.839+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c0b0550 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.840+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c1f4a80 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.841+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c3c012b70 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.842+0000 7f2c43e00640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:42.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.842+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2c3c08a410 con 0x7f2c4c0b39d0 2024-03-06T07:41:42.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.842+0000 7f2c4b400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:42.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.843+0000 7f2c4b400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2c38036000 tx=0x7f2c38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:42.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:42.847+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c3c059490 con 0x7f2c4c0b39d0 2024-03-06T07:41:43.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.041+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c4c04f330 con 0x7f2c2c06b4d0 2024-03-06T07:41:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.043+0000 7f2c43e00640 1 -- 172.21.15.97:0/3171708536 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13510 (secure 0 0 0) 0x7f2c4c04f330 con 0x7f2c2c06b4d0 2024-03-06T07:41:43.046 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:41:43.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c2c06b4d0 msgr2=0x7f2c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2c38036000 tx=0x7f2c38008030 comp rx=0 tx=0).stop 2024-03-06T07:41:43.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 msgr2=0x7f2c4c15c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 0x7f2c4c15c4f0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00d5f0 tx=0x7f2c3c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 shutdown_connections 2024-03-06T07:41:43.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2c4c15e0b0 0x7f2c4c1f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c4c15dcd0 0x7f2c4c0b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 --2- 172.21.15.97:0/3171708536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c4c0b39d0 0x7f2c4c15c4f0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.046+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 >> 172.21.15.97:0/3171708536 conn(0x7f2c4c06c0d0 msgr2=0x7f2c4c159560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:43.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.047+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 shutdown_connections 2024-03-06T07:41:43.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.047+0000 7f2c51400640 1 -- 172.21.15.97:0/3171708536 wait complete. 2024-03-06T07:41:43.068 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2024-03-06T07:41:41.172949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311048,"kb_used_data":15848,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754663672,"statfs":{"total":773094113280,"available":772775600128,"internally_reserved":0,"allocated":16228352,"data_stored":7780082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33881,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001732"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":407,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:41:37.530900+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:41:37.530900+0000","last_peered":"2024-03-06T07:41:37.530900+0000","last_clean":"2024-03-06T07:41:37.530900+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:41:37.530900+0000","last_undegraded":"2024-03-06T07:41:37.530900+0000","last_fullsized":"2024-03-06T07:41:37.530900+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542315,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37764,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334076,"statfs":{"total":96636764160,"available":96598093824,"internally_reserved":0,"allocated":2482176,"data_stored":1419262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542316,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":817176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1863680,"data_stored":816470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542316,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41316,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330524,"statfs":{"total":96636764160,"available":96594456576,"internally_reserved":0,"allocated":1859584,"data_stored":816448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":817300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542317,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1912832,"data_stored":848839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41328,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330512,"statfs":{"total":96636764160,"available":96594444288,"internally_reserved":0,"allocated":1871872,"data_stored":817142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542316,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":2486272,"data_stored":1427445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:41:43.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:41:43.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.270+0000 7ff316400640 1 Processor -- start 2024-03-06T07:41:43.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.270+0000 7ff316400640 1 -- start start 2024-03-06T07:41:43.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff316400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 0x7ff31015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff316400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff310070b10 0x7ff310070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff316400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31006cef0 con 0x7ff310144c00 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff316400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff310160370 con 0x7ff310070b10 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff316400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31006d1d0 con 0x7ff31015f9d0 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff30f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff310144c00 0x7ff31006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37134/0 (socket says 172.21.15.97:37134) 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.272+0000 7ff30f400640 1 -- 172.21.15.97:0/1106175401 learned_addr learned my addr 172.21.15.97:0/1106175401 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff307e00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 0x7ff31015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30fe00640 1 --2- 172.21.15.97:0/1106175401 >> v2:172.21.15.135:3300/0 conn(0x7ff310070b10 0x7ff310070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679504035 0 0) 0x7ff31006cef0 con 0x7ff310144c00 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f4003520 con 0x7ff310144c00 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295288386 0 0) 0x7ff2f4003520 con 0x7ff310144c00 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 msgr2=0x7ff31015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 0x7ff31015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 >> v2:172.21.15.135:3300/0 conn(0x7ff310070b10 msgr2=0x7ff310070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 --2- 172.21.15.97:0/1106175401 >> v2:172.21.15.135:3300/0 conn(0x7ff310070b10 0x7ff310070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31006d4b0 con 0x7ff310144c00 2024-03-06T07:41:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7ff300001670 con 0x7ff310144c00 2024-03-06T07:41:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 >> v1:172.21.15.97:6789/0 conn(0x7ff310144c00 legacy=0x7ff31006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.273+0000 7ff30ea00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f40039b0 0x7ff2f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30ea00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 0x7ff2f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30ea00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f4004d40 0x7ff2f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31006d1d0 con 0x7ff2f4004d40 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f400b9c0 con 0x7ff2f40039b0 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f400bb40 con 0x7ff31015f9d0 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30fe00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f4004d40 0x7ff2f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30f400640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f40039b0 0x7ff2f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff307e00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 0x7ff2f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30fe00640 1 -- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 msgr2=0x7ff2f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:43.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30fe00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff31015f9d0 0x7ff2f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30fe00640 1 -- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f40039b0 msgr2=0x7ff2f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:43.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30fe00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f40039b0 0x7ff2f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30fe00640 1 -- 172.21.15.97:0/1106175401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4003520 con 0x7ff2f4004d40 2024-03-06T07:41:43.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.274+0000 7ff30f400640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2f40039b0 0x7ff2f4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:41:43.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.276+0000 7ff30fe00640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f4004d40 0x7ff2f40051c0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff30800e170 tx=0x7ff2f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e2d67ba955e678 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:43.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.276+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308039070 con 0x7ff2f4004d40 2024-03-06T07:41:43.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.276+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30802cc20 con 0x7ff2f4004d40 2024-03-06T07:41:43.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.276+0000 7ff30ea00640 1 -- 172.21.15.97:0/1106175401 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308009c10 con 0x7ff2f4004d40 2024-03-06T07:41:43.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff316400640 1 -- 172.21.15.97:0/1106175401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f4004d40 msgr2=0x7ff2f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff316400640 1 --2- 172.21.15.97:0/1106175401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2f4004d40 0x7ff2f40051c0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff30800e170 tx=0x7ff2f4005830 comp rx=0 tx=0).stop 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff30fe00640 1 -- 172.21.15.97:0/1106175401 reap_dead start 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff316400640 1 -- 172.21.15.97:0/1106175401 shutdown_connections 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff316400640 1 -- 172.21.15.97:0/1106175401 >> 172.21.15.97:0/1106175401 conn(0x7ff3100b1730 msgr2=0x7ff3100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff316400640 1 -- 172.21.15.97:0/1106175401 shutdown_connections 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.277+0000 7ff316400640 1 -- 172.21.15.97:0/1106175401 wait complete. 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.279+0000 7ff316400640 1 Processor -- start 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.279+0000 7ff316400640 1 -- start start 2024-03-06T07:41:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff316400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 0x7ff3101425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff316400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff310142b30 0x7ff31015d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff316400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff31015dc10 0x7ff3101f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff316400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31006cb50 con 0x7ff31015b600 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff316400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff310160730 con 0x7ff31015dc10 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff316400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31006ccd0 con 0x7ff310142b30 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff30f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 0x7ff3101425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff30f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 0x7ff3101425f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47430/0 (socket says 172.21.15.97:47430) 2024-03-06T07:41:43.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff30f400640 1 -- 172.21.15.97:0/3122346513 learned_addr learned my addr 172.21.15.97:0/3122346513 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff307e00640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff310142b30 0x7ff31015d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.280+0000 7ff30fe00640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff31015dc10 0x7ff3101f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff30f400640 1 -- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff310142b30 msgr2=0x7ff31015d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff30f400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff310142b30 0x7ff31015d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff30f400640 1 -- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff31015dc10 msgr2=0x7ff3101f7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff30f400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff31015dc10 0x7ff3101f7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff30f400640 1 -- 172.21.15.97:0/3122346513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31015ece0 con 0x7ff31015b600 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff30f400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 0x7ff3101425f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7ff30000e8a0 tx=0x7ff30000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:43.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30000c870 con 0x7ff31015b600 2024-03-06T07:41:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300018070 con 0x7ff31015b600 2024-03-06T07:41:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31013fe40 con 0x7ff31015b600 2024-03-06T07:41:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30000cc70 con 0x7ff31015b600 2024-03-06T07:41:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.281+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31015e720 con 0x7ff31015b600 2024-03-06T07:41:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.282+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31004f330 con 0x7ff31015b600 2024-03-06T07:41:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.283+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff300013420 con 0x7ff31015b600 2024-03-06T07:41:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.284+0000 7ff307400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2ec06b500 0x7ff2ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.284+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff300089bb0 con 0x7ff31015b600 2024-03-06T07:41:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.284+0000 7ff307e00640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2ec06b500 0x7ff2ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.285+0000 7ff307e00640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2ec06b500 0x7ff2ec06d9c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff3101f72c0 tx=0x7ff2f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:43.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.289+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff300058cc0 con 0x7ff31015b600 2024-03-06T07:41:43.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.485+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff31015be20 con 0x7ff2ec06b500 2024-03-06T07:41:43.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.487+0000 7ff307400640 1 -- 172.21.15.97:0/3122346513 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13510 (secure 0 0 0) 0x7ff31015be20 con 0x7ff2ec06b500 2024-03-06T07:41:43.490 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:43.490 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:41:43.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2ec06b500 msgr2=0x7ff2ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2ec06b500 0x7ff2ec06d9c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff3101f72c0 tx=0x7ff2f8008040 comp rx=0 tx=0).stop 2024-03-06T07:41:43.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 msgr2=0x7ff3101425f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 0x7ff3101425f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7ff30000e8a0 tx=0x7ff30000ed60 comp rx=0 tx=0).stop 2024-03-06T07:41:43.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 shutdown_connections 2024-03-06T07:41:43.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7ff2ec06b500 0x7ff2ec06d9c0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff31015dc10 0x7ff3101f7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7ff310142b30 0x7ff31015d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 --2- 172.21.15.97:0/3122346513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff31015b600 0x7ff3101425f0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 >> 172.21.15.97:0/3122346513 conn(0x7ff3100b1730 msgr2=0x7ff31006e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:43.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 shutdown_connections 2024-03-06T07:41:43.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.489+0000 7ff316400640 1 -- 172.21.15.97:0/3122346513 wait complete. 2024-03-06T07:41:43.510 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2024-03-06T07:41:43.173517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311048,"kb_used_data":15848,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754663672,"statfs":{"total":773094113280,"available":772775600128,"internally_reserved":0,"allocated":16228352,"data_stored":7780082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33881,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.068603"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":407,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:41:37.530900+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:41:37.530900+0000","last_peered":"2024-03-06T07:41:37.530900+0000","last_clean":"2024-03-06T07:41:37.530900+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:41:37.530900+0000","last_undegraded":"2024-03-06T07:41:37.530900+0000","last_fullsized":"2024-03-06T07:41:37.530900+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542315,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37764,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334076,"statfs":{"total":96636764160,"available":96598093824,"internally_reserved":0,"allocated":2482176,"data_stored":1419262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542316,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":817176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1863680,"data_stored":816470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542316,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41316,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330524,"statfs":{"total":96636764160,"available":96594456576,"internally_reserved":0,"allocated":1859584,"data_stored":816448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":817300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542317,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1912832,"data_stored":848839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41328,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330512,"statfs":{"total":96636764160,"available":96594444288,"internally_reserved":0,"allocated":1871872,"data_stored":817142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542316,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":2486272,"data_stored":1427445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:41:43.511 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:41:43.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:41:43.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.707+0000 7f88f9200640 1 Processor -- start 2024-03-06T07:41:43.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.707+0000 7f88f9200640 1 -- start start 2024-03-06T07:41:43.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f9200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f415f9d0 0x7f88f415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f9200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f88f4070b10 0x7f88f4070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f9200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f406cef0 con 0x7f88f4144c00 2024-03-06T07:41:43.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f9200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f4160370 con 0x7f88f4070b10 2024-03-06T07:41:43.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f9200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f406d1d0 con 0x7f88f415f9d0 2024-03-06T07:41:43.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f88f4070b10 0x7f88f4070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f88f4070b10 0x7f88f4070f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47442/0 (socket says 172.21.15.97:47442) 2024-03-06T07:41:43.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 -- 172.21.15.97:0/3475211328 learned_addr learned my addr 172.21.15.97:0/3475211328 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:43.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f1400640 1 --2- 172.21.15.97:0/3475211328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f415f9d0 0x7f88f415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 -- 172.21.15.97:0/3475211328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f415f9d0 msgr2=0x7f88f415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:43.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 --2- 172.21.15.97:0/3475211328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f415f9d0 0x7f88f415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 -- 172.21.15.97:0/3475211328 >> v1:172.21.15.135:6789/0 conn(0x7f88f4144c00 legacy=0x7f88f406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:43.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.709+0000 7f88f2800640 1 -- 172.21.15.97:0/3475211328 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f406d4b0 con 0x7f88f4070b10 2024-03-06T07:41:43.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.711+0000 7f88f2800640 1 --2- 172.21.15.97:0/3475211328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4070f60 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009e50 tx=0x7f88ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee6b41d9cdef037 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:43.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.711+0000 7f88f0a00640 1 -- 172.21.15.97:0/3475211328 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec02ca70 con 0x7f88f4070b10 2024-03-06T07:41:43.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.711+0000 7f88f0a00640 1 -- 172.21.15.97:0/3475211328 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec02cc10 con 0x7f88f4070b10 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.711+0000 7f88f0a00640 1 -- 172.21.15.97:0/3475211328 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0343f0 con 0x7f88f4070b10 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.711+0000 7f88f9200640 1 -- 172.21.15.97:0/3475211328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 msgr2=0x7f88f4070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.711+0000 7f88f9200640 1 --2- 172.21.15.97:0/3475211328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4070f60 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009e50 tx=0x7f88ec02bed0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.712+0000 7f88f9200640 1 -- 172.21.15.97:0/3475211328 shutdown_connections 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.712+0000 7f88f9200640 1 --2- 172.21.15.97:0/3475211328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4070f60 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.712+0000 7f88f9200640 1 --2- 172.21.15.97:0/3475211328 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f415f9d0 0x7f88f415fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.712+0000 7f88f9200640 1 -- 172.21.15.97:0/3475211328 >> 172.21.15.97:0/3475211328 conn(0x7f88f40b1730 msgr2=0x7f88f40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.712+0000 7f88f9200640 1 -- 172.21.15.97:0/3475211328 shutdown_connections 2024-03-06T07:41:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.712+0000 7f88f9200640 1 -- 172.21.15.97:0/3475211328 wait complete. 2024-03-06T07:41:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.713+0000 7f88f9200640 1 Processor -- start 2024-03-06T07:41:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.713+0000 7f88f9200640 1 -- start start 2024-03-06T07:41:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f9200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f4144c00 0x7f88f413c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f9200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88f415f9d0 0x7f88f413cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f9200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f406cb50 con 0x7f88f4070b10 2024-03-06T07:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f9200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f4160730 con 0x7f88f415f9d0 2024-03-06T07:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f9200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f406ccd0 con 0x7f88f4144c00 2024-03-06T07:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.714+0000 7f88f1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47448/0 (socket says 172.21.15.97:47448) 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 -- 172.21.15.97:0/2457637679 learned_addr learned my addr 172.21.15.97:0/2457637679 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 -- 172.21.15.97:0/2457637679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f4144c00 msgr2=0x7f88f413c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f4144c00 0x7f88f413c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 -- 172.21.15.97:0/2457637679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88f415f9d0 msgr2=0x7f88f413cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88f415f9d0 0x7f88f413cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 -- 172.21.15.97:0/2457637679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f414ce40 con 0x7f88f4070b10 2024-03-06T07:41:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f1e00640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4139470 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f88e400c430 tx=0x7f88e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:43.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4011790 con 0x7f88f4070b10 2024-03-06T07:41:43.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4017070 con 0x7f88f4070b10 2024-03-06T07:41:43.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4011bb0 con 0x7f88f4070b10 2024-03-06T07:41:43.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.715+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f414d120 con 0x7f88f4070b10 2024-03-06T07:41:43.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.716+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f414d6e0 con 0x7f88f4070b10 2024-03-06T07:41:43.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.717+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f404f330 con 0x7f88f4070b10 2024-03-06T07:41:43.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.718+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88e401c460 con 0x7f88f4070b10 2024-03-06T07:41:43.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.718+0000 7f88ea000640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88d406b4d0 0x7f88d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:43.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.719+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f88e4089f60 con 0x7f88f4070b10 2024-03-06T07:41:43.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.719+0000 7f88f1400640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88d406b4d0 0x7f88d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:43.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.719+0000 7f88f1400640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88d406b4d0 0x7f88d406d990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f88f415e500 tx=0x7f88e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:43.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.723+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88e4058f50 con 0x7f88f4070b10 2024-03-06T07:41:43.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.915+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88f4072890 con 0x7f88d406b4d0 2024-03-06T07:41:43.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.916+0000 7f88ea000640 1 -- 172.21.15.97:0/2457637679 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13510 (secure 0 0 0) 0x7f88f4072890 con 0x7f88d406b4d0 2024-03-06T07:41:43.919 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:43.919 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:41:43.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.918+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88d406b4d0 msgr2=0x7f88d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.918+0000 7f88f9200640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88d406b4d0 0x7f88d406d990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f88f415e500 tx=0x7f88e0008040 comp rx=0 tx=0).stop 2024-03-06T07:41:43.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.918+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 msgr2=0x7f88f4139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:43.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.918+0000 7f88f9200640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4139470 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f88e400c430 tx=0x7f88e400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 shutdown_connections 2024-03-06T07:41:43.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f88d406b4d0 0x7f88d406d990 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88f415f9d0 0x7f88f413cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f88f4144c00 0x7f88f413c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 --2- 172.21.15.97:0/2457637679 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88f4070b10 0x7f88f4139470 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:43.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 >> 172.21.15.97:0/2457637679 conn(0x7f88f40b1730 msgr2=0x7f88f4142c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:43.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 shutdown_connections 2024-03-06T07:41:43.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:43.919+0000 7f88f9200640 1 -- 172.21.15.97:0/2457637679 wait complete. 2024-03-06T07:41:43.940 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2024-03-06T07:41:43.173517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311048,"kb_used_data":15848,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754663672,"statfs":{"total":773094113280,"available":772775600128,"internally_reserved":0,"allocated":16228352,"data_stored":7780082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33881,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.068603"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":407,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:41:37.530900+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:41:37.530900+0000","last_peered":"2024-03-06T07:41:37.530900+0000","last_clean":"2024-03-06T07:41:37.530900+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:41:37.530900+0000","last_undegraded":"2024-03-06T07:41:37.530900+0000","last_fullsized":"2024-03-06T07:41:37.530900+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542315,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37764,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334076,"statfs":{"total":96636764160,"available":96598093824,"internally_reserved":0,"allocated":2482176,"data_stored":1419262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542316,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":817176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1863680,"data_stored":816470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542316,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41316,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330524,"statfs":{"total":96636764160,"available":96594456576,"internally_reserved":0,"allocated":1859584,"data_stored":816448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":817300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542317,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1912832,"data_stored":848839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41328,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330512,"statfs":{"total":96636764160,"available":96594444288,"internally_reserved":0,"allocated":1871872,"data_stored":817142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542316,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":2486272,"data_stored":1427445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:41:43.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:41:44.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.135+0000 7fc71ac00640 1 Processor -- start 2024-03-06T07:41:44.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.136+0000 7fc71ac00640 1 -- start start 2024-03-06T07:41:44.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.137+0000 7fc71ac00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc71415f770 0x7fc71415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:44.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.137+0000 7fc71ac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7140b4390 0x7fc714160290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:44.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.137+0000 7fc71ac00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140b3040 con 0x7fc7141607d0 2024-03-06T07:41:44.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.137+0000 7fc71ac00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71415fb50 con 0x7fc71415f770 2024-03-06T07:41:44.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.137+0000 7fc71ac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140cd3e0 con 0x7fc7140b4390 2024-03-06T07:41:44.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc71415f770 0x7fc71415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:44.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc70b400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7140b4390 0x7fc714160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:44.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc71415f770 0x7fc71415fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:42054/0 (socket says 172.21.15.97:42054) 2024-03-06T07:41:44.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 -- 172.21.15.97:0/724885388 learned_addr learned my addr 172.21.15.97:0/724885388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:44.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc718c00640 1 --1- 172.21.15.97:0/724885388 >> v1:172.21.15.135:6789/0 conn(0x7fc7141607d0 0x7fc7140ce560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:55894/0 (socket says 172.21.15.97:55894) 2024-03-06T07:41:44.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 -- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7140b4390 msgr2=0x7fc714160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:44.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 --2- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7140b4390 0x7fc714160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 -- 172.21.15.97:0/724885388 >> v1:172.21.15.135:6789/0 conn(0x7fc7141607d0 legacy=0x7fc7140ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:44.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.138+0000 7fc713e00640 1 -- 172.21.15.97:0/724885388 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140cd6c0 con 0x7fc71415f770 2024-03-06T07:41:44.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.140+0000 7fc713e00640 1 --2- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc71415fd50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc704009870 tx=0x7fc70402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a09e8fd99557d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:44.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.140+0000 7fc713400640 1 -- 172.21.15.97:0/724885388 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704035020 con 0x7fc71415f770 2024-03-06T07:41:44.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.140+0000 7fc713400640 1 -- 172.21.15.97:0/724885388 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704031640 con 0x7fc71415f770 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.140+0000 7fc713400640 1 -- 172.21.15.97:0/724885388 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70402d040 con 0x7fc71415f770 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 -- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 msgr2=0x7fc71415fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 --2- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc71415fd50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc704009870 tx=0x7fc70402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 -- 172.21.15.97:0/724885388 shutdown_connections 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 --2- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7140b4390 0x7fc714160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 --2- 172.21.15.97:0/724885388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc71415fd50 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 -- 172.21.15.97:0/724885388 >> 172.21.15.97:0/724885388 conn(0x7fc7140b3870 msgr2=0x7fc7140b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:44.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 -- 172.21.15.97:0/724885388 shutdown_connections 2024-03-06T07:41:44.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.141+0000 7fc71ac00640 1 -- 172.21.15.97:0/724885388 wait complete. 2024-03-06T07:41:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.143+0000 7fc71ac00640 1 Processor -- start 2024-03-06T07:41:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.143+0000 7fc71ac00640 1 -- start start 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc71ac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 0x7fc7141f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc71ac00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc7140c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc71ac00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7141607d0 0x7fc7140c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc71ac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140b34a0 con 0x7fc7140b4390 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc71ac00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140b3320 con 0x7fc71415f770 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc71ac00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140d09b0 con 0x7fc7141607d0 2024-03-06T07:41:44.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 0x7fc7141f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 0x7fc7141f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44822/0 (socket says 172.21.15.97:44822) 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 -- 172.21.15.97:0/1380834195 learned_addr learned my addr 172.21.15.97:0/1380834195 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc70b400640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc7140c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc718c00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7141607d0 0x7fc7140c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 -- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7141607d0 msgr2=0x7fc7140c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7141607d0 0x7fc7140c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 -- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 msgr2=0x7fc7140c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc7140c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.144+0000 7fc713e00640 1 -- 172.21.15.97:0/1380834195 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc714155030 con 0x7fc7140b4390 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.145+0000 7fc713e00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 0x7fc7141f3ed0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fc704000c00 tx=0x7fc7040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.145+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704040070 con 0x7fc7140b4390 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.145+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7040399f0 con 0x7fc7140b4390 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.145+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704035040 con 0x7fc7140b4390 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.145+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc714143be0 con 0x7fc7140b4390 2024-03-06T07:41:44.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.145+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7141441a0 con 0x7fc7140b4390 2024-03-06T07:41:44.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.146+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71404f330 con 0x7fc7140b4390 2024-03-06T07:41:44.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.147+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc70404b060 con 0x7fc7140b4390 2024-03-06T07:41:44.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.148+0000 7fc710c00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6f006b4d0 0x7fc6f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:44.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.148+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc7040ab810 con 0x7fc7140b4390 2024-03-06T07:41:44.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.148+0000 7fc70b400640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6f006b4d0 0x7fc6f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:44.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.149+0000 7fc70b400640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6f006b4d0 0x7fc6f006d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc7140cbff0 tx=0x7fc6fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.152+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc70407a920 con 0x7fc7140b4390 2024-03-06T07:41:44.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.347+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7140ce130 con 0x7fc7140b4390 2024-03-06T07:41:44.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.349+0000 7fc710c00640 1 -- 172.21.15.97:0/1380834195 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+13148 (secure 0 0 0) 0x7fc70407a220 con 0x7fc7140b4390 2024-03-06T07:41:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:44.352 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":151,"fsid":"0b9abceb-2115-4321-96a2-348db0d0fbe4","created":"2024-03-06T07:27:21.773977+0000","modified":"2024-03-06T07:41:40.116970+0000","last_up_change":"2024-03-06T07:27:33.865196+0000","last_in_change":"2024-03-06T07:27:25.350454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:34.971154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f9b9c89-55e8-4987-838d-ff30970a00b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6819","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6821","nonce":1306424436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6827","nonce":1306424436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6831","nonce":1306424436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1306424436},{"type":"v1","addr":"172.21.15.97:6829","nonce":1306424436}]},"public_addr":"172.21.15.97:6821/1306424436","cluster_addr":"172.21.15.97:6827/1306424436","heartbeat_back_addr":"172.21.15.97:6831/1306424436","heartbeat_front_addr":"172.21.15.97:6829/1306424436","state":["exists","up"]},{"osd":1,"uuid":"49ce5f2a-0a8b-4315-8dab-e8cba83ffcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6809","nonce":2322813614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6811","nonce":2322813614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6815","nonce":2322813614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2322813614},{"type":"v1","addr":"172.21.15.97:6813","nonce":2322813614}]},"public_addr":"172.21.15.97:6809/2322813614","cluster_addr":"172.21.15.97:6811/2322813614","heartbeat_back_addr":"172.21.15.97:6815/2322813614","heartbeat_front_addr":"172.21.15.97:6813/2322813614","state":["exists","up"]},{"osd":2,"uuid":"9ec347a4-1598-4f0c-825f-6a5f130994d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6817","nonce":75411953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6820","nonce":75411953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6826","nonce":75411953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":75411953},{"type":"v1","addr":"172.21.15.97:6823","nonce":75411953}]},"public_addr":"172.21.15.97:6817/75411953","cluster_addr":"172.21.15.97:6820/75411953","heartbeat_back_addr":"172.21.15.97:6826/75411953","heartbeat_front_addr":"172.21.15.97:6823/75411953","state":["exists","up"]},{"osd":3,"uuid":"e2eeb450-3ac2-4a2f-89c2-9d82c39f094f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6801","nonce":3169996968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6803","nonce":3169996968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6807","nonce":3169996968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3169996968},{"type":"v1","addr":"172.21.15.97:6805","nonce":3169996968}]},"public_addr":"172.21.15.97:6801/3169996968","cluster_addr":"172.21.15.97:6803/3169996968","heartbeat_back_addr":"172.21.15.97:6807/3169996968","heartbeat_front_addr":"172.21.15.97:6805/3169996968","state":["exists","up"]},{"osd":4,"uuid":"80379238-cfbc-4c68-85b8-d71da22e550b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6809","nonce":3314712387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6811","nonce":3314712387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6815","nonce":3314712387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":3314712387},{"type":"v1","addr":"172.21.15.135:6813","nonce":3314712387}]},"public_addr":"172.21.15.135:6809/3314712387","cluster_addr":"172.21.15.135:6811/3314712387","heartbeat_back_addr":"172.21.15.135:6815/3314712387","heartbeat_front_addr":"172.21.15.135:6813/3314712387","state":["exists","up"]},{"osd":5,"uuid":"2ff0b5f7-e87d-411b-9a5a-79041279f3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6801","nonce":2374976391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6803","nonce":2374976391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6807","nonce":2374976391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2374976391},{"type":"v1","addr":"172.21.15.135:6805","nonce":2374976391}]},"public_addr":"172.21.15.135:6801/2374976391","cluster_addr":"172.21.15.135:6803/2374976391","heartbeat_back_addr":"172.21.15.135:6807/2374976391","heartbeat_front_addr":"172.21.15.135:6805/2374976391","state":["exists","up"]},{"osd":6,"uuid":"6b6b33d9-7d8a-4717-aae5-22e93e0c815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6825","nonce":3958261964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6827","nonce":3958261964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6831","nonce":3958261964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3958261964},{"type":"v1","addr":"172.21.15.135:6829","nonce":3958261964}]},"public_addr":"172.21.15.135:6825/3958261964","cluster_addr":"172.21.15.135:6827/3958261964","heartbeat_back_addr":"172.21.15.135:6831/3958261964","heartbeat_front_addr":"172.21.15.135:6829/3958261964","state":["exists","up"]},{"osd":7,"uuid":"ff93b254-4b0f-4580-b2e8-cac17574c5ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6817","nonce":3893802268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6819","nonce":3893802268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6823","nonce":3893802268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3893802268},{"type":"v1","addr":"172.21.15.135:6821","nonce":3893802268}]},"public_addr":"172.21.15.135:6817/3893802268","cluster_addr":"172.21.15.135:6819/3893802268","heartbeat_back_addr":"172.21.15.135:6823/3893802268","heartbeat_front_addr":"172.21.15.135:6821/3893802268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.613704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.911534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.616109+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.844041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.876481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.589920+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.622191+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:30.902406+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:6834/4190861064":"2024-03-07T07:41:37.098286+0000","172.21.15.97:6837/1171184135":"2024-03-07T07:41:37.098258+0000","172.21.15.97:6835/4190861064":"2024-03-07T07:41:37.098286+0000","172.21.15.97:6836/1171184135":"2024-03-07T07:41:37.098258+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:41:44.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6f006b4d0 msgr2=0x7fc6f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6f006b4d0 0x7fc6f006d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc7140cbff0 tx=0x7fc6fc008040 comp rx=0 tx=0).stop 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 msgr2=0x7fc7141f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 0x7fc7141f3ed0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fc704000c00 tx=0x7fc7040046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 shutdown_connections 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fc6f006b4d0 0x7fc6f006d990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fc7141607d0 0x7fc7140c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc71415f770 0x7fc7140c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 --2- 172.21.15.97:0/1380834195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7140b4390 0x7fc7141f3ed0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:44.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 >> 172.21.15.97:0/1380834195 conn(0x7fc7140b3870 msgr2=0x7fc71406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:44.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 shutdown_connections 2024-03-06T07:41:44.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:44.352+0000 7fc71ac00640 1 -- 172.21.15.97:0/1380834195 wait complete. 2024-03-06T07:41:45.373 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:41:45.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:45.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.597+0000 7f9e6e800640 1 Processor -- start 2024-03-06T07:41:45.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.598+0000 7f9e6e800640 1 -- start start 2024-03-06T07:41:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.599+0000 7f9e6e800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9e680716f0 0x7f9e6814bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.599+0000 7f9e6e800640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6814bbb0 con 0x7f9e6814b7d0 2024-03-06T07:41:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.599+0000 7f9e6e800640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e68071d40 con 0x7f9e6806d110 2024-03-06T07:41:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.599+0000 7f9e6e800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e680725e0 con 0x7f9e680716f0 2024-03-06T07:41:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.599+0000 7f9e67400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9e6814b7d0 0x7f9e6806ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38726/0 (socket says 172.21.15.97:38726) 2024-03-06T07:41:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.599+0000 7f9e67400640 1 -- 172.21.15.97:0/2066322837 learned_addr learned my addr 172.21.15.97:0/2066322837 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e67e00640 1 --2- 172.21.15.97:0/2066322837 >> v2:172.21.15.135:3300/0 conn(0x7f9e680716f0 0x7f9e6814bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e66a00640 1 -- 172.21.15.97:0/2066322837 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333574327 0 0) 0x7f9e6814bbb0 con 0x7f9e6814b7d0 2024-03-06T07:41:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e66a00640 1 -- 172.21.15.97:0/2066322837 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4c003520 con 0x7f9e6814b7d0 2024-03-06T07:41:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e67e00640 1 -- 172.21.15.97:0/2066322837 >> v1:172.21.15.135:6789/0 conn(0x7f9e6806d110 legacy=0x7f9e680b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e66a00640 1 -- 172.21.15.97:0/2066322837 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802748340 0 0) 0x7f9e68071d40 con 0x7f9e6806d110 2024-03-06T07:41:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e67e00640 1 -- 172.21.15.97:0/2066322837 >> v1:172.21.15.97:6789/0 conn(0x7f9e6814b7d0 legacy=0x7f9e6806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.600+0000 7f9e67e00640 1 -- 172.21.15.97:0/2066322837 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68072020 con 0x7f9e680716f0 2024-03-06T07:41:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.602+0000 7f9e67e00640 1 --2- 172.21.15.97:0/2066322837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 0x7f9e6814bda0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e6002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=558bcc6ac4053647 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.602+0000 7f9e66a00640 1 -- 172.21.15.97:0/2066322837 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60039070 con 0x7f9e680716f0 2024-03-06T07:41:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.602+0000 7f9e66a00640 1 -- 172.21.15.97:0/2066322837 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6002cc60 con 0x7f9e680716f0 2024-03-06T07:41:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.602+0000 7f9e66a00640 1 -- 172.21.15.97:0/2066322837 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60034640 con 0x7f9e680716f0 2024-03-06T07:41:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 -- 172.21.15.97:0/2066322837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 msgr2=0x7f9e6814bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 --2- 172.21.15.97:0/2066322837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 0x7f9e6814bda0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e6002beb0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 -- 172.21.15.97:0/2066322837 shutdown_connections 2024-03-06T07:41:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 --2- 172.21.15.97:0/2066322837 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 0x7f9e6814bda0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 -- 172.21.15.97:0/2066322837 >> 172.21.15.97:0/2066322837 conn(0x7f9e680cfa80 msgr2=0x7f9e680cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 -- 172.21.15.97:0/2066322837 shutdown_connections 2024-03-06T07:41:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.603+0000 7f9e6e800640 1 -- 172.21.15.97:0/2066322837 wait complete. 2024-03-06T07:41:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.605+0000 7f9e6e800640 1 Processor -- start 2024-03-06T07:41:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.605+0000 7f9e6e800640 1 -- start start 2024-03-06T07:41:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e6e800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e6806d110 0x7f9e6815c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e6e800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 0x7f9e681ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e6e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e6814b7d0 0x7f9e68145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e6e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680b2c60 con 0x7f9e6814b7d0 2024-03-06T07:41:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e6e800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e68071af0 con 0x7f9e680716f0 2024-03-06T07:41:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e6e800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680b3890 con 0x7f9e6806d110 2024-03-06T07:41:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e6814b7d0 0x7f9e68145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e6814b7d0 0x7f9e68145ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44826/0 (socket says 172.21.15.97:44826) 2024-03-06T07:41:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 -- 172.21.15.97:0/3738329626 learned_addr learned my addr 172.21.15.97:0/3738329626 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e5ea00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 0x7f9e681ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67400640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e6806d110 0x7f9e6815c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 -- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e6806d110 msgr2=0x7f9e6815c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e6806d110 0x7f9e6815c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 -- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 msgr2=0x7f9e681ea8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9e680716f0 0x7f9e681ea8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67e00640 1 -- 172.21.15.97:0/3738329626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e681e9800 con 0x7f9e6814b7d0 2024-03-06T07:41:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.606+0000 7f9e67400640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f9e6806d110 0x7f9e6815c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.607+0000 7f9e67e00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e6814b7d0 0x7f9e68145ce0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.607+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60039050 con 0x7f9e6814b7d0 2024-03-06T07:41:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.607+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6003a040 con 0x7f9e6814b7d0 2024-03-06T07:41:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.607+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60003bb0 con 0x7f9e6814b7d0 2024-03-06T07:41:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.607+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e681e9ae0 con 0x7f9e6814b7d0 2024-03-06T07:41:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.607+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e681ea0a0 con 0x7f9e6814b7d0 2024-03-06T07:41:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.608+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e6804f330 con 0x7f9e6814b7d0 2024-03-06T07:41:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.609+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e60004030 con 0x7f9e6814b7d0 2024-03-06T07:41:45.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.610+0000 7f9e5fe00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e4406b4d0 0x7f9e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.610+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9e600ad360 con 0x7f9e6814b7d0 2024-03-06T07:41:45.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.610+0000 7f9e5fe00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f9e44071430 0x7f9e44073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.610+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e6004be70 con 0x7f9e44071430 2024-03-06T07:41:45.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.610+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e600ad7f0 con 0x7f9e6814b7d0 2024-03-06T07:41:45.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.610+0000 7f9e67400640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e4406b4d0 0x7f9e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.611+0000 7f9e5ea00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f9e44071430 0x7f9e44073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.611+0000 7f9e67400640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e4406b4d0 0x7f9e4406d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9e58005880 tx=0x7f9e58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.611+0000 7f9e5ea00640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f9e44071430 0x7f9e44073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.612+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== osd.0 v2:172.21.15.97:6819/1306424436 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9e6004be70 con 0x7f9e44071430 2024-03-06T07:41:45.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.626+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 --> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9e680b4c10 con 0x7f9e44071430 2024-03-06T07:41:45.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e5fe00640 1 -- 172.21.15.97:0/3738329626 <== osd.0 v2:172.21.15.97:6819/1306424436 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9e680b4c10 con 0x7f9e44071430 2024-03-06T07:41:45.630 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:45.630 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:45.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f9e44071430 msgr2=0x7f9e44073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e6e800640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6819/1306424436,v1:172.21.15.97:6821/1306424436] conn(0x7f9e44071430 0x7f9e44073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e4406b4d0 msgr2=0x7f9e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e6e800640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f9e4406b4d0 0x7f9e4406d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9e58005880 tx=0x7f9e58009040 comp rx=0 tx=0).stop 2024-03-06T07:41:45.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e6814b7d0 msgr2=0x7f9e68145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e6e800640 1 --2- 172.21.15.97:0/3738329626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e6814b7d0 0x7f9e68145ce0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e60004a40 comp rx=0 tx=0).stop 2024-03-06T07:41:45.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.627+0000 7f9e67e00640 1 -- 172.21.15.97:0/3738329626 reap_dead start 2024-03-06T07:41:45.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.628+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 shutdown_connections 2024-03-06T07:41:45.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.628+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 >> 172.21.15.97:0/3738329626 conn(0x7f9e680cfa80 msgr2=0x7f9e68148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:45.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.628+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 shutdown_connections 2024-03-06T07:41:45.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.628+0000 7f9e6e800640 1 -- 172.21.15.97:0/3738329626 wait complete. 2024-03-06T07:41:45.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:41:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.858+0000 7f468fa00640 1 Processor -- start 2024-03-06T07:41:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.858+0000 7f468fa00640 1 -- start start 2024-03-06T07:41:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468fa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f468815cc70 0x7f4688162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468fa00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4688166ec0 con 0x7f46880cd5b0 2024-03-06T07:41:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468fa00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468806cbe0 con 0x7f4688145b70 2024-03-06T07:41:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468fa00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468806cec0 con 0x7f468815cc70 2024-03-06T07:41:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468d000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f46880cd5b0 0x7f46880cd990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38732/0 (socket says 172.21.15.97:38732) 2024-03-06T07:41:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468d000640 1 -- 172.21.15.97:0/1806193617 learned_addr learned my addr 172.21.15.97:0/1806193617 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468da00640 1 --2- 172.21.15.97:0/1806193617 >> v2:172.21.15.97:3300/0 conn(0x7f468815cc70 0x7f4688162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f4687400640 1 -- 172.21.15.97:0/1806193617 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204870722 0 0) 0x7f4688166ec0 con 0x7f46880cd5b0 2024-03-06T07:41:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f4687400640 1 -- 172.21.15.97:0/1806193617 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c003520 con 0x7f46880cd5b0 2024-03-06T07:41:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468da00640 1 -- 172.21.15.97:0/1806193617 >> v1:172.21.15.135:6789/0 conn(0x7f4688145b70 legacy=0x7f468815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.860+0000 7f468da00640 1 -- 172.21.15.97:0/1806193617 >> v1:172.21.15.97:6789/0 conn(0x7f46880cd5b0 legacy=0x7f46880cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.861+0000 7f468da00640 1 -- 172.21.15.97:0/1806193617 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468806d1a0 con 0x7f468815cc70 2024-03-06T07:41:45.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f468da00640 1 --2- 172.21.15.97:0/1806193617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f4688162fb0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f46800097a0 tx=0x7f468002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e1a07a9c58d61f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f4687400640 1 -- 172.21.15.97:0/1806193617 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680034070 con 0x7f468815cc70 2024-03-06T07:41:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f4687400640 1 -- 172.21.15.97:0/1806193617 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680030610 con 0x7f468815cc70 2024-03-06T07:41:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f4687400640 1 -- 172.21.15.97:0/1806193617 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46800309c0 con 0x7f468815cc70 2024-03-06T07:41:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f468fa00640 1 -- 172.21.15.97:0/1806193617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 msgr2=0x7f4688162fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f468fa00640 1 --2- 172.21.15.97:0/1806193617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f4688162fb0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f46800097a0 tx=0x7f468002b960 comp rx=0 tx=0).stop 2024-03-06T07:41:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f468fa00640 1 -- 172.21.15.97:0/1806193617 shutdown_connections 2024-03-06T07:41:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f468fa00640 1 --2- 172.21.15.97:0/1806193617 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f4688162fb0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.862+0000 7f468fa00640 1 -- 172.21.15.97:0/1806193617 >> 172.21.15.97:0/1806193617 conn(0x7f46880cc4f0 msgr2=0x7f46880cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.863+0000 7f468fa00640 1 -- 172.21.15.97:0/1806193617 shutdown_connections 2024-03-06T07:41:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.863+0000 7f468fa00640 1 -- 172.21.15.97:0/1806193617 wait complete. 2024-03-06T07:41:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.864+0000 7f468fa00640 1 Processor -- start 2024-03-06T07:41:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.864+0000 7f468fa00640 1 -- start start 2024-03-06T07:41:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.865+0000 7f468fa00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46880cd5b0 0x7f468815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.865+0000 7f468fa00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4688145b70 0x7f46880b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.865+0000 7f468fa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f46880b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.865+0000 7f468fa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4688163670 con 0x7f468815cc70 2024-03-06T07:41:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.865+0000 7f468fa00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46881634f0 con 0x7f4688145b70 2024-03-06T07:41:45.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.865+0000 7f468fa00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46881637f0 con 0x7f46880cd5b0 2024-03-06T07:41:45.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f46880b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f46880b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44842/0 (socket says 172.21.15.97:44842) 2024-03-06T07:41:45.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 -- 172.21.15.97:0/733200171 learned_addr learned my addr 172.21.15.97:0/733200171 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:45.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468d000640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46880cd5b0 0x7f468815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f4687e00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4688145b70 0x7f46880b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 -- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46880cd5b0 msgr2=0x7f468815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46880cd5b0 0x7f468815b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 -- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4688145b70 msgr2=0x7f46880b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4688145b70 0x7f46880b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 -- 172.21.15.97:0/733200171 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468815bcc0 con 0x7f468815cc70 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.866+0000 7f468da00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f46880b2c90 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f46800097a0 tx=0x7f46800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.867+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680034040 con 0x7f468815cc70 2024-03-06T07:41:45.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.867+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680030410 con 0x7f468815cc70 2024-03-06T07:41:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.867+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680006e10 con 0x7f468815cc70 2024-03-06T07:41:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.867+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46880b0490 con 0x7f468815cc70 2024-03-06T07:41:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.867+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46880b0a50 con 0x7f468815cc70 2024-03-06T07:41:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.868+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468804f330 con 0x7f468815cc70 2024-03-06T07:41:45.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.869+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4680037070 con 0x7f468815cc70 2024-03-06T07:41:45.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.870+0000 7f4684c00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f466406b500 0x7f466406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:45.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.870+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f46800ab910 con 0x7f468815cc70 2024-03-06T07:41:45.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.870+0000 7f468d000640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f466406b500 0x7f466406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:45.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.871+0000 7f468d000640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f466406b500 0x7f466406d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4678001560 tx=0x7f4678009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:45.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:45.874+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f468815bcc0 con 0x7f468815cc70 2024-03-06T07:41:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.072+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f468806e8d0 con 0x7f466406b500 2024-03-06T07:41:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.073+0000 7f4684c00640 1 -- 172.21.15.97:0/733200171 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f468806e8d0 con 0x7f466406b500 2024-03-06T07:41:46.076 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:41:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.075+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f466406b500 msgr2=0x7f466406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.075+0000 7f468fa00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f466406b500 0x7f466406d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4678001560 tx=0x7f4678009040 comp rx=0 tx=0).stop 2024-03-06T07:41:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.075+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 msgr2=0x7f46880b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.075+0000 7f468fa00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f46880b2c90 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f46800097a0 tx=0x7f46800046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 shutdown_connections 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f466406b500 0x7f466406d9c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f468815cc70 0x7f46880b2c90 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4688145b70 0x7f46880b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 --2- 172.21.15.97:0/733200171 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f46880cd5b0 0x7f468815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 >> 172.21.15.97:0/733200171 conn(0x7f46880cc4f0 msgr2=0x7f4688164be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 shutdown_connections 2024-03-06T07:41:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.076+0000 7f468fa00640 1 -- 172.21.15.97:0/733200171 wait complete. 2024-03-06T07:41:46.096 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:41:46.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.310+0000 7f74f9600640 1 Processor -- start 2024-03-06T07:41:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.310+0000 7f74f9600640 1 -- start start 2024-03-06T07:41:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f40b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f9600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f74f40723c0 0x7f74f40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f9600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40b4300 con 0x7f74f40b5090 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f9600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f40b45e0 con 0x7f74f40723c0 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40b48c0 con 0x7f74f4143a50 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f2000640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f74f40b5090 0x7f74f4071cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38738/0 (socket says 172.21.15.97:38738) 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f2000640 1 -- 172.21.15.97:0/3792752055 learned_addr learned my addr 172.21.15.97:0/3792752055 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.312+0000 7f74f3400640 1 --2- 172.21.15.97:0/3792752055 >> v2:172.21.15.97:3300/0 conn(0x7f74f40723c0 0x7f74f40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f2a00640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f40b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f1600640 1 -- 172.21.15.97:0/3792752055 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422919129 0 0) 0x7f74f40b4300 con 0x7f74f40b5090 2024-03-06T07:41:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f1600640 1 -- 172.21.15.97:0/3792752055 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d4003520 con 0x7f74f40b5090 2024-03-06T07:41:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f3400640 1 -- 172.21.15.97:0/3792752055 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 msgr2=0x7f74f40b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f3400640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f40b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f3400640 1 -- 172.21.15.97:0/3792752055 >> v1:172.21.15.97:6789/0 conn(0x7f74f40b5090 legacy=0x7f74f4071cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f3400640 1 -- 172.21.15.97:0/3792752055 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f40b4ba0 con 0x7f74f40723c0 2024-03-06T07:41:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.313+0000 7f74f2a00640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f40b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:41:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.314+0000 7f74f3400640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40723c0 0x7f74f40ce5c0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0097a0 tx=0x7f74ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee22bad69bd39a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.314+0000 7f74f1600640 1 -- 172.21.15.97:0/3792752055 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec035020 con 0x7f74f40723c0 2024-03-06T07:41:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f1600640 1 -- 172.21.15.97:0/3792752055 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec031830 con 0x7f74f40723c0 2024-03-06T07:41:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f1600640 1 -- 172.21.15.97:0/3792752055 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec031be0 con 0x7f74f40723c0 2024-03-06T07:41:46.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 -- 172.21.15.97:0/3792752055 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40723c0 msgr2=0x7f74f40ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40723c0 0x7f74f40ce5c0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0097a0 tx=0x7f74ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 -- 172.21.15.97:0/3792752055 shutdown_connections 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40723c0 0x7f74f40ce5c0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 --2- 172.21.15.97:0/3792752055 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f40b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 -- 172.21.15.97:0/3792752055 >> 172.21.15.97:0/3792752055 conn(0x7f74f40b27d0 msgr2=0x7f74f40b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 -- 172.21.15.97:0/3792752055 shutdown_connections 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.315+0000 7f74f9600640 1 -- 172.21.15.97:0/3792752055 wait complete. 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.317+0000 7f74f9600640 1 Processor -- start 2024-03-06T07:41:46.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.317+0000 7f74f9600640 1 -- start start 2024-03-06T07:41:46.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.318+0000 7f74f9600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74f40723c0 0x7f74f415bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.318+0000 7f74f9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40b5090 0x7f74f415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.318+0000 7f74f9600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f41f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.318+0000 7f74f9600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f4143e30 con 0x7f74f40b5090 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.318+0000 7f74f9600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40cdfa0 con 0x7f74f40723c0 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.318+0000 7f74f9600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f4135020 con 0x7f74f4143a50 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40b5090 0x7f74f415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40b5090 0x7f74f415a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44850/0 (socket says 172.21.15.97:44850) 2024-03-06T07:41:46.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 -- 172.21.15.97:0/552619310 learned_addr learned my addr 172.21.15.97:0/552619310 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2a00640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74f40723c0 0x7f74f415bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f3400640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f41f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 -- 172.21.15.97:0/552619310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 msgr2=0x7f74f41f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f74f4143a50 0x7f74f41f6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 -- 172.21.15.97:0/552619310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74f40723c0 msgr2=0x7f74f415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74f40723c0 0x7f74f415bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 -- 172.21.15.97:0/552619310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f415c2b0 con 0x7f74f40b5090 2024-03-06T07:41:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.319+0000 7f74f2000640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40b5090 0x7f74f415a560 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00cb40 tx=0x7f74dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.320+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc017070 con 0x7f74f40b5090 2024-03-06T07:41:46.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.320+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74dc00aca0 con 0x7f74f40b5090 2024-03-06T07:41:46.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.320+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f415c590 con 0x7f74f40b5090 2024-03-06T07:41:46.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.320+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc012620 con 0x7f74f40b5090 2024-03-06T07:41:46.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.320+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f415cb50 con 0x7f74f40b5090 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.321+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74f404f330 con 0x7f74f40b5090 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.322+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74dc01a070 con 0x7f74f40b5090 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74eaa00640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d006b4d0 0x7f74d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f74dc089d60 con 0x7f74f40b5090 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74f2a00640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d006b4d0 0x7f74d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74eaa00640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f74d0071430 0x7f74d0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74d0073f00 con 0x7f74d0071430 2024-03-06T07:41:46.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74dc08c460 con 0x7f74f40b5090 2024-03-06T07:41:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.323+0000 7f74f3400640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f74d0071430 0x7f74d0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.324+0000 7f74f2a00640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d006b4d0 0x7f74d006d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f74e4006fd0 tx=0x7f74e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.324+0000 7f74f3400640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f74d0071430 0x7f74d0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.325+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== osd.1 v2:172.21.15.97:6808/2322813614 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f74d0073f00 con 0x7f74d0071430 2024-03-06T07:41:46.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.339+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 --> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f74f40cfac0 con 0x7f74d0071430 2024-03-06T07:41:46.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74eaa00640 1 -- 172.21.15.97:0/552619310 <== osd.1 v2:172.21.15.97:6808/2322813614 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f74f40cfac0 con 0x7f74d0071430 2024-03-06T07:41:46.342 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:46.342 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:46.342 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:46.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f74d0071430 msgr2=0x7f74d0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6808/2322813614,v1:172.21.15.97:6809/2322813614] conn(0x7f74d0071430 0x7f74d0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d006b4d0 msgr2=0x7f74d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f74d006b4d0 0x7f74d006d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f74e4006fd0 tx=0x7f74e4008040 comp rx=0 tx=0).stop 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40b5090 msgr2=0x7f74f415a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 --2- 172.21.15.97:0/552619310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74f40b5090 0x7f74f415a560 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00cb40 tx=0x7f74dc00a060 comp rx=0 tx=0).stop 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f3400640 1 -- 172.21.15.97:0/552619310 reap_dead start 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 shutdown_connections 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 >> 172.21.15.97:0/552619310 conn(0x7f74f40b27d0 msgr2=0x7f74f4142050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 shutdown_connections 2024-03-06T07:41:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.340+0000 7f74f9600640 1 -- 172.21.15.97:0/552619310 wait complete. 2024-03-06T07:41:46.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:41:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.564+0000 7fd95c400640 1 Processor -- start 2024-03-06T07:41:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.564+0000 7fd95c400640 1 -- start start 2024-03-06T07:41:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd95c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 0x7fd9540c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd95c400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd954145b30 con 0x7fd9540c5880 2024-03-06T07:41:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd95c400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd954142ce0 con 0x7fd9540c4d10 2024-03-06T07:41:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd95c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540cbca0 con 0x7fd9540c4930 2024-03-06T07:41:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd95a400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd9540c5880 0x7fd954165b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38744/0 (socket says 172.21.15.97:38744) 2024-03-06T07:41:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd95a400640 1 -- 172.21.15.97:0/942803840 learned_addr learned my addr 172.21.15.97:0/942803840 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.566+0000 7fd959a00640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 0x7fd9540c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926654243 0 0) 0x7fd94c001110 con 0x7fd9540c5880 2024-03-06T07:41:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd93c003520 con 0x7fd9540c5880 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1217699153 0 0) 0x7fd93c003520 con 0x7fd9540c5880 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 msgr2=0x7fd9540c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 0x7fd9540c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 >> v1:172.21.15.135:6789/0 conn(0x7fd9540c4d10 legacy=0x7fd9540c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540cbf80 con 0x7fd9540c5880 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fd94c001840 con 0x7fd9540c5880 2024-03-06T07:41:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 >> v1:172.21.15.97:6789/0 conn(0x7fd9540c5880 legacy=0x7fd954165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 0x7fd93c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd93c003f60 0x7fd93c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd93c004d40 0x7fd93c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540cbca0 con 0x7fd93c003f60 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c00bb50 con 0x7fd93c004d40 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c00bd50 con 0x7fd9540c4930 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.567+0000 7fd959000640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd93c003f60 0x7fd93c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd959a00640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 0x7fd93c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd95a400640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd93c004d40 0x7fd93c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd959000640 1 -- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 msgr2=0x7fd93c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd959000640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd9540c4930 0x7fd93c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd959000640 1 -- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd93c004d40 msgr2=0x7fd93c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd959000640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd93c004d40 0x7fd93c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.568+0000 7fd959000640 1 -- 172.21.15.97:0/942803840 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c003520 con 0x7fd93c003f60 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.569+0000 7fd959000640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd93c003f60 0x7fd93c004800 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fd940001750 tx=0x7fd93c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fdaaf4e9d191b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.569+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940001370 con 0x7fd93c003f60 2024-03-06T07:41:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.569+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9400052d0 con 0x7fd93c003f60 2024-03-06T07:41:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.569+0000 7fd953e00640 1 -- 172.21.15.97:0/942803840 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940003830 con 0x7fd93c003f60 2024-03-06T07:41:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95c400640 1 -- 172.21.15.97:0/942803840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd93c003f60 msgr2=0x7fd93c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95c400640 1 --2- 172.21.15.97:0/942803840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd93c003f60 0x7fd93c004800 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fd940001750 tx=0x7fd93c005830 comp rx=0 tx=0).stop 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95a400640 1 -- 172.21.15.97:0/942803840 reap_dead start 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95c400640 1 -- 172.21.15.97:0/942803840 shutdown_connections 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95c400640 1 -- 172.21.15.97:0/942803840 >> 172.21.15.97:0/942803840 conn(0x7fd9540cc4f0 msgr2=0x7fd9540cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95c400640 1 -- 172.21.15.97:0/942803840 shutdown_connections 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.570+0000 7fd95c400640 1 -- 172.21.15.97:0/942803840 wait complete. 2024-03-06T07:41:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.572+0000 7fd95c400640 1 Processor -- start 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.572+0000 7fd95c400640 1 -- start start 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.572+0000 7fd95c400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd93c003f60 0x7fd954141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd95c400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 0x7fd9540c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd95c400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9540c8cc0 0x7fd9541f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd95c400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95406e600 con 0x7fd9540b1040 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd95c400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540ca3b0 con 0x7fd9540c8cc0 2024-03-06T07:41:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd95c400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540cc260 con 0x7fd93c003f60 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 0x7fd9540c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 0x7fd9540c8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44874/0 (socket says 172.21.15.97:44874) 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 -- 172.21.15.97:0/1051121642 learned_addr learned my addr 172.21.15.97:0/1051121642 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959a00640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd93c003f60 0x7fd954141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd95a400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9540c8cc0 0x7fd9541f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 -- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd93c003f60 msgr2=0x7fd954141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd93c003f60 0x7fd954141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 -- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9540c8cc0 msgr2=0x7fd9541f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9540c8cc0 0x7fd9541f5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.573+0000 7fd959000640 1 -- 172.21.15.97:0/1051121642 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540b2960 con 0x7fd9540b1040 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.574+0000 7fd959000640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 0x7fd9540c8780 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fd940001870 tx=0x7fd940001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.574+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940041070 con 0x7fd9540b1040 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.574+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940005480 con 0x7fd9540b1040 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.574+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940001370 con 0x7fd9540b1040 2024-03-06T07:41:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.574+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9540b2c40 con 0x7fd9540b1040 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.574+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9540b3200 con 0x7fd9540b1040 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.575+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95404f330 con 0x7fd9540b1040 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.576+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd940044070 con 0x7fd9540b1040 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.577+0000 7fd951600640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd93406b500 0x7fd93406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.577+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd9400aba10 con 0x7fd9540b1040 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.577+0000 7fd959a00640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd93406b500 0x7fd93406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:46.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.578+0000 7fd959a00640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd93406b500 0x7fd93406d9c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fd948007520 tx=0x7fd94800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.581+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd94007ac40 con 0x7fd9540b1040 2024-03-06T07:41:46.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.776+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9540b2710 con 0x7fd93406b500 2024-03-06T07:41:46.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.777+0000 7fd951600640 1 -- 172.21.15.97:0/1051121642 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd9540b2710 con 0x7fd93406b500 2024-03-06T07:41:46.780 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:41:46.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd93406b500 msgr2=0x7fd93406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd93406b500 0x7fd93406d9c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fd948007520 tx=0x7fd94800b040 comp rx=0 tx=0).stop 2024-03-06T07:41:46.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 msgr2=0x7fd9540c8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:46.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 0x7fd9540c8780 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fd940001870 tx=0x7fd940001120 comp rx=0 tx=0).stop 2024-03-06T07:41:46.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 shutdown_connections 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd93406b500 0x7fd93406d9c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9540c8cc0 0x7fd9541f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9540b1040 0x7fd9540c8780 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 --2- 172.21.15.97:0/1051121642 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd93c003f60 0x7fd954141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 >> 172.21.15.97:0/1051121642 conn(0x7fd9540cc4f0 msgr2=0x7fd9541e0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 shutdown_connections 2024-03-06T07:41:46.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:46.780+0000 7fd95c400640 1 -- 172.21.15.97:0/1051121642 wait complete. 2024-03-06T07:41:46.801 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:41:46.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:47.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.016+0000 7fe23a200640 1 Processor -- start 2024-03-06T07:41:47.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.016+0000 7fe23a200640 1 -- start start 2024-03-06T07:41:47.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe23a200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe23415adf0 0x7fe23406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe23a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23406e8e0 0x7fe23415eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe23a200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23415b1d0 con 0x7fe23415f020 2024-03-06T07:41:47.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe23a200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23406ed40 con 0x7fe23415adf0 2024-03-06T07:41:47.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe23a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340b2bd0 con 0x7fe23406e8e0 2024-03-06T07:41:47.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe233400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe23415adf0 0x7fe23406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe233400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe23415adf0 0x7fe23406e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44882/0 (socket says 172.21.15.97:44882) 2024-03-06T07:41:47.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe233400640 1 -- 172.21.15.97:0/3861395020 learned_addr learned my addr 172.21.15.97:0/3861395020 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.018+0000 7fe232a00640 1 --2- 172.21.15.97:0/3861395020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23406e8e0 0x7fe23415eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.019+0000 7fe233400640 1 -- 172.21.15.97:0/3861395020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23406e8e0 msgr2=0x7fe23415eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.019+0000 7fe233400640 1 --2- 172.21.15.97:0/3861395020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23406e8e0 0x7fe23415eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.019+0000 7fe233400640 1 -- 172.21.15.97:0/3861395020 >> v1:172.21.15.135:6789/0 conn(0x7fe23415f020 legacy=0x7fe2340716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.019+0000 7fe233400640 1 -- 172.21.15.97:0/3861395020 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2340b2eb0 con 0x7fe23415adf0 2024-03-06T07:41:47.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.020+0000 7fe233400640 1 --2- 172.21.15.97:0/3861395020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23415adf0 0x7fe23406e3a0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fe224009870 tx=0x7fe22402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4419af62d7b0bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.020+0000 7fe232000640 1 -- 172.21.15.97:0/3861395020 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224035020 con 0x7fe23415adf0 2024-03-06T07:41:47.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.020+0000 7fe232000640 1 -- 172.21.15.97:0/3861395020 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe224031640 con 0x7fe23415adf0 2024-03-06T07:41:47.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.020+0000 7fe232000640 1 -- 172.21.15.97:0/3861395020 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22402d040 con 0x7fe23415adf0 2024-03-06T07:41:47.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 -- 172.21.15.97:0/3861395020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23415adf0 msgr2=0x7fe23406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 --2- 172.21.15.97:0/3861395020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23415adf0 0x7fe23406e3a0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fe224009870 tx=0x7fe22402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 -- 172.21.15.97:0/3861395020 shutdown_connections 2024-03-06T07:41:47.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 --2- 172.21.15.97:0/3861395020 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23406e8e0 0x7fe23415eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 --2- 172.21.15.97:0/3861395020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23415adf0 0x7fe23406e3a0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 -- 172.21.15.97:0/3861395020 >> 172.21.15.97:0/3861395020 conn(0x7fe2340cfa80 msgr2=0x7fe2340cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 -- 172.21.15.97:0/3861395020 shutdown_connections 2024-03-06T07:41:47.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.021+0000 7fe23a200640 1 -- 172.21.15.97:0/3861395020 wait complete. 2024-03-06T07:41:47.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.022+0000 7fe23a200640 1 Processor -- start 2024-03-06T07:41:47.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.023+0000 7fe23a200640 1 -- start start 2024-03-06T07:41:47.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.023+0000 7fe23a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23406e8e0 0x7fe234143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.023+0000 7fe23a200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe23415adf0 0x7fe2341e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe23a200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23415f020 0x7fe2341e5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe23a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23415f850 con 0x7fe23406e8e0 2024-03-06T07:41:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe23a200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340b3470 con 0x7fe23415adf0 2024-03-06T07:41:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe23a200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340b3990 con 0x7fe23415f020 2024-03-06T07:41:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23406e8e0 0x7fe234143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23406e8e0 0x7fe234143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44894/0 (socket says 172.21.15.97:44894) 2024-03-06T07:41:47.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 -- 172.21.15.97:0/2081558192 learned_addr learned my addr 172.21.15.97:0/2081558192 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 -- 172.21.15.97:0/2081558192 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23415f020 msgr2=0x7fe2341e5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fe23415f020 0x7fe2341e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 -- 172.21.15.97:0/2081558192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe23415adf0 msgr2=0x7fe2341e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe232a00640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe23415adf0 0x7fe2341e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe23415adf0 0x7fe2341e70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.024+0000 7fe233400640 1 -- 172.21.15.97:0/2081558192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2341e4780 con 0x7fe23406e8e0 2024-03-06T07:41:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.025+0000 7fe233400640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23406e8e0 0x7fe234143ad0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fe22402b930 tx=0x7fe2240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.025+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224040070 con 0x7fe23406e8e0 2024-03-06T07:41:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.025+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2240399f0 con 0x7fe23406e8e0 2024-03-06T07:41:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.025+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2341e4a60 con 0x7fe23406e8e0 2024-03-06T07:41:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.025+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224035040 con 0x7fe23406e8e0 2024-03-06T07:41:47.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.025+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2341e5020 con 0x7fe23406e8e0 2024-03-06T07:41:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.026+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe23404f330 con 0x7fe23406e8e0 2024-03-06T07:41:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.027+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe22404b060 con 0x7fe23406e8e0 2024-03-06T07:41:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe22b400640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe21406b4d0 0x7fe21406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe2240ab9d0 con 0x7fe23406e8e0 2024-03-06T07:41:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe232a00640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe21406b4d0 0x7fe21406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe22b400640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fe214071430 0x7fe214073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe22404ae50 con 0x7fe214071430 2024-03-06T07:41:47.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2240b02a0 con 0x7fe23406e8e0 2024-03-06T07:41:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.028+0000 7fe233e00640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fe214071430 0x7fe214073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.029+0000 7fe232a00640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe21406b4d0 0x7fe21406d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe220006fd0 tx=0x7fe220008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.029+0000 7fe233e00640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fe214071430 0x7fe214073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.030+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== osd.2 v2:172.21.15.97:6816/75411953 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe22404ae50 con 0x7fe214071430 2024-03-06T07:41:47.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.044+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 --> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe23415e700 con 0x7fe214071430 2024-03-06T07:41:47.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe22b400640 1 -- 172.21.15.97:0/2081558192 <== osd.2 v2:172.21.15.97:6816/75411953 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe23415e700 con 0x7fe214071430 2024-03-06T07:41:47.048 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:47.048 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:47.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fe214071430 msgr2=0x7fe214073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe23a200640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6816/75411953,v1:172.21.15.97:6817/75411953] conn(0x7fe214071430 0x7fe214073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe21406b4d0 msgr2=0x7fe21406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe23a200640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fe21406b4d0 0x7fe21406d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe220006fd0 tx=0x7fe220008040 comp rx=0 tx=0).stop 2024-03-06T07:41:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23406e8e0 msgr2=0x7fe234143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.045+0000 7fe23a200640 1 --2- 172.21.15.97:0/2081558192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe23406e8e0 0x7fe234143ad0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fe22402b930 tx=0x7fe2240046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.046+0000 7fe233e00640 1 -- 172.21.15.97:0/2081558192 reap_dead start 2024-03-06T07:41:47.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.046+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 shutdown_connections 2024-03-06T07:41:47.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.046+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 >> 172.21.15.97:0/2081558192 conn(0x7fe2340cfa80 msgr2=0x7fe23415da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.046+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 shutdown_connections 2024-03-06T07:41:47.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.046+0000 7fe23a200640 1 -- 172.21.15.97:0/2081558192 wait complete. 2024-03-06T07:41:47.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:41:47.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.277+0000 7fd66d200640 1 Processor -- start 2024-03-06T07:41:47.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.277+0000 7fd66d200640 1 -- start start 2024-03-06T07:41:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd66d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd66d200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd6680c5040 0x7fd6680cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd66d200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6680ae380 con 0x7fd6681474c0 2024-03-06T07:41:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd66d200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6680ae660 con 0x7fd6680c5040 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd66d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6680ae940 con 0x7fd6681444e0 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd666800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd6680c5040 0x7fd6680cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd666800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd6680c5040 0x7fd6680cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44906/0 (socket says 172.21.15.97:44906) 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.279+0000 7fd666800640 1 -- 172.21.15.97:0/894538810 learned_addr learned my addr 172.21.15.97:0/894538810 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.280+0000 7fd665e00640 1 --2- 172.21.15.97:0/894538810 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.280+0000 7fd666800640 1 -- 172.21.15.97:0/894538810 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 msgr2=0x7fd6680cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.280+0000 7fd666800640 1 --2- 172.21.15.97:0/894538810 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.280+0000 7fd666800640 1 -- 172.21.15.97:0/894538810 >> v1:172.21.15.135:6789/0 conn(0x7fd6681474c0 legacy=0x7fd6680c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.280+0000 7fd666800640 1 -- 172.21.15.97:0/894538810 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6680aec20 con 0x7fd6680c5040 2024-03-06T07:41:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.281+0000 7fd666800640 1 --2- 172.21.15.97:0/894538810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680cd920 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fd660009870 tx=0x7fd66002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b740796cf6ddb0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.281+0000 7fd664a00640 1 -- 172.21.15.97:0/894538810 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660035020 con 0x7fd6680c5040 2024-03-06T07:41:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.281+0000 7fd664a00640 1 -- 172.21.15.97:0/894538810 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd660031640 con 0x7fd6680c5040 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.281+0000 7fd664a00640 1 -- 172.21.15.97:0/894538810 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66002d040 con 0x7fd6680c5040 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 -- 172.21.15.97:0/894538810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 msgr2=0x7fd6680cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 --2- 172.21.15.97:0/894538810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680cd920 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fd660009870 tx=0x7fd66002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 -- 172.21.15.97:0/894538810 shutdown_connections 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 --2- 172.21.15.97:0/894538810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680cd920 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 --2- 172.21.15.97:0/894538810 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 -- 172.21.15.97:0/894538810 >> 172.21.15.97:0/894538810 conn(0x7fd6680cc4f0 msgr2=0x7fd6680cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 -- 172.21.15.97:0/894538810 shutdown_connections 2024-03-06T07:41:47.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.282+0000 7fd66d200640 1 -- 172.21.15.97:0/894538810 wait complete. 2024-03-06T07:41:47.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.284+0000 7fd66d200640 1 Processor -- start 2024-03-06T07:41:47.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.284+0000 7fd66d200640 1 -- start start 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd66d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd66d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd66d200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6681474c0 0x7fd66815d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd66d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6681448c0 con 0x7fd6680c5040 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd66d200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd668147cb0 con 0x7fd6681474c0 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd66d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd668144a40 con 0x7fd6681444e0 2024-03-06T07:41:47.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd665e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd665e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44916/0 (socket says 172.21.15.97:44916) 2024-03-06T07:41:47.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd665e00640 1 -- 172.21.15.97:0/456081888 learned_addr learned my addr 172.21.15.97:0/456081888 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd665e00640 1 -- 172.21.15.97:0/456081888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 msgr2=0x7fd6680b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.285+0000 7fd665e00640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd665e00640 1 -- 172.21.15.97:0/456081888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6681474c0 msgr2=0x7fd66815d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd665e00640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6681474c0 0x7fd66815d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd665e00640 1 -- 172.21.15.97:0/456081888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66815de90 con 0x7fd6680c5040 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd665e00640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680c8c80 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c820 tx=0x7fd65800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658015070 con 0x7fd6680c5040 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd658004240 con 0x7fd6680c5040 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6680c9380 con 0x7fd6680c5040 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65800ce10 con 0x7fd6680c5040 2024-03-06T07:41:47.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.286+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6680c9940 con 0x7fd6680c5040 2024-03-06T07:41:47.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.288+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66804f330 con 0x7fd6680c5040 2024-03-06T07:41:47.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.288+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd658007830 con 0x7fd6680c5040 2024-03-06T07:41:47.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.289+0000 7fd65e000640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd64406b4d0 0x7fd64406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.289+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd658089ed0 con 0x7fd6680c5040 2024-03-06T07:41:47.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.290+0000 7fd665400640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd64406b4d0 0x7fd64406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.290+0000 7fd665400640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd64406b4d0 0x7fd64406d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd6681439d0 tx=0x7fd650009230 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.294+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd658058f50 con 0x7fd6680c5040 2024-03-06T07:41:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.491+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd668143b40 con 0x7fd64406b4d0 2024-03-06T07:41:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.491+0000 7fd65e000640 1 -- 172.21.15.97:0/456081888 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd668143b40 con 0x7fd64406b4d0 2024-03-06T07:41:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:41:47.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd64406b4d0 msgr2=0x7fd64406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd64406b4d0 0x7fd64406d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd6681439d0 tx=0x7fd650009230 comp rx=0 tx=0).stop 2024-03-06T07:41:47.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 msgr2=0x7fd6680c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680c8c80 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c820 tx=0x7fd65800cce0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 shutdown_connections 2024-03-06T07:41:47.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd64406b4d0 0x7fd64406d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6681474c0 0x7fd66815d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd6681444e0 0x7fd6680b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 --2- 172.21.15.97:0/456081888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd6680c5040 0x7fd6680c8c80 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 >> 172.21.15.97:0/456081888 conn(0x7fd6680cc4f0 msgr2=0x7fd66806e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 shutdown_connections 2024-03-06T07:41:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.494+0000 7fd66d200640 1 -- 172.21.15.97:0/456081888 wait complete. 2024-03-06T07:41:47.514 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:41:47.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:47.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.711+0000 7fb450400640 1 Processor -- start 2024-03-06T07:41:47.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.711+0000 7fb450400640 1 -- start start 2024-03-06T07:41:47.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.712+0000 7fb450400640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb44806eab0 0x7fb44815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.712+0000 7fb450400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb4480cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.712+0000 7fb450400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44815b1d0 con 0x7fb44815adf0 2024-03-06T07:41:47.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.712+0000 7fb450400640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4480b2bd0 con 0x7fb44806eab0 2024-03-06T07:41:47.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.712+0000 7fb450400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b2eb0 con 0x7fb44815f020 2024-03-06T07:41:47.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.713+0000 7fb44e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb4480cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.713+0000 7fb44d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb44806eab0 0x7fb44815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.713+0000 7fb44d000640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb44806eab0 0x7fb44815eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:42146/0 (socket says 172.21.15.97:42146) 2024-03-06T07:41:47.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.713+0000 7fb44da00640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb44815adf0 0x7fb44806e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.97:55952/0 (socket says 172.21.15.97:55952) 2024-03-06T07:41:47.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.713+0000 7fb44e400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb4480cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3301/0 says I am v2:172.21.15.97:36032/0 (socket says 172.21.15.97:36032) 2024-03-06T07:41:47.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.713+0000 7fb44d000640 1 -- 172.21.15.97:0/4143769780 learned_addr learned my addr 172.21.15.97:0/4143769780 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.714+0000 7fb44e400640 1 -- 172.21.15.97:0/4143769780 >> v2:172.21.15.135:3300/0 conn(0x7fb44806eab0 msgr2=0x7fb44815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.714+0000 7fb44e400640 1 --2- 172.21.15.97:0/4143769780 >> v2:172.21.15.135:3300/0 conn(0x7fb44806eab0 0x7fb44815eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.714+0000 7fb44e400640 1 -- 172.21.15.97:0/4143769780 >> v1:172.21.15.135:6789/0 conn(0x7fb44815adf0 legacy=0x7fb44806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.714+0000 7fb44e400640 1 -- 172.21.15.97:0/4143769780 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480b3190 con 0x7fb44815f020 2024-03-06T07:41:47.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.714+0000 7fb447e00640 1 -- 172.21.15.97:0/4143769780 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932533922 0 0) 0x7fb44815b1d0 con 0x7fb44815adf0 2024-03-06T07:41:47.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.715+0000 7fb44e400640 1 --2- 172.21.15.97:0/4143769780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb4480cd090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb440009d90 tx=0x7fb44002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d221d5720e0470fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.716+0000 7fb447e00640 1 -- 172.21.15.97:0/4143769780 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440039070 con 0x7fb44815f020 2024-03-06T07:41:47.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.716+0000 7fb447e00640 1 -- 172.21.15.97:0/4143769780 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44002cc50 con 0x7fb44815f020 2024-03-06T07:41:47.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.716+0000 7fb447e00640 1 -- 172.21.15.97:0/4143769780 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440034440 con 0x7fb44815f020 2024-03-06T07:41:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.716+0000 7fb450400640 1 -- 172.21.15.97:0/4143769780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 msgr2=0x7fb4480cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.716+0000 7fb450400640 1 --2- 172.21.15.97:0/4143769780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb4480cd090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb440009d90 tx=0x7fb44002bea0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.717+0000 7fb450400640 1 -- 172.21.15.97:0/4143769780 shutdown_connections 2024-03-06T07:41:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.717+0000 7fb450400640 1 --2- 172.21.15.97:0/4143769780 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb4480cd090 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.717+0000 7fb450400640 1 --2- 172.21.15.97:0/4143769780 >> v2:172.21.15.135:3300/0 conn(0x7fb44806eab0 0x7fb44815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.717+0000 7fb450400640 1 -- 172.21.15.97:0/4143769780 >> 172.21.15.97:0/4143769780 conn(0x7fb4480cfa80 msgr2=0x7fb4480cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.717+0000 7fb450400640 1 -- 172.21.15.97:0/4143769780 shutdown_connections 2024-03-06T07:41:47.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.717+0000 7fb450400640 1 -- 172.21.15.97:0/4143769780 wait complete. 2024-03-06T07:41:47.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.718+0000 7fb450400640 1 Processor -- start 2024-03-06T07:41:47.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.719+0000 7fb450400640 1 -- start start 2024-03-06T07:41:47.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb450400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb44806eab0 0x7fb448148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb450400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44815adf0 0x7fb448150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb450400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb44815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb450400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b3990 con 0x7fb44806eab0 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb450400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44815f850 con 0x7fb44815adf0 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb450400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44815f470 con 0x7fb44815f020 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb44806eab0 0x7fb448148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb44806eab0 0x7fb448148c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44926/0 (socket says 172.21.15.97:44926) 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 -- 172.21.15.97:0/3498802336 learned_addr learned my addr 172.21.15.97:0/3498802336 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 -- 172.21.15.97:0/3498802336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 msgr2=0x7fb44815b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb44815f020 0x7fb44815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 -- 172.21.15.97:0/3498802336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44815adf0 msgr2=0x7fb448150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44815adf0 0x7fb448150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.720+0000 7fb44da00640 1 -- 172.21.15.97:0/3498802336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4481520b0 con 0x7fb44806eab0 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.721+0000 7fb44da00640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb44806eab0 0x7fb448148c10 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00c4b0 tx=0x7fb43c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.721+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c018070 con 0x7fb44806eab0 2024-03-06T07:41:47.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.721+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c007480 con 0x7fb44806eab0 2024-03-06T07:41:47.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.721+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb448152390 con 0x7fb44806eab0 2024-03-06T07:41:47.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.721+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c0047e0 con 0x7fb44806eab0 2024-03-06T07:41:47.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.721+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44815d460 con 0x7fb44806eab0 2024-03-06T07:41:47.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.722+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb44815db50 con 0x7fb44806eab0 2024-03-06T07:41:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.723+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb43c01b070 con 0x7fb44806eab0 2024-03-06T07:41:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.724+0000 7fb445600640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb42806b500 0x7fb42806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.724+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb43c089230 con 0x7fb44806eab0 2024-03-06T07:41:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.724+0000 7fb44d000640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb42806b500 0x7fb42806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.724+0000 7fb445600640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fb428071460 0x7fb428073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.725+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb428073f30 con 0x7fb428071460 2024-03-06T07:41:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.725+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb43c090050 con 0x7fb44806eab0 2024-03-06T07:41:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.725+0000 7fb44e400640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fb428071460 0x7fb428073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.725+0000 7fb44d000640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb42806b500 0x7fb42806d9c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb43400b240 tx=0x7fb43400b690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.725+0000 7fb44e400640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fb428071460 0x7fb428073880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.726+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== osd.3 v2:172.21.15.97:6800/3169996968 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb428073f30 con 0x7fb428071460 2024-03-06T07:41:47.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.741+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 --> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb44804f330 con 0x7fb428071460 2024-03-06T07:41:47.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.741+0000 7fb445600640 1 -- 172.21.15.97:0/3498802336 <== osd.3 v2:172.21.15.97:6800/3169996968 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb44804f330 con 0x7fb428071460 2024-03-06T07:41:47.744 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:47.744 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:47.744 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:47.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fb428071460 msgr2=0x7fb428073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6800/3169996968,v1:172.21.15.97:6801/3169996968] conn(0x7fb428071460 0x7fb428073880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb42806b500 msgr2=0x7fb42806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb42806b500 0x7fb42806d9c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb43400b240 tx=0x7fb43400b690 comp rx=0 tx=0).stop 2024-03-06T07:41:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb44806eab0 msgr2=0x7fb448148c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 --2- 172.21.15.97:0/3498802336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb44806eab0 0x7fb448148c10 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00c4b0 tx=0x7fb43c00c970 comp rx=0 tx=0).stop 2024-03-06T07:41:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb44e400640 1 -- 172.21.15.97:0/3498802336 reap_dead start 2024-03-06T07:41:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 shutdown_connections 2024-03-06T07:41:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 >> 172.21.15.97:0/3498802336 conn(0x7fb4480cfa80 msgr2=0x7fb44806e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 shutdown_connections 2024-03-06T07:41:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.742+0000 7fb450400640 1 -- 172.21.15.97:0/3498802336 wait complete. 2024-03-06T07:41:47.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:41:47.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.958+0000 7f206fc00640 1 Processor -- start 2024-03-06T07:41:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.958+0000 7f206fc00640 1 -- start start 2024-03-06T07:41:47.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.960+0000 7f206fc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 0x7f206806e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.960+0000 7f206fc00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20680cc000 con 0x7f206815f120 2024-03-06T07:41:47.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.960+0000 7f206fc00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20680ce360 con 0x7f206815b070 2024-03-06T07:41:47.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.960+0000 7f206fc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20680ce640 con 0x7f206815ac90 2024-03-06T07:41:47.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.960+0000 7f206dc00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f206815f120 0x7f20680c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38748/0 (socket says 172.21.15.97:38748) 2024-03-06T07:41:47.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.960+0000 7f206dc00640 1 -- 172.21.15.97:0/2482588902 learned_addr learned my addr 172.21.15.97:0/2482588902 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:47.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f206d200640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 0x7f206806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160912495 0 0) 0x7f20680cc000 con 0x7f206815f120 2024-03-06T07:41:47.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f204c003520 con 0x7f206815f120 2024-03-06T07:41:47.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3879217557 0 0) 0x7f204c003520 con 0x7f206815f120 2024-03-06T07:41:47.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 msgr2=0x7f206806e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 0x7f206806e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 >> v1:172.21.15.135:6789/0 conn(0x7f206815b070 legacy=0x7f206815ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20680ce920 con 0x7f206815f120 2024-03-06T07:41:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f2060001a10 con 0x7f206815f120 2024-03-06T07:41:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 >> v1:172.21.15.97:6789/0 conn(0x7f206815f120 legacy=0x7f20680c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f204c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f204c004800 0x7f204c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 0x7f204c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20680ce640 con 0x7f204c0039e0 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c00baf0 con 0x7f204c004800 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.961+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c00bcf0 con 0x7f206815ac90 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f206d200640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f204c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f205fe00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f204c004800 0x7f204c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f206d200640 1 -- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 msgr2=0x7f204c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f206d200640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f206815ac90 0x7f204c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f206d200640 1 -- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f204c004800 msgr2=0x7f204c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f206d200640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f204c004800 0x7f204c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f206d200640 1 -- 172.21.15.97:0/2482588902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c003520 con 0x7f204c0039e0 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.962+0000 7f205fe00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f204c004800 0x7f204c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.963+0000 7f206d200640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f204c003e00 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f2058009ac0 tx=0x7f204c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134999cc3cfbfa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.963+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058039070 con 0x7f204c0039e0 2024-03-06T07:41:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.963+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205802c710 con 0x7f204c0039e0 2024-03-06T07:41:47.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.963+0000 7f2067e00640 1 -- 172.21.15.97:0/2482588902 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058009690 con 0x7f204c0039e0 2024-03-06T07:41:47.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206fc00640 1 -- 172.21.15.97:0/2482588902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 msgr2=0x7f204c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206fc00640 1 --2- 172.21.15.97:0/2482588902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f204c003e00 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f2058009ac0 tx=0x7f204c005830 comp rx=0 tx=0).stop 2024-03-06T07:41:47.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206dc00640 1 -- 172.21.15.97:0/2482588902 reap_dead start 2024-03-06T07:41:47.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206fc00640 1 -- 172.21.15.97:0/2482588902 shutdown_connections 2024-03-06T07:41:47.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206fc00640 1 -- 172.21.15.97:0/2482588902 >> 172.21.15.97:0/2482588902 conn(0x7f20680cc4f0 msgr2=0x7f20680cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:47.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206fc00640 1 -- 172.21.15.97:0/2482588902 shutdown_connections 2024-03-06T07:41:47.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.964+0000 7f206fc00640 1 -- 172.21.15.97:0/2482588902 wait complete. 2024-03-06T07:41:47.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.966+0000 7f206fc00640 1 Processor -- start 2024-03-06T07:41:47.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.966+0000 7f206fc00640 1 -- start start 2024-03-06T07:41:47.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206fc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f20680b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206fc00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20680af530 0x7f206816a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206fc00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20680ae3a0 0x7f20680ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206fc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206815f5a0 con 0x7f204c0039e0 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206fc00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20680ceee0 con 0x7f20680af530 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206fc00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206815f720 con 0x7f20680ae3a0 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f20680b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f20680b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44950/0 (socket says 172.21.15.97:44950) 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.967+0000 7f206d200640 1 -- 172.21.15.97:0/3481024036 learned_addr learned my addr 172.21.15.97:0/3481024036 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206dc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20680ae3a0 0x7f20680ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f205fe00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20680af530 0x7f206816a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206d200640 1 -- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20680ae3a0 msgr2=0x7f20680ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206d200640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20680ae3a0 0x7f20680ae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206d200640 1 -- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20680af530 msgr2=0x7f206816a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206d200640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20680af530 0x7f206816a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206d200640 1 -- 172.21.15.97:0/3481024036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206816a820 con 0x7f204c0039e0 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f206d200640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f20680b53e0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f2058009be0 tx=0x7f2058004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058039050 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205803a040 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.968+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058005bd0 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.969+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20680ad900 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.969+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20680adba0 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.970+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206804f330 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.971+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2058047020 con 0x7f204c0039e0 2024-03-06T07:41:47.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.971+0000 7f2065600640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f204406b4d0 0x7f204406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:47.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.972+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f20580ab840 con 0x7f204c0039e0 2024-03-06T07:41:47.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.972+0000 7f205fe00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f204406b4d0 0x7f204406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:47.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.972+0000 7f205fe00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f204406b4d0 0x7f204406d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2050007fd0 tx=0x7f2050009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:47.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:47.976+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f206816a820 con 0x7f204c0039e0 2024-03-06T07:41:48.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.170+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f20680cd1c0 con 0x7f204406b4d0 2024-03-06T07:41:48.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.170+0000 7f2065600640 1 -- 172.21.15.97:0/3481024036 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f20680cd1c0 con 0x7f204406b4d0 2024-03-06T07:41:48.173 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f204406b4d0 msgr2=0x7f204406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f204406b4d0 0x7f204406d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2050007fd0 tx=0x7f2050009040 comp rx=0 tx=0).stop 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 msgr2=0x7f20680b53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f20680b53e0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f2058009be0 tx=0x7f2058004730 comp rx=0 tx=0).stop 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 shutdown_connections 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f204406b4d0 0x7f204406d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20680ae3a0 0x7f20680ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20680af530 0x7f206816a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 --2- 172.21.15.97:0/3481024036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f204c0039e0 0x7f20680b53e0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 >> 172.21.15.97:0/3481024036 conn(0x7f20680cc4f0 msgr2=0x7f20680c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:48.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 shutdown_connections 2024-03-06T07:41:48.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.173+0000 7f206fc00640 1 -- 172.21.15.97:0/3481024036 wait complete. 2024-03-06T07:41:48.195 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:41:48.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:48.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.408+0000 7fb5c3200640 1 Processor -- start 2024-03-06T07:41:48.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.408+0000 7fb5c3200640 1 -- start start 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c3200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb5c415ad30 0x7fb5c415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c3200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5c40b5d50 0x7fb5c4072f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c3200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c406c3a0 con 0x7fb5c40b01a0 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c3200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c415b110 con 0x7fb5c40b5d50 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c3200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c415f700 con 0x7fb5c415ad30 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c1400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5c40b5d50 0x7fb5c4072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c1400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5c40b5d50 0x7fb5c4072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44966/0 (socket says 172.21.15.97:44966) 2024-03-06T07:41:48.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c2800640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb5c40b01a0 0x7fb5c40b05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38750/0 (socket says 172.21.15.97:38750) 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c1400640 1 -- 172.21.15.97:0/316130611 learned_addr learned my addr 172.21.15.97:0/316130611 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.410+0000 7fb5c1e00640 1 --2- 172.21.15.97:0/316130611 >> v2:172.21.15.135:3300/0 conn(0x7fb5c415ad30 0x7fb5c415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.411+0000 7fb5c1400640 1 -- 172.21.15.97:0/316130611 >> v2:172.21.15.135:3300/0 conn(0x7fb5c415ad30 msgr2=0x7fb5c415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.411+0000 7fb5c1400640 1 --2- 172.21.15.97:0/316130611 >> v2:172.21.15.135:3300/0 conn(0x7fb5c415ad30 0x7fb5c415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.411+0000 7fb5c1400640 1 -- 172.21.15.97:0/316130611 >> v1:172.21.15.97:6789/0 conn(0x7fb5c40b01a0 legacy=0x7fb5c40b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.411+0000 7fb5c1400640 1 -- 172.21.15.97:0/316130611 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c406c640 con 0x7fb5c40b5d50 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.411+0000 7fb5c0a00640 1 -- 172.21.15.97:0/316130611 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231993885 0 0) 0x7fb5c406c3a0 con 0x7fb5c40b01a0 2024-03-06T07:41:48.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.411+0000 7fb5c1e00640 1 --2- 172.21.15.97:0/316130611 >> v2:172.21.15.135:3300/0 conn(0x7fb5c415ad30 0x7fb5c415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.412+0000 7fb5c1400640 1 --2- 172.21.15.97:0/316130611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b5d50 0x7fb5c4072f40 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fb5c406fb00 tx=0x7fb5ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdcf4ec3f37544a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.412+0000 7fb5c0a00640 1 -- 172.21.15.97:0/316130611 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac035020 con 0x7fb5c40b5d50 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.412+0000 7fb5c0a00640 1 -- 172.21.15.97:0/316130611 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac031870 con 0x7fb5c40b5d50 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.412+0000 7fb5c0a00640 1 -- 172.21.15.97:0/316130611 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac031c40 con 0x7fb5c40b5d50 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 -- 172.21.15.97:0/316130611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b5d50 msgr2=0x7fb5c4072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 --2- 172.21.15.97:0/316130611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b5d50 0x7fb5c4072f40 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fb5c406fb00 tx=0x7fb5ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 -- 172.21.15.97:0/316130611 shutdown_connections 2024-03-06T07:41:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 --2- 172.21.15.97:0/316130611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b5d50 0x7fb5c4072f40 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 --2- 172.21.15.97:0/316130611 >> v2:172.21.15.135:3300/0 conn(0x7fb5c415ad30 0x7fb5c415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 -- 172.21.15.97:0/316130611 >> 172.21.15.97:0/316130611 conn(0x7fb5c40b4e60 msgr2=0x7fb5c40b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 -- 172.21.15.97:0/316130611 shutdown_connections 2024-03-06T07:41:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.413+0000 7fb5c3200640 1 -- 172.21.15.97:0/316130611 wait complete. 2024-03-06T07:41:48.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.415+0000 7fb5c3200640 1 Processor -- start 2024-03-06T07:41:48.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.415+0000 7fb5c3200640 1 -- start start 2024-03-06T07:41:48.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b01a0 0x7fb5c4159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c3200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5c40b5d50 0x7fb5c414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c3200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5c415ad30 0x7fb5c414c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c3200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4073480 con 0x7fb5c40b01a0 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c3200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4072a20 con 0x7fb5c415ad30 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c3200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4073600 con 0x7fb5c40b5d50 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b01a0 0x7fb5c4159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b01a0 0x7fb5c4159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44982/0 (socket says 172.21.15.97:44982) 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 -- 172.21.15.97:0/3584487529 learned_addr learned my addr 172.21.15.97:0/3584487529 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 -- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5c40b5d50 msgr2=0x7fb5c414bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb5c40b5d50 0x7fb5c414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 -- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5c415ad30 msgr2=0x7fb5c414c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5c415ad30 0x7fb5c414c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.416+0000 7fb5c1e00640 1 -- 172.21.15.97:0/3584487529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c415ea20 con 0x7fb5c40b01a0 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.417+0000 7fb5c1e00640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b01a0 0x7fb5c4159d30 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400c820 tx=0x7fb5b400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.417+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4015070 con 0x7fb5c40b01a0 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.417+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b4007a30 con 0x7fb5c40b01a0 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.417+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c4158c90 con 0x7fb5c40b01a0 2024-03-06T07:41:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.417+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4007de0 con 0x7fb5c40b01a0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.417+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c4159250 con 0x7fb5c40b01a0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.418+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5c404f330 con 0x7fb5c40b01a0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.419+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5b4018070 con 0x7fb5c40b01a0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.420+0000 7fb5ba000640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5a006b500 0x7fb5a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.420+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb5b4089db0 con 0x7fb5c40b01a0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.420+0000 7fb5c1400640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5a006b500 0x7fb5a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.420+0000 7fb5ba000640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7fb5a0071460 0x7fb5a0073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.420+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5b401be50 con 0x7fb5a0071460 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.420+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5b408c4b0 con 0x7fb5c40b01a0 2024-03-06T07:41:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.421+0000 7fb5c2800640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7fb5a0071460 0x7fb5a0073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.421+0000 7fb5c1400640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5a006b500 0x7fb5a006d9c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac036000 tx=0x7fb5ac030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.422+0000 7fb5c2800640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7fb5a0071460 0x7fb5a0073880 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.424+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== osd.4 v2:172.21.15.135:6808/3314712387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb5b401be50 con 0x7fb5a0071460 2024-03-06T07:41:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.438+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 --> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb5c4141fa0 con 0x7fb5a0071460 2024-03-06T07:41:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.439+0000 7fb5ba000640 1 -- 172.21.15.97:0/3584487529 <== osd.4 v2:172.21.15.135:6808/3314712387 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb5c4141fa0 con 0x7fb5a0071460 2024-03-06T07:41:48.442 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:48.442 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:48.442 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7fb5a0071460 msgr2=0x7fb5a0073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.135:6808/3314712387,v1:172.21.15.135:6809/3314712387] conn(0x7fb5a0071460 0x7fb5a0073880 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5a006b500 msgr2=0x7fb5a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb5a006b500 0x7fb5a006d9c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac036000 tx=0x7fb5ac030000 comp rx=0 tx=0).stop 2024-03-06T07:41:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b01a0 msgr2=0x7fb5c4159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 --2- 172.21.15.97:0/3584487529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5c40b01a0 0x7fb5c4159d30 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400c820 tx=0x7fb5b400cce0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c2800640 1 -- 172.21.15.97:0/3584487529 reap_dead start 2024-03-06T07:41:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 shutdown_connections 2024-03-06T07:41:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 >> 172.21.15.97:0/3584487529 conn(0x7fb5c40b4e60 msgr2=0x7fb5c40d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 shutdown_connections 2024-03-06T07:41:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.440+0000 7fb5c3200640 1 -- 172.21.15.97:0/3584487529 wait complete. 2024-03-06T07:41:48.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:41:48.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.670+0000 7fb97c600640 1 Processor -- start 2024-03-06T07:41:48.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.671+0000 7fb97c600640 1 -- start start 2024-03-06T07:41:48.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97c600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb9741444e0 0x7fb9740cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97c600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb9741474c0 0x7fb9740c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97c600640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740ae380 con 0x7fb9740c4e70 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97c600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9740ae660 con 0x7fb9741444e0 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97c600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9740ae940 con 0x7fb9741474c0 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb979c00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb9741444e0 0x7fb9740cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97a600640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb9740c4e70 0x7fb974161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38756/0 (socket says 172.21.15.97:38756) 2024-03-06T07:41:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.672+0000 7fb97a600640 1 -- 172.21.15.97:0/2280503267 learned_addr learned my addr 172.21.15.97:0/2280503267 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:48.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.673+0000 7fb979200640 1 --2- 172.21.15.97:0/2280503267 >> v2:172.21.15.135:3300/0 conn(0x7fb9741474c0 0x7fb9740c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.673+0000 7fb979c00640 1 -- 172.21.15.97:0/2280503267 >> v2:172.21.15.135:3300/0 conn(0x7fb9741474c0 msgr2=0x7fb9740c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.673+0000 7fb979c00640 1 --2- 172.21.15.97:0/2280503267 >> v2:172.21.15.135:3300/0 conn(0x7fb9741474c0 0x7fb9740c4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.673+0000 7fb979c00640 1 -- 172.21.15.97:0/2280503267 >> v1:172.21.15.97:6789/0 conn(0x7fb9740c4e70 legacy=0x7fb974161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.673+0000 7fb979c00640 1 -- 172.21.15.97:0/2280503267 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740aec20 con 0x7fb9741444e0 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.674+0000 7fb979c00640 1 --2- 172.21.15.97:0/2280503267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9741444e0 0x7fb9740cd010 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fb974068c90 tx=0x7fb96802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3137f5b6dcea61af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.674+0000 7fb973e00640 1 -- 172.21.15.97:0/2280503267 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968035020 con 0x7fb9741444e0 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.674+0000 7fb973e00640 1 -- 172.21.15.97:0/2280503267 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb968031830 con 0x7fb9741444e0 2024-03-06T07:41:48.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.674+0000 7fb973e00640 1 -- 172.21.15.97:0/2280503267 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968031c00 con 0x7fb9741444e0 2024-03-06T07:41:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 -- 172.21.15.97:0/2280503267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9741444e0 msgr2=0x7fb9740cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 --2- 172.21.15.97:0/2280503267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9741444e0 0x7fb9740cd010 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fb974068c90 tx=0x7fb96802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 -- 172.21.15.97:0/2280503267 shutdown_connections 2024-03-06T07:41:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 --2- 172.21.15.97:0/2280503267 >> v2:172.21.15.135:3300/0 conn(0x7fb9741474c0 0x7fb9740c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 --2- 172.21.15.97:0/2280503267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9741444e0 0x7fb9740cd010 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 -- 172.21.15.97:0/2280503267 >> 172.21.15.97:0/2280503267 conn(0x7fb9740cc4f0 msgr2=0x7fb9740cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 -- 172.21.15.97:0/2280503267 shutdown_connections 2024-03-06T07:41:48.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.675+0000 7fb97c600640 1 -- 172.21.15.97:0/2280503267 wait complete. 2024-03-06T07:41:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.677+0000 7fb97c600640 1 Processor -- start 2024-03-06T07:41:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.677+0000 7fb97c600640 1 -- start start 2024-03-06T07:41:48.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.678+0000 7fb97c600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 0x7fb97413d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.678+0000 7fb97c600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9741444e0 0x7fb97415d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.678+0000 7fb97c600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9741474c0 0x7fb9740c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.678+0000 7fb97c600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9741448c0 con 0x7fb9740c4e70 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.678+0000 7fb97c600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb974147cb0 con 0x7fb9741474c0 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.678+0000 7fb97c600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb974144a40 con 0x7fb9741444e0 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 0x7fb97413d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 0x7fb97413d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44996/0 (socket says 172.21.15.97:44996) 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 -- 172.21.15.97:0/3733777703 learned_addr learned my addr 172.21.15.97:0/3733777703 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:48.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 -- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9741444e0 msgr2=0x7fb97415d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb97a600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9741474c0 0x7fb9740c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9741444e0 0x7fb97415d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 -- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9741474c0 msgr2=0x7fb9740c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9741474c0 0x7fb9740c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 -- 172.21.15.97:0/3733777703 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740c92d0 con 0x7fb9740c4e70 2024-03-06T07:41:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.679+0000 7fb979c00640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 0x7fb97413d590 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fb968009890 tx=0x7fb9680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.680+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968041070 con 0x7fb9740c4e70 2024-03-06T07:41:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.680+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96803aa10 con 0x7fb9740c4e70 2024-03-06T07:41:48.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.680+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968035040 con 0x7fb9740c4e70 2024-03-06T07:41:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.680+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9740c95b0 con 0x7fb9740c4e70 2024-03-06T07:41:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.680+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97415dcb0 con 0x7fb9740c4e70 2024-03-06T07:41:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.681+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9740c59e0 con 0x7fb9740c4e70 2024-03-06T07:41:48.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.682+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb968038020 con 0x7fb9740c4e70 2024-03-06T07:41:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.683+0000 7fb971600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb95406b4d0 0x7fb95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.683+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb9680aba90 con 0x7fb9740c4e70 2024-03-06T07:41:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.683+0000 7fb979200640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb95406b4d0 0x7fb95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:48.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.684+0000 7fb979200640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb95406b4d0 0x7fb95406d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb97413d380 tx=0x7fb960008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:48.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.688+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9740c59e0 con 0x7fb9740c4e70 2024-03-06T07:41:48.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.881+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb97404f330 con 0x7fb95406b4d0 2024-03-06T07:41:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.881+0000 7fb971600640 1 -- 172.21.15.97:0/3733777703 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb97404f330 con 0x7fb95406b4d0 2024-03-06T07:41:48.884 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:41:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb95406b4d0 msgr2=0x7fb95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb95406b4d0 0x7fb95406d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb97413d380 tx=0x7fb960008040 comp rx=0 tx=0).stop 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 msgr2=0x7fb97413d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 0x7fb97413d590 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fb968009890 tx=0x7fb9680046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 shutdown_connections 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fb95406b4d0 0x7fb95406d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9741474c0 0x7fb9740c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fb9741444e0 0x7fb97415d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 --2- 172.21.15.97:0/3733777703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9740c4e70 0x7fb97413d590 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 >> 172.21.15.97:0/3733777703 conn(0x7fb9740cc4f0 msgr2=0x7fb9740722d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 shutdown_connections 2024-03-06T07:41:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:48.884+0000 7fb97c600640 1 -- 172.21.15.97:0/3733777703 wait complete. 2024-03-06T07:41:48.905 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:41:48.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:49.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.116+0000 7fdca8800640 1 Processor -- start 2024-03-06T07:41:49.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.117+0000 7fdca8800640 1 -- start start 2024-03-06T07:41:49.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca8800640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdca015e170 0x7fdca00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca8800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdca006e570 0x7fdca00b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca8800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 0x7fdca006cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca8800640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca006ea70 con 0x7fdca006e570 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca8800640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca00b3690 con 0x7fdca015e170 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca8800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00b3970 con 0x7fdca00b4990 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.118+0000 7fdca5400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdca006e570 0x7fdca00b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdca006e570 0x7fdca00b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45008/0 (socket says 172.21.15.97:45008) 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 -- 172.21.15.97:0/955650322 learned_addr learned my addr 172.21.15.97:0/955650322 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5e00640 1 --2- 172.21.15.97:0/955650322 >> v2:172.21.15.135:3300/0 conn(0x7fdca015e170 0x7fdca00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 -- 172.21.15.97:0/955650322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 msgr2=0x7fdca006cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 --2- 172.21.15.97:0/955650322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 0x7fdca006cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 -- 172.21.15.97:0/955650322 >> v2:172.21.15.135:3300/0 conn(0x7fdca015e170 msgr2=0x7fdca00ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:49.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 --2- 172.21.15.97:0/955650322 >> v2:172.21.15.135:3300/0 conn(0x7fdca015e170 0x7fdca00ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.119+0000 7fdca5400640 1 -- 172.21.15.97:0/955650322 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca00b3c50 con 0x7fdca006e570 2024-03-06T07:41:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.120+0000 7fdca5400640 1 --2- 172.21.15.97:0/955650322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca00b4450 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fdca00d0aa0 tx=0x7fdc8c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86ac7f39332c4f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.120+0000 7fdca4a00640 1 -- 172.21.15.97:0/955650322 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c0317d0 con 0x7fdca006e570 2024-03-06T07:41:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.120+0000 7fdca4a00640 1 -- 172.21.15.97:0/955650322 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c037070 con 0x7fdca006e570 2024-03-06T07:41:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.120+0000 7fdca4a00640 1 -- 172.21.15.97:0/955650322 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c031b90 con 0x7fdca006e570 2024-03-06T07:41:49.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 -- 172.21.15.97:0/955650322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 msgr2=0x7fdca00b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 --2- 172.21.15.97:0/955650322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca00b4450 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fdca00d0aa0 tx=0x7fdc8c02b870 comp rx=0 tx=0).stop 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 -- 172.21.15.97:0/955650322 shutdown_connections 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 --2- 172.21.15.97:0/955650322 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 0x7fdca006cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 --2- 172.21.15.97:0/955650322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca00b4450 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 --2- 172.21.15.97:0/955650322 >> v2:172.21.15.135:3300/0 conn(0x7fdca015e170 0x7fdca00ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 -- 172.21.15.97:0/955650322 >> 172.21.15.97:0/955650322 conn(0x7fdca00cfa80 msgr2=0x7fdca00cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 -- 172.21.15.97:0/955650322 shutdown_connections 2024-03-06T07:41:49.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.121+0000 7fdca8800640 1 -- 172.21.15.97:0/955650322 wait complete. 2024-03-06T07:41:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.123+0000 7fdca8800640 1 Processor -- start 2024-03-06T07:41:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.123+0000 7fdca8800640 1 -- start start 2024-03-06T07:41:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca8800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca0147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca8800640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 0x7fdca01440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca8800640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdca015e170 0x7fdca00aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca8800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca006df80 con 0x7fdca006e570 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca8800640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00b4dd0 con 0x7fdca015e170 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca8800640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00adda0 con 0x7fdca00b4990 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca0147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca0147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45010/0 (socket says 172.21.15.97:45010) 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.124+0000 7fdca5e00640 1 -- 172.21.15.97:0/336873647 learned_addr learned my addr 172.21.15.97:0/336873647 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5400640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 0x7fdca01440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca6800640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdca015e170 0x7fdca00aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5e00640 1 -- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 msgr2=0x7fdca01440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5e00640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fdca00b4990 0x7fdca01440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5e00640 1 -- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdca015e170 msgr2=0x7fdca00aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5e00640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdca015e170 0x7fdca00aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5e00640 1 -- 172.21.15.97:0/336873647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca0147990 con 0x7fdca006e570 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca5e00640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca0147450 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400c7f0 tx=0x7fdc9400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94015070 con 0x7fdca006e570 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94004220 con 0x7fdca006e570 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.125+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca0144ce0 con 0x7fdca006e570 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.126+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94003610 con 0x7fdca006e570 2024-03-06T07:41:49.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.126+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca01452a0 con 0x7fdca006e570 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.126+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdca004f330 con 0x7fdca006e570 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.128+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc940039c0 con 0x7fdca006e570 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.128+0000 7fdc9a000640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc9408a220 con 0x7fdca006e570 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdc9a000640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fdc80071430 0x7fdc80073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc94003e50 con 0x7fdc80071430 2024-03-06T07:41:49.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc94091050 con 0x7fdca006e570 2024-03-06T07:41:49.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdca5400640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdca6800640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fdc80071430 0x7fdc80073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.129+0000 7fdca5400640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc8006b4d0 0x7fdc8006d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.130+0000 7fdca6800640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fdc80071430 0x7fdc80073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.133+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== osd.5 v2:172.21.15.135:6800/2374976391 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc94003e50 con 0x7fdc80071430 2024-03-06T07:41:49.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.147+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 --> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdca00b2e30 con 0x7fdc80071430 2024-03-06T07:41:49.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.148+0000 7fdc9a000640 1 -- 172.21.15.97:0/336873647 <== osd.5 v2:172.21.15.135:6800/2374976391 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdca00b2e30 con 0x7fdc80071430 2024-03-06T07:41:49.151 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:49.151 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:49.151 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fdc80071430 msgr2=0x7fdc80073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.135:6800/2374976391,v1:172.21.15.135:6801/2374976391] conn(0x7fdc80071430 0x7fdc80073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc8006b4d0 msgr2=0x7fdc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fdc8006b4d0 0x7fdc8006d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c03d040 comp rx=0 tx=0).stop 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 msgr2=0x7fdca0147450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 --2- 172.21.15.97:0/336873647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdca006e570 0x7fdca0147450 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400c7f0 tx=0x7fdc9400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca6800640 1 -- 172.21.15.97:0/336873647 reap_dead start 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 shutdown_connections 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 >> 172.21.15.97:0/336873647 conn(0x7fdca00cfa80 msgr2=0x7fdca01582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 shutdown_connections 2024-03-06T07:41:49.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.149+0000 7fdca8800640 1 -- 172.21.15.97:0/336873647 wait complete. 2024-03-06T07:41:49.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:41:49.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.372+0000 7fd1b7400640 1 Processor -- start 2024-03-06T07:41:49.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.372+0000 7fd1b7400640 1 -- start start 2024-03-06T07:41:49.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1b7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1b00c4930 0x7fd1b00c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1b7400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b0145b30 con 0x7fd1b00c4d10 2024-03-06T07:41:49.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1b7400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b0142ce0 con 0x7fd1b00c5880 2024-03-06T07:41:49.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1b7400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00cbca0 con 0x7fd1b00c4930 2024-03-06T07:41:49.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1afe00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd1b00c4d10 0x7fd1b00c5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38758/0 (socket says 172.21.15.97:38758) 2024-03-06T07:41:49.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1afe00640 1 -- 172.21.15.97:0/222374032 learned_addr learned my addr 172.21.15.97:0/222374032 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:49.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.374+0000 7fd1b4a00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1b00c4930 0x7fd1b00c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948099223 0 0) 0x7fd19c001110 con 0x7fd1b00c4d10 2024-03-06T07:41:49.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd198003520 con 0x7fd1b00c4d10 2024-03-06T07:41:49.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2255419161 0 0) 0x7fd198003520 con 0x7fd1b00c4d10 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1b00c4930 msgr2=0x7fd1b00c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1b00c4930 0x7fd1b00c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 >> v1:172.21.15.135:6789/0 conn(0x7fd1b00c5880 legacy=0x7fd1b0165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b00cbf80 con 0x7fd1b00c4d10 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fd19c001840 con 0x7fd1b00c4d10 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 >> v1:172.21.15.97:6789/0 conn(0x7fd1b00c4d10 legacy=0x7fd1b00c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1980039e0 0x7fd198003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd198004800 0x7fd198004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1980051c0 0x7fd19800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00cbca0 con 0x7fd198004800 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19800c060 con 0x7fd1980039e0 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19800c260 con 0x7fd1980051c0 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.375+0000 7fd1b4a00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1b00c4930 0x7fd1b00c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1afe00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd198004800 0x7fd198004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1b4a00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1980039e0 0x7fd198003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1b5400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1980051c0 0x7fd19800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1afe00640 1 -- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1980051c0 msgr2=0x7fd19800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:49.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1afe00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1980051c0 0x7fd19800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1afe00640 1 -- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1980039e0 msgr2=0x7fd198003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:49.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1afe00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1980039e0 0x7fd198003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1afe00640 1 -- 172.21.15.97:0/222374032 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd198003520 con 0x7fd198004800 2024-03-06T07:41:49.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.376+0000 7fd1b5400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd1980051c0 0x7fd19800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:49.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1afe00640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd198004800 0x7fd198004c80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c4c0 tx=0x7fd198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2117cafabf79dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00c0e0 con 0x7fd198004800 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c030070 con 0x7fd198004800 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1af400640 1 -- 172.21.15.97:0/222374032 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c02ce60 con 0x7fd198004800 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1b7400640 1 -- 172.21.15.97:0/222374032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd198004800 msgr2=0x7fd198004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1b7400640 1 --2- 172.21.15.97:0/222374032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd198004800 0x7fd198004c80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c4c0 tx=0x7fd198005830 comp rx=0 tx=0).stop 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.378+0000 7fd1b5400640 1 -- 172.21.15.97:0/222374032 reap_dead start 2024-03-06T07:41:49.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.379+0000 7fd1b7400640 1 -- 172.21.15.97:0/222374032 shutdown_connections 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.379+0000 7fd1b7400640 1 -- 172.21.15.97:0/222374032 >> 172.21.15.97:0/222374032 conn(0x7fd1b00cc4f0 msgr2=0x7fd1b00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.379+0000 7fd1b7400640 1 -- 172.21.15.97:0/222374032 shutdown_connections 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.379+0000 7fd1b7400640 1 -- 172.21.15.97:0/222374032 wait complete. 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.380+0000 7fd1b7400640 1 Processor -- start 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.381+0000 7fd1b7400640 1 -- start start 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.381+0000 7fd1b7400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd198004800 0x7fd1b00af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 0x7fd1b0141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b7400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1b01d9280 0x7fd1b00c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b7400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b006e600 con 0x7fd1b01e2690 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b7400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00ca3b0 con 0x7fd1b01d9280 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b7400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00cc260 con 0x7fd198004800 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 0x7fd1b0141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 0x7fd1b0141860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45034/0 (socket says 172.21.15.97:45034) 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 -- 172.21.15.97:0/1162036489 learned_addr learned my addr 172.21.15.97:0/1162036489 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b4a00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd198004800 0x7fd1b00af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1b5400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1b01d9280 0x7fd1b00c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 -- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd198004800 msgr2=0x7fd1b00af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd198004800 0x7fd1b00af530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 -- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1b01d9280 msgr2=0x7fd1b00c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1b01d9280 0x7fd1b00c7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.382+0000 7fd1afe00640 1 -- 172.21.15.97:0/1162036489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b00cd5b0 con 0x7fd1b01e2690 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.383+0000 7fd1afe00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 0x7fd1b0141860 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c520 tx=0x7fd1b006f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.383+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c046070 con 0x7fd1b01e2690 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.383+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c0049e0 con 0x7fd1b01e2690 2024-03-06T07:41:49.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.383+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b00cd890 con 0x7fd1b01e2690 2024-03-06T07:41:49.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.383+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c004db0 con 0x7fd1b01e2690 2024-03-06T07:41:49.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.383+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b00cde50 con 0x7fd1b01e2690 2024-03-06T07:41:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.384+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b004f330 con 0x7fd1b01e2690 2024-03-06T07:41:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.385+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd19c007070 con 0x7fd1b01e2690 2024-03-06T07:41:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.386+0000 7fd1acc00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.387+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd19c0ab220 con 0x7fd1b01e2690 2024-03-06T07:41:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.387+0000 7fd1b4a00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.387+0000 7fd1b4a00640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd19006b4d0 0x7fd19006d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00cf950 tx=0x7fd1a4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.391+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1b0145b30 con 0x7fd1b01e2690 2024-03-06T07:41:49.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.585+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1b00c6280 con 0x7fd19006b4d0 2024-03-06T07:41:49.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.586+0000 7fd1acc00640 1 -- 172.21.15.97:0/1162036489 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd1b00c6280 con 0x7fd19006b4d0 2024-03-06T07:41:49.588 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:41:49.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd19006b4d0 msgr2=0x7fd19006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd19006b4d0 0x7fd19006d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00cf950 tx=0x7fd1a4008040 comp rx=0 tx=0).stop 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 msgr2=0x7fd1b0141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 0x7fd1b0141860 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c520 tx=0x7fd1b006f050 comp rx=0 tx=0).stop 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 shutdown_connections 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1b01d9280 0x7fd1b00c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b01e2690 0x7fd1b0141860 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 --2- 172.21.15.97:0/1162036489 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd198004800 0x7fd1b00af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 >> 172.21.15.97:0/1162036489 conn(0x7fd1b00cc4f0 msgr2=0x7fd1b0070b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:49.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 shutdown_connections 2024-03-06T07:41:49.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.589+0000 7fd1b7400640 1 -- 172.21.15.97:0/1162036489 wait complete. 2024-03-06T07:41:49.609 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:41:49.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:49.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.821+0000 7fabaa200640 1 Processor -- start 2024-03-06T07:41:49.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.821+0000 7fabaa200640 1 -- start start 2024-03-06T07:41:49.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fabaa200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faba4145980 0x7faba415fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fabaa200640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7faba415ff40 0x7faba406dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fabaa200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba4160350 con 0x7faba40ced20 2024-03-06T07:41:49.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fabaa200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba4070e40 con 0x7faba4145980 2024-03-06T07:41:49.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fabaa200640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40cf1e0 con 0x7faba415ff40 2024-03-06T07:41:49.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7faba3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faba4145980 0x7faba415fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fab9be00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7faba40ced20 0x7faba40712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38768/0 (socket says 172.21.15.97:38768) 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7faba3400640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faba4145980 0x7faba415fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45044/0 (socket says 172.21.15.97:45044) 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.823+0000 7fab9be00640 1 -- 172.21.15.97:0/1540125990 learned_addr learned my addr 172.21.15.97:0/1540125990 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba3e00640 1 --2- 172.21.15.97:0/1540125990 >> v2:172.21.15.135:3300/0 conn(0x7faba415ff40 0x7faba406dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba3400640 1 -- 172.21.15.97:0/1540125990 >> v2:172.21.15.135:3300/0 conn(0x7faba415ff40 msgr2=0x7faba406dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba3400640 1 --2- 172.21.15.97:0/1540125990 >> v2:172.21.15.135:3300/0 conn(0x7faba415ff40 0x7faba406dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba3400640 1 -- 172.21.15.97:0/1540125990 >> v1:172.21.15.97:6789/0 conn(0x7faba40ced20 legacy=0x7faba40712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba3400640 1 -- 172.21.15.97:0/1540125990 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40cf4c0 con 0x7faba4145980 2024-03-06T07:41:49.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba2a00640 1 -- 172.21.15.97:0/1540125990 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838974051 0 0) 0x7fab8c001110 con 0x7faba40ced20 2024-03-06T07:41:49.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.824+0000 7faba3e00640 1 --2- 172.21.15.97:0/1540125990 >> v2:172.21.15.135:3300/0 conn(0x7faba415ff40 0x7faba406dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.825+0000 7faba3400640 1 --2- 172.21.15.97:0/1540125990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba4145980 0x7faba415fa00 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7faba415f940 tx=0x7fab9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72bf91e9333fb1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.825+0000 7faba2a00640 1 -- 172.21.15.97:0/1540125990 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94035020 con 0x7faba4145980 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.825+0000 7faba2a00640 1 -- 172.21.15.97:0/1540125990 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab94031830 con 0x7faba4145980 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.825+0000 7faba2a00640 1 -- 172.21.15.97:0/1540125990 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94031c00 con 0x7faba4145980 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 -- 172.21.15.97:0/1540125990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba4145980 msgr2=0x7faba415fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 --2- 172.21.15.97:0/1540125990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba4145980 0x7faba415fa00 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7faba415f940 tx=0x7fab9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 -- 172.21.15.97:0/1540125990 shutdown_connections 2024-03-06T07:41:49.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 --2- 172.21.15.97:0/1540125990 >> v2:172.21.15.135:3300/0 conn(0x7faba415ff40 0x7faba406dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 --2- 172.21.15.97:0/1540125990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba4145980 0x7faba415fa00 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 -- 172.21.15.97:0/1540125990 >> 172.21.15.97:0/1540125990 conn(0x7faba40cfa80 msgr2=0x7faba40cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:49.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 -- 172.21.15.97:0/1540125990 shutdown_connections 2024-03-06T07:41:49.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.826+0000 7fabaa200640 1 -- 172.21.15.97:0/1540125990 wait complete. 2024-03-06T07:41:49.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.828+0000 7fabaa200640 1 Processor -- start 2024-03-06T07:41:49.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.828+0000 7fabaa200640 1 -- start start 2024-03-06T07:41:49.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7fabaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba40ced20 0x7faba40aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7fabaa200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faba4145980 0x7faba41f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7fabaa200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faba415ff40 0x7faba414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7fabaa200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4160740 con 0x7faba40ced20 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7fabaa200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4070ad0 con 0x7faba4145980 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7fabaa200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba41608c0 con 0x7faba415ff40 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7faba3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba40ced20 0x7faba40aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7faba3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba40ced20 0x7faba40aed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45056/0 (socket says 172.21.15.97:45056) 2024-03-06T07:41:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.829+0000 7faba3400640 1 -- 172.21.15.97:0/1252489735 learned_addr learned my addr 172.21.15.97:0/1252489735 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:49.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7faba3400640 1 -- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faba415ff40 msgr2=0x7faba414c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7faba3400640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7faba415ff40 0x7faba414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7faba3400640 1 -- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faba4145980 msgr2=0x7faba41f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7faba3400640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faba4145980 0x7faba41f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7faba3400640 1 -- 172.21.15.97:0/1252489735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba414ee40 con 0x7faba40ced20 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7faba3400640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba40ced20 0x7faba40aed60 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fab9402bee0 tx=0x7fab940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94041070 con 0x7faba40ced20 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.830+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9403aa30 con 0x7faba40ced20 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.831+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba414f120 con 0x7faba40ced20 2024-03-06T07:41:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.831+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94035040 con 0x7faba40ced20 2024-03-06T07:41:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.831+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba414f6e0 con 0x7faba40ced20 2024-03-06T07:41:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.831+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faba404f330 con 0x7faba40ced20 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.833+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab94038020 con 0x7faba40ced20 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.833+0000 7fab9b400640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab8006b4d0 0x7fab8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fab940ab6d0 con 0x7faba40ced20 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7fab9be00640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab8006b4d0 0x7fab8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7fab9b400640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fab80071430 0x7fab80073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab94031830 con 0x7fab80071430 2024-03-06T07:41:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab940b2050 con 0x7faba40ced20 2024-03-06T07:41:49.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7faba3e00640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fab80071430 0x7fab80073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:49.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.834+0000 7fab9be00640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab8006b4d0 0x7fab8006d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fab8c007fd0 tx=0x7fab8c007960 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.835+0000 7faba3e00640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fab80071430 0x7fab80073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:49.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.838+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== osd.6 v2:172.21.15.135:6824/3958261964 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fab94031830 con 0x7fab80071430 2024-03-06T07:41:49.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.852+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 --> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faba41f7670 con 0x7fab80071430 2024-03-06T07:41:49.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.853+0000 7fab9b400640 1 -- 172.21.15.97:0/1252489735 <== osd.6 v2:172.21.15.135:6824/3958261964 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7faba41f7670 con 0x7fab80071430 2024-03-06T07:41:49.855 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:49.856 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:49.856 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.853+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fab80071430 msgr2=0x7fab80073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.135:6824/3958261964,v1:172.21.15.135:6825/3958261964] conn(0x7fab80071430 0x7fab80073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab8006b4d0 msgr2=0x7fab8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fab8006b4d0 0x7fab8006d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fab8c007fd0 tx=0x7fab8c007960 comp rx=0 tx=0).stop 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba40ced20 msgr2=0x7faba40aed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 --2- 172.21.15.97:0/1252489735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faba40ced20 0x7faba40aed60 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fab9402bee0 tx=0x7fab940046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7faba3e00640 1 -- 172.21.15.97:0/1252489735 reap_dead start 2024-03-06T07:41:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 shutdown_connections 2024-03-06T07:41:49.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 >> 172.21.15.97:0/1252489735 conn(0x7faba40cfa80 msgr2=0x7faba40712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:49.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 shutdown_connections 2024-03-06T07:41:49.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:49.854+0000 7fabaa200640 1 -- 172.21.15.97:0/1252489735 wait complete. 2024-03-06T07:41:49.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:41:50.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.081+0000 7f955d200640 1 Processor -- start 2024-03-06T07:41:50.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.081+0000 7f955d200640 1 -- start start 2024-03-06T07:41:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f955d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f95580d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f955d200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f95580d0780 0x7f955815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f955d200640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955815c3a0 con 0x7f955815bfc0 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f955d200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95580cdc20 con 0x7f95580d0780 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f955d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95580cd660 con 0x7f95580b4390 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f95580d0780 0x7f955815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f95580d0780 0x7f955815fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45070/0 (socket says 172.21.15.97:45070) 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 -- 172.21.15.97:0/3879073973 learned_addr learned my addr 172.21.15.97:0/3879073973 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 -- 172.21.15.97:0/3879073973 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 msgr2=0x7f95580d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9555400640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f95580d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f95580d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 -- 172.21.15.97:0/3879073973 >> v1:172.21.15.135:6789/0 conn(0x7f955815bfc0 legacy=0x7f95580b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9556800640 1 -- 172.21.15.97:0/3879073973 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95580cd940 con 0x7f95580d0780 2024-03-06T07:41:50.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.083+0000 7f9555400640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f95580d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:41:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.084+0000 7f9556800640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95580d0780 0x7f955815fab0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f9550009e50 tx=0x7f955002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1220e6c8139b620f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.085+0000 7f9554a00640 1 -- 172.21.15.97:0/3879073973 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955002ca70 con 0x7f95580d0780 2024-03-06T07:41:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.085+0000 7f9554a00640 1 -- 172.21.15.97:0/3879073973 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955002cc10 con 0x7f95580d0780 2024-03-06T07:41:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.085+0000 7f9554a00640 1 -- 172.21.15.97:0/3879073973 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500343f0 con 0x7f95580d0780 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.085+0000 7f955d200640 1 -- 172.21.15.97:0/3879073973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95580d0780 msgr2=0x7f955815fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.085+0000 7f955d200640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95580d0780 0x7f955815fab0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f9550009e50 tx=0x7f955002bed0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.086+0000 7f955d200640 1 -- 172.21.15.97:0/3879073973 shutdown_connections 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.086+0000 7f955d200640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95580d0780 0x7f955815fab0 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.086+0000 7f955d200640 1 --2- 172.21.15.97:0/3879073973 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f95580d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.086+0000 7f955d200640 1 -- 172.21.15.97:0/3879073973 >> 172.21.15.97:0/3879073973 conn(0x7f95580b3800 msgr2=0x7f95580b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.086+0000 7f955d200640 1 -- 172.21.15.97:0/3879073973 shutdown_connections 2024-03-06T07:41:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.086+0000 7f955d200640 1 -- 172.21.15.97:0/3879073973 wait complete. 2024-03-06T07:41:50.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.088+0000 7f955d200640 1 Processor -- start 2024-03-06T07:41:50.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.088+0000 7f955d200640 1 -- start start 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f955d200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f955815a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f955d200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95580d0780 0x7f95581e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f955d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 0x7f9558149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f955d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95580b1d80 con 0x7f955815bfc0 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f955d200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95580cfd20 con 0x7f95580d0780 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f955d200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95580d0c10 con 0x7f95580b4390 2024-03-06T07:41:50.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.089+0000 7f9556800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 0x7f9558149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 0x7f9558149b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45080/0 (socket says 172.21.15.97:45080) 2024-03-06T07:41:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 -- 172.21.15.97:0/3979185819 learned_addr learned my addr 172.21.15.97:0/3979185819 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9555e00640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f955815a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9555400640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95580d0780 0x7f95581e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 -- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 msgr2=0x7f955815a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f955815a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 -- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95580d0780 msgr2=0x7f95581e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95580d0780 0x7f95581e4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 -- 172.21.15.97:0/3979185819 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955806ccf0 con 0x7f955815bfc0 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f9556800640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 0x7f9558149b40 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f9550036040 tx=0x7f9550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.090+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550043020 con 0x7f955815bfc0 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.091+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955003e450 con 0x7f955815bfc0 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.091+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955003e820 con 0x7f955815bfc0 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.091+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955806cfd0 con 0x7f955815bfc0 2024-03-06T07:41:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.091+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955806d590 con 0x7f955815bfc0 2024-03-06T07:41:50.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.092+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955804f330 con 0x7f955815bfc0 2024-03-06T07:41:50.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.093+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9550046020 con 0x7f955815bfc0 2024-03-06T07:41:50.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.094+0000 7f954e000640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f953806b4d0 0x7f953806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.094+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f95500ac940 con 0x7f955815bfc0 2024-03-06T07:41:50.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.094+0000 7f9555e00640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f953806b4d0 0x7f953806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.095+0000 7f9555e00640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f953806b4d0 0x7f953806d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f955806dcf0 tx=0x7f9548009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.098+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f955004d030 con 0x7f955815bfc0 2024-03-06T07:41:50.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.293+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f955815af00 con 0x7f953806b4d0 2024-03-06T07:41:50.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.293+0000 7f954e000640 1 -- 172.21.15.97:0/3979185819 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f955815af00 con 0x7f953806b4d0 2024-03-06T07:41:50.296 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:41:50.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f953806b4d0 msgr2=0x7f953806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f953806b4d0 0x7f953806d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f955806dcf0 tx=0x7f9548009040 comp rx=0 tx=0).stop 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 msgr2=0x7f9558149b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 0x7f9558149b40 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f9550036040 tx=0x7f9550004a40 comp rx=0 tx=0).stop 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 shutdown_connections 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f953806b4d0 0x7f953806d990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f955815bfc0 0x7f9558149b40 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95580d0780 0x7f95581e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 --2- 172.21.15.97:0/3979185819 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f95580b4390 0x7f955815a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 >> 172.21.15.97:0/3979185819 conn(0x7f95580b3800 msgr2=0x7f95580cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:50.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 shutdown_connections 2024-03-06T07:41:50.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.296+0000 7f955d200640 1 -- 172.21.15.97:0/3979185819 wait complete. 2024-03-06T07:41:50.316 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:41:50.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:41:50.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.514+0000 7f2067200640 1 Processor -- start 2024-03-06T07:41:50.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.514+0000 7f2067200640 1 -- start start 2024-03-06T07:41:50.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.516+0000 7f2067200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20600b3a10 0x7f206015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.516+0000 7f2067200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2060071a90 0x7f2060161fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.516+0000 7f2067200640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206015e590 con 0x7f2060161b80 2024-03-06T07:41:50.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.516+0000 7f2067200640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f206015f590 con 0x7f2060071a90 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.516+0000 7f2067200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206015ebc0 con 0x7f20600b3a10 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2060071a90 0x7f2060161fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2060071a90 0x7f2060161fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45086/0 (socket says 172.21.15.97:45086) 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f205f400640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2060161b80 0x7f2060071380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38772/0 (socket says 172.21.15.97:38772) 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 -- 172.21.15.97:0/4279500624 learned_addr learned my addr 172.21.15.97:0/4279500624 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f205fe00640 1 --2- 172.21.15.97:0/4279500624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20600b3a10 0x7f206015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 -- 172.21.15.97:0/4279500624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20600b3a10 msgr2=0x7f206015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:50.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 --2- 172.21.15.97:0/4279500624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20600b3a10 0x7f206015f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 -- 172.21.15.97:0/4279500624 >> v1:172.21.15.97:6789/0 conn(0x7f2060161b80 legacy=0x7f2060071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f2065200640 1 -- 172.21.15.97:0/4279500624 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2060162600 con 0x7f2060071a90 2024-03-06T07:41:50.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.517+0000 7f205ea00640 1 -- 172.21.15.97:0/4279500624 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884671482 0 0) 0x7f206015e590 con 0x7f2060161b80 2024-03-06T07:41:50.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.518+0000 7f2065200640 1 --2- 172.21.15.97:0/4279500624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2060071a90 0x7f2060161fe0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f2058009e50 tx=0x7f205802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110567ad2f9dd21e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.518+0000 7f205ea00640 1 -- 172.21.15.97:0/4279500624 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205802ca70 con 0x7f2060071a90 2024-03-06T07:41:50.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f205ea00640 1 -- 172.21.15.97:0/4279500624 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205802cc10 con 0x7f2060071a90 2024-03-06T07:41:50.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f205ea00640 1 -- 172.21.15.97:0/4279500624 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580343f0 con 0x7f2060071a90 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 -- 172.21.15.97:0/4279500624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2060071a90 msgr2=0x7f2060161fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 --2- 172.21.15.97:0/4279500624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2060071a90 0x7f2060161fe0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f2058009e50 tx=0x7f205802bed0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 -- 172.21.15.97:0/4279500624 shutdown_connections 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 --2- 172.21.15.97:0/4279500624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2060071a90 0x7f2060161fe0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 --2- 172.21.15.97:0/4279500624 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f20600b3a10 0x7f206015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 -- 172.21.15.97:0/4279500624 >> 172.21.15.97:0/4279500624 conn(0x7f20600b2220 msgr2=0x7f20600b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 -- 172.21.15.97:0/4279500624 shutdown_connections 2024-03-06T07:41:50.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.519+0000 7f2067200640 1 -- 172.21.15.97:0/4279500624 wait complete. 2024-03-06T07:41:50.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.521+0000 7f2067200640 1 Processor -- start 2024-03-06T07:41:50.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.521+0000 7f2067200640 1 -- start start 2024-03-06T07:41:50.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f2067200640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2060071a90 0x7f2060156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f2067200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20600b3a10 0x7f20601f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f2067200640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2060161b80 0x7f20601f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f2067200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20600b56a0 con 0x7f20600b3a10 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f2067200640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206015e850 con 0x7f2060161b80 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f2067200640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20600b3df0 con 0x7f2060071a90 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f205f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20600b3a10 0x7f20601f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f205f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20600b3a10 0x7f20601f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45092/0 (socket says 172.21.15.97:45092) 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.522+0000 7f205f400640 1 -- 172.21.15.97:0/3884678654 learned_addr learned my addr 172.21.15.97:0/3884678654 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205f400640 1 -- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2060071a90 msgr2=0x7f2060156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205fe00640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2060071a90 0x7f2060156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205f400640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f2060071a90 0x7f2060156700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205f400640 1 -- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2060161b80 msgr2=0x7f20601f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205f400640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2060161b80 0x7f20601f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205f400640 1 -- 172.21.15.97:0/3884678654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20601570a0 con 0x7f20600b3a10 2024-03-06T07:41:50.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f205f400640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20600b3a10 0x7f20601f2f90 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f204800c3f0 tx=0x7f204800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20480074e0 con 0x7f20600b3a10 2024-03-06T07:41:50.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20480047e0 con 0x7f20600b3a10 2024-03-06T07:41:50.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.523+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048004c10 con 0x7f20600b3a10 2024-03-06T07:41:50.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.524+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2060157380 con 0x7f20600b3a10 2024-03-06T07:41:50.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.524+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2060157940 con 0x7f20600b3a10 2024-03-06T07:41:50.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.524+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2060157d20 con 0x7f20600b3a10 2024-03-06T07:41:50.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.526+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2048011070 con 0x7f20600b3a10 2024-03-06T07:41:50.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.526+0000 7f2057e00640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203c06b4d0 0x7f203c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2048089ad0 con 0x7f20600b3a10 2024-03-06T07:41:50.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f205fe00640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203c06b4d0 0x7f203c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f2057e00640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7f203c071430 0x7f203c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f203c073f00 con 0x7f203c071430 2024-03-06T07:41:50.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f204808e2c0 con 0x7f20600b3a10 2024-03-06T07:41:50.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f205fe00640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203c06b4d0 0x7f203c06d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2050006fd0 tx=0x7f2050008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.527+0000 7f2065200640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7f203c071430 0x7f203c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.528+0000 7f2065200640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7f203c071430 0x7f203c073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.530+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== osd.7 v2:172.21.15.135:6816/3893802268 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f203c073f00 con 0x7f203c071430 2024-03-06T07:41:50.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.545+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 --> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f206004f330 con 0x7f203c071430 2024-03-06T07:41:50.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.546+0000 7f2057e00640 1 -- 172.21.15.97:0/3884678654 <== osd.7 v2:172.21.15.135:6816/3893802268 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f206004f330 con 0x7f203c071430 2024-03-06T07:41:50.548 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-06T07:41:50.548 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:41:50.549 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-06T07:41:50.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.546+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7f203c071430 msgr2=0x7f203c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.546+0000 7f2067200640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.135:6816/3893802268,v1:172.21.15.135:6817/3893802268] conn(0x7f203c071430 0x7f203c073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203c06b4d0 msgr2=0x7f203c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f203c06b4d0 0x7f203c06d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2050006fd0 tx=0x7f2050008040 comp rx=0 tx=0).stop 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20600b3a10 msgr2=0x7f20601f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 --2- 172.21.15.97:0/3884678654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20600b3a10 0x7f20601f2f90 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f204800c3f0 tx=0x7f204800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2065200640 1 -- 172.21.15.97:0/3884678654 reap_dead start 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 shutdown_connections 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 >> 172.21.15.97:0/3884678654 conn(0x7f20600b2220 msgr2=0x7f2060071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 shutdown_connections 2024-03-06T07:41:50.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.547+0000 7f2067200640 1 -- 172.21.15.97:0/3884678654 wait complete. 2024-03-06T07:41:50.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:41:50.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.761+0000 7f8c71600640 1 Processor -- start 2024-03-06T07:41:50.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.761+0000 7f8c71600640 1 -- start start 2024-03-06T07:41:50.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.762+0000 7f8c71600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 0x7f8c6c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.762+0000 7f8c71600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.762+0000 7f8c71600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c145b30 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.762+0000 7f8c71600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c142ce0 con 0x7f8c6c0c4930 2024-03-06T07:41:50.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.762+0000 7f8c71600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0cbca0 con 0x7f8c6c0c4d10 2024-03-06T07:41:50.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.763+0000 7f8c6aa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 0x7f8c6c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.763+0000 7f8c62000640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.763+0000 7f8c6aa00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 0x7f8c6c0c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.97:42272/0 (socket says 172.21.15.97:42272) 2024-03-06T07:41:50.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.763+0000 7f8c6aa00640 1 -- 172.21.15.97:0/344876464 learned_addr learned my addr 172.21.15.97:0/344876464 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:50.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.764+0000 7f8c62000640 1 -- 172.21.15.97:0/344876464 >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 msgr2=0x7f8c6c0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:50.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.764+0000 7f8c62000640 1 --2- 172.21.15.97:0/344876464 >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 0x7f8c6c0c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.764+0000 7f8c62000640 1 -- 172.21.15.97:0/344876464 >> v1:172.21.15.135:6789/0 conn(0x7f8c6c0c56b0 legacy=0x7f8c6c06d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.764+0000 7f8c62000640 1 -- 172.21.15.97:0/344876464 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c0cbf80 con 0x7f8c6c0c4d10 2024-03-06T07:41:50.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.764+0000 7f8c6aa00640 1 --2- 172.21.15.97:0/344876464 >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 0x7f8c6c0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:41:50.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.765+0000 7f8c62000640 1 --2- 172.21.15.97:0/344876464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0c5170 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c540097a0 tx=0x7f8c5402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15e048d398199fba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c6a000640 1 -- 172.21.15.97:0/344876464 <== mon.2 v2:172.21.15.135:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c540315a0 con 0x7f8c6c0c4d10 2024-03-06T07:41:50.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c6a000640 1 -- 172.21.15.97:0/344876464 <== mon.2 v2:172.21.15.135:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c54037070 con 0x7f8c6c0c4d10 2024-03-06T07:41:50.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c6a000640 1 -- 172.21.15.97:0/344876464 <== mon.2 v2:172.21.15.135:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c54031960 con 0x7f8c6c0c4d10 2024-03-06T07:41:50.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c71600640 1 -- 172.21.15.97:0/344876464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 msgr2=0x7f8c6c0c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c71600640 1 --2- 172.21.15.97:0/344876464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0c5170 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c540097a0 tx=0x7f8c5402b840 comp rx=0 tx=0).stop 2024-03-06T07:41:50.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c71600640 1 -- 172.21.15.97:0/344876464 shutdown_connections 2024-03-06T07:41:50.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.766+0000 7f8c71600640 1 --2- 172.21.15.97:0/344876464 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0c5170 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.767+0000 7f8c71600640 1 --2- 172.21.15.97:0/344876464 >> v2:172.21.15.135:3300/0 conn(0x7f8c6c0c4930 0x7f8c6c0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.767+0000 7f8c71600640 1 -- 172.21.15.97:0/344876464 >> 172.21.15.97:0/344876464 conn(0x7f8c6c0cc4f0 msgr2=0x7f8c6c0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:50.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.767+0000 7f8c71600640 1 -- 172.21.15.97:0/344876464 shutdown_connections 2024-03-06T07:41:50.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.767+0000 7f8c71600640 1 -- 172.21.15.97:0/344876464 wait complete. 2024-03-06T07:41:50.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.768+0000 7f8c71600640 1 Processor -- start 2024-03-06T07:41:50.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.768+0000 7f8c71600640 1 -- start start 2024-03-06T07:41:50.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.769+0000 7f8c71600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c0c4930 0x7f8c6c13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.769+0000 7f8c71600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.769+0000 7f8c71600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 0x7f8c6c1f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.769+0000 7f8c71600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0cc260 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.769+0000 7f8c71600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0ca3b0 con 0x7f8c6c0c4930 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.769+0000 7f8c71600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0c5b40 con 0x7f8c6c0c4d10 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 0x7f8c6c1f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 0x7f8c6c1f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45094/0 (socket says 172.21.15.97:45094) 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 -- 172.21.15.97:0/305266000 learned_addr learned my addr 172.21.15.97:0/305266000 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6aa00640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c0c4930 0x7f8c6c13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c62000640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 -- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 msgr2=0x7f8c6c0cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 -- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c0c4930 msgr2=0x7f8c6c13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c0c4930 0x7f8c6c13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.770+0000 7f8c6b400640 1 -- 172.21.15.97:0/305266000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c1f45e0 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.771+0000 7f8c6b400640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 0x7f8c6c1f3eb0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400c760 tx=0x7f8c6400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.771+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64003b70 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.771+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64003d30 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.771+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400aa90 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.771+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c1f48c0 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.771+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c1f4e80 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.772+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c04f330 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.773+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c6400ac30 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.774+0000 7f8c63400640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c4806b4d0 0x7f8c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.774+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8c64089ed0 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.774+0000 7f8c6aa00640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c4806b4d0 0x7f8c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.775+0000 7f8c6aa00640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c4806b4d0 0x7f8c4806d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c13f090 tx=0x7f8c5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:50.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.778+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c6c145b30 con 0x7f8c6c0c56b0 2024-03-06T07:41:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.957+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c6c144010 con 0x7f8c4806b4d0 2024-03-06T07:41:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.958+0000 7f8c63400640 1 -- 172.21.15.97:0/305266000 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8c6c144010 con 0x7f8c4806b4d0 2024-03-06T07:41:50.961 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:41:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c4806b4d0 msgr2=0x7f8c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c4806b4d0 0x7f8c4806d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c13f090 tx=0x7f8c5c008040 comp rx=0 tx=0).stop 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 msgr2=0x7f8c6c1f3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 0x7f8c6c1f3eb0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400c760 tx=0x7f8c6400cc20 comp rx=0 tx=0).stop 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 shutdown_connections 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f8c4806b4d0 0x7f8c4806d990 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c6c0c56b0 0x7f8c6c1f3eb0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f8c6c0c4d10 0x7f8c6c0cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 --2- 172.21.15.97:0/305266000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c0c4930 0x7f8c6c13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 >> 172.21.15.97:0/305266000 conn(0x7f8c6c0cc4f0 msgr2=0x7f8c6c0c61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 shutdown_connections 2024-03-06T07:41:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:50.961+0000 7f8c71600640 1 -- 172.21.15.97:0/305266000 wait complete. 2024-03-06T07:41:50.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:41:51.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.178+0000 7f5b05a00640 1 Processor -- start 2024-03-06T07:41:51.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.178+0000 7f5b05a00640 1 -- start start 2024-03-06T07:41:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5b05a00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5b0015f9d0 0x7f5b0015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5b05a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b00070b10 0x7f5b00070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5b05a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0006cef0 con 0x7f5b00144c00 2024-03-06T07:41:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5b05a00640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0006d1d0 con 0x7f5b0015f9d0 2024-03-06T07:41:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5b05a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0006d4b0 con 0x7f5b00070b10 2024-03-06T07:41:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5afea00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5b00144c00 0x7f5b0006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38784/0 (socket says 172.21.15.97:38784) 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5afea00640 1 -- 172.21.15.97:0/2754967903 learned_addr learned my addr 172.21.15.97:0/2754967903 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5afe000640 1 --2- 172.21.15.97:0/2754967903 >> v2:172.21.15.97:3300/0 conn(0x7f5b0015f9d0 0x7f5b0015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.180+0000 7f5afd600640 1 -- 172.21.15.97:0/2754967903 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950863803 0 0) 0x7f5b0006cef0 con 0x7f5b00144c00 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.181+0000 7f5aff400640 1 --2- 172.21.15.97:0/2754967903 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b00070b10 0x7f5b00070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.181+0000 7f5afd600640 1 -- 172.21.15.97:0/2754967903 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae4003520 con 0x7f5b00144c00 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.181+0000 7f5afe000640 1 -- 172.21.15.97:0/2754967903 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b00070b10 msgr2=0x7f5b00070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.181+0000 7f5afe000640 1 --2- 172.21.15.97:0/2754967903 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b00070b10 0x7f5b00070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.181+0000 7f5afe000640 1 -- 172.21.15.97:0/2754967903 >> v1:172.21.15.97:6789/0 conn(0x7f5b00144c00 legacy=0x7f5b0006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:51.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.181+0000 7f5afe000640 1 -- 172.21.15.97:0/2754967903 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0006d790 con 0x7f5b0015f9d0 2024-03-06T07:41:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.182+0000 7f5afe000640 1 --2- 172.21.15.97:0/2754967903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b0015f9d0 0x7f5b0015fe60 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80097a0 tx=0x7f5ae802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191e57c9c9b219cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.183+0000 7f5afd600640 1 -- 172.21.15.97:0/2754967903 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8035020 con 0x7f5b0015f9d0 2024-03-06T07:41:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.183+0000 7f5afd600640 1 -- 172.21.15.97:0/2754967903 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8031810 con 0x7f5b0015f9d0 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.183+0000 7f5afd600640 1 -- 172.21.15.97:0/2754967903 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8031bc0 con 0x7f5b0015f9d0 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.183+0000 7f5b05a00640 1 -- 172.21.15.97:0/2754967903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b0015f9d0 msgr2=0x7f5b0015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.183+0000 7f5b05a00640 1 --2- 172.21.15.97:0/2754967903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b0015f9d0 0x7f5b0015fe60 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80097a0 tx=0x7f5ae802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.184+0000 7f5b05a00640 1 -- 172.21.15.97:0/2754967903 shutdown_connections 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.184+0000 7f5b05a00640 1 --2- 172.21.15.97:0/2754967903 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b00070b10 0x7f5b00070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.184+0000 7f5b05a00640 1 --2- 172.21.15.97:0/2754967903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b0015f9d0 0x7f5b0015fe60 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.184+0000 7f5b05a00640 1 -- 172.21.15.97:0/2754967903 >> 172.21.15.97:0/2754967903 conn(0x7f5b000b1730 msgr2=0x7f5b000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.184+0000 7f5b05a00640 1 -- 172.21.15.97:0/2754967903 shutdown_connections 2024-03-06T07:41:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.184+0000 7f5b05a00640 1 -- 172.21.15.97:0/2754967903 wait complete. 2024-03-06T07:41:51.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.185+0000 7f5b05a00640 1 Processor -- start 2024-03-06T07:41:51.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.185+0000 7f5b05a00640 1 -- start start 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.186+0000 7f5b05a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b00070b10 0x7f5b0013b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.186+0000 7f5b05a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 0x7f5b001e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.186+0000 7f5b05a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b0015f9d0 0x7f5b0015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.186+0000 7f5b05a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0006ccd0 con 0x7f5b00144c00 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.186+0000 7f5b05a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0006cb50 con 0x7f5b00070b10 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.186+0000 7f5b05a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0006df90 con 0x7f5b0015f9d0 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 0x7f5b001e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:51.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 0x7f5b001e6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45120/0 (socket says 172.21.15.97:45120) 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 -- 172.21.15.97:0/3637950670 learned_addr learned my addr 172.21.15.97:0/3637950670 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 -- 172.21.15.97:0/3637950670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b0015f9d0 msgr2=0x7f5b0015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b0015f9d0 0x7f5b0015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 -- 172.21.15.97:0/3637950670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b00070b10 msgr2=0x7f5b0013b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b00070b10 0x7f5b0013b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 -- 172.21.15.97:0/3637950670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0013c6c0 con 0x7f5b00144c00 2024-03-06T07:41:51.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.187+0000 7f5afe000640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 0x7f5b001e6380 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f5ae802b8f0 tx=0x7f5ae80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:51.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.188+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8035040 con 0x7f5b00144c00 2024-03-06T07:41:51.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.188+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8003820 con 0x7f5b00144c00 2024-03-06T07:41:51.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.188+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8003bd0 con 0x7f5b00144c00 2024-03-06T07:41:51.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.188+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0013bb40 con 0x7f5b00144c00 2024-03-06T07:41:51.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.188+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0013c100 con 0x7f5b00144c00 2024-03-06T07:41:51.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.189+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0004f330 con 0x7f5b00144c00 2024-03-06T07:41:51.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.190+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ae8038020 con 0x7f5b00144c00 2024-03-06T07:41:51.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.191+0000 7f5af6a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5adc06b4d0 0x7f5adc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:51.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.191+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f5ae80ac450 con 0x7f5b00144c00 2024-03-06T07:41:51.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.191+0000 7f5afea00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5adc06b4d0 0x7f5adc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:51.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.191+0000 7f5afea00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5adc06b4d0 0x7f5adc06d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f5af0000f60 tx=0x7f5af00078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.195+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ae807b4e0 con 0x7f5b00144c00 2024-03-06T07:41:51.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.390+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b00072890 con 0x7f5adc06b4d0 2024-03-06T07:41:51.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.392+0000 7f5af6a00640 1 -- 172.21.15.97:0/3637950670 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13370 (secure 0 0 0) 0x7f5b00072890 con 0x7f5adc06b4d0 2024-03-06T07:41:51.394 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:41:51.395 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:41:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5adc06b4d0 msgr2=0x7f5adc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5adc06b4d0 0x7f5adc06d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f5af0000f60 tx=0x7f5af00078f0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 msgr2=0x7f5b001e6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 0x7f5b001e6380 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f5ae802b8f0 tx=0x7f5ae80046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 shutdown_connections 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f5adc06b4d0 0x7f5adc06d990 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f5b0015f9d0 0x7f5b0015b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b00144c00 0x7f5b001e6380 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 --2- 172.21.15.97:0/3637950670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b00070b10 0x7f5b0013b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 >> 172.21.15.97:0/3637950670 conn(0x7f5b000b1730 msgr2=0x7f5b0006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 shutdown_connections 2024-03-06T07:41:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:41:51.394+0000 7f5b05a00640 1 -- 172.21.15.97:0/3637950670 wait complete. 2024-03-06T07:41:51.416 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2024-03-06T07:41:51.175384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":311168,"kb_used_data":15944,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754663552,"statfs":{"total":773094113280,"available":772775477248,"internally_reserved":0,"allocated":16326656,"data_stored":7806608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.046814"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":411,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-06T07:41:40.921886+0000","last_change":"2024-03-06T07:27:37.168474+0000","last_active":"2024-03-06T07:41:40.921886+0000","last_peered":"2024-03-06T07:41:40.921886+0000","last_clean":"2024-03-06T07:41:40.921886+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:41:40.921886+0000","last_undegraded":"2024-03-06T07:41:40.921886+0000","last_fullsized":"2024-03-06T07:41:40.921886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:35.872692+0000","last_clean_scrub_stamp":"2024-03-06T07:27:35.872692+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:01.281825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37768,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334072,"statfs":{"total":96636764160,"available":96598089728,"internally_reserved":0,"allocated":2486272,"data_stored":1418602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542318,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37188,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542318,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37192,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542318,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37192,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542318,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":2486272,"data_stored":1418602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:41:51.419 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:27:35.872692+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=27, tm_sec=35, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=41, tm_sec=44, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:41:51.419 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:42:10.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:10.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:11.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:42:11.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.635+0000 7f6eba600640 1 Processor -- start 2024-03-06T07:42:11.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.635+0000 7f6eba600640 1 -- start start 2024-03-06T07:42:11.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eba600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb4144c00 0x7f6eb406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eba600640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6eb415f9d0 0x7f6eb415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eba600640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6eb40ce0a0 0x7f6eb40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eba600640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb4160370 con 0x7f6eb415f9d0 2024-03-06T07:42:11.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eba600640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb406ee70 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eba600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb406cef0 con 0x7f6eb4144c00 2024-03-06T07:42:11.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eaaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6eb415f9d0 0x7f6eb415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eaaa00640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6eb415f9d0 0x7f6eb415fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46002/0 (socket says 172.21.15.97:46002) 2024-03-06T07:42:11.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eaaa00640 1 -- 172.21.15.97:0/2285102752 learned_addr learned my addr 172.21.15.97:0/2285102752 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:42:11.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.637+0000 7f6eb3e00640 1 --2- 172.21.15.97:0/2285102752 >> v2:172.21.15.135:3300/0 conn(0x7f6eb40ce0a0 0x7f6eb40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eaaa00640 1 -- 172.21.15.97:0/2285102752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb4144c00 msgr2=0x7f6eb406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eb3400640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb4144c00 0x7f6eb406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eaaa00640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb4144c00 0x7f6eb406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eaaa00640 1 -- 172.21.15.97:0/2285102752 >> v2:172.21.15.135:3300/0 conn(0x7f6eb40ce0a0 msgr2=0x7f6eb40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eaaa00640 1 --2- 172.21.15.97:0/2285102752 >> v2:172.21.15.135:3300/0 conn(0x7f6eb40ce0a0 0x7f6eb40ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eaaa00640 1 -- 172.21.15.97:0/2285102752 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb406d1d0 con 0x7f6eb415f9d0 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eb3e00640 1 --2- 172.21.15.97:0/2285102752 >> v2:172.21.15.135:3300/0 conn(0x7f6eb40ce0a0 0x7f6eb40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.638+0000 7f6eb3400640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb4144c00 0x7f6eb406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.639+0000 7f6eaaa00640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb415f9d0 0x7f6eb415fe30 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40737b0 tx=0x7f6e9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab35c1ffffd2d0cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:11.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.639+0000 7f6eb2a00640 1 -- 172.21.15.97:0/2285102752 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c035020 con 0x7f6eb415f9d0 2024-03-06T07:42:11.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.639+0000 7f6eb2a00640 1 -- 172.21.15.97:0/2285102752 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c031870 con 0x7f6eb415f9d0 2024-03-06T07:42:11.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.639+0000 7f6eb2a00640 1 -- 172.21.15.97:0/2285102752 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c031c40 con 0x7f6eb415f9d0 2024-03-06T07:42:11.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 -- 172.21.15.97:0/2285102752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb415f9d0 msgr2=0x7f6eb415fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb415f9d0 0x7f6eb415fe30 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40737b0 tx=0x7f6e9c02b910 comp rx=0 tx=0).stop 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 -- 172.21.15.97:0/2285102752 shutdown_connections 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 --2- 172.21.15.97:0/2285102752 >> v2:172.21.15.135:3300/0 conn(0x7f6eb40ce0a0 0x7f6eb40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb415f9d0 0x7f6eb415fe30 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 --2- 172.21.15.97:0/2285102752 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb4144c00 0x7f6eb406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 -- 172.21.15.97:0/2285102752 >> 172.21.15.97:0/2285102752 conn(0x7f6eb40b1730 msgr2=0x7f6eb40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 -- 172.21.15.97:0/2285102752 shutdown_connections 2024-03-06T07:42:11.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.640+0000 7f6eba600640 1 -- 172.21.15.97:0/2285102752 wait complete. 2024-03-06T07:42:11.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.642+0000 7f6eba600640 1 Processor -- start 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.642+0000 7f6eba600640 1 -- start start 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eba600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 0x7f6eb40b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eba600640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb4144c00 0x7f6eb4139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eba600640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb415f9d0 0x7f6eb40b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eba600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4160730 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eba600640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb406d790 con 0x7f6eb4144c00 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eba600640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb406cb50 con 0x7f6eb415f9d0 2024-03-06T07:42:11.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eb3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 0x7f6eb40b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eb3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 0x7f6eb40b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46004/0 (socket says 172.21.15.97:46004) 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eb3400640 1 -- 172.21.15.97:0/1422721807 learned_addr learned my addr 172.21.15.97:0/1422721807 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.643+0000 7f6eaaa00640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb4144c00 0x7f6eb4139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3e00640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb415f9d0 0x7f6eb40b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3400640 1 -- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb415f9d0 msgr2=0x7f6eb40b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3400640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb415f9d0 0x7f6eb40b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3400640 1 -- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb4144c00 msgr2=0x7f6eb4139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3400640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb4144c00 0x7f6eb4139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3400640 1 -- 172.21.15.97:0/1422721807 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb413cda0 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eb3400640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 0x7f6eb40b3f90 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400cf60 tx=0x7f6ea400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:11.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea401a070 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea400d840 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea400dc10 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb413d080 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.644+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb415f100 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.645+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb404f330 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.646+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ea401d070 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.647+0000 7f6eabe00640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e9006b4d0 0x7f6e9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:11.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.647+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6ea408a6d0 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.648+0000 7f6eaaa00640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e9006b4d0 0x7f6e9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:11.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.648+0000 7f6eaaa00640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e9006b4d0 0x7f6e9006d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c036000 tx=0x7f6e9c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:11.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.652+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ea4059210 con 0x7f6eb40ce0a0 2024-03-06T07:42:11.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.843+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 --> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6eb4072890 con 0x7f6e9006b4d0 2024-03-06T07:42:11.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.844+0000 7f6eabe00640 1 -- 172.21.15.97:0/1422721807 <== mgr.4106 v2:172.21.15.97:6832/1969101468 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13380 (secure 0 0 0) 0x7f6e9c04c460 con 0x7f6e9006b4d0 2024-03-06T07:42:11.847 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-06T07:42:11.847 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:42:11.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e9006b4d0 msgr2=0x7f6e9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e9006b4d0 0x7f6e9006d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c036000 tx=0x7f6e9c008030 comp rx=0 tx=0).stop 2024-03-06T07:42:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 msgr2=0x7f6eb40b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 0x7f6eb40b3f90 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400cf60 tx=0x7f6ea400cf90 comp rx=0 tx=0).stop 2024-03-06T07:42:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 shutdown_connections 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f6e9006b4d0 0x7f6e9006d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f6eb415f9d0 0x7f6eb40b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eb4144c00 0x7f6eb4139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 --2- 172.21.15.97:0/1422721807 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6eb40ce0a0 0x7f6eb40b3f90 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 >> 172.21.15.97:0/1422721807 conn(0x7f6eb40b1730 msgr2=0x7f6eb40cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 shutdown_connections 2024-03-06T07:42:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:11.847+0000 7f6eba600640 1 -- 172.21.15.97:0/1422721807 wait complete. 2024-03-06T07:42:11.869 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-03-06T07:42:11.180768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":311160,"kb_used_data":15944,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754663560,"statfs":{"total":773094113280,"available":772775485440,"internally_reserved":0,"allocated":16326656,"data_stored":7806608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003363"},"pg_stats":[{"pgid":"1.0","version":"151'119","reported_seq":422,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-06T07:41:51.632142+0000","last_change":"2024-03-06T07:41:51.632219+0000","last_active":"2024-03-06T07:41:51.632142+0000","last_peered":"2024-03-06T07:41:51.632142+0000","last_clean":"2024-03-06T07:41:51.632142+0000","last_became_active":"2024-03-06T07:27:37.166248+0000","last_became_peered":"2024-03-06T07:27:37.166248+0000","last_unstale":"2024-03-06T07:41:51.632142+0000","last_undegraded":"2024-03-06T07:41:51.632142+0000","last_fullsized":"2024-03-06T07:41:51.632142+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'119","last_scrub_stamp":"2024-03-06T07:41:51.631956+0000","last_deep_scrub":"151'119","last_deep_scrub_stamp":"2024-03-06T07:41:51.631956+0000","last_clean_scrub_stamp":"2024-03-06T07:41:51.631956+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:32.753223+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37764,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334076,"statfs":{"total":96636764160,"available":96598093824,"internally_reserved":0,"allocated":2486272,"data_stored":1418602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542322,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37188,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542322,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37192,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542322,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37192,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":828234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41924,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329916,"statfs":{"total":96636764160,"available":96593833984,"internally_reserved":0,"allocated":2486272,"data_stored":1418602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:42:11.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:42:12.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.070+0000 7f30fe400640 1 Processor -- start 2024-03-06T07:42:12.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.070+0000 7f30fe400640 1 -- start start 2024-03-06T07:42:12.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30fe400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 0x7f30f8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30fe400640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f80b46c0 con 0x7f30f80b3f70 2024-03-06T07:42:12.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30fe400640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f80b49a0 con 0x7f30f815fc10 2024-03-06T07:42:12.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30fe400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8072c90 con 0x7f30f8160700 2024-03-06T07:42:12.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30f6a00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f30f80b3f70 0x7f30f80ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44256/0 (socket says 172.21.15.97:44256) 2024-03-06T07:42:12.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30f6a00640 1 -- 172.21.15.97:0/2382848088 learned_addr learned my addr 172.21.15.97:0/2382848088 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:42:12.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.072+0000 7f30f7e00640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 0x7f30f8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664319707 0 0) 0x7f30f80b46c0 con 0x7f30f80b3f70 2024-03-06T07:42:12.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc003520 con 0x7f30f80b3f70 2024-03-06T07:42:12.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1184082611 0 0) 0x7f30dc003520 con 0x7f30f80b3f70 2024-03-06T07:42:12.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 msgr2=0x7f30f8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:12.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 0x7f30f8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 >> v1:172.21.15.135:6789/0 conn(0x7f30f815fc10 legacy=0x7f30f815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f809dd80 con 0x7f30f80b3f70 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7f30e4001670 con 0x7f30f80b3f70 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 >> v1:172.21.15.97:6789/0 conn(0x7f30f80b3f70 legacy=0x7f30f80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 0x7f30dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30dc003f60 0x7f30dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8072c90 con 0x7f30dc004d40 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc00baa0 con 0x7f30dc003f60 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.073+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc00bca0 con 0x7f30f8160700 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7e00640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7400640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 0x7f30dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f6a00640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30dc003f60 0x7f30dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7e00640 1 -- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 msgr2=0x7f30dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:12.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7e00640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f8160700 0x7f30dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7e00640 1 -- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30dc003f60 msgr2=0x7f30dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7e00640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30dc003f60 0x7f30dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.074+0000 7f30f7e00640 1 -- 172.21.15.97:0/2382848088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc003520 con 0x7f30dc004d40 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.075+0000 7f30f7e00640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30dc0051c0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009fc0 tx=0x7f30dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da458689abf4f246 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.075+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0039070 con 0x7f30dc004d40 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f002cc20 con 0x7f30dc004d40 2024-03-06T07:42:12.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30f6000640 1 -- 172.21.15.97:0/2382848088 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0009c10 con 0x7f30dc004d40 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30fe400640 1 -- 172.21.15.97:0/2382848088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 msgr2=0x7f30dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30fe400640 1 --2- 172.21.15.97:0/2382848088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30dc0051c0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009fc0 tx=0x7f30dc005830 comp rx=0 tx=0).stop 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30f7e00640 1 -- 172.21.15.97:0/2382848088 reap_dead start 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30fe400640 1 -- 172.21.15.97:0/2382848088 shutdown_connections 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30fe400640 1 -- 172.21.15.97:0/2382848088 >> 172.21.15.97:0/2382848088 conn(0x7f30f80b1940 msgr2=0x7f30f80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30fe400640 1 -- 172.21.15.97:0/2382848088 shutdown_connections 2024-03-06T07:42:12.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.076+0000 7f30fe400640 1 -- 172.21.15.97:0/2382848088 wait complete. 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.078+0000 7f30fe400640 1 Processor -- start 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.078+0000 7f30fe400640 1 -- start start 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30fe400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30f80cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30fe400640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f81525a0 0x7f30f81473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30fe400640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f81f4230 0x7f30f80b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30fe400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f815c460 con 0x7f30dc004d40 2024-03-06T07:42:12.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30fe400640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f815c2e0 con 0x7f30f81525a0 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30fe400640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8070ff0 con 0x7f30f81f4230 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30f80cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30f80cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46020/0 (socket says 172.21.15.97:46020) 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.079+0000 7f30f7400640 1 -- 172.21.15.97:0/4168220796 learned_addr learned my addr 172.21.15.97:0/4168220796 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f6a00640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f81525a0 0x7f30f81473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7e00640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f81f4230 0x7f30f80b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7400640 1 -- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f81f4230 msgr2=0x7f30f80b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f81f4230 0x7f30f80b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7400640 1 -- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f81525a0 msgr2=0x7f30f81473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f81525a0 0x7f30f81473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7400640 1 -- 172.21.15.97:0/4168220796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f81e8870 con 0x7f30dc004d40 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f6a00640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f81525a0 0x7f30f81473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30f7400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30f80cb2c0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f30e8001eb0 tx=0x7f30e800e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:12.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80052e0 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e8018070 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f806ca30 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.080+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8003810 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.081+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f81478f0 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.082+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f804f330 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.083+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30e80039b0 con 0x7f30dc004d40 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.083+0000 7f30ef400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d806b500 0x7f30d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.084+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f30e8089bc0 con 0x7f30dc004d40 2024-03-06T07:42:12.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.084+0000 7f30f6a00640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d806b500 0x7f30d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.085+0000 7f30f6a00640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d806b500 0x7f30d806d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f30e4001eb0 tx=0x7f30e4001dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:12.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.089+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30e8058ce0 con 0x7f30dc004d40 2024-03-06T07:42:12.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.286+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f30f80b2d40 con 0x7f30dc004d40 2024-03-06T07:42:12.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.292+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v36)=0 v36) v1 ==== 123+0+0 (secure 0 0 0) 0x7f30e80585e0 con 0x7f30dc004d40 2024-03-06T07:42:12.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.294+0000 7f30ef400640 1 -- 172.21.15.97:0/4168220796 <== mon.0 v2:172.21.15.97:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f30e8005480 con 0x7f30dc004d40 2024-03-06T07:42:12.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d806b500 msgr2=0x7f30d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d806b500 0x7f30d806d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f30e4001eb0 tx=0x7f30e4001dc0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 msgr2=0x7f30f80cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30f80cb2c0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f30e8001eb0 tx=0x7f30e800e9e0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 shutdown_connections 2024-03-06T07:42:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7f30d806b500 0x7f30d806d9c0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7f30f81f4230 0x7f30f80b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30f81525a0 0x7f30f81473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 --2- 172.21.15.97:0/4168220796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30dc004d40 0x7f30f80cb2c0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 >> 172.21.15.97:0/4168220796 conn(0x7f30f80b1940 msgr2=0x7f30f80cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:42:12.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 shutdown_connections 2024-03-06T07:42:12.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.296+0000 7f30fe400640 1 -- 172.21.15.97:0/4168220796 wait complete. 2024-03-06T07:42:12.318 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:42:12.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:42:12.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.521+0000 7fd404a00640 1 Processor -- start 2024-03-06T07:42:12.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.521+0000 7fd404a00640 1 -- start start 2024-03-06T07:42:12.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.523+0000 7fd404a00640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd40015f9d0 0x7fd40015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.523+0000 7fd404a00640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40006cef0 con 0x7fd400144c00 2024-03-06T07:42:12.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.523+0000 7fd404a00640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40006d1d0 con 0x7fd400070b10 2024-03-06T07:42:12.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.523+0000 7fd404a00640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40006d4b0 con 0x7fd40015f9d0 2024-03-06T07:42:12.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.523+0000 7fd3fde00640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd400144c00 0x7fd40006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:44262/0 (socket says 172.21.15.97:44262) 2024-03-06T07:42:12.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.523+0000 7fd3fde00640 1 -- 172.21.15.97:0/1567394767 learned_addr learned my addr 172.21.15.97:0/1567394767 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-06T07:42:12.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fd400640 1 --2- 172.21.15.97:0/1567394767 >> v2:172.21.15.135:3300/0 conn(0x7fd40015f9d0 0x7fd40015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487939002 0 0) 0x7fd40006cef0 con 0x7fd400144c00 2024-03-06T07:42:12.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e4003520 con 0x7fd400144c00 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863386845 0 0) 0x7fd3e4003520 con 0x7fd400144c00 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 >> v2:172.21.15.135:3300/0 conn(0x7fd40015f9d0 msgr2=0x7fd40015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 --2- 172.21.15.97:0/1567394767 >> v2:172.21.15.135:3300/0 conn(0x7fd40015f9d0 0x7fd40015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 >> v1:172.21.15.135:6789/0 conn(0x7fd400070b10 legacy=0x7fd400070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40006d790 con 0x7fd400144c00 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3651546305 0 0) 0x7fd3f0001670 con 0x7fd400144c00 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 >> v1:172.21.15.97:6789/0 conn(0x7fd400144c00 legacy=0x7fd40006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4003a00 0x7fd3e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3e4004800 0x7fd3e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3e40051c0 0x7fd3e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40006d4b0 con 0x7fd3e4004800 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400c0c0 con 0x7fd3e4003a00 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.524+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400c2c0 con 0x7fd3e40051c0 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fd400640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3e4004800 0x7fd3e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fd400640 1 -- 172.21.15.97:0/1567394767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3e40051c0 msgr2=0x7fd3e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:12.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fd400640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd3e40051c0 0x7fd3e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fd400640 1 -- 172.21.15.97:0/1567394767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4003a00 msgr2=0x7fd3e4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:42:12.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fd400640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4003a00 0x7fd3e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fd400640 1 -- 172.21.15.97:0/1567394767 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e4003520 con 0x7fd3e4004800 2024-03-06T07:42:12.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.525+0000 7fd3fe800640 1 -- 172.21.15.97:0/1567394767 reap_dead start 2024-03-06T07:42:12.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.526+0000 7fd3fd400640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3e4004800 0x7fd3e4004c80 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009ac0 tx=0x7fd3e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79b14daae83860 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:12.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.526+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec039070 con 0x7fd3e4004800 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.526+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd3ec02c6d0 con 0x7fd3e4004800 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.526+0000 7fd3fca00640 1 -- 172.21.15.97:0/1567394767 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec009690 con 0x7fd3e4004800 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 -- 172.21.15.97:0/1567394767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3e4004800 msgr2=0x7fd3e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3e4004800 0x7fd3e4004c80 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009ac0 tx=0x7fd3e4005830 comp rx=0 tx=0).stop 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 -- 172.21.15.97:0/1567394767 shutdown_connections 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 --2- 172.21.15.97:0/1567394767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3e4004800 0x7fd3e4004c80 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 -- 172.21.15.97:0/1567394767 >> 172.21.15.97:0/1567394767 conn(0x7fd4000b1730 msgr2=0x7fd4000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 -- 172.21.15.97:0/1567394767 shutdown_connections 2024-03-06T07:42:12.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.527+0000 7fd404a00640 1 -- 172.21.15.97:0/1567394767 wait complete. 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.529+0000 7fd404a00640 1 Processor -- start 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.529+0000 7fd404a00640 1 -- start start 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd404a00640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4004800 0x7fd40015aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd404a00640 1 --2- >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4000ae220 0x7fd40013b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd404a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 0x7fd4001f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd404a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40006ccd0 con 0x7fd4001f3a10 2024-03-06T07:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd404a00640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40006cb50 con 0x7fd3e4004800 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd404a00640 1 -- --> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40006df90 con 0x7fd4000ae220 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd3fe800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 0x7fd4001f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd3fe800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 0x7fd4001f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46044/0 (socket says 172.21.15.97:46044) 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd3fe800640 1 -- 172.21.15.97:0/3418287605 learned_addr learned my addr 172.21.15.97:0/3418287605 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd3fd400640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4000ae220 0x7fd40013b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.530+0000 7fd3fde00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4004800 0x7fd40015aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fe800640 1 -- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4000ae220 msgr2=0x7fd40013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fe800640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4000ae220 0x7fd40013b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fe800640 1 -- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4004800 msgr2=0x7fd40015aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fe800640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4004800 0x7fd40015aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fe800640 1 -- 172.21.15.97:0/3418287605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4001f4400 con 0x7fd4001f3a10 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fd400640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4000ae220 0x7fd40013b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3fe800640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 0x7fd4001f3ec0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fd4000d0ac0 tx=0x7fd3f800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8017070 con 0x7fd4001f3a10 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd3f8005d60 con 0x7fd4001f3a10 2024-03-06T07:42:12.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f800a870 con 0x7fd4001f3a10 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4000ae700 con 0x7fd4001f3a10 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.531+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4000aecc0 con 0x7fd4001f3a10 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.533+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40004f330 con 0x7fd4001f3a10 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.533+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3f801a070 con 0x7fd4001f3a10 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.534+0000 7fd3f6000640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3e006b500 0x7fd3e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.534+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd3f80892f0 con 0x7fd4001f3a10 2024-03-06T07:42:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.535+0000 7fd3fde00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3e006b500 0x7fd3e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:42:12.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.535+0000 7fd3fde00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3e006b500 0x7fd3e006d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00011e0 tx=0x7fd3f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:42:12.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.539+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3f8058380 con 0x7fd4001f3a10 2024-03-06T07:42:12.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.771+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd400074be0 con 0x7fd4001f3a10 2024-03-06T07:42:12.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.772+0000 7fd3f6000640 1 -- 172.21.15.97:0/3418287605 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1329 (secure 0 0 0) 0x7fd3f8057c80 con 0x7fd4001f3a10 2024-03-06T07:42:12.774 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-06T07:42:12.775 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10215,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/2798928001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2798928001},{"type":"v1","addr":"172.21.15.97:6835","nonce":2798928001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[]} 2024-03-06T07:42:12.775 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 119 2024-03-06T07:42:12.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.774+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3e006b500 msgr2=0x7fd3e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.774+0000 7fd404a00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3e006b500 0x7fd3e006d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00011e0 tx=0x7fd3f0009040 comp rx=0 tx=0).stop 2024-03-06T07:42:12.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.774+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 msgr2=0x7fd4001f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:42:12.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.774+0000 7fd404a00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 0x7fd4001f3ec0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fd4000d0ac0 tx=0x7fd3f800cb90 comp rx=0 tx=0).stop 2024-03-06T07:42:12.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.774+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 shutdown_connections 2024-03-06T07:42:12.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.774+0000 7fd404a00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:6832/1969101468,v1:172.21.15.97:6833/1969101468] conn(0x7fd3e006b500 0x7fd3e006d9c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.775+0000 7fd404a00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4001f3a10 0x7fd4001f3ec0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.775+0000 7fd404a00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3301/0,v1:172.21.15.135:6790/0] conn(0x7fd4000ae220 0x7fd40013b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.775+0000 7fd404a00640 1 --2- 172.21.15.97:0/3418287605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3e4004800 0x7fd40015aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:42:12.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.775+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 >> 172.21.15.97:0/3418287605 conn(0x7fd4000b1730 msgr2=0x7fd4000af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:42:12.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.775+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 shutdown_connections 2024-03-06T07:42:12.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-06T07:42:12.775+0000 7fd404a00640 1 -- 172.21.15.97:0/3418287605 wait complete. 2024-03-06T07:42:12.796 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:42:12.796 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:42:12.796 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:42:12.797 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:12.826 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:42:12.826 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:42:12.826 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:12.897 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:42:12.897 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-03-06T07:42:12.900 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-03-06T07:42:12.904 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:42:12.905 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:42:12.905 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:12.988 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:42:12.988 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:42:12.988 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.042 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:42:13.042 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:42:13.042 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.091 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:42:13.092 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:42:13.092 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.157 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:42:13.157 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:42:13.158 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.220 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:42:13.221 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:42:13.221 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.286 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:42:13.286 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:42:13.287 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.357 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:42:13.357 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:42:13.357 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.413 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:42:13.414 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:42:13.414 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:42:13.414 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:13.487 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:42:13.487 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:42:13.488 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:16.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-03-06T07:42:19.589 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:42:19.590 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:42:19.590 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:42:19.591 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:19.625 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:42:19.625 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:42:19.625 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:19.668 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:42:19.669 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:42:19.669 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:42:19.698 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:42:19.699 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:42:19.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:42:19.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:42:19.802 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:42:20.114 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:42:20.114 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:42:20.277 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:42:20.278 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:42:20.435 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:42:20.435 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:42:20.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:42:20.609 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:42:20.890 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:42:20.890 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:42:21.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:42:21.030 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:42:21.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-03-06T07:42:21.174 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:42:21.174 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:42:21.344 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:42:21.346 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/data/mon.a.tgz 2024-03-06T07:42:21.346 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-06T07:42:21.403 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.i9IawXyXkD 2024-03-06T07:42:21.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.i9IawXyXkD 2024-03-06T07:42:21.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.i9IawXyXkD 2024-03-06T07:42:21.756 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.i9IawXyXkD is 426KB 2024-03-06T07:42:21.865 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.i9IawXyXkD 2024-03-06T07:42:21.918 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/data/mon.b.tgz 2024-03-06T07:42:21.918 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-06T07:42:21.976 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.vNbvdm15zv 2024-03-06T07:42:21.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vNbvdm15zv 2024-03-06T07:42:22.182 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.vNbvdm15zv 2024-03-06T07:42:22.327 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.vNbvdm15zv is 434KB 2024-03-06T07:42:22.434 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.vNbvdm15zv 2024-03-06T07:42:22.488 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/data/mon.c.tgz 2024-03-06T07:42:22.488 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-06T07:42:22.537 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.O81Fb9by0W 2024-03-06T07:42:22.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.O81Fb9by0W 2024-03-06T07:42:22.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.O81Fb9by0W 2024-03-06T07:42:22.862 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.O81Fb9by0W is 434KB 2024-03-06T07:42:22.972 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.O81Fb9by0W 2024-03-06T07:42:23.028 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:42:23.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:42:23.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:42:23.177 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:42:23.234 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:42:23.234 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:42:23.236 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/remote/smithi097/crash 2024-03-06T07:42:23.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:42:23.331 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/remote/smithi135/crash 2024-03-06T07:42:23.332 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:42:23.427 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:42:23.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:23.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:23.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:23.650 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:23.751 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:42:23.751 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:42:23.759 DEBUG:teuthology.orchestra.run.smithi135:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:42:23.845 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:42:23.846 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42870.log 2024-03-06T07:42:23.847 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73440.log 2024-03-06T07:42:23.849 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37294.log 2024-03-06T07:42:23.849 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.1.42870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42870.log.gz 2024-03-06T07:42:23.849 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43278.log 2024-03-06T07:42:23.849 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37494.log 2024-03-06T07:42:23.858 INFO:teuthology.orchestra.run.smithi135.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37294.log.gz 2024-03-06T07:42:23.858 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.3.43278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.43278.log.gz 2024-03-06T07:42:23.859 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:42:23.859 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37414.log 2024-03-06T07:42:23.859 INFO:teuthology.orchestra.run.smithi135.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37494.log.gz 2024-03-06T07:42:23.859 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:42:23.860 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37414.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:42:23.860 INFO:teuthology.orchestra.run.smithi135.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37414.log.gz 2024-03-06T07:42:23.860 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:42:23.860 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:42:23.860 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37534.log 2024-03-06T07:42:23.866 INFO:teuthology.orchestra.run.smithi135.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:42:23.866 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:42:23.867 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37454.log 2024-03-06T07:42:23.867 INFO:teuthology.orchestra.run.smithi135.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37534.log.gz 2024-03-06T07:42:23.867 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37334.log 2024-03-06T07:42:23.867 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37454.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37454.log.gz 2024-03-06T07:42:23.868 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37334.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37334.log.gz 2024-03-06T07:42:23.868 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:42:23.877 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37374.log 2024-03-06T07:42:23.877 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.37374.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37374.log.gz 2024-03-06T07:42:23.877 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37254.log 2024-03-06T07:42:23.877 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.37254.log: gzip -5 --verbose 92.7% -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:42:23.877 INFO:teuthology.orchestra.run.smithi135.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:42:23.877 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:42:23.882 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-mds.d.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37254.log.gz 2024-03-06T07:42:23.882 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:42:23.882 INFO:teuthology.orchestra.run.smithi135.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:42:23.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58486.log 2024-03-06T07:42:23.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51087.log 2024-03-06T07:42:23.887 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73440.log.gz 2024-03-06T07:42:23.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2024-03-06T07:42:23.887 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58486.log.gz 2024-03-06T07:42:23.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76940.log 2024-03-06T07:42:23.888 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51087.log.gz 2024-03-06T07:42:23.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43717.log 2024-03-06T07:42:23.888 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2024-03-06T07:42:23.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68335.log 2024-03-06T07:42:23.889 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76940.log.gz 2024-03-06T07:42:23.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50699.log 2024-03-06T07:42:23.889 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43717.log.gz 2024-03-06T07:42:23.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59159.log 2024-03-06T07:42:23.889 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68335.log.gz 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77454.log 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50699.log.gz 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.59159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59159.log.gz 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70397.log 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44852.log 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77454.log.gz 2024-03-06T07:42:23.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83344.log 2024-03-06T07:42:23.891 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70397.log.gz 2024-03-06T07:42:23.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67549.log 2024-03-06T07:42:23.891 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44852.log.gz 2024-03-06T07:42:23.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53230.log 2024-03-06T07:42:23.891 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83344.log.gz 2024-03-06T07:42:23.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71436.log 2024-03-06T07:42:23.892 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67549.log.gz 2024-03-06T07:42:23.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52435.log 2024-03-06T07:42:23.892 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53230.log.gz 2024-03-06T07:42:23.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43340.log 2024-03-06T07:42:23.892 INFO:teuthology.orchestra.run.smithi097.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.71436.log.gz 2024-03-06T07:42:23.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60909.log 2024-03-06T07:42:23.893 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52435.log.gz 2024-03-06T07:42:23.893 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55351.log 2024-03-06T07:42:23.893 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43340.log.gz 2024-03-06T07:42:23.893 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49568.log 2024-03-06T07:42:23.893 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60909.log.gz 2024-03-06T07:42:23.893 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51873.log 2024-03-06T07:42:23.894 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49568.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.55351.log.gz 2024-03-06T07:42:23.894 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82924.log 2024-03-06T07:42:23.894 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49568.log.gz 2024-03-06T07:42:23.894 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75038.log 2024-03-06T07:42:23.894 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51873.log.gz 2024-03-06T07:42:23.894 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62749.log 2024-03-06T07:42:23.895 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82924.log.gz 2024-03-06T07:42:23.895 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75038.log: gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.75038.log.gz --verbose 2024-03-06T07:42:23.895 INFO:teuthology.orchestra.run.smithi097.stderr: -- /var/log/ceph/ceph-client.admin.44600.log 2024-03-06T07:42:23.895 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87746.log 2024-03-06T07:42:23.895 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62749.log.gz 2024-03-06T07:42:23.895 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65091.log 2024-03-06T07:42:23.896 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44600.log.gz 2024-03-06T07:42:23.896 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81502.log 2024-03-06T07:42:23.896 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87746.log.gz 2024-03-06T07:42:23.896 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69851.log 2024-03-06T07:42:23.896 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65091.log.gz 2024-03-06T07:42:23.896 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81502.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81502.log.gz 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81207.log 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69851.log.gz 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66076.log 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85430.log 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81207.log.gz 2024-03-06T07:42:23.897 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42528.log 2024-03-06T07:42:23.898 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66076.log.gz 2024-03-06T07:42:23.898 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51381.log 2024-03-06T07:42:23.898 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85430.log.gz 2024-03-06T07:42:23.898 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80630.log 2024-03-06T07:42:23.898 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42528.log.gz 2024-03-06T07:42:23.898 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64056.log 2024-03-06T07:42:23.899 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51381.log.gz 2024-03-06T07:42:23.899 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.80630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80630.log.gz 2024-03-06T07:42:23.899 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46349.log 2024-03-06T07:42:23.900 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mon.c.log: 93.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:42:23.911 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81796.log 2024-03-06T07:42:23.911 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64056.log.gz 2024-03-06T07:42:23.911 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81754.log 2024-03-06T07:42:23.911 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46349.log.gz 2024-03-06T07:42:23.911 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53102.log 2024-03-06T07:42:23.911 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81796.log.gz 2024-03-06T07:42:23.912 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49131.log 2024-03-06T07:42:23.912 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81754.log.gz 2024-03-06T07:42:23.912 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51610.log 2024-03-06T07:42:23.912 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53102.log.gz 2024-03-06T07:42:23.912 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46139.log 2024-03-06T07:42:23.912 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49131.log.gz 2024-03-06T07:42:23.913 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52561.log 2024-03-06T07:42:23.913 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51610.log.gz 2024-03-06T07:42:23.913 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85220.log 2024-03-06T07:42:23.913 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46139.log.gz 2024-03-06T07:42:23.913 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2024-03-06T07:42:23.913 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52561.log.gz 2024-03-06T07:42:23.914 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63001.log 2024-03-06T07:42:23.914 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85220.log.gz 2024-03-06T07:42:23.914 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61334.log 2024-03-06T07:42:23.914 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-03-06T07:42:23.914 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74488.log 2024-03-06T07:42:23.914 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63001.log.gz 2024-03-06T07:42:23.915 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-03-06T07:42:23.915 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61334.log.gz 2024-03-06T07:42:23.915 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79356.log 2024-03-06T07:42:23.915 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74488.log.gz 2024-03-06T07:42:23.915 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64151.log 2024-03-06T07:42:23.915 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-03-06T07:42:23.916 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61747.log 2024-03-06T07:42:23.916 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79356.log.gz 2024-03-06T07:42:23.916 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70323.log 2024-03-06T07:42:23.916 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64151.log.gz 2024-03-06T07:42:23.916 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47636.log 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61747.log.gz 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70323.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.70323.log.gz --verbose 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr: -- /var/log/ceph/ceph-client.admin.50339.log 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87200.log 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47636.log.gz 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56737.log 2024-03-06T07:42:23.917 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50339.log.gz 2024-03-06T07:42:23.918 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52309.log 2024-03-06T07:42:23.918 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87200.log.gz 2024-03-06T07:42:23.918 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43728.log 2024-03-06T07:42:23.918 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56737.log.gz 2024-03-06T07:42:23.925 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71746.log 2024-03-06T07:42:23.925 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52309.log.gz 2024-03-06T07:42:23.926 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57396.log 2024-03-06T07:42:23.926 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43728.log.gz 2024-03-06T07:42:23.926 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78550.log 2024-03-06T07:42:23.926 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71746.log.gz 2024-03-06T07:42:23.926 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57396.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57396.log.gz 2024-03-06T07:42:23.926 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68623.log 2024-03-06T07:42:23.927 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68837.log 2024-03-06T07:42:23.927 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78550.log.gz 2024-03-06T07:42:23.927 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50562.log 2024-03-06T07:42:23.927 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68623.log.gz 2024-03-06T07:42:23.927 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58738.log 2024-03-06T07:42:23.927 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68837.log.gz 2024-03-06T07:42:23.928 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49736.log 2024-03-06T07:42:23.928 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50562.log.gz 2024-03-06T07:42:23.928 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54195.log 2024-03-06T07:42:23.928 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58738.log.gz 2024-03-06T07:42:23.928 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56569.log 2024-03-06T07:42:23.928 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49736.log.gz 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68939.log 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54195.log.gz 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47882.log 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56569.log.gz 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76340.log 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68939.log.gz 2024-03-06T07:42:23.929 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49844.log 2024-03-06T07:42:23.930 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47882.log.gz 2024-03-06T07:42:23.930 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74064.log 2024-03-06T07:42:23.930 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76340.log.gz 2024-03-06T07:42:23.934 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63043.log 2024-03-06T07:42:23.935 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49844.log.gz 2024-03-06T07:42:23.935 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59436.log 2024-03-06T07:42:23.935 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74064.log.gz 2024-03-06T07:42:23.935 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57354.log 2024-03-06T07:42:23.935 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63043.log.gz 2024-03-06T07:42:23.935 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57827.log 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59436.log.gz 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57354.log.gz 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49652.log 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57827.log.gz 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55849.log 2024-03-06T07:42:23.936 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-03-06T07:42:23.937 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59872.log 2024-03-06T07:42:23.937 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49652.log.gz 2024-03-06T07:42:23.937 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86509.log 2024-03-06T07:42:23.937 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55849.log.gz 2024-03-06T07:42:23.940 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84908.log 2024-03-06T07:42:23.940 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59872.log.gz 2024-03-06T07:42:23.940 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56695.log 2024-03-06T07:42:23.940 INFO:teuthology.orchestra.run.smithi097.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.86509.log.gz 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84319.log 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84908.log.gz 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65930.log 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56695.log.gz 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57522.log 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84319.log.gz 2024-03-06T07:42:23.941 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48591.log 2024-03-06T07:42:23.942 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65930.log.gz 2024-03-06T07:42:23.943 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83218.log 2024-03-06T07:42:23.944 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57522.log.gz 2024-03-06T07:42:23.944 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76424.log 2024-03-06T07:42:23.944 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48591.log.gz 2024-03-06T07:42:23.944 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63400.log 2024-03-06T07:42:23.944 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83218.log.gz 2024-03-06T07:42:23.944 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51255.log 2024-03-06T07:42:23.945 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76424.log.gz 2024-03-06T07:42:23.946 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60520.log 2024-03-06T07:42:23.947 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63400.log.gz 2024-03-06T07:42:23.947 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83050.log 2024-03-06T07:42:23.947 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51255.log.gz 2024-03-06T07:42:23.947 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46937.log 2024-03-06T07:42:23.947 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60520.log.gz 2024-03-06T07:42:23.948 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48134.log 2024-03-06T07:42:23.949 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83050.log.gz 2024-03-06T07:42:23.949 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57564.log 2024-03-06T07:42:23.949 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46937.log.gz 2024-03-06T07:42:23.950 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44049.log 2024-03-06T07:42:23.951 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48134.log.gz 2024-03-06T07:42:23.951 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61663.log 2024-03-06T07:42:23.951 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57564.log.gz 2024-03-06T07:42:23.952 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63484.log 2024-03-06T07:42:23.952 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44049.log.gz 2024-03-06T07:42:23.953 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58073.log 2024-03-06T07:42:23.953 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61663.log.gz 2024-03-06T07:42:23.954 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54927.log 2024-03-06T07:42:23.954 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63484.log.gz 2024-03-06T07:42:23.955 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65373.log 2024-03-06T07:42:23.955 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58073.log.gz 2024-03-06T07:42:23.956 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74236.log 2024-03-06T07:42:23.956 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54927.log.gz 2024-03-06T07:42:23.957 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83386.log 2024-03-06T07:42:23.957 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65373.log.gz 2024-03-06T07:42:23.958 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88182.log 2024-03-06T07:42:23.958 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74236.log.gz 2024-03-06T07:42:23.959 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57648.log 2024-03-06T07:42:23.959 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83386.log.gz 2024-03-06T07:42:23.959 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.88182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48793.log 2024-03-06T07:42:23.960 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88182.log.gz 2024-03-06T07:42:23.961 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54832.log 2024-03-06T07:42:23.961 INFO:teuthology.orchestra.run.smithi097.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57648.log.gz 2024-03-06T07:42:23.961 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46223.log 2024-03-06T07:42:23.962 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48793.log.gz 2024-03-06T07:42:23.963 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45074.log 2024-03-06T07:42:23.963 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54832.log.gz 2024-03-06T07:42:23.964 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65622.log 2024-03-06T07:42:23.964 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46223.log.gz 2024-03-06T07:42:23.964 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.45074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57083.log 2024-03-06T07:42:23.964 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45074.log.gz 2024-03-06T07:42:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82234.log 2024-03-06T07:42:23.966 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65622.log.gz 2024-03-06T07:42:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56653.log 2024-03-06T07:42:23.967 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57083.log.gz 2024-03-06T07:42:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58314.log 2024-03-06T07:42:23.967 INFO:teuthology.orchestra.run.smithi097.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.82234.log.gz 2024-03-06T07:42:23.969 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44058.log 2024-03-06T07:42:23.969 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56653.log.gz 2024-03-06T07:42:23.969 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74722.log 2024-03-06T07:42:23.969 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58314.log.gz 2024-03-06T07:42:23.970 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50186.log 2024-03-06T07:42:23.971 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44058.log.gz 2024-03-06T07:42:23.972 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79798.log 2024-03-06T07:42:23.972 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74722.log.gz 2024-03-06T07:42:23.972 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49089.log 2024-03-06T07:42:23.972 INFO:teuthology.orchestra.run.smithi097.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50186.log.gz 2024-03-06T07:42:23.973 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83866.log 2024-03-06T07:42:23.973 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79798.log.gz 2024-03-06T07:42:23.974 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49089.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49089.log.gz 2024-03-06T07:42:23.975 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48278.log 2024-03-06T07:42:23.975 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61831.log 2024-03-06T07:42:23.976 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83866.log.gz 2024-03-06T07:42:23.977 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69935.log 2024-03-06T07:42:23.977 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48278.log.gz 2024-03-06T07:42:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43716.log 2024-03-06T07:42:23.979 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61831.log.gz 2024-03-06T07:42:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67717.log 2024-03-06T07:42:23.979 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69935.log.gz 2024-03-06T07:42:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86425.log 2024-03-06T07:42:23.980 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43716.log.gz 2024-03-06T07:42:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82276.log 2024-03-06T07:42:23.982 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67717.log.gz 2024-03-06T07:42:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75125.log 2024-03-06T07:42:23.982 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86425.log.gz 2024-03-06T07:42:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56999.log 2024-03-06T07:42:23.984 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82276.log.gz 2024-03-06T07:42:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68665.log 2024-03-06T07:42:23.984 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75125.log.gz 2024-03-06T07:42:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52729.log 2024-03-06T07:42:23.986 INFO:teuthology.orchestra.run.smithi097.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56999.log.gz 2024-03-06T07:42:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42878.log 2024-03-06T07:42:23.987 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68665.log.gz 2024-03-06T07:42:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51129.log 2024-03-06T07:42:23.988 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52729.log.gz 2024-03-06T07:42:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72665.log 2024-03-06T07:42:23.989 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42878.log.gz 2024-03-06T07:42:23.989 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78601.log 2024-03-06T07:42:23.989 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51129.log.gz 2024-03-06T07:42:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43550.log 2024-03-06T07:42:23.991 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72665.log.gz 2024-03-06T07:42:23.991 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73272.log 2024-03-06T07:42:23.991 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78601.log.gz 2024-03-06T07:42:23.993 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86593.log 2024-03-06T07:42:23.993 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43550.log.gz 2024-03-06T07:42:23.993 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69191.log 2024-03-06T07:42:23.993 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73272.log.gz 2024-03-06T07:42:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83092.log 2024-03-06T07:42:23.995 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86593.log.gz 2024-03-06T07:42:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2024-03-06T07:42:23.995 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69191.log.gz 2024-03-06T07:42:23.997 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73841.log 2024-03-06T07:42:23.997 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83092.log.gz 2024-03-06T07:42:23.997 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87620.log 2024-03-06T07:42:23.997 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2024-03-06T07:42:23.998 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65560.log 2024-03-06T07:42:23.998 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73841.log.gz 2024-03-06T07:42:23.999 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66448.log 2024-03-06T07:42:23.999 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87620.log.gz 2024-03-06T07:42:23.999 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61579.log 2024-03-06T07:42:23.999 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65560.log.gz 2024-03-06T07:42:24.001 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44432.log 2024-03-06T07:42:24.001 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66448.log.gz 2024-03-06T07:42:24.001 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76382.log 2024-03-06T07:42:24.001 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61579.log.gz 2024-03-06T07:42:24.002 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76466.log 2024-03-06T07:42:24.002 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44432.log.gz 2024-03-06T07:42:24.003 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54153.log 2024-03-06T07:42:24.003 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76382.log.gz 2024-03-06T07:42:24.004 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51484.log 2024-03-06T07:42:24.004 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76466.log.gz 2024-03-06T07:42:24.005 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44390.log 2024-03-06T07:42:24.005 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54153.log.gz 2024-03-06T07:42:24.006 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77202.log 2024-03-06T07:42:24.006 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51484.log.gz 2024-03-06T07:42:24.007 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61491.log 2024-03-06T07:42:24.007 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44390.log.gz 2024-03-06T07:42:24.008 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74530.log 2024-03-06T07:42:24.008 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77202.log.gz 2024-03-06T07:42:24.009 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56157.log 2024-03-06T07:42:24.010 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61491.log.gz 2024-03-06T07:42:24.010 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61165.log 2024-03-06T07:42:24.010 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74530.log.gz 2024-03-06T07:42:24.011 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79939.log 2024-03-06T07:42:24.011 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56157.log.gz 2024-03-06T07:42:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83260.log 2024-03-06T07:42:24.012 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61165.log.gz 2024-03-06T07:42:24.012 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76820.log 2024-03-06T07:42:24.013 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79939.log.gz 2024-03-06T07:42:24.013 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53703.log 2024-03-06T07:42:24.014 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83260.log.gz 2024-03-06T07:42:24.015 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83530.log 2024-03-06T07:42:24.015 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.76820.log.gz 2024-03-06T07:42:24.016 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74909.log 2024-03-06T07:42:24.016 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53703.log.gz 2024-03-06T07:42:24.017 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58030.log 2024-03-06T07:42:24.017 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83530.log.gz 2024-03-06T07:42:24.018 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.74909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65702.log 2024-03-06T07:42:24.018 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74909.log.gz 2024-03-06T07:42:24.019 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59914.log 2024-03-06T07:42:24.019 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58030.log.gz 2024-03-06T07:42:24.020 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77869.log 2024-03-06T07:42:24.020 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65702.log.gz 2024-03-06T07:42:24.020 INFO:teuthology.orchestra.run.smithi135.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:42:24.021 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69299.log 2024-03-06T07:42:24.021 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59914.log.gz 2024-03-06T07:42:24.022 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67591.log 2024-03-06T07:42:24.022 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77869.log.gz 2024-03-06T07:42:24.023 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44936.log 2024-03-06T07:42:24.023 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69299.log.gz 2024-03-06T07:42:24.023 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78420.log 2024-03-06T07:42:24.024 INFO:teuthology.orchestra.run.smithi097.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67591.log.gz 2024-03-06T07:42:24.024 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74656.log 2024-03-06T07:42:24.025 INFO:teuthology.orchestra.run.smithi097.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.44936.log.gz 2024-03-06T07:42:24.025 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47966.log 2024-03-06T07:42:24.025 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78420.log.gz 2024-03-06T07:42:24.026 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68581.log 2024-03-06T07:42:24.027 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47966.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.74656.log.gz 2024-03-06T07:42:24.027 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65082.log 2024-03-06T07:42:24.027 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47966.log.gz 2024-03-06T07:42:24.028 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60741.log 2024-03-06T07:42:24.028 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68581.log.gz 2024-03-06T07:42:24.029 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85556.log 2024-03-06T07:42:24.030 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65082.log.gz 2024-03-06T07:42:24.030 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78941.log 2024-03-06T07:42:24.030 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60741.log.gz 2024-03-06T07:42:24.031 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77995.log 2024-03-06T07:42:24.031 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85556.log.gz 2024-03-06T07:42:24.032 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58612.log 2024-03-06T07:42:24.033 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78941.log.gz 2024-03-06T07:42:24.033 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66138.log 2024-03-06T07:42:24.033 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77995.log.gz 2024-03-06T07:42:24.034 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73883.log 2024-03-06T07:42:24.034 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58612.log.gz 2024-03-06T07:42:24.035 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71331.log 2024-03-06T07:42:24.035 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66138.log.gz 2024-03-06T07:42:24.036 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60657.log 2024-03-06T07:42:24.036 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73883.log.gz 2024-03-06T07:42:24.037 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84079.log 2024-03-06T07:42:24.037 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71331.log.gz 2024-03-06T07:42:24.038 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60657.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60657.log.gz 2024-03-06T07:42:24.038 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56873.log 2024-03-06T07:42:24.039 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66572.log 2024-03-06T07:42:24.040 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84079.log.gz 2024-03-06T07:42:24.040 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52247.log 2024-03-06T07:42:24.040 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56873.log.gz 2024-03-06T07:42:24.041 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86948.log 2024-03-06T07:42:24.041 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66572.log.gz 2024-03-06T07:42:24.042 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69233.log 2024-03-06T07:42:24.042 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52247.log.gz 2024-03-06T07:42:24.043 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80063.log 2024-03-06T07:42:24.043 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86948.log.gz 2024-03-06T07:42:24.044 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68539.log 2024-03-06T07:42:24.045 INFO:teuthology.orchestra.run.smithi097.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.69233.log.gz 2024-03-06T07:42:24.045 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82318.log 2024-03-06T07:42:24.046 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80063.log.gz 2024-03-06T07:42:24.046 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62168.log 2024-03-06T07:42:24.047 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68539.log.gz 2024-03-06T07:42:24.048 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48548.log 2024-03-06T07:42:24.048 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82318.log.gz 2024-03-06T07:42:24.049 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.62168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62168.log.gz 2024-03-06T07:42:24.049 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82108.log 2024-03-06T07:42:24.050 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61705.log 2024-03-06T07:42:24.050 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48548.log.gz 2024-03-06T07:42:24.051 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78378.log 2024-03-06T07:42:24.051 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82108.log.gz 2024-03-06T07:42:24.051 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77286.log 2024-03-06T07:42:24.051 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61705.log.gz 2024-03-06T07:42:24.052 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81460.log 2024-03-06T07:42:24.053 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78378.log.gz 2024-03-06T07:42:24.053 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77286.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77286.log.gz 2024-03-06T07:42:24.053 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46055.log 2024-03-06T07:42:24.054 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61873.log 2024-03-06T07:42:24.054 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81460.log.gz 2024-03-06T07:42:24.055 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74013.log 2024-03-06T07:42:24.055 INFO:teuthology.orchestra.run.smithi097.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46055.log.gz 2024-03-06T07:42:24.056 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77682.log 2024-03-06T07:42:24.057 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61873.log.gz 2024-03-06T07:42:24.058 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86383.log 2024-03-06T07:42:24.058 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74013.log.gz 2024-03-06T07:42:24.058 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44068.log 2024-03-06T07:42:24.059 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77682.log.gz 2024-03-06T07:42:24.059 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86383.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86383.log.gz 2024-03-06T07:42:24.060 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57167.log 2024-03-06T07:42:24.061 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57907.log 2024-03-06T07:42:24.061 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44068.log.gz 2024-03-06T07:42:24.061 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63085.log 2024-03-06T07:42:24.062 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57167.log.gz 2024-03-06T07:42:24.063 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77244.log 2024-03-06T07:42:24.063 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57907.log.gz 2024-03-06T07:42:24.063 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84824.log 2024-03-06T07:42:24.064 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63085.log.gz 2024-03-06T07:42:24.064 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77328.log 2024-03-06T07:42:24.064 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77244.log.gz 2024-03-06T07:42:24.066 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61981.log 2024-03-06T07:42:24.066 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84824.log.gz 2024-03-06T07:42:24.066 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77328.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77328.log.gz 2024-03-06T07:42:24.066 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80241.log 2024-03-06T07:42:24.067 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61440.log 2024-03-06T07:42:24.067 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61981.log.gz 2024-03-06T07:42:24.068 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50259.log 2024-03-06T07:42:24.068 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80241.log.gz 2024-03-06T07:42:24.069 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53913.log 2024-03-06T07:42:24.069 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61440.log.gz 2024-03-06T07:42:24.070 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75986.log 2024-03-06T07:42:24.070 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50259.log.gz 2024-03-06T07:42:24.071 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54662.log 2024-03-06T07:42:24.071 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53913.log.gz 2024-03-06T07:42:24.071 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75286.log 2024-03-06T07:42:24.072 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75986.log.gz 2024-03-06T07:42:24.072 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71269.log 2024-03-06T07:42:24.073 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54662.log.gz 2024-03-06T07:42:24.073 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54458.log 2024-03-06T07:42:24.073 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75286.log.gz 2024-03-06T07:42:24.074 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84530.log 2024-03-06T07:42:24.074 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71269.log.gz 2024-03-06T07:42:24.075 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86341.log 2024-03-06T07:42:24.076 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54458.log.gz 2024-03-06T07:42:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55891.log 2024-03-06T07:42:24.077 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84530.log.gz 2024-03-06T07:42:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86341.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.86341.log.gz 2024-03-06T07:42:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85262.log 2024-03-06T07:42:24.078 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78983.log 2024-03-06T07:42:24.079 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55891.log.gz 2024-03-06T07:42:24.079 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68879.log 2024-03-06T07:42:24.079 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85262.log.gz 2024-03-06T07:42:24.080 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78983.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.83008.log -- replaced with /var/log/ceph/ceph-client.admin.78983.log.gz 2024-03-06T07:42:24.080 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-06T07:42:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75734.log 2024-03-06T07:42:24.081 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68879.log.gz 2024-03-06T07:42:24.082 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82756.log 2024-03-06T07:42:24.082 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83008.log.gz 2024-03-06T07:42:24.083 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57041.log 2024-03-06T07:42:24.083 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75734.log.gz 2024-03-06T07:42:24.084 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75490.log 2024-03-06T07:42:24.084 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82756.log.gz 2024-03-06T07:42:24.085 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55724.log 2024-03-06T07:42:24.085 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57041.log.gz 2024-03-06T07:42:24.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82150.log 2024-03-06T07:42:24.086 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75490.log.gz 2024-03-06T07:42:24.087 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55724.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55724.log.gz 2024-03-06T07:42:24.087 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63928.log 2024-03-06T07:42:24.088 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43126.log 2024-03-06T07:42:24.089 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82150.log.gz 2024-03-06T07:42:24.089 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72959.log 2024-03-06T07:42:24.089 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63928.log.gz 2024-03-06T07:42:24.090 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77108.log 2024-03-06T07:42:24.090 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43126.log.gz 2024-03-06T07:42:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58906.log 2024-03-06T07:42:24.091 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72959.log.gz 2024-03-06T07:42:24.092 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74614.log 2024-03-06T07:42:24.092 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77108.log.gz 2024-03-06T07:42:24.093 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67381.log 2024-03-06T07:42:24.094 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.58906.log.gz 2024-03-06T07:42:24.095 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85868.log 2024-03-06T07:42:24.095 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74614.log.gz 2024-03-06T07:42:24.095 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69977.log 2024-03-06T07:42:24.095 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67381.log.gz 2024-03-06T07:42:24.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46895.log 2024-03-06T07:42:24.097 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85868.log.gz 2024-03-06T07:42:24.098 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60176.log 2024-03-06T07:42:24.098 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69977.log.gz 2024-03-06T07:42:24.098 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72749.log 2024-03-06T07:42:24.098 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46895.log.gz 2024-03-06T07:42:24.099 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60176.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.60176.log.gz 2024-03-06T07:42:24.099 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48488.log 2024-03-06T07:42:24.100 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58780.log 2024-03-06T07:42:24.101 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72749.log.gz 2024-03-06T07:42:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50031.log 2024-03-06T07:42:24.102 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48488.log.gz 2024-03-06T07:42:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55183.log 2024-03-06T07:42:24.103 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58780.log.gz 2024-03-06T07:42:24.103 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.50031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50031.log.gz 2024-03-06T07:42:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76982.log 2024-03-06T07:42:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68240.log 2024-03-06T07:42:24.105 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55183.log.gz 2024-03-06T07:42:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66386.log 2024-03-06T07:42:24.106 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76982.log.gz 2024-03-06T07:42:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56017.log 2024-03-06T07:42:24.106 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68240.log.gz 2024-03-06T07:42:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66386.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66386.log.gz 2024-03-06T07:42:24.107 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70154.log 2024-03-06T07:42:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77827.log 2024-03-06T07:42:24.109 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56017.log.gz 2024-03-06T07:42:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43075.log 2024-03-06T07:42:24.109 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70154.log.gz 2024-03-06T07:42:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73482.log 2024-03-06T07:42:24.111 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77827.log.gz 2024-03-06T07:42:24.111 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71940.log 2024-03-06T07:42:24.111 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43075.log.gz 2024-03-06T07:42:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79674.log 2024-03-06T07:42:24.112 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73482.log.gz 2024-03-06T07:42:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64012.log 2024-03-06T07:42:24.114 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71940.log.gz 2024-03-06T07:42:24.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76778.log 2024-03-06T07:42:24.115 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79674.log.gz 2024-03-06T07:42:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46811.log 2024-03-06T07:42:24.116 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64012.log.gz 2024-03-06T07:42:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68030.log 2024-03-06T07:42:24.117 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76778.log.gz 2024-03-06T07:42:24.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81375.log 2024-03-06T07:42:24.118 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46811.log.gz 2024-03-06T07:42:24.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69065.log 2024-03-06T07:42:24.119 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68030.log.gz 2024-03-06T07:42:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58356.log 2024-03-06T07:42:24.119 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81375.log.gz 2024-03-06T07:42:24.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47258.log 2024-03-06T07:42:24.121 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69065.log.gz 2024-03-06T07:42:24.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55309.log 2024-03-06T07:42:24.122 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58356.log.gz 2024-03-06T07:42:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49215.log 2024-03-06T07:42:24.122 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47258.log.gz 2024-03-06T07:42:24.123 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78462.log 2024-03-06T07:42:24.124 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55309.log.gz 2024-03-06T07:42:24.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43031.log 2024-03-06T07:42:24.124 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49215.log.gz 2024-03-06T07:42:24.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62875.log 2024-03-06T07:42:24.125 INFO:teuthology.orchestra.run.smithi097.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78462.log.gz 2024-03-06T07:42:24.126 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59516.log 2024-03-06T07:42:24.126 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43031.log.gz 2024-03-06T07:42:24.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62875.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62875.log.gz 2024-03-06T07:42:24.127 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85052.log 2024-03-06T07:42:24.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55225.log 2024-03-06T07:42:24.128 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59516.log.gz 2024-03-06T07:42:24.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71684.log 2024-03-06T07:42:24.130 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85052.log.gz 2024-03-06T07:42:24.130 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65077.log 2024-03-06T07:42:24.130 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55225.log.gz 2024-03-06T07:42:24.131 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42705.log 2024-03-06T07:42:24.131 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71684.log.gz 2024-03-06T07:42:24.132 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79596.log 2024-03-06T07:42:24.132 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65077.log.gz 2024-03-06T07:42:24.133 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50401.log 2024-03-06T07:42:24.133 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42705.log.gz 2024-03-06T07:42:24.134 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47924.log 2024-03-06T07:42:24.134 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79596.log.gz 2024-03-06T07:42:24.135 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47216.log 2024-03-06T07:42:24.135 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50401.log.gz 2024-03-06T07:42:24.136 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81880.log 2024-03-06T07:42:24.136 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47924.log.gz 2024-03-06T07:42:24.137 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49396.log 2024-03-06T07:42:24.137 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47216.log.gz 2024-03-06T07:42:24.138 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52645.log 2024-03-06T07:42:24.138 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81880.log.gz 2024-03-06T07:42:24.139 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82612.log 2024-03-06T07:42:24.139 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49396.log.gz 2024-03-06T07:42:24.140 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45984.log 2024-03-06T07:42:24.140 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52645.log.gz 2024-03-06T07:42:24.141 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82612.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82612.log.gz 2024-03-06T07:42:24.141 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86677.log 2024-03-06T07:42:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.45984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59830.log 2024-03-06T07:42:24.142 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45984.log.gz 2024-03-06T07:42:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81105.log 2024-03-06T07:42:24.143 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86677.log.gz 2024-03-06T07:42:24.144 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77412.log 2024-03-06T07:42:24.144 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59830.log.gz 2024-03-06T07:42:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76592.log 2024-03-06T07:42:24.145 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81105.log.gz 2024-03-06T07:42:24.146 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44055.log 2024-03-06T07:42:24.146 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77412.log.gz 2024-03-06T07:42:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43424.log 2024-03-06T07:42:24.147 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76592.log.gz 2024-03-06T07:42:24.148 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59578.log 2024-03-06T07:42:24.148 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44055.log.gz 2024-03-06T07:42:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54620.log 2024-03-06T07:42:24.149 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43424.log.gz 2024-03-06T07:42:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84445.log 2024-03-06T07:42:24.150 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59578.log.gz 2024-03-06T07:42:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54620.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54620.log.gz 2024-03-06T07:42:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49694.log 2024-03-06T07:42:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84445.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:42:24.152 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84445.log.gz 2024-03-06T07:42:24.153 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48713.log 2024-03-06T07:42:24.153 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49694.log.gz 2024-03-06T07:42:24.153 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81249.log 2024-03-06T07:42:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48713.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48713.log.gz 2024-03-06T07:42:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55057.log 2024-03-06T07:42:24.156 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54321.log 2024-03-06T07:42:24.156 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81249.log.gz 2024-03-06T07:42:24.157 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55057.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55057.log.gz 2024-03-06T07:42:24.157 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47132.log 2024-03-06T07:42:24.157 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:42:24.158 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86761.log 2024-03-06T07:42:24.158 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54321.log.gz 2024-03-06T07:42:24.159 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76028.log 2024-03-06T07:42:24.159 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47132.log.gz 2024-03-06T07:42:24.160 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86761.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86761.log.gz 2024-03-06T07:42:24.160 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80798.log 2024-03-06T07:42:24.161 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81670.log 2024-03-06T07:42:24.161 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76028.log.gz 2024-03-06T07:42:24.162 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52603.log 2024-03-06T07:42:24.162 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80798.log.gz 2024-03-06T07:42:24.163 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49005.log 2024-03-06T07:42:24.163 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81670.log.gz 2024-03-06T07:42:24.164 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64281.log 2024-03-06T07:42:24.164 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52603.log.gz 2024-03-06T07:42:24.165 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87956.log 2024-03-06T07:42:24.165 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49005.log.gz 2024-03-06T07:42:24.166 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67809.log 2024-03-06T07:42:24.166 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64281.log.gz 2024-03-06T07:42:24.166 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57690.log 2024-03-06T07:42:24.167 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87956.log.gz 2024-03-06T07:42:24.168 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51045.log 2024-03-06T07:42:24.168 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67809.log.gz 2024-03-06T07:42:24.169 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46307.log 2024-03-06T07:42:24.169 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57690.log.gz 2024-03-06T07:42:24.169 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47336.log 2024-03-06T07:42:24.170 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51045.log.gz 2024-03-06T07:42:24.171 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59279.log 2024-03-06T07:42:24.171 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46307.log.gz 2024-03-06T07:42:24.171 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44642.log 2024-03-06T07:42:24.172 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47336.log.gz 2024-03-06T07:42:24.172 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59279.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.44726.log 2024-03-06T07:42:24.173 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59279.log.gz 2024-03-06T07:42:24.173 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80157.log 2024-03-06T07:42:24.173 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44642.log.gz 2024-03-06T07:42:24.174 INFO:teuthology.orchestra.run.smithi135.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:42:24.175 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73925.log 2024-03-06T07:42:24.175 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44726.log.gz 2024-03-06T07:42:24.175 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60615.log 2024-03-06T07:42:24.176 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80157.log.gz 2024-03-06T07:42:24.176 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64407.log 2024-03-06T07:42:24.176 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73925.log.gz 2024-03-06T07:42:24.177 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72445.log 2024-03-06T07:42:24.177 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60615.log.gz 2024-03-06T07:42:24.178 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56611.log 2024-03-06T07:42:24.178 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64407.log.gz 2024-03-06T07:42:24.179 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76298.log 2024-03-06T07:42:24.179 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72445.log.gz 2024-03-06T07:42:24.180 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56611.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56611.log.gz 2024-03-06T07:42:24.180 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71075.log 2024-03-06T07:42:24.181 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63664.log 2024-03-06T07:42:24.181 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76298.log.gz 2024-03-06T07:42:24.181 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64962.log 2024-03-06T07:42:24.182 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71075.log.gz 2024-03-06T07:42:24.182 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58263.log 2024-03-06T07:42:24.183 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63664.log.gz 2024-03-06T07:42:24.183 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64962.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64962.log.gz 2024-03-06T07:42:24.184 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81838.log 2024-03-06T07:42:24.184 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55604.log 2024-03-06T07:42:24.184 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58263.log.gz 2024-03-06T07:42:24.185 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48008.log 2024-03-06T07:42:24.185 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81838.log.gz 2024-03-06T07:42:24.186 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.55604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48050.log 2024-03-06T07:42:24.186 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55604.log.gz 2024-03-06T07:42:24.187 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42979.log 2024-03-06T07:42:24.187 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48008.log.gz 2024-03-06T07:42:24.188 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51526.log 2024-03-06T07:42:24.188 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48050.log.gz 2024-03-06T07:42:24.189 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82192.log 2024-03-06T07:42:24.189 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42979.log.gz 2024-03-06T07:42:24.190 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66776.log 2024-03-06T07:42:24.190 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51526.log.gz 2024-03-06T07:42:24.191 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49526.log 2024-03-06T07:42:24.191 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82192.log.gz 2024-03-06T07:42:24.191 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73356.log 2024-03-06T07:42:24.192 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66776.log.gz 2024-03-06T07:42:24.193 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82444.log 2024-03-06T07:42:24.193 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49526.log.gz 2024-03-06T07:42:24.193 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73356.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73356.log.gz 2024-03-06T07:42:24.194 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81147.log 2024-03-06T07:42:24.194 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49778.log 2024-03-06T07:42:24.195 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82444.log.gz 2024-03-06T07:42:24.195 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49303.log 2024-03-06T07:42:24.196 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81147.log.gz 2024-03-06T07:42:24.196 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46685.log 2024-03-06T07:42:24.197 INFO:teuthology.orchestra.run.smithi097.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49778.log.gz 2024-03-06T07:42:24.197 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45853.log 2024-03-06T07:42:24.197 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49303.log.gz 2024-03-06T07:42:24.198 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55417.log 2024-03-06T07:42:24.198 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46685.log.gz 2024-03-06T07:42:24.199 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.2.45853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45853.log.gz 2024-03-06T07:42:24.199 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-03-06T07:42:24.200 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83572.log 2024-03-06T07:42:24.201 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55417.log.gz 2024-03-06T07:42:24.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51953.log 2024-03-06T07:42:24.201 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-03-06T07:42:24.202 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62791.log 2024-03-06T07:42:24.202 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83572.log.gz 2024-03-06T07:42:24.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51953.log.gz 2024-03-06T07:42:24.203 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51339.log 2024-03-06T07:42:24.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60867.log 2024-03-06T07:42:24.204 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62791.log.gz 2024-03-06T07:42:24.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68709.log 2024-03-06T07:42:24.205 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51339.log.gz 2024-03-06T07:42:24.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72529.log 2024-03-06T07:42:24.206 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60867.log.gz 2024-03-06T07:42:24.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75776.log 2024-03-06T07:42:24.207 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68709.log.gz 2024-03-06T07:42:24.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57251.log 2024-03-06T07:42:24.208 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72529.log.gz 2024-03-06T07:42:24.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54363.log 2024-03-06T07:42:24.209 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75776.log.gz 2024-03-06T07:42:24.210 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78037.log 2024-03-06T07:42:24.210 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57251.log.gz 2024-03-06T07:42:24.211 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57480.log 2024-03-06T07:42:24.211 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54363.log.gz 2024-03-06T07:42:24.212 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44066.log 2024-03-06T07:42:24.212 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78037.log.gz 2024-03-06T07:42:24.212 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76112.log 2024-03-06T07:42:24.213 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57480.log.gz 2024-03-06T07:42:24.213 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74572.log 2024-03-06T07:42:24.214 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44066.log.gz 2024-03-06T07:42:24.214 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53997.log 2024-03-06T07:42:24.215 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76112.log.gz 2024-03-06T07:42:24.215 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70443.log 2024-03-06T07:42:24.216 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74572.log.gz 2024-03-06T07:42:24.216 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74148.log 2024-03-06T07:42:24.216 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53997.log.gz 2024-03-06T07:42:24.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47840.log 2024-03-06T07:42:24.217 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70443.log.gz 2024-03-06T07:42:24.218 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54237.log 2024-03-06T07:42:24.218 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74148.log.gz 2024-03-06T07:42:24.219 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46475.log 2024-03-06T07:42:24.219 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47840.log.gz 2024-03-06T07:42:24.220 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85094.log 2024-03-06T07:42:24.220 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54237.log.gz 2024-03-06T07:42:24.221 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46475.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46475.log.gz 2024-03-06T07:42:24.221 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56095.log 2024-03-06T07:42:24.222 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44306.log 2024-03-06T07:42:24.222 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85094.log.gz 2024-03-06T07:42:24.223 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71878.log 2024-03-06T07:42:24.223 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56095.log.gz 2024-03-06T07:42:24.224 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85472.log 2024-03-06T07:42:24.224 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44306.log.gz 2024-03-06T07:42:24.225 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87872.log 2024-03-06T07:42:24.225 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71878.log.gz 2024-03-06T07:42:24.226 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81544.log 2024-03-06T07:42:24.226 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85472.log.gz 2024-03-06T07:42:24.227 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53829.log 2024-03-06T07:42:24.227 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87872.log.gz 2024-03-06T07:42:24.228 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42768.log 2024-03-06T07:42:24.228 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81544.log.gz 2024-03-06T07:42:24.229 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62959.log 2024-03-06T07:42:24.229 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53829.log.gz 2024-03-06T07:42:24.230 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69486.log 2024-03-06T07:42:24.230 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42768.log.gz 2024-03-06T07:42:24.231 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87242.log 2024-03-06T07:42:24.232 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62959.log.gz 2024-03-06T07:42:24.232 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.69486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69486.log.gz 2024-03-06T07:42:24.232 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62487.log 2024-03-06T07:42:24.233 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43168.log 2024-03-06T07:42:24.233 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87242.log.gz 2024-03-06T07:42:24.234 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82798.log 2024-03-06T07:42:24.234 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62487.log.gz 2024-03-06T07:42:24.235 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64641.log 2024-03-06T07:42:24.235 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43168.log.gz 2024-03-06T07:42:24.236 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80756.log 2024-03-06T07:42:24.236 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82798.log.gz 2024-03-06T07:42:24.237 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56406.log 2024-03-06T07:42:24.237 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64641.log.gz 2024-03-06T07:42:24.238 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76070.log 2024-03-06T07:42:24.238 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80756.log.gz 2024-03-06T07:42:24.239 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.56406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56406.log.gz 2024-03-06T07:42:24.239 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54969.log 2024-03-06T07:42:24.239 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43210.log 2024-03-06T07:42:24.240 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76070.log.gz 2024-03-06T07:42:24.240 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83950.log 2024-03-06T07:42:24.241 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54969.log.gz 2024-03-06T07:42:24.242 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54876.log 2024-03-06T07:42:24.242 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43210.log.gz 2024-03-06T07:42:24.243 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45553.log 2024-03-06T07:42:24.243 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83950.log.gz 2024-03-06T07:42:24.243 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63568.log 2024-03-06T07:42:24.244 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54876.log.gz 2024-03-06T07:42:24.244 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.45553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43732.log 2024-03-06T07:42:24.245 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45553.log.gz 2024-03-06T07:42:24.245 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46391.log 2024-03-06T07:42:24.246 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63568.log.gz 2024-03-06T07:42:24.246 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44474.log 2024-03-06T07:42:24.246 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43732.log.gz 2024-03-06T07:42:24.247 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72487.log 2024-03-06T07:42:24.247 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2024-03-06T07:42:24.248 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84572.log 2024-03-06T07:42:24.248 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44474.log.gz 2024-03-06T07:42:24.249 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74278.log 2024-03-06T07:42:24.249 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72487.log.gz 2024-03-06T07:42:24.250 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61396.log 2024-03-06T07:42:24.250 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.84572.log.gz 2024-03-06T07:42:24.251 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74278.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74278.log.gz 2024-03-06T07:42:24.251 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85826.log 2024-03-06T07:42:24.252 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84698.log 2024-03-06T07:42:24.252 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61396.log.gz 2024-03-06T07:42:24.253 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85910.log 2024-03-06T07:42:24.253 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85826.log.gz 2024-03-06T07:42:24.254 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75209.log 2024-03-06T07:42:24.254 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84698.log.gz 2024-03-06T07:42:24.255 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71789.log 2024-03-06T07:42:24.256 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85910.log.gz 2024-03-06T07:42:24.256 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52161.log 2024-03-06T07:42:24.256 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75209.log.gz 2024-03-06T07:42:24.257 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65238.log 2024-03-06T07:42:24.257 INFO:teuthology.orchestra.run.smithi097.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.71789.log.gz 2024-03-06T07:42:24.258 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52161.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:42:24.258 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52161.log.gz 2024-03-06T07:42:24.259 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69107.log 2024-03-06T07:42:24.259 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65238.log.gz 2024-03-06T07:42:24.260 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66900.log 2024-03-06T07:42:24.261 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53745.log 2024-03-06T07:42:24.261 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69107.log.gz 2024-03-06T07:42:24.262 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82024.log 2024-03-06T07:42:24.262 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66900.log.gz 2024-03-06T07:42:24.263 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59237.log 2024-03-06T07:42:24.263 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53745.log.gz 2024-03-06T07:42:24.264 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84163.log 2024-03-06T07:42:24.264 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82024.log.gz 2024-03-06T07:42:24.265 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73145.log 2024-03-06T07:42:24.265 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59237.log.gz 2024-03-06T07:42:24.266 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84163.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84163.log.gz 2024-03-06T07:42:24.266 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52687.log 2024-03-06T07:42:24.267 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62917.log 2024-03-06T07:42:24.268 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73145.log.gz 2024-03-06T07:42:24.268 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52687.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52687.log.gz 2024-03-06T07:42:24.269 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78132.log 2024-03-06T07:42:24.269 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87998.log 2024-03-06T07:42:24.269 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62917.log.gz 2024-03-06T07:42:24.270 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78132.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78132.log.gz 2024-03-06T07:42:24.270 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57606.log 2024-03-06T07:42:24.271 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72287.log 2024-03-06T07:42:24.272 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87998.log.gz 2024-03-06T07:42:24.272 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52771.log 2024-03-06T07:42:24.273 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57606.log.gz 2024-03-06T07:42:24.273 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54788.log 2024-03-06T07:42:24.274 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72287.log.gz 2024-03-06T07:42:24.275 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50961.log 2024-03-06T07:42:24.275 INFO:teuthology.orchestra.run.smithi097.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.52771.log.gz 2024-03-06T07:42:24.276 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77953.log 2024-03-06T07:42:24.276 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54788.log.gz 2024-03-06T07:42:24.277 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50961.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50961.log.gz 2024-03-06T07:42:24.277 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71393.log 2024-03-06T07:42:24.278 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61207.log 2024-03-06T07:42:24.278 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77953.log.gz 2024-03-06T07:42:24.279 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71393.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.71393.log.gz 2024-03-06T07:42:24.279 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.75366.log 2024-03-06T07:42:24.280 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85178.log 2024-03-06T07:42:24.281 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61207.log.gz 2024-03-06T07:42:24.281 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75366.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75366.log.gz 2024-03-06T07:42:24.282 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64491.log 2024-03-06T07:42:24.282 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55267.log 2024-03-06T07:42:24.283 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85178.log.gz 2024-03-06T07:42:24.284 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64491.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64491.log.gz 2024-03-06T07:42:24.284 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52034.log 2024-03-06T07:42:24.285 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78857.log 2024-03-06T07:42:24.285 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55267.log.gz 2024-03-06T07:42:24.286 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52034.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52034.log.gz 2024-03-06T07:42:24.286 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61533.log 2024-03-06T07:42:24.287 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60218.log 2024-03-06T07:42:24.288 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78857.log.gz 2024-03-06T07:42:24.288 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61533.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61533.log.gz 2024-03-06T07:42:24.288 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80199.log 2024-03-06T07:42:24.289 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54048.log 2024-03-06T07:42:24.290 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60218.log.gz 2024-03-06T07:42:24.291 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80199.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80199.log.gz 2024-03-06T07:42:24.291 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2024-03-06T07:42:24.292 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65331.log 2024-03-06T07:42:24.292 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54048.log.gz 2024-03-06T07:42:24.293 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52477.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2024-03-06T07:42:24.293 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87410.log 2024-03-06T07:42:24.294 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44810.log 2024-03-06T07:42:24.294 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65331.log.gz 2024-03-06T07:42:24.295 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87410.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87410.log.gz 2024-03-06T07:42:24.295 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82654.log 2024-03-06T07:42:24.296 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61084.log 2024-03-06T07:42:24.297 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44810.log.gz 2024-03-06T07:42:24.297 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82654.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82654.log.gz 2024-03-06T07:42:24.298 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82966.log 2024-03-06T07:42:24.298 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79025.log 2024-03-06T07:42:24.299 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61084.log.gz 2024-03-06T07:42:24.300 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82966.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82966.log.gz 2024-03-06T07:42:24.300 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44264.log 2024-03-06T07:42:24.301 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46769.log 2024-03-06T07:42:24.302 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44264.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79025.log.gz 2024-03-06T07:42:24.302 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44264.log.gz 2024-03-06T07:42:24.302 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79278.log 2024-03-06T07:42:24.303 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82696.log 2024-03-06T07:42:24.303 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46769.log.gz 2024-03-06T07:42:24.304 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.79278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73314.log 2024-03-06T07:42:24.304 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79278.log.gz 2024-03-06T07:42:24.305 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82696.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82696.log.gz 2024-03-06T07:42:24.305 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73657.log 2024-03-06T07:42:24.306 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46853.log 2024-03-06T07:42:24.307 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2%/var/log/ceph/ceph-client.admin.73657.log: -- replaced with /var/log/ceph/ceph-client.admin.73314.log.gz 2024-03-06T07:42:24.307 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73657.log.gz 2024-03-06T07:42:24.308 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57988.log 2024-03-06T07:42:24.308 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63844.log 2024-03-06T07:42:24.309 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57988.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46853.log.gz 2024-03-06T07:42:24.310 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57988.log.gz 2024-03-06T07:42:24.310 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83302.log 2024-03-06T07:42:24.311 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53272.log 2024-03-06T07:42:24.311 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63844.log.gz 2024-03-06T07:42:24.312 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83302.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83302.log.gz 2024-03-06T07:42:24.312 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73969.log 2024-03-06T07:42:24.313 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77580.log 2024-03-06T07:42:24.313 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53272.log.gz 2024-03-06T07:42:24.314 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58822.log 2024-03-06T07:42:24.315 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73969.log.gz 2024-03-06T07:42:24.316 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77580.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77580.log.gz 2024-03-06T07:42:24.316 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70055.log 2024-03-06T07:42:24.317 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63526.log 2024-03-06T07:42:24.317 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58822.log.gz 2024-03-06T07:42:24.318 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70055.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70055.log.gz 2024-03-06T07:42:24.318 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43298.log 2024-03-06T07:42:24.319 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44558.log 2024-03-06T07:42:24.319 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63526.log.gz 2024-03-06T07:42:24.320 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43298.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43298.log.gz 2024-03-06T07:42:24.320 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67423.log 2024-03-06T07:42:24.321 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67077.log 2024-03-06T07:42:24.322 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44558.log.gz 2024-03-06T07:42:24.322 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67423.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67423.log.gz 2024-03-06T07:42:24.323 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74362.log 2024-03-06T07:42:24.323 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62613.log 2024-03-06T07:42:24.324 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67077.log.gz 2024-03-06T07:42:24.325 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74362.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74362.log.gz 2024-03-06T07:42:24.325 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86719.log 2024-03-06T07:42:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72064.log 2024-03-06T07:42:24.326 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62613.log.gz 2024-03-06T07:42:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86719.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86719.log.gz 2024-03-06T07:42:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44057.log 2024-03-06T07:42:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75428.log 2024-03-06T07:42:24.329 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72064.log.gz 2024-03-06T07:42:24.329 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44057.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44057.log.gz 2024-03-06T07:42:24.330 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53398.log 2024-03-06T07:42:24.330 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67675.log 2024-03-06T07:42:24.331 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75428.log.gz 2024-03-06T07:42:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53398.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53398.log.gz 2024-03-06T07:42:24.332 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80325.log 2024-03-06T07:42:24.333 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87704.log 2024-03-06T07:42:24.333 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67675.log.gz/var/log/ceph/ceph-client.admin.80325.log: 2024-03-06T07:42:24.334 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80325.log.gz 2024-03-06T07:42:24.334 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83488.log 2024-03-06T07:42:24.335 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49173.log 2024-03-06T07:42:24.336 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87704.log.gz 2024-03-06T07:42:24.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83488.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83488.log.gz 2024-03-06T07:42:24.337 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52837.log 2024-03-06T07:42:24.337 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64828.log 2024-03-06T07:42:24.338 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52837.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49173.log.gz 2024-03-06T07:42:24.339 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52837.log.gz 2024-03-06T07:42:24.339 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61250.log 2024-03-06T07:42:24.340 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.64828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43731.log 2024-03-06T07:42:24.340 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64828.log.gz 2024-03-06T07:42:24.341 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61250.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61250.log.gz 2024-03-06T07:42:24.341 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49259.log 2024-03-06T07:42:24.342 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55975.log 2024-03-06T07:42:24.343 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43731.log.gz 2024-03-06T07:42:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49259.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49259.log.gz 2024-03-06T07:42:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85514.log 2024-03-06T07:42:24.344 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49438.log 2024-03-06T07:42:24.345 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55975.log.gz 2024-03-06T07:42:24.346 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85514.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85514.log.gz 2024-03-06T07:42:24.346 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86246.log 2024-03-06T07:42:24.347 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51736.log 2024-03-06T07:42:24.347 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49438.log.gz 2024-03-06T07:42:24.348 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86246.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86246.log.gz 2024-03-06T07:42:24.348 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51441.log 2024-03-06T07:42:24.349 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77725.log 2024-03-06T07:42:24.349 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51441.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51736.log.gz 2024-03-06T07:42:24.350 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51441.log.gz 2024-03-06T07:42:24.350 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81586.log 2024-03-06T07:42:24.351 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84276.log 2024-03-06T07:42:24.352 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77725.log.gz 2024-03-06T07:42:24.353 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81586.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81586.log.gz 2024-03-06T07:42:24.353 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64100.log 2024-03-06T07:42:24.354 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80840.log 2024-03-06T07:42:24.354 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84276.log.gz 2024-03-06T07:42:24.355 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64100.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64100.log.gz 2024-03-06T07:42:24.355 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85304.log 2024-03-06T07:42:24.356 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51568.log 2024-03-06T07:42:24.356 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80840.log.gz 2024-03-06T07:42:24.357 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85304.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85304.log.gz 2024-03-06T07:42:24.357 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82360.log 2024-03-06T07:42:24.358 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79452.log 2024-03-06T07:42:24.358 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51568.log.gz 2024-03-06T07:42:24.359 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85598.log 2024-03-06T07:42:24.360 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82360.log.gz 2024-03-06T07:42:24.361 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48963.log 2024-03-06T07:42:24.361 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79452.log.gz 2024-03-06T07:42:24.361 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86078.log 2024-03-06T07:42:24.362 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85598.log.gz 2024-03-06T07:42:24.363 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47504.log 2024-03-06T07:42:24.363 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48963.log.gz 2024-03-06T07:42:24.364 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86078.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86078.log.gz 2024-03-06T07:42:24.364 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53661.log 2024-03-06T07:42:24.365 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75944.log 2024-03-06T07:42:24.365 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47504.log.gz 2024-03-06T07:42:24.366 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66962.log 2024-03-06T07:42:24.366 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53661.log.gz 2024-03-06T07:42:24.367 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79736.log 2024-03-06T07:42:24.367 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75944.log.gz 2024-03-06T07:42:24.368 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82528.log 2024-03-06T07:42:24.368 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66962.log.gz 2024-03-06T07:42:24.369 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55682.log 2024-03-06T07:42:24.369 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79736.log.gz 2024-03-06T07:42:24.370 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81628.log 2024-03-06T07:42:24.370 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82528.log.gz 2024-03-06T07:42:24.371 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61621.log 2024-03-06T07:42:24.371 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55682.log.gz 2024-03-06T07:42:24.371 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81628.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.48921.log -- replaced with /var/log/ceph/ceph-client.admin.81628.log.gz 2024-03-06T07:42:24.372 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-06T07:42:24.373 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77066.log 2024-03-06T07:42:24.373 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61621.log.gz 2024-03-06T07:42:24.373 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60470.log 88.8% 2024-03-06T07:42:24.374 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48921.log.gz 2024-03-06T07:42:24.375 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57293.log 2024-03-06T07:42:24.375 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77066.log.gz 2024-03-06T07:42:24.376 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60470.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:42:24.376 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60470.log.gz 2024-03-06T07:42:24.377 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50506.log 2024-03-06T07:42:24.377 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57293.log.gz 2024-03-06T07:42:24.378 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81417.log 2024-03-06T07:42:24.379 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63169.log 2024-03-06T07:42:24.379 INFO:teuthology.orchestra.run.smithi097.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50506.log.gz 2024-03-06T07:42:24.380 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68415.log 2024-03-06T07:42:24.381 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81417.log.gz 2024-03-06T07:42:24.381 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55933.log 2024-03-06T07:42:24.382 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63169.log.gz 2024-03-06T07:42:24.383 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60783.log 2024-03-06T07:42:24.383 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68415.log.gz 2024-03-06T07:42:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49047.log 2024-03-06T07:42:24.384 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55933.log.gz 2024-03-06T07:42:24.385 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46433.log 2024-03-06T07:42:24.386 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60783.log.gz 2024-03-06T07:42:24.387 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82882.log 2024-03-06T07:42:24.387 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49047.log.gz 2024-03-06T07:42:24.388 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58972.log 2024-03-06T07:42:24.388 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46433.log.gz 2024-03-06T07:42:24.389 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75902.log 2024-03-06T07:42:24.389 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82882.log.gz 2024-03-06T07:42:24.390 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88040.log 2024-03-06T07:42:24.391 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58972.log.gz 2024-03-06T07:42:24.392 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86864.log 2024-03-06T07:42:24.392 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75902.log.gz 2024-03-06T07:42:24.393 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.88040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85640.log 2024-03-06T07:42:24.393 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88040.log.gz 2024-03-06T07:42:24.394 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46097.log 2024-03-06T07:42:24.395 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86864.log.gz 2024-03-06T07:42:24.395 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83782.log 2024-03-06T07:42:24.396 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85640.log.gz 2024-03-06T07:42:24.397 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84361.log 2024-03-06T07:42:24.397 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2024-03-06T07:42:24.398 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56831.log 2024-03-06T07:42:24.398 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83782.log.gz 2024-03-06T07:42:24.399 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60386.log 2024-03-06T07:42:24.399 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84361.log.gz 2024-03-06T07:42:24.400 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83134.log 2024-03-06T07:42:24.401 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56831.log.gz 2024-03-06T07:42:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53577.log 2024-03-06T07:42:24.402 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60386.log.gz 2024-03-06T07:42:24.403 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49484.log 2024-03-06T07:42:24.403 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83134.log.gz 2024-03-06T07:42:24.404 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60092.log 2024-03-06T07:42:24.404 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53577.log.gz 2024-03-06T07:42:24.405 INFO:teuthology.orchestra.run.smithi135.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:42:24.406 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66634.log 2024-03-06T07:42:24.406 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49484.log.gz 2024-03-06T07:42:24.407 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69682.log 2024-03-06T07:42:24.407 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60092.log.gz 2024-03-06T07:42:24.408 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66634.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66634.log.gz 2024-03-06T07:42:24.409 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61292.log 2024-03-06T07:42:24.410 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54704.log 2024-03-06T07:42:24.410 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69682.log.gz 2024-03-06T07:42:24.411 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84205.log 2024-03-06T07:42:24.411 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61292.log.gz 2024-03-06T07:42:24.412 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48446.log 2024-03-06T07:42:24.412 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54704.log.gz 2024-03-06T07:42:24.413 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79091.log 2024-03-06T07:42:24.414 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84205.log.gz 2024-03-06T07:42:24.415 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77024.log 2024-03-06T07:42:24.415 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48446.log.gz 2024-03-06T07:42:24.416 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60573.log 2024-03-06T07:42:24.416 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79091.log.gz 2024-03-06T07:42:24.417 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58570.log 2024-03-06T07:42:24.417 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77024.log.gz 2024-03-06T07:42:24.418 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61004.log 2024-03-06T07:42:24.419 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60573.log.gz 2024-03-06T07:42:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84037.log 2024-03-06T07:42:24.420 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58570.log.gz 2024-03-06T07:42:24.421 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85952.log 2024-03-06T07:42:24.421 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61004.log.gz 2024-03-06T07:42:24.422 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66262.log 2024-03-06T07:42:24.422 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84037.log.gz 2024-03-06T07:42:24.423 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42934.log 2024-03-06T07:42:24.423 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85952.log.gz 2024-03-06T07:42:24.424 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84614.log 2024-03-06T07:42:24.425 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66262.log.gz 2024-03-06T07:42:24.426 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46559.log 2024-03-06T07:42:24.426 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42934.log.gz 2024-03-06T07:42:24.427 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83176.log 2024-03-06T07:42:24.427 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84614.log.gz 2024-03-06T07:42:24.428 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50604.log 2024-03-06T07:42:24.428 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46559.log.gz 2024-03-06T07:42:24.429 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46643.log 2024-03-06T07:42:24.430 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83176.log.gz 2024-03-06T07:42:24.430 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71190.log 2024-03-06T07:42:24.431 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50604.log.gz 2024-03-06T07:42:24.431 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68795.log 2024-03-06T07:42:24.432 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46643.log.gz 2024-03-06T07:42:24.433 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37861.log 2024-03-06T07:42:24.433 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71190.log.gz 2024-03-06T07:42:24.434 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65890.log 2024-03-06T07:42:24.434 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68795.log.gz 2024-03-06T07:42:24.435 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log 2024-03-06T07:42:24.435 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37861.log.gz 2024-03-06T07:42:24.436 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57209.log 2024-03-06T07:42:24.437 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65890.log.gz 2024-03-06T07:42:24.438 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43721.log 2024-03-06T07:42:24.438 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42624.log.gz 2024-03-06T07:42:24.439 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58115.log 2024-03-06T07:42:24.439 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57209.log.gz 2024-03-06T07:42:24.440 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83740.log 2024-03-06T07:42:24.440 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43721.log.gz 2024-03-06T07:42:24.441 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74320.log 2024-03-06T07:42:24.441 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58115.log.gz 2024-03-06T07:42:24.442 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72169.log 2024-03-06T07:42:24.442 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83740.log.gz 2024-03-06T07:42:24.443 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73085.log 2024-03-06T07:42:24.444 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74320.log.gz 2024-03-06T07:42:24.444 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67339.log 2024-03-06T07:42:24.445 INFO:teuthology.orchestra.run.smithi097.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.72169.log.gz 2024-03-06T07:42:24.446 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43382.log 2024-03-06T07:42:24.446 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73085.log.gz 2024-03-06T07:42:24.447 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54279.log 2024-03-06T07:42:24.447 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67339.log.gz 2024-03-06T07:42:24.448 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45404.log 2024-03-06T07:42:24.448 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43382.log.gz 2024-03-06T07:42:24.449 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54279.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.71013.log -- replaced with /var/log/ceph/ceph-client.admin.54279.log.gz 2024-03-06T07:42:24.449 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-06T07:42:24.450 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.45404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53955.log 2024-03-06T07:42:24.450 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45404.log.gz 2024-03-06T07:42:24.451 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51297.log 2024-03-06T07:42:24.452 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71013.log.gz 2024-03-06T07:42:24.453 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77370.log 2024-03-06T07:42:24.453 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53955.log.gz 2024-03-06T07:42:24.453 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51003.log 2024-03-06T07:42:24.454 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51297.log.gz 2024-03-06T07:42:24.455 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77370.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77370.log.gzgzip 2024-03-06T07:42:24.455 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56219.log 2024-03-06T07:42:24.456 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52393.log 2024-03-06T07:42:24.456 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51003.log.gz 2024-03-06T07:42:24.457 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64575.log 2024-03-06T07:42:24.457 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56219.log.gz 2024-03-06T07:42:24.458 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72833.log 2024-03-06T07:42:24.459 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52393.log.gz 2024-03-06T07:42:24.459 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82486.log 2024-03-06T07:42:24.460 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.64575.log.gz 2024-03-06T07:42:24.461 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84992.log 2024-03-06T07:42:24.461 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72833.log.gz 2024-03-06T07:42:24.462 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75818.log 2024-03-06T07:42:24.462 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82486.log.gz 2024-03-06T07:42:24.463 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58444.log 2024-03-06T07:42:24.463 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84992.log.gz 2024-03-06T07:42:24.464 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79407.log 2024-03-06T07:42:24.465 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75818.log.gz 2024-03-06T07:42:24.466 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72225.log 2024-03-06T07:42:24.466 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58444.log.gz 2024-03-06T07:42:24.467 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73043.log 2024-03-06T07:42:24.467 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79407.log.gz 2024-03-06T07:42:24.468 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72225.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72225.log.gz 2024-03-06T07:42:24.468 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48362.log 2024-03-06T07:42:24.469 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73043.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73043.log.gz 2024-03-06T07:42:24.469 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62655.log 2024-03-06T07:42:24.470 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79494.log 2024-03-06T07:42:24.471 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48362.log.gz 2024-03-06T07:42:24.471 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62655.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62655.log.gz 2024-03-06T07:42:24.471 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64323.log 2024-03-06T07:42:24.472 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75167.log 2024-03-06T07:42:24.473 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79494.log.gz 2024-03-06T07:42:24.474 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86162.log 2024-03-06T07:42:24.474 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64323.log.gz 2024-03-06T07:42:24.475 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85388.log 2024-03-06T07:42:24.475 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75167.log.gz 2024-03-06T07:42:24.476 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85682.log 2024-03-06T07:42:24.476 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86162.log.gz 2024-03-06T07:42:24.477 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79877.log 2024-03-06T07:42:24.477 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85388.log.gz 2024-03-06T07:42:24.478 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64365.log 2024-03-06T07:42:24.478 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85682.log.gz 2024-03-06T07:42:24.479 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52203.log 2024-03-06T07:42:24.480 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64365.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79877.log.gz 2024-03-06T07:42:24.480 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86467.log 2024-03-06T07:42:24.481 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64365.log.gz 2024-03-06T07:42:24.482 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52203.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52203.log.gz 2024-03-06T07:42:24.482 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55011.log 2024-03-06T07:42:24.483 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66696.log 2024-03-06T07:42:24.483 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86467.log.gz 2024-03-06T07:42:24.484 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67633.log 2024-03-06T07:42:24.484 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55011.log.gz 2024-03-06T07:42:24.485 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63315.log 2024-03-06T07:42:24.485 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66696.log.gz 2024-03-06T07:42:24.486 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61789.log 2024-03-06T07:42:24.486 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0%/var/log/ceph/ceph-client.admin.63315.log: -- replaced with /var/log/ceph/ceph-client.admin.67633.log.gz 2024-03-06T07:42:24.487 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48835.log 2024-03-06T07:42:24.488 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63315.log.gz 2024-03-06T07:42:24.489 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73398.log 2024-03-06T07:42:24.489 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61789.log.gz 2024-03-06T07:42:24.489 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70158.log 2024-03-06T07:42:24.490 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48835.log.gz 2024-03-06T07:42:24.491 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67119.log 2024-03-06T07:42:24.491 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73398.log.gz 2024-03-06T07:42:24.492 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75860.log 2024-03-06T07:42:24.492 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70158.log.gz 2024-03-06T07:42:24.493 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67119.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67119.log.gzgzip 2024-03-06T07:42:24.493 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87830.log 2024-03-06T07:42:24.494 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88100.log 2024-03-06T07:42:24.495 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75860.log.gz 2024-03-06T07:42:24.496 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50867.log 2024-03-06T07:42:24.496 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87830.log.gz 2024-03-06T07:42:24.497 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.88100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78212.log 2024-03-06T07:42:24.497 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88100.log.gz 2024-03-06T07:42:24.498 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78336.log 2024-03-06T07:42:24.498 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50867.log.gz 2024-03-06T07:42:24.499 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67245.log 2024-03-06T07:42:24.499 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78212.log.gz 2024-03-06T07:42:24.500 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63744.log 2024-03-06T07:42:24.501 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78336.log.gz 2024-03-06T07:42:24.501 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70738.log 2024-03-06T07:42:24.502 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67245.log.gz 2024-03-06T07:42:24.502 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63744.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:42:24.503 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63744.log.gz 2024-03-06T07:42:24.503 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.70738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70738.log.gz 2024-03-06T07:42:24.504 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48633.log 2024-03-06T07:42:24.505 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60134.log 2024-03-06T07:42:24.506 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50126.log 2024-03-06T07:42:24.507 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48633.log.gz 2024-03-06T07:42:24.508 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68072.log 2024-03-06T07:42:24.508 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60134.log.gz 2024-03-06T07:42:24.509 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43730.log 2024-03-06T07:42:24.509 INFO:teuthology.orchestra.run.smithi097.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50126.log.gz 2024-03-06T07:42:24.510 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48879.log 2024-03-06T07:42:24.510 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68072.log.gz 2024-03-06T07:42:24.511 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60260.log 2024-03-06T07:42:24.512 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43730.log.gz 2024-03-06T07:42:24.513 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60302.log 2024-03-06T07:42:24.513 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48879.log.gz 2024-03-06T07:42:24.514 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47678.log 2024-03-06T07:42:24.514 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60260.log.gz 2024-03-06T07:42:24.515 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52519.log 2024-03-06T07:42:24.515 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60302.log.gz 2024-03-06T07:42:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76508.log 2024-03-06T07:42:24.516 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47678.log.gz 2024-03-06T07:42:24.518 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83824.log 2024-03-06T07:42:24.518 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52519.log.gz 2024-03-06T07:42:24.519 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48218.log 2024-03-06T07:42:24.519 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76508.log.gz 2024-03-06T07:42:24.520 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48320.log 2024-03-06T07:42:24.520 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83824.log.gz 2024-03-06T07:42:24.521 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80443.log 2024-03-06T07:42:24.521 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48218.log.gz 2024-03-06T07:42:24.522 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64449.log 2024-03-06T07:42:24.522 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48320.log.gz 2024-03-06T07:42:24.524 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57438.log 2024-03-06T07:42:24.524 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80443.log.gz 2024-03-06T07:42:24.525 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64449.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31419.log 2024-03-06T07:42:24.525 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64449.log.gz 2024-03-06T07:42:24.526 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53619.log 2024-03-06T07:42:24.526 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57438.log.gz 2024-03-06T07:42:24.527 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.tmp-client.admin.31419.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31419.log.gz 2024-03-06T07:42:24.527 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62392.log 2024-03-06T07:42:24.528 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86551.log 2024-03-06T07:42:24.528 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53619.log.gz 2024-03-06T07:42:24.529 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66510.log 2024-03-06T07:42:24.530 INFO:teuthology.orchestra.run.smithi097.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.62392.log.gz 2024-03-06T07:42:24.531 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84121.log 2024-03-06T07:42:24.531 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86551.log.gz 2024-03-06T07:42:24.532 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76154.log 2024-03-06T07:42:24.532 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66510.log.gz 2024-03-06T07:42:24.533 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47762.log 2024-03-06T07:42:24.533 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84121.log.gz 2024-03-06T07:42:24.534 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69809.log 2024-03-06T07:42:24.535 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76154.log.gz 2024-03-06T07:42:24.536 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65419.log 2024-03-06T07:42:24.536 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47762.log.gz 2024-03-06T07:42:24.537 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86635.log 2024-03-06T07:42:24.537 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69809.log.gz 2024-03-06T07:42:24.538 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81291.log 2024-03-06T07:42:24.539 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86635.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.65419.log.gz 2024-03-06T07:42:24.539 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83656.log 2024-03-06T07:42:24.539 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86635.log.gz 2024-03-06T07:42:24.540 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47090.log 2024-03-06T07:42:24.540 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81291.log.gz 2024-03-06T07:42:24.542 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50783.log 2024-03-06T07:42:24.542 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83656.log.gz 2024-03-06T07:42:24.543 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74446.log 2024-03-06T07:42:24.543 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47090.log.gz 2024-03-06T07:42:24.544 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83698.log 2024-03-06T07:42:24.544 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50783.log.gz 2024-03-06T07:42:24.545 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71622.log 2024-03-06T07:42:24.545 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74446.log.gz 2024-03-06T07:42:24.546 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72126.log 2024-03-06T07:42:24.546 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83698.log.gz 2024-03-06T07:42:24.548 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81922.log 2024-03-06T07:42:24.548 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71622.log.gz 2024-03-06T07:42:24.549 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76634.log 2024-03-06T07:42:24.549 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72126.log.gz 2024-03-06T07:42:24.550 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47378.log 2024-03-06T07:42:24.550 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81922.log.gz 2024-03-06T07:42:24.551 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73001.log 2024-03-06T07:42:24.551 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76634.log.gz 2024-03-06T07:42:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66200.log 2024-03-06T07:42:24.557 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47378.log.gz 2024-03-06T07:42:24.558 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2024-03-06T07:42:24.558 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73001.log.gz 2024-03-06T07:42:24.558 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48404.log 2024-03-06T07:42:24.558 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66200.log.gz 2024-03-06T07:42:24.558 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76718.log 2024-03-06T07:42:24.558 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2024-03-06T07:42:24.559 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50825.log 2024-03-06T07:42:24.559 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48404.log.gz 2024-03-06T07:42:24.559 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58654.log 2024-03-06T07:42:24.559 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76718.log.gz 2024-03-06T07:42:24.560 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85346.log 2024-03-06T07:42:24.560 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50825.log.gz 2024-03-06T07:42:24.561 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76676.log 2024-03-06T07:42:24.561 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58654.log.gz 2024-03-06T07:42:24.562 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71542.log 2024-03-06T07:42:24.562 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85346.log.gz 2024-03-06T07:42:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70957.log 2024-03-06T07:42:24.563 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76676.log.gz 2024-03-06T07:42:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85784.log 2024-03-06T07:42:24.566 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71542.log.gz 2024-03-06T07:42:24.566 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83908.log 2024-03-06T07:42:24.567 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85784.log: gzip 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70957.log.gz -5 2024-03-06T07:42:24.567 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.73188.log 2024-03-06T07:42:24.567 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85784.log.gz 2024-03-06T07:42:24.568 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53787.log 2024-03-06T07:42:24.568 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83908.log.gz 2024-03-06T07:42:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82066.log 2024-03-06T07:42:24.569 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73188.log.gz 2024-03-06T07:42:24.570 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78731.log 2024-03-06T07:42:24.571 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53787.log.gz 2024-03-06T07:42:24.572 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47420.log 2024-03-06T07:42:24.572 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82066.log.gz 2024-03-06T07:42:24.573 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43256.log 2024-03-06T07:42:24.573 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78731.log.gz 2024-03-06T07:42:24.574 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46727.log 2024-03-06T07:42:24.574 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47420.log.gz 2024-03-06T07:42:24.575 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77622.log 2024-03-06T07:42:24.575 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43256.log.gz 2024-03-06T07:42:24.577 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46727.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46727.log.gz 2024-03-06T07:42:24.577 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54746.log 2024-03-06T07:42:24.578 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65828.log 88.8% 2024-03-06T07:42:24.578 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77622.log.gz 2024-03-06T07:42:24.579 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73799.log 2024-03-06T07:42:24.579 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54746.log.gz 2024-03-06T07:42:24.580 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69893.log 2024-03-06T07:42:24.581 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65828.log.gz 2024-03-06T07:42:24.581 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53535.log 2024-03-06T07:42:24.582 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73799.log.gz 2024-03-06T07:42:24.583 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69893.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:42:24.583 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69893.log.gz 2024-03-06T07:42:24.584 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48176.log 2024-03-06T07:42:24.584 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53535.log.gz 2024-03-06T07:42:24.585 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51694.log 2024-03-06T07:42:24.586 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44348.log 2024-03-06T07:42:24.587 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48176.log.gz 2024-03-06T07:42:24.588 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63358.log 2024-03-06T07:42:24.588 INFO:teuthology.orchestra.run.smithi097.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51694.log.gz 2024-03-06T07:42:24.589 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59640.log 2024-03-06T07:42:24.589 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44348.log.gz 2024-03-06T07:42:24.590 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72917.log 2024-03-06T07:42:24.591 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63358.log.gz 2024-03-06T07:42:24.593 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59640.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59640.log.gz 2024-03-06T07:42:24.593 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5/var/log/ceph/ceph-client.admin.72917.log: --verbose -- /var/log/ceph/ceph-client.admin.51778.log 2024-03-06T07:42:24.593 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72917.log.gz 2024-03-06T07:42:24.594 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81982.log 2024-03-06T07:42:24.595 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48092.log 2024-03-06T07:42:24.595 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51778.log.gz 2024-03-06T07:42:24.597 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81982.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81982.log.gz 2024-03-06T07:42:24.597 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84487.log 2024-03-06T07:42:24.598 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.48092.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48092.log.gz 2024-03-06T07:42:24.598 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78899.log 2024-03-06T07:42:24.599 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77496.log 2024-03-06T07:42:24.600 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84487.log.gz 2024-03-06T07:42:24.601 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47174.log 2024-03-06T07:42:24.601 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78899.log.gz 2024-03-06T07:42:24.602 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58219.log 2024-03-06T07:42:24.602 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77496.log.gz 2024-03-06T07:42:24.603 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47174.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47174.log.gz 2024-03-06T07:42:24.604 INFO:teuthology.orchestra.run.smithi097.stderr:gzip/var/log/ceph/ceph-client.admin.58219.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67759.log 2024-03-06T07:42:24.605 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58219.log.gz 2024-03-06T07:42:24.605 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69705.log 2024-03-06T07:42:24.607 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82840.log 2024-03-06T07:42:24.607 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67759.log.gz 2024-03-06T07:42:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69705.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.69705.log.gz 2024-03-06T07:42:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50741.log 2024-03-06T07:42:24.609 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82840.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82840.log.gz 2024-03-06T07:42:24.610 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72875.log 2024-03-06T07:42:24.611 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.50741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76550.log 2024-03-06T07:42:24.611 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50741.log.gz 2024-03-06T07:42:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72875.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72875.log.gz 2024-03-06T07:42:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62330.log 2024-03-06T07:42:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76550.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76550.log.gz 2024-03-06T07:42:24.615 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65766.log 2024-03-06T07:42:24.616 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74194.log 2024-03-06T07:42:24.616 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62330.log.gz 2024-03-06T07:42:24.617 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66014.log 2024-03-06T07:42:24.617 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65766.log.gz 2024-03-06T07:42:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74194.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74194.log.gz 2024-03-06T07:42:24.618 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82402.log 2024-03-06T07:42:24.620 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75083.log 2024-03-06T07:42:24.620 INFO:teuthology.orchestra.run.smithi097.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66014.log.gz 2024-03-06T07:42:24.621 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52351.log 2024-03-06T07:42:24.621 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82402.log.gz 2024-03-06T07:42:24.623 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.75083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49610.log 89.3% 2024-03-06T07:42:24.623 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75083.log.gz 2024-03-06T07:42:24.623 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52351.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52351.log.gz 2024-03-06T07:42:24.624 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70157.log 2024-03-06T07:42:24.625 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86036.log 2024-03-06T07:42:24.625 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49610.log.gz 2024-03-06T07:42:24.626 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51171.log 2024-03-06T07:42:24.627 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70157.log.gz 2024-03-06T07:42:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86036.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86036.log.gz 2024-03-06T07:42:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:gzip/var/log/ceph/ceph-client.admin.51171.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70551.log 2024-03-06T07:42:24.630 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51171.log.gz 2024-03-06T07:42:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65092.log 2024-03-06T07:42:24.631 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43466.log 2024-03-06T07:42:24.631 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70551.log.gz 2024-03-06T07:42:24.633 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.65092.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65092.log.gz 2024-03-06T07:42:24.634 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60344.log 2024-03-06T07:42:24.634 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.43466.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43466.log.gz 2024-03-06T07:42:24.634 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74987.log 2024-03-06T07:42:24.636 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57125.log 2024-03-06T07:42:24.636 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60344.log.gz 2024-03-06T07:42:24.637 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46601.log 2024-03-06T07:42:24.637 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74987.log.gz 2024-03-06T07:42:24.638 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57125.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57125.log.gz 2024-03-06T07:42:24.639 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46601.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46601.log.gz 2024-03-06T07:42:24.640 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81063.log 2024-03-06T07:42:24.641 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-03-06T07:42:24.643 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66838.log 2024-03-06T07:42:24.643 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81063.log.gz 2024-03-06T07:42:24.643 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72791.log 2024-03-06T07:42:24.644 INFO:teuthology.orchestra.run.smithi097.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-03-06T07:42:24.645 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79536.log 2024-03-06T07:42:24.645 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66838.log.gz 2024-03-06T07:42:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59788.log 2024-03-06T07:42:24.646 INFO:teuthology.orchestra.run.smithi097.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72791.log.gz 2024-03-06T07:42:24.647 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.79536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62529.log 2024-03-06T07:42:24.647 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79536.log.gz 2024-03-06T07:42:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59788.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58696.log 2024-03-06T07:42:24.649 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59788.log.gz 2024-03-06T07:42:24.650 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62529.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62529.log.gz 2024-03-06T07:42:24.650 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84740.log 2024-03-06T07:42:24.651 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87494.log 2024-03-06T07:42:24.651 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58696.log.gz 2024-03-06T07:42:24.653 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84740.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:42:24.653 INFO:teuthology.orchestra.run.smithi097.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.84740.log.gz 2024-03-06T07:42:24.654 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87914.log 2024-03-06T07:42:24.655 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87494.log.gz 2024-03-06T07:42:24.657 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.87914.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87914.log.gz 2024-03-06T07:42:24.657 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46979.log 2024-03-06T07:42:24.659 INFO:teuthology.orchestra.run.smithi097.stderr:gzip 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:42:24.659 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87536.log 2024-03-06T07:42:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66324.log 2024-03-06T07:42:24.660 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46979.log.gz 2024-03-06T07:42:24.661 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88140.log 2024-03-06T07:42:24.661 INFO:teuthology.orchestra.run.smithi097.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87536.log.gz 2024-03-06T07:42:24.662 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.66324.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66324.log.gz 2024-03-06T07:42:24.663 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58398.log 2024-03-06T07:42:24.664 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.88140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68497.log 2024-03-06T07:42:24.664 INFO:teuthology.orchestra.run.smithi097.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88140.log.gz 2024-03-06T07:42:24.665 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47462.log 2024-03-06T07:42:24.665 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58398.log.gz 2024-03-06T07:42:24.666 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68497.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68497.log.gz 2024-03-06T07:42:24.667 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60699.log 2024-03-06T07:42:24.668 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47462.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56957.log 2024-03-06T07:42:24.668 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47462.log.gz 2024-03-06T07:42:24.669 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61915.log 2024-03-06T07:42:24.670 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60699.log.gz 2024-03-06T07:42:24.670 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.56957.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:42:24.671 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56957.log.gz 2024-03-06T07:42:24.672 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.61915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37731.log 2024-03-06T07:42:24.672 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.2.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.61915.log.gz 2024-03-06T07:42:24.673 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85136.log 2024-03-06T07:42:24.675 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37731.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37731.log.gz 2024-03-06T07:42:24.675 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70895.log 2024-03-06T07:42:24.676 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85136.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85136.log.gz 2024-03-06T07:42:24.678 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70895.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70895.log.gz 2024-03-06T07:42:24.678 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68198.log 2024-03-06T07:42:24.680 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53024.log 2024-03-06T07:42:24.681 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68198.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68198.log.gz 2024-03-06T07:42:24.681 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54538.log 2024-03-06T07:42:24.682 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.53024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53024.log.gz 2024-03-06T07:42:24.683 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76196.log 2024-03-06T07:42:24.684 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54538.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54538.log.gz 2024-03-06T07:42:24.685 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.76196.log: -- /var/log/ceph/ceph-client.admin.81021.log 2024-03-06T07:42:24.686 INFO:teuthology.orchestra.run.smithi097.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76196.log.gz 2024-03-06T07:42:24.686 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53314.log 2024-03-06T07:42:24.688 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81021.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81021.log.gz 2024-03-06T07:42:24.689 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44768.log 2024-03-06T07:42:24.690 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53314.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53314.log.gz 2024-03-06T07:42:24.690 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:42:24.691 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44768.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44768.log.gz 2024-03-06T07:42:24.693 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mds.c.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:42:24.694 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77785.log 2024-03-06T07:42:24.695 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78773.log 2024-03-06T07:42:24.697 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77785.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77785.log.gz 2024-03-06T07:42:24.697 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78773.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78773.log.gz 2024-03-06T07:42:24.698 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73577.log 2024-03-06T07:42:24.699 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47594.log 2024-03-06T07:42:24.701 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73577.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73577.log.gz 2024-03-06T07:42:24.702 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47594.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47594.log.gz 2024-03-06T07:42:24.703 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73757.log 2024-03-06T07:42:24.705 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87452.log 2024-03-06T07:42:24.705 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73757.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73757.log.gz 2024-03-06T07:42:24.706 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51652.log 2024-03-06T07:42:24.707 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87452.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87452.log.gz 2024-03-06T07:42:24.708 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67904.log 2024-03-06T07:42:24.710 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51652.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51652.log.gz 2024-03-06T07:42:24.710 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.67904.log: /var/log/ceph/ceph-client.admin.83428.log 2024-03-06T07:42:24.711 INFO:teuthology.orchestra.run.smithi097.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67904.log.gz 2024-03-06T07:42:24.712 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63127.log 2024-03-06T07:42:24.714 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83428.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83428.log.gz 2024-03-06T07:42:24.714 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60050.log/var/log/ceph/ceph-client.admin.63127.log: 2024-03-06T07:42:24.714 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63127.log.gz 2024-03-06T07:42:24.715 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63970.log 2024-03-06T07:42:24.716 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60050.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60050.log.gz 2024-03-06T07:42:24.717 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73230.log 2024-03-06T07:42:24.718 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63970.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63970.log.gz 2024-03-06T07:42:24.719 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57732.log 2024-03-06T07:42:24.720 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.73230.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73230.log.gz 2024-03-06T07:42:24.721 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70155.log 2024-03-06T07:42:24.721 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.57732.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57732.log.gz 2024-03-06T07:42:24.722 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58157.log 2024-03-06T07:42:24.723 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70155.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70155.log.gz 2024-03-06T07:42:24.725 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58157.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58157.log.gz 2024-03-06T07:42:24.725 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80283.log 2024-03-06T07:42:24.726 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84656.log 2024-03-06T07:42:24.728 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80283.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80283.log.gz 2024-03-06T07:42:24.728 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72002.log 2024-03-06T07:42:24.729 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84656.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84656.log.gz 2024-03-06T07:42:24.730 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76238.log 2024-03-06T07:42:24.731 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72002.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72002.log.gz 2024-03-06T07:42:24.732 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53871.log 2024-03-06T07:42:24.733 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.76238.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76238.log.gz 2024-03-06T07:42:24.734 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68114.log 2024-03-06T07:42:24.735 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53871.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53871.log.gz 2024-03-06T07:42:24.735 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72707.log 2024-03-06T07:42:24.736 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68114.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68114.log.gz 2024-03-06T07:42:24.737 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80001.log 2024-03-06T07:42:24.738 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72707.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72707.log.gz 2024-03-06T07:42:24.740 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80001.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80001.log.gz 2024-03-06T07:42:24.740 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78685.log 2024-03-06T07:42:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69023.log 2024-03-06T07:42:24.743 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78685.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78685.log.gz 2024-03-06T07:42:24.744 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68981.log 2024-03-06T07:42:24.745 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69023.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69023.log.gz 2024-03-06T07:42:24.745 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44516.log 2024-03-06T07:42:24.746 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68981.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68981.log.gz 2024-03-06T07:42:24.747 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51213.log 2024-03-06T07:42:24.748 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44516.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44516.log.gz 2024-03-06T07:42:24.749 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70833.log 2024-03-06T07:42:24.750 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.51213.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51213.log.gz 2024-03-06T07:42:24.751 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67507.log 2024-03-06T07:42:24.752 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70833.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70833.log.gz 2024-03-06T07:42:24.753 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.67507.log: /var/log/ceph/ceph-client.admin.60825.log 2024-03-06T07:42:24.753 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67507.log.gz 2024-03-06T07:42:24.754 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62571.log 2024-03-06T07:42:24.755 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60825.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60825.log.gz 2024-03-06T07:42:24.756 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2024-03-06T07:42:24.759 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62571.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62571.log.gz 2024-03-06T07:42:24.762 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67161.log 2024-03-06T07:42:24.762 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2024-03-06T07:42:24.762 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68156.log 2024-03-06T07:42:24.762 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67161.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67161.log.gz 2024-03-06T07:42:24.762 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87326.log 2024-03-06T07:42:24.763 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68156.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68156.log.gz 2024-03-06T07:42:24.763 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63442.log 2024-03-06T07:42:24.764 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87326.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87326.log.gz 2024-03-06T07:42:24.766 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63442.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63442.log.gz 2024-03-06T07:42:24.766 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63886.log 2024-03-06T07:42:24.768 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78815.log 2024-03-06T07:42:24.769 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63886.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63886.log.gz 2024-03-06T07:42:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62833.log 2024-03-06T07:42:24.770 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78815.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78815.log.gz 2024-03-06T07:42:24.771 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78506.log 2024-03-06T07:42:24.772 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.62833.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62833.log.gz 2024-03-06T07:42:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78506.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78506.log.gz 2024-03-06T07:42:24.774 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84950.log 2024-03-06T07:42:24.776 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53493.log 2024-03-06T07:42:24.777 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84950.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84950.log.gz 2024-03-06T07:42:24.778 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67203.log 2024-03-06T07:42:24.779 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53493.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53493.log.gz 2024-03-06T07:42:24.780 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.67203.log: /var/log/ceph/ceph-client.admin.80882.log 2024-03-06T07:42:24.780 INFO:teuthology.orchestra.run.smithi097.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67203.log.gz 2024-03-06T07:42:24.781 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87788.log 2024-03-06T07:42:24.782 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80882.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80882.log.gz 2024-03-06T07:42:24.783 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84782.log 2024-03-06T07:42:24.784 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87788.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87788.log.gz 2024-03-06T07:42:24.786 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84782.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84782.log.gz 2024-03-06T07:42:24.786 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55141.log 2024-03-06T07:42:24.788 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58528.log 2024-03-06T07:42:24.789 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55141.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55141.log.gz 2024-03-06T07:42:24.790 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58528.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58528.log.gz 2024-03-06T07:42:24.791 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44684.log 2024-03-06T07:42:24.792 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77538.log 2024-03-06T07:42:24.794 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44684.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44684.log.gz 2024-03-06T07:42:24.794 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59956.log 2024-03-06T07:42:24.795 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77538.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77538.log.gz 2024-03-06T07:42:24.796 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69149.log 2024-03-06T07:42:24.797 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59956.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59956.log.gz 2024-03-06T07:42:24.798 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87578.log 2024-03-06T07:42:24.799 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.69149.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69149.log.gz 2024-03-06T07:42:24.799 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68753.log 2024-03-06T07:42:24.801 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87578.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87578.log.gz 2024-03-06T07:42:24.801 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78643.log 2024-03-06T07:42:24.802 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.68753.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68753.log.gz 2024-03-06T07:42:24.803 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42824.log 2024-03-06T07:42:24.804 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78643.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78643.log.gz 2024-03-06T07:42:24.805 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81333.log 2024-03-06T07:42:24.805 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42824.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42824.log.gz 2024-03-06T07:42:24.806 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49354.log 2024-03-06T07:42:24.807 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81333.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81333.log.gz 2024-03-06T07:42:24.808 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87368.log 2024-03-06T07:42:24.809 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.49354.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49354.log.gz 2024-03-06T07:42:24.810 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67465.log 2024-03-06T07:42:24.811 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87368.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87368.log.gz 2024-03-06T07:42:24.812 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86819.log 2024-03-06T07:42:24.813 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67465.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67465.log.gz 2024-03-06T07:42:24.814 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87284.log 2024-03-06T07:42:24.814 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86819.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86819.log.gz 2024-03-06T07:42:24.815 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52119.log 2024-03-06T07:42:24.816 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87284.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87284.log.gz 2024-03-06T07:42:24.817 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71118.log 2024-03-06T07:42:24.818 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.52119.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52119.log.gz 2024-03-06T07:42:24.819 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80924.log 2024-03-06T07:42:24.820 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.71118.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.71118.log.gz 2024-03-06T07:42:24.821 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83993.log 2024-03-06T07:42:24.822 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.80924.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80924.log.gz 2024-03-06T07:42:24.823 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74106.log 2024-03-06T07:42:24.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83993.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83993.log.gz 2024-03-06T07:42:24.824 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81712.log 2024-03-06T07:42:24.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74106.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74106.log.gz 2024-03-06T07:42:24.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2024-03-06T07:42:24.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.81712.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81712.log.gz 2024-03-06T07:42:24.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60428.log 2024-03-06T07:42:24.829 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.46517.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2024-03-06T07:42:24.830 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67862.log 2024-03-06T07:42:24.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.60428.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60428.log.gz 2024-03-06T07:42:24.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82570.log 2024-03-06T07:42:24.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.67862.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67862.log.gz 2024-03-06T07:42:24.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.82570.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82570.log.gz 2024-03-06T07:42:24.834 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85724.log 2024-03-06T07:42:24.835 INFO:teuthology.orchestra.run.smithi135.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:42:24.836 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72571.log 2024-03-06T07:42:24.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85724.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85724.log.gz 2024-03-06T07:42:24.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78294.log 2024-03-06T07:42:24.840 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72571.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72571.log.gz 2024-03-06T07:42:24.840 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59374.log 2024-03-06T07:42:24.841 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.78294.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78294.log.gz 2024-03-06T07:42:24.841 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86990.log 2024-03-06T07:42:24.843 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.59374.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59374.log.gz 2024-03-06T07:42:24.843 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86120.log 2024-03-06T07:42:24.844 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86990.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86990.log.gz 2024-03-06T07:42:24.845 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63229.log 2024-03-06T07:42:24.846 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86120.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86120.log.gz 2024-03-06T07:42:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58864.log 2024-03-06T07:42:24.847 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63229.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63229.log.gz 2024-03-06T07:42:24.848 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83614.log 2024-03-06T07:42:24.849 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.58864.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58864.log.gz 2024-03-06T07:42:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47720.log 2024-03-06T07:42:24.851 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.83614.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83614.log.gz 2024-03-06T07:42:24.853 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.47720.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47720.log.gz 2024-03-06T07:42:24.853 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63272.log 2024-03-06T07:42:24.856 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.63272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53356.log 2024-03-06T07:42:24.856 INFO:teuthology.orchestra.run.smithi097.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63272.log.gz 2024-03-06T07:42:24.857 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85994.log 2024-03-06T07:42:24.859 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.53356.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53356.log.gz 2024-03-06T07:42:24.859 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54111.log 2024-03-06T07:42:24.860 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.85994.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85994.log.gz 2024-03-06T07:42:24.861 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64235.log 2024-03-06T07:42:24.862 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.54111.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54111.log.gz 2024-03-06T07:42:24.863 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86204.log 2024-03-06T07:42:24.864 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64235.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64235.log.gz 2024-03-06T07:42:24.864 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:42:24.866 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86204.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86204.log.gz 2024-03-06T07:42:24.866 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56915.log 2024-03-06T07:42:24.869 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.56915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77911.log 2024-03-06T07:42:24.870 INFO:teuthology.orchestra.run.smithi097.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56915.log.gz 2024-03-06T07:42:24.871 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74404.log 2024-03-06T07:42:24.872 INFO:teuthology.orchestra.run.smithi097.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:42:24.874 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.77911.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77911.log.gz 2024-03-06T07:42:24.875 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.74404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55099.log 2024-03-06T07:42:24.875 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74404.log.gz 2024-03-06T07:42:24.876 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87662.log 2024-03-06T07:42:24.877 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.55099.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55099.log.gz 2024-03-06T07:42:24.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72403.log 2024-03-06T07:42:24.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87662.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87662.log.gz 2024-03-06T07:42:24.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.72403.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72403.log.gz 2024-03-06T07:42:24.880 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64193.log 2024-03-06T07:42:24.882 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84866.log 2024-03-06T07:42:24.883 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64193.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64193.log.gz 2024-03-06T07:42:24.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84866.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84866.log.gz 2024-03-06T07:42:24.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70485.log 2024-03-06T07:42:24.886 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84403.log 2024-03-06T07:42:24.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.70485.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.70485.log.gz 2024-03-06T07:42:24.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.84403.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84403.log.gz 2024-03-06T07:42:24.889 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64533.log 2024-03-06T07:42:24.891 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86906.log 2024-03-06T07:42:24.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.64533.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64533.log.gz 2024-03-06T07:42:24.893 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.86906.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86906.log.gz 2024-03-06T07:42:24.894 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44065.log 2024-03-06T07:42:24.896 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.44065.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44065.log.gz 2024-03-06T07:42:24.930 INFO:teuthology.orchestra.run.smithi135.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:42:24.974 INFO:teuthology.orchestra.run.smithi135.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:42:25.118 INFO:teuthology.orchestra.run.smithi135.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:42:25.123 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-06T07:42:25.123 INFO:teuthology.orchestra.run.smithi135.stderr:real 0m1.316s 2024-03-06T07:42:25.123 INFO:teuthology.orchestra.run.smithi135.stderr:user 0m5.048s 2024-03-06T07:42:25.123 INFO:teuthology.orchestra.run.smithi135.stderr:sys 0m0.551s 2024-03-06T07:42:25.762 INFO:teuthology.orchestra.run.smithi097.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:42:25.762 INFO:teuthology.orchestra.run.smithi097.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:42:25.972 INFO:teuthology.orchestra.run.smithi097.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:42:26.044 INFO:teuthology.orchestra.run.smithi097.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:42:27.457 INFO:teuthology.orchestra.run.smithi097.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:42:32.054 INFO:teuthology.orchestra.run.smithi097.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:42:32.059 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-06T07:42:32.059 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m8.254s 2024-03-06T07:42:32.060 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m15.862s 2024-03-06T07:42:32.060 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m3.386s 2024-03-06T07:42:32.061 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:42:32.062 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/remote/smithi097/log 2024-03-06T07:42:32.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:42:34.922 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/remote/smithi135/log 2024-03-06T07:42:34.923 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:42:35.664 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:42:35.675 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:42:35.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:42:35.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:42:35.830 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:42:35.839 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:42:35.839 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:42:35.846 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:42:35.892 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:42:35.899 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:42:35.901 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:42:35.902 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-06T07:42:35.902 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +1777us[+1777us] +/- 91ms 2024-03-06T07:42:35.902 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 7 377 14 -954us[-1000us] +/- 62ms 2024-03-06T07:42:35.902 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -46us[ -46us] +/- 62ms 2024-03-06T07:42:35.902 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:42:35.908 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:42:35.908 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-03-06T07:42:35.909 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +1753us[+1761us] +/- 91ms 2024-03-06T07:42:35.909 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -1010us[-1003us] +/- 62ms 2024-03-06T07:42:35.909 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -89us[ -82us] +/- 62ms 2024-03-06T07:42:35.909 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:42:35.910 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:42:35.918 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:42:35.918 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:42:35.927 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:42:36.176 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:42:36.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:42:36.492 INFO:teuthology.orchestra.run.smithi097.stdout:type=AVC msg=audit(1709709480.963:197): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1 2024-03-06T07:42:36.494 DEBUG:teuthology.task.selinux:ubuntu@smithi097.front.sepia.ceph.com has 1 denials 2024-03-06T07:42:36.494 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:42:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:42:36.606 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 166, in teardown self.get_new_denials() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 215, in get_new_denials raise SELinuxError(node=remote, teuthology.exceptions.SELinuxError: SELinux denials found on ubuntu@smithi097.front.sepia.ceph.com: ['type=AVC msg=audit(1709709480.963:197): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-06T07:42:36.614 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:42:36.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:42:36.641 INFO:teuthology.task.internal:Duration was 1476.864752 seconds 2024-03-06T07:42:36.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:42:36.656 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:42:36.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:42:36.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:42:36.825 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:42:36.830 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:42:37.167 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:42:37.168 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-03-06T07:42:37.168 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:42:37.228 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-03-06T07:42:37.229 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:42:37.305 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:42:37.305 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:42:37.313 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:42:37.409 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:42:37.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:42:37.415 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:42:37.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:42:37.707 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:42:37.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:42:37.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:42:37.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:42:37.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:42:37.923 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:42:38.027 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-03-06T07:42:38.030 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-03-06T07:42:38.082 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:42:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:42:38.130 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:42:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:42:38.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:42:38.195 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:42:38.196 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/remote/smithi097 2024-03-06T07:42:38.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:42:38.308 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075/remote/smithi135 2024-03-06T07:42:38.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:42:38.421 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:42:38.422 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:42:38.429 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:42:38.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:42:38.502 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:42:38.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:42:38.520 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:42:38.520 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:42:38.530 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:42:38.577 INFO:teuthology.orchestra.run.smithi097.stdout: 524321 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:42 /home/ubuntu/cephtest 2024-03-06T07:42:38.588 INFO:teuthology.orchestra.run.smithi135.stdout: 393945 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:42 /home/ubuntu/cephtest 2024-03-06T07:42:38.591 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:42:38.603 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:42:38.683 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi097.front.sepia.ceph.com: ['type=AVC msg=audit(1709709480.963:197): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-06T07:42:38.684 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:42:38.708 DEBUG:teuthology.nuke:shortname: smithi097 2024-03-06T07:42:38.708 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:42:38.736 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:46.256764', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArYvpsmBmeK/TBDiqVWIuoAATGQa8GFzU77A+6ELgW99JRI/oLRgHOskkANHygDldGYO6XxqDib3DK/40vuNkg='} 2024-03-06T07:42:38.740 DEBUG:teuthology.nuke:shortname: smithi135 2024-03-06T07:42:38.740 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:42:38.768 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:46.255259', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPfyFDKab2OwyrP2BRUX2aSe+lz0hRLCl1Bj+JetLDX2qnn7gI3+3BuLidL6e2F9CEQCxVSFwMDe8F0xW2Qays='} 2024-03-06T07:42:38.797 INFO:teuthology.orchestra.console.smithi097:Power off 2024-03-06T07:42:38.798 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:42:38.819 INFO:teuthology.orchestra.console.smithi135:Power off 2024-03-06T07:42:38.821 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:42:38.835 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-03-06T07:42:38.835 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:38.854 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2024-03-06T07:42:38.855 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:38.869 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-06T07:42:38.883 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-06T07:42:42.871 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:42.884 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:42.983 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-06T07:42:42.994 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-06T07:42:46.985 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:46.996 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:47.098 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-06T07:42:47.106 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2024-03-06T07:42:47.206 INFO:teuthology.orchestra.console.smithi135:Power off completed 2024-03-06T07:42:51.099 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:42:51.208 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-03-06T07:42:51.309 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-03-06T07:42:51.414 INFO:teuthology.run:Summary data: client.0-kernel-sha1: b47552b3c0e3993e47c2c21f2f0d35fc872144cc client.1-kernel-sha1: b47552b3c0e3993e47c2c21f2f0d35fc872144cc description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1476.864751815796 failure_reason: 'Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a03aeca5d3a44ccab5acc938c0e009a status: fail success: false 2024-03-06T07:42:51.414 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:42:51.492 INFO:teuthology.run:FAIL